migrating to space_cache=2 and btrfs userspace commands
We're currently considering switching to "space_cache=v2" with noatime mount options for my lab server-workstations running RAID5.
* One has 13TB of data/metadata in a bunch of 6TB and 2TB disks totalling 26TB.
* Another has about 12TB data/metadata in uniformly sized 6TB disks totalling 24TB.
* Both of the arrays are on individually luks encrypted disks with btrfs on top of the luks.
* Both have "defaults,autodefrag" turned on in fstab.
We're starting to see large pauses during constant backups of millions of chunk files (using duplicacy backup) in the 24TB array.
Pauses sometimes take up to 20+ seconds in frequencies after every ~30secs of the end of the last pause. "btrfs-transacti" process consistently shows up as the blocking process/thread locking up filesystem IO. IO gets into the RAID5 array via nfsd. There are no disk or btrfs errors recorded. scrub last finished yesterday successfully.
After doing some research around the internet, we've come to the consideration above as described. Unfortunately the official documentation isn't clear on the following.
Official documentation URL - https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs(5)
1. How to migrate from default space_cache=v1 to space_cache=v2? It talks about the reverse, from v2 to v1!
2. If we use space_cache=v2, is it indeed still the case that the "btrfs" command will NOT work with the filesystem? So will our "btrfs scrub start /mount/point/..." cron jobs FAIL? I'm guessing the btrfs command comes from btrfs-progs which is currently v5.4.1-2 amd64, is that correct?
3. Any other ideas on how we can get rid of those annoying pauses with large backups into the array?
Thanks in advance!
DP
Quoted 2 lines · DanglingPointer
We're currently considering switching to "space_cache=v2" with noatime mount options for my lab server-workstations running RAID5.
Btrfs RAID5 is unsafe due to its write-hole problem.
Quoted 9 lines
* One has 13TB of data/metadata in a bunch of 6TB and 2TB disks totalling 26TB.
* Another has about 12TB data/metadata in uniformly sized 6TB disks totalling 24TB.
* Both of the arrays are on individually luks encrypted disks with btrfs on top of the luks.
* Both have "defaults,autodefrag" turned on in fstab.
Quoted 8 lines
We're starting to see large pauses during constant backups of millions of chunk files (using duplicacy backup) in the 24TB array.
Pauses sometimes take up to 20+ seconds in frequencies after every ~30secs of the end of the last pause. "btrfs-transacti" process consistently shows up as the blocking process/thread locking up filesystem IO. IO gets into the RAID5 array via nfsd. There are no disk or btrfs errors recorded. scrub last finished yesterday successfully.
Please provide the "echo l > /proc/sysrq-trigger" output when such pause happens.
If you're using qgroup (may be enabled by things like snapper), it may be the cause, as qgroup does its accounting when committing transaction.
If one transaction is super large, it can cause such problem.
You can test if qgroup is enabled by:
# btrfs qgroup show -prce <mnt>
Quoted 10 lines
After doing some research around the internet, we've come to the consideration above as described. Unfortunately the official documentation isn't clear on the following.
Official documentation URL -
https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs(5)1. How to migrate from default space_cache=v1 to space_cache=v2? It talks about the reverse, from v2 to v1!
Just mount with "space_cache=v2".
Quoted 2 lines
2. If we use space_cache=v2, is it indeed still the case that the "btrfs" command will NOT work with the filesystem?
Why would you think "btrfs" won't work on a btrfs?
Thanks,
Qu
Quoted 11 lines
So will our
"btrfs scrub start /mount/point/..." cron jobs FAIL? I'm guessing the btrfs command comes from btrfs-progs which is currently v5.4.1-2 amd64, is that correct?
3. Any other ideas on how we can get rid of those annoying pauses with large backups into the array?Thanks in advance!
DP
Quoted 10 lines · Qu Wenruo
Quoted 2 lines · DanglingPointer
2. If we use space_cache=v2, is it indeed still the case that the "btrfs" command will NOT work with the filesystem?
Why would you think "btrfs" won't work on a btrfs?
Maybe this?
man 5 btrfs, space_cache includes:
The btrfs(8) command currently only has read-only support for v2. A
read-write command may be run on a v2 filesystem by clearing the cache, running the command, and then remounting with space_cache=v2.
Signature · 1 line
Chris Murphy
Quoted 12 lines · Chris Murphy
Quoted 10 lines · Qu Wenruo
Quoted 2 lines · DanglingPointer
2. If we use space_cache=v2, is it indeed still the case that the "btrfs" command will NOT work with the filesystem?
Why would you think "btrfs" won't work on a btrfs?
Quoted 8 lines
Maybe this?
man 5 btrfs,
space_cacheincludes:The btrfs(8) command currently only has read-only support for v2. A
read-write command may be run on a v2 filesystem by clearing the cache, running the command, and then remounting with space_cache=v2.
Oh, that's only for offline tools writing into the fs, namingly "btrfs check --repair" and "mkfs.btrfs -R"
And I believe that sentence is now out-of-date after btrfs-progs v4.19, which pulls all the support for write time free space tree (v2 space cache).
I'll soon send out a patch to fix that.
Thanks,
Qu
Yep that's what I'm referring to here: https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs(5)
Thanks for the prompt respone Qu!
So just to confirm, our scheduled cron jobs to scrub will still work with space_cache=v2?
Quoted 34 lines · Qu Wenruo
Quoted 12 lines · Chris Murphy
Quoted 10 lines · Qu Wenruo
Quoted 2 lines · DanglingPointer
2. If we use space_cache=v2, is it indeed still the case that the "btrfs" command will NOT work with the filesystem?
Why would you think "btrfs" won't work on a btrfs?
Quoted 8 lines
Maybe this?
man 5 btrfs,
space_cacheincludes:The btrfs(8) command currently only has read-only support for v2. A read-write command may be run on a v2 filesystem by clearing the cache, running the command, and then remounting with space_cache=v2.
Oh, that's only for offline tools writing into the fs, namingly "btrfs check --repair" and "mkfs.btrfs -R"
And I believe that sentence is now out-of-date after btrfs-progs v4.19, which pulls all the support for write time free space tree (v2 space cache).
I'll soon send out a patch to fix that.
Thanks,
Qu
Quoted 7 lines · DanglingPointer
Yep that's what I'm referring to here:
https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs(5)Thanks for the prompt respone Qu!
So just to confirm, our scheduled cron jobs to scrub will still work with space_cache=v2?
Yep. No worry.
Just to be more accurate, btrfs-progs has two parts:
- Ioctl related part
Things like "scrub", "subvolume", "filesystem", "device" and etc are all ioctl related commands.
They just call kernel ioctls to do the job.
For those commands, they are way more independent, and seldomly get affected by btrfs-progs version.
- Offline read/write part
Things like "mkfs.btrfs", "btrfs check", "btrfs-convert" are the main commands in this part.
They relies on the btrfs-progs implementation on how to read/write btrfs.
That's why we always recommend the latest btrfs-progs for "btrfs check".
Thanks,
Qu
Quoted 40 lines · Qu Wenruo
Quoted 36 lines · Chris Murphy
Quoted 13 lines
Quoted 10 lines · Qu Wenruo
Quoted 2 lines · DanglingPointer
2. If we use space_cache=v2, is it indeed still the case that the "btrfs" command will NOT work with the filesystem?
Why would you think "btrfs" won't work on a btrfs?
Quoted 8 lines
Maybe this?
man 5 btrfs,
space_cacheincludes:The btrfs(8) command currently only has read-only support for v2. A read-write command may be run on a v2 filesystem by clearing the cache, running the command, and then remounting with space_cache=v2.
Oh, that's only for offline tools writing into the fs, namingly "btrfs check --repair" and "mkfs.btrfs -R"
And I believe that sentence is now out-of-date after btrfs-progs v4.19, which pulls all the support for write time free space tree (v2 space cache).
I'll soon send out a patch to fix that.
Thanks,
Qu
a) "echo l > /proc/sysrq-trigger"
The backup finished today already unfortunately and we are unlikely to run it again until we get an outage to remount the array with the space_cache=v2 and noatime mount options. Thanks for the command, we'll definitely use it if/when it happens again on the next large migration of data.
b) "sudo btrfs qgroup show -prce" ........
$ ERROR: can't list qgroups: quotas not enabled
So looks like it isn't enabled.
File sizes are between: 1,048,576 bytes and 16,777,216 bytes (Duplicacy backup defaults)
What classifies as a transaction? Any/All writes done in a 30sec interval? If 100 unique files were written in 30secs, is that 1 transaction or 100 transactions? Millions of files of the size range above were backed up.
c) "Just mount with "space_cache=v2""
Ok so no need to "clear_cache" the v1 cache, right?
I wrote this in the fstab but hadn't remounted yet until I can get an outage....
..."btrfs defaults,autodefrag,clear_cache,space_cache=v2,noatime 0 2"
Thanks again for your help Qu!
Quoted 70 lines · Qu Wenruo
Quoted 2 lines · DanglingPointer
We're currently considering switching to "space_cache=v2" with noatime mount options for my lab server-workstations running RAID5.
Btrfs RAID5 is unsafe due to its write-hole problem.
Quoted 9 lines
* One has 13TB of data/metadata in a bunch of 6TB and 2TB disks totalling 26TB.
* Another has about 12TB data/metadata in uniformly sized 6TB disks totalling 24TB.
* Both of the arrays are on individually luks encrypted disks with btrfs on top of the luks.
* Both have "defaults,autodefrag" turned on in fstab.Quoted 8 lines
We're starting to see large pauses during constant backups of millions of chunk files (using duplicacy backup) in the 24TB array.
Pauses sometimes take up to 20+ seconds in frequencies after every ~30secs of the end of the last pause. "btrfs-transacti" process consistently shows up as the blocking process/thread locking up filesystem IO. IO gets into the RAID5 array via nfsd. There are no disk or btrfs errors recorded. scrub last finished yesterday successfully.
Please provide the "echo l >
/proc/sysrq-trigger" output when such pause happens.If you're using qgroup (may be enabled by things like snapper), it may be the cause, as qgroup does its accounting when committing transaction.
If one transaction is super large, it can cause such problem.
You can test if qgroup is enabled by:
# btrfs qgroup show -prce <mnt>Quoted 10 lines
After doing some research around the internet, we've come to the consideration above as described. Unfortunately the official documentation isn't clear on the following.
Official documentation URL -
https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs(5)1. How to migrate from default space_cache=v1 to space_cache=v2? It talks about the reverse, from v2 to v1!
Just mount with "space_cache=v2".
Quoted 2 lines
2. If we use space_cache=v2, is it indeed still the case that the "btrfs" command will NOT work with the filesystem?
Why would you think "btrfs" won't work on a btrfs?
Thanks,
QuQuoted 11 lines
So will our
"btrfs scrub start /mount/point/..." cron jobs FAIL? I'm guessing the btrfs command comes from btrfs-progs which is currently v5.4.1-2 amd64, is that correct?
3. Any other ideas on how we can get rid of those annoying pauses with large backups into the array?Thanks in advance!
DP
Quoted 7 lines · DanglingPointer
a) "echo l >
/proc/sysrq-trigger"The backup finished today already unfortunately and we are unlikely to run it again until we get an outage to remount the array with the space_cache=v2 and noatime mount options.
Thanks for the command, we'll definitely use it if/when it happens again on the next large migration of data.
Just to avoid confusion, after that command, "dmesg" output is still needed, as that's where sysrq put its output.
Quoted 7 lines
b) "sudo btrfs qgroup show -prce" ........
$ ERROR: can't list qgroups: quotas not enabledSo looks like it isn't enabled.
One less thing to bother.
Quoted 3 lines
File sizes are between: 1,048,576 bytes and 16,777,216 bytes (Duplicacy backup defaults)
Between 1~16MiB, thus tons of small files.
Btrfs is not really good at handling tons of small files, as they generate a lot of metadata.
That may contribute to the hang.
Quoted 2 lines
What classifies as a transaction?
It's a little complex.
Technically it's a check point where before the checkpoint, all you see is old data, after the checkpoint, all you see is new data.
To end users, any data and metadata write will be included into one transaction (with proper dependency handled).
One way to finish (or commit) current transaction is to sync the fs, using "sync" command (sync all filesystems).
Quoted 2 lines
Any/All writes done in a 30sec
interval?
This the default commit interval. Almost all fses will try to commit its data/metadata to disk after a configurable interval.
The default one is 30s. That's also one way to commit current transaction.
Quoted 2 lines
If 100 unique files were written in 30secs, is that 1 transaction or 100 transactions?
It depends. As things like syncfs() and subvolume/snapshot creation may try to commit transaction.
But without those special operations, just writing 100 unique files using buffered write, it would only start one transaction, and when the 30s interval get hit, the transaction will be committed to disk.
Quoted 2 lines
Millions of files of the size range
above were backed up.
The amount of files may not force a transaction commit, if it doesn't trigger enough memory pressure, or free space pressure.
Anyway, the "echo l" sysrq would help us to locate what's taking so long time.
Quoted 5 lines
c) "Just mount with "space_cache=v2""
Ok so no need to "
clear_cache" the v1 cache, right?
Yes, and "clear_cache" won't really remove all the v1 cache anyway.
Thus it doesn't help much.
The only way to fully clear v1 cache is by using "btrfs check --clear-space-cache v1" on a *unmounted* btrfs.
Quoted 2 lines
I wrote this in the fstab but hadn't remounted yet until I can get an outage....
IMHO if you really want to test if v2 would help, you can just remount, no need to wait for a break.
Thanks,
Qu
Quoted 75 lines
..."btrfs defaults,autodefrag,clear_cache,space_cache=v2,noatime 0 2 > Thanks again for your help Qu!
Quoted 70 lines · Qu Wenruo
Quoted 2 lines · DanglingPointer
We're currently considering switching to "space_cache=v2" with noatime mount options for my lab server-workstations running RAID5.
Btrfs RAID5 is unsafe due to its write-hole problem.
Quoted 9 lines
* One has 13TB of data/metadata in a bunch of 6TB and 2TB disks totalling 26TB.
* Another has about 12TB data/metadata in uniformly sized 6TB disks totalling 24TB.
* Both of the arrays are on individually luks encrypted disks with btrfs on top of the luks.
* Both have "defaults,autodefrag" turned on in fstab.Quoted 8 lines
We're starting to see large pauses during constant backups of millions of chunk files (using duplicacy backup) in the 24TB array.
Pauses sometimes take up to 20+ seconds in frequencies after every ~30secs of the end of the last pause. "btrfs-transacti" process consistently shows up as the blocking process/thread locking up filesystem IO. IO gets into the RAID5 array via nfsd. There are no disk or btrfs errors recorded. scrub last finished yesterday successfully.
Please provide the "echo l >
/proc/sysrq-trigger" output when such pause happens.If you're using qgroup (may be enabled by things like snapper), it may be the cause, as qgroup does its accounting when committing transaction.
If one transaction is super large, it can cause such problem.
You can test if qgroup is enabled by:
# btrfs qgroup show -prce <mnt>Quoted 10 lines
After doing some research around the internet, we've come to the consideration above as described. Unfortunately the official documentation isn't clear on the following.
Official documentation URL -
https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs(5)1. How to migrate from default space_cache=v1 to space_cache=v2? It talks about the reverse, from v2 to v1!
Just mount with "space_cache=v2".
Quoted 2 lines
2. If we use space_cache=v2, is it indeed still the case that the "btrfs" command will NOT work with the filesystem?
Why would you think "btrfs" won't work on a btrfs?
Thanks,
QuQuoted 11 lines
So will our
"btrfs scrub start /mount/point/..." cron jobs FAIL? I'm guessing the btrfs command comes from btrfs-progs which is currently v5.4.1-2 amd64, is that correct?
3. Any other ideas on how we can get rid of those annoying pauses with large backups into the array?Thanks in advance!
DP
Hi Qu,
Just updating here that setting the mount option "space_cache=v2" and "noatime" completely SOLVED the performance problem! Basically like night and day!
These are my full fstab mount options...
btrfs defaults,autodefrag,space_cache=v2,noatime 0 2
Perhaps defaulting the space_cache=v2 should be considered? Why default to v1, what's the value of v1?
So for conclusion, for large multi-terrabyte arrays (in my case RAID5s), setting space_cache=v2 and noatime massively increases performance and eliminates the large long pauses in frequent intervals by "btrfs-transacti" blocking all IO.
Thanks Qu for your help!
Quoted 170 lines · Qu Wenruo
Quoted 7 lines · DanglingPointer
a) "echo l >
/proc/sysrq-trigger"The backup finished today already unfortunately and we are unlikely to run it again until we get an outage to remount the array with the space_cache=v2 and noatime mount options.
Thanks for the command, we'll definitely use it if/when it happens again on the next large migration of data.Just to avoid confusion, after that command, "dmesg" output is still needed, as that's where sysrq put its output.
Quoted 7 lines
b) "sudo btrfs qgroup show -prce" ........
$ ERROR: can't list qgroups: quotas not enabledSo looks like it isn't enabled.
One less thing to bother.
Quoted 3 lines
File sizes are between: 1,048,576 bytes and 16,777,216 bytes (Duplicacy backup defaults)
Between 1~16MiB, thus tons of small files.
Btrfs is not really good at handling tons of small files, as they generate a lot of metadata.
That may contribute to the hang.
Quoted 2 lines
What classifies as a transaction?
It's a little complex.
Technically it's a check point where before the checkpoint, all you see is old data, after the checkpoint, all you see is new data.
To end users, any data and metadata write will be included into one transaction (with proper dependency handled).
One way to finish (or commit) current transaction is to sync the fs, using "sync" command (sync all filesystems).
Quoted 2 lines
Any/All writes done in a 30sec
interval?This the default commit interval. Almost all fses will try to commit its data/metadata to disk after a configurable interval.
The default one is 30s. That's also one way to commit current transaction.
Quoted 2 lines
If 100 unique files were written in 30secs, is that 1 transaction or 100 transactions?
It depends. As things like
syncfs()and subvolume/snapshot creation may try to commit transaction.But without those special operations, just writing 100 unique files using buffered write, it would only start one transaction, and when the 30s interval get hit, the transaction will be committed to disk.
Quoted 2 lines
Millions of files of the size range
above were backed up.The amount of files may not force a transaction commit, if it doesn't trigger enough memory pressure, or free space pressure.
Anyway, the "echo l" sysrq would help us to locate what's taking so long time.
Quoted 5 lines
c) "Just mount with "space_cache=v2""
Ok so no need to "
clear_cache" the v1 cache, right?Yes, and "
clear_cache" won't really remove all the v1 cache anyway.Thus it doesn't help much.
The only way to fully clear v1 cache is by using "btrfs check --clear-space-cache v1" on a *unmounted* btrfs.
Quoted 2 lines
I wrote this in the fstab but hadn't remounted yet until I can get an outage....
IMHO if you really want to test if v2 would help, you can just remount, no need to wait for a break.
Thanks,
QuQuoted 75 lines
..."btrfs defaults,autodefrag,clear_cache,space_cache=v2,noatime 0 2 > Thanks again for your help Qu!
Quoted 70 lines · Qu Wenruo
Quoted 2 lines · DanglingPointer
We're currently considering switching to "space_cache=v2" with noatime mount options for my lab server-workstations running RAID5.
Btrfs RAID5 is unsafe due to its write-hole problem.
Quoted 9 lines
* One has 13TB of data/metadata in a bunch of 6TB and 2TB disks totalling 26TB.
* Another has about 12TB data/metadata in uniformly sized 6TB disks totalling 24TB.
* Both of the arrays are on individually luks encrypted disks with btrfs on top of the luks.
* Both have "defaults,autodefrag" turned on in fstab.Quoted 8 lines
We're starting to see large pauses during constant backups of millions of chunk files (using duplicacy backup) in the 24TB array.
Pauses sometimes take up to 20+ seconds in frequencies after every ~30secs of the end of the last pause. "btrfs-transacti" process consistently shows up as the blocking process/thread locking up filesystem IO. IO gets into the RAID5 array via nfsd. There are no disk or btrfs errors recorded. scrub last finished yesterday successfully.
Please provide the "echo l >
/proc/sysrq-trigger" output when such pause happens.If you're using qgroup (may be enabled by things like snapper), it may be the cause, as qgroup does its accounting when committing transaction.
If one transaction is super large, it can cause such problem.
You can test if qgroup is enabled by:
# btrfs qgroup show -prce <mnt>Quoted 10 lines
After doing some research around the internet, we've come to the consideration above as described. Unfortunately the official documentation isn't clear on the following.
Official documentation URL -
https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs(5)1. How to migrate from default space_cache=v1 to space_cache=v2? It talks about the reverse, from v2 to v1!
Just mount with "space_cache=v2".
Quoted 2 lines
2. If we use space_cache=v2, is it indeed still the case that the "btrfs" command will NOT work with the filesystem?
Why would you think "btrfs" won't work on a btrfs?
Thanks,
QuQuoted 11 lines
So will our
"btrfs scrub start /mount/point/..." cron jobs FAIL? I'm guessing the btrfs command comes from btrfs-progs which is currently v5.4.1-2 amd64, is that correct?
3. Any other ideas on how we can get rid of those annoying pauses with large backups into the array?Thanks in advance!
DP
Just as a point of data, I have a 96 TB array with RAID1 data, and RAID1C3 metadata.
I made the switch to space_cache=v2 some time ago, and I remember it made a huge difference when I did so! (It was RAID1 metadata at the time, as RAID1C3 was not available at the time.)
However, I also tried a check with '--clear-space-cache v1' at the time, and after waiting a literal whole day without it completing, I gave up, canceled it, and put it back into production. Is a --clear-space-cache v1 operation expected to take so long on such a large file system?
Thanks!
--Joshua Villwock
Quoted 178 lines · DanglingPointer
Hi Qu,
Just updating here that setting the mount option "space_cache=v2" and "noatime" completely SOLVED the performance problem!
Basically like night and day!These are my full fstab mount options...
btrfs defaults,autodefrag,space_cache=v2,noatime 0 2
Perhaps defaulting the space_cache=v2 should be considered? Why default to v1, what's the value of v1?
So for conclusion, for large multi-terrabyte arrays (in my case RAID5s), setting space_cache=v2 and noatime massively increases performance and eliminates the large long pauses in frequent intervals by "btrfs-transacti" blocking all IO.
Thanks Qu for your help!
Quoted 157 lines · Qu Wenruo
Quoted 7 lines · DanglingPointer
a) "echo l >
/proc/sysrq-trigger"The backup finished today already unfortunately and we are unlikely to run it again until we get an outage to remount the array with the space_cache=v2 and noatime mount options.
Thanks for the command, we'll definitely use it if/when it happens again on the next large migration of data.Just to avoid confusion, after that command, "dmesg" output is still needed, as that's where sysrq put its output.
Quoted 5 lines
b) "sudo btrfs qgroup show -prce" ........
$ ERROR: can't list qgroups: quotas not enabledSo looks like it isn't enabled.
One less thing to bother.
Quoted 2 lines
File sizes are between: 1,048,576 bytes and 16,777,216 bytes (Duplicacy backup defaults)
Between 1~16MiB, thus tons of small files.
Btrfs is not really good at handling tons of small files, as they generate a lot of metadata.
That may contribute to the hang.
Quoted 1 line
What classifies as a transaction?
It's a little complex.
Technically it's a check point where before the checkpoint, all you see is old data, after the checkpoint, all you see is new data.
To end users, any data and metadata write will be included into one transaction (with proper dependency handled).
One way to finish (or commit) current transaction is to sync the fs, using "sync" command (sync all filesystems).
Quoted 2 lines
Any/All writes done in a 30sec
interval?This the default commit interval. Almost all fses will try to commit its data/metadata to disk after a configurable interval.
The default one is 30s. That's also one way to commit current > transaction.
Quoted 2 lines
If 100 unique files were written in 30secs, is that 1 transaction or 100 transactions?
It depends. As things like
syncfs()and subvolume/snapshot creation may try to commit transaction.But without those special operations, just writing 100 unique files using buffered write, it would only start one transaction, and when the 30s interval get hit, the transaction will be committed to disk.
Quoted 2 lines
Millions of files of the size range
above were backed up.The amount of files may not force a transaction commit, if it doesn't trigger enough memory pressure, or free space pressure.
Anyway, the "echo l" sysrq would help us to locate what's taking so long time.
Quoted 3 lines
c) "Just mount with "space_cache=v2""
Ok so no need to "
clear_cache" the v1 cache, right?Yes, and "
clear_cache" won't really remove all the v1 cache anyway.Thus it doesn't help much.
The only way to fully clear v1 cache is by using "btrfs check --clear-space-cache v1" on a *unmounted* btrfs.
Quoted 2 lines
I wrote this in the fstab but hadn't remounted yet until I can get an outage....
IMHO if you really want to test if v2 would help, you can just remount, no need to wait for a break.
Thanks,
QuQuoted 4 lines
..."btrfs defaults,autodefrag,clear_cache,space_cache=v2,noatime 0 2 > Thanks again for your help Qu!
On 14/7/21 2:59 pm, Qu Wenruo wrote:
On 2021/7/13 下午11:38, DanglingPointer wrote:
We're currently considering switching to "space_cache=v2" with noatime mount options for my lab server-workstations running RAID5.Btrfs RAID5 is unsafe due to its write-hole problem.
* One has 13TB of data/metadata in a bunch of 6TB and 2TB disks totalling 26TB.
* Another has about 12TB data/metadata in uniformly sized 6TB disks totalling 24TB.
* Both of the arrays are on individually luks encrypted disks with btrfs on top of the luks.
* Both have "defaults,autodefrag" turned on in fstab.We're starting to see large pauses during constant backups of millions of chunk files (using duplicacy backup) in the 24TB array.
Pauses sometimes take up to 20+ seconds in frequencies after every ~30secs of the end of the last pause. "btrfs-transacti" process consistently shows up as the blocking process/thread locking up filesystem IO. IO gets into the RAID5 array via nfsd. There are no >>>> disk or btrfs errors recorded. scrub last finished yesterday successfully.
Please provide the "echo l >
/proc/sysrq-trigger" output when such >>> pause happens.If you're using qgroup (may be enabled by things like snapper), it may be the cause, as qgroup does its accounting when committing >>> transaction.
If one transaction is super large, it can cause such problem.
You can test if qgroup is enabled by:
# btrfs qgroup show -prce <mnt>After doing some research around the internet, we've come to the consideration above as described. Unfortunately the official documentation isn't clear on the following.
Official documentation URL -
https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs(5)1. How to migrate from default space_cache=v1 to space_cache=v2? It talks about the reverse, from v2 to v1!
Just mount with "space_cache=v2".
2. If we use space_cache=v2, is it indeed still the case that the "btrfs" command will NOT work with the filesystem?
Why would you think "btrfs" won't work on a btrfs?
Thanks,
QuSo will our
"btrfs scrub start /mount/point/..." cron jobs FAIL? I'm guessing the btrfs command comes from btrfs-progs which is currently >>>> v5.4.1-2 amd64, is that correct?
3. Any other ideas on how we can get rid of those annoying pauses with large backups into the array?Thanks in advance!
DP
Hi Joshua, on that system where you tried to run the "--clear-space-cache v1", when you gave up, did you continue using "space_cache=v2" on it?
Here are some more questions to add for anyone who can help educate us:...
Why would someone want to clear the space_cache v1?
What's the value of clearing the previous version space_cache before using the new version?
Why "clear" and not just "delete"? Won't "deleting" the whole previous space_cache files, blocks, whatever in the filesystem be faster then doing whatever "clear" does?
Am I missing out on something by not attempting to clear the previous version space_cache before using the new v2 version?
Quoted 183 lines · Joshua
Just as a point of data, I have a 96 TB array with RAID1 data, and RAID1C3 metadata.
I made the switch to space_cache=v2 some time ago, and I remember it made a huge difference when I did so! (It was RAID1 metadata at the time, as RAID1C3 was not available at the time.)
However, I also tried a check with '--clear-space-cache v1' at the time, and after waiting a literal whole day without it completing, I gave up, canceled it, and put it back into production. Is a --clear-space-cache v1 operation expected to take so long on such a large file system?
Thanks!
--Joshua VillwockQuoted 168 lines · DanglingPointer
Hi Qu,
Just updating here that setting the mount option "space_cache=v2" and "noatime" completely SOLVED the performance problem!
Basically like night and day!These are my full fstab mount options...
btrfs defaults,autodefrag,space_cache=v2,noatime 0 2
Perhaps defaulting the space_cache=v2 should be considered? Why default to v1, what's the value of v1?
So for conclusion, for large multi-terrabyte arrays (in my case RAID5s), setting space_cache=v2 and noatime massively increases performance and eliminates the large long pauses in frequent intervals by "btrfs-transacti" blocking all IO.
Thanks Qu for your help!
Quoted 147 lines · Qu Wenruo
Quoted 7 lines · DanglingPointer
a) "echo l >
/proc/sysrq-trigger"The backup finished today already unfortunately and we are unlikely to run it again until we get an outage to remount the array with the space_cache=v2 and noatime mount options.
Thanks for the command, we'll definitely use it if/when it happens again on the next large migration of data.Just to avoid confusion, after that command, "dmesg" output is still needed, as that's where sysrq put its output.
Quoted 5 lines
b) "sudo btrfs qgroup show -prce" ........
$ ERROR: can't list qgroups: quotas not enabledSo looks like it isn't enabled.
One less thing to bother.
Quoted 2 lines
File sizes are between: 1,048,576 bytes and 16,777,216 bytes (Duplicacy backup defaults)
Between 1~16MiB, thus tons of small files.
Btrfs is not really good at handling tons of small files, as they generate a lot of metadata.
That may contribute to the hang.
Quoted 1 line
What classifies as a transaction?
It's a little complex.
Technically it's a check point where before the checkpoint, all you see is old data, after the checkpoint, all you see is new data.
To end users, any data and metadata write will be included into one transaction (with proper dependency handled).
One way to finish (or commit) current transaction is to sync the fs, using "sync" command (sync all filesystems).
Quoted 2 lines
Any/All writes done in a 30sec
interval?This the default commit interval. Almost all fses will try to commit its data/metadata to disk after a configurable interval.
The default one is 30s. That's also one way to commit current > transaction.
Quoted 2 lines
If 100 unique files were written in 30secs, is that 1 transaction or 100 transactions?
It depends. As things like
syncfs()and subvolume/snapshot creation may try to commit transaction.But without those special operations, just writing 100 unique files using buffered write, it would only start one transaction, and when the 30s interval get hit, the transaction will be committed to disk.
Quoted 2 lines
Millions of files of the size range
above were backed up.The amount of files may not force a transaction commit, if it doesn't trigger enough memory pressure, or free space pressure.
Anyway, the "echo l" sysrq would help us to locate what's taking so long time.
Quoted 3 lines
c) "Just mount with "space_cache=v2""
Ok so no need to "
clear_cache" the v1 cache, right?Yes, and "
clear_cache" won't really remove all the v1 cache anyway.Thus it doesn't help much.
The only way to fully clear v1 cache is by using "btrfs check --clear-space-cache v1" on a *unmounted* btrfs.
Quoted 2 lines
I wrote this in the fstab but hadn't remounted yet until I can get an outage....
IMHO if you really want to test if v2 would help, you can just remount, no need to wait for a break.
Thanks,
QuQuoted 4 lines
..."btrfs defaults,autodefrag,clear_cache,space_cache=v2,noatime 0 2 > Thanks again for your help Qu!
On 14/7/21 2:59 pm, Qu Wenruo wrote:
On 2021/7/13 下午11:38, DanglingPointer wrote:
We're currently considering switching to "space_cache=v2" with noatime mount options for my lab server-workstations running RAID5.Btrfs RAID5 is unsafe due to its write-hole problem.
* One has 13TB of data/metadata in a bunch of 6TB and 2TB disks totalling 26TB.
* Another has about 12TB data/metadata in uniformly sized 6TB disks totalling 24TB.
* Both of the arrays are on individually luks encrypted disks with btrfs on top of the luks.
* Both have "defaults,autodefrag" turned on in fstab.We're starting to see large pauses during constant backups of millions of chunk files (using duplicacy backup) in the 24TB array.
Pauses sometimes take up to 20+ seconds in frequencies after every ~30secs of the end of the last pause. "btrfs-transacti" process consistently shows up as the blocking process/thread locking up filesystem IO. IO gets into the RAID5 array via nfsd. There are no >>>> disk or btrfs errors recorded. scrub last finished yesterday successfully.
Please provide the "echo l >
/proc/sysrq-trigger" output when such >>> pause happens.If you're using qgroup (may be enabled by things like snapper), it may be the cause, as qgroup does its accounting when committing >>> transaction.
If one transaction is super large, it can cause such problem.
You can test if qgroup is enabled by:
# btrfs qgroup show -prce <mnt>After doing some research around the internet, we've come to the consideration above as described. Unfortunately the official documentation isn't clear on the following.
Official documentation URL -
https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs(5)1. How to migrate from default space_cache=v1 to space_cache=v2? It talks about the reverse, from v2 to v1!
Just mount with "space_cache=v2".
2. If we use space_cache=v2, is it indeed still the case that the "btrfs" command will NOT work with the filesystem?
Why would you think "btrfs" won't work on a btrfs?
Thanks,
QuSo will our
"btrfs scrub start /mount/point/..." cron jobs FAIL? I'm guessing the btrfs command comes from btrfs-progs which is currently >>>> v5.4.1-2 amd64, is that correct?
3. Any other ideas on how we can get rid of those annoying pauses with large backups into the array?Thanks in advance!
DP
Quoted 8 lines · DanglingPointer
Hi Joshua, on that system where you tried to run the "--clear-space-cache v1", when you gave up, did you continue using "space_cache=v2" on it?
Here are some more questions to add for anyone who can help educate us:...
Why would someone want to clear the
space_cachev1?
Because it takes up space and we won't be able to delete them.
Quoted 3 lines
What's the value of clearing the previous version
space_cachebefore using the new version?
AFAIK, just save some space and reduce the root tree size.
The v1 cache exists as special files inside root tree (not accessible by end users).
Their existence takes up space and fragments the file system (one file is normally around 64K, and we have one such v1 file for each block group, you can see how many small files it has now)
Quoted 4 lines
Why "clear" and not just "delete"? Won't "deleting" the whole previous
space_cachefiles, blocks, whatever in the filesystem be faster then doing whatever "clear" does?
Just bad naming, and properly from me.
Indeed "delete" would be more proper here.
And we're indeed deleting them in "btrfs check --clear-space-cache v1", that's also why it's so slow.
If you have 20T used space, then the it would be around 20,000 block groups, meaning 20,000 64K files inside root tree, and deleting them one by one, and each deletion will cause a new transaction, no wonder it will be slow to hell.
Quoted 3 lines
Am I missing out on something by not attempting to clear the previous version
space_cachebefore using the new v2 version?
Except some wasted space, you're completely fine to skip the slow deletion.
This also means, I should enhance the deletion process to avoid too many transactions...
Thanks,
Qu
Quoted 208 lines · Joshua
Quoted 205 lines
Just as a point of data, I have a 96 TB array with RAID1 data, and RAID1C3 metadata.
I made the switch to space_cache=v2 some time ago, and I remember it made a huge difference when I did so!
(It was RAID1 metadata at the time, as RAID1C3 was not available at the time.)However, I also tried a check with '--clear-space-cache v1' at the time, and after waiting a literal whole day without it completing, I gave up, canceled it, and put it back into production. Is a --clear-space-cache v1 operation expected to take so long on such a large file system?
Thanks!
--Joshua VillwockJuly 15, 2021 9:40 AM, "DanglingPointer"
Quoted 182 lines · <[email protected]> wrote:
Hi Qu,
Just updating here that setting the mount option "space_cache=v2" and "noatime" completely SOLVED
the performance problem!
Basically like night and day!These are my full fstab mount options...
btrfs defaults,autodefrag,space_cache=v2,noatime 0 2
Perhaps defaulting the space_cache=v2 should be considered? Why default to v1, what's the value of
v1?So for conclusion, for large multi-terrabyte arrays (in my case RAID5s), setting space_cache=v2 and
noatime massively increases performance and eliminates the large long pauses in frequent intervals
by "btrfs-transacti" blocking all IO.Thanks Qu for your help!
Quoted 157 lines · Qu Wenruo
Quoted 8 lines · DanglingPointer
a) "echo l >
/proc/sysrq-trigger"The backup finished today already unfortunately and we are unlikely to run it again until we get an outage to remount the array with the space_cache=v2 and noatime mount options.
Thanks for the command, we'll definitely use it if/when it happens again
on the next large migration of data.Just to avoid confusion, after that command, "dmesg" output is still needed, as that's where sysrq put its output.
Quoted 5 lines
b) "sudo btrfs qgroup show -prce" ........
$ ERROR: can't list qgroups: quotas not enabledSo looks like it isn't enabled.
One less thing to bother.
Quoted 3 lines
File sizes are between: 1,048,576 bytes and 16,777,216 bytes (Duplicacy
backup defaults)Between 1~16MiB, thus tons of small files.
Btrfs is not really good at handling tons of small files, as they generate a lot of metadata.
That may contribute to the hang.
Quoted 1 line
What classifies as a transaction?
It's a little complex.
Technically it's a check point where before the checkpoint, all you see is old data, after the checkpoint, all you see is new data.
To end users, any data and metadata write will be included into one transaction (with proper dependency handled).
One way to finish (or commit) current transaction is to sync the fs, using "sync" command (sync all filesystems).
Quoted 2 lines
Any/All writes done in a 30sec
interval?This the default commit interval. Almost all fses will try to commit its
data/metadata to disk after a configurable interval.The default one is 30s. That's also one way to commit current > transaction.
Quoted 2 lines
If 100 unique files were written in 30secs, is that 1 transaction or 100 transactions?
It depends. As things like
syncfs()and subvolume/snapshot creation may try to commit transaction.But without those special operations, just writing 100 unique files using buffered write, it would only start one transaction, and when the 30s interval get hit, the transaction will be committed to disk.
Quoted 2 lines
Millions of files of the size range
above were backed up.The amount of files may not force a transaction commit, if it doesn't trigger enough memory pressure, or free space pressure.
Anyway, the "echo l" sysrq would help us to locate what's taking so long
time.Quoted 3 lines
c) "Just mount with "space_cache=v2""
Ok so no need to "
clear_cache" the v1 cache, right?Yes, and "
clear_cache" won't really remove all the v1 cache anyway.Thus it doesn't help much.
The only way to fully clear v1 cache is by using "btrfs check --clear-space-cache v1" on a *unmounted* btrfs.
Quoted 2 lines
I wrote this in the fstab but hadn't remounted yet until I can get an outage....
IMHO if you really want to test if v2 would help, you can just remount, no need to wait for a break.
Thanks,
QuQuoted 5 lines
..."btrfs defaults,autodefrag,clear_cache,space_cache=v2,noatime 0 2 >
Thanks again for your help Qu!On 14/7/21 2:59 pm, Qu Wenruo wrote:
On 2021/7/13 下午11:38, DanglingPointer wrote:
We're currently considering switching to "space_cache=v2" with noatime mount options for my lab server-workstations running RAID5.Btrfs RAID5 is unsafe due to its write-hole problem.
* One has 13TB of data/metadata in a bunch of 6TB and 2TB disks totalling 26TB.
* Another has about 12TB data/metadata in uniformly sized 6TB disks totalling 24TB.
* Both of the arrays are on individually luks encrypted disks with btrfs on top of the luks.
* Both have "defaults,autodefrag" turned on in fstab.We're starting to see large pauses during constant backups of millions of chunk files (using duplicacy backup) in the 24TB array.
Pauses sometimes take up to 20+ seconds in frequencies after every ~30secs of the end of the last pause. "btrfs-transacti" process consistently shows up as the blocking process/thread locking up filesystem IO. IO gets into the RAID5 array via nfsd. There are no
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
disk
or btrfs errors recorded. scrub last finished yesterday successfully.
Please provide the "echo l >
/proc/sysrq-trigger" output when suchQuoted 1 line
Quoted 1 line
Quoted 1 line
pause
happens.
If you're using qgroup (may be enabled by things like snapper), it may be the cause, as qgroup does its accounting when committing >>> transaction.
If one transaction is super large, it can cause such problem.
You can test if qgroup is enabled by:
# btrfs qgroup show -prce <mnt>After doing some research around the internet, we've come to the consideration above as described. Unfortunately the official documentation isn't clear on the following.
Official documentation URL -
https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs(5)1. How to migrate from default space_cache=v1 to space_cache=v2? It talks about the reverse, from v2 to v1!
Just mount with "space_cache=v2".
2. If we use space_cache=v2, is it indeed still the case that the "btrfs" command will NOT work with the filesystem?
Why would you think "btrfs" won't work on a btrfs?
Thanks,
QuSo will our
"btrfs scrub start /mount/point/..." cron jobs FAIL? I'm guessing the btrfs command comes from btrfs-progs which is currently >>>> v5.4.1-2
amd64, is that correct?
3. Any other ideas on how we can get rid of those annoying pauses with large backups into the array?Thanks in advance!
DP
Thanks Qu for the comprehensive response!
Quoted 264 lines · Qu Wenruo
Quoted 8 lines · DanglingPointer
Hi Joshua, on that system where you tried to run the "--clear-space-cache v1", when you gave up, did you continue using "space_cache=v2" on it?
Here are some more questions to add for anyone who can help educate us:...
Why would someone want to clear the
space_cachev1?Because it takes up space and we won't be able to delete them.
Quoted 3 lines
What's the value of clearing the previous version
space_cachebefore using the new version?AFAIK, just save some space and reduce the root tree size.
The v1 cache exists as special files inside root tree (not accessible by end users).
Their existence takes up space and fragments the file system (one file is normally around 64K, and we have one such v1 file for each block group, you can see how many small files it has now)
Quoted 4 lines
Why "clear" and not just "delete"? Won't "deleting" the whole previous
space_cachefiles, blocks, whatever in the filesystem be faster then doing whatever "clear" does?Just bad naming, and properly from me.
Indeed "delete" would be more proper here.
And we're indeed deleting them in "btrfs check --clear-space-cache v1", that's also why it's so slow.
If you have 20T used space, then the it would be around 20,000 block groups, meaning 20,000 64K files inside root tree, and deleting them one by one, and each deletion will cause a new transaction, no wonder it will be slow to hell.
Quoted 3 lines
Am I missing out on something by not attempting to clear the previous version
space_cachebefore using the new v2 version?Except some wasted space, you're completely fine to skip the slow deletion.
This also means, I should enhance the deletion process to avoid too many transactions...
Thanks,
QuQuoted 208 lines · Joshua
Quoted 205 lines
Just as a point of data, I have a 96 TB array with RAID1 data, and RAID1C3 metadata.
I made the switch to space_cache=v2 some time ago, and I remember it made a huge difference when I did so!
(It was RAID1 metadata at the time, as RAID1C3 was not available at the time.)However, I also tried a check with '--clear-space-cache v1' at the time, and after waiting a literal whole day without it completing, I gave up, canceled it, and put it back into production. Is a --clear-space-cache v1 operation expected to take so long on such a large file system?
Thanks!
--Joshua VillwockJuly 15, 2021 9:40 AM, "DanglingPointer"
Quoted 182 lines · <[email protected]> wrote:
Hi Qu,
Just updating here that setting the mount option "space_cache=v2" and "noatime" completely SOLVED
the performance problem!
Basically like night and day!These are my full fstab mount options...
btrfs defaults,autodefrag,space_cache=v2,noatime 0 2
Perhaps defaulting the space_cache=v2 should be considered? Why default to v1, what's the value of
v1?So for conclusion, for large multi-terrabyte arrays (in my case RAID5s), setting space_cache=v2 and
noatime massively increases performance and eliminates the large long pauses in frequent intervals
by "btrfs-transacti" blocking all IO.Thanks Qu for your help!
Quoted 157 lines · Qu Wenruo
Quoted 8 lines · DanglingPointer
a) "echo l >
/proc/sysrq-trigger"The backup finished today already unfortunately and we are unlikely to run it again until we get an outage to remount the array with the space_cache=v2 and noatime mount options.
Thanks for the command, we'll definitely use it if/when it happens again
on the next large migration of data.Just to avoid confusion, after that command, "dmesg" output is still needed, as that's where sysrq put its output.
Quoted 5 lines
b) "sudo btrfs qgroup show -prce" ........
$ ERROR: can't list qgroups: quotas not enabledSo looks like it isn't enabled.
One less thing to bother.
Quoted 3 lines
File sizes are between: 1,048,576 bytes and 16,777,216 bytes (Duplicacy
backup defaults)Between 1~16MiB, thus tons of small files.
Btrfs is not really good at handling tons of small files, as they generate a lot of metadata.
That may contribute to the hang.
Quoted 1 line
What classifies as a transaction?
It's a little complex.
Technically it's a check point where before the checkpoint, all you see is old data, after the checkpoint, all you see is new data.
To end users, any data and metadata write will be included into one transaction (with proper dependency handled).
One way to finish (or commit) current transaction is to sync the fs, using "sync" command (sync all filesystems).
Quoted 2 lines
Any/All writes done in a 30sec
interval?This the default commit interval. Almost all fses will try to commit its
data/metadata to disk after a configurable interval.The default one is 30s. That's also one way to commit current > transaction.
Quoted 2 lines
If 100 unique files were written in 30secs, is that 1 transaction or 100 transactions?
It depends. As things like
syncfs()and subvolume/snapshot creation may try to commit transaction.But without those special operations, just writing 100 unique files using buffered write, it would only start one transaction, and when the 30s interval get hit, the transaction will be committed to disk.
Quoted 2 lines
Millions of files of the size range
above were backed up.The amount of files may not force a transaction commit, if it doesn't trigger enough memory pressure, or free space pressure.
Anyway, the "echo l" sysrq would help us to locate what's taking so long
time.Quoted 3 lines
c) "Just mount with "space_cache=v2""
Ok so no need to "
clear_cache" the v1 cache, right?Yes, and "
clear_cache" won't really remove all the v1 cache anyway.Thus it doesn't help much.
The only way to fully clear v1 cache is by using "btrfs check --clear-space-cache v1" on a *unmounted* btrfs.
Quoted 2 lines
I wrote this in the fstab but hadn't remounted yet until I can get an outage....
IMHO if you really want to test if v2 would help, you can just remount, no need to wait for a break.
Thanks,
QuQuoted 5 lines
..."btrfs defaults,autodefrag,clear_cache,space_cache=v2,noatime 0 2 >
Thanks again for your help Qu!On 14/7/21 2:59 pm, Qu Wenruo wrote:
On 2021/7/13 下午11:38, DanglingPointer wrote:
We're currently considering switching to "space_cache=v2" with noatime mount options for my lab server-workstations running RAID5.Btrfs RAID5 is unsafe due to its write-hole problem.
* One has 13TB of data/metadata in a bunch of 6TB and 2TB disks totalling 26TB.
* Another has about 12TB data/metadata in uniformly sized 6TB disks totalling 24TB.
* Both of the arrays are on individually luks encrypted disks with btrfs on top of the luks.
* Both have "defaults,autodefrag" turned on in fstab.We're starting to see large pauses during constant backups of millions of chunk files (using duplicacy backup) in the 24TB array.
Pauses sometimes take up to 20+ seconds in frequencies after every ~30secs of the end of the last pause. "btrfs-transacti" process consistently shows up as the blocking process/thread locking up filesystem IO. IO gets into the RAID5 array via nfsd. There are no
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
disk
or btrfs errors recorded. scrub last finished yesterday successfully.
Please provide the "echo l >
/proc/sysrq-trigger" output when suchQuoted 1 line
Quoted 1 line
Quoted 1 line
pause
happens.
If you're using qgroup (may be enabled by things like snapper), it may be the cause, as qgroup does its accounting when committing >>> transaction.
If one transaction is super large, it can cause such problem.
You can test if qgroup is enabled by:
# btrfs qgroup show -prce <mnt>After doing some research around the internet, we've come to the consideration above as described. Unfortunately the official documentation isn't clear on the following.
Official documentation URL -
https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs(5)1. How to migrate from default space_cache=v1 to space_cache=v2? It talks about the reverse, from v2 to v1!
Just mount with "space_cache=v2".
2. If we use space_cache=v2, is it indeed still the case that the "btrfs" command will NOT work with the filesystem?
Why would you think "btrfs" won't work on a btrfs?
Thanks,
QuSo will our
"btrfs scrub start /mount/point/..." cron jobs FAIL? I'm guessing the btrfs command comes from btrfs-progs which is currently >>>> v5.4.1-2
amd64, is that correct?
3. Any other ideas on how we can get rid of those annoying pauses with large backups into the array?Thanks in advance!
DP
Qu, thanks indeed for your responses!
Quoted 38 lines · Qu Wenruo
Quoted 9 lines · DanglingPointer
Hi Joshua, on that system where you tried to run the "--clear-space-cache v1", when you gave up, did you continue using "space_cache=v2" on it?
Here are some more questions to add for anyone who can help educate us:...
Why would someone want to clear the
space_cachev1?Because it takes up space and we won't be able to delete them.
Quoted 3 lines
What's the value of clearing the previous version
space_cachebefore using the new version?AFAIK, just save some space and reduce the root tree size.
The v1 cache exists as special files inside root tree (not accessible by end users).
Their existence takes up space and fragments the file system (one file is normally around 64K, and we have one such v1 file for each block group, you can see how many small files it has now)
Quoted 4 lines
Why "clear" and not just "delete"? Won't "deleting" the whole previous
space_cachefiles, blocks, whatever in the filesystem be faster then doing whatever "clear" does?Just bad naming, and properly from me.
Indeed "delete" would be more proper here.
Quoted 7 lines
And we're indeed deleting them in "btrfs check --clear-space-cache v1", that's also why it's so slow.
If you have 20T used space, then the it would be around 20,000 block groups, meaning 20,000 64K files inside root tree, and deleting them one by one, and each deletion will cause a new transaction, no wonder it will be slow to hell.
If v1 space cache exists as special files in the root tree, could those be a contributing factor to the issue I was having in the past with mounts taking too long and dropping to recovery mode?
I discovered running btrfs fi defrag on each of my subvolumes (for the subvolume tree/extent tree, not the files) every couple weeks has reduced the mount time enough I don’t run into that issue on my massive FS anymore.
Could actually getting rid of those thousands of now-useless v1 entries help reduce mount times on such a massive FS, or would that be completely unrelated?
Thanks,
-Joshua
Quoted 220 lines
Quoted 3 lines
Am I missing out on something by not attempting to clear the previous version
space_cachebefore using the new v2 version?Except some wasted space, you're completely fine to skip the slow deletion.
This also means, I should enhance the deletion process to avoid too many transactions...
Thanks,
QuQuoted 208 lines
Quoted 208 lines · Joshua
Just as a point of data, I have a 96 TB array with RAID1 data, and RAID1C3 metadata.
I made the switch to space_cache=v2 some time ago, and I remember it made a huge difference when I did so!
(It was RAID1 metadata at the time, as RAID1C3 was not available at the time.)However, I also tried a check with '--clear-space-cache v1' at the time, and after waiting a literal whole day without it completing, I gave up, canceled it, and put it back into production. Is a --clear-space-cache v1 operation expected to take so long on such a large file system?
Thanks!
--Joshua VillwockJuly 15, 2021 9:40 AM, "DanglingPointer"
Quoted 182 lines · <[email protected]> wrote:
Hi Qu,
Just updating here that setting the mount option "space_cache=v2" and "noatime" completely SOLVED
the performance problem!
Basically like night and day!These are my full fstab mount options...
btrfs defaults,autodefrag,space_cache=v2,noatime 0 2
Perhaps defaulting the space_cache=v2 should be considered? Why default to v1, what's the value of
v1?So for conclusion, for large multi-terrabyte arrays (in my case RAID5s), setting space_cache=v2 and
noatime massively increases performance and eliminates the large long pauses in frequent intervals
by "btrfs-transacti" blocking all IO.Thanks Qu for your help!
Quoted 157 lines · Qu Wenruo
Quoted 8 lines · DanglingPointer
a) "echo l >
/proc/sysrq-trigger"The backup finished today already unfortunately and we are unlikely to run it again until we get an outage to remount the array with the space_cache=v2 and noatime mount options.
Thanks for the command, we'll definitely use it if/when it happens again
on the next large migration of data.Just to avoid confusion, after that command, "dmesg" output is still needed, as that's where sysrq put its output.
Quoted 5 lines
b) "sudo btrfs qgroup show -prce" ........
$ ERROR: can't list qgroups: quotas not enabledSo looks like it isn't enabled.
One less thing to bother.
Quoted 3 lines
File sizes are between: 1,048,576 bytes and 16,777,216 bytes (Duplicacy
backup defaults)Between 1~16MiB, thus tons of small files.
Btrfs is not really good at handling tons of small files, as they generate a lot of metadata.
That may contribute to the hang.
Quoted 1 line
What classifies as a transaction?
It's a little complex.
Technically it's a check point where before the checkpoint, all you see is old data, after the checkpoint, all you see is new data.
To end users, any data and metadata write will be included into one transaction (with proper dependency handled).
One way to finish (or commit) current transaction is to sync the fs, using "sync" command (sync all filesystems).
Quoted 2 lines
Any/All writes done in a 30sec
interval?This the default commit interval. Almost all fses will try to commit its
data/metadata to disk after a configurable interval.The default one is 30s. That's also one way to commit current > transaction.
Quoted 2 lines
If 100 unique files were written in 30secs, is that 1 transaction or 100 transactions?
It depends. As things like
syncfs()and subvolume/snapshot creation may try to commit transaction.But without those special operations, just writing 100 unique files using buffered write, it would only start one transaction, and when the 30s interval get hit, the transaction will be committed to disk.
Quoted 2 lines
Millions of files of the size range
above were backed up.The amount of files may not force a transaction commit, if it doesn't trigger enough memory pressure, or free space pressure.
Anyway, the "echo l" sysrq would help us to locate what's taking so long
time.Quoted 3 lines
c) "Just mount with "space_cache=v2""
Ok so no need to "
clear_cache" the v1 cache, right?Yes, and "
clear_cache" won't really remove all the v1 cache anyway.Thus it doesn't help much.
The only way to fully clear v1 cache is by using "btrfs check --clear-space-cache v1" on a *unmounted* btrfs.
Quoted 2 lines
I wrote this in the fstab but hadn't remounted yet until I can get an outage....
IMHO if you really want to test if v2 would help, you can just remount, no need to wait for a break.
Thanks,
QuQuoted 5 lines
..."btrfs defaults,autodefrag,clear_cache,space_cache=v2,noatime 0 2 >
Thanks again for your help Qu!On 14/7/21 2:59 pm, Qu Wenruo wrote:
On 2021/7/13 下午11:38, DanglingPointer wrote:
We're currently considering switching to "space_cache=v2" with noatime mount options for my lab server-workstations running RAID5.Btrfs RAID5 is unsafe due to its write-hole problem.
* One has 13TB of data/metadata in a bunch of 6TB and 2TB disks totalling 26TB.
* Another has about 12TB data/metadata in uniformly sized 6TB disks totalling 24TB.
* Both of the arrays are on individually luks encrypted disks with btrfs on top of the luks.
* Both have "defaults,autodefrag" turned on in fstab.We're starting to see large pauses during constant backups of millions of chunk files (using duplicacy backup) in the 24TB array.
Pauses sometimes take up to 20+ seconds in frequencies after every ~30secs of the end of the last pause. "btrfs-transacti" process consistently shows up as the blocking process/thread locking up filesystem IO. IO gets into the RAID5 array via nfsd. There are no
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
disk
or btrfs errors recorded. scrub last finished yesterday successfully.
Please provide the "echo l >
/proc/sysrq-trigger" output when suchQuoted 1 line
Quoted 1 line
Quoted 1 line
pause
happens.
If you're using qgroup (may be enabled by things like snapper), it may be the cause, as qgroup does its accounting when committing >>> transaction.
If one transaction is super large, it can cause such problem.
You can test if qgroup is enabled by:
# btrfs qgroup show -prce <mnt>After doing some research around the internet, we've come to the consideration above as described. Unfortunately the official documentation isn't clear on the following.
Official documentation URL -
https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs(5)1. How to migrate from default space_cache=v1 to space_cache=v2? It talks about the reverse, from v2 to v1!
Just mount with "space_cache=v2".
2. If we use space_cache=v2, is it indeed still the case that the "btrfs" command will NOT work with the filesystem?
Why would you think "btrfs" won't work on a btrfs?
Thanks,
QuSo will our
"btrfs scrub start /mount/point/..." cron jobs FAIL? I'm guessing the btrfs command comes from btrfs-progs which is currently >>>> v5.4.1-2
amd64, is that correct?
3. Any other ideas on how we can get rid of those annoying pauses with large backups into the array?Thanks in advance!
DP
Quoted 43 lines · Joshua Villwock
Qu, thanks indeed for your responses!
Quoted 41 lines · Qu Wenruo
Quoted 9 lines · DanglingPointer
Hi Joshua, on that system where you tried to run the "--clear-space-cache v1", when you gave up, did you continue using "space_cache=v2" on it?
Here are some more questions to add for anyone who can help educate us:...
Why would someone want to clear the
space_cachev1?Because it takes up space and we won't be able to delete them.
Quoted 3 lines
What's the value of clearing the previous version
space_cachebefore using the new version?AFAIK, just save some space and reduce the root tree size.
The v1 cache exists as special files inside root tree (not accessible by end users).
Their existence takes up space and fragments the file system (one file is normally around 64K, and we have one such v1 file for each block group, you can see how many small files it has now)
Quoted 4 lines
Why "clear" and not just "delete"? Won't "deleting" the whole previous
space_cachefiles, blocks, whatever in the filesystem be faster then doing whatever "clear" does?Just bad naming, and properly from me.
Indeed "delete" would be more proper here.
And we're indeed deleting them in "btrfs check --clear-space-cache v1", that's also why it's so slow.
Quoted 6 lines
Quoted 4 lines
If you have 20T used space, then the it would be around 20,000 block groups, meaning 20,000 64K files inside root tree, and deleting them one by one, and each deletion will cause a new transaction, no wonder it will be slow to hell.
If v1 space cache exists as special files in the root tree, could those be a contributing factor to the issue I was having in the past with mounts taking too long and dropping to recovery mode?
That's another known thing.
It's block group items in extent tree.
I have proposed skinny block group tree to greatly reduce the time needed to mount large fs.
But unfortunately it's not yet merged, and even merged, you will still need to do a lengthy convert to the new format.
Quoted 2 lines
I discovered running btrfs fi defrag on each of my subvolumes (for the subvolume tree/extent tree, not the files) every couple weeks has reduced the mount time enough I don’t run into that issue on my massive FS anymore.
Yes, defrag reduce the size of extent tree, but not good enough, especially when you have hundreds of thousands block groups, and every time mount needs to find the block group items scattering around the large extent tree, it will take quite some time.
Quoted 2 lines
Could actually getting rid of those thousands of now-useless v1 entries help reduce mount times on such a massive FS, or would that be completely unrelated?
It can help, but not a root cause.
Thanks,
Qu
Quoted 224 lines
Thanks,
-JoshuaQuoted 220 lines
Quoted 3 lines
Am I missing out on something by not attempting to clear the previous version
space_cachebefore using the new v2 version?Except some wasted space, you're completely fine to skip the slow deletion.
This also means, I should enhance the deletion process to avoid too many transactions...
Thanks,
QuQuoted 208 lines
Quoted 208 lines · Joshua
Just as a point of data, I have a 96 TB array with RAID1 data, and RAID1C3 metadata.
I made the switch to space_cache=v2 some time ago, and I remember it made a huge difference when I did so!
(It was RAID1 metadata at the time, as RAID1C3 was not available at the time.)However, I also tried a check with '--clear-space-cache v1' at the time, and after waiting a literal whole day without it completing, I gave up, canceled it, and put it back into production. Is a --clear-space-cache v1 operation expected to take so long on such a large file system?
Thanks!
--Joshua VillwockJuly 15, 2021 9:40 AM, "DanglingPointer"
Quoted 182 lines · <[email protected]> wrote:
Hi Qu,
Just updating here that setting the mount option "space_cache=v2" and "noatime" completely SOLVED
the performance problem!
Basically like night and day!These are my full fstab mount options...
btrfs defaults,autodefrag,space_cache=v2,noatime 0 2
Perhaps defaulting the space_cache=v2 should be considered? Why default to v1, what's the value of
v1?So for conclusion, for large multi-terrabyte arrays (in my case RAID5s), setting space_cache=v2 and
noatime massively increases performance and eliminates the large long pauses in frequent intervals
by "btrfs-transacti" blocking all IO.Thanks Qu for your help!
Quoted 157 lines · Qu Wenruo
Quoted 8 lines · DanglingPointer
a) "echo l >
/proc/sysrq-trigger"The backup finished today already unfortunately and we are unlikely to run it again until we get an outage to remount the array with the space_cache=v2 and noatime mount options.
Thanks for the command, we'll definitely use it if/when it happens again
on the next large migration of data.Just to avoid confusion, after that command, "dmesg" output is still needed, as that's where sysrq put its output.
Quoted 5 lines
b) "sudo btrfs qgroup show -prce" ........
$ ERROR: can't list qgroups: quotas not enabledSo looks like it isn't enabled.
One less thing to bother.
Quoted 3 lines
File sizes are between: 1,048,576 bytes and 16,777,216 bytes (Duplicacy
backup defaults)Between 1~16MiB, thus tons of small files.
Btrfs is not really good at handling tons of small files, as they generate a lot of metadata.
That may contribute to the hang.
Quoted 1 line
What classifies as a transaction?
It's a little complex.
Technically it's a check point where before the checkpoint, all you see is old data, after the checkpoint, all you see is new data.
To end users, any data and metadata write will be included into one transaction (with proper dependency handled).
One way to finish (or commit) current transaction is to sync the fs, using "sync" command (sync all filesystems).
Quoted 2 lines
Any/All writes done in a 30sec
interval?This the default commit interval. Almost all fses will try to commit its
data/metadata to disk after a configurable interval.The default one is 30s. That's also one way to commit current > transaction.
Quoted 2 lines
If 100 unique files were written in 30secs, is that 1 transaction or 100 transactions?
It depends. As things like
syncfs()and subvolume/snapshot creation may try to commit transaction.But without those special operations, just writing 100 unique files using buffered write, it would only start one transaction, and when the 30s interval get hit, the transaction will be committed to disk.
Quoted 2 lines
Millions of files of the size range
above were backed up.The amount of files may not force a transaction commit, if it doesn't trigger enough memory pressure, or free space pressure.
Anyway, the "echo l" sysrq would help us to locate what's taking so long
time.Quoted 3 lines
c) "Just mount with "space_cache=v2""
Ok so no need to "
clear_cache" the v1 cache, right?Yes, and "
clear_cache" won't really remove all the v1 cache anyway.Thus it doesn't help much.
The only way to fully clear v1 cache is by using "btrfs check --clear-space-cache v1" on a *unmounted* btrfs.
Quoted 2 lines
I wrote this in the fstab but hadn't remounted yet until I can get an outage....
IMHO if you really want to test if v2 would help, you can just remount, no need to wait for a break.
Thanks,
QuQuoted 5 lines
..."btrfs defaults,autodefrag,clear_cache,space_cache=v2,noatime 0 2 >
Thanks again for your help Qu!On 14/7/21 2:59 pm, Qu Wenruo wrote:
On 2021/7/13 下午11:38, DanglingPointer wrote:
We're currently considering switching to "space_cache=v2" with noatime mount options for my lab server-workstations running RAID5.Btrfs RAID5 is unsafe due to its write-hole problem.
* One has 13TB of data/metadata in a bunch of 6TB and 2TB disks totalling 26TB.
* Another has about 12TB data/metadata in uniformly sized 6TB disks totalling 24TB.
* Both of the arrays are on individually luks encrypted disks with btrfs on top of the luks.
* Both have "defaults,autodefrag" turned on in fstab.We're starting to see large pauses during constant backups of millions of chunk files (using duplicacy backup) in the 24TB array.
Pauses sometimes take up to 20+ seconds in frequencies after every ~30secs of the end of the last pause. "btrfs-transacti" process consistently shows up as the blocking process/thread locking up filesystem IO. IO gets into the RAID5 array via nfsd. There are no
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
disk
or btrfs errors recorded. scrub last finished yesterday successfully.
Please provide the "echo l >
/proc/sysrq-trigger" output when suchQuoted 1 line
Quoted 1 line
Quoted 1 line
pause
happens.
If you're using qgroup (may be enabled by things like snapper), it may be the cause, as qgroup does its accounting when committing >>> transaction.
If one transaction is super large, it can cause such problem.
You can test if qgroup is enabled by:
# btrfs qgroup show -prce <mnt>After doing some research around the internet, we've come to the consideration above as described. Unfortunately the official documentation isn't clear on the following.
Official documentation URL -
https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs(5)1. How to migrate from default space_cache=v1 to space_cache=v2? It talks about the reverse, from v2 to v1!
Just mount with "space_cache=v2".
2. If we use space_cache=v2, is it indeed still the case that the "btrfs" command will NOT work with the filesystem?
Why would you think "btrfs" won't work on a btrfs?
Thanks,
QuSo will our
"btrfs scrub start /mount/point/..." cron jobs FAIL? I'm guessing the btrfs command comes from btrfs-progs which is currently >>>> v5.4.1-2
amd64, is that correct?
3. Any other ideas on how we can get rid of those annoying pauses with large backups into the array?Thanks in advance!
DP
Quoted 7 lines · DanglingPointer
Hi Qu,
Just updating here that setting the mount option "space_cache=v2" and "noatime" completely SOLVED the performance problem! Basically like night and day!
Quoted 6 lines
These are my full fstab mount options...
btrfs defaults,autodefrag,space_cache=v2,noatime 0 2
Perhaps defaulting the space_cache=v2 should be considered?
We're already considering that.
Quoted 2 lines
Why default
to v1, what's the value of v1?
One of the problem in the past is the lack of write ability in btrfs-progs.
Now we're testing default it in mkfs.btrfs.
Thanks,
Qu
Quoted 187 lines
So for conclusion, for large multi-terrabyte arrays (in my case RAID5s), setting space_cache=v2 and noatime massively increases performance and eliminates the large long pauses in frequent intervals by "btrfs-transacti" blocking all IO.
Thanks Qu for your help!
Quoted 175 lines · Qu Wenruo
Quoted 7 lines · DanglingPointer
a) "echo l >
/proc/sysrq-trigger"The backup finished today already unfortunately and we are unlikely to run it again until we get an outage to remount the array with the space_cache=v2 and noatime mount options.
Thanks for the command, we'll definitely use it if/when it happens again on the next large migration of data.Just to avoid confusion, after that command, "dmesg" output is still needed, as that's where sysrq put its output.
Quoted 7 lines
b) "sudo btrfs qgroup show -prce" ........
$ ERROR: can't list qgroups: quotas not enabledSo looks like it isn't enabled.
One less thing to bother.
Quoted 3 lines
File sizes are between: 1,048,576 bytes and 16,777,216 bytes (Duplicacy backup defaults)
Between 1~16MiB, thus tons of small files.
Btrfs is not really good at handling tons of small files, as they generate a lot of metadata.
That may contribute to the hang.
Quoted 2 lines
What classifies as a transaction?
It's a little complex.
Technically it's a check point where before the checkpoint, all you see is old data, after the checkpoint, all you see is new data.
To end users, any data and metadata write will be included into one transaction (with proper dependency handled).
One way to finish (or commit) current transaction is to sync the fs, using "sync" command (sync all filesystems).
Quoted 2 lines
Any/All writes done in a 30sec
interval?This the default commit interval. Almost all fses will try to commit its data/metadata to disk after a configurable interval.
The default one is 30s. That's also one way to commit current transaction.
Quoted 2 lines
If 100 unique files were written in 30secs, is that 1 transaction or 100 transactions?
It depends. As things like
syncfs()and subvolume/snapshot creation may try to commit transaction.But without those special operations, just writing 100 unique files using buffered write, it would only start one transaction, and when the 30s interval get hit, the transaction will be committed to disk.
Quoted 2 lines
Millions of files of the size range
above were backed up.The amount of files may not force a transaction commit, if it doesn't trigger enough memory pressure, or free space pressure.
Anyway, the "echo l" sysrq would help us to locate what's taking so long time.
Quoted 5 lines
c) "Just mount with "space_cache=v2""
Ok so no need to "
clear_cache" the v1 cache, right?Yes, and "
clear_cache" won't really remove all the v1 cache anyway.Thus it doesn't help much.
The only way to fully clear v1 cache is by using "btrfs check --clear-space-cache v1" on a *unmounted* btrfs.
Quoted 2 lines
I wrote this in the fstab but hadn't remounted yet until I can get an outage....
IMHO if you really want to test if v2 would help, you can just remount, no need to wait for a break.
Thanks,
QuQuoted 79 lines
..."btrfs defaults,autodefrag,clear_cache,space_cache=v2,noatime 0 2 > Thanks again for your help Qu!
Quoted 74 lines · Qu Wenruo
Quoted 2 lines · DanglingPointer
We're currently considering switching to "space_cache=v2" with noatime mount options for my lab server-workstations running RAID5.
Btrfs RAID5 is unsafe due to its write-hole problem.
Quoted 12 lines
* One has 13TB of data/metadata in a bunch of 6TB and 2TB disks totalling 26TB.
* Another has about 12TB data/metadata in uniformly sized 6TB disks totalling 24TB.
* Both of the arrays are on individually luks encrypted disks with btrfs on top of the luks.
* Both have "defaults,autodefrag" turned on in fstab.We're starting to see large pauses during constant backups of millions of chunk files (using duplicacy backup) in the 24TB array.
Quoted 6 lines
Pauses sometimes take up to 20+ seconds in frequencies after every ~30secs of the end of the last pause. "btrfs-transacti" process consistently shows up as the blocking process/thread locking up filesystem IO. IO gets into the RAID5 array via nfsd. There are no disk
or btrfs errors recorded. scrub last finished yesterday successfully.Please provide the "echo l >
/proc/sysrq-trigger" output when such pause
happens.If you're using qgroup (may be enabled by things like snapper), it may be the cause, as qgroup does its accounting when committing transaction.
If one transaction is super large, it can cause such problem.
You can test if qgroup is enabled by:
# btrfs qgroup show -prce <mnt>Quoted 10 lines
After doing some research around the internet, we've come to the consideration above as described. Unfortunately the official documentation isn't clear on the following.
Official documentation URL -
https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs(5)1. How to migrate from default space_cache=v1 to space_cache=v2? It talks about the reverse, from v2 to v1!
Just mount with "space_cache=v2".
Quoted 2 lines
2. If we use space_cache=v2, is it indeed still the case that the "btrfs" command will NOT work with the filesystem?
Why would you think "btrfs" won't work on a btrfs?
Thanks,
QuQuoted 12 lines
So will our
"btrfs scrub start /mount/point/..." cron jobs FAIL? I'm guessing the btrfs command comes from btrfs-progs which is currently v5.4.1-2
amd64, is that correct?
3. Any other ideas on how we can get rid of those annoying pauses with large backups into the array?Thanks in advance!
DP