Still seeing high autodefrag IO with kernel 5.16.5
Hello all,
Even after the defrag patches that landed in 5.16.5, I am still seeing lots of cpu usage and disk writes to my SSD when autodefrag is enabled. I kinda expected slightly more IO during writes compared to 5.15, but what I am actually seeing is massive amounts of btrfs-cleaner i/o even when no programs are actively writing to the disk.
I can reproduce it quite reliably on my 2TB Btrfs Steam library partition. In my case, I was downloading Strange Brigade, which is a roughly 25GB download and 33.65GB on disk. Somewhere during the download, iostat will start reporting disk writes around 300 MB/s, even though Steam itself reports disk usage of 40-45MB/s. After the download finishes and nothing else is being written to disk, I still see around 90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs cleaner and other btrfs processes writing a lot of data.
I left it running for a while to see if it was just some maintenance tasks that Btrfs needed to do, but it just kept going. I tried to reboot, but it actually prevented me from properly rebooting. After systemd timed out, my system finally shutdown.
Here are my mount options:
rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/
I've disabled autodefrag again for now to save my SSD, but just wanted to say that there is still an issue. Have the defrag issues been fully fixed or are there more patches incoming despite what Reddit and Phoronix say? XD
Thanks!
Ben
Quoted 17 lines · Benjamin Xiao
Hello all,
Even after the defrag patches that landed in 5.16.5, I am still seeing lots of cpu usage and disk writes to my SSD when autodefrag is enabled. I kinda expected slightly more IO during writes compared to 5.15, but what I am actually seeing is massive amounts of btrfs-cleaner i/o even when no programs are actively writing to the disk.
I can reproduce it quite reliably on my 2TB Btrfs Steam library partition. In my case, I was downloading Strange Brigade, which is a roughly 25GB download and 33.65GB on disk. Somewhere during the download, iostat will start reporting disk writes around 300 MB/s, even though Steam itself reports disk usage of 40-45MB/s. After the download finishes and nothing else is being written to disk, I still see around 90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs cleaner and other btrfs processes writing a lot of data.
Quoted 7 lines
I left it running for a while to see if it was just some maintenance tasks that Btrfs needed to do, but it just kept going. I tried to reboot, but it actually prevented me from properly rebooting. After systemd timed out, my system finally shutdown.
Here are my mount options:
rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/
Compression, I guess that's the reason.
From the very beginning, btrfs defrag doesn't handle compressed extent well.
Even if a compressed extent is already at its maximum capacity, btrfs will still try to defrag it.
I believe the behavior is masked by other problems in older kernels thus not that obvious.
But after rework of defrag in v5.16, this behavior is more exposed.
There are patches to address the compression related problem, but not yet merged:
https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
Mind to test them to see if that's the case?
Thanks,
Qu
Quoted 11 lines
I've disabled autodefrag again for now to save my SSD, but just wanted to say that there is still an issue. Have the defrag issues been fully fixed or are there more patches incoming despite what Reddit and Phoronix say? XD
Thanks!
Ben
Quoted 34 lines · Qu Wenruo
Quoted 17 lines · Benjamin Xiao
Hello all,
Even after the defrag patches that landed in 5.16.5, I am still seeing lots of cpu usage and disk writes to my SSD when autodefrag is enabled. I kinda expected slightly more IO during writes compared to 5.15, but what I am actually seeing is massive amounts of btrfs-cleaner i/o even when no programs are actively writing to the disk.
I can reproduce it quite reliably on my 2TB Btrfs Steam library partition. In my case, I was downloading Strange Brigade, which is a roughly 25GB download and 33.65GB on disk. Somewhere during the download, iostat will start reporting disk writes around 300 MB/s, even though Steam itself reports disk usage of 40-45MB/s. After the download finishes and nothing else is being written to disk, I still see around 90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs cleaner and other btrfs processes writing a lot of data.
Quoted 8 lines
I left it running for a while to see if it was just some maintenance tasks that Btrfs needed to do, but it just kept going. I tried to reboot, but it actually prevented me from properly rebooting. After systemd timed out, my system finally shutdown.
Here are my mount options:
rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/Compression, I guess that's the reason.
From the very beginning, btrfs defrag doesn't handle compressed extent
well.
Quoted 7 lines
Even if a compressed extent is already at its maximum capacity, btrfs will still try to defrag it.
I believe the behavior is masked by other problems in older kernels thus not that obvious.
But after rework of defrag in v5.16, this behavior is more exposed.
And if possible, please try this diff on v5.15.x, and see if v5.15 is really doing less IO than v5.16.x.
The diff will solve a problem in the old code, where autodefrag is almost not working.
fs/btrfs/ioctl.c +0 -0
@@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode, struct
file *file,
continue;
}- if (!newer_than) {
- cluster = (PAGE_ALIGN(defrag_end) >>
- PAGE_SHIFT) - i;
- cluster = min(cluster, max_cluster);
- } else {
- cluster = max_cluster;
- } + cluster = (PAGE_ALIGN(defrag_end) >> PAGE_SHIFT) - i; + cluster = min(cluster, max_cluster);
if (i + cluster > ra_index) {
ra_index = max(i, ra_index);Quoted 21 lines
There are patches to address the compression related problem, but not yet merged:
https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
Mind to test them to see if that's the case?
Thanks,
QuQuoted 11 lines
I've disabled autodefrag again for now to save my SSD, but just wanted to say that there is still an issue. Have the defrag issues been fully fixed or are there more patches incoming despite what Reddit and Phoronix say? XD
Thanks!
Ben
Okay, I just compiled a custom Arch kernel with your patches applied. Will test soon. Besides enabling autodefrag and redownloading a game from Steam, what other sorts of tests should I do?
Ben
Quoted 96 lines · Qu Wenruo
Quoted 35 lines · Qu Wenruo
Quoted 17 lines · Benjamin Xiao
Hello all,
Even after the defrag patches that landed in 5.16.5, I am still seeing lots of cpu usage and disk writes to my SSD when autodefrag is enabled. I kinda expected slightly more IO during writes compared to 5.15, but what I am actually seeing is massive amounts of btrfs-cleaner i/o even when no programs are actively writing to the disk.
I can reproduce it quite reliably on my 2TB Btrfs Steam library partition. In my case, I was downloading Strange Brigade, which is a roughly 25GB download and 33.65GB on disk. Somewhere during the download, iostat will start reporting disk writes around 300 MB/s, even though Steam itself reports disk usage of 40-45MB/s. After the download finishes and nothing else is being written to disk, I still see around 90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs cleaner and other btrfs processes writing a lot of data.
Quoted 8 lines
I left it running for a while to see if it was just some maintenance tasks that Btrfs needed to do, but it just kept going. I tried to reboot, but it actually prevented me from properly rebooting. After systemd timed out, my system finally shutdown.
Here are my mount options:
rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/Compression, I guess that's the reason.
From the very beginning, btrfs defrag doesn't handle compressed
extent
well.Quoted 7 lines
Even if a compressed extent is already at its maximum capacity, btrfs will still try to defrag it.
I believe the behavior is masked by other problems in older kernels thus not that obvious.
But after rework of defrag in v5.16, this behavior is more exposed.
And if possible, please try this diff on v5.15.x, and see if v5.15 is really doing less IO than v5.16.x.
The diff will solve a problem in the old code, where autodefrag is almost not working.
fs/btrfs/ioctl.c +0
-0@@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode,struct
file *file,continue; }fs/btrfs/ioctl.c +2
-7-if (!newer_than) { - cluster = (PAGE_ALIGN (defrag_end) >> -PAGE_SHIFT ) - i; - cluster =min (cluster, max_cluster); - }else { - cluster = max_cluster; - } + cluster = (PAGE_ALIGN (defrag_end) >>PAGE_SHIFT ) - i; + cluster =min (cluster, max_cluster);if (i + cluster > ra_index) { ra_index =max (i, ra_index);Quoted 22 lines
There are patches to address the compression related problem, but not yet merged:
https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
Mind to test them to see if that's the case?
Thanks,
QuQuoted 11 lines
I've disabled autodefrag again for now to save my SSD, but just wanted to say that there is still an issue. Have the defrag issues been fully fixed or are there more patches incoming despite what Reddit and Phoronix say? XD
Thanks!
Ben
Quoted 3 lines · Benjamin Xiao
Okay, I just compiled a custom Arch kernel with your patches applied. Will test soon. Besides enabling autodefrag and redownloading a game from Steam, what other sorts of tests should I do?
As long as your workload can trigger the problem reliably, nothing else.
Thanks,
Qu
Quoted 102 lines
Ben
Quoted 95 lines · Qu Wenruo
Quoted 35 lines · Qu Wenruo
Quoted 17 lines · Benjamin Xiao
Hello all,
Even after the defrag patches that landed in 5.16.5, I am still seeing lots of cpu usage and disk writes to my SSD when autodefrag is enabled. I kinda expected slightly more IO during writes compared to 5.15, but what I am actually seeing is massive amounts of btrfs-cleaner i/o even when no programs are actively writing to the disk.
I can reproduce it quite reliably on my 2TB Btrfs Steam library partition. In my case, I was downloading Strange Brigade, which is a roughly 25GB download and 33.65GB on disk. Somewhere during the download, iostat will start reporting disk writes around 300 MB/s, even though Steam itself reports disk usage of 40-45MB/s. After the download finishes and nothing else is being written to disk, I still see around 90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs cleaner and other btrfs processes writing a lot of data.
Quoted 9 lines
I left it running for a while to see if it was just some maintenance tasks that Btrfs needed to do, but it just kept going. I tried to reboot, but it actually prevented me from properly rebooting. After systemd timed out, my system finally shutdown.
Here are my mount options:
rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/Compression, I guess that's the reason.
From the very beginning, btrfs defrag doesn't handle compressed extent well.
Quoted 7 lines
Even if a compressed extent is already at its maximum capacity, btrfs will still try to defrag it.
I believe the behavior is masked by other problems in older kernels thus not that obvious.
But after rework of defrag in v5.16, this behavior is more exposed.
And if possible, please try this diff on v5.15.x, and see if v5.15 is really doing less IO than v5.16.x.
The diff will solve a problem in the old code, where autodefrag is almost not working.
fs/btrfs/ioctl.c +2
-7@@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode, structfile *file, continue; } - if (!newer_than) { - cluster = (PAGE_ALIGN(defrag_end) >> - PAGE_SHIFT) - i; - cluster = min(cluster, max_cluster); - } else { - cluster = max_cluster; - } + cluster = (PAGE_ALIGN(defrag_end) >> PAGE_SHIFT) - i; + cluster = min(cluster, max_cluster); if (i + cluster > ra_index) {
ra_index=max(i, ra_index);Quoted 21 lines
There are patches to address the compression related problem, but not yet merged:
https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
Mind to test them to see if that's the case?
Thanks,
QuQuoted 11 lines
I've disabled autodefrag again for now to save my SSD, but just wanted to say that there is still an issue. Have the defrag issues been fully fixed or are there more patches incoming despite what Reddit and Phoronix say? XD
Thanks!
Ben
Okay, I just tested right now with my custom 5.16.5 kernel with your 3 patches applied. Redownloading the same game, I noticed that there was significantly less IO load during the download, which is great. It looked kinda bursty instead, with periods of no load, and then periods of load ranging in the 150-200MB/s range.
After the download, I am no longer getting that constant 90-150MB/s disk write from btrfs-cleaner, but I am seeing periodic bursts of it every 10 seconds or so. These bursts last for around 3 seconds and load is anywhere from 50-300MB/s.
Is this normal? Does autodefrag only defrag newly written data, or does it sometimes go back and run defrag on data written previously? I am gonna let it run for a bit to see if it eventually subsides.
Ben
Quoted 117 lines · Qu Wenruo
Quoted 3 lines · Benjamin Xiao
Okay, I just compiled a custom Arch kernel with your patches applied. Will test soon. Besides enabling autodefrag and redownloading a game from Steam, what other sorts of tests should I do?
As long as your workload can trigger the problem reliably, nothing else.
Thanks,
QuQuoted 105 lines
Ben
Quoted 98 lines · Qu Wenruo
Quoted 36 lines · Qu Wenruo
Quoted 17 lines · Benjamin Xiao
Hello all,
Even after the defrag patches that landed in 5.16.5, I am still seeing lots of cpu usage and disk writes to my SSD when autodefrag is enabled. I kinda expected slightly more IO during writes compared to 5.15, but what I am actually seeing is massive amounts of btrfs-cleaner i/o even when no programs are actively writing to the disk.
I can reproduce it quite reliably on my 2TB Btrfs Steam library partition. In my case, I was downloading Strange Brigade, which is a roughly 25GB download and 33.65GB on disk. Somewhere during the download, iostat will start reporting disk writes around 300 MB/s, even though Steam itself reports disk usage of 40-45MB/s. After the download finishes and nothing else is being written to disk, I still see around 90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs cleaner and other btrfs processes writing a lot of data.
Quoted 9 lines
I left it running for a while to see if it was just some maintenance tasks that Btrfs needed to do, but it just kept going. I tried to reboot, but it actually prevented me from properly rebooting. After systemd timed out, my system finally shutdown.
Here are my mount options:
rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/Compression, I guess that's the reason.
From the very beginning, btrfs defrag doesn't handle compressed
extent
well.Quoted 7 lines
Even if a compressed extent is already at its maximum capacity, btrfs will still try to defrag it.
I believe the behavior is masked by other problems in older kernels thus not that obvious.
But after rework of defrag in v5.16, this behavior is more exposed.
And if possible, please try this diff on v5.15.x, and see if v5.15 is really doing less IO than v5.16.x.
The diff will solve a problem in the old code, where autodefrag is almost not working.
fs/btrfs/ioctl.c +0
-0@@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode,struct
file *file,continue; }fs/btrfs/ioctl.c +1
-7-if (!newer_than) { - cluster = (PAGE_ALIGN (defrag_end) >> -PAGE_SHIFT ) - i; - cluster =min (cluster, max_cluster); - }else { - cluster = max_cluster; - } + cluster = (PAGE_ALIGN(defrag_end) >> PAGE_SHIFT) -i;
+ cluster =min(cluster, max_cluster);if (i + cluster > ra_index) { ra_index = max(i, ra_index);Quoted 21 lines
There are patches to address the compression related problem, but not yet merged:
https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
Mind to test them to see if that's the case?
Thanks,
QuQuoted 11 lines
I've disabled autodefrag again for now to save my SSD, but just wanted to say that there is still an issue. Have the defrag issues been fully fixed or are there more patches incoming despite what Reddit and Phoronix say? XD
Thanks!
Ben
There's definitely still an issue even with the patches. I ended up disabling autodefrag again and rebooting my computer after btrfs-cleaner wrote about 300-ish GB to my SSD.
The patch does help things out a bit compared to before where it was a constant non-stop stream of IO, but 300GB worth of extra writes for 33GB of actual data doesn't seem normal.
Ben
Quoted 128 lines · Benjamin Xiao
Okay, I just tested right now with my custom 5.16.5 kernel with your 3 patches applied. Redownloading the same game, I noticed that there was significantly less IO load during the download, which is great. It looked kinda bursty instead, with periods of no load, and then periods of load ranging in the 150-200MB/s range.
After the download, I am no longer getting that constant 90-150MB/s disk write from btrfs-cleaner, but I am seeing periodic bursts of it every 10 seconds or so. These bursts last for around 3 seconds and load is anywhere from 50-300MB/s.
Is this normal? Does autodefrag only defrag newly written data, or does it sometimes go back and run defrag on data written previously? I am gonna let it run for a bit to see if it eventually subsides.
Ben
Quoted 117 lines · Qu Wenruo
Quoted 3 lines · Benjamin Xiao
Okay, I just compiled a custom Arch kernel with your patches applied. Will test soon. Besides enabling autodefrag and redownloading a game from Steam, what other sorts of tests should I do?
As long as your workload can trigger the problem reliably, nothing else.
Thanks,
QuQuoted 105 lines
Ben
Quoted 98 lines · Qu Wenruo
Quoted 36 lines · Qu Wenruo
Quoted 17 lines · Benjamin Xiao
Hello all,
Even after the defrag patches that landed in 5.16.5, I am still seeing lots of cpu usage and disk writes to my SSD when autodefrag is enabled. I kinda expected slightly more IO during writes compared to 5.15, but what I am actually seeing is massive amounts of btrfs-cleaner i/o even when no programs are actively writing to the disk.
I can reproduce it quite reliably on my 2TB Btrfs Steam library partition. In my case, I was downloading Strange Brigade, which is a roughly 25GB download and 33.65GB on disk. Somewhere during the download, iostat will start reporting disk writes around 300 MB/s, even though Steam itself reports disk usage of 40-45MB/s. After the download finishes and nothing else is being written to disk, I still see around 90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs cleaner and other btrfs processes writing a lot of data.
Quoted 9 lines
I left it running for a while to see if it was just some maintenance tasks that Btrfs needed to do, but it just kept going. I tried to reboot, but it actually prevented me from properly rebooting. After systemd timed out, my system finally shutdown.
Here are my mount options:
rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/Compression, I guess that's the reason.
From the very beginning, btrfs defrag doesn't handle compressed
extent
well.Quoted 7 lines
Even if a compressed extent is already at its maximum capacity, btrfs will still try to defrag it.
I believe the behavior is masked by other problems in older kernels thus not that obvious.
But after rework of defrag in v5.16, this behavior is more exposed.
And if possible, please try this diff on v5.15.x, and see if v5.15 is really doing less IO than v5.16.x.
The diff will solve a problem in the old code, where autodefrag is almost not working.
fs/btrfs/ioctl.c +0
-0@@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode,struct
file *file,continue; }fs/btrfs/ioctl.c +1
-7-if (!newer_than) { - cluster = (PAGE_ALIGN (defrag_end) >> -PAGE_SHIFT ) - i; - cluster =min (cluster, max_cluster); - }else { - cluster = max_cluster; - } + cluster = (PAGE_ALIGN(defrag_end) >> PAGE_SHIFT) -i;
+ cluster =min(cluster, max_cluster);if (i + cluster > ra_index) { ra_index = max(i, ra_index);Quoted 21 lines
There are patches to address the compression related problem, but not yet merged:
https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
Mind to test them to see if that's the case?
Thanks,
QuQuoted 11 lines
I've disabled autodefrag again for now to save my SSD, but just wanted to say that there is still an issue. Have the defrag issues been fully fixed or are there more patches incoming despite what Reddit and Phoronix say? XD
Thanks!
Ben
Quoted 7 lines · Benjamin Xiao
There's definitely still an issue even with the patches. I ended up disabling autodefrag again and rebooting my computer after btrfs-cleaner wrote about 300-ish GB to my SSD.
The patch does help things out a bit compared to before where it was a constant non-stop stream of IO, but 300GB worth of extra writes for 33GB of actual data doesn't seem normal.
Have you tried v5.15.x with that diff?
Thanks,
Qu
Quoted 164 lines
Ben
Quoted 157 lines · Benjamin Xiao
Okay, I just tested right now with my custom 5.16.5 kernel with your 3 patches applied. Redownloading the same game, I noticed that there was significantly less IO load during the download, which is great. It looked kinda bursty instead, with periods of no load, and then periods of load ranging in the 150-200MB/s range.
After the download, I am no longer getting that constant 90-150MB/s disk write from btrfs-cleaner, but I am seeing periodic bursts of it every 10 seconds or so. These bursts last for around 3 seconds and load is anywhere from 50-300MB/s.
Is this normal? Does autodefrag only defrag newly written data, or does it sometimes go back and run defrag on data written previously? I am gonna let it run for a bit to see if it eventually subsides.
Ben
Quoted 136 lines · Qu Wenruo
Quoted 3 lines · Benjamin Xiao
Okay, I just compiled a custom Arch kernel with your patches applied. Will test soon. Besides enabling autodefrag and redownloading a game from Steam, what other sorts of tests should I do?
As long as your workload can trigger the problem reliably, nothing else.
Thanks,
QuQuoted 123 lines
Ben
Quoted 116 lines · Qu Wenruo
Quoted 52 lines · Qu Wenruo
Quoted 33 lines · Benjamin Xiao
Hello all,
Even after the defrag patches that landed in 5.16.5, I am still seeing
lots of cpu usage and disk writes to my SSD when autodefrag is enabled.
I kinda expected slightly more IO during writes compared to 5.15, but
what I am actually seeing is massive amounts of btrfs-cleaner i/o even
when no programs are actively writing to the disk.I can reproduce it quite reliably on my 2TB Btrfs Steam library partition. In my case, I was downloading Strange Brigade, which is a
roughly 25GB download and 33.65GB on disk. Somewhere during the download, iostat will start reporting disk writes around 300 MB/s, even
though Steam itself reports disk usage of 40-45MB/s. After the download
finishes and nothing else is being written to disk, I still see around
90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs
cleaner and other btrfs processes writing a lot of data.I left it running for a while to see if it was just some maintenance
tasks that Btrfs needed to do, but it just kept going. I tried to reboot, but it actually prevented me from properly rebooting. After
systemd timed out, my system finally shutdown.Quoted 5 lines
Here are my mount options:
rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/Compression, I guess that's the reason.
From the very beginning, btrfs defrag doesn't handle compressed extent
well.Even if a compressed extent is already at its maximum capacity, btrfs
will still try to defrag it.Quoted 5 lines
I believe the behavior is masked by other problems in older kernels thus
not that obvious.But after rework of defrag in v5.16, this behavior is more exposed.
And if possible, please try this diff on v5.15.x, and see if v5.15 is
really doing less IO than v5.16.x.The diff will solve a problem in the old code, where autodefrag is almost not working.
fs/btrfs/ioctl.c +2
-7@@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode, struct file *file, continue; } - if (!newer_than) { - cluster = (PAGE_ALIGN(defrag_end) >> - PAGE_SHIFT) - i; - cluster = min(cluster, max_cluster); - } else { - cluster = max_cluster; - } + cluster = (PAGE_ALIGN(defrag_end) >> PAGE_SHIFT) - i; + cluster = min(cluster, max_cluster);if (i + cluster >
ra_index) {ra_index=max(i, ra_index);Quoted 24 lines
There are patches to address the compression related problem, but not
yet merged:https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
Mind to test them to see if that's the case?
Thanks,
QuQuoted 13 lines
I've disabled autodefrag again for now to save my SSD, but just wanted
to say that there is still an issue. Have the defrag issues been fully
fixed or are there more patches incoming despite what Reddit and Phoronix say? XDThanks!
Ben
A message that is not in the archive (sent off-list or not yet received): SL2P216MB11112B447FB0400149D320C1AC2B9@SL2P216MB1111.KORP216.PROD.OUTLOOK.COM
Quoted 9 lines · Rylee Randall
I am experiencing the same issue on 5.16.7, near the end of a large steam game download btrfs-cleaner sits at the top of iotop, and shut downs take about ten minutes because of various btrfs hangs.
I compiled 5.15.21 with the mentioned patch and tried to recreate the issue and so far have been unable to. I seem to get far faster an dmore consistent write speeds from steam, and rather than btrfs-cleaner being the main source of io usage it is steam. btrfs-cleaner is far down the list along with various other btrfs- tasks.
Thanks for the report, this indeed looks like the bug in v5.15 that it doesn't defrag a lot of extents is not the root cause.
Mind to re-check with the following branch?
https://github.com/adam900710/linux/tree/autodefrag_fixes
It has one extra patch to emulate the older behavior of not using btrfs_get_em(), which can cause quite some problem for autodefrag.
Thanks,
Qu
Quoted 97 lines · Qu Wenruo
Quoted 95 lines · Qu Wenruo
Quoted 35 lines · Benjamin Xiao
Quoted 17 lines
Hello all,
Even after the defrag patches that landed in 5.16.5, I am still seeing lots of cpu usage and disk writes to my SSD when autodefrag is enabled. I kinda expected slightly more IO during writes compared to 5.15, but what I am actually seeing is massive amounts of btrfs-cleaner i/o even when no programs are actively writing to the disk.
I can reproduce it quite reliably on my 2TB Btrfs Steam library partition. In my case, I was downloading Strange Brigade, which is a roughly 25GB download and 33.65GB on disk. Somewhere during the download, iostat will start reporting disk writes around 300 MB/s, even though Steam itself reports disk usage of 40-45MB/s. After the download finishes and nothing else is being written to disk, I still see around 90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs cleaner and other btrfs processes writing a lot of data.
Quoted 9 lines
I left it running for a while to see if it was just some maintenance tasks that Btrfs needed to do, but it just kept going. I tried to reboot, but it actually prevented me from properly rebooting. After systemd timed out, my system finally shutdown.
Here are my mount options:
rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/Compression, I guess that's the reason.
From the very beginning, btrfs defrag doesn't handle compressed extent well.
Quoted 7 lines
Even if a compressed extent is already at its maximum capacity, btrfs will still try to defrag it.
I believe the behavior is masked by other problems in older kernels thus not that obvious.
But after rework of defrag in v5.16, this behavior is more exposed.
And if possible, please try this diff on v5.15.x, and see if v5.15 is really doing less IO than v5.16.x.
The diff will solve a problem in the old code, where autodefrag is almost not working.
fs/btrfs/ioctl.c +2
-7@@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode, structfile *file, continue; } - if (!newer_than) { - cluster = (PAGE_ALIGN(defrag_end) >> - PAGE_SHIFT) - i; - cluster = min(cluster, max_cluster); - } else { - cluster = max_cluster; - } + cluster = (PAGE_ALIGN(defrag_end) >> PAGE_SHIFT) - i; + cluster = min(cluster, max_cluster); if (i + cluster > ra_index) {
ra_index=max(i, ra_index);Quoted 21 lines
There are patches to address the compression related problem, but not yet merged:
https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
Mind to test them to see if that's the case?
Thanks,
QuQuoted 11 lines
I've disabled autodefrag again for now to save my SSD, but just wanted to say that there is still an issue. Have the defrag issues been fully fixed or are there more patches incoming despite what Reddit and Phoronix say? XD
Thanks!
Ben
Hi list,
I'm also experiencing high IO with autodefrag on 5.16.6 patched with:
- https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387,
- https://patchwork.kernel.org/project/linux-btrfs/list/?series=611509,
- btrfs-defrag-bring-back-the-old-file-extent-search-behavior.diff.
After about 24 hours uptime, btrfs-cleaner is most of the time at 90 % CPU usage
%CPU %MEM TIME+ COMMAND
77,6 0,0 29:03.25 btrfs-cleaner 19,1 0,0 3:40.22 kworker/u24:8-btrfs-endio-write 19,1 0,0 1:50.46 kworker/u24:27-btrfs-endio-write 18,4 0,0 4:14.82 kworker/u24:7-btrfs-endio-write 17,4 0,0 4:08.08 kworker/u24:10-events_unbound 17,4 0,0 4:23.41 kworker/u24:9-btrfs-delalloc 15,8 0,0 3:41.21 kworker/u24:3-btrfs-endio-write 15,1 0,0 2:12.61 kworker/u24:26-blkcg_punt_bio 14,8 0,0 3:40.13 kworker/u24:18-btrfs-delalloc 12,8 0,0 3:55.70 kworker/u24:20-btrfs-delalloc 12,5 0,0 4:01.79 kworker/u24:1-btrfs-delalloc 12,2 0,0 4:04.63 kworker/u24:22-btrfs-endio-write 11,5 0,0 1:11.90 kworker/u24:11-btrfs-endio-write 11,2 0,0 1:25.96 kworker/u24:0-blkcg_punt_bio 10,2 0,0 4:24.17 kworker/u24:24+events_unbound
...
Load average is 5,56, 6,88, 6,71 while just writing that mail.
And iostat shows:
avg-cpu: %user %nice %system %iowait %steal %idle
1,5% 0,0% 26,8% 6,3% 0,0% 65,4%
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn
kB_dscd Device
33893,50 0,0k 172,2M 57,8M 0,0k 1,7G
577,7M nvme0n1
169,40 0,0k 1,2M 0,0k 0,0k 12,5M
0,0k sda
178,90 0,0k 1,3M 0,0k 0,0k 12,5M
0,0k sdbI hope that helps.
Thanks,
Signature · 7 lines
Jean-Denis Girard SysNux Systèmes Linux en Polynésie française https://www.sysnux.pf/ Tél: +689 40.50.10.40 / GSM: +689 87.797.527 Le 05/02/2022 à 23:26, Qu Wenruo a écrit :
Quoted 125 lines · Rylee Randall
Quoted 9 lines
I am experiencing the same issue on 5.16.7, near the end of a large steam game download btrfs-cleaner sits at the top of iotop, and shut downs take about ten minutes because of various btrfs hangs.
I compiled 5.15.21 with the mentioned patch and tried to recreate the issue and so far have been unable to. I seem to get far faster an dmore consistent write speeds from steam, and rather than btrfs-cleaner being the main source of io usage it is steam. btrfs-cleaner is far down the list along with various other btrfs- tasks.
Thanks for the report, this indeed looks like the bug in v5.15 that it doesn't defrag a lot of extents is not the root cause.
Mind to re-check with the following branch?
https://github.com/adam900710/linux/tree/autodefrag_fixes
It has one extra patch to emulate the older behavior of not using
btrfs_get_em(), which can cause quite some problem for autodefrag.Thanks,
QuQuoted 98 lines · Qu Wenruo
Quoted 96 lines · Qu Wenruo
Quoted 36 lines · Benjamin Xiao
Quoted 17 lines
Hello all,
Even after the defrag patches that landed in 5.16.5, I am still seeing lots of cpu usage and disk writes to my SSD when autodefrag is enabled. I kinda expected slightly more IO during writes compared to 5.15, but what I am actually seeing is massive amounts of btrfs-cleaner i/o even when no programs are actively writing to the disk.
I can reproduce it quite reliably on my 2TB Btrfs Steam library partition. In my case, I was downloading Strange Brigade, which is a roughly 25GB download and 33.65GB on disk. Somewhere during the download, iostat will start reporting disk writes around 300 MB/s, even though Steam itself reports disk usage of 40-45MB/s. After the download finishes and nothing else is being written to disk, I still see around 90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs cleaner and other btrfs processes writing a lot of data.
Quoted 10 lines
I left it running for a while to see if it was just some maintenance tasks that Btrfs needed to do, but it just kept going. I tried to reboot, but it actually prevented me from properly rebooting. After systemd timed out, my system finally shutdown.
Here are my mount options:
rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/Compression, I guess that's the reason.
From the very beginning, btrfs defrag doesn't handle compressed extent well.
Quoted 7 lines
Even if a compressed extent is already at its maximum capacity, btrfs will still try to defrag it.
I believe the behavior is masked by other problems in older kernels thus not that obvious.
But after rework of defrag in v5.16, this behavior is more exposed.
And if possible, please try this diff on v5.15.x, and see if v5.15 is really doing less IO than v5.16.x.
The diff will solve a problem in the old code, where autodefrag is almost not working.
fs/btrfs/ioctl.c +2
-7@@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode, structfile *file, continue; } - if (!newer_than) { - cluster = (PAGE_ALIGN(defrag_end) >> - PAGE_SHIFT) - i; - cluster = min(cluster, max_cluster); - } else { - cluster = max_cluster; - } + cluster = (PAGE_ALIGN(defrag_end) >> PAGE_SHIFT) - i; + cluster = min(cluster, max_cluster); if (i + cluster > ra_index) {
ra_index=max(i, ra_index);Quoted 21 lines
There are patches to address the compression related problem, but not yet merged:
https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
Mind to test them to see if that's the case?
Thanks,
QuQuoted 11 lines
I've disabled autodefrag again for now to save my SSD, but just wanted to say that there is still an issue. Have the defrag issues been fully fixed or are there more patches incoming despite what Reddit and Phoronix say? XD
Thanks!
Ben
Quoted 7 lines · Jean-Denis Girard
Hi list,
I'm also experiencing high IO with autodefrag on 5.16.6 patched
with: -
https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387,
- https://patchwork.kernel.org/project/linux-btrfs/list/?series=611509,
Quoted 1 line
- btrfs-defrag-bring-back-the-old-file-extent-search-behavior.diff.
OK, btrfs_search_forward() doesn't help in this case I guess.
Quoted 27 lines
After about 24 hours uptime, btrfs-cleaner is most of the time at 90 % CPU usage
%CPU %MEM TIME+ COMMAND 77,6 0,0 29:03.25 btrfs-cleaner 19,1
0,0 3:40.22 kworker/u24:8-btrfs-endio-write 19,1 0,0 1:50.46
kworker/u24:27-btrfs-endio-write 18,4 0,0 4:14.82
kworker/u24:7-btrfs-endio-write 17,4 0,0 4:08.08
kworker/u24:10-events_unbound 17,4 0,0 4:23.41
kworker/u24:9-btrfs-delalloc 15,8 0,0 3:41.21
kworker/u24:3-btrfs-endio-write 15,1 0,0 2:12.61kworker/u24:26-blkcg_punt_bio 14,8 0,0 3:40.13 kworker/u24:18-btrfs-delalloc 12,8 0,0 3:55.70 kworker/u24:20-btrfs-delalloc 12,5 0,0 4:01.79 kworker/u24:1-btrfs-delalloc 12,2 0,0 4:04.63 kworker/u24:22-btrfs-endio-write 11,5 0,0 1:11.90 kworker/u24:11-btrfs-endio-write 11,2 0,0 1:25.96kworker/u24:0-blkcg_punt_bio 10,2 0,0 4:24.17
kworker/u24:24+events_unbound ...Load average is 5,56, 6,88, 6,71 while just writing that mail.
And iostat shows:
avg-cpu: %user %nice %system %iowait %steal %idle 1,5% 0,0%
26,8% 6,3% 0,0% 65,4%
Quoted 7 lines
tps kB_read/s kB_wrtn/s kB_dscd/s
kB_readkB_wrtnkB_dscdDevice 33893,50 0,0k 172,2M 57,8M0,0k 1,7G 577,7M nvme0n1 169,40 0,0k 1,2M 0,0k 0,0k 12,5M 0,0k sda 178,90 0,0k 1,3M 0,0k 0,0k 12,5M 0,0k sdbI hope that helps.
And what are the mount options?
From the CPU usage, it looks like there are some infinite loop, but I can't find out why by a quick glance.
And since there is no extra read, it looks like it's doing defrag again and again on the same inode and range.
Mind to apply attached debug.diff?
Then you still need to provide /sys/kernel/debug/tracing/trace, which can be very large and full of added debug info.
(This means this can slow down your system)
The example of my local test would result something like this in above trace file:
# tracer: nop
#
# entries-in-buffer/entries-written: 11/11 #P:16
#
# _-----=> irqs-off/BH-disabled
# / _----=> need-resched
# | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / _-=> migrate-disable # |||| / delay # TASK-PID CPU# ||||| TIMESTAMP FUNCTION # | | | ||||| | |
btrfs-cleaner-1770 [014] ..... 961.534329: btrfs_defrag_file:
entry r/i=5/257 start=0 len=131072 newer=7 thres=262144
btrfs-cleaner-1770 [014] ..... 961.534333:defrag_collect_targets: add target r/i=5/257 start=0 len=16384 em=0 len=16384 gen=7 newer=7
btrfs-cleaner-1770 [014] ..... 961.534335:defrag_collect_targets: add target r/i=5/257 start=16384 len=16384 em=16384 len=16384 gen=7 newer=7
btrfs-cleaner-1770 [014] ..... 961.534336:defrag_collect_targets: add target r/i=5/257 start=32768 len=16384 em=32768 len=16384 gen=7 newer=7
btrfs-cleaner-1770 [014] ..... 961.534337:defrag_collect_targets: add target r/i=5/257 start=49152 len=16384 em=49152 len=16384 gen=7 newer=7
btrfs-cleaner-1770 [014] ..... 961.534564:defrag_collect_targets: add target r/i=5/257 start=0 len=16384 em=0 len=16384 gen=7 newer=7
btrfs-cleaner-1770 [014] ..... 961.534567:defrag_collect_targets: add target r/i=5/257 start=16384 len=16384 em=16384 len=16384 gen=7 newer=7
btrfs-cleaner-1770 [014] ..... 961.534568:defrag_collect_targets: add target r/i=5/257 start=32768 len=16384 em=32768 len=16384 gen=7 newer=7
btrfs-cleaner-1770 [014] ..... 961.534569:defrag_collect_targets: add target r/i=5/257 start=49152 len=16384 em=49152 len=16384 gen=7 newer=7
btrfs-cleaner-1770 [014] ..... 961.534580: btrfs_defrag_file:
defrag target r/i=5/257 start=0 len=65536
btrfs-cleaner-1770 [014] ..... 961.534681:btrfs_run_defrag_inodes: defrag finish r/i=5/257 ret=0 defragged=16 last_scanned=131072 last_off=0 cycled=0
Thanks,
Qu
Quoted 3 lines
Thanks,
fs/btrfs/file.c +5 -0
@@ -304,6 +304,11 @@ static int __btrfs_run_defrag_inode(struct btrfs_fs_info *fs_info,sb_start_write (fs_info->sb); ret =btrfs_defrag_file (inode,NULL , &ctrl);sb_end_write (fs_info->sb); +trace_printk ("defrag finish r/i=%lld/%llu ret=%d defragged=%llu last_scanned=%llu last_off=%llu cycled=%d\n" , +BTRFS_I (inode)->root->root_key.objectid, +btrfs_ino (BTRFS_I (inode)), ret, ctrl.sectors_defragged, + ctrl.last_scanned, + defrag->last_offset, defrag->cycled);/* * if we filled the whole defrag batch, there * must be more work to do. Queue this defrag
fs/btrfs/ioctl.c +14 -0
@@ -1420,6 +1420,11 @@ static int defrag_collect_targets(struct btrfs_inode *inode, add: last_is_target = true; range_len =min (extent_map_end (em), start + len) - cur; + +trace_printk ("add target r/i=%lld/%llu start=%llu len=%llu em=%llu len=%llu gen=%lld newer=%llu\n" , + inode->root->root_key.objectid, +btrfs_ino (inode), cur, range_len, em->start, em->len, + em->generation, ctrl->newer_than);/* * This one is a good target, check if it can be merged into * last range of the target list.
@@ -1520,6 +1525,9 @@ static int defrag_one_locked_target(struct btrfs_inode *inode, EXTENT_DEFRAG, 0, 0, cached_state);set_extent_defrag (&inode->io_tree, start, start + len -1 , cached_state); +trace_printk ("defrag target r/i=%lld/%llu start=%llu len=%llu\n" , + inode->root->root_key.objectid, +btrfs_ino (inode), start, len);/* Update the page status */ for (i = start_index - first_index; i <= last_index - first_index; i++) {ClearPageChecked (pages[i]);
@@ -1786,6 +1794,12 @@ int btrfs_defrag_file(struct inode *inode, struct file_ra_state *ra,if (start_index < inode->i_mapping->writeback_index) inode->i_mapping->writeback_index = start_index; +trace_printk ("entry r/i=%lld/%llu start=%llu len=%llu newer=%llu thres=%u\n" , +BTRFS_I (inode)->root->root_key.objectid, +btrfs_ino (BTRFS_I (inode)), + ctrl->last_scanned, last_byte +1 - ctrl->last_scanned, + ctrl->newer_than, ctrl->extent_thresh); +while (ctrl->last_scanned < last_byte) {const unsigned long prev_sectors_defragged = ctrl->sectors_defragged;u64 cluster_end;
Le 06/02/2022 à 15:16, Qu Wenruo a écrit :
Quoted 1 line
And what are the mount options?
Here are the mount options:
/dev/nvme0n1p2 on / type btrfs (rw,noatime,nodiratime,seclabel,compress-force=zstd:3,ssd,discard=async,space_cache,autodefrag,skip_balance,subvolid=5,subvol=/) /dev/sdb on /home/Partage type btrfs (rw,noatime,nodiratime,seclabel,compress-force=zstd:3,space_cache=v2,autodefrag,skip_balance,subvolid=1061,subvol=/Partage) /dev/sdb on /home/Photos type btrfs (rw,noatime,nodiratime,seclabel,compress-force=zstd:3,space_cache=v2,autodefrag,skip_balance,subvolid=1062,subvol=/Photos)
...
I have 13 subvolumes on /dev/sdb.
Quoted 1 line
Mind to apply attached
debug.diff?
I can't try that immediately, but will do.
Thanks,
Signature · 4 lines
Jean-Denis Girard SysNux Systèmes Linux en Polynésie française https://www.sysnux.pf/ Tél: +689 40.50.10.40 / GSM: +689 87.797.527
Hi Qu,
Sorry for late reply, and I still have not applied the last diff you sent.
But I wanted to report that without rebooting (uptime is 3 days 7 hours) or doing anything special, my system now behaves much better. btrfs-cleaner sometimes appears in top, but not for long period, and CPU usage is low. iostat seems reasonnable too.
Not sure what to conclude...
Thanks,
Signature · 7 lines
Jean-Denis Girard SysNux Systèmes Linux en Polynésie française https://www.sysnux.pf/ Tél: +689 40.50.10.40 / GSM: +689 87.797.527 Le 06/02/2022 à 15:16, Qu Wenruo a écrit :
Quoted 10 lines · Jean-Denis Girard
Quoted 7 lines
Hi list,
I'm also experiencing high IO with autodefrag on 5.16.6 patched
with: -
https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387,
Quoted 29 lines
- https://patchwork.kernel.org/project/linux-btrfs/list/?series=611509,
Quoted 1 line
- btrfs-defrag-bring-back-the-old-file-extent-search-behavior.diff.
OK,
btrfs_search_forward()doesn't help in this case I guess.Quoted 24 lines
After about 24 hours uptime, btrfs-cleaner is most of the time at 90 % CPU usage
%CPU %MEM TIME+ COMMAND 77,6 0,0 29:03.25 btrfs-cleaner 19,1 0,0 3:40.22 kworker/u24:8-btrfs-endio-write 19,1 0,0 1:50.46 kworker/u24:27-btrfs-endio-write 18,4 0,0 4:14.82 kworker/u24:7-btrfs-endio-write 17,4 0,0 4:08.08 kworker/u24:10-events_unbound 17,4 0,0 4:23.41 kworker/u24:9-btrfs-delalloc 15,8 0,0 3:41.21 kworker/u24:3-btrfs-endio-write 15,1 0,0 2:12.61 kworker/u24:26-blkcg_punt_bio 14,8 0,0 3:40.13 kworker/u24:18-btrfs-delalloc 12,8 0,0 3:55.70 kworker/u24:20-btrfs-delalloc 12,5 0,0 4:01.79 kworker/u24:1-btrfs-delalloc 12,2 0,0 4:04.63 kworker/u24:22-btrfs-endio-write 11,5 0,0 1:11.90 kworker/u24:11-btrfs-endio-write 11,2 0,0 1:25.96 kworker/u24:0-blkcg_punt_bio 10,2 0,0 4:24.17 kworker/u24:24+events_unbound ...
Load average is 5,56, 6,88, 6,71 while just writing that mail.
And iostat shows:
hours uptime>> avg-cpu: %user %nice %system %iowait %st days eal
%idle 1,5% 0,0%
Quoted 79 lines
Quoted 9 lines
26,8% 6,3% 0,0% 65,4%
tps kB_read/s kB_wrtn/s kB_dscd/s kB_read
kB_wrtnkB_dscdDevice 33893,50 0,0k 172,2M 57,8M 0,0k 1,7G 577,7M nvme0n1 169,40 0,0k 1,2M 0,0k 0,0k 12,5M 0,0k sda 178,90 0,0k 1,3M 0,0k 0,0k 12,5M 0,0k sdbI hope that helps.
And what are the mount options?
From the CPU usage, it looks like there are some infinite loop, but I
can't find out why by a quick glance.And since there is no extra read, it looks like it's doing defrag again and again on the same inode and range.
Mind to apply attached
debug.diff?Then you still need to provide
/sys/kernel/debug/tracing/trace, which can be very large and full of added debug info.
(This means this can slow down your system)The example of my local test would result something like this in above trace file:
# tracer: nop
#
# entries-in-buffer/entries-written: 11/11 #P:16
#
# _-----=> irqs-off/BH-disabled # / _----=> need-resched# | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / _-=> migrate-disable # |||| / delay# TASK-PID CPU# ||||| TIMESTAMP FUNCTION
# | | | ||||| | |
btrfs-cleaner-1770 [014] ..... 961.534329:btrfs_defrag_file:
entry r/i=5/257 start=0 len=131072 newer=7 thres=262144
btrfs-cleaner-1770 [014] ..... 961.534333:defrag_collect_targets: add target r/i=5/257 start=0 len=16384 em=0 len=16384 gen=7 newer=7
btrfs-cleaner-1770 [014] ..... 961.534335:defrag_collect_targets: add target r/i=5/257 start=16384 len=16384 em=16384 len=16384 gen=7 newer=7
btrfs-cleaner-1770 [014] ..... 961.534336:defrag_collect_targets: add target r/i=5/257 start=32768 len=16384 em=32768 len=16384 gen=7 newer=7
btrfs-cleaner-1770 [014] ..... 961.534337:defrag_collect_targets: add target r/i=5/257 start=49152 len=16384 em=49152 len=16384 gen=7 newer=7
btrfs-cleaner-1770 [014] ..... 961.534564:defrag_collect_targets: add target r/i=5/257 start=0 len=16384 em=0 len=16384 gen=7 newer=7
btrfs-cleaner-1770 [014] ..... 961.534567:defrag_collect_targets: add target r/i=5/257 start=16384 len=16384 em=16384 len=16384 gen=7 newer=7
btrfs-cleaner-1770 [014] ..... 961.534568:defrag_collect_targets: add target r/i=5/257 start=32768 len=16384 em=32768 len=16384 gen=7 newer=7
btrfs-cleaner-1770 [014] ..... 961.534569:defrag_collect_targets: add target r/i=5/257 start=49152 len=16384 em=49152 len=16384 gen=7 newer=7
btrfs-cleaner-1770 [014] ..... 961.534580:btrfs_defrag_file:
defrag target r/i=5/257 start=0 len=65536
btrfs-cleaner-1770 [014] ..... 961.534681:btrfs_run_defrag_inodes: defrag finish r/i=5/257 ret=0 defragged=16 last_scanned=131072 last_off=0 cycled=0Thanks,
QuQuoted 3 lines
Thanks,
Quoted 8 lines · Jean-Denis Girard
Hi Qu,
Sorry for late reply, and I still have not applied the last diff you sent.
But I wanted to report that without rebooting (uptime is 3 days 7 hours) or doing anything special, my system now behaves much better. btrfs-cleaner sometimes appears in top, but not for long period, and CPU usage is low. iostat seems reasonnable too.
Well, at least it's some good news.
BTW I have updated that branch on github, feel free to try (better with the extra diff).
Just in case you have hit such situation again.
Meanwhile I'll try to rework autodefrag behavior to be more accurate.
Right now, autodefrag will try to scan the whole file if there is any small writ to it.
And there is some questionable check on whether autodefrag should rescan the whole file.
I want to change it to only scan ranges adjacent to the small write, so we can avoid possible problem related to the whole file scan behavior.
Thanks,
Qu
Quoted 5 lines
Not sure what to conclude...
Thanks,
Quoted 16 lines · Qu Wenruo
Quoted 9 lines · Rylee Randall
I am experiencing the same issue on 5.16.7, near the end of a large steam game download btrfs-cleaner sits at the top of iotop, and shut downs take about ten minutes because of various btrfs hangs.
I compiled 5.15.21 with the mentioned patch and tried to recreate the issue and so far have been unable to. I seem to get far faster an dmore consistent write speeds from steam, and rather than btrfs-cleaner being the main source of io usage it is steam. btrfs-cleaner is far down the list along with various other btrfs- tasks.
Thanks for the report, this indeed looks like the bug in v5.15 that it doesn't defrag a lot of extents is not the root cause.
Quoted 6 lines
Mind to re-check with the following branch?
https://github.com/adam900710/linux/tree/autodefrag_fixes
It has one extra patch to emulate the older behavior of not using
btrfs_get_em(), which can cause quite some problem for autodefrag.
And if anyone is still experiencing the problem even with that branch, there is a diff to do the ultimate debug (thus can slow down the system).
After applying this diff upon that branch, every auto defrag try will have several lines added to /sys/kernel/debug/tracing/trace. (Which can be pretty large).
And that trace file would greatly help us to locate which extent is defragged again and again.
Thanks,
Qu
Quoted 106 lines
Thanks,
QuQuoted 102 lines · Qu Wenruo
Quoted 100 lines · Qu Wenruo
Quoted 39 lines · Benjamin Xiao
Quoted 20 lines
Hello all,
Even after the defrag patches that landed in 5.16.5, I am still seeing lots of cpu usage and disk writes to my SSD when autodefrag is enabled.
I kinda expected slightly more IO during writes compared to 5.15, but what I am actually seeing is massive amounts of btrfs-cleaner i/o even when no programs are actively writing to the disk.I can reproduce it quite reliably on my 2TB Btrfs Steam library partition. In my case, I was downloading Strange Brigade, which is a roughly 25GB download and 33.65GB on disk. Somewhere during the download, iostat will start reporting disk writes around 300 MB/s, even
though Steam itself reports disk usage of 40-45MB/s. After the download
finishes and nothing else is being written to disk, I still see around 90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs cleaner and other btrfs processes writing a lot of data.Quoted 10 lines
I left it running for a while to see if it was just some maintenance tasks that Btrfs needed to do, but it just kept going. I tried to reboot, but it actually prevented me from properly rebooting. After systemd timed out, my system finally shutdown.
Here are my mount options:
rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/Compression, I guess that's the reason.
From the very beginning, btrfs defrag doesn't handle compressed extent well.
Quoted 8 lines
Even if a compressed extent is already at its maximum capacity, btrfs will still try to defrag it.
I believe the behavior is masked by other problems in older kernels thus
not that obvious.But after rework of defrag in v5.16, this behavior is more exposed.
And if possible, please try this diff on v5.15.x, and see if v5.15 is really doing less IO than v5.16.x.
The diff will solve a problem in the old code, where autodefrag is almost not working.
fs/btrfs/ioctl.c +2
-7@@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode, structfile *file, continue; } - if (!newer_than) { - cluster = (PAGE_ALIGN(defrag_end) >> - PAGE_SHIFT) - i; - cluster = min(cluster, max_cluster); - } else { - cluster = max_cluster; - } + cluster = (PAGE_ALIGN(defrag_end) >> PAGE_SHIFT) - i; + cluster = min(cluster, max_cluster); if (i + cluster > ra_index) {
ra_index=max(i, ra_index);Quoted 21 lines
There are patches to address the compression related problem, but not yet merged:
https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
Mind to test them to see if that's the case?
Thanks,
QuQuoted 11 lines
I've disabled autodefrag again for now to save my SSD, but just wanted to say that there is still an issue. Have the defrag issues been fully fixed or are there more patches incoming despite what Reddit and Phoronix say? XD
Thanks!
Ben
fs/btrfs/file.c +5 -0
@@ -304,6 +304,11 @@ static int __btrfs_run_defrag_inode(struct btrfs_fs_info *fs_info,sb_start_write (fs_info->sb); ret =btrfs_defrag_file (inode,NULL , &ctrl);sb_end_write (fs_info->sb); +trace_printk ("defrag finish r/i=%lld/%llu ret=%d defragged=%llu last_scanned=%llu last_off=%llu cycled=%d\n" , +BTRFS_I (inode)->root->root_key.objectid, +btrfs_ino (BTRFS_I (inode)), ret, ctrl.sectors_defragged, + ctrl.last_scanned, + defrag->last_offset, defrag->cycled);/* * if we filled the whole defrag batch, there * must be more work to do. Queue this defrag
fs/btrfs/ioctl.c +14 -0
@@ -1420,6 +1420,11 @@ static int defrag_collect_targets(struct btrfs_inode *inode, add: last_is_target = true; range_len =min (extent_map_end (em), start + len) - cur; + +trace_printk ("add target r/i=%lld/%llu start=%llu len=%llu em=%llu len=%llu gen=%lld newer=%llu\n" , + inode->root->root_key.objectid, +btrfs_ino (inode), cur, range_len, em->start, em->len, + em->generation, ctrl->newer_than);/* * This one is a good target, check if it can be merged into * last range of the target list.
@@ -1520,6 +1525,9 @@ static int defrag_one_locked_target(struct btrfs_inode *inode, EXTENT_DEFRAG, 0, 0, cached_state);set_extent_defrag (&inode->io_tree, start, start + len -1 , cached_state); +trace_printk ("defrag target r/i=%lld/%llu start=%llu len=%llu\n" , + inode->root->root_key.objectid, +btrfs_ino (inode), start, len);/* Update the page status */ for (i = start_index - first_index; i <= last_index - first_index; i++) {ClearPageChecked (pages[i]);
@@ -1786,6 +1794,12 @@ int btrfs_defrag_file(struct inode *inode, struct file_ra_state *ra,if (start_index < inode->i_mapping->writeback_index) inode->i_mapping->writeback_index = start_index; +trace_printk ("entry r/i=%lld/%llu start=%llu len=%llu newer=%llu thres=%u\n" , +BTRFS_I (inode)->root->root_key.objectid, +btrfs_ino (BTRFS_I (inode)), + ctrl->last_scanned, last_byte +1 - ctrl->last_scanned, + ctrl->newer_than, ctrl->extent_thresh); +while (ctrl->last_scanned < last_byte) {const unsigned long prev_sectors_defragged = ctrl->sectors_defragged;u64 cluster_end;
A message that is not in the archive (sent off-list or not yet received): SL2P216MB1111994F81CE0006D495511DAC2C9@SL2P216MB1111.KORP216.PROD.OUTLOOK.COM
Quoted 8 lines · Rylee Randall
I built a kernel from that branch and tried everything I could think of to try to reproduce the issue, but no matter what I tried btrfs-cleaner would only show up on iotop briefly every now and then, io did not hang and disk performance was as expected.
I think the branch resolves the issue, or at least prevents me from being able to reproduce.
Just in case, you can still use that diff, it will only populate ftrace event log buffer, which is much faster than dmesg log buffer.
And since you haven't seen huge IO/CPU usage, the extra debug message shouldn't cause any observable performance drop.
And in case you observed huge CPU usage from btrfs-cleaner under whatever load, you can direct grab the trace file, and no need to rebuild a kernel module (and may fail to reproduce in next try).
Thanks,
Qu
Quoted 158 lines · Qu Wenruo
Quoted 156 lines · Qu Wenruo
Quoted 16 lines · Rylee Randall
Quoted 9 lines
I am experiencing the same issue on 5.16.7, near the end of a large steam game download btrfs-cleaner sits at the top of iotop, and shut downs take about ten minutes because of various btrfs hangs.
I compiled 5.15.21 with the mentioned patch and tried to recreate the issue and so far have been unable to. I seem to get far faster an dmore consistent write speeds from steam, and rather than btrfs-cleaner being the main source of io usage it is steam. btrfs-cleaner is far down the list along with various other btrfs- tasks.
Thanks for the report, this indeed looks like the bug in v5.15 that it doesn't defrag a lot of extents is not the root cause.
Quoted 7 lines
Mind to re-check with the following branch?
It has one extra patch to emulate the older behavior of not using
btrfs_get_em(), which can cause quite some problem for autodefrag.And if anyone is still experiencing the problem even with that branch, there is a diff to do the ultimate debug (thus can slow down the system).
After applying this diff upon that branch, every auto defrag try will have several lines added to
/sys/kernel/debug/tracing/trace. (Which can be pretty large).And that trace file would greatly help us to locate which extent is defragged again and again.
Thanks,
QuQuoted 116 lines
Thanks,
QuQuoted 112 lines · Qu Wenruo
Quoted 110 lines · Qu Wenruo
Quoted 45 lines · Benjamin Xiao
Quoted 24 lines
Hello all,
Even after the defrag patches that landed in 5.16.5, I am still seeing
lots of cpu usage and disk writes to my SSD when autodefrag is enabled.
I kinda expected slightly more IO during writes compared to 5.15, but
what I am actually seeing is massive amounts of btrfs-cleaner i/o even
when no programs are actively writing to the disk.I can reproduce it quite reliably on my 2TB Btrfs Steam library partition. In my case, I was downloading Strange Brigade, which is a roughly 25GB download and 33.65GB on disk. Somewhere during the download, iostat will start reporting disk writes around 300 MB/s, even
though Steam itself reports disk usage of 40-45MB/s. After the download
finishes and nothing else is being written to disk, I still see around
90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs cleaner and other btrfs processes writing a lot of data.Quoted 11 lines
I left it running for a while to see if it was just some maintenance tasks that Btrfs needed to do, but it just kept going. I tried to reboot, but it actually prevented me from properly rebooting. After systemd timed out, my system finally shutdown.
Here are my mount options:
rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/Compression, I guess that's the reason.
From the very beginning, btrfs defrag doesn't handle compressed extent
well.Quoted 8 lines
Even if a compressed extent is already at its maximum capacity, btrfs will still try to defrag it.
I believe the behavior is masked by other problems in older kernels thus
not that obvious.But after rework of defrag in v5.16, this behavior is more exposed.
And if possible, please try this diff on v5.15.x, and see if v5.15 is really doing less IO than v5.16.x.
The diff will solve a problem in the old code, where autodefrag is almost not working.
fs/btrfs/ioctl.c +0
-0@@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode,struct
file *file,
continue;
}- if (!newer_than) {
- cluster = (PAGE_ALIGN(defrag_end)>> -PAGE_SHIFT) - i;
- cluster =min(cluster, max_cluster); - } else {
- cluster =max_cluster;
- }
+ cluster = (PAGE_ALIGN(defrag_end)>>PAGE_SHIFT) - i; + cluster =min(cluster, max_cluster);if (i + cluster >
ra_index) {ra_index=max(i, ra_index);Quoted 24 lines
There are patches to address the compression related problem, but not yet merged:
Mind to test them to see if that's the case?
Thanks,
QuQuoted 13 lines
I've disabled autodefrag again for now to save my SSD, but just wanted
to say that there is still an issue. Have the defrag issues been fully
fixed or are there more patches incoming despite what Reddit and Phoronix say? XDThanks!
Ben