Extreme slowdown
Hi all!
My BTRFS-FS ist getting really slow. Reading is ok, writing is slow and deleting is horrible slow.
There are many files and many links on the FS.
# btrfs filesystem df /srv/storage Data: total=3.09TB, used=3.07TB System, DUP: total=8.00MB, used=496.00KB System: total=4.00MB, used=0.00 Metadata, DUP: total=154.88GB, used=70.56GB Metadata: total=8.00MB, used=0.00
Maybe it's because there is so much Metadata and it needs so many seeks on the discs when deleting?
I'd like to delete some of the old Files but its so horrible slow that i think its maybe faster to copy all needed Data to a different Disc, killing the FS and move the Files back...
The machine is a QuadCore with 8GB RAM. Kernel is "3.1+for-linus".
Any hints how i could speed it up?
Tobias
Quoted 17 lines · Tobias
Hi all!
My BTRFS-FS ist getting really slow. Reading is ok, writing is slow and deleting is horrible slow.
There are many files and many links on the FS.
# btrfs filesystem df /srv/storage Data: total=3.09TB, used=3.07TB System, DUP: total=8.00MB, used=496.00KB System: total=4.00MB, used=0.00 Metadata, DUP: total=154.88GB, used=70.56GB Metadata: total=8.00MB, used=0.00Maybe it's because there is so much Metadata and it needs so many seeks on the discs when deleting?
Quoted 8 lines
I'd like to delete some of the old Files but its so horrible slow that i think its maybe faster to copy all needed Data to a different Disc, killing the FS and move the Files back...
The machine is a QuadCore with 8GB RAM. Kernel is "3.1+for-linus".
Any hints how i could speed it up?
I recommend to run defragment on your files, it should help you, although defrag maybe slow, too.
thanks,
liubo
Quoted 6 lines
Tobias
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · Tobias
Hi all!
My BTRFS-FS ist getting really slow. Reading is ok, writing is slow and deleting is horrible slow.
There are many files and many links on the FS.
# btrfs filesystem df /srv/storage Data: total=3.09TB, used=3.07TB
this is ... what, over 99% full?
The slow down is normal, somewhat. Same thing happens on zfs, which became slower when usage is above 80-90%.
Quoted 2 lines
Maybe it's because there is so much Metadata and it needs so many seeks on the discs when deleting?
I doubt it.
Quoted 7 lines
I'd like to delete some of the old Files but its so horrible slow that i think its maybe faster to copy all needed Data to a different Disc, killing the FS and move the Files back...
The machine is a QuadCore with 8GB RAM. Kernel is "3.1+for-linus".
Any hints how i could speed it up?
Try:
- mounting it with nodatacow:
https://btrfs.wiki.kernel.org/articles/f/a/q/FAQ_1fe9.html#Can_copy-on-write_be_turned_off_for_data_blocks.3F
- clobbering a big file:
https://btrfs.wiki.kernel.org/articles/f/a/q/FAQ_1fe9.html#if_your_device_is_large_.28.3E16gb.29
... until you have at least 20% free space available.
Signature · 1 line
Fajar
On Thu, Dec 15, 2011 at 8:19 PM, Fajar A. Nugraha <[email protected]> wrot= e:
Quoted 4 lines · Tobias
Quoted 3 lines
Hi all!
My BTRFS-FS ist getting really slow. Reading is ok, writing is slow =
and
Quoted 10 lines
Quoted 6 lines
deleting is horrible slow.
There are many files and many links on the FS.
# btrfs filesystem df /srv/storage Data: total=3D3.09TB, used=3D3.07TBthis is ... what, over 99% full?
The slow down is normal, somewhat. Same thing happens on zfs, which became slower when usage is above 80-90%.
I don't think "total" actually means "total space available" it increases as you use up more space.
Quoted 2 lines
Quoted 2 lines
Maybe it's because there is so much Metadata and it needs so many se=
eks on
Quoted 5 lines
Quoted 1 line
the discs when deleting?
I doubt it.
Quoted 1 line
I'd like to delete some of the old Files but its so horrible slow th=
at i
Quoted 1 line
Quoted 1 line
think its maybe faster to copy all needed Data to a different Disc, =
killing
Quoted 9 lines
Quoted 5 lines
the FS and move the Files back...
The machine is a QuadCore with 8GB RAM. Kernel is "3.1+for-linus".
Any hints how i could speed it up?
Try:
- mounting it with nodatacow:
https://btrfs.wiki.kernel.org/articles/f/a/q/FAQ_1fe9.html#Can_copy-o=
n-write_be_turned_off_for_data_blocks.3F
Quoted 2 lines
- clobbering a big file:
https://btrfs.wiki.kernel.org/articles/f/a/q/FAQ_1fe9.html#if_your_de=
vice_is_large_.28.3E16gb.29
Quoted 7 lines
... until you have at least 20% free space available.
--
Fajar
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs=
" in
Quoted 2 lines
the body of a message to [email protected]
More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 41 lines · Chester
Quoted 6 lines · Fajar A. Nugraha
Quoted 5 lines · Tobias
Hi all!
My BTRFS-FS ist getting really slow. Reading is ok, writing is slow and deleting is horrible slow.
Quoted 7 lines
Quoted 4 lines
There are many files and many links on the FS.
# btrfs filesystem df /srv/storage Data: total=3.09TB, used=3.07TBthis is ... what, over 99% full?
The slow down is normal, somewhat. Same thing happens on zfs, which became slower when usage is above 80-90%.I don't think "total" actually means "total space available" it increases as you use up more space.
Quoted 17 lines
Quoted 2 lines
Maybe it's because there is so much Metadata and it needs so many seeks on the discs when deleting?
I doubt it.
Quoted 7 lines
I'd like to delete some of the old Files but its so horrible slow that i think its maybe faster to copy all needed Data to a different Disc, killing the FS and move the Files back...
The machine is a QuadCore with 8GB RAM. Kernel is "3.1+for-linus".
Any hints how i could speed it up?
Try:
- mounting it with nodatacow:
https://btrfs.wiki.kernel.org/articles/f/a/q/FAQ_1fe9.html#Can_copy-on-write_be_turned_off_for_data_blocks.3F
- clobbering a big file:
https://btrfs.wiki.kernel.org/articles/f/a/q/FAQ_1fe9.html#if_your_device_is_large_.28.3E16gb.29Quoted 8 lines
... until you have at least 20% free space available.
--
Fajar
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Wouldn't it be more valuable to first collect enough info/data to debug the problem, before suggesting ways to get out of this situation?
AFAIK, the slowness could be due to several reasons:
1. Slow disk writes.
2. Too many things to commit to the disk.
3. Too many things to do to actually accomplish the writes.
4. Important threads suspending due to lack of resources.
5. Something else which I've missed above.
Is there some sort of profiling which we can enable, which can give us info about the speeds and quantities of read and write traffic on the fs? Aren't there btrfs statistics that we can print out, which can give us these info? If not, we probably should think about adding some.
P.S: btrfs n00b talking!
Regards,
Shyam
Am 16.12.2011 03:09, schrieb Liu Bo:
Quoted 7 lines · Tobias
Quoted 2 lines
Any hints how i could speed it up?
I recommend to run defragment on your files, it should help you, although defrag maybe slow, too.
When deleting files does it matter if the data is fragmented?
When i do normal file-access the speed of reading is ok so until not i didn't care about fragmentation. Especially because the data isn't changes after the first write.
Can i do a defrag on the metadata only?
Tobias
Am 16.12.2011 05:15, schrieb Shyam Prasad N:
Quoted 10 lines · Chester
Quoted 9 lines · Fajar A. Nugraha
Quoted 8 lines · Tobias
Quoted 7 lines
Hi all!
My BTRFS-FS ist getting really slow. Reading is ok, writing is slow and deleting is horrible slow.
There are many files and many links on the FS.
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
# btrfs filesystem df /srv/storage Data: total=3.09TB, used=3.07TBthis is ... what, over 99% full?
The slow down is normal, somewhat. Same thing happens on zfs, which became slower when usage is above 80-90%.I don't think "total" actually means "total space available" it increases as you use up more space.
Chester is right, the disk is a 8TB Array - the fs is less than half filled.
Quoted 4 lines
Wouldn't it be more valuable to first collect enough info/data to debug the problem, before suggesting ways to get out of this situation?
AFAIK, the slowness could be due to several reasons:
1. Slow disk writes.
That depends on what is slow for you. Its a SATA-Disc-Array so good linear Read/Write and low IOPs (compared to SSDs or SAS-Discs)
Quoted 1 line
2. Too many things to commit to the disk.
I think there should be very little data to write/commit when deleting files. Remember: writing is slow, but acceptable - deleting is the problem
Quoted 1 line
3. Too many things to do to actually accomplish the writes.
I just do rm -rf xxx. What has BTRFS to do to actually delete a file or directory?
Quoted 2 lines
Is there some sort of profiling which we can enable, which can give us info about the speeds and quantities of read and write traffic on the fs? Aren't there btrfs statistics that we can print out, which can give us these info? If not, we probably should think about adding some.
Good question - maybe a dev can say?
Tobias
Quoted 10 lines · Tobias
Am 16.12.2011 03:09, schrieb Liu Bo:
Quoted 7 lines · Tobias
Quoted 2 lines
Any hints how i could speed it up?
I recommend to run defragment on your files, it should help you, although defrag maybe slow, too.
When deleting files does it matter if the data is fragmented?
Yes. Deleting a file involves visiting the metadata for every
extent (i.e. fragment) and updating it to reduce the reference count and possibly free up the space -- which itself involves more writes, as the free space cache is updated.
Quoted 5 lines
When i do normal file-access the speed of reading is ok so until not i didn't care about fragmentation. Especially because the data isn't changes after the first write.
Can i do a defrag on the metadata only?
Hugo.
Signature · 3 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- UNIX: Italian pen maker. --- Tobias wrote (ao):
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines · Tobias
Quoted 4 lines
My BTRFS-FS ist getting really slow. Reading is ok, writing is slow and deleting is horrible slow.
There are many files and many links on the FS.
Do you happen to have (many) snapshots? Are btrfs kernel threads using a lot of cpu?
Sander
Signature · 2 lines
Humilis IT Services and Solutions http://www.humilis.net
Am 16.12.2011 10:19, schrieb Sander:
Quoted 9 lines
Tobias wrote (ao):
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines · Tobias
Quoted 4 lines
My BTRFS-FS ist getting really slow. Reading is ok, writing is slow and deleting is horrible slow.
There are many files and many links on the FS.
Do you happen to have (many) snapshots? Are btrfs kernel threads using a lot of cpu?
No Snapshots at all. Only compress-force=lzo, nothing else special.
The CPU is almost idle, IOwait is at 100%; looks likes the discs are accessing many small blocks. (seeking like hell...)
Output from iostat (average each over 120 seconds)
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdb 0.00 0.42 195.31 190.44 781.23 1657.90 12.65 64.91 168.90 3.42 338.60 2.51 96.64 sdb 0.00 0.50 75.00 336.57 300.00 2028.60 11.32 105.41 255.83 3.16 312.14 2.24 92.38 sdb 0.00 0.62 72.03 441.72 288.11 2164.88 9.55 116.39 226.27 4.20 262.48 1.89 97.29 sdb 0.00 0.20 1.07 468.74 4.26 2604.99 11.11 145.07 308.91 581.30 308.29 2.13 100.00 sdb 0.00 0.00 20.00 326.97 80.00 2114.03 12.65 38.56 111.41 58.02 114.67 2.83 98.28 sdb 0.00 0.00 5.02 563.35 20.08 3102.45 10.99 137.71 242.26 6.44 244.36 1.70 96.62
Tobias