Balance performance problem with 3.14.1
I've got a server/workstation (KDE desktop and file server) running kernel 3.14.1 from the Debian package 3.14-trunk-amd64.
It was running well until I decided to do a full balance of the BTRFS RAID-1 array of 3TB SATA disks (which hadn't been balanced before due to previous kernels performing badly with scrub or balance). I canceled the balance after about 5 days when it had been claiming to be about 65% done for a day while doing a lot of disk IO.
After canceling the balance the performance of the array has been poor. It has a cron job that runs twice a week to rsync data from a Maildir based mail server that currently has 2401473 Inodes in use according to ZFS on the mail server (unfortunately BTRFS won't tell me how many Inodes are in use). The cron job does an "rsync -va" type backup WITHOUT the -c option, so we're basically doing a recursive stat on all files and then transferring new files (Dovecot index files are excluded so files tend never to change). After the rsync is complete "cp -rl" is used to make a backup of the tree.
The cp -rl usually takes something less than 30 minutes (not long enough for me to even notice) but today cp has been running for 5.5 hours and seems to be about 3/5 done (9000/15000 subdirectories linked).
The backup script that runs rsync and cp is run at 2AM and usually finishes well before 9AM to avoid interfering with the workstation use of the system. Today the script is still running at 7:38PM and seems likely to run for some hours.
The system in question has a SSD for /, /home, and swap. The RAID-1 array of SATA disks is used for file serving and online backup. I am not aware of any performance problems with the SSD, but a reasonably fast Intel SSD used for light desktop use could probably run at 10% normal speed and still seem fast.
Signature · 2 lines
My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
Russell Coker posted on Thu, 08 May 2014 19:50:23 +1000 as excerpted:
Quoted 10 lines
I've got a server/workstation (KDE desktop and file server) running kernel 3.14.1 from the Debian package 3.14-trunk-amd64.
It was running well until I decided to do a full balance of the BTRFS RAID-1 array of 3TB SATA disks (which hadn't been balanced before due to previous kernels performing badly with scrub or balance). I canceled the balance after about 5 days when it had been claiming to be about 65% done for a day while doing a lot of disk IO.
After canceling the balance the performance of the array has been poor.
Not much help, but two comments:
1) I've seen other reports of performance problems after balance. Nobody seems to have a good reason as to why a balance might do that.
2) I've NOT seen anything like that, here. However, my btrfs are all rather small, under 50 GiB.
(FWIW, it /still/ seems weird to me calling a GB "small". It doesn't seem all /that/ long ago that I bought my first 1 GB drive, and it sure wasn't small nor inexpensive then! I guess I've officially joined the computer old-timers!)
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 9 lines · Russell Coker
I've got a server/workstation (KDE desktop and file server) running kernel 3.14.1 from the Debian package 3.14-trunk-amd64.
It was running well until I decided to do a full balance of the BTRFS RAID-1 array of 3TB SATA disks (which hadn't been balanced before due to previous kernels performing badly with scrub or balance). I canceled the balance after about 5 days when it had been claiming to be about 65% done for a day while doing a lot of disk IO.
Can I see dmesg/messages for the time that this was running for so long?
Quoted 9 lines
After canceling the balance the performance of the array has been poor. It has a cron job that runs twice a week to rsync data from a Maildir based mail server that currently has 2401473 Inodes in use according to ZFS on the mail server (unfortunately BTRFS won't tell me how many Inodes are in use). The cron job does an "rsync -va" type backup WITHOUT the -c option, so we're basically doing a recursive stat on all files and then transferring new files (Dovecot index files are excluded so files tend never to change). After the rsync is complete "cp -rl" is used to make a backup of the tree.
Quoted 4 lines
The cp -rl usually takes something less than 30 minutes (not long enough for me to even notice) but today cp has been running for 5.5 hours and seems to be about 3/5 done (9000/15000 subdirectories linked).
Sysrq+w while it's running, just one and then wait a while and then another. Thanks,
Josef
Quoted 9 lines · Josef Bacik
Quoted 8 lines · Russell Coker
I've got a server/workstation (KDE desktop and file server) running kernel 3.14.1 from the Debian package 3.14-trunk-amd64.
It was running well until I decided to do a full balance of the BTRFS RAID-1 array of 3TB SATA disks (which hadn't been balanced before due to previous kernels performing badly with scrub or balance). I canceled the balance after about 5 days when it had been claiming to be about 65% done for a day while doing a lot of disk IO.
Firstly after writing my previous message but before receiving your message the cron job finished and I umounted the filesystem and mounted it again. That freed 2G of RAM from cache (according to the "free" command) and the filesystem performed a lot better for most tasks (EG running a basic ls command didn't give a noticable delay).
Number of files: 2010493
Number of files transferred: 19609
Total file size: 216240165510 bytes
Total transferred file size: 2632884253 bytes
Literal data: 2632483826 bytes
Matched data: 400442 bytes
File list size: 98728269
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 452779
Total bytes received: 1743689596
sent 452779 bytes received 1743689596 bytes 150818.66 bytes/sec total size is 216240165510 speedup is 123.98
This morning the cron job is running again, above is the rsync output to show the nature of the data I'm dealing with. 2M files in 3.2 hours gives 174 files per second.
Quoted 1 line
Can I see dmesg/messages for the time that this was running for so long?
It's running now, cp has run for 21 minutes and copied 806/14938 directories, 5% done so it'll probably take 5 hours. There is nothing in the kernel message log since before the cron job started - that was when I turned off the monitor that is USB attached.
Now cp has been running for 2 hours 25 minutes and copied 7180 directories.
Quoted 6 lines
Quoted 3 lines
The cp -rl usually takes something less than 30 minutes (not long enough for me to even notice) but today cp has been running for 5.5 hours and seems to be about 3/5 done (9000/15000 subdirectories linked).
Sysrq+w while it's running, just one and then wait a while and then another. Thanks,
http://www.coker.com.au/bug/btrfs-slow-cp.gz
I've done a few of those, the log is at the above URL.
Signature · 2 lines
My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/