BTRFS, getting darn slower everyday

Hi,

I have 4 machines, all converted to BTRFS about 6 months ago, now all running Ubuntu Quantal with kernel 3.5.0-17

The matter is that all these machines are now getting slower and slower everyday, every disk access causing the disk to be 100% busy for long periods, to the point that I'm now seriously considering migrating everything back to ext4...

becomes truly unusable.

On one machine, I know have a typical complete boot time to a usable GUI that is over 4 minutes, with the HD still very busy for a couple more minutes afterwards, where it used to be around 35-40 seconds in ext4 !

Is there anything I could do to speed things back (without losing all my snapshots or doubling the size of data on disk)...?

I already had made the move back from BTRFS to ext4 about 18 months ago, I found it had improved so was back to BTRFS, and I wouldn't have to revert back again :-/

Any advice or help greatly appreciated.

TIA.

Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.

Am Sonntag, 7. Oktober 2012 schrieb Swâmi Petaramesh:

I experienced this on my oldest BTRFS filesystems on a ThinkPad T23 as well. Especially with fsync() based workloads like apt / aptitude dist- upgrade. I use eatmydata for these operations for now on this laptop.

A balance didn´t work. I considering to redo those as node and leaf size 16 KiB. But then anything above kernel 3.3 doesn´t resume reliably on the machine and I did not yet get into to finding out why. A kernel bisect would take ages on this machine unless I compile the kernels somewhere else.

Anyway, I think to better see what your issue is, it would be good to have some more information about free space and state of your BTRFS filesystem.

Such as

btrfs fi df (preferably with btrfs tools from Goffredo) btrfs fi show

for starters.

[1] See Goffredo Baroncelli, [PATCH][BTRFS-PROGS][V3] btrfs filesystem df last thursday:

You can pull this change from
http://cassiopea.homelinux.net/git/btrfs-progs-unstable.git
branch
disk_free

Thanks,

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

I don't think I miss any free space ;-)

(From one of my machines, but the others have rather the same architecture...)

# btrfs fi sh
failed to read /dev/sr0
Label: 'BTR_POOL'  uuid: 7266e630-ba72-404c-bfbd-759d3a6ce2e3
    Total devices 1 FS bytes used 278.55GB
    devid    1 size 912.26GB used 464.04GB path /dev/dm-1

Label: 'BTR_BOOT' uuid: b280100e-6a9a-4488-a207-d4e3f001b4c9
Total devices 1 FS bytes used 504.39MB
devid 1 size 1.00GB used 729.50MB path /dev/sda2

# btrfs fi df /boot
System: total=4.00MB, used=4.00KB
Data+Metadata: total=725.50MB, used=504.38MB

# btrfs fi df /
Data: total=426.01GB, used=273.00GB
System, DUP: total=8.00MB, used=56.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=19.00GB, used=5.55GB
Metadata: total=8.00MB, used=0.00

# df -h /boot /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       1,1G  505M  517M  50% /boot
-               913G  285G  602G  33% /

...and I have quite a lot of snapshots, as I use OpenSuSE excellent "snapper" utility :

# snapper list
Type   | #   | Pre # | Date                             | Cleanup       
| Description           | Userdata
-------+-----+-------+----------------------------------+----------------+-----------------------+---------
single | 0   |       |                                  |               
| current               |        
single | 33  |       | mer. 01 août 2012 09:17:01 CEST  | timeline      
| Precise               |        
pre    | 203 |       | lun. 27 août 2012 16:30:22 CEST  | empty-pre-post
| AvUp Precise->QuantA3 |        
post   | 213 | 203   | mar. 28 août 2012 10:15:02 CEST  | empty-pre-post
| ApUp Precise->QuantA3 |        
single | 265 |       | sam. 01 sept. 2012 08:17:02 CEST | timeline      
| QuantA3               |        
single | 443 |       | ven. 21 sept. 2012 00:17:02 CEST | timeline      
| QuantB1 K 3.5.0-15    |        
single | 456 |       | sam. 22 sept. 2012 11:17:01 CEST | timeline      
| timeline              |        
single | 463 |       | dim. 23 sept. 2012 10:17:03 CEST | timeline      
| QuantB1 K 3.5.0-15    |        
single | 473 |       | lun. 24 sept. 2012 00:17:01 CEST | timeline      
| QuantB1 K 3.5.0-15    |        
single | 486 |       | mar. 25 sept. 2012 08:17:01 CEST | timeline      
| QuantB1 K 3.5.0-15    |        
single | 495 |       | mer. 26 sept. 2012 09:17:11 CEST | timeline      
| QuantB1 K 3.5.0-15    |        
single | 505 |       | jeu. 27 sept. 2012 08:17:03 CEST | timeline      
| timeline              |        
single | 520 |       | ven. 28 sept. 2012 09:17:02 CEST | timeline      
| QuantB2 K 3.5.0-15    |        
single | 534 |       | sam. 29 sept. 2012 10:17:01 CEST | timeline      
| QuantB2 K 3.5.0-16    |        
single | 544 |       | dim. 30 sept. 2012 10:17:02 CEST | timeline      
| QuantB2 K 3.5.0-16    |        
single | 556 |       | lun. 01 oct. 2012 09:17:03 CEST  | timeline      
| QuantB2 K 3.5.0-16    |        
single | 568 |       | mar. 02 oct. 2012 08:17:01 CEST  | timeline      
| timeline              |        
single | 582 |       | mer. 03 oct. 2012 10:17:03 CEST  | timeline      
| timeline              |        
single | 591 |       | jeu. 04 oct. 2012 08:17:02 CEST  | timeline      
| timeline              |        
single | 602 |       | jeu. 04 oct. 2012 20:17:01 CEST  | timeline      
| timeline              |        
single | 603 |       | jeu. 04 oct. 2012 21:17:01 CEST  | timeline      
| timeline              |        
single | 604 |       | jeu. 04 oct. 2012 22:17:01 CEST  | timeline      
| timeline              |        
single | 605 |       | jeu. 04 oct. 2012 23:17:01 CEST  | timeline      
| timeline              |        
single | 608 |       | ven. 05 oct. 2012 10:17:04 CEST  | timeline      
| QuantB2 K 3.5.0-17    |        
single | 609 |       | ven. 05 oct. 2012 11:17:01 CEST  | timeline      
| timeline              |        
single | 610 |       | ven. 05 oct. 2012 12:17:02 CEST  | timeline      
| timeline              |        
single | 611 |       | ven. 05 oct. 2012 13:17:02 CEST  | timeline      
| timeline              |        
single | 612 |       | ven. 05 oct. 2012 14:17:01 CEST  | timeline      
| timeline              |        
single | 613 |       | ven. 05 oct. 2012 15:17:01 CEST  | timeline      
| timeline              |        
single | 614 |       | ven. 05 oct. 2012 19:17:02 CEST  | timeline      
| timeline              |        
single | 615 |       | ven. 05 oct. 2012 20:17:01 CEST  | timeline      
| timeline              |        
single | 616 |       | ven. 05 oct. 2012 22:18:48 CEST  | timeline      
| timeline              |        
single | 617 |       | ven. 05 oct. 2012 23:17:01 CEST  | timeline      
| timeline              |        
single | 618 |       | sam. 06 oct. 2012 10:17:03 CEST  | timeline      
| timeline              |        
single | 619 |       | sam. 06 oct. 2012 11:17:01 CEST  | timeline      
| timeline              |        
single | 620 |       | dim. 07 oct. 2012 00:17:03 CEST  | timeline      
| timeline              |        
single | 621 |       | dim. 07 oct. 2012 10:17:02 CEST  | timeline      
| timeline              |        
single | 622 |       | dim. 07 oct. 2012 11:17:02 CEST  | timeline      
| timeline              |        
single | 623 |       | dim. 07 oct. 2012 12:17:01 CEST  | timeline      
| timeline              |        
single | 624 |       | dim. 07 oct. 2012 13:17:01 CEST  | timeline      
| timeline              |        
single | 625 |       | dim. 07 oct. 2012 14:17:01 CEST  | timeline      
| timeline              |        
single | 626 |       | dim. 07 oct. 2012 15:17:01 CEST  | timeline      
| timeline              |

...or if you prefer, something like :

# btrfs su li /
ID 256 top level 5 path UBUNTU
ID 259 top level 5 path UBUNTU/@
ID 261 top level 5 path UBUNTU/@tmp
ID 262 top level 5 path UBUNTU/@home
ID 264 top level 5 path LMDE
ID 265 top level 5 path LMDE/@
ID 267 top level 5 path LMDE/@tmp
ID 268 top level 5 path LMDE/@home
ID 270 top level 5 path MINT
ID 271 top level 5 path MINT/@
ID 273 top level 5 path MINT/@tmp
ID 274 top level 5 path MINT/@home
ID 276 top level 5 path BODHI
ID 277 top level 5 path BODHI/@
ID 278 top level 5 path BODHI/@var
ID 279 top level 5 path BODHI/@tmp
ID 280 top level 5 path BODHI/@home
ID 282 top level 5 path STORAGE
ID 471 top level 5 path DATA/VBOX_HIDDEN
ID 472 top level 5 path DATA/VBOX_VMS
ID 505 top level 5 path LMDE/@/.snapshots
ID 506 top level 5 path LMDE/@home/.snapshots
ID 509 top level 5 path MINT/@/.snapshots
ID 510 top level 5 path MINT/@home/.snapshots
ID 515 top level 5 path UBUNTU/@/.snapshots
ID 516 top level 5 path UBUNTU/@home/.snapshots
ID 517 top level 5 path STORAGE/.snapshots
ID 628 top level 5 path DATA
ID 629 top level 5 path DATA/BIG
ID 630 top level 5 path DATA/BIG/michel
ID 1339 top level 5 path UBUNTU/@/.snapshots/33/snapshot
ID 1340 top level 5 path UBUNTU/@home/.snapshots/33/snapshot
ID 1341 top level 5 path STORAGE/.snapshots/35/snapshot
ID 1384 top level 5 path LMDE/@/.snapshots/2/snapshot
ID 1385 top level 5 path LMDE/@home/.snapshots/2/snapshot
ID 1387 top level 5 path MINT/@/.snapshots/2/snapshot
ID 1388 top level 5 path MINT/@home/.snapshots/2/snapshot
ID 1495 top level 5 path DATA/VBOX_HIDDEN/.snapshots
ID 1621 top level 5 path MINT/@/.snapshots/3/snapshot
ID 1622 top level 5 path MINT/@home/.snapshots/3/snapshot
ID 1651 top level 5 path MINT/@/.snapshots/4/snapshot
ID 1652 top level 5 path MINT/@home/.snapshots/4/snapshot
ID 1654 top level 5 path MINT/@/.snapshots/5/snapshot
ID 1655 top level 5 path MINT/@home/.snapshots/5/snapshot
ID 1657 top level 5 path MINT/@/.snapshots/6/snapshot
ID 1658 top level 5 path MINT/@home/.snapshots/6/snapshot
ID 1706 top level 5 path DATA/VBOX_HIDDEN/.snapshots/7/snapshot
ID 1874 top level 5 path UBUNTU/@home/.snapshots/203/snapshot
ID 1875 top level 5 path UBUNTU/@/.snapshots/203/snapshot
ID 1903 top level 5 path UBUNTU/@home/.snapshots/213/snapshot
ID 1904 top level 5 path UBUNTU/@/.snapshots/213/snapshot
ID 1980 top level 5 path LMDE/@/.snapshots/3/snapshot
ID 1981 top level 5 path LMDE/@home/.snapshots/3/snapshot
ID 1983 top level 5 path MINT/@/.snapshots/7/snapshot
ID 1984 top level 5 path MINT/@home/.snapshots/7/snapshot
ID 2058 top level 5 path UBUNTU/@/.snapshots/265/snapshot
ID 2059 top level 5 path UBUNTU/@home/.snapshots/263/snapshot
ID 2060 top level 5 path STORAGE/.snapshots/268/snapshot
ID 2184 top level 5 path LMDE/@/.snapshots/4/snapshot
ID 2185 top level 5 path LMDE/@home/.snapshots/4/snapshot
ID 2513 top level 5 path LMDE/@home/.snapshots/5/snapshot
ID 2514 top level 5 path LMDE/@/.snapshots/5/snapshot
ID 2515 top level 5 path LMDE/@/.snapshots/6/snapshot
ID 2516 top level 5 path LMDE/@home/.snapshots/6/snapshot
ID 2524 top level 5 path LMDE/@/.snapshots/9/snapshot
ID 2525 top level 5 path LMDE/@home/.snapshots/9/snapshot
ID 2527 top level 5 path LMDE/@/.snapshots/10/snapshot
ID 2528 top level 5 path LMDE/@home/.snapshots/10/snapshot
ID 2530 top level 5 path LMDE/@/.snapshots/11/snapshot
ID 2531 top level 5 path LMDE/@home/.snapshots/11/snapshot
ID 2533 top level 5 path LMDE/@/.snapshots/12/snapshot
ID 2534 top level 5 path LMDE/@home/.snapshots/12/snapshot
ID 2536 top level 5 path LMDE/@/.snapshots/13/snapshot
ID 2537 top level 5 path LMDE/@home/.snapshots/13/snapshot
ID 2539 top level 5 path LMDE/@/.snapshots/14/snapshot
ID 2540 top level 5 path LMDE/@home/.snapshots/14/snapshot
ID 2542 top level 5 path LMDE/@/.snapshots/15/snapshot
ID 2543 top level 5 path LMDE/@home/.snapshots/15/snapshot
ID 2545 top level 5 path LMDE/@/.snapshots/16/snapshot
ID 2546 top level 5 path LMDE/@home/.snapshots/16/snapshot
ID 2548 top level 5 path LMDE/@/.snapshots/17/snapshot
ID 2549 top level 5 path LMDE/@home/.snapshots/17/snapshot
ID 2551 top level 5 path LMDE/@/.snapshots/18/snapshot
ID 2552 top level 5 path LMDE/@home/.snapshots/18/snapshot
ID 2554 top level 5 path LMDE/@/.snapshots/19/snapshot
ID 2555 top level 5 path LMDE/@home/.snapshots/19/snapshot
ID 2557 top level 5 path LMDE/@/.snapshots/20/snapshot
ID 2558 top level 5 path LMDE/@home/.snapshots/20/snapshot
ID 2560 top level 5 path LMDE/@/.snapshots/21/snapshot
ID 2561 top level 5 path LMDE/@home/.snapshots/21/snapshot
ID 2563 top level 5 path LMDE/@/.snapshots/22/snapshot
ID 2564 top level 5 path LMDE/@home/.snapshots/22/snapshot
ID 2566 top level 5 path LMDE/@/.snapshots/23/snapshot
ID 2567 top level 5 path LMDE/@home/.snapshots/23/snapshot
ID 2569 top level 5 path LMDE/@/.snapshots/24/snapshot
ID 2570 top level 5 path LMDE/@home/.snapshots/24/snapshot
ID 2572 top level 5 path LMDE/@/.snapshots/25/snapshot
ID 2573 top level 5 path LMDE/@home/.snapshots/25/snapshot
ID 2575 top level 5 path LMDE/@/.snapshots/26/snapshot
ID 2576 top level 5 path LMDE/@home/.snapshots/26/snapshot
ID 2578 top level 5 path LMDE/@/.snapshots/27/snapshot
ID 2579 top level 5 path LMDE/@home/.snapshots/27/snapshot
ID 2581 top level 5 path LMDE/@/.snapshots/28/snapshot
ID 2582 top level 5 path LMDE/@home/.snapshots/28/snapshot
ID 2584 top level 5 path LMDE/@home/.snapshots/29/snapshot
ID 2585 top level 5 path LMDE/@/.snapshots/29/snapshot
ID 2586 top level 5 path LMDE/@/.snapshots/30/snapshot
ID 2587 top level 5 path LMDE/@home/.snapshots/30/snapshot
ID 2589 top level 5 path LMDE/@/.snapshots/31/snapshot
ID 2590 top level 5 path LMDE/@home/.snapshots/31/snapshot
ID 2676 top level 5 path UBUNTU/@/.snapshots/443/snapshot
ID 2677 top level 5 path UBUNTU/@home/.snapshots/442/snapshot
ID 2678 top level 5 path STORAGE/.snapshots/472/snapshot
ID 2682 top level 5 path MINT/@/.snapshots/8/snapshot
ID 2683 top level 5 path MINT/@home/.snapshots/8/snapshot
ID 2718 top level 5 path UBUNTU/@/.snapshots/456/snapshot
ID 2719 top level 5 path UBUNTU/@home/.snapshots/455/snapshot
ID 2720 top level 5 path STORAGE/.snapshots/486/snapshot
ID 2739 top level 5 path UBUNTU/@/.snapshots/463/snapshot
ID 2740 top level 5 path UBUNTU/@home/.snapshots/462/snapshot
ID 2741 top level 5 path STORAGE/.snapshots/493/snapshot
ID 2769 top level 5 path UBUNTU/@/.snapshots/473/snapshot
ID 2770 top level 5 path UBUNTU/@home/.snapshots/472/snapshot
ID 2771 top level 5 path STORAGE/.snapshots/503/snapshot
ID 2808 top level 5 path UBUNTU/@/.snapshots/486/snapshot
ID 2809 top level 5 path UBUNTU/@home/.snapshots/485/snapshot
ID 2810 top level 5 path STORAGE/.snapshots/516/snapshot
ID 2835 top level 5 path UBUNTU/@/.snapshots/495/snapshot
ID 2836 top level 5 path UBUNTU/@home/.snapshots/494/snapshot
ID 2837 top level 5 path STORAGE/.snapshots/525/snapshot
ID 2865 top level 5 path UBUNTU/@/.snapshots/505/snapshot
ID 2866 top level 5 path UBUNTU/@home/.snapshots/504/snapshot
ID 2867 top level 5 path STORAGE/.snapshots/535/snapshot
ID 2910 top level 5 path UBUNTU/@/.snapshots/520/snapshot
ID 2911 top level 5 path UBUNTU/@home/.snapshots/519/snapshot
ID 2912 top level 5 path STORAGE/.snapshots/550/snapshot
ID 2950 top level 5 path UBUNTU/@home/.snapshots/532/snapshot
ID 2951 top level 5 path STORAGE/.snapshots/563/snapshot
ID 2952 top level 5 path UBUNTU/@/.snapshots/534/snapshot
ID 2982 top level 5 path UBUNTU/@/.snapshots/544/snapshot
ID 2983 top level 5 path UBUNTU/@home/.snapshots/543/snapshot
ID 2984 top level 5 path STORAGE/.snapshots/574/snapshot
ID 3018 top level 5 path UBUNTU/@/.snapshots/556/snapshot
ID 3019 top level 5 path UBUNTU/@home/.snapshots/555/snapshot
ID 3020 top level 5 path STORAGE/.snapshots/586/snapshot
ID 3045 top level 5 path MINT/@/.snapshots/9/snapshot
ID 3046 top level 5 path MINT/@home/.snapshots/9/snapshot
ID 3060 top level 5 path UBUNTU/@/.snapshots/568/snapshot
ID 3061 top level 5 path UBUNTU/@home/.snapshots/567/snapshot
ID 3062 top level 5 path STORAGE/.snapshots/600/snapshot
ID 3069 top level 5 path LMDE/@/.snapshots/32/snapshot
ID 3070 top level 5 path LMDE/@home/.snapshots/32/snapshot
ID 3105 top level 5 path UBUNTU/@/.snapshots/582/snapshot
ID 3106 top level 5 path UBUNTU/@home/.snapshots/581/snapshot
ID 3107 top level 5 path STORAGE/.snapshots/615/snapshot
ID 3132 top level 5 path UBUNTU/@/.snapshots/591/snapshot
ID 3133 top level 5 path UBUNTU/@home/.snapshots/590/snapshot
ID 3134 top level 5 path STORAGE/.snapshots/624/snapshot
ID 3138 top level 5 path LMDE/@/.snapshots/33/snapshot
ID 3139 top level 5 path LMDE/@home/.snapshots/33/snapshot
ID 3168 top level 5 path UBUNTU/@/.snapshots/602/snapshot
ID 3169 top level 5 path UBUNTU/@home/.snapshots/601/snapshot
ID 3170 top level 5 path STORAGE/.snapshots/636/snapshot
ID 3171 top level 5 path UBUNTU/@/.snapshots/603/snapshot
ID 3172 top level 5 path UBUNTU/@home/.snapshots/602/snapshot
ID 3173 top level 5 path STORAGE/.snapshots/637/snapshot
ID 3174 top level 5 path UBUNTU/@/.snapshots/604/snapshot
ID 3175 top level 5 path UBUNTU/@home/.snapshots/603/snapshot
ID 3176 top level 5 path STORAGE/.snapshots/638/snapshot
ID 3177 top level 5 path UBUNTU/@/.snapshots/605/snapshot
ID 3178 top level 5 path UBUNTU/@home/.snapshots/604/snapshot
ID 3179 top level 5 path STORAGE/.snapshots/639/snapshot
ID 3186 top level 5 path UBUNTU/@/.snapshots/608/snapshot
ID 3187 top level 5 path UBUNTU/@home/.snapshots/607/snapshot
ID 3188 top level 5 path STORAGE/.snapshots/642/snapshot
ID 3189 top level 5 path UBUNTU/@/.snapshots/609/snapshot
ID 3190 top level 5 path UBUNTU/@home/.snapshots/608/snapshot
ID 3191 top level 5 path STORAGE/.snapshots/643/snapshot
ID 3192 top level 5 path UBUNTU/@/.snapshots/610/snapshot
ID 3193 top level 5 path UBUNTU/@home/.snapshots/609/snapshot
ID 3194 top level 5 path STORAGE/.snapshots/644/snapshot
ID 3195 top level 5 path UBUNTU/@/.snapshots/611/snapshot
ID 3196 top level 5 path UBUNTU/@home/.snapshots/610/snapshot
ID 3197 top level 5 path STORAGE/.snapshots/645/snapshot
ID 3198 top level 5 path UBUNTU/@/.snapshots/612/snapshot
ID 3199 top level 5 path UBUNTU/@home/.snapshots/611/snapshot
ID 3200 top level 5 path STORAGE/.snapshots/646/snapshot
ID 3201 top level 5 path UBUNTU/@/.snapshots/613/snapshot
ID 3202 top level 5 path UBUNTU/@home/.snapshots/612/snapshot
ID 3203 top level 5 path STORAGE/.snapshots/647/snapshot
ID 3204 top level 5 path UBUNTU/@/.snapshots/614/snapshot
ID 3205 top level 5 path UBUNTU/@home/.snapshots/613/snapshot
ID 3206 top level 5 path STORAGE/.snapshots/648/snapshot
ID 3207 top level 5 path UBUNTU/@/.snapshots/615/snapshot
ID 3208 top level 5 path UBUNTU/@home/.snapshots/614/snapshot
ID 3209 top level 5 path STORAGE/.snapshots/649/snapshot
ID 3210 top level 5 path UBUNTU/@/.snapshots/616/snapshot
ID 3211 top level 5 path UBUNTU/@home/.snapshots/615/snapshot
ID 3212 top level 5 path STORAGE/.snapshots/650/snapshot
ID 3213 top level 5 path UBUNTU/@/.snapshots/617/snapshot
ID 3214 top level 5 path UBUNTU/@home/.snapshots/616/snapshot
ID 3215 top level 5 path STORAGE/.snapshots/651/snapshot
ID 3216 top level 5 path UBUNTU/@/.snapshots/618/snapshot
ID 3217 top level 5 path UBUNTU/@home/.snapshots/617/snapshot
ID 3218 top level 5 path STORAGE/.snapshots/652/snapshot
ID 3219 top level 5 path UBUNTU/@/.snapshots/619/snapshot
ID 3220 top level 5 path UBUNTU/@home/.snapshots/618/snapshot
ID 3221 top level 5 path STORAGE/.snapshots/653/snapshot
ID 3222 top level 5 path UBUNTU/@/.snapshots/620/snapshot
ID 3223 top level 5 path UBUNTU/@home/.snapshots/619/snapshot
ID 3224 top level 5 path STORAGE/.snapshots/654/snapshot
ID 3225 top level 5 path UBUNTU/@/.snapshots/621/snapshot
ID 3226 top level 5 path UBUNTU/@home/.snapshots/620/snapshot
ID 3227 top level 5 path STORAGE/.snapshots/655/snapshot
ID 3228 top level 5 path UBUNTU/@/.snapshots/622/snapshot
ID 3229 top level 5 path UBUNTU/@home/.snapshots/621/snapshot
ID 3230 top level 5 path STORAGE/.snapshots/656/snapshot
ID 3231 top level 5 path UBUNTU/@/.snapshots/623/snapshot
ID 3232 top level 5 path UBUNTU/@home/.snapshots/622/snapshot
ID 3233 top level 5 path STORAGE/.snapshots/657/snapshot
ID 3234 top level 5 path UBUNTU/@/.snapshots/624/snapshot
ID 3235 top level 5 path UBUNTU/@home/.snapshots/623/snapshot
ID 3236 top level 5 path STORAGE/.snapshots/658/snapshot
ID 3237 top level 5 path UBUNTU/@/.snapshots/625/snapshot
ID 3238 top level 5 path UBUNTU/@home/.snapshots/624/snapshot
ID 3239 top level 5 path STORAGE/.snapshots/659/snapshot
ID 3240 top level 5 path UBUNTU/@/.snapshots/626/snapshot
ID 3241 top level 5 path UBUNTU/@home/.snapshots/625/snapshot
ID 3242 top level 5 path STORAGE/.snapshots/660/snapshot

Kind regards.

Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.

Am Sonntag, 7. Oktober 2012 schrieb Swâmi Petaramesh:

Well I could I know this beforehand?

Hmmm. Looks fine for me.

Where is this volume pool located on? On which drive(s)?

[…]

This could be 100 or more subvolumes / snapshots.

Maybe slowness could be related to this one.

It would be good to have some capture of this slowness with

vmstat 10

iostat -xd 10 /your/device

or something like that. Especially for the later, cause it shows the count of reads and writes.

Also how much RAM has the machine? What does free -m / cat /proc/meminfo tell.

Other than that I leave this too BTRFS developers to answer to. I think that this has become easier by the data you provided.

Thanks,

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Hi Martin,

All the concerned machines are laptops with a single physical HD...

That's a good estimation... I'm not sure if there is any theoretical reason why the number of snapshots of a given volume might slow down access to this volume ?

I will try to give you some figures. But when the machine starts really crawling, it becomes very hard to start any new program unless it's "finished being slow", so that's not so easy...

The 4 concerned machines have either 4GB or 2GB RAM. None is heavily loaded as far as RAM is concerned (just booting into a standard Ubuntu GUI and then accessing emailn, browser, office applications etc.)

Kind regards.

Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.

Absolutely. COW snapshots cause severe fragmentation (it's in their nature). "Easily" reproducible by snapshotting an fs every day. Been there, had that.

Am Sonntag, 9. Dezember 2012 schrieb Jan Engelhardt:

So such a workload is only possible with pleasing speed with SSD?

I have about 5-10 snapshots on my backup 2TB eSATA disk which I currently rsync to. This seems to work quite well still. But I have no idea how that would turn out with 100 snapshots or more. And about 1 TB is free, so at least I´d expect no serious freespace fragmentation like on the BTRFS in my SSD (extra post for that one).

Ciao,

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

CoW filesystem incurs fragmentation by its nature, not specifically snapshots. Even without snapshots, rewriting portions of existing files will write the new blocks not over the original ones, but elsewhere, thus increasing fragmentation.

With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

Is it to expect that somewhere in the future, BTRFS will be able to defragment itself without duplicating snapshot data ?

Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.

In the presence of snapshots that are modified, no, it's impossible
to fully defrag all the files.

Consider a file, AAABBBCCC, which is snapshotted, and then the
middle third is rewritten, so the snapshot shows AAAZZZCCC. You now have two files which share their first and last sections, but the centre section is different. One or other of those files *must* be fragmented -- you can't eliminate fragmentation here.

If you mean, will there be a defrag that does as well as it can
despite the limitation above, and which doesn't split snapshots into separate copies, then yes -- there's already patches out for that, but they've had a couple of problems.

Hugo.

=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
               --- emacs: Emacs Makes A Computer Slow. ---               

Of course, but I would agree with the poster that it would be important to "partially" defrag all the files, avoiding at least unneeded fragmentation... At least the fragmentation generated by normal writes

Right, of course.

In the "normal" case, the originals are deleted at some point, so the length of the CoW "chain" should be bound. However, with snapshots, there is, I wager to say, no limit to the chain length, so defragmenting a file also becomes more work.

One can of course use rsync without --inplace, which does a full file copy-up if the file has changed, which essentially disables CoW, and thus reduces fragmentation. Unfortunately, not all copy operations on one's machine do use rsync, so perhaps it would be nice to have this as a subvolume mount option as well.

Am Sonntag, 7. Oktober 2012 schrieb Swâmi Petaramesh:

As told in my previous post its better to first understand the problem. What might help here, additionally to what I suggested already, is some output of

vmstat 1

and

iostat -dx 1 /dev/sda

(or maybe even better 5 or 10 seconds interval)

where /dev/sda is device or better partition where the workload happens.

Also the amount of snapshots in use would be helpful I bet.

Ciao,

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

1. Convert to a 16k or 32k leafsize.
2. defragment (each non-trivial file) every now and again [eg. find / -size +16k -type f -exec btrfs filesystem defragment -vf '{}' ';' ]
3. Note that btrfs in kernel 3.6 (vs 3.5) does appear to be quite a bit quicker. I'm running Siduction kernels whilst Debian is on freeze at 3.2. I found 3.5 really quite awful - not sure why.

Make sure you discount other factors.

I'm not sure defrag goes so far as to consolidate files by moving bits out of the way yet, so you may find that the number of fragments in a file never reaches one.

Bear with it pls.

Hi again ;-)

How should I do this ? Can I do this on a live FS, and isn't this going to double my on-disk used space (I have active snapshots...)

I believed that the "autodefrag" mount option was supposed to avoid the need for manual defrags...?

Plus, with respect to snapshots, isn't this going to increase a lot my used disk space ?

I'm not sure that "a bit" quicker will be enough, when I see my BTRFS is currently about 8x slower than what ext4 user to...

Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.

Am Sonntag, 7. Oktober 2012 schrieb Swâmi Petaramesh:

I think you need to backup, reformat and restore from backup for now.

In theory yes. But AFAIR it is currently limited to certain usecases.

I don´t think so, but I will leave this to a developer to answer.

How did you measure this? Is this just the boot times you mentioned in the original article?

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

IIRC the defrag is not [fully] snapshot aware, so de-frag-ging a snapshot-ted file should break the COW link. Liu Bo is working to patch on that: see " Btrfs: use flag EXTENT_DEFRAG for snapshot-aware defrag" dated 2012/09/17.

Please could you clarify if you are using the "autodefrag" options when you have the performance problem ? Are you in position to to reduce the number of snapshot from hundreds to few ?

I use autodefrag on all volumes systematically, except on volumes that I use for really big files that would always be defragmenting (i.e. virtual machines, but I seldom use virtual machines, and my performance problem is general, I don't take VMs into consideration in this respect...)

I might do this as an experiment, but it would defeat one of the prominent purposes for which I use BTRFS, and the number of snapshots would grow immediately again, as I use the excellent OpenSuSE "snapper" tool, that makes a snapshot every single hour...

Kind regards.

Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.

The autodefrag option is per filesystem not per subvolume. The settings of the first subvolueme is used also for the other ones.

So, because you mount the same filesystem both on

subvol=UBUNTU/@,space_cache,autodefrag,compress=lzo,relatime 0 0

and

subvol=DATA/VBOX_HIDDEN,space_cache,compress=lzo,noatime 0 0

subvol=DATA/VBOX_VMS,space_cache,compress=lzo,noatime 0 0

the autodefrag option is enabled also for DATA/VBOX_HIDDEN and DATA/VBOM_VMS

I am not suggesting that as solution, but this would help to investigate the problem.

If you don't have any needs of the snapshot, you can delete it, wait that the cleaner
kernel thread does its job (it could require a bit of time), then reboot the machine.
If I am right the performance should goes high.

I fear that both the combination of autodefrag and the high number of snapshot could
be the root-cause of the the bad performance.

Ciao
Goffredo

Uh !

So there is no interest in creating several subvols, some for which files should be autodefragged, and some not ? That's to bad :-(

As a test, I will try to remove *most* of my snapshots and see if it helps...

Thanks for the suggestion !

Kind regards.

Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.

Hi again Goffredo,

I've removed, on one of my machines, all snapshots but three per subvol (keeping the oldests and newest), going from about 30 per subvol to 3, for the complete filesystem from 120+ to about a dozen.

Then I let btrfs-cleaner do its job

After that the machine boots to GUI in a bit less than 2 minutes, where it was more than 4 minutes previously.

The machine now seems much more reactive and swift.

So it seems that the number or active snapshots (or is it the number of subvols whatsoever ??) dramatically impacts performance...

Thanks for the suggestion.

Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.

Does the autodefrag options still alive ?
I believe that the snapshot is quite cheap, except if you update the shared files one at time. Which should be the case of the autodefrag. But it is only a my supposition....

Could you please try to avoid the autodefrag option in a machine with an high number of snapshot ? I am curios...

No way. 4 machines on each of which 2 to 4 different OSes are sharing the same BTRFS volume !

If I ever need to reformat/reinstall all this, the new format won't be BTRFS ! I would be pissed off to a point where I would stick to either ext4 or ZFS, all of which work nicely (I have a complex system that has been running very fine on ZFS for 1.5 year)...

No highly scientific measurement. I don't live with benchmarks, but with the impression that my machine is nicely swift or painfully slow.

When it comes to 4 minutes for boot, 40 seconds for opening a brower, 1+ minute for Thunderbird... 15 minutes for installing an innocent set of updates... I'm under the impression that it's painfully slow.

Kind regards.

Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.

Here is a graph to reinforce what Alex said about performance after 3.5.x. This graph was posted in #btrfs on freenode. At the end of the graph, the poster said that he upgraded from a 3.5.x to a 3.6.x

https://pipebreaker.pl/dump/loadl.png

I noticed the same thing going from 3.5.0 to 3.7.0rc2; system usage is much lower. Before this I would often find huge lags and then check and see btrfs-cleaner, btrfs-transactionhelpersomething, etc. taking up 100% io. So if your system usage is high, this will help.

Also possibly not relevant for you: on my desktop, (btrfs or ext4, with consumer disks) I find that deadline scheduler and nr_requests of 8 is vastly superior for reducing GUI lags. (default is scheduler cfq and 128 nr_requests). But for servers, I don't know what to recommend.

Am Sonntag, 7. Oktober 2012 schrieb Swâmi Petaramesh:

I forgot to mention mount options? Which one do you use?

Please always include all details that may be important.

space_cache, inode_cache?

Thanks,

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

root@tethys:/etc# grep btrfs fstab
/dev/VG1/BTR_POOL    /    btrfs   
subvol=UBUNTU/@,space_cache,autodefrag,compress=lzo,relatime        0 0
/dev/sda2        /boot    btrfs   
subvol=UBUNTU/@boot,space_cache,autodefrag,relatime            0 0
/dev/VG1/BTR_POOL    /tmp    btrfs   
subvol=UBUNTU/@tmp,space_cache,autodefrag,compress=lzo,relatime        0 0
/dev/VG1/BTR_POOL    /home    btrfs   
subvol=UBUNTU/@home,space_cache,autodefrag,compress=lzo,relatime    0 0
/dev/VG1/BTR_POOL    /storage    btrfs   
subvol=STORAGE,space_cache,autodefrag,compress=lzo,relatime    0 0
/dev/VG1/BTR_POOL    /data/BIG        btrfs   
subvol=DATA/BIG,space_cache,compress=lzo,noatime    0 0
/dev/VG1/BTR_POOL    /data/VBOX_HIDDEN    btrfs   
subvol=DATA/VBOX_HIDDEN,space_cache,compress=lzo,noatime    0 0
/dev/VG1/BTR_POOL    /data/VBOX_VMS        btrfs   
subvol=DATA/VBOX_VMS,space_cache,compress=lzo,noatime    0 0

Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.

Am Sonntag, 7. Oktober 2012 schrieb Swâmi Petaramesh:

Hmmm, is the slowness on all of these?

For the VBOX volumes I would have a possible explaination for slowness.

Unless the subvolume is mounted with nodatacow, which it isn´t according to above output, every write to the VM image goes to a new location. Thus this increases fragmentation. While autodefrag might mitigate some of these, it may not be enough. AFAIK its only efficient for certain use cases. So or so it does not appear tha tthe VBOX volumes use autodefrag.

Could you try filefrag -v on some of the bigger VM image files?

I also wonder about the absence of inode_cache, but this could either be default by now or BTRFS remembers whether what has initially been set. But you could try this one. Especially on the filesystems with lots of inodes.

Thanks,

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Can you get sysrq+w when you are seeing slowness? Usually bootup slow times means you don't have space_cache enabled or your cache is being evicted for some reason, can you check dmesg after bootup for messages related to space cache? Thanks,

Josef

I have a few :

Oct  8 15:27:26 tethys kernel: [16174.736603] btrfs: free space inode
generation (0) did not match free space cache generation (106988)
Oct  8 15:27:27 tethys kernel: [16175.976784] btrfs: free space inode
generation (0) did not match free space cache generation (30727)
Oct  8 15:27:28 tethys kernel: [16176.420719] btrfs: free space inode
generation (0) did not match free space cache generation (48040)
Oct  8 15:27:28 tethys kernel: [16176.710972] btrfs: free space inode

generation (0) did not match free space cache generation (30745)

...in syslog, but that's about all... and not during boot...

I used to have much much more of these in syslog, but solved it by booting once with the "clear_cache" option, that caused boot to be extremely slow, but seemed to fix it...

(Remember I have such issues on several machines, it is highly improbable that all of them would get their cache ignored...?)

Kind regards.

Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.

Well what happens is on a actually used fs it ends up being more fragmented than the amount we're allowed to preallocate for our space cache, and so we don't write anything out, so it's very likely that all of your machines could be hitting that. I put a patch into 3.6 to increase the cache size so that wouldn't happen as much, perhaps move to 3.6 and see if you see some improvements? Thanks,

Josef