abysmal performance

Currently on
commit 7cf96da3ec7ca225acf4f284b0e904a1f5f98821

Author: Tsutomu Itoh <[email protected]>
Date:   Mon Apr 25 19:43:53 2011 -0400

Btrfs: cleanup error handling in inode.c

merged into 2.6.38.4

I'm on a btrfs filesystem that has been used for some time. Let's say nine months. Very recently I noticed performance getting worse and worse. Most of the time it feels as if the system is just busy with iowait. Write and read performance during random access is mostly around 2MB/s, sometimes 1MB/s or slower. It's better for big files which can be read with about 6-9MB/s. The disk is a reasonably recent SATA disk (WDC_WD3200BEVT) so 30MB/s or 40MB/s linear reading should not be a problem.

rootfs 291G 242G 35G 88% /

I tried btrfs filesystem defragment -v / but did not notice any improvement after that.

Is this a known phenomenon? :-)

Excerpts from John Wyzer's message of 2011-04-29 10:46:08 -0400:

Sounds like you're hitting fragmentation, which we can confirm with latencytop. Please run latencytop while you're seeing poor performance and take a look at where you're spending most of your time.

-chris

y nine

/s,

ad with about

so 30MB/s

mprovement

ance

" in

Also, please note that 'btrfs filesystem defragment -v /' will defragment the directory structure, but not the files.

See:
https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#Defragmenting_a_dir=
ectory_doesn.27t_work

To defragment your entire volume, you'll need a command like:

# for file in $(find <PATH/TO/BTRFS/VOL/> -type f); do btrfs
filesystem defragment ${file}; done

There's also a similar command in the FAQ referenced above.

If you just want to see your fragmentation you can use the 'filefrag' program from e2fsprogs:

# for file in $(find <PATH/TO/BTRFS/VOL/> -type f); do filefrag
${file}; done | sort -n -k 2 | less
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Excerpts from Mitch Harder's message of Sat Apr 30 19:33:16 +0200 2011:

[...]

Thanks, I'm doing something like that at the moment (sorted the whole system according to atimes and mtimes and started defragmenting in order of recent access...)

However at this speed this will never end.

I'm willing to let it run some more nights however to see whether there will be an effect in the end.

By the way: does it make a difference to run defrag on one file at a time or on more?
At the moment I'm doing 100 files/directories per btrfs call...

Hmm.
Tried it and it gives me about 500000 lines of

FIBMAP: Invalid argument

and then:

large_file: 1 extent found

Is that the way it is supposed to work?
Just asking because this was part of a vmware disk image. Both the virtual machine and the rest of the host system are almost unusable once the VM ist started (even more unusable than without vmware :-D )

No. It sounds like the filefrag command is getting confused in the virtual environment.

Excerpts from Mitch Harder's message of Sun May 01 00:16:53 +0200 2011:

Misunderstanding :-) I tried filefrag _on_ a vmware disk image, not inside a virtual machine.
The whole btrfs story here is on a real machine.

Excerpts from John Wyzer's message of 2011-04-30 18:33:20 -0400:

The most important files to defrag are going to be your internal firefox files (I think in .mozilla), your sup database (in .sup) and your vmware images. I would definitely suggest that you try to narrow down if vmware is making the machine seem much slower after the defrag is done.

It might make sense to run the nocow ioctl on your vmware images, they are probably triggering lots of seeks.

You'll notice the machine is much slower after a reboot, this is because we have to do a lot of IO to recache the extent allocation tree. If you pull from the btrfs-unstable tree, you can use mount -o space_cache, which cuts down on the reading after a reboot dramatically.

If none of this works, we'll look at the files that you're fsyncing. That seems to be the bulk of your latencies.

-chris

Excerpts from John Wyzer's message of 2011-04-30 18:33:20 -0400:

Older filefrag uses fibmap, which we don't support (we use fiemap instead). If you update your e2fsprogs you should get a newer filefrag.

-chris

Suggest:

find /path/to/btrfs/vol -type f -exec btrfs filesystem defragment '{}' ';'

find /path/to/btrfs/vol -type f -exec filefrag '{}' ';'

If either command can take multiple filenames as parameters, then e.g.:

find /path/to/btrfs/vol -type f -execdir filefrag '+'

(significantly better because not a fork per file)

//Peter

Peter Stuge <[email protected]> wrote:

Hey,

defragging btrfs does not seem to work for me. I have run the filefrag command over the whole fs and (manually) tried to defrag a few heavily fragmented files, but I don't get it to work (it still has the same number of extends and they are horrently uncorrelated)

root@schleppi:~# filefrag
/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1
/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1: 72 extents found
root@schleppi:~# btrfs filesystem defrag
/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1
root@schleppi:~# filefrag
/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1
/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1: 72 extents found

I'm using Ubuntu Natty (2.6.38.4) and tried both btrfs-tools from Natty (201006xx) and from Debian experimental (git from 20101101). Both show the same symptoms. I don't think fragmentation is bad on this box (due to having an SSD), but my system at home is getting dog slow and I'd like to try that when I come home end of the week.

Best Regards,
Bernhard

You're not using compression on that filesystem are you? If so, be aware that the number of extents isn't going to change after defragmentation, although you should find that the _locations_ of those extents is contiguous.

Am 03.05.2011 13:00, schrieb cwillu:

Hi,

Actually I do run compression, but the location is not continguous afterwards (even after btrfs filesystem sync / or using -f):

root@schleppi:~# btrfs filesystem defrag -f
/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1
root@schleppi:~# btrfs filesystem sync /FSSync '/'
root@schleppi:~# filefrag -v
/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1
Filesystem type is: 9123683e
File size of /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 is
9361528 (2286 blocks, blocksize 4096)
 ext logical physical expected length flags
   0       0  4542111              32
   1      32  4542134  4542142     32
   2      64  4573263  4542165     32
   3      96  4573285  4573294     32
   4     128  4579639  4573316     32
   5     160  4579664  4579670     32
   6     192  4581178  4579695     32
   7     224  4579811  4581209     32
   8     256  4579836  4579842     32
   9     288  4579861  4579867     32
  10     320  4579884  4579892     32
  11     352  4580698  4579915     32
  12     384  4580720  4580729     32
  13     416  4580746  4580751     32
  14     448  4580768  4580777     32
  15     480  4580793  4580799     32
  16     512  4580819  4580824     32
  17     544  4581238  4580850     32
  18     576  4600396  4581269     32
  19     608  4600422  4600427     32
  20     640  4600447  4600453     32
  21     672  4600472  4600478     32
  22     704  4600498  4600503     32
  23     736  4600523  4600529     32
  24     768  4601483  4600554     32
  25     800  4601509  4601514     32
  26     832  4601534  4601540     32
  27     864  4601558  4601565     32
  28     896  4601583  4601589     32
  29     928  4601608  4601614     32
  30     960  4618420  4601639     32
  31     992  4618443  4618451     32
  32    1024  4541221  4618474     32
  33    1056  4618463  4541252     32
  34    1088  4618485  4618494     32
  35    1120  4618505  4618516     32
  36    1152  4579536  4618536     32
  37    1184  4579688  4579567     32
  38    1216  4579740  4579719     32
  39    1248  4618526  4579771     32
  40    1280  4618544  4618557     32
  41    1312  4618563  4618575     32
  42    1344  4618583  4618594     32
  43    1376  4618605  4618614     32
  44    1408  4618626  4618636     32
  45    1440  4618652  4618657     32
  46    1472  4618677  4618683     32
  47    1504  4618703  4618708     32
  48    1536  4618728  4618734     32
  49    1568  4618754  4618759     32
  50    1600  4618774  4618785     32
  51    1632  4618782  4618805     32
  52    1664  4561195  4618813     32
  53    1696  4600548  4561226     32
  54    1728  4618793  4600579     32
  55    1760  4618807  4618824     32
  56    1792  4539912  4618838     32
  57    1824  4542619  4539943     32
  58    1856  4556887  4542650     32
  59    1888  4601632  4556918     32
  60    1920  4558150  4601663     32
  61    1952  4561224  4558181     32
  62    1984  4618816  4561255     32
  63    2016  4618835  4618847     32
  64    2048  4618861  4618866     32
  65    2080  4618881  4618892     32
  66    2112  4618901  4618912     32
  67    2144  4618917  4618932     32
  68    2176  4539241  4618948     32
  69    2208  4539915  4539272     32
  70    2240  4539985  4539946     32
  71    2272  4540096  4540016     14 eof
/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1: 72 extents found

Best Regards,
Bernhard

Excerpts from Bernhard Schmidt's message of 2011-05-03 06:33:25 -0400:

Do you have compression on?

The file the defrag ioctl works is that it schedules things for defrag but doesn't force out the IO immediately unless you use -f.

So, to test the result of the defrag, you need to either wait a bit or run sync.

-chris

Am 03.05.2011 13:08, schrieb Chris Mason:

Yes. lzo to be exact.

Did so, no change. See my reply to cwillu for the data.

I usually mount my / without any compression option and did "mount -o remount,compress=lzo /" before. I cannot reboot at the moment and I did not find any option to disable compression again. There does not seem to be a "nocompress" or "compress=[none|off]" option. Is this correct?

Bernhard

Excerpts from Bernhard Schmidt's message of 2011-05-03 07:30:36 -0400:

Using compression is not a problem, but in order to reduce the maximum amount of ram we need to uncompress an extent, we enforce a max size on the extent. So you'll tend to have more extents, but they should be close together on disk.

Could you please do a filefrag -v on the file? Lets see how bad it really is.

-chris

Hi,

root@schleppi:~# filefrag -v
/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1
Filesystem type is: 9123683e
File size of /usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1 is
9361528 (2286 blocks, blocksize 4096)
 ext logical physical expected length flags
   0       0  4542111              32
   1      32  4542134  4542142     32
   2      64  4573263  4542165     32
   3      96  4573285  4573294     32
   4     128  4579639  4573316     32
   5     160  4579664  4579670     32
   6     192  4581178  4579695     32
   7     224  4579811  4581209     32
   8     256  4579836  4579842     32
   9     288  4579861  4579867     32
  10     320  4579884  4579892     32
  11     352  4580698  4579915     32
  12     384  4580720  4580729     32
  13     416  4580746  4580751     32
  14     448  4580768  4580777     32
  15     480  4580793  4580799     32
  16     512  4580819  4580824     32
  17     544  4581238  4580850     32
  18     576  4600396  4581269     32
  19     608  4600422  4600427     32
  20     640  4600447  4600453     32
  21     672  4600472  4600478     32
  22     704  4600498  4600503     32
  23     736  4600523  4600529     32
  24     768  4601483  4600554     32
  25     800  4601509  4601514     32
  26     832  4601534  4601540     32
  27     864  4601558  4601565     32
  28     896  4601583  4601589     32
  29     928  4601608  4601614     32
  30     960  4618420  4601639     32
  31     992  4618443  4618451     32
  32    1024  4541221  4618474     32
  33    1056  4618463  4541252     32
  34    1088  4618485  4618494     32
  35    1120  4618505  4618516     32
  36    1152  4579536  4618536     32
  37    1184  4579688  4579567     32
  38    1216  4579740  4579719     32
  39    1248  4618526  4579771     32
  40    1280  4618544  4618557     32
  41    1312  4618563  4618575     32
  42    1344  4618583  4618594     32
  43    1376  4618605  4618614     32
  44    1408  4618626  4618636     32
  45    1440  4618652  4618657     32
  46    1472  4618677  4618683     32
  47    1504  4618703  4618708     32
  48    1536  4618728  4618734     32
  49    1568  4618754  4618759     32
  50    1600  4618774  4618785     32
  51    1632  4618782  4618805     32
  52    1664  4561195  4618813     32
  53    1696  4600548  4561226     32
  54    1728  4618793  4600579     32
  55    1760  4618807  4618824     32
  56    1792  4539912  4618838     32
  57    1824  4542619  4539943     32
  58    1856  4556887  4542650     32
  59    1888  4601632  4556918     32
  60    1920  4558150  4601663     32
  61    1952  4561224  4558181     32
  62    1984  4618816  4561255     32
  63    2016  4618835  4618847     32
  64    2048  4618861  4618866     32
  65    2080  4618881  4618892     32
  66    2112  4618901  4618912     32
  67    2144  4618917  4618932     32
  68    2176  4539241  4618948     32
  69    2208  4539915  4539272     32
  70    2240  4539985  4539946     32
  71    2272  4540096  4540016     14 eof
/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.4/cc1: 72 extents found

Bernhard

Excerpts from Bernhard Schmidt's message of 2011-05-03 07:43:04 -0400:

Ok, looks like we could be doing a little better job when compression is on to build out a bigger extent. This shouldn't be causing trouble on an ssd at all but on your rotating disk it'll be slightly slower.

Still most of these extents are somewhat close together, this is roughly what mount -o ssd (which is enabled automatically when we detect a non-rotating drive) would try for.

The problematic files are going to have thousands of extents, this file should be fine.

-chris

Hi,

Thanks, I'll check on my system with rotating disks at home when I get back.

Bernhard

n is

e on

ghly

ile

t back.

I'd also be curious to see if mounting with "-o compress-force=3Dlzo" affects anything.

As I recall, the compress-force option was added because performance could be affected negatively when trying to optimize compression with "-o compress".
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

:

imum

ze on

d be

it

is

is

on

It does seem the case generally; on 2.6.39-rc5, writing to a fresh filesystem using rsync with BTRFS compression enabled, 128KB extents seem very common [1] (filefrag inconsistency noted).

Defragmenting with compression gives a nice linear extent [2]. It looks like it'll be a good win to prevent extents being split at writeout for the read case on rotational media.

Daniel

--- [1]

# filefrag sdhci.ppm
sdhci.ppm: 173 extents found

# filefrag -v sdhci.ppm
=46ilesystem type is: 9123683e
=46ile size of sdhci.ppm is 36838232 (8994 blocks, blocksize 4096)
 ext logical physical expected length flags
   0       0  7410681              32
   1      32  7564365  7410712     32
   2      64  7875283  7564396     32
   3      96  8197111  7875314     32
   4     128  8459255  8197142     32
   5     160 10072355  8459286     32
   6     192 10556407 10072386     32
   7     224 11088956 10556438     32
   8     256  9435820 11088987     32
   9     288  9769977  9435851     32
  10     320 10032121  9770008     32
  11     352 11342839 10032152     32
  12     384 12129270 11342870     32
  13     416 12653556 12129301     32
  14     448 12915698 12653587     32
  15     480 12915705 12915729     32
  16     512 13439988 12915736     32
  17     544 13702133 13440019     32
  18     576 14163816 13702164     32
  19     608 14163823 14163847     32
  20     640 14163830 14163854     32
  21     672 14163837 14163861     32
  22     704 14163843 14163868     32
  23     736 14163849 14163874     32
  24     768 14163859 14163880     32
  25     800 14163869 14163890     32
  26     832 14163879 14163900     32
  27     864 14163889 14163910     32
  28     896 14163899 14163920     32
  29     928 14163909 14163930     32
  30     960 14163919 14163940     32
  31     992 14163929 14163950     32
  32    1024 14163939 14163960     32
  33    1056 14163949 14163970     32
  34    1088 14163958 14163980     32
  35    1120 14163967 14163989     32
  36    1152 14163976 14163998     32
  37    1184 14163985 14164007     32
  38    1216 14163994 14164016     32
  39    1248 14164003 14164025     32
  40    1280 14164012 14164034     32
  41    1312 14164021 14164043     32
  42    1344 14164031 14164052     32
  43    1376 14164041 14164062     32
  44    1408 14164050 14164072     32
  45    1440 14164060 14164081     32
  46    1472 14164070 14164091     32
  47    1504 14164080 14164101     32
  48    1536 14164090 14164111     32
  49    1568 14164100 14164121     32
  50    1600 14164109 14164131     32
  51    1632 14164119 14164140     32
  52    1664 14164129 14164150     32
  53    1696 14164139 14164160     32
  54    1728 14164148 14164170     32
  55    1760 14164157 14164179     32
  56    1792 14164166 14164188     32
  57    1824 14164176 14164197     32
  58    1856 14164186 14164207     32
  59    1888 14164196 14164217     32
  60    1920 14164205 14164227     32
  61    1952 14164214 14164236     32
  62    1984 14164223 14164245     32
  63    2016 14164233 14164254     32
  64    2048 14164243 14164264     32
  65    2080 14164252 14164274     32
  66    2112 14164262 14164283     32
  67    2144 14164272 14164293     32
  68    2176 14164282 14164303     32
  69    2208 14164292 14164313     32
  70    2240 14164302 14164323     32
  71    2272 14164311 14164333     32
  72    2304 14164321 14164342     32
  73    2336 14164331 14164352     32
  74    2368 14164340 14164362     32
  75    2400 14164350 14164371     32
  76    2432 14164360 14164381     32
  77    2464 14164369 14164391     32
  78    2496 14164379 14164400     32
  79    2528 14164389 14164410     32
  80    2560 14164398 14164420     32
  81    2592 14164407 14164429     32
  82    2624 14164416 14164438     32
  83    2656 14164425 14164447     32
  84    2688 14164434 14164456     32
  85    2720 14164443 14164465     32
  86    2752 14164452 14164474     32
  87    2784 14164462 14164483     32
  88    2816 14164472 14164493     32
  89    2848 14164481 14164503     32
  90    2880 14164490 14164512     32
  91    2912 14164499 14164521     32
  92    2944 14164508 14164530     32
  93    2976 14164517 14164539     32
  94    3008 14164526 14164548     32
  95    3040 14164535 14164557     32
  96    3072 14164545 14164566     32
  97    3104 14164554 14164576     32
  98    3136 14164563 14164585     32
  99    3168 14164573 14164594     32
 100    3200 14164582 14164604     32
 101    3232 14164591 14164613     32
 102    3264 14164600 14164622     32
 103    3296 14164609 14164631     32
 104    3328 14164618 14164640     32
 105    3360 14164628 14164649     32
 106    3392 14164637 14164659     32
 107    3424 14164646 14164668     32
 108    3456 14164656 14164677     32
 109    3488 14164665 14164687     32
 110    3520 14164674 14164696     32
 111    3552 14164684 14164705     32
 112    3584 14164693 14164715     32
 113    3616 14164702 14164724     32
 114    3648 14164711 14164733     32
 115    3680 14164720 14164742     32
 116    3712 14164729 14164751     32
 117    3744 14164739 14164760     32
 118    3776 14164748 14164770     32
 119    3808 14164757 14164779     32
 120    3840 14164766 14164788     32
 121    3872 14164775 14164797     32
 122    3904 14164784 14164806     32
 123    3936 14164793 14164815     32
 124    3968 14164802 14164824     32
 125    4000 14164811 14164833     32
 126    4032 14164820 14164842     32
 127    4064 14164829 14164851     32
 128    4096 14164838 14164860     32
 129    4128 14164848 14164869     32
 130    4160 14164857 14164879     32
 131    4192 14164866 14164888     32
 132    4224 14164876 14164897     32
 133    4256 14164885 14164907     32
 134    4288 14164895 14164916     32
 135    4320 14164905 14164926     32
 136    4352 14164915 14164936     32
 137    4384 14164925 14164946     32
 138    4416 14164935 14164956     32
 139    4448 14164945 14164966     32
 140    4480 14164955 14164976     32
 141    4512 14164965 14164986     32
 142    4544 14164975 14164996     32
 143    4576 14164986 14165006     32
 144    4608 14164997 14165017     32
 145    4640 14165007 14165028     32
 146    4672 14165018 14165038     32
 147    4704 14165029 14165049     32
 148    4736 14165039 14165060     32
 149    4768 14165049 14165070    224
 150    4992 14165273              32
 151    5024 14165283 14165304     32
 152    5056 14165294 14165314     32
 153    5088 14165305 14165325     32
 154    5120 14165315 14165336     32
 155    5152 14165326 14165346     32
 156    5184 14165337 14165357     32
 157    5216 14165347 14165368     32
 158    5248 14165358 14165378     32
 159    5280 14165369 14165389    224
 160    5504 14165593              32
 161    5536 14165603 14165624     32
 162    5568 14165613 14165634     32
 163    5600 14165623 14165644     32
 164    5632 14165633 14165654     32
 165    5664 14165643 14165664     32
 166    5696 14165653 14165674     32
 167    5728 14165663 14165684    288
 168    6016 14165951              32
 169    6048 14165960 14165982     32
 170    6080 14165969 14165991     32
 171    6112 14165978 14166000     32
 172    6144 14165988 14166009   2850 eof
sdhci.ppm: 170 extents found

--- [2]

# btrfs filesystem defragment -c sdhci.ppm

# filefrag -v sdhci.ppm
=46ilesystem type is: 9123683e
=46ile size of sdhci.ppm is 36838232 (8994 blocks, blocksize 4096)
 ext logical physical expected length flags
   0       0 233041476            8994 eof
sdhci.ppm: 1 extent found
--=20
Daniel J Blueman
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Yes, 128KB extents are hardcoded in Btrfs right now.

There are two reasons cited in the comments for this:

(1) Ease the RAM required when spreading compression across several CPUs.
(2) Make sure the amount of IO required to do a random read is reasonably small.

For about 4 months, I've been playing locally with 2 patches to increase the extent size to 512KB.

I haven't noticed any issues running with these patches. However, I only have a Core2duo with 2 CPUs, so I'm probably not running into issues that someone with more CPUs might encounter.

I'll submit these patches to the list as an RFC so more people can at least see where this is done. But with my limited hardware, I can't assert this change is the best for everyone.

Excerpts from Mitch Harder's message of 2011-05-03 11:42:56 -0400:

The problem is just that any random read into the file will require reading the full 512KB in order to decompress the extent. And you need to make sure you have enough room in ram to represent the decompressed bytes in order to find the pages you care about.

The alternative is to keep the smaller compressed extent size and make the allocator work harder to find contiguous 128KB extents to store all of the file bytes. This will work out much better ;)

-chris

On 3 May 2011 19:30, Bernhard Schmidt <[email protected]> wrote:
[]

Can you try with the compression option enabled? Eg:

# filefrag foo.dat
foo.dat: 11 extents found

# find . -xdev -type f -print0 | xargs -0 btrfs filesystem defragment -c

# filefrag foo.dat
foo.dat: 1 extent found

Seems to work fine on 2.6.39-rc5; I mounted with '-o compress,clear_cache' though.

Daniel

Daniel J Blueman

Am 03.05.2011 16:54, schrieb Daniel J Blueman:

Hi,

Maybe I was expecting too much. I tried it on a file with 72 extends and was expecting for the count to go down to 1 (or very very few). This does not seem to happen with this particular file. I just tested another file (with 193 extends) and it was reduced to 5. defrag with -f, but without -c. Still mounted with compress=lzo.

However, the 72 frags file is not getting any better, no matter which flags I tried. No big problem at the moment, I've found an older (Ubuntu Maverick) based system with a rotating disk that had like 50000 extends for a single file. I expect defragging that will increase performance quite nicely :-)

Bernhard

Resending this to the list since I did not notice that I just responded to Chris Mason for the last emails...

Excerpts from John Wyzer's message of Sat Apr 30 08:57:06 +0200 2011: