Rename BTRfs to MuchSlowerFS ?

Hi list,

I don't trust theoretical benchmarks that much and prefer "real-life benchs" on the occasion, so here's mine:

Given 4 laptops, the most powerful of which was running BTRFS and the others ext3 or ext4, all machines running Ubuntu 11.04 Natty 32-bit with a stock Ubuntu 2.6.38-11 kernel, all machines were given the following FS-intensive task :

- Upgrade O.S. from Ubuntu 11.04 Natty to 11.10 Oneiric (beta), using a local packages apt-cacher.

Machines :

1/ Acer Aspire 3104 WLMi, AMD Sempron Mobile 3500+ @1.8 GHz, 1.5 GB RAM, 80 GB SATA HD, ext4 over standard partitions

2/ Asus EeePC 1005PE, Intel Atom N450 @1.66 GHz, 2 GB RAM, 250 GB SATA HD, ext3 over AES-128 encrypted LVM

3/ Compaq Mini CQ10-740SF, Intel Atom N455 @1.66 GHz, 1 GB RAM, 250 GB SATA HD, ext4 over standard partitions

4/ Dell XPS M1330, Intel Core 2 Duo T9300 @2.5 GHz, 2 GB RAM, 160 B SATA HD, BTRFS over AES-128 encrypted LVM

Results :

All 3 ext3 / ext4 machines took between 60 and 90 minutes to complete their upgrade.

BTRFS machine took 20 HOURS so far, still counting (ETA 15 minutes left).

Wow. Impressive.

That's because dpkg is known for using (f)sync very heavily. btrfs
honours the sync request in all cases, so it's much much slower than ext3, which doesn't. There's an LD_PRELOAD library called "eatmydata" that you can use to paper over the issue by running it as a wrapper around dpkg.

Hugo.

=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
         --- Always be sincere,  whether you mean it or not. ---         

I don't think it's *just* that.

I had the same problem, recently upgrading Debian from Stable to Testing, it was taking more than 24 hours. I stopped the upgrade, then resumed it this time via eatmydata, and it proceeded perhaps two orders of magnitude faster than before, finishing the remaining packages in 5 minutes or so.

Problem is, even when I stopped the upgrade and waited a considerable time, each 'sync' was still taking 5-7 seconds. No other disk activity in the system, no snapshot creation/deletion/cleanup going on either, just multiple consecutive syncs:

rm@rm:~$ time sync

real 0m4.772s
user 0m0.000s
sys 0m0.480s

rm@rm:~$ time sync

real 0m6.831s
user 0m0.000s
sys 0m0.472s

rm@rm:~$ time sync

real 0m8.069s
user 0m0.000s
sys 0m0.468s

rm@rm:~$ time sync

real 0m6.675s
user 0m0.000s
sys 0m0.464s

rm@rm:~$ time sync

real 0m4.293s
user 0m0.000s
sys 0m0.464s

rm@rm:~$ time sync

real 0m4.230s
user 0m0.000s
sys 0m0.472s

rm@rm:~$ time sync

real 0m6.924s
user 0m0.000s
sys 0m0.464s

To be fair, this was on the 2.6.39.2 kernel, and the performance seems to be somewhat better on 3.0 (though I didn't do tests like this one or any significant dpkg operations on it yet).

With respect,
Roman

On Monday 05 September 2011 16:20:00 Roman Mamedov wrote:
[...]

Same here with 3.1-rc4 in my Debian SID amd64 laptop with Vertex II SSD disk.

`--> time sync ; time sync; time sync; time sync;time sync ; time sync; time sync; time sync;time sync ; time sync; time sync; time sync

sync  0.00s user 0.09s system 3% cpu 2.509 total
sync  0.00s user 0.09s system 3% cpu 2.690 total
sync  0.00s user 0.09s system 4% cpu 2.219 total
sync  0.00s user 0.09s system 3% cpu 2.270 total
sync  0.00s user 0.08s system 3% cpu 2.120 total
sync  0.00s user 0.09s system 4% cpu 2.170 total
sync  0.00s user 0.09s system 4% cpu 2.120 total
sync  0.00s user 0.09s system 3% cpu 2.189 total
sync  0.00s user 0.08s system 3% cpu 2.341 total
sync  0.00s user 0.09s system 3% cpu 2.262 total
sync  0.00s user 0.10s system 2% cpu 3.967 total
sync  0.00s user 0.09s system 4% cpu 2.271 total

Elric Milon

others

tock

ensive

(dpkg-intensive workload)

e their

ft).

I see similar behaviour on a single-disk workstation. This is a consequ= ence of
dpkg's heavy use of fsync() to ensure that changes to the filesystem ar= e done
safely, to try to ensure that a sudden crash or power interruption does= n't
leave the machine in an inconsistent or unrecoverable state.

(You can see that btrfs performs speedily if fsync is disabled via, for example, eatmydata. Which is not a production-worthy workaround for = the
majority of cases, but usefully highlights what's going wrong.)

See also:
https://bugs.launchpad.net/ubuntu/+source/dpkg/+bug/607632
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=3D635993

Now, I don't know why btrfs is particularly slow with a (relatively?) fsync-heavy workload, though I do note that a commit went into 3.0 that improved some fsync workloads; see commit:
8e531cdfeb75269c6c5aae33651cca39707848da

However, in my own testing, it doesn't seem to have improved matters significantly.

There were also a number of fsync-related improvements made in 2.6.32; = perhaps
there have been regressions since then?

It seems clear that the dpkg developers consider this to be reasonable behaviour on their part; is there any scope for improvements to be made=
to
btrfs to cope better with this kind of workload? Or is this an unavoid= able
property of the datastructures it's using?

Cheers,
David
--=20
David McBride <[email protected]>
Department of Computing, Imperial College, London
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

I've seen similar problem on Ubuntu-11 + Aspire One (8GB of slow "SSD"). More specifically half of ubuntu install went very fast and when disk was ~50% free things _suddenly_ gone slow.

According to strace fsync() started to take ~10 seconds(!) and upper. It's not common filesystem behaviour, but a plain (and hard to catch?) bug.

I don't have the box anymore. The root cause it to be found by some brave soul yet.

I wonder if latencytop would get the profile of the issue.

Maybe, btrfs degrades only after certain (insanely large?) amount of fsync() calls?

  Sergei

").

I'm just about to give up and definitely quit using BTRFS. My system ha= s become=20
so slow after upgrade Ubuntu Natty =3D> Oneiric Beta, that's it's purel= y and=20
simply unusable (although I'm the kind of old, white-haired, experience= d, used=20
to prehistoric systems, thus very patient, IT guy...)

It'a most probable that all of my usage patterns, i.e. read my mail, br= owse=20
the web, etc, definitely do not correspond to what BTRFS was designed f= or.=20
(Sorry for the rant, but this really pisses me off...)

So I'm only wonderign whether I reformat my system to ext4 or ZFS, and = whether=20
I do it right now or on thursday...

--=20
Sw=E2mi Petaramesh <[email protected]> http://petaramesh.org PGP 907= 6E32E

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

D").

has become

ely and

ced, used

Compared to ext3/4, btrfs or zfs (on linux) in its current state would seem like snail. The only way I can get decent speed with btrfs on my laptop is after using sandforce-based SSD (which helps offset some of the slowness). But since I like lzo compression and snapshot feature, I'll keep on using btrfs on this one :)

browse

for.

d whether

Reading your post, at this point I'd actually recommend you stick with ext4. Both btrfs and zfs are great, but IMHO btrfs is not ready for daily use by "ordinary" user yet, while zfs is a memory hog (especially for laptops, which is part of the reason why I'm using btrfs instead of zfs on this one).

=46WIW I have another laptop with 4GB ram and Ubuntu + zfs root on it, and it would seem "stall" on some operations with no apparent cpu or disk activity. zfs compression and snapshot feature works great though (and much more stable compared to btrfs, which stiil doesn't have a functioning fsck), so if you have the resource to spare you might want to give zfs another try later.

--=20
=46ajar
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Le Mercredi 7 Septembre 2011 00:11:25 vous avez =E9crit :

h

I actually shifted back from BTRFS to ext4 and fell like having offered=
myself=20
a brand new computer, about 20 times faster, me happy ;-)

True, ZFS is excellent but a memory hog (and strongly advises using a 6= 4-bit=20
OS) but I was surprised to discover that BTRFS was such a memory eater = itself,=20
with kernel 3.0. My system was swapping like mad !

I use (kernel) ZFS on my 64-bit main machine and I'm plain happy with i= t, and=20
tried ZFS on my 32-bit laptop in the hope to get more performance for l= ess=20
memory ; alas I just got a memory-hungry system running damn slow... Fo= r the=20
time being I will stick to ZFS for 64-bit machines with >=3D 4GB RAM, a= nd to=20
ext4 for 32-bit systems with less RAM...

I don't feel that BTRFS gives any advantage in its current state of=20 development. Alas.

--=20
Sw=E2mi Petaramesh <[email protected]> http://petaramesh.org PGP 907= 6E32E

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

I'm using btrfs since one year now and it's quite fast. I don't feel an= y=20
differences to other filesystems. Never tried a benchmark but for my=20 daily work it's nice. I also never had any issues with the memory. Imho= =20
nowadays memory isn't a problem at all in desktop computers. I bought=20 8gb of memory 2 years before because it was so damn cheap. Never used=20 that mutch, but it was almost for free :)

The advantage to ext4 for me is the build in raid1 and the snapshots.=20 I'm using the snapshot feature for my local backups. I like it because=20 it's really easy and uses very few storage. A simple "Snapshot -> Rsync= =20
to a different disk -> Snapshot" script is the perfect local backup met= hod.

I really appreciate the work of the developers! Btrfs is great and I'm=20 110% sure it will become better and better over the next month.

Best regards,
=46elix

th

ed myself

64-bit

r itself,

it, and

less

=46or the

and to

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Your workload must be light :)

It might not be possible in some situations. For example, if:
- adding memory by yourself means voiding warranty (cause there's a warranty sticker covering memory slot cover screw), while going to official service shop means 1 full day wasted plus inflated price
- you're using $300 netbook with 1 or 2 GB RAM with only one memory slot

you've never used zfs have you :)

For that purpose, think "same feature as btrfs snapshot + rsync" but without needing rsync. This can be very useful as the process of rsync determining what data to transfer can be quite CPU/disk intensive.

btrfs is great, but not quite production-ready yet, with lack of working fsck being the greatest hurdle. I'm not complaining much about the speed at this point though, being still in dev and all, and since new features will have some performance cost.

Fajar

I recently repeatedly rsync whole partitions (>30GB) without ill effects. (ok - first sync took whole 1s).

Now I'm curious - how do zsf get data off the partition without rsync?

Regards

Wait, you mean you sync 30GB data to another partition in one second? It should not be possible for single HDD no matter what the filesystem is. Unless you're using SSD or many HDD in raid.

That's how I'm using btrfs btw, on SSD, so I pretty much don't see the "slowness" since the SSD is blazingly fast. Add to that the fact btrfs has the features I need (compression, snapshot), and more memory-efficient (compared to zfs), it's suitable for my needs.

First hit on Google: http://www.markround.com/archives/38-ZFS-Replication.html

As additinal info, zfs send/receive stream is at DMU layer, which (in over-simplified term) is similar to raw disk blocks in normal partition+ext4 setup. zfs keeps track of which blocks are used, so when given two different snapshot, it can easily find out which blocks are different. When using incremental send zfs only has to send those blocks. It doesn't have to explicitly re-examine which parts of the file is unmodified (thus not wasting disk, CPU, and network the way rsync does).

IIRC there was a proposal in this list sometime ago on implementing similar functionality (send/receive) in btrfs. No actual working code (yet) that I know of though.

Fajar

You misparse the sentence. First sync took 1s after rsync and next were much faster.

Regards