compression ratio
Hello all.
I`m new to btrfs and do some testing now.
What we have:
SL6 - kernel 2.6.32-131.2.1.el6.x86_64 mdadm RAID5 with 8 HDD - 27T partition.
Mount options is "noatime,noacl,compress-force"
I use scribe daemon to copy log files from 200 hosts to that partition for stress testing.
But I found what compression ratio is really small.
Partition is full of regular log files - plain text. Most of them a big ones(10-20Gb) and they grow in realtime.
du -hs /logs/
1.1T /logs/
And compressed size is(df -h):
908G
Regular gzip do a significant more ratio.
There is a problem here? File size? Realtime grow of a files?
Signature · 2 lines
Best regards, Proskurin Kirill
Hi,
Quoted 3 lines · Proskurin Kirill
What we have:
SL6 - kernel 2.6.32-131.2.1.el6.x86_64
mdadm RAID5 with 8 HDD - 27T partition.
btw .32 is very old
Quoted 5 lines
Mount options is "noatime,noacl,compress-force"
I use scribe daemon to copy log files from 200 hosts to that partition for stress testing.But I found what compression ratio is really small.
It is and it will be due to nature of compression method used and the constraints given:
* compression has to be fast, only real-time methods can be considered which have natural speed/ratio tradeoff
* you're using zlib (lzo was added recently), which is not that fast and does not have a great compression ratio
* there is maximum length of compressed data per round, it's hardcoded to 128KB by now
* zlib works in streaming mode, which compresses whole 128KB with dictionary reused
* lzo (as implemented in btrfs) however does not reuse dictionary and compresses each page separately; resulting size is even bigger than it would be when full 128KB were compressed
Quoted 2 lines
Partition is full of regular log files - plain text. Most of them a big ones(10-20Gb) and they grow in realtime.
Realtime growth should not be a problem.
The situation can be slightly improved if we use a method capable of streamed compression (ie. dictionary reuse). This is naturally supported by eg. quicklz and snappy.
There are a few other realtime methods (fastlz, lzrw, lzjb) which may be extended to support streaming compression, but with careful evaluation of the result.
david
Hallo, Proskurin,
Du meintest am 20.06.11:
Quoted 5 lines
I`m new to btrfs and do some testing now.
What we have:
SL6 - kernel 2.6.32-131.2.1.el6.x86_64
mdadm RAID5 with 8 HDD - 27T partition.
You should take a newer kernel. On my system I needed 2.6.38.5 and newer for btrfs; older kernels lead to unreadable partitions.
Viele Gruesse!
Helmut
Quoted 4 lines · Helmut Hullen
Hallo, Proskurin,
Du meintest am 20.06.11:
Quoted 8 lines
Quoted 5 lines
I`m new to btrfs and do some testing now.
What we have:
SL6 - kernel 2.6.32-131.2.1.el6.x86_64
mdadm RAID5 with 8 HDD - 27T partition.You should take a newer kernel. On my system I needed 2.6.38.5 and newer for btrfs; older kernels lead to unreadable partitions.
What kernel version is most recommended at this time?
Signature · 2 lines
Best regards, Proskurin Kirill
Quoted 10 lines · Proskurin Kirill
Quoted 7 lines · Helmut Hullen
Du meintest am 20.06.11:
Quoted 3 lines
What we have:
SL6 - kernel 2.6.32-131.2.1.el6.x86_64
mdadm RAID5 with 8 HDD - 27T partition.You should take a newer kernel. On my system I needed 2.6.38.5 and newer for btrfs; older kernels lead to unreadable partitions.
What kernel version is most recommended at this time?
It's always "the latest version you can manage to run". So,
currently, 2.6.39 or the 3.0-rc series.
Hugo.
Signature · 3 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- O tempura! O moresushi! --- On Mon, Jun 20, 2011 at 10:22 AM, Hugo Mills <[email protected]> wrote= :
Quoted 8 lines · Proskurin Kirill
Quoted 7 lines · Helmut Hullen
Quoted 6 lines
Du meintest am 20.06.11:
Quoted 3 lines
What we have:
SL6 - kernel 2.6.32-131.2.1.el6.x86_64
mdadm RAID5 with 8 HDD - 27T partition.You should take a newer kernel. On my system I needed 2.6.38.5 and =
newer
Quoted 11 lines
Quoted 3 lines
Quoted 1 line
for btrfs; older kernels lead to unreadable partitions.
What kernel version is most recommended at this time?
=A0 It's always "the latest version you can manage to run". So, currently, 2.6.39 or the 3.0-rc series.
=A0 Hugo.
have to audit the code yourself
--
=3D=3D=3D Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.=
org.uk =3D=3D=3D
Quoted 1 line
=A0PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org=
=2Euk
Quoted 1 line
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 --- O tempura! O moresushi! -=
--
Quoted 10 lines
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)iD8DBQFN/2W9IKyzvlFcI40RAgmuAKCEbTdS/ZjWzNMX3D5t8zNKbULJQACgyl6X e2SIUGnkDu1BFagOL2e5ymM=3D
=3DSO2f
-----END PGP SIGNATURE-----
Actually, the btrfs in the newest RHEL 2.6.32 kernels (and by extension, SL kernels) has been heavily backported, so it is fairly current, although probably not up to 2.6.39 level yet. It isn't the most current, but it is more current than most people realize, and should be fairly stable.
I think you'll want the RHEL 6.1 (or SL 6.1) 2.6.32 kernel to get the most recent Btrfs backporting.
I haven't seen any detailed information posted on the extent of the backporting. So, for right now, you're kind-of on your own to ascertain if the btrfs in your RHEL kernel is suitable for your needs.
I believe the original poster is getting confused by results of the 'df' command. Running 'df' will include a large amount of reserved space for metadata. The Btrfs Wiki has some insights on how to evaluate the space consumed by a Btrfs volume.
But, in general, running 'df' will not give you a complete picture of disk utilization with Btrfs.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html