data DUP
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Is there any particular reason why I can't use DUP for data?
When I try to set it with balance there is a kernel message:
btrfs: dup for data is not allowed
The glossary at https://btrfs.wiki.kernel.org/index.php/Glossary says:
Regular data cannot be assigned DUP level.
It is somewhat baffling that code and documentation exists to prevent this!
My current use case is an older hard drive I am putting backups on. Since my data fits in less than half of the drive, and drives get bad sectors (this has had several it has reallocated) using DUP would be useful.
(I realise making two partitions and RAID-1 with them would work, which makes the DUP restriction even sillier.)
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
iEYEARECAAYFAlFy978ACgkQmOOfHg372QTCXwCdGuWbRFY8cZoWNFZb1k3mFeZM nrkAoK9c5KddsJ1R57pAR1Lk89fPSo9Q
=ENfT
-----END PGP SIGNATURE-----
Quoted 4 lines · Roger Binns
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Is there any particular reason why I can't use DUP for data?
Technically, no. Performance is likely to suck if you use
rotational disks, and you may find some SSDs deduplicate blocks, making it fairly pointless for those devices.
Quoted 3 lines
When I try to set it with balance there is a kernel message:
btrfs: dup for data is not allowed
What kernel and userspace versions are you using? I thought the
restriction had been removed at some point (but possibly I'm just misremembering it).
Quoted 5 lines
The glossary at https://btrfs.wiki.kernel.org/index.php/Glossary says:
Regular data cannot be assigned DUP level.
It is somewhat baffling that code and documentation exists to prevent this!
I think it did get fixed at some point. The documentation doesn't
always get updated everywhere -- it can be quite hard to find every single implication of a feature when that feature gets changd/updated.
Hugo.
Signature · 3 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- UNIX: Spanish manufacturer of fire extinguishers. --- -----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 5 lines · Roger Binns
Quoted 1 line
Is there any particular reason why I can't use DUP for data?
Technically, no. Performance is likely to suck if you use rotational disks, and you may find some SSDs deduplicate blocks, making it fairly pointless for those devices.
Wanting dup is because I care about resilience in the face of errors over performance, so I don't actually care how bad performance sucks. I only put data on SSDs that I can afford to lose (usually backed up to Dropbox/github/spinning disks).
Quoted 7 lines
Quoted 3 lines
When I try to set it with balance there is a kernel message:
btrfs: dup for data is not allowed
What kernel and userspace versions are you using? I thought the restriction had been removed at some point (but possibly I'm just misremembering it).
Whatever Ubuntu 12.10 ships with. Kernel package is 3.5.0.27.43 and btrfs-tools is 0.19+20120328-7ubuntu1. Note the message came from the kernel so it would appear to be solely to blame for refusing my request.
Quoted 2 lines
it can be quite hard to find every single implication of a feature when that feature gets changd/updated.
I'm more amused that someone went to the trouble of putting in kernel detection and messages plus updating the documentation in order to prevent using DUP for data!
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
iEYEARECAAYFAlFzBWEACgkQmOOfHg372QT5twCeI5KOgHYT1wcPNPtni9TYKQaG aeIAmQGw/nL/ziTDAu4gUJIFPwCLI5td
=ABih
-----END PGP SIGNATURE-----
Quoted 9 lines · Roger Binns
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 4 lines · Roger Binns
Quoted 3 lines
When I try to set it with balance there is a kernel message:
btrfs: dup for data is not allowed
Quoted 7 lines
Quoted 3 lines
What kernel and userspace versions are you using? I thought the restriction had been removed at some point (but possibly I'm just misremembering it).
Whatever Ubuntu 12.10 ships with. Kernel package is 3.5.0.27.43 and btrfs-tools is 0.19+20120328-7ubuntu1. Note the message came from the kernel so it would appear to be solely to blame for refusing my request.
OK, I'm not 100% certain, but that's probably too early for the fix
I was thinking of. You should upgrade anyway -- there's been a number of serious bugs in btrfs fixed since then. Ubuntu has new kernels in a PPA at [1].
Quoted 6 lines
Quoted 2 lines
it can be quite hard to find every single implication of a feature when that feature gets changd/updated.
I'm more amused that someone went to the trouble of putting in kernel detection and messages plus updating the documentation in order to prevent using DUP for data!
It'd have been a mere couple of lines of code, and probably there to
match historical usage of mkfs (which I believe has also been dealt with).
Hugo.
[1] https://launchpad.net/~yofel/+archive/btrfs
Signature · 4 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- A linked list is still a binary tree. Just a ---
very unbalanced one. -----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 2 lines · Hugo Mills
You should upgrade anyway -- there's been a number of serious bugs in btrfs fixed since then.
13.04 is imminent so I'll pick up a newer kernel as part of that anyway. (Also Tanglu which I hope to move to intends to use the Ubuntu kernel anyway.)
In any event I am not worried. Bug fixes get backported. The probability of hitting any serious bug is low (others would likely be victims first) and worst case I have backups (snapshots, other machines, Google/Dropbox, DVDs & hard drives at other people's houses etc).
There are two major reasons I switched from ext4. The first is that everything is online, including adding and removing devices, checking data integrity etc.
The second is that data not silently lost. I had some bad sectors develop on ext4 spinning disk and the only way to properly recover was to do offline checks that would have taken ~ 24 hours! Finding out which filenames were involved was far too much effort.
My philosophy is that we have machines with billions of processor cycles per second. They can figure things out for themselves without requiring me to baby sit them!
I also run btrfs in a variety of configurations - raid0, raid1, sata, usb, hdd, ssd, single device, multi-device, bare, dmcrypt, machines on all the time, laptop, frequent suspend/resume, frequent power on/off. I've never experienced any problems with btrfs and use scrub for reassurance.
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
iEYEARECAAYFAlFzHkwACgkQmOOfHg372QSBBwCfYYJQm9l9ts9eWOvZIRUlTCWJ KsMAoLVsd1fRdV+T7KO7nVVGuFCGYN5a
=Y4by
-----END PGP SIGNATURE-----
Quoted 1 line · Roger Binns
Is there any particular reason why I can't use DUP for data?
IIRC separate DUP for metadata and data is not possible due to some technical reason (I don't remember nor can find the mail where this was mentioned).
DUP data is possible only in the --mixed mode so this
mkfs.btrfs --mixed -d dup -m dup /dev/ice
works.
david
Quoted 11 lines · Roger Binns
Quoted 2 lines · Hugo Mills
You should upgrade anyway -- there's been a number of serious bugs in btrfs fixed since then.
13.04 is imminent so I'll pick up a newer kernel as part of that anyway. (Also Tanglu which I hope to move to intends to use the Ubuntu kernel anyway.)
In any event I am not worried. Bug fixes get backported. The probability of hitting any serious bug is low (others would likely be victims first) and worst case I have backups (snapshots, other machines, Google/Dropbox, DVDs & hard drives at other people's houses etc).
Unfortunately, bugfixes in btrfs have tended to be *not* backported; aside from a few special cases, btrfs changes only go into new kernel 3.x releases. A few of the Linux distributions that officially support btrfs (Possibly Oracle, Red Hat, Suse? You would have to contact them for details...) may be backporting fixes themselves.
As a result, we recommend that btrfs users should generally run whatever the latest upstream kernel release is - in particular, you should try the latest kernel before reporting bugs in btrfs.
Signature · 1 line
Calvin Walton <[email protected]>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 2 lines · Calvin Walton
Unfortunately, bugfixes in btrfs have tended to be *not* backported; aside from a few special cases, ...
Your efforts to scare me are admirable, but have failed :-)
As btrfs development has progressed, the probability of a random user like me hitting bugs keeps decreasing. This is a reflection of the maturity of the code base, the increasing number of users, improved test suites, more eyes on the code, more diversity in uses etc. As far as I can see, backported bugfixes are made when the probability of a bug being encountered is significantly higher than the current probabilities, and that is why they are rare.
As for the severity of the rarely hit bugs, the COW nature of the data means there is unlikely to be corruption, and if there is then of the most recent activity. Additionally the checksums mean it is possible to proactively verify (online) that unexpected corruption hasn't been creeping in.
And if all that fails, I have multiple layers of backups.
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
iEYEARECAAYFAlF8X14ACgkQmOOfHg372QT5EQCfSbzZBRdm1KyljpoNpP5kHy15 iFoAn2eyair54JQdhnL2dfvqAJrVMWe4
=9zYA
-----END PGP SIGNATURE-----
Quoted 14 lines · Roger Binns
Quoted 2 lines · Calvin Walton
Unfortunately, bugfixes in btrfs have tended to be *not* backported; aside from a few special cases, ...
Your efforts to scare me are admirable, but have failed :-)
As btrfs development has progressed, the probability of a random user like me hitting bugs keeps decreasing. This is a reflection of the maturity of the code base, the increasing number of users, improved test suites, more eyes on the code, more diversity in uses etc. As far as I can see, backported bugfixes are made when the probability of a bug being encountered is significantly higher than the current probabilities, and that is why they are rare.
Quoted 7 lines
As for the severity of the rarely hit bugs, the COW nature of the data means there is unlikely to be corruption, and if there is then of the most recent activity. Additionally the checksums mean it is possible to proactively verify (online) that unexpected corruption hasn't been creeping in.
And if all that fails, I have multiple layers of backups.
I would be *very* hard-pressed to see it as an attempt to scare you - he's just saying what is (very consistently) said to others on this list: When using btrfs, run a recent kernel :P.
Honestly, even leaving aside the lack of backporting, there are other benefits to a recent kernel - things like cross-subvolume reflinks, btrfs device replace support being far more efficient than add/balance/remove/balance, and a bunch more.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 1 line · Alex Elsayed
When using btrfs, run a recent kernel :P.
Every software developer says that of what they produce. Newer is almost always better in many different axes.
Quoted 4 lines
Honestly, even leaving aside the lack of backporting, there are other benefits to a recent kernel - things like cross-subvolume reflinks, btrfs device replace support being far more efficient than add/balance/remove/balance, and a bunch more.
Those are all "features", none of which I use or have had to use yet.
If it will make you feel better I did upgrade some systems today to the most recent Ubuntu release which meant going from kernel 3.5 to 3.8.
Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
iEYEARECAAYFAlF83dUACgkQmOOfHg372QQeDACgz0oBnrYeg6fO5tFxUy9qonE9 HYIAoJWjT8z2sJ356YAph1NAyLKhcEBz
=ySAg
-----END PGP SIGNATURE-----