BTRFS, SSD and single metadata
Hi,
I created a BTRFS filesytem over LVM over LUKS encryption on an SSD [yes, I know...], and I noticed that the FS got created with metadata in "DUP" mode, contrary to what "man mkfs.btrfs" says for SSDs -> it would be supposed to be "SINGLE"...
Well I don't know if my system didn't identify the SSD because of the LVM+LUKS stack (however it mounts well by itself with the "ssd" flag and accepts the "discard" option [yes, I know...]), or if the manpage is obsolete or if this feature just doesn't work...?
The SSD being a Micron RealSSD C400
For both SSD preservation and data integrity, would it be advisable to change metadata to "SINGLE" using a rebalance, or if I'd better just leave things the way they are...?
TIA for any insight.
Signature · 5 lines
Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E Tout le malheur des hommes vient de ce qu'ils ne vivent pas dans _le_ monde, mais dans _leur_ monde. -- Héraclite.
Quoted 12 lines · Swâmi Petaramesh
Hi,
I created a BTRFS filesytem over LVM over LUKS encryption on an SSD [yes, I know...], and I noticed that the FS got created with metadata in "DUP" mode, contrary to what "man mkfs.btrfs" says for SSDs -> it would be supposed to be "SINGLE"...
Well I don't know if my system didn't identify the SSD because of the LVM+LUKS stack (however it mounts well by itself with the "ssd" flag and accepts the "discard" option [yes, I know...]), or if the manpage is obsolete or if this feature just doesn't work...?
Quoted 8 lines
The SSD being a Micron RealSSD C400
For both SSD preservation and data integrity, would it be advisable to change metadata to "SINGLE" using a rebalance, or if I'd better just leave things the way they are...?
TIA for any insight.
What mkfs.btrfs looks at is /sys/block/<whatever-device>/queue/rotational, if that is 1 it knows that the device isn't a SSD. I believe that LVM passes through whatever the next lower layer's value is, but dmcrypt (and by extension LUKS) always force it to a 1 (possibly to prevent programs from using heuristics for enabling discard)
Hi Austin, and thanks for your reply.
Quoted 7 lines · Le lundi 16 juin 2014, 07:09:55 Austin S Hemmelgarn a écrit :
What mkfs.btrfs looks at is /sys/block/<whatever-device>/queue/rotational, if that is 1 it knows that the device isn't a SSD. I believe that LVM passes through whatever the next lower layer's value is, but dmcrypt (and by extension LUKS) always force it to a 1 (possibly to prevent programs from using heuristics for enabling discard)
In the current "running condition", the system clearly sees this is *not* rotational, even thru the LVM/dmcrypt stack :
# mount | grep btrfs /dev/mapper/VG-LINUX on / type btrfs (rw,noatime,seclabel,compress=lzo,ssd,discard,space_cache,autodefrag)
# ll /dev/mapper/VGV-LINUX lrwxrwxrwx. 1 root root 7 16 juin 09:21 /dev/mapper/VG-LINUX -> ../dm-1
# cat /sys/block/dm-1/queue/rotational 0
...However, at mkfs.btrfs time, it migth well not have seen it, as I made it from a live USB key in which both the lvm.conf and crypttab had not been taylored to allow "trim" commands...
However, now that the FS is created, I still wonder whether I should use a rebalance to change the metadata from DUP to SINGLE, or if Id' better stay with DUP...
Kind regards.
Signature · 1 line
Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E
Quoted 28 lines · Swâmi Petaramesh
Hi Austin, and thanks for your reply.
Quoted 7 lines · Le lundi 16 juin 2014, 07:09:55 Austin S Hemmelgarn a écrit :
What mkfs.btrfs looks at is /sys/block/<whatever-device>/queue/rotational, if that is 1 it knows that the device isn't a SSD. I believe that LVM passes through whatever the next lower layer's value is, but dmcrypt (and by extension LUKS) always force it to a 1 (possibly to prevent programs from using heuristics for enabling discard)
In the current "running condition", the system clearly sees this is *not* rotational, even thru the LVM/dmcrypt stack :
# mount | grep btrfs /dev/mapper/VG-LINUX on / type btrfs (rw,noatime,seclabel,compress=lzo,ssd,discard,space_cache,autodefrag)# ll /dev/mapper/VGV-LINUX lrwxrwxrwx. 1 root root 7 16 juin 09:21 /dev/mapper/VG-LINUX -> ../dm-1# cat /sys/block/dm-1/queue/rotational 0...However, at mkfs.btrfs time, it migth well not have seen it, as I made it from a live USB key in which both the lvm.conf and crypttab had not been taylored to allow "trim" commands...
Quoted 7 lines
However, now that the FS is created, I still wonder whether I should use a rebalance to change the metadata from DUP to SINGLE, or if Id' better stay with DUP...
Kind regards.
I'd personally stay with the DUP profile, but then that's just me being paranoid. You will almost certainly get better performance using the SINGLE profile instead of DUP, but this is mostly due to it requiring fewer blocks to be encrypted by LUKS (Which is almost certainly your primary bottleneck unless you have some high-end crypto-accelerator card).
Quoted 5 lines · On Mon, 16 Jun 2014 07:23:14 Austin S Hemmelgarn wrote:
I'd personally stay with the DUP profile, but then that's just me being paranoid. You will almost certainly get better performance using the SINGLE profile instead of DUP, but this is mostly due to it requiring fewer blocks to be encrypted by LUKS (Which is almost certainly your primary bottleneck unless you have some high-end crypto-accelerator card).
On my Q8400 workstation running BTRFS over LUKS on an Intel SSD the primary bottleneck has always been BTRFS. The message I wrote earlier today about BTRFS fallocate() performance was on this system, I had BTRFS using kernel CPU time for periods of 10+ seconds without ANY disk IO - so LUKS wasn't a performance issue.
So far I've never seen LUKS be a performance bottleneck. When running LUKS on spinning media the disk seek performance will almost always be the bottleneck.
The worst case for LUKS is transferring large amounts of data such as contiguous reads. In a contiguous read test I'm seeing 120MB/s for LUKS on a SSD and 200MB/s for direct access to the same SSD. That is a reasonable difference, but it's not something I've been able to hit with any real-world use while BTRFS metadata performance is often an issue.
Signature · 2 lines
My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
Swâmi Petaramesh posted on Mon, 16 Jun 2014 09:54:01 +0200 as excerpted:
Quoted 9 lines
I created a BTRFS filesytem over LVM over LUKS encryption on an SSD [yes, I know...], and I noticed that the FS got created with metadata in "DUP" mode, contrary to what "man mkfs.btrfs" says for SSDs -> it would be supposed to be "SINGLE"...
Well I don't know if my system didn't identify the SSD because of the LVM+LUKS stack (however it mounts well by itself with the "ssd" flag and accepts the "discard" option [yes, I know...]), or if the manpage is obsolete or if this feature just doesn't work...?
Does btrfs automatically add the ssd mount option or do you have to add it? If you have to add it, that means btrfs isn't detecting the ssd, which would explain why mkfs.btrfs didn't detect it either... as you said very likely due to the LVM over LUKS stack.
I believe the detection is actually based on what the kernel reports. I may be mistaken and I'm not running stacked devices ATM in ordered to check them, but check /sys/block/<device>/queue/rotational. On my ssds, the value in that file is 0. On my spinning rust, it's 1. If that is indeed what it's looking at, you can verify that your hardware device is actually detected by the kernel as rotational 0, and then check the various layers of your stack and see where the rotational 0 (or the rotational file itself) gets dropped.
Quoted 7 lines
The SSD being a Micron RealSSD C400
For both SSD preservation and data integrity, would it be advisable to change metadata to "SINGLE" using a rebalance, or if I'd better just leave things the way they are...?
TIA for any insight.
That's a very good question, on which there has been some debate on this list.
The first question: Does your SSD firmware do compression and dedup, or not? IIRC the sandforce (I believe that's the firmware name) firmware does compression and dedup, but not all firmware does. I know a bullet- point feature of my SSDs (several Corsair Neutron 256 GB, FWIW) is that they do NOT do this sort of compression and dedup -- what the system tells them to save is what they save, and if you tell it to save a hundred copies of the same thing, that's what it does. (These SSDs are targeted at commercial usage and this is billed as a performance reliability feature.)
The reason originally given for defaulting to single mode metadata on SSDs was that it was due to this possible dedup -- dup-mode metadata might actually end up single-copy-only due to the firmware compression and dedup in any case. Between that and their typically smaller size, I guess the decision was that single was the best default.
However, it occurs to me that with the LUKS encryption layer, I'm not entirely sure if duplication at the btrfs level would end up as the same encrypted stream headed to the hardware in any case. If it would encrypt the two copies of the dup-mode metadata as different, then the hardware dedup/compression wouldn't work on it anyway. OTOH, if it encrypts them as the same stream headed to hardware, then again, it would matter.
Meanwhile... which is better and should you rebalance to single? In terms of data integrity, dup mode is definitely better, since if there's damage to one copy such that it doesn't pass checksum verification, you still have the other copy to read from... and to rebuild the damaged copy from.
OTOH, single does take less space, and performance should be slightly better. If you're keeping good backups anyway, or if the ssd's firmware might be mucking with things leaving you with only a single copy in any case, single mode could be a better choice.
FWIW, while most of my partitions are btrfs raid1 here, so the second copy is on a different physical device, /boot is an exception. I have a separate /boot on each device, pointed at by the grub loaded on each device so I can use the BIOS boot selector to choose which one I boot. That lets me keep a working /boot that I boot most of the time on one device, and a backup /boot on the other device, in case something goes wrong with the first.
So those dedicated /boot partitions are an exception to my normal btrfs raid1. They're both (working and primary backup) 256 MiB mixed-data/ metadata mode as they're so small, which means data and metadata must both be the same mode, and they're both set to dup mode.
Which means they effectively only hold (a bit under) 128 MiB worth of data, since it's dup mode for both data and metadata, but 128 MiB is fine for /boot, as long as I don't let too many kernels build up.
So I'd obviously recommend dup mode, unless you know your ssd's firmware is going to dedup it anyway. But that's just me. As I said, there has been some discussion about it on the list, and some people make other choices. You can of course dig in the list archives if you want to see previous threads on the subject, but ultimately, it's upto you. =:^)
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 2 lines · Le lundi 16 juin 2014, 12:16:33 Duncan a écrit :
Does btrfs automatically add the ssd mount option or do you have to add it? If you have to add it, that means btrfs isn't detecting the ssd,
First time I mounted the freshly created filesystem, it actually added the "ssd" option by itself. Thus indeed, it could see this was a SSD...
Quoted 6 lines
However, it occurs to me that with the LUKS encryption layer, I'm not entirely sure if duplication at the btrfs level would end up as the same encrypted stream headed to the hardware in any case. If it would encrypt the two copies of the dup-mode metadata as different, then the hardware dedup/compression wouldn't work on it anyway. OTOH, if it encrypts them as the same stream headed to hardware, then again, it would matter.
That makes an excellent point. 2 copies of the same binary data in different filesystem sectors, process thru LUKS, will create entirely different binary ciphertext, so for sure both metadata copies will always be different and cannot be "deduped" by SSD firmware...
Kind regards.
Signature · 6 lines
Swâmi Petaramesh <[email protected]> http://petaramesh.org PGP 9076E32E Les êtres humains, cernés par leurs besoins, s'agitent sans but comme des lapins pris au piège. Ainsi, moine, détourne-toi de ces besoins et trouve la liberté. -- Buddha Shakyamuni