zerofree btrfs support?

Hi.

Just wondered... was it ever planned (or is there some equivalent) to get support for btrfs in zerofree?

Thanks,
Chris.

Do you want zerofree for thin storage optimization, or for security?

For the former, you can use fstrim; this is enough on any modern SSD; on HDD you can rig the block device to simulate TRIM by writing zeroes. I'm sure one of dm-* can do this, if not -- should be easy to add, there's also qemu-nbd which allows control over discard, but incurs a performance penalty compared to playing with the block layer.

For zerofree for security, you'd need defrag (to dislodge partial pinned extents) first, and do a full balance to avoid data left in metadata nodes and in blocks beyond file ends (note that zerofree doesn't do this on traditional filesystems either).

Meow!

⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can.
⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener.
⠈⠳⣄⠀⠀⠀⠀ A master species delegates.

I don't think one can really use it for security (neither on SSD or HDD).
On both, zeroed blocks may still be readable by forensic measures.

So optimisation, i.e. digging holes in VM image files and make them sparse.

Writing zeros if of course possible... but rather ugly... one really needs to write *everything* while a smart tool could just zero those block groups that have been used (while everything else is still zero from the original image file).

TRIM/discard... not sure how far this is really a solution.

The first thing that comes to my mind is, that *if* the discard would propagate down below a dm-crypt layer (e.g. in my case there is:
SSD->partitions->dmcrypt->LUKS->btrfs->image-files-I-want-to-zero) it has effects on security, which is why dm-crypt per default blocks discard.

Some longer time ago I had a look at whether qemu would support that on it's own,... i.e. the guest and it's btrfs would normally use discard, but the image file below would mark the block as discarded and later on e can use some qemu-img command to dig holes into exactly those locations.
Back then it didn't seem to work.

But even if it would in the meantime, a proper zerofree implementation would be beneficial for all non-qemu/qcow2 users (e.g. if one uses raw images in qemu, the whole thing couldn't work but with really zeroing the blocks inside the guest.

Cheers,
Chris.

It is the solution in a great many of usage scenarios, don't know enough about your particular one, though.

Note you can use it on HDDs too, even without QEMU and the like: via using LVM "thin" volumes. I use that on a number of machines, the benefit is that since TRIMed areas are "stored nowhere", those partitions allow for incredibly fast block-level backups, as it doesn't have to physically read in all the free space, let alone any stale data in there. LVM snapshots are also way more efficient with thin volumes, which helps during backup.

Out of misguided paranoia. If your crypto is any good (and last I checked AES was good enough), there's really not a lot to gain for the "attacker" knowing which areas of the disk are used and which are not.

It works, just not with some of the QEMU virtualized disk device drivers. You don't need to use qemu-img to manually dig holes either, it's all automatic.

QEMU deallocates parts of its raw images for those areas which have been TRIM'ed by the guest. In fact I never use qcow2, always raw images only. Yet, boot a guest, issue fstrim, and see the raw file while still having the same size, show much lower actual disk usage in "du".

With respect,
Roman

Since we're on a btrfs mailing list, if you use qemu, you really want sparse format:raw instead of qcow2 or preallocated raw. This also works great with TRIM.

It works only with scsi and virtio-scsi drivers. Most qemu setups use either ide (ouch!) or virtio-blk.

You'd obviously want virtio-scsi; note that defconfig enables virtio-blk but not virtio-scsi; I assume most distribution kernels have both. It's a bit tedious to switch between the two as -blk is visible as /dev/vda while -scsi as /dev/sda.

Meow!

⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can.
⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener.
⠈⠳⣄⠀⠀⠀⠀ A master species delegates.

Well... my original question was whether someone could make zerofree support for btrfs (which I think would be best if someone who knows how btrfs really works)... thus I directed the question to this list and not to some of qemu :-)

Seems my libvirt created VMs use "sata" per default... and it does seem to work with that either in the meantime.

Thanks :-)

Chris.

Agreed, that's why I use RAW. QCOW2 would add a second layer of COW on top of Btrfs, which sounds like a nightmare. Even if you would run those files as NOCOW in Btrfs, somehow I feel FS-native COW is more efficient than emulating it in userspace with special format files.

It works with IDE as well.

With respect,
Roman

I've just seen there is even a nocow option "specifically" for btrfs... it seems however that it doesn't disable the CoW of qcow, but rather that of btrfs... (thus silently also the checksumming).

Does plain qcow2 really CoW on every write? I've always assumed it would only CoW when one makes snapshots or so...

Cheers,
Chris.

I was thinking about using those... but then I'd have to use loop device files I guess... which I also want to avoid.

I'm not an expert here... but a) I think it would be independent of AES and rather the encryption mode (e.g. XTS) which protects here or not... and b) we've seen too many attacks on crypto based on smart statistics and knowing which blocks on a medium are actually data or just "random crypto noise" (and you know that when using TRIM) can already tell a lot.
At least it could tell an attacker how much data there is on a fs.

You're right... seems like in older version one needed to set virtio- scsi as device driver (which I possible missed), but nowadays it even seems to work with sata.

Works with qcow2 as well... heck even Windows can do it (though it has no fstrim and it seems one needs to run defrag (which probably does next to defragmentation also what fstrim does).

Fine for me,... though non qemu users may still be interested in having
zerofree.

Cheers,
Chris.

The zerofree was requested, eg. here
https://bugzilla.kernel.org/show_bug.cgi?id=69121

and there was a discussion under proposed patches on fsdevel 3 years ago, message-id is [email protected]

tl;dr there's nothing like zeofree now, because.

Because the security reasons are not considered good enough and giving false sense of security. Clearing the qemu images sounds good but there's another way to do that via trim.

I have a prototype code for that and after the years, seeing the request again, I'm not against adding it as long as it's not advertised as a security feature.

The zeroing simply builds on top of the trim code, so it's merely adding the ioctl interface and passing down the desired operation.

Hey.

I'd expect that anyone in the security area should know that securely deleting data is not done by overwriting it (not even overwriting it multiple times may be enough).
So I don't think that it would be btrfs' or zerofree's duty do teach that to users.

The later's manpage doesn't advertise it for such purpose and even contains a (though perhaps a bit too vague) warning:

They should probably drop the first "can be used to make it difficult" sentence... and add that even overwriting multiple times is often not enough.

Well I think what would be really mandatory if such support is added to an 3rd party tools is, that it will definitely continue to work (without causing corruptions or so), even if btrfs changes.

And if it's just using existing btrfs kernel code (and zerofree itself would mostly do nothing)... than that seems quite promising. :-)

I personally don't need it that desperate anymore, since I got discard support working in my qemu... but others may still benefit from it, so if it's easy, why not!? :-)

Cheers,
Chris.