Skip to contentSkip to conversation

Why do we need these mount options?

DISCUSSION waxhead 5 participants 14 messages Jan 14, 2021 – Jan 16, 2021 Btrfs file system

lore
Share & tools
lore

waxhead

author
#0 lore raw

Howdy,

I was looking through the mount options and being a madman with strong opinions I can't help thinking that a lot of them does not really belong as mount options at all, but should rather be properties set on the subvolume - for example the toplevel subvolume.

And any options set on a child subvolume should override the parrent subvolume the way I see it.

By having a quick look - I don't see why these should be mount options at all.

autodefrag / noautodefrag
commit
compress / compress-force
datacow / nodatacow
datasum / nodatasum
discard / nodiscard
inode_cache / noinode_cache
space_cache / nospace_cache
sdd / ssd_spread / nossd / no_ssdspread
user_subvol_rm_allowed

Stuff like compress and nodatacow can be set with chattr so there is as far as I am aware three methods of setting compression for example.

Either by mount options in fstab, by chattr or by btrfs property set

I think it would be more consistent to have one interface for adjusting behavior.

As I asked before, the future plan to have different storage profiles on subvolumes seem to have been sneakily(?) removed from the wiki - if that is indeed a dropped goal I can see why it makes sense to keep the mount options, if not I think the mount options should go in favor of btrfs property set.

#1 lore raw

Hi,

Quoted 4 lines · waxhead

I was looking through the mount options and being a madman with strong opinions I can't help thinking that a lot of them does not really belong as mount options at all, but should rather be properties set on the subvolume - for example the toplevel subvolume.

I agree that some of them should not be there but mount options still have their own usecase. They can be set from the outside and are supposed to affect the whole filesystem mount lifetime.

However, they've been used as default values for some operations, which is something that points more to what you suggest. And as they're not persistent and need to be stored in /etc/fstab is also weighing for storage inside the fs.

Quoted 2 lines

And any options set on a child subvolume should override the parrent subvolume the way I see it.

Yeah, that's one of the ways how to do it and I see it that way as well. Property set closer to the object takes precedence, roughly

mount < subvolume < directory < file

but last time we had a discussion about that, the other oppinion was that mount options beat everything, perhaps because they can be set from the outside and forced to ovrride whatever is on the filesystem.

Quoted 13 lines

By having a quick look - I don't see why these should be mount options at all.

autodefrag / noautodefrag
commit
compress / compress-force
datacow / nodatacow
datasum / nodatasum
discard / nodiscard
inode_cache / noinode_cache
space_cache / nospace_cache
sdd / ssd_spread / nossd / no_ssdspread
user_subvol_rm_allowed

So there are historical reasons and interface limitations that led to current state and multiple ways to do things.

Per-inode attributes were originally private ioctl of ext2 that other filesystems adopted due to feature parity, and as the interface was bit-based, no additional values could be set eg. compression, limited number of bits, no precedence, inter-flag dependencies.

Quoted 7 lines

Stuff like compress and nodatacow can be set with chattr so there is as far as I am aware three methods of setting compression for example.

Either by mount options in fstab, by chattr or by btrfs property set

I think it would be more consistent to have one interface for adjusting behavior.

I agree with that and there's a proposal to unify that into the properties as interface once for all, accessible through the extended attributes. But there are much more ways how to do that wrong so it hasn't been implemented so far.

A suggestion for an inode flag here and there comes from time to time, fixing one problem each time. Repeating that would lead to a mess that can be demonstrated on the existing mount options, so we've been there and need to do it the right way.

Quoted 2 lines

As I asked before, the future plan to have different storage profiles on subvolumes seem to have been sneakily(?) removed from the wiki

I don't think the per-subvolume storage options were ever tracked on wiki, the closest match is per-subvolume mount options that's still there

https://btrfs.wiki.kernel.org/index.php/Project_ideas#Per-subvolume_mount_options

Quoted 3 lines

- if that is indeed a dropped goal I can see why it makes sense to keep the mount options, if not I think the mount options should go in favor of btrfs property set.

author
#2 lore raw
Quoted 8 lines · David Sterba

Hi,

Quoted 4 lines · waxhead

I was looking through the mount options and being a madman with strong opinions I can't help thinking that a lot of them does not really belong as mount options at all, but should rather be properties set on the subvolume - for example the toplevel subvolume.

Quoted 4 lines

I agree that some of them should not be there but mount options still have their own usecase. They can be set from the outside and are supposed to affect the whole filesystem mount lifetime. Yes, some of them. But not all, the ones I list for example can

perfectly well be set on the toplevel subvolume.

Quoted 8 lines

However, they've been used as default values for some operations, which is something that points more to what you suggest. And as they're not persistent and need to be stored in /etc/fstab is also weighing for storage inside the fs.

Quoted 2 lines

And any options set on a child subvolume should override the parrent subvolume the way I see it.

Quoted 9 lines

Yeah, that's one of the ways how to do it and I see it that way as well. Property set closer to the object takes precedence, roughly

mount < subvolume < directory < file

but last time we had a discussion about that, the other oppinion was that mount options beat everything, perhaps because they can be set from the outside and forced to ovrride whatever is on the filesystem.

Well I agree with that. Mount options should beat everything and just because of that I think that some mount options should be deprecated and instead be set per. subvolume.

Quoted 14 lines
Quoted 13 lines

By having a quick look - I don't see why these should be mount options at all.

autodefrag / noautodefrag
commit
compress / compress-force
datacow / nodatacow
datasum / nodatasum
discard / nodiscard
inode_cache / noinode_cache
space_cache / nospace_cache
sdd / ssd_spread / nossd / no_ssdspread
user_subvol_rm_allowed

Quoted 8 lines

So there are historical reasons and interface limitations that led to current state and multiple ways to do things.

Per-inode attributes were originally private ioctl of ext2 that other filesystems adopted due to feature parity, and as the interface was bit-based, no additional values could be set eg. compression, limited number of bits, no precedence, inter-flag dependencies.

Ok thanks, I was not aware of that.

Quoted 5 lines
Quoted 5 lines

Stuff like compress and nodatacow can be set with chattr so there is as far as I am aware three methods of setting compression for example.

Either by mount options in fstab, by chattr or by btrfs property set

Quoted 8 lines
Quoted 2 lines

I think it would be more consistent to have one interface for adjusting behavior.

I agree with that and there's a proposal to unify that into the properties as interface once for all, accessible through the extended attributes. But there are much more ways how to do that wrong so it hasn't been implemented so far.

Good to know, and by the way another nugget of entertainment is that with btrfs property set the parameters come after the object. Usually command->params->target is IMHO the better way to go. It seems a bit backwards.

Quoted 5 lines

A suggestion for an inode flag here and there comes from time to time, fixing one problem each time. Repeating that would lead to a mess that can be demonstrated on the existing mount options, so we've been there and need to do it the right way.

Quoted 9 lines
Quoted 2 lines

As I asked before, the future plan to have different storage profiles on subvolumes seem to have been sneakily(?) removed from the wiki

I don't think the per-subvolume storage options were ever tracked on wiki, the closest match is per-subvolume mount options that's still there

https://btrfs.wiki.kernel.org/index.php/Project_ideas#Per-subvolume_mount_options

Well how about this from our friends archive.org ?
http://web.archive.org/web/20200117205248/https://btrfs.wiki.kernel.org/index.php/Main_Page

Here it clearly states that object level mirroring and striping is planned. Maybe I misinterpret this , but I understand this as (amongst other things) configurable storage profiles per subvolume.

Quoted 3 lines
Quoted 3 lines

- if that is indeed a dropped goal I can see why it makes sense to keep the mount options, if not I think the mount options should go in favor of btrfs property set.

#3 lore raw
Quoted 4 lines · waxhead
Quoted 4 lines

I don't think the per-subvolume storage options were ever tracked on wiki, the closest match is per-subvolume mount options that's still there

Quoted 8 lines
Quoted 2 lines

https://btrfs.wiki.kernel.org/index.php/Project_ideas#Per-subvolume_mount_options

Well how about this from our friends archive.org ?
http://web.archive.org/web/20200117205248/https://btrfs.wiki.kernel.org/index.php/Main_Page

Here it clearly states that object level mirroring and striping is planned. Maybe I misinterpret this , but I understand this as (amongst other things) configurable storage profiles per subvolume.

I see. The list on the main page is supposed to list features that we could promise to be implemented "soon". For all the ideas there's the specific project page wher it does not matter too much when it will implemented, it's kind of a pool.

In the wiki edit that removed the object-level storage I also removed
(https://btrfs.wiki.kernel.org/index.php?title=Main_Page&diff=prev&oldid=33190)

* Online filesystem check
* Object-level mirroring and striping
* In-band deduplication (happens during writes)
* Hot data tracking and moving to faster devices (or provided on the generic VFS layer)

For each of the task there's nobody working on that, to my knowledge, though there was some interest and maybe RFC patches in the past.

The object-level storage idea/task can be added to the Project_ideas page, so it's not lost.

author
#4 lore raw
Quoted 27 lines · David Sterba
Quoted 4 lines · waxhead
Quoted 4 lines

I don't think the per-subvolume storage options were ever tracked on wiki, the closest match is per-subvolume mount options that's still there

Quoted 8 lines
Quoted 2 lines

https://btrfs.wiki.kernel.org/index.php/Project_ideas#Per-subvolume_mount_options

Well how about this from our friends archive.org ?
http://web.archive.org/web/20200117205248/https://btrfs.wiki.kernel.org/index.php/Main_Page

Here it clearly states that object level mirroring and striping is planned. Maybe I misinterpret this , but I understand this as (amongst other things) configurable storage profiles per subvolume.

I see. The list on the main page is supposed to list features that we could promise to be implemented "soon". For all the ideas there's the specific project page wher it does not matter too much when it will implemented, it's kind of a pool.

In the wiki edit that removed the object-level storage I also removed
(https://btrfs.wiki.kernel.org/index.php?title=Main_Page&diff=prev&oldid=33190)

* Online filesystem check
* Object-level mirroring and striping
* In-band deduplication (happens during writes)
* Hot data tracking and moving to faster devices (or provided on the generic VFS layer)

Quoted 6 lines

For each of the task there's nobody working on that, to my knowledge, though there was some interest and maybe RFC patches in the past.

The object-level storage idea/task can be added to the Project_ideas page, so it's not lost.

Okeydok... good to know! :)

#5 lore raw
In reply to David Sterba
Quoted 17 lines · David Sterba

Hi,

Quoted 4 lines · waxhead

I was looking through the mount options and being a madman with strong opinions I can't help thinking that a lot of them does not really belong as mount options at all, but should rather be properties set on the subvolume - for example the toplevel subvolume.

I agree that some of them should not be there but mount options still have their own usecase. They can be set from the outside and are supposed to affect the whole filesystem mount lifetime.

However, they've been used as default values for some operations, which is something that points more to what you suggest. And as they're not persistent and need to be stored in /etc/fstab is also weighing for storage inside the fs.

Quoted 7 lines
Quoted 2 lines

And any options set on a child subvolume should override the parrent subvolume the way I see it.

Yeah, that's one of the ways how to do it and I see it that way as well. Property set closer to the object takes precedence, roughly

mount < subvolume < directory < file

Wearing my grumpy sysadmin hat, I have occasionally wanted the mount options to override the subvolume and inode operations. Examples below.

Quoted 8 lines

but last time we had a discussion about that, the other oppinion was that mount options beat everything, perhaps because they can be set from the outside and forced to ovrride whatever is on the filesystem.

Quoted 4 lines

By having a quick look - I don't see why these should be mount options at all.

autodefrag / noautodefrag

That makes sense as an inode property--you only want autodefrag on a few files and they're usually easy to spot.

Quoted 1 line
Quoted 1 line

inode_cache / noinode_cache

That one is already gone as of v5.11-rc1.

Quoted 3 lines
Quoted 3 lines

commit
space_cache / nospace_cache
sdd / ssd_spread / nossd / no_ssdspread

How could those be anything other than filesystem-wide options?

Quoted 1 line
Quoted 1 line

discard / nodiscard

Maybe, but probably requires too much introspection in a fast path (we'd have to add a check for the last owner of a deleted extent to see if it had 'discard' set on some parent level).

On the other hand, I'm in favor of deprecating the whole discard option and going with fstrim instead. discard in its current form tends to increase write wear rather than decrease it, especially on metadata-heavy workloads. discard is roughly equivalent to running fstrim thousands of times a day, which is clearly bad for many (most? all?) SSDs.

It might be possible to make the discard mount option's behavior more sane (e.g. discard only full chunks, configurable minimum discard length, discard only within data chunks, discard only once per hour, etc).

Quoted 3 lines
Quoted 3 lines

compress / compress-force
datacow / nodatacow
datasum / nodatasum

Here's where I prefer the mount option over the more local attributes, because I'd like filesystem-level sysadmin overrides for those. i.e. disallow all users, even privileged ones, from being able to create files that don't have csums or compression on a filesystem.

It might be better to allow the per-inode/subvol properties to be set, but have a mount option to override them. I don't want to deal with legacy applications that might throw an error if they get an error return from a nodatacow chattr, so "silently drop the chattr" is better than "prevent chattr with error."

I also want to store backup / live-mountable copies of filesystems that have these attributes set, but that's a lot more complicated to implement. The filesystem can't just ignore a nodatasum bit on an inode once it has been set, and I can solve the problem above the filesystem by modifying btrfs receive or other backup code to strip those bits and store them somewhere else.

Something like this seems mandatory to have working crypto integrity in the future, but silent data corruption on cheap SSDs is bad enough to make it a requirement for plaintext storage now.

compress has some significant holes in its per-inode interface: no way to specify compress level in an inode property, no way to specify force on some files but not others.

Compress is far easier to override from a mount option--we just don't look at the inode bits in needs_compress and everything else still works the same as before.

Quoted 1 line
Quoted 1 line

user_subvol_rm_allowed

I'd like "user_subvol_create_disallowed" too. Unprivileged users can create subvols, and that breaks backups that rely on atomic btrfs snapshots. It could be a feature (it allows users to exclude parts of their home directory from backups) but most users I've met who have discovered this "feature" the hard way didn't enjoy it.

Historically I had other reasons to disallow subvol creates by unprivileged users, but they are mostly removed in 4.18, now that 'rmdir' works on an empty subvol.

Quoted 38 lines

So there are historical reasons and interface limitations that led to current state and multiple ways to do things.

Per-inode attributes were originally private ioctl of ext2 that other filesystems adopted due to feature parity, and as the interface was bit-based, no additional values could be set eg. compression, limited number of bits, no precedence, inter-flag dependencies.

Quoted 7 lines

Stuff like compress and nodatacow can be set with chattr so there is as far as I am aware three methods of setting compression for example.

Either by mount options in fstab, by chattr or by btrfs property set

I think it would be more consistent to have one interface for adjusting behavior.

I agree with that and there's a proposal to unify that into the properties as interface once for all, accessible through the extended attributes. But there are much more ways how to do that wrong so it hasn't been implemented so far.

A suggestion for an inode flag here and there comes from time to time, fixing one problem each time. Repeating that would lead to a mess that can be demonstrated on the existing mount options, so we've been there and need to do it the right way.

Quoted 2 lines

As I asked before, the future plan to have different storage profiles on subvolumes seem to have been sneakily(?) removed from the wiki

I don't think the per-subvolume storage options were ever tracked on wiki, the closest match is per-subvolume mount options that's still there

https://btrfs.wiki.kernel.org/index.php/Project_ideas#Per-subvolume_mount_options

Quoted 3 lines

- if that is indeed a dropped goal I can see why it makes sense to keep the mount options, if not I think the mount options should go in favor of btrfs property set.

author
#6 lore raw
Quoted 7 lines · Zygo Blaxell
Quoted 4 lines
Quoted 4 lines

commit
space_cache / nospace_cache
sdd / ssd_spread / nossd / no_ssdspread

How could those be anything other than filesystem-wide options?

Well being me, I tend to live in a fantasy world where BTRFS have complete world domination and has become the VFS layer.
As I have nagged about before on this list - I really think that the only sensible way forward for BTRFS (or dare I say BTRFS2) would be to make it possible to assign "storage device groups" where you can make certain btrfs device ids belong to group a,b,c, etc...

And with that it would be possible to assign a weight to subvolumes so that they would be preferred to be stored on group a (SSD's perhaps), while other subvolumes would be stored mostly or exlusively on HDD's, Fast HDD's, Archival HDD's etc... So maybe a bit over enthusiastic in thinking perhaps , but hopefully you see now why I think it is right that this is not filesystem-wide , but subvolume baseed properties.

Quoted 12 lines
Quoted 1 line
Quoted 1 line

discard / nodiscard

Maybe, but probably requires too much introspection in a fast path (we'd have to add a check for the last owner of a deleted extent to see if it had 'discard' set on some parent level).

On the other hand, I'm in favor of deprecating the whole discard option and going with fstrim instead. discard in its current form tends to increase write wear rather than decrease it, especially on metadata-heavy workloads. discard is roughly equivalent to running fstrim thousands of times a day, which is clearly bad for many (most? all?) SSDs.

Quoted 4 lines

It might be possible to make the discard mount option's behavior more sane (e.g. discard only full chunks, configurable minimum discard length, discard only within data chunks, discard only once per hour, etc).

Interesting, it might as well make sense to perhaps use the free space cache and a slow LRU mechanism e.g. "these chunks has not been in use for 64 hours/days" or something similar.

Quoted 9 lines
Quoted 3 lines
Quoted 3 lines

compress / compress-force
datacow / nodatacow
datasum / nodatasum

Here's where I prefer the mount option over the more local attributes, because I'd like filesystem-level sysadmin overrides for those. i.e. disallow all users, even privileged ones, from being able to create files that don't have csums or compression on a filesystem.

Then how about a mount option that allow only root to do certain things? e.g. a security restriction.

Quoted 9 lines
Quoted 2 lines
Quoted 2 lines

user_subvol_rm_allowed

I'd like "user_subvol_create_disallowed" too. Unprivileged users can create subvols, and that breaks backups that rely on atomic btrfs snapshots. It could be a feature (it allows users to exclude parts of their home directory from backups) but most users I've met who have discovered this "feature" the hard way didn't enjoy it.

Quoted 4 lines

Historically I had other reasons to disallow subvol creates by unprivileged users, but they are mostly removed in 4.18, now that 'rmdir' works on an empty subvol.

Again see above...

#7 lore raw
Quoted 16 lines · waxhead
Quoted 7 lines · Zygo Blaxell
Quoted 4 lines
Quoted 4 lines

commit
space_cache / nospace_cache
sdd / ssd_spread / nossd / no_ssdspread

How could those be anything other than filesystem-wide options?

Well being me, I tend to live in a fantasy world where BTRFS have complete world domination and has become the VFS layer.
As I have nagged about before on this list - I really think that the only sensible way forward for BTRFS (or dare I say BTRFS2) would be to make it possible to assign "storage device groups" where you can make certain btrfs device ids belong to group a,b,c, etc...

Quoted 6 lines

And with that it would be possible to assign a weight to subvolumes so that they would be preferred to be stored on group a (SSD's perhaps), while other subvolumes would be stored mostly or exlusively on HDD's, Fast HDD's, Archival HDD's etc... So maybe a bit over enthusiastic in thinking perhaps , but hopefully you see now why I think it is right that this is not filesystem-wide , but subvolume baseed properties.

Sure, that's all wonderful, but it has nothing to do with any of those mount options. ;)

commit sets a timer that forces a filesystem-wide sync() every now and then. space_cache picks one of the allocator implementations, also for the entire filesystem. ssd and related options affect the behavior of the metadata allocator and superblocks.

Quoted 12 lines
Quoted 12 lines
Quoted 1 line
Quoted 1 line

discard / nodiscard

Maybe, but probably requires too much introspection in a fast path (we'd have to add a check for the last owner of a deleted extent to see if it had 'discard' set on some parent level).

On the other hand, I'm in favor of deprecating the whole discard option and going with fstrim instead. discard in its current form tends to increase write wear rather than decrease it, especially on metadata-heavy workloads. discard is roughly equivalent to running fstrim thousands of times a day, which is clearly bad for many (most? all?) SSDs.

Quoted 7 lines
Quoted 4 lines

It might be possible to make the discard mount option's behavior more sane (e.g. discard only full chunks, configurable minimum discard length, discard only within data chunks, discard only once per hour, etc).

Interesting, it might as well make sense to perhaps use the free space cache and a slow LRU mechanism e.g. "these chunks has not been in use for 64 hours/days" or something similar.

That would add more writes, as the free space cache is an on-disk entity. It might make sense to maintain a 'discard tree', which lists extents that have been freed but not yet discarded or overwritten, to make fstrim more efficient. This wouldn't have to be very precise, just pointing to general regions of the disk (maybe even entire block groups) so fstrim doesn't issue discards to idle areas of the disk over and over.

Currently the discard extent list is stored in memory, so doing one discard per T time units would use more memory. This feature would be like discard=async, but 1) it would hold on to the pinned extents for a few hundred transactions instead of just one or two (subject to memory availability), and 2) it would be able to reclaim space from the discard list as free space, thus removing the need to issue a discard at all.

But that's really complicated, considering that a cron job that runs fstrim once an hour can do the same thing without all the complexity. On the other hand, I just ran fstrim on a test machine and it took 34 minutes, so maybe some complexity might be useful after all... :-O

Quoted 11 lines
Quoted 9 lines
Quoted 3 lines
Quoted 3 lines

compress / compress-force
datacow / nodatacow
datasum / nodatasum

Here's where I prefer the mount option over the more local attributes, because I'd like filesystem-level sysadmin overrides for those. i.e. disallow all users, even privileged ones, from being able to create files that don't have csums or compression on a filesystem.

Then how about a mount option that allow only root to do certain things? e.g. a security restriction.

No, I don't want root doing those things either. Most of the applications I want to bring to heel are already running as root.

Basically I want to say "every file on this filesystem shall be datacow and datasum" and (short of altering the mount option) no other kind of file can be created.

It might possibly make more sense to do this through tunefs--that way the filesystem couldn't ever have nodatacow files (new kernels would refuse, old kernels wouldn't be able to mount through a new feature flag).

Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line

user_subvol_rm_allowed

I'd like "user_subvol_create_disallowed" too. Unprivileged users can create subvols, and that breaks backups that rely on atomic btrfs snapshots. It could be a feature (it allows users to exclude parts of their home directory from backups) but most users I've met who have discovered this "feature" the hard way didn't enjoy it.

Quoted 5 lines
Quoted 4 lines

Historically I had other reasons to disallow subvol creates by unprivileged users, but they are mostly removed in 4.18, now that 'rmdir' works on an empty subvol.

Again see above...

Here, unlike above, I was already asking precisely for subvol create to be made root only.

That, or make snapshots recursive and atomic to avoid the accidental user data loss/corruption case.

author
#8 lore raw
Quoted 24 lines · Zygo Blaxell
Quoted 16 lines · waxhead
Quoted 7 lines · Zygo Blaxell
Quoted 4 lines
Quoted 4 lines

commit
space_cache / nospace_cache
sdd / ssd_spread / nossd / no_ssdspread

How could those be anything other than filesystem-wide options?

Well being me, I tend to live in a fantasy world where BTRFS have complete world domination and has become the VFS layer.
As I have nagged about before on this list - I really think that the only sensible way forward for BTRFS (or dare I say BTRFS2) would be to make it possible to assign "storage device groups" where you can make certain btrfs device ids belong to group a,b,c, etc...

Quoted 6 lines

And with that it would be possible to assign a weight to subvolumes so that they would be preferred to be stored on group a (SSD's perhaps), while other subvolumes would be stored mostly or exlusively on HDD's, Fast HDD's, Archival HDD's etc... So maybe a bit over enthusiastic in thinking perhaps , but hopefully you see now why I think it is right that this is not filesystem-wide , but subvolume baseed properties.

Quoted 8 lines

Sure, that's all wonderful, but it has nothing to do with any of those mount options. ;)

commit sets a timer that forces a filesystem-wide sync() every now and then. space_cache picks one of the allocator implementations, also for the entire filesystem. ssd and related options affect the behavior of the metadata allocator and superblocks.

Ok I understand the space_cache, but commit (the way I understand it) could be useful to keep stuff in memory for longer for certain subvolume.
ssd options could also be useful pr. subvolume *if* and that is a big if BTRFS sometime in the far future allows for storage device groups. However when that happens maybe everything is solid state , who knows ;)

Quoted 34 lines
Quoted 12 lines
Quoted 12 lines
Quoted 1 line
Quoted 1 line

discard / nodiscard

Maybe, but probably requires too much introspection in a fast path (we'd have to add a check for the last owner of a deleted extent to see if it had 'discard' set on some parent level).

On the other hand, I'm in favor of deprecating the whole discard option and going with fstrim instead. discard in its current form tends to increase write wear rather than decrease it, especially on metadata-heavy workloads. discard is roughly equivalent to running fstrim thousands of times a day, which is clearly bad for many (most? all?) SSDs.

Quoted 7 lines
Quoted 4 lines

It might be possible to make the discard mount option's behavior more sane (e.g. discard only full chunks, configurable minimum discard length, discard only within data chunks, discard only once per hour, etc).

Interesting, it might as well make sense to perhaps use the free space cache and a slow LRU mechanism e.g. "these chunks has not been in use for 64 hours/days" or something similar.

That would add more writes, as the free space cache is an on-disk entity. It might make sense to maintain a 'discard tree', which lists extents that have been freed but not yet discarded or overwritten, to make fstrim more efficient. This wouldn't have to be very precise, just pointing to general regions of the disk (maybe even entire block groups) so fstrim doesn't issue discards to idle areas of the disk over and over.

Currently the discard extent list is stored in memory, so doing one discard per T time units would use more memory. This feature would be like discard=async, but 1) it would hold on to the pinned extents for a few hundred transactions instead of just one or two (subject to memory availability), and 2) it would be able to reclaim space from the discard list as free space, thus removing the need to issue a discard at all.

Quoted 5 lines

But that's really complicated, considering that a cron job that runs fstrim once an hour can do the same thing without all the complexity. On the other hand, I just ran fstrim on a test machine and it took 34 minutes, so maybe some complexity might be useful after all... :-O

Thanks for the education. Inspired by this I ran fstrim on my main machine with 7 older and smaller SSD's. Have not run fstrim on it or maybe a year so it took 15-20 minutes to do. But I see your point. Minimizing writes to SSD's is a good thing, but if I am not mistaking SMR disks can benefit from discard as well, I don't know the details but maybe it would be more beneficial on real disks than SSD storage?

Quoted 15 lines
Quoted 11 lines
Quoted 9 lines
Quoted 3 lines
Quoted 3 lines

compress / compress-force
datacow / nodatacow
datasum / nodatasum

Here's where I prefer the mount option over the more local attributes, because I'd like filesystem-level sysadmin overrides for those. i.e. disallow all users, even privileged ones, from being able to create files that don't have csums or compression on a filesystem.

Then how about a mount option that allow only root to do certain things? e.g. a security restriction.

No, I don't want root doing those things either. Most of the applications I want to bring to heel are already running as root.

Quoted 8 lines

Basically I want to say "every file on this filesystem shall be datacow and datasum" and (short of altering the mount option) no other kind of file can be created.

It might possibly make more sense to do this through tunefs--that way the filesystem couldn't ever have nodatacow files (new kernels would refuse, old kernels wouldn't be able to mount through a new feature flag).

Allrighty point taken.

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 1 line
Quoted 1 line

user_subvol_rm_allowed

I'd like "user_subvol_create_disallowed" too. Unprivileged users can create subvols, and that breaks backups that rely on atomic btrfs snapshots. It could be a feature (it allows users to exclude parts of their home directory from backups) but most users I've met who have discovered this "feature" the hard way didn't enjoy it.

Historically I had other reasons to disallow subvol creates by unprivileged users, but they are mostly removed in 4.18, now that 'rmdir' works on an empty subvol.

Quoted 8 lines
Quoted 1 line

Again see above...

Here, unlike above, I was already asking precisely for subvol create to be made root only.

That, or make snapshots recursive and atomic to avoid the accidental user data loss/corruption case.

Allrighty again! :)

#9 lore raw
Quoted 30 lines · waxhead
Quoted 27 lines · Zygo Blaxell
Quoted 16 lines · waxhead
Quoted 7 lines · Zygo Blaxell
Quoted 4 lines
Quoted 4 lines

commit
space_cache / nospace_cache
sdd / ssd_spread / nossd / no_ssdspread

How could those be anything other than filesystem-wide options?

Well being me, I tend to live in a fantasy world where BTRFS have complete world domination and has become the VFS layer.
As I have nagged about before on this list - I really think that the only sensible way forward for BTRFS (or dare I say BTRFS2) would be to make it possible to assign "storage device groups" where you can make certain btrfs device ids belong to group a,b,c, etc...

Quoted 6 lines

And with that it would be possible to assign a weight to subvolumes so that they would be preferred to be stored on group a (SSD's perhaps), while other subvolumes would be stored mostly or exlusively on HDD's, Fast HDD's, Archival HDD's etc... So maybe a bit over enthusiastic in thinking perhaps , but hopefully you see now why I think it is right that this is not filesystem-wide , but subvolume baseed properties.

Sure, that's all wonderful, but it has nothing to do with any of those mount options. ;)

Quoted 7 lines
Quoted 5 lines

commit sets a timer that forces a filesystem-wide sync() every now and then. space_cache picks one of the allocator implementations, also for the entire filesystem. ssd and related options affect the behavior of the metadata allocator and superblocks.

Ok I understand the space_cache, but commit (the way I understand it) could be useful to keep stuff in memory for longer for certain subvolume.

That's probably something to implement mostly above the filesystem level, as it's VFS that controls most of writeback, and maybe memcg could be adapted to have per-cgroup writeback settings.

The btrfs transaction commit updates all of the filesystem trees at once (they are all children of the root subtree's root node). Changing that would be a significant restructuring.

Quoted 3 lines

ssd options could also be useful pr. subvolume *if* and that is a big if BTRFS sometime in the far future allows for storage device groups. However when that happens maybe everything is solid state , who knows ;)

First the ssd options have to do something that could be per-subvolume. Right now they do almost nothing, and the little they do is per-device.

There are various proposals for HSM, metadata-on-SSD, hot data tracking floating around. If they ever arrive at btrfs, they definitely won't be tacked onto the ssd mount option.

Quoted 39 lines
Quoted 34 lines
Quoted 12 lines
Quoted 12 lines
Quoted 1 line
Quoted 1 line

discard / nodiscard

Maybe, but probably requires too much introspection in a fast path (we'd have to add a check for the last owner of a deleted extent to see if it had 'discard' set on some parent level).

On the other hand, I'm in favor of deprecating the whole discard option and going with fstrim instead. discard in its current form tends to increase write wear rather than decrease it, especially on metadata-heavy workloads. discard is roughly equivalent to running fstrim thousands of times a day, which is clearly bad for many (most? all?) SSDs.

Quoted 7 lines
Quoted 4 lines

It might be possible to make the discard mount option's behavior more sane (e.g. discard only full chunks, configurable minimum discard length, discard only within data chunks, discard only once per hour, etc).

Interesting, it might as well make sense to perhaps use the free space cache and a slow LRU mechanism e.g. "these chunks has not been in use for 64 hours/days" or something similar.

That would add more writes, as the free space cache is an on-disk entity. It might make sense to maintain a 'discard tree', which lists extents that have been freed but not yet discarded or overwritten, to make fstrim more efficient. This wouldn't have to be very precise, just pointing to general regions of the disk (maybe even entire block groups) so fstrim doesn't issue discards to idle areas of the disk over and over.

Currently the discard extent list is stored in memory, so doing one discard per T time units would use more memory. This feature would be like discard=async, but 1) it would hold on to the pinned extents for a few hundred transactions instead of just one or two (subject to memory availability), and 2) it would be able to reclaim space from the discard list as free space, thus removing the need to issue a discard at all.

Quoted 5 lines

But that's really complicated, considering that a cron job that runs fstrim once an hour can do the same thing without all the complexity. On the other hand, I just ran fstrim on a test machine and it took 34 minutes, so maybe some complexity might be useful after all... :-O

Quoted 6 lines

Thanks for the education. Inspired by this I ran fstrim on my main machine with 7 older and smaller SSD's. Have not run fstrim on it or maybe a year so it took 15-20 minutes to do. But I see your point. Minimizing writes to SSD's is a good thing, but if I am not mistaking SMR disks can benefit from discard as well, I don't know the details but maybe it would be more beneficial on real disks than SSD storage?

Overwriting data at a given LBA implies discarding the data currently in the LBA, regardless of underlying storage. discard is useful on SSD, SMR, and also caching/thin LV layers if the LBAs are not going to be overwritten for a long time (thin LVs take less space, caches have more room for other data). There is no need to send a separate discard command if the LBAs in question are not part of a file at the time, and there is a high probability of overwriting the same LBA with new data very soon.

There are multiple variants of the discard operation in devices. One variant leaves the contents of the discarded blocks undefined. That's a nice feature, since the device is free to ignore the discard request if it might improve performance (i.e. there is no need to ensure the discard has immediate persistent effect, so it is acceptable for the drive to not flush discards to persistent storage right away, and merge the discard into some later write command to save one write cycle).

Most of the SSDs I've tested do a different variant of discard, where the discarded blocks are guaranteed to return zeros. That forces the drive to do at least one flash write for each discard. On such drives it's important to not send redundant discards from the filesystem, as those discards are necessarily wasting write cycles.

On SMR drives, redundant discards are even worse than on SSD. SMR drives have address translation layers like SSD, but they are stored on spinning disks. It will take milliseconds to move heads to record the discard in the translation layer, only to spend more milliseconds to write the replacement data in the same location immediately after. The disk platter has higher endurance than SSD, but can be orders of magnitude slower if there are too many translation layer updates.

Quoted 46 lines
Quoted 15 lines
Quoted 11 lines
Quoted 9 lines
Quoted 3 lines
Quoted 3 lines

compress / compress-force
datacow / nodatacow
datasum / nodatasum

Here's where I prefer the mount option over the more local attributes, because I'd like filesystem-level sysadmin overrides for those. i.e. disallow all users, even privileged ones, from being able to create files that don't have csums or compression on a filesystem.

Then how about a mount option that allow only root to do certain things? e.g. a security restriction.

No, I don't want root doing those things either. Most of the applications I want to bring to heel are already running as root.

Quoted 8 lines

Basically I want to say "every file on this filesystem shall be datacow and datasum" and (short of altering the mount option) no other kind of file can be created.

It might possibly make more sense to do this through tunefs--that way the filesystem couldn't ever have nodatacow files (new kernels would refuse, old kernels wouldn't be able to mount through a new feature flag).

Allrighty point taken.

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 1 line
Quoted 1 line

user_subvol_rm_allowed

I'd like "user_subvol_create_disallowed" too. Unprivileged users can create subvols, and that breaks backups that rely on atomic btrfs snapshots. It could be a feature (it allows users to exclude parts of their home directory from backups) but most users I've met who have discovered this "feature" the hard way didn't enjoy it.

Historically I had other reasons to disallow subvol creates by unprivileged users, but they are mostly removed in 4.18, now that 'rmdir' works on an empty subvol.

Quoted 8 lines
Quoted 1 line

Again see above...

Here, unlike above, I was already asking precisely for subvol create to be made root only.

That, or make snapshots recursive and atomic to avoid the accidental user data loss/corruption case.

Allrighty again! :)

#10 lore raw
In reply to Zygo Blaxell

15.01.2021 06:54, Zygo Blaxell пишет:

Quoted 7 lines

On the other hand, I'm in favor of deprecating the whole discard option and going with fstrim instead. discard in its current form tends to increase write wear rather than decrease it, especially on metadata-heavy workloads. discard is roughly equivalent to running fstrim thousands of times a day, which is clearly bad for many (most? all?) SSDs.

My (probably naive) understanding so far was that trim on SSD marks areas as "unused" which means SSD need to copy less residual data from erase block when reusing it. Assuming TRIM unit is (significantly) smaller than erase block.

I would appreciate if you elaborate how trim results in more write on SSD?

Or do you mean more writes from btrfs while performing discard?

#11 lore raw
Quoted 7 lines · Andrei Borzenkov

15.01.2021 06:54, Zygo Blaxell пишет:

Quoted 5 lines

On the other hand, I'm in favor of deprecating the whole discard option and going with fstrim instead. discard in its current form tends to increase write wear rather than decrease it, especially on metadata-heavy workloads. discard is roughly equivalent to running fstrim thousands of times a day, which is clearly bad for many (most? all?) SSDs.

Quoted 6 lines

My (probably naive) understanding so far was that trim on SSD marks areas as "unused" which means SSD need to copy less residual data from erase block when reusing it. Assuming TRIM unit is (significantly) smaller than erase block.

I would appreciate if you elaborate how trim results in more write on SSD?

The areas are not only marked as unused, but also zeroed. To keep the zeroing semantic, every discard must be persisted, thus requiring a write to the SSD's metadata (not btrfs metadata) area.

Meow!

Signature · 4 lines
⢀⣴⠾⠻⢶⣦⠀ .--[ Makefile ]
⣾⠁⢠⠒⠀⣿⡁ # beware of races
⢿⡄⠘⠷⠚⠋⠀ all: pillage burn
⠈⠳⣄⠀⠀⠀⠀ `----
#12 lore raw

16.01.2021 18:19, Adam Borowski пишет:

Quoted 13 lines · Andrei Borzenkov
Quoted 12 lines

15.01.2021 06:54, Zygo Blaxell пишет:

Quoted 5 lines

On the other hand, I'm in favor of deprecating the whole discard option and going with fstrim instead. discard in its current form tends to increase write wear rather than decrease it, especially on metadata-heavy workloads. discard is roughly equivalent to running fstrim thousands of times a day, which is clearly bad for many (most? all?) SSDs.

My (probably naive) understanding so far was that trim on SSD marks areas as "unused" which means SSD need to copy less residual data from erase block when reusing it. Assuming TRIM unit is (significantly) smaller than erase block.

Quoted 6 lines
Quoted 1 line

I would appreciate if you elaborate how trim results in more write on SSD?

The areas are not only marked as unused, but also zeroed. To keep the zeroing semantic, every discard must be persisted, thus requiring a write to the SSD's metadata (not btrfs metadata) area.

There is no requirement that TRIM did it. If device sets RZAT SUPPORTED bit, it should return zeroes for trimmed range, but there is no need to physically zero anything - simply return zeroes for areas marked as unallocated. Discard must be persisted in allocation table, but then every write must be persisted in allocation table anyway.

Moreover, to actually zero on TRIM either trim request must be issued for the full erase block or device must perform garbage collection.

Do you have any links that show that discards increase write load on physical media? I am really curious.

#13 lore raw
Quoted 21 lines · Andrei Borzenkov

16.01.2021 18:19, Adam Borowski пишет:

Quoted 13 lines · Andrei Borzenkov
Quoted 12 lines

15.01.2021 06:54, Zygo Blaxell пишет:

Quoted 5 lines

On the other hand, I'm in favor of deprecating the whole discard option and going with fstrim instead. discard in its current form tends to increase write wear rather than decrease it, especially on metadata-heavy workloads. discard is roughly equivalent to running fstrim thousands of times a day, which is clearly bad for many (most? all?) SSDs.

My (probably naive) understanding so far was that trim on SSD marks areas as "unused" which means SSD need to copy less residual data from erase block when reusing it. Assuming TRIM unit is (significantly) smaller than erase block.

Quoted 6 lines
Quoted 1 line

I would appreciate if you elaborate how trim results in more write on SSD?

The areas are not only marked as unused, but also zeroed. To keep the zeroing semantic, every discard must be persisted, thus requiring a write to the SSD's metadata (not btrfs metadata) area.

Quoted 5 lines

There is no requirement that TRIM did it. If device sets RZAT SUPPORTED bit, it should return zeroes for trimmed range, but there is no need to physically zero anything - simply return zeroes for areas marked as unallocated. Discard must be persisted in allocation table, but then every write must be persisted in allocation table anyway.

That is exactly the problem--the persistence is a write that counts against total drive wear. That is why TRIM variants that leave the contents of the discarded LBAs undefined are better than those which define the contents as zero.

The effect seems to be the equivalent of a small write, i.e. a 16K write might be the same cost as any length of contiguous discard. So it's OK to discard block-group-sized regions, but not OK to issue one discard for every metadata free page hole. Different drives have different ratios between these costs, so parity might occur at 4K or 256K depending on the drive.

AIUI there is a minimum discard length filter implemented in btrfs already, so maybe it just needs tuning?

Quoted 5 lines

Moreover, to actually zero on TRIM either trim request must be issued for the full erase block or device must perform garbage collection.

Do you have any links that show that discards increase write load on physical media? I am really curious.

I have no links, it's a directly observed result.

It's fairly straightforward to replicate: Set up a machine to do git checkouts of each Linux kernel tag in random order, in a loop (maybe multiple instances of this if needed to get the SSD device IO saturated). While that happens, watch the percentage used endurance indicator reported on the drives (smartctl -x). Wait for the indicator to increment twice, and measure the time between the first and second increment. Use a low-cost consumer or OEM SSD so you get results in less than a few hundred hours. Then mount -o discard=async and wait for two more increments. Assuming the workload produces constant amounts of IO over time, and the percentage used endurance indicator variable from SMART is not a complete lie, the time between increments should roughly indicate the wear rates of the different workloads.

In the field, we discovered this on CI builder workloads (lots of tiny files created, destroyed, and created again in rapid succession). They get almost double the SSD wear rate with discard on vs. discard off. We have monitoring on the p-u-e-i variable, and use it to project the date when 100% endurance will be reached. If that date lands within the date range when we want to be using the SSD, we get an alert. When discard is accidentally enabled on a CI server due to a configuration failure, we get an alert about a week later, as it shortens our drives' projected lifespan from more than 6 years to less than 4.

Other workloads are less sensitive to this. If the workload has fewer metadata updates, bigger files, and sequential writes, then discard doesn't have a negative effect--though to be fair, it doesn't seem to have a positive effect either, at least not by this measurement method.