Selective Compression/Encryption
Currently compression and I assume if encryption is implemented it is turned on or off during mount. There are however many times when a user may want to select which files/directories they want to compress or encrypt. This will also be helpful when implementing btrfs support in grub for example. We can say the disk can be compressed/encrypted except for /boot so compression/encryption doesn't have to be implemented in grub.
I was thinking of adding this functionality to the userspace application btrfstune. The way I was thinking of doing this is when btrfstune +c is applied to a directory or file the directory(and all its contents) or file will always be compressed reguardless of how the filesystem is mounted. The opposite would happen when btrfstune -c is used.
Would this be a reasonable thing to implement? Any suggestions before I start doing this?
Lee
Hi there,
Quoted 7 lines · Lee Trager
Currently compression and I assume if encryption is implemented it is turned on or off during mount. There are however many times when a user may want to select which files/directories they want to compress or encrypt. This will also be helpful when implementing btrfs support in grub for example. We can say the disk can be compressed/encrypted except for /boot so compression/encryption doesn't have to be implemented in grub.
Quoted 9 lines
I was thinking of adding this functionality to the userspace application btrfstune. The way I was thinking of doing this is when btrfstune +c is applied to a directory or file the directory(and all its contents) or file will always be compressed reguardless of how the filesystem is mounted. The opposite would happen when btrfstune -c is used.
Would this be a reasonable thing to implement? Any suggestions before I start doing this?
Things like compression or encription should be used at the "volume" level. So.. if a user wants a specific set of files or dirs ..they should create a mount-point/volume like:
private_volbigarchives_vol
and set those volumes as compressed or encripted volumes
Regarding usability, the best would be for the sub-volume creation tool to optionally allow passing encription/compression arguments.
and then:
should mount those volumes somewhere like: ~/Confidential or ~/Archives.
Basically, do it at the directory level (which in btrfs is at the sub-volume level).
File-level granularity is totally unmanageable in the long term.
Kind regards,
Signature · 1 line
Miguel Sousa Filipe
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 12 lines · Christian Hesse
Quoted 10 lines · Miguel Figueiredo Mascarenhas Sousa Filipe
Hi there,
Quoted 7 lines · Lee Trager
Currently compression and I assume if encryption is implemented it is turned on or off during mount. There are however many times when a user may want to select which files/directories they want to compress or encrypt. This will also be helpful when implementing btrfs support in grub for example. We can say the disk can be compressed/encrypted except for /boot so compression/encryption doesn't have to be implemented in grub.
Quoted 3 lines
You could just use an additional partition for /boot that has compression an encryption disabled...
Yes you could but many distributions are moving away from that such as ubuntu. Also while this works well on desktop and server environments where space is not an issue on embedded systems where free space is measured in kb or mb it starts becomming a huge issue.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
I was thinking of adding this functionality to the userspace application btrfstune. The way I was thinking of doing this is when btrfstune +c is applied to a directory or file the directory(and all its contents) or file will always be compressed reguardless of how the filesystem is mounted. The opposite would happen when btrfstune -c is used.
Would this be a reasonable thing to implement? Any suggestions before I start doing this?
Quoted 9 lines
Quoted 1 line
Things like compression or encription should be used at the "volume" level.
That was what I said some time ago when I asked why encryption support for btrfs is planned. On a volume level you can use dm-crypt and the fs can ignore that part completely.
The answer was that different users on a home partition could use their own encryption key. That sounds like volume level is out of bet. ;)
It does seem that doing it with volumes would limit user control and add lots of complexity to such a simple task.
Miguel, what is your reason for suggesting this be done at the volume level?
Quoted 12 lines
Quoted 12 lines
So.. if a user wants a specific set of files or dirs ..they should create a mount-point/volume like:
private_volbigarchives_voland set those volumes as compressed or encripted volumes
Regarding usability, the best would be for the sub-volume creation tool to optionally allow passing encription/compression arguments.
Quoted 12 lines
Quoted 8 lines
and then:
should mount those volumes somewhere like: ~/Confidential or ~/Archives.Basically, do it at the directory level (which in btrfs is at the sub-volume level).
File-level granularity is totally unmanageable in the long term.Kind regards,
Signature · 2 lines
Regards, Chris
Lee
Quoted 36 lines · Lee Trager
Quoted 12 lines · Christian Hesse
Quoted 10 lines · Miguel Figueiredo Mascarenhas Sousa Filipe
Hi there,
Quoted 7 lines · Lee Trager
Currently compression and I assume if encryption is implemented it is turned on or off during mount. There are however many times when a user may want to select which files/directories they want to compress or encrypt. This will also be helpful when implementing btrfs support in grub for example. We can say the disk can be compressed/encrypted except for /boot so compression/encryption doesn't have to be implemented in grub.
Quoted 3 lines
You could just use an additional partition for /boot that has compression an encryption disabled...
Yes you could but many distributions are moving away from that such as ubuntu. Also while this works well on desktop and server environments where space is not an issue on embedded systems where free space is measured in kb or mb it starts becomming a huge issue.
Quoted 15 lines
Quoted 10 lines
Quoted 8 lines
I was thinking of adding this functionality to the userspace application btrfstune. The way I was thinking of doing this is when btrfstune +c is applied to a directory or file the directory(and all its contents) or file will always be compressed reguardless of how the filesystem is mounted. The opposite would happen when btrfstune -c is used.
Would this be a reasonable thing to implement? Any suggestions before I start doing this?
Things like compression or encription should be used at the "volume" level.
That was what I said some time ago when I asked why encryption support for btrfs is planned. On a volume level you can use dm-crypt and the fs can ignore that part completely.
Quoted 7 lines
Quoted 3 lines
The answer was that different users on a home partition could use their own encryption key. That sounds like volume level is out of bet. ;)
It does seem that doing it with volumes would limit user control and add lots of complexity to such a simple task.
Miguel, what is your reason for suggesting this be done at the volume level?
Well, basically, the main issue I have with file level configuration, is maintenance nightmare in the long term.
Redundancy level settings (raid0,1,5,6..), and compression/encription, to me, seem like a sub-volume setting.
Its the right granularity to manage and mantain data and interact with it. If I want this folders compressed.. I can just move them to my compressed sub-vol.
If I want files: a, b/weird_path/D, and NSA/* confidential, I put them in my encripted sub-vol.
Imagine working on a confidencial project, where you add new files/folders practilly everyday, and every single day, you must remember to (and bother to do extra word so that..) set that file as a encripted one.
Also, by using sub-vols like this, and if redudancy is applied at the same level, things just play nice, I can have distinct mount-points that represent data that I want with a given set of redundancy/compression and encription rules.
It becomes easier to mantain, and use.
Just imagine having a "confidencial" or a "compressed" mountpoint/disk
ICON in you desktop, and just knowing that files in there are
encripted and/or compressed.
Versus, having to browser your file and look for a specific "encripted" or "compressed" flag, that the file explorer shows if they play nice...
Now... if its at the file level, It can be troublesome to look for allfiles encripted/compressed.
Also, knowing the encription key to each file (if it's set at the file level) ... argh!!
Doing this file based, seems to me that will lead to complication the long term maintenance of things.
In the long term, when someone picks up, or mounts a old, used system, I can't imagine the butload of work involved in finding out things like:
- what in this 4TB fs is compressed, what is encripted?
- (in case of redundancy at the file level): how many redundancy levels do I have, and which files respectively?
- why is rsync/scp 'ing this folder/BIG_NESTED_TREE_OF_FILES allways "stop" at folder/**/fileB and folder/z*/* ?
(might it be that some unsuspecting file in there has a trully beasty
compression ratio, and that 4GB fileB (in the middle of other 4GB files), is instead a 80Gb file compressed ?
I also don't like extended attributes, we're just creating a whole new orthogonal dimension/namespace were we store/manipulate information.. it also complicates backups (does the backup system understand and respects ext. attrs?)
It's messier doing these things with extended attributes or very custom tools, instead of levelaging/creating a volume management tool, for managing this, such tool will allready be there to manage the fisical storage <-> filesystems/volumes mappings and settings.. might has well leverage it.
Kind regards,
Quoted 28 lines
Quoted 20 lines
Quoted 20 lines
So.. if a user wants a specific set of files or dirs ..they should create a mount-point/volume like:
private_volbigarchives_voland set those volumes as compressed or encripted volumes
Regarding usability, the best would be for the sub-volume creation tool to optionally allow passing encription/compression arguments.
and then:
should mount those volumes somewhere like: ~/Confidential or ~/Archives.Basically, do it at the directory level (which in btrfs is at the sub-volume level).
File-level granularity is totally unmanageable in the long term.Quoted 5 lines
Quoted 1 line
Kind regards,
--
Regards,
ChrisLee
Signature · 1 line
Miguel Sousa Filipe
A message that is not in the archive (sent off-list or not yet received): 1228863790.8130.14.camel@mattos-laptop
Hi there,
Quoted 9 lines · Oliver Mattos
Hi,
It looks to me like if all the attributes were per file, and files automatically inherited settings from the parent, and users weren't allowed to change inherited permissions (only root could), then you could get the same effect as per volume controls.
The above could be enforced at the option of the sysAdmin\distro.
Quoted 8 lines
On the other hand if the attributes are per volume then you can't individually set attributes for individual files unless you create one volume per file, which doesn't sound like a clean solution.
As far as I can see, all the use cases that call for per-volume configuration are also satisfied by per file configuration with inheritance, but not vice versa.
You have convinced me.
If this works with default inheritance enabled, so that not only child/leaf files of a dir tree get the dir setting, but also new files, then, it fits my usage model and "wanted" features :)
Quoted 10 lines
An example where I would want compressed and uncompressed files in the same folder is where I have a Virtual Machine with two disks, each stored in a disk image on a btrfs partition - disk 1 is the root filesystem and compressed to save space, since it contains a lot of "zero" bytes, but disk 2 is the virtual swap disk, which isn't compressed due to needing fast low overhead access. Since both files are associated with the same virtual machine I want to keep them in the same folder on the same volume.
That's a good realistic scenario were per-volume settings indeed fail to behave "properly".
Quoted 158 lines
To resolve the problem with "random" attributes being set on random files, I propose a utility is created to reset all attributes on all files in a directory tree. Another utility could list all non-inherited (ie. not the same as the parent) attributes, then you can very quickly see which directories are compressed\encrypted.
Thanks
OliverQuoted 145 lines · Miguel Figueiredo Mascarenhas Sousa Filipe
Quoted 36 lines · Lee Trager
Quoted 12 lines · Christian Hesse
Quoted 10 lines · Miguel Figueiredo Mascarenhas Sousa Filipe
Hi there,
Quoted 7 lines · Lee Trager
Currently compression and I assume if encryption is implemented it is turned on or off during mount. There are however many times when a user may want to select which files/directories they want to compress or encrypt. This will also be helpful when implementing btrfs support in grub for example. We can say the disk can be compressed/encrypted except for /boot so compression/encryption doesn't have to be implemented in grub.
Quoted 3 lines
You could just use an additional partition for /boot that has compression an encryption disabled...
Yes you could but many distributions are moving away from that such as ubuntu. Also while this works well on desktop and server environments where space is not an issue on embedded systems where free space is measured in kb or mb it starts becomming a huge issue.
Quoted 15 lines
Quoted 10 lines
Quoted 8 lines
I was thinking of adding this functionality to the userspace application btrfstune. The way I was thinking of doing this is when btrfstune +c is applied to a directory or file the directory(and all its contents) or file will always be compressed reguardless of how the filesystem is mounted. The opposite would happen when btrfstune -c is used.
Would this be a reasonable thing to implement? Any suggestions before I start doing this?
Things like compression or encription should be used at the "volume" level.
That was what I said some time ago when I asked why encryption support for btrfs is planned. On a volume level you can use dm-crypt and the fs can ignore that part completely.
Quoted 7 lines
Quoted 3 lines
The answer was that different users on a home partition could use their own encryption key. That sounds like volume level is out of bet. ;)
It does seem that doing it with volumes would limit user control and add lots of complexity to such a simple task.
Miguel, what is your reason for suggesting this be done at the volume level?
Well, basically, the main issue I have with file level configuration, is maintenance nightmare in the long term.
Redundancy level settings (raid0,1,5,6..), and compression/encription, to me, seem like a sub-volume setting.
Its the right granularity to manage and mantain data and interact with it. If I want this folders compressed.. I can just move them to my compressed sub-vol.
If I want files: a, b/weird_path/D, and NSA/* confidential, I put them in my encripted sub-vol.Imagine working on a confidencial project, where you add new files/folders practilly everyday, and every single day, you must remember to (and bother to do extra word so that..) set that file as a encripted one.
Also, by using sub-vols like this, and if redudancy is applied at the same level, things just play nice, I can have distinct mount-points that represent data that I want with a given set of redundancy/compression and encription rules.
It becomes easier to mantain, and use.
Just imagine having a "confidencial" or a "compressed" mountpoint/disk
ICON in you desktop, and just knowing that files in there are
encripted and/or compressed.
Versus, having to browser your file and look for a specific "encripted" or "compressed" flag, that the file explorer shows if they play nice...Now... if its at the file level, It can be troublesome to look for allfiles encripted/compressed.
Also, knowing the encription key to each file (if it's set at the file level) ... argh!!Doing this file based, seems to me that will lead to complication the long term maintenance of things.
In the long term, when someone picks up, or mounts a old, used system, I can't imagine the butload of work involved in finding out things like:
- what in this 4TB fs is compressed, what is encripted?
- (in case of redundancy at the file level): how many redundancy levels do I have, and which files respectively?
- why is rsync/scp 'ing this folder/BIG_NESTED_TREE_OF_FILES allways "stop" at folder/**/fileB and folder/z*/* ?
(might it be that some unsuspecting file in there has a trully beasty
compression ratio, and that 4GB fileB (in the middle of other 4GB files), is instead a 80Gb file compressed ?I also don't like extended attributes, we're just creating a whole new orthogonal dimension/namespace were we store/manipulate information.. it also complicates backups (does the backup system understand and respects ext. attrs?)
It's messier doing these things with extended attributes or very custom tools, instead of levelaging/creating a volume management tool, for managing this, such tool will allready be there to manage the fisical storage <-> filesystems/volumes mappings and settings.. might has well leverage it.
Kind regards,
Quoted 28 lines
Quoted 20 lines
Quoted 20 lines
So.. if a user wants a specific set of files or dirs ..they should create a mount-point/volume like:
private_volbigarchives_voland set those volumes as compressed or encripted volumes
Regarding usability, the best would be for the sub-volume creation tool to optionally allow passing encription/compression arguments.
and then:
should mount those volumes somewhere like: ~/Confidential or ~/Archives.Basically, do it at the directory level (which in btrfs is at the sub-volume level).
File-level granularity is totally unmanageable in the long term.Quoted 5 lines
Quoted 1 line
Kind regards,
--
Regards,
ChrisLee
Signature · 1 line
Miguel Sousa Filipe
El Tue, 9 Dec 2008 13:09:51 -0500, "Lee Trager" <[email protected]> es= cribi=C3=B3:
Quoted 1 line
It does seem that doing it with volumes would limit user control and =
add
Quoted 1 line
lots of complexity to such a simple task.
IMHO, WRT compression it's the contrary. Compression on a per-file basi= s has
never been very succesful (just look at how many windows users use it) because it implies taking a decision for every file on the system. OTOH= ,
volume-level is just a single option to be enabled.
I'm of course not arguing that file-level compression shouldn't be poss= ible,
im just saying that is way more difficult to administer and that most p= eople
(including sysadmins) is most likely to use compression in a per-volume=
basis.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 1 line · Diego Calleja
El Tue, 9 Dec 2008 13:09:51 -0500, "Lee Trager" <[email protected]> =
escribi=C3=B3:
Quoted 2 lines
=20
Quoted 1 line
It does seem that doing it with volumes would limit user control and=
add
Quoted 3 lines
Quoted 1 line
lots of complexity to such a simple task.
=20
IMHO, WRT compression it's the contrary. Compression on a per-file ba=
sis has
Quoted 1 line
never been very succesful (just look at how many windows users use it=
)
Quoted 1 line
because it implies taking a decision for every file on the system. OT=
OH,
Quoted 3 lines
volume-level is just a single option to be enabled.
=20
I'm of course not arguing that file-level compression shouldn't be po=
ssible,
Quoted 1 line
im just saying that is way more difficult to administer and that most=
people
Quoted 1 line
(including sysadmins) is most likely to use compression in a per-volu=
me basis.
While I have not gotten far enough to prove it is feasible...
My idea on controlling features like compression is that the default mode is inherited from the parent in the directory tree. Thus you can turn it on/off at whatever granularity you want.
jim
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines
While I have not gotten far enough to prove it is feasible...
My idea on controlling features like compression is that the default mode is inherited from the parent in the directory tree. Thus you can turn it on/off at whatever granularity you want.
That seems like the ideal solution to me. Similar to how default ACLs work.
-John
Quoted 5 lines · jim owens
Quoted 1 line · Diego Calleja
El Tue, 9 Dec 2008 13:09:51 -0500, "Lee Trager" <[email protected]=
escribi=C3=B3:
Quoted 2 lines
=20
Quoted 1 line
It does seem that doing it with volumes would limit user control a=
nd add
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
lots of complexity to such a simple task.
=20
IMHO, WRT compression it's the contrary. Compression on a per-file =
basis has
Quoted 1 line
Quoted 1 line
never been very succesful (just look at how many windows users use =
it)
Quoted 1 line
Quoted 1 line
because it implies taking a decision for every file on the system. =
OTOH,
Quoted 3 lines
Quoted 3 lines
volume-level is just a single option to be enabled.
=20
I'm of course not arguing that file-level compression shouldn't be =
possible,
Quoted 1 line
Quoted 1 line
im just saying that is way more difficult to administer and that mo=
st people
Quoted 1 line
Quoted 1 line
(including sysadmins) is most likely to use compression in a per-vo=
lume basis.
Quoted 8 lines
=20
While I have not gotten far enough to prove it is feasible... =20
My idea on controlling features like compression is that the default mode is inherited from the parent in the directory tree. Thus you can turn it on/off at whatever granularity you want.
=20
I had planned to make the bits inheritable from the directory inode flags. There are two different discussions around xattrs for this. On= e
is using xattrs to store the flag, which I'd would rather avoid because it is checked in some performance critical places.
The second is using xattr programs to set the flag, which I don't reall= y
have an opinion on. The idea of having the flags backed up by backup programs or rsync is really nice, but do any of the backup programs actually copy out all the xattrs?
-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · Chris Mason
I had planned to make the bits inheritable from the directory inode flags. There are two different discussions around xattrs for this. One is using xattrs to store the flag, which I'd would rather avoid because it is checked in some performance critical places.
The second is using xattr programs to set the flag, which I don't really have an opinion on. The idea of having the flags backed up by backup programs or rsync is really nice, but do any of the backup programs actually copy out all the xattrs?
xfsdump does :) But I think especially the compressed bit is much better off in the bit for the set/get flags ioctls used by chattr / lsattr. These are implemented by all Linux filesystems, and even have a compressed bit allocated already. (and xfsdump of course backs them up, too ;-)) *run*
Quoted 11 lines · Christoph Hellwig
Quoted 9 lines · Chris Mason
I had planned to make the bits inheritable from the directory inode flags. There are two different discussions around xattrs for this. One is using xattrs to store the flag, which I'd would rather avoid because it is checked in some performance critical places.
The second is using xattr programs to set the flag, which I don't really have an opinion on. The idea of having the flags backed up by backup programs or rsync is really nice, but do any of the backup programs actually copy out all the xattrs?
Quoted 5 lines
xfsdump does :) But I think especially the compressed bit is much better off in the bit for the set/get flags ioctls used by chattr / lsattr. These are implemented by all Linux filesystems, and even have a compressed bit allocated already. (and xfsdump of course backs them up, too ;-)) *run*
The 'star' command has an -xattr option.
-- ljk
On Wednesday 10 December 2008 05:44:47 Chris Mason wrote:
[snip]
Quoted 4 lines
The second is using xattr programs to set the flag, which I don't really have an opinion on. The idea of having the flags backed up by backup programs or rsync is really nice, but do any of the backup programs actually copy out all the xattrs?
rsync has the -X option. rdiff-backup autodetects whether xattrs are supported.
Those are the programs I use; dunno about others.
-- Josh
Signature · 4 lines
Joshua J. Berry
"I haven't lost my mind -- it's backed up on tape somewhere."
-- /usr/games/fortuneSGF2aW5nIGEgMyBwb3NzaWJsZSBzdGF0ZXMgZm9yIGVhY2ggZmlsZSB3b3VsZCBzZWVtIHNlbnNp YmxlOg0KDQoxLiAgQ29tcHJlc3Npb24gRW5hYmxlZCAtIHRoaXMgZmlsZSBvciBmb2xkZXIgd2ls bCBiZSBjb21wcmVzc2VkLg0KMi4gIENvbXByZXNzaW9uIERpc2FibGVkIC0gdGhpcyBmaWxlIG9y IGZvbGRlciB3aWxsIG5ldmVyIGJlIGNvbXByZXNzZWQuDQozLiAgTm90IFNwZWNpZmllZCAtIFRo aXMgd2lsbCBpbmhlcml0IHRoZSBjb21wcmVzc2lvbiBzdGF0ZSBmcm9tIGl0J3MgcGFyZW50Lg0K DQpUbyBrZWVwIHRoaXMgc3lzdGVtIGluIGEgY29uc2lzdGVudCBzdGF0ZSwgaWYgYSBmb2xkZXIg aXMgY2hhbmdlZCBmcm9tIHNheSBzdGF0ZSAyIHRvIHN0YXRlIDEsIHRoZW4gYWxsIHRoZSBjaGls ZHJlbiBvZiB0aGF0IGZvbGRlciB3aG8gYXJlIGluIHN0YXRlIDMgbmVlZCB0byBiZSBjb21wcmVz c2VkIC0gcG90ZW50aWFsbHkgYSB2ZXJ5IHRpbWUgY29uc3VtaW5nIG9wZXJhdGlvbi4NCg0KSWYg dGhpcyBvcGVyYXRpb24gaXMgY2FuY2VsbGVkLCBvciBmYWlscywgdGhlbiB0aGUgY29tcHJlc3Np b24gc3RhdGVzIGluIHRoZSBmaWxlc3lzdGVtIHdpbGwgYmUgaW5jb25zaXN0ZW50LiAgTm8gaWRl YSBob3cgaXQncyBiZXN0IHRvIHNvbHZlIHRoYXQgLSBlaXRoZXIganVzdCBjb3BlIHdpdGggdGhl IGluY29uc2lzdGVuY3kgYW5kIGxldCBhIGZpbGVzeXN0ZW0gY2hlY2sgdGlkeSBpdCB1cCAocmVt ZW1iZXIgdGhpcyBpc3N1ZSB3b3VsZCBuZXZlciBlbmRhbmdlciBkYXRhLCBhbmQgb25seSBvY2N1 cnMgaW4gdGhlIGNhc2Ugb2YgYSBzeXN0ZW0gZmFpbHVyZSB3aGlsZSBjaGFuZ2luZyB0aGUgY29t cHJlc3Npb24gZmxhZ3Mgb24gYSBkaXJlY3RvcnkpLiAgQW4gYWx0ZXJuYXRpdmUgc29sdXRpb24g Y291bGQgYmUgdG8gbWFrZSB0aGUgYXR0cmlidXRlIGNoYW5nZSBhc2NocnluY2hyb25vdXMgYW5k IGhhdmUgYW4gIm9wZXJhdGlvbiBxdWV1ZSIsIHNvIGluIHRoZSBjYXNlIG9mIGEgc3lzdGVtIGZh aWx1cmUgdGhlIG9wZXJhdGlvbiBjYW4gY29udGludWUgd2hlcmUgaXQgbGVmdCBvZmYuDQo=
Quoted 6 lines · Miguel Figueiredo Mascarenhas Sousa Filipe
Things like compression or encription should be used at the "volume" level. So.. if a user wants a specific set of files or dirs ..they should create a mount-point/volume like:
private_volbigarchives_vol
Does a normal user have permissions to do this?
I could easily see the case where a normal user has a set of very compressible files in a directory. They shouldn't have to get a sysadmin go to the trouble of setting this up.
Jeremy
Quoted 6 lines · Jeremy Sanders
Quoted 4 lines · Miguel Figueiredo Mascarenhas Sousa Filipe
Things like compression or encription should be used at the "volume" level. So.. if a user wants a specific set of files or dirs ..they should create a mount-point/volume like:
Quoted 9 lines
Quoted 2 lines
private_volbigarchives_volDoes a normal user have permissions to do this?
I could easily see the case where a normal user has a set of very compressible files in a directory. They shouldn't have to get a sysadmin go to the trouble of setting this up.
that's a policy decision.
Distro's will probably allow a admin/wheel user to do such things.
Big deployments, shell servers, or university setups, will all have a policy decision and usualy hack up costumised tools for managing users quota, settings, passwds and things like that.
A costumised useradd would probably create such volumes for each user, for instance, and a costumised passwd, would reencript stored data with new passwd. (or something similar...)...
I know I did similar things when was a university sysadm, pam_useradd and pam_mountd were heavily used and costumised.
Kind regards,
Signature · 1 line
Miguel Sousa Filipe
Quoted 7 lines · Lee Trager
Currently compression and I assume if encryption is implemented it is turned on or off during mount. There are however many times when a user may want to select which files/directories they want to compress or encrypt. This will also be helpful when implementing btrfs support in grub for example. We can say the disk can be compressed/encrypted except for /boot so compression/encryption doesn't have to be implemented in grub.
Quoted 6 lines
I was thinking of adding this functionality to the userspace application btrfstune. The way I was thinking of doing this is when btrfstune +c is applied to a directory or file the directory(and all its contents) or file will always be compressed reguardless of how the filesystem is mounted. The opposite would happen when btrfstune -c is used.
This was my plan, but btrfstune probably isn't the best program to do it (the ext2 tune program is mostly aimed at the super block level things).
I think it would be better to make a setattr style program to call the ioctls. There is already a per file compression flag, and the code should already be checking it.
-chris
Would you suggest using the existing chattr/lsattr commands from e2fsprogs for userspace control and just add support at the kernel level?
Miguel suggested this be done at the volume level. Do you have any thoughts on that?
Thanks,
Lee
Quoted 24 lines · Chris Mason
Quoted 7 lines · Lee Trager
Currently compression and I assume if encryption is implemented it is turned on or off during mount. There are however many times when a user may want to select which files/directories they want to compress or encrypt. This will also be helpful when implementing btrfs support in grub for example. We can say the disk can be compressed/encrypted except for /boot so compression/encryption doesn't have to be implemented in grub.
Quoted 6 lines
I was thinking of adding this functionality to the userspace application btrfstune. The way I was thinking of doing this is when btrfstune +c is applied to a directory or file the directory(and all its contents) or file will always be compressed reguardless of how the filesystem is mounted. The opposite would happen when btrfstune -c is used.
This was my plan, but btrfstune probably isn't the best program to do it (the ext2 tune program is mostly aimed at the super block level things).
I think it would be better to make a setattr style program to call the ioctls. There is already a per file compression flag, and the code should already be checking it.
-chris
Quoted 15 lines · On Tuesday 09 December 2008 08:35:16 Chris Mason wrote:
Quoted 8 lines · Lee Trager
Currently compression and I assume if encryption is implemented it is turned on or off during mount. There are however many times when a user may want to select which files/directories they want to compress or encrypt. This will also be helpful when implementing btrfs support in grub for example. We can say the disk can be compressed/encrypted except for /boot so compression/encryption doesn't have to be implemented in grub.
Quoted 5 lines
I was thinking of adding this functionality to the userspace application btrfstune. The way I was thinking of doing this is when btrfstune +c is applied to a directory or file the directory(and all its contents) or file will always be compressed reguardless of how the filesystem is mounted. The opposite would happen when btrfstune -c is used.
Quoted 6 lines
This was my plan, but btrfstune probably isn't the best program to do it (the ext2 tune program is mostly aimed at the super block level things).
I think it would be better to make a setattr style program to call the ioctls. There is already a per file compression flag, and the code should already be checking it.
Is there some reason this can't be done with the existing extended attribute facilities?
It seems like xattrs would be preferable to some btrfs-specific tunable, as programs like rsync or backup tools would be able to preserve (and restore) these bits with no extra work required.
-- Josh
Signature · 4 lines
Joshua J. Berry
"I haven't lost my mind -- it's backed up on tape somewhere."
-- /usr/games/fortuneQuoted 18 lines · Joshua J. Berry
Quoted 17 lines · On Tuesday 09 December 2008 08:35:16 Chris Mason wrote:
Quoted 8 lines · Lee Trager
Currently compression and I assume if encryption is implemented it is turned on or off during mount. There are however many times when a user may want to select which files/directories they want to compress or encrypt. This will also be helpful when implementing btrfs support in grub for example. We can say the disk can be compressed/encrypted except for /boot so compression/encryption doesn't have to be implemented in grub.
Quoted 5 lines
I was thinking of adding this functionality to the userspace application btrfstune. The way I was thinking of doing this is when btrfstune +c is applied to a directory or file the directory(and all its contents) or file will always be compressed reguardless of how the filesystem is mounted. The opposite would happen when btrfstune -c is used.
This was my plan, but btrfstune probably isn't the best program to do it (the ext2 tune program is mostly aimed at the super block level things).
Quoted 7 lines
Quoted 3 lines
I think it would be better to make a setattr style program to call the ioctls. There is already a per file compression flag, and the code should already be checking it.
Is there some reason this can't be done with the existing extended attribute facilities?
It seems like xattrs would be preferable to some btrfs-specific tunable, as programs like rsync or backup tools would be able to preserve (and restore) these bits with no extra work required.
I had the same thought... that many btrfs per-file tunables would be better implemented in xattrs (I've read christoph's earlier objections and hope to get around those issues).
I have been working on changing the xattr code with the first step getting it functioning properly when selinux is enabled so we can see just how costly btrfs xattrs are in actual use.
jim
Quoted 1 line · jim owens
Joshua J. Berry wrote:
[ storing compression flags ]
Quoted 5 lines
Quoted 4 lines
It seems like xattrs would be preferable to some btrfs-specific tunable, as programs like rsync or backup tools would be able to preserve (and restore) these bits with no extra work required.
Quoted 7 lines
I had the same thought... that many btrfs per-file tunables would be better implemented in xattrs (I've read christoph's earlier objections and hope to get around those issues).
I have been working on changing the xattr code with the first step getting it functioning properly when selinux is enabled so we can see just how costly btrfs xattrs are in actual use.
Not really on topic, but how are things broken today with selinux?
-chris
btrfs with selinux
Quoted 6 lines · Chris Mason
Quoted 3 lines · jim owens
I have been working on changing the xattr code with the first step getting it functioning properly when selinux is enabled so we can see just how costly btrfs xattrs are in actual use.
Not really on topic, but how are things broken today with selinux?
With selinux enabled you can not create any files on a btrfs filesystem (as of dec9 git tree with fedora 9), even as root!
There are 2 things needed to make it work:
1) the /etc/selinux load-into-kernel database must be patched to recognize btrfs has xattrs. One of our security people, Paul Moore, has submitted it to the upstream refpolicy. But it won't be merged until I finish my testing.
After the database is patched, the dec9 git tree will allow file create on btrfs... but the selinux xattrs are not set. Thus "cp -a" will copy the files but all "selinux context" values are wrong.
2) I have btrfs patches to interface correctly with the LSM so we save the selinux context. I'll be sending them up as soon as I have finished testing.
jim
P.S. sane people just disable selinux on install :)
Quoted 26 lines · jim owens
Quoted 25 lines · Joshua J. Berry
Quoted 14 lines · On Tuesday 09 December 2008 08:35:16 Chris Mason wrote:
Quoted 8 lines · Lee Trager
Currently compression and I assume if encryption is implemented it is turned on or off during mount. There are however many times when a user may want to select which files/directories they want to compress or encrypt. This will also be helpful when implementing btrfs support in grub for example. We can say the disk can be compressed/encrypted except for /boot so compression/encryption doesn't have to be implemented in grub.
Quoted 5 lines
I was thinking of adding this functionality to the userspace application btrfstune. The way I was thinking of doing this is when btrfstune +c is applied to a directory or file the directory(and all its contents) or file will always be compressed reguardless of how the filesystem is mounted. The opposite would happen when btrfstune -c is used.
Quoted 6 lines
This was my plan, but btrfstune probably isn't the best program to do it (the ext2 tune program is mostly aimed at the super block level things).
I think it would be better to make a setattr style program to call the ioctls. There is already a per file compression flag, and the code should already be checking it.
Is there some reason this can't be done with the existing extended attribute facilities?
Quoted 7 lines
Quoted 3 lines
It seems like xattrs would be preferable to some btrfs-specific tunable, as programs like rsync or backup tools would be able to preserve (and restore) these bits with no extra work required.
I had the same thought... that many btrfs per-file tunables would be better implemented in xattrs (I've read christoph's earlier objections and hope to get around those issues).
I agree that implementing it in the xattr would be best but what userspace tools should be used to control it? lsattr/chattr or our own? lsattr/chattr does seem to do what we want for compression but it really has nothing for encryption. Should we also support attribute(a), no modification(i), etc support in xattrs?
Quoted 4 lines
I have been working on changing the xattr code with the first step getting it functioning properly when selinux is enabled so we can see just how costly btrfs xattrs are in actual use.
Could you send me to xattr code you are working on or will you be submitting patches soon?
Thanks,
Lee
Quoted 2 lines
jim
Quoted 31 lines · Lee Trager
Quoted 30 lines · jim owens
Quoted 28 lines · Joshua J. Berry
Quoted 14 lines · On Tuesday 09 December 2008 08:35:16 Chris Mason wrote:
Quoted 8 lines · Lee Trager
Currently compression and I assume if encryption is implemented it is turned on or off during mount. There are however many times when a user may want to select which files/directories they want to compress or encrypt. This will also be helpful when implementing btrfs support in grub for example. We can say the disk can be compressed/encrypted except for /boot so compression/encryption doesn't have to be implemented in grub.
Quoted 5 lines
I was thinking of adding this functionality to the userspace application btrfstune. The way I was thinking of doing this is when btrfstune +c is applied to a directory or file the directory(and all its contents) or file will always be compressed reguardless of how the filesystem is mounted. The opposite would happen when btrfstune -c is used.
Quoted 6 lines
This was my plan, but btrfstune probably isn't the best program to do it (the ext2 tune program is mostly aimed at the super block level things).
I think it would be better to make a setattr style program to call the ioctls. There is already a per file compression flag, and the code should already be checking it.
Is there some reason this can't be done with the existing extended attribute facilities?
It seems like xattrs would be preferable to some btrfs-specific tunable, as programs like rsync or backup tools would be able to preserve (and restore) these bits with no extra work required.
Quoted 8 lines
Quoted 3 lines
I had the same thought... that many btrfs per-file tunables would be better implemented in xattrs (I've read christoph's earlier objections and hope to get around those issues).
I agree that implementing it in the xattr would be best but what userspace tools should be used to control it? lsattr/chattr or our own? lsattr/chattr does seem to do what we want for compression but it really has nothing for encryption. Should we also support attribute(a), no modification(i), etc support in xattrs?
Just to clarify, I'm fine with xattrs as a user interface to set the per-inode flags. The flag for this should be stored in the inode though, and not as a btrfs xattr. There is already a flag field in the inode for this use (see NODATASUM and NODATACOW)
-chris