Skip to contentSkip to conversation

writing a plugin for reiserfs compression

DISCUSSION Roy Sigurd Karlsbakk 5 participants 8 messages Nov 1, 2001 – Nov 9, 2001

lore
Share & tools
lore
#0 lore raw

Andreas Dilger ([email protected]) wrote*:

Quoted 6 lines · Roy Sigurd Karlsbakk
Quoted 2 lines

Novell NetWare has a feature I really like. It's a file compression feature they've been having since version 4.0 (or 4.10) of the OS.

Yes, there is a patch for ext2 that does this as well.

ok...

I just thought there was a patch doing windows nt-like compress-em-all-realtime-and-get-doomed!

Quoted 7 lines
Quoted 7 lines

New attributes must be added somehow. 'ls' and 'find' and perhaps other files must be modified to take advantage of this. The compression job can be a simple script with something like

find . -type f ! --compressed ! --dont-compress / -exec fcomp {} \;

(and check can't compress and force compression).

There already exists a patch for reiserfs which uses the same interface to file attributes that ext2 and ext3 use.

ok? with batched compression?

Quoted 9 lines

Also, ext2 already has a "compressed", "do not compress", and "dirty" attributes. They are currently not all user modifyable for ext2 filesystems via chattr/lsattr, but that doesn't mean they cannot be on reiserfs.

Quoted 3 lines

There must be a way to access the compressed files directly to make backups more efficient - backing up already compressed files's a good thing.

Quoted 5 lines

Yes, there is also such an attribute for "raw" access I think.

Making the user-space interface and tools as compatible as possible is a good thing, IMHO, just like "ls", "cp", etc all work regardless of the underlying filesystem.

yes, but it's kinda nice to have some way of checking a file's attributes for a sysadmin...

Quoted 5 lines

As a note to whoever at namesys created the reiserfs patch to add the "notail" flag (overloading the "nodump" flag). I would much rather that a new "notail" flag be allocated for this. I will contact Ted Ted Ts'o to get a flag assigned. This will avoid any problems in the future, and may also be useful at some time for ext2.

Do you think the other flags I mentioned may be useful?

roy

#1 lore raw
Quoted 5 lines · Roy Sigurd Karlsbakk

Andreas Dilger ([email protected]) wrote*:

Quoted 1 line

Yes, there is a patch for ext2 that does this as well.

I just thought there was a patch doing windows nt-like compress-em-all-realtime-and-get-doomed!

I don't know what the actual heuristics for determining which files are compression with the ext2 patch. It is definitely NOT a compressed block device. Files are compressed in chunks (32kB?), so that it is possible to seek and do read-modify-write (e.g. appending to a file) without decompressing the entire file and/or recompressing it. This also protects against block corruption, since you would limit the amount of data lost to the end of the chunk after the bad spot.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

New attributes must be added somehow. 'ls' and 'find' and perhaps other files must be modified to take advantage of this. The compression job can be a simple script with something like

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

find . -type f ! --compressed ! --dont-compress / -exec fcomp {} \;

(and check can't compress and force compression).

There already exists a patch for reiserfs which uses the same interface to file attributes that ext2 and ext3 use.

ok? with batched compression?

No compression is there for either ext2 (without the patch), and not at all for reiserfs (AFAIK). All the patch does is give reiserfs the ABILITY to store per-inode attributes in a way compatible with the existing ext2 attributes. Since most people already have e2fsprogs installed (fsck, lsattr, chattr live there) it makes sense to use the same user-space tools to do the same thing, and even the same ioctl numbers/flag values.

Quoted 7 lines
Quoted 4 lines

Also, ext2 already has a "compressed", "do not compress", and "dirty" attributes. They are currently not all user modifyable for ext2 filesystems via chattr/lsattr, but that doesn't mean they cannot be on reiserfs.

yes, but it's kinda nice to have some way of checking a file's attributes for a sysadmin...

That's what "lsattr" is for. All I was getting at (when mentioning "ls" and "cp") is that where possible the user tools should be compatible, even if the underlying filesystem is different. I would rather avoid the case (currently being worked on) with ext2 and XFS ACL user tools being different, using different ioctls, but doing 99% of the same function. It is ugly to think "reiserattr" and "xfsattr" and "chattr/lsattr (ext2)", and "getacl/setacl", "xfsacl" commands instead of a single set of commands (and even kernel API) that hide the details from the user.

Quoted 1 line

Do you think the other flags I mentioned may be useful?

Yes, definitely disabling compression for a file is good. The "accessed in last 7 days flag" is questionable. This could be determined via the atime on Unix and doesn't need a separate flag. Also, the difference between "do not compress" and "can't compress" is very small. If it is found that the file is incompressible, you could just as easily set the "do not compress" flag.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

#2 lore raw
Quoted 10 lines
Quoted 2 lines

I just thought there was a patch doing windows nt-like compress-em-all-realtime-and-get-doomed!

I don't know what the actual heuristics for determining which files are compression with the ext2 patch. It is definitely NOT a compressed block device. Files are compressed in chunks (32kB?), so that it is possible to seek and do read-modify-write (e.g. appending to a file) without decompressing the entire file and/or recompressing it. This also protects against block corruption, since you would limit the amount of data lost to the end of the chunk after the bad spot.

But still... Are the files are compressed as they are created/modified on the filesystem? My main point was to avoid the compression overhead and just decompress the file at access time if it's compressed. Compression should (IMO) be done nightly.

Perhaps a file should be decompressed when it's modified and either (a) set scheduled to next nightly compression or (b) stay uncompressed the next <n> days. I mean - as a file is being modified, the chance is large that the file will be accessed pretty soon...

Quoted 6 lines

Yes, definitely disabling compression for a file is good. The "accessed in last 7 days flag" is questionable. This could be determined via the atime on Unix and doesn't need a separate flag. Also, the difference between "do not compress" and "can't compress" is very small. If it is found that the file is incompressible, you could just as easily set the "do not compress" flag.

I agree on the 'accessed in last <n> days'. It'd be better to check atime.

I'd still like to separate 'do not compress' and 'can't compress', as to show why the falg has been set - the former is set by the admin and the latter by the system.

roy

#3 lore raw
Quoted 10 lines · Roy Sigurd Karlsbakk
Quoted 10 lines
Quoted 3 lines

I just thought there was a patch doing windows nt-like compress-em-all-realtime-and-get-doomed!

I don't know what the actual heuristics for determining which files are compression with the ext2 patch. It is definitely NOT a compressed block device. Files are compressed in chunks (32kB?), so that it is possible to seek and do read-modify-write (e.g. appending to a file) without decompressing the entire file and/or recompressing it. This also protects against block corruption, since you would limit the amount of data lost to the end of the chunk after the bad spot.

It's a compilation option to never compress gz files. I guess it would be easy to add others (bz2, Z, jpg, mp3, ...) to the list?. The max chunk used for compression is 32KB and again this is configurable, as is the compression method/level used. The alogorithms are gzip (1-9), LZO, bz2.

Quoted 4 lines
Quoted 1 line

But still... Are the files are compressed as they are created/modified on the filesystem?

Only if the file is in a directory with the +c attribute set. You can have full control over the compression.

Note this transparent ext2 compression patch is only available for 2.2

Padraig.

#4 lore raw
Quoted 6 lines
Quoted 3 lines

But still... Are the files are compressed as they are created/modified on the filesystem?

Only if the file is in a directory with the +c attribute set. You can have full control over the compression.

Could this be modified to the 'wait <n> days' concept I mentioned earlier? I mean... Don't let the kernel modify them - ever. Let some cron-job do it...

Quoted 1 line

Note this transparent ext2 compression patch is only available for 2.2

Would it be hard to port to 2.4?

#5 lore raw
In reply to Padraig Brady
Quoted 3 lines · Roy Sigurd Karlsbakk
Quoted 1 line

Note this transparent ext2 compression patch is only available for 2.2

Would it be hard to port to 2.4?

AFAIK kinda yes. It relies on IO going through block buffer cache:
Buffer cache contains the compressed data, while the page cache has it all decompressed. This avoids excessive (de)compression (you need to compress data only when commiting page to disk).

With 2.4 Ext2 moved (almost?) entirely out of buffer cache, you'll need to create your own IO buffers. This is the only solution I came up with. Is there some other approach how to cope with this problem ?

Signature · 4 lines
Kind regards,
Robert Varga
------------------------------------------------------------------------------
[email protected]                                          http://hq.sk/~nite/gpgkey.txt
#6 lore raw
In reply to Roy Sigurd Karlsbakk
Quoted 29 lines · Roy Sigurd Karlsbakk
Quoted 11 lines
Quoted 3 lines

I just thought there was a patch doing windows nt-like compress-em-all-realtime-and-get-doomed!

I don't know what the actual heuristics for determining which files are compression with the ext2 patch. It is definitely NOT a compressed block device. Files are compressed in chunks (32kB?), so that it is possible to seek and do read-modify-write (e.g. appending to a file) without decompressing the entire file and/or recompressing it. This also protects against block corruption, since you would limit the amount of data lost to the end of the chunk after the bad spot.

But still... Are the files are compressed as they are created/modified on the filesystem? My main point was to avoid the compression overhead and just decompress the file at access time if it's compressed. Compression should (IMO) be done nightly.

Perhaps a file should be decompressed when it's modified and either (a) set scheduled to next nightly compression or (b) stay uncompressed the next <n> days. I mean - as a file is being modified, the chance is large that the file will be accessed pretty soon...

Quoted 6 lines

Yes, definitely disabling compression for a file is good. The "accessed in last 7 days flag" is questionable. This could be determined via the atime on Unix and doesn't need a separate flag. Also, the difference between "do not compress" and "can't compress" is very small. If it is found that the file is incompressible, you could just as easily set the "do not compress" flag.

Quoted 7 lines

I agree on the 'accessed in last <n> days'. It'd be better to check atime.

I'd still like to separate 'do not compress' and 'can't compress', as to show why the falg has been set - the former is set by the admin and the latter by the system.

roy

I think you'll find it easiest to code it such that doing anything to the file body, reading or writing, decompresses it. Fancier stuff can wait for later versions.

Hans

#7 lore raw
In reply to Roy Sigurd Karlsbakk
Quoted 4 lines · Nikita
Quoted 2 lines · Andreas Dilger

As a note to whoever at namesys created the reiserfs patch to add the "notail" flag (overloading the "nodump" flag). I would much rather

Quoted 11 lines

It was me. Agree completely that allocating new flag would be better. I just wanted "notail" to actually work and be accessible through standard utilities, because it's really useful. "nodump" looked like least useful of flags for me, because dump(8) doesn't work with reiserfs (not that it worked with ext2 reliably either). I actually tried to contact Remy Card and Theodore Tso, to discuss how [ls|ch]attr can be modified to support different file-systems, but to no avail.

Quoted 3 lines

that a new "notail" flag be allocated for this. I will contact Ted Ted Ts'o to get a flag assigned. This will avoid any problems in the future, and may also be useful at some time for ext2.

OK, FYI Nikita, Ted has allocated a EXT2_NOTAIL_FL flag for chattr/lsattr (value 0x00008000) which can be used for setting files/directories to be permanently notail. It is obviously up to the reiserfs code to handle this flag and inherit it for files created in a directory (e.g. /boot), but starting with e2fsprogs 1.26 chattr/lsattr it will be able to set/get this flag on ext2/ext3 (and reiserfs with your attributes patch).

Quoted 5 lines

I would rather like to see lsattr/chattr to become file-system independent. This requires that all file-systems use the same ioctl cmds to set and get bitmasks associated with inodes and provide somehow a mapping between symbolic name of an attribute and bitmask. Support for octal bitmask (a la chmod) in chattr is also an option.

There is nothing really ext2-specific to the chattr/lsattr programs. Yes, they use an ioctl and flag values assigned to ext2, but as you have shown it is also possible to use this ioctl on reiserfs without any problems. These commands are for simple file attributes only. If reiserfs has a need for specific attributes, then Ted can probably allocate a fs-specific range. If you want to store the values in a different format, you can always map in the ioctl, although I don't see a real need for that right now.

For more complex extended attributes, there are [gs]etextattr and [gs]etacl commands (I think) which the ext2 EA/ACL code uses, but again this is not ext2 specific. The author (Andreas Gruenbacher) is working with the XFS folks to support a common kernel API and allow the same user-space tools to work, even if the fs-internal and on-disk EA/ACL formats are different. However, I know for extended attributes that Hans has other plans (reiser4/sandbox syscall) so I don't know if this will be useful to you. Maybe still yes, if the same user programs can interface with the reiserfs syscall, and it may still be useful for ACL support.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/