ANNOUNCEMENT: Squashfs released (a highly compressed filesystem)
Hi,
First release of squashfs. Squashfs is a highly compressed read-only filesystem for Linux (kernel 2.4.x). It uses zlib compression to compress both files, inodes and directories. Inodes in the system are very small and all blocks are packed to minimise data overhead. Block sizes greater than 4K are supported up to a maximum of 32K.
Squashfs is intended for general read-only filesystem use, for archival use, and in embedded systems where low overhead is needed.
Squashfs is available from http://squashfs.sourceforge.net.
The patch file is currently against 2.4.19. There is further info on the filesystem design etc. in the README.
I'l be interested in getting any feedback, advice etc. on it.
Phill Lougher
Quoted 4 lines · Phillip Lougher
Hi,
First release of squashfs. Squashfs is a highly compressed read-only filesystem for Linux (kernel 2.4.x). It uses zlib compression to compress both files, inodes and directories. Inodes in the system are very small and all blocks are packed to minimise data overhead. Block sizes greater than 4K are supported up to a maximum of 32K.
Quoted 8 lines
Squashfs is intended for general read-only filesystem use, for archival use, and in embedded systems where low overhead is needed.
Squashfs is available from http://squashfs.sourceforge.net.
The patch file is currently against 2.4.19. There is further info on the filesystem design etc. in the README.
I'l be interested in getting any feedback, advice etc. on it.
What are the advantages of squashfs vs cramfs?
Quoted 8 lines · Samuel Flory
Quoted 6 lines · Phillip Lougher
Hi,
First release of squashfs. Squashfs is a highly compressed read-only filesystem for Linux (kernel 2.4.x). It uses zlib compression to compress both files, inodes and directories. Inodes in the system are very small and all blocks are packed to minimise data overhead. Block sizes greater than 4K are supported up to a maximum of 32K.
Squashfs is intended for general read-only filesystem use, for archival use, and in embedded systems where low overhead is needed.
Quoted 11 lines
Quoted 6 lines
Squashfs is available from http://squashfs.sourceforge.net.
The patch file is currently against 2.4.19. There is further info on the filesystem design etc. in the README.
I'l be interested in getting any feedback, advice etc. on it.
What are the advantages of squashfs vs cramfs?
Cramfs was the inspiration for squashfs. Squashfs basically gives better compression, bigger files/filesystem support, and more inode information.
1. Blocks upto 32K are supported - data is compressed in units of 32K which achieves better compression ratios than compressing in 4K blocks. Generally using bigger than 4K blocks are a bad idea, because the VFS
calls the filesystem in 4K pages. Squashfs explictly pushes the extra block data into the page cache.
2. Squashfs compresses inode and directory information in addition to file data. Inodes/directories generally compress down to 50%, or say on average 8 bytes or less per inode.
3. All fs data is packed on byte alignments, saving a couple of bytes per inode and directory.
4. Full 32 bit uids/guids are stored (4 bits stored in inode, uses a lookup table, to give 48 uids/16 gids). File sizes upto 2^32 are supported. Timestamp info is stored. Cramfs truncates uids to 16 bits, uids to 8 bits. Cramfs files sizes are upto 2^24. No timestamp info. Squashfs takes advantage of metadata compression to have more info with smaller metadata overhead.
5 Symbolic link contents/file indexes are stored inside the inode table, giving better compression than if they were compressed individually, or not compressed.
6. The mksquashfs program doesn't store/mmap the filesystem as it is created (it performs file duplicate checking against the partially written out compressed fs), and so allows larger filesystems to be created.
Further info on the fs is contained in the README...
Phillip
Quoted 17 lines · Phillip Lougher
Quoted 15 lines · Samuel Flory
Quoted 12 lines · Phillip Lougher
Hi,
First release of squashfs. Squashfs is a highly compressed read-only filesystem for Linux (kernel 2.4.x). It uses zlib compression to compress both files, inodes and directories. Inodes in the system are very small and all blocks are packed to minimise data overhead. Block sizes greater than 4K are supported up to a maximum of 32K.
Squashfs is intended for general read-only filesystem use, for archival use, and in embedded systems where low overhead is needed.
Squashfs is available from http://squashfs.sourceforge.net.
The patch file is currently against 2.4.19. There is further info on the filesystem design etc. in the README.
I'l be interested in getting any feedback, advice etc. on it.
Quoted 8 lines
Quoted 4 lines
What are the advantages of squashfs vs cramfs?
Cramfs was the inspiration for squashfs. Squashfs basically gives better compression, bigger files/filesystem support, and more inode information.
1. Blocks upto 32K are supported - data is compressed in units of 32K which achieves better compression ratios than compressing in 4K blocks. Generally using bigger than 4K blocks are a bad idea, because the VFS calls the filesystem in 4K pages. Squashfs explictly pushes the extra block data into the page cache.
I'm curious if you looked at ntfs-tng's code before implementing this. It's pretty darned optimal...
Quoted 1 line
2. Squashfs compresses inode and directory information in addition to file data. Inodes/directories generally compress down to 50%, or say on average 8 bytes or less per inode.
squashfs or mksquashfs?
A r/w compressed filesystem would be darned useful too :)
Jeff
Quoted 1 line
A r/w compressed filesystem would be darned useful too :)
mmap(2) is, err, hard. Not impossible, it means the file system has to support both compressed and uncompressed files, but it's interesting.
Signature · 2 lines
--- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
Quoted 6 lines · Larry McVoy
Quoted 3 lines
A r/w compressed filesystem would be darned useful too :)
mmap(2) is, err, hard.
The underlying filesystem format can make things easier on you... and given that compressing inevitably requires some amount of data copying, it's not terribly difficult. I wouldn't claim NTFS is anything close to well-designed, but supporting compression under Linux on NTFS is at least feasible and shouldn't require tons of thought. (I've looked at it when dicking around with NTFS-TNG)
Quoted 4 lines
Not impossible, it means the file system has to
support both compressed and uncompressed files, but it's interesting.
well, yeah... ;-)
Jeff
Quoted 4 lines · Larry McVoy
Quoted 1 line
A r/w compressed filesystem would be darned useful too :)
mmap(2) is, err, hard. Not impossible, it means the file system has to support both compressed and uncompressed files, but it's interesting.
You can also think of it as a step toward a hierarchical filesystem with the files:
1. uncompressed (with uncompressed inode)
2. compressed on line (real disk space allocated)
3. compressed nearline (only compressed inode on disk, with a reference to offline storage)
Obviously this is only for very large filesystems (we have one FS that is currently between 100-200 TB in size when you include the migrated storage).
Signature · 5 lines
------------------------------------------------------------------------- Jesse I Pollard, II Email: [email protected] Any opinions expressed are solely my own.
Quoted 14 lines · Jesse Pollard
Quoted 4 lines · Larry McVoy
Quoted 1 line
A r/w compressed filesystem would be darned useful too :)
mmap(2) is, err, hard. Not impossible, it means the file system has to support both compressed and uncompressed files, but it's interesting.
You can also think of it as a step toward a hierarchical filesystem with the files:
1. uncompressed (with uncompressed inode)
2. compressed on line (real disk space allocated)
3. compressed nearline (only compressed inode on disk, with a reference to offline storage)
Quoted 3 lines
Obviously this is only for very large filesystems (we have one FS that is currently between 100-200 TB in size when you include the migrated storage).
I think it's worth referencing e2compr here also, which is a patch that provides transparent compression for ext2 (& ext3?). Denis RICHARD (CC'd) is maintaining the 2.4 implementation. I've version 0.4.42 (against 2.4.16) mirrored here:
http://www.iol.ie/~padraiga/patches/e2compr-0.4.42-patch-2.4.16.gz
Note I've used it without problems (to access filesystems created with the kernel 2.2 version) here:
http://cvs.bofh.asn.au/e2compr/
Pádraig.
Quoted 23 lines · Padraig Brady
Quoted 16 lines · Jesse Pollard
Quoted 5 lines · Larry McVoy
Quoted 1 line
A r/w compressed filesystem would be darned useful too :)
mmap(2) is, err, hard. Not impossible, it means the file system has to support both compressed and uncompressed files, but it's interesting.
You can also think of it as a step toward a hierarchical filesystem with the files:
1. uncompressed (with uncompressed inode)
2. compressed on line (real disk space allocated)
3. compressed nearline (only compressed inode on disk, with a reference to offline storage)Quoted 3 lines
Obviously this is only for very large filesystems (we have one FS that is currently between 100-200 TB in size when you include the migrated storage).
Quoted 6 lines
I think it's worth referencing e2compr here also, which is a patch that provides transparent compression for ext2 (& ext3?). Denis RICHARD (CC'd) is maintaining the 2.4 implementation. I've version 0.4.42 (against 2.4.16) mirrored here:
http://www.iol.ie/~padraiga/patches/e2compr-0.4.42-patch-2.4.16.gz
A new version of the patch (0.4.43) is available here :
http://www.alizt.com
But I have not tested the patch on ext3.
Quoted 9 lines
Note I've used it without problems (to access filesystems created with the kernel 2.2 version) here:
http://cvs.bofh.asn.au/e2compr/Pádraig.
Signature · 4 lines
-----------------------------\--------------------------\ Denis RICHARD \ ALCATEL Business Systems \ mailto:[email protected] / Tel: +33(0)3 90 67 69 36 / -----------------------------/--------------------------/
Quoted 5 lines · Denis RICHARD
A new version of the patch (0.4.43) is available here :
http://www.alizt.com
But I have not tested the patch on ext3.
Any possibility for a forward port to 2.5?
Cheers,
Nicholas
Quoted 9 lines · Jeff Garzik
Quoted 3 lines · Phillip Lougher
2. Squashfs compresses inode and directory information in addition to file data. Inodes/directories generally compress down to 50%, or say on average 8 bytes or less per inode.
squashfs or mksquashfs?
mksquashfs...
Quoted 5 lines
A r/w compressed filesystem would be darned useful too :)
Jeff
A r/w compressed filesystem may be my next project... As a couple of people have mentioned there are compressed r/w filesystems already out there.
As you'll know, there are always tradeoffs with filesystem design, it is very difficult to get as good compression with a r/w fs than with a read only filesystem. I wanted to get maximum compression, and quite a few of the techniques I use rely on its read-only nature.
An append only (i.e. files can be added, but not modified), fs might be a useful compromise. With compressed metadata, any modification of files will inevitably achive different compression ratios, and so modification of metadata/files in place is not an option. Appending modified metadata/data brings you to log-structured (journalling) filesystems and compaction (log cleaning) requirements with consequent loss of compression.
Phillip
Quoted 4 lines
Quoted 18 lines · Phillip Lougher
Quoted 6 lines · Jeff Garzik
Quoted 3 lines · Phillip Lougher
2. Squashfs compresses inode and directory information in addition to file data. Inodes/directories generally compress down to 50%, or say on average 8 bytes or less per inode.
squashfs or mksquashfs?
mksquashfs...
Quoted 3 lines
A r/w compressed filesystem would be darned useful too :)
Jeff
A r/w compressed filesystem may be my next project... As a couple of people have mentioned there are compressed r/w filesystems already out there.
Quoted 5 lines
As you'll know, there are always tradeoffs with filesystem design, it is very difficult to get as good compression with a r/w fs than with a read only filesystem. I wanted to get maximum compression, and quite a few of the techniques I use rely on its read-only nature.
A compressed filesystem with dynamically updated random-access files will fragment the heck out of itself darn fast. (Seek into the middle of a file somewhere, write a block, seek somewhere else, write another block, repeat 1000 times... Keep in mind that the new compressed block of data will almost certainly not be the same size as the old one... It's a mess.)
My potential usage is: I've got a little linux distribution I put together called "firmware linux", which builds from source and outputs a zisofs image that gets loopback mounted as the root filesystem. (A very alpha version could be sucked off of http://216.143.22.141/firmware/fwl-0.8.tar, edit "build.sh" to specify the output directory, and then run it. The point is, the whole OS and all applications can be upgraded as one file. No package management, it's basically a big firmware image.)
The reason I used a zisofs instead of cramfs is that cramfs has a LOT of problems with big filesystems. (The finished root partition, with apache and samba and ntop and python and rsync and openssh and everything, is currently around 100 megs. Yeah, I can trim that down by quite a bit if I get time, I'm currently compilling and developing it under itself so I have gcc in there and the full set of man pages and everything...)
Mkcramfs seems to barf at somewhere around 16 megs, which is really limiting. AND it seems to try to open every file simultaneously (hardlink detection?) so it runs out of file handles. (Again, that could be adjusted under /proc somewhere, but isn't worth it.)
So it seems that the thing to test this against isn't cramfs, but zisofs. Have you looked at that?
I'll take a look myself and get back to you...
Rob
Signature · 2 lines
http://penguicon.sf.net - Terry Pratchett, Eric Raymond, Pete Abrams, Illiad, CmdrTaco, liquid nitrogen ice cream, and caffienated jello. Well why not?
Quoted 33 lines · Rob Landley
Quoted 18 lines · Phillip Lougher
Quoted 6 lines · Jeff Garzik
Quoted 3 lines · Phillip Lougher
2. Squashfs compresses inode and directory information in addition to file data. Inodes/directories generally compress down to 50%, or say on average 8 bytes or less per inode.
squashfs or mksquashfs?
mksquashfs...
Quoted 3 lines
A r/w compressed filesystem would be darned useful too :)
Jeff
A r/w compressed filesystem may be my next project... As a couple of people have mentioned there are compressed r/w filesystems already out there.
Quoted 5 lines
As you'll know, there are always tradeoffs with filesystem design, it is very difficult to get as good compression with a r/w fs than with a read only filesystem. I wanted to get maximum compression, and quite a few of the techniques I use rely on its read-only nature.
A compressed filesystem with dynamically updated random-access files will fragment the heck out of itself darn fast. (Seek into the middle of a file somewhere, write a block, seek somewhere else, write another block, repeat 1000 times... Keep in mind that the new compressed block of data will almost certainly not be the same size as the old one... It's a mess.)
My potential usage is: I've got a little linux distribution I put together called "firmware linux", which builds from source and outputs a zisofs image that gets loopback mounted as the root filesystem. (A very alpha version could be sucked off of http://216.143.22.141/firmware/fwl-0.8.tar, edit "
build.sh" to specify the output directory, and then run it. The point is, the whole OS and all applications can be upgraded as one file. No package management, it's basically a big firmware image.)The reason I used a zisofs instead of cramfs is that cramfs has a LOT of problems with big filesystems. (The finished root partition, with apache and samba and ntop and python and rsync and openssh and everything, is currently around 100 megs. Yeah, I can trim that down by quite a bit if I get time, I'm currently compilling and developing it under itself so I have gcc in there and the full set of man pages and everything...)
Quoted 8 lines
Mkcramfs seems to barf at somewhere around 16 megs, which is really limiting. AND it seems to try to open every file simultaneously (hardlink detection?) so it runs out of file handles. (Again, that could be adjusted under /proc somewhere, but isn't worth it.)
So it seems that the thing to test this against isn't cramfs, but zisofs. Have you looked at that?
I'll take a look myself and get back to you...
Rob
Hi,
I looked at both cramfs and ziosfs when writing squashfs. Zisofs is a nice fs, but tends to have greater overhead due to the isofs filesystem.
On tests I've found zisofs images to be between 5% (a single directory
with lots of small binaries) and 61% (lots of nested directories) bigger than the squashfs filesystem.
I believe that squashfs is useful for what you're doing. I'm a bit hesitant in saying that, because I'd rather people downloaded it and made up their own minds :-)
Thank you for trying it out, and I hope you like it. I'll obviously be interested in your thoughts on it.
Phillip
Quoted 6 lines · Phillip Lougher
4. Full 32 bit uids/guids are stored (4 bits stored in inode, uses a lookup table, to give 48 uids/16 gids). File sizes upto 2^32 are supported. Timestamp info is stored. Cramfs truncates uids to 16 bits, uids to 8 bits. Cramfs files sizes are upto 2^24. No timestamp info. Squashfs takes advantage of metadata compression to have more info with smaller metadata overhead.
Why limiting to 2GB? AFAIR you wanted to use a cramfs-like filesystem for backups. Are videos and large data bases not worth of backing up?
It seems to be good work. So I really wait for Al Viros comments ;-)
Regards
Ingo Oeser
Signature · 1 line
Science is what we can tell a computer. Art is everything else. --- D.E.Knuth
Quoted 8 lines · Ingo Oeser
Quoted 1 line · Phillip Lougher
File sizes upto 2^32 are supported.
Why limiting to 2GB? AFAIR you wanted to use a cramfs-like filesystem for backups. Are videos and large data bases not worth of backing up?
Why are files limited to 4GB? (2^32). Simply because I never thought anything bigger was needed :-) Initially files were limited to 2^24 (like cramfs), but because of compressed metadata I thought I could justify an extra 8 bits in the inode, in consideration of the extra functionality.
Going from 32 bits to 40 bits say is a bigger jump - an extra 2 bytes per file inode (1 byte for the extra file size, and 1 byte because the file start block pointer must increase by 1 byte as well). Considering the fact I've tried to squeeze every last byte out...
However, I could add the extra two bytes if people thought it was worth it.
Alternatively, squashfs uses different inodes per file type. I could add an extra "big file" type to deal with files bigger than 4GB. This would mean > 4GB files are supported, with only one extra byte per inode for smaller files. I'll think about it...
Quoted 1 line
It seems to be good work. So I really wait for Al Viros comments ;-)
Thanks!
Phillip
Quoted 3 lines
Regards
Ingo Oeser