Skip to contentSkip to conversation

Squashfs 3.3 released

ANN Phillip Lougher 5 participants 9 messages Nov 5, 2007 – Nov 21, 2007

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

I'm pleased to announce another release of Squashfs. This is the 22nd release in just over five years. Squashfs 3.3 has lots of nice improvements, both to the filesystem itself (bigger blocks and sparse files), but also to the Squashfs-tools Mksquashfs and Unsquashfs.

The next stage after this release is to fix the one remaining blocking issue (filesystem endianness), and then try to get Squashfs mainlined into the Linux kernel again.

The list of changes from the change-log are as follows:

1. Filesystem improvements:

1.1. Maximum block size has been increased to 1Mbyte, and the
default block size has been increased to 128 Kbytes. This improves compression.

1.2. Sparse files are now supported. Sparse files are files
which have large areas of unallocated data commonly called holes. These files are now detected by Squashfs and stored more efficiently. This improves compression and read performance for sparse files.

2. Mksquashfs improvements:

2.1. Exclude files have been extended to use wildcard pattern
matching and regular expressions. Support has also been added for non-anchored excludes, which means it is now possible to specify excludes which match anywhere in the filesystem (i.e. leaf files), rather than always having to specify exclude files starting from the root directory (anchored excludes).

2.2. Recovery files are now created when appending to existing
Squashfs filesystems. This allows the original filesystem to be recovered if Mksquashfs aborts unexpectedly (i.e. power failure).

3. Unsquashfs improvements:

3.1. Multiple extract files can now be specified on the
command line, and the files/directories to be extracted can now also be given in a file.

3.2. Extract files have been extended to use wildcard pattern
matching and regular expressions.

3.3. Filename printing has been enhanced and Unquashfs can
now display filenames with file attributes ('ls -l' style output).

3.4. A -stat option has been added which displays the filesystem
superblock information.

3.5. Unsquashfs now supports 1.x filesystems.

4. Miscellaneous improvements/bug fixes:

4.1. Squashfs kernel code improved to use SetPageError in
squashfs_readpage() if I/O error occurs.

4.2. Fixed Squashfs kernel code bug preventing file
seeking beyond 2GB.

4.3. Mksquashfs now detects file size changes between
first phase directory scan and second phase filesystem create.

Regards

Phillip Lougher

#1 lore raw
Quoted 10 lines · Phillip Lougher

I'm pleased to announce another release of Squashfs. This is the 22nd release in just over five years. Squashfs 3.3 has lots of nice improvements,
both to the filesystem itself (bigger blocks and sparse files), but also to the Squashfs-tools Mksquashfs and Unsquashfs.

The next stage after this release is to fix the one remaining blocking issue (filesystem endianness), and then try to get Squashfs mainlined into the Linux kernel again.

that would be very cool!
with my hat as debian kernel maintainer i'd be very relieved to see it mainlined. i don't know of any major distro that doesn't ship it.

thanks

Signature · 1 line
maks
#2 lore raw
Quoted 8 lines · maximilian attems
Quoted 4 lines · Phillip Lougher

The next stage after this release is to fix the one remaining blocking issue (filesystem endianness), and then try to get Squashfs mainlined into the Linux kernel again.

that would be very cool!

Yes, it would be cool :) Five years is a long time to maintain something out of tree, especially recently when there's been so many minor changes to the VFS interface between kernel releases.

Quoted 3 lines

with my hat as debian kernel maintainer i'd be very relieved to see it mainlined. i don't know of any major distro that doesn't ship it.

I don't know of any major distro that doesn't ship Squashfs either (except arguably Slackware). Putting my other hat on (one of the Ubuntu kernel maintainers) I don't think Squashfs has caused distros that many problems because it is an easy patch to apply (it doesn't touch that many kernel files), but it is always good to minimise the differences from the stock kernel.org kernel.

Phillip

#3 lore raw
Quoted 18 lines · Phillip Lougher
Quoted 8 lines · maximilian attems
Quoted 4 lines · Phillip Lougher

The next stage after this release is to fix the one remaining blocking issue (filesystem endianness), and then try to get Squashfs mainlined into the Linux kernel again.

that would be very cool!

Yes, it would be cool :) Five years is a long time to maintain something out of tree, especially recently when there's been so many minor changes to the VFS interface between kernel releases.

Quoted 3 lines

with my hat as debian kernel maintainer i'd be very relieved to see it mainlined. i don't know of any major distro that doesn't ship it.

Quoted 6 lines

I don't know of any major distro that doesn't ship Squashfs either (except arguably Slackware). Putting my other hat on (one of the Ubuntu kernel maintainers) I don't think Squashfs has caused distros that many problems because it is an easy patch to apply (it doesn't touch that many kernel files), but it is always good to minimise the differences from the stock kernel.org kernel.

The biggest problem we've seen with it (asides from having to rediff it every time we rebase when there isn't a newer upstream) is complaints along the lines of "my Fedora 7 kernel can't unpack squashfs images from Fedora 5"
(s/Fedora 5/other random older distros/ )

If the format is now stable however, it would be great to get it upstream.

Dave

Signature · 1 line
http://www.codemonkey.org.uk
#4 lore raw
Quoted 3 lines · Dave Jones

The biggest problem we've seen with it (asides from having to rediff it every time we rebase when there isn't a newer upstream)

Yes, this is mainly my fault. There was a gap of 10 months between the 3.2 release in January this year, and the latest in November. With the rate of new kernel releases this wasn't really acceptable because the January release was stuck with a patch for kernels no newer than 2.6.20. I received numerous complaints about it.

Some of you may be aware that I started work at Canonical, and this left almost no spare-time to work on Squashfs for 9 months.

Quoted 5 lines

is complaints
along the lines of "my Fedora 7 kernel can't unpack squashfs images from Fedora 5"
(s/Fedora 5/other random older distros/ )

Squashfs has backwards compatibility with older versions, and it should mount all older versions back to 2.0 (released May 2004). Unfortunately RedHat grabbed a CVS version of Squashfs just before the 3.0 release. This was development code, and release testing showed it had a bug where it couldn't mount older versions. It was fixed for
release.

Quoted 2 lines

If the format is now stable however, it would be great to get it upstream.

The move from the 2.0 format to the later 3.0 format was mainly forced by the demands of the Linux kernel mailing list when I first submitted it early 2005. There was no other way to incorporate demands for larger than 4GB filesystems, and provide support for "." and ".." in readdir without modifying the filesystem format.

Unfortunately the move to fixed little endian filesystem will involve another filesystem layout change. The current filesystem layout still uses packed bitfield structures, and it is impossible to swap these using the standard kernel swap macros. Removal of my routines that can properly swap packed bitfield structures is another change demanded by the Linux kernel mailing list.

Once the little endian work has been done, and hopefully once it is in the kernel, I don't anticipate any further layout changes.

Phillip

#5 lore raw
Quoted 6 lines · Phillip Lougher

Unfortunately the move to fixed little endian filesystem will involve another filesystem layout change. The current filesystem layout still uses packed bitfield structures, and it is impossible to swap these using the standard kernel swap macros. Removal of my routines that can properly swap packed bitfield structures is another change demanded by the Linux kernel mailing list.

The normal way to do it is to use shift and mask after doing the endian conversion. But the problem with bitfields is that they can have different kinds of layouts depending on the compiler or abi which is another reason to avoid them in ondisk/wire formats.

#6 lore raw
Quoted 8 lines · Christoph Hellwig
Quoted 1 line · Phillip Lougher

Unfortunately the move to fixed little endian filesystem will involve another filesystem layout change. The current filesystem layout still uses packed bitfield structures, and it is impossible to swap these using the standard kernel swap macros. Removal of my routines that can properly swap packed bitfield structures is another change demanded by the Linux kernel mailing list.

The normal way to do it is to use shift and mask after doing the endian conversion. But the problem with bitfields is that they can have different kinds of layouts depending on the compiler or abi which is another reason to avoid them in ondisk/wire formats.

Yes, the bitfields are packed differently on little and big endian architectures which mean they appear in different places in the structure. I want to move away from that mess when I move to little endian only.

Phillip

#7 lore raw
In reply to Phillip Lougher
Quoted 4 lines · Phillip Lougher

Hi,

I'm pleased to announce another release of Squashfs. This is the 22nd release in just over five years.

Thanks Phillip.

A tiny bug[fix] I always forgot to send... In fs/squashfs/inode.c, constants TASK_UNINTERRUPTIBLE and TASK_INTERRUPTIBLE are used, but they aren't sometimes defined (declared in linux/sched.h):

CC [M] fs/squashfs/inode.o
fs/squashfs/inode.c: In function 'squashfs_get_cached_block':
fs/squashfs/inode.c:367: error: 'TASK_UNINTERRUPTIBLE' undeclared (first use in this function) fs/squashfs/inode.c:367: error: (Each undeclared identifier is reported only once fs/squashfs/inode.c:367: error: for each function it appears in.) fs/squashfs/inode.c:367: warning: implicit declaration of function 'schedule' fs/squashfs/inode.c:404: error: 'TASK_INTERRUPTIBLE' undeclared (first use in this function) fs/squashfs/inode.c: In function 'release_cached_fragment':
fs/squashfs/inode.c:499: error: 'TASK_UNINTERRUPTIBLE' undeclared (first use in this function) fs/squashfs/inode.c:499: error: 'TASK_INTERRUPTIBLE' undeclared (first use in this function) fs/squashfs/inode.c: In function 'get_cached_fragment':
fs/squashfs/inode.c:522: error: 'TASK_UNINTERRUPTIBLE' undeclared (first use in this function) fs/squashfs/inode.c:559: error: 'TASK_INTERRUPTIBLE' undeclared (first use in this function)

I'm not exactly sure which config option is "at blame" (this is an i486-based UP generic-hardware config), but I'm not interested to know, either. The following trivial patch fixes it once for all.

fs/squashfs/inode.c 2007-07-12 14:57:53.000000000 +0400 renamed from fs/squashfs/inode.c 2007-07-12 14:57:22.000000000 +0400 +1 -0
@@ -31,6 +31,7 @@
 #include <linux/vfs.h>
 #include <linux/vmalloc.h>
+#include <linux/sched.h>
 #include <linux/smp_lock.h>
 
 #include "squashfs.h"
 

It was needed for v3.2 too.

Thanks.

/mjt

#8 lore raw
Quoted 5 lines · Michael Tokarev

A tiny bug[fix] I always forgot to send... In fs/squashfs/inode.c, constants TASK_UNINTERRUPTIBLE and TASK_INTERRUPTIBLE are used, but they aren't sometimes defined (declared in linux/sched.h):

Thanks - Squashfs gained a lot of #includes over time, many which I deemed were
unnecessary and removed in Squashfs 3.2. I obviously removed too many.
Fix applied to CVS.

Phillip