Squashfs: add XZ compression support
cover Squashfs: add XZ compression support
Hi,
Following the recent posting of patches by Lasse Collin to add XZ (LZMA2) support to the kernel (http://thread.gmane.org/gmane.linux.kernel/1071297),
I have added support for this to Squashfs.
Advantages of Squashfs XZ over the Squashfs LZMA implementation:
- Significantly better decompressor API supporting multi-call decoding, which requires less buffer overhead.
- Greater data robustness due to XZ's CRC32 check.
- BCJ filters which can produce smaller Squashfs images.
The following two patches add Squashfs kernel support. A git tree with these patches including Lasse Collin's patches is available here:
http://git.kernel.org/?p=linux/kernel/git/pkl/squashfs-xz.git;a=summary
XZ support has (obviously) also been added to the squashfs tools (Mksquashfs & Unsquashfs). These changes are available from the Squashfs CVS repository
(http://sourceforge.net/projects/squashfs/develop).
To build the Squashfs tools, edit the Makefile to enable XZ support (by default it is disabled).
XZ compression can be specified by using the -comp option, e.g. % mksquashfs xxx img.sqsh -comp xz
XZ BCJ filters (which can improve the compression of executable code on certain architectures) are supported by using the -Xbcj option, e.g.
% mksquashfs xxx img.sqsh -comp xz -Xbcj x86
will compress blocks using XZ with no filter, and then XZ with the x86 filter in turn, and choose the best compression.
Multiple filters can be specified which is useful in cases where the source file system has executable code from a mixture of different architectures, and again each filter will be tried for each block and the best compression used, e.g.
% mksquashfs xxx img.sqsh -comp xz -Xbcj x86,arm
will try both the x86 and arm BCJ filters.
Phillip
Quoted 4 lines · Phillip Lougher
XZ support has (obviously) also been added to the squashfs tools (Mksquashfs & Unsquashfs). These changes are available from the Squashfs CVS repository
(http://sourceforge.net/projects/squashfs/develop).
Do you have an estimate for when you will make a release with these changes and what will likely be the version number? I want to see if it makes sense to start tracking the development version for F15 or F16 and what to use for the version.
Quoted 11 lines · Bruno Wolff III
Quoted 3 lines · Phillip Lougher
XZ support has (obviously) also been added to the squashfs tools (Mksquashfs & Unsquashfs). These changes are available from the Squashfs CVS repository
(http://sourceforge.net/projects/squashfs/develop).Do you have an estimate for when you will make a release with these changes and what will likely be the version number? I want to see if it makes sense to start tracking the development version for F15 or F16 and what to use for the version.
Hmm, I don't know when F15 or F16 is destined to appear! (I guess roughly 3 to 6 months minimum time but correct me if I'm wrong).
Lasse collin's comment re: dictionary sizes threw a spanner in the works so to speak as I realised my current patches don't allow users to specify the dictionary size, which appears rather useful judging by Lasse's comments that a dictionary size of 50% of block size only impacts compression by 1%. This makes it rather obvious that allowing users to specify dictionary size in addition to block size is extremely desirable.
The rather unfortunate issue is that the current Squashfs file format doesn't allow compressor specific options to be stored, for example it allows compression type (i.e. gzip, lzma, lzo etc) to be stored, but there is nowhere to store a dictionary size.
So, this has led to a week of thinking about how to add compressor specific options to the Squashfs file format which is 100% compatible with pre-existing 4.0 file systems and file system code. I have settled on an extension which is 100% compatible and which fits in well with the file format (i.e. it is not a hack).
What does this mean in relation to your questions? Well basically it means in a week's time or before I'll be in a position to respin the kernel patches to include compressor specific options support (i.e. dictionary size), and there will be the corresponding CVS changes for the Squashfs tools.
As far as I'm concerned this makes a good point to make a new Squashfs tools release (very few people trust the CVS versions in my experience and always go for the last released version). I'll probably wait for a couple of weeks just in case CVS downloaders find any bugs (in practice I hardly ever get any bug reports from the CVS version). This means a Squashfs tools 4.2
release mid January. How does this fit in with the release schedules for
F15 and F16?
Phillip
Quoted 8 lines · Phillip Lougher
As far as I'm concerned this makes a good point to make a new Squashfs tools release (very few people trust the CVS versions in my experience and always go for the last released version). I'll probably wait for a couple of weeks just in case CVS downloaders find any bugs (in practice I hardly ever get any bug reports from the CVS version). This means a Squashfs tools 4.2
release mid January. How does this fit in with the release schedules for
F15 and F16?
For the kernel stuff, I need to wait until it lands. This looks good for making 2.6.38 which would make F15. If it doesn't then I'll continue to wait. (Fedora is fairly strict on using upstream for kernel stuff except where they have dedicated people as for video drivers.) (I may build some personal kernels with the patches so I can test live images more completely, but I can get by with testing squashfs-tools independently for now.)
For squashfs-tools, a mid January release sounds plenty fine. It could slip a lot and not cause a problem. I have some time off from work coming up next week and will get a 4.2 prerelease (fromCVS) into rawhide next week so we can get some testing there.
Thanks!