Skip to contentSkip to conversation

Squashfs 4.0 tools

DISCUSSION Stephen Hemminger 6 participants 13 messages Mar 24, 2009 – Apr 6, 2009

lore
Share & tools
lore
#0 lore raw

It is great that squashfs is finally in mainline kernel. But the squashfs tools are tightly bound to the version of the squashfs filesystem. The latest tools in Debian (and the download page) are limited to version 3.4 but the upstream kernel version is 4.0!

Do we have a cart with no horse here?

#1 lore raw
Quoted 6 lines · Stephen Hemminger

It is great that squashfs is finally in mainline kernel. But the
squashfs tools are tightly bound to the version of the squashfs filesystem.
The latest tools in Debian (and the download page) are limited to version 3.4 but the upstream kernel version is 4.0!

Do we have a cart with no horse here?

The horse is currently in training, or to put it more mundanely, the squashfs 4.0 tools are currently in Squashfs CVS :-)

I have been working on the 4.0 squashfs tools for the last couple of months making them (mksquashfs and unsquashfs) ready for a supported release. Now that Linux 2.6.29 has been released (3 hours ago), I need to make this release ASAP, which as everything is almost finished, should be sometime this week.

Phillip

#2 lore raw
Quoted 3 lines · Phillip Lougher
Quoted 2 lines · Stephen Hemminger

It is great that squashfs is finally in mainline kernel. But the squashfs tools are tightly bound to the version of the squashfs filesystem.

[]

Quoted 5 lines

I have been working on the 4.0 squashfs tools for the last couple of months making them (mksquashfs and unsquashfs) ready for a supported release. Now that Linux 2.6.29 has been released (3 hours ago), I need to make this release ASAP, which as everything is almost finished, should be sometime this week.

By the way, with this on-disk format change, is there a way to get 4.0 fs code working with 2.6.27 (long-time-stable) kernel? I tried to compile it on 2.6.27 but it fails due to some API changes and I'm not that deep into kernel to fix it...

Thanks!

#4 lore raw
In reply to Phillip Lougher
Quoted 8 lines · On Monday 23 March 2009 21:39:02 Phillip Lougher wrote:
Quoted 6 lines · Stephen Hemminger

It is great that squashfs is finally in mainline kernel. But the squashfs tools are tightly bound to the version of the squashfs filesystem. The latest tools in Debian (and the download page) are limited to version 3.4 but the upstream kernel version is 4.0!

Do we have a cart with no horse here?

Quoted 8 lines

The horse is currently in training, or to put it more mundanely, the squashfs 4.0 tools are currently in Squashfs CVS :-)

I have been working on the 4.0 squashfs tools for the last couple of months making them (mksquashfs and unsquashfs) ready for a supported release. Now that Linux 2.6.29 has been released (3 hours ago), I need to make this release ASAP, which as everything is almost finished, should be sometime this week.

Any updated ETA on this?

Rob

P.S I notice a squashfs-devel mailing list cc'd here, but there's no link to it from the squashfs.org page. Buried in the middle of a lot of text is a link to the sourceforge squashfs project page, which has a link to "forums" which are not the same as mailing lists.

Since I knew it's called "squashfs-devel" I could google for that and find
http://sourceforge.net/mailarchive/forum.php?forum_name=squashfs-devel

But it might be nice to at least _hint_ about its existence on the web page. The main page's "contact" section is your personal email and nothing else...

#5 lore raw
Quoted 4 lines · Rob Landley

P.S I notice a squashfs-devel mailing list cc'd here, but there's no link to it from the squashfs.org page.  Buried in the middle of a lot of text is a link to the sourceforge squashfs project page, which has a link to "forums" which are not the same as mailing lists.

There should be a link to it somewhere on the SF project page...

Quoted 2 lines

But it might be nice to at least _hint_ about its existence on the web page. The main page's "contact" section is your personal email and nothing else...

Yep. Fortunately we also have MAINTAINERS ;-)

| SQUASHFS FILE SYSTEM
| P:      Phillip Lougher
| M:      [email protected]
| L:      [email protected] (subscribers-only)
| W:      http://squashfs.org.uk
| S:      Maintained

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#6 lore raw
Quoted 22 lines · On Sun, 29 Mar 2009 11:02:19 +0200 Geert Uytterhoeven <[email protected]>…
Quoted 4 lines · Rob Landley

P.S I notice a squashfs-devel mailing list cc'd here, but there's no link to it from the squashfs.org page.  Buried in the middle of a lot of text is a link to the sourceforge squashfs project page, which has a link to "forums" which are not the same as mailing lists.

There should be a link to it somewhere on the SF project page...

Quoted 2 lines

But it might be nice to at least _hint_ about its existence on the web page. The main page's "contact" section is your personal email and nothing else...

Yep. Fortunately we also have MAINTAINERS ;-)

| SQUASHFS FILE SYSTEM
| P:      Phillip Lougher
| M:      [email protected]
| L:      [email protected] (subscribers-only)
| W:      http://squashfs.org.uk
| S:      Maintained

Gr{oetje,eeting}s,

Quoted 8 lines

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

I built it from cvs (imported into git), and it works okay.

#7 lore raw
Quoted 4 lines · Stephen Hemminger
Quoted 1 line · On Sun, 29 Mar 2009 11:02:19 +0200 Geert Uytterhoeven <[email protected]>…

On Sun, Mar 29, 2009 at 00:27, Rob Landley <[email protected]> wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

P.S I notice a squashfs-devel mailing list cc'd here, but there's no link to it from the squashfs.org page. Buried in the middle of a lot of text is a link to the sourceforge squashfs project page, which has a link to "forums" which are not the same as mailing lists.

There should be a link to it somewhere on the SF project page...

Quoted 2 lines

But it might be nice to at least _hint_ about its existence on the web page. The main page's "contact" section is your personal email and nothing else...

Squashfs is a sourceforge project, and it has a link to the main sourceforge page from the webpage. I've always thought that sufficient because users can get to the mailing lists, web forum, bug tracker etc. from that, and then choose what they think is the most appropriate way to report issues.

Missing information is simply that, and I rely on feedback from users to say if I've forgotten to add important or relevant information anywhere. It would be helpful if that feedback didn't include an implied criticism, that way we'll all get along better.

Quoted 3 lines

I built it from cvs (imported into git), and it works okay.

Good, this is what I suggested you do :-) CVS contains everything I've been working on for the last couple of months.

I intended to do a release last week, but, release testing found a number of bugs and other issues which had to be addressed, and which consequently delayed the release.

If you grabbed the tools from CVS more than a day ago you'll find there's problems with Unsquashfs and big-endian systems. The last bug fix (a deadlock issue in mksquashfs undiscovered since 3.4) went in just now, and so CVS should be what's in the release.

BTW Stephen, it would be helpful if you subscribed to the squashfs-devel mailing list, at the moment I've having to authorise all your posts which is a bit of a pain.

Phillip

#3 lore raw
In reply to Michael Tokarev
Quoted 5 lines · Michael Tokarev
Quoted 3 lines · Phillip Lougher
Quoted 2 lines · Stephen Hemminger

It is great that squashfs is finally in mainline kernel. But the squashfs tools are tightly bound to the version of the squashfs filesystem.

[]

Quoted 8 lines
Quoted 5 lines

I have been working on the 4.0 squashfs tools for the last couple of months making them (mksquashfs and unsquashfs) ready for a supported release. Now that Linux 2.6.29 has been released (3 hours ago), I need to make this release ASAP, which as everything is almost finished, should be sometime this week.

By the way, with this on-disk format change, is there a way to get 4.0 fs code working with 2.6.27 (long-time-stable) kernel?

I tried to compile it on 2.6.27 but it fails due to some API changes and I'm

Quoted 2 lines

not that deep into kernel to fix it...

Yes I know. The 4.0 patches only work on 2.6.28 or later AFAIK.

When I get time and if there's demand (you're the first person to ask and so there's obviously some demand), I'll backport the patches to earlier kernels.

Phillip

#8 lore raw
In reply to Stephen Hemminger
Quoted 1 line · On Monday 30 March 2009 19:55:24 Stephen Hemminger wrote:

I built it from cvs (imported into git), and it works okay.

I grabbed the cvs snapshot gentoo's been using, but I don't see any way to feed in a device list text file so I can create the /dev/console node root filesystems need without needing root access on the host to mknod.

genext2fs has the -D option, which takes:

/dev d 755 0 0 - - - - -
/dev/console c 640 0 0 5 1 0 0 -

gen_init_cpio takes:

dir /dev 755 0 0
nod /dev/console 660 0 0 c 5 1

Is there a squashfs option I'm missing...?

Rob

Signature · 1 line
GPLv3 is to GPLv2 what Attack of the Clones is to The Empire Strikes Back.
#9 lore raw
Quoted 10 lines · Rob Landley
Quoted 1 line · On Monday 30 March 2009 19:55:24 Stephen Hemminger wrote:

I built it from cvs (imported into git), and it works okay.

I grabbed the cvs snapshot gentoo's been using, but I don't see any way to feed in a device list text file so I can create the /dev/console node root filesystems need without needing root access on the host to mknod.

genext2fs has the -D option, which takes:

/dev d 755 0 0 - - - - -
/dev/console c 640 0 0 5 1 0 0 -

Quoted 7 lines

gen_init_cpio takes:

dir /dev 755 0 0
nod /dev/console 660 0 0 c 5 1

Is there a squashfs option I'm missing...?

No you're unfortunately not missing an option :-) That option doesn't exist.

But thanks for sending the email, a couple of people over the years have asked for something like this, and you've reminded me that this option doesn't exist in Mksquashfs...

I have thought about adding this option a couple of times, but it has always been dropped through lack of time. Efficiently adding the option isn't as easy as it sounds because the "fake" device nodes have to somehow be made available to the recursive directory scan routine that scans the source directories for files. Adding a list of fake device nodes which is scanned on every directory for potential inclusion in that directory is easy, but not terribly efficient.

I'll delay the release of the 4.0 tools and see if I can implement this option in the next day or two.

Phillip

#10 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Phillip" == Phillip Lougher <[email protected]> writes:

Hi,

Phillip> I have thought about adding this option a couple of times, Phillip> but it has always been dropped through lack of time. Phillip> Efficiently adding the option isn't as easy as it sounds Phillip> because the "fake" device nodes have to somehow be made Phillip> available to the recursive directory scan routine that scans Phillip> the source directories for files. Adding a list of fake Phillip> device nodes which is scanned on every directory for Phillip> potential inclusion in that directory is easy, but not Phillip> terribly efficient.

The normal solution for this is fakeroot, which is also what we're using for mksquashfs in buildroot:

http://git.buildroot.net/buildroot/tree/target/squashfs/squashfsroot.mk

Phillip> I'll delay the release of the 4.0 tools and see if I can Phillip> implement this option in the next day or two.

I would say rather get 4.0 out of the door as soon as possible and wait with such nice-to-have stuff for later.

Signature · 1 line
Bye, Peter Korsgaard
#11 lore raw
In reply to Phillip Lougher
Quoted 13 lines · Phillip Lougher
Quoted 12 lines · Rob Landley
Quoted 1 line · On Monday 30 March 2009 19:55:24 Stephen Hemminger wrote:

I built it from cvs (imported into git), and it works okay.

I grabbed the cvs snapshot gentoo's been using, but I don't see any way to feed in a device list text file so I can create the /dev/console node root filesystems need without needing root access on the host to mknod.

genext2fs has the -D option, which takes:

/dev d 755 0 0 - - - - -
/dev/console c 640 0 0 5 1 0 0 -

gen_init_cpio takes:

Quoted 7 lines
Quoted 5 lines

dir /dev 755 0 0
nod /dev/console 660 0 0 c 5 1

Is there a squashfs option I'm missing...?

I'll delay the release of the 4.0 tools and see if I can implement this option in the next day or two.

The Squashfs 4.0 tools have been released on sourceforge
(http://www.squashfs.org), and mksquashfs now includes support for fake devices and directories.

From README-4.0

Mksquashfs now supports pseudo files, these allow fake directories, character and block devices to be specified and added to the Squashfs filesystem being built, rather than requiring them to be present in the source directories. This, for example, allows device nodes to be added to the filesystem without requiring root access.

Two options are supported, -p allows one pseudo file to be specified on the command line, and -pf allows a pseudo file to be specified containing a list of pseudo definitions, one per line.

Pseudo device nodes are specified using 7 arguments

Filename type mode uid gid major minor

Where type is either
b - for block devices, and
c - for character devices

mode is the octal mode specifier, similar to that expected by chmod.

Uid and gid can be either specified as a decimal number, or by name.

For example:

/dev/chr_dev c 666 root root 100 1
/dev/blk_dev b 444 0 0 200 200

Directories are specified using 5 arguments

Filename type mode uid gid

Where type is d.

To ensure the pseudo definition is parsed as one argument and to ensure it isn't interpreted by the shell, the definition should be enclosed in quotes when using the -p option, i.e.

mksquashfs test test.sqsh -p "/dev/chrdev c 666 root 100 1"

If a directory given in a pseudo file path exists in the real source directories it doesn't have to be specified as a pseudo directory, if it doesn't exist in the source directories, it must be specified as a pseudo directory (otherwise any pseudo devices under the directory will be ignored).

For example

/dev d 777 root root
/dev/chr_dev c 666 root root 100 1

Phillip

#12 lore raw
Quoted 22 lines · On Sunday 05 April 2009 18:27:49 Phillip Lougher wrote:
Quoted 21 lines · Phillip Lougher
Quoted 19 lines · Rob Landley
Quoted 1 line · On Monday 30 March 2009 19:55:24 Stephen Hemminger wrote:

I built it from cvs (imported into git), and it works okay.

I grabbed the cvs snapshot gentoo's been using, but I don't see any way to feed in a device list text file so I can create the /dev/console node root filesystems need without needing root access on the host to mknod.

genext2fs has the -D option, which takes:

/dev d 755 0 0 - - - - -
/dev/console c 640 0 0 5 1 0 0 -

gen_init_cpio takes:

dir /dev 755 0 0
nod /dev/console 660 0 0 c 5 1

Is there a squashfs option I'm missing...?

Quoted 8 lines
Quoted 2 lines

I'll delay the release of the 4.0 tools and see if I can implement this option in the next day or two.

The Squashfs 4.0 tools have been released on sourceforge
(http://www.squashfs.org), and mksquashfs now includes support for fake devices and directories.

From README-4.0

Some suggested wordsmithing on the text you posted:

---

Mksquashfs now supports pseudo-files, which may be specified via the -p and -pf command line options.

Pseudo-files allow fake directories, character devices, and block devices to be added to the Squashfs filesystem being built without requiring them to be present in the source filesystem. This allows a non-root user to create a squashfs filesystem containing device nodes and directories (but files) with arbitrary ownership/permissions.

The -p option specifies an individual pseudo-file, and takes a single argument such as -p "/dev d 755 root root". The -p option may be repeated as necessary.

The -pf option specifies a file containing a list of pseudo-file definitions, one per line.

Pseudo-file definitions look like this:

/dev d 755 root root
/dev/chr_dev c 666 root root 100 1
/dev/blk_dev b 444 0 0 200 200

The first field is the complete path to the new entry in the squashfs filesystem. (This path cannot currently contain any whitespace space characters.) The second field is a single letter indicating the entry type. Currently supported types are d (directory), b (block device), and c (char device). The third, fourth, and fifth fields are (respectively) octal mode, user id, and group ID. UID and GID can be specified either by decimal number or using a name valid on the host filesystem.

Directory pseudo-file definitions only contain five fields. Device pseudo-file definitions include major and minor numbers in the sixth and seventh fields.

To ensure the pseudo-file definition is parsed as one argument, and to ensure it isn't interpreted by the shell, the definition should be enclosed in quotes when using the -p option, i.e.

mksquashfs test test.sqsh -p "/dev/chrdev c 666 root 100 1"

If a directory given in a pseudo file path exists in the real source directories it doesn't have to be specified as a pseudo directory, if it doesn't exist in the source directories, it must first be specified as a pseudo directory (otherwise any pseudo devices under the directory will be ignored).

For example

/dev d 755 root root
/dev/chr_dev c 666 root root 100 1

---

Thanks, looks good. I should be able to test it out in the morning.

Rob

Signature · 1 line
GPLv3 is to GPLv2 what Attack of the Clones is to The Empire Strikes Back.