Skip to contentSkip to conversation

cramfs in big endian

DISCUSSION Andi Drebes 4 participants 11 messages Nov 6, 2007 – Nov 11, 2007 VFS

lore
Share & tools
lore
#0 lore raw

Hi!
I'm currently trying to enable the cramfs to mount filesystems with a different endianness. All I have is an intel compatible machine that produces cramfs images in little endian format. I found a modified version of mkcramfs that is able to produce images in big endian mode. Unfortunately it seems to use a different compression algorithm (LZO) than the standard implementation (zlib).

I would be very glad if somebody could send me an image in big endian mode using the zlib deflate compression algorithm. I have uploaded a bz2 archive containing some testfiles. You can find it at [1].

Regards,
Andi

[1] http://drebesium.org/~hackbert/cramfs-testfiles.tar.bz2

#1 lore raw
Quoted 2 lines

I'm currently trying to enable the cramfs to mount filesystems with a different endianness.

I would suggest you to use squashfs instead of cramfs.

First, it's newer, it's better, it's actively developed, it doesn't have any limits like the bad cramfs. Moreover, it currently supports both endians.

(hurry up, as kernel people said in the past that squashfs should NEVER EVER support multiple endians, so the feature will be dropped from squashfs, in order to get it into mainline kernel more easily; if my informations are correct).

Tomas M

#2 lore raw

Hi!

Quoted 3 lines

I would suggest you to use squashfs instead of cramfs.
First, it's newer, it's better, it's actively developed, it doesn't have any limits like the bad cramfs.

I'm developing a new linux based firmware for my router which uses cramfs. Switching to squashfs still needs some time. Meanwhile, I have to work with cramfs. As the router uses the big endian format and as my machine works with the little endian format, I'm unable to mount the router's filesystem images.

Andi

#3 lore raw
Quoted 9 lines · Andi Drebes

Hi!

Quoted 3 lines

I would suggest you to use squashfs instead of cramfs.
First, it's newer, it's better, it's actively developed, it doesn't have any limits like the bad cramfs.

I'm developing a new linux based firmware for my router which uses cramfs. Switching to squashfs still needs some time. Meanwhile, I have to work with cramfs. As the router uses the big endian format and as my machine works with the little endian format, I'm unable to mount the router's filesystem images.

Making cramfs endianess-independent shouldn't be much work. Take a look at the helpers in fs/ufs/swab.h and use them for every ondisk access in cramfs. Drop me a not if you need some help.

#4 lore raw

Hi!

Quoted 3 lines

Making cramfs endianess-independent shouldn't be much work. Take a look at the helpers in fs/ufs/swab.h and use them for every ondisk access in cramfs. Drop me a not if you need some help.

I already started working on this feature some time ago. I think I am able to mount cramfs images with another endianness. But all I have to test this are images that use LZO compression istead of deflate.

In the original post I was just asking somebody possessing a big endian machine to create an image in big endian / deflate format. This would help me a lot. Later, I want to add support for multiple compression algorithms.

Thanks so far,
Andi

#5 lore raw
In reply to Christoph Hellwig
Quoted 11 lines · Christoph Hellwig
Quoted 6 lines · Andi Drebes

Hi!

Quoted 3 lines

I would suggest you to use squashfs instead of cramfs.
First, it's newer, it's better, it's actively developed, it doesn't have any limits like the bad cramfs.

I'm developing a new linux based firmware for my router which uses cramfs. Switching to squashfs still needs some time. Meanwhile, I have to work with cramfs. As the router uses the big endian format and as my machine works with the little endian format, I'm unable to mount the router's filesystem images.

Making cramfs endianess-independent shouldn't be much work. Take a look at the helpers in fs/ufs/swab.h and use them for every ondisk access in cramfs. Drop me a not if you need some help.

And it would suck. Hard.

Endian-independent code is slower than wrong-endian code, because of the necessary conditionals. Thus, you DO NOT WANT this(*).

The only way to not make it totally suck is to compile the code twice and do switching at a VERY high level -- in effect, creating two separate filesystem drivers.

-hpa

(*) Readonly filesystems like iso9660 have attempted to be *both*-endian, by having its datastructures in both forms at all times.
That is inefficient with space, but would be CPU-efficient in
theory. Unfortunately it causes output validation problems; apparently many iso9660 mastering programs output invalid bigendian information as it is never tested; consequently, many bigendian platforms read the littleendian information anyway.

#6 lore raw
Quoted 2 lines · H. Peter Anvin

Endian-independent code is slower than wrong-endian code, because of the necessary conditionals. Thus, you DO NOT WANT this(*).

I'd prefer not to have it either. But a someone (pinhead) was smart enough not to define an endianess for cramfs from the beginning we're stuck with it.

#7 lore raw
Quoted 7 lines · Christoph Hellwig
Quoted 1 line · H. Peter Anvin

Endian-independent code is slower than wrong-endian code, because of the necessary conditionals. Thus, you DO NOT WANT this(*).

I'd prefer not to have it either. But a someone (pinhead) was smart enough not to define an endianess for cramfs from the beginning we're stuck with it.

I thought cramfs was "always" defined as littleendian?

Either way... I thought the primary discussion was about squashfs.

-hpa

#8 lore raw
In reply to Christoph Hellwig
Quoted 6 lines
Quoted 2 lines

Endian-independent code is slower than wrong-endian code, because of the necessary conditionals. Thus, you DO NOT WANT this(*).

I'd prefer not to have it either. But a someone (pinhead) was smart enough not to define an endianess for cramfs from the beginning we're stuck with it.

Indeed, this is the problem. The readme file fs/cramfs/README states:

"One part of that is addressing endianness. The two options here are always use little-endian' (like ext2fs) or writer chooses endianness; kernel adapts at runtime'. Little-endian wins because of code simplicity and little CPU overhead even on big-endian machines."

Unfortunately, the better idea was never implemented. Further, there's no information about the endianness stored in the filesystem. Guessing it and mounting the filesystem isn't a clean solution. Even worse, there's no information about which compression algorithm was used to create the filesystem. Guessing the compression method may lead to serious problems.

So here is my proposal for future development of cramfs:
The should tell cramfs how to mount a filesystem. Therefore, the endianness and the compression method both have to be specified manually. If none is specified, cramfs will assume host endianness and that deflate can be used in order to decompress the contents. If something seems to be wrong with the filesystem (e.g. wrong magic), cramfs will guess the endianness and inform the user about the guess, but it won't mount the filesystem if it doesn't match the endianness specified or the host's one.

Andi

#9 lore raw
Quoted 8 lines · Andi Drebes

Indeed, this is the problem. The readme file fs/cramfs/README states:

"One part of that is addressing endianness. The two options here are always use little-endian' (like ext2fs) or writer chooses endianness; kernel adapts at runtime'. Little-endian wins because of code simplicity and little CPU overhead even on big-endian machines."

Quoted 5 lines

Unfortunately, the better idea was never implemented. Further, there's no information about the endianness stored in the filesystem. Guessing it and mounting the filesystem isn't a clean solution. Even worse, there's no information about which compression algorithm was used to create the filesystem. Guessing the compression method may lead to serious problems.

So here is my proposal for future development of cramfs:
The should tell cramfs how to mount a filesystem. Therefore, the endianness and the compression method both have to be specified manually. If none is specified, cramfs will assume host endianness and that deflate can be used in order to decompress the contents. If something seems to be wrong with the filesystem (e.g. wrong magic), cramfs will guess the endianness and inform the user about the guess, but it won't mount the filesystem if it doesn't match the endianness specified or the host's one.

What about simply deep-sixing cramfs and replacing it with squashfs or something else?

-hpa

#10 lore raw
Quoted 2 lines

What about simply deep-sixing cramfs and replacing it with squashfs or something else?

I think this is the long term solution. Cramfs isn't a very beautiful filesystem. It's a good candidate for removal. However, there are still some distributions that use cramfs for initrds. So removing it immediately isn't a good idea.

Andi