initramfs buffer spec -- second draft
This is an update to the initramfs buffer format spec I posted earlier. The changes are as follows:
a) Move the PAD() declarations around. It is now required that the cpio header is aligned on a multiple of 4 bytes, thereby removing a potential ambiguity in the previous specification.
b) Clearly specify that the data can be attached to any member of a hard link set.
As always, comments appreciated...
initramfs buffer format
-----------------------
Al Viro, H. Peter Anvin
Last revision: 2002-01-11
** DRAFT ** DRAFT ** DRAFT ** DRAFT ** DRAFT ** DRAFT **
Starting with kernel 2.5.x, the old "initial ramdisk" protocol is getting {replaced/complemented} with the new "initial ramfs" (initramfs) protocol. The initramfs contents is passed using the same memory buffer protocol used by the initrd protocol, but the contents is different. The initramfs buffer contains an archive which is expanded into a ramfs filesystem; this document details the format of the initramfs buffer format.
The initramfs buffer format is based around the "newc" CPIO format, and can be created with the cpio(1) utility. The cpio archive can be compressed using gzip(1). The simplest form of the initramfs buffer is thus a single .cpio.gz file.
The full format of the initramfs buffer is defined by the following grammar, where:
* is used to indicate "0 or more occurrences of"
(|) indicates alternatives
+ indicates concatenation
GZIP() indicates the gzip(1) of the operand
PAD(n) means padding with null bytes to an n-byte boundary
[QUESTION: is the padding relative to the start of the previous header, or is it an absolute address? Is it at all legal to have a header start on a non-multiple of 4?]
initramfs := ("\0" | cpio_archive | cpio_gzip_archive)*
cpio_gzip_archive := GZIP(cpio_archive)
cpio_archive := cpio_file* + (<nothing> | cpio_trailer)
cpio_file := PAD(4) + cpio_header + filename + "\0" + PAD(4) + data
cpio_trailer := PAD(4) + cpio_header + "TRAILER!!!\0" + PAD(4)
In human terms, the initramfs buffer contains a collection of compressed and/or uncompressed cpio archives (in the "newc" format); arbitrary amounts zero bytes (for padding) can be added between members.
The cpio "TRAILER!!!" entry (cpio end of file) is optional, but is not ignored; see "handling of hard links" below.
The structure of the cpio_header is as follows (all 8-byte entries contain 32-bit hexadecimal ASCII numbers):
Field name Field size Meaningc_magic 6 bytes The string "070701" or "070702"c_ino 8 bytes File inode numberc_mode 8 bytes File mode and permissionsc_uid 8 bytes File uidc_gid 8 bytes File gid
c_nlink 8 bytes Number of links c_mtime 8 bytes Modification time c_filesize 8 bytes Size of data field
c_maj 8 bytes Major part of file device numberc_min 8 bytes Minor part of file device numberc_rmaj 8 bytes Major part of device node referencec_rmin 8 bytes Minor part of device node referencec_namesize 8 bytes Length of filename, including final \0c_chksum 8 bytes CRC of data field if c_magic is 070702
The c_mode field matches the contents of st_mode returned by stat(2) on Linux, and encodes the file type and file permissions.
The c_filesize should be zero for any non-regular file.
If the filename is "TRAILER!!!" this is actually an end-of-file marker; the c_filesize for an end-of-file marker must be zero.
*** Handling of hard links
When a nondirectory with c_nlink > 1 is seen, the (c_maj,c_min,c_ino) tuple is looked up in a tuple buffer. If not found, it is entered in the tuple buffer and the entry is created as usual; if found, a hard link rather than a second copy of the file is created. It is not necessary (but permitted) to include a second copy of the file contents; if the file contents is not included, the c_filesize field should be set to zero to indicate no data section follows. If data is present, the previous instance of the file is overwritten; this allows the data-carrying instance of a file to occur anywhere in the sequence (GNU cpio is reported to attach the data to the last instance of a file only.)
When a "TRAILER!!!" end-of-file marker is seen, the tuple buffer is reset. This permits archives which are generated independently to be concatenated.
To combine file data from different sources (without having to regenerate the (c_maj,c_min,c_ino) fields), therefore, either one of the following techniques can be used:
a) Separate the different file data sources with a "TRAILER!!!" end-of-file marker, or
b) Make sure c_nlink == 1 for all nondirectory entries.
Signature · 3 lines
<[email protected]> at work, <[email protected]> in private! "Unix gives you enough rope to shoot yourself in the foot." http://www.zytor.com/~hpa/puzzle.txt <[email protected]>
First off, the documentation is great and the approach seems fundamentallly sound.
On January 12, 2002 09:04 am, H. Peter Anvin wrote:
[...]
Quoted 4 lines
PAD(n) means padding with null bytes to an n-byte boundary
[QUESTION: is the padding relative to the start of the previous header, or is it an absolute address? Is it at all legal to have a header start on a non-multiple of 4?]
I'll vote for the always/absolute rule.
[...]
Quoted 2 lines
The structure of the
cpio_headeris as follows (all 8-byte entries contain 32-bit hexadecimal ASCII numbers):
I thought there's a binary version of the cpio header. What is the point of the ascii encoding?
[...]
Quoted 7 lines
The
c_modefield matches the contents ofst_modereturned by stat(2) on Linux, and encodes the file type and file permissions.The
c_filesizeshould be zero for any non-regular file.If the filename is "TRAILER!!!" this is actually an end-of-file marker; the
c_filesizefor an end-of-file marker must be zero.
It sure looks ugly, but I suppose the c_filesize=zero is the real end-of-file marker. Did I mention it sure looks ugly?
--
Daniel
Quoted 2 lines · H. Peter Anvin
This is an update to the initramfs buffer format spec I posted earlier. The changes are as follows:
Comments. Endian issues are not specified, is the data little, big or vax endian?
What is the point of alignment? If the data starts as 4 byte aligned, the 6 byte magic string guarantees the data will be only 2 byte aligned. This isn't good for 32 or 64 bit architectures.
I do like having a c_magic that at least allows us to change things in the future if necessary.
Eric
Quoted 7 lines · Eric W. Biederman
Quoted 2 lines · H. Peter Anvin
This is an update to the initramfs buffer format spec I posted earlier. The changes are as follows:
Comments. Endian issues are not specified, is the data little, big or vax endian?
Data is what you put into files, byte-by-byte. Headers are ASCII.
Quoted 3 lines
What is the point of alignment? If the data starts as 4 byte aligned, the 6 byte magic string guarantees the data will be only 2 byte aligned. This isn't good for 32 or 64 bit architectures.
Both data and headers are aligned. And headers are ascii strings.
Quoted 10 lines · Alexander Viro
Quoted 7 lines · Eric W. Biederman
Quoted 2 lines · H. Peter Anvin
This is an update to the initramfs buffer format spec I posted earlier. The changes are as follows:
Comments. Endian issues are not specified, is the data little, big or vax endian?
Data is what you put into files, byte-by-byte. Headers are ASCII.
Encoding the numeric fields in ASCII/hex is a goofy wart on an otherwise nice design. What is the compelling reason? Bytesex isn't it: we should just pick one or the other and stick with it as we do in Ext2.
Why don't we fix cpio to write a consistent bytesex?
--
Daniel
Quoted 5 lines · Daniel Phillips
Encoding the numeric fields in ASCII/hex is a goofy wart on an otherwise nice design. What is the compelling reason? Bytesex isn't it: we should just pick one or the other and stick with it as we do in Ext2.
Why don't we fix cpio to write a consistent bytesex?
Because we want to use existing tools. It's a wart, but not compelling enough of one to rewrite the tools from scratch. (I would also change the EOA marker from "TRAILER!!!" to "" since a null filename would not interfere with the namespace.)
I don't think think this application alone is enough to add Yet Another Version of CPIO. However, if there are more compelling reasons to do so
for CPIO backup reasons itself I guess we could write it up and add it
to GNU cpio as "linux" format...
-hpa
Quoted 4 lines · H. Peter Anvin
Quoted 2 lines · Daniel Phillips
Encoding the numeric fields in ASCII/hex is a goofy wart on an otherwise
nice
Quoted 8 lines
Quoted 5 lines
design. What is the compelling reason? Bytesex isn't it: we should just pick one or the other and stick with it as we do in Ext2.
Why don't we fix cpio to write a consistent bytesex?
Because we want to use existing tools.
It's a mistake not to fix this tool. I'll post the cost in terms of bytes wasted shortly, pretty tough to argue with that, right?
Quoted 2 lines
It's a wart, but not compelling
enough of one to rewrite the tools from scratch.
Why would you rewrite from scratch?
Quoted 3 lines
(I would also change
the EOA marker from "TRAILER!!!" to "" since a null filename would not interfere with the namespace.)
Yes!
Quoted 4 lines
I don't think think this application alone is enough to add Yet Another Version of CPIO. However, if there are more compelling reasons to do so
for CPIO backup reasons itself I guess we could write it up and add it
to GNU cpio as "linux" format...
Oh, it is, really it is. It's not just any application, and GNU already has its own verion of cpio.
--
Daniel
Quoted 2 lines · Daniel Phillips
It's a mistake not to fix this tool. I'll post the cost in terms of bytes wasted shortly, pretty tough to argue with that, right?
No, it's actually very easy: squeezing 40 bytes out of file is not worth _any_ efforts.
Quoted 8 lines · Daniel Phillips
Quoted 4 lines
I don't think think this application alone is enough to add Yet Another Version of CPIO. However, if there are more compelling reasons to do so
for CPIO backup reasons itself I guess we could write it up and add it
to GNU cpio as "linux" format...Oh, it is, really it is. It's not just any application, and GNU already has its own verion of cpio.
But not their own data format.
-hpa
Quoted 6 lines · Daniel Phillips
Quoted 5 lines · H. Peter Anvin
Quoted 4 lines · Daniel Phillips
Encoding the numeric fields in ASCII/hex is a goofy wart on an otherwise nice design. What is the compelling reason? Bytesex isn't it: we should just pick one or the other and stick with it as we do in Ext2.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Why don't we fix cpio to write a consistent bytesex?
Because we want to use existing tools.
It's a mistake not to fix this tool. I'll post the cost in terms of bytes wasted shortly, pretty tough to argue with that, right?
Well, I doubt the difference will be more than a few bytes, if you compare the cpio archive sizes after compression with gzip.
Quoted 7 lines
Quoted 4 lines
I don't think think this application alone is enough to add Yet Another Version of CPIO. However, if there are more compelling reasons to do so
for CPIO backup reasons itself I guess we could write it up and add it
to GNU cpio as "linux" format...Oh, it is, really it is. It's not just any application, and GNU already has its own verion of cpio.
But then every person who wants to build a kernel will have to have the patched version of cpio until such a time it is part of the standard cpio tool (which may be "never"). I would much rather use the currently available tools than save 20 bytes off a 900kB kernel image.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
Quoted 7 lines · H. Peter Anvin
Quoted 6 lines · Daniel Phillips
Quoted 4 lines · You apparently
I don't think think this application alone is enough to add Yet Another Version of CPIO. However, if there are more compelling reasons to do so for CPIO backup reasons itself I guess we could write it up and add it to GNU cpio as "linux" format...
Quoted 6 lines
Quoted 2 lines
Oh, it is, really it is. It's not just any application, and GNU already has its own verion of cpio.
But not their own data format.
From the man page:
"The new ASCII format is portable between
different machine architectures and can be used on any size file system, but is not supported by all versions of cpio; currently, it is only supported by GNU and Unix System V R4."
--
Daniel
Quoted 9 lines · Andreas Dilger
Quoted 8 lines · Daniel Phillips
Quoted 7 lines · H. Peter Anvin
Quoted 5 lines · Daniel Phillips
Encoding the numeric fields in ASCII/hex is a goofy wart on an otherwise nice design. What is the compelling reason? Bytesex isn't it: we should just pick one or the other and stick with it as we do in Ext2.
Why don't we fix cpio to write a consistent bytesex?
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Because we want to use existing tools.
It's a mistake not to fix this tool. I'll post the cost in terms of bytes wasted shortly, pretty tough to argue with that, right?
Well, I doubt the difference will be more than a few bytes, if you compare the cpio archive sizes after compression with gzip.
Coming soon...
Side note: I have a hard time understanding the dual thinking that goes something like: "we have to save every nanosecond of CPU but wasting disk is ok because, um, disk is cheap, and everybody has more than they need anyway, and reading it takes zero time and oh yes, everybody has disks, don't they?"
Quoted 11 lines
Quoted 7 lines
Quoted 4 lines
I don't think think this application alone is enough to add Yet Another Version of CPIO. However, if there are more compelling reasons to do so
for CPIO backup reasons itself I guess we could write it up and add it
to GNU cpio as "linux" format...Oh, it is, really it is. It's not just any application, and GNU already has its own verion of cpio.
But then every person who wants to build a kernel will have to have the patched version of cpio until such a time it is part of the standard cpio tool...
If we go with little-endian then only big-endian architectures will need the patch, and they tend to need patches for lots of things anyway. Or if you like I'll write a little utility that goes through the file and byteswaps all the int fields.
Quoted 2 lines
(which may be "never"). I would much rather use the currently available tools than save 20 bytes off a 900kB kernel image.
What if it's more than 20 bytes?
--
Daniel
Quoted 9 lines · Daniel Phillips
Quoted 2 lines
From the man page:
"The new ASCII format is portable between
different machine architectures and can be used on any size file system, but is not supported by all versions of cpio; currently, it is only supported by GNU and Unix System V R4."
... which, between them, is virtually all Unices these days.
-hpa
Quoted 6 lines · Daniel Phillips
If we go with little-endian then only big-endian architectures will need the patch, and they tend to need patches for lots of things anyway. Or if you like I'll write a little utility that goes through the file and byteswaps all the int fields.
HUH?????????????????
-hpa
Quoted 8 lines · Daniel Phillips
Quoted 2 lines · Andreas Dilger
Well, I doubt the difference will be more than a few bytes, if you compare the cpio archive sizes after compression with gzip.
Side note: I have a hard time understanding the dual thinking that goes something like: "we have to save every nanosecond of CPU but wasting disk is ok because, um, disk is cheap, and everybody has more than they need anyway, and reading it takes zero time and oh yes, everybody has disks, don't they?"
OK, I agree somewhat that we need to save disk space, just as I agree we should reduce CPU usage. That said, would you want to save a few CPU cycles if (for example) it meant we didn't use the ELF binary format, and had to change? Yes, we went from a.out to ELF, but it was a major pain even when Linux was far less widely used.
Quoted 8 lines
Quoted 3 lines
But then every person who wants to build a kernel will have to have the patched version of cpio until such a time it is part of the standard cpio tool...
If we go with little-endian then only big-endian architectures will need the patch, and they tend to need patches for lots of things anyway. Or if you like I'll write a little utility that goes through the file and byteswaps all the int fields.
But the proposed cpio format (AFAIK) has ASCII numbers, which is what you were originally complaining about. I see that cpio(1) says that "by default, cpio creates binary format archives... and can read archives created on machines with a different byte-order".
Excluding alignment issues (which can also be handled relatively easily), is there a reason why we chose the ASCII format over binary, especially since the binary format _appears_ to be portable (assuming endian conversions at decoding time), despite warnings to the contrary?
Quoted 4 lines
Quoted 2 lines
(which may be "never"). I would much rather use the currently available tools than save 20 bytes off a 900kB kernel image.
What if it's more than 20 bytes?
Well, anything less than half a sector (or a network packet) isn't really measurable.
Well, a few quick tests show (GNU cpio version 2.4.2), with raw sizes in "blocks" as output by cpio, compressed sizes in bytes:
find <dir> | cpio -o -H <format> | gzip -9 | wc -c
dir bin (default) newc (proposed)
raw gzip raw gzip
/sbin 15121 3289678 12952 2769451
/etc 8822 689517 8996 693700/usr/local/sbin 1895 385461 1899 385764
The binary format reports lots of "truncating inode number", but for the purpose of initramfs, that is not an issue as we don't anticipate more than 64k files. I don't know why the /sbin test is so heavily in favour of the newc (ASCII) format, but I repeated it to confirm the numbers.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
Quoted 11 lines · Andreas Dilger
But the proposed cpio format (AFAIK) has ASCII numbers, which is what you were originally complaining about. I see that cpio(1) says that "by default, cpio creates binary format archives... and can read archives created on machines with a different byte-order".
Excluding alignment issues (which can also be handled relatively easily), is there a reason why we chose the ASCII format over binary, especially since the binary format _appears_ to be portable (assuming endian conversions at decoding time), despite warnings to the contrary?
The "binary" format of cpio is *ancient*. There is no binary equivalent to the "newc" (SVR4) format.
Quoted 5 lines
The binary format reports lots of "truncating inode number", but for the purpose of initramfs, that is not an issue as we don't anticipate more than 64k files. I don't know why the /sbin test is so heavily in favour of the newc (ASCII) format, but I repeated it to confirm the numbers.
There are way too many other problems with the ancient cpio formats. Not an option.
-hpa
Quoted 12 lines · Andreas Dilger
Quoted 1 line
Well, a few quick tests show (GNU cpio version 2.4.2), with raw sizes in "blocks" as output by cpio, compressed sizes in bytes:
find <dir> | cpio -o -H <format> | gzip -9 | wc -c
dir bin (default) newc (proposed)
raw gzip raw gzip
/sbin 15121 3289678 12952 2769451
/etc 8822 689517 8996 693700/usr/local/sbin1895 385461 1899 385764
Quoted 6 lines
The binary format reports lots of "truncating inode number", but for the purpose of initramfs, that is not an issue as we don't anticipate more than 64k files. I don't know why the /sbin test is so heavily in favour of the newc (ASCII) format, but I repeated it to confirm the numbers.
Probably because it does hard links.
-hpa
Quoted 11 lines · Alexander Viro
Quoted 7 lines · Eric W. Biederman
Quoted 2 lines · H. Peter Anvin
This is an update to the initramfs buffer format spec I posted earlier. The changes are as follows:
Comments. Endian issues are not specified, is the data little, big or vax endian?
Data is what you put into files, byte-by-byte. Headers are ASCII.
In a perfect world we would settle of one of big or little-endian and byte-swap as appropriate, as we do with, e.g., Ext2 filesystems. However it seems that cpio in its current form has no concept of byte-swapping. Cpio(1) can neither generate nor decode a cpio file in the 'foreign' byte sex. So if we are determined to use cpio as it stands, then we are stuck with the goofy ASCII encoding, does that sum up the situation?
Too bad about that, otherwise cpio seems quite reasonable.
I just can't get over those ascii encoding though, and I can't shake the feeling that relying on never having a file named TRAILER!!! is strange. It's gratuitous pollution of the namespace.
What was the reason for going with cpio again - so we can use standard tools? How hard would it be to fix cpio to get rid of the warts? What would we break? Is the problem that we would have to, ugh, go into user space or, eww, cooperate with non-kernel developers?
--
Daniel
Quoted 8 lines · Daniel Phillips
In a perfect world we would settle of one of big or little-endian and byte-swap as appropriate, as we do with, e.g., Ext2 filesystems. However it seems that cpio in its current form has no concept of byte-swapping. Cpio(1) can neither generate nor decode a cpio file in the 'foreign' byte sex. So if we are determined to use cpio as it stands, then we are stuck with the goofy ASCII encoding, does that sum up the situation?
Too bad about that, otherwise cpio seems quite reasonable.
I have to go back and look, isn't -Hcrc endian-neutral?
Signature · 3 lines
bill davidsen <[email protected]> CTO, TMR Associates, Inc Doing interesting things with little computers since 1979.
Quoted 11 lines · Alexander Viro
Quoted 7 lines · Eric W. Biederman
Quoted 2 lines · H. Peter Anvin
This is an update to the initramfs buffer format spec I posted earlier. The changes are as follows:
Comments. Endian issues are not specified, is the data little, big or vax endian?
Data is what you put into files, byte-by-byte. Headers are ASCII.
Is there a problem with the available tools, are they not capable of generating the binary version of the headers?
--
Daniel
Quoted 11 lines · Alexander Viro
Quoted 7 lines · Eric W. Biederman
Quoted 2 lines · H. Peter Anvin
This is an update to the initramfs buffer format spec I posted earlier. The changes are as follows:
Comments. Endian issues are not specified, is the data little, big or vax endian?
Data is what you put into files, byte-by-byte. Headers are ASCII.
Is there a problem with the available tools, are they not capable of generating the binary version of the headers?
--
Daniel
[offtopic] duplicate mails (was: initramfs buffer spec -- second draft)
In reply to Alexander ViroQuoted 1 line · Daniel Phillips
[duplicate stuff]
Sorry about the duplicates, this is some initiative that exim - running on my laptop - has decided, on its own, to undertake, I am research int. It has something to do with mails, apparently with some CC's undelivered, hanging around in the queue and runq sending duplicates of already-delievered copies.
For example, right now I have this hanging around in the queue, and I will eventually do a runq:
58h 1.4K 16Q3MS-0000lx-00 <[email protected]>
[email protected]
D [email protected]
D [email protected]
But now I'm worried exim will send to andrea and lkml again. Hmm.
Is this a bug?
--
Daniel
Quoted 7 lines · Daniel Phillips
Quoted 3 lines
The structure of the
cpio_headeris as follows (all 8-byte entries contain 32-bit hexadecimal ASCII numbers):I thought there's a binary version of the cpio header. What is the point of the ascii encoding?
Byte order independence. The binary version of cpio is ancient and obsolete. Unfortunately the SysV people didn't have the htons() etc macros of BSD, so they had no concept of portable binary formats.
Quoted 11 lines
Quoted 8 lines
The
c_modefield matches the contents ofst_modereturned by stat(2) on Linux, and encodes the file type and file permissions.The
c_filesizeshould be zero for any non-regular file.If the filename is "TRAILER!!!" this is actually an end-of-file marker; the
c_filesizefor an end-of-file marker must be zero.It sure looks ugly, but I suppose the c_filesize=zero is the real end-of-file marker. Did I mention it sure looks ugly?
c_filesize == 0 does *NOT* imply a end-of-archive marker. It is the filename "TRAILER!!!" that does. And yes, it's ugly.
-hpa
Quoted 4 lines · Eric W. Biederman
Comments. Endian issues are not specified, is the data little, big or vax endian?
Not applicable. There are no endian-specific binary structure in the format AT ALL. ASCII-coded fields are always bigendian.
Quoted 3 lines
What is the point of alignment? If the data starts as 4 byte aligned, the 6 byte magic string guarantees the data will be only 2 byte aligned. This isn't good for 32 or 64 bit architectures.
They're ASCII-coded, so it supposedly doesn't matter (yet, it's a bit daft, but blame the SysV people.) The alignment makes sure the *data* field is 4-byte aligned.
Quoted 2 lines
I do like having a
c_magicthat at least allows us to change things in the future if necessary.
It's pretty clear from a lot of the comments that a number of people haven't understood that the cpio encapsulation *THIS IS A CODIFICATION OF AN EXISTING FORMAT.*
-hpa
Followup to: <[email protected]> By author: [email protected] (Kai Henningsen) In newsgroup: linux.dev.kernel
Quoted 8 lines
The latest existing formal spec is probably POSIX 2001 (look under "pax"). An older POSIX version would have it under "cpio". You'll probably also find it there in Unix98 a.k.a. SuSv2. (POSIX 2001 (the Austin revision) supersedes all of those.)
It's a bit long to post here - probably exceeds fair use.
POSIX only specifies the "old ASCII" cpio format anyway, which is so limited as to be useless. POSIX specifies also specify "ustar" and "pax", two extended tar formats, neither of which is suitable for this application.
-hpa
Signature · 3 lines
<[email protected]> at work, <[email protected]> in private! "Unix gives you enough rope to shoot yourself in the foot." http://www.zytor.com/~hpa/puzzle.txt <[email protected]>
Quoted 9 lines · H. Peter Anvin
Quoted 3 lines · Eric W. Biederman
Comments. Endian issues are not specified, is the data little, big or vax endian?
Not applicable. There are no endian-specific binary structure in the format AT ALL. ASCII-coded fields are always bigendian.
O.k. Thanks, I missed that part. I just looked back and it is clear that there are 32 bit values encoded in hexadecimal. And I admit the bigendian (human readable) is strongly implied from the context.
Quoted 8 lines
Quoted 3 lines
What is the point of alignment? If the data starts as 4 byte aligned, the 6 byte magic string guarantees the data will be only 2 byte aligned. This isn't good for 32 or 64 bit architectures.
They're ASCII-coded, so it supposedly doesn't matter (yet, it's a bit daft, but blame the SysV people.) The alignment makes sure the *data* field is 4-byte aligned.
O.k. So the we have a bit of implied padding after the filename. And it is necessary to preserve this padding or we break with the prexisting format definition. You don't gain much with that as being 4 byte aligned on 64bit architectures, is not fully aligned.
Quoted 7 lines
Quoted 2 lines
I do like having a
c_magicthat at least allows us to change things in the future if necessary.It's pretty clear from a lot of the comments that a number of people haven't understood that the cpio encapsulation *THIS IS A CODIFICATION OF AN EXISTING FORMAT.*
Which we are reusing for a different purpose. And because of that we become trustees of our version of the format. To make it clear that someone else defines how this format works a reference to the appropriate specification is called for.
I admit I did a quick search earlier and I did not find this format specified, elsewhere.
The cases where initramfs will be used are some of the most operating specific cases I can imagine. To handle those cases it is necessary to support the full breadth of the capability of the operating system. So if initramfs is going to survive todays implementation of the linux kernel, or possibly be portable to other operating systems we must have an extensible format. It appears c_magic gives us that extensibility.
Eric
Quoted 4 lines · Eric W. Biederman
Which we are reusing for a different purpose. And because of that we become trustees of our version of the format. To make it clear that someone else defines how this format works a reference to the appropriate specification is called for.
We are using it for precisely the same purpose - to put a bunch of files on a filesystem.
Quoted 3 lines
The cases where initramfs will be used are some of the most operating specific cases I can imagine. To handle those cases it is necessary to support the full breadth of the capability of the operating system.
Huh? It's a bloody archive - collection of files and nothing else. What "capability of the operating system"?
Quoted 9 lines · Alexander Viro
Quoted 4 lines · Eric W. Biederman
Which we are reusing for a different purpose. And because of that we become trustees of our version of the format. To make it clear that someone else defines how this format works a reference to the appropriate specification is called for.
We are using it for precisely the same purpose - to put a bunch of files on a filesystem.
Anytime you are specifying semantics beyond what was in the original specification it isn't precisely the same case. Close enough not to matter yes but not precisely the same. The original cpio format does not specify compression or concatenation of images. It is not mandated that the cpio format handle the needs of everyones root filesystem.
Additionally we now have the potential of generating cpio files from the bootloaders. And bootloaders should be the kinds of programs that don't need constant maintenance or upgrading, (that is very destabilizing). So totally reworking the format is not a solution when we need to change something. Even if is ok for cpio in general.
This changing the format in incompatible ways when there is a new requirement does seem to be the traditional cpio method.
Quoted 6 lines
Quoted 3 lines
The cases where initramfs will be used are some of the most operating specific cases I can imagine. To handle those cases it is necessary to support the full breadth of the capability of the operating system.
Huh? It's a bloody archive - collection of files and nothing else. What "capability of the operating system"?
Exactly. But the standard unix stream of bytes does not cover everyones concept of files. Things like:
Symbolic Links
Device Nodes,
Resource Forks,
Device links,
Persistent mount points,
ACL's,
Persistent capabilities,
Are all partial exceptions to everything is the same kind of file. The cpio format as is doesn't handle all of these which is fine, but we may need some of these later, so we need someplace to expand to when if/when these kinds of things become important.
The startup process is likely to need everything the operating system can do, to handle some special case or the other. So if at some future date we support odd types of special files we will probably need to use them in the system startup code. We already require device nodes, and find symbolic links very helpful.
Further Linux is dynamic and always changing, so not having some elbow room for growth is just asking for trouble. All I noted is that the c_magic field exists so if/when the need arises we can handle really strange cases. With everyone in linux being able to use an initramfs as their root filesystem actually makes the odds of a change that requires special root filesystem support much more likely. Because you only have to change one filesystem.
All I am asking is two things. If we are not assuming guardianship for our variant of the cpio format we should reference those who do have guardianship, in the specification. We should be aware that the cpio format as it now exists may not handle all future needs so having a mechanism to extend the format when those needs arise without breaking all existing users is important.
Eric
[email protected] (Eric W. Biederman) wrote on 13.01.02 in <[email protected]>:
Quoted 2 lines
I admit I did a quick search earlier and I did not find this format specified, elsewhere.
The latest existing formal spec is probably POSIX 2001 (look under "pax"). An older POSIX version would have it under "cpio". You'll probably also find it there in Unix98 a.k.a. SuSv2. (POSIX 2001 (the Austin revision) supersedes all of those.)
It's a bit long to post here - probably exceeds fair use.
MfG Kai
Quoted 2 lines · Eric W. Biederman
Comments. Endian issues are not specified, is the data little, big or vax endian?
VAX is little endian.
Perhaps you mean PDP11?