Skip to contentSkip to conversation

initramfs: "crc" cpio format and INITRAMFS_PRESERVE_MTIME

PATCH Superseded David Disseldorp 3 participants 8 messages Nov 10, 2021

Subsystems (2)

v4 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v4 cover initramfs: "crc" cpio format and INITRAMFS_PRESERVE_MTIME

This patchset does some minor refactoring of cpio header magic checking and corrects documentation.
Patch 4/4 allows cpio entry mtime preservation to be disabled via a new INITRAMFS_PRESERVE_MTIME Kconfig option.

Changes since v3, following feedback from Martin Wilck:
- 4/4: keep vfs_utimes() call in do_copy() path
+ drop [PATCH v3 4/5] initramfs: use do_utime() wrapper consistently
+ add do_utime_path() helper
+ clean up timespec64 initialisation
- 4/4: move all mtime preservation logic to initramfs_mtime.h and drop separate .c
- 4/4: improve commit message

 .../early-userspace/buffer-format.rst         | 24 +++-----
 init/Kconfig                                  | 10 ++++
 init/initramfs.c                              | 57 +++----------------
 init/initramfs_mtime.h                        | 50 ++++++++++++++++
 4 files changed, 75 insertions(+), 66 deletions(-)
#1 lore raw

PATCH v4 1/4 initramfs: refactor do_header() cpio magic checks

do_header() is called for each cpio entry and fails if the first six bytes don't match "newc" magic. The magic check includes a special case error message if POSIX.1 ASCII (cpio -H odc) magic is detected. This special case POSIX.1 check can be nested under the "newc" mismatch code path to avoid calling memcmp() twice in a non-error case.

---
 init/initramfs.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
init/initramfs.c +4 -5
index 2f3d96dc3db6..2f79b3ec0b40 100644
@@ -257,12 +257,11 @@ static int __init do_collect(void)
 
 static int __init do_header(void)
 {
-	if (memcmp(collected, "070707", 6)==0) {
-		error("incorrect cpio method used: use -H newc option");
-		return 1;
-	}
 	if (memcmp(collected, "070701", 6)) {
-		error("no cpio magic");
+		if (memcmp(collected, "070707", 6) == 0)
+			error("incorrect cpio method used: use -H newc option");
+		else
+			error("no cpio magic");
 		return 1;
 	}
 	parse_header(collected);
Signature · 1 line
2.31.1
#2 lore raw

PATCH v4 2/4 initramfs: print helpful cpio error on "crc" magic

In reply to David Disseldorp

Contrary to the buffer-format.rst documentation, initramfs cpio extraction does not support "crc" archives, which carry "070702" header magic. Make it a little clearer that "newc" (magic="070701") is the only supported cpio format, by extending the POSIX.1 ASCII (magic="070707") specific error message to also cover "crc" magic.

---
 init/initramfs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
init/initramfs.c +1 -1
index 2f79b3ec0b40..44e692ae4646 100644
@@ -258,7 +258,7 @@ static int __init do_collect(void)
 static int __init do_header(void)
 {
 	if (memcmp(collected, "070701", 6)) {
-		if (memcmp(collected, "070707", 6) == 0)
+		if (memcmp(collected, "0707", 4) == 0)
 			error("incorrect cpio method used: use -H newc option");
 		else
 			error("no cpio magic");
Signature · 1 line
2.31.1
#3 lore raw

Re: [PATCH v4 2/4] initramfs: print helpful cpio error on "crc" magic

Quoted 5 lines · David Disseldorp

Contrary to the buffer-format.rst documentation, initramfs cpio extraction does not support "crc" archives, which carry "070702" header magic. Make it a little clearer that "newc" (magic="070701") is the only supported cpio format, by extending the POSIX.1 ASCII (magic="070707") specific error message to also cover "crc" magic.

Wouldn't it be easier to just add support? As far as I can tell from looking at documentation, the "crc" format is the same as newc, except that it uses some reserved bits to store the crc. Since we ignore those bits, we could just check for either 070701 or 070702.

Quoted 21 lines
---
 init/initramfs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
init/initramfs.c +1 -1
index 2f79b3ec0b40..44e692ae4646 100644
@@ -258,7 +258,7 @@ static int __init do_collect(void)
 static int __init do_header(void)
 {
 	if (memcmp(collected, "070701", 6)) {
-		if (memcmp(collected, "070707", 6) == 0)
+		if (memcmp(collected, "0707", 4) == 0)
 			error("incorrect cpio method used: use -H newc option");
 		else
 			error("no cpio magic");
Signature · 1 line
2.31.1
#4 lore raw

Re: [PATCH v4 2/4] initramfs: print helpful cpio error on "crc" magic

Quoted 8 lines · Matthew Wilcox
Quoted 5 lines · David Disseldorp

Contrary to the buffer-format.rst documentation, initramfs cpio extraction does not support "crc" archives, which carry "070702" header magic. Make it a little clearer that "newc" (magic="070701") is the only supported cpio format, by extending the POSIX.1 ASCII (magic="070707") specific error message to also cover "crc" magic.

Wouldn't it be easier to just add support?

Well, no, this patch already exists. :-)

Quoted 4 lines

As far as I can tell from
looking at documentation, the "crc" format is the same as newc, except that it uses some reserved bits to store the crc. Since we ignore those bits, we could just check for either 070701 or 070702.

Sure, it'd be pretty straightforward to implement "crc" format support, but I'm not sure how useful a 32-bit checksum would be... If we're going down this route, wouldn't proper IMA/EVM support make sense via some new cpio variant with space for the attributes (or as header/trailer like bootconfig)?
cc'ing Jeff, as I seem to recall him mentioning some work in this area.

Cheers, David

#5 lore raw

PATCH v4 3/4 docs: remove mention of "crc" cpio format support

In reply to David Disseldorp

init/initramfs.c only supports extraction of cpio archives carrying the "newc" header magic ("070701"). Remove statements indicating support for the "crc" cpio format.

---
 .../early-userspace/buffer-format.rst         | 24 +++++++------------
 1 file changed, 8 insertions(+), 16 deletions(-)
Documentation/driver-api/early-userspace/buffer-format.rst +8 -16
index 7f74e301fdf3..0df76bca444c 100644
@@ -14,10 +14,10 @@ is different.  The initramfs buffer contains an archive which is
 expanded into a ramfs filesystem; this document details the format of
 the initramfs buffer format.
 
-The initramfs buffer format is based around the "newc" or "crc" CPIO
-formats, and can be created with the cpio(1) utility.  The cpio
-archive can be compressed using gzip(1).  One valid version of an
-initramfs buffer is thus a single .cpio.gz file.
+The initramfs buffer format is based around the "newc" CPIO format, and
+can be created with the cpio(1) utility.  The cpio archive can be
+compressed using gzip(1).  One valid version of an initramfs buffer is
+thus a single .cpio.gz file.
 
 The full format of the initramfs buffer is defined by the following
 grammar, where::
@@ -40,9 +40,8 @@ grammar, where::
 
 
 In human terms, the initramfs buffer contains a collection of
-compressed and/or uncompressed cpio archives (in the "newc" or "crc"
-formats); arbitrary amounts zero bytes (for padding) can be added
-between members.
+compressed and/or uncompressed cpio archives (in the "newc" format),
+with arbitrary amount of zero-byte padding between members.
 
 The cpio "TRAILER!!!" entry (cpio end-of-archive) is optional, but is
 not ignored; see "handling of hard links" below.
@@ -55,7 +54,7 @@ by the ASCII string "000012ac"):
 ============= ================== ==============================================
 Field name    Field size	 Meaning
 ============= ================== ==============================================
-c_magic	      6 bytes		 The string "070701" or "070702"
+c_magic	      6 bytes		 The string "070701"
 c_ino	      8 bytes		 File inode number
 c_mode	      8 bytes		 File mode and permissions
 c_uid	      8 bytes		 File uid
@@ -68,8 +67,7 @@ c_min	      8 bytes		 Minor part of file device number
 c_rmaj	      8 bytes		 Major part of device node reference
 c_rmin	      8 bytes		 Minor part of device node reference
 c_namesize    8 bytes		 Length of filename, including final \0
-c_chksum      8 bytes		 Checksum of data field if c_magic is 070702;
-				 otherwise zero
+c_chksum      8 bytes		 Ignored; reserved for unsupported "crc" format
 ============= ================== ==============================================
 
 The c_mode field matches the contents of st_mode returned by stat(2)
@@ -78,12 +76,6 @@ on Linux, and encodes the file type and file permissions.
 The c_filesize should be zero for any file which is not a regular file
 or symlink.
 
-The c_chksum field contains a simple 32-bit unsigned sum of all the
-bytes in the data field.  cpio(1) refers to this as "crc", which is
-clearly incorrect (a cyclic redundancy check is a different and
-significantly stronger integrity check), however, this is the
-algorithm used.
-
 If the filename is "TRAILER!!!" this is actually an end-of-archive
 marker; the c_filesize for an end-of-archive marker must be zero.
 
Signature · 1 line
2.31.1
#6 lore raw

PATCH v4 4/4 initramfs: add INITRAMFS_PRESERVE_MTIME Kconfig option

In reply to David Disseldorp

initramfs cpio mtime preservation, as implemented in commit 889d51a10712 ("initramfs: add option to preserve mtime from initramfs cpio images"), uses a linked list to defer directory mtime processing until after all other items in the cpio archive have been processed. This is done to ensure that parent directory mtimes aren't overwritten via subsequent child creation. Contrary to the 889d51a10712 commit message, the mtime preservation behaviour is unconditional.

This change adds a new INITRAMFS_PRESERVE_MTIME Kconfig option, which can be used to disable on-by-default mtime retention and in turn speed up initramfs extraction, particularly for cpio archives with large directory counts.

Benchmarks with a one million directory cpio archive extracted 20 times demonstrated:
mean extraction time (s) std dev
INITRAMFS_PRESERVE_MTIME=y 3.808 0.006
INITRAMFS_PRESERVE_MTIME unset 3.056 0.004

The above extraction times were measured using ftrace (initcall_finish - initcall_start) values for populate_rootfs() with initramfs_async disabled.

---
 init/Kconfig           | 10 +++++++++
 init/initramfs.c       | 48 +++-------------------------------------
 init/initramfs_mtime.h | 50 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 63 insertions(+), 45 deletions(-)
 create mode 100644 init/initramfs_mtime.h
init/Kconfig +10 -0
index 21b1f4870c80..6c2b919f81a5 100644
@@ -1352,6 +1352,16 @@ config BOOT_CONFIG
 
 	  If unsure, say Y.
 
+config INITRAMFS_PRESERVE_MTIME
+	bool "Preserve cpio archive mtimes in initramfs"
+	default y
+	help
+	  Each entry in an initramfs cpio archive carries an mtime value. When
+	  enabled, extracted cpio items take this mtime, with directory mtime
+	  setting deferred until after creation of any child entries.
+
+	  If unsure, say Y.
+
 choice
 	prompt "Compiler optimization level"
 	default CC_OPTIMIZE_FOR_PERFORMANCE
init/initramfs.c +3 -45

69 lines not on this page · raw message

init/initramfs_mtime.h new file +50 -0

50 lines not on this page · raw message

Signature · 1 line
2.31.1
Reviewed-by on the v4 series
#7 lore raw
In reply to David Disseldorp
Quoted 18 lines · David Disseldorp

This patchset does some minor refactoring of cpio header magic checking
and corrects documentation.
Patch 4/4 allows cpio entry mtime preservation to be disabled via a new
INITRAMFS_PRESERVE_MTIME Kconfig option.

Changes since v3, following feedback from Martin Wilck:
- 4/4: keep vfs_utimes() call in do_copy() path
  + drop [PATCH v3 4/5] initramfs: use do_utime() wrapper consistently
  + add do_utime_path() helper
  + clean up timespec64 initialisation
- 4/4: move all mtime preservation logic to initramfs_mtime.h and drop
  separate .c
- 4/4: improve commit message

Quoted 7 lines

 .../early-userspace/buffer-format.rst         | 24 +++----- init/Kconfig                                  | 10 ++++ init/initramfs.c                              | 57 +++-------------- --
 init/initramfs_mtime.h                        | 50 ++++++++++++++++

 4 files changed, 75 insertions(+), 66 deletions(-)

For the set: