initramfs: "crc" cpio format and INITRAMFS_PRESERVE_MTIME
This patchset does some minor initramfs refactoring and allows cpio entry mtime preservation to be disabled via a new Kconfig INITRAMFS_PRESERVE_MTIME option.
Patches 3/5 to 5/5 implement support for creation and extraction of "crc" cpio archives, which carry file data checksums. Basic tests for this functionality can be found at
Changes since v4, following feedback from Matthew Wilcox:
- implement cpio "crc" archive creation and extraction
- add patch to fix gen_init_cpio short read handling
- drop now-unnecessary "crc" documentation and error msg changes
Changes since v3, following feedback from Martin Wilck:
- 4/4: keep vfs_utimes() call in do_copy() path
+ drop [PATCH v3 4/5] initramfs: use do_utime() wrapper consistently
+ add do_utime_path() helper
+ clean up timespec64 initialisation
- 4/4: move all mtime preservation logic to initramfs_mtime.h and drop separate .c
- 4/4: improve commit message
init/Kconfig | 10 +++++ init/initramfs.c | 89 +++++++++++++++------------------------- init/initramfs_mtime.h | 50 +++++++++++++++++++++++ usr/gen_init_cpio.c | 92 ++++++++++++++++++++++++++++++------------ 4 files changed, 159 insertions(+), 82 deletions(-)
PATCH v5 1/5 initramfs: refactor do_header() cpio magic checks
do_header() is called for each cpio entry and fails if the first six bytes don't match "newc" magic. The magic check includes a special case error message if POSIX.1 ASCII (cpio -H odc) magic is detected. This special case POSIX.1 check can be nested under the "newc" mismatch code path to avoid calling memcmp() twice in a non-error case.
--- init/initramfs.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
init/initramfs.c +4 -5
@@ -257,12 +257,11 @@ static int __init do_collect(void)static int __init do_header (void ) { -if (memcmp (collected,"070707" ,6 )==0 ) { -error ("incorrect cpio method used: use -H newc option" ); -return 1 ; - }if (memcmp (collected,"070701" ,6 )) { -error ("no cpio magic" ); +if (memcmp (collected,"070707" ,6 ) ==0 ) +error ("incorrect cpio method used: use -H newc option" ); +else +error ("no cpio magic" );return 1 ; }parse_header (collected);
Signature · 1 line
2.31.1
PATCH v5 2/5 initramfs: add INITRAMFS_PRESERVE_MTIME Kconfig option
In reply to David Disseldorpinitramfs cpio mtime preservation, as implemented in commit 889d51a10712 ("initramfs: add option to preserve mtime from initramfs cpio images"), uses a linked list to defer directory mtime processing until after all other items in the cpio archive have been processed. This is done to ensure that parent directory mtimes aren't overwritten via subsequent child creation. Contrary to the 889d51a10712 commit message, the mtime preservation behaviour is unconditional.
This change adds a new INITRAMFS_PRESERVE_MTIME Kconfig option, which can be used to disable on-by-default mtime retention and in turn speed up initramfs extraction, particularly for cpio archives with large directory counts.
Benchmarks with a one million directory cpio archive extracted 20 times demonstrated:
mean extraction time (s) std dev
INITRAMFS_PRESERVE_MTIME=y 3.808 0.006INITRAMFS_PRESERVE_MTIME unset 3.056 0.004
The above extraction times were measured using ftrace (initcall_finish - initcall_start) values for populate_rootfs() with initramfs_async disabled.
--- init/Kconfig | 10 +++++++++ init/initramfs.c | 48 +++------------------------------------- init/initramfs_mtime.h | 50 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 63 insertions(+), 45 deletions(-) create mode 100644 init/initramfs_mtime.h
init/Kconfig +10 -0
@@ -1357,6 +1357,16 @@ config BOOT_CONFIG If unsure, say Y. +config INITRAMFS_PRESERVE_MTIME + bool "Preserve cpio archive mtimes in initramfs" + default y + help + Each entry in an initramfs cpio archive carries an mtime value. When + enabled, extracted cpio items take this mtime, with directory mtime + setting deferred until after creation of any child entries. + + If unsure, say Y. + choice prompt "Compiler optimization level" default CC_OPTIMIZE_FOR_PERFORMANCE
init/initramfs.c +3 -45
69 lines not on this page · raw message
init/initramfs_mtime.h new file +50 -0
50 lines not on this page · raw message
Signature · 1 line
2.31.1
Re: [PATCH v5 2/5] initramfs: add INITRAMFS_PRESERVE_MTIME Kconfig option
Quoted 3 lines · David Disseldorp
init/initramfs.c +0
-0@@ -17,6 +17,8 @@
...
Quoted 10 lines
init/initramfs.c +0
-10-static void __init dir_add (const char *name,time64_t mtime) -{ -struct dir_entry *de =kmalloc (sizeof (struct dir_entry ),GFP_KERNEL ); -if (!de) -panic_show_mem ("can't allocate dir_entry buffer" ); -INIT_LIST_HEAD (&de->list); - de->name =kstrdup (name,GFP_KERNEL ); - de->mtime = mtime; -list_add (&de->list, &dir_list); -}
...
Quoted 2 lines
init/initramfs_mtime.h new file +0
-0
...
init/initramfs_mtime.h line 32
Quoted 10 lines
init/initramfs_mtime.h new file +10
-0+static void __init dir_add (const char *name,time64_t mtime) +{ +struct dir_entry *de =kmalloc (sizeof (struct dir_entry ),GFP_KERNEL ); +if (!de) +panic ("can't allocate dir_entry buffer" ); +INIT_LIST_HEAD (&de->list); + de->name =kstrdup (name,GFP_KERNEL ); + de->mtime = mtime; +list_add (&de->list, &dir_list); +}
I might as well fix the unhandled kstrdup() failure, rather than copying it here. I'll post another round which allocates the "name" buffer via the dir_entry kmalloc() call.
PATCH v5 3/5 gen_init_cpio: fix short read file handling
In reply to David DisseldorpWhen processing a "file" entry, gen_init_cpio attempts to allocate a buffer large enough to stage the entire contents of the source file. It then attempts to fill the buffer via a single read() call and subsequently writes out the entire buffer length, without checking that read() returned the full length, potentially writing uninitialized buffer memory.
Fix this by breaking up file I/O into 64k chunks and only writing the length returned by the prior read() call.
--- usr/gen_init_cpio.c | 44 +++++++++++++++++++++++++------------------- 1 file changed, 25 insertions(+), 19 deletions(-)
usr/gen_init_cpio.c +25 -19
@@ -20,6 +20,7 @@#define xstr (s) #s#define str (s) xstr(s) +#define MIN (a, b) ((a) < (b) ? (a) : (b))static unsigned int offset;static unsigned int ino =721 ;
@@ -297,9 +298,8 @@ static int cpio_mkfile(const char *name, const char *location, unsigned int nlinks) {char s[256 ]; -char *filebuf =NULL ;struct stat buf; -long size; +unsigned long size;int file =-1 ;int retval;int rc =-1 ;
@@ -326,22 +326,17 @@ static int cpio_mkfile(const char *name, const char *location, buf.st_mtime =0xffffffff ; } - filebuf =malloc (buf.st_size); -if (!filebuf) { -fprintf (stderr,"out of memory\n" ); - goto error; - } - - retval =read (file, filebuf, buf.st_size); -if (retval <0 ) { -fprintf (stderr,"Can not read %s file\n" , location); +if (buf.st_size >0xffffffff ) { +fprintf (stderr,"%s: Size exceeds maximum cpio file size\n" , + location); goto error; } size =0 ;for (i =1 ; i <= nlinks; i++) {/* data goes on last link */ -if (i == nlinks) size = buf.st_size; +if (i == nlinks) + size = buf.st_size;if (name[0 ] =='/' ) name++;
@@ -366,23 +361,34 @@ static int cpio_mkfile(const char *name, const char *location,push_string (name);push_pad (); -if (size) { -if (fwrite (filebuf, size,1 , stdout) !=1 ) { +while (size) { +unsigned char filebuf[65536 ]; +ssize_t this_read; +size_t this_size =MIN (size,sizeof (filebuf)); + + this_read =read (file, filebuf, this_size); +if (this_read <=0 || this_read > this_size) { +fprintf (stderr,"Can not read %s file\n" , location); + goto error; + } + +if (fwrite (filebuf, this_read,1 , stdout) !=1 ) {fprintf (stderr,"writing filebuf failed\n" ); goto error; } - offset += size; -push_pad (); + offset += this_read; + size -= this_read; } +push_pad (); name += namesize; } ino++; rc =0 ; - + error: -if (filebuf)free (filebuf); -if (file >=0 )close (file); +if (file >=0 ) +close (file);return rc; }
Signature · 1 line
2.31.1
PATCH v5 4/5 gen_init_cpio: support file checksum archiving
In reply to David DisseldorpDocumentation/driver-api/early-userspace/buffer-format.rst includes the specification for crc-enabled cpio archives. Implement support for this format in gen_init_cpio via a new '-c' parameter.
--- usr/gen_init_cpio.c | 54 +++++++++++++++++++++++++++++++++++++-------- 1 file changed, 45 insertions(+), 9 deletions(-)
usr/gen_init_cpio.c +45 -9
@@ -1,6 +1,8 @@// SPDX-License-Identifier: GPL-2.0 #include <stdio.h> #include <stdlib.h> +#include <stdint.h> +#include <stdbool.h> #include <sys/types.h> #include <sys/stat.h> #include <string.h>
@@ -25,6 +27,7 @@static unsigned int offset;static unsigned int ino =721 ;static time_t default_mtime; +static bool do_csum = false;struct file_handler {const char *type;
@@ -78,7 +81,7 @@ static void cpio_trailer(void) sprintf(s, "%s%08X%08X%08lX%08lX%08X%08lX" "%08X%08X%08X%08X%08X%08X%08X", - "070701", /* magic */ + do_csum ? "070702" : "070701", /* magic */ 0, /* ino */ 0, /* mode */ (long) 0, /* uid */
@@ -110,7 +113,7 @@ static int cpio_mkslink(const char *name, const char *target, name++; sprintf(s,"%s%08X%08X%08lX%08lX%08X%08lX" "%08X%08X%08X%08X%08X%08X%08X", - "070701", /* magic */ + do_csum ? "070702" : "070701", /* magic */ ino++, /* ino */ S_IFLNK | mode, /* mode */ (long) uid, /* uid */
@@ -159,7 +162,7 @@ static int cpio_mkgeneric(const char *name, unsigned int mode, name++; sprintf(s,"%s%08X%08X%08lX%08lX%08X%08lX" "%08X%08X%08X%08X%08X%08X%08X", - "070701", /* magic */ + do_csum ? "070702" : "070701", /* magic */ ino++, /* ino */ mode, /* mode */ (long) uid, /* uid */
@@ -253,7 +256,7 @@ static int cpio_mknod(const char *name, unsigned int mode, name++; sprintf(s,"%s%08X%08X%08lX%08lX%08X%08lX" "%08X%08X%08X%08X%08X%08X%08X", - "070701", /* magic */ + do_csum ? "070702" : "070701", /* magic */ ino++, /* ino */ mode, /* mode */ (long) uid, /* uid */
@@ -293,6 +296,29 @@ static int cpio_mknod_line(const char *line)return rc; } +static int cpio_mkfile_csum (int fd,unsigned long size,uint32_t *csum) +{ +while (size) { +unsigned char filebuf[65536 ]; +ssize_t this_read; +size_t i, this_size =MIN (size,sizeof (filebuf)); + + this_read =read (fd, filebuf, this_size); +if (this_read <=0 || this_read > this_size) +return -1 ; + +for (i =0 ; i < this_read; i++) + *csum += filebuf[i]; + + size -= this_read; + } +/* seek back to the start for data segment I/O */ +if (lseek (fd,0 ,SEEK_SET ) <0 ) +return -1 ; + +return 0 ; +} +static int cpio_mkfile (const char *name,const char *location,unsigned int mode,uid_t uid,gid_t gid,unsigned int nlinks)
@@ -305,6 +331,7 @@ static int cpio_mkfile(const char *name, const char *location,int rc =-1 ;int namesize;unsigned int i; +uint32_t csum =0 ; mode |=S_IFREG ;
@@ -332,6 +359,11 @@ static int cpio_mkfile(const char *name, const char *location, goto error; } +if (do_csum &&cpio_mkfile_csum (file, buf.st_size, &csum) <0 ) { +fprintf (stderr,"Failed to checksum file %s\n" , location); + goto error; + } + size =0 ;for (i =1 ; i <= nlinks; i++) {/* data goes on last link */
@@ -343,7 +375,7 @@ static int cpio_mkfile(const char *name, const char *location, namesize =strlen (name) +1 ; sprintf(s,"%s%08X%08X%08lX%08lX%08X%08lX" "%08lX%08X%08X%08X%08X%08X%08X", - "070701", /* magic */ + do_csum ? "070702" : "070701", /* magic */ ino, /* ino */ mode, /* mode */ (long) uid, /* uid */
@@ -356,7 +388,7 @@ static int cpio_mkfile(const char *name, const char *location,0 ,/* rmajor */ 0 ,/* rminor */ namesize,/* namesize */ - 0); /* chksum */ + size ? csum : 0); /* chksum */push_hdr (s);push_string (name);push_pad ();
@@ -464,7 +496,7 @@ static int cpio_mkfile_line(const char *line)static void usage (const char *prog) {fprintf (stderr,"Usage:\n" -"\t%s [-t <timestamp>] <cpio_list>\n" +"\t%s [-t <timestamp>] [-c] <cpio_list>\n" "\n" "<cpio_list> is a file containing newline separated entries that\n" "describe the files to be included in the initramfs archive:\n"
@@ -499,7 +531,8 @@ static void usage(const char *prog)"\n" "<timestamp> is time in seconds since Epoch that will be used\n" "as mtime for symlinks, special files and directories. The default\n" -"is to use the current time for these entries.\n" , +"is to use the current time for these entries.\n" +"-c: calculate and store 32-bit checksums for file data.\n" , prog); }
@@ -541,7 +574,7 @@ int main (int argc, char *argv[]) default_mtime =time (NULL );while (1 ) { -int opt =getopt (argc, argv,"t:h" ); +int opt =getopt (argc, argv,"t:ch" );char *invalid;if (opt ==-1 )
@@ -556,6 +589,9 @@ int main (int argc, char *argv[])exit (1 ); }break ; +case 'c' : + do_csum = true; +break ;case 'h' :case '?' :usage (argv[0 ]);
Signature · 1 line
2.31.1
PATCH v5 5/5 initramfs: support cpio extraction with file checksums
In reply to David DisseldorpAdd support for the extraction of crc-enabled "070702" cpio archives, as specified in Documentation/driver-api/early-userspace/buffer-format.rst. Fail extraction if the calculated file data checksum doesn't match the value carried in the header.
--- init/initramfs.c | 32 +++++++++++++++++++++++++------- 1 file changed, 25 insertions(+), 7 deletions(-)
init/initramfs.c +25 -7
@@ -19,8 +19,11 @@#include "initramfs_mtime.h" -static ssize_t __init xwrite (struct file *file,const char *p,size_t count, -loff_t *pos) +static __initdata bool csum_present; +static __initdata u32 io_csum; + +static ssize_t __init xwrite (struct file *file,const unsigned char *p, +size_t count,loff_t *pos) {ssize_t out =0 ;
@@ -35,6 +38,13 @@ static ssize_t __init xwrite(struct file *file, const char *p, size_t count, }else if (rv ==0 )break ; +if (csum_present) { +ssize_t i; + +for (i =0 ; i < rv; i++) + io_csum += p[i]; + } + p += rv; out += rv; count -= rv;
@@ -118,8 +128,6 @@ static void __init free_hash(void) } } -static __initdata time64_t mtime; -/* cpio header parsing */ static __initdata unsigned long ino, major, minor, nlink;
@@ -128,15 +136,17 @@ static __initdata unsigned long body_len, name_len;static __initdata uid_t uid;static __initdata gid_t gid;static __initdata unsigned rdev; +static __initdata time64_t mtime; +static __initdata u32 hdr_csum;static void __init parse_header (char *s) { -unsigned long parsed[12 ]; +unsigned long parsed[13 ];char buf[9 ];int i; buf[8 ] ='\0' ; -for (i =0 , s +=6 ; i <12 ; i++, s +=8 ) { +for (i =0 , s +=6 ; i <13 ; i++, s +=8 ) {memcpy (buf, s,8 ); parsed[i] =simple_strtoul (buf,NULL ,16 ); }
@@ -151,6 +161,7 @@ static void __init parse_header(char *s) minor = parsed[8 ]; rdev =new_encode_dev (MKDEV (parsed[9 ], parsed[10 ])); name_len = parsed[11 ]; + hdr_csum = parsed[12 ]; }/* FSM */
@@ -219,7 +230,11 @@ static int __init do_collect(void)static int __init do_header (void ) { -if (memcmp (collected,"070701" ,6 )) { +if (!memcmp (collected,"070701" ,6 )) { + csum_present = false; + }else if (!memcmp (collected,"070702" ,6 )) { + csum_present = true; + }else {if (memcmp (collected,"070707" ,6 ) ==0 )error ("incorrect cpio method used: use -H newc option" );else
@@ -314,6 +329,7 @@ static int __init do_name(void)if (IS_ERR (wfile))return 0 ; wfile_pos =0 ; + io_csum =0 ;vfs_fchown (wfile, uid, gid);vfs_fchmod (wfile, mode);
@@ -346,6 +362,8 @@ static int __init do_copy(void)do_utime_path (&wfile->f_path, mtime);fput (wfile); +if (csum_present && io_csum != hdr_csum) +error ("bad data checksum" );eat (body_len); state = SkipIt;return 0 ;
Signature · 1 line
2.31.1
Quoted 6 lines · David Disseldorp
This patchset does some minor initramfs refactoring and allows cpio entry mtime preservation to be disabled via a new
KconfigINITRAMFS_PRESERVE_MTIMEoption.
Patches 3/5 to 5/5 implement support for creation and extraction of "crc" cpio archives, which carry file data checksums. Basic tests for this functionality can be found at
Quoted 6 lines
Changes since v4, following feedback from Matthew Wilcox:
- implement cpio "crc" archive creation and extraction
- add patch to fixgen_init_cpioshort read handling
- drop now-unnecessary "crc" documentation and error msg changes
Ping, any feedback on this round?
Thanks, David
Quoted 15 lines
Changes since v3, following feedback from Martin Wilck:
- 4/4: keepvfs_utimes()call indo_copy()path
+ drop [PATCH v3 4/5] initramfs: usedo_utime()wrapper consistently
+ adddo_utime_path()helper
+ clean up timespec64 initialisation
- 4/4: move all mtime preservation logic toinitramfs_mtime.hand drop separate .c
- 4/4: improve commit messageinit/Kconfig | 10 +++++ init/initramfs.c | 89 +++++++++++++++------------------------- init/initramfs_mtime.h | 50 +++++++++++++++++++++++ usr/gen_init_cpio.c | 92 ++++++++++++++++++++++++++++++------------ 4 files changed, 159 insertions(+), 82 deletions(-)