mkfs: add ability to populate filesystem from directory
v11 cover mkfs: add ability to populate filesystem from directory
Currently the only way to pre populate an XFS partition is via the prototype file. While it works it has some limitations like:
- not allowed spaces in file names
- not preserving timestamps of original inodes
This series adds a new -P option to mkfs.xfs that allows users to populate a newly created filesystem directly from an existing directory. While similar to the prototype functionality, this doesn't require writing a prototype file.
The implementation preserves file and directory attributes (ownership, permissions, timestamps) from the source directory when copying content to the new filesystem.
[v1] -> [v2]
remove changes to protofile spec
ensure backward compatibility
[v2] -> [v3]
use inode_set_[acm]time() as suggested
avoid copying atime and ctime
they are often problematic for reproducibility, and
mtime is the important information to preserve anyway
[v3] -> [v4]
rewrite functionality to populate directly from an input directory this is similar to mkfs.ext4 option.
[v4] -> [v5]
reorder patch to make it easier to review
reflow to keep code below 80 chars
use _() macro in prints
add SPDX headers to new files
fix comment styling
move from typedef to structs
move direntry handling to own function
[v5] -> [v6]
rebase on 6.14
[v6] -> [v7]
move functionality to common -p flag
add noatime flag to skip atime copy and set to current time set ctime/crtime to current time
preserve hardlinks
preserve extended attributes for all file/dir types
add fsxattr to copied files/dirs
[v7] -> [v8]
changed directory source validation to use stat() instead of open() changed hardlink tracker to store inode numbers instead of inode pointers fixed path buffer handling for directory traversal
handle blocking FIFOs filetypes
handle hardlinks of symlinks
improve setup_proto and parse proto using structured xfs_proto_source type renamed noatime to preserve_atime with inverted logic
specify EBADF fgetxattr() and flistxattr() fallback for O_PATH fds switch to calloc() to initialize hardlinks_tracker
switch to reallocarray() for hardlinks_tracker resize
[v8] -> [v9]
squash commits in one
clarify the linear search implications in the commit log
fixed indentation and styling
fail early for invalid inputs
pass dir fd along to reduce amount of open()
switch from ino_t to xfs_ino_t
add support for socket files
copy fsxattr also for cases where we don't have an fd
[v9] -> [v10]
fix indentation, alignment and style
improve comments
avoid unneeded casts from ino_t to xfs_ino_t for source inodes avoid global hardlink_tracker as pointer
avoid mixing code and declarations
symlink target string now respects XFS_SYMLINK_MAXLEN
set root directory ownership/mode/attrs/fsxattrs from source dir
[v10] -> [v11]
fix indentation and comment style
avoid TOCTOU races by first opening file, then using fstat on it use openat() to avoid string manipulation
properly cleanup hardlink tracker
cleanup the henalde_direntry() switch statement
unify all file types except directories in a single create_inode() function
create a separate create_directory_inode() function
Luca Di Maio (1):
proto: add ability to populate a filesystem from a directory
man/man8/mkfs.xfs.8.in | 41 ++- mkfs/proto.c | 748 ++++++++++++++++++++++++++++++++++++++++- mkfs/proto.h | 18 +- mkfs/xfs_mkfs.c | 23 +- 4 files changed, 804 insertions(+), 26 deletions(-)
--
2.50.0
PATCH v11 1/1 proto: add ability to populate a filesystem from a directory
This patch implements the functionality to populate a newly created XFS filesystem directly from an existing directory structure.
It resuses existing protofile logic, it branches if input is a directory.
The population process steps are as follows:
- create the root inode before populating content
- recursively process nested directories
- handle regular files, directories, symlinks, char devices, block devices, sockets, fifos
- preserve attributes (ownership, permissions)
- preserve mtime timestamps from source files to maintain file history
- use current time for atime/ctime/crtime
- possible to specify atime=1 to preserve atime timestamps from source files
- preserve extended attributes and fsxattrs for all file types
- preserve hardlinks
At the moment, the implementation for the hardlink tracking is very simple, as it involves a linear search.
from my local testing using larger source directories (1.3mln inodes, ~400k hardlinks) the difference was actually just a few seconds (given that most of the time is doing i/o). We might want to revisit that in the future if this becomes a bottleneck.
This functionality makes it easier to create populated filesystems without having to mount them, it's particularly useful for reproducible builds.
--- man/man8/mkfs.xfs.8.in | 41 ++- mkfs/proto.c | 748 ++++++++++++++++++++++++++++++++++++++++- mkfs/proto.h | 18 +- mkfs/xfs_mkfs.c | 23 +- 4 files changed, 804 insertions(+), 26 deletions(-)
man/man8/mkfs.xfs.8.in +29 -12
@@ -28,7 +28,7 @@ mkfs.xfs \- construct an XFS filesystem .I naming_options ] [ .B \-p -.I protofile_options +.I prototype_options ] [ .B \-q ] [
@@ -977,30 +977,39 @@ option set. .PP .PD 0 .TP -.BI \-p " protofile_options" +.BI \-p " prototype_options" .TP .BI "Section Name: " [proto] .PD -These options specify the protofile parameters for populating the filesystem. +These options specify the prototype parameters for populating the filesystem. The valid -.I protofile_options +.I prototype_options are: .RS 1.2i .TP -.BI [file=] protofile +.BI [file=] The .B file= prefix is not required for this CLI argument for legacy reasons. If specified as a config file directive, the prefix is required. - +.TP +.BI [file=] directory If the optional .PD -.I protofile -argument is given, +.I prototype +argument is given, and it's a directory, .B mkfs.xfs -uses -.I protofile -as a prototype file and takes its directions from that file. +will populate the root file system with the contents of the given directory. +Content, timestamps (atime, mtime), attributes and extended attributes +are preserved for all file types. +.TP +.BI [file=] protofile +If the optional +.PD +.I prototype +argument is given, and points to a regular file, +.B mkfs.xfs +uses it as a prototype file and takes its directions from that file. The blocks and inodes specifiers in the .I protofile are provided for backwards compatibility, but are otherwise unused.
@@ -1136,8 +1145,16 @@ always terminated with the dollar ( .B $ ) token. .TP +.BI atime= value +If set to 1, when we're populating the root filesystem from a directory ( +.B file=directory +option) +access times are going to be preserved and are copied from the source files. +Set to 0 to set access times to the current time instead. +By default, this is set to 0. +.TP .BI slashes_are_spaces= value -If set to 1, slashes ("/") in the first token of each line of the protofile +If set to 1, slashes ("/") in the first token of each line of the prototype file are converted to spaces. This enables the creation of a filesystem containing filenames with spaces. By default, this is set to 0.
mkfs/proto.c +741 -7
807 lines not on this page · raw message
mkfs/proto.h +15 -3
@@ -6,9 +6,21 @@#ifndef MKFS_PROTO_H_ #define MKFS_PROTO_H_ -char *setup_proto (char *fname); -void parse_proto (struct xfs_mount *mp,struct fsxattr *fsx,char **pp, -int proto_slashes_are_spaces); +enum proto_source_type { +PROTO_SRC_NONE =0 , +PROTO_SRC_PROTOFILE , +PROTO_SRC_DIR +}; +struct proto_source { +enum proto_source_type type; +char *data; +}; + +struct proto_source setup_proto (char *fname); +void parse_proto (struct xfs_mount *mp,struct fsxattr *fsx, +struct proto_source *protosource, +int proto_slashes_are_spaces, +int proto_preserve_atime);void res_failed (int err);#endif /* MKFS_PROTO_H_ */
mkfs/xfs_mkfs.c +19 -4
@@ -123,6 +123,7 @@ enum {enum {P_FILE =0 , +P_ATIME ,P_SLASHES ,P_MAX_OPTS , };
@@ -714,6 +715,7 @@ static struct opt_params popts = { .ini_section = "proto", .subopts = { [P_FILE] = "file", + [P_ATIME] = "atime", [P_SLASHES] = "slashes_are_spaces", [P_MAX_OPTS] = NULL, },
@@ -722,6 +724,12 @@ static struct opt_params popts = { .conflicts = { { NULL, LAST_CONFLICT } }, .defaultval = SUBOPT_NEEDS_VAL, }, + { .index = P_ATIME, + .conflicts = { { NULL, LAST_CONFLICT } }, + .minval = 0, + .maxval = 1, + .defaultval = 1, + }, { .index = P_SLASHES, .conflicts = { { NULL, LAST_CONFLICT } }, .minval = 0,
@@ -1079,6 +1087,7 @@ struct cli_params {int lsunit;int is_supported;int proto_slashes_are_spaces; +int proto_atime;int data_concurrency;int log_concurrency;int rtvol_concurrency;
@@ -1206,6 +1215,7 @@ usage( void ) /* naming */ [-n size=num,version=2|ci,ftype=0|1,parent=0|1]]\n\ /* no-op info only */ [-N]\n\ /* prototype file */ [-p fname]\n\ +/* populate from directory */ [-p dirname,atime=0|1]\n\ /* quiet */ [-q]\n\ /* realtime subvol */ [-r extsize=num,size=num,rtdev=xxx,rgcount=n,rgsize=n,\n\ concurrency=num,zoned=0|1,start=n,reserved=n]\n\
@@ -2131,6 +2141,9 @@ proto_opts_parser(case P_SLASHES : cli->proto_slashes_are_spaces =getnum (value, opts, subopt);break ; +case P_ATIME : + cli->proto_atime =getnum (value, opts, subopt); +break ;case P_FILE : fallthrough;default :
@@ -5682,7 +5695,7 @@ main(int discard =1 ;int force_overwrite =0 ;int quiet =0 ; -char *protostring =NULL ; +struct proto_source protosource;int worst_freelist =0 ;struct libxfs_init xi = {
@@ -5832,8 +5845,6 @@ main(*/ cfgfile_parse (&cli); - protostring =setup_proto (cli.protofile); -/* * Extract as much of the valid config as we can from the CLI input * before opening the libxfs devices.
@@ -6010,7 +6021,11 @@ main(/* * Allocate the root inode and anything else in the proto file. */ -parse_proto (mp, &cli.fsx, &protostring, cli.proto_slashes_are_spaces); + protosource =setup_proto (cli.protofile); +parse_proto (mp, &cli.fsx, + &protosource, + cli.proto_slashes_are_spaces, + cli.proto_atime);/* * Protect ourselves against possible stupidity
--
2.50.0
Re: [PATCH v11 1/1] proto: add ability to populate a filesystem from a directory
Quoted 79 lines · Luca Di Maio
This patch implements the functionality to populate a newly created XFS filesystem directly from an existing directory structure.
It resuses existing protofile logic, it branches if input is a directory.
The population process steps are as follows:
- create the root inode before populating content
- recursively process nested directories
- handle regular files, directories, symlinks, char devices, block devices, sockets, fifos
- preserve attributes (ownership, permissions)
- preserve mtime timestamps from source files to maintain file history
- use current time for atime/ctime/crtime
- possible to specify atime=1 to preserve atime timestamps from source files
- preserve extended attributes and fsxattrs for all file types
- preserve hardlinksAt the moment, the implementation for the hardlink tracking is very simple, as it involves a linear search.
from my local testing using larger source directories (1.3mln inodes, ~400k hardlinks) the difference was actually just a few seconds (given that most of the time is doing i/o). We might want to revisit that in the future if this becomes a bottleneck.This functionality makes it easier to create populated filesystems without having to mount them, it's particularly useful for reproducible builds.
Signed-off-by Luca Di Maio <[email protected]>--- man/man8/mkfs.xfs.8.in | 41 ++- mkfs/proto.c | 748 ++++++++++++++++++++++++++++++++++++++++- mkfs/proto.h | 18 +- mkfs/xfs_mkfs.c | 23 +- 4 files changed, 804 insertions(+), 26 deletions(-)man/man8/mkfs.xfs.8.in +7
-6@@ -28,7 +28,7 @@ mkfs.xfs \- construct an XFS filesystem .I naming_options ] [ .B \-p -.I protofile_options +.I prototype_options ] [ .B \-q ] [@@ -977,30 +977,39 @@ option set. .PP .PD 0 .TP -.BI \-p " protofile_options" +.BI \-p " prototype_options" .TP .BI "Section Name: " [proto] .PD -These options specify the protofile parameters for populating the filesystem. +These options specify the prototype parameters for populating the filesystem. The valid -.I protofile_options +.I prototype_options are: .RS 1.2i .TP -.BI [file=] protofile +.BI [file=] The .B file= prefix is not required for this CLI argument for legacy reasons. If specified as a config file directive, the prefix is required. - +.TP +.BI [file=] directory If the optional
man/man8/mkfs.xfs.8.in option set. · line 1002
Quoted 10 lines
man/man8/mkfs.xfs.8.in +3
-5.PD -.I protofile -argument is given, +.I prototype +argument is given, and it's a directory, .B mkfs.xfs -uses -.I protofile -as a prototype file and takes its directions from that file. +will populate the root file system with the contents of the given directory.
"...of the given directory tree."
(It's a subtle hint that it recursively imports the whole tree, not one single directory)
Quoted 14 lines
man/man8/mkfs.xfs.8.in +10
-0+Content, timestamps (atime, mtime), attributes and extended attributes +are preserved for all file types. +.TP +.BI [file=] protofile +If the optional +.PD +.I prototype +argument is given, and points to a regular file, +.B mkfs.xfs +uses it as a prototype file and takes its directions from that file. The blocks and inodes specifiers in the .I protofile are provided for backwards compatibility, but are otherwise unused.
man/man8/mkfs.xfs.8.in always terminated with the dollar ( · line 1149
Quoted 5 lines
man/man8/mkfs.xfs.8.in +2
-0@@ -1136,8 +1145,16 @@ always terminated with the dollar ( .B $ ) token. .TP +.BI atime= value +If set to 1, when we're populating the root filesystem from a directory (
Who is "we"?
"If set to 1, mkfs will copy in access timestamps from the source files.
Otherwise, access timestamps will be set to the current time."
Quoted 71 lines
man/man8/mkfs.xfs.8.in +7
-1+.B file=directory +option) +access times are going to be preserved and are copied from the source files. +Set to 0 to set access times to the current time instead. +By default, this is set to 0. +.TP .BI slashes_are_spaces= value -If set to 1, slashes ("/") in the first token of each line of the protofile +If set to 1, slashes ("/") in the first token of each line of the prototype file are converted to spaces. This enables the creation of a filesystem containing filenames with spaces. By default, this is set to 0.mkfs/proto.c +21
-3@@ -5,6 +5,8 @@*/ #include "libxfs.h" +#include <dirent.h> +#include <sys/resource.h> #include <sys/stat.h> #include <sys/xattr.h> #include <linux/xattr.h> @@ -21,6 +23,11 @@ static void rsvfile(xfs_mount_t *mp, xfs_inode_t *ip, long long len);static int newregfile (char **pp,char **fname);static void rtinit (xfs_mount_t *mp);static off_t filesize (int fd); +static void populate_from_dir (struct xfs_mount *mp,struct fsxattr *fsxp, +char *source_dir); +static void walk_dir (struct xfs_mount *mp,struct xfs_inode *pip, +struct fsxattr *fsxp,char *path_buf); +static int preserve_atime;static int slashes_are_spaces;/* @@ -54,7 +61,7 @@ getnum(return i; } -char * +struct proto_source setup_proto (char *fname) {@@ -63,14 +70,40 @@ setup_proto(int fd;long size; -if (!fname) -return dflt; +struct proto_source result = {}; +struct stat statbuf; + +/* +* If no prototype path is supplied, use the default protofile which +* creates only a root directory. +*/ +if (!fname) { + result.type =PROTO_SRC_PROTOFILE ; + result.data = dflt; +return result; + } +if ((fd =open (fname,O_RDONLY )) <0 || (size =filesize (fd)) <0 ) {fprintf (stderr,_ ("%s: failed to open %s: %s\n" ), progname, fname,strerror (errno)); goto out_fail; }
mkfs/proto.c setup_proto( · line 101
Quoted 10 lines
mkfs/proto.c +10
-0+if (fstat (fd, &statbuf) <0 ) +fail (_ ("invalid or unreadable source path" ), errno); + +/* +* Handle directory inputs. +*/ +if (S_ISDIR (statbuf.st_mode)) { + result.type =PROTO_SRC_DIR ; + result.data = fname; +return result;
Er... this leaks the fd that you opened above.
Quoted 96 lines
mkfs/proto.c +55
-4+ } + +/* +* Else this is a protofile, let's handle traditionally. +*/ buf =malloc (size +1 );if (read (fd, buf, size) < size) { fprintf(stderr, _("%s: read failed on %s: %s\n"),@@ -90,7 +123,10 @@ setup_proto( (void )getnum (getstr (&buf),0 ,0 , false);/* block count */ (void )getnum (getstr (&buf),0 ,0 , false);/* inode count */ close (fd); -return buf; + + result.type =PROTO_SRC_PROTOFILE ; + result.data = buf; +return result; out_fail:if (fd >=0 )@@ -379,6 +415,13 @@ writeattr(int error; ret =fgetxattr (fd, attrname, valuebuf, valuelen); +/* +* In case of filedescriptors with O_PATH, fgetxattr() will fail with +* EBADF. +* Let's try to fallback to lgetxattr() using input path. +*/ +if (ret <0 && errno ==EBADF ) + ret =lgetxattr (fname, attrname, valuebuf, valuelen);if (ret <0 ) {if (errno ==EOPNOTSUPP )return ;@@ -425,6 +468,13 @@ writeattrs(fail (_ ("error allocating xattr name buffer" ), errno); ret =flistxattr (fd, namebuf,XATTR_LIST_MAX ); +/* +* In case of filedescriptors with O_PATH, flistxattr() will fail with +* EBADF. +* Let's try to fallback to llistxattr() using input path. +*/ +if (ret <0 && errno ==EBADF ) + ret =llistxattr (fname, namebuf,XATTR_LIST_MAX );if (ret <0 ) {if (errno ==EOPNOTSUPP ) goto out_namebuf;@@ -933,11 +983,27 @@ voidparse_proto ( xfs_mount_t *mp, struct fsxattr *fsx, - char **pp, - int proto_slashes_are_spaces) + struct proto_source *protosource, + int proto_slashes_are_spaces, + int proto_preserve_atime) { slashes_are_spaces = proto_slashes_are_spaces; -parseproto (mp,NULL , fsx, pp,NULL ); + preserve_atime = proto_preserve_atime; + +/* +* In case of a file input, we will use the prototype file logic else +* we will fallback to populate from dir. +*/ +switch (protosource->type) { +case PROTO_SRC_PROTOFILE : +parseproto (mp,NULL , fsx, &protosource->data,NULL ); +break ; +case PROTO_SRC_DIR : +populate_from_dir (mp, fsx, protosource->data); +break ; +case PROTO_SRC_NONE : +fail (_ ("invalid or unreadable source path" ),ENOENT ); + } }/* Create a sb-rooted metadata file. */ @@ -1172,3 +1238,671 @@ filesize(return -1 ;return stb.st_size; } + +/* Try to allow as many open directories as possible. */ +static void +bump_max_fds (void ) +{ +struct rlimit rlim = {}; +int ret; + + ret =getrlimit (RLIMIT_NOFILE , &rlim); +if (ret) +return ; + + rlim.rlim_cur = rlim.rlim_max;
mkfs/proto.c filesize( · line 1263
Quoted 10 lines
mkfs/proto.c +10
-0+ ret =setrlimit (RLIMIT_NOFILE , &rlim); +if (ret <0 ) +fprintf (stderr,_ ("%s: could not bump fd limit: [ %d - %s]\n" ), + progname, errno,strerror (errno)); +} + +static void +writefsxattrs ( +struct xfs_inode *ip, +struct fsxattr *fsxp)
^ one more tab to line these up, please
Quoted 4 lines
mkfs/proto.c +4
-0+{ + ip->i_projid = fsxp->fsx_projid; + ip->i_extsize = fsxp->fsx_extsize; + ip->i_diflags =xfs_flags2diflags (ip, fsxp->fsx_xflags);
mkfs/proto.c filesize( · line 1277
Quoted 10 lines
mkfs/proto.c +10
-0+if (xfs_has_v3inodes (ip->i_mount)) { + ip->i_diflags2 =xfs_flags2diflags2 (ip, fsxp->fsx_xflags); + ip->i_cowextsize = fsxp->fsx_cowextsize; + } +} + +static void +writetimestamps ( +struct xfs_inode *ip, +struct stat *statbuf)
^ one more tab to line these up, please
Quoted 19 lines
mkfs/proto.c +19
-0+{ +struct timespec64 ts; + +/* +* Copy timestamps from source file to destination inode. +* Usually reproducible archives will delete or not register +* atime and ctime, for example: +* https://www.gnu.org/software/tar/manual/html_section/Reproducibility.html +* hence we will only copy mtime, and let ctime/crtime be set to +* current time. +* atime will be copied over if atime is true. +*/ + ts.tv_sec = statbuf->st_mtim.tv_sec; + ts.tv_nsec = statbuf->st_mtim.tv_nsec; +inode_set_mtime_to_ts (VFS_I (ip), ts); + +/* +* In case of atime option, we will copy the atime timestamp +* from source.
mkfs/proto.c filesize( · line 1306
Quoted 10 lines
mkfs/proto.c +10
-0+*/ +if (preserve_atime) { + ts.tv_sec = statbuf->st_atim.tv_sec; + ts.tv_nsec = statbuf->st_atim.tv_nsec; +inode_set_atime_to_ts (VFS_I (ip), ts); + } +} + +struct hardlink { +ino_t src_ino;
^ one more tab to line these up, please
Quoted 65 lines
mkfs/proto.c +65
-0+xfs_ino_t dst_ino; +}; + +struct hardlinks { +size_t count; +size_t size; +struct hardlink *entries; +}; + +/* Growth strategy for hardlink tracking array */ +/* Double size for small arrays */ +#define HARDLINK_DEFAULT_GROWTH_FACTOR 2 +/* Grow by 25% for large arrays */ +#define HARDLINK_LARGE_GROWTH_FACTOR 0.25 +/* Threshold to switch growth strategies */ +#define HARDLINK_THRESHOLD 1024 +/* Initial allocation size */ +#define HARDLINK_TRACKER_INITIAL_SIZE 4096 + +/* +* Keep track of source inodes that are from hardlinks so we can retrieve them +* when needed to setup in destination. +*/ +static struct hardlinks hardlink_tracker = {0 }; + +static void +init_hardlink_tracker (void ) +{ + hardlink_tracker.size =HARDLINK_TRACKER_INITIAL_SIZE ; + hardlink_tracker.entries =calloc ( + hardlink_tracker.size, +sizeof (struct hardlink )); +if (!hardlink_tracker.entries) +fail (_ ("error allocating hardlinks tracking array" ), errno); +} + +static void +cleanup_hardlink_tracker (void ) +{ +free (hardlink_tracker.entries); + hardlink_tracker.entries =NULL ; + hardlink_tracker.count =0 ; + hardlink_tracker.size =0 ; +} + +static xfs_ino_t +get_hardlink_dst_inode ( +xfs_ino_t i_ino) +{ +for (size_t i =0 ; i < hardlink_tracker.count; i++) { +if (hardlink_tracker.entries[i].src_ino == i_ino) +return hardlink_tracker.entries[i].dst_ino; + } +return 0 ; +} + +static void +track_hardlink_inode ( +ino_t src_ino, +xfs_ino_t dst_ino) +{ +if (hardlink_tracker.count >= hardlink_tracker.size) { +/* +* double for smaller capacity. +* instead grow by 25% steps for larger capacities.
mkfs/proto.c filesize( · line 1381
Quoted 10 lines
mkfs/proto.c +10
-0+*/ +const size_t old_size = hardlink_tracker.size; +size_t new_size = old_size *HARDLINK_DEFAULT_GROWTH_FACTOR ; +if (old_size >HARDLINK_THRESHOLD ) + new_size = old_size + (old_size *HARDLINK_LARGE_GROWTH_FACTOR ); + +struct hardlink *resized_array =reallocarray ( + hardlink_tracker.entries, + new_size, +sizeof (struct hardlink ));
Continuations should be indented twice (not once) so that they don't line up with code that's in the body of an if statement:
Quoted 18 lines
mkfs/proto.c +18
-0+if (!resized_array) +fail (_ ("error enlarging hardlinks tracking array" ), errno); + +memset (&resized_array[old_size],0 , + (new_size - old_size) *sizeof (struct hardlink )); + + hardlink_tracker.entries = resized_array; + hardlink_tracker.size = new_size; + } + hardlink_tracker.entries[hardlink_tracker.count].src_ino = src_ino; + hardlink_tracker.entries[hardlink_tracker.count].dst_ino = dst_ino; + hardlink_tracker.count++; +} + +/* +* This function will first check in our tracker if the input hardlink has +* already been stored, if not report false so create_inode() can continue +* handling the inode as regularly, and later save the source inode in our
mkfs/proto.c filesize( · line 1409
Quoted 10 lines
mkfs/proto.c +10
-0+* buffer for future consumption. +*/ +static bool +handle_hardlink ( +struct xfs_mount *mp, +struct xfs_inode *pip, +struct xfs_name xname, +struct stat file_stat) +{ +int error;
^ one more tab to line these up, please
Quoted 42 lines
mkfs/proto.c +42
-0+xfs_ino_t dst_ino; +struct xfs_inode *ip; +struct xfs_trans *tp; +struct xfs_parent_args *ppargs =NULL ; + + tp =getres (mp,0 ); + ppargs =newpptr (mp); + dst_ino =get_hardlink_dst_inode (file_stat.st_ino); + +/* +* We didn't find the hardlink inode, this means it's the first time +* we see it, report error so create_inode() can continue handling the +* inode as a regular file type, and later save the source inode in our +* buffer for future consumption. +*/ +if (dst_ino ==0 ) +return false; + + error =libxfs_iget (mp,NULL , dst_ino,0 , &ip); +if (error) +fail (_ ("failed to get inode" ), error); + +/* +* In case the inode was already in our tracker we need to setup the +* hardlink and skip file copy. +*/ +libxfs_trans_ijoin (tp, pip,0 ); +libxfs_trans_ijoin (tp, ip,0 ); +newdirent (mp, tp, pip, &xname, ip, ppargs); + +/* +* Increment the link count +*/ +libxfs_bumplink (tp, ip); + +libxfs_trans_log_inode (tp, ip,XFS_ILOG_CORE ); + + error = -libxfs_trans_commit (tp); +if (error) +fail (_ ("Error encountered creating file from prototype file" ), error); + +libxfs_parent_finish (mp, ppargs);
mkfs/proto.c filesize( · line 1461
Quoted 10 lines
mkfs/proto.c +10
-0+libxfs_irele (ip); + +return true; +} + +static void +create_directory_inode( + struct xfs_mount *mp, + struct xfs_inode *pip, + struct fsxattr *fsxp,
^^ space before tab
mkfs/proto.c filesize( · line 1467
Quoted 6 lines
mkfs/proto.c +6
-0+int mode, + struct cred creds, + struct xfs_name xname, + int flags, + struct stat file_stat, + int fd,
^ space before tab
mkfs/proto.c filesize( · line 1472
Quoted 5 lines
mkfs/proto.c +5
-0+char *entryname, + char *path) +{ + + int error;
^ one more tab to line these up, please
Quoted 37 lines
mkfs/proto.c +37
-0+struct xfs_inode *ip; +struct xfs_trans *tp; +struct xfs_parent_args *ppargs =NULL ; + + tp =getres (mp,0 ); + ppargs =newpptr (mp); + + error =creatproto (&tp, pip, mode,0 , &creds, fsxp, &ip); +if (error) +fail (_ ("Inode allocation failed" ), error); + +libxfs_trans_ijoin (tp, pip,0 ); + +newdirent (mp, tp, pip, &xname, ip, ppargs); + +libxfs_bumplink (tp, pip); +libxfs_trans_log_inode (tp, pip,XFS_ILOG_CORE ); +newdirectory (mp, tp, ip, pip); + +/* +* Copy over timestamps. +*/ +writetimestamps (ip, &file_stat); + +libxfs_trans_log_inode (tp, ip, flags); + + error = -libxfs_trans_commit (tp); +if (error) +fail (_ ("Directory inode allocation failed." ), error); + +libxfs_parent_finish (mp, ppargs); + tp =NULL ; + +/* +* Copy over attributes. +*/ +writeattrs (ip, entryname, fd);
mkfs/proto.c filesize( · line 1519
Quoted 10 lines
mkfs/proto.c +10
-0+writefsxattrs (ip, fsxp); +close (fd); + +walk_dir (mp, ip, fsxp, path); + +libxfs_irele (ip); +} + +static void +create_inode(
Might want to call this create_nondir_inode to distinguish it from create_directory_inode.
mkfs/proto.c filesize( · line 1461
Quoted 3 lines
mkfs/proto.c +3
-0+struct xfs_mount *mp, + struct xfs_inode *pip, + struct fsxattr *fsxp,
^^ space before tab
mkfs/proto.c filesize( · line 1529
Quoted 7 lines
mkfs/proto.c +7
-0+int mode, + struct cred creds, + struct xfs_name xname, + int flags, + struct stat file_stat, + xfs_dev_t rdev, + int fd,
^ space before tab
mkfs/proto.c filesize( · line 1530
Quoted 1 line
mkfs/proto.c +1
-0+ char *fname)
Hrmm, is @xname the filename we're creating in pip, and @fname is the path to the original file so that we can copy the contents and various other attributes? If so, then maybe s/fname/src_fname/ to make this clearer?
mkfs/proto.c filesize( · line 1535
Quoted 5 lines
mkfs/proto.c +5
-0+{ + +char link_target[XFS_SYMLINK_MAXLEN ]; +ssize_t link_len; +int error;
^ one more tab to line these up, please
Quoted 85 lines
mkfs/proto.c +85
-0+struct xfs_inode *ip; +struct xfs_trans *tp; +struct xfs_parent_args *ppargs =NULL ; + +/* +* If handle_hardlink() returns true it means the hardlink has been +* correctly found and set, so we don't need to do anything else. +*/ +if (file_stat.st_nlink >1 &&handle_hardlink (mp, pip, xname, file_stat)) { +close (fd); +return ; + } +/* +* If instead we have an error it means the hardlink was not registered, +* so we proceed to treat it like a regular file, and save it to our +* tracker later. +*/ + tp =getres (mp,0 ); +/* +* In case of symlinks, we need to handle things a little differently. +* We need to read out our link target and act accordingly. +*/ +if (xname.type ==XFS_DIR3_FT_SYMLINK ) { + link_len =readlink (fname, link_target,XFS_SYMLINK_MAXLEN ); +if (link_len <0 ) +fail (_ ("could not resolve symlink" ), errno); +if (link_len >=PATH_MAX ) +fail (_ ("symlink target too long" ),ENAMETOOLONG ); + tp =getres (mp,XFS_B_TO_FSB (mp, link_len)); + } + ppargs =newpptr (mp); + + error =creatproto (&tp, pip, mode, rdev, &creds, fsxp, &ip); +if (error) +fail (_ ("Inode allocation failed" ), error); + +/* +* In case of symlinks, we now write it down, for other file types +* this is handled later before cleanup. +*/ +if (xname.type ==XFS_DIR3_FT_SYMLINK ) +writesymlink (tp, ip, link_target, link_len); + +libxfs_trans_ijoin (tp, pip,0 ); +newdirent (mp, tp, pip, &xname, ip, ppargs); + +/* +* Copy over timestamps. +*/ +writetimestamps (ip, &file_stat); + +libxfs_trans_log_inode (tp, ip, flags); + + error = -libxfs_trans_commit (tp); +if (error) +fail (_ ("Error encountered creating file from prototype file" ), error); + +libxfs_parent_finish (mp, ppargs); + +/* +* Copy over file content, attributes, extended attributes and +* timestamps. +*/ +if (fd >=0 ) { +/* We need to writefile only when not dealing with a symlink. */ +if (xname.type !=XFS_DIR3_FT_SYMLINK ) +writefile (ip, fname, fd); +writeattrs (ip, fname, fd); +close (fd); + } +/* +* We do fsxattr also for file types where we don't have an fd, +* for example FIFOs. +*/ +writefsxattrs (ip, fsxp); + +/* +* If we're here it means this is the first time we're encountering an +* hardlink, so we need to store it. +*/ +if (file_stat.st_nlink >1 ) +track_hardlink_inode (file_stat.st_ino, ip->i_ino); + +libxfs_irele (ip); +}
mkfs/proto.c filesize( · line 1630
Quoted 10 lines
mkfs/proto.c +10
-0+ +static void +handle_direntry ( +struct xfs_mount *mp, +struct xfs_inode *pip, +struct fsxattr *fsxp, +char *path_buf, +struct dirent *entry) +{ +char cur_path_buf[PATH_MAX ];
Hrmm. For each level of the source directory tree we allocate another PATH_MAX buffer on the stack just to snprintf from @path_buf (which is itself a stack buffer).
Even assuming the usual 8M thread stack, that means a directory structure more than ~2000 levels deep will overflow the stack and crash mkfs.
I really think you ought to consider allocating /one/ buffer at the start, passing (path_buf, path_len) down the stack, and then snprinting at the end of the buffer:
size_t avail = PATH_MAX - path_len;
size_t wrote = snprintf(path_buf + path_len, avail, "/%s", entry->d_name);
if (wrote > avail)
fail(path_buf, ENAMETOOLONG);
...
if (S_ISDIR(...)) {
create_directory_inode(..., path_buf, path_len + strlen(entry->d_name), ...);
...
path_buf[path_len] = 0;One buffer, much less memory usage.
mkfs/proto.c filesize( · line 1631
Quoted 1 line
mkfs/proto.c +1
-0+char *fname ="" ;
^ another tab to line things up with the
declarations
mkfs/proto.c filesize( · line 1637
Quoted 6 lines
mkfs/proto.c +6
-0+int pathfd,fd =-1 ; +int flags; +int majdev; +int mindev; +int mode; +int rdev =0 ;
^ space before tab
Quoted 39 lines
mkfs/proto.c +39
-0+struct stat file_stat; +struct xfs_name xname; + +/* Ensure we're within the limits of PATH_MAX. */ +if (strlen (path_buf) +strlen (entry->d_name) >=PATH_MAX ) +fail (_ ("path name too long" ),ENAMETOOLONG ); +/* +* Keep a reference to our current file path it will be useful +* afterwards for fds open with O_PATH where we will have to fallback +* to path based attribute handling. +*/ +if (snprintf (cur_path_buf,PATH_MAX ,"%s/%s" , + path_buf, entry->d_name) <0 ) +fail (_ ("path encoding error" ),-1 ); + + pathfd =open (path_buf,O_NOFOLLOW |O_PATH ); +if (pathfd <0 ){ +fprintf (stderr,_ ("%s: cannot open %s: %s\n" ), progname, + cur_path_buf,strerror (errno)); +exit (1 ); + } + +/* +* Symlinks and sockets will need to be opened with O_PATH to work, so +* we handle this special case. +*/ + fd =openat (pathfd, entry->d_name,O_NOFOLLOW |O_PATH ); +if (fd <0 ) { +fprintf (stderr,_ ("%s: cannot open %s: %s\n" ), progname, + cur_path_buf,strerror (errno)); +exit (1 ); + } + +if (fstat (fd, &file_stat) <0 ) { +fprintf (stderr,_ ("%s: cannot stat '%s': %s (errno=%d)\n" ), + progname, cur_path_buf,strerror (errno), errno); +exit (1 ); + } +
mkfs/proto.c filesize( · line 1686
Quoted 10 lines
mkfs/proto.c +10
-0+/* +* Regular files instead need to be reopened with broader flags so we +* check if that's the case and reopen those. +*/ +if (!S_ISSOCK (file_stat.st_mode) && + !S_ISLNK (file_stat.st_mode) && + !S_ISFIFO (file_stat.st_mode)) { +close (fd); + fd =openat (pathfd, entry->d_name, +O_NOFOLLOW |O_RDONLY |O_NOATIME );
Just out of curiosity, does O_NOATIME not work in the previous openat?
Quoted 49 lines
mkfs/proto.c +49
-0+if (fd <0 ) { +fprintf (stderr,_ ("%s: cannot open %s: %s\n" ), progname, + cur_path_buf,strerror (errno)); +exit (1 ); + } + } + +struct cred creds = { + .cr_uid = file_stat.st_uid, + .cr_gid = file_stat.st_gid, + }; + + xname.name = (unsigned char *)entry->d_name; + xname.len =strlen (entry->d_name); + xname.type =0 ; + mode = file_stat.st_mode; + flags =XFS_ILOG_CORE ; + +switch (file_stat.st_mode &S_IFMT ) { +case S_IFDIR : + xname.type =XFS_DIR3_FT_DIR ; +create_directory_inode (mp, pip, fsxp, mode, creds, xname, flags, + file_stat, fd, entry->d_name, cur_path_buf); +return ; +case S_IFREG : + xname.type =XFS_DIR3_FT_REG_FILE ; + fname = entry->d_name; +break ; +case S_IFCHR : + flags |=XFS_ILOG_DEV ; + xname.type =XFS_DIR3_FT_CHRDEV ; + majdev =major (file_stat.st_rdev); + mindev =minor (file_stat.st_rdev); + rdev =IRIX_MKDEV (majdev, mindev); + fname = entry->d_name; +break ; +case S_IFBLK : + flags |=XFS_ILOG_DEV ; + xname.type =XFS_DIR3_FT_BLKDEV ; + majdev =major (file_stat.st_rdev); + mindev =minor (file_stat.st_rdev); + rdev =IRIX_MKDEV (majdev, mindev); + fname = entry->d_name; +break ; +case S_IFLNK : +/* +* Being a symlink we opened the filedescriptor with O_PATH +* this will make flistxattr() and fgetxattr() fail wil EBADF, +* so we will need to fallback to llistxattr() and lgetxattr(),
mkfs/proto.c filesize( · line 1745
Quoted 10 lines
mkfs/proto.c +10
-0+* this will need the full path to the original file, not just +* the entry name. +*/ + xname.type =XFS_DIR3_FT_SYMLINK ; + fname = cur_path_buf; +break ; +case S_IFIFO : +/* +* Being a fifo we opened the filedescriptor with O_PATH +* this will make flistxattr() and fgetxattr() fail wil EBADF,
"fail with EBADF"...
Quoted 28 lines
mkfs/proto.c +28
-0+* so we will need to fallback to llistxattr() and lgetxattr(), +* this will need the full path to the original file, not just +* the entry name. +*/ + xname.type =XFS_DIR3_FT_FIFO ; + fname = cur_path_buf; +break ; +case S_IFSOCK : +/* +* Being a socket we opened the filedescriptor with O_PATH +* this will make flistxattr() and fgetxattr() fail wil EBADF, +* so we will need to fallback to llistxattr() and lgetxattr(), +* this will need the full path to the original file, not just +* the entry name. +*/ + xname.type =XFS_DIR3_FT_SOCK ; + fname = cur_path_buf; +break ; +default : +break ; + } + +create_inode (mp, pip, fsxp, mode, creds, xname, flags, file_stat, + rdev, fd, fname); +} + +/* +* Walk_dir will recursively list files and directories and populate the
mkfs/proto.c filesize( · line 1783
Quoted 10 lines
mkfs/proto.c +10
-0+* mountpoint *mp with them using handle_direntry(). +*/ +static void +walk_dir ( +struct xfs_mount *mp, +struct xfs_inode *pip, +struct fsxattr *fsxp, +char *path_buf) +{ +DIR *dir;
^ space before tab
Quoted 20 lines
mkfs/proto.c +20
-0+struct dirent *entry; + +/* +* Open input directory and iterate over all entries in it. +* when another directory is found, we will recursively call walk_dir. +*/ +if ((dir =opendir (path_buf)) ==NULL ) { +fprintf (stderr,_ ("%s: cannot open input dir: %s [%d - %s]\n" ), + progname, path_buf, errno,strerror (errno)); +exit (1 ); + } +while ((entry =readdir (dir)) !=NULL ) { +if (strcmp (entry->d_name,"." ) ==0 || +strcmp (entry->d_name,".." ) ==0 ) +continue ; + +handle_direntry (mp, pip, fsxp, path_buf, entry); + } +closedir (dir); +}
mkfs/proto.c filesize( · line 1813
Quoted 10 lines
mkfs/proto.c +10
-0+ +static void +populate_from_dir ( +struct xfs_mount *mp, +struct fsxattr *fsxp, +char *cur_path) +{ +int error; +int mode; +int fd =-1 ;
^ space before tab
Quoted 46 lines
mkfs/proto.c +46
-0+char path_buf[PATH_MAX ]; +struct stat file_stat; +struct xfs_inode *ip; +struct xfs_trans *tp; + +/* +* Initialize path_buf cur_path, strip trailing slashes they're +* automatically added when walking the dir. +*/ +if (strlen (cur_path) >1 && cur_path[strlen (cur_path)-1 ] =='/' ) + cur_path[strlen (cur_path)-1 ] ='\0' ; +if (snprintf (path_buf,PATH_MAX ,"%s" , cur_path) >=PATH_MAX ) +fail (_ ("path name too long" ),ENAMETOOLONG ); + +if (lstat (path_buf, &file_stat) <0 ) { +fprintf (stderr,_ ("%s: cannot stat '%s': %s (errno=%d)\n" ), + progname, path_buf,strerror (errno), errno); +exit (1 ); + } + fd =open (path_buf,O_NOFOLLOW |O_RDONLY |O_NOATIME ); +if (fd <0 ) { +fprintf (stderr,_ ("%s: cannot open %s: %s\n" ), + progname, path_buf,strerror (errno)); +exit (1 ); + } + +/* +* We first ensure we have the root inode. +*/ +struct cred creds = { + .cr_uid = file_stat.st_uid, + .cr_gid = file_stat.st_gid, + }; + mode = file_stat.st_mode; + + tp =getres (mp,0 ); + + error =creatproto (&tp,NULL , mode |S_IFDIR ,0 , &creds, fsxp, &ip); +if (error) +fail (_ ("Inode allocation failed" ), error); + + mp->m_sb.sb_rootino = ip->i_ino; +libxfs_log_sb (tp); +newdirectory (mp, tp, ip, ip); +libxfs_trans_log_inode (tp, ip,XFS_ILOG_CORE ); +
mkfs/proto.c filesize( · line 1869
Quoted 10 lines
mkfs/proto.c +10
-0+ error = -libxfs_trans_commit (tp); +if (error) +fail (_ ("Inode allocation failed" ), error); + +libxfs_parent_finish (mp,NULL ); + +/* +* Copy over attributes. +*/ +writeattrs (ip, path_buf, fd);
Nothing closes fd here; does it leak?
--D
Quoted 161 lines
mkfs/proto.c +39
-0+writefsxattrs (ip, fsxp); + +/* +* RT initialization. Do this here to ensure that the RT inodes get +* placed after the root inode. +*/ + error =create_metadir (mp); +if (error) +fail (_ ("Creation of the metadata directory inode failed" ), error); + +rtinit (mp); + +/* +* By nature of walk_dir() we could be opening a great number of fds +* for deeply nested directory trees. try to bump max fds limit. +*/ +bump_max_fds (); + +/* +* Initialize the hardlinks tracker. +*/ +init_hardlink_tracker (); +/* +* Now that we have a root inode, let's walk the input dir and populate +* the partition. +*/ +walk_dir (mp, ip, fsxp, path_buf); + +/* +* Cleanup hardlinks tracker. +*/ +cleanup_hardlink_tracker (); + +/* +* We free up our root inode only when we finished populating the root +* filesystem. +*/ +libxfs_irele (ip); +}mkfs/proto.h +15
-3@@ -6,9 +6,21 @@#ifndef MKFS_PROTO_H_ #define MKFS_PROTO_H_ -char *setup_proto (char *fname); -void parse_proto (struct xfs_mount *mp,struct fsxattr *fsx,char **pp, -int proto_slashes_are_spaces); +enum proto_source_type { +PROTO_SRC_NONE =0 , +PROTO_SRC_PROTOFILE , +PROTO_SRC_DIR +}; +struct proto_source { +enum proto_source_type type; +char *data; +}; + +struct proto_source setup_proto (char *fname); +void parse_proto (struct xfs_mount *mp,struct fsxattr *fsx, +struct proto_source *protosource, +int proto_slashes_are_spaces, +int proto_preserve_atime);void res_failed (int err);#endif /* MKFS_PROTO_H_ */ mkfs/xfs_mkfs.c +19
-5@@ -123,6 +123,7 @@ enum {enum {P_FILE =0 , +P_ATIME ,P_SLASHES ,P_MAX_OPTS , };@@ -714,6 +715,7 @@ static struct opt_params popts = { .ini_section = "proto", .subopts = { [P_FILE] = "file", + [P_ATIME] = "atime", [P_SLASHES] = "slashes_are_spaces", [P_MAX_OPTS] = NULL, },@@ -722,6 +724,12 @@ static struct opt_params popts = { .conflicts = { { NULL, LAST_CONFLICT } }, .defaultval = SUBOPT_NEEDS_VAL, }, + { .index = P_ATIME, + .conflicts = { { NULL, LAST_CONFLICT } }, + .minval = 0, + .maxval = 1, + .defaultval = 1, + }, { .index = P_SLASHES, .conflicts = { { NULL, LAST_CONFLICT } }, .minval = 0,@@ -1079,6 +1087,7 @@ struct cli_params {int lsunit;int is_supported;int proto_slashes_are_spaces; +int proto_atime;int data_concurrency;int log_concurrency;int rtvol_concurrency;@@ -1206,6 +1215,7 @@ usage( void ) /* naming */ [-n size=num,version=2|ci,ftype=0|1,parent=0|1]]\n\ /* no-op info only */ [-N]\n\ /* prototype file */ [-p fname]\n\ +/* populate from directory */ [-p dirname,atime=0|1]\n\ /* quiet */ [-q]\n\ /* realtime subvol */ [-r extsize=num,size=num,rtdev=xxx,rgcount=n,rgsize=n,\n\ concurrency=num,zoned=0|1,start=n,reserved=n]\n\@@ -2131,6 +2141,9 @@ proto_opts_parser(case P_SLASHES : cli->proto_slashes_are_spaces =getnum (value, opts, subopt);break ; +case P_ATIME : + cli->proto_atime =getnum (value, opts, subopt); +break ;case P_FILE : fallthrough;default :@@ -5682,7 +5695,7 @@ main(int discard =1 ;int force_overwrite =0 ;int quiet =0 ; -char *protostring =NULL ; +struct proto_source protosource;int worst_freelist =0 ;struct libxfs_init xi = {@@ -5832,8 +5845,6 @@ main(*/ cfgfile_parse (&cli); - protostring =setup_proto (cli.protofile); -/* * Extract as much of the valid config as we can from the CLI input * before opening the libxfs devices. @@ -6010,7 +6021,11 @@ main(/* * Allocate the root inode and anything else in the proto file. */ -parse_proto (mp, &cli.fsx, &protostring, cli.proto_slashes_are_spaces); + protosource =setup_proto (cli.protofile); +parse_proto (mp, &cli.fsx, + &protosource, + cli.proto_slashes_are_spaces, + cli.proto_atime);/* * Protect ourselves against possible stupidity -- 2.50.0
Re: [PATCH v11 1/1] proto: add ability to populate a filesystem from a directory
Thanks Darrick for the review!
Sorry again for this indentation mess, I'm going to basically align all function arguments and all the top-function declarations
I was a bit confused because elsewhere in the code is not like that so it's a bit difficult to infere
Offtopic: maybe we could also introduce an editorconfig setup? so that all various editors will be correctly set to see the tabs/spaces as needed (https://editorconfig.org/)
Back on topic:
Quoted 6 lines · Darrick J. Wong
Quoted 1 line
+will populate the root file system with the contents of the given directory.
"...of the given directory tree."
(It's a subtle hint that it recursively imports the whole tree, not one single directory)
Ack.
Quoted 22 lines
Quoted 14 lines
man/man8/mkfs.xfs.8.in +10
-0+Content, timestamps (atime, mtime), attributes and extended attributes +are preserved for all file types. +.TP +.BI [file=] protofile +If the optional +.PD +.I prototype +argument is given, and points to a regular file, +.B mkfs.xfs +uses it as a prototype file and takes its directions from that file. The blocks and inodes specifiers in the .I protofile are provided for backwards compatibility, but are otherwise unused.man/man8/mkfs.xfs.8.in always terminated with the dollar ( · line 1149
Quoted 5 lines
man/man8/mkfs.xfs.8.in +2
-0@@ -1136,8 +1145,16 @@ always terminated with the dollar ( .B $ ) token. .TP +.BI atime= value +If set to 1, when we're populating the root filesystem from a directory (Who is "we"?
Quoted 4 lines
"If set to 1, mkfs will copy in access timestamps from the source files.
Otherwise, access timestamps will be set to the current time."
Ack.
Quoted 8 lines
mkfs/proto.c setup_proto( · line 101
Quoted 5 lines
mkfs/proto.c +5
-0+*/ +if (S_ISDIR (statbuf.st_mode)) { + result.type =PROTO_SRC_DIR ; + result.data = fname; +return result;Er... this leaks the fd that you opened above.
Ack.
Quoted 6 lines
Review always terminated with the dollar ( · line 1156
Quoted 2 lines
(quoted diff) +2
-0+static void +create_inode(Might want to call this
create_nondir_inodeto distinguish it fromcreate_directory_inode.
Ack.
Quoted 7 lines
mkfs/proto.c filesize( · line 1530
Quoted 1 line
mkfs/proto.c +1
-0+ char *fname)Hrmm, is @xname the filename we're creating in pip, and @fname is the path to the original file so that we can copy the contents and various other attributes? If so, then maybe s/fname/src_fname/ to make this clearer?
Ack.
Quoted 37 lines
mkfs/proto.c filesize( · line 1630
Quoted 9 lines
mkfs/proto.c +9
-0+static void +handle_direntry ( +struct xfs_mount *mp, +struct xfs_inode *pip, +struct fsxattr *fsxp, +char *path_buf, +struct dirent *entry) +{ +char cur_path_buf[PATH_MAX ];Hrmm. For each level of the source directory tree we allocate another
PATH_MAXbuffer on the stack just to snprintf from @path_buf (which is itself a stack buffer).Even assuming the usual 8M thread stack, that means a directory structure more than ~2000 levels deep will overflow the stack and crash mkfs.
I really think you ought to consider allocating /one/ buffer at the start, passing (
path_buf,path_len) down the stack, and then snprinting at the end of the buffer:size_t avail = PATH_MAX - path_len; size_t wrote = snprintf(path_buf + path_len, avail, "/%s", entry->d_name); if (wrote > avail) fail(path_buf, ENAMETOOLONG); ... if (S_ISDIR(...)) { create_directory_inode(..., path_buf, path_len + strlen(entry->d_name), ...); ... path_buf[path_len] = 0;
Quoted 1 line
One buffer, much less memory usage.
Right, It was like this before but I've Inadvertently unoptimized this in V11 when switched to openat(), will fix in next path, sorry.
Quoted 8 lines
mkfs/proto.c filesize( · line 1686
Quoted 6 lines
mkfs/proto.c +6
-0+if (!S_ISSOCK (file_stat.st_mode) && + !S_ISLNK (file_stat.st_mode) && + !S_ISFIFO (file_stat.st_mode)) { +close (fd); + fd =openat (pathfd, entry->d_name, +O_NOFOLLOW |O_RDONLY |O_NOATIME );Just out of curiosity, does
O_NOATIMEnot work in the previous openat?
Actually on my test setup (mainly using docker/podman to test), opening with and without O_NOATIME when using O_PATH, does not change accesstime checking with stat, but also it works if I add it.
As a precautionary measure (not sure if podman/docker is messing with noatime) I'll add it, as it seems to work correctly.
Quoted 4 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ * this will make flistxattr() and fgetxattr() fail wil EBADF,"fail with EBADF"...
Ack.
Quoted 9 lines
mkfs/proto.c filesize( · line 1869
Quoted 4 lines
mkfs/proto.c +4
-0+/* +* Copy over attributes. +*/ +writeattrs (ip, path_buf, fd);Nothing closes fd here; does it leak?
--D
Ack.
L.
Re: [PATCH v11 1/1] proto: add ability to populate a filesystem from a directory
Quoted 9 lines · Luca Di Maio
Thanks Darrick for the review!
Sorry again for this indentation mess, I'm going to basically align all function arguments and all the top-function declarations
I was a bit confused because elsewhere in the code is not like that so it's a bit difficult to infereOfftopic: maybe we could also introduce an editorconfig setup? so that all various editors will be correctly set to see the tabs/spaces as needed (https://editorconfig.org/)
Hrmm, that /would/ be useful.
Quoted 3 lines
Back on topic:
On Tue, Jul 29, 2025 at 02:43:22PM -0700, Darrick J. Wong wrote:
<snipping>
Quoted 8 lines
Quoted 8 lines
mkfs/proto.c filesize( · line 1686
Quoted 6 lines
mkfs/proto.c +6
-0+if (!S_ISSOCK (file_stat.st_mode) && + !S_ISLNK (file_stat.st_mode) && + !S_ISFIFO (file_stat.st_mode)) { +close (fd); + fd =openat (pathfd, entry->d_name, +O_NOFOLLOW |O_RDONLY |O_NOATIME );Just out of curiosity, does
O_NOATIMEnot work in the previous openat?
[narrator: it doesn't]
Quoted 5 lines
Actually on my test setup (mainly using docker/podman to test), opening with and without
O_NOATIMEwhen usingO_PATH, does not change accesstime checking withstat, but also it works if I add it.
As a precautionary measure (not sure if podman/docker is messing with noatime) I'll add it, as it seems to work correctly.
On second thought I think you might leave the double opens because O_NOATIME is only allowed if the current user owns source file, or has CAP_FOWNER. If you're running mkfs as an unprivileged user trying to capture a rootfs (with uid 0 files) then O_NOATIME won't be allowed.
Maybe something along the lines of:
/* * Try to open the source file noatime to avoid a flood of * writes to the source fs, but we can fall back to plain * readonly mode if we don't have enough permission. */ fd = openat(pathfd, entry->d_name, O_NOFOLLOW | O_RDONLY | O_NOATIME); if (fd < 0) fd = openat(pathfd, entry->d_name, O_NOFOLLOW | O_RDONLY); if (fd < 0) /* whine and exit */
Just to see if you can open the source file without touching atime?
--D
Quoted 20 lines
Quoted 4 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ * this will make flistxattr() and fgetxattr() fail wil EBADF,"fail with EBADF"...
Ack.
Quoted 9 lines
mkfs/proto.c filesize( · line 1869
Quoted 4 lines
mkfs/proto.c +4
-0+/* +* Copy over attributes. +*/ +writeattrs (ip, path_buf, fd);Nothing closes fd here; does it leak?
--D
Ack.
L.
Re: [PATCH v11 1/1] proto: add ability to populate a filesystem from a directory
Quoted 7 lines · Darrick J. Wong
Quoted 6 lines · Luca Di Maio
Thanks Darrick for the review!
Sorry again for this indentation mess, I'm going to basically align all function arguments and all the top-function declarations
I was a bit confused because elsewhere in the code is not like that so it's a bit difficult to infere
Quoted 6 lines
Quoted 3 lines
Offtopic: maybe we could also introduce an editorconfig setup? so that all various editors will be correctly set to see the tabs/spaces as needed (https://editorconfig.org/)
Hrmm, that /would/ be useful.
Nice, will try to put something together for a future patch
Quoted 41 lines
Quoted 3 lines
Back on topic:
On Tue, Jul 29, 2025 at 02:43:22PM -0700, Darrick J. Wong wrote:
<snipping>
Quoted 8 lines
Quoted 8 lines
mkfs/proto.c filesize( · line 1686
Quoted 6 lines
mkfs/proto.c +6
-0+if (!S_ISSOCK (file_stat.st_mode) && + !S_ISLNK (file_stat.st_mode) && + !S_ISFIFO (file_stat.st_mode)) { +close (fd); + fd =openat (pathfd, entry->d_name, +O_NOFOLLOW |O_RDONLY |O_NOATIME );Just out of curiosity, does
O_NOATIMEnot work in the previous openat?[narrator: it doesn't]
Quoted 5 lines
Actually on my test setup (mainly using docker/podman to test), opening with and without
O_NOATIMEwhen usingO_PATH, does not change accesstime checking withstat, but also it works if I add it.
As a precautionary measure (not sure if podman/docker is messing with noatime) I'll add it, as it seems to work correctly.On second thought I think you might leave the double opens because
O_NOATIMEis only allowed if the current user owns source file, or hasCAP_FOWNER. If you're running mkfs as an unprivileged user trying to capture a rootfs (with uid 0 files) thenO_NOATIMEwon't be allowed.Maybe something along the lines of:
/* * Try to open the source file noatime to avoid a flood of * writes to the source fs, but we can fall back to plain * readonly mode if we don't have enough permission. */ fd = openat(pathfd, entry->d_name, O_NOFOLLOW | O_RDONLY | O_NOATIME); if (fd < 0) fd = openat(pathfd, entry->d_name, O_NOFOLLOW | O_RDONLY); if (fd < 0) /* whine and exit */
Quoted 3 lines
Just to see if you can open the source file without touching atime?
--D
Right, will do like this, didn't think about rootless/unprivileged mkfs
L.
Re: [PATCH v11 1/1] proto: add ability to populate a filesystem from a directory
Quoted 8 lines · Luca Di Maio
Quoted 7 lines · Darrick J. Wong
Quoted 6 lines · Luca Di Maio
Thanks Darrick for the review!
Sorry again for this indentation mess, I'm going to basically align all function arguments and all the top-function declarations
I was a bit confused because elsewhere in the code is not like that so it's a bit difficult to infere
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Offtopic: maybe we could also introduce an editorconfig setup? so that all various editors will be correctly set to see the tabs/spaces as needed (https://editorconfig.org/)
Hrmm, that /would/ be useful.
Nice, will try to put something together for a future patch
Either that or a indent/clang-format wrapper?
The kernel has scripts/Lindent, maybe someone should try to write one for xfs style if I don't get to it first.
<sigh>rustfmt<duck>
Quoted 49 lines
Quoted 41 lines
Quoted 3 lines
Back on topic:
On Tue, Jul 29, 2025 at 02:43:22PM -0700, Darrick J. Wong wrote:
<snipping>
Quoted 8 lines
Quoted 8 lines
mkfs/proto.c filesize( · line 1686
Quoted 6 lines
mkfs/proto.c +6
-0+if (!S_ISSOCK (file_stat.st_mode) && + !S_ISLNK (file_stat.st_mode) && + !S_ISFIFO (file_stat.st_mode)) { +close (fd); + fd =openat (pathfd, entry->d_name, +O_NOFOLLOW |O_RDONLY |O_NOATIME );Just out of curiosity, does
O_NOATIMEnot work in the previous openat?[narrator: it doesn't]
Quoted 5 lines
Actually on my test setup (mainly using docker/podman to test), opening with and without
O_NOATIMEwhen usingO_PATH, does not change accesstime checking withstat, but also it works if I add it.
As a precautionary measure (not sure if podman/docker is messing with noatime) I'll add it, as it seems to work correctly.On second thought I think you might leave the double opens because
O_NOATIMEis only allowed if the current user owns source file, or hasCAP_FOWNER. If you're running mkfs as an unprivileged user trying to capture a rootfs (with uid 0 files) thenO_NOATIMEwon't be allowed.Maybe something along the lines of:
/* * Try to open the source file noatime to avoid a flood of * writes to the source fs, but we can fall back to plain * readonly mode if we don't have enough permission. */ fd = openat(pathfd, entry->d_name, O_NOFOLLOW | O_RDONLY | O_NOATIME); if (fd < 0) fd = openat(pathfd, entry->d_name, O_NOFOLLOW | O_RDONLY); if (fd < 0) /* whine and exit */Quoted 3 lines
Just to see if you can open the source file without touching atime?
--D
Right, will do like this, didn't think about rootless/unprivileged mkfs
L.
Re: [PATCH v11 1/1] proto: add ability to populate a filesystem from a directory
Quoted 13 lines · Darrick J. Wong
Quoted 12 lines · Luca Di Maio
Quoted 11 lines · Darrick J. Wong
Quoted 9 lines · Luca Di Maio
Thanks Darrick for the review!
Sorry again for this indentation mess, I'm going to basically align all function arguments and all the top-function declarations
I was a bit confused because elsewhere in the code is not like that so it's a bit difficult to infereOfftopic: maybe we could also introduce an editorconfig setup? so that all various editors will be correctly set to see the tabs/spaces as needed (https://editorconfig.org/)
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Hrmm, that /would/ be useful.
Nice, will try to put something together for a future patch
Either that or a indent/clang-format wrapper?
Yes that is what I was thinking, there should be the option to create custom formatting rules, and stick with those, ideally with a pre-commit hook
Quoted 2 lines
The kernel has
scripts/Lindent, maybe someone should try to write one for xfs style if I don't get to it first.
I'm unfamiliar with those, I'll try to check them in the next days/weeks
L.
Re: [PATCH v11 1/1] proto: add ability to populate a filesystem from a directory
In reply to Darrick J. WongQuoted 13 lines · Darrick J. Wong
Quoted 12 lines · Luca Di Maio
Quoted 11 lines · Darrick J. Wong
Quoted 9 lines · Luca Di Maio
Thanks Darrick for the review!
Sorry again for this indentation mess, I'm going to basically align all function arguments and all the top-function declarations
I was a bit confused because elsewhere in the code is not like that so it's a bit difficult to infereOfftopic: maybe we could also introduce an editorconfig setup? so that all various editors will be correctly set to see the tabs/spaces as needed (https://editorconfig.org/)
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Hrmm, that /would/ be useful.
Nice, will try to put something together for a future patch
Either that or a indent/clang-format wrapper?
editorconfig and clang-format would be great, as far as I know clang-format is pretty flexible in its style configuration
Quoted 5 lines
The kernel has
scripts/Lindent, maybe someone should try to write one for xfs style if I don't get to it first.<sigh>rustfmt<duck>
:(
Signature · 1 line
- Andrey