Skip to contentSkip to conversation

mkfs: add ability to populate filesystem from directory

PATCH Superseded Luca Di Maio 3 participants 8 messages Apr 26, 2025 – May 2, 2025

v7 of 12 Superseded Latest is v12 →

lore
Share & tools
lore
#0 lore raw

v7 cover mkfs: add ability to populate filesystem from directory

Currently the only way to pre populate an XFS partition is via the prototype file. While it works it has some limitations like:
- not allowed spaces in file names
- not preserving timestamps of original inodes

This series adds a new -P option to mkfs.xfs that allows users to populate a newly created filesystem directly from an existing directory. While similar to the prototype functionality, this doesn't require writing a prototype file.
The implementation preserves file and directory attributes (ownership, permissions, timestamps) from the source directory when copying content to the new filesystem.

[v1] -> [v2]
remove changes to protofile spec
ensure backward compatibility
[v2] -> [v3]
use inode_set_[acm]time() as suggested
avoid copying atime and ctime
they are often problematic for reproducibility, and mtime is the important information to preserve anyway
[v3] -> [v4]
rewrite functionality to populate directly from an input directory this is similar to mkfs.ext4 option.
[v4] -> [v5]
reorder patch to make it easier to review
reflow to keep code below 80 chars
use _() macro in prints
add SPDX headers to new files
fix comment styling
move from typedef to structs
move direntry handling to own function
[v5] -> [v6]
rebase on 6.14
[v6] -> [v7]
move functionality to common -p flag
add noatime flag to skip atime copy and set to current time set ctime/crtime to current time
preserve hardlinks
preserve extended attributes for all file/dir types
add fsxattr to copied files/dirs

Luca Di Maio (2):
proto: add ability to populate a filesystem from a directory
mkfs: modify -p flag to populate a filesystem from a directory

 man/man8/mkfs.xfs.8.in |  41 ++-
 mkfs/proto.c           | 643 ++++++++++++++++++++++++++++++++++++++++-
 mkfs/proto.h           |   2 +-
 mkfs/xfs_mkfs.c        |  19 +-
 4 files changed, 682 insertions(+), 23 deletions(-)

--
2.49.0

#1 lore raw

PATCH v7 1/2 proto: add ability to populate a filesystem from a directory

This patch implements the functionality to populate a newly created XFS filesystem directly from an existing directory structure.

It resuses existing protofile logic, it branches if input is a directory.

The population process steps are as follows:
- create the root inode before populating content
- recursively process nested directories
- handle regular files, directories, symlinks, char devices, block devices, and fifos
- preserve attributes (ownership, permissions)
- preserve mtime timestamps from source files to maintain file history
- possible to specify noatime=1 to use current time also for atime
- use current time for ctime/crtime
- preserve extended attributes and fsxattrs for all file types
- preserve hardlinks

This functionality makes it easier to create populated filesystems without having to write protofiles manually.
It's particularly useful for reproducible builds.

---
 mkfs/proto.c | 653 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 mkfs/proto.h |   2 +-
 2 files changed, 646 insertions(+), 9 deletions(-)
mkfs/proto.c +645 -8
index 7f56a3d8..23f7998b 100644
@@ -5,11 +5,17 @@
  */
 
 #include "libxfs.h"
+#include "xfs_inode.h"
+#include <fcntl.h>
+#include <linux/limits.h>
+#include <stdio.h>
+#include <sys/resource.h>
 #include <sys/stat.h>
 #include <sys/xattr.h>
 #include <linux/xattr.h>
 #include "libfrog/convert.h"
 #include "proto.h"
+#include <dirent.h>
 
 /*
  * Prototypes for internal functions.
@@ -22,6 +28,11 @@ static int newregfile(char **pp, char **fname);
 static void rtinit(xfs_mount_t *mp);
 static off_t filesize(int fd);
 static int slashes_are_spaces;
+static int noatime;
+static void populate_from_dir(struct xfs_mount *mp, struct xfs_inode *pip,
+				struct fsxattr *fsxp, char *cur_path);
+static void walk_dir(struct xfs_mount *mp, struct xfs_inode *pip,
+				struct fsxattr *fsxp, char *cur_path);
 
 /*
  * Use this for block reservations needed for mkfs's conditions
@@ -65,6 +76,18 @@ setup_proto(
 
 	if (!fname)
 		return dflt;
+
+	/*
+	 * handle directory inputs
+	 * in this case we noop and let successive
+	 * parse_proto() to handle the directory
+	 * input.
+	 */
+	if ((fd = open(fname, O_DIRECTORY)) > 0) {
+		close(fd);
+		return fname;
+	}
+
 	if ((fd = open(fname, O_RDONLY)) < 0 || (size = filesize(fd)) < 0) {
 		fprintf(stderr, _("%s: failed to open %s: %s\n"),
 			progname, fname, strerror(errno));
@@ -380,9 +403,17 @@ writeattr(
 
 	ret = fgetxattr(fd, attrname, valuebuf, valuelen);
 	if (ret < 0) {
-		if (errno == EOPNOTSUPP)
-			return;
-		fail(_("error collecting xattr value"), errno);
+		/*
+		 * in case of filedescriptors with O_PATH, fgetxattr() will
+		 * fail. let's try to fallback to lgetxattr() using input
+		 * path.
+		 */
+		ret = lgetxattr(fname, attrname, valuebuf, valuelen);
+		if (ret < 0) {
+			if (errno == EOPNOTSUPP)
+				return;
+			fail(_("error collecting xattr value"), errno);
+		}
 	}
 	if (ret == 0)
 		return;
@@ -426,9 +457,17 @@ writeattrs(
 
 	ret = flistxattr(fd, namebuf, XATTR_LIST_MAX);
 	if (ret < 0) {
-		if (errno == EOPNOTSUPP)
-			goto out_namebuf;
-		fail(_("error collecting xattr names"), errno);
+		/*
+		 * in case of filedescriptors with O_PATH, flistxattr() will
+		 * fail. let's try to fallback to llistxattr() using input
+		 * path.
+		 */
+		ret = llistxattr(fname, namebuf, XATTR_LIST_MAX);
+		if (ret < 0) {
+			if (errno == EOPNOTSUPP)
+				goto out_namebuf;
+			fail(_("error collecting xattr names"), errno);
+		}
 	}
 
 	p = namebuf;
@@ -934,10 +973,24 @@ parse_proto(
 	xfs_mount_t	*mp,
 	struct fsxattr	*fsx,
 	char		**pp,
-	int		proto_slashes_are_spaces)
+	int proto_slashes_are_spaces,
+	int proto_noatime)
 {
 	slashes_are_spaces = proto_slashes_are_spaces;
-	parseproto(mp, NULL, fsx, pp, NULL);
+	noatime = proto_noatime;
+
+	/*
+	 * in case of a file input, we will use the prototype file logic
+	 * else we will fallback to populate from dir.
+	 */
+	int fd;
+	if ((fd = open(*pp,  O_DIRECTORY)) < 0) {
+		parseproto(mp, NULL, fsx, pp, NULL);
+		return;
+	}
+
+	close(fd);
+	populate_from_dir(mp, NULL, fsx, *pp);
 }
 
 /* Create a sb-rooted metadata file. */
@@ -1171,3 +1224,587 @@ filesize(
 		return -1;
 	return stb.st_size;
 }
+
+/* Try to allow as many memfds as possible. */
+static void
+bump_max_fds(void)
+{
+	struct rlimit rlim = {};
+	int ret;
+
+	ret = getrlimit(RLIMIT_NOFILE, &rlim);
+	if (!ret) {
+		rlim.rlim_cur = rlim.rlim_max;
+		setrlimit(RLIMIT_NOFILE, &rlim);
+	}
+}
+
+static void
+writefsxattrs(
+		struct fsxattr *fsxp,
+		struct xfs_inode *ip)
+{
+	ip->i_projid = fsxp->fsx_projid;
+	ip->i_extsize = fsxp->fsx_extsize;
+	ip->i_diflags = xfs_flags2diflags(ip, fsxp->fsx_xflags);
+	if (xfs_has_v3inodes(ip->i_mount)) {
+		ip->i_diflags2 = xfs_flags2diflags2(ip, fsxp->fsx_xflags);
+		ip->i_cowextsize = fsxp->fsx_cowextsize;
+	}
+}
+
+static void
+writetimestamps(
+		struct xfs_inode *ip,
+		struct stat statbuf)
+{
+	struct timespec64 ts;
+
+	/*
+	 * Copy timestamps from source file to destination inode.
+	 * Usually reproducible archives will delete or not register
+	 * atime and ctime, for example:
+	 *    https://www.gnu.org/software/tar/manual/html_section/Reproducibility.html
+	 * hence we will only copy mtime, and let ctime/crtime be set to
+	 * current time.
+	 * atime will be copied over if noatime is false.
+	 */
+	ts.tv_sec = statbuf.st_mtim.tv_sec;
+	ts.tv_nsec = statbuf.st_mtim.tv_nsec;
+	inode_set_mtime_to_ts(VFS_I(ip), ts);
+
+	/*
+	 * in case of noatime option, we will not copy the atime
+	 * timestamp from source, but let it be set from gettimeofday()
+	 */
+	if (!noatime) {
+		ts.tv_sec = statbuf.st_atim.tv_sec;
+		ts.tv_nsec = statbuf.st_atim.tv_nsec;
+		inode_set_atime_to_ts(VFS_I(ip), ts);
+	}
+
+	return;
+}
+
+struct hardlink {
+	unsigned long i_ino;
+	struct xfs_inode *existing_ip;
+};
+
+struct hardlinks {
+	int count;
+	size_t size;
+	struct hardlink *entries;
+};
+
+/*
+ * keep track of source inodes that are from hardlinks
+ * so we can retrieve them when needed to setup in
+ * destination.
+ */
+static struct hardlinks *hardlink_tracker = { 0 };
+
+static void
+init_hardlink_tracker(void) {
+	hardlink_tracker = malloc(sizeof(struct hardlinks));
+	if (!hardlink_tracker)
+		fail(_("error allocating hardlinks tracking array"), errno);
+	memset(hardlink_tracker, 0, sizeof(struct hardlinks));
+
+	hardlink_tracker->count = 0;
+	hardlink_tracker->size = PATH_MAX;
+
+	hardlink_tracker->entries = malloc(
+			hardlink_tracker->size * sizeof(struct hardlink));
+	if (!hardlink_tracker->entries)
+		fail(_("error allocating hardlinks tracking array"), errno);
+}
+
+static void
+cleanup_hardlink_tracker(void) {
+	/*
+	 * cleanup all pending inodes, call libxfs_irele() on them
+	 * before freeing memory.
+	 */
+	for (int i = 0; i < hardlink_tracker->count; i++)
+		libxfs_irele(hardlink_tracker->entries[i].existing_ip);
+
+	free(hardlink_tracker->entries);
+	free(hardlink_tracker);
+}
+
+static struct xfs_inode*
+get_hardlink_src_inode(
+		unsigned long i_ino)
+{
+	for (int i = 0; i < hardlink_tracker->count; i++) {
+		if (hardlink_tracker->entries[i].i_ino == i_ino) {
+			return hardlink_tracker->entries[i].existing_ip;
+		}
+	}
+	return NULL;
+}
+
+static void
+track_hardlink_inode(
+		unsigned long i_ino,
+		struct xfs_inode *ip)
+{
+	if (hardlink_tracker->count >= hardlink_tracker->size) {
+		/*
+		 * double for smaller capacity.
+		 * instead grow by 25% steps for larger capacities.
+		 */
+		const size_t old_size = hardlink_tracker->size;
+		size_t new_size = old_size * 2;
+		if (old_size > 1024)
+			new_size = old_size + (old_size / 4);
+
+		struct hardlink *resized_array = realloc(
+				hardlink_tracker->entries,
+				new_size * sizeof(struct hardlink));
+		if (!resized_array) {
+			fail(_("error enlarging hardlinks tracking array"), errno);
+		}
+		memset(&resized_array[old_size],
+				0, (new_size - old_size) * sizeof(struct hardlink));
+
+		hardlink_tracker->entries = resized_array;
+		hardlink_tracker->size = new_size;
+	}
+
+	hardlink_tracker->entries[hardlink_tracker->count].i_ino = i_ino;
+	hardlink_tracker->entries[hardlink_tracker->count].existing_ip = ip;
+	hardlink_tracker->count++;
+}
+
+static int
+handle_hardlink(
+		struct xfs_mount *mp,
+		struct xfs_inode *pip,
+		struct fsxattr *fsxp,
+		int mode,
+		struct cred creds,
+		struct xfs_name xname,
+		int flags,
+		struct stat file_stat,
+		xfs_dev_t rdev,
+		int fd,
+		char *fname,
+		char *path)
+{
+	int error;
+	struct xfs_parent_args *ppargs = NULL;
+	struct xfs_inode *ip;
+	struct xfs_trans *tp;
+	tp = getres(mp, 0);
+	ppargs = newpptr(mp);
+
+	ip = get_hardlink_src_inode(file_stat.st_ino);
+	if (!ip) {
+		/*
+		 * we didn't find the hardlink inode, this means
+		 * it's the first time we see it, report error
+		 * so create_file() can continue handling the inode
+		 * as a regular file type, and later save
+		 * *ip in our buffer for future consumption.
+		 */
+		return 1;
+	}
+	/*
+	* In case the inode was already in our tracker
+	* we need to setup the hardlink and skip file
+	* copy.
+	*/
+	libxfs_trans_ijoin(tp, pip, 0);
+	libxfs_trans_ijoin(tp, ip, 0);
+
+	newdirent(mp, tp, pip, &xname, ip, ppargs);
+
+	/*
+	 * Increment the link count
+	 */
+	libxfs_bumplink(tp, ip);
+
+	/*
+	 * we won't need fd for hardlinks
+	 * so we close and reset it.
+	 */
+	if (fd >= 0)
+		close(fd);
+
+	libxfs_trans_log_inode(tp, ip, flags);
+	error = -libxfs_trans_commit(tp);
+	if (error)
+		fail(_("Error encountered creating file from prototype file"), error);
+
+	libxfs_parent_finish(mp, ppargs);
+
+	return 0;
+}
+
+static void
+create_file(
+		struct xfs_mount *mp,
+		struct xfs_inode *pip,
+		struct fsxattr *fsxp,
+		int mode,
+		struct cred creds,
+		struct xfs_name xname,
+		int flags,
+		struct stat file_stat,
+		xfs_dev_t rdev,
+		int fd,
+		char *fname,
+		char *path)
+{
+
+	int error;
+	struct xfs_parent_args *ppargs = NULL;
+	struct xfs_inode *ip;
+	struct xfs_trans *tp;
+
+	if (file_stat.st_nlink > 1) {
+		error = handle_hardlink(mp, pip, fsxp, mode, creds,
+			    xname, flags, file_stat,
+			    rdev, fd, fname, path);
+		/*
+		 * if no error is reported it means the hardlink has
+		 * been correctly found and set, so we don't need to
+		 * do anything else.
+		 */
+		if (!error)
+			return;
+		/*
+		 * if instead we have an error it means the hardlink
+		 * was not registered, so we proceed to treat it like
+		 * a regular file, and save it to our tracker later.
+		 */
+	}
+
+	tp = getres(mp, 0);
+	ppargs = newpptr(mp);
+
+	error = creatproto(&tp, pip, mode, rdev, &creds, fsxp, &ip);
+	if (error)
+		fail(_("Inode allocation failed"), error);
+	libxfs_trans_ijoin(tp, pip, 0);
+	newdirent(mp, tp, pip, &xname, ip, ppargs);
+
+	/*
+	 * copy over timestamps
+	 */
+	writetimestamps(ip, file_stat);
+	libxfs_trans_log_inode(tp, ip, flags);
+	error = -libxfs_trans_commit(tp);
+	if (error)
+		fail(_("Error encountered creating file from prototype file"), error);
+
+	libxfs_parent_finish(mp, ppargs);
+
+	/*
+	 * copy over file content, attributes,
+	 * extended attributes and timestamps
+	 *
+	 * hardlinks will be skipped as fd will
+	 * be closed before this.
+	 */
+	if (fd >= 0) {
+		writefile(ip, fname, fd);
+		writeattrs(ip, fname, fd);
+		writefsxattrs(fsxp, ip);
+		close(fd);
+	}
+
+	if (file_stat.st_nlink > 1)
+		/*
+		 * if we're here it means this is the first time we're
+		 * encountering an hardlink, so we need to store it and
+		 * skpi libxfs_irele() to keep it around.
+		 */
+		track_hardlink_inode(file_stat.st_ino, ip);
+	else
+		/*
+		 * We release the inode pointer only if we're dealing
+		 * with a regular file, we need to keep the original
+		 * inode pointer for hardlinks, they'll be released
+		 * at the end of the lifecycle when we cleanup the
+		 * hardlink_tracker.
+		 */
+		libxfs_irele(ip);
+}
+
+static void
+handle_direntry(
+		struct xfs_mount *mp,
+		struct xfs_inode *pip,
+		struct fsxattr *fsxp,
+		char *cur_path,
+		struct dirent *entry)
+{
+	char link_target[PATH_MAX];
+	char path[PATH_MAX];
+	int error;
+	int fd = -1;
+	int flags;
+	int majdev;
+	int mindev;
+	int mode;
+	off_t len;
+	struct cred creds;
+	struct stat file_stat;
+	struct xfs_name xname;
+	struct xfs_parent_args *ppargs = NULL;
+	struct xfs_inode *ip;
+	struct xfs_trans *tp;
+
+	/*
+	 * Skip "." and ".." directories to avoid looping
+	 */
+	if (strcmp(entry->d_name, ".") == 0 ||
+	    strcmp(entry->d_name, "..") == 0) {
+		return;
+	}
+
+	/*
+	 * Create the full path to the original file or directory
+	 */
+	snprintf(path, sizeof(path), "%s/%s", cur_path, entry->d_name);
+
+	if (lstat(path, &file_stat) < 0) {
+		fprintf(stderr, _("%s (error accessing)\n"), entry->d_name);
+		exit(1);
+	}
+
+	/*
+	 * symlinks will need to be opened with O_PATH to work, so we handle this
+	 * special case.
+	 */
+	int open_flags = O_NOFOLLOW | O_RDONLY | O_NOATIME;
+	if ((file_stat.st_mode & S_IFMT) == S_IFLNK) {
+		open_flags = O_NOFOLLOW | O_PATH;
+	}
+	if ((fd = open(path, open_flags)) < 0) {
+		fprintf(stderr, _("%s: cannot open %s: %s\n"), progname, path,
+			strerror(errno));
+		exit(1);
+	}
+
+	memset(&creds, 0, sizeof(creds));
+	creds.cr_uid = file_stat.st_uid;
+	creds.cr_gid = file_stat.st_gid;
+	xname.name = (unsigned char *)entry->d_name;
+	xname.len = strlen(entry->d_name);
+	xname.type = 0;
+	mode = file_stat.st_mode;
+	flags = XFS_ILOG_CORE;
+	switch (file_stat.st_mode & S_IFMT) {
+	case S_IFDIR:
+		tp = getres(mp, 0);
+		ppargs = newpptr(mp);
+		error = creatproto(&tp, pip, mode, 0, &creds, fsxp, &ip);
+		if (error)
+			fail(_("Inode allocation failed"), error);
+		libxfs_trans_ijoin(tp, pip, 0);
+		xname.type = XFS_DIR3_FT_DIR;
+		newdirent(mp, tp, pip, &xname, ip, ppargs);
+		libxfs_bumplink(tp, pip);
+		libxfs_trans_log_inode(tp, pip, XFS_ILOG_CORE);
+		newdirectory(mp, tp, ip, pip);
+
+		/*
+		 * copy over timestamps
+		 */
+		writetimestamps(ip, file_stat);
+
+		libxfs_trans_log_inode(tp, ip, flags);
+		error = -libxfs_trans_commit(tp);
+		if (error)
+			fail(_("Directory inode allocation failed."), error);
+
+		libxfs_parent_finish(mp, ppargs);
+		tp = NULL;
+
+		/*
+		 * copy over attributes
+		 */
+		writeattrs(ip, entry->d_name, fd);
+		writefsxattrs(fsxp, ip);
+		close(fd);
+
+		walk_dir(mp, ip, fsxp, path);
+
+		libxfs_irele(ip);
+		break;
+	case S_IFLNK:
+		len = readlink(path, link_target, PATH_MAX - 1);
+		tp = getres(mp, XFS_B_TO_FSB(mp, len));
+		ppargs = newpptr(mp);
+		error = creatproto(&tp, pip, mode, 0, &creds, fsxp, &ip);
+		if (error)
+			fail(_("Inode allocation failed"), error);
+		writesymlink(tp, ip, link_target, len);
+		libxfs_trans_ijoin(tp, pip, 0);
+		xname.type = XFS_DIR3_FT_SYMLINK;
+		newdirent(mp, tp, pip, &xname, ip, ppargs);
+
+		/*
+		 * copy over timestamps
+		 */
+		writetimestamps(ip, file_stat);
+
+		libxfs_trans_log_inode(tp, ip, flags);
+		error = -libxfs_trans_commit(tp);
+		if (error)
+			fail(_("Error encountered creating file from prototype file"),
+			     error);
+		libxfs_parent_finish(mp, ppargs);
+
+		/*
+		 * copy over attributes
+		 *
+		 * being a symlink we opened the filedescriptor with O_PATH
+		 * this will make flistxattr() and fgetxattr() fail, so we
+		 * will need to fallback to llistxattr() and lgetxattr(), this
+		 * will need the full path to the original file, not just the
+		 * entry name.
+		 */
+		writeattrs(ip, path, fd);
+		writefsxattrs(fsxp, ip);
+		close(fd);
+
+		libxfs_irele(ip);
+		break;
+	case S_IFREG:
+		xname.type = XFS_DIR3_FT_REG_FILE;
+		create_file(mp, pip, fsxp, mode, creds, xname, flags, file_stat,
+			    0, fd, entry->d_name, path);
+		break;
+	case S_IFCHR:
+		xname.type = XFS_DIR3_FT_CHRDEV;
+		majdev = major(file_stat.st_rdev);
+		mindev = minor(file_stat.st_rdev);
+		create_file(mp, pip, fsxp, mode, creds, xname, flags, file_stat,
+			    IRIX_MKDEV(majdev, mindev), fd, entry->d_name,
+			    path);
+		break;
+	case S_IFBLK:
+		xname.type = XFS_DIR3_FT_BLKDEV;
+		majdev = major(file_stat.st_rdev);
+		mindev = minor(file_stat.st_rdev);
+		create_file(mp, pip, fsxp, mode, creds, xname, flags, file_stat,
+			    IRIX_MKDEV(majdev, mindev), fd, entry->d_name,
+			    path);
+		break;
+	case S_IFIFO:
+		flags |= XFS_ILOG_DEV;
+		create_file(mp, pip, fsxp, mode, creds, xname, flags, file_stat,
+			    0, fd, entry->d_name, path);
+		break;
+	default:
+		break;
+	}
+}
+
+/*
+ * walk_dir will recursively list files and directories
+ * and populate the mountpoint *mp with them using handle_direntry().
+ */
+static void
+walk_dir(
+		struct xfs_mount *mp,
+		struct xfs_inode *pip,
+		struct fsxattr *fsxp,
+		char *cur_path)
+{
+	DIR *dir;
+	struct dirent *entry;
+
+	/*
+	 * open input directory and iterate over all entries in it.
+	 * when another directory is found, we will recursively call
+	 * walk_dir.
+	 */
+	if ((dir = opendir(cur_path)) == NULL)
+		fail(_("cannot open input dir"), 1);
+	while ((entry = readdir(dir)) != NULL) {
+		handle_direntry(mp, pip, fsxp, cur_path, entry);
+	}
+	closedir(dir);
+}
+
+static void
+populate_from_dir(
+		struct xfs_mount *mp,
+		struct xfs_inode *pip,
+		struct fsxattr *fsxp,
+		char *cur_path)
+{
+	int error;
+	int mode;
+	struct cred creds;
+	struct xfs_inode *ip;
+	struct xfs_trans *tp;
+
+	/*
+	 * we first ensure we have the root inode
+	 */
+	memset(&creds, 0, sizeof(creds));
+	creds.cr_uid = 0;
+	creds.cr_gid = 0;
+	mode = S_IRWXU | S_IRGRP | S_IXGRP | S_IROTH | S_IXOTH;
+	tp = getres(mp, 0);
+	error = creatproto(&tp, pip, mode | S_IFDIR, 0, &creds, fsxp, &ip);
+	if (error)
+		fail(_("Inode allocation failed"), error);
+	pip = ip;
+	mp->m_sb.sb_rootino = ip->i_ino;
+	libxfs_log_sb(tp);
+	newdirectory(mp, tp, ip, pip);
+	libxfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
+	error = -libxfs_trans_commit(tp);
+	if (error)
+		fail(_("Inode allocation failed"), error);
+
+	libxfs_parent_finish(mp, NULL);
+
+	/*
+	 * RT initialization.  Do this here to ensure that
+	 * the RT inodes get placed after the root inode.
+	 */
+	error = create_metadir(mp);
+	if (error)
+		fail(_("Creation of the metadata directory inode failed"), error);
+
+	rtinit(mp);
+
+	/*
+	 * by nature of walk_dir() we could be opening
+	 * a great number of fds for deeply nested directory
+	 * trees.
+	 * try to bump max fds limit.
+	 */
+	bump_max_fds();
+
+	/*
+	 * initialize the hardlinks tracker
+	 */
+	init_hardlink_tracker();
+	/*
+	 * now that we have a root inode, let's
+	 * walk the input dir and populate the partition
+	 */
+	walk_dir(mp, ip, fsxp, cur_path);
+
+	/*
+	 * cleanup hardlinks tracker
+	 */
+	cleanup_hardlink_tracker();
+
+	/*
+	 * we free up our root inode
+	 * only when we finished populating the
+	 * root filesystem
+	 */
+	libxfs_irele(ip);
+}
mkfs/proto.h +1 -1
index be1ceb45..fea416f6 100644
@@ -8,7 +8,7 @@
 
 char *setup_proto(char *fname);
 void parse_proto(struct xfs_mount *mp, struct fsxattr *fsx, char **pp,
-		int proto_slashes_are_spaces);
+		int proto_slashes_are_spaces, int proto_noatime);
 void res_failed(int err);
 
 #endif /* MKFS_PROTO_H_ */

--
2.49.0

#2 lore raw

Re: [PATCH v7 1/2] proto: add ability to populate a filesystem from a directory

Quoted 64 lines · Luca Di Maio

This patch implements the functionality to populate a newly created XFS filesystem directly from an existing directory structure.

It resuses existing protofile logic, it branches if input is a directory.

The population process steps are as follows:
- create the root inode before populating content
- recursively process nested directories
- handle regular files, directories, symlinks, char devices, block devices, and fifos
- preserve attributes (ownership, permissions)
- preserve mtime timestamps from source files to maintain file history
- possible to specify noatime=1 to use current time also for atime
- use current time for ctime/crtime
- preserve extended attributes and fsxattrs for all file types
- preserve hardlinks

This functionality makes it easier to create populated filesystems without having to write protofiles manually.
It's particularly useful for reproducible builds.

---
 mkfs/proto.c | 653 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 mkfs/proto.h |   2 +-
 2 files changed, 646 insertions(+), 9 deletions(-)
mkfs/proto.c +11 -0
index 7f56a3d8..23f7998b 100644
@@ -5,11 +5,17 @@
  */
 
 #include "libxfs.h"
+#include "xfs_inode.h"
+#include <fcntl.h>
+#include <linux/limits.h>
+#include <stdio.h>
+#include <sys/resource.h>
 #include <sys/stat.h>
 #include <sys/xattr.h>
 #include <linux/xattr.h>
 #include "libfrog/convert.h"
 #include "proto.h"
+#include <dirent.h>
 
 /*
  * Prototypes for internal functions.
@@ -22,6 +28,11 @@ static int newregfile(char **pp, char **fname);
 static void rtinit(xfs_mount_t *mp);
 static off_t filesize(int fd);
 static int slashes_are_spaces;
+static int noatime;
+static void populate_from_dir(struct xfs_mount *mp, struct xfs_inode *pip,
+				struct fsxattr *fsxp, char *cur_path);
+static void walk_dir(struct xfs_mount *mp, struct xfs_inode *pip,
+				struct fsxattr *fsxp, char *cur_path);
 
 /*
  * Use this for block reservations needed for mkfs's conditions
@@ -65,6 +76,18 @@ setup_proto(
 

mkfs/proto.c setup_proto( · line 86

Quoted 10 lines
mkfs/proto.c +8 -0
 	if (!fname)
 		return dflt;
+
+	/*
+	 * handle directory inputs
+	 * in this case we noop and let successive
+	 * parse_proto() to handle the directory
+	 * input.
+	 */
+	if ((fd = open(fname, O_DIRECTORY)) > 0) {

0 is a valid (if unlikely) fd return value for open. But the bigger problem is...

mkfs/proto.c setup_proto( · line 88

Quoted 2 lines
mkfs/proto.c +2 -0
+		close(fd);
+		return fname;

...that now you've reduced the coherence of this function by making it return either a buffer containing the contents of a protofile minus the first two lines; a default protofile excerpt if there was no filename, or ... the path argument, if the path happened to point to a directory.

How does the caller (or parse_proto) figure out what they're supposed to do with the protostring? (More on this below.)

Quoted 6 lines
mkfs/proto.c +2 -0
+	}
+
 	if ((fd = open(fname, O_RDONLY)) < 0 || (size = filesize(fd)) < 0) {
 		fprintf(stderr, _("%s: failed to open %s: %s\n"),
 			progname, fname, strerror(errno));

mkfs/proto.c writeattr( · line 409

Quoted 10 lines
mkfs/proto.c +4 -3
@@ -380,9 +403,17 @@ writeattr(
 
 	ret = fgetxattr(fd, attrname, valuebuf, valuelen);
 	if (ret < 0) {
-		if (errno == EOPNOTSUPP)
-			return;
-		fail(_("error collecting xattr value"), errno);
+		/*
+		 * in case of filedescriptors with O_PATH, fgetxattr() will
+		 * fail. let's try to fallback to lgetxattr() using input
+		 * path.

Fail how? Shouldn't we gate this retry on the specific error code returned by fgetxattr on an O_PATH fd?

Quoted 13 lines
mkfs/proto.c +7 -0
+		 */
+		ret = lgetxattr(fname, attrname, valuebuf, valuelen);
+		if (ret < 0) {
+			if (errno == EOPNOTSUPP)
+				return;
+			fail(_("error collecting xattr value"), errno);
+		}
 	}
 	if (ret == 0)
 		return;
@@ -426,9 +457,17 @@ writeattrs(
 
 	ret = flistxattr(fd, namebuf, XATTR_LIST_MAX);

mkfs/proto.c writeattrs( · line 465

Quoted 10 lines
mkfs/proto.c +6 -3
 	if (ret < 0) {
-		if (errno == EOPNOTSUPP)
-			goto out_namebuf;
-		fail(_("error collecting xattr names"), errno);
+		/*
+		 * in case of filedescriptors with O_PATH, flistxattr() will
+		 * fail. let's try to fallback to llistxattr() using input
+		 * path.
+		 */
+		ret = llistxattr(fname, namebuf, XATTR_LIST_MAX);

Same here.

Quoted 16 lines
mkfs/proto.c +7 -1
+		if (ret < 0) {
+			if (errno == EOPNOTSUPP)
+				goto out_namebuf;
+			fail(_("error collecting xattr names"), errno);
+		}
 	}
 
 	p = namebuf;
@@ -934,10 +973,24 @@ parse_proto(
 	xfs_mount_t	*mp,
 	struct fsxattr	*fsx,
 	char		**pp,
-	int		proto_slashes_are_spaces)
+	int proto_slashes_are_spaces,
+	int proto_noatime)
 {

mkfs/proto.c parse_proto( · line 987

Quoted 10 lines
mkfs/proto.c +8 -1
 	slashes_are_spaces = proto_slashes_are_spaces;
-	parseproto(mp, NULL, fsx, pp, NULL);
+	noatime = proto_noatime;
+
+	/*
+	 * in case of a file input, we will use the prototype file logic
+	 * else we will fallback to populate from dir.
+	 */
+	int fd;
+	if ((fd = open(*pp,  O_DIRECTORY)) < 0) {

Urrrk, here we just open the protostring returned by setup_proto and passed in here as @pp? Recall that if -p file= pointed to a regular file, then protostring contains everything except the first two lines.

So if you pass in a corrupt protofile:

/stand/diskboot
4872 110
/etc/

this line will see /etc, the directory open succeeds, and now we start importing /etc. That isn't the documented behavior of a protofile.

Also, you open the passed-in path twice. This is a minor point for mkfs, but what if the path changes from a directory to a regular file in between the two open(..., O_DIRECTORY) calls?

mkfs/proto.c parse_proto( · line 993

Quoted 6 lines
mkfs/proto.c +6 -0
+		parseproto(mp, NULL, fsx, pp, NULL);
+		return;
+	}
+
+	close(fd);
+	populate_from_dir(mp, NULL, fsx, *pp);

I'm not sure why you don't pass in the opened directory here. I'm also not sure why the second parameter exists here; it's always NULL.

Quoted 16 lines
mkfs/proto.c +9 -0
 }
 
 /* Create a sb-rooted metadata file. */
@@ -1171,3 +1224,587 @@ filesize(
 		return -1;
 	return stb.st_size;
 }
+
+/* Try to allow as many memfds as possible. */
+static void
+bump_max_fds(void)
+{
+	struct rlimit rlim = {};
+	int ret;
+
+	ret = getrlimit(RLIMIT_NOFILE, &rlim);

mkfs/proto.c filesize( · line 1245

Quoted 10 lines
mkfs/proto.c +10 -0
+	if (!ret) {
+		rlim.rlim_cur = rlim.rlim_max;
+		setrlimit(RLIMIT_NOFILE, &rlim);
+	}
+}
+
+static void
+writefsxattrs(
+		struct fsxattr *fsxp,
+		struct xfs_inode *ip)

Strange indenting here, the usual parameter declaration convention in xfs is:

struct xfs_inode *ip

^ one tab ^ another tab

also we usually put the inode first because this function modifies the inode...

Quoted 4 lines
mkfs/proto.c +4 -0
+{
+	ip->i_projid = fsxp->fsx_projid;
+	ip->i_extsize = fsxp->fsx_extsize;
+	ip->i_diflags = xfs_flags2diflags(ip, fsxp->fsx_xflags);

mkfs/proto.c filesize( · line 1259

Quoted 10 lines
mkfs/proto.c +10 -0
+	if (xfs_has_v3inodes(ip->i_mount)) {
+		ip->i_diflags2 = xfs_flags2diflags2(ip, fsxp->fsx_xflags);
+		ip->i_cowextsize = fsxp->fsx_cowextsize;
+	}
+}
+
+static void
+writetimestamps(
+		struct xfs_inode *ip,
+		struct stat statbuf)

...like you do here.

mkfs/proto.c filesize( · line 1261

Quoted 2 lines
mkfs/proto.c +2 -0
+{
+	struct timespec64 ts;

^ tab indent here

Quoted 15 lines
mkfs/proto.c +15 -0
+
+	/*
+	 * Copy timestamps from source file to destination inode.
+	 * Usually reproducible archives will delete or not register
+	 * atime and ctime, for example:
+	 *    https://www.gnu.org/software/tar/manual/html_section/Reproducibility.html
+	 * hence we will only copy mtime, and let ctime/crtime be set to
+	 * current time.
+	 * atime will be copied over if noatime is false.
+	 */
+	ts.tv_sec = statbuf.st_mtim.tv_sec;
+	ts.tv_nsec = statbuf.st_mtim.tv_nsec;
+	inode_set_mtime_to_ts(VFS_I(ip), ts);
+
+	/*

mkfs/proto.c filesize( · line 1286

Quoted 10 lines
mkfs/proto.c +10 -0
+	 * in case of noatime option, we will not copy the atime
+	 * timestamp from source, but let it be set from gettimeofday()
+	 */
+	if (!noatime) {
+		ts.tv_sec = statbuf.st_atim.tv_sec;
+		ts.tv_nsec = statbuf.st_atim.tv_nsec;
+		inode_set_atime_to_ts(VFS_I(ip), ts);
+	}
+
+	return;

No need to explicitly return from the bottom of a void-return function.

mkfs/proto.c filesize( · line 1290

Quoted 4 lines
mkfs/proto.c +4 -0
+}
+
+struct hardlink {
+	unsigned long i_ino;

ino_t ? Since that's what stat returns.

also struct field declarations need a tab between the type and the field name.

mkfs/proto.c filesize( · line 1295

Quoted 5 lines
mkfs/proto.c +5 -0
+	struct xfs_inode *existing_ip;
+};
+
+struct hardlinks {
+	int count;

This can overflow to negative if there are more than 2^32 hardlinked files in the source filesystem. My guess is that this will be rare, but you ought to program against that anyway...

mkfs/proto.c filesize( · line 1296

Quoted 1 line
mkfs/proto.c +1 -0
+	size_t size;

...since I'm guessing the upper limit on this data structure is whatever can be held in size_t.

Quoted 8 lines
mkfs/proto.c +8 -0
+	struct hardlink *entries;
+};
+
+/*
+ * keep track of source inodes that are from hardlinks
+ * so we can retrieve them when needed to setup in
+ * destination.
+ */

mkfs/proto.c filesize( · line 1314

Quoted 10 lines
mkfs/proto.c +10 -0
+static struct hardlinks *hardlink_tracker = { 0 };
+
+static void
+init_hardlink_tracker(void) {
+	hardlink_tracker = malloc(sizeof(struct hardlinks));
+	if (!hardlink_tracker)
+		fail(_("error allocating hardlinks tracking array"), errno);
+	memset(hardlink_tracker, 0, sizeof(struct hardlinks));
+
+	hardlink_tracker->count = 0;

You just memset the object to zero, this isn't necessary.

(use calloc to elide the memset?)

mkfs/proto.c filesize( · line 1315

Quoted 1 line
mkfs/proto.c +1 -0
+	hardlink_tracker->size = PATH_MAX;

Wait, why is the size being set to PATH_MAX? Are we tracking path strings somehow?

Quoted 8 lines
mkfs/proto.c +8 -0
+
+	hardlink_tracker->entries = malloc(
+			hardlink_tracker->size * sizeof(struct hardlink));
+	if (!hardlink_tracker->entries)
+		fail(_("error allocating hardlinks tracking array"), errno);
+}
+
+static void

mkfs/proto.c filesize( · line 1333

Quoted 10 lines
mkfs/proto.c +10 -0
+cleanup_hardlink_tracker(void) {
+	/*
+	 * cleanup all pending inodes, call libxfs_irele() on them
+	 * before freeing memory.
+	 */
+	for (int i = 0; i < hardlink_tracker->count; i++)
+		libxfs_irele(hardlink_tracker->entries[i].existing_ip);
+
+	free(hardlink_tracker->entries);
+	free(hardlink_tracker);

Probably want to set hardlink_tracker back to NULL to prevent UAF.

mkfs/proto.c filesize( · line 1340

Quoted 7 lines
mkfs/proto.c +7 -0
+}
+
+static struct xfs_inode*
+get_hardlink_src_inode(
+		unsigned long i_ino)
+{
+	for (int i = 0; i < hardlink_tracker->count; i++) {

Urk, linear search. Oh well, I guess we can switch to a hashtable if we get complaints about issues.

mkfs/proto.c filesize( · line 1342

Quoted 2 lines
mkfs/proto.c +2 -0
+		if (hardlink_tracker->entries[i].i_ino == i_ino) {
+			return hardlink_tracker->entries[i].existing_ip;

/me notes that this pins the hardlinked xfs_inode objects in memory. That's a risky thing for an xfsprogs utility to do because there's no inode cache like there is in the kernel. In other words, xfs_iget creates a brand new xfs_inode object even for the same inumber.

I /think/ that's not technically an issue in mkfs because it's single-threaded and never goes back to an existing inode. But you might consider changing struct hardlink to:

/*
* Map an inumber in the source filesystem to an inumber in the new
* filesystem
*/

struct hardlink {
	ino_t		src_ino;
	xfs_ino_t	dst_ino;
};

since xfs_inodes are fairly large objects.

Quoted 13 lines
mkfs/proto.c +13 -0
+		}
+	}
+	return NULL;
+}
+
+static void
+track_hardlink_inode(
+		unsigned long i_ino,
+		struct xfs_inode *ip)
+{
+	if (hardlink_tracker->count >= hardlink_tracker->size) {
+		/*
+		 * double for smaller capacity.

mkfs/proto.c filesize( · line 1365

Quoted 10 lines
mkfs/proto.c +10 -0
+		 * instead grow by 25% steps for larger capacities.
+		 */
+		const size_t old_size = hardlink_tracker->size;
+		size_t new_size = old_size * 2;
+		if (old_size > 1024)
+			new_size = old_size + (old_size / 4);
+
+		struct hardlink *resized_array = realloc(
+				hardlink_tracker->entries,
+				new_size * sizeof(struct hardlink));

At this point, is it safe to use reallocarray? Or will that cause problems with musl?

Quoted 19 lines
mkfs/proto.c +19 -0
+		if (!resized_array) {
+			fail(_("error enlarging hardlinks tracking array"), errno);
+		}
+		memset(&resized_array[old_size],
+				0, (new_size - old_size) * sizeof(struct hardlink));
+
+		hardlink_tracker->entries = resized_array;
+		hardlink_tracker->size = new_size;
+	}
+
+	hardlink_tracker->entries[hardlink_tracker->count].i_ino = i_ino;
+	hardlink_tracker->entries[hardlink_tracker->count].existing_ip = ip;
+	hardlink_tracker->count++;
+}
+
+static int
+handle_hardlink(
+		struct xfs_mount *mp,
+		struct xfs_inode *pip,

mkfs/proto.c filesize( · line 1394

Quoted 10 lines
mkfs/proto.c +10 -0
+		struct fsxattr *fsxp,
+		int mode,
+		struct cred creds,
+		struct xfs_name xname,
+		int flags,
+		struct stat file_stat,
+		xfs_dev_t rdev,
+		int fd,
+		char *fname,
+		char *path)

How many of these parameters are actually needed to create a hardlink?

mkfs/proto.c filesize( · line 1399

Quoted 5 lines
mkfs/proto.c +5 -0
+{
+	int error;
+	struct xfs_parent_args *ppargs = NULL;
+	struct xfs_inode *ip;
+	struct xfs_trans *tp;

Same odd indenting style.

mkfs/proto.c filesize( · line 1401

Quoted 2 lines
mkfs/proto.c +2 -0
+	tp = getres(mp, 0);
+	ppargs = newpptr(mp);
	struct xfs_trans	*tp = getres(mp, 0);
	struct xfs_parent_args	*ppargs = newpptr(mp);

mkfs/proto.c filesize( · line 1410

Quoted 9 lines
mkfs/proto.c +9 -0
+
+	ip = get_hardlink_src_inode(file_stat.st_ino);
+	if (!ip) {
+		/*
+		 * we didn't find the hardlink inode, this means
+		 * it's the first time we see it, report error
+		 * so create_file() can continue handling the inode
+		 * as a regular file type, and later save
+		 * *ip in our buffer for future consumption.

Documentation of the return value should be in a comment attached to the top of the function.

Quoted 57 lines
mkfs/proto.c +57 -0
+		 */
+		return 1;
+	}
+	/*
+	* In case the inode was already in our tracker
+	* we need to setup the hardlink and skip file
+	* copy.
+	*/
+	libxfs_trans_ijoin(tp, pip, 0);
+	libxfs_trans_ijoin(tp, ip, 0);
+
+	newdirent(mp, tp, pip, &xname, ip, ppargs);
+
+	/*
+	 * Increment the link count
+	 */
+	libxfs_bumplink(tp, ip);
+
+	/*
+	 * we won't need fd for hardlinks
+	 * so we close and reset it.
+	 */
+	if (fd >= 0)
+		close(fd);
+
+	libxfs_trans_log_inode(tp, ip, flags);
+	error = -libxfs_trans_commit(tp);
+	if (error)
+		fail(_("Error encountered creating file from prototype file"), error);
+
+	libxfs_parent_finish(mp, ppargs);
+
+	return 0;
+}
+
+static void
+create_file(
+		struct xfs_mount *mp,
+		struct xfs_inode *pip,
+		struct fsxattr *fsxp,
+		int mode,
+		struct cred creds,
+		struct xfs_name xname,
+		int flags,
+		struct stat file_stat,
+		xfs_dev_t rdev,
+		int fd,
+		char *fname,
+		char *path)
+{
+
+	int error;
+	struct xfs_parent_args *ppargs = NULL;
+	struct xfs_inode *ip;
+	struct xfs_trans *tp;
+
+	if (file_stat.st_nlink > 1) {

mkfs/proto.c filesize( · line 1477

Quoted 10 lines
mkfs/proto.c +10 -0
+		error = handle_hardlink(mp, pip, fsxp, mode, creds,
+			    xname, flags, file_stat,
+			    rdev, fd, fname, path);
+		/*
+		 * if no error is reported it means the hardlink has
+		 * been correctly found and set, so we don't need to
+		 * do anything else.
+		 */
+		if (!error)
+			return;

You know, if you inverted the polarity of handle_hardlink's return value, you could do:

	if (file_stat.st_nlink > 1 && handle_hardlink(...)) {
		close(fd);
		return;
	}
Quoted 23 lines
mkfs/proto.c +23 -0
+		/*
+		 * if instead we have an error it means the hardlink
+		 * was not registered, so we proceed to treat it like
+		 * a regular file, and save it to our tracker later.
+		 */
+	}
+
+	tp = getres(mp, 0);
+	ppargs = newpptr(mp);
+
+	error = creatproto(&tp, pip, mode, rdev, &creds, fsxp, &ip);
+	if (error)
+		fail(_("Inode allocation failed"), error);
+	libxfs_trans_ijoin(tp, pip, 0);
+	newdirent(mp, tp, pip, &xname, ip, ppargs);
+
+	/*
+	 * copy over timestamps
+	 */
+	writetimestamps(ip, file_stat);
+	libxfs_trans_log_inode(tp, ip, flags);
+	error = -libxfs_trans_commit(tp);
+	if (error)

mkfs/proto.c filesize( · line 1510

Quoted 10 lines
mkfs/proto.c +10 -0
+		fail(_("Error encountered creating file from prototype file"), error);
+
+	libxfs_parent_finish(mp, ppargs);
+
+	/*
+	 * copy over file content, attributes,
+	 * extended attributes and timestamps
+	 *
+	 * hardlinks will be skipped as fd will
+	 * be closed before this.

Aren't hardlinks skipped because this function returns if handle_hardlink() returns 0?

mkfs/proto.c filesize( · line 1512

Quoted 2 lines
mkfs/proto.c +2 -0
+	 */
+	if (fd >= 0) {

Do callers actually pass us negative fd numbers?

Quoted 50 lines
mkfs/proto.c +50 -0
+		writefile(ip, fname, fd);
+		writeattrs(ip, fname, fd);
+		writefsxattrs(fsxp, ip);
+		close(fd);
+	}
+
+	if (file_stat.st_nlink > 1)
+		/*
+		 * if we're here it means this is the first time we're
+		 * encountering an hardlink, so we need to store it and
+		 * skpi libxfs_irele() to keep it around.
+		 */
+		track_hardlink_inode(file_stat.st_ino, ip);
+	else
+		/*
+		 * We release the inode pointer only if we're dealing
+		 * with a regular file, we need to keep the original
+		 * inode pointer for hardlinks, they'll be released
+		 * at the end of the lifecycle when we cleanup the
+		 * hardlink_tracker.
+		 */
+		libxfs_irele(ip);
+}
+
+static void
+handle_direntry(
+		struct xfs_mount *mp,
+		struct xfs_inode *pip,
+		struct fsxattr *fsxp,
+		char *cur_path,
+		struct dirent *entry)
+{
+	char link_target[PATH_MAX];
+	char path[PATH_MAX];
+	int error;
+	int fd = -1;
+	int flags;
+	int majdev;
+	int mindev;
+	int mode;
+	off_t len;
+	struct cred creds;
+	struct stat file_stat;
+	struct xfs_name xname;
+	struct xfs_parent_args *ppargs = NULL;
+	struct xfs_inode *ip;
+	struct xfs_trans *tp;
+
+	/*
+	 * Skip "." and ".." directories to avoid looping

mkfs/proto.c filesize( · line 1572

Quoted 10 lines
mkfs/proto.c +10 -0
+	 */
+	if (strcmp(entry->d_name, ".") == 0 ||
+	    strcmp(entry->d_name, "..") == 0) {
+		return;
+	}
+
+	/*
+	 * Create the full path to the original file or directory
+	 */
+	snprintf(path, sizeof(path), "%s/%s", cur_path, entry->d_name);

Urrk, check the return value here!

/me wonders if you should declare /one/ char path[PATH_MAX] at the top of the call chain and modify that as we walk down the directory tree, rather than declaring a new (4k) stack variable every time we walk down another level.

mkfs/proto.c filesize( · line 1584

Quoted 12 lines
mkfs/proto.c +12 -0
+
+	if (lstat(path, &file_stat) < 0) {
+		fprintf(stderr, _("%s (error accessing)\n"), entry->d_name);
+		exit(1);
+	}
+
+	/*
+	 * symlinks will need to be opened with O_PATH to work, so we handle this
+	 * special case.
+	 */
+	int open_flags = O_NOFOLLOW | O_RDONLY | O_NOATIME;
+	if ((file_stat.st_mode & S_IFMT) == S_IFLNK) {

S_ISLNK() ?

mkfs/proto.c filesize( · line 1591

Quoted 7 lines
mkfs/proto.c +7 -0
+		open_flags = O_NOFOLLOW | O_PATH;
+	}
+	if ((fd = open(path, open_flags)) < 0) {
+		fprintf(stderr, _("%s: cannot open %s: %s\n"), progname, path,
+			strerror(errno));
+		exit(1);
+	}

Not sure you want to open() on a fifo, those will block until someone opens the other end.

mkfs/proto.c filesize( · line 1599

Quoted 8 lines
mkfs/proto.c +8 -0
+
+	memset(&creds, 0, sizeof(creds));
+	creds.cr_uid = file_stat.st_uid;
+	creds.cr_gid = file_stat.st_gid;
+	xname.name = (unsigned char *)entry->d_name;
+	xname.len = strlen(entry->d_name);
+	xname.type = 0;
+	mode = file_stat.st_mode;

Most of these could be moved to the variable declarations:

	struct cred creds = {
		.cr_uid	= file-stat.st_uid,
		.cr_gid	= file_stat.st_gid,
	};

and now future programmers don't have to be careful about uninitialized variables.

Quoted 65 lines
mkfs/proto.c +65 -0
+	flags = XFS_ILOG_CORE;
+	switch (file_stat.st_mode & S_IFMT) {
+	case S_IFDIR:
+		tp = getres(mp, 0);
+		ppargs = newpptr(mp);
+		error = creatproto(&tp, pip, mode, 0, &creds, fsxp, &ip);
+		if (error)
+			fail(_("Inode allocation failed"), error);
+		libxfs_trans_ijoin(tp, pip, 0);
+		xname.type = XFS_DIR3_FT_DIR;
+		newdirent(mp, tp, pip, &xname, ip, ppargs);
+		libxfs_bumplink(tp, pip);
+		libxfs_trans_log_inode(tp, pip, XFS_ILOG_CORE);
+		newdirectory(mp, tp, ip, pip);
+
+		/*
+		 * copy over timestamps
+		 */
+		writetimestamps(ip, file_stat);
+
+		libxfs_trans_log_inode(tp, ip, flags);
+		error = -libxfs_trans_commit(tp);
+		if (error)
+			fail(_("Directory inode allocation failed."), error);
+
+		libxfs_parent_finish(mp, ppargs);
+		tp = NULL;
+
+		/*
+		 * copy over attributes
+		 */
+		writeattrs(ip, entry->d_name, fd);
+		writefsxattrs(fsxp, ip);
+		close(fd);
+
+		walk_dir(mp, ip, fsxp, path);
+
+		libxfs_irele(ip);
+		break;
+	case S_IFLNK:
+		len = readlink(path, link_target, PATH_MAX - 1);
+		tp = getres(mp, XFS_B_TO_FSB(mp, len));
+		ppargs = newpptr(mp);
+		error = creatproto(&tp, pip, mode, 0, &creds, fsxp, &ip);
+		if (error)
+			fail(_("Inode allocation failed"), error);
+		writesymlink(tp, ip, link_target, len);
+		libxfs_trans_ijoin(tp, pip, 0);
+		xname.type = XFS_DIR3_FT_SYMLINK;
+		newdirent(mp, tp, pip, &xname, ip, ppargs);
+
+		/*
+		 * copy over timestamps
+		 */
+		writetimestamps(ip, file_stat);
+
+		libxfs_trans_log_inode(tp, ip, flags);
+		error = -libxfs_trans_commit(tp);
+		if (error)
+			fail(_("Error encountered creating file from prototype file"),
+			     error);
+		libxfs_parent_finish(mp, ppargs);
+
+		/*
+		 * copy over attributes

mkfs/proto.c filesize( · line 1674

Quoted 10 lines
mkfs/proto.c +10 -0
+		 *
+		 * being a symlink we opened the filedescriptor with O_PATH
+		 * this will make flistxattr() and fgetxattr() fail, so we
+		 * will need to fallback to llistxattr() and lgetxattr(), this
+		 * will need the full path to the original file, not just the
+		 * entry name.
+		 */
+		writeattrs(ip, path, fd);
+		writefsxattrs(fsxp, ip);
+		close(fd);

PS: symlink (really, non-directory) files can be hardlinked too.

$ ln -s moo cow
$ ls -d cow
lrwxrwxrwx 1 djwong djwong 3 Apr 28 10:05 cow -> moo
$ ln cow bar
$ ls -d cow bar
lrwxrwxrwx 2 djwong djwong 3 Apr 28 10:05 bar -> moo
lrwxrwxrwx 2 djwong djwong 3 Apr 28 10:05 cow -> moo
Quoted 45 lines
mkfs/proto.c +45 -0
+
+		libxfs_irele(ip);
+		break;
+	case S_IFREG:
+		xname.type = XFS_DIR3_FT_REG_FILE;
+		create_file(mp, pip, fsxp, mode, creds, xname, flags, file_stat,
+			    0, fd, entry->d_name, path);
+		break;
+	case S_IFCHR:
+		xname.type = XFS_DIR3_FT_CHRDEV;
+		majdev = major(file_stat.st_rdev);
+		mindev = minor(file_stat.st_rdev);
+		create_file(mp, pip, fsxp, mode, creds, xname, flags, file_stat,
+			    IRIX_MKDEV(majdev, mindev), fd, entry->d_name,
+			    path);
+		break;
+	case S_IFBLK:
+		xname.type = XFS_DIR3_FT_BLKDEV;
+		majdev = major(file_stat.st_rdev);
+		mindev = minor(file_stat.st_rdev);
+		create_file(mp, pip, fsxp, mode, creds, xname, flags, file_stat,
+			    IRIX_MKDEV(majdev, mindev), fd, entry->d_name,
+			    path);
+		break;
+	case S_IFIFO:
+		flags |= XFS_ILOG_DEV;
+		create_file(mp, pip, fsxp, mode, creds, xname, flags, file_stat,
+			    0, fd, entry->d_name, path);
+		break;
+	default:
+		break;
+	}
+}
+
+/*
+ * walk_dir will recursively list files and directories
+ * and populate the mountpoint *mp with them using handle_direntry().
+ */
+static void
+walk_dir(
+		struct xfs_mount *mp,
+		struct xfs_inode *pip,
+		struct fsxattr *fsxp,
+		char *cur_path)
+{

mkfs/proto.c filesize( · line 1729

Quoted 10 lines
mkfs/proto.c +10 -0
+	DIR *dir;
+	struct dirent *entry;
+
+	/*
+	 * open input directory and iterate over all entries in it.
+	 * when another directory is found, we will recursively call
+	 * walk_dir.
+	 */
+	if ((dir = opendir(cur_path)) == NULL)
+		fail(_("cannot open input dir"), 1);

Please report *which* directory couldn't be opened.

Quoted 86 lines
mkfs/proto.c +81 -0
+	while ((entry = readdir(dir)) != NULL) {
+		handle_direntry(mp, pip, fsxp, cur_path, entry);
+	}
+	closedir(dir);
+}
+
+static void
+populate_from_dir(
+		struct xfs_mount *mp,
+		struct xfs_inode *pip,
+		struct fsxattr *fsxp,
+		char *cur_path)
+{
+	int error;
+	int mode;
+	struct cred creds;
+	struct xfs_inode *ip;
+	struct xfs_trans *tp;
+
+	/*
+	 * we first ensure we have the root inode
+	 */
+	memset(&creds, 0, sizeof(creds));
+	creds.cr_uid = 0;
+	creds.cr_gid = 0;
+	mode = S_IRWXU | S_IRGRP | S_IXGRP | S_IROTH | S_IXOTH;
+	tp = getres(mp, 0);
+	error = creatproto(&tp, pip, mode | S_IFDIR, 0, &creds, fsxp, &ip);
+	if (error)
+		fail(_("Inode allocation failed"), error);
+	pip = ip;
+	mp->m_sb.sb_rootino = ip->i_ino;
+	libxfs_log_sb(tp);
+	newdirectory(mp, tp, ip, pip);
+	libxfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
+	error = -libxfs_trans_commit(tp);
+	if (error)
+		fail(_("Inode allocation failed"), error);
+
+	libxfs_parent_finish(mp, NULL);
+
+	/*
+	 * RT initialization.  Do this here to ensure that
+	 * the RT inodes get placed after the root inode.
+	 */
+	error = create_metadir(mp);
+	if (error)
+		fail(_("Creation of the metadata directory inode failed"), error);
+
+	rtinit(mp);
+
+	/*
+	 * by nature of walk_dir() we could be opening
+	 * a great number of fds for deeply nested directory
+	 * trees.
+	 * try to bump max fds limit.
+	 */
+	bump_max_fds();
+
+	/*
+	 * initialize the hardlinks tracker
+	 */
+	init_hardlink_tracker();
+	/*
+	 * now that we have a root inode, let's
+	 * walk the input dir and populate the partition
+	 */
+	walk_dir(mp, ip, fsxp, cur_path);
+
+	/*
+	 * cleanup hardlinks tracker
+	 */
+	cleanup_hardlink_tracker();
+
+	/*
+	 * we free up our root inode
+	 * only when we finished populating the
+	 * root filesystem
+	 */
+	libxfs_irele(ip);
+}
mkfs/proto.h +0 -0
index be1ceb45..fea416f6 100644

mkfs/proto.h line 11

Quoted 5 lines
mkfs/proto.h +1 -1
@@ -8,7 +8,7 @@
 
 char *setup_proto(char *fname);
 void parse_proto(struct xfs_mount *mp, struct fsxattr *fsx, char **pp,
-		int proto_slashes_are_spaces);
+		int proto_slashes_are_spaces, int proto_noatime);

Kinda wondering if parse_proto should just take a const struct cli_params pointer.

--D

Quoted 6 lines
mkfs/proto.h +0 -1
 void res_failed(int err);
 
 #endif /* MKFS_PROTO_H_ */
--

2.49.0

#3 lore raw

Re: [PATCH v7 1/2] proto: add ability to populate a filesystem from a directory

Thanks again Darrick,

Quoted 28 lines · Darrick J. Wong

mkfs/proto.c setup_proto( · line 86

Quoted 1 line · Luca Di Maio
mkfs/proto.c +1 -0
+	if ((fd = open(fname, O_DIRECTORY)) > 0) {

0 is a valid (if unlikely) fd return value for open. But the bigger problem is...

mkfs/proto.c setup_proto( · line 88

Quoted 1 line
mkfs/proto.c +1 -0
+		return fname;

...that now you've reduced the coherence of this function by making it return either a buffer containing the contents of a protofile minus the first two lines; a default protofile excerpt if there was no filename, or ... the path argument, if the path happened to point to a directory.

How does the caller (or parse_proto) figure out what they're supposed to do with the protostring? (More on this below.)

mkfs/proto.c parse_proto( · line 987

Quoted 1 line
mkfs/proto.c +1 -0
+	if ((fd = open(*pp,  O_DIRECTORY)) < 0) {

Urrrk, here we just open the protostring returned by setup_proto and passed in here as @pp? Recall that if -p file= pointed to a regular file, then protostring contains everything except the first two lines.

So if you pass in a corrupt protofile:

/stand/diskboot
4872 110
/etc/

Quoted 6 lines

this line will see /etc, the directory open succeeds, and now we start importing /etc. That isn't the documented behavior of a protofile.

Also, you open the passed-in path twice. This is a minor point for mkfs, but what if the path changes from a directory to a regular file in between the two open(..., O_DIRECTORY) calls?

Right, this is confusing, I'll do a more structured return type for the various cases

Quoted 9 lines

mkfs/proto.c writeattr( · line 409

Quoted 5 lines
mkfs/proto.c +4 -1
-		fail(_("error collecting xattr value"), errno);
+		/*
+		 * in case of filedescriptors with O_PATH, fgetxattr() will
+		 * fail. let's try to fallback to lgetxattr() using input
+		 * path.

Fail how? Shouldn't we gate this retry on the specific error code returned by fgetxattr on an O_PATH fd?

Quoted 4 lines

mkfs/proto.c writeattrs( · line 465

Quoted 1 line
mkfs/proto.c +1 -0
+		ret = llistxattr(fname, namebuf, XATTR_LIST_MAX);

Same here.

Right, will fix and explain the error

Quoted 9 lines

mkfs/proto.c parse_proto( · line 993

Quoted 6 lines
mkfs/proto.c +6 -0
+		parseproto(mp, NULL, fsx, pp, NULL);
+		return;
+	}
+
+	close(fd);
+	populate_from_dir(mp, NULL, fsx, *pp);

I'm not sure why you don't pass in the opened directory here. I'm also not sure why the second parameter exists here; it's always NULL.

Right, will cleanup

Quoted 8 lines

mkfs/proto.c filesize( · line 1245

Quoted 4 lines
mkfs/proto.c +4 -0
+static void
+writefsxattrs(
+		struct fsxattr *fsxp,
+		struct xfs_inode *ip)

Strange indenting here, the usual parameter declaration convention in xfs is:

Quoted 7 lines

struct xfs_inode *ip

^ one tab ^ another tab

also we usually put the inode first because this function modifies the inode...

Ack will fix those

Quoted 9 lines

mkfs/proto.c filesize( · line 1290

Quoted 4 lines
mkfs/proto.c +4 -0
+}
+
+struct hardlink {
+	unsigned long i_ino;

ino_t ? Since that's what stat returns.

also struct field declarations need a tab between the type and the field name.

Ack

Quoted 10 lines

mkfs/proto.c filesize( · line 1295

Quoted 5 lines
mkfs/proto.c +5 -0
+	struct xfs_inode *existing_ip;
+};
+
+struct hardlinks {
+	int count;

This can overflow to negative if there are more than 2^32 hardlinked files in the source filesystem. My guess is that this will be rare, but you ought to program against that anyway...

Quoted 4 lines

mkfs/proto.c filesize( · line 1296

Quoted 1 line
mkfs/proto.c +1 -0
+	size_t size;

...since I'm guessing the upper limit on this data structure is whatever can be held in size_t.

Right, will upgrate it to size_t

Quoted 21 lines
Quoted 8 lines
mkfs/proto.c +8 -0
+	struct hardlink *entries;
+};
+
+/*
+ * keep track of source inodes that are from hardlinks
+ * so we can retrieve them when needed to setup in
+ * destination.
+ */

mkfs/proto.c filesize( · line 1314

Quoted 10 lines
mkfs/proto.c +10 -0
+static struct hardlinks *hardlink_tracker = { 0 };
+
+static void
+init_hardlink_tracker(void) {
+	hardlink_tracker = malloc(sizeof(struct hardlinks));
+	if (!hardlink_tracker)
+		fail(_("error allocating hardlinks tracking array"), errno);
+	memset(hardlink_tracker, 0, sizeof(struct hardlinks));
+
+	hardlink_tracker->count = 0;

You just memset the object to zero, this isn't necessary.

Quoted 1 line

(use calloc to elide the memset?)

Yep will switch to calloc, thanks

Quoted 5 lines

mkfs/proto.c filesize( · line 1315

Quoted 1 line
mkfs/proto.c +1 -0
+	hardlink_tracker->size = PATH_MAX;

Wait, why is the size being set to PATH_MAX? Are we tracking path strings somehow?

Right, I tought to just reuse a constant already defined, I'll just set it explicitly to a number, it will be resized most likely anyway.

Quoted 10 lines

mkfs/proto.c filesize( · line 1340

Quoted 7 lines
mkfs/proto.c +7 -0
+}
+
+static struct xfs_inode*
+get_hardlink_src_inode(
+		unsigned long i_ino)
+{
+	for (int i = 0; i < hardlink_tracker->count; i++) {

Urk, linear search. Oh well, I guess we can switch to a hashtable if we get complaints about issues.

Yea, if it's a problem I can implement an hashtable, from my local testing using larger source directories (1.3mln inodes, ~400k hardlinks) the difference was actually just a few seconds (given that most of the time is doing i/o)

If we get complains we can come back to this

Quoted 16 lines

mkfs/proto.c filesize( · line 1342

Quoted 2 lines
mkfs/proto.c +2 -0
+		if (hardlink_tracker->entries[i].i_ino == i_ino) {
+			return hardlink_tracker->entries[i].existing_ip;

/me notes that this pins the hardlinked xfs_inode objects in memory. That's a risky thing for an xfsprogs utility to do because there's no inode cache like there is in the kernel. In other words, xfs_iget creates a brand new xfs_inode object even for the same inumber.

I /think/ that's not technically an issue in mkfs because it's single-threaded and never goes back to an existing inode. But you might consider changing struct hardlink to:

/*
* Map an inumber in the source filesystem to an inumber in the new
* filesystem
*/

Quoted 6 lines
struct hardlink {
	ino_t		src_ino;
	xfs_ino_t	dst_ino;
};

since xfs_inodes are fairly large objects.

Ack, didn't know about libxfs_iget()
I'll switch to only store src/dst ino_t, thanks

Quoted 6 lines

mkfs/proto.c filesize( · line 1365

Quoted 3 lines
mkfs/proto.c +3 -0
+		struct hardlink *resized_array = realloc(
+				hardlink_tracker->entries,
+				new_size * sizeof(struct hardlink));

At this point, is it safe to use reallocarray? Or will that cause problems with musl?

I don't think it's a problem, reading around I see that modern versions of musl (since around 2020-2021) do include reallocarray(). I'll switch to that

Quoted 8 lines
Quoted 8 lines
mkfs/proto.c +8 -0
+static int
+handle_hardlink(
+		struct xfs_mount *mp,
+		struct xfs_inode *pip,
+		struct fsxattr *fsxp,
+		int mode,
+		struct cred creds,
+		struct xfs_name xname,
Quoted 8 lines

mkfs/proto.c filesize( · line 1394

Quoted 6 lines
mkfs/proto.c +6 -0
+		int flags,
+		struct stat file_stat,
+		xfs_dev_t rdev,
+		int fd,
+		char *fname,
+		char *path)

How many of these parameters are actually needed to create a hardlink?

Will cleanup

Quoted 11 lines

mkfs/proto.c filesize( · line 1477

Quoted 7 lines
mkfs/proto.c +7 -0
+		/*
+		 * if no error is reported it means the hardlink has
+		 * been correctly found and set, so we don't need to
+		 * do anything else.
+		 */
+		if (!error)
+			return;

You know, if you inverted the polarity of handle_hardlink's return value, you could do:

Quoted 4 lines
	if (file_stat.st_nlink > 1 && handle_hardlink(...)) {
		close(fd);
		return;
	}

Right, that's better, thanks

Quoted 11 lines

mkfs/proto.c filesize( · line 1510

Quoted 7 lines
mkfs/proto.c +7 -0
+
+	/*
+	 * copy over file content, attributes,
+	 * extended attributes and timestamps
+	 *
+	 * hardlinks will be skipped as fd will
+	 * be closed before this.

Aren't hardlinks skipped because this function returns if handle_hardlink() returns 0?

Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+	 */
+	if (fd >= 0) {

Do callers actually pass us negative fd numbers?

Hardlink of a FIFO should be fd=-1 as we're going to skip open() for FIFOs (as you suggested)

Quoted 8 lines

mkfs/proto.c filesize( · line 1572

Quoted 5 lines
mkfs/proto.c +5 -0
+
+	/*
+	 * Create the full path to the original file or directory
+	 */
+	snprintf(path, sizeof(path), "%s/%s", cur_path, entry->d_name);

Urrk, check the return value here!

Quoted 4 lines

/me wonders if you should declare /one/ char path[PATH_MAX] at the top of the call chain and modify that as we walk down the directory tree, rather than declaring a new (4k) stack variable every time we walk down another level.

Yes it's better to have one buffer PATH_MAX long and handle one instance of that, will modify accordingly

Quoted 6 lines
Quoted 6 lines
mkfs/proto.c +6 -0
+
+	if (lstat(path, &file_stat) < 0) {
+		fprintf(stderr, _("%s (error accessing)\n"), entry->d_name);
+		exit(1);
+	}
+
Quoted 8 lines

mkfs/proto.c filesize( · line 1584

Quoted 6 lines
mkfs/proto.c +6 -0
+	/*
+	 * symlinks will need to be opened with O_PATH to work, so we handle this
+	 * special case.
+	 */
+	int open_flags = O_NOFOLLOW | O_RDONLY | O_NOATIME;
+	if ((file_stat.st_mode & S_IFMT) == S_IFLNK) {

S_ISLNK() ?

Ack

Quoted 10 lines

mkfs/proto.c filesize( · line 1591

Quoted 7 lines
mkfs/proto.c +7 -0
+		open_flags = O_NOFOLLOW | O_PATH;
+	}
+	if ((fd = open(path, open_flags)) < 0) {
+		fprintf(stderr, _("%s: cannot open %s: %s\n"), progname, path,
+			strerror(errno));
+		exit(1);
+	}

Not sure you want to open() on a fifo, those will block until someone opens the other end.

Right, will fix

Quoted 11 lines

mkfs/proto.c filesize( · line 1599

Quoted 8 lines
mkfs/proto.c +8 -0
+
+	memset(&creds, 0, sizeof(creds));
+	creds.cr_uid = file_stat.st_uid;
+	creds.cr_gid = file_stat.st_gid;
+	xname.name = (unsigned char *)entry->d_name;
+	xname.len = strlen(entry->d_name);
+	xname.type = 0;
+	mode = file_stat.st_mode;

Most of these could be moved to the variable declarations:

Quoted 7 lines
	struct cred creds = {
		.cr_uid	= file-stat.st_uid,
		.cr_gid	= file_stat.st_gid,
	};

and now future programmers don't have to be careful about uninitialized variables.

Right, thanks

Quoted 15 lines

mkfs/proto.c filesize( · line 1674

Quoted 12 lines
mkfs/proto.c +12 -0
+		/*
+		 * copy over attributes
+		 *
+		 * being a symlink we opened the filedescriptor with O_PATH
+		 * this will make flistxattr() and fgetxattr() fail, so we
+		 * will need to fallback to llistxattr() and lgetxattr(), this
+		 * will need the full path to the original file, not just the
+		 * entry name.
+		 */
+		writeattrs(ip, path, fd);
+		writefsxattrs(fsxp, ip);
+		close(fd);

PS: symlink (really, non-directory) files can be hardlinked too.

Quoted 7 lines
$ ln -s moo cow
$ ls -d cow
lrwxrwxrwx 1 djwong djwong 3 Apr 28 10:05 cow -> moo
$ ln cow bar
$ ls -d cow bar
lrwxrwxrwx 2 djwong djwong 3 Apr 28 10:05 bar -> moo
lrwxrwxrwx 2 djwong djwong 3 Apr 28 10:05 cow -> moo

Right, I didn't think of this, will modify accordingly

Quoted 10 lines

mkfs/proto.c filesize( · line 1729

Quoted 8 lines
mkfs/proto.c +8 -0
+
+	/*
+	 * open input directory and iterate over all entries in it.
+	 * when another directory is found, we will recursively call
+	 * walk_dir.
+	 */
+	if ((dir = opendir(cur_path)) == NULL)
+		fail(_("cannot open input dir"), 1);

Please report *which* directory couldn't be opened.

Ack

Thanks for the review
L.

#4 lore raw

Re: [PATCH v7 1/2] proto: add ability to populate a filesystem from a directory

On the way back from meetings and long distance travel, so just chiming in with a few points.

One is that the patch split still doesn't make sense to me. Patch 1 adds all the infrastructure which remains unused, and then patch 2 trivially wires it up. Please merge the two.

Quoted 7 lines
Quoted 2 lines

Urk, linear search. Oh well, I guess we can switch to a hashtable if we get complaints about issues.

Yea, if it's a problem I can implement an hashtable, from my local testing using larger source directories (1.3mln inodes, ~400k hardlinks) the difference was actually just a few seconds (given that most of the time is doing i/o)

Please add this to the commit log, it's always good to know about the trade offs and the data supporting it when later looking at the commit history.

#5 lore raw

PATCH v7 2/2 mkfs: modify -p flag to populate a filesystem from a directory

In reply to Luca Di Maio

right now the -p flag only supports a file input. this patch will add support to input a directory.
on directory input, the populate functionality to copy files into the root filesystem.

add noatime flag to popts, that will let the user choose if copy the atime timestamps from source directory.

add documentation for new functionalities in man pages.

---
 man/man8/mkfs.xfs.8.in | 41 +++++++++++++++++++++++++++++------------
 mkfs/xfs_mkfs.c        | 19 +++++++++++++++++--
 2 files changed, 46 insertions(+), 14 deletions(-)
man/man8/mkfs.xfs.8.in +29 -12
index 37e3a88e..f06a3c9c 100644
@@ -28,7 +28,7 @@ mkfs.xfs \- construct an XFS filesystem
 .I naming_options
 ] [
 .B \-p
-.I protofile_options
+.I prototype_options
 ] [
 .B \-q
 ] [
@@ -977,30 +977,39 @@ option set.
 .PP
 .PD 0
 .TP
-.BI \-p " protofile_options"
+.BI \-p " prototype_options"
 .TP
 .BI "Section Name: " [proto]
 .PD
-These options specify the protofile parameters for populating the filesystem.
+These options specify the prototype parameters for populating the filesystem.
 The valid
-.I protofile_options
+.I prototype_options
 are:
 .RS 1.2i
 .TP
-.BI [file=] protofile
+.BI [file=]
 The
 .B file=
 prefix is not required for this CLI argument for legacy reasons.
 If specified as a config file directive, the prefix is required.
-
+.TP
+.BI [file=] directory
 If the optional
 .PD
-.I protofile
-argument is given,
+.I prototype
+argument is given, and it's a directory,
 .B mkfs.xfs
-uses
-.I protofile
-as a prototype file and takes its directions from that file.
+will populate the root file system with the contents of the given directory.
+Content, timestamps, attributes and extended attributes are preserved
+for all file types.
+.TP
+.BI [file=] protofile
+If the optional
+.PD
+.I prototype
+argument is given, and it's a file,
+.B mkfs.xfs
+uses it as a prototype file and takes its directions from that file.
 The blocks and inodes specifiers in the
 .I protofile
 are provided for backwards compatibility, but are otherwise unused.
@@ -1136,8 +1145,16 @@ always terminated with the dollar (
 .B $
 ) token.
 .TP
+.BI noatime= value
+If set to 1, when we're populating the root filesystem from a directory (
+.B file=directory
+option)
+access times are NOT preserved and are set to the current time instead.
+Set to 0 to preserve access times from source files.
+By default, this is set to 1.
+.TP
 .BI slashes_are_spaces= value
-If set to 1, slashes ("/") in the first token of each line of the protofile
+If set to 1, slashes ("/") in the first token of each line of the prototype file
 are converted to spaces.
 This enables the creation of a filesystem containing filenames with spaces.
 By default, this is set to 0.
mkfs/xfs_mkfs.c +17 -2
index 3f4455d4..1715e3fb 100644
@@ -121,6 +121,7 @@ enum {
 
 enum {
 	P_FILE = 0,
+	P_NOATIME,
 	P_SLASHES,
 	P_MAX_OPTS,
 };
@@ -709,6 +710,7 @@ static struct opt_params popts = {
 	.ini_section = "proto",
 	.subopts = {
 		[P_FILE] = "file",
+		[P_NOATIME] = "noatime",
 		[P_SLASHES] = "slashes_are_spaces",
 		[P_MAX_OPTS] = NULL,
 	},
@@ -717,6 +719,12 @@ static struct opt_params popts = {
 		  .conflicts = { { NULL, LAST_CONFLICT } },
 		  .defaultval = SUBOPT_NEEDS_VAL,
 		},
+		{ .index = P_NOATIME,
+		  .conflicts = { { NULL, LAST_CONFLICT } },
+		  .minval = 0,
+		  .maxval = 1,
+		  .defaultval = 1,
+		},
 		{ .index = P_SLASHES,
 		  .conflicts = { { NULL, LAST_CONFLICT } },
 		  .minval = 0,
@@ -1022,7 +1030,6 @@ struct cli_params {
 
 	char	*cfgfile;
 	char	*protofile;
-
 	enum fsprop_autofsck autofsck;
 
 	/* parameters that depend on sector/block size being validated. */
@@ -1045,6 +1052,7 @@ struct cli_params {
 	int	lsunit;
 	int	is_supported;
 	int	proto_slashes_are_spaces;
+	int	proto_noatime;
 	int	data_concurrency;
 	int	log_concurrency;
 	int	rtvol_concurrency;
@@ -1170,6 +1178,7 @@ usage( void )
 /* naming */		[-n size=num,version=2|ci,ftype=0|1,parent=0|1]]\n\
 /* no-op info only */	[-N]\n\
 /* prototype file */	[-p fname]\n\
+/* populate from directory */	[-p dirname]\n\
 /* quiet */		[-q]\n\
 /* realtime subvol */	[-r extsize=num,size=num,rtdev=xxx,rgcount=n,rgsize=n,\n\
 			    concurrency=num]\n\
@@ -2067,6 +2076,9 @@ proto_opts_parser(
 	case P_SLASHES:
 		cli->proto_slashes_are_spaces = getnum(value, opts, subopt);
 		break;
+	case P_NOATIME:
+		cli->proto_noatime = getnum(value, opts, subopt);
+		break;
 	case P_FILE:
 		fallthrough;
 	default:
@@ -5181,6 +5193,7 @@ main(
 		.log_concurrency = -1, /* auto detect non-mechanical ddev */
 		.rtvol_concurrency = -1, /* auto detect non-mechanical rtdev */
 		.autofsck = FSPROP_AUTOFSCK_UNSET,
+		.proto_noatime = 1,
 	};
 	struct mkfs_params	cfg = {};
 
@@ -5480,7 +5493,9 @@ main(
 	/*
 	 * Allocate the root inode and anything else in the proto file.
 	 */
-	parse_proto(mp, &cli.fsx, &protostring, cli.proto_slashes_are_spaces);
+	parse_proto(mp, &cli.fsx, &protostring,
+			cli.proto_slashes_are_spaces,
+			cli.proto_noatime);
 
 	/*
 	 * Protect ourselves against possible stupidity

--
2.49.0

#6 lore raw

Re: [PATCH v7 2/2] mkfs: modify -p flag to populate a filesystem from a directory

Quoted 57 lines · Luca Di Maio

right now the -p flag only supports a file input. this patch will add support to input a directory.
on directory input, the populate functionality to copy files into the root filesystem.

add noatime flag to popts, that will let the user choose if copy the atime timestamps from source directory.

add documentation for new functionalities in man pages.

---
 man/man8/mkfs.xfs.8.in | 41 +++++++++++++++++++++++++++++------------
 mkfs/xfs_mkfs.c        | 19 +++++++++++++++++--
 2 files changed, 46 insertions(+), 14 deletions(-)
man/man8/mkfs.xfs.8.in +7 -6
index 37e3a88e..f06a3c9c 100644
@@ -28,7 +28,7 @@ mkfs.xfs \- construct an XFS filesystem
 .I naming_options
 ] [
 .B \-p
-.I protofile_options
+.I prototype_options
 ] [
 .B \-q
 ] [
@@ -977,30 +977,39 @@ option set.
 .PP
 .PD 0
 .TP
-.BI \-p " protofile_options"
+.BI \-p " prototype_options"
 .TP
 .BI "Section Name: " [proto]
 .PD
-These options specify the protofile parameters for populating the filesystem.
+These options specify the prototype parameters for populating the filesystem.
 The valid
-.I protofile_options
+.I prototype_options
 are:
 .RS 1.2i
 .TP
-.BI [file=] protofile
+.BI [file=]
 The
 .B file=
 prefix is not required for this CLI argument for legacy reasons.
 If specified as a config file directive, the prefix is required.
-
+.TP
+.BI [file=] directory
 If the optional
 .PD

man/man8/mkfs.xfs.8.in option set. · line 1003

Quoted 10 lines
man/man8/mkfs.xfs.8.in +4 -5
-.I protofile
-argument is given,
+.I prototype
+argument is given, and it's a directory,
 .B mkfs.xfs
-uses
-.I protofile
-as a prototype file and takes its directions from that file.
+will populate the root file system with the contents of the given directory.
+Content, timestamps, attributes and extended attributes are preserved

I thought this only preserved the atime and mtime timestamps?

man/man8/mkfs.xfs.8.in option set. · line 1010

Quoted 7 lines
man/man8/mkfs.xfs.8.in +7 -0
+for all file types.
+.TP
+.BI [file=] protofile
+If the optional
+.PD
+.I prototype
+argument is given, and it's a file,

"If the optional prototype argument is given and points to a regular file..."

(I think, technically speaking, you store a protofile on a block device and pass that in, but let's not encourage that kind of absurdity)

Quoted 6 lines
man/man8/mkfs.xfs.8.in +2 -0
+.B mkfs.xfs
+uses it as a prototype file and takes its directions from that file.
 The blocks and inodes specifiers in the
 .I protofile
 are provided for backwards compatibility, but are otherwise unused.

man/man8/mkfs.xfs.8.in always terminated with the dollar ( · line 1153

Quoted 9 lines
man/man8/mkfs.xfs.8.in +6 -0
@@ -1136,8 +1145,16 @@ always terminated with the dollar (
 .B $
 ) token.
 .TP
+.BI noatime= value
+If set to 1, when we're populating the root filesystem from a directory (
+.B file=directory
+option)
+access times are NOT preserved and are set to the current time instead.
+Set to 0 to preserve access times from source files.

I would say "Set to 0 to copy access timestamps from source files".

Though if the default is going to be "do not copy atime from sourcev file" then the option to enable copying of atime ought to be named "atime".

Quoted 42 lines
man/man8/mkfs.xfs.8.in +3 -1
+By default, this is set to 1.
+.TP
 .BI slashes_are_spaces= value
-If set to 1, slashes ("/") in the first token of each line of the protofile
+If set to 1, slashes ("/") in the first token of each line of the prototype file
 are converted to spaces.
 This enables the creation of a filesystem containing filenames with spaces.
 By default, this is set to 0.
mkfs/xfs_mkfs.c +8 -0
index 3f4455d4..1715e3fb 100644
@@ -121,6 +121,7 @@ enum {
 
 enum {
 	P_FILE = 0,
+	P_NOATIME,
 	P_SLASHES,
 	P_MAX_OPTS,
 };
@@ -709,6 +710,7 @@ static struct opt_params popts = {
 	.ini_section = "proto",
 	.subopts = {
 		[P_FILE] = "file",
+		[P_NOATIME] = "noatime",
 		[P_SLASHES] = "slashes_are_spaces",
 		[P_MAX_OPTS] = NULL,
 	},
@@ -717,6 +719,12 @@ static struct opt_params popts = {
 		  .conflicts = { { NULL, LAST_CONFLICT } },
 		  .defaultval = SUBOPT_NEEDS_VAL,
 		},
+		{ .index = P_NOATIME,
+		  .conflicts = { { NULL, LAST_CONFLICT } },
+		  .minval = 0,
+		  .maxval = 1,
+		  .defaultval = 1,
+		},
 		{ .index = P_SLASHES,
 		  .conflicts = { { NULL, LAST_CONFLICT } },
 		  .minval = 0,

mkfs/xfs_mkfs.c struct cli_params { · line 1025

Quoted 4 lines
mkfs/xfs_mkfs.c +0 -1
@@ -1022,7 +1030,6 @@ struct cli_params {
 
 	char	*cfgfile;
 	char	*protofile;
-

Extraneous change?

Quoted 12 lines
mkfs/xfs_mkfs.c +1 -0
 	enum fsprop_autofsck autofsck;
 
 	/* parameters that depend on sector/block size being validated. */
@@ -1045,6 +1052,7 @@ struct cli_params {
 	int	lsunit;
 	int	is_supported;
 	int	proto_slashes_are_spaces;
+	int	proto_noatime;
 	int	data_concurrency;
 	int	log_concurrency;
 	int	rtvol_concurrency;

mkfs/xfs_mkfs.c usage( void ) · line 1181

Quoted 4 lines
mkfs/xfs_mkfs.c +1 -0
@@ -1170,6 +1178,7 @@ usage( void )
 /* naming */		[-n size=num,version=2|ci,ftype=0|1,parent=0|1]]\n\
 /* no-op info only */	[-N]\n\
 /* prototype file */	[-p fname]\n\
+/* populate from directory */	[-p dirname]\n\

/* populate from directory */ [-p dirname,noatime=0|1]

--D

Quoted 35 lines
mkfs/xfs_mkfs.c +7 -2
 /* quiet */		[-q]\n\
 /* realtime subvol */	[-r extsize=num,size=num,rtdev=xxx,rgcount=n,rgsize=n,\n\
 			    concurrency=num]\n\
@@ -2067,6 +2076,9 @@ proto_opts_parser(
 	case P_SLASHES:
 		cli->proto_slashes_are_spaces = getnum(value, opts, subopt);
 		break;
+	case P_NOATIME:
+		cli->proto_noatime = getnum(value, opts, subopt);
+		break;
 	case P_FILE:
 		fallthrough;
 	default:
@@ -5181,6 +5193,7 @@ main(
 		.log_concurrency = -1, /* auto detect non-mechanical ddev */
 		.rtvol_concurrency = -1, /* auto detect non-mechanical rtdev */
 		.autofsck = FSPROP_AUTOFSCK_UNSET,
+		.proto_noatime = 1,
 	};
 	struct mkfs_params	cfg = {};
 
@@ -5480,7 +5493,9 @@ main(
 	/*
 	 * Allocate the root inode and anything else in the proto file.
 	 */
-	parse_proto(mp, &cli.fsx, &protostring, cli.proto_slashes_are_spaces);
+	parse_proto(mp, &cli.fsx, &protostring,
+			cli.proto_slashes_are_spaces,
+			cli.proto_noatime);
 
 	/*
 	 * Protect ourselves against possible stupidity
--

2.49.0

#7 lore raw

Re: [PATCH v7 2/2] mkfs: modify -p flag to populate a filesystem from a directory

Thanks Darrick for the review

Quoted 4 lines · Darrick J. Wong
Quoted 1 line · Luca Di Maio

+Content, timestamps, attributes and extended attributes are preserved

I thought this only preserved the atime and mtime timestamps?

Right, I'll be more specific

Quoted 11 lines

man/man8/mkfs.xfs.8.in option set. · line 1010

Quoted 7 lines
man/man8/mkfs.xfs.8.in +7 -0
+for all file types.
+.TP
+.BI [file=] protofile
+If the optional
+.PD
+.I prototype
+argument is given, and it's a file,

"If the optional prototype argument is given and points to a regular file..."

Quoted 2 lines

(I think, technically speaking, you store a protofile on a block device and pass that in, but let's not encourage that kind of absurdity)

Will modify, thanks

Quoted 18 lines
Quoted 6 lines

+.B mkfs.xfs

man/man8/mkfs.xfs.8.in +1 -0
+uses it as a prototype file and takes its directions from that file.
 The blocks and inodes specifiers in the
 .I protofile
 are provided for backwards compatibility, but are otherwise unused.

man/man8/mkfs.xfs.8.in always terminated with the dollar ( · line 1153

Quoted 9 lines
man/man8/mkfs.xfs.8.in +6 -0
@@ -1136,8 +1145,16 @@ always terminated with the dollar (
 .B $
 ) token.
 .TP
+.BI noatime= value
+If set to 1, when we're populating the root filesystem from a directory (
+.B file=directory
+option)
+access times are NOT preserved and are set to the current time instead.
+Set to 0 to preserve access times from source files.

I would say "Set to 0 to copy access timestamps from source files".

Quoted 3 lines

Though if the default is going to be "do not copy atime from sourcev file" then the option to enable copying of atime ought to be named "atime".

I was also thinking atime=1 would be better to not invert the logic, I was going with the noatime option for consistency with mount options, but I assume this has nothing to do with that, so we can use atime

Will modify to be atime

Quoted 3 lines

Review always terminated with the dollar ( · line 1139

Quoted 1 line
(quoted diff) +0 -1
-

Extraneous change?

Yep sorry

Quoted 5 lines

mkfs/xfs_mkfs.c usage( void ) · line 1181

Quoted 1 line
mkfs/xfs_mkfs.c +1 -0
+/* populate from directory */	[-p dirname]\n\

/* populate from directory */ [-p dirname,noatime=0|1]

--D

Ack thanks