Skip to contentSkip to conversation

introduce defrag to xfs_spaceman

PATCH In review Wengang Wang 4 participants 60 messages Jul 9, 2024 – Aug 2, 2024

lore
Share & tools
lore
#0 lore raw

cover introduce defrag to xfs_spaceman

This patch set introduces defrag to xfs_spaceman command. It has the functionality and features below (also subject to be added to man page, so please review):

defrag [-f free_space] [-i idle_time] [-s segment_size] [-n] [-a]
defrag defragments the specified XFS file online non-exclusively. The target XFS doesn't have to (and must not) be unmunted. When defragmentation is in progress, file IOs are served 'in parallel'. reflink feature must be enabled in the XFS.

Defragmentation and file IOs

The target file is virtually devided into many small segments. Segments are the smallest units for defragmentation. Each segment is defragmented one by one in a lock->defragment->unlock->idle manner. File IOs are blocked when the target file is locked and are served during the defragmentation idle time (file is unlocked). Though the file IOs can't really go in parallel, they are not blocked long. The locking time basically depends on the segment size. Smaller segments usually take less locking time and thus IOs are blocked shorterly, bigger segments usually need more locking time and IOs are blocked longer. Check -s and -i options to balance the defragmentation and IO service.

Temporary file

A temporary file is used for the defragmentation. The temporary file is created in the same directory as the target file is and is named ".xfsdefrag_<pid>". It is a sparse file and contains a defragmentation segment at a time. The temporary file is removed automatically when defragmentation is done or is cancelled by ctrl-c. It remains in case kernel crashes when defragmentation is going on. In that case, the temporary file has to be removed manaully.

Free blocks consumption

Defragmenation works by (trying) allocating new (contiguous) blocks, copying data and then freeing old (non-contig) blocks. Usually the number of old blocks to free equals to the number the newly allocated blocks. As a finally result, defragmentation doesn't consume free blocks. Well, that is true if the target file is not sharing blocks with other files. In case the target file contains shared blocks, those shared blocks won't be freed back to filesystem as they are still owned by other files. So defragmenation allocates more blocks than it frees. For existing XFS, free blocks might be over- committed when reflink snapshots were created. To avoid causing the XFS running into low free blocks state, this defragmentation excludes (partially) shared segments when the file system free blocks reaches a shreshold. Check the -f option.

Safty and consistency

The defragmentation file is guanrantted safe and data consistent for ctrl-c and kernel crash.

First extent share

Current kernel has routine for each segment defragmentation detecting if the file is sharing blocks. It takes long in case the target file contains huge number of extents and the shared ones, if there is, are at the end. The First extent share feature works around above issue by making the first serveral blocks shared. Seeing the first blocks are shared, the kernel routine ends quickly. The side effect is that the "share" flag would remain traget file. This feature is enabled by default and can be disabled by -n option.

extsize and cowextsize

According to kernel implementation, extsize and cowextsize could have following impacts to defragmentation: 1) non-zero extsize causes separated block allocations for each extent in the segment and those blocks are not contiguous. The segment remains same number of extents after defragmention (no effect). 2) When extsize and/or cowextsize are too big, a lot of pre-allocated blocks remain in memory for a while. When new IO comes to whose pre-allocated blocks Copy on Write happens and causes the file fragmented.

Readahead

Readahead tries to fetch the data blocks for next segment with less locking in backgroud during idle time. This feature is disabled by default, use -a to enable it.

The command takes the following options:
-f free_space
The shreshold of XFS free blocks in MiB. When free blocks are less than this number, (partially) shared segments are excluded from defragmentation. Default number is 1024

-i idle_time
The time in milliseconds, defragmentation enters idle state for this long after defragmenting a segment and before handing the next. Default number is TOBEDONE.

-s segment_size
The size limitation in bytes of segments. Minimium number is 4MiB, default number is 16MiB.

-n Disable the First extent share feature. Enabled by default.

-a Enable readahead feature, disabled by default.

We tested with real customer metadump with some different 'idle_time's and found 250ms is good pratice sleep time. Here comes some number of the test:

Test: running of defrag on the image file which is used for the back end of a block device in a
virtual machine. At the same time, fio is running at the same time inside virtual machine on that block device.

block device type:   NVME
File size:           200GiB
paramters to defrag: free_space: 1024 idle_time: 250 First_extent_share: enabled readahead: disabled
Defrag run time:     223 minutes
Number of extents:   6745489(before) -> 203571(after)
Fio read latency:    15.72ms(without defrag) -> 14.53ms(during defrag)
Fio write latency:   32.21ms(without defrag) -> 20.03ms(during defrag)

Wengang Wang (9):
xfsprogs: introduce defrag command to spaceman spaceman/defrag: pick up segments from target file spaceman/defrag: defrag segments
spaceman/defrag: ctrl-c handler
spaceman/defrag: exclude shared segments on low free space spaceman/defrag: workaround kernel xfs_reflink_try_clear_inode_flag() spaceman/defrag: sleeps between segments
spaceman/defrag: readahead for better performance spaceman/defrag: warn on extsize

 spaceman/Makefile |   2 +-
 spaceman/defrag.c | 788 ++++++++++++++++++++++++++++++++++++++++++++++
 spaceman/init.c   |   1 +
 spaceman/space.h  |   1 +
 4 files changed, 791 insertions(+), 1 deletion(-)
 create mode 100644 spaceman/defrag.c
Signature · 1 line
2.39.3 (Apple Git-146)
#1 lore raw

PATCH 1/9 xfsprogs: introduce defrag command to spaceman

Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Non-exclusive defragment
Here we are introducing the non-exclusive manner to defragment a file, especially for huge files, without blocking IO to it long. Non-exclusive defragmentation divides the whole file into small segments. For each segment, we lock the file, defragment the segment and unlock the file. Defragmenting the small segment doesn’t take long. File IO requests can get served between defragmenting segments before blocked long. Also we put (user adjustable) idle time between defragmenting two consecutive segments to balance the defragmentation and file IOs.

The first patch in the set checks for valid target files

Valid target files to defrag must:
1. be accessible for read/write
2. be regular files
3. be in XFS filesystem
4. the containing XFS has reflink enabled. This is not checked before starting defragmentation, but error would be reported later.

---
 spaceman/Makefile |   2 +-
 spaceman/defrag.c | 198 ++++++++++++++++++++++++++++++++++++++++++++++
 spaceman/init.c   |   1 +
 spaceman/space.h  |   1 +
 4 files changed, 201 insertions(+), 1 deletion(-)
 create mode 100644 spaceman/defrag.c
spaceman/Makefile +1 -1
index 1f048d54..9c00b20a 100644
@@ -7,7 +7,7 @@ include $(TOPDIR)/include/builddefs
 
 LTCOMMAND = xfs_spaceman
 HFILES = init.h space.h
-CFILES = info.c init.c file.c health.c prealloc.c trim.c
+CFILES = info.c init.c file.c health.c prealloc.c trim.c defrag.c
 LSRCFILES = xfs_info.sh
 
 LLDLIBS = $(LIBXCMD) $(LIBFROG)
spaceman/defrag.c new file +198 -0

198 lines not on this page · raw message

spaceman/init.c +1 -0
index cf1ff3cb..396f965c 100644
@@ -35,6 +35,7 @@ init_commands(void)
 	trim_init();
 	freesp_init();
 	health_init();
+	defrag_init();
 }
 
 static int
spaceman/space.h +1 -0
index 723209ed..c288aeb9 100644
@@ -26,6 +26,7 @@ extern void	help_init(void);
 extern void	prealloc_init(void);
 extern void	quit_init(void);
 extern void	trim_init(void);
+extern void	defrag_init(void);
 #ifdef HAVE_GETFSMAP
 extern void	freesp_init(void);
 #else
Signature · 1 line
2.39.3 (Apple Git-146)
#2 lore raw

Re: [PATCH 1/9] xfsprogs: introduce defrag command to spaceman

Quoted 108 lines · Wengang Wang

Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Non-exclusive defragment
Here we are introducing the non-exclusive manner to defragment a file, especially for huge files, without blocking IO to it long. Non-exclusive defragmentation divides the whole file into small segments. For each segment, we lock the file, defragment the segment and unlock the file. Defragmenting the small segment doesn’t take long. File IO requests can get served between defragmenting segments before blocked long. Also we put (user adjustable) idle time between defragmenting two consecutive segments to balance the defragmentation and file IOs.

The first patch in the set checks for valid target files

Valid target files to defrag must:
1. be accessible for read/write
2. be regular files
3. be in XFS filesystem
4. the containing XFS has reflink enabled. This is not checked before starting defragmentation, but error would be reported later.

---
 spaceman/Makefile |   2 +-
 spaceman/defrag.c | 198 ++++++++++++++++++++++++++++++++++++++++++++++
 spaceman/init.c   |   1 +
 spaceman/space.h  |   1 +
 4 files changed, 201 insertions(+), 1 deletion(-)
 create mode 100644 spaceman/defrag.c
spaceman/Makefile +1 -1
index 1f048d54..9c00b20a 100644
@@ -7,7 +7,7 @@ include $(TOPDIR)/include/builddefs
 
 LTCOMMAND = xfs_spaceman
 HFILES = init.h space.h
-CFILES = info.c init.c file.c health.c prealloc.c trim.c
+CFILES = info.c init.c file.c health.c prealloc.c trim.c defrag.c
 LSRCFILES = xfs_info.sh
 
 LLDLIBS = $(LIBXCMD) $(LIBFROG)
spaceman/defrag.c new file +57 -0
new file mode 100644
index 00000000..c9732984
@@ -0,0 +1,198 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2024 Oracle.
+ * All Rights Reserved.
+ */
+
+#include "libxfs.h"
+#include <linux/fiemap.h>
+#include <linux/fsmap.h>
+#include "libfrog/fsgeom.h"
+#include "command.h"
+#include "init.h"
+#include "libfrog/paths.h"
+#include "space.h"
+#include "input.h"
+
+/* defrag segment size limit in units of 512 bytes */
+#define MIN_SEGMENT_SIZE_LIMIT 8192 /* 4MiB */
+#define DEFAULT_SEGMENT_SIZE_LIMIT 32768 /* 16MiB */
+static int g_segment_size_lmt = DEFAULT_SEGMENT_SIZE_LIMIT;
+
+/* size of the defrag target file */
+static off_t g_defrag_file_size = 0;
+
+/* stats for the target file extents before defrag */
+struct ext_stats {
+	long	nr_ext_total;
+	long	nr_ext_unwritten;
+	long	nr_ext_shared;
+};
+static struct ext_stats	g_ext_stats;
+
+/*
+ * check if the target is a valid file to defrag
+ * also store file size
+ * returns:
+ * true for yes and false for no
+ */
+static bool
+defrag_check_file(char *path)
+{
+	struct statfs statfs_s;
+	struct stat stat_s;
+
+	if (access(path, F_OK|W_OK) == -1) {
+		if (errno == ENOENT)
+			fprintf(stderr, "file \"%s\" doesn't exist\n", path);
+		else
+			fprintf(stderr, "no access to \"%s\", %s\n", path,
+				strerror(errno));
+		return false;
+	}
+
+	if (stat(path, &stat_s) == -1) {
+		fprintf(stderr, "failed to get file info on \"%s\":  %s\n",
+			path, strerror(errno));
+		return false;

spaceman/defrag.c line 67

Quoted 10 lines
spaceman/defrag.c new file +10 -0
+	}
+
+	g_defrag_file_size = stat_s.st_size;
+
+	if (!S_ISREG(stat_s.st_mode)) {
+		fprintf(stderr, "\"%s\" is not a regular file\n", path);
+		return false;
+	}
+
+	if (statfs(path, &statfs_s) == -1) {

statfs is deprecated, please use fstatvfs.

Quoted 9 lines
spaceman/defrag.c new file +9 -0
+		fprintf(stderr, "failed to get FS info on \"%s\":  %s\n",
+			path, strerror(errno));
+		return false;
+	}
+
+	if (statfs_s.f_type != XFS_SUPER_MAGIC) {
+		fprintf(stderr, "\"%s\" is not a xfs file\n", path);
+		return false;
+	}

spaceman/defrag.c line 86

Quoted 10 lines
spaceman/defrag.c new file +10 -0
+
+	return true;
+}
+
+/*
+ * defragment a file
+ * return 0 if successfully done, 1 otherwise
+ */
+static int
+defrag_xfs_defrag(char *file_path) {

defrag_xfs_path() ?

Quoted 8 lines
spaceman/defrag.c new file +8 -0
+	int	max_clone_us = 0, max_unshare_us = 0, max_punch_us = 0;
+	long	nr_seg_defrag = 0, nr_ext_defrag = 0;
+	int	scratch_fd = -1, defrag_fd = -1;
+	char	tmp_file_path[PATH_MAX+1];
+	char	*defrag_dir;
+	struct fsxattr	fsx;
+	int	ret = 0;
+

spaceman/defrag.c line 104

Quoted 10 lines
spaceman/defrag.c new file +10 -0
+	fsx.fsx_nextents = 0;
+	memset(&g_ext_stats, 0, sizeof(g_ext_stats));
+
+	if (!defrag_check_file(file_path)) {
+		ret = 1;
+		goto out;
+	}
+
+	defrag_fd = open(file_path, O_RDWR);
+	if (defrag_fd == -1) {

Not sure why you check the path before opening it -- all those file and statvfs attributes that you collect there can change (or the entire fs gets unmounted) until you've pinned the fs by opening the file.

spaceman/defrag.c line 115

Quoted 11 lines
spaceman/defrag.c new file +11 -0
+		fprintf(stderr, "Opening %s failed. %s\n", file_path,
+			strerror(errno));
+		ret = 1;
+		goto out;
+	}
+
+	defrag_dir = dirname(file_path);
+	snprintf(tmp_file_path, PATH_MAX, "%s/.xfsdefrag_%d", defrag_dir,
+		getpid());
+	tmp_file_path[PATH_MAX] = 0;
+	scratch_fd = open(tmp_file_path, O_CREAT|O_EXCL|O_RDWR, 0600);

O_TMPFILE? Then you don't have to do this .xfsdefrag_XXX stuff.

Quoted 25 lines
spaceman/defrag.c new file +25 -0
+	if (scratch_fd == -1) {
+		fprintf(stderr, "Opening temporary file %s failed. %s\n",
+			tmp_file_path, strerror(errno));
+		ret = 1;
+		goto out;
+	}
+out:
+	if (scratch_fd != -1) {
+		close(scratch_fd);
+		unlink(tmp_file_path);
+	}
+	if (defrag_fd != -1) {
+		ioctl(defrag_fd, FS_IOC_FSGETXATTR, &fsx);
+		close(defrag_fd);
+	}
+
+	printf("Pre-defrag %ld extents detected, %ld are \"unwritten\","
+		"%ld are \"shared\"\n",
+		g_ext_stats.nr_ext_total, g_ext_stats.nr_ext_unwritten,
+		g_ext_stats.nr_ext_shared);
+	printf("Tried to defragment %ld extents in %ld segments\n",
+		nr_ext_defrag, nr_seg_defrag);
+	printf("Time stats(ms): max clone: %d, max unshare: %d,"
+	       " max punch_hole: %d\n",
+	       max_clone_us/1000, max_unshare_us/1000, max_punch_us/1000);

spaceman/defrag.c line 150

Quoted 10 lines
spaceman/defrag.c new file +10 -0
+	printf("Post-defrag %u extents detected\n", fsx.fsx_nextents);
+	return ret;
+}
+
+
+static void defrag_help(void)
+{
+	printf(_(
+"\n"
+"Defragemnt files on XFS where reflink is enabled. IOs to the target files \n"

"Defragment"

Quoted 22 lines
spaceman/defrag.c new file +22 -0
+"can be served durning the defragmentations.\n"
+"\n"
+" -s segment_size    -- specify the segment size in MiB, minmum value is 4 \n"
+"                       default is 16\n"));
+}
+
+static cmdinfo_t defrag_cmd;
+
+static int
+defrag_f(int argc, char **argv)
+{
+	int	i;
+	int	c;
+
+	while ((c = getopt(argc, argv, "s:")) != EOF) {
+		switch(c) {
+		case 's':
+			g_segment_size_lmt = atoi(optarg) * 1024 * 1024 / 512;
+			if (g_segment_size_lmt < MIN_SEGMENT_SIZE_LIMIT) {
+				g_segment_size_lmt = MIN_SEGMENT_SIZE_LIMIT;
+				printf("Using minimium segment size %d\n",
+					g_segment_size_lmt);

spaceman/defrag.c line 182

Quoted 10 lines
spaceman/defrag.c new file +10 -0
+			}
+			break;
+		default:
+			command_usage(&defrag_cmd);
+			return 1;
+		}
+	}
+
+	for (i = 0; i < filecount; i++)
+		defrag_xfs_defrag(filetable[i].name);

Pass in the whole filetable[i] and then you've already got an open fd and some validation that it's an xfs filesystem.

spaceman/defrag.c line 193

Quoted 11 lines
spaceman/defrag.c new file +11 -0
+	return 0;
+}
+void defrag_init(void)
+{
+	defrag_cmd.name		= "defrag";
+	defrag_cmd.altname	= "dfg";
+	defrag_cmd.cfunc	= defrag_f;
+	defrag_cmd.argmin	= 0;
+	defrag_cmd.argmax	= 4;
+	defrag_cmd.args		= "[-s segment_size]";
+	defrag_cmd.flags	= CMD_FLAG_ONESHOT;

IIRC if you don't set CMD_FLAG_FOREIGN_OK then the command processor won't let this command get run against a non-xfs file.

--D

Quoted 33 lines
spaceman/defrag.c new file +5 -0
+	defrag_cmd.oneline	= _("Defragment XFS files");
+	defrag_cmd.help		= defrag_help;
+
+	add_command(&defrag_cmd);
+}
spaceman/init.c +1 -0
index cf1ff3cb..396f965c 100644
@@ -35,6 +35,7 @@ init_commands(void)
 	trim_init();
 	freesp_init();
 	health_init();
+	defrag_init();
 }
 
 static int
spaceman/space.h +1 -0
index 723209ed..c288aeb9 100644
@@ -26,6 +26,7 @@ extern void	help_init(void);
 extern void	prealloc_init(void);
 extern void	quit_init(void);
 extern void	trim_init(void);
+extern void	defrag_init(void);
 #ifdef HAVE_GETFSMAP
 extern void	freesp_init(void);
 #else
Signature · 1 line
2.39.3 (Apple Git-146)
#3 lore raw

Re: [PATCH 1/9] xfsprogs: introduce defrag command to spaceman

Hi Darrick,
Thanks for review, pls check in lines.

Quoted 115 lines · Darrick J. Wong
Quoted 112 lines · Wengang Wang

Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Non-exclusive defragment
Here we are introducing the non-exclusive manner to defragment a file, especially for huge files, without blocking IO to it long. Non-exclusive defragmentation divides the whole file into small segments. For each segment, we lock the file, defragment the segment and unlock the file. Defragmenting the small segment doesn’t take long. File IO requests can get served between defragmenting segments before blocked long. Also we put (user adjustable) idle time between defragmenting two consecutive segments to balance the defragmentation and file IOs.

The first patch in the set checks for valid target files

Valid target files to defrag must:
1. be accessible for read/write
2. be regular files
3. be in XFS filesystem
4. the containing XFS has reflink enabled. This is not checked before starting defragmentation, but error would be reported later.

---

spaceman/Makefile |   2 +-
spaceman/defrag.c | 198 ++++++++++++++++++++++++++++++++++++++++++++++
spaceman/init.c   |   1 +
spaceman/space.h  |   1 +
4 files changed, 201 insertions(+), 1 deletion(-)

create mode 100644 spaceman/defrag.c

spaceman/Makefile +0 -0
index 1f048d54..9c00b20a 100644
@@ -7,7 +7,7 @@ include $(TOPDIR)/include/builddefs
 

LTCOMMAND = xfs_spaceman
HFILES = init.h space.h

spaceman/Makefile +1 -1
-CFILES = info.c init.c file.c health.c prealloc.c trim.c
+CFILES = info.c init.c file.c health.c prealloc.c trim.c defrag.c

LSRCFILES = xfs_info.sh

LLDLIBS = $(LIBXCMD) $(LIBFROG)

spaceman/defrag.c new file +61 -0
new file mode 100644
index 00000000..c9732984
@@ -0,0 +1,198 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2024 Oracle.
+ * All Rights Reserved.
+ */
+
+#include "libxfs.h"
+#include <linux/fiemap.h>
+#include <linux/fsmap.h>
+#include "libfrog/fsgeom.h"
+#include "command.h"
+#include "init.h"
+#include "libfrog/paths.h"
+#include "space.h"
+#include "input.h"
+
+/* defrag segment size limit in units of 512 bytes */
+#define MIN_SEGMENT_SIZE_LIMIT 8192 /* 4MiB */
+#define DEFAULT_SEGMENT_SIZE_LIMIT 32768 /* 16MiB */
+static int g_segment_size_lmt = DEFAULT_SEGMENT_SIZE_LIMIT;
+
+/* size of the defrag target file */
+static off_t g_defrag_file_size = 0;
+
+/* stats for the target file extents before defrag */
+struct ext_stats {
+ long nr_ext_total;
+ long nr_ext_unwritten;
+ long nr_ext_shared;
+};
+static struct ext_stats g_ext_stats;
+
+/*
+ * check if the target is a valid file to defrag
+ * also store file size
+ * returns:
+ * true for yes and false for no
+ */
+static bool
+defrag_check_file(char *path)
+{
+ struct statfs statfs_s;
+ struct stat stat_s;
+
+ if (access(path, F_OK|W_OK) == -1) {
+ if (errno == ENOENT)
+ fprintf(stderr, "file \"%s\" doesn't exist\n", path);
+ else
+ fprintf(stderr, "no access to \"%s\", %s\n", path,
+ strerror(errno));
+ return false;
+ }
+
+ if (stat(path, &stat_s) == -1) {
+ fprintf(stderr, "failed to get file info on \"%s\":  %s\n",
+ path, strerror(errno));
+ return false;
+ }
+
+ g_defrag_file_size = stat_s.st_size;
+
Quoted 8 lines

spaceman/defrag.c line 67

Quoted 6 lines
spaceman/defrag.c new file +6 -0
+ if (!S_ISREG(stat_s.st_mode)) {
+ fprintf(stderr, "\"%s\" is not a regular file\n", path);
+ return false;
+ }
+
+ if (statfs(path, &statfs_s) == -1) {

statfs is deprecated, please use fstatvfs.

OK, will move to fstatvfs.

Quoted 14 lines
Quoted 14 lines
spaceman/defrag.c new file +13 -0
+ fprintf(stderr, "failed to get FS info on \"%s\":  %s\n",
+ path, strerror(errno));
+ return false;
+ }
+
+ if (statfs_s.f_type != XFS_SUPER_MAGIC) {
+ fprintf(stderr, "\"%s\" is not a xfs file\n", path);
+ return false;
+ }
+
+ return true;
+}
+
Quoted 8 lines

spaceman/defrag.c line 86

Quoted 6 lines
spaceman/defrag.c new file +6 -0
+/*
+ * defragment a file
+ * return 0 if successfully done, 1 otherwise
+ */
+static int
+defrag_xfs_defrag(char *file_path) {

defrag_xfs_path() ?

OK.

Quoted 15 lines
Quoted 15 lines
spaceman/defrag.c new file +14 -0
+ int max_clone_us = 0, max_unshare_us = 0, max_punch_us = 0;
+ long nr_seg_defrag = 0, nr_ext_defrag = 0;
+ int scratch_fd = -1, defrag_fd = -1;
+ char tmp_file_path[PATH_MAX+1];
+ char *defrag_dir;
+ struct fsxattr fsx;
+ int ret = 0;
+
+ fsx.fsx_nextents = 0;
+ memset(&g_ext_stats, 0, sizeof(g_ext_stats));
+
+ if (!defrag_check_file(file_path)) {
+ ret = 1;
+ goto out;
Quoted 8 lines

spaceman/defrag.c line 104

Quoted 4 lines
spaceman/defrag.c new file +4 -0
+ }
+
+ defrag_fd = open(file_path, O_RDWR);
+ if (defrag_fd == -1) {

Not sure why you check the path before opening it -- all those file and statvfs attributes that you collect there can change (or the entire fs gets unmounted) until you've pinned the fs by opening the file.

The idea comes from internal reviews hoping some explicit reasons why Defrag failed. Those reasons include:
1) if user has permission to access the target file.
2) if the species path exist (when moving to spaceman, spaceman takes care of it)
3) if the specified is a regular file
4) if the target file is a XFS file

Thing might change after checking and opening, but that’s very rare case and user is responsible for that change rather than this tool.

Quoted 13 lines

spaceman/defrag.c line 115

Quoted 12 lines
spaceman/defrag.c new file +11 -0
+ fprintf(stderr, "Opening %s failed. %s\n", file_path,
+ strerror(errno));
+ ret = 1;
+ goto out;
+ }
+
+ defrag_dir = dirname(file_path);
+ snprintf(tmp_file_path, PATH_MAX, "%s/.xfsdefrag_%d", defrag_dir,
+ getpid());
+ tmp_file_path[PATH_MAX] = 0;
+ scratch_fd = open(tmp_file_path, O_CREAT|O_EXCL|O_RDWR, 0600);
Quoted 2 lines

O_TMPFILE? Then you don't have to do this .xfsdefrag_XXX stuff.

My first first version was using O_TMPFILE. But clone failed somehow (Don’t remember the details). I retried O_TMPFILE, it’s working now. So will move to use O_TMPFILE.

Quoted 29 lines
Quoted 29 lines
spaceman/defrag.c new file +29 -0
+ if (scratch_fd == -1) {
+ fprintf(stderr, "Opening temporary file %s failed. %s\n",
+ tmp_file_path, strerror(errno));
+ ret = 1;
+ goto out;
+ }
+out:
+ if (scratch_fd != -1) {
+ close(scratch_fd);
+ unlink(tmp_file_path);
+ }
+ if (defrag_fd != -1) {
+ ioctl(defrag_fd, FS_IOC_FSGETXATTR, &fsx);
+ close(defrag_fd);
+ }
+
+ printf("Pre-defrag %ld extents detected, %ld are \"unwritten\","
+ "%ld are \"shared\"\n",
+ g_ext_stats.nr_ext_total, g_ext_stats.nr_ext_unwritten,
+ g_ext_stats.nr_ext_shared);
+ printf("Tried to defragment %ld extents in %ld segments\n",
+ nr_ext_defrag, nr_seg_defrag);
+ printf("Time stats(ms): max clone: %d, max unshare: %d,"
+        " max punch_hole: %d\n",
+        max_clone_us/1000, max_unshare_us/1000, max_punch_us/1000);
+ printf("Post-defrag %u extents detected\n", fsx.fsx_nextents);
+ return ret;
+}
+
Quoted 8 lines

spaceman/defrag.c line 150

Quoted 6 lines
spaceman/defrag.c new file +6 -0
+
+static void defrag_help(void)
+{
+ printf(_(
+"\n"
+"Defragemnt files on XFS where reflink is enabled. IOs to the target files \n"

"Defragment"

OK.

Quoted 28 lines
Quoted 28 lines
spaceman/defrag.c new file +27 -0
+"can be served durning the defragmentations.\n"
+"\n"
+" -s segment_size    -- specify the segment size in MiB, minmum value is 4 \n"
+"                       default is 16\n"));
+}
+
+static cmdinfo_t defrag_cmd;
+
+static int
+defrag_f(int argc, char **argv)
+{
+ int i;
+ int c;
+
+ while ((c = getopt(argc, argv, "s:")) != EOF) {
+ switch(c) {
+ case 's':
+ g_segment_size_lmt = atoi(optarg) * 1024 * 1024 / 512;
+ if (g_segment_size_lmt < MIN_SEGMENT_SIZE_LIMIT) {
+ g_segment_size_lmt = MIN_SEGMENT_SIZE_LIMIT;
+ printf("Using minimium segment size %d\n",
+ g_segment_size_lmt);
+ }
+ break;
+ default:
+ command_usage(&defrag_cmd);
+ return 1;
Quoted 8 lines

spaceman/defrag.c line 182

Quoted 5 lines
spaceman/defrag.c new file +5 -0
+ }
+ }
+
+ for (i = 0; i < filecount; i++)
+ defrag_xfs_defrag(filetable[i].name);

Pass in the whole filetable[i] and then you've already got an open fd and some validation that it's an xfs filesystem.

Good to know.

Quoted 16 lines

spaceman/defrag.c line 193

Quoted 12 lines
spaceman/defrag.c new file +11 -0
+ return 0;
+}
+void defrag_init(void)
+{
+ defrag_cmd.name = "defrag";
+ defrag_cmd.altname = "dfg";
+ defrag_cmd.cfunc = defrag_f;
+ defrag_cmd.argmin = 0;
+ defrag_cmd.argmax = 4;
+ defrag_cmd.args = "[-s segment_size]";
+ defrag_cmd.flags = CMD_FLAG_ONESHOT;

IIRC if you don't set CMD_FLAG_FOREIGN_OK then the command processor won't let this command get run against a non-xfs file.

OK.

Thanks,
Winging

Quoted 33 lines

--D

Quoted 31 lines
spaceman/defrag.c new file +5 -0
+ defrag_cmd.oneline = _("Defragment XFS files");
+ defrag_cmd.help = defrag_help;
+
+ add_command(&defrag_cmd);
+}
spaceman/init.c +1 -0
index cf1ff3cb..396f965c 100644
@@ -35,6 +35,7 @@ init_commands(void)
 trim_init();
 freesp_init();
 health_init();
+ defrag_init();
 }
 

static int

spaceman/space.h +1 -0
index 723209ed..c288aeb9 100644
@@ -26,6 +26,7 @@ extern void help_init(void);
 extern void prealloc_init(void);
 extern void quit_init(void);
 extern void trim_init(void);
+extern void defrag_init(void);
 #ifdef HAVE_GETFSMAP
 extern void freesp_init(void);
 #else
Signature · 1 line
2.39.3 (Apple Git-146)
#4 lore raw

Re: [PATCH 1/9] xfsprogs: introduce defrag command to spaceman

Quoted 248 lines · Wengang Wang

Hi Darrick,
Thanks for review, pls check in lines.

Quoted 115 lines · Darrick J. Wong
Quoted 112 lines · Wengang Wang

Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Non-exclusive defragment
Here we are introducing the non-exclusive manner to defragment a file, especially for huge files, without blocking IO to it long. Non-exclusive defragmentation divides the whole file into small segments. For each segment, we lock the file, defragment the segment and unlock the file. Defragmenting the small segment doesn’t take long. File IO requests can get served between defragmenting segments before blocked long. Also we put (user adjustable) idle time between defragmenting two consecutive segments to balance the defragmentation and file IOs.

The first patch in the set checks for valid target files

Valid target files to defrag must:
1. be accessible for read/write
2. be regular files
3. be in XFS filesystem
4. the containing XFS has reflink enabled. This is not checked before starting defragmentation, but error would be reported later.

---

spaceman/Makefile |   2 +-
spaceman/defrag.c | 198 ++++++++++++++++++++++++++++++++++++++++++++++
spaceman/init.c   |   1 +
spaceman/space.h  |   1 +
4 files changed, 201 insertions(+), 1 deletion(-)

create mode 100644 spaceman/defrag.c

spaceman/Makefile +0 -0
index 1f048d54..9c00b20a 100644
@@ -7,7 +7,7 @@ include $(TOPDIR)/include/builddefs
 

LTCOMMAND = xfs_spaceman
HFILES = init.h space.h

spaceman/Makefile +1 -1
-CFILES = info.c init.c file.c health.c prealloc.c trim.c
+CFILES = info.c init.c file.c health.c prealloc.c trim.c defrag.c

LSRCFILES = xfs_info.sh

LLDLIBS = $(LIBXCMD) $(LIBFROG)

spaceman/defrag.c new file +61 -0
new file mode 100644
index 00000000..c9732984
@@ -0,0 +1,198 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2024 Oracle.
+ * All Rights Reserved.
+ */
+
+#include "libxfs.h"
+#include <linux/fiemap.h>
+#include <linux/fsmap.h>
+#include "libfrog/fsgeom.h"
+#include "command.h"
+#include "init.h"
+#include "libfrog/paths.h"
+#include "space.h"
+#include "input.h"
+
+/* defrag segment size limit in units of 512 bytes */
+#define MIN_SEGMENT_SIZE_LIMIT 8192 /* 4MiB */
+#define DEFAULT_SEGMENT_SIZE_LIMIT 32768 /* 16MiB */
+static int g_segment_size_lmt = DEFAULT_SEGMENT_SIZE_LIMIT;
+
+/* size of the defrag target file */
+static off_t g_defrag_file_size = 0;
+
+/* stats for the target file extents before defrag */
+struct ext_stats {
+ long nr_ext_total;
+ long nr_ext_unwritten;
+ long nr_ext_shared;
+};
+static struct ext_stats g_ext_stats;
+
+/*
+ * check if the target is a valid file to defrag
+ * also store file size
+ * returns:
+ * true for yes and false for no
+ */
+static bool
+defrag_check_file(char *path)
+{
+ struct statfs statfs_s;
+ struct stat stat_s;
+
+ if (access(path, F_OK|W_OK) == -1) {
+ if (errno == ENOENT)
+ fprintf(stderr, "file \"%s\" doesn't exist\n", path);
+ else
+ fprintf(stderr, "no access to \"%s\", %s\n", path,
+ strerror(errno));
+ return false;
+ }
+
+ if (stat(path, &stat_s) == -1) {
+ fprintf(stderr, "failed to get file info on \"%s\":  %s\n",
+ path, strerror(errno));
+ return false;
+ }
+
+ g_defrag_file_size = stat_s.st_size;
+
Quoted 8 lines

spaceman/defrag.c line 67

Quoted 6 lines
spaceman/defrag.c new file +6 -0
+ if (!S_ISREG(stat_s.st_mode)) {
+ fprintf(stderr, "\"%s\" is not a regular file\n", path);
+ return false;
+ }
+
+ if (statfs(path, &statfs_s) == -1) {

statfs is deprecated, please use fstatvfs.

OK, will move to fstatvfs.

Quoted 14 lines
Quoted 14 lines
spaceman/defrag.c new file +13 -0
+ fprintf(stderr, "failed to get FS info on \"%s\":  %s\n",
+ path, strerror(errno));
+ return false;
+ }
+
+ if (statfs_s.f_type != XFS_SUPER_MAGIC) {
+ fprintf(stderr, "\"%s\" is not a xfs file\n", path);
+ return false;
+ }
+
+ return true;
+}
+
Quoted 8 lines

spaceman/defrag.c line 86

Quoted 6 lines
spaceman/defrag.c new file +6 -0
+/*
+ * defragment a file
+ * return 0 if successfully done, 1 otherwise
+ */
+static int
+defrag_xfs_defrag(char *file_path) {

defrag_xfs_path() ?

OK.

Quoted 15 lines
Quoted 15 lines
spaceman/defrag.c new file +14 -0
+ int max_clone_us = 0, max_unshare_us = 0, max_punch_us = 0;
+ long nr_seg_defrag = 0, nr_ext_defrag = 0;
+ int scratch_fd = -1, defrag_fd = -1;
+ char tmp_file_path[PATH_MAX+1];
+ char *defrag_dir;
+ struct fsxattr fsx;
+ int ret = 0;
+
+ fsx.fsx_nextents = 0;
+ memset(&g_ext_stats, 0, sizeof(g_ext_stats));
+
+ if (!defrag_check_file(file_path)) {
+ ret = 1;
+ goto out;
Quoted 8 lines

spaceman/defrag.c line 104

Quoted 4 lines
spaceman/defrag.c new file +4 -0
+ }
+
+ defrag_fd = open(file_path, O_RDWR);
+ if (defrag_fd == -1) {

Not sure why you check the path before opening it -- all those file and statvfs attributes that you collect there can change (or the entire fs gets unmounted) until you've pinned the fs by opening the file.

The idea comes from internal reviews hoping some explicit reasons why Defrag failed. Those reasons include:
1) if user has permission to access the target file.
2) if the species path exist (when moving to spaceman, spaceman takes care of it)
3) if the specified is a regular file
4) if the target file is a XFS file

Thing might change after checking and opening, but that’s very rare case and user is responsible for that change rather than this tool.

Quoted 13 lines

spaceman/defrag.c line 115

Quoted 12 lines
spaceman/defrag.c new file +11 -0
+ fprintf(stderr, "Opening %s failed. %s\n", file_path,
+ strerror(errno));
+ ret = 1;
+ goto out;
+ }
+
+ defrag_dir = dirname(file_path);
+ snprintf(tmp_file_path, PATH_MAX, "%s/.xfsdefrag_%d", defrag_dir,
+ getpid());
+ tmp_file_path[PATH_MAX] = 0;
+ scratch_fd = open(tmp_file_path, O_CREAT|O_EXCL|O_RDWR, 0600);
Quoted 2 lines

O_TMPFILE? Then you don't have to do this .xfsdefrag_XXX stuff.

My first first version was using O_TMPFILE. But clone failed somehow (Don’t remember the details). I retried O_TMPFILE, it’s working now. So will move to use O_TMPFILE.

Quoted 29 lines
Quoted 29 lines
spaceman/defrag.c new file +29 -0
+ if (scratch_fd == -1) {
+ fprintf(stderr, "Opening temporary file %s failed. %s\n",
+ tmp_file_path, strerror(errno));
+ ret = 1;
+ goto out;
+ }
+out:
+ if (scratch_fd != -1) {
+ close(scratch_fd);
+ unlink(tmp_file_path);
+ }
+ if (defrag_fd != -1) {
+ ioctl(defrag_fd, FS_IOC_FSGETXATTR, &fsx);
+ close(defrag_fd);
+ }
+
+ printf("Pre-defrag %ld extents detected, %ld are \"unwritten\","
+ "%ld are \"shared\"\n",
+ g_ext_stats.nr_ext_total, g_ext_stats.nr_ext_unwritten,
+ g_ext_stats.nr_ext_shared);
+ printf("Tried to defragment %ld extents in %ld segments\n",
+ nr_ext_defrag, nr_seg_defrag);
+ printf("Time stats(ms): max clone: %d, max unshare: %d,"
+        " max punch_hole: %d\n",
+        max_clone_us/1000, max_unshare_us/1000, max_punch_us/1000);
+ printf("Post-defrag %u extents detected\n", fsx.fsx_nextents);
+ return ret;
+}
+
Quoted 8 lines

spaceman/defrag.c line 150

Quoted 6 lines
spaceman/defrag.c new file +6 -0
+
+static void defrag_help(void)
+{
+ printf(_(
+"\n"
+"Defragemnt files on XFS where reflink is enabled. IOs to the target files \n"

"Defragment"

Quoted 39 lines

OK.

Quoted 36 lines
Quoted 23 lines
spaceman/defrag.c new file +22 -0
+"can be served durning the defragmentations.\n"
+"\n"
+" -s segment_size    -- specify the segment size in MiB, minmum value is 4 \n"
+"                       default is 16\n"));
+}
+
+static cmdinfo_t defrag_cmd;
+
+static int
+defrag_f(int argc, char **argv)
+{
+ int i;
+ int c;
+
+ while ((c = getopt(argc, argv, "s:")) != EOF) {
+ switch(c) {
+ case 's':
+ g_segment_size_lmt = atoi(optarg) * 1024 * 1024 / 512;
+ if (g_segment_size_lmt < MIN_SEGMENT_SIZE_LIMIT) {
+ g_segment_size_lmt = MIN_SEGMENT_SIZE_LIMIT;
+ printf("Using minimium segment size %d\n",
+ g_segment_size_lmt);

spaceman/defrag.c line 182

Quoted 10 lines
spaceman/defrag.c new file +10 -0
+ }
+ break;
+ default:
+ command_usage(&defrag_cmd);
+ return 1;
+ }
+ }
+
+ for (i = 0; i < filecount; i++)
+ defrag_xfs_defrag(filetable[i].name);

Pass in the whole filetable[i] and then you've already got an open fd and some validation that it's an xfs filesystem.

Filetable[I].xfd.fd doesn’t work well. UNSHARE returns “Bad file descriptor”, I am suspecting that fd is readonly.

So I have to write-open again.

Thanks,
Wengang

Quoted 56 lines

Good to know.

Quoted 16 lines

spaceman/defrag.c line 193

Quoted 12 lines
spaceman/defrag.c new file +11 -0
+ return 0;
+}
+void defrag_init(void)
+{
+ defrag_cmd.name = "defrag";
+ defrag_cmd.altname = "dfg";
+ defrag_cmd.cfunc = defrag_f;
+ defrag_cmd.argmin = 0;
+ defrag_cmd.argmax = 4;
+ defrag_cmd.args = "[-s segment_size]";
+ defrag_cmd.flags = CMD_FLAG_ONESHOT;

IIRC if you don't set CMD_FLAG_FOREIGN_OK then the command processor won't let this command get run against a non-xfs file.

OK.

Thanks,
Winging

Quoted 33 lines

--D

Quoted 31 lines
spaceman/defrag.c new file +5 -0
+ defrag_cmd.oneline = _("Defragment XFS files");
+ defrag_cmd.help = defrag_help;
+
+ add_command(&defrag_cmd);
+}
spaceman/init.c +1 -0
index cf1ff3cb..396f965c 100644
@@ -35,6 +35,7 @@ init_commands(void)
 trim_init();
 freesp_init();
 health_init();
+ defrag_init();
 }
 

static int

spaceman/space.h +1 -0
index 723209ed..c288aeb9 100644
@@ -26,6 +26,7 @@ extern void help_init(void);
 extern void prealloc_init(void);
 extern void quit_init(void);
 extern void trim_init(void);
+extern void defrag_init(void);
 #ifdef HAVE_GETFSMAP
 extern void freesp_init(void);
 #else
Signature · 1 line
2.39.3 (Apple Git-146)
#5 lore raw

Re: [PATCH 1/9] xfsprogs: introduce defrag command to spaceman

Quoted 288 lines · Wengang Wang
Quoted 248 lines · Wengang Wang

Hi Darrick,
Thanks for review, pls check in lines.

Quoted 115 lines · Darrick J. Wong
Quoted 112 lines · Wengang Wang

Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Non-exclusive defragment
Here we are introducing the non-exclusive manner to defragment a file, especially for huge files, without blocking IO to it long. Non-exclusive defragmentation divides the whole file into small segments. For each segment, we lock the file, defragment the segment and unlock the file. Defragmenting the small segment doesn’t take long. File IO requests can get served between defragmenting segments before blocked long. Also we put (user adjustable) idle time between defragmenting two consecutive segments to balance the defragmentation and file IOs.

The first patch in the set checks for valid target files

Valid target files to defrag must:
1. be accessible for read/write
2. be regular files
3. be in XFS filesystem
4. the containing XFS has reflink enabled. This is not checked before starting defragmentation, but error would be reported later.

---

spaceman/Makefile |   2 +-
spaceman/defrag.c | 198 ++++++++++++++++++++++++++++++++++++++++++++++
spaceman/init.c   |   1 +
spaceman/space.h  |   1 +
4 files changed, 201 insertions(+), 1 deletion(-)

create mode 100644 spaceman/defrag.c

spaceman/Makefile +0 -0
index 1f048d54..9c00b20a 100644
@@ -7,7 +7,7 @@ include $(TOPDIR)/include/builddefs
 

LTCOMMAND = xfs_spaceman
HFILES = init.h space.h

spaceman/Makefile +1 -1
-CFILES = info.c init.c file.c health.c prealloc.c trim.c
+CFILES = info.c init.c file.c health.c prealloc.c trim.c defrag.c

LSRCFILES = xfs_info.sh

LLDLIBS = $(LIBXCMD) $(LIBFROG)

spaceman/defrag.c new file +61 -0
new file mode 100644
index 00000000..c9732984
@@ -0,0 +1,198 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2024 Oracle.
+ * All Rights Reserved.
+ */
+
+#include "libxfs.h"
+#include <linux/fiemap.h>
+#include <linux/fsmap.h>
+#include "libfrog/fsgeom.h"
+#include "command.h"
+#include "init.h"
+#include "libfrog/paths.h"
+#include "space.h"
+#include "input.h"
+
+/* defrag segment size limit in units of 512 bytes */
+#define MIN_SEGMENT_SIZE_LIMIT 8192 /* 4MiB */
+#define DEFAULT_SEGMENT_SIZE_LIMIT 32768 /* 16MiB */
+static int g_segment_size_lmt = DEFAULT_SEGMENT_SIZE_LIMIT;
+
+/* size of the defrag target file */
+static off_t g_defrag_file_size = 0;
+
+/* stats for the target file extents before defrag */
+struct ext_stats {
+ long nr_ext_total;
+ long nr_ext_unwritten;
+ long nr_ext_shared;
+};
+static struct ext_stats g_ext_stats;
+
+/*
+ * check if the target is a valid file to defrag
+ * also store file size
+ * returns:
+ * true for yes and false for no
+ */
+static bool
+defrag_check_file(char *path)
+{
+ struct statfs statfs_s;
+ struct stat stat_s;
+
+ if (access(path, F_OK|W_OK) == -1) {
+ if (errno == ENOENT)
+ fprintf(stderr, "file \"%s\" doesn't exist\n", path);
+ else
+ fprintf(stderr, "no access to \"%s\", %s\n", path,
+ strerror(errno));
+ return false;
+ }
+
+ if (stat(path, &stat_s) == -1) {
+ fprintf(stderr, "failed to get file info on \"%s\":  %s\n",
+ path, strerror(errno));
+ return false;
+ }
+
+ g_defrag_file_size = stat_s.st_size;
+
Quoted 8 lines

spaceman/defrag.c line 67

Quoted 6 lines
spaceman/defrag.c new file +6 -0
+ if (!S_ISREG(stat_s.st_mode)) {
+ fprintf(stderr, "\"%s\" is not a regular file\n", path);
+ return false;
+ }
+
+ if (statfs(path, &statfs_s) == -1) {

statfs is deprecated, please use fstatvfs.

OK, will move to fstatvfs.

Quoted 14 lines
Quoted 14 lines
spaceman/defrag.c new file +13 -0
+ fprintf(stderr, "failed to get FS info on \"%s\":  %s\n",
+ path, strerror(errno));
+ return false;
+ }
+
+ if (statfs_s.f_type != XFS_SUPER_MAGIC) {
+ fprintf(stderr, "\"%s\" is not a xfs file\n", path);
+ return false;
+ }
+
+ return true;
+}
+
Quoted 8 lines

spaceman/defrag.c line 86

Quoted 6 lines
spaceman/defrag.c new file +6 -0
+/*
+ * defragment a file
+ * return 0 if successfully done, 1 otherwise
+ */
+static int
+defrag_xfs_defrag(char *file_path) {

defrag_xfs_path() ?

OK.

Quoted 15 lines
Quoted 15 lines
spaceman/defrag.c new file +14 -0
+ int max_clone_us = 0, max_unshare_us = 0, max_punch_us = 0;
+ long nr_seg_defrag = 0, nr_ext_defrag = 0;
+ int scratch_fd = -1, defrag_fd = -1;
+ char tmp_file_path[PATH_MAX+1];
+ char *defrag_dir;
+ struct fsxattr fsx;
+ int ret = 0;
+
+ fsx.fsx_nextents = 0;
+ memset(&g_ext_stats, 0, sizeof(g_ext_stats));
+
+ if (!defrag_check_file(file_path)) {
+ ret = 1;
+ goto out;
Quoted 8 lines

spaceman/defrag.c line 104

Quoted 4 lines
spaceman/defrag.c new file +4 -0
+ }
+
+ defrag_fd = open(file_path, O_RDWR);
+ if (defrag_fd == -1) {

Not sure why you check the path before opening it -- all those file and statvfs attributes that you collect there can change (or the entire fs gets unmounted) until you've pinned the fs by opening the file.

The idea comes from internal reviews hoping some explicit reasons why Defrag failed. Those reasons include:
1) if user has permission to access the target file.
2) if the species path exist (when moving to spaceman, spaceman takes care of it)
3) if the specified is a regular file
4) if the target file is a XFS file

Thing might change after checking and opening, but that’s very rare case and user is responsible for that change rather than this tool.

Quoted 13 lines

spaceman/defrag.c line 115

Quoted 12 lines
spaceman/defrag.c new file +11 -0
+ fprintf(stderr, "Opening %s failed. %s\n", file_path,
+ strerror(errno));
+ ret = 1;
+ goto out;
+ }
+
+ defrag_dir = dirname(file_path);
+ snprintf(tmp_file_path, PATH_MAX, "%s/.xfsdefrag_%d", defrag_dir,
+ getpid());
+ tmp_file_path[PATH_MAX] = 0;
+ scratch_fd = open(tmp_file_path, O_CREAT|O_EXCL|O_RDWR, 0600);
Quoted 2 lines

O_TMPFILE? Then you don't have to do this .xfsdefrag_XXX stuff.

My first first version was using O_TMPFILE. But clone failed somehow (Don’t remember the details). I retried O_TMPFILE, it’s working now. So will move to use O_TMPFILE.

Quoted 29 lines
Quoted 29 lines
spaceman/defrag.c new file +29 -0
+ if (scratch_fd == -1) {
+ fprintf(stderr, "Opening temporary file %s failed. %s\n",
+ tmp_file_path, strerror(errno));
+ ret = 1;
+ goto out;
+ }
+out:
+ if (scratch_fd != -1) {
+ close(scratch_fd);
+ unlink(tmp_file_path);
+ }
+ if (defrag_fd != -1) {
+ ioctl(defrag_fd, FS_IOC_FSGETXATTR, &fsx);
+ close(defrag_fd);
+ }
+
+ printf("Pre-defrag %ld extents detected, %ld are \"unwritten\","
+ "%ld are \"shared\"\n",
+ g_ext_stats.nr_ext_total, g_ext_stats.nr_ext_unwritten,
+ g_ext_stats.nr_ext_shared);
+ printf("Tried to defragment %ld extents in %ld segments\n",
+ nr_ext_defrag, nr_seg_defrag);
+ printf("Time stats(ms): max clone: %d, max unshare: %d,"
+        " max punch_hole: %d\n",
+        max_clone_us/1000, max_unshare_us/1000, max_punch_us/1000);
+ printf("Post-defrag %u extents detected\n", fsx.fsx_nextents);
+ return ret;
+}
+
Quoted 8 lines

spaceman/defrag.c line 150

Quoted 6 lines
spaceman/defrag.c new file +6 -0
+
+static void defrag_help(void)
+{
+ printf(_(
+"\n"
+"Defragemnt files on XFS where reflink is enabled. IOs to the target files \n"

"Defragment"

Quoted 39 lines

OK.

Quoted 36 lines
Quoted 23 lines
spaceman/defrag.c new file +22 -0
+"can be served durning the defragmentations.\n"
+"\n"
+" -s segment_size    -- specify the segment size in MiB, minmum value is 4 \n"
+"                       default is 16\n"));
+}
+
+static cmdinfo_t defrag_cmd;
+
+static int
+defrag_f(int argc, char **argv)
+{
+ int i;
+ int c;
+
+ while ((c = getopt(argc, argv, "s:")) != EOF) {
+ switch(c) {
+ case 's':
+ g_segment_size_lmt = atoi(optarg) * 1024 * 1024 / 512;
+ if (g_segment_size_lmt < MIN_SEGMENT_SIZE_LIMIT) {
+ g_segment_size_lmt = MIN_SEGMENT_SIZE_LIMIT;
+ printf("Using minimium segment size %d\n",
+ g_segment_size_lmt);

spaceman/defrag.c line 182

Quoted 10 lines
spaceman/defrag.c new file +10 -0
+ }
+ break;
+ default:
+ command_usage(&defrag_cmd);
+ return 1;
+ }
+ }
+
+ for (i = 0; i < filecount; i++)
+ defrag_xfs_defrag(filetable[i].name);

Pass in the whole filetable[i] and then you've already got an open fd and some validation that it's an xfs filesystem.

Quoted 3 lines

Filetable[I].xfd.fd doesn’t work well. UNSHARE returns “Bad file descriptor”, I am suspecting that fd is readonly.

So I have to write-open again.

Ah, ok. In that case, after you reopen the file, you ought to stat both of them and check that st_dev/st_ino match.

(Or just change spaceman to be able to open files O_RDWR?)

--D

Quoted 61 lines

Thanks,
Wengang

Quoted 56 lines

Good to know.

Quoted 16 lines

spaceman/defrag.c line 193

Quoted 12 lines
spaceman/defrag.c new file +11 -0
+ return 0;
+}
+void defrag_init(void)
+{
+ defrag_cmd.name = "defrag";
+ defrag_cmd.altname = "dfg";
+ defrag_cmd.cfunc = defrag_f;
+ defrag_cmd.argmin = 0;
+ defrag_cmd.argmax = 4;
+ defrag_cmd.args = "[-s segment_size]";
+ defrag_cmd.flags = CMD_FLAG_ONESHOT;

IIRC if you don't set CMD_FLAG_FOREIGN_OK then the command processor won't let this command get run against a non-xfs file.

OK.

Thanks,
Winging

Quoted 33 lines

--D

Quoted 31 lines
spaceman/defrag.c new file +5 -0
+ defrag_cmd.oneline = _("Defragment XFS files");
+ defrag_cmd.help = defrag_help;
+
+ add_command(&defrag_cmd);
+}
spaceman/init.c +1 -0
index cf1ff3cb..396f965c 100644
@@ -35,6 +35,7 @@ init_commands(void)
 trim_init();
 freesp_init();
 health_init();
+ defrag_init();
 }
 

static int

spaceman/space.h +1 -0
index 723209ed..c288aeb9 100644
@@ -26,6 +26,7 @@ extern void help_init(void);
 extern void prealloc_init(void);
 extern void quit_init(void);
 extern void trim_init(void);
+extern void defrag_init(void);
 #ifdef HAVE_GETFSMAP
 extern void freesp_init(void);
 #else
Signature · 1 line
2.39.3 (Apple Git-146)
#6 lore raw

PATCH 2/9 spaceman/defrag: pick up segments from target file

In reply to Wengang Wang

segments are the smallest unit to defragment.

A segment
1. Can't exceed size limit
2. contains some extents
3. the contained extents can't be "unwritten"
4. the contained extents must be contigous in file blocks

---
 spaceman/defrag.c | 204 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 204 insertions(+)
spaceman/defrag.c +204 -0
index c9732984..175cf461 100644
@@ -14,6 +14,32 @@
 #include "space.h"
 #include "input.h"
 
+#define MAPSIZE 512
+/* used to fetch bmap */
+struct getbmapx	g_mapx[MAPSIZE];
+/* current offset of the file in units of 512 bytes, used to fetch bmap */
+static long long 	g_offset = 0;
+/* index to indentify next extent, used to get next extent */
+static int		g_ext_next_idx = -1;
+
+/*
+ * segment, the smallest unit to defrag
+ * it includes some contiguous extents.
+ * no holes included,
+ * no unwritten extents included
+ * the size is limited by g_segment_size_lmt
+ */
+struct defrag_segment {
+	/* segment offset in units of 512 bytes */
+	long long	ds_offset;
+	/* length of segment in units of 512 bytes */
+	long long	ds_length;
+	/* number of extents in this segment */
+	int		ds_nr;
+	/* flag indicating if segment contains shared blocks */
+	bool		ds_shared;
+};
+
 /* defrag segment size limit in units of 512 bytes */
 #define MIN_SEGMENT_SIZE_LIMIT 8192 /* 4MiB */
 #define DEFAULT_SEGMENT_SIZE_LIMIT 32768 /* 16MiB */
@@ -78,6 +104,165 @@ defrag_check_file(char *path)
 	return true;
 }
 
+/*
+ * get next extent in the file.
+ * Note: next call will get the same extent unless move_next_extent() is called.
+ * returns:
+ * -1:	error happened.
+ * 0:	extent returned
+ * 1:	no more extent left
+ */
+static int
+defrag_get_next_extent(int fd, struct getbmapx *map_out)
+{
+	int err = 0, i;
+
+	/* when no extents are cached in g_mapx, fetch from kernel */
+	if (g_ext_next_idx == -1) {
+		g_mapx[0].bmv_offset = g_offset;
+		g_mapx[0].bmv_length = -1LL;
+		g_mapx[0].bmv_count = MAPSIZE;
+		g_mapx[0].bmv_iflags = BMV_IF_NO_HOLES | BMV_IF_PREALLOC;
+		err = ioctl(fd, XFS_IOC_GETBMAPX, g_mapx);
+		if (err == -1) {
+			perror("XFS_IOC_GETBMAPX failed");
+			goto out;
+		}
+		/* for stats */
+		g_ext_stats.nr_ext_total += g_mapx[0].bmv_entries;
+
+		/* no more extents */
+		if (g_mapx[0].bmv_entries == 0) {
+			err = 1;
+			goto out;
+		}
+
+		/* for stats */
+		for (i = 1; i <= g_mapx[0].bmv_entries; i++) {
+			if (g_mapx[i].bmv_oflags & BMV_OF_PREALLOC)
+				g_ext_stats.nr_ext_unwritten++;
+			if (g_mapx[i].bmv_oflags & BMV_OF_SHARED)
+				g_ext_stats.nr_ext_shared++;
+		}
+
+		g_ext_next_idx = 1;
+		g_offset = g_mapx[g_mapx[0].bmv_entries].bmv_offset +
+				g_mapx[g_mapx[0].bmv_entries].bmv_length;
+	}
+
+	map_out->bmv_offset = g_mapx[g_ext_next_idx].bmv_offset;
+	map_out->bmv_length = g_mapx[g_ext_next_idx].bmv_length;
+	map_out->bmv_oflags = g_mapx[g_ext_next_idx].bmv_oflags;
+out:
+	return err;
+}
+
+/*
+ * move to next extent
+ */
+static void
+defrag_move_next_extent()
+{
+	if (g_ext_next_idx == g_mapx[0].bmv_entries)
+		g_ext_next_idx = -1;
+	else
+		g_ext_next_idx += 1;
+}
+
+/*
+ * check if the given extent is a defrag target.
+ * no need to check for holes as we are using BMV_IF_NO_HOLES
+ */
+static bool
+defrag_is_target(struct getbmapx *mapx)
+{
+	/* unwritten */
+	if (mapx->bmv_oflags & BMV_OF_PREALLOC)
+		return false;
+	return mapx->bmv_length < g_segment_size_lmt;
+}
+
+static bool
+defrag_is_extent_shared(struct getbmapx *mapx)
+{
+	return !!(mapx->bmv_oflags & BMV_OF_SHARED);
+}
+
+/*
+ * get next segment to defragment.
+ * returns:
+ * -1	error happened.
+ * 0	segment returned.
+ * 1	no more segments to return
+ */
+static int
+defrag_get_next_segment(int fd, struct defrag_segment *out)
+{
+	struct getbmapx mapx;
+	int	ret;
+
+	out->ds_offset = 0;
+	out->ds_length = 0;
+	out->ds_nr = 0;
+	out->ds_shared = false;
+
+	do {
+		ret = defrag_get_next_extent(fd, &mapx);
+		if (ret != 0) {
+			/*
+			 * no more extetns, return current segment if its not
+			 * empty
+			*/
+			if (ret == 1 && out->ds_nr > 0)
+				ret = 0;
+			/* otherwise, error heppened, stop */
+			break;
+		}
+
+		/*
+		 * If the extent is not a defrag target, skip it.
+		 * go to next extent if the segment is empty;
+		 * otherwise return the segment.
+		 */
+		if (!defrag_is_target(&mapx)) {
+			defrag_move_next_extent();
+			if (out->ds_nr == 0)
+				continue;
+			else
+				break;
+		}
+
+		/* check for segment size limitation */
+		if (out->ds_length + mapx.bmv_length > g_segment_size_lmt)
+			break;
+
+		/* the segment is empty now, add this extent to it for sure */
+		if (out->ds_nr == 0) {
+			out->ds_offset = mapx.bmv_offset;
+			goto add_ext;
+		}
+
+		/*
+		 * the segment is not empty, check for hole since the last exent
+		 * if a hole exist before this extent, this extent can't be
+		 * added to the segment. return the segment
+		 */
+		if (out->ds_offset + out->ds_length != mapx.bmv_offset)
+			break;
+
+add_ext:
+		if (defrag_is_extent_shared(&mapx))
+			out->ds_shared = true;
+
+		out->ds_length += mapx.bmv_length;
+		out->ds_nr += 1;
+		defrag_move_next_extent();
+
+	} while (true);
+
+	return ret;
+}
+
 /*
  * defragment a file
  * return 0 if successfully done, 1 otherwise
@@ -92,6 +277,9 @@ defrag_xfs_defrag(char *file_path) {
 	struct fsxattr	fsx;
 	int	ret = 0;
 
+	g_offset = 0;
+	g_ext_next_idx = -1;
+
 	fsx.fsx_nextents = 0;
 	memset(&g_ext_stats, 0, sizeof(g_ext_stats));
 
@@ -119,6 +307,22 @@ defrag_xfs_defrag(char *file_path) {
 		ret = 1;
 		goto out;
 	}
+
+	do {
+		struct defrag_segment segment;
+
+		ret = defrag_get_next_segment(defrag_fd, &segment);
+		/* no more segments, we are done */
+		if (ret == 1) {
+			ret = 0;
+			break;
+		}
+		/* error happened when reading bmap, stop here */
+		if (ret == -1) {
+			ret = 1;
+			break;
+		}
+	} while (true);
 out:
 	if (scratch_fd != -1) {
 		close(scratch_fd);
Signature · 1 line
2.39.3 (Apple Git-146)
#7 lore raw

Re: [PATCH 2/9] spaceman/defrag: pick up segments from target fileOM

Quoted 4 lines · Wengang Wang

segments are the smallest unit to defragment.

A segment
1. Can't exceed size limit

What size limit? Do you mean a segment can't extend beyond EOF? Or did you actually mean RLIMIT_FSIZE?

Quoted 3 lines

2. contains some extents
3. the contained extents can't be "unwritten"
4. the contained extents must be contigous in file blocks

As in the segment cannot contain sparse holes?

I think what I"m reading here is that a segment cannot extend beyond EOF and must be completely filled with written extent mappings?

Is there an upper limit on the number of mappings per segment?

Quoted 10 lines
---
 spaceman/defrag.c | 204 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 204 insertions(+)
spaceman/defrag.c +0 -0
index c9732984..175cf461 100644

spaceman/defrag.c line 19

Quoted 6 lines
spaceman/defrag.c +3 -0
@@ -14,6 +14,32 @@
 #include "space.h"
 #include "input.h"
 
+#define MAPSIZE 512
+/* used to fetch bmap */
+struct getbmapx	g_mapx[MAPSIZE];

Each of these global arrays increases the bss segment size, which increases the overall footprint of xfs_spaceman, even when it's not being used to defragment files.

Could you switch this data to be dynamically allocated at the start of defrag_f and freed at the end?

spaceman/defrag.c line 21

Quoted 2 lines
spaceman/defrag.c +2 -0
+/* current offset of the file in units of 512 bytes, used to fetch bmap */
+static long long 	g_offset = 0;

Unnecessary space after the 'long long'.

Quoted 9 lines
spaceman/defrag.c +9 -0
+/* index to indentify next extent, used to get next extent */
+static int		g_ext_next_idx = -1;
+
+/*
+ * segment, the smallest unit to defrag
+ * it includes some contiguous extents.
+ * no holes included,
+ * no unwritten extents included
+ * the size is limited by g_segment_size_lmt

spaceman/defrag.c line 40

Quoted 10 lines
spaceman/defrag.c +10 -0
+ */
+struct defrag_segment {
+	/* segment offset in units of 512 bytes */
+	long long	ds_offset;
+	/* length of segment in units of 512 bytes */
+	long long	ds_length;
+	/* number of extents in this segment */
+	int		ds_nr;
+	/* flag indicating if segment contains shared blocks */
+	bool		ds_shared;

Maybe g_mapx belongs in here? Wait, does a bunch of contiguous written bmapx records comprise a segment, or is a segment created from (possibly a subection of) a particular written bmapx record?

Quoted 44 lines
spaceman/defrag.c +37 -0
+};
+
 /* defrag segment size limit in units of 512 bytes */
 #define MIN_SEGMENT_SIZE_LIMIT 8192 /* 4MiB */
 #define DEFAULT_SEGMENT_SIZE_LIMIT 32768 /* 16MiB */
@@ -78,6 +104,165 @@ defrag_check_file(char *path)
 	return true;
 }
 
+/*
+ * get next extent in the file.
+ * Note: next call will get the same extent unless move_next_extent() is called.
+ * returns:
+ * -1:	error happened.
+ * 0:	extent returned
+ * 1:	no more extent left
+ */
+static int
+defrag_get_next_extent(int fd, struct getbmapx *map_out)
+{
+	int err = 0, i;
+
+	/* when no extents are cached in g_mapx, fetch from kernel */
+	if (g_ext_next_idx == -1) {
+		g_mapx[0].bmv_offset = g_offset;
+		g_mapx[0].bmv_length = -1LL;
+		g_mapx[0].bmv_count = MAPSIZE;
+		g_mapx[0].bmv_iflags = BMV_IF_NO_HOLES | BMV_IF_PREALLOC;
+		err = ioctl(fd, XFS_IOC_GETBMAPX, g_mapx);
+		if (err == -1) {
+			perror("XFS_IOC_GETBMAPX failed");
+			goto out;
+		}
+		/* for stats */
+		g_ext_stats.nr_ext_total += g_mapx[0].bmv_entries;
+
+		/* no more extents */
+		if (g_mapx[0].bmv_entries == 0) {
+			err = 1;
+			goto out;
+		}
+
+		/* for stats */
+		for (i = 1; i <= g_mapx[0].bmv_entries; i++) {

spaceman/defrag.c defrag_check_file(char *path) · line 151

Quoted 10 lines
spaceman/defrag.c +10 -0
+			if (g_mapx[i].bmv_oflags & BMV_OF_PREALLOC)
+				g_ext_stats.nr_ext_unwritten++;
+			if (g_mapx[i].bmv_oflags & BMV_OF_SHARED)
+				g_ext_stats.nr_ext_shared++;
+		}
+
+		g_ext_next_idx = 1;
+		g_offset = g_mapx[g_mapx[0].bmv_entries].bmv_offset +
+				g_mapx[g_mapx[0].bmv_entries].bmv_length;
+	}

Huh. AFAICT, g_ext_next_idx/g_mapx effectively act as a cursor over the mappings for this file segment. In that case, shouldn't defrag_move_next_extent (which actually advances the cursor) be in charge of grabbing bmapx records from the kernel, and defrag_get_next_extent be the trivial helper to pass mappings to the consumer of the bmapx objects (aka defrag_get_next_segment)?

Quoted 97 lines
spaceman/defrag.c +97 -0
+
+	map_out->bmv_offset = g_mapx[g_ext_next_idx].bmv_offset;
+	map_out->bmv_length = g_mapx[g_ext_next_idx].bmv_length;
+	map_out->bmv_oflags = g_mapx[g_ext_next_idx].bmv_oflags;
+out:
+	return err;
+}
+
+/*
+ * move to next extent
+ */
+static void
+defrag_move_next_extent()
+{
+	if (g_ext_next_idx == g_mapx[0].bmv_entries)
+		g_ext_next_idx = -1;
+	else
+		g_ext_next_idx += 1;
+}
+
+/*
+ * check if the given extent is a defrag target.
+ * no need to check for holes as we are using BMV_IF_NO_HOLES
+ */
+static bool
+defrag_is_target(struct getbmapx *mapx)
+{
+	/* unwritten */
+	if (mapx->bmv_oflags & BMV_OF_PREALLOC)
+		return false;
+	return mapx->bmv_length < g_segment_size_lmt;
+}
+
+static bool
+defrag_is_extent_shared(struct getbmapx *mapx)
+{
+	return !!(mapx->bmv_oflags & BMV_OF_SHARED);
+}
+
+/*
+ * get next segment to defragment.
+ * returns:
+ * -1	error happened.
+ * 0	segment returned.
+ * 1	no more segments to return
+ */
+static int
+defrag_get_next_segment(int fd, struct defrag_segment *out)
+{
+	struct getbmapx mapx;
+	int	ret;
+
+	out->ds_offset = 0;
+	out->ds_length = 0;
+	out->ds_nr = 0;
+	out->ds_shared = false;
+
+	do {
+		ret = defrag_get_next_extent(fd, &mapx);
+		if (ret != 0) {
+			/*
+			 * no more extetns, return current segment if its not
+			 * empty
+			*/
+			if (ret == 1 && out->ds_nr > 0)
+				ret = 0;
+			/* otherwise, error heppened, stop */
+			break;
+		}
+
+		/*
+		 * If the extent is not a defrag target, skip it.
+		 * go to next extent if the segment is empty;
+		 * otherwise return the segment.
+		 */
+		if (!defrag_is_target(&mapx)) {
+			defrag_move_next_extent();
+			if (out->ds_nr == 0)
+				continue;
+			else
+				break;
+		}
+
+		/* check for segment size limitation */
+		if (out->ds_length + mapx.bmv_length > g_segment_size_lmt)
+			break;
+
+		/* the segment is empty now, add this extent to it for sure */
+		if (out->ds_nr == 0) {
+			out->ds_offset = mapx.bmv_offset;
+			goto add_ext;
+		}
+
+		/*
+		 * the segment is not empty, check for hole since the last exent
+		 * if a hole exist before this extent, this extent can't be
+		 * added to the segment. return the segment

spaceman/defrag.c defrag_check_file(char *path) · line 258

Quoted 10 lines
spaceman/defrag.c +10 -0
+		 */
+		if (out->ds_offset + out->ds_length != mapx.bmv_offset)
+			break;
+
+add_ext:
+		if (defrag_is_extent_shared(&mapx))
+			out->ds_shared = true;
+
+		out->ds_length += mapx.bmv_length;
+		out->ds_nr += 1;

OH, ok. So we walk the mappings for a file. If we can identify a run of contiguous written mappings, we define a segment to be the file range described by that run, up to whatever the maximum is (~4-16M). Each of these segments is defragmented (somehow). Is that correct?

Quoted 24 lines
spaceman/defrag.c +10 -0
+		defrag_move_next_extent();
+
+	} while (true);
+
+	return ret;
+}
+
 /*
  * defragment a file
  * return 0 if successfully done, 1 otherwise
@@ -92,6 +277,9 @@ defrag_xfs_defrag(char *file_path) {
 	struct fsxattr	fsx;
 	int	ret = 0;
 
+	g_offset = 0;
+	g_ext_next_idx = -1;
+
 	fsx.fsx_nextents = 0;
 	memset(&g_ext_stats, 0, sizeof(g_ext_stats));
 
@@ -119,6 +307,22 @@ defrag_xfs_defrag(char *file_path) {
 		ret = 1;
 		goto out;
 	}

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 319

Quoted 10 lines
spaceman/defrag.c +10 -0
+
+	do {
+		struct defrag_segment segment;
+
+		ret = defrag_get_next_segment(defrag_fd, &segment);
+		/* no more segments, we are done */
+		if (ret == 1) {
+			ret = 0;
+			break;
+		}

If you reverse the polarity of the 0/1 return values (aka return 1 if there is a segment and 0 if there is none) then you can shorten this loop to:

	struct defrag_segment	segment;
	int			ret;

	while ((ret = defrag_get_next_segment(...)) == 1) {
		/* process segment */
	}

	return ret;

--D

Quoted 13 lines
spaceman/defrag.c +6 -0
+		/* error happened when reading bmap, stop here */
+		if (ret == -1) {
+			ret = 1;
+			break;
+		}
+	} while (true);
 out:
 	if (scratch_fd != -1) {
 		close(scratch_fd);
Signature · 1 line
2.39.3 (Apple Git-146)
#8 lore raw

Re: [PATCH 2/9] spaceman/defrag: pick up segments from target fileOM

Quoted 12 lines · Darrick J. Wong
Quoted 4 lines · Wengang Wang

segments are the smallest unit to defragment.

A segment
1. Can't exceed size limit

What size limit? Do you mean a segment can't extend beyond EOF? Or did you actually mean RLIMIT_FSIZE?

We defrag things by share (non-contiguous) blocks and then UNSHARE them to get contiguous Blocks. The size limit is the limit of the range for which we issue a UNSHARE ioctl call. We put some contiguous extents (mappings) together (as a segment) for a UNSHARE call. We don’t hope that’s too big taking long time IO lock on file. Default size limit is 16MiB.

Quoted 6 lines
Quoted 4 lines

2. contains some extents
3. the contained extents can't be "unwritten"
4. the contained extents must be contigous in file blocks

As in the segment cannot contain sparse holes?

Holes are meaningless for unshare calls, so they are not included in segments.

Quoted 3 lines

I think what I"m reading here is that a segment cannot extend beyond EOF and must be completely filled with written extent mappings?

Yes, a segment as the unit/range for UNSHARE,
1) extents included in the segment must be contiguous by file offset
2) is not a hole
3) is not unwritten extents. Unwritten extents are meaningless for UNSHARE.

Quoted 24 lines

Is there an upper limit on the number of mappings per segment?

Quoted 10 lines

---
spaceman/defrag.c | 204 ++++++++++++++++++++++++++++++++++++++++++++++

1 file changed, 204 insertions(+)
spaceman/defrag.c +0 -0
index c9732984..175cf461 100644

spaceman/defrag.c line 19

Quoted 6 lines
spaceman/defrag.c +3 -0
@@ -14,6 +14,32 @@
 #include "space.h"
 #include "input.h"
 
+#define MAPSIZE 512
+/* used to fetch bmap */
+struct getbmapx g_mapx[MAPSIZE];

Each of these global arrays increases the bss segment size, which increases the overall footprint of xfs_spaceman, even when it's not being used to defragment files.

Quoted 3 lines

Could you switch this data to be dynamically allocated at the start of defrag_f and freed at the end?

Yes, I can.
Well I am wondering the binary size is concern this much?
After adding defrag, xfs_space has a size of 239K, I don’t think it’s too big.

$ ll -h spaceman/xfs_spaceman
-rwxrwxr-x 1 ubuntu ubuntu 239K Jul 10 17:20 spaceman/xfs_spaceman*
Quoted 5 lines

spaceman/defrag.c line 21

Quoted 2 lines
spaceman/defrag.c +2 -0
+/* current offset of the file in units of 512 bytes, used to fetch bmap */
+static long long  g_offset = 0;

Unnecessary space after the 'long long'.

ok.

Quoted 15 lines
Quoted 15 lines
spaceman/defrag.c +15 -0
+/* index to indentify next extent, used to get next extent */
+static int g_ext_next_idx = -1;
+
+/*
+ * segment, the smallest unit to defrag
+ * it includes some contiguous extents.
+ * no holes included,
+ * no unwritten extents included
+ * the size is limited by g_segment_size_lmt
+ */
+struct defrag_segment {
+ /* segment offset in units of 512 bytes */
+ long long ds_offset;
+ /* length of segment in units of 512 bytes */
+ long long ds_length;
Quoted 8 lines

spaceman/defrag.c line 40

Quoted 4 lines
spaceman/defrag.c +4 -0
+ /* number of extents in this segment */
+ int ds_nr;
+ /* flag indicating if segment contains shared blocks */
+ bool ds_shared;

Maybe g_mapx belongs in here? Wait, does a bunch of contiguous written bmapx records comprise a segment, or is a segment created from (possibly a subection of) a particular written bmapx record?

g_mapx is used to fetch the mappings from beginning of the file to the end. It,
1) can include at most 511 (MAPSIZE-1) extents/mappings
2) can fill more than one segment
3) can be a part of a segment

We are walking through the extents/mappings to form segments. g_mapx is used in walking through the extents/mappings. For forming a segment, pls see other functions.

Quoted 50 lines
Quoted 50 lines
spaceman/defrag.c +42 -0
+};
+
 /* defrag segment size limit in units of 512 bytes */
 #define MIN_SEGMENT_SIZE_LIMIT 8192 /* 4MiB */
 #define DEFAULT_SEGMENT_SIZE_LIMIT 32768 /* 16MiB */
@@ -78,6 +104,165 @@ defrag_check_file(char *path)
 return true;
 }
 
+/*
+ * get next extent in the file.
+ * Note: next call will get the same extent unless move_next_extent() is called.
+ * returns:
+ * -1: error happened.
+ * 0: extent returned
+ * 1: no more extent left
+ */
+static int
+defrag_get_next_extent(int fd, struct getbmapx *map_out)
+{
+ int err = 0, i;
+
+ /* when no extents are cached in g_mapx, fetch from kernel */
+ if (g_ext_next_idx == -1) {
+ g_mapx[0].bmv_offset = g_offset;
+ g_mapx[0].bmv_length = -1LL;
+ g_mapx[0].bmv_count = MAPSIZE;
+ g_mapx[0].bmv_iflags = BMV_IF_NO_HOLES | BMV_IF_PREALLOC;
+ err = ioctl(fd, XFS_IOC_GETBMAPX, g_mapx);
+ if (err == -1) {
+ perror("XFS_IOC_GETBMAPX failed");
+ goto out;
+ }
+ /* for stats */
+ g_ext_stats.nr_ext_total += g_mapx[0].bmv_entries;
+
+ /* no more extents */
+ if (g_mapx[0].bmv_entries == 0) {
+ err = 1;
+ goto out;
+ }
+
+ /* for stats */
+ for (i = 1; i <= g_mapx[0].bmv_entries; i++) {
+ if (g_mapx[i].bmv_oflags & BMV_OF_PREALLOC)
+ g_ext_stats.nr_ext_unwritten++;
+ if (g_mapx[i].bmv_oflags & BMV_OF_SHARED)
+ g_ext_stats.nr_ext_shared++;
+ }
Quoted 8 lines

spaceman/defrag.c defrag_check_file(char *path) · line 151

Quoted 5 lines
spaceman/defrag.c +5 -0
+
+ g_ext_next_idx = 1;
+ g_offset = g_mapx[g_mapx[0].bmv_entries].bmv_offset +
+ g_mapx[g_mapx[0].bmv_entries].bmv_length;
+ }

Huh. AFAICT, g_ext_next_idx/g_mapx effectively act as a cursor over the mappings for this file segment.

Yes, right.

Quoted 6 lines

In that case, shouldn't
defrag_move_next_extent (which actually advances the cursor) be in charge of grabbing bmapx records from the kernel, and defrag_get_next_extent be the trivial helper to pass mappings to the consumer of the bmapx objects (aka defrag_get_next_segment)?

Defrag_move_next_extent moves the cursor and defrag_get_next_extent
Pick the extent on cursor but doesn’t move the cursor.
There is case that if we add current extent (returned by defrag_get_next_extent)
to current segment, current segment would be too big. So we don’t add current extent
But will add it to next segment. For next segment, defrag_get_next_extent is called
To get the extent in question again.
So as summary on the cases we move cursor:
1). The extent on cursor is not target to defrag (unwritten or too big) 2). The extent is added in current segment.

Quoted 104 lines
Quoted 104 lines
spaceman/defrag.c +104 -0
+
+ map_out->bmv_offset = g_mapx[g_ext_next_idx].bmv_offset;
+ map_out->bmv_length = g_mapx[g_ext_next_idx].bmv_length;
+ map_out->bmv_oflags = g_mapx[g_ext_next_idx].bmv_oflags;
+out:
+ return err;
+}
+
+/*
+ * move to next extent
+ */
+static void
+defrag_move_next_extent()
+{
+ if (g_ext_next_idx == g_mapx[0].bmv_entries)
+ g_ext_next_idx = -1;
+ else
+ g_ext_next_idx += 1;
+}
+
+/*
+ * check if the given extent is a defrag target.
+ * no need to check for holes as we are using BMV_IF_NO_HOLES
+ */
+static bool
+defrag_is_target(struct getbmapx *mapx)
+{
+ /* unwritten */
+ if (mapx->bmv_oflags & BMV_OF_PREALLOC)
+ return false;
+ return mapx->bmv_length < g_segment_size_lmt;
+}
+
+static bool
+defrag_is_extent_shared(struct getbmapx *mapx)
+{
+ return !!(mapx->bmv_oflags & BMV_OF_SHARED);
+}
+
+/*
+ * get next segment to defragment.
+ * returns:
+ * -1 error happened.
+ * 0 segment returned.
+ * 1 no more segments to return
+ */
+static int
+defrag_get_next_segment(int fd, struct defrag_segment *out)
+{
+ struct getbmapx mapx;
+ int ret;
+
+ out->ds_offset = 0;
+ out->ds_length = 0;
+ out->ds_nr = 0;
+ out->ds_shared = false;
+
+ do {
+ ret = defrag_get_next_extent(fd, &mapx);
+ if (ret != 0) {
+ /*
+  * no more extetns, return current segment if its not
+  * empty
+ */
+ if (ret == 1 && out->ds_nr > 0)
+ ret = 0;
+ /* otherwise, error heppened, stop */
+ break;
+ }
+
+ /*
+  * If the extent is not a defrag target, skip it.
+  * go to next extent if the segment is empty;
+  * otherwise return the segment.
+  */
+ if (!defrag_is_target(&mapx)) {
+ defrag_move_next_extent();
+ if (out->ds_nr == 0)
+ continue;
+ else
+ break;
+ }
+
+ /* check for segment size limitation */
+ if (out->ds_length + mapx.bmv_length > g_segment_size_lmt)
+ break;
+
+ /* the segment is empty now, add this extent to it for sure */
+ if (out->ds_nr == 0) {
+ out->ds_offset = mapx.bmv_offset;
+ goto add_ext;
+ }
+
+ /*
+  * the segment is not empty, check for hole since the last exent
+  * if a hole exist before this extent, this extent can't be
+  * added to the segment. return the segment
+  */
+ if (out->ds_offset + out->ds_length != mapx.bmv_offset)
+ break;
+
+add_ext:
+ if (defrag_is_extent_shared(&mapx))
+ out->ds_shared = true;
Quoted 8 lines

spaceman/defrag.c defrag_check_file(char *path) · line 258

Quoted 3 lines
spaceman/defrag.c +3 -0
+
+ out->ds_length += mapx.bmv_length;
+ out->ds_nr += 1;

OH, ok. So we walk the mappings for a file. If we can identify a run of contiguous written mappings, we define a segment to be the file range described by that run, up to whatever the maximum is (~4-16M). Each of these segments is defragmented (somehow). Is that correct?

Yes, correct.

Quoted 39 lines
Quoted 24 lines
spaceman/defrag.c +10 -0
+ defrag_move_next_extent();
+
+ } while (true);
+
+ return ret;
+}
+
 /*
 * defragment a file
 * return 0 if successfully done, 1 otherwise
@@ -92,6 +277,9 @@ defrag_xfs_defrag(char *file_path) {
 struct fsxattr fsx;
 int ret = 0;
 
+ g_offset = 0;
+ g_ext_next_idx = -1;
+
 fsx.fsx_nextents = 0;
 memset(&g_ext_stats, 0, sizeof(g_ext_stats));
 
@@ -119,6 +307,22 @@ defrag_xfs_defrag(char *file_path) {
 ret = 1;
 goto out;
 }

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 319

Quoted 10 lines
spaceman/defrag.c +10 -0
+
+ do {
+ struct defrag_segment segment;
+
+ ret = defrag_get_next_segment(defrag_fd, &segment);
+ /* no more segments, we are done */
+ if (ret == 1) {
+ ret = 0;
+ break;
+ }

If you reverse the polarity of the 0/1 return values (aka return 1 if there is a segment and 0 if there is none) then you can shorten this loop to:

Quoted 9 lines

struct defrag_segment segment;
int ret;

while ((ret = defrag_get_next_segment(...)) == 1) { /* process segment */
}

return ret;

Yes, that might be better. What I was just thinking is the “0” usually mean a ‘good’ return :D Will consider this.

Thanks,
Wengang

Quoted 13 lines

--D

Quoted 11 lines
spaceman/defrag.c +6 -0
+ /* error happened when reading bmap, stop here */
+ if (ret == -1) {
+ ret = 1;
+ break;
+ }
+ } while (true);

out:
if (scratch_fd != -1) {
close(scratch_fd);

Signature · 1 line
2.39.3 (Apple Git-146)
#9 lore raw

Re: [PATCH 2/9] spaceman/defrag: pick up segments from target file

In reply to Wengang Wang
Quoted 18 lines · Wengang Wang

segments are the smallest unit to defragment.

A segment
1. Can't exceed size limit
2. contains some extents
3. the contained extents can't be "unwritten"
4. the contained extents must be contigous in file blocks

---
 spaceman/defrag.c | 204 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 204 insertions(+)
spaceman/defrag.c +0 -0
index c9732984..175cf461 100644

spaceman/defrag.c line 23

Quoted 10 lines
spaceman/defrag.c +7 -0
@@ -14,6 +14,32 @@
 #include "space.h"
 #include "input.h"
 
+#define MAPSIZE 512
+/* used to fetch bmap */
+struct getbmapx	g_mapx[MAPSIZE];
+/* current offset of the file in units of 512 bytes, used to fetch bmap */
+static long long 	g_offset = 0;
+/* index to indentify next extent, used to get next extent */
+static int		g_ext_next_idx = -1;

Please do not prefix global variables with "g_". This is not useful, and simply makes the code hard to read.

That said, it is much better to pass these as function parameters so they are specific to the mapping context and so are inherently thread safe.

spaceman/defrag.c line 31

Quoted 7 lines
spaceman/defrag.c +7 -0
+/*
+ * segment, the smallest unit to defrag
+ * it includes some contiguous extents.
+ * no holes included,
+ * no unwritten extents included
+ * the size is limited by g_segment_size_lmt
+ */

I have no idea what this comment is trying to tell me.

Quoted 60 lines
spaceman/defrag.c +53 -0
+struct defrag_segment {
+	/* segment offset in units of 512 bytes */
+	long long	ds_offset;
+	/* length of segment in units of 512 bytes */
+	long long	ds_length;
+	/* number of extents in this segment */
+	int		ds_nr;
+	/* flag indicating if segment contains shared blocks */
+	bool		ds_shared;
+};
+
 /* defrag segment size limit in units of 512 bytes */
 #define MIN_SEGMENT_SIZE_LIMIT 8192 /* 4MiB */
 #define DEFAULT_SEGMENT_SIZE_LIMIT 32768 /* 16MiB */
@@ -78,6 +104,165 @@ defrag_check_file(char *path)
 	return true;
 }
 
+/*
+ * get next extent in the file.
+ * Note: next call will get the same extent unless move_next_extent() is called.
+ * returns:
+ * -1:	error happened.
+ * 0:	extent returned
+ * 1:	no more extent left
+ */
+static int
+defrag_get_next_extent(int fd, struct getbmapx *map_out)
+{
+	int err = 0, i;
+
+	/* when no extents are cached in g_mapx, fetch from kernel */
+	if (g_ext_next_idx == -1) {
+		g_mapx[0].bmv_offset = g_offset;
+		g_mapx[0].bmv_length = -1LL;
+		g_mapx[0].bmv_count = MAPSIZE;
+		g_mapx[0].bmv_iflags = BMV_IF_NO_HOLES | BMV_IF_PREALLOC;
+		err = ioctl(fd, XFS_IOC_GETBMAPX, g_mapx);
+		if (err == -1) {
+			perror("XFS_IOC_GETBMAPX failed");
+			goto out;
+		}
+		/* for stats */
+		g_ext_stats.nr_ext_total += g_mapx[0].bmv_entries;
+
+		/* no more extents */
+		if (g_mapx[0].bmv_entries == 0) {
+			err = 1;
+			goto out;
+		}
+
+		/* for stats */
+		for (i = 1; i <= g_mapx[0].bmv_entries; i++) {
+			if (g_mapx[i].bmv_oflags & BMV_OF_PREALLOC)
+				g_ext_stats.nr_ext_unwritten++;
+			if (g_mapx[i].bmv_oflags & BMV_OF_SHARED)
+				g_ext_stats.nr_ext_shared++;
+		}
+
+		g_ext_next_idx = 1;

spaceman/defrag.c defrag_check_file(char *path) · line 158

Quoted 10 lines
spaceman/defrag.c +10 -0
+		g_offset = g_mapx[g_mapx[0].bmv_entries].bmv_offset +
+				g_mapx[g_mapx[0].bmv_entries].bmv_length;
+	}
+
+	map_out->bmv_offset = g_mapx[g_ext_next_idx].bmv_offset;
+	map_out->bmv_length = g_mapx[g_ext_next_idx].bmv_length;
+	map_out->bmv_oflags = g_mapx[g_ext_next_idx].bmv_oflags;
+out:
+	return err;
+}

Ok, so the global variables are just a bmap cache. That's a problem, because this cache is stale the moment XFS_IOC_GETBMAPX returns to userspace. Iterating it to decide exactly waht to do next will race with ongoing file modifications and so it's not going to be accurate....

Quoted 39 lines
spaceman/defrag.c +39 -0
+
+/*
+ * move to next extent
+ */
+static void
+defrag_move_next_extent()
+{
+	if (g_ext_next_idx == g_mapx[0].bmv_entries)
+		g_ext_next_idx = -1;
+	else
+		g_ext_next_idx += 1;
+}
+
+/*
+ * check if the given extent is a defrag target.
+ * no need to check for holes as we are using BMV_IF_NO_HOLES
+ */
+static bool
+defrag_is_target(struct getbmapx *mapx)
+{
+	/* unwritten */
+	if (mapx->bmv_oflags & BMV_OF_PREALLOC)
+		return false;
+	return mapx->bmv_length < g_segment_size_lmt;
+}
+
+static bool
+defrag_is_extent_shared(struct getbmapx *mapx)
+{
+	return !!(mapx->bmv_oflags & BMV_OF_SHARED);
+}
+
+/*
+ * get next segment to defragment.
+ * returns:
+ * -1	error happened.
+ * 0	segment returned.
+ * 1	no more segments to return
+ */

spaceman/defrag.c defrag_check_file(char *path) · line 207

Quoted 10 lines
spaceman/defrag.c +10 -0
+static int
+defrag_get_next_segment(int fd, struct defrag_segment *out)
+{
+	struct getbmapx mapx;
+	int	ret;
+
+	out->ds_offset = 0;
+	out->ds_length = 0;
+	out->ds_nr = 0;
+	out->ds_shared = false;

out->ds_nr is never set to anything but zero in this patch.

spaceman/defrag.c defrag_check_file(char *path) · line 213

Quoted 6 lines
spaceman/defrag.c +6 -0
+
+	do {
+		ret = defrag_get_next_extent(fd, &mapx);
+		if (ret != 0) {
+			/*
+			 * no more extetns, return current segment if its not

Typos everywhere.

Quoted 21 lines
spaceman/defrag.c +20 -0
+			 * empty
+			*/
+			if (ret == 1 && out->ds_nr > 0)
+				ret = 0;
+			/* otherwise, error heppened, stop */
+			break;
+		}
 
+
+		/*
+		 * If the extent is not a defrag target, skip it.
+		 * go to next extent if the segment is empty;
+		 * otherwise return the segment.
+		 */
+		if (!defrag_is_target(&mapx)) {
+			defrag_move_next_extent();
+			if (out->ds_nr == 0)
+				continue;
+			else
+				break;
+		}

spaceman/defrag.c defrag_check_file(char *path) · line 243

Quoted 10 lines
spaceman/defrag.c +10 -0
+
+		/* check for segment size limitation */
+		if (out->ds_length + mapx.bmv_length > g_segment_size_lmt)
+			break;
+
+		/* the segment is empty now, add this extent to it for sure */
+		if (out->ds_nr == 0) {
+			out->ds_offset = mapx.bmv_offset;
+			goto add_ext;
+		}

So this is essentially a filter for the getbmapx output that strips away unwritten extents and anything outside/larger than the target range.

spaceman/defrag.c defrag_check_file(char *path) · line 253

Quoted 10 lines
spaceman/defrag.c +10 -0
+
+		/*
+		 * the segment is not empty, check for hole since the last exent
+		 * if a hole exist before this extent, this extent can't be
+		 * added to the segment. return the segment
+		 */
+		if (out->ds_offset + out->ds_length != mapx.bmv_offset)
+			break;
+
+add_ext:

Why do you need a goto for this logic?

		/*
		 * the segment is not empty, check for hole since the last exent
		 * if a hole exist before this extent, this extent can't be
		 * added to the segment. return the segment
		 */
		if (out->ds_nr) {
			if (out->ds_offset + out->ds_length != mapx.bmv_offset)
				break;
		} else {
			out->ds_offset = mapx.bmv_offset;
		}
Quoted 35 lines
spaceman/defrag.c +12 -0
+		if (defrag_is_extent_shared(&mapx))
+			out->ds_shared = true;
+
+		out->ds_length += mapx.bmv_length;
+		out->ds_nr += 1;
+		defrag_move_next_extent();
+
+	} while (true);
 
+
+	return ret;
+}
+
 /*

* defragment a file
* return 0 if successfully done, 1 otherwise

spaceman/defrag.c +8 -0
@@ -92,6 +277,9 @@ defrag_xfs_defrag(char *file_path) {
 	struct fsxattr	fsx;
 	int	ret = 0;
 
+	g_offset = 0;
+	g_ext_next_idx = -1;
+
 	fsx.fsx_nextents = 0;
 	memset(&g_ext_stats, 0, sizeof(g_ext_stats));
 
@@ -119,6 +307,22 @@ defrag_xfs_defrag(char *file_path) {
 		ret = 1;
 		goto out;
 	}
+
+	do {
+		struct defrag_segment segment;
+
+		ret = defrag_get_next_segment(defrag_fd, &segment);

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 324

Quoted 10 lines
spaceman/defrag.c +10 -0
+		/* no more segments, we are done */
+		if (ret == 1) {
+			ret = 0;
+			break;
+		}
+		/* error happened when reading bmap, stop here */
+		if (ret == -1) {
+			ret = 1;
+			break;
+		}

ternary return values are nasty. Return a negative errno when a an error occurs, and -ENODATA when there are no more segments. Then you have

		if (ret < 0) {
			if (ret == -ENODATA)
				exit_value = 0;
			else
				exit_value = 1;
			break;
		}

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 325

Quoted 1 line
spaceman/defrag.c +1 -0
+	} while (true);

Not a fan of do {} while(true) loops.

WIth the above error handling changes, this becomes:

	do {
		struct defrag_segment segment;

		ret = defrag_get_next_segment(defrag_fd, &segment);
	} while (ret == 0);

	if (ret == 0 || ret == -ENODATA)
		exit_value = 0;
	else
		exit_value = 1;

Ok, so this is a linear iteration of all extents in the file that filters extents for the specific "segment" that is going to be processed. I still have no idea why fixed length segments are important, but "linear extent scan for filtering" seems somewhat expensive.

Indeed, if you used FIEMAP, you can pass a minimum segment length to filter out all the small extents. Iterating that extent list means all the ranges you need to defrag are in the holes of the returned mapping information. This would be much faster than an entire linear mapping to find all the regions with small extents that need defrag. The second step could then be doing a fine grained mapping of each region that we now know either contains fragmented data or holes....

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#10 lore raw

Re: [PATCH 2/9] spaceman/defrag: pick up segments from target file

Quoted 37 lines · Dave Chinner
Quoted 18 lines · Wengang Wang

segments are the smallest unit to defragment.

A segment
1. Can't exceed size limit
2. contains some extents
3. the contained extents can't be "unwritten"
4. the contained extents must be contigous in file blocks

---
spaceman/defrag.c | 204 ++++++++++++++++++++++++++++++++++++++++++++++

1 file changed, 204 insertions(+)
spaceman/defrag.c +0 -0
index c9732984..175cf461 100644

spaceman/defrag.c line 23

Quoted 10 lines
spaceman/defrag.c +7 -0
@@ -14,6 +14,32 @@
 #include "space.h"
 #include "input.h"
 
+#define MAPSIZE 512
+/* used to fetch bmap */
+struct getbmapx g_mapx[MAPSIZE];
+/* current offset of the file in units of 512 bytes, used to fetch bmap */
+static long long  g_offset = 0;
+/* index to indentify next extent, used to get next extent */
+static int g_ext_next_idx = -1;

Please do not prefix global variables with "g_". This is not useful, and simply makes the code hard to read.

Quoted 4 lines

That said, it is much better to pass these as function parameters so they are specific to the mapping context and so are inherently thread safe.

OK, will try to move them to function parameters, But I do see global variables used in xfsprog.

Quoted 9 lines

spaceman/defrag.c line 31

Quoted 7 lines
spaceman/defrag.c +7 -0
+/*
+ * segment, the smallest unit to defrag
+ * it includes some contiguous extents.
+ * no holes included,
+ * no unwritten extents included
+ * the size is limited by g_segment_size_lmt
+ */

I have no idea what this comment is trying to tell me.

OK.

Quoted 69 lines
Quoted 69 lines
spaceman/defrag.c +61 -0
+struct defrag_segment {
+ /* segment offset in units of 512 bytes */
+ long long ds_offset;
+ /* length of segment in units of 512 bytes */
+ long long ds_length;
+ /* number of extents in this segment */
+ int ds_nr;
+ /* flag indicating if segment contains shared blocks */
+ bool ds_shared;
+};
+
 /* defrag segment size limit in units of 512 bytes */
 #define MIN_SEGMENT_SIZE_LIMIT 8192 /* 4MiB */
 #define DEFAULT_SEGMENT_SIZE_LIMIT 32768 /* 16MiB */
@@ -78,6 +104,165 @@ defrag_check_file(char *path)
 return true;
 }
 
+/*
+ * get next extent in the file.
+ * Note: next call will get the same extent unless move_next_extent() is called.
+ * returns:
+ * -1: error happened.
+ * 0: extent returned
+ * 1: no more extent left
+ */
+static int
+defrag_get_next_extent(int fd, struct getbmapx *map_out)
+{
+ int err = 0, i;
+
+ /* when no extents are cached in g_mapx, fetch from kernel */
+ if (g_ext_next_idx == -1) {
+ g_mapx[0].bmv_offset = g_offset;
+ g_mapx[0].bmv_length = -1LL;
+ g_mapx[0].bmv_count = MAPSIZE;
+ g_mapx[0].bmv_iflags = BMV_IF_NO_HOLES | BMV_IF_PREALLOC;
+ err = ioctl(fd, XFS_IOC_GETBMAPX, g_mapx);
+ if (err == -1) {
+ perror("XFS_IOC_GETBMAPX failed");
+ goto out;
+ }
+ /* for stats */
+ g_ext_stats.nr_ext_total += g_mapx[0].bmv_entries;
+
+ /* no more extents */
+ if (g_mapx[0].bmv_entries == 0) {
+ err = 1;
+ goto out;
+ }
+
+ /* for stats */
+ for (i = 1; i <= g_mapx[0].bmv_entries; i++) {
+ if (g_mapx[i].bmv_oflags & BMV_OF_PREALLOC)
+ g_ext_stats.nr_ext_unwritten++;
+ if (g_mapx[i].bmv_oflags & BMV_OF_SHARED)
+ g_ext_stats.nr_ext_shared++;
+ }
+
+ g_ext_next_idx = 1;
+ g_offset = g_mapx[g_mapx[0].bmv_entries].bmv_offset +
+ g_mapx[g_mapx[0].bmv_entries].bmv_length;
+ }
+
+ map_out->bmv_offset = g_mapx[g_ext_next_idx].bmv_offset;
+ map_out->bmv_length = g_mapx[g_ext_next_idx].bmv_length;
+ map_out->bmv_oflags = g_mapx[g_ext_next_idx].bmv_oflags;
+out:
Quoted 8 lines

spaceman/defrag.c defrag_check_file(char *path) · line 158

Quoted 2 lines
spaceman/defrag.c +2 -0
+ return err;
+}

Ok, so the global variables are just a bmap cache. That's a problem, because this cache is stale the moment XFS_IOC_GETBMAPX returns to userspace. Iterating it to decide exactly waht to do next will race with ongoing file modifications and so it's not going to be accurate....

Yes, there is racing.
And even we do defrag basing on stale extents, there is harm to the file under defrag though it might not bring good defrag result.

Quoted 44 lines
Quoted 44 lines
spaceman/defrag.c +43 -0
+
+/*
+ * move to next extent
+ */
+static void
+defrag_move_next_extent()
+{
+ if (g_ext_next_idx == g_mapx[0].bmv_entries)
+ g_ext_next_idx = -1;
+ else
+ g_ext_next_idx += 1;
+}
+
+/*
+ * check if the given extent is a defrag target.
+ * no need to check for holes as we are using BMV_IF_NO_HOLES
+ */
+static bool
+defrag_is_target(struct getbmapx *mapx)
+{
+ /* unwritten */
+ if (mapx->bmv_oflags & BMV_OF_PREALLOC)
+ return false;
+ return mapx->bmv_length < g_segment_size_lmt;
+}
+
+static bool
+defrag_is_extent_shared(struct getbmapx *mapx)
+{
+ return !!(mapx->bmv_oflags & BMV_OF_SHARED);
+}
+
+/*
+ * get next segment to defragment.
+ * returns:
+ * -1 error happened.
+ * 0 segment returned.
+ * 1 no more segments to return
+ */
+static int
+defrag_get_next_segment(int fd, struct defrag_segment *out)
+{
+ struct getbmapx mapx;
Quoted 9 lines

spaceman/defrag.c defrag_check_file(char *path) · line 207

Quoted 6 lines
spaceman/defrag.c +6 -0
+ int ret;
+
+ out->ds_offset = 0;
+ out->ds_length = 0;
+ out->ds_nr = 0;
+ out->ds_shared = false;

out->ds_nr is never set to anything but zero in this patch.

It’s set at line 211 in the raw patch.

206 +add_ext:
207 + if (defrag_is_extent_shared(&mapx))
208 + out->ds_shared = true;
209 +

210 +               out->ds_length += mapx.bmv_length;
211 +               out->ds_nr += 1;
212 +               defrag_move_next_extent();
Quoted 8 lines

spaceman/defrag.c defrag_check_file(char *path) · line 213

Quoted 6 lines
spaceman/defrag.c +6 -0
+
+ do {
+ ret = defrag_get_next_extent(fd, &mapx);
+ if (ret != 0) {
+ /*
+  * no more extetns, return current segment if its not

Typos everywhere.

OK.

Quoted 28 lines
Quoted 28 lines
spaceman/defrag.c +26 -0
+  * empty
+ */
+ if (ret == 1 && out->ds_nr > 0)
+ ret = 0;
+ /* otherwise, error heppened, stop */
+ break;
+ }
 
+
+ /*
+  * If the extent is not a defrag target, skip it.
+  * go to next extent if the segment is empty;
+  * otherwise return the segment.
+  */
+ if (!defrag_is_target(&mapx)) {
+ defrag_move_next_extent();
+ if (out->ds_nr == 0)
+ continue;
+ else
+ break;
+ }
+
+ /* check for segment size limitation */
+ if (out->ds_length + mapx.bmv_length > g_segment_size_lmt)
+ break;
+
+ /* the segment is empty now, add this extent to it for sure */
Quoted 8 lines

spaceman/defrag.c defrag_check_file(char *path) · line 243

Quoted 4 lines
spaceman/defrag.c +4 -0
+ if (out->ds_nr == 0) {
+ out->ds_offset = mapx.bmv_offset;
+ goto add_ext;
+ }

So this is essentially a filter for the getbmapx output that strips away unwritten extents and anything outside/larger than the target range.

yes.

Quoted 14 lines

spaceman/defrag.c defrag_check_file(char *path) · line 253

Quoted 11 lines
spaceman/defrag.c +10 -0
+
+ /*
+  * the segment is not empty, check for hole since the last exent
+  * if a hole exist before this extent, this extent can't be
+  * added to the segment. return the segment
+  */
+ if (out->ds_offset + out->ds_length != mapx.bmv_offset)
+ break;
+
+add_ext:

Why do you need a goto for this logic?

Quoted 12 lines

/*
* the segment is not empty, check for hole since the last exent
* if a hole exist before this extent, this extent can't be
* added to the segment. return the segment
*/
if (out->ds_nr) {
if (out->ds_offset + out->ds_length != mapx.bmv_offset) break;
} else {
out->ds_offset = mapx.bmv_offset;
}

Above code also work.
The using of "goto add_ext” saved a “if" inside a “if” making the code clearer to me. But I can change it as you expected.

Quoted 50 lines
Quoted 35 lines
spaceman/defrag.c +12 -0
+ if (defrag_is_extent_shared(&mapx))
+ out->ds_shared = true;
+
+ out->ds_length += mapx.bmv_length;
+ out->ds_nr += 1;
+ defrag_move_next_extent();
+
+ } while (true);
 
+
+ return ret;
+}
+
 /*

* defragment a file
* return 0 if successfully done, 1 otherwise

spaceman/defrag.c +8 -0
@@ -92,6 +277,9 @@ defrag_xfs_defrag(char *file_path) {
 struct fsxattr fsx;
 int ret = 0;
 
+ g_offset = 0;
+ g_ext_next_idx = -1;
+
 fsx.fsx_nextents = 0;
 memset(&g_ext_stats, 0, sizeof(g_ext_stats));
 
@@ -119,6 +307,22 @@ defrag_xfs_defrag(char *file_path) {
 ret = 1;
 goto out;
 }
+
+ do {
+ struct defrag_segment segment;
+
+ ret = defrag_get_next_segment(defrag_fd, &segment);

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 324

Quoted 10 lines
spaceman/defrag.c +10 -0
+ /* no more segments, we are done */
+ if (ret == 1) {
+ ret = 0;
+ break;
+ }
+ /* error happened when reading bmap, stop here */
+ if (ret == -1) {
+ ret = 1;
+ break;
+ }

ternary return values are nasty. Return a negative errno when a an error occurs, and -ENODATA when there are no more segments. Then you have

Quoted 7 lines

if (ret < 0) {
if (ret == -ENODATA)
exit_value = 0;
else
exit_value = 1;
break;
}

Agreed.
I have modified defrag_get_next_segment() and defrag_get_next_extent() for V2, Making them return -1 for error and 0 for success. And the "no more extents/segements” is checked by looking at their length. Will send them out.

Quoted 10 lines

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 325

Quoted 2 lines
spaceman/defrag.c +1 -0
+ } while (true);

Not a fan of do {} while(true) loops.

WIth the above error handling changes, this becomes:

do {
struct defrag_segment segment;

Quoted 8 lines

ret = defrag_get_next_segment(defrag_fd, &segment); } while (ret == 0);

if (ret == 0 || ret == -ENODATA)
exit_value = 0;
else
exit_value = 1;

Yes, I making “big” changes to defrag_get_next_segment()/defrag_get_next_extent() Please review them when I sent them out.

Quoted 6 lines

Ok, so this is a linear iteration of all extents in the file that filters extents for the specific "segment" that is going to be processed. I still have no idea why fixed length segments are important, but "linear extent scan for filtering" seems somewhat expensive.

Hm… fixed length segments — actually not fixed length segments, but segment
size can’t exceed the limitation. So segment.ds_length <= LIMIT.
Larger segment take longer time (with filed locked) to defrag. The segment size limit is a way to balance the defrag and the parallel IO latency.

Quoted 9 lines

Indeed, if you used FIEMAP, you can pass a minimum segment length to filter out all the small extents. Iterating that extent list means all the ranges you need to defrag are in the holes of the returned mapping information. This would be much faster than an entire linear mapping to find all the regions with small extents that need defrag. The second step could then be doing a fine grained mapping of each region that we now know either contains fragmented data or holes....

Hm… just a question here:
As your way, say you set the filter length to 2048, all extents with 2048 or less blocks are to defragmented. What if the extent layout is like this:

1.    1
2.    2049
3.    2
4.    2050
5.    1
6.    2051

In above case, do you do defrag or not?

As I understand the situation, performance of defrag it’s self is not a critical concern here.

Thanks,
Wengang

#11 lore raw

Re: [PATCH 2/9] spaceman/defrag: pick up segments from target file

Quoted 8 lines · Wengang Wang
Quoted 5 lines

Ok, so this is a linear iteration of all extents in the file that filters extents for the specific "segment" that is going to be processed. I still have no idea why fixed length segments are important, but "linear extent scan for filtering" seems somewhat expensive.

Hm… fixed length segments — actually not fixed length segments, but segment size can’t exceed the limitation. So segment.ds_length <= LIMIT.

Which is effectively fixed length segments....

Quoted 3 lines

Larger segment take longer time (with filed locked) to defrag. The segment size limit is a way to balance the defrag and the parallel IO latency.

Yes, I know why you've done it. These were the same arguments made a while back for a new way of cloning files on XFS. We solved those problems just with a small change to the locking, and didn't need new ioctls or lots of new code just to solve the "clone blocks concurrent IO" problem.

I'm looking at this from exactly the same POV. The code presented is doing lots of complex, unusable stuff to work around the fact that UNSHARE blocks concurrent IO. I don't see any difference between CLONE and UNSHARE from the IO perspective - if anything UNSHARE can have looser rules than CLONE, because a concurrent write will either do the COW of a shared block itself, or hit the exclusive block that has already been unshared.

So if we fix these locking issues in the kernel, then the whole need for working around the IO concurrency problems with UNSHARE goes away and the userspace code becomes much, much simpler.

Quoted 13 lines
Quoted 8 lines

Indeed, if you used FIEMAP, you can pass a minimum segment length to filter out all the small extents. Iterating that extent list means all the ranges you need to defrag are in the holes of the returned mapping information. This would be much faster than an entire linear mapping to find all the regions with small extents that need defrag. The second step could then be doing a fine grained mapping of each region that we now know either contains fragmented data or holes....

Hm… just a question here:
As your way, say you set the filter length to 2048, all extents with 2048 or less blocks are to defragmented. What if the extent layout is like this:

Quoted 8 lines
1.    1
2.    2049
3.    2
4.    2050
5.    1
6.    2051

In above case, do you do defrag or not?

The filtering presenting in the patch above will not defrag any of this with a 2048 block segment side, because the second extent in each segment extend beyond the configured max segment length. IOWs, it ends up with a single extent per "2048 block segment", and that won't get defragged with the current algorithm.

As it is, this really isn't a common fragmentation pattern for a file that does not contain shared extents, so I wouldn't expect to ever need to decide if this needs to be defragged or not.

However, it is exactly the layout I would expect to see for cloned and modified filesystem image files. That is, the common layout for such a "cloned from golden image" Vm images is this:

1.    1		written
2.    2049	shared
3.    2		written
4.    2050	shared
5.    1		written
6.    2051	shared

i.e. there are large chunks of contiguous shared extents between the small individual COW block modifications that have been made to customise the image for the deployed VM.

Either way, if the segment/filter length is 2048 blocks, then this isn't a pattern that should be defragmented. If the segment/filter length is 4096 or larger, then yes, this pattern should definitely be defragmented.

Quoted 2 lines

As I understand the situation, performance of defrag it’s self is not a critical concern here.

Sure, but implementing a low performing, high CPU consumption, entirely single threaded defragmentation model that requires specific tuning in every different environment it is run in doesn't seem like the best idea to me.

I'm trying to work out if there is a faster, simpler way of achieving the same goal....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#12 lore raw

Re: [PATCH 2/9] spaceman/defrag: pick up segments from target file

Quoted 20 lines · Dave Chinner
Quoted 8 lines · Wengang Wang
Quoted 5 lines

Ok, so this is a linear iteration of all extents in the file that filters extents for the specific "segment" that is going to be processed. I still have no idea why fixed length segments are important, but "linear extent scan for filtering" seems somewhat expensive.

Hm… fixed length segments — actually not fixed length segments, but segment size can’t exceed the limitation. So segment.ds_length <= LIMIT.

Which is effectively fixed length segments....

Quoted 3 lines

Larger segment take longer time (with filed locked) to defrag. The segment size limit is a way to balance the defrag and the parallel IO latency.

Quoted 5 lines

Yes, I know why you've done it. These were the same arguments made a while back for a new way of cloning files on XFS. We solved those problems just with a small change to the locking, and didn't need new ioctls or lots of new code just to solve the "clone blocks concurrent IO" problem.

I didn’t check the code history, but I am thinking you solved the problem by allow reads to go while cloning is in progress? Correct me if I'm wrong. The problem we hit is (heart beat) write timeout.

Quoted 56 lines

I'm looking at this from exactly the same POV. The code presented is doing lots of complex, unusable stuff to work around the fact that UNSHARE blocks concurrent IO. I don't see any difference between CLONE and UNSHARE from the IO perspective - if anything UNSHARE can have looser rules than CLONE, because a concurrent write will either do the COW of a shared block itself, or hit the exclusive block that has already been unshared.

So if we fix these locking issues in the kernel, then the whole need for working around the IO concurrency problems with UNSHARE goes away and the userspace code becomes much, much simpler.

Quoted 13 lines
Quoted 8 lines

Indeed, if you used FIEMAP, you can pass a minimum segment length to filter out all the small extents. Iterating that extent list means all the ranges you need to defrag are in the holes of the returned mapping information. This would be much faster than an entire linear mapping to find all the regions with small extents that need defrag. The second step could then be doing a fine grained mapping of each region that we now know either contains fragmented data or holes....

Hm… just a question here:
As your way, say you set the filter length to 2048, all extents with 2048 or less blocks are to defragmented. What if the extent layout is like this:

Quoted 8 lines
1.    1
2.    2049
3.    2
4.    2050
5.    1
6.    2051

In above case, do you do defrag or not?

The filtering presenting in the patch above will not defrag any of this with a 2048 block segment side, because the second extent in each segment extend beyond the configured max segment length. IOWs, it ends up with a single extent per "2048 block segment", and that won't get defragged with the current algorithm.

As it is, this really isn't a common fragmentation pattern for a file that does not contain shared extents, so I wouldn't expect to ever need to decide if this needs to be defragged or not.

However, it is exactly the layout I would expect to see for cloned and modified filesystem image files. That is, the common layout for such a "cloned from golden image" Vm images is this:

1.    1 written
2.    2049 shared
3.    2 written
4.    2050 shared
5.    1 written
6.    2051 shared
Quoted 8 lines

i.e. there are large chunks of contiguous shared extents between the small individual COW block modifications that have been made to customise the image for the deployed VM.

Either way, if the segment/filter length is 2048 blocks, then this isn't a pattern that should be defragmented. If the segment/filter length is 4096 or larger, then yes, this pattern should definitely be defragmented.

Yes, true. We should focus on real case layout.

Quoted 4 lines
Quoted 3 lines

As I understand the situation, performance of defrag it’s self is not a critical concern here.

Quoted 8 lines

Sure, but implementing a low performing, high CPU consumption, entirely single threaded defragmentation model that requires specific tuning in every different environment it is run in doesn't seem like the best idea to me.

I'm trying to work out if there is a faster, simpler way of achieving the same goal....

Great!

Wengang

#13 lore raw

Re: [PATCH 2/9] spaceman/defrag: pick up segments from target file

Quoted 26 lines · Wengang Wang
Quoted 20 lines · Dave Chinner
Quoted 8 lines · Wengang Wang
Quoted 5 lines

Ok, so this is a linear iteration of all extents in the file that filters extents for the specific "segment" that is going to be processed. I still have no idea why fixed length segments are important, but "linear extent scan for filtering" seems somewhat expensive.

Hm… fixed length segments — actually not fixed length segments, but segment size can’t exceed the limitation. So segment.ds_length <= LIMIT.

Which is effectively fixed length segments....

Quoted 3 lines

Larger segment take longer time (with filed locked) to defrag. The segment size limit is a way to balance the defrag and the parallel IO latency.

Quoted 5 lines

Yes, I know why you've done it. These were the same arguments made a while back for a new way of cloning files on XFS. We solved those problems just with a small change to the locking, and didn't need new ioctls or lots of new code just to solve the "clone blocks concurrent IO" problem.

Quoted 3 lines

I didn’t check the code history, but I am thinking you solved the problem by allow reads to go while cloning is in progress? Correct me if I'm wrong. The problem we hit is (heart beat) write timeout.

The reason this worked (allowing shared reads through and not writes) was that the VM infrastructure this was being done for uses a sidecar write channel to redirect writes while a clone is being done. i.e. writes are not blocked by the clone in progress because they are being done to a different file.

When the clone completes, those writes are folded back into the original image file. e.g. see the qemu-img commit -b <backing file> <file with delta writes> which will fold writes to a sidecar write file back into the original backing file that was just cloned....

What I'm suggesting is that when you run an backing file defragmentation, you use the same sidecar write setup as cloning whilst the defrag is done. Reads go straight through to the backing file, and writes get written to a delta write file. When the defrag is done the delta write file gets folded back into the backing file.

But for this to work, UNSHARE needs to use shared read locking so that read IO can be directed through the file at the same time as the UNSHARE is running. If this works for CLONE to avoid read and write blocking whilst the operation is in progress, the same mechanism should be able to be used for UNSHARE, too. At this point defrag using CLONE+UNSHARE shouldn't ever block read IO and shouldn't block write IO for any significant period of time, either...

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#14 lore raw

Re: [PATCH 2/9] spaceman/defrag: pick up segments from target file

In reply to Dave Chinner
Quoted 14 lines · Dave Chinner

Yes, I know why you've done it. These were the same arguments made a while back for a new way of cloning files on XFS. We solved those problems just with a small change to the locking, and didn't need new ioctls or lots of new code just to solve the "clone blocks concurrent IO" problem.

I'm looking at this from exactly the same POV. The code presented is doing lots of complex, unusable stuff to work around the fact that UNSHARE blocks concurrent IO. I don't see any difference between CLONE and UNSHARE from the IO perspective - if anything UNSHARE can have looser rules than CLONE, because a concurrent write will either do the COW of a shared block itself, or hit the exclusive block that has already been unshared.

Quoted 3 lines

So if we fix these locking issues in the kernel, then the whole need for working around the IO concurrency problems with UNSHARE goes away and the userspace code becomes much, much simpler.

Btw, the main problem with unshare isn't just locking, but that is extremely inefficient. It synchronously reads one block at a time, which makes it very, very slow. That's purely a kernel implementation detail, of course.

#15 lore raw

Re: [PATCH 2/9] spaceman/defrag: pick up segments from target file

In reply to Dave Chinner
Quoted 34 lines · Dave Chinner
Quoted 8 lines · Wengang Wang
Quoted 5 lines

Ok, so this is a linear iteration of all extents in the file that filters extents for the specific "segment" that is going to be processed. I still have no idea why fixed length segments are important, but "linear extent scan for filtering" seems somewhat expensive.

Hm… fixed length segments — actually not fixed length segments, but segment size can’t exceed the limitation. So segment.ds_length <= LIMIT.

Which is effectively fixed length segments....

Quoted 3 lines

Larger segment take longer time (with filed locked) to defrag. The segment size limit is a way to balance the defrag and the parallel IO latency.

Yes, I know why you've done it. These were the same arguments made a while back for a new way of cloning files on XFS. We solved those problems just with a small change to the locking, and didn't need new ioctls or lots of new code just to solve the "clone blocks concurrent IO" problem.

I'm looking at this from exactly the same POV. The code presented is doing lots of complex, unusable stuff to work around the fact that UNSHARE blocks concurrent IO. I don't see any difference between CLONE and UNSHARE from the IO perspective - if anything UNSHARE can have looser rules than CLONE, because a concurrent write will either do the COW of a shared block itself, or hit the exclusive block that has already been unshared.

Quoted 12 lines

So if we fix these locking issues in the kernel, then the whole need for working around the IO concurrency problems with UNSHARE goes away and the userspace code becomes much, much simpler.

Quoted 8 lines
Quoted 8 lines

Indeed, if you used FIEMAP, you can pass a minimum segment length to filter out all the small extents. Iterating that extent list means all the ranges you need to defrag are in the holes of the returned mapping information. This would be much faster than an entire linear mapping to find all the regions with small extents that need defrag. The second step could then be doing a fine grained mapping of each region that we now know either contains fragmented data or holes....

Where can we pass a minimum segment length to filter out things? I don’t see that in the fiemap structure:

A fiemap request is encoded within struct fiemap:

struct fiemap {
__u64 fm_start; /* logical offset (inclusive) at
* which to start mapping (in) */
__u64 fm_length; /* logical length of mapping which
* userspace cares about (in) */
__u32 fm_flags; /* FIEMAP_FLAG_* flags for request (in/out) */ __u32 fm_mapped_extents; /* number of extents that were
* mapped (out) */
__u32 fm_extent_count; /* size of fm_extents array (in) */ __u32 fm_reserved;
struct fiemap_extent fm_extents[0]; /* array of mapped extents (out) */ };

Thanks,
Wengang

Quoted 61 lines
Quoted 5 lines

Hm… just a question here:
As your way, say you set the filter length to 2048, all extents with 2048 or less blocks are to defragmented. What if the extent layout is like this:

Quoted 8 lines
1.    1
2.    2049
3.    2
4.    2050
5.    1
6.    2051

In above case, do you do defrag or not?

The filtering presenting in the patch above will not defrag any of this with a 2048 block segment side, because the second extent in each segment extend beyond the configured max segment length. IOWs, it ends up with a single extent per "2048 block segment", and that won't get defragged with the current algorithm.

As it is, this really isn't a common fragmentation pattern for a file that does not contain shared extents, so I wouldn't expect to ever need to decide if this needs to be defragged or not.

However, it is exactly the layout I would expect to see for cloned and modified filesystem image files. That is, the common layout for such a "cloned from golden image" Vm images is this:

1.    1 written
2.    2049 shared
3.    2 written
4.    2050 shared
5.    1 written
6.    2051 shared

i.e. there are large chunks of contiguous shared extents between the small individual COW block modifications that have been made to customise the image for the deployed VM.

Either way, if the segment/filter length is 2048 blocks, then this isn't a pattern that should be defragmented. If the segment/filter length is 4096 or larger, then yes, this pattern should definitely be defragmented.

Quoted 2 lines

As I understand the situation, performance of defrag it’s self is not a critical concern here.

Sure, but implementing a low performing, high CPU consumption, entirely single threaded defragmentation model that requires specific tuning in every different environment it is run in doesn't seem like the best idea to me.

I'm trying to work out if there is a faster, simpler way of achieving the same goal....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#16 lore raw

Re: [PATCH 2/9] spaceman/defrag: pick up segments from target file

Quoted 11 lines · Wengang Wang
Quoted 9 lines · Dave Chinner
Quoted 8 lines
Quoted 8 lines

Indeed, if you used FIEMAP, you can pass a minimum segment length to filter out all the small extents. Iterating that extent list means all the ranges you need to defrag are in the holes of the returned mapping information. This would be much faster than an entire linear mapping to find all the regions with small extents that need defrag. The second step could then be doing a fine grained mapping of each region that we now know either contains fragmented data or holes....

Quoted 2 lines

Where can we pass a minimum segment length to filter out things? I don’t see that in the fiemap structure:

Oh, sorry, too many similar APIs - it's FITRIM that has a minimum length filter build into it. It's something we could add if we really need it.

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#17 lore raw

PATCH 3/9 spaceman/defrag: defrag segments

In reply to Wengang Wang

For each segment, the following steps are done trying to defrag it:

1. share the segment with a temporary file
2. unshare the segment in the target file. kernel simulates Cow on the whole segment complete the unshare (defrag).
3. release blocks from the tempoary file.

---
 spaceman/defrag.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 114 insertions(+)
spaceman/defrag.c +114 -0
index 175cf461..9f11e36b 100644
@@ -263,6 +263,40 @@ add_ext:
 	return ret;
 }
 
+/*
+ * check if the segment exceeds EoF.
+ * fix up the clone range and return true if EoF happens,
+ * return false otherwise.
+ */
+static bool
+defrag_clone_eof(struct file_clone_range *clone)
+{
+	off_t delta;
+
+	delta = clone->src_offset + clone->src_length - g_defrag_file_size;
+	if (delta > 0) {
+		clone->src_length = 0; // to the end
+		return true;
+	}
+	return false;
+}
+
+/*
+ * get the time delta since pre_time in ms.
+ * pre_time should contains values fetched by gettimeofday()
+ * cur_time is used to store current time by gettimeofday()
+ */
+static long long
+get_time_delta_us(struct timeval *pre_time, struct timeval *cur_time)
+{
+	long long us;
+
+	gettimeofday(cur_time, NULL);
+	us = (cur_time->tv_sec - pre_time->tv_sec) * 1000000;
+	us += (cur_time->tv_usec - pre_time->tv_usec);
+	return us;
+}
+
 /*
  * defragment a file
  * return 0 if successfully done, 1 otherwise
@@ -273,6 +307,7 @@ defrag_xfs_defrag(char *file_path) {
 	long	nr_seg_defrag = 0, nr_ext_defrag = 0;
 	int	scratch_fd = -1, defrag_fd = -1;
 	char	tmp_file_path[PATH_MAX+1];
+	struct file_clone_range clone;
 	char	*defrag_dir;
 	struct fsxattr	fsx;
 	int	ret = 0;
@@ -296,6 +331,8 @@ defrag_xfs_defrag(char *file_path) {
 		goto out;
 	}
 
+	clone.src_fd = defrag_fd;
+
 	defrag_dir = dirname(file_path);
 	snprintf(tmp_file_path, PATH_MAX, "%s/.xfsdefrag_%d", defrag_dir,
 		getpid());
@@ -309,7 +346,11 @@ defrag_xfs_defrag(char *file_path) {
 	}
 
 	do {
+		struct timeval t_clone, t_unshare, t_punch_hole;
 		struct defrag_segment segment;
+		long long seg_size, seg_off;
+		int time_delta;
+		bool stop;
 
 		ret = defrag_get_next_segment(defrag_fd, &segment);
 		/* no more segments, we are done */
@@ -322,6 +363,79 @@ defrag_xfs_defrag(char *file_path) {
 			ret = 1;
 			break;
 		}
+
+		/* we are done if the segment contains only 1 extent */
+		if (segment.ds_nr < 2)
+			continue;
+
+		/* to bytes */
+		seg_off = segment.ds_offset * 512;
+		seg_size = segment.ds_length * 512;
+
+		clone.src_offset = seg_off;
+		clone.src_length = seg_size;
+		clone.dest_offset = seg_off;
+
+		/* checks for EoF and fix up clone */
+		stop = defrag_clone_eof(&clone);
+		gettimeofday(&t_clone, NULL);
+		ret = ioctl(scratch_fd, FICLONERANGE, &clone);
+		if (ret != 0) {
+			fprintf(stderr, "FICLONERANGE failed %s\n",
+				strerror(errno));
+			break;
+		}
+
+		/* for time stats */
+		time_delta = get_time_delta_us(&t_clone, &t_unshare);
+		if (time_delta > max_clone_us)
+			max_clone_us = time_delta;
+
+		/* for defrag stats */
+		nr_ext_defrag += segment.ds_nr;
+
+		/*
+		 * For the shared range to be unshared via a copy-on-write
+		 * operation in the file to be defragged. This causes the
+		 * file needing to be defragged to have new extents allocated
+		 * and the data to be copied over and written out.
+		 */
+		ret = fallocate(defrag_fd, FALLOC_FL_UNSHARE_RANGE, seg_off,
+				seg_size);
+		if (ret != 0) {
+			fprintf(stderr, "UNSHARE_RANGE failed %s\n",
+				strerror(errno));
+			break;
+		}
+
+		/* for time stats */
+		time_delta = get_time_delta_us(&t_unshare, &t_punch_hole);
+		if (time_delta > max_unshare_us)
+			max_unshare_us = time_delta;
+
+		/*
+		 * Punch out the original extents we shared to the
+		 * scratch file so they are returned to free space.
+		 */
+		ret = fallocate(scratch_fd,
+			FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, seg_off,
+			seg_size);
+		if (ret != 0) {
+			fprintf(stderr, "PUNCH_HOLE failed %s\n",
+				strerror(errno));
+			break;
+		}
+
+		/* for defrag stats */
+		nr_seg_defrag += 1;
+
+		/* for time stats */
+		time_delta = get_time_delta_us(&t_punch_hole, &t_clone);
+		if (time_delta > max_punch_us)
+			max_punch_us = time_delta;
+
+		if (stop)
+			break;
 	} while (true);
 out:
 	if (scratch_fd != -1) {
Signature · 1 line
2.39.3 (Apple Git-146)
#18 lore raw

Re: [PATCH 3/9] spaceman/defrag: defrag segments

Quoted 58 lines · Wengang Wang

For each segment, the following steps are done trying to defrag it:

1. share the segment with a temporary file
2. unshare the segment in the target file. kernel simulates Cow on the whole segment complete the unshare (defrag).
3. release blocks from the tempoary file.

---
 spaceman/defrag.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 114 insertions(+)
spaceman/defrag.c +34 -0
index 175cf461..9f11e36b 100644
@@ -263,6 +263,40 @@ add_ext:
 	return ret;
 }
 
+/*
+ * check if the segment exceeds EoF.
+ * fix up the clone range and return true if EoF happens,
+ * return false otherwise.
+ */
+static bool
+defrag_clone_eof(struct file_clone_range *clone)
+{
+	off_t delta;
+
+	delta = clone->src_offset + clone->src_length - g_defrag_file_size;
+	if (delta > 0) {
+		clone->src_length = 0; // to the end
+		return true;
+	}
+	return false;
+}
+
+/*
+ * get the time delta since pre_time in ms.
+ * pre_time should contains values fetched by gettimeofday()
+ * cur_time is used to store current time by gettimeofday()
+ */
+static long long
+get_time_delta_us(struct timeval *pre_time, struct timeval *cur_time)
+{
+	long long us;
+
+	gettimeofday(cur_time, NULL);
+	us = (cur_time->tv_sec - pre_time->tv_sec) * 1000000;
+	us += (cur_time->tv_usec - pre_time->tv_usec);
+	return us;
+}
+
 /*
  * defragment a file
  * return 0 if successfully done, 1 otherwise

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 313

Quoted 7 lines
spaceman/defrag.c +1 -0
@@ -273,6 +307,7 @@ defrag_xfs_defrag(char *file_path) {
 	long	nr_seg_defrag = 0, nr_ext_defrag = 0;
 	int	scratch_fd = -1, defrag_fd = -1;
 	char	tmp_file_path[PATH_MAX+1];
+	struct file_clone_range clone;
 	char	*defrag_dir;
 	struct fsxattr	fsx;
 	int	ret = 0;

Now that I see this, you might want to straighten up the lines:

	struct fsxattr	fsx = { };
	long		nr_seg_defrag = 0, nr_ext_defrag = 0;

etc. Note the "= { }" bit that means you don't have to memset them to zero explicitly.

Quoted 32 lines
spaceman/defrag.c +13 -0
@@ -296,6 +331,8 @@ defrag_xfs_defrag(char *file_path) {
 		goto out;
 	}
 
+	clone.src_fd = defrag_fd;
+
 	defrag_dir = dirname(file_path);
 	snprintf(tmp_file_path, PATH_MAX, "%s/.xfsdefrag_%d", defrag_dir,
 		getpid());
@@ -309,7 +346,11 @@ defrag_xfs_defrag(char *file_path) {
 	}
 
 	do {
+		struct timeval t_clone, t_unshare, t_punch_hole;
 		struct defrag_segment segment;
+		long long seg_size, seg_off;
+		int time_delta;
+		bool stop;
 
 		ret = defrag_get_next_segment(defrag_fd, &segment);
 		/* no more segments, we are done */
@@ -322,6 +363,79 @@ defrag_xfs_defrag(char *file_path) {
 			ret = 1;
 			break;
 		}
+
+		/* we are done if the segment contains only 1 extent */
+		if (segment.ds_nr < 2)
+			continue;
+
+		/* to bytes */
+		seg_off = segment.ds_offset * 512;

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 382

Quoted 10 lines
spaceman/defrag.c +10 -0
+		seg_size = segment.ds_length * 512;
+
+		clone.src_offset = seg_off;
+		clone.src_length = seg_size;
+		clone.dest_offset = seg_off;
+
+		/* checks for EoF and fix up clone */
+		stop = defrag_clone_eof(&clone);
+		gettimeofday(&t_clone, NULL);
+		ret = ioctl(scratch_fd, FICLONERANGE, &clone);

Hm, should the top-level defrag_f function check in the filetable[i].fsgeom structure that the fs supports reflink?

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 385

Quoted 3 lines
spaceman/defrag.c +3 -0
+		if (ret != 0) {
+			fprintf(stderr, "FICLONERANGE failed %s\n",
+				strerror(errno));

Might be useful to include the file_path in the error message:

/opt/a: FICLONERANGE failed Software caused connection abort

(maybe also put a semicolon before the strerror message?)

Quoted 27 lines
spaceman/defrag.c +27 -0
+			break;
+		}
+
+		/* for time stats */
+		time_delta = get_time_delta_us(&t_clone, &t_unshare);
+		if (time_delta > max_clone_us)
+			max_clone_us = time_delta;
+
+		/* for defrag stats */
+		nr_ext_defrag += segment.ds_nr;
+
+		/*
+		 * For the shared range to be unshared via a copy-on-write
+		 * operation in the file to be defragged. This causes the
+		 * file needing to be defragged to have new extents allocated
+		 * and the data to be copied over and written out.
+		 */
+		ret = fallocate(defrag_fd, FALLOC_FL_UNSHARE_RANGE, seg_off,
+				seg_size);
+		if (ret != 0) {
+			fprintf(stderr, "UNSHARE_RANGE failed %s\n",
+				strerror(errno));
+			break;
+		}
+
+		/* for time stats */
+		time_delta = get_time_delta_us(&t_unshare, &t_punch_hole);

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 422

Quoted 10 lines
spaceman/defrag.c +10 -0
+		if (time_delta > max_unshare_us)
+			max_unshare_us = time_delta;
+
+		/*
+		 * Punch out the original extents we shared to the
+		 * scratch file so they are returned to free space.
+		 */
+		ret = fallocate(scratch_fd,
+			FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, seg_off,
+			seg_size);

Indentation here (two tabs for a continuation). Or just ftruncate scratch_fd to zero bytes? I think you have to do that for the EOF stuff to work, right?

--D

Quoted 23 lines
spaceman/defrag.c +16 -0
+		if (ret != 0) {
+			fprintf(stderr, "PUNCH_HOLE failed %s\n",
+				strerror(errno));
+			break;
+		}
+
+		/* for defrag stats */
+		nr_seg_defrag += 1;
+
+		/* for time stats */
+		time_delta = get_time_delta_us(&t_punch_hole, &t_clone);
+		if (time_delta > max_punch_us)
+			max_punch_us = time_delta;
+
+		if (stop)
+			break;
 	} while (true);
 out:
 	if (scratch_fd != -1) {
Signature · 1 line
2.39.3 (Apple Git-146)
#19 lore raw

Re: [PATCH 3/9] spaceman/defrag: defrag segments

Quoted 73 lines · Darrick J. Wong
Quoted 58 lines · Wengang Wang

For each segment, the following steps are done trying to defrag it:

1. share the segment with a temporary file
2. unshare the segment in the target file. kernel simulates Cow on the whole segment complete the unshare (defrag).
3. release blocks from the tempoary file.

---
spaceman/defrag.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++

1 file changed, 114 insertions(+)
spaceman/defrag.c +34 -0
index 175cf461..9f11e36b 100644
@@ -263,6 +263,40 @@ add_ext:
 return ret;
 }
 
+/*
+ * check if the segment exceeds EoF.
+ * fix up the clone range and return true if EoF happens,
+ * return false otherwise.
+ */
+static bool
+defrag_clone_eof(struct file_clone_range *clone)
+{
+ off_t delta;
+
+ delta = clone->src_offset + clone->src_length - g_defrag_file_size;
+ if (delta > 0) {
+ clone->src_length = 0; // to the end
+ return true;
+ }
+ return false;
+}
+
+/*
+ * get the time delta since pre_time in ms.
+ * pre_time should contains values fetched by gettimeofday()
+ * cur_time is used to store current time by gettimeofday()
+ */
+static long long
+get_time_delta_us(struct timeval *pre_time, struct timeval *cur_time)
+{
+ long long us;
+
+ gettimeofday(cur_time, NULL);
+ us = (cur_time->tv_sec - pre_time->tv_sec) * 1000000;
+ us += (cur_time->tv_usec - pre_time->tv_usec);
+ return us;
+}
+
 /*
 * defragment a file
 * return 0 if successfully done, 1 otherwise

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 313

Quoted 7 lines
spaceman/defrag.c +1 -0
@@ -273,6 +307,7 @@ defrag_xfs_defrag(char *file_path) {
 long nr_seg_defrag = 0, nr_ext_defrag = 0;
 int scratch_fd = -1, defrag_fd = -1;
 char tmp_file_path[PATH_MAX+1];
+ struct file_clone_range clone;
 char *defrag_dir;
 struct fsxattr fsx;
 int ret = 0;

Now that I see this, you might want to straighten up the lines:

Quoted 5 lines

struct fsxattr fsx = { };
long nr_seg_defrag = 0, nr_ext_defrag = 0;

etc. Note the "= { }" bit that means you don't have to memset them to zero explicitly.

Nice!

Quoted 38 lines
Quoted 38 lines
spaceman/defrag.c +18 -0
@@ -296,6 +331,8 @@ defrag_xfs_defrag(char *file_path) {
 goto out;
 }
 
+ clone.src_fd = defrag_fd;
+
 defrag_dir = dirname(file_path);
 snprintf(tmp_file_path, PATH_MAX, "%s/.xfsdefrag_%d", defrag_dir,
 getpid());
@@ -309,7 +346,11 @@ defrag_xfs_defrag(char *file_path) {
 }
 
 do {
+ struct timeval t_clone, t_unshare, t_punch_hole;
 struct defrag_segment segment;
+ long long seg_size, seg_off;
+ int time_delta;
+ bool stop;
 
 ret = defrag_get_next_segment(defrag_fd, &segment);
 /* no more segments, we are done */
@@ -322,6 +363,79 @@ defrag_xfs_defrag(char *file_path) {
 ret = 1;
 break;
 }
+
+ /* we are done if the segment contains only 1 extent */
+ if (segment.ds_nr < 2)
+ continue;
+
+ /* to bytes */
+ seg_off = segment.ds_offset * 512;
+ seg_size = segment.ds_length * 512;
+
+ clone.src_offset = seg_off;
+ clone.src_length = seg_size;
+ clone.dest_offset = seg_off;
Quoted 8 lines

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 382

Quoted 5 lines
spaceman/defrag.c +5 -0
+
+ /* checks for EoF and fix up clone */
+ stop = defrag_clone_eof(&clone);
+ gettimeofday(&t_clone, NULL);
+ ret = ioctl(scratch_fd, FICLONERANGE, &clone);

Hm, should the top-level defrag_f function check in the filetable[i].fsgeom structure that the fs supports reflink?

Yes, good to know.

Quoted 10 lines

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 385

Quoted 4 lines
spaceman/defrag.c +3 -0
+ if (ret != 0) {
+ fprintf(stderr, "FICLONERANGE failed %s\n",
+ strerror(errno));

Might be useful to include the file_path in the error message:

/opt/a: FICLONERANGE failed Software caused connection abort

(maybe also put a semicolon before the strerror message?)

OK.

Quoted 32 lines
Quoted 32 lines
spaceman/defrag.c +31 -0
+ break;
+ }
+
+ /* for time stats */
+ time_delta = get_time_delta_us(&t_clone, &t_unshare);
+ if (time_delta > max_clone_us)
+ max_clone_us = time_delta;
+
+ /* for defrag stats */
+ nr_ext_defrag += segment.ds_nr;
+
+ /*
+  * For the shared range to be unshared via a copy-on-write
+  * operation in the file to be defragged. This causes the
+  * file needing to be defragged to have new extents allocated
+  * and the data to be copied over and written out.
+  */
+ ret = fallocate(defrag_fd, FALLOC_FL_UNSHARE_RANGE, seg_off,
+ seg_size);
+ if (ret != 0) {
+ fprintf(stderr, "UNSHARE_RANGE failed %s\n",
+ strerror(errno));
+ break;
+ }
+
+ /* for time stats */
+ time_delta = get_time_delta_us(&t_unshare, &t_punch_hole);
+ if (time_delta > max_unshare_us)
+ max_unshare_us = time_delta;
+
+ /*
Quoted 8 lines

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 422

Quoted 6 lines
spaceman/defrag.c +6 -0
+  * Punch out the original extents we shared to the
+  * scratch file so they are returned to free space.
+  */
+ ret = fallocate(scratch_fd,
+ FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, seg_off,
+ seg_size);

Indentation here (two tabs for a continuation).

OK.

Quoted 4 lines

Or just ftruncate
scratch_fd to zero bytes? I think you have to do that for the EOF stuff to work, right?

I’d truncate the UNSHARE range only in the loop. EOF stuff would be truncated on (O_TMPFILE) file close. The EOF stuff would be used for another purpose, see [PATCH 6/9] spaceman/defrag: workaround kernel

Thanks,
Wengang

Quoted 23 lines

--D

Quoted 21 lines
spaceman/defrag.c +16 -0
+ if (ret != 0) {
+ fprintf(stderr, "PUNCH_HOLE failed %s\n",
+ strerror(errno));
+ break;
+ }
+
+ /* for defrag stats */
+ nr_seg_defrag += 1;
+
+ /* for time stats */
+ time_delta = get_time_delta_us(&t_punch_hole, &t_clone);
+ if (time_delta > max_punch_us)
+ max_punch_us = time_delta;
+
+ if (stop)
+ break;
 } while (true);

out:
if (scratch_fd != -1) {

Signature · 1 line
2.39.3 (Apple Git-146)
#20 lore raw

Re: [PATCH 3/9] spaceman/defrag: defrag segments

Quoted 132 lines · Wengang Wang
Quoted 74 lines · Darrick J. Wong
Quoted 58 lines · Wengang Wang

For each segment, the following steps are done trying to defrag it:

1. share the segment with a temporary file
2. unshare the segment in the target file. kernel simulates Cow on the whole segment complete the unshare (defrag).
3. release blocks from the tempoary file.

---
spaceman/defrag.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++

1 file changed, 114 insertions(+)
spaceman/defrag.c +34 -0
index 175cf461..9f11e36b 100644
@@ -263,6 +263,40 @@ add_ext:
 return ret;
 }
 
+/*
+ * check if the segment exceeds EoF.
+ * fix up the clone range and return true if EoF happens,
+ * return false otherwise.
+ */
+static bool
+defrag_clone_eof(struct file_clone_range *clone)
+{
+ off_t delta;
+
+ delta = clone->src_offset + clone->src_length - g_defrag_file_size;
+ if (delta > 0) {
+ clone->src_length = 0; // to the end
+ return true;
+ }
+ return false;
+}
+
+/*
+ * get the time delta since pre_time in ms.
+ * pre_time should contains values fetched by gettimeofday()
+ * cur_time is used to store current time by gettimeofday()
+ */
+static long long
+get_time_delta_us(struct timeval *pre_time, struct timeval *cur_time)
+{
+ long long us;
+
+ gettimeofday(cur_time, NULL);
+ us = (cur_time->tv_sec - pre_time->tv_sec) * 1000000;
+ us += (cur_time->tv_usec - pre_time->tv_usec);
+ return us;
+}
+
 /*
 * defragment a file
 * return 0 if successfully done, 1 otherwise

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 313

Quoted 7 lines
spaceman/defrag.c +1 -0
@@ -273,6 +307,7 @@ defrag_xfs_defrag(char *file_path) {
 long nr_seg_defrag = 0, nr_ext_defrag = 0;
 int scratch_fd = -1, defrag_fd = -1;
 char tmp_file_path[PATH_MAX+1];
+ struct file_clone_range clone;
 char *defrag_dir;
 struct fsxattr fsx;
 int ret = 0;

Now that I see this, you might want to straighten up the lines:

Quoted 5 lines

struct fsxattr fsx = { };
long nr_seg_defrag = 0, nr_ext_defrag = 0;

etc. Note the "= { }" bit that means you don't have to memset them to zero explicitly.

Nice!

Quoted 38 lines
Quoted 38 lines
spaceman/defrag.c +18 -0
@@ -296,6 +331,8 @@ defrag_xfs_defrag(char *file_path) {
 goto out;
 }
 
+ clone.src_fd = defrag_fd;
+
 defrag_dir = dirname(file_path);
 snprintf(tmp_file_path, PATH_MAX, "%s/.xfsdefrag_%d", defrag_dir,
 getpid());
@@ -309,7 +346,11 @@ defrag_xfs_defrag(char *file_path) {
 }
 
 do {
+ struct timeval t_clone, t_unshare, t_punch_hole;
 struct defrag_segment segment;
+ long long seg_size, seg_off;
+ int time_delta;
+ bool stop;
 
 ret = defrag_get_next_segment(defrag_fd, &segment);
 /* no more segments, we are done */
@@ -322,6 +363,79 @@ defrag_xfs_defrag(char *file_path) {
 ret = 1;
 break;
 }
+
+ /* we are done if the segment contains only 1 extent */
+ if (segment.ds_nr < 2)
+ continue;
+
+ /* to bytes */
+ seg_off = segment.ds_offset * 512;
+ seg_size = segment.ds_length * 512;
+
+ clone.src_offset = seg_off;
+ clone.src_length = seg_size;
+ clone.dest_offset = seg_off;
Quoted 8 lines

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 382

Quoted 5 lines
spaceman/defrag.c +5 -0
+
+ /* checks for EoF and fix up clone */
+ stop = defrag_clone_eof(&clone);
+ gettimeofday(&t_clone, NULL);
+ ret = ioctl(scratch_fd, FICLONERANGE, &clone);

Hm, should the top-level defrag_f function check in the filetable[i].fsgeom structure that the fs supports reflink?

Quoted 1 line

Yes, good to know.

It seems that xfs_fsop_geom doesn’t know about reflink?

Thanks,
Wengang

Quoted 93 lines
Quoted 11 lines

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 385

Quoted 5 lines
spaceman/defrag.c +3 -0
+ if (ret != 0) {
+ fprintf(stderr, "FICLONERANGE failed %s\n",
+ strerror(errno));

Might be useful to include the file_path in the error message:

/opt/a: FICLONERANGE failed Software caused connection abort

(maybe also put a semicolon before the strerror message?)

OK.

Quoted 32 lines
Quoted 32 lines
spaceman/defrag.c +31 -0
+ break;
+ }
+
+ /* for time stats */
+ time_delta = get_time_delta_us(&t_clone, &t_unshare);
+ if (time_delta > max_clone_us)
+ max_clone_us = time_delta;
+
+ /* for defrag stats */
+ nr_ext_defrag += segment.ds_nr;
+
+ /*
+  * For the shared range to be unshared via a copy-on-write
+  * operation in the file to be defragged. This causes the
+  * file needing to be defragged to have new extents allocated
+  * and the data to be copied over and written out.
+  */
+ ret = fallocate(defrag_fd, FALLOC_FL_UNSHARE_RANGE, seg_off,
+ seg_size);
+ if (ret != 0) {
+ fprintf(stderr, "UNSHARE_RANGE failed %s\n",
+ strerror(errno));
+ break;
+ }
+
+ /* for time stats */
+ time_delta = get_time_delta_us(&t_unshare, &t_punch_hole);
+ if (time_delta > max_unshare_us)
+ max_unshare_us = time_delta;
+
+ /*
Quoted 8 lines

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 422

Quoted 6 lines
spaceman/defrag.c +6 -0
+  * Punch out the original extents we shared to the
+  * scratch file so they are returned to free space.
+  */
+ ret = fallocate(scratch_fd,
+ FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, seg_off,
+ seg_size);

Indentation here (two tabs for a continuation).

OK.

Quoted 4 lines

Or just ftruncate
scratch_fd to zero bytes? I think you have to do that for the EOF stuff to work, right?

I’d truncate the UNSHARE range only in the loop. EOF stuff would be truncated on (O_TMPFILE) file close. The EOF stuff would be used for another purpose, see [PATCH 6/9] spaceman/defrag: workaround kernel

Thanks,
Wengang

Quoted 23 lines

--D

Quoted 21 lines
spaceman/defrag.c +16 -0
+ if (ret != 0) {
+ fprintf(stderr, "PUNCH_HOLE failed %s\n",
+ strerror(errno));
+ break;
+ }
+
+ /* for defrag stats */
+ nr_seg_defrag += 1;
+
+ /* for time stats */
+ time_delta = get_time_delta_us(&t_punch_hole, &t_clone);
+ if (time_delta > max_punch_us)
+ max_punch_us = time_delta;
+
+ if (stop)
+ break;
 } while (true);

out:
if (scratch_fd != -1) {

Signature · 1 line
2.39.3 (Apple Git-146)
#21 lore raw

Re: [PATCH 3/9] spaceman/defrag: defrag segments

Quoted 25 lines · Wengang Wang
Quoted 25 lines · Wengang Wang
Quoted 22 lines · Darrick J. Wong
Quoted 16 lines · Wengang Wang

For each segment, the following steps are done trying to defrag it:

1. share the segment with a temporary file
2. unshare the segment in the target file. kernel simulates Cow on the whole segment complete the unshare (defrag).
3. release blocks from the tempoary file.

---
spaceman/defrag.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++

1 file changed, 114 insertions(+)
spaceman/defrag.c +0 -0
index 175cf461..9f11e36b 100644

<snip>

Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
Quoted 11 lines
spaceman/defrag.c +7 -0
@@ -322,6 +363,79 @@ defrag_xfs_defrag(char *file_path) {
 ret = 1;
 break;
 }
+
+ /* we are done if the segment contains only 1 extent */
+ if (segment.ds_nr < 2)
+ continue;
+
+ /* to bytes */
+ seg_off = segment.ds_offset * 512;

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 382

Quoted 10 lines
spaceman/defrag.c +10 -0
+ seg_size = segment.ds_length * 512;
+
+ clone.src_offset = seg_off;
+ clone.src_length = seg_size;
+ clone.dest_offset = seg_off;
+
+ /* checks for EoF and fix up clone */
+ stop = defrag_clone_eof(&clone);
+ gettimeofday(&t_clone, NULL);
+ ret = ioctl(scratch_fd, FICLONERANGE, &clone);

Hm, should the top-level defrag_f function check in the filetable[i].fsgeom structure that the fs supports reflink?

Quoted 3 lines
Quoted 1 line

Yes, good to know.

It seems that xfs_fsop_geom doesn’t know about reflink?

XFS_FSOP_GEOM_FLAGS_REFLINK ?

--D

Quoted 98 lines

Thanks,
Wengang

Quoted 93 lines
Quoted 11 lines

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 385

Quoted 5 lines
spaceman/defrag.c +3 -0
+ if (ret != 0) {
+ fprintf(stderr, "FICLONERANGE failed %s\n",
+ strerror(errno));

Might be useful to include the file_path in the error message:

/opt/a: FICLONERANGE failed Software caused connection abort

(maybe also put a semicolon before the strerror message?)

OK.

Quoted 32 lines
Quoted 32 lines
spaceman/defrag.c +31 -0
+ break;
+ }
+
+ /* for time stats */
+ time_delta = get_time_delta_us(&t_clone, &t_unshare);
+ if (time_delta > max_clone_us)
+ max_clone_us = time_delta;
+
+ /* for defrag stats */
+ nr_ext_defrag += segment.ds_nr;
+
+ /*
+  * For the shared range to be unshared via a copy-on-write
+  * operation in the file to be defragged. This causes the
+  * file needing to be defragged to have new extents allocated
+  * and the data to be copied over and written out.
+  */
+ ret = fallocate(defrag_fd, FALLOC_FL_UNSHARE_RANGE, seg_off,
+ seg_size);
+ if (ret != 0) {
+ fprintf(stderr, "UNSHARE_RANGE failed %s\n",
+ strerror(errno));
+ break;
+ }
+
+ /* for time stats */
+ time_delta = get_time_delta_us(&t_unshare, &t_punch_hole);
+ if (time_delta > max_unshare_us)
+ max_unshare_us = time_delta;
+
+ /*
Quoted 8 lines

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 422

Quoted 6 lines
spaceman/defrag.c +6 -0
+  * Punch out the original extents we shared to the
+  * scratch file so they are returned to free space.
+  */
+ ret = fallocate(scratch_fd,
+ FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, seg_off,
+ seg_size);

Indentation here (two tabs for a continuation).

OK.

Quoted 4 lines

Or just ftruncate
scratch_fd to zero bytes? I think you have to do that for the EOF stuff to work, right?

I’d truncate the UNSHARE range only in the loop. EOF stuff would be truncated on (O_TMPFILE) file close. The EOF stuff would be used for another purpose, see [PATCH 6/9] spaceman/defrag: workaround kernel

Thanks,
Wengang

Quoted 23 lines

--D

Quoted 21 lines
spaceman/defrag.c +16 -0
+ if (ret != 0) {
+ fprintf(stderr, "PUNCH_HOLE failed %s\n",
+ strerror(errno));
+ break;
+ }
+
+ /* for defrag stats */
+ nr_seg_defrag += 1;
+
+ /* for time stats */
+ time_delta = get_time_delta_us(&t_punch_hole, &t_clone);
+ if (time_delta > max_punch_us)
+ max_punch_us = time_delta;
+
+ if (stop)
+ break;
 } while (true);

out:
if (scratch_fd != -1) {

Signature · 1 line
2.39.3 (Apple Git-146)
#22 lore raw

Re: [PATCH 3/9] spaceman/defrag: defrag segments

In reply to Wengang Wang
Quoted 66 lines · Wengang Wang

For each segment, the following steps are done trying to defrag it:

1. share the segment with a temporary file
2. unshare the segment in the target file. kernel simulates Cow on the whole segment complete the unshare (defrag).
3. release blocks from the tempoary file.

---
 spaceman/defrag.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 114 insertions(+)
spaceman/defrag.c +35 -0
index 175cf461..9f11e36b 100644
@@ -263,6 +263,40 @@ add_ext:
 	return ret;
 }
 
+/*
+ * check if the segment exceeds EoF.
+ * fix up the clone range and return true if EoF happens,
+ * return false otherwise.
+ */
+static bool
+defrag_clone_eof(struct file_clone_range *clone)
+{
+	off_t delta;
+
+	delta = clone->src_offset + clone->src_length - g_defrag_file_size;
+	if (delta > 0) {
+		clone->src_length = 0; // to the end
+		return true;
+	}
+	return false;
+}
+
+/*
+ * get the time delta since pre_time in ms.
+ * pre_time should contains values fetched by gettimeofday()
+ * cur_time is used to store current time by gettimeofday()
+ */
+static long long
+get_time_delta_us(struct timeval *pre_time, struct timeval *cur_time)
+{
+	long long us;
+
+	gettimeofday(cur_time, NULL);
+	us = (cur_time->tv_sec - pre_time->tv_sec) * 1000000;
+	us += (cur_time->tv_usec - pre_time->tv_usec);
+	return us;
+}
+
 /*
  * defragment a file
  * return 0 if successfully done, 1 otherwise
@@ -273,6 +307,7 @@ defrag_xfs_defrag(char *file_path) {
 	long	nr_seg_defrag = 0, nr_ext_defrag = 0;
 	int	scratch_fd = -1, defrag_fd = -1;
 	char	tmp_file_path[PATH_MAX+1];
+	struct file_clone_range clone;
 	char	*defrag_dir;
 	struct fsxattr	fsx;
 	int	ret = 0;

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 336

Quoted 6 lines
spaceman/defrag.c +2 -0
@@ -296,6 +331,8 @@ defrag_xfs_defrag(char *file_path) {
 		goto out;
 	}
 
+	clone.src_fd = defrag_fd;
+
 	defrag_dir = dirname(file_path);

Just a note: can you please call this the "source fd", not the "defrag_fd"? defrag_fd could mean either the source or the temporary scratch file we use as the defrag destination.

Quoted 16 lines
spaceman/defrag.c +4 -0
 	snprintf(tmp_file_path, PATH_MAX, "%s/.xfsdefrag_%d", defrag_dir,
 		getpid());
@@ -309,7 +346,11 @@ defrag_xfs_defrag(char *file_path) {
 	}
 
 	do {
+		struct timeval t_clone, t_unshare, t_punch_hole;
 		struct defrag_segment segment;
+		long long seg_size, seg_off;
+		int time_delta;
+		bool stop;
 
 		ret = defrag_get_next_segment(defrag_fd, &segment);
 		/* no more segments, we are done */
@@ -322,6 +363,79 @@ defrag_xfs_defrag(char *file_path) {
 			ret = 1;

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 373

Quoted 10 lines
spaceman/defrag.c +8 -0
 			break;
 		}
+
+		/* we are done if the segment contains only 1 extent */
+		if (segment.ds_nr < 2)
+			continue;
+
+		/* to bytes */
+		seg_off = segment.ds_offset * 512;
+		seg_size = segment.ds_length * 512;

Ugh. Do this in the mapping code that gets the extent info. Have it return bytes. Or just use FIEMAP because it uses byte ranges to begin with.

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 380

Quoted 7 lines
spaceman/defrag.c +7 -0
+
+		clone.src_offset = seg_off;
+		clone.src_length = seg_size;
+		clone.dest_offset = seg_off;
+
+		/* checks for EoF and fix up clone */
+		stop = defrag_clone_eof(&clone);

Ok, so we copy the segment map into clone args, and ...

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 387

Quoted 7 lines
spaceman/defrag.c +7 -0
+		gettimeofday(&t_clone, NULL);
+		ret = ioctl(scratch_fd, FICLONERANGE, &clone);
+		if (ret != 0) {
+			fprintf(stderr, "FICLONERANGE failed %s\n",
+				strerror(errno));
+			break;
+		}

clone the source to the scratch file. This blocks writes to the source file while it is in progress, but allows reads to pass through the source file as data is not changing.

Quoted 11 lines
spaceman/defrag.c +11 -0
+		/* for time stats */
+		time_delta = get_time_delta_us(&t_clone, &t_unshare);
+		if (time_delta > max_clone_us)
+			max_clone_us = time_delta;
+
+		/* for defrag stats */
+		nr_ext_defrag += segment.ds_nr;
+
+		/*
+		 * For the shared range to be unshared via a copy-on-write
+		 * operation in the file to be defragged. This causes the

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 409

Quoted 10 lines
spaceman/defrag.c +10 -0
+		 * file needing to be defragged to have new extents allocated
+		 * and the data to be copied over and written out.
+		 */
+		ret = fallocate(defrag_fd, FALLOC_FL_UNSHARE_RANGE, seg_off,
+				seg_size);
+		if (ret != 0) {
+			fprintf(stderr, "UNSHARE_RANGE failed %s\n",
+				strerror(errno));
+			break;
+		}

And now we unshare the source file. This blocks all IO to the source file.

Ok, so this is the fundamental problem this whole "segmented defrag" is trying to work around: FALLOC_FL_UNSHARE_RANGE blocks all read and write IO whilst it is in progress.

We had this same problem with FICLONERANGE taking snapshots of VM files - we changed the locking to take shared IO locks to allow reads to run while the clone was in progress. Because the Oracle Vm infrastructure uses a sidecar to redirect writes while a snapshot (clone) was in progress, no VM IO got blocked while the clone was in progress and so the applications inside the VM never even noticed a clone was taking place.

Why isn't the same infrastructure being used here?

FALLOC_FL_UNSHARE_RANGE is not changing data, nor is it freeing any data blocks. Yes, we are re-writing the data somewhere else, but in that case the original data is still intact in it's original location on disk and not being freed.

Hence if a read races with UNSHARE, it will hit a referenced extent containing the correct data regardless of whether it is in the old or new file. Hence we can likely use shared IO locking for UNSHARE, just like we do for FICLONERANGE.

At this point, if the Oracle VM infrastructure uses the sidecar write channel whilst the defrag is in progress, this whole algorithm simply becomes "for regions with extents smaller than X, clone and unshare the region".

The whole need for "idle time" goes away. The need for segment size control largely goes away. The need to tune the defrag algorithm to avoid IO latency and/or throughput issues goes away.

Quoted 8 lines
spaceman/defrag.c +8 -0
+
+		/* for time stats */
+		time_delta = get_time_delta_us(&t_unshare, &t_punch_hole);
+		if (time_delta > max_unshare_us)
+			max_unshare_us = time_delta;
+
+		/*
+		 * Punch out the original extents we shared to the

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 427

Quoted 10 lines
spaceman/defrag.c +10 -0
+		 * scratch file so they are returned to free space.
+		 */
+		ret = fallocate(scratch_fd,
+			FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, seg_off,
+			seg_size);
+		if (ret != 0) {
+			fprintf(stderr, "PUNCH_HOLE failed %s\n",
+				strerror(errno));
+			break;
+		}

This is unnecessary if there is lots of free space. You can leave this to the very end of defrag so that the source file defrag operation isn't slowed down by cleaning up all the fragmented extents....

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#23 lore raw

Re: [PATCH 3/9] spaceman/defrag: defrag segments

Quoted 74 lines · Dave Chinner
Quoted 69 lines · Wengang Wang

For each segment, the following steps are done trying to defrag it:

1. share the segment with a temporary file
2. unshare the segment in the target file. kernel simulates Cow on the whole segment complete the unshare (defrag).
3. release blocks from the tempoary file.

---
spaceman/defrag.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++

1 file changed, 114 insertions(+)
spaceman/defrag.c +35 -0
index 175cf461..9f11e36b 100644
@@ -263,6 +263,40 @@ add_ext:
 return ret;
 }
 
+/*
+ * check if the segment exceeds EoF.
+ * fix up the clone range and return true if EoF happens,
+ * return false otherwise.
+ */
+static bool
+defrag_clone_eof(struct file_clone_range *clone)
+{
+ off_t delta;
+
+ delta = clone->src_offset + clone->src_length - g_defrag_file_size;
+ if (delta > 0) {
+ clone->src_length = 0; // to the end
+ return true;
+ }
+ return false;
+}
+
+/*
+ * get the time delta since pre_time in ms.
+ * pre_time should contains values fetched by gettimeofday()
+ * cur_time is used to store current time by gettimeofday()
+ */
+static long long
+get_time_delta_us(struct timeval *pre_time, struct timeval *cur_time)
+{
+ long long us;
+
+ gettimeofday(cur_time, NULL);
+ us = (cur_time->tv_sec - pre_time->tv_sec) * 1000000;
+ us += (cur_time->tv_usec - pre_time->tv_usec);
+ return us;
+}
+
 /*
 * defragment a file
 * return 0 if successfully done, 1 otherwise
@@ -273,6 +307,7 @@ defrag_xfs_defrag(char *file_path) {
 long nr_seg_defrag = 0, nr_ext_defrag = 0;
 int scratch_fd = -1, defrag_fd = -1;
 char tmp_file_path[PATH_MAX+1];
+ struct file_clone_range clone;
 char *defrag_dir;
 struct fsxattr fsx;
 int ret = 0;
@@ -296,6 +331,8 @@ defrag_xfs_defrag(char *file_path) {
 goto out;
 }
 
Quoted 7 lines

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 336

Quoted 3 lines
spaceman/defrag.c +2 -0
+ clone.src_fd = defrag_fd;
+
 defrag_dir = dirname(file_path);

Just a note: can you please call this the "source fd", not the "defrag_fd"? defrag_fd could mean either the source or the temporary scratch file we use as the defrag destination.

I have no problem to change that. Just I was thinking there is no data moving happening from a file to another. The defrag_fd means the file under defrag and the temp file is with scratch_fd.

Quoted 23 lines
Quoted 23 lines
spaceman/defrag.c +8 -0
 snprintf(tmp_file_path, PATH_MAX, "%s/.xfsdefrag_%d", defrag_dir,
 getpid());
@@ -309,7 +346,11 @@ defrag_xfs_defrag(char *file_path) {
 }
 
 do {
+ struct timeval t_clone, t_unshare, t_punch_hole;
 struct defrag_segment segment;
+ long long seg_size, seg_off;
+ int time_delta;
+ bool stop;
 
 ret = defrag_get_next_segment(defrag_fd, &segment);
 /* no more segments, we are done */
@@ -322,6 +363,79 @@ defrag_xfs_defrag(char *file_path) {
 ret = 1;
 break;
 }
+
+ /* we are done if the segment contains only 1 extent */
+ if (segment.ds_nr < 2)
+ continue;
Quoted 8 lines

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 373

Quoted 4 lines
spaceman/defrag.c +4 -0
+
+ /* to bytes */
+ seg_off = segment.ds_offset * 512;
+ seg_size = segment.ds_length * 512;

Ugh. Do this in the mapping code that gets the extent info. Have it return bytes. Or just use FIEMAP because it uses byte ranges to begin with.

OK.

Quoted 40 lines

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 380

Quoted 8 lines
spaceman/defrag.c +7 -0
+
+ clone.src_offset = seg_off;
+ clone.src_length = seg_size;
+ clone.dest_offset = seg_off;
+
+ /* checks for EoF and fix up clone */
+ stop = defrag_clone_eof(&clone);

Ok, so we copy the segment map into clone args, and ...

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 387

Quoted 7 lines
spaceman/defrag.c +7 -0
+ gettimeofday(&t_clone, NULL);
+ ret = ioctl(scratch_fd, FICLONERANGE, &clone);
+ if (ret != 0) {
+ fprintf(stderr, "FICLONERANGE failed %s\n",
+ strerror(errno));
+ break;
+ }

clone the source to the scratch file. This blocks writes to the source file while it is in progress, but allows reads to pass through the source file as data is not changing.

Quoted 16 lines
spaceman/defrag.c +16 -0
+ /* for time stats */
+ time_delta = get_time_delta_us(&t_clone, &t_unshare);
+ if (time_delta > max_clone_us)
+ max_clone_us = time_delta;
+
+ /* for defrag stats */
+ nr_ext_defrag += segment.ds_nr;
+
+ /*
+  * For the shared range to be unshared via a copy-on-write
+  * operation in the file to be defragged. This causes the
+  * file needing to be defragged to have new extents allocated
+  * and the data to be copied over and written out.
+  */
+ ret = fallocate(defrag_fd, FALLOC_FL_UNSHARE_RANGE, seg_off,
+ seg_size);
Quoted 8 lines

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 409

Quoted 5 lines
spaceman/defrag.c +5 -0
+ if (ret != 0) {
+ fprintf(stderr, "UNSHARE_RANGE failed %s\n",
+ strerror(errno));
+ break;
+ }

And now we unshare the source file. This blocks all IO to the source file.

Yes, even fetching data from disk is done while IO is locked. I am wondering if we can fetch data without locking IO. That’s also why I added readahead in a later patch.

Quoted 25 lines

Ok, so this is the fundamental problem this whole "segmented defrag" is trying to work around: FALLOC_FL_UNSHARE_RANGE blocks all read and write IO whilst it is in progress.

We had this same problem with FICLONERANGE taking snapshots of VM files - we changed the locking to take shared IO locks to allow reads to run while the clone was in progress. Because the Oracle Vm infrastructure uses a sidecar to redirect writes while a snapshot (clone) was in progress, no VM IO got blocked while the clone was in progress and so the applications inside the VM never even noticed a clone was taking place.

Why isn't the same infrastructure being used here?

FALLOC_FL_UNSHARE_RANGE is not changing data, nor is it freeing any data blocks. Yes, we are re-writing the data somewhere else, but in that case the original data is still intact in it's original location on disk and not being freed.

Hence if a read races with UNSHARE, it will hit a referenced extent containing the correct data regardless of whether it is in the old or new file. Hence we can likely use shared IO locking for UNSHARE, just like we do for FICLONERANGE.

Quoted 9 lines

At this point, if the Oracle VM infrastructure uses the sidecar write channel whilst the defrag is in progress, this whole algorithm simply becomes "for regions with extents smaller than X, clone and unshare the region".

The whole need for "idle time" goes away. The need for segment size control largely goes away. The need to tune the defrag algorithm to avoid IO latency and/or throughput issues goes away.

As we tested, the write-redirecting doesn’t work well.

Quoted 15 lines
Quoted 15 lines
spaceman/defrag.c +15 -0
+
+ /* for time stats */
+ time_delta = get_time_delta_us(&t_unshare, &t_punch_hole);
+ if (time_delta > max_unshare_us)
+ max_unshare_us = time_delta;
+
+ /*
+  * Punch out the original extents we shared to the
+  * scratch file so they are returned to free space.
+  */
+ ret = fallocate(scratch_fd,
+ FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE, seg_off,
+ seg_size);
+ if (ret != 0) {
+ fprintf(stderr, "PUNCH_HOLE failed %s\n",
Quoted 8 lines

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 387

Quoted 3 lines
spaceman/defrag.c +3 -0
+ strerror(errno));
+ break;
+ }

This is unnecessary if there is lots of free space. You can leave this to the very end of defrag so that the source file defrag operation isn't slowed down by cleaning up all the fragmented extents....

Yes. Two considerations here:
1. The space use as you mentioned, the temp file might grow huge on a low space system.
2. Punching holes on the temp file doesn’t lock the file under defrag, so with the PUNCH_HOLE, we need a little bit less sleep time for each segment.

Thanks,
Wengang

#24 lore raw

PATCH 4/9 spaceman/defrag: ctrl-c handler

In reply to Wengang Wang

Add this handler to break the defrag better, so it has
1. the stats reporting
2. remove the temporary file

---
 spaceman/defrag.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
spaceman/defrag.c +10 -1
index 9f11e36b..61e47a43 100644
@@ -297,6 +297,13 @@ get_time_delta_us(struct timeval *pre_time, struct timeval *cur_time)
 	return us;
 }
 
+static volatile bool usedKilled = false;
+void defrag_sigint_handler(int dummy)
+{
+	usedKilled = true;
+	printf("Please wait until current segment is defragmented\n");
+};
+
 /*
  * defragment a file
  * return 0 if successfully done, 1 otherwise
@@ -345,6 +352,8 @@ defrag_xfs_defrag(char *file_path) {
 		goto out;
 	}
 
+	signal(SIGINT, defrag_sigint_handler);
+
 	do {
 		struct timeval t_clone, t_unshare, t_punch_hole;
 		struct defrag_segment segment;
@@ -434,7 +443,7 @@ defrag_xfs_defrag(char *file_path) {
 		if (time_delta > max_punch_us)
 			max_punch_us = time_delta;
 
-		if (stop)
+		if (stop || usedKilled)
 			break;
 	} while (true);
 out:
Signature · 1 line
2.39.3 (Apple Git-146)
#25 lore raw

Re: [PATCH 4/9] spaceman/defrag: ctrl-c handler

Quoted 14 lines · Wengang Wang

Add this handler to break the defrag better, so it has
1. the stats reporting
2. remove the temporary file

---
 spaceman/defrag.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
spaceman/defrag.c +0 -0
index 9f11e36b..61e47a43 100644

spaceman/defrag.c get_time_delta_us(struct timeval *pre_time, struct timeval *cur_time) · line 303

Quoted 7 lines
spaceman/defrag.c +4 -0
@@ -297,6 +297,13 @@ get_time_delta_us(struct timeval *pre_time, struct timeval *cur_time)
 	return us;
 }
 
+static volatile bool usedKilled = false;
+void defrag_sigint_handler(int dummy)
+{
+	usedKilled = true;

Not sure why some of these variables are camelCase and others not. Or why this global variable doesn't have a g_ prefix like the others?

spaceman/defrag.c get_time_delta_us(struct timeval *pre_time, struct timeval *cur_time) · line 304

Quoted 1 line
spaceman/defrag.c +1 -0
+	printf("Please wait until current segment is defragmented\n");

Is it actually safe to call printf from a signal handler? Handlers must be very careful about what they call -- regreSSHion was a result of openssh not getting this right.

(Granted spaceman isn't as critical...)

Also would you rather SIGINT merely terminate the spaceman process? I think the file locks drop on termination, right?

--D

Quoted 27 lines
spaceman/defrag.c +5 -1
+};
+
 /*
  * defragment a file
  * return 0 if successfully done, 1 otherwise
@@ -345,6 +352,8 @@ defrag_xfs_defrag(char *file_path) {
 		goto out;
 	}
 
+	signal(SIGINT, defrag_sigint_handler);
+
 	do {
 		struct timeval t_clone, t_unshare, t_punch_hole;
 		struct defrag_segment segment;
@@ -434,7 +443,7 @@ defrag_xfs_defrag(char *file_path) {
 		if (time_delta > max_punch_us)
 			max_punch_us = time_delta;
 
-		if (stop)
+		if (stop || usedKilled)
 			break;
 	} while (true);
 out:
Signature · 1 line
2.39.3 (Apple Git-146)
#26 lore raw

Re: [PATCH 4/9] spaceman/defrag: ctrl-c handler

Quoted 22 lines · Darrick J. Wong
Quoted 17 lines · Wengang Wang

Add this handler to break the defrag better, so it has
1. the stats reporting
2. remove the temporary file

---
spaceman/defrag.c | 11 ++++++++++-

1 file changed, 10 insertions(+), 1 deletion(-)
spaceman/defrag.c +0 -0
index 9f11e36b..61e47a43 100644
@@ -297,6 +297,13 @@ get_time_delta_us(struct timeval *pre_time, struct timeval *cur_time)
 return us;
 }
 
Quoted 8 lines

spaceman/defrag.c get_time_delta_us(struct timeval *pre_time, struct timeval *cur_time) · line 303

Quoted 4 lines
spaceman/defrag.c +4 -0
+static volatile bool usedKilled = false;
+void defrag_sigint_handler(int dummy)
+{
+ usedKilled = true;

Not sure why some of these variables are camelCase and others not. Or why this global variable doesn't have a g_ prefix like the others?

Yep, will change it to g_user_killed.

Quoted 8 lines

spaceman/defrag.c get_time_delta_us(struct timeval *pre_time, struct timeval *cur_time) · line 304

Quoted 1 line
spaceman/defrag.c +1 -0
+ printf("Please wait until current segment is defragmented\n");

Is it actually safe to call printf from a signal handler? Handlers must be very careful about what they call -- regreSSHion was a result of openssh not getting this right.

(Granted spaceman isn't as critical...)

As the ioctl of UNSHARE takes time, so the process would really stop a while After user’s kil. The message is used as a quick response to user. It’s not actually Has any functionality. If it’s not safe, we can remove the message.

Quoted 2 lines

Also would you rather SIGINT merely terminate the spaceman process? I think the file locks drop on termination, right?

Another purpose of the handler is that I want to show the stats like below even process is killed:

Pre-defrag 54699 extents detected, 0 are "unwritten",0 are "shared" Tried to defragment 54697 extents (939511808 bytes) in 57 segments Time stats(ms): max clone: 33, max unshare: 2254, max punch_hole: 286 Post-defrag 12617 extents detected

Thanks,
Winging

Quoted 28 lines

--D

Quoted 25 lines
spaceman/defrag.c +5 -1
+};
+
 /*
 * defragment a file
 * return 0 if successfully done, 1 otherwise
@@ -345,6 +352,8 @@ defrag_xfs_defrag(char *file_path) {
 goto out;
 }
 
+ signal(SIGINT, defrag_sigint_handler);
+
 do {
 struct timeval t_clone, t_unshare, t_punch_hole;
 struct defrag_segment segment;
@@ -434,7 +443,7 @@ defrag_xfs_defrag(char *file_path) {
 if (time_delta > max_punch_us)
 max_punch_us = time_delta;
 
- if (stop)
+ if (stop || usedKilled)
 break;
 } while (true);

out:

Signature · 1 line
2.39.3 (Apple Git-146)
#27 lore raw

Re: [PATCH 4/9] spaceman/defrag: ctrl-c handler

Quoted 39 lines · Wengang Wang
Quoted 22 lines · Darrick J. Wong
Quoted 17 lines · Wengang Wang

Add this handler to break the defrag better, so it has
1. the stats reporting
2. remove the temporary file

---
spaceman/defrag.c | 11 ++++++++++-

1 file changed, 10 insertions(+), 1 deletion(-)
spaceman/defrag.c +0 -0
index 9f11e36b..61e47a43 100644
@@ -297,6 +297,13 @@ get_time_delta_us(struct timeval *pre_time, struct timeval *cur_time)
 return us;
 }
 
Quoted 8 lines

spaceman/defrag.c get_time_delta_us(struct timeval *pre_time, struct timeval *cur_time) · line 303

Quoted 4 lines
spaceman/defrag.c +4 -0
+static volatile bool usedKilled = false;
+void defrag_sigint_handler(int dummy)
+{
+ usedKilled = true;

Not sure why some of these variables are camelCase and others not. Or why this global variable doesn't have a g_ prefix like the others?

Yep, will change it to g_user_killed.

Quoted 6 lines

spaceman/defrag.c get_time_delta_us(struct timeval *pre_time, struct timeval *cur_time) · line 304

Quoted 1 line
spaceman/defrag.c +1 -0
+ printf("Please wait until current segment is defragmented\n");

Is it actually safe to call printf from a signal handler? Handlers must be very careful about what they call -- regreSSHion was a result of openssh not getting this right.

Quoted 6 lines
Quoted 2 lines

(Granted spaceman isn't as critical...)

As the ioctl of UNSHARE takes time, so the process would really stop a while After user’s kil. The message is used as a quick response to user. It’s not actually Has any functionality. If it’s not safe, we can remove the message.

$ man signal-safety
Quoted 9 lines
Quoted 2 lines

Also would you rather SIGINT merely terminate the spaceman process? I think the file locks drop on termination, right?

Another purpose of the handler is that I want to show the stats like below even process is killed:

Pre-defrag 54699 extents detected, 0 are "unwritten",0 are "shared" Tried to defragment 54697 extents (939511808 bytes) in 57 segments Time stats(ms): max clone: 33, max unshare: 2254, max punch_hole: 286 Post-defrag 12617 extents detected

Ah, ok.

Quoted 33 lines

Thanks,
Winging

Quoted 28 lines

--D

Quoted 25 lines
spaceman/defrag.c +5 -1
+};
+
 /*
 * defragment a file
 * return 0 if successfully done, 1 otherwise
@@ -345,6 +352,8 @@ defrag_xfs_defrag(char *file_path) {
 goto out;
 }
 
+ signal(SIGINT, defrag_sigint_handler);
+
 do {
 struct timeval t_clone, t_unshare, t_punch_hole;
 struct defrag_segment segment;
@@ -434,7 +443,7 @@ defrag_xfs_defrag(char *file_path) {
 if (time_delta > max_punch_us)
 max_punch_us = time_delta;
 
- if (stop)
+ if (stop || usedKilled)
 break;
 } while (true);

out:

Signature · 1 line
2.39.3 (Apple Git-146)
#28 lore raw

Re: [PATCH 4/9] spaceman/defrag: ctrl-c handler

Quoted 44 lines · Darrick J. Wong
Quoted 39 lines · Wengang Wang
Quoted 22 lines · Darrick J. Wong
Quoted 17 lines · Wengang Wang

Add this handler to break the defrag better, so it has
1. the stats reporting
2. remove the temporary file

---
spaceman/defrag.c | 11 ++++++++++-

1 file changed, 10 insertions(+), 1 deletion(-)
spaceman/defrag.c +0 -0
index 9f11e36b..61e47a43 100644
@@ -297,6 +297,13 @@ get_time_delta_us(struct timeval *pre_time, struct timeval *cur_time)
 return us;
 }
 
Quoted 8 lines

spaceman/defrag.c get_time_delta_us(struct timeval *pre_time, struct timeval *cur_time) · line 303

Quoted 4 lines
spaceman/defrag.c +4 -0
+static volatile bool usedKilled = false;
+void defrag_sigint_handler(int dummy)
+{
+ usedKilled = true;

Not sure why some of these variables are camelCase and others not. Or why this global variable doesn't have a g_ prefix like the others?

Yep, will change it to g_user_killed.

Quoted 6 lines

spaceman/defrag.c get_time_delta_us(struct timeval *pre_time, struct timeval *cur_time) · line 304

Quoted 1 line
spaceman/defrag.c +1 -0
+ printf("Please wait until current segment is defragmented\n");

Is it actually safe to call printf from a signal handler? Handlers must be very careful about what they call -- regreSSHion was a result of openssh not getting this right.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

(Granted spaceman isn't as critical...)

As the ioctl of UNSHARE takes time, so the process would really stop a while After user’s kil. The message is used as a quick response to user. It’s not actually Has any functionality. If it’s not safe, we can remove the message.

$ man signal-safety

Yep, will remove the print.

Thanks,
Wengang

#29 lore raw

PATCH 5/9 spaceman/defrag: exclude shared segments on low free space

In reply to Wengang Wang

On some XFS, free blocks are over-committed to reflink copies. And those free blocks are not enough if CoW happens to all the shared blocks.

This defrag tool would exclude shared segments when free space is under shrethold.

---
 spaceman/defrag.c | 46 +++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 43 insertions(+), 3 deletions(-)
spaceman/defrag.c +43 -3
index 61e47a43..f8e6713c 100644
@@ -304,6 +304,29 @@ void defrag_sigint_handler(int dummy)
 	printf("Please wait until current segment is defragmented\n");
 };
 
+/*
+ * limitation of filesystem free space in bytes.
+ * when filesystem has less free space than this number, segments which contain
+ * shared extents are skipped. 1GiB by default
+ */
+static long	g_limit_free_bytes = 1024 * 1024 * 1024;
+
+/*
+ * check if the free space in the FS is less than the _limit_
+ * return true if so, false otherwise
+ */
+static bool
+defrag_fs_limit_hit(int fd)
+{
+	struct statfs statfs_s;
+
+	if (g_limit_free_bytes <= 0)
+		return false;
+
+	fstatfs(fd, &statfs_s);
+	return statfs_s.f_bsize * statfs_s.f_bavail < g_limit_free_bytes;
+}
+
 /*
  * defragment a file
  * return 0 if successfully done, 1 otherwise
@@ -377,6 +400,15 @@ defrag_xfs_defrag(char *file_path) {
 		if (segment.ds_nr < 2)
 			continue;
 
+		/*
+		 * When the segment is (partially) shared, defrag would
+		 * consume free blocks. We check the limit of FS free blocks
+		 * and skip defragmenting this segment in case the limit is
+		 * reached.
+		 */
+		if (segment.ds_shared && defrag_fs_limit_hit(defrag_fd))
+			continue;
+
 		/* to bytes */
 		seg_off = segment.ds_offset * 512;
 		seg_size = segment.ds_length * 512;
@@ -478,7 +510,11 @@ static void defrag_help(void)
 "can be served durning the defragmentations.\n"
 "\n"
 " -s segment_size    -- specify the segment size in MiB, minmum value is 4 \n"
-"                       default is 16\n"));
+"                       default is 16\n"
+" -f free_space      -- specify shrethod of the XFS free space in MiB, when\n"
+"                       XFS free space is lower than that, shared segments \n"
+"                       are excluded from defragmentation, 1024 by default\n"
+	));
 }
 
 static cmdinfo_t defrag_cmd;
@@ -489,7 +525,7 @@ defrag_f(int argc, char **argv)
 	int	i;
 	int	c;
 
-	while ((c = getopt(argc, argv, "s:")) != EOF) {
+	while ((c = getopt(argc, argv, "s:f:")) != EOF) {
 		switch(c) {
 		case 's':
 			g_segment_size_lmt = atoi(optarg) * 1024 * 1024 / 512;
@@ -499,6 +535,10 @@ defrag_f(int argc, char **argv)
 					g_segment_size_lmt);
 			}
 			break;
+		case 'f':
+			g_limit_free_bytes = atol(optarg) * 1024 * 1024;
+			break;
+
 		default:
 			command_usage(&defrag_cmd);
 			return 1;
@@ -516,7 +556,7 @@ void defrag_init(void)
 	defrag_cmd.cfunc	= defrag_f;
 	defrag_cmd.argmin	= 0;
 	defrag_cmd.argmax	= 4;
-	defrag_cmd.args		= "[-s segment_size]";
+	defrag_cmd.args		= "[-s segment_size] [-f free_space]";
 	defrag_cmd.flags	= CMD_FLAG_ONESHOT;
 	defrag_cmd.oneline	= _("Defragment XFS files");
 	defrag_cmd.help		= defrag_help;
Signature · 1 line
2.39.3 (Apple Git-146)
#30 lore raw

Re: [PATCH 5/9] spaceman/defrag: exclude shared segments on low free space

Quoted 2 lines · Wengang Wang

On some XFS, free blocks are over-committed to reflink copies. And those free blocks are not enough if CoW happens to all the shared blocks.

Hmmm. I think what you're trying to do here is avoid running a filesystem out of space because it defragmented files A, B, ... Z, each of which previously shared the same chunk of storage but now they don't because this defragger unshared them to reduce the extent count in those files. Right?

In that case, I wonder if it's a good idea to touch shared extents at all? Someone set those files to share space, that's probably a better performance optimization than reducing extent count.

That said, you /could/ also use GETFSMAP to find all the other owners of a shared extent. Then you can reflink the same extent to a scratch file, copy the contents to a new region in the scratch file, and use FIEDEDUPERANGE on each of A..Z to remap the new region into those files. Assuming the new region has fewer mappings than the old one it was copied from, you'll defragment A..Z while preserving the sharing factor.

I say that because I've written such a thing before; look for csp_evac_dedupe_fsmap in
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/commit/?h=defrag-freespace&id=785d2f024e31a0d0f52b04073a600f9139ef0b21

Quoted 1 line

This defrag tool would exclude shared segments when free space is under shrethold.

"threshold"

--D

Quoted 101 lines
---
 spaceman/defrag.c | 46 +++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 43 insertions(+), 3 deletions(-)
spaceman/defrag.c +43 -3
index 61e47a43..f8e6713c 100644
@@ -304,6 +304,29 @@ void defrag_sigint_handler(int dummy)
 	printf("Please wait until current segment is defragmented\n");
 };
 
+/*
+ * limitation of filesystem free space in bytes.
+ * when filesystem has less free space than this number, segments which contain
+ * shared extents are skipped. 1GiB by default
+ */
+static long	g_limit_free_bytes = 1024 * 1024 * 1024;
+
+/*
+ * check if the free space in the FS is less than the _limit_
+ * return true if so, false otherwise
+ */
+static bool
+defrag_fs_limit_hit(int fd)
+{
+	struct statfs statfs_s;
+
+	if (g_limit_free_bytes <= 0)
+		return false;
+
+	fstatfs(fd, &statfs_s);
+	return statfs_s.f_bsize * statfs_s.f_bavail < g_limit_free_bytes;
+}
+
 /*
  * defragment a file
  * return 0 if successfully done, 1 otherwise
@@ -377,6 +400,15 @@ defrag_xfs_defrag(char *file_path) {
 		if (segment.ds_nr < 2)
 			continue;
 
+		/*
+		 * When the segment is (partially) shared, defrag would
+		 * consume free blocks. We check the limit of FS free blocks
+		 * and skip defragmenting this segment in case the limit is
+		 * reached.
+		 */
+		if (segment.ds_shared && defrag_fs_limit_hit(defrag_fd))
+			continue;
+
 		/* to bytes */
 		seg_off = segment.ds_offset * 512;
 		seg_size = segment.ds_length * 512;
@@ -478,7 +510,11 @@ static void defrag_help(void)
 "can be served durning the defragmentations.\n"
 "\n"
 " -s segment_size    -- specify the segment size in MiB, minmum value is 4 \n"
-"                       default is 16\n"));
+"                       default is 16\n"
+" -f free_space      -- specify shrethod of the XFS free space in MiB, when\n"
+"                       XFS free space is lower than that, shared segments \n"
+"                       are excluded from defragmentation, 1024 by default\n"
+	));
 }
 
 static cmdinfo_t defrag_cmd;
@@ -489,7 +525,7 @@ defrag_f(int argc, char **argv)
 	int	i;
 	int	c;
 
-	while ((c = getopt(argc, argv, "s:")) != EOF) {
+	while ((c = getopt(argc, argv, "s:f:")) != EOF) {
 		switch(c) {
 		case 's':
 			g_segment_size_lmt = atoi(optarg) * 1024 * 1024 / 512;
@@ -499,6 +535,10 @@ defrag_f(int argc, char **argv)
 					g_segment_size_lmt);
 			}
 			break;
+		case 'f':
+			g_limit_free_bytes = atol(optarg) * 1024 * 1024;
+			break;
+
 		default:
 			command_usage(&defrag_cmd);
 			return 1;
@@ -516,7 +556,7 @@ void defrag_init(void)
 	defrag_cmd.cfunc	= defrag_f;
 	defrag_cmd.argmin	= 0;
 	defrag_cmd.argmax	= 4;
-	defrag_cmd.args		= "[-s segment_size]";
+	defrag_cmd.args		= "[-s segment_size] [-f free_space]";
 	defrag_cmd.flags	= CMD_FLAG_ONESHOT;
 	defrag_cmd.oneline	= _("Defragment XFS files");
 	defrag_cmd.help		= defrag_help;
Signature · 1 line
2.39.3 (Apple Git-146)
#31 lore raw

Re: [PATCH 5/9] spaceman/defrag: exclude shared segments on low free space

Quoted 14 lines · Darrick J. Wong
Quoted 2 lines · Wengang Wang

On some XFS, free blocks are over-committed to reflink copies. And those free blocks are not enough if CoW happens to all the shared blocks.

Hmmm. I think what you're trying to do here is avoid running a filesystem out of space because it defragmented files A, B, ... Z, each of which previously shared the same chunk of storage but now they don't because this defragger unshared them to reduce the extent count in those files. Right?

Yes.

Quoted 3 lines

In that case, I wonder if it's a good idea to touch shared extents at all? Someone set those files to share space, that's probably a better performance optimization than reducing extent count.

The question is that:
Are the shared parts are something to be overwritten frequently?
If they are, Copy-on-Write would make those shared parts fragmented.
In above case we should dedefrag those parts, otherwise, the defrag might doesn’t defrag at all. Otherwise the shared parts are not subjects to be overwritten frequently,
They are expected to remain in big extents, choosing proper segment size
Would skip those.

But yes, we can add a option to simply skip those share extents.

Quoted 7 lines

That said, you /could/ also use GETFSMAP to find all the other owners of a shared extent. Then you can reflink the same extent to a scratch file, copy the contents to a new region in the scratch file, and use FIEDEDUPERANGE on each of A..Z to remap the new region into those files. Assuming the new region has fewer mappings than the old one it was copied from, you'll defragment A..Z while preserving the sharing factor.

That’s not safe? Things may change after GETFSMAP.

Quoted 8 lines

I say that because I've written such a thing before; look for csp_evac_dedupe_fsmap in
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/commit/?h=defrag-freespace&id=785d2f024e31a0d0f52b04073a600f9139ef0b21

Quoted 1 line

This defrag tool would exclude shared segments when free space is under shrethold.

"threshold"

OK.

Thanks
Wengang

Quoted 105 lines

--D

Quoted 101 lines

---
spaceman/defrag.c | 46 +++++++++++++++++++++++++++++++++++++++++++---

1 file changed, 43 insertions(+), 3 deletions(-)
spaceman/defrag.c +32 -0
index 61e47a43..f8e6713c 100644
@@ -304,6 +304,29 @@ void defrag_sigint_handler(int dummy)
 printf("Please wait until current segment is defragmented\n");
 };
 
+/*
+ * limitation of filesystem free space in bytes.
+ * when filesystem has less free space than this number, segments which contain
+ * shared extents are skipped. 1GiB by default
+ */
+static long g_limit_free_bytes = 1024 * 1024 * 1024;
+
+/*
+ * check if the free space in the FS is less than the _limit_
+ * return true if so, false otherwise
+ */
+static bool
+defrag_fs_limit_hit(int fd)
+{
+ struct statfs statfs_s;
+
+ if (g_limit_free_bytes <= 0)
+ return false;
+
+ fstatfs(fd, &statfs_s);
+ return statfs_s.f_bsize * statfs_s.f_bavail < g_limit_free_bytes;
+}
+
 /*
 * defragment a file
 * return 0 if successfully done, 1 otherwise
@@ -377,6 +400,15 @@ defrag_xfs_defrag(char *file_path) {
 if (segment.ds_nr < 2)
 continue;
 
+ /*
+ * When the segment is (partially) shared, defrag would
+ * consume free blocks. We check the limit of FS free blocks
+ * and skip defragmenting this segment in case the limit is
+ * reached.
+ */
+ if (segment.ds_shared && defrag_fs_limit_hit(defrag_fd))
+ continue;
+
 /* to bytes */
 seg_off = segment.ds_offset * 512;
 seg_size = segment.ds_length * 512;
@@ -478,7 +510,11 @@ static void defrag_help(void)

"can be served durning the defragmentations.\n"
"\n"
" -s segment_size -- specify the segment size in MiB, minmum value is 4 \n"
-" default is 16\n"));
+" default is 16\n"
+" -f free_space -- specify shrethod of the XFS free space in MiB, when\n"
+" XFS free space is lower than that, shared segments \n"
+" are excluded from defragmentation, 1024 by default\n"
+ ));
}

static cmdinfo_t defrag_cmd;

spaceman/defrag.c +1 -1
@@ -489,7 +525,7 @@ defrag_f(int argc, char **argv)
 int i;
 int c;
 
- while ((c = getopt(argc, argv, "s:")) != EOF) {
+ while ((c = getopt(argc, argv, "s:f:")) != EOF) {
 switch(c) {

case 's':
g_segment_size_lmt = atoi(optarg) * 1024 * 1024 / 512;

spaceman/defrag.c +4 -0
@@ -499,6 +535,10 @@ defrag_f(int argc, char **argv)
 g_segment_size_lmt);
 }
 break;
+ case 'f':
+ g_limit_free_bytes = atol(optarg) * 1024 * 1024;
+ break;
+

default:
command_usage(&defrag_cmd);
return 1;

spaceman/defrag.c +1 -1
@@ -516,7 +556,7 @@ void defrag_init(void)
 defrag_cmd.cfunc = defrag_f;
 defrag_cmd.argmin = 0;
 defrag_cmd.argmax = 4;
- defrag_cmd.args = "[-s segment_size]";
+ defrag_cmd.args = "[-s segment_size] [-f free_space]";
 defrag_cmd.flags = CMD_FLAG_ONESHOT;
 defrag_cmd.oneline = _("Defragment XFS files");
 defrag_cmd.help = defrag_help;
Signature · 1 line
2.39.3 (Apple Git-146)
#32 lore raw

Re: [PATCH 5/9] spaceman/defrag: exclude shared segments on low free space

Quoted 29 lines · Wengang Wang
Quoted 14 lines · Darrick J. Wong
Quoted 2 lines · Wengang Wang

On some XFS, free blocks are over-committed to reflink copies. And those free blocks are not enough if CoW happens to all the shared blocks.

Hmmm. I think what you're trying to do here is avoid running a filesystem out of space because it defragmented files A, B, ... Z, each of which previously shared the same chunk of storage but now they don't because this defragger unshared them to reduce the extent count in those files. Right?

Yes.

Quoted 3 lines

In that case, I wonder if it's a good idea to touch shared extents at all? Someone set those files to share space, that's probably a better performance optimization than reducing extent count.

The question is that:
Are the shared parts are something to be overwritten frequently?
If they are, Copy-on-Write would make those shared parts fragmented.
In above case we should dedefrag those parts, otherwise, the defrag might doesn’t defrag at all. Otherwise the shared parts are not subjects to be overwritten frequently,
They are expected to remain in big extents, choosing proper segment size
Would skip those.

Quoted 1 line

But yes, we can add a option to simply skip those share extents.

Good enough for now, I think. :)

Quoted 9 lines
Quoted 7 lines

That said, you /could/ also use GETFSMAP to find all the other owners of a shared extent. Then you can reflink the same extent to a scratch file, copy the contents to a new region in the scratch file, and use FIEDEDUPERANGE on each of A..Z to remap the new region into those files. Assuming the new region has fewer mappings than the old one it was copied from, you'll defragment A..Z while preserving the sharing factor.

That’s not safe? Things may change after GETFSMAP.

It is if after you reflink the same extent to a scratch file, you then check that what was reflinked into that scratch file is the same space that you thought you were cloning. If not, truncate the scratch file and try the GETFSMAP again.

The dedupe should be safe because it doesn't remap unless the contents match.

--D

Quoted 119 lines
Quoted 8 lines

I say that because I've written such a thing before; look for csp_evac_dedupe_fsmap in
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git/commit/?h=defrag-freespace&id=785d2f024e31a0d0f52b04073a600f9139ef0b21

Quoted 1 line

This defrag tool would exclude shared segments when free space is under shrethold.

"threshold"

OK.

Thanks
Wengang

Quoted 105 lines

--D

Quoted 101 lines

---
spaceman/defrag.c | 46 +++++++++++++++++++++++++++++++++++++++++++---

1 file changed, 43 insertions(+), 3 deletions(-)
spaceman/defrag.c +32 -0
index 61e47a43..f8e6713c 100644
@@ -304,6 +304,29 @@ void defrag_sigint_handler(int dummy)
 printf("Please wait until current segment is defragmented\n");
 };
 
+/*
+ * limitation of filesystem free space in bytes.
+ * when filesystem has less free space than this number, segments which contain
+ * shared extents are skipped. 1GiB by default
+ */
+static long g_limit_free_bytes = 1024 * 1024 * 1024;
+
+/*
+ * check if the free space in the FS is less than the _limit_
+ * return true if so, false otherwise
+ */
+static bool
+defrag_fs_limit_hit(int fd)
+{
+ struct statfs statfs_s;
+
+ if (g_limit_free_bytes <= 0)
+ return false;
+
+ fstatfs(fd, &statfs_s);
+ return statfs_s.f_bsize * statfs_s.f_bavail < g_limit_free_bytes;
+}
+
 /*
 * defragment a file
 * return 0 if successfully done, 1 otherwise
@@ -377,6 +400,15 @@ defrag_xfs_defrag(char *file_path) {
 if (segment.ds_nr < 2)
 continue;
 
+ /*
+ * When the segment is (partially) shared, defrag would
+ * consume free blocks. We check the limit of FS free blocks
+ * and skip defragmenting this segment in case the limit is
+ * reached.
+ */
+ if (segment.ds_shared && defrag_fs_limit_hit(defrag_fd))
+ continue;
+
 /* to bytes */
 seg_off = segment.ds_offset * 512;
 seg_size = segment.ds_length * 512;
@@ -478,7 +510,11 @@ static void defrag_help(void)

"can be served durning the defragmentations.\n"
"\n"
" -s segment_size -- specify the segment size in MiB, minmum value is 4 \n"
-" default is 16\n"));
+" default is 16\n"
+" -f free_space -- specify shrethod of the XFS free space in MiB, when\n"
+" XFS free space is lower than that, shared segments \n"
+" are excluded from defragmentation, 1024 by default\n"
+ ));
}

static cmdinfo_t defrag_cmd;

spaceman/defrag.c +1 -1
@@ -489,7 +525,7 @@ defrag_f(int argc, char **argv)
 int i;
 int c;
 
- while ((c = getopt(argc, argv, "s:")) != EOF) {
+ while ((c = getopt(argc, argv, "s:f:")) != EOF) {
 switch(c) {

case 's':
g_segment_size_lmt = atoi(optarg) * 1024 * 1024 / 512;

spaceman/defrag.c +4 -0
@@ -499,6 +535,10 @@ defrag_f(int argc, char **argv)
 g_segment_size_lmt);
 }
 break;
+ case 'f':
+ g_limit_free_bytes = atol(optarg) * 1024 * 1024;
+ break;
+

default:
command_usage(&defrag_cmd);
return 1;

spaceman/defrag.c +1 -1
@@ -516,7 +556,7 @@ void defrag_init(void)
 defrag_cmd.cfunc = defrag_f;
 defrag_cmd.argmin = 0;
 defrag_cmd.argmax = 4;
- defrag_cmd.args = "[-s segment_size]";
+ defrag_cmd.args = "[-s segment_size] [-f free_space]";
 defrag_cmd.flags = CMD_FLAG_ONESHOT;
 defrag_cmd.oneline = _("Defragment XFS files");
 defrag_cmd.help = defrag_help;
Signature · 1 line
2.39.3 (Apple Git-146)
#33 lore raw

PATCH 6/9 spaceman/defrag: workaround kernel xfs_reflink_try_clear_inode_flag()

In reply to Wengang Wang

xfs_reflink_try_clear_inode_flag() takes very long in case file has huge number of extents and none of the extents are shared.

workaround:
share the first real extent so that xfs_reflink_try_clear_inode_flag() returns quickly to save cpu times and speed up defrag significantly.

---
 spaceman/defrag.c | 174 +++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 172 insertions(+), 2 deletions(-)
spaceman/defrag.c +172 -2
index f8e6713c..b5c5b187 100644
@@ -327,6 +327,155 @@ defrag_fs_limit_hit(int fd)
 	return statfs_s.f_bsize * statfs_s.f_bavail < g_limit_free_bytes;
 }
 
+static bool g_enable_first_ext_share = true;
+
+static int
+defrag_get_first_real_ext(int fd, struct getbmapx *mapx)
+{
+	int			err;
+
+	while (1) {
+		err = defrag_get_next_extent(fd, mapx);
+		if (err)
+			break;
+
+		defrag_move_next_extent();
+		if (!(mapx->bmv_oflags & BMV_OF_PREALLOC))
+			break;
+	}
+	return err;
+}
+
+static __u64 g_share_offset = -1ULL;
+static __u64 g_share_len = 0ULL;
+#define SHARE_MAX_SIZE 32768  /* 32KiB */
+
+/* share the first real extent with scrach */
+static void
+defrag_share_first_extent(int defrag_fd, int scratch_fd)
+{
+#define OFFSET_1PB 0x4000000000000LL
+	struct file_clone_range clone;
+	struct getbmapx mapx;
+	int	err;
+
+	if (g_enable_first_ext_share == false)
+		return;
+
+	err = defrag_get_first_real_ext(defrag_fd, &mapx);
+	if (err)
+		return;
+
+	clone.src_fd = defrag_fd;
+	clone.src_offset = mapx.bmv_offset * 512;
+	clone.src_length = mapx.bmv_length * 512;
+	/* shares at most SHARE_MAX_SIZE length */
+	if (clone.src_length > SHARE_MAX_SIZE)
+		clone.src_length = SHARE_MAX_SIZE;
+	clone.dest_offset = OFFSET_1PB + clone.src_offset;
+	/* if the first is extent is reaching the EoF, no need to share */
+	if (clone.src_offset + clone.src_length >= g_defrag_file_size)
+		return;
+	err = ioctl(scratch_fd, FICLONERANGE, &clone);
+	if (err != 0) {
+		fprintf(stderr, "cloning first extent failed: %s\n",
+			strerror(errno));
+		return;
+	}
+
+	/* safe the offset and length for re-share */
+	g_share_offset = clone.src_offset;
+	g_share_len = clone.src_length;
+}
+
+/* re-share the blocks we shared previous if then are no longer shared */
+static void
+defrag_reshare_blocks_in_front(int defrag_fd, int scratch_fd)
+{
+#define NR_GET_EXT 9
+	struct getbmapx mapx[NR_GET_EXT];
+	struct file_clone_range clone;
+	__u64	new_share_len;
+	int	idx, err;
+
+	if (g_enable_first_ext_share == false)
+		return;
+
+	if (g_share_len == 0ULL)
+		return;
+
+	/*
+	 * check if previous shareing still exist
+	 * we are done if (partially) so.
+	 */
+	mapx[0].bmv_offset = g_share_offset;
+	mapx[0].bmv_length = g_share_len;
+	mapx[0].bmv_count = NR_GET_EXT;
+	mapx[0].bmv_iflags = BMV_IF_NO_HOLES | BMV_IF_PREALLOC;
+	err = ioctl(defrag_fd, XFS_IOC_GETBMAPX, mapx);
+	if (err) {
+		fprintf(stderr, "XFS_IOC_GETBMAPX failed %s\n",
+			strerror(errno));
+		/* won't try share again */
+		g_share_len = 0ULL;
+		return;
+	}
+
+	if (mapx[0].bmv_entries == 0) {
+		/* shared blocks all became hole, won't try share again */
+		g_share_len = 0ULL;
+		return;
+	}
+
+	if (g_share_offset != 512 * mapx[1].bmv_offset) {
+		/* first shared block became hole, won't try share again */
+		g_share_len = 0ULL;
+		return;
+	}
+
+	/* we check up to only the first NR_GET_EXT - 1 extents */
+	for (idx = 1; idx <= mapx[0].bmv_entries; idx++) {
+		if (mapx[idx].bmv_oflags & BMV_OF_SHARED) {
+			/* some blocks still shared, done */
+			return;
+		}
+	}
+
+	/*
+	 * The previously shared blocks are no longer shared, re-share.
+	 * deallocate the blocks in scrath file first
+	 */
+	err = fallocate(scratch_fd,
+		FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE,
+		OFFSET_1PB + g_share_offset, g_share_len);
+	if (err != 0) {
+		fprintf(stderr, "punch hole failed %s\n",
+			strerror(errno));
+		g_share_len = 0;
+		return;
+	}
+
+	new_share_len = 512 * mapx[1].bmv_length;
+	if (new_share_len > SHARE_MAX_SIZE)
+		new_share_len = SHARE_MAX_SIZE;
+
+	clone.src_fd = defrag_fd;
+	/* keep starting offset unchanged */
+	clone.src_offset = g_share_offset;
+	clone.src_length = new_share_len;
+	clone.dest_offset = OFFSET_1PB + clone.src_offset;
+
+	err = ioctl(scratch_fd, FICLONERANGE, &clone);
+	if (err) {
+		fprintf(stderr, "FICLONERANGE failed %s\n",
+			strerror(errno));
+		g_share_len = 0;
+		return;
+	}
+
+	g_share_len = new_share_len;
+ }
+
 /*
  * defragment a file
  * return 0 if successfully done, 1 otherwise
@@ -377,6 +526,12 @@ defrag_xfs_defrag(char *file_path) {
 
 	signal(SIGINT, defrag_sigint_handler);
 
+	/*
+	 * share the first extent to work around kernel consuming time
+	 * in xfs_reflink_try_clear_inode_flag()
+	 */
+	defrag_share_first_extent(defrag_fd, scratch_fd);
+
 	do {
 		struct timeval t_clone, t_unshare, t_punch_hole;
 		struct defrag_segment segment;
@@ -454,6 +609,15 @@ defrag_xfs_defrag(char *file_path) {
 		if (time_delta > max_unshare_us)
 			max_unshare_us = time_delta;
 
+		/*
+		 * if unshare used more than 1 second, time is very possibly
+		 * used in checking if the file is sharing extents now.
+		 * to avoid that happen again we re-share the blocks in front
+		 * to workaround that.
+		 */
+		if (time_delta > 1000000)
+			defrag_reshare_blocks_in_front(defrag_fd, scratch_fd);
+
 		/*
 		 * Punch out the original extents we shared to the
 		 * scratch file so they are returned to free space.
@@ -514,6 +678,8 @@ static void defrag_help(void)
 " -f free_space      -- specify shrethod of the XFS free space in MiB, when\n"
 "                       XFS free space is lower than that, shared segments \n"
 "                       are excluded from defragmentation, 1024 by default\n"
+" -n                 -- disable the \"share first extent\" featue, it's\n"
+"                       enabled by default to speed up\n"
 	));
 }
 
@@ -525,7 +691,7 @@ defrag_f(int argc, char **argv)
 	int	i;
 	int	c;
 
-	while ((c = getopt(argc, argv, "s:f:")) != EOF) {
+	while ((c = getopt(argc, argv, "s:f:n")) != EOF) {
 		switch(c) {
 		case 's':
 			g_segment_size_lmt = atoi(optarg) * 1024 * 1024 / 512;
@@ -539,6 +705,10 @@ defrag_f(int argc, char **argv)
 			g_limit_free_bytes = atol(optarg) * 1024 * 1024;
 			break;
 
+		case 'n':
+			g_enable_first_ext_share = false;
+			break;
+
 		default:
 			command_usage(&defrag_cmd);
 			return 1;
@@ -556,7 +726,7 @@ void defrag_init(void)
 	defrag_cmd.cfunc	= defrag_f;
 	defrag_cmd.argmin	= 0;
 	defrag_cmd.argmax	= 4;
-	defrag_cmd.args		= "[-s segment_size] [-f free_space]";
+	defrag_cmd.args		= "[-s segment_size] [-f free_space] [-n]";
 	defrag_cmd.flags	= CMD_FLAG_ONESHOT;
 	defrag_cmd.oneline	= _("Defragment XFS files");
 	defrag_cmd.help		= defrag_help;
Signature · 1 line
2.39.3 (Apple Git-146)
#34 lore raw

Re: [PATCH 6/9] spaceman/defrag: workaround kernel xfs_reflink_try_clear_inode_flag()

Quoted 6 lines · Wengang Wang

xfs_reflink_try_clear_inode_flag() takes very long in case file has huge number of extents and none of the extents are shared.

workaround:
share the first real extent so that xfs_reflink_try_clear_inode_flag() returns quickly to save cpu times and speed up defrag significantly.

I wonder if a better solution would be to change xfs_reflink_unshare only to try to clear the reflink iflag if offset/len cover the entire file? It's a pity we can't set time budgets on fallocate requests.

--D

Quoted 236 lines
---
 spaceman/defrag.c | 174 +++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 172 insertions(+), 2 deletions(-)
spaceman/defrag.c +172 -2
index f8e6713c..b5c5b187 100644
@@ -327,6 +327,155 @@ defrag_fs_limit_hit(int fd)
 	return statfs_s.f_bsize * statfs_s.f_bavail < g_limit_free_bytes;
 }
 
+static bool g_enable_first_ext_share = true;
+
+static int
+defrag_get_first_real_ext(int fd, struct getbmapx *mapx)
+{
+	int			err;
+
+	while (1) {
+		err = defrag_get_next_extent(fd, mapx);
+		if (err)
+			break;
+
+		defrag_move_next_extent();
+		if (!(mapx->bmv_oflags & BMV_OF_PREALLOC))
+			break;
+	}
+	return err;
+}
+
+static __u64 g_share_offset = -1ULL;
+static __u64 g_share_len = 0ULL;
+#define SHARE_MAX_SIZE 32768  /* 32KiB */
+
+/* share the first real extent with scrach */
+static void
+defrag_share_first_extent(int defrag_fd, int scratch_fd)
+{
+#define OFFSET_1PB 0x4000000000000LL
+	struct file_clone_range clone;
+	struct getbmapx mapx;
+	int	err;
+
+	if (g_enable_first_ext_share == false)
+		return;
+
+	err = defrag_get_first_real_ext(defrag_fd, &mapx);
+	if (err)
+		return;
+
+	clone.src_fd = defrag_fd;
+	clone.src_offset = mapx.bmv_offset * 512;
+	clone.src_length = mapx.bmv_length * 512;
+	/* shares at most SHARE_MAX_SIZE length */
+	if (clone.src_length > SHARE_MAX_SIZE)
+		clone.src_length = SHARE_MAX_SIZE;
+	clone.dest_offset = OFFSET_1PB + clone.src_offset;
+	/* if the first is extent is reaching the EoF, no need to share */
+	if (clone.src_offset + clone.src_length >= g_defrag_file_size)
+		return;
+	err = ioctl(scratch_fd, FICLONERANGE, &clone);
+	if (err != 0) {
+		fprintf(stderr, "cloning first extent failed: %s\n",
+			strerror(errno));
+		return;
+	}
+
+	/* safe the offset and length for re-share */
+	g_share_offset = clone.src_offset;
+	g_share_len = clone.src_length;
+}
+
+/* re-share the blocks we shared previous if then are no longer shared */
+static void
+defrag_reshare_blocks_in_front(int defrag_fd, int scratch_fd)
+{
+#define NR_GET_EXT 9
+	struct getbmapx mapx[NR_GET_EXT];
+	struct file_clone_range clone;
+	__u64	new_share_len;
+	int	idx, err;
+
+	if (g_enable_first_ext_share == false)
+		return;
+
+	if (g_share_len == 0ULL)
+		return;
+
+	/*
+	 * check if previous shareing still exist
+	 * we are done if (partially) so.
+	 */
+	mapx[0].bmv_offset = g_share_offset;
+	mapx[0].bmv_length = g_share_len;
+	mapx[0].bmv_count = NR_GET_EXT;
+	mapx[0].bmv_iflags = BMV_IF_NO_HOLES | BMV_IF_PREALLOC;
+	err = ioctl(defrag_fd, XFS_IOC_GETBMAPX, mapx);
+	if (err) {
+		fprintf(stderr, "XFS_IOC_GETBMAPX failed %s\n",
+			strerror(errno));
+		/* won't try share again */
+		g_share_len = 0ULL;
+		return;
+	}
+
+	if (mapx[0].bmv_entries == 0) {
+		/* shared blocks all became hole, won't try share again */
+		g_share_len = 0ULL;
+		return;
+	}
+
+	if (g_share_offset != 512 * mapx[1].bmv_offset) {
+		/* first shared block became hole, won't try share again */
+		g_share_len = 0ULL;
+		return;
+	}
+
+	/* we check up to only the first NR_GET_EXT - 1 extents */
+	for (idx = 1; idx <= mapx[0].bmv_entries; idx++) {
+		if (mapx[idx].bmv_oflags & BMV_OF_SHARED) {
+			/* some blocks still shared, done */
+			return;
+		}
+	}
+
+	/*
+	 * The previously shared blocks are no longer shared, re-share.
+	 * deallocate the blocks in scrath file first
+	 */
+	err = fallocate(scratch_fd,
+		FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE,
+		OFFSET_1PB + g_share_offset, g_share_len);
+	if (err != 0) {
+		fprintf(stderr, "punch hole failed %s\n",
+			strerror(errno));
+		g_share_len = 0;
+		return;
+	}
+
+	new_share_len = 512 * mapx[1].bmv_length;
+	if (new_share_len > SHARE_MAX_SIZE)
+		new_share_len = SHARE_MAX_SIZE;
+
+	clone.src_fd = defrag_fd;
+	/* keep starting offset unchanged */
+	clone.src_offset = g_share_offset;
+	clone.src_length = new_share_len;
+	clone.dest_offset = OFFSET_1PB + clone.src_offset;
+
+	err = ioctl(scratch_fd, FICLONERANGE, &clone);
+	if (err) {
+		fprintf(stderr, "FICLONERANGE failed %s\n",
+			strerror(errno));
+		g_share_len = 0;
+		return;
+	}
+
+	g_share_len = new_share_len;
+ }
+
 /*
  * defragment a file
  * return 0 if successfully done, 1 otherwise
@@ -377,6 +526,12 @@ defrag_xfs_defrag(char *file_path) {
 
 	signal(SIGINT, defrag_sigint_handler);
 
+	/*
+	 * share the first extent to work around kernel consuming time
+	 * in xfs_reflink_try_clear_inode_flag()
+	 */
+	defrag_share_first_extent(defrag_fd, scratch_fd);
+
 	do {
 		struct timeval t_clone, t_unshare, t_punch_hole;
 		struct defrag_segment segment;
@@ -454,6 +609,15 @@ defrag_xfs_defrag(char *file_path) {
 		if (time_delta > max_unshare_us)
 			max_unshare_us = time_delta;
 
+		/*
+		 * if unshare used more than 1 second, time is very possibly
+		 * used in checking if the file is sharing extents now.
+		 * to avoid that happen again we re-share the blocks in front
+		 * to workaround that.
+		 */
+		if (time_delta > 1000000)
+			defrag_reshare_blocks_in_front(defrag_fd, scratch_fd);
+
 		/*
 		 * Punch out the original extents we shared to the
 		 * scratch file so they are returned to free space.
@@ -514,6 +678,8 @@ static void defrag_help(void)
 " -f free_space      -- specify shrethod of the XFS free space in MiB, when\n"
 "                       XFS free space is lower than that, shared segments \n"
 "                       are excluded from defragmentation, 1024 by default\n"
+" -n                 -- disable the \"share first extent\" featue, it's\n"
+"                       enabled by default to speed up\n"
 	));
 }
 
@@ -525,7 +691,7 @@ defrag_f(int argc, char **argv)
 	int	i;
 	int	c;
 
-	while ((c = getopt(argc, argv, "s:f:")) != EOF) {
+	while ((c = getopt(argc, argv, "s:f:n")) != EOF) {
 		switch(c) {
 		case 's':
 			g_segment_size_lmt = atoi(optarg) * 1024 * 1024 / 512;
@@ -539,6 +705,10 @@ defrag_f(int argc, char **argv)
 			g_limit_free_bytes = atol(optarg) * 1024 * 1024;
 			break;
 
+		case 'n':
+			g_enable_first_ext_share = false;
+			break;
+
 		default:
 			command_usage(&defrag_cmd);
 			return 1;
@@ -556,7 +726,7 @@ void defrag_init(void)
 	defrag_cmd.cfunc	= defrag_f;
 	defrag_cmd.argmin	= 0;
 	defrag_cmd.argmax	= 4;
-	defrag_cmd.args		= "[-s segment_size] [-f free_space]";
+	defrag_cmd.args		= "[-s segment_size] [-f free_space] [-n]";
 	defrag_cmd.flags	= CMD_FLAG_ONESHOT;
 	defrag_cmd.oneline	= _("Defragment XFS files");
 	defrag_cmd.help		= defrag_help;
Signature · 1 line
2.39.3 (Apple Git-146)
#35 lore raw

Re: [PATCH 6/9] spaceman/defrag: workaround kernel xfs_reflink_try_clear_inode_flag()

Quoted 8 lines · Darrick J. Wong
Quoted 3 lines · Wengang Wang

xfs_reflink_try_clear_inode_flag() takes very long in case file has huge number of extents and none of the extents are shared.

Quoted 7 lines
Quoted 3 lines

workaround:
share the first real extent so that xfs_reflink_try_clear_inode_flag() returns quickly to save cpu times and speed up defrag significantly.

I wonder if a better solution would be to change xfs_reflink_unshare only to try to clear the reflink iflag if offset/len cover the entire file? It's a pity we can't set time budgets on fallocate requests.

Yep.
Anyway the change, if there will be, would be in kernel. We can use -n option to disable this workaround in defrag.

Thanks,
Wengang

Quoted 239 lines

--D

Quoted 236 lines

---
spaceman/defrag.c | 174 +++++++++++++++++++++++++++++++++++++++++++++-

1 file changed, 172 insertions(+), 2 deletions(-)
spaceman/defrag.c +164 -0
index f8e6713c..b5c5b187 100644
@@ -327,6 +327,155 @@ defrag_fs_limit_hit(int fd)
 return statfs_s.f_bsize * statfs_s.f_bavail < g_limit_free_bytes;
 }
 
+static bool g_enable_first_ext_share = true;
+
+static int
+defrag_get_first_real_ext(int fd, struct getbmapx *mapx)
+{
+ int err;
+
+ while (1) {
+ err = defrag_get_next_extent(fd, mapx);
+ if (err)
+ break;
+
+ defrag_move_next_extent();
+ if (!(mapx->bmv_oflags & BMV_OF_PREALLOC))
+ break;
+ }
+ return err;
+}
+
+static __u64 g_share_offset = -1ULL;
+static __u64 g_share_len = 0ULL;
+#define SHARE_MAX_SIZE 32768  /* 32KiB */
+
+/* share the first real extent with scrach */
+static void
+defrag_share_first_extent(int defrag_fd, int scratch_fd)
+{
+#define OFFSET_1PB 0x4000000000000LL
+ struct file_clone_range clone;
+ struct getbmapx mapx;
+ int err;
+
+ if (g_enable_first_ext_share == false)
+ return;
+
+ err = defrag_get_first_real_ext(defrag_fd, &mapx);
+ if (err)
+ return;
+
+ clone.src_fd = defrag_fd;
+ clone.src_offset = mapx.bmv_offset * 512;
+ clone.src_length = mapx.bmv_length * 512;
+ /* shares at most SHARE_MAX_SIZE length */
+ if (clone.src_length > SHARE_MAX_SIZE)
+ clone.src_length = SHARE_MAX_SIZE;
+ clone.dest_offset = OFFSET_1PB + clone.src_offset;
+ /* if the first is extent is reaching the EoF, no need to share */
+ if (clone.src_offset + clone.src_length >= g_defrag_file_size)
+ return;
+ err = ioctl(scratch_fd, FICLONERANGE, &clone);
+ if (err != 0) {
+ fprintf(stderr, "cloning first extent failed: %s\n",
+ strerror(errno));
+ return;
+ }
+
+ /* safe the offset and length for re-share */
+ g_share_offset = clone.src_offset;
+ g_share_len = clone.src_length;
+}
+
+/* re-share the blocks we shared previous if then are no longer shared */
+static void
+defrag_reshare_blocks_in_front(int defrag_fd, int scratch_fd)
+{
+#define NR_GET_EXT 9
+ struct getbmapx mapx[NR_GET_EXT];
+ struct file_clone_range clone;
+ __u64 new_share_len;
+ int idx, err;
+
+ if (g_enable_first_ext_share == false)
+ return;
+
+ if (g_share_len == 0ULL)
+ return;
+
+ /*
+ * check if previous shareing still exist
+ * we are done if (partially) so.
+ */
+ mapx[0].bmv_offset = g_share_offset;
+ mapx[0].bmv_length = g_share_len;
+ mapx[0].bmv_count = NR_GET_EXT;
+ mapx[0].bmv_iflags = BMV_IF_NO_HOLES | BMV_IF_PREALLOC;
+ err = ioctl(defrag_fd, XFS_IOC_GETBMAPX, mapx);
+ if (err) {
+ fprintf(stderr, "XFS_IOC_GETBMAPX failed %s\n",
+ strerror(errno));
+ /* won't try share again */
+ g_share_len = 0ULL;
+ return;
+ }
+
+ if (mapx[0].bmv_entries == 0) {
+ /* shared blocks all became hole, won't try share again */
+ g_share_len = 0ULL;
+ return;
+ }
+
+ if (g_share_offset != 512 * mapx[1].bmv_offset) {
+ /* first shared block became hole, won't try share again */
+ g_share_len = 0ULL;
+ return;
+ }
+
+ /* we check up to only the first NR_GET_EXT - 1 extents */
+ for (idx = 1; idx <= mapx[0].bmv_entries; idx++) {
+ if (mapx[idx].bmv_oflags & BMV_OF_SHARED) {
+ /* some blocks still shared, done */
+ return;
+ }
+ }
+
+ /*
+ * The previously shared blocks are no longer shared, re-share.
+ * deallocate the blocks in scrath file first
+ */
+ err = fallocate(scratch_fd,
+ FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE,
+ OFFSET_1PB + g_share_offset, g_share_len);
+ if (err != 0) {
+ fprintf(stderr, "punch hole failed %s\n",
+ strerror(errno));
+ g_share_len = 0;
+ return;
+ }
+
+ new_share_len = 512 * mapx[1].bmv_length;
+ if (new_share_len > SHARE_MAX_SIZE)
+ new_share_len = SHARE_MAX_SIZE;
+
+ clone.src_fd = defrag_fd;
+ /* keep starting offset unchanged */
+ clone.src_offset = g_share_offset;
+ clone.src_length = new_share_len;
+ clone.dest_offset = OFFSET_1PB + clone.src_offset;
+
+ err = ioctl(scratch_fd, FICLONERANGE, &clone);
+ if (err) {
+ fprintf(stderr, "FICLONERANGE failed %s\n",
+ strerror(errno));
+ g_share_len = 0;
+ return;
+ }
+
+ g_share_len = new_share_len;
+ }
+
 /*
 * defragment a file
 * return 0 if successfully done, 1 otherwise
@@ -377,6 +526,12 @@ defrag_xfs_defrag(char *file_path) {
 
 signal(SIGINT, defrag_sigint_handler);
 
+ /*
+ * share the first extent to work around kernel consuming time
+ * in xfs_reflink_try_clear_inode_flag()
+ */
+ defrag_share_first_extent(defrag_fd, scratch_fd);
+
 do {
 struct timeval t_clone, t_unshare, t_punch_hole;
 struct defrag_segment segment;
@@ -454,6 +609,15 @@ defrag_xfs_defrag(char *file_path) {
 if (time_delta > max_unshare_us)
 max_unshare_us = time_delta;
 
+ /*
+ * if unshare used more than 1 second, time is very possibly
+ * used in checking if the file is sharing extents now.
+ * to avoid that happen again we re-share the blocks in front
+ * to workaround that.
+ */
+ if (time_delta > 1000000)
+ defrag_reshare_blocks_in_front(defrag_fd, scratch_fd);
+
 /*
 * Punch out the original extents we shared to the
 * scratch file so they are returned to free space.
@@ -514,6 +678,8 @@ static void defrag_help(void)

" -f free_space -- specify shrethod of the XFS free space in MiB, when\n"
" XFS free space is lower than that, shared segments \n"
" are excluded from defragmentation, 1024 by default\n"
+" -n -- disable the \"share first extent\" featue, it's\n"
+" enabled by default to speed up\n"
));
}

spaceman/defrag.c +1 -1
@@ -525,7 +691,7 @@ defrag_f(int argc, char **argv)
 int i;
 int c;
 
- while ((c = getopt(argc, argv, "s:f:")) != EOF) {
+ while ((c = getopt(argc, argv, "s:f:n")) != EOF) {
 switch(c) {

case 's':
g_segment_size_lmt = atoi(optarg) * 1024 * 1024 / 512;

spaceman/defrag.c +4 -0
@@ -539,6 +705,10 @@ defrag_f(int argc, char **argv)
 g_limit_free_bytes = atol(optarg) * 1024 * 1024;
 break;
 
+ case 'n':
+ g_enable_first_ext_share = false;
+ break;
+

default:
command_usage(&defrag_cmd);
return 1;

spaceman/defrag.c +1 -1
@@ -556,7 +726,7 @@ void defrag_init(void)
 defrag_cmd.cfunc = defrag_f;
 defrag_cmd.argmin = 0;
 defrag_cmd.argmax = 4;
- defrag_cmd.args = "[-s segment_size] [-f free_space]";
+ defrag_cmd.args = "[-s segment_size] [-f free_space] [-n]";
 defrag_cmd.flags = CMD_FLAG_ONESHOT;
 defrag_cmd.oneline = _("Defragment XFS files");
 defrag_cmd.help = defrag_help;
Signature · 1 line
2.39.3 (Apple Git-146)
#36 lore raw

Re: [PATCH 6/9] spaceman/defrag: workaround kernel xfs_reflink_try_clear_inode_flag()

In reply to Wengang Wang
Quoted 2 lines · Wengang Wang

xfs_reflink_try_clear_inode_flag() takes very long in case file has huge number of extents and none of the extents are shared.

Got a kernel profile showing how bad it is?

Quoted 4 lines

workaround:
share the first real extent so that xfs_reflink_try_clear_inode_flag() returns quickly to save cpu times and speed up defrag significantly.

That's nasty.

Let's fix the kernel code, not work around it in userspace.

I mean, it would be really easy to store if an extent is shared in the iext btree record for the extent. If we do an unshare operation, just do a single "find shared extents" pass on the extent tree and mark all the extents that are shared as shared. Then set a flag on the data fork saying it is tracking shared extents, and so when we share/unshare extents in that inode from then on, we set/clear that flag in the iext record. (i.e. it's an in-memory equivalent of the UNWRITTEN state flag).

Then after the first unshare, checking for nothing being shared is a walk of the iext btree over the given range, not a refcountbt walk. That should be much faster.

And we could make it even faster by adding a "shared extents" counter to the inode fork. i.e. the first scan that sets the flags also counts the shared extents, and we maintain that as we maintain the iin memory extent flags....

That makes the cost of xfs_reflink_try_clear_inode_flag() basically go to zero in these sorts of workloads. IMO, this is a much better solution to the problem than hacking around it in userspace...

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#37 lore raw

Re: [PATCH 6/9] spaceman/defrag: workaround kernel xfs_reflink_try_clear_inode_flag()

Quoted 9 lines · Dave Chinner
Quoted 2 lines · Wengang Wang

xfs_reflink_try_clear_inode_flag() takes very long in case file has huge number of extents and none of the extents are shared.

Got a kernel profile showing how bad it is?

It was more than 1.5 seconds (basing on 6.4 millions of extents) when I add debug code to measure it.

Quoted 23 lines
Quoted 5 lines

workaround:
share the first real extent so that xfs_reflink_try_clear_inode_flag() returns quickly to save cpu times and speed up defrag significantly.

That's nasty.

Let's fix the kernel code, not work around it in userspace.

I mean, it would be really easy to store if an extent is shared in the iext btree record for the extent. If we do an unshare operation, just do a single "find shared extents" pass on the extent tree and mark all the extents that are shared as shared. Then set a flag on the data fork saying it is tracking shared extents, and so when we share/unshare extents in that inode from then on, we set/clear that flag in the iext record. (i.e. it's an in-memory equivalent of the UNWRITTEN state flag).

Then after the first unshare, checking for nothing being shared is a walk of the iext btree over the given range, not a refcountbt walk. That should be much faster.

Quoted 9 lines

And we could make it even faster by adding a "shared extents" counter to the inode fork. i.e. the first scan that sets the flags also counts the shared extents, and we maintain that as we maintain the iin memory extent flags....

That makes the cost of xfs_reflink_try_clear_inode_flag() basically go to zero in these sorts of workloads. IMO, this is a much better solution to the problem than hacking around it in userspace...

Yes, fixing it in kernel is the best way to go.
Well, one consideration is that the customers don’t run on upstream kernel. They might run a much lower version. And some customers don’t want kernel upgrades if there are no security issues.
So can we have both?
1. Trying to fix kernel and
2. Keep the workaround in defrag usersapce?

Thanks,
Wengang

Quoted 1 line
#38 lore raw

Re: [PATCH 6/9] spaceman/defrag: workaround kernel xfs_reflink_try_clear_inode_flag()

In reply to Wengang Wang
Quoted 11 lines · Wengang Wang

xfs_reflink_try_clear_inode_flag() takes very long in case file has huge number of extents and none of the extents are shared.

workaround:
share the first real extent so that xfs_reflink_try_clear_inode_flag() returns quickly to save cpu times and speed up defrag significantly.

---
 spaceman/defrag.c | 174 +++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 172 insertions(+), 2 deletions(-)

I had some insight on this late last night. The source of the issue is that both the kernel and the defrag algorithm are walking forwards across the file. Hence as we get t higher offsets in the file during defrag which unshares shared ranges, we are moving the first shared range to be higher in the file.

Hence the act of unsharing the file in ascending offset order results in the ascending offset search for shared extents done by the kernel growing in time.

The solution to this is to make defrag work backwards through the file, so it leaves the low offset shared extents intact for the kernel to find until the defrag process unshares them. At which point the kernel will clear the reflink flag and the searching stops.

IOWs, we either need to change the kernel code to do reverse order shared extent searching, or change the defrag operation to work in reverse sequential order, and then the performance problem relating to unsharing having to determine if the file being defragged is still shared or not goes away.

That said, I still think that the fact the defrag is completely unsharing the source file is wrong. If we leave shared extents intact as we defrag the file, then this problem doesn't need solving at all because xfs_reflink_try_clear_inode_flag() will hit the same shared extent near the front of the file every time...

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#39 lore raw

PATCH 7/9 spaceman/defrag: sleeps between segments

In reply to Wengang Wang

Let user contol the time to sleep between segments (file unlocked) to balance defrag performance and file IO servicing time.

---
 spaceman/defrag.c | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)
spaceman/defrag.c +24 -2
index b5c5b187..415fe9c2 100644
@@ -311,6 +311,9 @@ void defrag_sigint_handler(int dummy)
  */
 static long	g_limit_free_bytes = 1024 * 1024 * 1024;
 
+/* sleep time in us between segments, overwritten by paramter */
+static int		g_idle_time = 250 * 1000;
+
 /*
  * check if the free space in the FS is less than the _limit_
  * return true if so, false otherwise
@@ -487,6 +490,7 @@ defrag_xfs_defrag(char *file_path) {
 	int	scratch_fd = -1, defrag_fd = -1;
 	char	tmp_file_path[PATH_MAX+1];
 	struct file_clone_range clone;
+	int	sleep_time_us = 0;
 	char	*defrag_dir;
 	struct fsxattr	fsx;
 	int	ret = 0;
@@ -574,6 +578,9 @@ defrag_xfs_defrag(char *file_path) {
 
 		/* checks for EoF and fix up clone */
 		stop = defrag_clone_eof(&clone);
+		if (sleep_time_us > 0)
+			usleep(sleep_time_us);
+
 		gettimeofday(&t_clone, NULL);
 		ret = ioctl(scratch_fd, FICLONERANGE, &clone);
 		if (ret != 0) {
@@ -587,6 +594,10 @@ defrag_xfs_defrag(char *file_path) {
 		if (time_delta > max_clone_us)
 			max_clone_us = time_delta;
 
+		/* sleeps if clone cost more than 500ms, slow FS */
+		if (time_delta >= 500000 && g_idle_time > 0)
+			usleep(g_idle_time);
+
 		/* for defrag stats */
 		nr_ext_defrag += segment.ds_nr;
 
@@ -641,6 +652,12 @@ defrag_xfs_defrag(char *file_path) {
 
 		if (stop || usedKilled)
 			break;
+
+		/*
+		 * no lock on target file when punching hole from scratch file,
+		 * so minus the time used for punching hole
+		 */
+		sleep_time_us = g_idle_time - time_delta;
 	} while (true);
 out:
 	if (scratch_fd != -1) {
@@ -678,6 +695,7 @@ static void defrag_help(void)
 " -f free_space      -- specify shrethod of the XFS free space in MiB, when\n"
 "                       XFS free space is lower than that, shared segments \n"
 "                       are excluded from defragmentation, 1024 by default\n"
+" -i idle_time       -- time in ms to be idle between segments, 250ms by default\n"
 " -n                 -- disable the \"share first extent\" featue, it's\n"
 "                       enabled by default to speed up\n"
 	));
@@ -691,7 +709,7 @@ defrag_f(int argc, char **argv)
 	int	i;
 	int	c;
 
-	while ((c = getopt(argc, argv, "s:f:n")) != EOF) {
+	while ((c = getopt(argc, argv, "s:f:ni")) != EOF) {
 		switch(c) {
 		case 's':
 			g_segment_size_lmt = atoi(optarg) * 1024 * 1024 / 512;
@@ -709,6 +727,10 @@ defrag_f(int argc, char **argv)
 			g_enable_first_ext_share = false;
 			break;
 
+		case 'i':
+			g_idle_time = atoi(optarg) * 1000;
+			break;
+
 		default:
 			command_usage(&defrag_cmd);
 			return 1;
@@ -726,7 +748,7 @@ void defrag_init(void)
 	defrag_cmd.cfunc	= defrag_f;
 	defrag_cmd.argmin	= 0;
 	defrag_cmd.argmax	= 4;
-	defrag_cmd.args		= "[-s segment_size] [-f free_space] [-n]";
+	defrag_cmd.args		= "[-s segment_size] [-f free_space] [-i idle_time] [-n]";
 	defrag_cmd.flags	= CMD_FLAG_ONESHOT;
 	defrag_cmd.oneline	= _("Defragment XFS files");
 	defrag_cmd.help		= defrag_help;
Signature · 1 line
2.39.3 (Apple Git-146)
#40 lore raw

Re: [PATCH 7/9] spaceman/defrag: sleeps between segments

Quoted 41 lines · Wengang Wang

Let user contol the time to sleep between segments (file unlocked) to balance defrag performance and file IO servicing time.

---
 spaceman/defrag.c | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)
spaceman/defrag.c +7 -0
index b5c5b187..415fe9c2 100644
@@ -311,6 +311,9 @@ void defrag_sigint_handler(int dummy)
  */
 static long	g_limit_free_bytes = 1024 * 1024 * 1024;
 
+/* sleep time in us between segments, overwritten by paramter */
+static int		g_idle_time = 250 * 1000;
+
 /*
  * check if the free space in the FS is less than the _limit_
  * return true if so, false otherwise
@@ -487,6 +490,7 @@ defrag_xfs_defrag(char *file_path) {
 	int	scratch_fd = -1, defrag_fd = -1;
 	char	tmp_file_path[PATH_MAX+1];
 	struct file_clone_range clone;
+	int	sleep_time_us = 0;
 	char	*defrag_dir;
 	struct fsxattr	fsx;
 	int	ret = 0;
@@ -574,6 +578,9 @@ defrag_xfs_defrag(char *file_path) {
 
 		/* checks for EoF and fix up clone */
 		stop = defrag_clone_eof(&clone);
+		if (sleep_time_us > 0)
+			usleep(sleep_time_us);
+
 		gettimeofday(&t_clone, NULL);
 		ret = ioctl(scratch_fd, FICLONERANGE, &clone);
 		if (ret != 0) {

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 597

Quoted 4 lines
spaceman/defrag.c +1 -0
@@ -587,6 +594,10 @@ defrag_xfs_defrag(char *file_path) {
 		if (time_delta > max_clone_us)
 			max_clone_us = time_delta;
 
+		/* sleeps if clone cost more than 500ms, slow FS */

Why half a second? I sense that what you're getting at is that you want to limit file io latency spikes in other programs by relaxing the defrag program, right? But the help screen doesn't say anything about "only if the clone lasts more than 500ms".

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 599

Quoted 2 lines
spaceman/defrag.c +2 -0
+		if (time_delta >= 500000 && g_idle_time > 0)
+			usleep(g_idle_time);

These days, I wonder if it makes more sense to provide a CPU utilization target and let the kernel figure out how much sleeping that is:

$ systemd-run -p 'CPUQuota=60%' xfs_spaceman -c 'defrag' /path/to/file

The tradeoff here is that we as application writers no longer have to implement these clunky sleeps ourselves, but then one has to turn on cpu accounting in systemd (if there even /is/ a systemd). Also I suppose we don't want this program getting throttled while it's holding a file lock.

--D

Quoted 35 lines
spaceman/defrag.c +9 -1
+
 		/* for defrag stats */
 		nr_ext_defrag += segment.ds_nr;
 
@@ -641,6 +652,12 @@ defrag_xfs_defrag(char *file_path) {
 
 		if (stop || usedKilled)
 			break;
+
+		/*
+		 * no lock on target file when punching hole from scratch file,
+		 * so minus the time used for punching hole
+		 */
+		sleep_time_us = g_idle_time - time_delta;
 	} while (true);
 out:
 	if (scratch_fd != -1) {
@@ -678,6 +695,7 @@ static void defrag_help(void)
 " -f free_space      -- specify shrethod of the XFS free space in MiB, when\n"
 "                       XFS free space is lower than that, shared segments \n"
 "                       are excluded from defragmentation, 1024 by default\n"
+" -i idle_time       -- time in ms to be idle between segments, 250ms by default\n"
 " -n                 -- disable the \"share first extent\" featue, it's\n"
 "                       enabled by default to speed up\n"
 	));
@@ -691,7 +709,7 @@ defrag_f(int argc, char **argv)
 	int	i;
 	int	c;
 
-	while ((c = getopt(argc, argv, "s:f:n")) != EOF) {
+	while ((c = getopt(argc, argv, "s:f:ni")) != EOF) {
 		switch(c) {
 		case 's':
 			g_segment_size_lmt = atoi(optarg) * 1024 * 1024 / 512;

spaceman/defrag.c defrag_f(int argc, char **argv) · line 731

Quoted 5 lines
spaceman/defrag.c +2 -0
@@ -709,6 +727,10 @@ defrag_f(int argc, char **argv)
 			g_enable_first_ext_share = false;
 			break;
 
+		case 'i':
+			g_idle_time = atoi(optarg) * 1000;

Should we complain if optarg is non-integer garbage? Or if g_idle_time is larger than 1s?

--D

Quoted 18 lines
spaceman/defrag.c +3 -1
+			break;
+
 		default:
 			command_usage(&defrag_cmd);
 			return 1;
@@ -726,7 +748,7 @@ void defrag_init(void)
 	defrag_cmd.cfunc	= defrag_f;
 	defrag_cmd.argmin	= 0;
 	defrag_cmd.argmax	= 4;
-	defrag_cmd.args		= "[-s segment_size] [-f free_space] [-n]";
+	defrag_cmd.args		= "[-s segment_size] [-f free_space] [-i idle_time] [-n]";
 	defrag_cmd.flags	= CMD_FLAG_ONESHOT;
 	defrag_cmd.oneline	= _("Defragment XFS files");
 	defrag_cmd.help		= defrag_help;
Signature · 1 line
2.39.3 (Apple Git-146)
#41 lore raw

Re: [PATCH 7/9] spaceman/defrag: sleeps between segments

Quoted 49 lines · Darrick J. Wong
Quoted 44 lines · Wengang Wang

Let user contol the time to sleep between segments (file unlocked) to balance defrag performance and file IO servicing time.

---
spaceman/defrag.c | 26 ++++++++++++++++++++++++--

1 file changed, 24 insertions(+), 2 deletions(-)
spaceman/defrag.c +7 -0
index b5c5b187..415fe9c2 100644
@@ -311,6 +311,9 @@ void defrag_sigint_handler(int dummy)
 */
 static long g_limit_free_bytes = 1024 * 1024 * 1024;
 
+/* sleep time in us between segments, overwritten by paramter */
+static int g_idle_time = 250 * 1000;
+
 /*
 * check if the free space in the FS is less than the _limit_
 * return true if so, false otherwise
@@ -487,6 +490,7 @@ defrag_xfs_defrag(char *file_path) {
 int scratch_fd = -1, defrag_fd = -1;
 char tmp_file_path[PATH_MAX+1];
 struct file_clone_range clone;
+ int sleep_time_us = 0;
 char *defrag_dir;
 struct fsxattr fsx;
 int ret = 0;
@@ -574,6 +578,9 @@ defrag_xfs_defrag(char *file_path) {
 
 /* checks for EoF and fix up clone */
 stop = defrag_clone_eof(&clone);
+ if (sleep_time_us > 0)
+ usleep(sleep_time_us);
+
 gettimeofday(&t_clone, NULL);
 ret = ioctl(scratch_fd, FICLONERANGE, &clone);
 if (ret != 0) {
@@ -587,6 +594,10 @@ defrag_xfs_defrag(char *file_path) {
 if (time_delta > max_clone_us)
 max_clone_us = time_delta;
 
Quoted 6 lines

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 597

Quoted 1 line
spaceman/defrag.c +1 -0
+ /* sleeps if clone cost more than 500ms, slow FS */

Why half a second? I sense that what you're getting at is that you want to limit file io latency spikes in other programs by relaxing the defrag program, right? But the help screen doesn't say anything about "only if the clone lasts more than 500ms".

This is an optional sleep for very slow FS with CLONE takes long. Actually, we don’t fall into this case in our local tests.

The main sleep is at above:

40 + if (sleep_time_us > 0)
41 + usleep(sleep_time_us);

Quoted 7 lines

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 599

Quoted 3 lines
spaceman/defrag.c +2 -0
+ if (time_delta >= 500000 && g_idle_time > 0)
+ usleep(g_idle_time);

These days, I wonder if it makes more sense to provide a CPU utilization target and let the kernel figure out how much sleeping that is:

Quoted 8 lines
$ systemd-run -p 'CPUQuota=60%' xfs_spaceman -c 'defrag' /path/to/file

The tradeoff here is that we as application writers no longer have to implement these clunky sleeps ourselves, but then one has to turn on cpu accounting in systemd (if there even /is/ a systemd). Also I suppose we don't want this program getting throttled while it's holding a file lock.

Yes, we hope less locking time as possible.

This slowness is mainly coming from slow disks. As we tested, when page cache
Is empty, CPU usages is only at about 6% on my VM (the real physical would be spindle disk).
It would be higher for NVMEs.
I’d like to provide user a way to make balance, say longer sleep time between segments to make IO serviced better.

Thanks,
Wengang

Quoted 64 lines

--D

Quoted 35 lines
spaceman/defrag.c +7 -0
+
 /* for defrag stats */
 nr_ext_defrag += segment.ds_nr;
 
@@ -641,6 +652,12 @@ defrag_xfs_defrag(char *file_path) {
 
 if (stop || usedKilled)
 break;
+
+ /*
+  * no lock on target file when punching hole from scratch file,
+  * so minus the time used for punching hole
+  */
+ sleep_time_us = g_idle_time - time_delta;
 } while (true);

out:
if (scratch_fd != -1) {

spaceman/defrag.c +0 -0
@@ -678,6 +695,7 @@ static void defrag_help(void)

" -f free_space -- specify shrethod of the XFS free space in MiB, when\n"
" XFS free space is lower than that, shared segments \n"
" are excluded from defragmentation, 1024 by default\n"
+" -i idle_time -- time in ms to be idle between segments, 250ms by default\n"
" -n -- disable the \"share first extent\" featue, it's\n"
" enabled by default to speed up\n"
));

spaceman/defrag.c +1 -1
@@ -691,7 +709,7 @@ defrag_f(int argc, char **argv)
 int i;
 int c;
 
- while ((c = getopt(argc, argv, "s:f:n")) != EOF) {
+ while ((c = getopt(argc, argv, "s:f:ni")) != EOF) {
 switch(c) {

case 's':
g_segment_size_lmt = atoi(optarg) * 1024 * 1024 / 512;

spaceman/defrag.c defrag_f(int argc, char **argv) · line 731

Quoted 5 lines
spaceman/defrag.c +2 -0
@@ -709,6 +727,10 @@ defrag_f(int argc, char **argv)
 g_enable_first_ext_share = false;
 break;
 
+ case 'i':
+ g_idle_time = atoi(optarg) * 1000;

Should we complain if optarg is non-integer garbage? Or if g_idle_time is larger than 1s?

--D

Quoted 16 lines
spaceman/defrag.c +2 -0
+ break;
+

default:
command_usage(&defrag_cmd);
return 1;

spaceman/defrag.c +1 -1
@@ -726,7 +748,7 @@ void defrag_init(void)
 defrag_cmd.cfunc = defrag_f;
 defrag_cmd.argmin = 0;
 defrag_cmd.argmax = 4;
- defrag_cmd.args = "[-s segment_size] [-f free_space] [-n]";
+ defrag_cmd.args = "[-s segment_size] [-f free_space] [-i idle_time] [-n]";
 defrag_cmd.flags = CMD_FLAG_ONESHOT;
 defrag_cmd.oneline = _("Defragment XFS files");
 defrag_cmd.help = defrag_help;
Signature · 1 line
2.39.3 (Apple Git-146)
#42 lore raw

Re: [PATCH 7/9] spaceman/defrag: sleeps between segments

In reply to Darrick J. Wong
Quoted 110 lines · Darrick J. Wong
Quoted 41 lines · Wengang Wang

Let user contol the time to sleep between segments (file unlocked) to balance defrag performance and file IO servicing time.

---
spaceman/defrag.c | 26 ++++++++++++++++++++++++--

1 file changed, 24 insertions(+), 2 deletions(-)
spaceman/defrag.c +7 -0
index b5c5b187..415fe9c2 100644
@@ -311,6 +311,9 @@ void defrag_sigint_handler(int dummy)
 */
 static long g_limit_free_bytes = 1024 * 1024 * 1024;
 
+/* sleep time in us between segments, overwritten by paramter */
+static int g_idle_time = 250 * 1000;
+
 /*
 * check if the free space in the FS is less than the _limit_
 * return true if so, false otherwise
@@ -487,6 +490,7 @@ defrag_xfs_defrag(char *file_path) {
 int scratch_fd = -1, defrag_fd = -1;
 char tmp_file_path[PATH_MAX+1];
 struct file_clone_range clone;
+ int sleep_time_us = 0;
 char *defrag_dir;
 struct fsxattr fsx;
 int ret = 0;
@@ -574,6 +578,9 @@ defrag_xfs_defrag(char *file_path) {
 
 /* checks for EoF and fix up clone */
 stop = defrag_clone_eof(&clone);
+ if (sleep_time_us > 0)
+ usleep(sleep_time_us);
+
 gettimeofday(&t_clone, NULL);
 ret = ioctl(scratch_fd, FICLONERANGE, &clone);
 if (ret != 0) {

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 597

Quoted 4 lines
spaceman/defrag.c +1 -0
@@ -587,6 +594,10 @@ defrag_xfs_defrag(char *file_path) {
 if (time_delta > max_clone_us)
 max_clone_us = time_delta;
 
+ /* sleeps if clone cost more than 500ms, slow FS */

Why half a second? I sense that what you're getting at is that you want to limit file io latency spikes in other programs by relaxing the defrag program, right? But the help screen doesn't say anything about "only if the clone lasts more than 500ms".

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 599

Quoted 2 lines
spaceman/defrag.c +2 -0
+ if (time_delta >= 500000 && g_idle_time > 0)
+ usleep(g_idle_time);

These days, I wonder if it makes more sense to provide a CPU utilization target and let the kernel figure out how much sleeping that is:

$ systemd-run -p 'CPUQuota=60%' xfs_spaceman -c 'defrag' /path/to/file

The tradeoff here is that we as application writers no longer have to implement these clunky sleeps ourselves, but then one has to turn on cpu accounting in systemd (if there even /is/ a systemd). Also I suppose we don't want this program getting throttled while it's holding a file lock.

--D

Quoted 38 lines
spaceman/defrag.c +7 -0
+
 /* for defrag stats */
 nr_ext_defrag += segment.ds_nr;
 
@@ -641,6 +652,12 @@ defrag_xfs_defrag(char *file_path) {
 
 if (stop || usedKilled)
 break;
+
+ /*
+  * no lock on target file when punching hole from scratch file,
+  * so minus the time used for punching hole
+  */
+ sleep_time_us = g_idle_time - time_delta;
 } while (true);

out:
if (scratch_fd != -1) {

spaceman/defrag.c +0 -0
@@ -678,6 +695,7 @@ static void defrag_help(void)

" -f free_space -- specify shrethod of the XFS free space in MiB, when\n"
" XFS free space is lower than that, shared segments \n"
" are excluded from defragmentation, 1024 by default\n"
+" -i idle_time -- time in ms to be idle between segments, 250ms by default\n"
" -n -- disable the \"share first extent\" featue, it's\n"
" enabled by default to speed up\n"
));

spaceman/defrag.c +1 -1
@@ -691,7 +709,7 @@ defrag_f(int argc, char **argv)
 int i;
 int c;
 
- while ((c = getopt(argc, argv, "s:f:n")) != EOF) {
+ while ((c = getopt(argc, argv, "s:f:ni")) != EOF) {
 switch(c) {

case 's':
g_segment_size_lmt = atoi(optarg) * 1024 * 1024 / 512;

spaceman/defrag.c +0 -0
@@ -709,6 +727,10 @@ defrag_f(int argc, char **argv)
 g_enable_first_ext_share = false;
 break;
 
Quoted 5 lines

spaceman/defrag.c defrag_f(int argc, char **argv) · line 731

Quoted 2 lines
spaceman/defrag.c +2 -0
+ case 'i':
+ g_idle_time = atoi(optarg) * 1000;

Should we complain if optarg is non-integer garbage? Or if g_idle_time is larger than 1s?

It’s users responsibility :D.

Thanks,
Wengang

Quoted 19 lines

--D

Quoted 16 lines
spaceman/defrag.c +2 -0
+ break;
+

default:
command_usage(&defrag_cmd);
return 1;

spaceman/defrag.c +1 -1
@@ -726,7 +748,7 @@ void defrag_init(void)
 defrag_cmd.cfunc = defrag_f;
 defrag_cmd.argmin = 0;
 defrag_cmd.argmax = 4;
- defrag_cmd.args = "[-s segment_size] [-f free_space] [-n]";
+ defrag_cmd.args = "[-s segment_size] [-f free_space] [-i idle_time] [-n]";
 defrag_cmd.flags = CMD_FLAG_ONESHOT;
 defrag_cmd.oneline = _("Defragment XFS files");
 defrag_cmd.help = defrag_help;
Signature · 1 line
2.39.3 (Apple Git-146)
#43 lore raw

PATCH 8/9 spaceman/defrag: readahead for better performance

In reply to Wengang Wang

Reading ahead take less lock on file compared to "unshare" the file via ioctl. Do readahead when defrag sleeps for better defrag performace and thus more file IO time.

---
 spaceman/defrag.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
spaceman/defrag.c +20 -1
index 415fe9c2..ab8508bb 100644
@@ -331,6 +331,18 @@ defrag_fs_limit_hit(int fd)
 }
 
 static bool g_enable_first_ext_share = true;
+static bool g_readahead = false;
+
+static void defrag_readahead(int defrag_fd, off64_t offset, size_t count)
+{
+	if (!g_readahead || g_idle_time <= 0)
+		return;
+
+	if (readahead(defrag_fd, offset, count) < 0) {
+		fprintf(stderr, "readahead failed: %s, errno=%d\n",
+			strerror(errno), errno);
+	}
+}
 
 static int
 defrag_get_first_real_ext(int fd, struct getbmapx *mapx)
@@ -578,6 +590,8 @@ defrag_xfs_defrag(char *file_path) {
 
 		/* checks for EoF and fix up clone */
 		stop = defrag_clone_eof(&clone);
+		defrag_readahead(defrag_fd, seg_off, seg_size);
+
 		if (sleep_time_us > 0)
 			usleep(sleep_time_us);
 
@@ -698,6 +712,7 @@ static void defrag_help(void)
 " -i idle_time       -- time in ms to be idle between segments, 250ms by default\n"
 " -n                 -- disable the \"share first extent\" featue, it's\n"
 "                       enabled by default to speed up\n"
+" -a                 -- do readahead to speed up defrag, disabled by default\n"
 	));
 }
 
@@ -709,7 +724,7 @@ defrag_f(int argc, char **argv)
 	int	i;
 	int	c;
 
-	while ((c = getopt(argc, argv, "s:f:ni")) != EOF) {
+	while ((c = getopt(argc, argv, "s:f:nia")) != EOF) {
 		switch(c) {
 		case 's':
 			g_segment_size_lmt = atoi(optarg) * 1024 * 1024 / 512;
@@ -731,6 +746,10 @@ defrag_f(int argc, char **argv)
 			g_idle_time = atoi(optarg) * 1000;
 			break;
 
+		case 'a':
+			g_readahead = true;
+			break;
+
 		default:
 			command_usage(&defrag_cmd);
 			return 1;
Signature · 1 line
2.39.3 (Apple Git-146)
#44 lore raw

Re: [PATCH 8/9] spaceman/defrag: readahead for better performance

Quoted 17 lines · Wengang Wang

Reading ahead take less lock on file compared to "unshare" the file via ioctl. Do readahead when defrag sleeps for better defrag performace and thus more file IO time.

---
 spaceman/defrag.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
spaceman/defrag.c +0 -0
index 415fe9c2..ab8508bb 100644
@@ -331,6 +331,18 @@ defrag_fs_limit_hit(int fd)
 }
 
 static bool g_enable_first_ext_share = true;

spaceman/defrag.c defrag_fs_limit_hit(int fd) · line 343

Quoted 10 lines
spaceman/defrag.c +10 -0
+static bool g_readahead = false;
+
+static void defrag_readahead(int defrag_fd, off64_t offset, size_t count)
+{
+	if (!g_readahead || g_idle_time <= 0)
+		return;
+
+	if (readahead(defrag_fd, offset, count) < 0) {
+		fprintf(stderr, "readahead failed: %s, errno=%d\n",
+			strerror(errno), errno);

Why is it worth reporting if readahead fails? Won't the unshare also fail? I'm also wondering why we wouldn't want readahead all the time?

--D

Quoted 46 lines
spaceman/defrag.c +10 -1
+	}
+}
 
 static int
 defrag_get_first_real_ext(int fd, struct getbmapx *mapx)
@@ -578,6 +590,8 @@ defrag_xfs_defrag(char *file_path) {
 
 		/* checks for EoF and fix up clone */
 		stop = defrag_clone_eof(&clone);
+		defrag_readahead(defrag_fd, seg_off, seg_size);
+
 		if (sleep_time_us > 0)
 			usleep(sleep_time_us);
 
@@ -698,6 +712,7 @@ static void defrag_help(void)
 " -i idle_time       -- time in ms to be idle between segments, 250ms by default\n"
 " -n                 -- disable the \"share first extent\" featue, it's\n"
 "                       enabled by default to speed up\n"
+" -a                 -- do readahead to speed up defrag, disabled by default\n"
 	));
 }
 
@@ -709,7 +724,7 @@ defrag_f(int argc, char **argv)
 	int	i;
 	int	c;
 
-	while ((c = getopt(argc, argv, "s:f:ni")) != EOF) {
+	while ((c = getopt(argc, argv, "s:f:nia")) != EOF) {
 		switch(c) {
 		case 's':
 			g_segment_size_lmt = atoi(optarg) * 1024 * 1024 / 512;
@@ -731,6 +746,10 @@ defrag_f(int argc, char **argv)
 			g_idle_time = atoi(optarg) * 1000;
 			break;
 
+		case 'a':
+			g_readahead = true;
+			break;
+
 		default:
 			command_usage(&defrag_cmd);
 			return 1;
Signature · 1 line
2.39.3 (Apple Git-146)
#45 lore raw

Re: [PATCH 8/9] spaceman/defrag: readahead for better performance

Quoted 21 lines · Darrick J. Wong
Quoted 16 lines · Wengang Wang

Reading ahead take less lock on file compared to "unshare" the file via ioctl. Do readahead when defrag sleeps for better defrag performace and thus more file IO time.

---
spaceman/defrag.c | 21 ++++++++++++++++++++-

1 file changed, 20 insertions(+), 1 deletion(-)
spaceman/defrag.c +0 -0
index 415fe9c2..ab8508bb 100644
@@ -331,6 +331,18 @@ defrag_fs_limit_hit(int fd)
 }
 
Quoted 14 lines

spaceman/defrag.c defrag_fs_limit_hit(int fd) · line 343

Quoted 11 lines
spaceman/defrag.c +10 -0
 static bool g_enable_first_ext_share = true;
+static bool g_readahead = false;
+
+static void defrag_readahead(int defrag_fd, off64_t offset, size_t count)
+{
+ if (!g_readahead || g_idle_time <= 0)
+ return;
+
+ if (readahead(defrag_fd, offset, count) < 0) {
+ fprintf(stderr, "readahead failed: %s, errno=%d\n",
+ strerror(errno), errno);

Why is it worth reporting if readahead fails? Won't the unshare also fail?

Yes, if readahead failed, later unshare should fail as I think.
I just want to capture error as soon as possible though readahead is not critical.

Quoted 1 line

I'm also wondering why we wouldn't want readahead all the time?

As per our tests, readahead on NVME doesn’t hehaved better. So I’d make it an option.

Thanks,
Wengang

Quoted 47 lines

--D

Quoted 44 lines
spaceman/defrag.c +2 -0
+ }
+}
 

static int
defrag_get_first_real_ext(int fd, struct getbmapx *mapx)

spaceman/defrag.c +2 -0
@@ -578,6 +590,8 @@ defrag_xfs_defrag(char *file_path) {
 
 /* checks for EoF and fix up clone */
 stop = defrag_clone_eof(&clone);
+ defrag_readahead(defrag_fd, seg_off, seg_size);
+
 if (sleep_time_us > 0)
 usleep(sleep_time_us);
 
@@ -698,6 +712,7 @@ static void defrag_help(void)

" -i idle_time -- time in ms to be idle between segments, 250ms by default\n"
" -n -- disable the \"share first extent\" featue, it's\n"
" enabled by default to speed up\n"
+" -a -- do readahead to speed up defrag, disabled by default\n"
));
}

spaceman/defrag.c +1 -1
@@ -709,7 +724,7 @@ defrag_f(int argc, char **argv)
 int i;
 int c;
 
- while ((c = getopt(argc, argv, "s:f:ni")) != EOF) {
+ while ((c = getopt(argc, argv, "s:f:nia")) != EOF) {
 switch(c) {

case 's':
g_segment_size_lmt = atoi(optarg) * 1024 * 1024 / 512;

spaceman/defrag.c +4 -0
@@ -731,6 +746,10 @@ defrag_f(int argc, char **argv)
 g_idle_time = atoi(optarg) * 1000;
 break;
 
+ case 'a':
+ g_readahead = true;
+ break;
+

default:
command_usage(&defrag_cmd);
return 1;

Signature · 1 line
2.39.3 (Apple Git-146)
#46 lore raw

Re: [PATCH 8/9] spaceman/defrag: readahead for better performance

In reply to Wengang Wang
Quoted 17 lines · Wengang Wang

Reading ahead take less lock on file compared to "unshare" the file via ioctl. Do readahead when defrag sleeps for better defrag performace and thus more file IO time.

---
 spaceman/defrag.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
spaceman/defrag.c +0 -0
index 415fe9c2..ab8508bb 100644
@@ -331,6 +331,18 @@ defrag_fs_limit_hit(int fd)
 }
 
 static bool g_enable_first_ext_share = true;

spaceman/defrag.c defrag_fs_limit_hit(int fd) · line 343

Quoted 10 lines
spaceman/defrag.c +10 -0
+static bool g_readahead = false;
+
+static void defrag_readahead(int defrag_fd, off64_t offset, size_t count)
+{
+	if (!g_readahead || g_idle_time <= 0)
+		return;
+
+	if (readahead(defrag_fd, offset, count) < 0) {
+		fprintf(stderr, "readahead failed: %s, errno=%d\n",
+			strerror(errno), errno);

This doesn't do what you think it does. readahead() only queues the first readahead chunk of the range given (a few pages at most). It does not cause readahead on the entire range, wait for page cache population, nor report IO errors that might have occurred during readahead.

There's almost no value to making this syscall, especially if the app is about to trigger a sequential read for the whole range. Readahead will occur naturally during that read operation (i.e. the UNSHARE copy), and the read will return IO errors unlike readahead().

If you want the page cache pre-populated before the unshare operation is done, then you need to use mmap() and madvise(MADV_POPULATE_READ). This will read the whole region into the page cache as if it was a sequential read, wait for it to complete and return any IO errors that might have occurred during the read.

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#47 lore raw

Re: [PATCH 8/9] spaceman/defrag: readahead for better performance

Quoted 30 lines · Dave Chinner
Quoted 25 lines · Wengang Wang

Reading ahead take less lock on file compared to "unshare" the file via ioctl. Do readahead when defrag sleeps for better defrag performace and thus more file IO time.

---
spaceman/defrag.c | 21 ++++++++++++++++++++-

1 file changed, 20 insertions(+), 1 deletion(-)
spaceman/defrag.c +8 -0
index 415fe9c2..ab8508bb 100644
@@ -331,6 +331,18 @@ defrag_fs_limit_hit(int fd)
 }
 
 static bool g_enable_first_ext_share = true;
+static bool g_readahead = false;
+
+static void defrag_readahead(int defrag_fd, off64_t offset, size_t count)
+{
+ if (!g_readahead || g_idle_time <= 0)
+ return;
+
+ if (readahead(defrag_fd, offset, count) < 0) {
Quoted 8 lines

spaceman/defrag.c defrag_fs_limit_hit(int fd) · line 343

Quoted 2 lines
spaceman/defrag.c +2 -0
+ fprintf(stderr, "readahead failed: %s, errno=%d\n",
+ strerror(errno), errno);

This doesn't do what you think it does. readahead() only queues the first readahead chunk of the range given (a few pages at most). It does not cause readahead on the entire range, wait for page cache population, nor report IO errors that might have occurred during readahead.

Is it a bug? As per the man page it should try to read _count_ bytes:

DESCRIPTION
readahead() initiates readahead on a file so that subsequent reads from that file will be satisfied from the cache, and not block on disk I/O (assuming the readahead was initiated early enough and that other activity on the system did not in the meantime flush pages from the cache).

The fd argument is a file descriptor identifying the file which is to be read. The offset argument specifies the starting point from which data is to be read and count specifies the number of bytes to be read. I/O is performed in whole pages, so that offset is effectively rounded down to a page boundary and bytes are read up to the next page boundary greater than or equal to (offset+count). reada‐ head() does not read beyond the end of the file. The file offset of the open file description referred to by fd is left unchanged.

Quoted 7 lines

There's almost no value to making this syscall, especially if the app is about to trigger a sequential read for the whole range. Readahead will occur naturally during that read operation (i.e. the UNSHARE copy), and the read will return IO errors unlike readahead().

Quoted 6 lines

If you want the page cache pre-populated before the unshare operation is done, then you need to use mmap() and madvise(MADV_POPULATE_READ). This will read the whole region into the page cache as if it was a sequential read, wait for it to complete and return any IO errors that might have occurred during the read.

As you know in the unshare path, fetching data from disk is done when IO is locked. (I am wondering if we can improve that.)
The main purpose of using readahead is that I want less (IO) lock time when fetching data from disk. Can we achieve that by using mmap and madvise()?

Thanks,
Wengang

#48 lore raw

Re: [PATCH 8/9] spaceman/defrag: readahead for better performance

Quoted 39 lines · Wengang Wang
Quoted 30 lines · Dave Chinner
Quoted 25 lines · Wengang Wang

Reading ahead take less lock on file compared to "unshare" the file via ioctl. Do readahead when defrag sleeps for better defrag performace and thus more file IO time.

---
spaceman/defrag.c | 21 ++++++++++++++++++++-

1 file changed, 20 insertions(+), 1 deletion(-)
spaceman/defrag.c +8 -0
index 415fe9c2..ab8508bb 100644
@@ -331,6 +331,18 @@ defrag_fs_limit_hit(int fd)
 }
 
 static bool g_enable_first_ext_share = true;
+static bool g_readahead = false;
+
+static void defrag_readahead(int defrag_fd, off64_t offset, size_t count)
+{
+ if (!g_readahead || g_idle_time <= 0)
+ return;
+
+ if (readahead(defrag_fd, offset, count) < 0) {
Quoted 8 lines

spaceman/defrag.c defrag_fs_limit_hit(int fd) · line 343

Quoted 2 lines
spaceman/defrag.c +2 -0
+ fprintf(stderr, "readahead failed: %s, errno=%d\n",
+ strerror(errno), errno);

This doesn't do what you think it does. readahead() only queues the first readahead chunk of the range given (a few pages at most). It does not cause readahead on the entire range, wait for page cache population, nor report IO errors that might have occurred during readahead.

Quoted 1 line

Is it a bug?

No.

Quoted 1 line

As per the man page it should try to read _count_ bytes:

No it doesn't. It says:

Quoted 3 lines

DESCRIPTION
readahead() initiates readahead on a file

         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

It says it -initiates- readahead. It doesn't mean it waits for readahead to complete or that it will readahead the whole range. It just starts readahead.

Quoted 13 lines
Quoted 6 lines

There's almost no value to making this syscall, especially if the app is about to trigger a sequential read for the whole range. Readahead will occur naturally during that read operation (i.e. the UNSHARE copy), and the read will return IO errors unlike readahead().

Quoted 6 lines

If you want the page cache pre-populated before the unshare operation is done, then you need to use mmap() and madvise(MADV_POPULATE_READ). This will read the whole region into the page cache as if it was a sequential read, wait for it to complete and return any IO errors that might have occurred during the read.

Quoted 2 lines

As you know in the unshare path, fetching data from disk is done when IO is locked. (I am wondering if we can improve that.)

Christoph pointed that out and some potential fixes back in the original discussion:

https://lore.kernel.org/linux-xfs/[email protected]/

Quoted 2 lines

The main purpose of using readahead is that I want less (IO) lock time when fetching data from disk. Can we achieve that by using mmap and madvise()?

Maybe, but you're still adding complexity to userspace as a work around for a kernel issue we should be fixing.

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#49 lore raw

Re: [PATCH 8/9] spaceman/defrag: readahead for better performance

Quoted 53 lines · Dave Chinner
Quoted 39 lines · Wengang Wang
Quoted 30 lines · Dave Chinner
Quoted 25 lines · Wengang Wang

Reading ahead take less lock on file compared to "unshare" the file via ioctl. Do readahead when defrag sleeps for better defrag performace and thus more file IO time.

---
spaceman/defrag.c | 21 ++++++++++++++++++++-

1 file changed, 20 insertions(+), 1 deletion(-)
spaceman/defrag.c +8 -0
index 415fe9c2..ab8508bb 100644
@@ -331,6 +331,18 @@ defrag_fs_limit_hit(int fd)
 }
 
 static bool g_enable_first_ext_share = true;
+static bool g_readahead = false;
+
+static void defrag_readahead(int defrag_fd, off64_t offset, size_t count)
+{
+ if (!g_readahead || g_idle_time <= 0)
+ return;
+
+ if (readahead(defrag_fd, offset, count) < 0) {
Quoted 8 lines

spaceman/defrag.c defrag_fs_limit_hit(int fd) · line 343

Quoted 2 lines
spaceman/defrag.c +2 -0
+ fprintf(stderr, "readahead failed: %s, errno=%d\n",
+ strerror(errno), errno);

This doesn't do what you think it does. readahead() only queues the first readahead chunk of the range given (a few pages at most). It does not cause readahead on the entire range, wait for page cache population, nor report IO errors that might have occurred during readahead.

Quoted 1 line

Is it a bug?

No.

Quoted 1 line

As per the man page it should try to read _count_ bytes:

No it doesn't. It says:

Quoted 1 line
Quoted 8 lines
Quoted 2 lines

DESCRIPTION
readahead() initiates readahead on a file

        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

It says it -initiates- readahead. It doesn't mean it waits for readahead to complete or that it will readahead the whole range. It just starts readahead.

I know it’s asynchronous operation. But when given enough time, it should complete.

# without considering your idea of completing the defrag as fast as possible
As we tested with 16MiB segment size limit, we were seeing the unshare takes about
350 ms including the disk data reading. And we are using 250 ms default sleep/idle time.
In my idea, during the 250 ms sleep time, a lot of block reads should be done.
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

There's almost no value to making this syscall, especially if the app is about to trigger a sequential read for the whole range. Readahead will occur naturally during that read operation (i.e. the UNSHARE copy), and the read will return IO errors unlike readahead().

If you want the page cache pre-populated before the unshare operation is done, then you need to use mmap() and madvise(MADV_POPULATE_READ). This will read the whole region into the page cache as if it was a sequential read, wait for it to complete and return any IO errors that might have occurred during the read.

Quoted 7 lines
Quoted 2 lines

As you know in the unshare path, fetching data from disk is done when IO is locked. (I am wondering if we can improve that.)

Christoph pointed that out and some potential fixes back in the original discussion:

https://lore.kernel.org/linux-xfs/[email protected]/

Yes, I read that. Thanks Christoph for that.

Quoted 7 lines
Quoted 3 lines

The main purpose of using readahead is that I want less (IO) lock time when fetching data from disk. Can we achieve that by using mmap and madvise()?

Maybe, but you're still adding complexity to userspace as a work around for a kernel issue we should be fixing.

Yes, true. But in case of kernel has no the related fixes and we have to use the defrag, working around is the way go, right.

Thanks,
Wengang

#50 lore raw

PATCH 9/9 spaceman/defrag: warn on extsize

In reply to Wengang Wang

According to current kernel implemenation, non-zero extsize might affect the result of defragmentation.
Just print a warning on that if non-zero extsize is set on file.

---
 spaceman/defrag.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
spaceman/defrag.c +12 -0
index ab8508bb..b6b89dd9 100644
@@ -526,6 +526,18 @@ defrag_xfs_defrag(char *file_path) {
 		goto out;
 	}
 
+       if (ioctl(defrag_fd, FS_IOC_FSGETXATTR, &fsx) < 0) {
+               fprintf(stderr, "FSGETXATTR failed %s\n",
+                       strerror(errno));
+               ret = 1;
+               goto out;
+       }
+
+       if (fsx.fsx_extsize != 0)
+               fprintf(stderr, "%s has extsize set %d. That might affect defrag "
+                       "according to kernel implementation\n",
+                       file_path, fsx.fsx_extsize);
+
 	clone.src_fd = defrag_fd;
 
 	defrag_dir = dirname(file_path);
Signature · 1 line
2.39.3 (Apple Git-146)
#51 lore raw

Re: [PATCH 9/9] spaceman/defrag: warn on extsize

Quoted 3 lines · Wengang Wang

According to current kernel implemenation, non-zero extsize might affect the result of defragmentation.
Just print a warning on that if non-zero extsize is set on file.

I'm not sure what's the point of warning vaguely about extent size hints? I'd have thought that would help reduce the number of extents; is that not the case?

Quoted 10 lines
---
 spaceman/defrag.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
spaceman/defrag.c +0 -0
index ab8508bb..b6b89dd9 100644

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 531

Quoted 6 lines
spaceman/defrag.c +3 -0
@@ -526,6 +526,18 @@ defrag_xfs_defrag(char *file_path) {
 		goto out;
 	}
 
+       if (ioctl(defrag_fd, FS_IOC_FSGETXATTR, &fsx) < 0) {
+               fprintf(stderr, "FSGETXATTR failed %s\n",
+                       strerror(errno));

Also we usually indent continuations by two tabs (not one) so that the continuation is more obvious:

		fprintf(stderr, "FSGETXATTR failed %s\n",
				strerror(errno));

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 538

Quoted 7 lines
spaceman/defrag.c +7 -0
+               ret = 1;
+               goto out;
+       }
+
+       if (fsx.fsx_extsize != 0)
+               fprintf(stderr, "%s has extsize set %d. That might affect defrag "
+                       "according to kernel implementation\n",

Format strings in userspace printf calls should be wrapped so that gettext can provide translated versions:

	fprintf(stderr, _("%s has extsize...\n"), file_path...);

(I know, xfsprogs isn't as consistent as it probably ought to be...)

--D

Quoted 9 lines
spaceman/defrag.c +2 -0
+                       file_path, fsx.fsx_extsize);
+
 	clone.src_fd = defrag_fd;
 
 	defrag_dir = dirname(file_path);
Signature · 1 line
2.39.3 (Apple Git-146)
#52 lore raw

Re: [PATCH 9/9] spaceman/defrag: warn on extsize

Quoted 12 lines · Darrick J. Wong
Quoted 3 lines · Wengang Wang

According to current kernel implemenation, non-zero extsize might affect the result of defragmentation.
Just print a warning on that if non-zero extsize is set on file.

I'm not sure what's the point of warning vaguely about extent size hints? I'd have thought that would help reduce the number of extents; is that not the case?

Not exactly.

Same 1G file with about 54K extents,

The one with 16K extsize, after defrag, it’s extents drops to 13K. And the one with 0 extsize, after defrag, it’s extents dropped to 22.

Above is tested with UEK6 (5.4 kernel). I can get the numbers on mainline if you want.

Quoted 21 lines
Quoted 11 lines

---
spaceman/defrag.c | 12 ++++++++++++

1 file changed, 12 insertions(+)
spaceman/defrag.c +0 -0
index ab8508bb..b6b89dd9 100644

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 531

Quoted 6 lines
spaceman/defrag.c +3 -0
@@ -526,6 +526,18 @@ defrag_xfs_defrag(char *file_path) {
 goto out;
 }
 
+       if (ioctl(defrag_fd, FS_IOC_FSGETXATTR, &fsx) < 0) {
+               fprintf(stderr, "FSGETXATTR failed %s\n",
+                       strerror(errno));

Also we usually indent continuations by two tabs (not one) so that the continuation is more obvious:

Quoted 2 lines

fprintf(stderr, "FSGETXATTR failed %s\n",
strerror(errno));

OK.

Quoted 9 lines

spaceman/defrag.c defrag_xfs_defrag(char *file_path) { · line 538

Quoted 8 lines
spaceman/defrag.c +7 -0
+               ret = 1;
+               goto out;
+       }
+
+       if (fsx.fsx_extsize != 0)
+               fprintf(stderr, "%s has extsize set %d. That might affect defrag "
+                       "according to kernel implementation\n",
Quoted 7 lines

Format strings in userspace printf calls should be wrapped so that gettext can provide translated versions:

fprintf(stderr, _("%s has extsize...\n"), file_path...);

(I know, xfsprogs isn't as consistent as it probably ought to be...)

OK.

Thanks,
Wengang

Quoted 9 lines

--D

Quoted 7 lines
spaceman/defrag.c +2 -0
+                       file_path, fsx.fsx_extsize);
+
 clone.src_fd = defrag_fd;
 
 defrag_dir = dirname(file_path);
Signature · 1 line
2.39.3 (Apple Git-146)
#53 lore raw

Re: [PATCH 9/9] spaceman/defrag: warn on extsize

Quoted 13 lines · Wengang Wang
Quoted 12 lines · Darrick J. Wong
Quoted 3 lines · Wengang Wang

According to current kernel implemenation, non-zero extsize might affect the result of defragmentation.
Just print a warning on that if non-zero extsize is set on file.

I'm not sure what's the point of warning vaguely about extent size hints? I'd have thought that would help reduce the number of extents; is that not the case?

Quoted 6 lines

Not exactly.

Same 1G file with about 54K extents,

The one with 16K extsize, after defrag, it’s extents drops to 13K. And the one with 0 extsize, after defrag, it’s extents dropped to 22.

extsize should not affect file contiguity like this at all. Are you measuring fragmentation correctly? i.e. a contiguous region from an larger extsize allocation that results in a bmap/fiemap output of three extents in a unwritten/written/unwritten is not fragmentation.

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#54 lore raw

Re: [PATCH 9/9] spaceman/defrag: warn on extsize

Quoted 22 lines · Dave Chinner
Quoted 17 lines · Wengang Wang
Quoted 12 lines · Darrick J. Wong
Quoted 3 lines · Wengang Wang

According to current kernel implemenation, non-zero extsize might affect the result of defragmentation.
Just print a warning on that if non-zero extsize is set on file.

I'm not sure what's the point of warning vaguely about extent size hints? I'd have thought that would help reduce the number of extents; is that not the case?

Not exactly.

Same 1G file with about 54K extents,

Quoted 7 lines
Quoted 2 lines

The one with 16K extsize, after defrag, it’s extents drops to 13K. And the one with 0 extsize, after defrag, it’s extents dropped to 22.

extsize should not affect file contiguity like this at all. Are you measuring fragmentation correctly? i.e. a contiguous region from an larger extsize allocation that results in a bmap/fiemap output of three extents in a unwritten/written/unwritten is not fragmentation.

I was using FS_IOC_FSGETXATTR to get the number of extents (fsx.fsx_nextents).
So if kernel doesn’t lie, I got it correctly. There was no unwritten extents in the files to defrag.

(As I mentioned somewhere else), though extsize is mainly used to align the number of blocks, it breaks delayed-allocations. In the unshare path, there are N allocations performed for the N extents respectively in the segment to be defragmented.

Thanks,
Wengang

#55 lore raw

Re: [PATCH 9/9] spaceman/defrag: warn on extsize

Quoted 25 lines · Wengang Wang
Quoted 25 lines · Dave Chinner
Quoted 19 lines · Wengang Wang
Quoted 12 lines · Darrick J. Wong
Quoted 3 lines · Wengang Wang

According to current kernel implemenation, non-zero extsize might affect the result of defragmentation.
Just print a warning on that if non-zero extsize is set on file.

I'm not sure what's the point of warning vaguely about extent size hints? I'd have thought that would help reduce the number of extents; is that not the case?

Not exactly.

Same 1G file with about 54K extents,

The one with 16K extsize, after defrag, it’s extents drops to 13K. And the one with 0 extsize, after defrag, it’s extents dropped to 22.

Quoted 7 lines
Quoted 4 lines

extsize should not affect file contiguity like this at all. Are you measuring fragmentation correctly? i.e. a contiguous region from an larger extsize allocation that results in a bmap/fiemap output of three extents in a unwritten/written/unwritten is not fragmentation.

I was using FS_IOC_FSGETXATTR to get the number of extents (fsx.fsx_nextents).
So if kernel doesn’t lie, I got it correctly. There was no unwritten extents in the files to defrag.

The kernel is not lying, and you've misunderstood what the kernel is reporting as an extent. The kernel reports the count of -individual extent records- it maintains, not the count of contiguous regions it is mapping. Have a look at the implementation of fsx.fsx_nextents in xfs_fill_fsxattr():

	if (ifp && !xfs_need_iread_extents(ifp))
                fa->fsx_nextents = xfs_iext_count(ifp);
        else
                fa->fsx_nextents = xfs_ifork_nextents(ifp);

We have:

inline xfs_extnum_t xfs_iext_count(struct xfs_ifork *ifp)

{
        return ifp->if_bytes / sizeof(struct xfs_iext_rec);
}

Which is the number of in-memory extents for the inode fork. Not only does that include unwritten extent records, it includes delayed allocation extents that don't even exist on disk.

And if we haven't read the extent list in from disk, we use:

static inline xfs_extnum_t xfs_ifork_nextents(struct xfs_ifork *ifp)

{
        if (!ifp)
                return 0;
        return ifp->if_nextents;
}

Which is a count of the on-disk extents for the inode fork which counts both written and unwritten extent records.

IOWs, both of these functions count unwritten extents as separate extents to written extents, even if they are contiguous. That means a single contiguous extent with an unwritten region in the middle of it:

	0	1	2	3
	+WWWWWWW+UUUUUUU+WWWWWWW+

Is reported as three extent records - {0,1,W}, {1,1,U}, {2,1,W} - and so fsx.fsx_nextents will report 3 extents despite the fact that file is *not* fragmented at all.

Hence interpretting fsx.fsx_nextents as a number that accurately reflects actual extent fragmentation levels is incorrect. If you have a sparse file or mixed written/unwritten regions, the extent count will be much higher than expected but it does not indicate that the file is fragmented at all.

Applications need to look at the actual extent map that is returned from FIEMAP to determine if there is significant fragmentation that can be addressed, not just the raw extent count.

Quoted 4 lines

(As I mentioned somewhere else), though extsize is mainly used to align the number of blocks, it breaks delayed-allocations. In the unshare path, there are N allocations performed for the N extents respectively in the segment to be defragmented.

That's largely irrelevant to the issue at hand. If there is sufficient free space in the filesystem, the allocator will first attempt and succeed at contiguous allocation. Hence the size of each allocation is irrelevant as they will be laid out contiguously given sufficient large contiguous free space.

Indeed, this is how allocation for direct IO works, and it doesn't have problems with fragmentation of files for single threaded sequential IO for the same reasons....

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#56 lore raw
In reply to Wengang Wang

[ Please keep documentation text to 80 columns. ]

[ Please run documentation through a spell checker - there are too many typos in this document to point them all out... ]

Quoted 2 lines · Wengang Wang

This patch set introduces defrag to xfs_spaceman command. It has the functionality and features below (also subject to be added to man page, so please review):

What's the use case for this?

Quoted 2 lines

defrag [-f free_space] [-i idle_time] [-s segment_size] [-n] [-a]
defrag defragments the specified XFS file online non-exclusively. The target XFS

What's "non-exclusively" mean? How is this different to what xfs_fsr does?

Quoted 2 lines

doesn't have to (and must not) be unmunted. When defragmentation is in progress, file IOs are served 'in parallel'. reflink feature must be enabled in the XFS.

xfs_fsr allows IO to occur in parallel to defrag.

Quoted 5 lines

Defragmentation and file IOs

The target file is virtually devided into many small segments. Segments are the smallest units for defragmentation. Each segment is defragmented one by one in a lock->defragment->unlock->idle manner.

Userspace can't easily lock the file to prevent concurrent access. So I'mnot sure what you are refering to here.

Quoted 2 lines

File IOs are blocked when the target file is locked and are served during the defragmentation idle time (file is unlocked).

What file IOs are being served in parallel? The defragmentation IO? something else?

Quoted 6 lines

Though
the file IOs can't really go in parallel, they are not blocked long. The locking time basically depends on the segment size. Smaller segments usually take less locking time and thus IOs are blocked shorterly, bigger segments usually need more locking time and IOs are blocked longer. Check -s and -i options to balance the defragmentation and IO service.

How is a user supposed to know what the correct values are for their storage, files, and workload? Algorithms should auto tune, not require users and administrators to use trial and error to find the best numbers to feed a given operation.

Quoted 8 lines

Temporary file

A temporary file is used for the defragmentation. The temporary file is created in the same directory as the target file is and is named ".xfsdefrag_<pid>". It is a sparse file and contains a defragmentation segment at a time. The temporary file is removed automatically when defragmentation is done or is cancelled by ctrl-c. It remains in case kernel crashes when defragmentation is going on. In that case, the temporary file has to be removed manaully.

O_TMPFILE, as Darrick has already requested.

Quoted 8 lines

Free blocks consumption

Defragmenation works by (trying) allocating new (contiguous) blocks, copying data and then freeing old (non-contig) blocks. Usually the number of old blocks to free equals to the number the newly allocated blocks. As a finally result, defragmentation doesn't consume free blocks. Well, that is true if the target file is not sharing blocks with other files.

This is really hard to read. Defragmentation will -always- consume free space while it is progress. It will always release the temporary space it consumes when it completes.

Quoted 3 lines

In case the target file contains shared blocks, those shared blocks won't be freed back to filesystem as they are still owned by other files. So defragmenation allocates more blocks than it frees.

So this is doing an unshare operation as well as defrag? That seems ... suboptimal. The whole point of sharing blocks is to minimise disk usage for duplicated data.

Quoted 4 lines

For existing XFS, free blocks might be over-
committed when reflink snapshots were created. To avoid causing the XFS running into low free blocks state, this defragmentation excludes (partially) shared segments when the file system free blocks reaches a shreshold. Check the -f option.

Again, how is the user supposed to know when they need to do this? If the answer is "they should always avoid defrag on low free space", then why is this an option?

Quoted 4 lines

Safty and consistency

The defragmentation file is guanrantted safe and data consistent for ctrl-c and kernel crash.

Which file is the "defragmentation file"? The source or the temp file?

Quoted 4 lines

First extent share

Current kernel has routine for each segment defragmentation detecting if the file is sharing blocks.

I have no idea what this means, or what interface this refers to.

Quoted 6 lines

It takes long in case the target file contains huge number of extents and the shared ones, if there is, are at the end. The First extent share feature works around above issue by making the first serveral blocks shared. Seeing the first blocks are shared, the kernel routine ends quickly. The side effect is that the "share" flag would remain traget file. This feature is enabled by default and can be disabled by -n option.

And from this description, I have no idea what this is doing, what problem it is trying to work around, or why we'd want to share blocks out of a file to speed up detection of whether there are shared blocks in the file. This description doesn't make any sense to me because I don't know what interface you are actually having performance issues with. Please reference the kernel code that is problematic, and explain why the existing kernel code is problematic and cannot be fixed.

Quoted 5 lines

extsize and cowextsize

According to kernel implementation, extsize and cowextsize could have following impacts to defragmentation: 1) non-zero extsize causes separated block allocations for each extent in the segment and those blocks are not contiguous.

Extent size hints do no such thing. The simply provide extent alignment guidelines and do not affect things like contiguous or multi-block allocation lengths.

Quoted 5 lines

The segment remains same
number of extents after defragmention (no effect). 2) When extsize and/or cowextsize are too big, a lot of pre-allocated blocks remain in memory for a while. When new IO comes to whose pre-allocated blocks Copy on Write happens and causes the file fragmented.

extsize based unwritten extents won't cause COW or cause fragmentation because they aren't shared and they are contiguous. I suspect that your definition of "fragmented" isn't taking into account that unwritten-written-unwritten over a contiguous range is *not* fragmentation. It's just a contiguous extent in different states, and this should really not be touched/changed by defragmentation.

check out xfs_fsr: it ensures that the pattern of unwritten/written blocks in the defragmented file is identical to the source. i.e. it preserves preallocation because the application/fs config wants it to be there....

Quoted 4 lines

Readahead

Readahead tries to fetch the data blocks for next segment with less locking in backgroud during idle time. This feature is disabled by default, use -a to enable it.

What are you reading ahead into? Kernel page cache or user buffers? Either way, it's hardly what I'd call "idle time" if the defrag process is using it to issue lots of read IO...

Quoted 5 lines

The command takes the following options:
-f free_space
The shreshold of XFS free blocks in MiB. When free blocks are less than this number, (partially) shared segments are excluded from defragmentation. Default number is 1024

When you are down to 4MB of free space in the filesystem, you shouldn't even be trying to run defrag because all the free space that will be left in the filesystem is single blocks. I would have expected this sort of number to be in a percentage of capacity, defaulting to something like 5% (which is where we start running low space algorithms in the kernel).

Quoted 3 lines

-i idle_time
The time in milliseconds, defragmentation enters idle state for this long after defragmenting a segment and before handing the next. Default number is TOBEDONE.

Yeah, I don't think this is something anyonce whould be expected to use or tune. If an idle time is needed, the defrag application should be selecting this itself.

Quoted 4 lines

-s segment_size
The size limitation in bytes of segments. Minimium number is 4MiB, default number is 16MiB.

Why were these numbers chosen? What happens if the file has ~32MB sized extents and the user wants the file to be returned to a single large contiguous extent it possible? i.e. how is the user supposed to know how to set this for any given file without first having examined the exact pattern of fragmentations in the file?

Quoted 1 line

-n Disable the First extent share feature. Enabled by default.

So confusing. Is the "feature disable flag" enabled by default, or is the feature enabled by default?

Quoted 1 line

-a Enable readahead feature, disabled by default.

Same confusion, but opposite logic.

I would highly recommend that you get a native english speaker to review, spell and grammar check the documentation before the next time you post it.

Quoted 11 lines

We tested with real customer metadump with some different 'idle_time's and found 250ms is good pratice sleep time. Here comes some number of the test:

Test: running of defrag on the image file which is used for the back end of a block device in a
virtual machine. At the same time, fio is running at the same time inside virtual machine on that block device.

block device type:   NVME
File size:           200GiB
paramters to defrag: free_space: 1024 idle_time: 250 First_extent_share: enabled readahead: disabled
Defrag run time:     223 minutes
Number of extents:   6745489(before) -> 203571(after)

So and average extent size of ~32kB before, 100MB after? How much of these are shared extents?

Runtime is 13380secs, so if we copied 200GiB in that time, the defrag ran at 16MB/s. That's not very fast.

What's the CPU utilisation of the defrag task and kernel side processing? What is the difference between "first_extent_share" enabled and disabled (both performance numbers and CPU usage)?

Quoted 2 lines
Fio read latency:    15.72ms(without defrag) -> 14.53ms(during defrag)
Fio write latency:   32.21ms(without defrag) -> 20.03ms(during defrag)

So the IO latency is *lower* when defrag is running? That doesn't make any sense, unless the fio throughput is massively reduced while defrag is running. What's the throughput change in the fio workload? What's the change in worst case latency for the fio workload? i.e. post the actual fio results so we can see the whole picture of the behaviour, not just a single cherry-picked number.

Really, though, I have to ask: why is this an xfs_spaceman command and not something built into the existing online defrag program we have (xfs_fsr)?

I'm sure I'll hav emore questions as I go through the code - I'll start at the userspace IO engine part of the patchset so I have some idea of what the defrag algorithm actually is...

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#57 lore raw
Quoted 6 lines · Dave Chinner

[ Please keep documentation text to 80 columns. ]

Yes. This is not a patch. I copied it from the man 8 output. It will be limited to 80 columns when sent as a patch.

Quoted 2 lines

[ Please run documentation through a spell checker - there are too many typos in this document to point them all out... ]

OK.

Quoted 6 lines · Wengang Wang
Quoted 2 lines

This patch set introduces defrag to xfs_spaceman command. It has the functionality and features below (also subject to be added to man page, so please review):

What's the use case for this?

This is the user space defrag as you suggested previously.

Please see the previous conversation for your reference:
https://patchwork.kernel.org/project/xfs/cover/[email protected]/

COPY STARTS —————————————>
I am copying your last comment there:

Quoted 2 lines · Wengang Wang

Hi Dave,
Yes, the user space defrag works and satisfies my requirement (almost no change from your example code).

That's good to know :)

Quoted 1 line

Let me know if you want it in xfsprog.

Yes, i think adding it as an xfs_spaceman command would be a good way for this defrag feature to be maintained for anyone who has need for it.

-Dave.
<———————————————— COPY ENDS

Quoted 7 lines
Quoted 3 lines

defrag [-f free_space] [-i idle_time] [-s segment_size] [-n] [-a]
defrag defragments the specified XFS file online non-exclusively. The target XFS

What's "non-exclusively" mean? How is this different to what xfs_fsr does?

I think you have seen the difference when you reviewing more of this set.
Well, if I read xfs_fsr code correctly, though xfs_fsr allow parallel writes, it looks have a problem(?) As I read the code, Xfs_fsr do the followings to defrag one file:
1) preallocating blocks to a temporary file hoping the temporary get same number of blocks as the file under defrag with with less extents.
2) copy data blocks from the file under defrag to the temporary file.
3) switch the extents between the two files.

For stage 2, it’s NOT copying data blocks in atomic manner. Take an example: there need two Read->write pair to complete the data copy, that is
Copy range 1 (read range 1 from the file under defrag to the temporary file) Copy range 2

If a new write come to the file (range 1) under defrag after copying range1 is done. After the defrag (xfs_fsr) finished, will the new write lose?

I didn’t look into the extents-switch code, don’t know if that check if the two files has same data contents. But even it does, it would be pretty slow with file locked.

Quoted 4 lines
Quoted 2 lines

doesn't have to (and must not) be unmunted. When defragmentation is in progress, file IOs are served 'in parallel'. reflink feature must be enabled in the XFS.

xfs_fsr allows IO to occur in parallel to defrag.

Pls see my concern above.

Quoted 9 lines
Quoted 6 lines

Defragmentation and file IOs

The target file is virtually devided into many small segments. Segments are the smallest units for defragmentation. Each segment is defragmented one by one in a lock->defragment->unlock->idle manner.

Userspace can't easily lock the file to prevent concurrent access. So I'mnot sure what you are refering to here.

The manner is not simply meant what is done at user space, but a whole thing in both user space and kernel space. The tool defrag a file segment by segment. The lock->defragment->unlock Is done by kernel in responding to the FALLOC_FL_UNSHARE_RANGE request from user space.

Quoted 6 lines
Quoted 3 lines

File IOs are blocked when the target file is locked and are served during the defragmentation idle time (file is unlocked).

What file IOs are being served in parallel? The defragmentation IO? something else?

Here the file IOs means the IOs request from user space applications including virtual machine Engine.

Quoted 8 lines
Quoted 7 lines

Though
the file IOs can't really go in parallel, they are not blocked long. The locking time basically depends on the segment size. Smaller segments usually take less locking time and thus IOs are blocked shorterly, bigger segments usually need more locking time and IOs are blocked longer. Check -s and -i options to balance the defragmentation and IO service.

Quoted 4 lines

How is a user supposed to know what the correct values are for their storage, files, and workload? Algorithms should auto tune, not require users and administrators to use trial and error to find the best numbers to feed a given operation.

In my option, user would need a way to control this according to their use case.
Any algorithms will restrict what user want to do.
Say, user want the defrag done as quick as possible regardless the resources it takes (CPU, IO and so on) when the production system is in a maintenance window. But when the production system is busy User want the defrag use less resources.
Another example, kernel (algorithms) never knows the maximum IO latency the user applications tolerate. But if you have some algorithms, please share.

And we provide default numbers for the options, they come from test practice though user might need to Change them for their own use case.

Quoted 11 lines
Quoted 9 lines

Temporary file

A temporary file is used for the defragmentation. The temporary file is created in the same directory as the target file is and is named ".xfsdefrag_<pid>". It is a sparse file and contains a defragmentation segment at a time. The temporary file is removed automatically when defragmentation is done or is cancelled by ctrl-c. It remains in case kernel crashes when defragmentation is going on. In that case, the temporary file has to be removed manaully.

O_TMPFILE, as Darrick has already requested.

OK. Will be it.

Quoted 10 lines
Quoted 9 lines

Free blocks consumption

Defragmenation works by (trying) allocating new (contiguous) blocks, copying data and then freeing old (non-contig) blocks. Usually the number of old blocks to free equals to the number the newly allocated blocks. As a finally result, defragmentation doesn't consume free blocks. Well, that is true if the target file is not sharing blocks with other files.

Quoted 3 lines

This is really hard to read. Defragmentation will -always- consume free space while it is progress. It will always release the temporary space it consumes when it completes.

I don’t think it’s always free blocks when it releases the temporary file. When the blocks were Original shared before defrag, the blocks won’t be freed.

Quoted 8 lines
Quoted 4 lines

In case the target file contains shared blocks, those shared blocks won't be freed back to filesystem as they are still owned by other files. So defragmenation allocates more blocks than it frees.

So this is doing an unshare operation as well as defrag? That seems ... suboptimal. The whole point of sharing blocks is to minimise disk usage for duplicated data.

That depends on user's need. If users think defrag is the first priority, it is. If users don’t think the disk saving is the most important, it is not. No matter what developers think.
What’s more, reflink (or sharing blocks) is not only used to minimize disk usage. Sometimes it’s Used as way to take snapshots. And those snapshots might won’t stay long.

And what’s more is that, the unshare operation is what you suggested :D

Quoted 9 lines
Quoted 5 lines

For existing XFS, free blocks might be over-
committed when reflink snapshots were created. To avoid causing the XFS running into low free blocks state, this defragmentation excludes (partially) shared segments when the file system free blocks reaches a shreshold. Check the -f option.

Again, how is the user supposed to know when they need to do this? If the answer is "they should always avoid defrag on low free space", then why is this an option?

I didn’t say "they should always avoid defrag on low free space”. And even we can’t say how low is Not tolerated by user, that depends on user use case. Though it’s an option, it has the default value Of 1GB. If users don’t set this option, that is "always avoid defrag on low free space”.

Quoted 8 lines
Quoted 5 lines

Safty and consistency

The defragmentation file is guanrantted safe and data consistent for ctrl-c and kernel crash.

Which file is the "defragmentation file"? The source or the temp file?

I don’t think there is "source concept" here. There is no data copy between files. “The defragmentation file” means the file under defrag, I will change it to “The file under defrag”. I don’t think users care about the temporary file at all.

Quoted 15 lines
Quoted 5 lines

First extent share

Current kernel has routine for each segment defragmentation detecting if the file is sharing blocks.

I have no idea what this means, or what interface this refers to.

Quoted 6 lines

It takes long in case the target file contains huge number of extents and the shared ones, if there is, are at the end. The First extent share feature works around above issue by making the first serveral blocks shared. Seeing the first blocks are shared, the kernel routine ends quickly. The side effect is that the "share" flag would remain traget file. This feature is enabled by default and can be disabled by -n option.

Quoted 8 lines

And from this description, I have no idea what this is doing, what problem it is trying to work around, or why we'd want to share blocks out of a file to speed up detection of whether there are shared blocks in the file. This description doesn't make any sense to me because I don't know what interface you are actually having performance issues with. Please reference the kernel code that is problematic, and explain why the existing kernel code is problematic and cannot be fixed.

I mentioned the kernel function name in patch 6. It is xfs_reflink_try_clear_inode_flag().

Quoted 10 lines
Quoted 6 lines

extsize and cowextsize

According to kernel implementation, extsize and cowextsize could have following impacts to defragmentation: 1) non-zero extsize causes separated block allocations for each extent in the segment and those blocks are not contiguous.

Extent size hints do no such thing. The simply provide extent alignment guidelines and do not affect things like contiguous or multi-block allocation lengths.

Extsize really make alignment on the number of blocks to allocate. But it affects more than that. When extsize is set, the allocations is not delayed allocation. xfs_reflink_unshare() does one allocation each extent. For a defrag segment containing N extents, there are N allocations.

Quoted 7 lines
Quoted 6 lines

The segment remains same
number of extents after defragmention (no effect). 2) When extsize and/or cowextsize are too big, a lot of pre-allocated blocks remain in memory for a while. When new IO comes to whose pre-allocated blocks Copy on Write happens and causes the file fragmented.

Quoted 7 lines

extsize based unwritten extents won't cause COW or cause fragmentation because they aren't shared and they are contiguous. I suspect that your definition of "fragmented" isn't taking into account that unwritten-written-unwritten over a contiguous range is *not* fragmentation. It's just a contiguous extent in different states, and this should really not be touched/changed by defragmentation.

Are you sure about that? In my option, take the buffer’s write as example,
During writeback, when the target block is found in Cow fork, Copy on Write just happens no matter If the block is really shared or not. Let’s see this simple example:
1) a file contains 4 blocks. file block 0, 1 and 2 are shared and block 3 is not share. Extsize on this file is 4 blocks.
2) a writeback come to file block 0, 1 and 2.
3) On seeing those 3 blocks are shared, kernel pre-allocate blocks in Cow fork. Extsize being 4 blocks, after alignment, 4 blocks (unwritten) are allocated in Cow fork.
4) data is written to 3 of the blocks in Cow fork. In IO done callback, those 3 blocks in Cow fork Is moved to data fork, the original 3 blocks in data fork are freed.

The Copy on Write is done, right?
But remember, there is 1 unwritten block left in the Cow fork.
In case now a new writeback come to file block 3, the kernel see there is a file block 3 in Cow fork, A new Copy on Write happens.

Quoted 6 lines

check out xfs_fsr: it ensures that the pattern of unwritten/written blocks in the defragmented file is identical to the source. i.e. it preserves preallocation because the application/fs config wants it to be there....

Quoted 6 lines
Quoted 4 lines

Readahead

Readahead tries to fetch the data blocks for next segment with less locking in backgroud during idle time. This feature is disabled by default, use -a to enable it.

What are you reading ahead into? Kernel page cache or user buffers?

Kernel page cache.

Quoted 3 lines

Either way, it's hardly what I'd call "idle time" if the defrag process is using it to issue lots of read IO...

During the “idle time”, the file is not (IOLOCK) locked though disk fetching might be happening.

Quoted 7 lines
Quoted 6 lines

The command takes the following options:
-f free_space
The shreshold of XFS free blocks in MiB. When free blocks are less than this number, (partially) shared segments are excluded from defragmentation. Default number is 1024

Quoted 6 lines

When you are down to 4MB of free space in the filesystem, you shouldn't even be trying to run defrag because all the free space that will be left in the filesystem is single blocks. I would have expected this sort of number to be in a percentage of capacity, defaulting to something like 5% (which is where we start running low space algorithms in the kernel).

I would like leave this to user. When user is doing defrag on low free space system, it won’t cause Problem to file system its self. At most the defrag fails during unshare when allocating blocks. You can’t prevent user from writing to new file when system is low free space either.

I don’t think a percentage is a good idea, say, for a 10TiB filesystem, 5% is 512GB. 512GB is pretty Enough to do things. And for a small one, say a 512 MB filesystem, 5% that’s 25MB, that’s too less. In above cases, limiting by a percentage would ether prevent user doing something that can be done Without any problem, or allow user to do something that might cause problem.
I’d think specifying a fixed safe size is better.

Quoted 8 lines
Quoted 4 lines

-i idle_time
The time in milliseconds, defragmentation enters idle state for this long after defragmenting a segment and before handing the next. Default number is TOBEDONE.

Yeah, I don't think this is something anyonce whould be expected to use or tune. If an idle time is needed, the defrag application should be selecting this itself.

I don’t think so, see my explain above.

Quoted 10 lines
Quoted 4 lines

-s segment_size
The size limitation in bytes of segments. Minimium number is 4MiB, default number is 16MiB.

Why were these numbers chosen? What happens if the file has ~32MB sized extents and the user wants the file to be returned to a single large contiguous extent it possible? i.e. how is the user supposed to know how to set this for any given file without first having examined the exact pattern of fragmentations in the file?

Why customer want the file to be returned to a single large contiguous extent?
A 32MB extent is pretty good to me. I didn’t here any customer complain about 32MB extents… And you know, whether we can defrag extents to a large one depends on not only the tool it’s self. It’s depends on the status of the filesystem, say if the filesystem is very fragmented too, say the AG size..

The 16MB is selected according our tests basing on a customer metadump. With 16MB segment size, The the defrag result is very good and the IO latency is acceptable too. With the default 16MB segment Size, 32MB extent is excluded from defrag.

If you have better default size, we can use that.

Quoted 5 lines
Quoted 2 lines

-n Disable the First extent share feature. Enabled by default.

So confusing. Is the "feature disable flag" enabled by default, or is the feature enabled by default?

Will change it to the following if it’s clear:
The "First extent share “ feature is enabled to default. User -n to disable it.

Quoted 8 lines
Quoted 2 lines

-a Enable readahead feature, disabled by default.

Same confusion, but opposite logic.

I would highly recommend that you get a native english speaker to review, spell and grammar check the documentation before the next time you post it.

OK, will try to do so.

Quoted 13 lines
Quoted 4 lines

We tested with real customer metadump with some different 'idle_time's and found 250ms is good pratice sleep time. Here comes some number of the test:

Quoted 8 lines

Test: running of defrag on the image file which is used for the back end of a block device in a
virtual machine. At the same time, fio is running at the same time inside virtual machine on that block device.

block device type:   NVME
File size:           200GiB
paramters to defrag: free_space: 1024 idle_time: 250 First_extent_share: enabled readahead: disabled
Defrag run time:     223 minutes
Number of extents:   6745489(before) -> 203571(after)
Quoted 2 lines

So and average extent size of ~32kB before, 100MB after? How much of these are shared extents?

Zero shared extents, but there are some unwritten ones.
A similar run stats is like this:
Pre-defrag 6654460 extents detected, 112228 are "unwritten",0 are "shared”
Tried to defragment 6393352 extents (181000359936 bytes) in 26032 segments Time stats(ms): max clone: 31, max unshare: 300, max punch_hole: 66 Post-defrag 282659 extents detected

Quoted 4 lines

Runtime is 13380secs, so if we copied 200GiB in that time, the defrag ran at 16MB/s. That's not very fast.

We are chasing the balance of defrag and parallel IO latency.

Quoted 3 lines

What's the CPU utilisation of the defrag task and kernel side processing? What is the difference between "first_extent_share" enabled and disabled (both performance numbers and CPU usage)?

On my test VM (spindle based disk I think). CPU usage is about 6% for
The defrag command. Kernel processes much lower.
I didn’t pay much attention to the CPU usage when “first_extent_share” is disabled. But think That caused very high CPU usages.

Quoted 7 lines
Quoted 3 lines
Fio read latency:    15.72ms(without defrag) -> 14.53ms(during defrag)
Fio write latency:   32.21ms(without defrag) -> 20.03ms(during defrag)

So the IO latency is *lower* when defrag is running? That doesn't make any sense, unless the fio throughput is massively reduced while defrag is running.

That’s reasonable. For the segments that defrag is done, the page cache remains.

Quoted 4 lines

What's the throughput change in the fio
workload? What's the change in worst case latency for the fio workload? i.e. post the actual fio results so we can see the whole picture of the behaviour, not just a single cherry-picked number.

Let me see if we have that saved.

Quoted 5 lines

Really, though, I have to ask: why is this an xfs_spaceman command and not something built into the existing online defrag program we have (xfs_fsr)?

Quotation from previous conversation:
“”””"

Quoted 1 line

Let me know if you want it in xfsprog.

Yes, i think adding it as an xfs_spaceman command would be a good way for this defrag feature to be maintained for anyone who has need for it.

-Dave.
“”””””

Thanks,
Wengang

Quoted 8 lines

I'm sure I'll hav emore questions as I go through the code - I'll start at the userspace IO engine part of the patchset so I have some idea of what the defrag algorithm actually is...

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#58 lore raw
Quoted 20 lines · Wengang Wang
Quoted 6 lines · Dave Chinner

[ Please keep documentation text to 80 columns. ]

Yes. This is not a patch. I copied it from the man 8 output. It will be limited to 80 columns when sent as a patch.

Quoted 2 lines

[ Please run documentation through a spell checker - there are too many typos in this document to point them all out... ]

OK.

Quoted 5 lines · Wengang Wang
Quoted 2 lines

This patch set introduces defrag to xfs_spaceman command. It has the functionality and features below (also subject to be added to man page, so please review):

Quoted 6 lines
Quoted 1 line

What's the use case for this?

This is the user space defrag as you suggested previously.

Please see the previous conversation for your reference:
https://patchwork.kernel.org/project/xfs/cover/[email protected]/

That's exactly what you should have put in the cover letter!

The cover letter is not for documenting the user interface of a new tool - that's what the patch in the patch set for the new man page should be doing.

The cover letter should contain references to past patch sets and discussions on the topic. The cover letter shoudl also contain a changelog that documents what is different in this new version of the patch set so reviewers know what you've changed since they last looked at it.

IOWs, the cover letter for explaining the use case, why the functionality is needed, important design/implementation decisions and the history of the patchset. It's meant to inform and remind readers of what has already happened to get to this point.

Quoted 7 lines

COPY STARTS —————————————>
I am copying your last comment there:

Quoted 2 lines · Wengang Wang

Hi Dave,
Yes, the user space defrag works and satisfies my requirement (almost no change from your example code).

Quoted 7 lines

That's good to know :)

Quoted 1 line

Let me know if you want it in xfsprog.

Yes, i think adding it as an xfs_spaceman command would be a good way for this defrag feature to be maintained for anyone who has need for it.

Sure, I might have said that 6 months ago. When presented with a completely new implementation in a new context months later, I might see things differently. Everyone is allowed to change their mind, opinions and theories as circumstances, evidence and contexts change.

Indeed when I look at this:

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

defrag [-f free_space] [-i idle_time] [-s segment_size] [-n] [-a]
defrag defragments the specified XFS file online non-exclusively. The target XFS

I didn't expect anything nearly as complex and baroque as this. All I was expecting was something like this to defrag a single range of a file:

xfs_spaceman -c "defrag <offset> <length>" <file>

As the control command, and then functionality for automated/periodic scanning and defrag would still end up being co-ordinated by the existing xfs_fsr code.

Quoted 12 lines
Quoted 3 lines

What's "non-exclusively" mean? How is this different to what xfs_fsr does?

I think you have seen the difference when you reviewing more of this set.
Well, if I read xfs_fsr code correctly, though xfs_fsr allow parallel writes, it looks have a problem(?) As I read the code, Xfs_fsr do the followings to defrag one file:
1) preallocating blocks to a temporary file hoping the temporary get same number of blocks as the file under defrag with with less extents.
2) copy data blocks from the file under defrag to the temporary file.
3) switch the extents between the two files.

Quoted 4 lines

For stage 2, it’s NOT copying data blocks in atomic manner. Take an example: there need two Read->write pair to complete the data copy, that is
Copy range 1 (read range 1 from the file under defrag to the temporary file) Copy range 2

I wasn't asking you to explain to me how the xfs_fsr algorithm works. What I was asking for was a definition of what "non-exclusively" means.

What xfs_fsr currently does meets my definition of "non-exclusive" - it does not rely on or require exclusive access to the file being defragmented except for the atomic extent swap at the end. However, using FICLONE/UNSHARE does require exclusive access to the file be defragmented for the entirity of those operations, so I don't have any real idea of why this new algorithm is explicitly described as "non-exclusive".

Defining terms so everyone has a common understanding is important.

Indeed, Given that we now have XFS_IOC_EXCHANGE_RANGE, I'm definitely starting to wonder if clone/unshare is actually the best way to do this now. I think we could make xfs_fsr do iterative small file region defrag using XFS_IOC_EXCHANGE_RANGE instead of 'whole file at once' as it does now. If we were also to make fsr aware of shared extents

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Defragmentation and file IOs

The target file is virtually devided into many small segments. Segments are the smallest units for defragmentation. Each segment is defragmented one by one in a lock->defragment->unlock->idle manner.

Quoted 6 lines
Quoted 2 lines

Userspace can't easily lock the file to prevent concurrent access. So I'mnot sure what you are refering to here.

The manner is not simply meant what is done at user space, but a whole thing in both user space and kernel space. The tool defrag a file segment by segment. The lock->defragment->unlock Is done by kernel in responding to the FALLOC_FL_UNSHARE_RANGE request from user space.

I'm still not sure what locking you are trying to describe. There are multiple layers of locking in the kernel, and we use them differently. Indeed, the algorithm you have described is actually

	FICLONERANGE
	IOLOCK shared
	ILOCK exclusive
	remap_file_range()
	IUNLOCK exclusive
	IOUNLOCK shared

	.....

	UNSHARE_RANGE
	IOLOCK exclusive
	MMAPLOCK exclusive
	<drain DIO in flight>
	ILOCK exclusive
	unshare_range()
	IUNLOCK exclusive
	MMAPUNLOCK exclusive
	IOUNLOCK shared

And so there isn't a single "lock -> defrag -> unlock" context occurring - there are multiple independent operations that have different kernel side locking contexts and there are no userspace side file locking contexts, either.

Quoted 23 lines
Quoted 6 lines
Quoted 3 lines

File IOs are blocked when the target file is locked and are served during the defragmentation idle time (file is unlocked).

What file IOs are being served in parallel? The defragmentation IO? something else?

Here the file IOs means the IOs request from user space applications including virtual machine Engine.

Quoted 8 lines
Quoted 7 lines

Though
the file IOs can't really go in parallel, they are not blocked long. The locking time basically depends on the segment size. Smaller segments usually take less locking time and thus IOs are blocked shorterly, bigger segments usually need more locking time and IOs are blocked longer. Check -s and -i options to balance the defragmentation and IO service.

Quoted 4 lines

How is a user supposed to know what the correct values are for their storage, files, and workload? Algorithms should auto tune, not require users and administrators to use trial and error to find the best numbers to feed a given operation.

Quoted 5 lines

In my option, user would need a way to control this according to their use case.
Any algorithms will restrict what user want to do.
Say, user want the defrag done as quick as possible regardless the resources it takes (CPU, IO and so on) when the production system is in a maintenance window. But when the production system is busy User want the defrag use less resources.

That's not for the defrag program to implement That's what we use resource control groups for. Things like memcgs, block IO cgroups, scheduler cgroups, etc. Administrators are used to restricting the resources used by applications with generic admin tools; asking them to learn how some random admin tool does it's own resrouce utilisation restriction that requires careful hand tuning for -one off admin events- is not the right way to solve this problem.

We should be making the admin tool go as fast as possible and consume as much resources as are available. This makes it fast out of the box, and lets the admins restrict the IO rate, CPU and memory usage to bring it down to an acceptible resource usage level for admin tasks on their systems.

Quoted 2 lines

Another example, kernel (algorithms) never knows the maximum IO latency the user applications tolerate. But if you have some algorithms, please share.

As I said - make it as fast and low latency as reasonably possible. If you have less than 10ms IO latency SLAs, the application isn't going to be running on sparse, software defined storage that may require hundreds of milliseconds of IO pauses during admin tasks. Hence design to a max fixed IO latency (say 100ms) and make the funcitonality run as fast as possible within that latency window.

If people need lower latency SLAs, then they shouldn't be running that application on sparse, COW based VM images. This is not a problem a defrag utility should be trying to solve.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

Free blocks consumption

Defragmenation works by (trying) allocating new (contiguous) blocks, copying data and then freeing old (non-contig) blocks. Usually the number of old blocks to free equals to the number the newly allocated blocks. As a finally result, defragmentation doesn't consume free blocks. Well, that is true if the target file is not sharing blocks with other files.

Quoted 6 lines
Quoted 3 lines

This is really hard to read. Defragmentation will -always- consume free space while it is progress. It will always release the temporary space it consumes when it completes.

I don’t think it’s always free blocks when it releases the temporary file. When the blocks were Original shared before defrag, the blocks won’t be freed.

I didn't make myself clear. If the blocks shared to the temp file are owned exclusively by the source file (i.e. they were COW'd from shared extents at some time in the past), then that is space that is temporarily required by the defragmentation process. UNSHARE creates a second, permanent copy of those blocks in the source file and closing of the temp file them makes the original exclusively owned blocks go away.

IOWs, defrag can temporarily consume an entire extra file's worth of space between the UNSHARE starting and the freeing of the temporary file when we are done with it. Freeing the temp file -always- releases this extra space, though I note that the implementation is to hole-punch it away after each segment has been processed.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

In case the target file contains shared blocks, those shared blocks won't be freed back to filesystem as they are still owned by other files. So defragmenation allocates more blocks than it frees.

Quoted 7 lines
Quoted 3 lines

So this is doing an unshare operation as well as defrag? That seems ... suboptimal. The whole point of sharing blocks is to minimise disk usage for duplicated data.

That depends on user's need. If users think defrag is the first priority, it is. If users don’t think the disk
saving is the most important, it is not. No matter what developers think.

                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

That's pretty ... dismissive.

I mean, you're flat out wrong. You make the assumption that a user knows exactly how every file that every application in their system has been created and knows exactly how best to defragment it.

That's just .... wrong.

Users and admins do not have intimate knowledge of how their applications do their stuff, and a lot of them don't even know that their systems are using file clones (i.e. reflink copies) instead of data copies extensively these days.

That is completely the wrong way to approach administration tools.

Our defragmentation policy for xfs_fsr is to leave the structure of the file as intact as possible. That means we replicate unwritten regions in the defragmented file. We actually -defragment unwritten extents- in xfs_fsr, not just written extents, and we do that because we have to assume that the unwritten extents exist for a good reason.

We don't expect the admin to make a decision as to whether unwritten extents should be replicated or defragged - we make the assumption that either the application or the admin has asked for them to exist in the first place.

It is similar for defragmenting files that are largely made up of shared extents. That layout exists for a reason, and it's not the place of the defragmentation operation to unilaterally decide layout policies for the admin and/or application that is using files with shared extents.

Hence the defrag operation must preserve the *observed intent* of the source file layout as much as possible and not require the admin or user to be sufficiently informed to make the right decision one way or another. We must attempt to preserve the status quo.

Hence if the file is largely shared, we must not unshare the entire file to defragment it unless that is the only way to reduce the fragmentation (e.g. resolve small interleaved shared and unshared extents). If there are reasonable sized shared extents, we should be leaving them alone and not unsharing them just to reduce the extent count by a handful of extents.

Quoted 2 lines

What’s more, reflink (or sharing blocks) is not only used to minimize disk usage. Sometimes it’s Used as way to take snapshots. And those snapshots might won’t stay long.

Yes, I know this. It doesn't change anything to do with how we defragment a file that contains shared blocks.

If you don't want the snapshot(s) to affect defragmentation, then don't run defrag while the snapshots are present. Otherwise, we want defrag to retain as much sharing between the snapshots and the source file because *minimising the space used by snapshots* is the whole point of using file clones for snapshots in the first place!

Quoted 1 line

And what’s more is that, the unshare operation is what you suggested :D

I suggested it as a mechanism to defrag regions of shared files with excessive fragmentation. I was not suggesting that "defrag == unshare".

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

For existing XFS, free blocks might be over-
committed when reflink snapshots were created. To avoid causing the XFS running into low free blocks state, this defragmentation excludes (partially) shared segments when the file system free blocks reaches a shreshold. Check the -f option.

Quoted 7 lines
Quoted 3 lines

Again, how is the user supposed to know when they need to do this? If the answer is "they should always avoid defrag on low free space", then why is this an option?

I didn’t say "they should always avoid defrag on low free space”. And even we can’t say how low is Not tolerated by user, that depends on user use case. Though it’s an option, it has the default value Of 1GB. If users don’t set this option, that is "always avoid defrag on low free space”.

You didn't answer my question: how is the user supposed to know when they should set this?

And, again, the followup question is: why does this need to be built into the defrag tool?

From a policy perspective, caring about the amount of free space in the filesystem isn't the job of a defragmentation operation. It should simply abort if it gets an ENOSPC error or fails to improve the layout of the file in question. Indeed, if it is obvious that there may not be enough free space in the filesystem to begin with thendon't run the defrag operation at all.

This is how xfs_fsr works - it tries to preallocate all the space it will need before it starts moving data. If it fails to preallocate all the space, it aborts. If it fails to find large enough contiguous free spaces to improve the layout of the file, it aborts.

IOWs, xfs_fsr policy is that it doesn't care about the amount of free space in the filesystem, it just cares if the result will improve the layout of the file. That's basically how any online background defrag operation should work - if the new layout is worse than the existing layout, or there isn't space for the new layout to be allocated, just abort.

Quoted 11 lines
Quoted 7 lines
Quoted 4 lines

Safty and consistency

The defragmentation file is guanrantted safe and data consistent for ctrl-c and kernel crash.

Which file is the "defragmentation file"? The source or the temp file?

I don’t think there is "source concept" here. There is no data copy between files. “The defragmentation file” means the file under defrag, I will change it to “The file under defrag”. I don’t think users care about the temporary file at all.

Define the terms you use rather than assuming the reader understands both the terminology you are using and the context in which you are using them.

.....

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

The command takes the following options:
-f free_space
The shreshold of XFS free blocks in MiB. When free blocks are less than this number, (partially) shared segments are excluded from defragmentation. Default number is 1024

Quoted 8 lines
Quoted 6 lines

When you are down to 4MB of free space in the filesystem, you shouldn't even be trying to run defrag because all the free space that will be left in the filesystem is single blocks. I would have expected this sort of number to be in a percentage of capacity, defaulting to something like 5% (which is where we start running low space algorithms in the kernel).

I would like leave this to user.

Again: How is the user going to know what to set this to? What problem is this avoiding that requires the user to change this in any way.

Quoted 2 lines

When user is doing defrag on low free space system, it won’t cause
Problem to file system its self. At most the defrag fails during unshare when allocating blocks.

Why would we even allow a user to run defrag near ENOSPC? It is a well known problem that finding contiguous free space when we are close to ENOSPC is difficult and so defrag often is unable to improve the situation when we are within a few percent of the filesysetm being full.

It is also a well known problem that defragmentation at low free space trades off contiguous free space for fragmented free space. Hence when we are at low free space, defrag makes the free space fragmetnation worse, which then results in all allocation in the filesystem getting worse and more fragmented. This is something we absolutely should be trying to avoid.

This is one of the reasons xfs_fsr tries to layout the entire file before doing any IO - when about 95% full, it's common for the new layout to be worse than the original file's layout because there isn't sufficient contiguous free space to improve the layout.

IOWs, running defragmentation when we are above 95% full is actively harmful to the longevity of the filesystem. Hence, on a fundamental level, having a low space threshold in a defragmentation tool is simply wrong - defragmentation should simply not be run when the filesystem is anywhere near full.

.....

Quoted 11 lines
Quoted 10 lines
Quoted 4 lines

-s segment_size
The size limitation in bytes of segments. Minimium number is 4MiB, default number is 16MiB.

Why were these numbers chosen? What happens if the file has ~32MB sized extents and the user wants the file to be returned to a single large contiguous extent it possible? i.e. how is the user supposed to know how to set this for any given file without first having examined the exact pattern of fragmentations in the file?

Quoted 3 lines

Why customer want the file to be returned to a single large contiguous extent? A 32MB extent is pretty good to me. I didn’t here any customer complain about 32MB extents…

There's a much wider world out there than just Oracle customers. Just because you aren't aware of other use cases that exist, it doesn't mean they don't exist. I know they exist, hence my question.

For example, extent size hints are used to guarantee that the data is aligned to the underlying storage correctly, and very large contiguous extents are required to avoid excessive seeks during sequential reads that result in critical SLA failures. Hence if a file is poorly laid out in this situation, defrag needs to return it to as few, maximally sized extents as it can. How does a user know what they'd need to set this segment size field to and so acheive the result they need?

Quoted 6 lines

And you know, whether we can defrag extents to a large one depends on not only the tool it’s self. It’s depends on the status of the filesystem, say if the filesystem is very fragmented too, say the AG size..

The 16MB is selected according our tests basing on a customer metadump. With 16MB segment size, The the defrag result is very good and the IO latency is acceptable too. With the default 16MB segment Size, 32MB extent is excluded from defrag.

Exactly my point: you have written a solution that works for a single filesystem in a single environment. However, the solution is so specific to the single problem you need to solve that it is not clear whether that functionality or defaults are valid outside of the specific problem case you've written it for and tested it on.

Quoted 1 line

If you have better default size, we can use that.

I'm not convinced that fixed size "segments" is even the right way to approach this problem. What needs to be done is dependent on the extent layout of the file, not how extents fit over some arbitrary fixed segment map....

Quoted 22 lines
Quoted 12 lines
Quoted 11 lines

We tested with real customer metadump with some different 'idle_time's and found 250ms is good pratice sleep time. Here comes some number of the test:

Test: running of defrag on the image file which is used for the back end of a block device in a
virtual machine. At the same time, fio is running at the same time inside virtual machine on that block device.

block device type:   NVME
File size:           200GiB
paramters to defrag: free_space: 1024 idle_time: 250 First_extent_share: enabled readahead: disabled
Defrag run time:     223 minutes
Number of extents:   6745489(before) -> 203571(after)
Quoted 2 lines

So and average extent size of ~32kB before, 100MB after? How much of these are shared extents?

Zero shared extents, but there are some unwritten ones.
A similar run stats is like this:
Pre-defrag 6654460 extents detected, 112228 are "unwritten",0 are "shared”
Tried to defragment 6393352 extents (181000359936 bytes) in 26032 segments Time stats(ms): max clone: 31, max unshare: 300, max punch_hole: 66 Post-defrag 282659 extents detected

Quoted 1 line
Quoted 5 lines
Quoted 3 lines

Runtime is 13380secs, so if we copied 200GiB in that time, the defrag ran at 16MB/s. That's not very fast.

We are chasing the balance of defrag and parallel IO latency.

My point is that stuff like CLONE and UNSHARE should be able to run much, much faster than this, even if some of the time is left idle for other IO.

i.e. we can clone extents at about 100,000/s. We can copy data through the page cache at 7-8GB/s on NVMe devices.

A full clone of the 6.6 million extents should only take about a minute.

A full page cache copy of the 200GB cloned file (i.e. via read/write syscalls) should easily run at >1GB/s, and so only take a couple of minutes to run.

IOWs, the actual IO and metadata modification side of things is really only about 5 minutes worth of CPU and IO.

Hence this defrag operation is roughly 100x slower than we should be able to run it at. We should be able to run it at close to those speeds whilst still allowing concurrent read access to the file.

If an admin then wants it to run at 16MB/s, it can be throttled to that speed using cgroups, ionice, etc.

i.e. I think you are trying to solve too many unnecessary problems here and not addressing the one thing it should do: defrag a file as fast and efficiently as possible.

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#59 lore raw
Quoted 533 lines · Dave Chinner
Quoted 20 lines · Wengang Wang
Quoted 6 lines · Dave Chinner

[ Please keep documentation text to 80 columns. ]

Yes. This is not a patch. I copied it from the man 8 output. It will be limited to 80 columns when sent as a patch.

Quoted 2 lines

[ Please run documentation through a spell checker - there are too many typos in this document to point them all out... ]

OK.

Quoted 5 lines · Wengang Wang
Quoted 2 lines

This patch set introduces defrag to xfs_spaceman command. It has the functionality and features below (also subject to be added to man page, so please review):

Quoted 6 lines
Quoted 1 line

What's the use case for this?

This is the user space defrag as you suggested previously.

Please see the previous conversation for your reference:
https://patchwork.kernel.org/project/xfs/cover/[email protected]/

That's exactly what you should have put in the cover letter!

The cover letter is not for documenting the user interface of a new tool - that's what the patch in the patch set for the new man page should be doing.

The cover letter should contain references to past patch sets and discussions on the topic. The cover letter shoudl also contain a changelog that documents what is different in this new version of the patch set so reviewers know what you've changed since they last looked at it.

IOWs, the cover letter for explaining the use case, why the functionality is needed, important design/implementation decisions and the history of the patchset. It's meant to inform and remind readers of what has already happened to get to this point.

Quoted 7 lines

COPY STARTS —————————————>
I am copying your last comment there:

Quoted 2 lines · Wengang Wang

Hi Dave,
Yes, the user space defrag works and satisfies my requirement (almost no change from your example code).

Quoted 7 lines

That's good to know :)

Quoted 1 line

Let me know if you want it in xfsprog.

Yes, i think adding it as an xfs_spaceman command would be a good way for this defrag feature to be maintained for anyone who has need for it.

Sure, I might have said that 6 months ago. When presented with a completely new implementation in a new context months later, I might see things differently. Everyone is allowed to change their mind, opinions and theories as circumstances, evidence and contexts change.

Indeed when I look at this:

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

defrag [-f free_space] [-i idle_time] [-s segment_size] [-n] [-a]
defrag defragments the specified XFS file online non-exclusively. The target XFS

I didn't expect anything nearly as complex and baroque as this. All I was expecting was something like this to defrag a single range of a file:

xfs_spaceman -c "defrag <offset> <length>" <file>

As the control command, and then functionality for automated/periodic scanning and defrag would still end up being co-ordinated by the existing xfs_fsr code.

Quoted 12 lines
Quoted 3 lines

What's "non-exclusively" mean? How is this different to what xfs_fsr does?

I think you have seen the difference when you reviewing more of this set.
Well, if I read xfs_fsr code correctly, though xfs_fsr allow parallel writes, it looks have a problem(?) As I read the code, Xfs_fsr do the followings to defrag one file:
1) preallocating blocks to a temporary file hoping the temporary get same number of blocks as the file under defrag with with less extents.
2) copy data blocks from the file under defrag to the temporary file.
3) switch the extents between the two files.

Quoted 4 lines

For stage 2, it’s NOT copying data blocks in atomic manner. Take an example: there need two Read->write pair to complete the data copy, that is
Copy range 1 (read range 1 from the file under defrag to the temporary file) Copy range 2

I wasn't asking you to explain to me how the xfs_fsr algorithm works. What I was asking for was a definition of what "non-exclusively" means.

What xfs_fsr currently does meets my definition of "non-exclusive" - it does not rely on or require exclusive access to the file being defragmented except for the atomic extent swap at the end. However, using FICLONE/UNSHARE does require exclusive access to the file be defragmented for the entirity of those operations, so I don't have any real idea of why this new algorithm is explicitly described as "non-exclusive".

Defining terms so everyone has a common understanding is important.

Indeed, Given that we now have XFS_IOC_EXCHANGE_RANGE, I'm definitely starting to wonder if clone/unshare is actually the best way to do this now. I think we could make xfs_fsr do iterative small file region defrag using XFS_IOC_EXCHANGE_RANGE instead of 'whole file at once' as it does now. If we were also to make fsr aware of shared extents

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Defragmentation and file IOs

The target file is virtually devided into many small segments. Segments are the smallest units for defragmentation. Each segment is defragmented one by one in a lock->defragment->unlock->idle manner.

Quoted 6 lines
Quoted 2 lines

Userspace can't easily lock the file to prevent concurrent access. So I'mnot sure what you are refering to here.

The manner is not simply meant what is done at user space, but a whole thing in both user space and kernel space. The tool defrag a file segment by segment. The lock->defragment->unlock Is done by kernel in responding to the FALLOC_FL_UNSHARE_RANGE request from user space.

I'm still not sure what locking you are trying to describe. There are multiple layers of locking in the kernel, and we use them differently. Indeed, the algorithm you have described is actually

FICLONERANGE
IOLOCK shared
ILOCK exclusive
remap_file_range()
IUNLOCK exclusive
IOUNLOCK shared

.....

UNSHARE_RANGE
IOLOCK exclusive
MMAPLOCK exclusive
<drain DIO in flight>
ILOCK exclusive
unshare_range()
IUNLOCK exclusive
MMAPUNLOCK exclusive
IOUNLOCK shared

And so there isn't a single "lock -> defrag -> unlock" context occurring - there are multiple independent operations that have different kernel side locking contexts and there are no userspace side file locking contexts, either.

Quoted 23 lines
Quoted 6 lines
Quoted 3 lines

File IOs are blocked when the target file is locked and are served during the defragmentation idle time (file is unlocked).

What file IOs are being served in parallel? The defragmentation IO? something else?

Here the file IOs means the IOs request from user space applications including virtual machine Engine.

Quoted 8 lines
Quoted 7 lines

Though
the file IOs can't really go in parallel, they are not blocked long. The locking time basically depends on the segment size. Smaller segments usually take less locking time and thus IOs are blocked shorterly, bigger segments usually need more locking time and IOs are blocked longer. Check -s and -i options to balance the defragmentation and IO service.

Quoted 4 lines

How is a user supposed to know what the correct values are for their storage, files, and workload? Algorithms should auto tune, not require users and administrators to use trial and error to find the best numbers to feed a given operation.

Quoted 5 lines

In my option, user would need a way to control this according to their use case.
Any algorithms will restrict what user want to do.
Say, user want the defrag done as quick as possible regardless the resources it takes (CPU, IO and so on) when the production system is in a maintenance window. But when the production system is busy User want the defrag use less resources.

That's not for the defrag program to implement That's what we use resource control groups for. Things like memcgs, block IO cgroups, scheduler cgroups, etc. Administrators are used to restricting the resources used by applications with generic admin tools; asking them to learn how some random admin tool does it's own resrouce utilisation restriction that requires careful hand tuning for -one off admin events- is not the right way to solve this problem.

We should be making the admin tool go as fast as possible and consume as much resources as are available. This makes it fast out of the box, and lets the admins restrict the IO rate, CPU and memory usage to bring it down to an acceptible resource usage level for admin tasks on their systems.

Quoted 2 lines

Another example, kernel (algorithms) never knows the maximum IO latency the user applications tolerate. But if you have some algorithms, please share.

As I said - make it as fast and low latency as reasonably possible. If you have less than 10ms IO latency SLAs, the application isn't going to be running on sparse, software defined storage that may require hundreds of milliseconds of IO pauses during admin tasks. Hence design to a max fixed IO latency (say 100ms) and make the funcitonality run as fast as possible within that latency window.

If people need lower latency SLAs, then they shouldn't be running that application on sparse, COW based VM images. This is not a problem a defrag utility should be trying to solve.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

Free blocks consumption

Defragmenation works by (trying) allocating new (contiguous) blocks, copying data and then freeing old (non-contig) blocks. Usually the number of old blocks to free equals to the number the newly allocated blocks. As a finally result, defragmentation doesn't consume free blocks. Well, that is true if the target file is not sharing blocks with other files.

Quoted 6 lines
Quoted 3 lines

This is really hard to read. Defragmentation will -always- consume free space while it is progress. It will always release the temporary space it consumes when it completes.

I don’t think it’s always free blocks when it releases the temporary file. When the blocks were Original shared before defrag, the blocks won’t be freed.

I didn't make myself clear. If the blocks shared to the temp file are owned exclusively by the source file (i.e. they were COW'd from shared extents at some time in the past), then that is space that is temporarily required by the defragmentation process. UNSHARE creates a second, permanent copy of those blocks in the source file and closing of the temp file them makes the original exclusively owned blocks go away.

IOWs, defrag can temporarily consume an entire extra file's worth of space between the UNSHARE starting and the freeing of the temporary file when we are done with it. Freeing the temp file -always- releases this extra space, though I note that the implementation is to hole-punch it away after each segment has been processed.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

In case the target file contains shared blocks, those shared blocks won't be freed back to filesystem as they are still owned by other files. So defragmenation allocates more blocks than it frees.

Quoted 7 lines
Quoted 3 lines

So this is doing an unshare operation as well as defrag? That seems ... suboptimal. The whole point of sharing blocks is to minimise disk usage for duplicated data.

That depends on user's need. If users think defrag is the first priority, it is. If users don’t think the disk
saving is the most important, it is not. No matter what developers think.

                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

That's pretty ... dismissive.

I mean, you're flat out wrong. You make the assumption that a user knows exactly how every file that every application in their system has been created and knows exactly how best to defragment it.

That's just .... wrong.

Users and admins do not have intimate knowledge of how their applications do their stuff, and a lot of them don't even know that their systems are using file clones (i.e. reflink copies) instead of data copies extensively these days.

That is completely the wrong way to approach administration tools.

Our defragmentation policy for xfs_fsr is to leave the structure of the file as intact as possible. That means we replicate unwritten regions in the defragmented file. We actually -defragment unwritten extents- in xfs_fsr, not just written extents, and we do that because we have to assume that the unwritten extents exist for a good reason.

We don't expect the admin to make a decision as to whether unwritten extents should be replicated or defragged - we make the assumption that either the application or the admin has asked for them to exist in the first place.

It is similar for defragmenting files that are largely made up of shared extents. That layout exists for a reason, and it's not the place of the defragmentation operation to unilaterally decide layout policies for the admin and/or application that is using files with shared extents.

Hence the defrag operation must preserve the *observed intent* of the source file layout as much as possible and not require the admin or user to be sufficiently informed to make the right decision one way or another. We must attempt to preserve the status quo.

Hence if the file is largely shared, we must not unshare the entire file to defragment it unless that is the only way to reduce the fragmentation (e.g. resolve small interleaved shared and unshared extents). If there are reasonable sized shared extents, we should be leaving them alone and not unsharing them just to reduce the extent count by a handful of extents.

Quoted 2 lines

What’s more, reflink (or sharing blocks) is not only used to minimize disk usage. Sometimes it’s Used as way to take snapshots. And those snapshots might won’t stay long.

Yes, I know this. It doesn't change anything to do with how we defragment a file that contains shared blocks.

If you don't want the snapshot(s) to affect defragmentation, then don't run defrag while the snapshots are present. Otherwise, we want defrag to retain as much sharing between the snapshots and the source file because *minimising the space used by snapshots* is the whole point of using file clones for snapshots in the first place!

Quoted 1 line

And what’s more is that, the unshare operation is what you suggested :D

I suggested it as a mechanism to defrag regions of shared files with excessive fragmentation. I was not suggesting that "defrag == unshare".

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

For existing XFS, free blocks might be over-
committed when reflink snapshots were created. To avoid causing the XFS running into low free blocks state, this defragmentation excludes (partially) shared segments when the file system free blocks reaches a shreshold. Check the -f option.

Quoted 7 lines
Quoted 3 lines

Again, how is the user supposed to know when they need to do this? If the answer is "they should always avoid defrag on low free space", then why is this an option?

I didn’t say "they should always avoid defrag on low free space”. And even we can’t say how low is Not tolerated by user, that depends on user use case. Though it’s an option, it has the default value Of 1GB. If users don’t set this option, that is "always avoid defrag on low free space”.

You didn't answer my question: how is the user supposed to know when they should set this?

And, again, the followup question is: why does this need to be built into the defrag tool?

From a policy perspective, caring about the amount of free space in the filesystem isn't the job of a defragmentation operation. It should simply abort if it gets an ENOSPC error or fails to improve the layout of the file in question. Indeed, if it is obvious that there may not be enough free space in the filesystem to begin with thendon't run the defrag operation at all.

This is how xfs_fsr works - it tries to preallocate all the space it will need before it starts moving data. If it fails to preallocate all the space, it aborts. If it fails to find large enough contiguous free spaces to improve the layout of the file, it aborts.

IOWs, xfs_fsr policy is that it doesn't care about the amount of free space in the filesystem, it just cares if the result will improve the layout of the file. That's basically how any online background defrag operation should work - if the new layout is worse than the existing layout, or there isn't space for the new layout to be allocated, just abort.

Quoted 11 lines
Quoted 7 lines
Quoted 4 lines

Safty and consistency

The defragmentation file is guanrantted safe and data consistent for ctrl-c and kernel crash.

Which file is the "defragmentation file"? The source or the temp file?

I don’t think there is "source concept" here. There is no data copy between files. “The defragmentation file” means the file under defrag, I will change it to “The file under defrag”. I don’t think users care about the temporary file at all.

Define the terms you use rather than assuming the reader understands both the terminology you are using and the context in which you are using them.

.....

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

The command takes the following options:
-f free_space
The shreshold of XFS free blocks in MiB. When free blocks are less than this number, (partially) shared segments are excluded from defragmentation. Default number is 1024

Quoted 8 lines
Quoted 6 lines

When you are down to 4MB of free space in the filesystem, you shouldn't even be trying to run defrag because all the free space that will be left in the filesystem is single blocks. I would have expected this sort of number to be in a percentage of capacity, defaulting to something like 5% (which is where we start running low space algorithms in the kernel).

I would like leave this to user.

Again: How is the user going to know what to set this to? What problem is this avoiding that requires the user to change this in any way.

Quoted 2 lines

When user is doing defrag on low free space system, it won’t cause
Problem to file system its self. At most the defrag fails during unshare when allocating blocks.

Why would we even allow a user to run defrag near ENOSPC? It is a well known problem that finding contiguous free space when we are close to ENOSPC is difficult and so defrag often is unable to improve the situation when we are within a few percent of the filesysetm being full.

It is also a well known problem that defragmentation at low free space trades off contiguous free space for fragmented free space. Hence when we are at low free space, defrag makes the free space fragmetnation worse, which then results in all allocation in the filesystem getting worse and more fragmented. This is something we absolutely should be trying to avoid.

This is one of the reasons xfs_fsr tries to layout the entire file before doing any IO - when about 95% full, it's common for the new layout to be worse than the original file's layout because there isn't sufficient contiguous free space to improve the layout.

IOWs, running defragmentation when we are above 95% full is actively harmful to the longevity of the filesystem. Hence, on a fundamental level, having a low space threshold in a defragmentation tool is simply wrong - defragmentation should simply not be run when the filesystem is anywhere near full.

.....

Quoted 11 lines
Quoted 10 lines
Quoted 4 lines

-s segment_size
The size limitation in bytes of segments. Minimium number is 4MiB, default number is 16MiB.

Why were these numbers chosen? What happens if the file has ~32MB sized extents and the user wants the file to be returned to a single large contiguous extent it possible? i.e. how is the user supposed to know how to set this for any given file without first having examined the exact pattern of fragmentations in the file?

Quoted 3 lines

Why customer want the file to be returned to a single large contiguous extent? A 32MB extent is pretty good to me. I didn’t here any customer complain about 32MB extents…

There's a much wider world out there than just Oracle customers. Just because you aren't aware of other use cases that exist, it doesn't mean they don't exist. I know they exist, hence my question.

For example, extent size hints are used to guarantee that the data is aligned to the underlying storage correctly, and very large contiguous extents are required to avoid excessive seeks during sequential reads that result in critical SLA failures. Hence if a file is poorly laid out in this situation, defrag needs to return it to as few, maximally sized extents as it can. How does a user know what they'd need to set this segment size field to and so acheive the result they need?

Quoted 6 lines

And you know, whether we can defrag extents to a large one depends on not only the tool it’s self. It’s depends on the status of the filesystem, say if the filesystem is very fragmented too, say the AG size..

The 16MB is selected according our tests basing on a customer metadump. With 16MB segment size, The the defrag result is very good and the IO latency is acceptable too. With the default 16MB segment Size, 32MB extent is excluded from defrag.

Exactly my point: you have written a solution that works for a single filesystem in a single environment. However, the solution is so specific to the single problem you need to solve that it is not clear whether that functionality or defaults are valid outside of the specific problem case you've written it for and tested it on.

Quoted 1 line

If you have better default size, we can use that.

I'm not convinced that fixed size "segments" is even the right way to approach this problem. What needs to be done is dependent on the extent layout of the file, not how extents fit over some arbitrary fixed segment map....

Quoted 22 lines
Quoted 12 lines
Quoted 11 lines

We tested with real customer metadump with some different 'idle_time's and found 250ms is good pratice sleep time. Here comes some number of the test:

Test: running of defrag on the image file which is used for the back end of a block device in a
virtual machine. At the same time, fio is running at the same time inside virtual machine on that block device.

block device type:   NVME
File size:           200GiB
paramters to defrag: free_space: 1024 idle_time: 250 First_extent_share: enabled readahead: disabled
Defrag run time:     223 minutes
Number of extents:   6745489(before) -> 203571(after)
Quoted 2 lines

So and average extent size of ~32kB before, 100MB after? How much of these are shared extents?

Zero shared extents, but there are some unwritten ones.
A similar run stats is like this:
Pre-defrag 6654460 extents detected, 112228 are "unwritten",0 are "shared”
Tried to defragment 6393352 extents (181000359936 bytes) in 26032 segments Time stats(ms): max clone: 31, max unshare: 300, max punch_hole: 66 Post-defrag 282659 extents detected

Quoted 1 line
Quoted 5 lines
Quoted 3 lines

Runtime is 13380secs, so if we copied 200GiB in that time, the defrag ran at 16MB/s. That's not very fast.

We are chasing the balance of defrag and parallel IO latency.

My point is that stuff like CLONE and UNSHARE should be able to run much, much faster than this, even if some of the time is left idle for other IO.

i.e. we can clone extents at about 100,000/s. We can copy data through the page cache at 7-8GB/s on NVMe devices.

A full clone of the 6.6 million extents should only take about a minute.

A full page cache copy of the 200GB cloned file (i.e. via read/write syscalls) should easily run at >1GB/s, and so only take a couple of minutes to run.

IOWs, the actual IO and metadata modification side of things is really only about 5 minutes worth of CPU and IO.

Hence this defrag operation is roughly 100x slower than we should be able to run it at. We should be able to run it at close to those speeds whilst still allowing concurrent read access to the file.

Quoted 7 lines

If an admin then wants it to run at 16MB/s, it can be throttled to that speed using cgroups, ionice, etc.

i.e. I think you are trying to solve too many unnecessary problems here and not addressing the one thing it should do: defrag a file as fast and efficiently as possible.

Thanks for all above replies.
For the performance, I am still thinking that the bottle neck is at the synchronous page by page disk reading.

Yes, I have to address/workaround something in kernel.
Let’s expect the related kernel fixings and then simplify the user space defrag code.