xfs_repair speedup changes

Hi everyone,

agami Systems started on a project to speed up xfs_repair before we knew that SGI was working on the same task. Similar to SGI's solution, our approach uses readahead to shorten the runtime. Agami also wanted to change the existing code as little as possible.

By releasing this patch, we hope to start a discussion which will lead to continued improvements in xfs_repair runtimes. Our patch has a couple of ideas which should benefit SGI's code. Using our NAS platform which has 4 CPUs and runs XFS over software RAID5, we have seen 5 to 8 times speedup, depending on resources allocated to a run. The test filesystem had 1.4TB of data with 24M files. Unfortunately, I have not been able to run the latest CVS code against our system due to kernel differences.

SGI's advantages
----------------
1. User space cache with maximum number of entries
a. means that xfs_repair will cause less interference
with other mounted filesystems.
b. allows tracking of cache behavior.
2. Rewrite phase7 to eliminate unnecessary transaction overhead.

agami's advantages
------------------
1. Doesn't depend on AIO & generic DIO working correctly. Will work with older linux kernels.
2. Parallelism model provides additional benefits
a. In phases 3 and 4, many threads can be used to prefetch
inode blocks regardless of AG count.
b. By processing one AG at a time, drives spend less time seeking
when multiple AGs are placed on a single drive due to the volume geometry.
c. By placing each prefetch in its own thread, more parallelism
is achieved especially when retrieving directory blocks.

Chandan Talukdar performed all the xfs_repair work over last summer. Because the work was done on an old base, I have ported it forward to a CVS date of May 17, 2006. I chose this date because it allows a cleaner patch to be delivered.

I would like to hear suggestions for how to proceed.

Michael Nishimoto

diff -Nru xfsprogs-old/include/builddefs.in xfsprogs-new2/include/builddefs.in
@@ -106,7 +106,7 @@
 
 GCFLAGS = $(OPTIMIZER) $(DEBUG) -funsigned-char -fno-strict-aliasing -Wall \
 	  -DVERSION=\"$(PKG_VERSION)\" -DLOCALEDIR=\"$(PKG_LOCALE_DIR)\"  \
-	  -DPACKAGE=\"$(PKG_NAME)\" -I$(TOPDIR)/include
+	  -DPACKAGE=\"$(PKG_NAME)\" -I$(TOPDIR)/include -pthread
 
 # First, Global, Platform, Local CFLAGS
 CFLAGS += $(FCFLAGS) $(GCFLAGS) $(PCFLAGS) $(LCFLAGS)

diff -Nru xfsprogs-old/libxfs/xfs.h xfsprogs-new2/libxfs/xfs.h
@@ -94,6 +94,7 @@
 #define xfs_itobp			libxfs_itobp
 #define xfs_ichgtime			libxfs_ichgtime
 #define xfs_bmapi			libxfs_bmapi
+#define xfs_bmapi_single		libxfs_bmapi_single
 #define xfs_bmap_finish			libxfs_bmap_finish
 #define xfs_bmap_del_free		libxfs_bmap_del_free
 #define xfs_bunmapi			libxfs_bunmapi

diff -Nru xfsprogs-old/repair/dino_chunks.c xfsprogs-new2/repair/dino_chunks.c
@@ -920,7 +920,38 @@
 	ino_tree_node_t *ino_rec, *first_ino_rec, *prev_ino_rec;
 
 	first_ino_rec = ino_rec = findfirst_inode_rec(agno);
+#ifdef PHASE_3_4
+	ino_tree_node_t *first_ra_rec, *ra_rec;
+	int iters = 0;
+	int chunklen = BBTOB(XFS_FSB_TO_BB(mp, XFS_IALLOC_BLOCKS(mp)));
+
+	first_ra_rec = ra_rec = first_ino_rec;
+	/*
+	 * before we start processing, insert 'rahead' number of
+	 * nodes into the read ahead queue.  call insert_nodes()
+	 * with readdirblks set to 1 implying that the directory
+	 * blocks of the directory inodes within this chunk should
+	 * also be read in.
+	 */
+	if (insert_nodes(rahead, agno, &first_ra_rec, &ra_rec, 1, chunklen))
+		do_error(_("failed to allocate memory.  aborting\n"));
+#endif
 	while (ino_rec != NULL)  {
+#ifdef PHASE_3_4
+		iters++;
+		/*
+		 * after a set of 'radelta' number of nodes have been processed,
+		 * insert another 'radelta' nodes into the read ahead queue.
+		 * call insert_nodes() with readdirblks set to 1 implying that
+		 * the directory blocks of the directory inodes within this
+		 * chunk should also be read in.
+		 */
+		if (iters % radelta == 0) {
+			if (insert_nodes(radelta, agno, &first_ra_rec,
+			                 &ra_rec, 1, chunklen))
+			  do_error(_("failed to allocate memory.  aborting\n"));
+		}
+#endif
 		/*
 		 * paranoia - step through inode records until we step
 		 * through a full allocation of inodes.  this could

diff -Nru xfsprogs-old/repair/globals.h xfsprogs-new2/repair/globals.h
@@ -193,4 +193,10 @@
 EXTERN __uint32_t	sb_unit;
 EXTERN __uint32_t	sb_width;
 
+/* Used to increase performance by doing readahead */
+
+EXTERN int		numthreads;
+EXTERN int		rahead;
+EXTERN int		radelta;
+
 #endif /* _XFS_REPAIR_GLOBAL_H */

diff -Nru xfsprogs-old/repair/Makefile xfsprogs-new2/repair/Makefile
@@ -14,13 +14,18 @@
 CFILES = agheader.c attr_repair.c avl.c avl64.c bmap.c dino_chunks.c \
 	dinode.c dir.c dir2.c dir_stack.c globals.c incore.c \
 	incore_bmc.c init.c incore_ext.c incore_ino.c io.c phase1.c \
-	phase2.c phase3.c phase4.c phase5.c phase6.c phase7.c rt.c sb.c \
-	scan.c versions.c xfs_repair.c
+	phase2.c phase3.c phase4.c phase5.c phase6.c phase7.c queue.c rt.c \
+	sb.c scan.c threads.c versions.c xfs_repair.c
 
-LLDLIBS = $(LIBXFS) $(LIBXLOG) $(LIBUUID)
+LLDLIBS = $(LIBXFS) $(LIBXLOG) $(LIBUUID) $(LIBPTHREAD)
 LTDEPENDENCIES = $(LIBXFS) $(LIBXLOG)
 LLDFLAGS = -static
 
+# PHASE_3_4: Enable read ahead for phase 3 and 4
+# PHASE_6: Enable read ahead for phase 6
+# PHASE_7: Enable read ahead for phase 7
+CFLAGS	+= -DPHASE_3_4 -DPHASE_6 -DPHASE_7
+
 default: $(LTCOMMAND)
 
 globals.o: globals.h

Hi Michael,

It's going to take a little bit of time for me to digest this patch and to see how it compares to the other work performed by us.

On the surface, it looks quite interesting and I'll benchmark and analyse the two to see how it compares and integrate the best solution for the majority of cases.

I'm confused on why the kernel should make any difference to running xfs_repair. You should be able to get the 2.8.18 xfsprogs tarball from the FTP site, compile and test it:
ftp://oss.sgi.com/projects/xfs/cmd_tars/xfsprogs_2.8.18-1.tar.gz

I'm currently working on converting the phase 2-5 block map to an extent based format which will improve memory consumption in addition to speeding it up in most cases.

The only other forseable change is trying to merge phase 3 and phase 6 with directory checking, but I'm not sure how practical/feasible this is and whether the amount of work will provide a significant performance increase.

Regards,
Barry.