Patches to reduce GFS2 fragmentation

Hi,

This is a revised version of my patches to improve GFS2 performance and reduce GFS2 fragmentation. As suggested by Steve Whitehouse, the first patch has been modified to base its distribution of resource groups on the number of journals. It also corrects the criteria on the final loop, changing "loops < 3" back to "loops < 2" as it should be. Also, I re-tested to make sure nothing had regressed. I updated the timing numbers below.

Also as suggested, the patch that kept file close from deleting rgrp reservations has been removed from the set, leaving only three patches. The removal of that patch caused fragmentation to climb 15 percent, but it's still much (41 percent) better than the original stock kernels. I agree that use cases like untar would make for much worse fragmentation problems if that patch were to remain.

I've re-tested with these three patches and found that performance and fragmentation are still good. Here are numbers:

STOCK RHEL7 kernel (none of these patches):

Run times:
Run 1 time: 2hr 40min 33sec
Run 2 time: 2hr 39min 52sec
Run 3 time: 2hr 39min 31sec
Run 4 time: 2hr 33min 57sec
Run 5 time: 2hr 41min 6sec

Total file extents (File fragmentation):

EXTENT COUNT FOR OUTPUT FILES =  744708
EXTENT COUNT FOR OUTPUT FILES =  749868
EXTENT COUNT FOR OUTPUT FILES =  721862
EXTENT COUNT FOR OUTPUT FILES =  635301
EXTENT COUNT FOR OUTPUT FILES =  689263

The times are bad and the fragmentation level is also bad. If I add just the first patch, "GFS2: Set of distributed preferences for rgrps" the performance improves, but the fragmentation is worse (I only did three iterations this time):

Run times:
Run 1 time: 2hr 2min 47sec
Run 2 time: 2hr 8min 37sec
Run 3 time: 2hr 10min 0sec

Total file extents (File fragmentation):

EXTENT COUNT FOR OUTPUT FILES =  1011217
EXTENT COUNT FOR OUTPUT FILES =  1025973
EXTENT COUNT FOR OUTPUT FILES =  1070163

With these three patches, both performance and fragmentation are good:

Run 1 time: 2hr 11min 11sec
Run 2 time: 2hr 8min 57sec
Run 3 time: 2hr 10min 43sec
Run 4 time: 2hr 12min 50sec
Run 5 time: 2hr 15min 32sec
Run 6 time: 2hr 10min 58sec

EXTENT COUNT FOR OUTPUT FILES =  424329
EXTENT COUNT FOR OUTPUT FILES =  430663
EXTENT COUNT FOR OUTPUT FILES =  430186
EXTENT COUNT FOR OUTPUT FILES =  421635
EXTENT COUNT FOR OUTPUT FILES =  416706
EXTENT COUNT FOR OUTPUT FILES =  418887

Regards,

Bob Peterson
Red Hat File Systems

---
Bob Peterson (3):
GFS2: Set of distributed preferences for rgrps
GFS2: Only increase rs_sizehint
GFS2: If we use up our block reservation, request more next time

 fs/gfs2/file.c   |  3 ++-
 fs/gfs2/incore.h |  1 +
 fs/gfs2/rgrp.c   | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++++----
 fs/gfs2/rgrp.h   |  1 +
 4 files changed, 68 insertions(+), 6 deletions(-)

1.9.3

This patch tries to use the journal numbers to evenly distribute which node prefers which resource group for block allocations. This is to help performance.

---
 fs/gfs2/incore.h |  1 +
 fs/gfs2/rgrp.c   | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 62 insertions(+), 5 deletions(-)

index 39e7e99..1b89918 100644
@@ -97,6 +97,7 @@ struct gfs2_rgrpd {
 #define GFS2_RDF_CHECK		0x10000000 /* check for unlinked inodes */
 #define GFS2_RDF_UPTODATE	0x20000000 /* rg is up to date */
 #define GFS2_RDF_ERROR		0x40000000 /* error in rg */
+#define GFS2_RDF_PREFERRED	0x80000000 /* This rgrp is preferred */
 #define GFS2_RDF_MASK		0xf0000000 /* mask for internal flags */
 	spinlock_t rd_rsspin;           /* protects reservation related vars */
 	struct rb_root rd_rstree;       /* multi-block reservation tree */

1.9.3

If an application does a sequence of (1) big write, (2) little write we don't necessarily want to reset the size hint based on the smaller size. The fact that they did any big writes implies they may do more, and therefore we should try to allocate bigger block reservations, even if the last few were small writes. Therefore this patch changes function gfs2_size_hint so that the size hint can only grow; it cannot shrink. This is especially important where there are multiple writers.

---
 fs/gfs2/file.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

index 80dd44d..5ebe568 100644
@@ -337,7 +337,8 @@ static void gfs2_size_hint(struct file *filep, loff_t offset, size_t size)
 	size_t blks = (size + sdp->sd_sb.sb_bsize - 1) >> sdp->sd_sb.sb_bsize_shift;
 	int hint = min_t(size_t, INT_MAX, blks);
 
-	atomic_set(&ip->i_res->rs_sizehint, hint);
+	if (hint > atomic_read(&ip->i_res->rs_sizehint))
+		atomic_set(&ip->i_res->rs_sizehint, hint);
 }
 
 /**

1.9.3

If we run out of blocks for a given multi-block allocation, we obviously did not reserve enough. We should reserve more blocks for the next reservation to reduce fragmentation. This patch increases the size hint for reservations when they run out.

---
 fs/gfs2/rgrp.c | 3 +++
 fs/gfs2/rgrp.h | 1 +
 2 files changed, 4 insertions(+)

index f4e4a0c..9150207 100644
@@ -2251,6 +2251,9 @@ static void gfs2_adjust_reservation(struct gfs2_inode *ip,
 			trace_gfs2_rs(rs, TRACE_RS_CLAIM);
 			if (rs->rs_free && !ret)
 				goto out;
+			/* We used up our block reservation, so we should
+			   reserve more blocks next time. */
+			atomic_add(RGRP_RSRV_ADDBLKS, &rs->rs_sizehint);
 		}
 		__rs_deltree(rs);
 	}

index 5d8f085..b104f4a 100644
@@ -20,6 +20,7 @@
  */
 #define RGRP_RSRV_MINBYTES 8
 #define RGRP_RSRV_MINBLKS ((u32)(RGRP_RSRV_MINBYTES * GFS2_NBBY))
+#define RGRP_RSRV_ADDBLKS 64
 
 struct gfs2_rgrpd;
 struct gfs2_sbd;

1.9.3