GFS2: Change minimum reservation size to 64 blocks

Hi,

I've been working on the problems of both slow performance and excessive file fragmentation, especially with regard to the RHEL7 kernel. I've created some patches (still forthcoming) to improve these things. For a long time, I thought the fragmentation issue was related to GFS2 not using its reservations efficiently. I wrote a patch that revises the calculations in function gfs2_write_calc_reserv, but it didn't seem to help. I also thought it might be releasing the reservations too early, but that turned out not to be the case. I've done extensive performance testing and determined that, even under the best circumstances (IOW by minimizing both inter and intra-node resource group contention), file fragmentation can be improved significantly by doubling the minimum block reservation size from 32 to 64 blocks. To test this, I did several two+ hour tests (real-life application) that uses lots of files in GFS2. After the test is complete, I totaled all the file extents (fragments) using filefrag.

Here are the results of five two-hour runs performed with new, revised calculations, and today's 32-block reservations:

EXTENT COUNT FOR OUTPUT FILES =  444352
EXTENT COUNT FOR OUTPUT FILES =  468425
EXTENT COUNT FOR OUTPUT FILES =  472464
EXTENT COUNT FOR OUTPUT FILES =  482224
EXTENT COUNT FOR OUTPUT FILES =  481174

Here are the results of five runs performed with the old calculations and 64-block reservations:

EXTENT COUNT FOR OUTPUT FILES =  365743
EXTENT COUNT FOR OUTPUT FILES =  398628
EXTENT COUNT FOR OUTPUT FILES =  404614
EXTENT COUNT FOR OUTPUT FILES =  401355
EXTENT COUNT FOR OUTPUT FILES =  384599

As you can see, by doubling the minimum reservation size, the files have about 20 percent fewer extents.

This patch, therefore, doubles the minimum block reservations.

Incidentally, if we don't take measures to minimize resource group contention (both inter and intra-node) the results are much worse. Here is the same test, done on a "stock" RHEL7 kernel:

EXTENT COUNT FOR OUTPUT FILES =  826314
EXTENT COUNT FOR OUTPUT FILES =  791406
EXTENT COUNT FOR OUTPUT FILES =  735760

Patch description:

The minimum block reservation size was 32 blocks. This patch doubles it to 64 blocks.

Regards,

Bob Peterson
Red Hat File Systems

---

index 5d8f085..e2058a7 100644
@@ -14,11 +14,9 @@
 #include <linux/uaccess.h>
 
 /* Since each block in the file system is represented by two bits in the
- * bitmap, one 64-bit word in the bitmap will represent 32 blocks.
- * By reserving 32 blocks at a time, we can optimize / shortcut how we search
- * through the bitmaps by looking a word at a time.
+ * bitmap, each 64-bit word in the bitmap represents 32 blocks.
  */
-#define RGRP_RSRV_MINBYTES 8
+#define RGRP_RSRV_MINBYTES 16 /* Reserve 64 blocks */
 #define RGRP_RSRV_MINBLKS ((u32)(RGRP_RSRV_MINBYTES * GFS2_NBBY))
 
 struct gfs2_rgrpd;

Hi,

Whether that makes sense to do or not depends upon the distribution of the file sizes in the workload, and also the way in which the files are written. I don't think there is any guarantee that a different workload would not produce a different result here. Just out of interest, what is the distribution of file sizes in this workload?

I'd much prefer to see an algorithm that is adaptive, rather than simply bumping up the default here. We do need to be able to cope with cases where the files are much smaller, and without adaptive sizing, we might land up creating small holes in the allocations, which would then cause problems for later allocations,

Steve.