Skip to contentSkip to conversation

Allow up to 1GB logs in mkfs.xfs

RFC PATCH In review David Chinner 4 participants 8 messages Feb 21, 2008 – Feb 28, 2008

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/2 Allow up to 1GB logs in mkfs.xfs

Increase the maximum log size supported by mkfs.

The log size can be increased easily in mkfs by changing a few defines and a couple of types to allow the log size to increase to 1GB. Theoretically, the log size can be made much, much larger than this (up to 2^32 sectors), but going beyond 2^30 *bytes* causes integer overflow issues in the kernel log code.

e.g. a 2GB log (2^31 bytes) will not mount because the space calculations in the kernel use "int" types and overflow:

STATIC int
xlog_space_left(xlog_t *log, int cycle, int bytes)

Hence logs larger than 2^30 will not work without kernel modifications. Therefore this change is limited to increasing the log size to what we can currently support in kernel space with needing kernel modifications.

---
 xfsprogs/include/xfs_fs.h |    8 ++++----
 xfsprogs/mkfs/xfs_mkfs.c  |   14 +++++++-------
 2 files changed, 11 insertions(+), 11 deletions(-)
xfsprogs/include/xfs_fs.h +4 -4
Index: xfs-cmds/xfsprogs/include/xfs_fs.h
===================================================================
@@ -248,10 +248,10 @@ typedef struct xfs_fsop_resblks {
  * Minimum and maximum sizes need for growth checks
  */
 #define XFS_MIN_AG_BLOCKS	64
-#define XFS_MIN_LOG_BLOCKS	512
-#define XFS_MAX_LOG_BLOCKS	(64 * 1024)
-#define XFS_MIN_LOG_BYTES	(256 * 1024)
-#define XFS_MAX_LOG_BYTES	(128 * 1024 * 1024)
+#define XFS_MIN_LOG_BLOCKS	512ULL
+#define XFS_MAX_LOG_BLOCKS	(1024 * 1024ULL)
+#define XFS_MIN_LOG_BYTES	(10 * 1024 * 1024ULL)
+#define XFS_MAX_LOG_BYTES	(1 * 1024 * 1024 * 1024ULL)
 
 /*
  * Structures for XFS_IOC_FSGROWFSDATA, XFS_IOC_FSGROWFSLOG & XFS_IOC_FSGROWFSRT
xfsprogs/mkfs/xfs_mkfs.c +7 -7
Index: xfs-cmds/xfsprogs/mkfs/xfs_mkfs.c
===================================================================
@@ -362,13 +362,13 @@ validate_log_size(__uint64_t logblocks, 
 	}
 	if (logblocks > XFS_MAX_LOG_BLOCKS) {
 		fprintf(stderr,
-	_("log size %lld blocks too large, maximum size is %d blocks\n"),
+	_("log size %lld blocks too large, maximum size is %lld blocks\n"),
 			(long long)logblocks, XFS_MAX_LOG_BLOCKS);
 		usage();
 	}
 	if ((logblocks << blocklog) > XFS_MAX_LOG_BYTES) {
 		fprintf(stderr,
-	_("log size %lld bytes too large, maximum size is %d bytes\n"),
+	_("log size %lld bytes too large, maximum size is %lld bytes\n"),
 			(long long)(logblocks << blocklog), XFS_MAX_LOG_BYTES);
 		usage();
 	}
@@ -1728,7 +1728,7 @@ reported by the device (%u).\n"),
 	min_logblocks = max_tr_res * XFS_MIN_LOG_FACTOR;
 	min_logblocks = MAX(XFS_MIN_LOG_BLOCKS, min_logblocks);
 	if (!logsize && dblocks >= (1024*1024*1024) >> blocklog)
-		min_logblocks = MAX(min_logblocks, (10*1024*1024)>>blocklog);
+		min_logblocks = MAX(min_logblocks, XFS_MIN_LOG_BYTES>>blocklog);
 	if (logsize && xi.logBBsize > 0 && logblocks > DTOBT(xi.logBBsize)) {
 		fprintf(stderr,
 _("size %s specified for log subvolume is too large, maximum is %lld blocks\n"),
@@ -1748,10 +1748,10 @@ _("size %s specified for log subvolume i
 		logblocks = 0;
 	} else if (loginternal && !logsize) {
 		/*
-		 * logblocks grows from min_logblocks to XFS_MAX_LOG_BLOCKS
-		 * at 128GB
-		 *
-		 * 2048 = 128GB / MAX_LOG_BYTES
+		 * With a 1GB max log size, default to maximum size
+		 * at 2TB. This keeps the same ratio from the older
+		 * max log size of 128M at 256GB fs size. IOWs,
+		 * the ratio of fs size to log size is 2048:1.
 		 */
 		logblocks = (dblocks << blocklog) / 2048;
 		logblocks = logblocks >> blocklog;
#1 lore raw
Quoted 12 lines · David Chinner

Increase the maximum log size supported by mkfs.

The log size can be increased easily in mkfs by changing a few defines and a couple of types to allow the log size to increase to 1GB. Theoretically, the log size can be made much, much larger than this (up to 2^32 sectors), but going beyond 2^30 *bytes* causes integer overflow issues in the kernel log code.

e.g. a 2GB log (2^31 bytes) will not mount because the space calculations in the kernel use "int" types and overflow:

Quoted 7 lines

STATIC int
xlog_space_left(xlog_t *log, int cycle, int bytes)

Hence logs larger than 2^30 will not work without kernel modifications. Therefore this change is limited to increasing the log size to what we can currently support in kernel space with needing kernel modifications.

I'm glad you got time to get around this, I didn't include it in the first batch as I was told it 'broke things'.

Looks good to me.

Cheers,

Signature · 1 line
Niv Sardi
#2 lore raw
Quoted 2 lines · Niv Sardi
Quoted 1 line · David Chinner

Increase the maximum log size supported by mkfs.

....

Quoted 7 lines
Quoted 4 lines

Hence logs larger than 2^30 will not work without kernel modifications. Therefore this change is limited to increasing the log size to what we can currently support in kernel space with needing kernel modifications.

I'm glad you got time to get around this, I didn't include it in the first batch as I was told it 'broke things'.

Right. There's all sorts of nasties lurking if we go over 1GB in size, but AFAICT up to 1GB is OK. There's still lots of validation needing to be done - that's why this is an "RFC" and not something ready for checkin yet. That's where more eye's and testers are handy...

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group
#3 lore raw
Quoted 8 lines · David Chinner
Quoted 2 lines · Niv Sardi
Quoted 1 line · David Chinner

Increase the maximum log size supported by mkfs.

....

Quoted 4 lines
Quoted 4 lines

Hence logs larger than 2^30 will not work without kernel modifications. Therefore this change is limited to increasing the log size to what we can currently support in kernel space with needing kernel modifications.

Quoted 9 lines
Quoted 2 lines

I'm glad you got time to get around this, I didn't include it in the first batch as I was told it 'broke things'.

Right. There's all sorts of nasties lurking if we go over 1GB in size, but AFAICT up to 1GB is OK. There's still lots of validation needing to be done - that's why this is an "RFC" and not something ready for checkin yet. That's where more eye's and testers are handy...

Dave, have you run any tests to see if we get any speed-ups with this change?

#4 lore raw
Quoted 12 lines · Lachlan McIlroy
Quoted 11 lines · David Chinner
Quoted 2 lines · Niv Sardi
Quoted 1 line · David Chinner

Increase the maximum log size supported by mkfs.

....

Quoted 6 lines
Quoted 4 lines

Hence logs larger than 2^30 will not work without kernel modifications. Therefore this change is limited to increasing the log size to what we can currently support in kernel space with needing kernel modifications.

I'm glad you got time to get around this, I didn't include it in the first batch as I was told it 'broke things'.

Quoted 8 lines
Quoted 5 lines

Right. There's all sorts of nasties lurking if we go over 1GB in size, but AFAICT up to 1GB is OK. There's still lots of validation needing to be done - that's why this is an "RFC" and not something ready for checkin yet. That's where more eye's and testers are handy...

Dave, have you run any tests to see if we get any speed-ups with this change?

Not yet - I've been running it through QA with different log sizes first to see if there's anything obviously broken with the kernel w.r.t. larger log sizes (nothing wrong so far).

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group
#5 lore raw
Quoted 11 lines · David Chinner
Quoted 7 lines · Lachlan McIlroy
Quoted 6 lines · David Chinner
Quoted 2 lines · Niv Sardi
Quoted 1 line · David Chinner

Increase the maximum log size supported by mkfs.

....

Quoted 2 lines
Quoted 2 lines

Hence logs larger than 2^30 will not work without kernel modifications.

Therefore this change is limited to increasing the

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

log size to what we can currently support in kernel space with needing kernel modifications.

Does anyone know of any work in mainline to address this?

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

I'm glad you got time to get around this, I didn't include it in the first batch as I was told it 'broke things'.

Right. There's all sorts of nasties lurking if we go over 1GB in size, but AFAICT up to 1GB is OK. There's still lots of validation needing to be done - that's why this is an "RFC" and not something ready for checkin yet. That's where more eye's and testers are handy...

Quoted 6 lines
Quoted 2 lines

Dave, have you run any tests to see if we get any speed-ups with this change?

Not yet - I've been running it through QA with different log sizes first to see if there's anything obviously broken with the kernel w.r.t. larger log sizes (nothing wrong so far).

What sort of improvements are we expecting? Log bound benchmarks will tail-push later than they do now, wont they? Which particular benchmarks are you thinking of running?

Thanks

Signature · 4 lines
 Mark Goodwin                                  [email protected]
 Engineering Manager for XFS and PCP    Phone: +61-3-99631937
 SGI Australian Software Group           Cell: +61-4-18969583
-------------------------------------------------------------
#6 lore raw
Quoted 26 lines · Mark Goodwin
Quoted 11 lines · David Chinner
Quoted 7 lines · Lachlan McIlroy
Quoted 6 lines · David Chinner
Quoted 2 lines · Niv Sardi
Quoted 1 line · David Chinner

Increase the maximum log size supported by mkfs.

....

Quoted 2 lines
Quoted 2 lines

Hence logs larger than 2^30 will not work without kernel modifications.

Therefore this change is limited to increasing the

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

log size to what we can currently support in kernel space with needing kernel modifications.

Does anyone know of any work in mainline to address this?

Quoted 10 lines
Quoted 9 lines
Quoted 7 lines
Quoted 2 lines

I'm glad you got time to get around this, I didn't include it in the first batch as I was told it 'broke things'.

Right. There's all sorts of nasties lurking if we go over 1GB in size, but AFAICT up to 1GB is OK. There's still lots of validation needing to be done - that's why this is an "RFC" and not something ready for checkin yet. That's where more eye's and testers are handy...

Dave, have you run any tests to see if we get any speed-ups with this change?

Quoted 8 lines
Quoted 3 lines

Not yet - I've been running it through QA with different log sizes first to see if there's anything obviously broken with the kernel w.r.t. larger log sizes (nothing wrong so far).

What sort of improvements are we expecting? Log bound benchmarks will tail-push later than they do now, wont they? Which particular benchmarks are you thinking of running?

Does a larger on-disk log mean we can have a lot more items in the AIL?

#7 lore raw
Quoted 1 line · Lachlan McIlroy

Does a larger on-disk log mean we can have a lot more items in the AIL?

Yes.

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group