Allow up to 1GB logs in mkfs.xfs
RFC PATCH 1/2 Allow up to 1GB logs in mkfs.xfs
Increase the maximum log size supported by mkfs.
The log size can be increased easily in mkfs by changing a few defines and a couple of types to allow the log size to increase to 1GB. Theoretically, the log size can be made much, much larger than this (up to 2^32 sectors), but going beyond 2^30 *bytes* causes integer overflow issues in the kernel log code.
e.g. a 2GB log (2^31 bytes) will not mount because the space calculations in the kernel use "int" types and overflow:
STATIC intxlog_space_left(xlog_t *log, int cycle, int bytes)
Hence logs larger than 2^30 will not work without kernel modifications. Therefore this change is limited to increasing the log size to what we can currently support in kernel space with needing kernel modifications.
--- xfsprogs/include/xfs_fs.h | 8 ++++---- xfsprogs/mkfs/xfs_mkfs.c | 14 +++++++------- 2 files changed, 11 insertions(+), 11 deletions(-)
xfsprogs/include/xfs_fs.h +4 -4
@@ -248,10 +248,10 @@ typedef struct xfs_fsop_resblks {* Minimum and maximum sizes need for growth checks */ #define XFS_MIN_AG_BLOCKS 64 -#define XFS_MIN_LOG_BLOCKS 512 -#define XFS_MAX_LOG_BLOCKS (64 * 1024) -#define XFS_MIN_LOG_BYTES (256 * 1024) -#define XFS_MAX_LOG_BYTES (128 * 1024 * 1024) +#define XFS_MIN_LOG_BLOCKS 512ULL +#define XFS_MAX_LOG_BLOCKS (1024 * 1024ULL) +#define XFS_MIN_LOG_BYTES (10 * 1024 * 1024ULL) +#define XFS_MAX_LOG_BYTES (1 * 1024 * 1024 * 1024ULL)/* * Structures for XFS_IOC_FSGROWFSDATA, XFS_IOC_FSGROWFSLOG & XFS_IOC_FSGROWFSRT
xfsprogs/mkfs/xfs_mkfs.c +7 -7
@@ -362,13 +362,13 @@ validate_log_size(__uint64_t logblocks, }if (logblocks >XFS_MAX_LOG_BLOCKS ) {fprintf (stderr, -_ ("log size %lld blocks too large, maximum size is %d blocks\n" ), +_ ("log size %lld blocks too large, maximum size is %lld blocks\n" ), (long long )logblocks,XFS_MAX_LOG_BLOCKS );usage (); }if ((logblocks << blocklog) >XFS_MAX_LOG_BYTES ) {fprintf (stderr, -_ ("log size %lld bytes too large, maximum size is %d bytes\n" ), +_ ("log size %lld bytes too large, maximum size is %lld bytes\n" ), (long long )(logblocks << blocklog),XFS_MAX_LOG_BYTES );usage (); }
@@ -1728,7 +1728,7 @@ reported by the device (%u).\n"), min_logblocks = max_tr_res *XFS_MIN_LOG_FACTOR ; min_logblocks =MAX (XFS_MIN_LOG_BLOCKS , min_logblocks);if (!logsize && dblocks >= (1024 *1024 *1024 ) >> blocklog) - min_logblocks =MAX (min_logblocks, (10 *1024 *1024 )>>blocklog); + min_logblocks =MAX (min_logblocks,XFS_MIN_LOG_BYTES >>blocklog);if (logsize && xi.logBBsize >0 && logblocks >DTOBT (xi.logBBsize)) { fprintf(stderr, _("size %s specified for log subvolume is too large, maximum is %lld blocks\n"),
@@ -1748,10 +1748,10 @@ _("size %s specified for log subvolume i logblocks =0 ; }else if (loginternal && !logsize) {/* -* logblocks grows from min_logblocks to XFS_MAX_LOG_BLOCKS -* at 128GB -* -* 2048 = 128GB / MAX_LOG_BYTES +* With a 1GB max log size, default to maximum size +* at 2TB. This keeps the same ratio from the older +* max log size of 128M at 256GB fs size. IOWs, +* the ratio of fs size to log size is 2048:1. */ logblocks = (dblocks << blocklog) /2048 ; logblocks = logblocks >> blocklog;
Quoted 12 lines · David Chinner
Increase the maximum log size supported by mkfs.
The log size can be increased easily in mkfs by changing a few defines and a couple of types to allow the log size to increase to 1GB. Theoretically, the log size can be made much, much larger than this (up to 2^32 sectors), but going beyond 2^30 *bytes* causes integer overflow issues in the kernel log code.
e.g. a 2GB log (2^31 bytes) will not mount because the space calculations in the kernel use "int" types and overflow:
Quoted 7 lines
STATIC int
xlog_space_left(xlog_t *log, int cycle, int bytes)Hence logs larger than 2^30 will not work without kernel modifications. Therefore this change is limited to increasing the log size to what we can currently support in kernel space with needing kernel modifications.
I'm glad you got time to get around this, I didn't include it in the first batch as I was told it 'broke things'.
Looks good to me.
Cheers,
Signature · 1 line
Niv Sardi
Quoted 2 lines · Niv Sardi
Quoted 1 line · David Chinner
Increase the maximum log size supported by mkfs.
....
Quoted 7 lines
Quoted 4 lines
Hence logs larger than 2^30 will not work without kernel modifications. Therefore this change is limited to increasing the log size to what we can currently support in kernel space with needing kernel modifications.
I'm glad you got time to get around this, I didn't include it in the first batch as I was told it 'broke things'.
Right. There's all sorts of nasties lurking if we go over 1GB in size, but AFAICT up to 1GB is OK. There's still lots of validation needing to be done - that's why this is an "RFC" and not something ready for checkin yet. That's where more eye's and testers are handy...
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Quoted 8 lines · David Chinner
Quoted 2 lines · Niv Sardi
Quoted 1 line · David Chinner
Increase the maximum log size supported by mkfs.
....
Quoted 4 lines
Quoted 4 lines
Hence logs larger than 2^30 will not work without kernel modifications. Therefore this change is limited to increasing the log size to what we can currently support in kernel space with needing kernel modifications.
Quoted 9 lines
Quoted 2 lines
I'm glad you got time to get around this, I didn't include it in the first batch as I was told it 'broke things'.
Right. There's all sorts of nasties lurking if we go over 1GB in size, but AFAICT up to 1GB is OK. There's still lots of validation needing to be done - that's why this is an "RFC" and not something ready for checkin yet. That's where more eye's and testers are handy...
Dave, have you run any tests to see if we get any speed-ups with this change?
Quoted 12 lines · Lachlan McIlroy
Quoted 11 lines · David Chinner
Quoted 2 lines · Niv Sardi
Quoted 1 line · David Chinner
Increase the maximum log size supported by mkfs.
....
Quoted 6 lines
Quoted 4 lines
Hence logs larger than 2^30 will not work without kernel modifications. Therefore this change is limited to increasing the log size to what we can currently support in kernel space with needing kernel modifications.
I'm glad you got time to get around this, I didn't include it in the first batch as I was told it 'broke things'.
Quoted 8 lines
Quoted 5 lines
Right. There's all sorts of nasties lurking if we go over 1GB in size, but AFAICT up to 1GB is OK. There's still lots of validation needing to be done - that's why this is an "RFC" and not something ready for checkin yet. That's where more eye's and testers are handy...
Dave, have you run any tests to see if we get any speed-ups with this change?
Not yet - I've been running it through QA with different log sizes first to see if there's anything obviously broken with the kernel w.r.t. larger log sizes (nothing wrong so far).
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Quoted 11 lines · David Chinner
Quoted 7 lines · Lachlan McIlroy
Quoted 6 lines · David Chinner
Quoted 2 lines · Niv Sardi
Quoted 1 line · David Chinner
Increase the maximum log size supported by mkfs.
....
Quoted 2 lines
Quoted 2 lines
Hence logs larger than 2^30 will not work without kernel modifications.
Therefore this change is limited to increasing the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
log size to what we can currently support in kernel space with needing kernel modifications.
Does anyone know of any work in mainline to address this?
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
I'm glad you got time to get around this, I didn't include it in the first batch as I was told it 'broke things'.
Right. There's all sorts of nasties lurking if we go over 1GB in size, but AFAICT up to 1GB is OK. There's still lots of validation needing to be done - that's why this is an "RFC" and not something ready for checkin yet. That's where more eye's and testers are handy...
Quoted 6 lines
Quoted 2 lines
Dave, have you run any tests to see if we get any speed-ups with this change?
Not yet - I've been running it through QA with different log sizes first to see if there's anything obviously broken with the kernel w.r.t. larger log sizes (nothing wrong so far).
What sort of improvements are we expecting? Log bound benchmarks will tail-push later than they do now, wont they? Which particular benchmarks are you thinking of running?
Thanks
Signature · 4 lines
Mark Goodwin [email protected] Engineering Manager for XFS and PCP Phone: +61-3-99631937 SGI Australian Software Group Cell: +61-4-18969583 -------------------------------------------------------------
Quoted 26 lines · Mark Goodwin
Quoted 11 lines · David Chinner
Quoted 7 lines · Lachlan McIlroy
Quoted 6 lines · David Chinner
Quoted 2 lines · Niv Sardi
Quoted 1 line · David Chinner
Increase the maximum log size supported by mkfs.
....
Quoted 2 lines
Quoted 2 lines
Hence logs larger than 2^30 will not work without kernel modifications.
Therefore this change is limited to increasing the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
log size to what we can currently support in kernel space with needing kernel modifications.
Does anyone know of any work in mainline to address this?
Quoted 10 lines
Quoted 9 lines
Quoted 7 lines
Quoted 2 lines
I'm glad you got time to get around this, I didn't include it in the first batch as I was told it 'broke things'.
Right. There's all sorts of nasties lurking if we go over 1GB in size, but AFAICT up to 1GB is OK. There's still lots of validation needing to be done - that's why this is an "RFC" and not something ready for checkin yet. That's where more eye's and testers are handy...
Dave, have you run any tests to see if we get any speed-ups with this change?
Quoted 8 lines
Quoted 3 lines
Not yet - I've been running it through QA with different log sizes first to see if there's anything obviously broken with the kernel w.r.t. larger log sizes (nothing wrong so far).
What sort of improvements are we expecting? Log bound benchmarks will tail-push later than they do now, wont they? Which particular benchmarks are you thinking of running?
Does a larger on-disk log mean we can have a lot more items in the AIL?
Quoted 1 line · Lachlan McIlroy
Does a larger on-disk log mean we can have a lot more items in the AIL?
Yes.
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group