dm-log.c: Work with devices with hard-sector-size > 512 bytes
RFC PATCH 1/1 dm-log.c: Work with devices with hard-sector-size > 512 bytes
Some testing here revealed that on-disk logs for dm-mirror devices don't work properly on devices with a hard-sector-size greater than 512 bytes (many zSeries/s390 systems have disks with a hard-sector-size of 4k).
This patch is suprisingly simple, and primarily changes from using hard- coded values of 1 sector to using the hard-sector-size reported in the target's "limits" structure.
Unfortunately, the on-disk bitset is hardcoded to be at sector 2, which is too small for any device with hard-sector-size > 1k. I've changed this offset to sector 64 (32k from start of device), which should be a safer choice. However, this necessitated a change of the disk-log version from 2 to 3, meaning the new code would not be compatible with any existing disk-logs. At the moment, I haven't added any code to deal with this scenario - I wanted to get some opinions on how to best handle this.
Thanks,
Signature · 95 lines
Kevin Corry [email protected] http://www.ibm.com/linux/ http://evms.sourceforge.net/ On-disk logs for dm-mirror devices are currently hard-coded to use 512 byte hard-sector-sizes. This patch fixes the disk-log code to use the log device's hard-sector-size where appropriate instead of 512. This also moves the actual bitset to offset 32k. As such, the log version has been incremented to 3, which is not backwards compatible with existing disk-logs. This also reallocates the clean_bits buffer. It's originally allocated in the core_ctr() routine, but is only as big as the actual bitset. Since we may read more from disk than just the size of the bitset, we need to reallocate the buffer to the appropriate size. Signed-Off-By: Kevin Corry <[email protected]> Index: 2.6.16-perfmon2-2/drivers/md/dm-log.c =================================================================== --- 2.6.16-perfmon2-2.orig/drivers/md/dm-log.c +++ 2.6.16-perfmon2-2/drivers/md/dm-log.c @@ -112,8 +112,8 @@ void dm_destroy_dirty_log(struct dirty_l /* * The on-disk version of the metadata. */ -#define MIRROR_DISK_VERSION 2 -#define LOG_OFFSET 2 +#define MIRROR_DISK_VERSION 3 +#define LOG_OFFSET 64 struct log_header { uint32_t magic; @@ -376,6 +376,8 @@ static int disk_ctr(struct dirty_log *lo size_t size; struct log_c *lc; struct dm_dev *dev; + unsigned int hardsect_size; + uint32_t *new_clean_bits; if (argc < 2 || argc > 3) { DMWARN("wrong number of arguments to disk mirror log"); @@ -393,19 +395,21 @@ static int disk_ctr(struct dirty_log *lo return r; } + hardsect_size = ti->limits.hardsect_size; + lc = (struct log_c *) log->context; lc->log_dev = dev; /* setup the disk header fields */ lc->header_location.bdev = lc->log_dev->bdev; lc->header_location.sector = 0; - lc->header_location.count = 1; + lc->header_location.count = hardsect_size >> SECTOR_SHIFT; /* * We can't read less than this amount, even though we'll * not be using most of this space. */ - lc->disk_header = vmalloc(1 << SECTOR_SHIFT); + lc->disk_header = vmalloc(hardsect_size); if (!lc->disk_header) goto bad; @@ -414,11 +418,23 @@ static int disk_ctr(struct dirty_log *lo lc->bits_location.sector = LOG_OFFSET; size = dm_round_up(lc->bitset_uint32_count * sizeof(uint32_t), - 1 << SECTOR_SHIFT); + hardsect_size); lc->bits_location.count = size >> SECTOR_SHIFT; + + /* Realloc the clean_bits buffer, since we may read more + * from disk than just the minimum size of the bitset. + */ + new_clean_bits = vmalloc(size); + if (!new_clean_bits) + goto bad; + vfree(lc->clean_bits); + lc->clean_bits = new_clean_bits; + memset(lc->clean_bits, -1, size); + return 0; bad: + vfree(lc->disk_header); dm_put_device(ti, lc->log_dev); core_dtr(log); return -ENOMEM;
Quoted 5 lines
Unfortunately, the on-disk bitset is hardcoded to be at sector 2, which is too small for any device with hard-sector-size > 1k. I've changed this offset to sector 64 (32k from start of device), which should be a safer choice. However, this necessitated a change of the disk-log
version
Quoted 3 lines
from 2 to 3, meaning the new code would not be compatible with any existing disk-logs. At the moment, I haven't added any code to deal with this scenario - I wanted to get some opinions on how to best handle
this.
Quoted 1 line
Would it really hurt to leave the sector offset just at 2? That way the bitset data starts at different byte offsets but that would be consistent with the device. A fix byte offset would only be needed if a device could change its hardsector size without loosing data, wouldn't it?
Regards,
Stefan Bader
Alasdair G Kergon
maintainerQuoted 3 lines · Kevin Corry
However, this necessitated a change of the disk-log version from 2 to 3, meaning the new code would not be compatible with any existing disk-logs.
Several points:
The patch should have no impact on existing users for whom everything is working OK - compatibility should be preserved.
People who need the patch are unable to use this code yet so there's no need to preserve compatibility for them.
We need to continue to support disk logs being loaded onto different machines/devices from where they were created.
If there's no alternative but to take the disruptive step of changing the header version number (not convinced of that yet) then we should fix other problems with the header at the same time e.g. LOG_OFFSET should only be used when creating a new log, not when reading an existing one (ie store it in the header); version should include compatibility mechanism (flags and/or major/minor) etc.
Alasdair
Signature · 1 line
[email protected]
Hi Stefan,
Quoted 9 lines · Stefan Bader
Quoted 7 lines
Unfortunately, the on-disk bitset is hardcoded to be at sector 2, which is too small for any device with hard-sector-size > 1k. I've changed this offset to sector 64 (32k from start of device), which should be a safer choice. However, this necessitated a change of the disk-log version from 2 to 3, meaning the new code would not be compatible with any existing disk-logs. At the moment, I haven't added any code to deal with this scenario - I wanted to get some opinions on how to best handle this.
Would it really hurt to leave the sector offset just at 2?
Do you mean 2 * hard-sector-size? As it's coded now, it will always be at offset 1k.
Quoted 3 lines
That way the bitset data starts at different byte offsets but that would be consistent with the device. A fix byte offset would only be needed if a device could change its hardsector size without loosing data, wouldn't it?
Yes, that would work. I guess I'm just not keen on placing metadata in a location that's dependent on some unrecorded property of the device. Having the log always start at a well-known location seems a lot simpler to me.
But, so we can compare, here's an updated patch that changes the bitset offset to (2 * hard-sector-size), which should be backwards compatible with existing mirror logs (assuming that almost all disks have 512 hard-sector-sizes).
Signature · 82 lines
Kevin Corry [email protected] http://www.ibm.com/linux/ http://evms.sourceforge.net/ On-disk logs for dm-mirror devices are currently hard-coded to use 512 byte hard-sector-sizes. This patch fixes the disk-log code to use the log device's hard-sector-size where appropriate instead of 512. This also reallocates the clean_bits buffer. It's originally allocated in the core_ctr() routine, but is only as big as the actual bitset. Since we may read more from disk than just the size of the bitset, we need to reallocate the buffer to the appropriate size. Signed-Off-By: Kevin Corry <[email protected]> Index: 2.6.16-perfmon2-2/drivers/md/dm-log.c =================================================================== --- 2.6.16-perfmon2-2.orig/drivers/md/dm-log.c +++ 2.6.16-perfmon2-2/drivers/md/dm-log.c @@ -376,6 +376,8 @@ static int disk_ctr(struct dirty_log *lo size_t size; struct log_c *lc; struct dm_dev *dev; + unsigned int hardsect_size; + uint32_t *new_clean_bits; if (argc < 2 || argc > 3) { DMWARN("wrong number of arguments to disk mirror log"); @@ -393,32 +395,46 @@ static int disk_ctr(struct dirty_log *lo return r; } + hardsect_size = ti->limits.hardsect_size; + lc = (struct log_c *) log->context; lc->log_dev = dev; /* setup the disk header fields */ lc->header_location.bdev = lc->log_dev->bdev; lc->header_location.sector = 0; - lc->header_location.count = 1; + lc->header_location.count = hardsect_size >> SECTOR_SHIFT; /* * We can't read less than this amount, even though we'll * not be using most of this space. */ - lc->disk_header = vmalloc(1 << SECTOR_SHIFT); + lc->disk_header = vmalloc(hardsect_size); if (!lc->disk_header) goto bad; /* setup the disk bitset fields */ lc->bits_location.bdev = lc->log_dev->bdev; - lc->bits_location.sector = LOG_OFFSET; + lc->bits_location.sector = LOG_OFFSET * (hardsect_size >> SECTOR_SHIFT); size = dm_round_up(lc->bitset_uint32_count * sizeof(uint32_t), - 1 << SECTOR_SHIFT); + hardsect_size); lc->bits_location.count = size >> SECTOR_SHIFT; + + /* Realloc the clean_bits buffer, since we may read more + * from disk than just the minimum size of the bitset. + */ + new_clean_bits = vmalloc(size); + if (!new_clean_bits) + goto bad; + vfree(lc->clean_bits); + lc->clean_bits = new_clean_bits; + memset(lc->clean_bits, -1, size); + return 0; bad: + vfree(lc->disk_header); dm_put_device(ti, lc->log_dev); core_dtr(log); return -ENOMEM;
Alasdair G Kergon
maintainerA couple of options:
Retain the existing on-disk format exactly, but adjust the offsets and sizes used in the reads and writes to avoid the problems (pre-reading other data if necessary to extend a write).
Add an option to create version 3 metadata storing the offset in the on-disk header but continue to use version 2 by default. Update userspace code to request version 3 (via new log parameter) for new mirror logs by default, but still permit version 2 for new logs. This way, people committed to existing logs can continue to use them (or schedule the update) and we maintain forwards and backwards compatibility for people that need that.
Alasdair
Signature · 1 line
[email protected]
Quoted 5 lines · Alasdair G Kergon
A couple of options:
Retain the existing on-disk format exactly, but adjust the offsets and sizes used in the reads and writes to avoid the problems (pre-reading other data if necessary to extend a write).
This method seems like it should be easy to implement. We could just extend the buffer for the disk-header and always read and write both the header and the log at the same time, and just point lc->clean_bits at the appropriate offset within that buffer. The log could then remain at offset 1k, and the version wouldn't have to change.
Quoted 2 lines
Add an option to create version 3 metadata storing the offset in the on-disk header but continue to use version 2 by default.
I think this is a better overall solution - less kludgy and more flexible.
Quoted 2 lines
Update userspace code to request version 3 (via new log parameter) for new mirror logs by default, but still permit version 2 for new logs.
Is there a situation where someone would still need to create a new version 2 log? I would think it would be far simpler to switch all new logs to use version 3, and revert back to version 2 only when the log offset recorded in the on-disk header is blank.
Or were you thinking of using this new log parameter to allow user-space to specify the offset of the log?
On a related note, shouldn't log_header.nr_regions be a uint64_t instead of a sector_t? The header_to_disk() and header_from_disk() routines already treat it as 64-bit.
Signature · 4 lines
Kevin Corry [email protected] http://www.ibm.com/linux/ http://evms.sourceforge.net/
Quoted 7 lines · Kevin Corry
Quoted 5 lines · Alasdair G Kergon
A couple of options:
Retain the existing on-disk format exactly, but adjust the offsets and sizes used in the reads and writes to avoid the problems (pre-reading other data if necessary to extend a write).
Quoted 5 lines
This method seems like it should be easy to implement. We could just extend the buffer for the disk-header and always read and write both the header and the log at the same time, and just point
lc->clean_bitsat the appropriate offset within that buffer. The log could then remain at offset 1k, and the version wouldn't have to change.
Sorry I haven't had a chance to get back to this in a while. I was finally able to finish up a new patch to fix this bug based on the above suggestions.
This patch maintains compatibility with the existing metadata layout by combining the disk-header buffer and clean-bits bitset buffer into a single buffer.
Signature · 169 lines
Kevin Corry [email protected] http://www.ibm.com/linux/ http://evms.sourceforge.net/ On-disk logs for dm-mirror devices are currently hard-coded to use 512 byte hard-sector-sizes. This patch fixes dm-log so it will work with devices with non-512-byte hard-sector-sizes. To maintain full compatibility, instead of moving the clean-bits bitset to a different location, this patch removes the separate buffer used for the bitset, and enlarges the disk-header buffer to encompass both the header and the bitset. The I/O routines for the bitset are removed, and the I/O routines for the disk-header now also read/write the bitset. Signed-Off-By: Kevin Corry <[email protected]> Index: 2.6.17-rc1/drivers/md/dm-log.c =================================================================== --- 2.6.17-rc1.orig/drivers/md/dm-log.c +++ 2.6.17-rc1/drivers/md/dm-log.c @@ -155,8 +155,6 @@ struct log_c { struct io_region header_location; struct log_header *disk_header; - - struct io_region bits_location; }; /* @@ -241,29 +239,6 @@ static inline int write_header(struct lo } /*---------------------------------------------------------------- - * Bits IO - *--------------------------------------------------------------*/ -static int read_bits(struct log_c *log) -{ - int r; - unsigned long ebits; - - r = dm_io_sync_vm(1, &log->bits_location, READ, - log->clean_bits, &ebits); - if (r) - return r; - - return 0; -} - -static int write_bits(struct log_c *log) -{ - unsigned long ebits; - return dm_io_sync_vm(1, &log->bits_location, WRITE, - log->clean_bits, &ebits); -} - -/*---------------------------------------------------------------- * core log constructor/destructor * * argv contains region_size followed optionally by [no]sync @@ -373,9 +348,11 @@ static int disk_ctr(struct dirty_log *lo unsigned int argc, char **argv) { int r; - size_t size; + size_t size, bitset_size; struct log_c *lc; struct dm_dev *dev; + uint32_t *clean_bits; + unsigned int hardsect_size; if (argc < 2 || argc > 3) { DMWARN("wrong number of arguments to disk mirror log"); @@ -396,26 +373,30 @@ static int disk_ctr(struct dirty_log *lo lc = (struct log_c *) log->context; lc->log_dev = dev; + hardsect_size = ti->limits.hardsect_size; + bitset_size = lc->bitset_uint32_count * sizeof(uint32_t); + /* setup the disk header fields */ lc->header_location.bdev = lc->log_dev->bdev; lc->header_location.sector = 0; - lc->header_location.count = 1; - /* - * We can't read less than this amount, even though we'll - * not be using most of this space. - */ - lc->disk_header = vmalloc(1 << SECTOR_SHIFT); + /* Include both the header and the bitset in one buffer. */ + size = dm_round_up((LOG_OFFSET << SECTOR_SHIFT) + bitset_size, + hardsect_size); + lc->header_location.count = size >> SECTOR_SHIFT; + + lc->disk_header = vmalloc(size); if (!lc->disk_header) goto bad; - /* setup the disk bitset fields */ - lc->bits_location.bdev = lc->log_dev->bdev; - lc->bits_location.sector = LOG_OFFSET; - - size = dm_round_up(lc->bitset_uint32_count * sizeof(uint32_t), - 1 << SECTOR_SHIFT); - lc->bits_location.count = size >> SECTOR_SHIFT; + /* Deallocate the clean_bits buffer that was allocated in core_ctr() + * and point it at the appropriate place in the disk_header buffer. + */ + clean_bits = lc->clean_bits; + lc->clean_bits = (void *)lc->disk_header + (LOG_OFFSET << SECTOR_SHIFT); + memcpy(lc->clean_bits, clean_bits, bitset_size); + vfree(clean_bits); + return 0; bad: @@ -429,6 +410,7 @@ static void disk_dtr(struct dirty_log *l struct log_c *lc = (struct log_c *) log->context; dm_put_device(lc->ti, lc->log_dev); vfree(lc->disk_header); + lc->clean_bits = NULL; core_dtr(log); } @@ -449,16 +431,11 @@ static int disk_resume(struct dirty_log struct log_c *lc = (struct log_c *) log->context; size_t size = lc->bitset_uint32_count * sizeof(uint32_t); - /* read the disk header */ + /* read the disk header and bitset */ r = read_header(lc); if (r) return r; - /* read the bits */ - r = read_bits(lc); - if (r) - return r; - /* set or clear any new bits */ if (lc->sync == NOSYNC) for (i = lc->header.nr_regions; i < lc->region_count; i++) @@ -473,15 +450,10 @@ static int disk_resume(struct dirty_log memcpy(lc->sync_bits, lc->clean_bits, size); lc->sync_count = count_bits32(lc->clean_bits, lc->bitset_uint32_count); - /* write the bits */ - r = write_bits(lc); - if (r) - return r; - /* set the correct number of regions in the header */ lc->header.nr_regions = lc->region_count; - /* write the new header */ + /* write the new header and bitset */ return write_header(lc); } @@ -518,7 +490,7 @@ static int disk_flush(struct dirty_log * if (!lc->touched) return 0; - r = write_bits(lc); + r = write_header(lc); if (!r) lc->touched = 0;