LVM2 topology autodetect support (patchset 2)
cover LVM2 topology autodetect support (patchset 2)
This is the second LVM2 topology patchset; it adds the ability to autodetect Linux's I/O topology attributes that are associated with a device.
It should be noted that the changes to lib/device/device.c were the basis for the upstream libblkid's topology support. Moving forward we may look to use libblkid directly but the topology-aware libblkkid isn't yet available upstream.
Mike Snitzer (5):
Add devices/data_alignment_offset_detection to lvm.conf. Add devices/data_alignment_detection to lvm.conf. Improve ability to lookup primary device associated with partition. Retrieve MD sysfs attributes for MD partitions.
Add MD, partition and topology tests to the LVM2 test-suite.
WHATS_NEW | 2 + doc/example.conf | 21 ++++- lib/config/defaults.h | 2 + lib/device/dev-md.c | 13 ++- lib/device/device.c | 193 +++++++++++++++++++++++++++++++++++++++ lib/device/device.h | 12 +++ lib/format_text/format-text.c | 6 +- lib/metadata/metadata.c | 27 ++++++ man/lvm.conf.5.in | 22 ++++- test/t-pvcreate-operation-md.sh | 91 ++++++++++++++++++ test/t-pvcreate-usage.sh | 12 +++ test/test-utils.sh | 2 +- 12 files changed, 392 insertions(+), 11 deletions(-) create mode 100644 test/t-pvcreate-operation-md.sh
PATCH 1/5 Add devices/data_alignment_offset_detection to lvm.conf.
If the pvcreate --dataalignmentoffset option is not specified the start of a PV's aligned data area will be shifted by the associated 'alignment_offset' exposed in sysfs (unless devices/data_alignment_offset_detection is disabled in lvm.conf).
--- WHATS_NEW | 1 + doc/example.conf | 9 +++++ lib/config/defaults.h | 1 + lib/device/device.c | 71 +++++++++++++++++++++++++++++++++++++++++ lib/device/device.h | 3 ++ lib/format_text/format-text.c | 6 +++- lib/metadata/metadata.c | 8 +++++ man/lvm.conf.5.in | 9 +++++- 8 files changed, 106 insertions(+), 2 deletions(-)
WHATS_NEW +1 -0
@@ -1,5 +1,6 @@ Version 2.02.51 - ================================ + Add devices/data_alignment_offset_detection to lvm.conf. Add --dataalignmentoffset to pvcreate to shift start of aligned data area. Fix _mda_setup() to not check first mda's size before pe_align rounding. Document -I option of clvmd in the man page.
doc/example.conf +9 -0
@@ -104,6 +104,15 @@ devices { # Set to 0 for the default alignment of 64KB or page size, if larger. data_alignment = 0 + # By default, the start of the PV's aligned data area will be shifted by + # the 'alignment_offset' exposed in sysfs. This offset is often 0 but + # may be non-zero; e.g.: certain 4KB sector drives that compensate for + # windows partitioning will have an alignment_offset of 3584 bytes + # (sector 7 is the lowest aligned logical block, the 4KB sectors start + # at LBA -1, and consequently sector 63 is aligned on a 4KB boundary). + # 1 enables; 0 disables. + data_alignment_offset_detection = 1 + # If, while scanning the system for PVs, LVM2 encounters a device-mapper # device that has its I/O suspended, it waits for it to become accessible. # Set this to 1 to skip such devices. This should only be needed
lib/config/defaults.h +1 -0
@@ -34,6 +34,7 @@#define DEFAULT_MD_COMPONENT_DETECTION 1#define DEFAULT_MD_CHUNK_ALIGNMENT 1#define DEFAULT_IGNORE_SUSPENDED_DEVICES 1 +#define DEFAULT_DATA_ALIGNMENT_OFFSET_DETECTION 1#define DEFAULT_LOCK_DIR "/var/lock/lvm"#define DEFAULT_LOCKING_LIB "liblvm2clusterlock.so"
lib/device/device.c +71 -0
74 lines not on this page · raw message
lib/device/device.h +3 -0
@@ -100,4 +100,7 @@ unsigned long dev_md_stripe_width(const char *sysfs_dir, struct device *dev);int is_partitioned_dev (struct device *dev); +unsigned long dev_alignment_offset (const char *sysfs_dir, +struct device *dev); +#endif
lib/format_text/format-text.c +5 -1
@@ -1772,7 +1772,11 @@ static int _text_pv_setup(const struct format_type *fmt,"%lu sectors (requested %lu sectors)" , pv_dev_name(pv), pv->pe_align, data_alignment); -set_pe_align_offset (pv, data_alignment_offset); +if (set_pe_align_offset (pv, data_alignment_offset) != data_alignment_offset && + data_alignment_offset) +log_warn ("WARNING: %s: Overriding data alignment offset to " +"%lu sectors (requested %lu sectors)" , +pv_dev_name (pv), pv->pe_align_offset, data_alignment_offset);if (pv->pe_align < pv->pe_align_offset) { log_error("%s: pe_align (%lu sectors) must not be less "
lib/metadata/metadata.c +8 -0
@@ -105,6 +105,14 @@ unsigned long set_pe_align_offset(struct physical_volume *pv,if (!pv->dev) goto out; +if (find_config_tree_bool (pv->fmt->cmd, +"devices/data_alignment_offset_detection" , +DEFAULT_DATA_ALIGNMENT_OFFSET_DETECTION )) + pv->pe_align_offset = +MAX (pv->pe_align_offset, +dev_alignment_offset (pv->fmt->cmd->sysfs_dir, + pv->dev)); +log_very_verbose ("%s: Setting PE alignment offset to %lu sectors." ,dev_name (pv->dev), pv->pe_align_offset);
man/lvm.conf.5.in +8 -1
@@ -142,10 +142,17 @@ when creating a new Physical Volume using the \fBlvm2\fP format. If a Physical Volume is placed directly upon an md device and \fBmd_chunk_alignment\fP is enabled this parameter is ignored. Set to 0 to use the default alignment of 64KB or the page size, if larger. +.IP +\fBdata_alignment_offset_detection\fP \(em If set to 1, and your kernel +provides topology information in sysfs for the Physical Volume, the +start of the aligned data area of the Physical Volume will be shifted +by the alignment_offset exposed in sysfs. .sp To see the location of the first Physical Extent of an existing Physical Volume use \fBpvs -o +pe_start\fP . It will be a multiple of the requested -\fBdata_alignment\fP. +\fBdata_alignment\fP plus the alignment_offset from +\fBdata_alignment_offset_detection\fP (if enabled) or the pvcreate +commandline. .TP \fBlog\fP \(em Default log settings .IP
Signature · 1 line
1.6.2.5
PATCH 2/5 Add devices/data_alignment_detection to lvm.conf.
In reply to Mike SnitzerAdds 'data_alignment_detection' config option to the devices section of lvm.conf. If your kernel provides topology information in sysfs (linux
Quoted 1 line
= 2.6.31) for the Physical Volume, the start of data area will be
aligned on a multiple of the ?minimum_io_size? or ?optimal_io_size? exposed in sysfs.
minimum_io_size is used if optimal_io_size is undefined (0). If both md_chunk_alignment and data_alignment_detection are enabled the result of data_alignment_detection is used.
--- WHATS_NEW | 1 + doc/example.conf | 12 ++++++- lib/config/defaults.h | 1 + lib/device/device.c | 75 +++++++++++++++++++++++++++++++++++++++++++--- lib/device/device.h | 6 ++++ lib/metadata/metadata.c | 19 ++++++++++++ man/lvm.conf.5.in | 13 +++++++- 7 files changed, 118 insertions(+), 9 deletions(-)
WHATS_NEW +1 -0
@@ -1,5 +1,6 @@ Version 2.02.51 - ================================ + Add devices/data_alignment_detection to lvm.conf. Add devices/data_alignment_offset_detection to lvm.conf. Add --dataalignmentoffset to pvcreate to shift start of aligned data area. Fix _mda_setup() to not check first mda's size before pe_align rounding.
doc/example.conf +10 -2
@@ -98,9 +98,17 @@ devices { # 1 enables; 0 disables. md_chunk_alignment = 1 + # By default, the start of a PV's data area will be aligned with + # the 'minimum_io_size' or 'optimal_io_size' exposed in sysfs. + # minimum_io_size is used if optimal_io_size is undefined (0). + # If md_chunk_alignment is enabled, that detects the optimal_io_size. + # This setting takes precedence over md_chunk_alignment. + # 1 enables; 0 disables. + data_alignment_detection = 1 + # Alignment (in KB) of start of data area when creating a new PV. - # If a PV is placed directly upon an md device and md_chunk_alignment is - # enabled this parameter is ignored. + # If a PV is placed directly upon an md device and md_chunk_alignment or + # data_alignment_detection is enabled this parameter is ignored. # Set to 0 for the default alignment of 64KB or page size, if larger. data_alignment = 0
lib/config/defaults.h +1 -0
@@ -35,6 +35,7 @@#define DEFAULT_MD_CHUNK_ALIGNMENT 1#define DEFAULT_IGNORE_SUSPENDED_DEVICES 1#define DEFAULT_DATA_ALIGNMENT_OFFSET_DETECTION 1 +#define DEFAULT_DATA_ALIGNMENT_DETECTION 1#define DEFAULT_LOCK_DIR "/var/lock/lvm"#define DEFAULT_LOCKING_LIB "liblvm2clusterlock.so"
lib/device/device.c +70 -5
109 lines not on this page · raw message
lib/device/device.h +6 -0
@@ -103,4 +103,10 @@ int is_partitioned_dev(struct device *dev);unsigned long dev_alignment_offset (const char *sysfs_dir,struct device *dev); +unsigned long dev_minimum_io_size (const char *sysfs_dir, +struct device *dev); + +unsigned long dev_optimal_io_size (const char *sysfs_dir, +struct device *dev); +#endif
lib/metadata/metadata.c +19 -0
@@ -86,6 +86,25 @@ unsigned long set_pe_align(struct physical_volume *pv, unsigned long data_alignmdev_md_stripe_width (pv->fmt->cmd->sysfs_dir, pv->dev)); +/* +* Align to topology's minimum_io_size or optimal_io_size if present +* - minimum_io_size - the smallest request the device can perform +* w/o incurring a read-modify-write penalty (e.g. MD's chunk size) +* - optimal_io_size - the device's preferred unit of receiving I/O +* (e.g. MD's stripe width) +*/ +if (find_config_tree_bool (pv->fmt->cmd, +"devices/data_alignment_detection" , +DEFAULT_DATA_ALIGNMENT_DETECTION )) { + pv->pe_align =MAX (pv->pe_align, +dev_minimum_io_size (pv->fmt->cmd->sysfs_dir, + pv->dev)); + + pv->pe_align =MAX (pv->pe_align, +dev_optimal_io_size (pv->fmt->cmd->sysfs_dir, + pv->dev)); + } +log_very_verbose ("%s: Setting PE alignment to %lu sectors." ,dev_name (pv->dev), pv->pe_align);
man/lvm.conf.5.in +11 -2
@@ -137,11 +137,20 @@ has been reused without wiping the md superblocks first. directly upon an md device, LVM2 will align its data blocks with the md device's stripe-width. .IP +\fBdata_alignment_detection\fP \(em If set to 1, and your kernel provides +topology information in sysfs for the Physical Volume, the start of data +area will be aligned on a multiple of the ?minimum_io_size? or +?optimal_io_size? exposed in sysfs. minimum_io_size is used if +optimal_io_size is undefined (0). If both \fBmd_chunk_alignment\fP and +\fBdata_alignment_detection\fP are enabled the result of +\fBdata_alignment_detection\fP is used. +.IP \fBdata_alignment\fP \(em Default alignment (in KB) of start of data area when creating a new Physical Volume using the \fBlvm2\fP format. If a Physical Volume is placed directly upon an md device and -\fBmd_chunk_alignment\fP is enabled this parameter is ignored. -Set to 0 to use the default alignment of 64KB or the page size, if larger. +\fBmd_chunk_alignment\fP or \fBdata_alignment_detection\fP is enabled +this parameter is ignored. Set to 0 to use the default alignment of +64KB or the page size, if larger. .IP \fBdata_alignment_offset_detection\fP \(em If set to 1, and your kernel provides topology information in sysfs for the Physical Volume, the
Signature · 1 line
1.6.2.5
PATCH 3/5 Improve ability to lookup primary device associated with partition
In reply to Mike SnitzerImprove lib/device/device.c:_primary_dev()'s ability to look up the primary device associated with all partitions; including blkext (e.g. partitions directly on MD). The same will also work for obscure sysfs paths; e.g.: paths with mangled names like the HP cciss driver
uses: /sys/block/cciss!c0d0/cciss!c0d0p1/
--- lib/device/device.c | 61 ++++++++++++++++++++++++++++++++++++++++++++++---- 1 files changed, 56 insertions(+), 5 deletions(-)
lib/device/device.c +56 -5
@@ -13,6 +13,7 @@* Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */ +#include <libgen.h> /* dirname, basename */ #include "lib.h" #include "lvm-types.h" #include "device.h"
@@ -289,21 +290,71 @@ static int _primary_dev(const char *sysfs_dir, struct device *dev, dev_t *result) { char path[PATH_MAX+1]; +char temp_path[PATH_MAX +1 ]; +char buffer[64 ];struct stat info; +FILE *fp; +int pri_maj, pri_min; +int ret =0 ;/* check if dev is a partition */ if (dm_snprintf (path,PATH_MAX ,"%s/dev/block/%d:%d/partition" , sysfs_dir, (int )MAJOR (dev->dev), (int )MINOR (dev->dev)) <0 ) {log_error ("dm_snprintf partition failed" ); -return 0 ; +return ret; }if (stat (path, &info) <0 ) -return 0 ; +return ret; - *result = dev->dev - - (MINOR (dev->dev) %max_partitions (MAJOR (dev->dev))); -return 1 ; +/* +* extract parent's path from the partition's symlink, e.g.: +* - readlink /sys/dev/block/259:0 = ../../block/md0/md0p1 +* - dirname ../../block/md0/md0p1 = ../../block/md0 +* - basename ../../block/md0/md0 = md0 +* Parent's 'dev' sysfs attribute = /sys/block/md0/dev +*/ +if (readlink (dirname (path), temp_path,PATH_MAX ) <0 ) { +log_sys_error ("readlink" , path); +return ret; + } + +if (dm_snprintf (path,PATH_MAX ,"%s/block/%s/dev" , + sysfs_dir,basename (dirname (temp_path))) <0 ) { +log_error ("dm_snprintf dev failed" ); +return ret; + } + +/* finally, parse 'dev' attribute and create corresponding dev_t */ +if (stat (path, &info) <0 ) { +log_error ("sysfs file %s does not exist" , path); +return ret; + } + + fp =fopen (path,"r" ); +if (!fp) { +log_sys_error ("fopen" , path); +return ret; + } + +if (!fgets (buffer,sizeof (buffer), fp)) { +log_sys_error ("fgets" , path); + goto out; + } + +if (sscanf (buffer,"%d:%d" , &pri_maj, &pri_min) !=2 ) { +log_error ("sysfs file %s not in expected MAJ:MIN format: %s" , + path, buffer); + goto out; + } + *result =makedev (pri_maj, pri_min); + ret =1 ; + +out: +if (fclose (fp)) +log_sys_error ("fclose" , path); + +return ret; } static unsigned long _dev_topology_attribute(const char *attribute,
Signature · 1 line
1.6.2.5
PATCH 4/5 Retrieve MD sysfs attributes for MD partitions
In reply to Mike SnitzerRename private _primary_dev() to a public get_primary_dev() and reuse it to allow retrieval of the MD sysfs attributes (raid level, etc) for MD partitions.
--- lib/device/dev-md.c | 13 +++++++++---- lib/device/device.c | 12 +++++++++--- lib/device/device.h | 3 +++ 3 files changed, 21 insertions(+), 7 deletions(-)
lib/device/dev-md.c +9 -4
@@ -131,16 +131,21 @@ static int _md_sysfs_attribute_snprintf(char *path, size_t size, const char *attribute) {struct stat info; +dev_t _dev = dev->dev;int ret =-1 ; -if (MAJOR (dev->dev) !=md_major ()) +if (!sysfs_dir || !*sysfs_dir)return ret; -if (!sysfs_dir || !*sysfs_dir) +check_md_major: +if (MAJOR (_dev) !=md_major ()) { +if (get_primary_dev (sysfs_dir, dev, &_dev)) + goto check_md_major;return ret; + } ret =dm_snprintf (path, size,"%s/dev/block/%d:%d/md/%s" , sysfs_dir, - (int )MAJOR (dev->dev), (int )MINOR (dev->dev), attribute); + (int )MAJOR (_dev), (int )MINOR (_dev), attribute);if (ret <0 ) {log_error ("dm_snprintf md %s failed" , attribute);return ret;
@@ -149,7 +154,7 @@ static int _md_sysfs_attribute_snprintf(char *path, size_t size,if (stat (path, &info) <0 ) {/* old sysfs structure */ ret =dm_snprintf (path, size,"%s/block/md%d/md/%s" , - sysfs_dir, (int )MINOR (dev->dev), attribute); + sysfs_dir, (int )MINOR (_dev), attribute);if (ret <0 ) {log_error ("dm_snprintf old md %s failed" , attribute);return ret;
lib/device/device.c +9 -3
@@ -286,8 +286,8 @@ int _get_partition_type(struct dev_mgr *dm, struct device *d)#ifdef linux -static int _primary_dev (const char *sysfs_dir, -struct device *dev,dev_t *result) +int get_primary_dev (const char *sysfs_dir, +struct device *dev,dev_t *result) {char path[PATH_MAX +1 ];char temp_path[PATH_MAX +1 ];
@@ -387,7 +387,7 @@ static unsigned long _dev_topology_attribute(const char *attribute,* or the device could be a partition */ if (stat (path, &info) <0 ) { -if (!_primary_dev (sysfs_dir, dev, &primary)) +if (!get_primary_dev (sysfs_dir, dev, &primary))return 0 ;/* get attribute from partition's primary device */
@@ -450,6 +450,12 @@ unsigned long dev_optimal_io_size(const char *sysfs_dir,#else +int get_primary_dev (const char *sysfs_dir, +struct device *dev,dev_t *result) +{ +return 0 ; +} +unsigned long dev_alignment_offset (const char *sysfs_dir,struct device *dev) {
lib/device/device.h +3 -0
@@ -100,6 +100,9 @@ unsigned long dev_md_stripe_width(const char *sysfs_dir, struct device *dev);int is_partitioned_dev (struct device *dev); +int get_primary_dev (const char *sysfs_dir, +struct device *dev,dev_t *result); +unsigned long dev_alignment_offset (const char *sysfs_dir,struct device *dev);
Signature · 1 line
1.6.2.5
PATCH 5/5 Add MD, partition and topology tests to the LVM2 test-suite
In reply to Mike SnitzerAdd MD, partition, and topology tests to the LVM2 test-suite.
Added MD devices to the filter in test-utils.sh:prepare_lvmconf()
--- test/t-pvcreate-operation-md.sh | 91 +++++++++++++++++++++++++++++++++++++++ test/t-pvcreate-usage.sh | 12 +++++ test/test-utils.sh | 2 +- 3 files changed, 104 insertions(+), 1 deletions(-) create mode 100644 test/t-pvcreate-operation-md.sh
test/t-pvcreate-operation-md.sh new file +91 -0
@@ -0,0 +1,91 @@ +# Copyright (C) 2009 Red Hat, Inc. All rights reserved. +# +# This copyrighted material is made available to anyone wishing to use, +# modify, copy, or redistribute it subject to the terms and conditions +# of the GNU General Public License v.2. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +# skip this test if mdadm or sfdisk aren't available +which mdadm || exit 200 +which sfdisk || exit 200 + +. ./test-utils.sh + +aux prepare_devs 2 + +# Have MD use a non-standard name to avoid colliding with an existing MD device +mddev=/dev/md_lvm_test0 + +cleanup_md() { + mdadm --stop $mddev + rm -f $mddev + teardown +} + +# create 2 disk MD raid0 array (stripe_width=128K) +[ -b "$mddev" ] && exit 200 +mdadm --create $mddev --auto=md --level 0 --raid-devices=2 --chunk 64 $dev1 $dev2 +trap 'aux cleanup_md' EXIT # cleanup this MD device at the end of the test + +# Test alignment of PV on MD without any MD-aware or topology-aware detection +# - should treat $mddev just like any other block device +pv_align="192.00K" +pvcreate --metadatasize 128k \ + --config 'devices {md_chunk_alignment=0 data_alignment_detection=0 data_alignment_offset_detection=0}' \ + $mddev +check_pv_field_ $mddev pe_start $pv_align + +# Test md_chunk_alignment independent of topology-aware detection +pv_align="256.00K" +pvcreate --metadatasize 128k \ + --config 'devices {data_alignment_detection=0 data_alignment_offset_detection=0}' \ + $mddev +check_pv_field_ $mddev pe_start $pv_align + +# Test newer topology-aware alignment detection +pv_align="256.00K" +pvcreate --metadatasize 128k \ + --config 'devices { md_chunk_alignment=0 }' $mddev +check_pv_field_ $mddev pe_start $pv_align + +# partition MD array directly, depends on blkext in Linux >= 2.6.28 +linux_minor=$(echo `uname -r` | cut -d'.' -f3 | cut -d'-' -f1) +if [ $linux_minor -gt 27 ]; then + # create one partition + sfdisk $mddev <<EOF +,,83 +EOF + # make sure partition on MD is _not_ removed + # - tests partition -> parent lookup via sysfs paths + not pvcreate --metadatasize 128k $mddev + + # verify alignment_offset is accounted for in pe_start + # - topology infrastructure is available in Linux >= 2.6.31 + # - also tests partition -> parent lookup via sysfs paths + + # Oh joy: need to lookup /sys/block/md127 rather than /sys/block/md_lvm_test0 + mddev_maj_min=$(ls -l $mddev | awk '{ print $5 $6 }' | tr , :) + mddev_p_sysfs_name=$(echo /sys/dev/block/${mddev_maj_min}/*p1) + base_mddev_p=`basename $mddev_p_sysfs_name` + mddev_p=/dev/${base_mddev_p} + + sysfs_alignment_offset=$(ls /sys/dev/block/${mddev_maj_min}/${base_mddev_p}/alignment_offset) + [ -f $sysfs_alignment_offset ] && \ + alignment_offset=`cat $sysfs_alignment_offset` || \ + alignment_offset=0 + + if [ "$alignment_offset" = "512" ]; then + pv_align="256.50K" + pvcreate --metadatasize 128k $mddev_p + check_pv_field_ $mddev_p pe_start $pv_align + pvremove $mddev_p + elif [ "$alignment_offset" = "2048" ]; then + pv_align="258.00K" + pvcreate --metadatasize 128k $mddev_p + check_pv_field_ $mddev_p pe_start $pv_align + pvremove $mddev_p + fi +fi
test/t-pvcreate-usage.sh +12 -0
@@ -116,6 +116,18 @@ check_pv_field_ $dev1 pe_start $pv_align pvcreate --metadatasize 128k --metadatacopies 2 --dataalignment 3.5k $dev1 check_pv_field_ $dev1 pe_start $pv_align +# data area is aligned to 64k by default, +# data area start is shifted by the specified alignment_offset +pv_align="195.50K" +pvcreate --metadatasize 128k --dataalignmentoffset 7s $dev1 +check_pv_field_ $dev1 pe_start $pv_align + +# 2nd metadata area is created without problems when +# data area start is shifted by the specified alignment_offset +pvcreate --metadatasize 128k --metadatacopies 2 --dataalignmentoffset 7s $dev1 +check_pv_field_ $dev1 pv_mda_count 2 +# FIXME: compare start of 2nd mda with and without --dataalignmentoffset + #COMM 'pv with LVM1 compatible data alignment can be convereted' #compatible == LVM1_PE_ALIGN == 64k pvcreate --dataalignment 256k $dev1
test/test-utils.sh +1 -1
@@ -182,7 +182,7 @@ prepare_lvmconf() { devices { dir = "$G_dev_" scan = "$G_dev_" - filter = [ "a/dev\/mirror/", "a/dev\/mapper\/.*pv[0-9_]*$/", "r/.*/" ] + filter = [ "a|/dev/md.*|", "a/dev\/mirror/", "a/dev\/mapper\/.*pv[0-9_]*$/", "r/.*/" ] cache_dir = "$G_root_/etc" sysfs_scan = 0 }
Signature · 1 line
1.6.2.5