Skip to contentSkip to conversation

Check for resized smaller devices, v1, rhbz753914

PATCH New Dave Wysochanski 2 participants 10 messages Nov 14, 2011 – Nov 15, 2011

lore
Share & tools
lore
#0 lore raw

cover Check for resized smaller devices, v1, rhbz753914

Addresses https://bugzilla.redhat.com/show_bug.cgi?id=753914

This patchset adds a check if a device has been resized since being added to LVM's control. The background of this patchset was customer case where the devices had been resized, but it was not detected by LVM until they tried to do an 'lvcreate' and got the cryptic "reload ioctl failed".

Start by adding a _check_pv_sizes() function which goes through all PVs in a VG and verifies whether the current device size is smaller than the recorded size of the PV.

Next, call _check_pv_sizes() from vg_read_internal() and flag a warning. We flag a warning here instead of failing to allow for a repair function to be added later.

Also call _check_pv_sizes() from vg_validate(). This will catch things like a pvcreate being done, and then later the device size changes and you try adding it to the VG. For now, flag this as an error and don't allow metadata updates if a PV like this is detected. We could relax this and just do a warning for now.

Finally, add a nightly test function, resize_dev, to assist in testing this, as well as a specific t-pv-resized.sh test which demonstrates vgextend failing when a PV gets sized smaller.

Further patches could include adding to 'pvck' and/or 'vgck' the ability to repair (provided no LVs would be affected, for any resized device, call 'pvresize --setphysicalvolumesize' using the current device size).

Ran against nightly tests with no failures, some warnings:
Testing with locking_type 1
## 91 tests 62:01 : 71 OK, 6 warnings, 0 failures; 14 skipped

#1 lore raw

PATCH 1/5 Add static _check_pv_sizes() function to check all pv sizes for consistency.

This function checks for the underlying device size < pv_size, and returns 0 (fail) if such a PV is found. It should always be the case that the device size >= pv_size (recorded at pvcreate time). Before this patch, LVM does not detect if the underlying device changes in size (for example, the PV is a LUN on a SAN, and gets resized) after pvcreate initialization.

This function should be called from vg_validate() and vg_read_internal() at the very least, barring some condition I have not thought of. Whether we want to warn or error out at those points is somewhat up for discussion (see later patches).

We could also just call this only from a check/repair code path such as pvck and/or vgck code path.

---
 lib/metadata/metadata.c |   28 ++++++++++++++++++++++++++++
 1 files changed, 28 insertions(+), 0 deletions(-)
lib/metadata/metadata.c +28 -0
index 4037c0e..3ee39f7 100644
@@ -2302,6 +2302,34 @@ static int _lv_validate_references_single(struct logical_volume *lv, void *data)
 	return r;
 }
 
+/*
+ * Check all pv sizes for consistency.
+ * It should never be the case that the device size is less than the pv_size.
+ * This can happen if for example, the PVs are on a SAN and they get resized.
+ * Often resizes are done and it is assumed subsystems such as LVM
+ * automatically detect the changes.
+ */
+static int _check_pv_sizes(struct volume_group *vg)
+{
+	struct physical_volume *pv;
+	struct pv_list *pvl;
+	int ret=1;
+
+	dm_list_iterate_items(pvl, &vg->pvs) {
+		pv = pvl->pv;
+		if (is_missing_pv(pv))
+			continue;
+
+		if (pv_dev_size(pv) < pv_size(pv)) {
+			log_error("Device %s size = %lu smaller than "
+				  "pv_size = %lu.  Was device resized?\n",
+				  pv_dev_name(pv), pv_dev_size(pv), pv_size(pv));
+			ret = 0;
+		}
+	}
+	return ret;
+}
+
 int vg_validate(struct volume_group *vg)
 {
 	struct pv_list *pvl;
Signature · 1 line
1.7.4.4
#2 lore raw

PATCH 1/5 Add static _check_pv_sizes() function to check all pv sizes for consistency.

Quoted 2 lines · Dave Wysochanski

This function should be called from vg_validate() and vg_read_internal() at the very least, barring some condition I have not thought of. Whether

--setphysicalvolumesize

Alasdair

#3 lore raw

PATCH 1/5 Add static _check_pv_sizes() function to check all pv sizes for consistency.

Quoted 8 lines · Alasdair G Kergon
Quoted 2 lines · Dave Wysochanski

This function should be called from vg_validate() and vg_read_internal() at the very least, barring some condition I have not thought of. Whether

--setphysicalvolumesize

Alasdair

Hmmm. So there's a reason we should allow someone to "pvresize --setphysicalvolumesize" a larger size than the device itself? What's the use case, or are you just not wanting to be restrictive?

It did not seem to me we ever want to allow it.

#4 lore raw

PATCH 2/5 Call _check_pv_sizes() from vg_read_internal(), print warning.

In reply to Dave Wysochanski

If we read a vg with any PVs that have changed sizes since pvcreate, flag it as a warning for now.

We probably want to provide a command to safely repair this condition, perhaps vgck and/or pvck.

---
 lib/metadata/metadata.c |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)
lib/metadata/metadata.c +5 -0
index 3ee39f7..56d78fa 100644
@@ -3358,6 +3358,11 @@ struct volume_group *vg_read_internal(struct cmd_context *cmd, const char *vgnam
 	if (!(vg = _vg_read(cmd, vgname, vgid, warnings, consistent, 0)))
 		return NULL;
 
+	if (!_check_pv_sizes(vg)) {
+		log_warn("WARNING: One or more PVs have changed sizes in %s.",
+			  vg->name);
+	}
+
 	if (!check_pv_segments(vg)) {
 		log_error(INTERNAL_ERROR "PV segments corrupted in %s.",
 			  vg->name);
Signature · 1 line
1.7.4.4
#5 lore raw

PATCH 3/5 Call _check_pv_sizes() from vg_validate().

In reply to Dave Wysochanski

Check PV sizes from vg_validate(), which gets called from pvck, as well as vg_write() (before any metadata changes). We'll need to relax this check if we want to add a repair function.

---
 lib/metadata/metadata.c |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)
lib/metadata/metadata.c +5 -0
index 56d78fa..8fd2d23 100644
@@ -2394,6 +2394,11 @@ int vg_validate(struct volume_group *vg)
 		}
 	}
 
+	if (!_check_pv_sizes(vg)) {
+		log_error("One or more PVs have changed sizes in %s.",
+			  vg->name);
+		r = 0;
+	}
 
 	if (!check_pv_segments(vg)) {
 		log_error(INTERNAL_ERROR "PV segments corrupted in %s.",
Signature · 1 line
1.7.4.4
#6 lore raw

PATCH 4/5 Add resize_dev() function to LVM nightly test library.

In reply to Dave Wysochanski

Add a resize device function to LVM nightly test library. This will help us test scenarios when devices get resized after being added to LVM control. This has been seen in customer environments, and today LVM does not detect this condition.

To implement resize_dev, we needed two things:
1. The count of devices originally created via prepare_pvs
2. The size of the devices originally created via prepare_pvs

When I looked into #1, a reasonable solution seemed to be to save a "NUM_DEVICES" to a file, similar to what is done with "LOOP".

For #2, the size of the loop device, divided by "NUM_DEVICES" would suffice. However, I discovered the actual size of the loop device had a bug because of the 'dd' statement. So I fixed that in this patch as well. You can easily see the bug if you dump 'loopsz' in prepare_devs during create with say "prepare_pvs 3 10".

---
 test/lib/aux.sh |   33 ++++++++++++++++++++++++++++++++-
 1 files changed, 32 insertions(+), 1 deletions(-)
test/lib/aux.sh +32 -1
index cde5581..25781fa 100644
@@ -103,6 +103,7 @@ teardown_devs() {
 		test -f LOOPFILE && rm -f $(cat LOOPFILE)
 	fi
 	rm -f DEVICES # devs is set in prepare_devs()
+	rm -f NUM_DEVICES
 	rm -f LOOP
 
 	# Attempt to remove any loop devices that failed to get torn down if earlier tests aborted
@@ -203,7 +204,7 @@ prepare_loop() {
         echo -n .
 
 	LOOPFILE="$PWD/test.img"
-	dd if=/dev/zero of="$LOOPFILE" bs=$((1024*1024)) count=0 seek=$(($size-1)) 2> /dev/null
+	dd if=/dev/zero of="$LOOPFILE" bs=$((1024*1024)) count=0 seek=$(($size)) 2> /dev/null
 	if LOOP=`losetup -s -f "$LOOPFILE" 2>/dev/null`; then
 		:
 	elif LOOP=`losetup -f` && losetup $LOOP "$LOOPFILE"; then
@@ -314,6 +315,7 @@ prepare_devs() {
 	#done
 
         echo $devs > DEVICES
+        echo $n > NUM_DEVICES
         echo "ok"
 }
 
@@ -339,6 +341,34 @@ enable_dev() {
 	finish_udev_transaction
 }
 
+resize_dev() {
+	local dev="$1"
+	local new_size="$2"
+	if [ $# -ne 2 ]; then
+		echo "Usage: resize_dev devname size_in_mb"
+		return 1
+	fi
+
+	local name=`echo "$dev" | sed -e 's,.*/,,'`
+	local dev_num=`echo "$name" | sed -e 's/.*pv//g'`
+	if ! loopsz=`blockdev --getsz $LOOP 2>/dev/null`; then
+		loopsz=`blockdev --getsize $LOOP 2>/dev/null`
+	fi
+	local new_size_sectors=$((new_size * 1024 * 2))
+	local max_size_sectors=$(($loopsz / $NUM_DEVICES))
+
+	if [ "$new_size_sectors" -gt "$max_size_sectors" ]; then
+		echo "resize_dev: new size $new_size_sectors sectors exceeds maximum size $max_size_sectors sectors"
+		return 1
+	fi
+	disable_dev $dev
+	init_udev_transaction
+	echo "0 $new_size_sectors linear $LOOP $((($dev_num-1)*$max_size_sectors))" > $name.table
+	dmsetup create -u TEST-$name $name $name.table || dmsetup load $name $name.table
+	dmsetup resume $name
+	finish_udev_transaction
+}
+
 backup_dev() {
 	for dev in "$@"; do
 		dd if=$dev of=$dev.backup bs=1024
@@ -482,6 +512,7 @@ target_at_least()
 }
 
 test -f DEVICES && devs=$(cat DEVICES)
+test -f NUM_DEVICES && NUM_DEVICES=$(cat NUM_DEVICES)
 test -f LOOP && LOOP=$(cat LOOP)
 
 "$@"
Signature · 1 line
1.7.4.4
#7 lore raw

PATCH 5/5 Add t-pv-resized nightly test, check for resized PV.

In reply to Dave Wysochanski

Add a check to ensure a vgextend or any operation which modifies metadata will fail after a PV gets resized smaller than when it was originally pvcreated.

---
 test/t-pv-resized.sh |   24 ++++++++++++++++++++++++
 1 files changed, 24 insertions(+), 0 deletions(-)
 create mode 100644 test/t-pv-resized.sh
test/t-pv-resized.sh new file +24 -0
new file mode 100644
index 0000000..92f6468
@@ -0,0 +1,24 @@
+#!/bin/sh
+# Copyright (C) 2008 Red Hat, Inc. All rights reserved.
+#
+# This copyrighted material is made available to anyone wishing to use,
+# modify, copy, or redistribute it subject to the terms and conditions
+# of the GNU General Public License v.2.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software Foundation,
+# Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+#
+# Test LVM tools when PV devices get resize (for example, on SAN).
+#
+. lib/test
+
+aux prepare_pvs 3 10
+
+vgcreate $vg $dev1 $dev2
+pvs -opv_name,pv_size,dev_size
+# Assume one PV gets sized smaller than at time of pvcreate
+aux resize_dev $dev3 5
+pvs -opv_name,pv_size,dev_size
+# Now, any operation like vgextend that writes metadata should fail
+not vgextend $vg $dev3
Signature · 1 line
1.7.4.4
#8 lore raw

cover Check for resized smaller devices, v1, rhbz753914

In reply to Dave Wysochanski

There are circumstances where sizes don't match and it's OK. There are circumstances where sizes don't match and it's not OK.

Warn, prompt or abort?

vg_validate is really about the internal consistency of the metadata, and I hesitate to extend it to check the underlying devices.

It's not a particularly cheap check either. It shouldn't be done while any devices are suspended.

A flag in each PV to say to ignore a size mismatch on an individual PV, controlled with pvchange and set when --setphysicalvolumesize overrides it, or alternatively just a single global lvm.conf setting?

Cache the sizes during the standard scanning we already do, perhaps and then check on (just the first?) vg read? Re-check next time VG lock is acquired? (Hook into VG cache this way?)

Alasdair

#9 lore raw

cover Check for resized smaller devices, v1, rhbz753914

Quoted 5 lines · Alasdair G Kergon

There are circumstances where sizes don't match and it's OK. There are circumstances where sizes don't match and it's not OK.

Warn, prompt or abort?

"sizes don't match" is a larger scope than what I was trying to tackle.

I don't think pv_size should ever be larger than dev_size, and today it causes strange failures with things like lvcreate.

However, it's perfectly fine for pv_size to be be smaller than dev_size (no LVM problems result).

Quoted 3 lines

vg_validate is really about the internal consistency of the metadata, and I hesitate to extend it to check the underlying devices.

Ok, that's reasonable to say vg_validate is restricted to internal checks. I was not sure exactly where we should call it.

Quoted 3 lines

It's not a particularly cheap check either. It shouldn't be done while any devices are suspended.

Yeah, and in probably 95% of times the check will pass so it's just overhead. This would argue for restricting the check to something like pvck or vgck.

What do you think about calling from pvck / vgck for now?

Quoted 4 lines

A flag in each PV to say to ignore a size mismatch on an individual PV, controlled with pvchange and set when --setphysicalvolumesize overrides it, or alternatively just a single global lvm.conf setting?

I would hope to not have to implement another flag or option but it may be necessary.

Quoted 4 lines

Cache the sizes during the standard scanning we already do, perhaps and then check on (just the first?) vg read? Re-check next time VG lock is acquired? (Hook into VG cache this way?)

I'll have to look into the scope of these types of changes. At a high level, it seems a reasonable approach, and may even improve performance.