Disable partitions scan for multipathed devices

PATCH In review Hannes Reinecke 3 participants 10 messages Mar 12, 2014 Device mapper

Share & tools

Hannes Reinecke

author

2014-03-12 11:48 UTC #0 lore raw

PATCH 1/1 Disable partitions scan for multipathed devices

When multipath devices are being used as disks for VM Guests any partition scanning / setup should be done within the VM Guest, not from host. So we need to switch off partitions scanning via kpartx there.
For this I've implemented a new feature 'no_partitions' which just serves as a notifier to kpartx to _not_ create partitions on these devices.

Signed-off-by Hannes Reinecke <[email protected]>

---
 drivers/md/dm-mpath.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

drivers/md/dm-mpath.c +10 -1

index 422a9fd..93154ec 100644
@@ -88,6 +88,7 @@ struct multipath {
 	unsigned saved_queue_if_no_path:1; /* Saved state during suspension */
 	unsigned retain_attached_hw_handler:1; /* If there's already a hw_handler present, don't change it. */
 	unsigned pg_init_disabled:1;	/* pg_init is not currently allowed */
+	unsigned no_partitions:1;	/* Disable partition scan for kpartx */
 
 	unsigned pg_init_retries;	/* Number of times to retry pg_init */
 	unsigned pg_init_count;		/* Number of times pg_init called */

@@ -818,6 +819,11 @@ static int parse_features(struct dm_arg_set *as, struct multipath *m)
 			continue;
 		}
 
+		if (!strcasecmp(arg_name, "no_partitions")) {
+			m->no_partitions = 1;
+			continue;
+		}
+
 		if (!strcasecmp(arg_name, "pg_init_retries") &&
 		    (argc >= 1)) {
 			r = dm_read_arg(_args + 1, as, &m->pg_init_retries, &ti->error);

@@ -1438,11 +1444,14 @@ static void multipath_status(struct dm_target *ti, status_type_t type,
 		DMEMIT("%u ", m->queue_if_no_path +
 			      (m->pg_init_retries > 0) * 2 +
 			      (m->pg_init_delay_msecs != DM_PG_INIT_DELAY_DEFAULT) * 2 +
-			      m->retain_attached_hw_handler);
+			      m->retain_attached_hw_handler +
+			      m->no_partitions);
 		if (m->queue_if_no_path)
 			DMEMIT("queue_if_no_path ");
 		if (m->pg_init_retries)
 			DMEMIT("pg_init_retries %u ", m->pg_init_retries);
+		if (m->no_partitions)
+			DMEMIT("no_partitions ");
 		if (m->pg_init_delay_msecs != DM_PG_INIT_DELAY_DEFAULT)
 			DMEMIT("pg_init_delay_msecs %u ", m->pg_init_delay_msecs);
 		if (m->retain_attached_hw_handler)

Signature · 1 line

1.7.12.4

Alasdair G Kergon

maintainer

2014-03-12 12:23 UTC #1 lore raw

Quoted 3 lines · Hannes Reinecke

For this I've implemented a new feature 'no_partitions' which just serves as a notifier to kpartx to _not_ create partitions on these devices.

This should be covered by the existing cookie flags (which udev rules already use, if you're using udev). It's not a multipath-specific problem but can apply to other targets too.

Alasdair

Hannes Reinecke

author

2014-03-12 12:27 UTC #2 lore raw

Quoted 9 lines · Alasdair G Kergon

Quoted 3 lines · Hannes Reinecke
For this I've implemented a new feature 'no_partitions' which just serves as a notifier to kpartx to _not_ create partitions on these devices.
This should be covered by the existing cookie flags (which udev rules already use, if you're using udev). It's not a multipath-specific problem but can apply to other targets too.

Ah.

How? To my knowledge the 'cookies' mechanism is to enable libdevmapper to wait until udev is done with device creation.

Can you elaborate here?

Cheers,

Hannes

Signature · 4 lines

Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Peter Rajnoha

2014-03-12 12:46 UTC #3 lore raw

Quoted 6 lines · Hannes Reinecke

Quoted 5 lines · Alasdair G Kergon
Quoted 3 lines · Hannes Reinecke
For this I've implemented a new feature 'no_partitions' which just serves as a notifier to kpartx to _not_ create partitions on these devices.

Quoted 9 lines

Quoted 4 lines
This should be covered by the existing cookie flags (which udev rules already use, if you're using udev). It's not a multipath-specific problem but can apply to other targets too.
Ah.
How? To my knowledge the 'cookies' mechanism is to enable libdevmapper to wait until udev is done with device creation.

Thing with calling kpartx in udev is that it's only used for mpath devices at the moment, not for all dm devices in general. We could add a hook to call kpartx for each dm device, not just limiting it to the mpath, but there has been no call for this yet...

As for the flags encoded in the cookie - the cookie has two parts - the lower 16 bits are used for synchronization identifier, the higher 16 bits are used for passing flags. Out of these 16 'flag bits', 8 bits are common for all DM devices and they're meant to be used for flagging DM devices in general and the rest 8 bits are meant to be used for specific DM subsystem use (see also
https://git.fedorahosted.org/cgit/lvm2.git/tree/libdm/libdevmapper.h#n1750)

The flags are set with the dm_set_cookie libdevmapper call.

The mpath already uses some bits from the "common" group and 1 flag from the subsystem specific group (that is applicable only to mpath). We could check if there's a combination we could reuse, but I think that in this case we should just introduce a new mpath subsystem specific flag that would direct udev rules to skip the kpartx call.

Now the question is whether there is anything else we need to skip besides the kpartx call so we cover everything we need for the "host/guest" scenario.

Signature · 1 line

Peter

Hannes Reinecke

author

2014-03-12 12:55 UTC #4 lore raw

Quoted 32 lines · Peter Rajnoha

Quoted 6 lines · Hannes Reinecke
Quoted 5 lines · Alasdair G Kergon
Quoted 3 lines · Hannes Reinecke
For this I've implemented a new feature 'no_partitions' which just serves as a notifier to kpartx to _not_ create partitions on these devices.
Quoted 9 lines
Quoted 4 lines
This should be covered by the existing cookie flags (which udev rules already use, if you're using udev). It's not a multipath-specific problem but can apply to other targets too.
Ah.
How? To my knowledge the 'cookies' mechanism is to enable libdevmapper to wait until udev is done with device creation.
Thing with calling kpartx in udev is that it's only used for mpath devices at the moment, not for all dm devices in general. We could add a hook to call kpartx for each dm device, not just limiting it to the mpath, but there has been no call for this yet...
As for the flags encoded in the cookie - the cookie has two parts - the lower 16 bits are used for synchronization identifier, the higher 16 bits are used for passing flags. Out of these 16 'flag bits', 8 bits are common for all DM devices and they're meant to be used for flagging DM devices in general and the rest 8 bits are meant to be used for specific DM subsystem use (see also
https://git.fedorahosted.org/cgit/lvm2.git/tree/libdm/libdevmapper.h#n1750)
The flags are set with the dm_set_cookie libdevmapper call.

Quoted 9 lines

The mpath already uses some bits from the "common" group and 1 flag from the subsystem specific group (that is applicable only to mpath). We could check if there's a combination we could reuse, but I think that in this case we should just introduce a new mpath subsystem specific flag that would direct udev rules to skip the kpartx call.
Now the question is whether there is anything else we need to skip besides the kpartx call so we cover everything we need for the "host/guest" scenario.

The only other problem I could think of is the ominous 'blkid' call problem.

Currently 'blkid' is run for every 'change' event,
And multipath is very keen on generating 'change' events under a variety of situations, _plus_ there is the in-kernel 'PATH_FAILED/PATH_REINSTATED' mechanism.
Each of which (except PATH_FAILED) will cause 'blkid' to be run by virtue of 13-dm-disk.rules.
(Actually, someone should expand that to cover 'PATH_REINSTATED', too)

So when that multipath device happens to have no dependencies left or all of those are failed, 'blkid' will hang until the device becomes useable again.
Which might be never, as is the case with multipath shutdown.

Hence it would be really useful if we had a 'do not issue any I/O, use the values from cache' flag, too.

Cheers,

Hannes

Signature · 4 lines

Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Peter Rajnoha

2014-03-12 13:02 UTC #5 lore raw

Quoted 53 lines · Hannes Reinecke

Quoted 32 lines · Peter Rajnoha
Quoted 6 lines · Hannes Reinecke
Quoted 5 lines · Alasdair G Kergon
Quoted 3 lines · Hannes Reinecke
For this I've implemented a new feature 'no_partitions' which just serves as a notifier to kpartx to _not_ create partitions on these devices.
Quoted 9 lines
Quoted 4 lines
This should be covered by the existing cookie flags (which udev rules already use, if you're using udev). It's not a multipath-specific problem but can apply to other targets too.
Ah.
How? To my knowledge the 'cookies' mechanism is to enable libdevmapper to wait until udev is done with device creation.
Thing with calling kpartx in udev is that it's only used for mpath devices at the moment, not for all dm devices in general. We could add a hook to call kpartx for each dm device, not just limiting it to the mpath, but there has been no call for this yet...
As for the flags encoded in the cookie - the cookie has two parts - the lower 16 bits are used for synchronization identifier, the higher 16 bits are used for passing flags. Out of these 16 'flag bits', 8 bits are common for all DM devices and they're meant to be used for flagging DM devices in general and the rest 8 bits are meant to be used for specific DM subsystem use (see also
https://git.fedorahosted.org/cgit/lvm2.git/tree/libdm/libdevmapper.h#n1750)
The flags are set with the dm_set_cookie libdevmapper call.
Quoted 9 lines
The mpath already uses some bits from the "common" group and 1 flag from the subsystem specific group (that is applicable only to mpath). We could check if there's a combination we could reuse, but I think that in this case we should just introduce a new mpath subsystem specific flag that would direct udev rules to skip the kpartx call.
Now the question is whether there is anything else we need to skip besides the kpartx call so we cover everything we need for the "host/guest" scenario.
The only other problem I could think of is the ominous 'blkid' call problem.
Currently 'blkid' is run for every 'change' event,
And multipath is very keen on generating 'change' events under a variety of situations, _plus_ there is the in-kernel 'PATH_FAILED/PATH_REINSTATED' mechanism.
Each of which (except PATH_FAILED) will cause 'blkid' to be run by virtue of 13-dm-disk.rules.
(Actually, someone should expand that to cover 'PATH_REINSTATED', too)

Quoted 3 lines

So when that multipath device happens to have no dependencies left or all of those are failed, 'blkid' will hang until the device becomes useable again.

This should be covered by latest changes in multipath - I revisited this a month ago with Ben and we've introduced some new udev rules to avoid running blkid in case we've run out of paths (there's a new 11-dm-mpath.rules). Though it's not yet upstream, but it has already been proposed on this list (we did this for RHEL originally to solve some problems we've been running into).

So this is already resolved, I hope.

Signature · 1 line

Peter

Peter Rajnoha

2014-03-12 13:06 UTC #6 lore raw

Quoted 58 lines · Peter Rajnoha

Quoted 53 lines · Hannes Reinecke
Quoted 32 lines · Peter Rajnoha
Quoted 6 lines · Hannes Reinecke
Quoted 5 lines · Alasdair G Kergon
Quoted 3 lines · Hannes Reinecke
For this I've implemented a new feature 'no_partitions' which just serves as a notifier to kpartx to _not_ create partitions on these devices.
Quoted 9 lines
Quoted 4 lines
This should be covered by the existing cookie flags (which udev rules already use, if you're using udev). It's not a multipath-specific problem but can apply to other targets too.
Ah.
How? To my knowledge the 'cookies' mechanism is to enable libdevmapper to wait until udev is done with device creation.
Thing with calling kpartx in udev is that it's only used for mpath devices at the moment, not for all dm devices in general. We could add a hook to call kpartx for each dm device, not just limiting it to the mpath, but there has been no call for this yet...
As for the flags encoded in the cookie - the cookie has two parts - the lower 16 bits are used for synchronization identifier, the higher 16 bits are used for passing flags. Out of these 16 'flag bits', 8 bits are common for all DM devices and they're meant to be used for flagging DM devices in general and the rest 8 bits are meant to be used for specific DM subsystem use (see also
https://git.fedorahosted.org/cgit/lvm2.git/tree/libdm/libdevmapper.h#n1750)
The flags are set with the dm_set_cookie libdevmapper call.
Quoted 9 lines
The mpath already uses some bits from the "common" group and 1 flag from the subsystem specific group (that is applicable only to mpath). We could check if there's a combination we could reuse, but I think that in this case we should just introduce a new mpath subsystem specific flag that would direct udev rules to skip the kpartx call.
Now the question is whether there is anything else we need to skip besides the kpartx call so we cover everything we need for the "host/guest" scenario.
The only other problem I could think of is the ominous 'blkid' call problem.
Currently 'blkid' is run for every 'change' event,
And multipath is very keen on generating 'change' events under a variety of situations, _plus_ there is the in-kernel 'PATH_FAILED/PATH_REINSTATED' mechanism.
Each of which (except PATH_FAILED) will cause 'blkid' to be run by virtue of 13-dm-disk.rules.
(Actually, someone should expand that to cover 'PATH_REINSTATED', too)
Quoted 3 lines
So when that multipath device happens to have no dependencies left or all of those are failed, 'blkid' will hang until the device becomes useable again.

Quoted 9 lines

This should be covered by latest changes in multipath - I revisited this a month ago with Ben and we've introduced some new udev rules to avoid running blkid in case we've run out of paths (there's a new 11-dm-mpath.rules). Though it's not yet upstream, but it has already been proposed on this list (we did this for RHEL originally to solve some problems we've been running into).
So this is already resolved, I hope.

https://www.redhat.com/archives/dm-devel/2014-February/msg00075.html
https://www.redhat.com/archives/dm-devel/2014-February/msg00076.html
https://www.redhat.com/archives/dm-devel/2014-February/msg00074.html

Signature · 1 line

Peter

Hannes Reinecke

author

2014-03-12 13:14 UTC #7 lore raw

Quoted 66 lines · Peter Rajnoha

Quoted 65 lines · Peter Rajnoha
Quoted 53 lines · Hannes Reinecke
Quoted 32 lines · Peter Rajnoha
Quoted 6 lines · Hannes Reinecke
Quoted 5 lines · Alasdair G Kergon
Quoted 3 lines · Hannes Reinecke
For this I've implemented a new feature 'no_partitions' which just serves as a notifier to kpartx to _not_ create partitions on these devices.
Quoted 9 lines
Quoted 4 lines
This should be covered by the existing cookie flags (which udev rules already use, if you're using udev). It's not a multipath-specific problem but can apply to other targets too.
Ah.
How? To my knowledge the 'cookies' mechanism is to enable libdevmapper to wait until udev is done with device creation.
Thing with calling kpartx in udev is that it's only used for mpath devices at the moment, not for all dm devices in general. We could add a hook to call kpartx for each dm device, not just limiting it to the mpath, but there has been no call for this yet...
As for the flags encoded in the cookie - the cookie has two parts - the lower 16 bits are used for synchronization identifier, the higher 16 bits are used for passing flags. Out of these 16 'flag bits', 8 bits are common for all DM devices and they're meant to be used for flagging DM devices in general and the rest 8 bits are meant to be used for specific DM subsystem use (see also
https://git.fedorahosted.org/cgit/lvm2.git/tree/libdm/libdevmapper.h#n1750)
The flags are set with the dm_set_cookie libdevmapper call.
Quoted 9 lines
The mpath already uses some bits from the "common" group and 1 flag from the subsystem specific group (that is applicable only to mpath). We could check if there's a combination we could reuse, but I think that in this case we should just introduce a new mpath subsystem specific flag that would direct udev rules to skip the kpartx call.
Now the question is whether there is anything else we need to skip besides the kpartx call so we cover everything we need for the "host/guest" scenario.
The only other problem I could think of is the ominous 'blkid' call problem.
Currently 'blkid' is run for every 'change' event,
And multipath is very keen on generating 'change' events under a variety of situations, _plus_ there is the in-kernel 'PATH_FAILED/PATH_REINSTATED' mechanism.
Each of which (except PATH_FAILED) will cause 'blkid' to be run by virtue of 13-dm-disk.rules.
(Actually, someone should expand that to cover 'PATH_REINSTATED', too)
Quoted 3 lines
So when that multipath device happens to have no dependencies left or all of those are failed, 'blkid' will hang until the device becomes useable again.
This should be covered by latest changes in multipath - I revisited this a month ago with Ben and we've introduced some new udev rules to avoid running blkid in case we've run out of paths (there's a new 11-dm-mpath.rules). Though it's not yet upstream, but it has already been proposed on this list (we did this for RHEL originally to solve some problems we've been running into).

Quoted 7 lines

Quoted 2 lines
So this is already resolved, I hope.
https://www.redhat.com/archives/dm-devel/2014-February/msg00075.html
https://www.redhat.com/archives/dm-devel/2014-February/msg00076.html
https://www.redhat.com/archives/dm-devel/2014-February/msg00074.html

Ah. Missed them. Will be including those.

_However_: The 'no_partition' feature has the advantage that you don't need to modify the multipath code; it'll just take whatever is in 'features' and pass it 1:1 into the device-mapper table.

When using cookies for that we'd need either a separate mechanism/flag for multipathing or filter out the features list. _And_ we need to modify libdevmapper _and_ multipath at the same time to make that work.
But if you insist ...

So can you make a patch for libdevmapper to add this cookie flag? I'll be checking the multipath-tools code ...

Cheers,

Hannes

Signature · 4 lines

Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Peter Rajnoha

2014-03-12 14:43 UTC #8 lore raw

Quoted 4 lines · Hannes Reinecke

_However_: The 'no_partition' feature has the advantage that you don't need to modify the multipath code; it'll just take whatever is in 'features' and pass it 1:1 into the device-mapper table.

But you still need some config file in userspace that will store the fact that some of the mpath devs need to be marked with this feature flag.

Quoted 5 lines

When using cookies for that we'd need either a separate mechanism/flag for multipathing or filter out the features list. _And_ we need to modify libdevmapper _and_ multipath at the same time to make that work.
But if you insist ...

If this is a subsystem specific (in this case mpath) flag, we don't need to change libdevmapper, only mpath userspace code.

Quoted 3 lines

So can you make a patch for libdevmapper to add this cookie flag? I'll be checking the multipath-tools code ...

I'll try to think about this a bit more since it would be beneficial to have such a thing for all block devices actually - any of them can be be used in a VM and we need to prevent scanning all such devices, not just mpath ones. A simple solution would be for users to define a udev rule (providing them a template) where they can match against some properties (e.g. DM name, uuid or WWID or whatever else is present for the block dev to match against in the rule).

This actually brings us to a general problem with making the device "private" in some way on the host system. AFAIK we still don't have a general rule for this! Everybody is patching this on his own. With all the support for various autoactivations and automounts, this can become quite dangerous then if not configured properly by each subsystem's mechanism to make the device private.

Signature · 1 line

Peter

Hannes Reinecke

author

2014-03-12 15:31 UTC #9 lore raw

Quoted 10 lines · Peter Rajnoha

Quoted 4 lines · Hannes Reinecke
_However_: The 'no_partition' feature has the advantage that you don't need to modify the multipath code; it'll just take whatever is in 'features' and pass it 1:1 into the device-mapper table.
But you still need some config file in userspace that will store the fact that some of the mpath devs need to be marked with this feature flag.

Yes. Multipath already has the 'multipath.conf' file, which is used to specify and/or modify multipath devices. Including the ability to set the 'features' string, which is passed 1:1 into the device-mapper core.

Quoted 9 lines

Quoted 5 lines
When using cookies for that we'd need either a separate mechanism/flag for multipathing or filter out the features list. _And_ we need to modify libdevmapper _and_ multipath at the same time to make that work.
But if you insist ...
If this is a subsystem specific (in this case mpath) flag, we don't need to change libdevmapper, only mpath userspace code.

Ah. Ok, that simplifies matters.

Quoted 12 lines

Quoted 3 lines
So can you make a patch for libdevmapper to add this cookie flag? I'll be checking the multipath-tools code ...
I'll try to think about this a bit more since it would be beneficial to have such a thing for all block devices actually - any of them can be be used in a VM and we need to prevent scanning all such devices, not just mpath ones. A simple solution would be for users to define a udev rule (providing them a template) where they can match against some properties (e.g. DM name, uuid or WWID or whatever else is present for the block dev to match against in the rule).

Quoted 7 lines

This actually brings us to a general problem with making the device "private" in some way on the host system. AFAIK we still don't have a general rule for this! Everybody is patching this on his own. With all the support for various autoactivations and automounts, this can become quite dangerous then if not configured properly by each subsystem's mechanism to make the device private.

Hehe. I've got a patch which implements a 'no_partition_scan' kernel parameter, disabling the in-kernel partition scanning code altogether. Especially useful if you're running on a root-on-multipath system, as then basically _all_ devices are handled by multipathing, so the in-kernel partition scan is pointless at best.
At worst it'll trigger tons of I/O errors on the passive path, slowing down the machine to an eventual halt as the serial console overflows.

And, btw, a udev rule won't work as the in-kernel scanning code
would be called nevertheless. You'd need to inhibit that, only then can you intercept the scanning.

I can post it here if you're interested ...

Cheers,

Hannes

Signature · 4 lines

Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)