Udev integration for device-mapper and its subsystems.
Hi,
I'd like to discuss these rules with you and any comments are welcome.
These are the prototypes of the rules we would like to use for device-mapper and its subsystems (LVM, multipath, etc.). We use a simple notification command "dmsetup udevnotify" that we have added. We send a notification when all the rules are completed, so we are sure that all the udev actions are complete.
The "DM_COOKIE" value you can see in the rules is just a random numeric value we set for every action (ioctl) and return as env var in uevents. When we want to group these devices together, we just set the same cookie value for every device in a group. We can notify the waiting process directly this way. We use this method instead of comparing queued/processed udev event numbers (this way we are sure we don't waste time waiting for unrelated events).
You can spot the use of the "last_rule", which seems to be controversial. The reason is that we require reliability and performance and we'd like to have more control over the rules that are applied for DM devices as well. Also, we would like to minimize any problems caused by some general rules creating wrong nodes or delaying the udev response for DM devices in any way (the process that fired the actions is waiting for the notification from udev rule!). And this is the part I'd like to discuss -- I know the usaage of "last_rule" is not perfect for the others, but we really need to satisfy the needs described above. There could be some system that requires the information about our devices, but we can resend the events for these systems if it is really necessary (for example the HAL). Simply, we need more control over this process.
We also need a separate place to define permissions and also a separate place to register the subsystems. Therefore, the rules are separated into several files as you can see below.
If you have any suggestions or critique, please, let me know.
Thank you very much for any comments in advance...
Peter
These rules are supposed to be installed in /lib/udev/rules.d dir:
=================================
10-dm-getname.rules (set DM_NAME and DM_UUID env vars used in rules later)
--------------------------------------------------------------------------
SUBSYSTEM!="block", GOTO="dm_end"
KERNEL!="dm-[0-9]*", GOTO="dm_end"
ACTION!="change", GOTO="dm_end"
# "dm" sysfs subdirectory is available in kernels 2.6.29 and later. # We have to check for its existence and use dmsetup tool instead
# to get the DM name and UUID if this subdirectory is not present.
# We use DM_NAME and DM_UUID in rules later.
TEST="dm", ENV{DM_NAME}="$attr{dm/name}", ENV{DM_UUID}="$attr{dm/uuid}"
TEST!="dm", IMPORT{program}="/sbin/dmsetup info -j %M -m %m -c --nameprefixes --noheadings --rows -o name,uuid"LABEL="dm_end"
12-dm.rules (create plain DM device nodes in /dev/mapper)
---------------------------------------------------------
KERNEL="device-mapper", NAME="exp_mapper/control"
SUBSYSTEM!="block", GOTO="dm_end"
KERNEL!="dm-[0-9]*", GOTO="dm_end"
ACTION!="change|remove", GOTO="dm_end"
ACTION="change", NAME="mapper/$env{DM_NAME}"
# If any DM subsystem is registered, we don't do any notificatons here. # It's the responsibility of the subsystem's rules to do this notification,
# so the waiting process is notified after all the required actions are done.
# We use the "last_rule" functionality, so we are sure that no other rule is
# processed afterwards.
ENV{DM_SUBSYS}="?*", GOTO="dm_end"ENV{DM_COOKIE}="?*", RUN+="/sbin/dmsetup udevnotify $env{DM_COOKIE}", OPTIONS+="last_rule"
LABEL="dm_end"
13-lvm.rules (create symlinks for LVM -- /dev/VG_NAME/LV_NAME)
--------------------------------------------------------------
SUBSYSTEM!="block", GOTO="lvm_end" KERNEL!="dm-[0-9]*", GOTO="lvm_end" ACTION!="change|remove", GOTO="lvm_end" ENV{DM_SUBSYS}!="LVM", GOTO="lvm_end"
# Split DM name into its its VG/LV/LAYER LVM constituents
ACTION="change", IMPORT{program}="/sbin/dmsetup lvmsplit $env{DM_NAME} --nameprefixes --noheadings --rows"
# Do not create symlinks for hidden subdevices.
ENV{DM_LV_NAME}="?*_mlog", GOTO="notify"
ENV{DM_LV_NAME}="?*_mimage_[0-9]*", GOTO="notify"
# Create symlinks for top level devices only.
ACTION="change", ENV{DM_VG_NAME}="?*", ENV{DM_LV_NAME}="?*", ENV{DM_LV_LAYER}!="?*", SYMLINK+="$env{DM_VG_NAME}/$env{DM_LV_NAME}" LABEL="notify", ENV{DM_COOKIE}="?*", RUN+="/sbin/dmsetup udevnotify $env{DM_COOKIE}", OPTIONS+="last_rule"
LABEL="lvm_end"
95-dm-fallback.rules
--------------------
SUBSYSTEM!="block", GOTO="dm_end"
KERNEL!="dm-[0-9]*", GOTO="dm_end"
ACTION!="change|remove", GOTO="dm_end"
# Just in case there is any unsent notification.
ENV{DM_COOKIE}="?*", RUN+="/sbin/dmsetup udevnotify $env{DM_COOKIE}", OPTIONS+="last_rule"
LABEL="dm_end"
These rules are supposed to be installed in /etc/udev/rules.d dir:
=================================
11-dm-permissions.rules
-----------------------
SUBSYSTEM!="block", GOTO="dm_end"
KERNEL!="dm-[0-9]*", GOTO="dm_end"
ACTION!="change", GOTO="dm_end"
# You can define your own permissions for DM devices here. # There are two environment variables set for you which you
# can use in your permission rules - DM_NAME and DM_UUID.
ENV{DM_NAME}="my_device", OWNER:="peter", GROUP:="peter", MODE:="644"# Default permissions for DM devices. KERNEL="dm-[0-9]*", OWNER:="root", GROUP:="root", MODE:="0600"
LABEL="dm_end"
11-dm-subsystem.rules
---------------------
SUBSYSTEM!="block", GOTO="dm_end"
KERNEL!="dm-[0-9]*", GOTO="dm_end"
ACTION!="change", GOTO="dm_end"
# Every DM subsystem registers its own prefixes here. # This is required so the notification is sent at right
# time and after the subsystem's udev rules are processed.
# We send the notification from the subsystem's rules then.
# (if we forget to do so, there is still a fallback rule
# that will send any residual notifications later)
ENV{DM_UUID}="LVM-?*", ENV{DM_SUBSYS}="LVM"
ENV{DM_UUID}="mpath-?*", ENV{DM_SUBSYS}="mpath"LABEL="dm_end"
40-multipath.rules
------------------
# multipath wants the devmaps presented as meaninglful device names # so name them after their devmap name SUBSYSTEM!="block", GOTO="end_mpath"
RUN+="socket:/org/kernel/dm/multipath_event"
KERNEL!="dm-[0-9]*", GOTO="end_mpath"
ENV{DM_SUBSYS}!="mpath", GOTO="end_mpath"
ACTION="add", PROGRAM!="/sbin/mpath_wait %M %m", GOTO="end_mpath"
ACTION="add", RUN+="/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p p' -j %M -m %m"
LABEL="end_mpath"
Quoted 7 lines · Peter Rajnoha
The "
DM_COOKIE" value you can see in the rules is just a random numeric value we set for every action (ioctl) and return as env var in uevents. When we want to group these devices together, we just set the same cookie value for every device in a group. We can notify the waiting process directly this way. We use this method instead of comparing queued/processed udev event numbers (this way we are sure we don't waste time waiting for unrelated events).
Sounds fine.
Quoted 9 lines
You can spot the use of the "
last_rule", which seems to be controversial. The reason is that we require reliability and performance and we'd like to have more control over the rules that are applied for DM devices as well. Also, we would like to minimize any problems caused by some general rules creating wrong nodes or delaying the udev response for DM devices in any way (the process that fired the actions is waiting for the notification from udev rule!). And this is the part I'd like to discuss -- I know the usaage of "last_rule" is not perfect for the others, but we really need to satisfy the needs described above.
You should not steal events, unless you are absolutely sure, that nothing else may be interested, or has to update its idea of the device, and I don't hink you can be sure of such a thing ever. I suggest you drop the last rule thing entirely, and If you see anything that goes wrong, let us know, and we will fix it.
Quoted 4 lines
There could be some system that requires the information about our devices, but we can resend the events for these systems if it is really necessary (for example the HAL). Simply, we need more control over this process.
That can not really work, you can not send out-of-band events to something.
Quoted 9 lines
12-dm.rules (create plain DM device nodes in
/dev/mapper)
---------------------------------------------------------
KERNEL="device-mapper", NAME="exp_mapper/control"SUBSYSTEM!="block", GOTO="dm_end"
KERNEL!="dm-[0-9]*", GOTO="dm_end"
ACTION!="change|remove", GOTO="dm_end"ACTION="change", NAME="mapper/$env{DM_NAME}"
Please do not rename kernel devices, they should match the kernel names. Only create SYMLINK+= to the kernel names, or change the kernel to ceate the proper names, which will not work I guess, so please leave them as they are today.
In general we do not want any unneeded disconnect from kernel names and /dev names, and dm block devices should stay as /dev/dm-* device nodes.
Quoted 8 lines
95-dm-fallback.rules
--------------------
SUBSYSTEM!="block", GOTO="dm_end"
KERNEL!="dm-[0-9]*", GOTO="dm_end"
ACTION!="change|remove", GOTO="dm_end"# Just in case there is any unsent notification.
ENV{DM_COOKIE}="?*", RUN+="/sbin/dmsetup udevnotify $env{DM_COOKIE}", OPTIONS+="last_rule"
Drop the last_rule stuff and this is not needed. :)
Quoted 1 line
These rules are supposed to be installed in
/etc/udev/rules.ddir:
Pleas don't install "documentation" in /etc. Document what is needed to set custom permissions, put examples in /usr/share/, but please do not drop "dead" files in /etc. Packages are not supposed to place anything in /etc. Ideally users area able to do "rm /etc/udev/rules.d/*" to start from scratch.
Quoted 6 lines
11-dm-subsystem.rules
---------------------
SUBSYSTEM!="block", GOTO="dm_end"
KERNEL!="dm-[0-9]*", GOTO="dm_end"
ACTION!="change", GOTO="dm_end"
Quoted 8 lines
# Every DM subsystem registers its own prefixes here. # This is required so the notification is sent at right
# time and after the subsystem's udev rules are processed. # We send the notification from the subsystem's rules then. # (if we forget to do so, there is still a fallback rule # that will send any residual notifications later) ENV{DM_UUID}="LVM-?*", ENV{DM_SUBSYS}="LVM" ENV{DM_UUID}="mpath-?*", ENV{DM_SUBSYS}="mpath"
Seems, stuff like this does not belong into /etc either.
Thanks,
Kay
Quoted 8 lines · Kay Sievers
Please do not rename kernel devices, they should match the kernel names. Only create SYMLINK+= to the kernel names, or change the kernel to ceate the proper names, which will not work I guess, so please leave them as they are today.
In general we do not want any unneeded disconnect from kernel names and /dev names, and dm block devices should stay as
/dev/dm-*device nodes.
The /dev/dm-* devices have been a support headache for us. I don't know how they first sneaked into some distros but I maintain that dm-* are internal kernel names only of use for debugging purposes and the average sysadmin should never have to encounter them. If I had my way they would never have become visible to userspace. (Unfortunately the size restriction on the field prevents us from replacing them with our preferred names within the kernel, unless that's changed since I last looked.)
There is nothing fundamental about dm-*, they are meaningless to the user, absolutely not persistent across boots and tools may need to change what they mean whenever there is some reconfiguration. Their presence causes no end of problems as they 'leak' out into initrds, fstabs, error messages etc. On a normal, functioning system, absolutely nothing in userspace should ever reference them.
The namespace for dm devices is currently *only* under /dev/mapper.
If people *really* think dm devices need to appear directly in a flat namespace in /dev (why?) then we should drop the /dev/mapper directory and move them to /dev/mapper-vg1-lvol1 for example. (The whole strength of udev is that it makes it easy to do something like that, in fact.)
Alternatively, we could see about making dm uuids mandatory and go for something like /dev/dm-<uuid>.
Alasdair
Signature · 1 line
[email protected]
Quoted 10 lines · Alasdair G Kergon
Quoted 8 lines · Kay Sievers
Please do not rename kernel devices, they should match the kernel names. Only create SYMLINK+= to the kernel names, or change the kernel to ceate the proper names, which will not work I guess, so please leave them as they are today.
In general we do not want any unneeded disconnect from kernel names and /dev names, and dm block devices should stay as
/dev/dm-*device nodes.
Quoted 8 lines
The
/dev/dm-*devices have been a support headache for us. I don't know how they first sneaked into some distros but I maintain that dm-* are internal kernel names only of use for debugging purposes and the average sysadmin should never have to encounter them. If I had my way they would never have become visible to userspace. (Unfortunately the size restriction on the field prevents us from replacing them with our preferred names within the kernel, unless that's changed since I last looked.)
There is no size limit anymore.
Quoted 6 lines
There is nothing fundamental about dm-*, they are meaningless to the user, absolutely not persistent across boots and tools may need to change what they mean whenever there is some reconfiguration. Their presence causes no end of problems as they 'leak' out into initrds, fstabs, error messages etc. On a normal, functioning system, absolutely nothing in userspace should ever reference them.
No, the kernel logs use these names, and therefore we want these names in /dev too.
Quoted 1 line
The namespace for dm devices is currently *only* under
/dev/mapper.
No, not the last years, since udev. The /dev/mapper/ nodes are racy and wrongly timed to use in hotplug setups. Most people who do hotplug-like stuff with dm use the /dev/by-id/dm-{uuid,name}* links because they are reliably created timing wise.
Quoted 4 lines
If people *really* think dm devices need to appear directly in a flat namespace in /dev (why?) then we should drop the
/dev/mapperdirectory and move them to/dev/mapper-vg1-lvol1for example. (The whole strength of udev is that it makes it easy to do something like that, in fact.)
Sure, but there is still not enough reason to be different from the kernel name. You support rename, that should never happen, that a device node needs to be renamed, if the kernel does not change the device name.
Quoted 2 lines
Alternatively, we could see about making dm uuids mandatory and go for something like /dev/dm-<uuid>.
Sounds fine, as long as the /dev names match the kernel devices. I suggest you change the kernel to use the names right away, or go for symlinks in /dev/mapper/, like some people already do with the dm/udev integration hack, which gets hopefully replaced by the new stuff you are coming up with.
Thanks,
Kay
Quoted 11 lines · Kay Sievers
Quoted 8 lines · Alasdair G Kergon
The
/dev/dm-*devices have been a support headache for us. I don't know how they first sneaked into some distros but I maintain that dm-* are internal kernel names only of use for debugging purposes and the average sysadmin should never have to encounter them. If I had my way they would never have become visible to userspace. (Unfortunately the size restriction on the field prevents us from replacing them with our preferred names within the kernel, unless that's changed since I last looked.)There is no size limit anymore.
I think that limit comes from 16 chars for block device name in register_blkdev() in kernel.
And also DISK_NAME_LEN is 32 (for genhd->disk_name, used by bdevname())...
(The dm device name has limit 128. How to set internal kernel name to properly to match dm name?)
(And DM device name is already in /sys/block/<dm-X>/dm/name, udev rules can use that value for symlink, no timing problem here.)
Quoted 2 lines
Quoted 2 lines
Alternatively, we could see about making dm uuids mandatory and go for something like /dev/dm-<uuid>.
This will break many userspace applications... If the uuid is set, userspace must use that uuid in all dm-ioctl calls to address device. And not all users of device mapper library set UUID, because it is not mandatory (an example is cryptsetup).
(So it is not quick switch because of need to fix all applications to work properly and set uuid for all dm devices... IMHO)
Milan
--
[email protected]
Quoted 11 lines · Milan Broz
Quoted 10 lines · Kay Sievers
Quoted 8 lines · Alasdair G Kergon
The
/dev/dm-*devices have been a support headache for us. I don't know how they first sneaked into some distros but I maintain that dm-* are internal kernel names only of use for debugging purposes and the average sysadmin should never have to encounter them. If I had my way they would never have become visible to userspace. (Unfortunately the size restriction on the field prevents us from replacing them with our preferred names within the kernel, unless that's changed since I last looked.)
Quoted 5 lines
Quoted 1 line
There is no size limit anymore.
I think that limit comes from 16 chars for block device name in
register_blkdev()in kernel.
And alsoDISK_NAME_LENis 32 (forgenhd->disk_name, used bybdevname())...
Ah, I see. The driver core used to have a limit of 20 chars, this has gone recently. Removing the entire copy of the device name in genhd should be easy, if that helps here. The name is in the kobject, and does not need to be copied to a static array in the same structure. Kobjects do not have any limit anymore.
Quoted 2 lines
(The dm device name has limit 128. How to set internal kernel name to properly to match dm name?)
Don't know, the name is probably not the right thing, because it can be changed, The uuid might fit better.
Quoted 2 lines
(And DM device name is already in /sys/block/<dm-X>/dm/name, udev rules can use that value for symlink, no timing problem here.)
Right, udev calls dmsetup since long to get this name and to create these links.
Kay
Quoted 2 lines · Kay Sievers
Don't know, the name is probably not the right thing, because it can be changed, The uuid might fit better.
But why would changing the name be a problem?
What is fundamental here is the device number - that's all. E.g. Anything in userspace that opens /dev/something has to check that what it actually opened still has the same device number, if it cares about avoiding races.
If we can extend this field in-kernel and put the user's preferred device name in there we'd gain the immediate big advantage of error messages produced throughout the kernel using a name that the sysadmin can translate into something they understand much more easily. "dm-vg1-lvol1" instead of "dm-13". And this would be a big help to people I believe. (Although the dm name can be very long, on most systems it's still quite short. The uuid on the other hand is long and typically randomly-generated and meaningless to the sysadmin.)
Then any userspace packages would by default pick up a meaningful name - something considerably more persistent and useful than dm-X.
So my latest proposal would be:
1) Change the kernel to allow longer names for block devices
2) Change dm to insert dm-<name> into this field instead of dm-X
3) Add an in-kernel interface to let dm change this name on request by the user
4) Adjust udev rules to create nodes as /dev/dm-<name>
(actually I'd still argue that /dev/subsystem/name is a better approach)
5) Change all dm-based userspace tools to handle /dev/dm-*
6) Change anything if there is anything in userspace that cares about those /dev names getting changed on a live object and needs to react differently to a CHANGE uevent.
And eventually perhaps:
7) Phase out /dev/mapper (in a suitably-compatible way as usual so we don't break anything)
8) Consider whether people are ready to stomach the phasing out of /dev/vg/lv or whether that should move to /dev/lvm/vg/lv so it doesn't pollute the flat /dev namespace. (LVM2 has always worked with just vg/lv and simply treats '/dev/' as an optional prefix, not a path.)
9) Also consider whether there's a better prefix than dm- to use on these devices. It's too easily confused with md. map- perhaps. Other ideas?
Alasdair
Signature · 1 line
[email protected]
Quoted 5 lines · Alasdair G Kergon
Quoted 2 lines · Kay Sievers
Don't know, the name is probably not the right thing, because it can be changed, The uuid might fit better.
But why would changing the name be a problem?
Because we would need to change the kernel name at the same time, including all the userspace rename processing, means "move" uevents.
There is the rule, that kernel block device names and /dev names match, and the kernel log shows device names which translate directly to the primary device nodes. DM is not an exception here, all block device behave like that , and we like to continue that, and not make needless rules just to be special here.
Also DM allows pretty stupid free-text device names, and names including spaces and control chars. This is not supported and users should not be able to provide such kernel names at all.
Udev btw, does not allow you to use spaces, or other unusual control characters in device names for several good reasons.
Quoted 1 line
What is fundamental here is the device number - that's all.
There are many things that match on name strings.
Quoted 3 lines
E.g. Anything in userspace that opens
/dev/somethinghas to check that what it actually opened still has the same device number, if it cares about avoiding races.
No, we don't go and change device node names for the same device number. You could do that with updated symlinks but not with the primary nodes.
Quoted 4 lines
If we can extend this field in-kernel and put the user's preferred device name in there we'd gain the immediate big advantage of error messages produced throughout the kernel using a name that the sysadmin can translate into something they understand much more easily.
Devices can have many names, also names which are not unique at all times on the same system. it might work in your specific case, but not for others.
Quoted 1 line
"dm-vg1-lvol1" instead of "dm-13".
Sure, if you get the kernel to have that as the name in /sys/block/, then it's fine. Btw, udev will replace your '|' characters with "_'. There is no free text in /dev today, especially not for user provided naming.
Quoted 7 lines
And this would be a big help to people I believe. (Although the dm name can be very long, on most systems it's still quite short. The uuid on the other hand is long and typically randomly-generated and meaningless to the sysadmin.)
Then any userspace packages would by default pick up a meaningful name - something considerably more persistent and useful than dm-X.
Sure, but again, if you want to do that for the primary device node and not for symlinks, please set the name in /sys/block/ to that name. We are not going to mess around here.
Quoted 2 lines
So my latest proposal would be:
1) Change the kernel to allow longer names for block devices
That should be easy.
Quoted 1 line
2) Change dm to insert dm-<name> into this field instead of dm-X
That will only work reliably if you can not change the name on the fly. Otherwise it will get complicated to keep track of the devices. Possible, but a real mess to get right.
Quoted 1 line
3) Add an in-kernel interface to let dm change this name on request by the user
It should be set before it is created, and not changed later. Also it is generally not recommended to let users specify any part of kernel device names. If you allow that, we will need to limit it to plain [a-zAZ-:.].
Quoted 2 lines
4) Adjust udev rules to create nodes as /dev/dm-<name> (actually I'd still argue that
/dev/subsystem/nameis a better approach)
No. The adjust the kernel device names, not the rules.
Quoted 1 line
5) Change all dm-based userspace tools to handle
/dev/dm-*
Sounds good to me.
Quoted 3 lines
6) Change anything if there is anything in userspace that cares about those /dev names getting changed on a live object and needs to react differently to a CHANGE uevent.
It would be "move", which I think you should not get yourself into the trouble supporting that.
Quoted 3 lines
And eventually perhaps:
7) Phase out/dev/mapper(in a suitably-compatible way as usual so we don't break anything)
Just keep it, what's wrong with it?
Quoted 7 lines
8) Consider whether people are ready to stomach the phasing out of
/dev/vg/lvor whether that should move to/dev/lvm/vg/lvso it doesn't pollute the flat /dev namespace. (LVM2 has always worked with just vg/lv and simply treats '/dev/' as an optional prefix, not a path.)9) Also consider whether there's a better prefix than dm- to use on these devices. It's too easily confused with md. map- perhaps. Other ideas?
I think it's fine as it is.
I think the easiest solution is just to keep the dm-* names as they are, and create proper symlinks in /dev/mapper/, which would be the first time for DM /dev names to be reliable, and people could just use them.
I'm not convinced at all, that you guys need to invent anything new here, before you have your stuff reliably working on a modern hotplug system.
And fancy renaming infrastructure is nothing anybody ever asked for. People want reliable multi-device auto-assembly, device auto-setup, and proper hooks to plug into the processing. The last thing we need here is some new special cases for DM, which do not fit into what all others are doing.
Thanks,
Kay
Quoted 1 line · Kay Sievers
including all the userspace rename processing, means "move" uevents.
So the concept for this already exists as "move". Good.
Quoted 1 line
Also DM allows pretty stupid free-text device names, and names
We took the view there was no need for any restrictions beyond '/' in the kernel and it was a matter for userspace to choose the namespace.
LVM on the other hand, does limit the alphabet pretty severely.
The workaround here would just be to map any characters outside the range udev permits into something else.
Quoted 6 lines
Quoted 3 lines
E.g. Anything in userspace that opens
/dev/somethinghas to check that what it actually opened still has the same device number, if it cares about avoiding races.No, we don't go and change device node names for the same device number. You could do that with updated symlinks but not with the primary nodes.
I'm merely pointing out that we don't pass direct references to objects between userspace and kernel, but references stored in a filesystem, and between passing a /dev/something reference into the kernel and the kernel resolving it into its device number and the corresponding object, the /dev filesystem could have changed so that it now points to a different block device: and we make it the responsibility of userspace code to check that this race did not occur.
Quoted 4 lines
Quoted 1 line
2) Change dm to insert dm-<name> into this field instead of dm-X
That will only work reliably if you can not change the name on the fly. Otherwise it will get complicated to keep track of the devices. Possible, but a real mess to get right.
I think renaming is an important facility for users and we should delve further into the implications of handling that.
Quoted 2 lines
device names. If you allow that, we will need to limit it to plain [a-zAZ-:.].
Plus numbers I presume?
LVM accepts [a-zA-Z0-9+_.-]
What about + and _ ? Would they also need to be escaped?
Quoted 4 lines
Quoted 3 lines
And eventually perhaps:
7) Phase out/dev/mapper(in a suitably-compatible way as usual so we don't break anything)Just keep it, what's wrong with it?
If we had /dev/map-vg1-lvol1 directly it would be pointless duplication.
Remember, when dm was written in the days before udev we had only /dev/mapper which we managed directly ourselves (or through devfs). I only became aware of /dev/dm-X when I started getting bug reports from users inadvertently using it for things they shouldn't.
The root of all these problems has been dm and udev each having a different emphasis on their device transactions.
(LVM/dm was designed around independent atomic transactions, which is what Peter has only now managed to retrofit into the udev framework.)
Quoted 3 lines
and proper hooks to plug into the processing. The last thing we need here is some new special cases for DM, which do not fit into what all others are doing.
dm is already different because the whole philosophy is built around facilitating arbitrary device reconfigurations. The fewer spurious things like /dev/dm-X we offer to users and developers to hang themselves with, the better.
This is why I get so cross when I receive bug reports of systems not booting because some tool didn't understand that and hard-coded a dynamic minor number into their initrd or fstab, which /dev/dm-X has been encouraging them to do.
Renaming without rebooting, unfortunately, is something our users do seem to want to be able to do and we need to find ways of making it easier for them. In LVM terms, it's clustered LVM that makes this particularly tricky for us, and my latest proposal is to store both the old name and the new name in the LVM metadata and keep checking for both names with the tools presenting the new name until such time as it's safe to make the change (e.g. after the next reboot if it's a root filesystem) and forget about the old name completely. (Currently LVM only supports renaming in-use LVs within VGs, not renaming VGs while they contain in-use LVs.)
Alasdair
Signature · 1 line
[email protected]
Quoted 4 lines · Alasdair G Kergon
Quoted 1 line · Kay Sievers
including all the userspace rename processing, means "move" uevents.
So the concept for this already exists as "move". Good.
Please stop dreaming.
Quoted 5 lines
Quoted 1 line
Also DM allows pretty stupid free-text device names, and names
We took the view there was no need for any restrictions beyond '/' in the kernel and it was a matter for userspace to choose the namespace.
And these days are over. /dev is not the thing anymore where everybody can mess around.
Quoted 4 lines
LVM on the other hand, does limit the alphabet pretty severely.
The workaround here would just be to map any characters outside the range udev permits into something else.
Sounds good.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
E.g. Anything in userspace that opens
/dev/somethinghas to check that what it actually opened still has the same device number, if it cares about avoiding races.No, we don't go and change device node names for the same device number. You could do that with updated symlinks but not with the primary nodes.
Quoted 7 lines
I'm merely pointing out that we don't pass direct references to objects between userspace and kernel, but references stored in a filesystem, and between passing a
/dev/somethingreference into the kernel and the kernel resolving it into its device number and the corresponding object, the /dev filesystem could have changed so that it now points to a different block device: and we make it the responsibility of userspace code to check that this race did not occur.
Please!
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
2) Change dm to insert dm-<name> into this field instead of dm-X
That will only work reliably if you can not change the name on the fly. Otherwise it will get complicated to keep track of the devices. Possible, but a real mess to get right.
Quoted 7 lines
I think renaming is an important facility for users and we should delve further into the implications of handling that.
Quoted 2 lines
device names. If you allow that, we will need to limit it to plain [a-zAZ-:.].
Plus numbers I presume?
Sure.
Quoted 2 lines
LVM accepts [a-zA-Z0-9+_.-]
What about + and _ ? Would they also need to be escaped?
"#+-.:=@_" and validated utf8 sequences are not escaped.
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
And eventually perhaps:
7) Phase out/dev/mapper(in a suitably-compatible way as usual so we don't break anything)Just keep it, what's wrong with it?
If we had
/dev/map-vg1-lvol1directly it would be pointless duplication.
It's your call, I just try to ask you to fix the stuff first and _ship_ it, before it break it even further as it already is. Thanks!
Quoted 4 lines
Remember, when dm was written in the days before udev we had only
/dev/mapperwhich we managed directly ourselves (or through devfs). I only became aware of/dev/dm-Xwhen I started getting bug reports from users inadvertently using it for things they shouldn't.
Sure, we are talking to you guys since years. What else should we do? Now you wake up after 5 years and think you want to change everything we do today? I'm confused.
Quoted 9 lines
The root of all these problems has been dm and udev each having a different emphasis on their device transactions.
(LVM/dm was designed around independent atomic transactions, which is what Peter has only now managed to retrofit into the udev framework.)Quoted 3 lines
and proper hooks to plug into the processing. The last thing we need here is some new special cases for DM, which do not fit into what all others are doing.
Quoted 8 lines
dm is already different because the whole philosophy is built around facilitating arbitrary device reconfigurations. The fewer spurious things like
/dev/dm-Xwe offer to users and developers to hang themselves with, the better.
This is why I get so cross when I receive bug reports of systems not booting because some tool didn't understand that and hard-coded a dynamic minor number into their initrd or fstab, which/dev/dm-Xhas been encouraging them to do.
And again: BLOCK DEVICES HAVE THE KERNEL NAME AS THE DEVICE NODE NAME.
Quoted 10 lines
Renaming without rebooting, unfortunately, is something our users do seem to want to be able to do and we need to find ways of making it easier for them. In LVM terms, it's clustered LVM that makes this particularly tricky for us, and my latest proposal is to store both the old name and the new name in the LVM metadata and keep checking for both names with the tools presenting the new name until such time as it's safe to make the change (e.g. after the next reboot if it's a root filesystem) and forget about the old name completely. (Currently LVM only supports renaming in-use LVs within VGs, not renaming VGs while they contain in-use LVs.)
Sure rename them. But if you want the device node renamed, rename the kernel devices too.
Thanks,
Kay
Quoted 8 lines · Kay Sievers
Quoted 4 lines · Alasdair G Kergon
Quoted 1 line
Also DM allows pretty stupid free-text device names, and names
We took the view there was no need for any restrictions beyond '/' in the kernel and it was a matter for userspace to choose the namespace.
And these days are over. /dev is not the thing anymore where everybody can mess around.
We had problems with some multipath vendors which named storage device paths using strange names/chars (according to LUN name, from hw).
Also dmraid sets uses names collected from proprietary metadata.
Change this means change to several tools. And there are probably still reasons to name devices according to hw provided names.
So if there is name limitation, we need to create some mapping which is acceptable both for udev and administrators/users...
Quoted 2 lines
Sure rename them. But if you want the device node renamed, rename the kernel devices too.
Sure, but it will have consequences which must be solved too.
E.g. the dm uuid can be max. 128 characters, I am sure that we can use that in kernel for internal device name.
(uuid is not just plain UUID string, it includes prefix of subsystem, like LVM- MPATH- etc.)
But how many userspace tools expects such long name in /sys/block?
People are using udev but also lvm, dmaid, multipath, kpartx, cryptsetup, etc.
We need to find some way how to fix that and not break these systems.
Maybe the symlink to /dev/dm-X is first step. This will switch dm to using udev, what is the primary goal now, thought.
and (later?),
- introduce mandatory uuid for dm device (or disable rename for devices which have no uuid set and keep the old dm-X name for them?).
- use dm uuid, so the internal kernel device name will be persistent (and avoid dm-X where X is dynamically allocated number)
(I think udev rules should be still the same here if written properly.)
This requires kernel changes outside DM, like remove device name limit in partitioning code.
- fix tools to "understand" the new names
* long name in /sys limitation
* fix mapping of not udev-valid characters
* probably some tweaks for tools which sometimes prefer to display symlinks instead of kernel name (if symlink exists)
(I mean e.g. lvm user expect device like /dev/VG/LV1 (symlink) for PV report and not /dev/dm-LVM-0124-9438-1238-8129.
If there is way to avoid these tweaks and keep /dev/<internal kernel name> then of course use that:-)
(I mean tools like blkid & Co.)
Please also note that clustered LVM can complicate things, the device name (created node in dev) must be the same on all nodes in cluster etc.
Milan
--
[email protected]
p.s.
I feel that this discussion heads to flame who is dreaming and slept last years... please, no:-)
Quoted 5 lines · Milan Broz
This requires kernel changes outside DM, like remove device name limit in partitioning code.
- fix tools to "understand" the new names
* long name in /sys limitation
* long name in /proc/partitions, we have many tools (fdisk, sfdisk, blockdev, losetup, libblkid ...) where is
sscanf(line, " %d %d %llu %128[^\n ]", ...)
or less than 128 bytes ;-(
Karel
Signature · 1 line
Karel Zak <[email protected]>
Quoted 13 lines · Milan Broz
Quoted 8 lines · Kay Sievers
Quoted 4 lines · Alasdair G Kergon
Quoted 1 line
Also DM allows pretty stupid free-text device names, and names
We took the view there was no need for any restrictions beyond '/' in the kernel and it was a matter for userspace to choose the namespace.
And these days are over. /dev is not the thing anymore where everybody can mess around.
We had problems with some multipath vendors which named storage device paths using strange names/chars (according to LUN name, from hw).
Quoted 7 lines
Also dmraid sets uses names collected from proprietary metadata.
Change this means change to several tools. And there are probably still reasons to name devices according to hw provided names.
So if there is name limitation, we need to create some mapping which is acceptable both for udev and administrators/users...
Sure. But there is a general rule, that the kernel does not take any device names from untrusted sources, which includes in this case un-mangled hardware strings. There is a reason, that every subsytem in the kernel uses a basename string and an appends an enumeration number. People can do all crazy names as symlinks in /dev, but not as device node names, or kernel device names.
Quoted 11 lines
Quoted 2 lines
Sure rename them. But if you want the device node renamed, rename the kernel devices too.
Sure, but it will have consequences which must be solved too.
E.g. the dm uuid can be max. 128 characters, I am sure that we can use that in kernel for internal device name.
(uuid is not just plain UUID string, it includes prefix of subsystem, like LVM- MPATH- etc.)But how many userspace tools expects such long name in
/sys/block?
I don't know. I'm not saying that this is how DM should do it. I'm just saying that the kernel block device name and the primary device node must always match.
And that a kernel device name can not really change during the lifetime of device, unless you want to go through a real pain managing userspace here. Even then, there can be no disconnect between the kernel name and the node name.
Here again, we have the working concept of simple enumerated kernel names, and meaningful symlinks to a node with the same name, and I absolutely fail why dm wants to be special here. It is like this for everything on a hotplug system. The current nodes in /dev/mapper just don't work at all. Some distros, who care about dynamic device management, even replaced them already with symlinks today.
Quoted 3 lines
People are using udev but also lvm, dmaid, multipath, kpartx, cryptsetup, etc.
We need to find some way how to fix that and not break these systems.
Sounds good. But the previous emails suggest something totally different, they even mention to change every name we have today.
Quoted 2 lines
Maybe the symlink to
/dev/dm-Xis first step. This will switch dm to using udev, what is the primary goal now, thought.
That's what I think it is.
Quoted 1 line
and (later?),
Well, add more links, as many as you like, there is no real limit. :)
Quoted 2 lines
- introduce mandatory uuid for dm device (or disable rename for devices which have no uuid set and keep the old dm-X name for them?).
Sounds nice to me.
Quoted 3 lines
- use dm uuid, so the internal kernel device name will be persistent (and avoid dm-X where X is dynamically allocated number)
(I think udev rules should be still the same here if written properly.)
That could work, yes.
Quoted 2 lines
This requires kernel changes outside DM, like remove device name limit in partitioning code.
Which is a good thing to do, yes.
Quoted 2 lines
- fix tools to "understand" the new names
* long name in /sys limitation
That should be easy to fix, and also worth to do, not only for dm.
Quoted 1 line
* fix mapping of not udev-valid characters
We just need to prevent people from adding names like "dm-rm -rf /", because some links in /dev are created from pretty much untrusted sources.
Quoted 2 lines
* probably some tweaks for tools which sometimes prefer to display symlinks instead of kernel name (if symlink exists)
They should probably just show all currently known names, if asked for. There is never only a single name for a device, besides maybe the uuid can be seen as one sometimes. Many other names are only valid at a certain point in time, or identify multiple devices at the same time.
Quoted 5 lines
(I mean e.g. lvm user expect device like
/dev/VG/LV1(symlink) for PV report and not/dev/dm-LVM-0124-9438-1238-8129.
If there is way to avoid these tweaks and keep /dev/<internal kernel name> then of course use that:-)
(I mean tools like blkid & Co.)
Sure, sounds good.
Quoted 2 lines
Please also note that clustered LVM can complicate things, the device name (created node in dev) must be the same on all nodes in cluster etc.
Oh, "it's just the device number that is important", "users have to cope with that" was an earlier statement in this thread. :)
Quoted 2 lines
I feel that this discussion heads to flame who is dreaming and slept last years... please, no:-)
Well, I'm looking for _integration_ and not for more hacks. I also don't look for new features in an almost completely broken subsystem regarding hotplug setups. This is about making the current stuff work, and not to introduce new features, or changes nobody needs at this moment.
Don't take that personally please, I'm just a bit annoyed by the stuff that is brought up here now, and all the words spoken, with no action taken, during the last five years.
I absolutely appreciate the effort, to change things now, and I'm happy that it looks like this time is something happen for real, but I also completely disagree with parts of the proposals made here.
And I seriously ask you guys to work on _integration_ and _fixes_, and not on new features or new names in /dev, or whatever else which just make things even more complicated as they already are.
And integration means, that the kernel device name, and the kernel error log, and the /dev nodes match. That's how Linux works today.
And meaningful device names, which can be many, and which can change during the device lifetime, are done by symlinks, and not by renaming device nodes.
Thanks, and sorry,
Kay
Quoted 12 lines · Milan Broz
Quoted 7 lines · Kay Sievers
Quoted 4 lines · Alasdair G Kergon
Quoted 1 line
Also DM allows pretty stupid free-text device names, and names
We took the view there was no need for any restrictions beyond '/' in the kernel and it was a matter for userspace to choose the namespace.
And these days are over. /dev is not the thing anymore where everybody can mess around.
We had problems with some multipath vendors which named storage device paths using strange names/chars (according to LUN name, from hw).
Quoted 8 lines
Also dmraid sets uses names collected from proprietary metadata.
Change this means change to several tools. And there are probably still reasons to name devices according to hw provided names.
So if there is name limitation, we need to create some mapping which is acceptable both for udev and administrators/users...
It's not a matter of the name limitation, it's a matter of the kernel device naming policy.
Every kernel device is named <prefix><iterator>. Full stop.
I really would think twice before changing this.
But back to the real discussion:
From what I've glanced here is that you're trying to solve the eternal device mapping question (yet again):
Which device (as logged in /var/log/messages) corresponds to which logical device as seen by the internal tools.
And similar I thought that this question was shoved over to userspace/sysadmin, as the in-kernel device names are strictly speaking valid for the running instance _only_. Even a reboot might trigger a different mapping scheme. And anybody really caring about this will have some sort of syslog managing tool, which then can easily be fed with the appropriate naming table.
And we had this discussion over and over again on the scsi mailing list (just look for 'stable device naming') and the consensus was that in-kernel device naming is _not_ stable and we should not try to make it so.
I would strongly suspect the same argument holds here, too. You're heading to a flamewar if you try to implement some in-kernel stable device naming.
Where is the problem with just having the 'dm-X' as real device nodes and everything undev /dev/mapper as symlinks to that?
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Markus Rex, HRB 16746 (AG Nürnberg)
Quoted 3 lines · Hannes Reinecke
Where is the problem with just having the 'dm-X' as real device nodes and everything undev
/dev/mapperas symlinks to that?
Right, we've been there so many times. The problem summarizes to: If you can not have *the name* you don't do any *name* at all, and you keep the enumerated one, and add as many additional names as needed by links, for your users to pick up. That's how all other things work today.
*The name* means: unique across clusters, systems, mirrors, backups, clones, everything, and it can not change during the lifetime of the device, and the kernel and userspace share the same name. That can certainly not be true for dm, so it should not try to mess around here and pretend to have *the name* which it does not have.
Thanks,
Kay
Thanks for the feedback. Will this model be more acceptable then?
Peter
/lib/udev/rules.d
========10-dm.rules
-----------
KERNEL="device-mapper", NAME="mapper/control"
SUBSYSTEM!="block", GOTO="dm_end"
KERNEL!="dm-[0-9]*", GOTO="dm_end"
# We supress the creation of nodes on ADD events, they must be created on CHANGE events only! ACTION="add", NAME=""
ACTION!="change", GOTO="dm_end"
TEST="dm", ENV{DM_NAME}="$attr{dm/name}", ENV{DM_UUID}="$attr{dm/uuid}"
TEST!="dm", IMPORT{program}="/sbin/dmsetup info -j %M -m %m -c --nameprefixes --noheadings --rows -o name,uuid"
ENV{DM_NAME}="?*", NAME="$kernel", SYMLINK+="mapper/$env{DM_NAME}"
LABEL="dm_end"
12-lvm.rules
------------
SUBSYSTEM!="block", GOTO="lvm_end"
KERNEL!="dm-[0-9]*", GOTO="lvm_end"
ACTION!="change", GOTO="lvm_end"
ENV{DM_UUID}!="LVM-?*", GOTO="lvm_end"
IMPORT{program}="/sbin/dmsetup lvmsplit $env{DM_NAME} --nameprefixes --noheadings --rows"
ENV{DM_LV_NAME}="?*_mlog", GOTO="lvm_end" ENV{DM_LV_NAME}="?*_mimage_[0-9]*", GOTO="lvm_end"
ENV{DM_VG_NAME}="?*", ENV{DM_LV_NAME}="?*", ENV{DM_LV_LAYER}!="?*", SYMLINK+="$env{DM_VG_NAME}/$env{DM_LV_NAME}"
LABEL="lvm_end"
95-dm-notify.rules
------------------
SUBSYSTEM!="block", GOTO="dm_end"
KERNEL!="dm-[0-9]*", GOTO="dm_end"
ACTION!="change|remove", GOTO="dm_end"
ENV{DM_COOKIE}="?*", RUN+="/sbin/dmsetup udevnotify $env{DM_COOKIE}"
LABEL="dm_end"
/etc/udev/rules.d
========11-dm-permissions.rules (I don't think this is a documentation or a dead file, sorry :))
-----------------------------------------------------------------
SUBSYSTEM!="block", GOTO="dm_end"
KERNEL!="dm-[0-9]*", GOTO="dm_end"
ACTION!="change", GOTO="dm_end"
#ENV{DM_NAME}="my_device", OWNER:="peter", GROUP:="peter", MODE:="644"
# Default permissions
KERNEL="dm-[0-9]*", OWNER:="root", GROUP:="root", MODE:="600"
LABEL="dm_end"
Quoted 8 lines · Peter Rajnoha
/lib/udev/rules.d
========> 10-dm.rules
-----------
KERNEL="device-mapper", NAME="mapper/control"SUBSYSTEM!="block", GOTO="dm_end"
KERNEL!="dm-[0-9]*", GOTO="dm_end"
Quoted 6 lines
# We supress the creation of nodes on ADD events, they must be created on CHANGE events only! ACTION="add", NAME=""
ACTION!="change", GOTO="dm_end"TEST="dm", ENV{DM_NAME}="$attr{dm/name}", ENV{DM_UUID}="$attr{dm/uuid}"
TEST!="dm", IMPORT{program}="/sbin/dmsetup info -j %M -m %m -c --nameprefixes --noheadings --rows -o name,uuid"
Wasn't there something that added the DM_* variables to the kernel directly? I remember seing something a while back. If that is still planned or possible, maybe you can just do thes both rules only if the values are not set?
ENV{DM_NAME}="", ENV{DM_NAME}="$attr{dm/name}", ... ENV{DM_NAME}="", IMPORT{program}= ...
Quoted 1 line
ENV{DM_NAME}="?*", NAME="$kernel", SYMLINK+="mapper/$env{DM_NAME}"
NAME="$kernel" is the default, it's not needed.
Quoted 38 lines
LABEL="dm_end"
12-lvm.rules
------------
SUBSYSTEM!="block", GOTO="lvm_end"
KERNEL!="dm-[0-9]*", GOTO="lvm_end"
ACTION!="change", GOTO="lvm_end"
ENV{DM_UUID}!="LVM-?*", GOTO="lvm_end"IMPORT{program}="/sbin/dmsetup lvmsplit $env{DM_NAME} --nameprefixes --noheadings --rows"
ENV{DM_LV_NAME}="?*_mlog", GOTO="lvm_end" ENV{DM_LV_NAME}="?*_mimage_[0-9]*", GOTO="lvm_end"
ENV{DM_VG_NAME}="?*", ENV{DM_LV_NAME}="?*", ENV{DM_LV_LAYER}!="?*", SYMLINK+="$env{DM_VG_NAME}/$env{DM_LV_NAME}"
LABEL="lvm_end"
95-dm-notify.rules
------------------
SUBSYSTEM!="block", GOTO="dm_end"
KERNEL!="dm-[0-9]*", GOTO="dm_end"
ACTION!="change|remove", GOTO="dm_end"ENV{DM_COOKIE}="?*", RUN+="/sbin/dmsetup udevnotify $env{DM_COOKIE}"
LABEL="dm_end"
/etc/udev/rules.d
========> 11-dm-permissions.rules (I don't think this is a documentation or a dead file, sorry :))
-----------------------------------------------------------------
SUBSYSTEM!="block", GOTO="dm_end"
KERNEL!="dm-[0-9]*", GOTO="dm_end"
ACTION!="change", GOTO="dm_end"
Quoted 6 lines
#ENV{DM_NAME}="my_device", OWNER:="peter", GROUP:="peter", MODE:="644"
# Default permissions
KERNEL="dm-[0-9]*", OWNER:="root", GROUP:="root", MODE:="600"LABEL="dm_end"
Yeah, looks great.
Thanks,
Kay
Quoted 10 lines · Kay Sievers
Quoted 2 lines
TEST="dm", ENV{DM_NAME}="$attr{dm/name}", ENV{DM_UUID}="$attr{dm/uuid}"
TEST!="dm", IMPORT{program}="/sbin/dmsetup info -j %M -m %m -c --nameprefixes --noheadings --rows -o name,uuid"Wasn't there something that added the DM_* variables to the kernel directly? I remember seing something a while back. If that is still planned or possible, maybe you can just do thes both rules only if the values are not set?
ENV{DM_NAME}="", ENV{DM_NAME}="$attr{dm/name}", ... ENV{DM_NAME}="", IMPORT{program}= ...
I think these are multipath uevents only. But uevents for plain DM devices don't have these vars set. But yes, we can save a little time in that particular situation.
Quoted 3 lines
Quoted 1 line
ENV{DM_NAME}="?*", NAME="$kernel", SYMLINK+="mapper/$env{DM_NAME}"
NAME="$kernel" is the default, it's not needed.
..ok
Quoted 1 line
Yeah, looks great.
Thanks! :)
Peter