Skip to contentSkip to conversation

1.1.0rc2: device-mapper: remove ioctl failed: Device or resource busy

dm-crypt Thomas Bächler 4 participants 12 messages Oct 20, 2009 – Oct 23, 2009 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

Okay, this report has a long story. I had reports of cryptsetup 1.0.7 failing to work after updates (I later found out that the cause was installing devicekit). I advised the affected people to try the 1.1.0 release candidate to get better debug output - but the bug was gone with 1.1.0. But something else happened with 1.1.0: On every opened LUKS volume, the message

device-mapper: remove ioctl failed: Device or resource busy

would appear (see http://bugs.archlinux.org/task/16735#comment51508).

Then I suggested adding the following udev rule (which I will probably make the default):

ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="dm-[0-9]*", PROGRAM="/sbin/dmsetup info -c --noopencount --noheadings -o name -j %M -m %m", RESULT=="temporary-cryptsetup-*", OPTIONS="last_rule"

(btw, is there a better way than calling dmsetup to get the name here?). Now, this fixes the problem with 1.0.7, as it prevents devicekit from blocking access to the device, but the above error message isn't gone in 1.1.0: http://bugs.archlinux.org/task/16735#comment51536

What is this remove error caused by, if not by some application being called from udev blocking access? Note that there are no temporary-cryptsetup-* volumes left after luksOpen.

Ideas welcome :)

#1 lore raw

Hi Thomas,

currently there is discussion between dm/lvm developers and some maintainers how to unify DM udev rules so these problems are fixed definitely. (Final state should be that applications using libdevmapper will use udev to create nodes and unified udev rules are part of libdevmapper package.)

CCing Peter - I am sure he can explain your questions better than me.

Quoted 11 lines · Thomas Bächler

Okay, this report has a long story. I had reports of cryptsetup 1.0.7 failing to work after updates (I later found out that the cause was installing devicekit). I advised the affected people to try the 1.1.0 release candidate to get better debug output - but the bug was gone with 1.1.0. But something else happened with 1.1.0: On every opened LUKS volume, the message

device-mapper: remove ioctl failed: Device or resource busy

would appear (see http://bugs.archlinux.org/task/16735#comment51508).

Quoted 8 lines

Then I suggested adding the following udev rule (which I will probably make the default):

ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="dm-[0-9]*", PROGRAM="/sbin/dmsetup info -c --noopencount --noheadings -o name -j %M -m %m", RESULT=="temporary-cryptsetup-*", OPTIONS="last_rule"

(btw, is there a better way than calling dmsetup to get the name here?).

BTW for recent kernel, name and UUID is in sysfs directly. We have already some rule in dm package.

Quoted 7 lines

Now, this fixes the problem with 1.0.7, as it prevents devicekit from blocking access to the device, but the above error message isn't gone in 1.1.0: http://bugs.archlinux.org/task/16735#comment51536

What is this remove error caused by, if not by some application being called from udev blocking access? Note that there are no temporary-cryptsetup-* volumes left after luksOpen.

No idea - but always I debubugged it, it was initiated from udev event through udev rule.

I'll add better debug message to next cryptsetup rc (including process
+ parent process name, I hope I can get it from /proc in debugging mode) into debug output to allow more easy debugging of that problem - I do not want to hide it by some workaround, it must be fixed.

(Surely nobody want an application randomly reading opened, decrypted internal keyslot device.)

Milan
--
[email protected]

#2 lore raw
Quoted 9 lines · Milan Broz

Hi Thomas,

currently there is discussion between dm/lvm developers and some maintainers how to unify DM udev rules so these problems are fixed definitely. (Final state should be that applications using libdevmapper will use udev to create nodes and unified udev rules are part of libdevmapper package.)

CCing Peter - I am sure he can explain your questions better than me.

I would be very interested in this, I'll look up the discussion.

Quoted 11 lines
Quoted 8 lines

Then I suggested adding the following udev rule (which I will probably make the default):

ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="dm-[0-9]*", PROGRAM="/sbin/dmsetup info -c --noopencount --noheadings -o name -j %M -m %m", RESULT=="temporary-cryptsetup-*", OPTIONS="last_rule"

(btw, is there a better way than calling dmsetup to get the name here?).

BTW for recent kernel, name and UUID is in sysfs directly. We have already some rule in dm package.

$ cat /sys/block/dm-0/dm/name
lvm_pv
$ cat /sys/block/dm-0/dm/suspended
0
$ cat /sys/block/dm-0/dm/uuid
CRYPT-4e80f75b-77d2-465c-ba8c-9480493dd717

This is awesome. How new does the kernel have to be? Some people are forcing me to support ancient kernels (like 2.6.27), while personally I'd rather only support latest stable.

Quoted 10 lines
Quoted 7 lines

Now, this fixes the problem with 1.0.7, as it prevents devicekit from blocking access to the device, but the above error message isn't gone in 1.1.0: http://bugs.archlinux.org/task/16735#comment51536

What is this remove error caused by, if not by some application being called from udev blocking access? Note that there are no temporary-cryptsetup-* volumes left after luksOpen.

No idea - but always I debubugged it, it was initiated from udev event through udev rule.

The rule above should effectively keep udev from spawning any process that will do anything to the device - but yes, we need to debug it. I have to see if I can set up a test case on the weekend so I can debug this - until now, I have not seen this problem myself.

Quoted 7 lines

I'll add better debug message to next cryptsetup rc (including process
+ parent process name, I hope I can get it from /proc in debugging mode) into debug output to allow more easy debugging of that problem - I do not want to hide it by some workaround, it must be fixed.

(Surely nobody want an application randomly reading opened, decrypted internal keyslot device.)

That would be great. For now I am happy that the affected users have working computers again :)

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw
Quoted 4 lines · Peter Rajnoha

Well, we are in the process of the changeover now. As Milan says, what we're trying to do is to centralize the udev support for DM devices. There were (are?) bits of the rules touching dm devices all over the udev rule sets from other packages.

Is a public discussion happening somewhere? I'd like to follow it or even participate, as I will be the one who has to fix up this mess in the end for Arch.

Quoted 17 lines
Quoted 4 lines
Quoted 4 lines

ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="dm-[0-9]*", PROGRAM="/sbin/dmsetup info -c --noopencount --noheadings -o name -j %M -m %m", RESULT=="temporary-cryptsetup-*", OPTIONS="last_rule"

Yes, something is definitely touching it. We have OPTIONS+="last_rule" in lvm/dm upstream udev rules for temporary-cryptsetup devices now. But the thing is that using last_rule is also a source of the complaints and people just do not want to see this (for example, this was also the first thing that Debian lvm maintainer removed and diverted from the upstream).

Frankly, as for me, I would use last_rule here (as well as for hidden lvm devices and all devices that are temporary, hidden or special in any way so it has no meaning to process any other rules for them further). I asked Kay Sievers (from udev) what's his opinion about this and I got simple and straight answer from him:

Quoted 4 lines

"I can not think of any valid use of last_rule. Also, it will still send out the events to listeners, as this can never be suppressed. last_rule should probably just be removed, as is breaks all sorts of things and creates an inconsistent behavior."

If there is no mechanism to tell udev to ignore a certain device, shouldn't it be possible to tell the dm subsystem to not raise a uevent for a device? I imagine setting a "temporary" flag.

Quoted 10 lines

I've posted patches on lvm-devel that introduce udev flags with which we can disable certain parts of the udev rules. Then these flags could be checked for and inappropriate parts of udev rules skipped (considering upstream):

DM_UDEV_FLAG_DISABLE_DISK_RULES - to disable 12-dm-disk.rules

DM_UDEV_FLAG_DISABLE_SUBSYSTEM_RULES - disable any subsystem rules, like 11-lvm.rules
for LVM... The same could be applied for any other subsystem supplying its own rules.

Quoted 5 lines

These flags could be set directly in libdevmapper for each device being processed (but that's only for kernels >= 2.6.31, because we use cookie values to pass this information and they were introduced only in 2.6.31). Or we can set these flags directly in udev rules (like I'm proposing now on lvm-devel where I removed last_rule for temporary-cryptsetup and like and set these flags directly).

I don't get how this is better than last_rule, except it is more work and adds extra complexity. All I want is that temporary devices are being left alone completely.

Your quote from Kay Sievers also mentions events being sent to listeners, which can "never be suppressed" - the fact is that we want/need to suppress them - what you suggest doesn't change that.

Anyway, looking at the udev manpage, there seems to be a "ignore_device" option, is that better than last_rule? Shouldn't it fit our situation better? Or is it yet another option we shouldn't use?

Quoted 4 lines

But such solution (I mean the one without last_rule) expects that everybody else will be playing device-mapper rules as well. It means always checking these flags and skip the rules if it is instructed to do so. This solution is much cleaner from udev perspective since it does not use that last_rule functionality...

Not only dm-specific rules but all rules everywhere need to be aware of this. There's also site/user-specific rules which could break and and and - as I said, extra complexity.

Quoted 4 lines

I know, the solution with OPTIONS+="last_rule" would be much easier, since it ignores any rules further, but at least this way we can spot all the places where dm devices are touched by the rules and we can send a notice to those maintainers. Otherwise this would be hidden forever.

Good point - but I still think that telling udev to leave the device alone completely feels like the right thing to do.

Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

What is this remove error caused by, if not by some application being called from udev blocking access? Note that there are no temporary-cryptsetup-* volumes left after luksOpen.

No idea - but always I debubugged it, it was initiated from udev event through udev rule.

Quoted 7 lines

..hmm, this is bound to the actual ruleset that is used in each distro. Each one should be cleaned and any help is a welcome, really. Also, if you spot any strange behaviour related to DM devices and udev, just feel free to CC me.

Also, I would be very glad if all the rules in all distros would be in sync with upstream ones. If we could achieve that, it would be great, so we would have one base and we don't have to solve various problems over and over again.

We're trying to do that, but I think there are some legacies in our packages - device-mapper didn't have upstream udev rules in the past, so we added one to get the /dev/mapper/* devices created. That rule is still there, and probably nobody checked if upstream included something themselves.

Quoted 1 line

..this is Arch Linux, right? I'll take a look...

Indeed.

Our udev rules are a big mess right now for various reasons I wanted to address soon. All I can say is that the only dm-specific rule is (included in the device-mapper package):

ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="dm-[0-9]*", PROGRAM="/sbin/dmsetup info -c --noopencount --noheadings -o name -j %M -m %m", NAME="mapper/%c", MODE="0600", SYMLINK+="disk/by-name/%c"

Then, the devicekit package also has some rules specific to dm, which I didn't look at yet (I guess they are just the upstream devicekit rules, we tend to not change them when possible).

The problematic rules are the ones that will just touch any block device - none of those has an explicit check to include or exclude dm-*. I think this should only be blkid, but I am not sure.

#5 lore raw
Quoted 10 lines · Thomas Bächler
Quoted 4 lines · Peter Rajnoha

Well, we are in the process of the changeover now. As Milan says, what we're trying to do is to centralize the udev support for DM devices. There were (are?) bits of the rules touching dm devices all over the udev rule sets from other packages.

Is a public discussion happening somewhere? I'd like to follow it or even participate, as I will be the one who has to fix up this mess in the end for Arch.

As for LVM/device-mapper related things, there's an "lvm-devel" mailing list and we can discuss the things there as well...

Quoted 11 lines
Quoted 10 lines

These flags could be set directly in libdevmapper for each device being processed
(but that's only for kernels >= 2.6.31, because we use cookie values to pass this
information and they were introduced only in 2.6.31). Or we can set these flags
directly in udev rules (like I'm proposing now on lvm-devel where I removed
last_rule for temporary-cryptsetup and like and set these flags directly).

Quoted 4 lines

I don't get how this is better than last_rule, except it is more work and adds extra complexity. All I want is that temporary devices are being left alone completely.

Well, it's also used to solve some specific situations where udev rules are not enough, e.g. we need specific lvm devices to be visible for a while and then make it invisible later on. If we used plain udev rules, we would set filter based on names only. But since these devices are not renamed in the process, we need to get this information somehow. Since we use those cookies for udev synchronisation, we reuse them to pass these flags into the udev rules, too...

I made it more general, so others can use it to pass any bit information to udev rules (we have 16 flags this way).

Quoted 4 lines

Your quote from Kay Sievers also mentions events being sent to listeners, which can "never be suppressed" - the fact is that we want/need to suppress them - what you suggest doesn't change that.

If I got him right, I think he meant direct listeners of the "KERNEL" udev events like udevd does. Yes, we can't do much here - if anybody listens to the events this way, he is on his own (if we listen to UDEV udev events, then these ones will have those env vars set, so one can check them).

Quoted 4 lines

Anyway, looking at the udev manpage, there seems to be a "ignore_device" option, is that better than last_rule? Shouldn't it fit our situation better? Or is it yet another option we shouldn't use?

Yes, ignore_device is another way how to suppress the events somehow. But this one will ignore all the rules irrespective of the sequence when it's called. When I tried this one first, I had a rule that sets the nodes and symlinks in /dev/mapper and just after that I called ignore_device, but it ignored everything. So this one can't be used either.

Quoted 21 lines
Quoted 8 lines

But such solution (I mean the one without last_rule) expects that everybody else
will be playing device-mapper rules as well. It means always checking these
flags and skip the rules if it is instructed to do so. This solution is much cleaner
from udev perspective since it does not use that last_rule functionality...

Not only dm-specific rules but all rules everywhere need to be aware of this. There's also site/user-specific rules which could break and and and - as I said, extra complexity.

Quoted 7 lines

I know, the solution with OPTIONS+="last_rule" would be much easier, since it
ignores any rules further, but at least this way we can spot all the places where
dm devices are touched by the rules and we can send a notice to those maintainers.
Otherwise this would be hidden forever.

Quoted 3 lines

Good point - but I still think that telling udev to leave the device alone completely feels like the right thing to do.

Hmm, it's hard to decide... But yes, it's a point to think of.

Quoted 34 lines
Quoted 13 lines
Quoted 5 lines
Quoted 3 lines

What is this remove error caused by, if not by some application being called from udev blocking access? Note that there are no temporary-cryptsetup-* volumes left after luksOpen.

No idea - but always I debubugged it, it was initiated from udev event through udev rule.

..hmm, this is bound to the actual ruleset that is used in each distro. Each
one should be cleaned and any help is a welcome, really. Also, if you spot any
strange behaviour related to DM devices and udev, just feel free to CC me.

Quoted 6 lines

Also, I would be very glad if all the rules in all distros would be in sync
with upstream ones. If we could achieve that, it would be great, so we would
have one base and we don't have to solve various problems over and over again.

We're trying to do that, but I think there are some legacies in our packages - device-mapper didn't have upstream udev rules in the past, so we added one to get the /dev/mapper/* devices created. That rule is still there, and probably nobody checked if upstream included something themselves.

Quoted 1 line

..this is Arch Linux, right? I'll take a look...

Indeed.

Our udev rules are a big mess right now for various reasons I wanted to address soon. All I can say is that the only dm-specific rule is (included in the device-mapper package):

Quoted 8 lines

ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="dm-[0-9]*", PROGRAM="/sbin/dmsetup info -c --noopencount --noheadings -o name -j %M -m %m", NAME="mapper/%c", MODE="0600", SYMLINK+="disk/by-name/%c"

Then, the devicekit package also has some rules specific to dm, which I didn't look at yet (I guess they are just the upstream devicekit rules, we tend to not change them when possible).

..that part should be removed as soon as we have all the things resolved and once we have fully functional udev support in Fedora rawhide. David Zeuthen who is responsible for devicekit said he would do that change then. So I believe that this will propagate to the upstream gradually as well.

Quoted 4 lines

The problematic rules are the ones that will just touch any block device
- none of those has an explicit check to include or exclude dm-*. I think this should only be blkid, but I am not sure.

Yes, indeed, blkid call is the source of most problems...

Peter

#6 lore raw
Quoted 4 lines · Peter Rajnoha

If I got him right, I think he meant direct listeners of the "KERNEL" udev events like udevd does. Yes, we can't do much here - if anybody listens to the events this way, he is on his own (if we listen to UDEV udev events, then these ones will have those env vars set, so one can check them).

Okay - which program would do that?

Quoted 5 lines

Yes, ignore_device is another way how to suppress the events somehow. But this one will ignore all the rules irrespective of the sequence when it's called. When I tried this one first, I had a rule that sets the nodes and symlinks in /dev/mapper and just after that I called ignore_device, but it ignored everything. So this one can't be used either.

Does it matter? If I remember correctly, libdevmapper creates the /dev/mapper/* nodes itself even if udev isn't there. So cryptsetup would create the temporary-cryptsetup-* node, access it and destroy it and udev would ignore everything else - sounds like a good solution to me.

I will deploy the rule mentioned in earlier posts as a workaroud for now and then see what is happening upstream - once the upstream rules are in a good state, I am more than willing to use those rather than distro-specific ones.

#7 lore raw
Quoted 10 lines · Thomas Bächler
Quoted 6 lines · Peter Rajnoha

If I got him right, I think he meant direct listeners of the "KERNEL" udev
events like udevd does. Yes, we can't do much here - if anybody listens to
the events this way, he is on his own (if we listen to UDEV udev events, then these ones will have those env vars set, so one can check them).

Okay - which program would do that?

Actually, I can't think of anyone listening to the events this way right now... If there's anybody, they really need to have a reason to do that.

Quoted 8 lines
Quoted 7 lines

Yes, ignore_device is another way how to suppress the events somehow. But this one will ignore all the rules irrespective of the sequence when it's called.
When I tried this one first, I had a rule that sets the nodes and symlinks
in /dev/mapper and just after that I called ignore_device, but it ignored everything. So this one can't be used either.

Quoted 4 lines

Does it matter? If I remember correctly, libdevmapper creates the /dev/mapper/* nodes itself even if udev isn't there. So cryptsetup would create the temporary-cryptsetup-* node, access it and destroy it and udev would ignore everything else - sounds like a good solution to me.

Yes, it does create the nodes itself if udev fails to do so. But I don't think this would be a clean solution. Either we create the nodes by udev or by libdevmapper itself (when the rules are not installed). The thing we kept the node/symlink creation code in libdevmapper even with udev support turned on - that's just a fallback action. And it would always give you a warning message like "<node> not set up by udev. Falling back to direct node creation.".

So I wouldn't go this way...

Quoted 6 lines

I will deploy the rule mentioned in earlier posts as a workaroud for now and then see what is happening upstream - once the upstream rules are in a good state, I am more than willing to use those rather than distro-specific ones.

Yes, sure. We have to do this until we have all the changes propagated.

Peter

#8 lore raw
In reply to Thomas Bächler
Quoted 11 lines · Thomas Bächler
Quoted 7 lines

I know, the solution with OPTIONS+="last_rule" would be much easier, since it
ignores any rules further, but at least this way we can spot all the places where
dm devices are touched by the rules and we can send a notice to those maintainers.
Otherwise this would be hidden forever.

Good point - but I still think that telling udev to leave the device alone completely feels like the right thing to do.

..ok, finally we kept the "last_rule" in the lvm/dm upstream. This is the safe way. Maybe later we can think of a solution where we will rely on the others so they will check the variables we set for them (if that's possible at all). But it would be too dangerous now, I have to admit.

You can check the rules we have in the upstream - if you have any comments or hints, just feel free to write me back...

Peter

#9 lore raw
Quoted 7 lines · Peter Rajnoha

..ok, finally we kept the "last_rule" in the lvm/dm upstream. This is the safe way. Maybe later we can think of a solution where we will rely on the others so they will check the variables we set for them (if that's possible at all). But it would be too dangerous now, I have to admit.

You can check the rules we have in the upstream - if you have any comments or hints, just feel free to write me back...

I am going to replace our own device-mapper rules with upstream rules (I think when we made our own rules, dm didn't have any, or I overlooked them).

#10 lore raw
Quoted 7 lines · Thomas Bächler
Quoted 6 lines · Peter Rajnoha

..ok, finally we kept the "last_rule" in the lvm/dm upstream. This is the safe way. Maybe later we can think of a solution where we will rely on the
others so they will check the variables we set for them (if that's possible at all). But it would be too dangerous now, I have to admit.

Quoted 8 lines
Quoted 3 lines

You can check the rules we have in the upstream - if you have any comments
or hints, just feel free to write me back...

I am going to replace our own device-mapper rules with upstream rules (I think when we made our own rules, dm didn't have any, or I overlooked them).

Well, official udev support is quite new (I think first rules were added sometime in August together with udev synchronisation interface in libdevmapper that could be used to synchronise udev actions with the code using libdevmapper itself, so there's no race between the udev creating the nodes and devmapper fallback - today only dmsetup and LVM uses this).

Maybe some important notes and problems we already know about and track:

- these rules change the layout in /dev a little:
- the nodes are created in /dev directly with the name of dm-X (this is internal kernel name that is not the same as actual DM name)
- the symlinks are created in /dev/mapper
(these ones use DM name -- like the old layout, but they're symlinks now,
that's the only difference)

- consenquences of using symlinks instead of nodes in /dev/mapper:
- mount utility follows those symlinks and writes mtab with dm-X name, not the actual DM name (so, for example, when someone uses "df" now, he will see those dm-X names instead of actual nice DM names)
- there's a known problem with GRUB2 that can't deal with symlinks in /dev/mapper (it's grub-probe utility that fails iirc, GRUB1 is OK though..)

- calling "udevadm trigger" itself - this one generates ADD udev events by default and we suppress the node creation on ADD. However, if the node already exists and you call udevadm trigger, you will loose the nodes and you will end up with dangling symlinks in /dev/mapper. This is because of how udev works.. This either needs to be fixed somehow in udev directly (I've written Kay about this specific situation already, but actually don't know about a solution right now :) ) or, the temporary solution is just to call "udevadm trigger --action=change" if you want those nodes and symlinks back.

OK, thanks... If you spot any other problems, please let me know. We have to catch all these initial bugs.

Peter

#11 lore raw
Quoted 5 lines · Peter Rajnoha

Maybe some important notes and problems we already know about and track:

- these rules change the layout in /dev a little:
- the nodes are created in /dev directly with the name of dm-X (this is internal kernel name that is not the same as actual DM name)

Is this really necessary? I tried to find out what the motivation is for that, and I came across a posting from you where you state:
"The reason for doing so is that creating nodes root /dev directory is
considered to be the "standard way"."

I'm currently on Ubuntu 9.04 and I don't have any /dev/dm-* devices, only /dev/mapper/* (eg. "raid5-root"), which have much more meaning for humans. The system works, so there seems to be no actual need for /dev/dm-* devices.

Maybe it's time to loosen the "standard way". I have the strong feeling that we're going backwards with respect to usability if eg. df reports meaningless /dev/dm-* names, so the human has to look up what this actually means.

Yours, Uwe

#12 lore raw
Quoted 6 lines · Uwe Menges

Peter Rajnoha wrote:

I'm currently on Ubuntu 9.04 and I don't have any /dev/dm-* devices, only /dev/mapper/* (eg. "raid5-root"), which have much more meaning for humans. The system works, so there seems to be no actual need for /dev/dm-* devices.

This was requirement from udev side.
Initially the "dm-X" device were never intended to be visible in userspace...

Anyway, seems that there is lot of discussion around this problem, please can we move this generic discussion to more appropriate list? (dm-devel, lvm-devel or udev/hotplug ?)

Quoted 4 lines

Maybe it's time to loosen the "standard way". I have the strong feeling that we're going backwards with respect to usability if eg. df reports meaningless /dev/dm-* names, so the human has to look up what this actually means.

I fully agree. Nobody want to display dm-X devices to users. (and btw df is fixed already upstream to not display these devices and prefer user friendly names)

But please read this thread first for context...
http://thread.gmane.org/gmane.linux.hotplug.devel/13876/focus=13877

Milan
--
[email protected]