Kernel update: "Failed to access temporary keystore device."
Hi all,
I just tried to upgrade my kernel from 3.10.48 to 3.14.15 (kernel.org). This is Debian wheezy. After the update, I get "Failed to access temporary keystore device." when trying to unlock my LUKS partitions. As far as I can tell I have not changed anything relevant in the kernel config, I just did a "make oldconfig" with the old kernel .config.
What is missing here or how do I debug this?
Gr"usse,
Arno
Signature · 4 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., Email: [email protected] GnuPG: ID: CB5D9718 FP: 12D6 C03B 1B30 33BB 13CF B774 E35C 5FA1 CB5D 9718 ---- A good decision is based on knowledge and not on numbers. - Plato
Quoted 6 lines · Arno Wagner
I just tried to upgrade my kernel from 3.10.48 to 3.14.15 (kernel.org). This is Debian wheezy. After the update, I get "Failed to access temporary keystore device." when trying to unlock my LUKS partitions. As far as I can tell I have not changed anything relevant in the kernel config, I just did a "make oldconfig" with the old kernel .config.
Hi,
for some strange reason I am tempting to ask if you read the FAQ but... ;-)
Well, seriously: this happens when temporary mapped keyslot device cannot be read (but kernel mapping was created successfully). Not common problem, I do not even remember someone reported this...
It seems like some udev/kernel compatibility problem (Debian has non-standard dm/lvm udev rules btw).
Either bad access rights to device node or device node is missing (the second is probably the issue).
It is possible you will need to use new udev or something.
Can you paste the command with added --debug?
Can you try to boot Debian provided kernel - does it work? (Anyway, I am using custom kernel in Debian for years without problem but I am using unstable repo.)
Milan
Quoted 7 lines · Milan Broz
Quoted 6 lines · Arno Wagner
I just tried to upgrade my kernel from 3.10.48 to 3.14.15 (kernel.org). This is Debian wheezy. After the update, I get "Failed to access temporary keystore device." when trying to unlock my LUKS partitions. As far as I can tell I have not changed anything relevant in the kernel config, I just did a "make oldconfig" with the old kernel .config.
Hi Milan,
Quoted 5 lines
Hi,
for some strange reason I am tempting to ask if you read the FAQ but... ;-)
I assure you, I did. The FAQ writer has never heard of this problem ;-)
Quoted 6 lines
Well, seriously: this happens when temporary mapped keyslot device cannot be read (but kernel mapping was created successfully). Not common problem, I do not even remember someone reported this...
It seems like some udev/kernel compatibility problem (Debian has non-standard dm/lvm udev rules btw).
One more reason not to like udev. It used to be that you just created the right devices manually and things worked...
Quoted 5 lines
Either bad access rights to device node or device node is missing (the second is probably the issue).
It is possible you will need to use new udev or something.Can you paste the command with added --debug?
See below, both for 1.6.1 and 1.6.5, which unloaks without error (well, without error that gets propagated to the user), but never creates the entry in /dev/mapper/. Likely a bug in 1.6.5, as it probably should tell the user that things went wrong.
Quoted 1 line
Can you try to boot Debian provided kernel - does it work?
Not easily. But it does work with 3.10.51, so the 3.2.x that Debian stable is stuck at should probably work too.
Come to think of it, I have /usr/src/linux pointing to a 3.4.67 source tree, as gcc kernel includes in Debian stable are really messed up with 3.5.x and later and I failed to fix it manually. (Sometimes I really wonder what the Kernel devs are thinking or whether they are thinking at all...) Could that be the problem?
Quoted 2 lines
(Anyway, I am using custom kernel in Debian for years without problem but I am using unstable repo.)
I usually run testing, except that I really do not want systemd, so until I am sure I can do that update without getting that atrocity, no update to jessy for me.
Anyways, if we do not figure this one out, I will just stay with 3.10.x, it is a longterm-kernel after all. I just tried 3.14.15 because I have some network issues and wanted to see whether they may be gone with a newer kernel.
Arno
1.6.5:
# cryptsetup 1.6.5 processing # "/home/wagner/tools/cryptsetup/cryptsetup-1.6.5/src/.libs/lt-cryptsetup # --debug luksOpen /dev/md10 c1" # Running command open. # Locking memory. # Installing SIGINT/SIGTERM handler. # Unblocking interruption on signal. # Allocating crypt device /dev/md10 context. # Trying to open and read device /dev/md10. # Initialising device-mapper backend library. # Trying to load LUKS1 crypt type from device /dev/md10. # Crypto backend (gcrypt 1.5.0, flawed whirlpool) initialized. # Reading LUKS header of size 1024 from device /dev/md10 # Key length 32, device size 419430272 sectors, header size 2050 sectors. # Timeout set to 0 miliseconds. # Password retry count set to 3. # Password verification disabled. # Iteration time set to 1000 miliseconds. # Activating volume c1 [keyslot -1] using [none] passphrase. # Detected kernel Linux 3.14.15 x86_64. # dm version OF [16384] (*1) # dm versions OF [16384] (*1) # Detected dm-verity version 1.2.0. # Detected dm-crypt version 1.13.0, dm-ioctl version 4.27.0. # Device-mapper backend running with UDEV support enabled. # dm status c1 OF [16384] (*1) # Interactive passphrase entry requested. Enter passphrase for /dev/md10: # Trying to open key slot 0 [ACTIVE]. # Reading key slot 0 area. # Using userspace crypto wrapper to access keyslot area. # Releasing crypt device /dev/md10 context. # Releasing device-mapper backend. # Unlocking memory. Command failed with code 5: Input/output error
1.6.1:
# cryptsetup 1.6.1 processing "cryptsetup --debug luksOpen /dev/md10 c1" # Running command open. # Locking memory. # Installing SIGINT/SIGTERM handler. # Unblocking interruption on signal. # Allocating crypt device /dev/md10 context. # Trying to open and read device /dev/md10. # Initialising device-mapper backend library. # Trying to load LUKS1 crypt type from device /dev/md10. # Crypto backend (gcrypt 1.5.0) initialized. # Reading LUKS header of size 1024 from device /dev/md10 # Key length 32, device size 419430272 sectors, header size 2050 sectors. # Timeout set to 0 miliseconds. # Password retry count set to 3. # Password verification disabled. # Iteration time set to 1000 miliseconds. # Activating volume c1 [keyslot -1] using [none] passphrase. # dm version OF [16384] (*1) # dm versions OF [16384] (*1) # Detected dm-verity version 1.2.0. # Detected dm-crypt version 1.13.0, dm-ioctl version 4.27.0. # Device-mapper backend running with UDEV support enabled. # dm status c1 OF [16384] (*1) # Interactive passphrase entry requested. Enter passphrase for /dev/md10: # Trying to open key slot 0 [ACTIVE]. # Reading key slot 0 area. # Calculated device size is 250 sectors (RW), offset 8. # DM-UUID is CRYPT-TEMP-temporary-cryptsetup-17411 # Udev cookie 0xd4dc8c7 (semid 9830400) created # Udev cookie 0xd4dc8c7 (semid 9830400) incremented to 1 # Udev cookie 0xd4dc8c7 (semid 9830400) incremented to 2 # Udev cookie 0xd4dc8c7 (semid 9830400) assigned to CREATE task(0) with # flags DISABLE_SUBSYSTEM_RULES DISABLE_DISK_RULES DISABLE_OTHER_RULES (0xe) # dm create temporary-cryptsetup-17411 CRYPT-TEMP-temporary-cryptsetup-17411 # OF [16384] (*1) # dm reload temporary-cryptsetup-17411 OFRW [16384] (*1) # dm resume temporary-cryptsetup-17411 OFRW [16384] (*1) # temporary-cryptsetup-17411: Stacking NODE_ADD (253,0) 0:6 0660 # [verify_udev] # temporary-cryptsetup-17411: Stacking NODE_READ_AHEAD 256 (flags=1) # Udev cookie 0xd4dc8c7 (semid 9830400) decremented to 1 # Udev cookie 0xd4dc8c7 (semid 9830400) waiting for zero # Udev cookie 0xd4dc8c7 (semid 9830400) destroyed # temporary-cryptsetup-17411: Processing NODE_ADD (253,0) 0:6 0660 # [verify_udev] # temporary-cryptsetup-17411: Processing NODE_READ_AHEAD 256 (flags=1) # temporary-cryptsetup-17411 (253:0): read ahead is 256 # temporary-cryptsetup-17411 (253:0): Setting read ahead to 256 Failed to access temporary keystore device. # Udev cookie 0xd4d53b6 (semid 9863168) created # Udev cookie 0xd4d53b6 (semid 9863168) incremented to 1 # Udev cookie 0xd4d53b6 (semid 9863168) incremented to 2 # Udev cookie 0xd4d53b6 (semid 9863168) assigned to REMOVE task(2) with # flags (0x0) # dm remove temporary-cryptsetup-17411 OFT [16384] (*1) # temporary-cryptsetup-17411: Stacking NODE_DEL [verify_udev] # Udev cookie 0xd4d53b6 (semid 9863168) decremented to 1 # Udev cookie 0xd4d53b6 (semid 9863168) waiting for zero # Udev cookie 0xd4d53b6 (semid 9863168) destroyed # temporary-cryptsetup-17411: Processing NODE_DEL [verify_udev] # Releasing crypt device /dev/md10 context. # Releasing device-mapper backend. # Unlocking memory. Command failed with code 5: Failed to access temporary keystore device.
Signature · 4 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., Email: [email protected] GnuPG: ID: CB5D9718 FP: 12D6 C03B 1B30 33BB 13CF B774 E35C 5FA1 CB5D 9718 ---- A good decision is based on knowledge and not on numbers. - Plato
Quoted 7 lines · Arno Wagner
Quoted 1 line
Can you paste the command with added --debug?
See below, both for 1.6.1 and 1.6.5, which unloaks without error (well, without error that gets propagated to the user), but never creates the entry in
/dev/mapper/. Likely a bug in 1.6.5, as it probably should tell the user that things went wrong.
The 1.6.5 uses different code here (it reads device directly when decrypting keyslot) and it need more user friendly error messages here, my bad...
Anyway, seems like in both cases read of device really returns I/O error while reading keyslot area.
Could you send me strace of the command?
(No need to enter correct password at all.)
BTW if not already there, it is another nice item to FAQ
- warn people that strace and similar debugging output can easily leak keys or passwords. And yes, people sometimes post these to lists :)
Quoted 11 lines
Quoted 2 lines
Can you try to boot Debian provided kernel - does it work?
Not easily. But it does work with 3.10.51, so the 3.2.x that Debian stable is stuck at should probably work too.
Come to think of it, I have
/usr/src/linuxpointing to a 3.4.67 source tree, as gcc kernel includes in Debian stable are really messed up with 3.5.x and later and I failed to fix it manually. (Sometimes I really wonder what the Kernel devs are thinking or whether they are thinking at all...) Could that be the problem?
Don't think so... kernel should use own includes while compiling and what's failing here is just plain read (I think).
Quoted 3 lines
I usually run testing, except that I really do not want systemd, so until I am sure I can do that update without getting that atrocity, no update to jessy for me.
There is a lot of discussion about this on debian devel, IIRC systemd-shim is possible the way to avoid systemd as init. (dunno if this will be supported).
Quoted 4 lines
Anyways, if we do not figure this one out, I will just stay with 3.10.x, it is a longterm-kernel after all. I just tried 3.14.15 because I have some network issues and wanted to see whether they may be gone with a newer kernel.
Well, it would be interesting to find what's wrong here.
You are using MD device - what kind of raid is that? (lsblk -t can say more info about storage stack topology as well).
Milan
Quoted 9 lines · Milan Broz
Quoted 7 lines · Arno Wagner
Quoted 1 line
Can you paste the command with added --debug?
See below, both for 1.6.1 and 1.6.5, which unloaks without error (well, without error that gets propagated to the user), but never creates the entry in
/dev/mapper/. Likely a bug in 1.6.5, as it probably should tell the user that things went wrong.
Quoted 8 lines
The 1.6.5 uses different code here (it reads device directly when decrypting keyslot) and it need more user friendly error messages here, my bad...
Anyway, seems like in both cases read of device really returns I/O error while reading keyslot area.
Could you send me strace of the command?
(No need to enter correct password at all.)
Looks like it. Strace output from a test container comes in separate email.
Quoted 4 lines
BTW if not already there, it is another nice item to FAQ
- warn people that strace and similar debugging output can easily leak keys or passwords. And yes, people sometimes post these to lists :)
Good idea. Added as Item 4.5 and to the warnings at the start.
Quoted 15 lines
Quoted 11 lines
Quoted 2 lines
Can you try to boot Debian provided kernel - does it work?
Not easily. But it does work with 3.10.51, so the 3.2.x that Debian stable is stuck at should probably work too.
Come to think of it, I have
/usr/src/linuxpointing to a 3.4.67 source tree, as gcc kernel includes in Debian stable are really messed up with 3.5.x and later and I failed to fix it manually. (Sometimes I really wonder what the Kernel devs are thinking or whether they are thinking at all...) Could that be the problem?Don't think so... kernel should use own includes while compiling and what's failing here is just plain read (I think).
Quoted 7 lines
Quoted 3 lines
I usually run testing, except that I really do not want systemd, so until I am sure I can do that update without getting that atrocity, no update to jessy for me.
There is a lot of discussion about this on debian devel, IIRC systemd-shim is possible the way to avoid systemd as init. (dunno if this will be supported).
We will see. I have a suspicion that the sudden long-term support for pre-systemd Debian is not an accident.
Quoted 6 lines
Quoted 4 lines
Anyways, if we do not figure this one out, I will just stay with 3.10.x, it is a longterm-kernel after all. I just tried 3.14.15 because I have some network issues and wanted to see whether they may be gone with a newer kernel.
Well, it would be interesting to find what's wrong here.
Ok, so lets keep poking at it.
Quoted 2 lines
You are using MD device - what kind of raid is that? (lsblk -t can say more info about storage stack topology as well).
It is a 3-way md RAID1 (on 2.5" laptop drives, about one firmware crash per year...).
"lsblk -t" does not say a lot:
NAME ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE
md10 0 4096 0 4096 512 1 128
Arno
Signature · 4 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., Email: [email protected] GnuPG: ID: CB5D9718 FP: 12D6 C03B 1B30 33BB 13CF B774 E35C 5FA1 CB5D 9718 ---- A good decision is based on knowledge and not on numbers. - Plato
Quoted 2 lines · Arno Wagner
We will see. I have a suspicion that the sudden long-term support for pre-systemd Debian is not an accident.
Actually that's completely unrelated. Also, Wheezy LTS will only happen if Squeeze LTS gains enough traction and volunteers.
Regards,
Signature · 1 line
Yves-Alexis
Quoted 10 lines · Yves-Alexis Perez
Quoted 2 lines · Arno Wagner
We will see. I have a suspicion that the sudden long-term support for pre-systemd Debian is not an accident.
Actually that's completely unrelated. Also, Wheezy LTS will only happen if Squeeze LTS gains enough traction and volunteers.
Regards,
Signature · 1 line
Yves-Alexis
I will form a definite opinion about that in a few years. At this time, I think the timing is highly suggestive.
Arno
Signature · 4 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., Email: [email protected] GnuPG: ID: CB5D9718 FP: 12D6 C03B 1B30 33BB 13CF B774 E35C 5FA1 CB5D 9718 ---- A good decision is based on knowledge and not on numbers. - Plato
Quoted 2 lines · Arno Wagner
I will form a definite opinion about that in a few years. At this time, I think the timing is highly suggestive.
I'm a member of the security team, which took the decision to try the LTS experiment, so I can assure you that's completely unrelated.
Now, maybe people will join the experiment because of that, but that's not the intention behind it.
Regards,
Signature · 1 line
Yves-Alexis
Hi,
just resolved this. I did indeed screw up the kernel config update. After a second try, cryptsetup works fine with stock kernel.org 3.14.15.
Gr"usse,
Arno
Quoted 86 lines · Arno Wagner
Quoted 9 lines · Milan Broz
Quoted 7 lines · Arno Wagner
Quoted 1 line
Can you paste the command with added --debug?
See below, both for 1.6.1 and 1.6.5, which unloaks without error (well, without error that gets propagated to the user), but never creates the entry in
/dev/mapper/. Likely a bug in 1.6.5, as it probably should tell the user that things went wrong.Quoted 8 lines
The 1.6.5 uses different code here (it reads device directly when decrypting keyslot) and it need more user friendly error messages here, my bad...
Anyway, seems like in both cases read of device really returns I/O error while reading keyslot area.
Could you send me strace of the command?
(No need to enter correct password at all.)Looks like it. Strace output from a test container comes in separate email.
Quoted 4 lines
BTW if not already there, it is another nice item to FAQ
- warn people that strace and similar debugging output can easily leak keys or passwords. And yes, people sometimes post these to lists :)Good idea. Added as Item 4.5 and to the warnings at the start.
Quoted 15 lines
Quoted 11 lines
Quoted 2 lines
Can you try to boot Debian provided kernel - does it work?
Not easily. But it does work with 3.10.51, so the 3.2.x that Debian stable is stuck at should probably work too.
Come to think of it, I have
/usr/src/linuxpointing to a 3.4.67 source tree, as gcc kernel includes in Debian stable are really messed up with 3.5.x and later and I failed to fix it manually. (Sometimes I really wonder what the Kernel devs are thinking or whether they are thinking at all...) Could that be the problem?Don't think so... kernel should use own includes while compiling and what's failing here is just plain read (I think).
Quoted 7 lines
Quoted 3 lines
I usually run testing, except that I really do not want systemd, so until I am sure I can do that update without getting that atrocity, no update to jessy for me.
There is a lot of discussion about this on debian devel, IIRC systemd-shim is possible the way to avoid systemd as init. (dunno if this will be supported).
We will see. I have a suspicion that the sudden long-term support for pre-systemd Debian is not an accident.
Quoted 6 lines
Quoted 4 lines
Anyways, if we do not figure this one out, I will just stay with 3.10.x, it is a longterm-kernel after all. I just tried 3.14.15 because I have some network issues and wanted to see whether they may be gone with a newer kernel.
Well, it would be interesting to find what's wrong here.
Ok, so lets keep poking at it.
Quoted 2 lines
You are using MD device - what kind of raid is that? (lsblk -t can say more info about storage stack topology as well).
It is a 3-way md RAID1 (on 2.5" laptop drives, about one firmware crash per year...).
"lsblk -t" does not say a lot:
NAME ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE
md10 0 4096 0 4096 512 1 128Arno
Signature · 8 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., Email: [email protected] GnuPG: ID: CB5D9718 FP: 12D6 C03B 1B30 33BB 13CF B774 E35C 5FA1 CB5D 9718 ---- A good decision is based on knowledge and not on numbers. - Plato _______________________________________________ dm-crypt mailing list [email protected] http://www.saout.de/mailman/listinfo/dm-crypt
Signature · 4 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., Email: [email protected] GnuPG: ID: CB5D9718 FP: 12D6 C03B 1B30 33BB 13CF B774 E35C 5FA1 CB5D 9718 ---- A good decision is based on knowledge and not on numbers. - Plato