dm-crypt LUKS and USB power management

Hi,

I have a encrypted USB disk at /dev/sda that worked nicely so far. However, recently some new (?) power management seems to break this. Disk spin down and power up still work, but at some point the USB disk disconnects. Searching what happens here, I found that
https://www.kernel.org/doc/Documentation/usb/power-management.txt
claims it's not uncommon that USB devices don't support power management properly and thus do a disconnect and reconnect there. Seem this is happening as dmesg reports:

  [    5.459493] usb 1-1.2: new high-speed USB device number 3 using dwc_otg
  [    5.459493] usb-storage 1-1.2:1.0: USB Mass Storage device detected
  [    5.561081] scsi0 : usb-storage 1-1.2:1.0
  [    5.561081] scsi 0:0:0:0: Direct-Access Disk Name 2BC1 PQ: 0 ANSI: 2 CCS
  [    5.562474] sd 0:0:0:0: [sda] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
  ....
  [ 1659.691793] usb 1-1.2: USB disconnect, device number 3
  [ 1659.699144] scsi 0:0:0:0: rejecting I/O to offline device
  [ 1659.699215] scsi 0:0:0:0: [sda] killing request
  [ 1659.703914] scsi 0:0:0:0: [sda] Unhandled error code
  [ 1659.708993] scsi 0:0:0:0: [sda]
  [ 1659.712500] Result: hostbyte=0x01 driverbyte=0x00
  [ 1659.717332] scsi 0:0:0:0: [sda] CDB:
  [ 1659.721162] cdb[0]=0x28: 28 00 00 04 31 7a 00 00 08 00
  [ 1659.726443] end_request: I/O error, dev sda, sector 274810
  [ 1663.759509] usb 1-1.2: new high-speed USB device number 4 using dwc_otg
  [ 1663.865049] usb-storage 1-1.2:1.0: USB Mass Storage device detected
  [ 1663.866221] scsi1 : usb-storage 1-1.2:1.0
  [ 1664.869823] scsi 1:0:0:0: Direct-Access Disk Name 2BC1 PQ: 0 ANSI: 2 CCS
  [ 1664.874690] sd 1:0:0:0: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
  [ 1664.881544] sd 1:0:0:0: [sdb] Write Protect is off
  [ 1664.885663] sd 1:0:0:0: [sdb] Mode Sense: 28 00 00 00
  [ 1664.886284] sd 1:0:0:0: [sdb] No Caching mode page found
  [ 1664.891160] sd 1:0:0:0: [sdb] Assuming drive cache: write through
  [ 1664.899793] sd 1:0:0:0: [sdb] No Caching mode page found
  [ 1664.902859] sd 1:0:0:0: [sdb] Assuming drive cache: write through
  [ 1664.985418]  sdb: sdb1
  [ 1664.988355] sd 1:0:0:0: [sdb] No Caching mode page found
  [ 1664.988410] sd 1:0:0:0: [sdb] Assuming drive cache: write through
  [ 1664.994563] sd 1:0:0:0: [sdb] Attached SCSI disk
  [ 1665.049694] Aborting journal on device dm-0-8.
  [ 1665.050009] Buffer I/O error on device dm-0, logical block 243826688
  [ 1665.055199] lost page write due to I/O error on dm-0
  [ 1665.060425] JBD2: Error -5 detected when updating journal superblock for dm-0-8.
  [ 1689.789480] Buffer I/O error on device dm-0, logical block 0
  [ 1689.793190] lost page write due to I/O error on dm-0
  [ 1689.798373] EXT4-fs error (device dm-0): __ext4_journal_start_sb:62: Detected aborted journal
  [ 1689.806990] EXT4-fs (dm-0): Remounting filesystem read-only
  [ 1689.812846] EXT4-fs (dm-0): previous I/O error to superblock detected
  [ 1689.819660] Buffer I/O error on device dm-0, logical block 0
  [ 1689.825145] lost page write due to I/O error on dm-0
  [ 1689.830320] EXT4-fs (dm-0): ext4_da_writepages: jbd2_start: 5120 pages, ino 13; err -30

I wonder if there are any ideas how dm-crypto or LUKS can handle this case. Or is the recommendation that the USB suspend should not happen at all for devices that are broken (ie. that disconnect and reconnect on resume)?
Even if all handles on /dev/sda are released internally, there is not really a guarantee that the devices comes back as /dev/sda are the disconnect/reconnect. However, the UUID should be the same, so that could be used to detect it's the same device and partition then and accesses get re-routed to it then transparently.

Thanks, Axel

I doubt if it could be done clean. Most targets in Device Mapper ask for careful unstacking.

Red Hat (leading Linux vendor) still seems to be recommending that.
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-si
ngle/Storage_Administration_Guide/#removing_devices

I would rather investigate the (flaky) USB device. First, does it happen only when Runtime PM is enabled ? If so, you should just blacklist it from Power Management. Many devices, under Linux, report (false) PM capabilities.

Given the large number of mailing lists I follow, I request you to CC
me in replies for quicker response

Couldn't LUKS/cryptsetup/dm-crypt set up a monitor for the disk peripheral at least? So it releases any connection to the device if it is diconnected. The device is gone anyway, so there is no gain in keeping any handles open. Internally, the higher layer file system driver should still get errors then. But the lower lever driver stack is no longer blocked. So a new device can become /dev/sda again and not dev/sdb because /dev/sda is still "somehow" active.

My current solution is using a hub with a dedicated power support to connect the USB/SATA adapter with the HDD. Then disk spindown can still be used, but there is no USB disconnect/reconnect. That solves the problem practically.
I did not find a way yet to disable power/idele management on the board USB ports. It's a Odroid C1 or C2 with Debian Jesse Kernel 3.14.29 or 3.14.65. The suggestion from
http://unix.stackexchange.com/questions/91027/how-to-disable-usb-autosuspend-on-kernel-3-7-10-or-above
does not work
and so far nobody else had a solution. Will keep searching...

Axel

That would be the wrong place to do it. Monitoring USB disconnect is the task of the USB driver, also because LUKS/LVM/etc. has no clue what hardware is used on the lower layers.

Regards,
Arno

Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: [email protected]
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
A good decision is based on knowledge and not on numbers. -- Plato

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier