Skip to contentSkip to conversation

ERROR: error removing the device '/dev/sdXN' - Inappropriate ioctl for device

DISCUSSION Martin 3 participants 10 messages Mar 30, 2015 – Apr 13, 2015 Btrfs file system

lore
Share & tools
lore

Martin

author
#0 lore raw

Help welcomed...

Trying to delete one device from a btrfs raid1 across 3 devices gives for my example:

btrfs device delete /dev/sdf5 /mnt/data2

ERROR: error removing the device '/dev/sdf5' - Inappropriate ioctl for device

That part of the filesystem is:

Label: 'btrfs_data2' uuid: 3aaee716-b98b-4c86-ba5a-53456994f152

        Total devices 3 FS bytes used 192.45GiB
        devid    1 size 206.47GiB used 206.02GiB path /dev/sdb5
        devid    2 size 206.47GiB used 206.47GiB path /dev/sdd5
        devid    3 size 206.47GiB used 206.47GiB path /dev/sdf5

All on: 3.14.10-gentoo

Before I do a mega-yank-up to the 3.18 kernel, is the above a known problem or am I missing something?

(Device delete needed due to that device throwing sector errors :-( )

Thanks,
Martin

#1 lore raw
Quoted 4 lines · Martin

btrfs device delete /dev/sdf5 /mnt/data2

ERROR: error removing the device '/dev/sdf5' - Inappropriate ioctl for device

Strange, that would mean that the ioctl number is not handled at all. I haven't found any change in device delete on the userspace side. This would mean that /mnt/data2 is not mounted. Did you try to do lazy unmount?

author
#2 lore raw
Quoted 10 lines · David Sterba
Quoted 4 lines · Martin

btrfs device delete /dev/sdf5 /mnt/data2

ERROR: error removing the device '/dev/sdf5' - Inappropriate ioctl for device

Strange, that would mean that the ioctl number is not handled at all. I haven't found any change in device delete on the userspace side. This would mean that /mnt/data2 is not mounted. Did you try to do lazy unmount?

Nope, that filesystem is still mounted and in use, but still a very good deduction...

Trying smartmonctl just now on that drive (/dev/sdf) gives errors suggesting that the drive is no longer responding at all :-(

There is also a steady repeat of:

kernel: sd 7:0:0:0: [sdf] CDB:
kernel: cdb[0]=0x2a: 2a 00 04 02 45 80 00 00 08 00
kernel: sd 7:0:0:0: [sdf] Unhandled error code
kernel: sd 7:0:0:0: [sdf]
kernel: Result: hostbyte=0x04 driverbyte=0x00

(Must watch that lot doesn't fill the log.)

Should I try a:

btrfs device delete missing /mnt/data2

or do I need to unmount or remount first?

... And the system is live and not to be rebooted if at all possible until the weekend...

Thanks,
Martin

#3 lore raw
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

btrfs device delete /dev/sdf5 /mnt/data2

ERROR: error removing the device '/dev/sdf5' - Inappropriate ioctl for device

very strange. 'btrfs fi show -m' shows btrfs fs(s) that are mounted.

- Anand

author
#4 lore raw
Quoted 7 lines · Anand Jain
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

btrfs device delete /dev/sdf5 /mnt/data2

ERROR: error removing the device '/dev/sdf5' - Inappropriate ioctl for device

very strange. 'btrfs fi show -m' shows btrfs fs(s) that are mounted.

Looks like my /dev/sdf isn't responding with anything useful at all... A firmware crash?... (This is for a 256GB SSD.)

# btrfs fi show -m

[...]

Label: 'btrfs_data2' uuid: 3aaee716-b98b-4c86-ba5a-53456994f152

        Total devices 3 FS bytes used 159.31GiB
        devid    1 size 206.47GiB used 206.02GiB path /dev/sdb5
        devid    2 size 206.47GiB used 206.47GiB path /dev/sdd5
        devid    3 size 206.47GiB used 206.47GiB path /dev/sdf5

btrfs-progs v3.19.1

# smartctl -i /dev/sdf
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.14.10-gentoo_s03a_11]
(local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor: /7:0:0:0
Product:

Compliance:           SPC-5
User Capacity:        600,332,565,813,390,450 bytes [600 PB]
Logical block size:   774843950 bytes
scsiModePageOffset: response length too short, resp_len=47 offset=50

bd_len=46
scsiModePageOffset: response length too short, resp_len=47 offset=50 bd_len=46

Quoted 1 line
Quoted 1 line

Terminate command early due to bad response to IEC mode page

A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

And btrfs is still running ok so far... I'll be swapping that device at the weekend. (First chance I have for a Sunday shutdown. :-( )

Still... For administering btrfs, it is a little disturbing not to be able to remove/delete a device unless that device is both mounted and working...

Or is that where the "missing" option comes in?

Comments welcomed,

Thanks,
Martin

#5 lore raw

Looks like an option to use devid to delete a device would have mitigated the issue. Also the error reported is no where near the reality. Will fix them.

Thanks for reporting.

Anand

Quoted 65 lines · Martin
Quoted 7 lines · Anand Jain
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

btrfs device delete /dev/sdf5 /mnt/data2

ERROR: error removing the device '/dev/sdf5' - Inappropriate ioctl for device

very strange. 'btrfs fi show -m' shows btrfs fs(s) that are mounted.

Looks like my /dev/sdf isn't responding with anything useful at all... A firmware crash?... (This is for a 256GB SSD.)

# btrfs fi show -m

[...]

Label: 'btrfs_data2' uuid: 3aaee716-b98b-4c86-ba5a-53456994f152

         Total devices 3 FS bytes used 159.31GiB
         devid    1 size 206.47GiB used 206.02GiB path /dev/sdb5
         devid    2 size 206.47GiB used 206.47GiB path /dev/sdd5
         devid    3 size 206.47GiB used 206.47GiB path /dev/sdf5

btrfs-progs v3.19.1

# smartctl -i /dev/sdf
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.14.10-gentoo_s03a_11]
(local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor: /7:0:0:0
Product:

Compliance:           SPC-5
User Capacity:        600,332,565,813,390,450 bytes [600 PB]
Logical block size:   774843950 bytes
scsiModePageOffset: response length too short, resp_len=47 offset=50

bd_len=46
scsiModePageOffset: response length too short, resp_len=47 offset=50 bd_len=46

Quoted 1 line
Quoted 1 line

Terminate command early due to bad response to IEC mode page

A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

And btrfs is still running ok so far... I'll be swapping that device at the weekend. (First chance I have for a Sunday shutdown. :-( )

Still... For administering btrfs, it is a little disturbing not to be able to remove/delete a device unless that device is both mounted and working...

Or is that where the "missing" option comes in?

Comments welcomed,

Thanks,
Martin

author
#6 lore raw

Anand,

Thanks for picking up.

The devid option sounds good.

Thanks,
Martin

Quoted 84 lines · Anand Jain

Looks like an option to use devid to delete a device would have mitigated the issue. Also the error reported is no where near the reality. Will fix them.

Thanks for reporting.

Anand

Quoted 67 lines · Martin
Quoted 8 lines · Anand Jain
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

btrfs device delete /dev/sdf5 /mnt/data2

ERROR: error removing the device '/dev/sdf5' - Inappropriate ioctl for
device

very strange. 'btrfs fi show -m' shows btrfs fs(s) that are mounted.

Looks like my /dev/sdf isn't responding with anything useful at all... A firmware crash?... (This is for a 256GB SSD.)

# btrfs fi show -m

[...]

Label: 'btrfs_data2' uuid: 3aaee716-b98b-4c86-ba5a-53456994f152

         Total devices 3 FS bytes used 159.31GiB
         devid    1 size 206.47GiB used 206.02GiB path /dev/sdb5
         devid    2 size 206.47GiB used 206.47GiB path /dev/sdd5
         devid    3 size 206.47GiB used 206.47GiB path /dev/sdf5

btrfs-progs v3.19.1

# smartctl -i /dev/sdf
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.14.10-gentoo_s03a_11]
(local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke,
www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor: /7:0:0:0
Product:

Compliance:           SPC-5
User Capacity:        600,332,565,813,390,450 bytes [600 PB]
Logical block size:   774843950 bytes
scsiModePageOffset: response length too short, resp_len=47 offset=50

bd_len=46
scsiModePageOffset: response length too short, resp_len=47 offset=50 bd_len=46

Quoted 1 line
Quoted 1 line

Terminate command early due to bad response to IEC mode page

A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

And btrfs is still running ok so far... I'll be swapping that device at the weekend. (First chance I have for a Sunday shutdown. :-( )

Still... For administering btrfs, it is a little disturbing not to be able to remove/delete a device unless that device is both mounted and working...

Or is that where the "missing" option comes in?

Comments welcomed,

Thanks,
Martin

Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#7 lore raw

Martin,

presumably kernel failed to read SB from disk /dev/sdf5, bit strange it saw 'Inappropriate ioctl for device', logically that don't explain.

Curios to know what does 'btrfs fi show -d' say ? and output of blkid if you could. The situation here is not same as destroying the SB though.

Thanks, Anand

Quoted 104 lines · Martin

Anand,

Thanks for picking up.

The devid option sounds good.

Thanks,
Martin

Quoted 75 lines · Anand Jain

Looks like an option to use devid to delete a device would have mitigated the issue. Also the error reported is no where near the reality. Will fix them.

Thanks for reporting.

Anand

Quoted 63 lines · Martin
Quoted 8 lines · Anand Jain
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

btrfs device delete /dev/sdf5 /mnt/data2

ERROR: error removing the device '/dev/sdf5' - Inappropriate ioctl for
device

very strange. 'btrfs fi show -m' shows btrfs fs(s) that are mounted.

Looks like my /dev/sdf isn't responding with anything useful at all... A firmware crash?... (This is for a 256GB SSD.)

# btrfs fi show -m

[...]

Label: 'btrfs_data2' uuid: 3aaee716-b98b-4c86-ba5a-53456994f152

          Total devices 3 FS bytes used 159.31GiB
          devid    1 size 206.47GiB used 206.02GiB path /dev/sdb5
          devid    2 size 206.47GiB used 206.47GiB path /dev/sdd5
          devid    3 size 206.47GiB used 206.47GiB path /dev/sdf5

btrfs-progs v3.19.1

# smartctl -i /dev/sdf
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.14.10-gentoo_s03a_11]
(local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke,
www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor: /7:0:0:0
Product:

Compliance:           SPC-5
User Capacity:        600,332,565,813,390,450 bytes [600 PB]
Logical block size:   774843950 bytes
scsiModePageOffset: response length too short, resp_len=47 offset=50

bd_len=46
scsiModePageOffset: response length too short, resp_len=47 offset=50 bd_len=46

Quoted 1 line
Quoted 1 line

Terminate command early due to bad response to IEC mode page

A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

And btrfs is still running ok so far... I'll be swapping that device at the weekend. (First chance I have for a Sunday shutdown. :-( )

Still... For administering btrfs, it is a little disturbing not to be able to remove/delete a device unless that device is both mounted and working...

Or is that where the "missing" option comes in?

Quoted 9 lines
Quoted 4 lines

Comments welcomed,

Thanks,
Martin

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#8 lore raw
Quoted 1 line · Anand Jain

btrfs fi show -d

That gives:

# btrfs fi show -d
warning, device 3 is missing
warning devid 3 not found already
warning, device 3 is missing
warning devid 3 not found already
Label: 'btrfs_root'  uuid: 92452e9a-2775-45c4-922c-f01b2afd51c2
        Total devices 3 FS bytes used 30.94GiB
        devid    1 size 24.00GiB used 24.00GiB path /dev/sda4
        devid    2 size 24.00GiB used 24.00GiB path /dev/sdc4
        devid    3 size 24.00GiB used 24.00GiB path /dev/sde4

Label: 'btrfs_data' uuid: d1b96638-be89-4291-8a40-f2f2e1dc5223

        Total devices 3 FS bytes used 95.74GiB
        devid    1 size 87.24GiB used 86.48GiB path /dev/sda5
        devid    2 size 87.24GiB used 87.24GiB path /dev/sdc5
        devid    3 size 87.24GiB used 87.24GiB path /dev/sde5

Label: 'btrfs_root2' uuid: 62603ce8-c333-4ca7-92f7-f8bdd712ab37
Total devices 3 FS bytes used 151.60MiB
devid 1 size 24.00GiB used 24.00GiB path /dev/sdb4
devid 2 size 24.00GiB used 24.00GiB path /dev/sdd4
*** Some devices missing

Label: 'btrfs_data2' uuid: 3aaee716-b98b-4c86-ba5a-53456994f152
Total devices 3 FS bytes used 159.34GiB
devid 1 size 206.47GiB used 206.02GiB path /dev/sdb5
devid 2 size 206.47GiB used 206.47GiB path /dev/sdd5
*** Some devices missing

btrfs-progs v3.19.1

And without the "-d":

# btrfs fi show
Label: 'btrfs_root'  uuid: 92452e9a-2775-45c4-922c-f01b2afd51c2
        Total devices 3 FS bytes used 30.94GiB
        devid    1 size 24.00GiB used 24.00GiB path /dev/sda4
        devid    2 size 24.00GiB used 24.00GiB path /dev/sdc4
        devid    3 size 24.00GiB used 24.00GiB path /dev/sde4

Label: 'btrfs_data' uuid: d1b96638-be89-4291-8a40-f2f2e1dc5223

        Total devices 3 FS bytes used 95.74GiB
        devid    1 size 87.24GiB used 86.48GiB path /dev/sda5
        devid    2 size 87.24GiB used 87.24GiB path /dev/sdc5
        devid    3 size 87.24GiB used 87.24GiB path /dev/sde5

Label: 'btrfs_root2' uuid: 62603ce8-c333-4ca7-92f7-f8bdd712ab37

        Total devices 3 FS bytes used 151.60MiB
        devid    1 size 24.00GiB used 24.00GiB path /dev/sdb4
        devid    2 size 24.00GiB used 24.00GiB path /dev/sdd4
        devid    3 size 24.00GiB used 24.00GiB path /dev/sdf4

Label: 'btrfs_data2' uuid: 3aaee716-b98b-4c86-ba5a-53456994f152

        Total devices 3 FS bytes used 159.34GiB
        devid    1 size 206.47GiB used 206.02GiB path /dev/sdb5
        devid    2 size 206.47GiB used 206.47GiB path /dev/sdd5
        devid    3 size 206.47GiB used 206.47GiB path /dev/sdf5

btrfs-progs v3.19.1

Interestingly, all the log messages about /dev/sdf are now no longer being repeated.

(And nope, not had a chance to swap that disk yet!)

Hence, should I do a "btrfs device delete missing /mnt/data2"?

Cheers,
Martin

#9 lore raw

Thanks Martin for helping with the data.

Unfortunately no matter what I try I couldn't reproduce the "Inappropriate ioctl for device"

But I got INVALD which is wrong as well. So I sent a patch for that. EIO is appropriate in this case.

Next, Passing a devid to delete a device - yes it would help patch is ready in my WS BUT - just noticed that error handling code nicely put the FS into readonly when there is commit failure so that's something which should be fixed along with this patch which I am attempting. Since as in your case with 3 disks raid1, your reconstructed radi1 will still be considered as healthy after taking out a disk.

Further, to recover from your situation you could try replace, that will work. OR if you are not planning to add another disk then, try this: (sorry needs a down time)

umount
remove failed disk from the system
mount -o degrade
btrfs dev del missing <mnt>
mount -o remount
(I hope you would check / mange the space availably part)

(By the way, those missing messages are user land fabricated,
btrfs-progs: 206efb60cbe3049e0d44c6da3c1909aeee18f813

so don't depend on that, to know what kernel knows probably you need /proc/fs/btrfs/devlist or sysfs posted in the ML before,

btrfs fi show -m

was written so to know actual kernel visibility but it was again crippled by above commit id.

David, should back out 206efb60cbe3049e0d44c6da3c1909aeee18f813 as mentioned before it will help.

Quoted 11 lines · Martin
Quoted 1 line · Anand Jain

btrfs fi show -d

That gives:

# btrfs fi show -d
warning, device 3 is missing
warning devid 3 not found already
warning, device 3 is missing
warning devid 3 not found already

David,

As commented before - you shouldn't have integrated the patch

915902c5002485fb13d27c4b699a73fb66cc0f09

Thanks, Anand

Quoted 75 lines

Label: 'btrfs_root' uuid: 92452e9a-2775-45c4-922c-f01b2afd51c2

         Total devices 3 FS bytes used 30.94GiB
         devid    1 size 24.00GiB used 24.00GiB path /dev/sda4
         devid    2 size 24.00GiB used 24.00GiB path /dev/sdc4
         devid    3 size 24.00GiB used 24.00GiB path /dev/sde4

Label: 'btrfs_data' uuid: d1b96638-be89-4291-8a40-f2f2e1dc5223

         Total devices 3 FS bytes used 95.74GiB
         devid    1 size 87.24GiB used 86.48GiB path /dev/sda5
         devid    2 size 87.24GiB used 87.24GiB path /dev/sdc5
         devid    3 size 87.24GiB used 87.24GiB path /dev/sde5

Label: 'btrfs_root2' uuid: 62603ce8-c333-4ca7-92f7-f8bdd712ab37
Total devices 3 FS bytes used 151.60MiB
devid 1 size 24.00GiB used 24.00GiB path /dev/sdb4
devid 2 size 24.00GiB used 24.00GiB path /dev/sdd4
*** Some devices missing

Label: 'btrfs_data2' uuid: 3aaee716-b98b-4c86-ba5a-53456994f152
Total devices 3 FS bytes used 159.34GiB
devid 1 size 206.47GiB used 206.02GiB path /dev/sdb5
devid 2 size 206.47GiB used 206.47GiB path /dev/sdd5
*** Some devices missing

btrfs-progs v3.19.1

And without the "-d":

# btrfs fi show
Label: 'btrfs_root'  uuid: 92452e9a-2775-45c4-922c-f01b2afd51c2
         Total devices 3 FS bytes used 30.94GiB
         devid    1 size 24.00GiB used 24.00GiB path /dev/sda4
         devid    2 size 24.00GiB used 24.00GiB path /dev/sdc4
         devid    3 size 24.00GiB used 24.00GiB path /dev/sde4

Label: 'btrfs_data' uuid: d1b96638-be89-4291-8a40-f2f2e1dc5223

         Total devices 3 FS bytes used 95.74GiB
         devid    1 size 87.24GiB used 86.48GiB path /dev/sda5
         devid    2 size 87.24GiB used 87.24GiB path /dev/sdc5
         devid    3 size 87.24GiB used 87.24GiB path /dev/sde5

Label: 'btrfs_root2' uuid: 62603ce8-c333-4ca7-92f7-f8bdd712ab37

         Total devices 3 FS bytes used 151.60MiB
         devid    1 size 24.00GiB used 24.00GiB path /dev/sdb4
         devid    2 size 24.00GiB used 24.00GiB path /dev/sdd4
         devid    3 size 24.00GiB used 24.00GiB path /dev/sdf4

Label: 'btrfs_data2' uuid: 3aaee716-b98b-4c86-ba5a-53456994f152

         Total devices 3 FS bytes used 159.34GiB
         devid    1 size 206.47GiB used 206.02GiB path /dev/sdb5
         devid    2 size 206.47GiB used 206.47GiB path /dev/sdd5
         devid    3 size 206.47GiB used 206.47GiB path /dev/sdf5

btrfs-progs v3.19.1

Interestingly, all the log messages about /dev/sdf are now no longer being repeated.

(And nope, not had a chance to swap that disk yet!)

Hence, should I do a "btrfs device delete missing /mnt/data2"?

Cheers,
Martin

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html