overlays on dd images of 4T drives

Hi, all --

Crossing my fingers that this gets through ... My last few messages have not come back to me on the list.

After a first attempt to assemble the two working and one wonky drives of my 4-disk array and then watching reads hang forever, I have now used dd to pull an image of each RAID partition

davidtg@gezebel:~> sudo fdisk -l /dev/sda
Disk /dev/sda: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors Disk model: ST4000DM000-1F21
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt
Disk identifier: 482CA0F0-663B-409D-A4DF-4A4B248E6D23

Device Start End Sectors Size Type
/dev/sda1 2048 7813774990 7813772943 3.7T Linux RAID
/dev/sda2 7813775360 7814037134 261775 127.8M Linux filesystem

diskfarm:/mnt/10Traid50md/tmp # ls -goh 4T*
-rw-r--r-- 1 4.0T Nov 12 02:05 '4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror'
-rw-r--r-- 1 4.0T Nov 12 02:05 '4Tsdb1.5YD9.dd-bs=256M-conv=sparse,noerror'
-rw-r--r-- 1 4.0T Nov 12 02:05 '4Tsdc1.5ZY3.dd-bs=256M-conv=sparse,noerror'
diskfarm:/mnt/10Traid50md/tmp # file 4Tsda1.EYNA.dd-bs\=256M-conv\=sparse\,noerror
4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror: Linux Software RAID version 1.2 (1) UUID=ca7008ef:90693dae:6c231ad7: 8b3f92d name=diskfarm:0 level=5 disks=4

so that I can use overlays to assemble the device and replay the XFS journal to get up and running again to finally be able to pull off this data.

After going back and forth across the deprecated wiki pages, I'm working through Irreversible Failure Recovery

https://raid.wiki.kernel.org/index.php/Irreversible_mdadm_failure_recovery

and have created

diskfarm:/mnt/10Traid50md/tmp # ls -goh overlay-sd* -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sda1 -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sdb1 -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sdc1 diskfarm:/mnt/10Traid50md/tmp # losetup -a /dev/loop1: [66326]:1059 (/mnt/10Traid50md/tmp/overlay-sdb1) /dev/loop2: [66326]:1060 (/mnt/10Traid50md/tmp/overlay-sdc1) /dev/loop0: [66326]:1058 (/mnt/10Traid50md/tmp/overlay-sda1)

my overlay files and loopback devices. But ...

It seems that blockdev does not like

diskfarm:/mnt/10Traid50md/tmp # blockdev --getsize ./overlay-sda1
blockdev: ioctl error on BLKGETSIZE: Inappropriate ioctl for device

my files and so it won't give me the size that I need for dmsetup in the next step.

How can I use these partition images as source for faking up my RAID array to recover the contents? I seem to have gotten myself stuck :-/

TIA & HAND

:-D

David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt

It must be run on block devices, not files, so try:

  blockdev --getsize /dev/loop0

With respect,
Roman

Hi, all --

...and then David T-G said...
%
% Crossing my fingers that this gets through ... My last few messages have % not come back to me on the list.

It looks like anything sent from my server is silently dropped. I've asked majordomo for help; wish me luck. Meanwhile, I have to send these through an alternate server :-/

This is even more fun because I'm only home every three days or so to be able to poke at this :-( Dang!

%
...
% to pull an image of each RAID partition
%

%   davidtg@gezebel:~> sudo fdisk -l /dev/sda
%   Disk /dev/sda: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
%   Disk model: ST4000DM000-1F21
...
%   diskfarm:/mnt/10Traid50md/tmp # ls -goh 4T*
%   -rw-r--r-- 1 4.0T Nov 12 02:05 '4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror'
%   -rw-r--r-- 1 4.0T Nov 12 02:05 '4Tsdb1.5YD9.dd-bs=256M-conv=sparse,noerror'
%   -rw-r--r-- 1 4.0T Nov 12 02:05 '4Tsdc1.5ZY3.dd-bs=256M-conv=sparse,noerror'
%   diskfarm:/mnt/10Traid50md/tmp # file 4Tsda1.EYNA.dd-bs\=256M-conv\=sparse\,noerror
%   4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror: Linux Software RAID version 1.2 (1) UUID=ca7008ef:90693dae:6c231ad7: 8b3f92d name=diskfarm:0 level=5 disks=4

%
% so that I can use overlays to assemble the device and replay the XFS
...
%
% and have created
%

%   diskfarm:/mnt/10Traid50md/tmp # ls -goh overlay-sd*
%   -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sda1
%   -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sdb1
%   -rw-r--r-- 1 4.0T Nov 12 02:05 overlay-sdc1
%   diskfarm:/mnt/10Traid50md/tmp # losetup -a
%   /dev/loop1: [66326]:1059 (/mnt/10Traid50md/tmp/overlay-sdb1)
%   /dev/loop2: [66326]:1060 (/mnt/10Traid50md/tmp/overlay-sdc1)
%   /dev/loop0: [66326]:1058 (/mnt/10Traid50md/tmp/overlay-sda1)

%
% my overlay files and loopback devices. But ...
%
% It seems that blockdev does not like
[snip]

Thanks to Roman for pointing out that I should look at the loopback devices rather than the overlay files. That gets me on my way again. But ... now I have problems with the device mapper:

diskfarm:/mnt/10Traid50md/tmp # echo $size
8388608000
diskfarm:/mnt/10Traid50md/tmp # echo "0 $size snapshot overlay-sda1 /dev/loop0 P 8" | dmsetup create ov-a
device-mapper: reload ioctl on ov-a failed: No such device
Command failed.

ARRRGH!

OK, so I'll go to my workstation and poke at the actual devices, with great trepidation, from there.

davidtg@gezebel:/mnt/data/tmp/4Traid> ls -goh
total 0
-rw-r--r-- 1 4.0T Nov 15 02:00 overlay-sda1
-rw-r--r-- 1 4.0T Nov 15 02:00 overlay-sdb1
-rw-r--r-- 1 4.0T Nov 15 02:00 overlay-sdc1

  davidtg@gezebel:/mnt/data/tmp/4Traid> parallel 'size=$(sudo blockdev --getsize {}); loop=$(sudo losetup -f --show -- overlay-{/}) ; echo $loop' ::: $DEVICES
  /dev/loop0
  /dev/loop2
  /dev/loop1

davidtg@gezebel:/mnt/data/tmp/4Traid> sudo losetup -a /dev/loop1: [66307]:624372298 (/mnt/data/tmp/4Traid/overlay-sdc1) /dev/loop2: [66307]:624372297 (/mnt/data/tmp/4Traid/overlay-sdb1) /dev/loop0: [66307]:624372296 (/mnt/data/tmp/4Traid/overlay-sda1) ### why are these out of order?!? *sigh*

Now, however, when I try to use the dev mapper to create the snapshot device I get

davidtg@gezebel:/mnt/data/tmp/4Traid> echo 0 sudo blockdev --getsize /dev/sda1 snapshot /dev/sda1 /dev/loop0 P 8 | sudo dmsetup create sda1
device-mapper: reload ioctl on sda1 failed: Device or resource busy
Command failed.

an error. It's simply plugged in! What could have it busy? Or is this a device mapper problem on my OpenSuSE LEAP 15.2 system?

I think we need the device mapper to create the snapshot dev to be able to write changes not to the original disks, so we're stuck there.

Soooo ... Let's go back to diskfarm and, throwing caution to the winds, attempt to assemble the image files copied over:

diskfarm:/mnt/10Traid50md/tmp # mdadm --assemble --force /dev/md4 4Tsda1* 4Tsda2* 4Tsda3*
mdadm: 4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror is not a block device.
mdadm: 4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror has no superblock - assembly aborted

OK, yeah, we could predict that. But it gets even more fun:

diskfarm:/mnt/10Traid50md/tmp # mdadm --assemble --force /dev/md4 /dev/loop{0,1,2}
mdadm: no recogniseable superblock on /dev/loop0
mdadm: /dev/loop0 has no superblock - assembly aborted

What?!? Where is my superblock? This is a bit-for-bit copy of the partition itself.

Time to fall back for more help from the RAID gods *sigh* Any further recommendations?

TIA again

:-D

David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt

Because you ran the command to create them with parallel. Don't do that.

Run mdadm -E /dev/loop{0,1,2}

Phillip, et al --

...and then Phillip Susi said...
%
% David T-G <[email protected]> writes:
%
% > davidtg@gezebel:/mnt/data/tmp/4Traid> parallel 'size=$(sudo blockdev
% > --getsize {}); loop=$(sudo losetup -f --show -- overlay-{/}) ; echo % > $loop' ::: $DEVICES
...
% > davidtg@gezebel:/mnt/data/tmp/4Traid> sudo losetup -a
...
% > ### why are these out of order?!? *sigh*
%
% Because you ran the command to create them with parallel. Don't do that.

Ah. Well, yeah; I'd like to not, not least because I don't have parallel on all machines and I don't know its grammar enough to know what it's really doing. But I also know that sometimes I should just shut up and follow the documentation :-)

And I would figure that parallel would process them in order since they should all take the same amount of time to run! But that's MUTEX play for ya.

%
% > OK, yeah, we could predict that. But it gets even more fun:
...
% > What?!? Where is my superblock? This is a bit-for-bit copy of the % > partition itself.
% >
% > Time to fall back for more help from the RAID gods *sigh* Any further % > recommendations?
%
% Run mdadm -E /dev/loop{0,1,2}

Well, that appears to agree:

diskfarm:/mnt/10Traid50md/tmp # mdadm -E /dev/loop{0,1,2}
mdadm: No md superblock detected on /dev/loop0.
mdadm: No md superblock detected on /dev/loop1.
mdadm: No md superblock detected on /dev/loop2. diskfarm:/mnt/10Traid50md/tmp # losetup -a /dev/loop1: [66326]:1059 (/mnt/10Traid50md/tmp/overlay-sdb1) /dev/loop2: [66326]:1060 (/mnt/10Traid50md/tmp/overlay-sdc1) /dev/loop0: [66326]:1058 (/mnt/10Traid50md/tmp/overlay-sda1)

But I'm confused, all right:

diskfarm:/mnt/10Traid50md/tmp # mdadm -E 4Tsda1.EYNA.dd-bs\=256M-conv\=sparse\,noerror 4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror:

	    Magic : a92b4efc
	  Version : 1.2
      Feature Map : 0x0
       Array UUID : ca7008ef:90693dae:6c231ad7:08b3f92d
	     Name : diskfarm:0  (local to host diskfarm)
    Creation Time : Mon Feb  6 05:56:35 2017
       Raid Level : raid5
     Raid Devices : 4

Avail Dev Size : 7813510799 sectors (3.64 TiB 4.00 TB)
Array Size : 11720265216 KiB (10.92 TiB 12.00 TB)
Used Dev Size : 7813510144 sectors (3.64 TiB 4.00 TB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=574835712 sectors
State : clean
Device UUID : f05a143b:50c9b024:36714b9a:44b6a159

Update Time : Sun Nov 7 01:00:00 2021
Checksum : 4adb8b59 - correct
Events : 128116

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 3
Array State : A.AA ('A' == active, '.' == missing, 'R' == replacing)

OH!!! Wait. The loopback device points to the overlay file, which is sparse. No wonder it has no superblock. When I was here on this host after having lots of device mapper issues, I tried to assemble the files and got nowhere.

Can I create loopback devices pointing to the image files, which will give me raw handles, and then assemble that?

Or am I stuck with struggling with the mapper on my workstation with the actual disks in it and wondering why sda1

davidtg@gezebel:/mnt/data/tmp/4Traid> sudo losetup -a /dev/loop1: [66307]:624372298 (/mnt/data/tmp/4Traid/overlay-sdc1) /dev/loop2: [66307]:624372297 (/mnt/data/tmp/4Traid/overlay-sdb1) /dev/loop0: [66307]:624372296 (/mnt/data/tmp/4Traid/overlay-sda1) davidtg@gezebel:/mnt/data/tmp/4Traid> echo 0 sudo blockdev --getsize /dev/sda1 snapshot /dev/sda1 /dev/loop0 P 8 | sudo dmsetup create sda1
device-mapper: reload ioctl on sda1 failed: Device or resource busy Command failed.

is busy?

Thanks again & HANN

:-D

David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt

Hi again, all --

...and then David T-G said...
%
...
%
% Can I create loopback devices pointing to the image files, which will % give me raw handles, and then assemble that?
[snip]

Apparently that's a "no":

diskfarm:/mnt/10Traid50md/tmp # losetup -r /dev/loop10 pwd/4Tsda1* diskfarm:/mnt/10Traid50md/tmp # losetup -r /dev/loop11 pwd/4Tsdb1* diskfarm:/mnt/10Traid50md/tmp # losetup -r /dev/loop12 pwd/4Tsdc1*

diskfarm:/mnt/10Traid50md/tmp # losetup -a | grep 4T
/dev/loop11: [66326]:1054 (/mnt/10Traid50md/tmp/4Tsdb1.5YD9.dd-bs=256M-conv=sparse,noerror) /dev/loop12: [66326]:1055 (/mnt/10Traid50md/tmp/4Tsdc1.5ZY3.dd-bs=256M-conv=sparse,noerror) /dev/loop10: [66326]:1028 (/mnt/10Traid50md/tmp/4Tsda1.EYNA.dd-bs=256M-conv=sparse,noerror)

  # hey ... i did those manually and in order! *sigh*

  diskfarm:/mnt/10Traid50md/tmp # mdadm --assemble /dev/md4 /dev/loop1{0,1,2}
  mdadm: /dev/loop10 is busy - skipping
  mdadm: /dev/loop11 is busy - skipping
  mdadm: /dev/loop12 is busy - skipping

Scratch that idea ...

:-D

David T-G
See http://justpickone.org/davidtg/email/
See http://justpickone.org/davidtg/tofu.txt