Open raid1 with luks encryption after a raid re-create

Hi.

I have a raid1 (mdadm-based) setup with two disks. I had them encrypted with luks. Everything was ok for 2 years. My PC had a problem and I had to mount this on a new PC.
When I tried to start the raid on the new PC it only started 1 of the disks because the other had been replaced on a different PC and had a different hostname (my original PC had a script to assemble the raid even with the two disks having a different hostname).

So I tried to fix this different hostname problem by re-creating the raid in the new PC using

mdadm -C /dev/md127 -l1 -n2 --assume-clean --metadata=1.2 /dev/sdb /dev/sdc1 --uuid=1d925c8d:8c8bb953:4e9070f7:43344cf9

mdadm: /dev/sdb appears to be part of a raid array:

    level=raid1 devices=2 ctime=Sun Feb 12 19:40:32 2012

mdadm: partition table exists on /dev/sdb but will be lost or

meaningless after creating array
mdadm: /dev/sdc1 appears to contain an ext2fs file system

    size=976760832K  mtime=Mon Aug 25 19:50:27 2014
mdadm: /dev/sdc1 appears to be part of a raid array:
    level=raid1 devices=2 ctime=Tue Aug 26 09:10:39 2014

Continue creating array? y
mdadm: array /dev/md127 started.

All appeared to be OK:

cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md127 : active (auto-read-only) raid1 sdc1[1] sdb[0]

      976629568 blocks super 1.2 [2/2] [UU]

unused devices: <none>

but now luks does not open the raid:

sudo cryptsetup luksOpen /dev/md127 raid1
Device /dev/md127 is not a valid LUKS device.

Any ideas on how to re-open the raid with luks?

Note: I thought there would be no problem with the create command because of this in the mdadm man page:
"Create Create a new array with per-device metadata (superblocks). Appropriate metadata is written to each device, and then the array comprising those devices is activated. A 'resync' process is started to make sure that the array is consistent (e.g. both sides of a mirror contain the same data) but *the content of the device is left otherwise untouched*. "

Thanks for any help you can provide.

Quoted 6 lines

It can sync in the wrong direction. And second, unfortunately, superblock format 1.2 is a dirty hack designed by incompetents. It places the RAID header 4k from the start of the device. For LUKS, this kills the first keyslow of misaligned. Unfortunately, this is also the default. No, that has not happened here or you would get the header.

Since the original raid was already 1.2 format, the location would already be the 4k from the start of the device. So where was LUKS info placed in terms of distance from the start of the device?

Quoted 5 lines

Quoted 1 line

Thanks for any help you can provide.

Ok, first stop writing to the disks. Second, make a full, binary backup of each disk. And third, try whether either disk works individually as degraded array.

1-Done.
2-Done. Dumped the first 2MB of each disk.

3-They appear as raid disks but again I cannot open the encryption.

Quoted 3 lines

If neither gives you a LUKS header, you can still search on the raw disks by looking for the LUKS signature. If that also fails, you are out of luck and all your data is gone.

The LUKS signature is simply 'LUKS'?

I grepped like this:

grep LUKS header_sdb_backup.dmp

is it the correct way to do it?
Did not find any match...

CC'ing to the list, as it serves as a sort-of archive of how to solve problems as well and there are several clueful and helpful people in it that may spot more things than I do.

On Sun, Nov 22, 2015 at 13:15:11 CET, Luis Alexandre wrote:
[...]

Quoted 9 lines

Quoted 6 lines

It can sync in the wrong direction. And second, unfortunately, superblock format 1.2 is a dirty hack designed by incompetents. It places the RAID header 4k from the start of the device. For LUKS, this kills the first keyslow of misaligned. Unfortunately, this is also the default. No, that has not happened here or you would get the header.

Since the original raid was already 1.2 format, the location would already be the 4k from the start of the device. So where was LUKS info placed in terms of distance from the start of the device?

Right at the start. That is why using superblock 0.90 or 1.0 with LUKS is a good diea as it becomes exceptionally unlikely that the MD-header damages the LUKS header.

Quoted 6 lines

Quoted 4 lines

Quoted 1 line

Thanks for any help you can provide.

Ok, first stop writing to the disks. Second, make a full, binary backup of each disk. And third, try whether either disk works individually as degraded array.

1-Done.
2-Done. Dumped the first 2MB of each disk.

Make that 1GB at least to be sure to capture the LUKS header in it if it is still anywhere.

Quoted 6 lines

3-They appear as raid disks but again I cannot open the encryption.

Quoted 3 lines

If neither gives you a LUKS header, you can still search on the raw disks by looking for the LUKS signature. If that also fails, you are out of luck and all your data is gone.

The LUKS signature is simply 'LUKS'?

Not quite. FAQ Item 6.12
(https://gitlab.com/cryptsetup/cryptsetup/wikis/FrequentlyAskedQuestions)
gives you a brief version, the LUKS on-disk spec gives everything.

Quoted 6 lines

I grepped like this:

grep LUKS header_sdb_backup.dmp

is it the correct way to do it?
Did not find any match...

Ok, lets repeat that with the full disks and including the full signature

hd /dev/sdx | grep "0 4c 55 4b 53 ba be 00 01"

with x one of your RAID disks. Do this for both. May take a while. This gives you the alignment as well. The "hd" start of a good luks header and container (header starts at offset 0) looks like this:

00000000 4c 55 4b 53 ba be 00 01 61 65 73 00 00 00 00 00 |LUKS....aes.....|

Only the first 6 bytes are fixed. Bytes 6 and 7 are the version of which there currently is onlyy "0001". This will always be aligned to a 512 byte boundary. Doing it this way has the advantage that you get the offset as well.

Regards,
Arno

Quoted 9 lines · Arno Wagner

Ok, lets repeat that with the full disks and including the full signature

hd /dev/sdx | grep "0 4c 55 4b 53 ba be 00 01"

with x one of your RAID disks. Do this for both. May take a while. This gives you the alignment as well. The "hd" start of a good luks header and container (header starts at offset 0) looks like this:

Quoted 5 lines

00000000 4c 55 4b 53 ba be 00 01 61 65 73 00 00 00 00 00 |LUKS....aes.....| Only the first 6 bytes are fixed. Bytes 6 and 7 are the version of which there currently is onlyy "0001". This will always be aligned to a 512 byte boundary. Doing it this way has the advantage that you get the offset as well.

found it in one of the disks:
08100000 4c 55 4b 53 ba be 00 01 61 65 73 00 00 00 00 00 |LUKS....aes.....|

Can you tell me how should I proceed now?

(the other is still being searched: the first one took a few seconds, this one is now over 1 hour search)

Many thanks,
Luis

Quoted 3 lines

Regards,
Arno

_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt

Quoted 51 lines

CC'ing to the list, as it serves as a sort-of archive of how to solve problems as well and there are several clueful and helpful people in it that may spot more things than I do.

On Sun, Nov 22, 2015 at 13:15:11 CET, Luis Alexandre wrote:
[...]

Quoted 9 lines

Quoted 6 lines

It can sync in the wrong direction. And second, unfortunately, superblock format 1.2 is a dirty hack designed by incompetents. It places the RAID header 4k from the start of the device. For LUKS, this kills the first keyslow of misaligned. Unfortunately, this is also the default. No, that has not happened here or you would get the header.

Since the original raid was already 1.2 format, the location would already be the 4k from the start of the device. So where was LUKS info placed in terms of distance from the start of the device?

Right at the start. That is why using superblock 0.90 or 1.0 with LUKS is a good diea as it becomes exceptionally unlikely that the MD-header damages the LUKS header.

Quoted 6 lines

Quoted 4 lines

Quoted 1 line

Thanks for any help you can provide.

Ok, first stop writing to the disks. Second, make a full, binary backup of each disk. And third, try whether either disk works individually as degraded array.

1-Done.
2-Done. Dumped the first 2MB of each disk.

Make that 1GB at least to be sure to capture the LUKS header in it if it is still anywhere.

Quoted 6 lines

3-They appear as raid disks but again I cannot open the encryption.

Quoted 3 lines

If neither gives you a LUKS header, you can still search on the raw disks by looking for the LUKS signature. If that also fails, you are out of luck and all your data is gone.

The LUKS signature is simply 'LUKS'?

Not quite. FAQ Item 6.12
(https://gitlab.com/cryptsetup/cryptsetup/wikis/FrequentlyAskedQuestions)
gives you a brief version, the LUKS on-disk spec gives everything.

Quoted 6 lines

I grepped like this:

grep LUKS header_sdb_backup.dmp

is it the correct way to do it?
Did not find any match...

Ok, lets repeat that with the full disks and including the full signature

hd /dev/sdx | grep "0 4c 55 4b 53 ba be 00 01"

with x one of your RAID disks. Do this for both. May take a while. This gives you the alignment as well. The "hd" start of a good luks header and container (header starts at offset 0) looks like this:

Quoted 6 lines

00000000 4c 55 4b 53 ba be 00 01 61 65 73 00 00 00 00 00 |LUKS....aes.....|
Only the first 6 bytes are fixed. Bytes 6 and 7 are the version of which there currently is onlyy "0001". This will always be aligned to a 512 byte boundary. Doing it this way has the advantage that you get the offset as well.

found it in one of the disks:
08100000 4c 55 4b 53 ba be 00 01 61 65 73 00 00 00 00 00 |LUKS....aes.....|

Can you tell me how should I proceed now?

(the other is still being searched: the first one took a few seconds, this one is now over 1 hour search)

Many thanks,
Luis

Quoted 3 lines

Regards,
Arno

_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt

Now to your question, once you know the offset of the header:
1.)Setup a loop device from your image (You can use an offset into the image where your loop device starts with sector 0) see --offset in losetup man page.

Inspect loopdevice if the LUKS Header now is on sector 0 2.)Try a cryptsetup luksopen in readonly mode

[...]

Quoted 4 lines

Now to your question, once you know the offset of the header:
1.)Setup a loop device from your image (You can use an offset into the image where your loop device starts with sector 0) see --offset in losetup man page.

Ah, yes. That would save copying it.

Quoted 3 lines

Inspect loopdevice if the LUKS Header now is on sector 0 2.)Try a cryptsetup luksopen in readonly mode

Good idea. With that it may be reasonaly safe to work with the original disk. I still would make a full backup before.

Regards,
Arno

Am 23.11.2015 um 04:35 schrieb Arno Wagner:

Quoted 7 lines · Sven Eschenberg

[...]

Quoted 4 lines

Now to your question, once you know the offset of the header:
1.)Setup a loop device from your image (You can use an offset into the image where your loop device starts with sector 0) see --offset in losetup man page.

Quoted 1 line

Ah, yes. That would save copying it.

That was the plan. In general using dmsetup to create a mapping manually should work too, if loop device support is missing - dmsetup is pretty cryptic to use though.

Quoted 10 lines

Quoted 2 lines

Inspect loopdevice if the LUKS Header now is on sector 0 2.)Try a cryptsetup luksopen in readonly mode

Good idea. With that it may be reasonaly safe to work with the original disk. I still would make a full backup before.

Regards,
Arno

Well, I thought about using the loop on the file while the physical disk stays unchanged. Otherwise it would be possible to work on the physical disk, and keep a safety image. No matter which way one chooses, always have a safety copy.

If the disk is having mechanical problems or something similiar one would of course use 2 images, one 'master binary backup' and the replica to work on.

Once mapping and opening works, one can choose to either copy out the files and backup (usually a good idea) or to create a copy in the manner you described. Possibly such an image could then be remerged onto a new clean array, if it is otherwise intact. Not without some remaining risks though.

Quoted 10 lines · Sven Eschenberg

Am 23.11.2015 um 04:35 schrieb Arno Wagner:

Quoted 7 lines · Sven Eschenberg

[...]

Quoted 4 lines

Now to your question, once you know the offset of the header:
1.)Setup a loop device from your image (You can use an offset into the image where your loop device starts with sector 0) see --offset in losetup man page.

Quoted 5 lines

Quoted 1 line

Ah, yes. That would save copying it.

That was the plan. In general using dmsetup to create a mapping manually should work too, if loop device support is missing - dmsetup is pretty cryptic to use though.

"Cryptic" is not good here...

I was not aware that losetup allows read-only mappings, or I would probably have looked at it too.

Excellent! So I learned something too!

Quoted 20 lines

Quoted 10 lines

Quoted 2 lines

Inspect loopdevice if the LUKS Header now is on sector 0 2.)Try a cryptsetup luksopen in readonly mode

Good idea. With that it may be reasonaly safe to work with the original disk. I still would make a full backup before.

Regards,
Arno

Well, I thought about using the loop on the file while the physical disk stays unchanged. Otherwise it would be possible to work on the physical disk, and keep a safety image. No matter which way one chooses, always have a safety copy.

If the disk is having mechanical problems or something similiar one would of course use 2 images, one 'master binary backup' and the replica to work on.

Quoted 5 lines

Once mapping and opening works, one can choose to either copy out the files and backup (usually a good idea) or to create a copy in the manner you described. Possibly such an image could then be remerged onto a new clean array, if it is otherwise intact. Not without some remaining risks though.

Indeed. And first things first, lets see whether that header is viable before goping any further.

Regards,
Arno

Dear Arno and Sven,

Thanks very much for your support.

I was able to open the copy I made with

tail -c +135266305 /dev/sdx > /dev/sdy
and
cryptsetup luksOpen /dev/sdy e1

and have all my data back.

Thanks again for your support.

Best regards,
Luis

Quoted 50 lines · Arno Wagner

Quoted 10 lines · Sven Eschenberg

Am 23.11.2015 um 04:35 schrieb Arno Wagner:

Quoted 7 lines · Sven Eschenberg

[...]

Quoted 4 lines

Now to your question, once you know the offset of the header:
1.)Setup a loop device from your image (You can use an offset into the image where your loop device starts with sector 0) see --offset in losetup man page.

Quoted 5 lines

Quoted 1 line

Ah, yes. That would save copying it.

That was the plan. In general using dmsetup to create a mapping manually should work too, if loop device support is missing - dmsetup is pretty cryptic to use though.

"Cryptic" is not good here...

I was not aware that losetup allows read-only mappings, or I would probably have looked at it too.

Excellent! So I learned something too!

Quoted 20 lines

Quoted 10 lines

Quoted 2 lines

Inspect loopdevice if the LUKS Header now is on sector 0 2.)Try a cryptsetup luksopen in readonly mode

Good idea. With that it may be reasonaly safe to work with the original disk. I still would make a full backup before.

Regards,
Arno

Well, I thought about using the loop on the file while the physical disk stays unchanged. Otherwise it would be possible to work on the physical disk, and keep a safety image. No matter which way one chooses, always have a safety copy.

If the disk is having mechanical problems or something similiar one would of course use 2 images, one 'master binary backup' and the replica to work on.

Quoted 5 lines

Once mapping and opening works, one can choose to either copy out the files and backup (usually a good idea) or to create a copy in the manner you described. Possibly such an image could then be remerged onto a new clean array, if it is otherwise intact. Not without some remaining risks though.

Quoted 6 lines

Indeed. And first things first, lets see whether that header is viable before goping any further.

Regards,
Arno

_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt

Dear Arno and Sven,

Thanks very much for your support.

I was able to open the copy I made with

tail -c +135266305 /dev/sdx > /dev/sdy
and
cryptsetup luksOpen /dev/sdy e1

and have all my data back.

Thanks again for your support.

Best regards,
Luis

Quoted 50 lines · Arno Wagner

Quoted 10 lines · Sven Eschenberg

Am 23.11.2015 um 04:35 schrieb Arno Wagner:

Quoted 7 lines · Sven Eschenberg

[...]

Quoted 4 lines

Now to your question, once you know the offset of the header:
1.)Setup a loop device from your image (You can use an offset into the image where your loop device starts with sector 0) see --offset in losetup man page.

Quoted 5 lines

Quoted 1 line

Ah, yes. That would save copying it.

That was the plan. In general using dmsetup to create a mapping manually should work too, if loop device support is missing - dmsetup is pretty cryptic to use though.

"Cryptic" is not good here...

I was not aware that losetup allows read-only mappings, or I would probably have looked at it too.

Excellent! So I learned something too!

Quoted 20 lines

Quoted 10 lines

Quoted 2 lines

Inspect loopdevice if the LUKS Header now is on sector 0 2.)Try a cryptsetup luksopen in readonly mode

Good idea. With that it may be reasonaly safe to work with the original disk. I still would make a full backup before.

Regards,
Arno

Well, I thought about using the loop on the file while the physical disk stays unchanged. Otherwise it would be possible to work on the physical disk, and keep a safety image. No matter which way one chooses, always have a safety copy.

If the disk is having mechanical problems or something similiar one would of course use 2 images, one 'master binary backup' and the replica to work on.

Quoted 5 lines

Once mapping and opening works, one can choose to either copy out the files and backup (usually a good idea) or to create a copy in the manner you described. Possibly such an image could then be remerged onto a new clean array, if it is otherwise intact. Not without some remaining risks though.

Quoted 6 lines

Indeed. And first things first, lets see whether that header is viable before goping any further.

Regards,
Arno

_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt

Hi.

I have a raid1 (mdadm-based) setup with two disks. I had them encrypted with luks. Everything was ok for 2 years. My PC had a problem and I had to mount this on a new PC.
When I tried to start the raid on the new PC it only started 1 of the disks because the other had been replaced on a different PC and had a different hostname (my original PC had a script to assemble the raid even with the two disks having a different hostname).

So I tried to fix this different hostname problem by re-creating the raid in the new PC using

mdadm -C /dev/md127 -l1 -n2 --assume-clean --metadata=1.2 /dev/sdb /dev/sdc1 --uuid=1d925c8d:8c8bb953:4e9070f7:43344cf9

mdadm: /dev/sdb appears to be part of a raid array:

     level=raid1 devices=2 ctime=Sun Feb 12 19:40:32 2012
mdadm: partition table exists on /dev/sdb but will be lost or
        meaningless after creating array
mdadm: /dev/sdc1 appears to contain an ext2fs file system
     size=976760832K  mtime=Mon Aug 25 19:50:27 2014
mdadm: /dev/sdc1 appears to be part of a raid array:
     level=raid1 devices=2 ctime=Tue Aug 26 09:10:39 2014

Continue creating array? y
mdadm: array /dev/md127 started.

All appeared to be OK:

cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md127 : active (auto-read-only) raid1 sdc1[1] sdb[0]

       976629568 blocks super 1.2 [2/2] [UU]

unused devices: <none>

but now luks does not open the raid:

sudo cryptsetup luksOpen /dev/md127 raid1
Device /dev/md127 is not a valid LUKS device.

Any ideas on how to re-open the raid with luks?

Note: I thought there would be no problem with the create command because of this in the mdadm man page:
"Create Create a new array with per-device metadata (superblocks). Appropriate metadata is written to each device, and then the array comprising those devices is activated. A 'resync' process is started to make sure that the array is consistent (e.g. both sides of a mirror contain the same data) but *the content of the device is left otherwise untouched*. "

Thanks for any help you can provide.
Luis

_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt