Connection lost during BTRFS move + resize
Good morning,
I couldn't find any definitive guidance on the list archives or Internet, so I want to double-check before giving up.
I tried to left-move and resize a btrfs partition on a USB-attached hard drive. My intention was to expand the partition from 2 TB to 3 TB on a 4TB drive. During the move, the USB cable came loose and the process failed.
From what I can tell, the partition was "moved" to its new location and correctly shows the usage, but the partition is not expanded to its new 3 TB limit. As one would expect, sudo mount -o ro /dev/sdb3 /mnt yields:
mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sdb3, missing codepage or helper program, or other error.
Although I know it's extremely dangerous, I nevertheless ran btrfs check --repair /dev/sdb3 and received:
Starting repair.
Opening filesystem to check...
checksum verify failed on 2160397959168 wanted 0x2d75ada8 found 0x55dc86b3
checksum verify failed on 2160397959168 wanted 0x5c57dcfd found 0xe722d853
checksum verify failed on 2160397959168 wanted 0x5c57dcfd found 0xe722d853
bad tree block 2160397959168, bytenr mismatch, want=2160397959168, have=8937084726424501725 Couldn't read tree root
ERROR: cannot open file system
As requested:
uname -a: Linux debian 5.15.0-1-amd64 #1 SMP Debian 5.15.3-1 (2021-11-18) x86_64 GNU/Linux btrfs --version: btrfs-progs v5.15
sudo btrfs fi show: Label: 'Backup' uuid: <drive UUID, still good>
Total devices 1 FS bytes used 1.71TiB
devid 1 size 1.82TiB used 1.73TiB path /dev/sdb3
dmesg: output at http://paste.debian.net/1221191/, valid until 2021-12-02
From other discussions, it seems like the partition's contents are gone for good, which is OK because it's a backup and I can start over. However, if I can recover the data, that would be nice, too, since I might have to dig up something from a while back.
With many thanks,
Quoted 8 lines · Borden
Good morning,
I couldn't find any definitive guidance on the list archives or Internet, so I want to double-check before giving up.
I tried to left-move and resize a btrfs partition on a USB-attached hard drive. My intention was to expand the partition from 2 TB to 3 TB on a 4TB drive. During the move, the USB cable came loose and the process failed.
The only tool I know of that can do this is gparted, so I assume you are using that. In this case, it has to umount the filesystem and manually copy data from the old start of the partition to the new start. Being interrupted in the middle leaves part of the filesystem in the wrong place ( and which parts is unknowable ), and so it is toast. This is one area where LVM has a significant advantage as its moves are interruption safe and automatically resumed on the next activation of the volume.
29 Nov 2021, 10:26 by [email protected]:
Quoted 9 lines
The only tool I know of that can do this is gparted, so I assume you are using that. In this case, it has to umount the filesystem and manually copy data from the old start of the partition to the new start. Being interrupted in the middle leaves part of the filesystem in the wrong place ( and which parts is unknowable ), and so it is toast. This is one area where LVM has a significant advantage as its moves are interruption safe and automatically resumed on the next activation of the volume.
This is the answer that I anticipated, and it's good to know now so I don't destroy data that I _cannot_ afford to lose later. So thank you.
For my own education/curiosity/intellectual banter: ddrescue, badblocks, rsync and other utilities have log files that track progress and allow it to resume if it's interrupted. Since resize operations work in the linear process you described, how hard would it be, theoretically, to implement a "needle position" in a move operation to allow a move to pick up where it left off?
Obviously, it wouldn't be 100% perfect, but if a recovery utility could look at the disk and say "partition starts here, skip a bit somewhere in the middle, continue here, stop there," surely that would be more efficient than trying to recover files with a low-level utility?
Quoted 6 lines · Borden
For my own education/curiosity/intellectual banter: ddrescue, badblocks, rsync and other utilities have log files that track progress and allow it to resume if it's interrupted. Since resize operations work in the linear process you described, how hard would it be, theoretically, to implement a "needle position" in a move operation to allow a move to pick up where it left off?
Theoretically it shouldn't be too hard. It's just a matter of deciding on a location where you can safely record the checkpoint information and then update the checkpoint between blocks. That's how LVM handles moves safely. In the worst case, you restart the move at the last checkpoint and just waste some time copying data that was already copied but not checkpointed.
29 Nov 2021, 11:20 by [email protected]:
Quoted 7 lines
Theoretically it shouldn't be too hard. It's just a matter of deciding on a location where you can safely record the checkpoint information and then update the checkpoint between blocks. That's how LVM handles moves safely. In the worst case, you restart the move at the last checkpoint and just waste some time copying data that was already copied but not checkpointed.
Thanks again. In those other utilities, since the logs get written to a plain text file of the user's choosing, and partition moving should be offline, anyhow, it would be reasonable to expect the user to provide a safe location to stash a checkpoint file.
And, of course, if the user chooses to manipulate the file and/or forego it altogether, their funeral.
Not sure if it's a feature worth requesting, but it would keep people like me from bothering the list with these Hail Mary support requests, so probably be net-cost negative.
Quoted 10 lines · Borden
29 Nov 2021, 10:26 by [email protected]:
Quoted 9 lines
The only tool I know of that can do this is gparted, so I assume you are using that. In this case, it has to umount the filesystem and manually copy data from the old start of the partition to the new start. Being interrupted in the middle leaves part of the filesystem in the wrong place ( and which parts is unknowable ), and so it is toast. This is one area where LVM has a significant advantage as its moves are interruption safe and automatically resumed on the next activation of the volume.
Quoted 6 lines
This is the answer that I anticipated, and it's good to know now so I don't destroy data that I _cannot_ afford to lose later. So thank you.
For my own education/curiosity/intellectual banter: ddrescue, badblocks, rsync and other utilities have log files that track progress and allow it to resume if it's interrupted. Since resize operations work in the linear process you described, how hard would it be, theoretically, to implement a "needle position" in a move operation to allow a move to pick up where it left off?
Obviously, it wouldn't be 100% perfect, but if a recovery utility could look at the disk and say "partition starts here, skip a bit somewhere in the middle, continue here, stop there," surely that would be more efficient than trying to recover files with a low-level utility?
I can't comment on that, and I don't know how the utility you were using works, but if it was copying blocks from higher disk addresses to lower ones, starting at the bottom, it is *possible* that it hadn't got beyond the first 1TB before it failed and the original filesystem is still untouched.
Did you try just resetting the original partition parameters manually (forcing them using something like GNU parted's mkpart - not a resize operation) to see whether the original filesystem could be mounted? It's just a long shot, of course.
Graham
Even if it wrote more than 1 TB and started overwriting the file system, it SHOULD be possible to find what data still needs to be written by searching backwards from the old partition's end point and the new partition's end point until you start finding identical data.
Matthew Warren
Quoted 31 lines · Graham Cobb
Quoted 10 lines · Borden
29 Nov 2021, 10:26 by [email protected]:
Quoted 9 lines
The only tool I know of that can do this is gparted, so I assume you are using that. In this case, it has to umount the filesystem and manually copy data from the old start of the partition to the new start. Being interrupted in the middle leaves part of the filesystem in the wrong place ( and which parts is unknowable ), and so it is toast. This is one area where LVM has a significant advantage as its moves are interruption safe and automatically resumed on the next activation of the volume.
Quoted 6 lines
This is the answer that I anticipated, and it's good to know now so I don't destroy data that I _cannot_ afford to lose later. So thank you.
For my own education/curiosity/intellectual banter: ddrescue, badblocks, rsync and other utilities have log files that track progress and allow it to resume if it's interrupted. Since resize operations work in the linear process you described, how hard would it be, theoretically, to implement a "needle position" in a move operation to allow a move to pick up where it left off?
Obviously, it wouldn't be 100% perfect, but if a recovery utility could look at the disk and say "partition starts here, skip a bit somewhere in the middle, continue here, stop there," surely that would be more efficient than trying to recover files with a low-level utility?
I can't comment on that, and I don't know how the utility you were using works, but if it was copying blocks from higher disk addresses to lower ones, starting at the bottom, it is *possible* that it hadn't got beyond the first 1TB before it failed and the original filesystem is still untouched.
Did you try just resetting the original partition parameters manually (forcing them using something like GNU parted's mkpart - not a resize operation) to see whether the original filesystem could be mounted? It's just a long shot, of course.
Graham
Quoted 13 lines · Phillip Susi
Quoted 8 lines · Borden
Good morning,
I couldn't find any definitive guidance on the list archives or Internet, so I want to double-check before giving up.
I tried to left-move and resize a btrfs partition on a USB-attached hard drive. My intention was to expand the partition from 2 TB to 3 TB on a 4TB drive. During the move, the USB cable came loose and the process failed.
Quoted 8 lines
The only tool I know of that can do this is gparted, so I assume you are using that. In this case, it has to umount the filesystem and manually copy data from the old start of the partition to the new start. Being interrupted in the middle leaves part of the filesystem in the wrong place ( and which parts is unknowable ), and so it is toast. This is one area where LVM has a significant advantage as its moves are interruption safe and automatically resumed on the next activation of the volume.
Whether LVM or Btrfs, you can just add the earlier partition to the storage pool. No need to move extents around, and near as I can tell no advantage of doing so.
i.e. if the btrfs is on vda2 and it's now desired to expand the file system "forward" into the space defined by vda1, just add it
btrfs device add /dev/vda1 /mnt
The command implies an abbreviated mkfs on vda1, and resizes the btrfs file system to encompass both devices. And it's also an interrupt safe operation unlike gparted move/resize which should come with dire warnings about the consequence for any interruption.
Signature · 1 line
Chris Murphy
Quoted 11 lines · Borden
29 Nov 2021, 10:26 by [email protected]:
Quoted 9 lines
The only tool I know of that can do this is gparted, so I assume you are using that. In this case, it has to umount the filesystem and manually copy data from the old start of the partition to the new start. Being interrupted in the middle leaves part of the filesystem in the wrong place ( and which parts is unknowable ), and so it is toast. This is one area where LVM has a significant advantage as its moves are interruption safe and automatically resumed on the next activation of the volume.
Quoted 3 lines
This is the answer that I anticipated, and it's good to know now so I don't destroy data that I _cannot_ afford to lose later. So thank you.
For my own education/curiosity/intellectual banter: ddrescue, badblocks, rsync and other utilities have log files that track progress and allow it to resume if it's interrupted. Since resize operations work in the linear process you described, how hard would it be, theoretically, to implement a "needle position" in a move operation to allow a move to pick up where it left off?
Trivial as a thought exercise, and not difficult for someone who can do some scripting or coding. Do a block by block comparison to find out where the "forward" (new location) writes ended. You can make a bunch of assumptions that at the moment of failure, only a specific contiguous set of sectors were in the process of receiving writes. So it's just a matter of finding where it left off. Optimization:
probably some minimal sampling you could do to get in the ballpark then tighten it up to a block by block to find out exactly where it stopped.
Instead of resuming the move, you can just use some device mapper commands to create a range (offset+length) for the two segments and make the two appear in a single virtual device (i.e. you delete the gap between the new range and old range). And now you can mount it and do whatever you want, back it up somewhere.
Quoted 1 line
Obviously, it wouldn't be 100% perfect, but if a recovery utility could look at the disk and say "partition starts here, skip a bit somewhere in the middle, continue here, stop there," surely that would be more efficient than trying to recover files with a low-level utility?
Yes.
Signature · 1 line
Chris Murphy