How to recover from failing btrffs send | btrfs receive?
So, I've veen running this for a few weeks, and soon should have something half decent to share for others to use.
Unfortunately, one of my backups is now failing like so:
btrfs send -p "$src_snap" "$src_newsnap" | btrfs receive "$dest_pool/"
+ btrfs send -p /mnt/btrfs_pool1/home_ro.20140209_12:00:01 home_ro.20140212_05:37:49
+ btrfs receive /mnt/btrfs_pool2//
At subvol home_ro.20140212_05:37:49
At snapshot home_ro.20140212_05:37:49
ERROR: rmdir o1845158-142-0 failed. No such file or directory
This looks like it got in an unfinished state it can't recover from.
This was with kernel 3.12.7.
Can I self fix this somehow, I know I can use rsync to make both sides the sames, but incremental send/receive will not work anymore after that, correct?
Except, not really. Now I'm confused.
legolas:/mnt/btrfs_pool1# rsync -avSH --delete --dry-run /mnt/btrfs_pool1/home_ro.20140209_12:00:01/. /mnt/btrfs_pool2/home_ro.20140209_12\:00\:01/. sending incremental file list ./
sent 116867233 bytes received 265427 bytes 92194.14 bytes/sec total size is 129135042766 speedup is 1102.47 (DRY RUN) legolas:/mnt/btrfs_pool1#
(I know I start the entire backup over from scratch, but for obvious reasons, restarting an entire backup from scratch each time I get an error isn't great since it could take hours or days to backup that much data)
Suggestions welcome :)
Thanks,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901Ok, let me try something else :)
Of those who are using btrfs send/receive, has anyone gotten in a state where incrementals will not apply anymore?
Thanks,
Marc
Quoted 48 lines · Marc MERLIN
So, I've veen running this for a few weeks, and soon should have something half decent to share for others to use.
Unfortunately, one of my backups is now failing like so:
btrfs send -p "$src_snap" "$src_newsnap" | btrfs receive "$dest_pool/"
+ btrfs send -p/mnt/btrfs_pool1/home_ro.20140209_12:00:01home_ro.20140212_05:37:49
+ btrfs receive/mnt/btrfs_pool2//
At subvol home_ro.20140212_05:37:49
At snapshot home_ro.20140212_05:37:49
ERROR: rmdir o1845158-142-0 failed. No such file or directoryThis looks like it got in an unfinished state it can't recover from.
This was with kernel 3.12.7.
Can I self fix this somehow, I know I can use rsync to make both sides the sames, but incremental send/receive will not work anymore after that, correct?
Except, not really. Now I'm confused.
legolas:/mnt/btrfs_pool1# rsync -avSH --delete --dry-run /mnt/btrfs_pool1/home_ro.20140209_12:00:01/. /mnt/btrfs_pool2/home_ro.20140209_12\:00\:01/. sending incremental file list ./sent 116867233 bytes received 265427 bytes 92194.14 bytes/sec total size is 129135042766 speedup is 1102.47 (DRY RUN) legolas:/mnt/btrfs_pool1#
(I know I start the entire backup over from scratch, but for obvious reasons, restarting an entire backup from scratch each time I get an error isn't great since it could take hours or days to backup that much data)
Suggestions welcome :)
Thanks,
MarcSignature · 8 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901PATCH 1/1 Btrfs: incremental send, fix invalid path after dir rename
This fixes yet one more case not caught by the commit titled:
Btrfs: fix infinite path build loops in incremental send
In this case, even before the initial full send, we have a directory which is a child of a directory with a higher inode number. Then we perform the initial send, and after we rename both the child and the parent, without moving them around. After doing these 2 renames, an incremental send sent a rename instruction for the child directory which contained an invalid "from" path (referenced the parent's old name, not the new one), which made the btrfs receive command fail.
Steps to reproduce:
$ mkfs.btrfs -f /dev/sdb3
$ mount /dev/sdb3 /mnt/btrfs
$ mkdir -p /mnt/btrfs/a/b
$ mkdir /mnt/btrfs/d
$ mkdir /mnt/btrfs/a/b/c
$ mv /mnt/btrfs/d /mnt/btrfs/a/b/c
$ btrfs subvolume snapshot -r /mnt/btrfs /mnt/btrfs/snap1
$ btrfs send /mnt/btrfs/snap1 -f /tmp/base.send
$ mv /mnt/btrfs/a/b/c /mnt/btrfs/a/b/x
$ mv /mnt/btrfs/a/b/x/d /mnt/btrfs/a/b/x/y
$ btrfs subvolume snapshot -r /mnt/btrfs /mnt/btrfs/snap2
$ btrfs send -p /mnt/btrfs/snap1 /mnt/btrfs/snap2 -f /tmp/incremental.send $ umout /mnt/btrfs
$ mkfs.btrfs -f /dev/sdb3
$ mount /dev/sdb3 /mnt/btrfs
$ btrfs receive /mnt/btrfs -f /tmp/base.send
$ btrfs receive /mnt/btrfs -f /tmp/incremental.sendThe second btrfs receive command failed with:
"ERROR: rename a/b/c/d -> a/b/x/y failed. No such file or directory"
A test case for xfstests follows.
--- fs/btrfs/send.c | 41 ++++++++++++++++++++++++++++++++++------- 1 file changed, 34 insertions(+), 7 deletions(-)
fs/btrfs/send.c +34 -7
@@ -2847,19 +2847,48 @@ static int apply_dir_move(struct send_ctx *sctx, struct pending_dir_move *pm) {struct fs_path *from_path =NULL ;struct fs_path *to_path =NULL ; +struct fs_path *name =NULL ;u64 orig_progress = sctx->send_progress;struct recorded_ref *cur; +u64 parent_ino, parent_gen;int ret; + name =fs_path_alloc (); from_path =fs_path_alloc (); -if (!from_path) -return -ENOMEM ; +if (!name || !from_path) { + ret = -ENOMEM ; + goto out; + } - sctx->send_progress = pm->ino; - ret =get_cur_path (sctx, pm->ino, pm->gen, from_path); + ret =del_waiting_dir_move (sctx, pm->ino); +ASSERT (ret ==0 ); + + ret =get_first_ref (sctx->parent_root, pm->ino, + &parent_ino, &parent_gen, name);if (ret <0 ) goto out; +if (parent_ino == sctx->cur_ino) { +/* child only renamed, not moved */ +ASSERT (parent_gen == sctx->cur_inode_gen); + ret =get_cur_path (sctx, sctx->cur_ino, sctx->cur_inode_gen, + from_path); +if (ret <0 ) + goto out; + ret =fs_path_add_path (from_path, name); +if (ret <0 ) + goto out; + }else { +/* child moved and maybe renamed too */ + sctx->send_progress = pm->ino; + ret =get_cur_path (sctx, pm->ino, pm->gen, from_path); +if (ret <0 ) + goto out; + } + +fs_path_free (name); + name =NULL ; + to_path =fs_path_alloc ();if (!to_path) { ret = -ENOMEM ;
@@ -2867,9 +2896,6 @@ static int apply_dir_move(struct send_ctx *sctx, struct pending_dir_move *pm) } sctx->send_progress = sctx->cur_ino +1 ; - ret =del_waiting_dir_move (sctx, pm->ino); -ASSERT (ret ==0 ); - ret =get_cur_path (sctx, pm->ino, pm->gen, to_path);if (ret <0 ) goto out;
@@ -2893,6 +2919,7 @@ static int apply_dir_move(struct send_ctx *sctx, struct pending_dir_move *pm) } out: +fs_path_free (name);fs_path_free (from_path);fs_path_free (to_path); sctx->send_progress = orig_progress;
Signature · 1 line
1.7.9.5
Re: How to recover from failing btrfs send | btrfs receive?
Hi Fillipe, I see you have another fix for btrfs send (attached below), as ell as your other patch on Jan 21st (neither are in my 3.12.7).
Quoted 2 lines
From my error below,
ERROR: rmdir o1845158-142-0 failed. No such file or directory
can you tell if I'm having a different problem than the two patches you sent?
More generally, could you hint how I can tell what this
rmdir o1845158-142-0
refers to, considering that it looks like a made up filename by btrfs send/receive?
I'm happy to go to 3.13.3 and apply both patches if you think it'll help.
Thanks,
Marc
Quoted 38 lines · Marc MERLIN
So, I've veen running this for a few weeks, and soon should have something half decent to share for others to use.
Unfortunately, one of my backups is now failing like so:
btrfs send -p "$src_snap" "$src_newsnap" | btrfs receive "$dest_pool/"
+ btrfs send -p/mnt/btrfs_pool1/home_ro.20140209_12:00:01home_ro.20140212_05:37:49
+ btrfs receive/mnt/btrfs_pool2//
At subvol home_ro.20140212_05:37:49
At snapshot home_ro.20140212_05:37:49
ERROR: rmdir o1845158-142-0 failed. No such file or directoryThis looks like it got in an unfinished state it can't recover from.
This was with kernel 3.12.7.
Can I self fix this somehow, I know I can use rsync to make both sides the sames, but incremental send/receive will not work anymore after that, correct?
Except, not really. Now I'm confused.
legolas:/mnt/btrfs_pool1# rsync -avSH --delete --dry-run /mnt/btrfs_pool1/home_ro.20140209_12:00:01/. /mnt/btrfs_pool2/home_ro.20140209_12\:00\:01/. sending incremental file list ./sent 116867233 bytes received 265427 bytes 92194.14 bytes/sec total size is 129135042766 speedup is 1102.47 (DRY RUN) legolas:/mnt/btrfs_pool1#
(I know I start the entire backup over from scratch, but for obvious reasons, restarting an entire backup from scratch each time I get an error isn't great since it could take hours or days to backup that much data)
Suggestions welcome :)
Thanks,
Marc
Quoted 126 lines · Filipe David Borba Manana
This fixes yet one more case not caught by the commit titled:
Btrfs: fix infinite path build loops in incremental send
In this case, even before the initial full send, we have a directory which is a child of a directory with a higher inode number. Then we perform the initial send, and after we rename both the child and the parent, without moving them around. After doing these 2 renames, an incremental send sent a rename instruction for the child directory which contained an invalid "from" path (referenced the parent's old name, not the new one), which made the btrfs receive command fail.
Steps to reproduce:
$ mkfs.btrfs -f /dev/sdb3 $ mount /dev/sdb3 /mnt/btrfs $ mkdir -p /mnt/btrfs/a/b $ mkdir /mnt/btrfs/d $ mkdir /mnt/btrfs/a/b/c $ mv /mnt/btrfs/d /mnt/btrfs/a/b/c $ btrfs subvolume snapshot -r /mnt/btrfs /mnt/btrfs/snap1 $ btrfs send /mnt/btrfs/snap1 -f /tmp/base.send $ mv /mnt/btrfs/a/b/c /mnt/btrfs/a/b/x $ mv /mnt/btrfs/a/b/x/d /mnt/btrfs/a/b/x/y $ btrfs subvolume snapshot -r /mnt/btrfs /mnt/btrfs/snap2 $ btrfs send -p /mnt/btrfs/snap1 /mnt/btrfs/snap2 -f /tmp/incremental.send$ umout /mnt/btrfs $ mkfs.btrfs -f /dev/sdb3 $ mount /dev/sdb3 /mnt/btrfs $ btrfs receive /mnt/btrfs -f /tmp/base.send $ btrfs receive /mnt/btrfs -f /tmp/incremental.sendThe second btrfs receive command failed with:
"ERROR: rename a/b/c/d -> a/b/x/y failed. No such file or directory"A test case for xfstests follows.
Signed-off-by Filipe David Borba Manana <[email protected]>--- fs/btrfs/send.c | 41 ++++++++++++++++++++++++++++++++++------- 1 file changed, 34 insertions(+), 7 deletions(-)fs/btrfs/send.c +34
-7@@ -2847,19 +2847,48 @@ static int apply_dir_move(struct send_ctx *sctx, struct pending_dir_move *pm) {struct fs_path *from_path =NULL ;struct fs_path *to_path =NULL ; +struct fs_path *name =NULL ;u64 orig_progress = sctx->send_progress;struct recorded_ref *cur; +u64 parent_ino, parent_gen;int ret; + name =fs_path_alloc (); from_path =fs_path_alloc (); -if (!from_path) -return -ENOMEM ; +if (!name || !from_path) { + ret = -ENOMEM ; + goto out; + } - sctx->send_progress = pm->ino; - ret =get_cur_path (sctx, pm->ino, pm->gen, from_path); + ret =del_waiting_dir_move (sctx, pm->ino); +ASSERT (ret ==0 ); + + ret =get_first_ref (sctx->parent_root, pm->ino, + &parent_ino, &parent_gen, name);if (ret <0 ) goto out; +if (parent_ino == sctx->cur_ino) { +/* child only renamed, not moved */ +ASSERT (parent_gen == sctx->cur_inode_gen); + ret =get_cur_path (sctx, sctx->cur_ino, sctx->cur_inode_gen, + from_path); +if (ret <0 ) + goto out; + ret =fs_path_add_path (from_path, name); +if (ret <0 ) + goto out; + }else { +/* child moved and maybe renamed too */ + sctx->send_progress = pm->ino; + ret =get_cur_path (sctx, pm->ino, pm->gen, from_path); +if (ret <0 ) + goto out; + } + +fs_path_free (name); + name =NULL ; + to_path =fs_path_alloc ();if (!to_path) { ret = -ENOMEM ;@@ -2867,9 +2896,6 @@ static int apply_dir_move(struct send_ctx *sctx, struct pending_dir_move *pm) } sctx->send_progress = sctx->cur_ino +1 ; - ret =del_waiting_dir_move (sctx, pm->ino); -ASSERT (ret ==0 ); - ret =get_cur_path (sctx, pm->ino, pm->gen, to_path);if (ret <0 ) goto out;@@ -2893,6 +2919,7 @@ static int apply_dir_move(struct send_ctx *sctx, struct pending_dir_move *pm) } out: +fs_path_free (name);fs_path_free (from_path);fs_path_free (to_path); sctx->send_progress = orig_progress;Signature · 6 lines
1.7.9.5 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901Re: How to recover from failing btrfs send | btrfs receive?
Quoted 2 lines · Marc MERLIN
Hi Fillipe, I see you have another fix for btrfs send (attached below), as ell as your other patch on Jan 21st (neither are in my 3.12.7).
Hi Marc,
Some replies below inlined.
Quoted 5 lines
From my error below,
Quoted 1 line
ERROR: rmdir o1845158-142-0 failed. No such file or directory
can you tell if I'm having a different problem than the two patches you sent?
I think it's a different problem.
The issues I have been fixing are about child directories with lower inode numbers then their parents and renaming/moving them.
Quoted 5 lines
More generally, could you hint how I can tell what this
rmdir o1845158-142-0
refers to, considering that it looks like a made up filename by btrfs send/receive?
Those ox-y-z names are for orphan inodes, and generated by btrfs send yes.
Since you don't seem to know the sequence of steps (filesystem ops) that lead to that issue, perhaps you can run 'tree -a --inodes' against the root of the subvolume and lets us know what the full output is. That might help in case it's one more case similar to those I have been fixing recently.
thanks
Quoted 181 lines
I'm happy to go to 3.13.3 and apply both patches if you think it'll help.
Thanks,
MarcQuoted 38 lines · Marc MERLIN
So, I've veen running this for a few weeks, and soon should have something half decent to share for others to use.
Unfortunately, one of my backups is now failing like so:
btrfs send -p "$src_snap" "$src_newsnap" | btrfs receive "$dest_pool/"
+ btrfs send -p/mnt/btrfs_pool1/home_ro.20140209_12:00:01home_ro.20140212_05:37:49
+ btrfs receive/mnt/btrfs_pool2//
At subvol home_ro.20140212_05:37:49
At snapshot home_ro.20140212_05:37:49
ERROR: rmdir o1845158-142-0 failed. No such file or directoryThis looks like it got in an unfinished state it can't recover from.
This was with kernel 3.12.7.
Can I self fix this somehow, I know I can use rsync to make both sides the sames, but incremental send/receive will not work anymore after that, correct?
Except, not really. Now I'm confused.
legolas:/mnt/btrfs_pool1# rsync -avSH --delete --dry-run /mnt/btrfs_pool1/home_ro.20140209_12:00:01/. /mnt/btrfs_pool2/home_ro.20140209_12\:00\:01/. sending incremental file list ./sent 116867233 bytes received 265427 bytes 92194.14 bytes/sec total size is 129135042766 speedup is 1102.47 (DRY RUN) legolas:/mnt/btrfs_pool1#
(I know I start the entire backup over from scratch, but for obvious reasons, restarting an entire backup from scratch each time I get an error isn't great since it could take hours or days to backup that much data)
Suggestions welcome :)
Thanks,
MarcQuoted 121 lines · Filipe David Borba Manana
This fixes yet one more case not caught by the commit titled:
Btrfs: fix infinite path build loops in incremental send
In this case, even before the initial full send, we have a directory which is a child of a directory with a higher inode number. Then we perform the initial send, and after we rename both the child and the parent, without moving them around. After doing these 2 renames, an incremental send sent a rename instruction for the child directory which contained an invalid "from" path (referenced the parent's old name, not the new one), which made the btrfs receive command fail.
Steps to reproduce:
$ mkfs.btrfs -f /dev/sdb3 $ mount /dev/sdb3 /mnt/btrfs $ mkdir -p /mnt/btrfs/a/b $ mkdir /mnt/btrfs/d $ mkdir /mnt/btrfs/a/b/c $ mv /mnt/btrfs/d /mnt/btrfs/a/b/c $ btrfs subvolume snapshot -r /mnt/btrfs /mnt/btrfs/snap1 $ btrfs send /mnt/btrfs/snap1 -f /tmp/base.send $ mv /mnt/btrfs/a/b/c /mnt/btrfs/a/b/x $ mv /mnt/btrfs/a/b/x/d /mnt/btrfs/a/b/x/y $ btrfs subvolume snapshot -r /mnt/btrfs /mnt/btrfs/snap2 $ btrfs send -p /mnt/btrfs/snap1 /mnt/btrfs/snap2 -f /tmp/incremental.send$ umout /mnt/btrfs $ mkfs.btrfs -f /dev/sdb3 $ mount /dev/sdb3 /mnt/btrfs $ btrfs receive /mnt/btrfs -f /tmp/base.send $ btrfs receive /mnt/btrfs -f /tmp/incremental.sendThe second btrfs receive command failed with:
"ERROR: rename a/b/c/d -> a/b/x/y failed. No such file or directory"A test case for xfstests follows.
Signed-off-by Filipe David Borba Manana <[email protected]>--- fs/btrfs/send.c | 41 ++++++++++++++++++++++++++++++++++------- 1 file changed, 34 insertions(+), 7 deletions(-)fs/btrfs/send.c +34
-8@@ -2847,19 +2847,48 @@ static int apply_dir_move(struct send_ctx *sctx, struct pending_dir_move *pm) {struct fs_path *from_path =NULL ;struct fs_path *to_path =NULL ; +struct fs_path *name =NULL ;u64 orig_progress = sctx->send_progress;struct recorded_ref *cur; +u64 parent_ino, parent_gen;int ret; + name =fs_path_alloc (); from_path =fs_path_alloc (); -if (!from_path) -return -ENOMEM ; +if (!name || !from_path) { + ret = -ENOMEM ; + goto out; + } - sctx->send_progress = pm->ino; - ret =get_cur_path (sctx, pm->ino, pm->gen, from_path); + ret =del_waiting_dir_move (sctx, pm->ino); +ASSERT (ret ==0 ); + + ret =get_first_ref (sctx->parent_root, pm->ino, + &parent_ino, &parent_gen, name);if (ret <0 ) goto out; +if (parent_ino == sctx->cur_ino) { +/* child only renamed, not moved */ +ASSERT (parent_gen == sctx->cur_inode_gen); + ret =get_cur_path (sctx, sctx->cur_ino, sctx->cur_inode_gen, + from_path); +if (ret <0 ) + goto out; + ret =fs_path_add_path (from_path, name); +if (ret <0 ) + goto out; + }else { +/* child moved and maybe renamed too */ + sctx->send_progress = pm->ino; + ret =get_cur_path (sctx, pm->ino, pm->gen, from_path); +if (ret <0 ) + goto out; + } + +fs_path_free (name); + name =NULL ; + to_path =fs_path_alloc ();if (!to_path) { ret = -ENOMEM ;@@ -2867,9 +2896,6 @@ static int apply_dir_move(struct send_ctx *sctx, struct pending_dir_move *pm) } sctx->send_progress = sctx->cur_ino +1 ; - ret =del_waiting_dir_move (sctx, pm->ino); -ASSERT (ret ==0 ); - ret =get_cur_path (sctx, pm->ino, pm->gen, to_path);if (ret <0 ) goto out;@@ -2893,6 +2919,7 @@ static int apply_dir_move(struct send_ctx *sctx, struct pending_dir_move *pm) } out: +fs_path_free (name);fs_path_free (from_path);fs_path_free (to_path); sctx->send_progress = orig_progress; --1.7.9.5
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
Signature · 5 lines
Filipe David Manana, "Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men."
Re: How to recover from failing btrfs send | btrfs receive?
Quoted 6 lines · Filipe David Manana
Quoted 2 lines · Marc MERLIN
Hi Fillipe, I see you have another fix for btrfs send (attached below), as ell as your other patch on Jan 21st (neither are in my 3.12.7).
Hi Marc,
Some replies below inlined.
The proper way to answer Email, thank you :)
Quoted 8 lines
Quoted 4 lines
From my error below,
Quoted 1 line
ERROR: rmdir o1845158-142-0 failed. No such file or directory
can you tell if I'm having a different problem than the two patches you sent?
I think it's a different problem.
The issues I have been fixing are about child directories with lower inode numbers then their parents and renaming/moving them.
Got it, thanks for letting me know.
Quoted 6 lines
Quoted 4 lines
More generally, could you hint how I can tell what this
rmdir o1845158-142-0
refers to, considering that it looks like a made up filename by btrfs send/receive?Those ox-y-z names are for orphan inodes, and generated by btrfs send yes.
Good to know, thanks.
Quoted 2 lines
Since you don't seem to know the sequence of steps (filesystem ops) that lead to that issue, perhaps you can run 'tree -a --inodes'
Nope, only had it once and it was on my home directory, which gets a lot of changes. If I had a way to know which filename o1845158-142-0 refers to, it would help :)
Ahh, I see that I do, grepped for inode 1818495 in tree -adf --inodes | less
You don't want me to send the output, it's megabytes worth, but here's what changed:
Before:
/mnt/btrfs_pool1/home_ro.20140209_12:00:01/merlin:
├── [1845158] ./gg-src/chromiumos/chroot/usr/share/zoneinfo/posix/America │ ├── [1845159] ./gg-src/chromiumos/chroot/usr/share/zoneinfo/posix/America/Argentina │ ├── [1845172] ./gg-src/chromiumos/chroot/usr/share/zoneinfo/posix/America/Indiana │ ├── [1845181] ./gg-src/chromiumos/chroot/usr/share/zoneinfo/posix/America/Kentucky │ └── [1845184] ./gg-src/chromiumos/chroot/usr/share/zoneinfo/posix/America/North_Dakota
In my new subvolume, I have this instead (directory is indeed gone, there are other 2 with the same name/hierachy, maybe even the same contents):
After:/mnt/btrfs_pool1/home/merlin:
legolas:~/gg-src$ tree -adf --inodes | grep zoneinfo/posix/America
│ ├── [4172524] ./ProdNG/PNG_test/chroots/borg/usr/share/zoneinfo/posix/America
│ │ ├── [4172733] ./ProdNG/PNG_test/chroots/borg/usr/share/zoneinfo/posix/America/Argentina
│ │ ├── [4172734] ./ProdNG/PNG_test/chroots/borg/usr/share/zoneinfo/posix/America/Indiana
│ │ ├── [4172735] ./ProdNG/PNG_test/chroots/borg/usr/share/zoneinfo/posix/America/Kentucky
│ │ └── [4172736] ./ProdNG/PNG_test/chroots/borg/usr/share/zoneinfo/posix/America/North_Dakota
├── [4188826] ./ProdNG/PNG_test/usr/share/zoneinfo/posix/America
│ ├── [4189034] ./ProdNG/PNG_test/usr/share/zoneinfo/posix/America/Argentina
│ ├── [4189035] ./ProdNG/PNG_test/usr/share/zoneinfo/posix/America/Indiana
│ ├── [4189036] ./ProdNG/PNG_test/usr/share/zoneinfo/posix/America/Kentucky
│ └── [4189037] ./ProdNG/PNG_test/usr/share/zoneinfo/posix/America/North_DakotaQuoted 3 lines
against the root of the subvolume and lets us know what the full output is. That might help in case it's one more case similar to those I have been fixing recently.
Does this help?
I still have the snapshot send/received failed at and the current volume, so it's not too hard for me to give you other diffs like that, but the full output would be very very large and not appropriate for posting here :)
Thanks,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Re: How to recover from failing btrfs send | btrfs receive?
Quoted 65 lines · Marc MERLIN
Quoted 6 lines · Filipe David Manana
Quoted 2 lines · Marc MERLIN
Hi Fillipe, I see you have another fix for btrfs send (attached below), as ell as your other patch on Jan 21st (neither are in my 3.12.7).
Hi Marc,
Some replies below inlined.The proper way to answer Email, thank you :)
Quoted 8 lines
Quoted 4 lines
From my error below,
Quoted 1 line
ERROR: rmdir o1845158-142-0 failed. No such file or directory
can you tell if I'm having a different problem than the two patches you sent?
I think it's a different problem.
The issues I have been fixing are about child directories with lower inode numbers then their parents and renaming/moving them.Got it, thanks for letting me know.
Quoted 6 lines
Quoted 4 lines
More generally, could you hint how I can tell what this
rmdir o1845158-142-0
refers to, considering that it looks like a made up filename by btrfs send/receive?Those ox-y-z names are for orphan inodes, and generated by btrfs send yes.
Good to know, thanks.
Quoted 2 lines
Since you don't seem to know the sequence of steps (filesystem ops) that lead to that issue, perhaps you can run 'tree -a --inodes'
Nope, only had it once and it was on my home directory, which gets a lot of changes. If I had a way to know which filename o1845158-142-0 refers to, it would help :)
Ahh, I see that I do, grepped for inode 1818495 in tree -adf --inodes | less
You don't want me to send the output, it's megabytes worth, but here's what changed:
Before:
/mnt/btrfs_pool1/home_ro.20140209_12:00:01/merlin:├── [1845158] ./gg-src/chromiumos/chroot/usr/share/zoneinfo/posix/America │ ├── [1845159] ./gg-src/chromiumos/chroot/usr/share/zoneinfo/posix/America/Argentina │ ├── [1845172] ./gg-src/chromiumos/chroot/usr/share/zoneinfo/posix/America/Indiana │ ├── [1845181] ./gg-src/chromiumos/chroot/usr/share/zoneinfo/posix/America/Kentucky │ └── [1845184] ./gg-src/chromiumos/chroot/usr/share/zoneinfo/posix/America/North_DakotaIn my new subvolume, I have this instead (directory is indeed gone, there are other 2 with the same name/hierachy, maybe even the same contents):
After:
/mnt/btrfs_pool1/home/merlin:legolas:~/gg-src$ tree -adf --inodes | grep zoneinfo/posix/America │ ├── [4172524] ./ProdNG/PNG_test/chroots/borg/usr/share/zoneinfo/posix/America │ │ ├── [4172733] ./ProdNG/PNG_test/chroots/borg/usr/share/zoneinfo/posix/America/Argentina │ │ ├── [4172734] ./ProdNG/PNG_test/chroots/borg/usr/share/zoneinfo/posix/America/Indiana │ │ ├── [4172735] ./ProdNG/PNG_test/chroots/borg/usr/share/zoneinfo/posix/America/Kentucky │ │ └── [4172736] ./ProdNG/PNG_test/chroots/borg/usr/share/zoneinfo/posix/America/North_Dakota ├── [4188826] ./ProdNG/PNG_test/usr/share/zoneinfo/posix/America │ ├── [4189034] ./ProdNG/PNG_test/usr/share/zoneinfo/posix/America/Argentina │ ├── [4189035] ./ProdNG/PNG_test/usr/share/zoneinfo/posix/America/Indiana │ ├── [4189036] ./ProdNG/PNG_test/usr/share/zoneinfo/posix/America/Kentucky │ └── [4189037] ./ProdNG/PNG_test/usr/share/zoneinfo/posix/America/North_Dakota
Quoted 5 lines
Quoted 3 lines
against the root of the subvolume and lets us know what the full output is. That might help in case it's one more case similar to those I have been fixing recently.
Does this help?
Well, not much.
In the meanwhile I've found a way to reproduce the same issue, i.e. send sending an rmdir operation for an orphan path that doesn't exist anymore, which involves multiple hardlinks against same file in a directory and deleting the directory. See the patch I just sent to the list titled: "[PATCH] Btrfs: send, don't send rmdir for same target multiple times".
Maybe your issue is different, triggered under different conditions and with different steps (no multiple hardlinks in the deleted directory).
I'll see if I come up with other ways of getting into that issue.
thanks
Quoted 12 lines
I still have the snapshot send/received failed at and the current volume, so it's not too hard for me to give you other diffs like that, but the full output would be very very large and not appropriate for posting here :)
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/
Signature · 5 lines
Filipe David Manana, "Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men."
Re: How to recover from failing btrfs send | btrfs receive?
Quoted 1 line · Filipe David Manana
I'll see if I come up with other ways of getting into that issue.
If you're collecting them, I found another bug, although it might not matter to most: if I put my laptop in S3 sleep during a send/receive, it reliably breaks the copy (this is disk to disk, not disk to network).
Not a problem for a server, but on a laptop, if you happen to have a background backup from disk1 to disk2 and you put the laptop to sleep, it will break the backup in a way that's not recoverable and you need to start back up from scratch.
btrfs send | btrfs receive gives:
Create a readonly snapshot of 'home' in './home_ro.20140216_21:03:53' At subvol home_ro.20140216_21:03:53
At subvol home_ro.20140216_21:03:53
ERROR: crc32 mismatch in command.
Error line 137 with status 234
If that helps, I can reproduce at will.
Thanks,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901btrfs send ioctl failed with -25: Inappropriate ioctl for device
Quoted 17 lines · Marc MERLIN
Quoted 1 line · Filipe David Manana
I'll see if I come up with other ways of getting into that issue.
If you're collecting them, I found another bug, although it might not matter to most: if I put my laptop in S3 sleep during a send/receive, it reliably breaks the copy (this is disk to disk, not disk to network).
Not a problem for a server, but on a laptop, if you happen to have a background backup from disk1 to disk2 and you put the laptop to sleep, it will break the backup in a way that's not recoverable and you need to start back up from scratch.
btrfs send | btrfs receive gives:
Create a readonly snapshot of 'home' in './home_ro.20140216_21:03:53' At subvol home_ro.20140216_21:03:53
At subvol home_ro.20140216_21:03:53
Quoted 4 lines
ERROR: crc32 mismatch in command.
Error line 137 with status 234If that helps, I can reproduce at will.
Mmmh, I may have found another problem.
I tried to init a new backup like so:
btrfs send "$src_newsnap" | btrfs receive "$dest_pool/"
And get:
+ btrfs send media_ro.20140222_11:12:53
+ btrfs receive /mnt/btrfs_bigbackup/
At subvol media_ro.20140222_11:12:53
ERROR: send ioctl failed with -25: Inappropriate ioctl for device
Or more simply:
gargamel:/mnt/btrfs_pool1# btrfs send media_ro.20140222_11:12:53 | less
At subvol media_ro.20140222_11:12:53
ERROR: send ioctl failed with -25: Inappropriate ioctl for device
gargamel:/mnt/btrfs_pool1# btrfs subvolume show media_ro.20140222_11:12:53
/mnt/btrfs_pool1/media_ro.20140222_11:12:53
Name: media_ro.20140222_11:12:53
uuid: 7e36a8a4-3aa2-7b4d-acbb-1ac79c52ae19
Parent uuid: 0e4dff73-1b21-0344-a7f1-79002d0eaa94
Creation time: 2014-02-22 11:12:56
Object ID: 8095
Generation (Gen): 6512
Gen at creation: 6512
Parent: 5
Top Level: 5
Flags: readonly
Snapshot(s):
gargamel:/mnt/btrfs_bigbackup# uname -r
3.13.3-amd64-i915-preempt-20140216
gargamel:/mnt/btrfs_bigbackup# btrfs --version
Btrfs v3.12
gargamel:/mnt/btrfs_bigbackup# btrfs fi show
Label: btrfs_boot uuid: e4c1daa8-9c39-4a59-b0a9-86297d397f3b
Total devices 1 FS bytes used 54.44GiB
devid 1 size 79.93GiB used 60.04GiB path /dev/mapper/cryptrootLabel: varlocalspace uuid: 9f46dbe2-1344-44c3-b0fb-af2888c34f18
Total devices 1 FS bytes used 464.76GiB
devid 1 size 1.63TiB used 471.04GiB path /dev/mapper/cryptraid0
Label: btrfs_pool1 uuid: 6358304a-2234-4243-b02d-4944c9af47d7
Total devices 1 FS bytes used 7.45TiB
devid 1 size 14.55TiB used 7.50TiB path /dev/mapper/dshelf1
Label: btrfs_pool2 uuid: cb9df6d3-a528-4afc-9a45-4fed5ec358d6
Total devices 1 FS bytes used 2.74TiB
devid 1 size 7.28TiB used 2.81TiB path /dev/mapper/dshelf2
Label: bigbackup uuid: 024ba4d0-dacb-438d-9f1b-eeb34083fe49
Total devices 5 FS bytes used 1.62MiB
devid 1 size 1.82TiB used 1.02GiB path /dev/dm-9
devid 2 size 1.82TiB used 2.00GiB path /dev/dm-6
devid 3 size 1.82TiB used 2.00GiB path /dev/dm-5
devid 4 size 1.82TiB used 1.01GiB path /dev/dm-7
devid 5 size 1.82TiB used 1.01GiB path /dev/mapper/crypt_sdq1Btrfs v3.12
Any ideas why btrfs send fails?
Thanks,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901