Skip to contentSkip to conversation

How to recover from failing btrffs send | btrfs receive?

PATCH In review Marc MERLIN 2 participants 9 messages Feb 12, 2014 – Feb 22, 2014 Btrfs file system

lore
Share & tools
lore
#0 lore raw

So, I've veen running this for a few weeks, and soon should have something half decent to share for others to use.

Unfortunately, one of my backups is now failing like so:

btrfs send -p "$src_snap" "$src_newsnap" | btrfs receive "$dest_pool/"
+ btrfs send -p /mnt/btrfs_pool1/home_ro.20140209_12:00:01 home_ro.20140212_05:37:49
+ btrfs receive /mnt/btrfs_pool2//
At subvol home_ro.20140212_05:37:49
At snapshot home_ro.20140212_05:37:49
ERROR: rmdir o1845158-142-0 failed. No such file or directory

This looks like it got in an unfinished state it can't recover from.

This was with kernel 3.12.7.

Can I self fix this somehow, I know I can use rsync to make both sides the sames, but incremental send/receive will not work anymore after that, correct?

Except, not really. Now I'm confused.

legolas:/mnt/btrfs_pool1# rsync -avSH --delete --dry-run  /mnt/btrfs_pool1/home_ro.20140209_12:00:01/. /mnt/btrfs_pool2/home_ro.20140209_12\:00\:01/.
sending incremental file list
./

sent 116867233 bytes received 265427 bytes 92194.14 bytes/sec total size is 129135042766 speedup is 1102.47 (DRY RUN) legolas:/mnt/btrfs_pool1#

(I know I start the entire backup over from scratch, but for obvious reasons, restarting an entire backup from scratch each time I get an error isn't great since it could take hours or days to backup that much data)

Suggestions welcome :)

Thanks,
Marc

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
#1 lore raw

Ok, let me try something else :)

Of those who are using btrfs send/receive, has anyone gotten in a state where incrementals will not apply anymore?

Thanks,
Marc

Quoted 48 lines · Marc MERLIN

So, I've veen running this for a few weeks, and soon should have something half decent to share for others to use.

Unfortunately, one of my backups is now failing like so:

btrfs send -p "$src_snap" "$src_newsnap" | btrfs receive "$dest_pool/"
+ btrfs send -p /mnt/btrfs_pool1/home_ro.20140209_12:00:01 home_ro.20140212_05:37:49
+ btrfs receive /mnt/btrfs_pool2//
At subvol home_ro.20140212_05:37:49
At snapshot home_ro.20140212_05:37:49
ERROR: rmdir o1845158-142-0 failed. No such file or directory

This looks like it got in an unfinished state it can't recover from.

This was with kernel 3.12.7.

Can I self fix this somehow, I know I can use rsync to make both sides the sames, but incremental send/receive will not work anymore after that, correct?

Except, not really. Now I'm confused.

legolas:/mnt/btrfs_pool1# rsync -avSH --delete --dry-run  /mnt/btrfs_pool1/home_ro.20140209_12:00:01/. /mnt/btrfs_pool2/home_ro.20140209_12\:00\:01/.
sending incremental file list
./

sent 116867233 bytes received 265427 bytes 92194.14 bytes/sec total size is 129135042766 speedup is 1102.47 (DRY RUN) legolas:/mnt/btrfs_pool1#

(I know I start the entire backup over from scratch, but for obvious reasons, restarting an entire backup from scratch each time I get an error isn't great since it could take hours or days to backup that much data)

Suggestions welcome :)

Thanks,
Marc

Signature · 8 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
#2 lore raw

PATCH 1/1 Btrfs: incremental send, fix invalid path after dir rename

This fixes yet one more case not caught by the commit titled:

Btrfs: fix infinite path build loops in incremental send

In this case, even before the initial full send, we have a directory which is a child of a directory with a higher inode number. Then we perform the initial send, and after we rename both the child and the parent, without moving them around. After doing these 2 renames, an incremental send sent a rename instruction for the child directory which contained an invalid "from" path (referenced the parent's old name, not the new one), which made the btrfs receive command fail.

Steps to reproduce:

    $ mkfs.btrfs -f /dev/sdb3
    $ mount /dev/sdb3 /mnt/btrfs
    $ mkdir -p /mnt/btrfs/a/b
    $ mkdir /mnt/btrfs/d
    $ mkdir /mnt/btrfs/a/b/c
    $ mv /mnt/btrfs/d /mnt/btrfs/a/b/c
    $ btrfs subvolume snapshot -r /mnt/btrfs /mnt/btrfs/snap1
    $ btrfs send /mnt/btrfs/snap1 -f /tmp/base.send
    $ mv /mnt/btrfs/a/b/c /mnt/btrfs/a/b/x
    $ mv /mnt/btrfs/a/b/x/d /mnt/btrfs/a/b/x/y
    $ btrfs subvolume snapshot -r /mnt/btrfs /mnt/btrfs/snap2
    $ btrfs send -p /mnt/btrfs/snap1 /mnt/btrfs/snap2 -f /tmp/incremental.send
    $ umout /mnt/btrfs
    $ mkfs.btrfs -f /dev/sdb3
    $ mount /dev/sdb3 /mnt/btrfs
    $ btrfs receive /mnt/btrfs -f /tmp/base.send
    $ btrfs receive /mnt/btrfs -f /tmp/incremental.send

The second btrfs receive command failed with:
"ERROR: rename a/b/c/d -> a/b/x/y failed. No such file or directory"

A test case for xfstests follows.

---
 fs/btrfs/send.c |   41 ++++++++++++++++++++++++++++++++++-------
 1 file changed, 34 insertions(+), 7 deletions(-)
fs/btrfs/send.c +34 -7
index b6d416c..7dbed58 100644
@@ -2847,19 +2847,48 @@ static int apply_dir_move(struct send_ctx *sctx, struct pending_dir_move *pm)
 {
 	struct fs_path *from_path = NULL;
 	struct fs_path *to_path = NULL;
+	struct fs_path *name = NULL;
 	u64 orig_progress = sctx->send_progress;
 	struct recorded_ref *cur;
+	u64 parent_ino, parent_gen;
 	int ret;
 
+	name = fs_path_alloc();
 	from_path = fs_path_alloc();
-	if (!from_path)
-		return -ENOMEM;
+	if (!name || !from_path) {
+		ret = -ENOMEM;
+		goto out;
+	}
 
-	sctx->send_progress = pm->ino;
-	ret = get_cur_path(sctx, pm->ino, pm->gen, from_path);
+	ret = del_waiting_dir_move(sctx, pm->ino);
+	ASSERT(ret == 0);
+
+	ret = get_first_ref(sctx->parent_root, pm->ino,
+			    &parent_ino, &parent_gen, name);
 	if (ret < 0)
 		goto out;
 
+	if (parent_ino == sctx->cur_ino) {
+		/* child only renamed, not moved */
+		ASSERT(parent_gen == sctx->cur_inode_gen);
+		ret = get_cur_path(sctx, sctx->cur_ino, sctx->cur_inode_gen,
+				   from_path);
+		if (ret < 0)
+			goto out;
+		ret = fs_path_add_path(from_path, name);
+		if (ret < 0)
+			goto out;
+	} else {
+		/* child moved and maybe renamed too */
+		sctx->send_progress = pm->ino;
+		ret = get_cur_path(sctx, pm->ino, pm->gen, from_path);
+		if (ret < 0)
+			goto out;
+	}
+
+	fs_path_free(name);
+	name = NULL;
+
 	to_path = fs_path_alloc();
 	if (!to_path) {
 		ret = -ENOMEM;
@@ -2867,9 +2896,6 @@ static int apply_dir_move(struct send_ctx *sctx, struct pending_dir_move *pm)
 	}
 
 	sctx->send_progress = sctx->cur_ino + 1;
-	ret = del_waiting_dir_move(sctx, pm->ino);
-	ASSERT(ret == 0);
-
 	ret = get_cur_path(sctx, pm->ino, pm->gen, to_path);
 	if (ret < 0)
 		goto out;
@@ -2893,6 +2919,7 @@ static int apply_dir_move(struct send_ctx *sctx, struct pending_dir_move *pm)
 	}
 
 out:
+	fs_path_free(name);
 	fs_path_free(from_path);
 	fs_path_free(to_path);
 	sctx->send_progress = orig_progress;
Signature · 1 line
1.7.9.5
#3 lore raw

Re: How to recover from failing btrfs send | btrfs receive?

Hi Fillipe, I see you have another fix for btrfs send (attached below), as ell as your other patch on Jan 21st (neither are in my 3.12.7).

Quoted 2 lines

From my error below,
ERROR: rmdir o1845158-142-0 failed. No such file or directory

can you tell if I'm having a different problem than the two patches you sent?

More generally, could you hint how I can tell what this
rmdir o1845158-142-0
refers to, considering that it looks like a made up filename by btrfs send/receive?

I'm happy to go to 3.13.3 and apply both patches if you think it'll help.

Thanks,
Marc

Quoted 38 lines · Marc MERLIN

So, I've veen running this for a few weeks, and soon should have something half decent to share for others to use.

Unfortunately, one of my backups is now failing like so:

btrfs send -p "$src_snap" "$src_newsnap" | btrfs receive "$dest_pool/"
+ btrfs send -p /mnt/btrfs_pool1/home_ro.20140209_12:00:01 home_ro.20140212_05:37:49
+ btrfs receive /mnt/btrfs_pool2//
At subvol home_ro.20140212_05:37:49
At snapshot home_ro.20140212_05:37:49
ERROR: rmdir o1845158-142-0 failed. No such file or directory

This looks like it got in an unfinished state it can't recover from.

This was with kernel 3.12.7.

Can I self fix this somehow, I know I can use rsync to make both sides the sames, but incremental send/receive will not work anymore after that, correct?

Except, not really. Now I'm confused.

legolas:/mnt/btrfs_pool1# rsync -avSH --delete --dry-run  /mnt/btrfs_pool1/home_ro.20140209_12:00:01/. /mnt/btrfs_pool2/home_ro.20140209_12\:00\:01/.
sending incremental file list
./

sent 116867233 bytes received 265427 bytes 92194.14 bytes/sec total size is 129135042766 speedup is 1102.47 (DRY RUN) legolas:/mnt/btrfs_pool1#

(I know I start the entire backup over from scratch, but for obvious reasons, restarting an entire backup from scratch each time I get an error isn't great since it could take hours or days to backup that much data)

Suggestions welcome :)

Thanks,
Marc

Quoted 126 lines · Filipe David Borba Manana

This fixes yet one more case not caught by the commit titled:

Btrfs: fix infinite path build loops in incremental send

In this case, even before the initial full send, we have a directory which is a child of a directory with a higher inode number. Then we perform the initial send, and after we rename both the child and the parent, without moving them around. After doing these 2 renames, an incremental send sent a rename instruction for the child directory which contained an invalid "from" path (referenced the parent's old name, not the new one), which made the btrfs receive command fail.

Steps to reproduce:

    $ mkfs.btrfs -f /dev/sdb3
    $ mount /dev/sdb3 /mnt/btrfs
    $ mkdir -p /mnt/btrfs/a/b
    $ mkdir /mnt/btrfs/d
    $ mkdir /mnt/btrfs/a/b/c
    $ mv /mnt/btrfs/d /mnt/btrfs/a/b/c
    $ btrfs subvolume snapshot -r /mnt/btrfs /mnt/btrfs/snap1
    $ btrfs send /mnt/btrfs/snap1 -f /tmp/base.send
    $ mv /mnt/btrfs/a/b/c /mnt/btrfs/a/b/x
    $ mv /mnt/btrfs/a/b/x/d /mnt/btrfs/a/b/x/y
    $ btrfs subvolume snapshot -r /mnt/btrfs /mnt/btrfs/snap2
    $ btrfs send -p /mnt/btrfs/snap1 /mnt/btrfs/snap2 -f /tmp/incremental.send
    $ umout /mnt/btrfs
    $ mkfs.btrfs -f /dev/sdb3
    $ mount /dev/sdb3 /mnt/btrfs
    $ btrfs receive /mnt/btrfs -f /tmp/base.send
    $ btrfs receive /mnt/btrfs -f /tmp/incremental.send

The second btrfs receive command failed with:
"ERROR: rename a/b/c/d -> a/b/x/y failed. No such file or directory"

A test case for xfstests follows.

---
 fs/btrfs/send.c |   41 ++++++++++++++++++++++++++++++++++-------
 1 file changed, 34 insertions(+), 7 deletions(-)
fs/btrfs/send.c +34 -7
index b6d416c..7dbed58 100644
@@ -2847,19 +2847,48 @@ static int apply_dir_move(struct send_ctx *sctx, struct pending_dir_move *pm)
 {
 	struct fs_path *from_path = NULL;
 	struct fs_path *to_path = NULL;
+	struct fs_path *name = NULL;
 	u64 orig_progress = sctx->send_progress;
 	struct recorded_ref *cur;
+	u64 parent_ino, parent_gen;
 	int ret;
 
+	name = fs_path_alloc();
 	from_path = fs_path_alloc();
-	if (!from_path)
-		return -ENOMEM;
+	if (!name || !from_path) {
+		ret = -ENOMEM;
+		goto out;
+	}
 
-	sctx->send_progress = pm->ino;
-	ret = get_cur_path(sctx, pm->ino, pm->gen, from_path);
+	ret = del_waiting_dir_move(sctx, pm->ino);
+	ASSERT(ret == 0);
+
+	ret = get_first_ref(sctx->parent_root, pm->ino,
+			    &parent_ino, &parent_gen, name);
 	if (ret < 0)
 		goto out;
 
+	if (parent_ino == sctx->cur_ino) {
+		/* child only renamed, not moved */
+		ASSERT(parent_gen == sctx->cur_inode_gen);
+		ret = get_cur_path(sctx, sctx->cur_ino, sctx->cur_inode_gen,
+				   from_path);
+		if (ret < 0)
+			goto out;
+		ret = fs_path_add_path(from_path, name);
+		if (ret < 0)
+			goto out;
+	} else {
+		/* child moved and maybe renamed too */
+		sctx->send_progress = pm->ino;
+		ret = get_cur_path(sctx, pm->ino, pm->gen, from_path);
+		if (ret < 0)
+			goto out;
+	}
+
+	fs_path_free(name);
+	name = NULL;
+
 	to_path = fs_path_alloc();
 	if (!to_path) {
 		ret = -ENOMEM;
@@ -2867,9 +2896,6 @@ static int apply_dir_move(struct send_ctx *sctx, struct pending_dir_move *pm)
 	}
 
 	sctx->send_progress = sctx->cur_ino + 1;
-	ret = del_waiting_dir_move(sctx, pm->ino);
-	ASSERT(ret == 0);
-
 	ret = get_cur_path(sctx, pm->ino, pm->gen, to_path);
 	if (ret < 0)
 		goto out;
@@ -2893,6 +2919,7 @@ static int apply_dir_move(struct send_ctx *sctx, struct pending_dir_move *pm)
 	}
 
 out:
+	fs_path_free(name);
 	fs_path_free(from_path);
 	fs_path_free(to_path);
 	sctx->send_progress = orig_progress;
Signature · 6 lines
1.7.9.5

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
#4 lore raw

Re: How to recover from failing btrfs send | btrfs receive?

Quoted 2 lines · Marc MERLIN

Hi Fillipe, I see you have another fix for btrfs send (attached below), as ell as your other patch on Jan 21st (neither are in my 3.12.7).

Hi Marc,
Some replies below inlined.

Quoted 5 lines

From my error below,

Quoted 1 line

ERROR: rmdir o1845158-142-0 failed. No such file or directory

can you tell if I'm having a different problem than the two patches you sent?

I think it's a different problem.
The issues I have been fixing are about child directories with lower inode numbers then their parents and renaming/moving them.

Quoted 5 lines

More generally, could you hint how I can tell what this
rmdir o1845158-142-0
refers to, considering that it looks like a made up filename by btrfs send/receive?

Those ox-y-z names are for orphan inodes, and generated by btrfs send yes.

Since you don't seem to know the sequence of steps (filesystem ops) that lead to that issue, perhaps you can run 'tree -a --inodes' against the root of the subvolume and lets us know what the full output is. That might help in case it's one more case similar to those I have been fixing recently.

thanks

Quoted 181 lines

I'm happy to go to 3.13.3 and apply both patches if you think it'll help.

Thanks,
Marc

Quoted 38 lines · Marc MERLIN

So, I've veen running this for a few weeks, and soon should have something half decent to share for others to use.

Unfortunately, one of my backups is now failing like so:

btrfs send -p "$src_snap" "$src_newsnap" | btrfs receive "$dest_pool/"
+ btrfs send -p /mnt/btrfs_pool1/home_ro.20140209_12:00:01 home_ro.20140212_05:37:49
+ btrfs receive /mnt/btrfs_pool2//
At subvol home_ro.20140212_05:37:49
At snapshot home_ro.20140212_05:37:49
ERROR: rmdir o1845158-142-0 failed. No such file or directory

This looks like it got in an unfinished state it can't recover from.

This was with kernel 3.12.7.

Can I self fix this somehow, I know I can use rsync to make both sides the sames, but incremental send/receive will not work anymore after that, correct?

Except, not really. Now I'm confused.

legolas:/mnt/btrfs_pool1# rsync -avSH --delete --dry-run  /mnt/btrfs_pool1/home_ro.20140209_12:00:01/. /mnt/btrfs_pool2/home_ro.20140209_12\:00\:01/.
sending incremental file list
./

sent 116867233 bytes received 265427 bytes 92194.14 bytes/sec total size is 129135042766 speedup is 1102.47 (DRY RUN) legolas:/mnt/btrfs_pool1#

(I know I start the entire backup over from scratch, but for obvious reasons, restarting an entire backup from scratch each time I get an error isn't great since it could take hours or days to backup that much data)

Suggestions welcome :)

Thanks,
Marc

Quoted 121 lines · Filipe David Borba Manana

This fixes yet one more case not caught by the commit titled:

Btrfs: fix infinite path build loops in incremental send

In this case, even before the initial full send, we have a directory which is a child of a directory with a higher inode number. Then we perform the initial send, and after we rename both the child and the parent, without moving them around. After doing these 2 renames, an incremental send sent a rename instruction for the child directory which contained an invalid "from" path (referenced the parent's old name, not the new one), which made the btrfs receive command fail.

Steps to reproduce:

    $ mkfs.btrfs -f /dev/sdb3
    $ mount /dev/sdb3 /mnt/btrfs
    $ mkdir -p /mnt/btrfs/a/b
    $ mkdir /mnt/btrfs/d
    $ mkdir /mnt/btrfs/a/b/c
    $ mv /mnt/btrfs/d /mnt/btrfs/a/b/c
    $ btrfs subvolume snapshot -r /mnt/btrfs /mnt/btrfs/snap1
    $ btrfs send /mnt/btrfs/snap1 -f /tmp/base.send
    $ mv /mnt/btrfs/a/b/c /mnt/btrfs/a/b/x
    $ mv /mnt/btrfs/a/b/x/d /mnt/btrfs/a/b/x/y
    $ btrfs subvolume snapshot -r /mnt/btrfs /mnt/btrfs/snap2
    $ btrfs send -p /mnt/btrfs/snap1 /mnt/btrfs/snap2 -f /tmp/incremental.send
    $ umout /mnt/btrfs
    $ mkfs.btrfs -f /dev/sdb3
    $ mount /dev/sdb3 /mnt/btrfs
    $ btrfs receive /mnt/btrfs -f /tmp/base.send
    $ btrfs receive /mnt/btrfs -f /tmp/incremental.send

The second btrfs receive command failed with:
"ERROR: rename a/b/c/d -> a/b/x/y failed. No such file or directory"

A test case for xfstests follows.

---
 fs/btrfs/send.c |   41 ++++++++++++++++++++++++++++++++++-------
 1 file changed, 34 insertions(+), 7 deletions(-)
fs/btrfs/send.c +34 -8
index b6d416c..7dbed58 100644
@@ -2847,19 +2847,48 @@ static int apply_dir_move(struct send_ctx *sctx, struct pending_dir_move *pm)
 {
      struct fs_path *from_path = NULL;
      struct fs_path *to_path = NULL;
+     struct fs_path *name = NULL;
      u64 orig_progress = sctx->send_progress;
      struct recorded_ref *cur;
+     u64 parent_ino, parent_gen;
      int ret;
 
+     name = fs_path_alloc();
      from_path = fs_path_alloc();
-     if (!from_path)
-             return -ENOMEM;
+     if (!name || !from_path) {
+             ret = -ENOMEM;
+             goto out;
+     }
 
-     sctx->send_progress = pm->ino;
-     ret = get_cur_path(sctx, pm->ino, pm->gen, from_path);
+     ret = del_waiting_dir_move(sctx, pm->ino);
+     ASSERT(ret == 0);
+
+     ret = get_first_ref(sctx->parent_root, pm->ino,
+                         &parent_ino, &parent_gen, name);
      if (ret < 0)
              goto out;
 
+     if (parent_ino == sctx->cur_ino) {
+             /* child only renamed, not moved */
+             ASSERT(parent_gen == sctx->cur_inode_gen);
+             ret = get_cur_path(sctx, sctx->cur_ino, sctx->cur_inode_gen,
+                                from_path);
+             if (ret < 0)
+                     goto out;
+             ret = fs_path_add_path(from_path, name);
+             if (ret < 0)
+                     goto out;
+     } else {
+             /* child moved and maybe renamed too */
+             sctx->send_progress = pm->ino;
+             ret = get_cur_path(sctx, pm->ino, pm->gen, from_path);
+             if (ret < 0)
+                     goto out;
+     }
+
+     fs_path_free(name);
+     name = NULL;
+
      to_path = fs_path_alloc();
      if (!to_path) {
              ret = -ENOMEM;
@@ -2867,9 +2896,6 @@ static int apply_dir_move(struct send_ctx *sctx, struct pending_dir_move *pm)
      }
 
      sctx->send_progress = sctx->cur_ino + 1;
-     ret = del_waiting_dir_move(sctx, pm->ino);
-     ASSERT(ret == 0);
-
      ret = get_cur_path(sctx, pm->ino, pm->gen, to_path);
      if (ret < 0)
              goto out;
@@ -2893,6 +2919,7 @@ static int apply_dir_move(struct send_ctx *sctx, struct pending_dir_move *pm)
      }
 
 out:
+     fs_path_free(name);
      fs_path_free(from_path);
      fs_path_free(to_path);
      sctx->send_progress = orig_progress;
--

1.7.9.5

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901

Signature · 5 lines
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."
#5 lore raw

Re: How to recover from failing btrfs send | btrfs receive?

Quoted 6 lines · Filipe David Manana
Quoted 2 lines · Marc MERLIN

Hi Fillipe, I see you have another fix for btrfs send (attached below), as ell as your other patch on Jan 21st (neither are in my 3.12.7).

Hi Marc,
Some replies below inlined.

The proper way to answer Email, thank you :)

Quoted 8 lines
Quoted 4 lines

From my error below,

Quoted 1 line

ERROR: rmdir o1845158-142-0 failed. No such file or directory

can you tell if I'm having a different problem than the two patches you sent?

I think it's a different problem.
The issues I have been fixing are about child directories with lower inode numbers then their parents and renaming/moving them.

Got it, thanks for letting me know.

Quoted 6 lines
Quoted 4 lines

More generally, could you hint how I can tell what this
rmdir o1845158-142-0
refers to, considering that it looks like a made up filename by btrfs send/receive?

Those ox-y-z names are for orphan inodes, and generated by btrfs send yes.

Good to know, thanks.

Quoted 2 lines

Since you don't seem to know the sequence of steps (filesystem ops) that lead to that issue, perhaps you can run 'tree -a --inodes'

Nope, only had it once and it was on my home directory, which gets a lot of changes. If I had a way to know which filename o1845158-142-0 refers to, it would help :)

Ahh, I see that I do, grepped for inode 1818495 in tree -adf --inodes | less

You don't want me to send the output, it's megabytes worth, but here's what changed:

Before:
/mnt/btrfs_pool1/home_ro.20140209_12:00:01/merlin:

├── [1845158]  ./gg-src/chromiumos/chroot/usr/share/zoneinfo/posix/America
│   ├── [1845159]  ./gg-src/chromiumos/chroot/usr/share/zoneinfo/posix/America/Argentina
│   ├── [1845172]  ./gg-src/chromiumos/chroot/usr/share/zoneinfo/posix/America/Indiana
│   ├── [1845181]  ./gg-src/chromiumos/chroot/usr/share/zoneinfo/posix/America/Kentucky
│   └── [1845184]  ./gg-src/chromiumos/chroot/usr/share/zoneinfo/posix/America/North_Dakota

In my new subvolume, I have this instead (directory is indeed gone, there are other 2 with the same name/hierachy, maybe even the same contents):

After:
/mnt/btrfs_pool1/home/merlin:

legolas:~/gg-src$  tree -adf --inodes | grep zoneinfo/posix/America
    │   ├── [4172524]  ./ProdNG/PNG_test/chroots/borg/usr/share/zoneinfo/posix/America
    │   │   ├── [4172733]  ./ProdNG/PNG_test/chroots/borg/usr/share/zoneinfo/posix/America/Argentina
    │   │   ├── [4172734]  ./ProdNG/PNG_test/chroots/borg/usr/share/zoneinfo/posix/America/Indiana
    │   │   ├── [4172735]  ./ProdNG/PNG_test/chroots/borg/usr/share/zoneinfo/posix/America/Kentucky
    │   │   └── [4172736]  ./ProdNG/PNG_test/chroots/borg/usr/share/zoneinfo/posix/America/North_Dakota
├── [4188826]  ./ProdNG/PNG_test/usr/share/zoneinfo/posix/America
│   ├── [4189034]  ./ProdNG/PNG_test/usr/share/zoneinfo/posix/America/Argentina
│   ├── [4189035]  ./ProdNG/PNG_test/usr/share/zoneinfo/posix/America/Indiana
│   ├── [4189036]  ./ProdNG/PNG_test/usr/share/zoneinfo/posix/America/Kentucky
│   └── [4189037]  ./ProdNG/PNG_test/usr/share/zoneinfo/posix/America/North_Dakota
Quoted 3 lines

against the root of the subvolume and lets us know what the full output is. That might help in case it's one more case similar to those I have been fixing recently.

Does this help?

I still have the snapshot send/received failed at and the current volume, so it's not too hard for me to give you other diffs like that, but the full output would be very very large and not appropriate for posting here :)

Thanks,
Marc

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  
#6 lore raw

Re: How to recover from failing btrfs send | btrfs receive?

Quoted 65 lines · Marc MERLIN
Quoted 6 lines · Filipe David Manana
Quoted 2 lines · Marc MERLIN

Hi Fillipe, I see you have another fix for btrfs send (attached below), as ell as your other patch on Jan 21st (neither are in my 3.12.7).

Hi Marc,
Some replies below inlined.

The proper way to answer Email, thank you :)

Quoted 8 lines
Quoted 4 lines

From my error below,

Quoted 1 line

ERROR: rmdir o1845158-142-0 failed. No such file or directory

can you tell if I'm having a different problem than the two patches you sent?

I think it's a different problem.
The issues I have been fixing are about child directories with lower inode numbers then their parents and renaming/moving them.

Got it, thanks for letting me know.

Quoted 6 lines
Quoted 4 lines

More generally, could you hint how I can tell what this
rmdir o1845158-142-0
refers to, considering that it looks like a made up filename by btrfs send/receive?

Those ox-y-z names are for orphan inodes, and generated by btrfs send yes.

Good to know, thanks.

Quoted 2 lines

Since you don't seem to know the sequence of steps (filesystem ops) that lead to that issue, perhaps you can run 'tree -a --inodes'

Nope, only had it once and it was on my home directory, which gets a lot of changes. If I had a way to know which filename o1845158-142-0 refers to, it would help :)

Ahh, I see that I do, grepped for inode 1818495 in tree -adf --inodes | less

You don't want me to send the output, it's megabytes worth, but here's what changed:

Before:
/mnt/btrfs_pool1/home_ro.20140209_12:00:01/merlin:

├── [1845158]  ./gg-src/chromiumos/chroot/usr/share/zoneinfo/posix/America
│   ├── [1845159]  ./gg-src/chromiumos/chroot/usr/share/zoneinfo/posix/America/Argentina
│   ├── [1845172]  ./gg-src/chromiumos/chroot/usr/share/zoneinfo/posix/America/Indiana
│   ├── [1845181]  ./gg-src/chromiumos/chroot/usr/share/zoneinfo/posix/America/Kentucky
│   └── [1845184]  ./gg-src/chromiumos/chroot/usr/share/zoneinfo/posix/America/North_Dakota

In my new subvolume, I have this instead (directory is indeed gone, there are other 2 with the same name/hierachy, maybe even the same contents):

After:
/mnt/btrfs_pool1/home/merlin:

legolas:~/gg-src$  tree -adf --inodes | grep zoneinfo/posix/America
    │   ├── [4172524]  ./ProdNG/PNG_test/chroots/borg/usr/share/zoneinfo/posix/America
    │   │   ├── [4172733]  ./ProdNG/PNG_test/chroots/borg/usr/share/zoneinfo/posix/America/Argentina
    │   │   ├── [4172734]  ./ProdNG/PNG_test/chroots/borg/usr/share/zoneinfo/posix/America/Indiana
    │   │   ├── [4172735]  ./ProdNG/PNG_test/chroots/borg/usr/share/zoneinfo/posix/America/Kentucky
    │   │   └── [4172736]  ./ProdNG/PNG_test/chroots/borg/usr/share/zoneinfo/posix/America/North_Dakota
├── [4188826]  ./ProdNG/PNG_test/usr/share/zoneinfo/posix/America
│   ├── [4189034]  ./ProdNG/PNG_test/usr/share/zoneinfo/posix/America/Argentina
│   ├── [4189035]  ./ProdNG/PNG_test/usr/share/zoneinfo/posix/America/Indiana
│   ├── [4189036]  ./ProdNG/PNG_test/usr/share/zoneinfo/posix/America/Kentucky
│   └── [4189037]  ./ProdNG/PNG_test/usr/share/zoneinfo/posix/America/North_Dakota
Quoted 5 lines
Quoted 3 lines

against the root of the subvolume and lets us know what the full output is. That might help in case it's one more case similar to those I have been fixing recently.

Does this help?

Well, not much.
In the meanwhile I've found a way to reproduce the same issue, i.e. send sending an rmdir operation for an orphan path that doesn't exist anymore, which involves multiple hardlinks against same file in a directory and deleting the directory. See the patch I just sent to the list titled: "[PATCH] Btrfs: send, don't send rmdir for same target multiple times".

Maybe your issue is different, triggered under different conditions and with different steps (no multiple hardlinks in the deleted directory).

I'll see if I come up with other ways of getting into that issue.

thanks

Quoted 12 lines

I still have the snapshot send/received failed at and the current volume, so it's not too hard for me to give you other diffs like that, but the full output would be very very large and not appropriate for posting here :)

Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/

Signature · 5 lines
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."
#7 lore raw

Re: How to recover from failing btrfs send | btrfs receive?

Quoted 1 line · Filipe David Manana

I'll see if I come up with other ways of getting into that issue.

If you're collecting them, I found another bug, although it might not matter to most: if I put my laptop in S3 sleep during a send/receive, it reliably breaks the copy (this is disk to disk, not disk to network).

Not a problem for a server, but on a laptop, if you happen to have a background backup from disk1 to disk2 and you put the laptop to sleep, it will break the backup in a way that's not recoverable and you need to start back up from scratch.

btrfs send | btrfs receive gives:
Create a readonly snapshot of 'home' in './home_ro.20140216_21:03:53' At subvol home_ro.20140216_21:03:53
At subvol home_ro.20140216_21:03:53

ERROR: crc32 mismatch in command.
Error line 137 with status 234

If that helps, I can reproduce at will.

Thanks,
Marc

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
#8 lore raw

btrfs send ioctl failed with -25: Inappropriate ioctl for device

Quoted 17 lines · Marc MERLIN
Quoted 1 line · Filipe David Manana

I'll see if I come up with other ways of getting into that issue.

If you're collecting them, I found another bug, although it might not matter to most: if I put my laptop in S3 sleep during a send/receive, it reliably breaks the copy (this is disk to disk, not disk to network).

Not a problem for a server, but on a laptop, if you happen to have a background backup from disk1 to disk2 and you put the laptop to sleep, it will break the backup in a way that's not recoverable and you need to start back up from scratch.

btrfs send | btrfs receive gives:
Create a readonly snapshot of 'home' in './home_ro.20140216_21:03:53' At subvol home_ro.20140216_21:03:53
At subvol home_ro.20140216_21:03:53

Quoted 4 lines

ERROR: crc32 mismatch in command.
Error line 137 with status 234

If that helps, I can reproduce at will.

Mmmh, I may have found another problem.

I tried to init a new backup like so:

btrfs send "$src_newsnap" | btrfs receive "$dest_pool/"

And get:
+ btrfs send media_ro.20140222_11:12:53
+ btrfs receive /mnt/btrfs_bigbackup/
At subvol media_ro.20140222_11:12:53
ERROR: send ioctl failed with -25: Inappropriate ioctl for device

Or more simply:

gargamel:/mnt/btrfs_pool1# btrfs send media_ro.20140222_11:12:53 | less
At subvol media_ro.20140222_11:12:53
ERROR: send ioctl failed with -25: Inappropriate ioctl for device
gargamel:/mnt/btrfs_pool1# btrfs subvolume show media_ro.20140222_11:12:53
/mnt/btrfs_pool1/media_ro.20140222_11:12:53
        Name:                   media_ro.20140222_11:12:53
        uuid:                   7e36a8a4-3aa2-7b4d-acbb-1ac79c52ae19
        Parent uuid:            0e4dff73-1b21-0344-a7f1-79002d0eaa94
        Creation time:          2014-02-22 11:12:56
        Object ID:              8095
        Generation (Gen):       6512
        Gen at creation:        6512
        Parent:                 5
        Top Level:              5
        Flags:                  readonly
        Snapshot(s):
gargamel:/mnt/btrfs_bigbackup# uname -r
3.13.3-amd64-i915-preempt-20140216
gargamel:/mnt/btrfs_bigbackup# btrfs --version
Btrfs v3.12
gargamel:/mnt/btrfs_bigbackup# btrfs fi show
Label: btrfs_boot  uuid: e4c1daa8-9c39-4a59-b0a9-86297d397f3b
        Total devices 1 FS bytes used 54.44GiB
        devid    1 size 79.93GiB used 60.04GiB path /dev/mapper/cryptroot

Label: varlocalspace uuid: 9f46dbe2-1344-44c3-b0fb-af2888c34f18
Total devices 1 FS bytes used 464.76GiB
devid 1 size 1.63TiB used 471.04GiB path /dev/mapper/cryptraid0

Label: btrfs_pool1 uuid: 6358304a-2234-4243-b02d-4944c9af47d7
Total devices 1 FS bytes used 7.45TiB
devid 1 size 14.55TiB used 7.50TiB path /dev/mapper/dshelf1

Label: btrfs_pool2 uuid: cb9df6d3-a528-4afc-9a45-4fed5ec358d6
Total devices 1 FS bytes used 2.74TiB
devid 1 size 7.28TiB used 2.81TiB path /dev/mapper/dshelf2

Label: bigbackup uuid: 024ba4d0-dacb-438d-9f1b-eeb34083fe49

        Total devices 5 FS bytes used 1.62MiB
        devid    1 size 1.82TiB used 1.02GiB path /dev/dm-9
        devid    2 size 1.82TiB used 2.00GiB path /dev/dm-6
        devid    3 size 1.82TiB used 2.00GiB path /dev/dm-5
        devid    4 size 1.82TiB used 1.01GiB path /dev/dm-7
        devid    5 size 1.82TiB used 1.01GiB path /dev/mapper/crypt_sdq1

Btrfs v3.12

Any ideas why btrfs send fails?

Thanks,
Marc

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901