btrfs: could not do orphan cleanup -116

DISCUSSION Maciej Marcin Piechotka 2 participants 7 messages Nov 7, 2011 – Nov 9, 2011 Btrfs file system

lore

Share & tools

lore

Maciej Marcin Piechotka

author

2011-11-07 13:17 UTC #0 lore raw

Hello,

When I booted my machine (after clean powerdown) the following message appeared:

[   32.757913] device fsid XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX devid 1
transid 40864 /dev/mapper/XXXXX-XXXXX
[   32.758466] btrfs: use lzo compression
[   32.758475] btrfs: enabling disk space caching
[   32.758483] btrfs: enabling inode map caching
[   32.758490] btrfs: enabling auto defrag
[   32.758497] btrfs: thread pool 2
[   34.024359] btrfs: could not do orphan cleanup -116
[   63.173382] btrfs: open_ctree failed

btrfsck has shown no errors.

Afterwards when I tried to mount filesystem it have shown:

[  116.012528] btrfs: enabling inode map caching
[  116.012534] btrfs: enabling auto defrag
[  116.012542] btrfs: thread pool 2

It looked like everything is working but to be sure I've run scrub:

scrub status for fsid XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
scrub started at Mon Nov 7 13:05:50 2011 and finished after 606
seconds
total bytes scrubbed: 32.37GB with 0 errors

PS.

# find include | xargs grep 116
(...)
include/asm-generic/errno.h:#define     ESTALE          116     /* Stale
NFS file handle */
(...)

I don't use NFS - I use LVM on LUKS. In any case - it have been done concurrently with mounting of other fs which had no such problems.

Josef Bacik

2011-11-07 15:29 UTC #1 lore raw

Quoted 30 lines · Maciej Marcin Piechotka

Hello,
When I booted my machine (after clean powerdown) the following message appeared:
[   32.757913] device fsid XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX devid 1
transid 40864 /dev/mapper/XXXXX-XXXXX
[   32.758466] btrfs: use lzo compression
[   32.758475] btrfs: enabling disk space caching
[   32.758483] btrfs: enabling inode map caching
[   32.758490] btrfs: enabling auto defrag
[   32.758497] btrfs: thread pool 2
[   34.024359] btrfs: could not do orphan cleanup -116
[   63.173382] btrfs: open_ctree failed
btrfsck has shown no errors.
Afterwards when I tried to mount filesystem it have shown:
[  116.012528] btrfs: enabling inode map caching
[  116.012534] btrfs: enabling auto defrag
[  116.012542] btrfs: thread pool 2
It looked like everything is working but to be sure I've run scrub:
scrub status for fsid XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
scrub started at Mon Nov 7 13:05:50 2011 and finished after 606
seconds
total bytes scrubbed: 32.37GB with 0 errors
PS.

Quoted 8 lines

# find include | xargs grep 116
(...)
include/asm-generic/errno.h:#define     ESTALE          116     /* Stale
NFS file handle */
(...)
I don't use NFS - I use LVM on LUKS. In any case - it have been done concurrently with mounting of other fs which had no such problems.

Yeah that's a bug with an earlier kernel, if you upgrade to 3.1 it should go away. Thanks,

Josef

Maciej Marcin Piechotka

author

2011-11-07 18:41 UTC #2 lore raw

Quoted 37 lines · Josef Bacik

Quoted 36 lines · Maciej Marcin Piechotka
Hello,
When I booted my machine (after clean powerdown) the following message appeared:
[   32.757913] device fsid XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX devid 1
transid 40864 /dev/mapper/XXXXX-XXXXX
[   32.758466] btrfs: use lzo compression
[   32.758475] btrfs: enabling disk space caching
[   32.758483] btrfs: enabling inode map caching
[   32.758490] btrfs: enabling auto defrag
[   32.758497] btrfs: thread pool 2
[   34.024359] btrfs: could not do orphan cleanup -116
[   63.173382] btrfs: open_ctree failed
btrfsck has shown no errors.
Afterwards when I tried to mount filesystem it have shown:
[  116.012528] btrfs: enabling inode map caching
[  116.012534] btrfs: enabling auto defrag
[  116.012542] btrfs: thread pool 2
It looked like everything is working but to be sure I've run scrub:
scrub status for fsid XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
scrub started at Mon Nov 7 13:05:50 2011 and finished after 606
seconds
total bytes scrubbed: 32.37GB with 0 errors
PS.
# find include | xargs grep 116
(...)
include/asm-generic/errno.h:#define     ESTALE          116     /* Stale
NFS file handle */
(...)

Quoted 7 lines

Quoted 2 lines
I don't use NFS - I use LVM on LUKS. In any case - it have been done concurrently with mounting of other fs which had no such problems.
Yeah that's a bug with an earlier kernel, if you upgrade to 3.1 it should go away. Thanks,
Josef

I'm using 'newer' kernel then 3.1 - I'm following Linus' tree (this kernel is few days old).

Regards

Josef Bacik

2011-11-07 19:01 UTC #3 lore raw

Quoted 41 lines · Maciej Marcin Piechotka

Quoted 40 lines · Josef Bacik
Quoted 38 lines · Maciej Marcin Piechotka
Hello,
When I booted my machine (after clean powerdown) the following message appeared:
[   32.757913] device fsid XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX devid 1
transid 40864 /dev/mapper/XXXXX-XXXXX
[   32.758466] btrfs: use lzo compression
[   32.758475] btrfs: enabling disk space caching
[   32.758483] btrfs: enabling inode map caching
[   32.758490] btrfs: enabling auto defrag
[   32.758497] btrfs: thread pool 2
[   34.024359] btrfs: could not do orphan cleanup -116
[   63.173382] btrfs: open_ctree failed
btrfsck has shown no errors.
Afterwards when I tried to mount filesystem it have shown:
[  116.012528] btrfs: enabling inode map caching
[  116.012534] btrfs: enabling auto defrag
[  116.012542] btrfs: thread pool 2
It looked like everything is working but to be sure I've run scrub:
scrub status for fsid XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
scrub started at Mon Nov 7 13:05:50 2011 and finished after 606
seconds
total bytes scrubbed: 32.37GB with 0 errors
PS.
# find include | xargs grep 116
(...)
include/asm-generic/errno.h:#define     ESTALE          116     /* Stale
NFS file handle */
(...)
I don't use NFS - I use LVM on LUKS. In any case - it have been done concurrently with mounting of other fs which had no such problems.

Quoted 9 lines

Quoted 4 lines
Yeah that's a bug with an earlier kernel, if you upgrade to 3.1 it should go away. Thanks,
Josef
I'm using 'newer' kernel then 3.1 - I'm following Linus' tree (this kernel is few days old).

Do you have the commit

a8c9e5769718d47e87cce40c9b84cab421804797
Btrfs: fix orphan cleanup regression

If not then the kernel isn't new enough. Thanks,

Josef

Maciej Marcin Piechotka

author

2011-11-09 15:43 UTC #4 lore raw

Quoted 52 lines · Josef Bacik

Quoted 41 lines · Maciej Marcin Piechotka
Quoted 40 lines · Josef Bacik
Quoted 38 lines · Maciej Marcin Piechotka
Hello,
When I booted my machine (after clean powerdown) the following message appeared:
[   32.757913] device fsid XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX devid 1
transid 40864 /dev/mapper/XXXXX-XXXXX
[   32.758466] btrfs: use lzo compression
[   32.758475] btrfs: enabling disk space caching
[   32.758483] btrfs: enabling inode map caching
[   32.758490] btrfs: enabling auto defrag
[   32.758497] btrfs: thread pool 2
[   34.024359] btrfs: could not do orphan cleanup -116
[   63.173382] btrfs: open_ctree failed
btrfsck has shown no errors.
Afterwards when I tried to mount filesystem it have shown:
[  116.012528] btrfs: enabling inode map caching
[  116.012534] btrfs: enabling auto defrag
[  116.012542] btrfs: thread pool 2
It looked like everything is working but to be sure I've run scrub:
scrub status for fsid XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
scrub started at Mon Nov 7 13:05:50 2011 and finished after 606
seconds
total bytes scrubbed: 32.37GB with 0 errors
PS.
# find include | xargs grep 116
(...)
include/asm-generic/errno.h:#define     ESTALE          116     /* Stale
NFS file handle */
(...)
I don't use NFS - I use LVM on LUKS. In any case - it have been done concurrently with mounting of other fs which had no such problems.
Quoted 9 lines
Quoted 4 lines
Yeah that's a bug with an earlier kernel, if you upgrade to 3.1 it should go away. Thanks,
Josef
I'm using 'newer' kernel then 3.1 - I'm following Linus' tree (this kernel is few days old).

Quoted 8 lines

Do you have the commit
a8c9e5769718d47e87cce40c9b84cab421804797
Btrfs: fix orphan cleanup regression
If not then the kernel isn't new enough. Thanks,
Josef

It have been merged in master for 3.2-rc1 (shortly before tag).

Regards

Josef Bacik

2011-11-09 16:14 UTC #5 lore raw

Quoted 58 lines · Maciej Marcin Piechotka

Quoted 57 lines · Josef Bacik
Quoted 41 lines · Maciej Marcin Piechotka
Quoted 40 lines · Josef Bacik
Quoted 38 lines · Maciej Marcin Piechotka
Hello,
When I booted my machine (after clean powerdown) the following message appeared:
[   32.757913] device fsid XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX devid 1
transid 40864 /dev/mapper/XXXXX-XXXXX
[   32.758466] btrfs: use lzo compression
[   32.758475] btrfs: enabling disk space caching
[   32.758483] btrfs: enabling inode map caching
[   32.758490] btrfs: enabling auto defrag
[   32.758497] btrfs: thread pool 2
[   34.024359] btrfs: could not do orphan cleanup -116
[   63.173382] btrfs: open_ctree failed
btrfsck has shown no errors.
Afterwards when I tried to mount filesystem it have shown:
[  116.012528] btrfs: enabling inode map caching
[  116.012534] btrfs: enabling auto defrag
[  116.012542] btrfs: thread pool 2
It looked like everything is working but to be sure I've run scrub:
scrub status for fsid XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
scrub started at Mon Nov 7 13:05:50 2011 and finished after 606
seconds
total bytes scrubbed: 32.37GB with 0 errors
PS.
# find include | xargs grep 116
(...)
include/asm-generic/errno.h:#define     ESTALE          116     /* Stale
NFS file handle */
(...)
I don't use NFS - I use LVM on LUKS. In any case - it have been done concurrently with mounting of other fs which had no such problems.
Quoted 9 lines
Quoted 4 lines
Yeah that's a bug with an earlier kernel, if you upgrade to 3.1 it should go away. Thanks,
Josef
I'm using 'newer' kernel then 3.1 - I'm following Linus' tree (this kernel is few days old).
Do you have the commit
a8c9e5769718d47e87cce40c9b84cab421804797
Btrfs: fix orphan cleanup regression

Quoted 8 lines

Quoted 3 lines
If not then the kernel isn't new enough. Thanks,
Josef
It have been merged in master for 3.2-rc1 (shortly before tag).
Regards

Does that mean you have that patch and are still hitting the problem? If so try this debug patch so I can see where it's coming from. Thanks,

Josef

fs/btrfs/inode.c +5 -1

index faf8a60..d72225a 100644
@@ -2153,14 +2173,17 @@ int btrfs_orphan_cleanup(struct btrfs_root *root)
 		found_key.offset = 0;
 		inode = btrfs_iget(root->fs_info->sb, &found_key, root, NULL);
 		ret = PTR_RET(inode);
-		if (ret && ret != -ESTALE)
+		if (ret && ret != -ESTALE) {
+			printk(KERN_ERR "ret is %d\n");
 			goto out;
+		}
 
 		/*
 		 * Inode is already gone but the orphan item is still there,
 		 * kill the orphan item.
 		 */
 		if (ret == -ESTALE) {
+			printk(KERN_ERR "bad inode, deleting orphan item\n");
 			trans = btrfs_start_transaction(root, 1);
 			if (IS_ERR(trans)) {
 				ret = PTR_ERR(trans);

@@ -2170,6 +2193,7 @@ int btrfs_orphan_cleanup(struct btrfs_root *root)
 						    found_key.objectid);
 			BUG_ON(ret);
 			btrfs_end_transaction(trans, root);
+			printk(KERN_ERR "successfully deleted orphan item\n");
 			continue;
 		}

Maciej Marcin Piechotka

author

2011-11-09 16:18 UTC #6 lore raw

Quoted 77 lines · Josef Bacik

Quoted 2 lines · Maciej Marcin Piechotka
Quoted 1 line · Josef Bacik
On Mon, Nov 07, 2011 at 06:41:44PM +0000, Maciej Marcin Piechotka
Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines · Josef Bacik
Quoted 1 line
On Mon, Nov 07, 2011 at 01:17:04PM +0000, Maciej Marcin
Quoted 3 lines · Piechotka
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Hello,
When I booted my machine (after clean powerdown) the
following message
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
appeared:
[   32.757913] device fsid
XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX devid 1
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
transid 40864 /dev/mapper/XXXXX-XXXXX
[   32.758466] btrfs: use lzo compression
[   32.758475] btrfs: enabling disk space caching
[   32.758483] btrfs: enabling inode map caching
[   32.758490] btrfs: enabling auto defrag
[   32.758497] btrfs: thread pool 2
[   34.024359] btrfs: could not do orphan cleanup -116
[   63.173382] btrfs: open_ctree failed
btrfsck has shown no errors.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Afterwards when I tried to mount filesystem it have shown:
[  116.012528] btrfs: enabling inode map caching
[  116.012534] btrfs: enabling auto defrag
[  116.012542] btrfs: thread pool 2
It looked like everything is working but to be sure I've run
scrub:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
scrub status for fsid XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
        scrub started at Mon Nov  7 13:05:50 2011 and
finished after 606
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
seconds
total bytes scrubbed: 32.37GB with 0 errors
PS.
# find include | xargs grep 116
(...)
include/asm-generic/errno.h:#define     ESTALE
116 /* Stale
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
NFS file handle */
(...)
I don't use NFS - I use LVM on LUKS. In any case - it have
been done
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
concurrently with mounting of other fs which had no such
problems.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Yeah that's a bug with an earlier kernel, if you upgrade to
3.1 it should go
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
away. Thanks,
Josef
I'm using 'newer' kernel then 3.1 - I'm following Linus' tree
(this
Quoted 14 lines
Quoted 11 lines
Quoted 2 lines
kernel is few days old).
Do you have the commit
a8c9e5769718d47e87cce40c9b84cab421804797
Btrfs: fix orphan cleanup regression
If not then the kernel isn't new enough. Thanks,
Josef
It have been merged in master for 3.2-rc1 (shortly before tag).

Quoted 8 lines

Quoted 2 lines
Regards
Does that mean you have that patch and are still hitting the problem? If so try
this debug patch so I can see where it's coming from. Thanks,
Josef

No. I was just informing that it entered tree a bit later then 3.1 was released - in fact shortly before 3.2-rc1 (I presume btrfs devs not necessary remember when Linus merges changes into tree although I have no experience with kernel development). I hit it only once before updating the kernel.

Best regards