Skip to contentSkip to conversation

WARNING: at fs/btrfs/inode.c:2193 btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()

DISCUSSION Martin Mailand 6 participants 13 messages Sep 10, 2011 – Sep 19, 2011 Btrfs file system

lore
Share & tools
lore

Martin Mailand

author
#0 lore raw

Hi
I am hitting this Warning reproducible, the workload is a ceph osd, kernel ist 3.1.0-rc5.

Best Regards,
martin

[ 5472.099766] ------------[ cut here ]------------
[ 5472.099833] WARNING: at fs/btrfs/inode.c:2193 btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()
[ 5472.099838] Hardware name: MS-96B3
[ 5472.099842] Modules linked in: radeon ttm drm_kms_helper drm i2c_algo_bit psmouse sp5100_tco edac_core lp shpchp serio_raw k8temp i2c_piix4 edac_mce_amd parport ahci pata_atiixp e1000e libahci btrfs zlib_deflate libcrc32c
[ 5472.099878] Pid: 2066, comm: kworker/1:1 Not tainted 3.1.0-rc5-custom #1
[ 5472.099882] Call Trace:
[ 5472.099898]  [<ffffffff81063c1f>] warn_slowpath_common+0x7f/0xc0
[ 5472.099907]  [<ffffffff81063c7a>] warn_slowpath_null+0x1a/0x20
[ 5472.099935]  [<ffffffffa003f420>] btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]
[ 5472.099961]  [<ffffffffa00380aa>] commit_fs_roots.clone.21+0xba/0x1a0 [btrfs]
[ 5472.099971]  [<ffffffff815db96e>] ? _raw_spin_lock+0xe/0x20
[ 5472.099997]  [<ffffffffa003966f>] btrfs_commit_transaction+0x3ef/0x870 [btrfs]
[ 5472.100065]  [<ffffffff81012871>] ? __switch_to+0x261/0x2f0
[ 5472.100084]  [<ffffffff81086bf0>] ? wake_up_bit+0x40/0x40
[ 5472.100120]  [<ffffffffa0039af0>] ? btrfs_commit_transaction+0x870/0x870 [btrfs]
[ 5472.100155]  [<ffffffffa0039b0f>] do_async_commit+0x1f/0x30 [btrfs]
[ 5472.100171]  [<ffffffff8108110d>] process_one_work+0x11d/0x430
[ 5472.100187]  [<ffffffff81081c69>] worker_thread+0x169/0x360
[ 5472.100203]  [<ffffffff81081b00>] ? manage_workers.clone.21+0x240/0x240
[ 5472.100220]  [<ffffffff81086496>] kthread+0x96/0xa0
[ 5472.100236]  [<ffffffff815e5bb4>] kernel_thread_helper+0x4/0x10
[ 5472.100253]  [<ffffffff81086400>] ? flush_kthread_worker+0xb0/0xb0
[ 5472.100269]  [<ffffffff815e5bb0>] ? gs_change+0x13/0x13
[ 5472.100279] ---[ end trace a8bae5767c2c3e55 ]---
#1 lore raw
Quoted 4 lines · Martin Mailand

Hi
I am hitting this Warning reproducible, the workload is a ceph osd, kernel ist 3.1.0-rc5.

Have posted a patch for this:

http://marc.info/?l=linux-btrfs&m=131547325515336&w=2

thanks,
liubo

Quoted 40 lines

Best Regards,
martin

[ 5472.099766] ------------[ cut here ]------------
[ 5472.099833] WARNING: at fs/btrfs/inode.c:2193
btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()
[ 5472.099838] Hardware name: MS-96B3
[ 5472.099842] Modules linked in: radeon ttm drm_kms_helper drm
i2c_algo_bit psmouse sp5100_tco edac_core lp shpchp serio_raw k8temp
i2c_piix4 edac_mce_amd parport ahci pata_atiixp e1000e libahci btrfs
zlib_deflate libcrc32c
[ 5472.099878] Pid: 2066, comm: kworker/1:1 Not tainted 3.1.0-rc5-custom #1
[ 5472.099882] Call Trace:
[ 5472.099898]  [<ffffffff81063c1f>] warn_slowpath_common+0x7f/0xc0
[ 5472.099907]  [<ffffffff81063c7a>] warn_slowpath_null+0x1a/0x20
[ 5472.099935]  [<ffffffffa003f420>] btrfs_orphan_commit_root+0xb0/0xc0
[btrfs]
[ 5472.099961]  [<ffffffffa00380aa>] commit_fs_roots.clone.21+0xba/0x1a0
[btrfs]
[ 5472.099971]  [<ffffffff815db96e>] ? _raw_spin_lock+0xe/0x20
[ 5472.099997]  [<ffffffffa003966f>]
btrfs_commit_transaction+0x3ef/0x870 [btrfs]
[ 5472.100065]  [<ffffffff81012871>] ? __switch_to+0x261/0x2f0
[ 5472.100084]  [<ffffffff81086bf0>] ? wake_up_bit+0x40/0x40
[ 5472.100120]  [<ffffffffa0039af0>] ?
btrfs_commit_transaction+0x870/0x870 [btrfs]
[ 5472.100155]  [<ffffffffa0039b0f>] do_async_commit+0x1f/0x30 [btrfs]
[ 5472.100171]  [<ffffffff8108110d>] process_one_work+0x11d/0x430
[ 5472.100187]  [<ffffffff81081c69>] worker_thread+0x169/0x360
[ 5472.100203]  [<ffffffff81081b00>] ? manage_workers.clone.21+0x240/0x240
[ 5472.100220]  [<ffffffff81086496>] kthread+0x96/0xa0
[ 5472.100236]  [<ffffffff815e5bb4>] kernel_thread_helper+0x4/0x10
[ 5472.100253]  [<ffffffff81086400>] ? flush_kthread_worker+0xb0/0xb0
[ 5472.100269]  [<ffffffff815e5bb0>] ? gs_change+0x13/0x13
[ 5472.100279] ---[ end trace a8bae5767c2c3e55 ]---
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#2 lore raw
Quoted 9 lines · Liu Bo
Quoted 4 lines · Martin Mailand

Hi
I am hitting this Warning reproducible, the workload is a ceph osd, kernel ist 3.1.0-rc5.

Have posted a patch for this:

http://marc.info/?l=linux-btrfs&m=131547325515336&w=2

We're still seeing this with -rc6, which includes 98c9942 and 65450aa.

I haven't looked at the reservation code in much detail. Is there anything I can do to help track this down?

Thanks-
sage

Quoted 51 lines

thanks,
liubo

Quoted 27 lines

Best Regards,
martin

[ 5472.099766] ------------[ cut here ]------------
[ 5472.099833] WARNING: at fs/btrfs/inode.c:2193
btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()
[ 5472.099838] Hardware name: MS-96B3
[ 5472.099842] Modules linked in: radeon ttm drm_kms_helper drm
i2c_algo_bit psmouse sp5100_tco edac_core lp shpchp serio_raw k8temp
i2c_piix4 edac_mce_amd parport ahci pata_atiixp e1000e libahci btrfs
zlib_deflate libcrc32c
[ 5472.099878] Pid: 2066, comm: kworker/1:1 Not tainted 3.1.0-rc5-custom #1
[ 5472.099882] Call Trace:
[ 5472.099898]  [<ffffffff81063c1f>] warn_slowpath_common+0x7f/0xc0
[ 5472.099907]  [<ffffffff81063c7a>] warn_slowpath_null+0x1a/0x20
[ 5472.099935]  [<ffffffffa003f420>] btrfs_orphan_commit_root+0xb0/0xc0
[btrfs]
[ 5472.099961]  [<ffffffffa00380aa>] commit_fs_roots.clone.21+0xba/0x1a0
[btrfs]
[ 5472.099971]  [<ffffffff815db96e>] ? _raw_spin_lock+0xe/0x20
[ 5472.099997]  [<ffffffffa003966f>]
btrfs_commit_transaction+0x3ef/0x870 [btrfs]
[ 5472.100065]  [<ffffffff81012871>] ? __switch_to+0x261/0x2f0
[ 5472.100084]  [<ffffffff81086bf0>] ? wake_up_bit+0x40/0x40
[ 5472.100120]  [<ffffffffa0039af0>] ?
btrfs_commit_transaction+0x870/0x870 [btrfs]
[ 5472.100155]  [<ffffffffa0039b0f>] do_async_commit+0x1f/0x30 [btrfs]
Quoted 13 lines
[ 5472.100171]  [<ffffffff8108110d>] process_one_work+0x11d/0x430
[ 5472.100187]  [<ffffffff81081c69>] worker_thread+0x169/0x360
[ 5472.100203]  [<ffffffff81081b00>] ? manage_workers.clone.21+0x240/0x240
[ 5472.100220]  [<ffffffff81086496>] kthread+0x96/0xa0
[ 5472.100236]  [<ffffffff815e5bb4>] kernel_thread_helper+0x4/0x10
[ 5472.100253]  [<ffffffff81086400>] ? flush_kthread_worker+0xb0/0xb0
[ 5472.100269]  [<ffffffff815e5bb0>] ? gs_change+0x13/0x13
[ 5472.100279] ---[ end trace a8bae5767c2c3e55 ]---
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
Quoted 7 lines · Sage Weil
Quoted 6 lines · Liu Bo
Quoted 4 lines · Martin Mailand

Hi
I am hitting this Warning reproducible, the workload is a ceph osd, kernel ist 3.1.0-rc5.

Quoted 9 lines
Quoted 3 lines

Have posted a patch for this:

http://marc.info/?l=linux-btrfs&m=131547325515336&w=2

We're still seeing this with -rc6, which includes 98c9942 and 65450aa.

I haven't looked at the reservation code in much detail. Is there anything I can do to help track this down?

This should be taken care of with all my enospc changes. You can pull them down from my btrfs-work tree as soon as kernel.org comes back from the dead :). Thanks,

Josef

#4 lore raw
Quoted 8 lines · Josef Bacik
Quoted 5 lines

We're still seeing this with -rc6, which includes 98c9942 and 65450aa.

I haven't looked at the reservation code in much detail. Is there anything I can do to help track this down?

This should be taken care of with all my enospc changes. You can pull them down from my btrfs-work tree as soon as kernel.org comes back from the dead :).

should you need it earlier, here's a copy:

git://repo.or.cz/linux-2.6/btrfs-unstable.git #git.kernel.org/josef/master

david

#5 lore raw
Quoted 7 lines · David Sterba
Quoted 6 lines · Josef Bacik
Quoted 5 lines

We're still seeing this with -rc6, which includes 98c9942 and 65450aa.

I haven't looked at the reservation code in much detail. Is there anything I can do to help track this down?

Quoted 6 lines
Quoted 2 lines

This should be taken care of with all my enospc changes. You can pull them down from my btrfs-work tree as soon as kernel.org comes back from the dead :).

should you need it earlier, here's a copy:

git://repo.or.cz/linux-2.6/btrfs-unstable.git #git.kernel.org/josef/master

Thanks! We'll do some testing today and see if it behaves better. :)

sage

author
#6 lore raw
In reply to Josef Bacik

Hi Josef,
after a quick test it seems that I do not hit this Warning any longer. But I got a new one.

[ 5241.839951] ------------[ cut here ]------------
[ 5241.839974] WARNING: at fs/btrfs/extent-tree.c:5715 btrfs_alloc_free_block+0xac/0x330 [btrfs]()
[ 5241.839979] Hardware name: MS-96B3
[ 5241.839982] Modules linked in: radeon ttm drm_kms_helper drm i2c_algo_bit psmouse k8temp sp5100_tco edac_core edac_mce_amd serio_raw shpchp i2c_piix4 lp parport ahci pata_atiixp libahci btrfs e1000e zlib_deflate libcrc32c
[ 5241.840068] Pid: 1568, comm: kworker/0:1 Tainted: G        W 3.1.0-rc6 #1
[ 5241.840072] Call Trace:
[ 5241.840084]  [<ffffffff81063d0f>] warn_slowpath_common+0x7f/0xc0
[ 5241.840101]  [<ffffffff81063d6a>] warn_slowpath_null+0x1a/0x20
[ 5241.840133]  [<ffffffffa002a9cc>] btrfs_alloc_free_block+0xac/0x330 [btrfs]
[ 5241.840152]  [<ffffffff8110d35a>] ? unlock_page+0x2a/0x40
[ 5241.840188]  [<ffffffffa0059268>] ? read_extent_buffer+0xa8/0x180 [btrfs]
[ 5241.840222]  [<ffffffffa0031c00>] ? verify_parent_transid+0x160/0x160 [btrfs]
[ 5241.840252]  [<ffffffffa001a0d2>] __btrfs_cow_block+0x122/0x4b0 [btrfs]
[ 5241.840283]  [<ffffffffa001a552>] btrfs_cow_block+0xf2/0x1f0 [btrfs]
[ 5241.840314]  [<ffffffffa001cb88>] push_leaf_left+0x108/0x180 [btrfs]
[ 5241.840344]  [<ffffffffa001fb78>] btrfs_del_items+0x2b8/0x440 [btrfs]
[ 5241.840379]  [<ffffffffa00300c2>] btrfs_del_csums+0x2d2/0x310 [btrfs]
[ 5241.840415]  [<ffffffffa00677a8>] ? btrfs_tree_unlock+0x28/0xb0 [btrfs]
[ 5241.840447]  [<ffffffffa002597a>] __btrfs_free_extent+0x48a/0x6f0 [btrfs]
[ 5241.840480]  [<ffffffffa0028c8d>] run_clustered_refs+0x21d/0x840 [btrfs]
[ 5241.840514]  [<ffffffffa002937a>] btrfs_run_delayed_refs+0xca/0x220 [btrfs]
[ 5241.840551]  [<ffffffffa0053576>] ? btrfs_run_ordered_operations+0x1d6/0x200 [btrfs]
[ 5241.840587]  [<ffffffffa0038fa3>] btrfs_commit_transaction+0x83/0x870 [btrfs]
[ 5241.840605]  [<ffffffff81012871>] ? __switch_to+0x261/0x2f0
[ 5241.840622]  [<ffffffff81086d70>] ? wake_up_bit+0x40/0x40
[ 5241.840656]  [<ffffffffa0039790>] ? btrfs_commit_transaction+0x870/0x870 [btrfs]
[ 5241.840691]  [<ffffffffa00397af>] do_async_commit+0x1f/0x30 [btrfs]
[ 5241.840708]  [<ffffffff8108110d>] process_one_work+0x11d/0x430
[ 5241.840724]  [<ffffffff81081dd9>] worker_thread+0x169/0x360
[ 5241.840741]  [<ffffffff81081c70>] ? manage_workers.clone.21+0x240/0x240
[ 5241.840758]  [<ffffffff81086616>] kthread+0x96/0xa0
[ 5241.840775]  [<ffffffff815f2434>] kernel_thread_helper+0x4/0x10
[ 5241.840792]  [<ffffffff81086580>] ? flush_kthread_worker+0xb0/0xb0
[ 5241.840808]  [<ffffffff815f2430>] ? gs_change+0x13/0x13
[ 5241.840819] ---[ end trace c8a580615cad6cb5 ]---

Best Regards,
Martin

Am 15.09.2011 21:50, schrieb Josef Bacik:

Quoted 27 lines · Sage Weil
Quoted 7 lines · Liu Bo
Quoted 6 lines · Martin Mailand
Quoted 4 lines

Hi
I am hitting this Warning reproducible, the workload is a ceph osd, kernel ist 3.1.0-rc5.

Quoted 9 lines
Quoted 3 lines

Have posted a patch for this:

http://marc.info/?l=linux-btrfs&m=131547325515336&w=2

We're still seeing this with -rc6, which includes 98c9942 and 65450aa.

I haven't looked at the reservation code in much detail. Is there anything I can do to help track this down?

This should be taken care of with all my enospc changes. You can pull them down from my btrfs-work tree as soon as kernel.org comes back from the dead :). Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#7 lore raw
Quoted 4 lines · Martin Mailand

Hi Josef,
after a quick test it seems that I do not hit this Warning any longer. But I got a new one.

Hmm looks like that may not be my newest stuff, is commit

57f499e1bb76ba3ebeb09cd12e9dac84baa5812b

in there? Specifically look at __btrfs_end_transaction in transaction.c and see if the line

trans->block_rsv = NULL;

is before the first while() loop. Thanks,

Josef

author
#8 lore raw

Hi Josef,
the commit is not in there, but the code looks like your post.

if (--trans->use_count) {
                trans->block_rsv = trans->orig_rsv;
                return 0;
        }

        trans->block_rsv = NULL;
        while (count < 4) {
                unsigned long cur = trans->delayed_ref_updates;
                trans->delayed_ref_updates = 0;

But on the other hand I am quite new to git, how could I get your latest commit?

Best Regards,
Martin

Am 16.09.2011 16:37, schrieb Josef Bacik:

Quoted 22 lines · Martin Mailand
Quoted 4 lines

Hi Josef,
after a quick test it seems that I do not hit this Warning any longer. But I got a new one.

Hmm looks like that may not be my newest stuff, is commit

57f499e1bb76ba3ebeb09cd12e9dac84baa5812b

in there? Specifically look at __btrfs_end_transaction in transaction.c and see if the line

trans->block_rsv = NULL;

is before the first while() loop. Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#9 lore raw
In reply to Sage Weil
Quoted 7 lines · Sage Weil
Quoted 6 lines · Liu Bo
Quoted 4 lines · Martin Mailand

Hi
I am hitting this Warning reproducible, the workload is a ceph osd, kernel ist 3.1.0-rc5.

Quoted 5 lines
Quoted 3 lines

Have posted a patch for this:

http://marc.info/?l=linux-btrfs&m=131547325515336&w=2

We're still seeing this with -rc6, which includes 98c9942 and 65450aa.

Me too, for the

WARNING: at fs/btrfs/extent-tree.c:5711 btrfs_alloc_free_block+0x180/0x350 [btrfs]()

case mentioned in the changelog. I optimistically dropped the ratelimit patch for that WARN_ON, but had to add it quickly back. Unfortunatelly I do not have a reliable reproducer. It justs starts sometime during xfstests, maybe after a few rounds.

david

#10 lore raw
Quoted 7 lines · David Sterba
Quoted 6 lines · Sage Weil
Quoted 5 lines · Liu Bo
Quoted 4 lines · Martin Mailand

Hi
I am hitting this Warning reproducible, the workload is a ceph osd, kernel ist 3.1.0-rc5.

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

Have posted a patch for this:

http://marc.info/?l=linux-btrfs&m=131547325515336&w=2

We're still seeing this with -rc6, which includes 98c9942 and 65450aa.

Me too, for the

WARNING: at fs/btrfs/extent-tree.c:5711 btrfs_alloc_free_block+0x180/0x350 [btrfs]()

FWIW, I'm seeing a slightly different case, while testing branch integration/btrfs-next (commit 2828cbd9620e03) from
git://repo.or.cz/linux-2.6/btrfs-unstable.git merged into branch master (commit c455ea4f122d21) from git://github.com/torvalds/linux.git

Under a heavy ceph write load, I see lots of these:

[ 2369.797044] ------------[ cut here ]------------
[ 2369.801759] WARNING: at fs/btrfs/extent-tree.c:5751 use_block_rsv+0x177/0x180 [btrfs]()
[ 2369.809864] Hardware name: X8DTH-i/6/iF/6F
[ 2369.814062] Modules linked in: loop btrfs zlib_deflate lzo_compress ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack
[ 2369.828671] btrfs: journal_info set but trans not join/nolock: 0x1
[ 2369.829040]  ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp scsi_transport_iscsi rds ib_ipoib rdma_ucm rdma_cm ib_ucm ib_uverbs ib_umad ib_cm iw_cm ib_addr ipv6 ib_sa dm_mirror dm_region_hash dm_log dm_multipath scsi_dh dm_mod video sbs sbshc pci_slot battery acpi_pad ac kvm sg sd_mod mlx4_ib ib_mad ib_core mlx4_en joydev mpt2sas tpm_tis tpm scsi_transport_sas mlx4_core cxgb4 button serio_raw raid_class tpm_bios ata_piix libata scsi_mod i2c_i801 ioatdma ehci_hcd iTCO_wdt uhci_hcd i2c_core i7core_edac iTCO_vendor_support edac_core pcspkr rtc nfs nfs_acl auth_rpcgss fscache lockd sunrpc tg3 bnx2 igb dca e1000
[ 2369.889908] Pid: 23744, comm: kworker/19:3 Tainted: G        W   3.1.0-rc6-00265-gf883c8c #33
[ 2369.898510] Call Trace:
[ 2369.901026]  [<ffffffff8104e02f>] warn_slowpath_common+0x7f/0xc0
[ 2369.907046]  [<ffffffff8104e08a>] warn_slowpath_null+0x1a/0x20
[ 2369.912935]  [<ffffffffa05c7fb7>] use_block_rsv+0x177/0x180 [btrfs]
[ 2369.919422]  [<ffffffffa05d122d>] btrfs_alloc_free_block+0x3d/0x220 [btrfs]
[ 2369.926431]  [<ffffffff810f1a01>] ? __set_page_dirty_nobuffers+0xe1/0x150
[ 2369.933265]  [<ffffffffa05ff3c1>] ? read_extent_buffer+0xc1/0x1a0 [btrfs]
[ 2369.940085]  [<ffffffffa05bf58e>] __btrfs_cow_block+0x11e/0x4d0 [btrfs]
[ 2369.946737]  [<ffffffffa0610091>] ? btrfs_tree_lock+0x161/0x2c0 [btrfs]
[ 2369.953391]  [<ffffffffa05bfefa>] btrfs_cow_block+0xea/0x200 [btrfs]
[ 2369.959756]  [<ffffffffa05c2b2f>] btrfs_search_slot+0x31f/0x720 [btrfs]
[ 2369.966362]  [<ffffffffa05d3a9b>] btrfs_del_csums+0x23b/0x320 [btrfs]
[ 2369.972833]  [<ffffffffa05cf41b>] __btrfs_free_extent+0x49b/0x720 [btrfs]
[ 2369.979660]  [<ffffffffa05cf969>] run_delayed_data_ref+0x159/0x160 [btrfs]
[ 2369.986591]  [<ffffffffa05cfa20>] run_one_delayed_ref+0xb0/0xd0 [btrfs]
[ 2369.993257]  [<ffffffffa05cfb0f>] run_clustered_refs+0xcf/0x240 [btrfs]
[ 2369.999928]  [<ffffffffa05cfd58>] btrfs_run_delayed_refs+0xd8/0x260 [btrfs]
[ 2370.006937]  [<ffffffffa05dde77>] btrfs_commit_transaction+0x87/0x8b0 [btrfs]
[ 2370.014238]  [<ffffffff81072930>] ? wake_up_bit+0x40/0x40
[ 2370.019710]  [<ffffffffa05de6a0>] ? btrfs_commit_transaction+0x8b0/0x8b0 [btrfs]
[ 2370.027180]  [<ffffffffa05de6bf>] do_async_commit+0x1f/0x30 [btrfs]
[ 2370.033578]  [<ffffffffa05de6a0>] ? btrfs_commit_transaction+0x8b0/0x8b0 [btrfs]
[ 2370.041039]  [<ffffffff8106c03f>] process_one_work+0x13f/0x490
[ 2370.046950]  [<ffffffff8106ddb7>] worker_thread+0x187/0x3e0
[ 2370.052679]  [<ffffffff8106dc30>] ? manage_workers+0x120/0x120
[ 2370.058682]  [<ffffffff810723a6>] kthread+0x96/0xa0
[ 2370.063690]  [<ffffffff8145c7f4>] kernel_thread_helper+0x4/0x10
[ 2370.069695]  [<ffffffff8145284a>] ? retint_restore_args+0xe/0xe
[ 2370.075720]  [<ffffffff81072310>] ? kthread_worker_fn+0x1d0/0x1d0
[ 2370.081919]  [<ffffffff8145c7f0>] ? gs_change+0xb/0xb
[ 2370.087103] ---[ end trace a6d5cd679d4e46b9 ]---

I don't know if it matters, but I also see lots of this sort of thing:

[ 2370.131721] btrfs: all snaps cleaned
[ 2370.241382] btrfs: btrfs_clean_old_snapshots to process 1 old snaps
[ 2370.275303] btrfs: btrfs_clean_old_snapshots to process 0 old snaps
[ 2370.275306] btrfs: all snaps cleaned
[ 2370.296461] btrfs: all snaps cleaned
[ 2370.639211] btrfs: btrfs_clean_old_snapshots to process 1 old snaps
[ 2370.688785] btrfs: all snaps cleaned
[ 2370.811568] btrfs: journal_info set but trans not join/nolock: 0x1
[ 2370.830958] btrfs: btrfs_clean_old_snapshots to process 0 old snaps
[ 2370.830962] btrfs: all snaps cleaned
[ 2371.040634] btrfs: journal_info set but trans not join/nolock: 0x1
[ 2371.057286] btrfs: journal_info set but trans not join/nolock: 0x1
[ 2371.267641] btrfs: btrfs_clean_old_snapshots to process 1 old snaps
[ 2371.285724] btrfs: journal_info set but trans not join/nolock: 0x1
[ 2371.301746] btrfs: journal_info set but trans not join/nolock: 0x1
[ 2371.328422] btrfs: all snaps cleaned
[ 2371.371326] btrfs: btrfs_clean_old_snapshots to process 1 old snaps
[ 2371.417955] btrfs: all snaps cleaned
[ 2371.498012] btrfs: btrfs_clean_old_snapshots to process 1 old snaps
[ 2371.530904] btrfs: btrfs_clean_old_snapshots to process 0 old snaps
[ 2371.530906] btrfs: all snaps cleaned
[ 2371.553309] btrfs: all snaps cleaned
[ 2371.574753] btrfs: btrfs_clean_old_snapshots to process 1 old snaps
[ 2371.625655] btrfs: all snaps cleaned

-- Jim

Quoted 9 lines

david
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#11 lore raw
Quoted 10 lines · Jim Schutt
Quoted 3 lines
WARNING: at fs/btrfs/extent-tree.c:5711 btrfs_alloc_free_block+0x180/0x350

[btrfs]()

FWIW, I'm seeing a slightly different case, while testing branch integration/btrfs-next (commit 2828cbd9620e03) from
git://repo.or.cz/linux-2.6/btrfs-unstable.git merged into branch master (commit c455ea4f122d21) from git://github.com/torvalds/linux.git

Under a heavy ceph write load, I see lots of these:

same warning from mainline fs/btrfs/extent-tree.c:5711 matching fs/btrfs/extent-tree.c:5751 in the btrfs-next branch and is ratelimited. It does not seem to be fixed by the patch, but as the branch contains other patches, it could be caused ty them, I didn't have a closer look. You can try with just Josef's branch on top of mainline if you have a good way to trigger the warnings.

Quoted 58 lines
[ 2369.797044] ------------[ cut here ]------------
[ 2369.801759] WARNING: at fs/btrfs/extent-tree.c:5751 
use_block_rsv+0x177/0x180 [btrfs]()
[ 2369.809864] Hardware name: X8DTH-i/6/iF/6F
[ 2369.814062] Modules linked in: loop btrfs zlib_deflate lzo_compress 
ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state 
nf_conntrack
[ 2369.828671] btrfs: journal_info set but trans not join/nolock: 0x1
[ 2369.829040]  ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables

bridge stp scsi_transport_iscsi rds ib_ipoib rdma_ucm rdma_cm ib_ucm ib_uverbs ib_umad ib_cm iw_cm ib_addr ipv6 ib_sa dm_mirror dm_region_hash dm_log dm_multipath scsi_dh dm_mod video sbs sbshc pci_slot battery acpi_pad ac kvm sg sd_mod mlx4_ib ib_mad ib_core mlx4_en joydev mpt2sas tpm_tis tpm scsi_transport_sas mlx4_core cxgb4 button serio_raw raid_class tpm_bios ata_piix libata scsi_mod i2c_i801 ioatdma ehci_hcd iTCO_wdt uhci_hcd i2c_core i7core_edac iTCO_vendor_support edac_core pcspkr rtc nfs nfs_acl auth_rpcgss fscache lockd sunrpc tg3 bnx2 igb dca e1000

[ 2369.889908] Pid: 23744, comm: kworker/19:3 Tainted: G        W   
3.1.0-rc6-00265-gf883c8c #33
[ 2369.898510] Call Trace:
[ 2369.901026]  [<ffffffff8104e02f>] warn_slowpath_common+0x7f/0xc0
[ 2369.907046]  [<ffffffff8104e08a>] warn_slowpath_null+0x1a/0x20
[ 2369.912935]  [<ffffffffa05c7fb7>] use_block_rsv+0x177/0x180 [btrfs]
[ 2369.919422]  [<ffffffffa05d122d>] btrfs_alloc_free_block+0x3d/0x220 
[btrfs]
[ 2369.926431]  [<ffffffff810f1a01>] ? __set_page_dirty_nobuffers+0xe1/0x150
[ 2369.933265]  [<ffffffffa05ff3c1>] ? read_extent_buffer+0xc1/0x1a0 [btrfs]
[ 2369.940085]  [<ffffffffa05bf58e>] __btrfs_cow_block+0x11e/0x4d0 [btrfs]
[ 2369.946737]  [<ffffffffa0610091>] ? btrfs_tree_lock+0x161/0x2c0 [btrfs]
[ 2369.953391]  [<ffffffffa05bfefa>] btrfs_cow_block+0xea/0x200 [btrfs]
[ 2369.959756]  [<ffffffffa05c2b2f>] btrfs_search_slot+0x31f/0x720 [btrfs]
[ 2369.966362]  [<ffffffffa05d3a9b>] btrfs_del_csums+0x23b/0x320 [btrfs]
[ 2369.972833]  [<ffffffffa05cf41b>] __btrfs_free_extent+0x49b/0x720 [btrfs]
[ 2369.979660]  [<ffffffffa05cf969>] run_delayed_data_ref+0x159/0x160 
[btrfs]
[ 2369.986591]  [<ffffffffa05cfa20>] run_one_delayed_ref+0xb0/0xd0 [btrfs]
[ 2369.993257]  [<ffffffffa05cfb0f>] run_clustered_refs+0xcf/0x240 [btrfs]
[ 2369.999928]  [<ffffffffa05cfd58>] btrfs_run_delayed_refs+0xd8/0x260 
[btrfs]
[ 2370.006937]  [<ffffffffa05dde77>] btrfs_commit_transaction+0x87/0x8b0 
[btrfs]
[ 2370.014238]  [<ffffffff81072930>] ? wake_up_bit+0x40/0x40
[ 2370.019710]  [<ffffffffa05de6a0>] ? btrfs_commit_transaction+0x8b0/0x8b0 
[btrfs]
[ 2370.027180]  [<ffffffffa05de6bf>] do_async_commit+0x1f/0x30 [btrfs]
[ 2370.033578]  [<ffffffffa05de6a0>] ? btrfs_commit_transaction+0x8b0/0x8b0 
[btrfs]
[ 2370.041039]  [<ffffffff8106c03f>] process_one_work+0x13f/0x490
[ 2370.046950]  [<ffffffff8106ddb7>] worker_thread+0x187/0x3e0
[ 2370.052679]  [<ffffffff8106dc30>] ? manage_workers+0x120/0x120
[ 2370.058682]  [<ffffffff810723a6>] kthread+0x96/0xa0
[ 2370.063690]  [<ffffffff8145c7f4>] kernel_thread_helper+0x4/0x10
[ 2370.069695]  [<ffffffff8145284a>] ? retint_restore_args+0xe/0xe
[ 2370.075720]  [<ffffffff81072310>] ? kthread_worker_fn+0x1d0/0x1d0
[ 2370.081919]  [<ffffffff8145c7f0>] ? gs_change+0xb/0xb
[ 2370.087103] ---[ end trace a6d5cd679d4e46b9 ]---
Quoted 5 lines

I don't know if it matters, but I also see lots of this sort of thing:

[ 2370.131721] btrfs: all snaps cleaned
[ 2370.241382] btrfs: btrfs_clean_old_snapshots to process 1 old snaps
[ 2370.275303] btrfs: btrfs_clean_old_snapshots to process 0 old snaps

Harmless debugging of

Quoted 3 lines
[ 2370.275306] btrfs: all snaps cleaned
[ 2370.296461] btrfs: all snaps cleaned
[ 2370.639211] btrfs: btrfs_clean_old_snapshots to process 1 old snaps

... slow snapshot deletion reported by BCrook.

Quoted 2 lines
[ 2370.688785] btrfs: all snaps cleaned
[ 2370.811568] btrfs: journal_info set but trans not join/nolock: 0x1

... btrfs/transaction.c:264 in start_transaction, but the patch was not correct and reported not only the WARN_ON case.

Thanks for the report, I'll clean the branch a bit and move the debugging or too experimental stuff into a separate branch.

david

#12 lore raw
In reply to Jim Schutt
Quoted 18 lines · Jim Schutt
Quoted 7 lines · David Sterba
Quoted 6 lines · Sage Weil
Quoted 5 lines · Liu Bo
Quoted 4 lines · Martin Mailand

Hi
I am hitting this Warning reproducible, the workload is a ceph osd, kernel ist 3.1.0-rc5.

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

Have posted a patch for this:

http://marc.info/?l=linux-btrfs&m=131547325515336&w=2

We're still seeing this with -rc6, which includes 98c9942 and 65450aa.

Me too, for the

WARNING: at fs/btrfs/extent-tree.c:5711
btrfs_alloc_free_block+0x180/0x350 [btrfs]()
Quoted 5 lines

FWIW, I'm seeing a slightly different case, while testing branch integration/btrfs-next (commit 2828cbd9620e03) from
git://repo.or.cz/linux-2.6/btrfs-unstable.git merged into branch master (commit c455ea4f122d21) from git://github.com/torvalds/linux.git

Ah yeah sorry I see what's going on here, I just missed a few places we call run_delayed_refs() where we can still have trans->block_rsv set. I will fix this and send a patch soon. Thanks,

Josef