kernel BUG at fs/btrfs/async-thread.c:605!
Hi,
In kernel 3.8-rc5, the following panics occurred when the mount was done by the degraded option.
# btrfs fi sh /dev/sdc8
Label: none uuid: fc63cd80-5ae2-4fbe-8795-2d526c937a56
Total devices 3 FS bytes used 20.98GB
devid 1 size 9.31GB used 9.31GB path /dev/sdd8
devid 2 size 9.31GB used 9.31GB path /dev/sdc8
*** Some devices missingBtrfs v0.20-rc1-37-g91d9eec
# mount -o degraded /dev/sdc8 /test1
564 static struct btrfs_worker_thread *find_worker(struct btrfs_workers *workers) 565 {
...
...
595 fallback:
596 fallback = NULL;
597 /*
598 * we have failed to find any workers, just 599 * return the first one we can find. 600 */
601 if (!list_empty(&workers->worker_list))
602 fallback = workers->worker_list.next;
603 if (!list_empty(&workers->idle_list))
604 fallback = workers->idle_list.next;
605 BUG_ON(!fallback); <---------------------- this !
606 worker = list_entry(fallback,
607 struct btrfs_worker_thread, worker_list);
-Tsutomu
===================================================================================
[ 7913.075890] btrfs: allowing degraded mounts [ 7913.075893] btrfs: disk space caching is enabled [ 7913.092031] Btrfs: too many missing devices, writeable mount is not allowed [ 7913.092297] ------------[ cut here ]------------ [ 7913.092313] kernel BUG at fs/btrfs/async-thread.c:605! [ 7913.092326] invalid opcode: 0000 [#1] SMP [ 7913.092342] Modules linked in: btrfs zlib_deflate crc32c libcrc32c nfsd lockd nfs_acl auth_rpcgss sunrpc 8021q garp stp llc cpufreq_ondemand cachefiles fscache ipv6 ext3 jbd dm_mirror dm_region_hash dm_log dm_mod uinput ppdev iTCO_wdt iTCO_vendor_support parport_pc parport sg acpi_cpufreq freq_table mperf coretemp kvm pcspkr i2c_i801 i2c_core lpc_ich mfd_core tg3 ptp pps_core shpchp pci_hotplug i3000_edac edac_core ext4 mbcache jbd2 crc16 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_piix libata megaraid_sas scsi_mod floppy [last unloaded: microcode] [ 7913.092575] CPU 0 [ 7913.092584] Pid: 3673, comm: btrfs-endio-wri Not tainted 3.8.0-rc5 #1 FUJITSU-SV PRIMERGY /D2399 [ 7913.092608] RIP: 0010:[<ffffffffa04670ef>] [<ffffffffa04670ef>] btrfs_queue_worker+0x10e/0x236 [btrfs] [ 7913.092663] RSP: 0018:ffff88019fc03c10 EFLAGS: 00010046 [ 7913.092676] RAX: 0000000000000000 RBX: ffff8801967b8a58 RCX: 0000000000000000 [ 7913.092894] RDX: 0000000000000000 RSI: ffff8801961239b8 RDI: ffff8801967b8ab8 [ 7913.093116] RBP: ffff88019fc03c50 R08: 0000000000000000 R09: ffff880198801180 [ 7913.093247] R10: ffffffffa045fda7 R11: 0000000000000003 R12: 0000000000000000 [ 7913.093247] R13: ffff8801961239b8 R14: ffff8801967b8ab8 R15: 0000000000000246 [ 7913.093247] FS: 0000000000000000(0000) GS:ffff88019fc00000(0000) knlGS:0000000000000000 [ 7913.093247] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 7913.093247] CR2: ffffffffff600400 CR3: 000000019575d000 CR4: 00000000000007f0 [ 7913.093247] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7913.093247] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7913.093247] Process btrfs-endio-wri (pid: 3673, threadinfo ffff8801939ca000, task ffff880195795b00) [ 7913.093247] Stack: [ 7913.093247] ffff8801967b8a88 ffff8801967b8a78 ffff88003fa0a600 ffff8801965ad0c0 [ 7913.093247] ffff88003fa0a600 0000000000000000 0000000000000000 0000000000000000 [ 7913.096183] ffff88019fc03c60 ffffffffa043e357 ffff88019fc03c70 ffffffff811526aa [ 7913.096183] Call Trace: [ 7913.096183] <IRQ> [ 7913.096183] [ 7913.096183] [<ffffffffa043e357>] end_workqueue_bio+0x79/0x7b [btrfs] [ 7913.096183] [<ffffffff811526aa>] bio_endio+0x2d/0x2f [ 7913.096183] [<ffffffffa045fdb2>] btrfs_end_bio+0x10b/0x122 [btrfs] [ 7913.096183] [<ffffffff811526aa>] bio_endio+0x2d/0x2f [ 7913.096183] [<ffffffff811c5e3f>] req_bio_endio+0x96/0x9f [ 7913.096183] [<ffffffff811c601d>] blk_update_request+0x1d5/0x3a4 [ 7913.096183] [<ffffffff811c620c>] blk_update_bidi_request+0x20/0x6f [ 7913.096183] [<ffffffff811c7a59>] blk_end_bidi_request+0x1f/0x5d [ 7913.096183] [<ffffffff811c7ad3>] blk_end_request+0x10/0x12 [ 7913.096183] [<ffffffffa001db50>] scsi_io_completion+0x207/0x4f3 [scsi_mod] [ 7913.096183] [<ffffffffa0016df9>] scsi_finish_command+0xec/0xf5 [scsi_mod] [ 7913.096183] [<ffffffffa001df50>] scsi_softirq_done+0xff/0x108 [scsi_mod] [ 7913.096183] [<ffffffff811ccb3a>] blk_done_softirq+0x7a/0x8e [ 7913.096183] [<ffffffff810475c3>] __do_softirq+0xd7/0x1ed [ 7913.096183] [<ffffffff813ead9c>] call_softirq+0x1c/0x30 [ 7913.096183] [<ffffffff81010ab6>] do_softirq+0x46/0x83 [ 7913.096183] [<ffffffff81047363>] irq_exit+0x49/0xb7 [ 7913.096183] [<ffffffff813eafd5>] do_IRQ+0x9d/0xb4 [ 7913.096183] [<ffffffffa0467217>] ? btrfs_queue_worker+0x236/0x236 [btrfs] [ 7913.096183] [<ffffffff813e2a2d>] common_interrupt+0x6d/0x6d [ 7913.096183] <EOI> [ 7913.096183] [ 7913.096183] [<ffffffff81069a5e>] ? sched_move_task+0x12e/0x13d [ 7913.096183] [<ffffffff8104a131>] ? ptrace_put_breakpoints+0x1/0x1e [ 7913.096183] [<ffffffff81044fc2>] ? do_exit+0x3d7/0x89d [ 7913.096183] [<ffffffffa0467217>] ? btrfs_queue_worker+0x236/0x236 [btrfs] [ 7913.096183] [<ffffffffa0467217>] ? btrfs_queue_worker+0x236/0x236 [btrfs] [ 7913.096183] [<ffffffff8105ca20>] kthread+0xbd/0xbd [ 7913.096183] [<ffffffff8105c963>] ? kthread_freezable_should_stop+0x65/0x65 [ 7913.096183] [<ffffffff813e99ec>] ret_from_fork+0x7c/0xb0 [ 7913.096183] [<ffffffff8105c963>] ? kthread_freezable_should_stop+0x65/0x65 [ 7913.096183] Code: 49 89 c7 0f 84 5f ff ff ff 48 8b 43 20 48 3b 45 c8 ba 00 00 00 00 4c 8b 63 30 48 0f 44 c2 4c 3b 65 c0 4c 0f 44 e0 4d 85 e4 75 04 <0f> 0b eb fe 49 83 ec 28 49 8d 44 24 40 48 89 45 c8 f0 41 ff 44 [ 7913.096183] RIP [<ffffffffa04670ef>] btrfs_queue_worker+0x10e/0x236 [btrfs] [ 7913.096183] RSP <ffff88019fc03c10>
Quoted 12 lines · Tsutomu Itoh
Hi,
In kernel 3.8-rc5, the following panics occurred when the mount was done by the degraded option.
# btrfs fi sh /dev/sdc8 Label: none uuid: fc63cd80-5ae2-4fbe-8795-2d526c937a56 Total devices 3 FS bytes used 20.98GB devid 1 size 9.31GB used 9.31GB path /dev/sdd8 devid 2 size 9.31GB used 9.31GB path /dev/sdc8 *** Some devices missing
Quoted 6 lines
Btrfs v0.20-rc1-37-g91d9eec
# mount -o degraded /dev/sdc8 /test1564 static struct
btrfs_worker_thread*find_worker(structbtrfs_workers*workers) 565 {
...
I'm new at this so just taking a guess, but maybe a patch below. :)
Hm, so we can't get here unless:
worker = next_worker(workers);
returned NULL. And it can't return NULL unless idle_list is empty, and we are not at the maximum nr. of threads, or the current worker list is empty.
So it's possible to return NULL from next_worker() if idle_list and worker_list are both empty, I think.
Quoted 9 lines
...
595 fallback:
596 fallback = NULL;
597 /*598 * we have failed to find any workers, just 599 * return the first one we can find. 600 */601 if (!list_empty(&workers->worker_list))
602 fallback =workers->worker_list.next;
it's possible that we got here *because* the worker_list was empty...
Quoted 1 line
603 if (!list_empty(&workers->idle_list))
... and that when we were called, this list was empty too.
Quoted 2 lines
604 fallback =
workers->idle_list.next;
605BUG_ON(!fallback); <---------------------- this !
Seems quite possible that there are no worker threads at all at this point. How could that happen...
Quoted 10 lines
606 worker = list_entry(fallback,
607 structbtrfs_worker_thread,worker_list);-Tsutomu
===================================================================================
[ 7913.075890] btrfs: allowing degraded mounts [ 7913.075893] btrfs: disk space caching is enabled [ 7913.092031] Btrfs: too many missing devices, writeable mount is not allowed
so this was supposed to fail the mount in open_ctree; it jumps to shutting down the worker threads. Which might result in no threads available.
Quoted 7 lines
[ 7913.092297] ------------[ cut here ]------------ [ 7913.092313] kernel BUG at fs/btrfs/async-thread.c:605! [ 7913.092326] invalid opcode: 0000 [#1] SMP [ 7913.092342] Modules linked in: btrfs zlib_deflate crc32c libcrc32c nfsd lockd nfs_acl auth_rpcgss sunrpc 8021q garp stp llc cpufreq_ondemand cachefiles fscache ipv6 ext3 jbd dm_mirror dm_region_hash dm_log dm_mod uinput ppdev iTCO_wdt iTCO_vendor_support parport_pc parport sg acpi_cpufreq freq_table mperf coretemp kvm pcspkr i2c_i801 i2c_core lpc_ich mfd_core tg3 ptp pps_core shpchp pci_hotplug i3000_edac edac_core ext4 mbcache jbd2 crc16 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_piix libata megaraid_sas scsi_mod floppy [last unloaded: microcode] [ 7913.092575] CPU 0 [ 7913.092584] Pid: 3673, comm: btrfs-endio-wri Not tainted 3.8.0-rc5 #1 FUJITSU-SV PRIMERGY /D2399 [ 7913.092608] RIP: 0010:[<ffffffffa04670ef>] [<ffffffffa04670ef>] btrfs_queue_worker+0x10e/0x236 [btrfs]
but this is already trying to do work, and has no workers to handle it.
The place we jump to is fail_block_groups, and before it is this comment:
/*
* make sure we're done with the btree inode before we stop our
* kthreads
*/
filemap_write_and_wait(fs_info->btree_inode->i_mapping);
invalidate_inode_pages2(fs_info->btree_inode->i_mapping);fail_block_groups:
btrfs_free_block_groups(fs_info);
if you move the fail_block_groups: target above the comment, does that fix it? (although I don't know yet what started IO . . . )
like this:
From: Eric Sandeen <[email protected]>
Make sure that we are always done with the btree_inode's mapping before we shut down the worker threads in open_ctree() error cases.
fs/btrfs/disk-io.c +1 -1
@@ -2689,6 +2689,7 @@ fail_trans_kthread: fail_cleaner:kthread_stop (fs_info->cleaner_kthread); +fail_block_groups:/* * make sure we're done with the btree inode before we stop our * kthreads
@@ -2696,7 +2697,6 @@ fail_cleaner:filemap_write_and_wait (fs_info->btree_inode->i_mapping);invalidate_inode_pages2 (fs_info->btree_inode->i_mapping); -fail_block_groups:btrfs_free_block_groups (fs_info); fail_tree_roots:
Just a guess; but I don't know what would have started writes already...
-Eric
Quoted 4 lines
[ 7913.092663] RSP: 0018:ffff88019fc03c10 EFLAGS: 00010046 [ 7913.092676] RAX: 0000000000000000 RBX: ffff8801967b8a58 RCX: 0000000000000000 [ 7913.092894] RDX: 0000000000000000 RSI: ffff8801961239b8 RDI: ffff8801967b8ab8 [ 7913.093116] RBP: ffff88019fc03c50 R08: 0000000000000000 R09: ffff880198801180
Quoted 8 lines
[ 7913.093247] R10: ffffffffa045fda7 R11: 0000000000000003 R12: 0000000000000000 [ 7913.093247] R13: ffff8801961239b8 R14: ffff8801967b8ab8 R15: 0000000000000246 [ 7913.093247] FS: 0000000000000000(0000) GS:ffff88019fc00000(0000) knlGS:0000000000000000 [ 7913.093247] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 7913.093247] CR2: ffffffffff600400 CR3: 000000019575d000 CR4: 00000000000007f0 [ 7913.093247] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7913.093247] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7913.093247] Process btrfs-endio-wri (pid: 3673, threadinfo ffff8801939ca000, task ffff880195795b00)
This was started by
btrfs_init_workers(&fs_info->endio_write_workers, "endio-write",
fs_info->thread_pool_size,
&fs_info->generic_worker);via open_ctree() before we jumped to fail_block_groups.
Quoted 47 lines
[ 7913.093247] Stack: [ 7913.093247] ffff8801967b8a88 ffff8801967b8a78 ffff88003fa0a600 ffff8801965ad0c0 [ 7913.093247] ffff88003fa0a600 0000000000000000 0000000000000000 0000000000000000 [ 7913.096183] ffff88019fc03c60 ffffffffa043e357 ffff88019fc03c70 ffffffff811526aa [ 7913.096183] Call Trace: [ 7913.096183] <IRQ> [ 7913.096183] [ 7913.096183] [<ffffffffa043e357>] end_workqueue_bio+0x79/0x7b [btrfs] [ 7913.096183] [<ffffffff811526aa>] bio_endio+0x2d/0x2f [ 7913.096183] [<ffffffffa045fdb2>] btrfs_end_bio+0x10b/0x122 [btrfs] [ 7913.096183] [<ffffffff811526aa>] bio_endio+0x2d/0x2f [ 7913.096183] [<ffffffff811c5e3f>] req_bio_endio+0x96/0x9f [ 7913.096183] [<ffffffff811c601d>] blk_update_request+0x1d5/0x3a4 [ 7913.096183] [<ffffffff811c620c>] blk_update_bidi_request+0x20/0x6f [ 7913.096183] [<ffffffff811c7a59>] blk_end_bidi_request+0x1f/0x5d [ 7913.096183] [<ffffffff811c7ad3>] blk_end_request+0x10/0x12 [ 7913.096183] [<ffffffffa001db50>] scsi_io_completion+0x207/0x4f3 [scsi_mod] [ 7913.096183] [<ffffffffa0016df9>] scsi_finish_command+0xec/0xf5 [scsi_mod] [ 7913.096183] [<ffffffffa001df50>] scsi_softirq_done+0xff/0x108 [scsi_mod] [ 7913.096183] [<ffffffff811ccb3a>] blk_done_softirq+0x7a/0x8e [ 7913.096183] [<ffffffff810475c3>] __do_softirq+0xd7/0x1ed [ 7913.096183] [<ffffffff813ead9c>] call_softirq+0x1c/0x30 [ 7913.096183] [<ffffffff81010ab6>] do_softirq+0x46/0x83 [ 7913.096183] [<ffffffff81047363>] irq_exit+0x49/0xb7 [ 7913.096183] [<ffffffff813eafd5>] do_IRQ+0x9d/0xb4 [ 7913.096183] [<ffffffffa0467217>] ? btrfs_queue_worker+0x236/0x236 [btrfs] [ 7913.096183] [<ffffffff813e2a2d>] common_interrupt+0x6d/0x6d [ 7913.096183] <EOI> [ 7913.096183] [ 7913.096183] [<ffffffff81069a5e>] ? sched_move_task+0x12e/0x13d [ 7913.096183] [<ffffffff8104a131>] ? ptrace_put_breakpoints+0x1/0x1e [ 7913.096183] [<ffffffff81044fc2>] ? do_exit+0x3d7/0x89d [ 7913.096183] [<ffffffffa0467217>] ? btrfs_queue_worker+0x236/0x236 [btrfs] [ 7913.096183] [<ffffffffa0467217>] ? btrfs_queue_worker+0x236/0x236 [btrfs] [ 7913.096183] [<ffffffff8105ca20>] kthread+0xbd/0xbd [ 7913.096183] [<ffffffff8105c963>] ? kthread_freezable_should_stop+0x65/0x65 [ 7913.096183] [<ffffffff813e99ec>] ret_from_fork+0x7c/0xb0 [ 7913.096183] [<ffffffff8105c963>] ? kthread_freezable_should_stop+0x65/0x65 [ 7913.096183] Code: 49 89 c7 0f 84 5f ff ff ff 48 8b 43 20 48 3b 45 c8 ba 00 00 00 00 4c 8b 63 30 48 0f 44 c2 4c 3b 65 c0 4c 0f 44 e0 4d 85 e4 75 04 <0f> 0b eb fe 49 83 ec 28 49 8d 44 24 40 48 89 45 c8 f0 41 ff 44 [ 7913.096183] RIP [<ffffffffa04670ef>] btrfs_queue_worker+0x10e/0x236 [btrfs] [ 7913.096183] RSP <ffff88019fc03c10>--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 28 lines · Eric Sandeen
Quoted 5 lines
===================================================================================
[ 7913.075890] btrfs: allowing degraded mounts [ 7913.075893] btrfs: disk space caching is enabled [ 7913.092031] Btrfs: too many missing devices, writeable mount is not allowedso this was supposed to fail the mount in
open_ctree; it jumps to shutting down the worker threads. Which might result in no threads available.Quoted 7 lines
[ 7913.092297] ------------[ cut here ]------------ [ 7913.092313] kernel BUG at fs/btrfs/async-thread.c:605! [ 7913.092326] invalid opcode: 0000 [#1] SMP [ 7913.092342] Modules linked in: btrfs zlib_deflate crc32c libcrc32c nfsd lockd nfs_acl auth_rpcgss sunrpc 8021q garp stp llc cpufreq_ondemand cachefiles fscache ipv6 ext3 jbd dm_mirror dm_region_hash dm_log dm_mod uinput ppdev iTCO_wdt iTCO_vendor_support parport_pc parport sg acpi_cpufreq freq_table mperf coretemp kvm pcspkr i2c_i801 i2c_core lpc_ich mfd_core tg3 ptp pps_core shpchp pci_hotplug i3000_edac edac_core ext4 mbcache jbd2 crc16 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_piix libata megaraid_sas scsi_mod floppy [last unloaded: microcode] [ 7913.092575] CPU 0 [ 7913.092584] Pid: 3673, comm: btrfs-endio-wri Not tainted 3.8.0-rc5 #1 FUJITSU-SV PRIMERGY /D2399 [ 7913.092608] RIP: 0010:[<ffffffffa04670ef>] [<ffffffffa04670ef>] btrfs_queue_worker+0x10e/0x236 [btrfs]but this is already trying to do work, and has no workers to handle it.
The place we jump to is
fail_block_groups, and before it is this comment:/* * make sure we're done with the btree inode before we stop our * kthreads */ filemap_write_and_wait(fs_info->btree_inode->i_mapping); invalidate_inode_pages2(fs_info->btree_inode->i_mapping);
Quoted 5 lines
fail_block_groups:btrfs_free_block_groups(fs_info);if you move the
fail_block_groups: target above the comment, does that fix it? (although I don't know yet what started IO . . . )
Reading the metadata of the tree root and Reading block group information started IO. so, I think this patch can fix the problem.
Quoted 31 lines
like this:
From: Eric Sandeen <[email protected]>
Make sure that we are always done with the btree_inode's mapping before we shut down the worker threads in
open_ctree()error cases.Signed-off-by Eric Sandeen <[email protected]>fs/btrfs/disk-io.c +1
-1@@ -2689,6 +2689,7 @@ fail_trans_kthread: fail_cleaner:kthread_stop (fs_info->cleaner_kthread); +fail_block_groups:/* * make sure we're done with the btree inode before we stop our * kthreads @@ -2696,7 +2697,6 @@ fail_cleaner:filemap_write_and_wait (fs_info->btree_inode->i_mapping);invalidate_inode_pages2 (fs_info->btree_inode->i_mapping); -fail_block_groups:btrfs_free_block_groups (fs_info); fail_tree_roots:
Quoted 1 line
Just a guess; but I don't know what would have started writes already...
I don't think it was write IO. It was just a soft interrupt caused by a metadata read IO, and this soft interrupt happened while btrfs-endio-write-workers was going to stop.
Thanks
Miao
Quoted 10 lines · Eric Sandeen
if you move the
fail_block_groups: target above the comment, does that fix it? (although I don't know yet what started IO . . . )like this:
From: Eric Sandeen <[email protected]>
Make sure that we are always done with the btree_inode's mapping before we shut down the worker threads in
open_ctree()error cases.
I reviewed your patch again, and found it just fix the above problem, it still have similar problems which are not fixed.
How about this one?
fs/btrfs/disk-io.c +1 -2
@@ -2728,13 +2728,13 @@ fail_cleaner:* kthreads */ filemap_write_and_wait (fs_info->btree_inode->i_mapping); -invalidate_inode_pages2 (fs_info->btree_inode->i_mapping); fail_block_groups:btrfs_free_block_groups (fs_info); fail_tree_roots:free_root_pointers (fs_info,1 ); +invalidate_inode_pages2 (fs_info->btree_inode->i_mapping); fail_sb_buffer:btrfs_stop_workers (&fs_info->generic_worker);
@@ -2755,7 +2755,6 @@ fail_alloc: fail_iput:btrfs_mapping_tree_free (&fs_info->mapping_tree); -invalidate_inode_pages2 (fs_info->btree_inode->i_mapping);iput (fs_info->btree_inode); fail_bdi:bdi_destroy (&fs_info->bdi);
Quoted 8 lines · Miao Xie
Quoted 7 lines · Eric Sandeen
if you move the
fail_block_groups: target above the comment, does that fix it? (although I don't know yet what started IO . . . )like this:
From: Eric Sandeen <[email protected]>
Quoted 7 lines
Quoted 3 lines
Make sure that we are always done with the btree_inode's mapping before we shut down the worker threads in
open_ctree()error cases.I reviewed your patch again, and found it just fix the above problem, it still have similar problems which are not fixed.
Can you explain the similar problems you found?
(Also, the reason I thought a write had been started was because the original panic was in comm: btrfs-endio-wr[iter])
Thanks,
-Eric
Quoted 30 lines
How about this one?
fs/btrfs/disk-io.c +1
-2@@ -2728,13 +2728,13 @@ fail_cleaner:* kthreads */ filemap_write_and_wait (fs_info->btree_inode->i_mapping); -invalidate_inode_pages2 (fs_info->btree_inode->i_mapping); fail_block_groups:btrfs_free_block_groups (fs_info); fail_tree_roots:free_root_pointers (fs_info,1 ); +invalidate_inode_pages2 (fs_info->btree_inode->i_mapping); fail_sb_buffer:btrfs_stop_workers (&fs_info->generic_worker);@@ -2755,7 +2755,6 @@ fail_alloc: fail_iput:btrfs_mapping_tree_free (&fs_info->mapping_tree); -invalidate_inode_pages2 (fs_info->btree_inode->i_mapping);iput (fs_info->btree_inode); fail_bdi:bdi_destroy (&fs_info->bdi);
Hi,
Quoted 13 lines · Miao Xie
Quoted 10 lines · Eric Sandeen
if you move the
fail_block_groups: target above the comment, does that fix it? (although I don't know yet what started IO . . . )like this:
From: Eric Sandeen <[email protected]>
Make sure that we are always done with the btree_inode's mapping before we shut down the worker threads in
open_ctree()error cases.
Quoted 4 lines
I reviewed your patch again, and found it just fix the above problem, it still have similar problems which are not fixed.
How about this one?
Thanks Eric and Miao.
But I can not reproduce this problem, yet.
('Btrfs: too many missing devices, writeable mount is not allowed' messages was
displayed, but not panic)
So, I can not test your patch, sorry.
Can you please explain similar problems, Miao?
Thanks,
Tsutomu
Quoted 35 lines
fs/btrfs/disk-io.c +1
-3@@ -2728,13 +2728,13 @@ fail_cleaner:* kthreads */ filemap_write_and_wait (fs_info->btree_inode->i_mapping); -invalidate_inode_pages2 (fs_info->btree_inode->i_mapping); fail_block_groups:btrfs_free_block_groups (fs_info); fail_tree_roots:free_root_pointers (fs_info,1 ); +invalidate_inode_pages2 (fs_info->btree_inode->i_mapping); fail_sb_buffer:btrfs_stop_workers (&fs_info->generic_worker);@@ -2755,7 +2755,6 @@ fail_alloc: fail_iput:btrfs_mapping_tree_free (&fs_info->mapping_tree); -invalidate_inode_pages2 (fs_info->btree_inode->i_mapping);iput (fs_info->btree_inode); fail_bdi:bdi_destroy (&fs_info->bdi); --To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 21 lines · Tsutomu Itoh
Hi,
Quoted 13 lines · Miao Xie
Quoted 10 lines · Eric Sandeen
if you move the
fail_block_groups: target above the comment, does that fix it? (although I don't know yet what started IO . . . )like this:
From: Eric Sandeen <[email protected]>
Make sure that we are always done with the btree_inode's mapping before we shut down the worker threads in
open_ctree()error cases.Quoted 4 lines
I reviewed your patch again, and found it just fix the above problem, it still have similar problems which are not fixed.
How about this one?
Quoted 7 lines
Thanks Eric and Miao.
But I can not reproduce this problem, yet.
('Btrfs: too many missing devices, writeable mount is not allowed' messages was
displayed, but not panic)
So, I can not test your patch, sorry.Can you please explain similar problems, Miao?
Before missing device check, there are several places where we read the metadata, such as reading chunk tree root, btrfs_read_chunk_tree, those functions may fail after submit a bio. If we don't wait until the bio end, and just stop the workers, the same problem will happen.
(invalidate_inode_pages2() will wait until the bio end, because it need lock the pages
which are going to be invalidated, and the page is locked if it is under disk read IO)
Thanks
Miao
Quoted 42 lines
Thanks,
TsutomuQuoted 35 lines
fs/btrfs/disk-io.c +1
-3@@ -2728,13 +2728,13 @@ fail_cleaner:* kthreads */ filemap_write_and_wait (fs_info->btree_inode->i_mapping); -invalidate_inode_pages2 (fs_info->btree_inode->i_mapping); fail_block_groups:btrfs_free_block_groups (fs_info); fail_tree_roots:free_root_pointers (fs_info,1 ); +invalidate_inode_pages2 (fs_info->btree_inode->i_mapping); fail_sb_buffer:btrfs_stop_workers (&fs_info->generic_worker);@@ -2755,7 +2755,6 @@ fail_alloc: fail_iput:btrfs_mapping_tree_free (&fs_info->mapping_tree); -invalidate_inode_pages2 (fs_info->btree_inode->i_mapping);iput (fs_info->btree_inode); fail_bdi:bdi_destroy (&fs_info->bdi); --To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 30 lines · Miao Xie
Quoted 21 lines · Tsutomu Itoh
Hi,
Quoted 13 lines · Miao Xie
Quoted 10 lines · Eric Sandeen
if you move the
fail_block_groups: target above the comment, does that fix it? (although I don't know yet what started IO . . . )like this:
From: Eric Sandeen <[email protected]>
Make sure that we are always done with the btree_inode's mapping before we shut down the worker threads in
open_ctree()error cases.Quoted 4 lines
I reviewed your patch again, and found it just fix the above problem, it still have similar problems which are not fixed.
How about this one?
Quoted 7 lines
Thanks Eric and Miao.
But I can not reproduce this problem, yet.
('Btrfs: too many missing devices, writeable mount is not allowed' messages was
displayed, but not panic)
So, I can not test your patch, sorry.Can you please explain similar problems, Miao?
Quoted 7 lines
Before missing device check, there are several places where we read the metadata, such as reading chunk tree root,
btrfs_read_chunk_tree, those functions may fail after submit a bio. If we don't wait until the bio end, and just stop the workers, the same problem will happen.(
invalidate_inode_pages2()will wait until the bio end, because it need lock the pages
which are going to be invalidated, and the page is locked if it is under disk read IO)
I understood.
My reproducer is not reproduce this problem yet. But the following messages were displayed when 'rmmod btrfs' command was executed.
[76378.723481] ============================================================================= [76378.723901] BUG btrfs_extent_buffer (Tainted: G B ): Objects remaining in btrfs_extent_buffer on kmem_cache_close() [76378.724333] ----------------------------------------------------------------------------- [76378.724333] [76378.724959] INFO: Slab 0xffffea00065c3280 objects=23 used=2 fp=0xffff8801970caac0 flags=0x8000000000004080 [76378.725391] Pid: 9156, comm: rmmod Tainted: G B 3.8.0-rc5 #1 [76378.725397] Call Trace: [76378.725403] [<ffffffff8111bc23>] slab_err+0xb0/0xd2
I think that this message means there is a possibility that I/O did not end normally.
and, after Miao's patch applied, this message is not displayed when rmmod was executed.
So, Miao's patch seems to fix the problem for me.
Thanks,
Tsutomu
Quoted 37 lines
Thanks
MiaoQuoted 33 lines
Thanks,
TsutomuQuoted 29 lines
fs/btrfs/disk-io.c +1
-2@@ -2728,13 +2728,13 @@ fail_cleaner:* kthreads */ filemap_write_and_wait (fs_info->btree_inode->i_mapping); -invalidate_inode_pages2 (fs_info->btree_inode->i_mapping); fail_block_groups:btrfs_free_block_groups (fs_info); fail_tree_roots:free_root_pointers (fs_info,1 ); +invalidate_inode_pages2 (fs_info->btree_inode->i_mapping); fail_sb_buffer:btrfs_stop_workers (&fs_info->generic_worker);@@ -2755,7 +2755,6 @@ fail_alloc: fail_iput:btrfs_mapping_tree_free (&fs_info->mapping_tree); -invalidate_inode_pages2 (fs_info->btree_inode->i_mapping);iput (fs_info->btree_inode); fail_bdi:bdi_destroy (&fs_info->bdi);
Hi, Eric
I want to send out my fix patch, but Could I add your Signed-off-by? because you found the key to solving the problem.
Thanks
Miao
Quoted 30 lines · Tsutomu Itoh
Quoted 9 lines
Quoted 1 line
Can you please explain similar problems, Miao?
Before missing device check, there are several places where we read the metadata, such as reading chunk tree root,
btrfs_read_chunk_tree, those functions may fail after submit a bio. If we don't wait until the bio end, and just stop the workers, the same problem will happen.(
invalidate_inode_pages2()will wait until the bio end, because it need lock the pages
which are going to be invalidated, and the page is locked if it is under disk read IO)I understood.
My reproducer is not reproduce this problem yet. But the following messages were displayed when 'rmmod btrfs' command was executed.
[76378.723481] ============================================================================= [76378.723901] BUG btrfs_extent_buffer (Tainted: G B ): Objects remaining in btrfs_extent_buffer on kmem_cache_close() [76378.724333] ----------------------------------------------------------------------------- [76378.724333] [76378.724959] INFO: Slab 0xffffea00065c3280 objects=23 used=2 fp=0xffff8801970caac0 flags=0x8000000000004080 [76378.725391] Pid: 9156, comm: rmmod Tainted: G B 3.8.0-rc5 #1 [76378.725397] Call Trace: [76378.725403] [<ffffffff8111bc23>] slab_err+0xb0/0xd2I think that this message means there is a possibility that I/O did not end normally.
and, after Miao's patch applied, this message is not displayed when rmmod was executed.So, Miao's patch seems to fix the problem for me.
[SNIP]
Quoted 31 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
fs/btrfs/disk-io.c +1
-2@@ -2728,13 +2728,13 @@ fail_cleaner:* kthreads */ filemap_write_and_wait (fs_info->btree_inode->i_mapping); -invalidate_inode_pages2 (fs_info->btree_inode->i_mapping); fail_block_groups:btrfs_free_block_groups (fs_info); fail_tree_roots:free_root_pointers (fs_info,1 ); +invalidate_inode_pages2 (fs_info->btree_inode->i_mapping); fail_sb_buffer:btrfs_stop_workers (&fs_info->generic_worker);@@ -2755,7 +2755,6 @@ fail_alloc: fail_iput:btrfs_mapping_tree_free (&fs_info->mapping_tree); -invalidate_inode_pages2 (fs_info->btree_inode->i_mapping);iput (fs_info->btree_inode); fail_bdi:bdi_destroy (&fs_info->bdi);
Quoted 4 lines · Miao Xie
Hi, Eric
I want to send out my fix patch, but Could I add your Signed-off-by? because you found the key to solving the problem.
I don't know if a signed-off-by chain is the right approach, but
don't worry about it. You can mention my first patch in the changelog if you like.
Thanks,
-Eric
Quoted 72 lines
Thanks
MiaoQuoted 30 lines · Tsutomu Itoh
Quoted 9 lines
Quoted 1 line
Can you please explain similar problems, Miao?
Before missing device check, there are several places where we read the metadata, such as reading chunk tree root,
btrfs_read_chunk_tree, those functions may fail after submit a bio. If we don't wait until the bio end, and just stop the workers, the same problem will happen.(
invalidate_inode_pages2()will wait until the bio end, because it need lock the pages
which are going to be invalidated, and the page is locked if it is under disk read IO)I understood.
My reproducer is not reproduce this problem yet. But the following messages were displayed when 'rmmod btrfs' command was executed.
[76378.723481] ============================================================================= [76378.723901] BUG btrfs_extent_buffer (Tainted: G B ): Objects remaining in btrfs_extent_buffer on kmem_cache_close() [76378.724333] ----------------------------------------------------------------------------- [76378.724333] [76378.724959] INFO: Slab 0xffffea00065c3280 objects=23 used=2 fp=0xffff8801970caac0 flags=0x8000000000004080 [76378.725391] Pid: 9156, comm: rmmod Tainted: G B 3.8.0-rc5 #1 [76378.725397] Call Trace: [76378.725403] [<ffffffff8111bc23>] slab_err+0xb0/0xd2I think that this message means there is a possibility that I/O did not end normally.
and, after Miao's patch applied, this message is not displayed when rmmod was executed.So, Miao's patch seems to fix the problem for me.
[SNIP]
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
fs/btrfs/disk-io.c +1
-1@@ -2728,13 +2728,13 @@ fail_cleaner:* kthreads */ filemap_write_and_wait (fs_info->btree_inode->i_mapping); -invalidate_inode_pages2 (fs_info->btree_inode->i_mapping); fail_block_groups:btrfs_free_block_groups (fs_info); fail_tree_roots:free_root_pointers (fs_info,1 ); +invalidate_inode_pages2 (fs_info->btree_inode->i_mapping); fail_sb_buffer:btrfs_stop_workers (&fs_info->generic_worker);@@ -2755,7 +2755,6 @@ fail_alloc: fail_iput:btrfs_mapping_tree_free (&fs_info->mapping_tree);Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
fs/btrfs/disk-io.c fail_alloc: · line 2760
Quoted 5 lines
fs/btrfs/disk-io.c +0
-1-invalidate_inode_pages2 (fs_info->btree_inode->i_mapping);iput (fs_info->btree_inode); fail_bdi:bdi_destroy (&fs_info->bdi);--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 25 lines · Tsutomu Itoh
Hi,
In kernel 3.8-rc5, the following panics occurred when the mount was done by the degraded option.
# btrfs fi sh /dev/sdc8 Label: none uuid: fc63cd80-5ae2-4fbe-8795-2d526c937a56 Total devices 3 FS bytes used 20.98GB devid 1 size 9.31GB used 9.31GB path /dev/sdd8 devid 2 size 9.31GB used 9.31GB path /dev/sdc8 *** Some devices missingBtrfs v0.20-rc1-37-g91d9eec
# mount -o degraded /dev/sdc8 /test1564 static struct
btrfs_worker_thread*find_worker(structbtrfs_workers*workers) 565 {
...
...
595 fallback:
596 fallback = NULL;
597 /*598 * we have failed to find any workers, just 599 * return the first one we can find. 600 */
Quoted 7 lines
601 if (!list_empty(&workers->worker_list))
602 fallback =workers->worker_list.next;
603 if (!list_empty(&workers->idle_list))
604 fallback =workers->idle_list.next;
605BUG_ON(!fallback); <---------------------- this !
606 worker = list_entry(fallback,
607 structbtrfs_worker_thread,worker_list);
If worker_list is not empty, we get a worker from this list; if worker_list is empty, it means all the workers in idle_list, we get the worker from idle_list.
So the above bug is introduced by the second if sentence. it should be "else if".
Thanks
Miao
Quoted 74 lines
-Tsutomu
===================================================================================
[ 7913.075890] btrfs: allowing degraded mounts [ 7913.075893] btrfs: disk space caching is enabled [ 7913.092031] Btrfs: too many missing devices, writeable mount is not allowed [ 7913.092297] ------------[ cut here ]------------ [ 7913.092313] kernel BUG at fs/btrfs/async-thread.c:605! [ 7913.092326] invalid opcode: 0000 [#1] SMP [ 7913.092342] Modules linked in: btrfs zlib_deflate crc32c libcrc32c nfsd lockd nfs_acl auth_rpcgss sunrpc 8021q garp stp llc cpufreq_ondemand cachefiles fscache ipv6 ext3 jbd dm_mirror dm_region_hash dm_log dm_mod uinput ppdev iTCO_wdt iTCO_vendor_support parport_pc parport sg acpi_cpufreq freq_table mperf coretemp kvm pcspkr i2c_i801 i2c_core lpc_ich mfd_core tg3 ptp pps_core shpchp pci_hotplug i3000_edac edac_core ext4 mbcache jbd2 crc16 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_piix libata megaraid_sas scsi_mod floppy [last unloaded: microcode] [ 7913.092575] CPU 0 [ 7913.092584] Pid: 3673, comm: btrfs-endio-wri Not tainted 3.8.0-rc5 #1 FUJITSU-SV PRIMERGY /D2399 [ 7913.092608] RIP: 0010:[<ffffffffa04670ef>] [<ffffffffa04670ef>] btrfs_queue_worker+0x10e/0x236 [btrfs] [ 7913.092663] RSP: 0018:ffff88019fc03c10 EFLAGS: 00010046 [ 7913.092676] RAX: 0000000000000000 RBX: ffff8801967b8a58 RCX: 0000000000000000 [ 7913.092894] RDX: 0000000000000000 RSI: ffff8801961239b8 RDI: ffff8801967b8ab8 [ 7913.093116] RBP: ffff88019fc03c50 R08: 0000000000000000 R09: ffff880198801180 [ 7913.093247] R10: ffffffffa045fda7 R11: 0000000000000003 R12: 0000000000000000 [ 7913.093247] R13: ffff8801961239b8 R14: ffff8801967b8ab8 R15: 0000000000000246 [ 7913.093247] FS: 0000000000000000(0000) GS:ffff88019fc00000(0000) knlGS:0000000000000000 [ 7913.093247] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 7913.093247] CR2: ffffffffff600400 CR3: 000000019575d000 CR4: 00000000000007f0 [ 7913.093247] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7913.093247] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7913.093247] Process btrfs-endio-wri (pid: 3673, threadinfo ffff8801939ca000, task ffff880195795b00) [ 7913.093247] Stack: [ 7913.093247] ffff8801967b8a88 ffff8801967b8a78 ffff88003fa0a600 ffff8801965ad0c0 [ 7913.093247] ffff88003fa0a600 0000000000000000 0000000000000000 0000000000000000 [ 7913.096183] ffff88019fc03c60 ffffffffa043e357 ffff88019fc03c70 ffffffff811526aa [ 7913.096183] Call Trace: [ 7913.096183] <IRQ> [ 7913.096183] [ 7913.096183] [<ffffffffa043e357>] end_workqueue_bio+0x79/0x7b [btrfs] [ 7913.096183] [<ffffffff811526aa>] bio_endio+0x2d/0x2f [ 7913.096183] [<ffffffffa045fdb2>] btrfs_end_bio+0x10b/0x122 [btrfs] [ 7913.096183] [<ffffffff811526aa>] bio_endio+0x2d/0x2f [ 7913.096183] [<ffffffff811c5e3f>] req_bio_endio+0x96/0x9f [ 7913.096183] [<ffffffff811c601d>] blk_update_request+0x1d5/0x3a4 [ 7913.096183] [<ffffffff811c620c>] blk_update_bidi_request+0x20/0x6f [ 7913.096183] [<ffffffff811c7a59>] blk_end_bidi_request+0x1f/0x5d [ 7913.096183] [<ffffffff811c7ad3>] blk_end_request+0x10/0x12 [ 7913.096183] [<ffffffffa001db50>] scsi_io_completion+0x207/0x4f3 [scsi_mod] [ 7913.096183] [<ffffffffa0016df9>] scsi_finish_command+0xec/0xf5 [scsi_mod] [ 7913.096183] [<ffffffffa001df50>] scsi_softirq_done+0xff/0x108 [scsi_mod] [ 7913.096183] [<ffffffff811ccb3a>] blk_done_softirq+0x7a/0x8e [ 7913.096183] [<ffffffff810475c3>] __do_softirq+0xd7/0x1ed [ 7913.096183] [<ffffffff813ead9c>] call_softirq+0x1c/0x30 [ 7913.096183] [<ffffffff81010ab6>] do_softirq+0x46/0x83 [ 7913.096183] [<ffffffff81047363>] irq_exit+0x49/0xb7 [ 7913.096183] [<ffffffff813eafd5>] do_IRQ+0x9d/0xb4 [ 7913.096183] [<ffffffffa0467217>] ? btrfs_queue_worker+0x236/0x236 [btrfs] [ 7913.096183] [<ffffffff813e2a2d>] common_interrupt+0x6d/0x6d [ 7913.096183] <EOI> [ 7913.096183] [ 7913.096183] [<ffffffff81069a5e>] ? sched_move_task+0x12e/0x13d [ 7913.096183] [<ffffffff8104a131>] ? ptrace_put_breakpoints+0x1/0x1e [ 7913.096183] [<ffffffff81044fc2>] ? do_exit+0x3d7/0x89d [ 7913.096183] [<ffffffffa0467217>] ? btrfs_queue_worker+0x236/0x236 [btrfs] [ 7913.096183] [<ffffffffa0467217>] ? btrfs_queue_worker+0x236/0x236 [btrfs] [ 7913.096183] [<ffffffff8105ca20>] kthread+0xbd/0xbd [ 7913.096183] [<ffffffff8105c963>] ? kthread_freezable_should_stop+0x65/0x65 [ 7913.096183] [<ffffffff813e99ec>] ret_from_fork+0x7c/0xb0 [ 7913.096183] [<ffffffff8105c963>] ? kthread_freezable_should_stop+0x65/0x65 [ 7913.096183] Code: 49 89 c7 0f 84 5f ff ff ff 48 8b 43 20 48 3b 45 c8 ba 00 00 00 00 4c 8b 63 30 48 0f 44 c2 4c 3b 65 c0 4c 0f 44 e0 4d 85 e4 75 04 <0f> 0b eb fe 49 83 ec 28 49 8d 44 24 40 48 89 45 c8 f0 41 ff 44 [ 7913.096183] RIP [<ffffffffa04670ef>] btrfs_queue_worker+0x10e/0x236 [btrfs] [ 7913.096183] RSP <ffff88019fc03c10>--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 35 lines · Miao Xie
Quoted 25 lines · Tsutomu Itoh
Hi,
In kernel 3.8-rc5, the following panics occurred when the mount was done by the degraded option.
# btrfs fi sh /dev/sdc8 Label: none uuid: fc63cd80-5ae2-4fbe-8795-2d526c937a56 Total devices 3 FS bytes used 20.98GB devid 1 size 9.31GB used 9.31GB path /dev/sdd8 devid 2 size 9.31GB used 9.31GB path /dev/sdc8 *** Some devices missingBtrfs v0.20-rc1-37-g91d9eec
# mount -o degraded /dev/sdc8 /test1564 static struct
btrfs_worker_thread*find_worker(structbtrfs_workers*workers) 565 {
...
...
595 fallback:
596 fallback = NULL;
597 /*598 * we have failed to find any workers, just 599 * return the first one we can find. 600 */Quoted 7 lines
601 if (!list_empty(&workers->worker_list))
602 fallback =workers->worker_list.next;
603 if (!list_empty(&workers->idle_list))
604 fallback =workers->idle_list.next;
605BUG_ON(!fallback); <---------------------- this !
606 worker = list_entry(fallback,
607 structbtrfs_worker_thread,worker_list);
Quoted 4 lines
If
worker_listis not empty, we get a worker from this list; ifworker_listis empty, it means all the workers inidle_list, we get the worker fromidle_list.So the above bug is introduced by the second if sentence. it should be "else if".
else if makes sense, but we cannot reach the BUG_ON unless both lists are empty, correct?
-Eric
Quoted 81 lines
Thanks
MiaoQuoted 69 lines
-Tsutomu
===================================================================================
[ 7913.075890] btrfs: allowing degraded mounts [ 7913.075893] btrfs: disk space caching is enabled [ 7913.092031] Btrfs: too many missing devices, writeable mount is not allowed [ 7913.092297] ------------[ cut here ]------------ [ 7913.092313] kernel BUG at fs/btrfs/async-thread.c:605! [ 7913.092326] invalid opcode: 0000 [#1] SMP [ 7913.092342] Modules linked in: btrfs zlib_deflate crc32c libcrc32c nfsd lockd nfs_acl auth_rpcgss sunrpc 8021q garp stp llc cpufreq_ondemand cachefiles fscache ipv6 ext3 jbd dm_mirror dm_region_hash dm_log dm_mod uinput ppdev iTCO_wdt iTCO_vendor_support parport_pc parport sg acpi_cpufreq freq_table mperf coretemp kvm pcspkr i2c_i801 i2c_core lpc_ich mfd_core tg3 ptp pps_core shpchp pci_hotplug i3000_edac edac_core ext4 mbcache jbd2 crc16 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_piix libata megaraid_sas scsi_mod floppy [last unloaded: microcode] [ 7913.092575] CPU 0 [ 7913.092584] Pid: 3673, comm: btrfs-endio-wri Not tainted 3.8.0-rc5 #1 FUJITSU-SV PRIMERGY /D2399 [ 7913.092608] RIP: 0010:[<ffffffffa04670ef>] [<ffffffffa04670ef>] btrfs_queue_worker+0x10e/0x236 [btrfs] [ 7913.092663] RSP: 0018:ffff88019fc03c10 EFLAGS: 00010046 [ 7913.092676] RAX: 0000000000000000 RBX: ffff8801967b8a58 RCX: 0000000000000000 [ 7913.092894] RDX: 0000000000000000 RSI: ffff8801961239b8 RDI: ffff8801967b8ab8 [ 7913.093116] RBP: ffff88019fc03c50 R08: 0000000000000000 R09: ffff880198801180 [ 7913.093247] R10: ffffffffa045fda7 R11: 0000000000000003 R12: 0000000000000000 [ 7913.093247] R13: ffff8801961239b8 R14: ffff8801967b8ab8 R15: 0000000000000246 [ 7913.093247] FS: 0000000000000000(0000) GS:ffff88019fc00000(0000) knlGS:0000000000000000 [ 7913.093247] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 7913.093247] CR2: ffffffffff600400 CR3: 000000019575d000 CR4: 00000000000007f0 [ 7913.093247] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7913.093247] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7913.093247] Process btrfs-endio-wri (pid: 3673, threadinfo ffff8801939ca000, task ffff880195795b00) [ 7913.093247] Stack: [ 7913.093247] ffff8801967b8a88 ffff8801967b8a78 ffff88003fa0a600 ffff8801965ad0c0 [ 7913.093247] ffff88003fa0a600 0000000000000000 0000000000000000 0000000000000000 [ 7913.096183] ffff88019fc03c60 ffffffffa043e357 ffff88019fc03c70 ffffffff811526aa [ 7913.096183] Call Trace: [ 7913.096183] <IRQ> [ 7913.096183] [ 7913.096183] [<ffffffffa043e357>] end_workqueue_bio+0x79/0x7b [btrfs] [ 7913.096183] [<ffffffff811526aa>] bio_endio+0x2d/0x2f [ 7913.096183] [<ffffffffa045fdb2>] btrfs_end_bio+0x10b/0x122 [btrfs] [ 7913.096183] [<ffffffff811526aa>] bio_endio+0x2d/0x2f [ 7913.096183] [<ffffffff811c5e3f>] req_bio_endio+0x96/0x9f [ 7913.096183] [<ffffffff811c601d>] blk_update_request+0x1d5/0x3a4 [ 7913.096183] [<ffffffff811c620c>] blk_update_bidi_request+0x20/0x6f [ 7913.096183] [<ffffffff811c7a59>] blk_end_bidi_request+0x1f/0x5d [ 7913.096183] [<ffffffff811c7ad3>] blk_end_request+0x10/0x12 [ 7913.096183] [<ffffffffa001db50>] scsi_io_completion+0x207/0x4f3 [scsi_mod] [ 7913.096183] [<ffffffffa0016df9>] scsi_finish_command+0xec/0xf5 [scsi_mod] [ 7913.096183] [<ffffffffa001df50>] scsi_softirq_done+0xff/0x108 [scsi_mod] [ 7913.096183] [<ffffffff811ccb3a>] blk_done_softirq+0x7a/0x8e [ 7913.096183] [<ffffffff810475c3>] __do_softirq+0xd7/0x1ed [ 7913.096183] [<ffffffff813ead9c>] call_softirq+0x1c/0x30 [ 7913.096183] [<ffffffff81010ab6>] do_softirq+0x46/0x83 [ 7913.096183] [<ffffffff81047363>] irq_exit+0x49/0xb7 [ 7913.096183] [<ffffffff813eafd5>] do_IRQ+0x9d/0xb4 [ 7913.096183] [<ffffffffa0467217>] ? btrfs_queue_worker+0x236/0x236 [btrfs] [ 7913.096183] [<ffffffff813e2a2d>] common_interrupt+0x6d/0x6d [ 7913.096183] <EOI> [ 7913.096183] [ 7913.096183] [<ffffffff81069a5e>] ? sched_move_task+0x12e/0x13d [ 7913.096183] [<ffffffff8104a131>] ? ptrace_put_breakpoints+0x1/0x1e [ 7913.096183] [<ffffffff81044fc2>] ? do_exit+0x3d7/0x89d [ 7913.096183] [<ffffffffa0467217>] ? btrfs_queue_worker+0x236/0x236 [btrfs] [ 7913.096183] [<ffffffffa0467217>] ? btrfs_queue_worker+0x236/0x236 [btrfs] [ 7913.096183] [<ffffffff8105ca20>] kthread+0xbd/0xbd [ 7913.096183] [<ffffffff8105c963>] ? kthread_freezable_should_stop+0x65/0x65 [ 7913.096183] [<ffffffff813e99ec>] ret_from_fork+0x7c/0xb0 [ 7913.096183] [<ffffffff8105c963>] ? kthread_freezable_should_stop+0x65/0x65 [ 7913.096183] Code: 49 89 c7 0f 84 5f ff ff ff 48 8b 43 20 48 3b 45 c8 ba 00 00 00 00 4c 8b 63 30 48 0f 44 c2 4c 3b 65 c0 4c 0f 44 e0 4d 85 e4 75 04 <0f> 0b eb fe 49 83 ec 28 49 8d 44 24 40 48 89 45 c8 f0 41 ff 44 [ 7913.096183] RIP [<ffffffffa04670ef>] btrfs_queue_worker+0x10e/0x236 [btrfs] [ 7913.096183] RSP <ffff88019fc03c10>Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 37 lines · Eric Sandeen
Quoted 35 lines · Miao Xie
Quoted 32 lines · Tsutomu Itoh
Hi,
In kernel 3.8-rc5, the following panics occurred when the mount was done by the degraded option.
# btrfs fi sh /dev/sdc8 Label: none uuid: fc63cd80-5ae2-4fbe-8795-2d526c937a56 Total devices 3 FS bytes used 20.98GB devid 1 size 9.31GB used 9.31GB path /dev/sdd8 devid 2 size 9.31GB used 9.31GB path /dev/sdc8 *** Some devices missingBtrfs v0.20-rc1-37-g91d9eec
# mount -o degraded /dev/sdc8 /test1564 static struct
btrfs_worker_thread*find_worker(structbtrfs_workers*workers) 565 {
...
...
595 fallback:
596 fallback = NULL;
597 /*598 * we have failed to find any workers, just 599 * return the first one we can find. 600 */601 if (!list_empty(&workers->worker_list))
602 fallback =workers->worker_list.next;
603 if (!list_empty(&workers->idle_list))
604 fallback =workers->idle_list.next;
605BUG_ON(!fallback); <---------------------- this !
606 worker = list_entry(fallback,
607 structbtrfs_worker_thread,worker_list);
Quoted 6 lines
Quoted 4 lines
If
worker_listis not empty, we get a worker from this list; ifworker_listis empty, it means all the workers inidle_list, we get the worker fromidle_list.So the above bug is introduced by the second if sentence. it should be "else if".
else if makes sense, but we cannot reach the
BUG_ONunless both lists are empty, correct?
You are right, I misread the code.
Thanks
Miao
Quoted 85 lines
-Eric
Quoted 81 lines
Thanks
MiaoQuoted 69 lines
-Tsutomu
===================================================================================
[ 7913.075890] btrfs: allowing degraded mounts [ 7913.075893] btrfs: disk space caching is enabled [ 7913.092031] Btrfs: too many missing devices, writeable mount is not allowed [ 7913.092297] ------------[ cut here ]------------ [ 7913.092313] kernel BUG at fs/btrfs/async-thread.c:605! [ 7913.092326] invalid opcode: 0000 [#1] SMP [ 7913.092342] Modules linked in: btrfs zlib_deflate crc32c libcrc32c nfsd lockd nfs_acl auth_rpcgss sunrpc 8021q garp stp llc cpufreq_ondemand cachefiles fscache ipv6 ext3 jbd dm_mirror dm_region_hash dm_log dm_mod uinput ppdev iTCO_wdt iTCO_vendor_support parport_pc parport sg acpi_cpufreq freq_table mperf coretemp kvm pcspkr i2c_i801 i2c_core lpc_ich mfd_core tg3 ptp pps_core shpchp pci_hotplug i3000_edac edac_core ext4 mbcache jbd2 crc16 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_piix libata megaraid_sas scsi_mod floppy [last unloaded: microcode] [ 7913.092575] CPU 0 [ 7913.092584] Pid: 3673, comm: btrfs-endio-wri Not tainted 3.8.0-rc5 #1 FUJITSU-SV PRIMERGY /D2399 [ 7913.092608] RIP: 0010:[<ffffffffa04670ef>] [<ffffffffa04670ef>] btrfs_queue_worker+0x10e/0x236 [btrfs] [ 7913.092663] RSP: 0018:ffff88019fc03c10 EFLAGS: 00010046 [ 7913.092676] RAX: 0000000000000000 RBX: ffff8801967b8a58 RCX: 0000000000000000 [ 7913.092894] RDX: 0000000000000000 RSI: ffff8801961239b8 RDI: ffff8801967b8ab8 [ 7913.093116] RBP: ffff88019fc03c50 R08: 0000000000000000 R09: ffff880198801180 [ 7913.093247] R10: ffffffffa045fda7 R11: 0000000000000003 R12: 0000000000000000 [ 7913.093247] R13: ffff8801961239b8 R14: ffff8801967b8ab8 R15: 0000000000000246 [ 7913.093247] FS: 0000000000000000(0000) GS:ffff88019fc00000(0000) knlGS:0000000000000000 [ 7913.093247] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 7913.093247] CR2: ffffffffff600400 CR3: 000000019575d000 CR4: 00000000000007f0 [ 7913.093247] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7913.093247] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 7913.093247] Process btrfs-endio-wri (pid: 3673, threadinfo ffff8801939ca000, task ffff880195795b00) [ 7913.093247] Stack: [ 7913.093247] ffff8801967b8a88 ffff8801967b8a78 ffff88003fa0a600 ffff8801965ad0c0 [ 7913.093247] ffff88003fa0a600 0000000000000000 0000000000000000 0000000000000000 [ 7913.096183] ffff88019fc03c60 ffffffffa043e357 ffff88019fc03c70 ffffffff811526aa [ 7913.096183] Call Trace: [ 7913.096183] <IRQ> [ 7913.096183] [ 7913.096183] [<ffffffffa043e357>] end_workqueue_bio+0x79/0x7b [btrfs] [ 7913.096183] [<ffffffff811526aa>] bio_endio+0x2d/0x2f [ 7913.096183] [<ffffffffa045fdb2>] btrfs_end_bio+0x10b/0x122 [btrfs] [ 7913.096183] [<ffffffff811526aa>] bio_endio+0x2d/0x2f [ 7913.096183] [<ffffffff811c5e3f>] req_bio_endio+0x96/0x9f [ 7913.096183] [<ffffffff811c601d>] blk_update_request+0x1d5/0x3a4 [ 7913.096183] [<ffffffff811c620c>] blk_update_bidi_request+0x20/0x6f [ 7913.096183] [<ffffffff811c7a59>] blk_end_bidi_request+0x1f/0x5d [ 7913.096183] [<ffffffff811c7ad3>] blk_end_request+0x10/0x12 [ 7913.096183] [<ffffffffa001db50>] scsi_io_completion+0x207/0x4f3 [scsi_mod] [ 7913.096183] [<ffffffffa0016df9>] scsi_finish_command+0xec/0xf5 [scsi_mod] [ 7913.096183] [<ffffffffa001df50>] scsi_softirq_done+0xff/0x108 [scsi_mod] [ 7913.096183] [<ffffffff811ccb3a>] blk_done_softirq+0x7a/0x8e [ 7913.096183] [<ffffffff810475c3>] __do_softirq+0xd7/0x1ed [ 7913.096183] [<ffffffff813ead9c>] call_softirq+0x1c/0x30 [ 7913.096183] [<ffffffff81010ab6>] do_softirq+0x46/0x83 [ 7913.096183] [<ffffffff81047363>] irq_exit+0x49/0xb7 [ 7913.096183] [<ffffffff813eafd5>] do_IRQ+0x9d/0xb4 [ 7913.096183] [<ffffffffa0467217>] ? btrfs_queue_worker+0x236/0x236 [btrfs] [ 7913.096183] [<ffffffff813e2a2d>] common_interrupt+0x6d/0x6d [ 7913.096183] <EOI> [ 7913.096183] [ 7913.096183] [<ffffffff81069a5e>] ? sched_move_task+0x12e/0x13d [ 7913.096183] [<ffffffff8104a131>] ? ptrace_put_breakpoints+0x1/0x1e [ 7913.096183] [<ffffffff81044fc2>] ? do_exit+0x3d7/0x89d [ 7913.096183] [<ffffffffa0467217>] ? btrfs_queue_worker+0x236/0x236 [btrfs] [ 7913.096183] [<ffffffffa0467217>] ? btrfs_queue_worker+0x236/0x236 [btrfs] [ 7913.096183] [<ffffffff8105ca20>] kthread+0xbd/0xbd [ 7913.096183] [<ffffffff8105c963>] ? kthread_freezable_should_stop+0x65/0x65 [ 7913.096183] [<ffffffff813e99ec>] ret_from_fork+0x7c/0xb0 [ 7913.096183] [<ffffffff8105c963>] ? kthread_freezable_should_stop+0x65/0x65 [ 7913.096183] Code: 49 89 c7 0f 84 5f ff ff ff 48 8b 43 20 48 3b 45 c8 ba 00 00 00 00 4c 8b 63 30 48 0f 44 c2 4c 3b 65 c0 4c 0f 44 e0 4d 85 e4 75 04 <0f> 0b eb fe 49 83 ec 28 49 8d 44 24 40 48 89 45 c8 f0 41 ff 44 [ 7913.096183] RIP [<ffffffffa04670ef>] btrfs_queue_worker+0x10e/0x236 [btrfs] [ 7913.096183] RSP <ffff88019fc03c10>Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html