CFQ Oops
I got an oops with CFQ (see below) while running the XFS QA test 133. I managed to bisect it down to commit a36e71f996e25d6213f57951f7ae1874086ec57e.
Quoted 1 line
From a quick glance at the code, I'd guess that prio_trees[..] happens to be
NULL, and so the rb_erase_init call on line 660 results in a NULL ptr deref.
657 if (!RB_EMPTY_NODE(&cfqq->rb_node))
658 cfq_rb_erase(&cfqq->rb_node, &cfqd->service_tree);
659 if (!RB_EMPTY_NODE(&cfqq->p_node))
660 rb_erase_init(&cfqq->p_node, &cfqd->prio_trees[cfqq->ioprio]);
661
662 BUG_ON(!cfqd->busy_queues);
663 cfqd->busy_queues--;
Josef 'Jeff' Sipek.
a36e71f996e25d6213f57951f7ae1874086ec57e is first bad commit commit a36e71f996e25d6213f57951f7ae1874086ec57e
Author: Jens Axboe <[email protected]> Date: Wed Apr 15 12:15:11 2009 +0200
cfq-iosched: add close cooperator code
If we have processes that are working in close proximity to each other on disk, we don't want to idle wait. Instead allow the close process to issue a request, getting better aggregate bandwidth. The anticipatory scheduler has similar checks, noop and deadline do not need it since they don't care about process <-> io mappings.
The code for CFQ is a little more involved though, since we split request queues into per-process contexts.
This fixes a performance problem with eg dump(8), since it uses several processes in some silly attempt to speed IO up. Even if dump(8) isn't really a valid case (it should be fixed by using CLONE_IO), there are other cases where we see close processes and where idling ends up hurting performance.
Credit goes to Jeff Moyer <[email protected]> for writing the initial implementation.
:040000 040000 2e905502bf3c466baae407bcd654fc36d015c83f b4aafd7edc811aed69fa44ddf00a29ece4f32a33 M block
BUG: unable to handle kernel NULL pointer dereference at (null) IP: [<ffffffff8119a594>] rb_erase+0x174/0x340 PGD f8843067 PUD f8842067 PMD 0 Oops: 0000 [#1] SMP last sysfs file: /sys/devices/pci0000:00/0000:00:0b.0/0000:01:03.0/local_cpus CPU 0 Modules linked in: xfs exportfs sco bridge stp llc bnep l2cap bluetooth
ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 dm_multipath uinput amd_rng ata_generic tg3 s2io libphy serio_raw i2c_amd8111 pcspkr i2c_amd756 pata_amd shpchp
Pid: 2926, comm: xfs_io Not tainted 2.6.30-rc3 #8 To be filled by O.E.M.
RIP: 0010:[<ffffffff8119a594>] [<ffffffff8119a594>] rb_erase+0x174/0x340 RSP: 0018:ffff8800788e9788 EFLAGS: 00010046 RAX: ffff8800f9dd12b1 RBX: ffff8800f9dd12b0 RCX: 0000000000000001 RDX: ffff8800f9dd12b0 RSI: 0000000000000000 RDI: 0000000000000000 RBP: ffff8800788e9798 R08: ffff8800f9dd1320 R09: 0000000000000000 R10: ffff8800f9dd12b0 R11: 0000000000000000 R12: ffff8800fa958a38 R13: ffff880074cc8e60 R14: ffff8800fa958a00 R15: ffff8800f9dd1320 FS: 00007f6262cc46f0(0000) GS:ffffc20000000000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000000 CR3: 00000000f5848000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process xfs_io (pid: 2926, threadinfo ffff8800788e8000, task ffff880074d0c4d0) Stack: ffff8800f9dd1320 ffff8800f9dd1350 ffff8800788e97d8 ffffffff81190a40 ffff8800fa958a00 ffff8800f9dd1320 ffff880074cc8e60 ffff8800fa958a00 ffff8800fa9208e0 ffff880074cc8e60 ffff8800788e9808 ffffffff81190b28 Call Trace: [<ffffffff81190a40>] cfq_remove_request+0x1e0/0x280 [<ffffffff81190b28>] cfq_dispatch_insert+0x48/0x90 [<ffffffff811917a5>] cfq_dispatch_requests+0x1a5/0x4d0 [<ffffffff81180842>] elv_next_request+0x162/0x200 [<ffffffff81196dba>] ? kobject_get+0x1a/0x30 [<ffffffff812156d2>] scsi_request_fn+0x62/0x560 [<ffffffff8138a5ff>] ? _spin_unlock_irqrestore+0x2f/0x40 [<ffffffff8118251a>] blk_start_queueing+0x1a/0x40 [<ffffffff81191f15>] cfq_insert_request+0x2d5/0x430 [<ffffffff81180a00>] elv_insert+0x120/0x2a0 [<ffffffff81180bfb>] __elv_add_request+0x7b/0xd0 [<ffffffff81183cf1>] __make_request+0x111/0x460 [<ffffffff81182bf5>] generic_make_request+0x3b5/0x490 [<ffffffff81182d40>] submit_bio+0x70/0xf0 [<ffffffff8110d86e>] dio_bio_submit+0x5e/0x90 [<ffffffff8110e215>] __blockdev_direct_IO+0x5c5/0xd60 [<ffffffffa019916e>] xfs_vm_direct_IO+0x10e/0x130 [xfs] [<ffffffffa01995a0>] ? xfs_get_blocks_direct+0x0/0x20 [xfs] [<ffffffffa01992f0>] ? xfs_end_io_direct+0x0/0x80 [xfs] [<ffffffff810a59ef>] generic_file_direct_write+0xbf/0x220 [<ffffffffa01a19bb>] xfs_write+0x3fb/0x9a0 [xfs] [<ffffffff810a4f6c>] ? filemap_fault+0x13c/0x440 [<ffffffffa019dafd>] xfs_file_aio_write+0x5d/0x70 [xfs] [<ffffffff810e16f1>] do_sync_write+0xf1/0x140 [<ffffffff8105b1d0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff8103cc7b>] ? finish_task_switch+0x5b/0xe0 [<ffffffffa0199190>] ? xfs_end_bio_written+0x0/0x30 [xfs] [<ffffffff81387ba0>] ? thread_return+0x3e/0x6ae [<ffffffff810e1ddb>] vfs_write+0xcb/0x190 [<ffffffff810e1f32>] sys_pwrite64+0x92/0xa0 [<ffffffff8100bf6b>] system_call_fastpath+0x16/0x1b Code: 7a ff ff ff 0f 1f 00 48 3b 78 10 0f 1f 40 00 0f 84 7a 01 00 00 48 89
48 08 66 0f 1f 44 00 00 e9 28 ff ff ff 0f 1f 00 48 8b 7b 08 <48> 8b 07 a8 01 0f 84 d1 00 00 00 48 8b 47 10 48 85 c0 74 09 f6
RIP [<ffffffff8119a594>] rb_erase+0x174/0x340
RSP <ffff8800788e9788>
CR2: 0000000000000000
---[ end trace 79edd488f3f3e8df ]---
Signature · 1 line
A computer without Microsoft is like chocolate cake without mustard.
Quoted 14 lines · Josef 'Jeff' Sipek
I got an oops with CFQ (see below) while running the XFS QA test 133. I managed to bisect it down to commit
a36e71f996e25d6213f57951f7ae1874086ec57e.From a quick glance at the code, I'd guess that prio_trees[..] happens to be NULL, and so the
rb_erase_initcall on line 660 results in a NULL ptr deref.657 if (!RB_EMPTY_NODE(&cfqq->rb_node))
658cfq_rb_erase(&cfqq->rb_node, &cfqd->service_tree);
659 if (!RB_EMPTY_NODE(&cfqq->p_node))
660rb_erase_init(&cfqq->p_node, &cfqd->prio_trees[cfqq->ioprio]);
661
662BUG_ON(!cfqd->busy_queues);
663 cfqd->busy_queues--;
Quoted 1 line
Josef 'Jeff' Sipek.
Yeah, it's a fixed issue, if only Linus would pull the patches. I guess he's away for a few days. If you pull:
git://git.kernel.dk/linux-2.6-block.git for-linus
into 2.6.30-rc3, then that should fix it.
Quoted 105 lines
a36e71f996e25d6213f57951f7ae1874086ec57eis first bad commit commita36e71f996e25d6213f57951f7ae1874086ec57eAuthor: Jens Axboe <[email protected]> Date: Wed Apr 15 12:15:11 2009 +0200cfq-iosched: add close cooperator code
If we have processes that are working in close proximity to each other on disk, we don't want to idle wait. Instead allow the close process to issue a request, getting better aggregate bandwidth. The anticipatory scheduler has similar checks, noop and deadline do not need it since they don't care about process <-> io mappings.
The code for CFQ is a little more involved though, since we split request queues into per-process contexts.
This fixes a performance problem with eg dump(8), since it uses several processes in some silly attempt to speed IO up. Even if dump(8) isn't really a valid case (it should be fixed by using
CLONE_IO), there are other cases where we see close processes and where idling ends up hurting performance.Credit goes to Jeff Moyer <[email protected]> for writing the initial implementation.
Signed-off-by Jens Axboe <[email protected]>:040000 040000
2e905502bf3c466baae407bcd654fc36d015c83fb4aafd7edc811aed69fa44ddf00a29ece4f32a33M blockBUG: unable to handle kernel NULL pointer dereference at (null) IP: [<ffffffff8119a594>] rb_erase+0x174/0x340 PGD f8843067 PUD f8842067 PMD 0 Oops: 0000 [#1] SMP last sysfs file: /sys/devices/pci0000:00/0000:00:0b.0/0000:01:03.0/local_cpus CPU 0 Modules linked in: xfs exportfs sco bridge stp llc bnep l2cap bluetooth
ip6t_REJECTnf_conntrack_ipv6ip6table_filterip6_tablesipv6dm_multipathuinputamd_rngata_generictg3 s2io libphyserio_rawi2c_amd8111pcspkri2c_amd756pata_amdshpchp
Pid: 2926, comm:xfs_ioNot tainted 2.6.30-rc3 #8 To be filled by O.E.M.RIP: 0010:[<ffffffff8119a594>] [<ffffffff8119a594>] rb_erase+0x174/0x340 RSP: 0018:ffff8800788e9788 EFLAGS: 00010046 RAX: ffff8800f9dd12b1 RBX: ffff8800f9dd12b0 RCX: 0000000000000001 RDX: ffff8800f9dd12b0 RSI: 0000000000000000 RDI: 0000000000000000 RBP: ffff8800788e9798 R08: ffff8800f9dd1320 R09: 0000000000000000 R10: ffff8800f9dd12b0 R11: 0000000000000000 R12: ffff8800fa958a38 R13: ffff880074cc8e60 R14: ffff8800fa958a00 R15: ffff8800f9dd1320 FS: 00007f6262cc46f0(0000) GS:ffffc20000000000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000000 CR3: 00000000f5848000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process xfs_io (pid: 2926, threadinfo ffff8800788e8000, task ffff880074d0c4d0) Stack: ffff8800f9dd1320 ffff8800f9dd1350 ffff8800788e97d8 ffffffff81190a40 ffff8800fa958a00 ffff8800f9dd1320 ffff880074cc8e60 ffff8800fa958a00 ffff8800fa9208e0 ffff880074cc8e60 ffff8800788e9808 ffffffff81190b28 Call Trace: [<ffffffff81190a40>] cfq_remove_request+0x1e0/0x280 [<ffffffff81190b28>] cfq_dispatch_insert+0x48/0x90 [<ffffffff811917a5>] cfq_dispatch_requests+0x1a5/0x4d0 [<ffffffff81180842>] elv_next_request+0x162/0x200 [<ffffffff81196dba>] ? kobject_get+0x1a/0x30 [<ffffffff812156d2>] scsi_request_fn+0x62/0x560 [<ffffffff8138a5ff>] ? _spin_unlock_irqrestore+0x2f/0x40 [<ffffffff8118251a>] blk_start_queueing+0x1a/0x40 [<ffffffff81191f15>] cfq_insert_request+0x2d5/0x430 [<ffffffff81180a00>] elv_insert+0x120/0x2a0 [<ffffffff81180bfb>] __elv_add_request+0x7b/0xd0 [<ffffffff81183cf1>] __make_request+0x111/0x460 [<ffffffff81182bf5>] generic_make_request+0x3b5/0x490 [<ffffffff81182d40>] submit_bio+0x70/0xf0 [<ffffffff8110d86e>] dio_bio_submit+0x5e/0x90 [<ffffffff8110e215>] __blockdev_direct_IO+0x5c5/0xd60 [<ffffffffa019916e>] xfs_vm_direct_IO+0x10e/0x130 [xfs] [<ffffffffa01995a0>] ? xfs_get_blocks_direct+0x0/0x20 [xfs] [<ffffffffa01992f0>] ? xfs_end_io_direct+0x0/0x80 [xfs] [<ffffffff810a59ef>] generic_file_direct_write+0xbf/0x220 [<ffffffffa01a19bb>] xfs_write+0x3fb/0x9a0 [xfs] [<ffffffff810a4f6c>] ? filemap_fault+0x13c/0x440 [<ffffffffa019dafd>] xfs_file_aio_write+0x5d/0x70 [xfs] [<ffffffff810e16f1>] do_sync_write+0xf1/0x140 [<ffffffff8105b1d0>] ? autoremove_wake_function+0x0/0x40 [<ffffffff8103cc7b>] ? finish_task_switch+0x5b/0xe0 [<ffffffffa0199190>] ? xfs_end_bio_written+0x0/0x30 [xfs] [<ffffffff81387ba0>] ? thread_return+0x3e/0x6ae [<ffffffff810e1ddb>] vfs_write+0xcb/0x190 [<ffffffff810e1f32>] sys_pwrite64+0x92/0xa0 [<ffffffff8100bf6b>] system_call_fastpath+0x16/0x1b Code: 7a ff ff ff 0f 1f 00 48 3b 78 10 0f 1f 40 00 0f 84 7a 01 00 00 48 8948 08 66 0f 1f 44 00 00 e9 28 ff ff ff 0f 1f 00 48 8b 7b 08 <48> 8b 07 a8 01 0f 84 d1 00 00 00 48 8b 47 10 48 85 c0 74 09 f6
RIP [<ffffffff8119a594>]rb_erase+0x174/0x340
RSP <ffff8800788e9788>
CR2: 0000000000000000---[ end trace 79edd488f3f3e8df ]---Signature · 1 line
A computer without Microsoft is like chocolate cake without mustard.
Signature · 1 line
Jens Axboe