Skip to contentSkip to conversation

core block bits for 2.6.37-rc1

GIT PULL Jens Axboe 5 participants 11 messages Oct 22, 2010 – Oct 24, 2010 Block layer

lore
Share & tools
lore

Jens Axboe

author
#0 lore raw

Hi Linus,

This first pull request is the core bits, meaning general block layer changes or core support. Should be clean this time, only 'weird bit' is the seemingly duplicate entry from Malahal. This is caused by the first patch being buggy (and later reverted), second patch used the same single line description.

Nothing really exciting in here. A good collection of fixes, some of which are marked for stable as well.

The biggest addition this time around is the block IO throttling support from Vivek.

Please pull.

git://git.kernel.dk/linux-2.6-block.git for-2.6.37/core

Christof Schmitt (1):
zfcp: Report scatter gather limit for DIX protection information

Corrado Zoccolo (1):
cfq: improve fsync performance for small files

Geert Uytterhoeven (1):
block: Turn bvec_k{un,}map_irq() into static inline functions

Jens Axboe (3):
core: match_dev_by_uuid() should not be marked __init
do_mounts: only enable PARTUUID for CONFIG_BLOCK
block: revert bad fix for memory hotplug causing bounces

Malahal Naineni (2):
block: set the bounce_pfn to the actual DMA limit rather than to max memory
block: set the bounce_pfn to the actual DMA limit rather than to max memory

Mark Lord (2):
block: Prevent hang_check firing during long I/O Fix compile error in blk-exec.c for !CONFIG_DETECT_HUNG_TASK

Martin K. Petersen (5):
Consolidate min_not_zero
block/scsi: Provide a limit on the number of integrity segments
block: Ensure physical block size is unsigned int
block: Fix double free in blk_integrity_unregister
block: Make the integrity mapped property a bio flag

Namhyung Kim (2):
block: fix an address space warning in blk-map.c
sg: fix a warning in blk_rq_aligned() call

San Mehat (1):
block: block_dump: Add number of sectors to debug output

Signed-off-by: Jan Kara (1):
block: Fix race during disk initialization

Vivek Goyal (16):
blk-cgroup: Kill the header printed at the start of blkio.weight_device file
blk-cgroup: Prepare the base for supporting more than one IO control policies
blk-cgroup: Introduce cgroup changes for throttling policy
blkio: Core implementation of throttle policy
blk-cgroup: cgroup changes for IOPS limit support
blkio: Implementation of IOPS limit logic
blkio: Documentation Update
blkio: Do not export throttle files if CONFIG_BLK_DEV_THROTTLING=n
blkio: deletion of a cgroup was causes oops
blkio: Add root group to td->tg_list
blkio: Recalculate the throttled bio dispatch time upon throttle limit change
blkio-throttle: Fix link failure failure on i386
blkio-throttle: There is no need to convert jiffies to milli seconds
blkio-throttle: limit max iops value to UINT_MAX
blkio-throttle: Fix possible multiplication overflow in iops calculations
cfq-iosched: Fix a gcc 4.5 warning and put some comments

Will Drewry (3):
block, partition: add partition_meta_info to hd_struct genhd, efi: add efi partition metadata to hd_structs
init: add support for root devices specified by partition UUID

Yasuaki Ishimatsu (1):
block: fix accounting bug on cross partition merges

 Documentation/cgroups/blkio-controller.txt |  106 +++-
 block/Kconfig                              |   12 +
 block/Makefile                             |    1 +
 block/blk-cgroup.c                         |  804 ++++++++++++++++----
 block/blk-cgroup.h                         |   87 ++-
 block/blk-core.c                           |   53 ++-
 block/blk-exec.c                           |    9 +-
 block/blk-integrity.c                      |   94 ++-
 block/blk-map.c                            |    5 +-
 block/blk-merge.c                          |   25 +-
 block/blk-settings.c                       |   12 +-
 block/blk-sysfs.c                          |   11 +
 block/blk-throttle.c                       | 1123 ++++++++++++++++++++++++++++
 block/blk.h                                |   12 -
 block/cfq-iosched.c                        |   39 +-
 block/cfq.h                                |    2 +-
 block/genhd.c                              |   30 +-
 block/ioctl.c                              |    2 +-
 drivers/block/drbd/drbd_receiver.c         |    1 -
 drivers/md/dm-snap.c                       |    2 -
 drivers/md/dm-table.c                      |    5 -
 drivers/s390/scsi/zfcp_scsi.c              |    1 +
 drivers/scsi/hosts.c                       |    1 +
 drivers/scsi/scsi_lib.c                    |   26 +-
 drivers/scsi/scsi_sysfs.c                  |    2 +
 drivers/scsi/sd_dif.c                      |   11 +-
 drivers/scsi/sg.c                          |    2 +-
 fs/jbd/commit.c                            |    2 +-
 fs/jbd2/commit.c                           |    2 +-
 fs/partitions/check.c                      |   35 +-
 fs/partitions/check.h                      |    3 +
 fs/partitions/efi.c                        |   25 +
 include/linux/bio.h                        |   15 +-
 include/linux/blk_types.h                  |    6 +-
 include/linux/blkdev.h                     |   66 ++-
 include/linux/elevator.h                   |    2 +
 include/linux/genhd.h                      |   54 ++-
 include/linux/kernel.h                     |   10 +
 include/linux/sched.h                      |    3 +
 include/scsi/scsi.h                        |    6 +
 include/scsi/scsi_host.h                   |    7 +
 init/Kconfig                               |    9 +-
 init/do_mounts.c                           |   70 ++
 43 files changed, 2494 insertions(+), 299 deletions(-)
 create mode 100644 block/blk-throttle.c
Signature · 1 line
Jens Axboe
#1 lore raw

Hi,

Quoted 8 lines · Jens Axboe

Hi Linus,

This first pull request is the core bits, meaning general block layer changes or core support. Should be clean this time, only 'weird bit' is the seemingly duplicate entry from Malahal. This is caused by the first patch being buggy (and later reverted), second patch used the same single line description.

Quoted 5 lines

Nothing really exciting in here. A good collection of fixes, some of which are marked for stable as well.

The biggest addition this time around is the block IO throttling support from Vivek.

The upstream block bits pulled in this merge window (or maybe the workqueue bits) are possibly the cause a boot crash on today's -tip, using a trivial x86 bootup test (64-bit allyesconfig):

[  116.064281] calling  hd_init+0x0/0x302 @ 1
[  116.068529] hd: no drives specified - use hd=cyl,head,sectors on kernel command line
[  116.076334] general protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC
[  116.080274] last sysfs file: 
[  116.080274] CPU 0 
[  116.080274] Modules linked in:
[  116.080274] 
[  116.080274] Pid: 1, comm: swapper Tainted: G        W   2.6.36-tip-03555-g825d9ec-dirty #51843 A8N-E/System Product Name
[  116.080274] RIP: 0010:[<ffffffff81064380>]  [<ffffffff81064380>] __ticket_spin_trylock+0x4/0x21
[  116.080274] RSP: 0018:ffff88003c417c10  EFLAGS: 00010082
[  116.080274] RAX: ffff88003c418000 RBX: 6b6b6b6b6b6b6b6a RCX: 0000000000000000
[  116.080274] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6b6a
[  116.080274] RBP: ffff88003c417c10 R08: 0000000000000002 R09: 0000000000000001
[  116.080274] R10: 0000000000000286 R11: ffff880032498738 R12: 6b6b6b6b6b6b6b82
[  116.080274] R13: 0000000000000286 R14: 6b6b6b6b6b6b6b6b R15: 0000000000000001
[  116.080274] FS:  0000000000000000(0000) GS:ffff88003e200000(0000) knlGS:0000000000000000
[  116.080274] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  116.080274] CR2: 0000000000000000 CR3: 0000000004071000 CR4: 00000000000006f0
[  116.080274] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  116.080274] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  116.080274] Process swapper (pid: 1, threadinfo ffff88003c416000, task ffff88003c418000)
[  116.080274] Stack:
[  116.080274]  ffff88003c417c30 ffffffff8168c6ee 6b6b6b6b6b6b6b6b 6b6b6b6b6b6b6b6a
[  116.080274] <0> ffff88003c417c70 ffffffff82d37a20 ffffffff810a1b65 ffff88003c418000
[  116.080274] <0> ffffffff82d3836b 6b6b6b6b6b6b6b6a ffff8800330fcc20 ffff88003c417cb8
[  116.080274] Call Trace:
[  116.080274]  [<ffffffff8168c6ee>] do_raw_spin_trylock+0x1f/0x41
[  116.080274]  [<ffffffff82d37a20>] _raw_spin_lock_irqsave+0x72/0xa4
[  116.080274]  [<ffffffff810a1b65>] ? lock_timer_base+0x2c/0x52
[  116.080274]  [<ffffffff82d3836b>] ? _raw_spin_unlock_irqrestore+0x55/0x72
[  116.080274]  [<ffffffff810a1b65>] lock_timer_base+0x2c/0x52
[  116.080274]  [<ffffffff810a1c43>] del_timer+0x2f/0x82
[  116.080274]  [<ffffffff810ac906>] ? wait_on_work+0x0/0xdb
[  116.080274]  [<ffffffff810aca18>] __cancel_work_timer+0x37/0x130
[  116.080274]  [<ffffffff810acb23>] cancel_delayed_work_sync+0x12/0x14
[  116.080274]  [<ffffffff8166974a>] throtl_shutdown_timer_wq+0x1c/0x1e
[  116.080274]  [<ffffffff8165dbec>] blk_sync_queue+0x3d/0x41
[  116.080274]  [<ffffffff8165f872>] blk_release_queue+0x1e/0x6a
[  116.080274]  [<ffffffff81673ce3>] kobject_release+0xf4/0x122
[  116.080274]  [<ffffffff81673bef>] ? kobject_release+0x0/0x122
[  116.080274]  [<ffffffff81674e7e>] kref_put+0x43/0x4d
[  116.080274]  [<ffffffff81673b46>] kobject_put+0x47/0x4c
[  116.080274]  [<ffffffff8165dc53>] blk_cleanup_queue+0x63/0x68
[  116.080274]  [<ffffffff84883c80>] ? hd_init+0x0/0x302
[  116.080274]  [<ffffffff84883f54>] hd_init+0x2d4/0x302
[  116.080274]  [<ffffffff81910778>] ? device_pm_unlock+0x15/0x17
[  116.080274]  [<ffffffff84883c80>] ? hd_init+0x0/0x302
[  116.080274]  [<ffffffff81002062>] do_one_initcall+0x57/0x15a
[  116.080274]  [<ffffffff8482f78b>] kernel_init+0x194/0x222
[  116.080274]  [<ffffffff8103ad04>] kernel_thread_helper+0x4/0x10
[  116.080274]  [<ffffffff82d38710>] ? restore_args+0x0/0x30
[  116.080274]  [<ffffffff8482f5f7>] ? kernel_init+0x0/0x222
[  116.080274]  [<ffffffff8103ad00>] ? kernel_thread_helper+0x0/0x10
[  116.080274] Code: ff ff c9 c3 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 0f b7 17 eb f5 c9 c3 55 48 89 e5 <8b> 07 89 c2 c1 c0 10 39 c2 8d 90 00 00 01 00 75 04 f0 0f b1 17 
[  116.080274] RIP  [<ffffffff81064380>] __ticket_spin_trylock+0x4/0x21
[  116.080274]  RSP <ffff88003c417c10>
[  116.080274] ---[ end trace e8df42e772bf6fed ]---
[  116.080274] Kernel panic - not syncing: Fatal exception
[  116.080274] Pid: 1, comm: swapper Tainted: G      D W   2.6.36-tip-03555-g825d9ec-dirty #51843
[  116.080274] Call Trace:
[  116.080274]  [<ffffffff82d34d9c>] panic+0x91/0x1b7
[  116.080274]  [<ffffffff81094c93>] ? kmsg_dump+0x18d/0x1a7
[  116.080274]  [<ffffffff82d38364>] ? _raw_spin_unlock_irqrestore+0x4e/0x72
[  116.080274]  [<ffffffff82d396af>] oops_end+0xd8/0xe8
[  116.080274]  [<ffffffff8103d6fd>] die+0x5a/0x63
[  116.080274]  [<ffffffff82d3924f>] do_general_protection+0x12a/0x132
[  116.080274]  [<ffffffff82d38740>] ? irq_return+0x0/0x10
[  116.080274]  [<ffffffff82d38965>] general_protection+0x25/0x30
[  116.080274]  [<ffffffff81064380>] ? __ticket_spin_trylock+0x4/0x21
[  116.080274]  [<ffffffff8168c6ee>] do_raw_spin_trylock+0x1f/0x41
[  116.080274]  [<ffffffff82d37a20>] _raw_spin_lock_irqsave+0x72/0xa4
[  116.080274]  [<ffffffff810a1b65>] ? lock_timer_base+0x2c/0x52
[  116.080274]  [<ffffffff82d3836b>] ? _raw_spin_unlock_irqrestore+0x55/0x72
[  116.080274]  [<ffffffff810a1b65>] lock_timer_base+0x2c/0x52
[  116.080274]  [<ffffffff810a1c43>] del_timer+0x2f/0x82
[  116.080274]  [<ffffffff810ac906>] ? wait_on_work+0x0/0xdb
[  116.080274]  [<ffffffff810aca18>] __cancel_work_timer+0x37/0x130
[  116.080274]  [<ffffffff810acb23>] cancel_delayed_work_sync+0x12/0x14
[  116.080274]  [<ffffffff8166974a>] throtl_shutdown_timer_wq+0x1c/0x1e
[  116.080274]  [<ffffffff8165dbec>] blk_sync_queue+0x3d/0x41
[  116.080274]  [<ffffffff8165f872>] blk_release_queue+0x1e/0x6a
[  116.080274]  [<ffffffff81673ce3>] kobject_release+0xf4/0x122
[  116.080274]  [<ffffffff81673bef>] ? kobject_release+0x0/0x122
[  116.080274]  [<ffffffff81674e7e>] kref_put+0x43/0x4d
[  116.080274]  [<ffffffff81673b46>] kobject_put+0x47/0x4c
[  116.080274]  [<ffffffff8165dc53>] blk_cleanup_queue+0x63/0x68
[  116.080274]  [<ffffffff84883c80>] ? hd_init+0x0/0x302
[  116.080274]  [<ffffffff84883f54>] hd_init+0x2d4/0x302
[  116.080274]  [<ffffffff81910778>] ? device_pm_unlock+0x15/0x17
[  116.080274]  [<ffffffff84883c80>] ? hd_init+0x0/0x302
[  116.080274]  [<ffffffff81002062>] do_one_initcall+0x57/0x15a
[  116.080274]  [<ffffffff8482f78b>] kernel_init+0x194/0x222
[  116.080274]  [<ffffffff8103ad04>] kernel_thread_helper+0x4/0x10
[  116.080274]  [<ffffffff82d38710>] ? restore_args+0x0/0x30
[  116.080274]  [<ffffffff8482f5f7>] ? kernel_init+0x0/0x222
[  116.080274]  [<ffffffff8103ad00>] ? kernel_thread_helper+0x0/0x10

(Note, the taint is there because there are a few other (unrelated and harmless) warnings in the bootup.)

Previous -tip testing narrows the regression down to between d4429f6 and ab34c02.

Going back to d4429f6 it boots fine.

I've also Cc:-ed Tejun as workqueue bits were pulled in that commit range as well and the crash is also in the workqueue code.

Thanks,

Ingo

#2 lore raw
Quoted 9 lines · Ingo Molnar

[  116.080274]
[  116.080274] Pid: 1, comm: swapper Tainted: G        W   2.6.36-tip-03555-g825d9ec-dirty #51843 A8N-E/System Product Name [  116.080274] RIP: 0010:[<ffffffff81064380>]  [<ffffffff81064380>] __ticket_spin_trylock+0x4/0x21
[  116.080274] RSP: 0018:ffff88003c417c10  EFLAGS: 00010082
[  116.080274] RAX: ffff88003c418000 RBX: 6b6b6b6b6b6b6b6a RCX: 0000000000000000
[  116.080274] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6b6a
[  116.080274] RBP: ffff88003c417c10 R08: 0000000000000002 R09: 0000000000000001
[  116.080274] R10: 0000000000000286 R11: ffff880032498738 R12: 6b6b6b6b6b6b6b82

And we obviously have that "6b" pattern for a use-after free with slab poisoning. Jens, have you tried with slab debugging?

Linus

#3 lore raw
In reply to Ingo Molnar
Quoted 7 lines · Ingo Molnar
[  116.080274]  [<ffffffff8165dc53>] blk_cleanup_queue+0x63/0x68
[  116.080274]  [<ffffffff84883c80>] ? hd_init+0x0/0x302
[  116.080274]  [<ffffffff84883f54>] hd_init+0x2d4/0x302
[  116.080274]  [<ffffffff81910778>] ? device_pm_unlock+0x15/0x17
[  116.080274]  [<ffffffff84883c80>] ? hd_init+0x0/0x302
[  116.080274]  [<ffffffff81002062>] do_one_initcall+0x57/0x15a
[  116.080274]  [<ffffffff8482f78b>] kernel_init+0x194/0x222

Btw., another data point, the crash goes away when the ancient XT-HD driver is turned off:

# CONFIG_BLK_DEV_HD is not set

I'm not sure whether the bug is limited to this driver alone though.

Ingo

author
#4 lore raw
In reply to Ingo Molnar
Quoted 69 lines · Ingo Molnar

Hi,

Quoted 8 lines · Jens Axboe

Hi Linus,

This first pull request is the core bits, meaning general block layer changes or core support. Should be clean this time, only 'weird bit' is the seemingly duplicate entry from Malahal. This is caused by the first patch being buggy (and later reverted), second patch used the same single line description.

Quoted 5 lines

Nothing really exciting in here. A good collection of fixes, some of which are marked for stable as well.

The biggest addition this time around is the block IO throttling support from Vivek.

The upstream block bits pulled in this merge window (or maybe the workqueue bits) are possibly the cause a boot crash on today's -tip, using a trivial x86 bootup test (64-bit allyesconfig):

[  116.064281] calling  hd_init+0x0/0x302 @ 1
[  116.068529] hd: no drives specified - use hd=cyl,head,sectors on kernel command line
[  116.076334] general protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC
[  116.080274] last sysfs file: 
[  116.080274] CPU 0 
[  116.080274] Modules linked in:
[  116.080274] 
[  116.080274] Pid: 1, comm: swapper Tainted: G        W   2.6.36-tip-03555-g825d9ec-dirty #51843 A8N-E/System Product Name
[  116.080274] RIP: 0010:[<ffffffff81064380>]  [<ffffffff81064380>] __ticket_spin_trylock+0x4/0x21
[  116.080274] RSP: 0018:ffff88003c417c10  EFLAGS: 00010082
[  116.080274] RAX: ffff88003c418000 RBX: 6b6b6b6b6b6b6b6a RCX: 0000000000000000
[  116.080274] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6b6a
[  116.080274] RBP: ffff88003c417c10 R08: 0000000000000002 R09: 0000000000000001
[  116.080274] R10: 0000000000000286 R11: ffff880032498738 R12: 6b6b6b6b6b6b6b82
[  116.080274] R13: 0000000000000286 R14: 6b6b6b6b6b6b6b6b R15: 0000000000000001
[  116.080274] FS:  0000000000000000(0000) GS:ffff88003e200000(0000) knlGS:0000000000000000
[  116.080274] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  116.080274] CR2: 0000000000000000 CR3: 0000000004071000 CR4: 00000000000006f0
[  116.080274] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  116.080274] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  116.080274] Process swapper (pid: 1, threadinfo ffff88003c416000, task ffff88003c418000)
[  116.080274] Stack:
[  116.080274]  ffff88003c417c30 ffffffff8168c6ee 6b6b6b6b6b6b6b6b 6b6b6b6b6b6b6b6a
[  116.080274] <0> ffff88003c417c70 ffffffff82d37a20 ffffffff810a1b65 ffff88003c418000
[  116.080274] <0> ffffffff82d3836b 6b6b6b6b6b6b6b6a ffff8800330fcc20 ffff88003c417cb8
[  116.080274] Call Trace:
[  116.080274]  [<ffffffff8168c6ee>] do_raw_spin_trylock+0x1f/0x41
[  116.080274]  [<ffffffff82d37a20>] _raw_spin_lock_irqsave+0x72/0xa4
[  116.080274]  [<ffffffff810a1b65>] ? lock_timer_base+0x2c/0x52
[  116.080274]  [<ffffffff82d3836b>] ? _raw_spin_unlock_irqrestore+0x55/0x72
[  116.080274]  [<ffffffff810a1b65>] lock_timer_base+0x2c/0x52
[  116.080274]  [<ffffffff810a1c43>] del_timer+0x2f/0x82
[  116.080274]  [<ffffffff810ac906>] ? wait_on_work+0x0/0xdb
[  116.080274]  [<ffffffff810aca18>] __cancel_work_timer+0x37/0x130
[  116.080274]  [<ffffffff810acb23>] cancel_delayed_work_sync+0x12/0x14
[  116.080274]  [<ffffffff8166974a>] throtl_shutdown_timer_wq+0x1c/0x1e
[  116.080274]  [<ffffffff8165dbec>] blk_sync_queue+0x3d/0x41
[  116.080274]  [<ffffffff8165f872>] blk_release_queue+0x1e/0x6a
[  116.080274]  [<ffffffff81673ce3>] kobject_release+0xf4/0x122
[  116.080274]  [<ffffffff81673bef>] ? kobject_release+0x0/0x122
[  116.080274]  [<ffffffff81674e7e>] kref_put+0x43/0x4d
[  116.080274]  [<ffffffff81673b46>] kobject_put+0x47/0x4c
[  116.080274]  [<ffffffff8165dc53>] blk_cleanup_queue+0x63/0x68
[  116.080274]  [<ffffffff84883c80>] ? hd_init+0x0/0x302
[  116.080274]  [<ffffffff84883f54>] hd_init+0x2d4/0x302
[  116.080274]  [<ffffffff81910778>] ? device_pm_unlock+0x15/0x17
Quoted 8 lines
[  116.080274]  [<ffffffff84883c80>] ? hd_init+0x0/0x302
[  116.080274]  [<ffffffff81002062>] do_one_initcall+0x57/0x15a
[  116.080274]  [<ffffffff8482f78b>] kernel_init+0x194/0x222
[  116.080274]  [<ffffffff8103ad04>] kernel_thread_helper+0x4/0x10
[  116.080274]  [<ffffffff82d38710>] ? restore_args+0x0/0x30
[  116.080274]  [<ffffffff8482f5f7>] ? kernel_init+0x0/0x222
[  116.080274]  [<ffffffff8103ad00>] ? kernel_thread_helper+0x0/0x10
[  116.080274] Code: ff ff c9 c3 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 0f b7 17 eb f5 c9 c3 55 48 89 e5 <8b> 07 89 c2 c1 c0 10 39 c2 8d 90 00 00 01 00 75 04 f0 0f b1 17 

Looks like a fairly straight forward case of uninitialized memory and blk_sync_queue() -> throtl_shutdown_timer() -> cancel_delayed_work_sync().

Will get that fixed up.

Signature · 1 line
Jens Axboe
author
#5 lore raw
Quoted 79 lines · Jens Axboe
Quoted 69 lines · Ingo Molnar

Hi,

Quoted 8 lines · Jens Axboe

Hi Linus,

This first pull request is the core bits, meaning general block layer changes or core support. Should be clean this time, only 'weird bit' is the seemingly duplicate entry from Malahal. This is caused by the first patch being buggy (and later reverted), second patch used the same single line description.

Quoted 5 lines

Nothing really exciting in here. A good collection of fixes, some of which are marked for stable as well.

The biggest addition this time around is the block IO throttling support from Vivek.

The upstream block bits pulled in this merge window (or maybe the workqueue bits) are possibly the cause a boot crash on today's -tip, using a trivial x86 bootup test (64-bit allyesconfig):

[  116.064281] calling  hd_init+0x0/0x302 @ 1
[  116.068529] hd: no drives specified - use hd=cyl,head,sectors on kernel command line
[  116.076334] general protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC
[  116.080274] last sysfs file: 
[  116.080274] CPU 0 
[  116.080274] Modules linked in:
[  116.080274] 
[  116.080274] Pid: 1, comm: swapper Tainted: G        W   2.6.36-tip-03555-g825d9ec-dirty #51843 A8N-E/System Product Name
[  116.080274] RIP: 0010:[<ffffffff81064380>]  [<ffffffff81064380>] __ticket_spin_trylock+0x4/0x21
[  116.080274] RSP: 0018:ffff88003c417c10  EFLAGS: 00010082
[  116.080274] RAX: ffff88003c418000 RBX: 6b6b6b6b6b6b6b6a RCX: 0000000000000000
[  116.080274] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6b6a
[  116.080274] RBP: ffff88003c417c10 R08: 0000000000000002 R09: 0000000000000001
[  116.080274] R10: 0000000000000286 R11: ffff880032498738 R12: 6b6b6b6b6b6b6b82
[  116.080274] R13: 0000000000000286 R14: 6b6b6b6b6b6b6b6b R15: 0000000000000001
[  116.080274] FS:  0000000000000000(0000) GS:ffff88003e200000(0000) knlGS:0000000000000000
[  116.080274] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  116.080274] CR2: 0000000000000000 CR3: 0000000004071000 CR4: 00000000000006f0
[  116.080274] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  116.080274] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  116.080274] Process swapper (pid: 1, threadinfo ffff88003c416000, task ffff88003c418000)
[  116.080274] Stack:
[  116.080274]  ffff88003c417c30 ffffffff8168c6ee 6b6b6b6b6b6b6b6b 6b6b6b6b6b6b6b6a
[  116.080274] <0> ffff88003c417c70 ffffffff82d37a20 ffffffff810a1b65 ffff88003c418000
[  116.080274] <0> ffffffff82d3836b 6b6b6b6b6b6b6b6a ffff8800330fcc20 ffff88003c417cb8
[  116.080274] Call Trace:
[  116.080274]  [<ffffffff8168c6ee>] do_raw_spin_trylock+0x1f/0x41
[  116.080274]  [<ffffffff82d37a20>] _raw_spin_lock_irqsave+0x72/0xa4
[  116.080274]  [<ffffffff810a1b65>] ? lock_timer_base+0x2c/0x52
[  116.080274]  [<ffffffff82d3836b>] ? _raw_spin_unlock_irqrestore+0x55/0x72
[  116.080274]  [<ffffffff810a1b65>] lock_timer_base+0x2c/0x52
[  116.080274]  [<ffffffff810a1c43>] del_timer+0x2f/0x82
[  116.080274]  [<ffffffff810ac906>] ? wait_on_work+0x0/0xdb
[  116.080274]  [<ffffffff810aca18>] __cancel_work_timer+0x37/0x130
[  116.080274]  [<ffffffff810acb23>] cancel_delayed_work_sync+0x12/0x14
[  116.080274]  [<ffffffff8166974a>] throtl_shutdown_timer_wq+0x1c/0x1e
[  116.080274]  [<ffffffff8165dbec>] blk_sync_queue+0x3d/0x41
[  116.080274]  [<ffffffff8165f872>] blk_release_queue+0x1e/0x6a
[  116.080274]  [<ffffffff81673ce3>] kobject_release+0xf4/0x122
[  116.080274]  [<ffffffff81673bef>] ? kobject_release+0x0/0x122
[  116.080274]  [<ffffffff81674e7e>] kref_put+0x43/0x4d
[  116.080274]  [<ffffffff81673b46>] kobject_put+0x47/0x4c
[  116.080274]  [<ffffffff8165dc53>] blk_cleanup_queue+0x63/0x68
[  116.080274]  [<ffffffff84883c80>] ? hd_init+0x0/0x302
[  116.080274]  [<ffffffff84883f54>] hd_init+0x2d4/0x302
[  116.080274]  [<ffffffff81910778>] ? device_pm_unlock+0x15/0x17
Quoted 8 lines
[  116.080274]  [<ffffffff84883c80>] ? hd_init+0x0/0x302
[  116.080274]  [<ffffffff81002062>] do_one_initcall+0x57/0x15a
[  116.080274]  [<ffffffff8482f78b>] kernel_init+0x194/0x222
[  116.080274]  [<ffffffff8103ad04>] kernel_thread_helper+0x4/0x10
[  116.080274]  [<ffffffff82d38710>] ? restore_args+0x0/0x30
[  116.080274]  [<ffffffff8482f5f7>] ? kernel_init+0x0/0x222
[  116.080274]  [<ffffffff8103ad00>] ? kernel_thread_helper+0x0/0x10
[  116.080274] Code: ff ff c9 c3 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 0f b7 17 eb f5 c9 c3 55 48 89 e5 <8b> 07 89 c2 c1 c0 10 39 c2 8d 90 00 00 01 00 75 04 f0 0f b1 17 
Quoted 5 lines

Looks like a fairly straight forward case of uninitialized memory and blk_sync_queue() -> throtl_shutdown_timer() -> cancel_delayed_work_sync().

Will get that fixed up.

It frees q->td in blk_cleanup_queue(), but doesn't clear q->td. When the final put happens, blk_sync_queue() is called and then ends up doing the cancel_delayed_work_sync() on freed memory.

Two possible fixes:

- Clear ->td when the queue is goin dead. May require other ->td == NULL checks in the code, so I opted for:

- Move the free to when the queue is really going away, post doing the blk_sync_queue() call.

The below should fix it.

block/blk-core.c +0 -2
index 4514146..51efd83 100644
@@ -462,8 +462,6 @@ void blk_cleanup_queue(struct request_queue *q)
 	if (q->elevator)
 		elevator_exit(q->elevator);
 
-	blk_throtl_exit(q);
-
 	blk_put_queue(q);
 }
 EXPORT_SYMBOL(blk_cleanup_queue);
block/blk-sysfs.c +2 -0
index da8a8a4..013457f 100644
@@ -471,6 +471,8 @@ static void blk_release_queue(struct kobject *kobj)
 
 	blk_sync_queue(q);
 
+	blk_throtl_exit(q);
+
 	if (rl->rq_pool)
 		mempool_destroy(rl->rq_pool);
 
Signature · 1 line
Jens Axboe
Tested-by
#6 lore raw
Quoted 14 lines · Jens Axboe
Quoted 4 lines

Looks like a fairly straight forward case of uninitialized memory and blk_sync_queue() -> throtl_shutdown_timer() -> cancel_delayed_work_sync().

Will get that fixed up.

It frees q->td in blk_cleanup_queue(), but doesn't clear q->td. When the final put happens, blk_sync_queue() is called and then ends up doing the cancel_delayed_work_sync() on freed memory.

Two possible fixes:

- Clear ->td when the queue is goin dead. May require other ->td == NULL checks in the code, so I opted for:

Quoted 6 lines

- Move the free to when the queue is really going away, post doing the blk_sync_queue() call.

The below should fix it.

This did the trick, thanks Jens!

Ingo

author
#7 lore raw

[GIT PULL] Throtl bug (was Re: [origin tree boot failure] Re: [GIT PULL] core block bits for 2.6.37-rc1)

Quoted 17 lines · Ingo Molnar
Quoted 14 lines · Jens Axboe
Quoted 4 lines

Looks like a fairly straight forward case of uninitialized memory and blk_sync_queue() -> throtl_shutdown_timer() -> cancel_delayed_work_sync().

Will get that fixed up.

It frees q->td in blk_cleanup_queue(), but doesn't clear q->td. When the final put happens, blk_sync_queue() is called and then ends up doing the cancel_delayed_work_sync() on freed memory.

Two possible fixes:

- Clear ->td when the queue is goin dead. May require other ->td == NULL checks in the code, so I opted for:

Quoted 8 lines
Quoted 6 lines

- Move the free to when the queue is really going away, post doing the blk_sync_queue() call.

The below should fix it.

This did the trick, thanks Jens!

Great, thanks for testing/reporting! I added your reported/tested-by.

Linus, please pull this single fix, better get this out the door since I'll be travelling very shortly.

git://git.kernel.dk/linux-2.6-block.git for-2.6.37/core

Jens Axboe (1):
block: fix use-after-free bug in blk throttle code

 block/blk-core.c  |    2 --
 block/blk-sysfs.c |    2 ++
 2 files changed, 2 insertions(+), 2 deletions(-)
Signature · 1 line
Jens Axboe
#8 lore raw

Re: [GIT PULL] Throtl bug (was Re: [origin tree boot failure] Re: [GIT PULL] core block bits for 2.6.37-rc1)

Quoted 33 lines · Jens Axboe
Quoted 17 lines · Ingo Molnar
Quoted 14 lines · Jens Axboe
Quoted 4 lines

Looks like a fairly straight forward case of uninitialized memory and blk_sync_queue() -> throtl_shutdown_timer() -> cancel_delayed_work_sync().

Will get that fixed up.

It frees q->td in blk_cleanup_queue(), but doesn't clear q->td. When the final put happens, blk_sync_queue() is called and then ends up doing the cancel_delayed_work_sync() on freed memory.

Two possible fixes:

- Clear ->td when the queue is goin dead. May require other ->td == NULL checks in the code, so I opted for:

Quoted 8 lines
Quoted 6 lines

- Move the free to when the queue is really going away, post doing the blk_sync_queue() call.

The below should fix it.

This did the trick, thanks Jens!

Great, thanks for testing/reporting! I added your reported/tested-by.

Linus, please pull this single fix, better get this out the door since I'll be travelling very shortly.

Quoted 9 lines

git://git.kernel.dk/linux-2.6-block.git for-2.6.37/core

Jens Axboe (1):
block: fix use-after-free bug in blk throttle code

 block/blk-core.c  |    2 --
 block/blk-sysfs.c |    2 ++
 2 files changed, 2 insertions(+), 2 deletions(-)

I have here very similar bug.
Must have been caused by this patch series. I pulled that tree, but that didn't affect anything.

System oopses/panics on removal of any hotplugable device. (reproduced with xD, MemoryStick, and USB mass storage).

Here is backtrace for MemoryStick card:

<6>[   24.138665] r592: IRQ: card removed
<1>[   24.228293] BUG: unable to handle kernel NULL pointer dereference at 00000000000001f8
<1>[   24.228966] IP: [<00000000000001f8>] 0x1f8
<4>[   24.230739] PGD 0 
<0>[   24.231182] Oops: 0010 [#1] PREEMPT SMP 
<0>[   24.231182] last sysfs file: /sys/devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda/sda3/alignment_offset
<4>[   24.231182] CPU 1 
<4>[   24.231182] Modules linked in: dm_crypt firewire_net usb_storage usb_libusual cpufreq_powersave cpufreq_conservative cpufreq_userspace uvcvideo videodev v4l2_compat_ioctl32 acpi_cpufreq iwl3945 iwlcore snd_hda_codec_realtek mac80211 mperf r852 iTCO_wdt coretemp uhci_hcd sm_common ir_lirc_codec mspro_block snd_hda_intel ms_block ehci_hcd sdhci_pci lirc_dev joydev sbp2 nand snd_hda_codec cfg80211 firewire_ohci sdhci ir_sony_decoder ieee1394 nand_ids usbcore r592 ir_jvc_decoder snd_hwdep mmc_core nand_ecc ir_rc6_decoder ene_ir snd_pcm tg3 ir_rc5_decoder firewire_core mtd battery memstick ac ir_nec_decoder psmouse snd_page_alloc libphy sunrpc ir_core sg evdev serio_raw dm_mirror dm_region_hash dm_log dm_mod nouveau ttm drm_kms_helper drm i2c_algo_bit thermal video
<4>[   32.881606] 
<4>[   32.881606] Pid: 543, comm: kworker/u:4 Not tainted 2.6.36+ #191 Nettiling/Aspire 5720     
<4>[   32.881606] RIP: 0010:[<00000000000001f8>]  [<00000000000001f8>] 0x1f8
<4>[   32.881606] RSP: 0018:ffff880037a03ab8  EFLAGS: 00010086
<4>[   32.881606] RAX: ffff88007c0ebc00 RBX: ffff880037af9470 RCX: 0000000000000000
<4>[   32.881606] RDX: 0000000000000019 RSI: 0000000000000001 RDI: ffff880037af9470
<4>[   32.881606] RBP: ffff880037a03ad0 R08: 0000000000000000 R09: 0000000000000001
<4>[   32.881606] R10: 00000000000002f0 R11: 0000000000000000 R12: ffff880037af9470
<4>[   32.881606] R13: ffff880075d6a870 R14: ffff880075bfb560 R15: 0000000000000282
<4>[   32.881606] FS:  0000000000000000(0000) GS:ffff88007f200000(0000) knlGS:0000000000000000
<4>[   32.881606] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<4>[   32.881606] CR2: 00000000000001f8 CR3: 000000007a046000 CR4: 00000000000006e0
<4>[   32.881606] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>[   32.881606] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>[   32.881606] Process kworker/u:4 (pid: 543, threadinfo ffff880037a02000, task ffff88007c5b0000)
<0>[   32.881606] Stack:
<4>[   32.881606]  ffffffff811c42a2 ffff880037a03af0 ffff880037af9470 ffff880037a03af0
<4>[   32.881606] <0> ffffffff811c525a ffff880077250040 ffff880077250040 ffff880037a03b10
<4>[   32.881606] <0> ffffffff811cebb2 ffff880075d6a800 ffff880075d6a8a8 ffff880037a03b30
<0>[   32.881606] Call Trace:
<4>[   32.881606]  [<ffffffff811c42a2>] ? elv_drain_elevator+0x22/0x70
<4>[   32.881606]  [<ffffffff811c525a>] elv_quiesce_start+0x3a/0xc0
<4>[   32.881606]  [<ffffffff811cebb2>] disk_replace_part_tbl+0x42/0x70
<4>[   32.881606]  [<ffffffff811cec63>] disk_release+0x23/0x50
<4>[   32.881606]  [<ffffffff81273c42>] device_release+0x22/0x90
<4>[   32.881606]  [<ffffffff811daced>] kobject_release+0x8d/0x1a0
<4>[   32.881606]  [<ffffffff811dac60>] ? kobject_release+0x0/0x1a0
<4>[   32.881606]  [<ffffffff811dc257>] kref_put+0x37/0x70
<4>[   32.881606]  [<ffffffff811dab67>] kobject_put+0x27/0x60
<4>[   32.881606]  [<ffffffff811cef42>] put_disk+0x12/0x20
<4>[   32.881606]  [<ffffffffa0627663>] mspro_block_disk_release+0xa3/0xb0 [mspro_block]
<4>[   32.881606]  [<ffffffffa062773d>] mspro_block_remove+0xcd/0x140 [mspro_block]
<4>[   32.881606]  [<ffffffffa01d42b5>] memstick_device_remove+0x35/0x60 [memstick]
<4>[   32.881606]  [<ffffffff81277630>] __device_release_driver+0x70/0xe0
<4>[   32.881606]  [<ffffffff8127779a>] device_release_driver+0x2a/0x40
<4>[   32.881606]  [<ffffffff812769b5>] bus_remove_device+0xb5/0x120
<4>[   32.881606]  [<ffffffff81274817>] device_del+0x127/0x1d0
<4>[   32.881606]  [<ffffffff812748dd>] device_unregister+0x1d/0x60
<4>[   32.881606]  [<ffffffffa01d5071>] memstick_check+0x241/0x360 [memstick]
<4>[   32.881606]  [<ffffffff8105a740>] process_one_work+0x1c0/0x4d0
<4>[   32.881606]  [<ffffffff8105a6e2>] ? process_one_work+0x162/0x4d0
<4>[   32.881606]  [<ffffffffa01d4e30>] ? memstick_check+0x0/0x360 [memstick]
<4>[   32.881606]  [<ffffffff8105ae36>] worker_thread+0x156/0x410
<4>[   32.881606]  [<ffffffff8105ace0>] ? worker_thread+0x0/0x410
<4>[   32.881606]  [<ffffffff8105ed66>] kthread+0xb6/0xc0
<4>[   32.881606]  [<ffffffff81037fa6>] ? finish_task_switch+0x46/0xe0
<4>[   32.881606]  [<ffffffff81003c14>] kernel_thread_helper+0x4/0x10
<4>[   32.881606]  [<ffffffff8105ecb0>] ? kthread+0x0/0xc0
<4>[   32.881606]  [<ffffffff81003c10>] ? kernel_thread_helper+0x0/0x10
<0>[   32.881606] Code:  Bad RIP value.
<1>[   32.881606] RIP  [<00000000000001f8>] 0x1f8
<4>[   32.881606]  RSP <ffff880037a03ab8>
<0>[   32.881606] CR2: 00000000000001f8
<4>[   32.881606] ---[ end trace ca0206dec4457aff ]---

Best regards,
Maxim Levitsky

#10 lore raw
In reply to Jens Axboe
Quoted 120 lines · Jens Axboe
Quoted 79 lines · Jens Axboe
Quoted 69 lines · Ingo Molnar

Hi,

Quoted 8 lines · Jens Axboe

Hi Linus,

This first pull request is the core bits, meaning general block layer changes or core support. Should be clean this time, only 'weird bit' is the seemingly duplicate entry from Malahal. This is caused by the first patch being buggy (and later reverted), second patch used the same single line description.

Quoted 5 lines

Nothing really exciting in here. A good collection of fixes, some of which are marked for stable as well.

The biggest addition this time around is the block IO throttling support from Vivek.

The upstream block bits pulled in this merge window (or maybe the workqueue bits) are possibly the cause a boot crash on today's -tip, using a trivial x86 bootup test (64-bit allyesconfig):

[  116.064281] calling  hd_init+0x0/0x302 @ 1
[  116.068529] hd: no drives specified - use hd=cyl,head,sectors on kernel command line
[  116.076334] general protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC
[  116.080274] last sysfs file: 
[  116.080274] CPU 0 
[  116.080274] Modules linked in:
[  116.080274] 
[  116.080274] Pid: 1, comm: swapper Tainted: G        W   2.6.36-tip-03555-g825d9ec-dirty #51843 A8N-E/System Product Name
[  116.080274] RIP: 0010:[<ffffffff81064380>]  [<ffffffff81064380>] __ticket_spin_trylock+0x4/0x21
[  116.080274] RSP: 0018:ffff88003c417c10  EFLAGS: 00010082
[  116.080274] RAX: ffff88003c418000 RBX: 6b6b6b6b6b6b6b6a RCX: 0000000000000000
[  116.080274] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6b6a
[  116.080274] RBP: ffff88003c417c10 R08: 0000000000000002 R09: 0000000000000001
[  116.080274] R10: 0000000000000286 R11: ffff880032498738 R12: 6b6b6b6b6b6b6b82
[  116.080274] R13: 0000000000000286 R14: 6b6b6b6b6b6b6b6b R15: 0000000000000001
[  116.080274] FS:  0000000000000000(0000) GS:ffff88003e200000(0000) knlGS:0000000000000000
[  116.080274] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  116.080274] CR2: 0000000000000000 CR3: 0000000004071000 CR4: 00000000000006f0
[  116.080274] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  116.080274] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  116.080274] Process swapper (pid: 1, threadinfo ffff88003c416000, task ffff88003c418000)
[  116.080274] Stack:
[  116.080274]  ffff88003c417c30 ffffffff8168c6ee 6b6b6b6b6b6b6b6b 6b6b6b6b6b6b6b6a
[  116.080274] <0> ffff88003c417c70 ffffffff82d37a20 ffffffff810a1b65 ffff88003c418000
[  116.080274] <0> ffffffff82d3836b 6b6b6b6b6b6b6b6a ffff8800330fcc20 ffff88003c417cb8
[  116.080274] Call Trace:
[  116.080274]  [<ffffffff8168c6ee>] do_raw_spin_trylock+0x1f/0x41
[  116.080274]  [<ffffffff82d37a20>] _raw_spin_lock_irqsave+0x72/0xa4
[  116.080274]  [<ffffffff810a1b65>] ? lock_timer_base+0x2c/0x52
[  116.080274]  [<ffffffff82d3836b>] ? _raw_spin_unlock_irqrestore+0x55/0x72
[  116.080274]  [<ffffffff810a1b65>] lock_timer_base+0x2c/0x52
[  116.080274]  [<ffffffff810a1c43>] del_timer+0x2f/0x82
[  116.080274]  [<ffffffff810ac906>] ? wait_on_work+0x0/0xdb
[  116.080274]  [<ffffffff810aca18>] __cancel_work_timer+0x37/0x130
[  116.080274]  [<ffffffff810acb23>] cancel_delayed_work_sync+0x12/0x14
[  116.080274]  [<ffffffff8166974a>] throtl_shutdown_timer_wq+0x1c/0x1e
[  116.080274]  [<ffffffff8165dbec>] blk_sync_queue+0x3d/0x41
[  116.080274]  [<ffffffff8165f872>] blk_release_queue+0x1e/0x6a
[  116.080274]  [<ffffffff81673ce3>] kobject_release+0xf4/0x122
[  116.080274]  [<ffffffff81673bef>] ? kobject_release+0x0/0x122
[  116.080274]  [<ffffffff81674e7e>] kref_put+0x43/0x4d
[  116.080274]  [<ffffffff81673b46>] kobject_put+0x47/0x4c
[  116.080274]  [<ffffffff8165dc53>] blk_cleanup_queue+0x63/0x68
[  116.080274]  [<ffffffff84883c80>] ? hd_init+0x0/0x302
[  116.080274]  [<ffffffff84883f54>] hd_init+0x2d4/0x302
[  116.080274]  [<ffffffff81910778>] ? device_pm_unlock+0x15/0x17
Quoted 8 lines
[  116.080274]  [<ffffffff84883c80>] ? hd_init+0x0/0x302
[  116.080274]  [<ffffffff81002062>] do_one_initcall+0x57/0x15a
[  116.080274]  [<ffffffff8482f78b>] kernel_init+0x194/0x222
[  116.080274]  [<ffffffff8103ad04>] kernel_thread_helper+0x4/0x10
[  116.080274]  [<ffffffff82d38710>] ? restore_args+0x0/0x30
[  116.080274]  [<ffffffff8482f5f7>] ? kernel_init+0x0/0x222
[  116.080274]  [<ffffffff8103ad00>] ? kernel_thread_helper+0x0/0x10
[  116.080274] Code: ff ff c9 c3 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 0f b7 17 eb f5 c9 c3 55 48 89 e5 <8b> 07 89 c2 c1 c0 10 39 c2 8d 90 00 00 01 00 75 04 f0 0f b1 17 
Quoted 5 lines

Looks like a fairly straight forward case of uninitialized memory and blk_sync_queue() -> throtl_shutdown_timer() -> cancel_delayed_work_sync().

Will get that fixed up.

It frees q->td in blk_cleanup_queue(), but doesn't clear q->td. When the final put happens, blk_sync_queue() is called and then ends up doing the cancel_delayed_work_sync() on freed memory.

Two possible fixes:

- Clear ->td when the queue is goin dead. May require other ->td == NULL checks in the code, so I opted for:

- Move the free to when the queue is really going away, post doing the blk_sync_queue() call.

The below should fix it.

block/blk-core.c +0 -2
index 4514146..51efd83 100644
@@ -462,8 +462,6 @@ void blk_cleanup_queue(struct request_queue *q)
 	if (q->elevator)
 		elevator_exit(q->elevator);
 
-	blk_throtl_exit(q);
-
 	blk_put_queue(q);
 }
 EXPORT_SYMBOL(blk_cleanup_queue);
block/blk-sysfs.c +0 -0
index da8a8a4..013457f 100644

block/blk-sysfs.c static void blk_release_queue(struct kobject *kobj) · line 477

Quoted 7 lines
block/blk-sysfs.c +2 -0
@@ -471,6 +471,8 @@ static void blk_release_queue(struct kobject *kobj)
 
 	blk_sync_queue(q);
 
+	blk_throtl_exit(q);
+
 	if (rl->rq_pool)
 		mempool_destroy(rl->rq_pool);

Thanks for the fix Jens. I had done testing with pulling out a usb key from a running system to check for hot remove/ blk_cleanup_queue() path and not sure why didn't I catch it.

I have got one little concern here. blk_throtl_exit() takes requeust queue spin locks and relies on the fact that q->queue_lock is still around.

IIUC, in blk_release_queue(), there is no gurantee that driver has not freed up the memory associated with spin lock (If it is a driver provided spin lock).

Checking for q->td in throtl_shutdown_timer_wq(), might be a fix but it has the potential to be racy as throtl_shutdown_timer_wq() does not take spin lock and I guess it can't take spin lock to check for q->td, as it is called in blk_release_queue-> blk_sync_queue path and it is not guranteed if spin lock is still around.

So may be we need to come up with a method to make sure driver does not release queue lock until all the users of queue are gone and one can safely assume q->queue_lock is valid in blk_release_queue().

Or may be make q->td rcu protected. It is already spin lock protected and it kind of will become messy to access it under rcu lock in throtl_shutdown_timer_wq(), and under q->queue_lock in rest of the places. Ofcourse freeing of q->td will be after waiting through call_rcu().

Thanks
Vivek

#9 lore raw

Re: [GIT PULL] Throtl bug (was Re: [origin tree boot failure] Re: [GIT PULL] core block bits for 2.6.37-rc1)

In reply to Maxim Levitsky
Quoted 106 lines · Maxim Levitsky
Quoted 33 lines · Jens Axboe
Quoted 17 lines · Ingo Molnar
Quoted 14 lines · Jens Axboe
Quoted 4 lines

Looks like a fairly straight forward case of uninitialized memory and blk_sync_queue() -> throtl_shutdown_timer() -> cancel_delayed_work_sync().

Will get that fixed up.

It frees q->td in blk_cleanup_queue(), but doesn't clear q->td. When the final put happens, blk_sync_queue() is called and then ends up doing the cancel_delayed_work_sync() on freed memory.

Two possible fixes:

- Clear ->td when the queue is goin dead. May require other ->td == NULL checks in the code, so I opted for:

Quoted 8 lines
Quoted 6 lines

- Move the free to when the queue is really going away, post doing the blk_sync_queue() call.

The below should fix it.

This did the trick, thanks Jens!

Great, thanks for testing/reporting! I added your reported/tested-by.

Linus, please pull this single fix, better get this out the door since I'll be travelling very shortly.

Quoted 9 lines

git://git.kernel.dk/linux-2.6-block.git for-2.6.37/core

Jens Axboe (1):
block: fix use-after-free bug in blk throttle code

 block/blk-core.c  |    2 --
 block/blk-sysfs.c |    2 ++
 2 files changed, 2 insertions(+), 2 deletions(-)

I have here very similar bug.
Must have been caused by this patch series. I pulled that tree, but that didn't affect anything.

System oopses/panics on removal of any hotplugable device. (reproduced with xD, MemoryStick, and USB mass storage).

Here is backtrace for MemoryStick card:

<6>[   24.138665] r592: IRQ: card removed
<1>[   24.228293] BUG: unable to handle kernel NULL pointer dereference at 00000000000001f8
<1>[   24.228966] IP: [<00000000000001f8>] 0x1f8
<4>[   24.230739] PGD 0 
<0>[   24.231182] Oops: 0010 [#1] PREEMPT SMP 
<0>[   24.231182] last sysfs file: /sys/devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda/sda3/alignment_offset
<4>[   24.231182] CPU 1 
<4>[   24.231182] Modules linked in: dm_crypt firewire_net usb_storage usb_libusual cpufreq_powersave cpufreq_conservative cpufreq_userspace uvcvideo videodev v4l2_compat_ioctl32 acpi_cpufreq iwl3945 iwlcore snd_hda_codec_realtek mac80211 mperf r852 iTCO_wdt coretemp uhci_hcd sm_common ir_lirc_codec mspro_block snd_hda_intel ms_block ehci_hcd sdhci_pci lirc_dev joydev sbp2 nand snd_hda_codec cfg80211 firewire_ohci sdhci ir_sony_decoder ieee1394 nand_ids usbcore r592 ir_jvc_decoder snd_hwdep mmc_core nand_ecc ir_rc6_decoder ene_ir snd_pcm tg3 ir_rc5_decoder firewire_core mtd battery memstick ac ir_nec_decoder psmouse snd_page_alloc libphy sunrpc ir_core sg evdev serio_raw dm_mirror dm_region_hash dm_log dm_mod nouveau ttm drm_kms_helper drm i2c_algo_bit thermal video
<4>[   32.881606] 
<4>[   32.881606] Pid: 543, comm: kworker/u:4 Not tainted 2.6.36+ #191 Nettiling/Aspire 5720     
<4>[   32.881606] RIP: 0010:[<00000000000001f8>]  [<00000000000001f8>] 0x1f8
<4>[   32.881606] RSP: 0018:ffff880037a03ab8  EFLAGS: 00010086
<4>[   32.881606] RAX: ffff88007c0ebc00 RBX: ffff880037af9470 RCX: 0000000000000000
<4>[   32.881606] RDX: 0000000000000019 RSI: 0000000000000001 RDI: ffff880037af9470
<4>[   32.881606] RBP: ffff880037a03ad0 R08: 0000000000000000 R09: 0000000000000001
<4>[   32.881606] R10: 00000000000002f0 R11: 0000000000000000 R12: ffff880037af9470
<4>[   32.881606] R13: ffff880075d6a870 R14: ffff880075bfb560 R15: 0000000000000282
<4>[   32.881606] FS:  0000000000000000(0000) GS:ffff88007f200000(0000) knlGS:0000000000000000
<4>[   32.881606] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<4>[   32.881606] CR2: 00000000000001f8 CR3: 000000007a046000 CR4: 00000000000006e0
<4>[   32.881606] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>[   32.881606] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>[   32.881606] Process kworker/u:4 (pid: 543, threadinfo ffff880037a02000, task ffff88007c5b0000)
<0>[   32.881606] Stack:
<4>[   32.881606]  ffffffff811c42a2 ffff880037a03af0 ffff880037af9470 ffff880037a03af0
<4>[   32.881606] <0> ffffffff811c525a ffff880077250040 ffff880077250040 ffff880037a03b10
<4>[   32.881606] <0> ffffffff811cebb2 ffff880075d6a800 ffff880075d6a8a8 ffff880037a03b30
<0>[   32.881606] Call Trace:
<4>[   32.881606]  [<ffffffff811c42a2>] ? elv_drain_elevator+0x22/0x70
<4>[   32.881606]  [<ffffffff811c525a>] elv_quiesce_start+0x3a/0xc0
<4>[   32.881606]  [<ffffffff811cebb2>] disk_replace_part_tbl+0x42/0x70
<4>[   32.881606]  [<ffffffff811cec63>] disk_release+0x23/0x50
<4>[   32.881606]  [<ffffffff81273c42>] device_release+0x22/0x90
<4>[   32.881606]  [<ffffffff811daced>] kobject_release+0x8d/0x1a0
<4>[   32.881606]  [<ffffffff811dac60>] ? kobject_release+0x0/0x1a0
<4>[   32.881606]  [<ffffffff811dc257>] kref_put+0x37/0x70
<4>[   32.881606]  [<ffffffff811dab67>] kobject_put+0x27/0x60
<4>[   32.881606]  [<ffffffff811cef42>] put_disk+0x12/0x20
<4>[   32.881606]  [<ffffffffa0627663>] mspro_block_disk_release+0xa3/0xb0 [mspro_block]
<4>[   32.881606]  [<ffffffffa062773d>] mspro_block_remove+0xcd/0x140 [mspro_block]
<4>[   32.881606]  [<ffffffffa01d42b5>] memstick_device_remove+0x35/0x60 [memstick]
<4>[   32.881606]  [<ffffffff81277630>] __device_release_driver+0x70/0xe0
<4>[   32.881606]  [<ffffffff8127779a>] device_release_driver+0x2a/0x40
<4>[   32.881606]  [<ffffffff812769b5>] bus_remove_device+0xb5/0x120
<4>[   32.881606]  [<ffffffff81274817>] device_del+0x127/0x1d0
<4>[   32.881606]  [<ffffffff812748dd>] device_unregister+0x1d/0x60
<4>[   32.881606]  [<ffffffffa01d5071>] memstick_check+0x241/0x360 [memstick]
<4>[   32.881606]  [<ffffffff8105a740>] process_one_work+0x1c0/0x4d0
<4>[   32.881606]  [<ffffffff8105a6e2>] ? process_one_work+0x162/0x4d0
<4>[   32.881606]  [<ffffffffa01d4e30>] ? memstick_check+0x0/0x360 [memstick]
<4>[   32.881606]  [<ffffffff8105ae36>] worker_thread+0x156/0x410
<4>[   32.881606]  [<ffffffff8105ace0>] ? worker_thread+0x0/0x410
<4>[   32.881606]  [<ffffffff8105ed66>] kthread+0xb6/0xc0
<4>[   32.881606]  [<ffffffff81037fa6>] ? finish_task_switch+0x46/0xe0
Quoted 9 lines
<4>[   32.881606]  [<ffffffff81003c14>] kernel_thread_helper+0x4/0x10
<4>[   32.881606]  [<ffffffff8105ecb0>] ? kthread+0x0/0xc0
<4>[   32.881606]  [<ffffffff81003c10>] ? kernel_thread_helper+0x0/0x10
<0>[   32.881606] Code:  Bad RIP value.
<1>[   32.881606] RIP  [<00000000000001f8>] 0x1f8
<4>[   32.881606]  RSP <ffff880037a03ab8>
<0>[   32.881606] CR2: 00000000000001f8
<4>[   32.881606] ---[ end trace ca0206dec4457aff ]---

Looking at the backtrace and commit messages, it might be coming from following commit.

commit 7681bfeeccff5efa9eb29bf09249a3c400b15327

Author: Yasuaki Ishimatsu <[email protected]>
Date:   Tue Oct 19 09:05:00 2010 +0200

block: fix accounting bug on cross partition merges

Looks like we have freed the request queue in mspro_block_remove() and then we are calling mspro_block_disk_release() which ends up accessing request queue in disk_replace_part_tbl(). So use-after-free case.

CCing Yasuaki Ishimatsu <[email protected]>.

Thanks
Vivek