Skip to contentSkip to conversation

Core block IO bits for 4.2

GIT PULL Jens Axboe 5 participants 7 messages Jun 25, 2015 – Jun 29, 2015 Device mapper

lore
Share & tools
lore

Jens Axboe

author
#0 lore raw

Hi Linus,

This is the core block pull request for 4.2. Nothing really major in here, mostly a collection of smaller optimizations and cleanups, mixed with various fixes. In more detail, this pull request contains:

- Addition of policy specific data to blkcg for block cgroups. From Arianna Avanzini.

- Various cleanups around command types from Christoph.

- Cleanup of the suspend block I/O path from Christoph.

- Plugging updates from Shaohua and Jeff Moyer, for blk-mq.

- Eliminating atomic inc/dec of both remaining IO count and reference count in a bio. From me.

- Fixes for SG gap and chunk size support for data-less (discards) IO, so we can merge these better. From me.

- Small restructuring of blk-mq shared tag support, freeing drivers from iterating hardware queues. From Keith Busch.

- A few cfq-iosched tweaks, from Tahsin Erdogan and me. Makes the IOPS mode the default for non-rotational storage.

Please pull!

git://git.kernel.dk/linux-block.git for-4.2/core

----------------------------------------------------------------
Arianna Avanzini (1):
block, cgroup: implement policy-specific per-blkcg data

Christoph Hellwig (11):
block: rename REQ_TYPE_SPECIAL to REQ_TYPE_DRV_PRIV
block: move REQ_TYPE_ATA_TASKFILE and REQ_TYPE_ATA_PC to ide.h
block: move REQ_TYPE_SENSE to the ide driver
block: remove REQ_TYPE_PM_SHUTDOWN
block: move PM request support to IDE
nbd: stop using req->cmd
block: use an atomic_t for mq_freeze_depth
block: remove BIO_EOPNOTSUPP
block: remove unused BIO_RW_BLOCK and BIO_EOF flags
suspend: simplify block I/O handling
block, dm: don't copy bios for request clones

Jarod Wilson (1):
block: export blkdev_reread_part() and __blkdev_reread_part()

Jeff Moyer (1):
blk-mq: fix plugging in blk_sq_make_request

Jens Axboe (10):
bio: skip atomic inc/dec of ->bi_remaining for non-chains
bio: skip atomic inc/dec of ->bi_cnt for most use cases
block: collapse bio bit space
block: only honor SG gap prevention for merges that contain data
block: don't honor chunk sizes for data-less IO
block: add blk_set_queue_dying() to blkdev.h
cfq-iosched: fix the setting of IOPS mode on SSDs
cfq-iosched: move group scheduling functions under ifdef
cfq-iosched: fix sysfs oops when attempting to read unconfigured weights
cfq-iosched: fix other locations where blkcg_to_cfqgd() can return NULL

Julia Lawall (1):
block: fix returnvar.cocci warnings

Keith Busch (1):
blk-mq: Shared tag enhancements

Mike Snitzer (1):
block: remove management of bi_remaining when restoring original bi_end_io

Ming Lei (1):
block: replace trylock with mutex_lock in blkdev_reread_part()

Shaohua Li (5):
blk: clean up plug
sched: always use blk_schedule_flush_plug in io_schedule_out
blk-mq: avoid re-initialize request which is failed in direct dispatch
blk-mq: do limited block plug for multiple queue case
blk-mq: make plug work for mutiple disks and queues

Steven Rostedt (1):
blktrace: Add blktrace.c to BLOCK LAYER in MAINTAINERS file

Tahsin Erdogan (1):
block: Make CFQ default to IOPS mode on SSDs

 MAINTAINERS                  |   1 +
 block/bio-integrity.c        |   4 +-
 block/bio.c                  |  77 +++++++++++--------
 block/blk-cgroup.c           |  92 ++++++++++++++++++++---
 block/blk-cgroup.h           |  40 ++++++++--
 block/blk-core.c             | 136 +++++++++-------------------------
 block/blk-exec.c             |  10 ---
 block/blk-merge.c            |   3 +-
 block/blk-mq-tag.c           |  38 ++++++++++
 block/blk-mq-tag.h           |   1 +
 block/blk-mq.c               | 160 +++++++++++++++++++++++++---------------
 block/blk.h                  |   5 +-
 block/bounce.c               |   3 -
 block/cfq-iosched.c          | 125 +++++++++++++++++++++++++------
 block/elevator.c             |   2 +
 block/ioctl.c                |  37 ++++++++--
 drivers/block/nbd.c          |  50 ++++++-------
 drivers/block/paride/pd.c    |   4 +-
 drivers/block/sx8.c          |   4 +-
 drivers/block/virtio_blk.c   |   6 +-
 drivers/ide/ide-atapi.c      |  10 +--
 drivers/ide/ide-cd.c         |  10 +--
 drivers/ide/ide-cd_ioctl.c   |   2 +-
 drivers/ide/ide-devsets.c    |   2 +-
 drivers/ide/ide-eh.c         |   4 +-
 drivers/ide/ide-floppy.c     |   8 +-
 drivers/ide/ide-io.c         |  12 +--
 drivers/ide/ide-ioctls.c     |   2 +-
 drivers/ide/ide-park.c       |   4 +-
 drivers/ide/ide-pm.c         |  56 ++++++++++----
 drivers/ide/ide-tape.c       |   6 +-
 drivers/ide/ide-taskfile.c   |   2 +-
 drivers/md/bcache/io.c       |   2 +-
 drivers/md/bcache/request.c  |   2 +-
 drivers/md/dm-cache-target.c |   6 --
 drivers/md/dm-raid1.c        |   2 -
 drivers/md/dm-snap.c         |   1 -
 drivers/md/dm-table.c        |  25 ++++---
 drivers/md/dm-thin.c         |   9 +--
 drivers/md/dm-verity.c       |   2 +-
 drivers/md/dm.c              | 171 ++++++++++---------------------------------
 drivers/md/dm.h              |   5 +-
 fs/btrfs/disk-io.c           |  13 +---
 fs/btrfs/extent_io.c         |   2 -
 fs/btrfs/volumes.c           |  18 ++---
 fs/btrfs/volumes.h           |   2 -
 fs/buffer.c                  |  13 +---
 fs/ext4/page-io.c            |   1 -
 fs/nilfs2/segbuf.c           |  12 ---
 fs/xfs/xfs_aops.c            |   1 -
 include/linux/bio.h          |  17 ++++-
 include/linux/blk-mq.h       |   4 +
 include/linux/blk_types.h    |  25 ++++---
 include/linux/blkdev.h       |  45 ++----------
 include/linux/elevator.h     |   2 +
 include/linux/fs.h           |   3 +
 include/linux/ide.h          |  27 +++++++
 include/linux/swap.h         |   1 -
 include/uapi/linux/nbd.h     |   2 -
 kernel/power/Makefile        |   3 +-
 kernel/power/block_io.c      | 103 --------------------------
 kernel/power/power.h         |   9 ---
 kernel/power/swap.c          | 159 ++++++++++++++++++++++++++++++----------
 kernel/sched/core.c          |   5 +-
 mm/page_io.c                 |   2 +-
 65 files changed, 853 insertions(+), 757 deletions(-)
 delete mode 100644 kernel/power/block_io.c
Signature · 1 line
Jens Axboe
#1 lore raw

Hi Jens,

Quoted 1 line · Jens Axboe

block, dm: don't copy bios for request clones

this change should not be pushed to mainline yet.

Firstly, Christoph has a newer version of the patch that fixes silent data corruption problem:
https://www.redhat.com/archives/dm-devel/2015-May/msg00229.html

And the new version still depends on LLDDs to always complete requests to the end when error happens, while block API doesn't enforce such a requirement. If the assumption is ever broken, the inconsistency between request and bio (e.g. rq->__sector and rq->bio) will cause silent data corruption:
https://www.redhat.com/archives/dm-devel/2015-June/msg00022.html

Signature · 1 line
Jun'ichi Nomura, NEC Corporation
#2 lore raw
Quoted 7 lines · Junichi Nomura

Hi Jens,

Quoted 1 line · Jens Axboe

block, dm: don't copy bios for request clones

this change should not be pushed to mainline yet.

Hmm. I merged all of Jens' pull requests today, so it's now in my tree. What do?

Linus

#3 lore raw
Quoted 7 lines · Linus Torvalds
Quoted 4 lines · On Jun 25, 2015 23:17, "Junichi Nomura" <[email protected]…
Quoted 1 line · Jens Axboe

block, dm: don't copy bios for request clones

this change should not be pushed to mainline yet.

Hmm. I merged all of Jens' pull requests today, so it's now in my tree. What do?

Then I think following 3 commits should be reverted.

2d76fff dm: cleanup methods that requeue requests
cbc4e3c dm: do not allocate any mempools for blk-mq request-based DM 5f1b670 block, dm: don't copy bios for request clones

Reverting 5f1b670 involves conflicts in drvers/md/dm.c for these 2 hunks due to diff contexts difference. It's ok to revert those changes ignoring the contexts.

@@ -1089,8 +1038,6 @@ static void free_rq_clone(struct request *clone, bool must_be_mapped)

          WARN_ON_ONCE(must_be_mapped && !clone->q);

- blk_rq_unprep_clone(clone);
-

          if (md->type == DM_TYPE_MQ_REQUEST_BASED)
                  /* stacked on blk-mq queue(s) */
                  tio->ti->type->release_clone_rq(clone);

@@ -1973,11 +1889,7 @@ static int map_request(struct dm_rq_target_io *tio, struct request *rq,

                  }
                  if (IS_ERR(clone))
                          return DM_MAPIO_REQUEUE;
  -               if (setup_clone(clone, rq, tio, GFP_ATOMIC)) {
  -                       /* -ENOMEM */
  -                       ti->type->release_clone_rq(clone);
  -                       return DM_MAPIO_REQUEUE;
  -               }
  +               setup_clone(clone, rq, tio);
          }
   
          switch (r) {
Signature · 1 line
Jun'ichi Nomura, NEC Corporation
#4 lore raw

[git pull] device mapper fixes for 4.2-rc1

Quoted 11 lines · Junichi Nomura
Quoted 7 lines · Linus Torvalds
Quoted 4 lines · On Jun 25, 2015 23:17, "Junichi Nomura" <[email protected]…
Quoted 1 line · Jens Axboe

block, dm: don't copy bios for request clones

this change should not be pushed to mainline yet.

Hmm. I merged all of Jens' pull requests today, so it's now in my tree. What do?

Then I think following 3 commits should be reverted.

Quoted 7 lines

2d76fff dm: cleanup methods that requeue requests
cbc4e3c dm: do not allocate any mempools for blk-mq request-based DM 5f1b670 block, dm: don't copy bios for request clones

Reverting 5f1b670 involves conflicts in drvers/md/dm.c for these 2 hunks due to diff contexts difference. It's ok to revert those changes ignoring the contexts.

That is more than I'd care to put on Linus at this point. I've prepared a DM pull request that fixes things up. Apologies for not pressing this request-based DM partial completion issue further, it was an oversight on my part. We'll have to get it fixed up properly and revisit for a future release.

Linus,

The following changes since commit e262f34741522e0d821642e5449c6eeb512723fc:

dm stats: add support for request-based DM devices (2015-06-17 12:40:41 -0400)

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm tags/dm-4.2-fixes

for you to fetch changes up to b5451e456840af027b794afc2c7c84c2a17f569b:

dm cache policy smq: fix "default" version to be 1.4.0 (2015-06-26 10:14:28 -0400)

----------------------------------------------------------------
- Revert block and DM core changes that removed request-based DM's ability to handle partial request completions -- otherwise with the current SCSI LLDs these changes could lead to silent data corruption.

- Fix two DM version bumps that were missing from the initial 4.2 DM pull request (enabled userspace lvm2 to know certain changes have been made).

----------------------------------------------------------------
Mike Snitzer (4):
Revert "dm: do not allocate any mempools for blk-mq request-based DM" Revert "block, dm: don't copy bios for request clones"
dm: bump the ioctl version to 4.32.0
dm cache policy smq: fix "default" version to be 1.4.0

 block/blk-core.c                 |  94 +++++++++++++++--
 drivers/md/dm-cache-policy-smq.c |   2 +-
 drivers/md/dm-table.c            |  29 ++----
 drivers/md/dm.c                  | 220 +++++++++++++++++++++++++++------------
 drivers/md/dm.h                  |   5 +-
 include/linux/blk_types.h        |   2 -
 include/linux/blkdev.h           |   6 +-
 include/uapi/linux/dm-ioctl.h    |   4 +-
 8 files changed, 256 insertions(+), 106 deletions(-)
author
#5 lore raw
In reply to Junichi Nomura
Quoted 9 lines · Junichi Nomura

Hi Jens,

Quoted 1 line · Jens Axboe

block, dm: don't copy bios for request clones

this change should not be pushed to mainline yet.

Firstly, Christoph has a newer version of the patch that fixes silent data corruption problem:

Quoted 8 lines
   https://www.redhat.com/archives/dm-devel/2015-May/msg00229.html

And the new version still depends on LLDDs to always complete requests to the end when error happens, while block API doesn't enforce such a requirement. If the assumption is ever broken, the inconsistency between request and bio (e.g. rq->__sector and rq->bio) will cause silent data corruption:

   https://www.redhat.com/archives/dm-devel/2015-June/msg00022.html

Why was none of that sent to the block tree?

In any case, Linus, pick up Mike's pull request to fix this up.

Signature · 1 line
Jens Axboe
#6 lore raw
Quoted 3 lines · Jens Axboe

Why was none of that sent to the block tree?

In any case, Linus, pick up Mike's pull request to fix this up.

I didn't manage to driver the discussion forward, and hadn't really grasped it was in the block tree. For some reason I assumed it only was in Mikes dm WIP tree.