NVMe changes for 4.5-rc1

Hi Linus,

Last branch for this series is the nvme changes. It's in a separate branch to avoid splitting too much between core and NVMe changes, since NVMe is still helping drive some blk-mq changes. That said, not a huge amount of core changes in here. The grunt of the work is the continued split of the code.

Note that pulling this in will conflict with master, since the code was forked off pretty early, and we had a good chunk of nvme fixes later in the 4.4 cycle.

Please pull!

git://git.kernel.dk/linux-block.git for-4.5/nvme

----------------------------------------------------------------
Arnd Bergmann (1):
nvme: fix another 32-bit build warning

Christoph Hellwig (45):
nvme: move struct nvme_iod to pci.c
nvme: split command submission helpers out of pci.c
nvme: use offset instead of a struct for registers
nvme: split nvme_trans_device_id_page
nvme: use vendor it from identify
nvme: split a new struct nvme_ctrl out of struct nvme_dev
nvme: simplify nvme_setup_prps calling convention
nvme: refactor nvme_queue_rq
nvme: factor out a nvme_unmap_data helper
nvme: move nvme_error_status to common code
nvme: move nvme_setup_flush and nvme_setup_rw to common code
nvme: split __nvme_submit_sync_cmd
nvme: move block_device_operations and ns/ctrl freeing to common code
nvme: add explicit quirk handling
nvme: move remaining CC setup into nvme_enable_ctrl
nvme: move nvme_{enable,disable,shutdown}_ctrl to common code
nvme: add a common helper to read Identify Controller data
nvme: move the call to nvme_init_identify earlier
nvme: move namespace scanning to common code
nvme: move chardev and sysfs interface to common code
nvme: refactor set_queue_count
NVMe: fix build with CONFIG_NVM enabled
block: defer timeouts to a workqueue
nvme: only ignore hardware errors in nvme_create_io_queues
nvme: only add a controller to dev_list after it's been fully initialized
nvme: don't take the I/O queue q_lock in nvme_timeout
nvme: merge nvme_abort_req and nvme_timeout
nvme: add NVME_SC_CANCELLED
nvme: simplify resets
nvme: merge probe_work and reset_work
nvme: remove dead controllers from a work item
nvme: switch abort_limit to an atomic_t
nvme: fix admin queue depth
nvme: factor out a few helpers from req_completion
nvme: switch delete SQ/CQ to blk_execute_rq_nowait
nvme: switch abort to blk_execute_rq_nowait
nvme: special case AEN requests
nvme: simplify completion handling
nvme: properly free resources for cancelled command
nvme: meta_sg doesn't have to be an array
nvme: merge iod and cmd_info
block: remove REQ_NO_TIMEOUT flag
nvme: synchronize access to ctrl->namespaces
nvme: fixes for NVME_IOCTL_IO_CMD on the char device
nvme: make SG_IO support optional

Dan Carpenter (2):
blk-integrity: checking for NULL instead of IS_ERR
nvme: precedence bug in nvme_pr_clear()

Keith Busch (16):
nvme: use the block layer for userspace passthrough metadata
blk-integrity: empty implementation when disabled
nvme: protect against simultaneous shutdown invocations
nvme: do not restart the request timeout if we're resetting the controller
NVMe: Implement namespace list scanning
NVMe: Use unbounded work queue for all work
NVMe: Remove device management handles on remove
NVMe: Simplify metadata setup
NVMe: Add pci error handlers
NVMe: Export namespace attributes to sysfs
NVMe: Fix admin queue ring wrap
NVMe: Use a retryable error code on reset
NVMe: Remove queue freezing on resets
NVMe: IO queue deletion re-write
NVMe: Shutdown controller only for power-off
NVMe: Export NVMe attributes to sysfs group

Mike Frysinger (1):
uapi: update install list after nvme.h rename

Sagi Grimberg (1):
nvme: Move nvme_freeze/unfreeze_queues to nvme core

Sudip Mukherjee (1):
PCI/AER: include header file

 block/bio-integrity.c               |   13 +-
 block/blk-core.c                    |    8 +
 block/blk-mq.c                      |   13 +-
 block/blk-timeout.c                 |   11 +-
 block/blk.h                         |    2 +-
 drivers/nvme/host/Kconfig           |   11 +
 drivers/nvme/host/Makefile          |    3 +-
 drivers/nvme/host/core.c            | 1463 +++++++++++++++++++
 drivers/nvme/host/lightnvm.c        |   35 +-
 drivers/nvme/host/nvme.h            |  242 +++-
 drivers/nvme/host/pci.c             | 2706 ++++++++++-------------------------
 drivers/nvme/host/scsi.c            |  212 +--
 drivers/target/target_core_iblock.c |    4 +-
 include/linux/aer.h                 |    1 +
 include/linux/bio.h                 |   32 +-
 include/linux/blk_types.h           |    2 -
 include/linux/blkdev.h              |    1 +
 include/linux/nvme.h                |   27 +-
 include/uapi/linux/Kbuild           |    2 +-
 19 files changed, 2598 insertions(+), 2190 deletions(-)
 create mode 100644 drivers/nvme/host/core.c

Jens Axboe

Not just conflict, but conflict in bad ways. I don't think I'll be able to fix it up sanely.

In particular, commit b5875222de2f ("NVMe: IO ending fixes on surprise removal") by Keith Busch added this to nvme_dev_remove():

    if (nvme_io_incapable(dev)) {
        /*
         * If the device is not capable of IO (surprise hot-removal,
         * for example), we need to quiesce prior to deleting the
         * namespaces. This will end outstanding requests and prevent
         * attempts to sync dirty data.
         */
        nvme_dev_shutdown(dev);
    }

and in your branch we now have:

- nvme_dev_shutdown() is now nvme_dev_disable(dev, false). Fine.

- nvme_dev_remove() got renamed to nvme_remove_namespaces(), but also
lost the "dev" argument (it takes "struct nvme_ctrl *ctrl" now).

I think I will end up doing the merge by just dropping that part of the surprise removal commit, and letting you and Keith work out what the real solution is..

Linus

&dev->ctrl

That's fine, thanks for merging!

Jens Axboe

The latter is the cleanest approach, we'll get it fixed up.

Is that like a Canadian duel?

Jens Axboe

Looks like IO ending is the only conflict. The cleanup below will get the functionality back.

You might notice nvme_dev_disable is called twice for a surprise removal, case, but that's okay.
---

index 72ef832..5d31cfc 100644
@@ -2118,6 +2118,15 @@ static void nvme_remove(struct pci_dev *pdev)
 	pci_set_drvdata(pdev, NULL);
 	flush_work(&dev->reset_work);
 	flush_work(&dev->scan_work);
+
+	/*
+	 * If the controller can't do IO (surprise removal, for example), we
+	 * need to disable prior to deleting namespaces. This ends outstanding
+	 * requests and prevents attempts to sync dirty data.
+	 */
+	if (nvme_io_incapable(&dev->ctrl))
+		nvme_dev_disable(dev, true);
+
 	nvme_remove_namespaces(&dev->ctrl);
 	nvme_uninit_ctrl(&dev->ctrl);
 	nvme_dev_disable(dev, true);

--

Oh sorry, but something's actually off with this. I tested removal w/ IO using O_DIRECT by mistake, but the failure this was supposed to fix requires using page cache. I disabled direct in the fio profile, and now getting a hung surprise removal.

I'll have to circle back on this on Monday.