NVMe: Reduce spinlock contention in IO timeout disposition path

PATCH New Sam Bradshaw 1 participant 1 message Nov 17, 2014

Subsystems (2)

Block layer NVMe

lore

Share & tools

lore

Sam Bradshaw

author

2014-11-17 22:54 UTC #0 lore raw

PATCH 1/1 NVMe: Reduce spinlock contention in IO timeout disposition path

When workload queue depths exceed hardware queue depths, the kthread is constantly woken to resubmit queued IOs as cmdids free up. The kthread routine also walks the entire list of cmdids to disposition any timeouts for each queue. This sequence introduces latency variation by extending queue spinlock hold times and is unnecessarily coupled with kthread wakeups. This patch causes the kthread routine to only disposition cmdid timouts if a minimum duration has passed since the last timeout check.

Since multiple devices share a single kthread, this patch has the added benefit of improved latency consistency by reducing the unnecessary spinlock contention introduced by "noisy neighbors" which trigger high rates of kthread wakes.

Signed-off-by Sam Bradshaw <sbradshaw at micron.com>

---

drivers/block/nvme-core.c +10 -1

index 00fa5d2..399d3ac 100644
@@ -1965,9 +1965,17 @@ static int nvme_submit_async_req(struct nvme_queue *nvmeq)
 static int nvme_kthread(void *data)
 {
 	struct nvme_dev *dev, *next;
+	unsigned long do_check, next_check = jiffies;
 
 	while (!kthread_should_stop()) {
 		set_current_state(TASK_INTERRUPTIBLE);
+
+		do_check = 0;
+		if (time_after(jiffies, next_check)) {
+			do_check = 1;
+			next_check = jiffies + HZ;
+		}
+
 		spin_lock(&dev_list_lock);
 		list_for_each_entry_safe(dev, next, &dev_list, node) {
 			int i;

@@ -1992,7 +2000,8 @@ static int nvme_kthread(void *data)
 				if (nvmeq->q_suspended)
 					goto unlock;
 				nvme_process_cq(nvmeq);
-				nvme_cancel_ios(nvmeq, true);
+				if (do_check)
+					nvme_cancel_ios(nvmeq, true);
 				nvme_resubmit_bios(nvmeq);
 				nvme_resubmit_iods(nvmeq);