Skip to contentSkip to conversation

fix interrupt swamp in NVMe

PATCH In review [email protected] 10 participants 30 messages Aug 20, 2019 – Aug 24, 2019

Subsystems (7)
SCSI Block layer PCI Library code +3 more subsystems
lore
Share & tools
lore
#0 lore raw

cover fix interrupt swamp in NVMe

From: Long Li <[email protected]>

This patch set tries to fix interrupt swamp in NVMe devices.

On large systems with many CPUs, a number of CPUs may share one NVMe hardware queue. It may have this situation where several CPUs are issuing I/Os, and all the I/Os are returned on the CPU where the hardware queue is bound to. This may result in that CPU swamped by interrupts and stay in interrupt mode for extended time while other CPUs continue to issue I/O. This can trigger Watchdog and RCU timeout, and make the system unresponsive.

This patch set addresses this by enforcing scheduling and throttling I/O when CPU is starved in this situation.

Long Li (3):
sched: define a function to report the number of context switches on a
CPU
sched: export idle_cpu()
nvme: complete request in work queue on CPU with flooded interrupts

 drivers/nvme/host/core.c | 57 +++++++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/nvme.h |  1 +
 include/linux/sched.h    |  2 ++
 kernel/sched/core.c      |  7 +++++
 4 files changed, 66 insertions(+), 1 deletion(-)
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/3 sched: define a function to report the number of context switches on a CPU

From: Long Li <[email protected]>

The number of context switches on a CPU is useful to determine how busy this CPU is on processing IRQs. Export this information so it can be used by device drivers.

---
 include/linux/sched.h | 1 +
 kernel/sched/core.c   | 6 ++++++
 2 files changed, 7 insertions(+)
include/linux/sched.h +1 -0
index 9b35aff09f70..575f1ef7b159 100644
@@ -1500,6 +1500,7 @@ current_restore_flags(unsigned long orig_flags, unsigned long flags)
 
 extern int cpuset_cpumask_can_shrink(const struct cpumask *cur, const struct cpumask *trial);
 extern int task_can_attach(struct task_struct *p, const struct cpumask *cs_cpus_allowed);
+extern u64 get_cpu_rq_switches(int cpu);
 #ifdef CONFIG_SMP
 extern void do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask);
 extern int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask);
kernel/sched/core.c +6 -0
index 4a8e7207cafa..1a76f0e97c2d 100644
@@ -1143,6 +1143,12 @@ int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask)
 }
 EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr);
 
+u64 get_cpu_rq_switches(int cpu)
+{
+	return cpu_rq(cpu)->nr_switches;
+}
+EXPORT_SYMBOL_GPL(get_cpu_rq_switches);
+
 void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
 {
 #ifdef CONFIG_SCHED_DEBUG
Signature · 1 line
2.17.1
maintainer
#2 lore raw

Re: [PATCH 1/3] sched: define a function to report the number of context switches on a CPU

Quoted 5 lines · On Mon, Aug 19, 2019 at 11:14:27PM -0700, [email protected] wrote:

From: Long Li <[email protected]>

The number of context switches on a CPU is useful to determine how busy this CPU is on processing IRQs. Export this information so it can be used by device drivers.

Please do explain that; because I'm not seeing how number of switches relates to processing IRQs _at_all_!

#3 lore raw

RE: [PATCH 1/3] sched: define a function to report the number of context switches on a CPU

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Subject: Re: [PATCH 1/3] sched: define a function to report the number of context switches on a CPU

On Mon, Aug 19, 2019 at 11:14:27PM -0700, [email protected] wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

From: Long Li <[email protected]>

The number of context switches on a CPU is useful to determine how busy this CPU is on processing IRQs. Export this information so it can be used by device drivers.

Please do explain that; because I'm not seeing how number of switches relates to processing IRQs _at_all_!

Some kernel components rely on context switch to progress, for example watchdog and RCU. On a CPU with reasonable interrupt load, it continues to make context switches, normally a number of switches per seconds.

While observing a CPU with heavy interrupt loads, I see that it spends all its time in IRQ and softIRQ, and not to get a chance to do a switch (calling __schedule()) for a long time. This will result in system unresponsive at times. The purpose is to find out if this CPU is in this state, and implement some throttling mechanism to help reduce the number of interrupts. I think the number of switches is not accurate for detecting this condition in the most precise way, but maybe it's good enough.

I agree this may not be the best way. If you have other idea on detecting a CPU is swamped by interrupts, please point me to where to look at.

Thanks

Long

maintainer
#4 lore raw

Re: [PATCH 1/3] sched: define a function to report the number of context switches on a CPU

Quoted 11 lines · Long Li
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

Subject: Re: [PATCH 1/3] sched: define a function to report the number of context switches on a CPU

Quoted 5 lines · On Mon, Aug 19, 2019 at 11:14:27PM -0700, [email protected] wrote:

From: Long Li <[email protected]>

The number of context switches on a CPU is useful to determine how busy this CPU is on processing IRQs. Export this information so it can be used by device drivers.

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Please do explain that; because I'm not seeing how number of switches relates to processing IRQs _at_all_!

Some kernel components rely on context switch to progress, for example watchdog and RCU. On a CPU with reasonable interrupt load, it continues to make context switches, normally a number of switches per seconds.

That isn't true; RCU is perfectly fine with a single task always running and not making context switches, and so is the watchdog.

maintainer
#5 lore raw

Re: [PATCH 1/3] sched: define a function to report the number of context switches on a CPU

In reply to [email protected]

kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1150

Quoted 5 lines · On Mon, Aug 19, 2019 at 11:14:27PM -0700, [email protected] wrote:
kernel/sched/core.c +5 -0
+u64 get_cpu_rq_switches(int cpu)
+{
+	return cpu_rq(cpu)->nr_switches;
+}
+EXPORT_SYMBOL_GPL(get_cpu_rq_switches);

Also, that is broken on 32bit.

#6 lore raw

PATCH 2/3 sched: export idle_cpu()

In reply to [email protected]

From: Long Li <[email protected]>

This function is useful for device drivers to check if this CPU has work to do in process context.

---
 include/linux/sched.h | 1 +
 kernel/sched/core.c   | 1 +
 2 files changed, 2 insertions(+)
include/linux/sched.h +1 -0
index 575f1ef7b159..a209941c1770 100644
@@ -1501,6 +1501,7 @@ current_restore_flags(unsigned long orig_flags, unsigned long flags)
 extern int cpuset_cpumask_can_shrink(const struct cpumask *cur, const struct cpumask *trial);
 extern int task_can_attach(struct task_struct *p, const struct cpumask *cs_cpus_allowed);
 extern u64 get_cpu_rq_switches(int cpu);
+extern int idle_cpu(int cpu);
 #ifdef CONFIG_SMP
 extern void do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask);
 extern int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask);
kernel/sched/core.c +1 -0
index 1a76f0e97c2d..d1cedfb38174 100644
@@ -4023,6 +4023,7 @@ int idle_cpu(int cpu)
 
 	return 1;
 }
+EXPORT_SYMBOL_GPL(idle_cpu);
 
 /**
  * available_idle_cpu - is a given CPU idle for enqueuing work.
Signature · 1 line
2.17.1
#7 lore raw

PATCH 3/3 nvme: complete request in work queue on CPU with flooded interrupts

In reply to [email protected]

From: Long Li <[email protected]>

When a NVMe hardware queue is mapped to several CPU queues, it is possible that the CPU this hardware queue is bound to is flooded by returning I/O for other CPUs.

For example, consider the following scenario:
1. CPU 0, 1, 2 and 3 share the same hardware queue
2. the hardware queue interrupts CPU 0 for I/O response
3. processes from CPU 1, 2 and 3 keep sending I/Os

CPU 0 may be flooded with interrupts from NVMe device that are I/O responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible that CPU 0 spends all the time serving NVMe and other system interrupts, but doesn't have a chance to run in process context.

To fix this, CPU 0 can schedule a work to complete the I/O request when it detects the scheduler is not making progress. This serves multiple purposes:

1. This CPU has to be scheduled to complete the request. The other CPUs can't issue more I/Os until some previous I/Os are completed. This helps this CPU get out of NVMe interrupts.

2. This acts a throttling mechanisum for NVMe devices, in that it can not starve a CPU while servicing I/Os from other CPUs.

3. This CPU can make progress on RCU and other work items on its queue.

---
 drivers/nvme/host/core.c | 57 +++++++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/nvme.h |  1 +
 2 files changed, 57 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +56 -1
index 6a9dd68c0f4f..576bb6fce293 100644
@@ -28,6 +28,7 @@
 #include <linux/t10-pi.h>
 #include <linux/pm_qos.h>
 #include <asm/unaligned.h>
+#include <linux/sched.h>
 
 #define CREATE_TRACE_POINTS
 #include "trace.h"
@@ -97,6 +98,15 @@ static dev_t nvme_chr_devt;
 static struct class *nvme_class;
 static struct class *nvme_subsys_class;
 
+/*
+ * The following are for detecting if this CPU is flooded with interrupts.
+ * The timestamp for the last context switch is recorded. If that is at least
+ * MAX_SCHED_TIMEOUT ago, try to recover from interrupt flood
+ */
+static DEFINE_PER_CPU(u64, last_switch);
+static DEFINE_PER_CPU(u64, last_clock);
+#define MAX_SCHED_TIMEOUT 2000000000	// 2 seconds in ns
+
 static int nvme_revalidate_disk(struct gendisk *disk);
 static void nvme_put_subsystem(struct nvme_subsystem *subsys);
 static void nvme_remove_invalid_namespaces(struct nvme_ctrl *ctrl,
@@ -260,9 +270,54 @@ static void nvme_retry_req(struct request *req)
 	blk_mq_delay_kick_requeue_list(req->q, delay);
 }
 
+static void nvme_complete_rq_work(struct work_struct *work)
+{
+	struct nvme_request *nvme_rq =
+		container_of(work, struct nvme_request, work);
+	struct request *req = blk_mq_rq_from_pdu(nvme_rq);
+
+	nvme_complete_rq(req);
+}
+
+
 void nvme_complete_rq(struct request *req)
 {
-	blk_status_t status = nvme_error_status(req);
+	blk_status_t status;
+	int cpu;
+	u64 switches;
+	struct nvme_request *nvme_rq;
+
+	if (!in_interrupt())
+		goto skip_check;
+
+	nvme_rq = nvme_req(req);
+	cpu = smp_processor_id();
+	if (idle_cpu(cpu))
+		goto skip_check;
+
+	/* Check if this CPU is flooded with interrupts */
+	switches = get_cpu_rq_switches(cpu);
+	if (this_cpu_read(last_switch) == switches) {
+		/*
+		 * If this CPU hasn't made a context switch in
+		 * MAX_SCHED_TIMEOUT ns (and it's not idle), schedule a work to
+		 * complete this I/O. This forces this CPU run non-interrupt
+		 * code and throttle the other CPU issuing the I/O
+		 */
+		if (sched_clock() - this_cpu_read(last_clock)
+				> MAX_SCHED_TIMEOUT) {
+			INIT_WORK(&nvme_rq->work, nvme_complete_rq_work);
+			schedule_work_on(cpu, &nvme_rq->work);
+			return;
+		}
+
+	} else {
+		this_cpu_write(last_switch, switches);
+		this_cpu_write(last_clock, sched_clock());
+	}
+
+skip_check:
+	status = nvme_error_status(req);
 
 	trace_nvme_complete_rq(req);
 
drivers/nvme/host/nvme.h +1 -0
index 0a4a7f5e0de7..a8876e69e476 100644
@@ -113,6 +113,7 @@ struct nvme_request {
 	u8			flags;
 	u16			status;
 	struct nvme_ctrl	*ctrl;
+	struct work_struct	work;
 };
 
 /*
Signature · 1 line
2.17.1
maintainer
#8 lore raw

Re: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts

Quoted 11 lines · On Mon, Aug 19, 2019 at 11:14:29PM -0700, [email protected] wrote:

From: Long Li <[email protected]>

When a NVMe hardware queue is mapped to several CPU queues, it is possible that the CPU this hardware queue is bound to is flooded by returning I/O for other CPUs.

For example, consider the following scenario:
1. CPU 0, 1, 2 and 3 share the same hardware queue
2. the hardware queue interrupts CPU 0 for I/O response
3. processes from CPU 1, 2 and 3 keep sending I/Os

Quoted 4 lines

CPU 0 may be flooded with interrupts from NVMe device that are I/O responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible that CPU 0 spends all the time serving NVMe and other system interrupts, but doesn't have a chance to run in process context.

Ideally -- and there is some code to affect this, the load-balancer will move tasks away from this CPU.

Quoted 2 lines

To fix this, CPU 0 can schedule a work to complete the I/O request when it detects the scheduler is not making progress. This serves multiple purposes:

Suppose the task waiting for the IO completion is a RT task, and you've just queued it to a regular work. This is an instant priority inversion.

Quoted 7 lines

1. This CPU has to be scheduled to complete the request. The other CPUs can't issue more I/Os until some previous I/Os are completed. This helps this CPU get out of NVMe interrupts.

2. This acts a throttling mechanisum for NVMe devices, in that it can not starve a CPU while servicing I/Os from other CPUs.

Quoted 7 lines

3. This CPU can make progress on RCU and other work items on its queue.

---
 drivers/nvme/host/core.c | 57 +++++++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/nvme.h |  1 +
 2 files changed, 57 insertions(+), 1 deletion(-)

WTH does this live in the NVME driver? Surely something like this should be in the block layer. I'm thinking there's fiber channel connected storage that should be able to trigger much the same issues.

Quoted 34 lines
drivers/nvme/host/core.c +22 -1
index 6a9dd68c0f4f..576bb6fce293 100644
@@ -260,9 +270,54 @@ static void nvme_retry_req(struct request *req)
 	blk_mq_delay_kick_requeue_list(req->q, delay);
 }
 
+static void nvme_complete_rq_work(struct work_struct *work)
+{
+	struct nvme_request *nvme_rq =
+		container_of(work, struct nvme_request, work);
+	struct request *req = blk_mq_rq_from_pdu(nvme_rq);
+
+	nvme_complete_rq(req);
+}
+
+
 void nvme_complete_rq(struct request *req)
 {
-	blk_status_t status = nvme_error_status(req);
+	blk_status_t status;
+	int cpu;
+	u64 switches;
+	struct nvme_request *nvme_rq;
+
+	if (!in_interrupt())
+		goto skip_check;
+
+	nvme_rq = nvme_req(req);
+	cpu = smp_processor_id();
+	if (idle_cpu(cpu))
+		goto skip_check;

drivers/nvme/host/core.c static void nvme_retry_req(struct request *req) · line 306

Quoted 10 lines
drivers/nvme/host/core.c +10 -0
+
+	/* Check if this CPU is flooded with interrupts */
+	switches = get_cpu_rq_switches(cpu);
+	if (this_cpu_read(last_switch) == switches) {
+		/*
+		 * If this CPU hasn't made a context switch in
+		 * MAX_SCHED_TIMEOUT ns (and it's not idle), schedule a work to
+		 * complete this I/O. This forces this CPU run non-interrupt
+		 * code and throttle the other CPU issuing the I/O
+		 */

What if there was only a single task on that CPU? Then we'd never need/want to context switch in the first place.

AFAICT all this is just a whole bunch of gruesome hacks piled on top one another.

drivers/nvme/host/core.c static void nvme_retry_req(struct request *req) · line 319

Quoted 13 lines
drivers/nvme/host/core.c +13 -0
+		if (sched_clock() - this_cpu_read(last_clock)
+				> MAX_SCHED_TIMEOUT) {
+			INIT_WORK(&nvme_rq->work, nvme_complete_rq_work);
+			schedule_work_on(cpu, &nvme_rq->work);
+			return;
+		}
+
+	} else {
+		this_cpu_write(last_switch, switches);
+		this_cpu_write(last_clock, sched_clock());
+	}
+
+skip_check:

Aside from everything else; this is just sodding poor coding style. What is wrong with something like:

	if (nvme_complete_throttle(...))
		return;
Quoted 4 lines
drivers/nvme/host/core.c +1 -0
+	status = nvme_error_status(req);
 
 	trace_nvme_complete_rq(req);
 
#9 lore raw

RE: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts

Quoted 24 lines
Quoted 24 lines
Quoted 24 lines

Subject: Re: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts

Quoted 11 lines · On Mon, Aug 19, 2019 at 11:14:29PM -0700, [email protected] wrote:

From: Long Li <[email protected]>

When a NVMe hardware queue is mapped to several CPU queues, it is possible that the CPU this hardware queue is bound to is flooded by returning I/O for other CPUs.

For example, consider the following scenario:
1. CPU 0, 1, 2 and 3 share the same hardware queue 2. the hardware queue interrupts CPU 0 for I/O response 3. processes from CPU 1, 2 and 3 keep sending I/Os

Quoted 4 lines

CPU 0 may be flooded with interrupts from NVMe device that are I/O responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible that CPU 0 spends all the time serving NVMe and other system interrupts, but doesn't have a chance to run in process context.

Ideally -- and there is some code to affect this, the load-balancer will move tasks away from this CPU.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

To fix this, CPU 0 can schedule a work to complete the I/O request when it detects the scheduler is not making progress. This serves multiple

purposes:

Suppose the task waiting for the IO completion is a RT task, and you've just queued it to a regular work. This is an instant priority inversion.

This is a choice. We can either not "lock up" the CPU, or finish the I/O on time from IRQ handler. I think throttling only happens in extreme conditions, which is rare. The purpose is to make the whole system responsive and happy.

Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines

1. This CPU has to be scheduled to complete the request. The other CPUs can't issue more I/Os until some previous I/Os are completed. This helps this CPU get out of NVMe interrupts.

2. This acts a throttling mechanisum for NVMe devices, in that it can not starve a CPU while servicing I/Os from other CPUs.

3. This CPU can make progress on RCU and other work items on its queue.

---
 drivers/nvme/host/core.c | 57
+++++++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/nvme.h |  1 +
 2 files changed, 57 insertions(+), 1 deletion(-)
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

WTH does this live in the NVME driver? Surely something like this should be in the block layer. I'm thinking there's fiber channel connected storage that should be able to trigger much the same issues.

Yes this can be done in block layer. I'm not sure the best way to accomplish this so implemented a NVMe patch to help test. The test results are promising in that we are getting 99.5% of performance while avoided CPU lockup. The challenge is to find a way to throttle a fast storage device.

Quoted 75 lines
Quoted 75 lines
Quoted 75 lines
Quoted 29 lines
drivers/nvme/host/core.c index renamed from drivers/nvme/host/core.c +0 -0

6a9dd68c0f4f..576bb6fce293 100644

drivers/nvme/host/core.c +18 -1
@@ -260,9 +270,54 @@ static void nvme_retry_req(struct request *req)
 	blk_mq_delay_kick_requeue_list(req->q, delay);  }
 
+static void nvme_complete_rq_work(struct work_struct *work) {
+	struct nvme_request *nvme_rq =
+		container_of(work, struct nvme_request, work);
+	struct request *req = blk_mq_rq_from_pdu(nvme_rq);
+
+	nvme_complete_rq(req);
+}
+
+
 void nvme_complete_rq(struct request *req)  {
-	blk_status_t status = nvme_error_status(req);
+	blk_status_t status;
+	int cpu;
+	u64 switches;
+	struct nvme_request *nvme_rq;
+
+	if (!in_interrupt())
+		goto skip_check;
+
+	nvme_rq = nvme_req(req);

drivers/nvme/host/core.c static void nvme_retry_req(struct request *req) · line 300

Quoted 10 lines
drivers/nvme/host/core.c +10 -0
+	cpu = smp_processor_id();
+	if (idle_cpu(cpu))
+		goto skip_check;
+
+	/* Check if this CPU is flooded with interrupts */
+	switches = get_cpu_rq_switches(cpu);
+	if (this_cpu_read(last_switch) == switches) {
+		/*
+		 * If this CPU hasn't made a context switch in
+		 * MAX_SCHED_TIMEOUT ns (and it's not idle), schedule a

work to

drivers/nvme/host/core.c static void nvme_retry_req(struct request *req) · line 306

Quoted 3 lines
drivers/nvme/host/core.c +3 -0
+		 * complete this I/O. This forces this CPU run non-interrupt
+		 * code and throttle the other CPU issuing the I/O
+		 */

What if there was only a single task on that CPU? Then we'd never need/want to context switch in the first place.

AFAICT all this is just a whole bunch of gruesome hacks piled on top one another.

drivers/nvme/host/core.c static void nvme_retry_req(struct request *req)

Quoted 3 lines
drivers/nvme/host/core.c +3 -0
+		if (sched_clock() - this_cpu_read(last_clock)
+				> MAX_SCHED_TIMEOUT) {
+			INIT_WORK(&nvme_rq->work,

nvme_complete_rq_work);

drivers/nvme/host/core.c static void nvme_retry_req(struct request *req) · line 319

Quoted 10 lines
drivers/nvme/host/core.c +10 -0
+			schedule_work_on(cpu, &nvme_rq->work);
+			return;
+		}
+
+	} else {
+		this_cpu_write(last_switch, switches);
+		this_cpu_write(last_clock, sched_clock());
+	}
+
+skip_check:

Aside from everything else; this is just sodding poor coding style. What is wrong with something like:

	if (nvme_complete_throttle(...))
		return;
Quoted 4 lines
drivers/nvme/host/core.c +1 -0
+	status = nvme_error_status(req);
 
 	trace_nvme_complete_rq(req);
 
maintainer
#10 lore raw

Re: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts

Quoted 24 lines · Long Li
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines

Subject: Re: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts

Quoted 11 lines · On Mon, Aug 19, 2019 at 11:14:29PM -0700, [email protected] wrote:

From: Long Li <[email protected]>

When a NVMe hardware queue is mapped to several CPU queues, it is possible that the CPU this hardware queue is bound to is flooded by returning I/O for other CPUs.

For example, consider the following scenario:
1. CPU 0, 1, 2 and 3 share the same hardware queue 2. the hardware queue interrupts CPU 0 for I/O response 3. processes from CPU 1, 2 and 3 keep sending I/Os

Quoted 4 lines

CPU 0 may be flooded with interrupts from NVMe device that are I/O responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible that CPU 0 spends all the time serving NVMe and other system interrupts, but doesn't have a chance to run in process context.

Ideally -- and there is some code to affect this, the load-balancer will move tasks away from this CPU.

Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

To fix this, CPU 0 can schedule a work to complete the I/O request when it detects the scheduler is not making progress. This serves multiple

purposes:

Suppose the task waiting for the IO completion is a RT task, and you've just queued it to a regular work. This is an instant priority inversion.

This is a choice. We can either not "lock up" the CPU, or finish the I/O on time from IRQ handler. I think throttling only happens in extreme conditions, which is rare. The purpose is to make the whole system responsive and happy.

Can you please use a sane MUA.. this is unreadable garbage.

maintainer
#11 lore raw

Re: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts

In reply to [email protected]
Quoted 20 lines

From: Long Li <[email protected]>

When a NVMe hardware queue is mapped to several CPU queues, it is possible that the CPU this hardware queue is bound to is flooded by returning I/O for other CPUs.

For example, consider the following scenario:
1. CPU 0, 1, 2 and 3 share the same hardware queue
2. the hardware queue interrupts CPU 0 for I/O response
3. processes from CPU 1, 2 and 3 keep sending I/Os

CPU 0 may be flooded with interrupts from NVMe device that are I/O responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible that CPU 0 spends all the time serving NVMe and other system interrupts, but doesn't have a chance to run in process context.

To fix this, CPU 0 can schedule a work to complete the I/O request when it detects the scheduler is not making progress. This serves multiple purposes:

Quoted 8 lines

1. This CPU has to be scheduled to complete the request. The other CPUs can't issue more I/Os until some previous I/Os are completed. This helps this CPU get out of NVMe interrupts.

2. This acts a throttling mechanisum for NVMe devices, in that it can not starve a CPU while servicing I/Os from other CPUs.

3. This CPU can make progress on RCU and other work items on its queue.

The problem is indeed real, but this is the wrong approach in my mind.

We already have irqpoll which takes care proper budgeting polling cycles and not hogging the cpu.

I've sent rfc for this particular problem before [1]. At the time IIRC, Christoph suggested that we will poll the first batch directly from the irq context and reap the rest in irqpoll handler.

[1]: http://lists.infradead.org/pipermail/linux-nvme/2016-October/006497.html

How about something like this instead:
--

drivers/nvme/host/pci.c +27 -1
index 71127a366d3c..84bf16d75109 100644
@@ -24,6 +24,7 @@
 #include <linux/io-64-nonatomic-lo-hi.h>
 #include <linux/sed-opal.h>
 #include <linux/pci-p2pdma.h>
+#include <linux/irq_poll.h>
 
 #include "trace.h"
 #include "nvme.h"
@@ -32,6 +33,7 @@
 #define CQ_SIZE(q)     ((q)->q_depth * sizeof(struct nvme_completion))
 
 #define SGES_PER_PAGE  (PAGE_SIZE / sizeof(struct nvme_sgl_desc))
+#define NVME_POLL_BUDGET_IRQ   256
 
 /*
  * These can be higher, but we need to ensure that any command doesn't
@@ -189,6 +191,7 @@ struct nvme_queue {
        u32 *dbbuf_cq_db;
        u32 *dbbuf_sq_ei;
        u32 *dbbuf_cq_ei;
+       struct irq_poll iop;
        struct completion delete_done;
 };
 
@@ -1015,6 +1018,23 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq, u16 *start,
        return found;
 }
 
+static int nvme_irqpoll_handler(struct irq_poll *iop, int budget)
+{
+       struct nvme_queue *nvmeq = container_of(iop, struct nvme_queue, iop);
+       struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
+       u16 start, end;
+       int completed;
+
+       completed = nvme_process_cq(nvmeq, &start, &end, budget);
+       nvme_complete_cqes(nvmeq, start, end);
+       if (completed < budget) {
+               irq_poll_complete(&nvmeq->iop);
+               enable_irq(pci_irq_vector(pdev, nvmeq->cq_vector));
+       }
+
+       return completed;
+}
+
 static irqreturn_t nvme_irq(int irq, void *data)
 {
        struct nvme_queue *nvmeq = data;
@@ -1028,12 +1048,16 @@ static irqreturn_t nvme_irq(int irq, void *data)
        rmb();
        if (nvmeq->cq_head != nvmeq->last_cq_head)
                ret = IRQ_HANDLED;
-       nvme_process_cq(nvmeq, &start, &end, -1);
+       nvme_process_cq(nvmeq, &start, &end, NVME_POLL_BUDGET_IRQ);
        nvmeq->last_cq_head = nvmeq->cq_head;
        wmb();
 
        if (start != end) {
                nvme_complete_cqes(nvmeq, start, end);
+               if (nvme_cqe_pending(nvmeq)) {
+                       disable_irq_nosync(irq);
+                       irq_poll_sched(&nvmeq->iop);
+               }
                return IRQ_HANDLED;
        }
 
@@ -1347,6 +1371,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)
 
 static void nvme_free_queue(struct nvme_queue *nvmeq)
 {
+       irq_poll_disable(&nvmeq->iop);
        dma_free_coherent(nvmeq->dev->dev, CQ_SIZE(nvmeq),
                                (void *)nvmeq->cqes, nvmeq->cq_dma_addr);
        if (!nvmeq->sq_cmds)
@@ -1481,6 +1506,7 @@ static int nvme_alloc_queue(struct nvme_dev *dev, int qid, int depth)
        nvmeq->dev = dev;
        spin_lock_init(&nvmeq->sq_lock);
        spin_lock_init(&nvmeq->cq_poll_lock);
+       irq_poll_init(&nvmeq->iop, NVME_POLL_BUDGET_IRQ, nvme_irqpoll_handler);
        nvmeq->cq_head = 0;
        nvmeq->cq_phase = 1;
        nvmeq->q_db = &dev->dbs[qid * 2 * dev->db_stride];

--

#12 lore raw

RE: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts

Quoted 33 lines
Quoted 33 lines
Quoted 33 lines

Subject: Re: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts

Quoted 11 lines

From: Long Li <[email protected]>

When a NVMe hardware queue is mapped to several CPU queues, it is possible that the CPU this hardware queue is bound to is flooded by returning I/O for other CPUs.

For example, consider the following scenario:
1. CPU 0, 1, 2 and 3 share the same hardware queue 2. the hardware queue interrupts CPU 0 for I/O response 3. processes from CPU 1, 2 and 3 keep sending I/Os

Quoted 7 lines

CPU 0 may be flooded with interrupts from NVMe device that are I/O responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible that CPU 0 spends all the time serving NVMe and other system interrupts, but doesn't have a chance to run in process context.

To fix this, CPU 0 can schedule a work to complete the I/O request when it detects the scheduler is not making progress. This serves multiple

purposes:

Quoted 9 lines

1. This CPU has to be scheduled to complete the request. The other CPUs can't issue more I/Os until some previous I/Os are completed. This helps this CPU get out of NVMe interrupts.

2. This acts a throttling mechanisum for NVMe devices, in that it can not starve a CPU while servicing I/Os from other CPUs.

3. This CPU can make progress on RCU and other work items on its queue.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

The problem is indeed real, but this is the wrong approach in my mind.

We already have irqpoll which takes care proper budgeting polling cycles and not hogging the cpu.

I've sent rfc for this particular problem before [1]. At the time IIRC, Christoph suggested that we will poll the first batch directly from the irq context and reap the rest in irqpoll handler.

Thanks for the pointer. I will test and report back.

Quoted 102 lines
Quoted 102 lines
Quoted 102 lines

[1]:
https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.
infradead.org%2Fpipermail%2Flinux-nvme%2F2016- October%2F006497.html&amp;data=02%7C01%7Clongli%40microsoft.com% 7C0ebf36eff15c4182116608d725948b93%7C72f988bf86f141af91ab2d7cd011d b47%7C1%7C0%7C637019192254250361&amp;sdata=fJ%2Fkc8HLSmfzaY3BY E66zlZKD6FjcXgMJZzVGCVqI%2FU%3D&amp;reserved=0

How about something like this instead:
--

drivers/nvme/host/pci.c index renamed from drivers/nvme/host/pci.c +0 -0

71127a366d3c..84bf16d75109 100644

drivers/nvme/host/pci.c +19 -0
@@ -24,6 +24,7 @@
 #include <linux/io-64-nonatomic-lo-hi.h>
 #include <linux/sed-opal.h>
 #include <linux/pci-p2pdma.h>
+#include <linux/irq_poll.h>
 
 #include "trace.h"
 #include "nvme.h"
@@ -32,6 +33,7 @@
 #define CQ_SIZE(q)     ((q)->q_depth * sizeof(struct nvme_completion))
 
 #define SGES_PER_PAGE  (PAGE_SIZE / sizeof(struct nvme_sgl_desc))
+#define NVME_POLL_BUDGET_IRQ   256
 
 /*
  * These can be higher, but we need to ensure that any command doesn't
@@ -189,6 +191,7 @@ struct nvme_queue {
        u32 *dbbuf_cq_db;
        u32 *dbbuf_sq_ei;
        u32 *dbbuf_cq_ei;
+       struct irq_poll iop;
        struct completion delete_done;
 };
 
@@ -1015,6 +1018,23 @@ static inline int nvme_process_cq(struct
 nvme_queue *nvmeq, u16 *start,
        return found;
 }
 
+static int nvme_irqpoll_handler(struct irq_poll *iop, int budget) {
+       struct nvme_queue *nvmeq = container_of(iop, struct nvme_queue,
 iop);
+       struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
+       u16 start, end;
+       int completed;
+
+       completed = nvme_process_cq(nvmeq, &start, &end, budget);
+       nvme_complete_cqes(nvmeq, start, end);
+       if (completed < budget) {
+               irq_poll_complete(&nvmeq->iop);
+               enable_irq(pci_irq_vector(pdev, nvmeq->cq_vector));
+       }
+
+       return completed;
+}
+
 static irqreturn_t nvme_irq(int irq, void *data)
 {
        struct nvme_queue *nvmeq = data; @@ -1028,12 +1048,16 @@ static

irqreturn_t nvme_irq(int irq, void *data)

        rmb();
        if (nvmeq->cq_head != nvmeq->last_cq_head)
                ret = IRQ_HANDLED;
drivers/nvme/host/pci.c +6 -1
-       nvme_process_cq(nvmeq, &start, &end, -1);
+       nvme_process_cq(nvmeq, &start, &end, NVME_POLL_BUDGET_IRQ);
        nvmeq->last_cq_head = nvmeq->cq_head;
        wmb();
 
        if (start != end) {
                nvme_complete_cqes(nvmeq, start, end);
+               if (nvme_cqe_pending(nvmeq)) {
+                       disable_irq_nosync(irq);
+                       irq_poll_sched(&nvmeq->iop);
+               }
                return IRQ_HANDLED;
        }
@@ -1347,6 +1371,7 @@ static enum blk_eh_timer_return
 nvme_timeout(struct request *req, bool reserved)
 
 static void nvme_free_queue(struct nvme_queue *nvmeq)
 {
+       irq_poll_disable(&nvmeq->iop);
        dma_free_coherent(nvmeq->dev->dev, CQ_SIZE(nvmeq),
                                (void *)nvmeq->cqes, nvmeq->cq_dma_addr);
        if (!nvmeq->sq_cmds)
drivers/nvme/host/pci.c +1 -0
@@ -1481,6 +1506,7 @@ static int nvme_alloc_queue(struct nvme_dev
 *dev, int qid, int depth)
        nvmeq->dev = dev;
        spin_lock_init(&nvmeq->sq_lock);
        spin_lock_init(&nvmeq->cq_poll_lock);
+       irq_poll_init(&nvmeq->iop, NVME_POLL_BUDGET_IRQ,
 nvme_irqpoll_handler);
        nvmeq->cq_head = 0;
        nvmeq->cq_phase = 1;
        nvmeq->q_db = &dev->dbs[qid * 2 * dev->db_stride];

--

#13 lore raw

RE: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts

Quoted 42 lines
Quoted 42 lines
Quoted 42 lines

Subject: RE: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts

Quoted 1 line
Quoted 1 line
Quoted 1 line

Subject: Re: [PATCH 3/3] nvme: complete request in work queue on

CPU

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines

with flooded interrupts

Quoted 12 lines

From: Long Li <[email protected]>

When a NVMe hardware queue is mapped to several CPU queues, it is possible that the CPU this hardware queue is bound to is flooded by returning I/O for other CPUs.

For example, consider the following scenario:
1. CPU 0, 1, 2 and 3 share the same hardware queue 2. the hardware queue interrupts CPU 0 for I/O response 3. processes from CPU 1, 2 and
3 keep sending I/Os

Quoted 8 lines

CPU 0 may be flooded with interrupts from NVMe device that are I/O responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible that CPU 0 spends all the time serving NVMe and other system interrupts, but doesn't have a chance to run in process context.

To fix this, CPU 0 can schedule a work to complete the I/O request when it detects the scheduler is not making progress. This serves multiple

purposes:

Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

1. This CPU has to be scheduled to complete the request. The other CPUs can't issue more I/Os until some previous I/Os are completed. This helps this CPU get out of NVMe interrupts.

2. This acts a throttling mechanisum for NVMe devices, in that it can not starve a CPU while servicing I/Os from other CPUs.

3. This CPU can make progress on RCU and other work items on its

queue.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

The problem is indeed real, but this is the wrong approach in my mind.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

We already have irqpoll which takes care proper budgeting polling cycles and not hogging the cpu.

I've sent rfc for this particular problem before [1]. At the time IIRC, Christoph suggested that we will poll the first batch directly from the irq context and reap the rest in irqpoll handler.

Thanks for the pointer. I will test and report back.

Sagi,

Here are the test results.

Benchmark command:
fio --bs=4k --ioengine=libaio --iodepth=64 --filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1:/dev/nvme4n1:/dev/nvme5n1:/dev/nvme6n1:/dev/nvme7n1:/dev/nvme8n1:/dev/nvme9n1 --direct=1 --runtime=90 --numjobs=80 --rw=randread --name=test --group_reporting --gtod_reduce=1

With your patch: 1720k IOPS
With threaded interrupts: 1320k IOPS
With just interrupts: 3720k IOPS

Interrupts are the fastest but we need to find a way to throttle it.

Thanks

Long

Quoted 107 lines
Quoted 107 lines
Quoted 107 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

[1]:
https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fl

ists.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

infradead.org%2Fpipermail%2Flinux-nvme%2F2016- October%2F006497.html&amp;data=02%7C01%7Clongli%40microsoft.co

m%

Quoted 1 line
Quoted 1 line
Quoted 1 line

7C0ebf36eff15c4182116608d725948b93%7C72f988bf86f141af91ab2d7cd0

11d

Quoted 1 line
Quoted 1 line
Quoted 1 line

b47%7C1%7C0%7C637019192254250361&amp;sdata=fJ%2Fkc8HLSmfzaY

3BY

Quoted 55 lines
Quoted 55 lines
Quoted 55 lines

E66zlZKD6FjcXgMJZzVGCVqI%2FU%3D&amp;reserved=0

How about something like this instead:
--

drivers/nvme/host/pci.c index renamed from drivers/nvme/host/pci.c +0 -0

71127a366d3c..84bf16d75109 100644

drivers/nvme/host/pci.c +2 -0
@@ -24,6 +24,7 @@
 #include <linux/io-64-nonatomic-lo-hi.h>
 #include <linux/sed-opal.h>
 #include <linux/pci-p2pdma.h>
+#include <linux/irq_poll.h>
 
 #include "trace.h"
 #include "nvme.h"
@@ -32,6 +33,7 @@
 #define CQ_SIZE(q)     ((q)->q_depth * sizeof(struct nvme_completion))
 
 #define SGES_PER_PAGE  (PAGE_SIZE / sizeof(struct nvme_sgl_desc))
+#define NVME_POLL_BUDGET_IRQ   256
 
 /*
  * These can be higher, but we need to ensure that any command
doesn't @@ -189,6 +191,7 @@ struct nvme_queue {
        u32 *dbbuf_cq_db;
        u32 *dbbuf_sq_ei;
        u32 *dbbuf_cq_ei;
drivers/nvme/host/pci.c +18 -0
+       struct irq_poll iop;
        struct completion delete_done;  };
@@ -1015,6 +1018,23 @@ static inline int nvme_process_cq(struct
 nvme_queue *nvmeq, u16 *start,
        return found;
 }
 
+static int nvme_irqpoll_handler(struct irq_poll *iop, int budget) {
+       struct nvme_queue *nvmeq = container_of(iop, struct
+nvme_queue,
 iop);
+       struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
+       u16 start, end;
+       int completed;
+
+       completed = nvme_process_cq(nvmeq, &start, &end, budget);
+       nvme_complete_cqes(nvmeq, start, end);
+       if (completed < budget) {
+               irq_poll_complete(&nvmeq->iop);
+               enable_irq(pci_irq_vector(pdev, nvmeq->cq_vector));
+       }
+
+       return completed;
+}
+
 static irqreturn_t nvme_irq(int irq, void *data)
Quoted 8 lines
Quoted 8 lines

drivers/nvme/host/pci.c

Quoted 8 lines
 {
        struct nvme_queue *nvmeq = data; @@ -1028,12 +1048,16 @@
static irqreturn_t nvme_irq(int irq, void *data)
        rmb();
        if (nvmeq->cq_head != nvmeq->last_cq_head)
                ret = IRQ_HANDLED;
drivers/nvme/host/pci.c +1 -1
-       nvme_process_cq(nvmeq, &start, &end, -1);
+       nvme_process_cq(nvmeq, &start, &end,

NVME_POLL_BUDGET_IRQ);

Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
        nvmeq->last_cq_head = nvmeq->cq_head;
        wmb();

        if (start != end) {
                nvme_complete_cqes(nvmeq, start, end);
drivers/nvme/host/pci.c +6 -0
+               if (nvme_cqe_pending(nvmeq)) {
+                       disable_irq_nosync(irq);
+                       irq_poll_sched(&nvmeq->iop);
+               }
                return IRQ_HANDLED;
        }
@@ -1347,6 +1371,7 @@ static enum blk_eh_timer_return
 nvme_timeout(struct request *req, bool reserved)
 
 static void nvme_free_queue(struct nvme_queue *nvmeq)  {
+       irq_poll_disable(&nvmeq->iop);
        dma_free_coherent(nvmeq->dev->dev, CQ_SIZE(nvmeq),
                                (void *)nvmeq->cqes, nvmeq->cq_dma_addr);
        if (!nvmeq->sq_cmds)
@@ -1481,6 +1506,7 @@ static int nvme_alloc_queue(struct nvme_dev
 *dev, int qid, int depth)
        nvmeq->dev = dev;
        spin_lock_init(&nvmeq->sq_lock);
        spin_lock_init(&nvmeq->cq_poll_lock);
+       irq_poll_init(&nvmeq->iop, NVME_POLL_BUDGET_IRQ,
 nvme_irqpoll_handler);
        nvmeq->cq_head = 0;
        nvmeq->cq_phase = 1;
        nvmeq->q_db = &dev->dbs[qid * 2 * dev->db_stride];

--

maintainer
#14 lore raw

Re: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts

Quoted 5 lines

Sagi,

Here are the test results.

Quoted 8 lines

Benchmark command:
fio --bs=4k --ioengine=libaio --iodepth=64 --filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1:/dev/nvme4n1:/dev/nvme5n1:/dev/nvme6n1:/dev/nvme7n1:/dev/nvme8n1:/dev/nvme9n1 --direct=1 --runtime=90 --numjobs=80 --rw=randread --name=test --group_reporting --gtod_reduce=1

With your patch: 1720k IOPS
With threaded interrupts: 1320k IOPS
With just interrupts: 3720k IOPS

Interrupts are the fastest but we need to find a way to throttle it.

This is the workload that generates the flood?
If so I did not expect that this would be the perf difference..

If completions keep pounding on the cpu, I would expect irqpoll to simply keep running forever and poll the cqs. There is no fundamental reason why polling would be faster in an interrupt, what matters could be:
1. we reschedule more than we need to
2. we don't reap enough completions in every poll round, which will trigger rearming the interrupt and then when it fires reschedule another softirq...

Maybe we need to take care of some irq_poll optimizations?

Does this (untested) patch make any difference?
--

lib/irq_poll.c +11 -18
index 2f17b488d58e..0e94183eba15 100644
@@ -12,7 +12,8 @@
 #include <linux/irq_poll.h>
 #include <linux/delay.h>
 
-static unsigned int irq_poll_budget __read_mostly = 256;
+static unsigned int irq_poll_budget __read_mostly = 3000;
+unsigned int __read_mostly irqpoll_budget_usecs = 2000;
 
 static DEFINE_PER_CPU(struct list_head, blk_cpu_iopoll);
 
@@ -77,32 +78,26 @@ EXPORT_SYMBOL(irq_poll_complete);
 
 static void __latent_entropy irq_poll_softirq(struct softirq_action *h)
 {
-       struct list_head *list = this_cpu_ptr(&blk_cpu_iopoll);
-       int rearm = 0, budget = irq_poll_budget;
-       unsigned long start_time = jiffies;
+       struct list_head *irqpoll_list = this_cpu_ptr(&blk_cpu_iopoll);
+       unsigned int budget = irq_poll_budget;
+       unsigned long time_limit =
+               jiffies + usecs_to_jiffies(irqpoll_budget_usecs);
+       LIST_HEAD(list);
 
        local_irq_disable();
+       list_splice_init(irqpoll_list, &list);
+       local_irq_enable();
 
-       while (!list_empty(list)) {
+       while (!list_empty(&list)) {
                struct irq_poll *iop;
                int work, weight;
 
-               /*
-                * If softirq window is exhausted then punt.
-                */
-               if (budget <= 0 || time_after(jiffies, start_time)) {
-                       rearm = 1;
-                       break;
-               }
-
-               local_irq_enable();
-
                /* Even though interrupts have been re-enabled, this
                 * access is safe because interrupts can only add new
                 * entries to the tail of this list, and only ->poll()
                 * calls can remove this head entry from the list.
                 */
-               iop = list_entry(list->next, struct irq_poll, list);
+               iop = list_first_entry(&list, struct irq_poll, list);
 
                weight = iop->weight;
                work = 0;
@@ -111,8 +106,6 @@ static void __latent_entropy irq_poll_softirq(struct softirq_action *h)
 
                budget -= work;
 
-               local_irq_disable();
-
                /*
                 * Drivers must not modify the iopoll state, if they
                 * consume their assigned weight (or more, some drivers 

can't
@@ -125,11 +118,21 @@ static void __latent_entropy irq_poll_softirq(struct softirq_action *h)

                        if (test_bit(IRQ_POLL_F_DISABLE, &iop->state))
                                __irq_poll_complete(iop);
                        else
-                               list_move_tail(&iop->list, list);
+                               list_move_tail(&iop->list, &list);
                }

+
+ /*
+ * If softirq window is exhausted then punt.
+ */
+ if (budget <= 0 || time_after_eq(jiffies, time_limit))

+                       break;
        }

- if (rearm)
+ local_irq_disable();
+

+       list_splice_tail_init(irqpoll_list, &list);
+       list_splice(&list, irqpoll_list);
+       if (!list_empty(irqpoll_list))
                __raise_softirq_irqoff(IRQ_POLL_SOFTIRQ);

        local_irq_enable();

--

#15 lore raw

RE: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts

Quoted 28 lines
Quoted 28 lines
Quoted 28 lines

Subject: Re: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts

Quoted 7 lines

Sagi,

Here are the test results.

Benchmark command:
fio --bs=4k --ioengine=libaio --iodepth=64
--

filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1:/d ev/nv

Quoted 1 line

me4n1:/dev/nvme5n1:/dev/nvme6n1:/dev/nvme7n1:/dev/nvme8n1:/dev /nvme9n1

Quoted 8 lines

--direct=1 --runtime=90 --numjobs=80 --rw=randread --name=test --group_reporting --gtod_reduce=1

With your patch: 1720k IOPS
With threaded interrupts: 1320k IOPS
With just interrupts: 3720k IOPS

Interrupts are the fastest but we need to find a way to throttle it.

This is the workload that generates the flood?
If so I did not expect that this would be the perf difference..

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

If completions keep pounding on the cpu, I would expect irqpoll to simply keep running forever and poll the cqs. There is no fundamental reason why polling would be faster in an interrupt, what matters could be:
1. we reschedule more than we need to
2. we don't reap enough completions in every poll round, which will trigger rearming the interrupt and then when it fires reschedule another softirq...

Yes I think it's the rescheduling that takes some. With the patch there are lots of ksoftirqd activities. (compared to nearly none without the patch) A 90 seconds FIO run shows a big difference of context switches on all CPUs:
With patch: 5755849
Without patch: 1462931

Quoted 94 lines
Quoted 94 lines
Quoted 94 lines

Maybe we need to take care of some irq_poll optimizations?

Does this (untested) patch make any difference?
--

lib/irq_poll.c index 2f17b488d58e..0e94183eba15 renamed from lib/irq_poll.c +0 -0

100644

lib/irq_poll.c +11 -18
@@ -12,7 +12,8 @@
 #include <linux/irq_poll.h>
 #include <linux/delay.h>
 
-static unsigned int irq_poll_budget __read_mostly = 256;
+static unsigned int irq_poll_budget __read_mostly = 3000; unsigned int
+__read_mostly irqpoll_budget_usecs = 2000;
 
 static DEFINE_PER_CPU(struct list_head, blk_cpu_iopoll);
 
@@ -77,32 +78,26 @@ EXPORT_SYMBOL(irq_poll_complete);
 
 static void __latent_entropy irq_poll_softirq(struct softirq_action *h)
 {
-       struct list_head *list = this_cpu_ptr(&blk_cpu_iopoll);
-       int rearm = 0, budget = irq_poll_budget;
-       unsigned long start_time = jiffies;
+       struct list_head *irqpoll_list = this_cpu_ptr(&blk_cpu_iopoll);
+       unsigned int budget = irq_poll_budget;
+       unsigned long time_limit =
+               jiffies + usecs_to_jiffies(irqpoll_budget_usecs);
+       LIST_HEAD(list);
 
        local_irq_disable();
+       list_splice_init(irqpoll_list, &list);
+       local_irq_enable();
 
-       while (!list_empty(list)) {
+       while (!list_empty(&list)) {
                struct irq_poll *iop;
                int work, weight;
 
-               /*
-                * If softirq window is exhausted then punt.
-                */
-               if (budget <= 0 || time_after(jiffies, start_time)) {
-                       rearm = 1;
-                       break;
-               }
-
-               local_irq_enable();
-
                /* Even though interrupts have been re-enabled, this
                 * access is safe because interrupts can only add new
                 * entries to the tail of this list, and only ->poll()
                 * calls can remove this head entry from the list.
                 */
-               iop = list_entry(list->next, struct irq_poll, list);
+               iop = list_first_entry(&list, struct irq_poll, list);
 
                weight = iop->weight;
                work = 0;
@@ -111,8 +106,6 @@ static void __latent_entropy irq_poll_softirq(struct
 softirq_action *h)
 
                budget -= work;
 
-               local_irq_disable();
-
                /*
                 * Drivers must not modify the iopoll state, if they

* consume their assigned weight (or more, some drivers can't @@
-125,11 +118,21 @@ static void __latent_entropy irq_poll_softirq(struct softirq_action *h)

                        if (test_bit(IRQ_POLL_F_DISABLE, &iop->state))
                                __irq_poll_complete(iop);
                        else
lib/irq_poll.c +12 -2
-                               list_move_tail(&iop->list, list);
+                               list_move_tail(&iop->list, &list);
                }
+
+               /*
+                * If softirq window is exhausted then punt.
+                */
+               if (budget <= 0 || time_after_eq(jiffies, time_limit))
+                       break;
        }
 
-       if (rearm)
+       local_irq_disable();
+
+       list_splice_tail_init(irqpoll_list, &list);
+       list_splice(&list, irqpoll_list);
+       if (!list_empty(irqpoll_list))
                __raise_softirq_irqoff(IRQ_POLL_SOFTIRQ);
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
        local_irq_enable();

--

It's got slightly better IOPS. With this 2nd patch, the number of context switches is at 5445863 (~5% improvement over the 1st patch).

#16 lore raw

Re: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts

In reply to Sagi Grimberg
Quoted 27 lines · Sagi Grimberg
Quoted 27 lines

From: Long Li <[email protected]>

When a NVMe hardware queue is mapped to several CPU queues, it is possible that the CPU this hardware queue is bound to is flooded by returning I/O for other CPUs.

For example, consider the following scenario:
1. CPU 0, 1, 2 and 3 share the same hardware queue
2. the hardware queue interrupts CPU 0 for I/O response
3. processes from CPU 1, 2 and 3 keep sending I/Os

CPU 0 may be flooded with interrupts from NVMe device that are I/O responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible that CPU 0 spends all the time serving NVMe and other system interrupts, but doesn't have a chance to run in process context.

To fix this, CPU 0 can schedule a work to complete the I/O request when it detects the scheduler is not making progress. This serves multiple purposes:

1. This CPU has to be scheduled to complete the request. The other CPUs can't issue more I/Os until some previous I/Os are completed. This helps this CPU get out of NVMe interrupts.

2. This acts a throttling mechanisum for NVMe devices, in that it can not starve a CPU while servicing I/Os from other CPUs.

Quoted 6 lines
Quoted 1 line

3. This CPU can make progress on RCU and other work items on its queue.

The problem is indeed real, but this is the wrong approach in my mind.

We already have irqpoll which takes care proper budgeting polling cycles and not hogging the cpu.

The issue isn't unique to NVMe, and can be any fast devices which interrupts CPU too frequently, meantime the interrupt/softirq handler may take a bit much time, then CPU is easy to be lockup by the interrupt/sofirq handler, especially in case that multiple submission CPUs vs. single completion CPU.

Some SCSI devices has the same problem too.

Could we consider to add one generic mechanism to cover this kind of problem?

One approach I thought of is to allocate one backup thread for handling such interrupt, which can be marked as IRQF_BACKUP_THREAD by drivers.

Inside do_IRQ(), irqtime is accounted, before calling action->handler(), check if this CPU has taken too long time for handling IRQ(interrupt or softirq) and see if this CPU could be lock up. If yes, wakeup the backup thread to handle the interrupt for avoiding lockup this CPU.

The threaded interrupt framework is there, and this way could be easier to implement. Meantime most time the handler is run in interrupt context and we may avoid the performance loss when CPU isn't busy enough.

Any comment on this approach?

Thanks,
Ming

#17 lore raw

RE: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts

Quoted 51 lines
Quoted 51 lines
Quoted 51 lines

Subject: Re: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts

Quoted 12 lines · Sagi Grimberg
Quoted 12 lines

From: Long Li <[email protected]>

When a NVMe hardware queue is mapped to several CPU queues, it is possible that the CPU this hardware queue is bound to is flooded by returning I/O for other CPUs.

For example, consider the following scenario:
1. CPU 0, 1, 2 and 3 share the same hardware queue 2. the hardware queue interrupts CPU 0 for I/O response 3. processes from CPU 1, 2 and 3 keep sending I/Os

Quoted 7 lines
Quoted 7 lines

CPU 0 may be flooded with interrupts from NVMe device that are I/O responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible that CPU 0 spends all the time serving NVMe and other system interrupts, but doesn't have a chance to run in process context.

To fix this, CPU 0 can schedule a work to complete the I/O request when it detects the scheduler is not making progress. This serves

multiple purposes:

Quoted 9 lines
Quoted 9 lines

1. This CPU has to be scheduled to complete the request. The other CPUs can't issue more I/Os until some previous I/Os are completed. This helps this CPU get out of NVMe interrupts.

2. This acts a throttling mechanisum for NVMe devices, in that it can not starve a CPU while servicing I/Os from other CPUs.

3. This CPU can make progress on RCU and other work items on its

queue.

Quoted 5 lines

The problem is indeed real, but this is the wrong approach in my mind.

We already have irqpoll which takes care proper budgeting polling cycles and not hogging the cpu.

The issue isn't unique to NVMe, and can be any fast devices which interrupts CPU too frequently, meantime the interrupt/softirq handler may take a bit much time, then CPU is easy to be lockup by the interrupt/sofirq handler, especially in case that multiple submission CPUs vs. single completion CPU.

Some SCSI devices has the same problem too.

Could we consider to add one generic mechanism to cover this kind of problem?

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

One approach I thought of is to allocate one backup thread for handling such interrupt, which can be marked as IRQF_BACKUP_THREAD by drivers.

Inside do_IRQ(), irqtime is accounted, before calling action->handler(), check if this CPU has taken too long time for handling IRQ(interrupt or softirq) and see if this CPU could be lock up. If yes, wakeup the backup

How do you know if this CPU is spending all the time in do_IRQ()?

Is it something like:
If (IRQ_time /elapsed_time > a threshold value)
wake up the backup thread

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

thread to handle the interrupt for avoiding lockup this CPU.

The threaded interrupt framework is there, and this way could be easier to implement. Meantime most time the handler is run in interrupt context and we may avoid the performance loss when CPU isn't busy enough.

Any comment on this approach?

Thanks,
Ming

#18 lore raw

Re: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts

Quoted 58 lines · Long Li
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines

Subject: Re: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts

Quoted 12 lines · Sagi Grimberg
Quoted 12 lines

From: Long Li <[email protected]>

When a NVMe hardware queue is mapped to several CPU queues, it is possible that the CPU this hardware queue is bound to is flooded by returning I/O for other CPUs.

For example, consider the following scenario:
1. CPU 0, 1, 2 and 3 share the same hardware queue 2. the hardware queue interrupts CPU 0 for I/O response 3. processes from CPU 1, 2 and 3 keep sending I/Os

Quoted 7 lines
Quoted 7 lines

CPU 0 may be flooded with interrupts from NVMe device that are I/O responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible that CPU 0 spends all the time serving NVMe and other system interrupts, but doesn't have a chance to run in process context.

To fix this, CPU 0 can schedule a work to complete the I/O request when it detects the scheduler is not making progress. This serves

multiple purposes:

Quoted 9 lines
Quoted 9 lines

1. This CPU has to be scheduled to complete the request. The other CPUs can't issue more I/Os until some previous I/Os are completed. This helps this CPU get out of NVMe interrupts.

2. This acts a throttling mechanisum for NVMe devices, in that it can not starve a CPU while servicing I/Os from other CPUs.

3. This CPU can make progress on RCU and other work items on its

queue.

Quoted 5 lines

The problem is indeed real, but this is the wrong approach in my mind.

We already have irqpoll which takes care proper budgeting polling cycles and not hogging the cpu.

The issue isn't unique to NVMe, and can be any fast devices which interrupts CPU too frequently, meantime the interrupt/softirq handler may take a bit much time, then CPU is easy to be lockup by the interrupt/sofirq handler, especially in case that multiple submission CPUs vs. single completion CPU.

Some SCSI devices has the same problem too.

Could we consider to add one generic mechanism to cover this kind of problem?

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

One approach I thought of is to allocate one backup thread for handling such interrupt, which can be marked as IRQF_BACKUP_THREAD by drivers.

Inside do_IRQ(), irqtime is accounted, before calling action->handler(), check if this CPU has taken too long time for handling IRQ(interrupt or softirq) and see if this CPU could be lock up. If yes, wakeup the backup

Quoted 5 lines

How do you know if this CPU is spending all the time in do_IRQ()?

Is it something like:
If (IRQ_time /elapsed_time > a threshold value)
wake up the backup thread

Yeah, the above could work in theory.

Another approach I thought of is to monitor average irq gap time on each CPU.

We could use EWMA(Exponential Weighted Moving Average) to do it simply, such as:

curr_irq_gap(cpu) = current start time of do_IRQ() on 'cpu' -
end time of last do_IRQ() on 'cpu'
avg_irq_gap(cpu) = weight_prev * avg_irq_gap(cpu) + weight_curr * curr_irq_gap(cpu)

note:
weight_prev + weight_curr = 1

When avg_irq_gap(cpu) is close to one small enough threshold, we think irq flood is detected.

'weight_prev' could be chosen as one big enough value for avoiding short-time flood.

Thanks,
Ming

#19 lore raw
In reply to [email protected]
Quoted 21 lines · On Tue, Aug 20, 2019 at 2:14 PM <[email protected]> wrote:

From: Long Li <[email protected]>

This patch set tries to fix interrupt swamp in NVMe devices.

On large systems with many CPUs, a number of CPUs may share one NVMe hardware queue. It may have this situation where several CPUs are issuing I/Os, and all the I/Os are returned on the CPU where the hardware queue is bound to. This may result in that CPU swamped by interrupts and stay in interrupt mode for extended time while other CPUs continue to issue I/O. This can trigger Watchdog and RCU timeout, and make the system unresponsive.

This patch set addresses this by enforcing scheduling and throttling I/O when CPU is starved in this situation.

Long Li (3):
sched: define a function to report the number of context switches on a
CPU
sched: export idle_cpu()
nvme: complete request in work queue on CPU with flooded interrupts

Quoted 5 lines
 drivers/nvme/host/core.c | 57 +++++++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/nvme.h |  1 +
 include/linux/sched.h    |  2 ++
 kernel/sched/core.c      |  7 +++++
 4 files changed, 66 insertions(+), 1 deletion(-)

Another simpler solution may be to complete request in threaded interrupt handler for this case. Meantime allow scheduler to run the interrupt thread handler on CPUs specified by the irq affinity mask, which was discussed by the following link:

https://lore.kernel.org/lkml/[email protected]/

Could you try the above solution and see if the lockup can be avoided? John Garry
should have workable patch.

Thanks,
Ming Lei

#20 lore raw
Quoted 33 lines · Ming Lei
Quoted 21 lines · On Tue, Aug 20, 2019 at 2:14 PM <[email protected]> wrote:

From: Long Li <[email protected]>

This patch set tries to fix interrupt swamp in NVMe devices.

On large systems with many CPUs, a number of CPUs may share one NVMe hardware queue. It may have this situation where several CPUs are issuing I/Os, and all the I/Os are returned on the CPU where the hardware queue is bound to. This may result in that CPU swamped by interrupts and stay in interrupt mode for extended time while other CPUs continue to issue I/O. This can trigger Watchdog and RCU timeout, and make the system unresponsive.

This patch set addresses this by enforcing scheduling and throttling I/O when CPU is starved in this situation.

Long Li (3):
sched: define a function to report the number of context switches on a
CPU
sched: export idle_cpu()
nvme: complete request in work queue on CPU with flooded interrupts

Quoted 5 lines
 drivers/nvme/host/core.c | 57 +++++++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/nvme.h |  1 +
 include/linux/sched.h    |  2 ++
 kernel/sched/core.c      |  7 +++++
 4 files changed, 66 insertions(+), 1 deletion(-)

Another simpler solution may be to complete request in threaded interrupt handler for this case. Meantime allow scheduler to run the interrupt thread handler on CPUs specified by the irq affinity mask, which was discussed by the following link:

Quoted 5 lines

https://lore.kernel.org/lkml/[email protected]/

Could you try the above solution and see if the lockup can be avoided? John Garry
should have workable patch.

Yeah, so we experimented with changing the interrupt handling in the SCSI driver I maintain to use a threaded handler IRQ handler plus patch below, and saw a significant throughput boost:

--->8

Subject: [PATCH] genirq: Add support to allow thread to use hard irq affinity

Currently the cpu allowed mask for the threaded part of a threaded irq handler will be set to the effective affinity of the hard irq.

Typically the effective affinity of the hard irq will be for a single cpu. As such, the threaded handler would always run on the same cpu as the hard irq.

We have seen scenarios in high data-rate throughput testing that the cpu handling the interrupt can be totally saturated handling both the hard interrupt and threaded handler parts, limiting throughput.

Add IRQF_IRQ_AFFINITY flag to allow the driver requesting the threaded interrupt to decide on the policy of which cpu the threaded handler may run.

include/linux/interrupt.h +0 -0
index 5b8328a99b2a..48e8b955989a 100644
@@ -61,6 +61,9 @@
  *                interrupt handler after suspending interrupts. For 

system
* wakeup devices users need to implement wakeup
detection in
* their interrupt handlers.
+ * IRQF_IRQ_AFFINITY - Use the hard interrupt affinity for setting the cpu
+ * allowed mask for the threaded handler of a threaded interrupt
+ * handler, rather than the effective hard irq affinity.
*/
#define IRQF_SHARED 0x00000080
#define IRQF_PROBE_SHARED 0x00000100
@@ -74,6 +77,7 @@
#define IRQF_NO_THREAD 0x00010000
#define IRQF_EARLY_RESUME 0x00020000
#define IRQF_COND_SUSPEND 0x00040000
+#define IRQF_IRQ_AFFINITY 0x00080000

#define IRQF_TIMER (__IRQF_TIMER | IRQF_NO_SUSPEND | IRQF_NO_THREAD)

kernel/irq/manage.c +5 -1
index e8f7f179bf77..cb483a055512 100644
@@ -966,9 +966,13 @@ irq_thread_check_affinity(struct irq_desc *desc, struct irqaction *action)
 	 * mask pointer. For CPU_MASK_OFFSTACK=n this is optimized out.
 	 */
 	if (cpumask_available(desc->irq_common_data.affinity)) {
+		struct irq_data *irq_data = &desc->irq_data;
 		const struct cpumask *m;
 
-		m = irq_data_get_effective_affinity_mask(&desc->irq_data);
+		if (action->flags & IRQF_IRQ_AFFINITY)
+			m = desc->irq_common_data.affinity;
+		else
+			m = irq_data_get_effective_affinity_mask(irq_data);
 		cpumask_copy(mask, m);
 	} else {
 		valid = false;
Signature · 6 lines
2.17.1

As Ming mentioned in that same thread, we could even make this policy for managed interrupts.

Cheers,
John
Quoted 6 lines

Thanks,
Ming Lei

.

maintainer
#21 lore raw
Quoted 19 lines · John Garry
kernel/irq/manage.c +5 -1
index e8f7f179bf77..cb483a055512 100644
@@ -966,9 +966,13 @@ irq_thread_check_affinity(struct irq_desc *desc, 
 struct irqaction *action)
  	 * mask pointer. For CPU_MASK_OFFSTACK=n this is optimized out.
  	 */
  	if (cpumask_available(desc->irq_common_data.affinity)) {
+		struct irq_data *irq_data = &desc->irq_data;
  		const struct cpumask *m;
 
-		m = irq_data_get_effective_affinity_mask(&desc->irq_data);
+		if (action->flags & IRQF_IRQ_AFFINITY)
+			m = desc->irq_common_data.affinity;
+		else
+			m = irq_data_get_effective_affinity_mask(irq_data);
  		cpumask_copy(mask, m);
  	} else {
Quoted 6 lines

valid = false;

Signature · 4 lines
2.17.1

As Ming mentioned in that same thread, we could even make this policy 
for managed interrupts.

Ack, I really like this option!

#22 lore raw
In reply to John Garry
Quoted 73 lines
Quoted 73 lines
Quoted 73 lines

Subject: Re: [PATCH 0/3] fix interrupt swamp in NVMe

Quoted 7 lines · Ming Lei
Quoted 6 lines · On Tue, Aug 20, 2019 at 2:14 PM <[email protected]> wrote:

From: Long Li <[email protected]>

This patch set tries to fix interrupt swamp in NVMe devices.

On large systems with many CPUs, a number of CPUs may share one

NVMe

Quoted 2 lines
Quoted 2 lines

hardware queue. It may have this situation where several CPUs are issuing I/Os, and all the I/Os are returned on the CPU where the

hardware queue is bound to.

Quoted 3 lines
Quoted 3 lines

This may result in that CPU swamped by interrupts and stay in interrupt mode for extended time while other CPUs continue to issue I/O. This can trigger Watchdog and RCU timeout, and make the system

unresponsive.

Quoted 17 lines
Quoted 10 lines

This patch set addresses this by enforcing scheduling and throttling I/O when CPU is starved in this situation.

Long Li (3):
sched: define a function to report the number of context switches on a
CPU
sched: export idle_cpu()
nvme: complete request in work queue on CPU with flooded interrupts

Quoted 6 lines
 drivers/nvme/host/core.c | 57
+++++++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/nvme.h |  1 +
 include/linux/sched.h    |  2 ++
 kernel/sched/core.c      |  7 +++++
 4 files changed, 66 insertions(+), 1 deletion(-)
Quoted 6 lines

Another simpler solution may be to complete request in threaded interrupt handler for this case. Meantime allow scheduler to run the interrupt thread handler on CPUs specified by the irq affinity mask, which was discussed by the following link:

https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flor
e

Quoted 1 line

.kernel.org%2Flkml%2Fe0e9478e-62a5-ca24-3b12-

58f7d056383e%40huawei.com

Quoted 1 line

%2F&amp;data=02%7C01%7Clongli%40microsoft.com%7Cc7f46d3e273f45

176d1c08

Quoted 1 line

d7254cc69e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370188 8401588

Quoted 1 line

9866&amp;sdata=h5k6HoGoyDxuhmDfuKLZUwgmw17PU%2BT%2FCbawfxV Er3U%3D&amp;

Quoted 5 lines

reserved=0

Could you try the above solution and see if the lockup can be avoided? John Garry
should have workable patch.

Yeah, so we experimented with changing the interrupt handling in the SCSI driver I maintain to use a threaded handler IRQ handler plus patch below, and saw a significant throughput boost:

--->8

Subject: [PATCH] genirq: Add support to allow thread to use hard irq affinity

Currently the cpu allowed mask for the threaded part of a threaded irq handler will be set to the effective affinity of the hard irq.

Typically the effective affinity of the hard irq will be for a single cpu. As such, the threaded handler would always run on the same cpu as the hard irq.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

We have seen scenarios in high data-rate throughput testing that the cpu handling the interrupt can be totally saturated handling both the hard interrupt and threaded handler parts, limiting throughput.

Add IRQF_IRQ_AFFINITY flag to allow the driver requesting the threaded interrupt to decide on the policy of which cpu the threaded handler may run.

Thanks for pointing me to this patch. This fixed the interrupt swamp and make the system stable.

However I'm seeing reduced performance when using threaded interrupts.

Here are the test results on a system with 80 CPUs and 10 NVMe disks (32 hardware queues for each disk) Benchmark tool is FIO, I/O pattern: 4k random reads on all NVMe disks, with queue depth = 64, num of jobs = 80, direct=1

With threaded interrupts: 1320k IOPS
With just interrupts: 3720k IOPS
With just interrupts and my patch: 3700k IOPS

At the peak IOPS, the overall CPU usage is at around 98-99%. I think the cost of doing wake up and context switch for NVMe threaded IRQ handler takes some CPU away.

In this test, I made the following change to make use of IRQF_IRQ_AFFINITY for NVMe:

drivers/pci/irq.c +1 -1
index a1de501a2729..3fb30d16464e 100644
@@ -86,7 +86,7 @@ int pci_request_irq(struct pci_dev *dev, unsigned int nr, irq_handler_t handler,
        va_list ap;
        int ret;
        char *devname;
-       unsigned long irqflags = IRQF_SHARED;
+       unsigned long irqflags = IRQF_SHARED | IRQF_IRQ_AFFINITY;
 
        if (!handler)
                irqflags |= IRQF_ONESHOT;

Thanks

Long

Quoted 64 lines
Quoted 64 lines
Quoted 64 lines
include/linux/interrupt.h index renamed from include/linux/interrupt.h +0 -0

5b8328a99b2a..48e8b955989a 100644

include/linux/interrupt.h +0 -0
@@ -61,6 +61,9 @@
  *                interrupt handler after suspending interrupts. For

system
* wakeup devices users need to implement wakeup
detection in
* their interrupt handlers.
+ * IRQF_IRQ_AFFINITY - Use the hard interrupt affinity for setting the cpu
+ * allowed mask for the threaded handler of a threaded
interrupt
+ * handler, rather than the effective hard irq affinity.
*/
#define IRQF_SHARED 0x00000080
#define IRQF_PROBE_SHARED 0x00000100

include/linux/interrupt.h +1 -0
@@ -74,6 +77,7 @@
 #define IRQF_NO_THREAD		0x00010000
 #define IRQF_EARLY_RESUME	0x00020000
 #define IRQF_COND_SUSPEND	0x00040000
+#define IRQF_IRQ_AFFINITY	0x00080000
 
 #define IRQF_TIMER		(__IRQF_TIMER | IRQF_NO_SUSPEND |
 IRQF_NO_THREAD)
kernel/irq/manage.c index renamed from kernel/irq/manage.c +0 -0

e8f7f179bf77..cb483a055512 100644

kernel/irq/manage.c +5 -1
@@ -966,9 +966,13 @@ irq_thread_check_affinity(struct irq_desc *desc,
 struct irqaction *action)
 	 * mask pointer. For CPU_MASK_OFFSTACK=n this is optimized out.
 	 */
 	if (cpumask_available(desc->irq_common_data.affinity)) {
+		struct irq_data *irq_data = &desc->irq_data;
 		const struct cpumask *m;
 
-		m = irq_data_get_effective_affinity_mask(&desc-
 >irq_data);
+		if (action->flags & IRQF_IRQ_AFFINITY)
+			m = desc->irq_common_data.affinity;
+		else
+			m = irq_data_get_effective_affinity_mask(irq_data);
 		cpumask_copy(mask, m);
kernel/irq/manage.c +0 -1
 	} else {
 		valid = false;
--

2.17.1

As Ming mentioned in that same thread, we could even make this policy for managed interrupts.

Cheers,
John

Quoted 6 lines

Thanks,
Ming Lei

.

#23 lore raw
Quoted 84 lines · Long Li
Quoted 73 lines
Quoted 73 lines
Quoted 73 lines

Subject: Re: [PATCH 0/3] fix interrupt swamp in NVMe

Quoted 7 lines · Ming Lei
Quoted 6 lines · On Tue, Aug 20, 2019 at 2:14 PM <[email protected]> wrote:

From: Long Li <[email protected]>

This patch set tries to fix interrupt swamp in NVMe devices.

On large systems with many CPUs, a number of CPUs may share one

NVMe

Quoted 2 lines
Quoted 2 lines

hardware queue. It may have this situation where several CPUs are issuing I/Os, and all the I/Os are returned on the CPU where the

hardware queue is bound to.

Quoted 3 lines
Quoted 3 lines

This may result in that CPU swamped by interrupts and stay in interrupt mode for extended time while other CPUs continue to issue I/O. This can trigger Watchdog and RCU timeout, and make the system

unresponsive.

Quoted 17 lines
Quoted 10 lines

This patch set addresses this by enforcing scheduling and throttling I/O when CPU is starved in this situation.

Long Li (3):
sched: define a function to report the number of context switches on a
CPU
sched: export idle_cpu()
nvme: complete request in work queue on CPU with flooded interrupts

Quoted 6 lines
 drivers/nvme/host/core.c | 57
+++++++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/nvme.h |  1 +
 include/linux/sched.h    |  2 ++
 kernel/sched/core.c      |  7 +++++
 4 files changed, 66 insertions(+), 1 deletion(-)
Quoted 6 lines

Another simpler solution may be to complete request in threaded interrupt handler for this case. Meantime allow scheduler to run the interrupt thread handler on CPUs specified by the irq affinity mask, which was discussed by the following link:

https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flor
e

Quoted 1 line

.kernel.org%2Flkml%2Fe0e9478e-62a5-ca24-3b12-

58f7d056383e%40huawei.com

Quoted 1 line

%2F&amp;data=02%7C01%7Clongli%40microsoft.com%7Cc7f46d3e273f45

176d1c08

Quoted 1 line

d7254cc69e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370188 8401588

Quoted 1 line

9866&amp;sdata=h5k6HoGoyDxuhmDfuKLZUwgmw17PU%2BT%2FCbawfxV Er3U%3D&amp;

Quoted 5 lines

reserved=0

Could you try the above solution and see if the lockup can be avoided? John Garry
should have workable patch.

Yeah, so we experimented with changing the interrupt handling in the SCSI driver I maintain to use a threaded handler IRQ handler plus patch below, and saw a significant throughput boost:

--->8

Subject: [PATCH] genirq: Add support to allow thread to use hard irq affinity

Currently the cpu allowed mask for the threaded part of a threaded irq handler will be set to the effective affinity of the hard irq.

Typically the effective affinity of the hard irq will be for a single cpu. As such, the threaded handler would always run on the same cpu as the hard irq.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

We have seen scenarios in high data-rate throughput testing that the cpu handling the interrupt can be totally saturated handling both the hard interrupt and threaded handler parts, limiting throughput.

Add IRQF_IRQ_AFFINITY flag to allow the driver requesting the threaded interrupt to decide on the policy of which cpu the threaded handler may run.

Thanks for pointing me to this patch. This fixed the interrupt swamp and make the system stable.

Quoted 8 lines

However I'm seeing reduced performance when using threaded interrupts.

Here are the test results on a system with 80 CPUs and 10 NVMe disks (32 hardware queues for each disk) Benchmark tool is FIO, I/O pattern: 4k random reads on all NVMe disks, with queue depth = 64, num of jobs = 80, direct=1

With threaded interrupts: 1320k IOPS
With just interrupts: 3720k IOPS
With just interrupts and my patch: 3700k IOPS

This gap looks too big wrt. threaded interrupts vs. interrupts.

Quoted 3 lines

At the peak IOPS, the overall CPU usage is at around 98-99%. I think the cost of doing wake up and context switch for NVMe threaded IRQ handler takes some CPU away.

In theory, it shouldn't be so because most of times the thread should be running on CPUs of this hctx, and the wakeup cost shouldn't be so big. Maybe there is performance problem somewhere wrt. threaded interrupt.

Could you share us your test script and environment? I will see if I can reproduce it in my environment.

Quoted 8 lines

In this test, I made the following change to make use of IRQF_IRQ_AFFINITY for NVMe:

drivers/pci/irq.c +0 -0
index a1de501a2729..3fb30d16464e 100644

drivers/pci/irq.c int pci_request_irq(struct pci_dev *dev, unsigned int nr, irq_handler_t handler, · line 92

Quoted 8 lines
drivers/pci/irq.c +1 -1
@@ -86,7 +86,7 @@ int pci_request_irq(struct pci_dev *dev, unsigned int nr, irq_handler_t handler,
        va_list ap;
        int ret;
        char *devname;
-       unsigned long irqflags = IRQF_SHARED;
+       unsigned long irqflags = IRQF_SHARED | IRQF_IRQ_AFFINITY;
 
        if (!handler)
                irqflags |= IRQF_ONESHOT;

I don't see why IRQF_IRQ_AFFINITY is needed.

John, could you explain it a bit why you need changes on IRQF_IRQ_AFFINITY?

The following patch should be enough:

kernel/irq/manage.c +5 -1
index e8f7f179bf77..1e7cffc1c20c 100644
@@ -968,7 +968,11 @@ irq_thread_check_affinity(struct irq_desc *desc, struct irqaction *action)
 	if (cpumask_available(desc->irq_common_data.affinity)) {
 		const struct cpumask *m;
 
-		m = irq_data_get_effective_affinity_mask(&desc->irq_data);
+		if (irqd_affinity_is_managed(&desc->irq_data))
+			m = desc->irq_common_data.affinity;
+		else
+			m = irq_data_get_effective_affinity_mask(
+					&desc->irq_data);
 		cpumask_copy(mask, m);
 	} else {
 		valid = false;

Thanks,
Ming

#24 lore raw
Quoted 120 lines · Ming Lei
Quoted 84 lines · Long Li
Quoted 73 lines
Quoted 73 lines
Quoted 73 lines

Subject: Re: [PATCH 0/3] fix interrupt swamp in NVMe

Quoted 7 lines · Ming Lei
Quoted 6 lines · On Tue, Aug 20, 2019 at 2:14 PM <[email protected]> wrote:

From: Long Li <[email protected]>

This patch set tries to fix interrupt swamp in NVMe devices.

On large systems with many CPUs, a number of CPUs may share one

NVMe

Quoted 2 lines
Quoted 2 lines

hardware queue. It may have this situation where several CPUs are issuing I/Os, and all the I/Os are returned on the CPU where the

hardware queue is bound to.

Quoted 3 lines
Quoted 3 lines

This may result in that CPU swamped by interrupts and stay in interrupt mode for extended time while other CPUs continue to issue I/O. This can trigger Watchdog and RCU timeout, and make the system

unresponsive.

Quoted 17 lines
Quoted 10 lines

This patch set addresses this by enforcing scheduling and throttling I/O when CPU is starved in this situation.

Long Li (3):
sched: define a function to report the number of context switches on a
CPU
sched: export idle_cpu()
nvme: complete request in work queue on CPU with flooded interrupts

Quoted 6 lines
 drivers/nvme/host/core.c | 57
+++++++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/nvme.h |  1 +
 include/linux/sched.h    |  2 ++
 kernel/sched/core.c      |  7 +++++
 4 files changed, 66 insertions(+), 1 deletion(-)
Quoted 6 lines

Another simpler solution may be to complete request in threaded interrupt handler for this case. Meantime allow scheduler to run the interrupt thread handler on CPUs specified by the irq affinity mask, which was discussed by the following link:

https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flor
e

Quoted 1 line

.kernel.org%2Flkml%2Fe0e9478e-62a5-ca24-3b12-

58f7d056383e%40huawei.com

Quoted 1 line

%2F&amp;data=02%7C01%7Clongli%40microsoft.com%7Cc7f46d3e273f45

176d1c08

Quoted 1 line

d7254cc69e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370188 8401588

Quoted 1 line

9866&amp;sdata=h5k6HoGoyDxuhmDfuKLZUwgmw17PU%2BT%2FCbawfxV Er3U%3D&amp;

Quoted 5 lines

reserved=0

Could you try the above solution and see if the lockup can be avoided? John Garry
should have workable patch.

Yeah, so we experimented with changing the interrupt handling in the SCSI driver I maintain to use a threaded handler IRQ handler plus patch below, and saw a significant throughput boost:

--->8

Subject: [PATCH] genirq: Add support to allow thread to use hard irq affinity

Currently the cpu allowed mask for the threaded part of a threaded irq handler will be set to the effective affinity of the hard irq.

Typically the effective affinity of the hard irq will be for a single cpu. As such, the threaded handler would always run on the same cpu as the hard irq.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

We have seen scenarios in high data-rate throughput testing that the cpu handling the interrupt can be totally saturated handling both the hard interrupt and threaded handler parts, limiting throughput.

Add IRQF_IRQ_AFFINITY flag to allow the driver requesting the threaded interrupt to decide on the policy of which cpu the threaded handler may run.

Thanks for pointing me to this patch. This fixed the interrupt swamp and make the system stable.

Quoted 8 lines

However I'm seeing reduced performance when using threaded interrupts.

Here are the test results on a system with 80 CPUs and 10 NVMe disks (32 hardware queues for each disk) Benchmark tool is FIO, I/O pattern: 4k random reads on all NVMe disks, with queue depth = 64, num of jobs = 80, direct=1

With threaded interrupts: 1320k IOPS
With just interrupts: 3720k IOPS
With just interrupts and my patch: 3700k IOPS

This gap looks too big wrt. threaded interrupts vs. interrupts.

Quoted 3 lines

At the peak IOPS, the overall CPU usage is at around 98-99%. I think the cost of doing wake up and context switch for NVMe threaded IRQ handler takes some CPU away.

In theory, it shouldn't be so because most of times the thread should be running on CPUs of this hctx, and the wakeup cost shouldn't be so big. Maybe there is performance problem somewhere wrt. threaded interrupt.

Could you share us your test script and environment? I will see if I can reproduce it in my environment.

Quoted 13 lines

In this test, I made the following change to make use of IRQF_IRQ_AFFINITY for NVMe:

drivers/pci/irq.c +1 -1
index a1de501a2729..3fb30d16464e 100644
@@ -86,7 +86,7 @@ int pci_request_irq(struct pci_dev *dev, unsigned int nr, irq_handler_t handler,
        va_list ap;
        int ret;
        char *devname;
-       unsigned long irqflags = IRQF_SHARED;
+       unsigned long irqflags = IRQF_SHARED | IRQF_IRQ_AFFINITY;
 
Quoted 7 lines

drivers/pci/irq.c int pci_request_irq(struct pci_dev *dev, unsigned int nr, irq_handler_t handler, · line 92

Quoted 3 lines
drivers/pci/irq.c +0 -0
        if (!handler)
                irqflags |= IRQF_ONESHOT;

I don't see why IRQF_IRQ_AFFINITY is needed.

John, could you explain it a bit why you need changes on IRQF_IRQ_AFFINITY?

Hi Ming,

The patch I shared was my original solution, based on the driver setting flag IRQF_IRQ_AFFINITY to request the threaded handler uses the irq affinity mask for the handler cpu allowed mask.

If we want to make this decision based only on whether the irq is managed or not, then we can drop the flag and just make the change as you suggest, below.

Thanks,
John

Quoted 27 lines

The following patch should be enough:

kernel/irq/manage.c +5 -1
index e8f7f179bf77..1e7cffc1c20c 100644
@@ -968,7 +968,11 @@ irq_thread_check_affinity(struct irq_desc *desc, struct irqaction *action)
 	if (cpumask_available(desc->irq_common_data.affinity)) {
 		const struct cpumask *m;
 
-		m = irq_data_get_effective_affinity_mask(&desc->irq_data);
+		if (irqd_affinity_is_managed(&desc->irq_data))
+			m = desc->irq_common_data.affinity;
+		else
+			m = irq_data_get_effective_affinity_mask(
+					&desc->irq_data);
 		cpumask_copy(mask, m);
 	} else {
 		valid = false;

Thanks,
Ming

.

#25 lore raw
In reply to Ming Lei
Quoted 121 lines
Quoted 121 lines
Quoted 121 lines

Subject: Re: [PATCH 0/3] fix interrupt swamp in NVMe

Quoted 10 lines · Long Li
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

Subject: Re: [PATCH 0/3] fix interrupt swamp in NVMe

Quoted 7 lines · Ming Lei
Quoted 6 lines · On Tue, Aug 20, 2019 at 2:14 PM <[email protected]> wrote:

From: Long Li <[email protected]>

This patch set tries to fix interrupt swamp in NVMe devices.

On large systems with many CPUs, a number of CPUs may share

one

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

NVMe

Quoted 3 lines
Quoted 3 lines

hardware queue. It may have this situation where several CPUs are issuing I/Os, and all the I/Os are returned on the CPU where the

hardware queue is bound to.

Quoted 4 lines
Quoted 4 lines

This may result in that CPU swamped by interrupts and stay in interrupt mode for extended time while other CPUs continue to issue I/O. This can trigger Watchdog and RCU timeout, and make the system

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

unresponsive.

Quoted 6 lines
Quoted 6 lines

This patch set addresses this by enforcing scheduling and throttling I/O when CPU is starved in this situation.

Long Li (3):
sched: define a function to report the number of context switches

on a

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 11 lines

CPU
sched: export idle_cpu()
nvme: complete request in work queue on CPU with flooded
interrupts

 drivers/nvme/host/core.c | 57
+++++++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/nvme.h |  1 +
 include/linux/sched.h    |  2 ++
 kernel/sched/core.c      |  7 +++++
 4 files changed, 66 insertions(+), 1 deletion(-)

Another simpler solution may be to complete request in threaded interrupt handler for this case. Meantime allow scheduler to run the interrupt thread handler on CPUs specified by the irq affinity mask, which was discussed by the following link:

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

https://lor
e

Quoted 1 line

.kernel.org%2Flkml%2Fe0e9478e-62a5-ca24-3b12-

58f7d056383e%40huawei.com

Quoted 1 line

%2F&amp;data=02%7C01%7Clongli%40microsoft.com%7Cc7f46d3e2

73f45

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

176d1c08

Quoted 1 line

d7254cc69e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63

70188

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

8401588

Quoted 1 line

9866&amp;sdata=h5k6HoGoyDxuhmDfuKLZUwgmw17PU%2BT%2FCb

awfxV

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Er3U%3D&amp;

Quoted 3 lines

reserved=0

Could you try the above solution and see if the lockup can be

avoided?

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

John Garry
should have workable patch.

Yeah, so we experimented with changing the interrupt handling in the SCSI driver I maintain to use a threaded handler IRQ handler plus patch below, and saw a significant throughput boost:

--->8

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Subject: [PATCH] genirq: Add support to allow thread to use hard irq affinity

Currently the cpu allowed mask for the threaded part of a threaded irq handler will be set to the effective affinity of the hard irq.

Typically the effective affinity of the hard irq will be for a single cpu. As such, the threaded handler would always run on the

same cpu as the hard irq.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

We have seen scenarios in high data-rate throughput testing that the cpu handling the interrupt can be totally saturated handling both the hard interrupt and threaded handler parts, limiting

throughput.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Add IRQF_IRQ_AFFINITY flag to allow the driver requesting the threaded interrupt to decide on the policy of which cpu the threaded

handler may run.

Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Thanks for pointing me to this patch. This fixed the interrupt swamp and

make the system stable.

Quoted 2 lines

However I'm seeing reduced performance when using threaded

interrupts.

Quoted 9 lines

Here are the test results on a system with 80 CPUs and 10 NVMe disks (32 hardware queues for each disk) Benchmark tool is FIO, I/O pattern:
4k random reads on all NVMe disks, with queue depth = 64, num of jobs = 80, direct=1

With threaded interrupts: 1320k IOPS
With just interrupts: 3720k IOPS
With just interrupts and my patch: 3700k IOPS

This gap looks too big wrt. threaded interrupts vs. interrupts.

Quoted 2 lines

At the peak IOPS, the overall CPU usage is at around 98-99%. I think the

cost of doing wake up and context switch for NVMe threaded IRQ handler takes some CPU away.

Quoted 1 line
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

In theory, it shouldn't be so because most of times the thread should be running on CPUs of this hctx, and the wakeup cost shouldn't be so big. Maybe there is performance problem somewhere wrt. threaded interrupt.

Could you share us your test script and environment? I will see if I can reproduce it in my environment.

Ming, do you have access to L80s_v2 in Azure? This test needs to run on that VM size.

Here is the command to benchmark it:

fio --bs=4k --ioengine=libaio --iodepth=128 --filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1:/dev/nvme4n1:/dev/nvme5n1:/dev/nvme6n1:/dev/nvme7n1:/dev/nvme8n1:/dev/nvme9n1 --direct=1 --runtime=120 --numjobs=80 --rw=randread --name=test --group_reporting --gtod_reduce=1

Thanks

Long

Quoted 50 lines
Quoted 50 lines
Quoted 50 lines
Quoted 2 lines

In this test, I made the following change to make use of

IRQF_IRQ_AFFINITY for NVMe:

drivers/pci/irq.c int pci_request_irq(struct pci_dev *dev, unsigned int

Quoted 6 lines
drivers/pci/irq.c index renamed from drivers/pci/irq.c +0 -0

a1de501a2729..3fb30d16464e 100644

drivers/pci/irq.c +0 -0
@@ -86,7 +86,7 @@ int pci_request_irq(struct pci_dev *dev, unsigned int

nr, irq_handler_t handler,

drivers/pci/irq.c int pci_request_irq(struct pci_dev *dev, unsigned int · line 92

Quoted 9 lines
drivers/pci/irq.c +1 -1
        va_list ap;
        int ret;
        char *devname;
-       unsigned long irqflags = IRQF_SHARED;
+       unsigned long irqflags = IRQF_SHARED | IRQF_IRQ_AFFINITY;
 
        if (!handler)
                irqflags |= IRQF_ONESHOT;

I don't see why IRQF_IRQ_AFFINITY is needed.

John, could you explain it a bit why you need changes on IRQF_IRQ_AFFINITY?

The following patch should be enough:

kernel/irq/manage.c index renamed from kernel/irq/manage.c +0 -0

e8f7f179bf77..1e7cffc1c20c 100644

kernel/irq/manage.c +5 -1
@@ -968,7 +968,11 @@ irq_thread_check_affinity(struct irq_desc *desc,
 struct irqaction *action)
 	if (cpumask_available(desc->irq_common_data.affinity)) {
 		const struct cpumask *m;
 
-		m = irq_data_get_effective_affinity_mask(&desc-
 >irq_data);
+		if (irqd_affinity_is_managed(&desc->irq_data))
+			m = desc->irq_common_data.affinity;
+		else
+			m = irq_data_get_effective_affinity_mask(
+					&desc->irq_data);
 		cpumask_copy(mask, m);
	} else {
		valid = false;

Thanks,
Ming

#26 lore raw
Quoted 125 lines · Long Li
Quoted 125 lines
Quoted 125 lines
Quoted 125 lines

Subject: Re: [PATCH 0/3] fix interrupt swamp in NVMe

Quoted 10 lines · Long Li
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

Subject: Re: [PATCH 0/3] fix interrupt swamp in NVMe

Quoted 7 lines · Ming Lei
Quoted 6 lines · On Tue, Aug 20, 2019 at 2:14 PM <[email protected]> wrote:

From: Long Li <[email protected]>

This patch set tries to fix interrupt swamp in NVMe devices.

On large systems with many CPUs, a number of CPUs may share

one

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

NVMe

Quoted 3 lines
Quoted 3 lines

hardware queue. It may have this situation where several CPUs are issuing I/Os, and all the I/Os are returned on the CPU where the

hardware queue is bound to.

Quoted 4 lines
Quoted 4 lines

This may result in that CPU swamped by interrupts and stay in interrupt mode for extended time while other CPUs continue to issue I/O. This can trigger Watchdog and RCU timeout, and make the system

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

unresponsive.

Quoted 6 lines
Quoted 6 lines

This patch set addresses this by enforcing scheduling and throttling I/O when CPU is starved in this situation.

Long Li (3):
sched: define a function to report the number of context switches

on a

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 11 lines

CPU
sched: export idle_cpu()
nvme: complete request in work queue on CPU with flooded
interrupts

 drivers/nvme/host/core.c | 57
+++++++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/nvme.h |  1 +
 include/linux/sched.h    |  2 ++
 kernel/sched/core.c      |  7 +++++
 4 files changed, 66 insertions(+), 1 deletion(-)

Another simpler solution may be to complete request in threaded interrupt handler for this case. Meantime allow scheduler to run the interrupt thread handler on CPUs specified by the irq affinity mask, which was discussed by the following link:

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

https://lor
e

Quoted 1 line

.kernel.org%2Flkml%2Fe0e9478e-62a5-ca24-3b12-

58f7d056383e%40huawei.com

Quoted 1 line

%2F&amp;data=02%7C01%7Clongli%40microsoft.com%7Cc7f46d3e2

73f45

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

176d1c08

Quoted 1 line

d7254cc69e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63

70188

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

8401588

Quoted 1 line

9866&amp;sdata=h5k6HoGoyDxuhmDfuKLZUwgmw17PU%2BT%2FCb

awfxV

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Er3U%3D&amp;

Quoted 3 lines

reserved=0

Could you try the above solution and see if the lockup can be

avoided?

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

John Garry
should have workable patch.

Yeah, so we experimented with changing the interrupt handling in the SCSI driver I maintain to use a threaded handler IRQ handler plus patch below, and saw a significant throughput boost:

--->8

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Subject: [PATCH] genirq: Add support to allow thread to use hard irq affinity

Currently the cpu allowed mask for the threaded part of a threaded irq handler will be set to the effective affinity of the hard irq.

Typically the effective affinity of the hard irq will be for a single cpu. As such, the threaded handler would always run on the

same cpu as the hard irq.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

We have seen scenarios in high data-rate throughput testing that the cpu handling the interrupt can be totally saturated handling both the hard interrupt and threaded handler parts, limiting

throughput.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Add IRQF_IRQ_AFFINITY flag to allow the driver requesting the threaded interrupt to decide on the policy of which cpu the threaded

handler may run.

Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Thanks for pointing me to this patch. This fixed the interrupt swamp and

make the system stable.

Quoted 2 lines

However I'm seeing reduced performance when using threaded

interrupts.

Quoted 9 lines

Here are the test results on a system with 80 CPUs and 10 NVMe disks (32 hardware queues for each disk) Benchmark tool is FIO, I/O pattern:
4k random reads on all NVMe disks, with queue depth = 64, num of jobs = 80, direct=1

With threaded interrupts: 1320k IOPS
With just interrupts: 3720k IOPS
With just interrupts and my patch: 3700k IOPS

This gap looks too big wrt. threaded interrupts vs. interrupts.

Quoted 2 lines

At the peak IOPS, the overall CPU usage is at around 98-99%. I think the

cost of doing wake up and context switch for NVMe threaded IRQ handler takes some CPU away.

Quoted 1 line

In theory, it shouldn't be so because most of times the thread should be running on CPUs of this hctx, and the wakeup cost shouldn't be so big. Maybe there is performance problem somewhere wrt. threaded interrupt.

Quoted 9 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Could you share us your test script and environment? I will see if I can reproduce it in my environment.

Ming, do you have access to L80s_v2 in Azure? This test needs to run on that VM size.

Here is the command to benchmark it:

fio --bs=4k --ioengine=libaio --iodepth=128 --filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1:/dev/nvme4n1:/dev/nvme5n1:/dev/nvme6n1:/dev/nvme7n1:/dev/nvme8n1:/dev/nvme9n1 --direct=1 --runtime=120 --numjobs=80 --rw=randread --name=test --group_reporting --gtod_reduce=1

I can reproduce the issue on one machine(96 cores) with 4 NVMes(32 queues), so each queue is served on 3 CPUs.

IOPS drops > 20% when 'use_threaded_interrupts' is enabled. From fio log, CPU context switch is increased a lot.

Thanks,
Ming

#27 lore raw
Quoted 11 lines · Ming Lei
Quoted 4 lines · Long Li

Here is the command to benchmark it:

fio --bs=4k --ioengine=libaio --iodepth=128 --filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1:/dev/nvme4n1:/dev/nvme5n1:/dev/nvme6n1:/dev/nvme7n1:/dev/nvme8n1:/dev/nvme9n1 --direct=1 --runtime=120 --numjobs=80 --rw=randread --name=test --group_reporting --gtod_reduce=1

I can reproduce the issue on one machine(96 cores) with 4 NVMes(32 queues), so each queue is served on 3 CPUs.

IOPS drops > 20% when 'use_threaded_interrupts' is enabled. From fio log, CPU context switch is increased a lot.

Interestingly use_threaded_interrupts shows a marginal improvement on my machine with the same fio profile. It was only 5 NVMes, but they've one queue per-cpu on 112 cores.

#28 lore raw
Quoted 11 lines · Keith Busch
Quoted 9 lines · Ming Lei
Quoted 4 lines · Long Li

Here is the command to benchmark it:

fio --bs=4k --ioengine=libaio --iodepth=128 --filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1:/dev/nvme4n1:/dev/nvme5n1:/dev/nvme6n1:/dev/nvme7n1:/dev/nvme8n1:/dev/nvme9n1 --direct=1 --runtime=120 --numjobs=80 --rw=randread --name=test --group_reporting --gtod_reduce=1

I can reproduce the issue on one machine(96 cores) with 4 NVMes(32 queues), so each queue is served on 3 CPUs.

Quoted 6 lines
Quoted 2 lines

IOPS drops > 20% when 'use_threaded_interrupts' is enabled. From fio log, CPU context switch is increased a lot.

Interestingly use_threaded_interrupts shows a marginal improvement on my machine with the same fio profile. It was only 5 NVMes, but they've one queue per-cpu on 112 cores.

Not investigate it yet.

BTW, my fio test is only done on the single hw queue via 'taskset -c $cpu_list_of_the_queue',
without applying the threaded interrupt affinity patch. NVMe is Optane.

The same issue can be reproduced after I force to use 1:1 mapping via passing 'possible_cpus=32' kernel cmd line.

Maybe related with kernel options, so attache the one I used, and basically it is
a subset of RHEL8 kernel.

Thanks,
Ming Lei

#29 lore raw
In reply to Keith Busch
Quoted 10 lines · Keith Busch
Quoted 9 lines · Ming Lei
Quoted 4 lines · Long Li

Here is the command to benchmark it:

fio --bs=4k --ioengine=libaio --iodepth=128 --filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1:/dev/nvme4n1:/dev/nvme5n1:/dev/nvme6n1:/dev/nvme7n1:/dev/nvme8n1:/dev/nvme9n1 --direct=1 --runtime=120 --numjobs=80 --rw=randread --name=test --group_reporting --gtod_reduce=1

I can reproduce the issue on one machine(96 cores) with 4 NVMes(32 queues), so each queue is served on 3 CPUs.

Quoted 6 lines
Quoted 2 lines

IOPS drops > 20% when 'use_threaded_interrupts' is enabled. From fio log, CPU context switch is increased a lot.

Interestingly use_threaded_interrupts shows a marginal improvement on my machine with the same fio profile. It was only 5 NVMes, but they've one queue per-cpu on 112 cores.

Which is not surprising because the thread and the hard interrupt are on the same CPU and there is just that little overhead of the context switch.

The thing is that this really depends on how the scheduler decides to place the interrupt thread.

If you have a queue for several CPUs, then depending on the load situation allowing a multi-cpu affinity for the thread can cause lots of task migration.

But restricting the irq thread to the CPU on which the interrupt is affine can also starve that CPU. There is no universal rule for that.

Tracing should tell.

Thanks,

tglx