fix interrupt swamp in NVMe
Subsystems (7)
cover fix interrupt swamp in NVMe
From: Long Li <[email protected]>
This patch set tries to fix interrupt swamp in NVMe devices.
On large systems with many CPUs, a number of CPUs may share one NVMe hardware queue. It may have this situation where several CPUs are issuing I/Os, and all the I/Os are returned on the CPU where the hardware queue is bound to. This may result in that CPU swamped by interrupts and stay in interrupt mode for extended time while other CPUs continue to issue I/O. This can trigger Watchdog and RCU timeout, and make the system unresponsive.
This patch set addresses this by enforcing scheduling and throttling I/O when CPU is starved in this situation.
Long Li (3):
sched: define a function to report the number of context switches on a
CPU
sched: export idle_cpu()
nvme: complete request in work queue on CPU with flooded interrupts
drivers/nvme/host/core.c | 57 +++++++++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 1 + include/linux/sched.h | 2 ++ kernel/sched/core.c | 7 +++++ 4 files changed, 66 insertions(+), 1 deletion(-)
Signature · 1 line
2.17.1
PATCH 1/3 sched: define a function to report the number of context switches on a CPU
From: Long Li <[email protected]>
The number of context switches on a CPU is useful to determine how busy this CPU is on processing IRQs. Export this information so it can be used by device drivers.
--- include/linux/sched.h | 1 + kernel/sched/core.c | 6 ++++++ 2 files changed, 7 insertions(+)
include/linux/sched.h +1 -0
@@ -1500,6 +1500,7 @@ current_restore_flags(unsigned long orig_flags, unsigned long flags)extern int cpuset_cpumask_can_shrink (const struct cpumask *cur,const struct cpumask *trial);extern int task_can_attach (struct task_struct *p,const struct cpumask *cs_cpus_allowed); +extern u64 get_cpu_rq_switches (int cpu);#ifdef CONFIG_SMP extern void do_set_cpus_allowed (struct task_struct *p,const struct cpumask *new_mask);extern int set_cpus_allowed_ptr (struct task_struct *p,const struct cpumask *new_mask);
kernel/sched/core.c +6 -0
@@ -1143,6 +1143,12 @@ int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) }EXPORT_SYMBOL_GPL (set_cpus_allowed_ptr); +u64 get_cpu_rq_switches (int cpu) +{ +return cpu_rq (cpu)->nr_switches; +} +EXPORT_SYMBOL_GPL (get_cpu_rq_switches); +void set_task_cpu (struct task_struct *p,unsigned int new_cpu) {#ifdef CONFIG_SCHED_DEBUG
Signature · 1 line
2.17.1
Peter Zijlstra
maintainerRe: [PATCH 1/3] sched: define a function to report the number of context switches on a CPU
Quoted 5 lines · On Mon, Aug 19, 2019 at 11:14:27PM -0700, [email protected] wrote:
From: Long Li <[email protected]>
The number of context switches on a CPU is useful to determine how busy this CPU is on processing IRQs. Export this information so it can be used by device drivers.
Please do explain that; because I'm not seeing how number of switches relates to processing IRQs _at_all_!
RE: [PATCH 1/3] sched: define a function to report the number of context switches on a CPU
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Subject: Re: [PATCH 1/3] sched: define a function to report the number of context switches on a CPU
On Mon, Aug 19, 2019 at 11:14:27PM -0700, [email protected] wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
From: Long Li <[email protected]>
The number of context switches on a CPU is useful to determine how busy this CPU is on processing IRQs. Export this information so it can be used by device drivers.
Please do explain that; because I'm not seeing how number of switches relates to processing IRQs _at_all_!
Some kernel components rely on context switch to progress, for example watchdog and RCU. On a CPU with reasonable interrupt load, it continues to make context switches, normally a number of switches per seconds.
While observing a CPU with heavy interrupt loads, I see that it spends all its time in IRQ and softIRQ, and not to get a chance to do a switch (calling __schedule()) for a long time. This will result in system unresponsive at times. The purpose is to find out if this CPU is in this state, and implement some throttling mechanism to help reduce the number of interrupts. I think the number of switches is not accurate for detecting this condition in the most precise way, but maybe it's good enough.
I agree this may not be the best way. If you have other idea on detecting a CPU is swamped by interrupts, please point me to where to look at.
Thanks
Long
Peter Zijlstra
maintainerRe: [PATCH 1/3] sched: define a function to report the number of context switches on a CPU
Quoted 11 lines · Long Li
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Subject: Re: [PATCH 1/3] sched: define a function to report the number of context switches on a CPU
Quoted 5 lines · On Mon, Aug 19, 2019 at 11:14:27PM -0700, [email protected] wrote:
From: Long Li <[email protected]>
The number of context switches on a CPU is useful to determine how busy this CPU is on processing IRQs. Export this information so it can be used by device drivers.
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Please do explain that; because I'm not seeing how number of switches relates to processing IRQs _at_all_!
Some kernel components rely on context switch to progress, for example watchdog and RCU. On a CPU with reasonable interrupt load, it continues to make context switches, normally a number of switches per seconds.
That isn't true; RCU is perfectly fine with a single task always running and not making context switches, and so is the watchdog.
Peter Zijlstra
maintainerRe: [PATCH 1/3] sched: define a function to report the number of context switches on a CPU
In reply to [email protected]kernel/sched/core.c int set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask) · line 1150
Quoted 5 lines · On Mon, Aug 19, 2019 at 11:14:27PM -0700, [email protected] wrote:
kernel/sched/core.c +5
-0+u64 get_cpu_rq_switches (int cpu) +{ +return cpu_rq (cpu)->nr_switches; +} +EXPORT_SYMBOL_GPL (get_cpu_rq_switches);
Also, that is broken on 32bit.
PATCH 2/3 sched: export idle_cpu()
In reply to [email protected]From: Long Li <[email protected]>
This function is useful for device drivers to check if this CPU has work to do in process context.
--- include/linux/sched.h | 1 + kernel/sched/core.c | 1 + 2 files changed, 2 insertions(+)
include/linux/sched.h +1 -0
@@ -1501,6 +1501,7 @@ current_restore_flags(unsigned long orig_flags, unsigned long flags)extern int cpuset_cpumask_can_shrink (const struct cpumask *cur,const struct cpumask *trial);extern int task_can_attach (struct task_struct *p,const struct cpumask *cs_cpus_allowed);extern u64 get_cpu_rq_switches (int cpu); +extern int idle_cpu (int cpu);#ifdef CONFIG_SMP extern void do_set_cpus_allowed (struct task_struct *p,const struct cpumask *new_mask);extern int set_cpus_allowed_ptr (struct task_struct *p,const struct cpumask *new_mask);
kernel/sched/core.c +1 -0
@@ -4023,6 +4023,7 @@ int idle_cpu(int cpu)return 1 ; } +EXPORT_SYMBOL_GPL (idle_cpu);/** * available_idle_cpu - is a given CPU idle for enqueuing work.
Signature · 1 line
2.17.1
PATCH 3/3 nvme: complete request in work queue on CPU with flooded interrupts
In reply to [email protected]From: Long Li <[email protected]>
When a NVMe hardware queue is mapped to several CPU queues, it is possible that the CPU this hardware queue is bound to is flooded by returning I/O for other CPUs.
For example, consider the following scenario:
1. CPU 0, 1, 2 and 3 share the same hardware queue
2. the hardware queue interrupts CPU 0 for I/O response
3. processes from CPU 1, 2 and 3 keep sending I/Os
CPU 0 may be flooded with interrupts from NVMe device that are I/O responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible that CPU 0 spends all the time serving NVMe and other system interrupts, but doesn't have a chance to run in process context.
To fix this, CPU 0 can schedule a work to complete the I/O request when it detects the scheduler is not making progress. This serves multiple purposes:
1. This CPU has to be scheduled to complete the request. The other CPUs can't issue more I/Os until some previous I/Os are completed. This helps this CPU get out of NVMe interrupts.
2. This acts a throttling mechanisum for NVMe devices, in that it can not starve a CPU while servicing I/Os from other CPUs.
3. This CPU can make progress on RCU and other work items on its queue.
--- drivers/nvme/host/core.c | 57 +++++++++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 1 + 2 files changed, 57 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +56 -1
@@ -28,6 +28,7 @@#include <linux/t10-pi.h> #include <linux/pm_qos.h> #include <asm/unaligned.h> +#include <linux/sched.h> #define CREATE_TRACE_POINTS #include "trace.h"
@@ -97,6 +98,15 @@ static dev_t nvme_chr_devt;static struct class *nvme_class;static struct class *nvme_subsys_class; +/* +* The following are for detecting if this CPU is flooded with interrupts. +* The timestamp for the last context switch is recorded. If that is at least +* MAX_SCHED_TIMEOUT ago, try to recover from interrupt flood +*/ +static DEFINE_PER_CPU(u64, last_switch); +static DEFINE_PER_CPU(u64, last_clock); +#define MAX_SCHED_TIMEOUT 2000000000 // 2 seconds in ns +static int nvme_revalidate_disk (struct gendisk *disk);static void nvme_put_subsystem (struct nvme_subsystem *subsys); static void nvme_remove_invalid_namespaces(struct nvme_ctrl *ctrl,
@@ -260,9 +270,54 @@ static void nvme_retry_req(struct request *req)blk_mq_delay_kick_requeue_list (req->q, delay); } +static void nvme_complete_rq_work (struct work_struct *work) +{ +struct nvme_request *nvme_rq = + container_of(work, struct nvme_request, work); +struct request *req =blk_mq_rq_from_pdu (nvme_rq); + +nvme_complete_rq (req); +} + +void nvme_complete_rq (struct request *req) { -blk_status_t status =nvme_error_status (req); +blk_status_t status; +int cpu; +u64 switches; +struct nvme_request *nvme_rq; + +if (!in_interrupt ()) + goto skip_check; + + nvme_rq =nvme_req (req); + cpu =smp_processor_id (); +if (idle_cpu (cpu)) + goto skip_check; + +/* Check if this CPU is flooded with interrupts */ + switches =get_cpu_rq_switches (cpu); +if (this_cpu_read (last_switch) == switches) { +/* +* If this CPU hasn't made a context switch in +* MAX_SCHED_TIMEOUT ns (and it's not idle), schedule a work to +* complete this I/O. This forces this CPU run non-interrupt +* code and throttle the other CPU issuing the I/O +*/ +if (sched_clock () -this_cpu_read (last_clock) + >MAX_SCHED_TIMEOUT ) { +INIT_WORK (&nvme_rq->work, nvme_complete_rq_work); +schedule_work_on (cpu, &nvme_rq->work); +return ; + } + + }else { +this_cpu_write (last_switch, switches); +this_cpu_write (last_clock,sched_clock ()); + } + +skip_check: + status =nvme_error_status (req);trace_nvme_complete_rq (req);
drivers/nvme/host/nvme.h +1 -0
@@ -113,6 +113,7 @@ struct nvme_request {u8 flags;u16 status;struct nvme_ctrl *ctrl; +struct work_struct work; };/*
Signature · 1 line
2.17.1
Peter Zijlstra
maintainerRe: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts
Quoted 11 lines · On Mon, Aug 19, 2019 at 11:14:29PM -0700, [email protected] wrote:
From: Long Li <[email protected]>
When a NVMe hardware queue is mapped to several CPU queues, it is possible that the CPU this hardware queue is bound to is flooded by returning I/O for other CPUs.
For example, consider the following scenario:
1. CPU 0, 1, 2 and 3 share the same hardware queue
2. the hardware queue interrupts CPU 0 for I/O response
3. processes from CPU 1, 2 and 3 keep sending I/Os
Quoted 4 lines
CPU 0 may be flooded with interrupts from NVMe device that are I/O responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible that CPU 0 spends all the time serving NVMe and other system interrupts, but doesn't have a chance to run in process context.
Ideally -- and there is some code to affect this, the load-balancer will move tasks away from this CPU.
Quoted 2 lines
To fix this, CPU 0 can schedule a work to complete the I/O request when it detects the scheduler is not making progress. This serves multiple purposes:
Suppose the task waiting for the IO completion is a RT task, and you've just queued it to a regular work. This is an instant priority inversion.
Quoted 7 lines
1. This CPU has to be scheduled to complete the request. The other CPUs can't issue more I/Os until some previous I/Os are completed. This helps this CPU get out of NVMe interrupts.
2. This acts a throttling mechanisum for NVMe devices, in that it can not starve a CPU while servicing I/Os from other CPUs.
Quoted 7 lines
3. This CPU can make progress on RCU and other work items on its queue.
Signed-off-by Long Li <[email protected]>--- drivers/nvme/host/core.c | 57 +++++++++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 1 + 2 files changed, 57 insertions(+), 1 deletion(-)
WTH does this live in the NVME driver? Surely something like this should be in the block layer. I'm thinking there's fiber channel connected storage that should be able to trigger much the same issues.
Quoted 34 lines
drivers/nvme/host/core.c +22
-1@@ -260,9 +270,54 @@ static void nvme_retry_req(struct request *req)blk_mq_delay_kick_requeue_list (req->q, delay); } +static void nvme_complete_rq_work (struct work_struct *work) +{ +struct nvme_request *nvme_rq = + container_of(work, struct nvme_request, work); +struct request *req =blk_mq_rq_from_pdu (nvme_rq); + +nvme_complete_rq (req); +} + +void nvme_complete_rq (struct request *req) { -blk_status_t status =nvme_error_status (req); +blk_status_t status; +int cpu; +u64 switches; +struct nvme_request *nvme_rq; + +if (!in_interrupt ()) + goto skip_check; + + nvme_rq =nvme_req (req); + cpu =smp_processor_id (); +if (idle_cpu (cpu)) + goto skip_check;
drivers/nvme/host/core.c static void nvme_retry_req(struct request *req) · line 306
Quoted 10 lines
drivers/nvme/host/core.c +10
-0+ +/* Check if this CPU is flooded with interrupts */ + switches =get_cpu_rq_switches (cpu); +if (this_cpu_read (last_switch) == switches) { +/* +* If this CPU hasn't made a context switch in +* MAX_SCHED_TIMEOUT ns (and it's not idle), schedule a work to +* complete this I/O. This forces this CPU run non-interrupt +* code and throttle the other CPU issuing the I/O +*/
What if there was only a single task on that CPU? Then we'd never need/want to context switch in the first place.
AFAICT all this is just a whole bunch of gruesome hacks piled on top one another.
drivers/nvme/host/core.c static void nvme_retry_req(struct request *req) · line 319
Quoted 13 lines
drivers/nvme/host/core.c +13
-0+if (sched_clock () -this_cpu_read (last_clock) + >MAX_SCHED_TIMEOUT ) { +INIT_WORK (&nvme_rq->work, nvme_complete_rq_work); +schedule_work_on (cpu, &nvme_rq->work); +return ; + } + + }else { +this_cpu_write (last_switch, switches); +this_cpu_write (last_clock,sched_clock ()); + } + +skip_check:
Aside from everything else; this is just sodding poor coding style. What is wrong with something like:
if (nvme_complete_throttle(...)) return;
Quoted 4 lines
drivers/nvme/host/core.c +1
-0+ status =nvme_error_status (req);trace_nvme_complete_rq (req);
RE: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Subject: Re: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts
Quoted 11 lines · On Mon, Aug 19, 2019 at 11:14:29PM -0700, [email protected] wrote:
From: Long Li <[email protected]>
When a NVMe hardware queue is mapped to several CPU queues, it is possible that the CPU this hardware queue is bound to is flooded by returning I/O for other CPUs.
For example, consider the following scenario:
1. CPU 0, 1, 2 and 3 share the same hardware queue 2. the hardware queue interrupts CPU 0 for I/O response 3. processes from CPU 1, 2 and 3 keep sending I/OsQuoted 4 lines
CPU 0 may be flooded with interrupts from NVMe device that are I/O responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible that CPU 0 spends all the time serving NVMe and other system interrupts, but doesn't have a chance to run in process context.
Ideally -- and there is some code to affect this, the load-balancer will move tasks away from this CPU.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
To fix this, CPU 0 can schedule a work to complete the I/O request when it detects the scheduler is not making progress. This serves multiple
purposes:
Suppose the task waiting for the IO completion is a RT task, and you've just queued it to a regular work. This is an instant priority inversion.
This is a choice. We can either not "lock up" the CPU, or finish the I/O on time from IRQ handler. I think throttling only happens in extreme conditions, which is rare. The purpose is to make the whole system responsive and happy.
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
1. This CPU has to be scheduled to complete the request. The other CPUs can't issue more I/Os until some previous I/Os are completed. This helps this CPU get out of NVMe interrupts.
2. This acts a throttling mechanisum for NVMe devices, in that it can not starve a CPU while servicing I/Os from other CPUs.
3. This CPU can make progress on RCU and other work items on its queue.
Signed-off-by Long Li <[email protected]>--- drivers/nvme/host/core.c | 57 +++++++++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 1 + 2 files changed, 57 insertions(+), 1 deletion(-)
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
WTH does this live in the NVME driver? Surely something like this should be in the block layer. I'm thinking there's fiber channel connected storage that should be able to trigger much the same issues.
Yes this can be done in block layer. I'm not sure the best way to accomplish this so implemented a NVMe patch to help test. The test results are promising in that we are getting 99.5% of performance while avoided CPU lockup. The challenge is to find a way to throttle a fast storage device.
Quoted 75 lines
Quoted 75 lines
Quoted 75 lines
Quoted 29 lines
drivers/nvme/host/core.c index renamed from drivers/nvme/host/core.c +0
-0
6a9dd68c0f4f..576bb6fce293100644drivers/nvme/host/core.c +18
-1@@ -260,9 +270,54 @@ static void nvme_retry_req(struct request *req)blk_mq_delay_kick_requeue_list (req->q, delay); } +static void nvme_complete_rq_work (struct work_struct *work) { +struct nvme_request *nvme_rq = + container_of(work, struct nvme_request, work); +struct request *req =blk_mq_rq_from_pdu (nvme_rq); + +nvme_complete_rq (req); +} + +void nvme_complete_rq (struct request *req) { -blk_status_t status =nvme_error_status (req); +blk_status_t status; +int cpu; +u64 switches; +struct nvme_request *nvme_rq; + +if (!in_interrupt ()) + goto skip_check; + + nvme_rq =nvme_req (req);drivers/nvme/host/core.c static void nvme_retry_req(struct request *req) · line 300
Quoted 10 lines
drivers/nvme/host/core.c +10
-0+ cpu =smp_processor_id (); +if (idle_cpu (cpu)) + goto skip_check; + +/* Check if this CPU is flooded with interrupts */ + switches =get_cpu_rq_switches (cpu); +if (this_cpu_read (last_switch) == switches) { +/* +* If this CPU hasn't made a context switch in +* MAX_SCHED_TIMEOUT ns (and it's not idle), schedule a work to
drivers/nvme/host/core.c static void nvme_retry_req(struct request *req) · line 306
Quoted 3 lines
drivers/nvme/host/core.c +3
-0+* complete this I/O. This forces this CPU run non-interrupt +* code and throttle the other CPU issuing the I/O +*/ What if there was only a single task on that CPU? Then we'd never need/want to context switch in the first place.
AFAICT all this is just a whole bunch of gruesome hacks piled on top one another.
drivers/nvme/host/core.c static void nvme_retry_req(struct request *req)
Quoted 3 lines
drivers/nvme/host/core.c +3
-0+if (sched_clock () -this_cpu_read (last_clock) + >MAX_SCHED_TIMEOUT ) { + INIT_WORK(&nvme_rq->work,
nvme_complete_rq_work);drivers/nvme/host/core.c static void nvme_retry_req(struct request *req) · line 319
Quoted 10 lines
drivers/nvme/host/core.c +10
-0+schedule_work_on (cpu, &nvme_rq->work); +return ; + } + + }else { +this_cpu_write (last_switch, switches); +this_cpu_write (last_clock,sched_clock ()); + } + +skip_check:Aside from everything else; this is just sodding poor coding style. What is wrong with something like:
if (nvme_complete_throttle(...)) return;Quoted 4 lines
drivers/nvme/host/core.c +1
-0+ status =nvme_error_status (req);trace_nvme_complete_rq (req);
Peter Zijlstra
maintainerRe: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts
Quoted 24 lines · Long Li
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Subject: Re: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts
Quoted 11 lines · On Mon, Aug 19, 2019 at 11:14:29PM -0700, [email protected] wrote:
From: Long Li <[email protected]>
When a NVMe hardware queue is mapped to several CPU queues, it is possible that the CPU this hardware queue is bound to is flooded by returning I/O for other CPUs.
For example, consider the following scenario:
1. CPU 0, 1, 2 and 3 share the same hardware queue 2. the hardware queue interrupts CPU 0 for I/O response 3. processes from CPU 1, 2 and 3 keep sending I/OsQuoted 4 lines
CPU 0 may be flooded with interrupts from NVMe device that are I/O responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible that CPU 0 spends all the time serving NVMe and other system interrupts, but doesn't have a chance to run in process context.
Ideally -- and there is some code to affect this, the load-balancer will move tasks away from this CPU.
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
To fix this, CPU 0 can schedule a work to complete the I/O request when it detects the scheduler is not making progress. This serves multiple
purposes:
Suppose the task waiting for the IO completion is a RT task, and you've just queued it to a regular work. This is an instant priority inversion.
This is a choice. We can either not "lock up" the CPU, or finish the I/O on time from IRQ handler. I think throttling only happens in extreme conditions, which is rare. The purpose is to make the whole system responsive and happy.
Can you please use a sane MUA.. this is unreadable garbage.
Sagi Grimberg
maintainerRe: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts
In reply to [email protected]Quoted 20 lines
From: Long Li <[email protected]>
When a NVMe hardware queue is mapped to several CPU queues, it is possible that the CPU this hardware queue is bound to is flooded by returning I/O for other CPUs.
For example, consider the following scenario:
1. CPU 0, 1, 2 and 3 share the same hardware queue
2. the hardware queue interrupts CPU 0 for I/O response
3. processes from CPU 1, 2 and 3 keep sending I/OsCPU 0 may be flooded with interrupts from NVMe device that are I/O responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible that CPU 0 spends all the time serving NVMe and other system interrupts, but doesn't have a chance to run in process context.
To fix this, CPU 0 can schedule a work to complete the I/O request when it detects the scheduler is not making progress. This serves multiple purposes:
Quoted 8 lines
1. This CPU has to be scheduled to complete the request. The other CPUs can't issue more I/Os until some previous I/Os are completed. This helps this CPU get out of NVMe interrupts.
2. This acts a throttling mechanisum for NVMe devices, in that it can not starve a CPU while servicing I/Os from other CPUs.
3. This CPU can make progress on RCU and other work items on its queue.
The problem is indeed real, but this is the wrong approach in my mind.
We already have irqpoll which takes care proper budgeting polling cycles and not hogging the cpu.
I've sent rfc for this particular problem before [1]. At the time IIRC, Christoph suggested that we will poll the first batch directly from the irq context and reap the rest in irqpoll handler.
[1]: http://lists.infradead.org/pipermail/linux-nvme/2016-October/006497.html
How about something like this instead:
--
drivers/nvme/host/pci.c +27 -1
@@ -24,6 +24,7 @@#include <linux/io-64-nonatomic-lo-hi.h> #include <linux/sed-opal.h> #include <linux/pci-p2pdma.h> +#include <linux/irq_poll.h> #include "trace.h" #include "nvme.h"
@@ -32,6 +33,7 @@#define CQ_SIZE (q) ((q)->q_depth * sizeof(struct nvme_completion))#define SGES_PER_PAGE (PAGE_SIZE / sizeof(struct nvme_sgl_desc)) +#define NVME_POLL_BUDGET_IRQ 256/* * These can be higher, but we need to ensure that any command doesn't
@@ -189,6 +191,7 @@ struct nvme_queue {u32 *dbbuf_cq_db;u32 *dbbuf_sq_ei;u32 *dbbuf_cq_ei; +struct irq_poll iop;struct completion delete_done; };
@@ -1015,6 +1018,23 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq, u16 *start,return found; } +static int nvme_irqpoll_handler (struct irq_poll *iop,int budget) +{ + struct nvme_queue *nvmeq = container_of(iop, struct nvme_queue, iop); +struct pci_dev *pdev =to_pci_dev (nvmeq->dev->dev); +u16 start, end; +int completed; + + completed =nvme_process_cq (nvmeq, &start, &end, budget); +nvme_complete_cqes (nvmeq, start, end); +if (completed < budget) { +irq_poll_complete (&nvmeq->iop); +enable_irq (pci_irq_vector (pdev, nvmeq->cq_vector)); + } + +return completed; +} +static irqreturn_t nvme_irq (int irq,void *data) {struct nvme_queue *nvmeq = data;
@@ -1028,12 +1048,16 @@ static irqreturn_t nvme_irq(int irq, void *data)rmb ();if (nvmeq->cq_head != nvmeq->last_cq_head) ret =IRQ_HANDLED ; -nvme_process_cq (nvmeq, &start, &end,-1 ); +nvme_process_cq (nvmeq, &start, &end,NVME_POLL_BUDGET_IRQ ); nvmeq->last_cq_head = nvmeq->cq_head;wmb ();if (start != end) {nvme_complete_cqes (nvmeq, start, end); +if (nvme_cqe_pending (nvmeq)) { +disable_irq_nosync (irq); +irq_poll_sched (&nvmeq->iop); + }return IRQ_HANDLED ; }
@@ -1347,6 +1371,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)static void nvme_free_queue (struct nvme_queue *nvmeq) { +irq_poll_disable (&nvmeq->iop);dma_free_coherent (nvmeq->dev->dev,CQ_SIZE (nvmeq), (void *)nvmeq->cqes, nvmeq->cq_dma_addr);if (!nvmeq->sq_cmds)
@@ -1481,6 +1506,7 @@ static int nvme_alloc_queue(struct nvme_dev *dev, int qid, int depth) nvmeq->dev = dev;spin_lock_init (&nvmeq->sq_lock);spin_lock_init (&nvmeq->cq_poll_lock); +irq_poll_init (&nvmeq->iop,NVME_POLL_BUDGET_IRQ , nvme_irqpoll_handler); nvmeq->cq_head =0 ; nvmeq->cq_phase =1 ; nvmeq->q_db = &dev->dbs[qid *2 * dev->db_stride];
--
RE: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts
Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
Subject: Re: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts
Quoted 11 lines
From: Long Li <[email protected]>
When a NVMe hardware queue is mapped to several CPU queues, it is possible that the CPU this hardware queue is bound to is flooded by returning I/O for other CPUs.
For example, consider the following scenario:
1. CPU 0, 1, 2 and 3 share the same hardware queue 2. the hardware queue interrupts CPU 0 for I/O response 3. processes from CPU 1, 2 and 3 keep sending I/OsQuoted 7 lines
CPU 0 may be flooded with interrupts from NVMe device that are I/O responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible that CPU 0 spends all the time serving NVMe and other system interrupts, but doesn't have a chance to run in process context.
To fix this, CPU 0 can schedule a work to complete the I/O request when it detects the scheduler is not making progress. This serves multiple
purposes:
Quoted 9 lines
1. This CPU has to be scheduled to complete the request. The other CPUs can't issue more I/Os until some previous I/Os are completed. This helps this CPU get out of NVMe interrupts.
2. This acts a throttling mechanisum for NVMe devices, in that it can not starve a CPU while servicing I/Os from other CPUs.
3. This CPU can make progress on RCU and other work items on its queue.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The problem is indeed real, but this is the wrong approach in my mind.
We already have irqpoll which takes care proper budgeting polling cycles and not hogging the cpu.
I've sent rfc for this particular problem before [1]. At the time IIRC, Christoph suggested that we will poll the first batch directly from the irq context and reap the rest in irqpoll handler.
Thanks for the pointer. I will test and report back.
Quoted 102 lines
Quoted 102 lines
Quoted 102 lines
[1]:
https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.
infradead.org%2Fpipermail%2Flinux-nvme%2F2016- October%2F006497.html&data=02%7C01%7Clongli%40microsoft.com% 7C0ebf36eff15c4182116608d725948b93%7C72f988bf86f141af91ab2d7cd011d b47%7C1%7C0%7C637019192254250361&sdata=fJ%2Fkc8HLSmfzaY3BY E66zlZKD6FjcXgMJZzVGCVqI%2FU%3D&reserved=0How about something like this instead:
--drivers/nvme/host/pci.c index renamed from drivers/nvme/host/pci.c +0
-0
71127a366d3c..84bf16d75109100644drivers/nvme/host/pci.c +19
-0@@ -24,6 +24,7 @@#include <linux/io-64-nonatomic-lo-hi.h> #include <linux/sed-opal.h> #include <linux/pci-p2pdma.h> +#include <linux/irq_poll.h> #include "trace.h" #include "nvme.h" @@ -32,6 +33,7 @@#define CQ_SIZE (q) ((q)->q_depth * sizeof(struct nvme_completion))#define SGES_PER_PAGE (PAGE_SIZE / sizeof(struct nvme_sgl_desc)) +#define NVME_POLL_BUDGET_IRQ 256/* * These can be higher, but we need to ensure that any command doesn't @@ -189,6 +191,7 @@ struct nvme_queue {u32 *dbbuf_cq_db;u32 *dbbuf_sq_ei;u32 *dbbuf_cq_ei; +struct irq_poll iop;struct completion delete_done; };@@ -1015,6 +1018,23 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq, u16 *start, return found; } +static int nvme_irqpoll_handler (struct irq_poll *iop,int budget) { + struct nvme_queue *nvmeq = container_of(iop, struct nvme_queue, iop); +struct pci_dev *pdev =to_pci_dev (nvmeq->dev->dev); +u16 start, end; +int completed; + + completed =nvme_process_cq (nvmeq, &start, &end, budget); +nvme_complete_cqes (nvmeq, start, end); +if (completed < budget) { +irq_poll_complete (&nvmeq->iop); +enable_irq (pci_irq_vector (pdev, nvmeq->cq_vector)); + } + +return completed; +} +static irqreturn_t nvme_irq (int irq,void *data) { struct nvme_queue *nvmeq = data; @@ -1028,12 +1048,16 @@ static
irqreturn_tnvme_irq(int irq, void *data)rmb(); if (nvmeq->cq_head != nvmeq->last_cq_head) ret = IRQ_HANDLED;drivers/nvme/host/pci.c +6
-1-nvme_process_cq (nvmeq, &start, &end,-1 ); +nvme_process_cq (nvmeq, &start, &end,NVME_POLL_BUDGET_IRQ ); nvmeq->last_cq_head = nvmeq->cq_head;wmb ();if (start != end) {nvme_complete_cqes (nvmeq, start, end); +if (nvme_cqe_pending (nvmeq)) { +disable_irq_nosync (irq); +irq_poll_sched (&nvmeq->iop); + }return IRQ_HANDLED ; }@@ -1347,6 +1371,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)static void nvme_free_queue (struct nvme_queue *nvmeq) { +irq_poll_disable (&nvmeq->iop);dma_free_coherent (nvmeq->dev->dev,CQ_SIZE (nvmeq), (void *)nvmeq->cqes, nvmeq->cq_dma_addr);if (!nvmeq->sq_cmds)drivers/nvme/host/pci.c +1
-0@@ -1481,6 +1506,7 @@ static int nvme_alloc_queue(struct nvme_dev *dev, int qid, int depth) nvmeq->dev = dev;spin_lock_init (&nvmeq->sq_lock);spin_lock_init (&nvmeq->cq_poll_lock); +irq_poll_init (&nvmeq->iop,NVME_POLL_BUDGET_IRQ , nvme_irqpoll_handler); nvmeq->cq_head =0 ;nvmeq->cq_phase = 1; nvmeq->q_db = &dev->dbs[qid * 2 * dev->db_stride];--
RE: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts
Quoted 42 lines
Quoted 42 lines
Quoted 42 lines
Subject: RE: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts
Quoted 1 line
Quoted 1 line
Quoted 1 line
Subject: Re: [PATCH 3/3] nvme: complete request in work queue on
CPU
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
with flooded interrupts
Quoted 12 lines
From: Long Li <[email protected]>
When a NVMe hardware queue is mapped to several CPU queues, it is possible that the CPU this hardware queue is bound to is flooded by returning I/O for other CPUs.
For example, consider the following scenario:
1. CPU 0, 1, 2 and 3 share the same hardware queue 2. the hardware queue interrupts CPU 0 for I/O response 3. processes from CPU 1, 2 and
3 keep sending I/OsQuoted 8 lines
CPU 0 may be flooded with interrupts from NVMe device that are I/O responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible that CPU 0 spends all the time serving NVMe and other system interrupts, but doesn't have a chance to run in process context.
To fix this, CPU 0 can schedule a work to complete the I/O request when it detects the scheduler is not making progress. This serves multiple
purposes:
Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
1. This CPU has to be scheduled to complete the request. The other CPUs can't issue more I/Os until some previous I/Os are completed. This helps this CPU get out of NVMe interrupts.
2. This acts a throttling mechanisum for NVMe devices, in that it can not starve a CPU while servicing I/Os from other CPUs.
3. This CPU can make progress on RCU and other work items on its
queue.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The problem is indeed real, but this is the wrong approach in my mind.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
We already have irqpoll which takes care proper budgeting polling cycles and not hogging the cpu.
I've sent rfc for this particular problem before [1]. At the time IIRC, Christoph suggested that we will poll the first batch directly from the irq context and reap the rest in irqpoll handler.
Thanks for the pointer. I will test and report back.
Sagi,
Here are the test results.
Benchmark command:
fio --bs=4k --ioengine=libaio --iodepth=64 --filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1:/dev/nvme4n1:/dev/nvme5n1:/dev/nvme6n1:/dev/nvme7n1:/dev/nvme8n1:/dev/nvme9n1 --direct=1 --runtime=90 --numjobs=80 --rw=randread --name=test --group_reporting --gtod_reduce=1
With your patch: 1720k IOPS
With threaded interrupts: 1320k IOPS
With just interrupts: 3720k IOPS
Interrupts are the fastest but we need to find a way to throttle it.
Thanks
Long
Quoted 107 lines
Quoted 107 lines
Quoted 107 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
[1]:
https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
infradead.org%2Fpipermail%2Flinux-nvme%2F2016- October%2F006497.html&data=02%7C01%7Clongli%40microsoft.co
m%
Quoted 1 line
Quoted 1 line
Quoted 1 line
7C0ebf36eff15c4182116608d725948b93%7C72f988bf86f141af91ab2d7cd0
11d
Quoted 1 line
Quoted 1 line
Quoted 1 line
b47%7C1%7C0%7C637019192254250361&sdata=fJ%2Fkc8HLSmfzaY
3BY
Quoted 55 lines
Quoted 55 lines
Quoted 55 lines
E66zlZKD6FjcXgMJZzVGCVqI%2FU%3D&reserved=0
How about something like this instead:
--drivers/nvme/host/pci.c index renamed from drivers/nvme/host/pci.c +0
-0
71127a366d3c..84bf16d75109100644drivers/nvme/host/pci.c +2
-0@@ -24,6 +24,7 @@#include <linux/io-64-nonatomic-lo-hi.h> #include <linux/sed-opal.h> #include <linux/pci-p2pdma.h> +#include <linux/irq_poll.h> #include "trace.h" #include "nvme.h" @@ -32,6 +33,7 @@#define CQ_SIZE (q) ((q)->q_depth * sizeof(struct nvme_completion))#define SGES_PER_PAGE (PAGE_SIZE / sizeof(struct nvme_sgl_desc)) +#define NVME_POLL_BUDGET_IRQ 256/* * These can be higher, but we need to ensure that any command doesn't @@ -189,6 +191,7 @@ struct nvme_queue { u32 *dbbuf_cq_db; u32 *dbbuf_sq_ei; u32 *dbbuf_cq_ei;drivers/nvme/host/pci.c +18
-0+struct irq_poll iop;struct completion delete_done; };@@ -1015,6 +1018,23 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq, u16 *start, return found; } +static int nvme_irqpoll_handler (struct irq_poll *iop,int budget) { + struct nvme_queue *nvmeq = container_of(iop, struct +nvme_queue, iop); +struct pci_dev *pdev =to_pci_dev (nvmeq->dev->dev); +u16 start, end; +int completed; + + completed =nvme_process_cq (nvmeq, &start, &end, budget); +nvme_complete_cqes (nvmeq, start, end); +if (completed < budget) { +irq_poll_complete (&nvmeq->iop); +enable_irq (pci_irq_vector (pdev, nvmeq->cq_vector)); + } + +return completed; +} +static irqreturn_t nvme_irq (int irq,void *data)Quoted 8 lines
Quoted 8 lines
drivers/nvme/host/pci.c
Quoted 8 lines
{ struct nvme_queue *nvmeq = data; @@ -1028,12 +1048,16 @@ static irqreturn_t nvme_irq(int irq, void *data) rmb(); if (nvmeq->cq_head != nvmeq->last_cq_head) ret = IRQ_HANDLED;drivers/nvme/host/pci.c +1
-1-nvme_process_cq (nvmeq, &start, &end,-1 ); + nvme_process_cq(nvmeq, &start, &end,
NVME_POLL_BUDGET_IRQ);Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
nvmeq->last_cq_head = nvmeq->cq_head; wmb(); if (start != end) { nvme_complete_cqes(nvmeq, start, end);drivers/nvme/host/pci.c +6
-0+if (nvme_cqe_pending (nvmeq)) { +disable_irq_nosync (irq); +irq_poll_sched (&nvmeq->iop); + }return IRQ_HANDLED ; }@@ -1347,6 +1371,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)static void nvme_free_queue (struct nvme_queue *nvmeq) { +irq_poll_disable (&nvmeq->iop);dma_free_coherent (nvmeq->dev->dev,CQ_SIZE (nvmeq), (void *)nvmeq->cqes, nvmeq->cq_dma_addr);if (!nvmeq->sq_cmds)@@ -1481,6 +1506,7 @@ static int nvme_alloc_queue(struct nvme_dev *dev, int qid, int depth) nvmeq->dev = dev;spin_lock_init (&nvmeq->sq_lock);spin_lock_init (&nvmeq->cq_poll_lock); +irq_poll_init (&nvmeq->iop,NVME_POLL_BUDGET_IRQ , nvme_irqpoll_handler); nvmeq->cq_head =0 ;nvmeq->cq_phase = 1; nvmeq->q_db = &dev->dbs[qid * 2 * dev->db_stride];--
Sagi Grimberg
maintainerRe: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts
Quoted 5 lines
Sagi,
Here are the test results.
Quoted 8 lines
Benchmark command:
fio --bs=4k --ioengine=libaio --iodepth=64 --filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1:/dev/nvme4n1:/dev/nvme5n1:/dev/nvme6n1:/dev/nvme7n1:/dev/nvme8n1:/dev/nvme9n1 --direct=1 --runtime=90 --numjobs=80 --rw=randread --name=test --group_reporting --gtod_reduce=1With your patch: 1720k IOPS
With threaded interrupts: 1320k IOPS
With just interrupts: 3720k IOPSInterrupts are the fastest but we need to find a way to throttle it.
This is the workload that generates the flood?
If so I did not expect that this would be the perf difference..
If completions keep pounding on the cpu, I would expect irqpoll to simply keep running forever and poll the cqs. There is no fundamental reason why polling would be faster in an interrupt, what matters could be:
1. we reschedule more than we need to
2. we don't reap enough completions in every poll round, which will trigger rearming the interrupt and then when it fires reschedule another softirq...
Maybe we need to take care of some irq_poll optimizations?
Does this (untested) patch make any difference?
--
lib/irq_poll.c +11 -18
@@ -12,7 +12,8 @@#include <linux/irq_poll.h> #include <linux/delay.h> -static unsigned int irq_poll_budget__read_mostly =256 ; +static unsigned int irq_poll_budget__read_mostly =3000 ; +unsigned int __read_mostly irqpoll_budget_usecs =2000 ; static DEFINE_PER_CPU(struct list_head, blk_cpu_iopoll);
@@ -77,32 +78,26 @@ EXPORT_SYMBOL(irq_poll_complete); static void __latent_entropy irq_poll_softirq(struct softirq_action *h) { -struct list_head *list =this_cpu_ptr (&blk_cpu_iopoll); -int rearm =0 , budget = irq_poll_budget; -unsigned long start_time = jiffies; +struct list_head *irqpoll_list =this_cpu_ptr (&blk_cpu_iopoll); +unsigned int budget = irq_poll_budget; +unsigned long time_limit = + jiffies +usecs_to_jiffies (irqpoll_budget_usecs); +LIST_HEAD (list);local_irq_disable (); +list_splice_init (irqpoll_list, &list); +local_irq_enable (); -while (!list_empty (list)) { +while (!list_empty (&list)) {struct irq_poll *iop;int work, weight; -/* -* If softirq window is exhausted then punt. -*/ -if (budget <=0 ||time_after (jiffies, start_time)) { - rearm =1 ; -break ; - } - -local_irq_enable (); -/* Even though interrupts have been re-enabled, this * access is safe because interrupts can only add new * entries to the tail of this list, and only ->poll() * calls can remove this head entry from the list. */ - iop = list_entry(list->next, struct irq_poll, list); + iop = list_first_entry(&list, struct irq_poll, list); weight = iop->weight; work =0 ;
@@ -111,8 +106,6 @@ static void __latent_entropy irq_poll_softirq(struct softirq_action *h) budget -= work; -local_irq_disable (); -/* * Drivers must not modify the iopoll state, if they * consume their assigned weight (or more, some drivers
can't
@@ -125,11 +118,21 @@ static void __latent_entropy irq_poll_softirq(struct softirq_action *h)
if (test_bit(IRQ_POLL_F_DISABLE, &iop->state))
__irq_poll_complete(iop);
else
- list_move_tail(&iop->list, list);
+ list_move_tail(&iop->list, &list);
}+
+ /*
+ * If softirq window is exhausted then punt.
+ */
+ if (budget <= 0 || time_after_eq(jiffies, time_limit))
+ break;
}- if (rearm)
+ local_irq_disable();
+
+ list_splice_tail_init(irqpoll_list, &list);
+ list_splice(&list, irqpoll_list);
+ if (!list_empty(irqpoll_list))
__raise_softirq_irqoff(IRQ_POLL_SOFTIRQ);
local_irq_enable();--
RE: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Subject: Re: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts
Quoted 7 lines
Sagi,
Here are the test results.
Benchmark command:
fio --bs=4k --ioengine=libaio --iodepth=64
--filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1:/d ev/nv
Quoted 1 line
me4n1:/dev/nvme5n1:/dev/nvme6n1:/dev/nvme7n1:/dev/nvme8n1:/dev /nvme9n1
Quoted 8 lines
--direct=1 --runtime=90 --numjobs=80 --rw=randread --name=test --group_reporting --gtod_reduce=1
With your patch: 1720k IOPS
With threaded interrupts: 1320k IOPS
With just interrupts: 3720k IOPSInterrupts are the fastest but we need to find a way to throttle it.
This is the workload that generates the flood?
If so I did not expect that this would be the perf difference..
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
If completions keep pounding on the cpu, I would expect irqpoll to simply keep running forever and poll the cqs. There is no fundamental reason why polling would be faster in an interrupt, what matters could be:
1. we reschedule more than we need to
2. we don't reap enough completions in every poll round, which will trigger rearming the interrupt and then when it fires reschedule another softirq...
Yes I think it's the rescheduling that takes some. With the patch there are lots of ksoftirqd activities. (compared to nearly none without the patch) A 90 seconds FIO run shows a big difference of context switches on all CPUs:
With patch: 5755849
Without patch: 1462931
Quoted 94 lines
Quoted 94 lines
Quoted 94 lines
Maybe we need to take care of some
irq_polloptimizations?Does this (untested) patch make any difference?
--lib/irq_poll.c index 2f17b488d58e..0e94183eba15 renamed from lib/irq_poll.c +0
-0100644
lib/irq_poll.c +11
-18@@ -12,7 +12,8 @@#include <linux/irq_poll.h> #include <linux/delay.h> -static unsigned int irq_poll_budget__read_mostly =256 ; +static unsigned int irq_poll_budget__read_mostly =3000 ;unsigned int +__read_mostly irqpoll_budget_usecs =2000 ; static DEFINE_PER_CPU(struct list_head, blk_cpu_iopoll);@@ -77,32 +78,26 @@ EXPORT_SYMBOL(irq_poll_complete); static void __latent_entropy irq_poll_softirq(struct softirq_action *h) { -struct list_head *list =this_cpu_ptr (&blk_cpu_iopoll); -int rearm =0 , budget = irq_poll_budget; -unsigned long start_time = jiffies; +struct list_head *irqpoll_list =this_cpu_ptr (&blk_cpu_iopoll); +unsigned int budget = irq_poll_budget; +unsigned long time_limit = + jiffies +usecs_to_jiffies (irqpoll_budget_usecs); +LIST_HEAD (list);local_irq_disable (); +list_splice_init (irqpoll_list, &list); +local_irq_enable (); -while (!list_empty (list)) { +while (!list_empty (&list)) {struct irq_poll *iop;int work, weight; -/* -* If softirq window is exhausted then punt. -*/ -if (budget <=0 ||time_after (jiffies, start_time)) { - rearm =1 ; -break ; - } - -local_irq_enable (); -/* Even though interrupts have been re-enabled, this * access is safe because interrupts can only add new * entries to the tail of this list, and only ->poll() * calls can remove this head entry from the list. */ - iop = list_entry(list->next, struct irq_poll, list); + iop = list_first_entry(&list, struct irq_poll, list); weight = iop->weight; work =0 ;@@ -111,8 +106,6 @@ static void __latent_entropy irq_poll_softirq(struct softirq_action *h) budget -= work; -local_irq_disable (); -/* * Drivers must not modify the iopoll state, if they * consume their assigned weight (or more, some drivers can't @@
-125,11 +118,21 @@ static void__latent_entropyirq_poll_softirq(struct softirq_action *h)if (test_bit(IRQ_POLL_F_DISABLE, &iop->state)) __irq_poll_complete(iop); elselib/irq_poll.c +12
-2-list_move_tail (&iop->list, list); +list_move_tail (&iop->list, &list); } + +/* +* If softirq window is exhausted then punt. +*/ +if (budget <=0 ||time_after_eq (jiffies, time_limit)) +break ; } -if (rearm) +local_irq_disable (); + +list_splice_tail_init (irqpoll_list, &list); +list_splice (&list, irqpoll_list); +if (!list_empty (irqpoll_list))__raise_softirq_irqoff (IRQ_POLL_SOFTIRQ );
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
local_irq_enable();--
It's got slightly better IOPS. With this 2nd patch, the number of context switches is at 5445863 (~5% improvement over the 1st patch).
Re: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts
In reply to Sagi GrimbergQuoted 27 lines · Sagi Grimberg
Quoted 27 lines
From: Long Li <[email protected]>
When a NVMe hardware queue is mapped to several CPU queues, it is possible that the CPU this hardware queue is bound to is flooded by returning I/O for other CPUs.
For example, consider the following scenario:
1. CPU 0, 1, 2 and 3 share the same hardware queue
2. the hardware queue interrupts CPU 0 for I/O response
3. processes from CPU 1, 2 and 3 keep sending I/OsCPU 0 may be flooded with interrupts from NVMe device that are I/O responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible that CPU 0 spends all the time serving NVMe and other system interrupts, but doesn't have a chance to run in process context.
To fix this, CPU 0 can schedule a work to complete the I/O request when it detects the scheduler is not making progress. This serves multiple purposes:
1. This CPU has to be scheduled to complete the request. The other CPUs can't issue more I/Os until some previous I/Os are completed. This helps this CPU get out of NVMe interrupts.
2. This acts a throttling mechanisum for NVMe devices, in that it can not starve a CPU while servicing I/Os from other CPUs.
Quoted 6 lines
Quoted 1 line
3. This CPU can make progress on RCU and other work items on its queue.
The problem is indeed real, but this is the wrong approach in my mind.
We already have irqpoll which takes care proper budgeting polling cycles and not hogging the cpu.
The issue isn't unique to NVMe, and can be any fast devices which interrupts CPU too frequently, meantime the interrupt/softirq handler may take a bit much time, then CPU is easy to be lockup by the interrupt/sofirq handler, especially in case that multiple submission CPUs vs. single completion CPU.
Some SCSI devices has the same problem too.
Could we consider to add one generic mechanism to cover this kind of problem?
One approach I thought of is to allocate one backup thread for handling such interrupt, which can be marked as IRQF_BACKUP_THREAD by drivers.
Inside do_IRQ(), irqtime is accounted, before calling action->handler(), check if this CPU has taken too long time for handling IRQ(interrupt or softirq) and see if this CPU could be lock up. If yes, wakeup the backup thread to handle the interrupt for avoiding lockup this CPU.
The threaded interrupt framework is there, and this way could be easier to implement. Meantime most time the handler is run in interrupt context and we may avoid the performance loss when CPU isn't busy enough.
Any comment on this approach?
Thanks,
Ming
RE: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Subject: Re: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts
Quoted 12 lines · Sagi Grimberg
Quoted 12 lines
From: Long Li <[email protected]>
When a NVMe hardware queue is mapped to several CPU queues, it is possible that the CPU this hardware queue is bound to is flooded by returning I/O for other CPUs.
For example, consider the following scenario:
1. CPU 0, 1, 2 and 3 share the same hardware queue 2. the hardware queue interrupts CPU 0 for I/O response 3. processes from CPU 1, 2 and 3 keep sending I/OsQuoted 7 lines
Quoted 7 lines
CPU 0 may be flooded with interrupts from NVMe device that are I/O responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible that CPU 0 spends all the time serving NVMe and other system interrupts, but doesn't have a chance to run in process context.
To fix this, CPU 0 can schedule a work to complete the I/O request when it detects the scheduler is not making progress. This serves
multiple purposes:
Quoted 9 lines
Quoted 9 lines
1. This CPU has to be scheduled to complete the request. The other CPUs can't issue more I/Os until some previous I/Os are completed. This helps this CPU get out of NVMe interrupts.
2. This acts a throttling mechanisum for NVMe devices, in that it can not starve a CPU while servicing I/Os from other CPUs.
3. This CPU can make progress on RCU and other work items on its
queue.
Quoted 5 lines
The problem is indeed real, but this is the wrong approach in my mind.
We already have irqpoll which takes care proper budgeting polling cycles and not hogging the cpu.
The issue isn't unique to NVMe, and can be any fast devices which interrupts CPU too frequently, meantime the interrupt/softirq handler may take a bit much time, then CPU is easy to be lockup by the interrupt/sofirq handler, especially in case that multiple submission CPUs vs. single completion CPU.
Some SCSI devices has the same problem too.
Could we consider to add one generic mechanism to cover this kind of problem?
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
One approach I thought of is to allocate one backup thread for handling such interrupt, which can be marked as
IRQF_BACKUP_THREADby drivers.Inside
do_IRQ(), irqtime is accounted, before callingaction->handler(), check if this CPU has taken too long time for handling IRQ(interrupt or softirq) and see if this CPU could be lock up. If yes, wakeup the backup
How do you know if this CPU is spending all the time in do_IRQ()?
Is it something like:
If (IRQ_time /elapsed_time > a threshold value)
wake up the backup thread
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
thread to handle the interrupt for avoiding lockup this CPU.
The threaded interrupt framework is there, and this way could be easier to implement. Meantime most time the handler is run in interrupt context and we may avoid the performance loss when CPU isn't busy enough.
Any comment on this approach?
Thanks,
Ming
Re: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts
Quoted 58 lines · Long Li
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Subject: Re: [PATCH 3/3] nvme: complete request in work queue on CPU with flooded interrupts
Quoted 12 lines · Sagi Grimberg
Quoted 12 lines
From: Long Li <[email protected]>
When a NVMe hardware queue is mapped to several CPU queues, it is possible that the CPU this hardware queue is bound to is flooded by returning I/O for other CPUs.
For example, consider the following scenario:
1. CPU 0, 1, 2 and 3 share the same hardware queue 2. the hardware queue interrupts CPU 0 for I/O response 3. processes from CPU 1, 2 and 3 keep sending I/OsQuoted 7 lines
Quoted 7 lines
CPU 0 may be flooded with interrupts from NVMe device that are I/O responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible that CPU 0 spends all the time serving NVMe and other system interrupts, but doesn't have a chance to run in process context.
To fix this, CPU 0 can schedule a work to complete the I/O request when it detects the scheduler is not making progress. This serves
multiple purposes:
Quoted 9 lines
Quoted 9 lines
1. This CPU has to be scheduled to complete the request. The other CPUs can't issue more I/Os until some previous I/Os are completed. This helps this CPU get out of NVMe interrupts.
2. This acts a throttling mechanisum for NVMe devices, in that it can not starve a CPU while servicing I/Os from other CPUs.
3. This CPU can make progress on RCU and other work items on its
queue.
Quoted 5 lines
The problem is indeed real, but this is the wrong approach in my mind.
We already have irqpoll which takes care proper budgeting polling cycles and not hogging the cpu.
The issue isn't unique to NVMe, and can be any fast devices which interrupts CPU too frequently, meantime the interrupt/softirq handler may take a bit much time, then CPU is easy to be lockup by the interrupt/sofirq handler, especially in case that multiple submission CPUs vs. single completion CPU.
Some SCSI devices has the same problem too.
Could we consider to add one generic mechanism to cover this kind of problem?
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
One approach I thought of is to allocate one backup thread for handling such interrupt, which can be marked as
IRQF_BACKUP_THREADby drivers.Inside
do_IRQ(), irqtime is accounted, before callingaction->handler(), check if this CPU has taken too long time for handling IRQ(interrupt or softirq) and see if this CPU could be lock up. If yes, wakeup the backup
Quoted 5 lines
How do you know if this CPU is spending all the time in
do_IRQ()?Is it something like:
If (IRQ_time/elapsed_time > a threshold value)
wake up the backup thread
Yeah, the above could work in theory.
Another approach I thought of is to monitor average irq gap time on each CPU.
We could use EWMA(Exponential Weighted Moving Average) to do it simply, such as:
curr_irq_gap(cpu) = current start time of do_IRQ() on 'cpu' -
end time of last do_IRQ() on 'cpu'
avg_irq_gap(cpu) = weight_prev * avg_irq_gap(cpu) + weight_curr * curr_irq_gap(cpu)
note:
weight_prev + weight_curr = 1
When avg_irq_gap(cpu) is close to one small enough threshold, we think irq flood is detected.
'weight_prev' could be chosen as one big enough value for avoiding short-time flood.
Thanks,
Ming
Quoted 21 lines · On Tue, Aug 20, 2019 at 2:14 PM <[email protected]> wrote:
From: Long Li <[email protected]>
This patch set tries to fix interrupt swamp in NVMe devices.
On large systems with many CPUs, a number of CPUs may share one NVMe hardware queue. It may have this situation where several CPUs are issuing I/Os, and all the I/Os are returned on the CPU where the hardware queue is bound to. This may result in that CPU swamped by interrupts and stay in interrupt mode for extended time while other CPUs continue to issue I/O. This can trigger Watchdog and RCU timeout, and make the system unresponsive.
This patch set addresses this by enforcing scheduling and throttling I/O when CPU is starved in this situation.
Long Li (3):
sched: define a function to report the number of context switches on a
CPU
sched: exportidle_cpu()
nvme: complete request in work queue on CPU with flooded interrupts
Quoted 5 lines
drivers/nvme/host/core.c | 57 +++++++++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 1 + include/linux/sched.h | 2 ++ kernel/sched/core.c | 7 +++++ 4 files changed, 66 insertions(+), 1 deletion(-)
Another simpler solution may be to complete request in threaded interrupt handler for this case. Meantime allow scheduler to run the interrupt thread handler on CPUs specified by the irq affinity mask, which was discussed by the following link:
https://lore.kernel.org/lkml/[email protected]/
Could you try the above solution and see if the lockup can be avoided? John Garry
should have workable patch.
Thanks,
Ming Lei
Quoted 33 lines · Ming Lei
Quoted 21 lines · On Tue, Aug 20, 2019 at 2:14 PM <[email protected]> wrote:
From: Long Li <[email protected]>
This patch set tries to fix interrupt swamp in NVMe devices.
On large systems with many CPUs, a number of CPUs may share one NVMe hardware queue. It may have this situation where several CPUs are issuing I/Os, and all the I/Os are returned on the CPU where the hardware queue is bound to. This may result in that CPU swamped by interrupts and stay in interrupt mode for extended time while other CPUs continue to issue I/O. This can trigger Watchdog and RCU timeout, and make the system unresponsive.
This patch set addresses this by enforcing scheduling and throttling I/O when CPU is starved in this situation.
Long Li (3):
sched: define a function to report the number of context switches on a
CPU
sched: exportidle_cpu()
nvme: complete request in work queue on CPU with flooded interruptsQuoted 5 lines
drivers/nvme/host/core.c | 57 +++++++++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 1 + include/linux/sched.h | 2 ++ kernel/sched/core.c | 7 +++++ 4 files changed, 66 insertions(+), 1 deletion(-)Another simpler solution may be to complete request in threaded interrupt handler for this case. Meantime allow scheduler to run the interrupt thread handler on CPUs specified by the irq affinity mask, which was discussed by the following link:
Quoted 5 lines
https://lore.kernel.org/lkml/[email protected]/
Could you try the above solution and see if the lockup can be avoided? John Garry
should have workable patch.
Yeah, so we experimented with changing the interrupt handling in the SCSI driver I maintain to use a threaded handler IRQ handler plus patch below, and saw a significant throughput boost:
--->8
Subject: [PATCH] genirq: Add support to allow thread to use hard irq affinity
Currently the cpu allowed mask for the threaded part of a threaded irq handler will be set to the effective affinity of the hard irq.
Typically the effective affinity of the hard irq will be for a single cpu. As such, the threaded handler would always run on the same cpu as the hard irq.
We have seen scenarios in high data-rate throughput testing that the cpu handling the interrupt can be totally saturated handling both the hard interrupt and threaded handler parts, limiting throughput.
Add IRQF_IRQ_AFFINITY flag to allow the driver requesting the threaded interrupt to decide on the policy of which cpu the threaded handler may run.
include/linux/interrupt.h +0 -0
@@ -61,6 +61,9 @@* interrupt handler after suspending interrupts. For
system
* wakeup devices users need to implement wakeup
detection in
* their interrupt handlers.
+ * IRQF_IRQ_AFFINITY - Use the hard interrupt affinity for setting the cpu
+ * allowed mask for the threaded handler of a threaded interrupt
+ * handler, rather than the effective hard irq affinity.
*/
#define IRQF_SHARED 0x00000080
#define IRQF_PROBE_SHARED 0x00000100
@@ -74,6 +77,7 @@
#define IRQF_NO_THREAD 0x00010000
#define IRQF_EARLY_RESUME 0x00020000
#define IRQF_COND_SUSPEND 0x00040000
+#define IRQF_IRQ_AFFINITY 0x00080000
#define IRQF_TIMER (__IRQF_TIMER | IRQF_NO_SUSPEND | IRQF_NO_THREAD)
kernel/irq/manage.c +5 -1
@@ -966,9 +966,13 @@ irq_thread_check_affinity(struct irq_desc *desc, struct irqaction *action)* mask pointer. For CPU_MASK_OFFSTACK=n this is optimized out. */ if (cpumask_available (desc->irq_common_data.affinity)) { +struct irq_data *irq_data = &desc->irq_data;const struct cpumask *m; - m =irq_data_get_effective_affinity_mask (&desc->irq_data); +if (action->flags &IRQF_IRQ_AFFINITY ) + m = desc->irq_common_data.affinity; +else + m =irq_data_get_effective_affinity_mask (irq_data);cpumask_copy (mask, m); }else { valid = false;
Signature · 6 lines
2.17.1 As Ming mentioned in that same thread, we could even make this policy for managed interrupts. Cheers, John
Quoted 6 lines
Thanks,
Ming Lei.
Keith Busch
maintainerQuoted 19 lines · John Garry
kernel/irq/manage.c +5
-1@@ -966,9 +966,13 @@ irq_thread_check_affinity(struct irq_desc *desc,struct irqaction *action) * mask pointer. For CPU_MASK_OFFSTACK=n this is optimized out. */ if (cpumask_available (desc->irq_common_data.affinity)) { +struct irq_data *irq_data = &desc->irq_data;const struct cpumask *m; - m =irq_data_get_effective_affinity_mask (&desc->irq_data); +if (action->flags &IRQF_IRQ_AFFINITY ) + m = desc->irq_common_data.affinity; +else + m =irq_data_get_effective_affinity_mask (irq_data);cpumask_copy (mask, m); }else {
Quoted 6 lines
valid = false;
Signature · 4 lines
2.17.1 As Ming mentioned in that same thread, we could even make this policy for managed interrupts.
Ack, I really like this option!
Quoted 73 lines
Quoted 73 lines
Quoted 73 lines
Subject: Re: [PATCH 0/3] fix interrupt swamp in NVMe
Quoted 7 lines · Ming Lei
Quoted 6 lines · On Tue, Aug 20, 2019 at 2:14 PM <[email protected]> wrote:
From: Long Li <[email protected]>
This patch set tries to fix interrupt swamp in NVMe devices.
On large systems with many CPUs, a number of CPUs may share one
NVMe
Quoted 2 lines
Quoted 2 lines
hardware queue. It may have this situation where several CPUs are issuing I/Os, and all the I/Os are returned on the CPU where the
hardware queue is bound to.
Quoted 3 lines
Quoted 3 lines
This may result in that CPU swamped by interrupts and stay in interrupt mode for extended time while other CPUs continue to issue I/O. This can trigger Watchdog and RCU timeout, and make the system
unresponsive.
Quoted 17 lines
Quoted 10 lines
This patch set addresses this by enforcing scheduling and throttling I/O when CPU is starved in this situation.
Long Li (3):
sched: define a function to report the number of context switches on a
CPU
sched: exportidle_cpu()
nvme: complete request in work queue on CPU with flooded interruptsQuoted 6 lines
drivers/nvme/host/core.c | 57 +++++++++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 1 + include/linux/sched.h | 2 ++ kernel/sched/core.c | 7 +++++ 4 files changed, 66 insertions(+), 1 deletion(-)Quoted 6 lines
Another simpler solution may be to complete request in threaded interrupt handler for this case. Meantime allow scheduler to run the interrupt thread handler on CPUs specified by the irq affinity mask, which was discussed by the following link:
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flor
eQuoted 1 line
.kernel.org%2Flkml%2Fe0e9478e-62a5-ca24-3b12-
58f7d056383e%40huawei.com
Quoted 1 line
%2F&data=02%7C01%7Clongli%40microsoft.com%7Cc7f46d3e273f45
176d1c08Quoted 1 line
d7254cc69e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370188 8401588
Quoted 1 line
9866&sdata=h5k6HoGoyDxuhmDfuKLZUwgmw17PU%2BT%2FCbawfxV Er3U%3D&
Quoted 5 lines
reserved=0
Could you try the above solution and see if the lockup can be avoided? John Garry
should have workable patch.Yeah, so we experimented with changing the interrupt handling in the SCSI driver I maintain to use a threaded handler IRQ handler plus patch below, and saw a significant throughput boost:
--->8
Subject: [PATCH] genirq: Add support to allow thread to use hard irq affinity
Currently the cpu allowed mask for the threaded part of a threaded irq handler will be set to the effective affinity of the hard irq.
Typically the effective affinity of the hard irq will be for a single cpu. As such, the threaded handler would always run on the same cpu as the hard irq.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
We have seen scenarios in high data-rate throughput testing that the cpu handling the interrupt can be totally saturated handling both the hard interrupt and threaded handler parts, limiting throughput.
Add
IRQF_IRQ_AFFINITYflag to allow the driver requesting the threaded interrupt to decide on the policy of which cpu the threaded handler may run.Signed-off-by John Garry <[email protected]>
Thanks for pointing me to this patch. This fixed the interrupt swamp and make the system stable.
However I'm seeing reduced performance when using threaded interrupts.
Here are the test results on a system with 80 CPUs and 10 NVMe disks (32 hardware queues for each disk) Benchmark tool is FIO, I/O pattern: 4k random reads on all NVMe disks, with queue depth = 64, num of jobs = 80, direct=1
With threaded interrupts: 1320k IOPS
With just interrupts: 3720k IOPS
With just interrupts and my patch: 3700k IOPS
At the peak IOPS, the overall CPU usage is at around 98-99%. I think the cost of doing wake up and context switch for NVMe threaded IRQ handler takes some CPU away.
In this test, I made the following change to make use of IRQF_IRQ_AFFINITY for NVMe:
drivers/pci/irq.c +1 -1
@@ -86,7 +86,7 @@ int pci_request_irq(struct pci_dev *dev, unsigned int nr, irq_handler_t handler,va_list ap;int ret;char *devname; -unsigned long irqflags =IRQF_SHARED ; +unsigned long irqflags =IRQF_SHARED |IRQF_IRQ_AFFINITY ;if (!handler) irqflags |=IRQF_ONESHOT ;
Thanks
Long
Quoted 64 lines
Quoted 64 lines
Quoted 64 lines
include/linux/interrupt.h index renamed from include/linux/interrupt.h +0
-0
5b8328a99b2a..48e8b955989a100644include/linux/interrupt.h +0
-0@@ -61,6 +61,9 @@* interrupt handler after suspending interrupts. For system
* wakeup devices users need to implement wakeup
detection in
* their interrupt handlers.
+ *IRQF_IRQ_AFFINITY- Use the hard interrupt affinity for setting the cpu
+ * allowed mask for the threaded handler of a threaded
interrupt
+ * handler, rather than the effective hard irq affinity.
*/
#defineIRQF_SHARED0x00000080
#defineIRQF_PROBE_SHARED0x00000100include/linux/interrupt.h +1
-0@@ -74,6 +77,7 @@#define IRQF_NO_THREAD 0x00010000#define IRQF_EARLY_RESUME 0x00020000#define IRQF_COND_SUSPEND 0x00040000 +#define IRQF_IRQ_AFFINITY 0x00080000#define IRQF_TIMER (__IRQF_TIMER | IRQF_NO_SUSPEND | IRQF_NO_THREAD)kernel/irq/manage.c index renamed from kernel/irq/manage.c +0
-0
e8f7f179bf77..cb483a055512100644kernel/irq/manage.c +5
-1@@ -966,9 +966,13 @@ irq_thread_check_affinity(struct irq_desc *desc,struct irqaction *action) * mask pointer. For CPU_MASK_OFFSTACK=n this is optimized out. */ if (cpumask_available (desc->irq_common_data.affinity)) { +struct irq_data *irq_data = &desc->irq_data;const struct cpumask *m; - m =irq_data_get_effective_affinity_mask (&desc- >irq_data); +if (action->flags &IRQF_IRQ_AFFINITY ) + m = desc->irq_common_data.affinity; +else + m =irq_data_get_effective_affinity_mask (irq_data);cpumask_copy (mask, m);kernel/irq/manage.c +0
-1}else { valid = false; --2.17.1
As Ming mentioned in that same thread, we could even make this policy for managed interrupts.
Cheers,
JohnQuoted 6 lines
Thanks,
Ming Lei.
Quoted 84 lines · Long Li
Quoted 73 lines
Quoted 73 lines
Quoted 73 lines
Subject: Re: [PATCH 0/3] fix interrupt swamp in NVMe
Quoted 7 lines · Ming Lei
Quoted 6 lines · On Tue, Aug 20, 2019 at 2:14 PM <[email protected]> wrote:
From: Long Li <[email protected]>
This patch set tries to fix interrupt swamp in NVMe devices.
On large systems with many CPUs, a number of CPUs may share one
NVMe
Quoted 2 lines
Quoted 2 lines
hardware queue. It may have this situation where several CPUs are issuing I/Os, and all the I/Os are returned on the CPU where the
hardware queue is bound to.
Quoted 3 lines
Quoted 3 lines
This may result in that CPU swamped by interrupts and stay in interrupt mode for extended time while other CPUs continue to issue I/O. This can trigger Watchdog and RCU timeout, and make the system
unresponsive.
Quoted 17 lines
Quoted 10 lines
This patch set addresses this by enforcing scheduling and throttling I/O when CPU is starved in this situation.
Long Li (3):
sched: define a function to report the number of context switches on a
CPU
sched: exportidle_cpu()
nvme: complete request in work queue on CPU with flooded interruptsQuoted 6 lines
drivers/nvme/host/core.c | 57 +++++++++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 1 + include/linux/sched.h | 2 ++ kernel/sched/core.c | 7 +++++ 4 files changed, 66 insertions(+), 1 deletion(-)Quoted 6 lines
Another simpler solution may be to complete request in threaded interrupt handler for this case. Meantime allow scheduler to run the interrupt thread handler on CPUs specified by the irq affinity mask, which was discussed by the following link:
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flor
eQuoted 1 line
.kernel.org%2Flkml%2Fe0e9478e-62a5-ca24-3b12-
58f7d056383e%40huawei.com
Quoted 1 line
%2F&data=02%7C01%7Clongli%40microsoft.com%7Cc7f46d3e273f45
176d1c08Quoted 1 line
d7254cc69e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370188 8401588
Quoted 1 line
9866&sdata=h5k6HoGoyDxuhmDfuKLZUwgmw17PU%2BT%2FCbawfxV Er3U%3D&
Quoted 5 lines
reserved=0
Could you try the above solution and see if the lockup can be avoided? John Garry
should have workable patch.Yeah, so we experimented with changing the interrupt handling in the SCSI driver I maintain to use a threaded handler IRQ handler plus patch below, and saw a significant throughput boost:
--->8
Subject: [PATCH] genirq: Add support to allow thread to use hard irq affinity
Currently the cpu allowed mask for the threaded part of a threaded irq handler will be set to the effective affinity of the hard irq.
Typically the effective affinity of the hard irq will be for a single cpu. As such, the threaded handler would always run on the same cpu as the hard irq.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
We have seen scenarios in high data-rate throughput testing that the cpu handling the interrupt can be totally saturated handling both the hard interrupt and threaded handler parts, limiting throughput.
Add
IRQF_IRQ_AFFINITYflag to allow the driver requesting the threaded interrupt to decide on the policy of which cpu the threaded handler may run.Signed-off-by John Garry <[email protected]>Thanks for pointing me to this patch. This fixed the interrupt swamp and make the system stable.
Quoted 8 lines
However I'm seeing reduced performance when using threaded interrupts.
Here are the test results on a system with 80 CPUs and 10 NVMe disks (32 hardware queues for each disk) Benchmark tool is FIO, I/O pattern: 4k random reads on all NVMe disks, with queue depth = 64, num of jobs = 80, direct=1
With threaded interrupts: 1320k IOPS
With just interrupts: 3720k IOPS
With just interrupts and my patch: 3700k IOPS
This gap looks too big wrt. threaded interrupts vs. interrupts.
Quoted 3 lines
At the peak IOPS, the overall CPU usage is at around 98-99%. I think the cost of doing wake up and context switch for NVMe threaded IRQ handler takes some CPU away.
In theory, it shouldn't be so because most of times the thread should be running on CPUs of this hctx, and the wakeup cost shouldn't be so big. Maybe there is performance problem somewhere wrt. threaded interrupt.
Could you share us your test script and environment? I will see if I can reproduce it in my environment.
Quoted 8 lines
In this test, I made the following change to make use of
IRQF_IRQ_AFFINITYfor NVMe:drivers/pci/irq.c +0
-0
drivers/pci/irq.c int pci_request_irq(struct pci_dev *dev, unsigned int nr, irq_handler_t handler, · line 92
Quoted 8 lines
drivers/pci/irq.c +1
-1@@ -86,7 +86,7 @@ int pci_request_irq(struct pci_dev *dev, unsigned int nr, irq_handler_t handler,va_list ap;int ret;char *devname; -unsigned long irqflags =IRQF_SHARED ; +unsigned long irqflags =IRQF_SHARED |IRQF_IRQ_AFFINITY ;if (!handler) irqflags |=IRQF_ONESHOT ;
I don't see why IRQF_IRQ_AFFINITY is needed.
John, could you explain it a bit why you need changes on IRQF_IRQ_AFFINITY?
The following patch should be enough:
kernel/irq/manage.c +5 -1
@@ -968,7 +968,11 @@ irq_thread_check_affinity(struct irq_desc *desc, struct irqaction *action)if (cpumask_available (desc->irq_common_data.affinity)) {const struct cpumask *m; - m =irq_data_get_effective_affinity_mask (&desc->irq_data); +if (irqd_affinity_is_managed (&desc->irq_data)) + m = desc->irq_common_data.affinity; +else + m =irq_data_get_effective_affinity_mask ( + &desc->irq_data);cpumask_copy (mask, m); }else { valid = false;
Thanks,
Ming
Quoted 120 lines · Ming Lei
Quoted 84 lines · Long Li
Quoted 73 lines
Quoted 73 lines
Quoted 73 lines
Subject: Re: [PATCH 0/3] fix interrupt swamp in NVMe
Quoted 7 lines · Ming Lei
Quoted 6 lines · On Tue, Aug 20, 2019 at 2:14 PM <[email protected]> wrote:
From: Long Li <[email protected]>
This patch set tries to fix interrupt swamp in NVMe devices.
On large systems with many CPUs, a number of CPUs may share one
NVMe
Quoted 2 lines
Quoted 2 lines
hardware queue. It may have this situation where several CPUs are issuing I/Os, and all the I/Os are returned on the CPU where the
hardware queue is bound to.
Quoted 3 lines
Quoted 3 lines
This may result in that CPU swamped by interrupts and stay in interrupt mode for extended time while other CPUs continue to issue I/O. This can trigger Watchdog and RCU timeout, and make the system
unresponsive.
Quoted 17 lines
Quoted 10 lines
This patch set addresses this by enforcing scheduling and throttling I/O when CPU is starved in this situation.
Long Li (3):
sched: define a function to report the number of context switches on a
CPU
sched: exportidle_cpu()
nvme: complete request in work queue on CPU with flooded interruptsQuoted 6 lines
drivers/nvme/host/core.c | 57 +++++++++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 1 + include/linux/sched.h | 2 ++ kernel/sched/core.c | 7 +++++ 4 files changed, 66 insertions(+), 1 deletion(-)Quoted 6 lines
Another simpler solution may be to complete request in threaded interrupt handler for this case. Meantime allow scheduler to run the interrupt thread handler on CPUs specified by the irq affinity mask, which was discussed by the following link:
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flor
eQuoted 1 line
.kernel.org%2Flkml%2Fe0e9478e-62a5-ca24-3b12-
58f7d056383e%40huawei.com
Quoted 1 line
%2F&data=02%7C01%7Clongli%40microsoft.com%7Cc7f46d3e273f45
176d1c08Quoted 1 line
d7254cc69e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370188 8401588
Quoted 1 line
9866&sdata=h5k6HoGoyDxuhmDfuKLZUwgmw17PU%2BT%2FCbawfxV Er3U%3D&
Quoted 5 lines
reserved=0
Could you try the above solution and see if the lockup can be avoided? John Garry
should have workable patch.Yeah, so we experimented with changing the interrupt handling in the SCSI driver I maintain to use a threaded handler IRQ handler plus patch below, and saw a significant throughput boost:
--->8
Subject: [PATCH] genirq: Add support to allow thread to use hard irq affinity
Currently the cpu allowed mask for the threaded part of a threaded irq handler will be set to the effective affinity of the hard irq.
Typically the effective affinity of the hard irq will be for a single cpu. As such, the threaded handler would always run on the same cpu as the hard irq.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
We have seen scenarios in high data-rate throughput testing that the cpu handling the interrupt can be totally saturated handling both the hard interrupt and threaded handler parts, limiting throughput.
Add
IRQF_IRQ_AFFINITYflag to allow the driver requesting the threaded interrupt to decide on the policy of which cpu the threaded handler may run.Signed-off-by John Garry <[email protected]>Thanks for pointing me to this patch. This fixed the interrupt swamp and make the system stable.
Quoted 8 lines
However I'm seeing reduced performance when using threaded interrupts.
Here are the test results on a system with 80 CPUs and 10 NVMe disks (32 hardware queues for each disk) Benchmark tool is FIO, I/O pattern: 4k random reads on all NVMe disks, with queue depth = 64, num of jobs = 80, direct=1
With threaded interrupts: 1320k IOPS
With just interrupts: 3720k IOPS
With just interrupts and my patch: 3700k IOPSThis gap looks too big wrt. threaded interrupts vs. interrupts.
Quoted 3 lines
At the peak IOPS, the overall CPU usage is at around 98-99%. I think the cost of doing wake up and context switch for NVMe threaded IRQ handler takes some CPU away.
In theory, it shouldn't be so because most of times the thread should be running on CPUs of this hctx, and the wakeup cost shouldn't be so big. Maybe there is performance problem somewhere wrt. threaded interrupt.
Could you share us your test script and environment? I will see if I can reproduce it in my environment.
Quoted 13 lines
In this test, I made the following change to make use of
IRQF_IRQ_AFFINITYfor NVMe:drivers/pci/irq.c +1
-1@@ -86,7 +86,7 @@ int pci_request_irq(struct pci_dev *dev, unsigned int nr, irq_handler_t handler,va_list ap;int ret;char *devname; -unsigned long irqflags =IRQF_SHARED ; +unsigned long irqflags =IRQF_SHARED |IRQF_IRQ_AFFINITY ;
Quoted 7 lines
drivers/pci/irq.c int pci_request_irq(struct pci_dev *dev, unsigned int nr, irq_handler_t handler, · line 92
Quoted 3 lines
drivers/pci/irq.c +0
-0if (!handler) irqflags |=IRQF_ONESHOT ;I don't see why
IRQF_IRQ_AFFINITYis needed.John, could you explain it a bit why you need changes on
IRQF_IRQ_AFFINITY?
Hi Ming,
The patch I shared was my original solution, based on the driver setting flag IRQF_IRQ_AFFINITY to request the threaded handler uses the irq affinity mask for the handler cpu allowed mask.
If we want to make this decision based only on whether the irq is managed or not, then we can drop the flag and just make the change as you suggest, below.
Thanks,
John
Quoted 27 lines
The following patch should be enough:
kernel/irq/manage.c +5
-1@@ -968,7 +968,11 @@ irq_thread_check_affinity(struct irq_desc *desc, struct irqaction *action)if (cpumask_available (desc->irq_common_data.affinity)) {const struct cpumask *m; - m =irq_data_get_effective_affinity_mask (&desc->irq_data); +if (irqd_affinity_is_managed (&desc->irq_data)) + m = desc->irq_common_data.affinity; +else + m =irq_data_get_effective_affinity_mask ( + &desc->irq_data);cpumask_copy (mask, m); }else { valid = false;Thanks,
Ming.
Quoted 121 lines
Quoted 121 lines
Quoted 121 lines
Subject: Re: [PATCH 0/3] fix interrupt swamp in NVMe
Quoted 10 lines · Long Li
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Subject: Re: [PATCH 0/3] fix interrupt swamp in NVMe
Quoted 7 lines · Ming Lei
Quoted 6 lines · On Tue, Aug 20, 2019 at 2:14 PM <[email protected]> wrote:
From: Long Li <[email protected]>
This patch set tries to fix interrupt swamp in NVMe devices.
On large systems with many CPUs, a number of CPUs may share
one
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
NVMe
Quoted 3 lines
Quoted 3 lines
hardware queue. It may have this situation where several CPUs are issuing I/Os, and all the I/Os are returned on the CPU where the
hardware queue is bound to.
Quoted 4 lines
Quoted 4 lines
This may result in that CPU swamped by interrupts and stay in interrupt mode for extended time while other CPUs continue to issue I/O. This can trigger Watchdog and RCU timeout, and make the system
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
unresponsive.
Quoted 6 lines
Quoted 6 lines
This patch set addresses this by enforcing scheduling and throttling I/O when CPU is starved in this situation.
Long Li (3):
sched: define a function to report the number of context switcheson a
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 11 lines
CPU
sched: exportidle_cpu()
nvme: complete request in work queue on CPU with flooded
interruptsdrivers/nvme/host/core.c | 57 +++++++++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 1 + include/linux/sched.h | 2 ++ kernel/sched/core.c | 7 +++++ 4 files changed, 66 insertions(+), 1 deletion(-)Another simpler solution may be to complete request in threaded interrupt handler for this case. Meantime allow scheduler to run the interrupt thread handler on CPUs specified by the irq affinity mask, which was discussed by the following link:
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
.kernel.org%2Flkml%2Fe0e9478e-62a5-ca24-3b12-
58f7d056383e%40huawei.com
Quoted 1 line
%2F&data=02%7C01%7Clongli%40microsoft.com%7Cc7f46d3e2
73f45
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
176d1c08Quoted 1 line
d7254cc69e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63
70188
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
8401588
Quoted 1 line
9866&sdata=h5k6HoGoyDxuhmDfuKLZUwgmw17PU%2BT%2FCb
awfxV
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Er3U%3D&
Quoted 3 lines
reserved=0
Could you try the above solution and see if the lockup can be
avoided?
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 2 lines
John Garry
should have workable patch.Yeah, so we experimented with changing the interrupt handling in the SCSI driver I maintain to use a threaded handler IRQ handler plus patch below, and saw a significant throughput boost:
--->8
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Subject: [PATCH] genirq: Add support to allow thread to use hard irq affinity
Currently the cpu allowed mask for the threaded part of a threaded irq handler will be set to the effective affinity of the hard irq.
Typically the effective affinity of the hard irq will be for a single cpu. As such, the threaded handler would always run on the
same cpu as the hard irq.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
We have seen scenarios in high data-rate throughput testing that the cpu handling the interrupt can be totally saturated handling both the hard interrupt and threaded handler parts, limiting
throughput.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Add
IRQF_IRQ_AFFINITYflag to allow the driver requesting the threaded interrupt to decide on the policy of which cpu the threadedhandler may run.
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Signed-off-by John Garry <[email protected]>Thanks for pointing me to this patch. This fixed the interrupt swamp and
make the system stable.
Quoted 2 lines
However I'm seeing reduced performance when using threaded
interrupts.
Quoted 9 lines
Here are the test results on a system with 80 CPUs and 10 NVMe disks (32 hardware queues for each disk) Benchmark tool is FIO, I/O pattern:
4k random reads on all NVMe disks, with queue depth = 64, num of jobs = 80, direct=1With threaded interrupts: 1320k IOPS
With just interrupts: 3720k IOPS
With just interrupts and my patch: 3700k IOPSThis gap looks too big wrt. threaded interrupts vs. interrupts.
Quoted 2 lines
At the peak IOPS, the overall CPU usage is at around 98-99%. I think the
cost of doing wake up and context switch for NVMe threaded IRQ handler takes some CPU away.
Quoted 1 line
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
In theory, it shouldn't be so because most of times the thread should be running on CPUs of this hctx, and the wakeup cost shouldn't be so big. Maybe there is performance problem somewhere wrt. threaded interrupt.
Could you share us your test script and environment? I will see if I can reproduce it in my environment.
Ming, do you have access to L80s_v2 in Azure? This test needs to run on that VM size.
Here is the command to benchmark it:
fio --bs=4k --ioengine=libaio --iodepth=128 --filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1:/dev/nvme4n1:/dev/nvme5n1:/dev/nvme6n1:/dev/nvme7n1:/dev/nvme8n1:/dev/nvme9n1 --direct=1 --runtime=120 --numjobs=80 --rw=randread --name=test --group_reporting --gtod_reduce=1
Thanks
Long
Quoted 50 lines
Quoted 50 lines
Quoted 50 lines
Quoted 2 lines
In this test, I made the following change to make use of
IRQF_IRQ_AFFINITYfor NVMe:drivers/pci/irq.c int pci_request_irq(struct pci_dev *dev, unsigned int
Quoted 6 lines
drivers/pci/irq.c index renamed from drivers/pci/irq.c +0
-0
a1de501a2729..3fb30d16464e100644drivers/pci/irq.c +0
-0@@ -86,7 +86,7 @@ int pci_request_irq(struct pci_dev *dev, unsigned intnr,
irq_handler_thandler,drivers/pci/irq.c int pci_request_irq(struct pci_dev *dev, unsigned int · line 92
Quoted 9 lines
drivers/pci/irq.c +1
-1va_list ap;int ret;char *devname; -unsigned long irqflags =IRQF_SHARED ; +unsigned long irqflags =IRQF_SHARED |IRQF_IRQ_AFFINITY ;if (!handler) irqflags |=IRQF_ONESHOT ;I don't see why
IRQF_IRQ_AFFINITYis needed.John, could you explain it a bit why you need changes on
IRQF_IRQ_AFFINITY?The following patch should be enough:
kernel/irq/manage.c index renamed from kernel/irq/manage.c +0
-0
e8f7f179bf77..1e7cffc1c20c100644kernel/irq/manage.c +5
-1@@ -968,7 +968,11 @@ irq_thread_check_affinity(struct irq_desc *desc, struct irqaction *action)if (cpumask_available (desc->irq_common_data.affinity)) {const struct cpumask *m; - m =irq_data_get_effective_affinity_mask (&desc- >irq_data); +if (irqd_affinity_is_managed (&desc->irq_data)) + m = desc->irq_common_data.affinity; +else + m =irq_data_get_effective_affinity_mask ( + &desc->irq_data);cpumask_copy (mask, m);} else { valid = false;Thanks,
Ming
Quoted 125 lines · Long Li
Quoted 125 lines
Quoted 125 lines
Quoted 125 lines
Subject: Re: [PATCH 0/3] fix interrupt swamp in NVMe
Quoted 10 lines · Long Li
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Subject: Re: [PATCH 0/3] fix interrupt swamp in NVMe
Quoted 7 lines · Ming Lei
Quoted 6 lines · On Tue, Aug 20, 2019 at 2:14 PM <[email protected]> wrote:
From: Long Li <[email protected]>
This patch set tries to fix interrupt swamp in NVMe devices.
On large systems with many CPUs, a number of CPUs may share
one
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
NVMe
Quoted 3 lines
Quoted 3 lines
hardware queue. It may have this situation where several CPUs are issuing I/Os, and all the I/Os are returned on the CPU where the
hardware queue is bound to.
Quoted 4 lines
Quoted 4 lines
This may result in that CPU swamped by interrupts and stay in interrupt mode for extended time while other CPUs continue to issue I/O. This can trigger Watchdog and RCU timeout, and make the system
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
unresponsive.
Quoted 6 lines
Quoted 6 lines
This patch set addresses this by enforcing scheduling and throttling I/O when CPU is starved in this situation.
Long Li (3):
sched: define a function to report the number of context switcheson a
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 11 lines
CPU
sched: exportidle_cpu()
nvme: complete request in work queue on CPU with flooded
interruptsdrivers/nvme/host/core.c | 57 +++++++++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 1 + include/linux/sched.h | 2 ++ kernel/sched/core.c | 7 +++++ 4 files changed, 66 insertions(+), 1 deletion(-)Another simpler solution may be to complete request in threaded interrupt handler for this case. Meantime allow scheduler to run the interrupt thread handler on CPUs specified by the irq affinity mask, which was discussed by the following link:
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
.kernel.org%2Flkml%2Fe0e9478e-62a5-ca24-3b12-
58f7d056383e%40huawei.com
Quoted 1 line
%2F&data=02%7C01%7Clongli%40microsoft.com%7Cc7f46d3e2
73f45
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
176d1c08Quoted 1 line
d7254cc69e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63
70188
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
8401588
Quoted 1 line
9866&sdata=h5k6HoGoyDxuhmDfuKLZUwgmw17PU%2BT%2FCb
awfxV
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Er3U%3D&
Quoted 3 lines
reserved=0
Could you try the above solution and see if the lockup can be
avoided?
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 2 lines
John Garry
should have workable patch.Yeah, so we experimented with changing the interrupt handling in the SCSI driver I maintain to use a threaded handler IRQ handler plus patch below, and saw a significant throughput boost:
--->8
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Subject: [PATCH] genirq: Add support to allow thread to use hard irq affinity
Currently the cpu allowed mask for the threaded part of a threaded irq handler will be set to the effective affinity of the hard irq.
Typically the effective affinity of the hard irq will be for a single cpu. As such, the threaded handler would always run on the
same cpu as the hard irq.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
We have seen scenarios in high data-rate throughput testing that the cpu handling the interrupt can be totally saturated handling both the hard interrupt and threaded handler parts, limiting
throughput.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Add
IRQF_IRQ_AFFINITYflag to allow the driver requesting the threaded interrupt to decide on the policy of which cpu the threadedhandler may run.
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Signed-off-by John Garry <[email protected]>Thanks for pointing me to this patch. This fixed the interrupt swamp and
make the system stable.
Quoted 2 lines
However I'm seeing reduced performance when using threaded
interrupts.
Quoted 9 lines
Here are the test results on a system with 80 CPUs and 10 NVMe disks (32 hardware queues for each disk) Benchmark tool is FIO, I/O pattern:
4k random reads on all NVMe disks, with queue depth = 64, num of jobs = 80, direct=1With threaded interrupts: 1320k IOPS
With just interrupts: 3720k IOPS
With just interrupts and my patch: 3700k IOPSThis gap looks too big wrt. threaded interrupts vs. interrupts.
Quoted 2 lines
At the peak IOPS, the overall CPU usage is at around 98-99%. I think the
cost of doing wake up and context switch for NVMe threaded IRQ handler takes some CPU away.
Quoted 1 line
In theory, it shouldn't be so because most of times the thread should be running on CPUs of this hctx, and the wakeup cost shouldn't be so big. Maybe there is performance problem somewhere wrt. threaded interrupt.
Quoted 9 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Could you share us your test script and environment? I will see if I can reproduce it in my environment.
Ming, do you have access to
L80s_v2in Azure? This test needs to run on that VM size.Here is the command to benchmark it:
fio --bs=4k --ioengine=libaio --iodepth=128 --filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1:/dev/nvme4n1:/dev/nvme5n1:/dev/nvme6n1:/dev/nvme7n1:/dev/nvme8n1:/dev/nvme9n1 --direct=1 --runtime=120 --numjobs=80 --rw=randread --name=test --group_reporting --gtod_reduce=1
I can reproduce the issue on one machine(96 cores) with 4 NVMes(32 queues), so each queue is served on 3 CPUs.
IOPS drops > 20% when 'use_threaded_interrupts' is enabled. From fio log, CPU context switch is increased a lot.
Thanks,
Ming
Quoted 11 lines · Ming Lei
Quoted 4 lines · Long Li
Here is the command to benchmark it:
fio --bs=4k --ioengine=libaio --iodepth=128 --filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1:/dev/nvme4n1:/dev/nvme5n1:/dev/nvme6n1:/dev/nvme7n1:/dev/nvme8n1:/dev/nvme9n1 --direct=1 --runtime=120 --numjobs=80 --rw=randread --name=test --group_reporting --gtod_reduce=1
I can reproduce the issue on one machine(96 cores) with 4 NVMes(32 queues), so each queue is served on 3 CPUs.
IOPS drops > 20% when '
use_threaded_interrupts' is enabled. From fio log, CPU context switch is increased a lot.
Interestingly use_threaded_interrupts shows a marginal improvement on my machine with the same fio profile. It was only 5 NVMes, but they've one queue per-cpu on 112 cores.
Quoted 11 lines · Keith Busch
Quoted 9 lines · Ming Lei
Quoted 4 lines · Long Li
Here is the command to benchmark it:
fio --bs=4k --ioengine=libaio --iodepth=128 --filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1:/dev/nvme4n1:/dev/nvme5n1:/dev/nvme6n1:/dev/nvme7n1:/dev/nvme8n1:/dev/nvme9n1 --direct=1 --runtime=120 --numjobs=80 --rw=randread --name=test --group_reporting --gtod_reduce=1
I can reproduce the issue on one machine(96 cores) with 4 NVMes(32 queues), so each queue is served on 3 CPUs.
Quoted 6 lines
Quoted 2 lines
IOPS drops > 20% when '
use_threaded_interrupts' is enabled. From fio log, CPU context switch is increased a lot.Interestingly
use_threaded_interruptsshows a marginal improvement on my machine with the same fio profile. It was only 5 NVMes, but they've one queue per-cpu on 112 cores.
Not investigate it yet.
BTW, my fio test is only done on the single hw queue via 'taskset -c $cpu_list_of_the_queue',
without applying the threaded interrupt affinity patch. NVMe is Optane.
The same issue can be reproduced after I force to use 1:1 mapping via passing 'possible_cpus=32' kernel cmd line.
Maybe related with kernel options, so attache the one I used, and basically it is
a subset of RHEL8 kernel.
Thanks,
Ming Lei
Quoted 10 lines · Keith Busch
Quoted 9 lines · Ming Lei
Quoted 4 lines · Long Li
Here is the command to benchmark it:
fio --bs=4k --ioengine=libaio --iodepth=128 --filename=/dev/nvme0n1:/dev/nvme1n1:/dev/nvme2n1:/dev/nvme3n1:/dev/nvme4n1:/dev/nvme5n1:/dev/nvme6n1:/dev/nvme7n1:/dev/nvme8n1:/dev/nvme9n1 --direct=1 --runtime=120 --numjobs=80 --rw=randread --name=test --group_reporting --gtod_reduce=1
I can reproduce the issue on one machine(96 cores) with 4 NVMes(32 queues), so each queue is served on 3 CPUs.
Quoted 6 lines
Quoted 2 lines
IOPS drops > 20% when '
use_threaded_interrupts' is enabled. From fio log, CPU context switch is increased a lot.Interestingly
use_threaded_interruptsshows a marginal improvement on my machine with the same fio profile. It was only 5 NVMes, but they've one queue per-cpu on 112 cores.
Which is not surprising because the thread and the hard interrupt are on the same CPU and there is just that little overhead of the context switch.
The thing is that this really depends on how the scheduler decides to place the interrupt thread.
If you have a queue for several CPUs, then depending on the load situation allowing a multi-cpu affinity for the thread can cause lots of task migration.
But restricting the irq thread to the CPU on which the interrupt is affine can also starve that CPU. There is no universal rule for that.
Tracing should tell.
Thanks,
tglx