Documentation: Add managed interrupts
RFC PATCH 1/1 Documentation: Add managed interrupts
I stumbled upon "isolcpus=managed_irq" which is the last piece which can only be handled by isolcpus= and has no runtime knob. I knew roughly what managed interrupts should do but I lacked some details how it is used and what the managed_irq sub parameter means in practise.
This documents what we have as of today and how it works. I added some examples how the parameter affects the configuration. Did I miss something?
Given that the spreading as computed group_cpus_evenly() does not take the mask of isolated CPUs into account I'm not sure how relevant the managed_irq argument is. The virtio_scsi driver has no way to limit the interrupts and I don't see this for the nvme. Even if the number of queues can be reduced to two (as in the example) it is still spread evenly in the system instead and the isolated CPUs are not taken into account.
To make this worse, you can even argue further whether or not the application on the isolated CPU wants to receive the interrupt directly or would prefer not to.
Given all this, I am not sure if it makes sense to add 'io_queue' to the mix or if it could be incorporated into 'managed_irq'.
One more point: Given that isolcpus= is marked deprecated as of commit
b0d40d2b22fe4 ("sched/isolation: Document isolcpus= boot parameter flags, mark it deprecated")
and the 'managed_irq' is evaluated at device's probe time it would require additional callbacks to re-evaluate the situation. Probably for 'io_queue', too. Does is make sense or should we simply drop the "deprecation" notice and allowing using it long term? Dynamic partitions work with cpusets, there this (managed_irq) limitation but is it really? And if static partition is the use case why bother.
--- Documentation/core-api/irq/index.rst | 1 + Documentation/core-api/irq/managed_irq.rst | 116 +++++++++++++++++++++ 2 files changed, 117 insertions(+) create mode 100644 Documentation/core-api/irq/managed_irq.rst
Documentation/core-api/irq/index.rst +1 -0
@@ -9,3 +9,4 @@ IRQs irq-affinity irq-domain irqflags-tracing + managed_irq
Documentation/core-api/irq/managed_irq.rst new file +116 -0
116 lines not on this page · raw message
Signature · 1 line
2.53.0
Jonathan Corbet
maintainerQuoted 8 lines · Sebastian Andrzej Siewior
I stumbled upon "isolcpus=managed_irq" which is the last piece which can only be handled by isolcpus= and has no runtime knob. I knew roughly what managed interrupts should do but I lacked some details how it is used and what the
managed_irqsub parameter means in practise.This documents what we have as of today and how it works. I added some examples how the parameter affects the configuration. Did I miss something?
There's been a lot of silence on this one... should I pick this one up, or are there other plans for it...?
Thanks,
jon
Thomas Gleixner
Acked-byQuoted 6 lines · Jonathan Corbet
Quoted 3 lines
This documents what we have as of today and how it works. I added some examples how the parameter affects the configuration. Did I miss something?
There's been a lot of silence on this one... should I pick this one up, or are there other plans for it...?
Looks good to me.
Aaron Tomlin
Reviewed-byQuoted 11 lines · Jonathan Corbet
Quoted 8 lines · Sebastian Andrzej Siewior
I stumbled upon "isolcpus=managed_irq" which is the last piece which can only be handled by isolcpus= and has no runtime knob. I knew roughly what managed interrupts should do but I lacked some details how it is used and what the
managed_irqsub parameter means in practise.This documents what we have as of today and how it works. I added some examples how the parameter affects the configuration. Did I miss something?
Quoted 6 lines
There's been a lot of silence on this one... should I pick this one up, or are there other plans for it...?
Thanks,
jon
Hi Jonathan,
Sorry.
Signature · 1 line
Aaron Tomlin
Quoted 109 lines · Sebastian Andrzej Siewior
I stumbled upon "isolcpus=managed_irq" which is the last piece which can only be handled by isolcpus= and has no runtime knob. I knew roughly what managed interrupts should do but I lacked some details how it is used and what the
managed_irqsub parameter means in practise.This documents what we have as of today and how it works. I added some examples how the parameter affects the configuration. Did I miss something?
Given that the spreading as computed
group_cpus_evenly()does not take the mask of isolated CPUs into account I'm not sure how relevant themanaged_irqargument is. Thevirtio_scsidriver has no way to limit the interrupts and I don't see this for the nvme. Even if the number of queues can be reduced to two (as in the example) it is still spread evenly in the system instead and the isolated CPUs are not taken into account.
To make this worse, you can even argue further whether or not the application on the isolated CPU wants to receive the interrupt directly or would prefer not to.Given all this, I am not sure if it makes sense to add '
io_queue' to the mix or if it could be incorporated into 'managed_irq'.One more point: Given that isolcpus= is marked deprecated as of commit
b0d40d2b22fe4("sched/isolation: Document isolcpus= boot parameter flags, mark it deprecated")and the '
managed_irq' is evaluated at device's probe time it would require additional callbacks to re-evaluate the situation. Probably for 'io_queue', too. Does is make sense or should we simply drop the "deprecation" notice and allowing using it long term? Dynamic partitions work with cpusets, there this (managed_irq) limitation but is it really? And if static partition is the use case why bother.Signed-off-by Sebastian Andrzej Siewior <[email protected]>--- Documentation/core-api/irq/index.rst | 1 + Documentation/core-api/irq/managed_irq.rst | 116 +++++++++++++++++++++ 2 files changed, 117 insertions(+) create mode 100644 Documentation/core-api/irq/managed_irq.rstDocumentation/core-api/irq/index.rst +1
-0@@ -9,3 +9,4 @@ IRQs irq-affinity irq-domain irqflags-tracing + managed_irqDocumentation/core-api/irq/managed_irq.rst new file +52
-0@@ -0,0 +1,116 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=========================== +Affinity managed interrupts +=========================== + +The IRQ core provides support for managing interrupts according to a specified +CPU affinity. Under normal operation, an interrupt is associated with a +particular CPU. If that CPU is taken offline, the interrupt is migrated to +another online CPU. + +Devices with large numbers of interrupt vectors can stress the available vector +space. For example, an NVMe device with 128 I/O queues typically requests one +interrupt per queue on systems with at least 128 CPUs. Two such devices +therefore request 256 interrupts. On x86, the interrupt vector space is +notoriously low, providing only 256 vectors per CPU, and the kernel reserves a +subset of these, further reducing the number available for device interrupts. +In practice this is not an issue because the interrupts are distributed across +many CPUs, so each CPU only receives a small number of vectors. + +During system suspend, however, all secondary CPUs are taken offline and all +interrupts are migrated to the single CPU that remains online. This can exhaust +the available interrupt vectors on that CPU and cause the suspend operation to +fail. + +Affinity‑managed interrupts address this limitation. Each interrupt is assigned +a CPU affinity mask that specifies the set of CPUs on which the interrupt may +be targeted. When a CPU in the mask goes offline, the interrupt is moved to the +next CPU in the mask. If the last CPU in the mask goes offline, the interrupt +is shut down. Drivers using affinity‑managed interrupts must ensure that the +associated queue is quiesced before the interrupt is disabled so that no +further interrupts are generated. When a CPU in the affinity mask comes back +online, the interrupt is re‑enabled. + +Implementation +-------------- + +Devices must provide per‑instance interrupts, such as per‑I/O‑queue interrupts +for storage devices like NVMe. The driver allocates interrupt vectors with the +required affinity settings using struct irq_affinity. For MSI‑X devices, this +is done via pci_alloc_irq_vectors_affinity() with the PCI_IRQ_AFFINITY flag +set. + +Based on the provided affinity information, the IRQ core attempts to spread the +interrupts evenly across the system. The affinity masks are computed during +this allocation step, but the final IRQ assignment is performed when +request_irq() is invoked. + +Isolated CPUs +------------- + +The affinity of managed interrupts is handled entirely in the kernel and cannot
Documentation/core-api/irq/managed_irq.rst line 62
Quoted 10 lines
Documentation/core-api/irq/managed_irq.rst new file +10
-0+be modified from user space through the /proc interfaces. The managed_irq +sub‑parameter of the isolcpus boot option specifies a CPU mask that managed +interrupts should attempt to avoid. This isolation is best‑effort and only +applies if the automatically assigned interrupt mask also contains online CPUs +outside the avoided mask. If the requested mask contains only isolated CPUs, +the setting has no effect. + +CPUs listed in the avoided mask remain part of the interrupt’s affinity mask. +This means that if all non‑isolated CPUs go offline while isolated CPUs remain +online, the interrupt will be assigned to one of the isolated CPUs.
Maybe you can add:
In reality it is fine because IO isn't supposed to submit from isolated CPUs.
Quoted 44 lines
Documentation/core-api/irq/managed_irq.rst new file +44
-0+ +The following examples assume a system with 8 CPUs. + +- A QEMU instance is booted with "-device virtio-scsi-pci". + The MSI‑X device exposes 11 interrupts: 3 "management" interrupts and 8 + "queue" interrupts. The driver requests the 8 queue interrupts, each of which + is affine to exactly one CPU. If that CPU goes offline, the interrupt is shut + down. + + Assuming interrupt 48 is one of the queue interrupts, the following appears:: + + /proc/irq/48/effective_affinity_list:7 + /proc/irq/48/smp_affinity_list:7 + + This indicates that the interrupt is served only by CPU7. Shutting down CPU7 + does not migrate the interrupt to another CPU:: + + /proc/irq/48/effective_affinity_list:0 + /proc/irq/48/smp_affinity_list:7 + + This can be verified via the debugfs interface + (/sys/kernel/debug/irq/irqs/48). The dstate field will include + IRQD_IRQ_DISABLED, IRQD_IRQ_MASKED and IRQD_MANAGED_SHUTDOWN. + +- A QEMU instance is booted with "-device virtio-scsi-pci,num_queues=2" + and the kernel command line includes: + "irqaffinity=0,1 isolcpus=domain,2-7 isolcpus=managed_irq,1-3,5-7". + The MSI‑X device exposes 5 interrupts: 3 management interrupts and 2 queue + interrupts. The management interrupts follow the irqaffinity= setting. The + queue interrupts are spread across available CPUs:: + + /proc/irq/47/effective_affinity_list:0 + /proc/irq/47/smp_affinity_list:0-3 + /proc/irq/48/effective_affinity_list:4 + /proc/irq/48/smp_affinity_list:4-7 + + The two queue interrupts are evenly distributed. Interrupt 48 is placed on CPU4 + because the managed_irq mask avoids CPUs 5–7 when possible. + + Replacing the managed_irq argument with "isolcpus=managed_irq,1-3,4-5,7" + results in:: + + /proc/irq/48/effective_affinity_list:6 + /proc/irq/48/smp_affinity_list:4-7
Documentation/core-api/irq/managed_irq.rst line 116
Quoted 10 lines
Documentation/core-api/irq/managed_irq.rst new file +10
-0+ + Interrupt 48 is now served on CPU6 because the system avoids CPUs 4, 5 and + 7. If CPU6 is taken offline, the interrupt migrates to one of the "isolated" + CPUs:: + + /proc/irq/48/effective_affinity_list:7 + /proc/irq/48/smp_affinity_list:4-7 + + The interrupt is shut down once all CPUs listed in its smp_affinity mask are + offline.
Nice document, with or without the above change:
Thanks,
Jonathan Corbet
maintainerQuoted 11 lines · Thomas Gleixner
Quoted 6 lines · Jonathan Corbet
Quoted 3 lines
This documents what we have as of today and how it works. I added some examples how the parameter affects the configuration. Did I miss something?
There's been a lot of silence on this one... should I pick this one up, or are there other plans for it...?
Looks good to me.
Acked-by Thomas Gleixner <[email protected]>
OK, I've applied it, thanks.
jon
Quoted 7 lines · Sebastian Andrzej Siewior
One more point: Given that isolcpus= is marked deprecated as of commit
b0d40d2b22fe4("sched/isolation: Document isolcpus= boot parameter flags, mark it deprecated")and the '
managed_irq' is evaluated at device's probe time it would require additional callbacks to re-evaluate the situation. Probably for 'io_queue', too. Does is make sense or should we simply drop the "deprecation" notice and allowing using it long term?
AIUI the deprecation notice is more for isolcpus=domain, i.e. the scheduler part, but it's still relevant for e.g. managed_irq. FWIW Openshift uses:
isolcpus=managed_irq,<cpulist>
nohz_full=<cpulist>
and cpusets for dynamically isolating CPUs from the scheduler.
Quoted 11 lines · Valentin Schneider
Quoted 7 lines · Sebastian Andrzej Siewior
One more point: Given that isolcpus= is marked deprecated as of commit
b0d40d2b22fe4("sched/isolation: Document isolcpus= boot parameter flags, mark it deprecated")and the '
managed_irq' is evaluated at device's probe time it would require additional callbacks to re-evaluate the situation. Probably for 'io_queue', too. Does is make sense or should we simply drop the "deprecation" notice and allowing using it long term?AIUI the deprecation notice is more for isolcpus=domain, i.e. the scheduler part, but it's still relevant for e.g.
managed_irq.
If that is the case, then the deprecation notice should explicitly target the "domain" flag. Also note that "domain" is the default if no flag is specified, so it is a bit messy. It is odd to deprecate a (default) component of a feature and not have a plan how that component will ever be removed.
The documentation of "domain" already strongly advises to use cpusets instead. Is that not enough? If so, the deprecation should be dropped.
If there is a strong wish to remove the "domain" boot functionality, then there should be a new boot arg that can be used for "managed_irq" and "nohz" features, i.e. get users off the deprecated isolcpus so that isolcpus can log deprecation notices and be removed someday.
John Ogness
Quoted 7 lines · Ming Lei
Documentation/core-api/irq/managed_irq.rst line 62
Quoted 3 lines
Documentation/core-api/irq/managed_irq.rst new file +3
-0+CPUs listed in the avoided mask remain part of the interrupt’s affinity mask. +This means that if all non‑isolated CPUs go offline while isolated CPUs remain +online, the interrupt will be assigned to one of the isolated CPUs.Maybe you can add:
In reality it is fine because IO isn't supposed to submit from isolated CPUs.
You can argue both way. And I have some vague memory that block will schedule kworker and there was some work to use unbound worker instead of _this_ CPU. I just don't know what happens with interrupt and this is probably the one thing you can't configure.
Sebastian
Quoted 9 lines · Valentin Schneider
Quoted 7 lines · Sebastian Andrzej Siewior
One more point: Given that isolcpus= is marked deprecated as of commit
b0d40d2b22fe4("sched/isolation: Document isolcpus= boot parameter flags, mark it deprecated")and the '
managed_irq' is evaluated at device's probe time it would require additional callbacks to re-evaluate the situation. Probably for 'io_queue', too. Does is make sense or should we simply drop the "deprecation" notice and allowing using it long term?
Quoted 7 lines
AIUI the deprecation notice is more for isolcpus=domain, i.e. the scheduler part, but it's still relevant for e.g.
managed_irq. FWIW Openshift uses:isolcpus=managed_irq,<cpulist>
nohz_full=<cpulist>and cpusets for dynamically isolating CPUs from the scheduler.
For the managed_irq you could argue that this could also use some runtime configuration at which point isolcpus= would have a runtime counterpart and could be removed.
After going through all this I concluded that it makes hardly sense since you would require callbacks in every driver using it or other magic "to reconfigure" but it already makes little sense using it.
Either way, I don't see anything wrong with using isolcpus=domain if you have a static setup and need/ want reconfigure at runtime.
Sebastian
Quoted 2 lines · John Ogness
If that is the case, then the deprecation notice should explicitly target the "domain" flag. Also note that "domain" is the default if no
I am going to submit a patch removing that deprecation note. Frederic was pro and I did not collect a single argument why removing it would be a good thing.
Quoted 1 line
John Ogness
Sebastian
Quoted 10 lines · Sebastian Andrzej Siewior
For the
managed_irqyou could argue that this could also use some runtime configuration at which point isolcpus= would have a runtime counterpart and could be removed.
After going through all this I concluded that it makes hardly sense since you would require callbacks in every driver using it or other magic "to reconfigure" but it already makes little sense using it.Either way, I don't see anything wrong with using isolcpus=domain if you have a static setup and need/ want reconfigure at runtime.
Hi Sebastian,
I completely agree.
Signature · 1 line
Aaron Tomlin
Quoted 8 lines · Sebastian Andrzej Siewior
Quoted 6 lines · Ming Lei
Documentation/core-api/irq/managed_irq.rst line 62
Quoted 3 lines
Documentation/core-api/irq/managed_irq.rst new file +3
-0+CPUs listed in the avoided mask remain part of the interrupt’s affinity mask. +This means that if all non‑isolated CPUs go offline while isolated CPUs remain +online, the interrupt will be assigned to one of the isolated CPUs.Maybe you can add:
Quoted 5 lines
Quoted 1 line
In reality it is fine because IO isn't supposed to submit from isolated CPUs.
You can argue both way. And I have some vague memory that block will schedule kworker and there was some work to use unbound worker instead of _this_ CPU.
It is actually to rule out isolated cpus from hctx->cpu_mask, please see blk_mq_map_swqueue().
Also unbound worker won't be scheduled on isolated CPU too, if I remember correctly.
Quoted 2 lines
I just don't know what happens with interrupt and this is probably the one thing you can't configure.
Interrupt control is determined by irq.effective_affinity, as shown in your documentation.
Or you may provide more detail about the question.
Thanks,
Ming