isolation: limit msix vectors to housekeeping CPUs
Subsystems (5)
v4 cover isolation: limit msix vectors to housekeeping CPUs
This is a follow-up posting for "[PATCH v3 0/4] isolation: limit msix vectors to housekeeping CPUs".
Issue
=====
With the current implementation device drivers while creating their MSIX ? ? ? ? vectors only take num_online_cpus() into consideration which works quite well ? for a non-RT environment, but in an RT environment that has a large number of ? isolated CPUs and very few housekeeping CPUs this could lead to a problem. ? ? The problem will be triggered when something like tuned will try to move all ? ? the IRQs from isolated CPUs to the limited number of housekeeping CPUs to ? ? ? prevent interruptions for a latency-sensitive workload that will be running on the isolated CPUs. This failure is caused because of the per CPU vector ? ? ? ? limitation. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
Proposed Fix
============
In this patch-set, the following changes are proposed:
- A generic API housekeeping_num_online_cpus() which is meant to return the online housekeeping CPUs based on the hk_flag passed by the caller.
- i40e: Specifically for the i40e driver the num_online_cpus() used in ? i40e_init_msix() to calculate numbers msix vectors is replaced with the above defined API that returns the online housekeeping CPUs that are meant to handle managed IRQ jobs.
- pci_alloc_irq_vector(): With the help of housekeeping_num_online_cpus() the max_vecs passed in pci_alloc_irq_vector() is restricted only to the online
? housekeeping CPUs (designated for managed IRQ jobs) strictly in an RT
environment. However, if the min_vecs exceeds the online housekeeping CPUs, max_vecs is limited based on the min_vecs instead.
Future Work
===========
- In the previous upstream discussion [1], it was decided that it would be better if we can have a generic framework that can be consumed by all the drivers to fix this kind of issue. However, it will be a long term work, and since there are RT workloads that are getting impacted by the reported issue. We agreed upon the proposed per-device approach for now.
Testing
=======
Functionality:
- To test that the issue is resolved with i40e change I added a tracepoint ? in i40e_init_msix() to find the number of CPUs derived for vector creation ? with and without tuned's realtime-virtual-host profile. As per expectation ? with the profile applied I was only getting the number of housekeeping CPUs ? and all available CPUs without it. Another way to verify is by checking the number of IRQs that get created corresponding to a impacted device. Similarly did a few more tests with different modes eg with only nohz_full, isolcpus etc.
Performance:
- To analyze the performance impact I have targetted the change introduced in ? pci_alloc_irq_vectors() and compared the results against a vanilla kernel ? (5.9.0-rc3) results.
? Setup Information:
? + I had a couple of 24-core machines connected back to back via a couple of ? ? mlx5 NICs and I analyzed the average bitrate for server-client TCP and
UDP transmission via iperf.
? + To minimize the Bitrate variation of iperf TCP and UDP stream test I have ? ? applied the tuned's network-throughput profile and disabled HT. ?Test Information:
? + For the environment that had no isolated CPUs:
? ? I have tested with single stream and 24 streams (same as that of online ? ? CPUs).
? + For the environment that had 20 isolated CPUs:
? ? I have tested with single stream, 4 streams (same as that the number of ? ? housekeeping) and 24 streams (same as that of online CPUs).
?Results:
? # UDP Stream Test:
? + There was no degradation observed in UDP stream tests in both
environments. (With isolated CPUs and without isolated CPUs after the introduction of the patches).
? # TCP Stream Test - No isolated CPUs:
? + No noticeable degradation was observed.
? # TCP Stream Test - With isolated CPUs:
? + Multiple Stream (4) ?- Average degradation of around 5-6% ? + Multiple Stream (24) - Average degradation of around 2-3% ? + Single Stream ? ? ? ?- Even on a vanilla kernel the Bitrate observed
for a TCP single stream test seem to vary significantly across different runs (eg. the % variation between the best and the worst case on a vanilla kernel was around 8-10%). A similar variation was observed with the kernel that included my patches. No additional degradation was observed.
If there are any suggestions for more performance evaluation, I would be happy to discuss/perform them.
Changes from v3[2]:
==================
- Moved the logic to limit the max_vecs from pci_alloc_irq_vectors() to pci_alloc_irq_vectors_affinity() as that's the exported interface and drivers using this API also need to be fixed (suggestion from Bjorn Helgaas).
Changes from v2[3]:
==================
- Renamed hk_num_online_cpus() with housekeeping_num_online_cpus() to keep the naming convention consistent (based on a suggestion from Peter Zijlstra and Frederic Weisbecker).
- Added an argument "enum hk_flags" to the housekeeping_num_online_cpus() API to make it more usable in different use-cases (based on a suggestion from Frederic Weisbecker).
- Replaced cpumask_weight(cpu_online_mask) with num_online_cpus() (suggestion from Bjorn Helgaas).
- Modified patch commit messages and comment based on Bjorn Helgaas's suggestion.
Changes from v1[4]:
==================
Patch1: ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
- Replaced num_houskeeeping_cpus() with hk_num_online_cpus() and started
using the cpumask corresponding to HK_FLAG_MANAGED_IRQ to derive the number
of online housekeeping CPUs. This is based on Frederic Weisbecker's
suggestion. ? ? ? ? ?
- Since the hk_num_online_cpus() is self-explanatory, got rid of ? ? ? ? ? ?
? the comment that was added previously. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
Patch2: ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
- Added a new patch that is meant to enable managed IRQ isolation for
nohz_full CPUs. This is based on Frederic Weisbecker's suggestion.? ? ? ? ? ? ?
Patch4 (PCI): ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
- For cases where the min_vecs exceeds the online housekeeping CPUs, instead
of skipping modification to max_vecs, started restricting it based on the
min_vecs. This is based on a suggestion from Marcelo Tosatti.? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??
[1] https://lore.kernel.org/lkml/20200922095440.GA5217 at lenoir/
[2] https://lore.kernel.org/lkml/20200925182654.224004-1-nitesh at redhat.com/
[3] https://lore.kernel.org/lkml/20200923181126.223766-1-nitesh at redhat.com/
[4] https://lore.kernel.org/lkml/20200909150818.313699-1-nitesh at redhat.com/
Nitesh Narayan Lal (4):
sched/isolation: API to get number of housekeeping CPUs sched/isolation: Extend nohz_full to isolate managed IRQs
i40e: Limit msix vectors to housekeeping CPUs
PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
drivers/net/ethernet/intel/i40e/i40e_main.c | 3 ++- drivers/pci/msi.c | 18 ++++++++++++++++++ include/linux/sched/isolation.h | 9 +++++++++ kernel/sched/isolation.c | 2 +- 4 files changed, 30 insertions(+), 2 deletions(-)
PATCH v4 1/4 sched/isolation: API to get number of housekeeping CPUs
Introduce a new API housekeeping_num_online_cpus(), that can be used to retrieve the number of online housekeeping CPUs based on the housekeeping flag passed by the caller.
Some of the consumers for this API are the device drivers that were previously relying only on num_online_cpus() to determine the number of MSIX vectors to create. In real-time environments to minimize interruptions to isolated CPUs, all device-specific IRQ vectors are often moved to the housekeeping CPUs, having excess vectors could cause housekeeping CPU to run out of IRQ vectors.
--- include/linux/sched/isolation.h | 9 +++++++++ 1 file changed, 9 insertions(+)
include/linux/sched/isolation.h +9 -0
@@ -57,4 +57,13 @@ static inline bool housekeeping_cpu(int cpu, enum hk_flags flags) return true; } +static inline unsigned int housekeeping_num_online_cpus (enum hk_flags flags) +{ +#ifdef CONFIG_CPU_ISOLATION +if (static_branch_unlikely (&housekeeping_overridden)) +return cpumask_weight (housekeeping_cpumask (flags)); +#endif +return num_online_cpus (); +} +#endif /* _LINUX_SCHED_ISOLATION_H */
Signature · 1 line
2.18.2
PATCH v4 2/4 sched/isolation: Extend nohz_full to isolate managed IRQs
In reply to Nitesh Narayan LalExtend nohz_full feature set to include isolation from managed IRQS. This is required specifically for setups that only uses nohz_full and still requires isolation for maintaining lower latency for the listed CPUs.
--- kernel/sched/isolation.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/isolation.c +1 -1
@@ -141,7 +141,7 @@ static int __init housekeeping_nohz_full_setup(char *str)unsigned int flags; flags =HK_FLAG_TICK |HK_FLAG_WQ |HK_FLAG_TIMER |HK_FLAG_RCU | -HK_FLAG_MISC |HK_FLAG_KTHREAD ; +HK_FLAG_MISC |HK_FLAG_KTHREAD |HK_FLAG_MANAGED_IRQ ;return housekeeping_setup (str, flags); }
Signature · 1 line
2.18.2
Peter Zijlstra
maintainerRe: [PATCH v4 2/4] sched/isolation: Extend nohz_full to isolate managed IRQs
Quoted 1 line · Nitesh Narayan Lal
Extend
nohz_fullfeature set to include isolation from managed IRQS. This
So you say it's for managed-irqs, the feature is actually called MANAGED_IRQ, but, AFAICT, it does *NOT* in fact affect managed IRQs.
Also, as per Thomas' earlier points, managed-irqs are in fact perfectly fine and don't need help at at...
Quoted 25 lines
is required specifically for setups that only uses
nohz_fulland still requires isolation for maintaining lower latency for the listed CPUs.Suggested-by Frederic Weisbecker <[email protected]>Signed-off-by Nitesh Narayan Lal <[email protected]>--- kernel/sched/isolation.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/sched/isolation.c +1
-1@@ -141,7 +141,7 @@ static int __init housekeeping_nohz_full_setup(char *str)unsigned int flags; flags =HK_FLAG_TICK |HK_FLAG_WQ |HK_FLAG_TIMER |HK_FLAG_RCU | -HK_FLAG_MISC |HK_FLAG_KTHREAD ; +HK_FLAG_MISC |HK_FLAG_KTHREAD |HK_FLAG_MANAGED_IRQ ;return housekeeping_setup (str, flags); }Signature · 1 line
2.18.2
Re: [PATCH v4 2/4] sched/isolation: Extend nohz_full to isolate managed IRQs
Quoted 13 lines · Peter Zijlstra
Quoted 1 line · Nitesh Narayan Lal
Extend
nohz_fullfeature set to include isolation from managed IRQS. ThisSo you say it's for managed-irqs, the feature is actually called
MANAGED_IRQ, but, AFAICT, it does *NOT* in fact affect managed IRQs.Also, as per Thomas' earlier points, managed-irqs are in fact perfectly fine and don't need help at at...
Quoted 4 lines
is required specifically for setups that only uses
nohz_fulland still requires isolation for maintaining lower latency for the listed CPUs.Suggested-by Frederic Weisbecker <[email protected]>
Ah and yes there is this tag :-p
So that's my bad, I really thought this thing was about managed IRQ. The problem is that I can't find a single documentation about them so I'm too clueless on that matter.
Thanks.
Quoted 21 lines
Quoted 21 lines
Signed-off-by Nitesh Narayan Lal <[email protected]>--- kernel/sched/isolation.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/sched/isolation.c +1
-1@@ -141,7 +141,7 @@ static int __init housekeeping_nohz_full_setup(char *str)unsigned int flags; flags =HK_FLAG_TICK |HK_FLAG_WQ |HK_FLAG_TIMER |HK_FLAG_RCU | -HK_FLAG_MISC |HK_FLAG_KTHREAD ; +HK_FLAG_MISC |HK_FLAG_KTHREAD |HK_FLAG_MANAGED_IRQ ;return housekeeping_setup (str, flags); }Signature · 1 line
2.18.2
Re: [PATCH v4 2/4] sched/isolation: Extend nohz_full to isolate managed IRQs
Quoted 12 lines · Frederic Weisbecker
Quoted 11 lines · Peter Zijlstra
Quoted 1 line · Nitesh Narayan Lal
Extend
nohz_fullfeature set to include isolation from managed IRQS. ThisSo you say it's for managed-irqs, the feature is actually called
MANAGED_IRQ, but, AFAICT, it does *NOT* in fact affect managed IRQs.Also, as per Thomas' earlier points, managed-irqs are in fact perfectly fine and don't need help at at...
Quoted 3 lines
is required specifically for setups that only uses
nohz_fulland still requires isolation for maintaining lower latency for the listed CPUs.
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Suggested-by Frederic Weisbecker <[email protected]>Ah and yes there is this tag :-p
So that's my bad, I really thought this thing was about managed IRQ. The problem is that I can't find a single documentation about them so I'm too clueless on that matter.
I am also confused with this terminology.
So my bad for not taking care of this.
Signature · 10 lines
Thanks Nitesh -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20201023/4cfec365/attachment.asc>
Re: [PATCH v4 2/4] sched/isolation: Extend nohz_full to isolate managed IRQs
In reply to Peter ZijlstraQuoted 4 lines · Peter Zijlstra
Quoted 1 line · Nitesh Narayan Lal
Extend
nohz_fullfeature set to include isolation from managed IRQS. ThisSo you say it's for managed-irqs, the feature is actually called
MANAGED_IRQ, but, AFAICT, it does *NOT* in fact affect managed IRQs.
Ah my bad! I should replace the managed IRQS with MANAGED_IRQ. I can send another version with this fixed.
Quoted 3 lines
Also, as per Thomas' earlier points, managed-irqs are in fact perfectly fine and don't need help at at...
Since the introduction of
"genirq, sched/isolation: Isolate from handling managed interrupts"
Within irq_do_set_affinity(), it is ensured that for managed intrrupts as well, the isolated CPUs are removed from the affinity mask.
Hence, IMHO before this change managed interrupts were affecting the isolated CPUs.
My intent of having this change is to basically allow isolation for nohz_full CPUs even when we don't have something like isolcpus. Does that make sense?
Quoted 26 lines
Quoted 26 lines
is required specifically for setups that only uses
nohz_fulland still requires isolation for maintaining lower latency for the listed CPUs.Suggested-by Frederic Weisbecker <[email protected]>Signed-off-by Nitesh Narayan Lal <[email protected]>--- kernel/sched/isolation.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/sched/isolation.c +1
-1@@ -141,7 +141,7 @@ static int __init housekeeping_nohz_full_setup(char *str)unsigned int flags; flags =HK_FLAG_TICK |HK_FLAG_WQ |HK_FLAG_TIMER |HK_FLAG_RCU | -HK_FLAG_MISC |HK_FLAG_KTHREAD ; +HK_FLAG_MISC |HK_FLAG_KTHREAD |HK_FLAG_MANAGED_IRQ ;return housekeeping_setup (str, flags); }Signature · 1 line
2.18.2
Signature · 9 lines
Nitesh -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20201023/0559be1a/attachment-0001.asc>
PATCH v4 3/4 i40e: Limit msix vectors to housekeeping CPUs
In reply to Nitesh Narayan LalIf we have isolated CPUs designated to perform real-time tasks, to keep the latency overhead to a minimum for real-time CPUs IRQ vectors are moved to housekeeping CPUs from the userspace. Creating MSIX vectors only based on the online CPUs could lead to exhaustion of housekeeping CPU IRQ vectors in such environments.
This patch prevents i40e to create vectors only based on online CPUs by retrieving the online housekeeping CPUs that are designated to perform managed IRQ jobs.
--- drivers/net/ethernet/intel/i40e/i40e_main.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/net/ethernet/intel/i40e/i40e_main.c +2 -1
@@ -5,6 +5,7 @@#include <linux/of_net.h> #include <linux/pci.h> #include <linux/bpf.h> +#include <linux/sched/isolation.h> #include <generated/utsrelease.h> /* Local includes */
@@ -11002,7 +11003,7 @@ static int i40e_init_msix(struct i40e_pf *pf)* will use any remaining vectors to reach as close as we can to the * number of online CPUs. */ - cpus =num_online_cpus (); + cpus =housekeeping_num_online_cpus (HK_FLAG_MANAGED_IRQ ); pf->num_lan_msix =min_t (int, cpus, vectors_left /2 ); vectors_left -= pf->num_lan_msix;
Signature · 1 line
2.18.2
PATCH v4 4/4 PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
In reply to Nitesh Narayan LalIf we have isolated CPUs dedicated for use by real-time tasks, we try to move IRQs to housekeeping CPUs from the userspace to reduce latency overhead on the isolated CPUs.
If we allocate too many IRQ vectors, moving them all to housekeeping CPUs may exceed per-CPU vector limits.
When we have isolated CPUs, limit the number of vectors allocated by pci_alloc_irq_vectors() to the minimum number required by the driver, or to one per housekeeping CPU if that is larger.
--- drivers/pci/msi.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
drivers/pci/msi.c +18 -0
@@ -23,6 +23,7 @@#include <linux/slab.h> #include <linux/irqdomain.h> #include <linux/of_irq.h> +#include <linux/sched/isolation.h> #include "pci.h"
@@ -1191,8 +1192,25 @@ int pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs, struct irq_affinity *affd) {struct irq_affinity msi_default_affd = {0 }; +unsigned int hk_cpus;int nvecs = -ENOSPC ; + hk_cpus =housekeeping_num_online_cpus (HK_FLAG_MANAGED_IRQ ); + +/* +* If we have isolated CPUs for use by real-time tasks, to keep the +* latency overhead to a minimum, device-specific IRQ vectors are moved +* to the housekeeping CPUs from the userspace by changing their +* affinity mask. Limit the vector usage to keep housekeeping CPUs from +* running out of IRQ vectors. +*/ +if (hk_cpus <num_online_cpus ()) { +if (hk_cpus < min_vecs) + max_vecs = min_vecs; +else if (hk_cpus < max_vecs) + max_vecs = hk_cpus; + } +if (flags &PCI_IRQ_AFFINITY ) {if (!affd) affd = &msi_default_affd;
Signature · 1 line
2.18.2
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
[to: Christoph in case he has comments, since I think he wrote this code]
Quoted 4 lines · Nitesh Narayan Lal
If we have isolated CPUs dedicated for use by real-time tasks, we try to move IRQs to housekeeping CPUs from the userspace to reduce latency overhead on the isolated CPUs.
Quoted 8 lines
If we allocate too many IRQ vectors, moving them all to housekeeping CPUs may exceed per-CPU vector limits.
When we have isolated CPUs, limit the number of vectors allocated by
pci_alloc_irq_vectors()to the minimum number required by the driver, or to one per housekeeping CPU if that is larger.Signed-off-by Nitesh Narayan Lal <[email protected]>
Quoted 45 lines
--- drivers/pci/msi.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)drivers/pci/msi.c +18
-0@@ -23,6 +23,7 @@#include <linux/slab.h> #include <linux/irqdomain.h> #include <linux/of_irq.h> +#include <linux/sched/isolation.h> #include "pci.h" @@ -1191,8 +1192,25 @@ int pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs, struct irq_affinity *affd) {struct irq_affinity msi_default_affd = {0 }; +unsigned int hk_cpus;int nvecs = -ENOSPC ; + hk_cpus =housekeeping_num_online_cpus (HK_FLAG_MANAGED_IRQ ); + +/* +* If we have isolated CPUs for use by real-time tasks, to keep the +* latency overhead to a minimum, device-specific IRQ vectors are moved +* to the housekeeping CPUs from the userspace by changing their +* affinity mask. Limit the vector usage to keep housekeeping CPUs from +* running out of IRQ vectors. +*/ +if (hk_cpus <num_online_cpus ()) { +if (hk_cpus < min_vecs) + max_vecs = min_vecs; +else if (hk_cpus < max_vecs) + max_vecs = hk_cpus; + } +if (flags &PCI_IRQ_AFFINITY ) {if (!affd) affd = &msi_default_affd;Signature · 1 line
2.18.2
Christoph Hellwig
Acked-byRe: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 1 line · Bjorn Helgaas
[to: Christoph in case he has comments, since I think he wrote this code]
I think I actually suggested this a few iterations back.
Quoted 10 lines
drivers/pci/msi.c int pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs, · line 1207
Quoted 10 lines
drivers/pci/msi.c +10
-0+ hk_cpus =housekeeping_num_online_cpus (HK_FLAG_MANAGED_IRQ ); + +/* +* If we have isolated CPUs for use by real-time tasks, to keep the +* latency overhead to a minimum, device-specific IRQ vectors are moved +* to the housekeeping CPUs from the userspace by changing their +* affinity mask. Limit the vector usage to keep housekeeping CPUs from +* running out of IRQ vectors. +*/ +if (hk_cpus <num_online_cpus ()) {
I woukd have moved the assignment to hk_cpus below the comment and just above the if, but that is really just a minor style preference.
Otherwise this looks good:
Peter Zijlstra
maintainerRe: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
In reply to Nitesh Narayan LalQuoted 39 lines · Nitesh Narayan Lal
If we have isolated CPUs dedicated for use by real-time tasks, we try to move IRQs to housekeeping CPUs from the userspace to reduce latency overhead on the isolated CPUs.
If we allocate too many IRQ vectors, moving them all to housekeeping CPUs may exceed per-CPU vector limits.
When we have isolated CPUs, limit the number of vectors allocated by
pci_alloc_irq_vectors()to the minimum number required by the driver, or to one per housekeeping CPU if that is larger.Signed-off-by Nitesh Narayan Lal <[email protected]>--- drivers/pci/msi.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)drivers/pci/msi.c +6
-0@@ -23,6 +23,7 @@#include <linux/slab.h> #include <linux/irqdomain.h> #include <linux/of_irq.h> +#include <linux/sched/isolation.h> #include "pci.h" @@ -1191,8 +1192,25 @@ int pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs, struct irq_affinity *affd) {struct irq_affinity msi_default_affd = {0 }; +unsigned int hk_cpus;int nvecs = -ENOSPC ; + hk_cpus =housekeeping_num_online_cpus (HK_FLAG_MANAGED_IRQ ); + +/* +* If we have isolated CPUs for use by real-time tasks, to keep the
drivers/pci/msi.c int pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs, · line 1211
Quoted 10 lines
drivers/pci/msi.c +10
-0+* latency overhead to a minimum, device-specific IRQ vectors are moved +* to the housekeeping CPUs from the userspace by changing their +* affinity mask. Limit the vector usage to keep housekeeping CPUs from +* running out of IRQ vectors. +*/ +if (hk_cpus <num_online_cpus ()) { +if (hk_cpus < min_vecs) + max_vecs = min_vecs; +else if (hk_cpus < max_vecs) + max_vecs = hk_cpus;
is that:
max_vecs = clamp(hk_cpus, min_vecs, max_vecs);
Also, do we really need to have that conditional on hk_cpus < num_online_cpus()? That is, why can't we do this unconditionally?
And what are the (desired) semantics vs hotplug? Using a cpumask without excluding hotplug is racy.
Quoted 8 lines
drivers/pci/msi.c +2
-0+ } +if (flags &PCI_IRQ_AFFINITY ) {if (!affd) affd = &msi_default_affd;Signature · 1 line
2.18.2
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 52 lines · Peter Zijlstra
Quoted 39 lines · Nitesh Narayan Lal
If we have isolated CPUs dedicated for use by real-time tasks, we try to move IRQs to housekeeping CPUs from the userspace to reduce latency overhead on the isolated CPUs.
If we allocate too many IRQ vectors, moving them all to housekeeping CPUs may exceed per-CPU vector limits.
When we have isolated CPUs, limit the number of vectors allocated by
pci_alloc_irq_vectors()to the minimum number required by the driver, or to one per housekeeping CPU if that is larger.Signed-off-by Nitesh Narayan Lal <[email protected]>--- drivers/pci/msi.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)drivers/pci/msi.c +6
-0@@ -23,6 +23,7 @@#include <linux/slab.h> #include <linux/irqdomain.h> #include <linux/of_irq.h> +#include <linux/sched/isolation.h> #include "pci.h" @@ -1191,8 +1192,25 @@ int pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs, struct irq_affinity *affd) {struct irq_affinity msi_default_affd = {0 }; +unsigned int hk_cpus;int nvecs = -ENOSPC ; + hk_cpus =housekeeping_num_online_cpus (HK_FLAG_MANAGED_IRQ ); + +/* +* If we have isolated CPUs for use by real-time tasks, to keep the drivers/pci/msi.c int pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs, · line 1211
Quoted 10 lines
drivers/pci/msi.c +10
-0+* latency overhead to a minimum, device-specific IRQ vectors are moved +* to the housekeeping CPUs from the userspace by changing their +* affinity mask. Limit the vector usage to keep housekeeping CPUs from +* running out of IRQ vectors. +*/ +if (hk_cpus <num_online_cpus ()) { +if (hk_cpus < min_vecs) + max_vecs = min_vecs; +else if (hk_cpus < max_vecs) + max_vecs = hk_cpus;is that:
Quoted 1 line
max_vecs = clamp(hk_cpus, min_vecs, max_vecs);
Yes, I think this will do.
Quoted 3 lines
Also, do we really need to have that conditional on
hk_cpus<num_online_cpus()? That is, why can't we do this unconditionally?
FWIU most of the drivers using this API already restricts the number of vectors based on the num_online_cpus, if we do it unconditionally we can unnecessary duplicate the restriction for cases where we don't have any isolated CPUs.
Also, different driver seems to take different factors into consideration along with num_online_cpus while finding the max_vecs to request, for example in the case of mlx5:MLX5_CAP_GEN(dev, num_ports) * num_online_cpus() + ?????????????? MLX5_EQ_VEC_COMP_BASE
Having hk_cpus < num_online_cpus() helps us ensure that we are only changing the behavior when we have isolated CPUs.
Does that make sense?
Quoted 3 lines
And what are the (desired) semantics vs hotplug? Using a cpumask without excluding hotplug is racy.
The housekeeping_mask should still remain constant, isn't? In any case, I can double check this.
Quoted 9 lines
Quoted 9 lines
drivers/pci/msi.c +2
-0+ } +if (flags &PCI_IRQ_AFFINITY ) {if (!affd) affd = &msi_default_affd;Signature · 1 line
2.18.2
Signature · 10 lines
Thanks Nitesh -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20201018/0e2907a5/attachment-0001.asc>
Peter Zijlstra
maintainerRe: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 21 lines · Nitesh Narayan Lal
Quoted 16 lines
Quoted 4 lines
drivers/pci/msi.c +4
-0+ hk_cpus =housekeeping_num_online_cpus (HK_FLAG_MANAGED_IRQ ); + +/* +* If we have isolated CPUs for use by real-time tasks, to keep the drivers/pci/msi.c int pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs, · line 1211
Quoted 10 lines
drivers/pci/msi.c +10
-0+* latency overhead to a minimum, device-specific IRQ vectors are moved +* to the housekeeping CPUs from the userspace by changing their +* affinity mask. Limit the vector usage to keep housekeeping CPUs from +* running out of IRQ vectors. +*/ +if (hk_cpus <num_online_cpus ()) { +if (hk_cpus < min_vecs) + max_vecs = min_vecs; +else if (hk_cpus < max_vecs) + max_vecs = hk_cpus;is that:
Quoted 1 line
max_vecs = clamp(hk_cpus, min_vecs, max_vecs);Yes, I think this will do.
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
Also, do we really need to have that conditional on
hk_cpus<num_online_cpus()? That is, why can't we do this unconditionally?FWIU most of the drivers using this API already restricts the number of vectors based on the
num_online_cpus, if we do it unconditionally we can unnecessary duplicate the restriction for cases where we don't have any isolated CPUs.
unnecessary isn't really a concern here, this is a slow path. What's important is code clarity.
Quoted 10 lines
Also, different driver seems to take different factors into consideration along with
num_online_cpuswhile finding themax_vecsto request, for example in the case of mlx5:MLX5_CAP_GEN(dev, num_ports)*num_online_cpus()+ ??????????????MLX5_EQ_VEC_COMP_BASEHaving
hk_cpus<num_online_cpus()helps us ensure that we are only changing the behavior when we have isolated CPUs.Does that make sense?
That seems to want to allocate N interrupts per cpu (plus some random static amount, which seems weird, but whatever). This patch breaks that.
So I think it is important to figure out what that driver really wants in the nohz_full case. If it wants to retain N interrupts per CPU, and only reduce the number of CPUs, the proposed interface is wrong.
Quoted 5 lines
Quoted 2 lines
And what are the (desired) semantics vs hotplug? Using a cpumask without excluding hotplug is racy.
The
housekeeping_maskshould still remain constant, isn't? In any case, I can double check this.
The goal is very much to have that dynamically configurable.
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 39 lines · Peter Zijlstra
Quoted 21 lines · Nitesh Narayan Lal
Quoted 16 lines
Quoted 4 lines
drivers/pci/msi.c +4
-0+ hk_cpus =housekeeping_num_online_cpus (HK_FLAG_MANAGED_IRQ ); + +/* +* If we have isolated CPUs for use by real-time tasks, to keep the drivers/pci/msi.c int pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs, · line 1211
Quoted 10 lines
drivers/pci/msi.c +10
-0+* latency overhead to a minimum, device-specific IRQ vectors are moved +* to the housekeeping CPUs from the userspace by changing their +* affinity mask. Limit the vector usage to keep housekeeping CPUs from +* running out of IRQ vectors. +*/ +if (hk_cpus <num_online_cpus ()) { +if (hk_cpus < min_vecs) + max_vecs = min_vecs; +else if (hk_cpus < max_vecs) + max_vecs = hk_cpus;is that:
Quoted 1 line
max_vecs = clamp(hk_cpus, min_vecs, max_vecs);Yes, I think this will do.
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
Also, do we really need to have that conditional on
hk_cpus<num_online_cpus()? That is, why can't we do this unconditionally?FWIU most of the drivers using this API already restricts the number of vectors based on the
num_online_cpus, if we do it unconditionally we can unnecessary duplicate the restriction for cases where we don't have any isolated CPUs.unnecessary isn't really a concern here, this is a slow path. What's important is code clarity.
Quoted 6 lines
Also, different driver seems to take different factors into consideration along with
num_online_cpuswhile finding themax_vecsto request, for example in the case of mlx5:MLX5_CAP_GEN(dev, num_ports)*num_online_cpus()+ ??????????????MLX5_EQ_VEC_COMP_BASE
Quoted 7 lines
Quoted 4 lines
Having
hk_cpus<num_online_cpus()helps us ensure that we are only changing the behavior when we have isolated CPUs.Does that make sense?
That seems to want to allocate N interrupts per cpu (plus some random static amount, which seems weird, but whatever). This patch breaks that.
On purpose. For the isolated CPUs we don't want network device interrupts (in this context).
Quoted 3 lines
So I think it is important to figure out what that driver really wants in the
nohz_fullcase. If it wants to retain N interrupts per CPU, and only reduce the number of CPUs, the proposed interface is wrong.
It wants N interrupts per non-isolated (AKA housekeeping) CPU. Zero interrupts for isolated interrupts.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
And what are the (desired) semantics vs hotplug? Using a cpumask without excluding hotplug is racy.
The
housekeeping_maskshould still remain constant, isn't? In any case, I can double check this.The goal is very much to have that dynamically configurable.
Yes, but this patch is a fix for customer bug in the old, static on-boot isolation CPU configuration.
---
Discussing the dynamic configuration (not this patch!) case:
Would need to enable/disable interrupts for a particular device on a per-CPU basis. Such interface does not exist yet.
Perhaps that is what you are looking for when writing "proposed interface is wrong" Peter?
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 2 lines · Marcelo Tosatti
Quoted 1 line · Peter Zijlstra
On Sun, Oct 18, 2020 at 02:14:46PM -0400, Nitesh Narayan Lal wrote:
[...]
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Also, do we really need to have that conditional on
hk_cpus<num_online_cpus()? That is, why can't we do this unconditionally?FWIU most of the drivers using this API already restricts the number of vectors based on the
num_online_cpus, if we do it unconditionally we can unnecessary duplicate the restriction for cases where we don't have any isolated CPUs.unnecessary isn't really a concern here, this is a slow path. What's important is code clarity.
Right, I can skip that check then.
Quoted 13 lines
Quoted 7 lines
Quoted 7 lines
Also, different driver seems to take different factors into consideration along with
num_online_cpuswhile finding themax_vecsto request, for example in the case of mlx5:MLX5_CAP_GEN(dev, num_ports)*num_online_cpus()+ ??????????????MLX5_EQ_VEC_COMP_BASEQuoted 6 lines
Quoted 4 lines
Having
hk_cpus<num_online_cpus()helps us ensure that we are only changing the behavior when we have isolated CPUs.Does that make sense?
That seems to want to allocate N interrupts per cpu (plus some random static amount, which seems weird, but whatever). This patch breaks that.
Quoted 8 lines
On purpose. For the isolated CPUs we don't want network device interrupts (in this context).
Quoted 3 lines
So I think it is important to figure out what that driver really wants in the
nohz_fullcase. If it wants to retain N interrupts per CPU, and only reduce the number of CPUs, the proposed interface is wrong.It wants N interrupts per non-isolated (AKA housekeeping) CPU. Zero interrupts for isolated interrupts.
Right, otherwise we may end up in a situation where we run out of per CPU vectors while we move the IRQs from isolated CPUs to housekeeping.
Signature · 10 lines
Thanks Nitesh -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20201019/c70b8a10/attachment-0001.asc>
Peter Zijlstra
maintainerRe: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
In reply to Marcelo TosattiQuoted 5 lines · Marcelo Tosatti
Quoted 3 lines
So I think it is important to figure out what that driver really wants in the
nohz_fullcase. If it wants to retain N interrupts per CPU, and only reduce the number of CPUs, the proposed interface is wrong.It wants N interrupts per non-isolated (AKA housekeeping) CPU.
Then the patch is wrong and the interface needs changing from @min_vecs, @max_vecs to something that expresses the N*nr_cpus relation.
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Marcelo Tosatti
Quoted 3 lines
So I think it is important to figure out what that driver really wants in the
nohz_fullcase. If it wants to retain N interrupts per CPU, and only reduce the number of CPUs, the proposed interface is wrong.It wants N interrupts per non-isolated (AKA housekeeping) CPU.
Then the patch is wrong and the interface needs changing from @min_vecs, @max_vecs to something that expresses the N*nr_cpus relation.
Reading Marcelo's comment again I think what is really expected is 1 interrupt per non-isolated (housekeeping) CPU (not N interrupts).
My bad that I missed it initially.
Signature · 10 lines
Thanks Nitesh -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20201020/3d789976/attachment-0001.asc>
Peter Zijlstra
maintainerRe: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 6 lines · Nitesh Narayan Lal
Quoted 4 lines · Peter Zijlstra
Quoted 3 lines · Marcelo Tosatti
Quoted 3 lines
So I think it is important to figure out what that driver really wants in the
nohz_fullcase. If it wants to retain N interrupts per CPU, and only reduce the number of CPUs, the proposed interface is wrong.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
It wants N interrupts per non-isolated (AKA housekeeping) CPU.
Then the patch is wrong and the interface needs changing from @min_vecs, @max_vecs to something that expresses the N*nr_cpus relation.
Reading Marcelo's comment again I think what is really expected is 1 interrupt per non-isolated (housekeeping) CPU (not N interrupts).
Then what is the point of them asking for N*nr_cpus when there is no isolation?
Either everybody wants 1 interrupts per CPU and we can do the clamp unconditionally, in which case we should go fix this user, or they want multiple per cpu and we should go fix the interface.
It cannot be both.
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 11 lines · Peter Zijlstra
Quoted 10 lines · Nitesh Narayan Lal
Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Marcelo Tosatti
Quoted 3 lines
So I think it is important to figure out what that driver really wants in the
nohz_fullcase. If it wants to retain N interrupts per CPU, and only reduce the number of CPUs, the proposed interface is wrong.It wants N interrupts per non-isolated (AKA housekeeping) CPU.
Then the patch is wrong and the interface needs changing from @min_vecs, @max_vecs to something that expresses the N*nr_cpus relation.
Reading Marcelo's comment again I think what is really expected is 1 interrupt per non-isolated (housekeeping) CPU (not N interrupts).
Quoted 8 lines
Then what is the point of them asking for N*nr_cpus when there is no isolation?
Either everybody wants 1 interrupts per CPU and we can do the clamp unconditionally, in which case we should go fix this user, or they want multiple per cpu and we should go fix the interface.
It cannot be both.
Based on my understanding I don't think this is consistent, the number of interrupts any driver can request varies to an extent that some consumer of this API even request just one interrupt for its use.
This was one of the reasons why I thought of having a conditional restriction.
But I agree there is a lack of consistency.
Signature · 9 lines
Nitesh -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20201020/807f3acc/attachment-0001.asc>
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 20 lines · Nitesh Narayan Lal
Quoted 11 lines · Peter Zijlstra
Quoted 10 lines · Nitesh Narayan Lal
Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Marcelo Tosatti
Quoted 3 lines
So I think it is important to figure out what that driver really wants in the
nohz_fullcase. If it wants to retain N interrupts per CPU, and only reduce the number of CPUs, the proposed interface is wrong.It wants N interrupts per non-isolated (AKA housekeeping) CPU.
Then the patch is wrong and the interface needs changing from @min_vecs, @max_vecs to something that expresses the N*nr_cpus relation.
Reading Marcelo's comment again I think what is really expected is 1 interrupt per non-isolated (housekeeping) CPU (not N interrupts).
Quoted 8 lines
Then what is the point of them asking for N*nr_cpus when there is no isolation?
Either everybody wants 1 interrupts per CPU and we can do the clamp unconditionally, in which case we should go fix this user, or they want multiple per cpu and we should go fix the interface.
It cannot be both.
Quoted 9 lines
Based on my understanding I don't think this is consistent, the number of interrupts any driver can request varies to an extent that some consumer of this API even request just one interrupt for its use.
This was one of the reasons why I thought of having a conditional restriction.
But I agree there is a lack of consistency.
Hi Peter,
So based on the suggestions from you and Thomas, I think something like the following should do the job within pci_alloc_irq_vectors_affinity():
+ ? ? ? if (!pci_is_managed(dev) && (hk_cpus < num_online_cpus()))
+ ? ? ? ? ? ? ? max_vecs = clamp(hk_cpus, min_vecs, max_vecs);
I do know that you didn't like the usage of "hk_cpus < num_online_cpus()" and to an extent I agree that it does degrade the code clarity.
However, since there is a certain inconsistency in the number of vectors that drivers request through this API IMHO we will need this, otherwise we could cause an impact on the drivers even in setups that doesn't have any isolated CPUs.
If you agree, I can send the next version of the patch-set.
Signature · 10 lines
Thanks Nitesh -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20201022/42e01657/attachment.asc>
Peter Zijlstra
maintainerRe: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 10 lines · Nitesh Narayan Lal
Hi Peter,
So based on the suggestions from you and Thomas, I think something like the following should do the job within
pci_alloc_irq_vectors_affinity():+ ? ? ? if (!pci_is_managed(dev) && (
hk_cpus<num_online_cpus()))
+ ? ? ? ? ? ? ?max_vecs=clamp(hk_cpus, min_vecs, max_vecs);I do know that you didn't like the usage of "
hk_cpus<num_online_cpus()" and to an extent I agree that it does degrade the code clarity.
It's not just code clarity; I simply don't understand it. It feels like a band-aid that breaks thing.
At the very least it needs a ginormous (and coherent) comment that explains:
- the interface
- the usage
- this hack
Quoted 4 lines
However, since there is a certain inconsistency in the number of vectors that drivers request through this API IMHO we will need this, otherwise we could cause an impact on the drivers even in setups that doesn't have any isolated CPUs.
So shouldn't we then fix the drivers / interface first, to get rid of this inconsistency?
Quoted 1 line
If you agree, I can send the next version of the patch-set.
Well, it's not just me you have to convince.
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 15 lines · Peter Zijlstra
Quoted 10 lines · Nitesh Narayan Lal
Hi Peter,
So based on the suggestions from you and Thomas, I think something like the following should do the job within
pci_alloc_irq_vectors_affinity():+ ? ? ? if (!pci_is_managed(dev) && (
hk_cpus<num_online_cpus()))
+ ? ? ? ? ? ? ?max_vecs=clamp(hk_cpus, min_vecs, max_vecs);I do know that you didn't like the usage of "
hk_cpus<num_online_cpus()" and to an extent I agree that it does degrade the code clarity.It's not just code clarity; I simply don't understand it. It feels like a band-aid that breaks thing.
Quoted 6 lines
At the very least it needs a ginormous (and coherent) comment that explains:
- the interface
- the usage
- this hack
That make sense.
Quoted 8 lines
Quoted 5 lines
However, since there is a certain inconsistency in the number of vectors that drivers request through this API IMHO we will need this, otherwise we could cause an impact on the drivers even in setups that doesn't have any isolated CPUs.
So shouldn't we then fix the drivers / interface first, to get rid of this inconsistency?
Considering we agree that excess vector is a problem that needs to be solved across all the drivers and that you are comfortable with the other three patches in the set. If I may suggest the following:
- We can pick those three patches for now, as that will atleast fix a ? driver that is currently impacting RT workloads. Is that a fair ? expectation?
- In the meanwhile, I will start looking into individual drivers that ? consume this API to find out if there is a co-relation that can be ? derived between the max_vecs and number of CPUs. If that exists then I ? can go ahead and tweak the API's max_vecs accordingly. However, if this ? is absolutely random then I can come up with a sane comment ? before this check that covers the list of items you suggested.
- I also want to explore the comments made by Thomas which may take ? some time.
Signature · 10 lines
Thanks Nitesh -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20201023/2ac925b4/attachment-0001.asc>
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 5 lines · Nitesh Narayan Lal
Quoted 4 lines · Peter Zijlstra
On Thu, Oct 22, 2020 at 01:47:14PM -0400, Nitesh Narayan Lal wrote:
So shouldn't we then fix the drivers / interface first, to get rid of this inconsistency?
Quoted 7 lines
Considering we agree that excess vector is a problem that needs to be solved across all the drivers and that you are comfortable with the other three patches in the set. If I may suggest the following:
- We can pick those three patches for now, as that will atleast fix a ? driver that is currently impacting RT workloads. Is that a fair ? expectation?
No. Blindly reducing the maximum vectors to the number of housekeeping CPUs is patently wrong. The PCI core _cannot_ just nilly willy decide what the right number of interrupts for this situation is.
Many of these drivers need more than queue interrupts, admin, error interrupt and some operate best with seperate RX/TX interrupts per queue. They all can "work" with a single PCI interrupt of course, but the price you pay is performance.
An isolated setup, which I'm familiar with, has two housekeeping CPUs. So far I restricted the number of network queues with a module argument to two, which allocates two management interrupts for the device and two interrupts (RX/TX) per queue, i.e. a total of six.
Now I reduced the number of available interrupts to two according to your hack, which makes it use one queue RX/TX combined and one management interrupt. Guess what happens? Network performance tanks to the points that it breaks a carefully crafted setup.
The same applies to a device which is application specific and wants one channel including an interrupt per isolated application core. Today I can isolate 8 out of 12 CPUs and let the device create 8 channels and set one interrupt and channel affine to each isolated CPU. With your hack, I get only 4 interrupts and channels. Fail!
You cannot declare that all this is perfectly fine, just because it does not matter for your particular use case.
So without information from the driver which tells what the best number of interrupts is with a reduced number of CPUs, this cutoff will cause more problems than it solves. Regressions guaranteed.
Managed interrupts base their interrupt allocation and spreading on information which is handed in by the individual driver and not on crude assumptions. They are not imposing restrictions on the use case.
It's perfectly fine for isolated work to save a data set to disk after computation has finished and that just works with the per-cpu I/O queue which is otherwise completely silent. All isolated workers can do the same in parallel without trampling on each other toes by competing for a reduced number of queues which are affine to the housekeeper CPUs.
Unfortunately network multi-queue is substantially different from block multi-queue (as I learned in this conversation), so the concept cannot be applied one-to-one to networking as is. But there are certainly part of it which can be reused.
This needs a lot more thought than just these crude hacks.
Especially under the aspect that there are talks about making isolation runtime switchable. Are you going to rmmod/insmod the i40e network driver to do so? That's going to work fine if you do that reconfiguration over network...
Thanks,
tglx
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 22 lines · Thomas Gleixner
Quoted 5 lines · Nitesh Narayan Lal
Quoted 4 lines · Peter Zijlstra
On Thu, Oct 22, 2020 at 01:47:14PM -0400, Nitesh Narayan Lal wrote:
So shouldn't we then fix the drivers / interface first, to get rid of this inconsistency?Quoted 7 lines
Considering we agree that excess vector is a problem that needs to be solved across all the drivers and that you are comfortable with the other three patches in the set. If I may suggest the following:
- We can pick those three patches for now, as that will atleast fix a ? driver that is currently impacting RT workloads. Is that a fair ? expectation?
No. Blindly reducing the maximum vectors to the number of housekeeping CPUs is patently wrong. The PCI core _cannot_ just nilly willy decide what the right number of interrupts for this situation is.
Many of these drivers need more than queue interrupts, admin, error interrupt and some operate best with seperate RX/TX interrupts per queue. They all can "work" with a single PCI interrupt of course, but the price you pay is performance.
Quoted 4 lines
An isolated setup, which I'm familiar with, has two housekeeping CPUs. So far I restricted the number of network queues with a module argument to two, which allocates two management interrupts for the device and two interrupts (RX/TX) per queue, i.e. a total of six.
Does it somehow take num_online_cpus() into consideration while deciding the number of interrupts to create?
Quoted 6 lines
Now I reduced the number of available interrupts to two according to your hack, which makes it use one queue RX/TX combined and one management interrupt. Guess what happens? Network performance tanks to the points that it breaks a carefully crafted setup.
Quoted 8 lines
The same applies to a device which is application specific and wants one channel including an interrupt per isolated application core. Today I can isolate 8 out of 12 CPUs and let the device create 8 channels and set one interrupt and channel affine to each isolated CPU. With your hack, I get only 4 interrupts and channels. Fail!
You cannot declare that all this is perfectly fine, just because it does not matter for your particular use case.
I agree that does sound wrong.
Quoted 4 lines
So without information from the driver which tells what the best number of interrupts is with a reduced number of CPUs, this cutoff will cause more problems than it solves. Regressions guaranteed.
Indeed.
I think one commonality among the drivers at the moment is the usage of num_online_cpus() to determine the vectors to create.
So, maybe instead of doing this kind of restrictions in a generic level API, it will make more sense to do this on a per-device basis by replacing the number of online CPUs with the housekeeping CPUs?
This is what I have done in the i40e patch.
But that still sounds hackish and will impact the performance.
Quoted 4 lines
Managed interrupts base their interrupt allocation and spreading on information which is handed in by the individual driver and not on crude assumptions. They are not imposing restrictions on the use case.
Right, FWIU it is irq_do_set_affinity that prevents the spreading of managed interrupts to isolated CPUs if HK_FLAG_MANAGED_IRQ is enabled, isn't?
Quoted 11 lines
It's perfectly fine for isolated work to save a data set to disk after computation has finished and that just works with the per-cpu I/O queue which is otherwise completely silent. All isolated workers can do the same in parallel without trampling on each other toes by competing for a reduced number of queues which are affine to the housekeeper CPUs.
Unfortunately network multi-queue is substantially different from block multi-queue (as I learned in this conversation), so the concept cannot be applied one-to-one to networking as is. But there are certainly part of it which can be reused.
So this is one of the areas that I don't understand well yet and will explore now.
Quoted 2 lines
This needs a lot more thought than just these crude hacks.
Got it.
I am indeed not in the favor of pushing a solution that has a possibility of regressing/breaking things afterward.
Quoted 5 lines
Especially under the aspect that there are talks about making isolation runtime switchable. Are you going to rmmod/insmod the i40e network driver to do so? That's going to work fine if you do that reconfiguration over network...
That's an interesting point. However, for some of the drivers which uses something like cpu_online/possible_mask while creating its affinity mask reloading will still associate jobs to the isolated CPUs.
Signature · 10 lines
Thanks Nitesh -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20201026/4be817af/attachment.asc>
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 8 lines · Nitesh Narayan Lal
Quoted 4 lines · Thomas Gleixner
An isolated setup, which I'm familiar with, has two housekeeping CPUs. So far I restricted the number of network queues with a module argument to two, which allocates two management interrupts for the device and two interrupts (RX/TX) per queue, i.e. a total of six.
Does it somehow take
num_online_cpus()into consideration while deciding the number of interrupts to create?
No, I just tell it to create two queues :)
Quoted 8 lines
Quoted 3 lines
So without information from the driver which tells what the best number of interrupts is with a reduced number of CPUs, this cutoff will cause more problems than it solves. Regressions guaranteed.
Indeed.
I think one commonality among the drivers at the moment is the usage ofnum_online_cpus()to determine the vectors to create.
Quoted 6 lines
So, maybe instead of doing this kind of restrictions in a generic level API, it will make more sense to do this on a per-device basis by replacing the number of online CPUs with the housekeeping CPUs?
This is what I have done in the i40e patch.
But that still sounds hackish and will impact the performance.
You want an interface which allows the driver to say:
I need N interrupts for general management and ideally M interrupts per queue.
This is similar to the way drivers tell the core code about their requirements for managed interrupts for the spreading calculation.
Quoted 7 lines
Quoted 3 lines
Managed interrupts base their interrupt allocation and spreading on information which is handed in by the individual driver and not on crude assumptions. They are not imposing restrictions on the use case.
Right, FWIU it is
irq_do_set_affinitythat prevents the spreading of managed interrupts to isolated CPUs ifHK_FLAG_MANAGED_IRQis enabled, isn't?
No. Spreading takes possible CPUs into account. HK_FLAG_MANAGED_IRQ does not influence spreading at all.
It only handles the case that an interrupt is affine to more than one CPUs and the resulting affinity mask spawns both housekeeping and isolated CPUs. It then steers the interrupt to the housekeeping CPUs (as long as there is one online).
Thanks,
tglx
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
In reply to Thomas GleixnerQuoted 33 lines · Thomas Gleixner
Quoted 5 lines · Nitesh Narayan Lal
Quoted 4 lines · Peter Zijlstra
On Thu, Oct 22, 2020 at 01:47:14PM -0400, Nitesh Narayan Lal wrote:
So shouldn't we then fix the drivers / interface first, to get rid of this inconsistency?Quoted 7 lines
Considering we agree that excess vector is a problem that needs to be solved across all the drivers and that you are comfortable with the other three patches in the set. If I may suggest the following:
- We can pick those three patches for now, as that will atleast fix a ? driver that is currently impacting RT workloads. Is that a fair ? expectation?
No. Blindly reducing the maximum vectors to the number of housekeeping CPUs is patently wrong. The PCI core _cannot_ just nilly willy decide what the right number of interrupts for this situation is.
Many of these drivers need more than queue interrupts, admin, error interrupt and some operate best with seperate RX/TX interrupts per queue. They all can "work" with a single PCI interrupt of course, but the price you pay is performance.
An isolated setup, which I'm familiar with, has two housekeeping CPUs. So far I restricted the number of network queues with a module argument to two, which allocates two management interrupts for the device and two interrupts (RX/TX) per queue, i.e. a total of six.
Now I reduced the number of available interrupts to two according to your hack, which makes it use one queue RX/TX combined and one management interrupt. Guess what happens? Network performance tanks to the points that it breaks a carefully crafted setup.
Quoted 5 lines
The same applies to a device which is application specific and wants one channel including an interrupt per isolated application core. Today I can isolate 8 out of 12 CPUs and let the device create 8 channels and set one interrupt and channel affine to each isolated CPU. With your hack, I get only 4 interrupts and channels. Fail!
Good point.
Quoted 6 lines
You cannot declare that all this is perfectly fine, just because it does not matter for your particular use case.
So without information from the driver which tells what the best number of interrupts is with a reduced number of CPUs, this cutoff will cause more problems than it solves. Regressions guaranteed.
One might want to move from one interrupt per isolated app core to zero, or vice versa. It seems that "best number of interrupts is with reduced number of CPUs" information, is therefore in userspace, not in driver...
No?
Quoted 7 lines
Managed interrupts base their interrupt allocation and spreading on information which is handed in by the individual driver and not on crude assumptions. They are not imposing restrictions on the use case.
It's perfectly fine for isolated work to save a data set to disk after computation has finished and that just works with the per-cpu I/O queue which is otherwise completely silent.
Userspace could only change the mask of interrupts which are not triggered by requests from the local CPU (admin, error, mgmt, etc), to avoid the vector exhaustion problem.
However, there is no explicit way for userspace to know that, as far as i know.
130: 34845 0 0 0 0 0 0 0 IR-PCI-MSI 33554433-edge nvme0q1 131: 0 27062 0 0 0 0 0 0 IR-PCI-MSI 33554434-edge nvme0q2 132: 0 0 24393 0 0 0 0 0 IR-PCI-MSI 33554435-edge nvme0q3 133: 0 0 0 24313 0 0 0 0 IR-PCI-MSI 33554436-edge nvme0q4 134: 0 0 0 0 20608 0 0 0 IR-PCI-MSI 33554437-edge nvme0q5 135: 0 0 0 0 0 22163 0 0 IR-PCI-MSI 33554438-edge nvme0q6 136: 0 0 0 0 0 0 23020 0 IR-PCI-MSI 33554439-edge nvme0q7 137: 0 0 0 0 0 0 0 24285 IR-PCI-MSI 33554440-edge nvme0q8
Can that be retrieved from PCI-MSI information, or drivers have to inform this?
Quoted 19 lines
All isolated workers can do the
same in parallel without trampling on each other toes by competing for a reduced number of queues which are affine to the housekeeper CPUs.Unfortunately network multi-queue is substantially different from block multi-queue (as I learned in this conversation), so the concept cannot be applied one-to-one to networking as is. But there are certainly part of it which can be reused.
This needs a lot more thought than just these crude hacks.
Especially under the aspect that there are talks about making isolation runtime switchable. Are you going to rmmod/insmod the i40e network driver to do so? That's going to work fine if you do that reconfiguration over network...
Thanks,
tglx
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 9 lines · Marcelo Tosatti
Quoted 3 lines · Thomas Gleixner
So without information from the driver which tells what the best number of interrupts is with a reduced number of CPUs, this cutoff will cause more problems than it solves. Regressions guaranteed.
One might want to move from one interrupt per isolated app core to zero, or vice versa. It seems that "best number of interrupts is with reduced number of CPUs" information, is therefore in userspace, not in driver...
How does userspace know about the driver internals? Number of management interrupts, optimal number of interrupts per queue?
Quoted 24 lines
Quoted 7 lines
Managed interrupts base their interrupt allocation and spreading on information which is handed in by the individual driver and not on crude assumptions. They are not imposing restrictions on the use case.
It's perfectly fine for isolated work to save a data set to disk after computation has finished and that just works with the per-cpu I/O queue which is otherwise completely silent.
Userspace could only change the mask of interrupts which are not triggered by requests from the local CPU (admin, error, mgmt, etc), to avoid the vector exhaustion problem.
However, there is no explicit way for userspace to know that, as far as i know.
130: 34845 0 0 0 0 0 0 0 IR-PCI-MSI 33554433-edge nvme0q1 131: 0 27062 0 0 0 0 0 0 IR-PCI-MSI 33554434-edge nvme0q2 132: 0 0 24393 0 0 0 0 0 IR-PCI-MSI 33554435-edge nvme0q3 133: 0 0 0 24313 0 0 0 0 IR-PCI-MSI 33554436-edge nvme0q4 134: 0 0 0 0 20608 0 0 0 IR-PCI-MSI 33554437-edge nvme0q5 135: 0 0 0 0 0 22163 0 0 IR-PCI-MSI 33554438-edge nvme0q6 136: 0 0 0 0 0 0 23020 0 IR-PCI-MSI 33554439-edge nvme0q7 137: 0 0 0 0 0 0 0 24285 IR-PCI-MSI 33554440-edge nvme0q8
Quoted 2 lines
Can that be retrieved from PCI-MSI information, or drivers have to inform this?
The driver should use a different name for the admin queues.
Thanks,
tglx
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 38 lines · Thomas Gleixner
Quoted 9 lines · Marcelo Tosatti
Quoted 3 lines · Thomas Gleixner
So without information from the driver which tells what the best number of interrupts is with a reduced number of CPUs, this cutoff will cause more problems than it solves. Regressions guaranteed.
One might want to move from one interrupt per isolated app core to zero, or vice versa. It seems that "best number of interrupts is with reduced number of CPUs" information, is therefore in userspace, not in driver...
How does userspace know about the driver internals? Number of management interrupts, optimal number of interrupts per queue?
Quoted 24 lines
Quoted 7 lines
Managed interrupts base their interrupt allocation and spreading on information which is handed in by the individual driver and not on crude assumptions. They are not imposing restrictions on the use case.
It's perfectly fine for isolated work to save a data set to disk after computation has finished and that just works with the per-cpu I/O queue which is otherwise completely silent.
Userspace could only change the mask of interrupts which are not triggered by requests from the local CPU (admin, error, mgmt, etc), to avoid the vector exhaustion problem.
However, there is no explicit way for userspace to know that, as far as i know.
130: 34845 0 0 0 0 0 0 0 IR-PCI-MSI 33554433-edge nvme0q1 131: 0 27062 0 0 0 0 0 0 IR-PCI-MSI 33554434-edge nvme0q2 132: 0 0 24393 0 0 0 0 0 IR-PCI-MSI 33554435-edge nvme0q3 133: 0 0 0 24313 0 0 0 0 IR-PCI-MSI 33554436-edge nvme0q4 134: 0 0 0 0 20608 0 0 0 IR-PCI-MSI 33554437-edge nvme0q5 135: 0 0 0 0 0 22163 0 0 IR-PCI-MSI 33554438-edge nvme0q6 136: 0 0 0 0 0 0 23020 0 IR-PCI-MSI 33554439-edge nvme0q7 137: 0 0 0 0 0 0 0 24285 IR-PCI-MSI 33554440-edge nvme0q8
Quoted 4 lines
Quoted 2 lines
Can that be retrieved from PCI-MSI information, or drivers have to inform this?
The driver should use a different name for the admin queues.
Works for me.
Sounds more like a heuristic which can break, so documenting this as an "interface" seems appropriate.
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
In reply to Thomas GleixnerQuoted 6 lines · Thomas Gleixner
Quoted 5 lines · Marcelo Tosatti
Quoted 3 lines · Thomas Gleixner
So without information from the driver which tells what the best number of interrupts is with a reduced number of CPUs, this cutoff will cause more problems than it solves. Regressions guaranteed.
Quoted 8 lines
Quoted 4 lines
One might want to move from one interrupt per isolated app core to zero, or vice versa. It seems that "best number of interrupts is with reduced number of CPUs" information, is therefore in userspace, not in driver...
How does userspace know about the driver internals? Number of management interrupts, optimal number of interrupts per queue?
I guess this is the problem solved in part by the queue management work that would make queues a thing that userspace is aware of.
Are there drivers which use more than one interrupt per queue? I know drivers have multiple management interrupts.. and I guess some drivers do combined 1 interrupt per pair of Tx/Rx.. It's also plausible to to have multiple queues for one interrupt .. I'm not sure how a single queue with multiple interrupts would work though.
Quoted 33 lines
Quoted 24 lines
Quoted 7 lines
Managed interrupts base their interrupt allocation and spreading on information which is handed in by the individual driver and not on crude assumptions. They are not imposing restrictions on the use case.
It's perfectly fine for isolated work to save a data set to disk after computation has finished and that just works with the per-cpu I/O queue which is otherwise completely silent.
Userspace could only change the mask of interrupts which are not triggered by requests from the local CPU (admin, error, mgmt, etc), to avoid the vector exhaustion problem.
However, there is no explicit way for userspace to know that, as far as i know.
130: 34845 0 0 0 0 0 0 0 IR-PCI-MSI 33554433-edge nvme0q1 131: 0 27062 0 0 0 0 0 0 IR-PCI-MSI 33554434-edge nvme0q2 132: 0 0 24393 0 0 0 0 0 IR-PCI-MSI 33554435-edge nvme0q3 133: 0 0 0 24313 0 0 0 0 IR-PCI-MSI 33554436-edge nvme0q4 134: 0 0 0 0 20608 0 0 0 IR-PCI-MSI 33554437-edge nvme0q5 135: 0 0 0 0 0 22163 0 0 IR-PCI-MSI 33554438-edge nvme0q6 136: 0 0 0 0 0 0 23020 0 IR-PCI-MSI 33554439-edge nvme0q7 137: 0 0 0 0 0 0 0 24285 IR-PCI-MSI 33554440-edge nvme0q8Quoted 2 lines
Can that be retrieved from PCI-MSI information, or drivers have to inform this?
The driver should use a different name for the admin queues.
Thanks,
tglx
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 4 lines · Jacob Keller
Quoted 2 lines · Thomas Gleixner
How does userspace know about the driver internals? Number of management interrupts, optimal number of interrupts per queue?
Quoted 8 lines
I guess this is the problem solved in part by the queue management work that would make queues a thing that userspace is aware of.
Are there drivers which use more than one interrupt per queue? I know drivers have multiple management interrupts.. and I guess some drivers do combined 1 interrupt per pair of Tx/Rx.. It's also plausible to to have multiple queues for one interrupt .. I'm not sure how a single queue with multiple interrupts would work though.
For block there is always one interrupt per queue. Some Network drivers seem to have seperate RX and TX interrupts per queue.
Thanks,
tglx
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 14 lines · Thomas Gleixner
Quoted 4 lines · Jacob Keller
Quoted 2 lines · Thomas Gleixner
How does userspace know about the driver internals? Number of management interrupts, optimal number of interrupts per queue?
Quoted 8 lines
I guess this is the problem solved in part by the queue management work that would make queues a thing that userspace is aware of.
Are there drivers which use more than one interrupt per queue? I know drivers have multiple management interrupts.. and I guess some drivers do combined 1 interrupt per pair of Tx/Rx.. It's also plausible to to have multiple queues for one interrupt .. I'm not sure how a single queue with multiple interrupts would work though.
Quoted 7 lines
For block there is always one interrupt per queue. Some Network drivers seem to have seperate RX and TX interrupts per queue.
Thanks,
tglx
That's true when thinking of Tx and Rx as a single queue. Another way to think about it is "one rx queue" and "one tx queue" each with their own interrupt...
Even if there are devices which force there to be exactly queue pairs, you could still think of them as separate entities?
Hmm.
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 8 lines · Jacob Keller
Quoted 7 lines · Thomas Gleixner
Quoted 5 lines · Jacob Keller
Are there drivers which use more than one interrupt per queue? I know drivers have multiple management interrupts.. and I guess some drivers do combined 1 interrupt per pair of Tx/Rx.. It's also plausible to to have multiple queues for one interrupt .. I'm not sure how a single queue with multiple interrupts would work though.
Quoted 8 lines
Quoted 2 lines
For block there is always one interrupt per queue. Some Network drivers seem to have seperate RX and TX interrupts per queue.
That's true when thinking of Tx and Rx as a single queue. Another way to think about it is "one rx queue" and "one tx queue" each with their own interrupt...
Even if there are devices which force there to be exactly queue pairs, you could still think of them as separate entities?
Interesting thought.
But as Jakub explained networking queues are fundamentally different from block queues on the RX side. For block the request issued on queue X will raise the complete interrupt on queue X.
For networking the TX side will raise the TX interrupt on the queue on which the packet was queued obviously or should I say hopefully. :)
But incoming packets will be directed to some receive queue based on a hash or whatever crystallball logic the firmware decided to implement.
Which makes this not really suitable for the managed interrupt and spreading approach which is used by block-mq. Hrm...
But I still think that for curing that isolation stuff we want at least some information from the driver. Alternative solution would be to grant the allocation of interrupts and queues and have some sysfs knob to shut down queues at runtime. If that shutdown results in releasing the queue interrupt (via free_irq()) then the vector exhaustion problem goes away.
Needs more thought and information (for network oblivious folks like me).
Thanks,
tglx
Jakub Kicinski
maintainerRe: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 33 lines · On Mon, 26 Oct 2020 22:50:45 +0100 Thomas Gleixner wrote:
Quoted 8 lines · Jacob Keller
Quoted 7 lines · Thomas Gleixner
Quoted 5 lines · Jacob Keller
Are there drivers which use more than one interrupt per queue? I know drivers have multiple management interrupts.. and I guess some drivers do combined 1 interrupt per pair of Tx/Rx.. It's also plausible to to have multiple queues for one interrupt .. I'm not sure how a single queue with multiple interrupts would work though.
Quoted 8 lines
Quoted 2 lines
For block there is always one interrupt per queue. Some Network drivers seem to have seperate RX and TX interrupts per queue.
That's true when thinking of Tx and Rx as a single queue. Another way to think about it is "one rx queue" and "one tx queue" each with their own interrupt...
Even if there are devices which force there to be exactly queue pairs, you could still think of them as separate entities?
Interesting thought.
But as Jakub explained networking queues are fundamentally different from block queues on the RX side. For block the request issued on queue X will raise the complete interrupt on queue X.
For networking the TX side will raise the TX interrupt on the queue on which the packet was queued obviously or should I say hopefully. :)
But incoming packets will be directed to some receive queue based on a hash or whatever crystallball logic the firmware decided to implement.
Which makes this not really suitable for the managed interrupt and spreading approach which is used by block-mq. Hrm...
Quoted 8 lines
But I still think that for curing that isolation stuff we want at least some information from the driver. Alternative solution would be to grant the allocation of interrupts and queues and have some sysfs knob to shut down queues at runtime. If that shutdown results in releasing the queue interrupt (via
free_irq()) then the vector exhaustion problem goes away.Needs more thought and information (for network oblivious folks like me).
One piece of information that may be useful is that even tho the RX packets may be spread semi-randomly the user space can still control which queues are included in the mechanism. There is an indirection table in the HW which allows to weigh queues differently, or exclude selected queues from the spreading. Other mechanisms exist to filter flows onto specific queues.
IOW just because a core has an queue/interrupt does not mean that interrupt will ever fire, provided its excluded from RSS.
Another piece is that by default we suggest drivers allocate 8 RX queues, and online_cpus TX queues. The number of queues can be independently controlled via ethtool -L. Drivers which can't support separate queues will default to online_cpus queue pairs, and let ethtool -L only set the "combined" parameter.
There are drivers which always allocate online_cpus interrupts, and then some of them will go unused if #qs < #cpus.
My unpopular opinion is that for networking devices all the heuristics we may come up with are going to be a dead end. We need an explicit API to allow users placing queues on cores, and use managed IRQs for data queues. (I'm assuming that managed IRQs will let us reliably map a MSI-X vector to a core :))
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 20 lines · Jakub Kicinski
Quoted 8 lines · On Mon, 26 Oct 2020 22:50:45 +0100 Thomas Gleixner wrote:
But I still think that for curing that isolation stuff we want at least some information from the driver. Alternative solution would be to grant the allocation of interrupts and queues and have some sysfs knob to shut down queues at runtime. If that shutdown results in releasing the queue interrupt (via
free_irq()) then the vector exhaustion problem goes away.Needs more thought and information (for network oblivious folks like me).
One piece of information that may be useful is that even tho the RX packets may be spread semi-randomly the user space can still control which queues are included in the mechanism. There is an indirection table in the HW which allows to weigh queues differently, or exclude selected queues from the spreading. Other mechanisms exist to filter flows onto specific queues.
IOW just because a core has an queue/interrupt does not mean that interrupt will ever fire, provided its excluded from RSS.
Quoted 8 lines
Another piece is that by default we suggest drivers allocate 8 RX queues, and
online_cpusTX queues. The number of queues can be independently controlled via ethtool -L. Drivers which can't support separate queues will default toonline_cpusqueue pairs, and let ethtool -L only set the "combined" parameter.There are drivers which always allocate
online_cpusinterrupts, and then some of them will go unused if #qs < #cpus.
Thanks for the enlightment.
Quoted 2 lines
My unpopular opinion is that for networking devices all the heuristics we may come up with are going to be a dead end.
I agree. Heuristics suck.
Quoted 3 lines
We need an explicit API to allow users placing queues on cores, and use managed IRQs for data queues. (I'm assuming that managed IRQs will let us reliably map a MSI-X vector to a core :))
Yes, they allow you to do that. That will need some tweaks to theway they work today (coming from the strict block mq semantics). You also need to be aware that managed irqs have also strict semantics vs. CPU hotplug. If the last CPU in the managed affinity set goes down then the interrupt is shut down by the irq core which means that you need to quiesce the associated queue before that happens. When the first CPU comes online again the interrupt is reenabled, so the queue should be able to handle it or has ensured that the device does not raise one before it is able to do so.
Thanks,
tglx
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
In reply to Jakub KicinskiQuoted 50 lines · Jakub Kicinski
Quoted 33 lines · On Mon, 26 Oct 2020 22:50:45 +0100 Thomas Gleixner wrote:
Quoted 8 lines · Jacob Keller
Quoted 7 lines · Thomas Gleixner
Quoted 5 lines · Jacob Keller
Are there drivers which use more than one interrupt per queue? I know drivers have multiple management interrupts.. and I guess some drivers do combined 1 interrupt per pair of Tx/Rx.. It's also plausible to to have multiple queues for one interrupt .. I'm not sure how a single queue with multiple interrupts would work though.
Quoted 8 lines
Quoted 2 lines
For block there is always one interrupt per queue. Some Network drivers seem to have seperate RX and TX interrupts per queue.
That's true when thinking of Tx and Rx as a single queue. Another way to think about it is "one rx queue" and "one tx queue" each with their own interrupt...
Even if there are devices which force there to be exactly queue pairs, you could still think of them as separate entities?
Interesting thought.
But as Jakub explained networking queues are fundamentally different from block queues on the RX side. For block the request issued on queue X will raise the complete interrupt on queue X.
For networking the TX side will raise the TX interrupt on the queue on which the packet was queued obviously or should I say hopefully. :)
But incoming packets will be directed to some receive queue based on a hash or whatever crystallball logic the firmware decided to implement.
Which makes this not really suitable for the managed interrupt and spreading approach which is used by block-mq. Hrm...
Quoted 8 lines
But I still think that for curing that isolation stuff we want at least some information from the driver. Alternative solution would be to grant the allocation of interrupts and queues and have some sysfs knob to shut down queues at runtime. If that shutdown results in releasing the queue interrupt (via
free_irq()) then the vector exhaustion problem goes away.Needs more thought and information (for network oblivious folks like me).
One piece of information that may be useful is that even tho the RX packets may be spread semi-randomly the user space can still control which queues are included in the mechanism. There is an indirection table in the HW which allows to weigh queues differently, or exclude selected queues from the spreading. Other mechanisms exist to filter flows onto specific queues.
Quoted 9 lines
IOW just because a core has an queue/interrupt does not mean that interrupt will ever fire, provided its excluded from RSS.
Another piece is that by default we suggest drivers allocate 8 RX queues, and
online_cpusTX queues. The number of queues can be independently controlled via ethtool -L. Drivers which can't support separate queues will default toonline_cpusqueue pairs, and let ethtool -L only set the "combined" parameter.
I know the Intel drivers usually have defaulted to trying to maintain queue pairs. I do not believe this is technically a HW restriction, but it is heavily built into the way the drivers work today.
Quoted 4 lines
There are drivers which always allocate
online_cpusinterrupts, and then some of them will go unused if #qs < #cpus.
Right.
Quoted 6 lines
My unpopular opinion is that for networking devices all the heuristics we may come up with are going to be a dead end. We need an explicit API to allow users placing queues on cores, and use managed IRQs for data queues. (I'm assuming that managed IRQs will let us reliably map a MSI-X vector to a core :))
I don't think it is that unpopular... This is the direction I'd like to see us go as well.
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
In reply to Thomas GleixnerQuoted 16 lines · Thomas Gleixner
Quoted 7 lines · Jacob Keller
Quoted 6 lines · Thomas Gleixner
Quoted 5 lines · Jacob Keller
Are there drivers which use more than one interrupt per queue? I know drivers have multiple management interrupts.. and I guess some drivers do combined 1 interrupt per pair of Tx/Rx.. It's also plausible to to have multiple queues for one interrupt .. I'm not sure how a single queue with multiple interrupts would work though.
Quoted 8 lines
Quoted 2 lines
For block there is always one interrupt per queue. Some Network drivers seem to have seperate RX and TX interrupts per queue.
That's true when thinking of Tx and Rx as a single queue. Another way to think about it is "one rx queue" and "one tx queue" each with their own interrupt...
Even if there are devices which force there to be exactly queue pairs, you could still think of them as separate entities?
Quoted 8 lines
Interesting thought.
But as Jakub explained networking queues are fundamentally different from block queues on the RX side. For block the request issued on queue X will raise the complete interrupt on queue X.
For networking the TX side will raise the TX interrupt on the queue on which the packet was queued obviously or should I say hopefully. :)
This is my impression as well.
Quoted 11 lines
But incoming packets will be directed to some receive queue based on a hash or whatever crystallball logic the firmware decided to implement.
Which makes this not really suitable for the managed interrupt and spreading approach which is used by block-mq. Hrm...
But I still think that for curing that isolation stuff we want at least some information from the driver. Alternative solution would be to grant the allocation of interrupts and queues and have some sysfs knob to shut down queues at runtime. If that shutdown results in releasing the queue interrupt (via
free_irq()) then the vector exhaustion problem goes away.
I think this is close to what I and Marcelo were discussing earlier today privately.
I don't think there is currently a way to control the enablement/disablement of interrupts from the userspace.
I think in terms of the idea we need something similar to what i40e does, that is shutdown all IRQs when CPU is suspended and restores the interrupt schema when the CPU is back online.
The two key difference will be that this API needs to be generic and also needs to be exposed to the userspace through something like sysfs as you have mentioned.
Signature · 10 lines
Thanks Nitesh -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20201026/25b2a722/attachment.asc>
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 7 lines · Nitesh Narayan Lal
Quoted 5 lines · Thomas Gleixner
But I still think that for curing that isolation stuff we want at least some information from the driver. Alternative solution would be to grant the allocation of interrupts and queues and have some sysfs knob to shut down queues at runtime. If that shutdown results in releasing the queue interrupt (via
free_irq()) then the vector exhaustion problem goes away.
Quoted 5 lines
I think this is close to what I and Marcelo were discussing earlier today privately.
I don't think there is currently a way to control the enablement/disablement of interrupts from the userspace.
You cannot just disable the interrupt. You need to make sure that the associated queue is shutdown or quiesced _before_ the interrupt is shut down.
Thanks,
tglx
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 14 lines · Thomas Gleixner
Quoted 7 lines · Nitesh Narayan Lal
Quoted 5 lines · Thomas Gleixner
But I still think that for curing that isolation stuff we want at least some information from the driver. Alternative solution would be to grant the allocation of interrupts and queues and have some sysfs knob to shut down queues at runtime. If that shutdown results in releasing the queue interrupt (via
free_irq()) then the vector exhaustion problem goes away.Quoted 5 lines
I think this is close to what I and Marcelo were discussing earlier today privately.
I don't think there is currently a way to control the enablement/disablement of interrupts from the userspace.
Quoted 8 lines
You cannot just disable the interrupt. You need to make sure that the associated queue is shutdown or quiesced _before_ the interrupt is shut down.
Thanks,
tglx
Could this be handled with a callback to the driver/hw? I know Intel HW should support this type of quiesce/shutdown.
Thanks,
Jake
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 11 lines · Jacob Keller
Quoted 7 lines · Thomas Gleixner
Quoted 2 lines · Nitesh Narayan Lal
I don't think there is currently a way to control the enablement/disablement of interrupts from the userspace.
You cannot just disable the interrupt. You need to make sure that the associated queue is shutdown or quiesced _before_ the interrupt is shut down.
Could this be handled with a callback to the driver/hw? I know Intel HW should support this type of quiesce/shutdown.
We can't have a callback from the interrupt shutdown code as you have to wait for the queue to drain packets in flight. Something like this
mark queue as going down (no more tx queueing) tell hardware not to route RX packets to it consume pending RX
wait for already queued TX packets to be sent
Look what the block people did. They have a common multi-instance hotplug state and they register each context (queue) as an instance. The hotplug core invokes the corresponding callbacks when bringing a CPU up or when shutting it down.
Thanks,
tglx
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
In reply to Nitesh Narayan LalQuoted 41 lines · Nitesh Narayan Lal
Quoted 16 lines · Thomas Gleixner
Quoted 7 lines · Jacob Keller
Quoted 6 lines · Thomas Gleixner
Quoted 5 lines · Jacob Keller
Are there drivers which use more than one interrupt per queue? I know drivers have multiple management interrupts.. and I guess some drivers do combined 1 interrupt per pair of Tx/Rx.. It's also plausible to to have multiple queues for one interrupt .. I'm not sure how a single queue with multiple interrupts would work though.
Quoted 8 lines
Quoted 2 lines
For block there is always one interrupt per queue. Some Network drivers seem to have seperate RX and TX interrupts per queue.
That's true when thinking of Tx and Rx as a single queue. Another way to think about it is "one rx queue" and "one tx queue" each with their own interrupt...
Even if there are devices which force there to be exactly queue pairs, you could still think of them as separate entities?
Quoted 8 lines
Interesting thought.
But as Jakub explained networking queues are fundamentally different from block queues on the RX side. For block the request issued on queue X will raise the complete interrupt on queue X.
For networking the TX side will raise the TX interrupt on the queue on which the packet was queued obviously or should I say hopefully. :)
This is my impression as well.
Quoted 11 lines
But incoming packets will be directed to some receive queue based on a hash or whatever crystallball logic the firmware decided to implement.
Which makes this not really suitable for the managed interrupt and spreading approach which is used by block-mq. Hrm...
But I still think that for curing that isolation stuff we want at least some information from the driver. Alternative solution would be to grant the allocation of interrupts and queues and have some sysfs knob to shut down queues at runtime. If that shutdown results in releasing the queue interrupt (via
free_irq()) then the vector exhaustion problem goes away.
Quoted 5 lines
I think this is close to what I and Marcelo were discussing earlier today privately.
I don't think there is currently a way to control the enablement/disablement of interrupts from the userspace.
As long as the interrupt obeys the "trigger when request has been performed by local CPU" rule (#1) (for MSI type interrupts, where driver allocates one I/O interrupt per CPU), don't see a need for the interface.
For other types of interrupts, interrupt controller should be programmed to not include the isolated CPU on its "destination CPU list".
About the block VS network discussion, what we are trying to do at skb level (Paolo Abeni CC'ed, author of the suggestion) is to use RPS to avoid skbs from being queued to a CPU (on RX), and to queue skbs on housekeeping CPUs for processing (TX).
However, if per-CPU interrupts are not disabled, then the (for example) network device is free to include the CPU in its list of destinations. Which would require one to say, configure RPS (or whatever mechanism is distributing interrupts).
Hum, it would feel safer (rather than trust the #1 rule to be valid in all cases) to ask the driver to disable the interrupt (after shutting down queue) for that particular CPU.
BTW, Thomas, software is free to configure a particular MSI-X interrupt to point to any CPU:
10.11 MESSAGE SIGNALLED INTERRUPTS
The PCI Local Bus Specification, Rev 2.2 (www.pcisig.com) introduces the concept of message signalled interrupts. As the specification indicates:
?Message signalled interrupts (MSI) is an optional feature that enables PCI devices to request service by writing a system-specified message to a system-specified address (PCI DWORD memory write transaction). The transaction address specifies the message destination while the transaction data specifies the message. System software is expected to initialize the message destination and message during device configuration, allocating one or more non-shared messages to each MSI capable function.?
Fields in the Message Address Register are as follows:
1. Bits 31-20 ? These bits contain a fixed value for interrupt messages (0FEEH). This value locates interrupts at the 1-MByte area with a base address of 4G ? 18M. All accesses to this region are directed as interrupt messages. Care must to be taken to ensure that no other device claims the region as I/O space.
2. Destination ID ? This field contains an 8-bit destination ID. It identifies the message?s target processor(s). The destination ID corresponds to bits 63:56 of the I/O APIC Redirection Table Entry if the IOAPIC is used to dispatch the interrupt to the processor(s).
---
So taking the example where computation happens while isolated and later stored via block interface, aren't we restricting the usage scenarios by enforcing the "per-CPU queue has interrupt pointing to owner CPU" rule?
Quoted 14 lines
I think in terms of the idea we need something similar to what i40e does, that is shutdown all IRQs when CPU is suspended and restores the interrupt schema when the CPU is back online.
The two key difference will be that this API needs to be generic and also needs to be exposed to the userspace through something like sysfs as you have mentioned.
Signature · 2 lines
Thanks Nitesh
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 5 lines · Marcelo Tosatti
On Mon, Oct 26, 2020 at 06:22:29PM -0400, Nitesh Narayan Lal wrote:
However, if per-CPU interrupts are not disabled, then the (for example) network device is free to include the CPU in its list of destinations. Which would require one to say, configure RPS (or whatever mechanism is distributing interrupts).
And why is that a problem? If that's possible then you can prevent getting RX interrupts already today.
Quoted 8 lines
Hum, it would feel safer (rather than trust the #1 rule to be valid in all cases) to ask the driver to disable the interrupt (after shutting down queue) for that particular CPU.
BTW, Thomas, software is free to configure a particular MSI-X interrupt to point to any CPU:
10.11 MESSAGE SIGNALLED INTERRUPTS
I know how MSI works :)
Quoted 4 lines
So taking the example where computation happens while isolated and later stored via block interface, aren't we restricting the usage scenarios by enforcing the "per-CPU queue has interrupt pointing to owner CPU" rule?
No. For block this is the ideal configuration (think locality) and it prevents vector exhaustion. If you make these interrupts freely routable then you bring back the vector exhaustion problem right away.
Now we already established that networking has different requirements, so you have to come up with a different solution for it which allows to work for all use cases.
Thanks,
tglx
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
In reply to Peter ZijlstraQuoted 7 lines · Peter Zijlstra
Quoted 5 lines
Quoted 2 lines
And what are the (desired) semantics vs hotplug? Using a cpumask without excluding hotplug is racy.
The
housekeeping_maskshould still remain constant, isn't? In any case, I can double check this.The goal is very much to have that dynamically configurable.
Right but unfortunately we are not there before a little while. And the existing code in these drivers allocating vectors doesn't even take into account hotplug as you spotted. So I agreed to let Nitesh fix this issue on top of the existing code until he can look into providing some infrastructure for these kind of vectors allocation. The first step would be to consolidate similar code from other drivers, then maybe handle hotplug and later dynamic housekeeping.
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
In reply to Nitesh Narayan Laldrivers/pci/msi.c int pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs, · line 1206
Quoted 10 lines · Nitesh Narayan Lal
drivers/pci/msi.c +9
-0+ hk_cpus =housekeeping_num_online_cpus (HK_FLAG_MANAGED_IRQ ); + +/* +* If we have isolated CPUs for use by real-time tasks, to keep the +* latency overhead to a minimum, device-specific IRQ vectors are moved +* to the housekeeping CPUs from the userspace by changing their +* affinity mask. Limit the vector usage to keep housekeeping CPUs from +* running out of IRQ vectors. +*/
This is not true for managed interrupts. The interrupts affinity of those cannot be changed by user space.
drivers/pci/msi.c int pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs, · line 1212
Quoted 6 lines
drivers/pci/msi.c +6
-0+if (hk_cpus <num_online_cpus ()) { +if (hk_cpus < min_vecs) + max_vecs = min_vecs; +else if (hk_cpus < max_vecs) + max_vecs = hk_cpus; + }
So now with that assume a 16 core machine (HT off for simplicity)
17 Requested interrupts (1 general, 16 queues)
Managed interrupts will allocate
1 general interrupt which is free movable by user space 16 managed interrupts for queues (one per CPU)
This allows the driver to have 16 queues, i.e. one queue per CPU. These interrupts are only used when an application on a CPU issues I/O.
With the above change this will result
1 general interrupt which is free movable by user space 1 managed interrupts (possible affinity to all 16 CPUs, but routed
to housekeeping CPU as long as there is one online)
So the device is now limited to a single queue which also affects the housekeeping CPUs because now they have to share a single queue.
With larger machines this gets even worse.
So no. This needs way more thought for managed interrupts and you cannot do that at the PCI layer. Only the affinity spreading mechanism can do the right thing here.
Thanks,
tglx
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 13 lines · Thomas Gleixner
drivers/pci/msi.c int pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs, · line 1206
Quoted 10 lines · Nitesh Narayan Lal
drivers/pci/msi.c +9
-0+ hk_cpus =housekeeping_num_online_cpus (HK_FLAG_MANAGED_IRQ ); + +/* +* If we have isolated CPUs for use by real-time tasks, to keep the +* latency overhead to a minimum, device-specific IRQ vectors are moved +* to the housekeeping CPUs from the userspace by changing their +* affinity mask. Limit the vector usage to keep housekeeping CPUs from +* running out of IRQ vectors. +*/ This is not true for managed interrupts. The interrupts affinity of those cannot be changed by user space.
Ah Yes. Perhaps
s/IRQs/non-managed IRQ ?
Quoted 11 lines
drivers/pci/msi.c int pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs, · line 1212
Quoted 7 lines
drivers/pci/msi.c +6
-0+if (hk_cpus <num_online_cpus ()) { +if (hk_cpus < min_vecs) + max_vecs = min_vecs; +else if (hk_cpus < max_vecs) + max_vecs = hk_cpus; + }So now with that assume a 16 core machine (HT off for simplicity)
17 Requested interrupts (1 general, 16 queues)
Quoted 7 lines
Managed interrupts will allocate
1 general interrupt which is free movable by user space 16 managed interrupts for queues (one per CPU)
This allows the driver to have 16 queues, i.e. one queue per CPU. These interrupts are only used when an application on a CPU issues I/O.
Correct.
Quoted 11 lines
With the above change this will result
1 general interrupt which is free movable by user space 1 managed interrupts (possible affinity to all 16 CPUs, but routed
to housekeeping CPU as long as there is one online)So the device is now limited to a single queue which also affects the housekeeping CPUs because now they have to share a single queue.
With larger machines this gets even worse.
Yes, the change can impact the performance, however, if we don't do that we may have a latency impact instead. Specifically, on larger systems where most of the CPUs are isolated as we will definitely fail in moving all of the IRQs away from the isolated CPUs to the housekeeping.
Quoted 3 lines
So no. This needs way more thought for managed interrupts and you cannot do that at the PCI layer.
Maybe we should not be doing anything in the case of managed IRQs as they are anyways pinned to the housekeeping CPUs as long as we have the 'managed_irq' option included in the kernel cmdline.
Quoted 2 lines
Only the affinity spreading mechanism can do
the right thing here.
I can definitely explore this further.
However, IMHO we would still need a logic to prevent the devices from creating excess vectors.
Do you agree?
Quoted 5 lines
Thanks,
tglx
Signature · 10 lines
Thanks Nitesh -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20201020/c4247b8a/attachment.asc>
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 10 lines · Nitesh Narayan Lal
Quoted 9 lines · Thomas Gleixner
With the above change this will result
1 general interrupt which is free movable by user space 1 managed interrupts (possible affinity to all 16 CPUs, but routed
to housekeeping CPU as long as there is one online)So the device is now limited to a single queue which also affects the housekeeping CPUs because now they have to share a single queue.
Quoted 6 lines
Quoted 1 line
With larger machines this gets even worse.
Yes, the change can impact the performance, however, if we don't do that we may have a latency impact instead. Specifically, on larger systems where most of the CPUs are isolated as we will definitely fail in moving all of the IRQs away from the isolated CPUs to the housekeeping.
For non managed interrupts I agree.
Quoted 6 lines
Quoted 2 lines
So no. This needs way more thought for managed interrupts and you cannot do that at the PCI layer.
Maybe we should not be doing anything in the case of managed IRQs as they are anyways pinned to the housekeeping CPUs as long as we have the '
managed_irq' option included in the kernel cmdline.
Exactly. For the PCI side this vector limiting has to be restricted to the non managed case.
Quoted 7 lines
Quoted 2 lines
Only the affinity spreading mechanism can do
the right thing here.I can definitely explore this further.
However, IMHO we would still need a logic to prevent the devices from creating excess vectors.
Managed interrupts are preventing exactly that by pinning the interrupts and queues to one or a set of CPUs, which prevents vector exhaustion on CPU hotplug.
Non-managed, yes that is and always was a problem. One of the reasons why managed interrupts exist.
Thanks,
tglx
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 10 lines · Thomas Gleixner
Quoted 2 lines · Nitesh Narayan Lal
However, IMHO we would still need a logic to prevent the devices from creating excess vectors.
Managed interrupts are preventing exactly that by pinning the interrupts and queues to one or a set of CPUs, which prevents vector exhaustion on CPU hotplug.
Non-managed, yes that is and always was a problem. One of the reasons why managed interrupts exist.
But why is this only a problem for isolation? The very same problem exists vs. CPU hotplug and therefore hibernation.
On x86 we have at max. 204 vectors available for device interrupts per CPU. So assumed the only device interrupt in use is networking then any machine which has more than 204 network interrupts (queues, aux ...) active will prevent the machine from hibernation.
Aside of that it's silly to have multiple queues targeted at a single CPU in case of hotplug. And that's not a theoretical problem. Some power management schemes shut down sockets when the utilization of a system is low enough, e.g. outside of working hours.
The whole point of multi-queue is to have locality so that traffic from a CPU goes through the CPU local queue. What's the point of having two or more queues on a CPU in case of hotplug?
The right answer to this is to utilize managed interrupts and have according logic in your network driver to handle CPU hotplug. When a CPU goes down, then the queue which is associated to that CPU is quiesced and the interrupt core shuts down the relevant interrupt instead of moving it to an online CPU (which causes the whole vector exhaustion problem on x86). When the CPU comes online again, then the interrupt is reenabled in the core and the driver reactivates the queue.
Thanks,
tglx
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 10 lines · Thomas Gleixner
Quoted 9 lines · Thomas Gleixner
Quoted 2 lines · Nitesh Narayan Lal
However, IMHO we would still need a logic to prevent the devices from creating excess vectors.
Managed interrupts are preventing exactly that by pinning the interrupts and queues to one or a set of CPUs, which prevents vector exhaustion on CPU hotplug.
Non-managed, yes that is and always was a problem. One of the reasons why managed interrupts exist.
Quoted 7 lines
But why is this only a problem for isolation? The very same problem exists vs. CPU hotplug and therefore hibernation.
On x86 we have at max. 204 vectors available for device interrupts per CPU. So assumed the only device interrupt in use is networking then any machine which has more than 204 network interrupts (queues, aux ...) active will prevent the machine from hibernation.
Yes, that is indeed the case.
Quoted 10 lines
Aside of that it's silly to have multiple queues targeted at a single CPU in case of hotplug. And that's not a theoretical problem. Some power management schemes shut down sockets when the utilization of a system is low enough, e.g. outside of working hours.
The whole point of multi-queue is to have locality so that traffic from a CPU goes through the CPU local queue. What's the point of having two or more queues on a CPU in case of hotplug?
Quoted 7 lines
The right answer to this is to utilize managed interrupts and have according logic in your network driver to handle CPU hotplug. When a CPU goes down, then the queue which is associated to that CPU is quiesced and the interrupt core shuts down the relevant interrupt instead of moving it to an online CPU (which causes the whole vector exhaustion problem on x86). When the CPU comes online again, then the interrupt is reenabled in the core and the driver reactivates the queue.
IIRC then i40e does have something like that where it suspends all IRQs before hibernation and restores them when the CPU is back online.
I am not particularly sure about the other drivers.
This brings me to another discussion that Peter initiated that is to perform the proposed restriction without any condition for all non-managed IRQs.
Something on the lines:
+?????? if (!pci_is_managed(dev))
+?????????????? max_vecs = clamp(hk_cpus, min_vecs, max_vecs);
I am not particularly sure about this because I am not sure what kind of performance penalty this will have on the drivers in general and if that will be acceptable at all. Any thoughts?
However, this still doesn't solve the generic problem, and an ideal solution will be something that you suggested.
Will it be sensible to think about having a generic API that can be consumed by all the drivers and that can do both the things you mentioned?
Quoted 7 lines
Thanks,
tglx
Signature · 10 lines
Thanks Nitesh -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20201021/23faabd1/attachment-0001.asc>
Jakub Kicinski
maintainerRe: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
In reply to Thomas GleixnerQuoted 29 lines · On Wed, 21 Oct 2020 22:25:48 +0200 Thomas Gleixner wrote:
Quoted 10 lines · Thomas Gleixner
Quoted 2 lines · Nitesh Narayan Lal
However, IMHO we would still need a logic to prevent the devices from creating excess vectors.
Managed interrupts are preventing exactly that by pinning the interrupts and queues to one or a set of CPUs, which prevents vector exhaustion on CPU hotplug.
Non-managed, yes that is and always was a problem. One of the reasons why managed interrupts exist.
But why is this only a problem for isolation? The very same problem exists vs. CPU hotplug and therefore hibernation.
On x86 we have at max. 204 vectors available for device interrupts per CPU. So assumed the only device interrupt in use is networking then any machine which has more than 204 network interrupts (queues, aux ...) active will prevent the machine from hibernation.
Aside of that it's silly to have multiple queues targeted at a single CPU in case of hotplug. And that's not a theoretical problem. Some power management schemes shut down sockets when the utilization of a system is low enough, e.g. outside of working hours.
The whole point of multi-queue is to have locality so that traffic from a CPU goes through the CPU local queue. What's the point of having two or more queues on a CPU in case of hotplug?
Quoted 7 lines
The right answer to this is to utilize managed interrupts and have according logic in your network driver to handle CPU hotplug. When a CPU goes down, then the queue which is associated to that CPU is quiesced and the interrupt core shuts down the relevant interrupt instead of moving it to an online CPU (which causes the whole vector exhaustion problem on x86). When the CPU comes online again, then the interrupt is reenabled in the core and the driver reactivates the queue.
I think Mellanox folks made some forays into managed irqs, but I don't remember/can't find the details now.
For networking the locality / queue per core does not always work, since the incoming traffic is usually spread based on a hash. Many applications perform better when network processing is done on a small subset of CPUs, and application doesn't get interrupted every 100us. So we do need extra user control here.
We have a bit of a uAPI problem since people had grown to depend on IRQ == queue == NAPI to configure their systems. "The right way" out would be a proper API which allows associating queues with CPUs rather than IRQs, then we can use managed IRQs and solve many other problems.
Such new API has been in the works / discussions for a while now.
(Magnus keep me honest here, if you disagree the queue API solves this.)
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 38 lines · Jakub Kicinski
Quoted 29 lines · On Wed, 21 Oct 2020 22:25:48 +0200 Thomas Gleixner wrote:
Quoted 10 lines · Thomas Gleixner
Quoted 2 lines · Nitesh Narayan Lal
However, IMHO we would still need a logic to prevent the devices from creating excess vectors.
Managed interrupts are preventing exactly that by pinning the interrupts and queues to one or a set of CPUs, which prevents vector exhaustion on CPU hotplug.
Non-managed, yes that is and always was a problem. One of the reasons why managed interrupts exist.
But why is this only a problem for isolation? The very same problem exists vs. CPU hotplug and therefore hibernation.
On x86 we have at max. 204 vectors available for device interrupts per CPU. So assumed the only device interrupt in use is networking then any machine which has more than 204 network interrupts (queues, aux ...) active will prevent the machine from hibernation.
Aside of that it's silly to have multiple queues targeted at a single CPU in case of hotplug. And that's not a theoretical problem. Some power management schemes shut down sockets when the utilization of a system is low enough, e.g. outside of working hours.
The whole point of multi-queue is to have locality so that traffic from a CPU goes through the CPU local queue. What's the point of having two or more queues on a CPU in case of hotplug?
Quoted 7 lines
The right answer to this is to utilize managed interrupts and have according logic in your network driver to handle CPU hotplug. When a CPU goes down, then the queue which is associated to that CPU is quiesced and the interrupt core shuts down the relevant interrupt instead of moving it to an online CPU (which causes the whole vector exhaustion problem on x86). When the CPU comes online again, then the interrupt is reenabled in the core and the driver reactivates the queue.
Quoted 3 lines
I think Mellanox folks made some forays into managed irqs, but I don't remember/can't find the details now.
I remember looking into this a few years ago, and not getting very far either.
Quoted 11 lines
For networking the locality / queue per core does not always work, since the incoming traffic is usually spread based on a hash. Many applications perform better when network processing is done on a small subset of CPUs, and application doesn't get interrupted every 100us. So we do need extra user control here.
We have a bit of a uAPI problem since people had grown to depend on IRQ == queue == NAPI to configure their systems. "The right way" out would be a proper API which allows associating queues with CPUs rather than IRQs, then we can use managed IRQs and solve many other problems.
I think we (Intel) hit some of the same issues you mention.
I know I personally would like to see something that lets a lot of the current driver-specific policy be moved out. I think it should be possible to significantly simplify the abstraction used by the drivers.
Quoted 4 lines
Such new API has been in the works / discussions for a while now.
(Magnus keep me honest here, if you disagree the queue API solves this.)
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
In reply to Jakub KicinskiQuoted 9 lines · Jakub Kicinski
Quoted 7 lines · On Wed, 21 Oct 2020 22:25:48 +0200 Thomas Gleixner wrote:
The right answer to this is to utilize managed interrupts and have according logic in your network driver to handle CPU hotplug. When a CPU goes down, then the queue which is associated to that CPU is quiesced and the interrupt core shuts down the relevant interrupt instead of moving it to an online CPU (which causes the whole vector exhaustion problem on x86). When the CPU comes online again, then the interrupt is reenabled in the core and the driver reactivates the queue.
Quoted 5 lines
I think Mellanox folks made some forays into managed irqs, but I don't remember/can't find the details now.
For networking the locality / queue per core does not always work, since the incoming traffic is usually spread based on a hash. Many
That makes it problematic and is fundamentally different from block I/O.
Quoted 3 lines
applications perform better when network processing is done on a small subset of CPUs, and application doesn't get interrupted every 100us. So we do need extra user control here.
Ok.
Quoted 6 lines
We have a bit of a uAPI problem since people had grown to depend on IRQ == queue == NAPI to configure their systems. "The right way" out would be a proper API which allows associating queues with CPUs rather than IRQs, then we can use managed IRQs and solve many other problems.
Such new API has been in the works / discussions for a while now.
If there is anything which needs to be done/extended on the irq side please let me know.
Thanks
tglx
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
In reply to Thomas GleixnerQuoted 20 lines · Thomas Gleixner
Quoted 10 lines · Thomas Gleixner
Quoted 2 lines · Nitesh Narayan Lal
However, IMHO we would still need a logic to prevent the devices from creating excess vectors.
Managed interrupts are preventing exactly that by pinning the interrupts and queues to one or a set of CPUs, which prevents vector exhaustion on CPU hotplug.
Non-managed, yes that is and always was a problem. One of the reasons why managed interrupts exist.
But why is this only a problem for isolation? The very same problem exists vs. CPU hotplug and therefore hibernation.
On x86 we have at max. 204 vectors available for device interrupts per CPU. So assumed the only device interrupt in use is networking then any machine which has more than 204 network interrupts (queues, aux ...) active will prevent the machine from hibernation.
Quoted 4 lines
Aside of that it's silly to have multiple queues targeted at a single CPU in case of hotplug. And that's not a theoretical problem. Some power management schemes shut down sockets when the utilization of a system is low enough, e.g. outside of working hours.
Exactly. It seems the proper way to do handle this is to disable individual vectors rather than moving them. And that is needed for dynamic isolate / unisolate anyway...
Quoted 11 lines
The whole point of multi-queue is to have locality so that traffic from a CPU goes through the CPU local queue. What's the point of having two or more queues on a CPU in case of hotplug?
The right answer to this is to utilize managed interrupts and have according logic in your network driver to handle CPU hotplug. When a CPU goes down, then the queue which is associated to that CPU is quiesced and the interrupt core shuts down the relevant interrupt instead of moving it to an online CPU (which causes the whole vector exhaustion problem on x86). When the CPU comes online again, then the interrupt is reenabled in the core and the driver reactivates the queue.
Aha... But it would be necessary to do that from userspace (for runtime isolate/unisolate).
Re: [PATCH v4 4/4] PCI: Limit pci_alloc_irq_vectors() to housekeeping CPUs
Quoted 11 lines · Marcelo Tosatti
Quoted 7 lines · Thomas Gleixner
The right answer to this is to utilize managed interrupts and have according logic in your network driver to handle CPU hotplug. When a CPU goes down, then the queue which is associated to that CPU is quiesced and the interrupt core shuts down the relevant interrupt instead of moving it to an online CPU (which causes the whole vector exhaustion problem on x86). When the CPU comes online again, then the interrupt is reenabled in the core and the driver reactivates the queue.
Aha... But it would be necessary to do that from userspace (for runtime isolate/unisolate).
For anything which uses managed interrupts this is a non-problem and userspace has absolutely no business with it.
Isolation does not shut down queues, at least not the block multi-queue ones which are only active when I/O is issued from that isolated CPU.
So transitioning out of isolation requires no action at all.
Transitioning in or changing the housekeeping mask needs some trivial tweak to handle the case where there is an overlap in the cpuset of a queue (housekeeping and isolated). This is handled already for setup and affinity changes, but of course not for runtime isolation mask changes, but that's a trivial thing to do.
What's more interesting is how to deal with the network problem where there is no guarantee that the "response" ends up on the same queue as the "request" which is what the block people rely on. And that problem is not really an interrupt affinity problem in the first place.
Thanks,
tglx
Quoted 11 lines · Nitesh Narayan Lal
Nitesh Narayan Lal (4):
sched/isolation: API to get number of housekeeping CPUs sched/isolation: Extendnohz_fullto isolate managed IRQs
i40e: Limit msix vectors to housekeeping CPUs
PCI: Limitpci_alloc_irq_vectors()to housekeeping CPUsdrivers/net/ethernet/intel/i40e/i40e_main.c | 3 ++- drivers/pci/msi.c | 18 ++++++++++++++++++ include/linux/sched/isolation.h | 9 +++++++++ kernel/sched/isolation.c | 2 +- 4 files changed, 30 insertions(+), 2 deletions(-)
Peter, if you're ok with the set, I guess this should go through the scheduler tree?
Thanks.
Quoted 12 lines · Frederic Weisbecker
Quoted 11 lines · Nitesh Narayan Lal
Nitesh Narayan Lal (4):
sched/isolation: API to get number of housekeeping CPUs sched/isolation: Extendnohz_fullto isolate managed IRQs
i40e: Limit msix vectors to housekeeping CPUs
PCI: Limitpci_alloc_irq_vectors()to housekeeping CPUsdrivers/net/ethernet/intel/i40e/i40e_main.c | 3 ++- drivers/pci/msi.c | 18 ++++++++++++++++++ include/linux/sched/isolation.h | 9 +++++++++ kernel/sched/isolation.c | 2 +- 4 files changed, 30 insertions(+), 2 deletions(-)
Quoted 6 lines
Peter, if you're ok with the set, I guess this should go through the scheduler tree?
Thanks.
Hi Peter,
I wanted follow up to check if you have any concerns/suggestions that I should address in this patch-set before this can be picked?
Signature · 10 lines
Thanks Nitesh -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20201008/74292894/attachment.asc>