fix ia64 build failure when CONFIG_SFC=m
Subsystems (2)
PATCH 1/1 fix ia64 build failure when CONFIG_SFC=m
CONFIG_SFC=m uses topology_core_siblings() which, for ia64, expects cpu_core_map to be exported. It is not. This patch exports the needed symbol.
Index: contention_unroll/arch/ia64/kernel/smpboot.c
=================================--- contention_unroll.orig/arch/ia64/kernel/smpboot.c 2008-07-28 18:19:56.000000000 -0500
+++ contention_unroll/arch/ia64/kernel/smpboot.c 2008-07-28 20:25:54.240198328 -0500
@@ -138,6 +138,7 @@ cpumask_t cpu_possible_map = CPU_MASK_NO
EXPORT_SYMBOL(cpu_possible_map);
cpumask_t cpu_core_map[NR_CPUS] __cacheline_aligned;
+EXPORT_SYMBOL(cpu_core_map);
DEFINE_PER_CPU_SHARED_ALIGNED(cpumask_t, cpu_sibling_map); EXPORT_PER_CPU_SYMBOL(cpu_sibling_map);
Quoted 3 lines
CONFIG_SFC=musestopology_core_siblings()which, for ia64, expectscpu_core_mapto be exported. It is not. This patch exports the needed symbol.
Ben,
Before I rush to apply this (or one of the other identical patches that I've received) ... I'd like to ponder on whether this is the right thing to do.
Looking at the code in drivers/net/sfc/efx.c I see that you are using this to compute the number of RX queues based on the number of packages with the git commit comment saying:
Using multiple cores in the same package to handle received traffic does not appear to provide a performance benefit. Therefore use CPU topology information to count CPU packages and use that as the default number of RX queues and interrupts. We rely on interrupt balancing to spread the interrupts across packages.
I have some questions on this:
1) Did you measure on FSB based systems? ... It's kind of sad that the extra cores are not helping and I wonder why. Do AMD and QPI based systems have this same limitation?
2) Should Linux have an API to give you a useful number of threads, rather than make you dig through the topology data structures? At some point cpumask_t data structure is going to be too big for the kernel stack.
3) Does interrupt balancing always do the right thing to spread your interrupts across packages?
4) In a hotplug system would you want to adjust the number of threads if cpus were added or removed?
-Tony
Quoted 20 lines · Tony
Quoted 3 lines
CONFIG_SFC=musestopology_core_siblings()which, for ia64, expectscpu_core_mapto be exported. It is not. This patch exports the needed symbol.Ben,
Before I rush to apply this (or one of the other identical patches that I've received) ... I'd like to ponder on whether this is the right thing to do.
Looking at the code in
drivers/net/sfc/efx.cI see that you are using this to compute the number of RX queues based on the number of packages with the git commit comment saying:Using multiple cores in the same package to handle received traffic does not appear to provide a performance benefit. Therefore use CPU topology information to count CPU packages and use that as the default number of RX queues and interrupts. We rely on interrupt balancing to spread the interrupts across packages.
Quoted 5 lines
I have some questions on this:
1) Did you measure on FSB based systems? ... It's kind of sad that the extra cores are not helping and I wonder why. Do AMD and QPI based systems have this same limitation?
This heuristic has been in the out-of-tree driver for some time, mostly based on experience with Intel x86 systems but apparently resulting in good performance on both Intel and AMD multi-core systems tested here. We do not have any IA64 multi-core systems. I think a single core in each package can generally saturate the memory bus and this is why spreading the load wider is not useful.
Quoted 3 lines
2) Should Linux have an API to give you a useful number of threads, rather than make you dig through the topology data structures? At some point
cpumask_tdata structure is going to be too big for the kernel stack.
Since that's all we want to know, it would certainly simplify this driver and might be useful to others that implement RSS.
Quoted 2 lines
3) Does interrupt balancing always do the right thing to spread your interrupts across packages?
No, if we want to be sure then we do interrupt pinning afterwards.
Quoted 2 lines
4) In a hotplug system would you want to adjust the number of threads if cpus were added or removed?
That would make sense, but I don't think it can be done without disrupting network traffic.
Ben.
Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare Communications Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.
David Miller
maintainerFrom: Ben Hutchings <[email protected]>
Date: Tue, 29 Jul 2008 18:48:41 +0100
Quoted 2 lines
I think a single core in each package can generally saturate the memory bus and this is why spreading the load wider is not useful.
I disagree, especially for routing and firewall setups.
You want as many cpu threads as possible, even on the same core, doing the routing and firewall lookups through the various datastructures, in parallel.
Quoted 11 lines · David Miller
From: Ben Hutchings <[email protected]>
Date: Tue, 29 Jul 2008 18:48:41 +0100Quoted 2 lines
I think a single core in each package can generally saturate the memory bus and this is why spreading the load wider is not useful.
I disagree, especially for routing and firewall setups.
You want as many cpu threads as possible, even on the same core, doing the routing and firewall lookups through the various datastructures, in parallel.
So far as I can see, hardly anyone is doing routing at 10G speeds on Linux. If they were, the horrible interaction with LRO would presumably have been found and fixed earlier. Besides which, internal benchmarking showed that we could route 1500-byte packets bidirectionally at very near line rate using one core on each of two multi-core packages. (I can give more details if you want.) The default can in any case be overridden using the rss_cpus module parameter. If you want to add a core network setting for that, I think we'd be happy to use it.
Ben.
Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare Communications Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.
David Miller
maintainerFrom: Ben Hutchings <[email protected]>
Date: Wed, 30 Jul 2008 14:35:29 +0100
Quoted 3 lines
So far as I can see, hardly anyone is doing routing at 10G speeds on Linux. If they were, the horrible interaction with LRO would presumably have been found and fixed earlier.
This assumes a precise intersection between those doing 10G routing and their using chips with LRO capable drivers.
All of the 10G equipment I have here and do testing on use drivers that lack LRO support.