cpumask tree
Subsystems (42)
Rusty Russell
Reviewed-by Grant Grundler Tested-by Tony Luck Acked-by Ingo Molnar Acked-by Ingo Molnar Reviewed-by Grant Grundler Acked-by Ingo Molnar Acked-by Greg Kroah-Hartman Acked-by Ingo Molnar Acked-by David S. Miller Acked-by Hoang-Nam Nguyen Tested-by Hoang-Nam Nguyen Acked-by Hoang-Nam Nguyen Tested-by Hoang-Nam Nguyen Acked-by Ingo Molnar Acked-by Ingo Molnar Acked-by Ingo Molnar Acked-by Hirokazu Takata Acked-by Ingo Molnar Acked-by James Morris Acked-by Mike Frysinger Acked-by Steven Rostedt Reviewed-by Christoph LameterOK, this is the bulk of the conversion to the new cpumask operators. The x86-specific parts (the most aggressive large-NR_CPUS arch) are going via Ingo's tree.
Goals:
1) Get cpumasks off the stack when CONFIG_CPUMASK_OFFSTACK. This should be achieved for core & x86 by 2.6.29. cpumask_var_t is struct cpumask[1] for CONFIG_CPUMASK_OFFSTACK=n (and alloc_cpumask_var et al. a noop).
2) Convert to new cpumask functions which only go to nr_cpu_ids for large NR_CPUS, so booting huge configured kernels on small machines doesn't suck.
3) Allocate smaller cpumasks when nr_cpu_ids < NR_CPUS, when CONFIG_CPUMASK_OFFSTACK=y. This requires (2) to be completed.
4) Use cpumask_var_t for static cpumasks as well, or raw bitmaps if we really have to. The former will aave space for nr_cpu_ids << NR_CPUS.
5) Ban on-stack cpumasks (to ensure (1) doesn't get reverted) and cpumask assignment (for (3)) by making struct cpumask undefined when CONFIG_CPUMASK_OFFSTACK=y. This means (4) needs to be finished.
Between this and Ingo's tree, we achieve (1) and part of (2) and (4). Completing the work is expected by 2.6.30.
Note that we can't stop people creating bitmaps of NR_CPUS on the stack and using to_cpumask() on them. But at least it should stand out.
Cheers,
Rusty.
The following changes since commit 6a94cb73064c952255336cc57731904174b2c58f:
Linus Torvalds (1):
Merge branch 'for-linus' of git://oss.sgi.com/xfs/xfs
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/rusty/linux-2.6-cpumask.git master
Li Zefan (1):
cpumask: fix bogus kernel-doc
Mike Travis (4):
cpumask: Add alloc_cpumask_var_node()
cpumask: documentation for cpumask_var_t
cpumask: add sysfs displays for configured and disabled cpu maps
sysfs: add documentation to cputopology.txt for system cpumasks
Rusty Russell (64):
cpumask: centralize cpu_online_map and cpu_possible_map
cpumask: change cpumask_scnprintf, cpumask_parse_user, cpulist_parse, and cpulist_scnprintf to take pointers.
cpumask: make irq_set_affinity() take a const struct cpumask
cpumask: convert struct clock_event_device to cpumask pointers.
cpumask: Add CONFIG_CPUMASK_OFFSTACK
cpumask: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: Use all NR_CPUS bits unless CONFIG_CPUMASK_OFFSTACK
cpumask: x86: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: sparc: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: sh: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: powerpc: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: IA64: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: Mips: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: alpha: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: cpu_coregroup_mask(): x86
cpumask: cpu_coregroup_mask(): sparc
cpumask: cpu_coregroup_mask(): s390
cpumask: Replace cpu_coregroup_map with cpu_coregroup_mask
Merge branch 'master' of git://git.kernel.org/.../torvalds/linux-2.6
cpumask: make CONFIG_NR_CPUS always valid.
bitmap: test for constant as well as small size for inline versions
bitmap: fix seq_bitmap and seq_cpumask to take const pointer
cpumask: switch over to cpu_online/possible/active/present_mask: core
cpumask: make cpumask.h eat its own dogfood.
cpumask: make set_cpu_*/init_cpu_* out-of-line
cpumask: smp_call_function_many()
cpumask: arch_send_call_function_ipi_mask: core
cpumask: use for_each_online_cpu() in drivers/infiniband/hw/ehca/ehca_irq.c
cpumask: use new cpumask API in drivers/infiniband/hw/ehca
cpumask: use new cpumask API in drivers/infiniband/hw/ipath
cpumask: Use nr_cpu_ids in seq_cpumask
Merge branch 'master' of git://git.kernel.org/.../torvalds/linux-2.6
cpumask: Remove IA64 definition of total_cpus now it's in core code
percpu: fix percpu accessors to potentially !cpu_possible() cpus: pnpbios
percpu: fix percpu accessors to potentially !cpu_possible() cpus: m32r
cpumask: prepare for iterators to only go to nr_cpu_ids/nr_cpumask_bits.: core
cpumask: Use accessors code in core
parisc: remove gratuitous cpu_online_map declaration.
avr32: define __fls
blackfin: define __fls
m68k: define __fls
m68knommu: define __fls
bitmap: find_last_bit()
cpumask: Use find_last_bit()
cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): sparc
cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): s390
cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): powerpc
cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): ia64
cpumask: convert kernel trace functions
cpumask: convert kernel trace functions further
cpumask: remove any_online_cpu() users: kernel/
cpumask: remove any_online_cpu() users: mm/
cpumask: convert kernel/compat.c
cpumask: convert kernel/workqueue.c
cpumask: convert kernel time functions
cpumask: convert kernel/irq
cpumask: convert RCU implementations
cpumask: convert kernel/profile.c
cpumask: convert kernel/cpu.c
cpumask: convert rest of files in kernel/
cpumask: convert mm/
cpumask: replace for_each_cpu_mask_nr with for_each_cpu in kernel/time/
cpumask: zero extra bits in alloc_cpumask_var_node
cpumask: CONFIG_DISABLE_OBSOLETE_CPUMASK_FUNCTIONS
Documentation/cputopology.txt | 48 ++++ arch/alpha/include/asm/smp.h | 1 - arch/alpha/include/asm/topology.h | 17 ++ arch/alpha/kernel/irq.c | 5 +- arch/alpha/kernel/process.c | 2 + arch/alpha/kernel/setup.c | 5 + arch/alpha/kernel/smp.c | 7 +- arch/alpha/kernel/sys_dp264.c | 8 +- arch/alpha/kernel/sys_titan.c | 4 +- arch/arm/common/gic.c | 4 +- arch/arm/kernel/irq.c | 2 +- arch/arm/kernel/smp.c | 10 - arch/arm/mach-at91/at91rm9200_time.c | 3 +- arch/arm/mach-at91/at91sam926x_time.c | 2 +- arch/arm/mach-davinci/time.c | 2 +- arch/arm/mach-imx/time.c | 2 +- arch/arm/mach-ixp4xx/common.c | 2 +- arch/arm/mach-msm/timer.c | 2 +- arch/arm/mach-ns9xxx/time-ns9360.c | 2 +- arch/arm/mach-omap1/time.c | 2 +- arch/arm/mach-omap1/timer32k.c | 2 +- arch/arm/mach-omap2/timer-gp.c | 2 +- arch/arm/mach-pxa/time.c | 2 +- arch/arm/mach-realview/core.c | 2 +- arch/arm/mach-realview/localtimer.c | 4 +- arch/arm/mach-sa1100/time.c | 2 +- arch/arm/mach-versatile/core.c | 2 +- arch/arm/oprofile/op_model_mpcore.c | 4 +- arch/arm/plat-mxc/time.c | 2 +- arch/arm/plat-orion/time.c | 2 +- arch/avr32/include/asm/bitops.h | 5 + arch/avr32/kernel/time.c | 2 +- arch/blackfin/include/asm/bitops.h | 1 + arch/blackfin/kernel/time-ts.c | 2 +- arch/cris/arch-v32/kernel/irq.c | 4 +- arch/cris/arch-v32/kernel/smp.c | 4 - arch/cris/include/asm/smp.h | 1 - arch/ia64/hp/sim/hpsim_irq.c | 2 +- arch/ia64/include/asm/smp.h | 1 - arch/ia64/include/asm/topology.h | 9 +- arch/ia64/kernel/acpi.c | 3 +- arch/ia64/kernel/iosapic.c | 35 ++-- arch/ia64/kernel/irq.c | 9 +- arch/ia64/kernel/msi_ia64.c | 12 +- arch/ia64/kernel/smpboot.c | 10 +- arch/ia64/kernel/topology.c | 2 +- arch/ia64/sn/kernel/irq.c | 6 +- arch/ia64/sn/kernel/msi_sn.c | 7 +- arch/ia64/sn/kernel/sn2/sn_hwperf.c | 27 +-- arch/m32r/Kconfig | 1 + arch/m32r/kernel/smpboot.c | 8 +- arch/m68knommu/include/asm/bitops.h | 1 + arch/m68knommu/platform/coldfire/pit.c | 2 +- arch/mips/include/asm/irq.h | 3 +- arch/mips/include/asm/mach-ip27/topology.h | 4 +- arch/mips/include/asm/smp.h | 3 - arch/mips/jazz/irq.c | 2 +- arch/mips/kernel/cevt-bcm1480.c | 4 +- arch/mips/kernel/cevt-ds1287.c | 2 +- arch/mips/kernel/cevt-gt641xx.c | 2 +- arch/mips/kernel/cevt-r4k.c | 2 +- arch/mips/kernel/cevt-sb1250.c | 4 +- arch/mips/kernel/cevt-smtc.c | 2 +- arch/mips/kernel/cevt-txx9.c | 2 +- arch/mips/kernel/i8253.c | 2 +- arch/mips/kernel/irq-gic.c | 6 +- arch/mips/kernel/smp-cmp.c | 6 +- arch/mips/kernel/smp-mt.c | 2 +- arch/mips/kernel/smp.c | 7 +- arch/mips/kernel/smtc.c | 6 +- arch/mips/mti-malta/malta-smtc.c | 6 +- arch/mips/nxp/pnx8550/common/time.c | 1 + arch/mips/pmc-sierra/yosemite/smp.c | 6 +- arch/mips/sgi-ip27/ip27-smp.c | 2 +- arch/mips/sgi-ip27/ip27-timer.c | 2 +- arch/mips/sibyte/bcm1480/irq.c | 8 +- arch/mips/sibyte/bcm1480/smp.c | 8 +- arch/mips/sibyte/sb1250/irq.c | 8 +- arch/mips/sibyte/sb1250/smp.c | 8 +- arch/mips/sni/time.c | 2 +- arch/parisc/Kconfig | 1 + arch/parisc/include/asm/smp.h | 2 - arch/parisc/kernel/irq.c | 6 +- arch/parisc/kernel/smp.c | 15 -- arch/powerpc/include/asm/topology.h | 12 +- arch/powerpc/kernel/irq.c | 2 +- arch/powerpc/kernel/smp.c | 4 - arch/powerpc/kernel/time.c | 2 +- arch/powerpc/platforms/cell/spu_priv1_mmio.c | 6 +- arch/powerpc/platforms/cell/spufs/sched.c | 4 +- arch/powerpc/platforms/pseries/xics.c | 4 +- arch/powerpc/sysdev/mpic.c | 4 +- arch/powerpc/sysdev/mpic.h | 2 +- arch/s390/Kconfig | 1 + arch/s390/include/asm/topology.h | 2 + arch/s390/kernel/smp.c | 6 - arch/s390/kernel/time.c | 2 +- arch/s390/kernel/topology.c | 5 + arch/sh/include/asm/smp.h | 2 +- arch/sh/include/asm/topology.h | 1 + arch/sh/kernel/smp.c | 10 +- arch/sh/kernel/timers/timer-broadcast.c | 2 +- arch/sh/kernel/timers/timer-tmu.c | 2 +- arch/sparc/include/asm/smp_32.h | 2 - arch/sparc/include/asm/topology_64.h | 13 +- arch/sparc/kernel/irq_64.c | 11 +- arch/sparc/kernel/of_device_64.c | 4 +- arch/sparc/kernel/pci_msi.c | 4 +- arch/sparc/kernel/smp_32.c | 6 +- arch/sparc/kernel/smp_64.c | 4 - arch/sparc/kernel/sparc_ksyms_32.c | 4 - arch/sparc/kernel/time_64.c | 2 +- arch/um/kernel/smp.c | 7 - arch/um/kernel/time.c | 2 +- arch/x86/include/asm/pci.h | 10 +- arch/x86/include/asm/topology.h | 36 ++- arch/x86/kernel/apic.c | 8 +- arch/x86/kernel/cpu/intel_cacheinfo.c | 4 +- arch/x86/kernel/hpet.c | 8 +- arch/x86/kernel/i8253.c | 2 +- arch/x86/kernel/io_apic.c | 78 ++++---- arch/x86/kernel/irq_32.c | 2 +- arch/x86/kernel/irq_64.c | 2 +- arch/x86/kernel/mfgpt_32.c | 2 +- arch/x86/kernel/setup_percpu.c | 10 +- arch/x86/kernel/smpboot.c | 17 +- arch/x86/kernel/vmiclock_32.c | 2 +- arch/x86/lguest/boot.c | 2 +- arch/x86/mach-voyager/voyager_smp.c | 7 - arch/x86/xen/time.c | 2 +- block/blk.h | 4 +- drivers/base/cpu.c | 46 ++++- drivers/base/node.c | 4 +- drivers/base/topology.c | 4 +- drivers/clocksource/tcb_clksrc.c | 2 +- drivers/infiniband/hw/ehca/ehca_irq.c | 17 +- drivers/infiniband/hw/ipath/ipath_file_ops.c | 8 +- drivers/parisc/iosapic.c | 7 +- drivers/pci/pci-sysfs.c | 4 +- drivers/pci/probe.c | 4 +- drivers/pnp/pnpbios/bioscalls.c | 2 +- drivers/xen/events.c | 6 +- fs/seq_file.c | 3 +- include/asm-generic/topology.h | 14 ++- include/asm-m32r/smp.h | 2 - include/asm-m68k/bitops.h | 5 + include/linux/bitmap.h | 35 ++-- include/linux/bitops.h | 13 +- include/linux/clockchips.h | 4 +- include/linux/cpumask.h | 295 +++++++++++++------------- include/linux/interrupt.h | 6 +- include/linux/irq.h | 3 +- include/linux/rcuclassic.h | 4 +- include/linux/seq_file.h | 7 +- include/linux/smp.h | 18 +- include/linux/stop_machine.h | 6 +- include/linux/threads.h | 16 +- include/linux/tick.h | 4 +- init/Kconfig | 9 + init/main.c | 13 +- kernel/compat.c | 49 +++-- kernel/cpu.c | 145 +++++++++---- kernel/cpuset.c | 4 +- kernel/irq/chip.c | 2 +- kernel/irq/manage.c | 31 ++-- kernel/irq/migration.c | 14 +- kernel/irq/proc.c | 59 ++++-- kernel/kexec.c | 2 +- kernel/power/poweroff.c | 2 +- kernel/profile.c | 38 +++-- kernel/rcuclassic.c | 32 ++-- kernel/rcupreempt.c | 19 +- kernel/rcutorture.c | 27 ++- kernel/sched.c | 10 +- kernel/sched_stats.h | 2 +- kernel/smp.c | 145 +++++-------- kernel/softirq.c | 2 +- kernel/softlockup.c | 10 +- kernel/stop_machine.c | 8 +- kernel/taskstats.c | 41 +++-- kernel/time/clockevents.c | 2 + kernel/time/clocksource.c | 9 +- kernel/time/tick-broadcast.c | 113 +++++----- kernel/time/tick-common.c | 18 +- kernel/trace/ring_buffer.c | 42 +++-- kernel/trace/trace.c | 68 ++++--- kernel/trace/trace.h | 2 +- kernel/trace/trace_boot.c | 2 +- kernel/trace/trace_functions_graph.c | 2 +- kernel/trace/trace_hw_branches.c | 6 +- kernel/trace/trace_power.c | 2 +- kernel/trace/trace_sysprof.c | 13 +- kernel/workqueue.c | 26 ++- lib/Kconfig | 15 ++ lib/Makefile | 1 + lib/cpumask.c | 62 +++++- lib/find_last_bit.c | 45 ++++ mm/pdflush.c | 16 ++- mm/slab.c | 2 +- mm/slub.c | 20 +- mm/vmscan.c | 4 +- mm/vmstat.c | 4 +- security/selinux/selinuxfs.c | 2 +- 203 files changed, 1393 insertions(+), 1056 deletions(-) create mode 100644 lib/find_last_bit.c
commit 98a79d6a50181ca1ecf7400eda01d5dc1bc0dbf0
Author: Rusty Russell <[email protected]> Date: Sat Dec 13 21:19:41 2008 +1030
cpumask: centralize cpu_online_map and cpu_possible_map
Impact: cleanup
Each SMP arch defines these themselves. Move them to a central location.
Twists:
1) Some archs (m32, parisc, s390) set possible_map to all 1, so we add a CONFIG_INIT_ALL_POSSIBLE for this rather than break them.
2) mips and sparc32 '#define cpu_possible_map phys_cpu_present_map'. Those archs simply have phys_cpu_present_map replaced everywhere.
3) Alpha defined cpu_possible_map to cpu_present_map; this is tricky so I just manipulate them both in sync.
4) IA64, cris and m32r have gratuitous 'extern cpumask_t cpu_possible_map' declarations.
commit 29c0177e6a4ac094302bed54a1d4bbb6b740a9ef
Author: Rusty Russell <[email protected]> Date: Sat Dec 13 21:20:25 2008 +1030
cpumask: change cpumask_scnprintf, cpumask_parse_user, cpulist_parse, and cpulist_scnprintf to take pointers.
Impact: change calling convention of existing cpumask APIs
Most cpumask functions started with cpus_: these have been replaced by cpumask_ ones which take struct cpumask pointers as expected.
These four functions don't have good replacement names; fortunately they're rarely used, so we just change them over.
commit 0de26520c7cabf36e1de090ea8092f011a6106ce
Author: Rusty Russell <[email protected]> Date: Sat Dec 13 21:20:26 2008 +1030
cpumask: make irq_set_affinity() take a const struct cpumask
Impact: change existing irq_chip API
Not much point with gentle transition here: the struct irq_chip's setaffinity method signature needs to change.
Fortunately, not widely used code, but hits a few architectures.
Note: In irq_select_affinity() I save a temporary in by mangling irq_desc[irq].affinity directly. Ingo, does this break anything?
(Folded in fix from KOSAKI Motohiro)
commit 320ab2b0b1e08e3805a3e1084a2f0eb1938d5d67
Author: Rusty Russell <[email protected]> Date: Sat Dec 13 21:20:26 2008 +1030
cpumask: convert struct clock_event_device to cpumask pointers.
Impact: change calling convention of existing clock_event APIs
struct clock_event_timer's cpumask field gets changed to take pointer, as does the ->broadcast function.
Another single-patch change. For safety, we BUG_ON() in clockevents_register_device() if it's not set.
commit aab46da0520af9c99b7802cebe4f14a81ff39415
Author: Rusty Russell <[email protected]> Date: Sat Dec 13 21:20:27 2008 +1030
cpumask: Add CONFIG_CPUMASK_OFFSTACK
Impact: Add config option to enable code in cpumask.h
Currently it can be set if DEBUG_PER_CPU_MAPS, or set specifically by an arch.
commit f0b848ce6fe9062d504d997e9e97fe0f87d57217
Author: Rusty Russell <[email protected]> Date: Sat Dec 13 21:20:27 2008 +1030
cpumask: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
Impact: New APIs
The old node_to_cpumask/node_to_pcibus returned a cpumask_t: these return a pointer to a struct cpumask. Part of removing cpumasks from the stack.
This defines them in the generic non-NUMA case.
commit 7be7585393d311866653564fbcd10a3232773c0b
Author: Rusty Russell <[email protected]> Date: Sat Dec 13 21:20:28 2008 +1030
cpumask: Use all NR_CPUS bits unless CONFIG_CPUMASK_OFFSTACK
Impact: futureproof as we convert more code to new APIs
The old cpumask operators treat all NR_CPUS bits as relevent, the new ones use nr_cpumask_bits. For large NR_CPUS and small nr_cpu_ids, this makes a difference.
However, mixing the two can cause problems with undefined bits. An arch which sets CONFIG_CPUMASK_OFFSTACK should have converted across to the new operators, so it's safe in that case.
(Thanks to Stephen Rothwell for bisecting the initial unused-bits bug, and Mike Travis for this solution).
commit 7b4967c532045a1983d6d4af5c69cc7c5109f62b
Author: Mike Travis <[email protected]> Date: Fri Dec 19 16:56:37 2008 +1030
cpumask: Add alloc_cpumask_var_node()
Impact: New API
This will be needed in x86 code to allocate the domain and old_domain cpumasks on the same node as where the containing irq_cfg struct is allocated.
(Also fixes double-dump_stack on rare CONFIG_DEBUG_PER_CPU_MAPS case)
commit ec26b805879c7e77865b39ee91b737985e80006d
Author: Mike Travis <[email protected]> Date: Fri Dec 19 16:56:52 2008 +1030
cpumask: documentation for cpumask_var_t
Impact: New kerneldoc comments
Additional documentation added to all the alloc_cpumask and free_cpumask functions.
commit e057d7aea9d8f2a46cd440d8bfb72245d4e72d79
Author: Mike Travis <[email protected]> Date: Mon Dec 15 20:26:48 2008 -0800
cpumask: add sysfs displays for configured and disabled cpu maps
Impact: add new sysfs files.
Add sysfs files "kernel_max" and "offline" to display the max CPU index allowed (NR_CPUS-1), and the map of cpus that are offline.
Cpus can be offlined via HOTPLUG, disabled by the BIOS ACPI tables, or if they exceed the number of cpus allowed by the NR_CPUS config option, or the "maxcpus=NUM" kernel start parameter.
The "possible_cpus=NUM" parameter can also extend the number of possible cpus allowed, in which case the cpus not present at startup will be in the offline state. (These cpus can be HOTPLUGGED ON after system startup [pending a follow-on patch to provide the capability via the /sys/devices/sys/cpu/cpuN/online mechanism to bring them online.])
By design, the "offlined cpus > possible cpus" display will always use the following formats:
* all possible cpus online: "x$" or "x-y$"
* some possible cpus offline: ".*,x$" or ".*,x-y$"
where:
x == number of possible cpus (nr_cpu_ids); and y == number of cpus >= NR_CPUS or maxcpus (if y > x).
One use of this feature is for distros to select (or configure) the appropriate kernel to install for the resident system.
Notes:
* cpus offlined <= possible cpus will be printed for all architectures.
* cpus offlined > possible cpus will only be printed for arches that set 'total_cpus' [X86 only in this patch].
Based on tip/cpus4096 + .../rusty/linux-2.6-for-ingo.git/master +
x86-only-patches sent 12/15.
commit d62720ade82c5e5b8f9585e5ed02c89573ebf111
Author: Mike Travis <[email protected]> Date: Wed Dec 17 14:14:30 2008 -0800
sysfs: add documentation to cputopology.txt for system cpumasks
Add information to cputopology.txt explaining the output of various system cpumask's.
commit 393d68fb9929817cde7ab31c82d66fcb28ad35fc
Author: Rusty Russell <[email protected]> Date: Fri Dec 26 22:23:38 2008 +1030
cpumask: x86: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
Impact: New APIs
The old node_to_cpumask/node_to_pcibus returned a cpumask_t: these return a pointer to a struct cpumask. Part of removing cpumasks from the stack.
Also makes __pcibus_to_node take a const pointer.
commit 96d76a74870d5f11ce2abdd09a8dcdc401d714d1
Author: Rusty Russell <[email protected]> Date: Fri Dec 26 22:23:38 2008 +1030
cpumask: sparc: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
Impact: New APIs
The old node_to_cpumask/node_to_pcibus returned a cpumask_t: these return a pointer to a struct cpumask. Part of removing cpumasks from the stack.
commit 7479a2939df4957ba794cce814379b6d10914bdc
Author: Rusty Russell <[email protected]> Date: Fri Dec 26 22:23:39 2008 +1030
cpumask: sh: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
Impact: New APIs
The old node_to_cpumask/node_to_pcibus returned a cpumask_t: these return a pointer to a struct cpumask. Part of removing cpumasks from the stack.
commit 86c6f274f52c3e991d429869780945c0790e7b65
Author: Rusty Russell <[email protected]> Date: Fri Dec 26 22:23:39 2008 +1030
cpumask: powerpc: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
Impact: New APIs
The old node_to_cpumask/node_to_pcibus returned a cpumask_t: these return a pointer to a struct cpumask. Part of removing cpumasks from the stack.
(Also replaces powerpc internal uses of node_to_cpumask).
commit fbb776c3ca4501d5a2821bf1e9bceefcaec7ae47
Author: Rusty Russell <[email protected]> Date: Fri Dec 26 22:23:40 2008 +1030
cpumask: IA64: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
Impact: New APIs
The old node_to_cpumask/node_to_pcibus returned a cpumask_t: these return a pointer to a struct cpumask. Part of removing cpumasks from the stack.
We can also use the new for_each_cpu_and() to avoid a temporary cpumask, and a gratuitous test in sn_topology_show.
(Includes fix from KOSAKI Motohiro <[email protected]>)
commit b4a2f916a8326065816a0743dd1b0ca2ffd18f5f
Author: Rusty Russell <[email protected]> Date: Fri Dec 26 22:23:40 2008 +1030
cpumask: Mips: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
Impact: New APIs
The old node_to_cpumask/node_to_pcibus returned a cpumask_t: these return a pointer to a struct cpumask. Part of removing cpumasks from the stack.
commit 2258a5bb1064351b552aceaff29393967d694fa3
Author: Rusty Russell <[email protected]> Date: Fri Dec 26 22:23:41 2008 +1030
cpumask: alpha: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
Impact: New APIs
The old node_to_cpumask/node_to_pcibus returned a cpumask_t: these return a pointer to a struct cpumask. Part of removing cpumasks from the stack.
I'm not sure the existing code even compiles, but new version is straightforward.
commit 030bb203e01db12e3f2866799f4f03a114d06349
Author: Rusty Russell <[email protected]> Date: Fri Dec 26 22:23:41 2008 +1030
cpumask: cpu_coregroup_mask(): x86
Impact: New API
Like cpu_coregroup_map, but returns a (const) pointer.
commit a0ae09b46a516f05ea76e3419ad43c46f52c1165
Author: Rusty Russell <[email protected]> Date: Fri Dec 26 22:23:42 2008 +1030
cpumask: cpu_coregroup_mask(): sparc
Like cpu_coregroup_map, but returns a (const) pointer.
Compile-tested on sparc64 (defconfig).
commit 9be3eec2c83848a1ca57ebad13c63c95d0df01e2
Author: Rusty Russell <[email protected]> Date: Fri Dec 26 22:23:42 2008 +1030
cpumask: cpu_coregroup_mask(): s390
Like cpu_coregroup_map, but returns a (const) pointer.
Compile-tested on s390 (defconfig).
commit be4d638c1597580ed2294d899d9f1a2cd10e462c
Author: Rusty Russell <[email protected]> Date: Fri Dec 26 22:23:43 2008 +1030
cpumask: Replace cpu_coregroup_map with cpu_coregroup_mask
cpu_coregroup_map returned a cpumask_t: it's going away.
(Note, the sched part of this patch won't apply meaningfully to the sched tree, but I'm posting it to show the goal).
commit 33edcf133ba93ecba2e4b6472e97b689895d805c
Merge: be4d638... 3c92ec8...
Author: Rusty Russell <[email protected]>
Date: Tue Dec 30 08:02:35 2008 +1030
Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6
commit 278d1ed65e25d80af7c3a112d707b3f70516ddb4
Author: Rusty Russell <[email protected]> Date: Tue Dec 30 09:05:12 2008 +1030
cpumask: make CONFIG_NR_CPUS always valid.
Impact: cleanup
Currently we have NR_CPUS, which is 1 on UP, and CONFIG_NR_CPUS on SMP. If we make CONFIG_NR_CPUS always valid (and always 1 on !SMP), we can skip the middleman.
This also allows us to find and check all the unaudited NR_CPUS usage as we prepare for v. large NR_CPUS.
To avoid breaking every arch, we cheat and do this for the moment in the header if the arch doesn't.
commit 4b0bc0bca83f3fb7cf920e2ec80684c15d2269c0
Author: Rusty Russell <[email protected]> Date: Tue Dec 30 09:05:13 2008 +1030
bitmap: test for constant as well as small size for inline versions
Impact: reduce text size
bitmap_zero et al have a fastpath for nbits <= BITS_PER_LONG, but this should really only apply where the nbits is known at compile time.
This only saves about 1200 bytes on an allyesconfig kernel, but with cpumasks going variable that number will increase.
text data bss dec hex filename
35327852 5035607 6782976 47146435 2cf65c3 vmlinux-before
35326640 5035607 6782976 47145223 2cf6107 vmlinux-after
commit cb78a0ce69fad2026825f957e24e2d9cda1ec9f1
Author: Rusty Russell <[email protected]> Date: Tue Dec 30 09:05:14 2008 +1030
bitmap: fix seq_bitmap and seq_cpumask to take const pointer
Impact: cleanup
seq_bitmap just calls bitmap_scnprintf on the bits: that arg can be const. Similarly, seq_cpumask just calls seq_bitmap.
commit b3199c025d1646e25e7d1d640dd605db251dccf8
Author: Rusty Russell <[email protected]> Date: Tue Dec 30 09:05:14 2008 +1030
cpumask: switch over to cpu_online/possible/active/present_mask: core
Impact: cleanup
This implements the obsolescent cpu_online_map in terms of cpu_online_mask, rather than the other way around. Same for the other maps.
The documentation comments are also updated to refer to _mask rather than _map.
commit ae7a47e72e1a0b5e2b46d1596bc2c22942a73023
Author: Rusty Russell <[email protected]> Date: Tue Dec 30 09:05:15 2008 +1030
cpumask: make cpumask.h eat its own dogfood.
Changes:
1) cpumask_t to struct cpumask,
2) cpus_weight_nr to cpumask_weight,
3) cpu_isset to cpumask_test_cpu,
4) ->bits to cpumask_bits()
5) cpu_*_map to cpu_*_mask.
6) for_each_cpu_mask_nr to for_each_cpu
commit 3fa41520696fec2815e2d88fbcccdda77ba4d693
Author: Rusty Russell <[email protected]> Date: Tue Dec 30 09:05:16 2008 +1030
cpumask: make set_cpu_*/init_cpu_* out-of-line
They're only for use in boot/cpu hotplug code anyway, and this avoids the use of deprecated cpu_*_map.
Stephen Rothwell points out that gcc 4.2.4 (on powerpc at least) didn't like the cast away of const anyway:
include/linux/cpumask.h: In function 'set_cpu_possible':
include/linux/cpumask.h:1052: warning: passing argument 2 of 'cpumask_set_cpu' discards qualifiers from pointer target type
So this kills two birds with one stone.
commit 54b11e6d57a10aa9d0009efd93873e17bffd5d30
Author: Rusty Russell <[email protected]> Date: Tue Dec 30 09:05:16 2008 +1030
cpumask: smp_call_function_many()
Impact: Implementation change to remove cpumask_t from stack.
Actually change smp_call_function_mask() to smp_call_function_many(). We avoid cpumasks on the stack in this version.
(S390 has its own version, but that's going away apparently).
We have to do some dancing to figure out if 0 or 1 other cpus are in the mask supplied and the online mask without allocating a tmp cpumask. It's still fairly cheap.
We allocate the cpumask at the end of the call_function_data
structure: if allocation fails we fallback to smp_call_function_single rather than using the baroque quiescing code (which needs a cpumask on stack).
(Thanks to Hiroshi Shimamoto for spotting several bugs in previous versions!)
commit ce47d974f71af26d00832e83a43ac79bec272d99
Author: Rusty Russell <[email protected]> Date: Tue Dec 30 09:05:17 2008 +1030
cpumask: arch_send_call_function_ipi_mask: core
Impact: new API to reduce stack usage
We're weaning the core code off handing cpumask's around on-stack. This introduces arch_send_call_function_ipi_mask().
commit 259c4ddd00237e5072921afa15a900839643fd98
Author: Rusty Russell <[email protected]> Date: Tue Dec 30 09:05:17 2008 +1030
cpumask: use for_each_online_cpu() in drivers/infiniband/hw/ehca/ehca_irq.c
Impact: cleanup
In future, accessing cpu numbers beyond nr_cpu_ids (the runtime limit) will be undefined. We can avoid future problems by using for_each_online_cpu() here.
commit b29179c3d32021d79c11ece7199a1da41d31b1b7
Author: Rusty Russell <[email protected]> Date: Tue Dec 30 09:05:18 2008 +1030
cpumask: use new cpumask API in drivers/infiniband/hw/ehca
Impact: cleanup
We're moving from handing around cpumask_t's to handing around struct cpumask *'s. cpus_*, cpumask_t and cpu_*_map are deprecated: convert to cpumask_*, cpu_*_mask.
commit cbe31f02f5b5536f17dd978118e25052af528071
Author: Rusty Russell <[email protected]> Date: Tue Dec 30 09:05:18 2008 +1030
cpumask: use new cpumask API in drivers/infiniband/hw/ipath
Impact: cleanup
We're moving from handing around cpumask_t's to handing around struct cpumask *'s. cpus_*, cpumask_t and cpu_*_map are deprecated: convert to cpumask_*, cpu_*_mask.
commit e12f0102ac81d660c9f801d0a0e10ccf4537a9de
Author: Rusty Russell <[email protected]> Date: Tue Dec 30 09:05:19 2008 +1030
cpumask: Use nr_cpu_ids in seq_cpumask
Impact: cleanup, futureproof
nr_cpu_ids is the (badly named) runtime limit on possible CPU numbers; ie. the variable version of NR_CPUS.
With the new cpumask operators, only bits less than this are defined. So we should use it everywhere, rather than NR_CPUS. Eventually this will make it possible to allocate cpumasks of the minimal length at runtime.
commit 2ca1a615835d9f4990f42102ab1f2ef434e7e89c
Merge: e12f010... 6a94cb7...
Author: Rusty Russell <[email protected]>
Date: Wed Dec 31 23:05:57 2008 +1030
Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6
Conflicts:
arch/x86/kernel/io_apic.c
commit f320786063a9d1f885d2cf34ab44aa69c1d88f43
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:13 2009 +1030
cpumask: Remove IA64 definition of total_cpus now it's in core code
Impact: fix IA64 compile
Fortunately, they have exactly the same semantics.
commit e9690a6e4b1615cb0102e425e04b7ce29e7858e2
Author: Li Zefan <[email protected]> Date: Wed Dec 31 16:45:50 2008 +0800
cpumask: fix bogus kernel-doc
Impact: fix kernel-doc
alloc_bootmem_cpumask_var() returns avoid.
commit 6aaa8ce523c7ce954b81b8c0b3e32c8be599af8d
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:14 2009 +1030
percpu: fix percpu accessors to potentially !cpu_possible() cpus: pnpbios
Impact: CPU iterator bugfixes
Percpu areas are only allocated for possible cpus. In general, you shouldn't access random cpu's percpu areas.
commit 9e2f913df70b378379a358a44e7d286f7b765e8e
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:14 2009 +1030
percpu: fix percpu accessors to potentially !cpu_possible() cpus: m32r
Impact: CPU iterator bugfixes
Percpu areas are only allocated for possible cpus. In general, you shouldn't access random cpu's percpu areas.
commit 4f4b6c1a94a8735bbdc030a2911cf395495645b6
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:15 2009 +1030
cpumask: prepare for iterators to only go to nr_cpu_ids/nr_cpumask_bits.: core
Impact: cleanup
In future, all cpumask ops will only be valid (in general) for bit numbers < nr_cpu_ids. So use that instead of NR_CPUS in iterators and other comparisons.
This is always safe: no cpu number can be >= nr_cpu_ids, and nr_cpu_ids is initialized to NR_CPUS at boot.
commit 915441b601e6662e79f6c958e7be307967a96977
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:15 2009 +1030
cpumask: Use accessors code in core
Impact: use new API
cpu_*_map are going away in favour of cpu_*_mask, but const pointers. So we have accessors where we really do want to frob them. Archs will also need the (trivial) conversion before we can finally remove cpu_*_map.
commit 165ac433fa3f01ba99b29972f3adc283d03b0f17
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:16 2009 +1030
parisc: remove gratuitous cpu_online_map declaration.
This is defined in linux/cpumask.h (included in this file already), and this is now defined differently.
commit 96b8d4c19d797200b973caab57ca842531184c13
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:16 2009 +1030
avr32: define __fls
Like fls, but can't be handed 0 and returns the bit number.
(I broke this arch in linux-next by using __fls in generic code).
commit ccec25ff69d5f48c7a088c16fe2dc7e11d9e87fe
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:17 2009 +1030
blackfin: define __fls
Like fls, but can't be handed 0 and returns the bit number.
(I broke this arch in linux-next by using __fls in generic code).
commit 434ae514c23047db87a8bbf39cebc9e1767aea44
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:18 2009 +1030
m68k: define __fls
Like fls, but can't be handed 0 and returns the bit number.
(I broke this arch in linux-next by using __fls in generic code).
commit 0db5d3d2f58804edb394e8008c7d9744110338a2
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:18 2009 +1030
m68knommu: define __fls
Like fls, but can't be handed 0 and returns the bit number.
(I broke this arch in linux-next by using __fls in generic code).
commit ab53d472e785e51fdfc08fc1d66252c1153e6c0f
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:19 2009 +1030
bitmap: find_last_bit()
Impact: New API
As the name suggests. For the moment everyone uses the generic one.
commit e0c0ba736547e81c4f986ce192307c549d214167
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:19 2009 +1030
cpumask: Use find_last_bit()
Impact: cleanup
There's one obvious place to use it: to find the highest possible cpu.
commit 78fd744f827586615da5b387fa9f0af1888601b6
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:20 2009 +1030
cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): sparc
Impact: New API
The old topology_core_siblings() and topology_thread_siblings() return a cpumask_t; these new ones return a (const) struct cpumask *.
commit 2bb23a63f22f0e2d91fee93ff5ca9c29e180b146
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:20 2009 +1030
cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): s390
Impact: New API
The old topology_core_siblings() and topology_thread_siblings() return a cpumask_t; these new ones return a (const) struct cpumask *.
commit 9150641dd17fe9e213ab3391c8ebfc228daa2d9d
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:21 2009 +1030
cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): powerpc
Impact: New API
The old topology_core_siblings() and topology_thread_siblings() return a cpumask_t; these new ones return a (const) struct cpumask *.
commit 333af15341b2f6cd813c054e1b441d7b6d8e9318
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:21 2009 +1030
cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): ia64
Impact: New API
The old topology_core_siblings() and topology_thread_siblings() return a cpumask_t; these new ones return a (const) struct cpumask *.
commit 9e01c1b74c9531e301c900edaa92a99fcb7738f2
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:22 2009 +1030
cpumask: convert kernel trace functions
Impact: Reduce future memory usage, use new cpumask API.
(Eventually, cpumask_var_t will be allocated based on nr_cpu_ids, not NR_CPUS).
Convert kernel trace functions to use struct cpumask API:
1) Use cpumask_copy/cpumask_test_cpu/for_each_cpu.
2) Use cpumask_var_t and alloc_cpumask_var/free_cpumask_var everywhere.
3) Use on_each_cpu instead of playing with current->cpus_allowed.
commit 4462344ee9ea9224d026801b877887f2f39774a3
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:23 2009 +1030
cpumask: convert kernel trace functions further
Impact: Reduce future memory usage, use new cpumask API.
Since the last patch was created and acked, more old cpumask users slipped into kernel/trace.
Mostly trivial conversions, except struct trace_iterator's "started" member becomes a cpumask_var_t.
commit f1fc057c79cb2d27602fb3ad08a031f13459ef27
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:23 2009 +1030
cpumask: remove any_online_cpu() users: kernel/
Impact: Remove obsolete API usage
any_online_cpu() is a good name, but it takes a cpumask_t, not a pointer.
There are several places where any_online_cpu() doesn't really want a mask arg at all. Replace all callers with cpumask_any() and cpumask_any_and().
commit 3e597945384dee1457240158eb81e3afb90b68c2
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:24 2009 +1030
cpumask: remove any_online_cpu() users: mm/
Impact: Remove obsolete API usage
any_online_cpu() is a good name, but it takes a cpumask_t, not a pointer.
There are several places where any_online_cpu() doesn't really want a mask arg at all. Replace all callers with cpumask_any() and cpumask_any_and().
commit a45185d2d7108b01b90b9e0293377be4d6346dde
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:24 2009 +1030
cpumask: convert kernel/compat.c
Impact: Reduce stack usage, use new cpumask API.
Straightforward conversion; cpumasks' size is given by cpumask_size() (now a variable rather than fixed) and on-stack cpu masks use cpumask_var_t.
commit e7577c50f2fb2d1c167e2c04a4b4c2cc042acb82
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:25 2009 +1030
cpumask: convert kernel/workqueue.c
Impact: Reduce memory usage, use new cpumask API.
cpu_populated_map becomes a cpumask_var_t, and cpu_singlethread_map is simply a cpumask pointer: it's simply the cpumask containing the first possible CPU anyway.
commit 6b954823c24f04ed026a8517f6bab5abda279db8
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:25 2009 +1030
cpumask: convert kernel time functions
Impact: Use new APIs
Convert kernel/time functions to use struct cpumask *.
Note the ugly bitmap declarations in tick-broadcast.c. These should be cpumask_var_t, but there was no obvious initialization function to put the alloc_cpumask_var() calls in. This was safe.
(Eventually 'struct cpumask' will be undefined for CONFIG_CPUMASK_OFFSTACK, so we use a bitmap here to show we really mean it).
commit d036e67b40f52bdd95392390108defbac7e53837
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:26 2009 +1030
cpumask: convert kernel/irq
Impact: Reduce stack usage, use new cpumask API. ALPHA mod!
Main change is that irq_default_affinity becomes a cpumask_var_t, so treat it as a pointer (this effects alpha).
commit bd232f97b30f6bb630efa136a777647545db3039
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:26 2009 +1030
cpumask: convert RCU implementations
Impact: use new cpumask API.
rcu_ctrlblk contains a cpumask, and it's highly optimized so I don't want a cpumask_var_t (ie. a pointer) for the CONFIG_CPUMASK_OFFSTACK case. It could use a dangling bitmap, and be allocated in __rcu_init to save memory, but for the moment we use a bitmap.
(Eventually 'struct cpumask' will be undefined for CONFIG_CPUMASK_OFFSTACK, so we use a bitmap here to show we really mean it).
We remove on-stack cpumasks, using cpumask_var_t for rcu_torture_shuffle_tasks() and for_each_cpu_and in force_quiescent_state().
commit c309b917cab55799ea489d7b5f1b77025d9f8462
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:27 2009 +1030
cpumask: convert kernel/profile.c
Impact: Reduce kernel memory usage, use new cpumask API.
Avoid a static cpumask_t for prof_cpu_mask, and an on-stack cpumask_t in prof_cpu_mask_write_proc. Both become cpumask_var_t.
prof_cpu_mask is only allocated when profiling is on, but the NULL checks are optimized out by gcc for the !CPUMASK_OFFSTACK case.
Also removed some strange and unnecessary casts.
commit e0b582ec56f1a1d8b30ebf340a7b91fb09f26c8c
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:28 2009 +1030
cpumask: convert kernel/cpu.c
Impact: Reduce kernel stack and memory usage, use new cpumask API.
Use cpumask_var_t for take_cpu_down() stack var, and frozen_cpus.
Note that notify_cpu_starting() can be called before core_initcall allocates frozen_cpus, but the NULL check is optimized out by gcc for the CONFIG_CPUMASK_OFFSTACK=n case.
commit 41c7bb9588904eb060a95bcad47bd3804a1ece25
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:28 2009 +1030
cpumask: convert rest of files in kernel/
Impact: Reduce stack usage, use new cpumask API.
Mainly changing cpumask_t to 'struct cpumask' and similar simple API conversion. Two conversions worth mentioning:
1) we use cpumask_any_but to avoid a temporary in kernel/softlockup.c,
2) Use cpumask_var_t in taskstats_user_cmd().
commit 174596a0b9f21e8844d70566a6bb29bf48a87750
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:29 2009 +1030
cpumask: convert mm/
Impact: Use new API
Convert kernel mm functions to use struct cpumask.
We skip include/linux/percpu.h and mm/allocpercpu.c, which are in flux.
commit 5db0e1e9e0f30f160b832a0b5cd1131954bf4f6e
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:29 2009 +1030
cpumask: replace for_each_cpu_mask_nr with for_each_cpu in kernel/time/
Impact: cleanup
Simple replacement, now the _nr is redundant.
commit 2a53008033189ed09bfe241c6b33811ba4ce980d
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:30 2009 +1030
cpumask: zero extra bits in alloc_cpumask_var_node
Impact: extra safety checks during transition
When CONFIG_CPUMASKS_OFFSTACK is set, the new cpumask_ operators only use bits up to nr_cpu_ids, not NR_CPUS. Using the old cpus_ operators on these masks can mean accessing undefined bits.
After some discussion, Mike and I decided to err on the side of caution; we zero the "undefined" bits in alloc_cpumask_var_node() until all the old cpumask functions are removed.
commit 8c384cdee3e04d6194a2c2b192b624754f990835
Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:30 2009 +1030
cpumask: CONFIG_DISABLE_OBSOLETE_CPUMASK_FUNCTIONS
Impact: new debug CONFIG options
This helps find unconverted code. It currently breaks compile horribly, but we never wanted a flag day so that's expected.
Quoted 4 lines · Rusty Russell
OK, this is the bulk of the conversion to the new cpumask operators. The x86-specific parts (the most aggressive large-NR_CPUS arch) are going via Ingo's tree.
This gets lots of conflicts for me. Some of them look simple enough, but not all. io_apic.c gets lots of nasty conflicts, and it _looks_ like I should just pick the version of the file that I already have (because the only thing that comes in from that is yet another merge commit), but kernel/sched.c also gets conflicts in areas with FIXME's etc.
Rusty, Ingo, can you work this out? I pushed out my current tree.
Linus
Quoted 6 lines · Linus Torvalds
Quoted 4 lines · Rusty Russell
OK, this is the bulk of the conversion to the new cpumask operators. The x86-specific parts (the most aggressive large-NR_CPUS arch) are going via Ingo's tree.
Quoted 7 lines
This gets lots of conflicts for me. Some of them look simple enough, but not all.
io_apic.cgets lots of nasty conflicts, and it _looks_ like I should just pick the version of the file that I already have (because the only thing that comes in from that is yet another merge commit), butkernel/sched.calso gets conflicts in areas with FIXME's etc.Rusty, Ingo, can you work this out? I pushed out my current tree.
yes, we have those conflicts all resolved already in the second phase of the tip/cpus4096 changes: Mike did all those difficult conflict resolutions over the hollidays and i pulled it yesterday.
The end result looks nice as a tree but it is not fully cooked yet: since i pulled yesterday i found a couple of build and runtime test failures with Rusty's latest cpumask tree:
- architectures that have no __fls (8 out of 21) fail to build:
arch/cris
arch/frv
arch/h8300
arch/m32r
arch/m68k
arch/mn10300
arch/xtensa- there's a new circular locking lockdep splat in CPU hotplug tests when the var-cpumask code is enabled. (needs a handful of default-off options enabled in the .config)
So i didnt want to push the second phase to you until those known bugs are sorted out - i think we need some more time for that - a day or two at most.
Linus, would you like to pull that, despite the pending regressions? Can send a pull request right away.
Rusty, would it be fine with you if we did all the remaining bits via tip/cpus4096? It's your tree and your bits and we wanted to send our remaining bits after your tree went to Linus but the conflict resolutions from Mike are valuable so i think we should reconsider the ordering.
All the commits you sent to Linus in this pull request are already included in tip/cpus4096, the conflicts that Linus hit are all non-trivial and Mike resolved them correctly and it merges cleanly with Linus's latest tree:
earth4:~/tip> git pull
git://git.kernel.org/pub/scm/linux/kernel/git/rusty/linux-2.6-cpumask.git master From git://git.kernel.org/pub/scm/linux/kernel/git/rusty/linux-2.6-cpumask
* branch master -> FETCH_HEAD
Already up-to-date.
The pending diff is:
108 files changed, 1442 insertions(+), 979 deletions(-)
which is pretty OK and straightforward.
With that Rusty and Mike has done 99% of the cpumask conversions, so the most difficult phase of the conversion should be dealt with in .29. Cool stuff.
Ingo
Quoted 3 lines · Ingo Molnar
Linus, would you like to pull that, despite the pending regressions? Can send a pull request right away.
No, if there are known pending regressions, please get those fixed first.
Linus
[git pull] cpus4096 tree, part 3
Quoted 7 lines · Linus Torvalds
Quoted 3 lines · Ingo Molnar
Linus, would you like to pull that, despite the pending regressions? Can send a pull request right away.
No, if there are known pending regressions, please get those fixed first.
ok. The pending regressions are all fixed now, and i've just finished my standard tests on the latest tree and all the tests passed fine.
Please pull the cpus4096-for-linus-3 git tree from:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git cpus4096-for-linus-3
Thanks,
Ingo
------------------>
Cyrill Gorcunov (1):
x86: setup_per_cpu_areas() cleanup
Ingo Molnar (2):
cpumask: convert RCU implementations, fix
ia64: cpumask fix for is_affinity_mask_valid()
Li Zefan (1):
cpumask: fix bogus kernel-doc
Mike Travis (9):
cpumask: Add alloc_cpumask_var_node()
cpumask: documentation for cpumask_var_t
cpumask: add sysfs displays for configured and disabled cpu maps
sysfs: add documentation to cputopology.txt for system cpumasks
x86: enable cpus display of kernel_max and offlined cpus
sched: put back some stack hog changes that were undone in kernel/sched.c
x86: cleanup some remaining usages of NR_CPUS where s/b nr_cpu_ids
cpumask: use alloc_cpumask_var_node where appropriate
cpumask: fix compile error when CONFIG_NR_CPUS is not defined
Rusty Russell (63):
cpumask: x86: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: sparc: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: sh: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: powerpc: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: IA64: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: Mips: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: alpha: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: cpu_coregroup_mask(): x86
cpumask: cpu_coregroup_mask(): sparc
cpumask: cpu_coregroup_mask(): s390
cpumask: Replace cpu_coregroup_map with cpu_coregroup_mask
cpumask: make CONFIG_NR_CPUS always valid.
bitmap: test for constant as well as small size for inline versions
bitmap: fix seq_bitmap and seq_cpumask to take const pointer
cpumask: switch over to cpu_online/possible/active/present_mask: core
cpumask: make cpumask.h eat its own dogfood.
cpumask: make set_cpu_*/init_cpu_* out-of-line
cpumask: smp_call_function_many()
cpumask: arch_send_call_function_ipi_mask: core
cpumask: use for_each_online_cpu() in drivers/infiniband/hw/ehca/ehca_irq.c
cpumask: use new cpumask API in drivers/infiniband/hw/ehca
cpumask: use new cpumask API in drivers/infiniband/hw/ipath
cpumask: Use nr_cpu_ids in seq_cpumask
cpumask: Remove IA64 definition of total_cpus now it's in core code
percpu: fix percpu accessors to potentially !cpu_possible() cpus: pnpbios
percpu: fix percpu accessors to potentially !cpu_possible() cpus: m32r
cpumask: prepare for iterators to only go to nr_cpu_ids/nr_cpumask_bits.: core
cpumask: Use accessors code in core
parisc: remove gratuitous cpu_online_map declaration.
avr32: define __fls
blackfin: define __fls
m68k: define __fls
m68knommu: define __fls
bitmap: find_last_bit()
cpumask: Use find_last_bit()
cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): sparc
cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): s390
cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): powerpc
cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): ia64
cpumask: convert kernel trace functions
cpumask: convert kernel trace functions further
cpumask: remove any_online_cpu() users: kernel/
cpumask: remove any_online_cpu() users: mm/
cpumask: convert kernel/compat.c
cpumask: convert kernel/workqueue.c
cpumask: convert kernel time functions
cpumask: convert kernel/irq
cpumask: convert RCU implementations
cpumask: convert kernel/profile.c
cpumask: convert kernel/cpu.c
cpumask: convert rest of files in kernel/
cpumask: convert mm/
cpumask: replace for_each_cpu_mask_nr with for_each_cpu in kernel/time/
cpumask: zero extra bits in alloc_cpumask_var_node
cpumask: CONFIG_DISABLE_OBSOLETE_CPUMASK_FUNCTIONS
cris: define __fls
frv: define __fls
h8300: define __fls
m32r: define __fls
mn10300: define __fls
xtensa: define __fls
x86: use cpumask_var_t in acpi/boot.c
cpumask: convert shared_cpu_map in acpi_processor* structs to cpumask_var_t
Sergio Luis (1):
x86: mark get_cpu_leaves() with __cpuinit annotation
Documentation/cputopology.txt | 48 ++++++ arch/alpha/include/asm/topology.h | 17 ++ arch/alpha/kernel/irq.c | 3 +- arch/alpha/kernel/setup.c | 5 + arch/avr32/include/asm/bitops.h | 5 + arch/blackfin/include/asm/bitops.h | 1 + arch/cris/include/asm/bitops.h | 1 + arch/h8300/include/asm/bitops.h | 1 + arch/ia64/include/asm/irq.h | 2 +- arch/ia64/include/asm/topology.h | 9 +- arch/ia64/kernel/acpi.c | 3 +- arch/ia64/kernel/iosapic.c | 23 ++-- arch/ia64/kernel/irq.c | 4 +- arch/ia64/sn/kernel/sn2/sn_hwperf.c | 27 ++-- arch/m32r/kernel/smpboot.c | 2 +- arch/m68knommu/include/asm/bitops.h | 1 + arch/mips/include/asm/mach-ip27/topology.h | 4 +- arch/parisc/include/asm/smp.h | 2 - arch/powerpc/include/asm/topology.h | 12 +- arch/powerpc/platforms/cell/spu_priv1_mmio.c | 6 +- arch/powerpc/platforms/cell/spufs/sched.c | 4 +- arch/s390/include/asm/topology.h | 2 + arch/s390/kernel/topology.c | 5 + arch/sh/include/asm/topology.h | 1 + arch/sparc/include/asm/topology_64.h | 13 +- arch/sparc/kernel/of_device_64.c | 2 +- arch/sparc/kernel/pci_msi.c | 2 +- arch/x86/include/asm/es7000/apic.h | 32 +---- arch/x86/include/asm/lguest.h | 2 +- arch/x86/include/asm/numaq/apic.h | 4 +- arch/x86/include/asm/pci.h | 10 +- arch/x86/include/asm/summit/apic.h | 42 +---- arch/x86/include/asm/topology.h | 36 +++-- arch/x86/kernel/acpi/boot.c | 31 +++- arch/x86/kernel/apic.c | 4 +- arch/x86/kernel/cpu/common.c | 2 +- arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c | 28 +++- arch/x86/kernel/cpu/cpufreq/powernow-k7.c | 9 + arch/x86/kernel/cpu/cpufreq/powernow-k8.c | 24 ++- arch/x86/kernel/cpu/intel_cacheinfo.c | 2 +- arch/x86/kernel/cpuid.c | 2 +- arch/x86/kernel/io_apic.c | 6 +- arch/x86/kernel/msr.c | 2 +- arch/x86/kernel/reboot.c | 4 +- arch/x86/kernel/setup_percpu.c | 33 ++--- arch/x86/kernel/smpboot.c | 15 ++- arch/x86/mach-voyager/voyager_smp.c | 7 +- block/blk.h | 4 +- drivers/acpi/processor_core.c | 14 ++- drivers/acpi/processor_perflib.c | 28 ++-- drivers/acpi/processor_throttling.c | 80 ++++++---- drivers/base/cpu.c | 44 +++++ drivers/infiniband/hw/ehca/ehca_irq.c | 17 +- drivers/infiniband/hw/ipath/ipath_file_ops.c | 8 +- drivers/pnp/pnpbios/bioscalls.c | 2 +- fs/seq_file.c | 3 +- include/acpi/processor.h | 4 +- include/asm-frv/bitops.h | 13 ++ include/asm-m32r/bitops.h | 1 + include/asm-m68k/bitops.h | 5 + include/asm-mn10300/bitops.h | 11 ++ include/asm-xtensa/bitops.h | 11 ++ include/linux/bitmap.h | 35 +++-- include/linux/bitops.h | 13 ++- include/linux/cpumask.h | 221 +++++++++++--------------- include/linux/interrupt.h | 2 +- include/linux/rcuclassic.h | 4 +- include/linux/seq_file.h | 7 +- include/linux/smp.h | 18 ++- include/linux/stop_machine.h | 6 +- include/linux/threads.h | 16 +- include/linux/tick.h | 4 +- init/main.c | 13 +- kernel/compat.c | 49 ++++--- kernel/cpu.c | 144 ++++++++++++------ kernel/irq/manage.c | 11 +- kernel/irq/proc.c | 34 +++-- kernel/kexec.c | 2 +- kernel/power/poweroff.c | 2 +- kernel/profile.c | 38 +++-- kernel/rcuclassic.c | 32 ++-- kernel/rcupreempt.c | 19 ++- kernel/rcutorture.c | 27 ++-- kernel/sched.c | 53 ++----- kernel/sched_rt.c | 3 +- kernel/smp.c | 145 +++++++----------- kernel/softirq.c | 2 +- kernel/softlockup.c | 10 +- kernel/stop_machine.c | 8 +- kernel/taskstats.c | 39 +++-- kernel/time/clocksource.c | 9 +- kernel/time/tick-broadcast.c | 115 +++++++------- kernel/time/tick-common.c | 6 +- kernel/trace/ring_buffer.c | 42 +++-- kernel/trace/trace.c | 72 ++++++--- kernel/trace/trace.h | 2 +- kernel/trace/trace_boot.c | 2 +- kernel/trace/trace_functions_graph.c | 2 +- kernel/trace/trace_hw_branches.c | 6 +- kernel/trace/trace_power.c | 2 +- kernel/trace/trace_sysprof.c | 13 +-- kernel/workqueue.c | 26 ++-- lib/Kconfig | 8 + lib/Makefile | 1 + lib/cpumask.c | 62 +++++++- lib/find_last_bit.c | 45 ++++++ mm/pdflush.c | 16 ++- mm/slab.c | 2 +- mm/slub.c | 20 ++- mm/vmscan.c | 4 +- mm/vmstat.c | 4 +- security/selinux/selinuxfs.c | 2 +- 112 files changed, 1285 insertions(+), 878 deletions(-) create mode 100644 lib/find_last_bit.c
Documentation/cputopology.txt +48 -0
@@ -31,3 +31,51 @@ not defined by include/asm-XXX/topology.h: 2) core_id: 0 3) thread_siblings: just the given CPU 4) core_siblings: just the given CPU + +Additionally, cpu topology information is provided under +/sys/devices/system/cpu and includes these files. The internal +source for the output is in brackets ("[]"). + + kernel_max: the maximum cpu index allowed by the kernel configuration. + [NR_CPUS-1] + + offline: cpus that are not online because they have been + HOTPLUGGED off (see cpu-hotplug.txt) or exceed the limit + of cpus allowed by the kernel configuration (kernel_max + above). [~cpu_online_mask + cpus >= NR_CPUS] + + online: cpus that are online and being scheduled [cpu_online_mask] + + possible: cpus that have been allocated resources and can be + brought online if they are present. [cpu_possible_mask] + + present: cpus that have been identified as being present in the + system. [cpu_present_mask] + +The format for the above output is compatible with cpulist_parse() +[see <linux/cpumask.h>]. Some examples follow. + +In this example, there are 64 cpus in the system but cpus 32-63 exceed +the kernel max which is limited to 0..31 by the NR_CPUS config option +being 32. Note also that cpus 2 and 4-31 are not online but could be +brought online as they are both present and possible. + + kernel_max: 31 + offline: 2,4-31,32-63 + online: 0-1,3 + possible: 0-31 + present: 0-31 + +In this example, the NR_CPUS config option is 128, but the kernel was +started with possible_cpus=144. There are 4 cpus in the system and cpu2 +was manually taken offline (and is the only cpu that can be brought +online.) + + kernel_max: 127 + offline: 2,4-127,128-143 + online: 0-1,3 + possible: 0-127 + present: 0-3 + +See cpu-hotplug.txt for the possible_cpus=NUM kernel start parameter +as well as more information on the various cpumask's.
arch/alpha/include/asm/topology.h +17 -0
@@ -39,7 +39,24 @@ static inline cpumask_t node_to_cpumask(int node) return node_cpu_mask; } +extern struct cpumask node_to_cpumask_map[]; +/* FIXME: This is dumb, recalculating every time. But simple. */ +static const struct cpumask *cpumask_of_node (int node) +{ +int cpu; + +cpumask_clear (&node_to_cpumask_map[node]); + +for_each_online_cpu (cpu) { +if (cpu_to_node (cpu) == node) +cpumask_set_cpu (cpu, node_to_cpumask_map[node]); + } + +return &node_to_cpumask_map[node]; +} +#define pcibus_to_cpumask (bus) (cpu_online_map) +#define cpumask_of_pcibus (bus) (cpu_online_mask)#endif /* !CONFIG_NUMA */ # include <asm-generic/topology.h>
arch/alpha/kernel/irq.c +2 -1
@@ -50,7 +50,8 @@ int irq_select_affinity(unsigned int irq)if (!irq_desc[irq].chip->set_affinity || irq_user_affinity[irq])return 1 ; -while (!cpu_possible (cpu) || !cpu_isset (cpu, irq_default_affinity)) +while (!cpu_possible (cpu) || + !cpumask_test_cpu (cpu, irq_default_affinity)) cpu = (cpu < (NR_CPUS -1 ) ? cpu +1 :0 ); last_cpu = cpu;
arch/alpha/kernel/setup.c +5 -0
@@ -79,6 +79,11 @@ int alpha_l3_cacheshape;unsigned long alpha_verbose_mcheck =CONFIG_VERBOSE_MCHECK_ON ;#endif +#ifdef CONFIG_NUMA +struct cpumask node_to_cpumask_map[MAX_NUMNODES ]__read_mostly ; +EXPORT_SYMBOL (node_to_cpumask_map); +#endif +/* Which processor we booted from. */ int boot_cpuid;
arch/avr32/include/asm/bitops.h +5 -0
@@ -263,6 +263,11 @@ static inline int fls(unsigned long word)return 32 - result; } +static inline int __fls (unsigned long word) +{ +return fls (word) -1 ; +} +unsigned long find_first_zero_bit (const unsigned long *addr,unsigned long size); unsigned long find_next_zero_bit(const unsigned long *addr,
arch/blackfin/include/asm/bitops.h +1 -0
@@ -213,6 +213,7 @@ static __inline__ int __test_bit(int nr, const void *addr)#endif /* __KERNEL__ */ #include <asm-generic/bitops/fls.h> +#include <asm-generic/bitops/__fls.h> #include <asm-generic/bitops/fls64.h> #endif /* _BLACKFIN_BITOPS_H */
arch/cris/include/asm/bitops.h +1 -0
@@ -148,6 +148,7 @@ static inline int test_and_change_bit(int nr, volatile unsigned long *addr)#define ffs kernel_ffs#include <asm-generic/bitops/fls.h> +#include <asm-generic/bitops/__fls.h> #include <asm-generic/bitops/fls64.h> #include <asm-generic/bitops/hweight.h> #include <asm-generic/bitops/find.h>
arch/h8300/include/asm/bitops.h +1 -0
@@ -207,6 +207,7 @@ static __inline__ unsigned long __ffs(unsigned long word)#endif /* __KERNEL__ */ #include <asm-generic/bitops/fls.h> +#include <asm-generic/bitops/__fls.h> #include <asm-generic/bitops/fls64.h> #endif /* _H8300_BITOPS_H */
arch/ia64/include/asm/irq.h +1 -1
@@ -27,7 +27,7 @@ irq_canonicalize (int irq) }extern void set_irq_affinity_info (unsigned int irq,int dest,int redir); -bool is_affinity_mask_valid (cpumask_t cpumask); +bool is_affinity_mask_valid (cpumask_var_t cpumask);#define is_affinity_mask_valid is_affinity_mask_valid
arch/ia64/include/asm/topology.h +8 -1
@@ -34,6 +34,7 @@* Returns a bitmask of CPUs on Node 'node'. */ #define node_to_cpumask (node) (node_to_cpu_mask[node]) +#define cpumask_of_node (node) (&node_to_cpu_mask[node])/* * Returns the number of the node containing Node 'nid'.
@@ -45,7 +46,7 @@/* * Returns the number of the first CPU on Node 'node'. */ -#define node_to_first_cpu (node) (first_cpu(node_to_cpumask(node))) +#define node_to_first_cpu (node) (cpumask_first(cpumask_of_node(node)))/* * Determines the node for a given pci bus
@@ -109,6 +110,8 @@ void build_cpu_to_node_map(void);#define topology_core_id (cpu) (cpu_data(cpu)->core_id)#define topology_core_siblings (cpu) (cpu_core_map[cpu])#define topology_thread_siblings (cpu) (per_cpu(cpu_sibling_map, cpu)) +#define topology_core_cpumask (cpu) (&cpu_core_map[cpu]) +#define topology_thread_cpumask (cpu) (&per_cpu(cpu_sibling_map, cpu))#define smt_capable () (smp_num_siblings > 1)#endif
@@ -119,6 +122,10 @@ extern void arch_fix_phys_package_id(int num, u32 slot); node_to_cpumask(pcibus_to_node(bus)) \ ) +#define cpumask_of_pcibus (bus) (pcibus_to_node(bus) == -1 ? \ + cpu_all_mask : \ + cpumask_from_node(pcibus_to_node(bus))) +#include <asm-generic/topology.h> #endif /* _ASM_IA64_TOPOLOGY_H */
arch/ia64/kernel/acpi.c +1 -2
@@ -202,7 +202,6 @@ char *__init __acpi_map_table(unsigned long phys_addr, unsigned long size)Boot-time Table Parsing -------------------------------------------------------------------------- */ -static int total_cpus__initdata ;static int available_cpus__initdata ;struct acpi_table_madt *acpi_madt__initdata ;static u8 has_8259;
@@ -1001,7 +1000,7 @@ acpi_map_iosapic(acpi_handle handle, u32 depth, void *context, void **ret) node =pxm_to_node (pxm);if (node >=MAX_NUMNODES || !node_online (node) || -cpus_empty (node_to_cpumask (node))) +cpumask_empty (cpumask_of_node (node)))return AE_OK ;/* We know a gsi to node mapping! */
arch/ia64/kernel/iosapic.c +11 -12
@@ -695,32 +695,31 @@ get_target_cpu (unsigned int gsi, int irq)#ifdef CONFIG_NUMA {int num_cpus, cpu_index, iosapic_index, numa_cpu, i =0 ; -cpumask_t cpu_mask; +const struct cpumask *cpu_mask; iosapic_index =find_iosapic (gsi);if (iosapic_index <0 || iosapic_lists[iosapic_index].node ==MAX_NUMNODES ) goto skip_numa_setup; - cpu_mask =node_to_cpumask (iosapic_lists[iosapic_index].node); -cpus_and (cpu_mask, cpu_mask, domain); -for_each_cpu_mask (numa_cpu, cpu_mask) { -if (!cpu_online (numa_cpu)) -cpu_clear (numa_cpu, cpu_mask); + cpu_mask =cpumask_of_node (iosapic_lists[iosapic_index].node); + num_cpus =0 ; +for_each_cpu_and (numa_cpu, cpu_mask, &domain) { +if (cpu_online (numa_cpu)) + num_cpus++; } - num_cpus =cpus_weight (cpu_mask); -if (!num_cpus) goto skip_numa_setup;/* Use irq assignment to distribute across cpus in node */ cpu_index = irq % num_cpus; -for (numa_cpu =first_cpu (cpu_mask) ; i < cpu_index ; i++) - numa_cpu =next_cpu (numa_cpu, cpu_mask); +for_each_cpu_and (numa_cpu, cpu_mask, &domain) +if (cpu_online (numa_cpu) && i++ >= cpu_index) +break ; -if (numa_cpu !=NR_CPUS ) +if (numa_cpu < nr_cpu_ids)return cpu_physical_id (numa_cpu); } skip_numa_setup:
@@ -731,7 +730,7 @@ skip_numa_setup:* case of NUMA.) */ do { -if (++cpu >=NR_CPUS ) +if (++cpu >= nr_cpu_ids) cpu =0 ; }while (!cpu_online (cpu) || !cpu_isset (cpu, domain));
arch/ia64/kernel/irq.c +2 -2
@@ -112,11 +112,11 @@ void set_irq_affinity_info (unsigned int irq, int hwid, int redir) } } -bool is_affinity_mask_valid (cpumask_t cpumask) +bool is_affinity_mask_valid (cpumask_var_t cpumask) {if (ia64_platform_is ("sn2" )) {/* Only allow one CPU to be specified in the smp_affinity mask */ -if (cpus_weight (cpumask) !=1 ) +if (cpumask_weight (cpumask) !=1 )return false; }return true;
arch/ia64/sn/kernel/sn2/sn_hwperf.c +12 -15
@@ -385,7 +385,6 @@ static int sn_topology_show(struct seq_file *s, void *d)int j;const char *slabname;int ordinal; -cpumask_t cpumask;char slice;struct cpuinfo_ia64 *c;struct sn_hwperf_port_info *ptdata;
@@ -473,23 +472,21 @@ static int sn_topology_show(struct seq_file *s, void *d)* CPUs on this node, if any */ if (!SN_HWPERF_IS_IONODE (obj)) { - cpumask =node_to_cpumask (ordinal); -for_each_online_cpu (i) { -if (cpu_isset (i, cpumask)) { - slice ='a' +cpuid_to_slice (i); - c =cpu_data (i); -seq_printf (s,"cpu %d %s%c local" -" freq %luMHz, arch ia64" , - i, obj->location, slice, - c->proc_freq /1000000 ); -for_each_online_cpu (j) { -seq_printf (s, j ?":%d" :", dist %d" , -node_distance ( +for_each_cpu_and (i, cpu_online_mask, +cpumask_of_node (ordinal)) { + slice ='a' +cpuid_to_slice (i); + c =cpu_data (i); +seq_printf (s,"cpu %d %s%c local" +" freq %luMHz, arch ia64" , + i, obj->location, slice, + c->proc_freq /1000000 ); +for_each_online_cpu (j) { +seq_printf (s, j ?":%d" :", dist %d" , +node_distance (cpu_to_node (i),cpu_to_node (j))); - } -seq_putc (s,'\n' ); } +seq_putc (s,'\n' ); } } }
arch/m32r/kernel/smpboot.c +1 -1
@@ -592,7 +592,7 @@ int setup_profiling_timer(unsigned int multiplier)* accounting. At that time they also adjust their APIC timers * accordingly. */ -for (i =0 ; i <NR_CPUS ; ++i) +for_each_possible_cpu (i)per_cpu (prof_multiplier, i) = multiplier;return 0 ;
arch/m68knommu/include/asm/bitops.h +1 -0
@@ -331,6 +331,7 @@ found_middle:#endif /* __KERNEL__ */ #include <asm-generic/bitops/fls.h> +#include <asm-generic/bitops/__fls.h> #include <asm-generic/bitops/fls64.h> #endif /* _M68KNOMMU_BITOPS_H */
arch/mips/include/asm/mach-ip27/topology.h +3 -1
@@ -25,11 +25,13 @@ extern struct cpuinfo_ip27 sn_cpu_info[NR_CPUS];#define cpu_to_node (cpu) (sn_cpu_info[(cpu)].p_nodeid)#define parent_node (node) (node)#define node_to_cpumask (node) (hub_data(node)->h_cpus) -#define node_to_first_cpu (node) (first_cpu(node_to_cpumask(node))) +#define cpumask_of_node (node) (&hub_data(node)->h_cpus) +#define node_to_first_cpu (node) (cpumask_first(cpumask_of_node(node)))struct pci_bus ;extern int pcibus_to_node (struct pci_bus *);#define pcibus_to_cpumask (bus) (cpu_online_map) +#define cpumask_of_pcibus (bus) (cpu_online_mask)extern unsigned char __node_distances[MAX_COMPACT_NODES ][MAX_COMPACT_NODES ];
arch/parisc/include/asm/smp.h +0 -2
@@ -16,8 +16,6 @@#include <linux/cpumask.h> typedef unsigned long address_t ; -extern cpumask_t cpu_online_map; -/* * Private routines/data
arch/powerpc/include/asm/topology.h +9 -3
@@ -22,11 +22,11 @@ static inline cpumask_t node_to_cpumask(int node)return numa_cpumask_lookup_table[node]; } +#define cpumask_of_node (node) (&numa_cpumask_lookup_table[node]) +static inline int node_to_first_cpu (int node) { -cpumask_t tmp; - tmp =node_to_cpumask (node); -return first_cpu (tmp); +return cpumask_first (cpumask_of_node (node)); }int of_node_to_nid (struct device_node *device);
@@ -46,6 +46,10 @@ static inline int pcibus_to_node(struct pci_bus *bus) node_to_cpumask(pcibus_to_node(bus)) \ ) +#define cpumask_of_pcibus (bus) (pcibus_to_node(bus) == -1 ? \ + cpu_all_mask : \ + cpumask_of_node(pcibus_to_node(bus))) +/* sched_domains SD_NODE_INIT for PPC64 machines */ #define SD_NODE_INIT (struct sched_domain) { \ .parent = NULL, \
@@ -108,6 +112,8 @@ static inline void sysfs_remove_device_from_node(struct sys_device *dev,#define topology_thread_siblings (cpu) (per_cpu(cpu_sibling_map, cpu))#define topology_core_siblings (cpu) (per_cpu(cpu_core_map, cpu)) +#define topology_thread_cpumask (cpu) (&per_cpu(cpu_sibling_map, cpu)) +#define topology_core_cpumask (cpu) (&per_cpu(cpu_core_map, cpu))#define topology_core_id (cpu) (cpu_to_core_id(cpu))#endif #endif
arch/powerpc/platforms/cell/spu_priv1_mmio.c +3 -3
@@ -80,10 +80,10 @@ static void cpu_affinity_set(struct spu *spu, int cpu)u64 route;if (nr_cpus_node (spu->node)) { -cpumask_t spumask =node_to_cpumask (spu->node); -cpumask_t cpumask =node_to_cpumask (cpu_to_node (cpu)); +const struct cpumask *spumask =cpumask_of_node (spu->node), + *cpumask =cpumask_of_node (cpu_to_node (cpu)); -if (!cpus_intersects (spumask, cpumask)) +if (!cpumask_intersects (spumask, cpumask))return ; }
arch/powerpc/platforms/cell/spufs/sched.c +2 -2
@@ -166,9 +166,9 @@ void spu_update_sched_info(struct spu_context *ctx)static int __node_allowed (struct spu_context *ctx,int node) {if (nr_cpus_node (node)) { -cpumask_t mask =node_to_cpumask (node); +const struct cpumask *mask =cpumask_of_node (node); -if (cpus_intersects (mask, ctx->cpus_allowed)) +if (cpumask_intersects (mask, &ctx->cpus_allowed))return 1 ; }
arch/s390/include/asm/topology.h +2 -0
@@ -6,10 +6,12 @@#define mc_capable () (1)cpumask_t cpu_coregroup_map (unsigned int cpu); +const struct cpumask *cpu_coregroup_mask (unsigned int cpu);extern cpumask_t cpu_core_map[NR_CPUS ];#define topology_core_siblings (cpu) (cpu_core_map[cpu]) +#define topology_core_cpumask (cpu) (&cpu_core_map[cpu])int topology_set_cpu_management (int fc);void topology_schedule_update (void );
arch/s390/kernel/topology.c +5 -0
@@ -97,6 +97,11 @@ cpumask_t cpu_coregroup_map(unsigned int cpu)return mask; } +const struct cpumask *cpu_coregroup_mask (unsigned int cpu) +{ +return &cpu_core_map[cpu]; +} +static void add_cpus_to_core (struct tl_cpu *tl_cpu,struct core_info *core) {unsigned int cpu;
arch/sh/include/asm/topology.h +1 -0
@@ -32,6 +32,7 @@#define parent_node (node) ((void)(node),0)#define node_to_cpumask (node) ((void)node, cpu_online_map) +#define cpumask_of_node (node) ((void)node, cpu_online_mask)#define node_to_first_cpu (node) ((void)(node),0)#define pcibus_to_node (bus) ((void)(bus), -1)
arch/sparc/include/asm/topology_64.h +9 -4
@@ -16,8 +16,12 @@ static inline cpumask_t node_to_cpumask(int node) {return numa_cpumask_lookup_table[node]; } +#define cpumask_of_node (node) (&numa_cpumask_lookup_table[node]) -/* Returns a pointer to the cpumask of CPUs on Node 'node'. */ +/* +* Returns a pointer to the cpumask of CPUs on Node 'node'. +* Deprecated: use "const struct cpumask *mask = cpumask_of_node(node)" +*/ #define node_to_cpumask_ptr (v, node) \ cpumask_t *v = &(numa_cpumask_lookup_table[node])
@@ -26,9 +30,7 @@ static inline cpumask_t node_to_cpumask(int node)static inline int node_to_first_cpu (int node) { -cpumask_t tmp; - tmp =node_to_cpumask (node); -return first_cpu (tmp); +return cpumask_first (cpumask_of_node (node)); }struct pci_bus ;
@@ -77,10 +79,13 @@ static inline int pcibus_to_node(struct pci_bus *pbus)#define topology_core_id (cpu) (cpu_data(cpu).core_id)#define topology_core_siblings (cpu) (cpu_core_map[cpu])#define topology_thread_siblings (cpu) (per_cpu(cpu_sibling_map, cpu)) +#define topology_core_cpumask (cpu) (&cpu_core_map[cpu]) +#define topology_thread_cpumask (cpu) (&per_cpu(cpu_sibling_map, cpu))#define mc_capable () (sparc64_multi_core)#define smt_capable () (sparc64_multi_core)#endif /* CONFIG_SMP */ #define cpu_coregroup_map (cpu) (cpu_core_map[cpu]) +#define cpu_coregroup_mask (cpu) (&cpu_core_map[cpu])#endif /* _ASM_SPARC64_TOPOLOGY_H */
arch/sparc/kernel/of_device_64.c +1 -1
@@ -778,7 +778,7 @@ static unsigned int __init build_one_device_irq(struct of_device *op, out: nid =of_node_to_nid (dp);if (nid !=-1 ) { -cpumask_t numa_mask =node_to_cpumask (nid); +cpumask_t numa_mask = *cpumask_of_node (nid);irq_set_affinity (irq, &numa_mask); }
arch/sparc/kernel/pci_msi.c +1 -1
@@ -286,7 +286,7 @@ static int bringup_one_msi_queue(struct pci_pbm_info *pbm, nid = pbm->numa_node;if (nid !=-1 ) { -cpumask_t numa_mask =node_to_cpumask (nid); +cpumask_t numa_mask = *cpumask_of_node (nid);irq_set_affinity (irq, &numa_mask); }
arch/x86/include/asm/es7000/apic.h +3 -29
@@ -157,7 +157,7 @@ cpu_mask_to_apicid_cluster(const struct cpumask *cpumask) num_bits_set =cpumask_weight (cpumask);/* Return id to all */ -if (num_bits_set ==NR_CPUS ) +if (num_bits_set == nr_cpu_ids)return 0xFF ;/* * The cpus in the mask must all be on the apic cluster. If are not
@@ -190,7 +190,7 @@ static inline unsigned int cpu_mask_to_apicid(const cpumask_t *cpumask) num_bits_set =cpus_weight (*cpumask);/* Return id to all */ -if (num_bits_set ==NR_CPUS ) +if (num_bits_set == nr_cpu_ids)return cpu_to_logical_apicid (0 );/* * The cpus in the mask must all be on the apic cluster. If are not
@@ -218,9 +218,6 @@ static inline unsigned int cpu_mask_to_apicid(const cpumask_t *cpumask)static inline unsigned int cpu_mask_to_apicid_and (const struct cpumask *inmask,const struct cpumask *andmask) { -int num_bits_set; -int cpus_found =0 ; -int cpu;int apicid =cpu_to_logical_apicid (0 );cpumask_var_t cpumask;
@@ -229,31 +226,8 @@ static inline unsigned int cpu_mask_to_apicid_and(const struct cpumask *inmask,cpumask_and (cpumask, inmask, andmask);cpumask_and (cpumask, cpumask, cpu_online_mask); + apicid =cpu_mask_to_apicid (cpumask); - num_bits_set =cpumask_weight (cpumask); -/* Return id to all */ -if (num_bits_set ==NR_CPUS ) - goto exit; -/* -* The cpus in the mask must all be on the apic cluster. If are not -* on the same apicid cluster return default value of TARGET_CPUS. -*/ - cpu =cpumask_first (cpumask); - apicid =cpu_to_logical_apicid (cpu); -while (cpus_found < num_bits_set) { -if (cpumask_test_cpu (cpu, cpumask)) { -int new_apicid =cpu_to_logical_apicid (cpu); -if (apicid_cluster (apicid) != -apicid_cluster (new_apicid)){ -printk ("%s: Not a valid mask!\n" , __func__); -return cpu_to_logical_apicid (0 ); - } - apicid = new_apicid; - cpus_found++; - } - cpu++; - } -exit:free_cpumask_var (cpumask);return apicid; }
arch/x86/include/asm/lguest.h +1 -1
@@ -15,7 +15,7 @@#define SHARED_SWITCHER_PAGES \ DIV_ROUND_UP(end_switcher_text - start_switcher_text, PAGE_SIZE)/* Pages for switcher itself, then two pages per cpu */ -#define TOTAL_SWITCHER_PAGES (SHARED_SWITCHER_PAGES + 2 * NR_CPUS) +#define TOTAL_SWITCHER_PAGES (SHARED_SWITCHER_PAGES + 2 * nr_cpu_ids)/* We map at -4M for ease of mapping into the guest (one PTE page). */ #define SWITCHER_ADDR 0xFFC00000
arch/x86/include/asm/numaq/apic.h +2 -2
@@ -63,8 +63,8 @@ static inline physid_mask_t ioapic_phys_id_map(physid_mask_t phys_map)extern u8 cpu_2_logical_apicid[];static inline int cpu_to_logical_apicid (int cpu) { -if (cpu >=NR_CPUS ) -return BAD_APICID ; +if (cpu >= nr_cpu_ids) +return BAD_APICID ;return (int )cpu_2_logical_apicid[cpu]; }
arch/x86/include/asm/pci.h +8 -2
@@ -102,9 +102,9 @@ extern void pci_iommu_alloc(void);#ifdef CONFIG_NUMA /* Returns the node based on pci bus */ -static inline int __pcibus_to_node (struct pci_bus *bus) +static inline int __pcibus_to_node (const struct pci_bus *bus) { -struct pci_sysdata *sd = bus->sysdata; +const struct pci_sysdata *sd = bus->sysdata;return sd->node; }
@@ -113,6 +113,12 @@ static inline cpumask_t __pcibus_to_cpumask(struct pci_bus *bus) {return node_to_cpumask (__pcibus_to_node (bus)); } + +static inline const struct cpumask * +cpumask_of_pcibus (const struct pci_bus *bus) +{ +return cpumask_of_node (__pcibus_to_node (bus)); +}#endif #endif /* _ASM_X86_PCI_H */
arch/x86/include/asm/summit/apic.h +8 -34
79 lines not on this page · raw message
arch/x86/include/asm/topology.h +24 -12
@@ -61,13 +61,19 @@ static inline int cpu_to_node(int cpu)* * Side note: this function creates the returned cpumask on the stack * so with a high NR_CPUS count, excessive stack space is used. The -* node_to_cpumask_ptr function should be used whenever possible. +* cpumask_of_node function should be used whenever possible. */ static inline cpumask_t node_to_cpumask (int node) {return node_to_cpumask_map[node]; } +/* Returns a bitmask of CPUs on Node 'node'. */ +static inline const struct cpumask *cpumask_of_node (int node) +{ +return &node_to_cpumask_map[node]; +} +#else /* CONFIG_X86_64 */ /* Mappings between node number and cpus on that node. */
@@ -82,7 +88,7 @@ DECLARE_EARLY_PER_CPU(int, x86_cpu_to_node_map);#ifdef CONFIG_DEBUG_PER_CPU_MAPS extern int cpu_to_node (int cpu);extern int early_cpu_to_node (int cpu); -extern const cpumask_t *_node_to_cpumask_ptr (int node); +extern const cpumask_t *cpumask_of_node (int node);extern cpumask_t node_to_cpumask (int node);#else /* !CONFIG_DEBUG_PER_CPU_MAPS */
@@ -103,7 +109,7 @@ static inline int early_cpu_to_node(int cpu) }/* Returns a pointer to the cpumask of CPUs on Node 'node'. */ -static inline const cpumask_t *_node_to_cpumask_ptr (int node) +static inline const cpumask_t *cpumask_of_node (int node) {return &node_to_cpumask_map[node]; }
@@ -116,12 +122,15 @@ static inline cpumask_t node_to_cpumask(int node)#endif /* !CONFIG_DEBUG_PER_CPU_MAPS */ -/* Replace default node_to_cpumask_ptr with optimized version */ +/* +* Replace default node_to_cpumask_ptr with optimized version +* Deprecated: use "const struct cpumask *mask = cpumask_of_node(node)" +*/ #define node_to_cpumask_ptr (v, node) \ - const cpumask_t *v = _node_to_cpumask_ptr(node) + const cpumask_t *v = cpumask_of_node(node)#define node_to_cpumask_ptr_next (v, node) \ - v = _node_to_cpumask_ptr(node) + v = cpumask_of_node(node)#endif /* CONFIG_X86_64 */
@@ -187,7 +196,7 @@ extern int __node_distance(int, int);#define cpu_to_node (cpu) 0#define early_cpu_to_node (cpu) 0 -static inline const cpumask_t *_node_to_cpumask_ptr (int node) +static inline const cpumask_t *cpumask_of_node (int node) {return &cpu_online_map; }
@@ -200,12 +209,15 @@ static inline int node_to_first_cpu(int node) return first_cpu(cpu_online_map); } -/* Replace default node_to_cpumask_ptr with optimized version */ +/* +* Replace default node_to_cpumask_ptr with optimized version +* Deprecated: use "const struct cpumask *mask = cpumask_of_node(node)" +*/ #define node_to_cpumask_ptr (v, node) \ - const cpumask_t *v = _node_to_cpumask_ptr(node) + const cpumask_t *v = cpumask_of_node(node)#define node_to_cpumask_ptr_next (v, node) \ - v = _node_to_cpumask_ptr(node) + v = cpumask_of_node(node)#endif #include <asm-generic/topology.h>
@@ -214,12 +226,12 @@ static inline int node_to_first_cpu(int node)/* Returns the number of the first CPU on Node 'node'. */ static inline int node_to_first_cpu (int node) { -node_to_cpumask_ptr (mask, node); -return first_cpu (*mask); +return cpumask_first (cpumask_of_node (node)); }#endif extern cpumask_t cpu_coregroup_map (int cpu); +extern const struct cpumask *cpu_coregroup_mask (int cpu);#ifdef ENABLE_TOPO_DEFINES #define topology_physical_package_id (cpu) (cpu_data(cpu).phys_proc_id)
arch/x86/kernel/acpi/boot.c +23 -8
@@ -538,9 +538,10 @@ static int __cpuinit _acpi_map_lsapic(acpi_handle handle, int *pcpu)struct acpi_buffer buffer = {ACPI_ALLOCATE_BUFFER ,NULL };union acpi_object *obj;struct acpi_madt_local_apic *lapic; -cpumask_t tmp_map, new_map; +cpumask_var_t tmp_map, new_map;u8 physid;int cpu; +int retval = -ENOMEM ;if (ACPI_FAILURE (acpi_evaluate_object (handle,"_MAT" ,NULL , &buffer)))return -EINVAL ;
@@ -569,23 +570,37 @@ static int __cpuinit _acpi_map_lsapic(acpi_handle handle, int *pcpu) buffer.length =ACPI_ALLOCATE_BUFFER ; buffer.pointer =NULL ; - tmp_map = cpu_present_map; +if (!alloc_cpumask_var (&tmp_map,GFP_KERNEL )) + goto out; + +if (!alloc_cpumask_var (&new_map,GFP_KERNEL )) + goto free_tmp_map; + +cpumask_copy (tmp_map, cpu_present_mask);acpi_register_lapic (physid, lapic->lapic_flags &ACPI_MADT_ENABLED );/* * If mp_register_lapic successfully generates a new logical cpu * number, then the following will get us exactly what was mapped */ -cpus_andnot (new_map, cpu_present_map, tmp_map); -if (cpus_empty (new_map)) { +cpumask_andnot (new_map, cpu_present_mask, tmp_map); +if (cpumask_empty (new_map)) {printk ("Unable to map lapic to logical cpu number\n" ); -return -EINVAL ; + retval = -EINVAL ; + goto free_new_map; } - cpu =first_cpu (new_map); + cpu =cpumask_first (new_map); *pcpu = cpu; -return 0 ; + retval =0 ; + +free_new_map: +free_cpumask_var (new_map); +free_tmp_map: +free_cpumask_var (tmp_map); +out: +return retval; }/* wrapper to silence section mismatch warning */
@@ -598,7 +613,7 @@ EXPORT_SYMBOL(acpi_map_lsapic);int acpi_unmap_lsapic (int cpu) {per_cpu (x86_cpu_to_apicid, cpu) =-1 ; -cpu_clear (cpu, cpu_present_map); +set_cpu_present (cpu, false); num_processors--;return (0 );
arch/x86/kernel/apic.c +2 -2
@@ -140,7 +140,7 @@ static int lapic_next_event(unsigned long delta, struct clock_event_device *evt);static void lapic_timer_setup (enum clock_event_mode mode,struct clock_event_device *evt); -static void lapic_timer_broadcast (const cpumask_t *mask); +static void lapic_timer_broadcast (const struct cpumask *mask);static void apic_pm_activate (void );/*
@@ -453,7 +453,7 @@ static void lapic_timer_setup(enum clock_event_mode mode,/* * Local APIC timer broadcast function */ -static void lapic_timer_broadcast (const cpumask_t *mask) +static void lapic_timer_broadcast (const struct cpumask *mask) {#ifdef CONFIG_SMP send_IPI_mask (mask,LOCAL_TIMER_VECTOR );
arch/x86/kernel/cpu/common.c +1 -1
@@ -355,7 +355,7 @@ void __cpuinit detect_ht(struct cpuinfo_x86 *c)printk (KERN_INFO "CPU: Hyper-Threading is disabled\n" ); }else if (smp_num_siblings >1 ) { -if (smp_num_siblings >NR_CPUS ) { +if (smp_num_siblings > nr_cpu_ids) {printk (KERN_WARNING "CPU: Unsupported number of siblings %d" , smp_num_siblings); smp_num_siblings =1 ;
arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c +25 -3
@@ -517,6 +517,17 @@ acpi_cpufreq_guess_freq(struct acpi_cpufreq_data *data, unsigned int cpu) } } +static void free_acpi_perf_data (void ) +{ +unsigned int i; + +/* Freeing a NULL pointer is OK, and alloc_percpu zeroes. */ +for_each_possible_cpu (i) +free_cpumask_var (per_cpu_ptr (acpi_perf_data, i) + ->shared_cpu_map); +free_percpu (acpi_perf_data); +} +/* * acpi_cpufreq_early_init - initialize ACPI P-States library *
@@ -527,6 +538,7 @@ acpi_cpufreq_guess_freq(struct acpi_cpufreq_data *data, unsigned int cpu)*/ static int __init acpi_cpufreq_early_init (void ) { +unsigned int i;dprintk ("acpi_cpufreq_early_init\n" ); acpi_perf_data = alloc_percpu(struct acpi_processor_performance);
@@ -534,6 +546,16 @@ static int __init acpi_cpufreq_early_init(void)dprintk ("Memory allocation error for acpi_perf_data.\n" );return -ENOMEM ; } +for_each_possible_cpu (i) { +if (!alloc_cpumask_var_node ( + &per_cpu_ptr (acpi_perf_data, i)->shared_cpu_map, +GFP_KERNEL ,cpu_to_node (i))) { + +/* Freeing a NULL pointer is OK: alloc_percpu zeroes. */ +free_acpi_perf_data (); +return -ENOMEM ; + } + }/* Do initialization in ACPI core */ acpi_processor_preregister_performance (acpi_perf_data);
@@ -604,9 +626,9 @@ static int acpi_cpufreq_cpu_init(struct cpufreq_policy *policy)*/ if (policy->shared_type ==CPUFREQ_SHARED_TYPE_ALL || policy->shared_type ==CPUFREQ_SHARED_TYPE_ANY ) { - policy->cpus = perf->shared_cpu_map; +cpumask_copy (&policy->cpus, perf->shared_cpu_map); } - policy->related_cpus = perf->shared_cpu_map; +cpumask_copy (&policy->related_cpus, perf->shared_cpu_map);#ifdef CONFIG_SMP dmi_check_system (sw_any_bug_dmi_table);
@@ -795,7 +817,7 @@ static int __init acpi_cpufreq_init(void) ret =cpufreq_register_driver (&acpi_cpufreq_driver);if (ret) -free_percpu (acpi_perf_data); +free_acpi_perf_data ();return ret; }
arch/x86/kernel/cpu/cpufreq/powernow-k7.c +9 -0
@@ -310,6 +310,12 @@ static int powernow_acpi_init(void) goto err0; } +if (!alloc_cpumask_var (&acpi_processor_perf->shared_cpu_map, +GFP_KERNEL )) { + retval = -ENOMEM ; + goto err05; + } +if (acpi_processor_register_performance (acpi_processor_perf,0 )) { retval = -EIO ; goto err1;
@@ -412,6 +418,8 @@ static int powernow_acpi_init(void) err2:acpi_processor_unregister_performance (acpi_processor_perf,0 ); err1: +free_cpumask_var (acpi_processor_perf->shared_cpu_map); +err05:kfree (acpi_processor_perf); err0: printk(KERN_WARNING PFX "ACPI perflib can not be used in this platform\n");
@@ -652,6 +660,7 @@ static int powernow_cpu_exit (struct cpufreq_policy *policy) {#ifdef CONFIG_X86_POWERNOW_K7_ACPI if (acpi_processor_perf) {acpi_processor_unregister_performance (acpi_processor_perf,0 ); +free_cpumask_var (acpi_processor_perf->shared_cpu_map);kfree (acpi_processor_perf); }#endif
arch/x86/kernel/cpu/cpufreq/powernow-k8.c +15 -9
@@ -766,7 +766,7 @@ static void powernow_k8_acpi_pst_values(struct powernow_k8_data *data, unsignedstatic int powernow_k8_cpu_init_acpi (struct powernow_k8_data *data) {struct cpufreq_frequency_table *powernow_table; -int ret_val; +int ret_val = -ENODEV ;if (acpi_processor_register_performance (&data->acpi_data, data->cpu)) {dprintk ("register performance failed: bad ACPI data\n" );
@@ -815,6 +815,13 @@ static int powernow_k8_cpu_init_acpi(struct powernow_k8_data *data)/* notify BIOS that we exist */ acpi_processor_notify_smm (THIS_MODULE ); +if (!alloc_cpumask_var (&data->acpi_data.shared_cpu_map,GFP_KERNEL )) { + printk(KERN_ERR PFX + "unable to alloc powernow_k8_data cpumask\n"); + ret_val = -ENOMEM ; + goto err_out_mem; + } +return 0 ; err_out_mem:
@@ -826,7 +833,7 @@ err_out:/* data->acpi_data.state_count informs us at ->exit() whether ACPI was used */ data->acpi_data.state_count =0 ; -return -ENODEV ; +return ret_val; }static int fill_powernow_table_pstate (struct powernow_k8_data *data,struct cpufreq_frequency_table *powernow_table)
@@ -929,6 +936,7 @@ static void powernow_k8_cpu_exit_acpi(struct powernow_k8_data *data) {if (data->acpi_data.state_count)acpi_processor_unregister_performance (&data->acpi_data, data->cpu); +free_cpumask_var (data->acpi_data.shared_cpu_map); }#else
@@ -1134,7 +1142,8 @@ static int __cpuinit powernowk8_cpu_init(struct cpufreq_policy *pol) data->cpu = pol->cpu; data->currpstate =HW_PSTATE_INVALID ; -if (powernow_k8_cpu_init_acpi (data)) { + rc =powernow_k8_cpu_init_acpi (data); +if (rc) {/* * Use the PSB BIOS structure. This is only availabe on * an UP version, and is deprecated by AMD.
@@ -1152,20 +1161,17 @@ static int __cpuinit powernowk8_cpu_init(struct cpufreq_policy *pol)"ACPI maintainers and complain to your BIOS " "vendor.\n");#endif -kfree (data); -return -ENODEV ; + goto err_out; }if (pol->cpu !=0 ) { printk(KERN_ERR FW_BUG PFX "No ACPI _PSS objects for ""CPU other than CPU0. Complain to your BIOS " "vendor.\n"); -kfree (data); -return -ENODEV ; + goto err_out; } rc =find_psb_table (data);if (rc) { -kfree (data); -return -ENODEV ; + goto err_out; } }
arch/x86/kernel/cpu/intel_cacheinfo.c +1 -1
@@ -534,7 +534,7 @@ static void __cpuinit free_cache_attributes(unsigned int cpu)per_cpu (cpuid4_info, cpu) =NULL ; } -static void get_cpu_leaves (void *_retval) +static void __cpuinit get_cpu_leaves (void *_retval) {int j, *retval = _retval, cpu =smp_processor_id ();
arch/x86/kernel/cpuid.c +1 -1
@@ -121,7 +121,7 @@ static int cpuid_open(struct inode *inode, struct file *file)lock_kernel (); cpu =iminor (file->f_path.dentry->d_inode); -if (cpu >=NR_CPUS || !cpu_online (cpu)) { +if (cpu >= nr_cpu_ids || !cpu_online (cpu)) { ret = -ENXIO ;/* No such CPU */ goto out; }
arch/x86/kernel/io_apic.c +3 -3
@@ -214,11 +214,11 @@ static struct irq_cfg *get_one_free_irq_cfg(int cpu) cfg =kzalloc_node (sizeof (*cfg),GFP_ATOMIC , node);if (cfg) { -/* FIXME: needs alloc_cpumask_var_node() */ -if (!alloc_cpumask_var (&cfg->domain,GFP_ATOMIC )) { +if (!alloc_cpumask_var_node (&cfg->domain,GFP_ATOMIC , node)) {kfree (cfg); cfg =NULL ; - }else if (!alloc_cpumask_var (&cfg->old_domain,GFP_ATOMIC )) { + }else if (!alloc_cpumask_var_node (&cfg->old_domain, +GFP_ATOMIC , node)) {free_cpumask_var (cfg->domain);kfree (cfg); cfg =NULL ;
arch/x86/kernel/msr.c +1 -1
@@ -136,7 +136,7 @@ static int msr_open(struct inode *inode, struct file *file)lock_kernel (); cpu =iminor (file->f_path.dentry->d_inode); -if (cpu >=NR_CPUS || !cpu_online (cpu)) { +if (cpu >= nr_cpu_ids || !cpu_online (cpu)) { ret = -ENXIO ;/* No such CPU */ goto out; }
arch/x86/kernel/reboot.c +2 -2
@@ -501,7 +501,7 @@ void native_machine_shutdown(void)#ifdef CONFIG_X86_32 /* See if there has been given a command line override */ -if ((reboot_cpu !=-1 ) && (reboot_cpu <NR_CPUS ) && +if ((reboot_cpu !=-1 ) && (reboot_cpu < nr_cpu_ids) &&cpu_online (reboot_cpu)) reboot_cpu_id = reboot_cpu;#endif
@@ -511,7 +511,7 @@ void native_machine_shutdown(void) reboot_cpu_id =smp_processor_id ();/* Make certain I only run on the appropriate processor */ -set_cpus_allowed_ptr (current, &cpumask_of_cpu (reboot_cpu_id)); +set_cpus_allowed_ptr (current,cpumask_of (reboot_cpu_id));/* O.K Now that I'm on the appropriate processor, * stop all of the others.
arch/x86/kernel/setup_percpu.c +12 -21
@@ -153,12 +153,10 @@ void __init setup_per_cpu_areas(void) align = max_t(unsigned long, PAGE_SIZE, align); size =roundup (old_size, align); -printk (KERN_INFO -"NR_CPUS:%d nr_cpumask_bits:%d nr_cpu_ids:%d nr_node_ids:%d\n" , +pr_info ("NR_CPUS:%d nr_cpumask_bits:%d nr_cpu_ids:%d nr_node_ids:%d\n" ,NR_CPUS , nr_cpumask_bits, nr_cpu_ids, nr_node_ids); -printk (KERN_INFO "PERCPU: Allocating %zd bytes of per cpu data\n" , - size); +pr_info ("PERCPU: Allocating %zd bytes of per cpu data\n" , size);for_each_possible_cpu (cpu) {#ifndef CONFIG_NEED_MULTIPLE_NODES
@@ -169,22 +167,15 @@ void __init setup_per_cpu_areas(void)if (!node_online (node) || !NODE_DATA (node)) { ptr =__alloc_bootmem (size, align,__pa (MAX_DMA_ADDRESS )); -printk (KERN_INFO -"cpu %d has no node %d or node-local memory\n" , +pr_info ("cpu %d has no node %d or node-local memory\n" , cpu, node); -if (ptr) -printk (KERN_DEBUG -"per cpu data for cpu%d at %016lx\n" , - cpu,__pa (ptr)); - } -else { +pr_debug ("per cpu data for cpu%d at %016lx\n" , + cpu,__pa (ptr)); + }else { ptr =__alloc_bootmem_node (NODE_DATA (node), size, align,__pa (MAX_DMA_ADDRESS )); -if (ptr) -printk (KERN_DEBUG -"per cpu data for cpu%d on node%d " -"at %016lx\n" , - cpu, node,__pa (ptr)); +pr_debug ("per cpu data for cpu%d on node%d at %016lx\n" , + cpu, node,__pa (ptr)); }#endif per_cpu_offset (cpu) = ptr - __per_cpu_start;
@@ -339,25 +330,25 @@ static const cpumask_t cpu_mask_none;/* * Returns a pointer to the bitmask of CPUs on Node 'node'. */ -const cpumask_t *_node_to_cpumask_ptr (int node) +const cpumask_t *cpumask_of_node (int node) {if (node_to_cpumask_map ==NULL ) {printk (KERN_WARNING -"_node_to_cpumask_ptr(%d): no node_to_cpumask_map!\n" , +"cpumask_of_node(%d): no node_to_cpumask_map!\n" , node);dump_stack ();return (const cpumask_t *)&cpu_online_map; }if (node >= nr_node_ids) {printk (KERN_WARNING -"_node_to_cpumask_ptr(%d): node > nr_node_ids(%d)\n" , +"cpumask_of_node(%d): node > nr_node_ids(%d)\n" , node, nr_node_ids);dump_stack ();return &cpu_mask_none; }return &node_to_cpumask_map[node]; } -EXPORT_SYMBOL (_node_to_cpumask_ptr); +EXPORT_SYMBOL (cpumask_of_node);/* * Returns a bitmask of CPUs on Node 'node'.
arch/x86/kernel/smpboot.c +11 -4
@@ -496,7 +496,7 @@ void __cpuinit set_cpu_sibling_map(int cpu) }/* maps the cpu to the sched domain representing multi-core */ -cpumask_t cpu_coregroup_map (int cpu) +const struct cpumask *cpu_coregroup_mask (int cpu) {struct cpuinfo_x86 *c = &cpu_data (cpu);/*
@@ -504,9 +504,14 @@ cpumask_t cpu_coregroup_map(int cpu)* And for power savings, we return cpu_core_map */ if (sched_mc_power_savings || sched_smt_power_savings) -return per_cpu (cpu_core_map, cpu); +return &per_cpu (cpu_core_map, cpu);else -return c->llc_shared_map; +return &c->llc_shared_map; +} + +cpumask_t cpu_coregroup_map (int cpu) +{ +return *cpu_coregroup_mask (cpu); }static void impress_friends (void )
@@ -1149,7 +1154,7 @@ static void __init smp_cpu_index_default(void)for_each_possible_cpu (i) { c = &cpu_data (i);/* mark all to hotplug */ - c->cpu_index =NR_CPUS ; + c->cpu_index = nr_cpu_ids; } }
@@ -1293,6 +1298,8 @@ __init void prefill_possible_map(void)else possible = setup_possible_cpus; + total_cpus =max_t (int, possible, num_processors + disabled_cpus); +if (possible >CONFIG_NR_CPUS ) { printk(KERN_WARNING "%d Processors exceeds NR_CPUS limit of %d\n",
arch/x86/mach-voyager/voyager_smp.c +3 -4
@@ -357,9 +357,8 @@ void __init find_smp_config(void)printk ("VOYAGER SMP: Boot cpu is %d\n" , boot_cpu_id);/* initialize the CPU structures (moved from smp_boot_cpus) */ -for (i =0 ; i <NR_CPUS ; i++) { +for (i =0 ; i < nr_cpu_ids; i++) cpu_irq_affinity[i] = ~0 ; - } cpu_online_map =cpumask_of_cpu (boot_cpu_id);/* The boot CPU must be extended */
@@ -1227,7 +1226,7 @@ int setup_profiling_timer(unsigned int multiplier)* new values until the next timer interrupt in which they do process * accounting. */ -for (i =0 ; i <NR_CPUS ; ++i) +for (i =0 ; i < nr_cpu_ids; ++i)per_cpu (prof_multiplier, i) = multiplier;return 0 ;
@@ -1257,7 +1256,7 @@ void __init voyager_smp_intr_init(void)int i;/* initialize the per cpu irq mask to all disabled */ -for (i =0 ; i <NR_CPUS ; i++) +for (i =0 ; i < nr_cpu_ids; i++) vic_irq_mask[i] =0xFFFF ;VIC_SET_GATE (VIC_CPI_LEVEL0 , vic_cpi_interrupt);
block/blk.h +2 -2
@@ -99,8 +99,8 @@ static inline int queue_congestion_off_threshold(struct request_queue *q)static inline int blk_cpu_to_group (int cpu) {#ifdef CONFIG_SCHED_MC -cpumask_t mask =cpu_coregroup_map (cpu); -return first_cpu (mask); +const struct cpumask *mask =cpu_coregroup_mask (cpu); +return cpumask_first (mask);#elif defined(CONFIG_SCHED_SMT )return first_cpu (per_cpu (cpu_sibling_map, cpu));#else
drivers/acpi/processor_core.c +10 -4
@@ -826,6 +826,11 @@ static int acpi_processor_add(struct acpi_device *device)if (!pr)return -ENOMEM ; +if (!alloc_cpumask_var (&pr->throttling.shared_cpu_map,GFP_KERNEL )) { +kfree (pr); +return -ENOMEM ; + } + pr->handle = device->handle;strcpy (acpi_device_name (device),ACPI_PROCESSOR_DEVICE_NAME );strcpy (acpi_device_class (device),ACPI_PROCESSOR_CLASS );
@@ -845,10 +850,8 @@ static int acpi_processor_remove(struct acpi_device *device, int type) pr =acpi_driver_data (device); -if (pr->id >= nr_cpu_ids) { -kfree (pr); -return 0 ; - } +if (pr->id >= nr_cpu_ids) + goto free;if (type ==ACPI_BUS_REMOVAL_EJECT ) {if (acpi_processor_handle_eject (pr))
@@ -873,6 +876,9 @@ static int acpi_processor_remove(struct acpi_device *device, int type)per_cpu (processors, pr->id) =NULL ;per_cpu (processor_device_array, pr->id) =NULL ; + +free: +free_cpumask_var (pr->throttling.shared_cpu_map);kfree (pr);return 0 ;
drivers/acpi/processor_perflib.c +16 -12
@@ -588,12 +588,15 @@ int acpi_processor_preregister_performance(int count, count_target;int retval =0 ;unsigned int i, j; -cpumask_t covered_cpus; +cpumask_var_t covered_cpus;struct acpi_processor *pr;struct acpi_psd_package *pdomain;struct acpi_processor *match_pr;struct acpi_psd_package *match_pdomain; +if (!alloc_cpumask_var (&covered_cpus,GFP_KERNEL )) +return -ENOMEM ; +mutex_lock (&performance_mutex); retval =0 ;
@@ -617,7 +620,7 @@ int acpi_processor_preregister_performance( } pr->performance =percpu_ptr (performance, i); -cpu_set (i, pr->performance->shared_cpu_map); +cpumask_set_cpu (i, pr->performance->shared_cpu_map);if (acpi_processor_get_psd (pr)) { retval = -EINVAL ;continue ;
@@ -650,18 +653,18 @@ int acpi_processor_preregister_performance( } } -cpus_clear (covered_cpus); +cpumask_clear (covered_cpus);for_each_possible_cpu (i) { pr =per_cpu (processors, i);if (!pr)continue ; -if (cpu_isset (i, covered_cpus)) +if (cpumask_test_cpu (i, covered_cpus))continue ; pdomain = &(pr->performance->domain_info); -cpu_set (i, pr->performance->shared_cpu_map); -cpu_set (i, covered_cpus); +cpumask_set_cpu (i, pr->performance->shared_cpu_map); +cpumask_set_cpu (i, covered_cpus);if (pdomain->num_processors <=1 )continue ;
@@ -699,8 +702,8 @@ int acpi_processor_preregister_performance( goto err_ret; } -cpu_set (j, covered_cpus); -cpu_set (j, pr->performance->shared_cpu_map); +cpumask_set_cpu (j, covered_cpus); +cpumask_set_cpu (j, pr->performance->shared_cpu_map); count++; }
@@ -718,8 +721,8 @@ int acpi_processor_preregister_performance( match_pr->performance->shared_type = pr->performance->shared_type; - match_pr->performance->shared_cpu_map = - pr->performance->shared_cpu_map; +cpumask_copy (match_pr->performance->shared_cpu_map, + pr->performance->shared_cpu_map); } }
@@ -731,14 +734,15 @@ err_ret:/* Assume no coordination on any error parsing domain info */ if (retval) { -cpus_clear (pr->performance->shared_cpu_map); -cpu_set (i, pr->performance->shared_cpu_map); +cpumask_clear (pr->performance->shared_cpu_map); +cpumask_set_cpu (i, pr->performance->shared_cpu_map); pr->performance->shared_type =CPUFREQ_SHARED_TYPE_ALL ; } pr->performance =NULL ;/* Will be set for real in register */ }mutex_unlock (&performance_mutex); +free_cpumask_var (covered_cpus);return retval; }EXPORT_SYMBOL (acpi_processor_preregister_performance);
drivers/acpi/processor_throttling.c +52 -28
209 lines not on this page · raw message
drivers/base/cpu.c +44 -0
54 lines not on this page · raw message
drivers/infiniband/hw/ehca/ehca_irq.c +8 -9
@@ -659,12 +659,12 @@ static inline int find_next_online_cpu(struct ehca_comp_pool *pool)WARN_ON_ONCE (!in_interrupt ());if (ehca_debug_level >=3 ) -ehca_dmp (&cpu_online_map,sizeof (cpumask_t),"" ); +ehca_dmp (cpu_online_mask,cpumask_size (),"" );spin_lock_irqsave (&pool->last_cpu_lock, flags); - cpu =next_cpu_nr (pool->last_cpu, cpu_online_map); + cpu =cpumask_next (pool->last_cpu, cpu_online_mask);if (cpu >= nr_cpu_ids) - cpu =first_cpu (cpu_online_map); + cpu =cpumask_first (cpu_online_mask); pool->last_cpu = cpu;spin_unlock_irqrestore (&pool->last_cpu_lock, flags);
@@ -855,7 +855,7 @@ static int __cpuinit comp_pool_callback(struct notifier_block *nfb,case CPU_UP_CANCELED_FROZEN :ehca_gen_dbg ("CPU: %x (CPU_CANCELED)" , cpu); cct =per_cpu_ptr (pool->cpu_comp_tasks, cpu); -kthread_bind (cct->task,any_online_cpu (cpu_online_map)); +kthread_bind (cct->task,cpumask_any (cpu_online_mask));destroy_comp_task (pool, cpu);break ;case CPU_ONLINE :
@@ -902,7 +902,7 @@ int ehca_create_comp_pool(void)return -ENOMEM ;spin_lock_init (&pool->last_cpu_lock); - pool->last_cpu =any_online_cpu (cpu_online_map); + pool->last_cpu =cpumask_any (cpu_online_mask); pool->cpu_comp_tasks = alloc_percpu(struct ehca_cpu_comp_task);if (pool->cpu_comp_tasks ==NULL ) {
@@ -934,10 +934,9 @@ void ehca_destroy_comp_pool(void)unregister_hotcpu_notifier (&comp_pool_callback_nb); -for (i =0 ; i <NR_CPUS ; i++) { -if (cpu_online (i)) -destroy_comp_task (pool, i); - } +for_each_online_cpu (i) +destroy_comp_task (pool, i); +free_percpu (pool->cpu_comp_tasks);kfree (pool); }
drivers/infiniband/hw/ipath/ipath_file_ops.c +4 -4
@@ -1679,7 +1679,7 @@ static int find_best_unit(struct file *fp,* InfiniPath chip to that processor (we assume reasonable connectivity, * for now). This code assumes that if affinity has been set * before this point, that at most one cpu is set; for now this -* is reasonable. I check for both cpus_empty() and cpus_full(), +* is reasonable. I check for both cpumask_empty() and cpumask_full(), * in case some kernel variant sets none of the bits when no * affinity is set. 2.6.11 and 12 kernels have all present * cpus set. Some day we'll have to fix it up further to handle
@@ -1688,11 +1688,11 @@ static int find_best_unit(struct file *fp,* information. There may be some issues with dual core numbering * as well. This needs more work prior to release. */ -if (!cpus_empty (current->cpus_allowed) && - !cpus_full (current->cpus_allowed)) { +if (!cpumask_empty (¤t->cpus_allowed) && + !cpumask_full (¤t->cpus_allowed)) {int ncpus =num_online_cpus (), curcpu =-1 , nset =0 ;for (i =0 ; i < ncpus; i++) -if (cpu_isset (i, current->cpus_allowed)) { +if (cpumask_test_cpu (i, ¤t->cpus_allowed)) {ipath_cdbg (PROC ,"%s[%u] affinity set for " "cpu %d/%d\n" , current->comm, current->pid, i, ncpus);
drivers/pnp/pnpbios/bioscalls.c +1 -1
@@ -481,7 +481,7 @@ void pnpbios_calls_init(union pnp_bios_install_struct *header)set_base (bad_bios_desc,__va ((unsigned long )0x40 <<4 ));_set_limit ((char *)&bad_bios_desc,4095 - (0x40 <<4 )); -for (i =0 ; i <NR_CPUS ; i++) { +for_each_possible_cpu (i) {struct desc_struct *gdt =get_cpu_gdt_table (i);if (!gdt)continue ;
fs/seq_file.c +2 -1
@@ -468,7 +468,8 @@ int seq_dentry(struct seq_file *m, struct dentry *dentry, char *esc)return -1 ; } -int seq_bitmap (struct seq_file *m,unsigned long *bits,unsigned int nr_bits) +int seq_bitmap (struct seq_file *m,const unsigned long *bits, +unsigned int nr_bits) {if (m->count < m->size) { int len = bitmap_scnprintf(m->buf + m->count,
include/acpi/processor.h +2 -2
@@ -127,7 +127,7 @@ struct acpi_processor_performance {unsigned int state_count;struct acpi_processor_px *states;struct acpi_psd_package domain_info; -cpumask_t shared_cpu_map; +cpumask_var_t shared_cpu_map;unsigned int shared_type; };
@@ -172,7 +172,7 @@ struct acpi_processor_throttling {unsigned int state_count;struct acpi_processor_tx_tss *states_tss;struct acpi_tsd_package domain_info; -cpumask_t shared_cpu_map; +cpumask_var_t shared_cpu_map;int (*acpi_processor_get_throttling) (struct acpi_processor * pr);int (*acpi_processor_set_throttling) (struct acpi_processor * pr,int state);
include/asm-frv/bitops.h +13 -0
@@ -339,6 +339,19 @@ int __ffs(unsigned long x)return 31 - bit; } +/** +* __fls - find last (most-significant) set bit in a long word +* @word: the word to search +* +* Undefined if no set bit exists, so code should check against 0 first. +*/ +static inline unsigned long __fls (unsigned long word) +{ +unsigned long bit; + asm("scan %1,gr0,%0" :"=r" (bit) :"r" (word)); +return bit; +} +/* * special slimline version of fls() for calculating ilog2_u32() * - note: no protection against n == 0
include/asm-m32r/bitops.h +1 -0
@@ -251,6 +251,7 @@ static __inline__ int test_and_change_bit(int nr, volatile void * addr)#include <asm-generic/bitops/ffz.h> #include <asm-generic/bitops/__ffs.h> #include <asm-generic/bitops/fls.h> +#include <asm-generic/bitops/__fls.h> #include <asm-generic/bitops/fls64.h> #ifdef __KERNEL__
include/asm-m68k/bitops.h +5 -0
@@ -315,6 +315,11 @@ static inline int fls(int x)return 32 - cnt; } +static inline int __fls (int x) +{ +return fls (x) -1 ; +} +#include <asm-generic/bitops/fls64.h> #include <asm-generic/bitops/sched.h> #include <asm-generic/bitops/hweight.h>
include/asm-mn10300/bitops.h +11 -0
@@ -196,6 +196,17 @@ int fls(int x) }/** +* __fls - find last (most-significant) set bit in a long word +* @word: the word to search +* +* Undefined if no set bit exists, so code should check against 0 first. +*/ +static inline unsigned long __fls (unsigned long word) +{ +return __ilog2_u32 (word); +} + +/** * ffs - find first bit set * @x: the word to search *
include/asm-xtensa/bitops.h +11 -0
@@ -82,6 +82,16 @@ static inline int fls (unsigned int x) return 32 - __cntlz(x); } +/** +* __fls - find last (most-significant) set bit in a long word +* @word: the word to search +* +* Undefined if no set bit exists, so code should check against 0 first. +*/ +static inline unsigned long __fls (unsigned long word) +{ +return 31 -__cntlz (word); +}#else /* Use the generic implementation if we don't have the nsa/nsau instructions. */
@@ -90,6 +100,7 @@ static inline int fls (unsigned int x)# include <asm-generic/bitops/__ffs.h> # include <asm-generic/bitops/ffz.h> # include <asm-generic/bitops/fls.h> +# include <asm-generic/bitops/__fls.h> #endif
include/linux/bitmap.h +19 -16
@@ -137,9 +137,12 @@ extern void bitmap_copy_le(void *dst, const unsigned long *src, int nbits); (1UL<<((nbits) % BITS_PER_LONG))-1 : ~0UL \ ) +#define small_const_nbits (nbits) \ + (__builtin_constant_p(nbits) && (nbits) <= BITS_PER_LONG) +static inline void bitmap_zero (unsigned long *dst,int nbits) { -if (nbits <=BITS_PER_LONG ) +if (small_const_nbits (nbits)) *dst =0UL ;else {int len =BITS_TO_LONGS (nbits) *sizeof (unsigned long );
@@ -150,7 +153,7 @@ static inline void bitmap_zero(unsigned long *dst, int nbits)static inline void bitmap_fill (unsigned long *dst,int nbits) {size_t nlongs =BITS_TO_LONGS (nbits); -if (nlongs >1 ) { +if (!small_const_nbits (nbits)) {int len = (nlongs -1 ) *sizeof (unsigned long );memset (dst,0xff , len); }
@@ -160,7 +163,7 @@ static inline void bitmap_fill(unsigned long *dst, int nbits)static inline void bitmap_copy (unsigned long *dst,const unsigned long *src,int nbits) { -if (nbits <=BITS_PER_LONG ) +if (small_const_nbits (nbits)) *dst = *src;else {int len =BITS_TO_LONGS (nbits) *sizeof (unsigned long );
@@ -171,7 +174,7 @@ static inline void bitmap_copy(unsigned long *dst, const unsigned long *src,static inline void bitmap_and (unsigned long *dst,const unsigned long *src1,const unsigned long *src2,int nbits) { -if (nbits <=BITS_PER_LONG ) +if (small_const_nbits (nbits)) *dst = *src1 & *src2;else __bitmap_and (dst, src1, src2, nbits);
@@ -180,7 +183,7 @@ static inline void bitmap_and(unsigned long *dst, const unsigned long *src1,static inline void bitmap_or (unsigned long *dst,const unsigned long *src1,const unsigned long *src2,int nbits) { -if (nbits <=BITS_PER_LONG ) +if (small_const_nbits (nbits)) *dst = *src1 | *src2;else __bitmap_or (dst, src1, src2, nbits);
@@ -189,7 +192,7 @@ static inline void bitmap_or(unsigned long *dst, const unsigned long *src1,static inline void bitmap_xor (unsigned long *dst,const unsigned long *src1,const unsigned long *src2,int nbits) { -if (nbits <=BITS_PER_LONG ) +if (small_const_nbits (nbits)) *dst = *src1 ^ *src2;else __bitmap_xor (dst, src1, src2, nbits);
@@ -198,7 +201,7 @@ static inline void bitmap_xor(unsigned long *dst, const unsigned long *src1,static inline void bitmap_andnot (unsigned long *dst,const unsigned long *src1,const unsigned long *src2,int nbits) { -if (nbits <=BITS_PER_LONG ) +if (small_const_nbits (nbits)) *dst = *src1 & ~(*src2);else __bitmap_andnot (dst, src1, src2, nbits);
@@ -207,7 +210,7 @@ static inline void bitmap_andnot(unsigned long *dst, const unsigned long *src1,static inline void bitmap_complement (unsigned long *dst,const unsigned long *src,int nbits) { -if (nbits <=BITS_PER_LONG ) +if (small_const_nbits (nbits)) *dst = ~(*src) &BITMAP_LAST_WORD_MASK (nbits);else __bitmap_complement (dst, src, nbits);
@@ -216,7 +219,7 @@ static inline void bitmap_complement(unsigned long *dst, const unsigned long *srstatic inline int bitmap_equal (const unsigned long *src1,const unsigned long *src2,int nbits) { -if (nbits <=BITS_PER_LONG ) +if (small_const_nbits (nbits))return ! ((*src1 ^ *src2) &BITMAP_LAST_WORD_MASK (nbits));else return __bitmap_equal (src1, src2, nbits);
@@ -225,7 +228,7 @@ static inline int bitmap_equal(const unsigned long *src1,static inline int bitmap_intersects (const unsigned long *src1,const unsigned long *src2,int nbits) { -if (nbits <=BITS_PER_LONG ) +if (small_const_nbits (nbits))return ((*src1 & *src2) &BITMAP_LAST_WORD_MASK (nbits)) !=0 ;else return __bitmap_intersects (src1, src2, nbits);
@@ -234,7 +237,7 @@ static inline int bitmap_intersects(const unsigned long *src1,static inline int bitmap_subset (const unsigned long *src1,const unsigned long *src2,int nbits) { -if (nbits <=BITS_PER_LONG ) +if (small_const_nbits (nbits))return ! ((*src1 & ~(*src2)) &BITMAP_LAST_WORD_MASK (nbits));else return __bitmap_subset (src1, src2, nbits);
@@ -242,7 +245,7 @@ static inline int bitmap_subset(const unsigned long *src1,static inline int bitmap_empty (const unsigned long *src,int nbits) { -if (nbits <=BITS_PER_LONG ) +if (small_const_nbits (nbits))return ! (*src &BITMAP_LAST_WORD_MASK (nbits));else return __bitmap_empty (src, nbits);
@@ -250,7 +253,7 @@ static inline int bitmap_empty(const unsigned long *src, int nbits)static inline int bitmap_full (const unsigned long *src,int nbits) { -if (nbits <=BITS_PER_LONG ) +if (small_const_nbits (nbits))return ! (~(*src) &BITMAP_LAST_WORD_MASK (nbits));else return __bitmap_full (src, nbits);
@@ -258,7 +261,7 @@ static inline int bitmap_full(const unsigned long *src, int nbits)static inline int bitmap_weight (const unsigned long *src,int nbits) { -if (nbits <=BITS_PER_LONG ) +if (small_const_nbits (nbits))return hweight_long (*src &BITMAP_LAST_WORD_MASK (nbits));return __bitmap_weight (src, nbits); }
@@ -266,7 +269,7 @@ static inline int bitmap_weight(const unsigned long *src, int nbits)static inline void bitmap_shift_right (unsigned long *dst,const unsigned long *src,int n,int nbits) { -if (nbits <=BITS_PER_LONG ) +if (small_const_nbits (nbits)) *dst = *src >> n;else __bitmap_shift_right (dst, src, n, nbits);
@@ -275,7 +278,7 @@ static inline void bitmap_shift_right(unsigned long *dst,static inline void bitmap_shift_left (unsigned long *dst,const unsigned long *src,int n,int nbits) { -if (nbits <=BITS_PER_LONG ) +if (small_const_nbits (nbits)) *dst = (*src << n) &BITMAP_LAST_WORD_MASK (nbits);else __bitmap_shift_left (dst, src, n, nbits);
include/linux/bitops.h +12 -1
@@ -134,9 +134,20 @@ extern unsigned long find_first_bit(const unsigned long *addr,*/ extern unsigned long find_first_zero_bit (const unsigned long *addr,unsigned long size); -#endif /* CONFIG_GENERIC_FIND_FIRST_BIT */ +#ifdef CONFIG_GENERIC_FIND_LAST_BIT +/** +* find_last_bit - find the last set bit in a memory region +* @addr: The address to start the search at +* @size: The maximum size to search +* +* Returns the bit number of the first set bit, or size. +*/ +extern unsigned long find_last_bit (const unsigned long *addr, +unsigned long size); +#endif /* CONFIG_GENERIC_FIND_LAST_BIT */ +#ifdef CONFIG_GENERIC_FIND_NEXT_BIT /**
include/linux/cpumask.h +91 -130
359 lines not on this page · raw message
include/linux/interrupt.h +1 -1
@@ -109,7 +109,7 @@ extern void enable_irq(unsigned int irq);#if defined(CONFIG_SMP ) && defined(CONFIG_GENERIC_HARDIRQS ) -extern cpumask_t irq_default_affinity; +extern cpumask_var_t irq_default_affinity;extern int irq_set_affinity (unsigned int irq,const struct cpumask *cpumask);extern int irq_can_set_affinity (unsigned int irq);
include/linux/rcuclassic.h +2 -2
@@ -59,8 +59,8 @@ struct rcu_ctrlblk {int signaled;spinlock_t lock____cacheline_internodealigned_in_smp ; -cpumask_t cpumask;/* CPUs that need to switch in order */ -/* for current batch to proceed. */ +DECLARE_BITMAP (cpumask,NR_CPUS );/* CPUs that need to switch for */ +/* current batch to proceed. */ }____cacheline_internodealigned_in_smp ;/* Is batch a before batch b ? */
include/linux/seq_file.h +4 -3
@@ -50,10 +50,11 @@ int seq_path(struct seq_file *, struct path *, char *);int seq_dentry (struct seq_file *,struct dentry *,char *);int seq_path_root (struct seq_file *m,struct path *path,struct path *root,char *esc); -int seq_bitmap (struct seq_file *m,unsigned long *bits,unsigned int nr_bits); -static inline int seq_cpumask (struct seq_file *m,cpumask_t *mask) +int seq_bitmap (struct seq_file *m,const unsigned long *bits, +unsigned int nr_bits); +static inline int seq_cpumask (struct seq_file *m,const struct cpumask *mask) { -return seq_bitmap (m, mask->bits,NR_CPUS ); +return seq_bitmap (m, mask->bits, nr_cpu_ids); }static inline int seq_nodemask (struct seq_file *m,nodemask_t *mask)
include/linux/smp.h +11 -7
@@ -21,6 +21,9 @@ struct call_single_data {u16 priv; }; +/* total number of cpus in this system (may exceed NR_CPUS) */ +extern unsigned int total_cpus; +#ifdef CONFIG_SMP #include <linux/preempt.h>
@@ -64,15 +67,16 @@ extern void smp_cpus_done(unsigned int max_cpus);* Call a function on all other processors */ int smp_call_function (void (*func)(void *info),void *info,int wait); -/* Deprecated: use smp_call_function_many() which uses a cpumask ptr. */ -int smp_call_function_mask (cpumask_t mask,void (*func)(void *info),void *info, -int wait); +void smp_call_function_many (const struct cpumask *mask, +void (*func)(void *info),void *info,bool wait); -static inline void smp_call_function_many (const struct cpumask *mask, -void (*func)(void *info),void *info, -int wait) +/* Deprecated: Use smp_call_function_many which takes a pointer to the mask. */ +static inline int +smp_call_function_mask (cpumask_t mask,void (*func)(void *info),void *info, +int wait) { -smp_call_function_mask (*mask, func, info, wait); +smp_call_function_many (&mask, func, info, wait); +return 0 ; } int smp_call_function_single(int cpuid, void (*func) (void *info), void *info,
include/linux/stop_machine.h +3 -3
@@ -23,7 +23,7 @@* * This can be thought of as a very heavy write lock, equivalent to * grabbing every spinlock in the kernel. */ -int stop_machine (int (*fn)(void *),void *data,const cpumask_t *cpus); +int stop_machine (int (*fn)(void *),void *data,const struct cpumask *cpus);/** * __stop_machine: freeze the machine on all CPUs and run this function
@@ -34,11 +34,11 @@ int stop_machine(int (*fn)(void *), void *data, const cpumask_t *cpus);* Description: This is a special version of the above, which assumes cpus * won't come or go while it's being called. Used by hotplug cpu. */ -int __stop_machine (int (*fn)(void *),void *data,const cpumask_t *cpus); +int __stop_machine (int (*fn)(void *),void *data,const struct cpumask *cpus);#else static inline int stop_machine (int (*fn)(void *),void *data, -const cpumask_t *cpus) +const struct cpumask *cpus) {int ret;local_irq_disable ();
include/linux/threads.h +8 -8
@@ -8,17 +8,17 @@*/ /* -* Maximum supported processors that can run under SMP. This value is -* set via configure setting. The maximum is equal to the size of the -* bitmasks used on that platform, i.e. 32 or 64. Setting this smaller -* saves quite a bit of memory. +* Maximum supported processors. Setting this smaller saves quite a +* bit of memory. Use nr_cpu_ids instead of this except for static bitmaps. */ -#ifdef CONFIG_SMP -#define NR_CPUS CONFIG_NR_CPUS -#else -#define NR_CPUS 1 +#ifndef CONFIG_NR_CPUS +/* FIXME: This should be fixed in the arch's Kconfig */ +#define CONFIG_NR_CPUS 1#endif +/* Places which use this should consider cpumask_var_t. */ +#define NR_CPUS CONFIG_NR_CPUS +#define MIN_THREADS_LEFT_FOR_ROOT 4/*
include/linux/tick.h +2 -2
@@ -84,10 +84,10 @@ static inline void tick_cancel_sched_timer(int cpu) { }# ifdef CONFIG_GENERIC_CLOCKEVENTS_BROADCAST extern struct tick_device *tick_get_broadcast_device (void ); -extern cpumask_t *tick_get_broadcast_mask (void ); +extern struct cpumask *tick_get_broadcast_mask (void );# ifdef CONFIG_TICK_ONESHOT -extern cpumask_t *tick_get_broadcast_oneshot_mask (void ); +extern struct cpumask *tick_get_broadcast_oneshot_mask (void );# endif # endif /* BROADCAST */
init/main.c +4 -9
@@ -371,12 +371,7 @@ EXPORT_SYMBOL(nr_cpu_ids);/* An arch may set nr_cpu_ids earlier if needed, so this would be redundant */ static void __init setup_nr_cpu_ids (void ) { -int cpu, highest_cpu =0 ; - -for_each_possible_cpu (cpu) - highest_cpu = cpu; - - nr_cpu_ids = highest_cpu +1 ; + nr_cpu_ids =find_last_bit (cpumask_bits (cpu_possible_mask),NR_CPUS ) +1 ; }#ifndef CONFIG_HAVE_SETUP_PER_CPU_AREA
@@ -518,9 +513,9 @@ static void __init boot_cpu_init(void) {int cpu =smp_processor_id ();/* Mark the boot cpu "present", "online" etc for SMP and UP case */ -cpu_set (cpu, cpu_online_map); -cpu_set (cpu, cpu_present_map); -cpu_set (cpu, cpu_possible_map); +set_cpu_online (cpu, true); +set_cpu_present (cpu, true); +set_cpu_possible (cpu, true); }void __init __weak smp_setup_processor_id (void )
kernel/compat.c +30 -19
86 lines not on this page · raw message
kernel/cpu.c +99 -45
260 lines not on this page · raw message
kernel/irq/manage.c +9 -2
@@ -16,8 +16,15 @@#include "internals.h" #ifdef CONFIG_SMP +cpumask_var_t irq_default_affinity; -cpumask_t irq_default_affinity =CPU_MASK_ALL ; +static int init_irq_default_affinity (void ) +{ +alloc_cpumask_var (&irq_default_affinity,GFP_KERNEL ); +cpumask_setall (irq_default_affinity); +return 0 ; +} +core_initcall (init_irq_default_affinity);/** * synchronize_irq - wait for pending IRQ handlers (on other CPUs)
@@ -127,7 +134,7 @@ int do_irq_select_affinity(unsigned int irq, struct irq_desc *desc) desc->status &= ~IRQ_AFFINITY_SET ; } -cpumask_and (&desc->affinity, cpu_online_mask, &irq_default_affinity); +cpumask_and (&desc->affinity, cpu_online_mask, irq_default_affinity); set_affinity: desc->chip->set_affinity (irq, &desc->affinity);
kernel/irq/proc.c +22 -12
@@ -20,7 +20,7 @@ static struct proc_dir_entry *root_irq_dir;static int irq_affinity_proc_show (struct seq_file *m,void *v) {struct irq_desc *desc =irq_to_desc ((long )m->private); -cpumask_t *mask = &desc->affinity; +const struct cpumask *mask = &desc->affinity;#ifdef CONFIG_GENERIC_PENDING_IRQ if (desc->status &IRQ_MOVE_PENDING )
@@ -54,7 +54,7 @@ static ssize_t irq_affinity_proc_write(struct file *file,if (err) goto free_cpumask; -if (!is_affinity_mask_valid (*new_value)) { +if (!is_affinity_mask_valid (new_value)) { err = -EINVAL ; goto free_cpumask; }
@@ -93,7 +93,7 @@ static const struct file_operations irq_affinity_proc_fops = {static int default_affinity_show (struct seq_file *m,void *v) { -seq_cpumask (m, &irq_default_affinity); +seq_cpumask (m, irq_default_affinity);seq_putc (m,'\n' );return 0 ; }
@@ -101,27 +101,37 @@ static int default_affinity_show(struct seq_file *m, void *v)static ssize_t default_affinity_write (struct file *file,const char __user *buffer,size_t count,loff_t *ppos) { -cpumask_t new_value; +cpumask_var_t new_value;int err; - err =cpumask_parse_user (buffer, count, &new_value); +if (!alloc_cpumask_var (&new_value,GFP_KERNEL )) +return -ENOMEM ; + + err =cpumask_parse_user (buffer, count, new_value);if (err) -return err; + goto out; -if (!is_affinity_mask_valid (new_value)) -return -EINVAL ; +if (!is_affinity_mask_valid (new_value)) { + err = -EINVAL ; + goto out; + }/* * Do not allow disabling IRQs completely - it's a too easy * way to make the system unusable accidentally :-) At least * one online CPU still has to be targeted. */ -if (!cpus_intersects (new_value, cpu_online_map)) -return -EINVAL ; +if (!cpumask_intersects (new_value, cpu_online_mask)) { + err = -EINVAL ; + goto out; + } - irq_default_affinity = new_value; +cpumask_copy (irq_default_affinity, new_value); + err = count; -return count; +out: +free_cpumask_var (new_value); +return err; }static int default_affinity_open (struct inode *inode,struct file *file)
kernel/kexec.c +1 -1
@@ -1116,7 +1116,7 @@ void crash_save_cpu(struct pt_regs *regs, int cpu)struct elf_prstatus prstatus;u32 *buf; -if ((cpu <0 ) || (cpu >=NR_CPUS )) +if ((cpu <0 ) || (cpu >= nr_cpu_ids))return ;/* Using ELF notes here is opportunistic.
kernel/power/poweroff.c +1 -1
@@ -27,7 +27,7 @@ static DECLARE_WORK(poweroff_work, do_poweroff);static void handle_poweroff (int key,struct tty_struct *tty) {/* run sysrq poweroff on boot cpu */ -schedule_work_on (first_cpu (cpu_online_map), &poweroff_work); +schedule_work_on (cpumask_first (cpu_online_mask), &poweroff_work); }static struct sysrq_key_op sysrq_poweroff_op = {
kernel/profile.c +25 -13
@@ -45,7 +45,7 @@ static unsigned long prof_len, prof_shift;int prof_on__read_mostly ;EXPORT_SYMBOL_GPL (prof_on); -static cpumask_t prof_cpu_mask =CPU_MASK_ALL ; +static cpumask_var_t prof_cpu_mask;#ifdef CONFIG_SMP static DEFINE_PER_CPU(struct profile_hit *[2], cpu_profile_hits); static DEFINE_PER_CPU(int, cpu_profile_flip);
@@ -113,9 +113,13 @@ int __ref profile_init(void) buffer_bytes = prof_len*sizeof (atomic_t);if (!slab_is_available ()) { prof_buffer =alloc_bootmem (buffer_bytes); +alloc_bootmem_cpumask_var (&prof_cpu_mask);return 0 ; } +if (!alloc_cpumask_var (&prof_cpu_mask,GFP_KERNEL )) +return -ENOMEM ; + prof_buffer =kzalloc (buffer_bytes,GFP_KERNEL );if (prof_buffer)return 0 ;
@@ -128,6 +132,7 @@ int __ref profile_init(void)if (prof_buffer)return 0 ; +free_cpumask_var (prof_cpu_mask);return -ENOMEM ; }
@@ -386,13 +391,15 @@ out_free:return NOTIFY_BAD ;case CPU_ONLINE :case CPU_ONLINE_FROZEN : -cpu_set (cpu, prof_cpu_mask); +if (prof_cpu_mask !=NULL ) +cpumask_set_cpu (cpu, prof_cpu_mask);break ;case CPU_UP_CANCELED :case CPU_UP_CANCELED_FROZEN :case CPU_DEAD :case CPU_DEAD_FROZEN : -cpu_clear (cpu, prof_cpu_mask); +if (prof_cpu_mask !=NULL ) +cpumask_clear_cpu (cpu, prof_cpu_mask);if (per_cpu (cpu_profile_hits, cpu)[0 ]) { page =virt_to_page (per_cpu (cpu_profile_hits, cpu)[0 ]);per_cpu (cpu_profile_hits, cpu)[0 ] =NULL ;
@@ -430,7 +437,8 @@ void profile_tick(int type)if (type ==CPU_PROFILING && timer_hook)timer_hook (regs); -if (!user_mode (regs) &&cpu_isset (smp_processor_id (), prof_cpu_mask)) +if (!user_mode (regs) && prof_cpu_mask !=NULL && +cpumask_test_cpu (smp_processor_id (), prof_cpu_mask))profile_hit (type, (void *)profile_pc (regs)); }
@@ -442,7 +450,7 @@ void profile_tick(int type)static int prof_cpu_mask_read_proc (char *page,char **start,off_t off,int count,int *eof,void *data) { -int len =cpumask_scnprintf (page, count, (cpumask_t *)data); +int len =cpumask_scnprintf (page, count, data);if (count - len <2 )return -EINVAL ; len +=sprintf (page + len,"\n" );
@@ -452,16 +460,20 @@ static int prof_cpu_mask_read_proc(char *page, char **start, off_t off,static int prof_cpu_mask_write_proc (struct file *file,const char __user *buffer,unsigned long count,void *data) { -cpumask_t *mask = (cpumask_t *)data; +struct cpumask *mask = data;unsigned long full_count = count, err; -cpumask_t new_value; +cpumask_var_t new_value; - err =cpumask_parse_user (buffer, count, &new_value); -if (err) -return err; +if (!alloc_cpumask_var (&new_value,GFP_KERNEL )) +return -ENOMEM ; - *mask = new_value; -return full_count; + err =cpumask_parse_user (buffer, count, new_value); +if (!err) { +cpumask_copy (mask, new_value); + err = full_count; + } +free_cpumask_var (new_value); +return err; }void create_prof_cpu_mask (struct proc_dir_entry *root_irq_dir)
@@ -472,7 +484,7 @@ void create_prof_cpu_mask(struct proc_dir_entry *root_irq_dir) entry =create_proc_entry ("prof_cpu_mask" ,0600 , root_irq_dir);if (!entry)return ; - entry->data = (void *)&prof_cpu_mask; + entry->data = prof_cpu_mask; entry->read_proc = prof_cpu_mask_read_proc; entry->write_proc = prof_cpu_mask_write_proc; }
kernel/rcuclassic.c +17 -15
@@ -63,14 +63,14 @@ static struct rcu_ctrlblk rcu_ctrlblk = { .completed = -300, .pending = -300, .lock = __SPIN_LOCK_UNLOCKED(&rcu_ctrlblk.lock), - .cpumask = CPU_MASK_NONE, + .cpumask = CPU_BITS_NONE, };static struct rcu_ctrlblk rcu_bh_ctrlblk = { .cur =-300 , .completed =-300 , .pending =-300 , .lock =__SPIN_LOCK_UNLOCKED (&rcu_bh_ctrlblk.lock), - .cpumask =CPU_MASK_NONE , + .cpumask =CPU_BITS_NONE , }; DEFINE_PER_CPU(struct rcu_data, rcu_data) = { 0L };
@@ -85,7 +85,6 @@ static void force_quiescent_state(struct rcu_data *rdp, struct rcu_ctrlblk *rcp) {int cpu; -cpumask_t cpumask;unsigned long flags;set_need_resched ();
@@ -96,10 +95,10 @@ static void force_quiescent_state(struct rcu_data *rdp,* Don't send IPI to itself. With irqs disabled, * rdp->cpu is the current cpu. * -* cpu_online_map is updated by the _cpu_down() +* cpu_online_mask is updated by the _cpu_down() * using __stop_machine(). Since we're in irqs disabled * section, __stop_machine() is not exectuting, hence -* the cpu_online_map is stable. +* the cpu_online_mask is stable. * * However, a cpu might have been offlined _just_ before * we disabled irqs while entering here.
@@ -107,13 +106,14 @@ static void force_quiescent_state(struct rcu_data *rdp,* notification, leading to the offlined cpu's bit * being set in the rcp->cpumask. * -* Hence cpumask = (rcp->cpumask & cpu_online_map) to prevent +* Hence cpumask = (rcp->cpumask & cpu_online_mask) to prevent * sending smp_reschedule() to an offlined CPU. */ -cpus_and (cpumask, rcp->cpumask, cpu_online_map); -cpu_clear (rdp->cpu, cpumask); -for_each_cpu_mask_nr (cpu, cpumask) -smp_send_reschedule (cpu); +for_each_cpu_and (cpu, +to_cpumask (rcp->cpumask), cpu_online_mask) { +if (cpu != rdp->cpu) +smp_send_reschedule (cpu); + } }spin_unlock_irqrestore (&rcp->lock, flags); }
@@ -193,7 +193,7 @@ static void print_other_cpu_stall(struct rcu_ctrlblk *rcp)printk (KERN_ERR "INFO: RCU detected CPU stalls:" );for_each_possible_cpu (cpu) { -if (cpu_isset (cpu, rcp->cpumask))
Lines of this file are not shown. View the raw message.
kernel/sched.c +15 -38
53 lines not on this page · raw message
kernel/smp.c +55 -90
145 lines not on this page · raw message
kernel/time/tick-broadcast.c +59 -56
115 lines not on this page · raw message
kernel/trace/ring_buffer.c +25 -17
42 lines not on this page · raw message
kernel/trace/trace.c +45 -27
72 lines not on this page · raw message
lib/cpumask.c +59 -3
62 lines not on this page · raw message
lib/find_last_bit.c new file +45 -0
45 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
Re: [git pull] cpus4096 tree, part 3
Quoted 3 lines · Ingo Molnar
ok. The pending regressions are all fixed now, and i've just finished my standard tests on the latest tree and all the tests passed fine.
Ok, pulled and pushed out.
Has anybody looked at what the stack size is with MAXSMP set with an allyesconfig? And what areas are still problematic, if any? Are we going to have some code-paths that still essentially have 1kB+ of stack space just because they haven't been converted and still have the cpu mask on stack?
Linus
Re: [git pull] cpus4096 tree, part 3
Quoted 6 lines · Linus Torvalds
Quoted 3 lines · Ingo Molnar
ok. The pending regressions are all fixed now, and i've just finished my standard tests on the latest tree and all the tests passed fine.
Ok, pulled and pushed out.
thanks!
Quoted 5 lines
Has anybody looked at what the stack size is with MAXSMP set with an allyesconfig? And what areas are still problematic, if any? Are we going to have some code-paths that still essentially have 1kB+ of stack space just because they haven't been converted and still have the cpu mask on stack?
ok, indeed testing of that is in order now.
I'll check what the worst-case runtime stack footprint is for an allyesconfig 64-bit bootup - that should be the worst-case scenario on x86. We have a low number of leftover places, but the serious ones like the IPI paths, which triggered stack overflows in the past, got all fixed.
The test is underway with:
CONFIG_64BIT=y
CONFIG_NR_CPUS=4096
CONFIG_MAXSMP=y
Ingo
Re: [git pull] cpus4096 tree, part 3
Quoted 8 lines · Ingo Molnar
Quoted 6 lines
Has anybody looked at what the stack size is with MAXSMP set with an allyesconfig? And what areas are still problematic, if any? Are we going to have some code-paths that still essentially have 1kB+ of stack space just because they haven't been converted and still have the cpu mask on stack?
ok, indeed testing of that is in order now.
Well, since I can compile a allyesconfig pretty quickly, I did the static part. It looks better than it used to, and I think most of the huge stacks are totally unrealted to cpu masks. But not all.
But it looks like we have a few:
- flush_tlb_current_task:
cpumask_t cpu_mask; - flush_tlb_mm: cpumask_t cpu_mask; - local_cpus_show: cpumask_t mask; - local_cpulist_show: cpumask_t mask;
- acpi_cpufreq_target:
cpumask_t online_policy_cpus
and then we have a number of things that have "struct cpufreq_policy" on the stack, and those things have two cpumask_t's in each.
The rest of the high-stack-usage cases - from a _very_ quick look - seem to be unrelated to CPU masks, but in the "more than 1kB of stack" group about a third (wild handwaving eyeballing) of them do seem to be related to cpumask.
And those things (VM, ACPI and PCI) are easily triggered in real use and don't need odd hardware. So I think they need to be fixed before 2.6.29, otherwise I'll have to disable MAXSMP and again limit MAX_CPU back to 128 or something.
Linus
Re: [git pull] cpus4096 tree, part 3
Quoted 4 lines · Linus Torvalds
And those things (VM, ACPI and PCI) are easily triggered in real use and don't need odd hardware. So I think they need to be fixed before 2.6.29, otherwise I'll have to disable MAXSMP and again limit
MAX_CPUback to 128 or something.
agreed.
Ingo
Re: [git pull] cpus4096 tree, part 3
In reply to Linus TorvaldsQuoted 21 lines · On Sunday 04 January 2009 07:26:03 Linus Torvalds wrote:
Quoted 8 lines · Ingo Molnar
Quoted 6 lines
Has anybody looked at what the stack size is with MAXSMP set with an allyesconfig? And what areas are still problematic, if any? Are we going to have some code-paths that still essentially have 1kB+ of stack space just because they haven't been converted and still have the cpu mask on stack?
ok, indeed testing of that is in order now.
Well, since I can compile a allyesconfig pretty quickly, I did the static part. It looks better than it used to, and I think most of the huge stacks are totally unrealted to cpu masks. But not all.
But it looks like we have a few:
-
flush_tlb_current_task:cpumask_t cpu_mask; - flush_tlb_mm: cpumask_t cpu_mask;
...
Quoted 2 lines
-
acpi_cpufreq_target:
cpumask_tonline_policy_cpus
Mike? These are x86-specific...
Quoted 4 lines
-
local_cpus_show:cpumask_t mask; - local_cpulist_show: cpumask_t mask;
Yes, this removal is still in my queue. I'll double-check that all the
archs have the new "cpumask_of_pcibus". (cpumask:replace-and-remove-pcibus_to_cpumask.patch "cpumask: remove the now-obsoleted pcibus_to_cpumask()").
Quoted 2 lines
and then we have a number of things that have "struct
cpufreq_policy" on the stack, and those things have two cpumask_t's in each.
Yep, we have the conversion for that too. Mike, it's cpumask:convert-drivers_acpi.patch "cpumask: convert struct cpufreq_policy to cpumask_var_t."
Quoted 4 lines
The rest of the high-stack-usage cases - from a _very_ quick look - seem to be unrelated to CPU masks, but in the "more than 1kB of stack" group about a third (wild handwaving eyeballing) of them do seem to be related to cpumask.
Mike was tracking this; I think he has a script to set NR_CPUS small then large and dump the changes.
Cheers,
Rusty.
Re: [git pull] cpus4096 tree, part 3
Quoted 18 lines · Rusty Russell
Quoted 11 lines · On Sunday 04 January 2009 07:26:03 Linus Torvalds wrote:
Quoted 6 lines · Ingo Molnar
Quoted 5 lines
Has anybody looked at what the stack size is with MAXSMP set with an allyesconfig? And what areas are still problematic, if any? Are we going to have some code-paths that still essentially have 1kB+ of stack space just because they haven't been converted and still have the cpu mask on stack?
ok, indeed testing of that is in order now.
Well, since I can compile a allyesconfig pretty quickly, I did the static part. It looks better than it used to, and I think most of the huge stacks are totally unrealted to cpu masks. But not all.
Quoted 6 lines
But it looks like we have a few:
-
flush_tlb_current_task:cpumask_t cpu_mask; - flush_tlb_mm: cpumask_t cpu_mask;
Quoted 5 lines
...
Quoted 2 lines
-
acpi_cpufreq_target:
cpumask_tonline_policy_cpusMike? These are x86-specific...
I've been testing the heck out of it... ;-)
Quoted 5 lines
Quoted 5 lines
-
local_cpus_show:cpumask_t mask; - local_cpulist_show: cpumask_t mask;
Yes, these are in my "real soon now" patchset. Trivial.
Quoted 9 lines
Yes, this removal is still in my queue. I'll double-check that all the
archs have the new "cpumask_of_pcibus". (cpumask:replace-and-remove-pcibus_to_cpumask.patch "cpumask: remove the now-obsoletedpcibus_to_cpumask()").Quoted 2 lines
and then we have a number of things that have "struct
cpufreq_policy" on the stack, and those things have two cpumask_t's in each.Yep, we have the conversion for that too. Mike, it's cpumask:convert-drivers_acpi.patch "cpumask: convert struct
cpufreq_policytocpumask_var_t."
That's part of what I'm testing above.
Quoted 7 lines
Quoted 4 lines
The rest of the high-stack-usage cases - from a _very_ quick look - seem to be unrelated to CPU masks, but in the "more than 1kB of stack" group about a third (wild handwaving eyeballing) of them do seem to be related to cpumask.
Mike was tracking this; I think he has a script to set
NR_CPUSsmall then large and dump the changes.
It's looking pretty good, only 11 > 1k and 19 more > 512.
Thanks,
Mike
Re: [git pull] cpus4096 tree, part 3
In reply to Ingo MolnarQuoted 5 lines · Ingo Molnar
The test is underway with:
CONFIG_64BIT=y
CONFIG_NR_CPUS=4096
CONFIG_MAXSMP=y
the worst-case stack footprint i was able to trigger on an allyesconfig bootup (instrumented via CONFIG_STACK_TRACER=y and the 'stacktrace' boot option) was 5.6K:
# ls -l /debug/tracing/*stack* -rw-r--r-- 1 root root 0 2009-01-03 22:42 /debug/tracing/stack_max_size -r--r--r-- 1 root root 0 2009-01-03 22:42 /debug/tracing/stack_trace
# cat /debug/tracing/*stack*
5624
Depth Size Location (38 entries)
----- ---- --------
0) 5496 232 lookup_address+0x51/0x20e
1) 5264 496 __change_page_attr_set_clr+0xa7/0xa15
2) 4768 128 kernel_map_pages+0x121/0x140
3) 4640 224 get_page_from_freelist+0x4bb/0x6ec
4) 4416 160 __alloc_pages_internal+0x150/0x4af
5) 4256 48 alloc_pages_current+0xbe/0xc7
6) 4208 16 alloc_slab_page+0x1e/0x6e
7) 4192 64 new_slab+0x51/0x1e2
8) 4128 96 __slab_alloc+0x260/0x3fa
9) 4032 80 kmem_cache_alloc+0xa8/0x120
10) 3952 48 radix_tree_preload+0x38/0x86
11) 3904 64 add_to_page_cache_locked+0x51/0xed
12) 3840 32 add_to_page_cache_lru+0x31/0x6b
13) 3808 64 find_or_create_page+0x56/0x7d
14) 3744 80 __getblk+0x136/0x262
15) 3664 32 __bread+0x13/0x82
16) 3632 64 ext3_get_branch+0x7b/0xef
17) 3568 368 ext3_get_blocks_handle+0xa2/0x907
18) 3200 96 ext3_get_block+0xc3/0x101
19) 3104 224 do_mpage_readpage+0x1ad/0x4d0
20) 2880 240 mpage_readpages+0xb6/0xf9
21) 2640 16 ext3_readpages+0x1f/0x21
22) 2624 144 __do_page_cache_readahead+0x147/0x1bd
23) 2480 48 do_page_cache_readahead+0x5b/0x68
24) 2432 112 filemap_fault+0x176/0x34b
25) 2320 160 __do_fault+0x58/0x410
26) 2160 176 handle_mm_fault+0x4b2/0xa3e
27) 1984 800 do_page_fault+0x86d/0xcec
28) 1184 208 page_fault+0x25/0x30
29) 976 16 clear_user+0x30/0x38
30) 960 288 load_elf_binary+0x61c/0x18f0
31) 672 80 search_binary_handler+0xd9/0x279
32) 592 176 load_script+0x1b3/0x1c8
33) 416 80 search_binary_handler+0xd9/0x279
34) 336 96 do_execve+0x1df/0x296
35) 240 64 sys_execve+0x43/0x5e
36) 176 176 stub_execve+0x6a/0xc0None of the cpumask_t callsites showed up - but i agree that they should be fixed, 512 bytes of cpumask_t on the stack is not good no matter how one looks at it, it will only make an already strained kernel stack footprint situation worse.
This one looks a bit high:
1) 5264 496 __change_page_attr_set_clr+0xa7/0xa15
(Venki, Suresh and Arjan Cc:-ed)
This one isnt too nice either:
27) 1984 800 do_page_fault+0x86d/0xcec
not sure why it happens - will investigate tomorrow, it's getting late here.
Ingo
Re: [git pull] cpus4096 tree, part 3
Quoted 5 lines · Ingo Molnar
This one looks a bit high:
1) 5264 496
__change_page_attr_set_clr+0xa7/0xa15
Quoted 8 lines
(Venki, Suresh and Arjan Cc:-ed)
This one isnt too nice either:
27) 1984 800
do_page_fault+0x86d/0xcecnot sure why it happens - will investigate tomorrow, it's getting late here.
The most common case tends to be insane gcc inlining (vmalloc_fault and spurious_fault), and then gcc not re-using stack slots even if they have no overlap in usage. Some people continue to claim that gcc reuses them, but it's definitely not the case in any complex situation, so I suspect the re-use is probably purely for some simple spilling case, not for variables allocated on the stack.
do_page_fault() in particular has a lot of gunk in it for the special cases.
What happened to Nick's cleanup patch to do_page_fault (a month or two ago? I complained about some of the issues in his first version and asked for some further cleanups, but I think that whole discussion ended with him saying "I am going to add those changes that you suggested (in fact, I already have)".
And then I didn't see anything further. Maybe I just missed the end result. Or maybe we have it in some -mm branch or something?
Linus
Re: [git pull] cpus4096 tree, part 3
Quoted 8 lines · Linus Torvalds
What happened to Nick's cleanup patch to
do_page_fault(a month or two ago? I complained about some of the issues in his first version and asked for some further cleanups, but I think that whole discussion ended with him saying "I am going to add those changes that you suggested (in fact, I already have)".And then I didn't see anything further. Maybe I just missed the end result. Or maybe we have it in some -mm branch or something?
they would have been in tip/x86/mm and would be upstream now had Nick re-sent a v2 series but that never happened. I think they might have fallen victim to a serious attention deficit caused by the SLQB patch ;-)
Ingo
Re: [git pull] cpus4096 tree, part 3
Quoted 9 lines · Ingo Molnar
Quoted 6 lines · Linus Torvalds
What happened to Nick's cleanup patch to
do_page_fault(a month or two ago? I complained about some of the issues in his first version and asked for some further cleanups, but I think that whole discussion ended with him saying "I am going to add those changes that you suggested (in fact, I already have)".
Quoted 6 lines
Quoted 2 lines
And then I didn't see anything further. Maybe I just missed the end result. Or maybe we have it in some -mm branch or something?
they would have been in tip/x86/mm and would be upstream now had Nick re-sent a v2 series but that never happened. I think they might have fallen victim to a serious attention deficit caused by the SLQB patch ;-)
Well, I already added Linus's suggestions but didn't submit it because there was a bit of work going on in that file as far as I could see, both in the x86 tree and in -mm:
It isn't a big deal to resolve either way, but I don't want to make Andrew's life harder.
[Yes OK now I'm the guilty one of pushing in an x86 patch not via the x86 tree ;) This one is easy to break in pieces, but I didn't want to create a dependency between the trees]
I didn't really consider it to be urgent, so I was waiting for that patch to go in, but I was still hoping to get this into 2.6.29... This is what it looks like now with your suggestions, and just merged it to your current tree (untested).
I'll cc the linux-arch list here too, because it might be nice to keep these things as structurally similar as possible (and they'll all want to look at the -mm patch above, although I'll probably end up having to write the patches!).
---
Optimise x86's do_page_fault (C entry point for the page fault path).
gcc isn't _all_ that smart about spilling registers to stack or reusing stack slots, even with branch annotations. do_page_fault contained a lot of functionality, so split unlikely paths into their own functions, and mark them as noinline just to be sure. I consider this actually to be somewhat of a cleanup too: the main function now contains about half the number of lines.
Also, ensure the order of arguments to functions is always the same: regs, addr, error_code. This can reduce code size a tiny bit, and just looks neater too.
Add a couple of branch annotations.
One real behavioural difference this makes is that the OOM-init-task case will no longer loop around the page fault handler, but will return to userspace and presumably retry the fault. Effectively the same macro-behaviour, but it is a notable difference. Such change in behaviour should disappear after the "call oom killer from page fault" patch.
Before:
do_page_fault:
subq $360, %rsp #,
After:
do_page_fault:
subq $56, %rsp #,
bloat-o-meter:
add/remove: 8/0 grow/shrink: 0/1 up/down: 2222/-1680 (542)
function old new delta __bad_area_nosemaphore - 506 +506 no_context - 474 +474 vmalloc_fault - 424 +424 spurious_fault - 358 +358 mm_fault_error - 272 +272 bad_area_access_error - 89 +89 bad_area - 89 +89 bad_area_nosemaphore - 10 +10
do_page_fault 2464 784 -1680
Yes, the total size increases by 542 bytes, due to the extra function calls. But these will very rarely be called (except for vmalloc_fault) in a normal workload. Importantly, do_page_fault is less than 1/3rd it's original size. Existing gotos and branch hints did move a lot of the infrequently used text out of the fastpath, but that's even further improved after this patch.
--- arch/x86/mm/fault.c | 458 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 265 insertions(+), 193 deletions(-)
arch/x86/mm/fault.c +265 -193
@@ -91,8 +91,8 @@ static inline int notify_page_fault(stru* * Opcode checker based on code by Richard Brunner */ -static int is_prefetch (struct pt_regs *regs,unsigned long addr, -unsigned long error_code) +static int is_prefetch (struct pt_regs *regs,unsigned long error_code, +unsigned long addr) {unsigned char *instr;int scan_more =1 ;
@@ -409,15 +409,15 @@ static void show_fault_oops(struct pt_re }#ifdef CONFIG_X86_64 -static noinline void pgtable_bad(unsigned long address, struct pt_regs *regs, -unsigned long error_code) +static noinline void pgtable_bad(struct pt_regs *regs, +unsigned long error_code,unsigned long address) {unsigned long flags =oops_begin ();int sig =SIGKILL ; -struct task_struct *tsk; +struct task_struct *tsk = current;printk (KERN_ALERT "%s: Corrupted page table at address %lx\n" , - current->comm, address); + tsk->comm, address);dump_pagetable (address); tsk = current; tsk->thread.cr2 = address;
@@ -429,6 +429,200 @@ static noinline void pgtable_bad(unsigne }#endif +static noinline void no_context(struct pt_regs *regs, +unsigned long error_code,unsigned long address) +{ +struct task_struct *tsk = current; +#ifdef CONFIG_X86_64 +unsigned long flags; +int sig; +#endif + +/* Are we prepared to handle this kernel fault? */ +if (fixup_exception (regs)) +return ; + +/* +* X86_32 +* Valid to do another page fault here, because if this fault +* had been triggered by is_prefetch fixup_exception would have +* handled it. +* +* X86_64 +* Hall of shame of CPU/BIOS bugs. +*/ +if (is_prefetch (regs, error_code, address)) +return ; + +if (is_errata93 (regs, address)) +return ; + +/* +* Oops. The kernel tried to access some bad page. We'll have to +* terminate things with extreme prejudice. +*/ +#ifdef CONFIG_X86_32 +bust_spinlocks (1 ); +#else + flags =oops_begin (); +#endif + +show_fault_oops (regs, error_code, address); + + tsk->thread.cr2 = address; + tsk->thread.trap_no =14 ; + tsk->thread.error_code = error_code; + +#ifdef CONFIG_X86_32 +die ("Oops" , regs, error_code); +bust_spinlocks (0 ); +do_exit (SIGKILL ); +#else + sig =SIGKILL ; +if (__die ("Oops" , regs, error_code)) + sig =0 ; +/* Executive summary in case the body of the oops scrolled away */ +printk (KERN_EMERG "CR2: %016lx\n" , address); +oops_end (flags, regs, sig); +#endif +} + +static void __bad_area_nosemaphore (struct pt_regs *regs, +unsigned long error_code,unsigned long address, +int si_code) +{ +struct task_struct *tsk = current; + +/* User mode accesses just cause a SIGSEGV */ +if (error_code &PF_USER ) { +/* +* It's possible to have interrupts off here. +*/ +local_irq_enable (); + +/* +* Valid to do another page fault here because this one came +* from user space. +*/ +if (is_prefetch (regs, error_code, address)) +return ; + +if (is_errata100 (regs, address)) +return ; + +if (show_unhandled_signals &&unhandled_signal (tsk,SIGSEGV ) && +printk_ratelimit ()) { +printk ( +"%s%s[%d]: segfault at %lx ip %p sp %p error %lx" , +task_pid_nr (tsk) >1 ?KERN_INFO :KERN_EMERG , + tsk->comm,task_pid_nr (tsk), address, + (void *) regs->ip, (void *) regs->sp, error_code); +print_vma_addr (" in " , regs->ip); +printk ("\n" ); + } + + tsk->thread.cr2 = address; +/* Kernel addresses are always protection faults */ + tsk->thread.error_code = error_code | (address >=TASK_SIZE ); + tsk->thread.trap_no =14 ; +force_sig_info_fault (SIGSEGV , si_code, address, tsk); +return ; + } + +if (is_f00f_bug (regs, address)) +return ; + +no_context (regs, error_code, address); +} + +static noinline void bad_area_nosemaphore(struct pt_regs *regs, +unsigned long error_code,unsigned long address) +{ +__bad_area_nosemaphore (regs, error_code, address,SEGV_MAPERR ); +} + +static void __bad_area (struct pt_regs *regs, +unsigned long error_code,unsigned long address, +int si_code) +{ +struct mm_struct *mm = current->mm; + +/* +* Something tried to access memory that isn't in our memory map.. +* Fix it, but check if it's kernel or user first.. +*/ +up_read (&mm->mmap_sem); + +__bad_area_nosemaphore (regs, error_code, address, si_code); +} + +static noinline void bad_area(struct pt_regs *regs, +unsigned long error_code,unsigned long address) +{ +__bad_area (regs, error_code, address,SEGV_MAPERR ); +} + +static noinline void bad_area_access_error(struct pt_regs *regs, +unsigned long error_code,unsigned long address) +{ +__bad_area (regs, error_code, address,SEGV_ACCERR ); +} + +/* TODO: fixup for "mm-invoke-oom-killer-from-page-fault.patch" */ +static void out_of_memory (struct pt_regs *regs, +unsigned long error_code,unsigned long address) +{ +struct task_struct *tsk = current; +struct mm_struct *mm = tsk->mm; +/* +* We ran out of memory, or some other thing happened to us that made +* us unable to handle the page fault gracefully. +*/ +up_read (&mm->mmap_sem); +if (is_global_init (tsk)) { +yield (); +return ; + } + +printk ("VM: killing process %s\n" , tsk->comm); +if (error_code &PF_USER ) +do_group_exit (SIGKILL ); +no_context (regs, error_code, address); +} + +static void do_sigbus (struct pt_regs *regs, +unsigned long error_code,unsigned long address) +{ +struct task_struct *tsk = current; +struct mm_struct *mm = tsk->mm; + +up_read (&mm->mmap_sem); + +/* Kernel mode? Handle exceptions or die */ +if (!(error_code &PF_USER )) +no_context (regs, error_code, address); +#ifdef CONFIG_X86_32 +/* User space => ok to do another page fault */ +if (is_prefetch (regs, error_code, address)) +return ; +#endif + tsk->thread.cr2 = address; + tsk->thread.error_code = error_code; + tsk->thread.trap_no =14 ; +force_sig_info_fault (SIGBUS ,BUS_ADRERR , address, tsk); +} + +static noinline void mm_fault_error(struct pt_regs *regs, +unsigned long error_code,unsigned long address,unsigned int fault) +{ +if (fault &VM_FAULT_OOM ) +out_of_memory (regs, error_code, address); +else if (fault &VM_FAULT_SIGBUS ) +do_sigbus (regs, error_code, address); +else +BUG (); +} +static int spurious_fault_check (unsigned long error_code,pte_t *pte) {if ((error_code &PF_WRITE ) && !pte_write (*pte))
@@ -448,8 +642,8 @@ static int spurious_fault_check(unsigned* There are no security implications to leaving a stale TLB when * increasing the permissions on a page. */ -static int spurious_fault (unsigned long address, -unsigned long error_code) +static noinline int spurious_fault(unsigned long error_code, +unsigned long address) {pgd_t *pgd;pud_t *pud;
@@ -494,7 +688,7 @@ static int spurious_fault(unsigned long* * This assumes no large pages in there. */ -static int vmalloc_fault (unsigned long address) +static noinline int vmalloc_fault(unsigned long address) {#ifdef CONFIG_X86_32 unsigned long pgd_paddr;
@@ -573,6 +767,25 @@ static int vmalloc_fault(unsigned long aint show_unhandled_signals =1 ; +static inline int access_error (unsigned long error_code,int write, +struct vm_area_struct *vma) +{ +if (write) { +/* write, present and write, not present */ +if (unlikely (!(vma->vm_flags &VM_WRITE ))) +return 1 ; + }else if (unlikely (error_code &PF_PROT )) { +/* read, present */ +return 1 ; + }else { +/* read, not present */ +if (unlikely (!(vma->vm_flags & (VM_READ |VM_EXEC |VM_WRITE )))) +return 1 ; + } + +return 0 ; +} +/* * This routine handles page faults. It determines the address, * and the problem, and then passes it off to one of the appropriate
@@ -583,16 +796,12 @@ asmlinkage#endif void __kprobes do_page_fault (struct pt_regs *regs,unsigned long error_code) { +unsigned long address;struct task_struct *tsk;struct mm_struct *mm;struct vm_area_struct *vma; -unsigned long address; -int write, si_code; +int write;int fault; -#ifdef CONFIG_X86_64 -unsigned long flags; -int sig; -#endif tsk = current; mm = tsk->mm;
@@ -601,9 +810,7 @@ void __kprobes do_page_fault(struct pt_r/* get the address */ address =read_cr2 (); - si_code =SEGV_MAPERR ; - -if (notify_page_fault (regs)) +if (unlikely (notify_page_fault (regs)))return ;if (unlikely (kmmio_fault (regs, address)))return ;
@@ -638,10 +845,10 @@ void __kprobes do_page_fault(struct pt_r* Don't take the mm semaphore here. If we fixup a prefetch * fault we could otherwise deadlock. */ - goto bad_area_nosemaphore; +bad_area_nosemaphore (regs, error_code, address); +return ; } -/* * It's safe to allow irq's after cr2 has been saved and the * vmalloc fault has been handled.
@@ -657,17 +864,18 @@ void __kprobes do_page_fault(struct pt_r#ifdef CONFIG_X86_64 if (unlikely (error_code &PF_RSVD )) -pgtable_bad (address, regs, error_code); +pgtable_bad (regs, error_code, address);#endif /* * If we're in an interrupt, have no user context or are running in an * atomic region then we must not take the fault. */ -if (unlikely (in_atomic () || !mm)) - goto bad_area_nosemaphore; +if (unlikely (in_atomic () || !mm)) { +bad_area_nosemaphore (regs, error_code, address); +return ; + } -again:/* * When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the
@@ -684,20 +892,26 @@ again:* source. If this is invalid we can skip the address space check, * thus avoiding the deadlock. */ -if (!down_read_trylock (&mm->mmap_sem)) { +if (unlikely (!down_read_trylock (&mm->mmap_sem))) {if ((error_code &PF_USER ) ==0 && - !search_exception_tables (regs->ip)) - goto bad_area_nosemaphore; + !search_exception_tables (regs->ip)) { +bad_area_nosemaphore (regs, error_code, address); +return ; + }down_read (&mm->mmap_sem); } vma =find_vma (mm, address); -if (!vma) - goto bad_area; -if (vma->vm_start <= address) +if (unlikely (!vma)) { +bad_area (regs, error_code, address); +return ; + } +if (likely (vma->vm_start <= address)) goto good_area; -if (!(vma->vm_flags &VM_GROWSDOWN )) - goto bad_area; +if (unlikely (!(vma->vm_flags &VM_GROWSDOWN ))) { +bad_area (regs, error_code, address); +return ; + }if (error_code &PF_USER ) {/* * Accessing the stack below %sp is always a bug.
@@ -705,31 +919,25 @@ again:* and pusha to work. ("enter $65535,$31" pushes * 32 pointers and then decrements %sp by 65535.) */ -if (address +65536 +32 *sizeof (unsigned long ) < regs->sp) - goto bad_area; +if (unlikely (address +65536 +32 *sizeof (unsigned long ) < regs->sp)) { +bad_area (regs, error_code, address); +return ; + } } -if (expand_stack (vma, address)) - goto bad_area; -/* -* Ok, we have a good vm_area for this memory access, so -* we can handle it.. -*/ +if (unlikely (expand_stack (vma, address))) { +bad_area (regs, error_code, address); +return ; + } + +/* +* Ok, we have a good vm_area for this memory access, so +* we can handle it.. +*/ good_area: - si_code =SEGV_ACCERR ; - write =0 ; -switch (error_code & (PF_PROT |PF_WRITE )) { -default :/* 3: write, present */ -/* fall through */ -case PF_WRITE :/* write, not present */ -if (!(vma->vm_flags &VM_WRITE )) - goto bad_area; - write++; -break ; -case PF_PROT :/* read, present */ - goto bad_area; -case 0 :/* read, not present */ -if (!(vma->vm_flags & (VM_READ |VM_EXEC |VM_WRITE ))) - goto bad_area; + write = error_code &PF_WRITE ; +if (unlikely (access_error (error_code, write, vma))) { +bad_area_access_error (regs, error_code, address); +return ; }/*
@@ -739,11 +947,8 @@ good_area:*/ fault =handle_mm_fault (mm, vma, address, write);if (unlikely (fault &VM_FAULT_ERROR )) { -if (fault &VM_FAULT_OOM ) - goto out_of_memory; -else if (fault &VM_FAULT_SIGBUS ) - goto do_sigbus; -BUG (); +mm_fault_error (regs, error_code, address, fault); +return ; }if (fault &VM_FAULT_MAJOR ) tsk->maj_flt++;
@@ -761,139 +966,6 @@ good_area: }#endif up_read (&mm->mmap_sem); -return ; - -/* -* Something tried to access memory that isn't in our memory map.. -* Fix it, but check if it's kernel or user first.. -*/ -bad_area: -up_read (&mm->mmap_sem); - -bad_area_nosemaphore: -/* User mode accesses just cause a SIGSEGV */ -if (error_code &PF_USER ) { -/* -* It's possible to have interrupts off here. -*/ -local_irq_enable (); - -/* -* Valid to do another page fault here because this one came -* from user space. -*/ -if (is_prefetch (regs, address, error_code)) -return ; - -if (is_errata100 (regs, address)) -return ; - -if (show_unhandled_signals &&unhandled_signal (tsk,SIGSEGV ) && -printk_ratelimit ()) { -printk ( -"%s%s[%d]: segfault at %lx ip %p sp %p error %lx" , -task_pid_nr (tsk) >1 ?KERN_INFO :KERN_EMERG , - tsk->comm,task_pid_nr (tsk), address, - (void *) regs->ip, (void *) regs->sp, error_code); -print_vma_addr (" in " , regs->ip); -printk ("\n" ); - } - - tsk->thread.cr2 = address; -/* Kernel addresses are always protection faults */ - tsk->thread.error_code = error_code | (address >=TASK_SIZE ); - tsk->thread.trap_no =14 ; -force_sig_info_fault (SIGSEGV , si_code, address, tsk); -return ; - } - -if (is_f00f_bug (regs, address)) -return ; - -no_context: -/* Are we prepared to handle this kernel fault? */ -if (fixup_exception (regs)) -return ; - -/* -* X86_32 -* Valid to do another page fault here, because if this fault -* had been triggered by is_prefetch fixup_exception would have -* handled it. -* -* X86_64 -* Hall of shame of CPU/BIOS bugs. -*/ -if (is_prefetch (regs, address, error_code)) -return ; - -if (is_errata93 (regs, address)) -return ; - -/* -* Oops. The kernel tried to access some bad page. We'll have to -* terminate things with extreme prejudice. -*/ -#ifdef CONFIG_X86_32 -bust_spinlocks (1 ); -#else - flags =oops_begin (); -#endif - -show_fault_oops (regs, error_code, address); - - tsk->thread.cr2 = address; - tsk->thread.trap_no =14 ; - tsk->thread.error_code = error_code; - -#ifdef CONFIG_X86_32 -die ("Oops" , regs, error_code); -bust_spinlocks (0 ); -do_exit (SIGKILL ); -#else - sig =SIGKILL ; -if (__die ("Oops" , regs, error_code)) - sig =0 ; -/* Executive summary in case the body of the oops scrolled away */ -printk (KERN_EMERG "CR2: %016lx\n" , address); -oops_end (flags, regs, sig); -#endif - -/* -* We ran out of memory, or some other thing happened to us that made -* us unable to handle the page fault gracefully. -*/ -out_of_memory: -up_read (&mm->mmap_sem); -if (is_global_init (tsk)) { -yield (); -/* -* Re-lookup the vma - in theory the vma tree might -* have changed: -*/ - goto again; - } - -printk ("VM: killing process %s\n" , tsk->comm); -if (error_code &PF_USER ) -do_group_exit (SIGKILL ); - goto no_context; - -do_sigbus: -up_read (&mm->mmap_sem); - -/* Kernel mode? Handle exceptions or die */ -if (!(error_code &PF_USER )) - goto no_context; -#ifdef CONFIG_X86_32 -/* User space => ok to do another page fault */ -if (is_prefetch (regs, address, error_code)) -return ; -#endif - tsk->thread.cr2 = address; - tsk->thread.error_code = error_code; - tsk->thread.trap_no =14 ; -force_sig_info_fault (SIGBUS ,BUS_ADRERR , address, tsk); }DEFINE_SPINLOCK (pgd_lock);
Re: [git pull] cpus4096 tree, part 3
Really cc linux-arch this time
Quoted 35 lines · Nick Piggin
Quoted 9 lines · Ingo Molnar
Quoted 6 lines · Linus Torvalds
What happened to Nick's cleanup patch to
do_page_fault(a month or two ago? I complained about some of the issues in his first version and asked for some further cleanups, but I think that whole discussion ended with him saying "I am going to add those changes that you suggested (in fact, I already have)".Quoted 6 lines
Quoted 2 lines
And then I didn't see anything further. Maybe I just missed the end result. Or maybe we have it in some -mm branch or something?
they would have been in tip/x86/mm and would be upstream now had Nick re-sent a v2 series but that never happened. I think they might have fallen victim to a serious attention deficit caused by the SLQB patch ;-)
Well, I already added Linus's suggestions but didn't submit it because there was a bit of work going on in that file as far as I could see, both in the x86 tree and in -mm:
It isn't a big deal to resolve either way, but I don't want to make Andrew's life harder.
[Yes OK now I'm the guilty one of pushing in an x86 patch not via the x86 tree ;) This one is easy to break in pieces, but I didn't want to create a dependency between the trees]
I didn't really consider it to be urgent, so I was waiting for that patch to go in, but I was still hoping to get this into 2.6.29... This is what it looks like now with your suggestions, and just merged it to your current tree (untested).
Quoted 4 lines
I'll cc the linux-arch list here too, because it might be nice to keep these things as structurally similar as possible (and they'll all want to look at the -mm patch above, although I'll probably end up having to write the patches!).
---
Optimise x86's do_page_fault (C entry point for the page fault path).
gcc isn't _all_ that smart about spilling registers to stack or reusing stack slots, even with branch annotations. do_page_fault contained a lot of functionality, so split unlikely paths into their own functions, and mark them as noinline just to be sure. I consider this actually to be somewhat of a cleanup too: the main function now contains about half the number of lines.
Also, ensure the order of arguments to functions is always the same: regs, addr, error_code. This can reduce code size a tiny bit, and just looks neater too.
Add a couple of branch annotations.
One real behavioural difference this makes is that the OOM-init-task case will no longer loop around the page fault handler, but will return to userspace and presumably retry the fault. Effectively the same macro-behaviour, but it is a notable difference. Such change in behaviour should disappear after the "call oom killer from page fault" patch.
Before:
do_page_fault:
subq $360, %rsp #,
After:
do_page_fault:
subq $56, %rsp #,
bloat-o-meter:
add/remove: 8/0 grow/shrink: 0/1 up/down: 2222/-1680 (542)
function old new delta __bad_area_nosemaphore - 506 +506 no_context - 474 +474 vmalloc_fault - 424 +424 spurious_fault - 358 +358 mm_fault_error - 272 +272 bad_area_access_error - 89 +89 bad_area - 89 +89 bad_area_nosemaphore - 10 +10
do_page_fault 2464 784 -1680
Yes, the total size increases by 542 bytes, due to the extra function calls. But these will very rarely be called (except for vmalloc_fault) in a normal workload. Importantly, do_page_fault is less than 1/3rd it's original size. Existing gotos and branch hints did move a lot of the infrequently used text out of the fastpath, but that's even further improved after this patch.
--- arch/x86/mm/fault.c | 458 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 265 insertions(+), 193 deletions(-)
arch/x86/mm/fault.c +265 -193
@@ -91,8 +91,8 @@ static inline int notify_page_fault(stru* * Opcode checker based on code by Richard Brunner */ -static int is_prefetch (struct pt_regs *regs,unsigned long addr, -unsigned long error_code) +static int is_prefetch (struct pt_regs *regs,unsigned long error_code, +unsigned long addr) {unsigned char *instr;int scan_more =1 ;
@@ -409,15 +409,15 @@ static void show_fault_oops(struct pt_re }#ifdef CONFIG_X86_64 -static noinline void pgtable_bad(unsigned long address, struct pt_regs *regs, -unsigned long error_code) +static noinline void pgtable_bad(struct pt_regs *regs, +unsigned long error_code,unsigned long address) {unsigned long flags =oops_begin ();int sig =SIGKILL ; -struct task_struct *tsk; +struct task_struct *tsk = current;printk (KERN_ALERT "%s: Corrupted page table at address %lx\n" , - current->comm, address); + tsk->comm, address);dump_pagetable (address); tsk = current; tsk->thread.cr2 = address;
@@ -429,6 +429,200 @@ static noinline void pgtable_bad(unsigne }#endif +static noinline void no_context(struct pt_regs *regs, +unsigned long error_code,unsigned long address) +{ +struct task_struct *tsk = current; +#ifdef CONFIG_X86_64 +unsigned long flags; +int sig; +#endif + +/* Are we prepared to handle this kernel fault? */ +if (fixup_exception (regs)) +return ; + +/* +* X86_32 +* Valid to do another page fault here, because if this fault +* had been triggered by is_prefetch fixup_exception would have +* handled it. +* +* X86_64 +* Hall of shame of CPU/BIOS bugs. +*/ +if (is_prefetch (regs, error_code, address)) +return ; + +if (is_errata93 (regs, address)) +return ; + +/* +* Oops. The kernel tried to access some bad page. We'll have to +* terminate things with extreme prejudice. +*/ +#ifdef CONFIG_X86_32 +bust_spinlocks (1 ); +#else + flags =oops_begin (); +#endif + +show_fault_oops (regs, error_code, address); + + tsk->thread.cr2 = address; + tsk->thread.trap_no =14 ; + tsk->thread.error_code = error_code; + +#ifdef CONFIG_X86_32 +die ("Oops" , regs, error_code); +bust_spinlocks (0 ); +do_exit (SIGKILL ); +#else + sig =SIGKILL ; +if (__die ("Oops" , regs, error_code)) + sig =0 ; +/* Executive summary in case the body of the oops scrolled away */ +printk (KERN_EMERG "CR2: %016lx\n" , address); +oops_end (flags, regs, sig); +#endif +} + +static void __bad_area_nosemaphore (struct pt_regs *regs, +unsigned long error_code,unsigned long address, +int si_code) +{ +struct task_struct *tsk = current; + +/* User mode accesses just cause a SIGSEGV */ +if (error_code &PF_USER ) { +/* +* It's possible to have interrupts off here. +*/ +local_irq_enable (); + +/* +* Valid to do another page fault here because this one came +* from user space. +*/ +if (is_prefetch (regs, error_code, address)) +return ; + +if (is_errata100 (regs, address)) +return ; + +if (show_unhandled_signals &&unhandled_signal (tsk,SIGSEGV ) && +printk_ratelimit ()) { +printk ( +"%s%s[%d]: segfault at %lx ip %p sp %p error %lx" , +task_pid_nr (tsk) >1 ?KERN_INFO :KERN_EMERG , + tsk->comm,task_pid_nr (tsk), address, + (void *) regs->ip, (void *) regs->sp, error_code); +print_vma_addr (" in " , regs->ip); +printk ("\n" ); + } + + tsk->thread.cr2 = address; +/* Kernel addresses are always protection faults */ + tsk->thread.error_code = error_code | (address >=TASK_SIZE ); + tsk->thread.trap_no =14 ; +force_sig_info_fault (SIGSEGV , si_code, address, tsk); +return ; + } + +if (is_f00f_bug (regs, address)) +return ; + +no_context (regs, error_code, address); +} + +static noinline void bad_area_nosemaphore(struct pt_regs *regs, +unsigned long error_code,unsigned long address) +{ +__bad_area_nosemaphore (regs, error_code, address,SEGV_MAPERR ); +} + +static void __bad_area (struct pt_regs *regs, +unsigned long error_code,unsigned long address, +int si_code) +{ +struct mm_struct *mm = current->mm; + +/* +* Something tried to access memory that isn't in our memory map.. +* Fix it, but check if it's kernel or user first.. +*/ +up_read (&mm->mmap_sem); + +__bad_area_nosemaphore (regs, error_code, address, si_code); +} + +static noinline void bad_area(struct pt_regs *regs, +unsigned long error_code,unsigned long address) +{ +__bad_area (regs, error_code, address,SEGV_MAPERR ); +} + +static noinline void bad_area_access_error(struct pt_regs *regs, +unsigned long error_code,unsigned long address) +{ +__bad_area (regs, error_code, address,SEGV_ACCERR ); +} + +/* TODO: fixup for "mm-invoke-oom-killer-from-page-fault.patch" */ +static void out_of_memory (struct pt_regs *regs, +unsigned long error_code,unsigned long address) +{ +struct task_struct *tsk = current; +struct mm_struct *mm = tsk->mm; +/* +* We ran out of memory, or some other thing happened to us that made +* us unable to handle the page fault gracefully. +*/ +up_read (&mm->mmap_sem); +if (is_global_init (tsk)) { +yield (); +return ; + } + +printk ("VM: killing process %s\n" , tsk->comm); +if (error_code &PF_USER ) +do_group_exit (SIGKILL ); +no_context (regs, error_code, address); +} + +static void do_sigbus (struct pt_regs *regs, +unsigned long error_code,unsigned long address) +{ +struct task_struct *tsk = current; +struct mm_struct *mm = tsk->mm; + +up_read (&mm->mmap_sem); + +/* Kernel mode? Handle exceptions or die */ +if (!(error_code &PF_USER )) +no_context (regs, error_code, address); +#ifdef CONFIG_X86_32 +/* User space => ok to do another page fault */ +if (is_prefetch (regs, error_code, address)) +return ; +#endif + tsk->thread.cr2 = address; + tsk->thread.error_code = error_code; + tsk->thread.trap_no =14 ; +force_sig_info_fault (SIGBUS ,BUS_ADRERR , address, tsk); +} + +static noinline void mm_fault_error(struct pt_regs *regs, +unsigned long error_code,unsigned long address,unsigned int fault) +{ +if (fault &VM_FAULT_OOM ) +out_of_memory (regs, error_code, address); +else if (fault &VM_FAULT_SIGBUS ) +do_sigbus (regs, error_code, address); +else +BUG (); +} +static int spurious_fault_check (unsigned long error_code,pte_t *pte) {if ((error_code &PF_WRITE ) && !pte_write (*pte))
@@ -448,8 +642,8 @@ static int spurious_fault_check(unsigned* There are no security implications to leaving a stale TLB when * increasing the permissions on a page. */ -static int spurious_fault (unsigned long address, -unsigned long error_code) +static noinline int spurious_fault(unsigned long error_code, +unsigned long address) {pgd_t *pgd;pud_t *pud;
@@ -494,7 +688,7 @@ static int spurious_fault(unsigned long* * This assumes no large pages in there. */ -static int vmalloc_fault (unsigned long address) +static noinline int vmalloc_fault(unsigned long address) {#ifdef CONFIG_X86_32 unsigned long pgd_paddr;
@@ -573,6 +767,25 @@ static int vmalloc_fault(unsigned long aint show_unhandled_signals =1 ; +static inline int access_error (unsigned long error_code,int write, +struct vm_area_struct *vma) +{ +if (write) { +/* write, present and write, not present */ +if (unlikely (!(vma->vm_flags &VM_WRITE ))) +return 1 ; + }else if (unlikely (error_code &PF_PROT )) { +/* read, present */ +return 1 ; + }else { +/* read, not present */ +if (unlikely (!(vma->vm_flags & (VM_READ |VM_EXEC |VM_WRITE )))) +return 1 ; + } + +return 0 ; +} +/* * This routine handles page faults. It determines the address, * and the problem, and then passes it off to one of the appropriate
@@ -583,16 +796,12 @@ asmlinkage#endif void __kprobes do_page_fault (struct pt_regs *regs,unsigned long error_code) { +unsigned long address;struct task_struct *tsk;struct mm_struct *mm;struct vm_area_struct *vma; -unsigned long address; -int write, si_code; +int write;int fault; -#ifdef CONFIG_X86_64 -unsigned long flags; -int sig; -#endif tsk = current; mm = tsk->mm;
@@ -601,9 +810,7 @@ void __kprobes do_page_fault(struct pt_r/* get the address */ address =read_cr2 (); - si_code =SEGV_MAPERR ; - -if (notify_page_fault (regs)) +if (unlikely (notify_page_fault (regs)))return ;if (unlikely (kmmio_fault (regs, address)))return ;
@@ -638,10 +845,10 @@ void __kprobes do_page_fault(struct pt_r* Don't take the mm semaphore here. If we fixup a prefetch * fault we could otherwise deadlock. */ - goto bad_area_nosemaphore; +bad_area_nosemaphore (regs, error_code, address); +return ; } -/* * It's safe to allow irq's after cr2 has been saved and the * vmalloc fault has been handled.
@@ -657,17 +864,18 @@ void __kprobes do_page_fault(struct pt_r#ifdef CONFIG_X86_64 if (unlikely (error_code &PF_RSVD )) -pgtable_bad (address, regs, error_code); +pgtable_bad (regs, error_code, address);#endif /* * If we're in an interrupt, have no user context or are running in an * atomic region then we must not take the fault. */ -if (unlikely (in_atomic () || !mm)) - goto bad_area_nosemaphore; +if (unlikely (in_atomic () || !mm)) { +bad_area_nosemaphore (regs, error_code, address); +return ; + } -again:/* * When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the
@@ -684,20 +892,26 @@ again:* source. If this is invalid we can skip the address space check, * thus avoiding the deadlock. */ -if (!down_read_trylock (&mm->mmap_sem)) { +if (unlikely (!down_read_trylock (&mm->mmap_sem))) {if ((error_code &PF_USER ) ==0 && - !search_exception_tables (regs->ip)) - goto bad_area_nosemaphore; + !search_exception_tables (regs->ip)) { +bad_area_nosemaphore (regs, error_code, address); +return ; + }down_read (&mm->mmap_sem); } vma =find_vma (mm, address); -if (!vma) - goto bad_area; -if (vma->vm_start <= address) +if (unlikely (!vma)) { +bad_area (regs, error_code, address); +return ; + } +if (likely (vma->vm_start <= address)) goto good_area; -if (!(vma->vm_flags &VM_GROWSDOWN )) - goto bad_area; +if (unlikely (!(vma->vm_flags &VM_GROWSDOWN ))) { +bad_area (regs, error_code, address); +return ; + }if (error_code &PF_USER ) {/* * Accessing the stack below %sp is always a bug.
@@ -705,31 +919,25 @@ again:* and pusha to work. ("enter $65535,$31" pushes * 32 pointers and then decrements %sp by 65535.) */ -if (address +65536 +32 *sizeof (unsigned long ) < regs->sp) - goto bad_area; +if (unlikely (address +65536 +32 *sizeof (unsigned long ) < regs->sp)) { +bad_area (regs, error_code, address); +return ; + } } -if (expand_stack (vma, address)) - goto bad_area; -/* -* Ok, we have a good vm_area for this memory access, so -* we can handle it.. -*/ +if (unlikely (expand_stack (vma, address))) { +bad_area (regs, error_code, address); +return ; + } + +/* +* Ok, we have a good vm_area for this memory access, so +* we can handle it.. +*/ good_area: - si_code =SEGV_ACCERR ; - write =0 ; -switch (error_code & (PF_PROT |PF_WRITE )) { -default :/* 3: write, present */ -/* fall through */ -case PF_WRITE :/* write, not present */ -if (!(vma->vm_flags &VM_WRITE )) - goto bad_area; - write++; -break ; -case PF_PROT :/* read, present */ - goto bad_area; -case 0 :/* read, not present */ -if (!(vma->vm_flags & (VM_READ |VM_EXEC |VM_WRITE ))) - goto bad_area; + write = error_code &PF_WRITE ; +if (unlikely (access_error (error_code, write, vma))) { +bad_area_access_error (regs, error_code, address); +return ; }/*
@@ -739,11 +947,8 @@ good_area:*/ fault =handle_mm_fault (mm, vma, address, write);if (unlikely (fault &VM_FAULT_ERROR )) { -if (fault &VM_FAULT_OOM ) - goto out_of_memory; -else if (fault &VM_FAULT_SIGBUS ) - goto do_sigbus; -BUG (); +mm_fault_error (regs, error_code, address, fault); +return ; }if (fault &VM_FAULT_MAJOR ) tsk->maj_flt++;
@@ -761,139 +966,6 @@ good_area: }#endif up_read (&mm->mmap_sem); -return ; - -/* -* Something tried to access memory that isn't in our memory map.. -* Fix it, but check if it's kernel or user first.. -*/ -bad_area: -up_read (&mm->mmap_sem); - -bad_area_nosemaphore: -/* User mode accesses just cause a SIGSEGV */ -if (error_code &PF_USER ) { -/* -* It's possible to have interrupts off here. -*/ -local_irq_enable (); - -/* -* Valid to do another page fault here because this one came -* from user space. -*/ -if (is_prefetch (regs, address, error_code)) -return ; - -if (is_errata100 (regs, address)) -return ; - -if (show_unhandled_signals &&unhandled_signal (tsk,SIGSEGV ) && -printk_ratelimit ()) { -printk ( -"%s%s[%d]: segfault at %lx ip %p sp %p error %lx" , -task_pid_nr (tsk) >1 ?KERN_INFO :KERN_EMERG , - tsk->comm,task_pid_nr (tsk), address, - (void *) regs->ip, (void *) regs->sp, error_code); -print_vma_addr (" in " , regs->ip); -printk ("\n" ); - } - - tsk->thread.cr2 = address; -/* Kernel addresses are always protection faults */ - tsk->thread.error_code = error_code | (address >=TASK_SIZE ); - tsk->thread.trap_no =14 ; -force_sig_info_fault (SIGSEGV , si_code, address, tsk); -return ; - } - -if (is_f00f_bug (regs, address)) -return ; - -no_context: -/* Are we prepared to handle this kernel fault? */ -if (fixup_exception (regs)) -return ; - -/* -* X86_32 -* Valid to do another page fault here, because if this fault -* had been triggered by is_prefetch fixup_exception would have -* handled it. -* -* X86_64 -* Hall of shame of CPU/BIOS bugs. -*/ -if (is_prefetch (regs, address, error_code)) -return ; - -if (is_errata93 (regs, address)) -return ; - -/* -* Oops. The kernel tried to access some bad page. We'll have to -* terminate things with extreme prejudice. -*/ -#ifdef CONFIG_X86_32 -bust_spinlocks (1 ); -#else - flags =oops_begin (); -#endif - -show_fault_oops (regs, error_code, address); - - tsk->thread.cr2 = address; - tsk->thread.trap_no =14 ; - tsk->thread.error_code = error_code; - -#ifdef CONFIG_X86_32 -die ("Oops" , regs, error_code); -bust_spinlocks (0 ); -do_exit (SIGKILL ); -#else - sig =SIGKILL ; -if (__die ("Oops" , regs, error_code)) - sig =0 ; -/* Executive summary in case the body of the oops scrolled away */ -printk (KERN_EMERG "CR2: %016lx\n" , address); -oops_end (flags, regs, sig); -#endif - -/* -* We ran out of memory, or some other thing happened to us that made -* us unable to handle the page fault gracefully. -*/ -out_of_memory: -up_read (&mm->mmap_sem); -if (is_global_init (tsk)) { -yield (); -/* -* Re-lookup the vma - in theory the vma tree might -* have changed: -*/ - goto again; - } - -printk ("VM: killing process %s\n" , tsk->comm); -if (error_code &PF_USER ) -do_group_exit (SIGKILL ); - goto no_context; - -do_sigbus: -up_read (&mm->mmap_sem); - -/* Kernel mode? Handle exceptions or die */ -if (!(error_code &PF_USER )) - goto no_context; -#ifdef CONFIG_X86_32 -/* User space => ok to do another page fault */ -if (is_prefetch (regs, address, error_code)) -return ; -#endif - tsk->thread.cr2 = address; - tsk->thread.error_code = error_code; - tsk->thread.trap_no =14 ; -force_sig_info_fault (SIGBUS ,BUS_ADRERR , address, tsk); }DEFINE_SPINLOCK (pgd_lock);
Andrew Morton
maintainerRe: [git pull] cpus4096 tree, part 3
Quoted 38 lines · On Mon, 5 Jan 2009 02:16:30 +0100 Nick Piggin <[email protected]> wrote:
Really cc linux-arch this time
Quoted 35 lines · Nick Piggin
Quoted 9 lines · Ingo Molnar
Quoted 6 lines · Linus Torvalds
What happened to Nick's cleanup patch to
do_page_fault(a month or two ago? I complained about some of the issues in his first version and asked for some further cleanups, but I think that whole discussion ended with him saying "I am going to add those changes that you suggested (in fact, I already have)".Quoted 6 lines
Quoted 2 lines
And then I didn't see anything further. Maybe I just missed the end result. Or maybe we have it in some -mm branch or something?
they would have been in tip/x86/mm and would be upstream now had Nick re-sent a v2 series but that never happened. I think they might have fallen victim to a serious attention deficit caused by the SLQB patch ;-)
Well, I already added Linus's suggestions but didn't submit it because there was a bit of work going on in that file as far as I could see, both in the x86 tree and in -mm:
It isn't a big deal to resolve either way, but I don't want to make Andrew's life harder.
[Yes OK now I'm the guilty one of pushing in an x86 patch not via the x86 tree ;) This one is easy to break in pieces, but I didn't want to create a dependency between the trees]
I didn't really consider it to be urgent, so I was waiting for that patch to go in, but I was still hoping to get this into 2.6.29... This is what it looks like now with your suggestions, and just merged it to your current tree (untested).
Quoted 7 lines
Quoted 4 lines
I'll cc the linux-arch list here too, because it might be nice to keep these things as structurally similar as possible (and they'll all want to look at the -mm patch above, although I'll probably end up having to write the patches!).
---
Optimise x86'sdo_page_fault(C entry point for the page fault path).
It took rather a lot of hunting to find this email. Please do try to make the email subject match the final patch's title?
Quoted 136 lines
* This routine handles page faults. It determines the address,
* and the problem, and then passes it off to one of the appropriate(quoted diff) +37
-37@@ -583,16 +796,12 @@ asmlinkage #endif void __kprobes do_page_fault(struct pt_regs *regs, unsigned long error_code) { + unsigned long address; struct task_struct *tsk; struct mm_struct *mm; struct vm_area_struct *vma; - unsigned long address; - int write, si_code; + int write; int fault; -#ifdef CONFIG_X86_64 - unsigned long flags; - int sig; -#endif tsk = current; mm = tsk->mm;@@ -601,9 +810,7 @@ void __kprobes do_page_fault(struct pt_r /* get the address */ address = read_cr2(); - si_code = SEGV_MAPERR; - - if (notify_page_fault(regs)) + if (unlikely(notify_page_fault(regs))) return; if (unlikely(kmmio_fault(regs, address))) return;@@ -638,10 +845,10 @@ void __kprobes do_page_fault(struct pt_r * Don't take the mm semaphore here. If we fixup a prefetch * fault we could otherwise deadlock. */ - goto bad_area_nosemaphore; + bad_area_nosemaphore(regs, error_code, address); + return; } - /* * It's safe to allow irq's after cr2 has been saved and the * vmalloc fault has been handled.@@ -657,17 +864,18 @@ void __kprobes do_page_fault(struct pt_r #ifdef CONFIG_X86_64 if (unlikely(error_code & PF_RSVD)) - pgtable_bad(address, regs, error_code); + pgtable_bad(regs, error_code, address); #endif /* * If we're in an interrupt, have no user context or are running in an * atomic region then we must not take the fault. */ - if (unlikely(in_atomic() || !mm)) - goto bad_area_nosemaphore; + if (unlikely(in_atomic() || !mm)) { + bad_area_nosemaphore(regs, error_code, address); + return; + } -again: /* * When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the@@ -684,20 +892,26 @@ again: * source. If this is invalid we can skip the address space check, * thus avoiding the deadlock. */ - if (!down_read_trylock(&mm->mmap_sem)) { + if (unlikely(!down_read_trylock(&mm->mmap_sem))) { if ((error_code & PF_USER) == 0 && - !search_exception_tables(regs->ip)) - goto bad_area_nosemaphore; + !search_exception_tables(regs->ip)) { + bad_area_nosemaphore(regs, error_code, address); + return; + } down_read(&mm->mmap_sem); } vma = find_vma(mm, address); - if (!vma) - goto bad_area; - if (vma->vm_start <= address) + if (unlikely(!vma)) { + bad_area(regs, error_code, address); + return; + } + if (likely(vma->vm_start <= address)) goto good_area; - if (!(vma->vm_flags & VM_GROWSDOWN)) - goto bad_area; + if (unlikely(!(vma->vm_flags & VM_GROWSDOWN))) { + bad_area(regs, error_code, address); + return; + } if (error_code & PF_USER) { /* * Accessing the stack below %sp is always a bug.@@ -705,31 +919,25 @@ again: * and pusha to work. ("enter $65535,$31" pushes * 32 pointers and then decrements %sp by 65535.) */ - if (address + 65536 + 32 * sizeof(unsigned long) < regs->sp) - goto bad_area; + if (unlikely(address + 65536 + 32 * sizeof(unsigned long) < regs->sp)) { + bad_area(regs, error_code, address); + return; + } } - if (expand_stack(vma, address)) - goto bad_area; -/* - * Ok, we have a good vm_area for this memory access, so - * we can handle it.. - */ + if (unlikely(expand_stack(vma, address))) { + bad_area(regs, error_code, address); + return; + } + + /* + * Ok, we have a good vm_area for this memory access, so + * we can handle it.. + */ good_area: - si_code = SEGV_ACCERR; - write = 0; - switch (error_code & (PF_PROT|PF_WRITE)) { - default: /* 3: write, present */ - /* fall through */ - case PF_WRITE: /* write, not present */
Review again: · line 937
Quoted 10 lines
(quoted diff) +1
-9- if (!(vma->vm_flags & VM_WRITE)) - goto bad_area; - write++; - break; - case PF_PROT: /* read, present */ - goto bad_area; - case 0: /* read, not present */ - if (!(vma->vm_flags & (VM_READ | VM_EXEC | VM_WRITE))) - goto bad_area; + write = error_code & PF_WRITE;
What's going on here? We set `error_code' to PF_WRITE, which is some x86-specific thing.
Review good_area: · line 948
Quoted 9 lines
(quoted diff) +3
-0+ if (unlikely(access_error(error_code, write, vma))) { + bad_area_access_error(regs, error_code, address); + return; } /*@@ -739,11 +947,8 @@ good_area: */ fault = handle_mm_fault(mm, vma, address, write);
and then pass it into handle_mm_fault(), which is expecting a bunch of flags in the FAULT_FLAG_foo domain.
IOW, the code will accidentally set FAULT_FLAG_NONLINEAR!.
Methinks we want something like this,
arch/x86/mm/fault.c renamed from arch/x86/mm/fault.c~fix-x86-optimise-x86s-do_page_fault-c-entry-point-for-the-page-fault-path +1 -1
@@ -942,7 +942,7 @@ void __kprobes do_page_fault(struct pt_r* we can handle it.. */ good_area: - write = error_code &PF_WRITE ; + write = (error_code &PF_WRITE ) ?FAULT_FLAG_WRITE :0 ;if (unlikely (access_error (error_code, write, vma))) {bad_area_access_error (regs, error_code, address);return ;
_
but why did the current code pass testing at all??
Re: [git pull] cpus4096 tree, part 3
Quoted 4 lines · Andrew Morton
Review
Quoted 1 line
(quoted diff) +1
-0+ write = error_code & PF_WRITE;What's going on here? We set `error_code' to
PF_WRITE, which is some x86-specific thing.
No. We set "write" to non-zero if it was a write fault.
Quoted 4 lines
Quoted 1 line
fault =
handle_mm_fault(mm, vma, address, write);and then pass it into
handle_mm_fault(), which is expecting a bunch of flags in theFAULT_FLAG_foodomain.
No. "handle_mm_fault()" takes an integer that is non-zero if it's a write, zero if it's a read. That's how it has _always_ worked.
I don't see where you find that FAULT_FLAG_foo thing. That's much deeper down, when people do things like
unsigned int flags = FAULT_FLAG_NONLINEAR |
(write_access ? FAULT_FLAG_WRITE : 0);
based on that whole "write_access" flag.
Linus
Andrew Morton
maintainerRe: [git pull] cpus4096 tree, part 3
Quoted 18 lines · On Mon, 26 Jan 2009 11:09:59 -0800 (PST) Linus Torvalds…
Quoted 4 lines · Andrew Morton
Review
Quoted 1 line
(quoted diff) +1
-0+ write = error_code & PF_WRITE;What's going on here? We set `error_code' to
PF_WRITE, which is some x86-specific thing.No. We set "write" to non-zero if it was a write fault.
Quoted 4 lines
Quoted 1 line
fault =
handle_mm_fault(mm, vma, address, write);and then pass it into
handle_mm_fault(), which is expecting a bunch of flags in theFAULT_FLAG_foodomain.No. "
handle_mm_fault()" takes an integer that is non-zero if it's a write, zero if it's a read. That's how it has _always_ worked.
Quoted 8 lines
I don't see where you find that
FAULT_FLAG_foothing. That's much deeper down, when people do things likeunsigned int flags =
FAULT_FLAG_NONLINEAR|
(write_access?FAULT_FLAG_WRITE: 0);based on that whole "
write_access" flag.
OK, thanks. It's actually page_fault-retry-with-nopage_retry.patch which got those things confused, and then confused me. I'll go address that in the other thread..
Re: [git pull] cpus4096 tree, part 3
In reply to Andrew MortonQuoted 1 line · Andrew Morton
but why did the current code pass testing at all??
i queued it up a week ago and beyond a same-day breakage i reported to Nick (and which he fixed) this commit was problem-free and passed all testing here.
Does it cause problems for you? If yes then please describe the kind of problems.
Note: i see that -mm modifies a few other details of the x86 pagefault handling path (there a pagefault-retry patch in there) - so there might be contextual interactions there. But this particular cleanup/improvement from Nick is working fine on a wide range of systems here.
Btw., regarding pagefault retry. The bits that are in -mm currently i find a bit ugly:
Quoted 19 lines
+++
a/arch/x86/mm/fault.c(quoted diff) +2
-1@@ -799,7 +799,7 @@ void __kprobes do_page_fault(struct pt_r struct vm_area_struct *vma; int write; int fault; - unsigned int retry_flag = FAULT_FLAG_RETRY; + int retry_flag = 1; tsk = current; mm = tsk->mm;@@ -951,6 +951,7 @@ good_area: } write |= retry_flag; + /* * If for any reason at all we couldn't handle the fault, * make sure we exit gracefully rather than endlessly redo
Review good_area: · line 977
Quoted 10 lines
(quoted diff) +2
-2@@ -969,8 +970,8 @@ good_area: * be removed or changed after the retry. */ if (fault & VM_FAULT_RETRY) { - if (write & FAULT_FLAG_RETRY) { - retry_flag &= ~FAULT_FLAG_RETRY; + if (retry_flag) { + retry_flag = 0; goto retry; } BUG();
as this complicates every architecture with a 'can the fault be retried' logic and open-coded retry loop.
But that logic is rather repetitive and once an architecture filters out all its special in-kernel sources of faults and the hw quirks it has, the handling of pte faults is rather generic and largely offloaded into handle_pte_fault() already.
So when this patch was submitted a few weeks ago i suggested that retry should be done purely in mm/memory.c instead, and the low level code should at most be refactored to suit this method, but not complicated any further.
Any deep reasons for why such a more generic approach is not desirable?
Ingo
Andrew Morton
maintainerRe: [git pull] cpus4096 tree, part 3
Quoted 43 lines · On Mon, 26 Jan 2009 21:09:57 +0100 Ingo Molnar <[email protected]> wrote:
...
Btw., regarding pagefault retry. The bits that are in -mm currently i find a bit ugly:
Quoted 19 lines
+++
a/arch/x86/mm/fault.c(quoted diff) +2
-1@@ -799,7 +799,7 @@ void __kprobes do_page_fault(struct pt_r struct vm_area_struct *vma; int write; int fault; - unsigned int retry_flag = FAULT_FLAG_RETRY; + int retry_flag = 1; tsk = current; mm = tsk->mm;@@ -951,6 +951,7 @@ good_area: } write |= retry_flag; + /* * If for any reason at all we couldn't handle the fault, * make sure we exit gracefully rather than endlessly redoReview good_area: · line 977
Quoted 10 lines
(quoted diff) +2
-2@@ -969,8 +970,8 @@ good_area: * be removed or changed after the retry. */ if (fault & VM_FAULT_RETRY) { - if (write & FAULT_FLAG_RETRY) { - retry_flag &= ~FAULT_FLAG_RETRY; + if (retry_flag) { + retry_flag = 0; goto retry; } BUG();as this complicates every architecture with a 'can the fault be retried' logic and open-coded retry loop.
But that logic is rather repetitive and once an architecture filters out all its special in-kernel sources of faults and the hw quirks it has, the handling of pte faults is rather generic and largely offloaded into
handle_pte_fault()already.
Quoted 7 lines
So when this patch was submitted a few weeks ago i suggested that retry should be done purely in
mm/memory.cinstead, and the low level code should at most be refactored to suit this method, but not complicated any further.Any deep reasons for why such a more generic approach is not desirable?
Let's cc the people who wrote it.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [git pull] cpus4096 tree, part 3
Quoted 2 lines · Ying Han
Thank you Ingo and Andrew for the comments. I will take a look into it ASAP and updates it here.
Note, my objection wasnt a hard NAK - just an observation. If all things considered Andrew still favors the VM_FAULT_RETRY approach then that's fine too i guess.
It's just that a quick look gave me the feeling of a retry flag tacked on to an existing codepath [and all the micro-overhead and complexity that this brings], instead of a clean refactoring of pagefault handling functionality into a higher MM level retry loop.
So the alternative has to be looked at and rejected because it's technically inferior - not because it's more difficult to implement. (which it certainly is)
Ingo
Andrew Morton
maintainerRe: [git pull] cpus4096 tree, part 3
Quoted 10 lines · On Tue, 27 Jan 2009 00:21:39 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 2 lines · Ying Han
Thank you Ingo and Andrew for the comments. I will take a look into it ASAP and updates it here.
Note, my objection wasnt a hard NAK - just an observation. If all things considered Andrew still favors the
VM_FAULT_RETRYapproach then that's fine too i guess.
Quoted 9 lines
It's just that a quick look gave me the feeling of a retry flag tacked on to an existing codepath [and all the micro-overhead and complexity that this brings], instead of a clean refactoring of pagefault handling functionality into a higher MM level retry loop.
So the alternative has to be looked at and rejected because it's technically inferior - not because it's more difficult to implement. (which it certainly is)
I have wobbly feelings about this patch. There are your issues, and a long string of problems and fixes. And my recent half-assed linux-next-related fix which I didn't really think about.
It all needs a revisit/rereview/reunderstand cycle.
Re: [git pull] cpus4096 tree, part 3
In reply to Nick PigginQuoted 21 lines · Nick Piggin
Quoted 9 lines · Ingo Molnar
Quoted 6 lines · Linus Torvalds
What happened to Nick's cleanup patch to
do_page_fault(a month or two ago? I complained about some of the issues in his first version and asked for some further cleanups, but I think that whole discussion ended with him saying "I am going to add those changes that you suggested (in fact, I already have)".Quoted 6 lines
Quoted 2 lines
And then I didn't see anything further. Maybe I just missed the end result. Or maybe we have it in some -mm branch or something?
they would have been in tip/x86/mm and would be upstream now had Nick re-sent a v2 series but that never happened. I think they might have fallen victim to a serious attention deficit caused by the SLQB patch ;-)
Well, I already added Linus's suggestions but didn't submit it because there was a bit of work going on in that file as far as I could see, both in the x86 tree and in -mm:
Quoted 8 lines
It isn't a big deal to resolve either way, but I don't want to make Andrew's life harder.
[Yes OK now I'm the guilty one of pushing in an x86 patch not via the x86 tree ;) This one is easy to break in pieces, but I didn't want to create a dependency between the trees]
That's OK, and the oom-killer patch impact on x86 was incidental, so it was correct to push it via -mm IMO.
Now that the bits that went in via Andrew's tree upstream, there's a handful of new conflicts in the patch - so would you mind to (re-)send a merged up patch against latest -git?
Ingo
Re: [git pull] cpus4096 tree, part 3
In reply to Linus TorvaldsQuoted 7 lines · Linus Torvalds
Quoted 2 lines · Ingo Molnar
ok. The pending regressions are all fixed now, and i've just finished my standard tests on the latest tree and all the tests passed fine.
Ok, pulled and pushed out.
Quoted 7 lines
Has anybody looked at what the stack size is with MAXSMP set with an allyesconfig? And what areas are still problematic, if any? Are we going to have some code-paths that still essentially have 1kB+ of stack space just because they haven't been converted and still have the cpu mask on stack?
Linus
Hi Linus,
Yes, I do periodically collect stats for memory and stack usage. Here is a recent stack summary (not "allyes", but most all trace/debug options turned on). It shows the stack growth from a 128 NR_CPUS config to a MAXSMP (4096 NR_CPUS) config. Most of the changes to correct these "stack hogs" have been sitting in a queue until the changes affecting non-x86 architectures have been accepted (which you just did), though some are because of new code from the merge activity.
Rusty has introduced a config option that disables the old cpumask_t which really highlights where the offenders still are. Ultimately, that should prevent any new stack hogs from being introduced, but it won't be settable until 2.6.30 time frame.
====== Stack (-l 500)
1 - 128-defconfig
2 - 4k-defconfig
.1. .2. ..final..
0 +1640 1640 . acpi_cpufreq_target
0 +1368 1368 . cpufreq_add_dev
0 +1344 1344 . store_scaling_governor
0 +1328 1328 . store_scaling_min_freq
0 +1328 1328 . store_scaling_max_freq
0 +1328 1328 . cpufreq_update_policy
0 +1328 1328 . cpu_has_cpufreq
0 +1048 1048 . get_cur_val
0 +1032 1032 . local_cpus_show
0 +1032 1032 . local_cpulist_show
0 +1024 1024 . pci_bus_show_cpuaffinity
0 +808 808 . cpuset_write_resmask
0 +736 736 . update_flag
0 +648 648 . init_intel_cacheinfo
0 +640 640 . cpuset_attach
0 +584 584 . shmem_getpage
0 +584 584 . __percpu_alloc_mask
0 +552 552 . smp_call_function_many
0 +536 536 . pci_device_probe
0 +536 536 . native_flush_tlb_others
0 +536 536 . cpuset_common_file_read
0 +520 520 . show_related_cpus
0 +520 520 . show_affected_cpus
0 +520 520 . get_measured_perf
0 +520 520 . flush_tlb_page
0 +520 520 . cpuset_can_attach
0 +512 512 . flush_tlb_mm
0 +512 512 . flush_tlb_current_task
0 +512 512 . find_lowest_rq
0 +512 512 . acpi_processor_ffh_cstate_probe====== Text/Data ()
Overall memory reservation looks like this:
.1. .2. ..final.. 5799936 +4096 5804032 +0.07% TextSize 3772416 +139264 3911680 +3.69% DataSize 8822784 +1234944 10057728 +13% BssSize 2445312 +794624 3239936 +32% InitSize 1884160 +4096 1888256 +0.22% PerCPU 143360 +708608 851968 +494% OtherSize 22867968 +2885632 25753600 +112% Totals
I will update these with the latest changes (and use a allyesconfig config) and post them again soon.
Thanks,
Mike
Quoted 9 lines · On Saturday 03 January 2009 07:08:40 Ingo Molnar wrote:
- architectures that have no
__fls(8 out of 21) fail to build:arch/cris arch/frv arch/h8300 arch/m32r arch/m68k arch/mn10300 arch/xtensa
Fixes pushed, m68k should be OK tho; is this actual compile test? You have to look in include/asm-m68k to see __fls.
Quoted 4 lines
Rusty, would it be fine with you if we did all the remaining bits via tip/cpus4096? It's your tree and your bits and we wanted to send our remaining bits after your tree went to Linus but the conflict resolutions from Mike are valuable so i think we should reconsider the ordering.
Yeah, no reason for us to do the merge twice. As long as it ends upstream, I'm a happy camper.
Thanks,
Rusty.
Quoted 11 lines · Rusty Russell
Quoted 9 lines · On Saturday 03 January 2009 07:08:40 Ingo Molnar wrote:
- architectures that have no
__fls(8 out of 21) fail to build:arch/cris arch/frv arch/h8300 arch/m32r arch/m68k arch/mn10300 arch/xtensa
Quoted 2 lines
Fixes pushed, m68k should be OK tho; is this actual compile test? You have to look in
include/asm-m68kto see__fls.
yeah, i stopped the tests after the first two build failures - the rest is a grep result from arch/*/, that's why include/asm-m68k/ was left out.
Quoted 8 lines
Quoted 5 lines
Rusty, would it be fine with you if we did all the remaining bits via tip/cpus4096? It's your tree and your bits and we wanted to send our remaining bits after your tree went to Linus but the conflict resolutions from Mike are valuable so i think we should reconsider the ordering.
Yeah, no reason for us to do the merge twice. As long as it ends upstream, I'm a happy camper.
great - lets do it that way then. I have pulled your fixes into the cpus4096 tree:
5ece5c5: xtensa: define __fls
5c134da: mn10300: define __fls
16a2062: m32r: define __fls
9ddabc2: h8300: define __fls
ee38e51: frv: define __fls
0999769: cris: define __fls
Once we have figured out the CPU-hotplug lockdep splat (possibly due to Mike's changes not yours) i'll send it to Linus. Thanks,
Ingo
PATCH 1/1 ia64: cpumask fix for is_affinity_mask_valid()
Quoted 31 lines · Ingo Molnar
Quoted 11 lines · Rusty Russell
Quoted 9 lines · On Saturday 03 January 2009 07:08:40 Ingo Molnar wrote:
- architectures that have no
__fls(8 out of 21) fail to build:arch/cris arch/frv arch/h8300 arch/m32r arch/m68k arch/mn10300 arch/xtensaQuoted 2 lines
Fixes pushed, m68k should be OK tho; is this actual compile test? You have to look in
include/asm-m68kto see__fls.yeah, i stopped the tests after the first two build failures - the rest is a grep result from
arch/*/, that's whyinclude/asm-m68k/was left out.Quoted 8 lines
Quoted 5 lines
Rusty, would it be fine with you if we did all the remaining bits via tip/cpus4096? It's your tree and your bits and we wanted to send our remaining bits after your tree went to Linus but the conflict resolutions from Mike are valuable so i think we should reconsider the ordering.
Yeah, no reason for us to do the merge twice. As long as it ends upstream, I'm a happy camper.
great - lets do it that way then. I have pulled your fixes into the cpus4096 tree:
Quoted 6 lines
5ece5c5: xtensa: define
__fls
5c134da: mn10300: define__fls
16a2062: m32r: define__fls
9ddabc2: h8300: define__fls
ee38e51: frv: define__fls
0999769: cris: define__fls
ok, these architectures build fine now.
There's one new build failure due to cpumask changes: ia64. I have fixed it via the patch below. (if it looks good to you i'll carry it via tip/cpus4096, ok?)
Breakage came via this commit i think:
d036e67: cpumask: convert kernel/irq
[ia64 is the only architecture that re-defines default_affinity_write().]
Ingo
----------------------->
Quoted 1 line
From
f1faf35ad450c3d648138b74b9f58b283dc19c5cMon Sep 17 00:00:00 2001
From: Ingo Molnar <[email protected]>
Date: Sat, 3 Jan 2009 12:50:46 +0100
Subject: [PATCH] ia64: cpumask fix for is_affinity_mask_valid()
Impact: build fix on ia64
ia64's default_affinity_write() still had old cpumask_t usage:
/home/mingo/tip/kernel/irq/proc.c: In function `default_affinity_write':
/home/mingo/tip/kernel/irq/proc.c:114: error: incompatible type for argument 1 of `is_affinity_mask_valid' make[3]: *** [kernel/irq/proc.o] Error 1
make[3]: *** Waiting for unfinished jobs....
update it to cpumask_var_t.
--- arch/ia64/include/asm/irq.h | 2 +- arch/ia64/kernel/irq.c | 4 ++-- kernel/irq/proc.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)
arch/ia64/include/asm/irq.h +1 -1
@@ -27,7 +27,7 @@ irq_canonicalize (int irq) }extern void set_irq_affinity_info (unsigned int irq,int dest,int redir); -bool is_affinity_mask_valid (cpumask_t cpumask); +bool is_affinity_mask_valid (cpumask_var_t cpumask);#define is_affinity_mask_valid is_affinity_mask_valid
arch/ia64/kernel/irq.c +2 -2
@@ -112,11 +112,11 @@ void set_irq_affinity_info (unsigned int irq, int hwid, int redir) } } -bool is_affinity_mask_valid (cpumask_t cpumask) +bool is_affinity_mask_valid (cpumask_var_t cpumask) {if (ia64_platform_is ("sn2" )) {/* Only allow one CPU to be specified in the smp_affinity mask */ -if (cpus_weight (cpumask) !=1 ) +if (cpumask_weight (cpumask) !=1 )return false; }return true;
kernel/irq/proc.c +1 -1
@@ -54,7 +54,7 @@ static ssize_t irq_affinity_proc_write(struct file *file,if (err) goto free_cpumask; -if (!is_affinity_mask_valid (*new_value)) { +if (!is_affinity_mask_valid (new_value)) { err = -EINVAL ; goto free_cpumask; }
PATCH 1/1 cpumask: convert RCU implementations, fix
there's a build warning regression as well on CONFIG_RCU_CLASSIC:
kernel/rcuclassic.c: In function ‘rcu_start_batch’:
kernel/rcuclassic.c:397: warning: passing argument 1 of ‘cpumask_andnot’ from incompatible pointer type
caused by:
bd232f9: cpumask: convert RCU implementations
fixed via the patch below. (i'll carry this in cpus4096 too, ok?)
Ingo
---------------------->
Quoted 1 line
From
9caab32c0d69e569e33caa03845cec73ae701f32Mon Sep 17 00:00:00 2001
From: Ingo Molnar <[email protected]>
Date: Sat, 3 Jan 2009 13:16:09 +0100
Subject: [PATCH] cpumask: convert RCU implementations, fix
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Impact: cleanup
This warning:
kernel/rcuclassic.c: In function ‘rcu_start_batch’:
kernel/rcuclassic.c:397: warning: passing argument 1 of ‘cpumask_andnot’ from incompatible pointer type
triggers because one usage site of rcp->cpumask was not converted to to_cpumask(rcp->cpumask). There's no ill effects of this bug.
--- kernel/rcuclassic.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)
kernel/rcuclassic.c +2 -1
@@ -394,7 +394,8 @@ static void rcu_start_batch(struct rcu_ctrlblk *rcp)* unnecessarily. */ smp_mb (); -cpumask_andnot (&rcp->cpumask, cpu_online_mask, nohz_cpu_mask); +cpumask_andnot (to_cpumask (rcp->cpumask), + cpu_online_mask, nohz_cpu_mask); rcp->signaled =0 ; }
Re: [PATCH] ia64: cpumask fix for is_affinity_mask_valid()
In reply to Ingo MolnarQuoted 1 line · On Saturday 03 January 2009 22:29:04 Ingo Molnar wrote:
update it to
cpumask_var_t.
Please change it to a const struct cpumask_t *.
It should be const, and of course 'const cpumask_var_t' isn't the same.
Thanks,
Rusty.
Re: [PATCH] ia64: cpumask fix for is_affinity_mask_valid()
Quoted 9 lines · Rusty Russell
Quoted 1 line · On Saturday 03 January 2009 22:29:04 Ingo Molnar wrote:
update it to
cpumask_var_t.Please change it to a const struct
cpumask_t*.It should be const, and of course 'const
cpumask_var_t' isn't the same.Thanks,
Rusty.
I have a fixup patch coming to correct that and remove an unneeded cpumask off the stack.
Mike
Re: [PATCH] ia64: cpumask fix for is_affinity_mask_valid()
Quoted 5 lines · Mike Travis
Quoted 4 lines · Rusty Russell
Quoted 1 line · On Saturday 03 January 2009 22:29:04 Ingo Molnar wrote:
update it to
cpumask_var_t.Please change it to a const struct
cpumask_t*.
yeah.
Quoted 8 lines
Quoted 5 lines
It should be const, and of course 'const
cpumask_var_t' isn't the same.Thanks,
Rusty.I have a fixup patch coming to correct that and remove an unneeded cpumask off the stack.
ok, will wait for that.
Ingo
Quoted 36 lines · Ingo Molnar
Quoted 10 lines · Rusty Russell
Quoted 9 lines · On Saturday 03 January 2009 07:08:40 Ingo Molnar wrote:
- architectures that have no
__fls(8 out of 21) fail to build:arch/cris arch/frv arch/h8300 arch/m32r arch/m68k arch/mn10300 arch/xtensaQuoted 2 lines
Fixes pushed, m68k should be OK tho; is this actual compile test? You have to look in
include/asm-m68kto see__fls.yeah, i stopped the tests after the first two build failures - the rest is a grep result from
arch/*/, that's whyinclude/asm-m68k/was left out.Quoted 7 lines
Quoted 5 lines
Rusty, would it be fine with you if we did all the remaining bits via tip/cpus4096? It's your tree and your bits and we wanted to send our remaining bits after your tree went to Linus but the conflict resolutions from Mike are valuable so i think we should reconsider the ordering.
Yeah, no reason for us to do the merge twice. As long as it ends upstream, I'm a happy camper.
great - lets do it that way then. I have pulled your fixes into the cpus4096 tree:
5ece5c5: xtensa: define
__fls
5c134da: mn10300: define__fls
16a2062: m32r: define__fls
9ddabc2: h8300: define__fls
ee38e51: frv: define__fls
0999769: cris: define__fls
Quoted 4 lines
Once we have figured out the CPU-hotplug lockdep splat (possibly due to Mike's changes not yours) i'll send it to Linus. Thanks,
Ingo
Thanks! Am working on that now.
Mike
Quoted 6 lines · Mike Travis
Quoted 4 lines
Once we have figured out the CPU-hotplug lockdep splat (possibly due to Mike's changes not yours) i'll send it to Linus. Thanks,
Ingo
Thanks! Am working on that now.
do you suspect any of the commits? I'm bisecting it right now but if you have bisected it already i wont repeat it.
Ingo
Quoted 5 lines · Ingo Molnar
Quoted 3 lines · Mike Travis
Quoted 3 lines
Once we have figured out the CPU-hotplug lockdep splat (possibly due to Mike's changes not yours) i'll send it to Linus. Thanks,
Quoted 7 lines
Quoted 2 lines
Quoted 1 line
Ingo
Thanks! Am working on that now.
do you suspect any of the commits? I'm bisecting it right now but if you have bisected it already i wont repeat it.
Ingo
I haven't done that yet, I'm just now getting your config to work pre-patches.
But yes, I suspect one of the changes to use "work_on_cpu" -- this usually causes the 2nd call to get_online_cpus().
Thanks,
Mike
Quoted 12 lines · Mike Travis
Quoted 11 lines · Ingo Molnar
Quoted 5 lines · Mike Travis
Quoted 4 lines
Once we have figured out the CPU-hotplug lockdep splat (possibly due to Mike's changes not yours) i'll send it to Linus. Thanks,
Ingo
Thanks! Am working on that now.
do you suspect any of the commits? I'm bisecting it right now but if you have bisected it already i wont repeat it.
Quoted 6 lines
Quoted 1 line
Ingo
I haven't done that yet, I'm just now getting your config to work pre-patches.
But yes, I suspect one of the changes to use "
work_on_cpu" -- this usually causes the 2nd call toget_online_cpus().
i suspect it's:
| commit 2d22bd5e74519854458ad372a89006e65f45e628 | Author: Mike Travis <[email protected]> | Date: Wed Dec 31 18:08:46 2008 -0800 | | x86: cleanup remaining cpumask_t code in microcode_core.c
as the microcode is loaded during CPU onlining.
Ingo
Quoted 27 lines · Ingo Molnar
Quoted 8 lines · Mike Travis
Quoted 7 lines · Ingo Molnar
Quoted 5 lines · Mike Travis
Quoted 4 lines
Once we have figured out the CPU-hotplug lockdep splat (possibly due to Mike's changes not yours) i'll send it to Linus. Thanks,
Ingo
Thanks! Am working on that now.
Quoted 8 lines
Quoted 4 lines
do you suspect any of the commits? I'm bisecting it right now but if you have bisected it already i wont repeat it.
Ingo
I haven't done that yet, I'm just now getting your config to work pre-patches.
But yes, I suspect one of the changes to use "
work_on_cpu" -- this usually causes the 2nd call toget_online_cpus().i suspect it's:
| commit 2d22bd5e74519854458ad372a89006e65f45e628 | Author: Mike Travis <[email protected]> | Date: Wed Dec 31 18:08:46 2008 -0800 | | x86: cleanup remaining cpumask_t code in microcode_core.c
Quoted 3 lines
as the microcode is loaded during CPU onlining.
Ingo
I think you're right. You need the coincidence of both writing to the /sys/devices/system/cpu/cpuX/online file along with the cpu coming up quickly enough to attempt to load the microcode before the write has returned. (Or something similar.)
Btw, I'm resending your "fix ia64" patch shortly.
Thanks,
Mike
Quoted 9 lines · Ingo Molnar
i suspect it's:
| commit 2d22bd5e74519854458ad372a89006e65f45e628 | Author: Mike Travis <[email protected]> | Date: Wed Dec 31 18:08:46 2008 -0800 | | x86: cleanup remaining cpumask_t code in microcode_core.cas the microcode is loaded during CPU onlining.
yep, that's the bad one. Should i revert it or do you have a safe fix in mind?
Ingo
Quoted 10 lines · Ingo Molnar
Quoted 8 lines · Ingo Molnar
i suspect it's:
| commit 2d22bd5e74519854458ad372a89006e65f45e628 | Author: Mike Travis <[email protected]> | Date: Wed Dec 31 18:08:46 2008 -0800 | | x86: cleanup remaining cpumask_t code in microcode_core.c
Quoted 6 lines
Quoted 1 line
as the microcode is loaded during CPU onlining.
yep, that's the bad one. Should i revert it or do you have a safe fix in mind?
Ingo
Probably revert for now. There are a few more following patches that also use 'work_on_cpu' so a better (more global?) fix should be used.
Any thought on using a recursive lock for cpu-hotplug-lock? (At least for get_online_cpus()?)
Thanks,
Mike
Quoted 16 lines · Mike Travis
Quoted 15 lines · Ingo Molnar
Quoted 9 lines · Ingo Molnar
i suspect it's:
| commit 2d22bd5e74519854458ad372a89006e65f45e628 | Author: Mike Travis <[email protected]> | Date: Wed Dec 31 18:08:46 2008 -0800 | | x86: cleanup remaining cpumask_t code in microcode_core.cas the microcode is loaded during CPU onlining.
yep, that's the bad one. Should i revert it or do you have a safe fix in mind?
Quoted 7 lines
Quoted 1 line
Ingo
Probably revert for now. There are a few more following patches that also use '
work_on_cpu' so a better (more global?) fix should be used.Any thought on using a recursive lock for cpu-hotplug-lock? (At least for
get_online_cpus()?)
but the problem has nothing to do with self-recursion. Take a look at the lockdep warning i posted (also below) - the locks are simply taken in the wrong order.
your change adds this cpu_hotplug.lock usage:
[ 43.652000] -> #1 (&cpu_hotplug.lock){--..}:
[ 43.652000] [<ffffffff8027a7c0>] __lock_acquire+0xf10/0x1360
[ 43.652000] [<ffffffff8027aca9>] lock_acquire+0x99/0xd0
[ 43.652000] [<ffffffff809b5e4a>] __mutex_lock_common+0xaa/0x450
[ 43.652000] [<ffffffff809b62cf>] mutex_lock_nested+0x3f/0x50
[ 43.652000] [<ffffffff802516ba>] get_online_cpus+0x3a/0x50
[ 43.652000] [<ffffffff802648dc>] work_on_cpu+0x6c/0xc0
[ 43.652000] [<ffffffff8022b2a2>] mc_sysdev_add+0x92/0xa0
[ 43.652000] [<ffffffff8050a800>] sysdev_driver_register+0xb0/0x140
[ 43.652000] [<ffffffff8163c792>] microcode_init+0xb2/0x13b
[ 43.652000] [<ffffffff8020a041>] do_one_initcall+0x41/0x180
[ 43.652000] [<ffffffff8162e6cb>] kernel_init+0x145/0x19d
[ 43.652000] [<ffffffff802146aa>] child_rip+0xa/0x20
[ 43.652000] [<ffffffffffffffff>] 0xffffffffffffffffwhich nests the inside sysdev_drivers_lock - which is wrong [sysdev_drivers_lock is a pretty lowlevel lock that generally nests inside the CPU hotplug lock].
If you want to use work_on_cpu() it should be done on a higher level, so that sysdev_drivers_lock is taken after the hotplug lock.
Ingo
[ 43.376051] lockdep: fixing up alternatives.
[ 43.380007] SMP alternatives: switching to UP code
[ 43.616014] CPU0 attaching NULL sched-domain.
[ 43.620068] CPU1 attaching NULL sched-domain.
[ 43.644482] CPU0 attaching NULL sched-domain.
[ 43.648264]
[ 43.648265] =======================================================
[ 43.652000] [ INFO: possible circular locking dependency detected ]
[ 43.652000] 2.6.28-05081-geeff031-dirty #37
[ 43.652000] -------------------------------------------------------
[ 43.652000] S99local/1238 is trying to acquire lock:
[ 43.652000] (sysdev_drivers_lock){--..}, at: [<ffffffff8050a52d>] sysdev_unregister+0x1d/0x80
[ 43.652000]
[ 43.652000] but task is already holding lock:
[ 43.652000] (&cpu_hotplug.lock){--..}, at: [<ffffffff802515d7>] cpu_hotplug_begin+0x27/0x60
[ 43.652000]
[ 43.652000] which lock already depends on the new lock.
[ 43.652000]
[ 43.652000]
[ 43.652000] the existing dependency chain (in reverse order) is:
[ 43.652000]
[ 43.652000] -> #1 (&cpu_hotplug.lock){--..}:
[ 43.652000] [<ffffffff8027a7c0>] __lock_acquire+0xf10/0x1360
[ 43.652000] [<ffffffff8027aca9>] lock_acquire+0x99/0xd0
[ 43.652000] [<ffffffff809b5e4a>] __mutex_lock_common+0xaa/0x450
[ 43.652000] [<ffffffff809b62cf>] mutex_lock_nested+0x3f/0x50
[ 43.652000] [<ffffffff802516ba>] get_online_cpus+0x3a/0x50
[ 43.652000] [<ffffffff802648dc>] work_on_cpu+0x6c/0xc0
[ 43.652000] [<ffffffff8022b2a2>] mc_sysdev_add+0x92/0xa0
[ 43.652000] [<ffffffff8050a800>] sysdev_driver_register+0xb0/0x140
[ 43.652000] [<ffffffff8163c792>] microcode_init+0xb2/0x13b
[ 43.652000] [<ffffffff8020a041>] do_one_initcall+0x41/0x180
[ 43.652000] [<ffffffff8162e6cb>] kernel_init+0x145/0x19d
[ 43.652000] [<ffffffff802146aa>] child_rip+0xa/0x20
[ 43.652000] [<ffffffffffffffff>] 0xffffffffffffffff
[ 43.652000]
[ 43.652000] -> #0 (sysdev_drivers_lock){--..}:
[ 43.652000] [<ffffffff8027a89c>] __lock_acquire+0xfec/0x1360
[ 43.652000] [<ffffffff8027aca9>] lock_acquire+0x99/0xd0
[ 43.652000] [<ffffffff809b5e4a>] __mutex_lock_common+0xaa/0x450
[ 43.652000] [<ffffffff809b62cf>] mutex_lock_nested+0x3f/0x50
[ 43.652000] [<ffffffff8050a52d>] sysdev_unregister+0x1d/0x80
[ 43.652000] [<ffffffff809af9d1>] mce_cpu_callback+0xce/0x101
[ 43.652000] [<ffffffff809bbb75>] notifier_call_chain+0x65/0xa0
[ 43.652000] [<ffffffff8026d696>] raw_notifier_call_chain+0x16/0x20
[ 43.652000] [<ffffffff80964a00>] _cpu_down+0x240/0x350
[ 43.652000] [<ffffffff80964b8b>] cpu_down+0x7b/0xa0
[ 43.652000] [<ffffffff80966268>] store_online+0x48/0xa0
[ 43.652000] [<ffffffff80509e90>] sysdev_store+0x20/0x30
[ 43.652000] [<ffffffff80335ddf>] sysfs_write_file+0xcf/0x140
[ 43.652000] [<ffffffff802dc1f7>] vfs_write+0xc7/0x150
[ 43.652000] [<ffffffff802dc375>] sys_write+0x55/0x90
[ 43.652000] [<ffffffff802133ca>] system_call_fastpath+0x16/0x1b
[ 43.652000] [<ffffffffffffffff>] 0xffffffffffffffff
[ 43.652000]
[ 43.652000] other info that might help us debug this:
[ 43.652000]
[ 43.652000] 3 locks held by S99local/1238:
[ 43.652000] #0: (&buffer->mutex){--..}, at: [<ffffffff80335d58>] sysfs_write_file+0x48/0x140
[ 43.652000] #1: (cpu_add_remove_lock){--..}, at: [<ffffffff80964b3f>] cpu_down+0x2f/0xa0
[ 43.652000] #2: (&cpu_hotplug.lock){--..}, at: [<ffffffff802515d7>] cpu_hotplug_begin+0x27/0x60
[ 43.652000]
[ 43.652000] stack backtrace:
[ 43.652000] Pid: 1238, comm: S99local Not tainted 2.6.28-05081-geeff031-dirty #37
[ 43.652000] Call Trace:
[ 43.652000] [<ffffffff80277f24>] print_circular_bug_tail+0xa4/0x100
[ 43.652000] [<ffffffff8027a89c>] __lock_acquire+0xfec/0x1360
[ 43.652000] [<ffffffff8027aca9>] lock_acquire+0x99/0xd0
[ 43.652000] [<ffffffff8050a52d>] ? sysdev_unregister+0x1d/0x80
[ 43.652000] [<ffffffff809b5e4a>] __mutex_lock_common+0xaa/0x450
[ 43.652000] [<ffffffff8050a52d>] ? sysdev_unregister+0x1d/0x80
[ 43.652000] [<ffffffff8050a52d>] ? sysdev_unregister+0x1d/0x80
[ 43.652000] [<ffffffff809b62cf>] mutex_lock_nested+0x3f/0x50
[ 43.652000] [<ffffffff8050a52d>] sysdev_unregister+0x1d/0x80
[ 43.652000] [<ffffffff809af9d1>] mce_cpu_callback+0xce/0x101
[ 43.652000] [<ffffffff809bbb75>] notifier_call_chain+0x65/0xa0
[ 43.652000] [<ffffffff8026d696>] raw_notifier_call_chain+0x16/0x20
[ 43.652000] [<ffffffff80964a00>] _cpu_down+0x240/0x350
[ 43.652000] [<ffffffff809b4763>] ? wait_for_common+0xe3/0x1b0
[ 43.652000] [<ffffffff80964b8b>] cpu_down+0x7b/0xa0
[ 43.652000] [<ffffffff80966268>] store_online+0x48/0xa0
[ 43.652000] [<ffffffff80509e90>] sysdev_store+0x20/0x30
[ 43.652000] [<ffffffff80335ddf>] sysfs_write_file+0xcf/0x140
[ 43.652000] [<ffffffff802dc1f7>] vfs_write+0xc7/0x150
[ 43.652000] [<ffffffff802dc375>] sys_write+0x55/0x90
[ 43.652000] [<ffffffff802133ca>] system_call_fastpath+0x16/0x1b
[ 43.652104] device: 'msr1': device_unregister
[ 43.656005] PM: Removing info for No Bus:msr1Quoted 45 lines · Ingo Molnar
Quoted 15 lines · Mike Travis
Quoted 14 lines · Ingo Molnar
Quoted 9 lines · Ingo Molnar
i suspect it's:
| commit 2d22bd5e74519854458ad372a89006e65f45e628 | Author: Mike Travis <[email protected]> | Date: Wed Dec 31 18:08:46 2008 -0800 | | x86: cleanup remaining cpumask_t code in microcode_core.cas the microcode is loaded during CPU onlining.
yep, that's the bad one. Should i revert it or do you have a safe fix in mind?
Quoted 6 lines
Quoted 1 line
Ingo
Probably revert for now. There are a few more following patches that also use '
work_on_cpu' so a better (more global?) fix should be used.Any thought on using a recursive lock for cpu-hotplug-lock? (At least for
get_online_cpus()?)but the problem has nothing to do with self-recursion. Take a look at the lockdep warning i posted (also below) - the locks are simply taken in the wrong order.
your change adds this cpu_hotplug.lock usage:
[ 43.652000] -> #1 (&cpu_hotplug.lock){--..}: [ 43.652000] [<ffffffff8027a7c0>] __lock_acquire+0xf10/0x1360 [ 43.652000] [<ffffffff8027aca9>] lock_acquire+0x99/0xd0 [ 43.652000] [<ffffffff809b5e4a>] __mutex_lock_common+0xaa/0x450 [ 43.652000] [<ffffffff809b62cf>] mutex_lock_nested+0x3f/0x50 [ 43.652000] [<ffffffff802516ba>] get_online_cpus+0x3a/0x50 [ 43.652000] [<ffffffff802648dc>] work_on_cpu+0x6c/0xc0 [ 43.652000] [<ffffffff8022b2a2>] mc_sysdev_add+0x92/0xa0 [ 43.652000] [<ffffffff8050a800>] sysdev_driver_register+0xb0/0x140 [ 43.652000] [<ffffffff8163c792>] microcode_init+0xb2/0x13b [ 43.652000] [<ffffffff8020a041>] do_one_initcall+0x41/0x180 [ 43.652000] [<ffffffff8162e6cb>] kernel_init+0x145/0x19d [ 43.652000] [<ffffffff802146aa>] child_rip+0xa/0x20 [ 43.652000] [<ffffffffffffffff>] 0xffffffffffffffff
Quoted 8 lines
which nests the inside
sysdev_drivers_lock- which is wrong [sysdev_drivers_lockis a pretty lowlevel lock that generally nests inside the CPU hotplug lock].If you want to use
work_on_cpu()it should be done on a higher level, so thatsysdev_drivers_lockis taken after the hotplug lock.Ingo
Ok, thanks, I will look in that direction.
Mike
Quoted 89 lines
[ 43.376051] lockdep: fixing up alternatives.
[ 43.380007] SMP alternatives: switching to UP code
[ 43.616014] CPU0 attaching NULL sched-domain.
[ 43.620068] CPU1 attaching NULL sched-domain.
[ 43.644482] CPU0 attaching NULL sched-domain.
[ 43.648264]
[ 43.648265] =======================================================
[ 43.652000] [ INFO: possible circular locking dependency detected ]
[ 43.652000] 2.6.28-05081-geeff031-dirty #37
[ 43.652000] -------------------------------------------------------
[ 43.652000] S99local/1238 is trying to acquire lock:
[ 43.652000] (sysdev_drivers_lock){--..}, at: [<ffffffff8050a52d>] sysdev_unregister+0x1d/0x80
[ 43.652000]
[ 43.652000] but task is already holding lock:
[ 43.652000] (&cpu_hotplug.lock){--..}, at: [<ffffffff802515d7>] cpu_hotplug_begin+0x27/0x60
[ 43.652000]
[ 43.652000] which lock already depends on the new lock.
[ 43.652000]
[ 43.652000]
[ 43.652000] the existing dependency chain (in reverse order) is:
[ 43.652000]
[ 43.652000] -> #1 (&cpu_hotplug.lock){--..}:
[ 43.652000] [<ffffffff8027a7c0>] __lock_acquire+0xf10/0x1360
[ 43.652000] [<ffffffff8027aca9>] lock_acquire+0x99/0xd0
[ 43.652000] [<ffffffff809b5e4a>] __mutex_lock_common+0xaa/0x450
[ 43.652000] [<ffffffff809b62cf>] mutex_lock_nested+0x3f/0x50
[ 43.652000] [<ffffffff802516ba>] get_online_cpus+0x3a/0x50
[ 43.652000] [<ffffffff802648dc>] work_on_cpu+0x6c/0xc0
[ 43.652000] [<ffffffff8022b2a2>] mc_sysdev_add+0x92/0xa0
[ 43.652000] [<ffffffff8050a800>] sysdev_driver_register+0xb0/0x140
[ 43.652000] [<ffffffff8163c792>] microcode_init+0xb2/0x13b
[ 43.652000] [<ffffffff8020a041>] do_one_initcall+0x41/0x180
[ 43.652000] [<ffffffff8162e6cb>] kernel_init+0x145/0x19d
[ 43.652000] [<ffffffff802146aa>] child_rip+0xa/0x20
[ 43.652000] [<ffffffffffffffff>] 0xffffffffffffffff
[ 43.652000]
[ 43.652000] -> #0 (sysdev_drivers_lock){--..}:
[ 43.652000] [<ffffffff8027a89c>] __lock_acquire+0xfec/0x1360
[ 43.652000] [<ffffffff8027aca9>] lock_acquire+0x99/0xd0
[ 43.652000] [<ffffffff809b5e4a>] __mutex_lock_common+0xaa/0x450
[ 43.652000] [<ffffffff809b62cf>] mutex_lock_nested+0x3f/0x50
[ 43.652000] [<ffffffff8050a52d>] sysdev_unregister+0x1d/0x80
[ 43.652000] [<ffffffff809af9d1>] mce_cpu_callback+0xce/0x101
[ 43.652000] [<ffffffff809bbb75>] notifier_call_chain+0x65/0xa0
[ 43.652000] [<ffffffff8026d696>] raw_notifier_call_chain+0x16/0x20
[ 43.652000] [<ffffffff80964a00>] _cpu_down+0x240/0x350
[ 43.652000] [<ffffffff80964b8b>] cpu_down+0x7b/0xa0
[ 43.652000] [<ffffffff80966268>] store_online+0x48/0xa0
[ 43.652000] [<ffffffff80509e90>] sysdev_store+0x20/0x30
[ 43.652000] [<ffffffff80335ddf>] sysfs_write_file+0xcf/0x140
[ 43.652000] [<ffffffff802dc1f7>] vfs_write+0xc7/0x150
[ 43.652000] [<ffffffff802dc375>] sys_write+0x55/0x90
[ 43.652000] [<ffffffff802133ca>] system_call_fastpath+0x16/0x1b
[ 43.652000] [<ffffffffffffffff>] 0xffffffffffffffff
[ 43.652000]
[ 43.652000] other info that might help us debug this:
[ 43.652000]
[ 43.652000] 3 locks held by S99local/1238:
[ 43.652000] #0: (&buffer->mutex){--..}, at: [<ffffffff80335d58>] sysfs_write_file+0x48/0x140
[ 43.652000] #1: (cpu_add_remove_lock){--..}, at: [<ffffffff80964b3f>] cpu_down+0x2f/0xa0
[ 43.652000] #2: (&cpu_hotplug.lock){--..}, at: [<ffffffff802515d7>] cpu_hotplug_begin+0x27/0x60
[ 43.652000]
[ 43.652000] stack backtrace:
[ 43.652000] Pid: 1238, comm: S99local Not tainted 2.6.28-05081-geeff031-dirty #37
[ 43.652000] Call Trace:
[ 43.652000] [<ffffffff80277f24>] print_circular_bug_tail+0xa4/0x100
[ 43.652000] [<ffffffff8027a89c>] __lock_acquire+0xfec/0x1360
[ 43.652000] [<ffffffff8027aca9>] lock_acquire+0x99/0xd0
[ 43.652000] [<ffffffff8050a52d>] ? sysdev_unregister+0x1d/0x80
[ 43.652000] [<ffffffff809b5e4a>] __mutex_lock_common+0xaa/0x450
[ 43.652000] [<ffffffff8050a52d>] ? sysdev_unregister+0x1d/0x80
[ 43.652000] [<ffffffff8050a52d>] ? sysdev_unregister+0x1d/0x80
[ 43.652000] [<ffffffff809b62cf>] mutex_lock_nested+0x3f/0x50
[ 43.652000] [<ffffffff8050a52d>] sysdev_unregister+0x1d/0x80
[ 43.652000] [<ffffffff809af9d1>] mce_cpu_callback+0xce/0x101
[ 43.652000] [<ffffffff809bbb75>] notifier_call_chain+0x65/0xa0
[ 43.652000] [<ffffffff8026d696>] raw_notifier_call_chain+0x16/0x20
[ 43.652000] [<ffffffff80964a00>] _cpu_down+0x240/0x350
[ 43.652000] [<ffffffff809b4763>] ? wait_for_common+0xe3/0x1b0
[ 43.652000] [<ffffffff80964b8b>] cpu_down+0x7b/0xa0
[ 43.652000] [<ffffffff80966268>] store_online+0x48/0xa0
[ 43.652000] [<ffffffff80509e90>] sysdev_store+0x20/0x30
[ 43.652000] [<ffffffff80335ddf>] sysfs_write_file+0xcf/0x140
[ 43.652000] [<ffffffff802dc1f7>] vfs_write+0xc7/0x150
[ 43.652000] [<ffffffff802dc375>] sys_write+0x55/0x90
[ 43.652000] [<ffffffff802133ca>] system_call_fastpath+0x16/0x1b
[ 43.652104] device: 'msr1': device_unregister
[ 43.656005] PM: Removing info for No Bus:msr1Quoted 4 lines · Mike Travis
Quoted 2 lines
yep, that's the bad one. Should i revert it or do you have a safe fix in mind?
Probably revert for now. [...]
done.
But -tip testing found another bug today as well, a boot crash with certain (rare) 64-bit configs:
[ 1.588202] ACPI: PCI Interrupt Link [LNKA] (IRQs<1>BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 [ 1.588012] IP: [<ffffffff80239778>] find_busiest_group+0x198/0xa20
[ 1.588048] Call Trace: [ 1.588049] <IRQ> <0> [<ffffffff80240f96>] rebalance_domains+0x196/0x5e0 [ 1.588052] [<ffffffff80270b15>] ? lock_release_holdtime+0x35/0x1e0 [ 1.588055] [<ffffffff80983f70>] ? _spin_unlock_irq+0x30/0x40 [ 1.588058] [<ffffffff8024300e>] run_rebalance_domains+0x4e/0x120 [ 1.588060] [<ffffffff8024f80c>] __do_softirq+0xac/0x190 [ 1.588063] [<ffffffff8020d13c>] call_softirq+0x1c/0x30 [ 1.588066] [<ffffffff8020ef35>] do_softirq+0x75/0xa0 [ 1.588067] [<ffffffff8024f47d>] irq_exit+0x9d/0xb0 [ 1.588069] [<ffffffff80984f5d>] smp_apic_timer_interrupt+0x8d/0xc3 [ 1.588071] [<ffffffff8020cb73>] apic_timer_interrupt+0x13/0x20
i just bisected it back to:
| 74c5409893751c400547184751410c61930043b2 is first bad commit | commit 74c5409893751c400547184751410c61930043b2 | Author: Mike Travis <[email protected]> | Date: Wed Dec 31 18:08:45 2008 -0800 | | x86: cleanup remaining cpumask_t ops in smpboot code | | Impact: Reduce memory usage and use new cpumask API.
this is in the final pieces of changes you did after pulling Rusty's tree:
26e2013: x86: setup_per_cpu_areas() cleanup
44aa683: cpumask: fix compile error when CONFIG_NR_CPUS is not defined
eeff031: cpumask: use alloc_cpumask_var_node where appropriate 40fbcb0: cpumask: convert shared_cpu_map in acpi_processor* structs to cpumask_var 197e99c: x86: use cpumask_var_t in acpi/boot.c
2d22bd5: x86: cleanup remaining cpumask_t code in microcode_core.c 22022f5: x86: cleanup remaining cpumask_t code in mce_amd_64.c
b5f3096: x86: cleanup some remaining usages of NR_CPUS where s/b nr_cpu_ids
efb897c: sched: put back some stack hog changes that were undone in kernel/sched.c 74c5409: x86: cleanup remaining cpumask_t ops in smpboot code 8627b2a: x86: enable cpus display of kernel_max and offlined cpus 095fb96: Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/ru
i think i'll just rebase the tail portion of cpus4096 starting at 8627b2a
- this keeps most of the history intact and avoids these ugly reverts.
Also, while bisecting this window of commits i found that neither would build successfully due to a typo - and the typo is fixed in 095fb96. So since we rebase this portion anyway due to excessive amount of bugs, i'll make it fully bisectable by rebasing right at 095fb96, backmerge the fixlet from eeff031 and redo the whole series dropping the two bad patches. Since this portion of the tree has no appreciable testing value the rebase is the right thing to do here.
Ingo
Quoted 7 lines · Ingo Molnar
Also, while bisecting this window of commits i found that neither would build successfully due to a typo - and the typo is fixed in 095fb96. So since we rebase this portion anyway due to excessive amount of bugs, i'll make it fully bisectable by rebasing right at 095fb96, backmerge the fixlet from eeff031 and redo the whole series dropping the two bad patches. Since this portion of the tree has no appreciable testing value the rebase is the right thing to do here.
okay, i've done this now and pushed out the resulting tree to tip/cpus4096-v2. It is a no-content-changed reshuffle of the tail ~10 commits of tip/cpus4096:
$ git diff cpus4096..cpus4096-v2 $
i dropped the two broken patches i bisected today, and backmerged a fixlet to make it more bisectable, and reordered fixes next to the merge point - and the new changes now come after that.
this finally is something that has no known regressions and looks pushable to Linus. I've started the -tip tests, lets see how well this holds up now.
Ingo
Quoted 63 lines · Ingo Molnar
Quoted 3 lines · Mike Travis
Quoted 2 lines
yep, that's the bad one. Should i revert it or do you have a safe fix in mind?
Probably revert for now. [...]
done.
But -tip testing found another bug today as well, a boot crash with certain (rare) 64-bit configs:
[ 1.588202] ACPI: PCI Interrupt Link [LNKA] (IRQs<1>BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 [ 1.588012] IP: [<ffffffff80239778>] find_busiest_group+0x198/0xa20[ 1.588048] Call Trace: [ 1.588049] <IRQ> <0> [<ffffffff80240f96>] rebalance_domains+0x196/0x5e0 [ 1.588052] [<ffffffff80270b15>] ? lock_release_holdtime+0x35/0x1e0 [ 1.588055] [<ffffffff80983f70>] ? _spin_unlock_irq+0x30/0x40 [ 1.588058] [<ffffffff8024300e>] run_rebalance_domains+0x4e/0x120 [ 1.588060] [<ffffffff8024f80c>] __do_softirq+0xac/0x190 [ 1.588063] [<ffffffff8020d13c>] call_softirq+0x1c/0x30 [ 1.588066] [<ffffffff8020ef35>] do_softirq+0x75/0xa0 [ 1.588067] [<ffffffff8024f47d>] irq_exit+0x9d/0xb0 [ 1.588069] [<ffffffff80984f5d>] smp_apic_timer_interrupt+0x8d/0xc3 [ 1.588071] [<ffffffff8020cb73>] apic_timer_interrupt+0x13/0x20i just bisected it back to:
| 74c5409893751c400547184751410c61930043b2 is first bad commit | commit 74c5409893751c400547184751410c61930043b2 | Author: Mike Travis <[email protected]> | Date: Wed Dec 31 18:08:45 2008 -0800 | | x86: cleanup remaining cpumask_t ops in smpboot code | | Impact: Reduce memory usage and use new cpumask API.this is in the final pieces of changes you did after pulling Rusty's tree:
26e2013: x86:
setup_per_cpu_areas()cleanup
44aa683: cpumask: fix compile error whenCONFIG_NR_CPUSis not defined
eeff031: cpumask: usealloc_cpumask_var_nodewhere appropriate 40fbcb0: cpumask: convertshared_cpu_mapin acpi_processor* structs tocpumask_var197e99c: x86: usecpumask_var_tinacpi/boot.c
2d22bd5: x86: cleanup remainingcpumask_tcode inmicrocode_core.c22022f5: x86: cleanup remainingcpumask_tcode inmce_amd_64.c
b5f3096: x86: cleanup some remaining usages ofNR_CPUSwhere s/bnr_cpu_ids
efb897c: sched: put back some stack hog changes that were undone inkernel/sched.c74c5409: x86: cleanup remainingcpumask_tops in smpboot code 8627b2a: x86: enable cpus display ofkernel_maxand offlined cpus 095fb96: Merge branch 'master' ofgit://git.kernel.org/pub/scm/linux/kernel/git/rui think i'll just rebase the tail portion of cpus4096 starting at 8627b2a
- this keeps most of the history intact and avoids these ugly reverts.Also, while bisecting this window of commits i found that neither would build successfully due to a typo - and the typo is fixed in 095fb96. So since we rebase this portion anyway due to excessive amount of bugs, i'll make it fully bisectable by rebasing right at 095fb96, backmerge the fixlet from eeff031 and redo the whole series dropping the two bad patches. Since this portion of the tree has no appreciable testing value the rebase is the right thing to do here.
Quoted 1 line
Ingo
Ok, thanks. Still working through my queue... I'll re-pull when you've got your part done.
Mike
Quoted 7 lines · On Saturday 03 January 2009 06:36:33 Linus Torvalds wrote:
Quoted 4 lines · Rusty Russell
OK, this is the bulk of the conversion to the new cpumask operators. The x86-specific parts (the most aggressive large-NR_CPUS arch) are going via Ingo's tree.
Quoted 7 lines
This gets lots of conflicts for me. Some of them look simple enough, but not all.
io_apic.cgets lots of nasty conflicts, and it _looks_ like I should just pick the version of the file that I already have (because the only thing that comes in from that is yet another merge commit), butkernel/sched.calso gets conflicts in areas with FIXME's etc.Rusty, Ingo, can you work this out? I pushed out my current tree.
Yes, some went via Ingo's tree and there are some known overlaps.
I'll clean it up and resend pull req this weekend.
Thanks,
Rusty.
Quoted 4 lines · Rusty Russell
commit
d036e67b40f52bdd95392390108defbac7e53837Author: Rusty Russell <[email protected]> Date: Thu Jan 1 10:12:26 2009 +1030
Quoted 9 lines
cpumask: convert kernel/irq
Impact: Reduce stack usage, use new cpumask API. ALPHA mod!
Main change is that
irq_default_affinitybecomes acpumask_var_t, so treat it as a pointer (this effects alpha).Signed-off-by Rusty Russell <[email protected]>
Which contains:
Quoted 8 lines
kernel/irq/manage.c +0
-0@@ -16,8 +16,15 @@#include "internals.h" #ifdef CONFIG_SMP
kernel/irq/manage.c line 27
Quoted 10 lines
kernel/irq/manage.c +8
-1+cpumask_var_t irq_default_affinity; -cpumask_t irq_default_affinity =CPU_MASK_ALL ; +static int init_irq_default_affinity (void ) +{ +alloc_cpumask_var (&irq_default_affinity,GFP_KERNEL ); +cpumask_setall (irq_default_affinity); +return 0 ; +} +core_initcall (init_irq_default_affinity);
I think core_initcall is too late to be initializing irq_default_affinity. This happens way after init_IRQ() is called and for my target (mips/cavium_octeon) after the timer and SMP related irqs are setup.
I had been setting irq_default_affinity in init_IRQ(), and I could probably do it later with no real problem, but this seems wrong to me. Data that is potentially used in interrupt configuration and processing should be initialized before it is used.
David Daney