Skip to contentSkip to conversation

cpumask tree

GIT PULL In review Rusty Russell 7 participants 46 messages Jan 1, 2009 – Jan 26, 2009 1 message not in the archive

Subsystems (42)
lore
Share & tools
lore

Rusty Russell

author Reviewed-by Grant Grundler Tested-by Tony Luck Acked-by Ingo Molnar Acked-by Ingo Molnar Reviewed-by Grant Grundler Acked-by Ingo Molnar Acked-by Greg Kroah-Hartman Acked-by Ingo Molnar Acked-by David S. Miller Acked-by Hoang-Nam Nguyen Tested-by Hoang-Nam Nguyen Acked-by Hoang-Nam Nguyen Tested-by Hoang-Nam Nguyen Acked-by Ingo Molnar Acked-by Ingo Molnar Acked-by Ingo Molnar Acked-by Hirokazu Takata Acked-by Ingo Molnar Acked-by James Morris Acked-by Mike Frysinger Acked-by Steven Rostedt Reviewed-by Christoph Lameter
#0 lore raw

OK, this is the bulk of the conversion to the new cpumask operators. The x86-specific parts (the most aggressive large-NR_CPUS arch) are going via Ingo's tree.

Goals:
1) Get cpumasks off the stack when CONFIG_CPUMASK_OFFSTACK. This should be achieved for core & x86 by 2.6.29. cpumask_var_t is struct cpumask[1] for CONFIG_CPUMASK_OFFSTACK=n (and alloc_cpumask_var et al. a noop).
2) Convert to new cpumask functions which only go to nr_cpu_ids for large NR_CPUS, so booting huge configured kernels on small machines doesn't suck.
3) Allocate smaller cpumasks when nr_cpu_ids < NR_CPUS, when CONFIG_CPUMASK_OFFSTACK=y. This requires (2) to be completed.
4) Use cpumask_var_t for static cpumasks as well, or raw bitmaps if we really have to. The former will aave space for nr_cpu_ids << NR_CPUS.
5) Ban on-stack cpumasks (to ensure (1) doesn't get reverted) and cpumask assignment (for (3)) by making struct cpumask undefined when CONFIG_CPUMASK_OFFSTACK=y. This means (4) needs to be finished.

Between this and Ingo's tree, we achieve (1) and part of (2) and (4). Completing the work is expected by 2.6.30.

Note that we can't stop people creating bitmaps of NR_CPUS on the stack and using to_cpumask() on them. But at least it should stand out.

Cheers,
Rusty.

The following changes since commit 6a94cb73064c952255336cc57731904174b2c58f:
Linus Torvalds (1):
Merge branch 'for-linus' of git://oss.sgi.com/xfs/xfs

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/rusty/linux-2.6-cpumask.git master

Li Zefan (1):
cpumask: fix bogus kernel-doc

Mike Travis (4):
cpumask: Add alloc_cpumask_var_node()
cpumask: documentation for cpumask_var_t
cpumask: add sysfs displays for configured and disabled cpu maps
sysfs: add documentation to cputopology.txt for system cpumasks

Rusty Russell (64):
cpumask: centralize cpu_online_map and cpu_possible_map
cpumask: change cpumask_scnprintf, cpumask_parse_user, cpulist_parse, and cpulist_scnprintf to take pointers.
cpumask: make irq_set_affinity() take a const struct cpumask
cpumask: convert struct clock_event_device to cpumask pointers.
cpumask: Add CONFIG_CPUMASK_OFFSTACK
cpumask: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: Use all NR_CPUS bits unless CONFIG_CPUMASK_OFFSTACK
cpumask: x86: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: sparc: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: sh: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: powerpc: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: IA64: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: Mips: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: alpha: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: cpu_coregroup_mask(): x86
cpumask: cpu_coregroup_mask(): sparc
cpumask: cpu_coregroup_mask(): s390
cpumask: Replace cpu_coregroup_map with cpu_coregroup_mask
Merge branch 'master' of git://git.kernel.org/.../torvalds/linux-2.6
cpumask: make CONFIG_NR_CPUS always valid.
bitmap: test for constant as well as small size for inline versions
bitmap: fix seq_bitmap and seq_cpumask to take const pointer
cpumask: switch over to cpu_online/possible/active/present_mask: core
cpumask: make cpumask.h eat its own dogfood.
cpumask: make set_cpu_*/init_cpu_* out-of-line
cpumask: smp_call_function_many()
cpumask: arch_send_call_function_ipi_mask: core
cpumask: use for_each_online_cpu() in drivers/infiniband/hw/ehca/ehca_irq.c
cpumask: use new cpumask API in drivers/infiniband/hw/ehca
cpumask: use new cpumask API in drivers/infiniband/hw/ipath
cpumask: Use nr_cpu_ids in seq_cpumask
Merge branch 'master' of git://git.kernel.org/.../torvalds/linux-2.6
cpumask: Remove IA64 definition of total_cpus now it's in core code
percpu: fix percpu accessors to potentially !cpu_possible() cpus: pnpbios
percpu: fix percpu accessors to potentially !cpu_possible() cpus: m32r
cpumask: prepare for iterators to only go to nr_cpu_ids/nr_cpumask_bits.: core
cpumask: Use accessors code in core
parisc: remove gratuitous cpu_online_map declaration.
avr32: define __fls
blackfin: define __fls
m68k: define __fls
m68knommu: define __fls
bitmap: find_last_bit()
cpumask: Use find_last_bit()
cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): sparc
cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): s390
cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): powerpc
cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): ia64
cpumask: convert kernel trace functions
cpumask: convert kernel trace functions further
cpumask: remove any_online_cpu() users: kernel/
cpumask: remove any_online_cpu() users: mm/
cpumask: convert kernel/compat.c
cpumask: convert kernel/workqueue.c
cpumask: convert kernel time functions
cpumask: convert kernel/irq
cpumask: convert RCU implementations
cpumask: convert kernel/profile.c
cpumask: convert kernel/cpu.c
cpumask: convert rest of files in kernel/
cpumask: convert mm/
cpumask: replace for_each_cpu_mask_nr with for_each_cpu in kernel/time/
cpumask: zero extra bits in alloc_cpumask_var_node
cpumask: CONFIG_DISABLE_OBSOLETE_CPUMASK_FUNCTIONS

 Documentation/cputopology.txt                |   48 ++++
 arch/alpha/include/asm/smp.h                 |    1 -   
 arch/alpha/include/asm/topology.h            |   17 ++  
 arch/alpha/kernel/irq.c                      |    5 +-  
 arch/alpha/kernel/process.c                  |    2 +   
 arch/alpha/kernel/setup.c                    |    5 +   
 arch/alpha/kernel/smp.c                      |    7 +-  
 arch/alpha/kernel/sys_dp264.c                |    8 +-  
 arch/alpha/kernel/sys_titan.c                |    4 +-  
 arch/arm/common/gic.c                        |    4 +-  
 arch/arm/kernel/irq.c                        |    2 +-  
 arch/arm/kernel/smp.c                        |   10 -   
 arch/arm/mach-at91/at91rm9200_time.c         |    3 +-  
 arch/arm/mach-at91/at91sam926x_time.c        |    2 +-  
 arch/arm/mach-davinci/time.c                 |    2 +-  
 arch/arm/mach-imx/time.c                     |    2 +-  
 arch/arm/mach-ixp4xx/common.c                |    2 +-  
 arch/arm/mach-msm/timer.c                    |    2 +-  
 arch/arm/mach-ns9xxx/time-ns9360.c           |    2 +-  
 arch/arm/mach-omap1/time.c                   |    2 +-  
 arch/arm/mach-omap1/timer32k.c               |    2 +-  
 arch/arm/mach-omap2/timer-gp.c               |    2 +-  
 arch/arm/mach-pxa/time.c                     |    2 +-  
 arch/arm/mach-realview/core.c                |    2 +-  
 arch/arm/mach-realview/localtimer.c          |    4 +-  
 arch/arm/mach-sa1100/time.c                  |    2 +-  
 arch/arm/mach-versatile/core.c               |    2 +-  
 arch/arm/oprofile/op_model_mpcore.c          |    4 +-  
 arch/arm/plat-mxc/time.c                     |    2 +-  
 arch/arm/plat-orion/time.c                   |    2 +-  
 arch/avr32/include/asm/bitops.h              |    5 +   
 arch/avr32/kernel/time.c                     |    2 +-  
 arch/blackfin/include/asm/bitops.h           |    1 +   
 arch/blackfin/kernel/time-ts.c               |    2 +-  
 arch/cris/arch-v32/kernel/irq.c              |    4 +-  
 arch/cris/arch-v32/kernel/smp.c              |    4 -   
 arch/cris/include/asm/smp.h                  |    1 -   
 arch/ia64/hp/sim/hpsim_irq.c                 |    2 +-  
 arch/ia64/include/asm/smp.h                  |    1 -   
 arch/ia64/include/asm/topology.h             |    9 +-  
 arch/ia64/kernel/acpi.c                      |    3 +-  
 arch/ia64/kernel/iosapic.c                   |   35 ++--
 arch/ia64/kernel/irq.c                       |    9 +-  
 arch/ia64/kernel/msi_ia64.c                  |   12 +-  
 arch/ia64/kernel/smpboot.c                   |   10 +-  
 arch/ia64/kernel/topology.c                  |    2 +-  
 arch/ia64/sn/kernel/irq.c                    |    6 +-  
 arch/ia64/sn/kernel/msi_sn.c                 |    7 +-  
 arch/ia64/sn/kernel/sn2/sn_hwperf.c          |   27 +-- 
 arch/m32r/Kconfig                            |    1 +   
 arch/m32r/kernel/smpboot.c                   |    8 +-  
 arch/m68knommu/include/asm/bitops.h          |    1 +   
 arch/m68knommu/platform/coldfire/pit.c       |    2 +-  
 arch/mips/include/asm/irq.h                  |    3 +-  
 arch/mips/include/asm/mach-ip27/topology.h   |    4 +-  
 arch/mips/include/asm/smp.h                  |    3 -   
 arch/mips/jazz/irq.c                         |    2 +-  
 arch/mips/kernel/cevt-bcm1480.c              |    4 +-  
 arch/mips/kernel/cevt-ds1287.c               |    2 +-  
 arch/mips/kernel/cevt-gt641xx.c              |    2 +-  
 arch/mips/kernel/cevt-r4k.c                  |    2 +-  
 arch/mips/kernel/cevt-sb1250.c               |    4 +-  
 arch/mips/kernel/cevt-smtc.c                 |    2 +-  
 arch/mips/kernel/cevt-txx9.c                 |    2 +-  
 arch/mips/kernel/i8253.c                     |    2 +-  
 arch/mips/kernel/irq-gic.c                   |    6 +-  
 arch/mips/kernel/smp-cmp.c                   |    6 +-  
 arch/mips/kernel/smp-mt.c                    |    2 +-  
 arch/mips/kernel/smp.c                       |    7 +-  
 arch/mips/kernel/smtc.c                      |    6 +-  
 arch/mips/mti-malta/malta-smtc.c             |    6 +-  
 arch/mips/nxp/pnx8550/common/time.c          |    1 +   
 arch/mips/pmc-sierra/yosemite/smp.c          |    6 +-  
 arch/mips/sgi-ip27/ip27-smp.c                |    2 +-  
 arch/mips/sgi-ip27/ip27-timer.c              |    2 +-  
 arch/mips/sibyte/bcm1480/irq.c               |    8 +-  
 arch/mips/sibyte/bcm1480/smp.c               |    8 +-  
 arch/mips/sibyte/sb1250/irq.c                |    8 +-  
 arch/mips/sibyte/sb1250/smp.c                |    8 +-  
 arch/mips/sni/time.c                         |    2 +-  
 arch/parisc/Kconfig                          |    1 +   
 arch/parisc/include/asm/smp.h                |    2 -   
 arch/parisc/kernel/irq.c                     |    6 +-  
 arch/parisc/kernel/smp.c                     |   15 --  
 arch/powerpc/include/asm/topology.h          |   12 +-  
 arch/powerpc/kernel/irq.c                    |    2 +-  
 arch/powerpc/kernel/smp.c                    |    4 -   
 arch/powerpc/kernel/time.c                   |    2 +-  
 arch/powerpc/platforms/cell/spu_priv1_mmio.c |    6 +-  
 arch/powerpc/platforms/cell/spufs/sched.c    |    4 +-  
 arch/powerpc/platforms/pseries/xics.c        |    4 +-  
 arch/powerpc/sysdev/mpic.c                   |    4 +-  
 arch/powerpc/sysdev/mpic.h                   |    2 +-  
 arch/s390/Kconfig                            |    1 +   
 arch/s390/include/asm/topology.h             |    2 +   
 arch/s390/kernel/smp.c                       |    6 -   
 arch/s390/kernel/time.c                      |    2 +-  
 arch/s390/kernel/topology.c                  |    5 +   
 arch/sh/include/asm/smp.h                    |    2 +-  
 arch/sh/include/asm/topology.h               |    1 +   
 arch/sh/kernel/smp.c                         |   10 +-  
 arch/sh/kernel/timers/timer-broadcast.c      |    2 +-  
 arch/sh/kernel/timers/timer-tmu.c            |    2 +-  
 arch/sparc/include/asm/smp_32.h              |    2 -   
 arch/sparc/include/asm/topology_64.h         |   13 +-  
 arch/sparc/kernel/irq_64.c                   |   11 +-  
 arch/sparc/kernel/of_device_64.c             |    4 +-  
 arch/sparc/kernel/pci_msi.c                  |    4 +-  
 arch/sparc/kernel/smp_32.c                   |    6 +-  
 arch/sparc/kernel/smp_64.c                   |    4 -   
 arch/sparc/kernel/sparc_ksyms_32.c           |    4 -   
 arch/sparc/kernel/time_64.c                  |    2 +-  
 arch/um/kernel/smp.c                         |    7 -   
 arch/um/kernel/time.c                        |    2 +-  
 arch/x86/include/asm/pci.h                   |   10 +-  
 arch/x86/include/asm/topology.h              |   36 ++- 
 arch/x86/kernel/apic.c                       |    8 +-  
 arch/x86/kernel/cpu/intel_cacheinfo.c        |    4 +-  
 arch/x86/kernel/hpet.c                       |    8 +-  
 arch/x86/kernel/i8253.c                      |    2 +-  
 arch/x86/kernel/io_apic.c                    |   78 ++++----
 arch/x86/kernel/irq_32.c                     |    2 +-      
 arch/x86/kernel/irq_64.c                     |    2 +-      
 arch/x86/kernel/mfgpt_32.c                   |    2 +-      
 arch/x86/kernel/setup_percpu.c               |   10 +-      
 arch/x86/kernel/smpboot.c                    |   17 +-      
 arch/x86/kernel/vmiclock_32.c                |    2 +-      
 arch/x86/lguest/boot.c                       |    2 +-      
 arch/x86/mach-voyager/voyager_smp.c          |    7 -       
 arch/x86/xen/time.c                          |    2 +-      
 block/blk.h                                  |    4 +-      
 drivers/base/cpu.c                           |   46 ++++-   
 drivers/base/node.c                          |    4 +-      
 drivers/base/topology.c                      |    4 +-      
 drivers/clocksource/tcb_clksrc.c             |    2 +-      
 drivers/infiniband/hw/ehca/ehca_irq.c        |   17 +-      
 drivers/infiniband/hw/ipath/ipath_file_ops.c |    8 +-      
 drivers/parisc/iosapic.c                     |    7 +-      
 drivers/pci/pci-sysfs.c                      |    4 +-      
 drivers/pci/probe.c                          |    4 +-      
 drivers/pnp/pnpbios/bioscalls.c              |    2 +-      
 drivers/xen/events.c                         |    6 +-      
 fs/seq_file.c                                |    3 +-      
 include/asm-generic/topology.h               |   14 ++-     
 include/asm-m32r/smp.h                       |    2 -       
 include/asm-m68k/bitops.h                    |    5 +       
 include/linux/bitmap.h                       |   35 ++--    
 include/linux/bitops.h                       |   13 +-      
 include/linux/clockchips.h                   |    4 +-      
 include/linux/cpumask.h                      |  295 +++++++++++++-------------
 include/linux/interrupt.h                    |    6 +-                        
 include/linux/irq.h                          |    3 +-                        
 include/linux/rcuclassic.h                   |    4 +-                        
 include/linux/seq_file.h                     |    7 +-                        
 include/linux/smp.h                          |   18 +-                        
 include/linux/stop_machine.h                 |    6 +-                        
 include/linux/threads.h                      |   16 +-                        
 include/linux/tick.h                         |    4 +-                        
 init/Kconfig                                 |    9 +                         
 init/main.c                                  |   13 +-                        
 kernel/compat.c                              |   49 +++--                     
 kernel/cpu.c                                 |  145 +++++++++----             
 kernel/cpuset.c                              |    4 +-                        
 kernel/irq/chip.c                            |    2 +-                        
 kernel/irq/manage.c                          |   31 ++--                      
 kernel/irq/migration.c                       |   14 +-                        
 kernel/irq/proc.c                            |   59 ++++--                    
 kernel/kexec.c                               |    2 +-                        
 kernel/power/poweroff.c                      |    2 +-                        
 kernel/profile.c                             |   38 +++--                     
 kernel/rcuclassic.c                          |   32 ++--                      
 kernel/rcupreempt.c                          |   19 +-                        
 kernel/rcutorture.c                          |   27 ++-                       
 kernel/sched.c                               |   10 +-
 kernel/sched_stats.h                         |    2 +-
 kernel/smp.c                                 |  145 +++++--------
 kernel/softirq.c                             |    2 +-
 kernel/softlockup.c                          |   10 +-
 kernel/stop_machine.c                        |    8 +-
 kernel/taskstats.c                           |   41 +++--
 kernel/time/clockevents.c                    |    2 +
 kernel/time/clocksource.c                    |    9 +-
 kernel/time/tick-broadcast.c                 |  113 +++++-----
 kernel/time/tick-common.c                    |   18 +-
 kernel/trace/ring_buffer.c                   |   42 +++--
 kernel/trace/trace.c                         |   68 ++++---
 kernel/trace/trace.h                         |    2 +-
 kernel/trace/trace_boot.c                    |    2 +-
 kernel/trace/trace_functions_graph.c         |    2 +-
 kernel/trace/trace_hw_branches.c             |    6 +-
 kernel/trace/trace_power.c                   |    2 +-
 kernel/trace/trace_sysprof.c                 |   13 +-
 kernel/workqueue.c                           |   26 ++-
 lib/Kconfig                                  |   15 ++
 lib/Makefile                                 |    1 +
 lib/cpumask.c                                |   62 +++++-
 lib/find_last_bit.c                          |   45 ++++
 mm/pdflush.c                                 |   16 ++-
 mm/slab.c                                    |    2 +-
 mm/slub.c                                    |   20 +-
 mm/vmscan.c                                  |    4 +-
 mm/vmstat.c                                  |    4 +-
 security/selinux/selinuxfs.c                 |    2 +-
 203 files changed, 1393 insertions(+), 1056 deletions(-)
 create mode 100644 lib/find_last_bit.c

commit 98a79d6a50181ca1ecf7400eda01d5dc1bc0dbf0

Author: Rusty Russell <[email protected]>
Date:   Sat Dec 13 21:19:41 2008 +1030

cpumask: centralize cpu_online_map and cpu_possible_map

Impact: cleanup

Each SMP arch defines these themselves. Move them to a central location.

Twists:
1) Some archs (m32, parisc, s390) set possible_map to all 1, so we add a CONFIG_INIT_ALL_POSSIBLE for this rather than break them.

2) mips and sparc32 '#define cpu_possible_map phys_cpu_present_map'. Those archs simply have phys_cpu_present_map replaced everywhere.

3) Alpha defined cpu_possible_map to cpu_present_map; this is tricky so I just manipulate them both in sync.

4) IA64, cris and m32r have gratuitous 'extern cpumask_t cpu_possible_map' declarations.

commit 29c0177e6a4ac094302bed54a1d4bbb6b740a9ef

Author: Rusty Russell <[email protected]>
Date:   Sat Dec 13 21:20:25 2008 +1030

cpumask: change cpumask_scnprintf, cpumask_parse_user, cpulist_parse, and cpulist_scnprintf to take pointers.

Impact: change calling convention of existing cpumask APIs

Most cpumask functions started with cpus_: these have been replaced by cpumask_ ones which take struct cpumask pointers as expected.

These four functions don't have good replacement names; fortunately they're rarely used, so we just change them over.

commit 0de26520c7cabf36e1de090ea8092f011a6106ce

Author: Rusty Russell <[email protected]>
Date:   Sat Dec 13 21:20:26 2008 +1030

cpumask: make irq_set_affinity() take a const struct cpumask

Impact: change existing irq_chip API

Not much point with gentle transition here: the struct irq_chip's setaffinity method signature needs to change.

Fortunately, not widely used code, but hits a few architectures.

Note: In irq_select_affinity() I save a temporary in by mangling irq_desc[irq].affinity directly. Ingo, does this break anything?

(Folded in fix from KOSAKI Motohiro)

commit 320ab2b0b1e08e3805a3e1084a2f0eb1938d5d67

Author: Rusty Russell <[email protected]>
Date:   Sat Dec 13 21:20:26 2008 +1030

cpumask: convert struct clock_event_device to cpumask pointers.

Impact: change calling convention of existing clock_event APIs

struct clock_event_timer's cpumask field gets changed to take pointer, as does the ->broadcast function.

Another single-patch change. For safety, we BUG_ON() in clockevents_register_device() if it's not set.

commit aab46da0520af9c99b7802cebe4f14a81ff39415

Author: Rusty Russell <[email protected]>
Date:   Sat Dec 13 21:20:27 2008 +1030

cpumask: Add CONFIG_CPUMASK_OFFSTACK

Impact: Add config option to enable code in cpumask.h

Currently it can be set if DEBUG_PER_CPU_MAPS, or set specifically by an arch.

commit f0b848ce6fe9062d504d997e9e97fe0f87d57217

Author: Rusty Russell <[email protected]>
Date:   Sat Dec 13 21:20:27 2008 +1030

cpumask: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask

Impact: New APIs

The old node_to_cpumask/node_to_pcibus returned a cpumask_t: these return a pointer to a struct cpumask. Part of removing cpumasks from the stack.

This defines them in the generic non-NUMA case.

Cc Benjamin Herrenschmidt <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Tony Luck <[email protected]>
Cc Ralf Baechle <[email protected]>
Cc Richard Henderson <[email protected]>

commit 7be7585393d311866653564fbcd10a3232773c0b

Author: Rusty Russell <[email protected]>
Date:   Sat Dec 13 21:20:28 2008 +1030

cpumask: Use all NR_CPUS bits unless CONFIG_CPUMASK_OFFSTACK

Impact: futureproof as we convert more code to new APIs

The old cpumask operators treat all NR_CPUS bits as relevent, the new ones use nr_cpumask_bits. For large NR_CPUS and small nr_cpu_ids, this makes a difference.

However, mixing the two can cause problems with undefined bits. An arch which sets CONFIG_CPUMASK_OFFSTACK should have converted across to the new operators, so it's safe in that case.

(Thanks to Stephen Rothwell for bisecting the initial unused-bits bug, and Mike Travis for this solution).

commit 7b4967c532045a1983d6d4af5c69cc7c5109f62b

Author: Mike Travis <[email protected]>
Date:   Fri Dec 19 16:56:37 2008 +1030

cpumask: Add alloc_cpumask_var_node()

Impact: New API

This will be needed in x86 code to allocate the domain and old_domain cpumasks on the same node as where the containing irq_cfg struct is allocated.

(Also fixes double-dump_stack on rare CONFIG_DEBUG_PER_CPU_MAPS case)

commit ec26b805879c7e77865b39ee91b737985e80006d

Author: Mike Travis <[email protected]>
Date:   Fri Dec 19 16:56:52 2008 +1030

cpumask: documentation for cpumask_var_t

Impact: New kerneldoc comments

Additional documentation added to all the alloc_cpumask and free_cpumask functions.

commit e057d7aea9d8f2a46cd440d8bfb72245d4e72d79

Author: Mike Travis <[email protected]>
Date:   Mon Dec 15 20:26:48 2008 -0800

cpumask: add sysfs displays for configured and disabled cpu maps

Impact: add new sysfs files.

Add sysfs files "kernel_max" and "offline" to display the max CPU index allowed (NR_CPUS-1), and the map of cpus that are offline.

Cpus can be offlined via HOTPLUG, disabled by the BIOS ACPI tables, or if they exceed the number of cpus allowed by the NR_CPUS config option, or the "maxcpus=NUM" kernel start parameter.

The "possible_cpus=NUM" parameter can also extend the number of possible cpus allowed, in which case the cpus not present at startup will be in the offline state. (These cpus can be HOTPLUGGED ON after system startup [pending a follow-on patch to provide the capability via the /sys/devices/sys/cpu/cpuN/online mechanism to bring them online.])

By design, the "offlined cpus > possible cpus" display will always use the following formats:

* all possible cpus online: "x$" or "x-y$"
* some possible cpus offline: ".*,x$" or ".*,x-y$"

where:
x == number of possible cpus (nr_cpu_ids); and y == number of cpus >= NR_CPUS or maxcpus (if y > x).

One use of this feature is for distros to select (or configure) the appropriate kernel to install for the resident system.

Notes:
* cpus offlined <= possible cpus will be printed for all architectures.
* cpus offlined > possible cpus will only be printed for arches that set 'total_cpus' [X86 only in this patch].

Based on tip/cpus4096 + .../rusty/linux-2.6-for-ingo.git/master +
x86-only-patches sent 12/15.

commit d62720ade82c5e5b8f9585e5ed02c89573ebf111

Author: Mike Travis <[email protected]>
Date:   Wed Dec 17 14:14:30 2008 -0800

sysfs: add documentation to cputopology.txt for system cpumasks

Add information to cputopology.txt explaining the output of various system cpumask's.

commit 393d68fb9929817cde7ab31c82d66fcb28ad35fc

Author: Rusty Russell <[email protected]>
Date:   Fri Dec 26 22:23:38 2008 +1030

cpumask: x86: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask

Impact: New APIs

The old node_to_cpumask/node_to_pcibus returned a cpumask_t: these return a pointer to a struct cpumask. Part of removing cpumasks from the stack.

Also makes __pcibus_to_node take a const pointer.

commit 96d76a74870d5f11ce2abdd09a8dcdc401d714d1

Author: Rusty Russell <[email protected]>
Date:   Fri Dec 26 22:23:38 2008 +1030

cpumask: sparc: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask

Impact: New APIs

The old node_to_cpumask/node_to_pcibus returned a cpumask_t: these return a pointer to a struct cpumask. Part of removing cpumasks from the stack.

commit 7479a2939df4957ba794cce814379b6d10914bdc

Author: Rusty Russell <[email protected]>
Date:   Fri Dec 26 22:23:39 2008 +1030

cpumask: sh: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask

Impact: New APIs

The old node_to_cpumask/node_to_pcibus returned a cpumask_t: these return a pointer to a struct cpumask. Part of removing cpumasks from the stack.

commit 86c6f274f52c3e991d429869780945c0790e7b65

Author: Rusty Russell <[email protected]>
Date:   Fri Dec 26 22:23:39 2008 +1030

cpumask: powerpc: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask

Impact: New APIs

The old node_to_cpumask/node_to_pcibus returned a cpumask_t: these return a pointer to a struct cpumask. Part of removing cpumasks from the stack.

(Also replaces powerpc internal uses of node_to_cpumask).

Cc Benjamin Herrenschmidt <[email protected]>

commit fbb776c3ca4501d5a2821bf1e9bceefcaec7ae47

Author: Rusty Russell <[email protected]>
Date:   Fri Dec 26 22:23:40 2008 +1030

cpumask: IA64: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask

Impact: New APIs

The old node_to_cpumask/node_to_pcibus returned a cpumask_t: these return a pointer to a struct cpumask. Part of removing cpumasks from the stack.

We can also use the new for_each_cpu_and() to avoid a temporary cpumask, and a gratuitous test in sn_topology_show.

(Includes fix from KOSAKI Motohiro <[email protected]>)

Cc Tony Luck <[email protected]>
Cc KOSAKI Motohiro <[email protected]>

commit b4a2f916a8326065816a0743dd1b0ca2ffd18f5f

Author: Rusty Russell <[email protected]>
Date:   Fri Dec 26 22:23:40 2008 +1030

cpumask: Mips: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask

Impact: New APIs

The old node_to_cpumask/node_to_pcibus returned a cpumask_t: these return a pointer to a struct cpumask. Part of removing cpumasks from the stack.

commit 2258a5bb1064351b552aceaff29393967d694fa3

Author: Rusty Russell <[email protected]>
Date:   Fri Dec 26 22:23:41 2008 +1030

cpumask: alpha: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask

Impact: New APIs

The old node_to_cpumask/node_to_pcibus returned a cpumask_t: these return a pointer to a struct cpumask. Part of removing cpumasks from the stack.

I'm not sure the existing code even compiles, but new version is straightforward.

Cc Richard Henderson <[email protected]>

commit 030bb203e01db12e3f2866799f4f03a114d06349

Author: Rusty Russell <[email protected]>
Date:   Fri Dec 26 22:23:41 2008 +1030

cpumask: cpu_coregroup_mask(): x86

Impact: New API

Like cpu_coregroup_map, but returns a (const) pointer.

commit a0ae09b46a516f05ea76e3419ad43c46f52c1165

Author: Rusty Russell <[email protected]>
Date:   Fri Dec 26 22:23:42 2008 +1030

cpumask: cpu_coregroup_mask(): sparc

Like cpu_coregroup_map, but returns a (const) pointer.

Compile-tested on sparc64 (defconfig).

commit 9be3eec2c83848a1ca57ebad13c63c95d0df01e2

Author: Rusty Russell <[email protected]>
Date:   Fri Dec 26 22:23:42 2008 +1030

cpumask: cpu_coregroup_mask(): s390

Like cpu_coregroup_map, but returns a (const) pointer.

Compile-tested on s390 (defconfig).

commit be4d638c1597580ed2294d899d9f1a2cd10e462c

Author: Rusty Russell <[email protected]>
Date:   Fri Dec 26 22:23:43 2008 +1030

cpumask: Replace cpu_coregroup_map with cpu_coregroup_mask

cpu_coregroup_map returned a cpumask_t: it's going away.

(Note, the sched part of this patch won't apply meaningfully to the sched tree, but I'm posting it to show the goal).

commit 33edcf133ba93ecba2e4b6472e97b689895d805c
Merge: be4d638... 3c92ec8...
Author: Rusty Russell <[email protected]>
Date: Tue Dec 30 08:02:35 2008 +1030

Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6

commit 278d1ed65e25d80af7c3a112d707b3f70516ddb4

Author: Rusty Russell <[email protected]>
Date:   Tue Dec 30 09:05:12 2008 +1030

cpumask: make CONFIG_NR_CPUS always valid.

Impact: cleanup

Currently we have NR_CPUS, which is 1 on UP, and CONFIG_NR_CPUS on SMP. If we make CONFIG_NR_CPUS always valid (and always 1 on !SMP), we can skip the middleman.

This also allows us to find and check all the unaudited NR_CPUS usage as we prepare for v. large NR_CPUS.

To avoid breaking every arch, we cheat and do this for the moment in the header if the arch doesn't.

commit 4b0bc0bca83f3fb7cf920e2ec80684c15d2269c0

Author: Rusty Russell <[email protected]>
Date:   Tue Dec 30 09:05:13 2008 +1030

bitmap: test for constant as well as small size for inline versions

Impact: reduce text size

bitmap_zero et al have a fastpath for nbits <= BITS_PER_LONG, but this should really only apply where the nbits is known at compile time.

This only saves about 1200 bytes on an allyesconfig kernel, but with cpumasks going variable that number will increase.

text data bss dec hex filename
35327852 5035607 6782976 47146435 2cf65c3 vmlinux-before
35326640 5035607 6782976 47145223 2cf6107 vmlinux-after

commit cb78a0ce69fad2026825f957e24e2d9cda1ec9f1

Author: Rusty Russell <[email protected]>
Date:   Tue Dec 30 09:05:14 2008 +1030

bitmap: fix seq_bitmap and seq_cpumask to take const pointer

Impact: cleanup

seq_bitmap just calls bitmap_scnprintf on the bits: that arg can be const. Similarly, seq_cpumask just calls seq_bitmap.

commit b3199c025d1646e25e7d1d640dd605db251dccf8

Author: Rusty Russell <[email protected]>
Date:   Tue Dec 30 09:05:14 2008 +1030

cpumask: switch over to cpu_online/possible/active/present_mask: core

Impact: cleanup

This implements the obsolescent cpu_online_map in terms of cpu_online_mask, rather than the other way around. Same for the other maps.

The documentation comments are also updated to refer to _mask rather than _map.

commit ae7a47e72e1a0b5e2b46d1596bc2c22942a73023

Author: Rusty Russell <[email protected]>
Date:   Tue Dec 30 09:05:15 2008 +1030

cpumask: make cpumask.h eat its own dogfood.

Changes:
1) cpumask_t to struct cpumask,
2) cpus_weight_nr to cpumask_weight,
3) cpu_isset to cpumask_test_cpu,
4) ->bits to cpumask_bits()
5) cpu_*_map to cpu_*_mask.
6) for_each_cpu_mask_nr to for_each_cpu

commit 3fa41520696fec2815e2d88fbcccdda77ba4d693

Author: Rusty Russell <[email protected]>
Date:   Tue Dec 30 09:05:16 2008 +1030

cpumask: make set_cpu_*/init_cpu_* out-of-line

They're only for use in boot/cpu hotplug code anyway, and this avoids the use of deprecated cpu_*_map.

Stephen Rothwell points out that gcc 4.2.4 (on powerpc at least) didn't like the cast away of const anyway:

include/linux/cpumask.h: In function 'set_cpu_possible':
include/linux/cpumask.h:1052: warning: passing argument 2 of 'cpumask_set_cpu' discards qualifiers from pointer target type

So this kills two birds with one stone.

commit 54b11e6d57a10aa9d0009efd93873e17bffd5d30

Author: Rusty Russell <[email protected]>
Date:   Tue Dec 30 09:05:16 2008 +1030

cpumask: smp_call_function_many()

Impact: Implementation change to remove cpumask_t from stack.

Actually change smp_call_function_mask() to smp_call_function_many(). We avoid cpumasks on the stack in this version.

(S390 has its own version, but that's going away apparently).

We have to do some dancing to figure out if 0 or 1 other cpus are in the mask supplied and the online mask without allocating a tmp cpumask. It's still fairly cheap.

We allocate the cpumask at the end of the call_function_data
structure: if allocation fails we fallback to smp_call_function_single rather than using the baroque quiescing code (which needs a cpumask on stack).

(Thanks to Hiroshi Shimamoto for spotting several bugs in previous versions!)

commit ce47d974f71af26d00832e83a43ac79bec272d99

Author: Rusty Russell <[email protected]>
Date:   Tue Dec 30 09:05:17 2008 +1030

cpumask: arch_send_call_function_ipi_mask: core

Impact: new API to reduce stack usage

We're weaning the core code off handing cpumask's around on-stack. This introduces arch_send_call_function_ipi_mask().

commit 259c4ddd00237e5072921afa15a900839643fd98

Author: Rusty Russell <[email protected]>
Date:   Tue Dec 30 09:05:17 2008 +1030

cpumask: use for_each_online_cpu() in drivers/infiniband/hw/ehca/ehca_irq.c

Impact: cleanup

In future, accessing cpu numbers beyond nr_cpu_ids (the runtime limit) will be undefined. We can avoid future problems by using for_each_online_cpu() here.

commit b29179c3d32021d79c11ece7199a1da41d31b1b7

Author: Rusty Russell <[email protected]>
Date:   Tue Dec 30 09:05:18 2008 +1030

cpumask: use new cpumask API in drivers/infiniband/hw/ehca

Impact: cleanup

We're moving from handing around cpumask_t's to handing around struct cpumask *'s. cpus_*, cpumask_t and cpu_*_map are deprecated: convert to cpumask_*, cpu_*_mask.

commit cbe31f02f5b5536f17dd978118e25052af528071

Author: Rusty Russell <[email protected]>
Date:   Tue Dec 30 09:05:18 2008 +1030

cpumask: use new cpumask API in drivers/infiniband/hw/ipath

Impact: cleanup

We're moving from handing around cpumask_t's to handing around struct cpumask *'s. cpus_*, cpumask_t and cpu_*_map are deprecated: convert to cpumask_*, cpu_*_mask.

commit e12f0102ac81d660c9f801d0a0e10ccf4537a9de

Author: Rusty Russell <[email protected]>
Date:   Tue Dec 30 09:05:19 2008 +1030

cpumask: Use nr_cpu_ids in seq_cpumask

Impact: cleanup, futureproof

nr_cpu_ids is the (badly named) runtime limit on possible CPU numbers; ie. the variable version of NR_CPUS.

With the new cpumask operators, only bits less than this are defined. So we should use it everywhere, rather than NR_CPUS. Eventually this will make it possible to allocate cpumasks of the minimal length at runtime.

commit 2ca1a615835d9f4990f42102ab1f2ef434e7e89c
Merge: e12f010... 6a94cb7...
Author: Rusty Russell <[email protected]>
Date: Wed Dec 31 23:05:57 2008 +1030

Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6

Conflicts:

arch/x86/kernel/io_apic.c

commit f320786063a9d1f885d2cf34ab44aa69c1d88f43

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:13 2009 +1030

cpumask: Remove IA64 definition of total_cpus now it's in core code

Impact: fix IA64 compile

Fortunately, they have exactly the same semantics.

commit e9690a6e4b1615cb0102e425e04b7ce29e7858e2

Author: Li Zefan <[email protected]>
Date:   Wed Dec 31 16:45:50 2008 +0800

cpumask: fix bogus kernel-doc

Impact: fix kernel-doc

alloc_bootmem_cpumask_var() returns avoid.

commit 6aaa8ce523c7ce954b81b8c0b3e32c8be599af8d

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:14 2009 +1030

percpu: fix percpu accessors to potentially !cpu_possible() cpus: pnpbios

Impact: CPU iterator bugfixes

Percpu areas are only allocated for possible cpus. In general, you shouldn't access random cpu's percpu areas.

commit 9e2f913df70b378379a358a44e7d286f7b765e8e

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:14 2009 +1030

percpu: fix percpu accessors to potentially !cpu_possible() cpus: m32r

Impact: CPU iterator bugfixes

Percpu areas are only allocated for possible cpus. In general, you shouldn't access random cpu's percpu areas.

commit 4f4b6c1a94a8735bbdc030a2911cf395495645b6

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:15 2009 +1030

cpumask: prepare for iterators to only go to nr_cpu_ids/nr_cpumask_bits.: core

Impact: cleanup

In future, all cpumask ops will only be valid (in general) for bit numbers < nr_cpu_ids. So use that instead of NR_CPUS in iterators and other comparisons.

This is always safe: no cpu number can be >= nr_cpu_ids, and nr_cpu_ids is initialized to NR_CPUS at boot.

commit 915441b601e6662e79f6c958e7be307967a96977

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:15 2009 +1030

cpumask: Use accessors code in core

Impact: use new API

cpu_*_map are going away in favour of cpu_*_mask, but const pointers. So we have accessors where we really do want to frob them. Archs will also need the (trivial) conversion before we can finally remove cpu_*_map.

commit 165ac433fa3f01ba99b29972f3adc283d03b0f17

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:16 2009 +1030

parisc: remove gratuitous cpu_online_map declaration.

This is defined in linux/cpumask.h (included in this file already), and this is now defined differently.

commit 96b8d4c19d797200b973caab57ca842531184c13

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:16 2009 +1030

avr32: define __fls

Like fls, but can't be handed 0 and returns the bit number.

(I broke this arch in linux-next by using __fls in generic code).

commit ccec25ff69d5f48c7a088c16fe2dc7e11d9e87fe

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:17 2009 +1030

blackfin: define __fls

Like fls, but can't be handed 0 and returns the bit number.

(I broke this arch in linux-next by using __fls in generic code).

commit 434ae514c23047db87a8bbf39cebc9e1767aea44

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:18 2009 +1030

m68k: define __fls

Like fls, but can't be handed 0 and returns the bit number.

(I broke this arch in linux-next by using __fls in generic code).

commit 0db5d3d2f58804edb394e8008c7d9744110338a2

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:18 2009 +1030

m68knommu: define __fls

Like fls, but can't be handed 0 and returns the bit number.

(I broke this arch in linux-next by using __fls in generic code).

commit ab53d472e785e51fdfc08fc1d66252c1153e6c0f

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:19 2009 +1030

bitmap: find_last_bit()

Impact: New API

As the name suggests. For the moment everyone uses the generic one.

commit e0c0ba736547e81c4f986ce192307c549d214167

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:19 2009 +1030

cpumask: Use find_last_bit()

Impact: cleanup

There's one obvious place to use it: to find the highest possible cpu.

commit 78fd744f827586615da5b387fa9f0af1888601b6

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:20 2009 +1030

cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): sparc

Impact: New API

The old topology_core_siblings() and topology_thread_siblings() return a cpumask_t; these new ones return a (const) struct cpumask *.

commit 2bb23a63f22f0e2d91fee93ff5ca9c29e180b146

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:20 2009 +1030

cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): s390

Impact: New API

The old topology_core_siblings() and topology_thread_siblings() return a cpumask_t; these new ones return a (const) struct cpumask *.

commit 9150641dd17fe9e213ab3391c8ebfc228daa2d9d

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:21 2009 +1030

cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): powerpc

Impact: New API

The old topology_core_siblings() and topology_thread_siblings() return a cpumask_t; these new ones return a (const) struct cpumask *.

commit 333af15341b2f6cd813c054e1b441d7b6d8e9318

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:21 2009 +1030

cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): ia64

Impact: New API

The old topology_core_siblings() and topology_thread_siblings() return a cpumask_t; these new ones return a (const) struct cpumask *.

commit 9e01c1b74c9531e301c900edaa92a99fcb7738f2

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:22 2009 +1030

cpumask: convert kernel trace functions

Impact: Reduce future memory usage, use new cpumask API.

(Eventually, cpumask_var_t will be allocated based on nr_cpu_ids, not NR_CPUS).

Convert kernel trace functions to use struct cpumask API:
1) Use cpumask_copy/cpumask_test_cpu/for_each_cpu.
2) Use cpumask_var_t and alloc_cpumask_var/free_cpumask_var everywhere.
3) Use on_each_cpu instead of playing with current->cpus_allowed.

commit 4462344ee9ea9224d026801b877887f2f39774a3

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:23 2009 +1030

cpumask: convert kernel trace functions further

Impact: Reduce future memory usage, use new cpumask API.

Since the last patch was created and acked, more old cpumask users slipped into kernel/trace.

Mostly trivial conversions, except struct trace_iterator's "started" member becomes a cpumask_var_t.

commit f1fc057c79cb2d27602fb3ad08a031f13459ef27

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:23 2009 +1030

cpumask: remove any_online_cpu() users: kernel/

Impact: Remove obsolete API usage

any_online_cpu() is a good name, but it takes a cpumask_t, not a pointer.

There are several places where any_online_cpu() doesn't really want a mask arg at all. Replace all callers with cpumask_any() and cpumask_any_and().

commit 3e597945384dee1457240158eb81e3afb90b68c2

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:24 2009 +1030

cpumask: remove any_online_cpu() users: mm/

Impact: Remove obsolete API usage

any_online_cpu() is a good name, but it takes a cpumask_t, not a pointer.

There are several places where any_online_cpu() doesn't really want a mask arg at all. Replace all callers with cpumask_any() and cpumask_any_and().

commit a45185d2d7108b01b90b9e0293377be4d6346dde

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:24 2009 +1030

cpumask: convert kernel/compat.c

Impact: Reduce stack usage, use new cpumask API.

Straightforward conversion; cpumasks' size is given by cpumask_size() (now a variable rather than fixed) and on-stack cpu masks use cpumask_var_t.

commit e7577c50f2fb2d1c167e2c04a4b4c2cc042acb82

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:25 2009 +1030

cpumask: convert kernel/workqueue.c

Impact: Reduce memory usage, use new cpumask API.

cpu_populated_map becomes a cpumask_var_t, and cpu_singlethread_map is simply a cpumask pointer: it's simply the cpumask containing the first possible CPU anyway.

commit 6b954823c24f04ed026a8517f6bab5abda279db8

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:25 2009 +1030

cpumask: convert kernel time functions

Impact: Use new APIs

Convert kernel/time functions to use struct cpumask *.

Note the ugly bitmap declarations in tick-broadcast.c. These should be cpumask_var_t, but there was no obvious initialization function to put the alloc_cpumask_var() calls in. This was safe.

(Eventually 'struct cpumask' will be undefined for CONFIG_CPUMASK_OFFSTACK, so we use a bitmap here to show we really mean it).

commit d036e67b40f52bdd95392390108defbac7e53837

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:26 2009 +1030

cpumask: convert kernel/irq

Impact: Reduce stack usage, use new cpumask API. ALPHA mod!

Main change is that irq_default_affinity becomes a cpumask_var_t, so treat it as a pointer (this effects alpha).

commit bd232f97b30f6bb630efa136a777647545db3039

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:26 2009 +1030

cpumask: convert RCU implementations

Impact: use new cpumask API.

rcu_ctrlblk contains a cpumask, and it's highly optimized so I don't want a cpumask_var_t (ie. a pointer) for the CONFIG_CPUMASK_OFFSTACK case. It could use a dangling bitmap, and be allocated in __rcu_init to save memory, but for the moment we use a bitmap.

(Eventually 'struct cpumask' will be undefined for CONFIG_CPUMASK_OFFSTACK, so we use a bitmap here to show we really mean it).

We remove on-stack cpumasks, using cpumask_var_t for rcu_torture_shuffle_tasks() and for_each_cpu_and in force_quiescent_state().

commit c309b917cab55799ea489d7b5f1b77025d9f8462

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:27 2009 +1030

cpumask: convert kernel/profile.c

Impact: Reduce kernel memory usage, use new cpumask API.

Avoid a static cpumask_t for prof_cpu_mask, and an on-stack cpumask_t in prof_cpu_mask_write_proc. Both become cpumask_var_t.

prof_cpu_mask is only allocated when profiling is on, but the NULL checks are optimized out by gcc for the !CPUMASK_OFFSTACK case.

Also removed some strange and unnecessary casts.

commit e0b582ec56f1a1d8b30ebf340a7b91fb09f26c8c

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:28 2009 +1030

cpumask: convert kernel/cpu.c

Impact: Reduce kernel stack and memory usage, use new cpumask API.

Use cpumask_var_t for take_cpu_down() stack var, and frozen_cpus.

Note that notify_cpu_starting() can be called before core_initcall allocates frozen_cpus, but the NULL check is optimized out by gcc for the CONFIG_CPUMASK_OFFSTACK=n case.

commit 41c7bb9588904eb060a95bcad47bd3804a1ece25

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:28 2009 +1030

cpumask: convert rest of files in kernel/

Impact: Reduce stack usage, use new cpumask API.

Mainly changing cpumask_t to 'struct cpumask' and similar simple API conversion. Two conversions worth mentioning:

1) we use cpumask_any_but to avoid a temporary in kernel/softlockup.c,
2) Use cpumask_var_t in taskstats_user_cmd().

commit 174596a0b9f21e8844d70566a6bb29bf48a87750

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:29 2009 +1030

cpumask: convert mm/

Impact: Use new API

Convert kernel mm functions to use struct cpumask.

We skip include/linux/percpu.h and mm/allocpercpu.c, which are in flux.

commit 5db0e1e9e0f30f160b832a0b5cd1131954bf4f6e

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:29 2009 +1030

cpumask: replace for_each_cpu_mask_nr with for_each_cpu in kernel/time/

Impact: cleanup

Simple replacement, now the _nr is redundant.

commit 2a53008033189ed09bfe241c6b33811ba4ce980d

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:30 2009 +1030

cpumask: zero extra bits in alloc_cpumask_var_node

Impact: extra safety checks during transition

When CONFIG_CPUMASKS_OFFSTACK is set, the new cpumask_ operators only use bits up to nr_cpu_ids, not NR_CPUS. Using the old cpus_ operators on these masks can mean accessing undefined bits.

After some discussion, Mike and I decided to err on the side of caution; we zero the "undefined" bits in alloc_cpumask_var_node() until all the old cpumask functions are removed.

commit 8c384cdee3e04d6194a2c2b192b624754f990835

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:30 2009 +1030

cpumask: CONFIG_DISABLE_OBSOLETE_CPUMASK_FUNCTIONS

Impact: new debug CONFIG options

This helps find unconverted code. It currently breaks compile horribly, but we never wanted a flag day so that's expected.

#1 lore raw
Quoted 4 lines · Rusty Russell

OK, this is the bulk of the conversion to the new cpumask operators. The x86-specific parts (the most aggressive large-NR_CPUS arch) are going via Ingo's tree.

This gets lots of conflicts for me. Some of them look simple enough, but not all. io_apic.c gets lots of nasty conflicts, and it _looks_ like I should just pick the version of the file that I already have (because the only thing that comes in from that is yet another merge commit), but kernel/sched.c also gets conflicts in areas with FIXME's etc.

Rusty, Ingo, can you work this out? I pushed out my current tree.

Linus

#2 lore raw
Quoted 6 lines · Linus Torvalds
Quoted 4 lines · Rusty Russell

OK, this is the bulk of the conversion to the new cpumask operators. The x86-specific parts (the most aggressive large-NR_CPUS arch) are going via Ingo's tree.

Quoted 7 lines

This gets lots of conflicts for me. Some of them look simple enough, but not all. io_apic.c gets lots of nasty conflicts, and it _looks_ like I should just pick the version of the file that I already have (because the only thing that comes in from that is yet another merge commit), but kernel/sched.c also gets conflicts in areas with FIXME's etc.

Rusty, Ingo, can you work this out? I pushed out my current tree.

yes, we have those conflicts all resolved already in the second phase of the tip/cpus4096 changes: Mike did all those difficult conflict resolutions over the hollidays and i pulled it yesterday.

The end result looks nice as a tree but it is not fully cooked yet: since i pulled yesterday i found a couple of build and runtime test failures with Rusty's latest cpumask tree:

- architectures that have no __fls (8 out of 21) fail to build:

     arch/cris
     arch/frv
     arch/h8300
     arch/m32r
     arch/m68k
     arch/mn10300
     arch/xtensa

- there's a new circular locking lockdep splat in CPU hotplug tests when the var-cpumask code is enabled. (needs a handful of default-off options enabled in the .config)

So i didnt want to push the second phase to you until those known bugs are sorted out - i think we need some more time for that - a day or two at most.

Linus, would you like to pull that, despite the pending regressions? Can send a pull request right away.

Rusty, would it be fine with you if we did all the remaining bits via tip/cpus4096? It's your tree and your bits and we wanted to send our remaining bits after your tree went to Linus but the conflict resolutions from Mike are valuable so i think we should reconsider the ordering.

All the commits you sent to Linus in this pull request are already included in tip/cpus4096, the conflicts that Linus hit are all non-trivial and Mike resolved them correctly and it merges cleanly with Linus's latest tree:

earth4:~/tip> git pull
git://git.kernel.org/pub/scm/linux/kernel/git/rusty/linux-2.6-cpumask.git master From git://git.kernel.org/pub/scm/linux/kernel/git/rusty/linux-2.6-cpumask
* branch master -> FETCH_HEAD
Already up-to-date.

The pending diff is:

  108 files changed, 1442 insertions(+), 979 deletions(-)

which is pretty OK and straightforward.

With that Rusty and Mike has done 99% of the cpumask conversions, so the most difficult phase of the conversion should be dealt with in .29. Cool stuff.

Ingo

#3 lore raw
Quoted 3 lines · Ingo Molnar

Linus, would you like to pull that, despite the pending regressions? Can send a pull request right away.

No, if there are known pending regressions, please get those fixed first.

Linus

#45 lore raw
In reply to Linus Torvalds
Quoted 7 lines · On Saturday 03 January 2009 06:36:33 Linus Torvalds wrote:
Quoted 4 lines · Rusty Russell

OK, this is the bulk of the conversion to the new cpumask operators. The x86-specific parts (the most aggressive large-NR_CPUS arch) are going via Ingo's tree.

Quoted 7 lines

This gets lots of conflicts for me. Some of them look simple enough, but not all. io_apic.c gets lots of nasty conflicts, and it _looks_ like I should just pick the version of the file that I already have (because the only thing that comes in from that is yet another merge commit), but kernel/sched.c also gets conflicts in areas with FIXME's etc.

Rusty, Ingo, can you work this out? I pushed out my current tree.

Yes, some went via Ingo's tree and there are some known overlaps.

I'll clean it up and resend pull req this weekend.

Thanks,
Rusty.

#26 lore raw
In reply to Ingo Molnar
Quoted 9 lines · On Saturday 03 January 2009 07:08:40 Ingo Molnar wrote:

- architectures that have no __fls (8 out of 21) fail to build:

     arch/cris
     arch/frv
     arch/h8300
     arch/m32r
     arch/m68k
     arch/mn10300
     arch/xtensa

Fixes pushed, m68k should be OK tho; is this actual compile test? You have to look in include/asm-m68k to see __fls.

Quoted 4 lines

Rusty, would it be fine with you if we did all the remaining bits via tip/cpus4096? It's your tree and your bits and we wanted to send our remaining bits after your tree went to Linus but the conflict resolutions from Mike are valuable so i think we should reconsider the ordering.

Yeah, no reason for us to do the merge twice. As long as it ends upstream, I'm a happy camper.

Thanks,
Rusty.

#27 lore raw
Quoted 11 lines · Rusty Russell
Quoted 9 lines · On Saturday 03 January 2009 07:08:40 Ingo Molnar wrote:

- architectures that have no __fls (8 out of 21) fail to build:

     arch/cris
     arch/frv
     arch/h8300
     arch/m32r
     arch/m68k
     arch/mn10300
     arch/xtensa
Quoted 2 lines

Fixes pushed, m68k should be OK tho; is this actual compile test? You have to look in include/asm-m68k to see __fls.

yeah, i stopped the tests after the first two build failures - the rest is a grep result from arch/*/, that's why include/asm-m68k/ was left out.

Quoted 8 lines
Quoted 5 lines

Rusty, would it be fine with you if we did all the remaining bits via tip/cpus4096? It's your tree and your bits and we wanted to send our remaining bits after your tree went to Linus but the conflict resolutions from Mike are valuable so i think we should reconsider the ordering.

Yeah, no reason for us to do the merge twice. As long as it ends upstream, I'm a happy camper.

great - lets do it that way then. I have pulled your fixes into the cpus4096 tree:

5ece5c5: xtensa: define __fls
5c134da: mn10300: define __fls
16a2062: m32r: define __fls
9ddabc2: h8300: define __fls
ee38e51: frv: define __fls
0999769: cris: define __fls

Once we have figured out the CPU-hotplug lockdep splat (possibly due to Mike's changes not yours) i'll send it to Linus. Thanks,

Ingo

#28 lore raw

PATCH 1/1 ia64: cpumask fix for is_affinity_mask_valid()

Quoted 31 lines · Ingo Molnar
Quoted 11 lines · Rusty Russell
Quoted 9 lines · On Saturday 03 January 2009 07:08:40 Ingo Molnar wrote:

- architectures that have no __fls (8 out of 21) fail to build:

     arch/cris
     arch/frv
     arch/h8300
     arch/m32r
     arch/m68k
     arch/mn10300
     arch/xtensa
Quoted 2 lines

Fixes pushed, m68k should be OK tho; is this actual compile test? You have to look in include/asm-m68k to see __fls.

yeah, i stopped the tests after the first two build failures - the rest is a grep result from arch/*/, that's why include/asm-m68k/ was left out.

Quoted 8 lines
Quoted 5 lines

Rusty, would it be fine with you if we did all the remaining bits via tip/cpus4096? It's your tree and your bits and we wanted to send our remaining bits after your tree went to Linus but the conflict resolutions from Mike are valuable so i think we should reconsider the ordering.

Yeah, no reason for us to do the merge twice. As long as it ends upstream, I'm a happy camper.

great - lets do it that way then. I have pulled your fixes into the cpus4096 tree:

Quoted 6 lines

5ece5c5: xtensa: define __fls
5c134da: mn10300: define __fls
16a2062: m32r: define __fls
9ddabc2: h8300: define __fls
ee38e51: frv: define __fls
0999769: cris: define __fls

ok, these architectures build fine now.

There's one new build failure due to cpumask changes: ia64. I have fixed it via the patch below. (if it looks good to you i'll carry it via tip/cpus4096, ok?)

Breakage came via this commit i think:

d036e67: cpumask: convert kernel/irq

[ia64 is the only architecture that re-defines default_affinity_write().]

Ingo

----------------------->

Quoted 1 line

From f1faf35ad450c3d648138b74b9f58b283dc19c5c Mon Sep 17 00:00:00 2001

From: Ingo Molnar <[email protected]>
Date: Sat, 3 Jan 2009 12:50:46 +0100
Subject: [PATCH] ia64: cpumask fix for is_affinity_mask_valid()

Impact: build fix on ia64

ia64's default_affinity_write() still had old cpumask_t usage:

/home/mingo/tip/kernel/irq/proc.c: In function `default_affinity_write':
/home/mingo/tip/kernel/irq/proc.c:114: error: incompatible type for argument 1 of `is_affinity_mask_valid' make[3]: *** [kernel/irq/proc.o] Error 1
make[3]: *** Waiting for unfinished jobs....

update it to cpumask_var_t.

---
 arch/ia64/include/asm/irq.h |    2 +-
 arch/ia64/kernel/irq.c      |    4 ++--
 kernel/irq/proc.c           |    2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)
arch/ia64/include/asm/irq.h +1 -1
index 3627116..36429a5 100644
@@ -27,7 +27,7 @@ irq_canonicalize (int irq)
 }
 
 extern void set_irq_affinity_info (unsigned int irq, int dest, int redir);
-bool is_affinity_mask_valid(cpumask_t cpumask);
+bool is_affinity_mask_valid(cpumask_var_t cpumask);
 
 #define is_affinity_mask_valid is_affinity_mask_valid
 
arch/ia64/kernel/irq.c +2 -2
index 0b6db53..95ff16c 100644
@@ -112,11 +112,11 @@ void set_irq_affinity_info (unsigned int irq, int hwid, int redir)
 	}
 }
 
-bool is_affinity_mask_valid(cpumask_t cpumask)
+bool is_affinity_mask_valid(cpumask_var_t cpumask)
 {
 	if (ia64_platform_is("sn2")) {
 		/* Only allow one CPU to be specified in the smp_affinity mask */
-		if (cpus_weight(cpumask) != 1)
+		if (cpumask_weight(cpumask) != 1)
 			return false;
 	}
 	return true;
kernel/irq/proc.c +1 -1
index 2abd3a7..aae3f74 100644
@@ -54,7 +54,7 @@ static ssize_t irq_affinity_proc_write(struct file *file,
 	if (err)
 		goto free_cpumask;
 
-	if (!is_affinity_mask_valid(*new_value)) {
+	if (!is_affinity_mask_valid(new_value)) {
 		err = -EINVAL;
 		goto free_cpumask;
 	}
#29 lore raw

PATCH 1/1 cpumask: convert RCU implementations, fix

there's a build warning regression as well on CONFIG_RCU_CLASSIC:

kernel/rcuclassic.c: In function ‘rcu_start_batch’:
kernel/rcuclassic.c:397: warning: passing argument 1 of ‘cpumask_andnot’ from incompatible pointer type

caused by:

bd232f9: cpumask: convert RCU implementations

fixed via the patch below. (i'll carry this in cpus4096 too, ok?)

Ingo

---------------------->

Quoted 1 line

From 9caab32c0d69e569e33caa03845cec73ae701f32 Mon Sep 17 00:00:00 2001

From: Ingo Molnar <[email protected]>
Date: Sat, 3 Jan 2009 13:16:09 +0100
Subject: [PATCH] cpumask: convert RCU implementations, fix
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

Impact: cleanup

This warning:

kernel/rcuclassic.c: In function ‘rcu_start_batch’:
kernel/rcuclassic.c:397: warning: passing argument 1 of ‘cpumask_andnot’ from incompatible pointer type

triggers because one usage site of rcp->cpumask was not converted to to_cpumask(rcp->cpumask). There's no ill effects of this bug.

---
 kernel/rcuclassic.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
kernel/rcuclassic.c +2 -1
index 6ec495f..490934f 100644
@@ -394,7 +394,8 @@ static void rcu_start_batch(struct rcu_ctrlblk *rcp)
 		 * unnecessarily.
 		 */
 		smp_mb();
-		cpumask_andnot(&rcp->cpumask, cpu_online_mask, nohz_cpu_mask);
+		cpumask_andnot(to_cpumask(rcp->cpumask),
+			       cpu_online_mask, nohz_cpu_mask);
 
 		rcp->signaled = 0;
 	}
#33 lore raw
In reply to Ingo Molnar
Quoted 36 lines · Ingo Molnar
Quoted 10 lines · Rusty Russell
Quoted 9 lines · On Saturday 03 January 2009 07:08:40 Ingo Molnar wrote:

- architectures that have no __fls (8 out of 21) fail to build:

     arch/cris
     arch/frv
     arch/h8300
     arch/m32r
     arch/m68k
     arch/mn10300
     arch/xtensa
Quoted 2 lines

Fixes pushed, m68k should be OK tho; is this actual compile test? You have to look in include/asm-m68k to see __fls.

yeah, i stopped the tests after the first two build failures - the rest is a grep result from arch/*/, that's why include/asm-m68k/ was left out.

Quoted 7 lines
Quoted 5 lines

Rusty, would it be fine with you if we did all the remaining bits via tip/cpus4096? It's your tree and your bits and we wanted to send our remaining bits after your tree went to Linus but the conflict resolutions from Mike are valuable so i think we should reconsider the ordering.

Yeah, no reason for us to do the merge twice. As long as it ends upstream, I'm a happy camper.

great - lets do it that way then. I have pulled your fixes into the cpus4096 tree:

5ece5c5: xtensa: define __fls
5c134da: mn10300: define __fls
16a2062: m32r: define __fls
9ddabc2: h8300: define __fls
ee38e51: frv: define __fls
0999769: cris: define __fls

Quoted 4 lines

Once we have figured out the CPU-hotplug lockdep splat (possibly due to Mike's changes not yours) i'll send it to Linus. Thanks,

Ingo

Thanks! Am working on that now.

Mike

#34 lore raw
Quoted 6 lines · Mike Travis
Quoted 4 lines

Once we have figured out the CPU-hotplug lockdep splat (possibly due to Mike's changes not yours) i'll send it to Linus. Thanks,

Ingo

Thanks! Am working on that now.

do you suspect any of the commits? I'm bisecting it right now but if you have bisected it already i wont repeat it.

Ingo

#35 lore raw
Quoted 5 lines · Ingo Molnar
Quoted 3 lines · Mike Travis
Quoted 3 lines

Once we have figured out the CPU-hotplug lockdep splat (possibly due to Mike's changes not yours) i'll send it to Linus. Thanks,

Quoted 7 lines
Quoted 2 lines
Quoted 1 line

Ingo

Thanks! Am working on that now.

do you suspect any of the commits? I'm bisecting it right now but if you have bisected it already i wont repeat it.

Ingo

I haven't done that yet, I'm just now getting your config to work pre-patches.

But yes, I suspect one of the changes to use "work_on_cpu" -- this usually causes the 2nd call to get_online_cpus().

Thanks,
Mike

#36 lore raw
Quoted 12 lines · Mike Travis
Quoted 11 lines · Ingo Molnar
Quoted 5 lines · Mike Travis
Quoted 4 lines

Once we have figured out the CPU-hotplug lockdep splat (possibly due to Mike's changes not yours) i'll send it to Linus. Thanks,

Ingo

Thanks! Am working on that now.

do you suspect any of the commits? I'm bisecting it right now but if you have bisected it already i wont repeat it.

Quoted 6 lines
Quoted 1 line

Ingo

I haven't done that yet, I'm just now getting your config to work pre-patches.

But yes, I suspect one of the changes to use "work_on_cpu" -- this usually causes the 2nd call to get_online_cpus().

i suspect it's:

| commit 2d22bd5e74519854458ad372a89006e65f45e628
| Author: Mike Travis <[email protected]>
| Date:   Wed Dec 31 18:08:46 2008 -0800
|
|     x86: cleanup remaining cpumask_t code in microcode_core.c

as the microcode is loaded during CPU onlining.

Ingo

#37 lore raw
Quoted 27 lines · Ingo Molnar
Quoted 8 lines · Mike Travis
Quoted 7 lines · Ingo Molnar
Quoted 5 lines · Mike Travis
Quoted 4 lines

Once we have figured out the CPU-hotplug lockdep splat (possibly due to Mike's changes not yours) i'll send it to Linus. Thanks,

Ingo

Thanks! Am working on that now.

Quoted 8 lines
Quoted 4 lines

do you suspect any of the commits? I'm bisecting it right now but if you have bisected it already i wont repeat it.

Ingo

I haven't done that yet, I'm just now getting your config to work pre-patches.

But yes, I suspect one of the changes to use "work_on_cpu" -- this usually causes the 2nd call to get_online_cpus().

i suspect it's:

| commit 2d22bd5e74519854458ad372a89006e65f45e628
| Author: Mike Travis <[email protected]>
| Date:   Wed Dec 31 18:08:46 2008 -0800
|
|     x86: cleanup remaining cpumask_t code in microcode_core.c
Quoted 3 lines

as the microcode is loaded during CPU onlining.

Ingo

I think you're right. You need the coincidence of both writing to the /sys/devices/system/cpu/cpuX/online file along with the cpu coming up quickly enough to attempt to load the microcode before the write has returned. (Or something similar.)

Btw, I'm resending your "fix ia64" patch shortly.

Thanks,
Mike

#38 lore raw
In reply to Ingo Molnar
Quoted 9 lines · Ingo Molnar

i suspect it's:

| commit 2d22bd5e74519854458ad372a89006e65f45e628
| Author: Mike Travis <[email protected]>
| Date:   Wed Dec 31 18:08:46 2008 -0800
|
|     x86: cleanup remaining cpumask_t code in microcode_core.c

as the microcode is loaded during CPU onlining.

yep, that's the bad one. Should i revert it or do you have a safe fix in mind?

Ingo

#39 lore raw
Quoted 10 lines · Ingo Molnar
Quoted 8 lines · Ingo Molnar

i suspect it's:

| commit 2d22bd5e74519854458ad372a89006e65f45e628
| Author: Mike Travis <[email protected]>
| Date:   Wed Dec 31 18:08:46 2008 -0800
|
|     x86: cleanup remaining cpumask_t code in microcode_core.c
Quoted 6 lines
Quoted 1 line

as the microcode is loaded during CPU onlining.

yep, that's the bad one. Should i revert it or do you have a safe fix in mind?

Ingo

Probably revert for now. There are a few more following patches that also use 'work_on_cpu' so a better (more global?) fix should be used.

Any thought on using a recursive lock for cpu-hotplug-lock? (At least for get_online_cpus()?)

Thanks,
Mike

#40 lore raw
Quoted 16 lines · Mike Travis
Quoted 15 lines · Ingo Molnar
Quoted 9 lines · Ingo Molnar

i suspect it's:

| commit 2d22bd5e74519854458ad372a89006e65f45e628
| Author: Mike Travis <[email protected]>
| Date:   Wed Dec 31 18:08:46 2008 -0800
|
|     x86: cleanup remaining cpumask_t code in microcode_core.c

as the microcode is loaded during CPU onlining.

yep, that's the bad one. Should i revert it or do you have a safe fix in mind?

Quoted 7 lines
Quoted 1 line

Ingo

Probably revert for now. There are a few more following patches that also use 'work_on_cpu' so a better (more global?) fix should be used.

Any thought on using a recursive lock for cpu-hotplug-lock? (At least for get_online_cpus()?)

but the problem has nothing to do with self-recursion. Take a look at the lockdep warning i posted (also below) - the locks are simply taken in the wrong order.

your change adds this cpu_hotplug.lock usage:

[   43.652000] -> #1 (&cpu_hotplug.lock){--..}:
[   43.652000]        [<ffffffff8027a7c0>] __lock_acquire+0xf10/0x1360
[   43.652000]        [<ffffffff8027aca9>] lock_acquire+0x99/0xd0
[   43.652000]        [<ffffffff809b5e4a>] __mutex_lock_common+0xaa/0x450
[   43.652000]        [<ffffffff809b62cf>] mutex_lock_nested+0x3f/0x50
[   43.652000]        [<ffffffff802516ba>] get_online_cpus+0x3a/0x50
[   43.652000]        [<ffffffff802648dc>] work_on_cpu+0x6c/0xc0
[   43.652000]        [<ffffffff8022b2a2>] mc_sysdev_add+0x92/0xa0
[   43.652000]        [<ffffffff8050a800>] sysdev_driver_register+0xb0/0x140
[   43.652000]        [<ffffffff8163c792>] microcode_init+0xb2/0x13b
[   43.652000]        [<ffffffff8020a041>] do_one_initcall+0x41/0x180
[   43.652000]        [<ffffffff8162e6cb>] kernel_init+0x145/0x19d
[   43.652000]        [<ffffffff802146aa>] child_rip+0xa/0x20
[   43.652000]        [<ffffffffffffffff>] 0xffffffffffffffff

which nests the inside sysdev_drivers_lock - which is wrong [sysdev_drivers_lock is a pretty lowlevel lock that generally nests inside the CPU hotplug lock].

If you want to use work_on_cpu() it should be done on a higher level, so that sysdev_drivers_lock is taken after the hotplug lock.

Ingo

[   43.376051] lockdep: fixing up alternatives.
[   43.380007] SMP alternatives: switching to UP code
[   43.616014] CPU0 attaching NULL sched-domain.
[   43.620068] CPU1 attaching NULL sched-domain.
[   43.644482] CPU0 attaching NULL sched-domain.
[   43.648264] 
[   43.648265] =======================================================
[   43.652000] [ INFO: possible circular locking dependency detected ]
[   43.652000] 2.6.28-05081-geeff031-dirty #37
[   43.652000] -------------------------------------------------------
[   43.652000] S99local/1238 is trying to acquire lock:
[   43.652000]  (sysdev_drivers_lock){--..}, at: [<ffffffff8050a52d>] sysdev_unregister+0x1d/0x80
[   43.652000] 
[   43.652000] but task is already holding lock:
[   43.652000]  (&cpu_hotplug.lock){--..}, at: [<ffffffff802515d7>] cpu_hotplug_begin+0x27/0x60
[   43.652000] 
[   43.652000] which lock already depends on the new lock.
[   43.652000] 
[   43.652000] 
[   43.652000] the existing dependency chain (in reverse order) is:
[   43.652000] 
[   43.652000] -> #1 (&cpu_hotplug.lock){--..}:
[   43.652000]        [<ffffffff8027a7c0>] __lock_acquire+0xf10/0x1360
[   43.652000]        [<ffffffff8027aca9>] lock_acquire+0x99/0xd0
[   43.652000]        [<ffffffff809b5e4a>] __mutex_lock_common+0xaa/0x450
[   43.652000]        [<ffffffff809b62cf>] mutex_lock_nested+0x3f/0x50
[   43.652000]        [<ffffffff802516ba>] get_online_cpus+0x3a/0x50
[   43.652000]        [<ffffffff802648dc>] work_on_cpu+0x6c/0xc0
[   43.652000]        [<ffffffff8022b2a2>] mc_sysdev_add+0x92/0xa0
[   43.652000]        [<ffffffff8050a800>] sysdev_driver_register+0xb0/0x140
[   43.652000]        [<ffffffff8163c792>] microcode_init+0xb2/0x13b
[   43.652000]        [<ffffffff8020a041>] do_one_initcall+0x41/0x180
[   43.652000]        [<ffffffff8162e6cb>] kernel_init+0x145/0x19d
[   43.652000]        [<ffffffff802146aa>] child_rip+0xa/0x20
[   43.652000]        [<ffffffffffffffff>] 0xffffffffffffffff
[   43.652000] 
[   43.652000] -> #0 (sysdev_drivers_lock){--..}:
[   43.652000]        [<ffffffff8027a89c>] __lock_acquire+0xfec/0x1360
[   43.652000]        [<ffffffff8027aca9>] lock_acquire+0x99/0xd0
[   43.652000]        [<ffffffff809b5e4a>] __mutex_lock_common+0xaa/0x450
[   43.652000]        [<ffffffff809b62cf>] mutex_lock_nested+0x3f/0x50
[   43.652000]        [<ffffffff8050a52d>] sysdev_unregister+0x1d/0x80
[   43.652000]        [<ffffffff809af9d1>] mce_cpu_callback+0xce/0x101
[   43.652000]        [<ffffffff809bbb75>] notifier_call_chain+0x65/0xa0
[   43.652000]        [<ffffffff8026d696>] raw_notifier_call_chain+0x16/0x20
[   43.652000]        [<ffffffff80964a00>] _cpu_down+0x240/0x350
[   43.652000]        [<ffffffff80964b8b>] cpu_down+0x7b/0xa0
[   43.652000]        [<ffffffff80966268>] store_online+0x48/0xa0
[   43.652000]        [<ffffffff80509e90>] sysdev_store+0x20/0x30
[   43.652000]        [<ffffffff80335ddf>] sysfs_write_file+0xcf/0x140
[   43.652000]        [<ffffffff802dc1f7>] vfs_write+0xc7/0x150
[   43.652000]        [<ffffffff802dc375>] sys_write+0x55/0x90
[   43.652000]        [<ffffffff802133ca>] system_call_fastpath+0x16/0x1b
[   43.652000]        [<ffffffffffffffff>] 0xffffffffffffffff
[   43.652000] 
[   43.652000] other info that might help us debug this:
[   43.652000] 
[   43.652000] 3 locks held by S99local/1238:
[   43.652000]  #0:  (&buffer->mutex){--..}, at: [<ffffffff80335d58>] sysfs_write_file+0x48/0x140
[   43.652000]  #1:  (cpu_add_remove_lock){--..}, at: [<ffffffff80964b3f>] cpu_down+0x2f/0xa0
[   43.652000]  #2:  (&cpu_hotplug.lock){--..}, at: [<ffffffff802515d7>] cpu_hotplug_begin+0x27/0x60
[   43.652000] 
[   43.652000] stack backtrace:
[   43.652000] Pid: 1238, comm: S99local Not tainted 2.6.28-05081-geeff031-dirty #37
[   43.652000] Call Trace:
[   43.652000]  [<ffffffff80277f24>] print_circular_bug_tail+0xa4/0x100
[   43.652000]  [<ffffffff8027a89c>] __lock_acquire+0xfec/0x1360
[   43.652000]  [<ffffffff8027aca9>] lock_acquire+0x99/0xd0
[   43.652000]  [<ffffffff8050a52d>] ? sysdev_unregister+0x1d/0x80
[   43.652000]  [<ffffffff809b5e4a>] __mutex_lock_common+0xaa/0x450
[   43.652000]  [<ffffffff8050a52d>] ? sysdev_unregister+0x1d/0x80
[   43.652000]  [<ffffffff8050a52d>] ? sysdev_unregister+0x1d/0x80
[   43.652000]  [<ffffffff809b62cf>] mutex_lock_nested+0x3f/0x50
[   43.652000]  [<ffffffff8050a52d>] sysdev_unregister+0x1d/0x80
[   43.652000]  [<ffffffff809af9d1>] mce_cpu_callback+0xce/0x101
[   43.652000]  [<ffffffff809bbb75>] notifier_call_chain+0x65/0xa0
[   43.652000]  [<ffffffff8026d696>] raw_notifier_call_chain+0x16/0x20
[   43.652000]  [<ffffffff80964a00>] _cpu_down+0x240/0x350
[   43.652000]  [<ffffffff809b4763>] ? wait_for_common+0xe3/0x1b0
[   43.652000]  [<ffffffff80964b8b>] cpu_down+0x7b/0xa0
[   43.652000]  [<ffffffff80966268>] store_online+0x48/0xa0
[   43.652000]  [<ffffffff80509e90>] sysdev_store+0x20/0x30
[   43.652000]  [<ffffffff80335ddf>] sysfs_write_file+0xcf/0x140
[   43.652000]  [<ffffffff802dc1f7>] vfs_write+0xc7/0x150
[   43.652000]  [<ffffffff802dc375>] sys_write+0x55/0x90
[   43.652000]  [<ffffffff802133ca>] system_call_fastpath+0x16/0x1b
[   43.652104] device: 'msr1': device_unregister
[   43.656005] PM: Removing info for No Bus:msr1
#41 lore raw
Quoted 45 lines · Ingo Molnar
Quoted 15 lines · Mike Travis
Quoted 14 lines · Ingo Molnar
Quoted 9 lines · Ingo Molnar

i suspect it's:

| commit 2d22bd5e74519854458ad372a89006e65f45e628
| Author: Mike Travis <[email protected]>
| Date:   Wed Dec 31 18:08:46 2008 -0800
|
|     x86: cleanup remaining cpumask_t code in microcode_core.c

as the microcode is loaded during CPU onlining.

yep, that's the bad one. Should i revert it or do you have a safe fix in mind?

Quoted 6 lines
Quoted 1 line

Ingo

Probably revert for now. There are a few more following patches that also use 'work_on_cpu' so a better (more global?) fix should be used.

Any thought on using a recursive lock for cpu-hotplug-lock? (At least for get_online_cpus()?)

but the problem has nothing to do with self-recursion. Take a look at the lockdep warning i posted (also below) - the locks are simply taken in the wrong order.

your change adds this cpu_hotplug.lock usage:

[   43.652000] -> #1 (&cpu_hotplug.lock){--..}:
[   43.652000]        [<ffffffff8027a7c0>] __lock_acquire+0xf10/0x1360
[   43.652000]        [<ffffffff8027aca9>] lock_acquire+0x99/0xd0
[   43.652000]        [<ffffffff809b5e4a>] __mutex_lock_common+0xaa/0x450
[   43.652000]        [<ffffffff809b62cf>] mutex_lock_nested+0x3f/0x50
[   43.652000]        [<ffffffff802516ba>] get_online_cpus+0x3a/0x50
[   43.652000]        [<ffffffff802648dc>] work_on_cpu+0x6c/0xc0
[   43.652000]        [<ffffffff8022b2a2>] mc_sysdev_add+0x92/0xa0
[   43.652000]        [<ffffffff8050a800>] sysdev_driver_register+0xb0/0x140
[   43.652000]        [<ffffffff8163c792>] microcode_init+0xb2/0x13b
[   43.652000]        [<ffffffff8020a041>] do_one_initcall+0x41/0x180
[   43.652000]        [<ffffffff8162e6cb>] kernel_init+0x145/0x19d
[   43.652000]        [<ffffffff802146aa>] child_rip+0xa/0x20
[   43.652000]        [<ffffffffffffffff>] 0xffffffffffffffff
Quoted 8 lines

which nests the inside sysdev_drivers_lock - which is wrong [sysdev_drivers_lock is a pretty lowlevel lock that generally nests inside the CPU hotplug lock].

If you want to use work_on_cpu() it should be done on a higher level, so that sysdev_drivers_lock is taken after the hotplug lock.

Ingo

Ok, thanks, I will look in that direction.

Mike

Quoted 89 lines
[   43.376051] lockdep: fixing up alternatives.
[   43.380007] SMP alternatives: switching to UP code
[   43.616014] CPU0 attaching NULL sched-domain.
[   43.620068] CPU1 attaching NULL sched-domain.
[   43.644482] CPU0 attaching NULL sched-domain.
[   43.648264] 
[   43.648265] =======================================================
[   43.652000] [ INFO: possible circular locking dependency detected ]
[   43.652000] 2.6.28-05081-geeff031-dirty #37
[   43.652000] -------------------------------------------------------
[   43.652000] S99local/1238 is trying to acquire lock:
[   43.652000]  (sysdev_drivers_lock){--..}, at: [<ffffffff8050a52d>] sysdev_unregister+0x1d/0x80
[   43.652000] 
[   43.652000] but task is already holding lock:
[   43.652000]  (&cpu_hotplug.lock){--..}, at: [<ffffffff802515d7>] cpu_hotplug_begin+0x27/0x60
[   43.652000] 
[   43.652000] which lock already depends on the new lock.
[   43.652000] 
[   43.652000] 
[   43.652000] the existing dependency chain (in reverse order) is:
[   43.652000] 
[   43.652000] -> #1 (&cpu_hotplug.lock){--..}:
[   43.652000]        [<ffffffff8027a7c0>] __lock_acquire+0xf10/0x1360
[   43.652000]        [<ffffffff8027aca9>] lock_acquire+0x99/0xd0
[   43.652000]        [<ffffffff809b5e4a>] __mutex_lock_common+0xaa/0x450
[   43.652000]        [<ffffffff809b62cf>] mutex_lock_nested+0x3f/0x50
[   43.652000]        [<ffffffff802516ba>] get_online_cpus+0x3a/0x50
[   43.652000]        [<ffffffff802648dc>] work_on_cpu+0x6c/0xc0
[   43.652000]        [<ffffffff8022b2a2>] mc_sysdev_add+0x92/0xa0
[   43.652000]        [<ffffffff8050a800>] sysdev_driver_register+0xb0/0x140
[   43.652000]        [<ffffffff8163c792>] microcode_init+0xb2/0x13b
[   43.652000]        [<ffffffff8020a041>] do_one_initcall+0x41/0x180
[   43.652000]        [<ffffffff8162e6cb>] kernel_init+0x145/0x19d
[   43.652000]        [<ffffffff802146aa>] child_rip+0xa/0x20
[   43.652000]        [<ffffffffffffffff>] 0xffffffffffffffff
[   43.652000] 
[   43.652000] -> #0 (sysdev_drivers_lock){--..}:
[   43.652000]        [<ffffffff8027a89c>] __lock_acquire+0xfec/0x1360
[   43.652000]        [<ffffffff8027aca9>] lock_acquire+0x99/0xd0
[   43.652000]        [<ffffffff809b5e4a>] __mutex_lock_common+0xaa/0x450
[   43.652000]        [<ffffffff809b62cf>] mutex_lock_nested+0x3f/0x50
[   43.652000]        [<ffffffff8050a52d>] sysdev_unregister+0x1d/0x80
[   43.652000]        [<ffffffff809af9d1>] mce_cpu_callback+0xce/0x101
[   43.652000]        [<ffffffff809bbb75>] notifier_call_chain+0x65/0xa0
[   43.652000]        [<ffffffff8026d696>] raw_notifier_call_chain+0x16/0x20
[   43.652000]        [<ffffffff80964a00>] _cpu_down+0x240/0x350
[   43.652000]        [<ffffffff80964b8b>] cpu_down+0x7b/0xa0
[   43.652000]        [<ffffffff80966268>] store_online+0x48/0xa0
[   43.652000]        [<ffffffff80509e90>] sysdev_store+0x20/0x30
[   43.652000]        [<ffffffff80335ddf>] sysfs_write_file+0xcf/0x140
[   43.652000]        [<ffffffff802dc1f7>] vfs_write+0xc7/0x150
[   43.652000]        [<ffffffff802dc375>] sys_write+0x55/0x90
[   43.652000]        [<ffffffff802133ca>] system_call_fastpath+0x16/0x1b
[   43.652000]        [<ffffffffffffffff>] 0xffffffffffffffff
[   43.652000] 
[   43.652000] other info that might help us debug this:
[   43.652000] 
[   43.652000] 3 locks held by S99local/1238:
[   43.652000]  #0:  (&buffer->mutex){--..}, at: [<ffffffff80335d58>] sysfs_write_file+0x48/0x140
[   43.652000]  #1:  (cpu_add_remove_lock){--..}, at: [<ffffffff80964b3f>] cpu_down+0x2f/0xa0
[   43.652000]  #2:  (&cpu_hotplug.lock){--..}, at: [<ffffffff802515d7>] cpu_hotplug_begin+0x27/0x60
[   43.652000] 
[   43.652000] stack backtrace:
[   43.652000] Pid: 1238, comm: S99local Not tainted 2.6.28-05081-geeff031-dirty #37
[   43.652000] Call Trace:
[   43.652000]  [<ffffffff80277f24>] print_circular_bug_tail+0xa4/0x100
[   43.652000]  [<ffffffff8027a89c>] __lock_acquire+0xfec/0x1360
[   43.652000]  [<ffffffff8027aca9>] lock_acquire+0x99/0xd0
[   43.652000]  [<ffffffff8050a52d>] ? sysdev_unregister+0x1d/0x80
[   43.652000]  [<ffffffff809b5e4a>] __mutex_lock_common+0xaa/0x450
[   43.652000]  [<ffffffff8050a52d>] ? sysdev_unregister+0x1d/0x80
[   43.652000]  [<ffffffff8050a52d>] ? sysdev_unregister+0x1d/0x80
[   43.652000]  [<ffffffff809b62cf>] mutex_lock_nested+0x3f/0x50
[   43.652000]  [<ffffffff8050a52d>] sysdev_unregister+0x1d/0x80
[   43.652000]  [<ffffffff809af9d1>] mce_cpu_callback+0xce/0x101
[   43.652000]  [<ffffffff809bbb75>] notifier_call_chain+0x65/0xa0
[   43.652000]  [<ffffffff8026d696>] raw_notifier_call_chain+0x16/0x20
[   43.652000]  [<ffffffff80964a00>] _cpu_down+0x240/0x350
[   43.652000]  [<ffffffff809b4763>] ? wait_for_common+0xe3/0x1b0
[   43.652000]  [<ffffffff80964b8b>] cpu_down+0x7b/0xa0
[   43.652000]  [<ffffffff80966268>] store_online+0x48/0xa0
[   43.652000]  [<ffffffff80509e90>] sysdev_store+0x20/0x30
[   43.652000]  [<ffffffff80335ddf>] sysfs_write_file+0xcf/0x140
[   43.652000]  [<ffffffff802dc1f7>] vfs_write+0xc7/0x150
[   43.652000]  [<ffffffff802dc375>] sys_write+0x55/0x90
[   43.652000]  [<ffffffff802133ca>] system_call_fastpath+0x16/0x1b
[   43.652104] device: 'msr1': device_unregister
[   43.656005] PM: Removing info for No Bus:msr1
#42 lore raw
In reply to Mike Travis
Quoted 4 lines · Mike Travis
Quoted 2 lines

yep, that's the bad one. Should i revert it or do you have a safe fix in mind?

Probably revert for now. [...]

done.

But -tip testing found another bug today as well, a boot crash with certain (rare) 64-bit configs:

[    1.588202] ACPI: PCI Interrupt Link [LNKA] (IRQs<1>BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
[    1.588012] IP: [<ffffffff80239778>] find_busiest_group+0x198/0xa20
[    1.588048] Call Trace:
[    1.588049]  <IRQ> <0> [<ffffffff80240f96>] rebalance_domains+0x196/0x5e0
[    1.588052]  [<ffffffff80270b15>] ? lock_release_holdtime+0x35/0x1e0
[    1.588055]  [<ffffffff80983f70>] ? _spin_unlock_irq+0x30/0x40
[    1.588058]  [<ffffffff8024300e>] run_rebalance_domains+0x4e/0x120
[    1.588060]  [<ffffffff8024f80c>] __do_softirq+0xac/0x190
[    1.588063]  [<ffffffff8020d13c>] call_softirq+0x1c/0x30
[    1.588066]  [<ffffffff8020ef35>] do_softirq+0x75/0xa0
[    1.588067]  [<ffffffff8024f47d>] irq_exit+0x9d/0xb0
[    1.588069]  [<ffffffff80984f5d>] smp_apic_timer_interrupt+0x8d/0xc3
[    1.588071]  [<ffffffff8020cb73>] apic_timer_interrupt+0x13/0x20

i just bisected it back to:

| 74c5409893751c400547184751410c61930043b2 is first bad commit
| commit 74c5409893751c400547184751410c61930043b2
| Author: Mike Travis <[email protected]>
| Date:   Wed Dec 31 18:08:45 2008 -0800
|
|     x86: cleanup remaining cpumask_t ops in smpboot code
|    
|     Impact: Reduce memory usage and use new cpumask API.

this is in the final pieces of changes you did after pulling Rusty's tree:

26e2013: x86: setup_per_cpu_areas() cleanup
44aa683: cpumask: fix compile error when CONFIG_NR_CPUS is not defined
eeff031: cpumask: use alloc_cpumask_var_node where appropriate 40fbcb0: cpumask: convert shared_cpu_map in acpi_processor* structs to cpumask_var 197e99c: x86: use cpumask_var_t in acpi/boot.c
2d22bd5: x86: cleanup remaining cpumask_t code in microcode_core.c 22022f5: x86: cleanup remaining cpumask_t code in mce_amd_64.c
b5f3096: x86: cleanup some remaining usages of NR_CPUS where s/b nr_cpu_ids
efb897c: sched: put back some stack hog changes that were undone in kernel/sched.c 74c5409: x86: cleanup remaining cpumask_t ops in smpboot code 8627b2a: x86: enable cpus display of kernel_max and offlined cpus 095fb96: Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/ru

i think i'll just rebase the tail portion of cpus4096 starting at 8627b2a
- this keeps most of the history intact and avoids these ugly reverts.

Also, while bisecting this window of commits i found that neither would build successfully due to a typo - and the typo is fixed in 095fb96. So since we rebase this portion anyway due to excessive amount of bugs, i'll make it fully bisectable by rebasing right at 095fb96, backmerge the fixlet from eeff031 and redo the whole series dropping the two bad patches. Since this portion of the tree has no appreciable testing value the rebase is the right thing to do here.

Ingo

#43 lore raw
Quoted 7 lines · Ingo Molnar

Also, while bisecting this window of commits i found that neither would build successfully due to a typo - and the typo is fixed in 095fb96. So since we rebase this portion anyway due to excessive amount of bugs, i'll make it fully bisectable by rebasing right at 095fb96, backmerge the fixlet from eeff031 and redo the whole series dropping the two bad patches. Since this portion of the tree has no appreciable testing value the rebase is the right thing to do here.

okay, i've done this now and pushed out the resulting tree to tip/cpus4096-v2. It is a no-content-changed reshuffle of the tail ~10 commits of tip/cpus4096:

 $ git diff cpus4096..cpus4096-v2
 $

i dropped the two broken patches i bisected today, and backmerged a fixlet to make it more bisectable, and reordered fixes next to the merge point - and the new changes now come after that.

this finally is something that has no known regressions and looks pushable to Linus. I've started the -tip tests, lets see how well this holds up now.

Ingo

#44 lore raw
In reply to Ingo Molnar
Quoted 63 lines · Ingo Molnar
Quoted 3 lines · Mike Travis
Quoted 2 lines

yep, that's the bad one. Should i revert it or do you have a safe fix in mind?

Probably revert for now. [...]

done.

But -tip testing found another bug today as well, a boot crash with certain (rare) 64-bit configs:

[    1.588202] ACPI: PCI Interrupt Link [LNKA] (IRQs<1>BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
[    1.588012] IP: [<ffffffff80239778>] find_busiest_group+0x198/0xa20
[    1.588048] Call Trace:
[    1.588049]  <IRQ> <0> [<ffffffff80240f96>] rebalance_domains+0x196/0x5e0
[    1.588052]  [<ffffffff80270b15>] ? lock_release_holdtime+0x35/0x1e0
[    1.588055]  [<ffffffff80983f70>] ? _spin_unlock_irq+0x30/0x40
[    1.588058]  [<ffffffff8024300e>] run_rebalance_domains+0x4e/0x120
[    1.588060]  [<ffffffff8024f80c>] __do_softirq+0xac/0x190
[    1.588063]  [<ffffffff8020d13c>] call_softirq+0x1c/0x30
[    1.588066]  [<ffffffff8020ef35>] do_softirq+0x75/0xa0
[    1.588067]  [<ffffffff8024f47d>] irq_exit+0x9d/0xb0
[    1.588069]  [<ffffffff80984f5d>] smp_apic_timer_interrupt+0x8d/0xc3
[    1.588071]  [<ffffffff8020cb73>] apic_timer_interrupt+0x13/0x20

i just bisected it back to:

| 74c5409893751c400547184751410c61930043b2 is first bad commit
| commit 74c5409893751c400547184751410c61930043b2
| Author: Mike Travis <[email protected]>
| Date:   Wed Dec 31 18:08:45 2008 -0800
|
|     x86: cleanup remaining cpumask_t ops in smpboot code
|    
|     Impact: Reduce memory usage and use new cpumask API.

this is in the final pieces of changes you did after pulling Rusty's tree:

26e2013: x86: setup_per_cpu_areas() cleanup
44aa683: cpumask: fix compile error when CONFIG_NR_CPUS is not defined
eeff031: cpumask: use alloc_cpumask_var_node where appropriate 40fbcb0: cpumask: convert shared_cpu_map in acpi_processor* structs to cpumask_var 197e99c: x86: use cpumask_var_t in acpi/boot.c
2d22bd5: x86: cleanup remaining cpumask_t code in microcode_core.c 22022f5: x86: cleanup remaining cpumask_t code in mce_amd_64.c
b5f3096: x86: cleanup some remaining usages of NR_CPUS where s/b nr_cpu_ids
efb897c: sched: put back some stack hog changes that were undone in kernel/sched.c 74c5409: x86: cleanup remaining cpumask_t ops in smpboot code 8627b2a: x86: enable cpus display of kernel_max and offlined cpus 095fb96: Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/ru

i think i'll just rebase the tail portion of cpus4096 starting at 8627b2a
- this keeps most of the history intact and avoids these ugly reverts.

Also, while bisecting this window of commits i found that neither would build successfully due to a typo - and the typo is fixed in 095fb96. So since we rebase this portion anyway due to excessive amount of bugs, i'll make it fully bisectable by rebasing right at 095fb96, backmerge the fixlet from eeff031 and redo the whole series dropping the two bad patches. Since this portion of the tree has no appreciable testing value the rebase is the right thing to do here.

Quoted 1 line

Ingo

Ok, thanks. Still working through my queue... I'll re-pull when you've got your part done.

Mike

#4 lore raw

[git pull] cpus4096 tree, part 3

In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds
Quoted 3 lines · Ingo Molnar

Linus, would you like to pull that, despite the pending regressions? Can send a pull request right away.

No, if there are known pending regressions, please get those fixed first.

ok. The pending regressions are all fixed now, and i've just finished my standard tests on the latest tree and all the tests passed fine.

Please pull the cpus4096-for-linus-3 git tree from:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git cpus4096-for-linus-3

Thanks,

Ingo

------------------>
Cyrill Gorcunov (1):
x86: setup_per_cpu_areas() cleanup

Ingo Molnar (2):
cpumask: convert RCU implementations, fix
ia64: cpumask fix for is_affinity_mask_valid()

Li Zefan (1):
cpumask: fix bogus kernel-doc

Mike Travis (9):
cpumask: Add alloc_cpumask_var_node()
cpumask: documentation for cpumask_var_t
cpumask: add sysfs displays for configured and disabled cpu maps
sysfs: add documentation to cputopology.txt for system cpumasks
x86: enable cpus display of kernel_max and offlined cpus
sched: put back some stack hog changes that were undone in kernel/sched.c
x86: cleanup some remaining usages of NR_CPUS where s/b nr_cpu_ids
cpumask: use alloc_cpumask_var_node where appropriate
cpumask: fix compile error when CONFIG_NR_CPUS is not defined

Rusty Russell (63):
cpumask: x86: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: sparc: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: sh: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: powerpc: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: IA64: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: Mips: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: alpha: Introduce cpumask_of_{node,pcibus} to replace {node,pcibus}_to_cpumask
cpumask: cpu_coregroup_mask(): x86
cpumask: cpu_coregroup_mask(): sparc
cpumask: cpu_coregroup_mask(): s390
cpumask: Replace cpu_coregroup_map with cpu_coregroup_mask
cpumask: make CONFIG_NR_CPUS always valid.
bitmap: test for constant as well as small size for inline versions
bitmap: fix seq_bitmap and seq_cpumask to take const pointer
cpumask: switch over to cpu_online/possible/active/present_mask: core
cpumask: make cpumask.h eat its own dogfood.
cpumask: make set_cpu_*/init_cpu_* out-of-line
cpumask: smp_call_function_many()
cpumask: arch_send_call_function_ipi_mask: core
cpumask: use for_each_online_cpu() in drivers/infiniband/hw/ehca/ehca_irq.c
cpumask: use new cpumask API in drivers/infiniband/hw/ehca
cpumask: use new cpumask API in drivers/infiniband/hw/ipath
cpumask: Use nr_cpu_ids in seq_cpumask
cpumask: Remove IA64 definition of total_cpus now it's in core code
percpu: fix percpu accessors to potentially !cpu_possible() cpus: pnpbios
percpu: fix percpu accessors to potentially !cpu_possible() cpus: m32r
cpumask: prepare for iterators to only go to nr_cpu_ids/nr_cpumask_bits.: core
cpumask: Use accessors code in core
parisc: remove gratuitous cpu_online_map declaration.
avr32: define __fls
blackfin: define __fls
m68k: define __fls
m68knommu: define __fls
bitmap: find_last_bit()
cpumask: Use find_last_bit()
cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): sparc
cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): s390
cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): powerpc
cpumask: Introduce topology_core_cpumask()/topology_thread_cpumask(): ia64
cpumask: convert kernel trace functions
cpumask: convert kernel trace functions further
cpumask: remove any_online_cpu() users: kernel/
cpumask: remove any_online_cpu() users: mm/
cpumask: convert kernel/compat.c
cpumask: convert kernel/workqueue.c
cpumask: convert kernel time functions
cpumask: convert kernel/irq
cpumask: convert RCU implementations
cpumask: convert kernel/profile.c
cpumask: convert kernel/cpu.c
cpumask: convert rest of files in kernel/
cpumask: convert mm/
cpumask: replace for_each_cpu_mask_nr with for_each_cpu in kernel/time/
cpumask: zero extra bits in alloc_cpumask_var_node
cpumask: CONFIG_DISABLE_OBSOLETE_CPUMASK_FUNCTIONS
cris: define __fls
frv: define __fls
h8300: define __fls
m32r: define __fls
mn10300: define __fls
xtensa: define __fls
x86: use cpumask_var_t in acpi/boot.c
cpumask: convert shared_cpu_map in acpi_processor* structs to cpumask_var_t

Sergio Luis (1):
x86: mark get_cpu_leaves() with __cpuinit annotation

 Documentation/cputopology.txt                |   48 ++++++
 arch/alpha/include/asm/topology.h            |   17 ++
 arch/alpha/kernel/irq.c                      |    3 +-
 arch/alpha/kernel/setup.c                    |    5 +
 arch/avr32/include/asm/bitops.h              |    5 +
 arch/blackfin/include/asm/bitops.h           |    1 +
 arch/cris/include/asm/bitops.h               |    1 +
 arch/h8300/include/asm/bitops.h              |    1 +
 arch/ia64/include/asm/irq.h                  |    2 +-
 arch/ia64/include/asm/topology.h             |    9 +-
 arch/ia64/kernel/acpi.c                      |    3 +-
 arch/ia64/kernel/iosapic.c                   |   23 ++--
 arch/ia64/kernel/irq.c                       |    4 +-
 arch/ia64/sn/kernel/sn2/sn_hwperf.c          |   27 ++--
 arch/m32r/kernel/smpboot.c                   |    2 +-
 arch/m68knommu/include/asm/bitops.h          |    1 +
 arch/mips/include/asm/mach-ip27/topology.h   |    4 +-
 arch/parisc/include/asm/smp.h                |    2 -
 arch/powerpc/include/asm/topology.h          |   12 +-
 arch/powerpc/platforms/cell/spu_priv1_mmio.c |    6 +-
 arch/powerpc/platforms/cell/spufs/sched.c    |    4 +-
 arch/s390/include/asm/topology.h             |    2 +
 arch/s390/kernel/topology.c                  |    5 +
 arch/sh/include/asm/topology.h               |    1 +
 arch/sparc/include/asm/topology_64.h         |   13 +-
 arch/sparc/kernel/of_device_64.c             |    2 +-
 arch/sparc/kernel/pci_msi.c                  |    2 +-
 arch/x86/include/asm/es7000/apic.h           |   32 +----
 arch/x86/include/asm/lguest.h                |    2 +-
 arch/x86/include/asm/numaq/apic.h            |    4 +-
 arch/x86/include/asm/pci.h                   |   10 +-
 arch/x86/include/asm/summit/apic.h           |   42 +----
 arch/x86/include/asm/topology.h              |   36 +++--
 arch/x86/kernel/acpi/boot.c                  |   31 +++-
 arch/x86/kernel/apic.c                       |    4 +-
 arch/x86/kernel/cpu/common.c                 |    2 +-
 arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c   |   28 +++-
 arch/x86/kernel/cpu/cpufreq/powernow-k7.c    |    9 +
 arch/x86/kernel/cpu/cpufreq/powernow-k8.c    |   24 ++-
 arch/x86/kernel/cpu/intel_cacheinfo.c        |    2 +-
 arch/x86/kernel/cpuid.c                      |    2 +-
 arch/x86/kernel/io_apic.c                    |    6 +-
 arch/x86/kernel/msr.c                        |    2 +-
 arch/x86/kernel/reboot.c                     |    4 +-
 arch/x86/kernel/setup_percpu.c               |   33 ++---
 arch/x86/kernel/smpboot.c                    |   15 ++-
 arch/x86/mach-voyager/voyager_smp.c          |    7 +-
 block/blk.h                                  |    4 +-
 drivers/acpi/processor_core.c                |   14 ++-
 drivers/acpi/processor_perflib.c             |   28 ++--
 drivers/acpi/processor_throttling.c          |   80 ++++++----
 drivers/base/cpu.c                           |   44 +++++
 drivers/infiniband/hw/ehca/ehca_irq.c        |   17 +-
 drivers/infiniband/hw/ipath/ipath_file_ops.c |    8 +-
 drivers/pnp/pnpbios/bioscalls.c              |    2 +-
 fs/seq_file.c                                |    3 +-
 include/acpi/processor.h                     |    4 +-
 include/asm-frv/bitops.h                     |   13 ++
 include/asm-m32r/bitops.h                    |    1 +
 include/asm-m68k/bitops.h                    |    5 +
 include/asm-mn10300/bitops.h                 |   11 ++
 include/asm-xtensa/bitops.h                  |   11 ++
 include/linux/bitmap.h                       |   35 +++--
 include/linux/bitops.h                       |   13 ++-
 include/linux/cpumask.h                      |  221 +++++++++++---------------
 include/linux/interrupt.h                    |    2 +-
 include/linux/rcuclassic.h                   |    4 +-
 include/linux/seq_file.h                     |    7 +-
 include/linux/smp.h                          |   18 ++-
 include/linux/stop_machine.h                 |    6 +-
 include/linux/threads.h                      |   16 +-
 include/linux/tick.h                         |    4 +-
 init/main.c                                  |   13 +-
 kernel/compat.c                              |   49 ++++---
 kernel/cpu.c                                 |  144 ++++++++++++------
 kernel/irq/manage.c                          |   11 +-
 kernel/irq/proc.c                            |   34 +++--
 kernel/kexec.c                               |    2 +-
 kernel/power/poweroff.c                      |    2 +-
 kernel/profile.c                             |   38 +++--
 kernel/rcuclassic.c                          |   32 ++--
 kernel/rcupreempt.c                          |   19 ++-
 kernel/rcutorture.c                          |   27 ++--
 kernel/sched.c                               |   53 ++-----
 kernel/sched_rt.c                            |    3 +-
 kernel/smp.c                                 |  145 +++++++-----------
 kernel/softirq.c                             |    2 +-
 kernel/softlockup.c                          |   10 +-
 kernel/stop_machine.c                        |    8 +-
 kernel/taskstats.c                           |   39 +++--
 kernel/time/clocksource.c                    |    9 +-
 kernel/time/tick-broadcast.c                 |  115 +++++++-------
 kernel/time/tick-common.c                    |    6 +-
 kernel/trace/ring_buffer.c                   |   42 +++--
 kernel/trace/trace.c                         |   72 ++++++---
 kernel/trace/trace.h                         |    2 +-
 kernel/trace/trace_boot.c                    |    2 +-
 kernel/trace/trace_functions_graph.c         |    2 +-
 kernel/trace/trace_hw_branches.c             |    6 +-
 kernel/trace/trace_power.c                   |    2 +-
 kernel/trace/trace_sysprof.c                 |   13 +--
 kernel/workqueue.c                           |   26 ++--
 lib/Kconfig                                  |    8 +
 lib/Makefile                                 |    1 +
 lib/cpumask.c                                |   62 +++++++-
 lib/find_last_bit.c                          |   45 ++++++
 mm/pdflush.c                                 |   16 ++-
 mm/slab.c                                    |    2 +-
 mm/slub.c                                    |   20 ++-
 mm/vmscan.c                                  |    4 +-
 mm/vmstat.c                                  |    4 +-
 security/selinux/selinuxfs.c                 |    2 +-
 112 files changed, 1285 insertions(+), 878 deletions(-)
 create mode 100644 lib/find_last_bit.c
Documentation/cputopology.txt +48 -0
index bd699da..45932ec 100644
@@ -31,3 +31,51 @@ not defined by include/asm-XXX/topology.h:
 2) core_id: 0
 3) thread_siblings: just the given CPU
 4) core_siblings: just the given CPU
+
+Additionally, cpu topology information is provided under
+/sys/devices/system/cpu and includes these files.  The internal
+source for the output is in brackets ("[]").
+
+    kernel_max: the maximum cpu index allowed by the kernel configuration.
+		[NR_CPUS-1]
+
+    offline:	cpus that are not online because they have been
+		HOTPLUGGED off (see cpu-hotplug.txt) or exceed the limit
+		of cpus allowed by the kernel configuration (kernel_max
+		above). [~cpu_online_mask + cpus >= NR_CPUS]
+
+    online:	cpus that are online and being scheduled [cpu_online_mask]
+
+    possible:	cpus that have been allocated resources and can be
+		brought online if they are present. [cpu_possible_mask]
+
+    present:	cpus that have been identified as being present in the
+		system. [cpu_present_mask]
+
+The format for the above output is compatible with cpulist_parse()
+[see <linux/cpumask.h>].  Some examples follow.
+
+In this example, there are 64 cpus in the system but cpus 32-63 exceed
+the kernel max which is limited to 0..31 by the NR_CPUS config option
+being 32.  Note also that cpus 2 and 4-31 are not online but could be
+brought online as they are both present and possible.
+
+     kernel_max: 31
+        offline: 2,4-31,32-63
+         online: 0-1,3
+       possible: 0-31
+        present: 0-31
+
+In this example, the NR_CPUS config option is 128, but the kernel was
+started with possible_cpus=144.  There are 4 cpus in the system and cpu2
+was manually taken offline (and is the only cpu that can be brought
+online.)
+
+     kernel_max: 127
+        offline: 2,4-127,128-143
+         online: 0-1,3
+       possible: 0-127
+        present: 0-3
+
+See cpu-hotplug.txt for the possible_cpus=NUM kernel start parameter
+as well as more information on the various cpumask's.
arch/alpha/include/asm/topology.h +17 -0
index 149532e..b4f284c 100644
@@ -39,7 +39,24 @@ static inline cpumask_t node_to_cpumask(int node)
 	return node_cpu_mask;
 }
 
+extern struct cpumask node_to_cpumask_map[];
+/* FIXME: This is dumb, recalculating every time.  But simple. */
+static const struct cpumask *cpumask_of_node(int node)
+{
+	int cpu;
+
+	cpumask_clear(&node_to_cpumask_map[node]);
+
+	for_each_online_cpu(cpu) {
+		if (cpu_to_node(cpu) == node)
+			cpumask_set_cpu(cpu, node_to_cpumask_map[node]);
+	}
+
+	return &node_to_cpumask_map[node];
+}
+
 #define pcibus_to_cpumask(bus)	(cpu_online_map)
+#define cpumask_of_pcibus(bus)	(cpu_online_mask)
 
 #endif /* !CONFIG_NUMA */
 # include <asm-generic/topology.h>
arch/alpha/kernel/irq.c +2 -1
index d0f1620..703731a 100644
@@ -50,7 +50,8 @@ int irq_select_affinity(unsigned int irq)
 	if (!irq_desc[irq].chip->set_affinity || irq_user_affinity[irq])
 		return 1;
 
-	while (!cpu_possible(cpu) || !cpu_isset(cpu, irq_default_affinity))
+	while (!cpu_possible(cpu) ||
+	       !cpumask_test_cpu(cpu, irq_default_affinity))
 		cpu = (cpu < (NR_CPUS-1) ? cpu + 1 : 0);
 	last_cpu = cpu;
 
arch/alpha/kernel/setup.c +5 -0
index a449e99..02bee69 100644
@@ -79,6 +79,11 @@ int alpha_l3_cacheshape;
 unsigned long alpha_verbose_mcheck = CONFIG_VERBOSE_MCHECK_ON;
 #endif
 
+#ifdef CONFIG_NUMA
+struct cpumask node_to_cpumask_map[MAX_NUMNODES] __read_mostly;
+EXPORT_SYMBOL(node_to_cpumask_map);
+#endif
+
 /* Which processor we booted from.  */
 int boot_cpuid;
 
arch/avr32/include/asm/bitops.h +5 -0
index 1a50b69..f7dd5f7 100644
@@ -263,6 +263,11 @@ static inline int fls(unsigned long word)
 	return 32 - result;
 }
 
+static inline int __fls(unsigned long word)
+{
+	return fls(word) - 1;
+}
+
 unsigned long find_first_zero_bit(const unsigned long *addr,
 				  unsigned long size);
 unsigned long find_next_zero_bit(const unsigned long *addr,
arch/blackfin/include/asm/bitops.h +1 -0
index b39a175..c428e41 100644
@@ -213,6 +213,7 @@ static __inline__ int __test_bit(int nr, const void *addr)
 #endif				/* __KERNEL__ */
 
 #include <asm-generic/bitops/fls.h>
+#include <asm-generic/bitops/__fls.h>
 #include <asm-generic/bitops/fls64.h>
 
 #endif				/* _BLACKFIN_BITOPS_H */
arch/cris/include/asm/bitops.h +1 -0
index c0e62f8..9e69cfb 100644
@@ -148,6 +148,7 @@ static inline int test_and_change_bit(int nr, volatile unsigned long *addr)
 #define ffs kernel_ffs
 
 #include <asm-generic/bitops/fls.h>
+#include <asm-generic/bitops/__fls.h>
 #include <asm-generic/bitops/fls64.h>
 #include <asm-generic/bitops/hweight.h>
 #include <asm-generic/bitops/find.h>
arch/h8300/include/asm/bitops.h +1 -0
index cb18e3b..cb9ddf5 100644
@@ -207,6 +207,7 @@ static __inline__ unsigned long __ffs(unsigned long word)
 #endif /* __KERNEL__ */
 
 #include <asm-generic/bitops/fls.h>
+#include <asm-generic/bitops/__fls.h>
 #include <asm-generic/bitops/fls64.h>
 
 #endif /* _H8300_BITOPS_H */
arch/ia64/include/asm/irq.h +1 -1
index 3627116..36429a5 100644
@@ -27,7 +27,7 @@ irq_canonicalize (int irq)
 }
 
 extern void set_irq_affinity_info (unsigned int irq, int dest, int redir);
-bool is_affinity_mask_valid(cpumask_t cpumask);
+bool is_affinity_mask_valid(cpumask_var_t cpumask);
 
 #define is_affinity_mask_valid is_affinity_mask_valid
 
arch/ia64/include/asm/topology.h +8 -1
index a3cc9f6..76a33a9 100644
@@ -34,6 +34,7 @@
  * Returns a bitmask of CPUs on Node 'node'.
  */
 #define node_to_cpumask(node) (node_to_cpu_mask[node])
+#define cpumask_of_node(node) (&node_to_cpu_mask[node])
 
 /*
  * Returns the number of the node containing Node 'nid'.
@@ -45,7 +46,7 @@
 /*
  * Returns the number of the first CPU on Node 'node'.
  */
-#define node_to_first_cpu(node) (first_cpu(node_to_cpumask(node)))
+#define node_to_first_cpu(node) (cpumask_first(cpumask_of_node(node)))
 
 /*
  * Determines the node for a given pci bus
@@ -109,6 +110,8 @@ void build_cpu_to_node_map(void);
 #define topology_core_id(cpu)			(cpu_data(cpu)->core_id)
 #define topology_core_siblings(cpu)		(cpu_core_map[cpu])
 #define topology_thread_siblings(cpu)		(per_cpu(cpu_sibling_map, cpu))
+#define topology_core_cpumask(cpu)		(&cpu_core_map[cpu])
+#define topology_thread_cpumask(cpu)		(&per_cpu(cpu_sibling_map, cpu))
 #define smt_capable() 				(smp_num_siblings > 1)
 #endif
 
@@ -119,6 +122,10 @@ extern void arch_fix_phys_package_id(int num, u32 slot);
 					node_to_cpumask(pcibus_to_node(bus)) \
 				)
 
+#define cpumask_of_pcibus(bus)	(pcibus_to_node(bus) == -1 ?		\
+				 cpu_all_mask :				\
+				 cpumask_from_node(pcibus_to_node(bus)))
+
 #include <asm-generic/topology.h>
 
 #endif /* _ASM_IA64_TOPOLOGY_H */
arch/ia64/kernel/acpi.c +1 -2
index bd7acc7..0553648 100644
@@ -202,7 +202,6 @@ char *__init __acpi_map_table(unsigned long phys_addr, unsigned long size)
                             Boot-time Table Parsing
    -------------------------------------------------------------------------- */
 
-static int total_cpus __initdata;
 static int available_cpus __initdata;
 struct acpi_table_madt *acpi_madt __initdata;
 static u8 has_8259;
@@ -1001,7 +1000,7 @@ acpi_map_iosapic(acpi_handle handle, u32 depth, void *context, void **ret)
 	node = pxm_to_node(pxm);
 
 	if (node >= MAX_NUMNODES || !node_online(node) ||
-	    cpus_empty(node_to_cpumask(node)))
+	    cpumask_empty(cpumask_of_node(node)))
 		return AE_OK;
 
 	/* We know a gsi to node mapping! */
arch/ia64/kernel/iosapic.c +11 -12
index c8adecd..5cfd3d9 100644
@@ -695,32 +695,31 @@ get_target_cpu (unsigned int gsi, int irq)
 #ifdef CONFIG_NUMA
 	{
 		int num_cpus, cpu_index, iosapic_index, numa_cpu, i = 0;
-		cpumask_t cpu_mask;
+		const struct cpumask *cpu_mask;
 
 		iosapic_index = find_iosapic(gsi);
 		if (iosapic_index < 0 ||
 		    iosapic_lists[iosapic_index].node == MAX_NUMNODES)
 			goto skip_numa_setup;
 
-		cpu_mask = node_to_cpumask(iosapic_lists[iosapic_index].node);
-		cpus_and(cpu_mask, cpu_mask, domain);
-		for_each_cpu_mask(numa_cpu, cpu_mask) {
-			if (!cpu_online(numa_cpu))
-				cpu_clear(numa_cpu, cpu_mask);
+		cpu_mask = cpumask_of_node(iosapic_lists[iosapic_index].node);
+		num_cpus = 0;
+		for_each_cpu_and(numa_cpu, cpu_mask, &domain) {
+			if (cpu_online(numa_cpu))
+				num_cpus++;
 		}
 
-		num_cpus = cpus_weight(cpu_mask);
-
 		if (!num_cpus)
 			goto skip_numa_setup;
 
 		/* Use irq assignment to distribute across cpus in node */
 		cpu_index = irq % num_cpus;
 
-		for (numa_cpu = first_cpu(cpu_mask) ; i < cpu_index ; i++)
-			numa_cpu = next_cpu(numa_cpu, cpu_mask);
+		for_each_cpu_and(numa_cpu, cpu_mask, &domain)
+			if (cpu_online(numa_cpu) && i++ >= cpu_index)
+				break;
 
-		if (numa_cpu != NR_CPUS)
+		if (numa_cpu < nr_cpu_ids)
 			return cpu_physical_id(numa_cpu);
 	}
 skip_numa_setup:
@@ -731,7 +730,7 @@ skip_numa_setup:
 	 * case of NUMA.)
 	 */
 	do {
-		if (++cpu >= NR_CPUS)
+		if (++cpu >= nr_cpu_ids)
 			cpu = 0;
 	} while (!cpu_online(cpu) || !cpu_isset(cpu, domain));
 
arch/ia64/kernel/irq.c +2 -2
index 0b6db53..95ff16c 100644
@@ -112,11 +112,11 @@ void set_irq_affinity_info (unsigned int irq, int hwid, int redir)
 	}
 }
 
-bool is_affinity_mask_valid(cpumask_t cpumask)
+bool is_affinity_mask_valid(cpumask_var_t cpumask)
 {
 	if (ia64_platform_is("sn2")) {
 		/* Only allow one CPU to be specified in the smp_affinity mask */
-		if (cpus_weight(cpumask) != 1)
+		if (cpumask_weight(cpumask) != 1)
 			return false;
 	}
 	return true;
arch/ia64/sn/kernel/sn2/sn_hwperf.c +12 -15
index 636588e..be33947 100644
@@ -385,7 +385,6 @@ static int sn_topology_show(struct seq_file *s, void *d)
 	int j;
 	const char *slabname;
 	int ordinal;
-	cpumask_t cpumask;
 	char slice;
 	struct cpuinfo_ia64 *c;
 	struct sn_hwperf_port_info *ptdata;
@@ -473,23 +472,21 @@ static int sn_topology_show(struct seq_file *s, void *d)
 		 * CPUs on this node, if any
 		 */
 		if (!SN_HWPERF_IS_IONODE(obj)) {
-			cpumask = node_to_cpumask(ordinal);
-			for_each_online_cpu(i) {
-				if (cpu_isset(i, cpumask)) {
-					slice = 'a' + cpuid_to_slice(i);
-					c = cpu_data(i);
-					seq_printf(s, "cpu %d %s%c local"
-						" freq %luMHz, arch ia64",
-						i, obj->location, slice,
-						c->proc_freq / 1000000);
-					for_each_online_cpu(j) {
-						seq_printf(s, j ? ":%d" : ", dist %d",
-							node_distance(
+			for_each_cpu_and(i, cpu_online_mask,
+					 cpumask_of_node(ordinal)) {
+				slice = 'a' + cpuid_to_slice(i);
+				c = cpu_data(i);
+				seq_printf(s, "cpu %d %s%c local"
+					   " freq %luMHz, arch ia64",
+					   i, obj->location, slice,
+					   c->proc_freq / 1000000);
+				for_each_online_cpu(j) {
+					seq_printf(s, j ? ":%d" : ", dist %d",
+						   node_distance(
 						    	cpu_to_node(i),
 						    	cpu_to_node(j)));
-					}
-					seq_putc(s, '\n');
 				}
+				seq_putc(s, '\n');
 			}
 		}
 	}
arch/m32r/kernel/smpboot.c +1 -1
index 0f06b37..2547d6c 100644
@@ -592,7 +592,7 @@ int setup_profiling_timer(unsigned int multiplier)
 	 * accounting. At that time they also adjust their APIC timers
 	 * accordingly.
 	 */
-	for (i = 0; i < NR_CPUS; ++i)
+	for_each_possible_cpu(i)
 		per_cpu(prof_multiplier, i) = multiplier;
 
 	return 0;
arch/m68knommu/include/asm/bitops.h +1 -0
index 6f3685e..9d3cbe5 100644
@@ -331,6 +331,7 @@ found_middle:
 #endif /* __KERNEL__ */
 
 #include <asm-generic/bitops/fls.h>
+#include <asm-generic/bitops/__fls.h>
 #include <asm-generic/bitops/fls64.h>
 
 #endif /* _M68KNOMMU_BITOPS_H */
arch/mips/include/asm/mach-ip27/topology.h +3 -1
index 1fb959f..55d4815 100644
@@ -25,11 +25,13 @@ extern struct cpuinfo_ip27 sn_cpu_info[NR_CPUS];
 #define cpu_to_node(cpu)	(sn_cpu_info[(cpu)].p_nodeid)
 #define parent_node(node)	(node)
 #define node_to_cpumask(node)	(hub_data(node)->h_cpus)
-#define node_to_first_cpu(node)	(first_cpu(node_to_cpumask(node)))
+#define cpumask_of_node(node)	(&hub_data(node)->h_cpus)
+#define node_to_first_cpu(node)	(cpumask_first(cpumask_of_node(node)))
 struct pci_bus;
 extern int pcibus_to_node(struct pci_bus *);
 
 #define pcibus_to_cpumask(bus)	(cpu_online_map)
+#define cpumask_of_pcibus(bus)	(cpu_online_mask)
 
 extern unsigned char __node_distances[MAX_COMPACT_NODES][MAX_COMPACT_NODES];
 
arch/parisc/include/asm/smp.h +0 -2
index 409e698..6ef4b78 100644
@@ -16,8 +16,6 @@
 #include <linux/cpumask.h>
 typedef unsigned long address_t;
 
-extern cpumask_t cpu_online_map;
-
 
 /*
  *	Private routines/data
arch/powerpc/include/asm/topology.h +9 -3
index 373fca3..3752585 100644
@@ -22,11 +22,11 @@ static inline cpumask_t node_to_cpumask(int node)
 	return numa_cpumask_lookup_table[node];
 }
 
+#define cpumask_of_node(node) (&numa_cpumask_lookup_table[node])
+
 static inline int node_to_first_cpu(int node)
 {
-	cpumask_t tmp;
-	tmp = node_to_cpumask(node);
-	return first_cpu(tmp);
+	return cpumask_first(cpumask_of_node(node));
 }
 
 int of_node_to_nid(struct device_node *device);
@@ -46,6 +46,10 @@ static inline int pcibus_to_node(struct pci_bus *bus)
 					node_to_cpumask(pcibus_to_node(bus)) \
 				)
 
+#define cpumask_of_pcibus(bus)	(pcibus_to_node(bus) == -1 ?		\
+				 cpu_all_mask :				\
+				 cpumask_of_node(pcibus_to_node(bus)))
+
 /* sched_domains SD_NODE_INIT for PPC64 machines */
 #define SD_NODE_INIT (struct sched_domain) {		\
 	.parent			= NULL,			\
@@ -108,6 +112,8 @@ static inline void sysfs_remove_device_from_node(struct sys_device *dev,
 
 #define topology_thread_siblings(cpu)	(per_cpu(cpu_sibling_map, cpu))
 #define topology_core_siblings(cpu)	(per_cpu(cpu_core_map, cpu))
+#define topology_thread_cpumask(cpu)	(&per_cpu(cpu_sibling_map, cpu))
+#define topology_core_cpumask(cpu)	(&per_cpu(cpu_core_map, cpu))
 #define topology_core_id(cpu)		(cpu_to_core_id(cpu))
 #endif
 #endif
arch/powerpc/platforms/cell/spu_priv1_mmio.c +3 -3
index 906a0a2..1410443 100644
@@ -80,10 +80,10 @@ static void cpu_affinity_set(struct spu *spu, int cpu)
 	u64 route;
 
 	if (nr_cpus_node(spu->node)) {
-		cpumask_t spumask = node_to_cpumask(spu->node);
-		cpumask_t cpumask = node_to_cpumask(cpu_to_node(cpu));
+		const struct cpumask *spumask = cpumask_of_node(spu->node),
+			*cpumask = cpumask_of_node(cpu_to_node(cpu));
 
-		if (!cpus_intersects(spumask, cpumask))
+		if (!cpumask_intersects(spumask, cpumask))
 			return;
 	}
 
arch/powerpc/platforms/cell/spufs/sched.c +2 -2
index 2ad914c..6a0ad19 100644
@@ -166,9 +166,9 @@ void spu_update_sched_info(struct spu_context *ctx)
 static int __node_allowed(struct spu_context *ctx, int node)
 {
 	if (nr_cpus_node(node)) {
-		cpumask_t mask = node_to_cpumask(node);
+		const struct cpumask *mask = cpumask_of_node(node);
 
-		if (cpus_intersects(mask, ctx->cpus_allowed))
+		if (cpumask_intersects(mask, &ctx->cpus_allowed))
 			return 1;
 	}
 
arch/s390/include/asm/topology.h +2 -0
index d96c916..c93eb50 100644
@@ -6,10 +6,12 @@
 #define mc_capable()	(1)
 
 cpumask_t cpu_coregroup_map(unsigned int cpu);
+const struct cpumask *cpu_coregroup_mask(unsigned int cpu);
 
 extern cpumask_t cpu_core_map[NR_CPUS];
 
 #define topology_core_siblings(cpu)	(cpu_core_map[cpu])
+#define topology_core_cpumask(cpu)	(&cpu_core_map[cpu])
 
 int topology_set_cpu_management(int fc);
 void topology_schedule_update(void);
arch/s390/kernel/topology.c +5 -0
index 90e9ba1..cc362c9 100644
@@ -97,6 +97,11 @@ cpumask_t cpu_coregroup_map(unsigned int cpu)
 	return mask;
 }
 
+const struct cpumask *cpu_coregroup_mask(unsigned int cpu)
+{
+	return &cpu_core_map[cpu];
+}
+
 static void add_cpus_to_core(struct tl_cpu *tl_cpu, struct core_info *core)
 {
 	unsigned int cpu;
arch/sh/include/asm/topology.h +1 -0
index 279d9cc..066f0fb 100644
@@ -32,6 +32,7 @@
 #define parent_node(node)	((void)(node),0)
 
 #define node_to_cpumask(node)	((void)node, cpu_online_map)
+#define cpumask_of_node(node)	((void)node, cpu_online_mask)
 #define node_to_first_cpu(node)	((void)(node),0)
 
 #define pcibus_to_node(bus)	((void)(bus), -1)
arch/sparc/include/asm/topology_64.h +9 -4
index 001c040..b8a65b6 100644
@@ -16,8 +16,12 @@ static inline cpumask_t node_to_cpumask(int node)
 {
 	return numa_cpumask_lookup_table[node];
 }
+#define cpumask_of_node(node) (&numa_cpumask_lookup_table[node])
 
-/* Returns a pointer to the cpumask of CPUs on Node 'node'. */
+/*
+ * Returns a pointer to the cpumask of CPUs on Node 'node'.
+ * Deprecated: use "const struct cpumask *mask = cpumask_of_node(node)"
+ */
 #define node_to_cpumask_ptr(v, node)		\
 		cpumask_t *v = &(numa_cpumask_lookup_table[node])
 
@@ -26,9 +30,7 @@ static inline cpumask_t node_to_cpumask(int node)
 
 static inline int node_to_first_cpu(int node)
 {
-	cpumask_t tmp;
-	tmp = node_to_cpumask(node);
-	return first_cpu(tmp);
+	return cpumask_first(cpumask_of_node(node));
 }
 
 struct pci_bus;
@@ -77,10 +79,13 @@ static inline int pcibus_to_node(struct pci_bus *pbus)
 #define topology_core_id(cpu)			(cpu_data(cpu).core_id)
 #define topology_core_siblings(cpu)		(cpu_core_map[cpu])
 #define topology_thread_siblings(cpu)		(per_cpu(cpu_sibling_map, cpu))
+#define topology_core_cpumask(cpu)		(&cpu_core_map[cpu])
+#define topology_thread_cpumask(cpu)		(&per_cpu(cpu_sibling_map, cpu))
 #define mc_capable()				(sparc64_multi_core)
 #define smt_capable()				(sparc64_multi_core)
 #endif /* CONFIG_SMP */
 
 #define cpu_coregroup_map(cpu)			(cpu_core_map[cpu])
+#define cpu_coregroup_mask(cpu)			(&cpu_core_map[cpu])
 
 #endif /* _ASM_SPARC64_TOPOLOGY_H */
arch/sparc/kernel/of_device_64.c +1 -1
index 322046c..4873f28 100644
@@ -778,7 +778,7 @@ static unsigned int __init build_one_device_irq(struct of_device *op,
 out:
 	nid = of_node_to_nid(dp);
 	if (nid != -1) {
-		cpumask_t numa_mask = node_to_cpumask(nid);
+		cpumask_t numa_mask = *cpumask_of_node(nid);
 
 		irq_set_affinity(irq, &numa_mask);
 	}
arch/sparc/kernel/pci_msi.c +1 -1
index 0d0cd81..4ef282e 100644
@@ -286,7 +286,7 @@ static int bringup_one_msi_queue(struct pci_pbm_info *pbm,
 
 	nid = pbm->numa_node;
 	if (nid != -1) {
-		cpumask_t numa_mask = node_to_cpumask(nid);
+		cpumask_t numa_mask = *cpumask_of_node(nid);
 
 		irq_set_affinity(irq, &numa_mask);
 	}
arch/x86/include/asm/es7000/apic.h +3 -29
index 51ac123..bc53d5e 100644
@@ -157,7 +157,7 @@ cpu_mask_to_apicid_cluster(const struct cpumask *cpumask)
 
 	num_bits_set = cpumask_weight(cpumask);
 	/* Return id to all */
-	if (num_bits_set == NR_CPUS)
+	if (num_bits_set == nr_cpu_ids)
 		return 0xFF;
 	/*
 	 * The cpus in the mask must all be on the apic cluster.  If are not
@@ -190,7 +190,7 @@ static inline unsigned int cpu_mask_to_apicid(const cpumask_t *cpumask)
 
 	num_bits_set = cpus_weight(*cpumask);
 	/* Return id to all */
-	if (num_bits_set == NR_CPUS)
+	if (num_bits_set == nr_cpu_ids)
 		return cpu_to_logical_apicid(0);
 	/*
 	 * The cpus in the mask must all be on the apic cluster.  If are not
@@ -218,9 +218,6 @@ static inline unsigned int cpu_mask_to_apicid(const cpumask_t *cpumask)
 static inline unsigned int cpu_mask_to_apicid_and(const struct cpumask *inmask,
 						  const struct cpumask *andmask)
 {
-	int num_bits_set;
-	int cpus_found = 0;
-	int cpu;
 	int apicid = cpu_to_logical_apicid(0);
 	cpumask_var_t cpumask;
 
@@ -229,31 +226,8 @@ static inline unsigned int cpu_mask_to_apicid_and(const struct cpumask *inmask,
 
 	cpumask_and(cpumask, inmask, andmask);
 	cpumask_and(cpumask, cpumask, cpu_online_mask);
+	apicid = cpu_mask_to_apicid(cpumask);
 
-	num_bits_set = cpumask_weight(cpumask);
-	/* Return id to all */
-	if (num_bits_set == NR_CPUS)
-		goto exit;
-	/*
-	 * The cpus in the mask must all be on the apic cluster.  If are not
-	 * on the same apicid cluster return default value of TARGET_CPUS.
-	 */
-	cpu = cpumask_first(cpumask);
-	apicid = cpu_to_logical_apicid(cpu);
-	while (cpus_found < num_bits_set) {
-		if (cpumask_test_cpu(cpu, cpumask)) {
-			int new_apicid = cpu_to_logical_apicid(cpu);
-			if (apicid_cluster(apicid) !=
-					apicid_cluster(new_apicid)){
-				printk ("%s: Not a valid mask!\n", __func__);
-				return cpu_to_logical_apicid(0);
-			}
-			apicid = new_apicid;
-			cpus_found++;
-		}
-		cpu++;
-	}
-exit:
 	free_cpumask_var(cpumask);
 	return apicid;
 }
arch/x86/include/asm/lguest.h +1 -1
index d28a507..1caf576 100644
@@ -15,7 +15,7 @@
 #define SHARED_SWITCHER_PAGES \
 	DIV_ROUND_UP(end_switcher_text - start_switcher_text, PAGE_SIZE)
 /* Pages for switcher itself, then two pages per cpu */
-#define TOTAL_SWITCHER_PAGES (SHARED_SWITCHER_PAGES + 2 * NR_CPUS)
+#define TOTAL_SWITCHER_PAGES (SHARED_SWITCHER_PAGES + 2 * nr_cpu_ids)
 
 /* We map at -4M for ease of mapping into the guest (one PTE page). */
 #define SWITCHER_ADDR 0xFFC00000
arch/x86/include/asm/numaq/apic.h +2 -2
index c80f00d..bf37bc4 100644
@@ -63,8 +63,8 @@ static inline physid_mask_t ioapic_phys_id_map(physid_mask_t phys_map)
 extern u8 cpu_2_logical_apicid[];
 static inline int cpu_to_logical_apicid(int cpu)
 {
-       if (cpu >= NR_CPUS)
-	       return BAD_APICID;
+	if (cpu >= nr_cpu_ids)
+		return BAD_APICID;
 	return (int)cpu_2_logical_apicid[cpu];
 }
 
arch/x86/include/asm/pci.h +8 -2
index 66834c4..a977de2 100644
@@ -102,9 +102,9 @@ extern void pci_iommu_alloc(void);
 
 #ifdef CONFIG_NUMA
 /* Returns the node based on pci bus */
-static inline int __pcibus_to_node(struct pci_bus *bus)
+static inline int __pcibus_to_node(const struct pci_bus *bus)
 {
-	struct pci_sysdata *sd = bus->sysdata;
+	const struct pci_sysdata *sd = bus->sysdata;
 
 	return sd->node;
 }
@@ -113,6 +113,12 @@ static inline cpumask_t __pcibus_to_cpumask(struct pci_bus *bus)
 {
 	return node_to_cpumask(__pcibus_to_node(bus));
 }
+
+static inline const struct cpumask *
+cpumask_of_pcibus(const struct pci_bus *bus)
+{
+	return cpumask_of_node(__pcibus_to_node(bus));
+}
 #endif
 
 #endif /* _ASM_X86_PCI_H */
arch/x86/include/asm/summit/apic.h +8 -34

79 lines not on this page · raw message

arch/x86/include/asm/topology.h +24 -12
index 79e31e9..4e2f2e0 100644
@@ -61,13 +61,19 @@ static inline int cpu_to_node(int cpu)
  *
  * Side note: this function creates the returned cpumask on the stack
  * so with a high NR_CPUS count, excessive stack space is used.  The
- * node_to_cpumask_ptr function should be used whenever possible.
+ * cpumask_of_node function should be used whenever possible.
  */
 static inline cpumask_t node_to_cpumask(int node)
 {
 	return node_to_cpumask_map[node];
 }
 
+/* Returns a bitmask of CPUs on Node 'node'. */
+static inline const struct cpumask *cpumask_of_node(int node)
+{
+	return &node_to_cpumask_map[node];
+}
+
 #else /* CONFIG_X86_64 */
 
 /* Mappings between node number and cpus on that node. */
@@ -82,7 +88,7 @@ DECLARE_EARLY_PER_CPU(int, x86_cpu_to_node_map);
 #ifdef CONFIG_DEBUG_PER_CPU_MAPS
 extern int cpu_to_node(int cpu);
 extern int early_cpu_to_node(int cpu);
-extern const cpumask_t *_node_to_cpumask_ptr(int node);
+extern const cpumask_t *cpumask_of_node(int node);
 extern cpumask_t node_to_cpumask(int node);
 
 #else	/* !CONFIG_DEBUG_PER_CPU_MAPS */
@@ -103,7 +109,7 @@ static inline int early_cpu_to_node(int cpu)
 }
 
 /* Returns a pointer to the cpumask of CPUs on Node 'node'. */
-static inline const cpumask_t *_node_to_cpumask_ptr(int node)
+static inline const cpumask_t *cpumask_of_node(int node)
 {
 	return &node_to_cpumask_map[node];
 }
@@ -116,12 +122,15 @@ static inline cpumask_t node_to_cpumask(int node)
 
 #endif /* !CONFIG_DEBUG_PER_CPU_MAPS */
 
-/* Replace default node_to_cpumask_ptr with optimized version */
+/*
+ * Replace default node_to_cpumask_ptr with optimized version
+ * Deprecated: use "const struct cpumask *mask = cpumask_of_node(node)"
+ */
 #define node_to_cpumask_ptr(v, node)		\
-		const cpumask_t *v = _node_to_cpumask_ptr(node)
+		const cpumask_t *v = cpumask_of_node(node)
 
 #define node_to_cpumask_ptr_next(v, node)	\
-			   v = _node_to_cpumask_ptr(node)
+			   v = cpumask_of_node(node)
 
 #endif /* CONFIG_X86_64 */
 
@@ -187,7 +196,7 @@ extern int __node_distance(int, int);
 #define	cpu_to_node(cpu)	0
 #define	early_cpu_to_node(cpu)	0
 
-static inline const cpumask_t *_node_to_cpumask_ptr(int node)
+static inline const cpumask_t *cpumask_of_node(int node)
 {
 	return &cpu_online_map;
 }
@@ -200,12 +209,15 @@ static inline int node_to_first_cpu(int node)
 	return first_cpu(cpu_online_map);
 }
 
-/* Replace default node_to_cpumask_ptr with optimized version */
+/*
+ * Replace default node_to_cpumask_ptr with optimized version
+ * Deprecated: use "const struct cpumask *mask = cpumask_of_node(node)"
+ */
 #define node_to_cpumask_ptr(v, node)		\
-		const cpumask_t *v = _node_to_cpumask_ptr(node)
+		const cpumask_t *v = cpumask_of_node(node)
 
 #define node_to_cpumask_ptr_next(v, node)	\
-			   v = _node_to_cpumask_ptr(node)
+			   v = cpumask_of_node(node)
 #endif
 
 #include <asm-generic/topology.h>
@@ -214,12 +226,12 @@ static inline int node_to_first_cpu(int node)
 /* Returns the number of the first CPU on Node 'node'. */
 static inline int node_to_first_cpu(int node)
 {
-	node_to_cpumask_ptr(mask, node);
-	return first_cpu(*mask);
+	return cpumask_first(cpumask_of_node(node));
 }
 #endif
 
 extern cpumask_t cpu_coregroup_map(int cpu);
+extern const struct cpumask *cpu_coregroup_mask(int cpu);
 
 #ifdef ENABLE_TOPO_DEFINES
 #define topology_physical_package_id(cpu)	(cpu_data(cpu).phys_proc_id)
arch/x86/kernel/acpi/boot.c +23 -8
index 65d0b72..29dc0c8 100644
@@ -538,9 +538,10 @@ static int __cpuinit _acpi_map_lsapic(acpi_handle handle, int *pcpu)
 	struct acpi_buffer buffer = { ACPI_ALLOCATE_BUFFER, NULL };
 	union acpi_object *obj;
 	struct acpi_madt_local_apic *lapic;
-	cpumask_t tmp_map, new_map;
+	cpumask_var_t tmp_map, new_map;
 	u8 physid;
 	int cpu;
+	int retval = -ENOMEM;
 
 	if (ACPI_FAILURE(acpi_evaluate_object(handle, "_MAT", NULL, &buffer)))
 		return -EINVAL;
@@ -569,23 +570,37 @@ static int __cpuinit _acpi_map_lsapic(acpi_handle handle, int *pcpu)
 	buffer.length = ACPI_ALLOCATE_BUFFER;
 	buffer.pointer = NULL;
 
-	tmp_map = cpu_present_map;
+	if (!alloc_cpumask_var(&tmp_map, GFP_KERNEL))
+		goto out;
+
+	if (!alloc_cpumask_var(&new_map, GFP_KERNEL))
+		goto free_tmp_map;
+
+	cpumask_copy(tmp_map, cpu_present_mask);
 	acpi_register_lapic(physid, lapic->lapic_flags & ACPI_MADT_ENABLED);
 
 	/*
 	 * If mp_register_lapic successfully generates a new logical cpu
 	 * number, then the following will get us exactly what was mapped
 	 */
-	cpus_andnot(new_map, cpu_present_map, tmp_map);
-	if (cpus_empty(new_map)) {
+	cpumask_andnot(new_map, cpu_present_mask, tmp_map);
+	if (cpumask_empty(new_map)) {
 		printk ("Unable to map lapic to logical cpu number\n");
-		return -EINVAL;
+		retval = -EINVAL;
+		goto free_new_map;
 	}
 
-	cpu = first_cpu(new_map);
+	cpu = cpumask_first(new_map);
 
 	*pcpu = cpu;
-	return 0;
+	retval = 0;
+
+free_new_map:
+	free_cpumask_var(new_map);
+free_tmp_map:
+	free_cpumask_var(tmp_map);
+out:
+	return retval;
 }
 
 /* wrapper to silence section mismatch warning */
@@ -598,7 +613,7 @@ EXPORT_SYMBOL(acpi_map_lsapic);
 int acpi_unmap_lsapic(int cpu)
 {
 	per_cpu(x86_cpu_to_apicid, cpu) = -1;
-	cpu_clear(cpu, cpu_present_map);
+	set_cpu_present(cpu, false);
 	num_processors--;
 
 	return (0);
arch/x86/kernel/apic.c +2 -2
index 6b7f824..9958924 100644
@@ -140,7 +140,7 @@ static int lapic_next_event(unsigned long delta,
 			    struct clock_event_device *evt);
 static void lapic_timer_setup(enum clock_event_mode mode,
 			      struct clock_event_device *evt);
-static void lapic_timer_broadcast(const cpumask_t *mask);
+static void lapic_timer_broadcast(const struct cpumask *mask);
 static void apic_pm_activate(void);
 
 /*
@@ -453,7 +453,7 @@ static void lapic_timer_setup(enum clock_event_mode mode,
 /*
  * Local APIC timer broadcast function
  */
-static void lapic_timer_broadcast(const cpumask_t *mask)
+static void lapic_timer_broadcast(const struct cpumask *mask)
 {
 #ifdef CONFIG_SMP
 	send_IPI_mask(mask, LOCAL_TIMER_VECTOR);
arch/x86/kernel/cpu/common.c +1 -1
index 42e0853..3f95a40 100644
@@ -355,7 +355,7 @@ void __cpuinit detect_ht(struct cpuinfo_x86 *c)
 		printk(KERN_INFO  "CPU: Hyper-Threading is disabled\n");
 	} else if (smp_num_siblings > 1) {
 
-		if (smp_num_siblings > NR_CPUS) {
+		if (smp_num_siblings > nr_cpu_ids) {
 			printk(KERN_WARNING "CPU: Unsupported number of siblings %d",
 					smp_num_siblings);
 			smp_num_siblings = 1;
arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c +25 -3
index 88ea02d..28102ad 100644
@@ -517,6 +517,17 @@ acpi_cpufreq_guess_freq(struct acpi_cpufreq_data *data, unsigned int cpu)
 	}
 }
 
+static void free_acpi_perf_data(void)
+{
+	unsigned int i;
+
+	/* Freeing a NULL pointer is OK, and alloc_percpu zeroes. */
+	for_each_possible_cpu(i)
+		free_cpumask_var(per_cpu_ptr(acpi_perf_data, i)
+				 ->shared_cpu_map);
+	free_percpu(acpi_perf_data);
+}
+
 /*
  * acpi_cpufreq_early_init - initialize ACPI P-States library
  *
@@ -527,6 +538,7 @@ acpi_cpufreq_guess_freq(struct acpi_cpufreq_data *data, unsigned int cpu)
  */
 static int __init acpi_cpufreq_early_init(void)
 {
+	unsigned int i;
 	dprintk("acpi_cpufreq_early_init\n");
 
 	acpi_perf_data = alloc_percpu(struct acpi_processor_performance);
@@ -534,6 +546,16 @@ static int __init acpi_cpufreq_early_init(void)
 		dprintk("Memory allocation error for acpi_perf_data.\n");
 		return -ENOMEM;
 	}
+	for_each_possible_cpu(i) {
+		if (!alloc_cpumask_var_node(
+			&per_cpu_ptr(acpi_perf_data, i)->shared_cpu_map,
+			GFP_KERNEL, cpu_to_node(i))) {
+
+			/* Freeing a NULL pointer is OK: alloc_percpu zeroes. */
+			free_acpi_perf_data();
+			return -ENOMEM;
+		}
+	}
 
 	/* Do initialization in ACPI core */
 	acpi_processor_preregister_performance(acpi_perf_data);
@@ -604,9 +626,9 @@ static int acpi_cpufreq_cpu_init(struct cpufreq_policy *policy)
 	 */
 	if (policy->shared_type == CPUFREQ_SHARED_TYPE_ALL ||
 	    policy->shared_type == CPUFREQ_SHARED_TYPE_ANY) {
-		policy->cpus = perf->shared_cpu_map;
+		cpumask_copy(&policy->cpus, perf->shared_cpu_map);
 	}
-	policy->related_cpus = perf->shared_cpu_map;
+	cpumask_copy(&policy->related_cpus, perf->shared_cpu_map);
 
 #ifdef CONFIG_SMP
 	dmi_check_system(sw_any_bug_dmi_table);
@@ -795,7 +817,7 @@ static int __init acpi_cpufreq_init(void)
 
 	ret = cpufreq_register_driver(&acpi_cpufreq_driver);
 	if (ret)
-		free_percpu(acpi_perf_data);
+		free_acpi_perf_data();
 
 	return ret;
 }
arch/x86/kernel/cpu/cpufreq/powernow-k7.c +9 -0
index 7c7d56b..1b446d7 100644
@@ -310,6 +310,12 @@ static int powernow_acpi_init(void)
 		goto err0;
 	}
 
+	if (!alloc_cpumask_var(&acpi_processor_perf->shared_cpu_map,
+								GFP_KERNEL)) {
+		retval = -ENOMEM;
+		goto err05;
+	}
+
 	if (acpi_processor_register_performance(acpi_processor_perf, 0)) {
 		retval = -EIO;
 		goto err1;
@@ -412,6 +418,8 @@ static int powernow_acpi_init(void)
 err2:
 	acpi_processor_unregister_performance(acpi_processor_perf, 0);
 err1:
+	free_cpumask_var(acpi_processor_perf->shared_cpu_map);
+err05:
 	kfree(acpi_processor_perf);
 err0:
 	printk(KERN_WARNING PFX "ACPI perflib can not be used in this platform\n");
@@ -652,6 +660,7 @@ static int powernow_cpu_exit (struct cpufreq_policy *policy) {
 #ifdef CONFIG_X86_POWERNOW_K7_ACPI
 	if (acpi_processor_perf) {
 		acpi_processor_unregister_performance(acpi_processor_perf, 0);
+		free_cpumask_var(acpi_processor_perf->shared_cpu_map);
 		kfree(acpi_processor_perf);
 	}
 #endif
arch/x86/kernel/cpu/cpufreq/powernow-k8.c +15 -9
index 7f05f44..c3c9adb 100644
@@ -766,7 +766,7 @@ static void powernow_k8_acpi_pst_values(struct powernow_k8_data *data, unsigned
 static int powernow_k8_cpu_init_acpi(struct powernow_k8_data *data)
 {
 	struct cpufreq_frequency_table *powernow_table;
-	int ret_val;
+	int ret_val = -ENODEV;
 
 	if (acpi_processor_register_performance(&data->acpi_data, data->cpu)) {
 		dprintk("register performance failed: bad ACPI data\n");
@@ -815,6 +815,13 @@ static int powernow_k8_cpu_init_acpi(struct powernow_k8_data *data)
 	/* notify BIOS that we exist */
 	acpi_processor_notify_smm(THIS_MODULE);
 
+	if (!alloc_cpumask_var(&data->acpi_data.shared_cpu_map, GFP_KERNEL)) {
+		printk(KERN_ERR PFX
+				"unable to alloc powernow_k8_data cpumask\n");
+		ret_val = -ENOMEM;
+		goto err_out_mem;
+	}
+
 	return 0;
 
 err_out_mem:
@@ -826,7 +833,7 @@ err_out:
 	/* data->acpi_data.state_count informs us at ->exit() whether ACPI was used */
 	data->acpi_data.state_count = 0;
 
-	return -ENODEV;
+	return ret_val;
 }
 
 static int fill_powernow_table_pstate(struct powernow_k8_data *data, struct cpufreq_frequency_table *powernow_table)
@@ -929,6 +936,7 @@ static void powernow_k8_cpu_exit_acpi(struct powernow_k8_data *data)
 {
 	if (data->acpi_data.state_count)
 		acpi_processor_unregister_performance(&data->acpi_data, data->cpu);
+	free_cpumask_var(data->acpi_data.shared_cpu_map);
 }
 
 #else
@@ -1134,7 +1142,8 @@ static int __cpuinit powernowk8_cpu_init(struct cpufreq_policy *pol)
 	data->cpu = pol->cpu;
 	data->currpstate = HW_PSTATE_INVALID;
 
-	if (powernow_k8_cpu_init_acpi(data)) {
+	rc = powernow_k8_cpu_init_acpi(data);
+	if (rc) {
 		/*
 		 * Use the PSB BIOS structure. This is only availabe on
 		 * an UP version, and is deprecated by AMD.
@@ -1152,20 +1161,17 @@ static int __cpuinit powernowk8_cpu_init(struct cpufreq_policy *pol)
 			       "ACPI maintainers and complain to your BIOS "
 			       "vendor.\n");
 #endif
-			kfree(data);
-			return -ENODEV;
+			goto err_out;
 		}
 		if (pol->cpu != 0) {
 			printk(KERN_ERR FW_BUG PFX "No ACPI _PSS objects for "
 			       "CPU other than CPU0. Complain to your BIOS "
 			       "vendor.\n");
-			kfree(data);
-			return -ENODEV;
+			goto err_out;
 		}
 		rc = find_psb_table(data);
 		if (rc) {
-			kfree(data);
-			return -ENODEV;
+			goto err_out;
 		}
 	}
 
arch/x86/kernel/cpu/intel_cacheinfo.c +1 -1
index c6ecda6..48533d7 100644
@@ -534,7 +534,7 @@ static void __cpuinit free_cache_attributes(unsigned int cpu)
 	per_cpu(cpuid4_info, cpu) = NULL;
 }
 
-static void get_cpu_leaves(void *_retval)
+static void __cpuinit get_cpu_leaves(void *_retval)
 {
 	int j, *retval = _retval, cpu = smp_processor_id();
 
arch/x86/kernel/cpuid.c +1 -1
index 72cefd1..62a3c23 100644
@@ -121,7 +121,7 @@ static int cpuid_open(struct inode *inode, struct file *file)
 	lock_kernel();
 
 	cpu = iminor(file->f_path.dentry->d_inode);
-	if (cpu >= NR_CPUS || !cpu_online(cpu)) {
+	if (cpu >= nr_cpu_ids || !cpu_online(cpu)) {
 		ret = -ENXIO;	/* No such CPU */
 		goto out;
 	}
arch/x86/kernel/io_apic.c +3 -3
index 62ecfc9..37715bb 100644
@@ -214,11 +214,11 @@ static struct irq_cfg *get_one_free_irq_cfg(int cpu)
 
 	cfg = kzalloc_node(sizeof(*cfg), GFP_ATOMIC, node);
 	if (cfg) {
-		/* FIXME: needs alloc_cpumask_var_node() */
-		if (!alloc_cpumask_var(&cfg->domain, GFP_ATOMIC)) {
+		if (!alloc_cpumask_var_node(&cfg->domain, GFP_ATOMIC, node)) {
 			kfree(cfg);
 			cfg = NULL;
-		} else if (!alloc_cpumask_var(&cfg->old_domain, GFP_ATOMIC)) {
+		} else if (!alloc_cpumask_var_node(&cfg->old_domain,
+							  GFP_ATOMIC, node)) {
 			free_cpumask_var(cfg->domain);
 			kfree(cfg);
 			cfg = NULL;
arch/x86/kernel/msr.c +1 -1
index 82a7c7e..7262666 100644
@@ -136,7 +136,7 @@ static int msr_open(struct inode *inode, struct file *file)
 	lock_kernel();
 	cpu = iminor(file->f_path.dentry->d_inode);
 
-	if (cpu >= NR_CPUS || !cpu_online(cpu)) {
+	if (cpu >= nr_cpu_ids || !cpu_online(cpu)) {
 		ret = -ENXIO;	/* No such CPU */
 		goto out;
 	}
arch/x86/kernel/reboot.c +2 -2
index 39643b1..bc1535e 100644
@@ -501,7 +501,7 @@ void native_machine_shutdown(void)
 
 #ifdef CONFIG_X86_32
 	/* See if there has been given a command line override */
-	if ((reboot_cpu != -1) && (reboot_cpu < NR_CPUS) &&
+	if ((reboot_cpu != -1) && (reboot_cpu < nr_cpu_ids) &&
 		cpu_online(reboot_cpu))
 		reboot_cpu_id = reboot_cpu;
 #endif
@@ -511,7 +511,7 @@ void native_machine_shutdown(void)
 		reboot_cpu_id = smp_processor_id();
 
 	/* Make certain I only run on the appropriate processor */
-	set_cpus_allowed_ptr(current, &cpumask_of_cpu(reboot_cpu_id));
+	set_cpus_allowed_ptr(current, cpumask_of(reboot_cpu_id));
 
 	/* O.K Now that I'm on the appropriate processor,
 	 * stop all of the others.
arch/x86/kernel/setup_percpu.c +12 -21
index 0b63b08..a4b619c 100644
@@ -153,12 +153,10 @@ void __init setup_per_cpu_areas(void)
 	align = max_t(unsigned long, PAGE_SIZE, align);
 	size = roundup(old_size, align);
 
-	printk(KERN_INFO
-		"NR_CPUS:%d nr_cpumask_bits:%d nr_cpu_ids:%d nr_node_ids:%d\n",
+	pr_info("NR_CPUS:%d nr_cpumask_bits:%d nr_cpu_ids:%d nr_node_ids:%d\n",
 		NR_CPUS, nr_cpumask_bits, nr_cpu_ids, nr_node_ids);
 
-	printk(KERN_INFO "PERCPU: Allocating %zd bytes of per cpu data\n",
-			  size);
+	pr_info("PERCPU: Allocating %zd bytes of per cpu data\n", size);
 
 	for_each_possible_cpu(cpu) {
 #ifndef CONFIG_NEED_MULTIPLE_NODES
@@ -169,22 +167,15 @@ void __init setup_per_cpu_areas(void)
 		if (!node_online(node) || !NODE_DATA(node)) {
 			ptr = __alloc_bootmem(size, align,
 					 __pa(MAX_DMA_ADDRESS));
-			printk(KERN_INFO
-			       "cpu %d has no node %d or node-local memory\n",
+			pr_info("cpu %d has no node %d or node-local memory\n",
 				cpu, node);
-			if (ptr)
-				printk(KERN_DEBUG
-					"per cpu data for cpu%d at %016lx\n",
-					 cpu, __pa(ptr));
-		}
-		else {
+			pr_debug("per cpu data for cpu%d at %016lx\n",
+				 cpu, __pa(ptr));
+		} else {
 			ptr = __alloc_bootmem_node(NODE_DATA(node), size, align,
 							__pa(MAX_DMA_ADDRESS));
-			if (ptr)
-				printk(KERN_DEBUG
-					"per cpu data for cpu%d on node%d "
-					"at %016lx\n",
-					cpu, node, __pa(ptr));
+			pr_debug("per cpu data for cpu%d on node%d at %016lx\n",
+				cpu, node, __pa(ptr));
 		}
 #endif
 		per_cpu_offset(cpu) = ptr - __per_cpu_start;
@@ -339,25 +330,25 @@ static const cpumask_t cpu_mask_none;
 /*
  * Returns a pointer to the bitmask of CPUs on Node 'node'.
  */
-const cpumask_t *_node_to_cpumask_ptr(int node)
+const cpumask_t *cpumask_of_node(int node)
 {
 	if (node_to_cpumask_map == NULL) {
 		printk(KERN_WARNING
-			"_node_to_cpumask_ptr(%d): no node_to_cpumask_map!\n",
+			"cpumask_of_node(%d): no node_to_cpumask_map!\n",
 			node);
 		dump_stack();
 		return (const cpumask_t *)&cpu_online_map;
 	}
 	if (node >= nr_node_ids) {
 		printk(KERN_WARNING
-			"_node_to_cpumask_ptr(%d): node > nr_node_ids(%d)\n",
+			"cpumask_of_node(%d): node > nr_node_ids(%d)\n",
 			node, nr_node_ids);
 		dump_stack();
 		return &cpu_mask_none;
 	}
 	return &node_to_cpumask_map[node];
 }
-EXPORT_SYMBOL(_node_to_cpumask_ptr);
+EXPORT_SYMBOL(cpumask_of_node);
 
 /*
  * Returns a bitmask of CPUs on Node 'node'.
arch/x86/kernel/smpboot.c +11 -4
index 31869bf..6bd4d9b 100644
@@ -496,7 +496,7 @@ void __cpuinit set_cpu_sibling_map(int cpu)
 }
 
 /* maps the cpu to the sched domain representing multi-core */
-cpumask_t cpu_coregroup_map(int cpu)
+const struct cpumask *cpu_coregroup_mask(int cpu)
 {
 	struct cpuinfo_x86 *c = &cpu_data(cpu);
 	/*
@@ -504,9 +504,14 @@ cpumask_t cpu_coregroup_map(int cpu)
 	 * And for power savings, we return cpu_core_map
 	 */
 	if (sched_mc_power_savings || sched_smt_power_savings)
-		return per_cpu(cpu_core_map, cpu);
+		return &per_cpu(cpu_core_map, cpu);
 	else
-		return c->llc_shared_map;
+		return &c->llc_shared_map;
+}
+
+cpumask_t cpu_coregroup_map(int cpu)
+{
+	return *cpu_coregroup_mask(cpu);
 }
 
 static void impress_friends(void)
@@ -1149,7 +1154,7 @@ static void __init smp_cpu_index_default(void)
 	for_each_possible_cpu(i) {
 		c = &cpu_data(i);
 		/* mark all to hotplug */
-		c->cpu_index = NR_CPUS;
+		c->cpu_index = nr_cpu_ids;
 	}
 }
 
@@ -1293,6 +1298,8 @@ __init void prefill_possible_map(void)
 	else
 		possible = setup_possible_cpus;
 
+	total_cpus = max_t(int, possible, num_processors + disabled_cpus);
+
 	if (possible > CONFIG_NR_CPUS) {
 		printk(KERN_WARNING
 			"%d Processors exceeds NR_CPUS limit of %d\n",
arch/x86/mach-voyager/voyager_smp.c +3 -4
index a5bc054..9840b7e 100644
@@ -357,9 +357,8 @@ void __init find_smp_config(void)
 	printk("VOYAGER SMP: Boot cpu is %d\n", boot_cpu_id);
 
 	/* initialize the CPU structures (moved from smp_boot_cpus) */
-	for (i = 0; i < NR_CPUS; i++) {
+	for (i = 0; i < nr_cpu_ids; i++)
 		cpu_irq_affinity[i] = ~0;
-	}
 	cpu_online_map = cpumask_of_cpu(boot_cpu_id);
 
 	/* The boot CPU must be extended */
@@ -1227,7 +1226,7 @@ int setup_profiling_timer(unsigned int multiplier)
 	 * new values until the next timer interrupt in which they do process
 	 * accounting.
 	 */
-	for (i = 0; i < NR_CPUS; ++i)
+	for (i = 0; i < nr_cpu_ids; ++i)
 		per_cpu(prof_multiplier, i) = multiplier;
 
 	return 0;
@@ -1257,7 +1256,7 @@ void __init voyager_smp_intr_init(void)
 	int i;
 
 	/* initialize the per cpu irq mask to all disabled */
-	for (i = 0; i < NR_CPUS; i++)
+	for (i = 0; i < nr_cpu_ids; i++)
 		vic_irq_mask[i] = 0xFFFF;
 
 	VIC_SET_GATE(VIC_CPI_LEVEL0, vic_cpi_interrupt);
block/blk.h +2 -2
index d2e49af..6e1ed40 100644
@@ -99,8 +99,8 @@ static inline int queue_congestion_off_threshold(struct request_queue *q)
 static inline int blk_cpu_to_group(int cpu)
 {
 #ifdef CONFIG_SCHED_MC
-	cpumask_t mask = cpu_coregroup_map(cpu);
-	return first_cpu(mask);
+	const struct cpumask *mask = cpu_coregroup_mask(cpu);
+	return cpumask_first(mask);
 #elif defined(CONFIG_SCHED_SMT)
 	return first_cpu(per_cpu(cpu_sibling_map, cpu));
 #else
drivers/acpi/processor_core.c +10 -4
index 3494836..0cc2fd3 100644
@@ -826,6 +826,11 @@ static int acpi_processor_add(struct acpi_device *device)
 	if (!pr)
 		return -ENOMEM;
 
+	if (!alloc_cpumask_var(&pr->throttling.shared_cpu_map, GFP_KERNEL)) {
+		kfree(pr);
+		return -ENOMEM;
+	}
+
 	pr->handle = device->handle;
 	strcpy(acpi_device_name(device), ACPI_PROCESSOR_DEVICE_NAME);
 	strcpy(acpi_device_class(device), ACPI_PROCESSOR_CLASS);
@@ -845,10 +850,8 @@ static int acpi_processor_remove(struct acpi_device *device, int type)
 
 	pr = acpi_driver_data(device);
 
-	if (pr->id >= nr_cpu_ids) {
-		kfree(pr);
-		return 0;
-	}
+	if (pr->id >= nr_cpu_ids)
+		goto free;
 
 	if (type == ACPI_BUS_REMOVAL_EJECT) {
 		if (acpi_processor_handle_eject(pr))
@@ -873,6 +876,9 @@ static int acpi_processor_remove(struct acpi_device *device, int type)
 
 	per_cpu(processors, pr->id) = NULL;
 	per_cpu(processor_device_array, pr->id) = NULL;
+
+free:
+	free_cpumask_var(pr->throttling.shared_cpu_map);
 	kfree(pr);
 
 	return 0;
drivers/acpi/processor_perflib.c +16 -12
index 0d7b772..846e227 100644
@@ -588,12 +588,15 @@ int acpi_processor_preregister_performance(
 	int count, count_target;
 	int retval = 0;
 	unsigned int i, j;
-	cpumask_t covered_cpus;
+	cpumask_var_t covered_cpus;
 	struct acpi_processor *pr;
 	struct acpi_psd_package *pdomain;
 	struct acpi_processor *match_pr;
 	struct acpi_psd_package *match_pdomain;
 
+	if (!alloc_cpumask_var(&covered_cpus, GFP_KERNEL))
+		return -ENOMEM;
+
 	mutex_lock(&performance_mutex);
 
 	retval = 0;
@@ -617,7 +620,7 @@ int acpi_processor_preregister_performance(
 		}
 
 		pr->performance = percpu_ptr(performance, i);
-		cpu_set(i, pr->performance->shared_cpu_map);
+		cpumask_set_cpu(i, pr->performance->shared_cpu_map);
 		if (acpi_processor_get_psd(pr)) {
 			retval = -EINVAL;
 			continue;
@@ -650,18 +653,18 @@ int acpi_processor_preregister_performance(
 		}
 	}
 
-	cpus_clear(covered_cpus);
+	cpumask_clear(covered_cpus);
 	for_each_possible_cpu(i) {
 		pr = per_cpu(processors, i);
 		if (!pr)
 			continue;
 
-		if (cpu_isset(i, covered_cpus))
+		if (cpumask_test_cpu(i, covered_cpus))
 			continue;
 
 		pdomain = &(pr->performance->domain_info);
-		cpu_set(i, pr->performance->shared_cpu_map);
-		cpu_set(i, covered_cpus);
+		cpumask_set_cpu(i, pr->performance->shared_cpu_map);
+		cpumask_set_cpu(i, covered_cpus);
 		if (pdomain->num_processors <= 1)
 			continue;
 
@@ -699,8 +702,8 @@ int acpi_processor_preregister_performance(
 				goto err_ret;
 			}
 
-			cpu_set(j, covered_cpus);
-			cpu_set(j, pr->performance->shared_cpu_map);
+			cpumask_set_cpu(j, covered_cpus);
+			cpumask_set_cpu(j, pr->performance->shared_cpu_map);
 			count++;
 		}
 
@@ -718,8 +721,8 @@ int acpi_processor_preregister_performance(
 
 			match_pr->performance->shared_type = 
 					pr->performance->shared_type;
-			match_pr->performance->shared_cpu_map =
-				pr->performance->shared_cpu_map;
+			cpumask_copy(match_pr->performance->shared_cpu_map,
+				     pr->performance->shared_cpu_map);
 		}
 	}
 
@@ -731,14 +734,15 @@ err_ret:
 
 		/* Assume no coordination on any error parsing domain info */
 		if (retval) {
-			cpus_clear(pr->performance->shared_cpu_map);
-			cpu_set(i, pr->performance->shared_cpu_map);
+			cpumask_clear(pr->performance->shared_cpu_map);
+			cpumask_set_cpu(i, pr->performance->shared_cpu_map);
 			pr->performance->shared_type = CPUFREQ_SHARED_TYPE_ALL;
 		}
 		pr->performance = NULL; /* Will be set for real in register */
 	}
 
 	mutex_unlock(&performance_mutex);
+	free_cpumask_var(covered_cpus);
 	return retval;
 }
 EXPORT_SYMBOL(acpi_processor_preregister_performance);
drivers/acpi/processor_throttling.c +52 -28

209 lines not on this page · raw message

drivers/base/cpu.c +44 -0

54 lines not on this page · raw message

drivers/infiniband/hw/ehca/ehca_irq.c +8 -9
index 757035e..3128a50 100644
@@ -659,12 +659,12 @@ static inline int find_next_online_cpu(struct ehca_comp_pool *pool)
 
 	WARN_ON_ONCE(!in_interrupt());
 	if (ehca_debug_level >= 3)
-		ehca_dmp(&cpu_online_map, sizeof(cpumask_t), "");
+		ehca_dmp(cpu_online_mask, cpumask_size(), "");
 
 	spin_lock_irqsave(&pool->last_cpu_lock, flags);
-	cpu = next_cpu_nr(pool->last_cpu, cpu_online_map);
+	cpu = cpumask_next(pool->last_cpu, cpu_online_mask);
 	if (cpu >= nr_cpu_ids)
-		cpu = first_cpu(cpu_online_map);
+		cpu = cpumask_first(cpu_online_mask);
 	pool->last_cpu = cpu;
 	spin_unlock_irqrestore(&pool->last_cpu_lock, flags);
 
@@ -855,7 +855,7 @@ static int __cpuinit comp_pool_callback(struct notifier_block *nfb,
 	case CPU_UP_CANCELED_FROZEN:
 		ehca_gen_dbg("CPU: %x (CPU_CANCELED)", cpu);
 		cct = per_cpu_ptr(pool->cpu_comp_tasks, cpu);
-		kthread_bind(cct->task, any_online_cpu(cpu_online_map));
+		kthread_bind(cct->task, cpumask_any(cpu_online_mask));
 		destroy_comp_task(pool, cpu);
 		break;
 	case CPU_ONLINE:
@@ -902,7 +902,7 @@ int ehca_create_comp_pool(void)
 		return -ENOMEM;
 
 	spin_lock_init(&pool->last_cpu_lock);
-	pool->last_cpu = any_online_cpu(cpu_online_map);
+	pool->last_cpu = cpumask_any(cpu_online_mask);
 
 	pool->cpu_comp_tasks = alloc_percpu(struct ehca_cpu_comp_task);
 	if (pool->cpu_comp_tasks == NULL) {
@@ -934,10 +934,9 @@ void ehca_destroy_comp_pool(void)
 
 	unregister_hotcpu_notifier(&comp_pool_callback_nb);
 
-	for (i = 0; i < NR_CPUS; i++) {
-		if (cpu_online(i))
-			destroy_comp_task(pool, i);
-	}
+	for_each_online_cpu(i)
+		destroy_comp_task(pool, i);
+
 	free_percpu(pool->cpu_comp_tasks);
 	kfree(pool);
 }
drivers/infiniband/hw/ipath/ipath_file_ops.c +4 -4
index 239d4e8..2317398 100644
@@ -1679,7 +1679,7 @@ static int find_best_unit(struct file *fp,
 	 * InfiniPath chip to that processor (we assume reasonable connectivity,
 	 * for now).  This code assumes that if affinity has been set
 	 * before this point, that at most one cpu is set; for now this
-	 * is reasonable.  I check for both cpus_empty() and cpus_full(),
+	 * is reasonable.  I check for both cpumask_empty() and cpumask_full(),
 	 * in case some kernel variant sets none of the bits when no
 	 * affinity is set.  2.6.11 and 12 kernels have all present
 	 * cpus set.  Some day we'll have to fix it up further to handle
@@ -1688,11 +1688,11 @@ static int find_best_unit(struct file *fp,
 	 * information.  There may be some issues with dual core numbering
 	 * as well.  This needs more work prior to release.
 	 */
-	if (!cpus_empty(current->cpus_allowed) &&
-	    !cpus_full(current->cpus_allowed)) {
+	if (!cpumask_empty(&current->cpus_allowed) &&
+	    !cpumask_full(&current->cpus_allowed)) {
 		int ncpus = num_online_cpus(), curcpu = -1, nset = 0;
 		for (i = 0; i < ncpus; i++)
-			if (cpu_isset(i, current->cpus_allowed)) {
+			if (cpumask_test_cpu(i, &current->cpus_allowed)) {
 				ipath_cdbg(PROC, "%s[%u] affinity set for "
 					   "cpu %d/%d\n", current->comm,
 					   current->pid, i, ncpus);
drivers/pnp/pnpbios/bioscalls.c +1 -1
index 7ff8244..7e6b5a3 100644
@@ -481,7 +481,7 @@ void pnpbios_calls_init(union pnp_bios_install_struct *header)
 
 	set_base(bad_bios_desc, __va((unsigned long)0x40 << 4));
 	_set_limit((char *)&bad_bios_desc, 4095 - (0x40 << 4));
-	for (i = 0; i < NR_CPUS; i++) {
+	for_each_possible_cpu(i) {
 		struct desc_struct *gdt = get_cpu_gdt_table(i);
 		if (!gdt)
 			continue;
fs/seq_file.c +2 -1
index 99d8b8c..b569ff1 100644
@@ -468,7 +468,8 @@ int seq_dentry(struct seq_file *m, struct dentry *dentry, char *esc)
 	return -1;
 }
 
-int seq_bitmap(struct seq_file *m, unsigned long *bits, unsigned int nr_bits)
+int seq_bitmap(struct seq_file *m, const unsigned long *bits,
+				   unsigned int nr_bits)
 {
 	if (m->count < m->size) {
 		int len = bitmap_scnprintf(m->buf + m->count,
include/acpi/processor.h +2 -2
index 3795590..0574add 100644
@@ -127,7 +127,7 @@ struct acpi_processor_performance {
 	unsigned int state_count;
 	struct acpi_processor_px *states;
 	struct acpi_psd_package domain_info;
-	cpumask_t shared_cpu_map;
+	cpumask_var_t shared_cpu_map;
 	unsigned int shared_type;
 };
 
@@ -172,7 +172,7 @@ struct acpi_processor_throttling {
 	unsigned int state_count;
 	struct acpi_processor_tx_tss *states_tss;
 	struct acpi_tsd_package domain_info;
-	cpumask_t shared_cpu_map;
+	cpumask_var_t shared_cpu_map;
 	int (*acpi_processor_get_throttling) (struct acpi_processor * pr);
 	int (*acpi_processor_set_throttling) (struct acpi_processor * pr,
 					      int state);
include/asm-frv/bitops.h +13 -0
index 39456ba..287f6f6 100644
@@ -339,6 +339,19 @@ int __ffs(unsigned long x)
 	return 31 - bit;
 }
 
+/**
+ * __fls - find last (most-significant) set bit in a long word
+ * @word: the word to search
+ *
+ * Undefined if no set bit exists, so code should check against 0 first.
+ */
+static inline unsigned long __fls(unsigned long word)
+{
+	unsigned long bit;
+	asm("scan %1,gr0,%0" : "=r"(bit) : "r"(word));
+	return bit;
+}
+
 /*
  * special slimline version of fls() for calculating ilog2_u32()
  * - note: no protection against n == 0
include/asm-m32r/bitops.h +1 -0
index 6dc9b81..aaddf0d 100644
@@ -251,6 +251,7 @@ static __inline__ int test_and_change_bit(int nr, volatile void * addr)
 #include <asm-generic/bitops/ffz.h>
 #include <asm-generic/bitops/__ffs.h>
 #include <asm-generic/bitops/fls.h>
+#include <asm-generic/bitops/__fls.h>
 #include <asm-generic/bitops/fls64.h>
 
 #ifdef __KERNEL__
include/asm-m68k/bitops.h +5 -0
index 3e81064..9bde784 100644
@@ -315,6 +315,11 @@ static inline int fls(int x)
 	return 32 - cnt;
 }
 
+static inline int __fls(int x)
+{
+	return fls(x) - 1;
+}
+
 #include <asm-generic/bitops/fls64.h>
 #include <asm-generic/bitops/sched.h>
 #include <asm-generic/bitops/hweight.h>
include/asm-mn10300/bitops.h +11 -0
index cc6d40c..0b610f4 100644
@@ -196,6 +196,17 @@ int fls(int x)
 }
 
 /**
+ * __fls - find last (most-significant) set bit in a long word
+ * @word: the word to search
+ *
+ * Undefined if no set bit exists, so code should check against 0 first.
+ */
+static inline unsigned long __fls(unsigned long word)
+{
+	return __ilog2_u32(word);
+}
+
+/**
  * ffs - find first bit set
  * @x: the word to search
  *
include/asm-xtensa/bitops.h +11 -0
index 23261e8..6c39303 100644
@@ -82,6 +82,16 @@ static inline int fls (unsigned int x)
 	return 32 - __cntlz(x);
 }
 
+/**
+ * __fls - find last (most-significant) set bit in a long word
+ * @word: the word to search
+ *
+ * Undefined if no set bit exists, so code should check against 0 first.
+ */
+static inline unsigned long __fls(unsigned long word)
+{
+	return 31 - __cntlz(word);
+}
 #else
 
 /* Use the generic implementation if we don't have the nsa/nsau instructions. */
@@ -90,6 +100,7 @@ static inline int fls (unsigned int x)
 # include <asm-generic/bitops/__ffs.h>
 # include <asm-generic/bitops/ffz.h>
 # include <asm-generic/bitops/fls.h>
+# include <asm-generic/bitops/__fls.h>
 
 #endif
 
include/linux/bitmap.h +19 -16
index a08c33a..2878811 100644
@@ -137,9 +137,12 @@ extern void bitmap_copy_le(void *dst, const unsigned long *src, int nbits);
 		(1UL<<((nbits) % BITS_PER_LONG))-1 : ~0UL		\
 )
 
+#define small_const_nbits(nbits) \
+	(__builtin_constant_p(nbits) && (nbits) <= BITS_PER_LONG)
+
 static inline void bitmap_zero(unsigned long *dst, int nbits)
 {
-	if (nbits <= BITS_PER_LONG)
+	if (small_const_nbits(nbits))
 		*dst = 0UL;
 	else {
 		int len = BITS_TO_LONGS(nbits) * sizeof(unsigned long);
@@ -150,7 +153,7 @@ static inline void bitmap_zero(unsigned long *dst, int nbits)
 static inline void bitmap_fill(unsigned long *dst, int nbits)
 {
 	size_t nlongs = BITS_TO_LONGS(nbits);
-	if (nlongs > 1) {
+	if (!small_const_nbits(nbits)) {
 		int len = (nlongs - 1) * sizeof(unsigned long);
 		memset(dst, 0xff,  len);
 	}
@@ -160,7 +163,7 @@ static inline void bitmap_fill(unsigned long *dst, int nbits)
 static inline void bitmap_copy(unsigned long *dst, const unsigned long *src,
 			int nbits)
 {
-	if (nbits <= BITS_PER_LONG)
+	if (small_const_nbits(nbits))
 		*dst = *src;
 	else {
 		int len = BITS_TO_LONGS(nbits) * sizeof(unsigned long);
@@ -171,7 +174,7 @@ static inline void bitmap_copy(unsigned long *dst, const unsigned long *src,
 static inline void bitmap_and(unsigned long *dst, const unsigned long *src1,
 			const unsigned long *src2, int nbits)
 {
-	if (nbits <= BITS_PER_LONG)
+	if (small_const_nbits(nbits))
 		*dst = *src1 & *src2;
 	else
 		__bitmap_and(dst, src1, src2, nbits);
@@ -180,7 +183,7 @@ static inline void bitmap_and(unsigned long *dst, const unsigned long *src1,
 static inline void bitmap_or(unsigned long *dst, const unsigned long *src1,
 			const unsigned long *src2, int nbits)
 {
-	if (nbits <= BITS_PER_LONG)
+	if (small_const_nbits(nbits))
 		*dst = *src1 | *src2;
 	else
 		__bitmap_or(dst, src1, src2, nbits);
@@ -189,7 +192,7 @@ static inline void bitmap_or(unsigned long *dst, const unsigned long *src1,
 static inline void bitmap_xor(unsigned long *dst, const unsigned long *src1,
 			const unsigned long *src2, int nbits)
 {
-	if (nbits <= BITS_PER_LONG)
+	if (small_const_nbits(nbits))
 		*dst = *src1 ^ *src2;
 	else
 		__bitmap_xor(dst, src1, src2, nbits);
@@ -198,7 +201,7 @@ static inline void bitmap_xor(unsigned long *dst, const unsigned long *src1,
 static inline void bitmap_andnot(unsigned long *dst, const unsigned long *src1,
 			const unsigned long *src2, int nbits)
 {
-	if (nbits <= BITS_PER_LONG)
+	if (small_const_nbits(nbits))
 		*dst = *src1 & ~(*src2);
 	else
 		__bitmap_andnot(dst, src1, src2, nbits);
@@ -207,7 +210,7 @@ static inline void bitmap_andnot(unsigned long *dst, const unsigned long *src1,
 static inline void bitmap_complement(unsigned long *dst, const unsigned long *src,
 			int nbits)
 {
-	if (nbits <= BITS_PER_LONG)
+	if (small_const_nbits(nbits))
 		*dst = ~(*src) & BITMAP_LAST_WORD_MASK(nbits);
 	else
 		__bitmap_complement(dst, src, nbits);
@@ -216,7 +219,7 @@ static inline void bitmap_complement(unsigned long *dst, const unsigned long *sr
 static inline int bitmap_equal(const unsigned long *src1,
 			const unsigned long *src2, int nbits)
 {
-	if (nbits <= BITS_PER_LONG)
+	if (small_const_nbits(nbits))
 		return ! ((*src1 ^ *src2) & BITMAP_LAST_WORD_MASK(nbits));
 	else
 		return __bitmap_equal(src1, src2, nbits);
@@ -225,7 +228,7 @@ static inline int bitmap_equal(const unsigned long *src1,
 static inline int bitmap_intersects(const unsigned long *src1,
 			const unsigned long *src2, int nbits)
 {
-	if (nbits <= BITS_PER_LONG)
+	if (small_const_nbits(nbits))
 		return ((*src1 & *src2) & BITMAP_LAST_WORD_MASK(nbits)) != 0;
 	else
 		return __bitmap_intersects(src1, src2, nbits);
@@ -234,7 +237,7 @@ static inline int bitmap_intersects(const unsigned long *src1,
 static inline int bitmap_subset(const unsigned long *src1,
 			const unsigned long *src2, int nbits)
 {
-	if (nbits <= BITS_PER_LONG)
+	if (small_const_nbits(nbits))
 		return ! ((*src1 & ~(*src2)) & BITMAP_LAST_WORD_MASK(nbits));
 	else
 		return __bitmap_subset(src1, src2, nbits);
@@ -242,7 +245,7 @@ static inline int bitmap_subset(const unsigned long *src1,
 
 static inline int bitmap_empty(const unsigned long *src, int nbits)
 {
-	if (nbits <= BITS_PER_LONG)
+	if (small_const_nbits(nbits))
 		return ! (*src & BITMAP_LAST_WORD_MASK(nbits));
 	else
 		return __bitmap_empty(src, nbits);
@@ -250,7 +253,7 @@ static inline int bitmap_empty(const unsigned long *src, int nbits)
 
 static inline int bitmap_full(const unsigned long *src, int nbits)
 {
-	if (nbits <= BITS_PER_LONG)
+	if (small_const_nbits(nbits))
 		return ! (~(*src) & BITMAP_LAST_WORD_MASK(nbits));
 	else
 		return __bitmap_full(src, nbits);
@@ -258,7 +261,7 @@ static inline int bitmap_full(const unsigned long *src, int nbits)
 
 static inline int bitmap_weight(const unsigned long *src, int nbits)
 {
-	if (nbits <= BITS_PER_LONG)
+	if (small_const_nbits(nbits))
 		return hweight_long(*src & BITMAP_LAST_WORD_MASK(nbits));
 	return __bitmap_weight(src, nbits);
 }
@@ -266,7 +269,7 @@ static inline int bitmap_weight(const unsigned long *src, int nbits)
 static inline void bitmap_shift_right(unsigned long *dst,
 			const unsigned long *src, int n, int nbits)
 {
-	if (nbits <= BITS_PER_LONG)
+	if (small_const_nbits(nbits))
 		*dst = *src >> n;
 	else
 		__bitmap_shift_right(dst, src, n, nbits);
@@ -275,7 +278,7 @@ static inline void bitmap_shift_right(unsigned long *dst,
 static inline void bitmap_shift_left(unsigned long *dst,
 			const unsigned long *src, int n, int nbits)
 {
-	if (nbits <= BITS_PER_LONG)
+	if (small_const_nbits(nbits))
 		*dst = (*src << n) & BITMAP_LAST_WORD_MASK(nbits);
 	else
 		__bitmap_shift_left(dst, src, n, nbits);
include/linux/bitops.h +12 -1
index 024f2b0..6182913 100644
@@ -134,9 +134,20 @@ extern unsigned long find_first_bit(const unsigned long *addr,
  */
 extern unsigned long find_first_zero_bit(const unsigned long *addr,
 					 unsigned long size);
-
 #endif /* CONFIG_GENERIC_FIND_FIRST_BIT */
 
+#ifdef CONFIG_GENERIC_FIND_LAST_BIT
+/**
+ * find_last_bit - find the last set bit in a memory region
+ * @addr: The address to start the search at
+ * @size: The maximum size to search
+ *
+ * Returns the bit number of the first set bit, or size.
+ */
+extern unsigned long find_last_bit(const unsigned long *addr,
+				   unsigned long size);
+#endif /* CONFIG_GENERIC_FIND_LAST_BIT */
+
 #ifdef CONFIG_GENERIC_FIND_NEXT_BIT
 
 /**
include/linux/cpumask.h +91 -130

359 lines not on this page · raw message

include/linux/interrupt.h +1 -1
index 990355f..0702c4d 100644
@@ -109,7 +109,7 @@ extern void enable_irq(unsigned int irq);
 
 #if defined(CONFIG_SMP) && defined(CONFIG_GENERIC_HARDIRQS)
 
-extern cpumask_t irq_default_affinity;
+extern cpumask_var_t irq_default_affinity;
 
 extern int irq_set_affinity(unsigned int irq, const struct cpumask *cpumask);
 extern int irq_can_set_affinity(unsigned int irq);
include/linux/rcuclassic.h +2 -2
index 301dda8..f3f697d 100644
@@ -59,8 +59,8 @@ struct rcu_ctrlblk {
 	int	signaled;
 
 	spinlock_t	lock	____cacheline_internodealigned_in_smp;
-	cpumask_t	cpumask; /* CPUs that need to switch in order    */
-				 /* for current batch to proceed.        */
+	DECLARE_BITMAP(cpumask, NR_CPUS); /* CPUs that need to switch for */
+					  /* current batch to proceed.     */
 } ____cacheline_internodealigned_in_smp;
 
 /* Is batch a before batch b ? */
include/linux/seq_file.h +4 -3
index b3dfa72..40ea505 100644
@@ -50,10 +50,11 @@ int seq_path(struct seq_file *, struct path *, char *);
 int seq_dentry(struct seq_file *, struct dentry *, char *);
 int seq_path_root(struct seq_file *m, struct path *path, struct path *root,
 		  char *esc);
-int seq_bitmap(struct seq_file *m, unsigned long *bits, unsigned int nr_bits);
-static inline int seq_cpumask(struct seq_file *m, cpumask_t *mask)
+int seq_bitmap(struct seq_file *m, const unsigned long *bits,
+				   unsigned int nr_bits);
+static inline int seq_cpumask(struct seq_file *m, const struct cpumask *mask)
 {
-	return seq_bitmap(m, mask->bits, NR_CPUS);
+	return seq_bitmap(m, mask->bits, nr_cpu_ids);
 }
 
 static inline int seq_nodemask(struct seq_file *m, nodemask_t *mask)
include/linux/smp.h +11 -7
index 6e7ba16..b824669 100644
@@ -21,6 +21,9 @@ struct call_single_data {
 	u16 priv;
 };
 
+/* total number of cpus in this system (may exceed NR_CPUS) */
+extern unsigned int total_cpus;
+
 #ifdef CONFIG_SMP
 
 #include <linux/preempt.h>
@@ -64,15 +67,16 @@ extern void smp_cpus_done(unsigned int max_cpus);
  * Call a function on all other processors
  */
 int smp_call_function(void(*func)(void *info), void *info, int wait);
-/* Deprecated: use smp_call_function_many() which uses a cpumask ptr. */
-int smp_call_function_mask(cpumask_t mask, void(*func)(void *info), void *info,
-				int wait);
+void smp_call_function_many(const struct cpumask *mask,
+			    void (*func)(void *info), void *info, bool wait);
 
-static inline void smp_call_function_many(const struct cpumask *mask,
-					  void (*func)(void *info), void *info,
-					  int wait)
+/* Deprecated: Use smp_call_function_many which takes a pointer to the mask. */
+static inline int
+smp_call_function_mask(cpumask_t mask, void(*func)(void *info), void *info,
+		       int wait)
 {
-	smp_call_function_mask(*mask, func, info, wait);
+	smp_call_function_many(&mask, func, info, wait);
+	return 0;
 }
 
 int smp_call_function_single(int cpuid, void (*func) (void *info), void *info,
include/linux/stop_machine.h +3 -3
index faf1519..74d59a6 100644
@@ -23,7 +23,7 @@
  *
  * This can be thought of as a very heavy write lock, equivalent to
  * grabbing every spinlock in the kernel. */
-int stop_machine(int (*fn)(void *), void *data, const cpumask_t *cpus);
+int stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus);
 
 /**
  * __stop_machine: freeze the machine on all CPUs and run this function
@@ -34,11 +34,11 @@ int stop_machine(int (*fn)(void *), void *data, const cpumask_t *cpus);
  * Description: This is a special version of the above, which assumes cpus
  * won't come or go while it's being called.  Used by hotplug cpu.
  */
-int __stop_machine(int (*fn)(void *), void *data, const cpumask_t *cpus);
+int __stop_machine(int (*fn)(void *), void *data, const struct cpumask *cpus);
 #else
 
 static inline int stop_machine(int (*fn)(void *), void *data,
-			       const cpumask_t *cpus)
+			       const struct cpumask *cpus)
 {
 	int ret;
 	local_irq_disable();
include/linux/threads.h +8 -8
index 38d1a5d..052b12b 100644
@@ -8,17 +8,17 @@
  */
 
 /*
- * Maximum supported processors that can run under SMP.  This value is
- * set via configure setting.  The maximum is equal to the size of the
- * bitmasks used on that platform, i.e. 32 or 64.  Setting this smaller
- * saves quite a bit of memory.
+ * Maximum supported processors.  Setting this smaller saves quite a
+ * bit of memory.  Use nr_cpu_ids instead of this except for static bitmaps.
  */
-#ifdef CONFIG_SMP
-#define NR_CPUS		CONFIG_NR_CPUS
-#else
-#define NR_CPUS		1
+#ifndef CONFIG_NR_CPUS
+/* FIXME: This should be fixed in the arch's Kconfig */
+#define CONFIG_NR_CPUS	1
 #endif
 
+/* Places which use this should consider cpumask_var_t. */
+#define NR_CPUS		CONFIG_NR_CPUS
+
 #define MIN_THREADS_LEFT_FOR_ROOT 4
 
 /*
include/linux/tick.h +2 -2
index b6ec818..469b82d 100644
@@ -84,10 +84,10 @@ static inline void tick_cancel_sched_timer(int cpu) { }
 
 # ifdef CONFIG_GENERIC_CLOCKEVENTS_BROADCAST
 extern struct tick_device *tick_get_broadcast_device(void);
-extern cpumask_t *tick_get_broadcast_mask(void);
+extern struct cpumask *tick_get_broadcast_mask(void);
 
 #  ifdef CONFIG_TICK_ONESHOT
-extern cpumask_t *tick_get_broadcast_oneshot_mask(void);
+extern struct cpumask *tick_get_broadcast_oneshot_mask(void);
 #  endif
 
 # endif /* BROADCAST */
init/main.c +4 -9
index ad8f9f5..cd168eb 100644
@@ -371,12 +371,7 @@ EXPORT_SYMBOL(nr_cpu_ids);
 /* An arch may set nr_cpu_ids earlier if needed, so this would be redundant */
 static void __init setup_nr_cpu_ids(void)
 {
-	int cpu, highest_cpu = 0;
-
-	for_each_possible_cpu(cpu)
-		highest_cpu = cpu;
-
-	nr_cpu_ids = highest_cpu + 1;
+	nr_cpu_ids = find_last_bit(cpumask_bits(cpu_possible_mask),NR_CPUS) + 1;
 }
 
 #ifndef CONFIG_HAVE_SETUP_PER_CPU_AREA
@@ -518,9 +513,9 @@ static void __init boot_cpu_init(void)
 {
 	int cpu = smp_processor_id();
 	/* Mark the boot cpu "present", "online" etc for SMP and UP case */
-	cpu_set(cpu, cpu_online_map);
-	cpu_set(cpu, cpu_present_map);
-	cpu_set(cpu, cpu_possible_map);
+	set_cpu_online(cpu, true);
+	set_cpu_present(cpu, true);
+	set_cpu_possible(cpu, true);
 }
 
 void __init __weak smp_setup_processor_id(void)
kernel/compat.c +30 -19

86 lines not on this page · raw message

kernel/cpu.c +99 -45

260 lines not on this page · raw message

kernel/irq/manage.c +9 -2
index 61c4a9b..cd0cd8d 100644
@@ -16,8 +16,15 @@
 #include "internals.h"
 
 #ifdef CONFIG_SMP
+cpumask_var_t irq_default_affinity;
 
-cpumask_t irq_default_affinity = CPU_MASK_ALL;
+static int init_irq_default_affinity(void)
+{
+	alloc_cpumask_var(&irq_default_affinity, GFP_KERNEL);
+	cpumask_setall(irq_default_affinity);
+	return 0;
+}
+core_initcall(init_irq_default_affinity);
 
 /**
  *	synchronize_irq - wait for pending IRQ handlers (on other CPUs)
@@ -127,7 +134,7 @@ int do_irq_select_affinity(unsigned int irq, struct irq_desc *desc)
 			desc->status &= ~IRQ_AFFINITY_SET;
 	}
 
-	cpumask_and(&desc->affinity, cpu_online_mask, &irq_default_affinity);
+	cpumask_and(&desc->affinity, cpu_online_mask, irq_default_affinity);
 set_affinity:
 	desc->chip->set_affinity(irq, &desc->affinity);
 
kernel/irq/proc.c +22 -12
index d2c0e5e..aae3f74 100644
@@ -20,7 +20,7 @@ static struct proc_dir_entry *root_irq_dir;
 static int irq_affinity_proc_show(struct seq_file *m, void *v)
 {
 	struct irq_desc *desc = irq_to_desc((long)m->private);
-	cpumask_t *mask = &desc->affinity;
+	const struct cpumask *mask = &desc->affinity;
 
 #ifdef CONFIG_GENERIC_PENDING_IRQ
 	if (desc->status & IRQ_MOVE_PENDING)
@@ -54,7 +54,7 @@ static ssize_t irq_affinity_proc_write(struct file *file,
 	if (err)
 		goto free_cpumask;
 
-	if (!is_affinity_mask_valid(*new_value)) {
+	if (!is_affinity_mask_valid(new_value)) {
 		err = -EINVAL;
 		goto free_cpumask;
 	}
@@ -93,7 +93,7 @@ static const struct file_operations irq_affinity_proc_fops = {
 
 static int default_affinity_show(struct seq_file *m, void *v)
 {
-	seq_cpumask(m, &irq_default_affinity);
+	seq_cpumask(m, irq_default_affinity);
 	seq_putc(m, '\n');
 	return 0;
 }
@@ -101,27 +101,37 @@ static int default_affinity_show(struct seq_file *m, void *v)
 static ssize_t default_affinity_write(struct file *file,
 		const char __user *buffer, size_t count, loff_t *ppos)
 {
-	cpumask_t new_value;
+	cpumask_var_t new_value;
 	int err;
 
-	err = cpumask_parse_user(buffer, count, &new_value);
+	if (!alloc_cpumask_var(&new_value, GFP_KERNEL))
+		return -ENOMEM;
+
+	err = cpumask_parse_user(buffer, count, new_value);
 	if (err)
-		return err;
+		goto out;
 
-	if (!is_affinity_mask_valid(new_value))
-		return -EINVAL;
+	if (!is_affinity_mask_valid(new_value)) {
+		err = -EINVAL;
+		goto out;
+	}
 
 	/*
 	 * Do not allow disabling IRQs completely - it's a too easy
 	 * way to make the system unusable accidentally :-) At least
 	 * one online CPU still has to be targeted.
 	 */
-	if (!cpus_intersects(new_value, cpu_online_map))
-		return -EINVAL;
+	if (!cpumask_intersects(new_value, cpu_online_mask)) {
+		err = -EINVAL;
+		goto out;
+	}
 
-	irq_default_affinity = new_value;
+	cpumask_copy(irq_default_affinity, new_value);
+	err = count;
 
-	return count;
+out:
+	free_cpumask_var(new_value);
+	return err;
 }
 
 static int default_affinity_open(struct inode *inode, struct file *file)
kernel/kexec.c +1 -1
index ac0fde7..3fb855a 100644
@@ -1116,7 +1116,7 @@ void crash_save_cpu(struct pt_regs *regs, int cpu)
 	struct elf_prstatus prstatus;
 	u32 *buf;
 
-	if ((cpu < 0) || (cpu >= NR_CPUS))
+	if ((cpu < 0) || (cpu >= nr_cpu_ids))
 		return;
 
 	/* Using ELF notes here is opportunistic.
kernel/power/poweroff.c +1 -1
index 72016f0..9789083 100644
@@ -27,7 +27,7 @@ static DECLARE_WORK(poweroff_work, do_poweroff);
 static void handle_poweroff(int key, struct tty_struct *tty)
 {
 	/* run sysrq poweroff on boot cpu */
-	schedule_work_on(first_cpu(cpu_online_map), &poweroff_work);
+	schedule_work_on(cpumask_first(cpu_online_mask), &poweroff_work);
 }
 
 static struct sysrq_key_op	sysrq_poweroff_op = {
kernel/profile.c +25 -13
index 4cb7d68..d18e2d2 100644
@@ -45,7 +45,7 @@ static unsigned long prof_len, prof_shift;
 int prof_on __read_mostly;
 EXPORT_SYMBOL_GPL(prof_on);
 
-static cpumask_t prof_cpu_mask = CPU_MASK_ALL;
+static cpumask_var_t prof_cpu_mask;
 #ifdef CONFIG_SMP
 static DEFINE_PER_CPU(struct profile_hit *[2], cpu_profile_hits);
 static DEFINE_PER_CPU(int, cpu_profile_flip);
@@ -113,9 +113,13 @@ int __ref profile_init(void)
 	buffer_bytes = prof_len*sizeof(atomic_t);
 	if (!slab_is_available()) {
 		prof_buffer = alloc_bootmem(buffer_bytes);
+		alloc_bootmem_cpumask_var(&prof_cpu_mask);
 		return 0;
 	}
 
+	if (!alloc_cpumask_var(&prof_cpu_mask, GFP_KERNEL))
+		return -ENOMEM;
+
 	prof_buffer = kzalloc(buffer_bytes, GFP_KERNEL);
 	if (prof_buffer)
 		return 0;
@@ -128,6 +132,7 @@ int __ref profile_init(void)
 	if (prof_buffer)
 		return 0;
 
+	free_cpumask_var(prof_cpu_mask);
 	return -ENOMEM;
 }
 
@@ -386,13 +391,15 @@ out_free:
 		return NOTIFY_BAD;
 	case CPU_ONLINE:
 	case CPU_ONLINE_FROZEN:
-		cpu_set(cpu, prof_cpu_mask);
+		if (prof_cpu_mask != NULL)
+			cpumask_set_cpu(cpu, prof_cpu_mask);
 		break;
 	case CPU_UP_CANCELED:
 	case CPU_UP_CANCELED_FROZEN:
 	case CPU_DEAD:
 	case CPU_DEAD_FROZEN:
-		cpu_clear(cpu, prof_cpu_mask);
+		if (prof_cpu_mask != NULL)
+			cpumask_clear_cpu(cpu, prof_cpu_mask);
 		if (per_cpu(cpu_profile_hits, cpu)[0]) {
 			page = virt_to_page(per_cpu(cpu_profile_hits, cpu)[0]);
 			per_cpu(cpu_profile_hits, cpu)[0] = NULL;
@@ -430,7 +437,8 @@ void profile_tick(int type)
 
 	if (type == CPU_PROFILING && timer_hook)
 		timer_hook(regs);
-	if (!user_mode(regs) && cpu_isset(smp_processor_id(), prof_cpu_mask))
+	if (!user_mode(regs) && prof_cpu_mask != NULL &&
+	    cpumask_test_cpu(smp_processor_id(), prof_cpu_mask))
 		profile_hit(type, (void *)profile_pc(regs));
 }
 
@@ -442,7 +450,7 @@ void profile_tick(int type)
 static int prof_cpu_mask_read_proc(char *page, char **start, off_t off,
 			int count, int *eof, void *data)
 {
-	int len = cpumask_scnprintf(page, count, (cpumask_t *)data);
+	int len = cpumask_scnprintf(page, count, data);
 	if (count - len < 2)
 		return -EINVAL;
 	len += sprintf(page + len, "\n");
@@ -452,16 +460,20 @@ static int prof_cpu_mask_read_proc(char *page, char **start, off_t off,
 static int prof_cpu_mask_write_proc(struct file *file,
 	const char __user *buffer,  unsigned long count, void *data)
 {
-	cpumask_t *mask = (cpumask_t *)data;
+	struct cpumask *mask = data;
 	unsigned long full_count = count, err;
-	cpumask_t new_value;
+	cpumask_var_t new_value;
 
-	err = cpumask_parse_user(buffer, count, &new_value);
-	if (err)
-		return err;
+	if (!alloc_cpumask_var(&new_value, GFP_KERNEL))
+		return -ENOMEM;
 
-	*mask = new_value;
-	return full_count;
+	err = cpumask_parse_user(buffer, count, new_value);
+	if (!err) {
+		cpumask_copy(mask, new_value);
+		err = full_count;
+	}
+	free_cpumask_var(new_value);
+	return err;
 }
 
 void create_prof_cpu_mask(struct proc_dir_entry *root_irq_dir)
@@ -472,7 +484,7 @@ void create_prof_cpu_mask(struct proc_dir_entry *root_irq_dir)
 	entry = create_proc_entry("prof_cpu_mask", 0600, root_irq_dir);
 	if (!entry)
 		return;
-	entry->data = (void *)&prof_cpu_mask;
+	entry->data = prof_cpu_mask;
 	entry->read_proc = prof_cpu_mask_read_proc;
 	entry->write_proc = prof_cpu_mask_write_proc;
 }
kernel/rcuclassic.c +17 -15
index c03ca3e..490934f 100644
@@ -63,14 +63,14 @@ static struct rcu_ctrlblk rcu_ctrlblk = {
 	.completed = -300,
 	.pending = -300,
 	.lock = __SPIN_LOCK_UNLOCKED(&rcu_ctrlblk.lock),
-	.cpumask = CPU_MASK_NONE,
+	.cpumask = CPU_BITS_NONE,
 };
 static struct rcu_ctrlblk rcu_bh_ctrlblk = {
 	.cur = -300,
 	.completed = -300,
 	.pending = -300,
 	.lock = __SPIN_LOCK_UNLOCKED(&rcu_bh_ctrlblk.lock),
-	.cpumask = CPU_MASK_NONE,
+	.cpumask = CPU_BITS_NONE,
 };
 
 DEFINE_PER_CPU(struct rcu_data, rcu_data) = { 0L };
@@ -85,7 +85,6 @@ static void force_quiescent_state(struct rcu_data *rdp,
 			struct rcu_ctrlblk *rcp)
 {
 	int cpu;
-	cpumask_t cpumask;
 	unsigned long flags;
 
 	set_need_resched();
@@ -96,10 +95,10 @@ static void force_quiescent_state(struct rcu_data *rdp,
 		 * Don't send IPI to itself. With irqs disabled,
 		 * rdp->cpu is the current cpu.
 		 *
-		 * cpu_online_map is updated by the _cpu_down()
+		 * cpu_online_mask is updated by the _cpu_down()
 		 * using __stop_machine(). Since we're in irqs disabled
 		 * section, __stop_machine() is not exectuting, hence
-		 * the cpu_online_map is stable.
+		 * the cpu_online_mask is stable.
 		 *
 		 * However,  a cpu might have been offlined _just_ before
 		 * we disabled irqs while entering here.
@@ -107,13 +106,14 @@ static void force_quiescent_state(struct rcu_data *rdp,
 		 * notification, leading to the offlined cpu's bit
 		 * being set in the rcp->cpumask.
 		 *
-		 * Hence cpumask = (rcp->cpumask & cpu_online_map) to prevent
+		 * Hence cpumask = (rcp->cpumask & cpu_online_mask) to prevent
 		 * sending smp_reschedule() to an offlined CPU.
 		 */
-		cpus_and(cpumask, rcp->cpumask, cpu_online_map);
-		cpu_clear(rdp->cpu, cpumask);
-		for_each_cpu_mask_nr(cpu, cpumask)
-			smp_send_reschedule(cpu);
+		for_each_cpu_and(cpu,
+				  to_cpumask(rcp->cpumask), cpu_online_mask) {
+			if (cpu != rdp->cpu)
+				smp_send_reschedule(cpu);
+		}
 	}
 	spin_unlock_irqrestore(&rcp->lock, flags);
 }
@@ -193,7 +193,7 @@ static void print_other_cpu_stall(struct rcu_ctrlblk *rcp)
 
 	printk(KERN_ERR "INFO: RCU detected CPU stalls:");
 	for_each_possible_cpu(cpu) {
-		if (cpu_isset(cpu, rcp->cpumask))
[… more lines in the raw message]

Lines of this file are not shown. View the raw message.

kernel/rcupreempt.c +10 -9
index 0498265..f9dc8f3 100644

Lines of this file are not shown. View the raw message.

kernel/rcutorture.c +15 -12
index b310655..3245b40 100644

Lines of this file are not shown. View the raw message.

kernel/sched.c +15 -38

53 lines not on this page · raw message

kernel/sched_rt.c +2 -1
index 833b6d4..954e1a8 100644

Lines of this file are not shown. View the raw message.

kernel/smp.c +55 -90

145 lines not on this page · raw message

kernel/softirq.c +1 -1
index 670c1ec..bdbe9de 100644

Lines of this file are not shown. View the raw message.

kernel/softlockup.c +4 -6
index 1ab790c..d9188c6 100644

Lines of this file are not shown. View the raw message.

kernel/stop_machine.c +4 -4
index 24e8cea..286c417 100644

Lines of this file are not shown. View the raw message.

kernel/taskstats.c +24 -15
index 6d7dc4e..888adbc 100644

Lines of this file are not shown. View the raw message.

kernel/time/clocksource.c +5 -4
index 9ed2eec..ca89e15 100644

Lines of this file are not shown. View the raw message.

kernel/time/tick-broadcast.c +59 -56

115 lines not on this page · raw message

kernel/time/tick-common.c +3 -3
index f8372be..63e05d4 100644

Lines of this file are not shown. View the raw message.

kernel/trace/ring_buffer.c +25 -17

42 lines not on this page · raw message

kernel/trace/trace.c +45 -27

72 lines not on this page · raw message

kernel/trace/trace.h +1 -1
index cc7a4f8..4d3d381 100644

Lines of this file are not shown. View the raw message.

kernel/trace/trace_boot.c +1 -1
index 3ccebde..366c8c3 100644

Lines of this file are not shown. View the raw message.

kernel/trace/trace_functions_graph.c +1 -1
index 4bf39fc..930c08e 100644

Lines of this file are not shown. View the raw message.

kernel/trace/trace_hw_branches.c +3 -3
index b6a3e20..649df22 100644

Lines of this file are not shown. View the raw message.

kernel/trace/trace_power.c +1 -1
index a7172a3..7bda248 100644

Lines of this file are not shown. View the raw message.

kernel/trace/trace_sysprof.c +3 -10
index a5779bd..eaca5ad 100644

Lines of this file are not shown. View the raw message.

kernel/workqueue.c +14 -12
index 4952322..2f44583 100644

Lines of this file are not shown. View the raw message.

lib/Kconfig +8 -0
index 2ba43c4..03c2c24 100644

Lines of this file are not shown. View the raw message.

lib/Makefile +1 -0
index 80fe8a3..32b0e64 100644

Lines of this file are not shown. View the raw message.

lib/cpumask.c +59 -3

62 lines not on this page · raw message

lib/find_last_bit.c new file +45 -0

45 lines not on this page · raw message

mm/pdflush.c +13 -3
index a0a14c4..15de509 100644

Lines of this file are not shown. View the raw message.

mm/slab.c +1 -1
index f97e564..ddc41f3 100644

Lines of this file are not shown. View the raw message.

mm/slub.c +11 -9
index 0d861c3..f0e2892 100644

Lines of this file are not shown. View the raw message.

mm/vmscan.c +2 -2
index 62e7f62..d196f46 100644

Lines of this file are not shown. View the raw message.

mm/vmstat.c +2 -2
index c3ccfda..9114974 100644

Lines of this file are not shown. View the raw message.

security/selinux/selinuxfs.c +1 -1
index c863036..e552099 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Part of the diff is not shown here. View the raw message.

#5 lore raw

Re: [git pull] cpus4096 tree, part 3

Quoted 3 lines · Ingo Molnar

ok. The pending regressions are all fixed now, and i've just finished my standard tests on the latest tree and all the tests passed fine.

Ok, pulled and pushed out.

Has anybody looked at what the stack size is with MAXSMP set with an allyesconfig? And what areas are still problematic, if any? Are we going to have some code-paths that still essentially have 1kB+ of stack space just because they haven't been converted and still have the cpu mask on stack?

Linus

#6 lore raw

Re: [git pull] cpus4096 tree, part 3

Quoted 6 lines · Linus Torvalds
Quoted 3 lines · Ingo Molnar

ok. The pending regressions are all fixed now, and i've just finished my standard tests on the latest tree and all the tests passed fine.

Ok, pulled and pushed out.

thanks!

Quoted 5 lines

Has anybody looked at what the stack size is with MAXSMP set with an allyesconfig? And what areas are still problematic, if any? Are we going to have some code-paths that still essentially have 1kB+ of stack space just because they haven't been converted and still have the cpu mask on stack?

ok, indeed testing of that is in order now.

I'll check what the worst-case runtime stack footprint is for an allyesconfig 64-bit bootup - that should be the worst-case scenario on x86. We have a low number of leftover places, but the serious ones like the IPI paths, which triggered stack overflows in the past, got all fixed.

The test is underway with:

CONFIG_64BIT=y
CONFIG_NR_CPUS=4096
CONFIG_MAXSMP=y

Ingo

#7 lore raw

Re: [git pull] cpus4096 tree, part 3

Quoted 8 lines · Ingo Molnar
Quoted 6 lines

Has anybody looked at what the stack size is with MAXSMP set with an allyesconfig? And what areas are still problematic, if any? Are we going to have some code-paths that still essentially have 1kB+ of stack space just because they haven't been converted and still have the cpu mask on stack?

ok, indeed testing of that is in order now.

Well, since I can compile a allyesconfig pretty quickly, I did the static part. It looks better than it used to, and I think most of the huge stacks are totally unrealted to cpu masks. But not all.

But it looks like we have a few:

- flush_tlb_current_task:

	cpumask_t cpu_mask;
 - flush_tlb_mm:
	cpumask_t cpu_mask;
 - local_cpus_show:
	cpumask_t mask;
 - local_cpulist_show:
	cpumask_t mask;

- acpi_cpufreq_target:
cpumask_t online_policy_cpus

and then we have a number of things that have "struct cpufreq_policy" on the stack, and those things have two cpumask_t's in each.

The rest of the high-stack-usage cases - from a _very_ quick look - seem to be unrelated to CPU masks, but in the "more than 1kB of stack" group about a third (wild handwaving eyeballing) of them do seem to be related to cpumask.

And those things (VM, ACPI and PCI) are easily triggered in real use and don't need odd hardware. So I think they need to be fixed before 2.6.29, otherwise I'll have to disable MAXSMP and again limit MAX_CPU back to 128 or something.

Linus

#25 lore raw

Re: [git pull] cpus4096 tree, part 3

In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds
Quoted 2 lines · Ingo Molnar

ok. The pending regressions are all fixed now, and i've just finished my standard tests on the latest tree and all the tests passed fine.

Ok, pulled and pushed out.

Quoted 7 lines

Has anybody looked at what the stack size is with MAXSMP set with an allyesconfig? And what areas are still problematic, if any? Are we going to have some code-paths that still essentially have 1kB+ of stack space just because they haven't been converted and still have the cpu mask on stack?

Linus

Hi Linus,

Yes, I do periodically collect stats for memory and stack usage. Here is a recent stack summary (not "allyes", but most all trace/debug options turned on). It shows the stack growth from a 128 NR_CPUS config to a MAXSMP (4096 NR_CPUS) config. Most of the changes to correct these "stack hogs" have been sitting in a queue until the changes affecting non-x86 architectures have been accepted (which you just did), though some are because of new code from the merge activity.

Rusty has introduced a config option that disables the old cpumask_t which really highlights where the offenders still are. Ultimately, that should prevent any new stack hogs from being introduced, but it won't be settable until 2.6.30 time frame.

====== Stack (-l 500)
1 - 128-defconfig
2 - 4k-defconfig

.1. .2. ..final..

    0  +1640 1640      .  acpi_cpufreq_target
    0  +1368 1368      .  cpufreq_add_dev
    0  +1344 1344      .  store_scaling_governor
    0  +1328 1328      .  store_scaling_min_freq
    0  +1328 1328      .  store_scaling_max_freq
    0  +1328 1328      .  cpufreq_update_policy
    0  +1328 1328      .  cpu_has_cpufreq
    0  +1048 1048      .  get_cur_val
    0  +1032 1032      .  local_cpus_show
    0  +1032 1032      .  local_cpulist_show
    0  +1024 1024      .  pci_bus_show_cpuaffinity
    0   +808 808      .  cpuset_write_resmask
    0   +736 736      .  update_flag
    0   +648 648      .  init_intel_cacheinfo
    0   +640 640      .  cpuset_attach
    0   +584 584      .  shmem_getpage
    0   +584 584      .  __percpu_alloc_mask
    0   +552 552      .  smp_call_function_many
    0   +536 536      .  pci_device_probe
    0   +536 536      .  native_flush_tlb_others
    0   +536 536      .  cpuset_common_file_read
    0   +520 520      .  show_related_cpus
    0   +520 520      .  show_affected_cpus
    0   +520 520      .  get_measured_perf
    0   +520 520      .  flush_tlb_page
    0   +520 520      .  cpuset_can_attach
    0   +512 512      .  flush_tlb_mm
    0   +512 512      .  flush_tlb_current_task
    0   +512 512      .  find_lowest_rq
    0   +512 512      .  acpi_processor_ffh_cstate_probe

====== Text/Data ()

Overall memory reservation looks like this:

      .1.       .2.    ..final..
  5799936     +4096   5804032 +0.07%  TextSize
  3772416   +139264   3911680 +3.69%  DataSize
  8822784  +1234944  10057728   +13%  BssSize
  2445312   +794624   3239936   +32%  InitSize
  1884160     +4096   1888256 +0.22%  PerCPU
   143360   +708608    851968  +494%  OtherSize
 22867968  +2885632  25753600  +112%  Totals

I will update these with the latest changes (and use a allyesconfig config) and post them again soon.

Thanks,
Mike

#11 lore raw

Re: [git pull] cpus4096 tree, part 3

In reply to Ingo Molnar
Quoted 5 lines · Ingo Molnar

The test is underway with:

CONFIG_64BIT=y
CONFIG_NR_CPUS=4096
CONFIG_MAXSMP=y

the worst-case stack footprint i was able to trigger on an allyesconfig bootup (instrumented via CONFIG_STACK_TRACER=y and the 'stacktrace' boot option) was 5.6K:

 # ls -l /debug/tracing/*stack*
 -rw-r--r-- 1 root root 0 2009-01-03 22:42 /debug/tracing/stack_max_size
 -r--r--r-- 1 root root 0 2009-01-03 22:42 /debug/tracing/stack_trace
 # cat /debug/tracing/*stack*
 5624
        Depth   Size      Location    (38 entries)
        -----   ----      --------
  0)     5496     232   lookup_address+0x51/0x20e
  1)     5264     496   __change_page_attr_set_clr+0xa7/0xa15
  2)     4768     128   kernel_map_pages+0x121/0x140
  3)     4640     224   get_page_from_freelist+0x4bb/0x6ec
  4)     4416     160   __alloc_pages_internal+0x150/0x4af
  5)     4256      48   alloc_pages_current+0xbe/0xc7
  6)     4208      16   alloc_slab_page+0x1e/0x6e
  7)     4192      64   new_slab+0x51/0x1e2
  8)     4128      96   __slab_alloc+0x260/0x3fa
  9)     4032      80   kmem_cache_alloc+0xa8/0x120
 10)     3952      48   radix_tree_preload+0x38/0x86
 11)     3904      64   add_to_page_cache_locked+0x51/0xed
 12)     3840      32   add_to_page_cache_lru+0x31/0x6b
 13)     3808      64   find_or_create_page+0x56/0x7d
 14)     3744      80   __getblk+0x136/0x262
 15)     3664      32   __bread+0x13/0x82
 16)     3632      64   ext3_get_branch+0x7b/0xef
 17)     3568     368   ext3_get_blocks_handle+0xa2/0x907
 18)     3200      96   ext3_get_block+0xc3/0x101
 19)     3104     224   do_mpage_readpage+0x1ad/0x4d0
 20)     2880     240   mpage_readpages+0xb6/0xf9
 21)     2640      16   ext3_readpages+0x1f/0x21
 22)     2624     144   __do_page_cache_readahead+0x147/0x1bd
 23)     2480      48   do_page_cache_readahead+0x5b/0x68
 24)     2432     112   filemap_fault+0x176/0x34b
 25)     2320     160   __do_fault+0x58/0x410
 26)     2160     176   handle_mm_fault+0x4b2/0xa3e
 27)     1984     800   do_page_fault+0x86d/0xcec
 28)     1184     208   page_fault+0x25/0x30
 29)      976      16   clear_user+0x30/0x38
 30)      960     288   load_elf_binary+0x61c/0x18f0
 31)      672      80   search_binary_handler+0xd9/0x279
 32)      592     176   load_script+0x1b3/0x1c8
 33)      416      80   search_binary_handler+0xd9/0x279
 34)      336      96   do_execve+0x1df/0x296
 35)      240      64   sys_execve+0x43/0x5e
 36)      176     176   stub_execve+0x6a/0xc0

None of the cpumask_t callsites showed up - but i agree that they should be fixed, 512 bytes of cpumask_t on the stack is not good no matter how one looks at it, it will only make an already strained kernel stack footprint situation worse.

This one looks a bit high:

1) 5264 496 __change_page_attr_set_clr+0xa7/0xa15

(Venki, Suresh and Arjan Cc:-ed)

This one isnt too nice either:

27) 1984 800 do_page_fault+0x86d/0xcec

not sure why it happens - will investigate tomorrow, it's getting late here.

Ingo

#8 lore raw

Re: [git pull] cpus4096 tree, part 3

In reply to Linus Torvalds
Quoted 4 lines · Linus Torvalds

And those things (VM, ACPI and PCI) are easily triggered in real use and don't need odd hardware. So I think they need to be fixed before 2.6.29, otherwise I'll have to disable MAXSMP and again limit MAX_CPU back to 128 or something.

agreed.

Ingo

#12 lore raw

Re: [git pull] cpus4096 tree, part 3

In reply to Ingo Molnar
Quoted 5 lines · Ingo Molnar

This one looks a bit high:

1) 5264 496 __change_page_attr_set_clr+0xa7/0xa15

Quoted 8 lines

(Venki, Suresh and Arjan Cc:-ed)

This one isnt too nice either:

27) 1984 800 do_page_fault+0x86d/0xcec

not sure why it happens - will investigate tomorrow, it's getting late here.

The most common case tends to be insane gcc inlining (vmalloc_fault and spurious_fault), and then gcc not re-using stack slots even if they have no overlap in usage. Some people continue to claim that gcc reuses them, but it's definitely not the case in any complex situation, so I suspect the re-use is probably purely for some simple spilling case, not for variables allocated on the stack.

do_page_fault() in particular has a lot of gunk in it for the special cases.

What happened to Nick's cleanup patch to do_page_fault (a month or two ago? I complained about some of the issues in his first version and asked for some further cleanups, but I think that whole discussion ended with him saying "I am going to add those changes that you suggested (in fact, I already have)".

And then I didn't see anything further. Maybe I just missed the end result. Or maybe we have it in some -mm branch or something?

Linus

#13 lore raw

Re: [git pull] cpus4096 tree, part 3

Quoted 8 lines · Linus Torvalds

What happened to Nick's cleanup patch to do_page_fault (a month or two ago? I complained about some of the issues in his first version and asked for some further cleanups, but I think that whole discussion ended with him saying "I am going to add those changes that you suggested (in fact, I already have)".

And then I didn't see anything further. Maybe I just missed the end result. Or maybe we have it in some -mm branch or something?

they would have been in tip/x86/mm and would be upstream now had Nick re-sent a v2 series but that never happened. I think they might have fallen victim to a serious attention deficit caused by the SLQB patch ;-)

Ingo

#9 lore raw

Re: [git pull] cpus4096 tree, part 3

In reply to Linus Torvalds
Quoted 21 lines · On Sunday 04 January 2009 07:26:03 Linus Torvalds wrote:
Quoted 8 lines · Ingo Molnar
Quoted 6 lines

Has anybody looked at what the stack size is with MAXSMP set with an allyesconfig? And what areas are still problematic, if any? Are we going to have some code-paths that still essentially have 1kB+ of stack space just because they haven't been converted and still have the cpu mask on stack?

ok, indeed testing of that is in order now.

Well, since I can compile a allyesconfig pretty quickly, I did the static part. It looks better than it used to, and I think most of the huge stacks are totally unrealted to cpu masks. But not all.

But it looks like we have a few:

- flush_tlb_current_task:

	cpumask_t cpu_mask;
 - flush_tlb_mm:
	cpumask_t cpu_mask;

...

Quoted 2 lines

- acpi_cpufreq_target:
cpumask_t online_policy_cpus

Mike? These are x86-specific...

Quoted 4 lines

- local_cpus_show:

	cpumask_t mask;
 - local_cpulist_show:
	cpumask_t mask;

Yes, this removal is still in my queue. I'll double-check that all the
archs have the new "cpumask_of_pcibus". (cpumask:replace-and-remove-pcibus_to_cpumask.patch "cpumask: remove the now-obsoleted pcibus_to_cpumask()").

Quoted 2 lines

and then we have a number of things that have "struct cpufreq_policy" on the stack, and those things have two cpumask_t's in each.

Yep, we have the conversion for that too. Mike, it's cpumask:convert-drivers_acpi.patch "cpumask: convert struct cpufreq_policy to cpumask_var_t."

Quoted 4 lines

The rest of the high-stack-usage cases - from a _very_ quick look - seem to be unrelated to CPU masks, but in the "more than 1kB of stack" group about a third (wild handwaving eyeballing) of them do seem to be related to cpumask.

Mike was tracking this; I think he has a script to set NR_CPUS small then large and dump the changes.

Cheers,
Rusty.

#30 lore raw

Re: [PATCH] ia64: cpumask fix for is_affinity_mask_valid()

In reply to Ingo Molnar
Quoted 1 line · On Saturday 03 January 2009 22:29:04 Ingo Molnar wrote:

update it to cpumask_var_t.

Please change it to a const struct cpumask_t *.

It should be const, and of course 'const cpumask_var_t' isn't the same.

Thanks,
Rusty.

#31 lore raw

Re: [PATCH] ia64: cpumask fix for is_affinity_mask_valid()

Quoted 9 lines · Rusty Russell
Quoted 1 line · On Saturday 03 January 2009 22:29:04 Ingo Molnar wrote:

update it to cpumask_var_t.

Please change it to a const struct cpumask_t *.

It should be const, and of course 'const cpumask_var_t' isn't the same.

Thanks,
Rusty.

I have a fixup patch coming to correct that and remove an unneeded cpumask off the stack.

Mike

#10 lore raw

Re: [git pull] cpus4096 tree, part 3

In reply to Rusty Russell
Quoted 18 lines · Rusty Russell
Quoted 11 lines · On Sunday 04 January 2009 07:26:03 Linus Torvalds wrote:
Quoted 6 lines · Ingo Molnar
Quoted 5 lines

Has anybody looked at what the stack size is with MAXSMP set with an allyesconfig? And what areas are still problematic, if any? Are we going to have some code-paths that still essentially have 1kB+ of stack space just because they haven't been converted and still have the cpu mask on stack?

ok, indeed testing of that is in order now.

Well, since I can compile a allyesconfig pretty quickly, I did the static part. It looks better than it used to, and I think most of the huge stacks are totally unrealted to cpu masks. But not all.

Quoted 6 lines

But it looks like we have a few:

- flush_tlb_current_task:

	cpumask_t cpu_mask;
 - flush_tlb_mm:
	cpumask_t cpu_mask;
Quoted 5 lines

...

Quoted 2 lines

- acpi_cpufreq_target:
cpumask_t online_policy_cpus

Mike? These are x86-specific...

I've been testing the heck out of it... ;-)

Quoted 5 lines
Quoted 5 lines

- local_cpus_show:

	cpumask_t mask;
 - local_cpulist_show:
	cpumask_t mask;

Yes, these are in my "real soon now" patchset. Trivial.

Quoted 9 lines

Yes, this removal is still in my queue. I'll double-check that all the
archs have the new "cpumask_of_pcibus". (cpumask:replace-and-remove-pcibus_to_cpumask.patch "cpumask: remove the now-obsoleted pcibus_to_cpumask()").

Quoted 2 lines

and then we have a number of things that have "struct cpufreq_policy" on the stack, and those things have two cpumask_t's in each.

Yep, we have the conversion for that too. Mike, it's cpumask:convert-drivers_acpi.patch "cpumask: convert struct cpufreq_policy to cpumask_var_t."

That's part of what I'm testing above.

Quoted 7 lines
Quoted 4 lines

The rest of the high-stack-usage cases - from a _very_ quick look - seem to be unrelated to CPU masks, but in the "more than 1kB of stack" group about a third (wild handwaving eyeballing) of them do seem to be related to cpumask.

Mike was tracking this; I think he has a script to set NR_CPUS small then large and dump the changes.

It's looking pretty good, only 11 > 1k and 19 more > 512.

Thanks,
Mike

#32 lore raw

Re: [PATCH] ia64: cpumask fix for is_affinity_mask_valid()

In reply to Mike Travis
Quoted 5 lines · Mike Travis
Quoted 4 lines · Rusty Russell
Quoted 1 line · On Saturday 03 January 2009 22:29:04 Ingo Molnar wrote:

update it to cpumask_var_t.

Please change it to a const struct cpumask_t *.

yeah.

Quoted 8 lines
Quoted 5 lines

It should be const, and of course 'const cpumask_var_t' isn't the same.

Thanks,
Rusty.

I have a fixup patch coming to correct that and remove an unneeded cpumask off the stack.

ok, will wait for that.

Ingo

#14 lore raw

Re: [git pull] cpus4096 tree, part 3

In reply to Ingo Molnar
Quoted 9 lines · Ingo Molnar
Quoted 6 lines · Linus Torvalds

What happened to Nick's cleanup patch to do_page_fault (a month or two ago? I complained about some of the issues in his first version and asked for some further cleanups, but I think that whole discussion ended with him saying "I am going to add those changes that you suggested (in fact, I already have)".

Quoted 6 lines
Quoted 2 lines

And then I didn't see anything further. Maybe I just missed the end result. Or maybe we have it in some -mm branch or something?

they would have been in tip/x86/mm and would be upstream now had Nick re-sent a v2 series but that never happened. I think they might have fallen victim to a serious attention deficit caused by the SLQB patch ;-)

Well, I already added Linus's suggestions but didn't submit it because there was a bit of work going on in that file as far as I could see, both in the x86 tree and in -mm:

(http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.28-rc2/2.6.28-rc2-mm1/broken-out/mm-invoke-oom-killer-from-page-fault.patch)

It isn't a big deal to resolve either way, but I don't want to make Andrew's life harder.

[Yes OK now I'm the guilty one of pushing in an x86 patch not via the x86 tree ;) This one is easy to break in pieces, but I didn't want to create a dependency between the trees]

I didn't really consider it to be urgent, so I was waiting for that patch to go in, but I was still hoping to get this into 2.6.29... This is what it looks like now with your suggestions, and just merged it to your current tree (untested).

I'll cc the linux-arch list here too, because it might be nice to keep these things as structurally similar as possible (and they'll all want to look at the -mm patch above, although I'll probably end up having to write the patches!).

---

Optimise x86's do_page_fault (C entry point for the page fault path).

gcc isn't _all_ that smart about spilling registers to stack or reusing stack slots, even with branch annotations. do_page_fault contained a lot of functionality, so split unlikely paths into their own functions, and mark them as noinline just to be sure. I consider this actually to be somewhat of a cleanup too: the main function now contains about half the number of lines.

Also, ensure the order of arguments to functions is always the same: regs, addr, error_code. This can reduce code size a tiny bit, and just looks neater too.

Add a couple of branch annotations.

One real behavioural difference this makes is that the OOM-init-task case will no longer loop around the page fault handler, but will return to userspace and presumably retry the fault. Effectively the same macro-behaviour, but it is a notable difference. Such change in behaviour should disappear after the "call oom killer from page fault" patch.

Before:
do_page_fault:

          subq    $360, %rsp      #,

After:
do_page_fault:

          subq    $56, %rsp       #,

bloat-o-meter:
add/remove: 8/0 grow/shrink: 0/1 up/down: 2222/-1680 (542)

  function                                     old     new   delta
  __bad_area_nosemaphore                         -     506    +506
  no_context                                     -     474    +474
  vmalloc_fault                                  -     424    +424
  spurious_fault                                 -     358    +358
  mm_fault_error                                 -     272    +272
  bad_area_access_error                          -      89     +89
  bad_area                                       -      89     +89
  bad_area_nosemaphore                           -      10     +10

do_page_fault 2464 784 -1680

Yes, the total size increases by 542 bytes, due to the extra function calls. But these will very rarely be called (except for vmalloc_fault) in a normal workload. Importantly, do_page_fault is less than 1/3rd it's original size. Existing gotos and branch hints did move a lot of the infrequently used text out of the fastpath, but that's even further improved after this patch.

---
 arch/x86/mm/fault.c |  458 ++++++++++++++++++++++++++++++----------------------
 1 file changed, 265 insertions(+), 193 deletions(-)
arch/x86/mm/fault.c +265 -193
Index: linux-2.6/arch/x86/mm/fault.c
===================================================================
@@ -91,8 +91,8 @@ static inline int notify_page_fault(stru
  *
  * Opcode checker based on code by Richard Brunner
  */
-static int is_prefetch(struct pt_regs *regs, unsigned long addr,
-		       unsigned long error_code)
+static int is_prefetch(struct pt_regs *regs, unsigned long error_code,
+			unsigned long addr)
 {
 	unsigned char *instr;
 	int scan_more = 1;
@@ -409,15 +409,15 @@ static void show_fault_oops(struct pt_re
 }
 
 #ifdef CONFIG_X86_64
-static noinline void pgtable_bad(unsigned long address, struct pt_regs *regs,
-				 unsigned long error_code)
+static noinline void pgtable_bad(struct pt_regs *regs,
+			 unsigned long error_code, unsigned long address)
 {
 	unsigned long flags = oops_begin();
 	int sig = SIGKILL;
-	struct task_struct *tsk;
+	struct task_struct *tsk = current;
 
 	printk(KERN_ALERT "%s: Corrupted page table at address %lx\n",
-	       current->comm, address);
+	       tsk->comm, address);
 	dump_pagetable(address);
 	tsk = current;
 	tsk->thread.cr2 = address;
@@ -429,6 +429,200 @@ static noinline void pgtable_bad(unsigne
 }
 #endif
 
+static noinline void no_context(struct pt_regs *regs,
+			unsigned long error_code, unsigned long address)
+{
+	struct task_struct *tsk = current;
+#ifdef CONFIG_X86_64
+	unsigned long flags;
+	int sig;
+#endif
+
+	/* Are we prepared to handle this kernel fault?  */
+	if (fixup_exception(regs))
+		return;
+
+	/*
+	 * X86_32
+	 * Valid to do another page fault here, because if this fault
+	 * had been triggered by is_prefetch fixup_exception would have
+	 * handled it.
+	 *
+	 * X86_64
+	 * Hall of shame of CPU/BIOS bugs.
+	 */
+	if (is_prefetch(regs, error_code, address))
+		return;
+
+	if (is_errata93(regs, address))
+		return;
+
+	/*
+	 * Oops. The kernel tried to access some bad page. We'll have to
+	 * terminate things with extreme prejudice.
+	 */
+#ifdef CONFIG_X86_32
+	bust_spinlocks(1);
+#else
+	flags = oops_begin();
+#endif
+
+	show_fault_oops(regs, error_code, address);
+
+	tsk->thread.cr2 = address;
+	tsk->thread.trap_no = 14;
+	tsk->thread.error_code = error_code;
+
+#ifdef CONFIG_X86_32
+	die("Oops", regs, error_code);
+	bust_spinlocks(0);
+	do_exit(SIGKILL);
+#else
+	sig = SIGKILL;
+	if (__die("Oops", regs, error_code))
+		sig = 0;
+	/* Executive summary in case the body of the oops scrolled away */
+	printk(KERN_EMERG "CR2: %016lx\n", address);
+	oops_end(flags, regs, sig);
+#endif
+}
+
+static void __bad_area_nosemaphore(struct pt_regs *regs,
+			unsigned long error_code, unsigned long address,
+			int si_code)
+{
+	struct task_struct *tsk = current;
+
+	/* User mode accesses just cause a SIGSEGV */
+	if (error_code & PF_USER) {
+		/*
+		 * It's possible to have interrupts off here.
+		 */
+		local_irq_enable();
+
+		/*
+		 * Valid to do another page fault here because this one came
+		 * from user space.
+		 */
+		if (is_prefetch(regs, error_code, address))
+			return;
+
+		if (is_errata100(regs, address))
+			return;
+
+		if (show_unhandled_signals && unhandled_signal(tsk, SIGSEGV) &&
+		    printk_ratelimit()) {
+			printk(
+			"%s%s[%d]: segfault at %lx ip %p sp %p error %lx",
+			task_pid_nr(tsk) > 1 ? KERN_INFO : KERN_EMERG,
+			tsk->comm, task_pid_nr(tsk), address,
+			(void *) regs->ip, (void *) regs->sp, error_code);
+			print_vma_addr(" in ", regs->ip);
+			printk("\n");
+		}
+
+		tsk->thread.cr2 = address;
+		/* Kernel addresses are always protection faults */
+		tsk->thread.error_code = error_code | (address >= TASK_SIZE);
+		tsk->thread.trap_no = 14;
+		force_sig_info_fault(SIGSEGV, si_code, address, tsk);
+		return;
+	}
+
+	if (is_f00f_bug(regs, address))
+		return;
+
+	no_context(regs, error_code, address);
+}
+
+static noinline void bad_area_nosemaphore(struct pt_regs *regs,
+			unsigned long error_code, unsigned long address)
+{
+	__bad_area_nosemaphore(regs, error_code, address, SEGV_MAPERR);
+}
+
+static void __bad_area(struct pt_regs *regs,
+			unsigned long error_code, unsigned long address,
+			int si_code)
+{
+	struct mm_struct *mm = current->mm;
+
+	/*
+	 * Something tried to access memory that isn't in our memory map..
+	 * Fix it, but check if it's kernel or user first..
+	 */
+	up_read(&mm->mmap_sem);
+
+	__bad_area_nosemaphore(regs, error_code, address, si_code);
+}
+
+static noinline void bad_area(struct pt_regs *regs,
+			unsigned long error_code, unsigned long address)
+{
+	__bad_area(regs, error_code, address, SEGV_MAPERR);
+}
+
+static noinline void bad_area_access_error(struct pt_regs *regs,
+			unsigned long error_code, unsigned long address)
+{
+	__bad_area(regs, error_code, address, SEGV_ACCERR);
+}
+
+/* TODO: fixup for "mm-invoke-oom-killer-from-page-fault.patch" */
+static void out_of_memory(struct pt_regs *regs,
+			unsigned long error_code, unsigned long address)
+{
+	struct task_struct *tsk = current;
+	struct mm_struct *mm = tsk->mm;
+	/*
+	 * We ran out of memory, or some other thing happened to us that made
+	 * us unable to handle the page fault gracefully.
+	 */
+	up_read(&mm->mmap_sem);
+	if (is_global_init(tsk)) {
+		yield();
+		return;
+	}
+
+	printk("VM: killing process %s\n", tsk->comm);
+	if (error_code & PF_USER)
+		do_group_exit(SIGKILL);
+	no_context(regs, error_code, address);
+}
+
+static void do_sigbus(struct pt_regs *regs,
+			unsigned long error_code, unsigned long address)
+{
+	struct task_struct *tsk = current;
+	struct mm_struct *mm = tsk->mm;
+
+	up_read(&mm->mmap_sem);
+
+	/* Kernel mode? Handle exceptions or die */
+	if (!(error_code & PF_USER))
+		no_context(regs, error_code, address);
+#ifdef CONFIG_X86_32
+	/* User space => ok to do another page fault */
+	if (is_prefetch(regs, error_code, address))
+		return;
+#endif
+	tsk->thread.cr2 = address;
+	tsk->thread.error_code = error_code;
+	tsk->thread.trap_no = 14;
+	force_sig_info_fault(SIGBUS, BUS_ADRERR, address, tsk);
+}
+
+static noinline void mm_fault_error(struct pt_regs *regs,
+		unsigned long error_code, unsigned long address, unsigned int fault)
+{
+	if (fault & VM_FAULT_OOM)
+		out_of_memory(regs, error_code, address);
+	else if (fault & VM_FAULT_SIGBUS)
+		do_sigbus(regs, error_code, address);
+	else
+		BUG();
+}
+
 static int spurious_fault_check(unsigned long error_code, pte_t *pte)
 {
 	if ((error_code & PF_WRITE) && !pte_write(*pte))
@@ -448,8 +642,8 @@ static int spurious_fault_check(unsigned
  * There are no security implications to leaving a stale TLB when
  * increasing the permissions on a page.
  */
-static int spurious_fault(unsigned long address,
-			  unsigned long error_code)
+static noinline int spurious_fault(unsigned long error_code,
+				unsigned long address)
 {
 	pgd_t *pgd;
 	pud_t *pud;
@@ -494,7 +688,7 @@ static int spurious_fault(unsigned long
  *
  * This assumes no large pages in there.
  */
-static int vmalloc_fault(unsigned long address)
+static noinline int vmalloc_fault(unsigned long address)
 {
 #ifdef CONFIG_X86_32
 	unsigned long pgd_paddr;
@@ -573,6 +767,25 @@ static int vmalloc_fault(unsigned long a
 
 int show_unhandled_signals = 1;
 
+static inline int access_error(unsigned long error_code, int write,
+				struct vm_area_struct *vma)
+{
+	if (write) {
+		/* write, present and write, not present */
+		if (unlikely(!(vma->vm_flags & VM_WRITE)))
+			return 1;
+	} else if (unlikely(error_code & PF_PROT)) {
+		/* read, present */
+		return 1;
+	} else {
+		/* read, not present */
+		if (unlikely(!(vma->vm_flags & (VM_READ | VM_EXEC | VM_WRITE))))
+			return 1;
+	}
+
+	return 0;
+}
+
 /*
  * This routine handles page faults.  It determines the address,
  * and the problem, and then passes it off to one of the appropriate
@@ -583,16 +796,12 @@ asmlinkage
 #endif
 void __kprobes do_page_fault(struct pt_regs *regs, unsigned long error_code)
 {
+	unsigned long address;
 	struct task_struct *tsk;
 	struct mm_struct *mm;
 	struct vm_area_struct *vma;
-	unsigned long address;
-	int write, si_code;
+	int write;
 	int fault;
-#ifdef CONFIG_X86_64
-	unsigned long flags;
-	int sig;
-#endif
 
 	tsk = current;
 	mm = tsk->mm;
@@ -601,9 +810,7 @@ void __kprobes do_page_fault(struct pt_r
 	/* get the address */
 	address = read_cr2();
 
-	si_code = SEGV_MAPERR;
-
-	if (notify_page_fault(regs))
+	if (unlikely(notify_page_fault(regs)))
 		return;
 	if (unlikely(kmmio_fault(regs, address)))
 		return;
@@ -638,10 +845,10 @@ void __kprobes do_page_fault(struct pt_r
 		 * Don't take the mm semaphore here. If we fixup a prefetch
 		 * fault we could otherwise deadlock.
 		 */
-		goto bad_area_nosemaphore;
+		bad_area_nosemaphore(regs, error_code, address);
+		return;
 	}
 
-
 	/*
 	 * It's safe to allow irq's after cr2 has been saved and the
 	 * vmalloc fault has been handled.
@@ -657,17 +864,18 @@ void __kprobes do_page_fault(struct pt_r
 
 #ifdef CONFIG_X86_64
 	if (unlikely(error_code & PF_RSVD))
-		pgtable_bad(address, regs, error_code);
+		pgtable_bad(regs, error_code, address);
 #endif
 
 	/*
 	 * If we're in an interrupt, have no user context or are running in an
 	 * atomic region then we must not take the fault.
 	 */
-	if (unlikely(in_atomic() || !mm))
-		goto bad_area_nosemaphore;
+	if (unlikely(in_atomic() || !mm)) {
+		bad_area_nosemaphore(regs, error_code, address);
+		return;
+	}
 
-again:
 	/*
 	 * When running in the kernel we expect faults to occur only to
 	 * addresses in user space.  All other faults represent errors in the
@@ -684,20 +892,26 @@ again:
 	 * source.  If this is invalid we can skip the address space check,
 	 * thus avoiding the deadlock.
 	 */
-	if (!down_read_trylock(&mm->mmap_sem)) {
+	if (unlikely(!down_read_trylock(&mm->mmap_sem))) {
 		if ((error_code & PF_USER) == 0 &&
-		    !search_exception_tables(regs->ip))
-			goto bad_area_nosemaphore;
+		    !search_exception_tables(regs->ip)) {
+			bad_area_nosemaphore(regs, error_code, address);
+			return;
+		}
 		down_read(&mm->mmap_sem);
 	}
 
 	vma = find_vma(mm, address);
-	if (!vma)
-		goto bad_area;
-	if (vma->vm_start <= address)
+	if (unlikely(!vma)) {
+		bad_area(regs, error_code, address);
+		return;
+	}
+	if (likely(vma->vm_start <= address))
 		goto good_area;
-	if (!(vma->vm_flags & VM_GROWSDOWN))
-		goto bad_area;
+	if (unlikely(!(vma->vm_flags & VM_GROWSDOWN))) {
+		bad_area(regs, error_code, address);
+		return;
+	}
 	if (error_code & PF_USER) {
 		/*
 		 * Accessing the stack below %sp is always a bug.
@@ -705,31 +919,25 @@ again:
 		 * and pusha to work.  ("enter $65535,$31" pushes
 		 * 32 pointers and then decrements %sp by 65535.)
 		 */
-		if (address + 65536 + 32 * sizeof(unsigned long) < regs->sp)
-			goto bad_area;
+		if (unlikely(address + 65536 + 32 * sizeof(unsigned long) < regs->sp)) {
+			bad_area(regs, error_code, address);
+			return;
+		}
 	}
-	if (expand_stack(vma, address))
-		goto bad_area;
-/*
- * Ok, we have a good vm_area for this memory access, so
- * we can handle it..
- */
+	if (unlikely(expand_stack(vma, address))) {
+		bad_area(regs, error_code, address);
+		return;
+	}
+
+	/*
+	 * Ok, we have a good vm_area for this memory access, so
+	 * we can handle it..
+	 */
 good_area:
-	si_code = SEGV_ACCERR;
-	write = 0;
-	switch (error_code & (PF_PROT|PF_WRITE)) {
-	default:	/* 3: write, present */
-		/* fall through */
-	case PF_WRITE:		/* write, not present */
-		if (!(vma->vm_flags & VM_WRITE))
-			goto bad_area;
-		write++;
-		break;
-	case PF_PROT:		/* read, present */
-		goto bad_area;
-	case 0:			/* read, not present */
-		if (!(vma->vm_flags & (VM_READ | VM_EXEC | VM_WRITE)))
-			goto bad_area;
+	write = error_code & PF_WRITE;
+	if (unlikely(access_error(error_code, write, vma))) {
+		bad_area_access_error(regs, error_code, address);
+		return;
 	}
 
 	/*
@@ -739,11 +947,8 @@ good_area:
 	 */
 	fault = handle_mm_fault(mm, vma, address, write);
 	if (unlikely(fault & VM_FAULT_ERROR)) {
-		if (fault & VM_FAULT_OOM)
-			goto out_of_memory;
-		else if (fault & VM_FAULT_SIGBUS)
-			goto do_sigbus;
-		BUG();
+		mm_fault_error(regs, error_code, address, fault);
+		return;
 	}
 	if (fault & VM_FAULT_MAJOR)
 		tsk->maj_flt++;
@@ -761,139 +966,6 @@ good_area:
 	}
 #endif
 	up_read(&mm->mmap_sem);
-	return;
-
-/*
- * Something tried to access memory that isn't in our memory map..
- * Fix it, but check if it's kernel or user first..
- */
-bad_area:
-	up_read(&mm->mmap_sem);
-
-bad_area_nosemaphore:
-	/* User mode accesses just cause a SIGSEGV */
-	if (error_code & PF_USER) {
-		/*
-		 * It's possible to have interrupts off here.
-		 */
-		local_irq_enable();
-
-		/*
-		 * Valid to do another page fault here because this one came
-		 * from user space.
-		 */
-		if (is_prefetch(regs, address, error_code))
-			return;
-
-		if (is_errata100(regs, address))
-			return;
-
-		if (show_unhandled_signals && unhandled_signal(tsk, SIGSEGV) &&
-		    printk_ratelimit()) {
-			printk(
-			"%s%s[%d]: segfault at %lx ip %p sp %p error %lx",
-			task_pid_nr(tsk) > 1 ? KERN_INFO : KERN_EMERG,
-			tsk->comm, task_pid_nr(tsk), address,
-			(void *) regs->ip, (void *) regs->sp, error_code);
-			print_vma_addr(" in ", regs->ip);
-			printk("\n");
-		}
-
-		tsk->thread.cr2 = address;
-		/* Kernel addresses are always protection faults */
-		tsk->thread.error_code = error_code | (address >= TASK_SIZE);
-		tsk->thread.trap_no = 14;
-		force_sig_info_fault(SIGSEGV, si_code, address, tsk);
-		return;
-	}
-
-	if (is_f00f_bug(regs, address))
-		return;
-
-no_context:
-	/* Are we prepared to handle this kernel fault?  */
-	if (fixup_exception(regs))
-		return;
-
-	/*
-	 * X86_32
-	 * Valid to do another page fault here, because if this fault
-	 * had been triggered by is_prefetch fixup_exception would have
-	 * handled it.
-	 *
-	 * X86_64
-	 * Hall of shame of CPU/BIOS bugs.
-	 */
-	if (is_prefetch(regs, address, error_code))
-		return;
-
-	if (is_errata93(regs, address))
-		return;
-
-/*
- * Oops. The kernel tried to access some bad page. We'll have to
- * terminate things with extreme prejudice.
- */
-#ifdef CONFIG_X86_32
-	bust_spinlocks(1);
-#else
-	flags = oops_begin();
-#endif
-
-	show_fault_oops(regs, error_code, address);
-
-	tsk->thread.cr2 = address;
-	tsk->thread.trap_no = 14;
-	tsk->thread.error_code = error_code;
-
-#ifdef CONFIG_X86_32
-	die("Oops", regs, error_code);
-	bust_spinlocks(0);
-	do_exit(SIGKILL);
-#else
-	sig = SIGKILL;
-	if (__die("Oops", regs, error_code))
-		sig = 0;
-	/* Executive summary in case the body of the oops scrolled away */
-	printk(KERN_EMERG "CR2: %016lx\n", address);
-	oops_end(flags, regs, sig);
-#endif
-
-/*
- * We ran out of memory, or some other thing happened to us that made
- * us unable to handle the page fault gracefully.
- */
-out_of_memory:
-	up_read(&mm->mmap_sem);
-	if (is_global_init(tsk)) {
-		yield();
-		/*
-		 * Re-lookup the vma - in theory the vma tree might
-		 * have changed:
-		 */
-		goto again;
-	}
-
-	printk("VM: killing process %s\n", tsk->comm);
-	if (error_code & PF_USER)
-		do_group_exit(SIGKILL);
-	goto no_context;
-
-do_sigbus:
-	up_read(&mm->mmap_sem);
-
-	/* Kernel mode? Handle exceptions or die */
-	if (!(error_code & PF_USER))
-		goto no_context;
-#ifdef CONFIG_X86_32
-	/* User space => ok to do another page fault */
-	if (is_prefetch(regs, address, error_code))
-		return;
-#endif
-	tsk->thread.cr2 = address;
-	tsk->thread.error_code = error_code;
-	tsk->thread.trap_no = 14;
-	force_sig_info_fault(SIGBUS, BUS_ADRERR, address, tsk);
 }
 
 DEFINE_SPINLOCK(pgd_lock);
#15 lore raw

Re: [git pull] cpus4096 tree, part 3

Really cc linux-arch this time

Quoted 35 lines · Nick Piggin
Quoted 9 lines · Ingo Molnar
Quoted 6 lines · Linus Torvalds

What happened to Nick's cleanup patch to do_page_fault (a month or two ago? I complained about some of the issues in his first version and asked for some further cleanups, but I think that whole discussion ended with him saying "I am going to add those changes that you suggested (in fact, I already have)".

Quoted 6 lines
Quoted 2 lines

And then I didn't see anything further. Maybe I just missed the end result. Or maybe we have it in some -mm branch or something?

they would have been in tip/x86/mm and would be upstream now had Nick re-sent a v2 series but that never happened. I think they might have fallen victim to a serious attention deficit caused by the SLQB patch ;-)

Well, I already added Linus's suggestions but didn't submit it because there was a bit of work going on in that file as far as I could see, both in the x86 tree and in -mm:

(http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.28-rc2/2.6.28-rc2-mm1/broken-out/mm-invoke-oom-killer-from-page-fault.patch)

It isn't a big deal to resolve either way, but I don't want to make Andrew's life harder.

[Yes OK now I'm the guilty one of pushing in an x86 patch not via the x86 tree ;) This one is easy to break in pieces, but I didn't want to create a dependency between the trees]

I didn't really consider it to be urgent, so I was waiting for that patch to go in, but I was still hoping to get this into 2.6.29... This is what it looks like now with your suggestions, and just merged it to your current tree (untested).

Quoted 4 lines

I'll cc the linux-arch list here too, because it might be nice to keep these things as structurally similar as possible (and they'll all want to look at the -mm patch above, although I'll probably end up having to write the patches!).

---
Optimise x86's do_page_fault (C entry point for the page fault path).

gcc isn't _all_ that smart about spilling registers to stack or reusing stack slots, even with branch annotations. do_page_fault contained a lot of functionality, so split unlikely paths into their own functions, and mark them as noinline just to be sure. I consider this actually to be somewhat of a cleanup too: the main function now contains about half the number of lines.

Also, ensure the order of arguments to functions is always the same: regs, addr, error_code. This can reduce code size a tiny bit, and just looks neater too.

Add a couple of branch annotations.

One real behavioural difference this makes is that the OOM-init-task case will no longer loop around the page fault handler, but will return to userspace and presumably retry the fault. Effectively the same macro-behaviour, but it is a notable difference. Such change in behaviour should disappear after the "call oom killer from page fault" patch.

Before:
do_page_fault:

          subq    $360, %rsp      #,

After:
do_page_fault:

          subq    $56, %rsp       #,

bloat-o-meter:
add/remove: 8/0 grow/shrink: 0/1 up/down: 2222/-1680 (542)

  function                                     old     new   delta
  __bad_area_nosemaphore                         -     506    +506
  no_context                                     -     474    +474
  vmalloc_fault                                  -     424    +424
  spurious_fault                                 -     358    +358
  mm_fault_error                                 -     272    +272
  bad_area_access_error                          -      89     +89
  bad_area                                       -      89     +89
  bad_area_nosemaphore                           -      10     +10

do_page_fault 2464 784 -1680

Yes, the total size increases by 542 bytes, due to the extra function calls. But these will very rarely be called (except for vmalloc_fault) in a normal workload. Importantly, do_page_fault is less than 1/3rd it's original size. Existing gotos and branch hints did move a lot of the infrequently used text out of the fastpath, but that's even further improved after this patch.

---
 arch/x86/mm/fault.c |  458 ++++++++++++++++++++++++++++++----------------------
 1 file changed, 265 insertions(+), 193 deletions(-)
arch/x86/mm/fault.c +265 -193
Index: linux-2.6/arch/x86/mm/fault.c
===================================================================
@@ -91,8 +91,8 @@ static inline int notify_page_fault(stru
  *
  * Opcode checker based on code by Richard Brunner
  */
-static int is_prefetch(struct pt_regs *regs, unsigned long addr,
-		       unsigned long error_code)
+static int is_prefetch(struct pt_regs *regs, unsigned long error_code,
+			unsigned long addr)
 {
 	unsigned char *instr;
 	int scan_more = 1;
@@ -409,15 +409,15 @@ static void show_fault_oops(struct pt_re
 }
 
 #ifdef CONFIG_X86_64
-static noinline void pgtable_bad(unsigned long address, struct pt_regs *regs,
-				 unsigned long error_code)
+static noinline void pgtable_bad(struct pt_regs *regs,
+			 unsigned long error_code, unsigned long address)
 {
 	unsigned long flags = oops_begin();
 	int sig = SIGKILL;
-	struct task_struct *tsk;
+	struct task_struct *tsk = current;
 
 	printk(KERN_ALERT "%s: Corrupted page table at address %lx\n",
-	       current->comm, address);
+	       tsk->comm, address);
 	dump_pagetable(address);
 	tsk = current;
 	tsk->thread.cr2 = address;
@@ -429,6 +429,200 @@ static noinline void pgtable_bad(unsigne
 }
 #endif
 
+static noinline void no_context(struct pt_regs *regs,
+			unsigned long error_code, unsigned long address)
+{
+	struct task_struct *tsk = current;
+#ifdef CONFIG_X86_64
+	unsigned long flags;
+	int sig;
+#endif
+
+	/* Are we prepared to handle this kernel fault?  */
+	if (fixup_exception(regs))
+		return;
+
+	/*
+	 * X86_32
+	 * Valid to do another page fault here, because if this fault
+	 * had been triggered by is_prefetch fixup_exception would have
+	 * handled it.
+	 *
+	 * X86_64
+	 * Hall of shame of CPU/BIOS bugs.
+	 */
+	if (is_prefetch(regs, error_code, address))
+		return;
+
+	if (is_errata93(regs, address))
+		return;
+
+	/*
+	 * Oops. The kernel tried to access some bad page. We'll have to
+	 * terminate things with extreme prejudice.
+	 */
+#ifdef CONFIG_X86_32
+	bust_spinlocks(1);
+#else
+	flags = oops_begin();
+#endif
+
+	show_fault_oops(regs, error_code, address);
+
+	tsk->thread.cr2 = address;
+	tsk->thread.trap_no = 14;
+	tsk->thread.error_code = error_code;
+
+#ifdef CONFIG_X86_32
+	die("Oops", regs, error_code);
+	bust_spinlocks(0);
+	do_exit(SIGKILL);
+#else
+	sig = SIGKILL;
+	if (__die("Oops", regs, error_code))
+		sig = 0;
+	/* Executive summary in case the body of the oops scrolled away */
+	printk(KERN_EMERG "CR2: %016lx\n", address);
+	oops_end(flags, regs, sig);
+#endif
+}
+
+static void __bad_area_nosemaphore(struct pt_regs *regs,
+			unsigned long error_code, unsigned long address,
+			int si_code)
+{
+	struct task_struct *tsk = current;
+
+	/* User mode accesses just cause a SIGSEGV */
+	if (error_code & PF_USER) {
+		/*
+		 * It's possible to have interrupts off here.
+		 */
+		local_irq_enable();
+
+		/*
+		 * Valid to do another page fault here because this one came
+		 * from user space.
+		 */
+		if (is_prefetch(regs, error_code, address))
+			return;
+
+		if (is_errata100(regs, address))
+			return;
+
+		if (show_unhandled_signals && unhandled_signal(tsk, SIGSEGV) &&
+		    printk_ratelimit()) {
+			printk(
+			"%s%s[%d]: segfault at %lx ip %p sp %p error %lx",
+			task_pid_nr(tsk) > 1 ? KERN_INFO : KERN_EMERG,
+			tsk->comm, task_pid_nr(tsk), address,
+			(void *) regs->ip, (void *) regs->sp, error_code);
+			print_vma_addr(" in ", regs->ip);
+			printk("\n");
+		}
+
+		tsk->thread.cr2 = address;
+		/* Kernel addresses are always protection faults */
+		tsk->thread.error_code = error_code | (address >= TASK_SIZE);
+		tsk->thread.trap_no = 14;
+		force_sig_info_fault(SIGSEGV, si_code, address, tsk);
+		return;
+	}
+
+	if (is_f00f_bug(regs, address))
+		return;
+
+	no_context(regs, error_code, address);
+}
+
+static noinline void bad_area_nosemaphore(struct pt_regs *regs,
+			unsigned long error_code, unsigned long address)
+{
+	__bad_area_nosemaphore(regs, error_code, address, SEGV_MAPERR);
+}
+
+static void __bad_area(struct pt_regs *regs,
+			unsigned long error_code, unsigned long address,
+			int si_code)
+{
+	struct mm_struct *mm = current->mm;
+
+	/*
+	 * Something tried to access memory that isn't in our memory map..
+	 * Fix it, but check if it's kernel or user first..
+	 */
+	up_read(&mm->mmap_sem);
+
+	__bad_area_nosemaphore(regs, error_code, address, si_code);
+}
+
+static noinline void bad_area(struct pt_regs *regs,
+			unsigned long error_code, unsigned long address)
+{
+	__bad_area(regs, error_code, address, SEGV_MAPERR);
+}
+
+static noinline void bad_area_access_error(struct pt_regs *regs,
+			unsigned long error_code, unsigned long address)
+{
+	__bad_area(regs, error_code, address, SEGV_ACCERR);
+}
+
+/* TODO: fixup for "mm-invoke-oom-killer-from-page-fault.patch" */
+static void out_of_memory(struct pt_regs *regs,
+			unsigned long error_code, unsigned long address)
+{
+	struct task_struct *tsk = current;
+	struct mm_struct *mm = tsk->mm;
+	/*
+	 * We ran out of memory, or some other thing happened to us that made
+	 * us unable to handle the page fault gracefully.
+	 */
+	up_read(&mm->mmap_sem);
+	if (is_global_init(tsk)) {
+		yield();
+		return;
+	}
+
+	printk("VM: killing process %s\n", tsk->comm);
+	if (error_code & PF_USER)
+		do_group_exit(SIGKILL);
+	no_context(regs, error_code, address);
+}
+
+static void do_sigbus(struct pt_regs *regs,
+			unsigned long error_code, unsigned long address)
+{
+	struct task_struct *tsk = current;
+	struct mm_struct *mm = tsk->mm;
+
+	up_read(&mm->mmap_sem);
+
+	/* Kernel mode? Handle exceptions or die */
+	if (!(error_code & PF_USER))
+		no_context(regs, error_code, address);
+#ifdef CONFIG_X86_32
+	/* User space => ok to do another page fault */
+	if (is_prefetch(regs, error_code, address))
+		return;
+#endif
+	tsk->thread.cr2 = address;
+	tsk->thread.error_code = error_code;
+	tsk->thread.trap_no = 14;
+	force_sig_info_fault(SIGBUS, BUS_ADRERR, address, tsk);
+}
+
+static noinline void mm_fault_error(struct pt_regs *regs,
+		unsigned long error_code, unsigned long address, unsigned int fault)
+{
+	if (fault & VM_FAULT_OOM)
+		out_of_memory(regs, error_code, address);
+	else if (fault & VM_FAULT_SIGBUS)
+		do_sigbus(regs, error_code, address);
+	else
+		BUG();
+}
+
 static int spurious_fault_check(unsigned long error_code, pte_t *pte)
 {
 	if ((error_code & PF_WRITE) && !pte_write(*pte))
@@ -448,8 +642,8 @@ static int spurious_fault_check(unsigned
  * There are no security implications to leaving a stale TLB when
  * increasing the permissions on a page.
  */
-static int spurious_fault(unsigned long address,
-			  unsigned long error_code)
+static noinline int spurious_fault(unsigned long error_code,
+				unsigned long address)
 {
 	pgd_t *pgd;
 	pud_t *pud;
@@ -494,7 +688,7 @@ static int spurious_fault(unsigned long
  *
  * This assumes no large pages in there.
  */
-static int vmalloc_fault(unsigned long address)
+static noinline int vmalloc_fault(unsigned long address)
 {
 #ifdef CONFIG_X86_32
 	unsigned long pgd_paddr;
@@ -573,6 +767,25 @@ static int vmalloc_fault(unsigned long a
 
 int show_unhandled_signals = 1;
 
+static inline int access_error(unsigned long error_code, int write,
+				struct vm_area_struct *vma)
+{
+	if (write) {
+		/* write, present and write, not present */
+		if (unlikely(!(vma->vm_flags & VM_WRITE)))
+			return 1;
+	} else if (unlikely(error_code & PF_PROT)) {
+		/* read, present */
+		return 1;
+	} else {
+		/* read, not present */
+		if (unlikely(!(vma->vm_flags & (VM_READ | VM_EXEC | VM_WRITE))))
+			return 1;
+	}
+
+	return 0;
+}
+
 /*
  * This routine handles page faults.  It determines the address,
  * and the problem, and then passes it off to one of the appropriate
@@ -583,16 +796,12 @@ asmlinkage
 #endif
 void __kprobes do_page_fault(struct pt_regs *regs, unsigned long error_code)
 {
+	unsigned long address;
 	struct task_struct *tsk;
 	struct mm_struct *mm;
 	struct vm_area_struct *vma;
-	unsigned long address;
-	int write, si_code;
+	int write;
 	int fault;
-#ifdef CONFIG_X86_64
-	unsigned long flags;
-	int sig;
-#endif
 
 	tsk = current;
 	mm = tsk->mm;
@@ -601,9 +810,7 @@ void __kprobes do_page_fault(struct pt_r
 	/* get the address */
 	address = read_cr2();
 
-	si_code = SEGV_MAPERR;
-
-	if (notify_page_fault(regs))
+	if (unlikely(notify_page_fault(regs)))
 		return;
 	if (unlikely(kmmio_fault(regs, address)))
 		return;
@@ -638,10 +845,10 @@ void __kprobes do_page_fault(struct pt_r
 		 * Don't take the mm semaphore here. If we fixup a prefetch
 		 * fault we could otherwise deadlock.
 		 */
-		goto bad_area_nosemaphore;
+		bad_area_nosemaphore(regs, error_code, address);
+		return;
 	}
 
-
 	/*
 	 * It's safe to allow irq's after cr2 has been saved and the
 	 * vmalloc fault has been handled.
@@ -657,17 +864,18 @@ void __kprobes do_page_fault(struct pt_r
 
 #ifdef CONFIG_X86_64
 	if (unlikely(error_code & PF_RSVD))
-		pgtable_bad(address, regs, error_code);
+		pgtable_bad(regs, error_code, address);
 #endif
 
 	/*
 	 * If we're in an interrupt, have no user context or are running in an
 	 * atomic region then we must not take the fault.
 	 */
-	if (unlikely(in_atomic() || !mm))
-		goto bad_area_nosemaphore;
+	if (unlikely(in_atomic() || !mm)) {
+		bad_area_nosemaphore(regs, error_code, address);
+		return;
+	}
 
-again:
 	/*
 	 * When running in the kernel we expect faults to occur only to
 	 * addresses in user space.  All other faults represent errors in the
@@ -684,20 +892,26 @@ again:
 	 * source.  If this is invalid we can skip the address space check,
 	 * thus avoiding the deadlock.
 	 */
-	if (!down_read_trylock(&mm->mmap_sem)) {
+	if (unlikely(!down_read_trylock(&mm->mmap_sem))) {
 		if ((error_code & PF_USER) == 0 &&
-		    !search_exception_tables(regs->ip))
-			goto bad_area_nosemaphore;
+		    !search_exception_tables(regs->ip)) {
+			bad_area_nosemaphore(regs, error_code, address);
+			return;
+		}
 		down_read(&mm->mmap_sem);
 	}
 
 	vma = find_vma(mm, address);
-	if (!vma)
-		goto bad_area;
-	if (vma->vm_start <= address)
+	if (unlikely(!vma)) {
+		bad_area(regs, error_code, address);
+		return;
+	}
+	if (likely(vma->vm_start <= address))
 		goto good_area;
-	if (!(vma->vm_flags & VM_GROWSDOWN))
-		goto bad_area;
+	if (unlikely(!(vma->vm_flags & VM_GROWSDOWN))) {
+		bad_area(regs, error_code, address);
+		return;
+	}
 	if (error_code & PF_USER) {
 		/*
 		 * Accessing the stack below %sp is always a bug.
@@ -705,31 +919,25 @@ again:
 		 * and pusha to work.  ("enter $65535,$31" pushes
 		 * 32 pointers and then decrements %sp by 65535.)
 		 */
-		if (address + 65536 + 32 * sizeof(unsigned long) < regs->sp)
-			goto bad_area;
+		if (unlikely(address + 65536 + 32 * sizeof(unsigned long) < regs->sp)) {
+			bad_area(regs, error_code, address);
+			return;
+		}
 	}
-	if (expand_stack(vma, address))
-		goto bad_area;
-/*
- * Ok, we have a good vm_area for this memory access, so
- * we can handle it..
- */
+	if (unlikely(expand_stack(vma, address))) {
+		bad_area(regs, error_code, address);
+		return;
+	}
+
+	/*
+	 * Ok, we have a good vm_area for this memory access, so
+	 * we can handle it..
+	 */
 good_area:
-	si_code = SEGV_ACCERR;
-	write = 0;
-	switch (error_code & (PF_PROT|PF_WRITE)) {
-	default:	/* 3: write, present */
-		/* fall through */
-	case PF_WRITE:		/* write, not present */
-		if (!(vma->vm_flags & VM_WRITE))
-			goto bad_area;
-		write++;
-		break;
-	case PF_PROT:		/* read, present */
-		goto bad_area;
-	case 0:			/* read, not present */
-		if (!(vma->vm_flags & (VM_READ | VM_EXEC | VM_WRITE)))
-			goto bad_area;
+	write = error_code & PF_WRITE;
+	if (unlikely(access_error(error_code, write, vma))) {
+		bad_area_access_error(regs, error_code, address);
+		return;
 	}
 
 	/*
@@ -739,11 +947,8 @@ good_area:
 	 */
 	fault = handle_mm_fault(mm, vma, address, write);
 	if (unlikely(fault & VM_FAULT_ERROR)) {
-		if (fault & VM_FAULT_OOM)
-			goto out_of_memory;
-		else if (fault & VM_FAULT_SIGBUS)
-			goto do_sigbus;
-		BUG();
+		mm_fault_error(regs, error_code, address, fault);
+		return;
 	}
 	if (fault & VM_FAULT_MAJOR)
 		tsk->maj_flt++;
@@ -761,139 +966,6 @@ good_area:
 	}
 #endif
 	up_read(&mm->mmap_sem);
-	return;
-
-/*
- * Something tried to access memory that isn't in our memory map..
- * Fix it, but check if it's kernel or user first..
- */
-bad_area:
-	up_read(&mm->mmap_sem);
-
-bad_area_nosemaphore:
-	/* User mode accesses just cause a SIGSEGV */
-	if (error_code & PF_USER) {
-		/*
-		 * It's possible to have interrupts off here.
-		 */
-		local_irq_enable();
-
-		/*
-		 * Valid to do another page fault here because this one came
-		 * from user space.
-		 */
-		if (is_prefetch(regs, address, error_code))
-			return;
-
-		if (is_errata100(regs, address))
-			return;
-
-		if (show_unhandled_signals && unhandled_signal(tsk, SIGSEGV) &&
-		    printk_ratelimit()) {
-			printk(
-			"%s%s[%d]: segfault at %lx ip %p sp %p error %lx",
-			task_pid_nr(tsk) > 1 ? KERN_INFO : KERN_EMERG,
-			tsk->comm, task_pid_nr(tsk), address,
-			(void *) regs->ip, (void *) regs->sp, error_code);
-			print_vma_addr(" in ", regs->ip);
-			printk("\n");
-		}
-
-		tsk->thread.cr2 = address;
-		/* Kernel addresses are always protection faults */
-		tsk->thread.error_code = error_code | (address >= TASK_SIZE);
-		tsk->thread.trap_no = 14;
-		force_sig_info_fault(SIGSEGV, si_code, address, tsk);
-		return;
-	}
-
-	if (is_f00f_bug(regs, address))
-		return;
-
-no_context:
-	/* Are we prepared to handle this kernel fault?  */
-	if (fixup_exception(regs))
-		return;
-
-	/*
-	 * X86_32
-	 * Valid to do another page fault here, because if this fault
-	 * had been triggered by is_prefetch fixup_exception would have
-	 * handled it.
-	 *
-	 * X86_64
-	 * Hall of shame of CPU/BIOS bugs.
-	 */
-	if (is_prefetch(regs, address, error_code))
-		return;
-
-	if (is_errata93(regs, address))
-		return;
-
-/*
- * Oops. The kernel tried to access some bad page. We'll have to
- * terminate things with extreme prejudice.
- */
-#ifdef CONFIG_X86_32
-	bust_spinlocks(1);
-#else
-	flags = oops_begin();
-#endif
-
-	show_fault_oops(regs, error_code, address);
-
-	tsk->thread.cr2 = address;
-	tsk->thread.trap_no = 14;
-	tsk->thread.error_code = error_code;
-
-#ifdef CONFIG_X86_32
-	die("Oops", regs, error_code);
-	bust_spinlocks(0);
-	do_exit(SIGKILL);
-#else
-	sig = SIGKILL;
-	if (__die("Oops", regs, error_code))
-		sig = 0;
-	/* Executive summary in case the body of the oops scrolled away */
-	printk(KERN_EMERG "CR2: %016lx\n", address);
-	oops_end(flags, regs, sig);
-#endif
-
-/*
- * We ran out of memory, or some other thing happened to us that made
- * us unable to handle the page fault gracefully.
- */
-out_of_memory:
-	up_read(&mm->mmap_sem);
-	if (is_global_init(tsk)) {
-		yield();
-		/*
-		 * Re-lookup the vma - in theory the vma tree might
-		 * have changed:
-		 */
-		goto again;
-	}
-
-	printk("VM: killing process %s\n", tsk->comm);
-	if (error_code & PF_USER)
-		do_group_exit(SIGKILL);
-	goto no_context;
-
-do_sigbus:
-	up_read(&mm->mmap_sem);
-
-	/* Kernel mode? Handle exceptions or die */
-	if (!(error_code & PF_USER))
-		goto no_context;
-#ifdef CONFIG_X86_32
-	/* User space => ok to do another page fault */
-	if (is_prefetch(regs, address, error_code))
-		return;
-#endif
-	tsk->thread.cr2 = address;
-	tsk->thread.error_code = error_code;
-	tsk->thread.trap_no = 14;
-	force_sig_info_fault(SIGBUS, BUS_ADRERR, address, tsk);
 }
 
 DEFINE_SPINLOCK(pgd_lock);
#24 lore raw

Re: [git pull] cpus4096 tree, part 3

In reply to Nick Piggin
Quoted 21 lines · Nick Piggin
Quoted 9 lines · Ingo Molnar
Quoted 6 lines · Linus Torvalds

What happened to Nick's cleanup patch to do_page_fault (a month or two ago? I complained about some of the issues in his first version and asked for some further cleanups, but I think that whole discussion ended with him saying "I am going to add those changes that you suggested (in fact, I already have)".

Quoted 6 lines
Quoted 2 lines

And then I didn't see anything further. Maybe I just missed the end result. Or maybe we have it in some -mm branch or something?

they would have been in tip/x86/mm and would be upstream now had Nick re-sent a v2 series but that never happened. I think they might have fallen victim to a serious attention deficit caused by the SLQB patch ;-)

Well, I already added Linus's suggestions but didn't submit it because there was a bit of work going on in that file as far as I could see, both in the x86 tree and in -mm:

Quoted 8 lines

(http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.28-rc2/2.6.28-rc2-mm1/broken-out/mm-invoke-oom-killer-from-page-fault.patch)

It isn't a big deal to resolve either way, but I don't want to make Andrew's life harder.

[Yes OK now I'm the guilty one of pushing in an x86 patch not via the x86 tree ;) This one is easy to break in pieces, but I didn't want to create a dependency between the trees]

That's OK, and the oom-killer patch impact on x86 was incidental, so it was correct to push it via -mm IMO.

Now that the bits that went in via Andrew's tree upstream, there's a handful of new conflicts in the patch - so would you mind to (re-)send a merged up patch against latest -git?

Ingo

#46 lore raw
In reply to Rusty Russell
Quoted 4 lines · Rusty Russell

commit d036e67b40f52bdd95392390108defbac7e53837

Author: Rusty Russell <[email protected]>
Date:   Thu Jan 1 10:12:26 2009 +1030
Quoted 9 lines

cpumask: convert kernel/irq

Impact: Reduce stack usage, use new cpumask API. ALPHA mod!

Main change is that irq_default_affinity becomes a cpumask_var_t, so treat it as a pointer (this effects alpha).

Which contains:

Quoted 8 lines
kernel/irq/manage.c +0 -0
index 61c4a9b..cd0cd8d 100644
@@ -16,8 +16,15 @@
 #include "internals.h"
 
 #ifdef CONFIG_SMP

kernel/irq/manage.c line 27

Quoted 10 lines
kernel/irq/manage.c +8 -1
+cpumask_var_t irq_default_affinity;
 
-cpumask_t irq_default_affinity = CPU_MASK_ALL;
+static int init_irq_default_affinity(void)
+{
+       alloc_cpumask_var(&irq_default_affinity, GFP_KERNEL);
+       cpumask_setall(irq_default_affinity);
+       return 0;
+}
+core_initcall(init_irq_default_affinity);

I think core_initcall is too late to be initializing irq_default_affinity. This happens way after init_IRQ() is called and for my target (mips/cavium_octeon) after the timer and SMP related irqs are setup.

I had been setting irq_default_affinity in init_IRQ(), and I could probably do it later with no real problem, but this seems wrong to me. Data that is potentially used in interrupt configuration and processing should be initialized before it is used.

David Daney

maintainer
#16 lore raw

Re: [git pull] cpus4096 tree, part 3

In reply to Nick Piggin
Quoted 38 lines · On Mon, 5 Jan 2009 02:16:30 +0100 Nick Piggin <[email protected]> wrote:

Really cc linux-arch this time

Quoted 35 lines · Nick Piggin
Quoted 9 lines · Ingo Molnar
Quoted 6 lines · Linus Torvalds

What happened to Nick's cleanup patch to do_page_fault (a month or two ago? I complained about some of the issues in his first version and asked for some further cleanups, but I think that whole discussion ended with him saying "I am going to add those changes that you suggested (in fact, I already have)".

Quoted 6 lines
Quoted 2 lines

And then I didn't see anything further. Maybe I just missed the end result. Or maybe we have it in some -mm branch or something?

they would have been in tip/x86/mm and would be upstream now had Nick re-sent a v2 series but that never happened. I think they might have fallen victim to a serious attention deficit caused by the SLQB patch ;-)

Well, I already added Linus's suggestions but didn't submit it because there was a bit of work going on in that file as far as I could see, both in the x86 tree and in -mm:

(http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.28-rc2/2.6.28-rc2-mm1/broken-out/mm-invoke-oom-killer-from-page-fault.patch)

It isn't a big deal to resolve either way, but I don't want to make Andrew's life harder.

[Yes OK now I'm the guilty one of pushing in an x86 patch not via the x86 tree ;) This one is easy to break in pieces, but I didn't want to create a dependency between the trees]

I didn't really consider it to be urgent, so I was waiting for that patch to go in, but I was still hoping to get this into 2.6.29... This is what it looks like now with your suggestions, and just merged it to your current tree (untested).

Quoted 7 lines
Quoted 4 lines

I'll cc the linux-arch list here too, because it might be nice to keep these things as structurally similar as possible (and they'll all want to look at the -mm patch above, although I'll probably end up having to write the patches!).

---
Optimise x86's do_page_fault (C entry point for the page fault path).

It took rather a lot of hunting to find this email. Please do try to make the email subject match the final patch's title?

Quoted 136 lines

* This routine handles page faults. It determines the address,
* and the problem, and then passes it off to one of the appropriate

(quoted diff) +37 -37
@@ -583,16 +796,12 @@ asmlinkage
 #endif
 void __kprobes do_page_fault(struct pt_regs *regs, unsigned long error_code)
 {
+	unsigned long address;
 	struct task_struct *tsk;
 	struct mm_struct *mm;
 	struct vm_area_struct *vma;
-	unsigned long address;
-	int write, si_code;
+	int write;
 	int fault;
-#ifdef CONFIG_X86_64
-	unsigned long flags;
-	int sig;
-#endif
 
 	tsk = current;
 	mm = tsk->mm;
@@ -601,9 +810,7 @@ void __kprobes do_page_fault(struct pt_r
 	/* get the address */
 	address = read_cr2();
 
-	si_code = SEGV_MAPERR;
-
-	if (notify_page_fault(regs))
+	if (unlikely(notify_page_fault(regs)))
 		return;
 	if (unlikely(kmmio_fault(regs, address)))
 		return;
@@ -638,10 +845,10 @@ void __kprobes do_page_fault(struct pt_r
 		 * Don't take the mm semaphore here. If we fixup a prefetch
 		 * fault we could otherwise deadlock.
 		 */
-		goto bad_area_nosemaphore;
+		bad_area_nosemaphore(regs, error_code, address);
+		return;
 	}
 
-
 	/*
 	 * It's safe to allow irq's after cr2 has been saved and the
 	 * vmalloc fault has been handled.
@@ -657,17 +864,18 @@ void __kprobes do_page_fault(struct pt_r
 
 #ifdef CONFIG_X86_64
 	if (unlikely(error_code & PF_RSVD))
-		pgtable_bad(address, regs, error_code);
+		pgtable_bad(regs, error_code, address);
 #endif
 
 	/*
 	 * If we're in an interrupt, have no user context or are running in an
 	 * atomic region then we must not take the fault.
 	 */
-	if (unlikely(in_atomic() || !mm))
-		goto bad_area_nosemaphore;
+	if (unlikely(in_atomic() || !mm)) {
+		bad_area_nosemaphore(regs, error_code, address);
+		return;
+	}
 
-again:
 	/*
 	 * When running in the kernel we expect faults to occur only to
 	 * addresses in user space.  All other faults represent errors in the
@@ -684,20 +892,26 @@ again:
 	 * source.  If this is invalid we can skip the address space check,
 	 * thus avoiding the deadlock.
 	 */
-	if (!down_read_trylock(&mm->mmap_sem)) {
+	if (unlikely(!down_read_trylock(&mm->mmap_sem))) {
 		if ((error_code & PF_USER) == 0 &&
-		    !search_exception_tables(regs->ip))
-			goto bad_area_nosemaphore;
+		    !search_exception_tables(regs->ip)) {
+			bad_area_nosemaphore(regs, error_code, address);
+			return;
+		}
 		down_read(&mm->mmap_sem);
 	}
 
 	vma = find_vma(mm, address);
-	if (!vma)
-		goto bad_area;
-	if (vma->vm_start <= address)
+	if (unlikely(!vma)) {
+		bad_area(regs, error_code, address);
+		return;
+	}
+	if (likely(vma->vm_start <= address))
 		goto good_area;
-	if (!(vma->vm_flags & VM_GROWSDOWN))
-		goto bad_area;
+	if (unlikely(!(vma->vm_flags & VM_GROWSDOWN))) {
+		bad_area(regs, error_code, address);
+		return;
+	}
 	if (error_code & PF_USER) {
 		/*
 		 * Accessing the stack below %sp is always a bug.
@@ -705,31 +919,25 @@ again:
 		 * and pusha to work.  ("enter $65535,$31" pushes
 		 * 32 pointers and then decrements %sp by 65535.)
 		 */
-		if (address + 65536 + 32 * sizeof(unsigned long) < regs->sp)
-			goto bad_area;
+		if (unlikely(address + 65536 + 32 * sizeof(unsigned long) < regs->sp)) {
+			bad_area(regs, error_code, address);
+			return;
+		}
 	}
-	if (expand_stack(vma, address))
-		goto bad_area;
-/*
- * Ok, we have a good vm_area for this memory access, so
- * we can handle it..
- */
+	if (unlikely(expand_stack(vma, address))) {
+		bad_area(regs, error_code, address);
+		return;
+	}
+
+	/*
+	 * Ok, we have a good vm_area for this memory access, so
+	 * we can handle it..
+	 */
 good_area:
-	si_code = SEGV_ACCERR;
-	write = 0;
-	switch (error_code & (PF_PROT|PF_WRITE)) {
-	default:	/* 3: write, present */
-		/* fall through */
-	case PF_WRITE:		/* write, not present */

Review again: · line 937

Quoted 10 lines
(quoted diff) +1 -9
-		if (!(vma->vm_flags & VM_WRITE))
-			goto bad_area;
-		write++;
-		break;
-	case PF_PROT:		/* read, present */
-		goto bad_area;
-	case 0:			/* read, not present */
-		if (!(vma->vm_flags & (VM_READ | VM_EXEC | VM_WRITE)))
-			goto bad_area;
+	write = error_code & PF_WRITE;

What's going on here? We set `error_code' to PF_WRITE, which is some x86-specific thing.

Review good_area: · line 948

Quoted 9 lines
(quoted diff) +3 -0
+	if (unlikely(access_error(error_code, write, vma))) {
+		bad_area_access_error(regs, error_code, address);
+		return;
 	}
 
 	/*
@@ -739,11 +947,8 @@ good_area:
 	 */
 	fault = handle_mm_fault(mm, vma, address, write);

and then pass it into handle_mm_fault(), which is expecting a bunch of flags in the FAULT_FLAG_foo domain.

IOW, the code will accidentally set FAULT_FLAG_NONLINEAR!.

Methinks we want something like this,

arch/x86/mm/fault.c renamed from arch/x86/mm/fault.c~fix-x86-optimise-x86s-do_page_fault-c-entry-point-for-the-page-fault-path +1 -1
@@ -942,7 +942,7 @@ void __kprobes do_page_fault(struct pt_r
 	 * we can handle it..
 	 */
 good_area:
-	write = error_code & PF_WRITE;
+	write = (error_code & PF_WRITE) ? FAULT_FLAG_WRITE : 0;
 	if (unlikely(access_error(error_code, write, vma))) {
 		bad_area_access_error(regs, error_code, address);
 		return;

_

but why did the current code pass testing at all??

#17 lore raw

Re: [git pull] cpus4096 tree, part 3

Quoted 4 lines · Andrew Morton

Review

Quoted 1 line
(quoted diff) +1 -0
+	write = error_code & PF_WRITE;

What's going on here? We set `error_code' to PF_WRITE, which is some x86-specific thing.

No. We set "write" to non-zero if it was a write fault.

Quoted 4 lines
Quoted 1 line

fault = handle_mm_fault(mm, vma, address, write);

and then pass it into handle_mm_fault(), which is expecting a bunch of flags in the FAULT_FLAG_foo domain.

No. "handle_mm_fault()" takes an integer that is non-zero if it's a write, zero if it's a read. That's how it has _always_ worked.

I don't see where you find that FAULT_FLAG_foo thing. That's much deeper down, when people do things like

unsigned int flags = FAULT_FLAG_NONLINEAR |
(write_access ? FAULT_FLAG_WRITE : 0);

based on that whole "write_access" flag.

Linus

maintainer
#18 lore raw

Re: [git pull] cpus4096 tree, part 3

Quoted 18 lines · On Mon, 26 Jan 2009 11:09:59 -0800 (PST) Linus Torvalds…
Quoted 4 lines · Andrew Morton

Review

Quoted 1 line
(quoted diff) +1 -0
+	write = error_code & PF_WRITE;

What's going on here? We set `error_code' to PF_WRITE, which is some x86-specific thing.

No. We set "write" to non-zero if it was a write fault.

Quoted 4 lines
Quoted 1 line

fault = handle_mm_fault(mm, vma, address, write);

and then pass it into handle_mm_fault(), which is expecting a bunch of flags in the FAULT_FLAG_foo domain.

No. "handle_mm_fault()" takes an integer that is non-zero if it's a write, zero if it's a read. That's how it has _always_ worked.

Quoted 8 lines

I don't see where you find that FAULT_FLAG_foo thing. That's much deeper down, when people do things like

unsigned int flags = FAULT_FLAG_NONLINEAR |
(write_access ? FAULT_FLAG_WRITE : 0);

based on that whole "write_access" flag.

OK, thanks. It's actually page_fault-retry-with-nopage_retry.patch which got those things confused, and then confused me. I'll go address that in the other thread..

#19 lore raw

Re: [git pull] cpus4096 tree, part 3

In reply to Andrew Morton
Quoted 1 line · Andrew Morton

but why did the current code pass testing at all??

i queued it up a week ago and beyond a same-day breakage i reported to Nick (and which he fixed) this commit was problem-free and passed all testing here.

Does it cause problems for you? If yes then please describe the kind of problems.

Note: i see that -mm modifies a few other details of the x86 pagefault handling path (there a pagefault-retry patch in there) - so there might be contextual interactions there. But this particular cleanup/improvement from Nick is working fine on a wide range of systems here.

Btw., regarding pagefault retry. The bits that are in -mm currently i find a bit ugly:

Quoted 19 lines

+++ a/arch/x86/mm/fault.c

(quoted diff) +2 -1
@@ -799,7 +799,7 @@ void __kprobes do_page_fault(struct pt_r
 	struct vm_area_struct *vma;
 	int write;
 	int fault;
-	unsigned int retry_flag = FAULT_FLAG_RETRY;
+	int retry_flag = 1;
 
 	tsk = current;
 	mm = tsk->mm;
@@ -951,6 +951,7 @@ good_area:
 	}
 
 	write |= retry_flag;
+
 	/*
 	 * If for any reason at all we couldn't handle the fault,
 	 * make sure we exit gracefully rather than endlessly redo

Review good_area: · line 977

Quoted 10 lines
(quoted diff) +2 -2
@@ -969,8 +970,8 @@ good_area:
 	 * be removed or changed after the retry.
 	 */
 	if (fault & VM_FAULT_RETRY) {
-		if (write & FAULT_FLAG_RETRY) {
-			retry_flag &= ~FAULT_FLAG_RETRY;
+		if (retry_flag) {
+			retry_flag = 0;
 			goto retry;
 		}
 		BUG();

as this complicates every architecture with a 'can the fault be retried' logic and open-coded retry loop.

But that logic is rather repetitive and once an architecture filters out all its special in-kernel sources of faults and the hw quirks it has, the handling of pte faults is rather generic and largely offloaded into handle_pte_fault() already.

So when this patch was submitted a few weeks ago i suggested that retry should be done purely in mm/memory.c instead, and the low level code should at most be refactored to suit this method, but not complicated any further.

Any deep reasons for why such a more generic approach is not desirable?

Ingo

maintainer
#20 lore raw

Re: [git pull] cpus4096 tree, part 3

Quoted 43 lines · On Mon, 26 Jan 2009 21:09:57 +0100 Ingo Molnar <[email protected]> wrote:

...

Btw., regarding pagefault retry. The bits that are in -mm currently i find a bit ugly:

Quoted 19 lines

+++ a/arch/x86/mm/fault.c

(quoted diff) +2 -1
@@ -799,7 +799,7 @@ void __kprobes do_page_fault(struct pt_r
 	struct vm_area_struct *vma;
 	int write;
 	int fault;
-	unsigned int retry_flag = FAULT_FLAG_RETRY;
+	int retry_flag = 1;
 
 	tsk = current;
 	mm = tsk->mm;
@@ -951,6 +951,7 @@ good_area:
 	}
 
 	write |= retry_flag;
+
 	/*
 	 * If for any reason at all we couldn't handle the fault,
 	 * make sure we exit gracefully rather than endlessly redo

Review good_area: · line 977

Quoted 10 lines
(quoted diff) +2 -2
@@ -969,8 +970,8 @@ good_area:
 	 * be removed or changed after the retry.
 	 */
 	if (fault & VM_FAULT_RETRY) {
-		if (write & FAULT_FLAG_RETRY) {
-			retry_flag &= ~FAULT_FLAG_RETRY;
+		if (retry_flag) {
+			retry_flag = 0;
 			goto retry;
 		}
 		BUG();

as this complicates every architecture with a 'can the fault be retried' logic and open-coded retry loop.

But that logic is rather repetitive and once an architecture filters out all its special in-kernel sources of faults and the hw quirks it has, the handling of pte faults is rather generic and largely offloaded into handle_pte_fault() already.

Quoted 7 lines

So when this patch was submitted a few weeks ago i suggested that retry should be done purely in mm/memory.c instead, and the low level code should at most be refactored to suit this method, but not complicated any further.

Any deep reasons for why such a more generic approach is not desirable?

Let's cc the people who wrote it.

#22 lore raw

Re: [git pull] cpus4096 tree, part 3

In reply to a message not in the archive
Quoted 2 lines · Ying Han

Thank you Ingo and Andrew for the comments. I will take a look into it ASAP and updates it here.

Note, my objection wasnt a hard NAK - just an observation. If all things considered Andrew still favors the VM_FAULT_RETRY approach then that's fine too i guess.

It's just that a quick look gave me the feeling of a retry flag tacked on to an existing codepath [and all the micro-overhead and complexity that this brings], instead of a clean refactoring of pagefault handling functionality into a higher MM level retry loop.

So the alternative has to be looked at and rejected because it's technically inferior - not because it's more difficult to implement. (which it certainly is)

Ingo

maintainer
#23 lore raw

Re: [git pull] cpus4096 tree, part 3

Quoted 10 lines · On Tue, 27 Jan 2009 00:21:39 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 2 lines · Ying Han

Thank you Ingo and Andrew for the comments. I will take a look into it ASAP and updates it here.

Note, my objection wasnt a hard NAK - just an observation. If all things considered Andrew still favors the VM_FAULT_RETRY approach then that's fine too i guess.

Quoted 9 lines

It's just that a quick look gave me the feeling of a retry flag tacked on to an existing codepath [and all the micro-overhead and complexity that this brings], instead of a clean refactoring of pagefault handling functionality into a higher MM level retry loop.

So the alternative has to be looked at and rejected because it's technically inferior - not because it's more difficult to implement. (which it certainly is)

I have wobbly feelings about this patch. There are your issues, and a long string of problems and fixes. And my recent half-assed linux-next-related fix which I didn't really think about.

It all needs a revisit/rereview/reunderstand cycle.