cpumask v4 - bitmap and cpumask cleanup
Subsystems (4)
PATCH 1/1 cpumask v4 - bitmap and cpumask cleanup
Paul Jackson [email protected] 21 April 2004 Bitmap and Cpumask Cleanup
This is the fourth version of my cleanup of cpumasks.
This set of 17 patches applies against a vanilla 2.6.5 kernel.
The actual patches will be posted in another hour or two, as followups to this initial post.
Primay goals:
The primary goal of this patch set is to simplify the code for cpumask_t and (later) nodemask_t, make them easier to use, and reduce code duplication.
Several flavors of cpumask have been reduced to one, with some local special handling to optimize for that vast majority of systems which have less than 32 CPUs.
The bitmap operations upon which cpumasks depend have been optimized a bit more, with a more careful mix of inline and outofline code.
By simplifying masks to a single file, it should also be easier to add other such mask types, such as the nodemask that Matthew Dobson has waiting in the wings, just by copying cpumask.h and making a few global edits.
Compared to the third version (8 April) of this patch set:
1) Doesn't include Matthew Dobson's nodemask patch. Matthew will submit that patch separately, after the fate of this cleanup patch is resolved.
2) Has one typo fixed in a comment.
What's next:
I will seek out the feedback of the architecture maintainers over the next few days to this patch, and continue local testing. Then it should be ready to submit to Andrew.
What's new (comparing 2.6.5 to this patch set):
1) Some 27 files matching the pattern include/*/*mask*.h are replaced with the single file include/linux/cpumask.h The variety of arch-specific redirect headers for various cpumask implementation flavors is gone.
2) The bitmap operations (bitmap.h, bitmap.c) are optimized for systems of less than 32 CPUs.
3) The cpumask operations are now just a thin layer on top of the bitmask and a few other operations. A cpumask is a bitmask of exactly NR_CPUS bits, wrapped in a structure.
4) bitmap_complement and cpumask_complement now take two args, source and target, instead of working in place.
5) Some uses of these macros elsewhere in the kernel were fine tuned.
6) On ia64, find_first_bit and find_first_zero_bit are uninlined - saving quite a bit of kernel text space. The architectures: alpha, parisc, ppc, sh, sparc, sparc64 have this same bit find code, and might also want to uninline.
7) Comments in bitmap.h and cpumask.h list available ops for ease of browsing.
8) The MASK_ALL macro zeros out unused bits on multiword bitmaps.
9) This patch includes Bill Irwin's very recent rewrite of the bitmap_shift operators to avoid assuming some fixed upper bound on bitmap sizes.
10) A few more mask macros have been added, to make it easier to code mask manipulations correctly and easily. They provide xor, andnot, intersects and subset operators.
Bug fixes:
1) *_complement macros don't leave unused high bits set
2) MASK_ALL for sizes > 1 word, but not exact word multiple, doesn't have unused high bits set
3) Explicit, documented semantics for handling these unused high bits.
4) A few missing const attributes in bitmap & bitops added.
5) The (Hamming) cpumask weight macros were using the bitops hweight*() macros, which don't mask high unused bits - fixed to use the bitmap weight macro which does this masking.
Do to the rather limited use so far of cpumask macros, I am not aware of anything that these bugs would actually break in current mm or linus kernels.
Testing so far:
Kernels have been built for i386 and ia64, and booted and minimally tested for ia64. Kernel text size has been compared and found to typically get a little smaller for these two arch's. Correct function of bit operations has been tested for a variety of NR_CPUS values in a user level framework. Gcc compiler versions 2.95.3, 3.2.3 and 3.3.2 have been built successfully.
Risks:
This patch set makes changes in bitmap and cpumask handling code in several architectures. I could have broken something.
Other affected arch's, need to be tested and reviewed. The other arch's seeing the most changes were i386, ppc64 and x86_64, but specific changes to alpha, arm, mips, sparc64 and sh aloo need review.
This patch set removes several more optimized variants of mask, which were targeted for optimum performance on various architectures, and replaces that with essentially a single variant. Preliminary analysis on the one architecture I cared most about (large ia64) shows no noticeable degradation in the quality of the code generated for mask operations, but other architectures or more demanding uses may require further optimization work.
This patch set might have dependencies on specific gcc or toolchain versions. Portions have worked with both gcc 2.95.3 and 3.3.2 inside a user level scaffolding on i386. Kernels have been built and booted on ia64 using gcc 3.2.3 and built on i386 using gcc 3.3.2. But non-trivial use is made of gcc extensions, so certain archs or toolchains might not build, or might not produce sufficiently optimal code, or might not even work.
This patch set changes the semantics of cpumasks slightly. If all mask operations are performed with correct parameters, and no bits are intentionally set outside the range of bits covered by such a mask (typically this means outside the range [0 ... NR_CPUS-1] for cpumask_t), then there should be no noticeable difference. Code that (improperly) sets bits outside that valid range may notice differences in when such bits affect the results of other mask operations.
Patches 13 through 16 (see list of patches, below) in particular attempt to "cleanup" some cpumask manipulation code in various sparc and intel arch's. These patches are not essential - if they get in the way of other work, or are too much distraction, or break something, any of them can be dropped. At a minimum, they require review and approval from the affected arch's.
================ Patch Details ================
Here's what each patch does:
[Patch 1 of 17] cpumask v4 - Document bitmap.c bit model.
Document the bitmap bit model, including handling of unused bits, and operation preconditions and postconditions.
[Patch 2 of 17] cpumask v4 - Dont generate nonzero unused bits in bitmap
Tighten up bitmap so it does not generate nonzero bits in the unused tail if it is not given any on input.
[Patch 3 of 17] cpumask v4 - New bitmap operators and two op complement
Add intersects, subset, xor and andnot operators. Change bitmap_complement to take two operands.
[Patch 4 of 17] cpumask v4 - two missing 'const' qualifiers in bitops/bitmap
Add a couple of missing 'const' qualifiers on
bitops test_bit and bitmap_equal args.
[Patch 5 of 17] cpumask v4 - Optimize and extend bitmap.
This bitmap improvements make it a suitable basis for fully supporting cpumask_t and nodemask_t. Inline macros with compile-time checks enable generating tight code on both small and large systems (large meaning cpumask_t requires more than one unsigned long's worth of bits).
The existing bitmap_<op> macros in lib/bitmap.c are renamed to __bitmap_<op>, and wrappers for each bitmap_<op> are exposed in include/linux/bitmap.h
This patch _includes_ Bill Irwins rewrite of the bitmap_shift operators to not require a fixed length intermediate bitmap.
Improved comments list each available operator for easy browsing.
[Patch 6 of 17] cpumask v4 - Uninline find_next_bit on ia64
Move the page of code (~700 bytes of instructions) for find_next_bit and find_next_zero_bit from inline in include/asm-ia64/bitops.h to a real function in arch/ia64/lib/bitops.c, leaving a declaration and macro wrapper behind.
The other arch's with almost this same code might want to also uninline it: alpha, parisc, ppc, sh, sparc, sparc64.
These are too big to inline.
[Patch 7 of 17] cpumask v4 - Rewrite cpumask.h to use bitmap directly.
Major rewrite of cpumask to use a single implementation, as a struct-wrapped bitmap.
This patch leaves some 26 include/asm-*/cpumask*.h header files orphaned - to be removed next patch.
Some nine cpumask macros for const variants and to coerce and promote between an unsigned long and a cpumask are obsolete. Simple emulation wrappers are provided in this patch, which can be removed once each of the 3 archs (i386, ppc64, x86_64) using them are recoded in follow-on patches to not need them.
The CPU_MASK_ALL macro now avoids leaving possible garbage one bits in any unused portion of the high word.
An improved comment lists all available operators, for convenient browsing.
[Patch 8 of 17] cpumask v4 - Remove 26 no longer used cpumask headers.
With the cpumask rewrite in the previous patch, these various include/asm-*/cpumask*.h headers are no longer used.
[Patch 9 of 17] cpumask v4 - Recode obsolete cpumask macros - arch i386
Remove by recoding all uses of the obsolete cpumask const, coerce and promote macros.
[Patch 10 of 17] cpumask v4 - Recode obsolete cpumask macros - arch ppc64
Remove by recoding all uses of the obsolete cpumask const, coerce and promote macros.
[Patch 11 of 17] cpumask v4 - Recode obsolete cpumask macros - arch x86_64
Remove by recoding all uses of the obsolete cpumask const, coerce and promote macros.
[Patch 12 of 17] cpumask v4 - Remove obsolete emulation from cpumask.h
Now that the emulation of the obsolete cpumask macros is no longer needed, remove it from cpumask.h
[Patch 13 of 17] cpumask v4 - Simplify some sparc64 cpumask loop code
Make use of for_each_cpu_mask() macro to simplify and optimize a couple of sparc64 per-CPU loops. This code change has _not_ been tested or reviewed. Feedback welcome. There is non-trivial risk that I still don't understand the logic here.
[Patch 14 of 17] cpumask v4 - Optimize i386 cpumask macro usage.
Optimize a bit of cpumask code for asm-i386/mach-es7000 Code untested, unreviewed. Feedback welcome.
[Patch 15 of 17] cpumask v4 - Convert physids_complement() to use both args
Provide for specifying distinct source and dest args to the physids_complement(). No one actually uses this macro yet. The physid_mask type would be a good candidate to convert to using this new mask ADT as a base.
[Patch 16 of 17] cpumask v4 - Remove cpumask hack from asm-x86_64/topology.h
This file had the cpumask cpu_online_map as type unsigned long, instead of type cpumask_t, for no good reason that I could see. So I changed it. Everywhere else, cpu_online_map is already of type cpumask_t.
[Patch 17 of 17] cpumask v4 - Cpumask code clarification in kernel/sched.c
Clarify and slightly optimize set_cpus_allowed() cpumask check
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373PATCH 1/1 cpumask v4 - Recode obsolete cpumask macros - arch x86_64
mask11-cpumask-x86-64-fixup - Remove/recode obsolete cpumask macros from arch x86_64
Remove by recoding all uses of the obsolete cpumask const, coerce and promote macros.
Diffstat Patch_9_of_23:
arch/x86_64/kernel/io_apic.c | 2 +- arch/x86_64/kernel/pci-gart.c | 4 ++-- arch/x86_64/kernel/smp.c | 2 +- include/asm-x86_64/smp.h | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-)
arch/x86_64/kernel/io_apic.c +1 -1
@@ -1368,7 +1368,7 @@unsigned long flags;unsigned int dest; - dest =cpu_mask_to_apicid (mk_cpumask_const (mask)); + dest =cpu_mask_to_apicid (mask);/* * Only the first 8 bits are valid.
arch/x86_64/kernel/pci-gart.c +2 -2
@@ -149,7 +149,7 @@ {unsigned long flags;int bus = dev ? dev->bus->number :-1 ; -cpumask_const_t bus_cpumask =pcibus_to_cpumask (bus); +cpumask_t bus_cpumask =pcibus_to_cpumask (bus);int flushed =0 ;int i;
@@ -159,7 +159,7 @@u32 w;if (!northbridges[i])continue ; -if (bus >=0 && !(cpu_isset_const (i, bus_cpumask))) +if (bus >=0 && !(cpu_isset (i, bus_cpumask)))continue ;pci_write_config_dword (northbridges[i],0x9c , northbridge_flush_word[i] |1 );
arch/x86_64/kernel/smp.c +1 -1
@@ -94,7 +94,7 @@static inline void send_IPI_mask (cpumask_t cpumask,int vector) { -unsigned long mask =cpus_coerce (cpumask); +unsigned long mask =cpus_addr (cpumask)[0 ];unsigned long cfg;unsigned long flags;
include/asm-x86_64/smp.h +2 -2
@@ -95,9 +95,9 @@#define TARGET_CPUS 1#ifndef ASSEMBLY -static inline unsigned int cpu_mask_to_apicid (cpumask_const_t cpumask) +static inline unsigned int cpu_mask_to_apicid (cpumask_t cpumask) { -return cpus_coerce_const (cpumask); +return cpus_addr (cpumask)[0 ]; }#endif
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373PATCH 1/1 cpumask v4 - Remove obsolete emulation from cpumask.h
In reply to Paul Jacksonmask12-rm-old-cpumask-emul - Remove obsolete cpumask emulation from cpumask.h
Now that the emulation of the obsolete cpumask macros is no longer needed, remove it from cpumask.h
include/linux/cpumask.h +0 -12
@@ -315,16 +315,4 @@#define for_each_online_cpu (cpu) \ for_each_cpu_mask(cpu, cpu_online_map) -/* Begin obsolete cpumask operator emulation */ -#define cpu_isset_const (a,b) cpu_isset(a,b) -#define cpumask_const_t cpumask_t -#define cpus_coerce (m) (cpus_addr(m)[0]) -#define cpus_coerce_const cpus_coerce -#define cpus_promote (x) ({ cpumask_t m; m.bits[0] = x; m; }) -#define cpus_weight_const cpus_weight -#define first_cpu_const first_cpu -#define mk_cpumask_const (x) x -#define next_cpu_const next_cpu -/* End of obsolete cpumask operator emulation */ -#endif /* __LINUX_CPUMASK_H */
%diffstat
cpumask.h | 12 ------------ 1 files changed, 12 deletions(-)
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373PATCH 1/1 cpumask v4 - Cpumask code clarification in kernel/sched.c
In reply to Paul Jacksonmask17-cpumask-sched-refine - Cpumask code clarification in kernel/sched.c
Clarify and slightly optimize set_cpus_allowed() cpumask check
kernel/sched.c +1 -1
@@ -2722,7 +2722,7 @@runqueue_t *rq; rq =task_rq_lock (p, &flags); -if (any_online_cpu (new_mask) ==NR_CPUS ) { +if (!cpus_intersects (new_mask, cpu_online_map)) { ret = -EINVAL ; goto out; }
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373PATCH 1/1 cpumask v4 - New bitmap operators and two op complement
In reply to Paul Jacksonmask3-bitmap-newops - New bitmap operators and two op complement
Add intersects, subset, xor and andnot operators. Change bitmap_complement to take two operands.
include/asm-generic/cpumask_array.h +1 -1
@@ -17,7 +17,7 @@#define cpus_and (dst,src1,src2) bitmap_and((dst).mask,(src1).mask, (src2).mask, NR_CPUS)#define cpus_or (dst,src1,src2) bitmap_or((dst).mask, (src1).mask, (src2).mask, NR_CPUS)#define cpus_clear (map) bitmap_clear((map).mask, NR_CPUS) -#define cpus_complement (map) bitmap_complement((map).mask, NR_CPUS) +#define cpus_complement (map) bitmap_complement((map).mask, (map).mask, NR_CPUS)#define cpus_equal (map1, map2) bitmap_equal((map1).mask, (map2).mask, NR_CPUS)#define cpus_empty (map) bitmap_empty(map.mask, NR_CPUS)#define cpus_addr (map) ((map).mask)
include/asm-i386/mpspec.h +1 -1
@@ -53,7 +53,7 @@#define physids_and (dst, src1, src2) bitmap_and((dst).mask, (src1).mask, (src2).mask, MAX_APICS)#define physids_or (dst, src1, src2) bitmap_or((dst).mask, (src1).mask, (src2).mask, MAX_APICS)#define physids_clear (map) bitmap_clear((map).mask, MAX_APICS) -#define physids_complement (map) bitmap_complement((map).mask, MAX_APICS) +#define physids_complement (map) bitmap_complement((map).mask, (map).mask, MAX_APICS)#define physids_empty (map) bitmap_empty((map).mask, MAX_APICS)#define physids_equal (map1, map2) bitmap_equal((map1).mask, (map2).mask, MAX_APICS)#define physids_weight (map) bitmap_weight((map).mask, MAX_APICS)
include/asm-x86_64/mpspec.h +1 -1
@@ -212,7 +212,7 @@#define physids_and (dst, src1, src2) bitmap_and((dst).mask, (src1).mask, (src2).mask, MAX_APICS)#define physids_or (dst, src1, src2) bitmap_or((dst).mask, (src1).mask, (src2).mask, MAX_APICS)#define physids_clear (map) bitmap_clear((map).mask, MAX_APICS) -#define physids_complement (map) bitmap_complement((map).mask, MAX_APICS) +#define physids_complement (map) bitmap_complement((map).mask, (map).mask, MAX_APICS)#define physids_empty (map) bitmap_empty((map).mask, MAX_APICS)#define physids_equal (map1, map2) bitmap_equal((map1).mask, (map2).mask, MAX_APICS)#define physids_weight (map) bitmap_weight((map).mask, MAX_APICS)
include/linux/bitmap.h +10 -2
@@ -13,8 +13,8 @@int bitmap_empty (const unsigned long *bitmap,int bits);int bitmap_full (const unsigned long *bitmap,int bits);int bitmap_equal (const unsigned long *bitmap1, -unsigned long *bitmap2,int bits); -void bitmap_complement (unsigned long *bitmap,int bits); +const unsigned long *bitmap2,int bits); +void bitmap_complement (unsigned long *dst,const unsigned long *src,int bits);static inline void bitmap_clear (unsigned long *bitmap,int bits) {
@@ -40,6 +40,14 @@ const unsigned long *bitmap2, int bits);void bitmap_or (unsigned long *dst,const unsigned long *bitmap1,const unsigned long *bitmap2,int bits); +void bitmap_xor (unsigned long *dst,const unsigned long *bitmap1, +const unsigned long *bitmap2,int bits); +void bitmap_andnot (unsigned long *dst,const unsigned long *bitmap1, +const unsigned long *bitmap2,int bits); +int bitmap_intersects (const unsigned long *bitmap1, +const unsigned long *bitmap2,int bits); +int bitmap_subset (const unsigned long *bitmap1, +const unsigned long *bitmap2,int bits);int bitmap_weight (const unsigned long *bitmap,int bits);int bitmap_scnprintf (char *buf,unsigned int buflen,const unsigned long *maskp,int bits);
lib/bitmap.c +57 -3
77 lines not on this page · raw message
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373PATCH 1/1 cpumask v4 - Simplify some sparc64 cpumask loop code
In reply to Paul Jacksonmask13-cpumask-sparc-simplify - Simplify some sparc64 cpumask loop code
Make use of for_each_cpu_mask() macro to simplify and optimize a couple of sparc64 per-CPU loops. This code change has _not_ been tested or reviewed. Feedback welcome. There is non-trivial risk that I still don't understand the logic here.
arch/sparc64/kernel/smp.c +24 -42
@@ -406,14 +406,8 @@int i; __asm__ __volatile__("rdpr %%pstate, %0" :"=r" (pstate)); -for (i =0 ; i <NR_CPUS ; i++) { -if (cpu_isset (i, mask)) { -spitfire_xcall_helper (data0, data1, data2, pstate, i); -cpu_clear (i, mask); -if (cpus_empty (mask)) -break ; - } - } +for_each_cpu_mask (i, mask) +spitfire_xcall_helper (data0, data1, data2, pstate, i); }/* Cheetah now allows to send the whole 64-bytes of data in the interrupt
@@ -456,25 +450,19 @@ nack_busy_id =0 ; { -cpumask_t work_mask = mask;int i; -for (i =0 ; i <NR_CPUS ; i++) { -if (cpu_isset (i, work_mask)) { -u64 target = (i <<14 ) |0x70 ; - -if (!is_jalapeno) - target |= (nack_busy_id <<24 ); - __asm__ __volatile__( -"stxa %%g0, [%0] %1\n\t" -"membar #Sync\n\t" - :/* no outputs */ - :"r" (target),"i" (ASI_INTR_W )); - nack_busy_id++; -cpu_clear (i, work_mask); -if (cpus_empty (work_mask)) -break ; - } +for_each_cpu_mask (i, mask) { +u64 target = (i <<14 ) |0x70 ; + +if (!is_jalapeno) + target |= (nack_busy_id <<24 ); + __asm__ __volatile__( +"stxa %%g0, [%0] %1\n\t" +"membar #Sync\n\t" + :/* no outputs */ + :"r" (target),"i" (ASI_INTR_W )); + nack_busy_id++; } }
@@ -507,7 +495,6 @@printk ("CPU[%d]: mondo stuckage result[%016lx]\n" ,smp_processor_id (), dispatch_stat); }else { -cpumask_t work_mask = mask;int i, this_busy_nack =0 ;/* Delay some random time with interrupts enabled
@@ -518,22 +505,17 @@/* Clear out the mask bits for cpus which did not * NACK us. */ -for (i =0 ; i <NR_CPUS ; i++) { -if (cpu_isset (i, work_mask)) { -u64 check_mask; - -if (is_jalapeno) - check_mask = (0x2UL << (2 *i)); -else - check_mask = (0x2UL << - this_busy_nack); -if ((dispatch_stat & check_mask) ==0 ) -cpu_clear (i, mask); - this_busy_nack +=2 ; -cpu_clear (i, work_mask); -if (cpus_empty (work_mask)) -break ; - } +for_each_cpu_mask (i, mask) { +u64 check_mask; + +if (is_jalapeno) + check_mask = (0x2UL << (2 *i)); +else + check_mask = (0x2UL << + this_busy_nack); +if ((dispatch_stat & check_mask) ==0 ) +cpu_clear (i, mask); + this_busy_nack +=2 ; } goto retry;
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373PATCH 1/1 cpumask v4 - Convert physids_complement() to use both args
In reply to Paul Jacksonmask15-cpumask-physids-complement - Convert physids_complement() to really use both args
Provide for specifying distinct source and dest args to the physids_complement(). No one actually uses this macro yet.
The physid_mask type would be a good candidate to convert to
using this new mask ADT as a base.
include/asm-i386/mpspec.h +1 -1
@@ -53,7 +53,7 @@#define physids_and (dst, src1, src2) bitmap_and((dst).mask, (src1).mask, (src2).mask, MAX_APICS)#define physids_or (dst, src1, src2) bitmap_or((dst).mask, (src1).mask, (src2).mask, MAX_APICS)#define physids_clear (map) bitmap_clear((map).mask, MAX_APICS) -#define physids_complement (map) bitmap_complement((map).mask, (map).mask, MAX_APICS) +#define physids_complement (dst, src) bitmap_complement((dst).mask, (src).mask, MAX_APICS)#define physids_empty (map) bitmap_empty((map).mask, MAX_APICS)#define physids_equal (map1, map2) bitmap_equal((map1).mask, (map2).mask, MAX_APICS)#define physids_weight (map) bitmap_weight((map).mask, MAX_APICS)
include/asm-x86_64/mpspec.h +1 -1
@@ -212,7 +212,7 @@#define physids_and (dst, src1, src2) bitmap_and((dst).mask, (src1).mask, (src2).mask, MAX_APICS)#define physids_or (dst, src1, src2) bitmap_or((dst).mask, (src1).mask, (src2).mask, MAX_APICS)#define physids_clear (map) bitmap_clear((map).mask, MAX_APICS) -#define physids_complement (map) bitmap_complement((map).mask, (map).mask, MAX_APICS) +#define physids_complement (dst, src) bitmap_complement((dst).mask, (src).mask, MAX_APICS)#define physids_empty (map) bitmap_empty((map).mask, MAX_APICS)#define physids_equal (map1, map2) bitmap_equal((map1).mask, (map2).mask, MAX_APICS)#define physids_weight (map) bitmap_weight((map).mask, MAX_APICS)
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373PATCH 1/1 cpumask v4 - Remove cpumask hack from asm-x86_64/topology.h
In reply to Paul Jacksonmask16-cpumask-x86-64-online - Remove cpumask hack from asm-x86_64/topology.h
This file had the cpumask cpu_online_map as type
unsigned long, instead of type cpumask_t, for no good reason that I could see. So I changed it. Everywhere else, cpu_online_map is already of type cpumask_t.
include/asm-x86_64/topology.h +5 -5
@@ -10,18 +10,18 @@/* Map the K8 CPU local memory controllers to a simple 1:1 CPU:NODE topology */ extern int fake_node; -/* This is actually a cpumask_t, but doesn't matter because we don't have ->BITS_PER_LONG CPUs */ -extern unsigned long cpu_online_map; +extern cpumask_t cpu_online_map;#define cpu_to_node (cpu) (fake_node ? 0 : (cpu))#define parent_node (node) (node)#define node_to_first_cpu (node) (fake_node ? 0 : (node))#define node_to_cpumask (node) (fake_node ? cpu_online_map : (1UL << (node))) -static inline unsigned long pcibus_to_cpumask (int bus) +static inline cpumask_t pcibus_to_cpumask (int bus) { -return mp_bus_to_cpumask[bus] & cpu_online_map; +cpumask_t tmp; +cpus_and (tmp, mp_bus_to_cpumask[bus], cpu_online_map); +return tmp; }#define NODE_BALANCE_RATE 30/* CHECKME */
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373PATCH 1/1 cpumask v4 - Uninline find_next_bit on ia64
In reply to Paul Jacksonmask6-unline-find-next-bit-ia64 - Uninline find_next_bit on ia64
Move the page of code (~700 bytes of instructions) for find_next_bit and find_next_zero_bit from inline in include/asm-ia64/bitops.h to a real function in arch/ia64/lib/bitops.c, leaving a declaration and macro wrapper behind.
The other arch's with almost this same code might want to also uninline it: alpha, parisc, ppc, sh, sparc, sparc64.
These are too big to inline.
include/asm-ia64/bitops.h +11 -83
@@ -11,7 +11,7 @@#include <linux/compiler.h> #include <linux/types.h> - +#include <asm/bitops.h> #include <asm/intrinsics.h> /**
@@ -236,7 +236,7 @@ }/** -* test_and_change_bit - Change a bit and return its new value +* test_and_change_bit - Change a bit and return its old value * @nr: Bit to set * @addr: Address to count from *
@@ -359,93 +359,21 @@#endif /* __KERNEL__ */ -/* -* Find next zero bit in a bitmap reasonably efficiently.. -*/ -static inline int -find_next_zero_bit (void *addr,unsigned long size,unsigned long offset) -{ -unsigned long *p = ((unsigned long *) addr) + (offset >>6 ); -unsigned long result = offset & ~63UL ; -unsigned long tmp; - -if (offset >= size) -return size; - size -= result; - offset &=63UL ; -if (offset) { - tmp = *(p++); - tmp |= ~0UL >> (64 -offset); -if (size <64 ) - goto found_first; -if (~tmp) - goto found_middle; - size -=64 ; - result +=64 ; - } -while (size & ~63UL ) { -if (~(tmp = *(p++))) - goto found_middle; - result +=64 ; - size -=64 ; - } -if (!size) -return result; - tmp = *p; -found_first: - tmp |= ~0UL << size; -if (tmp == ~0UL )/* any bits zero? */ -return result + size;/* nope */ -found_middle: -return result +ffz (tmp); -} +extern int __find_next_zero_bit (void *addr,unsigned long size, \ +unsigned long offset); +extern int __find_next_bit (const void *addr,unsigned long size, \ +unsigned long offset); + +#define find_next_zero_bit (addr, size, offset) \ + __find_next_zero_bit((addr), (size), (offset)) +#define find_next_bit (addr, size, offset) \ + __find_next_bit((addr), (size), (offset))/* * The optimizer actually does good code for this case.. */ #define find_first_zero_bit (addr, size) find_next_zero_bit((addr), (size), 0) -/* -* Find next bit in a bitmap reasonably efficiently.. -*/ -static inline int -find_next_bit (const void *addr,unsigned long size,unsigned long offset) -{ -unsigned long *p = ((unsigned long *) addr) + (offset >>6 ); -unsigned long result = offset & ~63UL ; -unsigned long tmp; - -if (offset >= size) -return size; - size -= result; - offset &=63UL ; -if (offset) { - tmp = *(p++); - tmp &= ~0UL << offset; -if (size <64 ) - goto found_first; -if (tmp) - goto found_middle; - size -=64 ; - result +=64 ; - } -while (size & ~63UL ) { -if ((tmp = *(p++))) - goto found_middle; - result +=64 ; - size -=64 ; - } -if (!size) -return result; - tmp = *p; - found_first: - tmp &= ~0UL >> (64 -size); -if (tmp ==0UL )/* Are any bits set? */ -return result + size;/* Nope. */ - found_middle: -return result +__ffs (tmp); -} -#define find_first_bit (addr, size) find_next_bit((addr), (size), 0)#ifdef __KERNEL__
include/asm-cris/bitops.h +1 -1
@@ -169,7 +169,7 @@return retval; }/** -* test_and_change_bit - Change a bit and return its new value +* test_and_change_bit - Change a bit and return its old value * @nr: Bit to change * @addr: Address to count from *
include/asm-i386/bitops.h +1 -1
@@ -212,7 +212,7 @@ }/** -* test_and_change_bit - Change a bit and return its new value +* test_and_change_bit - Change a bit and return its old value * @nr: Bit to change * @addr: Address to count from *
include/asm-mips/bitops.h +2 -2
@@ -296,7 +296,7 @@ }/* -* test_and_change_bit - Change a bit and return its new value +* test_and_change_bit - Change a bit and return its old value * @nr: Bit to change * @addr: Address to count from *
@@ -567,7 +567,7 @@ }/* -* test_and_change_bit - Change a bit and return its new value +* test_and_change_bit - Change a bit and return its old value * @nr: Bit to change * @addr: Address to count from *
include/asm-x86_64/bitops.h +1 -1
@@ -204,7 +204,7 @@ }/** -* test_and_change_bit - Change a bit and return its new value +* test_and_change_bit - Change a bit and return its old value * @nr: Bit to change * @addr: Address to count from *
arch/ia64/lib/bitop.c +88 -0
88 lines not on this page · raw message
arch/ia64/lib/Makefile +1 -1
@@ -6,7 +6,7 @@ lib-y := __divsi3.o __udivsi3.o __modsi3.o __umodsi3.o \ __divdi3.o __udivdi3.o __moddi3.o __umoddi3.o \ - checksum.o clear_page.o csum_partial_copy.o copy_page.o \ + bitop.o checksum.o clear_page.o csum_partial_copy.o copy_page.o \ clear_user.o strncpy_from_user.o strlen_user.o strnlen_user.o \ flush.o ip_fast_csum.o do_csum.o \ memset.o strlen.o swiotlb.o
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373PATCH 1/1 cpumask v4 - Recode obsolete cpumask macros - arch i386
In reply to Paul Jacksonmask9-cpumask-i386-fixup - Remove/recode obsolete cpumask macros from arch i386
Remove by recoding all uses of the obsolete cpumask const, coerce and promote macros.
Diffstat Patch_7_of_23:
arch/i386/kernel/io_apic.c | 2 +- arch/i386/kernel/smp.c | 2 +- arch/i386/mach-voyager/voyager_smp.c | 30 +++++++++++++++--------------- include/asm-i386/genapic.h | 2 +- include/asm-i386/mach-bigsmp/mach_apic.h | 8 ++++---- include/asm-i386/mach-default/mach_apic.h | 10 +++++----- include/asm-i386/mach-es7000/mach_apic.h | 10 +++++----- include/asm-i386/mach-numaq/mach_apic.h | 2 +- include/asm-i386/mach-summit/mach_apic.h | 8 ++++---- include/asm-i386/mach-visws/mach_apic.h | 4 ++-- 10 files changed, 39 insertions(+), 39 deletions(-)
arch/i386/kernel/io_apic.c +1 -1
@@ -264,7 +264,7 @@struct irq_pin_list *entry = irq_2_pin + irq;unsigned int apicid_value; - apicid_value =cpu_mask_to_apicid (mk_cpumask_const (cpumask)); + apicid_value =cpu_mask_to_apicid (cpumask);/* Prepare to do the io_apic_write */ apicid_value = apicid_value <<24 ;spin_lock_irqsave (&ioapic_lock, flags);
arch/i386/kernel/smp.c +1 -1
@@ -160,7 +160,7 @@*/ inline void send_IPI_mask_bitmask (cpumask_t cpumask,int vector) { -unsigned long mask =cpus_coerce (cpumask); +unsigned long mask =cpus_addr (cpumask)[0 ];unsigned long cfg;unsigned long flags;
arch/i386/mach-voyager/voyager_smp.c +15 -15
@@ -154,7 +154,7 @@ send_CPI_allbutself(__u8 cpi) {__u8 cpu =smp_processor_id (); -__u32 mask =cpus_coerce (cpu_online_map) & ~(1 << cpu); +__u32 mask =cpus_addr (cpu_online_map)[0 ] & ~(1 << cpu);send_CPI (mask, cpi); }
@@ -403,11 +403,11 @@/* set up everything for just this CPU, we can alter * this as we start the other CPUs later */ /* now get the CPU disposition from the extended CMOS */ - phys_cpu_present_map =cpus_promote (voyager_extended_cmos_read (VOYAGER_PROCESSOR_PRESENT_MASK )); -cpus_coerce (phys_cpu_present_map) |=voyager_extended_cmos_read (VOYAGER_PROCESSOR_PRESENT_MASK +1 ) <<8 ; -cpus_coerce (phys_cpu_present_map) |=voyager_extended_cmos_read (VOYAGER_PROCESSOR_PRESENT_MASK +2 ) <<16 ; -cpus_coerce (phys_cpu_present_map) |=voyager_extended_cmos_read (VOYAGER_PROCESSOR_PRESENT_MASK +3 ) <<24 ; -printk ("VOYAGER SMP: phys_cpu_present_map = 0x%lx\n" ,cpus_coerce (phys_cpu_present_map)); +cpus_addr (phys_cpu_present_map)[0 ] =voyager_extended_cmos_read (VOYAGER_PROCESSOR_PRESENT_MASK ); +cpus_addr (phys_cpu_present_map)[0 ] |=voyager_extended_cmos_read (VOYAGER_PROCESSOR_PRESENT_MASK +1 ) <<8 ; +cpus_addr (phys_cpu_present_map)[0 ] |=voyager_extended_cmos_read (VOYAGER_PROCESSOR_PRESENT_MASK +2 ) <<16 ; +cpus_addr (phys_cpu_present_map)[0 ] |=voyager_extended_cmos_read (VOYAGER_PROCESSOR_PRESENT_MASK +3 ) <<24 ; +printk ("VOYAGER SMP: phys_cpu_present_map = 0x%lx\n" ,cpus_addr (phys_cpu_present_map)[0 ]);/* Here we set up the VIC to enable SMP */ /* enable the CPIs by writing the base vector to their register */ outb (VIC_DEFAULT_CPI_BASE ,VIC_CPI_BASE_REGISTER );
@@ -709,12 +709,12 @@/* now that the cat has probed the Voyager System Bus, sanity * check the cpu map */ if ( ((voyager_quad_processors | voyager_extended_vic_processors) - &cpus_coerce (phys_cpu_present_map)) !=cpus_coerce (phys_cpu_present_map)) { + &cpus_addr (phys_cpu_present_map)[0 ]) !=cpus_addr (phys_cpu_present_map)[0 ]) {/* should panic */ printk ("\n\n***WARNING*** Sanity check of CPU present map FAILED\n" ); } }else if (voyager_level ==4 ) - voyager_extended_vic_processors =cpus_coerce (phys_cpu_present_map); + voyager_extended_vic_processors =cpus_addr (phys_cpu_present_map)[0 ];/* this sets up the idle task to run on the current cpu */ voyager_extended_cpus =1 ;
@@ -912,7 +912,7 @@if (!cpumask)BUG (); -if ((cpumask &cpus_coerce (cpu_online_map)) != cpumask) +if ((cpumask &cpus_addr (cpu_online_map)[0 ]) != cpumask)BUG ();if (cpumask & (1 <<smp_processor_id ()))BUG ();
@@ -955,7 +955,7 @@preempt_disable (); - cpu_mask =cpus_coerce (mm->cpu_vm_mask) & ~(1 <<smp_processor_id ()); + cpu_mask =cpus_addr (mm->cpu_vm_mask)[0 ] & ~(1 <<smp_processor_id ());local_flush_tlb ();if (cpu_mask)flush_tlb_others (cpu_mask, mm,FLUSH_ALL );
@@ -971,7 +971,7 @@preempt_disable (); - cpu_mask =cpus_coerce (mm->cpu_vm_mask) & ~(1 <<smp_processor_id ()); + cpu_mask =cpus_addr (mm->cpu_vm_mask)[0 ] & ~(1 <<smp_processor_id ());if (current->active_mm == mm) {if (current->mm)
@@ -992,7 +992,7 @@preempt_disable (); - cpu_mask =cpus_coerce (mm->cpu_vm_mask) & ~(1 <<smp_processor_id ()); + cpu_mask =cpus_addr (mm->cpu_vm_mask)[0 ] & ~(1 <<smp_processor_id ());if (current->active_mm == mm) {if (current->mm)__flush_tlb_one (va);
@@ -1101,7 +1101,7 @@ int wait) {struct call_data_struct data; -__u32 mask =cpus_coerce (cpu_online_map); +__u32 mask =cpus_addr (cpu_online_map)[0 ]; mask &= ~(1 <<smp_processor_id ());
@@ -1789,9 +1789,9 @@unsigned long irq_mask =1 << irq;int cpu; - real_mask =cpus_coerce (mask) & voyager_extended_vic_processors; + real_mask =cpus_addr (mask)[0 ] & voyager_extended_vic_processors; -if (cpus_coerce (mask) ==0 ) +if (cpus_addr (mask)[0 ] ==0 )/* can't have no cpu's to accept the interrupt -- extremely * bad things will happen */ return ;
include/asm-i386/genapic.h +1 -1
@@ -62,7 +62,7 @@unsigned (*get_apic_id)(unsigned long x);unsigned long apic_id_mask; -unsigned int (*cpu_mask_to_apicid)(cpumask_const_t cpumask); +unsigned int (*cpu_mask_to_apicid)(cpumask_t cpumask);/* ipi */ void (*send_IPI_mask)(cpumask_t mask,int vector);
include/asm-i386/mach-bigsmp/mach_apic.h +4 -4
@@ -140,14 +140,14 @@return (1 ); } -static inline unsigned int cpu_mask_to_apicid (cpumask_const_t cpumask) +static inline unsigned int cpu_mask_to_apicid (cpumask_t cpumask) {int num_bits_set;int cpus_found =0 ;int cpu;int apicid; - num_bits_set =cpus_weight_const (cpumask); + num_bits_set =cpus_weight (cpumask);/* Return id to all */ if (num_bits_set ==NR_CPUS )return (int )0xFF ;
@@ -155,10 +155,10 @@* The cpus in the mask must all be on the apic cluster. If are not * on the same apicid cluster return default value of TARGET_CPUS. */ - cpu =first_cpu_const (cpumask); + cpu =first_cpu (cpumask); apicid =cpu_to_logical_apicid (cpu);while (cpus_found < num_bits_set) { -if (cpu_isset_const (cpu, cpumask)) { +if (cpu_isset (cpu, cpumask)) {int new_apicid =cpu_to_logical_apicid (cpu);if (apicid_cluster (apicid) !=apicid_cluster (new_apicid)){
include/asm-i386/mach-default/mach_apic.h +5 -5
@@ -5,12 +5,12 @@#define APIC_DFR_VALUE (APIC_DFR_FLAT) -static inline cpumask_const_t target_cpus (void ) +static inline cpumask_t target_cpus (void ) {#ifdef CONFIG_SMP -return mk_cpumask_const (cpu_online_map); +return cpu_online_map;#else -return mk_cpumask_const (cpumask_of_cpu (0 )); +return cpumask_of_cpu (0 );#endif }#define TARGET_CPUS (target_cpus())
@@ -118,9 +118,9 @@return physid_isset (GET_APIC_ID (apic_read (APIC_ID )), phys_cpu_present_map); } -static inline unsigned int cpu_mask_to_apicid (cpumask_const_t cpumask) +static inline unsigned int cpu_mask_to_apicid (cpumask_t cpumask) { -return cpus_coerce_const (cpumask); +return cpus_addr (cpumask)[0 ]; }static inline void enable_apic_mode (void )
include/asm-i386/mach-es7000/mach_apic.h +5 -5
@@ -89,7 +89,7 @@int apic = bios_cpu_apicid[smp_processor_id ()];printk ("Enabling APIC mode: %s. Using %d I/O APICs, target cpus %lx\n" , (apic_version[apic] ==0x14 ) ? -"Physical Cluster" :"Logical Cluster" , nr_ioapics,cpus_coerce (TARGET_CPUS )); +"Physical Cluster" :"Logical Cluster" , nr_ioapics,cpus_addr (TARGET_CPUS )[0 ]); }static inline int multi_timer_check (int apic,int irq)
@@ -159,14 +159,14 @@return (1 ); } -static inline unsigned int cpu_mask_to_apicid (cpumask_const_t cpumask) +static inline unsigned int cpu_mask_to_apicid (cpumask_t cpumask) {int num_bits_set;int cpus_found =0 ;int cpu;int apicid; - num_bits_set =cpus_weight_const (cpumask); + num_bits_set =cpus_weight (cpumask);/* Return id to all */ if (num_bits_set ==NR_CPUS )return 0xFF ;
@@ -174,10 +174,10 @@* The cpus in the mask must all be on the apic cluster. If are not * on the same apicid cluster return default value of TARGET_CPUS. */ - cpu =first_cpu_const (cpumask); + cpu =first_cpu (cpumask); apicid =cpu_to_logical_apicid (cpu);while (cpus_found < num_bits_set) { -if (cpu_isset_const (cpu, cpumask)) { +if (cpu_isset (cpu, cpumask)) {int new_apicid =cpu_to_logical_apicid (cpu);if (apicid_cluster (apicid) !=apicid_cluster (new_apicid)){
include/asm-i386/mach-numaq/mach_apic.h +1 -1
@@ -136,7 +136,7 @@* We use physical apicids here, not logical, so just return the default * physical broadcast to stop people from breaking us */ -static inline unsigned int cpu_mask_to_apicid (cpumask_const_t cpumask) +static inline unsigned int cpu_mask_to_apicid (cpumask_t cpumask) {return (int )0xF ; }
include/asm-i386/mach-summit/mach_apic.h +4 -4
@@ -140,14 +140,14 @@ { } -static inline unsigned int cpu_mask_to_apicid (cpumask_const_t cpumask) +static inline unsigned int cpu_mask_to_apicid (cpumask_t cpumask) {int num_bits_set;int cpus_found =0 ;int cpu;int apicid; - num_bits_set =cpus_weight_const (cpumask); + num_bits_set =cpus_weight (cpumask);/* Return id to all */ if (num_bits_set ==NR_CPUS )return (int )0xFF ;
@@ -155,10 +155,10 @@* The cpus in the mask must all be on the apic cluster. If are not * on the same apicid cluster return default value of TARGET_CPUS. */ - cpu =first_cpu_const (cpumask); + cpu =first_cpu (cpumask); apicid =cpu_to_logical_apicid (cpu);while (cpus_found < num_bits_set) { -if (cpu_isset_const (cpu, cpumask)) { +if (cpu_isset (cpu, cpumask)) {int new_apicid =cpu_to_logical_apicid (cpu);if (apicid_cluster (apicid) !=apicid_cluster (new_apicid)){
include/asm-i386/mach-visws/mach_apic.h +2 -2
@@ -84,9 +84,9 @@return physid_isset (boot_cpu_physical_apicid, phys_cpu_present_map); } -static inline unsigned int cpu_mask_to_apicid (cpumask_const_t cpumask) +static inline unsigned int cpu_mask_to_apicid (cpumask_t cpumask) { -return cpus_coerce_const (cpumask); +return cpus_addr (cpumask)[0 ]; }static inline u32 phys_pkg_id (u32 cpuid_apic,int index_msb)
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373PATCH 1/1 cpumask v4 - Optimize i386 cpumask macro usage.
In reply to Paul Jacksonmask14-cpumask-i386-simplify - Optimize i386 cpumask macro usage.
Optimize a bit of cpumask code for asm-i386/mach-es7000 Code untested, unreviewed. Feedback welcome.
include/asm-i386/mach-es7000/mach_ipi.h +2 -3
@@ -10,9 +10,8 @@static inline void send_IPI_allbutself (int vector) { -cpumask_t mask =cpumask_of_cpu (smp_processor_id ()); -cpus_complement (mask); -cpus_and (mask, mask, cpu_online_map); +cpumask_t mask = cpu_online_map; +cpu_clear (smp_processor_id (), mask);if (!cpus_empty (mask))send_IPI_mask (mask, vector); }
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373PATCH 1/1 cpumask v4 - two missing 'const' qualifiers in bitops/bitmap
In reply to Paul Jacksonmask4-bitmap-const - two missing 'const' qualifiers in bitops/bitmap
Add a couple of missing 'const' qualifiers on bitops test_bit and bitmap_equal args.
include/asm-generic/bitops.h +1 -1
@@ -42,7 +42,7 @@return retval; } -extern __inline__int test_bit (int nr,long * addr) +extern __inline__int test_bit (int nr,const unsigned long * addr) {int mask;
lib/bitmap.c +1 -1
@@ -65,7 +65,7 @@EXPORT_SYMBOL (bitmap_full);int bitmap_equal (const unsigned long *bitmap1, -unsigned long *bitmap2,int bits) +const unsigned long *bitmap2,int bits) {int k, lim = bits/BITS_PER_LONG ;for (k =0 ; k < lim; ++k)
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373PATCH 1/1 cpumask v4 - Dont generate nonzero unused bits in bitmap
In reply to Paul Jacksonmask2-bitmap-complement - Dont generate nonzero unused bits in bitmap
Tighten up bitmap so it does not generate nonzero bits in the unused tail if it is not given any on input.
lib/bitmap.c +9 -8
@@ -26,10 +26,10 @@* carefully filter out these unused bits from impacting their * results. * -* Except for bitmap_complement, these operations hold to a -* slightly stronger rule: if you don't input any bitmaps to -* these ops that have some unused bits set, then they won't -* output any set unused bits in output bitmaps. +* These operations actually hold to a slightly stronger rule: +* if you don't input any bitmaps to these ops that have some +* unused bits set, then they won't output any set unused bits +* in output bitmaps. */ #define MAX_BITMAP_BITS 512U/* for ia64 NR_CPUS maximum */
@@ -83,11 +83,12 @@void bitmap_complement (unsigned long *bitmap,int bits) { -int k; -int nr =BITS_TO_LONGS (bits); - -for (k =0 ; k < nr; ++k) +int k, lim = bits/BITS_PER_LONG ; +for (k =0 ; k < lim; ++k) bitmap[k] = ~bitmap[k]; + +if (bits %BITS_PER_LONG ) + bitmap[k] = ~bitmap[k] & ((1UL << (bits %BITS_PER_LONG )) -1 ); }EXPORT_SYMBOL (bitmap_complement);
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373PATCH 1/1 cpumask v4 - Optimize and extend bitmap.
In reply to Paul Jacksonmask5-bitmap-extensions - Optimize and extend bitmap.
This bitmap improvements make it a suitable basis for fully supporting cpumask_t and nodemask_t. Inline macros with compile-time checks enable generating tight code on both small and large systems (large meaning cpumask_t requires more than one unsigned long's worth of bits).
The existing bitmap_<op> macros in lib/bitmap.c are renamed to __bitmap_<op>, and wrappers for each bitmap_<op> are exposed in include/linux/bitmap.h
This patch _includes_ Bill Irwins rewrite of the bitmap_shift operators to not require a fixed length intermediate bitmap.
Improved comments list each available operator for easy browsing.
include/linux/bitmap.h +230 -39
@@ -1,59 +1,250 @@ -#ifndef __LINUX_BITMAP_H -#define __LINUX_BITMAP_H +#ifndef __LINUX__bitmap_H +#define __LINUX__bitmap_H#ifndef __ASSEMBLY__ -#include <linux/config.h> -#include <linux/compiler.h> #include <linux/types.h> -#include <linux/kernel.h> #include <linux/bitops.h> #include <linux/string.h> -int bitmap_empty (const unsigned long *bitmap,int bits); -int bitmap_full (const unsigned long *bitmap,int bits); -int bitmap_equal (const unsigned long *bitmap1, -const unsigned long *bitmap2,int bits); -void bitmap_complement (unsigned long *dst,const unsigned long *src,int bits); +/* +* bitmaps provide bit arrays that consume one or more unsigned +* longs. The bitmap interface and available operations are listed +* here, in bitmap.h +* +* Function implementations generic to all architectures are in +* lib/bitmap.c. Functions implementations that are architecture +* specific are in various include/asm-<arch>/bitops.h headers +* and other arch/<arch> specific files. +* +* See lib/bitmap.c for more details. +*/ + +/* +* The available bitmap operations and their rough meaning in the +* case that the bitmap is a single unsigned long are thus: +* +* bitmap_clear(dst, nbits) *dst = 0UL +* bitmap_fill(dst, nbits) *dst = ~0UL +* bitmap_copy(dst, src, nbits) *dst = *src +* bitmap_and(dst, src1, src2, nbits) *dst = *src1 & *src2 +* bitmap_or(dst, src1, src2, nbits) *dst = *src1 | *src2 +* bitmap_xor(dst, src1, src2, nbits) *dst = *src1 ^ *src2 +* bitmap_andnot(dst, src1, src2, nbits) *dst = *src1 & ~(*src2) +* bitmap_complement(dst, src, nbits) *dst = ~(*src) +* bitmap_equal(src1, src2, nbits) Are *src1 and *src2 equal? +* bitmap_intersects(src1, src2, nbits) Do *src1 and *src2 overlap? +* bitmap_subset(src1, src2, nbits) Is *src1 a subset of *src2? +* bitmap_empty(src, nbits) Are all bits zero in *src? +* bitmap_full(src, nbits) Are all bits set in *src? +* bitmap_weight(src, nbits) Hamming Weight: number set bits +* bitmap_shift_right(dst, src, n, nbits) *dst = *src >> n +* bitmap_shift_left(dst, src, n, nbits) *dst = *src << n +* bitmap_scnprintf(buf, len, src, nbits) Print bitmap src to buf +* bitmap_parse(ubuf, ulen, dst, nbits) Parse bitmap dst from buf +*/ + +/* +* Also the following operations in asm/bitops.h apply to bitmaps. +* +* set_bit(bit, addr) *addr |= bit +* clear_bit(bit, addr) *addr &= ~bit +* change_bit(bit, addr) *addr ^= bit +* test_bit(bit, addr) Is bit set in *addr? +* test_and_set_bit(bit, addr) Set bit and return old value +* test_and_clear_bit(bit, addr) Clear bit and return old value +* test_and_change_bit(bit, addr) Change bit and return old value +* find_first_zero_bit(addr, nbits) Position first zero bit in *addr +* find_first_bit(addr, nbits) Position first set bit in *addr +* find_next_zero_bit(addr, nbits, bit) Position next zero bit in *addr >= bit +* find_next_bit(addr, nbits, bit) Position next set bit in *addr >= bit +*/ + +/* +* The DECLARE_BITMAP(name,bits) macro, in linux/types.h, can be used +* to declare an array named 'name' of just enough unsigned longs to +* contain all bit positions from 0 to 'bits' - 1. +*/ + +/* +* lib/bitmap.c provides these functions: +*/ + +extern int __bitmap_empty (const unsigned long *bitmap,int bits); +extern int __bitmap_full (const unsigned long *bitmap,int bits); +extern int __bitmap_equal (const unsigned long *bitmap1, +const unsigned long *bitmap2,int bits); +extern void __bitmap_complement (unsigned long *dst,const unsigned long *src, +int bits); +extern void __bitmap_shift_right (unsigned long *dst, +const unsigned long *src,int shift,int bits); +extern void __bitmap_shift_left (unsigned long *dst, +const unsigned long *src,int shift,int bits); +extern void __bitmap_and (unsigned long *dst,const unsigned long *bitmap1, +const unsigned long *bitmap2,int bits); +extern void __bitmap_or (unsigned long *dst,const unsigned long *bitmap1, +const unsigned long *bitmap2,int bits); +extern void __bitmap_xor (unsigned long *dst,const unsigned long *bitmap1, +const unsigned long *bitmap2,int bits); +extern void __bitmap_andnot (unsigned long *dst,const unsigned long *bitmap1, +const unsigned long *bitmap2,int bits); +extern int __bitmap_intersects (const unsigned long *bitmap1, +const unsigned long *bitmap2,int bits); +extern int __bitmap_subset (const unsigned long *bitmap1, +const unsigned long *bitmap2,int bits); +extern int __bitmap_weight (const unsigned long *bitmap,int bits); + +extern int bitmap_scnprintf (char *buf,unsigned int len, +const unsigned long *src,int nbits); +extern int bitmap_parse (const char __user *ubuf,unsigned int ulen, +unsigned long *dst,int nbits); + +#define BITMAP_LAST_WORD_MASK (nbits) \ +( \ + ((nbits) % BITS_PER_LONG) ? \ + (1UL<<((nbits) % BITS_PER_LONG))-1 : ~0UL \ +) -static inline void bitmap_clear (unsigned long *bitmap,int bits) +static inline void bitmap_clear (unsigned long *dst,int nbits) { -CLEAR_BITMAP ((unsigned long *)bitmap, bits); +if (nbits <=BITS_PER_LONG ) + *dst =0UL ; +else +CLEAR_BITMAP (dst, nbits); } -static inline void bitmap_fill (unsigned long *bitmap,int bits) +static inline void bitmap_fill (unsigned long *dst,int nbits) { -memset (bitmap,0xff ,BITS_TO_LONGS (bits)*sizeof (unsigned long )); +size_t nlongs =BITS_TO_LONGS (nbits); +if (nlongs >1 ) { +int len = (nlongs -1 ) *sizeof (unsigned long ); +memset (dst,0xff , len); + } + dst[nlongs -1 ] =BITMAP_LAST_WORD_MASK (nbits); } -static inline void bitmap_copy (unsigned long *dst, -const unsigned long *src,int bits) +static inline void bitmap_copy (unsigned long *dst,const unsigned long *src, +int nbits) { -memcpy (dst, src,BITS_TO_LONGS (bits)*sizeof (unsigned long )); +if (nbits <=BITS_PER_LONG ) + *dst = *src; +else { +int len =BITS_TO_LONGS (nbits) *sizeof (unsigned long ); +memcpy (dst, src, len); + } } -void bitmap_shift_right (unsigned long *dst, -const unsigned long *src,int shift,int bits); -void bitmap_shift_left (unsigned long *dst, -const unsigned long *src,int shift,int bits); -void bitmap_and (unsigned long *dst,const unsigned long *bitmap1, -const unsigned long *bitmap2,int bits); -void bitmap_or (unsigned long *dst,const unsigned long *bitmap1, -const unsigned long *bitmap2,int bits); -void bitmap_xor (unsigned long *dst,const unsigned long *bitmap1, -const unsigned long *bitmap2,int bits); -void bitmap_andnot (unsigned long *dst,const unsigned long *bitmap1, -const unsigned long *bitmap2,int bits); -int bitmap_intersects (const unsigned long *bitmap1, -const unsigned long *bitmap2,int bits); -int bitmap_subset (const unsigned long *bitmap1, -const unsigned long *bitmap2,int bits); -int bitmap_weight (const unsigned long *bitmap,int bits); -int bitmap_scnprintf (char *buf,unsigned int buflen, -const unsigned long *maskp,int bits); -int bitmap_parse (const char __user *ubuf,unsigned int ubuflen, -unsigned long *maskp,int bits); +static inline void bitmap_and (unsigned long *dst,const unsigned long *src1, +const unsigned long *src2,int nbits) +{ +if (nbits <=BITS_PER_LONG ) + *dst = *src1 & *src2; +else +__bitmap_and (dst, src1, src2, nbits); +} + +static inline void bitmap_or (unsigned long *dst,const unsigned long *src1, +const unsigned long *src2,int nbits) +{ +if (nbits <=BITS_PER_LONG ) + *dst = *src1 | *src2; +else +__bitmap_or (dst, src1, src2, nbits); +} + +static inline void bitmap_xor (unsigned long *dst,const unsigned long *src1, +const unsigned long *src2,int nbits) +{ +if (nbits <=BITS_PER_LONG ) + *dst = *src1 ^ *src2; +else +__bitmap_xor (dst, src1, src2, nbits); +} + +static inline void bitmap_andnot (unsigned long *dst,const unsigned long *src1, +const unsigned long *src2,int nbits) +{ +if (nbits <=BITS_PER_LONG ) + *dst = *src1 & ~(*src2); +else +__bitmap_andnot (dst, src1, src2, nbits); +} + +static inline void bitmap_complement (unsigned long *dst,const unsigned long *src, +int nbits) +{ +if (nbits <=BITS_PER_LONG ) + *dst = ~(*src) &BITMAP_LAST_WORD_MASK (nbits); +else +__bitmap_complement (dst, src, nbits); +} + +static inline int bitmap_equal (const unsigned long *src1, +const unsigned long *src2,int nbits) +{ +if (nbits <=BITS_PER_LONG ) +return ! ((*src1 ^ *src2) &BITMAP_LAST_WORD_MASK (nbits)); +else +return __bitmap_equal (src1, src2, nbits); +} + +static inline int bitmap_intersects (const unsigned long *src1, +const unsigned long *src2,int nbits) +{ +if (nbits <=BITS_PER_LONG ) +return ((*src1 & *src2) &BITMAP_LAST_WORD_MASK (nbits)) !=0 ; +else +return __bitmap_intersects (src1, src2, nbits); +} + +static inline int bitmap_subset (const unsigned long *src1, +const unsigned long *src2,int nbits) +{ +if (nbits <=BITS_PER_LONG ) +return ! ((*src1 & ~(*src2)) &BITMAP_LAST_WORD_MASK (nbits)); +else +return __bitmap_subset (src1, src2, nbits); +} + +static inline int bitmap_empty (const unsigned long *src,int nbits) +{ +if (nbits <=BITS_PER_LONG ) +return ! (*src &BITMAP_LAST_WORD_MASK (nbits)); +else +return __bitmap_empty (src, nbits); +} + +static inline int bitmap_full (const unsigned long *src,int nbits) +{ +if (nbits <=BITS_PER_LONG ) +return ! (~(*src) &BITMAP_LAST_WORD_MASK (nbits)); +else +return __bitmap_full (src, nbits); +} + +static inline int bitmap_weight (const unsigned long *src,int nbits) +{ +return __bitmap_weight (src, nbits); +} + +static inline void bitmap_shift_right (unsigned long *dst, +const unsigned long *src,int n,int nbits) +{ +if (nbits <=BITS_PER_LONG ) + *dst = *src >> n; +else +__bitmap_shift_right (dst, src, n, nbits); +} + +static inline void bitmap_shift_left (unsigned long *dst, +const unsigned long *src,int n,int nbits) +{ +if (nbits <=BITS_PER_LONG ) + *dst = (*src << n) &BITMAP_LAST_WORD_MASK (nbits); +else +__bitmap_shift_left (dst, src, n, nbits); +}#endif /* __ASSEMBLY__ */ -#endif /* __LINUX_BITMAP_H */ +#endif /* __LINUX__bitmap_H */
lib/bitmap.c +91 -62
291 lines not on this page · raw message
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373PATCH 1/1 cpumask v4 - Rewrite cpumask.h to use bitmap directly.
In reply to Paul Jacksonmask7-new-cpumask-h - Rewrite cpumask.h to use bitmap directly.
Major rewrite of cpumask to use a single implementation, as a struct-wrapped bitmap.
This patch leaves some 26 include/asm-*/cpumask*.h header files orphaned - to be removed next patch.
Some nine cpumask macros for const variants and to coerce and promote between an unsigned long and a cpumask are obsolete. Simple emulation wrappers are provided in this patch, which can be removed once each of the 3 archs (i386, ppc64, x86_64) using them are recoded in follow-on patches to not need them.
The CPU_MASK_ALL macro now avoids leaving possible garbage one bits in any unused portion of the high word.
An improved comment lists all available operators, for convenient browsing.
include/linux/cpumask.h +313 -27
@@ -1,44 +1,330 @@#ifndef __LINUX_CPUMASK_H#define __LINUX_CPUMASK_H +/* +* Cpumasks provide a bitmap suitable for representing the +* set of CPU's in a system, one bit position per CPU number. +* +* See detailed comments in the file linux/bitmap.h describing the +* data type on which these cpumasks are based. +* +* For details of cpumask_scnprintf() and cpumask_parse(), +* see bitmap_scnprintf() and bitmap_parse() in lib/bitmap.c. +* +* The available cpumask operations are: +* +* void cpu_set(cpu, mask) turn on bit 'cpu' in mask +* void cpu_clear(cpu, mask) turn off bit 'cpu' in mask +* void cpus_setall(mask) set all bits +* void cpus_clear(mask) clear all bits +* int cpu_isset(cpu, mask) true iff bit 'cpu' set in mask +* int cpu_test_and_set(cpu, mask) test and set bit 'cpu' in mask +* +* void cpus_and(dst, src1, src2) dst = src1 & src2 [intersection] +* void cpus_or(dst, src1, src2) dst = src1 | src2 [union] +* void cpus_xor(dst, src1, src2) dst = src1 ^ src2 +* void cpus_andnot(dst, src1, src2) dst = src1 & ~src2 +* void cpus_complement(dst, src) dst = ~src +* +* int cpus_equal(mask1, mask2) Does mask1 == mask2? +* int cpus_intersects(mask1, mask2) Do mask1 and mask2 intersect? +* int cpus_subset(mask1, mask2) Is mask1 a subset of mask2? +* int cpus_empty(mask) Is mask empty (no bits sets)? +* int cpus_full(mask) Is mask full (all bits sets)? +* int cpus_weight(mask) Hamming weigh - number of set bits +* +* void cpus_shift_right(dst, src, n) Shift right +* void cpus_shift_left(dst, src, n) Shift left +* +* int first_cpu(mask) Number lowest set bit, or NR_CPUS +* int next_cpu(cpu, mask) Next cpu past 'cpu', or NR_CPUS +* +* cpumask_t cpumask_of_cpu(cpu) Return cpumask with bit 'cpu' set +* CPU_MASK_ALL Initializer - all bits set +* CPU_MASK_NONE Initializer - no bits set +* unsigned long *cpus_addr(mask) Array of unsigned long's in mask +* +* int cpumask_scnprintf(buf, len, mask) Format cpumask for printing +* int cpumask_parse(ubuf, ulen, mask) Parse ascii string as cpumask +* +* int num_online_cpus() Number of online CPUs +* int num_possible_cpus() Number of all possible CPUs +* int cpu_online(cpu) Is some cpu online? +* int cpu_possible(cpu) Is some cpu possible? +* void cpu_set_online(cpu) set cpu in cpu_online_map +* void cpu_set_offline(cpu) clear cpu in cpu_online_map +* int any_online_cpu(mask) First online cpu in mask +* +* for_each_cpu_mask(cpu, mask) for-loop cpu over mask +* for_each_cpu(cpu) for-loop cpu over cpu_possible_map +* for_each_online_cpu(cpu) for-loop cpu over cpu_online_map +*/ +#include <linux/threads.h> #include <linux/bitmap.h> -#include <asm/cpumask.h> #include <asm/bug.h> -#ifdef CONFIG_SMP +typedef struct { DECLARE_BITMAP(bits, NR_CPUS); } cpumask_t; +extern cpumask_t _unused_cpumask_arg_; -extern cpumask_t cpu_online_map; -extern cpumask_t cpu_possible_map; +#define cpu_set (cpu, dst) __cpu_set((cpu), &(dst)) +static inline void __cpu_set (int cpu,volatile cpumask_t *dstp) +{ +if (cpu <NR_CPUS ) +set_bit (cpu, dstp->bits); +} + +#define cpu_clear (cpu, dst) __cpu_clear((cpu), &(dst)) +static inline void __cpu_clear (int cpu,volatile cpumask_t *dstp) +{ +clear_bit (cpu, dstp->bits); +} + +#define cpus_setall (dst) __cpus_setall(&(dst), NR_CPUS) +static inline void __cpus_setall (cpumask_t *dstp,int nbits) +{ +bitmap_fill (dstp->bits, nbits); +} + +#define cpus_clear (dst) __cpus_clear(&(dst), NR_CPUS) +static inline void __cpus_clear (cpumask_t *dstp,int nbits) +{ +bitmap_clear (dstp->bits, nbits); +} + +#define cpu_isset (cpu, cpumask) __cpu_isset((cpu), &(cpumask)) +static inline int __cpu_isset (int cpu,const volatile cpumask_t *addr) +{ +return test_bit (cpu, addr->bits); +} + +#define cpu_test_and_set (cpu, cpumask) __cpu_test_and_set((cpu), &(cpumask)) +static inline int __cpu_test_and_set (int cpu,cpumask_t *addr) +{ +if (cpu <NR_CPUS ) +return test_and_set_bit (cpu, addr->bits); +else +return 0 ; +} + +#define cpus_and (dst, src1, src2) __cpus_and(&(dst), &(src1), &(src2), NR_CPUS) +static inline void __cpus_and (cpumask_t *dstp,cpumask_t *src1p, +cpumask_t *src2p,int nbits) +{ +bitmap_and (dstp->bits, src1p->bits, src2p->bits, nbits); +} + +#define cpus_or (dst, src1, src2) __cpus_or(&(dst), &(src1), &(src2), NR_CPUS) +static inline void __cpus_or (cpumask_t *dstp,cpumask_t *src1p, +cpumask_t *src2p,int nbits) +{ +bitmap_or (dstp->bits, src1p->bits, src2p->bits, nbits); +} + +#define cpus_xor (dst, src1, src2) __cpus_xor(&(dst), &(src1), &(src2), NR_CPUS) +static inline void __cpus_xor (cpumask_t *dstp,cpumask_t *src1p, +cpumask_t *src2p,int nbits) +{ +bitmap_xor (dstp->bits, src1p->bits, src2p->bits, nbits); +} + +#define cpus_andnot (dst, src1, src2) \ + __cpus_andnot(&(dst), &(src1), &(src2), NR_CPUS) +static inline void __cpus_andnot (cpumask_t *dstp,cpumask_t *src1p, +cpumask_t *src2p,int nbits) +{ +bitmap_andnot (dstp->bits, src1p->bits, src2p->bits, nbits); +} + +#define cpus_complement (dst, src) __cpus_complement(&(dst), &(src), NR_CPUS) +static inline void __cpus_complement (cpumask_t *dstp, +cpumask_t *srcp,int nbits) +{ +bitmap_complement (dstp->bits, srcp->bits, nbits); +} + +#define cpus_equal (src1, src2) __cpus_equal(&(src1), &(src2), NR_CPUS) +static inline int __cpus_equal (cpumask_t *src1p, +cpumask_t *src2p,int nbits) +{ +return bitmap_equal (src1p->bits, src2p->bits, nbits); +} + +#define cpus_intersects (src1, src2) __cpus_intersects(&(src1), &(src2), NR_CPUS) +static inline int __cpus_intersects (cpumask_t *src1p, +cpumask_t *src2p,int nbits) +{ +return bitmap_intersects (src1p->bits, src2p->bits, nbits); +} + +#define cpus_subset (src1, src2) __cpus_subset(&(src1), &(src2), NR_CPUS) +static inline int __cpus_subset (cpumask_t *src1p, +cpumask_t *src2p,int nbits) +{ +return bitmap_subset (src1p->bits, src2p->bits, nbits); +} + +#define cpus_empty (src) __cpus_empty(&(src), NR_CPUS) +static inline int __cpus_empty (cpumask_t *srcp,int nbits) +{ +return bitmap_empty (srcp->bits, nbits); +} + +#define cpus_full (cpumask) __cpus_full(&(cpumask), NR_CPUS) +static inline int __cpus_full (cpumask_t *srcp,int nbits) +{ +return bitmap_full (srcp->bits, nbits); +} + +#define cpus_weight (cpumask) __cpus_weight(&(cpumask), NR_CPUS) +static inline int __cpus_weight (cpumask_t *srcp,int nbits) +{ +return bitmap_weight (srcp->bits, nbits); +} + +#define cpus_shift_right (dst, src, n) \ + __cpus_shift_right(&(dst), &(src), (n), NR_CPUS) +static inline void __cpus_shift_right (cpumask_t *dstp, +cpumask_t *srcp,int n,int nbits) +{ +bitmap_shift_right (dstp->bits, srcp->bits, n, nbits); +} + +#define cpus_shift_left (dst, src, n) \ + __cpus_shift_left(&(dst), &(src), (n), NR_CPUS) +static inline void __cpus_shift_left (cpumask_t *dstp, +cpumask_t *srcp,int n,int nbits) +{ +bitmap_shift_left (dstp->bits, srcp->bits, n, nbits); +} + +#define first_cpu (src) __first_cpu(&(src), NR_CPUS) +static inline int __first_cpu (cpumask_t *srcp,int nbits) +{ +return find_first_bit (srcp->bits, nbits); +} -#define num_online_cpus () cpus_weight(cpu_online_map) -#define num_possible_cpus () cpus_weight(cpu_possible_map) -#define cpu_online (cpu) cpu_isset(cpu, cpu_online_map) -#define cpu_possible (cpu) cpu_isset(cpu, cpu_possible_map) - -#define for_each_cpu_mask (cpu, mask) \ - for (cpu = first_cpu_const(mk_cpumask_const(mask)); \ - cpu < NR_CPUS; \ - cpu = next_cpu_const(cpu, mk_cpumask_const(mask))) +#define next_cpu (n, src) __next_cpu((n), &(src), NR_CPUS) +static inline int __next_cpu (int n,cpumask_t *srcp,int nbits) +{ +return find_next_bit (srcp->bits, nbits, n+1 ); +} + +#define cpumask_of_cpu (cpu) \ +({ \ + typeof(_unused_cpumask_arg_) m; \ + int c = cpu; \ + if (sizeof(m) == sizeof(unsigned long)) { \ + if (c < NR_CPUS) \ + m.bits[0] = 1UL<<c; \ + } else { \ + cpus_clear(m); \ + cpu_set(c, m); \ + } \ + m; \ +}) + +#define CPU_MASK_LAST_WORD BITMAP_LAST_WORD_MASK(NR_CPUS) + +#if NR_CPUS <=BITS_PER_LONG + +#define CPU_MASK_ALL \ +{ { \ + [BITS_TO_LONGS(NR_CPUS)-1] = CPU_MASK_LAST_WORD \ +} } -#define for_each_cpu (cpu) for_each_cpu_mask(cpu, cpu_possible_map) -#define for_each_online_cpu (cpu) for_each_cpu_mask(cpu, cpu_online_map)#else -#define cpu_online_map cpumask_of_cpu(0) -#define cpu_possible_map cpumask_of_cpu(0) -#define num_online_cpus () 1 -#define num_possible_cpus () 1 -#define cpu_online (cpu) ({ BUG_ON((cpu) != 0); 1; }) -#define cpu_possible (cpu) ({ BUG_ON((cpu) != 0); 1; }) -#define for_each_cpu (cpu) for (cpu = 0; cpu < 1; cpu++) -#define for_each_online_cpu (cpu) for (cpu = 0; cpu < 1; cpu++) +#define CPU_MASK_ALL \ +{ { \ + [0 ... BITS_TO_LONGS(NR_CPUS)-2] = ~0UL, \ + [BITS_TO_LONGS(NR_CPUS)-1] = CPU_MASK_LAST_WORD \ +} } +#endif -#define cpumask_scnprintf (buf, buflen, map) \ - bitmap_scnprintf(buf, buflen, cpus_addr(map), NR_CPUS) +#define CPU_MASK_NONE \ +{ { \ + [0 ... BITS_TO_LONGS(NR_CPUS)-1] = 0UL \ +} } + +#define cpus_addr (src) ((src).bits) + +#define cpumask_scnprintf (buf, len, src) \ + __cpumask_scnprintf((buf), (len), &(src), NR_CPUS) +static inline int __cpumask_scnprintf (char *buf,int len, +cpumask_t *srcp,int nbits) +{ +return bitmap_scnprintf (buf, len, srcp->bits, nbits); +} + +#define cpumask_parse (ubuf, ulen, src) \ + __cpumask_parse((ubuf), (ulen), &(src), NR_CPUS) +static inline int __cpumask_parse (const char __user *buf,int len, +cpumask_t *srcp,int nbits) +{ +return bitmap_parse (buf, len, srcp->bits, nbits); +} + +/* +* The following particular system cpumasks and operations +* on them manage all (possible) and online cpus. +*/ + +extern cpumask_t cpu_online_map; +extern cpumask_t cpu_possible_map; + +#ifdef CONFIG_SMP + +#define num_online_cpus () cpus_weight(cpu_online_map) +#define num_possible_cpus () cpus_weight(cpu_possible_map) +#define cpu_online (cpu) cpu_isset((cpu), cpu_online_map) +#define cpu_possible (cpu) cpu_isset((cpu), cpu_possible_map) +#define cpu_set_online (cpu) cpu_set((cpu), cpu_online_map) +#define cpu_set_offline (cpu) cpu_clear((cpu), cpu_online_map) + +#define any_online_cpu (mask) \ +({ \ + cpumask_t m; \ + cpus_and(m, mask, cpu_online_map); \ + first_cpu(m); \ +}) + +#define for_each_cpu_mask (cpu, mask) \ + for (cpu = first_cpu(mask); \ + cpu < NR_CPUS; \ + cpu = next_cpu(cpu, mask)) + +#else /* !CONFIG_SMP */ + +#define num_online_cpus () 1 +#define num_possible_cpus () 1 +#define cpu_online (cpu) ({ BUG_ON((cpu) != 0); 1; }) +#define cpu_possible (cpu) ({ BUG_ON((cpu) != 0); 1; }) +#define cpu_set_online (cpu) ({ BUG_ON((cpu) != 0); }) +#define cpu_set_offline (cpu) ({ BUG(); }) + +#define any_online_cpu (mask) 0 + +#define for_each_cpu_mask (cpu, mask) for (cpu = 0; cpu < 1; cpu++) + +#endif /* CONFIG_SMP */ + +#define for_each_cpu (cpu) \ + for_each_cpu_mask(cpu, cpu_possible_map) +#define for_each_online_cpu (cpu) \ + for_each_cpu_mask(cpu, cpu_online_map) -#define cpumask_parse (buf, buflen, map) \ - bitmap_parse(buf, buflen, cpus_addr(map), NR_CPUS) +/* Begin obsolete cpumask operator emulation */ +#define cpu_isset_const (a,b) cpu_isset(a,b) +#define cpumask_const_t cpumask_t +#define cpus_coerce (m) (cpus_addr(m)[0]) +#define cpus_coerce_const cpus_coerce +#define cpus_promote (x) ({ cpumask_t m; m.bits[0] = x; m; }) +#define cpus_weight_const cpus_weight +#define first_cpu_const first_cpu +#define mk_cpumask_const (x) x +#define next_cpu_const next_cpu +/* End of obsolete cpumask operator emulation */ #endif /* __LINUX_CPUMASK_H */
kernel/sched.c +5 -0
@@ -2349,6 +2349,11 @@return retval; } +#ifndef CONFIG_SMP +cpumask_t cpu_online_map =CPU_MASK_ALL ; +cpumask_t cpu_possible_map =CPU_MASK_ALL ; +#endif +/** * sys_sched_getaffinity - get the cpu affinity of a process * @pid: pid of the process
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373PATCH 1/1 cpumask v4 - Document bitmap.c bit model
In reply to Paul Jacksonmask1-bitmap-comment - Document bitmap.c bit model.
Document the bitmap bit model, including handling of unused bits, and operation preconditions and postconditions.
lib/bitmap.c +20 -0
@@ -12,6 +12,26 @@#include <asm/bitops.h> #include <asm/uaccess.h> +/* +* bitmaps provide an array of bits, implemented using an an +* array of unsigned longs. The number of valid bits in a +* given bitmap need not be an exact multiple of BITS_PER_LONG. +* +* The possible unused bits in the last, partially used word +* of a bitmap are 'don't care'. The implementation makes +* no particular effort to keep them zero. It ensures that +* their value will not affect the results of any operation. +* The bitmap operations that return Boolean (bitmap_empty, +* for example) or scalar (bitmap_weight, for example) results +* carefully filter out these unused bits from impacting their +* results. +* +* Except for bitmap_complement, these operations hold to a +* slightly stronger rule: if you don't input any bitmaps to +* these ops that have some unused bits set, then they won't +* output any set unused bits in output bitmaps. +*/ +#define MAX_BITMAP_BITS 512U/* for ia64 NR_CPUS maximum */ int bitmap_empty (const unsigned long *bitmap,int bits)
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373PATCH 1/1 cpumask v4 - Recode obsolete cpumask macros - arch ppc64
In reply to Paul Jacksonmask10-cpumask-ppc64-fixup - Remove/recode obsolete cpumask macros from arch ppc64
Remove by recoding all uses of the obsolete cpumask const, coerce and promote macros.
Diffstat Patch_8_of_23:
open_pic.c | 8 ++++---- rtasd.c | 6 +++--- 2 files changed, 7 insertions(+), 7 deletions(-)
arch/ppc64/kernel/open_pic.c +4 -4
@@ -592,7 +592,7 @@void openpic_init_processor (u_int cpumask) {openpic_write (&OpenPIC->Global.Processor_Initialization, -physmask (cpumask &cpus_coerce (cpu_online_map))); +physmask (cpumask &cpus_addr (cpu_online_map)[0 ])); }#ifdef CONFIG_SMP
@@ -626,7 +626,7 @@CHECK_THIS_CPU ;check_arg_ipi (ipi);openpic_write (&OpenPIC->THIS_CPU.IPI_Dispatch (ipi), -physmask (cpumask &cpus_coerce (cpu_online_map))); +physmask (cpumask &cpus_addr (cpu_online_map)[0 ])); }void openpic_request_IPIs (void )
@@ -712,7 +712,7 @@ {check_arg_timer (timer);openpic_write (&OpenPIC->Global.Timer[timer].Destination, -physmask (cpumask &cpus_coerce (cpu_online_map))); +physmask (cpumask &cpus_addr (cpu_online_map)[0 ])); }
@@ -837,7 +837,7 @@cpumask_t tmp;cpus_and (tmp, cpumask, cpu_online_map); -openpic_mapirq (irq_nr - open_pic_irq_offset,physmask (cpus_coerce (tmp))); +openpic_mapirq (irq_nr - open_pic_irq_offset,physmask (cpus_addr (tmp)[0 ])); }#ifdef CONFIG_SMP
arch/ppc64/kernel/rtasd.c +3 -3
@@ -411,7 +411,7 @@ }lock_cpu_hotplug (); - cpu =first_cpu_const (mk_cpumask_const (cpu_online_map)); + cpu =first_cpu (cpu_online_map);for (;;) {set_cpus_allowed (current,cpumask_of_cpu (cpu));do_event_scan (event_scan);
@@ -425,9 +425,9 @@schedule_timeout ((HZ *60 /rtas_event_scan_rate) /2 );lock_cpu_hotplug (); - cpu =next_cpu_const (cpu,mk_cpumask_const (cpu_online_map)); + cpu =next_cpu (cpu, cpu_online_map);if (cpu ==NR_CPUS ) - cpu =first_cpu_const (mk_cpumask_const (cpu_online_map)); + cpu =first_cpu (cpu_online_map); } error_vfree:
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373PATCH 1/1 cpumask v4 - Remove 26 no longer used cpumask headers.
In reply to Paul Jacksonmask8-remove-old-cpumask-files - Remove 26 no longer used cpumask headers.
With the cpumask rewrite in the previous patch, these various include/asm-*/cpumask*.h headers are no longer used.
include/asm-alpha/cpumask.h +0 -6
@@ -1,6 +0,0 @@ -#ifndef _ASM_ALPHA_CPUMASK_H -#define _ASM_ALPHA_CPUMASK_H - -#include <asm-generic/cpumask.h> - -#endif /* _ASM_ALPHA_CPUMASK_H */
include/asm-arm/cpumask.h +0 -6
@@ -1,6 +0,0 @@ -#ifndef _ASM_ARM_CPUMASK_H -#define _ASM_ARM_CPUMASK_H - -#include <asm-generic/cpumask.h> - -#endif /* _ASM_ARM_CPUMASK_H */
include/asm-arm26/cpumask.h +0 -6
@@ -1,6 +0,0 @@ -#ifndef _ASM_ARM26_CPUMASK_H -#define _ASM_ARM26_CPUMASK_H - -#include <asm-generic/cpumask.h> - -#endif /* _ASM_ARM26_CPUMASK_H */
include/asm-cris/cpumask.h +0 -6
@@ -1,6 +0,0 @@ -#ifndef _ASM_CRIS_CPUMASK_H -#define _ASM_CRIS_CPUMASK_H - -#include <asm-generic/cpumask.h> - -#endif /* _ASM_CRIS_CPUMASK_H */
include/asm-generic/cpumask.h +0 -40
@@ -1,40 +0,0 @@ -#ifndef __ASM_GENERIC_CPUMASK_H -#define __ASM_GENERIC_CPUMASK_H - -#include <linux/config.h> -#include <linux/kernel.h> -#include <linux/threads.h> -#include <linux/types.h> -#include <linux/bitmap.h> - -#if NR_CPUS >BITS_PER_LONG &&NR_CPUS !=1 -#define CPU_ARRAY_SIZE BITS_TO_LONGS(NR_CPUS) - -struct cpumask -{ -unsigned long mask[CPU_ARRAY_SIZE ]; -}; - -typedef struct cpumask cpumask_t ; - -#else -typedef unsigned long cpumask_t ; -#endif - -#ifdef CONFIG_SMP -#if NR_CPUS >BITS_PER_LONG -#include <asm-generic/cpumask_array.h> -#else -#include <asm-generic/cpumask_arith.h> -#endif -#else -#include <asm-generic/cpumask_up.h> -#endif - -#if NR_CPUS <=4 *BITS_PER_LONG -#include <asm-generic/cpumask_const_value.h> -#else -#include <asm-generic/cpumask_const_reference.h> -#endif - -#endif /* __ASM_GENERIC_CPUMASK_H */
include/asm-generic/cpumask_arith.h +0 -49
49 lines not on this page · raw message
include/asm-generic/cpumask_array.h +0 -54
54 lines not on this page · raw message
include/asm-generic/cpumask_const_reference.h +0 -29
@@ -1,29 +0,0 @@ -#ifndef __ASM_GENERIC_CPUMASK_CONST_REFERENCE_H -#define __ASM_GENERIC_CPUMASK_CONST_REFERENCE_H - -struct cpumask_ref { -const cpumask_t *val; -}; - -typedef const struct cpumask_ref cpumask_const_t ; - -#define mk_cpumask_const (map) ((cpumask_const_t){ &(map) }) -#define cpu_isset_const (cpu, map) cpu_isset(cpu, *(map).val) - -#define cpus_and_const (dst,src1,src2) cpus_and(dst,*(src1).val,*(src2).val) -#define cpus_or_const (dst,src1,src2) cpus_or(dst,*(src1).val,*(src2).val) - -#define cpus_equal_const (map1, map2) cpus_equal(*(map1).val, *(map2).val) - -#define cpus_copy_const (map1, map2) bitmap_copy((map1).mask, (map2).val->mask, NR_CPUS) - -#define cpus_empty_const (map) cpus_empty(*(map).val) -#define cpus_weight_const (map) cpus_weight(*(map).val) -#define first_cpu_const (map) first_cpu(*(map).val) -#define next_cpu_const (cpu, map) next_cpu(cpu, *(map).val) - -/* only ever use this for things that are _never_ used on large boxen */ -#define cpus_coerce_const (map) cpus_coerce(*(map).val) -#define any_online_cpu_const (map) any_online_cpu(*(map).val) - -#endif /* __ASM_GENERIC_CPUMASK_CONST_REFERENCE_H */
include/asm-generic/cpumask_const_value.h +0 -21
@@ -1,21 +0,0 @@ -#ifndef __ASM_GENERIC_CPUMASK_CONST_VALUE_H -#define __ASM_GENERIC_CPUMASK_CONST_VALUE_H - -typedef const cpumask_t cpumask_const_t ; - -#define mk_cpumask_const (map) (map) -#define cpu_isset_const (cpu, map) cpu_isset(cpu, map) -#define cpus_and_const (dst,src1,src2) cpus_and(dst, src1, src2) -#define cpus_or_const (dst,src1,src2) cpus_or(dst, src1, src2) -#define cpus_equal_const (map1, map2) cpus_equal(map1, map2) -#define cpus_empty_const (map) cpus_empty(map) -#define cpus_copy_const (map1, map2) do { map1 = (cpumask_t)map2; } while (0) -#define cpus_weight_const (map) cpus_weight(map) -#define first_cpu_const (map) first_cpu(map) -#define next_cpu_const (cpu, map) next_cpu(cpu, map) - -/* only ever use this for things that are _never_ used on large boxen */ -#define cpus_coerce_const (map) cpus_coerce(map) -#define any_online_cpu_const (map) any_online_cpu(map) - -#endif /* __ASM_GENERIC_CPUMASK_CONST_VALUE_H */
include/asm-generic/cpumask_up.h +0 -59
59 lines not on this page · raw message
include/asm-h8300/cpumask.h +0 -6
@@ -1,6 +0,0 @@ -#ifndef _ASM_H8300_CPUMASK_H -#define _ASM_H8300_CPUMASK_H - -#include <asm-generic/cpumask.h> - -#endif /* _ASM_H8300_CPUMASK_H */
include/asm-i386/cpumask.h +0 -6
@@ -1,6 +0,0 @@ -#ifndef _ASM_I386_CPUMASK_H -#define _ASM_I386_CPUMASK_H - -#include <asm-generic/cpumask.h> - -#endif /* _ASM_I386_CPUMASK_H */
include/asm-m68k/cpumask.h +0 -6
@@ -1,6 +0,0 @@ -#ifndef _ASM_M68K_CPUMASK_H -#define _ASM_M68K_CPUMASK_H - -#include <asm-generic/cpumask.h> - -#endif /* _ASM_M68K_CPUMASK_H */
include/asm-m68knommu/cpumask.h +0 -6
@@ -1,6 +0,0 @@ -#ifndef _ASM_M68KNOMMU_CPUMASK_H -#define _ASM_M68KNOMMU_CPUMASK_H - -#include <asm-generic/cpumask.h> - -#endif /* _ASM_M68KNOMMU_CPUMASK_H */
include/asm-mips/cpumask.h +0 -6
@@ -1,6 +0,0 @@ -#ifndef _ASM_MIPS_CPUMASK_H -#define _ASM_MIPS_CPUMASK_H - -#include <asm-generic/cpumask.h> - -#endif /* _ASM_MIPS_CPUMASK_H */
include/asm-parisc/cpumask.h +0 -6
@@ -1,6 +0,0 @@ -#ifndef _ASM_PARISC_CPUMASK_H -#define _ASM_PARISC_CPUMASK_H - -#include <asm-generic/cpumask.h> - -#endif /* _ASM_PARISC_CPUMASK_H */
include/asm-ppc/cpumask.h +0 -6
@@ -1,6 +0,0 @@ -#ifndef _ASM_PPC_CPUMASK_H -#define _ASM_PPC_CPUMASK_H - -#include <asm-generic/cpumask.h> - -#endif /* _ASM_PPC_CPUMASK_H */
include/asm-ppc64/cpumask.h +0 -6
@@ -1,6 +0,0 @@ -#ifndef _ASM_PPC64_CPUMASK_H -#define _ASM_PPC64_CPUMASK_H - -#include <asm-generic/cpumask.h> - -#endif /* _ASM_PPC64_CPUMASK_H */
include/asm-s390/cpumask.h +0 -6
@@ -1,6 +0,0 @@ -#ifndef _ASM_S390_CPUMASK_H -#define _ASM_S390_CPUMASK_H - -#include <asm-generic/cpumask.h> - -#endif /* _ASM_S390_CPUMASK_H */
include/asm-sh/cpumask.h +0 -6
@@ -1,6 +0,0 @@ -#ifndef _ASM_SH_CPUMASK_H -#define _ASM_SH_CPUMASK_H - -#include <asm-generic/cpumask.h> - -#endif /* _ASM_SH_CPUMASK_H */
include/asm-sparc/cpumask.h +0 -6
@@ -1,6 +0,0 @@ -#ifndef _ASM_SPARC_CPUMASK_H -#define _ASM_SPARC_CPUMASK_H - -#include <asm-generic/cpumask.h> - -#endif /* _ASM_SPARC_CPUMASK_H */
include/asm-sparc64/cpumask.h +0 -6
@@ -1,6 +0,0 @@ -#ifndef _ASM_SPARC64_CPUMASK_H -#define _ASM_SPARC64_CPUMASK_H - -#include <asm-generic/cpumask.h> - -#endif /* _ASM_SPARC64_CPUMASK_H */
include/asm-um/cpumask.h +0 -6
@@ -1,6 +0,0 @@ -#ifndef _ASM_UM_CPUMASK_H -#define _ASM_UM_CPUMASK_H - -#include <asm-generic/cpumask.h> - -#endif /* _ASM_UM_CPUMASK_H */
include/asm-v850/cpumask.h +0 -6
@@ -1,6 +0,0 @@ -#ifndef _ASM_V850_CPUMASK_H -#define _ASM_V850_CPUMASK_H - -#include <asm-generic/cpumask.h> - -#endif /* _ASM_V850_CPUMASK_H */
include/asm-x86_64/cpumask.h +0 -6
@@ -1,6 +0,0 @@ -#ifndef _ASM_X86_64_CPUMASK_H -#define _ASM_X86_64_CPUMASK_H - -#include <asm-generic/cpumask.h> - -#endif /* _ASM_X86_64_CPUMASK_H */
include/asm-ia64/cpumask.h +0 -6
@@ -1,6 +0,0 @@ -#ifndef _ASM_IA64_CPUMASK_H -#define _ASM_IA64_CPUMASK_H - -#include <asm-generic/cpumask.h> - -#endif /* _ASM_IA64_CPUMASK_H */
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373Hi Paul,
This patch gets the new bitmap stuff to compile and pass a boot-test
on Opteron.
I especially like the cleaned-up, orthogonal API and your addition of the missing bitmap_andnot() function. Good work!
Regards,
Joe
"Money can buy bandwidth, but latency is forever" -- John Mashey
arch/x86_64/kernel/smpboot.c +1 -1
@@ -827,7 +827,7 @@if (apicid == boot_cpu_id)continue ; -if (!cpu_isset (apicid, phys_cpu_present_map)) +if (!physid_isset (apicid, phys_cpu_present_map))continue ;if ((max_cpus >=0 ) && (max_cpus <= cpucount+1 ))continue ;
include/asm-x86_64/smp.h +0 -2
@@ -60,7 +60,6 @@extern cpumask_t cpu_callout_map;#define cpu_possible_map cpu_callout_map -#define cpu_online (cpu) cpu_isset(cpu, cpu_online_map)static inline int num_booting_cpus (void ) {
@@ -85,7 +84,6 @@#define safe_smp_processor_id () (disable_apic ? 0 : x86_apicid_to_cpu[hard_smp_processor_id()]) -#define cpu_online (cpu) cpu_isset(cpu, cpu_online_map)#endif /* !ASSEMBLY */ #define NO_PROC_ID 0xFF/* No processor magic marker */
Quoted 2 lines
This patch gets the new bitmap stuff to compile and pass a boot-test on Opteron.
Ok - thanks for testing it. I'll add that patch. And thanks for the favorable feedback.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373