Skip to content

percpu: Consistent per cpu operations V3

Christoph Lameter linux-ia64 linux-mm linux-parisc linux-pm linux-sh linux-watchdog lkml netdev sparclinux 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest41 patches 3 repliesIn review

V2->V3 from the cover letter

- Rediff patches
- Fix breakage caused by mips patches. Add a mips patch to convert from local_t.
- Update some descriptions.

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/41mm: Replace __get_cpu_var uses with this_cpu_ptrlib/radix-tree.c mm/memcontrol.c mm/memory-failure.c +5 more+14 -140 replies
2/41tracing: Replace __get_cpu_var uses with this_cpu_ptrinclude/linux/kprobes.h kernel/trace/ftrace.c kernel/trace/trace.c+4 -40 replies
3/41percpu: Replace __get_cpu_var with this_cpu_ptrinclude/linux/percpu.h+1 -10 replies
4/41kernel misc: Replace __get_cpu_var useskernel/printk/printk.c kernel/smp.c+7 -70 replies
5/41drivers/char/random: Replace __get_cpu_var usesdrivers/char/random.c+1 -10 replies
6/41drivers/cpuidle: Replace __get_cpu_var uses for address calculationdrivers/cpuidle/governors/ladder.c drivers/cpuidle/governors/menu.c+5 -50 replies
7/41drivers/oprofile: Replace __get_cpu_var uses for address calculationdrivers/oprofile/cpu_buffer.c drivers/oprofile/timer_int.c+6 -60 replies
8/41drivers/leds: Replace __get_cpu_var use through this_cpu_ptrdrivers/leds/trigger/ledtrig-cpu.c+1 -10 replies
9/41drivers/clocksource: Replace __get_cpu_var used for address calculationdrivers/clocksource/metag_generic.c+1 -10 replies
10/41staging/zsmalloc: Replace instances of using __get_cpu_var for address calculationdrivers/staging/zsmalloc/zsmalloc-main.c+2 -20 replies
11/41parisc: Replace __get_cpu_var uses for address calculationarch/parisc/lib/memcpy.c arch/parisc/mm/fault.c+2 -20 replies
12/41metag: Replace __get_cpu_var uses for address calculationarch/metag/kernel/perf/perf_event.c+7 -70 replies
13/41drivers/net/ethernet/tile: Replace __get_cpu_var uses for address calculationdrivers/net/ethernet/tile/tilegx.c drivers/net/ethernet/tile/tilepro.c+13 -130 replies
14/41percpu: Add raw_cpu_opsarch/x86/include/asm/percpu.h arch/x86/include/asm/preempt.h include/asm-generic/percpu.h +1 more+264 -2020 replies
15/41mm: Use raw_cpu ops for determining current NUMA nodeinclude/linux/topology.h+2 -20 replies
16/41modules: Use raw_cpu_write for initialization of per cpu refcount.kernel/module.c+1 -10 replies
17/41net: Replace __this_cpu_inc in route.c with raw_cpu_incnet/ipv4/route.c+1 -11 reply
18/41percpu: Add preemption checks to __this_cpu opsinclude/linux/percpu.h lib/smp_processor_id.c+45 -140 replies
19/41time: Replace __get_cpu_var usesdrivers/clocksource/dummy_timer.c kernel/hrtimer.c kernel/irq_work.c +6 more+34 -340 replies
20/41scheduler: Replace __get_cpu_var with this_cpu_ptrinclude/linux/kernel_stat.h kernel/events/callchain.c kernel/events/core.c +5 more+22 -220 replies
21/41block: Replace __this_cpu_ptr with raw_cpu_ptrfs/buffer.c fs/ext4/mballoc.c+2 -20 replies
22/41rcu: Replace __this_cpu_ptr uses with raw_cpu_ptrkernel/rcu/tree.c kernel/rcu/tree_plugin.h+5 -50 replies
23/41watchdog: Replace __raw_get_cpu_var useskernel/watchdog.c+5 -50 replies
24/41net: Replace get_cpu_var through this_cpu_ptrinclude/net/netfilter/nf_conntrack.h include/net/snmp.h net/core/dev.c +8 more+20 -201 reply
25/41md: Replace __this_cpu_ptr with raw_cpu_ptrdrivers/md/dm-stats.c+1 -10 replies
26/41irqchips: Replace __this_cpu_ptr usesdrivers/irqchip/irq-gic.c kernel/irq/chip.c+6 -60 replies
27/41x86: Replace __get_cpu_var usesarch/x86/include/asm/debugreg.h arch/x86/include/asm/perf_event_p4.h arch/x86/include/asm/uv/uv_hub.h +29 more+142 -1421 reply
28/41arm: Replace __this_cpu_ptr with raw_cpu_ptrarch/arm/kernel/smp_twd.c arch/arm/mach-msm/timer.c+7 -70 replies
29/41MIPS: Replace __get_cpu_var uses in FPU emulator.arch/mips/include/asm/fpu_emulator.h arch/mips/math-emu/cp1emu.c+10 -100 replies
30/41mips: Replace __get_cpu_var usesarch/mips/cavium-octeon/octeon-irq.c arch/mips/kernel/kprobes.c arch/mips/kernel/perf_event_mipsxx.c +1 more+26 -260 replies
31/41s390: rename __this_cpu_ptr to raw_cpu_ptrarch/s390/include/asm/percpu.h+8 -80 replies
32/41ia64: Replace __get_cpu_var usesarch/alpha/kernel/perf_event.c arch/ia64/include/asm/hw_irq.h arch/ia64/include/asm/sn/arch.h +11 more+40 -400 replies
33/41powerpc: Replace __get_cpu_var usesarch/powerpc/include/asm/cputime.h arch/powerpc/include/asm/hardirq.h arch/powerpc/include/asm/tlbflush.h +28 more+91 -910 replies
34/41sparc: Replace __get_cpu_var usesarch/sparc/include/asm/cpudata_32.h arch/sparc/include/asm/cpudata_64.h arch/sparc/kernel/kprobes.c +7 more+35 -350 replies
35/41tile: Replace __get_cpu_var usesarch/tile/include/asm/irqflags.h arch/tile/include/asm/mmu_context.h arch/tile/kernel/irq.c +9 more+31 -300 replies
36/41blackfin: Replace __get_cpu_var usesarch/blackfin/include/asm/ipipe.h arch/blackfin/kernel/perf_event.c arch/blackfin/mach-common/ints-priority.c +1 more+11 -110 replies
37/41avr32: Replace __get_cpu_var with __this_cpu_writearch/avr32/kernel/kprobes.c+1 -10 replies
38/41alpha: Replace __get_cpu_vararch/alpha/kernel/perf_event.c arch/alpha/kernel/time.c+10 -100 replies
39/41sh: Replace __get_cpu_var usesarch/sh/kernel/hw_breakpoint.c arch/sh/kernel/kprobes.c arch/sh/kernel/localtimer.c +2 more+23 -230 replies
40/41Remove __get_cpu_var and __raw_get_cpu_var macros [only in 3.15]include/asm-generic/percpu.h+0 -50 replies
41/41percpu: Remove __this_cpu_ptrinclude/asm-generic/percpu.h+0 -30 replies

Reviews trailers per patch

18 of 41 patches have a Reviewed-by or Acked-by.

PatchRafael J. WysockiIngo MolnarDavid S. MillerMasami HiramatsuBryan WuJames HoganHans-Christian EgtvedtChris MetcalfRichard HendersonH. Peter Anvin
1/41
2/41A-b
3/41
4/41
5/41
6/41A-b
7/41
8/41A-b
9/41A-b
10/41
11/41
12/41A-b
13/41A-b
14/41A-b
15/41A-b
16/41A-b
17/41A-b A-b
18/41A-b
19/41
20/41A-b
21/41
22/41
23/41
24/41A-b
25/41
26/41
27/41A-b A-b
28/41
29/41
30/41
31/41
32/41
33/41
34/41A-b
35/41A-b
36/41
37/41A-b
38/41A-b
39/41
40/41
41/41

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 10 reviewers.

Read the v1 thread

Cover letter [PATCH 00/41] percpu: Consistent per cpu operations V3

V2->V3:
- Rediff patches
- Fix breakage caused by mips patches. Add a mips patch to convert from local_t.
- Update some descriptions.

V1->V2:
- Move legacy definition for __this_cpu_ptr into include/asm-generic/percpu.h so that users bypassing include/linux/percpu.h do not break (affects tile and s390)
- Merge raw_cpu_ops core and the patch to rename x86 __this_cpu primitives into one. Otherwise breakage will occur since x86 __this_cpu ops will fall back to generic ops which is not tolerated well by the preempt hackery in x86.
- Add notes to each patch that depends on another to avoid mismerges. Add acks etc.
- Use quilt-0.61 with the bug fix that ensures all mailing lists receive the postings intended for them.

The kernel has never been audited to ensure that this_cpu operations are consistently used throughout the kernel. The code generated in many places can be improved through the use of this_cpu operations (which uses a segment register for relocation of per cpu offsets instead of performing address calculations).

The patch set also addresses various consistency issues in general with the per cpu macros.

A. The semantics of __this_cpu_ptr() differs from this_cpu_ptr only
because checks are skipped. This is typically shown through a raw_ prefix. So this patch set changes the places where __this_cpu_ptr() is used to raw_cpu_ptr().

B. There has been the long term wish by some that __this_cpu operations
would check for preemption. However, there are cases where preemption checks need to be skipped. This patch set adds raw_cpu operations that do not check for preemption and then adds preemption checks to the __this_cpu operations.

C. The use of __get_cpu_var is always a reference to a percpu variable
that can also be handled via a this_cpu operation. This patch set replaces all uses of __get_cpu_var with this_cpu operations.

D. We can then use this_cpu RMW operations in various places replacing
sequences of instructions by a single one.

E. The use of this_cpu operations throughout will allow other arches than
x86 to implement optimized references and RMV operations to work with per cpu local data.

F. The use of this_cpu operations opens up the possibility to
further optimize code that relies on synchronization through per cpu data.

The patch set works in a couple of stages:

I. Patches 1-13 are patches that simply replace uses of __get_cpu_var
with this_cpu_ptr. They do not depend on any changes to the percpu code. No preemption tests are skipped if they are applied.

II. Patch 14 adds the additional raw_cpu operations and raw_cpu_ptr().
Also converts the existing __this_cpu_xx_# primitive in the x86 code to raw_cpu_xx_#.

III. Patch 15-17 use the raw_cpu operations in places that would give
us false positives once they are enabled.

IV. Patch 18 adds preemption checks to __this_cpu operations to allow
checking if preemption is properly disabled when these functions are used.

V. Patches 19-39 are conversion patches that use this_cpu operations
in various kernel subsystems/drivers or arch code.

VI. Patches 40/41 remove no longer used functions (__this_cpu_ptr
and __get_cpu_var). These should only be applied after all the conversion patches have made it and after we have done additional passes through the kernel to ensure that none of the uses of these functions remain.