Skip to content

percpu: Consistent per cpu operations V4

Christoph Lameter linux-ia64 linux-mm linux-parisc linux-pm linux-sh linux-watchdog lkml netdev sparclinux 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest48 patches 32 repliesIn review

V3->V4 from the cover letter

- Rediff patches
- Put the patches first that define the new API.
- Add patches to convert the __get_cpu_var stuff added in 3.14

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/48percpu: Add raw_cpu_opsarch/x86/include/asm/percpu.h arch/x86/include/asm/preempt.h include/asm-generic/percpu.h +1 more+264 -2020 replies
2/48mm: Use raw_cpu ops for determining current NUMA nodeinclude/linux/topology.h+2 -20 replies
3/48modules: Use raw_cpu_write for initialization of per cpu refcount.kernel/module.c+1 -10 replies
4/48net: Replace __this_cpu_inc in route.c with raw_cpu_incnet/ipv4/route.c+1 -10 replies
5/48percpu: Add preemption checks to __this_cpu opsinclude/linux/percpu.h lib/smp_processor_id.c+45 -144 replies
6/48mm: Replace __get_cpu_var uses with this_cpu_ptrlib/radix-tree.c mm/memcontrol.c mm/memory-failure.c +6 more+15 -150 replies
7/48tracing: Replace __get_cpu_var uses with this_cpu_ptrinclude/linux/kprobes.h kernel/trace/ftrace.c kernel/trace/trace.c+4 -40 replies
8/48percpu: Replace __get_cpu_var with this_cpu_ptrinclude/linux/percpu.h+1 -10 replies
9/48kernel misc: Replace __get_cpu_var useskernel/printk/printk.c kernel/smp.c+7 -70 replies
10/48drivers/char/random: Replace __get_cpu_var usesdrivers/char/random.c+1 -10 replies
11/48drivers/cpuidle: Replace __get_cpu_var uses for address calculationdrivers/cpuidle/governors/ladder.c drivers/cpuidle/governors/menu.c+5 -50 replies
12/48drivers/oprofile: Replace __get_cpu_var uses for address calculationdrivers/oprofile/cpu_buffer.c drivers/oprofile/timer_int.c+6 -60 replies
13/48drivers/leds: Replace __get_cpu_var use through this_cpu_ptrdrivers/leds/trigger/ledtrig-cpu.c+1 -10 replies
14/48drivers/clocksource: Replace __get_cpu_var used for address calculationdrivers/clocksource/metag_generic.c+1 -10 replies
15/48parisc: Replace __get_cpu_var uses for address calculationarch/parisc/lib/memcpy.c arch/parisc/mm/fault.c+2 -20 replies
16/48metag: Replace __get_cpu_var uses for address calculationarch/metag/kernel/perf/perf_event.c+7 -70 replies
17/48drivers/net/ethernet/tile: Replace __get_cpu_var uses for address calculationdrivers/net/ethernet/tile/tilegx.c drivers/net/ethernet/tile/tilepro.c+13 -130 replies
18/48drivers/net/ethernet/tile: __get_cpu_var call introduced in 3.14drivers/net/ethernet/tile/tilegx.c+1 -10 replies
19/48tilegx: Another case of get_cpu_vardrivers/net/ethernet/tile/tilegx.c+1 -10 replies
20/48time: Replace __get_cpu_var usesdrivers/clocksource/dummy_timer.c kernel/hrtimer.c kernel/irq_work.c +6 more+34 -341 reply
21/48scheduler: Replace __get_cpu_var with this_cpu_ptrinclude/linux/kernel_stat.h kernel/events/callchain.c kernel/events/core.c +5 more+22 -220 replies
22/48tick-sched: Fix two new uses of __get_cpu_ptrkernel/time/tick-sched.c+2 -21 reply
23/48block: Replace __this_cpu_ptr with raw_cpu_ptrfs/ext4/mballoc.c+1 -10 replies
24/48rcu: Replace __this_cpu_ptr uses with raw_cpu_ptrkernel/rcu/tree.c kernel/rcu/tree_plugin.h+5 -51 reply
25/48watchdog: Replace __raw_get_cpu_var useskernel/watchdog.c+5 -50 replies
26/48net: Replace get_cpu_var through this_cpu_ptrinclude/net/netfilter/nf_conntrack.h include/net/snmp.h net/core/dev.c +8 more+19 -190 replies
27/48md: Replace __this_cpu_ptr with raw_cpu_ptrdrivers/md/dm-stats.c+1 -10 replies
28/48irqchips: Replace __this_cpu_ptr usesdrivers/irqchip/irq-gic.c kernel/irq/chip.c+6 -60 replies
29/48x86: Replace __get_cpu_var usesarch/x86/include/asm/debugreg.h arch/x86/include/asm/perf_event_p4.h arch/x86/include/asm/uv/uv_hub.h +29 more+142 -1420 replies
30/48x86: Change __get_cpu_var calls introduced in 3.14arch/x86/kernel/cpu/perf_event_intel_rapl.c kernel/sched/deadline.c kernel/time/tick-broadcast.c+8 -80 replies
31/48uv: Replace __get_cpu_vararch/x86/include/asm/uv/uv_hub.h+1 -110 replies
32/48arm: Replace __this_cpu_ptr with raw_cpu_ptrarch/arm/kernel/smp_twd.c arch/arm/mach-msm/timer.c+7 -70 replies
33/48MIPS: Replace __get_cpu_var uses in FPU emulator.arch/mips/include/asm/fpu_emulator.h arch/mips/math-emu/cp1emu.c+10 -100 replies
34/48mips: Replace __get_cpu_var usesarch/mips/cavium-octeon/octeon-irq.c arch/mips/kernel/kprobes.c arch/mips/kernel/perf_event_mipsxx.c +1 more+26 -260 replies
35/48s390: rename __this_cpu_ptr to raw_cpu_ptrarch/s390/include/asm/percpu.h+8 -80 replies
36/48s390: Replace __get_cpu_var usesarch/s390/include/asm/cputime.h arch/s390/include/asm/irq.h arch/s390/kernel/irq.c +7 more+28 -280 replies
37/48s390: Handle new __get_cpu_var calls added in 3.14arch/s390/kernel/perf_cpum_sf.c+8 -80 replies
38/48ia64: Replace __get_cpu_var usesarch/alpha/kernel/perf_event.c arch/ia64/include/asm/hw_irq.h arch/ia64/include/asm/sn/arch.h +10 more+39 -390 replies
39/48powerpc: Replace __get_cpu_var usesarch/powerpc/include/asm/cputime.h arch/powerpc/include/asm/hardirq.h arch/powerpc/include/asm/tlbflush.h +28 more+89 -8913 replies
40/48powerpc: Handle new __get_cpu_var calls in 3.14arch/powerpc/kernel/irq.c arch/powerpc/kernel/kgdb.c arch/powerpc/kernel/mce.c +2 more+17 -170 replies
41/48sparc: Replace __get_cpu_var usesarch/sparc/include/asm/cpudata_32.h arch/sparc/include/asm/cpudata_64.h arch/sparc/kernel/kprobes.c +7 more+35 -350 replies
42/48tile: Replace __get_cpu_var usesarch/tile/include/asm/irqflags.h arch/tile/include/asm/mmu_context.h arch/tile/kernel/irq.c +9 more+31 -300 replies
43/48blackfin: Replace __get_cpu_var usesarch/blackfin/include/asm/ipipe.h arch/blackfin/kernel/perf_event.c arch/blackfin/mach-common/ints-priority.c +1 more+11 -110 replies
44/48avr32: Replace __get_cpu_var with __this_cpu_writearch/avr32/kernel/kprobes.c+1 -10 replies
45/48alpha: Replace __get_cpu_vararch/alpha/kernel/perf_event.c arch/alpha/kernel/time.c+10 -100 replies
46/48sh: Replace __get_cpu_var usesarch/sh/kernel/hw_breakpoint.c arch/sh/kernel/kprobes.c arch/sh/kernel/localtimer.c +2 more+23 -230 replies
47/48Remove __get_cpu_var and __raw_get_cpu_var macros [only in 3.16]include/asm-generic/percpu.h+0 -50 replies
48/48percpu: Remove __this_cpu_ptrinclude/asm-generic/percpu.h+0 -30 replies

Reviews trailers per patch

22 of 48 patches have a Reviewed-by or Acked-by.

PatchDavid S. MillerIngo MolnarRafael J. WysockiJames HoganChris MetcalfHans-Christian EgtvedtRichard HendersonHeiko CarstensH. Peter AnvinBryan WuMasami HiramatsuThomas GleixnerPaul E. McKenney
1/48A-b
2/48A-b
3/48A-b
4/48A-b A-b
5/48A-b
6/48
7/48A-b
8/48
9/48
10/48
11/48A-b
12/48
13/48A-b
14/48A-b
15/48
16/48A-b
17/48A-b
18/48
19/48
20/48R-b
21/48A-b
22/48R-b
23/48
24/48A-b
25/48
26/48A-b
27/48
28/48
29/48A-b A-b
30/48
31/48
32/48
33/48
34/48
35/48
36/48A-b
37/48
38/48
39/48
40/48
41/48A-b
42/48A-b
43/48
44/48A-b
45/48A-b
46/48
47/48
48/48

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 13 reviewers.

Read the v1 thread

Cover letter [PATCH 00/48] percpu: Consistent per cpu operations V4

Can we please get this merged? The first patch alone would at least define the functions required to enable the merging of the rest in any order and through any tree.

There is a git tree that can be pulled at

https://git.kernel.org/cgit/linux/kernel/git/christoph/percpu.git/

It has the following branches:

minimal Just the first 5 patches to get the preemption checks going
most All the conversion but not the removal of functionality in 47/48
all All these patches

V3->V4:
- Rediff patches
- Put the patches first that define the new API.
- Add patches to convert the __get_cpu_var stuff added in 3.14

V2->V3:
- Rediff patches
- Fix breakage caused by mips patches. Add a mips patch to convert from local_t.
- Update some descriptions.

V1->V2:
- Move legacy definition for __this_cpu_ptr into include/asm-generic/percpu.h so that users bypassing include/linux/percpu.h do not break (affects tile and s390)
- Merge raw_cpu_ops core and the patch to rename x86 __this_cpu primitives into one. Otherwise breakage will occur since x86 __this_cpu ops will fall back to generic ops which is not tolerated well by the preempt hackery in x86.
- Add notes to each patch that depends on another to avoid mismerges. Add acks etc.
- Use quilt-0.61 with the bug fix that ensures all mailing lists receive the postings intended for them.

The kernel has never been audited to ensure that this_cpu operations are consistently used throughout the kernel. The code generated in many places can be improved through the use of this_cpu operations (which uses a segment register for relocation of per cpu offsets instead of performing address calculations).

The patch set also addresses various consistency issues in general with the per cpu macros.

A. The semantics of __this_cpu_ptr() differs from this_cpu_ptr only
because checks are skipped. This is typically shown through a raw_ prefix. So this patch set changes the places where __this_cpu_ptr() is used to raw_cpu_ptr().

B. There has been the long term wish by some that __this_cpu operations
would check for preemption. However, there are cases where preemption checks need to be skipped. This patch set adds raw_cpu operations that do not check for preemption and then adds preemption checks to the __this_cpu operations.

C. The use of __get_cpu_var is always a reference to a percpu variable
that can also be handled via a this_cpu operation. This patch set replaces all uses of __get_cpu_var with this_cpu operations.

D. We can then use this_cpu RMW operations in various places replacing
sequences of instructions by a single one.

E. The use of this_cpu operations throughout will allow other arches than
x86 to implement optimized references and RMV operations to work with per cpu local data.

F. The use of this_cpu operations opens up the possibility to
further optimize code that relies on synchronization through per cpu data.

The patch set works in a couple of stages:

I. Patch 1 adds the additional raw_cpu operations and raw_cpu_ptr().
Also converts the existing __this_cpu_xx_# primitive in the x86 code to raw_cpu_xx_#.

II. Patch 2-4 use the raw_cpu operations in places that would give
us false positives once they are enabled.

III. Patch 5 adds preemption checks to __this_cpu operations to allow
checking if preemption is properly disabled when these functions are used.

IV. Patches 6-20 are patches that simply replace uses of __get_cpu_var
with this_cpu_ptr. They do not depend on any changes to the percpu code. No preemption tests are skipped if they are applied.

V. Patches 21-46 are conversion patches that use this_cpu operations
in various kernel subsystems/drivers or arch code.

VI. Patches 47/48 remove no longer used functions (__this_cpu_ptr
and __get_cpu_var). These should only be applied after all the conversion patches have made it and after we have done additional passes through the kernel to ensure that none of the uses of these functions remain.