Skip to content

RFC CPU ops and a rework of per cpu data handling on x86_64

[email protected] lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest44 of 45 patches 75 repliesIn review

44 of 45 patches are in this archive. The cover letter announces 45.

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/45ACPI: Avoid references to impossible processors.drivers/acpi/processor_core.c+7 -74 replies
2/45cpu alloc: Simple version of the allocator (static allocations)include/linux/percpu.h include/linux/vmstat.h mm/Kconfig +3 more+280 -20 replies
3/45Generic CPU operations: Core pieceinclude/linux/percpu.h+156 -04 replies
4/45cpu alloc: Use in SLUBarch/x86/Kconfig include/linux/slub_def.h mm/slub.c+52 -1875 replies
5/45cpu alloc: Remove SLUB fieldsinclude/linux/slub_def.h mm/slub.c+16 -350 replies
6/45cpu alloc: page allocator conversioninclude/linux/mm.h include/linux/mmzone.h mm/page_alloc.c +1 more+73 -1190 replies
7/45cpu_alloc: Implement dynamically extendable cpu areasinclude/linux/mm.h mm/Kconfig mm/cpu_alloc.c+300 -110 replies
8/45cpu alloc: x86 supportarch/x86/Kconfig arch/x86/kernel/vmlinux_32.lds.S arch/x86/kernel/vmlinux_64.lds.S +4 more+64 -225 replies
9/45cpu alloc: IA64 supportarch/ia64/Kconfig arch/ia64/kernel/vmlinux.lds.S include/asm-ia64/pgtable.h+41 -60 replies
10/45cpu_alloc: Sparc64 supportarch/sparc64/Kconfig arch/sparc64/kernel/vmlinux.lds.S include/asm-sparc64/pgtable.h+19 -00 replies
11/45cpu alloc: percpu_counter conversionlib/percpu_counter.c+6 -60 replies
12/45cpu alloc: crash_notes conversionarch/ia64/kernel/crash.c drivers/base/cpu.c kernel/kexec.c+4 -42 replies
13/45cpu alloc: workqueue conversionkernel/workqueue.c+14 -130 replies
14/45cpu alloc: ACPI cstate handling conversionarch/x86/kernel/acpi/cstate.c arch/x86/kernel/cpu/cpufreq/acpi-cpufreq.c drivers/acpi/processor_perflib.c+11 -90 replies
15/45cpu alloc: genhd statistics conversioninclude/linux/genhd.h+6 -100 replies
16/45cpu alloc: blktrace conversionblock/blktrace.c+4 -40 replies
17/45cpu alloc: SRCUkernel/rcutorture.c kernel/srcu.c+10 -140 replies
18/45cpu alloc: XFS countersfs/xfs/xfs_mount.c+8 -164 replies
19/45cpu alloc: NFS statisticsfs/nfs/iostat.h fs/nfs/super.c+5 -59 replies
20/45cpu alloc: neigbour statisticsinclude/net/neighbour.h net/core/neighbour.c+7 -100 replies
21/45cpu alloc: tcp statisticsnet/ipv4/tcp.c+5 -50 replies
22/45cpu alloc: convert scatchesnet/ipv4/ipcomp.c net/ipv6/ipcomp6.c+26 -260 replies
23/45cpu alloc: dmaengine conversiondrivers/dma/dmaengine.c include/linux/dmaengine.h+20 -342 replies
24/45cpu alloc: convert loopback statisticsdrivers/net/loopback.c+6 -80 replies
25/45cpu alloc: veth conversiondrivers/net/veth.c+5 -50 replies
26/45cpu alloc: Chelsio statistics conversiondrivers/net/chelsio/sge.c+7 -60 replies
27/45cpu alloc: convert mib handling to cpu allocinclude/net/ip.h include/net/snmp.h net/dccp/proto.c +5 more+55 -490 replies
28/45cpu_alloc: convert network socketsnet/core/sock.c+4 -40 replies
29/45cpu alloc: Use for infinibanddrivers/infiniband/hw/ehca/ehca_irq.c+11 -110 replies
30/45cpu alloc: Use in the crypto subsystem.crypto/async_tx/async_tx.c+8 -70 replies
31/45cpu alloc: Remove the allocpercpu functionalityinclude/linux/percpu.h mm/Makefile mm/allocpercpu.c+0 -2080 replies
32/45Module handling: Use CPU_xx ops to dynamically allocate countersinclude/linux/module.h kernel/module.c+12 -180 replies
33/45x86_64: Use CPU ops for nmi alert counterarch/x86/kernel/nmi_64.c+4 -40 replies
34/45x86_64: Fold percpu area into the cpu area.arch/x86/kernel/setup64.c arch/x86/kernel/vmlinux_64.lds.S include/asm-generic/sections.h +1 more+32 -180 replies
35/45X86_64: Declare pda as per cpu data thereby moving it into the cpu areaarch/x86/kernel/head64.c arch/x86/kernel/setup64.c arch/x86/kernel/smpboot_64.c +2 more+17 -180 replies
36/45X86_64: Place pda first in cpu area.arch/x86/kernel/setup64.c include/asm-generic/vmlinux.lds.h include/asm-x86/percpu_64.h+7 -10 replies
37/45x86_64: Support for fast per cpu operationsarch/x86/Kconfig include/asm-x86/percpu_64.h+266 -07 replies
38/45x86_64: Remove obsolete per_cpu offset calculationsarch/x86/kernel/smpboot_64.c include/asm-x86/percpu_64.h+9 -130 replies
39/45x86_64: Remove the data_offset field from the pda.arch/x86/kernel/asm-offsets_64.c arch/x86/kernel/entry_64.S arch/x86/kernel/setup64.c +1 more+3 -110 replies
40/45x86_64: Provide per_cpu_var definitioninclude/asm-x86/percpu_64.h+12 -60 replies
41/45VM statistics: Use CPU opsinclude/linux/vmstat.h+4 -60 replies
43/45x86_64: Add a CPU_OR to support or_pda()include/asm-x86/pda.h include/asm-x86/percpu_64.h+31 -340 replies
44/45Remove local_t supportDocumentation/local_ops.txt arch/frv/kernel/local.h include/asm-alpha/local.h +26 more+1 -14392 replies
45/45Modules: Hack to handle symbols that have a zero valuekernel/module.c+8 -73 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [rfc 00/45] [RFC] CPU ops and a rework of per cpu data handling on x86_64

This is a pretty early draft stage of the patch. It works on x86_64 only. Its a bit massive so I'd like to have some feedback before proceeding (and maybe some help)?.

The support for other arches was not tested yet.

The patch establishes a new set of cpu operations that allow to exploit single instruction atomicity to allow per cpu variable modifications without disabling/enabling preempt or interrupts and without the need to do an offset calculation in order to determine the location of the variable on the current processor.

It then implements these operations on x86_64 after consolidating per cpu access for allocpercpu, percpu and pda. All per cpu data is then accessible via gs segment override.

This results in a reduction in code size of the kernel and in more efficient operation of per cpu access.

Before:
text data bss dec hex filename
4041907 512371 1302360 5856638 595d7e vmlinux

After (this includes the code added for the cpu allocator!):

text data bss dec hex filename
3861532 527715 1298072 5687319 56c817 vmlinux

On x86_64 the segment override results in the following change for a simple vm counter increment:

Before:

mov    %gs:0x8,%rdx		Get smp_processor_id
mov    tableoffset,%rax		Get table base
incq   varoffset(%rax,%rdx,1)	Perform the operation with a complex lookup

adding the var offset

An interrupt or a reschedule action can move the execution thread to another processor if interrupt or preempt is not disabled. Then the variable of the wrong processor may be updated in a racy way.

After:

incq %gs:varoffset(%rip)

Single instruction that is safe from interrupts or moving of the execution thread. It will reliably operate on the current processors data area.

Other platforms can also perform address relocation plus atomic ops on a memory location. Exploiting of the atomicity of instructions vs interrupts is therefore possible and will reduce the cpu op processing overhead.

F.e on IA64 we have per cpu virtual mapping of the per cpu area. If we add an offset to the per cpu area variable address then we can guarantee that we always hit the per cpu areas local to a processor.

Other platforms (SPARC?) have registers that can be used to form addresses. If the cpu area address is in one of those then atomic per cpu modifications can be generated for those platforms in the same way.

Slub best performance in the fast fastpath goes from 47 cycles to 41 cycles through the use of the segment override.