Cmpxchg and xchg support for cpu ops
Subsystems (5)
cover Cmpxchg and xchg support for cpu ops
This is a patchset on top of the thiscpuops upgrade patchset. It adds cmpxchg and xchg support and uses that to improve various kernel subsystem.
For vm statistics we can avoid interrupt enable/disable which reduces the latencies of the vm counter updates to a fraction.
The slub fastpaths and slowpaths can decrease their cycle count. In particular it is possible to improve the performance of the free slowpath.
RFC 2/8 --- include/linux/percpu.h | 31 +++---------------------------- 1 file changed, 3 insertions(+), 28 deletions(-)
include/linux/percpu.h +3 -28
@@ -361,14 +361,7 @@ do { \# define __this_cpu_xchg (pcp, nval) __pcpu_size_call_return2(__this_cpu_xchg_, (pcp), nval)#endif -#define _this_cpu_generic_xchg (pcp, nval) \ -({ typeof(pcp) ret__; \ - preempt_disable(); \ - ret__ = __this_cpu_read(pcp); \ - __this_cpu_write(pcp, nval); \ - preempt_enable(); \ - ret__; \ -}) +#define _this_cpu_generic_xchg (pcp, nval) xchg(__this_cpu_ptr(&(pcp)), nval)#ifndef this_cpu_xchg# ifndef this_cpu_xchg_1
@@ -386,15 +379,7 @@ do { \# define this_cpu_xchg (pcp, nval) __pcpu_size_call_return2(this_cpu_xchg_, (pcp), nval)#endif -#define _this_cpu_generic_cmpxchg (pcp, oval, nval) \ -({ typeof(pcp) ret__; \ - preempt_disable(); \ - ret__ = __this_cpu_read(pcp); \ - if (ret__ == (oval)) \ - __this_cpu_write(pcp, nval); \ - preempt_enable(); \ - ret__; \ -}) +#define _this_cpu_generic_cmpxchg (pcp, oval, nval) cmpxchg(__this_cpu_ptr(&(pcp)), oval, nval);#ifndef this_cpu_cmpxchg# ifndef this_cpu_cmpxchg_1
@@ -892,17 +877,7 @@ do { \# define irqsafe_cpu_xor (pcp, val) __pcpu_size_call(irqsafe_cpu_xor_, (val))#endif -#define irqsafe_cpu_generic_cmpxchg (pcp, oval, nval) \ -({ \ - typeof(pcp) ret__; \ - unsigned long flags; \ - local_irq_save(flags); \ - ret__ = __this_cpu_read(pcp); \ - if (ret__ == (oval)) \ - __this_cpu_write(pcp, nval); \ - local_irq_restore(flags); \ - ret__; \ -}) +#define irqsafe_cpu_generic_cmpxchg (pcp, oval, nval) cmpxchg(__this_cpu_ptr(&(pcp), oval, nval)#ifndef irqsafe_cpu_cmpxchg# ifndef irqsafe_cpu_cmpxchg_1
RFC 4/8 irq_work: Use per cpu atomics instead of regular atomics
In reply to Christoph LameterThe irq work queue is a per cpu object and it is sufficient for synchronization if per cpu atomics are used. Doing so simplifies the code and reduces the overhead of the code.
Before:
[email protected]$ size kernel/irq_work.o text data bss dec hex filename 451 8 1 460 1cc kernel/irq_work.o
After:
[email protected]$ size kernel/irq_work.o text data bss dec hex filename 438 8 1 447 1bf kernel/irq_work.o
--- kernel/irq_work.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
kernel/irq_work.c +9 -9
@@ -77,21 +77,21 @@ void __weak arch_irq_work_raise(void)*/ static void __irq_work_queue (struct irq_work *entry) { -struct irq_work **head, *next; +struct irq_work *next; - head = &get_cpu_var (irq_work_list); +preempt_disable ();do { - next = *head; + next =__this_cpu_read (irq_work_list);/* Can assign non-atomic because we keep the flags set. */ entry->next =next_flags (next,IRQ_WORK_FLAGS ); - }while (cmpxchg (head, next, entry) != next); + }while (this_cpu_cmpxchg (irq_work_list, next, entry) != next);/* The list was empty, raise self-interrupt to start processing. */ if (!irq_work_next (entry))arch_irq_work_raise (); -put_cpu_var (irq_work_list); +preempt_enable (); }/*
@@ -120,16 +120,16 @@ EXPORT_SYMBOL_GPL(irq_work_queue);*/ void irq_work_run (void ) { -struct irq_work *list, **head; +struct irq_work *list; - head = &__get_cpu_var (irq_work_list); -if (*head ==NULL ) +if (this_cpu_read (irq_work_list) ==NULL )return ;BUG_ON (!in_irq ());BUG_ON (!irqs_disabled ()); - list =xchg (head,NULL ); + list =this_cpu_xchg (irq_work_list,NULL ); +while (list !=NULL ) {struct irq_work *entry = list;
RFC 1/8 percpu: generic this_cpu_cmpxchg() and this_cpu_cmpxchg_double support
In reply to Christoph LameterProvide arch code to create the (local atomic) instructions.
V2->V3:
- Clean up some parameters
- Provide implementation of irqsafe_cpu_cmpxchg
--- include/linux/percpu.h | 258 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 257 insertions(+), 1 deletion(-)
include/linux/percpu.h +257 -1
@@ -259,6 +259,22 @@ extern void __bad_size_call_parameter(vo ret__; \ }) +/* Special handling for cmpxchg_double */ +#define __pcpu_size_call_return_int (stem, pcp, ...) \ +({ \ + int ret__; \ + __verify_pcpu_ptr(pcp); \ + switch(sizeof(*pcp)) { \ + case 1: ret__ = stem##1(pcp, __VA_ARGS__);break; \ + case 2: ret__ = stem##2(pcp, __VA_ARGS__);break; \ + case 4: ret__ = stem##4(pcp, __VA_ARGS__);break; \ + case 8: ret__ = stem##8(pcp, __VA_ARGS__);break; \ + default: \ + __bad_size_call_parameter();break; \ + } \ + ret__; \ +}) +#define __pcpu_size_call (stem, variable, ...) \ do { \ __verify_pcpu_ptr(&(variable)); \
@@ -322,6 +338,185 @@ do { \# define this_cpu_read (pcp) __pcpu_size_call_return(this_cpu_read_, (pcp))#endif +#define __this_cpu_generic_xchg (pcp, nval) \ +({ typeof(pcp) ret__; \ + ret__ = __this_cpu_read(pcp); \ + __this_cpu_write(pcp, nval); \ + ret__; \ +}) + +#ifndef __this_cpu_xchg +# ifndef __this_cpu_xchg_1 +# define __this_cpu_xchg_1 (pcp, nval) __this_cpu_generic_xchg(pcp, nval) +# endif +# ifndef __this_cpu_xchg_2 +# define __this_cpu_xchg_2 (pcp, nval) __this_cpu_generic_xchg(pcp, nval) +# endif +# ifndef __this_cpu_xchg_4 +# define __this_cpu_xchg_4 (pcp, nval) __this_cpu_generic_xchg(pcp, nval) +# endif +# ifndef __this_cpu_xchg_8 +# define __this_cpu_xchg_8 (pcp, nval) __this_cpu_generic_xchg(pcp, nval) +# endif +# define __this_cpu_xchg (pcp, nval) __pcpu_size_call_return2(__this_cpu_xchg_, (pcp), nval) +#endif + +#define _this_cpu_generic_xchg (pcp, nval) \ +({ typeof(pcp) ret__; \ + preempt_disable(); \ + ret__ = __this_cpu_read(pcp); \ + __this_cpu_write(pcp, nval); \ + preempt_enable(); \ + ret__; \ +}) + +#ifndef this_cpu_xchg +# ifndef this_cpu_xchg_1 +# define this_cpu_xchg_1 (pcp, nval) _this_cpu_generic_xchg(pcp, nval) +# endif +# ifndef this_cpu_xchg_2 +# define this_cpu_xchg_2 (pcp, nval) _this_cpu_generic_xchg(pcp, nval) +# endif +# ifndef this_cpu_xchg_4 +# define this_cpu_xchg_4 (pcp, nval) _this_cpu_generic_xchg(pcp, nval) +# endif +# ifndef this_cpu_xchg_8 +# define this_cpu_xchg_8 (pcp, nval) _this_cpu_generic_xchg(pcp, nval) +# endif +# define this_cpu_xchg (pcp, nval) __pcpu_size_call_return2(this_cpu_xchg_, (pcp), nval) +#endif + +#define _this_cpu_generic_cmpxchg (pcp, oval, nval) \ +({ typeof(pcp) ret__; \ + preempt_disable(); \ + ret__ = __this_cpu_read(pcp); \ + if (ret__ == (oval)) \ + __this_cpu_write(pcp, nval); \ + preempt_enable(); \ + ret__; \ +}) + +#ifndef this_cpu_cmpxchg +# ifndef this_cpu_cmpxchg_1 +# define this_cpu_cmpxchg_1 (pcp, oval, nval) _this_cpu_generic_cmpxchg(pcp, oval, nval) +# endif +# ifndef this_cpu_cmpxchg_2 +# define this_cpu_cmpxchg_2 (pcp, oval, nval) _this_cpu_generic_cmpxchg(pcp, oval, nval) +# endif +# ifndef this_cpu_cmpxchg_4 +# define this_cpu_cmpxchg_4 (pcp, oval, nval) _this_cpu_generic_cmpxchg(pcp, oval, nval) +# endif +# ifndef this_cpu_cmpxchg_8 +# define this_cpu_cmpxchg_8 (pcp, oval, nval) _this_cpu_generic_cmpxchg(pcp, oval, nval) +# endif +# define this_cpu_cmpxchg (pcp, oval, nval) __pcpu_size_call_return2(this_cpu_cmpxchg_, (pcp), oval, nval) +#endif + +#define __this_cpu_generic_cmpxchg (pcp, oval, nval) \ +({ \ + typeof(pcp) ret__; \ + ret__ = __this_cpu_read(pcp); \ + if (ret__ == (oval)) \ + __this_cpu_write(pcp, nval); \ + ret__; \ +}) + +#ifndef __this_cpu_cmpxchg +# ifndef __this_cpu_cmpxchg_1 +# define __this_cpu_cmpxchg_1 (pcp, oval, nval) __this_cpu_generic_cmpxchg(pcp, oval, nval) +# endif +# ifndef __this_cpu_cmpxchg_2 +# define __this_cpu_cmpxchg_2 (pcp, oval, nval) __this_cpu_generic_cmpxchg(pcp, oval, nval) +# endif +# ifndef __this_cpu_cmpxchg_4 +# define __this_cpu_cmpxchg_4 (pcp, oval, nval) __this_cpu_generic_cmpxchg(pcp, oval, nval) +# endif +# ifndef __this_cpu_cmpxchg_8 +# define __this_cpu_cmpxchg_8 (pcp, oval, nval) __this_cpu_generic_cmpxchg(pcp, oval, nval) +# endif +# define __this_cpu_cmpxchg (pcp, oval, nval) __pcpu_size_call_return2(\ + __this_cpu_cmpxchg_, (pcp), oval, nval) +#endif + +/* +* cmpxchg_double replaces two adjacent scalars at once. The first parameter +* passed is a percpu pointer, not a scalar like the other this_cpu +* operations. This is so because the function operates on two scalars +* (must be of same size). A truth value is returned to indicate success or +* failure (since a double register result is difficult to handle). +* There is very limited hardware support for these operations. So only certain +* sizes may work. +*/ +#define __this_cpu_generic_cmpxchg_double (pcp, oval1, oval2, nval1, nval2) \ +({ \ + typeof(oval2) * __percpu pcp2 = (typeof(oval2) *)((pcp) + 1); \ + int __ret = 0; \ + if (__this_cpu_read(*pcp) == (oval1) && \ + __this_cpu_read(*pcp2) == (oval2)) { \ + __this_cpu_write(*pcp, (nval1)); \ + __this_cpu_write(*pcp2, (nval2)); \ + __ret = 1; \ + } \ + (__ret); \ +}) + +#ifndef __this_cpu_cmpxchg_double +# ifndef __this_cpu_cmpxchg_double_1 +# define __this_cpu_cmpxchg_double_1 (pcp, oval1, oval2, nval1, nval2) \ + __this_cpu_generic_cmpxchg_double(pcp, oval1, oval2, nval1, nval2) +# endif +# ifndef __this_cpu_cmpxchg_double_2 +# define __this_cpu_cmpxchg_double_2 (pcp, oval1, oval2, nval1, nval2) \ + __this_cpu_generic_cmpxchg_double(pcp, oval1, oval2, nval1, nval2) +# endif +# ifndef __this_cpu_cmpxchg_double_4 +# define __this_cpu_cmpxchg_double_4 (pcp, oval1, oval2, nval1, nval2) \ + __this_cpu_generic_cmpxchg_double(pcp, oval1, oval2, nval1, nval2) +# endif +# ifndef __this_cpu_cmpxchg_double_8 +# define __this_cpu_cmpxchg_double_8 (pcp, oval1, oval2, nval1, nval2) \ + __this_cpu_generic_cmpxchg_double(pcp, oval1, oval2, nval1, nval2) +# endif +# define __this_cpu_cmpxchg_double (pcp, oval1, oval2, nval1, nval2) \ + __pcpu_size_call_return_int(__this_cpu_cmpxchg_double_, (pcp), \ + oval1, oval2, nval1, nval2) +#endif + +#define _this_cpu_generic_cmpxchg_double (pcp, oval1, oval2, nval1, nval2) \ +({ \ + int ret__; \ + preempt_disable(); \ + ret__ = __this_cpu_generic_cmpxchg_double(pcp, \ + oval1, oval2, nval1, nval2); \ + preempt_enable(); \ + ret__; \ +}) + +#ifndef this_cpu_cmpxchg_double +# ifndef this_cpu_cmpxchg_double_1 +# define this_cpu_cmpxchg_double_1 (pcp, oval1, oval2, nval1, nval2) \ + _this_cpu_generic_cmpxchg_double(pcp, oval1, oval2, nval1, nval2) +# endif +# ifndef this_cpu_cmpxchg_double_2 +# define this_cpu_cmpxchg_double_2 (pcp, oval1, oval2, nval1, nval2) \ + _this_cpu_generic_cmpxchg_double(pcp, oval1, oval2, nval1, nval2) +# endif +# ifndef this_cpu_cmpxchg_double_4 +# define this_cpu_cmpxchg_double_4 (pcp, oval1, oval2, nval1, nval2) \ + _this_cpu_generic_cmpxchg_double(pcp, oval1, oval2, nval1, nval2) +# endif +# ifndef this_cpu_cmpxchg_double_8 +# define this_cpu_cmpxchg_double_8 (pcp, oval1, oval2, nval1, nval2) \ + _this_cpu_generic_cmpxchg_double(pcp, oval1, oval2, nval1, nval2) +# endif +# define this_cpu_cmpxchg_double (pcp, oval1, oval2, nval1, nval2) \ + __pcpu_size_call_return_int(this_cpu_cmpxchg_double_, (pcp), \ + oval1, oval2, nval1, nval2) +#endif + + + +#define _this_cpu_generic_to_op (pcp, val, op) \ do { \ preempt_disable(); \
@@ -610,7 +805,7 @@ do { \* IRQ safe versions of the per cpu RMW operations. Note that these operations * are *not* safe against modification of the same variable from another * processors (which one gets when using regular atomic operations) -. They are guaranteed to be atomic vs. local interrupts and +* They are guaranteed to be atomic vs. local interrupts and * preemption only. */ #define irqsafe_cpu_generic_to_op (pcp, val, op) \
@@ -697,4 +892,65 @@ do { \# define irqsafe_cpu_xor (pcp, val) __pcpu_size_call(irqsafe_cpu_xor_, (val))#endif +#define irqsafe_cpu_generic_cmpxchg (pcp, oval, nval) \ +({ \ + typeof(pcp) ret__; \ + unsigned long flags; \ + local_irq_save(flags); \ + ret__ = __this_cpu_read(pcp); \ + if (ret__ == (oval)) \ + __this_cpu_write(pcp, nval); \ + local_irq_restore(flags); \ + ret__; \ +}) + +#ifndef irqsafe_cpu_cmpxchg +# ifndef irqsafe_cpu_cmpxchg_1 +# define irqsafe_cpu_cmpxchg_1 (pcp, oval, nval) irqsafe_cpu_generic_cmpxchg(pcp, oval, nval) +# endif +# ifndef irqsafe_cpu_cmpxchg_2 +# define irqsafe_cpu_cmpxchg_2 (pcp, oval, nval) irqsafe_cpu_generic_cmpxchg(pcp, oval, nval) +# endif +# ifndef irqsafe_cpu_cmpxchg_4 +# define irqsafe_cpu_cmpxchg_4 (pcp, oval, nval) irqsafe_cpu_generic_cmpxchg(pcp, oval, nval) +# endif +# ifndef irqsafe_cpu_cmpxchg_8 +# define irqsafe_cpu_cmpxchg_8 (pcp, oval, nval) irqsafe_cpu_generic_cmpxchg(pcp, oval, nval) +# endif +# define irqsafe_cpu_cmpxchg (pcp, oval, nval) __pcpu_size_call_return2(irqsafe_cpu_cmpxchg_, (pcp), oval, nval) +#endif + +#define irqsafe_generic_cpu_cmpxchg_double (pcp, oval1, oval2, nval1, nval2) \ +({ \ + int ret__; \ + unsigned long flags; \ + local_irq_save(flags); \ + ret__ = __this_cpu_generic_cmpxchg_double(pcp, \ + oval1, oval2, nval1, nval2); \ + local_irq_restore(flags); \ + ret__; \ +}) + +#ifndef irqsafe_cpu_cmpxchg_double +# ifndef irqsafe_cpu_cmpxchg_double_1 +# define irqsafe_cpu_cmpxchg_double_1 (pcp, oval1, oval2, nval1, nval2) \ + irqsafe_generic_cpu_cmpxchg_double(pcp, oval1, oval2, nval1, nval2) +# endif +# ifndef irqsafe_cpu_cmpxchg_double_2 +# define irqsafe_cpu_cmpxchg_double_2 (pcp, oval1, oval2, nval1, nval2) \ + irqsafe_generic_cpu_cmpxchg_double(pcp, oval1, oval2, nval1, nval2) +# endif +# ifndef irqsafe_cpu_cmpxchg_double_4 +# define irqsafe_cpu_cmpxchg_double_4 (pcp, oval1, oval2, nval1, nval2) \ + irqsafe_generic_cpu_cmpxchg_double(pcp, oval1, oval2, nval1, nval2) +# endif +# ifndef irqsafe_cpu_cmpxchg_double_8 +# define irqsafe_cpu_cmpxchg_double_8 (pcp, oval1, oval2, nval1, nval2) \ + irqsafe_generic_cpu_cmpxchg_double(pcp, oval1, oval2, nval1, nval2) +# endif +# define irqsafe_cpu_cmpxchg_double (pcp, oval1, oval2, nval1, nval2) \ + __pcpu_size_call_return_int(irqsafe_cpu_cmpxchg_double_, (pcp), \ + oval1, oval2, nval1, nval2) +#endif +#endif /* __LINUX_PERCPU_H */
RFC 8/8 slub: [RFC] Partially lockless freepath slowpath
In reply to Christoph LameterThe slub slow freepath is frequently invoked since fast frees are only possible for objects from the current slab page. Optimization of the slowpath is therefore necessary to increase freeing performance.
This patch supplies a partially lockless slowpath. It addresses the performance issues related to cycle count in the slow path but not issues that may arise because cache hotness (that are tracked differently in SLAB).
In the fastpaths we use a cmpxchg_local with segment prefix to perform freelist insertion. We can provide a similar approach for the slowpath but there we must use a regular cmpxchg with lock prefix since frees to a page may occur from multiple processors.
The cmpxchg only updates the freelist in the page struct. We also maintain an object counter (inuse) in the page structure. That counter is decremented in a racy way. This means that we may miss a decrement and the counter may be higher that the actual number of used objects in the slab. The counter is not used for the determination if the page is filled up though. Thus the page will cycle via the partial list back to slab_alloc. The counter is then fixed in allocation processing because allocation takes the whole list for the per cpu allocation list.
Serialization via the slab_lock() is still performed for any situation in which the freelist needs to be shrunk. Thus holding the slab_lock prevents the fastpath from zapping the freelist. This can be used to guarantee that no new object are allocated from a slab during free.
Results show that the slowpath performance is improved by around 40% to 100%.
10000 Allocations then 10000 frees test
Before (no lockless patches):
10000 times kmalloc(8) -> 207 cycles kfree -> 156 cycles 10000 times kmalloc(16) -> 208 cycles kfree -> 158 cycles 10000 times kmalloc(32) -> 257 cycles kfree -> 159 cycles 10000 times kmalloc(64) -> 383 cycles kfree -> 169 cycles 10000 times kmalloc(128) -> 375 cycles kfree -> 170 cycles 10000 times kmalloc(256) -> 869 cycles kfree -> 187 cycles 10000 times kmalloc(512) -> 1129 cycles kfree -> 307 cycles 10000 times kmalloc(1024) -> 2087 cycles kfree -> 554 cycles 10000 times kmalloc(2048) -> 3912 cycles kfree -> 588 cycles 10000 times kmalloc(4096) -> 7584 cycles kfree -> 664 cycles 10000 times kmalloc(8192) -> 7927 cycles kfree -> 903 cycles 10000 times kmalloc(16384) -> 8625 cycles kfree -> 1308 cycles
After (ll fastpath and slowpath):
10000 times kmalloc(8) -> 125 cycles kfree -> 95 cycles 10000 times kmalloc(16) -> 81 cycles kfree -> 109 cycles 10000 times kmalloc(32) -> 114 cycles kfree -> 101 cycles 10000 times kmalloc(64) -> 193 cycles kfree -> 110 cycles 10000 times kmalloc(128) -> 323 cycles kfree -> 124 cycles 10000 times kmalloc(256) -> 808 cycles kfree -> 141 cycles 10000 times kmalloc(512) -> 1051 cycles kfree -> 264 cycles 10000 times kmalloc(1024) -> 2026 cycles kfree -> 523 cycles 10000 times kmalloc(2048) -> 3970 cycles kfree -> 581 cycles 10000 times kmalloc(4096) -> 7677 cycles kfree -> 683 cycles 10000 times kmalloc(8192) -> 8022 cycles kfree -> 946 cycles 10000 times kmalloc(16384) -> 8641 cycles kfree -> 1286 cycles
10000 (alloc + free) test
Before:
10000 times kmalloc(8)/kfree -> 180 cycles
10000 times kmalloc(16)/kfree -> 180 cycles
10000 times kmalloc(32)/kfree -> 187 cycles
10000 times kmalloc(64)/kfree -> 186 cycles
10000 times kmalloc(128)/kfree -> 190 cycles
10000 times kmalloc(256)/kfree -> 188 cycles
10000 times kmalloc(512)/kfree -> 197 cycles
10000 times kmalloc(1024)/kfree -> 189 cycles
10000 times kmalloc(2048)/kfree -> 190 cycles
10000 times kmalloc(4096)/kfree -> 190 cycles
10000 times kmalloc(8192)/kfree -> 192 cycles
10000 times kmalloc(16384)/kfree -> 758 cycles
After:
10000 times kmalloc(8)/kfree -> 72 cycles
10000 times kmalloc(16)/kfree -> 83 cycles
10000 times kmalloc(32)/kfree -> 72 cycles
10000 times kmalloc(64)/kfree -> 72 cycles
10000 times kmalloc(128)/kfree -> 83 cycles
10000 times kmalloc(256)/kfree -> 93 cycles
10000 times kmalloc(512)/kfree -> 77 cycles
10000 times kmalloc(1024)/kfree -> 76 cycles
10000 times kmalloc(2048)/kfree -> 87 cycles
10000 times kmalloc(4096)/kfree -> 75 cycles
10000 times kmalloc(8192)/kfree -> 77 cycles
10000 times kmalloc(16384)/kfree -> 754 cycles
Concurrent alloc/free on all cpus:
Before:
Kmalloc N*(alloc free)(8): 0=176 1=177 2=176 3=176 4=184 5=176 6=176 7=176 Average=177 Kmalloc N*(alloc free)(16): 0=176 1=176 2=176 3=176 4=176 5=182 6=176 7=182 Average=177 Kmalloc N*(alloc free)(32): 0=178 1=178 2=177 3=178 4=177 5=182 6=178 7=184 Average=179 Kmalloc N*(alloc free)(64): 0=176 1=176 2=176 3=176 4=176 5=182 6=176 7=182 Average=177 Kmalloc N*(alloc free)(128): 0=176 1=178 2=176 3=176 4=176 5=176 6=176 7=182 Average=177 Kmalloc N*(alloc free)(256): 0=176 1=178 2=178 3=178 4=176 5=184 6=178 7=178 Average=178 Kmalloc N*(alloc free)(512): 0=178 1=178 2=178 3=178 4=178 5=182 6=178 7=184 Average=179 Kmalloc N*(alloc free)(1024): 0=178 1=178 2=178 3=188 4=178 5=178 6=178 7=184 Average=180 Kmalloc N*(alloc free)(2048): 0=400 1=177 2=178 3=176 4=282 5=185 6=233 7=237 Average=233 Kmalloc N*(alloc free)(4096): 0=178 1=178 2=178 3=178 4=178 5=184 6=178 7=183 Average=179
After:
Kmalloc N*(alloc free)(8): 0=73 1=73 2=73 3=71 4=71 5=71 6=71 7=75 Average=72 Kmalloc N*(alloc free)(16): 0=74 1=71 2=71 3=72 4=71 5=73 6=71 7=73 Average=72 Kmalloc N*(alloc free)(32): 0=73 1=71 2=71 3=71 4=71 5=71 6=72 7=71 Average=71 Kmalloc N*(alloc free)(64): 0=71 1=74 2=71 3=71 4=73 5=73 6=71 7=71 Average=72 Kmalloc N*(alloc free)(128): 0=71 1=71 2=81 3=73 4=71 5=71 6=75 7=75 Average=73 Kmalloc N*(alloc free)(256): 0=72 1=76 2=76 3=72 4=76 5=76 6=76 7=76 Average=75 Kmalloc N*(alloc free)(512): 0=76 1=76 2=76 3=76 4=72 5=72 6=76 7=76 Average=75 Kmalloc N*(alloc free)(1024): 0=76 1=76 2=76 3=76 4=77 5=76 6=168 7=77 Average=88 Kmalloc N*(alloc free)(2048): 0=81 1=81 2=81 3=81 4=77 5=77 6=72 7=76 Average=78 Kmalloc N*(alloc free)(4096): 0=99 1=76 2=76 3=76 4=77 5=94 6=72 7=76 Average=81
WARNING: The patch is not mature yet. There are unresolved issues around freelist traversal and fallback for arches not supporting cmpxchg etc.
The resulting kernel so far survived initial testing in kvm with the in-kernel memory allocator benchmarks and hackbench from user space.
--- mm/slub.c | 65 +++++++++++++++++++++++++++++++++++--------------------------- 1 file changed, 37 insertions(+), 28 deletions(-)
mm/slub.c +37 -28
@@ -1579,6 +1579,10 @@ static void deactivate_slab(struct kmem_if (page->freelist)stat (s,DEACTIVATE_REMOTE_FREES ); +else +/* Fix up results of any racy updates */ + page->inuse = page->objects; +/* * Merge cpu freelist into slab freelist. Typically we get here * because both freelists are empty. So this is unlikely
@@ -1586,6 +1590,7 @@ static void deactivate_slab(struct kmem_*/ while (unlikely (c->freelist)) {void **object; +void *prior; tail =0 ;/* Hot objects. Put the slab first */
@@ -1594,8 +1599,11 @@ static void deactivate_slab(struct kmem_ c->freelist =get_freepointer (s, c->freelist);/* And put onto the regular freelist */ -set_freepointer (s, object, page->freelist); - page->freelist = object; +redo: + prior = page->freelist; +set_freepointer (s, object, prior); +if (cmpxchg (&page->freelist, prior, object) != prior) + goto redo; page->inuse--; } c->page =NULL ;
@@ -1763,15 +1771,14 @@ static void *__slab_alloc(struct kmem_castat (s,ALLOC_REFILL ); load_freelist: - object = c->page->freelist; + c->page->inuse = c->page->objects; + object =xchg (&c->page->freelist,NULL );if (unlikely (!object)) goto another_slab;if (kmem_cache_debug (s)) goto debug; c->freelist =get_freepointer (s, object); - c->page->inuse = c->page->objects; - c->page->freelist =NULL ; c->node =page_to_nid (c->page); unlock_out:slab_unlock (c->page);
@@ -1970,40 +1977,48 @@ static void __slab_free(struct kmem_cach {void *prior;void **object = (void *)x; -#ifdef CONFIG_CMPXCHG_LOCAL unsigned long flags; -local_irq_save (flags); -#endif -slab_lock (page);stat (s,FREE_SLOWPATH ); -if (kmem_cache_debug (s)) goto debug; checks_ok: prior = page->freelist;set_freepointer (s, object, prior); - page->freelist = object; - page->inuse--; +if (cmpxchg (&page->freelist, prior, object) != prior) + goto checks_ok; -if (unlikely (PageSlubFrozen (page))) { +/* Racy update */ +if (unlikely (PageSlubFrozen (page) || (--page->inuse && prior))) {stat (s,FREE_FROZEN ); - goto out_unlock; +return ; } -if (unlikely (!page->inuse)) - goto slab_empty; +#ifdef CONFIG_CMPXCHG_LOCAL +local_irq_save (flags); +#endif +slab_lock (page);/* Locking prevents reduction of free list */ + +if (PageSlubFrozen (page))/* If page has been exempted by now yield */ + goto out_unlock; + +/* +* Still objects in use but those may be gone at any point now since +* we are not locking out the freepath. +*/ /* * Objects left in the slab. If it was not on the partial list before * then add it. */ -if (unlikely (!prior)) { -add_partial (get_node (s,page_to_nid (page)), page,1 ); -stat (s,FREE_ADD_PARTIAL ); - } +add_partial (get_node (s,page_to_nid (page)), page,1 ); +if (!page->inuse) +/* They are indeed gone and we need to remove the page from the partial list again */ + goto slab_empty; + +/* Objects left and slab on the partial list */ out_unlock:slab_unlock (page);#ifdef CONFIG_CMPXCHG_LOCAL
@@ -2012,13 +2027,7 @@ out_unlock:return ; slab_empty: -if (prior) { -/* -* Slab still on the partial list. -*/ -remove_partial (s, page); -stat (s,FREE_REMOVE_PARTIAL ); - } +remove_partial (s, page);slab_unlock (page);#ifdef CONFIG_CMPXCHG_LOCAL local_irq_restore (flags);
@@ -2029,7 +2038,7 @@ slab_empty: debug:if (!free_debug_processing (s, page, x, addr)) - goto out_unlock; +return ; goto checks_ok; }
RFC 7/8 slub: Add PageSlubPartial
In reply to Christoph LameterThe condition for a page being on the partial list is established by a set of combinations of values in the page struct (inuse > 0 && freelist != NULL).
With the lockless updates that set may become temporarily incoherent. Use an explit flag to signal that a page is on a partial list and allow multiple adds and removes from the partial list.
--- include/linux/page-flags.h | 2 ++ mm/slub.c | 25 +++++++++++++++---------- 2 files changed, 17 insertions(+), 10 deletions(-)
include/linux/page-flags.h +2 -0
@@ -128,6 +128,7 @@ enum pageflags {/* SLUB */ PG_slub_frozen = PG_active, + PG_slub_partial = PG_error, };#ifndef __GENERATING_BOUNDS_H
@@ -214,6 +215,7 @@ PAGEFLAG(SwapBacked, swapbacked) __CLEAR __PAGEFLAG(SlobFree, slob_free) __PAGEFLAG(SlubFrozen, slub_frozen) +__PAGEFLAG(SlubPartial, slub_partial) TESTSCFLAG(SlubPartial, slub_partial)/* * Private page markings that may be used by the filesystem that owns the page
mm/slub.c +15 -10
@@ -1308,13 +1308,15 @@ static __always_inline int slab_trylock(static void add_partial (struct kmem_cache_node *n,struct page *page,int tail) { -spin_lock (&n->list_lock); - n->nr_partial++; -if (tail) -list_add_tail (&page->lru, &n->partial); -else -list_add (&page->lru, &n->partial); -spin_unlock (&n->list_lock); +if (!TestSetPageSlubPartial (page)) { +spin_lock (&n->list_lock); + n->nr_partial++; +if (tail) +list_add_tail (&page->lru, &n->partial); +else +list_add (&page->lru, &n->partial); +spin_unlock (&n->list_lock); + } } static inline void __remove_partial(struct kmem_cache_node *n,
@@ -1322,15 +1324,18 @@ static inline void __remove_partial(stru {list_del (&page->lru); n->nr_partial--; +__ClearPageSlubPartial (page); }static void remove_partial (struct kmem_cache *s,struct page *page) {struct kmem_cache_node *n =get_node (s,page_to_nid (page)); -spin_lock (&n->list_lock); -__remove_partial (n, page); -spin_unlock (&n->list_lock); +if (TestClearPageSlubPartial (page)) { +spin_lock (&n->list_lock); +__remove_partial (n, page); +spin_unlock (&n->list_lock); + } }/*
RFC 6/8 Lockless (and preemptless) fastpaths for slub
In reply to Christoph LameterUse the this_cpu_cmpxchg_double functionality to implement a lockless allocation algorithm on arches that support fast this_cpu_ops.
Each of the per cpu pointers is paired with a transaction id that ensures that updates of the per cpu information can only occur in sequence on a certain cpu.
A transaction id is a "long" integer that is comprised of an event number and the cpu number. The event number is incremented for every change to the per cpu state. This means that the cmpxchg instruction can verify for an update that nothing interfered and that we are updating the percpu structure for the processor where we picked up the information and that we are also currently on that processor when we update the information.
This results in a significant decrease of the overhead in the fastpaths. It also makes it easy to adopt the fast path for realtime kernels since this is lockless and does not require that the use of the current per cpu area over the critical section. It is only important that the per cpu area is current at the beginning of the critical section and at that end.
So there is no need even to disable preemption which will make the allocations scale well in a RT environment.
Test results show that the fastpath cycle count is reduced by up to ~ 40% (alloc/free test goes from ~140 cycles down to ~80). The slowpath for kfree adds a few cycles.
Sadly this does nothing for the slowpath which is where the main issues with performance in slub are but the best case performance rises significantly. (For that see the next patches)
Kmalloc: alloc/free test
Before:
10000 times kmalloc(8)/kfree -> 142 cycles
10000 times kmalloc(16)/kfree -> 142 cycles
10000 times kmalloc(32)/kfree -> 142 cycles
10000 times kmalloc(64)/kfree -> 142 cycles
10000 times kmalloc(128)/kfree -> 142 cycles
10000 times kmalloc(256)/kfree -> 140 cycles
10000 times kmalloc(512)/kfree -> 140 cycles
10000 times kmalloc(1024)/kfree -> 144 cycles
10000 times kmalloc(2048)/kfree -> 144 cycles
10000 times kmalloc(4096)/kfree -> 144 cycles
10000 times kmalloc(8192)/kfree -> 144 cycles
10000 times kmalloc(16384)/kfree -> 913 cycles
After:
10000 times kmalloc(8)/kfree -> 81 cycles
10000 times kmalloc(16)/kfree -> 81 cycles
10000 times kmalloc(32)/kfree -> 81 cycles
10000 times kmalloc(64)/kfree -> 81 cycles
10000 times kmalloc(128)/kfree -> 81 cycles
10000 times kmalloc(256)/kfree -> 87 cycles
10000 times kmalloc(512)/kfree -> 87 cycles
10000 times kmalloc(1024)/kfree -> 87 cycles
10000 times kmalloc(2048)/kfree -> 84 cycles
10000 times kmalloc(4096)/kfree -> 81 cycles
10000 times kmalloc(8192)/kfree -> 81 cycles
10000 times kmalloc(16384)/kfree -> 927 cycles
Kmalloc: Repeatedly allocate then free test
Before:
10000 times kmalloc(8) -> 102 cycles kfree -> 111 cycles 10000 times kmalloc(16) -> 101 cycles kfree -> 111 cycles 10000 times kmalloc(32) -> 120 cycles kfree -> 114 cycles 10000 times kmalloc(64) -> 161 cycles kfree -> 130 cycles 10000 times kmalloc(128) -> 284 cycles kfree -> 129 cycles 10000 times kmalloc(256) -> 410 cycles kfree -> 134 cycles 10000 times kmalloc(512) -> 312 cycles kfree -> 197 cycles 10000 times kmalloc(1024) -> 377 cycles kfree -> 494 cycles 10000 times kmalloc(2048) -> 571 cycles kfree -> 522 cycles 10000 times kmalloc(4096) -> 674 cycles kfree -> 565 cycles 10000 times kmalloc(8192) -> 836 cycles kfree -> 648 cycles 10000 times kmalloc(16384) -> 1201 cycles kfree -> 775 cycles
After:
10000 times kmalloc(8) -> 69 cycles kfree -> 115 cycles 10000 times kmalloc(16) -> 73 cycles kfree -> 115 cycles 10000 times kmalloc(32) -> 86 cycles kfree -> 119 cycles 10000 times kmalloc(64) -> 122 cycles kfree -> 125 cycles 10000 times kmalloc(128) -> 247 cycles kfree -> 132 cycles 10000 times kmalloc(256) -> 375 cycles kfree -> 137 cycles 10000 times kmalloc(512) -> 283 cycles kfree -> 183 cycles 10000 times kmalloc(1024) -> 316 cycles kfree -> 504 cycles 10000 times kmalloc(2048) -> 516 cycles kfree -> 531 cycles 10000 times kmalloc(4096) -> 610 cycles kfree -> 570 cycles 10000 times kmalloc(8192) -> 759 cycles kfree -> 651 cycles 10000 times kmalloc(16384) -> 1169 cycles kfree -> 778 cycles
--- include/linux/slub_def.h | 5 - mm/slub.c | 228 ++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 219 insertions(+), 14 deletions(-)
include/linux/slub_def.h +4 -1
@@ -36,7 +36,10 @@ enum stat_item { NR_SLUB_STAT_ITEMS };struct kmem_cache_cpu { -void **freelist;/* Pointer to first free per cpu object */ +void **freelist;/* Pointer to next available object */ +#ifdef CONFIG_CMPXCHG_LOCAL +unsigned long tid;/* Globally unique transaction id */ +#endif struct page *page;/* The slab from which we are allocating */ int node;/* The node of the page (or -1 for debug) */ #ifdef CONFIG_SLUB_STATS
mm/slub.c +215 -13
338 lines not on this page · raw message
RFC 5/8 vmstat: User per cpu atomics to avoid interrupt disable / enable
In reply to Christoph LameterCurrently the operations to increment vm counters must disable interrupts in order to not mess up their housekeeping of counters.
So use this_cpu_cmpxchg() to avoid the overhead. Since we can no longer count on preremption being disabled we still have some minor issues. The fetching of the counter thresholds is racy.
A threshold from another cpu may be applied if we happen to be rescheduled on another cpu. However, the following vmstat operation will then bring the counter again under the threshold limit.
The operations for __xxx_zone_state are not changed since the caller has taken care of the synchronization needs (and therefore the cycle count is even less than the optimized version for the irq disable case provided here).
The optimization using this_cpu_cmpxchg will only be used if the arch supports efficient this_cpu_ops (must have CONFIG_CMPXCHG_LOCAL set!)
The use of this_cpu_cmpxchg reduces the cycle count for the counter operations by %80 (inc_zone_page_state goes from 170 cycles to 32).
--- mm/vmstat.c | 101 +++++++++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 87 insertions(+), 14 deletions(-)
mm/vmstat.c +87 -14
@@ -185,20 +185,6 @@ void __mod_zone_page_state(struct zone *EXPORT_SYMBOL (__mod_zone_page_state);/* -* For an unknown interrupt state -*/ -void mod_zone_page_state (struct zone *zone,enum zone_stat_item item, -int delta) -{ -unsigned long flags; - -local_irq_save (flags); -__mod_zone_page_state (zone, item, delta); -local_irq_restore (flags); -} -EXPORT_SYMBOL (mod_zone_page_state); - -/* * Optimized increment and decrement functions. * * These are only for a single page and therefore can take a struct page *
@@ -265,6 +251,92 @@ void __dec_zone_page_state(struct page * }EXPORT_SYMBOL (__dec_zone_page_state); +#ifdef CONFIG_CMPXCHG_LOCAL +/* +* If we have cmpxchg_local support then we do not need to incur the overhead +* that comes with local_irq_save/restore if we use this_cpu_cmpxchg. +* +* mod_state() modifies the zone counter state through atomic per cpu +* operations. +* +* Overstep mode specifies how overstep should handled: +* 0 No overstepping +* 1 Overstepping half of threshold +* -1 Overstepping minus half of threshold +*/ +static inline void mod_state (struct zone *zone, +enum zone_stat_item item,int delta,int overstep_mode) +{ +struct per_cpu_pageset __percpu *pcp = zone->pageset; +s8 __percpu *p = pcp->vm_stat_diff + item; +long o, n, t, z; + +do { + z =0 ;/* overflow to zone counters */ + +/* +* The fetching of the stat_threshold is racy. We may apply +* a counter threshold to the wrong the cpu if we get +* rescheduled while executing here. However, the following +* will apply the threshold again and therefore bring the +* counter under the threshold. +*/ + t =this_cpu_read (pcp->stat_threshold); + + o =this_cpu_read (*p); + n = delta + o; + +if (n > t || n < -t) { +int os = overstep_mode * (t >>1 ) ; + +/* Overflow must be added to zone counters */ + z = n + os; + n = -os; + } + }while (this_cpu_cmpxchg (*p, o, n) != o); + +if (z) +zone_page_state_add (z, zone, item); +} + +void mod_zone_page_state (struct zone *zone,enum zone_stat_item item, +int delta) +{ +mod_state (zone, item, delta,0 ); +} +EXPORT_SYMBOL (mod_zone_page_state); + +void inc_zone_state (struct zone *zone,enum zone_stat_item item) +{ +mod_state (zone, item,1 ,1 ); +} + +void inc_zone_page_state (struct page *page,enum zone_stat_item item) +{ +mod_state (page_zone (page), item,1 ,1 ); +} +EXPORT_SYMBOL (inc_zone_page_state); + +void dec_zone_page_state (struct page *page,enum zone_stat_item item) +{ +mod_state (page_zone (page), item,-1 ,-1 ); +} +EXPORT_SYMBOL (dec_zone_page_state); +#else +/* +* Use interrupt disable to serialize counter updates +*/ +void mod_zone_page_state (struct zone *zone,enum zone_stat_item item, +int delta) +{ +unsigned long flags; + +local_irq_save (flags); +__mod_zone_page_state (zone, item, delta); +local_irq_restore (flags); +} +EXPORT_SYMBOL (mod_zone_page_state); +void inc_zone_state (struct zone *zone,enum zone_stat_item item) {unsigned long flags;
@@ -295,6 +367,7 @@ void dec_zone_page_state(struct page *pa local_irq_restore(flags); }EXPORT_SYMBOL (dec_zone_page_state); +#endif /* * Update the zone counters for one cpu.
RFC 3/8 x86: this_cpu_cmpxchg and this_cpu_cmpxchg_double operations
In reply to Christoph LameterProvide support as far as the hardware capabilities of the x86 cpus allow.
V1->V2:
- Mark %rdx clobbering during cmpxchg16b
- Provide emulation of cmpxchg16b for early AMD processors
--- arch/x86/Kconfig.cpu | 3 arch/x86/include/asm/percpu.h | 183 +++++++++++++++++++++++++++++++++++++++++- arch/x86/lib/Makefile | 1 arch/x86/lib/cmpxchg16b_emu.S | 55 ++++++++++++ 4 files changed, 238 insertions(+), 4 deletions(-)
arch/x86/lib/Makefile +1 -0
@@ -42,4 +42,5 @@ else lib-y += memmove_64.o memset_64.o lib-y += copy_user_64.o rwlock_64.o copy_user_nocache_64.o lib-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem_64.o + lib-y += cmpxchg16b_emu.o endif
arch/x86/lib/cmpxchg16b_emu.S new file +55 -0
55 lines not on this page · raw message
arch/x86/include/asm/percpu.h +179 -4
205 lines not on this page · raw message
arch/x86/Kconfig.cpu +3 -0
@@ -310,6 +310,9 @@ config X86_INTERNODE_CACHE_SHIFT config X86_CMPXCHG def_bool X86_64 || (X86_32 && !M386) +config CMPXCHG_LOCAL + def_bool X86_64 || (X86_32 && !M386) + config X86_L1_CACHE_SHIFT int default "7" if MPENTIUM4 || MPSC
RFC 2/8 Fallback to atomic xchg, cmpxchg
In reply to Christoph LameterSorry this should have been:
Subject: this_cpu_xchg/cmpxchg: Fall back to atomic xchg, cmpxchg
It may be better if we fall back to the full operation instead of simulation through irq disable etc.
--- include/linux/percpu.h | 31 +++---------------------------- 1 file changed, 3 insertions(+), 28 deletions(-)
include/linux/percpu.h +3 -28
@@ -361,14 +361,7 @@ do { \# define __this_cpu_xchg (pcp, nval) __pcpu_size_call_return2(__this_cpu_xchg_, (pcp), nval)#endif -#define _this_cpu_generic_xchg (pcp, nval) \ -({ typeof(pcp) ret__; \ - preempt_disable(); \ - ret__ = __this_cpu_read(pcp); \ - __this_cpu_write(pcp, nval); \ - preempt_enable(); \ - ret__; \ -}) +#define _this_cpu_generic_xchg (pcp, nval) xchg(__this_cpu_ptr(&(pcp)), nval)#ifndef this_cpu_xchg# ifndef this_cpu_xchg_1
@@ -386,15 +379,7 @@ do { \# define this_cpu_xchg (pcp, nval) __pcpu_size_call_return2(this_cpu_xchg_, (pcp), nval)#endif -#define _this_cpu_generic_cmpxchg (pcp, oval, nval) \ -({ typeof(pcp) ret__; \ - preempt_disable(); \ - ret__ = __this_cpu_read(pcp); \ - if (ret__ == (oval)) \ - __this_cpu_write(pcp, nval); \ - preempt_enable(); \ - ret__; \ -}) +#define _this_cpu_generic_cmpxchg (pcp, oval, nval) cmpxchg(__this_cpu_ptr(&(pcp)), oval, nval);#ifndef this_cpu_cmpxchg# ifndef this_cpu_cmpxchg_1
@@ -892,17 +877,7 @@ do { \# define irqsafe_cpu_xor (pcp, val) __pcpu_size_call(irqsafe_cpu_xor_, (val))#endif -#define irqsafe_cpu_generic_cmpxchg (pcp, oval, nval) \ -({ \ - typeof(pcp) ret__; \ - unsigned long flags; \ - local_irq_save(flags); \ - ret__ = __this_cpu_read(pcp); \ - if (ret__ == (oval)) \ - __this_cpu_write(pcp, nval); \ - local_irq_restore(flags); \ - ret__; \ -}) +#define irqsafe_cpu_generic_cmpxchg (pcp, oval, nval) cmpxchg(__this_cpu_ptr(&(pcp), oval, nval)#ifndef irqsafe_cpu_cmpxchg# ifndef irqsafe_cpu_cmpxchg_1
Hi,
[ Sorry for the delay everyone. ]
Christoph, do you mind sending me a series that you think is stable enough for linux-next? The numbers are pretty impressive and I'd love to give them some testing locally and in linux-next.
How should we coordinate the per-CPU ops patches, btw? We had pretty good experience with the per-cpu allocator patches where Tejun merged the patches in his tree and I pulled it in slab.git. Who handles the per-cpu ops?
Pekka
Tejun Heo
maintainerHello,
Quoted 10 lines · Pekka Enberg
[ Sorry for the delay everyone. ]
Christoph, do you mind sending me a series that you think is stable enough for linux-next? The numbers are pretty impressive and I'd love to give them some testing locally and in linux-next.
How should we coordinate the per-CPU ops patches, btw? We had pretty good experience with the per-cpu allocator patches where Tejun merged the patches in his tree and I pulled it in slab.git. Who handles the per-cpu ops?
I'll be happy to apply the per-cpu ops part. Christoph, would it be okay to apply patches from this posting?
Thanks.
Signature · 1 line
tejun
Quoted 3 lines · Pekka Enberg
Christoph, do you mind sending me a series that you think is stable enough for linux-next? The numbers are pretty impressive and I'd love to give them some testing locally and in linux-next.
I'd like to see the cpu ops *updates* patchset getting into -next and then be hopefully merged for the next relased. The cpuops advanced patchset (this one) can go into -next *after* the next merge phase.
Quoted 2 lines · Tejun Heo
I'll be happy to apply the per-cpu ops part. Christoph, would it be okay to apply patches from this posting?
No. Please focus merging efforts on the cpu ops updates patchset. There is a reason that there is an "rfc" in the title of this patchset (cpu ops avanced).
Re: [rfc: cpuops adv V1 3/8] x86: this_cpu_cmpxchg and this_cpu_cmpxchg_double operations
In reply to Christoph LameterQuoted 9 lines · Christoph Lameter
Provide support as far as the hardware capabilities of the x86 cpus allow.
+/*
+ * Beware: xchg on x86 has an implied lock prefix. There will be the cost of
+ * full lock semantics even though they are not needed.
+ */
Perhaps we can use cmpxchg instead of xchg to avoid this? costs one more instruction but may be worth it.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [rfc: cpuops adv V1 3/8] x86: this_cpu_cmpxchg and this_cpu_cmpxchg_double operations
Quoted 6 lines · Avi Kivity
Quoted 5 lines · Christoph Lameter
Provide support as far as the hardware capabilities of the x86 cpus allow.
Quoted 8 lines
Quoted 5 lines
+/*
+ * Beware: xchg on x86 has an implied lock prefix. There will be the cost of
+ * full lock semantics even though they are not needed.
+ */Perhaps we can use cmpxchg instead of xchg to avoid this? costs one more instruction but may be worth it.
Hmmm... Maybe good since I also need a xchg_double. And xchg_double can only be realized with cmpxchg16b. Using cmpxchg would make it consistent.
Re: [rfc: cpuops adv V1 3/8] x86: this_cpu_cmpxchg and this_cpu_cmpxchg_double operations
Quoted 14 lines · Christoph Lameter
Quoted 12 lines · Avi Kivity
Quoted 10 lines · Christoph Lameter
Provide support as far as the hardware capabilities of the x86 cpus allow.
+/*
+ * Beware: xchg on x86 has an implied lock prefix. There will be the cost of
+ * full lock semantics even though they are not needed.
+ */
Quoted 5 lines
Quoted 2 lines
Perhaps we can use cmpxchg instead of xchg to avoid this? costs one more instruction but may be worth it.
Hmmm... Maybe good since I also need a
xchg_double. Andxchg_doublecan only be realized with cmpxchg16b. Using cmpxchg would make it consistent.
I don't think we need to worry about consistency, this is an implementation not an interface.
We have three choices:
xchg %1, %0
atomic, one instruction
1: cmpxchg %2, %0
jnz 1b
two non-atomic instructions, potential mispredicted jump, extra clobber (%1 == "=a")
mov %0, %1
1: cmpxchg %2, %0
jnz 1b
three non-atomic instructions, no mispredict, extra clobber
Signature · 1 line
error compiling committee.c: too many arguments to function