bitops: add _local bitops
Subsystems (6)
PATCH v2 1/1 bitops: add _local bitops
kvm needs to update some hypervisor variables atomically in a sense that the operation can't be interrupted in the middle. However the hypervisor always runs on the same CPU so it does not need any memory barrier or lock prefix.
Add _local bitops for this purpose: define them
as non-atomics for x86 and (for now) atomics for
everyone else.
Uses are not restricted to virtualization: they might be useful to communicate with an interrupt handler if we know that it's running on the same CPU.
---
Changes from v1:
- Fix x86 version so it includes optimization barriers as suggested by Peter Anvin
- Fix documentation as suggested by Avi Kivity
- Remove duplicate files as suggested by Arnd Bergmann
Link to discussion preceding v1:
http://www.spinics.net/lists/kvm/msg72241.html
Documentation/atomic_ops.txt | 19 ++++ arch/x86/include/asm/bitops.h | 132 +++++++++++++++++++++++++++++ include/asm-generic/bitops.h | 1 + include/asm-generic/bitops/local-atomic.h | 114 +++++++++++++++++++++++++ include/linux/bitops.h | 8 ++ 5 files changed, 274 insertions(+), 0 deletions(-) create mode 100644 include/asm-generic/bitops/local-atomic.h
Documentation/atomic_ops.txt +19 -0
@@ -520,6 +520,25 @@ The __clear_bit_unlock version is non-atomic, however it still implements unlock barrier semantics. This can be useful if the lock itself is protecting the other bits in the word. +Local versions of the bitmask operations are also provided. They are used in +contexts where the operations need to be performed atomically with respect to +the local CPU, but no other CPU accesses the bitmask. This assumption makes it +possible to avoid the need for SMP protection and use less expensive atomic +operations in the implementation. +They have names similar to the above bitmask operation interfaces, +except that _local is appended to the interface name. + + void set_bit_local(unsigned long nr, volatile unsigned long *addr); + void clear_bit_local(unsigned long nr, volatile unsigned long *addr); + void change_bit_local(unsigned long nr, volatile unsigned long *addr); + int test_and_set_bit_local(unsigned long nr, volatile unsigned long *addr); + int test_and_clear_bit_local(unsigned long nr, volatile unsigned long *addr); + int test_and_change_bit_local(unsigned long nr, volatile unsigned long *addr); + +These local variants are useful for example if the bitmask may be accessed from +a local intrerrupt, or from a hypervisor on the same CPU if running in a VM. +These local variants also do not have any special memory barrier semantics. + Finally, there are non-atomic versions of the bitmask operations provided. They are used in contexts where some other higher-level SMP locking scheme is being used to protect the bitmask, and thus less
arch/x86/include/asm/bitops.h +132 -0
174 lines not on this page · raw message
include/asm-generic/bitops.h +1 -0
@@ -40,5 +40,6 @@#include <asm-generic/bitops/non-atomic.h> #include <asm-generic/bitops/le.h> #include <asm-generic/bitops/ext2-atomic.h> +#include <asm-generic/bitops/local-atomic.h> #endif /* __ASM_GENERIC_BITOPS_H */
include/asm-generic/bitops/local-atomic.h new file +114 -0
114 lines not on this page · raw message
include/linux/bitops.h +8 -0
@@ -197,5 +197,13 @@ extern unsigned long find_last_bit(const unsigned long *addr, unsigned long size);#endif +/** +* Include the generic version for local atomics unless +* an architecture overrides it. +* */ +#ifndef ARCH_HAS_BITOPS_LOCAL_ATOMIC +#include "include/asm-generic/bitops/local-atomic.h" +#endif +#endif /* __KERNEL__ */ #endif
Signature · 1 line
MST
Quoted 9 lines · Michael S. Tsirkin
kvm needs to update some hypervisor variables atomically in a sense that the operation can't be interrupted in the middle. However the hypervisor always runs on the same CPU so it does not need any memory barrier or lock prefix.
Add
_localbitops for this purpose: define them as non-atomics for x86 and (for now) atomics for everyone else.
Have you tried to use the this_cpu_ops for that purpose? They create the per cpu atomic instructions that you want without a lock prefix and can also relocate the per cpu pointer to the correct processor via a segment register prefix.
There are no bit operations provided right now but those can either be improvised using this_cpu_cmpxchg or added.
Quoted 12 lines · Christoph Lameter
Quoted 9 lines · Michael S. Tsirkin
kvm needs to update some hypervisor variables atomically in a sense that the operation can't be interrupted in the middle. However the hypervisor always runs on the same CPU so it does not need any memory barrier or lock prefix.
Add
_localbitops for this purpose: define them as non-atomics for x86 and (for now) atomics for everyone else.
Quoted 7 lines
Have you tried to use the
this_cpu_opsfor that purpose? They create the per cpu atomic instructions that you want without a lock prefix and can also relocate the per cpu pointer to the correct processor via a segment register prefix.There are no bit operations provided right now but those can either be improvised using
this_cpu_cmpxchgor added.
this_cpu_xchg() should be sufficient, since only bit zero has any meaning in our use case (so xchg with zero is equivalent to test_and_clear_bit).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 18 lines · Avi Kivity
Quoted 17 lines · Christoph Lameter
Quoted 9 lines · Michael S. Tsirkin
kvm needs to update some hypervisor variables atomically in a sense that the operation can't be interrupted in the middle. However the hypervisor always runs on the same CPU so it does not need any memory barrier or lock prefix.
Add
_localbitops for this purpose: define them as non-atomics for x86 and (for now) atomics for everyone else.Have you tried to use the
this_cpu_opsfor that purpose? They create the per cpu atomic instructions that you want without a lock prefix and can also relocate the per cpu pointer to the correct processor via a segment register prefix.
Quoted 6 lines
Quoted 2 lines
There are no bit operations provided right now but those can either be improvised using
this_cpu_cmpxchgor added.
this_cpu_xchg()should be sufficient, since only bit zero has any meaning in our use case (so xchg with zero is equivalent totest_and_clear_bit).
Yes it should work. No idea how it'd perform:arch/x86/include/asm/percpu.h implies it's expensive. My latest version simply documents what __test_and_clear does anyway.
Which was indicated is acceptable ...
Did you change your mind?
Quoted 3 lines
Signature · 2 lines
-- error compiling committee.c: too many arguments to function
Quoted 22 lines · Michael S. Tsirkin
Quoted 21 lines · Avi Kivity
Quoted 19 lines · Christoph Lameter
Quoted 9 lines · Michael S. Tsirkin
kvm needs to update some hypervisor variables atomically in a sense that the operation can't be interrupted in the middle. However the hypervisor always runs on the same CPU so it does not need any memory barrier or lock prefix.
Add
_localbitops for this purpose: define them as non-atomics for x86 and (for now) atomics for everyone else.Have you tried to use the
this_cpu_opsfor that purpose? They create the per cpu atomic instructions that you want without a lock prefix and can also relocate the per cpu pointer to the correct processor via a segment register prefix.There are no bit operations provided right now but those can either be improvised using
this_cpu_cmpxchgor added.
Quoted 6 lines
Quoted 3 lines
this_cpu_xchg()should be sufficient, since only bit zero has any meaning in our use case (so xchg with zero is equivalent totest_and_clear_bit).Yes it should work. No idea how it'd perform:
arch/x86/include/asm/percpu.himplies it's expensive.
It says a naive implementation is expensive, whereas the implementation actually used (a cmpxchg loop) is better.
Quoted 4 lines
My latest version
simply documents what__test_and_cleardoes anyway. Which was indicated is acceptable ...
Did you change your mind?
No.
Signature · 1 line
error compiling committee.c: too many arguments to function