percpu_ref: remove unnecessary ACCESS_ONCE() in percpu_ref_tryget_live()
Tejun Heo
maintainerPATCH 1/2 percpu_ref: remove unnecessary ACCESS_ONCE() in percpu_ref_tryget_live()
__ref_is_percpu() needs the implied ACCESS_ONCE() in lockless_dereference() on @ref->percpu_count_ptr because the value is tested for !__PERCPU_REF_ATOMIC, which may be set asynchronously, and then used as a pointer. If the compiler generates a separate fetch when using it as a pointer, __PERCPU_REF_ATOMIC may be set in between contaminating the pointer value.
percpu_ref_tryget_live() also uses ACCESS_ONCE() to test __PERCPU_REF_DEAD; however, there's no reason for this. I just copied ACCESS_ONCE() usage blindly from __ref_is_percpu(). All it does is confusing people trying to understand what's going on.
This patch removes the unnecessary ACCESS_ONCE() usage from percpu_ref_tryget_live() and adds a comment explaining why __ref_is_percpu() needs it.
--- include/linux/percpu-refcount.h | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-)
include/linux/percpu-refcount.h +17 -3
@@ -128,8 +128,22 @@ static inline void percpu_ref_kill(strucstatic inline bool __ref_is_percpu (struct percpu_ref *ref,unsigned long __percpu **percpu_countp) { -/* paired with smp_store_release() in percpu_ref_reinit() */ -unsigned long percpu_ptr =lockless_dereference (ref->percpu_count_ptr); +unsigned long percpu_ptr; + +/* +* The value of @ref->percpu_count_ptr is tested for +* !__PERCPU_REF_ATOMIC, which may be set asynchronously, and then +* used as a pointer. If the compiler generates a separate fetch +* when using it as a pointer, __PERCPU_REF_ATOMIC may be set in +* between contaminating the pointer value, meaning that +* ACCESS_ONCE() is required when fetching it. +* +* Also, we need a data dependency barrier to be paired with +* smp_store_release() in __percpu_ref_switch_to_percpu(). +* +* Use lockless deref which contains both. +*/ + percpu_ptr =lockless_dereference (ref->percpu_count_ptr);/* * Theoretically, the following could test just ATOMIC; however,
@@ -233,7 +247,7 @@ static inline bool percpu_ref_tryget_livif (__ref_is_percpu (ref, &percpu_count)) {this_cpu_inc (*percpu_count); ret = true; - }else if (!(ACCESS_ONCE (ref->percpu_count_ptr) & __PERCPU_REF_DEAD)) { + }else if (!(ref->percpu_count_ptr & __PERCPU_REF_DEAD)) { ret =atomic_long_inc_not_zero (&ref->count); }
Tejun Heo
maintainerPATCH 2/2 percpu_ref: implement percpu_ref_is_dying()
Implement percpu_ref_is_dying() which tests whether the ref is dying or dead. This is useful to determine the current state when a percpu_ref is used as a cyclic on/off switch via kill and reinit.
---
Hello,
This will be used by the scheduled cgroup writeback support. I'm applying these two to percpu/for-3.20.
Thanks.
include/linux/percpu-refcount.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
include/linux/percpu-refcount.h +14 -0
@@ -295,6 +295,20 @@ static inline void percpu_ref_put(struct }/** +* percpu_ref_is_dying - test whether a percpu refcount is dying or dead +* @ref: percpu_ref to test +* +* Returns %true if @ref is dying or dead. +* +* This function is safe to call as long as @ref is between init and exit +* and the caller is responsible for synchronizing against state changes. +*/ +static inline bool percpu_ref_is_dying (struct percpu_ref *ref) +{ +return ref->percpu_count_ptr & __PERCPU_REF_DEAD; +} + +/** * percpu_ref_is_zero - test whether a percpu refcount reached zero * @ref: percpu_ref to test *
Re: [PATCH 2/2 percpu/for-3.20] percpu_ref: implement percpu_ref_is_dying()
Quoted 3 lines · Tejun Heo
Implement
percpu_ref_is_dying()which tests whether the ref is dying or dead. This is useful to determine the current state when apercpu_refis used as a cyclic on/off switch via kill and reinit.
Quoted 7 lines · Tejun Heo
__ref_is_percpu()needs the impliedACCESS_ONCE()inlockless_dereference()on @ref->percpu_count_ptr because the value is tested for !__PERCPU_REF_ATOMIC, which may be set asynchronously, and then used as a pointer. If the compiler generates a separate fetch when using it as a pointer,__PERCPU_REF_ATOMICmay be set in between contaminating the pointer value.
Quoted 8 lines
percpu_ref_tryget_live()also usesACCESS_ONCE()to test__PERCPU_REF_DEAD; however, there's no reason for this. I just copiedACCESS_ONCE()usage blindly from__ref_is_percpu(). All it does is confusing people trying to understand what's going on.This patch removes the unnecessary
ACCESS_ONCE()usage frompercpu_ref_tryget_live()and adds a comment explaining why__ref_is_percpu()needs it.
lockless_dereference() is new - cool! I've been wanting that without even realizing it.