percpu_ref & block: reduce memory footprint of percpu_ref in fast path
Subsystems (5)
v6 cover percpu_ref & block: reduce memory footprint of percpu_ref in fast path
Hi,
The 1st patch removes memory footprint of percpu_ref in fast path from 7 words to 2 words, since it is often used in fast path and embedded in user struct.
The 2nd patch moves .q_usage_counter to 1st cacheline of 'request_queue'.
Simple test on null_blk shows ~2% IOPS boost on one 16cores(two threads per core) machine, dual socket/numa.
V6:
- drop the 1st patch which adds percpu_ref_is_initialized() for MD only, since Christoph doesn't like it
V5:
- fix memory leak on ref->data, only percpu_ref_exit() of patch 2 is modified.
V4:
- rename percpu_ref_inited as percpu_ref_is_initialized
V3:
- fix kernel oops on MD
- add patch for avoiding to use percpu-refcount internal from md code
- pass Red Hat CKI test which is done by Veronika Kabatova
V2:
- pass 'gfp' to kzalloc() for fixing block/027 failure reported by kernel test robot
- protect percpu_ref_is_zero() with destroying percpu-refcount by spin lock
Ming Lei (2):
percpu_ref: reduce memory footprint of percpu_ref in fast path
block: move 'q_usage_counter' into front of 'request_queue'
drivers/infiniband/sw/rdmavt/mr.c | 2 +- include/linux/blkdev.h | 3 +- include/linux/percpu-refcount.h | 45 ++++------ lib/percpu-refcount.c | 131 ++++++++++++++++++++++-------- 4 files changed, 118 insertions(+), 63 deletions(-)
Signature · 1 line
2.25.2
PATCH v6 1/2 percpu_ref: reduce memory footprint of percpu_ref in fast path
'struct percpu_ref' is often embedded into one user structure, and the instance is usually referenced in fast path, however actually only 'percpu_count_ptr' is needed in fast path.
So move other fields into one new structure of 'percpu_ref_data', and allocate it dynamically via kzalloc(), then memory footprint of 'percpu_ref' in fast path is reduced a lot and becomes suitable to put into hot cacheline of user structure.
--- drivers/infiniband/sw/rdmavt/mr.c | 2 +- include/linux/percpu-refcount.h | 45 ++++------ lib/percpu-refcount.c | 131 ++++++++++++++++++++++-------- 3 files changed, 116 insertions(+), 62 deletions(-)
drivers/infiniband/sw/rdmavt/mr.c +1 -1
@@ -499,7 +499,7 @@ static int rvt_check_refs(struct rvt_mregion *mr, const char *t)rvt_pr_err (rdi,"%s timeout mr %p pd %p lkey %x refcount %ld\n" , t, mr, mr->pd, mr->lkey, -atomic_long_read (&mr->refcount.count)); +atomic_long_read (&mr->refcount.data->count));rvt_get_mr (mr);return -EBUSY ; }
include/linux/percpu-refcount.h +17 -28
91 lines not on this page · raw message
lib/percpu-refcount.c +98 -33
258 lines not on this page · raw message
Signature · 1 line
2.25.2
Tejun Heo
maintainerRe: [PATCH V6 1/2] percpu_ref: reduce memory footprint of percpu_ref in fast path
Quoted 22 lines · Ming Lei
include/linux/percpu-refcount.h +2
-6@@ -92,18 +92,23 @@ enum { PERCPU_REF_ALLOW_REINIT = 1 << 2, }; -struct percpu_ref { +struct percpu_ref_data {atomic_long_t count; -/* -* The low bit of the pointer indicates whether the ref is in percpu -* mode; if set, then get/put will manipulate the atomic_t. -*/ -unsigned long percpu_count_ptr;percpu_ref_func_t *release;percpu_ref_func_t *confirm_switch;bool force_atomic:1 ;bool allow_reinit:1 ;struct rcu_head rcu; +struct percpu_ref *ref;
include/linux/percpu-refcount.h enum { · line 112
Quoted 10 lines
include/linux/percpu-refcount.h +9
-0+}; + +struct percpu_ref { +/* +* The low bit of the pointer indicates whether the ref is in percpu +* mode; if set, then get/put will manipulate the atomic_t. +*/ +unsigned long percpu_count_ptr; +struct percpu_ref_data *data; };
Can you please add a comment explaining why the two structs are split?
Thanks.
Signature · 1 line
tejun
Re: [PATCH V6 1/2] percpu_ref: reduce memory footprint of percpu_ref in fast path
Quoted 27 lines · Tejun Heo
Quoted 26 lines · Ming Lei
include/linux/percpu-refcount.h +6
-6@@ -92,18 +92,23 @@ enum { PERCPU_REF_ALLOW_REINIT = 1 << 2, }; -struct percpu_ref { +struct percpu_ref_data {atomic_long_t count; -/* -* The low bit of the pointer indicates whether the ref is in percpu -* mode; if set, then get/put will manipulate the atomic_t. -*/ -unsigned long percpu_count_ptr;percpu_ref_func_t *release;percpu_ref_func_t *confirm_switch;bool force_atomic:1 ;bool allow_reinit:1 ;struct rcu_head rcu; +struct percpu_ref *ref; +}; + +struct percpu_ref { +/*
Quoted 8 lines
include/linux/percpu-refcount.h enum { · line 112
Quoted 6 lines
include/linux/percpu-refcount.h +5
-0+* The low bit of the pointer indicates whether the ref is in percpu +* mode; if set, then get/put will manipulate the atomic_t. +*/ +unsigned long percpu_count_ptr; +struct percpu_ref_data *data; };Can you please add a comment explaining why the two structs are split?
Sure, will do that in next version.
Thanks,
Ming
PATCH v6 2/2 block: move 'q_usage_counter' into front of 'request_queue'
In reply to Ming LeiThe field of 'q_usage_counter' is always fetched in fast path of every block driver, and move it into front of 'request_queue', so it can be fetched into 1st cacheline of 'request_queue' instance.
--- include/linux/blkdev.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
include/linux/blkdev.h +2 -1
@@ -397,6 +397,8 @@ struct request_queue {struct request *last_merge;struct elevator_queue *elevator; +struct percpu_ref q_usage_counter; +struct blk_queue_stats *stats;struct rq_qos *rq_qos;
@@ -569,7 +571,6 @@ struct request_queue {* percpu_ref_kill() and percpu_ref_reinit(). */ struct mutex mq_freeze_lock; -struct percpu_ref q_usage_counter;struct blk_mq_tag_set *tag_set;struct list_head tag_set_list;
Signature · 1 line
2.25.2