percpu_ref & block: reduce memory footprint of percpu_ref in fast path
Subsystems (6)
v5 cover percpu_ref & block: reduce memory footprint of percpu_ref in fast path
Hi,
The 1st patch removes memory footprint of percpu_ref in fast path from 7 words to 2 words, since it is often used in fast path and embedded in user struct.
The 2nd patch moves .q_usage_counter to 1st cacheline of 'request_queue'.
Simple test on null_blk shows ~2% IOPS boost on one 16cores(two threads per core) machine, dual socket/numa.
V5:
- fix memory leak on ref->data, only percpu_ref_exit() of patch 2 is modified.
V4:
- rename percpu_ref_inited as percpu_ref_is_initialized
V3:
- fix kernel oops on MD
- add patch for avoiding to use percpu-refcount internal from md code
- pass Red Hat CKI test which is done by Veronika Kabatova
V2:
- pass 'gfp' to kzalloc() for fixing block/027 failure reported by kernel test robot
- protect percpu_ref_is_zero() with destroying percpu-refcount by spin lock
Ming Lei (3):
percpu_ref: add percpu_ref_is_initialized for MD
percpu_ref: reduce memory footprint of percpu_ref in fast path
block: move 'q_usage_counter' into front of 'request_queue'
drivers/infiniband/sw/rdmavt/mr.c | 2 +- drivers/md/md.c | 2 +- include/linux/blkdev.h | 3 +- include/linux/percpu-refcount.h | 46 ++++------ lib/percpu-refcount.c | 137 +++++++++++++++++++++++------- 5 files changed, 126 insertions(+), 64 deletions(-)
Signature · 1 line
2.25.2
PATCH v5 1/3 percpu_ref: add percpu_ref_is_initialized for MD
MD code uses perpcu-refcount internal to check if this percpu-refcount variable is initialized, this way is a hack.
Add percpu_ref_is_initialized for MD so that the hack can be avoided.
--- drivers/md/md.c | 2 +- include/linux/percpu-refcount.h | 1 + lib/percpu-refcount.c | 6 ++++++ 3 files changed, 8 insertions(+), 1 deletion(-)
drivers/md/md.c +1 -1
@@ -5631,7 +5631,7 @@ static void no_op(struct percpu_ref *r) {}int mddev_init_writes_pending (struct mddev *mddev) { -if (mddev->writes_pending.percpu_count_ptr) +if (percpu_ref_is_initialized (&mddev->writes_pending))return 0 ;if (percpu_ref_init (&mddev->writes_pending, no_op,PERCPU_REF_ALLOW_REINIT ,GFP_KERNEL ) <0 )
include/linux/percpu-refcount.h +1 -0
@@ -109,6 +109,7 @@ struct percpu_ref {int __must_check percpu_ref_init (struct percpu_ref *ref,percpu_ref_func_t *release,unsigned int flags,gfp_t gfp); +bool percpu_ref_is_initialized (struct percpu_ref *ref);void percpu_ref_exit (struct percpu_ref *ref);void percpu_ref_switch_to_atomic (struct percpu_ref *ref,percpu_ref_func_t *confirm_switch);
lib/percpu-refcount.c +6 -0
@@ -93,6 +93,12 @@ int percpu_ref_init(struct percpu_ref *ref, percpu_ref_func_t *release, }EXPORT_SYMBOL_GPL (percpu_ref_init); +bool percpu_ref_is_initialized (struct percpu_ref *ref) +{ +return percpu_count_ptr (ref) !=NULL ; +} +EXPORT_SYMBOL_GPL (percpu_ref_is_initialized); +/** * percpu_ref_exit - undo percpu_ref_init() * @ref: percpu_ref to exit
Signature · 1 line
2.25.2
Re: [PATCH V5 1/3] percpu_ref: add percpu_ref_is_initialized for MD
Quoted 27 lines · Ming Lei
MD code uses perpcu-refcount internal to check if this percpu-refcount variable is initialized, this way is a hack.
Add
percpu_ref_is_initializedfor MD so that the hack can be avoided.Acked-by Song Liu <[email protected]>Suggested-by Jens Axboe <[email protected]>Tested-by Veronika Kabatova <[email protected]>Cc Song Liu <[email protected]>Cc Sagi Grimberg <[email protected]>Cc Tejun Heo <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Jens Axboe <[email protected]>Cc Bart Van Assche <[email protected]>Signed-off-by Ming Lei <[email protected]>--- drivers/md/md.c | 2 +- include/linux/percpu-refcount.h | 1 + lib/percpu-refcount.c | 6 ++++++ 3 files changed, 8 insertions(+), 1 deletion(-)drivers/md/md.c +0
-0
drivers/md/md.c static void no_op(struct percpu_ref *r) {} · line 5635
Quoted 6 lines
drivers/md/md.c +1
-1@@ -5631,7 +5631,7 @@ static void no_op(struct percpu_ref *r) {}int mddev_init_writes_pending (struct mddev *mddev) { -if (mddev->writes_pending.percpu_count_ptr) +if (percpu_ref_is_initialized (&mddev->writes_pending))return 0 ;
I'd much rather use a flag in the containing mddev structure then exposing this new "API". That mddev code is pretty gross to be honest, we should just initialize the percpu_ref once instea of such a hack.
PATCH v5 2/3 percpu_ref: reduce memory footprint of percpu_ref in fast path
In reply to Ming Lei'struct percpu_ref' is often embedded into one user structure, and the instance is usually referenced in fast path, however actually only 'percpu_count_ptr' is needed in fast path.
So move other fields into one new structure of 'percpu_ref_data', and allocate it dynamically via kzalloc(), then memory footprint of 'percpu_ref' in fast path is reduced a lot and becomes suitable to put into hot cacheline of user structure.
--- drivers/infiniband/sw/rdmavt/mr.c | 2 +- include/linux/percpu-refcount.h | 45 ++++------ lib/percpu-refcount.c | 131 ++++++++++++++++++++++-------- 3 files changed, 116 insertions(+), 62 deletions(-)
drivers/infiniband/sw/rdmavt/mr.c +1 -1
@@ -499,7 +499,7 @@ static int rvt_check_refs(struct rvt_mregion *mr, const char *t)rvt_pr_err (rdi,"%s timeout mr %p pd %p lkey %x refcount %ld\n" , t, mr, mr->pd, mr->lkey, -atomic_long_read (&mr->refcount.count)); +atomic_long_read (&mr->refcount.data->count));rvt_get_mr (mr);return -EBUSY ; }
include/linux/percpu-refcount.h +17 -28
91 lines not on this page · raw message
lib/percpu-refcount.c +98 -33
260 lines not on this page · raw message
Signature · 1 line
2.25.2
PATCH v5 3/3 block: move 'q_usage_counter' into front of 'request_queue'
In reply to Ming LeiThe field of 'q_usage_counter' is always fetched in fast path of every block driver, and move it into front of 'request_queue', so it can be fetched into 1st cacheline of 'request_queue' instance.
--- include/linux/blkdev.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
include/linux/blkdev.h +2 -1
@@ -397,6 +397,8 @@ struct request_queue {struct request *last_merge;struct elevator_queue *elevator; +struct percpu_ref q_usage_counter; +struct blk_queue_stats *stats;struct rq_qos *rq_qos;
@@ -569,7 +571,6 @@ struct request_queue {* percpu_ref_kill() and percpu_ref_reinit(). */ struct mutex mq_freeze_lock; -struct percpu_ref q_usage_counter;struct blk_mq_tag_set *tag_set;struct list_head tag_set_list;
Signature · 1 line
2.25.2