percpu_ref & block: reduce memory footprint of percpu_ref in fast path
Subsystems (5)
v7 cover percpu_ref & block: reduce memory footprint of percpu_ref in fast path
Hi,
The 1st patch removes memory footprint of percpu_ref in fast path from 7 words to 2 words, since it is often used in fast path and embedded in user struct.
The 2nd patch moves .q_usage_counter to 1st cacheline of 'request_queue'.
Simple test on null_blk shows ~2% IOPS boost on one 16cores(two threads per core) machine, dual socket/numa.
V7:
- add comments about reason for struct split
V6:
- drop the 1st patch which adds percpu_ref_is_initialized() for MD only, since Christoph doesn't like it
V5:
- fix memory leak on ref->data, only percpu_ref_exit() of patch 2 is modified.
V4:
- rename percpu_ref_inited as percpu_ref_is_initialized
V3:
- fix kernel oops on MD
- add patch for avoiding to use percpu-refcount internal from md code
- pass Red Hat CKI test which is done by Veronika Kabatova
V2:
- pass 'gfp' to kzalloc() for fixing block/027 failure reported by kernel test robot
- protect percpu_ref_is_zero() with destroying percpu-refcount by spin lock
Ming Lei (2):
percpu_ref: reduce memory footprint of percpu_ref in fast path
block: move 'q_usage_counter' into front of 'request_queue'
drivers/infiniband/sw/rdmavt/mr.c | 2 +- include/linux/blkdev.h | 3 +- include/linux/percpu-refcount.h | 52 ++++++------ lib/percpu-refcount.c | 131 ++++++++++++++++++++++-------- 4 files changed, 125 insertions(+), 63 deletions(-)
Signature · 1 line
2.25.2
PATCH v7 1/2 percpu_ref: reduce memory footprint of percpu_ref in fast path
'struct percpu_ref' is often embedded into one user structure, and the instance is usually referenced in fast path, however actually only 'percpu_count_ptr' is needed in fast path.
So move other fields into one new structure of 'percpu_ref_data', and allocate it dynamically via kzalloc(), then memory footprint of 'percpu_ref' in fast path is reduced a lot and becomes suitable to put into hot cacheline of user structure.
--- drivers/infiniband/sw/rdmavt/mr.c | 2 +- include/linux/percpu-refcount.h | 52 ++++++------ lib/percpu-refcount.c | 131 ++++++++++++++++++++++-------- 3 files changed, 123 insertions(+), 62 deletions(-)
drivers/infiniband/sw/rdmavt/mr.c +1 -1
@@ -499,7 +499,7 @@ static int rvt_check_refs(struct rvt_mregion *mr, const char *t)rvt_pr_err (rdi,"%s timeout mr %p pd %p lkey %x refcount %ld\n" , t, mr, mr->pd, mr->lkey, -atomic_long_read (&mr->refcount.count)); +atomic_long_read (&mr->refcount.data->count));rvt_get_mr (mr);return -EBUSY ; }
include/linux/percpu-refcount.h +24 -28
98 lines not on this page · raw message
lib/percpu-refcount.c +98 -33
258 lines not on this page · raw message
Signature · 1 line
2.25.2
Re: [PATCH V7 1/2] percpu_ref: reduce memory footprint of percpu_ref in fast path
Quoted 9 lines · Ming Lei
'struct
percpu_ref' is often embedded into one user structure, and the instance is usually referenced in fast path, however actually only 'percpu_count_ptr' is needed in fast path.So move other fields into one new structure of '
percpu_ref_data', and allocate it dynamically viakzalloc(), then memory footprint of 'percpu_ref' in fast path is reduced a lot and becomes suitable to put into hot cacheline of user structure.
Quoted 8 lines
Tested-by Veronika Kabatova <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Cc Sagi Grimberg <[email protected]>Cc Tejun Heo <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Jens Axboe <[email protected]>Cc Bart Van Assche <[email protected]>Signed-off-by Ming Lei <[email protected]>
Thanks.
Signature · 1 line
tejun
PATCH v7 2/2 block: move 'q_usage_counter' into front of 'request_queue'
In reply to Ming LeiThe field of 'q_usage_counter' is always fetched in fast path of every block driver, and move it into front of 'request_queue', so it can be fetched into 1st cacheline of 'request_queue' instance.
--- include/linux/blkdev.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
include/linux/blkdev.h +2 -1
@@ -397,6 +397,8 @@ struct request_queue {struct request *last_merge;struct elevator_queue *elevator; +struct percpu_ref q_usage_counter; +struct blk_queue_stats *stats;struct rq_qos *rq_qos;
@@ -569,7 +571,6 @@ struct request_queue {* percpu_ref_kill() and percpu_ref_reinit(). */ struct mutex mq_freeze_lock; -struct percpu_ref q_usage_counter;struct blk_mq_tag_set *tag_set;struct list_head tag_set_list;
Signature · 1 line
2.25.2
Quoted 6 lines · Ming Lei
Hi,
The 1st patch removes memory footprint of
percpu_refin fast path from 7 words to 2 words, since it is often used in fast path and embedded in user struct.
Quoted 8 lines
The 2nd patch moves .q_usage_counter to 1st cacheline of '
request_queue'.Simple test on
null_blkshows ~2% IOPS boost on one 16cores(two threads per core) machine, dual socket/numa.V7:
- add comments about reason for struct split
Hello Jens
Can you consider to merge the patchset in block tree if you are fine?
Thanks,
Ming
Quoted 11 lines · Ming Lei
Hi,
The 1st patch removes memory footprint of
percpu_refin fast path from 7 words to 2 words, since it is often used in fast path and embedded in user struct.The 2nd patch moves .q_usage_counter to 1st cacheline of '
request_queue'.Simple test on
null_blkshows ~2% IOPS boost on one 16cores(two threads per core) machine, dual socket/numa.
Applied, thanks.
Signature · 1 line
Jens Axboe