percpu_ref & block: reduce memory footprint of percpu_ref in fast path
Subsystems (5)
v2 cover percpu_ref & block: reduce memory footprint of percpu_ref in fast path
Hi,
The 1st patch removes memory footprint of percpu_ref in fast path from 7 words to 2 words, since it is often used in fast path and embedded in user struct.
The 2nd patch moves .q_usage_counter to 1st cacheline of 'request_queue'.
Simple test on null_blk shows ~2% IOPS boost on one 16cores(two threads per core) machine, dual socket/numa.
V2:
- pass 'gfp' to kzalloc() for fixing block/027 failure reported by kernel test robot
- protect percpu_ref_is_zero() with destroying percpu-refcount by spin lock
Ming Lei (2):
percpu_ref: reduce memory footprint of percpu_ref in fast path
block: move 'q_usage_counter' into front of 'request_queue'
drivers/infiniband/sw/rdmavt/mr.c | 2 +- include/linux/blkdev.h | 3 +- include/linux/percpu-refcount.h | 45 ++++------- lib/percpu-refcount.c | 128 ++++++++++++++++++++++-------- 4 files changed, 115 insertions(+), 63 deletions(-)
Signature · 1 line
2.25.2
PATCH v2 1/2 percpu_ref: reduce memory footprint of percpu_ref in fast path
'struct percpu_ref' is often embedded into one user structure, and the instance is usually referenced in fast path, however actually only 'percpu_count_ptr' is needed in fast path.
So move other fields into one new structure of 'percpu_ref_data', and allocate it dynamically via kzalloc(), then memory footprint of 'percpu_ref' in fast path is reduced a lot and becomes suitable to put into hot cacheline of user structure.
--- drivers/infiniband/sw/rdmavt/mr.c | 2 +- include/linux/percpu-refcount.h | 45 ++++------- lib/percpu-refcount.c | 128 ++++++++++++++++++++++-------- 3 files changed, 113 insertions(+), 62 deletions(-)
drivers/infiniband/sw/rdmavt/mr.c +1 -1
@@ -499,7 +499,7 @@ static int rvt_check_refs(struct rvt_mregion *mr, const char *t)rvt_pr_err (rdi,"%s timeout mr %p pd %p lkey %x refcount %ld\n" , t, mr, mr->pd, mr->lkey, -atomic_long_read (&mr->refcount.count)); +atomic_long_read (&mr->refcount.data->count));rvt_get_mr (mr);return -EBUSY ; }
include/linux/percpu-refcount.h +17 -28
91 lines not on this page · raw message
lib/percpu-refcount.c +95 -33
255 lines not on this page · raw message
Signature · 1 line
2.25.2
Re: [PATCH V2 1/2] percpu_ref: reduce memory footprint of percpu_ref in fast path
Quoted 8 lines · Ming Lei
'struct
percpu_ref' is often embedded into one user structure, and the instance is usually referenced in fast path, however actually only 'percpu_count_ptr' is needed in fast path.So move other fields into one new structure of '
percpu_ref_data', and allocate it dynamically viakzalloc(), then memory footprint of 'percpu_ref' in fast path is reduced a lot and becomes suitable to put into hot cacheline of user structure.
This looks good:
PATCH v2 2/2 block: move 'q_usage_counter' into front of 'request_queue'
In reply to Ming LeiThe field of 'q_usage_counter' is always fetched in fast path of every block driver, and move it into front of 'request_queue', so it can be fetched into 1st cacheline of 'request_queue' instance.
--- include/linux/blkdev.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
include/linux/blkdev.h +2 -1
@@ -397,6 +397,8 @@ struct request_queue {struct request *last_merge;struct elevator_queue *elevator; +struct percpu_ref q_usage_counter; +struct blk_queue_stats *stats;struct rq_qos *rq_qos;
@@ -567,7 +569,6 @@ struct request_queue {* percpu_ref_kill() and percpu_ref_reinit(). */ struct mutex mq_freeze_lock; -struct percpu_ref q_usage_counter;struct blk_mq_tag_set *tag_set;struct list_head tag_set_list;
Signature · 1 line
2.25.2
Re: [PATCH V2 2/2] block: move 'q_usage_counter' into front of 'request_queue'
Quoted 3 lines · Ming Lei
The field of '
q_usage_counter' is always fetched in fast path of every block driver, and move it into front of 'request_queue', so it can be fetched into 1st cacheline of 'request_queue' instance.
Looks good,
Quoted 18 lines · Ming Lei
Hi,
The 1st patch removes memory footprint of
percpu_refin fast path from 7 words to 2 words, since it is often used in fast path and embedded in user struct.The 2nd patch moves .q_usage_counter to 1st cacheline of '
request_queue'.Simple test on
null_blkshows ~2% IOPS boost on one 16cores(two threads per core) machine, dual socket/numa.V2:
- pass 'gfp' tokzalloc()for fixing block/027 failure reported by kernel test robot
- protectpercpu_ref_is_zero()with destroying percpu-refcount by spin lock
Quoted 3 lines
Ming Lei (2):
percpu_ref: reduce memory footprint ofpercpu_refin fast path
block: move 'q_usage_counter' into front of 'request_queue'
Applied, thanks.
Signature · 1 line
Jens Axboe