Skip to contentSkip to conversation

percpu_ref & block: reduce memory footprint of percpu_ref in fast path

PATCH Applied Ming Lei 3 participants 6 messages Sep 2, 2020

Subsystems (5)

v2 of 7 Applied Latest is v7 →

lore
Share & tools
lore

Ming Lei

author
#0 lore raw

v2 cover percpu_ref & block: reduce memory footprint of percpu_ref in fast path

Hi,

The 1st patch removes memory footprint of percpu_ref in fast path from 7 words to 2 words, since it is often used in fast path and embedded in user struct.

The 2nd patch moves .q_usage_counter to 1st cacheline of 'request_queue'.

Simple test on null_blk shows ~2% IOPS boost on one 16cores(two threads per core) machine, dual socket/numa.

V2:
- pass 'gfp' to kzalloc() for fixing block/027 failure reported by kernel test robot
- protect percpu_ref_is_zero() with destroying percpu-refcount by spin lock

Ming Lei (2):
percpu_ref: reduce memory footprint of percpu_ref in fast path
block: move 'q_usage_counter' into front of 'request_queue'

 drivers/infiniband/sw/rdmavt/mr.c |   2 +-
 include/linux/blkdev.h            |   3 +-
 include/linux/percpu-refcount.h   |  45 ++++-------
 lib/percpu-refcount.c             | 128 ++++++++++++++++++++++--------
 4 files changed, 115 insertions(+), 63 deletions(-)
Cc Sagi Grimberg <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Jens Axboe <[email protected]>
Cc Bart Van Assche <[email protected]>
Signature · 1 line
2.25.2
author
#1 lore raw

PATCH v2 1/2 percpu_ref: reduce memory footprint of percpu_ref in fast path

'struct percpu_ref' is often embedded into one user structure, and the instance is usually referenced in fast path, however actually only 'percpu_count_ptr' is needed in fast path.

So move other fields into one new structure of 'percpu_ref_data', and allocate it dynamically via kzalloc(), then memory footprint of 'percpu_ref' in fast path is reduced a lot and becomes suitable to put into hot cacheline of user structure.

Cc Sagi Grimberg <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Jens Axboe <[email protected]>
Cc Bart Van Assche <[email protected]>
---
 drivers/infiniband/sw/rdmavt/mr.c |   2 +-
 include/linux/percpu-refcount.h   |  45 ++++-------
 lib/percpu-refcount.c             | 128 ++++++++++++++++++++++--------
 3 files changed, 113 insertions(+), 62 deletions(-)
drivers/infiniband/sw/rdmavt/mr.c +1 -1
index 2f7c25fea44a..8490fdb9c91e 100644
@@ -499,7 +499,7 @@ static int rvt_check_refs(struct rvt_mregion *mr, const char *t)
 		rvt_pr_err(rdi,
 			   "%s timeout mr %p pd %p lkey %x refcount %ld\n",
 			   t, mr, mr->pd, mr->lkey,
-			   atomic_long_read(&mr->refcount.count));
+			   atomic_long_read(&mr->refcount.data->count));
 		rvt_get_mr(mr);
 		return -EBUSY;
 	}
include/linux/percpu-refcount.h +17 -28

91 lines not on this page · raw message

lib/percpu-refcount.c +95 -33

255 lines not on this page · raw message

Signature · 1 line
2.25.2
Reviewed-by on v2 1/2
#2 lore raw

Re: [PATCH V2 1/2] percpu_ref: reduce memory footprint of percpu_ref in fast path

Quoted 8 lines · Ming Lei

'struct percpu_ref' is often embedded into one user structure, and the instance is usually referenced in fast path, however actually only 'percpu_count_ptr' is needed in fast path.

So move other fields into one new structure of 'percpu_ref_data', and allocate it dynamically via kzalloc(), then memory footprint of 'percpu_ref' in fast path is reduced a lot and becomes suitable to put into hot cacheline of user structure.

This looks good:

author
#3 lore raw

PATCH v2 2/2 block: move 'q_usage_counter' into front of 'request_queue'

In reply to Ming Lei

The field of 'q_usage_counter' is always fetched in fast path of every block driver, and move it into front of 'request_queue', so it can be fetched into 1st cacheline of 'request_queue' instance.

Cc Sagi Grimberg <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Jens Axboe <[email protected]>
Cc Bart Van Assche <[email protected]>
---
 include/linux/blkdev.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
include/linux/blkdev.h +2 -1
index d0d61bc81615..7575fa0aae6e 100644
@@ -397,6 +397,8 @@ struct request_queue {
 	struct request		*last_merge;
 	struct elevator_queue	*elevator;
 
+	struct percpu_ref	q_usage_counter;
+
 	struct blk_queue_stats	*stats;
 	struct rq_qos		*rq_qos;
 
@@ -567,7 +569,6 @@ struct request_queue {
 	 * percpu_ref_kill() and percpu_ref_reinit().
 	 */
 	struct mutex		mq_freeze_lock;
-	struct percpu_ref	q_usage_counter;
 
 	struct blk_mq_tag_set	*tag_set;
 	struct list_head	tag_set_list;
Signature · 1 line
2.25.2
Reviewed-by on v2 2/2
#4 lore raw

Re: [PATCH V2 2/2] block: move 'q_usage_counter' into front of 'request_queue'

Quoted 3 lines · Ming Lei

The field of 'q_usage_counter' is always fetched in fast path of every block driver, and move it into front of 'request_queue', so it can be fetched into 1st cacheline of 'request_queue' instance.

Looks good,

maintainer Applied
#5 lore raw
In reply to Ming Lei
Quoted 18 lines · Ming Lei

Hi,

The 1st patch removes memory footprint of percpu_ref in fast path from 7 words to 2 words, since it is often used in fast path and embedded in user struct.

The 2nd patch moves .q_usage_counter to 1st cacheline of 'request_queue'.

Simple test on null_blk shows ~2% IOPS boost on one 16cores(two threads per core) machine, dual socket/numa.

V2:
- pass 'gfp' to kzalloc() for fixing block/027 failure reported by kernel test robot
- protect percpu_ref_is_zero() with destroying percpu-refcount by spin lock

Quoted 3 lines

Ming Lei (2):
percpu_ref: reduce memory footprint of percpu_ref in fast path
block: move 'q_usage_counter' into front of 'request_queue'

Applied, thanks.

Signature · 1 line
Jens Axboe