Skip to contentSkip to conversation

percpu_ref & block: reduce memory footprint of percpu_ref in fast path

PATCH Applied Ming Lei 3 participants 6 messages Oct 1, 2020 – Oct 6, 2020

Subsystems (5)
lore
Share & tools
lore

Ming Lei

author
#0 lore raw

v7 cover percpu_ref & block: reduce memory footprint of percpu_ref in fast path

Hi,

The 1st patch removes memory footprint of percpu_ref in fast path from 7 words to 2 words, since it is often used in fast path and embedded in user struct.

The 2nd patch moves .q_usage_counter to 1st cacheline of 'request_queue'.

Simple test on null_blk shows ~2% IOPS boost on one 16cores(two threads per core) machine, dual socket/numa.

V7:
- add comments about reason for struct split

V6:
- drop the 1st patch which adds percpu_ref_is_initialized() for MD only, since Christoph doesn't like it

V5:
- fix memory leak on ref->data, only percpu_ref_exit() of patch 2 is modified.

V4:
- rename percpu_ref_inited as percpu_ref_is_initialized

V3:
- fix kernel oops on MD
- add patch for avoiding to use percpu-refcount internal from md code
- pass Red Hat CKI test which is done by Veronika Kabatova

V2:
- pass 'gfp' to kzalloc() for fixing block/027 failure reported by kernel test robot
- protect percpu_ref_is_zero() with destroying percpu-refcount by spin lock

Ming Lei (2):
percpu_ref: reduce memory footprint of percpu_ref in fast path
block: move 'q_usage_counter' into front of 'request_queue'

 drivers/infiniband/sw/rdmavt/mr.c |   2 +-
 include/linux/blkdev.h            |   3 +-
 include/linux/percpu-refcount.h   |  52 ++++++------
 lib/percpu-refcount.c             | 131 ++++++++++++++++++++++--------
 4 files changed, 125 insertions(+), 63 deletions(-)
Cc Veronika Kabatova <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Jens Axboe <[email protected]>
Cc Bart Van Assche <[email protected]>
Signature · 1 line
2.25.2
author
#1 lore raw

PATCH v7 1/2 percpu_ref: reduce memory footprint of percpu_ref in fast path

'struct percpu_ref' is often embedded into one user structure, and the instance is usually referenced in fast path, however actually only 'percpu_count_ptr' is needed in fast path.

So move other fields into one new structure of 'percpu_ref_data', and allocate it dynamically via kzalloc(), then memory footprint of 'percpu_ref' in fast path is reduced a lot and becomes suitable to put into hot cacheline of user structure.

Cc Sagi Grimberg <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Jens Axboe <[email protected]>
Cc Bart Van Assche <[email protected]>
---
 drivers/infiniband/sw/rdmavt/mr.c |   2 +-
 include/linux/percpu-refcount.h   |  52 ++++++------
 lib/percpu-refcount.c             | 131 ++++++++++++++++++++++--------
 3 files changed, 123 insertions(+), 62 deletions(-)
drivers/infiniband/sw/rdmavt/mr.c +1 -1
index 2f7c25fea44a..8490fdb9c91e 100644
@@ -499,7 +499,7 @@ static int rvt_check_refs(struct rvt_mregion *mr, const char *t)
 		rvt_pr_err(rdi,
 			   "%s timeout mr %p pd %p lkey %x refcount %ld\n",
 			   t, mr, mr->pd, mr->lkey,
-			   atomic_long_read(&mr->refcount.count));
+			   atomic_long_read(&mr->refcount.data->count));
 		rvt_get_mr(mr);
 		return -EBUSY;
 	}
include/linux/percpu-refcount.h +24 -28

98 lines not on this page · raw message

lib/percpu-refcount.c +98 -33

258 lines not on this page · raw message

Signature · 1 line
2.25.2
maintainer Acked-by on v7 1/2
#2 lore raw

Re: [PATCH V7 1/2] percpu_ref: reduce memory footprint of percpu_ref in fast path

Quoted 9 lines · Ming Lei

'struct percpu_ref' is often embedded into one user structure, and the instance is usually referenced in fast path, however actually only 'percpu_count_ptr' is needed in fast path.

So move other fields into one new structure of 'percpu_ref_data', and allocate it dynamically via kzalloc(), then memory footprint of 'percpu_ref' in fast path is reduced a lot and becomes suitable to put into hot cacheline of user structure.

Quoted 8 lines
Cc Sagi Grimberg <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Jens Axboe <[email protected]>
Cc Bart Van Assche <[email protected]>

Thanks.

Signature · 1 line
tejun
author
#3 lore raw

PATCH v7 2/2 block: move 'q_usage_counter' into front of 'request_queue'

In reply to Ming Lei

The field of 'q_usage_counter' is always fetched in fast path of every block driver, and move it into front of 'request_queue', so it can be fetched into 1st cacheline of 'request_queue' instance.

Cc Sagi Grimberg <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Jens Axboe <[email protected]>
Cc Bart Van Assche <[email protected]>
---
 include/linux/blkdev.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
include/linux/blkdev.h +2 -1
index d5a3e1a4c2f7..67935b3bef6c 100644
@@ -397,6 +397,8 @@ struct request_queue {
 	struct request		*last_merge;
 	struct elevator_queue	*elevator;
 
+	struct percpu_ref	q_usage_counter;
+
 	struct blk_queue_stats	*stats;
 	struct rq_qos		*rq_qos;
 
@@ -569,7 +571,6 @@ struct request_queue {
 	 * percpu_ref_kill() and percpu_ref_reinit().
 	 */
 	struct mutex		mq_freeze_lock;
-	struct percpu_ref	q_usage_counter;
 
 	struct blk_mq_tag_set	*tag_set;
 	struct list_head	tag_set_list;
Signature · 1 line
2.25.2
author
#4 lore raw
In reply to Ming Lei
Quoted 6 lines · Ming Lei

Hi,

The 1st patch removes memory footprint of percpu_ref in fast path from 7 words to 2 words, since it is often used in fast path and embedded in user struct.

Quoted 8 lines

The 2nd patch moves .q_usage_counter to 1st cacheline of 'request_queue'.

Simple test on null_blk shows ~2% IOPS boost on one 16cores(two threads per core) machine, dual socket/numa.

V7:
- add comments about reason for struct split

Hello Jens

Can you consider to merge the patchset in block tree if you are fine?

Thanks,
Ming

maintainer Applied
#5 lore raw
In reply to Ming Lei
Quoted 11 lines · Ming Lei

Hi,

The 1st patch removes memory footprint of percpu_ref in fast path from 7 words to 2 words, since it is often used in fast path and embedded in user struct.

The 2nd patch moves .q_usage_counter to 1st cacheline of 'request_queue'.

Simple test on null_blk shows ~2% IOPS boost on one 16cores(two threads per core) machine, dual socket/numa.

Applied, thanks.

Signature · 1 line
Jens Axboe