Skip to contentSkip to conversation

percpu_ref & block: reduce memory footprint of percpu_ref in fast path

PATCH Superseded Ming Lei 2 participants 5 messages Sep 27, 2020 – Sep 28, 2020

Subsystems (6)

v5 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Ming Lei

author
#0 lore raw

v5 cover percpu_ref & block: reduce memory footprint of percpu_ref in fast path

Hi,

The 1st patch removes memory footprint of percpu_ref in fast path from 7 words to 2 words, since it is often used in fast path and embedded in user struct.

The 2nd patch moves .q_usage_counter to 1st cacheline of 'request_queue'.

Simple test on null_blk shows ~2% IOPS boost on one 16cores(two threads per core) machine, dual socket/numa.

V5:
- fix memory leak on ref->data, only percpu_ref_exit() of patch 2 is modified.

V4:
- rename percpu_ref_inited as percpu_ref_is_initialized

V3:
- fix kernel oops on MD
- add patch for avoiding to use percpu-refcount internal from md code
- pass Red Hat CKI test which is done by Veronika Kabatova

V2:
- pass 'gfp' to kzalloc() for fixing block/027 failure reported by kernel test robot
- protect percpu_ref_is_zero() with destroying percpu-refcount by spin lock

Ming Lei (3):
percpu_ref: add percpu_ref_is_initialized for MD
percpu_ref: reduce memory footprint of percpu_ref in fast path
block: move 'q_usage_counter' into front of 'request_queue'

 drivers/infiniband/sw/rdmavt/mr.c |   2 +-
 drivers/md/md.c                   |   2 +-
 include/linux/blkdev.h            |   3 +-
 include/linux/percpu-refcount.h   |  46 ++++------
 lib/percpu-refcount.c             | 137 +++++++++++++++++++++++-------
 5 files changed, 126 insertions(+), 64 deletions(-)
Cc Veronika Kabatova <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Jens Axboe <[email protected]>
Cc Bart Van Assche <[email protected]>
Signature · 1 line
2.25.2
author
#1 lore raw

PATCH v5 1/3 percpu_ref: add percpu_ref_is_initialized for MD

MD code uses perpcu-refcount internal to check if this percpu-refcount variable is initialized, this way is a hack.

Add percpu_ref_is_initialized for MD so that the hack can be avoided.

---
 drivers/md/md.c                 | 2 +-
 include/linux/percpu-refcount.h | 1 +
 lib/percpu-refcount.c           | 6 ++++++
 3 files changed, 8 insertions(+), 1 deletion(-)
drivers/md/md.c +1 -1
index de8419b7ae98..241ff618d84e 100644
@@ -5631,7 +5631,7 @@ static void no_op(struct percpu_ref *r) {}
 
 int mddev_init_writes_pending(struct mddev *mddev)
 {
-	if (mddev->writes_pending.percpu_count_ptr)
+	if (percpu_ref_is_initialized(&mddev->writes_pending))
 		return 0;
 	if (percpu_ref_init(&mddev->writes_pending, no_op,
 			    PERCPU_REF_ALLOW_REINIT, GFP_KERNEL) < 0)
include/linux/percpu-refcount.h +1 -0
index 87d8a38bdea1..2dfdf870f2d0 100644
@@ -109,6 +109,7 @@ struct percpu_ref {
 int __must_check percpu_ref_init(struct percpu_ref *ref,
 				 percpu_ref_func_t *release, unsigned int flags,
 				 gfp_t gfp);
+bool percpu_ref_is_initialized(struct percpu_ref *ref);
 void percpu_ref_exit(struct percpu_ref *ref);
 void percpu_ref_switch_to_atomic(struct percpu_ref *ref,
 				 percpu_ref_func_t *confirm_switch);
lib/percpu-refcount.c +6 -0
index 0ba686b8fe57..db2ec682e0f7 100644
@@ -93,6 +93,12 @@ int percpu_ref_init(struct percpu_ref *ref, percpu_ref_func_t *release,
 }
 EXPORT_SYMBOL_GPL(percpu_ref_init);
 
+bool percpu_ref_is_initialized(struct percpu_ref *ref)
+{
+	return percpu_count_ptr(ref) != NULL;
+}
+EXPORT_SYMBOL_GPL(percpu_ref_is_initialized);
+
 /**
  * percpu_ref_exit - undo percpu_ref_init()
  * @ref: percpu_ref to exit
Signature · 1 line
2.25.2
#2 lore raw

Re: [PATCH V5 1/3] percpu_ref: add percpu_ref_is_initialized for MD

Quoted 27 lines · Ming Lei

MD code uses perpcu-refcount internal to check if this percpu-refcount variable is initialized, this way is a hack.

Add percpu_ref_is_initialized for MD so that the hack can be avoided.

---
 drivers/md/md.c                 | 2 +-
 include/linux/percpu-refcount.h | 1 +
 lib/percpu-refcount.c           | 6 ++++++
 3 files changed, 8 insertions(+), 1 deletion(-)
drivers/md/md.c +0 -0
index de8419b7ae98..241ff618d84e 100644

drivers/md/md.c static void no_op(struct percpu_ref *r) {} · line 5635

Quoted 6 lines
drivers/md/md.c +1 -1
@@ -5631,7 +5631,7 @@ static void no_op(struct percpu_ref *r) {}
 
 int mddev_init_writes_pending(struct mddev *mddev)
 {
-	if (mddev->writes_pending.percpu_count_ptr)
+	if (percpu_ref_is_initialized(&mddev->writes_pending))
 		return 0;

I'd much rather use a flag in the containing mddev structure then exposing this new "API". That mddev code is pretty gross to be honest, we should just initialize the percpu_ref once instea of such a hack.

author
#3 lore raw

PATCH v5 2/3 percpu_ref: reduce memory footprint of percpu_ref in fast path

In reply to Ming Lei

'struct percpu_ref' is often embedded into one user structure, and the instance is usually referenced in fast path, however actually only 'percpu_count_ptr' is needed in fast path.

So move other fields into one new structure of 'percpu_ref_data', and allocate it dynamically via kzalloc(), then memory footprint of 'percpu_ref' in fast path is reduced a lot and becomes suitable to put into hot cacheline of user structure.

Cc Sagi Grimberg <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Jens Axboe <[email protected]>
Cc Bart Van Assche <[email protected]>
---
 drivers/infiniband/sw/rdmavt/mr.c |   2 +-
 include/linux/percpu-refcount.h   |  45 ++++------
 lib/percpu-refcount.c             | 131 ++++++++++++++++++++++--------
 3 files changed, 116 insertions(+), 62 deletions(-)
drivers/infiniband/sw/rdmavt/mr.c +1 -1
index 2f7c25fea44a..8490fdb9c91e 100644
@@ -499,7 +499,7 @@ static int rvt_check_refs(struct rvt_mregion *mr, const char *t)
 		rvt_pr_err(rdi,
 			   "%s timeout mr %p pd %p lkey %x refcount %ld\n",
 			   t, mr, mr->pd, mr->lkey,
-			   atomic_long_read(&mr->refcount.count));
+			   atomic_long_read(&mr->refcount.data->count));
 		rvt_get_mr(mr);
 		return -EBUSY;
 	}
include/linux/percpu-refcount.h +17 -28

91 lines not on this page · raw message

lib/percpu-refcount.c +98 -33

260 lines not on this page · raw message

Signature · 1 line
2.25.2
author
#4 lore raw

PATCH v5 3/3 block: move 'q_usage_counter' into front of 'request_queue'

In reply to Ming Lei

The field of 'q_usage_counter' is always fetched in fast path of every block driver, and move it into front of 'request_queue', so it can be fetched into 1st cacheline of 'request_queue' instance.

Cc Sagi Grimberg <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Jens Axboe <[email protected]>
Cc Bart Van Assche <[email protected]>
---
 include/linux/blkdev.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
include/linux/blkdev.h +2 -1
index d5a3e1a4c2f7..67935b3bef6c 100644
@@ -397,6 +397,8 @@ struct request_queue {
 	struct request		*last_merge;
 	struct elevator_queue	*elevator;
 
+	struct percpu_ref	q_usage_counter;
+
 	struct blk_queue_stats	*stats;
 	struct rq_qos		*rq_qos;
 
@@ -569,7 +571,6 @@ struct request_queue {
 	 * percpu_ref_kill() and percpu_ref_reinit().
 	 */
 	struct mutex		mq_freeze_lock;
-	struct percpu_ref	q_usage_counter;
 
 	struct blk_mq_tag_set	*tag_set;
 	struct list_head	tag_set_list;
Signature · 1 line
2.25.2