Move usages of struct __call_single_data to call_single_data_t
v2 cover Move usages of struct __call_single_data to call_single_data_t
Changes since RFCv1:
- request->csd moved to the middle of the struct, without size impact
- type change happens in a different patch (thanks Jens Axboe!)
- Improved the third patch to also update the .h file.
Leonardo Bras (3):
blk-mq: Move csd inside struct request so it's 32-byte aligned
blk-mq: Change request->csd type to call_single_data_t
smp: Change signatures to use call_single_data_t
include/linux/blk-mq.h | 10 +++++----- include/linux/smp.h | 2 +- kernel/smp.c | 4 ++-- kernel/up.c | 2 +- 4 files changed, 9 insertions(+), 9 deletions(-)
Signature · 1 line
2.40.1
RFC PATCH v2 1/3 blk-mq: Move csd inside struct request so it's 32-byte aligned
Currently, request->csd has type struct __call_single_data.
call_single_data_t is defined in include/linux/smp.h :
/* Use __aligned() to avoid to use 2 cache lines for 1 csd */ typedef struct __call_single_data call_single_data_t
__aligned(sizeof(struct __call_single_data));
As the comment above the typedef suggests, having struct __call_single_data split between 2 cachelines causes the need to fetch / invalidate / bounce 2 cachelines instead of 1 when the cpu receiving the request gets to run the requested function. This is usually bad for performance, due to one extra memory access and 1 extra cacheline usage.
As an example with a 64-bit machine withCONFIG_BLK_RQ_ALLOC_TIME=yCONFIG_BLK_WBT=yCONFIG_BLK_DEV_INTEGRITY=yCONFIG_BLK_INLINE_ENCRYPTION=y
Will output pahole with:
struct request {
[...]
union {
struct __call_single_data csd; /* 240 32 */
u64 fifo_time; /* 240 8 */
}; /* 240 32 */[...]
}
At this config, and any cacheline size between 32 and 256, will cause csd to be split between 2 cachelines: csd->node (16 bytes) in the first cacheline, and csd->func (8 bytes) & csd->info (8 bytes) in the second.
During blk_mq_complete_send_ipi(), csd->func and csd->info are getting changed, and when it calls __smp_call_single_queue() csd->node will get changed.
On the cpu which got the request, csd->func and csd->info get read by __flush_smp_call_function_queue() and csd->node gets changed by csd_unlock(), meaning the two cachelines containing csd will get accessed.
To avoid this, it would be necessary to make sure request->csd is placed somewhere else in the struct, so it is always in a single cacheline, while avoiding the introduction of any hole in the struct. In order to achieve this, move request->csd to after 'struct block_device *part'.
The rationale of this change is that:
- There will be no CONFIG_*-dependent field before csd, so there is no chance of having unexpected holes on given configs.
- On 64-bit machines, csd will be at byte 96, and
- On 32-bit machines, csd will be at byte 64.
This means after this change, request->csd will always be cacheline aligned for cachelines >= 32-bytes (64-bit) and cachelines >= 16-bytes (32-bits), as long as struct request is cacheline aligned.
In above change, the struct request size is not supposed to change in any configuration.
--- include/linux/blk-mq.h | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
include/linux/blk-mq.h +5 -5
@@ -105,6 +105,11 @@ struct request { };struct block_device *part; + +union { +struct __call_single_data csd; +u64 fifo_time; + };#ifdef CONFIG_BLK_RQ_ALLOC_TIME /* Time that the first bio started allocating this request. */ u64 alloc_time_ns;
@@ -189,11 +194,6 @@ struct request { } flush; }; -union { -struct __call_single_data csd; -u64 fifo_time; - }; -/* * completion callback. */
Signature · 1 line
2.40.1
RFC PATCH v2 2/3 blk-mq: Change request->csd type to call_single_data_t
In reply to Leonardo BrasCurrently, request->csd has type struct __call_single_data.
call_single_data_t is defined in include/linux/smp.h :
/* Use __aligned() to avoid to use 2 cache lines for 1 csd */ typedef struct __call_single_data call_single_data_t
__aligned(sizeof(struct __call_single_data));
This is proposed this way so it avoids doing 2 cacheline bounce when running a csd request.
Changing request->csd to the aligned typedef was previously attempted in commit 966a967116e6 ("smp: Avoid using two cache lines for struct call_single_data") but at the time the union that contains csd was positioned near the top of struct request, only below a struct list_head, and this caused the issue of holes summing up 24 extra bytes in struct request.
The struct size was restored back to normal by
commit 4ccafe032005 ("block: unalign call_single_data in struct request") but it caused the csd to be possibly split in 2 cachelines again.
Since request->csd was moved to be always 32-byte aligned inside 'struct request', changing it to the aligned typedef should not cause any increased size or hole in the struct.
Since this is the last use of the unaligned struct outside smp code, having this changed will allow us to change smp_call_function_single_async() to only accept the aligned typedef, and avoid undesired extra cacheline bounce in future code.
--- include/linux/blk-mq.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/blk-mq.h +1 -1
@@ -107,7 +107,7 @@ struct request {struct block_device *part;union { -struct __call_single_data csd; +call_single_data_t csd;u64 fifo_time; };#ifdef CONFIG_BLK_RQ_ALLOC_TIME
Signature · 1 line
2.40.1
RFC PATCH v2 3/3 smp: Change signatures to use call_single_data_t
In reply to Leonardo BrasEvery caller of smp_call_function_single_async() now makes use of call_single_data_t, which is a size-aligned typedef of struct __call_single_data.
Changing smp_call_function_single_async() csd parameter to call_single_data_t makes possible to warn future callers if they are using an unaligned csd, which can cause it to be split between 2 cachelines, which is usually bad for performance.
Also, for the same reason, change generic_exec_single().
--- include/linux/smp.h | 2 +- kernel/smp.c | 4 ++-- kernel/up.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)
include/linux/smp.h +1 -1
@@ -53,7 +53,7 @@ int smp_call_function_single(int cpuid, smp_call_func_t func, void *info,void on_each_cpu_cond_mask (smp_cond_func_t cond_func,smp_call_func_t func,void *info,bool wait,const struct cpumask *mask); -int smp_call_function_single_async (int cpu,struct __call_single_data *csd); +int smp_call_function_single_async (int cpu,call_single_data_t *csd);/* * Cpus stopping functions in panic. All have default weak definitions.
kernel/smp.c +2 -2
@@ -361,7 +361,7 @@ void __smp_call_single_queue(int cpu, struct llist_node *node)* for execution on the given CPU. data must already have * ->func, ->info, and ->flags set. */ -static int generic_exec_single (int cpu,struct __call_single_data *csd) +static int generic_exec_single (int cpu,call_single_data_t *csd) {if (cpu ==smp_processor_id ()) {smp_call_func_t func = csd->func;
@@ -645,7 +645,7 @@ EXPORT_SYMBOL(smp_call_function_single);* * Return: %0 on success or negative errno value on error */ -int smp_call_function_single_async (int cpu,struct __call_single_data *csd) +int smp_call_function_single_async (int cpu,call_single_data_t *csd) {int err =0 ;
kernel/up.c +1 -1
@@ -25,7 +25,7 @@ int smp_call_function_single(int cpu, void (*func) (void *info), void *info, }EXPORT_SYMBOL (smp_call_function_single); -int smp_call_function_single_async (int cpu,struct __call_single_data *csd) +int smp_call_function_single_async (int cpu,call_single_data_t *csd) {unsigned long flags;
Signature · 1 line
2.40.1
Friendly ping
Quoted 20 lines · Leonardo Bras
Changes since RFCv1:
-request->csdmoved to the middle of the struct, without size impact
- type change happens in a different patch (thanks Jens Axboe!)
- Improved the third patch to also update the .h file.Leonardo Bras (3):
blk-mq: Move csd inside struct request so it's 32-byte aligned
blk-mq: Changerequest->csdtype tocall_single_data_t
smp: Change signatures to usecall_single_data_tinclude/linux/blk-mq.h | 10 +++++----- include/linux/smp.h | 2 +- kernel/smp.c | 4 ++-- kernel/up.c | 2 +- 4 files changed, 9 insertions(+), 9 deletions(-)--
2.40.1
Quoted 23 lines · Leonardo Bras Soares Passos
Friendly ping
Quoted 20 lines · Leonardo Bras
Changes since RFCv1:
-request->csdmoved to the middle of the struct, without size impact
- type change happens in a different patch (thanks Jens Axboe!)
- Improved the third patch to also update the .h file.Leonardo Bras (3):
blk-mq: Move csd inside struct request so it's 32-byte aligned
blk-mq: Changerequest->csdtype tocall_single_data_t
smp: Change signatures to usecall_single_data_tinclude/linux/blk-mq.h | 10 +++++----- include/linux/smp.h | 2 +- kernel/smp.c | 4 ++-- kernel/up.c | 2 +- 4 files changed, 9 insertions(+), 9 deletions(-)--
2.40.1
Hello Jens,
I still want your feedback on this series :)
I think I addressed every issue of RFCv1, but if you have any other feedback, please let me know.
Thanks!
Leo
Quoted 27 lines · Leonardo Brás
Quoted 23 lines · Leonardo Bras Soares Passos
Friendly ping
Quoted 20 lines · Leonardo Bras
Changes since RFCv1:
-request->csdmoved to the middle of the struct, without size impact
- type change happens in a different patch (thanks Jens Axboe!)
- Improved the third patch to also update the .h file.Leonardo Bras (3):
blk-mq: Move csd inside struct request so it's 32-byte aligned
blk-mq: Changerequest->csdtype tocall_single_data_t
smp: Change signatures to usecall_single_data_tinclude/linux/blk-mq.h | 10 +++++----- include/linux/smp.h | 2 +- kernel/smp.c | 4 ++-- kernel/up.c | 2 +- 4 files changed, 9 insertions(+), 9 deletions(-)--
2.40.1Hello Jens,
Quoted 7 lines
I still want your feedback on this series :)
I think I addressed every issue of RFCv1, but if you have any other feedback, please let me know.
Thanks!
Leo
Hello Jens Axboe,
Please provide feedback on this series!
Are you ok with those changes?
What's your opinion on them?
Thanks!
Leo
Quoted 38 lines · Leonardo Brás
Quoted 27 lines · Leonardo Brás
Quoted 23 lines · Leonardo Bras Soares Passos
Friendly ping
Quoted 20 lines · Leonardo Bras
Changes since RFCv1:
-request->csdmoved to the middle of the struct, without size impact
- type change happens in a different patch (thanks Jens Axboe!)
- Improved the third patch to also update the .h file.Leonardo Bras (3):
blk-mq: Move csd inside struct request so it's 32-byte aligned
blk-mq: Changerequest->csdtype tocall_single_data_t
smp: Change signatures to usecall_single_data_tinclude/linux/blk-mq.h | 10 +++++----- include/linux/smp.h | 2 +- kernel/smp.c | 4 ++-- kernel/up.c | 2 +- 4 files changed, 9 insertions(+), 9 deletions(-)--
2.40.1Hello Jens,
Quoted 7 lines
I still want your feedback on this series :)
I think I addressed every issue of RFCv1, but if you have any other feedback, please let me know.
Thanks!
LeoHello Jens Axboe,
Quoted 8 lines
Please provide feedback on this series!
Are you ok with those changes?
What's your opinion on them?Thanks!
Leo
Hello,
FYI, there is no csd in struct request anymore in block/for-next branch, which is deleted by this commit:
commit 660e802c76c89e871c29cd3174c07c8d23e39c35
Author: Chengming Zhou <[email protected]> Date: Mon Jul 17 12:00:55 2023 +0800
blk-mq: use percpu csd to remote complete instead of per-rq csd
If request need to be completed remotely, we insert it into percpu llist, and smp_call_function_single_async() if llist is empty previously.
We don't need to use per-rq csd, percpu csd is enough. And the size of struct request is decreased by 24 bytes.
This way is cleaner, and looks correct, given block softirq is guaranteed to be scheduled to consume the list if one new request is added to this percpu list, either smp_call_function_single_async() returns -EBUSY or 0.
Quoted 69 lines · Chengming Zhou
Quoted 38 lines · Leonardo Brás
Quoted 27 lines · Leonardo Brás
Quoted 23 lines · Leonardo Bras Soares Passos
Friendly ping
Quoted 20 lines · Leonardo Bras
Changes since RFCv1:
-request->csdmoved to the middle of the struct, without size impact
- type change happens in a different patch (thanks Jens Axboe!)
- Improved the third patch to also update the .h file.Leonardo Bras (3):
blk-mq: Move csd inside struct request so it's 32-byte aligned
blk-mq: Changerequest->csdtype tocall_single_data_t
smp: Change signatures to usecall_single_data_tinclude/linux/blk-mq.h | 10 +++++----- include/linux/smp.h | 2 +- kernel/smp.c | 4 ++-- kernel/up.c | 2 +- 4 files changed, 9 insertions(+), 9 deletions(-)--
2.40.1Hello Jens,
Quoted 7 lines
I still want your feedback on this series :)
I think I addressed every issue of RFCv1, but if you have any other feedback, please let me know.
Thanks!
LeoHello Jens Axboe,
Quoted 8 lines
Please provide feedback on this series!
Are you ok with those changes?
What's your opinion on them?Thanks!
LeoHello,
FYI, there is no csd in struct request anymore in
block/for-nextbranch, which is deleted by this commit:commit
660e802c76c89e871c29cd3174c07c8d23e39c35Author: Chengming Zhou <[email protected]> Date: Mon Jul 17 12:00:55 2023 +0800blk-mq: use percpu csd to remote complete instead of per-rq csd
If request need to be completed remotely, we insert it into percpu llist, and
smp_call_function_single_async()if llist is empty previously.We don't need to use per-rq csd, percpu csd is enough. And the size of struct request is decreased by 24 bytes.
This way is cleaner, and looks correct, given block softirq is guaranteed to be scheduled to consume the list if one new request is added to this percpu list, either
smp_call_function_single_async()returns-EBUSYor 0.
Quoted 6 lines
Signed-off-by Chengming Zhou <[email protected]>Reviewed-by Ming Lei <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Oh, thanks for the heads-up!
I will send reviewed version of patch 3.
I suppose it can go on top of block/for-next, since the above patch is there. Does that work for you Jens Axboe?
Thanks!
Leo
Jens Axboe
maintainerQuoted 78 lines · Leonardo Br?s
Quoted 69 lines · Chengming Zhou
Quoted 38 lines · Leonardo Br?s
Quoted 27 lines · Leonardo Br?s
Quoted 23 lines · Leonardo Bras Soares Passos
Friendly ping
Quoted 20 lines · On Sat, May 20, 2023 at 2:30?AM Leonardo Bras <[email protected]> wrote:
Changes since RFCv1:
-request->csdmoved to the middle of the struct, without size impact
- type change happens in a different patch (thanks Jens Axboe!)
- Improved the third patch to also update the .h file.Leonardo Bras (3):
blk-mq: Move csd inside struct request so it's 32-byte aligned
blk-mq: Changerequest->csdtype tocall_single_data_t
smp: Change signatures to usecall_single_data_tinclude/linux/blk-mq.h | 10 +++++----- include/linux/smp.h | 2 +- kernel/smp.c | 4 ++-- kernel/up.c | 2 +- 4 files changed, 9 insertions(+), 9 deletions(-)--
2.40.1Hello Jens,
Quoted 7 lines
I still want your feedback on this series :)
I think I addressed every issue of RFCv1, but if you have any other feedback, please let me know.
Thanks!
LeoHello Jens Axboe,
Quoted 8 lines
Please provide feedback on this series!
Are you ok with those changes?
What's your opinion on them?Thanks!
LeoHello,
FYI, there is no csd in struct request anymore in
block/for-nextbranch, which is deleted by this commit:commit
660e802c76c89e871c29cd3174c07c8d23e39c35Author: Chengming Zhou <[email protected]> Date: Mon Jul 17 12:00:55 2023 +0800blk-mq: use percpu csd to remote complete instead of per-rq csd
If request need to be completed remotely, we insert it into percpu llist, and
smp_call_function_single_async()if llist is empty previously.We don't need to use per-rq csd, percpu csd is enough. And the size of struct request is decreased by 24 bytes.
This way is cleaner, and looks correct, given block softirq is guaranteed to be scheduled to consume the list if one new request is added to this percpu list, either
smp_call_function_single_async()returns-EBUSYor 0.Quoted 6 lines
Signed-off-by Chengming Zhou <[email protected]>Reviewed-by Ming Lei <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Quoted 5 lines
Oh, thanks for the heads-up!
I will send reviewed version of patch 3.I suppose it can go on top of
block/for-next, since the above patch is there. Does that work for you Jens Axboe?
Just send it against Linus's tree, it's all upstream now.
Signature · 1 line
Jens Axboe
Quoted 79 lines · Jens Axboe
Quoted 78 lines · Leonardo Br?s
Quoted 75 lines · Chengming Zhou
Quoted 38 lines · Leonardo Br?s
Quoted 27 lines · Leonardo Br?s
Quoted 23 lines · Leonardo Bras Soares Passos
Friendly ping
Quoted 20 lines · On Sat, May 20, 2023 at 2:30?AM Leonardo Bras <[email protected]> wrote:
Changes since RFCv1:
-request->csdmoved to the middle of the struct, without size impact
- type change happens in a different patch (thanks Jens Axboe!)
- Improved the third patch to also update the .h file.Leonardo Bras (3):
blk-mq: Move csd inside struct request so it's 32-byte aligned
blk-mq: Changerequest->csdtype tocall_single_data_t
smp: Change signatures to usecall_single_data_tinclude/linux/blk-mq.h | 10 +++++----- include/linux/smp.h | 2 +- kernel/smp.c | 4 ++-- kernel/up.c | 2 +- 4 files changed, 9 insertions(+), 9 deletions(-)--
2.40.1Hello Jens,
Quoted 7 lines
I still want your feedback on this series :)
I think I addressed every issue of RFCv1, but if you have any other feedback, please let me know.
Thanks!
LeoHello Jens Axboe,
Quoted 8 lines
Please provide feedback on this series!
Are you ok with those changes?
What's your opinion on them?Thanks!
LeoHello,
FYI, there is no csd in struct request anymore in
block/for-nextbranch, which is deleted by this commit:commit
660e802c76c89e871c29cd3174c07c8d23e39c35Author: Chengming Zhou <[email protected]> Date: Mon Jul 17 12:00:55 2023 +0800blk-mq: use percpu csd to remote complete instead of per-rq csd
If request need to be completed remotely, we insert it into percpu llist, and
smp_call_function_single_async()if llist is empty previously.We don't need to use per-rq csd, percpu csd is enough. And the size of struct request is decreased by 24 bytes.
This way is cleaner, and looks correct, given block softirq is guaranteed to be scheduled to consume the list if one new request is added to this percpu list, either
smp_call_function_single_async()returns-EBUSYor 0.Signed-off-by Chengming Zhou <[email protected]>Reviewed-by Ming Lei <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Quoted 8 lines
Quoted 5 lines
Oh, thanks for the heads-up!
I will send reviewed version of patch 3.I suppose it can go on top of
block/for-next, since the above patch is there. Does that work for you Jens Axboe?Just send it against Linus's tree, it's all upstream now.
Sure,
Thanks!
Leo