ksmbd: smbd: create MR pool
Subsystems (2)
PATCH 1/2 ksmbd: smbd: create MR pool
Create a memory region pool because rdma_rw_ctx_init() uses memory registration if memory registration yields better performance than using multiple SGE entries.
--- fs/ksmbd/transport_rdma.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-)
fs/ksmbd/transport_rdma.c +22 -1
@@ -428,6 +428,7 @@ static void free_transport(struct smb_direct_transport *t)if (t->qp) {ib_drain_qp (t->qp); +ib_mr_pool_destroy (t->qp, &t->qp->rdma_mrs);ib_destroy_qp (t->qp); }
@@ -1708,7 +1709,9 @@ static int smb_direct_init_params(struct smb_direct_transport *t, cap->max_send_sge =SMB_DIRECT_MAX_SEND_SGES ; cap->max_recv_sge =SMB_DIRECT_MAX_RECV_SGES ; cap->max_inline_data =0 ; - cap->max_rdma_ctxs =0 ; + cap->max_rdma_ctxs = +rdma_rw_mr_factor (device, t->cm_id->port_num, max_pages) * + smb_direct_max_outstanding_rw_ops;return 0 ; }
@@ -1790,6 +1793,7 @@ static int smb_direct_create_qpair(struct smb_direct_transport *t, {int ret;struct ib_qp_init_attr qp_attr; +int pages_per_rw; t->pd =ib_alloc_pd (t->cm_id->device,0 );if (IS_ERR (t->pd)) {
@@ -1837,6 +1841,23 @@ static int smb_direct_create_qpair(struct smb_direct_transport *t, t->qp = t->cm_id->qp; t->cm_id->event_handler = smb_direct_cm_handler; + pages_per_rw =DIV_ROUND_UP (t->max_rdma_rw_size,PAGE_SIZE ) +1 ; +if (pages_per_rw > t->cm_id->device->attrs.max_sgl_rd) { +int pages_per_mr, mr_count; + + pages_per_mr =min_t (int, pages_per_rw, + t->cm_id->device->attrs.max_fast_reg_page_list_len); + mr_count =DIV_ROUND_UP (pages_per_rw, pages_per_mr) * +atomic_read (&t->rw_avail_ops); + ret =ib_mr_pool_init (t->qp, &t->qp->rdma_mrs, mr_count, +IB_MR_TYPE_MEM_REG , pages_per_mr,0 ); +if (ret) { +pr_err ("failed to init mr pool count %d pages %d\n" , + mr_count, pages_per_mr); + goto err; + } + } +return 0 ; err:if (t->qp) {
Signature · 1 line
2.25.1
PATCH 2/2 ksmbd: smbd: change the default maximum read/write, receive size
Due to restriction that cannot handle multiple buffer descriptor structures, decrease the maximum read/write size for Windows clients.
And set the maximum fragmented receive size in consideration of the receive queue size.
--- fs/ksmbd/transport_rdma.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
fs/ksmbd/transport_rdma.c +4 -2
@@ -80,7 +80,7 @@ static int smb_direct_max_fragmented_recv_size = 1024 * 1024;/* The maximum single-message size which can be received */ static int smb_direct_max_receive_size =8192 ; -static int smb_direct_max_read_write_size =1024 *1024 ; +static int smb_direct_max_read_write_size =1048512 ;static int smb_direct_max_outstanding_rw_ops =8 ;
@@ -1908,7 +1908,9 @@ static int smb_direct_prepare(struct ksmbd_transport *t) st->max_send_size =min_t (int, st->max_send_size,le32_to_cpu (req->max_receive_size)); st->max_fragmented_send_size = -le32_to_cpu (req->max_fragmented_size); +le32_to_cpu (req->max_fragmented_size); + st->max_fragmented_recv_size = + (st->recv_credit_max * st->max_recv_size) /2 ; ret =smb_direct_send_negotiate_response (st, ret); out:
Signature · 1 line
2.25.1
2022-01-07 14:45 GMT+09:00, Hyunchul Lee <[email protected]>:
Quoted 5 lines
Create a memory region pool because
rdma_rw_ctx_init()uses memory registration if memory registration yields better performance than using multiple SGE entries.Signed-off-by Hyunchul Lee <[email protected]>
Re: [PATCH 2/2] ksmbd: smbd: change the default maximum read/write, receive size
In reply to Hyunchul Lee2022-01-07 14:45 GMT+09:00, Hyunchul Lee <[email protected]>:
Quoted 8 lines
Due to restriction that cannot handle multiple buffer descriptor structures, decrease the maximum read/write size for Windows clients.
And set the maximum fragmented receive size in consideration of the receive queue size.
Signed-off-by Hyunchul Lee <[email protected]>
Re: [PATCH 2/2] ksmbd: smbd: change the default maximum read/write, receive size
Do you have more detail on what the negotiated readsize/writesize would be for Windows clients with this size? for Linux clients?
It looked like it would still be 4MB at first glance (although in theory some Windows could do 8MB) ... I may have missed something
Quoted 11 lines · Namjae Jeon
2022-01-07 14:45 GMT+09:00, Hyunchul Lee <[email protected]>:
Quoted 8 lines
Due to restriction that cannot handle multiple buffer descriptor structures, decrease the maximum read/write size for Windows clients.
And set the maximum fragmented receive size in consideration of the receive queue size.
Signed-off-by Hyunchul Lee <[email protected]>Acked-by Namjae Jeon <[email protected]>
Signature · 3 lines
Thanks, Steve
Namjae Jeon
maintainerRe: [PATCH 2/2] ksmbd: smbd: change the default maximum read/write, receive size
2022-01-09 15:44 GMT+09:00, Steve French <[email protected]>:
Quoted 2 lines
Do you have more detail on what the negotiated readsize/writesize would be for Windows clients with this size? for Linux clients?
Hyunchul, Please answer.
Quoted 3 lines
It looked like it would still be 4MB at first glance (although in theory some Windows could do 8MB) ... I may have missed something
I understood that multiple-buffer descriptor support was required to set a read/write size of 1MB or more. As I know, Hyunchul is currently working on it.
It seems to be set to the smaller of max read/write size in smb-direct negotiate and max read/write size in smb2 negotiate.
Hyunchul, I have one question more, How did you get 1048512 setting value ?
Quoted 21 lines · Namjae Jeon
Quoted 11 lines
2022-01-07 14:45 GMT+09:00, Hyunchul Lee <[email protected]>:
Quoted 8 lines
Due to restriction that cannot handle multiple buffer descriptor structures, decrease the maximum read/write size for Windows clients.
And set the maximum fragmented receive size in consideration of the receive queue size.
Signed-off-by Hyunchul Lee <[email protected]>Acked-by Namjae Jeon <[email protected]>--
Thanks,Steve
Re: [PATCH 2/2] ksmbd: smbd: change the default maximum read/write, receive size
2022년 1월 9일 (일) 오후 9:56, Namjae Jeon <[email protected]>님이 작성:
Quoted 6 lines
2022-01-09 15:44 GMT+09:00, Steve French <[email protected]>:
Quoted 2 lines
Do you have more detail on what the negotiated readsize/writesize would be for Windows clients with this size? for Linux clients?
Hyunchul, Please answer.
For a Linux client, if connected using smb-direct, the size will be 1048512. But connected with multichannel, the size will be 4MB instead of 1048512. And this causes problems because the read/write size is bigger than 1048512. It looks like a bug. I have to limit the ksmbd's SMB2 maximum read/write size for a test.
For Windows clients, the actual read/write size is less than 1048512.
Quoted 11 lines
Quoted 3 lines
It looked like it would still be 4MB at first glance (although in theory some Windows could do 8MB) ... I may have missed something
I understood that multiple-buffer descriptor support was required to set a read/write size of 1MB or more. As I know, Hyunchul is currently working on it.
It seems to be set to the smaller of max read/write size in smb-direct negotiate and max read/write size in smb2 negotiate.Hyunchul, I have one question more, How did you get 1048512 setting value ?
Quoted 1 line
I remember when the size was 1MB, Windows clients requested read/write with 1048512 and 64.
Quoted 15 lines
Quoted 15 lines · Namjae Jeon
Quoted 11 lines
2022-01-07 14:45 GMT+09:00, Hyunchul Lee <[email protected]>:
Quoted 8 lines
Due to restriction that cannot handle multiple buffer descriptor structures, decrease the maximum read/write size for Windows clients.
And set the maximum fragmented receive size in consideration of the receive queue size.
Signed-off-by Hyunchul Lee <[email protected]>Acked-by Namjae Jeon <[email protected]>
Quoted 5 lines
Quoted 5 lines
--
Thanks,Steve
--
Thanks,
Hyunchul
Re: [PATCH 2/2] ksmbd: smbd: change the default maximum read/write, receive size
I was concerned because I saw significant improvements in large i/o (file copy to or from the server) to Windows and Azure going to 1MB (negotiated max read/write size), then slightly better to 2MB and slightly better still to 4MB (was hard to show gain with 8MB in my earlier tests though)
Quoted 60 lines · Hyunchul Lee
2022년 1월 9일 (일) 오후 9:56, Namjae Jeon <[email protected]>님이 작성:
Quoted 6 lines
2022-01-09 15:44 GMT+09:00, Steve French <[email protected]>:
Quoted 2 lines
Do you have more detail on what the negotiated readsize/writesize would be for Windows clients with this size? for Linux clients?
Hyunchul, Please answer.
For a Linux client, if connected using smb-direct, the size will be 1048512. But connected with multichannel, the size will be 4MB instead of 1048512. And this causes problems because the read/write size is bigger than 1048512. It looks like a bug. I have to limit the ksmbd's SMB2 maximum read/write size for a test.
For Windows clients, the actual read/write size is less than 1048512.
Quoted 11 lines
Quoted 3 lines
It looked like it would still be 4MB at first glance (although in theory some Windows could do 8MB) ... I may have missed something
I understood that multiple-buffer descriptor support was required to set a read/write size of 1MB or more. As I know, Hyunchul is currently working on it.
It seems to be set to the smaller of max read/write size in smb-direct negotiate and max read/write size in smb2 negotiate.Hyunchul, I have one question more, How did you get 1048512 setting value ?
Quoted 1 line
I remember when the size was 1MB, Windows clients requested read/write with 1048512 and 64.
Quoted 15 lines
Quoted 15 lines · Namjae Jeon
Quoted 11 lines
2022-01-07 14:45 GMT+09:00, Hyunchul Lee <[email protected]>:
Quoted 8 lines
Due to restriction that cannot handle multiple buffer descriptor structures, decrease the maximum read/write size for Windows clients.
And set the maximum fragmented receive size in consideration of the receive queue size.
Signed-off-by Hyunchul Lee <[email protected]>Acked-by Namjae Jeon <[email protected]>Quoted 5 lines
Quoted 5 lines
--
Thanks,Steve
--
Thanks,
Hyunchul
Signature · 3 lines
Thanks, Steve
Re: [PATCH 2/2] ksmbd: smbd: change the default maximum read/write, receive size
2022년 1월 10일 (월) 오전 10:43, Steve French <[email protected]>님이 작성:
Quoted 7 lines
I was concerned because I saw significant improvements in large i/o (file copy to or from the server) to Windows and Azure going to 1MB (negotiated max read/write size), then slightly better to 2MB and slightly better still to 4MB (was hard to show gain with 8MB in my earlier tests though)
This patch limits the size only when the SMB Direct protocol is used. If handling multiple buffer descriptors is implemented, we can increase the size.
Quoted 68 lines · Hyunchul Lee
Quoted 52 lines
2022년 1월 9일 (일) 오후 9:56, Namjae Jeon <[email protected]>님이 작성:
Quoted 6 lines
2022-01-09 15:44 GMT+09:00, Steve French <[email protected]>:
Quoted 2 lines
Do you have more detail on what the negotiated readsize/writesize would be for Windows clients with this size? for Linux clients?
Hyunchul, Please answer.
For a Linux client, if connected using smb-direct, the size will be 1048512. But connected with multichannel, the size will be 4MB instead of 1048512. And this causes problems because the read/write size is bigger than 1048512. It looks like a bug. I have to limit the ksmbd's SMB2 maximum read/write size for a test.
For Windows clients, the actual read/write size is less than 1048512.
Quoted 11 lines
Quoted 3 lines
It looked like it would still be 4MB at first glance (although in theory some Windows could do 8MB) ... I may have missed something
I understood that multiple-buffer descriptor support was required to set a read/write size of 1MB or more. As I know, Hyunchul is currently working on it.
It seems to be set to the smaller of max read/write size in smb-direct negotiate and max read/write size in smb2 negotiate.Hyunchul, I have one question more, How did you get 1048512 setting value ?
Quoted 1 line
I remember when the size was 1MB, Windows clients requested read/write with 1048512 and 64.
Quoted 18 lines
Quoted 18 lines · Namjae Jeon
Quoted 11 lines
2022-01-07 14:45 GMT+09:00, Hyunchul Lee <[email protected]>:
Quoted 8 lines
Due to restriction that cannot handle multiple buffer descriptor structures, decrease the maximum read/write size for Windows clients.
And set the maximum fragmented receive size in consideration of the receive queue size.
Signed-off-by Hyunchul Lee <[email protected]>Acked-by Namjae Jeon <[email protected]>--
Thanks,Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Steve
--
Thanks,
Hyunchul--
Thanks,Steve
Signature · 2 lines
Thanks, Hyunchul
Namjae Jeon
maintainerRe: [PATCH 2/2] ksmbd: smbd: change the default maximum read/write, receive size
In reply to Hyunchul Lee2022-01-10 10:37 GMT+09:00, Hyunchul Lee <[email protected]>:
Quoted 15 lines
2022년 1월 9일 (일) 오후 9:56, Namjae Jeon <[email protected]>님이 작성:
Quoted 6 lines
2022-01-09 15:44 GMT+09:00, Steve French <[email protected]>:
Quoted 2 lines
Do you have more detail on what the negotiated readsize/writesize would be for Windows clients with this size? for Linux clients?
Hyunchul, Please answer.
For a Linux client, if connected using smb-direct, the size will be 1048512. But connected with multichannel, the size will be 4MB instead of 1048512. And this causes problems because the read/write size is bigger than 1048512. It looks like a bug. I have to limit the ksmbd's SMB2 maximum read/write size for a test.
Quoted 2 lines
For Windows clients, the actual read/write size is less than 1048512.
In the case of my Chelsio device, Need to set it to about 512K(512*1024 - 64) for it to work.
The 1048512 value seems insufficient to cover all devices. Is there any other way to set the minimum read/write value? Calibrate this minimum value by looking at
the device information? For example variables in ib_dev->attrs.
Quoted 45 lines
Quoted 4 lines
Quoted 4 lines
It looked like it would still be 4MB at first glance (although in theory some Windows could do 8MB) ... I may have missed something
Quoted 9 lines
I understood that multiple-buffer descriptor support was required to set a read/write size of 1MB or more. As I know, Hyunchul is currently working on it.
It seems to be set to the smaller of max read/write size in smb-direct negotiate and max read/write size in smb2 negotiate.Hyunchul, I have one question more, How did you get 1048512 setting value ?
Quoted 1 line
I remember when the size was 1MB, Windows clients requested read/write with 1048512 and 64.
Quoted 16 lines
Quoted 16 lines
Quoted 11 lines · Namjae Jeon
2022-01-07 14:45 GMT+09:00, Hyunchul Lee <[email protected]>:
Quoted 8 lines
Due to restriction that cannot handle multiple buffer descriptor structures, decrease the maximum read/write size for Windows clients.
And set the maximum fragmented receive size in consideration of the receive queue size.
Signed-off-by Hyunchul Lee <[email protected]>Acked-by Namjae Jeon <[email protected]>Quoted 5 lines
Quoted 5 lines
--
Thanks,Steve
--
Thanks,
Hyunchul
Re: [PATCH 2/2] ksmbd: smbd: change the default maximum read/write, receive size
2022년 1월 18일 (화) 오전 8:33, Namjae Jeon <[email protected]>님이 작성:
Quoted 17 lines
2022-01-10 10:37 GMT+09:00, Hyunchul Lee <[email protected]>:
Quoted 15 lines
2022년 1월 9일 (일) 오후 9:56, Namjae Jeon <[email protected]>님이 작성:
Quoted 6 lines
2022-01-09 15:44 GMT+09:00, Steve French <[email protected]>:
Quoted 2 lines
Do you have more detail on what the negotiated readsize/writesize would be for Windows clients with this size? for Linux clients?
Hyunchul, Please answer.
For a Linux client, if connected using smb-direct, the size will be 1048512. But connected with multichannel, the size will be 4MB instead of 1048512. And this causes problems because the read/write size is bigger than 1048512. It looks like a bug. I have to limit the ksmbd's SMB2 maximum read/write size for a test.
Quoted 9 lines
Quoted 2 lines
For Windows clients, the actual read/write size is less than 1048512.
In the case of my Chelsio device, Need to set it to about 512K(512*1024 - 64) for it to work.
The 1048512 value seems insufficient to cover all devices. Is there any other way to set the minimum read/write value? Calibrate this minimum value by looking at
the device information? For example variables inib_dev->attrs.
Let me check it. But I think multiple buffer descriptors caused this problem. because of a client's device limitation, a client seems to send a read/write request with multiple buffer descriptors for sending 1048512 bytes.
To check this assumption, can you tell me the buffer descriptors' content when the default read/write size is 1048512?
Quoted 45 lines
Quoted 45 lines
Quoted 4 lines
Quoted 4 lines
It looked like it would still be 4MB at first glance (although in theory some Windows could do 8MB) ... I may have missed something
Quoted 9 lines
I understood that multiple-buffer descriptor support was required to set a read/write size of 1MB or more. As I know, Hyunchul is currently working on it.
It seems to be set to the smaller of max read/write size in smb-direct negotiate and max read/write size in smb2 negotiate.Hyunchul, I have one question more, How did you get 1048512 setting value ?
Quoted 1 line
I remember when the size was 1MB, Windows clients requested read/write with 1048512 and 64.
Quoted 16 lines
Quoted 16 lines
Quoted 11 lines · Namjae Jeon
2022-01-07 14:45 GMT+09:00, Hyunchul Lee <[email protected]>:
Quoted 8 lines
Due to restriction that cannot handle multiple buffer descriptor structures, decrease the maximum read/write size for Windows clients.
And set the maximum fragmented receive size in consideration of the receive queue size.
Signed-off-by Hyunchul Lee <[email protected]>Acked-by Namjae Jeon <[email protected]>Quoted 5 lines
Quoted 5 lines
--
Thanks,Steve
--
Thanks,
Hyunchul
Signature · 2 lines
Thanks, Hyunchul