ksmbd: reduce smb direct max read/write size
Subsystems (2)
Namjae Jeon
maintainerPATCH 1/3 ksmbd: reduce smb direct max read/write size
To support RDMA in chelsio NICs, Reduce smb direct read/write size to about 512KB. With this change, we have checked that a single buffer descriptor was sent from Windows client, and intel X722 was operated at that size.
--- fs/ksmbd/transport_rdma.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/ksmbd/transport_rdma.c +1 -1
@@ -80,7 +80,7 @@ static int smb_direct_max_fragmented_recv_size = 1024 * 1024;/* The maximum single-message size which can be received */ static int smb_direct_max_receive_size =8192 ; -static int smb_direct_max_read_write_size =1048512 ; +static int smb_direct_max_read_write_size =524224 ;static int smb_direct_max_outstanding_rw_ops =8 ;
Signature · 1 line
2.25.1
Namjae Jeon
maintainerPATCH 2/3 ksmbd: fix same UniqueId for dot and dotdot entries
ksmbd sets the inode number to UniqueId. However, the same UniqueId for dot and dotdot entry is set to the inode number of the parent inode. This patch set them using the current inode and parent inode.
--- fs/ksmbd/smb_common.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
fs/ksmbd/smb_common.c +4 -1
@@ -308,14 +308,17 @@ int ksmbd_populate_dot_dotdot_entries(struct ksmbd_work *work, int info_level, for (i = 0; i < 2; i++) { struct kstat kstat; struct ksmbd_kstat ksmbd_kstat; + struct dentry *dentry; if (!dir->dot_dotdot[i]) { /* fill dot entry info */ if (i == 0) { d_info->name = "."; d_info->name_len = 1; + dentry = dir->filp->f_path.dentry; } else { d_info->name = ".."; d_info->name_len = 2; + dentry = dir->filp->f_path.dentry->d_parent; } if (!match_pattern(d_info->name, d_info->name_len,
@@ -327,7 +330,7 @@ int ksmbd_populate_dot_dotdot_entries(struct ksmbd_work *work, int info_level, ksmbd_kstat.kstat = &kstat;ksmbd_vfs_fill_dentry_attrs (work, user_ns, - dir->filp->f_path.dentry->d_parent, + dentry, &ksmbd_kstat); rc =fn (conn, info_level, d_info, &ksmbd_kstat);if (rc)
Signature · 1 line
2.25.1
Namjae Jeon
maintainerPATCH 3/3 ksmbd: don't align last entry offset in smb2 query directory
In reply to Namjae JeonWhen checking smb2 query directory packets from other servers, OutputBufferLength is different with ksmbd. Other servers add an unaligned next offset to OutputBufferLength for the last entry.
--- fs/ksmbd/smb2pdu.c | 7 ++++--- fs/ksmbd/vfs.h | 1 + 2 files changed, 5 insertions(+), 3 deletions(-)
fs/ksmbd/smb2pdu.c +4 -3
@@ -3422,9 +3422,9 @@ static int smb2_populate_readdir_entry(struct ksmbd_conn *conn, int info_level, goto free_conv_name; } - struct_sz =readdir_info_level_struct_sz (info_level); - next_entry_offset =ALIGN (struct_sz -1 + conv_len, -KSMBD_DIR_INFO_ALIGNMENT ); + struct_sz =readdir_info_level_struct_sz (info_level) -1 + conv_len; + next_entry_offset =ALIGN (struct_sz,KSMBD_DIR_INFO_ALIGNMENT ); + d_info->last_entry_off_align = next_entry_offset - struct_sz;if (next_entry_offset > d_info->out_buf_len) { d_info->out_buf_len =0 ;
@@ -3976,6 +3976,7 @@ int smb2_query_dir(struct ksmbd_work *work) ((struct file_directory_info *) ((char *)rsp->Buffer + d_info.last_entry_offset)) ->NextEntryOffset =0 ; + d_info.data_count -= d_info.last_entry_off_align; rsp->StructureSize =cpu_to_le16 (9 ); rsp->OutputBufferOffset =cpu_to_le16 (72 );
fs/ksmbd/vfs.h +1 -0
@@ -47,6 +47,7 @@ struct ksmbd_dir_info {int last_entry_offset;bool hide_dot_file;int flags; +int last_entry_off_align; };struct ksmbd_readdir_data {
Signature · 1 line
2.25.1
Tom Talpey
reviewerQuoted 4 lines · Namjae Jeon
To support RDMA in chelsio NICs, Reduce smb direct read/write size to about 512KB. With this change, we have checked that a single buffer descriptor was sent from Windows client, and intel X722 was operated at that size.
I am guessing that the larger payload required a fast-register of a page count which was larger than the adapter supports? Can you provide more detail?
Also, what exactly does "single buffer descriptor from Windows client" mean, and why is it relevant?
Confused,
Tom.
Quoted 18 lines
Signed-off-by Namjae Jeon <[email protected]>--- fs/ksmbd/transport_rdma.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/ksmbd/transport_rdma.c +1
-1@@ -80,7 +80,7 @@ static int smb_direct_max_fragmented_recv_size = 1024 * 1024;/* The maximum single-message size which can be received */ static int smb_direct_max_receive_size =8192 ; -static int smb_direct_max_read_write_size =1048512 ; +static int smb_direct_max_read_write_size =524224 ;static int smb_direct_max_outstanding_rw_ops =8 ;
Namjae Jeon
maintainer2022-01-31 4:04 GMT+09:00, Tom Talpey <[email protected]>:
Quoted 9 lines · Namjae Jeon
Quoted 4 lines
To support RDMA in chelsio NICs, Reduce smb direct read/write size to about 512KB. With this change, we have checked that a single buffer descriptor was sent from Windows client, and intel X722 was operated at that size.
I am guessing that the larger payload required a fast-register of a page count which was larger than the adapter supports? Can you provide more detail?
Windows client can send multiple Buffer Descriptor V1 structure elements to server.
ksmbd server doesn't support it yet. So it can handle only single element. We have known that Windows sends multiple elements according to the size of smb direct max read/write size. For Melanox adapters, 1MB size, and Chelsea O, 512KB seems to be the threshold. I thought that windows would send a single buffer descriptor element when setting the adapter's max_fast_reg_page_list_len value to read/write size, but it was not.
chelsio's max_fast_reg_page_list_len: 128
mellanox's max_fast_reg_page_list_len: 511
I don't know exactly what factor Windows client uses to send multiple elements. Even in MS-SMB2, It is not described. So I am trying to set the minimum read/write size until multiple elements handling is supported.
Quoted 3 lines
Also, what exactly does "single buffer descriptor from Windows client" mean, and why is it relevant?
Windows can send an array of one or more Buffer Descriptor V1 structures, i.e. multiple elements. Currently, ksmbd can handle only one Buffer Descriptor V1 structure element.
If there's anything I've missed, please let me know.
Quoted 3 lines
Confused,
Tom.
Thanks!
Quoted 21 lines
Quoted 20 lines
Signed-off-by Namjae Jeon <[email protected]>--- fs/ksmbd/transport_rdma.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/ksmbd/transport_rdma.c +1
-1@@ -80,7 +80,7 @@ static int smb_direct_max_fragmented_recv_size = 1024 *1024 ;/* The maximum single-message size which can be received */ static int smb_direct_max_receive_size =8192 ; -static int smb_direct_max_read_write_size =1048512 ; +static int smb_direct_max_read_write_size =524224 ;static int smb_direct_max_outstanding_rw_ops =8 ;
Tom Talpey
reviewerQuoted 7 lines · Namjae Jeon
2022-01-31 4:04 GMT+09:00, Tom Talpey <[email protected]>:
Quoted 6 lines · Namjae Jeon
Quoted 4 lines
To support RDMA in chelsio NICs, Reduce smb direct read/write size to about 512KB. With this change, we have checked that a single buffer descriptor was sent from Windows client, and intel X722 was operated at that size.
Quoted 6 lines
Quoted 3 lines
I am guessing that the larger payload required a fast-register of a page count which was larger than the adapter supports? Can you provide more detail?
Windows client can send multiple Buffer Descriptor V1 structure elements to server.
ksmbd server doesn't support it yet. So it can handle only single element.
Oh! So it's a bug in ksmbd which isn't supporting the protocol. Presumably this will be fixed in the future, and this patch would be reversed.
In any case, the smaller size is purely a workaround which permits it to interoperate with the Windows client. It's not actually a fix, and has nothing fundamentally to do with Chelsio or Intel NICs.
The patch needs to say these. How about
"ksmbd does not support more than one Buffer Descriptor V1 element in an smbdirect protocol request. Reducing the maximum read/write size to about 512KB allows interoperability with Windows over a wider variety of RDMA NICs, as an interim workaround."
Quoted 4 lines
We have known that Windows sends multiple elements according to the size of smb direct max read/write size. For Melanox adapters, 1MB size, and Chelsea O, 512KB seems to be the threshold. I thought that windows would send a single buffer descriptor element when setting the
Quoted 8 lines
adapter's
max_fast_reg_page_list_lenvalue to read/write size, but it was not.
chelsio'smax_fast_reg_page_list_len: 128
mellanox'smax_fast_reg_page_list_len: 511
I don't know exactly what factor Windows client uses to send multiple elements. Even in MS-SMB2, It is not described. So I am trying to set the minimum read/write size until multiple elements handling is supported.
The protocol documents are about the protocol, and they intentionally avoid specifying the behavior of each implementation. You could ask the dochelp folks, but you may not get a clear answer, because as you can see, "it depends" :)
In practice, a client will probably try to pack as many pages into a single registration (memory handle) as possible. This will depend on the memory layout, the adapter capabilities, and the way the client was actually coded (fast-register has very different requirements from other memreg models). I take it the Linux smbdirect client does not trigger this issue?
Is there some reason you can't currently support multiple descriptors? Or is it simply deferred for now?
Tom.
Quoted 33 lines
Quoted 2 lines
Also, what exactly does "single buffer descriptor from Windows client" mean, and why is it relevant?
Windows can send an array of one or more Buffer Descriptor V1 structures, i.e. multiple elements. Currently, ksmbd can handle only one Buffer Descriptor V1 structure element.
If there's anything I've missed, please let me know.
Quoted 3 lines
Confused,
Tom.Thanks!
Quoted 21 lines
Quoted 20 lines
Signed-off-by Namjae Jeon <[email protected]>--- fs/ksmbd/transport_rdma.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/ksmbd/transport_rdma.c +1
-1@@ -80,7 +80,7 @@ static int smb_direct_max_fragmented_recv_size = 1024 *1024 ;/* The maximum single-message size which can be received */ static int smb_direct_max_receive_size =8192 ; -static int smb_direct_max_read_write_size =1048512 ; +static int smb_direct_max_read_write_size =524224 ;static int smb_direct_max_outstanding_rw_ops =8 ;
Namjae Jeon
maintainer2022-01-31 12:18 GMT+09:00, Tom Talpey <[email protected]>:
Quoted 11 lines · Namjae Jeon
Quoted 10 lines
2022-01-31 4:04 GMT+09:00, Tom Talpey <[email protected]>:
Quoted 9 lines · Namjae Jeon
Quoted 4 lines
To support RDMA in chelsio NICs, Reduce smb direct read/write size to about 512KB. With this change, we have checked that a single buffer descriptor was sent from Windows client, and intel X722 was operated at that size.
I am guessing that the larger payload required a fast-register of a page count which was larger than the adapter supports? Can you provide more detail?
Quoted 8 lines
Quoted 4 lines
Windows client can send multiple Buffer Descriptor V1 structure elements to server.
ksmbd server doesn't support it yet. So it can handle only single element.Oh! So it's a bug in ksmbd which isn't supporting the protocol. Presumably this will be fixed in the future, and this patch would be reversed.
Right. Work to support it is in progress, Need the time to complete it.
Quoted 4 lines
In any case, the smaller size is purely a workaround which permits it to interoperate with the Windows client. It's not actually a fix, and has nothing fundamentally to do with Chelsio or Intel NICs.
Right.
Quoted 7 lines
The patch needs to say these. How about
"ksmbd does not support more than one Buffer Descriptor V1 element in an smbdirect protocol request. Reducing the maximum read/write size to about 512KB allows interoperability with Windows over a wider variety of RDMA NICs, as an interim workaround."
Thanks:) I will update patch description.
Quoted 14 lines
Quoted 13 lines
We have known that Windows sends multiple elements according to the size of smb direct max read/write size. For Melanox adapters, 1MB size, and Chelsea O, 512KB seems to be the threshold. I thought that windows would send a single buffer descriptor element when setting the adapter's
max_fast_reg_page_list_lenvalue to read/write size, but it was not.
chelsio'smax_fast_reg_page_list_len: 128
mellanox'smax_fast_reg_page_list_len: 511
I don't know exactly what factor Windows client uses to send multiple elements. Even in MS-SMB2, It is not described. So I am trying to set the minimum read/write size until multiple elements handling is supported.
Quoted 4 lines
The protocol documents are about the protocol, and they intentionally avoid specifying the behavior of each implementation. You could ask the dochelp folks, but you may not get a clear answer, because as you can see, "it depends" :)
Okay.
Quoted 7 lines
In practice, a client will probably try to pack as many pages into a single registration (memory handle) as possible. This will depend on the memory layout, the adapter capabilities, and the way the client was actually coded (fast-register has very different requirements from other memreg models). I take it the Linux smbdirect client does not trigger this issue?
Yes, Because linux smbdirect client send one Buffer Descriptor V1 element:) So there is no issue between linux smbdirect client and ksmbd.
Quoted 3 lines
Is there some reason you can't currently support multiple descriptors? Or is it simply deferred for now?
smbdirect codes of ksmbd are referenced from the linux cifs client's one. and we found that windows client could send more than one element later. It is in progress. I want that ksmbd work RDMA with Windows over a wider variety of RDMA NICs in the meantime.
Thanks!
Quoted 38 lines
Tom.
Quoted 34 lines
Quoted 2 lines
Also, what exactly does "single buffer descriptor from Windows client" mean, and why is it relevant?
Windows can send an array of one or more Buffer Descriptor V1 structures, i.e. multiple elements. Currently, ksmbd can handle only one Buffer Descriptor V1 structure element.
If there's anything I've missed, please let me know.
Quoted 3 lines
Confused,
Tom.Thanks!
Quoted 22 lines
Quoted 21 lines
Signed-off-by Namjae Jeon <[email protected]>--- fs/ksmbd/transport_rdma.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/ksmbd/transport_rdma.c +1
-1@@ -80,7 +80,7 @@ static int smb_direct_max_fragmented_recv_size = 1024 *1024 ;/* The maximum single-message size which can be received */ static int smb_direct_max_receive_size =8192 ; -static int smb_direct_max_read_write_size =1048512 ; +static int smb_direct_max_read_write_size =524224 ;static int smb_direct_max_outstanding_rw_ops = 8;