Implement direct user I/O interfaces for RDMA
Subsystems (3)
cover Implement direct user I/O interfaces for RDMA
From: Long Li <[email protected]>
This patchset implements direct user I/O through RDMA.
In normal code path (even with cache=none), CIFS copies I/O data from user-space to kernel-space for security reasons.
With this patchset, a new mounting option is introduced to have CIFS pin the user-space buffer into memory and performs I/O through RDMA. This avoids memory copy, at the cost of added security risk.
This patchset is RFC. The work is in progress, do not merge.
Long Li (9):
Introduce offset for the 1st page in data transfer structures Change wdata alloc to support direct pages
Change rdata alloc to support direct pages
Change function to support offset when reading pages Change RDMA send to regonize page offset in the 1st page Change RDMA recv to support offset in the 1st page Support page offset in memory regsitrations Implement no-copy file I/O interfaces
Introduce cache=rdma moutning option
fs/cifs/cifs_fs_sb.h | 2 + fs/cifs/cifsfs.c | 19 +++ fs/cifs/cifsfs.h | 3 + fs/cifs/cifsglob.h | 6 + fs/cifs/cifsproto.h | 4 +- fs/cifs/cifssmb.c | 10 +- fs/cifs/connect.c | 13 +- fs/cifs/dir.c | 5 + fs/cifs/file.c | 351 ++++++++++++++++++++++++++++++++++++++++++---- fs/cifs/inode.c | 4 +- fs/cifs/smb2ops.c | 2 +- fs/cifs/smb2pdu.c | 22 ++- fs/cifs/smbdirect.c | 132 ++++++++++------- fs/cifs/smbdirect.h | 2 +- fs/read_write.c | 7 + include/linux/ratelimit.h | 2 +- 16 files changed, 489 insertions(+), 95 deletions(-)
Signature · 1 line
2.7.4
Tom Talpey
reviewerQuoted 10 lines · Long Li
From: Long Li <[email protected]>
This patchset implements direct user I/O through RDMA.
In normal code path (even with cache=none), CIFS copies I/O data from user-space to kernel-space for security reasons.
With this patchset, a new mounting option is introduced to have CIFS pin the user-space buffer into memory and performs I/O through RDMA. This avoids memory copy, at the cost of added security risk.
What's the security risk? This type of direct i/o behavior is not uncommon, and can certainly be made safe, using the appropriate memory registration and protection domains. Any risk needs to be stated explicitly, and mitigation provided, or at least described.
Tom.
Quoted 34 lines
This patchset is RFC. The work is in progress, do not merge.
Long Li (9):
Introduce offset for the 1st page in data transfer structures Change wdata alloc to support direct pages
Change rdata alloc to support direct pages
Change function to support offset when reading pages Change RDMA send to regonize page offset in the 1st page Change RDMA recv to support offset in the 1st page Support page offset in memory regsitrations Implement no-copy file I/O interfaces
Introduce cache=rdma moutning optionfs/cifs/cifs_fs_sb.h | 2 + fs/cifs/cifsfs.c | 19 +++ fs/cifs/cifsfs.h | 3 + fs/cifs/cifsglob.h | 6 + fs/cifs/cifsproto.h | 4 +- fs/cifs/cifssmb.c | 10 +- fs/cifs/connect.c | 13 +- fs/cifs/dir.c | 5 + fs/cifs/file.c | 351 ++++++++++++++++++++++++++++++++++++++++++---- fs/cifs/inode.c | 4 +- fs/cifs/smb2ops.c | 2 +- fs/cifs/smb2pdu.c | 22 ++- fs/cifs/smbdirect.c | 132 ++++++++++------- fs/cifs/smbdirect.h | 2 +- fs/read_write.c | 7 + include/linux/ratelimit.h | 2 +- 16 files changed, 489 insertions(+), 95 deletions(-)
Quoted 11 lines
Subject: Re: [RFC PATCH 00/09] Implement direct user I/O interfaces for RDMA
Quoted 7 lines · Long Li
From: Long Li <[email protected]>
This patchset implements direct user I/O through RDMA.
In normal code path (even with cache=none), CIFS copies I/O data from user-space to kernel-space for security reasons.
Quoted 8 lines
Quoted 3 lines
With this patchset, a new mounting option is introduced to have CIFS pin the user-space buffer into memory and performs I/O through RDMA. This avoids memory copy, at the cost of added security risk.
What's the security risk? This type of direct i/o behavior is not uncommon, and can certainly be made safe, using the appropriate memory registration and protection domains. Any risk needs to be stated explicitly, and mitigation provided, or at least described.
I think it's an assumption that user-mode buffer can't be trusted, so CIFS always copies them into internal buffers, and calculate signature and encryption based on protocol used.
With the direct buffer, the user can potentially modify the buffer when signature or encryption is in progress or after they are done.
I also want to point out that, I choose to implement .read_iter and .write_iter from file_operations to implement direct I/O (CIFS is already doing this for O_DIRECT, so following this code path will avoid a big mess up). The ideal choice is to implement .direct_IO from address_space_operations that I think eventually we want to move to.
Quoted 38 lines
Tom.
Quoted 16 lines
This patchset is RFC. The work is in progress, do not merge.
Long Li (9):
Introduce offset for the 1st page in data transfer structures Change wdata alloc to support direct pages
Change rdata alloc to support direct pages
Change function to support offset when reading pages Change RDMA send to regonize page offset in the 1st page Change RDMA recv to support offset in the 1st page Support page offset in memory regsitrations Implement no-copy file I/O interfaces
Introduce cache=rdma moutning optionQuoted 9 lines
fs/cifs/cifs_fs_sb.h | 2 + fs/cifs/cifsfs.c | 19 +++ fs/cifs/cifsfs.h | 3 + fs/cifs/cifsglob.h | 6 + fs/cifs/cifsproto.h | 4 +- fs/cifs/cifssmb.c | 10 +- fs/cifs/connect.c | 13 +- fs/cifs/dir.c | 5 + fs/cifs/file.c | 351++++++++++++++++++++++++++++++++++++++++++----
Quoted 9 lines
fs/cifs/inode.c | 4 +- fs/cifs/smb2ops.c | 2 +- fs/cifs/smb2pdu.c | 22 ++- fs/cifs/smbdirect.c | 132 ++++++++++------- fs/cifs/smbdirect.h | 2 +- fs/read_write.c | 7 + include/linux/ratelimit.h | 2 +- 16 files changed, 489 insertions(+), 95 deletions(-)
Quoted 1 line · Long Li
I also want to point out that, I choose to implement .read_iter and .write_iter from
file_operationsto implement direct I/O (CIFS is already doing this forO_DIRECT, so following this code path will avoid a big mess up). The ideal choice is to implement .direct_IO fromaddress_space_operationsthat I think eventually we want to move to.
No, the direct_IO address space operation is the mess. We're moving away from it.
Tom Talpey
reviewerQuoted 15 lines · Long Li
Quoted 15 lines
Subject: Re: [RFC PATCH 00/09] Implement direct user I/O interfaces for RDMA
Quoted 10 lines · Long Li
From: Long Li <[email protected]>
This patchset implements direct user I/O through RDMA.
In normal code path (even with cache=none), CIFS copies I/O data from user-space to kernel-space for security reasons.
With this patchset, a new mounting option is introduced to have CIFS pin the user-space buffer into memory and performs I/O through RDMA. This avoids memory copy, at the cost of added security risk.
Quoted 8 lines
Quoted 4 lines
What's the security risk? This type of direct i/o behavior is not uncommon, and can certainly be made safe, using the appropriate memory registration and protection domains. Any risk needs to be stated explicitly, and mitigation provided, or at least described.
I think it's an assumption that user-mode buffer can't be trusted, so CIFS always copies them into internal buffers, and calculate signature and encryption based on protocol used.
With the direct buffer, the user can potentially modify the buffer when signature or encryption is in progress or after they are done.
I don't agree that the legacy copying behavior is because the buffer is "untrusted". The buffer is the user's data, there's no trust issue here. If the user application modifies the buffer while it's being sent, it's a violation of the API contract, and the only victim is the application itself. Same applies for receiving data. And as pointed out, most all storage layers, file and block both, use this strategy for direct i/o.
Regarding signing, if the application alters the data then the integrity hash will simply do its job and catch the application in the act. Again, nothing suffers but the application.
Regarding encryption, I assume you're proposing to encrypt and decrypt the data in a kernel buffer, effectively a copy. So in fact, in the encryption case there's no need to pin and map the user buffer at all.
I'll mention however that Windows takes the path of not performing RDMA placement when encrypting data. It saves nothing, and even adds some overhead, because of the need to touch the buffer anyway to manage the encryption/decryption.
Bottom line - no security implication for using user buffers directly.
Tom.
Quoted 40 lines
I also want to point out that, I choose to implement .read_iter and .write_iter from
file_operationsto implement direct I/O (CIFS is already doing this forO_DIRECT, so following this code path will avoid a big mess up). The ideal choice is to implement .direct_IO fromaddress_space_operationsthat I think eventually we want to move to.Quoted 28 lines
Tom.
Quoted 16 lines
This patchset is RFC. The work is in progress, do not merge.
Long Li (9):
Introduce offset for the 1st page in data transfer structures Change wdata alloc to support direct pages
Change rdata alloc to support direct pages
Change function to support offset when reading pages Change RDMA send to regonize page offset in the 1st page Change RDMA recv to support offset in the 1st page Support page offset in memory regsitrations Implement no-copy file I/O interfaces
Introduce cache=rdma moutning optionQuoted 9 lines
fs/cifs/cifs_fs_sb.h | 2 + fs/cifs/cifsfs.c | 19 +++ fs/cifs/cifsfs.h | 3 + fs/cifs/cifsglob.h | 6 + fs/cifs/cifsproto.h | 4 +- fs/cifs/cifssmb.c | 10 +- fs/cifs/connect.c | 13 +- fs/cifs/dir.c | 5 + fs/cifs/file.c | 351Quoted 10 lines
++++++++++++++++++++++++++++++++++++++++++----
Quoted 9 lines
fs/cifs/inode.c | 4 +- fs/cifs/smb2ops.c | 2 +- fs/cifs/smb2pdu.c | 22 ++- fs/cifs/smbdirect.c | 132 ++++++++++------- fs/cifs/smbdirect.h | 2 +- fs/read_write.c | 7 + include/linux/ratelimit.h | 2 +- 16 files changed, 489 insertions(+), 95 deletions(-)
Quoted 1 line
N�����r��y���b�X��ǧv�^�){.n�+����{��ٚ�{ay�
ʇڙ�,j��f���h���z�
�w���
���j:+v���w�j�m��������zZ+�����ݢj"��!tml=
Quoted 1 line
Quoted 4 lines · Tom Talpey
What's the security risk? This type of direct i/o behavior is not uncommon, and can certainly be made safe, using the appropriate memory registration and protection domains. Any risk needs to be stated explicitly, and mitigation provided, or at least described.
And in fact it is the same behavior you'll see on NFS over RDMA, or a block device or any local fs over SRP/iSER/NVMe over Fabrics..
RFC PATCH 1/9 Introduce offset for the 1st page in data transfer structures
In reply to Long LiFrom: Long Li <[email protected]>
Currently CIFS allocates its own pages for data transfer, they don't need offset since it's always 0 in the 1st page.
Direct data transfer needs to define an offset because user-data may not start on the page boundary
--- fs/cifs/cifsglob.h | 5 +++++ 1 file changed, 5 insertions(+)
fs/cifs/cifsglob.h +5 -0
@@ -176,6 +176,7 @@ struct smb_rqst {struct kvec *rq_iov;/* array of kvecs */ unsigned int rq_nvec;/* number of kvecs in array */ struct page **rq_pages;/* pointer to array of page ptrs */ +unsigned int rq_offset;/* the offset to the 1st page */ unsigned int rq_npages;/* number pages in array */ unsigned int rq_pagesz;/* page size to use */ unsigned int rq_tailsz;/* length of last page */
@@ -1167,8 +1168,10 @@ struct cifs_readdata {struct kvec iov[2 ];#ifdef CONFIG_CIFS_SMB_DIRECT struct smbd_mr *mr; +struct page **direct_pages;#endif unsigned int pagesz; +unsigned int page_offset;unsigned int tailsz;unsigned int credits;unsigned int nr_pages;
@@ -1192,8 +1195,10 @@ struct cifs_writedata {int result;#ifdef CONFIG_CIFS_SMB_DIRECT struct smbd_mr *mr; +struct page **direct_pages;#endif unsigned int pagesz; +unsigned int page_offset;unsigned int tailsz;unsigned int credits;unsigned int nr_pages;
Signature · 1 line
2.7.4
Steve French
AppliedRe: [RFC PATCH 01/09] Introduce offset for the 1st page in data transfer structures
merged into cifs-2.6.git for-next
Quoted 54 lines · Long Li
From: Long Li <[email protected]>
Currently CIFS allocates its own pages for data transfer, they don't need offset since it's always 0 in the 1st page.
Direct data transfer needs to define an offset because user-data may not start on the page boundary
Signed-off-by Long Li <[email protected]>--- fs/cifs/cifsglob.h | 5 +++++ 1 file changed, 5 insertions(+)fs/cifs/cifsglob.h +5
-1@@ -176,6 +176,7 @@ struct smb_rqst {struct kvec *rq_iov;/* array of kvecs */ unsigned int rq_nvec;/* number of kvecs in array */ struct page **rq_pages;/* pointer to array of page ptrs */ +unsigned int rq_offset;/* the offset to the 1st page */ unsigned int rq_npages;/* number pages in array */ unsigned int rq_pagesz;/* page size to use */ unsigned int rq_tailsz;/* length of last page */ @@ -1167,8 +1168,10 @@ struct cifs_readdata {struct kvec iov[2 ];#ifdef CONFIG_CIFS_SMB_DIRECT struct smbd_mr *mr; +struct page **direct_pages;#endif unsigned int pagesz; +unsigned int page_offset;unsigned int tailsz;unsigned int credits;unsigned int nr_pages;@@ -1192,8 +1195,10 @@ struct cifs_writedata {int result;#ifdef CONFIG_CIFS_SMB_DIRECT struct smbd_mr *mr; +struct page **direct_pages;#endif unsigned int pagesz; +unsigned int page_offset;unsigned int tailsz;unsigned int credits;unsigned int nr_pages; --2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-cifs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
Thanks, Steve
RFC PATCH 2/9 Change wdata alloc to support direct pages
In reply to Long LiFrom: Long Li <[email protected]>
When using direct pages from user space, there is no need to allocate pages.
Just ping those user pages for RDMA.
--- fs/cifs/cifsproto.h | 2 +- fs/cifs/cifssmb.c | 10 +++++++--- fs/cifs/file.c | 4 ++-- 3 files changed, 10 insertions(+), 6 deletions(-)
fs/cifs/cifsproto.h +1 -1
@@ -523,7 +523,7 @@ int cifs_readv_receive(struct TCP_Server_Info *server, struct mid_q_entry *mid);int cifs_async_writev (struct cifs_writedata *wdata,void (*release)(struct kref *kref));void cifs_writev_complete (struct work_struct *work); -struct cifs_writedata *cifs_writedata_alloc (unsigned int nr_pages, +struct cifs_writedata *cifs_writedata_alloc (unsigned int nr_pages,struct page **direct_pages,work_func_t complete);void cifs_writedata_release (struct kref *refcount); int cifs_query_mf_symlink(unsigned int xid, struct cifs_tcon *tcon,
fs/cifs/cifssmb.c +7 -3
@@ -1983,7 +1983,7 @@ cifs_writev_requeue(struct cifs_writedata *wdata) tailsz = rest_len - (nr_pages -1 ) *PAGE_SIZE ; } - wdata2 =cifs_writedata_alloc (nr_pages, cifs_writev_complete); + wdata2 =cifs_writedata_alloc (nr_pages,NULL , cifs_writev_complete);if (!wdata2) { rc = -ENOMEM ;break ;
@@ -2067,12 +2067,16 @@ cifs_writev_complete(struct work_struct *work) }struct cifs_writedata * -cifs_writedata_alloc (unsigned int nr_pages,work_func_t complete) +cifs_writedata_alloc (unsigned int nr_pages,struct page **direct_pages,work_func_t complete) {struct cifs_writedata *wdata;/* writedata + number of page pointers */ - wdata =kzalloc (sizeof (*wdata) + +if (direct_pages) { + wdata =kzalloc (sizeof (*wdata),GFP_NOFS ); + wdata->direct_pages = direct_pages; + }else + wdata =kzalloc (sizeof (*wdata) +sizeof (struct page *) * nr_pages,GFP_NOFS );if (wdata !=NULL ) {kref_init (&wdata->refcount);
fs/cifs/file.c +2 -2
@@ -1965,7 +1965,7 @@ wdata_alloc_and_fillpages(pgoff_t tofind, struct address_space *mapping, {struct cifs_writedata *wdata; - wdata =cifs_writedata_alloc ((unsigned int )tofind, + wdata =cifs_writedata_alloc ((unsigned int )tofind,NULL , cifs_writev_complete);if (!wdata)return NULL ;
@@ -2554,7 +2554,7 @@ cifs_write_from_iter(loff_t offset, size_t len, struct iov_iter *from,break ; nr_pages =get_numpages (wsize, len, &cur_len); - wdata =cifs_writedata_alloc (nr_pages, + wdata =cifs_writedata_alloc (nr_pages,NULL , cifs_uncached_writev_complete);if (!wdata) { rc = -ENOMEM ;
Signature · 1 line
2.7.4
Tom Talpey
reviewerRe: [RFC PATCH 02/09] Change wdata alloc to support direct pages
Quoted 5 lines · Long Li
From: Long Li <[email protected]>
When using direct pages from user space, there is no need to allocate pages.
Just ping those user pages for RDMA.
Did you mean "pin" those user pages? If so, where does that pinning occur, it's not in this patch.
Perhaps this should just say "point to" those user pages.
I also don't think this is necessarily only "for RDMA". Perhaps there are other transport scenarios where this is advantageous.
Quoted 76 lines
Signed-off-by Long Li <[email protected]>--- fs/cifs/cifsproto.h | 2 +- fs/cifs/cifssmb.c | 10 +++++++--- fs/cifs/file.c | 4 ++-- 3 files changed, 10 insertions(+), 6 deletions(-)fs/cifs/cifsproto.h +1
-1@@ -523,7 +523,7 @@ int cifs_readv_receive(struct TCP_Server_Info *server, struct mid_q_entry *mid);int cifs_async_writev (struct cifs_writedata *wdata,void (*release)(struct kref *kref));void cifs_writev_complete (struct work_struct *work); -struct cifs_writedata *cifs_writedata_alloc (unsigned int nr_pages, +struct cifs_writedata *cifs_writedata_alloc (unsigned int nr_pages,struct page **direct_pages,work_func_t complete);void cifs_writedata_release (struct kref *refcount); int cifs_query_mf_symlink(unsigned int xid, struct cifs_tcon *tcon,fs/cifs/cifssmb.c +7
-3@@ -1983,7 +1983,7 @@ cifs_writev_requeue(struct cifs_writedata *wdata) tailsz = rest_len - (nr_pages -1 ) *PAGE_SIZE ; } - wdata2 =cifs_writedata_alloc (nr_pages, cifs_writev_complete); + wdata2 =cifs_writedata_alloc (nr_pages,NULL , cifs_writev_complete);if (!wdata2) { rc = -ENOMEM ;break ;@@ -2067,12 +2067,16 @@ cifs_writev_complete(struct work_struct *work) }struct cifs_writedata * -cifs_writedata_alloc (unsigned int nr_pages,work_func_t complete) +cifs_writedata_alloc (unsigned int nr_pages,struct page **direct_pages,work_func_t complete) {struct cifs_writedata *wdata;/* writedata + number of page pointers */ - wdata =kzalloc (sizeof (*wdata) + +if (direct_pages) { + wdata =kzalloc (sizeof (*wdata),GFP_NOFS ); + wdata->direct_pages = direct_pages; + }else + wdata =kzalloc (sizeof (*wdata) +sizeof (struct page *) * nr_pages,GFP_NOFS );if (wdata !=NULL ) {kref_init (&wdata->refcount);fs/cifs/file.c +2
-2@@ -1965,7 +1965,7 @@ wdata_alloc_and_fillpages(pgoff_t tofind, struct address_space *mapping, {struct cifs_writedata *wdata; - wdata =cifs_writedata_alloc ((unsigned int )tofind, + wdata =cifs_writedata_alloc ((unsigned int )tofind,NULL , cifs_writev_complete);if (!wdata)return NULL ;@@ -2554,7 +2554,7 @@ cifs_write_from_iter(loff_t offset, size_t len, struct iov_iter *from,break ; nr_pages =get_numpages (wsize, len, &cur_len); - wdata =cifs_writedata_alloc (nr_pages, + wdata =cifs_writedata_alloc (nr_pages,NULL , cifs_uncached_writev_complete);if (!wdata) { rc = -ENOMEM ;
RFC PATCH 3/9 Change rdata alloc to support direct pages
In reply to Long LiFrom: Long Li <[email protected]>
There is no need to allocate pages when using pages directly from user buffer
--- fs/cifs/file.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
fs/cifs/file.c +8 -5
@@ -2880,11 +2880,15 @@ cifs_strict_writev(struct kiocb *iocb, struct iov_iter *from) }static struct cifs_readdata * -cifs_readdata_alloc (unsigned int nr_pages,work_func_t complete) +cifs_readdata_alloc (unsigned int nr_pages,struct page **direct_pages,work_func_t complete) {struct cifs_readdata *rdata; - rdata =kzalloc (sizeof (*rdata) + (sizeof (struct page *) * nr_pages), +if (direct_pages) { + rdata =kzalloc (sizeof (*rdata),GFP_KERNEL ); + rdata->direct_pages = direct_pages; + }else + rdata =kzalloc (sizeof (*rdata) + (sizeof (struct page *) * nr_pages),GFP_KERNEL );if (rdata !=NULL ) {kref_init (&rdata->refcount);
@@ -3095,14 +3099,13 @@ cifs_send_async_read(loff_t offset, size_t len, struct cifsFileInfo *open_file, npages =DIV_ROUND_UP (cur_len,PAGE_SIZE );/* allocate a readdata struct */ - rdata =cifs_readdata_alloc (npages, + rdata =cifs_readdata_alloc (npages,NULL , cifs_uncached_readv_complete);if (!rdata) {add_credits_and_wake_if (server, credits,0 ); rc = -ENOMEM ;break ; } - rc =cifs_read_allocate_pages (rdata, npages);if (rc) goto error;
@@ -3770,7 +3773,7 @@ static int cifs_readpages(struct file *file, struct address_space *mapping,break ; } - rdata =cifs_readdata_alloc (nr_pages, cifs_readv_complete); + rdata =cifs_readdata_alloc (nr_pages,NULL , cifs_readv_complete);if (!rdata) {/* best to give up if we're out of mem */ list_for_each_entry_safe (page, tpage, &tmplist, lru) {
Signature · 1 line
2.7.4
RFC PATCH 4/9 Change function to support offset when reading pages
In reply to Long LiFrom: Long Li <[email protected]>
It's possible that we may want to read data into an offset to the 1st page, change the functions to pass the offset to transport.
--- fs/cifs/cifsproto.h | 2 +- fs/cifs/connect.c | 4 ++-- fs/cifs/file.c | 4 ++-- fs/cifs/smb2ops.c | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-)
fs/cifs/cifsproto.h +1 -1
@@ -197,7 +197,7 @@ extern void dequeue_mid(struct mid_q_entry *mid, bool malformed);extern int cifs_read_from_socket (struct TCP_Server_Info *server,char *buf,unsigned int to_read);extern int cifs_read_page_from_socket (struct TCP_Server_Info *server, -struct page *page,unsigned int to_read); +struct page *page,unsigned int page_offset,unsigned int to_read);extern int cifs_setup_cifs_sb (struct smb_vol *pvolume_info,struct cifs_sb_info *cifs_sb);extern int cifs_match_super (struct super_block *,void *);
fs/cifs/connect.c +2 -2
@@ -591,11 +591,11 @@ cifs_read_from_socket(struct TCP_Server_Info *server, char *buf, }int -cifs_read_page_from_socket (struct TCP_Server_Info *server,struct page *page, +cifs_read_page_from_socket (struct TCP_Server_Info *server,struct page *page,unsigned int page_offset,unsigned int to_read) {struct msghdr smb_msg; -struct bio_vec bv = {.bv_page = page, .bv_len = to_read}; +struct bio_vec bv = {.bv_page = page, .bv_len = to_read, .bv_offset = page_offset};iov_iter_bvec (&smb_msg.msg_iter,READ |ITER_BVEC , &bv,1 , to_read);return cifs_readv_from_socket (server, &smb_msg); }
fs/cifs/file.c +2 -2
@@ -3044,7 +3044,7 @@ uncached_fill_pages(struct TCP_Server_Info *server, result = n;#endif else - result =cifs_read_page_from_socket (server, page, n); + result =cifs_read_page_from_socket (server, page, page_offset, n);if (result <0 )break ;
@@ -3614,7 +3614,7 @@ readpages_fill_pages(struct TCP_Server_Info *server, result = n;#endif else - result =cifs_read_page_from_socket (server, page, n); + result =cifs_read_page_from_socket (server, page,0 , n);if (result <0 )break ;
fs/cifs/smb2ops.c +1 -1
@@ -2387,7 +2387,7 @@ read_data_into_pages(struct TCP_Server_Info *server, struct page **pages,zero_user (page, len,PAGE_SIZE - len); len =0 ; } - length =cifs_read_page_from_socket (server, page, n); + length =cifs_read_page_from_socket (server, page,0 , n);if (length <0 )return length; server->total_read += length;
Signature · 1 line
2.7.4
RFC PATCH 5/9 Change RDMA send to regonize page offset in the 1st page
In reply to Long LiFrom: Long Li <[email protected]>
When doing RDMA send, the offset needs to be checked as data may start in an offset in the 1st page.
--- fs/cifs/smb2pdu.c | 3 ++- fs/cifs/smbdirect.c | 25 +++++++++++++++++++------ 2 files changed, 21 insertions(+), 7 deletions(-)
fs/cifs/smb2pdu.c +2 -1
@@ -3015,7 +3015,8 @@ smb2_async_writev(struct cifs_writedata *wdata, rqst.rq_iov = iov; rqst.rq_nvec =2 ; - rqst.rq_pages = wdata->pages; + rqst.rq_pages = wdata->direct_pages ? wdata->direct_pages : wdata->pages; + rqst.rq_offset = wdata->page_offset; rqst.rq_npages = wdata->nr_pages; rqst.rq_pagesz = wdata->pagesz; rqst.rq_tailsz = wdata->tailsz;
fs/cifs/smbdirect.c +19 -6
@@ -2084,8 +2084,10 @@ int smbd_send(struct smbd_connection *info, struct smb_rqst *rqst)/* add in the page array if there is one */ if (rqst->rq_npages) { - buflen += rqst->rq_pagesz * (rqst->rq_npages -1 ); - buflen += rqst->rq_tailsz; +if (rqst->rq_npages ==1 ) + buflen += rqst->rq_tailsz; +else + buflen += rqst->rq_pagesz * (rqst->rq_npages -1 ) - rqst->rq_offset + rqst->rq_tailsz; } if (buflen + sizeof(struct smbd_data_transfer) >
@@ -2182,8 +2184,19 @@ int smbd_send(struct smbd_connection *info, struct smb_rqst *rqst)/* now sending pages if there are any */ for (i =0 ; i < rqst->rq_npages; i++) { - buflen = (i == rqst->rq_npages-1 ) ? - rqst->rq_tailsz : rqst->rq_pagesz; +unsigned int offset =0 ; +if (i ==0 ) + offset = rqst->rq_offset; +if (rqst->rq_npages ==1 || i == rqst->rq_npages-1 ) + buflen = rqst->rq_tailsz; +else { +/* We have at least two pages, and this is not the last page */ +if (i ==0 ) + buflen = rqst->rq_pagesz - rqst->rq_offset; +else + buflen = rqst->rq_pagesz; + } + nvecs = (buflen + max_iov_size -1 ) / max_iov_size;log_write (INFO ,"sending pages buflen=%d nvecs=%d\n" , buflen, nvecs);
@@ -2194,9 +2207,9 @@ int smbd_send(struct smbd_connection *info, struct smb_rqst *rqst) remaining_data_length -= size;log_write (INFO ,"sending pages i=%d offset=%d size=%d" " remaining_data_length=%d\n" , - i, j*max_iov_size, size, remaining_data_length); + i, j*max_iov_size+offset, size, remaining_data_length); rc =smbd_post_send_page ( - info, rqst->rq_pages[i], j*max_iov_size, + info, rqst->rq_pages[i], j*max_iov_size + offset, size, remaining_data_length);if (rc) goto done;
Signature · 1 line
2.7.4
Tom Talpey
reviewerRe: [RFC PATCH 05/09] Change RDMA send to regonize page offset in the 1st page
Quoted 1 line · Long Li
From: Long Li <[email protected]>
There's a typo "recognize" in the patch title
Quoted 2 lines
When doing RDMA send, the offset needs to be checked as data may start in an offset in the 1st page.
Doesn't this patch alter the generic smb2pdu.c code too? I think this should note "any" send, not just RDMA?
Tom.
Quoted 73 lines
Signed-off-by Long Li <[email protected]>--- fs/cifs/smb2pdu.c | 3 ++- fs/cifs/smbdirect.c | 25 +++++++++++++++++++------ 2 files changed, 21 insertions(+), 7 deletions(-)fs/cifs/smb2pdu.c +2
-1@@ -3015,7 +3015,8 @@ smb2_async_writev(struct cifs_writedata *wdata, rqst.rq_iov = iov; rqst.rq_nvec =2 ; - rqst.rq_pages = wdata->pages; + rqst.rq_pages = wdata->direct_pages ? wdata->direct_pages : wdata->pages; + rqst.rq_offset = wdata->page_offset; rqst.rq_npages = wdata->nr_pages; rqst.rq_pagesz = wdata->pagesz; rqst.rq_tailsz = wdata->tailsz;fs/cifs/smbdirect.c +19
-6@@ -2084,8 +2084,10 @@ int smbd_send(struct smbd_connection *info, struct smb_rqst *rqst)/* add in the page array if there is one */ if (rqst->rq_npages) { - buflen += rqst->rq_pagesz * (rqst->rq_npages -1 ); - buflen += rqst->rq_tailsz; +if (rqst->rq_npages ==1 ) + buflen += rqst->rq_tailsz; +else + buflen += rqst->rq_pagesz * (rqst->rq_npages -1 ) - rqst->rq_offset + rqst->rq_tailsz; } if (buflen + sizeof(struct smbd_data_transfer) >@@ -2182,8 +2184,19 @@ int smbd_send(struct smbd_connection *info, struct smb_rqst *rqst)/* now sending pages if there are any */ for (i =0 ; i < rqst->rq_npages; i++) { - buflen = (i == rqst->rq_npages-1 ) ? - rqst->rq_tailsz : rqst->rq_pagesz; +unsigned int offset =0 ; +if (i ==0 ) + offset = rqst->rq_offset; +if (rqst->rq_npages ==1 || i == rqst->rq_npages-1 ) + buflen = rqst->rq_tailsz; +else { +/* We have at least two pages, and this is not the last page */ +if (i ==0 ) + buflen = rqst->rq_pagesz - rqst->rq_offset; +else + buflen = rqst->rq_pagesz; + } + nvecs = (buflen + max_iov_size -1 ) / max_iov_size;log_write (INFO ,"sending pages buflen=%d nvecs=%d\n" , buflen, nvecs);@@ -2194,9 +2207,9 @@ int smbd_send(struct smbd_connection *info, struct smb_rqst *rqst) remaining_data_length -= size;log_write (INFO ,"sending pages i=%d offset=%d size=%d" " remaining_data_length=%d\n" , - i, j*max_iov_size, size, remaining_data_length); + i, j*max_iov_size+offset, size, remaining_data_length); rc =smbd_post_send_page ( - info, rqst->rq_pages[i], j*max_iov_size, + info, rqst->rq_pages[i], j*max_iov_size + offset, size, remaining_data_length);if (rc) goto done;
RE: [RFC PATCH 05/09] Change RDMA send to regonize page offset in the 1st page
Quoted 6 lines
Subject: Re: [RFC PATCH 05/09] Change RDMA send to regonize page offset in the 1st page
Quoted 1 line · Long Li
From: Long Li <[email protected]>
Quoted 8 lines
There's a typo "recognize" in the patch title
Quoted 2 lines
When doing RDMA send, the offset needs to be checked as data may start in an offset in the 1st page.
Doesn't this patch alter the generic
smb2pdu.ccode too? I think this should note "any" send, not just RDMA?
Yes, but for TCP the direct_pages and page_offset are always NULL and 0 when cache=rdma is in mounting options, so it doesn't really affect he TCP code path in this patch set.
This behavior can be changed, and it will work with TCP too.
Quoted 86 lines
Tom.
Quoted 12 lines
Signed-off-by Long Li <[email protected]>--- fs/cifs/smb2pdu.c | 3 ++- fs/cifs/smbdirect.c | 25 +++++++++++++++++++------ 2 files changed, 21 insertions(+), 7 deletions(-)fs/cifs/smb2pdu.c index renamed from fs/cifs/smb2pdu.c +0
-05097f28..fdcf97e 100644
fs/cifs/smb2pdu.c smb2_async_writev(struct cifs_writedata *wdata, · line 3018
Quoted 5 lines
fs/cifs/smb2pdu.c +1
-1@@ -3015,7 +3015,8 @@ smb2_async_writev(struct cifs_writedata *wdata, rqst.rq_iov = iov; rqst.rq_nvec =2 ; - rqst.rq_pages = wdata->pages; + rqst.rq_pages = wdata->direct_pages ? wdata->direct_pages :
wdata->pages;Quoted 28 lines
fs/cifs/smb2pdu.c +1
-0+ rqst.rq_offset = wdata->page_offset; rqst.rq_npages = wdata->nr_pages; rqst.rq_pagesz = wdata->pagesz; rqst.rq_tailsz = wdata->tailsz;fs/cifs/smbdirect.c index renamed from fs/cifs/smbdirect.c +0
-0b0a1955..b46586d 100644
fs/cifs/smbdirect.c +6
-2@@ -2084,8 +2084,10 @@ int smbd_send(struct smbd_connection *info, struct smb_rqst *rqst)/* add in the page array if there is one */ if (rqst->rq_npages) { - buflen += rqst->rq_pagesz * (rqst->rq_npages -1 ); - buflen += rqst->rq_tailsz; +if (rqst->rq_npages ==1 ) + buflen += rqst->rq_tailsz; +else + buflen += rqst->rq_pagesz * (rqst->rq_npages -1 ) - +rqst->rq_offset + rqst->rq_tailsz; } if (buflen + sizeof(struct smbd_data_transfer) > @@ -2182,8 +2184,19 @@ int smbd_send(struct smbd_connection *info, struct
smb_rqst*rqst)/* now sending pages if there are any */
for (i = 0; i <rqst->rq_npages; i++) {fs/cifs/smbdirect.c int smbd_send(struct smbd_connection *info, struct smb_rqst *rqst)
Quoted 9 lines
fs/cifs/smbdirect.c +7
-2- buflen = (i == rqst->rq_npages-1 ) ? - rqst->rq_tailsz : rqst->rq_pagesz; +unsigned int offset =0 ; +if (i ==0 ) + offset = rqst->rq_offset; +if (rqst->rq_npages ==1 || i == rqst->rq_npages-1 ) + buflen = rqst->rq_tailsz; +else { +/* We have at least two pages, and this is not the last page */
fs/cifs/smbdirect.c int smbd_send(struct smbd_connection *info,
Quoted 10 lines
fs/cifs/smbdirect.c +6
-0+if (i ==0 ) + buflen = rqst->rq_pagesz - rqst->rq_offset; +else + buflen = rqst->rq_pagesz; + } + nvecs = (buflen + max_iov_size -1 ) / max_iov_size;log_write (INFO ,"sending pages buflen=%d nvecs=%d\n" , buflen, nvecs);@@ -2194,9 +2207,9 @@ int smbd_send(struct smbd_connection *info,struct
smb_rqst*rqst)fs/cifs/smbdirect.c int smbd_send(struct smbd_connection *info, · line 2208
Quoted 2 lines
fs/cifs/smbdirect.c +0
-0remaining_data_length -= size; log_write(INFO, "sending pages i=%d offset=%dsize=%d"
fs/cifs/smbdirect.c int smbd_send(struct smbd_connection *info, · line 2197
Quoted 2 lines
fs/cifs/smbdirect.c +0
-1" remaining_data_length=%d\n", - i, j*max_iov_size, size,
remaining_data_length);fs/cifs/smbdirect.c int smbd_send(struct smbd_connection *info, · line 2210
Quoted 1 line
fs/cifs/smbdirect.c +1
-0+ i, j*max_iov_size+offset, size,
remaining_data_length);fs/cifs/smbdirect.c int smbd_send(struct smbd_connection *info, struct smb_rqst *rqst)
Quoted 3 lines
fs/cifs/smbdirect.c +1
-1rc = smbd_post_send_page( - info, rqst->rq_pages[i], j*max_iov_size, + info, rqst->rq_pages[i], j*max_iov_size +offset,
Quoted 4 lines
fs/cifs/smbdirect.c +0
-0size, remaining_data_length);if (rc) goto done;
RFC PATCH 6/9 Change RDMA recv to support offset in the 1st page
In reply to Long LiFrom: Long Li <[email protected]>
The actaul data buffer may start with an offset in the 1st page, modify RDMA recv function to read the data to the correct buffer.
--- fs/cifs/smbdirect.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
fs/cifs/smbdirect.c +7 -6
@@ -1978,7 +1978,7 @@ static int smbd_recv_buf(struct smbd_connection *info, char *buf,* return value: actual data read */ static int smbd_recv_page (struct smbd_connection *info, -struct page *page,unsigned int to_read) +struct page *page,unsigned int page_offset,unsigned int to_read) {int ret;char *to_address;
@@ -1989,10 +1989,10 @@ static int smbd_recv_page(struct smbd_connection *info, info->reassembly_data_length >= to_read || info->transport_status != SMBD_CONNECTED);if (ret) -return 0 ; +return ret;/* now we can read from reassembly queue and not sleep */ - to_address =kmap_atomic (page); + to_address = (char *)kmap_atomic (page) + page_offset;log_read (INFO ,"reading from page=%p address=%p to_read=%d\n" , page, to_address, to_read);
@@ -2012,7 +2012,7 @@ int smbd_recv(struct smbd_connection *info, struct msghdr *msg) {char *buf;struct page *page; -unsigned int to_read; +unsigned int to_read, page_offset;int rc;switch (msg->msg_iter.type) {
@@ -2024,15 +2024,16 @@ int smbd_recv(struct smbd_connection *info, struct msghdr *msg)case READ |ITER_BVEC : page = msg->msg_iter.bvec->bv_page; + page_offset = msg->msg_iter.bvec->bv_offset; to_read = msg->msg_iter.bvec->bv_len; - rc =smbd_recv_page (info, page, to_read); + rc =smbd_recv_page (info, page, page_offset, to_read);break ;default :/* It's a bug in upper layer to get there */ cifs_dbg (VFS ,"CIFS: invalid msg type %d\n" , msg->msg_iter.type); - rc = -EIO ; + rc = -EINVAL ; }/* SMBDirect will read it all or nothing */
Signature · 1 line
2.7.4
RFC PATCH 7/9 Support page offset in memory regsitrations
In reply to Long LiFrom: Long Li <[email protected]>
Now momory registration needs to recognize offset in page when direct transfer is used.
--- fs/cifs/smb2pdu.c | 19 +++++++++----- fs/cifs/smbdirect.c | 74 +++++++++++++++++++++++++++++++---------------------- fs/cifs/smbdirect.h | 2 +- 3 files changed, 57 insertions(+), 38 deletions(-)
fs/cifs/smb2pdu.c +13 -6
@@ -2586,6 +2586,8 @@ smb2_new_read_req(void **buf, unsigned int *total_len, req->MinimumCount =0 ; req->Length =cpu_to_le32 (io_parms->length); req->Offset =cpu_to_le64 (io_parms->offset); +if (!rdata->tailsz) + rdata->tailsz = rdata->pagesz;#ifdef CONFIG_CIFS_SMB_DIRECT /* * If we want to do a RDMA write, fill in and append
@@ -2601,8 +2603,8 @@ smb2_new_read_req(void **buf, unsigned int *total_len,rcu_read_lock ();rcu_dereference (server->smbd_conn); rdata->mr =smbd_register_mr ( - server->smbd_conn, rdata->pages, - rdata->nr_pages, rdata->tailsz, + server->smbd_conn, rdata->direct_pages ? rdata->direct_pages : rdata->pages, + rdata->nr_pages, rdata->page_offset, rdata->tailsz, true, need_invalidate);rcu_read_unlock ();if (!rdata->mr)
@@ -2967,6 +2969,8 @@ smb2_async_writev(struct cifs_writedata *wdata, req->DataOffset =cpu_to_le16 ( offsetof(struct smb2_write_req , Buffer)); req->RemainingBytes =0 ; +if (!wdata->tailsz) + wdata->tailsz = wdata->pagesz;#ifdef CONFIG_CIFS_SMB_DIRECT /* * If we want to do a server RDMA read, fill in and append
@@ -2981,8 +2985,8 @@ smb2_async_writev(struct cifs_writedata *wdata,rcu_read_lock ();rcu_dereference (server->smbd_conn); wdata->mr =smbd_register_mr ( - server->smbd_conn, wdata->pages, - wdata->nr_pages, wdata->tailsz, + server->smbd_conn, wdata->direct_pages ? wdata->direct_pages : wdata->pages, + wdata->nr_pages, wdata->page_offset, wdata->tailsz, false, need_invalidate);rcu_read_unlock ();if (!wdata->mr) {
@@ -2991,8 +2995,11 @@ smb2_async_writev(struct cifs_writedata *wdata, } req->Length =0 ; req->DataOffset =0 ; - req->RemainingBytes = -cpu_to_le32 ((wdata->nr_pages-1 )*PAGE_SIZE + wdata->tailsz); +if (wdata->nr_pages >1 ) + req->RemainingBytes = +cpu_to_le32 ((wdata->nr_pages-1 )*PAGE_SIZE - wdata->page_offset + wdata->tailsz); +else + req->RemainingBytes =cpu_to_le32 (wdata->tailsz); req->Channel =SMB2_CHANNEL_RDMA_V1_INVALIDATE ;if (need_invalidate) req->Channel =SMB2_CHANNEL_RDMA_V1 ;
fs/cifs/smbdirect.c +43 -31
112 lines not on this page · raw message
fs/cifs/smbdirect.h +1 -1
@@ -314,7 +314,7 @@ struct smbd_mr {/* Interfaces to register and deregister MR for RDMA read/write */ struct smbd_mr *smbd_register_mr (struct smbd_connection *info,struct page *pages[],int num_pages, -int tailsz,bool writing,bool need_invalidate); +int offset,int tailsz,bool writing,bool need_invalidate);int smbd_deregister_mr (struct smbd_mr *mr);#else
Signature · 1 line
2.7.4
RFC PATCH 8/9 Implement direct file I/O interfaces
In reply to Long LiFrom: Long Li <[email protected]>
Implement the main filesystem interface for doing read and write. These functions don't copy the user data into a kenrel buffer for data transfer. Pages are directly pinned and passed to the RDMA transport.
--- fs/cifs/cifsfs.c | 19 ++++ fs/cifs/cifsfs.h | 3 + fs/cifs/file.c | 322 ++++++++++++++++++++++++++++++++++++++++++++++++++++--- 3 files changed, 329 insertions(+), 15 deletions(-)
fs/cifs/cifsfs.c +19 -0
@@ -1118,6 +1118,25 @@ const struct file_operations cifs_file_direct_ops = { .fallocate = cifs_fallocate, }; +const struct file_operations cifs_file_direct_rdma_ops = { + .read_iter = cifs_direct_readv, + .write_iter = cifs_direct_writev, + .open = cifs_open, + .release = cifs_close, + .lock = cifs_lock, + .fsync = cifs_fsync, + .flush = cifs_flush, + .mmap = cifs_file_mmap, + .splice_read = generic_file_splice_read, + .splice_write = iter_file_splice_write, + .unlocked_ioctl = cifs_ioctl, + .copy_file_range = cifs_copy_file_range, + .clone_file_range = cifs_clone_file_range, + .llseek = cifs_llseek, + .setlease = cifs_setlease, + .fallocate = cifs_fallocate, +}; +const struct file_operations cifs_file_nobrl_ops = { .read_iter = cifs_loose_read_iter, .write_iter = cifs_file_write_iter,
fs/cifs/cifsfs.h +3 -0
@@ -94,6 +94,7 @@ extern const struct inode_operations cifs_dfs_referral_inode_operations;/* Functions related to files and directories */ extern const struct file_operations cifs_file_ops;extern const struct file_operations cifs_file_direct_ops;/* if directio mnt */ +extern const struct file_operations cifs_file_direct_rdma_ops;/* if directio mnt */ extern const struct file_operations cifs_file_strict_ops;/* if strictio mnt */ extern const struct file_operations cifs_file_nobrl_ops;/* no brlocks */ extern const struct file_operations cifs_file_direct_nobrl_ops;
@@ -102,8 +103,10 @@ extern int cifs_open(struct inode *inode, struct file *file);extern int cifs_close (struct inode *inode,struct file *file);extern int cifs_closedir (struct inode *inode,struct file *file);extern ssize_t cifs_user_readv (struct kiocb *iocb,struct iov_iter *to); +extern ssize_t cifs_direct_readv (struct kiocb *iocb,struct iov_iter *to);extern ssize_t cifs_strict_readv (struct kiocb *iocb,struct iov_iter *to);extern ssize_t cifs_user_writev (struct kiocb *iocb,struct iov_iter *from); +extern ssize_t cifs_direct_writev (struct kiocb *iocb,struct iov_iter *from);extern ssize_t cifs_strict_writev (struct kiocb *iocb,struct iov_iter *from);extern int cifs_lock (struct file *,int ,struct file_lock *);extern int cifs_fsync (struct file *,loff_t ,loff_t ,int );
fs/cifs/file.c +307 -15
382 lines not on this page · raw message
Signature · 1 line
2.7.4
RFC PATCH 9/9 Introduce cache=rdma moutning option
In reply to Long LiFrom: Long Li <[email protected]>
When cache=rdma is enabled on mount options, CIFS do not allocate internal data buffer pages for I/O, data is read/writen directly to user memory via RDMA.
--- fs/cifs/cifs_fs_sb.h | 2 ++ fs/cifs/cifsglob.h | 1 + fs/cifs/connect.c | 9 +++++++++ fs/cifs/dir.c | 5 +++++ fs/cifs/file.c | 4 ++++ fs/cifs/inode.c | 4 +++- 6 files changed, 24 insertions(+), 1 deletion(-)
fs/cifs/cifs_fs_sb.h +2 -0
@@ -51,6 +51,8 @@*/ #define CIFS_MOUNT_UID_FROM_ACL 0x2000000/* try to get UID via special SID */ +#define CIFS_MOUNT_DIRECT_RDMA 0x4000000 +struct cifs_sb_info {struct rb_root tlink_tree;spinlock_t tlink_tree_lock;
fs/cifs/cifsglob.h +1 -0
@@ -518,6 +518,7 @@ struct smb_vol { bool server_ino:1; /* use inode numbers from server ie UniqueId */ bool direct_io:1; bool strict_io:1; /* strict cache behavior */ + bool rdma_io:1; bool remap:1; /* set to remap seven reserved chars in filenames */ bool sfu_remap:1; /* remap seven reserved chars ala SFU */ bool posix_paths:1; /* unset to not ask for posix pathnames. */
fs/cifs/connect.c +9 -0
@@ -282,6 +282,7 @@ enum { Opt_cache_loose, Opt_cache_strict, Opt_cache_none, + Opt_cache_rdma, Opt_cache_err };
@@ -289,6 +290,7 @@ static const match_table_t cifs_cacheflavor_tokens = { { Opt_cache_loose, "loose" }, { Opt_cache_strict, "strict" }, { Opt_cache_none, "none" }, + { Opt_cache_rdma, "rdma" }, { Opt_cache_err, NULL } };
@@ -1128,6 +1130,9 @@ cifs_parse_cache_flavor(char *value, struct smb_vol *vol) vol->direct_io = true; vol->strict_io = false;break ; +case Opt_cache_rdma: + vol->rdma_io = true; +break ;default :cifs_dbg (VFS ,"bad cache= option: %s\n" , value);return 1 ;
@@ -3612,6 +3617,10 @@ int cifs_setup_cifs_sb(struct smb_vol *pvolume_info,cifs_dbg (FYI ,"mounting share using direct i/o\n" ); cifs_sb->mnt_cifs_flags |=CIFS_MOUNT_DIRECT_IO ; } +if (pvolume_info->rdma_io) { +cifs_dbg (VFS ,"mounting share using rdma i/o\n" ); + cifs_sb->mnt_cifs_flags |=CIFS_MOUNT_DIRECT_RDMA ; + }if (pvolume_info->mfsymlinks) {if (pvolume_info->sfu_emul) {/*
fs/cifs/dir.c +5 -0
@@ -557,6 +557,11 @@ cifs_atomic_open(struct inode *inode, struct dentry *direntry, file->f_op = &cifs_file_direct_ops; } +if (file->f_flags &O_DIRECT && +CIFS_SB (inode->i_sb)->mnt_cifs_flags &CIFS_MOUNT_DIRECT_RDMA ) + file->f_op = &cifs_file_direct_rdma_ops; + + file_info =cifs_new_fileinfo (&fid, file, tlink, oplock);if (file_info ==NULL ) {if (server->ops->close)
fs/cifs/file.c +4 -0
@@ -492,6 +492,10 @@ int cifs_open(struct inode *inode, struct file *file) file->f_op = &cifs_file_direct_ops; } +if (file->f_flags &O_DIRECT && + cifs_sb->mnt_cifs_flags &CIFS_MOUNT_DIRECT_RDMA ) + file->f_op = &cifs_file_direct_rdma_ops; +if (server->oplocks) oplock =REQ_OPLOCK ;else
fs/cifs/inode.c +3 -1
@@ -44,7 +44,9 @@ static void cifs_set_ops(struct inode *inode)switch (inode->i_mode &S_IFMT ) {case S_IFREG : inode->i_op = &cifs_file_inode_ops; -if (cifs_sb->mnt_cifs_flags &CIFS_MOUNT_DIRECT_IO ) { +if (cifs_sb->mnt_cifs_flags &CIFS_MOUNT_DIRECT_RDMA ) + inode->i_fop = &cifs_file_direct_rdma_ops; +else if (cifs_sb->mnt_cifs_flags &CIFS_MOUNT_DIRECT_IO ) {if (cifs_sb->mnt_cifs_flags &CIFS_MOUNT_NO_BRL ) inode->i_fop = &cifs_file_direct_nobrl_ops;else
Signature · 1 line
2.7.4
Re: [RFC PATCH 09/09] Introduce cache=rdma moutning option
Quoted 4 lines · Long Li
From: Long Li <[email protected]>
When cache=rdma is enabled on mount options, CIFS do not allocate internal data buffer pages for I/O, data is read/writen directly to user memory via RDMA.
I don't think this should be an option. For direct I/O without signing or encryption CIFS should always use get_user_pages, with or without RDMA.
RE: [RFC PATCH 09/09] Introduce cache=rdma moutning option
Quoted 11 lines
Subject: Re: [RFC PATCH 09/09] Introduce cache=rdma moutning option
Quoted 4 lines · Long Li
From: Long Li <[email protected]>
When cache=rdma is enabled on mount options, CIFS do not allocate internal data buffer pages for I/O, data is read/writen directly to user
memory via RDMA.
I don't think this should be an option. For direct I/O without signing or encryption CIFS should always use
get_user_pages, with or without RDMA.
Yes this should be done for all transport. If there are no objections, I'll send patches to change this.
Re: [RFC PATCH 09/09] Introduce cache=rdma moutning option
Quoted 5 lines · Long Li via samba-technical
Quoted 5 lines
Subject: Re: [RFC PATCH 09/09] Introduce cache=rdma moutning option
Quoted 2 lines · Long Li
From: Long Li <[email protected]>
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
When cache=rdma is enabled on mount options, CIFS do not allocate internal data buffer pages for I/O, data is read/writen directly to user
memory via RDMA.
I don't think this should be an option. For direct I/O without signing or encryption CIFS should always use
get_user_pages, with or without RDMA.Yes this should be done for all transport. If there are no objections, I'll send patches to change this.
Would this help/change performance much?
Signature · 3 lines
Thanks, Steve
RE: [RFC PATCH 09/09] Introduce cache=rdma moutning option
Quoted 14 lines
Subject: Re: [RFC PATCH 09/09] Introduce cache=rdma moutning option
Quoted 10 lines · Long Li via samba-technical
Quoted 10 lines
Subject: Re: [RFC PATCH 09/09] Introduce cache=rdma moutning option
Quoted 5 lines · Long Li
From: Long Li <[email protected]>
When cache=rdma is enabled on mount options, CIFS do not allocate internal data buffer pages for I/O, data is read/writen directly to user
memory via RDMA.
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
I don't think this should be an option. For direct I/O without signing or encryption CIFS should always use
get_user_pages, with orwithout RDMA.
Quoted 2 lines
Yes this should be done for all transport. If there are no objections, I'll send
patches to change this.
Would this help/change performance much?
On RDMA, it helps with I/O latency and reduces CPU usage on certain I/O patterns.
But I haven't tested on TCP. Maybe it will help a little bit.
Quoted 7 lines
--
Thanks,Steve
Tom Talpey
reviewerRe: [RFC PATCH 09/09] Introduce cache=rdma moutning option
Quoted 18 lines · Long Li
Quoted 18 lines
Subject: Re: [RFC PATCH 09/09] Introduce cache=rdma moutning option
Quoted 5 lines · Long Li via samba-technical
Quoted 5 lines
Subject: Re: [RFC PATCH 09/09] Introduce cache=rdma moutning option
Quoted 2 lines · Long Li
From: Long Li <[email protected]>
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
When cache=rdma is enabled on mount options, CIFS do not allocate internal data buffer pages for I/O, data is read/writen directly to user
memory via RDMA.
I don't think this should be an option. For direct I/O without signing or encryption CIFS should always use
get_user_pages, with orwithout RDMA.
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Yes this should be done for all transport. If there are no objections, I'll send
patches to change this.
Would this help/change performance much?
On RDMA, it helps with I/O latency and reduces CPU usage on certain I/O patterns.
But I haven't tested on TCP. Maybe it will help a little bit.
Well, when the application requests direct i/o on a TCP connection, you definitely don't want to cache it! So even if the performance is different, correctness would dictate doing this.
You probably don't need to pin the buffer in the TCP case, which might be worth avoiding.
Tom.