server-side support for "inter" SSC copy
Subsystems (2)
v5 cover server-side support for "inter" SSC copy
This patch series adds support for NFSv4.2 copy offload feature allowing copy between two different NFS servers.
This functionality depends on the VFS ability to support generic copy_file_range() where a copy is done between an NFS file and a local file system. This is on top of Amir's VFS generic copy offload series.
This feature is enabled by the kernel module parameter -- inter_copy_offload_enable -- and by default is disabled. There is also a kernel compile configuration of NFSD_V4_2_INTER_SSC that adds dependency on the NFS client side functions called from the server.
These patches work on top of existing async intra copy offload patches. For the "inter" SSC, the implementation only supports asynchronous inter copy.
On the source server, upon receiving a COPY_NOTIFY, it generate a unique stateid that's kept in the global list. Upon receiving a READ with a stateid, the code checks the normal list of open stateid and now additionally, it'll check the copy state list as well before deciding to either fail with BAD_STATEID or find one that matches. The stored stateid is only valid to be used for the first time with a choosen lease period (90s currently). When the source server received an OFFLOAD_CANCEL, it will remove the stateid from the global list. Otherwise, the copy stateid is removed upon the removal of its "parent" stateid (open/lock/delegation stateid).
On the destination server, upon receiving a COPY request, the server establishes the necessary clientid/session with the source server. It calls into the NFS client code to establish the necessary open stateid, filehandle, file description (without doing an NFS open). Then the server calls into the copy_file_range() to preform the copy where the source file will issue NFS READs and then do local file system writes (this depends on the VFS ability to do cross device copy_file_range().
v5 fixes:
-- Remove unused nl4_ulr/nl4_name types
-- default to warn_on_once
-- move 2 chucks of use of stid into previous patch
-- in the idr store stateid ptr instead of copy or copy_notify
-- add sc_type to distinguish copy and copy_notify stateid (for that need to re-work data structures to encapsulate stateid_t with sc_type) -- rework how initialization of copy and copy_notify stateid is done w respect to adding it to the global list
-- during compound check to allow for stale_fh make sure that saved fh was set – done
-- separated the copy_notify xdr code into its own patch
-- need to insure the parent stateid is valid for copy_notify accesses. inpreprocess_op take a reference on the parent stateid and hold it. on theput_stid, check if there are any copy_notifies associated and account for them
during decrementing a reference count. add to the laundromat, traversing copy
notify list of the client pointer, if entry is passed a lease period, then
remove it.
Olga Kornievskaia (9):
NFSD fill-in netloc4 structure
NFSD add ca_source_server<> to COPY
NFSD return nfs4_stid in nfs4_preprocess_stateid_op NFSD COPY_NOTIFY xdr
NFSD add COPY_NOTIFY operation
NFSD check stateids against copy stateids
NFSD generalize nfsd4_compound_state flag names
NFSD: allow inter server COPY to have a STALE source server fh NFSD add nfs4 inter ssc to nfsd4_copy
fs/nfsd/Kconfig | 10 ++ fs/nfsd/nfs4proc.c | 453 +++++++++++++++++++++++++++++++++++++++++++++++----- fs/nfsd/nfs4state.c | 179 ++++++++++++++++++--- fs/nfsd/nfs4xdr.c | 154 +++++++++++++++++- fs/nfsd/nfsd.h | 32 ++++ fs/nfsd/nfsfh.h | 5 +- fs/nfsd/nfssvc.c | 6 + fs/nfsd/state.h | 36 ++++- fs/nfsd/xdr4.h | 39 +++-- 9 files changed, 840 insertions(+), 74 deletions(-)
Signature · 1 line
1.8.3.1
PATCH v5 1/9 NFSD fill-in netloc4 structure
From: Olga Kornievskaia <[email protected]>
nfs.4 defines nfs42_netaddr structure that represents netloc4.
Populate needed fields from the sockaddr structure.
This will be used by flexfiles and 4.2 inter copy
--- fs/nfsd/nfsd.h | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+)
fs/nfsd/nfsd.h +32 -0
@@ -19,6 +19,7 @@#include <linux/sunrpc/svc.h> #include <linux/sunrpc/svc_xprt.h> #include <linux/sunrpc/msg_prot.h> +#include <linux/sunrpc/addr.h> #include <uapi/linux/nfsd/debug.h>
@@ -386,6 +387,37 @@ static inline bool nfsd4_spo_must_allow(struct svc_rqst *rqstp)extern const u32 nfsd_suppattrs[3 ][3 ]; +static inline u32 nfsd4_set_netaddr (struct sockaddr *addr, +struct nfs42_netaddr *netaddr) +{ +struct sockaddr_in *sin = (struct sockaddr_in *)addr; +struct sockaddr_in6 *sin6 = (struct sockaddr_in6 *)addr; +unsigned int port; +size_t ret_addr, ret_port; + +switch (addr->sa_family) { +case AF_INET : + port =ntohs (sin->sin_port); +sprintf (netaddr->netid,"tcp" ); + netaddr->netid_len =3 ; +break ; +case AF_INET6 : + port =ntohs (sin6->sin6_port); +sprintf (netaddr->netid,"tcp6" ); + netaddr->netid_len =4 ; +break ; +default : +return nfserr_inval; + } + ret_addr =rpc_ntop (addr, netaddr->addr,sizeof (netaddr->addr)); + ret_port =snprintf (netaddr->addr + ret_addr, +RPCBIND_MAXUADDRLEN +1 - ret_addr, +".%u.%u" , port >>8 , port &0xff ); +WARN_ON (ret_port >=RPCBIND_MAXUADDRLEN +1 - ret_addr); + netaddr->addr_len = ret_addr + ret_port; +return 0 ; +} +static inline bool bmval_is_subset (const u32 *bm1,const u32 *bm2) { return !((bm1[0] & ~bm2[0]) ||
Signature · 1 line
1.8.3.1
kbuild test robot
build botRe: [PATCH v5 1/9] NFSD fill-in netloc4 structure
Thank you for the patch! Yet something to improve:
Hi Olga,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on nfsd/nfsd-next]
[cannot apply to v5.3-rc3 next-20190809]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Olga-Kornievskaia/server-side-support-for-inter-SSC-copy/20190811-120551 base: git://linux-nfs.org/~bfields/linux.git nfsd-next config: x86_64-lkp (attached as .config) compiler: gcc-7 (Debian 7.4.0-10) 7.4.0 reproduce: # save the attached .config to linux build tree make ARCH=x86_64
If you fix the issue, kindly add following tag
All error/warnings (new ones prefixed by >>):
In file included from fs/nfsd/state.h:42:0,
from fs/nfsd/trace.h:128,
from fs/nfsd/trace.c:3:
>> fs/nfsd/nfsd.h:391:16: warning: 'struct nfs42_netaddr' declared inside parameter list will not be visible outside of this definition or declaration
struct nfs42_netaddr *netaddr)
^~~~~~~~~~~~~
fs/nfsd/nfsd.h: In function 'nfsd4_set_netaddr':
>> fs/nfsd/nfsd.h:401:18: error: dereferencing pointer to incomplete type 'struct nfs42_netaddr'
sprintf(netaddr->netid, "tcp");
^~vim +401 fs/nfsd/nfsd.h
389
390 static inline u32 nfsd4_set_netaddr(struct sockaddr *addr,
> 391 struct nfs42_netaddr *netaddr)
392 {
393 struct sockaddr_in *sin = (struct sockaddr_in *)addr;
394 struct sockaddr_in6 *sin6 = (struct sockaddr_in6 *)addr;
395 unsigned int port;
396 size_t ret_addr, ret_port;
397
398 switch (addr->sa_family) {
399 case AF_INET:
400 port = ntohs(sin->sin_port);
> 401 sprintf(netaddr->netid, "tcp"); 402 netaddr->netid_len = 3;
403 break;
404 case AF_INET6:
405 port = ntohs(sin6->sin6_port);
406 sprintf(netaddr->netid, "tcp6");
407 netaddr->netid_len = 4;
408 break;
409 default:
410 return nfserr_inval;
411 }
412 ret_addr = rpc_ntop(addr, netaddr->addr, sizeof(netaddr->addr));
413 ret_port = snprintf(netaddr->addr + ret_addr,
414 RPCBIND_MAXUADDRLEN + 1 - ret_addr,
415 ".%u.%u", port >> 8, port & 0xff);
416 WARN_ON(ret_port >= RPCBIND_MAXUADDRLEN + 1 - ret_addr);
417 netaddr->addr_len = ret_addr + ret_port;
418 return 0;
419 }
420
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Re: [PATCH v5 1/9] NFSD fill-in netloc4 structure
Hi Bruce,
I believe I'm getting this because as it has been before NFSD patches depend on client-side patches (or specifically) it needed
https://patchwork.kernel.org/patch/10649667/
I should have included the patch in both the client and server patch series. How should I correct this?
Also, in general (to run things) nfsd patches need the client side patches to be functional destination server (destination server nfs client piece).
Quoted 71 lines · kbuild test robot
Hi Olga,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on nfsd/nfsd-next]
[cannot apply to v5.3-rc3 next-20190809]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]url: https://github.com/0day-ci/linux/commits/Olga-Kornievskaia/server-side-support-for-inter-SSC-copy/20190811-120551 base: git://linux-nfs.org/~bfields/linux.git nfsd-next config: x86_64-lkp (attached as .config) compiler: gcc-7 (Debian 7.4.0-10) 7.4.0 reproduce: # save the attached .config to linux build tree make ARCH=x86_64If you fix the issue, kindly add following tag
Reported-by kbuild test robot <[email protected]>All error/warnings (new ones prefixed by >>):
In file included from fs/nfsd/state.h:42:0, from fs/nfsd/trace.h:128, from fs/nfsd/trace.c:3:Quoted 1 line
Quoted 1 line
fs/nfsd/nfsd.h:391:16: warning: 'structnfs42_netaddr' declared inside parameter list will not be visible outside of this definition or declarationstruct
nfs42_netaddr*netaddr)
^~~~~~~~~~~~~
fs/nfsd/nfsd.h: In function 'nfsd4_set_netaddr':Quoted 1 line
Quoted 1 line
fs/nfsd/nfsd.h:401:18: error: dereferencing pointer to incomplete type 'structnfs42_netaddr'sprintf(netaddr->netid, "tcp"); ^~vim +401
fs/nfsd/nfsd.h389
390 static inline u32 nfsd4_set_netaddr(struct sockaddr *addr,Quoted 1 line
391 struct
nfs42_netaddr*netaddr)392 { 393 struct sockaddr_in *sin = (struct sockaddr_in *)addr; 394 struct sockaddr_in6 *sin6 = (struct sockaddr_in6 *)addr; 395 unsigned int port; 396 size_t ret_addr, ret_port; 397 398 switch (addr->sa_family) { 399 case AF_INET: 400 port = ntohs(sin->sin_port);Quoted 1 line
401
sprintf(netaddr->netid, "tcp");402 netaddr->netid_len = 3; 403 break; 404 case AF_INET6: 405 port = ntohs(sin6->sin6_port); 406 sprintf(netaddr->netid, "tcp6"); 407 netaddr->netid_len = 4; 408 break; 409 default: 410 return nfserr_inval; 411 } 412 ret_addr = rpc_ntop(addr, netaddr->addr, sizeof(netaddr->addr)); 413 ret_port = snprintf(netaddr->addr + ret_addr, 414 RPCBIND_MAXUADDRLEN + 1 - ret_addr, 415 ".%u.%u", port >> 8, port & 0xff); 416 WARN_ON(ret_port >= RPCBIND_MAXUADDRLEN + 1 - ret_addr); 417 netaddr->addr_len = ret_addr + ret_port; 418 return 0; 419 } 420---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Re: [PATCH v5 1/9] NFSD fill-in netloc4 structure
Quoted 6 lines · Olga Kornievskaia
Hi Bruce,
I believe I'm getting this because as it has been before NFSD patches depend on client-side patches (or specifically) it needed
https://patchwork.kernel.org/patch/10649667/
Quoted 6 lines
I should have included the patch in both the client and server patch series. How should I correct this?
Also, in general (to run things) nfsd patches need the client side patches to be functional destination server (destination server nfs client piece).
I think I'd just sent it all as one series. You can note in the cover letter that the first X patches are for the client and the rest for the server, if you'd like.
--b.
Quoted 72 lines · kbuild test robot
Quoted 71 lines
Hi Olga,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on nfsd/nfsd-next]
[cannot apply to v5.3-rc3 next-20190809]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]url: https://github.com/0day-ci/linux/commits/Olga-Kornievskaia/server-side-support-for-inter-SSC-copy/20190811-120551 base: git://linux-nfs.org/~bfields/linux.git nfsd-next config: x86_64-lkp (attached as .config) compiler: gcc-7 (Debian 7.4.0-10) 7.4.0 reproduce: # save the attached .config to linux build tree make ARCH=x86_64If you fix the issue, kindly add following tag
Reported-by kbuild test robot <[email protected]>All error/warnings (new ones prefixed by >>):
In file included from fs/nfsd/state.h:42:0, from fs/nfsd/trace.h:128, from fs/nfsd/trace.c:3:Quoted 1 line
Quoted 1 line
fs/nfsd/nfsd.h:391:16: warning: 'structnfs42_netaddr' declared inside parameter list will not be visible outside of this definition or declarationstruct
nfs42_netaddr*netaddr)
^~~~~~~~~~~~~
fs/nfsd/nfsd.h: In function 'nfsd4_set_netaddr':Quoted 1 line
Quoted 1 line
fs/nfsd/nfsd.h:401:18: error: dereferencing pointer to incomplete type 'structnfs42_netaddr'sprintf(netaddr->netid, "tcp"); ^~vim +401
fs/nfsd/nfsd.h389
390 static inline u32 nfsd4_set_netaddr(struct sockaddr *addr,Quoted 1 line
391 struct
nfs42_netaddr*netaddr)392 { 393 struct sockaddr_in *sin = (struct sockaddr_in *)addr; 394 struct sockaddr_in6 *sin6 = (struct sockaddr_in6 *)addr; 395 unsigned int port; 396 size_t ret_addr, ret_port; 397 398 switch (addr->sa_family) { 399 case AF_INET: 400 port = ntohs(sin->sin_port);Quoted 1 line
401
sprintf(netaddr->netid, "tcp");402 netaddr->netid_len = 3; 403 break; 404 case AF_INET6: 405 port = ntohs(sin6->sin6_port); 406 sprintf(netaddr->netid, "tcp6"); 407 netaddr->netid_len = 4; 408 break; 409 default: 410 return nfserr_inval; 411 } 412 ret_addr = rpc_ntop(addr, netaddr->addr, sizeof(netaddr->addr)); 413 ret_port = snprintf(netaddr->addr + ret_addr, 414 RPCBIND_MAXUADDRLEN + 1 - ret_addr, 415 ".%u.%u", port >> 8, port & 0xff); 416 WARN_ON(ret_port >= RPCBIND_MAXUADDRLEN + 1 - ret_addr); 417 netaddr->addr_len = ret_addr + ret_port; 418 return 0; 419 } 420---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
PATCH v5 2/9 NFSD add ca_source_server<> to COPY
In reply to Olga KornievskaiaDecode the ca_source_server list that's sent but only use the first one. Presence of non-zero list indicates an "inter" copy.
--- fs/nfsd/nfs4xdr.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++++-- fs/nfsd/xdr4.h | 12 ++++++----- 2 files changed, 69 insertions(+), 7 deletions(-)
fs/nfsd/nfs4xdr.c +62 -2
@@ -40,6 +40,7 @@#include <linux/utsname.h> #include <linux/pagemap.h> #include <linux/sunrpc/svcauth_gss.h> +#include <linux/sunrpc/addr.h> #include "idmap.h" #include "acl.h"
@@ -1732,11 +1733,47 @@ static __be32 nfsd4_decode_reclaim_complete(struct nfsd4_compoundargs *argp, strDECODE_TAIL ; } +static __be32 nfsd4_decode_nl4_server (struct nfsd4_compoundargs *argp, +struct nl4_server *ns) +{ +DECODE_HEAD ; +struct nfs42_netaddr *naddr; + +READ_BUF (4 ); + ns->nl4_type =be32_to_cpup (p++); + +/* currently support for 1 inter-server source server */ +switch (ns->nl4_type) { +case NL4_NETADDR : + naddr = &ns->u.nl4_addr; + +READ_BUF (4 ); + naddr->netid_len =be32_to_cpup (p++); +if (naddr->netid_len >RPCBIND_MAXNETIDLEN ) + goto xdr_error; + +READ_BUF (naddr->netid_len +4 );/* 4 for uaddr len */ +COPYMEM (naddr->netid, naddr->netid_len); + + naddr->addr_len =be32_to_cpup (p++); +if (naddr->addr_len >RPCBIND_MAXUADDRLEN ) + goto xdr_error; + +READ_BUF (naddr->addr_len); +COPYMEM (naddr->addr, naddr->addr_len); +break ; +default : + goto xdr_error; + } +DECODE_TAIL ; +} +static __be32 nfsd4_decode_copy (struct nfsd4_compoundargs *argp,struct nfsd4_copy *copy) {DECODE_HEAD ; -unsigned int tmp; +struct nl4_server *ns_dummy; +int i, count; status =nfsd4_decode_stateid (argp, ©->cp_src_stateid);if (status)
@@ -1751,8 +1788,31 @@ static __be32 nfsd4_decode_reclaim_complete(struct nfsd4_compoundargs *argp, str p =xdr_decode_hyper (p, ©->cp_count); p++;/* ca_consecutive: we always do consecutive copies */ copy->cp_synchronous =be32_to_cpup (p++); - tmp =be32_to_cpup (p);/* Source server list not supported */ + count =be32_to_cpup (p++); + + copy->cp_intra = false; +if (count ==0 ) {/* intra-server copy */ + copy->cp_intra = true; + goto intra; + } +/* decode all the supplied server addresses but use first */ + status =nfsd4_decode_nl4_server (argp, ©->cp_src); +if (status) +return status; + + ns_dummy =kmalloc (sizeof (struct nl4_server ),GFP_KERNEL ); +if (ns_dummy ==NULL ) +return nfserrno (-ENOMEM ); +for (i =0 ; i < count -1 ; i++) { + status =nfsd4_decode_nl4_server (argp, ns_dummy); +if (status) { +kfree (ns_dummy); +return status; + } + } +kfree (ns_dummy); +intra:DECODE_TAIL ; }
fs/nfsd/xdr4.h +7 -5
@@ -519,11 +519,13 @@ struct nfsd42_write_res {struct nfsd4_copy {/* request */ -stateid_t cp_src_stateid; -stateid_t cp_dst_stateid; -u64 cp_src_pos; -u64 cp_dst_pos; -u64 cp_count; +stateid_t cp_src_stateid; +stateid_t cp_dst_stateid; +u64 cp_src_pos; +u64 cp_dst_pos; +u64 cp_count; +struct nl4_server cp_src; +bool cp_intra;/* both */ bool cp_synchronous;
Signature · 1 line
1.8.3.1
kbuild test robot
build botRe: [PATCH v5 2/9] NFSD add ca_source_server<> to COPY
Thank you for the patch! Yet something to improve:
Hi Olga,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on nfsd/nfsd-next]
[cannot apply to v5.3-rc3 next-20190809]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Olga-Kornievskaia/server-side-support-for-inter-SSC-copy/20190811-120551 base: git://linux-nfs.org/~bfields/linux.git nfsd-next config: x86_64-lkp (attached as .config) compiler: gcc-7 (Debian 7.4.0-10) 7.4.0 reproduce: # save the attached .config to linux build tree make ARCH=x86_64
If you fix the issue, kindly add following tag
All errors (new ones prefixed by >>):
In file included from fs/nfsd/nfsctl.c:23:0:
fs/nfsd/nfsd.h:391:16: warning: 'struct nfs42_netaddr' declared inside parameter list will not be visible outside of this definition or declaration
struct nfs42_netaddr *netaddr)
^~~~~~~~~~~~~
fs/nfsd/nfsd.h: In function 'nfsd4_set_netaddr':
fs/nfsd/nfsd.h:401:18: error: dereferencing pointer to incomplete type 'struct nfs42_netaddr'
sprintf(netaddr->netid, "tcp");
^~
In file included from fs/nfsd/pnfs.h:10:0,
from fs/nfsd/nfsctl.c:27:
fs/nfsd/xdr4.h: At top level:
>> fs/nfsd/xdr4.h:527:20: error: field 'cp_src' has incomplete type
struct nl4_server cp_src;
^~~~~~
--
In file included from fs/nfsd/state.h:42:0,
from fs/nfsd/xdr4.h:40,
from fs/nfsd/nfs4xdr.c:47:
fs/nfsd/nfsd.h:391:16: warning: 'struct nfs42_netaddr' declared inside parameter list will not be visible outside of this definition or declaration
struct nfs42_netaddr *netaddr)
^~~~~~~~~~~~~
fs/nfsd/nfsd.h: In function 'nfsd4_set_netaddr':
fs/nfsd/nfsd.h:401:18: error: dereferencing pointer to incomplete type 'struct nfs42_netaddr'
sprintf(netaddr->netid, "tcp");
^~
In file included from fs/nfsd/nfs4xdr.c:47:0:
fs/nfsd/xdr4.h: At top level:
>> fs/nfsd/xdr4.h:527:20: error: field 'cp_src' has incomplete type
struct nl4_server cp_src;
^~~~~~
fs/nfsd/nfs4xdr.c: In function 'nfsd4_decode_nl4_server':
>> fs/nfsd/nfs4xdr.c:1743:4: error: dereferencing pointer to incomplete type 'struct nl4_server'
ns->nl4_type = be32_to_cpup(p++);
^~
>> fs/nfsd/nfs4xdr.c:1747:7: error: 'NL4_NETADDR' undeclared (first use in this function); did you mean 'RTM_GETADDR'?
case NL4_NETADDR:
^~~~~~~~~~~
RTM_GETADDR
fs/nfsd/nfs4xdr.c:1747:7: note: each undeclared identifier is reported only once for each function it appears in
>> fs/nfsd/nfs4xdr.c:1751:8: error: dereferencing pointer to incomplete type 'struct nfs42_netaddr'
naddr->netid_len = be32_to_cpup(p++);
^~
fs/nfsd/nfs4xdr.c: In function 'nfsd4_decode_copy':
>> fs/nfsd/nfs4xdr.c:1804:28: error: invalid application of 'sizeof' to incomplete type 'struct nl4_server'
ns_dummy = kmalloc(sizeof(struct nl4_server), GFP_KERNEL);
^~~~~~vim +/cp_src +527 fs/nfsd/xdr4.h
519
520 struct nfsd4_copy {
521 /* request */
522 stateid_t cp_src_stateid;
523 stateid_t cp_dst_stateid;
524 u64 cp_src_pos;
525 u64 cp_dst_pos;
526 u64 cp_count;
> 527 struct nl4_server cp_src;
528 bool cp_intra;
529
530 /* both */
531 bool cp_synchronous;
532
533 /* response */
534 struct nfsd42_write_res cp_res;
535
536 /* for cb_offload */
537 struct nfsd4_callback cp_cb;
538 __be32 nfserr;
539 struct knfsd_fh fh;
540
541 struct nfs4_client *cp_clp;
542
543 struct file *file_src;
544 struct file *file_dst;
545
546 stateid_t cp_stateid;
547
548 struct list_head copies;
549 struct task_struct *copy_task;
550 refcount_t refcount;
551 bool stopped;
552 };
553
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
kbuild test robot
build botRe: [PATCH v5 2/9] NFSD add ca_source_server<> to COPY
In reply to Olga KornievskaiaThank you for the patch! Perhaps something to improve:
Hi Olga,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on nfsd/nfsd-next]
[cannot apply to v5.3-rc3 next-20190809]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Olga-Kornievskaia/server-side-support-for-inter-SSC-copy/20190811-120551 base: git://linux-nfs.org/~bfields/linux.git nfsd-next reproduce: # apt-get install sparse # sparse version: v0.6.1-rc1-7-g2b96cd8-dirty make ARCH=x86_64 allmodconfig make C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__'
If you fix the issue, kindly add following tag
sparse warnings: (new ones prefixed by >>)
include/linux/sched.h:609:43: sparse: sparse: bad integer constant expression include/linux/sched.h:609:73: sparse: sparse: invalid named zero-width bitfield value' include/linux/sched.h:610:43: sparse: sparse: bad integer constant expression include/linux/sched.h:610:67: sparse: sparse: invalid named zero-width bitfield bucket_id' fs/nfsd/nfs4xdr.c:1743:11: sparse: sparse: using member 'nl4_type' in incomplete struct nl4_server fs/nfsd/nfs4xdr.c:1746:19: sparse: sparse: using member 'nl4_type' in incomplete struct nl4_server fs/nfsd/nfs4xdr.c:1747:14: sparse: sparse: undefined identifier 'NL4_NETADDR' fs/nfsd/nfs4xdr.c:1748:28: sparse: sparse: using member 'u' in incomplete struct nl4_server fs/nfsd/nfs4xdr.c:1751:22: sparse: sparse: using member 'netid_len' in incomplete struct nfs42_netaddr fs/nfsd/nfs4xdr.c:1752:26: sparse: sparse: using member 'netid_len' in incomplete struct nfs42_netaddr fs/nfsd/nfs4xdr.c:1755:17: sparse: sparse: using member 'netid_len' in incomplete struct nfs42_netaddr fs/nfsd/nfs4xdr.c:1755:17: sparse: sparse: using member 'netid_len' in incomplete struct nfs42_netaddr fs/nfsd/nfs4xdr.c:1755:17: sparse: sparse: using member 'netid_len' in incomplete struct nfs42_netaddr fs/nfsd/nfs4xdr.c:1756:17: sparse: sparse: using member 'netid' in incomplete struct nfs42_netaddr fs/nfsd/nfs4xdr.c:1756:17: sparse: sparse: using member 'netid_len' in incomplete struct nfs42_netaddr fs/nfsd/nfs4xdr.c:1758:22: sparse: sparse: using member 'addr_len' in incomplete struct nfs42_netaddr fs/nfsd/nfs4xdr.c:1759:26: sparse: sparse: using member 'addr_len' in incomplete struct nfs42_netaddr fs/nfsd/nfs4xdr.c:1762:17: sparse: sparse: using member 'addr_len' in incomplete struct nfs42_netaddr fs/nfsd/nfs4xdr.c:1762:17: sparse: sparse: using member 'addr_len' in incomplete struct nfs42_netaddr fs/nfsd/nfs4xdr.c:1762:17: sparse: sparse: using member 'addr_len' in incomplete struct nfs42_netaddr fs/nfsd/nfs4xdr.c:1763:17: sparse: sparse: using member 'addr' in incomplete struct nfs42_netaddr fs/nfsd/nfs4xdr.c:1763:17: sparse: sparse: using member 'addr_len' in incomplete struct nfs42_netaddr
>> fs/nfsd/nfs4xdr.c:1747:14: sparse: sparse: incompatible types for 'case' statement
fs/nfsd/nfs4xdr.c:1747:14: sparse: sparse: Expected constant expression in case statement
vim +/case +1747 fs/nfsd/nfs4xdr.c
1735
1736 static __be32 nfsd4_decode_nl4_server(struct nfsd4_compoundargs *argp,
1737 struct nl4_server *ns)
1738 {
1739 DECODE_HEAD;
1740 struct nfs42_netaddr *naddr;
1741
1742 READ_BUF(4);
1743 ns->nl4_type = be32_to_cpup(p++);
1744
1745 /* currently support for 1 inter-server source server */
1746 switch (ns->nl4_type) {
> 1747 case NL4_NETADDR: 1748 naddr = &ns->u.nl4_addr; 1749 1750 READ_BUF(4); 1751 naddr->netid_len = be32_to_cpup(p++); 1752 if (naddr->netid_len > RPCBIND_MAXNETIDLEN) 1753 goto xdr_error; 1754 1755 READ_BUF(naddr->netid_len + 4); /* 4 for uaddr len */ > 1756 COPYMEM(naddr->netid, naddr->netid_len);
1757
1758 naddr->addr_len = be32_to_cpup(p++);
1759 if (naddr->addr_len > RPCBIND_MAXUADDRLEN)
1760 goto xdr_error;
1761
1762 READ_BUF(naddr->addr_len);
1763 COPYMEM(naddr->addr, naddr->addr_len);
1764 break;
1765 default:
1766 goto xdr_error;
1767 }
1768 DECODE_TAIL;
1769 }
1770
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
PATCH v5 3/9 NFSD return nfs4_stid in nfs4_preprocess_stateid_op
In reply to Olga KornievskaiaNeeded for copy to add nfs4_cp_state to the nfs4_stid.
--- fs/nfsd/nfs4proc.c | 24 ++++++++++++++---------- fs/nfsd/nfs4state.c | 11 ++++++++--- fs/nfsd/state.h | 3 ++- 3 files changed, 24 insertions(+), 14 deletions(-)
fs/nfsd/nfs4proc.c +14 -10
@@ -782,7 +782,8 @@ static __be32 nfsd4_do_lookupp(struct svc_rqst *rqstp, struct svc_fh *fh)/* check stateid */ status =nfs4_preprocess_stateid_op (rqstp, cstate, &cstate->current_fh, &read->rd_stateid,RD_STATE , - &read->rd_filp, &read->rd_tmp_file); + &read->rd_filp, &read->rd_tmp_file, +NULL );if (status) {dprintk ("NFSD: nfsd4_read: couldn't process stateid!\n" ); goto out;
@@ -954,7 +955,7 @@ static __be32 nfsd4_do_lookupp(struct svc_rqst *rqstp, struct svc_fh *fh)if (setattr->sa_iattr.ia_valid &ATTR_SIZE ) { status =nfs4_preprocess_stateid_op (rqstp, cstate, &cstate->current_fh, &setattr->sa_stateid, -WR_STATE ,NULL ,NULL ); +WR_STATE ,NULL ,NULL ,NULL );if (status) {dprintk ("NFSD: nfsd4_setattr: couldn't process stateid!\n" );return status;
@@ -1005,7 +1006,7 @@ static __be32 nfsd4_do_lookupp(struct svc_rqst *rqstp, struct svc_fh *fh)trace_nfsd_write_start (rqstp, &cstate->current_fh, write->wr_offset, cnt); status =nfs4_preprocess_stateid_op (rqstp, cstate, &cstate->current_fh, - stateid,WR_STATE , &filp,NULL ); + stateid,WR_STATE , &filp,NULL ,NULL );if (status) {dprintk ("NFSD: nfsd4_write: couldn't process stateid!\n" );return status;
@@ -1032,7 +1033,8 @@ static __be32 nfsd4_do_lookupp(struct svc_rqst *rqstp, struct svc_fh *fh)static __be32 nfsd4_verify_copy (struct svc_rqst *rqstp,struct nfsd4_compound_state *cstate,stateid_t *src_stateid,struct file **src, -stateid_t *dst_stateid,struct file **dst) +stateid_t *dst_stateid,struct file **dst, +struct nfs4_stid **stid) {__be32 status;
@@ -1040,14 +1042,16 @@ static __be32 nfsd4_do_lookupp(struct svc_rqst *rqstp, struct svc_fh *fh)return nfserr_nofilehandle; status =nfs4_preprocess_stateid_op (rqstp, cstate, &cstate->save_fh, - src_stateid,RD_STATE , src,NULL ); + src_stateid,RD_STATE , src,NULL , +NULL );if (status) {dprintk ("NFSD: %s: couldn't process src stateid!\n" , __func__); goto out; } status =nfs4_preprocess_stateid_op (rqstp, cstate, &cstate->current_fh, - dst_stateid,WR_STATE , dst,NULL ); + dst_stateid,WR_STATE , dst,NULL , + stid);if (status) {dprintk ("NFSD: %s: couldn't process dst stateid!\n" , __func__); goto out_put_src;
@@ -1078,7 +1082,7 @@ static __be32 nfsd4_do_lookupp(struct svc_rqst *rqstp, struct svc_fh *fh)__be32 status; status =nfsd4_verify_copy (rqstp, cstate, &clone->cl_src_stateid, &src, - &clone->cl_dst_stateid, &dst); + &clone->cl_dst_stateid, &dst,NULL );if (status) goto out;
@@ -1265,7 +1269,7 @@ static int nfsd4_do_async_copy(void *data) status =nfsd4_verify_copy (rqstp, cstate, ©->cp_src_stateid, ©->file_src, ©->cp_dst_stateid, - ©->file_dst); + ©->file_dst,NULL );if (status) goto out;
@@ -1351,7 +1355,7 @@ struct nfsd4_copy * status =nfs4_preprocess_stateid_op (rqstp, cstate, &cstate->current_fh, &fallocate->falloc_stateid, -WR_STATE , &file,NULL ); +WR_STATE , &file,NULL ,NULL );if (status != nfs_ok) {dprintk ("NFSD: nfsd4_fallocate: couldn't process stateid!\n" );return status;
@@ -1410,7 +1414,7 @@ struct nfsd4_copy * status =nfs4_preprocess_stateid_op (rqstp, cstate, &cstate->current_fh, &seek->seek_stateid, -RD_STATE , &file,NULL ); +RD_STATE , &file,NULL ,NULL );if (status) {dprintk ("NFSD: nfsd4_seek: couldn't process stateid!\n" );return status;
fs/nfsd/nfs4state.c +8 -3
@@ -5583,7 +5583,8 @@ static __be32 nfsd4_validate_stateid(struct nfs4_client *cl, stateid_t *stateid)__be32 nfs4_preprocess_stateid_op (struct svc_rqst *rqstp,struct nfsd4_compound_state *cstate,struct svc_fh *fhp, -stateid_t *stateid,int flags,struct file **filpp,bool *tmp_file) +stateid_t *stateid,int flags,struct file **filpp, +bool *tmp_file,struct nfs4_stid **cstid) {struct inode *ino =d_inode (fhp->fh_dentry);struct net *net =SVC_NET (rqstp);
@@ -5634,8 +5635,12 @@ static __be32 nfsd4_validate_stateid(struct nfs4_client *cl, stateid_t *stateid)if (!status && filpp) status =nfs4_check_file (rqstp, fhp, s, filpp, tmp_file, flags); out: -if (s) -nfs4_put_stid (s); +if (s) { +if (!status && cstid) + *cstid = s; +else +nfs4_put_stid (s); + }return status; }
fs/nfsd/state.h +2 -1
@@ -616,7 +616,8 @@ struct nfsd4_blocked_lock {extern __be32 nfs4_preprocess_stateid_op (struct svc_rqst *rqstp,struct nfsd4_compound_state *cstate,struct svc_fh *fhp, -stateid_t *stateid,int flags,struct file **filp,bool *tmp_file); +stateid_t *stateid,int flags,struct file **filp, +bool *tmp_file,struct nfs4_stid **cstid);__be32 nfsd4_lookup_stateid (struct nfsd4_compound_state *cstate,stateid_t *stateid,unsigned char typemask,struct nfs4_stid **s,struct nfsd_net *nn);
Signature · 1 line
1.8.3.1
PATCH v5 4/9 NFSD COPY_NOTIFY xdr
In reply to Olga Kornievskaia--- fs/nfsd/nfs4proc.c | 28 +++++++++++++++++ fs/nfsd/nfs4xdr.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++++-- fs/nfsd/xdr4.h | 13 ++++++++ 3 files changed, 129 insertions(+), 2 deletions(-)
fs/nfsd/nfs4proc.c +28 -0
@@ -1347,6 +1347,13 @@ struct nfsd4_copy * }static __be32 +nfsd4_copy_notify (struct svc_rqst *rqstp,struct nfsd4_compound_state *cstate, +union nfsd4_op_u *u) +{ +return nfserr_notsupp; +} + +static __be32 nfsd4_fallocate (struct svc_rqst *rqstp,struct nfsd4_compound_state *cstate,struct nfsd4_fallocate *fallocate,int flags) {
@@ -2299,6 +2306,21 @@ static inline u32 nfsd4_offload_status_rsize(struct svc_rqst *rqstp, 1 /* osr_complete<1> optional 0 for now */) * sizeof(__be32); } +static inline u32 nfsd4_copy_notify_rsize (struct svc_rqst *rqstp, +struct nfsd4_op *op) +{ +return (op_encode_hdr_size + +3 /* cnr_lease_time */ + +1 /* We support one cnr_source_server */ + +1 /* cnr_stateid seq */ + + op_encode_stateid_maxsz/* cnr_stateid */ + +1 /* num cnr_source_server*/ + +1 /* nl4_type */ + +1 /* nl4 size */ + +XDR_QUADLEN (NFS4_OPAQUE_LIMIT )/*nl4_loc + nl4_loc_sz */ ) + *sizeof (__be32); +} +#ifdef CONFIG_NFSD_PNFS static inline u32 nfsd4_getdeviceinfo_rsize (struct svc_rqst *rqstp,struct nfsd4_op *op) {
@@ -2723,6 +2745,12 @@ static inline u32 nfsd4_seek_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op) .op_name = "OP_OFFLOAD_CANCEL", .op_rsize_bop = nfsd4_only_status_rsize, }, + [OP_COPY_NOTIFY] = { + .op_func = nfsd4_copy_notify, + .op_flags = OP_MODIFIES_SOMETHING, + .op_name = "OP_COPY_NOTIFY", + .op_rsize_bop = nfsd4_copy_notify_rsize, + }, };/**
fs/nfsd/nfs4xdr.c +88 -2
120 lines not on this page · raw message
fs/nfsd/xdr4.h +13 -0
@@ -571,6 +571,18 @@ struct nfsd4_offload_status {u32 status; }; +struct nfsd4_copy_notify { +/* request */ +stateid_t cpn_src_stateid; +struct nl4_server cpn_dst; + +/* response */ +stateid_t cpn_cnr_stateid; +u64 cpn_sec; +u32 cpn_nsec; +struct nl4_server cpn_src; +}; +struct nfsd4_op {int opnum;const struct nfsd4_operation * opdesc;
@@ -630,6 +642,7 @@ struct nfsd4_op {struct nfsd4_clone clone;struct nfsd4_copy copy;struct nfsd4_offload_status offload_status; +struct nfsd4_copy_notify copy_notify;struct nfsd4_seek seek; } u;struct nfs4_replay * replay;
Signature · 1 line
1.8.3.1
kbuild test robot
build botRe: [PATCH v5 4/9] NFSD COPY_NOTIFY xdr
Thank you for the patch! Yet something to improve:
Hi Olga,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on nfsd/nfsd-next]
[cannot apply to v5.3-rc3 next-20190809]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Olga-Kornievskaia/server-side-support-for-inter-SSC-copy/20190811-120551 base: git://linux-nfs.org/~bfields/linux.git nfsd-next config: x86_64-lkp (attached as .config) compiler: gcc-7 (Debian 7.4.0-10) 7.4.0 reproduce: # save the attached .config to linux build tree make ARCH=x86_64
If you fix the issue, kindly add following tag
All errors (new ones prefixed by >>):
In file included from fs/nfsd/nfsctl.c:23:0:
fs/nfsd/nfsd.h:391:16: warning: 'struct nfs42_netaddr' declared inside parameter list will not be visible outside of this definition or declaration
struct nfs42_netaddr *netaddr)
^~~~~~~~~~~~~
fs/nfsd/nfsd.h: In function 'nfsd4_set_netaddr':
fs/nfsd/nfsd.h:401:18: error: dereferencing pointer to incomplete type 'struct nfs42_netaddr'
sprintf(netaddr->netid, "tcp");
^~
In file included from fs/nfsd/pnfs.h:10:0,
from fs/nfsd/nfsctl.c:27:
fs/nfsd/xdr4.h: At top level:
fs/nfsd/xdr4.h:527:20: error: field 'cp_src' has incomplete type
struct nl4_server cp_src;
^~~~~~
>> fs/nfsd/xdr4.h:577:20: error: field 'cpn_dst' has incomplete type
struct nl4_server cpn_dst;
^~~~~~~
>> fs/nfsd/xdr4.h:583:20: error: field 'cpn_src' has incomplete type
struct nl4_server cpn_src;
^~~~~~~
--
In file included from fs/nfsd/state.h:42:0,
from fs/nfsd/xdr4.h:40,
from fs/nfsd/nfs4xdr.c:47:
fs/nfsd/nfsd.h:391:16: warning: 'struct nfs42_netaddr' declared inside parameter list will not be visible outside of this definition or declaration
struct nfs42_netaddr *netaddr)
^~~~~~~~~~~~~
fs/nfsd/nfsd.h: In function 'nfsd4_set_netaddr':
fs/nfsd/nfsd.h:401:18: error: dereferencing pointer to incomplete type 'struct nfs42_netaddr'
sprintf(netaddr->netid, "tcp");
^~
In file included from fs/nfsd/nfs4xdr.c:47:0:
fs/nfsd/xdr4.h: At top level:
fs/nfsd/xdr4.h:527:20: error: field 'cp_src' has incomplete type
struct nl4_server cp_src;
^~~~~~
>> fs/nfsd/xdr4.h:577:20: error: field 'cpn_dst' has incomplete type
struct nl4_server cpn_dst;
^~~~~~~
>> fs/nfsd/xdr4.h:583:20: error: field 'cpn_src' has incomplete type
struct nl4_server cpn_src;
^~~~~~~
fs/nfsd/nfs4xdr.c: In function 'nfsd4_decode_nl4_server':
fs/nfsd/nfs4xdr.c:1743:4: error: dereferencing pointer to incomplete type 'struct nl4_server'
ns->nl4_type = be32_to_cpup(p++);
^~
fs/nfsd/nfs4xdr.c:1747:7: error: 'NL4_NETADDR' undeclared (first use in this function); did you mean 'RTM_GETADDR'?
case NL4_NETADDR:
^~~~~~~~~~~
RTM_GETADDR
fs/nfsd/nfs4xdr.c:1747:7: note: each undeclared identifier is reported only once for each function it appears in
fs/nfsd/nfs4xdr.c:1751:8: error: dereferencing pointer to incomplete type 'struct nfs42_netaddr'
naddr->netid_len = be32_to_cpup(p++);
^~
fs/nfsd/nfs4xdr.c: In function 'nfsd4_decode_copy':
fs/nfsd/nfs4xdr.c:1804:28: error: invalid application of 'sizeof' to incomplete type 'struct nl4_server'
ns_dummy = kmalloc(sizeof(struct nl4_server), GFP_KERNEL);
^~~~~~
fs/nfsd/nfs4xdr.c: In function 'nfsd42_encode_nl4_server':
fs/nfsd/nfs4xdr.c:4338:7: error: 'NL4_NETADDR' undeclared (first use in this function); did you mean 'RTM_GETADDR'?
case NL4_NETADDR:
^~~~~~~~~~~
RTM_GETADDR
In file included from include/linux/sunrpc/svcauth_gss.h:15:0,
from fs/nfsd/nfs4xdr.c:42:
fs/nfsd/nfs4xdr.c:4346:21: error: dereferencing pointer to incomplete type 'struct nfs42_netaddr'
(XDR_QUADLEN(addr->netid_len) * 4) +
^
include/linux/sunrpc/xdr.h:27:28: note: in definition of macro 'XDR_QUADLEN'
#define XDR_QUADLEN(l) (((l) + 3) >> 2)
^
fs/nfsd/nfs4xdr.c: In function 'nfsd4_encode_copy_notify':
fs/nfsd/nfs4xdr.c:4432:1: warning: control reaches end of non-void function [-Wreturn-type]
}
^
fs/nfsd/nfs4xdr.c: In function 'nfsd4_decode_copy_notify':
fs/nfsd/nfs4xdr.c:1836:1: warning: control reaches end of non-void function [-Wreturn-type]
}
^vim +/cpn_dst +577 fs/nfsd/xdr4.h
519
520 struct nfsd4_copy {
521 /* request */
522 stateid_t cp_src_stateid;
523 stateid_t cp_dst_stateid;
524 u64 cp_src_pos;
525 u64 cp_dst_pos;
526 u64 cp_count;
> 527 struct nl4_server cp_src;
528 bool cp_intra;
529
530 /* both */
531 bool cp_synchronous;
532
533 /* response */
534 struct nfsd42_write_res cp_res;
535
536 /* for cb_offload */
537 struct nfsd4_callback cp_cb;
538 __be32 nfserr;
539 struct knfsd_fh fh;
540
541 struct nfs4_client *cp_clp;
542
543 struct file *file_src;
544 struct file *file_dst;
545
546 stateid_t cp_stateid;
547
548 struct list_head copies;
549 struct task_struct *copy_task;
550 refcount_t refcount;
551 bool stopped;
552 };
553
554 struct nfsd4_seek {
555 /* request */
556 stateid_t seek_stateid;
557 loff_t seek_offset;
558 u32 seek_whence;
559
560 /* response */
561 u32 seek_eof;
562 loff_t seek_pos;
563 };
564
565 struct nfsd4_offload_status {
566 /* request */
567 stateid_t stateid;
568
569 /* response */
570 u64 count;
571 u32 status;
572 };
573
574 struct nfsd4_copy_notify {
575 /* request */
576 stateid_t cpn_src_stateid;
> 577 struct nl4_server cpn_dst;
578
579 /* response */
580 stateid_t cpn_cnr_stateid;
581 u64 cpn_sec;
582 u32 cpn_nsec;
> 583 struct nl4_server cpn_src; 584 };
585
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
PATCH v5 5/9 NFSD add COPY_NOTIFY operation
In reply to Olga KornievskaiaIntroducing the COPY_NOTIFY operation.
Create a new unique stateid that will keep track of the copy state and the upcoming READs that will use that stateid. Keep it in the list associated with parent stateid. When putting a reference on a stateid, check if there are associated copy notify stateids, if so, account for it and remove them on last reference.
Laundromat thread will traverse globally stored copy notify stateid and notice if any haven't been referenced in the lease period, if so, it'll remove them.
Return single netaddr to advertise to the copy.
--- fs/nfsd/nfs4proc.c | 47 ++++++++++++++++++-- fs/nfsd/nfs4state.c | 122 ++++++++++++++++++++++++++++++++++++++++++++++------ fs/nfsd/state.h | 30 ++++++++++++- fs/nfsd/xdr4.h | 2 +- 4 files changed, 182 insertions(+), 19 deletions(-)
fs/nfsd/nfs4proc.c +44 -3
@@ -37,6 +37,7 @@#include <linux/falloc.h> #include <linux/slab.h> #include <linux/kthread.h> +#include <linux/sunrpc/addr.h> #include "idmap.h" #include "cache.h"
@@ -1229,7 +1230,7 @@ static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)static void cleanup_async_copy (struct nfsd4_copy *copy) { -nfs4_free_cp_state (copy); +nfs4_free_copy_state (copy);fput (copy->file_dst);fput (copy->file_src);spin_lock (©->cp_clp->async_lock);
@@ -1283,7 +1284,7 @@ static int nfsd4_do_async_copy(void *data) async_copy =kzalloc (sizeof (struct nfsd4_copy ),GFP_KERNEL );if (!async_copy) goto out; -if (!nfs4_init_cp_state (nn, copy)) { +if (!nfs4_init_copy_state (nn, copy)) {kfree (async_copy); goto out; }
@@ -1350,7 +1351,47 @@ struct nfsd4_copy * nfsd4_copy_notify(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, union nfsd4_op_u *u) { - return nfserr_notsupp; + struct nfsd4_copy_notify *cn = &u->copy_notify; +__be32 status; +struct nfsd_net *nn =net_generic (SVC_NET (rqstp), nfsd_net_id); +struct nfs4_stid *stid; +struct nfs4_cpntf_state *cps; +struct nfs4_client *clp = cstate->clp; + + status =nfs4_preprocess_stateid_op (rqstp, cstate, &cstate->current_fh, + &cn->cpn_src_stateid,RD_STATE ,NULL , +NULL , &stid); +if (status) +return status; + + cn->cpn_sec = nn->nfsd4_lease; + cn->cpn_nsec =0 ; + + status =nfserrno (-ENOMEM ); + cps =nfs4_alloc_init_cpntf_state (nn, stid); +if (!cps) + goto out_err; +memcpy (&cn->cpn_cnr_stateid, &cps->cp_stateid.stid,sizeof (stateid_t)); + +/* For now, only return one server address in cpn_src, the +* address used by the client to connect to this server. +*/ + cn->cpn_src.nl4_type =NL4_NETADDR ; + status =nfsd4_set_netaddr ((struct sockaddr *)&rqstp->rq_daddr, + &cn->cpn_src.u.nl4_addr); +WARN_ON_ONCE (status); +if (status) { +free_cpntf_state (nn, cps); + goto out; + } +spin_lock (&clp->cpntf_lock); +list_add (&cps->cpntf, &clp->copy_notifies); +spin_unlock (&clp->cpntf_lock); +out: +return status; +out_err: +nfs4_put_stid (stid); + goto out; } static __be32
fs/nfsd/nfs4state.c +109 -13
206 lines not on this page · raw message
fs/nfsd/state.h +28 -2
@@ -56,6 +56,13 @@stateid_opaque_t si_opaque; } stateid_t; +typedef struct { +stateid_t stid; +#define NFS4_COPY_STID 1 +#define NFS4_COPYNOTIFY_STID 2 +unsigned char sc_type; +}copy_stateid_t ; +#define STATEID_FMT "(%08x/%08x/%08x/%08x)"#define STATEID_VAL (s) \ (s)->si_opaque.so_clid.cl_boot, \
@@ -96,6 +103,8 @@ struct nfs4_stid {#define NFS4_REVOKED_DELEG_STID 16#define NFS4_CLOSED_DELEG_STID 32#define NFS4_LAYOUT_STID 64 +struct list_head sc_cp_list; +size_t sc_cp_list_size;unsigned char sc_type;stateid_t sc_stateid;spinlock_t sc_lock;
@@ -104,6 +113,18 @@ struct nfs4_stid {void (*sc_free)(struct nfs4_stid *); }; +/* Keep a list of stateids issued by the COPY_NOTIFY, associate it with the +* parent OPEN/LOCK/DELEG stateid. +*/ +struct nfs4_cpntf_state { +copy_stateid_t cp_stateid; +struct list_head cp_list;/* per parent nfs4_stid */ +struct nfs4_stid *cp_p_stid;/* pointer to parent */ +struct list_head cpntf;/* list of copy_notifies */ +time_t cpntf_time;/* last time stateid used */ +struct nfsd_net *net; +}; +/* * Represents a delegation stateid. The nfs4_client holds references to these * and they are put when it is being destroyed or when the delegation is
@@ -367,6 +388,8 @@ struct nfs4_client {struct net *net;struct list_head async_copies;/* list of async copies */ spinlock_t async_lock;/* lock for async copies */ +struct list_head copy_notifies;/* list of copy notify stids */ +spinlock_t cpntf_lock;/* lock for copy_notifies */ };/* struct nfs4_client_reset
@@ -623,8 +646,10 @@ __be32 nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate, struct nfs4_stid **s, struct nfsd_net *nn);struct nfs4_stid *nfs4_alloc_stid (struct nfs4_client *cl,struct kmem_cache *slab,void (*sc_free)(struct nfs4_stid *)); -int nfs4_init_cp_state (struct nfsd_net *nn,struct nfsd4_copy *copy); -void nfs4_free_cp_state (struct nfsd4_copy *copy); +int nfs4_init_copy_state (struct nfsd_net *nn,struct nfsd4_copy *copy); +void nfs4_free_copy_state (struct nfsd4_copy *copy); +struct nfs4_cpntf_state *nfs4_alloc_init_cpntf_state (struct nfsd_net *nn, +struct nfs4_stid *p_stid);void nfs4_unhash_stid (struct nfs4_stid *s);void nfs4_put_stid (struct nfs4_stid *s);void nfs4_inc_and_copy_stateid (stateid_t *dst,struct nfs4_stid *stid);
@@ -654,6 +679,7 @@ extern struct nfs4_client_reclaim *nfs4_client_to_reclaim(struct xdr_netobj nameextern void nfs4_put_copy (struct nfsd4_copy *copy);extern struct nfsd4_copy *find_async_copy (struct nfs4_client *clp,stateid_t *staetid); +extern void free_cpntf_state (struct nfsd_net *nn,struct nfs4_cpntf_state *cps);static inline void get_nfs4_file (struct nfs4_file *fi) {refcount_inc (&fi->fi_ref);
fs/nfsd/xdr4.h +1 -1
@@ -543,7 +543,7 @@ struct nfsd4_copy {struct file *file_src;struct file *file_dst; -stateid_t cp_stateid; +copy_stateid_t cp_stateid;struct list_head copies;struct task_struct *copy_task;
Signature · 1 line
1.8.3.1
kbuild test robot
build botRe: [PATCH v5 5/9] NFSD add COPY_NOTIFY operation
Thank you for the patch! Yet something to improve:
Hi Olga,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on nfsd/nfsd-next]
[cannot apply to v5.3-rc3 next-20190809]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Olga-Kornievskaia/server-side-support-for-inter-SSC-copy/20190811-120551 base: git://linux-nfs.org/~bfields/linux.git nfsd-next config: x86_64-lkp (attached as .config) compiler: gcc-7 (Debian 7.4.0-10) 7.4.0 reproduce: # save the attached .config to linux build tree make ARCH=x86_64
If you fix the issue, kindly add following tag
All errors (new ones prefixed by >>):
In file included from fs/nfsd/state.h:42:0,
from fs/nfsd/xdr4.h:40,
from fs/nfsd/nfs4proc.c:44:
fs/nfsd/nfsd.h:391:16: warning: 'struct nfs42_netaddr' declared inside parameter list will not be visible outside of this definition or declaration
struct nfs42_netaddr *netaddr)
^~~~~~~~~~~~~
fs/nfsd/nfsd.h: In function 'nfsd4_set_netaddr':
fs/nfsd/nfsd.h:401:18: error: dereferencing pointer to incomplete type 'struct nfs42_netaddr'
sprintf(netaddr->netid, "tcp");
^~
In file included from fs/nfsd/nfs4proc.c:44:0:
fs/nfsd/xdr4.h: At top level:
fs/nfsd/xdr4.h:527:20: error: field 'cp_src' has incomplete type
struct nl4_server cp_src;
^~~~~~
fs/nfsd/xdr4.h:577:20: error: field 'cpn_dst' has incomplete type
struct nl4_server cpn_dst;
^~~~~~~
fs/nfsd/xdr4.h:583:20: error: field 'cpn_src' has incomplete type
struct nl4_server cpn_src;
^~~~~~~
fs/nfsd/nfs4proc.c: In function 'nfsd4_copy_notify':
>> fs/nfsd/nfs4proc.c:1379:25: error: 'NL4_NETADDR' undeclared (first use in this function); did you mean 'RTM_GETADDR'?
cn->cpn_src.nl4_type = NL4_NETADDR;
^~~~~~~~~~~
RTM_GETADDR fs/nfsd/nfs4proc.c:1379:25: note: each undeclared identifier is reported only once for each function it appears in
vim +1379 fs/nfsd/nfs4proc.c
1349
1350 static __be32
1351 nfsd4_copy_notify(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
1352 union nfsd4_op_u *u)
1353 {
1354 struct nfsd4_copy_notify *cn = &u->copy_notify;
1355 __be32 status;
1356 struct nfsd_net *nn = net_generic(SVC_NET(rqstp), nfsd_net_id);
1357 struct nfs4_stid *stid;
1358 struct nfs4_cpntf_state *cps;
1359 struct nfs4_client *clp = cstate->clp;
1360
1361 status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->current_fh,
1362 &cn->cpn_src_stateid, RD_STATE, NULL,
1363 NULL, &stid);
1364 if (status)
1365 return status;
1366
1367 cn->cpn_sec = nn->nfsd4_lease;
1368 cn->cpn_nsec = 0;
1369
1370 status = nfserrno(-ENOMEM);
1371 cps = nfs4_alloc_init_cpntf_state(nn, stid);
1372 if (!cps)
1373 goto out_err;
1374 memcpy(&cn->cpn_cnr_stateid, &cps->cp_stateid.stid, sizeof(stateid_t));
1375
1376 /* For now, only return one server address in cpn_src, the
1377 * address used by the client to connect to this server.
1378 */
> 1379 cn->cpn_src.nl4_type = NL4_NETADDR;
1380 status = nfsd4_set_netaddr((struct sockaddr *)&rqstp->rq_daddr,
1381 &cn->cpn_src.u.nl4_addr);
1382 WARN_ON_ONCE(status);
1383 if (status) {
1384 free_cpntf_state(nn, cps);
1385 goto out;
1386 }
1387 spin_lock(&clp->cpntf_lock);
1388 list_add(&cps->cpntf, &clp->copy_notifies);
1389 spin_unlock(&clp->cpntf_lock);
1390 out:
1391 return status;
1392 out_err:
1393 nfs4_put_stid(stid);
1394 goto out;
1395 }
1396
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Re: [PATCH v5 5/9] NFSD add COPY_NOTIFY operation
In reply to Olga KornievskaiaQuoted 233 lines · Olga Kornievskaia
Introducing the
COPY_NOTIFYoperation.Create a new unique stateid that will keep track of the copy state and the upcoming READs that will use that stateid. Keep it in the list associated with parent stateid. When putting a reference on a stateid, check if there are associated copy notify stateids, if so, account for it and remove them on last reference.
Laundromat thread will traverse globally stored copy notify stateid and notice if any haven't been referenced in the lease period, if so, it'll remove them.
Return single netaddr to advertise to the copy.
Signed-off-by Andy Adamson <[email protected]>Signed-off-by Olga Kornievskaia <[email protected]>--- fs/nfsd/nfs4proc.c | 47 ++++++++++++++++++-- fs/nfsd/nfs4state.c | 122 ++++++++++++++++++++++++++++++++++++++++++++++------ fs/nfsd/state.h | 30 ++++++++++++- fs/nfsd/xdr4.h | 2 +- 4 files changed, 182 insertions(+), 19 deletions(-)fs/nfsd/nfs4proc.c +44
-3@@ -37,6 +37,7 @@#include <linux/falloc.h> #include <linux/slab.h> #include <linux/kthread.h> +#include <linux/sunrpc/addr.h> #include "idmap.h" #include "cache.h" @@ -1229,7 +1230,7 @@ static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)static void cleanup_async_copy (struct nfsd4_copy *copy) { -nfs4_free_cp_state (copy); +nfs4_free_copy_state (copy);fput (copy->file_dst);fput (copy->file_src);spin_lock (©->cp_clp->async_lock);@@ -1283,7 +1284,7 @@ static int nfsd4_do_async_copy(void *data) async_copy =kzalloc (sizeof (struct nfsd4_copy ),GFP_KERNEL );if (!async_copy) goto out; -if (!nfs4_init_cp_state (nn, copy)) { +if (!nfs4_init_copy_state (nn, copy)) {kfree (async_copy); goto out; }@@ -1350,7 +1351,47 @@ struct nfsd4_copy * nfsd4_copy_notify(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, union nfsd4_op_u *u) { - return nfserr_notsupp; + struct nfsd4_copy_notify *cn = &u->copy_notify; +__be32 status; +struct nfsd_net *nn =net_generic (SVC_NET (rqstp), nfsd_net_id); +struct nfs4_stid *stid; +struct nfs4_cpntf_state *cps; +struct nfs4_client *clp = cstate->clp; + + status =nfs4_preprocess_stateid_op (rqstp, cstate, &cstate->current_fh, + &cn->cpn_src_stateid,RD_STATE ,NULL , +NULL , &stid); +if (status) +return status; + + cn->cpn_sec = nn->nfsd4_lease; + cn->cpn_nsec =0 ; + + status =nfserrno (-ENOMEM ); + cps =nfs4_alloc_init_cpntf_state (nn, stid); +if (!cps) + goto out_err; +memcpy (&cn->cpn_cnr_stateid, &cps->cp_stateid.stid,sizeof (stateid_t)); + +/* For now, only return one server address in cpn_src, the +* address used by the client to connect to this server. +*/ + cn->cpn_src.nl4_type =NL4_NETADDR ; + status =nfsd4_set_netaddr ((struct sockaddr *)&rqstp->rq_daddr, + &cn->cpn_src.u.nl4_addr); +WARN_ON_ONCE (status); +if (status) { +free_cpntf_state (nn, cps); + goto out; + } +spin_lock (&clp->cpntf_lock); +list_add (&cps->cpntf, &clp->copy_notifies); +spin_unlock (&clp->cpntf_lock); +out: +return status; +out_err: +nfs4_put_stid (stid); + goto out; } static __be32fs/nfsd/nfs4state.c +74
-11@@ -720,6 +720,7 @@ struct nfs4_stid *nfs4_alloc_stid(struct nfs4_client *cl, struct kmem_cache *sla/* Will be incremented before return to client: */ refcount_set (&stid->sc_count,1 );spin_lock_init (&stid->sc_lock); +INIT_LIST_HEAD (&stid->sc_cp_list);/* * It shouldn't be a problem to reuse an opaque stateid value. @@ -739,33 +740,89 @@ struct nfs4_stid *nfs4_alloc_stid(struct nfs4_client *cl, struct kmem_cache *sla/* * Create a unique stateid_t to represent each COPY. */ -int nfs4_init_cp_state (struct nfsd_net *nn,struct nfsd4_copy *copy) +static int nfs4_init_cp_state (struct nfsd_net *nn,copy_stateid_t *stid, +unsigned char sc_type) {int new_id; + stid->stid.si_opaque.so_clid.cl_boot = nn->boot_time; + stid->stid.si_opaque.so_clid.cl_id = nn->s2s_cp_cl_id; + stid->sc_type = sc_type; +idr_preload (GFP_KERNEL );spin_lock (&nn->s2s_cp_lock); - new_id =idr_alloc_cyclic (&nn->s2s_cp_stateids, copy,0 ,0 ,GFP_NOWAIT ); + new_id =idr_alloc_cyclic (&nn->s2s_cp_stateids, stid,0 ,0 ,GFP_NOWAIT ); + stid->stid.si_opaque.so_id = new_id;spin_unlock (&nn->s2s_cp_lock);idr_preload_end ();if (new_id <0 )return 0 ; - copy->cp_stateid.si_opaque.so_id = new_id; - copy->cp_stateid.si_opaque.so_clid.cl_boot = nn->boot_time; - copy->cp_stateid.si_opaque.so_clid.cl_id = nn->s2s_cp_cl_id;return 1 ; } -void nfs4_free_cp_state (struct nfsd4_copy *copy) +int nfs4_init_copy_state (struct nfsd_net *nn,struct nfsd4_copy *copy) { -struct nfsd_net *nn; +return nfs4_init_cp_state (nn, ©->cp_stateid,NFS4_COPY_STID ); +} - nn =net_generic (copy->cp_clp->net, nfsd_net_id); +struct nfs4_cpntf_state *nfs4_alloc_init_cpntf_state (struct nfsd_net *nn, +struct nfs4_stid *p_stid) +{ +struct nfs4_cpntf_state *cps; + + cps =kzalloc (sizeof (struct nfs4_cpntf_state ),GFP_KERNEL ); +if (!cps) +return NULL ; + cps->cp_p_stid = p_stid; + cps->cpntf_time =get_seconds (); + cps->net = nn; +if (!nfs4_init_cp_state (nn, &cps->cp_stateid,NFS4_COPYNOTIFY_STID )) + goto out_free; +spin_lock (&p_stid->sc_lock); +list_add (&cps->cp_list, &p_stid->sc_cp_list); + p_stid->sc_cp_list_size++; +spin_unlock (&p_stid->sc_lock); +return cps; +out_free: +kfree (cps); +return NULL ; +} +void _free_copy_cpntf_stateid (struct nfsd_net *nn,stateid_t *stid) +{spin_lock (&nn->s2s_cp_lock); -idr_remove (&nn->s2s_cp_stateids, copy->cp_stateid.si_opaque.so_id); +idr_remove (&nn->s2s_cp_stateids, stid->si_opaque.so_id);spin_unlock (&nn->s2s_cp_lock); } +void nfs4_free_copy_state (struct nfsd4_copy *copy) +{ +struct nfsd_net *nn; + + nn =net_generic (copy->cp_clp->net, nfsd_net_id); +_free_copy_cpntf_stateid (nn, ©->cp_stateid.stid); +} + +static void nfs4_free_cpntf_statelist (struct net *net,struct nfs4_stid *stid) +{ +struct nfs4_cpntf_state *cps; +struct nfsd_net *nn; + + nn =net_generic (net, nfsd_net_id); +spin_lock (&stid->sc_lock); +while (!list_empty (&stid->sc_cp_list)) { + cps =list_first_entry (&stid->sc_cp_list, + struct nfs4_cpntf_state, cp_list); + stid->sc_cp_list_size--; +list_del (&cps->cp_list); +_free_copy_cpntf_stateid (nn, &cps->cp_stateid.stid); +spin_lock (&stid->sc_client->cpntf_lock); +list_del (&cps->cpntf); +spin_unlock (&stid->sc_client->cpntf_lock); +kfree (cps); + } +spin_unlock (&stid->sc_lock); +} +static struct nfs4_ol_stateid *nfs4_alloc_open_stateid (struct nfs4_client *clp) {struct nfs4_stid *stid;@@ -905,15 +962,25 @@ static void block_delegations(struct knfsd_fh *fh) {struct nfs4_file *fp = s->sc_file;struct nfs4_client *clp = s->sc_client; +size_t size =0 ; + +spin_lock (&s->sc_lock); + size = s->sc_cp_list_size; +spin_unlock (&s->sc_lock);might_lock (&clp->cl_lock);if (!refcount_dec_and_lock (&s->sc_count, &clp->cl_lock)) { -wake_up_all (&close_wq); -return ; +if (!refcount_sub_and_test_checked (s->sc_cp_list_size, + &s->sc_count)) { +refcount_add_checked (s->sc_cp_list_size, &s->sc_count);
fs/nfsd/nfs4state.c static void block_delegations(struct knfsd_fh *fh) · line 986
Quoted 10 lines
fs/nfsd/nfs4state.c +4
-0+wake_up_all (&close_wq); +return ; + } }idr_remove (&clp->cl_stateids, s->sc_stateid.si_opaque.so_id);spin_unlock (&clp->cl_lock); +nfs4_free_cpntf_statelist (clp->net, s); s->sc_free (s);if (fp)put_nfs4_file (fp);
While this passes my testing, in theory this allows for the race that we get the copy notify size but then offload_cancel arrive and change the value. Then refcount_sub_and test_check would have an incorrect value (can subtract larger than an actual reference count). I have no solution for that as there is no refcount_sub_and_lock() that will allow to decrement by a multiple under a lock. Thoughts?
Quoted 172 lines
fs/nfsd/nfs4state.c +31
-2@@ -1881,6 +1948,8 @@ static struct nfs4_client *alloc_client(struct xdr_netobj name)#endif INIT_LIST_HEAD (&clp->async_copies);spin_lock_init (&clp->async_lock); +INIT_LIST_HEAD (&clp->copy_notifies); +spin_lock_init (&clp->cpntf_lock);spin_lock_init (&clp->cl_lock);rpc_init_wait_queue (&clp->cl_cb_waitq,"Backchannel slot table" );return clp;@@ -2909,7 +2978,8 @@ static bool client_has_state(struct nfs4_client *clp)#endif || !list_empty(&clp->cl_delegations) || !list_empty (&clp->cl_sessions) - || !list_empty (&clp->async_copies); + || !list_empty (&clp->async_copies) + || !list_empty (&clp->copy_notifies); } static __be32 copy_impl_id(struct nfs4_client *clp,@@ -5184,9 +5254,10 @@ static bool clients_still_reclaiming(struct nfsd_net *nn)struct nfs4_delegation *dp;struct nfs4_ol_stateid *stp;struct nfsd4_blocked_lock *nbl; -struct list_head *pos, *next, reaplist; +struct list_head *pos, *next, *ppos, *pnext, reaplist, cpntflist;time_t cutoff =get_seconds () - nn->nfsd4_lease;time_t t, new_timeo = nn->nfsd4_lease; +struct nfs4_cpntf_state *cps;dprintk ("NFSD: laundromat service - starting\n" );@@ -5197,9 +5268,19 @@ static bool clients_still_reclaiming(struct nfsd_net *nn)dprintk ("NFSD: end of grace period\n" );nfsd4_end_grace (nn);INIT_LIST_HEAD (&reaplist); +INIT_LIST_HEAD (&cpntflist); +spin_lock (&nn->client_lock);list_for_each_safe (pos, next, &nn->client_lru) { clp = list_entry(pos, struct nfs4_client, cl_lru); +spin_lock (&clp->cpntf_lock); +list_for_each_safe (ppos, pnext, &clp->copy_notifies) { + cps = list_entry(ppos, struct nfs4_cpntf_state, cpntf); +if (!time_after ((unsigned long )cps->cpntf_time, + (unsigned long )cutoff)) +list_move (&cps->cpntf, &cpntflist); + } +spin_unlock (&clp->cpntf_lock);if (time_after ((unsigned long )clp->cl_time, (unsigned long )cutoff)) { t = clp->cl_time - cutoff; new_timeo =min (new_timeo, t);@@ -5213,6 +5294,11 @@ static bool clients_still_reclaiming(struct nfsd_net *nn)list_add (&clp->cl_lru, &reaplist); }spin_unlock (&nn->client_lock); +list_for_each_safe (pos, next, &cpntflist) { + cps = list_entry(pos, struct nfs4_cpntf_state, cpntf); +list_del_init (&cps->cpntf); +free_cpntf_state (cps->net, cps); + }list_for_each_safe (pos, next, &reaplist) { clp = list_entry(pos, struct nfs4_client, cl_lru); dprintk("NFSD: purging unused client (clientid %08x)\n",@@ -5576,6 +5662,16 @@ static __be32 nfsd4_validate_stateid(struct nfs4_client *cl, stateid_t *stateid)return 0 ; } +void free_cpntf_state (struct nfsd_net *nn,struct nfs4_cpntf_state *cps) +{ +spin_lock (&cps->cp_p_stid->sc_lock); +list_del (&cps->cp_list); + cps->cp_p_stid->sc_cp_list_size--; +spin_unlock (&cps->cp_p_stid->sc_lock); +_free_copy_cpntf_stateid (nn, &cps->cp_stateid.stid); +nfs4_put_stid (cps->cp_p_stid); +kfree (cps); +}/* * Checks for stateid operations fs/nfsd/state.h +28
-2@@ -56,6 +56,13 @@stateid_opaque_t si_opaque; } stateid_t; +typedef struct { +stateid_t stid; +#define NFS4_COPY_STID 1 +#define NFS4_COPYNOTIFY_STID 2 +unsigned char sc_type; +}copy_stateid_t ; +#define STATEID_FMT "(%08x/%08x/%08x/%08x)"#define STATEID_VAL (s) \ (s)->si_opaque.so_clid.cl_boot, \@@ -96,6 +103,8 @@ struct nfs4_stid {#define NFS4_REVOKED_DELEG_STID 16#define NFS4_CLOSED_DELEG_STID 32#define NFS4_LAYOUT_STID 64 +struct list_head sc_cp_list; +size_t sc_cp_list_size;unsigned char sc_type;stateid_t sc_stateid;spinlock_t sc_lock;@@ -104,6 +113,18 @@ struct nfs4_stid {void (*sc_free)(struct nfs4_stid *); }; +/* Keep a list of stateids issued by the COPY_NOTIFY, associate it with the +* parent OPEN/LOCK/DELEG stateid. +*/ +struct nfs4_cpntf_state { +copy_stateid_t cp_stateid; +struct list_head cp_list;/* per parent nfs4_stid */ +struct nfs4_stid *cp_p_stid;/* pointer to parent */ +struct list_head cpntf;/* list of copy_notifies */ +time_t cpntf_time;/* last time stateid used */ +struct nfsd_net *net; +}; +/* * Represents a delegation stateid. The nfs4_client holds references to these * and they are put when it is being destroyed or when the delegation is @@ -367,6 +388,8 @@ struct nfs4_client {struct net *net;struct list_head async_copies;/* list of async copies */ spinlock_t async_lock;/* lock for async copies */ +struct list_head copy_notifies;/* list of copy notify stids */ +spinlock_t cpntf_lock;/* lock for copy_notifies */ };/* struct nfs4_client_reset @@ -623,8 +646,10 @@ __be32 nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate, struct nfs4_stid **s, struct nfsd_net *nn);struct nfs4_stid *nfs4_alloc_stid (struct nfs4_client *cl,struct kmem_cache *slab,void (*sc_free)(struct nfs4_stid *)); -int nfs4_init_cp_state (struct nfsd_net *nn,struct nfsd4_copy *copy); -void nfs4_free_cp_state (struct nfsd4_copy *copy); +int nfs4_init_copy_state (struct nfsd_net *nn,struct nfsd4_copy *copy); +void nfs4_free_copy_state (struct nfsd4_copy *copy); +struct nfs4_cpntf_state *nfs4_alloc_init_cpntf_state (struct nfsd_net *nn, +struct nfs4_stid *p_stid);void nfs4_unhash_stid (struct nfs4_stid *s);void nfs4_put_stid (struct nfs4_stid *s);void nfs4_inc_and_copy_stateid (stateid_t *dst,struct nfs4_stid *stid);@@ -654,6 +679,7 @@ extern struct nfs4_client_reclaim *nfs4_client_to_reclaim(struct xdr_netobj nameextern void nfs4_put_copy (struct nfsd4_copy *copy);extern struct nfsd4_copy *find_async_copy (struct nfs4_client *clp,stateid_t *staetid); +extern void free_cpntf_state (struct nfsd_net *nn,struct nfs4_cpntf_state *cps);static inline void get_nfs4_file (struct nfs4_file *fi) {refcount_inc (&fi->fi_ref);fs/nfsd/xdr4.h +1
-2@@ -543,7 +543,7 @@ struct nfsd4_copy {struct file *file_src;struct file *file_dst; -stateid_t cp_stateid; +copy_stateid_t cp_stateid;struct list_head copies;struct task_struct *copy_task; --1.8.3.1
Re: [PATCH v5 5/9] NFSD add COPY_NOTIFY operation
Quoted 245 lines · Olga Kornievskaia
Quoted 242 lines · Olga Kornievskaia
Introducing the
COPY_NOTIFYoperation.Create a new unique stateid that will keep track of the copy state and the upcoming READs that will use that stateid. Keep it in the list associated with parent stateid. When putting a reference on a stateid, check if there are associated copy notify stateids, if so, account for it and remove them on last reference.
Laundromat thread will traverse globally stored copy notify stateid and notice if any haven't been referenced in the lease period, if so, it'll remove them.
Return single netaddr to advertise to the copy.
Signed-off-by Andy Adamson <[email protected]>Signed-off-by Olga Kornievskaia <[email protected]>--- fs/nfsd/nfs4proc.c | 47 ++++++++++++++++++-- fs/nfsd/nfs4state.c | 122 ++++++++++++++++++++++++++++++++++++++++++++++------ fs/nfsd/state.h | 30 ++++++++++++- fs/nfsd/xdr4.h | 2 +- 4 files changed, 182 insertions(+), 19 deletions(-)fs/nfsd/nfs4proc.c +44
-3@@ -37,6 +37,7 @@#include <linux/falloc.h> #include <linux/slab.h> #include <linux/kthread.h> +#include <linux/sunrpc/addr.h> #include "idmap.h" #include "cache.h" @@ -1229,7 +1230,7 @@ static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)static void cleanup_async_copy (struct nfsd4_copy *copy) { -nfs4_free_cp_state (copy); +nfs4_free_copy_state (copy);fput (copy->file_dst);fput (copy->file_src);spin_lock (©->cp_clp->async_lock);@@ -1283,7 +1284,7 @@ static int nfsd4_do_async_copy(void *data) async_copy =kzalloc (sizeof (struct nfsd4_copy ),GFP_KERNEL );if (!async_copy) goto out; -if (!nfs4_init_cp_state (nn, copy)) { +if (!nfs4_init_copy_state (nn, copy)) {kfree (async_copy); goto out; }@@ -1350,7 +1351,47 @@ struct nfsd4_copy * nfsd4_copy_notify(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, union nfsd4_op_u *u) { - return nfserr_notsupp; + struct nfsd4_copy_notify *cn = &u->copy_notify; +__be32 status; +struct nfsd_net *nn =net_generic (SVC_NET (rqstp), nfsd_net_id); +struct nfs4_stid *stid; +struct nfs4_cpntf_state *cps; +struct nfs4_client *clp = cstate->clp; + + status =nfs4_preprocess_stateid_op (rqstp, cstate, &cstate->current_fh, + &cn->cpn_src_stateid,RD_STATE ,NULL , +NULL , &stid); +if (status) +return status; + + cn->cpn_sec = nn->nfsd4_lease; + cn->cpn_nsec =0 ; + + status =nfserrno (-ENOMEM ); + cps =nfs4_alloc_init_cpntf_state (nn, stid); +if (!cps) + goto out_err; +memcpy (&cn->cpn_cnr_stateid, &cps->cp_stateid.stid,sizeof (stateid_t)); + +/* For now, only return one server address in cpn_src, the +* address used by the client to connect to this server. +*/ + cn->cpn_src.nl4_type =NL4_NETADDR ; + status =nfsd4_set_netaddr ((struct sockaddr *)&rqstp->rq_daddr, + &cn->cpn_src.u.nl4_addr); +WARN_ON_ONCE (status); +if (status) { +free_cpntf_state (nn, cps); + goto out; + } +spin_lock (&clp->cpntf_lock); +list_add (&cps->cpntf, &clp->copy_notifies); +spin_unlock (&clp->cpntf_lock); +out: +return status; +out_err: +nfs4_put_stid (stid); + goto out; } static __be32fs/nfsd/nfs4state.c +78
-11@@ -720,6 +720,7 @@ struct nfs4_stid *nfs4_alloc_stid(struct nfs4_client *cl, struct kmem_cache *sla/* Will be incremented before return to client: */ refcount_set (&stid->sc_count,1 );spin_lock_init (&stid->sc_lock); +INIT_LIST_HEAD (&stid->sc_cp_list);/* * It shouldn't be a problem to reuse an opaque stateid value. @@ -739,33 +740,89 @@ struct nfs4_stid *nfs4_alloc_stid(struct nfs4_client *cl, struct kmem_cache *sla/* * Create a unique stateid_t to represent each COPY. */ -int nfs4_init_cp_state (struct nfsd_net *nn,struct nfsd4_copy *copy) +static int nfs4_init_cp_state (struct nfsd_net *nn,copy_stateid_t *stid, +unsigned char sc_type) {int new_id; + stid->stid.si_opaque.so_clid.cl_boot = nn->boot_time; + stid->stid.si_opaque.so_clid.cl_id = nn->s2s_cp_cl_id; + stid->sc_type = sc_type; +idr_preload (GFP_KERNEL );spin_lock (&nn->s2s_cp_lock); - new_id =idr_alloc_cyclic (&nn->s2s_cp_stateids, copy,0 ,0 ,GFP_NOWAIT ); + new_id =idr_alloc_cyclic (&nn->s2s_cp_stateids, stid,0 ,0 ,GFP_NOWAIT ); + stid->stid.si_opaque.so_id = new_id;spin_unlock (&nn->s2s_cp_lock);idr_preload_end ();if (new_id <0 )return 0 ; - copy->cp_stateid.si_opaque.so_id = new_id; - copy->cp_stateid.si_opaque.so_clid.cl_boot = nn->boot_time; - copy->cp_stateid.si_opaque.so_clid.cl_id = nn->s2s_cp_cl_id;return 1 ; } -void nfs4_free_cp_state (struct nfsd4_copy *copy) +int nfs4_init_copy_state (struct nfsd_net *nn,struct nfsd4_copy *copy) { -struct nfsd_net *nn; +return nfs4_init_cp_state (nn, ©->cp_stateid,NFS4_COPY_STID ); +} - nn =net_generic (copy->cp_clp->net, nfsd_net_id); +struct nfs4_cpntf_state *nfs4_alloc_init_cpntf_state (struct nfsd_net *nn, +struct nfs4_stid *p_stid) +{ +struct nfs4_cpntf_state *cps; + + cps =kzalloc (sizeof (struct nfs4_cpntf_state ),GFP_KERNEL ); +if (!cps) +return NULL ; + cps->cp_p_stid = p_stid; + cps->cpntf_time =get_seconds (); + cps->net = nn; +if (!nfs4_init_cp_state (nn, &cps->cp_stateid,NFS4_COPYNOTIFY_STID )) + goto out_free; +spin_lock (&p_stid->sc_lock); +list_add (&cps->cp_list, &p_stid->sc_cp_list); + p_stid->sc_cp_list_size++; +spin_unlock (&p_stid->sc_lock); +return cps; +out_free: +kfree (cps); +return NULL ; +} +void _free_copy_cpntf_stateid (struct nfsd_net *nn,stateid_t *stid) +{spin_lock (&nn->s2s_cp_lock); -idr_remove (&nn->s2s_cp_stateids, copy->cp_stateid.si_opaque.so_id); +idr_remove (&nn->s2s_cp_stateids, stid->si_opaque.so_id);spin_unlock (&nn->s2s_cp_lock); } +void nfs4_free_copy_state (struct nfsd4_copy *copy) +{ +struct nfsd_net *nn; + + nn =net_generic (copy->cp_clp->net, nfsd_net_id); +_free_copy_cpntf_stateid (nn, ©->cp_stateid.stid); +} + +static void nfs4_free_cpntf_statelist (struct net *net,struct nfs4_stid *stid) +{ +struct nfs4_cpntf_state *cps; +struct nfsd_net *nn; + + nn =net_generic (net, nfsd_net_id); +spin_lock (&stid->sc_lock); +while (!list_empty (&stid->sc_cp_list)) { + cps =list_first_entry (&stid->sc_cp_list, + struct nfs4_cpntf_state, cp_list); + stid->sc_cp_list_size--; +list_del (&cps->cp_list); +_free_copy_cpntf_stateid (nn, &cps->cp_stateid.stid); +spin_lock (&stid->sc_client->cpntf_lock); +list_del (&cps->cpntf); +spin_unlock (&stid->sc_client->cpntf_lock); +kfree (cps); + } +spin_unlock (&stid->sc_lock); +} +static struct nfs4_ol_stateid *nfs4_alloc_open_stateid (struct nfs4_client *clp) {struct nfs4_stid *stid;@@ -905,15 +962,25 @@ static void block_delegations(struct knfsd_fh *fh) {struct nfs4_file *fp = s->sc_file;struct nfs4_client *clp = s->sc_client; +size_t size =0 ; + +spin_lock (&s->sc_lock); + size = s->sc_cp_list_size; +spin_unlock (&s->sc_lock);might_lock (&clp->cl_lock);if (!refcount_dec_and_lock (&s->sc_count, &clp->cl_lock)) { -wake_up_all (&close_wq); -return ; +if (!refcount_sub_and_test_checked (s->sc_cp_list_size, + &s->sc_count)) { +refcount_add_checked (s->sc_cp_list_size, &s->sc_count); +wake_up_all (&close_wq); +return ; + } }idr_remove (&clp->cl_stateids, s->sc_stateid.si_opaque.so_id);spin_unlock (&clp->cl_lock); +nfs4_free_cpntf_statelist (clp->net, s); s->sc_free (s);if (fp)
Quoted 8 lines
fs/nfsd/nfs4state.c static void block_delegations(struct knfsd_fh *fh) · line 986
Quoted 1 line
fs/nfsd/nfs4state.c +0
-0put_nfs4_file (fp);While this passes my testing, in theory this allows for the race that we get the copy notify size but then
offload_cancelarrive and change the value. Thenrefcount_sub_andtest_checkwould have an incorrect value (can subtract larger than an actual reference count). I have no solution for that as there is norefcount_sub_and_lock()that will allow to decrement by a multiple under a lock. Thoughts?
I tried not to use the client's cl_lock but instead use a specific lock to protect the copy notifies stateid on the stateid list. But since stateid's reference counter (sc_count) is protected by it, I think by getting rid of the special lock and using cl_lock will solve the problem of coordinating access between the sc_count and the copy_notify stateid list. Are the any problems with using such a big lock?
Quoted 173 lines
Quoted 173 lines
fs/nfsd/nfs4state.c +31
-2@@ -1881,6 +1948,8 @@ static struct nfs4_client *alloc_client(struct xdr_netobj name)#endif INIT_LIST_HEAD (&clp->async_copies);spin_lock_init (&clp->async_lock); +INIT_LIST_HEAD (&clp->copy_notifies); +spin_lock_init (&clp->cpntf_lock);spin_lock_init (&clp->cl_lock);rpc_init_wait_queue (&clp->cl_cb_waitq,"Backchannel slot table" );return clp;@@ -2909,7 +2978,8 @@ static bool client_has_state(struct nfs4_client *clp)#endif || !list_empty(&clp->cl_delegations) || !list_empty (&clp->cl_sessions) - || !list_empty (&clp->async_copies); + || !list_empty (&clp->async_copies) + || !list_empty (&clp->copy_notifies); } static __be32 copy_impl_id(struct nfs4_client *clp,@@ -5184,9 +5254,10 @@ static bool clients_still_reclaiming(struct nfsd_net *nn)struct nfs4_delegation *dp;struct nfs4_ol_stateid *stp;struct nfsd4_blocked_lock *nbl; -struct list_head *pos, *next, reaplist; +struct list_head *pos, *next, *ppos, *pnext, reaplist, cpntflist;time_t cutoff =get_seconds () - nn->nfsd4_lease;time_t t, new_timeo = nn->nfsd4_lease; +struct nfs4_cpntf_state *cps;dprintk ("NFSD: laundromat service - starting\n" );@@ -5197,9 +5268,19 @@ static bool clients_still_reclaiming(struct nfsd_net *nn)dprintk ("NFSD: end of grace period\n" );nfsd4_end_grace (nn);INIT_LIST_HEAD (&reaplist); +INIT_LIST_HEAD (&cpntflist); +spin_lock (&nn->client_lock);list_for_each_safe (pos, next, &nn->client_lru) { clp = list_entry(pos, struct nfs4_client, cl_lru); +spin_lock (&clp->cpntf_lock); +list_for_each_safe (ppos, pnext, &clp->copy_notifies) { + cps = list_entry(ppos, struct nfs4_cpntf_state, cpntf); +if (!time_after ((unsigned long )cps->cpntf_time, + (unsigned long )cutoff)) +list_move (&cps->cpntf, &cpntflist); + } +spin_unlock (&clp->cpntf_lock);if (time_after ((unsigned long )clp->cl_time, (unsigned long )cutoff)) { t = clp->cl_time - cutoff; new_timeo =min (new_timeo, t);@@ -5213,6 +5294,11 @@ static bool clients_still_reclaiming(struct nfsd_net *nn)list_add (&clp->cl_lru, &reaplist); }spin_unlock (&nn->client_lock); +list_for_each_safe (pos, next, &cpntflist) { + cps = list_entry(pos, struct nfs4_cpntf_state, cpntf); +list_del_init (&cps->cpntf); +free_cpntf_state (cps->net, cps); + }list_for_each_safe (pos, next, &reaplist) { clp = list_entry(pos, struct nfs4_client, cl_lru); dprintk("NFSD: purging unused client (clientid %08x)\n",@@ -5576,6 +5662,16 @@ static __be32 nfsd4_validate_stateid(struct nfs4_client *cl, stateid_t *stateid)return 0 ; } +void free_cpntf_state (struct nfsd_net *nn,struct nfs4_cpntf_state *cps) +{ +spin_lock (&cps->cp_p_stid->sc_lock); +list_del (&cps->cp_list); + cps->cp_p_stid->sc_cp_list_size--; +spin_unlock (&cps->cp_p_stid->sc_lock); +_free_copy_cpntf_stateid (nn, &cps->cp_stateid.stid); +nfs4_put_stid (cps->cp_p_stid); +kfree (cps); +}/* * Checks for stateid operations fs/nfsd/state.h +28
-2@@ -56,6 +56,13 @@stateid_opaque_t si_opaque; } stateid_t; +typedef struct { +stateid_t stid; +#define NFS4_COPY_STID 1 +#define NFS4_COPYNOTIFY_STID 2 +unsigned char sc_type; +}copy_stateid_t ; +#define STATEID_FMT "(%08x/%08x/%08x/%08x)"#define STATEID_VAL (s) \ (s)->si_opaque.so_clid.cl_boot, \@@ -96,6 +103,8 @@ struct nfs4_stid {#define NFS4_REVOKED_DELEG_STID 16#define NFS4_CLOSED_DELEG_STID 32#define NFS4_LAYOUT_STID 64 +struct list_head sc_cp_list; +size_t sc_cp_list_size;unsigned char sc_type;stateid_t sc_stateid;spinlock_t sc_lock;@@ -104,6 +113,18 @@ struct nfs4_stid {void (*sc_free)(struct nfs4_stid *); }; +/* Keep a list of stateids issued by the COPY_NOTIFY, associate it with the +* parent OPEN/LOCK/DELEG stateid. +*/ +struct nfs4_cpntf_state { +copy_stateid_t cp_stateid; +struct list_head cp_list;/* per parent nfs4_stid */ +struct nfs4_stid *cp_p_stid;/* pointer to parent */ +struct list_head cpntf;/* list of copy_notifies */ +time_t cpntf_time;/* last time stateid used */ +struct nfsd_net *net; +}; +/* * Represents a delegation stateid. The nfs4_client holds references to these * and they are put when it is being destroyed or when the delegation is @@ -367,6 +388,8 @@ struct nfs4_client {struct net *net;struct list_head async_copies;/* list of async copies */ spinlock_t async_lock;/* lock for async copies */ +struct list_head copy_notifies;/* list of copy notify stids */ +spinlock_t cpntf_lock;/* lock for copy_notifies */ };/* struct nfs4_client_reset @@ -623,8 +646,10 @@ __be32 nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate, struct nfs4_stid **s, struct nfsd_net *nn);struct nfs4_stid *nfs4_alloc_stid (struct nfs4_client *cl,struct kmem_cache *slab,void (*sc_free)(struct nfs4_stid *)); -int nfs4_init_cp_state (struct nfsd_net *nn,struct nfsd4_copy *copy); -void nfs4_free_cp_state (struct nfsd4_copy *copy); +int nfs4_init_copy_state (struct nfsd_net *nn,struct nfsd4_copy *copy); +void nfs4_free_copy_state (struct nfsd4_copy *copy); +struct nfs4_cpntf_state *nfs4_alloc_init_cpntf_state (struct nfsd_net *nn, +struct nfs4_stid *p_stid);void nfs4_unhash_stid (struct nfs4_stid *s);void nfs4_put_stid (struct nfs4_stid *s);void nfs4_inc_and_copy_stateid (stateid_t *dst,struct nfs4_stid *stid);@@ -654,6 +679,7 @@ extern struct nfs4_client_reclaim *nfs4_client_to_reclaim(struct xdr_netobj nameextern void nfs4_put_copy (struct nfsd4_copy *copy);extern struct nfsd4_copy *find_async_copy (struct nfs4_client *clp,stateid_t *staetid); +extern void free_cpntf_state (struct nfsd_net *nn,struct nfs4_cpntf_state *cps);static inline void get_nfs4_file (struct nfs4_file *fi) {refcount_inc (&fi->fi_ref);fs/nfsd/xdr4.h +1
-2@@ -543,7 +543,7 @@ struct nfsd4_copy {struct file *file_src;struct file *file_dst; -stateid_t cp_stateid; +copy_stateid_t cp_stateid;struct list_head copies;struct task_struct *copy_task; --1.8.3.1
Re: [PATCH v5 5/9] NFSD add COPY_NOTIFY operation
Quoted 9 lines · Olga Kornievskaia
Quoted 6 lines · Olga Kornievskaia
While this passes my testing, in theory this allows for the race that we get the copy notify size but then
offload_cancelarrive and change the value. Thenrefcount_sub_andtest_checkwould have an incorrect value (can subtract larger than an actual reference count). I have no solution for that as there is norefcount_sub_and_lock()that will allow to decrement by a multiple under a lock. Thoughts?
Quoted 7 lines
I tried not to use the client's
cl_lockbut instead use a specific lock to protect the copy notifies stateid on the stateid list. But since stateid's reference counter (sc_count) is protected by it, I think by getting rid of the special lock and usingcl_lockwill solve the problem of coordinating access between thesc_countand thecopy_notifystateid list. Are the any problems with using such a big lock?
Probably not. But it can be confusing when a single lock is used for several different things. A comment explaining why you need it might help.
--b.
Re: [PATCH v5 5/9] NFSD add COPY_NOTIFY operation
Quoted 18 lines · J. Bruce Fields
Quoted 9 lines · Olga Kornievskaia
Quoted 6 lines · Olga Kornievskaia
While this passes my testing, in theory this allows for the race that we get the copy notify size but then
offload_cancelarrive and change the value. Thenrefcount_sub_andtest_checkwould have an incorrect value (can subtract larger than an actual reference count). I have no solution for that as there is norefcount_sub_and_lock()that will allow to decrement by a multiple under a lock. Thoughts?Quoted 7 lines
I tried not to use the client's
cl_lockbut instead use a specific lock to protect the copy notifies stateid on the stateid list. But since stateid's reference counter (sc_count) is protected by it, I think by getting rid of the special lock and usingcl_lockwill solve the problem of coordinating access between thesc_countand thecopy_notifystateid list. Are the any problems with using such a big lock?
Quoted 3 lines
Probably not. But it can be confusing when a single lock is used for several different things. A comment explaining why you need it might help.
(By which I mean, a comment in the code.)
--b.
Re: [PATCH v5 5/9] NFSD add COPY_NOTIFY operation
In reply to J. Bruce FieldsQuoted 19 lines · J. Bruce Fields
Quoted 9 lines · Olga Kornievskaia
Quoted 6 lines · Olga Kornievskaia
While this passes my testing, in theory this allows for the race that we get the copy notify size but then
offload_cancelarrive and change the value. Thenrefcount_sub_andtest_checkwould have an incorrect value (can subtract larger than an actual reference count). I have no solution for that as there is norefcount_sub_and_lock()that will allow to decrement by a multiple under a lock. Thoughts?Quoted 7 lines
I tried not to use the client's
cl_lockbut instead use a specific lock to protect the copy notifies stateid on the stateid list. But since stateid's reference counter (sc_count) is protected by it, I think by getting rid of the special lock and usingcl_lockwill solve the problem of coordinating access between thesc_countand thecopy_notifystateid list. Are the any problems with using such a big lock?
Quoted 3 lines
Probably not. But it can be confusing when a single lock is used for several different things. A comment explaining why you need it might help.
While holding the client's cl_lock to manipulate the list of copy notify stateids solves the refcount problem. It generates a different problem for the laundromat thread. There, client list is traversed already holding the cl_lock, so I can't call routines to free copy_notify stateid because in turn it calls nfs4_put_stid() which wants to take the cl_lock. Putting the copy_notify stateid on the reaplist and then I lose a pointer to the client structure that I need to take the lock. Then it seems the nfs4_cpntf_state structure would need to keep a pointer to the client structure but then I get a problem of making sure the nfs4_client structure isn't going away and because it even a bigger mess.
I think I need to remove the code in the laundromat that looks for the not referenced copy_notifies stateid and just rely on cleaning on the removal of the stateid (basically what I originally had). Or I need to rely on the client to always send FREE_STATEID. I don't see other options, do you?
Quoted 2 lines
--b.
Re: [PATCH v5 5/9] NFSD add COPY_NOTIFY operation
Quoted 37 lines · Olga Kornievskaia
Quoted 19 lines · J. Bruce Fields
Quoted 9 lines · Olga Kornievskaia
Quoted 6 lines · Olga Kornievskaia
While this passes my testing, in theory this allows for the race that we get the copy notify size but then
offload_cancelarrive and change the value. Thenrefcount_sub_andtest_checkwould have an incorrect value (can subtract larger than an actual reference count). I have no solution for that as there is norefcount_sub_and_lock()that will allow to decrement by a multiple under a lock. Thoughts?Quoted 7 lines
I tried not to use the client's
cl_lockbut instead use a specific lock to protect the copy notifies stateid on the stateid list. But since stateid's reference counter (sc_count) is protected by it, I think by getting rid of the special lock and usingcl_lockwill solve the problem of coordinating access between thesc_countand thecopy_notifystateid list. Are the any problems with using such a big lock?Quoted 3 lines
Probably not. But it can be confusing when a single lock is used for several different things. A comment explaining why you need it might help.
While holding the client's
cl_lockto manipulate the list of copy notify stateids solves the refcount problem. It generates a different problem for the laundromat thread. There, client list is traversed already holding thecl_lock, so I can't call routines to freecopy_notifystateid because in turn it callsnfs4_put_stid()which wants to take thecl_lock. Putting thecopy_notifystateid on the reaplist and then I lose a pointer to the client structure that I need to take the lock. Then it seems thenfs4_cpntf_statestructure would need to keep a pointer to the client structure but then I get a problem of making sure thenfs4_clientstructure isn't going away and because it even a bigger mess.
Quoted 5 lines
I think I need to remove the code in the laundromat that looks for the not referenced
copy_notifiesstateid and just rely on cleaning on the removal of the stateid (basically what I originally had). Or I need to rely on the client to always sendFREE_STATEID. I don't see other options, do you?
Ignore this Bruce. Trond gave me a good idea and gets me unstuck.
Quoted 3 lines
Quoted 3 lines
--b.
Re: [PATCH v5 5/9] NFSD add COPY_NOTIFY operation
Quoted 40 lines · Olga Kornievskaia
Quoted 37 lines · Olga Kornievskaia
Quoted 19 lines · J. Bruce Fields
Quoted 9 lines · Olga Kornievskaia
Quoted 6 lines · Olga Kornievskaia
While this passes my testing, in theory this allows for the race that we get the copy notify size but then
offload_cancelarrive and change the value. Thenrefcount_sub_andtest_checkwould have an incorrect value (can subtract larger than an actual reference count). I have no solution for that as there is norefcount_sub_and_lock()that will allow to decrement by a multiple under a lock. Thoughts?Quoted 7 lines
I tried not to use the client's
cl_lockbut instead use a specific lock to protect the copy notifies stateid on the stateid list. But since stateid's reference counter (sc_count) is protected by it, I think by getting rid of the special lock and usingcl_lockwill solve the problem of coordinating access between thesc_countand thecopy_notifystateid list. Are the any problems with using such a big lock?Quoted 3 lines
Probably not. But it can be confusing when a single lock is used for several different things. A comment explaining why you need it might help.
While holding the client's
cl_lockto manipulate the list of copy notify stateids solves the refcount problem. It generates a different problem for the laundromat thread. There, client list is traversed already holding thecl_lock, so I can't call routines to freecopy_notifystateid because in turn it callsnfs4_put_stid()which wants to take thecl_lock. Putting thecopy_notifystateid on the reaplist and then I lose a pointer to the client structure that I need to take the lock. Then it seems thenfs4_cpntf_statestructure would need to keep a pointer to the client structure but then I get a problem of making sure thenfs4_clientstructure isn't going away and because it even a bigger mess.
Quoted 7 lines
Quoted 5 lines
I think I need to remove the code in the laundromat that looks for the not referenced
copy_notifiesstateid and just rely on cleaning on the removal of the stateid (basically what I originally had). Or I need to rely on the client to always sendFREE_STATEID. I don't see other options, do you?Ignore this Bruce. Trond gave me a good idea and gets me unstuck.
Hi Bruce,
I'm stuck again. The idea that Trond gave me is to instead of storing the pointer to the stateid, (copy) store the stateid_t structure itself and then use it to look it up the appropriate nfs4_stid.
The problem with that is when nfsd4_lookup_stateid() is called it takes is a compound state (cstate) which has a client pointer and during the lookup it's verified that the client looking up the stateid is the same that generate the stateid which is not the case with copy offload.
I tried also saving a cl_clientid and using that to lookup the nfs4_client that's needed for the stateid lookup but I'm not sure that's possible. lookup_clientid() calls find_client_in_id_table() and always passes "false" for sessions args. Original client has minor version 2 and then the check if (clp->minor_versions != sessions) fails. I don't understand what this logic is suppose to check.
Should I be writing special version of the lookup_clientid that ignores that check (when called in the path of the copy_notify verification)? Or any other ideas of how to get passed this would be appreciated it.
Thank you.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
--b.
Re: [PATCH v5 5/9] NFSD add COPY_NOTIFY operation
Quoted 17 lines · Olga Kornievskaia
I'm stuck again. The idea that Trond gave me is to instead of storing the pointer to the stateid, (copy) store the
stateid_tstructure itself and then use it to look it up the appropriatenfs4_stid.The problem with that is when
nfsd4_lookup_stateid()is called it takes is a compound state (cstate) which has a client pointer and during the lookup it's verified that the client looking up the stateid is the same that generate the stateid which is not the case with copy offload.I tried also saving a
cl_clientidand using that to lookup thenfs4_clientthat's needed for the stateid lookup but I'm not sure that's possible.lookup_clientid()callsfind_client_in_id_table()and always passes "false" for sessions args. Original client has minor version 2 and then the check if (clp->minor_versions!= sessions) fails. I don't understand what this logic is suppose to check.
Quoted 4 lines
Should I be writing special version of the
lookup_clientidthat ignores that check (when called in the path of thecopy_notifyverification)? Or any other ideas of how to get passed this would be appreciated it.
Yeah, I think you may want to do the clientid lookup by hand under cl_lock, and take the cl_ref?
Or turn things around and ensure that copy stateid's are destroyed before clients are.
--b.
PATCH v5 6/9 NFSD check stateids against copy stateids
In reply to Olga KornievskaiaIncoming stateid (used by a READ) could be a saved copy stateid. Update the copy notify timestamp (cpntf_time) with current time.
--- fs/nfsd/nfs4state.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)
fs/nfsd/nfs4state.c +39 -0
@@ -5672,6 +5672,43 @@ void free_cpntf_state(struct nfsd_net *nn, struct nfs4_cpntf_state *cps)nfs4_put_stid (cps->cp_p_stid);kfree (cps); } +/* +* A READ from an inter server to server COPY will have a +* copy stateid. Return the parent nfs4_stid. +*/ +static __be32 _find_cpntf_state (struct nfsd_net *nn,stateid_t *st, +struct nfs4_cpntf_state **cps) +{ +struct nfs4_cpntf_state *state =NULL ; + +if (st->si_opaque.so_clid.cl_id != nn->s2s_cp_cl_id) +return nfserr_bad_stateid; +spin_lock (&nn->s2s_cp_lock); + state =idr_find (&nn->s2s_cp_stateids, st->si_opaque.so_id); +if (state) +refcount_inc (&state->cp_p_stid->sc_count); +spin_unlock (&nn->s2s_cp_lock); +if (!state) +return nfserr_bad_stateid; + *cps = state; +return 0 ; +} + +static __be32 find_cpntf_state (struct nfsd_net *nn,stateid_t *st, +struct nfs4_stid **stid) +{ +__be32 status; +struct nfs4_cpntf_state *cps =NULL ; + + status =_find_cpntf_state (nn, st, &cps); +if (status) +return status; + + cps->cpntf_time =get_seconds (); + *stid = cps->cp_p_stid; + +return nfs_ok; +}/* * Checks for stateid operations
@@ -5704,6 +5741,8 @@ void free_cpntf_state(struct nfsd_net *nn, struct nfs4_cpntf_state *cps) status =nfsd4_lookup_stateid (cstate, stateid,NFS4_DELEG_STID |NFS4_OPEN_STID |NFS4_LOCK_STID , &s, nn); +if (status == nfserr_bad_stateid) + status =find_cpntf_state (nn, stateid, &s);if (status)return status; status = nfsd4_stid_check_stateid_generation(stateid, s,
Signature · 1 line
1.8.3.1
PATCH v5 7/9 NFSD generalize nfsd4_compound_state flag names
In reply to Olga KornievskaiaFrom: Olga Kornievskaia <[email protected]>
Allow for sid_flag field non-stateid use.
--- fs/nfsd/nfs4proc.c | 8 ++++---- fs/nfsd/nfs4state.c | 7 ++++--- fs/nfsd/xdr4.h | 6 +++--- 3 files changed, 11 insertions(+), 10 deletions(-)
fs/nfsd/nfs4proc.c +4 -4
@@ -531,9 +531,9 @@ static __be32 nfsd4_open_omfg(struct svc_rqst *rqstp, struct nfsd4_compound_statreturn nfserr_restorefh;fh_dup2 (&cstate->current_fh, &cstate->save_fh); -if (HAS_STATE_ID (cstate,SAVED_STATE_ID_FLAG )) { +if (HAS_CSTATE_FLAG (cstate,SAVED_STATE_ID_FLAG )) {memcpy (&cstate->current_stateid, &cstate->save_stateid,sizeof (stateid_t)); -SET_STATE_ID (cstate,CURRENT_STATE_ID_FLAG ); +SET_CSTATE_FLAG (cstate,CURRENT_STATE_ID_FLAG ); }return nfs_ok; }
@@ -543,9 +543,9 @@ static __be32 nfsd4_open_omfg(struct svc_rqst *rqstp, struct nfsd4_compound_stat union nfsd4_op_u *u) {fh_dup2 (&cstate->save_fh, &cstate->current_fh); -if (HAS_STATE_ID (cstate,CURRENT_STATE_ID_FLAG )) { +if (HAS_CSTATE_FLAG (cstate,CURRENT_STATE_ID_FLAG )) {memcpy (&cstate->save_stateid, &cstate->current_stateid,sizeof (stateid_t)); -SET_STATE_ID (cstate,SAVED_STATE_ID_FLAG ); +SET_CSTATE_FLAG (cstate,SAVED_STATE_ID_FLAG ); }return nfs_ok; }
fs/nfsd/nfs4state.c +4 -3
@@ -7900,7 +7900,8 @@ static int nfs4_state_create_net(struct net *net)static void get_stateid (struct nfsd4_compound_state *cstate,stateid_t *stateid) { -if (HAS_STATE_ID (cstate,CURRENT_STATE_ID_FLAG ) &&CURRENT_STATEID (stateid)) +if (HAS_CSTATE_FLAG (cstate,CURRENT_STATE_ID_FLAG ) && +CURRENT_STATEID (stateid))memcpy (stateid, &cstate->current_stateid,sizeof (stateid_t)); }
@@ -7909,14 +7910,14 @@ static int nfs4_state_create_net(struct net *net) {if (cstate->minorversion) {memcpy (&cstate->current_stateid, stateid,sizeof (stateid_t)); -SET_STATE_ID (cstate,CURRENT_STATE_ID_FLAG ); +SET_CSTATE_FLAG (cstate,CURRENT_STATE_ID_FLAG ); } }void clear_current_stateid (struct nfsd4_compound_state *cstate) { -CLEAR_STATE_ID (cstate,CURRENT_STATE_ID_FLAG ); +CLEAR_CSTATE_FLAG (cstate,CURRENT_STATE_ID_FLAG ); }/*
fs/nfsd/xdr4.h +3 -3
@@ -46,9 +46,9 @@#define CURRENT_STATE_ID_FLAG (1<<0)#define SAVED_STATE_ID_FLAG (1<<1) -#define SET_STATE_ID (c, f) ((c)->sid_flags |= (f)) -#define HAS_STATE_ID (c, f) ((c)->sid_flags & (f)) -#define CLEAR_STATE_ID (c, f) ((c)->sid_flags &= ~(f)) +#define SET_CSTATE_FLAG (c, f) ((c)->sid_flags |= (f)) +#define HAS_CSTATE_FLAG (c, f) ((c)->sid_flags & (f)) +#define CLEAR_CSTATE_FLAG (c, f) ((c)->sid_flags &= ~(f))struct nfsd4_compound_state {struct svc_fh current_fh;
Signature · 1 line
1.8.3.1
PATCH v5 8/9 NFSD: allow inter server COPY to have a STALE source server fh
In reply to Olga KornievskaiaThe inter server to server COPY source server filehandle is a foreign filehandle as the COPY is sent to the destination server.
--- fs/nfsd/Kconfig | 10 +++++++++ fs/nfsd/nfs4proc.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++---- fs/nfsd/nfsfh.h | 5 ++++- fs/nfsd/xdr4.h | 1 + 4 files changed, 70 insertions(+), 5 deletions(-)
fs/nfsd/Kconfig +10 -0
@@ -132,6 +132,16 @@ config NFSD_FLEXFILELAYOUT If unsure, say N. +config NFSD_V4_2_INTER_SSC + bool "NFSv4.2 inter server to server COPY" + depends on NFSD_V4 && NFS_V4_1 && NFS_V4_2 + help + This option enables support for NFSv4.2 inter server to + server copy where the destination server calls the NFSv4.2 + client to read the data to copy from the source server. + + If unsure, say N. + config NFSD_V4_SECURITY_LABEL bool "Provide Security Label support for NFSv4 server" depends on NFSD_V4 && SECURITY
fs/nfsd/nfs4proc.c +55 -4
92 lines not on this page · raw message
fs/nfsd/nfsfh.h +4 -1
@@ -35,7 +35,7 @@ static inline ino_t u32_to_ino_t(__u32 uino)bool fh_locked;/* inode locked by us */ bool fh_want_write;/* remount protection taken */ - +int fh_flags;/* FH flags */ #ifdef CONFIG_NFSD_V3 bool fh_post_saved;/* post-op attrs saved */ bool fh_pre_saved;/* pre-op attrs saved */
@@ -56,6 +56,9 @@ static inline ino_t u32_to_ino_t(__u32 uino)#endif /* CONFIG_NFSD_V3 */ } svc_fh; +#define NFSD4_FH_FOREIGN (1<<0) +#define SET_FH_FLAG (c, f) ((c)->fh_flags |= (f)) +#define HAS_FH_FLAG (c, f) ((c)->fh_flags & (f))enum nfsd_fsid {FSID_DEV =0 ,
fs/nfsd/xdr4.h +1 -0
@@ -221,6 +221,7 @@ struct nfsd4_lookup {struct nfsd4_putfh {u32 pf_fhlen;/* request */ char *pf_fhval;/* request */ +bool no_verify;/* represents foreigh fh */ };struct nfsd4_open {
Signature · 1 line
1.8.3.1
PATCH v5 9/9 NFSD add nfs4 inter ssc to nfsd4_copy
In reply to Olga KornievskaiaFrom: Olga Kornievskaia <[email protected]>
Given a universal address, mount the source server from the destination server. Use an internal mount. Call the NFS client nfs42_ssc_open to obtain the NFS struct file suitable for nfsd_copy_range.
Ability to do "inter" server-to-server depends on the an nfsd kernel parameter "inter_copy_offload_enable".
--- fs/nfsd/nfs4proc.c | 295 ++++++++++++++++++++++++++++++++++++++++++++++++---- fs/nfsd/nfs4state.c | 6 +- fs/nfsd/nfssvc.c | 6 ++ fs/nfsd/state.h | 3 + fs/nfsd/xdr4.h | 5 + 5 files changed, 291 insertions(+), 24 deletions(-)
fs/nfsd/nfs4proc.c +274 -21
@@ -1153,6 +1153,209 @@ void nfsd4_shutdown_copy(struct nfs4_client *clp)while ((copy =nfsd4_get_copy (clp)) !=NULL )nfsd4_stop_copy (copy); } +#ifdef CONFIG_NFSD_V4_2_INTER_SSC + +extern struct file *nfs42_ssc_open (struct vfsmount *ss_mnt, +struct nfs_fh *src_fh, +nfs4_stateid *stateid); +extern void nfs42_ssc_close (struct file *filep); + +extern void nfs_sb_deactive (struct super_block *sb); + +#define NFSD42_INTERSSC_MOUNTOPS "vers=4.2,addr=%s,sec=sys" + +/** +* Support one copy source server for now. +*/ +static __be32 +nfsd4_interssc_connect (struct nl4_server *nss,struct svc_rqst *rqstp, +struct vfsmount **mount) +{ +struct file_system_type *type; +struct vfsmount *ss_mnt; +struct nfs42_netaddr *naddr; +struct sockaddr_storage tmp_addr; +size_t tmp_addrlen, match_netid_len =3 ; +char *startsep ="" , *endsep ="" , *match_netid ="tcp" ; +char *ipaddr, *dev_name, *raw_data; +int len, raw_len, status = -EINVAL ; + + naddr = &nss->u.nl4_addr; + tmp_addrlen =rpc_uaddr2sockaddr (SVC_NET (rqstp), naddr->addr, + naddr->addr_len, + (struct sockaddr *)&tmp_addr, +sizeof (tmp_addr)); +if (tmp_addrlen ==0 ) + goto out_err; + +if (tmp_addr.ss_family ==AF_INET6 ) { + startsep ="[" ; + endsep ="]" ; + match_netid ="tcp6" ; + match_netid_len =4 ; + } + +if (naddr->netid_len != match_netid_len || +strncmp (naddr->netid, match_netid, naddr->netid_len)) + goto out_err; + +/* Construct the raw data for the vfs_kern_mount call */ + len =RPC_MAX_ADDRBUFLEN +1 ; + ipaddr =kzalloc (len,GFP_KERNEL ); +if (!ipaddr) + goto out_err; + +rpc_ntop ((struct sockaddr *)&tmp_addr, ipaddr, len); + +/* 2 for ipv6 endsep and startsep. 3 for ":/" and trailing '/0'*/ + + raw_len =strlen (NFSD42_INTERSSC_MOUNTOPS ) +strlen (ipaddr); + raw_data =kzalloc (raw_len,GFP_KERNEL ); +if (!raw_data) + goto out_free_ipaddr; + +snprintf (raw_data, raw_len,NFSD42_INTERSSC_MOUNTOPS , ipaddr); + + status = -ENODEV ; + type =get_fs_type ("nfs" ); +if (!type) + goto out_free_rawdata; + +/* Set the server:<export> for the vfs_kern_mount call */ + dev_name =kzalloc (len +5 ,GFP_KERNEL ); +if (!dev_name) + goto out_free_rawdata; +snprintf (dev_name, len +5 ,"%s%s%s:/" , startsep, ipaddr, endsep); + +/* Use an 'internal' mount: SB_KERNMOUNT -> MNT_INTERNAL */ + ss_mnt =vfs_kern_mount (type,SB_KERNMOUNT , dev_name, raw_data); +module_put (type->owner); +if (IS_ERR (ss_mnt)) + goto out_free_devname; + + status =0 ; + *mount = ss_mnt; + +out_free_devname: +kfree (dev_name); +out_free_rawdata: +kfree (raw_data); +out_free_ipaddr: +kfree (ipaddr); +out_err: +return status; +} + +static void +nfsd4_interssc_disconnect (struct vfsmount *ss_mnt) +{ +nfs_sb_deactive (ss_mnt->mnt_sb); +mntput (ss_mnt); +} + +/** +* nfsd4_setup_inter_ssc +* +* Verify COPY destination stateid. +* Connect to the source server with NFSv4.1. +* Create the source struct file for nfsd_copy_range. +* Called with COPY cstate: +* SAVED_FH: source filehandle +* CURRENT_FH: destination filehandle +* +* Returns errno (not nfserrxxx) +*/ +static __be32 +nfsd4_setup_inter_ssc (struct svc_rqst *rqstp, +struct nfsd4_compound_state *cstate, +struct nfsd4_copy *copy,struct vfsmount **mount) +{ +struct svc_fh *s_fh =NULL ; +stateid_t *s_stid = ©->cp_src_stateid; +__be32 status = -EINVAL ; + +/* Verify the destination stateid and set dst struct file*/ + status =nfs4_preprocess_stateid_op (rqstp, cstate, &cstate->current_fh, + ©->cp_dst_stateid, +WR_STATE , ©->file_dst,NULL , +NULL ); +if (status) + goto out; + + status =nfsd4_interssc_connect (©->cp_src, rqstp, mount); +if (status) + goto out; + + s_fh = &cstate->save_fh; + + copy->c_fh.size = s_fh->fh_handle.fh_size; +memcpy (copy->c_fh.data, &s_fh->fh_handle.fh_base, copy->c_fh.size); + copy->stateid.seqid = s_stid->si_generation; +memcpy (copy->stateid.other, (void *)&s_stid->si_opaque, +sizeof (stateid_opaque_t)); + + status =0 ; +out: +return status; +} + +static void +nfsd4_cleanup_inter_ssc (struct vfsmount *ss_mnt,struct file *src, +struct file *dst) +{ +nfs42_ssc_close (src); +fput (src); +fput (dst); +mntput (ss_mnt); +} + +#else /* CONFIG_NFSD_V4_2_INTER_SSC */ + +static __be32 +nfsd4_setup_inter_ssc (struct svc_rqst *rqstp, +struct nfsd4_compound_state *cstate, +struct nfsd4_copy *copy, +struct vfsmount **mount) +{ + *mount =NULL ; +return -EINVAL ; +} + +static void +nfsd4_cleanup_inter_ssc (struct vfsmount *ss_mnt,struct file *src, +struct file *dst) +{ +} + +static void +nfsd4_interssc_disconnect (struct vfsmount *ss_mnt) +{ +} + +static struct file *nfs42_ssc_open (struct vfsmount *ss_mnt, +struct nfs_fh *src_fh, +nfs4_stateid *stateid) +{ +return NULL ; +} +#endif /* CONFIG_NFSD_V4_2_INTER_SSC */ + +static __be32 +nfsd4_setup_intra_ssc (struct svc_rqst *rqstp, +struct nfsd4_compound_state *cstate, +struct nfsd4_copy *copy) +{ +return nfsd4_verify_copy (rqstp, cstate, ©->cp_src_stateid, + ©->file_src, ©->cp_dst_stateid, + ©->file_dst,NULL ); +} + +static void +nfsd4_cleanup_intra_ssc (struct file *src,struct file *dst) +{ +fput (src); +fput (dst); +}static void nfsd4_cb_offload_release (struct nfsd4_callback *cb) {
@@ -1217,12 +1420,16 @@ static __be32 nfsd4_do_copy(struct nfsd4_copy *copy, bool sync) status = nfs_ok; } -fput (copy->file_src); -fput (copy->file_dst); +if (!copy->cp_intra)/* Inter server SSC */ +nfsd4_cleanup_inter_ssc (copy->ss_mnt, copy->file_src, + copy->file_dst); +else +nfsd4_cleanup_intra_ssc (copy->file_src, copy->file_dst); +return status; } -static void dup_copy_fields (struct nfsd4_copy *src,struct nfsd4_copy *dst) +static int dup_copy_fields (struct nfsd4_copy *src,struct nfsd4_copy *dst) { dst->cp_src_pos = src->cp_src_pos; dst->cp_dst_pos = src->cp_dst_pos;
@@ -1232,8 +1439,17 @@ static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)memcpy (&dst->fh, &src->fh,sizeof (src->fh)); dst->cp_clp = src->cp_clp; dst->file_dst =get_file (src->file_dst); - dst->file_src =get_file (src->file_src); + dst->cp_intra = src->cp_intra; +if (src->cp_intra)/* for inter, file_src doesn't exist yet */ + dst->file_src =get_file (src->file_src);memcpy (&dst->cp_stateid, &src->cp_stateid,sizeof (src->cp_stateid)); +memcpy (&dst->cp_src, &src->cp_src,sizeof (struct nl4_server )); +memcpy (&dst->stateid, &src->stateid,sizeof (src->stateid)); +memcpy (&dst->c_fh, &src->c_fh,sizeof (src->c_fh)); + dst->ss_mnt = src->ss_mnt; + +return 0 ; + }static void cleanup_async_copy (struct nfsd4_copy *copy)
@@ -1252,7 +1468,18 @@ static int nfsd4_do_async_copy(void *data)struct nfsd4_copy *copy = (struct nfsd4_copy *)data;struct nfsd4_copy *cb_copy; +if (!copy->cp_intra) {/* Inter server SSC */ + copy->file_src =nfs42_ssc_open (copy->ss_mnt, ©->c_fh, + ©->stateid); +if (IS_ERR (copy->file_src)) { + copy->nfserr = nfserr_offload_denied; +nfsd4_interssc_disconnect (copy->ss_mnt); + goto do_callback; + } + } + copy->nfserr =nfsd4_do_copy (copy,0 ); +do_callback: cb_copy =kzalloc (sizeof (struct nfsd4_copy ),GFP_KERNEL );if (!cb_copy) goto out;
@@ -1276,11 +1503,20 @@ static int nfsd4_do_async_copy(void *data)__be32 status;struct nfsd4_copy *async_copy =NULL ; - status =nfsd4_verify_copy (rqstp, cstate, ©->cp_src_stateid, - ©->file_src, ©->cp_dst_stateid, - ©->file_dst,NULL ); -if (status) - goto out; +if (!copy->cp_intra) {/* Inter server SSC */ +if (!inter_copy_offload_enable || copy->cp_synchronous) { + status = nfserr_notsupp; + goto out; + } + status =nfsd4_setup_inter_ssc (rqstp, cstate, copy, + ©->ss_mnt); +if (status) +return nfserr_offload_denied; + }else { + status =nfsd4_setup_intra_ssc (rqstp, cstate, copy); +if (status) +return status; + } copy->cp_clp = cstate->clp; memcpy(©->fh, &cstate->current_fh.fh_handle,
@@ -1291,15 +1527,15 @@ static int nfsd4_do_async_copy(void *data) status =nfserrno (-ENOMEM ); async_copy =kzalloc (sizeof (struct nfsd4_copy ),GFP_KERNEL );if (!async_copy) - goto out; -if (!nfs4_init_copy_state (nn, copy)) { -kfree (async_copy); - goto out; - } + goto out_err; +if (!nfs4_init_copy_state (nn, copy)) + goto out_err;refcount_set (&async_copy->refcount,1 );memcpy (©->cp_res.cb_stateid, ©->cp_stateid,sizeof (copy->cp_stateid)); -dup_copy_fields (copy, async_copy); + status =dup_copy_fields (copy, async_copy); +if (status) + goto out_err; async_copy->copy_task =kthread_create (nfsd4_do_async_copy, async_copy,"%s" ,"copy thread" );if (IS_ERR (async_copy->copy_task))
@@ -1310,12 +1546,16 @@ static int nfsd4_do_async_copy(void *data)spin_unlock (&async_copy->cp_clp->async_lock);wake_up_process (async_copy->copy_task); status = nfs_ok; - }else + }else { status =nfsd4_do_copy (copy,1 ); + } out:return status; out_err:cleanup_async_copy (async_copy); + status =nfserrno (-ENOMEM ); +if (!copy->cp_intra) +nfsd4_interssc_disconnect (copy->ss_mnt); goto out; }
@@ -1326,7 +1566,7 @@ struct nfsd4_copy *spin_lock (&clp->async_lock);list_for_each_entry (copy, &clp->async_copies, copies) { -if (memcmp (©->cp_stateid, stateid,NFS4_STATEID_SIZE )) +if (memcmp (©->cp_stateid.stid, stateid,NFS4_STATEID_SIZE ))continue ;refcount_inc (©->refcount);spin_unlock (&clp->async_lock);
@@ -1347,12 +1587,25 @@ struct nfsd4_copy *struct nfs4_client *clp = cstate->clp; copy =find_async_copy (clp, &os->stateid); -if (copy) +if (!copy) { +struct nfs4_cpntf_state *cps =NULL ; +struct nfsd_net *nn =net_generic (SVC_NET (rqstp), nfsd_net_id); + + status =find_internal_cpntf_state (nn, &os->stateid, &cps); +if (status) +return status; +if (cps) { +spin_lock (&clp->cpntf_lock); +list_del (&cps->cpntf); +spin_unlock (&clp->cpntf_lock); +nfs4_put_stid (cps->cp_p_stid); +free_cpntf_state (nn, cps); +return nfs_ok; + } + }else nfsd4_stop_copy (copy); -else - status = nfserr_bad_stateid; -return status; +return nfs_ok; } static __be32
fs/nfsd/nfs4state.c +3 -3
@@ -5676,8 +5676,8 @@ void free_cpntf_state(struct nfsd_net *nn, struct nfs4_cpntf_state *cps)* A READ from an inter server to server COPY will have a * copy stateid. Return the parent nfs4_stid. */ -static __be32 _find_cpntf_state (struct nfsd_net *nn,stateid_t *st, -struct nfs4_cpntf_state **cps) +__be32 find_internal_cpntf_state (struct nfsd_net *nn,stateid_t *st, +struct nfs4_cpntf_state **cps) {struct nfs4_cpntf_state *state =NULL ;
@@ -5700,7 +5700,7 @@ static __be32 find_cpntf_state(struct nfsd_net *nn, stateid_t *st,__be32 status;struct nfs4_cpntf_state *cps =NULL ; - status =_find_cpntf_state (nn, st, &cps); + status =find_internal_cpntf_state (nn, st, &cps);if (status)return status;
fs/nfsd/nfssvc.c +6 -0
@@ -30,6 +30,12 @@#define NFSDDBG_FACILITY NFSDDBG_SVC +bool inter_copy_offload_enable; +EXPORT_SYMBOL_GPL (inter_copy_offload_enable); +module_param (inter_copy_offload_enable, bool,0644 ); +MODULE_PARM_DESC (inter_copy_offload_enable, +"Enable inter server to server copy offload. Default: false" ); +extern struct svc_program nfsd_program;static int nfsd (void *vrqstp);#if defined(CONFIG_NFSD_V2_ACL ) || defined(CONFIG_NFSD_V3_ACL )
fs/nfsd/state.h +3 -0
@@ -680,6 +680,9 @@ extern struct nfs4_client_reclaim *nfs4_client_to_reclaim(struct xdr_netobj nameextern struct nfsd4_copy *find_async_copy (struct nfs4_client *clp,stateid_t *staetid);extern void free_cpntf_state (struct nfsd_net *nn,struct nfs4_cpntf_state *cps); +extern __be32 find_internal_cpntf_state (struct nfsd_net *nn,stateid_t *st, +struct nfs4_cpntf_state **cps); +static inline void get_nfs4_file (struct nfs4_file *fi) {refcount_inc (&fi->fi_ref);
fs/nfsd/xdr4.h +5 -0
@@ -550,7 +550,12 @@ struct nfsd4_copy {struct task_struct *copy_task;refcount_t refcount;bool stopped; + +struct vfsmount *ss_mnt; +struct nfs_fh c_fh; +nfs4_stateid stateid; }; +extern bool inter_copy_offload_enable;struct nfsd4_seek {/* request */
Signature · 1 line
1.8.3.1
kbuild test robot
build botRe: [PATCH v5 9/9] NFSD add nfs4 inter ssc to nfsd4_copy
Thank you for the patch! Yet something to improve:
Hi Olga,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on nfsd/nfsd-next]
[cannot apply to v5.3-rc3 next-20190809]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Olga-Kornievskaia/server-side-support-for-inter-SSC-copy/20190811-120551 base: git://linux-nfs.org/~bfields/linux.git nfsd-next config: x86_64-lkp (attached as .config) compiler: gcc-7 (Debian 7.4.0-10) 7.4.0 reproduce: # save the attached .config to linux build tree make ARCH=x86_64
If you fix the issue, kindly add following tag
All errors (new ones prefixed by >>):
In file included from fs//nfsd/state.h:42:0,
from fs//nfsd/xdr4.h:40,
from fs//nfsd/nfs4proc.c:44:
fs//nfsd/nfsd.h:391:16: warning: 'struct nfs42_netaddr' declared inside parameter list will not be visible outside of this definition or declaration
struct nfs42_netaddr *netaddr)
^~~~~~~~~~~~~
fs//nfsd/nfsd.h: In function 'nfsd4_set_netaddr':
fs//nfsd/nfsd.h:401:18: error: dereferencing pointer to incomplete type 'struct nfs42_netaddr'
sprintf(netaddr->netid, "tcp");
^~
In file included from fs//nfsd/nfs4proc.c:44:0:
fs//nfsd/xdr4.h: At top level:
fs//nfsd/xdr4.h:528:20: error: field 'cp_src' has incomplete type
struct nl4_server cp_src;
^~~~~~
fs//nfsd/xdr4.h:583:20: error: field 'cpn_dst' has incomplete type
struct nl4_server cpn_dst;
^~~~~~~
fs//nfsd/xdr4.h:589:20: error: field 'cpn_src' has incomplete type
struct nl4_server cpn_src;
^~~~~~~
fs//nfsd/nfs4proc.c: In function 'dup_copy_fields':
>> fs//nfsd/nfs4proc.c:1446:44: error: invalid application of 'sizeof' to incomplete type 'struct nl4_server'
memcpy(&dst->cp_src, &src->cp_src, sizeof(struct nl4_server));
^~~~~~
fs//nfsd/nfs4proc.c: In function 'nfsd4_copy_notify':
fs//nfsd/nfs4proc.c:1640:25: error: 'NL4_NETADDR' undeclared (first use in this function); did you mean 'RTM_GETADDR'?
cn->cpn_src.nl4_type = NL4_NETADDR;
^~~~~~~~~~~
RTM_GETADDR fs//nfsd/nfs4proc.c:1640:25: note: each undeclared identifier is reported only once for each function it appears in
vim +1446 fs//nfsd/nfs4proc.c
1431
1432 static int dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)
1433 {
1434 dst->cp_src_pos = src->cp_src_pos;
1435 dst->cp_dst_pos = src->cp_dst_pos;
1436 dst->cp_count = src->cp_count;
1437 dst->cp_synchronous = src->cp_synchronous;
1438 memcpy(&dst->cp_res, &src->cp_res, sizeof(src->cp_res));
1439 memcpy(&dst->fh, &src->fh, sizeof(src->fh));
1440 dst->cp_clp = src->cp_clp;
1441 dst->file_dst = get_file(src->file_dst);
1442 dst->cp_intra = src->cp_intra;
1443 if (src->cp_intra) /* for inter, file_src doesn't exist yet */
1444 dst->file_src = get_file(src->file_src);
1445 memcpy(&dst->cp_stateid, &src->cp_stateid, sizeof(src->cp_stateid));
> 1446 memcpy(&dst->cp_src, &src->cp_src, sizeof(struct nl4_server));
1447 memcpy(&dst->stateid, &src->stateid, sizeof(src->stateid));
1448 memcpy(&dst->c_fh, &src->c_fh, sizeof(src->c_fh));
1449 dst->ss_mnt = src->ss_mnt;
1450
1451 return 0;
1452
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
What do we know about the status of NFSv4.2 and COPY support on Netapp or other servers? In either the single-server or inter-server cases?
--b.
Quoted 3 lines · J. Bruce Fields
What do we know about the status of NFSv4.2 and COPY support on Netapp or other servers? In either the single-server or inter-server cases?
I'm unaware of any other (current/on-going) implementations. Netapp is interested in having (implementing) this 4.2 feature though (single server case for sure not sure about the inter-server).
Quoted 2 lines
--b.
Quoted 9 lines
Olga Kornievskaia wrote:
Quoted 3 lines · J. Bruce Fields
What do we know about the status of NFSv4.2 and COPY support on Netapp or other servers? In either the single-server or inter-server cases?
I'm unaware of any other (current/on-going) implementations. Netapp is interested in having (implementing) this 4.2 feature though (single server case for sure not sure about the inter-server).
Just fyi, I have implemented the single-server case for FreeBSD. (The code is currently in a projects area of the FreeBSD subversion repository, which makes it a little awkward to set up for testing at this point, but if anyone is interested, just email me.)
I haven't yet implemented the async case, but plan on doing so soon, rick
Quoted 7 lines · Rick Macklem
Quoted 7 lines
Olga Kornievskaia wrote:
Quoted 3 lines · J. Bruce Fields
What do we know about the status of NFSv4.2 and COPY support on Netapp or other servers? In either the single-server or inter-server cases?
Quoted 8 lines
Quoted 3 lines
I'm unaware of any other (current/on-going) implementations. Netapp is interested in having (implementing) this 4.2 feature though (single server case for sure not sure about the inter-server).
Just fyi, I have implemented the single-server case for FreeBSD. (The code is currently in a projects area of the FreeBSD subversion repository, which makes it a little awkward to set up for testing at this point, but if anyone is interested, just email me.)
That's great Rick. Have you done any interoperability with the linux implementation? Will the code go into the main freeBSD release (or is already there)?
Quoted 1 line
I haven't yet implemented the async case, but plan on doing so soon, rick
Quoted 12 lines · Olga Kornievskaia
Quoted 11 lines · Rick Macklem
Quoted 10 lines
Olga Kornievskaia wrote:
Quoted 3 lines · J. Bruce Fields
What do we know about the status of NFSv4.2 and COPY support on Netapp or other servers? In either the single-server or inter-server cases?
I'm unaware of any other (current/on-going) implementations. Netapp is interested in having (implementing) this 4.2 feature though (single server case for sure not sure about the inter-server).
Quoted 6 lines
Quoted 4 lines
Just fyi, I have implemented the single-server case for FreeBSD. (The code is currently in a projects area of the FreeBSD subversion repository, which makes it a little awkward to set up for testing at this point, but if anyone is interested, just email me.)
That's great Rick. Have you done any interoperability with the linux
implementation?
Not yet, but I am planning on doing so soon.
Quoted 2 lines
Will the code go into the main freeBSD release (or is already there)?
Not there yet. My target is FreeBSD13.
rick
Quoted 1 line
I haven't yet implemented the async case, but plan on doing so soon, rick