Do not hold clnt_fd_lock mutex during connect
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): [email protected]
v2 cover Do not hold clnt_fd_lock mutex during connect
The original patch was split in 3 new patches, addressing some concerns
brough in the first version, about thread safety of data accessed without the lock held.
It was also added an extra change to save the errno value before calling
syslog.
Original description of what the problem corrects follows:
An user reports that their application connects to multiple servers
through a rpc interface using libtirpc. When one of the servers misbehaves (goes down ungracefully or has a delay of a few seconds in the traffic flow), it was observed that the traffic from the client to other servers is decreased by the traffic anomaly of the failing server, i.e. traffic decreases or goes to 0 in all the servers.
When investigated further, specifically into the behavior of the libtirpc
at the time of the issue, it was observed that all of the application threads specifically interacting with libtirpc were locked into one single lock inside the libtirpc library. This was a race condition which had resulted in a deadlock and hence the resultant dip/stoppage of traffic.
As an experiment, the user removed the libtirpc from the application build
and used the standard glibc library for rpc communication. In that case, everything worked perfectly even in the time of the issue of server nodes misbehaving.
Paulo Andrade (3):
Make it clear rpc_createerr is thread safe
Record errno value before calling syslog
Do not hold a global mutex during connect
src/clnt_vc.c | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-)
Signature · 1 line
1.8.3.1
PATCH v2 1/3 Make it clear rpc_createerr is thread safe
Avoid hidding it under a macro, and also avoid multiple function
calls when accessing structure fields.
--- src/clnt_vc.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)
src/clnt_vc.c +11 -6
@@ -72,6 +72,8 @@#define CMGROUP_MAX 16#define SCM_CREDS 0x03/* process creds (struct cmsgcred) */ +#undef rpc_createerr/* make it clear it is a thread safe variable */ +/* * Credentials structure, used to verify the identity of a peer * process that has sent us a message. This is allocated by the
@@ -188,10 +190,11 @@ clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz) cl = (CLIENT *)mem_alloc (sizeof (*cl)); ct = (struct ct_data *)mem_alloc (sizeof (*ct));if ((cl == (CLIENT *)NULL ) || (ct == (struct ct_data *)NULL )) { +struct rpc_createerr *ce = &get_rpc_createerr (); (void )syslog (LOG_ERR , clnt_vc_errstr, clnt_vc_str, __no_mem_str); - rpc_createerr.cf_stat =RPC_SYSTEMERROR ; - rpc_createerr.cf_error.re_errno = errno; + ce->cf_stat =RPC_SYSTEMERROR ; + ce->cf_error.re_errno = errno; goto err; } ct->ct_addr.buf =NULL ;
@@ -235,15 +238,17 @@ clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz) slen =sizeof ss;if (getpeername (fd, (struct sockaddr *)&ss, &slen) <0 ) {if (errno !=ENOTCONN ) { - rpc_createerr.cf_stat =RPC_SYSTEMERROR ; - rpc_createerr.cf_error.re_errno = errno; +struct rpc_createerr *ce = &get_rpc_createerr (); + ce->cf_stat =RPC_SYSTEMERROR ; + ce->cf_error.re_errno = errno;mutex_unlock (&clnt_fd_lock);thr_sigsetmask (SIG_SETMASK , &(mask),NULL ); goto err; }if (connect (fd, (struct sockaddr *)raddr->buf, raddr->len) <0 ){ - rpc_createerr.cf_stat =RPC_SYSTEMERROR ; - rpc_createerr.cf_error.re_errno = errno; +struct rpc_createerr *ce = &get_rpc_createerr (); + ce->cf_stat =RPC_SYSTEMERROR ; + ce->cf_error.re_errno = errno;mutex_unlock (&clnt_fd_lock);thr_sigsetmask (SIG_SETMASK , &(mask),NULL ); goto err;
Signature · 1 line
1.8.3.1
Re: [Libtirpc-devel] [PATCH 1/3] Make it clear rpc_createerr is thread safe
Quoted 2 lines · Paulo Andrade
Avoid hidding it under a macro, and also avoid multiple function
calls when accessing structure fields.
I like this approach, the use of the macro was confusing to me. Including that define must always be the case though.
If others agree with this there probably needs to be follow up series with similar changes for the rest of the source.
I guess the main argument against it is that library users have come to expect using the macro and could be confused by this change, the exact opposite to the point of the change.
Thoughts anyone?
Quoted 57 lines
Signed-off-by Paulo Andrade <[email protected]>--- src/clnt_vc.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)src/clnt_vc.c +1
-0@@ -72,6 +72,8 @@#define CMGROUP_MAX 16#define SCM_CREDS 0x03/* process creds (struct cmsgcred) */ +#undef rpc_createerr/* make it clear it is a thread safe variable */
src/clnt_vc.c +10
-6+/* * Credentials structure, used to verify the identity of a peer * process that has sent us a message. This is allocated by the @@ -188,10 +190,11 @@ clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz) cl = (CLIENT *)mem_alloc (sizeof (*cl)); ct = (struct ct_data *)mem_alloc (sizeof (*ct));if ((cl == (CLIENT *)NULL ) || (ct == (struct ct_data *)NULL )) { +struct rpc_createerr *ce = &get_rpc_createerr (); (void )syslog (LOG_ERR , clnt_vc_errstr, clnt_vc_str, __no_mem_str); - rpc_createerr.cf_stat =RPC_SYSTEMERROR ; - rpc_createerr.cf_error.re_errno = errno; + ce->cf_stat =RPC_SYSTEMERROR ; + ce->cf_error.re_errno = errno; goto err; } ct->ct_addr.buf =NULL ;@@ -235,15 +238,17 @@ clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz) slen =sizeof ss;if (getpeername (fd, (struct sockaddr *)&ss, &slen) <0 ) {if (errno !=ENOTCONN ) { - rpc_createerr.cf_stat =RPC_SYSTEMERROR ; - rpc_createerr.cf_error.re_errno = errno; +struct rpc_createerr *ce = &get_rpc_createerr (); + ce->cf_stat =RPC_SYSTEMERROR ; + ce->cf_error.re_errno = errno;mutex_unlock (&clnt_fd_lock);thr_sigsetmask (SIG_SETMASK , &(mask),NULL ); goto err; }if (connect (fd, (struct sockaddr *)raddr->buf, raddr->len) <0 ){ - rpc_createerr.cf_stat =RPC_SYSTEMERROR ; - rpc_createerr.cf_error.re_errno = errno; +struct rpc_createerr *ce = &get_rpc_createerr (); + ce->cf_stat =RPC_SYSTEMERROR ; + ce->cf_error.re_errno = errno;mutex_unlock (&clnt_fd_lock);thr_sigsetmask (SIG_SETMASK , &(mask),NULL );goto err;
Re: [Libtirpc-devel] [PATCH 1/3] Make it clear rpc_createerr is thread safe
In reply to Paulo AndradeQuoted 4 lines · Paulo Andrade
Avoid hidding it under a macro, and also avoid multiple function
calls when accessing structure fields.Signed-off-by Paulo Andrade <[email protected]>
Committed...
steved.
Quoted 54 lines
--- src/clnt_vc.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)src/clnt_vc.c +11
-6@@ -72,6 +72,8 @@#define CMGROUP_MAX 16#define SCM_CREDS 0x03/* process creds (struct cmsgcred) */ +#undef rpc_createerr/* make it clear it is a thread safe variable */ +/* * Credentials structure, used to verify the identity of a peer * process that has sent us a message. This is allocated by the @@ -188,10 +190,11 @@ clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz) cl = (CLIENT *)mem_alloc (sizeof (*cl)); ct = (struct ct_data *)mem_alloc (sizeof (*ct));if ((cl == (CLIENT *)NULL ) || (ct == (struct ct_data *)NULL )) { +struct rpc_createerr *ce = &get_rpc_createerr (); (void )syslog (LOG_ERR , clnt_vc_errstr, clnt_vc_str, __no_mem_str); - rpc_createerr.cf_stat =RPC_SYSTEMERROR ; - rpc_createerr.cf_error.re_errno = errno; + ce->cf_stat =RPC_SYSTEMERROR ; + ce->cf_error.re_errno = errno; goto err; } ct->ct_addr.buf =NULL ;@@ -235,15 +238,17 @@ clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz) slen =sizeof ss;if (getpeername (fd, (struct sockaddr *)&ss, &slen) <0 ) {if (errno !=ENOTCONN ) { - rpc_createerr.cf_stat =RPC_SYSTEMERROR ; - rpc_createerr.cf_error.re_errno = errno; +struct rpc_createerr *ce = &get_rpc_createerr (); + ce->cf_stat =RPC_SYSTEMERROR ; + ce->cf_error.re_errno = errno;mutex_unlock (&clnt_fd_lock);thr_sigsetmask (SIG_SETMASK , &(mask),NULL ); goto err; }if (connect (fd, (struct sockaddr *)raddr->buf, raddr->len) <0 ){ - rpc_createerr.cf_stat =RPC_SYSTEMERROR ; - rpc_createerr.cf_error.re_errno = errno; +struct rpc_createerr *ce = &get_rpc_createerr (); + ce->cf_stat =RPC_SYSTEMERROR ; + ce->cf_error.re_errno = errno;mutex_unlock (&clnt_fd_lock);thr_sigsetmask (SIG_SETMASK , &(mask),NULL ); goto err;
PATCH v2 2/3 Record errno value before calling syslog
In reply to Paulo AndradeUnlikely to change, but stay in the safe side.
--- src/clnt_vc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
src/clnt_vc.c +2 -2
@@ -191,10 +191,10 @@ clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz) ct = (struct ct_data *)mem_alloc (sizeof (*ct));if ((cl == (CLIENT *)NULL ) || (ct == (struct ct_data *)NULL )) {struct rpc_createerr *ce = &get_rpc_createerr (); - (void )syslog (LOG_ERR , clnt_vc_errstr, - clnt_vc_str, __no_mem_str); ce->cf_stat =RPC_SYSTEMERROR ; ce->cf_error.re_errno = errno; + (void )syslog (LOG_ERR , clnt_vc_errstr, + clnt_vc_str, __no_mem_str); goto err; } ct->ct_addr.buf =NULL ;
Signature · 1 line
1.8.3.1
Re: [Libtirpc-devel] [PATCH 2/3] Record errno value before calling syslog
Quoted 3 lines · Paulo Andrade
Unlikely to change, but stay in the safe side.
Signed-off-by Paulo Andrade <[email protected]>
Committed..
steved.
Quoted 22 lines
--- src/clnt_vc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)src/clnt_vc.c +2
-2@@ -191,10 +191,10 @@ clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz) ct = (struct ct_data *)mem_alloc (sizeof (*ct));if ((cl == (CLIENT *)NULL ) || (ct == (struct ct_data *)NULL )) {struct rpc_createerr *ce = &get_rpc_createerr (); - (void )syslog (LOG_ERR , clnt_vc_errstr, - clnt_vc_str, __no_mem_str); ce->cf_stat =RPC_SYSTEMERROR ; ce->cf_error.re_errno = errno; + (void )syslog (LOG_ERR , clnt_vc_errstr, + clnt_vc_str, __no_mem_str); goto err; } ct->ct_addr.buf =NULL ;
PATCH v2 3/3 Do not hold a global mutex during connect
In reply to Paulo Andrade A multi-threaded application, connecting to multiple rpc servers,
may dead lock if the connect call stalls on a non responsive server.
--- src/clnt_vc.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
src/clnt_vc.c +3 -4
@@ -233,15 +233,16 @@ clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz)assert (vc_cv != (cond_t *)NULL );/* -* XXX - fvdl connecting while holding a mutex? +* Do not hold mutex during connect */ +mutex_unlock (&clnt_fd_lock); + slen =sizeof ss;if (getpeername (fd, (struct sockaddr *)&ss, &slen) <0 ) {if (errno !=ENOTCONN ) {struct rpc_createerr *ce = &get_rpc_createerr (); ce->cf_stat =RPC_SYSTEMERROR ; ce->cf_error.re_errno = errno; -mutex_unlock (&clnt_fd_lock);thr_sigsetmask (SIG_SETMASK , &(mask),NULL ); goto err; }
@@ -249,12 +250,10 @@ clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz)struct rpc_createerr *ce = &get_rpc_createerr (); ce->cf_stat =RPC_SYSTEMERROR ; ce->cf_error.re_errno = errno; -mutex_unlock (&clnt_fd_lock);thr_sigsetmask (SIG_SETMASK , &(mask),NULL ); goto err; } } -mutex_unlock (&clnt_fd_lock);if (!__rpc_fd2sockinfo (fd, &si)) goto err;thr_sigsetmask (SIG_SETMASK , &(mask),NULL );
Signature · 1 line
1.8.3.1
Re: [Libtirpc-devel] [PATCH 3/3] Do not hold a global mutex during connect
Quoted 2 lines · Paulo Andrade
A multi-threaded application, connecting to multiple rpc servers,
may dead lock if the connect call stalls on a non responsive server.
It's occurred to me that the mutex may be held over the connect(2) call to prevent concurrent calls to connect(2) using the same fd.
That's a race and is a lot harder to deal with.
Comments, thoughts anyone?
I guess it's time to have a look at the connect(2) source ....
Quoted 42 lines
Signed-off-by Paulo Andrade <[email protected]>--- src/clnt_vc.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)src/clnt_vc.c +3
-4@@ -233,15 +233,16 @@ clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz)assert (vc_cv != (cond_t *)NULL );/* -* XXX - fvdl connecting while holding a mutex? +* Do not hold mutex during connect */ +mutex_unlock (&clnt_fd_lock); + slen =sizeof ss;if (getpeername (fd, (struct sockaddr *)&ss, &slen) <0 ) {if (errno !=ENOTCONN ) {struct rpc_createerr *ce = &get_rpc_createerr (); ce->cf_stat =RPC_SYSTEMERROR ; ce->cf_error.re_errno = errno; -mutex_unlock (&clnt_fd_lock);thr_sigsetmask (SIG_SETMASK , &(mask),NULL ); goto err; }@@ -249,12 +250,10 @@ clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz)struct rpc_createerr *ce = &get_rpc_createerr (); ce->cf_stat =RPC_SYSTEMERROR ; ce->cf_error.re_errno = errno; -mutex_unlock (&clnt_fd_lock);thr_sigsetmask (SIG_SETMASK , &(mask),NULL ); goto err; } } -mutex_unlock (&clnt_fd_lock);if (!__rpc_fd2sockinfo (fd, &si)) goto err;thr_sigsetmask (SIG_SETMASK , &(mask),NULL );
Re: [Libtirpc-devel] [PATCH 3/3] Do not hold a global mutex during connect
In reply to Paulo AndradeQuoted 4 lines · Paulo Andrade
A multi-threaded application, connecting to multiple rpc servers,
may dead lock if the connect call stalls on a non responsive server.Signed-off-by Paulo Andrade <[email protected]>
committed...
steved.
Quoted 41 lines
--- src/clnt_vc.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)src/clnt_vc.c +3
-4@@ -233,15 +233,16 @@ clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz)assert (vc_cv != (cond_t *)NULL );/* -* XXX - fvdl connecting while holding a mutex? +* Do not hold mutex during connect */ +mutex_unlock (&clnt_fd_lock); + slen =sizeof ss;if (getpeername (fd, (struct sockaddr *)&ss, &slen) <0 ) {if (errno !=ENOTCONN ) {struct rpc_createerr *ce = &get_rpc_createerr (); ce->cf_stat =RPC_SYSTEMERROR ; ce->cf_error.re_errno = errno; -mutex_unlock (&clnt_fd_lock);thr_sigsetmask (SIG_SETMASK , &(mask),NULL ); goto err; }@@ -249,12 +250,10 @@ clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz)struct rpc_createerr *ce = &get_rpc_createerr (); ce->cf_stat =RPC_SYSTEMERROR ; ce->cf_error.re_errno = errno; -mutex_unlock (&clnt_fd_lock);thr_sigsetmask (SIG_SETMASK , &(mask),NULL ); goto err; } } -mutex_unlock (&clnt_fd_lock);if (!__rpc_fd2sockinfo (fd, &si)) goto err;thr_sigsetmask (SIG_SETMASK , &(mask),NULL );