Do not hold clnt_fd_lock mutex during connect
Subsystems (2)
PATCH 1/1 Do not hold clnt_fd_lock mutex during connect
An user reports that their application connects to multiple servers
through a rpc interface using libtirpc. When one of the servers misbehaves (goes down ungracefully or has a delay of a few seconds in the traffic flow), it was observed that the traffic from the client to other servers is decreased by the traffic anomaly of the failing server, i.e. traffic decreases or goes to 0 in all the servers.
When investigated further, specifically into the behavior of the libtirpc
at the time of the issue, it was observed that all of the application threads specifically interacting with libtirpc were locked into one single lock inside the libtirpc library. This was a race condition which had resulted in a deadlock and hence the resultant dip/stoppage of traffic.
As an experiment, the user removed the libtirpc from the application build
and used the standard glibc library for rpc communication. In that case, everything worked perfectly even in the time of the issue of server nodes misbehaving.
--- src/clnt_vc.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)
src/clnt_vc.c +2 -6
@@ -229,27 +229,23 @@ clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz) }else assert (vc_cv != (cond_t *)NULL ); -/* -* XXX - fvdl connecting while holding a mutex? -*/ +mutex_unlock (&clnt_fd_lock); + slen =sizeof ss;if (getpeername (fd, (struct sockaddr *)&ss, &slen) <0 ) {if (errno !=ENOTCONN ) { rpc_createerr.cf_stat =RPC_SYSTEMERROR ; rpc_createerr.cf_error.re_errno = errno; -mutex_unlock (&clnt_fd_lock);thr_sigsetmask (SIG_SETMASK , &(mask),NULL ); goto err; }if (connect (fd, (struct sockaddr *)raddr->buf, raddr->len) <0 ){ rpc_createerr.cf_stat =RPC_SYSTEMERROR ; rpc_createerr.cf_error.re_errno = errno; -mutex_unlock (&clnt_fd_lock);thr_sigsetmask (SIG_SETMASK , &(mask),NULL ); goto err; } } -mutex_unlock (&clnt_fd_lock);if (!__rpc_fd2sockinfo (fd, &si)) goto err;thr_sigsetmask (SIG_SETMASK , &(mask),NULL );
Signature · 1 line
1.8.3.1
Quoted 13 lines · Paulo Andrade
An user reports that their application connects to multiple servers
through a rpc interface using libtirpc. When one of the servers misbehaves (goes down ungracefully or has a delay of a few seconds in the traffic flow), it was observed that the traffic from the client to other servers is decreased by the traffic anomaly of the failing server, i.e. traffic decreases or goes to 0 in all the servers.When investigated further, specifically into the behavior of the libtirpc
at the time of the issue, it was observed that all of the application threads specifically interacting with libtirpc were locked into one single lock inside the libtirpc library. This was a race condition which had resulted in a deadlock and hence the resultant dip/stoppage of traffic.
Quoted 4 lines
As an experiment, the user removed the libtirpc from the application build
and used the standard glibc library for rpc communication. In that case, everything worked perfectly even in the time of the issue of server nodes misbehaving.
I recommend simplifying this.
It should be a concise description of what is wrong and how this patch resolves it.
The description of the investigation will probably make reading the history more difficult when trying to find changes at later times so less is more I think.
Quoted 31 lines
Signed-off-by Paulo Andrade <[email protected]>--- src/clnt_vc.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)src/clnt_vc.c +2
-3@@ -229,27 +229,23 @@ clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz) }else assert (vc_cv != (cond_t *)NULL ); -/* -* XXX - fvdl connecting while holding a mutex? -*/ +mutex_unlock (&clnt_fd_lock); + slen =sizeof ss;if (getpeername (fd, (struct sockaddr *)&ss, &slen) <0 ) {if (errno !=ENOTCONN ) { rpc_createerr.cf_stat =RPC_SYSTEMERROR ; rpc_createerr.cf_error.re_errno = errno;-mutex_unlock (&clnt_fd_lock);thr_sigsetmask (SIG_SETMASK , &(mask),NULL ); goto err; }if (connect (fd, (struct sockaddr *)raddr->buf, raddr->len) <0 ){ rpc_createerr.cf_stat =RPC_SYSTEMERROR ;
src/clnt_vc.c clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz) · line 251
Quoted 10 lines
src/clnt_vc.c +0
-2rpc_createerr.cf_error.re_errno = errno; -mutex_unlock (&clnt_fd_lock);thr_sigsetmask (SIG_SETMASK , &(mask),NULL ); goto err; } } -mutex_unlock (&clnt_fd_lock);if (!__rpc_fd2sockinfo (fd, &si)) goto err;thr_sigsetmask (SIG_SETMASK , &(mask),NULL );
We will need to review the code in the other clnt_*_create() functions for this to be a complete resolution for the problem.
Ian
Quoted 35 lines · Paulo Andrade
An user reports that their application connects to multiple servers
through a rpc interface using libtirpc. When one of the servers misbehaves (goes down ungracefully or has a delay of a few seconds in the traffic flow), it was observed that the traffic from the client to other servers is decreased by the traffic anomaly of the failing server, i.e. traffic decreases or goes to 0 in all the servers.When investigated further, specifically into the behavior of the libtirpc
at the time of the issue, it was observed that all of the application threads specifically interacting with libtirpc were locked into one single lock inside the libtirpc library. This was a race condition which had resulted in a deadlock and hence the resultant dip/stoppage of traffic.As an experiment, the user removed the libtirpc from the application build
and used the standard glibc library for rpc communication. In that case, everything worked perfectly even in the time of the issue of server nodes misbehaving.Signed-off-by Paulo Andrade <[email protected]>--- src/clnt_vc.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)src/clnt_vc.c +1
-3@@ -229,27 +229,23 @@ clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz) }else assert (vc_cv != (cond_t *)NULL ); -/* -* XXX - fvdl connecting while holding a mutex? -*/ +mutex_unlock (&clnt_fd_lock);
src/clnt_vc.c clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz) · line 241
Quoted 10 lines
src/clnt_vc.c +1
-1+ slen =sizeof ss;if (getpeername (fd, (struct sockaddr *)&ss, &slen) <0 ) {if (errno !=ENOTCONN ) { rpc_createerr.cf_stat =RPC_SYSTEMERROR ; rpc_createerr.cf_error.re_errno = errno; -mutex_unlock (&clnt_fd_lock);thr_sigsetmask (SIG_SETMASK , &(mask),NULL ); goto err; }
Oh, right, the mutex is probably needed to ensure that errno is reliable.
src/clnt_vc.c clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz) · line 243
Quoted 2 lines
src/clnt_vc.c +0
-0if (connect (fd, (struct sockaddr *)raddr->buf, raddr->len) <0 ){
But this is probably where the caller is blocking so a small variation of this patch should achieve the required result.
btw, I had a quick look at some of the other code and so far it looks like they lead to clnt_tp_create() or clnt_dg_create() calls.
clnt_dg_create() is not connection oriented so it doesn't have the same mutex lock problem.
So this patch might be all that's needed.
Ian
Quoted 32 lines · Ian Kent
Quoted 31 lines · Paulo Andrade
An user reports that their application connects to multiple servers
through a rpc interface using libtirpc. When one of the servers misbehaves (goes down ungracefully or has a delay of a few seconds in the traffic flow), it was observed that the traffic from the client to other servers is decreased by the traffic anomaly of the failing server, i.e. traffic decreases or goes to 0 in all the servers.When investigated further, specifically into the behavior of the libtirpc
at the time of the issue, it was observed that all of the application threads specifically interacting with libtirpc were locked into one single lock inside the libtirpc library. This was a race condition which had resulted in a deadlock and hence the resultant dip/stoppage of traffic.As an experiment, the user removed the libtirpc from the application build
and used the standard glibc library for rpc communication. In that case, everything worked perfectly even in the time of the issue of server nodes misbehaving.Signed-off-by Paulo Andrade <[email protected]>--- src/clnt_vc.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)src/clnt_vc.c +0
-0@@ -229,27 +229,23 @@ clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz) }else assert (vc_cv != (cond_t *)NULL );
Quoted 16 lines
Quoted 4 lines
src/clnt_vc.c +1
-3-/* -* XXX - fvdl connecting while holding a mutex? -*/ +mutex_unlock (&clnt_fd_lock);src/clnt_vc.c clnt_vc_create(fd, raddr, prog, vers, sendsz, recvsz) · line 241
Quoted 10 lines
src/clnt_vc.c +1
-1+ slen =sizeof ss;if (getpeername (fd, (struct sockaddr *)&ss, &slen) <0 ) {if (errno !=ENOTCONN ) { rpc_createerr.cf_stat =RPC_SYSTEMERROR ; rpc_createerr.cf_error.re_errno = errno; -mutex_unlock (&clnt_fd_lock);thr_sigsetmask (SIG_SETMASK , &(mask),NULL ); goto err; }Oh, right, the mutex is probably needed to ensure that errno is reliable.
I realized later how dumb this comment was so I checked.
pThreads does provide a per-thread errno so there's no reason I can see to hold this lock over the getpeername() and connect() calls.
Ian