SUNRPC connect timeout case network request delay
Subsystems (2)
Hi,
When I use the nfs at RHEL with kernel-2.6.31 and nfs-utils-1.2.0, I got a problem, but i don't sure.
The process:
Step1: [22:40:9] Open file /mnt/nfs/file(OK)
Step2: [22:40:9] Lock file
[22:40:9] Lock file success
Step3: [22:40:9] Reboot the nfs server
If reboot success, put <enter> to continue
Step4: [22:42:16] Write data to file
[22:42:16] Write data success
Step5: [22:42:16] Unlock file
[22:46:30] Unlock file success.
Step6: [22:46:30] Close file /mnt/nfs/file
[22:46:30] Close fiel /mnt/nfs/file successThe problem is at step5, unlock file takes 4 min, it's a long time than expected. When traceing the kernel, I find SUNRPC call call_connect timeout many times, one timeout is 1min.
I think it's a problem of kernel, but i don't know why, can someone help me ?
thanks,
Mi Jinlong
Quoted 19 lines · Mi Jinlong
Hi,
When I use the nfs at RHEL with kernel-2.6.31 and nfs-utils-1.2.0, I got a problem, but i don't sure.
The process:
Step1: [22:40:9] Open file /mnt/nfs/file(OK)
Step2: [22:40:9] Lock file[22:40:9] Lock file success Step3: [22:40:9] Reboot the nfs server If reboot success, put<enter> to continue Step4: [22:42:16] Write data to file [22:42:16] Write data success Step5: [22:42:16] Unlock file [22:46:30] Unlock file success. Step6: [22:46:30] Close file /mnt/nfs/file [22:46:30] Close fiel /mnt/nfs/file success
Quoted 3 lines
The problem is at step5, unlock file takes 4 min, it's a long time than expected. When traceing the kernel, I find SUNRPC call
call_connecttimeout many times, one timeout is 1min.
The kernel's TCP reconnect logic will retry until it succeeds, without letting the upper level make progress. For some reason, it is having difficulty reconnecting with your server.
Quoted 1 line
I think it's a problem of kernel, but i don't know why, can someone help me ?
# sudo rpcdebug -m rpc -s xprt trans
run your test, and then look in your kernel log. It might tell you what error is occurring. Another good thing to do is capture a network trace with
# sudo tcpdump -s 1540 -w /tmp/raw
while you run your test.
This is similar to the symptoms of other connect problems I've encountered recently.
Signature · 1 line
chuck[dot]lever[at]oracle[dot]com
Hi chuck,
Thanks for your reply.
Chuck Lever =E5=86=99=E9=81=93:
Quoted 9 lines · Mi Jinlong
Quoted 8 lines
Hi,
Step4: [22:42:16] Write data to file[22:42:16] Write data success Step5: [22:42:16] Unlock file [22:46:30] Unlock file success. Step6: [22:46:30] Close file /mnt/nfs/file [22:46:30] Close fiel /mnt/nfs/file success
Quoted 3 lines
Quoted 3 lines
The problem is at step5, unlock file takes 4 min, it's a long time than expected.
When traceing the kernel, I find SUNRPC callcall_connecttimeout ma=
ny
Quoted 4 lines
Quoted 2 lines
times,
one timeout is 1min.=20
The kernel's TCP reconnect logic will retry until it succeeds, withou=
t
Quoted 7 lines
letting the upper level make progress. For some reason, it is having difficulty reconnecting with your server.
=20Quoted 2 lines
I think it's a problem of kernel, but i don't know why, can someone help me ?
=20
# sudo rpcdebug -m rpc -s xprt trans
After running this command, I got some important messages that I think=
=2E
RPC: xs_connect delayed xprt for 3 seconds
...
RPC: xs_connect delayed xprt for 6 seconds
...
RPC: xs_connect delayed xprt for 12 seconds
...
RPC: xs_connect delayed xprt for 24 seconds
...
...
RPC: xs_connect delayed xprt for 300 seconds
=20
This message is printed at xs_connect, and the delay time is double the= re.
IMO, when some data translate over through a socket, the socket should = be released.
But, it seems the socket isn't released through those messages above.=20 Is it wrong, or there are some other reasons ?
At the latest kernel, this bug was fix by patch
"NFS/RPC: fix problems with reestablish_timeout and related code."
But I don't sure about this.
thanks,
Mi Jinlong
Quoted 17 lines · Mi Jinlong
Hi chuck,
Thanks for your reply.
Chuck Lever =E5=86=99=E9=81=93:
Quoted 12 lines · Mi Jinlong
Quoted 11 lines
Hi,
Step4: [22:42:16] Write data to file[22:42:16] Write data success Step5: [22:42:16] Unlock file [22:46:30] Unlock file success. Step6: [22:46:30] Close file /mnt/nfs/file [22:46:30] Close fiel /mnt/nfs/file successThe problem is at step5, unlock file takes 4 min, it's a long time than expected.
When traceing the kernel, I find SUNRPC callcall_connecttimeout m=
any
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
times,
one timeout is 1min.The kernel's TCP reconnect logic will retry until it succeeds, witho=
ut
Quoted 1 line
Quoted 1 line
letting the upper level make progress. For some reason, it is havin=
g
Quoted 8 lines
Quoted 6 lines
difficulty reconnecting with your server.
Quoted 2 lines
I think it's a problem of kernel, but i don't know why, can someone help me ?
# sudo rpcdebug -m rpc -s xprt transAfter running this command, I got some important messages that I th=
ink.
Quoted 12 lines
RPC:
xs_connectdelayed xprt for 3 seconds
...
RPC:xs_connectdelayed xprt for 6 seconds
...
RPC:xs_connectdelayed xprt for 12 seconds
...
RPC:xs_connectdelayed xprt for 24 seconds
...
...
RPC:xs_connectdelayed xprt for 300 seconds
Quoted 1 line
This message is printed at
xs_connect, and the delay time is double t=
here.
Quoted 1 line
IMO, when some data translate over through a socket, the socket shoul=
d be released.
Quoted 2 lines
But, it seems the socket isn't released through those messages above. Is it wrong, or there are some other reasons ?
The code is trying to connect, but the ->connect call isn't working=20 somehow. The code backs off by doubling the timeout each time, so that= =20
the connect attempts don't overload the server.
This tells us that the code is attempting to connect, but not why the=20 connect attempt is failing.
Quoted 3 lines
At the latest kernel, this bug was fix by patch
"NFS/RPC: fix problems withreestablish_timeoutand related code."
But I don't sure about this.
--=20
chuck[dot]lever[at]oracle[dot]com
Hi,
Thanks for your reply.
Chuck Lever =E5=86=99=E9=81=93:
Quoted 15 lines · Mi Jinlong
Quoted 14 lines
Hi chuck,
Thanks for your reply.
Chuck Lever =E5=86=99=E9=81=93:
Quoted 9 lines · Mi Jinlong
Quoted 8 lines
Hi,
Step4: [22:42:16] Write data to file[22:42:16] Write data success Step5: [22:42:16] Unlock file [22:46:30] Unlock file success. Step6: [22:46:30] Close file /mnt/nfs/file [22:46:30] Close fiel /mnt/nfs/file success
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The problem is at step5, unlock file takes 4 min, it's a long time than expected.
When traceing the kernel, I find SUNRPC callcall_connecttimeout =
many
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
times,
one timeout is 1min.The kernel's TCP reconnect logic will retry until it succeeds, with=
out
Quoted 1 line
Quoted 1 line
Quoted 1 line
letting the upper level make progress. For some reason, it is havi=
ng
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
difficulty reconnecting with your server.
Quoted 1 line
I think it's a problem of kernel, but i don't know why, can someon=
e
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
help me ?
# sudo rpcdebug -m rpc -s xprt transAfter running this command, I got some important messages that I t=
hink.
Quoted 12 lines
Quoted 12 lines
RPC:
xs_connectdelayed xprt for 3 seconds
...
RPC:xs_connectdelayed xprt for 6 seconds
...
RPC:xs_connectdelayed xprt for 12 seconds
...
RPC:xs_connectdelayed xprt for 24 seconds
...
...
RPC:xs_connectdelayed xprt for 300 seconds
Quoted 3 lines
Quoted 3 lines
This message is printed at
xs_connect, and the delay time is double there.
IMO, when some data translate over through a socket, the socket shou=
ld
Quoted 2 lines
Quoted 2 lines
be released.
But, it seems the socket isn't released through those messages above=
=2E
Quoted 4 lines
Quoted 1 line
Is it wrong, or there are some other reasons ?
=20
The code is trying to connect, but the ->connect call isn't working somehow. The code backs off by doubling the timeout each time, so th=
at
Quoted 4 lines
the connect attempts don't overload the server.
=20
This tells us that the code is attempting to connect, but not why the connect attempt is failing.
When reading the kernel codes, I find a problem at function xs_tcp_cl=
ose.
....
772 static void xs_tcp_close(struct rpc_xprt *xprt)
773 {
774 if (test_and_clear_bit(XPRT_CONNECTION_CLOSE, &xprt->st=
ate))
775 xs_close(xprt);
776 else
777 xs_tcp_shutdown(xprt);
778 }
... If a task call xs_tcp_close to close the xprt's sock, many times it o=
nly call=20
xs_tcp_shutdown to using the next layer's close function to close the=
socket
connection.
But after close the socket connection, the socket also exist, so the =
socket may
be reused. Is it a problem ? I think after xs_tcp_shutdown, the socke=
t should
be released.
=20
thanks,
Mi Jinlong
Quoted 72 lines · Mi Jinlong
Hi,
Thanks for your reply.
Quoted 50 lines · Chuck Lever
Quoted 42 lines · Mi Jinlong
Hi chuck,
Thanks for your reply.
Quoted 15 lines · Chuck Lever
Quoted 8 lines · Mi Jinlong
Hi,
Step4: [22:42:16] Write data to file[22:42:16] Write data success Step5: [22:42:16] Unlock file [22:46:30] Unlock file success. Step6: [22:46:30] Close file /mnt/nfs/file [22:46:30] Close fiel /mnt/nfs/file successQuoted 5 lines
The problem is at step5, unlock file takes 4 min, it's a long time than expected.
When traceing the kernel, I find SUNRPC callcall_connecttimeout many times,
one timeout is 1min.Quoted 8 lines
The kernel's TCP reconnect logic will retry until it succeeds, without letting the upper level make progress. For some reason, it is having difficulty reconnecting with your server.
Quoted 2 lines
I think it's a problem of kernel, but i don't know why, can someone help me ?
# sudo rpcdebug -m rpc -s xprt transAfter running this command, I got some important messages that I think.
RPC:
xs_connectdelayed xprt for 3 seconds
...
RPC:xs_connectdelayed xprt for 6 seconds
...
RPC:xs_connectdelayed xprt for 12 seconds
...
RPC:xs_connectdelayed xprt for 24 seconds
...
...
RPC:xs_connectdelayed xprt for 300 secondsQuoted 6 lines
This message is printed at
xs_connect, and the delay time is double there.
IMO, when some data translate over through a socket, the socket should be released.
But, it seems the socket isn't released through those messages above. Is it wrong, or there are some other reasons ?Quoted 6 lines
The code is trying to connect, but the ->connect call isn't working somehow. The code backs off by doubling the timeout each time, so that the connect attempts don't overload the server.
This tells us that the code is attempting to connect, but not why the connect attempt is failing.
When reading the kernel codes, I find a problem at function
xs_tcp_close..... 772 static void xs_tcp_close(struct rpc_xprt *xprt) 773 { 774 if (test_and_clear_bit(XPRT_CONNECTION_CLOSE, &xprt->state)) 775 xs_close(xprt); 776 else 777 xs_tcp_shutdown(xprt); 778 } ...
Quoted 6 lines
If a task call
xs_tcp_closeto close the xprt's sock, many times it only callxs_tcp_shutdownto using the next layer's close function to close the socket connection.
But after close the socket connection, the socket also exist, so the socket may be reused. Is it a problem ? I think afterxs_tcp_shutdown, the socket should be released.
No it shouldn't. The whole point of the current code is to allow the RPC client to _reuse_ the same port without having to wait for a TIME_WAIT.
The reason why we want to do that is because a lot of servers key their duplicate reply caches on the port number. See
http://www.connectathon.org/talks96/werme1.html
Trond