Skip to contentSkip to conversation

SUNRPC connect timeout case network request delay

RFC Mi Jinlong 3 participants 6 messages Mar 4, 2010 – Mar 12, 2010

Subsystems (2)
lore
Share & tools
lore

Mi Jinlong

author
#0 lore raw

Hi,

When I use the nfs at RHEL with kernel-2.6.31 and nfs-utils-1.2.0, I got a problem, but i don't sure.

The process:

Step1: [22:40:9] Open file /mnt/nfs/file(OK)
Step2: [22:40:9] Lock file

       [22:40:9] Lock file success
Step3: [22:40:9] Reboot the nfs server
       If reboot success, put <enter> to continue
Step4: [22:42:16] Write data to file
       [22:42:16] Write data success
Step5: [22:42:16] Unlock file
       [22:46:30] Unlock file success.
Step6: [22:46:30] Close file /mnt/nfs/file
       [22:46:30] Close fiel /mnt/nfs/file success

The problem is at step5, unlock file takes 4 min, it's a long time than expected. When traceing the kernel, I find SUNRPC call call_connect timeout many times, one timeout is 1min.

I think it's a problem of kernel, but i don't know why, can someone help me ?

thanks,
Mi Jinlong

#1 lore raw
Quoted 19 lines · Mi Jinlong

Hi,

When I use the nfs at RHEL with kernel-2.6.31 and nfs-utils-1.2.0, I got a problem, but i don't sure.

The process:

Step1: [22:40:9] Open file /mnt/nfs/file(OK)
Step2: [22:40:9] Lock file

        [22:40:9] Lock file success
Step3: [22:40:9] Reboot the nfs server
        If reboot success, put<enter>  to continue
Step4: [22:42:16] Write data to file
        [22:42:16] Write data success
Step5: [22:42:16] Unlock file
        [22:46:30] Unlock file success.
Step6: [22:46:30] Close file /mnt/nfs/file
        [22:46:30] Close fiel /mnt/nfs/file success
Quoted 3 lines

The problem is at step5, unlock file takes 4 min, it's a long time than expected. When traceing the kernel, I find SUNRPC call call_connect timeout many times, one timeout is 1min.

The kernel's TCP reconnect logic will retry until it succeeds, without letting the upper level make progress. For some reason, it is having difficulty reconnecting with your server.

Quoted 1 line

I think it's a problem of kernel, but i don't know why, can someone help me ?

# sudo rpcdebug -m rpc -s xprt trans

run your test, and then look in your kernel log. It might tell you what error is occurring. Another good thing to do is capture a network trace with

# sudo tcpdump -s 1540 -w /tmp/raw

while you run your test.

This is similar to the symptoms of other connect problems I've encountered recently.

Signature · 1 line
chuck[dot]lever[at]oracle[dot]com
author
#2 lore raw

Hi chuck,

Thanks for your reply.

Chuck Lever =E5=86=99=E9=81=93:

Quoted 9 lines · Mi Jinlong
Quoted 8 lines

Hi,
Step4: [22:42:16] Write data to file

        [22:42:16] Write data success
Step5: [22:42:16] Unlock file
        [22:46:30] Unlock file success.
Step6: [22:46:30] Close file /mnt/nfs/file
        [22:46:30] Close fiel /mnt/nfs/file success
Quoted 3 lines
Quoted 3 lines

The problem is at step5, unlock file takes 4 min, it's a long time than expected.
When traceing the kernel, I find SUNRPC call call_connect timeout ma=

ny

Quoted 4 lines
Quoted 2 lines

times,
one timeout is 1min.

=20
The kernel's TCP reconnect logic will retry until it succeeds, withou=

t

Quoted 7 lines

letting the upper level make progress. For some reason, it is having difficulty reconnecting with your server.
=20

Quoted 2 lines

I think it's a problem of kernel, but i don't know why, can someone help me ?

=20

# sudo rpcdebug -m rpc -s xprt trans

After running this command, I got some important messages that I think=
=2E

RPC: xs_connect delayed xprt for 3 seconds
...
RPC: xs_connect delayed xprt for 6 seconds
...
RPC: xs_connect delayed xprt for 12 seconds
...
RPC: xs_connect delayed xprt for 24 seconds
...
...
RPC: xs_connect delayed xprt for 300 seconds
=20
This message is printed at xs_connect, and the delay time is double the= re.
IMO, when some data translate over through a socket, the socket should = be released.
But, it seems the socket isn't released through those messages above.=20 Is it wrong, or there are some other reasons ?

At the latest kernel, this bug was fix by patch
"NFS/RPC: fix problems with reestablish_timeout and related code."
But I don't sure about this.

thanks,
Mi Jinlong

#3 lore raw
Quoted 17 lines · Mi Jinlong

Hi chuck,

Thanks for your reply.

Chuck Lever =E5=86=99=E9=81=93:

Quoted 12 lines · Mi Jinlong
Quoted 11 lines

Hi,
Step4: [22:42:16] Write data to file

         [22:42:16] Write data success
Step5: [22:42:16] Unlock file
         [22:46:30] Unlock file success.
Step6: [22:46:30] Close file /mnt/nfs/file
         [22:46:30] Close fiel /mnt/nfs/file success

The problem is at step5, unlock file takes 4 min, it's a long time than expected.
When traceing the kernel, I find SUNRPC call call_connect timeout m=

any

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

times,
one timeout is 1min.

The kernel's TCP reconnect logic will retry until it succeeds, witho=

ut

Quoted 1 line
Quoted 1 line

letting the upper level make progress. For some reason, it is havin=

g

Quoted 8 lines
Quoted 6 lines

difficulty reconnecting with your server.

Quoted 2 lines

I think it's a problem of kernel, but i don't know why, can someone help me ?

# sudo rpcdebug -m rpc -s xprt trans

After running this command, I got some important messages that I th=

ink.

Quoted 12 lines

RPC: xs_connect delayed xprt for 3 seconds
...
RPC: xs_connect delayed xprt for 6 seconds
...
RPC: xs_connect delayed xprt for 12 seconds
...
RPC: xs_connect delayed xprt for 24 seconds
...
...
RPC: xs_connect delayed xprt for 300 seconds

Quoted 1 line

This message is printed at xs_connect, and the delay time is double t=

here.

Quoted 1 line

IMO, when some data translate over through a socket, the socket shoul=

d be released.

Quoted 2 lines

But, it seems the socket isn't released through those messages above. Is it wrong, or there are some other reasons ?

The code is trying to connect, but the ->connect call isn't working=20 somehow. The code backs off by doubling the timeout each time, so that= =20
the connect attempts don't overload the server.

This tells us that the code is attempting to connect, but not why the=20 connect attempt is failing.

Quoted 3 lines

At the latest kernel, this bug was fix by patch
"NFS/RPC: fix problems with reestablish_timeout and related code."
But I don't sure about this.

--=20
chuck[dot]lever[at]oracle[dot]com

author
#4 lore raw

Hi,

Thanks for your reply.

Chuck Lever =E5=86=99=E9=81=93:

Quoted 15 lines · Mi Jinlong
Quoted 14 lines

Hi chuck,

Thanks for your reply.

Chuck Lever =E5=86=99=E9=81=93:

Quoted 9 lines · Mi Jinlong
Quoted 8 lines

Hi,
Step4: [22:42:16] Write data to file

         [22:42:16] Write data success
Step5: [22:42:16] Unlock file
         [22:46:30] Unlock file success.
Step6: [22:46:30] Close file /mnt/nfs/file
         [22:46:30] Close fiel /mnt/nfs/file success
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

The problem is at step5, unlock file takes 4 min, it's a long time than expected.
When traceing the kernel, I find SUNRPC call call_connect timeout =

many

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

times,
one timeout is 1min.

The kernel's TCP reconnect logic will retry until it succeeds, with=

out

Quoted 1 line
Quoted 1 line
Quoted 1 line

letting the upper level make progress. For some reason, it is havi=

ng

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

difficulty reconnecting with your server.

Quoted 1 line

I think it's a problem of kernel, but i don't know why, can someon=

e

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

help me ?

# sudo rpcdebug -m rpc -s xprt trans

After running this command, I got some important messages that I t=

hink.

Quoted 12 lines
Quoted 12 lines

RPC: xs_connect delayed xprt for 3 seconds
...
RPC: xs_connect delayed xprt for 6 seconds
...
RPC: xs_connect delayed xprt for 12 seconds
...
RPC: xs_connect delayed xprt for 24 seconds
...
...
RPC: xs_connect delayed xprt for 300 seconds

Quoted 3 lines
Quoted 3 lines

This message is printed at xs_connect, and the delay time is double there.
IMO, when some data translate over through a socket, the socket shou=

ld

Quoted 2 lines
Quoted 2 lines

be released.
But, it seems the socket isn't released through those messages above=

=2E

Quoted 4 lines
Quoted 1 line

Is it wrong, or there are some other reasons ?

=20
The code is trying to connect, but the ->connect call isn't working somehow. The code backs off by doubling the timeout each time, so th=

at

Quoted 4 lines

the connect attempts don't overload the server.
=20
This tells us that the code is attempting to connect, but not why the connect attempt is failing.

When reading the kernel codes, I find a problem at function xs_tcp_cl=
ose.

    ....
    772 static void xs_tcp_close(struct rpc_xprt *xprt)
    773 {
    774         if (test_and_clear_bit(XPRT_CONNECTION_CLOSE, &xprt->st=
ate))
    775                 xs_close(xprt);
    776         else
    777                 xs_tcp_shutdown(xprt);
    778 }
     ...

If a task call xs_tcp_close to close the xprt's sock, many times it o=
nly call=20
xs_tcp_shutdown to using the next layer's close function to close the=
socket
connection.
But after close the socket connection, the socket also exist, so the =
socket may
be reused. Is it a problem ? I think after xs_tcp_shutdown, the socke=
t should
be released.
=20
thanks,
Mi Jinlong

#5 lore raw
Quoted 72 lines · Mi Jinlong

Hi,

Thanks for your reply.

Quoted 50 lines · Chuck Lever
Quoted 42 lines · Mi Jinlong

Hi chuck,

Thanks for your reply.

Quoted 15 lines · Chuck Lever
Quoted 8 lines · Mi Jinlong

Hi,
Step4: [22:42:16] Write data to file

         [22:42:16] Write data success
Step5: [22:42:16] Unlock file
         [22:46:30] Unlock file success.
Step6: [22:46:30] Close file /mnt/nfs/file
         [22:46:30] Close fiel /mnt/nfs/file success
Quoted 5 lines

The problem is at step5, unlock file takes 4 min, it's a long time than expected.
When traceing the kernel, I find SUNRPC call call_connect timeout many times,
one timeout is 1min.

Quoted 8 lines

The kernel's TCP reconnect logic will retry until it succeeds, without letting the upper level make progress. For some reason, it is having difficulty reconnecting with your server.

Quoted 2 lines

I think it's a problem of kernel, but i don't know why, can someone help me ?

# sudo rpcdebug -m rpc -s xprt trans

After running this command, I got some important messages that I think.

RPC: xs_connect delayed xprt for 3 seconds
...
RPC: xs_connect delayed xprt for 6 seconds
...
RPC: xs_connect delayed xprt for 12 seconds
...
RPC: xs_connect delayed xprt for 24 seconds
...
...
RPC: xs_connect delayed xprt for 300 seconds

Quoted 6 lines

This message is printed at xs_connect, and the delay time is double there.
IMO, when some data translate over through a socket, the socket should be released.
But, it seems the socket isn't released through those messages above. Is it wrong, or there are some other reasons ?

Quoted 6 lines

The code is trying to connect, but the ->connect call isn't working somehow. The code backs off by doubling the timeout each time, so that the connect attempts don't overload the server.

This tells us that the code is attempting to connect, but not why the connect attempt is failing.

When reading the kernel codes, I find a problem at function xs_tcp_close.

    ....
    772 static void xs_tcp_close(struct rpc_xprt *xprt)
    773 {
    774         if (test_and_clear_bit(XPRT_CONNECTION_CLOSE, &xprt->state))
    775                 xs_close(xprt);
    776         else
    777                 xs_tcp_shutdown(xprt);
    778 }
     ...
Quoted 6 lines

If a task call xs_tcp_close to close the xprt's sock, many times it only call xs_tcp_shutdown to using the next layer's close function to close the socket connection.
But after close the socket connection, the socket also exist, so the socket may be reused. Is it a problem ? I think after xs_tcp_shutdown, the socket should be released.

No it shouldn't. The whole point of the current code is to allow the RPC client to _reuse_ the same port without having to wait for a TIME_WAIT.

The reason why we want to do that is because a lot of servers key their duplicate reply caches on the port number. See
http://www.connectathon.org/talks96/werme1.html

Trond