Weird TCP hang when doing loopback NFS (wireshark traces attached)

DISCUSSION Trond Myklebust 4 participants 6 messages Mar 1, 2015 – Mar 2, 2015 2 messages not in the archive

Subsystems (4)

Networking Network drivers NFS client NFS server

lore

Share & tools

lore

Trond Myklebust

author

2015-03-01 19:14 UTC #0 lore raw

Hi,

When doing testing of NFSv3 loopback mounts (client and server are on the same IP address), I'm seeing a very reproducible hang in which the client stops receiving data from the server. The TCP connection is still marked as established, and the server appears to continue to receive and send data, however the client does not.

So far, I've reproduced on both v4.0-rc1, and the Fedora v3.18.7 kernel.

The reproducer is simply to loopback mount using NFSv3, and then run the 'fsx' filesystem exerciser. I'm usually able to trigger the hang with "fsx -N 100000 foobar".

I've attached a couple of wireshark trace of a few frames just before and during the hang in case it jogs any memories.

Cheers
Trond

A message that is not in the archive (sent off-list or not yet received): 1425237291.24845.13.camel-7I+n7zu2hftEKMMhf/[email protected]

Trond Myklebust

author

2015-03-02 00:52 UTC #2 lore raw

Hi Bruce,

Quoted 8 lines · Trond Myklebust

Hi,
When doing testing of NFSv3 loopback mounts (client and server are on the same IP address), I'm seeing a very reproducible hang in which the client stops receiving data from the server. The TCP connection is still marked as established, and the server appears to continue to receive and send data, however the client does not.

Quoted 8 lines

So far, I've reproduced on both v4.0-rc1, and the Fedora v3.18.7 kernel.
The reproducer is simply to loopback mount using NFSv3, and then run the 'fsx' filesystem exerciser. I'm usually able to trigger the hang with "fsx -N 100000 foobar".
I've attached a couple of wireshark trace of a few frames just before and during the hang in case it jogs any memories.

This bug appears to go away when I disable the splice()-based reads by clearing the RQ_SPLICE_OK flag.

I noticed that it always involved a combination of a READ and a truncating SETATTR call. Are you sure that it is safe to share pagecache pages directly with sendpage() in this way? As far as I can tell, there is no locking to prevent them from being modified while in the TCP send queue.

Cheers
Trond

Signature · 7 lines

Trond Myklebust
Linux NFS client maintainer, PrimaryData
trond.myklebust-7I+n7zu2hftEKMMhf/[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Bruce James Fields

2015-03-02 01:06 UTC #3 lore raw

Quoted 21 lines · Trond Myklebust

Hi Bruce,
Quoted 8 lines · Trond Myklebust
Hi,
When doing testing of NFSv3 loopback mounts (client and server are on the same IP address), I'm seeing a very reproducible hang in which the client stops receiving data from the server. The TCP connection is still marked as established, and the server appears to continue to receive and send data, however the client does not.
Quoted 8 lines
So far, I've reproduced on both v4.0-rc1, and the Fedora v3.18.7 kernel.
The reproducer is simply to loopback mount using NFSv3, and then run the 'fsx' filesystem exerciser. I'm usually able to trigger the hang with "fsx -N 100000 foobar".
I've attached a couple of wireshark trace of a few frames just before and during the hang in case it jogs any memories.

Quoted 8 lines

This bug appears to go away when I disable the splice()-based reads by clearing the RQ_SPLICE_OK flag.
I noticed that it always involved a combination of a READ and a truncating SETATTR call. Are you sure that it is safe to share pagecache pages directly with sendpage() in this way? As far as I can tell, there is no locking to prevent them from being modified while in the TCP send queue.

This is the stable-pages problem that we've had forever, isn't it? Or is this a different problem?

--b.

Trond Myklebust

2015-03-02 01:20 UTC #4 lore raw

Quoted 25 lines · Bruce James Fields

Quoted 24 lines · Trond Myklebust
Hi Bruce,
Quoted 8 lines · Trond Myklebust
Hi,
When doing testing of NFSv3 loopback mounts (client and server are on the same IP address), I'm seeing a very reproducible hang in which the client stops receiving data from the server. The TCP connection is still marked as established, and the server appears to continue to receive and send data, however the client does not.
Quoted 8 lines
So far, I've reproduced on both v4.0-rc1, and the Fedora v3.18.7 kernel.
The reproducer is simply to loopback mount using NFSv3, and then run the 'fsx' filesystem exerciser. I'm usually able to trigger the hang with "fsx -N 100000 foobar".
I've attached a couple of wireshark trace of a few frames just before and during the hang in case it jogs any memories.
This bug appears to go away when I disable the splice()-based reads by clearing the RQ_SPLICE_OK flag.

Quoted 8 lines

Quoted 5 lines
I noticed that it always involved a combination of a READ and a truncating SETATTR call. Are you sure that it is safe to share pagecache pages directly with sendpage() in this way? As far as I can tell, there is no locking to prevent them from being modified while in the TCP send queue.
This is the stable-pages problem that we've had forever, isn't it? Or is this a different problem?

It is causing the TCP socket to hang, so it goes beyond the usual stable pages issue.

Signature · 3 lines

Trond Myklebust
Linux NFS client maintainer, PrimaryData
[email protected]

Trond Myklebust

2015-03-02 04:31 UTC #5 lore raw

Quoted 32 lines · Trond Myklebust

Quoted 31 lines · Bruce James Fields
Quoted 29 lines · Trond Myklebust
Hi Bruce,
Quoted 8 lines · Trond Myklebust
Hi,
When doing testing of NFSv3 loopback mounts (client and server are on the same IP address), I'm seeing a very reproducible hang in which the client stops receiving data from the server. The TCP connection is still marked as established, and the server appears to continue to receive and send data, however the client does not.
Quoted 8 lines
So far, I've reproduced on both v4.0-rc1, and the Fedora v3.18.7 kernel.
The reproducer is simply to loopback mount using NFSv3, and then run the 'fsx' filesystem exerciser. I'm usually able to trigger the hang with "fsx -N 100000 foobar".
I've attached a couple of wireshark trace of a few frames just before and during the hang in case it jogs any memories.
This bug appears to go away when I disable the splice()-based reads by clearing the RQ_SPLICE_OK flag.
I noticed that it always involved a combination of a READ and a truncating SETATTR call. Are you sure that it is safe to share pagecache pages directly with sendpage() in this way? As far as I can tell, there is no locking to prevent them from being modified while in the TCP send queue.

Quoted 6 lines

Quoted 2 lines
This is the stable-pages problem that we've had forever, isn't it? Or is this a different problem?
It is causing the TCP socket to hang, so it goes beyond the usual stable pages issue.

Confirming that clearing RQ_SPLICE_OK fixes the issue on all kernel that I've tested so far.

Signature · 3 lines

Trond Myklebust
Linux NFS client maintainer, PrimaryData
[email protected]

A message that is not in the archive (sent off-list or not yet received): CAHQdGtQnbPWYhdvwTGJKUD4mt8x_rmQjCH3AO4X17Y4RBSpUQQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

Bruce James Fields

2015-03-02 19:58 UTC #7 lore raw

Quoted 37 lines · Trond Myklebust

Quoted 35 lines · Trond Myklebust
Quoted 33 lines · Bruce James Fields
Quoted 21 lines · Trond Myklebust
Hi Bruce,
Quoted 8 lines · Trond Myklebust
Hi,
When doing testing of NFSv3 loopback mounts (client and server are on the same IP address), I'm seeing a very reproducible hang in which the client stops receiving data from the server. The TCP connection is still marked as established, and the server appears to continue to receive and send data, however the client does not.
Quoted 8 lines
So far, I've reproduced on both v4.0-rc1, and the Fedora v3.18.7 kernel.
The reproducer is simply to loopback mount using NFSv3, and then run the 'fsx' filesystem exerciser. I'm usually able to trigger the hang with "fsx -N 100000 foobar".
I've attached a couple of wireshark trace of a few frames just before and during the hang in case it jogs any memories.
Quoted 8 lines
This bug appears to go away when I disable the splice()-based reads by clearing the RQ_SPLICE_OK flag.
I noticed that it always involved a combination of a READ and a truncating SETATTR call. Are you sure that it is safe to share pagecache pages directly with sendpage() in this way? As far as I can tell, there is no locking to prevent them from being modified while in the TCP send queue.
This is the stable-pages problem that we've had forever, isn't it? Or is this a different problem?

Quoted 6 lines

Quoted 3 lines
It is causing the TCP socket to hang, so it goes beyond the usual stable pages issue.
Confirming that clearing RQ_SPLICE_OK fixes the issue on all kernel that I've tested so far.

Well, if the problem is a race with truncate then I guess it may have something to do with sending pages that are no longer part of the page cache?

I'd think that the get_page() in nfsd_splice_actor would prevent the page being put to any other use until the network layer was done with it, so that at worst the client would see garbage. But I don't begin to understand how truncation actually works....

The zero-copy v3 code has been there since 2002, if I'm reading the history right, so if it's really a fundamental problem with the approach then I wonder how it's survived so long.

I haven't tried to reproduce yet.

--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html