Skip to contentSkip to conversation

Hard to debug NFS loss of connectivity

DISCUSSION Emmanuel Florac 3 participants 12 messages Sep 5, 2013 – Sep 11, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi list, I have a serious problem I've never met before. Here is the setup:

The NFS server is running Debian 6 amd64 but with a plain vanilla 3.2.50 kernel. it shares a large 81 TB volume (XFS over LVM on hardware RAID6) through nfs without any particular options. Here is a glimpse of /etc/exports:

/mnt/raid 10.1.1.0/255.255.255.128(fsid=1,rw,no_root_squash,async,no_subtree_check)

On the other side is a VMWare ESX VM running Ubuntu 12.04LTS, kernel 3.2.0-52 Ubuntu amd64 mounting the share. From the fstab:

10.1.1.99:/mnt/raid /server nfs rw,hard,intr 0 0

The problem is as follow: stat'ing files on the VM makes the NFS connection drop. For instance:

find /server -type f -ls

It works for a while, then stops responding. The NFS mount is frozen. The network link is OK; I still can ssh from the server to the VM and back, I can wget from the VM to the server, ping the server from the VM, etc. Only NFS is affected.

Restarting NFS on the server does nothing to unfreeze the mount. Using nfs4 instead of nfs3 does nothing. The only remedy is to reboot the VM. There isn't any error in dmesg, /var/log/syslog or /var/log/messages in the VM nor the server.

I've tried rebooting the server on a 3.9.7 kernel. Same thing. Of course there isn't any data corruption of any sort. Running "find /mnt/raid -type f -ls" on the server works perfectly and lists about 25000 files without the slightest trouble.

It works equally well if I mount the NFS share on the server itself.

Now it's becoming crazier: When I run the find command as previously said, it freezes always on the same file, for instance :

/server/folder1/folder2/folder3/folder4/.svn/somefile

However, if after a fresh reboot I do

stat /server/folder1/folder2/folder3/folder4/.svn/somefile

no problem. Even doing this:

cd /server/folder1/folder2/folder3/folder4/ && find . -type f -ls

works. However this

cd /server/folder1/folder2/folder3/ && find . -type f -ls

doesn't fly. It freezes at exactly the same point. In the first test (running directly from /server) it freezes after successfully listing 10000 files. In the last test it freezes after only 25 files.
So apparently it's not about the number of files.

Now I'm stuck. Out of going through tcpdump, I have absolutely not the faintest idea about what's going on, except I tend to think that's some Ubuntu kernel bug.

Any hint, idea, etc would be extremely welcome. Even some debugging method less painful than digging through huge tcpdumps would be nice :)

Signature · 6 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------
#1 lore raw
Quoted 64 lines · Emmanuel Florac

Hi list, I have a serious problem I've never met before. Here is the setup:

The NFS server is running Debian 6 amd64 but with a plain vanilla 3.2.50 kernel. it shares a large 81 TB volume (XFS over LVM on hardware RAID6) through nfs without any particular options. Here is a glimpse of /etc/exports:

/mnt/raid 10.1.1.0/255.255.255.128(fsid=1,rw,no_root_squash,async,no_subtree_check)

On the other side is a VMWare ESX VM running Ubuntu 12.04LTS, kernel 3.2.0-52 Ubuntu amd64 mounting the share. From the fstab:

10.1.1.99:/mnt/raid /server nfs rw,hard,intr 0 0

The problem is as follow: stat'ing files on the VM makes the NFS connection drop. For instance:

find /server -type f -ls

It works for a while, then stops responding. The NFS mount is frozen. The network link is OK; I still can ssh from the server to the VM and back, I can wget from the VM to the server, ping the server from the VM, etc. Only NFS is affected.

Restarting NFS on the server does nothing to unfreeze the mount. Using nfs4 instead of nfs3 does nothing. The only remedy is to reboot the VM. There isn't any error in dmesg, /var/log/syslog or /var/log/messages in the VM nor the server.

I've tried rebooting the server on a 3.9.7 kernel. Same thing. Of course there isn't any data corruption of any sort. Running "find /mnt/raid -type f -ls" on the server works perfectly and lists about 25000 files without the slightest trouble.

It works equally well if I mount the NFS share on the server itself.

Now it's becoming crazier: When I run the find command as previously said, it freezes always on the same file, for instance :

/server/folder1/folder2/folder3/folder4/.svn/somefile

However, if after a fresh reboot I do

stat /server/folder1/folder2/folder3/folder4/.svn/somefile

no problem. Even doing this:

cd /server/folder1/folder2/folder3/folder4/ && find . -type f -ls

works. However this

cd /server/folder1/folder2/folder3/ && find . -type f -ls

doesn't fly. It freezes at exactly the same point. In the first test (running directly from /server) it freezes after successfully listing 10000 files. In the last test it freezes after only 25 files.
So apparently it's not about the number of files.

Quoted 7 lines

Now I'm stuck. Out of going through tcpdump, I have absolutely not the faintest idea about what's going on, except I tend to think that's some Ubuntu kernel bug.

Any hint, idea, etc would be extremely welcome. Even some debugging method less painful than digging through huge tcpdumps would be nice :)

Well, it sounds like you have a reproducer that shouldn't be *too* huge (the test where it freezes after stat'ing 25 files).

What do you see on the network in that case?

Are you literally using just tcpdump? Wireshark will give more (and easier to read) information.

Does the server stop responding at some point, or reply with an error? Or does the getattr reply on the problem file look odd in any way?

--b.

#2 lore raw

Le Thu, 5 Sep 2013 16:45:36 -0400 vous écriviez:

Quoted 4 lines

Well, it sounds like you have a reproducer that shouldn't be *too* huge (the test where it freezes after stat'ing 25 files).

What do you see on the network in that case?

I didn't look yet what's actually happening, out of the drop in network throughput.

Quoted 2 lines

Are you literally using just tcpdump? Wireshark will give more (and easier to read) information.

I didn't install tcpdump yet, but isn't wireshark with a GUI? I can't run anything with a GUI, this is a remote site behind several ssh portals. So I should run tshark, then get my hands on the results to analyze them with wireshark on my PC. Will try that.

Quoted 1 line

Does the server stop responding at some point, or reply with an error?

No response. I don't know if the server actually stops responding or if something's wrong on the client side. Unfortunately I don't have access to any other NFS client on this network, out of this VM and the server itself. When rebooting the VM reconnects to the server all by itself, so the server most probably is OK at all times.

Quoted 1 line

Or does the getattr reply on the problem file look odd in any way?

Not odd at all; it just stops after a particular file, though this file and the following files can be accessed OK before I run the failing test.

regards,

Signature · 6 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------
#3 lore raw
Quoted 10 lines · Emmanuel Florac

Le Thu, 5 Sep 2013 16:45:36 -0400 vous écriviez:

Quoted 4 lines

Well, it sounds like you have a reproducer that shouldn't be *too* huge (the test where it freezes after stat'ing 25 files).

What do you see on the network in that case?

I didn't look yet what's actually happening, out of the drop in network throughput.

Quoted 7 lines
Quoted 2 lines

Are you literally using just tcpdump? Wireshark will give more (and easier to read) information.

I didn't install tcpdump yet, but isn't wireshark with a GUI? I can't run anything with a GUI, this is a remote site behind several ssh portals. So I should run tshark, then get my hands on the results to analyze them with wireshark on my PC. Will try that.

Right, I just do "tcpdump -s0 -wtmp.pcap -i<interface>" and then run wireshark on tmp.pcap.

Quoted 8 lines
Quoted 1 line

Does the server stop responding at some point, or reply with an error?

No response. I don't know if the server actually stops responding or if something's wrong on the client side. Unfortunately I don't have access to any other NFS client on this network, out of this VM and the server itself. When rebooting the VM reconnects to the server all by itself, so the server most probably is OK at all times.

Quoted 5 lines
Quoted 1 line

Or does the getattr reply on the problem file look odd in any way?

Not odd at all; it just stops after a particular file, though this file and the following files can be accessed OK before I run the failing test.

I was asking about the on-the-wire errors and getattr replies here, not the application system calls.

--b.

#4 lore raw

Le Thu, 5 Sep 2013 17:40:02 -0400
"J. Bruce Fields" <[email protected]> écrivait:

Quoted 3 lines

I was asking about the on-the-wire errors and getattr replies here, not the application system calls.

OK, I've done the dump; I've kept just the last few working calls, then the failure calls, and filtered for NFS traffic (there are SSH and LACP frames interspersed here). I have absolutely no idea about what's going on there :) Any light from the network savvy?

Signature · 6 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------
#5 lore raw
Quoted 11 lines · Emmanuel Florac

Le Thu, 5 Sep 2013 17:40:02 -0400
"J. Bruce Fields" <[email protected]> écrivait:

Quoted 3 lines

I was asking about the on-the-wire errors and getattr replies here, not the application system calls.

OK, I've done the dump; I've kept just the last few working calls, then the failure calls, and filtered for NFS traffic (there are SSH and LACP frames interspersed here). I have absolutely no idea about what's going on there :) Any light from the network savvy?

Weird. Things look normal up through frame 14, which is a READDIRPLUS reply. Then the server resends the reply after .2s, and and the client resends its call shortly thereafter (but without acking the latest reply). And then the rest of the trace is resends of the reply.

So it looks like the client stopped ACKing the server's replies?

You may also have filtered out some TCP ACKs, which makes this harder to work out.

Looks like a networking problem, but I don't know.

--b.

#6 lore raw

Le Fri, 6 Sep 2013 12:07:35 -0400
"J. Bruce Fields" <[email protected]> écrivait:

Quoted 10 lines

Weird. Things look normal up through frame 14, which is a READDIRPLUS reply. Then the server resends the reply after .2s, and and the client resends its call shortly thereafter (but without acking the latest reply). And then the rest of the trace is resends of the reply.

So it looks like the client stopped ACKing the server's replies?

You may also have filtered out some TCP ACKs, which makes this harder to work out.

Ah yes, my bad, one TCP ACK was filtered out. Here I've kept trafic between the two machines but ssh.

Here I was capturing from the server, maybe I should try capturing on the client side?

Signature · 6 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------
#7 lore raw
Quoted 9 lines · Emmanuel Florac

Le Fri, 6 Sep 2013 12:07:35 -0400
"J. Bruce Fields" <[email protected]> écrivait:

Quoted 6 lines

Weird. Things look normal up through frame 14, which is a READDIRPLUS reply. Then the server resends the reply after .2s, and and the client resends its call shortly thereafter (but without acking the latest reply). And then the rest of the trace is resends of the reply.

Quoted 7 lines
Quoted 4 lines

So it looks like the client stopped ACKing the server's replies?

You may also have filtered out some TCP ACKs, which makes this harder to work out.

Ah yes, my bad, one TCP ACK was filtered out. Here I've kept trafic between the two machines but ssh.

Huh, no idea. You can see the server retransmitting the readdir plus reply, and still no ACKs from the client.

Quoted 2 lines

Here I was capturing from the server, maybe I should try capturing on the client side?

Sure, maybe.

Honestly looks like a network problem, if it weren't for the failing on the filesystem operation each time.

Hm, it may just be the first packet of a certain size. In fact it's the first frame > 1500 bytes in that trace. Is there some problem with jumbo frame configuration on your network?

--b.

#8 lore raw

Le Tue, 10 Sep 2013 09:28:41 -0400
"J. Bruce Fields" <[email protected]> écrivait:

Quoted 9 lines

Sure, maybe.

Honestly looks like a network problem, if it weren't for the failing on the filesystem operation each time.

Hm, it may just be the first packet of a certain size. In fact it's the first frame > 1500 bytes in that trace. Is there some problem with jumbo frame configuration on your network?

Interesting idea. I could try forcing both interfaces to 1500 mtu and see what's happening. Oh, maybe the VMWare host interface /isn't/ set to 9000? I'll check that. Thank you.

Signature · 6 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------
#9 lore raw
In reply to Emmanuel Florac

Emmanuel Florac wrote:

Le Thu, 5 Sep 2013 17:40:02 -0400
"J. Bruce Fields" <[email protected]> écrivait:

Quoted 3 lines

I was asking about the on-the-wire errors and getattr replies here, not the application system calls.

OK, I've done the dump; I've kept just the last few working calls, then the failure calls, and filtered for NFS traffic (there are SSH and LACP frames interspersed here). I have absolutely no idea about what's going on there :) Any light from the network savvy?

I suggest you filter on port=2049 instead of proto=nfs.

#10 lore raw

Re: Hard to debug NFS loss of connectivity: problem solved

In reply to Emmanuel Florac

Le Thu, 5 Sep 2013 19:18:00 +0200
Emmanuel Florac <[email protected]> écrivait:

Quoted 3 lines

Any hint, idea, etc would be extremely welcome. Even some debugging method less painful than digging through huge tcpdumps would be nice :)

I post this answer in the faint hope that it may avoid a long and painful week of testing to someone else :)

The problem comes from the virtual network adapter "Intel pro 1000" using the e1000e linux driver. The Intel hardware supports a MTU of 4078 max. On a physical machine, it will fail to set up a greater value with something like:

# ifconfig eth0 mtu 4079
SIOCSIFMTU: Invalid argument

However the virtual e1000 from VMWare ESX silently accept any value; then it will mostly work, and fail in mysterious way and only with certain protocols (such as nfs).

In the case of VMWare virtual machines, using a vmnet3 virtual network adapter works fine with a mtu of 9000 under NFS. Therefore the problem is solved.

For the sake of completeness and curiosity, I'll try and see what happens with a KVM e1000 virtual device.

Signature · 6 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------
#11 lore raw

Re: Hard to debug NFS loss of connectivity: problem solved

Quoted 15 lines · Emmanuel Florac

Le Thu, 5 Sep 2013 19:18:00 +0200
Emmanuel Florac <[email protected]> écrivait:

Quoted 3 lines

Any hint, idea, etc would be extremely welcome. Even some debugging method less painful than digging through huge tcpdumps would be nice :)

I post this answer in the faint hope that it may avoid a long and painful week of testing to someone else :)

The problem comes from the virtual network adapter "Intel pro 1000" using the e1000e linux driver. The Intel hardware supports a MTU of 4078 max. On a physical machine, it will fail to set up a greater value with something like:

Quoted 6 lines
# ifconfig eth0 mtu 4079
SIOCSIFMTU: Invalid argument

However the virtual e1000 from VMWare ESX silently accept any value; then it will mostly work, and fail in mysterious way and only with certain protocols (such as nfs).

Ah-hah!

Thanks for the followup.--b.

Quoted 19 lines

In the case of VMWare virtual machines, using a vmnet3 virtual network adapter works fine with a mtu of 9000 under NFS. Therefore the problem is solved.

For the sake of completeness and curiosity, I'll try and see what happens with a KVM e1000 virtual device.

Signature · 10 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html