Skip to contentSkip to conversation

read(2) hangs on the client side

DISCUSSION Haakon Riiser 2 participants 14 messages May 8, 2005 – May 15, 2005

Subsystems (2)
NFS client NFS server
lore
Share & tools
lore

Haakon Riiser

author
#0 lore raw

I have noticed that there is a specific case that can lock up the client while doing read(2), and it seems to be a race condition that only occures in a very specific situation. I will try to describe this in as much detail as possible, since it is unlikely that you'll be able to reproduce the bug for yourselves:

A big file (~ 250 MB) is shared on the NFS server. The NFS server also acts as a Samba server, so that Windows machines can use it. One of the Windows machines is running the eMule P2P client, and it makes some of the Samba-hosted files available on the eMule network.

The hang has _only_ happened when I try to access a shared file via NFS while it is _simultaneously_ being accessed (via Samba) by the eMule machine. To reproduce the hang from the NFS client's side, I use this C program:

        #include <sys/types.h>
        #include <sys/stat.h>
        #include <fcntl.h>
        #include <unistd.h>

        int main(int argc, char *argv[])
        {
                for (;;) {
                        char buf[4096];
                        int fd = open(argv[1], O_RDONLY);
                        read(fd, buf, 4096);
                        close(fd);
                }
                return 0;
        }

It always hangs in the read() call, usually in the first iteration, and after it does, _nothing_ can kill it, not even SIGKILL, and until the next reboot, _any_ NFS operation -- even stat() -- on the accessed file will now hang.

I have no idea how Samba + eMule's access patterns look like, but I know with 100 % certainty that it is the cause. If I move the file in question out of eMule's shared directory, I can never hang the NFS client no matter how long I run the above program. If I move it back in, it hangs almost instantly. :-(

Note that only the Linux NFS client machine is seemingly affected by this -- both the NFS/Samba server and the eMule-running Samba client are doing just fine while the hang happens on the Linux client.

Some system info:

NFS client:
Slackware 10.1
Linux 2.6.11
nfs-utils 1.0.7
glibc 2.3.4
util-linux 2.12p

NFS server:
Fedora Core 3 (fully updated)
Linux 2.6.11-1.14_FC3
nfs-utils-1.0.6-52
glibc-2.3.5-0.fc3.1
Samba 3.0.15pre2-1
util-linux-2.12a-24.2

Any help in further analysis would be greatly appreciated!

Signature · 11 lines
 Haakon


-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.
Get your fingers limbered up and give it your best shot. 4 great events, 4
opportunities to win big! Highest score wins.NEC IT Guy Games. Play to
win an NEC 61 plasma display. Visit http://www.necitguy.com/?r=20
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#1 lore raw

su den 08.05.2005 Klokka 13:33 (+0200) skreiv Haakon Riiser:

Quoted 5 lines

I have no idea how Samba + eMule's access patterns look like, but I know with 100 % certainty that it is the cause. If I move the file in question out of eMule's shared directory, I can never hang the NFS client no matter how long I run the above program. If I move it back in, it hangs almost instantly. :-(

Are oplocks enabled on the Samba server?

Cheers,
Trond

-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.
Get your fingers limbered up and give it your best shot. 4 great events, 4 opportunities to win big! Highest score wins.NEC IT Guy Games. Play to win an NEC 61 plasma display. Visit http://www.necitguy.com/?r=20
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

author
#2 lore raw

Trond,

Quoted 9 lines

su den 08.05.2005 Klokka 13:33 (+0200) skreiv Haakon Riiser:

Quoted 5 lines

I have no idea how Samba + eMule's access patterns look like, but I know with 100 % certainty that it is the cause. If I move the file in question out of eMule's shared directory, I can never hang the NFS client no matter how long I run the above program. If I move it back in, it hangs almost instantly. :-(

Are oplocks enabled on the Samba server?

Probably. I haven't touched the oplock setting, but according to smb.conf(5), kernel oplocks are enabled by default, and they translate to no-ops when the kernel doesn't support them. How do I determine if I have kernel support?

Signature · 11 lines
 Haakon


-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.
Get your fingers limbered up and give it your best shot. 4 great events, 4
opportunities to win big! Highest score wins.NEC IT Guy Games. Play to
win an NEC 61 plasma display. Visit http://www.necitguy.com/?r=20
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#3 lore raw

su den 08.05.2005 Klokka 15:21 (+0200) skreiv Haakon Riiser:

Quoted 6 lines
Quoted 1 line

Are oplocks enabled on the Samba server?

Probably. I haven't touched the oplock setting, but according to smb.conf(5), kernel oplocks are enabled by default, and they translate to no-ops when the kernel doesn't support them. How do I determine if I have kernel support?

Kernel support for oplocks has been in Linux since version 2.4 - see the "lease" locks on the fcntl manpage. They haven't been maintained for quite some while, though until Andy started working on the NFSv4 server delegation code (in 2.6.11 or so).

Anyhow, try turning oplocks off in smb.conf and see if the hangs disappear.

Cheers,
Trond

-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.
Get your fingers limbered up and give it your best shot. 4 great events, 4 opportunities to win big! Highest score wins.NEC IT Guy Games. Play to win an NEC 61 plasma display. Visit http://www.necitguy.com/?r=20
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

author
#4 lore raw

Trond,

Quoted 2 lines

Anyhow, try turning oplocks off in smb.conf and see if the hangs disappear.

Still hangs. I also tried enabling fake oplocks, but that didn't make any difference either. Just so you know: I made sure to stop all Samba processes (service smb stop) so I /know/ that the new oplock settings were in effect.

Signature · 11 lines
 Haakon


-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.
Get your fingers limbered up and give it your best shot. 4 great events, 4
opportunities to win big! Highest score wins.NEC IT Guy Games. Play to
win an NEC 61 plasma display. Visit http://www.necitguy.com/?r=20
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#5 lore raw

su den 08.05.2005 Klokka 16:37 (+0200) skreiv Haakon Riiser:

Quoted 9 lines

Trond,

Quoted 2 lines

Anyhow, try turning oplocks off in smb.conf and see if the hangs disappear.

Still hangs. I also tried enabling fake oplocks, but that didn't make any difference either. Just so you know: I made sure to stop all Samba processes (service smb stop) so I /know/ that the new oplock settings were in effect.

Do you have a tcpdump of the traffic?

Cheers,
Trond

-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.
Get your fingers limbered up and give it your best shot. 4 great events, 4 opportunities to win big! Highest score wins.NEC IT Guy Games. Play to win an NEC 61 plasma display. Visit http://www.necitguy.com/?r=20
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

author
#6 lore raw

Trond,

Quoted 1 line

Do you have a tcpdump of the traffic?

Should I run tcpdump on the server or the client, or perhaps it doesn't matter?

Signature · 11 lines
 Haakon


-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.
Get your fingers limbered up and give it your best shot. 4 great events, 4
opportunities to win big! Highest score wins.NEC IT Guy Games. Play to
win an NEC 61 plasma display. Visit http://www.necitguy.com/?r=20
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#7 lore raw

m=C3=A5 den 09.05.2005 Klokka 16:13 (+0200) skreiv Haakon Riiser:

Quoted 6 lines

Trond,
=20

Quoted 1 line

Do you have a tcpdump of the traffic?

=20
Should I run tcpdump on the server or the client, or perhaps it doesn't matter?

Please try on the client first.

Cheers,
Trond

-------------------------------------------------------
This SF.Net email is sponsored by Oracle Space Sweepstakes Want to be the first software developer in space? Enter now for the Oracle Space Sweepstakes!
http://ads.osdn.com/?ad_id=7393&alloc_id=16281&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

author
#8 lore raw

Trond,

Quoted 4 lines
Quoted 2 lines

Should I run tcpdump on the server or the client, or perhaps it doesn't matter?

Please try on the client first.

Done. I tcpdump'd all data (-s0) to file, and opened it up in Ethereal to make some sense of it. I think these packets are the ones of interest to you: (10.0.0.2 is the NFS server and 10.0.0.4 is the NFS client)

-------------------------------------------------------------------------------

No.     Time        Source                Destination           Protocol Info
     51 8.020185    10.0.0.4              10.0.0.2              NFS      V3 READ Call (Reply In 52), FH:0x2322d19a Offset:0 Len:4096

Frame 51 (190 bytes on wire, 190 bytes captured)
Ethernet II, Src: 00:01:02:fa:af:5f, Dst: 00:50:04:55:8c:44
Internet Protocol, Src Addr: 10.0.0.4 (10.0.0.4), Dst Addr: 10.0.0.2 (10.0.0.2) Transmission Control Protocol, Src Port: 799 (799), Dst Port: nfsd (2049), Seq: 2048, Ack: 6256, Len: 124 Remote Procedure Call, Type:Call XID:0x70cf1c9d
Network File System, READ Call FH:0x2322d19a Offset:0 Len:4096
Program Version: 3
V3 Procedure: READ (6)
file
offset: 0
count: 4096

No.     Time        Source                Destination           Protocol Info
     52 8.020731    10.0.0.2              10.0.0.4              NFS      V3 READ Reply (Call In 51) Error:Unknown error:10008

Frame 52 (186 bytes on wire, 186 bytes captured)
Ethernet II, Src: 00:50:04:55:8c:44, Dst: 00:01:02:fa:af:5f
Internet Protocol, Src Addr: 10.0.0.2 (10.0.0.2), Dst Addr: 10.0.0.4 (10.0.0.4) Transmission Control Protocol, Src Port: nfsd (2049), Dst Port: 799 (799), Seq: 6256, Ack: 2172, Len: 120 Remote Procedure Call, Type:Reply XID:0x70cf1c9d
Network File System, READ Reply Error:Unknown error:10008
Program Version: 3
V3 Procedure: READ (6)
Status: ERR_JUKEBOX (10008)
file_attributes

No.     Time        Source                Destination           Protocol Info
     53 8.059968    10.0.0.4              10.0.0.2              TCP      799 > nfsd [ACK] Seq=2172 Ack=6376 Win=17376 Len=0 TSV=4294774881 TSER=1928447303

Frame 53 (66 bytes on wire, 66 bytes captured)
Ethernet II, Src: 00:01:02:fa:af:5f, Dst: 00:50:04:55:8c:44
Internet Protocol, Src Addr: 10.0.0.4 (10.0.0.4), Dst Addr: 10.0.0.2 (10.0.0.2) Transmission Control Protocol, Src Port: 799 (799), Dst Port: nfsd (2049), Seq: 2172, Ack: 6376, Len: 0
-------------------------------------------------------------------------------

As you can see in the above, I'm running NFS over TCP, but I have also tried UDP, and it still hangs. Anyway, have you seen this before? ERR_JUKEBOX (10008) has only three hits on Google, and all of them point to the patches to Ethereal that made it recognize this error. :-)

Signature · 11 lines
 Haakon


-------------------------------------------------------
This SF.Net email is sponsored by Oracle Space Sweepstakes
Want to be the first software developer in space?
Enter now for the Oracle Space Sweepstakes!
http://ads.osdn.com/?ad_id=7393&alloc_id=16281&op=click
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#9 lore raw

su den 15.05.2005 Klokka 00:39 (+0200) skreiv Haakon Riiser:

Quoted 20 lines

Frame 52 (186 bytes on wire, 186 bytes captured)
Ethernet II, Src: 00:50:04:55:8c:44, Dst: 00:01:02:fa:af:5f
Internet Protocol, Src Addr: 10.0.0.2 (10.0.0.2), Dst Addr: 10.0.0.4 (10.0.0.4) Transmission Control Protocol, Src Port: nfsd (2049), Dst Port: 799 (799), Seq: 6256, Ack: 2172, Len: 120 Remote Procedure Call, Type:Reply XID:0x70cf1c9d
Network File System, READ Reply Error:Unknown error:10008
Program Version: 3
V3 Procedure: READ (6)
Status: ERR_JUKEBOX (10008)
file_attributes

No.     Time        Source                Destination           Protocol Info
     53 8.059968    10.0.0.4              10.0.0.2              TCP      799 > nfsd [ACK] Seq=2172 Ack=6376 Win=17376 Len=0 TSV=4294774881 TSER=1928447303

Frame 53 (66 bytes on wire, 66 bytes captured)
Ethernet II, Src: 00:01:02:fa:af:5f, Dst: 00:50:04:55:8c:44
Internet Protocol, Src Addr: 10.0.0.4 (10.0.0.4), Dst Addr: 10.0.0.2 (10.0.0.2) Transmission Control Protocol, Src Port: 799 (799), Dst Port: nfsd (2049), Seq: 2172, Ack: 6376, Len: 0
-------------------------------------------------------------------------------

Quoted 5 lines

As you can see in the above, I'm running NFS over TCP, but I have also tried UDP, and it still hangs. Anyway, have you seen this before? ERR_JUKEBOX (10008) has only three hits on Google, and all of them point to the patches to Ethereal that made it recognize this error. :-)

Why didn't you try the NFSv3 RFC? 8-)

NFS3ERR_JUKEBOX
The server initiated the request, but was not able to complete it in a timely fashion. The client should wait and then try the request with a new RPC transaction ID. For example, this error should be returned from a server that supports hierarchical storage and receives a request to process a file that has been migrated. In this case, the server should start the immigration process and respond to client with this error.

In other words, this is an error used by the server to say "I'm busy, and cannot retrieve the data you want. Please try again later".

Are you absolutely sure that you managed to disable oplocks on the samba server? 'cos this is the error an NFS server will usually return when it is waiting for the samba server (or an NFSv4 server) to break the lease on a file so that it can be opened.

Cheers,
Trond

-------------------------------------------------------
This SF.Net email is sponsored by Oracle Space Sweepstakes Want to be the first software developer in space? Enter now for the Oracle Space Sweepstakes!
http://ads.osdn.com/?ad_id=7393&alloc_id=16281&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

author
#10 lore raw

Trond,

Quoted 12 lines

Why didn't you try the NFSv3 RFC? 8-)

NFS3ERR_JUKEBOX
The server initiated the request, but was not able to complete it in a timely fashion. The client should wait and then try the request with a new RPC transaction ID. For example, this error should be returned from a server that supports hierarchical storage and receives a request to process a file that has been migrated. In this case, the server should start the immigration process and respond to client with this error.

Quoted 7 lines

In other words, this is an error used by the server to say "I'm busy, and cannot retrieve the data you want. Please try again later".

Are you absolutely sure that you managed to disable oplocks on the samba server? 'cos this is the error an NFS server will usually return when it is waiting for the samba server (or an NFSv4 server) to break the lease on a file so that it can be opened.

I did set 'kernel oplocks = no' globally and restarted Samba, but I didn't look for locked files with smbstatus. I'll try again and investigate a little more this time. In the meantime, here's what smbstatus shows:

Locked files:
Pid DenyMode Access R/W Oplock Name
--------------------------------------------------------
32475 DENY_NONE 0x20089 RDONLY EXCLUSIVE+BATCH foo Sun May 15 08:13:23 2005

This lock goes on and off all the time, which probably explains why this error doesn't happen every single time I try to open the file. But why does ERR_JUKEBOX cause the client to hang? This lock _is_ soon released, so shouldn't NFS wake up once that happens?

Signature · 11 lines
 Haakon


-------------------------------------------------------
This SF.Net email is sponsored by Oracle Space Sweepstakes
Want to be the first software developer in space?
Enter now for the Oracle Space Sweepstakes!
http://ads.osdn.com/?ad_id=7393&alloc_id=16281&op=click
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#11 lore raw

su den 15.05.2005 Klokka 08:21 (+0200) skreiv Haakon Riiser:

Quoted 10 lines

I did set 'kernel oplocks = no' globally and restarted Samba, but I didn't look for locked files with smbstatus. I'll try again and investigate a little more this time. In the meantime, here's what smbstatus shows:

Locked files:
Pid DenyMode Access R/W Oplock Name
--------------------------------------------------------
32475 DENY_NONE 0x20089 RDONLY EXCLUSIVE+BATCH foo Sun May 15 08:13:23 2005

Quoted 4 lines

This lock goes on and off all the time, which probably explains why this error doesn't happen every single time I try to open the file. But why does ERR_JUKEBOX cause the client to hang? This lock _is_ soon released, so shouldn't NFS wake up once that happens?

I suspect there were hanging locks. Some of those lease code changes look pretty suspicious, so I think I want to have a chat with Andy & co as soon as I get back to Michigan.

In the mean time, please try rebooting the server, and just turn off the lease code using 'sysctl -w fs.leases-enable=0'.

Cheers,
Trond

Cheers,
Trond

-------------------------------------------------------
This SF.Net email is sponsored by Oracle Space Sweepstakes Want to be the first software developer in space? Enter now for the Oracle Space Sweepstakes!
http://ads.osdn.com/?ad_id=7393&alloc_id=16281&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

author
#12 lore raw

Trond,

I actually managed to fix the hang with a third oplock option in smb.conf that I missed before: 'oplocks = no'. The last time, I tried toggling fake oplocks and kernel oplocks, but neither of those made any difference. The plain 'oplocks' parameter did though! I've been running my test program for almost 25 minutes, and still no hang.

Thanks for your persistence in tracking this down. I'll have oplocks disabled for now.

Quoted 3 lines

I suspect there were hanging locks. Some of those lease code changes look pretty suspicious, so I think I want to have a chat with Andy & co as soon as I get back to Michigan.

So this is, after all, a bug in NFS? Strange that it hasn't hit more people. Searching for similar problems on Google Groups turned up nothing.

Signature · 11 lines
 Haakon


-------------------------------------------------------
This SF.Net email is sponsored by Oracle Space Sweepstakes
Want to be the first software developer in space?
Enter now for the Oracle Space Sweepstakes!
http://ads.osdn.com/?ad_id=7393&alloc_id=16281&op=click
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#13 lore raw

su den 15.05.2005 Klokka 08:54 (+0200) skreiv Haakon Riiser:

Quoted 7 lines
Quoted 3 lines

I suspect there were hanging locks. Some of those lease code changes look pretty suspicious, so I think I want to have a chat with Andy & co as soon as I get back to Michigan.

So this is, after all, a bug in NFS? Strange that it hasn't hit more people. Searching for similar problems on Google Groups turned up nothing.

I suspect rather that it is a bug in the VFS' lease code. That code was jiggled around a bit during the NFSv4 delegation code development.

Cheers,
Trond

-------------------------------------------------------
This SF.Net email is sponsored by Oracle Space Sweepstakes Want to be the first software developer in space? Enter now for the Oracle Space Sweepstakes!
http://ads.osdn.com/?ad_id=7393&alloc_id=16281&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs