Skip to contentSkip to conversation

ls hangs on NFS share from Apple Xserve

DISCUSSION [email protected] 2 participants 10 messages Sep 8, 2004 – Sep 9, 2004

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

A couple of months ago I reported a bug that occurs with Linux NFS client and Mac OS X NFS server. Here's the bug report:

http://sourceforge.net/tracker/index.php?func=detail&aid=964204&group_id=14&atid=100014

And here are other docs related to this problem:

http://discussions.info.apple.com/[email protected]@.689495bc
http://discussions.info.apple.com/[email protected]@.6897b10d/2
http://groups.google.ch/groups?hl=de&lr=&ie=UTF-8&selm=9c87e8e6.0405280036.16f3c991%40posting.google.com
http://algesten.blogspot.com/2004/07/mac-os-x-server-nfs-unique-cookie-bug.html
http://sources.redhat.com/bugzilla/show_bug.cgi?id=353

Martin Algesten reported that this might be caused by non-unique NFS cookies in the same READDIRPLUS reply from the Apple NFS server but in our environment with MacOS X 10.3.5 and SuSE Linux 9.0 (Kernel 2.4.21-243-smp4G, glibc-2.3.2-88) I cannot confirm this. The NFS cookies are unique in the same READDIRPLUS reply but not unique during the same NFS session.

We don't have problems accessing the MacOS X 10.3.5 NFS server from our Solaris clients and also don't have problems with Solaris, Linux and NetApp NFS servers from Linux clients.

I assume this is a bug with the Linux Kernel NFS implementation or the glibc library which provides the getdents64 call that loops. Apple says it's a Linux bug.

According to SuSE the patch linux-2.4.21-15-seekdir.dif from http://client.linux-nfs.org/Linux-2.4.x/2.4.21/
is already included in their kernel.

Could someone who is familiar with NFS and the Linux NFS or glibc code analyze and solve this problem? I can provide network traffic and strace logs if you don't have the equipment to reproduce this problem.

Thanks in advance.

Regards,
Bernd

-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#1 lore raw

P=E5 on , 08/09/2004 klokka 04:42, skreiv [email protected]:

Quoted 1 line

Martin Algesten reported that this might be caused by non-unique NFS cook=

ies

Quoted 1 line

in the same READDIRPLUS reply from the Apple NFS server but in our enviro=

nment

Quoted 1 line

with MacOS X 10.3.5 and SuSE Linux 9.0 (Kernel 2.4.21-243-smp4G, glibc-2.=

3.2-88)

Quoted 3 lines

I cannot confirm this. The NFS cookies are unique in the same READDIRPLUS reply but not unique during the same NFS session.
=20

What does this mean? That 2 different READDIRPLUS entries for the same directory may return the same cookie? That would a server bug...

Cheers,
Trond

-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

author
#2 lore raw

Hi Trond,

Quoted 10 lines
Quoted 6 lines

Martin Algesten reported that this might be caused by non-unique NFS cookies in the same READDIRPLUS reply from the Apple NFS server but in our environment with MacOS X 10.3.5 and SuSE Linux 9.0 (Kernel 2.4.21-243-smp4G, glibc-2.3.2-88) I cannot confirm this. The NFS cookies are unique in the same READDIRPLUS reply but not unique during the same NFS session.

What does this mean? That 2 different READDIRPLUS entries for the same directory may return the same cookie? That would a server bug...

The problem reported by Martin Algesten was: His MacOS X 10.3.4 NFS server sent a duplicate Cookie within the _same_ READDIRPLUS reply. See:

http://algesten.blogspot.com/index.html#109103599929132337

I can't observe that on our MacOS X 10.3.5 server. I only see that while traversing a NFS share with ls -lR on a Linux client the same Cookie ID appears in different READDIRPLUS replies of different directories. Because our Solaris NFS servers do the same and Linux NFS client works fine with it, I guess this is allowed.

So I think the endless looping ls -lR from a Linux NFS client on a Mac OS X 10.3.5 NFS share I reported is not related to that bug reported by Martin Algesten.

Here is the ethereal snoop of such an incident:

http://www.nies.ch/download/apple-linux-nfs-loop.tar.gz

I did a ls -lR on a MacOS X NFS share and in a certain directory it hung. A strace on the ls process shows that it loops over these system calls:

getdents64(5, /* 3 entries */, 4096) = 128 lstat64("/share/dir/file1.txt", {st_mode=S_IFREG|0444, st_size=8550, ...}) = 0
getxattr("/share/dir/file1.txt", "system.posix_acl_access", (nil), 0) = -1 EOPNOTSUPP (Operation not supported) lstat64("/share/dir/file2.txt", {st_mode=S_IFREG|0444, st_size=6570, ...}) = 0 getxattr("/share/dir/file2.txt", "system.posix_acl_access", (nil), 0) = -1 EOPNOTSUPP (Operation not supported) lstat64("/share/dir/file3.txt", {st_mode=S_IFREG|0444, st_size=23411, ...}) = 0 getxattr("/share/dir/file3.txt", "system.posix_acl_access", (nil), 0) = -1 EOPNOTSUPP (Operation not supported)

This does not only happen with ls. It happens with all commands that want to read a certain directory (e.g. tar). A quick workaround is to create a new file in the directory where it loops.

Regards,
Bernd

-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#3 lore raw

As far as I can see from your traces, 2 consecutive READDIRs on the same directory is giving totally different results...

In your TCP file, the first time the Linux client reads through the directory which begins with "Autorun.inf", the entry "install.sh" (for instance) has cookie 292. The second time round, it has cookie 252.

Our client will not work with such a server.

Cheers,
Trond

-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

author
#4 lore raw
Quoted 8 lines · Trond Myklebust

As far as I can see from your traces, 2 consecutive READDIRs on the same directory is giving totally different results...

In your TCP file, the first time the Linux client reads through the directory which begins with "Autorun.inf", the entry "install.sh" (for instance) has cookie 292. The second time round, it has cookie 252.

Our client will not work with such a server.

Are you sure it reads the same directory twice? A ls -lR shouldn't do this and I ran it only once per traffic capture. Between the UDP and NFS traffics I unmounted the directory. The share contains directories which contents are sometimes similar.

The server is working fine. Only on about one of thousand directories the client loops and if Client A loops over one directory, client B does not.

Regards,
Bernd

-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#5 lore raw

P=E5 on , 08/09/2004 klokka 16:07, skreiv Bernd Nies:

Quoted 1 line

Are you sure it reads the same directory twice? A ls -lR shouldn't do=20

Yes. Check the tcpdumps.

Quoted 1 line

this and I ran it only once per traffic capture. Between the UDP and

NFS=20

Quoted 2 lines

traffics I unmounted the directory. The share contains directories which=20 contents are sometimes similar.

Huh? Of course the client may end up reading in the directory multiple times.

Unless you have allocated a large buffer for it, readdir() will end up calling multiple times down into the kernel, so the read of an entire directory is NOT going to be atomic.
If the mtime on the directory has changed in the meantime, the cached version may be flushed out, and the directory read in again. If the server has screwed up the cookies, then that will confuse the client.

Cheers,
Trond

-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

author
#6 lore raw
Quoted 7 lines

Unless you have allocated a large buffer for it, readdir() will end up calling multiple times down into the kernel, so the read of an entire directory is NOT going to be atomic.
If the mtime on the directory has changed in the meantime, the cached version may be flushed out, and the directory read in again. If the server has screwed up the cookies, then that will confuse the client.

OK, right. I tracked down one incident captured here:

http://www.nies.ch/download/apple-nfs-loop-1dir.dump.gz

A READDIRPLUS reports a directory entry "." with cookie 276 (Frame 157) and the next READDIRPLUS reports a file "package-use.html" with same cookie 276 (Frame 161). This is WRONG by the server, right?

But why is it working then without problems on a Solaris NFS client?

Well, seems that I have to call Apple support and listen to one hour iTunes again ...

Thanks a lot.

Regards,
Bernd

-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#7 lore raw

P=E5 on , 08/09/2004 klokka 17:51, skreiv Bernd Nies:

Quoted 9 lines

OK, right. I tracked down one incident captured here:
=20
http://www.nies.ch/download/apple-nfs-loop-1dir.dump.gz
=20
A READDIRPLUS reports a directory entry "." with cookie 276 (Frame 157)=20 and the next READDIRPLUS reports a file "package-use.html" with same=20 cookie 276 (Frame 161). This is WRONG by the server, right? =20
But why is it working then without problems on a Solaris NFS client?

They probably do not rely exclusively on the cookie to track where they are in the READDIR stream: for instance it is possible to cache the filename too, and to use that as a backup solution if the server is being cranky.
We could possibly add this sort of information to Linux too.

Note though, this scheme fails for the case where the user wants to use telldir()/seekdir(), so if you have an older glibc that uses the horrible heuristic algorithm, you will still see odd behaviour. More obviously: it also breaks down if the filename you cached was one of the entries that got deleted...

Cheers,
Trond

-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#8 lore raw

Hi Trond,

Quoted 5 lines

They probably do not rely exclusively on the cookie to track where they are in the READDIR stream: for instance it is possible to cache the filename too, and to use that as a backup solution if the server is being cranky.
We could possibly add this sort of information to Linux too.

That's probably a good idea. At least the network file system connecting each flavours of Unix should be consistent and reliable. Maybe this bug also exists in other BSD derivates. Sun developed NFS. Is the Solaris client and server code available?

I filed a bug in OpenDarwins Bugzilla:

http://www.opendarwin.org/bugzilla/show_bug.cgi?id=2201

Seems that Mac OS X NFS implementation is a kind of buggy, because I found other problems with it:

http://discussions.info.apple.com/[email protected]@.6899e34d
http://discussions.info.apple.com/[email protected]@.6899e2eb
http://discussions.info.apple.com/[email protected]@.6899e444

Regards,
Bernd

-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#9 lore raw

P=E5 to , 09/09/2004 klokka 04:08, skreiv [email protected]:

Quoted 1 line

Sun developed NFS. Is the Solaris client and server code available?

You'll have to wait until they GPL Solaris 8-)

Cheers,
Trond

-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs