NFS problems with Linux-2.4

Hello all,

I am having some problems with NFS which I suspect may be a bug in the 2.4 kernels. I can probably come up with a small testcase, but before I do that I would like to describe the problem and see if it is something that is susposed to work. Perhaps I simply do not understand the guarantees that NFS makes.

The setup is like this. I have two machines which share an NFS mounted directory. The NFS server is a network appliance box. Machine A does an fopen/fwrite/fclose to create a file on the NFS filesystem. It then sends a message to machine B. Machine B then attemps to fopen the file, but fopen fails (as does stat). If I add code that sleeps for a couple of seconds and retries the fopen then everything works.

I have seen the problem on both IA64 machines running the kernel 2.4.18 from Red Hats Advanced Server and on x86 machines running Red Hats 2.4.7-10smp kernel. I have not tried other linux kernels (I am not root), but I have run the same program under Solaris (sparc) and have never observed this.

The IA64 and x86 machines were on different networks and using different network appliance servers. The IA64 /proc/mounts entry is:

na1:/vol/h1/home /remote/na1h1home nfs rw,v3,rsize=4096,wsize=4096,hard,intr,udp,lock,addr=na1 0 0

and the x86 entry is:

na1-rtp:/vol/vol0/home/jlnance /home/jlnance nfs rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=na1-rtp 0 0

If you would like more information, please let me know.

Thanks,

Jim

Could you please try with a newer kernel. The close-to-open cache consistency fixes are a relatively recent addition to the Linux NFS client. I dunno if RedHat's 2.4.18 kernel has them.

2.4.7 certainly does not.

Cheers,
Trond

I tried again with the 2.4.20 based kernel that Red Hat released yesterday (2.4.20-13.7bigmem). The problem that I was seeing occurs less frequently there, but it still happens.

I have attached a program which can reproduce this. If you run it under 2.4.7 it fails instantly. If you use 2.4.20 it may take a minute or so but it will also fail.

Thanks,

Jim

PS: Do you know if there is any way to work around this problem from
within my program?

----------------------------------------------------------------------------
Jim Nance                                                           Synopsys
(919) 425-7219  Do you have sweet iced tea?        jlnance at synopsys.com
                No, but there's sugar on the table.

/* This program demonstrates a problem with the close/open consistency
* of NFS file systems under Linux. It fails very rapidy with Red Hats
* 2.4.7-10smp kernel. This kernel was known to have bugs. It also fails
* with Red Hats 2.4.20-13.7bigmem kernel, which was thought to have this
* bug fixed. For my testcase both linux machines were talking to a
* network applicance file server and mounted like this:
*
* na1-rtp:/vol/vol0/home/jlnance /home/jlnance nfs rw,v3,rsize=4096,\
* wsize=4096,hard,intr,udp,lock,addr=na1-rtp 0 0
*
* This program needs to be run on 2 machines, assume hostnames A & B.
* A and B need to share an NFS mounted file system.
*
* On machine A:
* cd /some/nfs/path/common/to/both
* ./p1 s
*
* On machine B:
* cd /some/nfs/path/common/to/both
* ./p1 c A
*
* After a while you may see output similar to:

 *   cayman> ./p1 s
 *   Failed to find #0 which client wrote
 *   Failed on file number 483

*/

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <netdb.h>

#define PORT 12387
#define FLEN 16

void die()
{
perror("");
exit(-1);
}

void Write(int fd, char *buff, size_t len)
{

  for(;;) {
    int nsent=write(fd, buff, len);
    if(nsent==0)
      exit(0);
    if(nsent==-1) {
      if(errno!=EINTR)
        die();
    } else {
      buff += nsent;
      len  -= nsent;
      if(len==0) {
        return;
      }
    }

}
}

void Read(int fd, char *buff, size_t len)
{

  for(;;) {
    int nread=read(fd, buff, len);
    if(nread==0)
      exit(0);
    if(nread==-1) {
      if(errno!=EINTR)
        die();
    } else {
      buff += nread;
      len  -= nread;
      if(len==0) {
        return;
      }
    }

}
}

int server()
{
int sock = socket(AF_INET, SOCK_STREAM, 0);

  if(sock==-1) die(); else {
    struct sockaddr_in name;
    int                on = 1;
    name.sin_family       = AF_INET;
    name.sin_addr.s_addr  = htonl(INADDR_ANY);
    name.sin_port         = htons(PORT);

    setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, sizeof on);
    if(bind(sock, (struct sockaddr*)&name, sizeof(name))==-1) die(); else {
      if(listen(sock, 1)==-1) die(); else {
        int tsock = accept(sock, 0, 0);
        if(tsock!=-1) {
          int cnt;

          for(cnt=0; cnt<100000; cnt++) {
            int  fd;
            char dummy;
            char number[FLEN];
            struct stat sbuf;
            /*sprintf(number, "#%d", cnt);*/
            sprintf(number, "#%d", 0);
            Write(tsock, number, sizeof(number));
            Read(tsock, &dummy, 1);
            if(stat(number, &sbuf)) {
              fprintf(stderr, "Failed to find %s which client wrote\n", number);
              fprintf(stderr, "Failed on file number %d\n", cnt);
              exit(-2);
            }
            unlink(number);
          }
        }
      }
    }

}

return 0;
}

int client(char *server)
{
struct hostent *info = gethostbyname(server);

  if(!info) die(); else {
    int rsocket = socket(AF_INET, SOCK_STREAM, 0);
    if(rsocket==-1) die(); else {
      struct sockaddr_in name;
      name.sin_family = AF_INET;
      name.sin_port   = htons(PORT);
      memcpy(&name.sin_addr, info->h_addr_list[0], sizeof(struct in_addr));
      if(connect(rsocket, (struct sockaddr*)&name, sizeof(name))==-1)
        die();
      else {
        for(;;) {
          int  fd;
          char fname[FLEN];
          char tname[FLEN+8];

          Read(rsocket, fname, sizeof(fname));
          strcpy(tname, fname);
          strcat(tname, ".tmp");

          fd = open(tname, O_WRONLY|O_CREAT, 0600);
          if(fd==-1) die();

          Write(fd, fname, sizeof(fname)); /* Junk data */
          close(fd);

          rename(tname, fname);

          Write(rsocket, fname, 1); /* Tells the server we are done */
        }
      }
    }

}

return 0;
}

void usage(char *prog)
{
fprintf(stderr, "Usage:\n");
fprintf(stderr, " %s s\n", prog);
fprintf(stderr, " %s c servername\n", prog);
fprintf(stderr, " Run 1 of each in the same NFS directory on 2 different "

        "machines\n Two processes total\n");

exit(-1);
}

int main(int ac, char **av)
{

  if(ac<2) {
    usage(av[0]);
  } if(av[1][0]=='s') {
    return server();
  }else if(ac<3) {
    usage(av[0]);
  } else if(av[1][0]=='c') {
    return client(av[2]);
  } else {
    usage(av[0]);

}

return -1;
}

Sorry. stat doesn't obey close-to-open. It relies on standard attribute caching. close-to-open means "open()" (and only "open()") checks data cache consistency...

Cheers,
Trond

Trond,
Thanks for the info. Here is a section of the man page for open.
Is the information it gives correct wrt using link & stat?

O_EXCL When used with O_CREAT, if the file already exists
it is an error and the open will fail. In this con text, a symbolic link exists, regardless of where its points to. O_EXCL is broken on NFS file sys tems, programs which rely on it for performing
locking tasks will contain a race condition. The
solution for performing atomic file locking using a lockfile is to create a unique file on the same fs (e.g., incorporating hostname and pid), use link(2) to make a link to the lockfile. If link() returns 0, the lock is successful. Otherwise, use stat(2) on the unique file to check if its link count has increased to 2, in which case the lock is also suc cessful.

Thanks,

Jim

> Thanks for the info. Here is a section of the man page for
> open.
> Is the information it gives correct wrt using link & stat?

Yes. Attempting to link a file will normally update the cached attributes, so stat() will give correct results.

Cheers,
Trond

Hi Trond,
I rewrote my test program so that it uses open() instead of stat().
I also changed it so that it does not rename the file after it writing it. This should only leave close, open, and unlink calls. The program still fails for me after running for a minute or so:

cayman> ./p1 s
Failed to find #0 which client wrote
Failed on file number 10202

Again, this is with 2.4.20 kernel. It fails much faster with a 2.4.7.

Thanks,

Jim

----------------------------------------------------------------------------
Jim Nance                                                           Synopsys
(919) 425-7219  Do you have sweet iced tea?        [email protected]
                No, but there's sugar on the table.

/* This program demonstrates a problem with the close/open consistency
* of NFS file systems under Linux. It fails very rapidy with Red Hats
* 2.4.7-10smp kernel. This kernel was known to have bugs. It also fails
* with Red Hats 2.4.20-13.7bigmem kernel, which was thought to have this
* bug fixed. For my testcase both linux machines were talking to a
* network applicance file server and mounted like this:
*
* na1-rtp:/vol/vol0/home/jlnance /home/jlnance nfs rw,v3,rsize=4096,\
* wsize=4096,hard,intr,udp,lock,addr=na1-rtp 0 0
*
* This program needs to be run on 2 machines, assume hostnames A & B.
* A and B need to share an NFS mounted file system.
*
* On machine A:
* cd /some/nfs/path/common/to/both
* ./p1 s
*
* On machine B:
* cd /some/nfs/path/common/to/both
* ./p1 c A
*
* After a while you may see output similar to:

 *   cayman> ./p1 s
 *   Failed to find #0 which client wrote
 *   Failed on file number 483

*/

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <netdb.h>

#define PORT 12387
#define FLEN 16

void die()
{
perror("");
exit(-1);
}

void Write(int fd, char *buff, size_t len)
{

  for(;;) {
    int nsent=write(fd, buff, len);
    if(nsent==0)
      exit(0);
    if(nsent==-1) {
      if(errno!=EINTR)
        die();
    } else {
      buff += nsent;
      len  -= nsent;
      if(len==0) {
        return;
      }
    }

}
}

void Read(int fd, char *buff, size_t len)
{

  for(;;) {
    int nread=read(fd, buff, len);
    if(nread==0)
      exit(0);
    if(nread==-1) {
      if(errno!=EINTR)
        die();
    } else {
      buff += nread;
      len  -= nread;
      if(len==0) {
        return;
      }
    }

}
}

int server()
{
int sock = socket(AF_INET, SOCK_STREAM, 0);

  if(sock==-1) die(); else {
    struct sockaddr_in name;
    int                on = 1;
    name.sin_family       = AF_INET;
    name.sin_addr.s_addr  = htonl(INADDR_ANY);
    name.sin_port         = htons(PORT);

    setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, sizeof on);
    if(bind(sock, (struct sockaddr*)&name, sizeof(name))==-1) die(); else {
      if(listen(sock, 1)==-1) die(); else {
        int tsock = accept(sock, 0, 0);
        if(tsock!=-1) {
          int cnt;

          for(cnt=0; cnt<100000; cnt++) {
            int  fd;
            char dummy;
            char number[FLEN];
            struct stat sbuf;
            /*sprintf(number, "#%d", cnt);*/
            sprintf(number, "#%d", 0);
            Write(tsock, number, sizeof(number));
            Read(tsock, &dummy, 1);
#if 0
            if(stat(number, &sbuf)) {
              fprintf(stderr, "Failed to find %s which client wrote\n", number);
              fprintf(stderr, "Failed on file number %d\n", cnt);
              exit(-2);
            }
#else
            fd = open(number, O_RDONLY, 0);
            if(fd<0) {
              fprintf(stderr, "Failed to find %s which client wrote\n", number);
              fprintf(stderr, "Failed on file number %d\n", cnt);
              exit(-2);
            }
#endif
            close(fd);
            unlink(number);
          }
        }
      }
    }

}

return 0;
}

int client(char *server)
{
struct hostent *info = gethostbyname(server);

  if(!info) die(); else {
    int rsocket = socket(AF_INET, SOCK_STREAM, 0);
    if(rsocket==-1) die(); else {
      struct sockaddr_in name;
      name.sin_family = AF_INET;
      name.sin_port   = htons(PORT);
      memcpy(&name.sin_addr, info->h_addr_list[0], sizeof(struct in_addr));
      if(connect(rsocket, (struct sockaddr*)&name, sizeof(name))==-1)
        die();
      else {
        for(;;) {
          int  fd;
          char fname[FLEN];
          char tname[FLEN+8];

          Read(rsocket, fname, sizeof(fname));
          strcpy(tname, fname);
          /*strcat(tname, ".tmp");*/

          fd = open(tname, O_WRONLY|O_CREAT, 0600);
          if(fd==-1) die();

          Write(fd, fname, sizeof(fname)); /* Junk data */
          close(fd);

          /*rename(tname, fname);*/

          Write(rsocket, fname, 1); /* Tells the server we are done */
        }
      }
    }

}

return 0;
}

void usage(char *prog)
{
fprintf(stderr, "Usage:\n");
fprintf(stderr, " %s s\n", prog);
fprintf(stderr, " %s c servername\n", prog);
fprintf(stderr, " Run 1 of each in the same NFS directory on 2 different "

        "machines\n Two processes total\n");

exit(-1);
}

int main(int ac, char **av)
{

  if(ac<2) {
    usage(av[0]);
  } if(av[1][0]=='s') {
    return server();
  }else if(ac<3) {
    usage(av[0]);
  } else if(av[1][0]=='c') {
    return client(av[2]);
  } else {
    usage(av[0]);

}

return -1;
}

Hello,

While its true that this program, p1.c with the rename, does not really check cto cache consistency it still should work (IMHO)..

After some investigation it turns out that the timestamps that is used in handling negative lookups was not being updated
as precisely as it could be... And since this timestamps is also used with other caches (namely the readdir cache), ISTM, that keeping in sync (as much as possible) with the server is a good thing...

Please try the attached patch....

SteveD.

@@ -1096,6 +1096,12 @@
 	NFS_READTIME(inode) = fattr->timestamp;
 
 	/*
+	 * If need be, update the last time the mtime was changed
+	 */
+	if (time_after(fattr->timestamp, NFS_MTIME_UPDATE(inode)))
+		NFS_MTIME_UPDATE(inode) = fattr->timestamp;
+
+	/*
 	 * Note: NFS_CACHE_ISIZE(inode) reflects the state of the cache.
 	 *       NOT inode->i_size!!!
 	 */

@@ -1142,8 +1148,6 @@
 	inode->i_atime = new_atime;
 
 	if (NFS_CACHE_MTIME(inode) != new_mtime) {
-		if (invalid)
-			NFS_MTIME_UPDATE(inode) = fattr->timestamp;
 		NFS_CACHE_MTIME(inode) = new_mtime;
 		inode->i_mtime = nfs_time_to_secs(new_mtime);
 	}

> Hello, While its true that this program, p1.c with the rename, > does not really check cto cache consistency it still should > work (IMHO)..

> After some investigation it turns out that the timestamps that > is used in handling negative lookups was not being updated as > precisely as it could be... And since this timestamps is also > used with other caches (namely the readdir cache), ISTM, that > keeping in sync (as much as possible) with the server is a good > thing...

Steve,
That patch is wrong. I've already told you why...

Cheers,
Trond

-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a relational database is painful, don't do it! Check out ObjectStore. Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Huh?

Advanced server is currently at 2.4.9-e.16 -

perhaps this is a self-compiled kernel?

Joe

jlnance> I have seen the problem on both IA64 machines running the jlnance> kernel 2.4.18 from Red Hats Advanced Server

jjs> Huh?
jjs> Advanced server is currently at 2.4.9-e.16 - jjs> perhaps this is a self-compiled kernel?

Advanced Server on IA64 is at 2.4.18, NOT 2.4.9.

- Roland

Ah -

Thanks for the info, I didn't know about
the version skew between platforms...

Joe

AS for IA64 isnt the same kernel as for IA32