EACCES race when opening just-created file

With the Linux kernel as both NFS client and server, I'm seeing a race condition where open() can return EACCES for a just-created file. Is this expected behavior? POSIX implicitly allows it, but I failed to locate any discussion of it. I have attached a test program; here, it witnesses the race for roughly a dozen of the 4096 files it creates (different files each time). One router separates client from server, with overall ping times as follows:

100 packets transmitted, 100 received, 0% packet loss, time 99039ms rtt min/avg/max/mdev = 0.173/0.259/4.447/0.422 ms

This server has three similar clients, each of which witnesses the race. I did not reproduce this in a different NFS setup where client and server were virtual machines on the same box. Those are the only two environments I have tested so far.

-- Server Details
[nm@gcc45 0:0 00:36:42 ~]$ uname -a
Linux gcc45 3.16.0-4-686-pae #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04) i686 GNU/Linux [nm@gcc45 0:0 00:36:37 ~]$ cat /proc/fs/nfs/exports
# Version 1.1
# Path Client(Flags) # IPs

/home   gcc24.tetaneutral.net(rw,no_root_squash,async,wdelay,no_subtree_check,uuid=6797c43c:b0f64b64:bfca4840:f15789eb,sec=1)
/home   gcc23.tetaneutral.net(rw,no_root_squash,async,wdelay,no_subtree_check,uuid=6797c43c:b0f64b64:bfca4840:f15789eb,sec=1)
/home   gcc22.tetaneutral.net(rw,no_root_squash,async,wdelay,no_subtree_check,uuid=6797c43c:b0f64b64:bfca4840:f15789eb,sec=1)

-- Client Details
[nm@erpro8-fsf3 0:1 00:37:28 nfs]$ uname -a
Linux erpro8-fsf3 4.1.4 #1 SMP PREEMPT Mon Aug 3 14:22:54 PDT 2015 mips64 GNU/Linux
[nm@erpro8-fsf3 0:1 00:37:49 nfs]$ mount|grep nfs
gcc45.tetaneutral.net:/home on /home type nfs4 (rw,relatime,vers=4.0,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp6,port=0,timeo=600,retrans=2,sec=sys,clientaddr=2a03:7220:8083:9d00::1,local_lock=none,addr=2a03:7220:8081:8100::1)

Thanks,
nm

/*
* Demonstrates open() returning EACCES for a recently-created file on NFS. One
* thread creates files nfs0 .. nfs4095 in the CWD, and another thread
* continually attempts to open(O_RDWR) the latest file. Prints a message
* whenever the second thread sees open() fail in a way other than ENOENT.
* Affected NFS configurations will print such messages for a selection of the
* files, and immune configurations will print nothing.
*
* I abstracted this test case from a malfunction in the PostgreSQL database,
* where one process opened a file just created in another process. (See
* src/interfaces/ecpg cases thread/thread and thread/prep.)
*/

#include <errno.h>
#include <fcntl.h>
#include <limits.h>
#include <pthread.h>
#include <stdio.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

/* no reader/writer synchronization; occasional disagreement is okay */ static int pos;

static void *
creator_task(void *arg)

{
    for (pos = 0; pos < (1 << 12); ++pos)
    {
	char buf[128];
	int fd;

	sprintf(buf, "nfs%d", pos);
	fd = open(buf, O_RDWR | O_CREAT | O_EXCL, 0600);
	if (fd < 0)
	    printf("could not create %s: %s\n", buf, strerror(errno));
	else if (close(fd) != 0)
	    printf("could not close %s after creation: %s\n",
		   buf, strerror(errno));
    }

    pos = -1;

    return NULL;
}

static void *
tester_task(void *arg)

{
    while (pos >= 0)
    {
	char buf[128];
	int fd;

	sprintf(buf, "nfs%d", pos);
	fd = open(buf, O_RDWR);
	if (fd < 0)
	{
	    if (errno != ENOENT)
		printf("could not open %s: %s\n", buf, strerror(errno));
	}
	else if (close(fd) != 0)
	    printf("could not close %s: %s\n", buf, strerror(errno));
    }

    return NULL;
}

int
main(int argc, char **argv)

{
    pthread_t creator;

    pthread_create(&creator, NULL, creator_task, 0);
    tester_task(NULL);
    pthread_join(creator, NULL);

    return 0;
}

This is due to a limitation of NFSv3 and NFSv4.0. When you do an exclusive create, the atime and mtime get set to a particular set of values (the verifier), and the client is expected to override those values with a follow-on SETATTR call once it gets those values back. This is to prevent problems during certain server reboot scenarios.

The NFS server also sets the mode to 0000 during this time (since the client can't even set the mode during the operation).

NFSv4.1 is less subject to this problem. You may want to try using that and see if it's any better.

Jeff Layton <[email protected]>

Interesting; that fully explains it. Thanks.

I will look into that.

Mounting with vers=4.1 eliminated the unwanted behavior. Thanks for the tip.