regression, bisected: getcwd() ENOENT on NFS4...
Subsystems (2)
Since 2.6.30-rc, I've been experiencing various issues relating to getcwd() returning ENOENT on NFS4 clients. I used an over-complicated but reliable reproducer [1] (on Karmic RC against a 2.6.32-rc5 NFS4 server) to bisect [2].
The impact of this regression is moderate (side-effects range from benign to failure), so we should get a fix into 2.6.32 if at all possible and strongly consider a 2.6.31 stable update.
Thanks,
Daniel
--- [1]
$ apt-get source apt $ cd apt-* $ ./configure && make [snip] sh: getcwd() failed: No such file or directory
--- [2]
a65318bf3afc93ce49227e849d213799b072c5fd is first bad commit commit a65318bf3afc93ce49227e849d213799b072c5fd
Author: Trond Myklebust <[email protected]> Date: Wed Mar 11 14:10:28 2009 -0400
NFSv4: Simplify some cache consistency post-op GETATTRs
Certain asynchronous operations such as write() do not expect (or care) that other metadata such as the file owner, mode, acls, ... change. All they want to do is update and/or check the change attribute, ctime, and mtime.
By skipping the file owner and group update, we also avoid having to do a potential idmapper upcall for these asynchronous RPC calls.
:040000 040000 0c89c426fff3ad249757e3f77327ea79902df64b6c96fe34f548df08cd11771fbcc839b44a5607e1 M fs
:040000 040000 6ce4d50124b9dd214c54a42bd922a2b9da10aca216f551da881e2d2faeb786e5c47cfe0a21f42ade M include
Signature · 1 line
Daniel J Blueman
Quoted 20 lines · Daniel J Blueman
Since 2.6.30-rc, I've been experiencing various issues relating to
getcwd()returning ENOENT on NFS4 clients. I used an over-complicated but reliable reproducer [1] (on Karmic RC against a 2.6.32-rc5 NFS4 server) to bisect [2].The impact of this regression is moderate (side-effects range from benign to failure), so we should get a fix into 2.6.32 if at all possible and strongly consider a 2.6.31 stable update.
Thanks,
Daniel--- [1]
$ apt-get source apt $ cd apt-* $ ./configure && make [snip] sh: getcwd() failed: No such file or directory
Quoted 8 lines
--- [2]
a65318bf3afc93ce49227e849d213799b072c5fdis first bad commit commita65318bf3afc93ce49227e849d213799b072c5fdAuthor: Trond Myklebust <[email protected]> Date: Wed Mar 11 14:10:28 2009 -0400NFSv4: Simplify some cache consistency post-op GETATTRs
I'm having a lot of trouble seeing how this patch could result in ENOENT. All it should be doing is reducing the frequency with which we update some of the inode metadata.
Have you ever been able to capture one of these errors using strace?
Cheers
Trond
Signature · 6 lines
Trond Myklebust Linux NFS client maintainer NetApp [email protected] www.netapp.com
Hi Trond,
Quoted 3 lines · Trond Myklebust
Quoted 2 lines · Daniel J Blueman
Since 2.6.30-rc, I've been experiencing various issues relating to
getcwd()returning ENOENT on NFS4 clients. I used an over-complicate=
d
Quoted 25 lines
Quoted 25 lines
but reliable reproducer [1] (on Karmic RC against a 2.6.32-rc5 NFS4 server) to bisect [2].
The impact of this regression is moderate (side-effects range from benign to failure), so we should get a fix into 2.6.32 if at all possible and strongly consider a 2.6.31 stable update.
Thanks,
=A0 Daniel--- [1]
$ apt-get source apt $ cd apt-* $ ./configure && make [snip] sh: getcwd() failed: No such file or directory--- [2]
a65318bf3afc93ce49227e849d213799b072c5fdis first bad commit commita65318bf3afc93ce49227e849d213799b072c5fd
Author: Trond Myklebust <[email protected]>
Date: =A0 Wed Mar 11 14:10:28 2009 -0400
Quoted 4 lines
Quoted 1 line
=A0 =A0 NFSv4: Simplify some cache consistency post-op GETATTRs
I'm having a lot of trouble seeing how this patch could result in ENOENT. All it should be doing is reducing the frequency with which w=
e
Quoted 3 lines
update some of the inode metadata.
Have you ever been able to capture one of these errors using strace?
Backing this patch out by hand against stock 2.6.32-rc5 (w/ 2.6.32-rc5 on server) corrects the behaviour. It's readily reproducible [1]; using 2.6.30, the issue is not seen, thus is a regression.
To observe the change to user-level behaviour (after the reproducer com= mands):
# make clean # strace -ffe getcwd make -n >list [pid 3829] getcwd(0x7fffa269a380, 4096) =3D -1 ENOENT (No such file or= directory) make: getcwd: No such file or directory
Would this help for me to log this via a bugzilla.kernel.org ticket?
Thanks,
Daniel
--- [1]
booting eg:
http://mira.sunsite.utk.edu/ubuntu-releases/karmic/ubuntu-9.10-desktop-=
amd64.iso
$ sudo bash # apt-get install build-essential # apt-get build-dep apt # mount server:/ /mnt -tnfs4 && cd /mnt # apt-get source apt # cd apt-0.7.23.1ubuntu2 # ./configure && make -> "getcwd: No such file or directory" messages observed with cited patch and not without --=20 Daniel J Blueman
Quoted 60 lines · Daniel J Blueman
Hi Trond,
Quoted 28 lines · Trond Myklebust
Quoted 27 lines · Daniel J Blueman
Since 2.6.30-rc, I've been experiencing various issues relating to
getcwd()returning ENOENT on NFS4 clients. I used an over-complicated but reliable reproducer [1] (on Karmic RC against a 2.6.32-rc5 NFS4 server) to bisect [2].The impact of this regression is moderate (side-effects range from benign to failure), so we should get a fix into 2.6.32 if at all possible and strongly consider a 2.6.31 stable update.
Thanks,
Daniel--- [1]
$ apt-get source apt $ cd apt-* $ ./configure && make [snip] sh: getcwd() failed: No such file or directory--- [2]
a65318bf3afc93ce49227e849d213799b072c5fdis first bad commit commita65318bf3afc93ce49227e849d213799b072c5fd
Author: Trond Myklebust <[email protected]>
Date: Wed Mar 11 14:10:28 2009 -0400Quoted 7 lines
Quoted 1 line
NFSv4: Simplify some cache consistency post-op GETATTRs
I'm having a lot of trouble seeing how this patch could result in ENOENT. All it should be doing is reducing the frequency with which we update some of the inode metadata.
Have you ever been able to capture one of these errors using strace?
Backing this patch out by hand against stock 2.6.32-rc5 (w/ 2.6.32-rc5 on server) corrects the behaviour. It's readily reproducible [1]; using 2.6.30, the issue is not seen, thus is a regression.
To observe the change to user-level behaviour (after the reproducer commands):
# make clean # strace -ffe getcwd make -n >list [pid 3829] getcwd(0x7fffa269a380, 4096) = -1 ENOENT (No such file or directory) make: getcwd: No such file or directoryWould this help for me to log this via a bugzilla.kernel.org ticket?
Thanks,
Daniel--- [1]
booting eg:
http://mira.sunsite.utk.edu/ubuntu-releases/karmic/ubuntu-9.10-desktop-amd64.iso
Quoted 9 lines
$ sudo bash # apt-get install build-essential # apt-get build-dep apt # mount server:/ /mnt -tnfs4 && cd /mnt # apt-get source apt # cd apt-0.7.23.1ubuntu2 # ./configure && make -> "getcwd: No such file or directory" messages observed with cited patch and not without
For continuity with the mailing list thread, I've created a bug report of this at:
http://bugzilla.kernel.org/show_bug.cgi?id=14541
Signature · 1 line
Daniel J Blueman
Quoted 72 lines · Daniel J Blueman
Quoted 60 lines · Daniel J Blueman
Hi Trond,
Quoted 28 lines · Trond Myklebust
Quoted 27 lines · Daniel J Blueman
Since 2.6.30-rc, I've been experiencing various issues relating to
getcwd()returning ENOENT on NFS4 clients. I used an over-complicated but reliable reproducer [1] (on Karmic RC against a 2.6.32-rc5 NFS4 server) to bisect [2].The impact of this regression is moderate (side-effects range from benign to failure), so we should get a fix into 2.6.32 if at all possible and strongly consider a 2.6.31 stable update.
Thanks,
Daniel--- [1]
$ apt-get source apt $ cd apt-* $ ./configure && make [snip] sh: getcwd() failed: No such file or directory--- [2]
a65318bf3afc93ce49227e849d213799b072c5fdis first bad commit commita65318bf3afc93ce49227e849d213799b072c5fdAuthor: Trond Myklebust <[email protected]> Date: Wed Mar 11 14:10:28 2009 -0400Quoted 7 lines
Quoted 1 line
NFSv4: Simplify some cache consistency post-op GETATTRs
I'm having a lot of trouble seeing how this patch could result in ENOENT. All it should be doing is reducing the frequency with which we update some of the inode metadata.
Have you ever been able to capture one of these errors using strace?
Backing this patch out by hand against stock 2.6.32-rc5 (w/ 2.6.32-rc5 on server) corrects the behaviour. It's readily reproducible [1]; using 2.6.30, the issue is not seen, thus is a regression.
To observe the change to user-level behaviour (after the reproducer commands):
# make clean # strace -ffe getcwd make -n >list [pid 3829] getcwd(0x7fffa269a380, 4096) = -1 ENOENT (No such file or directory) make: getcwd: No such file or directoryWould this help for me to log this via a bugzilla.kernel.org ticket?
Thanks,
Daniel--- [1]
booting eg:
http://mira.sunsite.utk.edu/ubuntu-releases/karmic/ubuntu-9.10-desktop-amd64.isoQuoted 9 lines
$ sudo bash # apt-get install build-essential # apt-get build-dep apt # mount server:/ /mnt -tnfs4 && cd /mnt # apt-get source apt # cd apt-0.7.23.1ubuntu2 # ./configure && make -> "getcwd: No such file or directory" messages observed with cited patch and not without
Quoted 4 lines
For continuity with the mailing list thread, I've created a bug report of this at:
I just committed the following patch into the above bugzilla entry. I hope it suffices to fix the bug.
Cheers
Trond
-------------------------------------------------------------------
NFSv4: Fix a cache validation bug which causes getcwd() to return ENOENT
From: Trond Myklebust <[email protected]>
Changeset a65318bf3afc93ce49227e849d213799b072c5fd (NFSv4: Simplify some cache consistency post-op GETATTRs) incorrectly changed the getattr bitmap for readdir().
This causes the readdir() function to fail to return a fileid/inode number, which again exposed a bug in the NFS readdir code that causes spurious ENOENT errors to appear in applications (see
http://bugzilla.kernel.org/show_bug.cgi?id=14541).
The immediate band aid is to revert the incorrect bitmap change, but more long term, we should change the NFS readdir code to cope with the fact that NFSv4 servers are not required to support fileids/inode numbers.
---
fs/nfs/nfs4proc.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
fs/nfs/nfs4proc.c +1 -1
@@ -2767,7 +2767,7 @@ static int _nfs4_proc_readdir(struct dentry *dentry, struct rpc_cred *cred, .pages = &page, .pgbase = 0, .count = count, - .bitmask = NFS_SERVER(dentry->d_inode)->cache_consistency_bitmask, + .bitmask = NFS_SERVER(dentry->d_inode)->attr_bitmask, };struct nfs4_readdir_res res;struct rpc_message msg = {
Quoted 112 lines · Trond Myklebust
Quoted 72 lines · Daniel J Blueman
Quoted 60 lines · Daniel J Blueman
Hi Trond,
Quoted 28 lines · Trond Myklebust
Quoted 27 lines · Daniel J Blueman
Since 2.6.30-rc, I've been experiencing various issues relating to
getcwd()returning ENOENT on NFS4 clients. I used an over-complicated but reliable reproducer [1] (on Karmic RC against a 2.6.32-rc5 NFS4 server) to bisect [2].The impact of this regression is moderate (side-effects range from benign to failure), so we should get a fix into 2.6.32 if at all possible and strongly consider a 2.6.31 stable update.
Thanks,
Daniel--- [1]
$ apt-get source apt $ cd apt-* $ ./configure && make [snip] sh: getcwd() failed: No such file or directory--- [2]
a65318bf3afc93ce49227e849d213799b072c5fdis first bad commit commita65318bf3afc93ce49227e849d213799b072c5fd
Author: Trond Myklebust <[email protected]>
Date: Wed Mar 11 14:10:28 2009 -0400Quoted 7 lines
Quoted 1 line
NFSv4: Simplify some cache consistency post-op GETATTRs
I'm having a lot of trouble seeing how this patch could result in ENOENT. All it should be doing is reducing the frequency with which we update some of the inode metadata.
Have you ever been able to capture one of these errors using strace?
Backing this patch out by hand against stock 2.6.32-rc5 (w/ 2.6.32-rc5 on server) corrects the behaviour. It's readily reproducible [1]; using 2.6.30, the issue is not seen, thus is a regression.
To observe the change to user-level behaviour (after the reproducer commands):
# make clean # strace -ffe getcwd make -n >list [pid 3829] getcwd(0x7fffa269a380, 4096) = -1 ENOENT (No such file or directory) make: getcwd: No such file or directoryWould this help for me to log this via a bugzilla.kernel.org ticket?
Thanks,
Daniel--- [1]
booting eg:
http://mira.sunsite.utk.edu/ubuntu-releases/karmic/ubuntu-9.10-desktop-amd64.isoQuoted 9 lines
$ sudo bash # apt-get install build-essential # apt-get build-dep apt # mount server:/ /mnt -tnfs4 && cd /mnt # apt-get source apt # cd apt-0.7.23.1ubuntu2 # ./configure && make -> "getcwd: No such file or directory" messages observed with cited patch and not withoutQuoted 4 lines
For continuity with the mailing list thread, I've created a bug report of this at:
I just committed the following patch into the above bugzilla entry. I hope it suffices to fix the bug.
Cheers
Trond
-------------------------------------------------------------------
NFSv4: Fix a cache validation bug which causesgetcwd()to return ENOENT
From: Trond Myklebust <[email protected]>Changeset
a65318bf3afc93ce49227e849d213799b072c5fd(NFSv4: Simplify some cache consistency post-op GETATTRs) incorrectly changed the getattr bitmap forreaddir().
This causes thereaddir()function to fail to return a fileid/inode number, which again exposed a bug in the NFS readdir code that causes spurious ENOENT errors to appear in applications (see
http://bugzilla.kernel.org/show_bug.cgi?id=14541).The immediate band aid is to revert the incorrect bitmap change, but more long term, we should change the NFS readdir code to cope with the fact that NFSv4 servers are not required to support fileids/inode numbers.
Signed-off-by Trond Myklebust <[email protected]>---
fs/nfs/nfs4proc.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)fs/nfs/nfs4proc.c +0
-0
fs/nfs/nfs4proc.c static int _nfs4_proc_readdir(struct dentry *dentry, struct rpc_cred *cred, · line 2773
Quoted 8 lines
fs/nfs/nfs4proc.c +1
-1@@ -2767,7 +2767,7 @@ static int _nfs4_proc_readdir(struct dentry *dentry, struct rpc_cred *cred, .pages = &page, .pgbase = 0, .count = count, - .bitmask = NFS_SERVER(dentry->d_inode)->cache_consistency_bitmask, + .bitmask = NFS_SERVER(dentry->d_inode)->attr_bitmask, }; struct nfs4_readdir_res res; struct rpc_message msg = {
This fixes the behaviour and passes some heavy testing with two good test-cases, with 2.6.32-rc6. As well, this would be good value for the stable stream. I've sync'd the bugzilla report.
Thanks for your work on this, Trond!
Daniel
Signature · 1 line
Daniel J Blueman
Hi Trond,
Quoted 11 lines · Daniel J Blueman
Quoted 9 lines · Trond Myklebust
Quoted 8 lines · Daniel J Blueman
Quoted 6 lines · Daniel J Blueman
Hi Trond,
Quoted 2 lines · Trond Myklebust
Quoted 1 line · Daniel J Blueman
Since 2.6.30-rc, I've been experiencing various issues relating=
to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
getcwd()returning ENOENT on NFS4 clients. I used an over-compl=
icated
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
but reliable reproducer [1] (on Karmic RC against a 2.6.32-rc5 =
NFS4
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
server) to bisect [2].
The impact of this regression is moderate (side-effects range f=
rom
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
benign to failure), so we should get a fix into 2.6.32 if at al=
l
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
possible and strongly consider a 2.6.31 stable update.
Thanks,
=A0 Daniel--- [1]
$ apt-get source apt $ cd apt-* $ ./configure && make [snip] sh: getcwd() failed: No such file or directory--- [2]
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
a65318bf3afc93ce49227e849d213799b072c5fdis first bad commit commita65318bf3afc93ce49227e849d213799b072c5fd
Author: Trond Myklebust <[email protected]>
Date: =A0 Wed Mar 11 14:10:28 2009 -0400=A0 =A0 NFSv4: Simplify some cache consistency post-op GETATTRs
I'm having a lot of trouble seeing how this patch could result i=
n
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
ENOENT. All it should be doing is reducing the frequency with wh=
ich we
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
update some of the inode metadata.
Have you ever been able to capture one of these errors using str=
ace?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Backing this patch out by hand against stock 2.6.32-rc5 (w/ 2.6.3=
2-rc5
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
on server) corrects the behaviour. It's readily reproducible [1]; using 2.6.30, the issue is not seen, thus is a regression.
To observe the change to user-level behaviour (after the reproduc=
er commands):
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
# make clean # strace -ffe getcwd make -n >list [pid =A03829] getcwd(0x7fffa269a380, 4096) =3D -1 ENOENT (No such=
file or directory)
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
make: getcwd: No such file or directory
Would this help for me to log this via a bugzilla.kernel.org tick=
et?
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Thanks,
=A0Daniel--- [1]
booting eg:
http://mira.sunsite.utk.edu/ubuntu-releases/karmic/ubuntu-9.10-de=
sktop-amd64.iso
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
$ sudo bash # apt-get install build-essential # apt-get build-dep apt # mount server:/ /mnt -tnfs4 && cd /mnt # apt-get source apt # cd apt-0.7.23.1ubuntu2 # ./configure && make =A0-> "getcwd: No such file or directory" messages observed with =
cited
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
patch and not without
For continuity with the mailing list thread, I've created a bug rep=
ort
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
of this at:
I just committed the following patch into the above bugzilla entry. =
I
Quoted 6 lines
Quoted 6 lines
hope it suffices to fix the bug.
Cheers
=A0Trond
-------------------------------------------------------------------
NFSv4: Fix a cache validation bug which causesgetcwd()to return EN=
OENT
Quoted 3 lines
Quoted 3 lines
From: Trond Myklebust <[email protected]>
Changeset
a65318bf3afc93ce49227e849d213799b072c5fd(NFSv4: Simplify =
some
Quoted 4 lines
Quoted 4 lines
cache consistency post-op GETATTRs) incorrectly changed the getattr bitmap for
readdir().
This causes thereaddir()function to fail to return a fileid/inode number, which again exposed a bug in the NFS readdir co=
de that
Quoted 4 lines
Quoted 4 lines
causes spurious ENOENT errors to appear in applications (see
http://bugzilla.kernel.org/show_bug.cgi?id=3D14541).The immediate band aid is to revert the incorrect bitmap change, but=
more
Quoted 2 lines
Quoted 2 lines
long term, we should change the NFS readdir code to cope with the fact that NFSv4 servers are not required to support fileids/inode nu=
mbers.
Quoted 8 lines
Quoted 8 lines
Signed-off-by Trond Myklebust <[email protected]>---
=A0fs/nfs/nfs4proc.c | =A0 =A02 +-
=A01 files changed, 1 insertions(+), 1 deletions(-)
Quoted 5 lines
fs/nfs/nfs4proc.c static int _nfs4_proc_readdir(struct dentry *d=
Quoted 5 lines
fs/nfs/nfs4proc.c +0
-0@@ -2767,7 +2767,7 @@ static int _nfs4_proc_readdir(struct dentry *d=
entry, struct rpc_cred *cred,
Quoted 4 lines
fs/nfs/nfs4proc.c static int _nfs4_proc_readdir(struct dentry *d= · line 2770
Quoted 4 lines
fs/nfs/nfs4proc.c +0
-1=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0.pages =3D &page, =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0.pgbase =3D 0, =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0.count =3D count, - =A0 =A0 =A0 =A0 =A0 =A0 =A0 .bitmask =3D NFS_SERVER(dentry->d_inod=
e)->cache_consistency_bitmask,
Quoted 1 line
fs/nfs/nfs4proc.c static int _nfs4_proc_readdir(struct dentry *d= · line 2770
Quoted 1 line
fs/nfs/nfs4proc.c +1
-0+ =A0 =A0 =A0 =A0 =A0 =A0 =A0 .bitmask =3D NFS_SERVER(dentry->d_inod=
e)->attr_bitmask,
Quoted 7 lines
fs/nfs/nfs4proc.c static int _nfs4_proc_readdir(struct dentry *d= · line 2773
Quoted 4 lines
fs/nfs/nfs4proc.c +0
-0=A0 =A0 =A0 =A0}; =A0 =A0 =A0 =A0struct nfs4_readdir_res res; =A0 =A0 =A0 =A0struct rpc_message msg =3D {This fixes the behaviour and passes some heavy testing with two good test-cases, with 2.6.32-rc6. As well, this would be good value for th=
e
Quoted 1 line
stable stream. I've sync'd the bugzilla report.
Is there opportunity to get this regression fix into 2.6.32-rc8, since -rc8 may be the (pen)ultimate before final?
Thanks,
Daniel
--=20
Daniel J Blueman
Quoted 10 lines · Daniel J Blueman
Hi Trond,
Quoted 3 lines · Daniel J Blueman
This fixes the behaviour and passes some heavy testing with two good test-cases, with 2.6.32-rc6. As well, this would be good value for the stable stream. I've sync'd the bugzilla report.
Is there opportunity to get this regression fix into 2.6.32-rc8, since -rc8 may be the (pen)ultimate before final?
It has been committed in the linux-next tree for testing for a few days now. The plan is to send it on to Linus and [email protected] in a couple more days.
Cheers
Trond