Btrfs, NFS (v3) and ESTALE
Dear all;
On a cluster of ~35 machines used for batch processing, which all mount via NFS (v3) a BTRFS export, I am experiencing issues that are causing NFS clients to occasionally produce Stale NFS handle errors on accessing this file system. I would be interested to know if this is possibly related to use of BTRFS, or is mere coincidence.
Background:
- The NFS server is running 2.6.33, with a btrfs file system created under the same kernel.
- The file system is mounted as:
/dev/md2 /work btrfs rw,noatime,nodiratime 0 0
- The file system is exported as:
/work <world>(rw,wdelay,root_squash,no_subtree_check)
- Clients are mostly 2.6.35, however, problems have also been
seen with 2.6.32.
- Clients mount (from /proc/mounts)
vc-fs0:/work /work nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.29.146.16,mountvers=3,mountport=51102,mountproto=udp,addr=172.29.146.16 0 0
The problem manifests itself when issuing a job to the cluster, of ~120 tasks on 30 nodes. We will occasionally find that a machine reports NFS stale filehandle errors when trying to stat a directory. The directory will not have been deleted during the lifetime of the job, however some (eg 30) sub-directories will have been created.
The erros are Usually seen from a machine that has not done any work.
For example:
(2.6.35:)
vcfe0:~$ ls -l /work >/dev/null --launch job (doesn't do anything on vcfe, uses different nodes)-- ... time passes (unknown how long) ...
vcfe0:~$ ls -l /work >/dev/null ls: cannot access /work/marta-cip-test: Stale NFS file handle ls: cannot access /work/andrea-test-ais: Stale NFS file handle
(2.6.35:)
vc-r210-0:~$ ls -l /work >/dev/null vc-r210-0:~$
(2.6.32:)
b36048:~$ ls -l /work/ >/dev/null ls: cannot access /work/marta-cip-test: Stale NFS file handle ls: cannot access /work/andrea-test-ais: Stale NFS file handle
Two separate machines are seeing the same stale file handles. b36048 hadn't even touched /work for some considerable time before doing that ls.
performing `touch /work/andrea-test-ais' on the client will allow the client machine to stat the directory again, however, doing it on the file server does not.
performing `echo 2 > /proc/sys/vm/drop_caches' on the client will sometimes solve the problem for that client [but not always].
I've not yet found a reliable way to reproduce the problem, other than running large jobs (we aren't running small ones at the moment, so can't say if it is related to size)
I would be interested to know if anyone believes this may be related to the use of btrfs, (or even a configuration / nfs cache coherency problem).
Some extra anecdotal evidence:
I don't recall this being an issue before we upgraded all the compute nodes to 2.6.35. Previously they used 2.6.33, but an upgrade was forced due to an nfs bug under high write loads. However, it may be that the nature of the jobs that we are running now has changed slightly too.
Kind regards,
..david
Hi David,
Quoted 3 lines · David Flynn
Dear all;
On a cluster of ~35 machines used for batch processing, which all mou=
nt
Quoted 1 line
via NFS (v3) a BTRFS export, I am experiencing issues that are causin=
g
Quoted 1 line
NFS clients to occasionally produce Stale NFS handle errors on access=
ing
Quoted 1 line
this file system. =A0I would be interested to know if this is possibl=
y
Quoted 4 lines
related to use of BTRFS, or is mere coincidence.
Background:
=A0- The NFS server is running 2.6.33, with a btrfs file system creat=
ed
Quoted 7 lines
=A0 =A0under the same kernel.
=A0- The file system is mounted as:
=A0 =A0/dev/md2 /work btrfs rw,noatime,nodiratime 0 0=A0- The file system is exported as:
=A0 =A0/work =A0 =A0 =A0 =A0 =A0 <world>(rw,wdelay,root_squash,no_sub=
tree_check)
Quoted 6 lines
=A0- Clients are mostly 2.6.35, however, problems have also been =A0 =A0seen with 2.6.32.
=A0- Clients mount (from
/proc/mounts)
=A0 =A0vc-fs0:/work /work nfs rw,relatime,vers=3D3,rsize=3D1048576,ws=
ize=3D1048576,namlen=3D255,hard,proto=3Dtcp,timeo=3D600,retrans=3D2,sec= =3Dsys,mountaddr=3D172.29.146.16,mountvers=3D3,mountport=3D51102,mountp= roto=3Dudp,addr=3D172.29.146.16 0 0
Quoted 2 lines
The problem manifests itself when issuing a job to the cluster, of ~1=
20
Quoted 1 line
tasks on 30 nodes. =A0We will occasionally find that a machine report=
s
Quoted 21 lines
NFS stale filehandle errors when trying to stat a directory. =A0The directory will not have been deleted during the lifetime of the job, however some (eg 30) sub-directories will have been created.
The erros are Usually seen from a machine that has not done any work.
For example:
(2.6.35:)
vcfe0:~$ ls -l /work >/dev/null --launch job (doesn't do anything on vcfe, uses different nodes)-- ... time passes (unknown how long) ...vcfe0:~$ ls -l /work >/dev/null ls: cannot access /work/marta-cip-test: Stale NFS file handle ls: cannot access /work/andrea-test-ais: Stale NFS file handle(2.6.35:)
vc-r210-0:~$ ls -l /work >/dev/null vc-r210-0:~$
Quoted 6 lines
(2.6.32:)
b36048:~$ ls -l /work/ >/dev/null ls: cannot access /work/marta-cip-test: Stale NFS file handle ls: cannot access /work/andrea-test-ais: Stale NFS file handleTwo separate machines are seeing the same stale file handles. =A0b360=
48
Quoted 1 line
hadn't even touched /work for some considerable time before doing tha=
t
Quoted 10 lines
ls.
performing `touch /work/andrea-test-ais' on the client will allow the client machine to stat the directory again, however, doing it on the file server does not.
performing `echo 2 >
/proc/sys/vm/drop_caches' on the client will sometimes solve the problem for that client [but not always].I've not yet found a reliable way to reproduce the problem, other tha=
n
Quoted 1 line
running large jobs (we aren't running small ones at the moment, so ca=
n't
Quoted 3 lines
say if it is related to size)
I would be interested to know if anyone believes this may be related =
to
Quoted 1 line
the use of btrfs, (or even a configuration / nfs cache coherency prob=
lem).
Quoted 3 lines
Some extra anecdotal evidence:
=A0I don't recall this being an issue before we upgraded all the comp=
ute
Quoted 1 line
=A0nodes to 2.6.35. =A0Previously they used 2.6.33, but an upgrade wa=
s
Quoted 1 line
=A0forced due to an nfs bug under high write loads. =A0However, it ma=
y be
Quoted 2 lines
=A0that the nature of the jobs that we are running now has changed =A0slightly too.
I was experiencing a similar pattern of ESTALE issues with NFS with 2.6.33 (IIRC) and cached data on ext4, and could reproduce it from time to time performing kernel rebuilds over NFS.
I've CC'd Trond on the full email to see if it rings a bell. The best outcome may be if we write a micro-reproducer which exploits this race using cached data.
Thanks,
Daniel
--=20
Daniel J Blueman
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
I wonder how difficult it would be to use GFS storage allocation, so the right way to cluster btrfs would be at the block device level instead of file system level.
Like the way btrfs can coexist in ext4 partition, but coexisting with GFS.
Quoted 4 lines · David Flynn
Dear all;
On a cluster of ~35 machines used for batch processing, which all mount via NFS (v3) a BTRFS export, I am experiencing
Quoted 7 lines · * Daniel J Blueman ([email protected]) wrote:
I was experiencing a similar pattern of ESTALE issues with NFS with 2.6.33 (IIRC) and cached data on ext4, and could reproduce it from time to time performing kernel rebuilds over NFS.
I've CC'd Trond on the full email to see if it rings a bell. The best outcome may be if we write a micro-reproducer which exploits this race using cached data.
I've recently seen quite a concrete case, which may be interesting:
NB, this is not an exact transcript
## step1: build a binary to use (out of tree build, touches: depends
## files, object files, binaries)
vcfe:some/dir/bin$ make
## step2: launch a job on the cluster that uses the binaries in dir/bin
## but does not touch any other files in dir/bin
vcfe:some/dir$ sbatch -N4 my_job.sh
## step3: let time pass (job completed, came back next day) ##
vcfe:some/dir$ ls -l bin
< many stale filehandle errors >
In actual fact, steps 1 and 2 were repeated several times (happened to be bisecting something) with out issue, then the following day step 3 revealed a problem.
Now all writes to dir/bin occurred on vcfe, other computers only accessed it for the binary. Other computers will have created extra directories in "some/dir/".
stale filehandle errors were resolved by:
echo 2 > /proc/sys/vm/drop_caches
A quick summary of the setup:
- nfs client was 2.6.35, mounting with nfsv3
- nfs server was 2.6.33, exporting a btrfs filesystem (noatime,nodiratime)
I'd be very interested if anyone has any further thoughts on the issue.
Kind regards,
..david