-EXDEV between mounts that are same fs
Subsystems (2)
Hi,
I have a somewhat odd use case that I think has some room for improvement.
I have a single fs exported on the server (/export/data). On the client I have mount /export/data/a and /export/data/b. Doing a rename("/export/data/a/foo", "/export/data/b/foo") results in -EXDEV.
Now this is a bit annoying from a program point of view as it is very difficult to detect this beforehand. Doing a stat() on source and destination indicates that they are the same fs. So it would seem POSIX would guarantee I would not get a -EXDEV in that case. I assume this is simply a local, client-side check that determines that two different mount entries are involved.
So, would it be possible to actually make the kernel realise that the source and target are in face the same fs on the server (so a NFS rename will succeed)? Or at least make the kernel present different device st_dev for the two trees.
Rgds
Signature · 17 lines
-- Pierre Ossman Linux kernel, MMC maintainer http://www.kernel.org PulseAudio, core developer http://pulseaudio.org rdesktop, core developer http://www.rdesktop.org ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs _______________________________________________ Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead. http://vger.kernel.org/vger-lists.html#linux-nfs
A message that is not in the archive (sent off-list or not yet received): 20080209200503.6a11e88b-mgABNEgzgxm+PRNnhPf8W5YgPPQkE1Si@public.gmane.org
Quoted 8 lines · Pierre Ossman
Hi,
I have a somewhat odd use case that I think has some room for improvement.
I have a single fs exported on the server (/export/data). On the client I have mount /export/data/a and /export/data/b. Doing a
rename("/export/data/a/foo", "/export/data/b/foo")results in-EXDEV.
I'm afraid the answer is likely "don't do that!"
Quoted 8 lines
Now this is a bit annoying from a program point of view as it is very difficult to detect this beforehand. Doing a
stat()on source and destination indicates that they are the same fs. So it would seem POSIX would guarantee I would not get a-EXDEVin that case. I assume this is simply a local, client-side check that determines that two different mount entries are involved.
Quoted 4 lines
So, would it be possible to actually make the kernel realise that the source and target are in face the same fs on the server (so a NFS rename will succeed)? Or at least make the kernel present different device
st_devfor the two trees.
I believe the same thing happens even with --bind mounts of the same local filesystem, and that it's by design. See this discussion:
http://marc.info/?t=119801816800006&r=1&w=2
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
Quoted 12 lines · On Sat, 9 Feb 2008 14:10:21 -0500 "J. Bruce Fields" <[email protected]>…
Quoted 8 lines · Pierre Ossman
Hi,
I have a somewhat odd use case that I think has some room for improvement.
I have a single fs exported on the server (/export/data). On the client I have mount /export/data/a and /export/data/b. Doing a
rename("/export/data/a/foo", "/export/data/b/foo")results in-EXDEV.I'm afraid the answer is likely "don't do that!"
I don't think I find that an acceptable answer. Giving all systems complete access to your entire server side storage isn't really acceptable in many situations.
Quoted 11 lines
Quoted 5 lines
So, would it be possible to actually make the kernel realise that the source and target are in face the same fs on the server (so a NFS rename will succeed)? Or at least make the kernel present different device
st_devfor the two trees.I believe the same thing happens even with --bind mounts of the same local filesystem, and that it's by design. See this discussion:
http://marc.info/?t=119801816800006&r=1&w=2
They make some sound arguments for it, so I guess the optimisation is out of the question. I also couldn't find anything in POSIX linking rename() and st_dev, so it should be fine from a strict interpretation of the spec. Still, it is a bit unexpected to get -EXDEV when st_dev is the same for both points.
I though the NFS client allocates "fake" st_dev numbers to guarantee the st_dev/st_ino uniqueness, so how come each mount point doesn't get a new one?
Rgds
Signature · 17 lines
-- Pierre Ossman Linux kernel, MMC maintainer http://www.kernel.org PulseAudio, core developer http://pulseaudio.org rdesktop, core developer http://www.rdesktop.org ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs _______________________________________________ Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead. http://vger.kernel.org/vger-lists.html#linux-nfs
A message that is not in the archive (sent off-list or not yet received): 20080209213511.59ff4e2c-mgABNEgzgxm+PRNnhPf8W5YgPPQkE1Si@public.gmane.org
Quoted 9 lines · Pierre Ossman
Quoted 9 lines
Quoted 5 lines
So, would it be possible to actually make the kernel realise that the source and target are in face the same fs on the server (so a NFS rename will succeed)? Or at least make the kernel present different device
st_devfor the two trees.I believe the same thing happens even with --bind mounts of the same local filesystem, and that it's by design. See this discussion:
Quoted 6 lines
Quoted 2 lines
http://marc.info/?t=119801816800006&r=1&w=2They make some sound arguments for it, so I guess the optimisation is out of the question. I also couldn't find anything in POSIX linking
rename()andst_dev, so it should be fine from a strict interpretation of the spec. Still, it is a bit unexpected to get-EXDEVwhenst_devis the same for both points.I though the NFS client allocates "fake"
st_devnumbers to guarantee the st_dev/st_ino uniqueness, so how come each mount point doesn't get a new one?
It has the same syntax and properties as a bind mount. If you don't like it, then fix the VFS.
Trond
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
A message that is not in the archive (sent off-list or not yet received): 1202589611.11035.0.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org
Quoted 4 lines · On Sat, 09 Feb 2008 15:40:11 -0500 Trond Myklebust…
It has the same syntax and properties as a bind mount. If you don't like it, then fix the VFS.
So it detects that it is the same server and server side fs and reuses the super block?
Quoted 4 lines
Those semantics are _exactly_ the same as if you were to mount --bind the same filesystem to two different mountpoints. This is not an NFS bug, but rather a deliberate design within the VFS in order to avoid mutex locking deadlocks during rename.
Do you have a reference for that? The previous thread linked in suggested it was a security feature.
Rgds
Signature · 17 lines
-- Pierre Ossman Linux kernel, MMC maintainer http://www.kernel.org PulseAudio, core developer http://pulseaudio.org rdesktop, core developer http://www.rdesktop.org ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs _______________________________________________ Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead. http://vger.kernel.org/vger-lists.html#linux-nfs
Quoted 9 lines · Pierre Ossman
Quoted 6 lines · On Sat, 9 Feb 2008 14:10:21 -0500 "J. Bruce Fields" <[email protected]>…
Quoted 5 lines · Pierre Ossman
Hi,
I have a somewhat odd use case that I think has some room for improvement.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
I have a single fs exported on the server (/export/data). On the client I have mount /export/data/a and /export/data/b. Doing a
rename("/export/data/a/foo", "/export/data/b/foo")results in-EXDEV.I'm afraid the answer is likely "don't do that!"
I don't think I find that an acceptable answer. Giving all systems complete access to your entire server side storage isn't really acceptable in many situations.
I'd be curious to know what you're actually trying to accomplish. If you're mounting subdirectories of the same server filesystem, then you need to be aware that, for example, anyone with access to the network can probably access that whole filesystem by guessing filehandles (which is very easy).
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
Quoted 9 lines · On Sat, 9 Feb 2008 18:14:35 -0500 "J. Bruce Fields" <[email protected]>…
Quoted 2 lines
I don't think I find that an acceptable answer. Giving all systems complete access to your entire server side storage isn't really acceptable in many situations.
I'd be curious to know what you're actually trying to accomplish. If you're mounting subdirectories of the same server filesystem, then you need to be aware that, for example, anyone with access to the network can probably access that whole filesystem by guessing filehandles (which is very easy).
I thought the subtree check prevented that?
The point of the setup is of course to avoid exposing more information than is necessary, while still having the benefit of having a single pool of disk space for all exported resources.
Rgds
Signature · 17 lines
-- Pierre Ossman Linux kernel, MMC maintainer http://www.kernel.org PulseAudio, core developer http://pulseaudio.org rdesktop, core developer http://www.rdesktop.org ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs _______________________________________________ Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead. http://vger.kernel.org/vger-lists.html#linux-nfs
A message that is not in the archive (sent off-list or not yet received): 20080210131224.1d4ae39f-mgABNEgzgxm+PRNnhPf8W5YgPPQkE1Si@public.gmane.org
Quoted 6 lines · Pierre Ossman
Quoted 3 lines · On Sat, 9 Feb 2008 18:14:35 -0500 "J. Bruce Fields" <[email protected]>…
Quoted 2 lines
I don't think I find that an acceptable answer. Giving all systems complete access to your entire server side storage isn't really acceptable in many situations.
Quoted 8 lines
Quoted 6 lines
I'd be curious to know what you're actually trying to accomplish. If you're mounting subdirectories of the same server filesystem, then you need to be aware that, for example, anyone with access to the network can probably access that whole filesystem by guessing filehandles (which is very easy).
I thought the subtree check prevented that?
Yes, at the expense of odd behavior in a couple corner cases: the parent directory is encoded in the filehandle, so, for example, a client holding a file open will get ESTALE if it moves to a different directory.
Quoted 3 lines
The point of the setup is of course to avoid exposing more information than is necessary, while still having the benefit of having a single pool of disk space for all exported resources.
I notice that btrfs claims support for "subvolumes"--multiple filesystem namespaces sharing the same storage. Seems like a useful feature for this situation. I don't know whether it'd be possible to add to other filesystems.
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
Quoted 8 lines · J. Bruce Fields
Quoted 3 lines
The point of the setup is of course to avoid exposing more information than is necessary, while still having the benefit of having a single pool of disk space for all exported resources.
I notice that btrfs claims support for "subvolumes"--multiple filesystem namespaces sharing the same storage. Seems like a useful feature for this situation. I don't know whether it'd be possible to add to other filesystems.
That stuff is done using a code sniplet from me, which does nothing but allowing to mount into a subdirectory of a filesystem. It will look exactly the same as bind mounts. I have hacked up implementations for ext2 and xfs aswell and in the form btfs does it currently it would be doable for any filesystem. But to actually make it useful you'd need quotas for each subvolume which is something currently only xfs could provide (if I could be arsed into finally implementing my multiple subvolumes proposal fully).
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
Quoted 5 lines · Christoph Hellwig
Quoted 4 lines · J. Bruce Fields
Quoted 3 lines
The point of the setup is of course to avoid exposing more information than is necessary, while still having the benefit of having a single pool of disk space for all exported resources.
Quoted 8 lines
Quoted 4 lines
I notice that btrfs claims support for "subvolumes"--multiple filesystem namespaces sharing the same storage. Seems like a useful feature for this situation. I don't know whether it'd be possible to add to other filesystems.
That stuff is done using a code sniplet from me, which does nothing but allowing to mount into a subdirectory of a filesystem. It will look exactly the same as bind mounts.
To get the equivalent of subtree checking, we also need a reference from each inode back to the subvolume it belongs to, so we can determine the subvolume from a filehandle. I assume that's a more intrusive change.
--b.
Quoted 7 lines
I have hacked up implementations for
ext2 and xfs aswell and in the form btfs does it currently it would be doable for any filesystem. But to actually make it useful you'd need quotas for each subvolume which is something currently only xfs could provide (if I could be arsed into finally implementing my multiple subvolumes proposal fully).
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
Quoted 3 lines · J. Bruce Fields
To get the equivalent of subtree checking, we also need a reference from each inode back to the subvolume it belongs to, so we can determine the subvolume from a filehandle. I assume that's a more intrusive change.
XFS actually has the infrastructure for that by having a project id that's inheritable to child directories/files. But to actuall make that useable we'd have to encode it in the fsid. I must admit I haven't thought that far yet.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
Quoted 9 lines · Christoph Hellwig
Quoted 3 lines · J. Bruce Fields
To get the equivalent of subtree checking, we also need a reference from each inode back to the subvolume it belongs to, so we can determine the subvolume from a filehandle. I assume that's a more intrusive change.
XFS actually has the infrastructure for that by having a project id that's inheritable to child directories/files. But to actuall make that useable we'd have to encode it in the fsid. I must admit I haven't thought that far yet.
It'd be useful for nfsd. So if you have any design notes or anything I'd be interested.
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
Quoted 7 lines · J. Bruce Fields
Quoted 4 lines
XFS actually has the infrastructure for that by having a project id that's inheritable to child directories/files. But to actuall make that useable we'd have to encode it in the fsid. I must admit I haven't thought that far yet.
It'd be useful for nfsd. So if you have any design notes or anything I'd be interested.
http://verein.lst.de/~hch/xfs/xfs-multiple-containers.txt
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
Quoted 10 lines · Christoph Hellwig
Quoted 7 lines · J. Bruce Fields
Quoted 4 lines
XFS actually has the infrastructure for that by having a project id that's inheritable to child directories/files. But to actuall make that useable we'd have to encode it in the fsid. I must admit I haven't thought that far yet.
It'd be useful for nfsd. So if you have any design notes or anything I'd be interested.
Excellent, thanks, yes that's just what we'd want....
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
Quoted 3 lines · J. Bruce Fields
Excellent, thanks, yes that's just what we'd want....
The proposal still lacks the idea of encoding the projid in the file handle. But I think we could do that quite easily, but I'd do it after my planned next round of the export interface cleanups, so that a filehandle can see at mount time how much of the filehandle space it has left and refuse the export if it doesn't fit (e.g. exporting with various 64bit inode fields and NFSv2, or projids on NFSv2)
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
Quoted 11 lines · Pierre Ossman
Hi,
I have a somewhat odd use case that I think has some room for improvement.
I have a single fs exported on the server (/export/data). On the client I have mount /export/data/a and /export/data/b. Doing a
rename("/export/data/a/foo", "/export/data/b/foo")results in-EXDEV.Now this is a bit annoying from a program point of view as it is very difficult to detect this beforehand. Doing a
stat()on source and destination indicates that they are the same fs. So it would seem POSIX would guarantee I would not get a-EXDEVin that case. I assume this is simply a local, client-side check that determines that two different mount entries are involved.So, would it be possible to actually make the kernel realise that the source and target are in face the same fs on the server (so a NFS rename will succeed)? Or at least make the kernel present different device
st_devfor the two trees.Rgds
Those semantics are _exactly_ the same as if you were to mount --bind the same filesystem to two different mountpoints. This is not an NFS bug, but rather a deliberate design within the VFS in order to avoid mutex locking deadlocks during rename.
Cheers
Trond
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs