client side see wrong directory entries with nfs_export on overlayfs
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): CAOQ4uxgtD1F_S_tGoAKo7egqbXRGa-sM6SL-Ab7K62LN0FYBSA@mail.gmail.com
Quoted 9 lines
2018-03-27 13:24 GMT+03:00 Dai Qizhi <[email protected]>:
Quoted 2 lines
Dear Amir Goldstein,
Hi Dai,
Do you mind if I re-posting your question to overlayfs mailing list. There are other people out there that would encounter the same problem and will benefit from my answer.
no problem.
Quoted 45 lines
Quoted 38 lines
I tested overlayfs with
nfs_exportfeature , the kernel is built by ubuntu , 4.16.0-041600rc7, http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16-rc7/the problem is , when i export two different overlay mount points on server side, then mount them use nfs on the client side, the client side views the two differnet mount points as the same.
on server side:
none on
/mnt/m1type overlay (rw,relatime,lowerdir=m1/ro,upperdir=m1/rw,workdir= m1/w,index=on,nfs_export=on)
none on/mnt/m2type overlay (rw,relatime,lowerdir=m2/ro,upperdir=m2/rw,workdir= m2/w,index=on,nfs_export=on)
[root@localhost data]# exportfs -rv
exporting *:/mnt/m4
exporting *:/mnt/m3
exporting *:/mnt/m2
exporting *:/mnt/m1
[root@localhost data]# ls/mnt/m1this_is_m1
[root@localhost data]# ls/mnt/m2this_is_m2on client side:
[root@localhost mnt]# mount 192.168.0.1:/mnt/m1 m1
[root@localhost mnt]# mount 192.168.0.1:/mnt/m2 m2
[root@localhost mnt]# mount |grep 192
192.168.0.1:/mnt/m1 on/mnt/m1type nfs (rw,relatime,vers=3,rsize=524288,wsize=5 24288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0. 1,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=192.168.0.1) 192.168.0.1:/mnt/m1 on/mnt/m2type nfs (rw,relatime,vers=3,rsize=524288,wsize=5 24288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0. 1,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=192.168.0.1)Quoted 5 lines
[root@localhost mnt]# ls
/mnt/m1this_is_m1
[root@localhost mnt]# ls/mnt/m2this_is_m1
[root@localhost mnt]#
Quoted 3 lines
Please refer to the man page of exports(5), the section about 'fsid' describes this problem related to exporting file systems that are not on a block device, such as overlayfs.
yes, export different mount point with different fsid= option works as expected.
Quoted 7 lines
If you are interested to know if there is a way to fix this that does not involve manually
configuring different fsid per export, I will have to consult with the NFS experts, so please
reply to this message with CC to <[email protected]> and <[email protected]>
when exporting parent directory with crossmnt option and mount differnet overlayfs under that directory, we encounter the same problem on client side..
Quoted 3 lines
Thanks,
Amir.
Best regards,
Dai Qizhi
On Wed, Mar 28, 2018 at 4:14 AM, Dai Qizhi <[email protected]> wrote:
[...]
Quoted 61 lines
Quoted 45 lines
Quoted 38 lines
I tested overlayfs with
nfs_exportfeature , the kernel is built by ubuntu , 4.16.0-041600rc7, http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16-rc7/the problem is , when i export two different overlay mount points on server side, then mount them use nfs on the client side, the client side views the two differnet mount points as the same.
on server side:
none on
/mnt/m1type overlay (rw,relatime,lowerdir=m1/ro,upperdir=m1/rw,workdir= m1/w,index=on,nfs_export=on)
none on/mnt/m2type overlay (rw,relatime,lowerdir=m2/ro,upperdir=m2/rw,workdir= m2/w,index=on,nfs_export=on)
[root@localhost data]# exportfs -rv
exporting *:/mnt/m4
exporting *:/mnt/m3
exporting *:/mnt/m2
exporting *:/mnt/m1
[root@localhost data]# ls/mnt/m1this_is_m1
[root@localhost data]# ls/mnt/m2this_is_m2on client side:
[root@localhost mnt]# mount 192.168.0.1:/mnt/m1 m1
[root@localhost mnt]# mount 192.168.0.1:/mnt/m2 m2
[root@localhost mnt]# mount |grep 192
192.168.0.1:/mnt/m1 on/mnt/m1type nfs (rw,relatime,vers=3,rsize=524288,wsize=5 24288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0. 1,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=192.168.0.1) 192.168.0.1:/mnt/m1 on/mnt/m2type nfs (rw,relatime,vers=3,rsize=524288,wsize=5 24288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0. 1,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=192.168.0.1)Quoted 5 lines
[root@localhost mnt]# ls
/mnt/m1this_is_m1
[root@localhost mnt]# ls/mnt/m2this_is_m1
[root@localhost mnt]#Quoted 3 lines
Please refer to the man page of exports(5), the section about 'fsid' describes this problem related to exporting file systems that are not on a block device, such as overlayfs.
yes, export different mount point with different fsid= option works as expected.
Quoted 7 lines
If you are interested to know if there is a way to fix this that does not involve manually
configuring different fsid per export, I will have to consult with the NFS experts, so please
reply to this message with CC to <[email protected]> and <[email protected]>
Quoted 3 lines
when exporting parent directory with crossmnt option and mount differnet overlayfs under that directory, we encounter the same problem on client side..
I see. Jeff, Bruce, is there a school book solution to this issue?
Is there a way for a non blockdev export to automatically identify itself to knfsd? After all, the tuple ("overlayfs";<overlay root file handle>) should be unique on the server. <overlay root file handle> contains (struct ovl_fh) the upper fs UUID and the upper root dir file handle.
Technically, if there is no out of tree fs on the system that is using the value OVL_FILEID (0xfb) for file handle type <overlay root file handle> itself would be unique.
Thanks,
Amir.
Jeff Layton
maintainerQuoted 98 lines · Amir Goldstein
On Wed, Mar 28, 2018 at 4:14 AM, Dai Qizhi <[email protected]> wrote:
[...]Quoted 79 lines
Quoted 57 lines
Quoted 50 lines
I tested overlayfs with
nfs_exportfeature , the kernel is built by
ubuntu , 4.16.0-041600rc7, http://kernel.ubuntu.com/~kernel-ppa
/mainline/v4.16-rc7/the problem is , when i export two different overlay mount points on
server side, then mount them use nfs on the client side, the client side
views the two differnet mount points as the same.on server side:
none on
/mnt/m1type overlay (rw,relatime,lowerdir=m1/ro,upperdir=m1/rw,workdir= m1/w,index=on,nfs_export=on)
none on/mnt/m2type overlay (rw,relatime,lowerdir=m2/ro,upperdir=m2/rw,workdir= m2/w,index=on,nfs_export=on) [root@localhost data]# exportfs -rv exporting *:/mnt/m4
exporting *:/mnt/m3
exporting *:/mnt/m2
exporting *:/mnt/m1
[root@localhost data]# ls/mnt/m1this_is_m1
[root@localhost data]# ls/mnt/m2this_is_m2on client side:
[root@localhost mnt]# mount 192.168.0.1:/mnt/m1 m1 [root@localhost mnt]# mount 192.168.0.1:/mnt/m2 m2 [root@localhost mnt]# mount |grep 192 192.168.0.1:/mnt/m1 on
/mnt/m1type nfs (rw,relatime,vers=3,rsize=524288,wsize=5 24288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mou ntaddr=192.168.0. 1,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,ad dr=192.168.0.1)
192.168.0.1:/mnt/m1 on/mnt/m2type nfs (rw,relatime,vers=3,rsize=524288,wsize=5 24288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mou ntaddr=192.168.0. 1,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,ad dr=192.168.0.1)Quoted 5 lines
[root@localhost mnt]# ls
/mnt/m1this_is_m1
[root@localhost mnt]# ls/mnt/m2this_is_m1
[root@localhost mnt]#Quoted 5 lines
Please refer to the man page of exports(5), the section about 'fsid' describes
this problem related to exporting file systems that are not on a block device,
such as overlayfs.yes, export different mount point with different fsid= option works as
expected.Quoted 10 lines
If you are interested to know if there is a way to fix this that does
not involve manually
configuring different fsid per export, I will have to consult with the
NFS experts, so please
reply to this message with CC to <[email protected]> and
<[email protected]>Quoted 5 lines
when exporting parent directory with crossmnt option and mount differnet overlayfs
under that directory, we encounter the same problem on client side..I see. Jeff, Bruce, is there a school book solution to this issue?
Is there a way for a non blockdev export to automatically identify itself
to knfsd? After all, the tuple ("overlayfs";<overlay root file handle>) should
be unique on the server. <overlay root file handle> contains (structovl_fh)
the upper fs UUID and the upper root dir file handle.
Quoted 8 lines
Technically, if there is no out of tree fs on the system that is using the
valueOVL_FILEID(0xfb) for file handle type <overlay root file handle>
itself would be unique.Thanks,
Amir.
tl;dr: not currently, which is why when I did the reexport patches a few years ago, they _required_ that you manually set the fsid= export option.
Longer story:
Long, long ago, the fsid for the export was almost always determined by the device major/minor tuple. That became really problematic whenever devices got reordered after adding a disk to the system and rebooting. So, Neil Brown added the ability to determine the fsid from the libblkdev uuid (see nfs-utils commit e91ff0175602c, and kernel commits from around that time).
In principle, you could do something similar for overlayfs: add a new FSID_* type for overlayfs that can reliably determine a unique fsid for different overlays. That would require kernel and userland patches, of course...
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 121 lines · Jeff Layton
Quoted 98 lines · Amir Goldstein
On Wed, Mar 28, 2018 at 4:14 AM, Dai Qizhi <[email protected]> wrote:
[...]Quoted 79 lines
Quoted 57 lines
Quoted 50 lines
I tested overlayfs with
nfs_exportfeature , the kernel is built by
ubuntu , 4.16.0-041600rc7, http://kernel.ubuntu.com/~kernel-ppa
/mainline/v4.16-rc7/the problem is , when i export two different overlay mount points on
server side, then mount them use nfs on the client side, the client side
views the two differnet mount points as the same.on server side:
none on
/mnt/m1type overlay (rw,relatime,lowerdir=m1/ro,upperdir=m1/rw,workdir= m1/w,index=on,nfs_export=on)
none on/mnt/m2type overlay (rw,relatime,lowerdir=m2/ro,upperdir=m2/rw,workdir= m2/w,index=on,nfs_export=on) [root@localhost data]# exportfs -rv exporting *:/mnt/m4
exporting *:/mnt/m3
exporting *:/mnt/m2
exporting *:/mnt/m1
[root@localhost data]# ls/mnt/m1this_is_m1
[root@localhost data]# ls/mnt/m2this_is_m2on client side:
[root@localhost mnt]# mount 192.168.0.1:/mnt/m1 m1 [root@localhost mnt]# mount 192.168.0.1:/mnt/m2 m2 [root@localhost mnt]# mount |grep 192 192.168.0.1:/mnt/m1 on
/mnt/m1type nfs (rw,relatime,vers=3,rsize=524288,wsize=5 24288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mou ntaddr=192.168.0. 1,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,ad dr=192.168.0.1)
192.168.0.1:/mnt/m1 on/mnt/m2type nfs (rw,relatime,vers=3,rsize=524288,wsize=5 24288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mou ntaddr=192.168.0. 1,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,ad dr=192.168.0.1)Quoted 5 lines
[root@localhost mnt]# ls
/mnt/m1this_is_m1
[root@localhost mnt]# ls/mnt/m2this_is_m1
[root@localhost mnt]#Quoted 5 lines
Please refer to the man page of exports(5), the section about 'fsid' describes
this problem related to exporting file systems that are not on a block device,
such as overlayfs.yes, export different mount point with different fsid= option works as
expected.Quoted 10 lines
If you are interested to know if there is a way to fix this that does
not involve manually
configuring different fsid per export, I will have to consult with the
NFS experts, so please
reply to this message with CC to <[email protected]> and
<[email protected]>Quoted 5 lines
when exporting parent directory with crossmnt option and mount differnet overlayfs
under that directory, we encounter the same problem on client side..I see. Jeff, Bruce, is there a school book solution to this issue?
Is there a way for a non blockdev export to automatically identify itself
to knfsd? After all, the tuple ("overlayfs";<overlay root file handle>) should
be unique on the server. <overlay root file handle> contains (structovl_fh)
the upper fs UUID and the upper root dir file handle.Quoted 8 lines
Technically, if there is no out of tree fs on the system that is using the
valueOVL_FILEID(0xfb) for file handle type <overlay root file handle>
itself would be unique.Thanks,
Amir.tl;dr: not currently, which is why when I did the reexport patches a few years ago, they _required_ that you manually set the fsid= export option.
Longer story:
Long, long ago, the fsid for the export was almost always determined by the device major/minor tuple. That became really problematic whenever devices got reordered after adding a disk to the system and rebooting. So, Neil Brown added the ability to determine the fsid from the libblkdev uuid (see nfs-utils commit
e91ff0175602c, and kernel commits from around that time).
Quoted 5 lines
In principle, you could do something similar for overlayfs: add a new FSID_* type for overlayfs that can reliably determine a unique fsid for different overlays. That would require kernel and userland patches, of course...
That's good to know, but I guess I won't be doing any of that anytime soon. Thanks!
Amir.
Quoted 7 lines · Jeff Layton
Long, long ago, the fsid for the export was almost always determined by the device major/minor tuple. That became really problematic whenever devices got reordered after adding a disk to the system and rebooting. So, Neil Brown added the ability to determine the fsid from the libblkdev uuid (see nfs-utils commit
e91ff0175602c, and kernel commits from around that time).
I recently got burned by this because it appears that bind-mounted XFS exports are somehow different from bind-mounted ext4 exports? I.e. I was able to export one bind mounted filesystem (under an NFS4 root export) with no fsid, but when I tried to add another one (also a bind mount under the same root), the export failed in an odd way. The only difference was in the first case the underlying filesystem was ext4 and the other XFS. I'll post a question about this later, but for now ...
This seems like bad design. Wouldn't it make more sense to require that an fsid be assigned to all exports? Then there's no question in the user's mind of do I need an fsid or don't I?