can overlayfs work wit NFS v4 as lower fs?
Hi,
I'm using overlayfs with NFS v3 as a lower (mounted read-only) and a tmpfs as an upper filesystem. This works nicely since several years. So I get a writeable directory tree on an nfs client even the nfs was mounted ro.
But then I mount the lower filesystem as NFS v4 (4.2) it does not work. Is this combination supposed to work?
I tested this with kernels up to 5.2 yet but still no success.
Here's my current test setup:
NFS server kernel 5.2.0-0.bpo.2-amd64 (Debian 10 with newer kernel) NFS client kernel 5.2.0-0.bpo.2-amd64 (Debian 10 with newer kernel)
On the NFS server I did once
# mkdir /files/scratch/t/etc; echo test-buster > /files/scratch/t/etc/test1
/etc/exports: /files/scratch 11.22.33.128/25(async,ro,no_subtree_check,no_root_squash)
I can do more tests if you tell me what I can change.
Here are the two tests result, one using nfs v3 the other using NFS v4.
P.S.: After more research I think this may be a problem of a
filesystem missing xattr support. Do you have any experiences using overlayfs using NFS v4 as lower fs?
Any hints would be appreciated.
regards Thomas
NFS v3 mount
============
Result: read and write works
+ mkdir -p /b/lower /a/upper /a/work /b/merged
+ mount -t nfs -overs=3,ro,noatime buster:/files/scratch/t /b/lower
mount -t overlay -olowerdir=$lower,upperdir=$upper,workdir=$work,default_permissions overlay $merged || exit 88
+ cat /b/merged/etc/test1
test-buster
+ sleep 3
echo "write to merged" > $merged/etc/test1
+ echo 'write to merged'
+ sleep 3
echo "write new file" > $merged/etc/test3
+ echo 'write new file'
both files test1 and test3 exists with corect content.
Here's the journalctl output of the overlayfs debug messages:
Sep 16 13:45:04 suenner kernel: overlay: mkdir(work/work, 040000) = 0 Sep 16 13:45:04 suenner kernel: overlay: tmpfile(work/work, 0100000) = 0 Sep 16 13:45:04 suenner kernel: overlay: setxattr(work/work, "trusted.overlay.opaque", "0", 1, 0x0) = 0 Sep 16 13:45:04 suenner kernel: overlay: open(00000000b7f8278e[etc/test1/l], 0100000) -> (00000000df256fc7, 0401100000) Sep 16 13:45:07 suenner kernel: overlay: setxattr(a/upper, "trusted.overlay.impure", "y", 1, 0x0) = 0 Sep 16 13:45:07 suenner kernel: overlay: mkdir(work/#7, 040000) = 0 Sep 16 13:45:07 suenner kernel: [168B blob data] Sep 16 13:45:07 suenner kernel: overlay: rename(work/#7, upper/etc, 0x0) Sep 16 13:45:07 suenner kernel: overlay: setxattr(upper/etc, "trusted.overlay.impure", "y", 1, 0x0) = 0 Sep 16 13:45:07 suenner kernel: overlay: tmpfile(work/work, 0100644) = 0 Sep 16 13:45:07 suenner kernel: [174B blob data] Sep 16 13:45:07 suenner kernel: overlay: link(work/#150994, etc/test1) = 0 Sep 16 13:45:07 suenner kernel: overlay: open(000000001998a069[etc/test1/u], 0100001) -> (0000000009d1ebd4, 0401100001) Sep 16 13:45:10 suenner kernel: overlay: create(etc/test3, 0100666) = 0 Sep 16 13:45:10 suenner kernel: overlay: open(0000000009d1ebd4[etc/test3/u], 0100001) -> (000000001998a069, 0401100001)
The mount info
buster:/files/scratch/t on /b/lower type nfs (ro,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=11.22.33.211,mountvers=3,mountport=45356,mountproto=udp,local_lock=none,addr=11.22.33.211) overlay on /b/merged type overlay (rw,relatime,lowerdir=/b/lower,upperdir=/a/upper,workdir=/a/work,default_permissions)
-----------------------------------------------------------------------------------------------
NFS v4 mount
============
Result: read works, write does not work
+ mkdir -p /b/lower /a/upper /a/work /b/merged
+ mount -t nfs -oro,noatime buster:/files/scratch/t /b/lower
+ mount -t overlay -olowerdir=/b/lower,upperdir=/a/upper,workdir=/a/work,default_permissions overlay /b/merged
+ cat /b/merged/etc/test1
test-buster
+ sleep 3
echo "write to merged" > $merged/etc/test1
+ echo 'write to merged'/home/lange/fairesearch/xfai/overlay-test: line 32: /b/merged/etc/test1: Operation not supported
+ sleep 3
echo "write new file" > $merged/etc/test3
+ echo 'write new file'/home/lange/fairesearch/xfai/overlay-test: line 34: /b/merged/etc/test3: Operation not supported
Here's the journalctl output of the overlayfs debug messages:
Sep 16 13:40:57 suenner kernel: overlay: mkdir(work/work, 040000) = 0 Sep 16 13:40:57 suenner kernel: overlay: tmpfile(work/work, 0100000) = 0 Sep 16 13:40:57 suenner kernel: overlay: setxattr(work/work, "trusted.overlay.opaque", "0", 1, 0x0) = 0 Sep 16 13:40:57 suenner kernel: overlay: open(00000000ca5c0310[etc/test1/l], 0100000) -> (0000000023701f29, 0401100000) Sep 16 13:41:00 suenner kernel: overlay: setxattr(a/upper, "trusted.overlay.impure", "y", 1, 0x0) = 0 Sep 16 13:41:00 suenner kernel: overlay: mkdir(work/#3, 040000) = 0 Sep 16 13:41:00 suenner kernel: overlay: rmdir(work/#3) = 0 Sep 16 13:41:03 suenner kernel: overlay: mkdir(work/#4, 040000) = 0 Sep 16 13:41:03 suenner kernel: overlay: rmdir(work/#4) = 0
The mount info:
buster:/files/scratch/t on /b/lower type nfs4 (ro,noatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=11.22.33.136,local_lock=none,addr=11.22.33.211) overlay on /b/merged type overlay (rw,relatime,lowerdir=/b/lower,upperdir=/a/upper,workdir=/a/work,default_permissions)
A manual test writing to the files using strace shows:
openat(AT_FDCWD, "/b/merged/etc/test1", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EOPNOTSUPP (Operation not supported) openat(AT_FDCWD, "/b/merged/etc/test3", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EOPNOTSUPP (Operation not supported)
Quoted 21 lines · On Mon, Sep 16, 2019 at 3:25 PM <[email protected]> wrote:
Hi,
I'm using overlayfs with NFS v3 as a lower (mounted read-only) and a tmpfs as an upper filesystem. This works nicely since several years. So I get a writeable directory tree on an nfs client even the nfs was mounted ro.
But then I mount the lower filesystem as NFS v4 (4.2) it does not work. Is this combination supposed to work?
I tested this with kernels up to 5.2 yet but still no success.Here's my current test setup:
NFS server kernel 5.2.0-0.bpo.2-amd64 (Debian 10 with newer kernel) NFS client kernel 5.2.0-0.bpo.2-amd64 (Debian 10 with newer kernel)
On the NFS server I did once
# mkdir /files/scratch/t/etc; echo test-buster > /files/scratch/t/etc/test1
Quoted 8 lines
/etc/exports: /files/scratch 11.22.33.128/25(async,ro,no_subtree_check,no_root_squash)I can do more tests if you tell me what I can change.
Here are the two tests result, one using nfs v3 the other using NFS v4.
P.S.: After more research I think this may be a problem of a
filesystem missing xattr support. Do you have any experiences using overlayfs using NFS v4 as lower fs?
This is most probably about nfs4 acl support. Does "noacl" export
option fix it?
Thanks,
Miklos
Quoted 1 line
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Miklos Szeredi
This is most probably about nfs4 acl support. Does "noacl" export
option fix it?
Unfortunately it's not possible to disable acl support for NFS v4 (only in Debian?). That's what I read in some articles.
I've added no_acl to the exports line:
# exportfs -v /files/scratch 134.95.9.128/25(ro,async,wdelay,no_root_squash,no_subtree_check,no_acl,sec=sys,ro,secure,no_root_squash,no_all_squash)
But this command does not show the no_acl option:
# cat /proc/fs/nfs/exports # Version 1.1 # Path Client(Flags) # IPs /files 134.95.9.128/25(ro,no_root_squash,sync,no_wdelay,no_subtree_check,v4root,uuid=9d4520b3:676a426f:92251f2f:b0e40f3b,sec=390003:390004:390005:1) / 134.95.9.128/25(ro,no_root_squash,sync,no_wdelay,no_subtree_check,v4root,fsid=0,uuid=9d4520b3:676a426f:92251f2f:b0e40f3b,sec=390003:390004:390005:1) /files/scratch 134.95.9.128/25(ro,no_root_squash,async,wdelay,no_subtree_check,uuid=9d4520b3:676a426f:92251f2f:b0e40f3b,sec=1)
I've also added -onoacl the the mount command on client side. Still the same results, when I want to write to a file in overlayfs
13415 openat(AT_FDCWD, "/b/merged/etc/test1", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EOPNOTSUPP (Operation not supported)
I've checked the access to the "original" file in the lower (NFS v4 mounted) directory, by checking the results of getfacl and nfs4_getfacl.
NFS v3, exportfs with no_acl
============================
+ strace -etrace=getxattr getfacl /b/lower/etc/test1getxattr("/b/lower/etc/test1", "system.posix_acl_access", 0x7ffcb6b56160, 132) = -1 EOPNOTSUPP (Operation not supported)
getfacl: Removing leading '/' from absolute path names
suenner[~]# strace -etrace=getxattr nfs4_getfacl /b/lower/etc/test1 getxattr("/b/lower/etc/test1", "system.nfs4_acl", NULL, 0) = -1 EOPNOTSUPP (Operation not supported)
NFS v4 using mount -onoacl, exportfs with no_acl
================================================
+ strace -etrace=getxattr getfacl /b/lower/etc/test1getxattr("/b/lower/etc/test1", "system.posix_acl_access", 0x7fff39fd9cd0, 132) = -1 EOPNOTSUPP (Operation not supported)
+ strace -etrace=getxattr nfs4_getfacl /b/lower/etc/test1getxattr("/b/lower/etc/test1", "system.nfs4_acl", NULL, 0) = 80
getxattr("/b/lower/etc/test1", "system.nfs4_acl", "\0\0\0\3\0\0\0\0\0\0\0\0\0\26\1\207\0\0\0\6OWNER@\0\0\0\0\0", 80) = 80
This shows that disabling ACL in NFS v4 does not work. Does this disturb the behaviour of overlayfs?
Signature · 1 line
regards Thomas
Quoted 50 lines · Thomas Lange
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines · Miklos Szeredi
This is most probably about nfs4 acl support. Does "noacl" export
option fix it?Unfortunately it's not possible to disable acl support for NFS v4 (only in Debian?). That's what I read in some articles.
I've added
no_aclto the exports line:# exportfs -v /files/scratch 134.95.9.128/25(ro,async,wdelay,no_root_squash,no_subtree_check,no_acl,sec=sys,ro,secure,no_root_squash,no_all_squash)But this command does not show the
no_acloption:# cat /proc/fs/nfs/exports # Version 1.1 # Path Client(Flags) # IPs /files 134.95.9.128/25(ro,no_root_squash,sync,no_wdelay,no_subtree_check,v4root,uuid=9d4520b3:676a426f:92251f2f:b0e40f3b,sec=390003:390004:390005:1) / 134.95.9.128/25(ro,no_root_squash,sync,no_wdelay,no_subtree_check,v4root,fsid=0,uuid=9d4520b3:676a426f:92251f2f:b0e40f3b,sec=390003:390004:390005:1) /files/scratch 134.95.9.128/25(ro,no_root_squash,async,wdelay,no_subtree_check,uuid=9d4520b3:676a426f:92251f2f:b0e40f3b,sec=1)I've also added -onoacl the the mount command on client side. Still the same results, when I want to write to a file in overlayfs
13415
openat(AT_FDCWD, "/b/merged/etc/test1", O_WRONLY|O_CREAT|O_TRUNC, 0666)= -1 EOPNOTSUPP (Operation not supported)I've checked the access to the "original" file in the lower (NFS v4 mounted) directory, by checking the results of getfacl and
nfs4_getfacl.NFS v3, exportfs with
no_acl
============================
+ strace -etrace=getxattr getfacl /b/lower/etc/test1getxattr("/b/lower/etc/test1", "system.posix_acl_access", 0x7ffcb6b56160, 132)= -1 EOPNOTSUPP (Operation not supported)
getfacl: Removing leading '/' from absolute path namessuenner[~]# strace -etrace=getxattr
nfs4_getfacl/b/lower/etc/test1getxattr("/b/lower/etc/test1", "system.nfs4_acl", NULL, 0)= -1 EOPNOTSUPP (Operation not supported)NFS v4 using mount -onoacl, exportfs with
no_acl
================================================+ strace -etrace=getxattr getfacl /b/lower/etc/test1
getxattr("/b/lower/etc/test1", "system.posix_acl_access", 0x7fff39fd9cd0, 132)= -1 EOPNOTSUPP (Operation not supported)
Quoted 7 lines
+ strace -etrace=getxattr
nfs4_getfacl/b/lower/etc/test1getxattr("/b/lower/etc/test1", "system.nfs4_acl", NULL, 0)= 80
getxattr("/b/lower/etc/test1", "system.nfs4_acl", "\0\0\0\3\0\0\0\0\0\0\0\0\0\26\1\207\0\0\0\6OWNER@\0\0\0\0\0", 80) = 80This shows that disabling ACL in NFS v4 does not work. Does this disturb the behaviour of overlayfs?
The problem is that overlayfs tries to copy-up the system.nfs4_acl, but fails (no other filesystem supports this xattr). Similar thing happens with "cp --preserv=xattr".
Looking back...
Ah, the way to disable these is to disable acl on the exported filesystem. I.e.
mount -oremount,noacl $EXPORTED_FS
Doesn't work if that filesystem is used for other purposes that need acls.
Thanks,
Miklos
Quoted 61 lines · Miklos Szeredi
Quoted 50 lines · Thomas Lange
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines · Miklos Szeredi
This is most probably about nfs4 acl support. Does "noacl" export
option fix it?Unfortunately it's not possible to disable acl support for NFS v4 (only in Debian?). That's what I read in some articles.
I've added
no_aclto the exports line:# exportfs -v /files/scratch 134.95.9.128/25(ro,async,wdelay,no_root_squash,no_subtree_check,no_acl,sec=sys,ro,secure,no_root_squash,no_all_squash)But this command does not show the
no_acloption:# cat /proc/fs/nfs/exports # Version 1.1 # Path Client(Flags) # IPs /files 134.95.9.128/25(ro,no_root_squash,sync,no_wdelay,no_subtree_check,v4root,uuid=9d4520b3:676a426f:92251f2f:b0e40f3b,sec=390003:390004:390005:1) / 134.95.9.128/25(ro,no_root_squash,sync,no_wdelay,no_subtree_check,v4root,fsid=0,uuid=9d4520b3:676a426f:92251f2f:b0e40f3b,sec=390003:390004:390005:1) /files/scratch 134.95.9.128/25(ro,no_root_squash,async,wdelay,no_subtree_check,uuid=9d4520b3:676a426f:92251f2f:b0e40f3b,sec=1)I've also added -onoacl the the mount command on client side. Still the same results, when I want to write to a file in overlayfs
13415
openat(AT_FDCWD, "/b/merged/etc/test1", O_WRONLY|O_CREAT|O_TRUNC, 0666)= -1 EOPNOTSUPP (Operation not supported)I've checked the access to the "original" file in the lower (NFS v4 mounted) directory, by checking the results of getfacl and
nfs4_getfacl.NFS v3, exportfs with
no_acl
============================
+ strace -etrace=getxattr getfacl /b/lower/etc/test1getxattr("/b/lower/etc/test1", "system.posix_acl_access", 0x7ffcb6b56160, 132)= -1 EOPNOTSUPP (Operation not supported)
getfacl: Removing leading '/' from absolute path namessuenner[~]# strace -etrace=getxattr
nfs4_getfacl/b/lower/etc/test1getxattr("/b/lower/etc/test1", "system.nfs4_acl", NULL, 0)= -1 EOPNOTSUPP (Operation not supported)NFS v4 using mount -onoacl, exportfs with
no_acl
================================================+ strace -etrace=getxattr getfacl /b/lower/etc/test1
getxattr("/b/lower/etc/test1", "system.posix_acl_access", 0x7fff39fd9cd0, 132)= -1 EOPNOTSUPP (Operation not supported)Quoted 7 lines
+ strace -etrace=getxattr
nfs4_getfacl/b/lower/etc/test1getxattr("/b/lower/etc/test1", "system.nfs4_acl", NULL, 0)= 80
getxattr("/b/lower/etc/test1", "system.nfs4_acl", "\0\0\0\3\0\0\0\0\0\0\0\0\0\26\1\207\0\0\0\6OWNER@\0\0\0\0\0", 80) = 80This shows that disabling ACL in NFS v4 does not work. Does this disturb the behaviour of overlayfs?
Quoted 7 lines
The problem is that overlayfs tries to copy-up the system.nfs4_acl, but fails (no other filesystem supports this xattr). Similar thing happens with "cp --preserv=xattr".
Looking back...
The other thread discussing this issue:
https://lore.kernel.org/linux-nfs/[email protected]/
Thanks,
Miklos
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Miklos Szeredi
Ah, the way to disable these is to disable acl on the exported filesystem. I.e. mount -oremount,noacl $EXPORTED_FS
That works perfect for me. Thanks a lot for your help.
It would be nice to add a note about this in the kernel overlayfs.txt. What about this:
The lower filesystem can be any filesystem supported by Linux and does
-not need to be writable. The lower filesystem can even be another +not need to be writable. If you are using an NFS v4 mount as lower +filesystem, you should disable acl on the exported filesystem. +The lower filesystem can even be another
Signature · 1 line
regards Thomas
Quoted 11 lines · Thomas Lange
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines · Miklos Szeredi
Ah, the way to disable these is to disable acl on the exported filesystem. I.e. mount -oremount,noacl $EXPORTED_FS
That works perfect for me. Thanks a lot for your help.
It would be nice to add a note about this in the kernel overlayfs.txt. What about this:
The lower filesystem can be any filesystem supported by Linux and does
Review
Quoted 4 lines
(quoted diff) +3
-1-not need to be writable. The lower filesystem can even be another +not need to be writable. If you are using an NFS v4 mount as lower +filesystem, you should disable acl on the exported filesystem. +The lower filesystem can even be another
Just started working on documentation update to explain this and related issues.
Thanks,
Miklos