Skip to contentSkip to conversation

can overlayfs work wit NFS v4 as lower fs?

DISCUSSION [email protected] 3 participants 7 messages Sep 16, 2019 – Sep 17, 2019

lore
Share & tools
lore
#0 lore raw

Hi,

I'm using overlayfs with NFS v3 as a lower (mounted read-only) and a tmpfs as an upper filesystem. This works nicely since several years. So I get a writeable directory tree on an nfs client even the nfs was mounted ro.

But then I mount the lower filesystem as NFS v4 (4.2) it does not work. Is this combination supposed to work?
I tested this with kernels up to 5.2 yet but still no success.

Here's my current test setup:

NFS server kernel 5.2.0-0.bpo.2-amd64 (Debian 10 with newer kernel) NFS client kernel 5.2.0-0.bpo.2-amd64 (Debian 10 with newer kernel)

On the NFS server I did once

# mkdir /files/scratch/t/etc; echo test-buster > /files/scratch/t/etc/test1

/etc/exports: /files/scratch 11.22.33.128/25(async,ro,no_subtree_check,no_root_squash)

I can do more tests if you tell me what I can change.
Here are the two tests result, one using nfs v3 the other using NFS v4.

P.S.: After more research I think this may be a problem of a
filesystem missing xattr support. Do you have any experiences using overlayfs using NFS v4 as lower fs?

Any hints would be appreciated.

regards Thomas

NFS v3 mount
============
Result: read and write works

+ mkdir -p /b/lower /a/upper /a/work /b/merged
+ mount -t nfs -overs=3,ro,noatime buster:/files/scratch/t /b/lower
mount -t overlay -olowerdir=$lower,upperdir=$upper,workdir=$work,default_permissions overlay $merged || exit 88
+ cat /b/merged/etc/test1
test-buster
+ sleep 3
echo "write to merged" > $merged/etc/test1
+ echo 'write to merged'
+ sleep 3
echo "write new file" > $merged/etc/test3
+ echo 'write new file'

both files test1 and test3 exists with corect content.

Here's the journalctl output of the overlayfs debug messages:

Sep 16 13:45:04 suenner kernel: overlay: mkdir(work/work, 040000) = 0
Sep 16 13:45:04 suenner kernel: overlay: tmpfile(work/work, 0100000) = 0
Sep 16 13:45:04 suenner kernel: overlay: setxattr(work/work, "trusted.overlay.opaque", "0", 1, 0x0) = 0
Sep 16 13:45:04 suenner kernel: overlay: open(00000000b7f8278e[etc/test1/l], 0100000) -> (00000000df256fc7, 0401100000)
Sep 16 13:45:07 suenner kernel: overlay: setxattr(a/upper, "trusted.overlay.impure", "y", 1, 0x0) = 0
Sep 16 13:45:07 suenner kernel: overlay: mkdir(work/#7, 040000) = 0
Sep 16 13:45:07 suenner kernel: [168B blob data]
Sep 16 13:45:07 suenner kernel: overlay: rename(work/#7, upper/etc, 0x0)
Sep 16 13:45:07 suenner kernel: overlay: setxattr(upper/etc, "trusted.overlay.impure", "y", 1, 0x0) = 0
Sep 16 13:45:07 suenner kernel: overlay: tmpfile(work/work, 0100644) = 0
Sep 16 13:45:07 suenner kernel: [174B blob data]
Sep 16 13:45:07 suenner kernel: overlay: link(work/#150994, etc/test1) = 0
Sep 16 13:45:07 suenner kernel: overlay: open(000000001998a069[etc/test1/u], 0100001) -> (0000000009d1ebd4, 0401100001)
Sep 16 13:45:10 suenner kernel: overlay: create(etc/test3, 0100666) = 0
Sep 16 13:45:10 suenner kernel: overlay: open(0000000009d1ebd4[etc/test3/u], 0100001) -> (000000001998a069, 0401100001)

The mount info
buster:/files/scratch/t on /b/lower type nfs (ro,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=11.22.33.211,mountvers=3,mountport=45356,mountproto=udp,local_lock=none,addr=11.22.33.211) overlay on /b/merged type overlay (rw,relatime,lowerdir=/b/lower,upperdir=/a/upper,workdir=/a/work,default_permissions)

-----------------------------------------------------------------------------------------------
NFS v4 mount
============
Result: read works, write does not work

+ mkdir -p /b/lower /a/upper /a/work /b/merged
+ mount -t nfs -oro,noatime buster:/files/scratch/t /b/lower
+ mount -t overlay -olowerdir=/b/lower,upperdir=/a/upper,workdir=/a/work,default_permissions overlay /b/merged

+ cat /b/merged/etc/test1
test-buster
+ sleep 3
echo "write to merged" > $merged/etc/test1
+ echo 'write to merged'
/home/lange/fairesearch/xfai/overlay-test: line 32: /b/merged/etc/test1: Operation not supported
+ sleep 3
echo "write new file" > $merged/etc/test3
+ echo 'write new file'
/home/lange/fairesearch/xfai/overlay-test: line 34: /b/merged/etc/test3: Operation not supported

Here's the journalctl output of the overlayfs debug messages:

Sep 16 13:40:57 suenner kernel: overlay: mkdir(work/work, 040000) = 0
Sep 16 13:40:57 suenner kernel: overlay: tmpfile(work/work, 0100000) = 0
Sep 16 13:40:57 suenner kernel: overlay: setxattr(work/work, "trusted.overlay.opaque", "0", 1, 0x0) = 0
Sep 16 13:40:57 suenner kernel: overlay: open(00000000ca5c0310[etc/test1/l], 0100000) -> (0000000023701f29, 0401100000)
Sep 16 13:41:00 suenner kernel: overlay: setxattr(a/upper, "trusted.overlay.impure", "y", 1, 0x0) = 0
Sep 16 13:41:00 suenner kernel: overlay: mkdir(work/#3, 040000) = 0
Sep 16 13:41:00 suenner kernel: overlay: rmdir(work/#3) = 0
Sep 16 13:41:03 suenner kernel: overlay: mkdir(work/#4, 040000) = 0
Sep 16 13:41:03 suenner kernel: overlay: rmdir(work/#4) = 0

The mount info:
buster:/files/scratch/t on /b/lower type nfs4 (ro,noatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=11.22.33.136,local_lock=none,addr=11.22.33.211) overlay on /b/merged type overlay (rw,relatime,lowerdir=/b/lower,upperdir=/a/upper,workdir=/a/work,default_permissions)

A manual test writing to the files using strace shows:

openat(AT_FDCWD, "/b/merged/etc/test1", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EOPNOTSUPP (Operation not supported)
openat(AT_FDCWD, "/b/merged/etc/test3", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EOPNOTSUPP (Operation not supported)
#1 lore raw
Quoted 21 lines · On Mon, Sep 16, 2019 at 3:25 PM <[email protected]> wrote:

Hi,

I'm using overlayfs with NFS v3 as a lower (mounted read-only) and a tmpfs as an upper filesystem. This works nicely since several years. So I get a writeable directory tree on an nfs client even the nfs was mounted ro.

But then I mount the lower filesystem as NFS v4 (4.2) it does not work. Is this combination supposed to work?
I tested this with kernels up to 5.2 yet but still no success.

Here's my current test setup:

NFS server kernel 5.2.0-0.bpo.2-amd64 (Debian 10 with newer kernel) NFS client kernel 5.2.0-0.bpo.2-amd64 (Debian 10 with newer kernel)

On the NFS server I did once

# mkdir /files/scratch/t/etc; echo test-buster > /files/scratch/t/etc/test1
Quoted 8 lines

/etc/exports: /files/scratch 11.22.33.128/25(async,ro,no_subtree_check,no_root_squash)

I can do more tests if you tell me what I can change.
Here are the two tests result, one using nfs v3 the other using NFS v4.

P.S.: After more research I think this may be a problem of a
filesystem missing xattr support. Do you have any experiences using overlayfs using NFS v4 as lower fs?

This is most probably about nfs4 acl support. Does "noacl" export
option fix it?

Thanks,
Miklos

Quoted 1 line
#2 lore raw
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Miklos Szeredi

This is most probably about nfs4 acl support. Does "noacl" export
option fix it?

Unfortunately it's not possible to disable acl support for NFS v4 (only in Debian?). That's what I read in some articles.

I've added no_acl to the exports line:

# exportfs -v
/files/scratch  134.95.9.128/25(ro,async,wdelay,no_root_squash,no_subtree_check,no_acl,sec=sys,ro,secure,no_root_squash,no_all_squash)

But this command does not show the no_acl option:

# cat /proc/fs/nfs/exports
# Version 1.1
# Path Client(Flags) # IPs
/files  134.95.9.128/25(ro,no_root_squash,sync,no_wdelay,no_subtree_check,v4root,uuid=9d4520b3:676a426f:92251f2f:b0e40f3b,sec=390003:390004:390005:1)
/       134.95.9.128/25(ro,no_root_squash,sync,no_wdelay,no_subtree_check,v4root,fsid=0,uuid=9d4520b3:676a426f:92251f2f:b0e40f3b,sec=390003:390004:390005:1)
/files/scratch  134.95.9.128/25(ro,no_root_squash,async,wdelay,no_subtree_check,uuid=9d4520b3:676a426f:92251f2f:b0e40f3b,sec=1)

I've also added -onoacl the the mount command on client side. Still the same results, when I want to write to a file in overlayfs

13415 openat(AT_FDCWD, "/b/merged/etc/test1", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EOPNOTSUPP (Operation not supported)

I've checked the access to the "original" file in the lower (NFS v4 mounted) directory, by checking the results of getfacl and nfs4_getfacl.

NFS v3, exportfs with no_acl
============================
+ strace -etrace=getxattr getfacl /b/lower/etc/test1
getxattr("/b/lower/etc/test1", "system.posix_acl_access", 0x7ffcb6b56160, 132) = -1 EOPNOTSUPP (Operation not supported)
getfacl: Removing leading '/' from absolute path names

suenner[~]# strace -etrace=getxattr nfs4_getfacl /b/lower/etc/test1 getxattr("/b/lower/etc/test1", "system.nfs4_acl", NULL, 0) = -1 EOPNOTSUPP (Operation not supported)

NFS v4 using mount -onoacl, exportfs with no_acl
================================================

+ strace -etrace=getxattr getfacl /b/lower/etc/test1
getxattr("/b/lower/etc/test1", "system.posix_acl_access", 0x7fff39fd9cd0, 132) = -1 EOPNOTSUPP (Operation not supported)

+ strace -etrace=getxattr nfs4_getfacl /b/lower/etc/test1
getxattr("/b/lower/etc/test1", "system.nfs4_acl", NULL, 0) = 80
getxattr("/b/lower/etc/test1", "system.nfs4_acl", "\0\0\0\3\0\0\0\0\0\0\0\0\0\26\1\207\0\0\0\6OWNER@\0\0\0\0\0", 80) = 80

This shows that disabling ACL in NFS v4 does not work. Does this disturb the behaviour of overlayfs?

Signature · 1 line
regards Thomas
#3 lore raw
Quoted 50 lines · Thomas Lange
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines · Miklos Szeredi

This is most probably about nfs4 acl support. Does "noacl" export
option fix it?

Unfortunately it's not possible to disable acl support for NFS v4 (only in Debian?). That's what I read in some articles.

I've added no_acl to the exports line:

# exportfs -v
/files/scratch  134.95.9.128/25(ro,async,wdelay,no_root_squash,no_subtree_check,no_acl,sec=sys,ro,secure,no_root_squash,no_all_squash)

But this command does not show the no_acl option:

# cat /proc/fs/nfs/exports
# Version 1.1
# Path Client(Flags) # IPs
/files  134.95.9.128/25(ro,no_root_squash,sync,no_wdelay,no_subtree_check,v4root,uuid=9d4520b3:676a426f:92251f2f:b0e40f3b,sec=390003:390004:390005:1)
/       134.95.9.128/25(ro,no_root_squash,sync,no_wdelay,no_subtree_check,v4root,fsid=0,uuid=9d4520b3:676a426f:92251f2f:b0e40f3b,sec=390003:390004:390005:1)
/files/scratch  134.95.9.128/25(ro,no_root_squash,async,wdelay,no_subtree_check,uuid=9d4520b3:676a426f:92251f2f:b0e40f3b,sec=1)

I've also added -onoacl the the mount command on client side. Still the same results, when I want to write to a file in overlayfs

13415 openat(AT_FDCWD, "/b/merged/etc/test1", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EOPNOTSUPP (Operation not supported)

I've checked the access to the "original" file in the lower (NFS v4 mounted) directory, by checking the results of getfacl and nfs4_getfacl.

NFS v3, exportfs with no_acl
============================
+ strace -etrace=getxattr getfacl /b/lower/etc/test1
getxattr("/b/lower/etc/test1", "system.posix_acl_access", 0x7ffcb6b56160, 132) = -1 EOPNOTSUPP (Operation not supported)
getfacl: Removing leading '/' from absolute path names

suenner[~]# strace -etrace=getxattr nfs4_getfacl /b/lower/etc/test1 getxattr("/b/lower/etc/test1", "system.nfs4_acl", NULL, 0) = -1 EOPNOTSUPP (Operation not supported)

NFS v4 using mount -onoacl, exportfs with no_acl
================================================

+ strace -etrace=getxattr getfacl /b/lower/etc/test1
getxattr("/b/lower/etc/test1", "system.posix_acl_access", 0x7fff39fd9cd0, 132) = -1 EOPNOTSUPP (Operation not supported)

Quoted 7 lines

+ strace -etrace=getxattr nfs4_getfacl /b/lower/etc/test1
getxattr("/b/lower/etc/test1", "system.nfs4_acl", NULL, 0) = 80
getxattr("/b/lower/etc/test1", "system.nfs4_acl", "\0\0\0\3\0\0\0\0\0\0\0\0\0\26\1\207\0\0\0\6OWNER@\0\0\0\0\0", 80) = 80

This shows that disabling ACL in NFS v4 does not work. Does this disturb the behaviour of overlayfs?

The problem is that overlayfs tries to copy-up the system.nfs4_acl, but fails (no other filesystem supports this xattr). Similar thing happens with "cp --preserv=xattr".

Looking back...

https://lore.kernel.org/linux-fsdevel/CAHpGcMKHjic6L+J0qvMYNG9hVCcDO1hEpx4BiEk0ZCKDV39BmA@mail.gmail.com/

Ah, the way to disable these is to disable acl on the exported filesystem. I.e.

mount -oremount,noacl $EXPORTED_FS

Doesn't work if that filesystem is used for other purposes that need acls.

Thanks,
Miklos

#4 lore raw
Quoted 61 lines · Miklos Szeredi
Quoted 50 lines · Thomas Lange
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines · Miklos Szeredi

This is most probably about nfs4 acl support. Does "noacl" export
option fix it?

Unfortunately it's not possible to disable acl support for NFS v4 (only in Debian?). That's what I read in some articles.

I've added no_acl to the exports line:

# exportfs -v
/files/scratch  134.95.9.128/25(ro,async,wdelay,no_root_squash,no_subtree_check,no_acl,sec=sys,ro,secure,no_root_squash,no_all_squash)

But this command does not show the no_acl option:

# cat /proc/fs/nfs/exports
# Version 1.1
# Path Client(Flags) # IPs
/files  134.95.9.128/25(ro,no_root_squash,sync,no_wdelay,no_subtree_check,v4root,uuid=9d4520b3:676a426f:92251f2f:b0e40f3b,sec=390003:390004:390005:1)
/       134.95.9.128/25(ro,no_root_squash,sync,no_wdelay,no_subtree_check,v4root,fsid=0,uuid=9d4520b3:676a426f:92251f2f:b0e40f3b,sec=390003:390004:390005:1)
/files/scratch  134.95.9.128/25(ro,no_root_squash,async,wdelay,no_subtree_check,uuid=9d4520b3:676a426f:92251f2f:b0e40f3b,sec=1)

I've also added -onoacl the the mount command on client side. Still the same results, when I want to write to a file in overlayfs

13415 openat(AT_FDCWD, "/b/merged/etc/test1", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EOPNOTSUPP (Operation not supported)

I've checked the access to the "original" file in the lower (NFS v4 mounted) directory, by checking the results of getfacl and nfs4_getfacl.

NFS v3, exportfs with no_acl
============================
+ strace -etrace=getxattr getfacl /b/lower/etc/test1
getxattr("/b/lower/etc/test1", "system.posix_acl_access", 0x7ffcb6b56160, 132) = -1 EOPNOTSUPP (Operation not supported)
getfacl: Removing leading '/' from absolute path names

suenner[~]# strace -etrace=getxattr nfs4_getfacl /b/lower/etc/test1 getxattr("/b/lower/etc/test1", "system.nfs4_acl", NULL, 0) = -1 EOPNOTSUPP (Operation not supported)

NFS v4 using mount -onoacl, exportfs with no_acl
================================================

+ strace -etrace=getxattr getfacl /b/lower/etc/test1
getxattr("/b/lower/etc/test1", "system.posix_acl_access", 0x7fff39fd9cd0, 132) = -1 EOPNOTSUPP (Operation not supported)

Quoted 7 lines

+ strace -etrace=getxattr nfs4_getfacl /b/lower/etc/test1
getxattr("/b/lower/etc/test1", "system.nfs4_acl", NULL, 0) = 80
getxattr("/b/lower/etc/test1", "system.nfs4_acl", "\0\0\0\3\0\0\0\0\0\0\0\0\0\26\1\207\0\0\0\6OWNER@\0\0\0\0\0", 80) = 80

This shows that disabling ACL in NFS v4 does not work. Does this disturb the behaviour of overlayfs?

Quoted 7 lines

The problem is that overlayfs tries to copy-up the system.nfs4_acl, but fails (no other filesystem supports this xattr). Similar thing happens with "cp --preserv=xattr".

Looking back...

https://lore.kernel.org/linux-fsdevel/CAHpGcMKHjic6L+J0qvMYNG9hVCcDO1hEpx4BiEk0ZCKDV39BmA@mail.gmail.com/

The other thread discussing this issue:

https://lore.kernel.org/linux-nfs/[email protected]/

Thanks,
Miklos

#5 lore raw
In reply to Miklos Szeredi
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Miklos Szeredi

Ah, the way to disable these is to disable acl on the exported filesystem. I.e. mount -oremount,noacl $EXPORTED_FS

That works perfect for me. Thanks a lot for your help.

It would be nice to add a note about this in the kernel overlayfs.txt. What about this:

The lower filesystem can be any filesystem supported by Linux and does
-not need to be writable. The lower filesystem can even be another +not need to be writable. If you are using an NFS v4 mount as lower +filesystem, you should disable acl on the exported filesystem. +The lower filesystem can even be another

Signature · 1 line
regards Thomas
#6 lore raw
Quoted 11 lines · Thomas Lange
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines · Miklos Szeredi

Ah, the way to disable these is to disable acl on the exported filesystem. I.e. mount -oremount,noacl $EXPORTED_FS

That works perfect for me. Thanks a lot for your help.

It would be nice to add a note about this in the kernel overlayfs.txt. What about this:

The lower filesystem can be any filesystem supported by Linux and does

Review

Quoted 4 lines
(quoted diff) +3 -1
-not need to be writable.  The lower filesystem can even be another
+not need to be writable. If you are using an NFS v4 mount as lower
+filesystem, you should disable acl on the exported filesystem.
+The lower filesystem can even be another

Just started working on documentation update to explain this and related issues.

Thanks,
Miklos