ceph: allow encrypting a directory while not having Ax caps
RFC PATCH 1/1 ceph: allow encrypting a directory while not having Ax caps
If a client doesn't have Fx caps on a directory, it will get errors while trying encrypt it:
ceph: handle_cap_grant: cap grant attempt to change fscrypt_auth on non-I_NEW inode (old len 0 new len 48) fscrypt (ceph, inode 1099511627812): Error -105 getting encryption context
A simple way to reproduce this is to use two clients:
client1 # mkdir /mnt/mydir
client2 # ls /mnt/mydir
client1 # fscrypt encrypt /mnt/mydir
client1 # echo hello > /mnt/mydir/world
This happens because, in __ceph_setattr(), we only initialize ci->fscrypt_auth if we have Ax. If we don't have, we'll need to do that later, in handle_cap_grant().
---
Hi!
To be honest, I'm not really sure about the conditions in the 'if': shall I bother checking it's really a dir and that it is empty?
Cheers,
--
Luís
fs/ceph/caps.c | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-)
fs/ceph/caps.c +23 -3
@@ -3511,9 +3511,29 @@ static void handle_cap_grant(struct inode *inode,from_kuid (&init_user_ns, inode->i_uid), from_kgid(&init_user_ns, inode->i_gid));#if IS_ENABLED (CONFIG_FS_ENCRYPTION ) -if (ci->fscrypt_auth_len != extra_info->fscrypt_auth_len || -memcmp (ci->fscrypt_auth, extra_info->fscrypt_auth, - ci->fscrypt_auth_len)) +if ((ci->fscrypt_auth_len ==0 ) && + (extra_info->fscrypt_auth_len >0 ) && +S_ISDIR (inode->i_mode) && + (ci->i_rsubdirs + ci->i_rfiles ==1 )) { +/* +* We'll get here when setting up an encrypted directory +* but we don't have Fx in that directory, i.e. other +* clients have accessed this directory too. +*/ + ci->fscrypt_auth =kmemdup (extra_info->fscrypt_auth, + extra_info->fscrypt_auth_len, +GFP_KERNEL ); +if (ci->fscrypt_auth) { + inode->i_flags |=S_ENCRYPTED ; + ci->fscrypt_auth_len = extra_info->fscrypt_auth_len; + }else { +pr_err ("Failed to alloc memory for %llx.%llx fscrypt_auth\n" , +ceph_vinop (inode)); + } +dout ("ino %llx.%llx is now encrypted\n" ,ceph_vinop (inode)); + }else if (ci->fscrypt_auth_len != extra_info->fscrypt_auth_len || +memcmp (ci->fscrypt_auth, extra_info->fscrypt_auth, + ci->fscrypt_auth_len))pr_warn_ratelimited ("%s: cap grant attempt to change fscrypt_auth on non-I_NEW inode (old len %d new len %d)\n" , __func__, ci->fscrypt_auth_len, extra_info->fscrypt_auth_len);#endif
Quoted 60 lines · Luís Henriques
If a client doesn't have Fx caps on a directory, it will get errors while trying encrypt it:
ceph:
handle_cap_grant: cap grant attempt to changefscrypt_authon non-I_NEW inode (old len 0 new len 48) fscrypt (ceph, inode 1099511627812): Error -105 getting encryption contextA simple way to reproduce this is to use two clients:
client1 # mkdir
/mnt/mydirclient2 # ls
/mnt/mydirclient1 # fscrypt encrypt
/mnt/mydir
client1 # echo hello >/mnt/mydir/worldThis happens because, in
__ceph_setattr(), we only initializeci->fscrypt_authif we have Ax. If we don't have, we'll need to do that later, inhandle_cap_grant().Signed-off-by Luís Henriques <[email protected]>---
Hi!To be honest, I'm not really sure about the conditions in the 'if': shall I bother checking it's really a dir and that it is empty?
Cheers,
--
Luísfs/ceph/caps.c | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-)fs/ceph/caps.c +16
-3@@ -3511,9 +3511,29 @@ static void handle_cap_grant(struct inode *inode,from_kuid (&init_user_ns, inode->i_uid), from_kgid(&init_user_ns, inode->i_gid));#if IS_ENABLED (CONFIG_FS_ENCRYPTION ) -if (ci->fscrypt_auth_len != extra_info->fscrypt_auth_len || -memcmp (ci->fscrypt_auth, extra_info->fscrypt_auth, - ci->fscrypt_auth_len)) +if ((ci->fscrypt_auth_len ==0 ) && + (extra_info->fscrypt_auth_len >0 ) && +S_ISDIR (inode->i_mode) && + (ci->i_rsubdirs + ci->i_rfiles ==1 )) { +/* +* We'll get here when setting up an encrypted directory +* but we don't have Fx in that directory, i.e. other +* clients have accessed this directory too. +*/ + ci->fscrypt_auth =kmemdup (extra_info->fscrypt_auth, + extra_info->fscrypt_auth_len, +GFP_KERNEL ); +if (ci->fscrypt_auth) { + inode->i_flags |=S_ENCRYPTED ; + ci->fscrypt_auth_len = extra_info->fscrypt_auth_len; + }else {
fs/ceph/caps.c static void handle_cap_grant(struct inode *inode, · line 3539
Quoted 10 lines
fs/ceph/caps.c +7
-0+pr_err ("Failed to alloc memory for %llx.%llx fscrypt_auth\n" , + ceph_vinop(inode)); + } +dout ("ino %llx.%llx is now encrypted\n" ,ceph_vinop (inode)); + }else if (ci->fscrypt_auth_len != extra_info->fscrypt_auth_len || +memcmp (ci->fscrypt_auth, extra_info->fscrypt_auth, + ci->fscrypt_auth_len))pr_warn_ratelimited ("%s: cap grant attempt to change fscrypt_auth on non-I_NEW inode (old len %d new len %d)\n" , __func__, ci->fscrypt_auth_len, extra_info->fscrypt_auth_len);#endif
Hi Luis,
Thanks for your time on this bug.
IMO we should fix this in ceph_fill_inode():
995 #ifdef CONFIG_FS_ENCRYPTION
996 if (iinfo->fscrypt_auth_len && (inode->i_state & I_NEW)) { 997 kfree(ci->fscrypt_auth);
998 ci->fscrypt_auth_len = iinfo->fscrypt_auth_len; 999 ci->fscrypt_auth = iinfo->fscrypt_auth; 1000 iinfo->fscrypt_auth = NULL;
1001 iinfo->fscrypt_auth_len = 0;
1002 inode_set_flags(inode, S_ENCRYPTED, S_ENCRYPTED); 1003 }
1004 #endif
The setattr will get a reply from MDS including the fscrypt auth info, I think the kclient just drop it here.
If we fix it in handle_cap_grant() I am afraid this bug still exists. What if there is no any new caps will be issued or revoked recently and then access to the directory ?
Thanks
- Xiubo
Quoted 1 line
Quoted 90 lines · Xiubo Li
Quoted 60 lines · Luís Henriques
If a client doesn't have Fx caps on a directory, it will get errors while trying encrypt it:
ceph:
handle_cap_grant: cap grant attempt to changefscrypt_authon non-I_NEW inode (old len 0 new len 48) fscrypt (ceph, inode 1099511627812): Error -105 getting encryption contextA simple way to reproduce this is to use two clients:
client1 # mkdir
/mnt/mydirclient2 # ls
/mnt/mydirclient1 # fscrypt encrypt
/mnt/mydir
client1 # echo hello >/mnt/mydir/worldThis happens because, in
__ceph_setattr(), we only initializeci->fscrypt_authif we have Ax. If we don't have, we'll need to do that later, inhandle_cap_grant().Signed-off-by Luís Henriques <[email protected]>---
Hi!To be honest, I'm not really sure about the conditions in the 'if': shall I bother checking it's really a dir and that it is empty?
Cheers,
--
Luísfs/ceph/caps.c | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-)fs/ceph/caps.c +16
-3@@ -3511,9 +3511,29 @@ static void handle_cap_grant(struct inode *inode,from_kuid (&init_user_ns, inode->i_uid), from_kgid(&init_user_ns, inode->i_gid));#if IS_ENABLED (CONFIG_FS_ENCRYPTION ) -if (ci->fscrypt_auth_len != extra_info->fscrypt_auth_len || -memcmp (ci->fscrypt_auth, extra_info->fscrypt_auth, - ci->fscrypt_auth_len)) +if ((ci->fscrypt_auth_len ==0 ) && + (extra_info->fscrypt_auth_len >0 ) && +S_ISDIR (inode->i_mode) && + (ci->i_rsubdirs + ci->i_rfiles ==1 )) { +/* +* We'll get here when setting up an encrypted directory +* but we don't have Fx in that directory, i.e. other +* clients have accessed this directory too. +*/ + ci->fscrypt_auth =kmemdup (extra_info->fscrypt_auth, + extra_info->fscrypt_auth_len, +GFP_KERNEL ); +if (ci->fscrypt_auth) { + inode->i_flags |=S_ENCRYPTED ; + ci->fscrypt_auth_len = extra_info->fscrypt_auth_len; + }else {fs/ceph/caps.c static void handle_cap_grant(struct inode *inode, · line 3539
Quoted 10 lines
fs/ceph/caps.c +7
-0+pr_err ("Failed to alloc memory for %llx.%llx fscrypt_auth\n" , + ceph_vinop(inode)); + } +dout ("ino %llx.%llx is now encrypted\n" ,ceph_vinop (inode)); + }else if (ci->fscrypt_auth_len != extra_info->fscrypt_auth_len || +memcmp (ci->fscrypt_auth, extra_info->fscrypt_auth, + ci->fscrypt_auth_len))pr_warn_ratelimited ("%s: cap grant attempt to change fscrypt_auth on non-I_NEW inode (old len %d new len %d)\n" , __func__, ci->fscrypt_auth_len, extra_info->fscrypt_auth_len);#endif Hi Luis,
Thanks for your time on this bug.
IMO we should fix this in
ceph_fill_inode():995 #ifdef
CONFIG_FS_ENCRYPTION
996 if (iinfo->fscrypt_auth_len&& (inode->i_state&I_NEW)) { 997kfree(ci->fscrypt_auth);
998ci->fscrypt_auth_len=iinfo->fscrypt_auth_len; 999ci->fscrypt_auth=iinfo->fscrypt_auth; 1000iinfo->fscrypt_auth= NULL;
1001iinfo->fscrypt_auth_len= 0;
1002inode_set_flags(inode, S_ENCRYPTED, S_ENCRYPTED); 1003 }
1004 #endif
Quoted 2 lines
The setattr will get a reply from MDS including the fscrypt auth info, I think the kclient just drop it here.
I've done some testing and I don't really see this code kfree'ing a valid fscrypt_auth here. However, I guess it is possible to fix this issue here too, but in a different way, by changing that 'if' condition to:
if (iinfo->fscrypt_auth_len &&
((inode->i_state & I_NEW) || (ci->fscrypt_auth_len == 0))) {
...
}I'm not really sure if this is sane though. When we loose the 'Ax' caps (another client as accessed the directory we're encrypting), we also seem to loose the I_NEW state. Using the above code seems to work for the testcase in my patch, but I'm not sure it won't break something else.
Cheers,
--
Luís
Quoted 10 lines
If we fix it in
handle_cap_grant()I am afraid this bug still exists. What if there is no any new caps will be issued or revoked recently and then access to the directory ?Thanks
- Xiubo
Quoted 1 line
Quoted 100 lines · Luís Henriques
Quoted 88 lines · Xiubo Li
Quoted 60 lines · Luís Henriques
If a client doesn't have Fx caps on a directory, it will get errors while trying encrypt it:
ceph:
handle_cap_grant: cap grant attempt to changefscrypt_authon non-I_NEW inode (old len 0 new len 48) fscrypt (ceph, inode 1099511627812): Error -105 getting encryption contextA simple way to reproduce this is to use two clients:
client1 # mkdir
/mnt/mydirclient2 # ls
/mnt/mydirclient1 # fscrypt encrypt
/mnt/mydir
client1 # echo hello >/mnt/mydir/worldThis happens because, in
__ceph_setattr(), we only initializeci->fscrypt_authif we have Ax. If we don't have, we'll need to do that later, inhandle_cap_grant().Signed-off-by Luís Henriques <[email protected]>---
Hi!To be honest, I'm not really sure about the conditions in the 'if': shall I bother checking it's really a dir and that it is empty?
Cheers,
--
Luísfs/ceph/caps.c | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-)fs/ceph/caps.c +16
-3@@ -3511,9 +3511,29 @@ static void handle_cap_grant(struct inode *inode,from_kuid (&init_user_ns, inode->i_uid), from_kgid(&init_user_ns, inode->i_gid));#if IS_ENABLED (CONFIG_FS_ENCRYPTION ) -if (ci->fscrypt_auth_len != extra_info->fscrypt_auth_len || -memcmp (ci->fscrypt_auth, extra_info->fscrypt_auth, - ci->fscrypt_auth_len)) +if ((ci->fscrypt_auth_len ==0 ) && + (extra_info->fscrypt_auth_len >0 ) && +S_ISDIR (inode->i_mode) && + (ci->i_rsubdirs + ci->i_rfiles ==1 )) { +/* +* We'll get here when setting up an encrypted directory +* but we don't have Fx in that directory, i.e. other +* clients have accessed this directory too. +*/ + ci->fscrypt_auth =kmemdup (extra_info->fscrypt_auth, + extra_info->fscrypt_auth_len, +GFP_KERNEL ); +if (ci->fscrypt_auth) { + inode->i_flags |=S_ENCRYPTED ; + ci->fscrypt_auth_len = extra_info->fscrypt_auth_len; + }else {fs/ceph/caps.c static void handle_cap_grant(struct inode *inode, · line 3539
Quoted 10 lines
fs/ceph/caps.c +7
-0+pr_err ("Failed to alloc memory for %llx.%llx fscrypt_auth\n" , + ceph_vinop(inode)); + } +dout ("ino %llx.%llx is now encrypted\n" ,ceph_vinop (inode)); + }else if (ci->fscrypt_auth_len != extra_info->fscrypt_auth_len || +memcmp (ci->fscrypt_auth, extra_info->fscrypt_auth, + ci->fscrypt_auth_len))pr_warn_ratelimited ("%s: cap grant attempt to change fscrypt_auth on non-I_NEW inode (old len %d new len %d)\n" , __func__, ci->fscrypt_auth_len, extra_info->fscrypt_auth_len);#endif Hi Luis,
Thanks for your time on this bug.
IMO we should fix this in
ceph_fill_inode():995 #ifdef
CONFIG_FS_ENCRYPTION
996 if (iinfo->fscrypt_auth_len&& (inode->i_state&I_NEW)) { 997kfree(ci->fscrypt_auth);
998ci->fscrypt_auth_len=iinfo->fscrypt_auth_len; 999ci->fscrypt_auth=iinfo->fscrypt_auth;
1000iinfo->fscrypt_auth= NULL;
1001iinfo->fscrypt_auth_len= 0;
1002inode_set_flags(inode, S_ENCRYPTED, S_ENCRYPTED); 1003 }
1004 #endifQuoted 2 lines
The setattr will get a reply from MDS including the fscrypt auth info, I think the kclient just drop it here.
I've done some testing and I don't really see this code kfree'ing a valid
fscrypt_authhere. However, I guess it is possible to fix this issue here too, but in a different way, by changing that 'if' condition to:if (iinfo->fscrypt_auth_len && ((inode->i_state & I_NEW) || (ci->fscrypt_auth_len == 0))) { ... }
Quoted 4 lines
I'm not really sure if this is sane though. When we loose the 'Ax' caps (another client as accessed the directory we're encrypting), we also seem to loose the
I_NEWstate. Using the above code seems to work for the testcase in my patch, but I'm not sure it won't break something else.
It should be okay IMO.
The I_NEW is for new created directories, such as for mkdir request,etc. But currently the code didn't consider the setattr case.
Please send you patch let's check and discuss there.
Thanks!
- Xiubo
Quoted 12 lines
Cheers,
--
LuísQuoted 8 lines
If we fix it in
handle_cap_grant()I am afraid this bug still exists. What if there is no any new caps will be issued or revoked recently and then access to the directory ?Thanks
- Xiubo