fix quota subdir mounts
cover fix quota subdir mounts
Hi!
As reported recently in the ceph-users mailing-list[1], the kernel client behaves differently from the fuse client regarding mounting subdirs where quotas are in effect. I've also created a bug to track this issue[2].
The following patches are a possible way of fixing this issue. The performance impact should be close to zero if the mount is done in the CephFS root inode. When we're mounting subdirs, we may have extra queries to the MDSs, depending on how many extra realms we'll need to loop through.
Warning: this is just an RFC, and the patches are only lightly tested!
[1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-February/033357.html
[2] https://tracker.ceph.com/issues/38482
Luis Henriques (2):
ceph: factor out ceph_lookup_inode()
ceph: quota: fix quota subdir mounts
fs/ceph/caps.c | 2 +- fs/ceph/export.c | 14 +++++++++++++- fs/ceph/quota.c | 30 +++++++++++++++++++++++++++--- fs/ceph/snap.c | 3 +++ fs/ceph/super.h | 3 +++ 5 files changed, 47 insertions(+), 5 deletions(-)
RFC PATCH 1/2 ceph: factor out ceph_lookup_inode()
This function will be used by __fh_to_dentry and by the quotas code, to find quota realm inodes that are not visible in the mountpoint. The only functional change is that an error is also returned if ceph_mdsc_do_request() fails.
--- fs/ceph/export.c | 14 +++++++++++++- fs/ceph/super.h | 1 + 2 files changed, 14 insertions(+), 1 deletion(-)
fs/ceph/export.c +13 -1
@@ -59,7 +59,7 @@ static int ceph_encode_fh(struct inode *inode, u32 *rawfh, int *max_len,return type; } -static struct dentry *__fh_to_dentry (struct super_block *sb,u64 ino) +struct inode *ceph_lookup_inode (struct super_block *sb,u64 ino) {struct ceph_mds_client *mdsc =ceph_sb_to_client (sb)->mdsc;struct inode *inode;
@@ -92,12 +92,24 @@ static struct dentry *__fh_to_dentry(struct super_block *sb, u64 ino)ceph_mdsc_put_request (req);if (!inode)return ERR_PTR (-ESTALE ); +if (err) +return ERR_PTR (err);if (inode->i_nlink ==0 ) {iput (inode);return ERR_PTR (-ESTALE ); } } +return inode; +} + +static struct dentry *__fh_to_dentry (struct super_block *sb,u64 ino) +{ +struct inode *inode =ceph_lookup_inode (sb, ino); + +if (IS_ERR (inode)) +return ERR_CAST (inode); +return d_obtain_alias (inode); }
fs/ceph/super.h +1 -0
@@ -1061,6 +1061,7 @@ extern long ceph_ioctl(struct file *file, unsigned int cmd, unsigned long arg);/* export.c */ extern const struct export_operations ceph_export_ops; +struct inode *ceph_lookup_inode (struct super_block *sb,u64 ino);/* locks.c */ extern __init void ceph_flock_init (void );
RFC PATCH 2/2 ceph: quota: fix quota subdir mounts
In reply to Luis HenriquesThe CephFS kernel client doesn't enforce quotas that are set in a directory that isn't visible in the mount point. For example, given the path '/dir1/dir2', if quotas are set in 'dir1' and the mount is done in with
mount -t ceph <server>:<port>:/dir1/ /mnt
then the client can't access the 'dir1' inode from the quota realm dir2 belongs to.
This patch fixes this by simply doing an MDS LOOKUPINO Op and grabbing a reference to it (so that it doesn't disappear again). This also requires an extra field in ceph_snap_realm so that we know we have to release that reference when destroying the realm.
Links: https://tracker.ceph.com/issues/3848
--- fs/ceph/caps.c | 2 +- fs/ceph/quota.c | 30 +++++++++++++++++++++++++++--- fs/ceph/snap.c | 3 +++ fs/ceph/super.h | 2 ++ 4 files changed, 33 insertions(+), 4 deletions(-)
fs/ceph/caps.c +1 -1
@@ -1035,7 +1035,7 @@ static void drop_inode_snap_realm(struct ceph_inode_info *ci)list_del_init (&ci->i_snap_realm_item); ci->i_snap_realm_counter++; ci->i_snap_realm =NULL ; -if (realm->ino == ci->i_vino.ino) +if ((realm->ino == ci->i_vino.ino) && !realm->own_inode) realm->inode =NULL ;spin_unlock (&realm->inodes_with_caps_lock); ceph_put_snap_realm(ceph_sb_to_client(ci->vfs_inode.i_sb)->mdsc,
fs/ceph/quota.c +27 -3
@@ -22,7 +22,16 @@ void ceph_adjust_quota_realms_count(struct inode *inode, bool inc)static inline bool ceph_has_realms_with_quotas (struct inode *inode) {struct ceph_mds_client *mdsc =ceph_inode_to_client (inode)->mdsc; -return atomic64_read (&mdsc->quotarealms_count) >0 ; +struct super_block *sb = mdsc->fsc->sb; + +if (atomic64_read (&mdsc->quotarealms_count) >0 ) +return true; +/* if root is the real CephFS root, we don't have quota realms */ +if (sb->s_root->d_inode && + (sb->s_root->d_inode->i_ino ==CEPH_INO_ROOT )) +return false; +/* otherwise, we can't know for sure */ +return true; } void ceph_handle_quota(struct ceph_mds_client *mdsc,
@@ -166,6 +175,7 @@ static bool check_quota_exceeded(struct inode *inode, enum quota_check_op op,return false;down_read (&mdsc->snap_rwsem); +restart: realm =ceph_inode (inode)->i_snap_realm;if (realm)ceph_get_snap_realm (mdsc, realm);
@@ -176,8 +186,22 @@ static bool check_quota_exceeded(struct inode *inode, enum quota_check_op op,spin_lock (&realm->inodes_with_caps_lock); in = realm->inode ?igrab (realm->inode) :NULL ;spin_unlock (&realm->inodes_with_caps_lock); -if (!in) -break ; +if (!in) { +up_read (&mdsc->snap_rwsem); + in =ceph_lookup_inode (inode->i_sb, realm->ino); +down_read (&mdsc->snap_rwsem); +if (IS_ERR (in)) { +pr_warn ("Can't lookup inode %llx (err: %ld)\n" , + realm->ino,PTR_ERR (in)); +break ; + } +spin_lock (&realm->inodes_with_caps_lock); + realm->inode = in; + realm->own_inode = true; +spin_unlock (&realm->inodes_with_caps_lock); +ceph_put_snap_realm (mdsc, realm); + goto restart; + } ci =ceph_inode (in);spin_lock (&ci->i_ceph_lock);
fs/ceph/snap.c +3 -0
@@ -117,6 +117,7 @@ static struct ceph_snap_realm *ceph_create_snap_realm(atomic_set (&realm->nref,1 );/* for caller */ realm->ino = ino; + realm->own_inode = false;INIT_LIST_HEAD (&realm->children);INIT_LIST_HEAD (&realm->child_item);INIT_LIST_HEAD (&realm->empty_item);
@@ -184,6 +185,8 @@ static void __destroy_snap_realm(struct ceph_mds_client *mdsc,kfree (realm->prior_parent_snaps);kfree (realm->snaps);ceph_put_snap_context (realm->cached_context); +if (realm->own_inode && realm->inode) +iput (realm->inode);kfree (realm); }
fs/ceph/super.h +2 -0
@@ -764,6 +764,8 @@ struct ceph_snap_realm {atomic_t nref;struct rb_node node; +bool own_inode;/* true if we hold a ref to the inode */ +u64 created, seq;u64 parent_ino;u64 parent_since;/* snapid when our current parent became so */
Re: [RFC PATCH 2/2] ceph: quota: fix quota subdir mounts
Quoted 9 lines · Luis Henriques
The CephFS kernel client doesn't enforce quotas that are set in a directory that isn't visible in the mount point. For example, given the path '/dir1/dir2', if quotas are set in 'dir1' and the mount is done in with
mount -t ceph <server>:<port>:/dir1/ /mnt
then the client can't access the 'dir1' inode from the quota realm dir2 belongs to.
Quoted 6 lines
This patch fixes this by simply doing an MDS LOOKUPINO Op and grabbing a reference to it (so that it doesn't disappear again). This also requires an extra field in
ceph_snap_realmso that we know we have to release that reference when destroying the realm.
This bug looks unrelated to the patch description. Are you sure it's correct?
Quoted 111 lines
Reported-by Hendrik Peyerl <[email protected]>Signed-off-by Luis Henriques <[email protected]>--- fs/ceph/caps.c | 2 +- fs/ceph/quota.c | 30 +++++++++++++++++++++++++++--- fs/ceph/snap.c | 3 +++ fs/ceph/super.h | 2 ++ 4 files changed, 33 insertions(+), 4 deletions(-)fs/ceph/caps.c +1
-1@@ -1035,7 +1035,7 @@ static void drop_inode_snap_realm(struct ceph_inode_info *ci)list_del_init (&ci->i_snap_realm_item); ci->i_snap_realm_counter++; ci->i_snap_realm =NULL ; -if (realm->ino == ci->i_vino.ino) +if ((realm->ino == ci->i_vino.ino) && !realm->own_inode) realm->inode =NULL ;spin_unlock (&realm->inodes_with_caps_lock); ceph_put_snap_realm(ceph_sb_to_client(ci->vfs_inode.i_sb)->mdsc,fs/ceph/quota.c +27
-3@@ -22,7 +22,16 @@ void ceph_adjust_quota_realms_count(struct inode *inode, bool inc)static inline bool ceph_has_realms_with_quotas (struct inode *inode) {struct ceph_mds_client *mdsc =ceph_inode_to_client (inode)->mdsc; -return atomic64_read (&mdsc->quotarealms_count) >0 ; +struct super_block *sb = mdsc->fsc->sb; + +if (atomic64_read (&mdsc->quotarealms_count) >0 ) +return true; +/* if root is the real CephFS root, we don't have quota realms */ +if (sb->s_root->d_inode && + (sb->s_root->d_inode->i_ino ==CEPH_INO_ROOT )) +return false; +/* otherwise, we can't know for sure */ +return true; } void ceph_handle_quota(struct ceph_mds_client *mdsc,@@ -166,6 +175,7 @@ static bool check_quota_exceeded(struct inode *inode, enum quota_check_op op,return false;down_read (&mdsc->snap_rwsem); +restart: realm =ceph_inode (inode)->i_snap_realm;if (realm)ceph_get_snap_realm (mdsc, realm);@@ -176,8 +186,22 @@ static bool check_quota_exceeded(struct inode *inode, enum quota_check_op op,spin_lock (&realm->inodes_with_caps_lock); in = realm->inode ?igrab (realm->inode) :NULL ;spin_unlock (&realm->inodes_with_caps_lock); -if (!in) -break ; +if (!in) { +up_read (&mdsc->snap_rwsem); + in =ceph_lookup_inode (inode->i_sb, realm->ino); +down_read (&mdsc->snap_rwsem); +if (IS_ERR (in)) { +pr_warn ("Can't lookup inode %llx (err: %ld)\n" , + realm->ino,PTR_ERR (in)); +break ; + } +spin_lock (&realm->inodes_with_caps_lock); + realm->inode = in; + realm->own_inode = true; +spin_unlock (&realm->inodes_with_caps_lock); +ceph_put_snap_realm (mdsc, realm); + goto restart; + } ci =ceph_inode (in);spin_lock (&ci->i_ceph_lock);fs/ceph/snap.c +3
-0@@ -117,6 +117,7 @@ static struct ceph_snap_realm *ceph_create_snap_realm(atomic_set (&realm->nref,1 );/* for caller */ realm->ino = ino; + realm->own_inode = false;INIT_LIST_HEAD (&realm->children);INIT_LIST_HEAD (&realm->child_item);INIT_LIST_HEAD (&realm->empty_item);@@ -184,6 +185,8 @@ static void __destroy_snap_realm(struct ceph_mds_client *mdsc,kfree (realm->prior_parent_snaps);kfree (realm->snaps);ceph_put_snap_context (realm->cached_context); +if (realm->own_inode && realm->inode) +iput (realm->inode);kfree (realm); }fs/ceph/super.h +2
-0@@ -764,6 +764,8 @@ struct ceph_snap_realm {atomic_t nref;struct rb_node node; +bool own_inode;/* true if we hold a ref to the inode */ +u64 created, seq;u64 parent_ino;u64 parent_since;/* snapid when our current parent became so */
Signature · 1 line
Jeff Layton <[email protected]>
Re: [RFC PATCH 2/2] ceph: quota: fix quota subdir mounts
Quoted 15 lines · Jeff Layton
Quoted 14 lines · Luis Henriques
The CephFS kernel client doesn't enforce quotas that are set in a directory that isn't visible in the mount point. For example, given the path '/dir1/dir2', if quotas are set in 'dir1' and the mount is done in with
mount -t ceph <server>:<port>:/dir1/ /mnt
then the client can't access the 'dir1' inode from the quota realm dir2 belongs to.
This patch fixes this by simply doing an MDS LOOKUPINO Op and grabbing a reference to it (so that it doesn't disappear again). This also requires an extra field in
ceph_snap_realmso that we know we have to release that reference when destroying the realm.
Quoted 4 lines
Quoted 1 line
This bug looks unrelated to the patch description. Are you sure it's correct?
Ups! Looks like I truncated the bug number. Sorry about that, here's the correct link:
https://tracker.ceph.com/issues/38482
Cheers,
Signature · 1 line
Luis
Quoted 111 lines
Quoted 111 lines
Reported-by Hendrik Peyerl <[email protected]>Signed-off-by Luis Henriques <[email protected]>--- fs/ceph/caps.c | 2 +- fs/ceph/quota.c | 30 +++++++++++++++++++++++++++--- fs/ceph/snap.c | 3 +++ fs/ceph/super.h | 2 ++ 4 files changed, 33 insertions(+), 4 deletions(-)fs/ceph/caps.c +1
-1@@ -1035,7 +1035,7 @@ static void drop_inode_snap_realm(struct ceph_inode_info *ci)list_del_init (&ci->i_snap_realm_item); ci->i_snap_realm_counter++; ci->i_snap_realm =NULL ; -if (realm->ino == ci->i_vino.ino) +if ((realm->ino == ci->i_vino.ino) && !realm->own_inode) realm->inode =NULL ;spin_unlock (&realm->inodes_with_caps_lock); ceph_put_snap_realm(ceph_sb_to_client(ci->vfs_inode.i_sb)->mdsc,fs/ceph/quota.c +27
-3@@ -22,7 +22,16 @@ void ceph_adjust_quota_realms_count(struct inode *inode, bool inc)static inline bool ceph_has_realms_with_quotas (struct inode *inode) {struct ceph_mds_client *mdsc =ceph_inode_to_client (inode)->mdsc; -return atomic64_read (&mdsc->quotarealms_count) >0 ; +struct super_block *sb = mdsc->fsc->sb; + +if (atomic64_read (&mdsc->quotarealms_count) >0 ) +return true; +/* if root is the real CephFS root, we don't have quota realms */ +if (sb->s_root->d_inode && + (sb->s_root->d_inode->i_ino ==CEPH_INO_ROOT )) +return false; +/* otherwise, we can't know for sure */ +return true; } void ceph_handle_quota(struct ceph_mds_client *mdsc,@@ -166,6 +175,7 @@ static bool check_quota_exceeded(struct inode *inode, enum quota_check_op op,return false;down_read (&mdsc->snap_rwsem); +restart: realm =ceph_inode (inode)->i_snap_realm;if (realm)ceph_get_snap_realm (mdsc, realm);@@ -176,8 +186,22 @@ static bool check_quota_exceeded(struct inode *inode, enum quota_check_op op,spin_lock (&realm->inodes_with_caps_lock); in = realm->inode ?igrab (realm->inode) :NULL ;spin_unlock (&realm->inodes_with_caps_lock); -if (!in) -break ; +if (!in) { +up_read (&mdsc->snap_rwsem); + in =ceph_lookup_inode (inode->i_sb, realm->ino); +down_read (&mdsc->snap_rwsem); +if (IS_ERR (in)) { +pr_warn ("Can't lookup inode %llx (err: %ld)\n" , + realm->ino,PTR_ERR (in)); +break ; + } +spin_lock (&realm->inodes_with_caps_lock); + realm->inode = in; + realm->own_inode = true; +spin_unlock (&realm->inodes_with_caps_lock); +ceph_put_snap_realm (mdsc, realm); + goto restart; + } ci =ceph_inode (in);spin_lock (&ci->i_ceph_lock);fs/ceph/snap.c +3
-0@@ -117,6 +117,7 @@ static struct ceph_snap_realm *ceph_create_snap_realm(atomic_set (&realm->nref,1 );/* for caller */ realm->ino = ino; + realm->own_inode = false;INIT_LIST_HEAD (&realm->children);INIT_LIST_HEAD (&realm->child_item);INIT_LIST_HEAD (&realm->empty_item);@@ -184,6 +185,8 @@ static void __destroy_snap_realm(struct ceph_mds_client *mdsc,kfree (realm->prior_parent_snaps);kfree (realm->snaps);ceph_put_snap_context (realm->cached_context); +if (realm->own_inode && realm->inode) +iput (realm->inode);kfree (realm); }fs/ceph/super.h +2
-0@@ -764,6 +764,8 @@ struct ceph_snap_realm {atomic_t nref;struct rb_node node; +bool own_inode;/* true if we hold a ref to the inode */ +u64 created, seq;u64 parent_ino;u64 parent_since;/* snapid when our current parent became so */
Re: [RFC PATCH 2/2] ceph: quota: fix quota subdir mounts
In reply to Luis HenriquesQuoted 7 lines · Luis Henriques
The CephFS kernel client doesn't enforce quotas that are set in a directory that isn't visible in the mount point. For example, given the path '/dir1/dir2', if quotas are set in 'dir1' and the mount is done in with
mount -t ceph <server>:<port>:/dir1/ /mnt
Quoted 8 lines
then the client can't access the 'dir1' inode from the quota realm dir2 belongs to.
This patch fixes this by simply doing an MDS LOOKUPINO Op and grabbing a reference to it (so that it doesn't disappear again). This also requires an extra field in
ceph_snap_realmso that we know we have to release that reference when destroying the realm.
This may cause circle reference if somehow an inode owned by snaprealm get moved into mount subdir (other clients do rename). how about holding these inodes in mds_client?
Quoted 112 lines
Links: https://tracker.ceph.com/issues/3848
Reported-by Hendrik Peyerl <[email protected]>Signed-off-by Luis Henriques <[email protected]>--- fs/ceph/caps.c | 2 +- fs/ceph/quota.c | 30 +++++++++++++++++++++++++++--- fs/ceph/snap.c | 3 +++ fs/ceph/super.h | 2 ++ 4 files changed, 33 insertions(+), 4 deletions(-)fs/ceph/caps.c +1
-1@@ -1035,7 +1035,7 @@ static void drop_inode_snap_realm(struct ceph_inode_info *ci)list_del_init (&ci->i_snap_realm_item); ci->i_snap_realm_counter++; ci->i_snap_realm =NULL ; -if (realm->ino == ci->i_vino.ino) +if ((realm->ino == ci->i_vino.ino) && !realm->own_inode) realm->inode =NULL ;spin_unlock (&realm->inodes_with_caps_lock); ceph_put_snap_realm(ceph_sb_to_client(ci->vfs_inode.i_sb)->mdsc,fs/ceph/quota.c +27
-3@@ -22,7 +22,16 @@ void ceph_adjust_quota_realms_count(struct inode *inode, bool inc)static inline bool ceph_has_realms_with_quotas (struct inode *inode) {struct ceph_mds_client *mdsc =ceph_inode_to_client (inode)->mdsc; -return atomic64_read (&mdsc->quotarealms_count) >0 ; +struct super_block *sb = mdsc->fsc->sb; + +if (atomic64_read (&mdsc->quotarealms_count) >0 ) +return true; +/* if root is the real CephFS root, we don't have quota realms */ +if (sb->s_root->d_inode && + (sb->s_root->d_inode->i_ino ==CEPH_INO_ROOT )) +return false; +/* otherwise, we can't know for sure */ +return true; } void ceph_handle_quota(struct ceph_mds_client *mdsc,@@ -166,6 +175,7 @@ static bool check_quota_exceeded(struct inode *inode, enum quota_check_op op,return false;down_read (&mdsc->snap_rwsem); +restart: realm =ceph_inode (inode)->i_snap_realm;if (realm)ceph_get_snap_realm (mdsc, realm);@@ -176,8 +186,22 @@ static bool check_quota_exceeded(struct inode *inode, enum quota_check_op op,spin_lock (&realm->inodes_with_caps_lock); in = realm->inode ?igrab (realm->inode) :NULL ;spin_unlock (&realm->inodes_with_caps_lock); -if (!in) -break ; +if (!in) { +up_read (&mdsc->snap_rwsem); + in =ceph_lookup_inode (inode->i_sb, realm->ino); +down_read (&mdsc->snap_rwsem); +if (IS_ERR (in)) { +pr_warn ("Can't lookup inode %llx (err: %ld)\n" , + realm->ino,PTR_ERR (in)); +break ; + } +spin_lock (&realm->inodes_with_caps_lock); + realm->inode = in; + realm->own_inode = true; +spin_unlock (&realm->inodes_with_caps_lock); +ceph_put_snap_realm (mdsc, realm); + goto restart; + } ci =ceph_inode (in);spin_lock (&ci->i_ceph_lock);fs/ceph/snap.c +3
-0@@ -117,6 +117,7 @@ static struct ceph_snap_realm *ceph_create_snap_realm(atomic_set (&realm->nref,1 );/* for caller */ realm->ino = ino; + realm->own_inode = false;INIT_LIST_HEAD (&realm->children);INIT_LIST_HEAD (&realm->child_item);INIT_LIST_HEAD (&realm->empty_item);@@ -184,6 +185,8 @@ static void __destroy_snap_realm(struct ceph_mds_client *mdsc,kfree (realm->prior_parent_snaps);kfree (realm->snaps);ceph_put_snap_context (realm->cached_context); +if (realm->own_inode && realm->inode) +iput (realm->inode);kfree (realm); }fs/ceph/super.h +2
-0@@ -764,6 +764,8 @@ struct ceph_snap_realm {atomic_t nref;struct rb_node node; +bool own_inode;/* true if we hold a ref to the inode */ +u64 created, seq;u64 parent_ino;u64 parent_since;/* snapid when our current parent became so */
Re: [RFC PATCH 2/2] ceph: quota: fix quota subdir mounts
Quoted 17 lines · Zheng
Quoted 7 lines · Luis Henriques
The CephFS kernel client doesn't enforce quotas that are set in a directory that isn't visible in the mount point. For example, given the path '/dir1/dir2', if quotas are set in 'dir1' and the mount is done in with
mount -t ceph <server>:<port>:/dir1/ /mnt
Quoted 8 lines
then the client can't access the 'dir1' inode from the quota realm dir2 belongs to.
This patch fixes this by simply doing an MDS LOOKUPINO Op and grabbing a reference to it (so that it doesn't disappear again). This also requires an extra field in
ceph_snap_realmso that we know we have to release that reference when destroying the realm.
Quoted 3 lines
This may cause circle reference if somehow an inode owned by snaprealm get moved into mount subdir (other clients do rename). how about holding these inodes in
mds_client?
It seems to make sense. But this means that this inode reference will live until the filesystem is umounted. And what if another client deletes that inode? Will we know about that? /me needs to think about that a bit more.
Cheers,
Signature · 1 line
Luis
Quoted 114 lines
Quoted 113 lines
Links: https://tracker.ceph.com/issues/3848
Reported-by Hendrik Peyerl <[email protected]>Signed-off-by Luis Henriques <[email protected]>--- fs/ceph/caps.c | 2 +- fs/ceph/quota.c | 30 +++++++++++++++++++++++++++--- fs/ceph/snap.c | 3 +++ fs/ceph/super.h | 2 ++ 4 files changed, 33 insertions(+), 4 deletions(-)fs/ceph/caps.c +1
-1@@ -1035,7 +1035,7 @@ static void drop_inode_snap_realm(struct ceph_inode_info *ci)list_del_init (&ci->i_snap_realm_item); ci->i_snap_realm_counter++; ci->i_snap_realm =NULL ; -if (realm->ino == ci->i_vino.ino) +if ((realm->ino == ci->i_vino.ino) && !realm->own_inode) realm->inode =NULL ;spin_unlock (&realm->inodes_with_caps_lock); ceph_put_snap_realm(ceph_sb_to_client(ci->vfs_inode.i_sb)->mdsc,fs/ceph/quota.c +27
-3@@ -22,7 +22,16 @@ void ceph_adjust_quota_realms_count(struct inode *inode, bool inc)static inline bool ceph_has_realms_with_quotas (struct inode *inode) {struct ceph_mds_client *mdsc =ceph_inode_to_client (inode)->mdsc; -return atomic64_read (&mdsc->quotarealms_count) >0 ; +struct super_block *sb = mdsc->fsc->sb; + +if (atomic64_read (&mdsc->quotarealms_count) >0 ) +return true; +/* if root is the real CephFS root, we don't have quota realms */ +if (sb->s_root->d_inode && + (sb->s_root->d_inode->i_ino ==CEPH_INO_ROOT )) +return false; +/* otherwise, we can't know for sure */ +return true; } void ceph_handle_quota(struct ceph_mds_client *mdsc,@@ -166,6 +175,7 @@ static bool check_quota_exceeded(struct inode *inode, enum quota_check_op op,return false;down_read (&mdsc->snap_rwsem); +restart: realm =ceph_inode (inode)->i_snap_realm;if (realm)ceph_get_snap_realm (mdsc, realm);@@ -176,8 +186,22 @@ static bool check_quota_exceeded(struct inode *inode, enum quota_check_op op,spin_lock (&realm->inodes_with_caps_lock); in = realm->inode ?igrab (realm->inode) :NULL ;spin_unlock (&realm->inodes_with_caps_lock); -if (!in) -break ; +if (!in) { +up_read (&mdsc->snap_rwsem); + in =ceph_lookup_inode (inode->i_sb, realm->ino); +down_read (&mdsc->snap_rwsem); +if (IS_ERR (in)) { +pr_warn ("Can't lookup inode %llx (err: %ld)\n" , + realm->ino,PTR_ERR (in)); +break ; + } +spin_lock (&realm->inodes_with_caps_lock); + realm->inode = in; + realm->own_inode = true; +spin_unlock (&realm->inodes_with_caps_lock); +ceph_put_snap_realm (mdsc, realm); + goto restart; + } ci =ceph_inode (in);spin_lock (&ci->i_ceph_lock);fs/ceph/snap.c +3
-0@@ -117,6 +117,7 @@ static struct ceph_snap_realm *ceph_create_snap_realm(atomic_set (&realm->nref,1 );/* for caller */ realm->ino = ino; + realm->own_inode = false;INIT_LIST_HEAD (&realm->children);INIT_LIST_HEAD (&realm->child_item);INIT_LIST_HEAD (&realm->empty_item);@@ -184,6 +185,8 @@ static void __destroy_snap_realm(struct ceph_mds_client *mdsc,kfree (realm->prior_parent_snaps);kfree (realm->snaps);ceph_put_snap_context (realm->cached_context); +if (realm->own_inode && realm->inode) +iput (realm->inode);kfree (realm); }fs/ceph/super.h +2
-0@@ -764,6 +764,8 @@ struct ceph_snap_realm {atomic_t nref;struct rb_node node; +bool own_inode;/* true if we hold a ref to the inode */ +u64 created, seq;u64 parent_ino;u64 parent_since;/* snapid when our current parent became so */
Re: [RFC PATCH 2/2] ceph: quota: fix quota subdir mounts
In reply to Yan, ZhengQuoted 17 lines · Zheng
Quoted 7 lines · Luis Henriques
The CephFS kernel client doesn't enforce quotas that are set in a directory that isn't visible in the mount point. For example, given the path '/dir1/dir2', if quotas are set in 'dir1' and the mount is done in with
mount -t ceph <server>:<port>:/dir1/ /mnt
Quoted 8 lines
then the client can't access the 'dir1' inode from the quota realm dir2 belongs to.
This patch fixes this by simply doing an MDS LOOKUPINO Op and grabbing a reference to it (so that it doesn't disappear again). This also requires an extra field in
ceph_snap_realmso that we know we have to release that reference when destroying the realm.
Quoted 3 lines
This may cause circle reference if somehow an inode owned by snaprealm get moved into mount subdir (other clients do rename). how about holding these inodes in
mds_client?
Ok, before proceeded any further I wanted to make sure that what you were suggesting was something like the patch below. It simply keeps a list of inodes in ceph_mds_client until the filesystem is umounted, iput()ing them at that point.
I'm sure I'm missing another place where the reference should be dropped, but I couldn't figure it out yet. It can't be ceph_destroy_inode; drop_inode_snap_realm is a possibility, but what if the inode becomes visible in the meantime? Well, I'll continue thinking about it.
Function get_quota_realm() should have a similar construct to lookup inodes. But it's a bit more tricky, because ceph_quota_is_same_realm() requires snap_rwsem to be held for the 2 invocations of get_quota_realm.
Cheers,
Signature · 14 lines
Luis From a429a4c167186781bd235a25d72be893baf9e029 Mon Sep 17 00:00:00 2001 From: Luis Henriques <[email protected]> Date: Wed, 6 Mar 2019 17:58:04 +0000 Subject: [PATCH] ceph: quota: fix quota subdir mounts (II) Signed-off-by: Luis Henriques <[email protected]> --- fs/ceph/mds_client.c | 14 ++++++++++++++ fs/ceph/mds_client.h | 2 ++ fs/ceph/quota.c | 34 ++++++++++++++++++++++++++++++---- fs/ceph/super.h | 2 ++ 4 files changed, 48 insertions(+), 4 deletions(-)
fs/ceph/mds_client.c +14 -0
@@ -3656,6 +3656,8 @@ int ceph_mdsc_init(struct ceph_fs_client *fsc) mdsc->max_sessions =0 ; mdsc->stopping =0 ;atomic64_set (&mdsc->quotarealms_count,0 ); +INIT_LIST_HEAD (&mdsc->quotarealms_inodes_list); +spin_lock_init (&mdsc->quotarealms_inodes_lock); mdsc->last_snap_seq =0 ;init_rwsem (&mdsc->snap_rwsem); mdsc->snap_realms =RB_ROOT ;
@@ -3726,9 +3728,21 @@ static void wait_requests(struct ceph_mds_client *mdsc)*/ void ceph_mdsc_pre_umount (struct ceph_mds_client *mdsc) { +struct ceph_inode_info *ci; +dout ("pre_umount\n" ); mdsc->stopping =1 ; +spin_lock (&mdsc->quotarealms_inodes_lock); +while (!list_empty (&mdsc->quotarealms_inodes_list)) { + ci =list_first_entry (&mdsc->quotarealms_inodes_list, + struct ceph_inode_info, + i_quotarealms_inode_item); +list_del (&ci->i_quotarealms_inode_item); +iput (&ci->vfs_inode); + } +spin_unlock (&mdsc->quotarealms_inodes_lock); +lock_unlock_sessions (mdsc);ceph_flush_dirty_caps (mdsc);wait_requests (mdsc);
fs/ceph/mds_client.h +2 -0
@@ -329,6 +329,8 @@ struct ceph_mds_client {int stopping;/* true if shutting down */ atomic64_t quotarealms_count;/* # realms with quota */ +struct list_head quotarealms_inodes_list; +spinlock_t quotarealms_inodes_lock;/* * snap_rwsem will cover cap linkage into snaprealms, and
fs/ceph/quota.c +30 -4
@@ -22,7 +22,16 @@ void ceph_adjust_quota_realms_count(struct inode *inode, bool inc)static inline bool ceph_has_realms_with_quotas (struct inode *inode) {struct ceph_mds_client *mdsc =ceph_inode_to_client (inode)->mdsc; -return atomic64_read (&mdsc->quotarealms_count) >0 ; +struct super_block *sb = mdsc->fsc->sb; + +if (atomic64_read (&mdsc->quotarealms_count) >0 ) +return true; +/* if root is the real CephFS root, we don't have quota realms */ +if (sb->s_root->d_inode && + (sb->s_root->d_inode->i_ino ==CEPH_INO_ROOT )) +return false; +/* otherwise, we can't know for sure */ +return true; } void ceph_handle_quota(struct ceph_mds_client *mdsc,
@@ -166,6 +175,7 @@ static bool check_quota_exceeded(struct inode *inode, enum quota_check_op op,return false;down_read (&mdsc->snap_rwsem); +restart: realm =ceph_inode (inode)->i_snap_realm;if (realm)ceph_get_snap_realm (mdsc, realm);
@@ -176,9 +186,25 @@ static bool check_quota_exceeded(struct inode *inode, enum quota_check_op op,spin_lock (&realm->inodes_with_caps_lock); in = realm->inode ?igrab (realm->inode) :NULL ;spin_unlock (&realm->inodes_with_caps_lock); -if (!in) -break ; - +if (!in) { +up_read (&mdsc->snap_rwsem); + in =ceph_lookup_inode (inode->i_sb, realm->ino); +down_read (&mdsc->snap_rwsem); +if (IS_ERR (in)) { +pr_warn ("Can't lookup inode %llx (err: %ld)\n" , + realm->ino,PTR_ERR (in)); +break ; + } +spin_lock (&mdsc->quotarealms_inodes_lock); +list_add (&ceph_inode (in)->i_quotarealms_inode_item, + &mdsc->quotarealms_inodes_list); +spin_unlock (&mdsc->quotarealms_inodes_lock); +spin_lock (&realm->inodes_with_caps_lock); + realm->inode = in; +spin_unlock (&realm->inodes_with_caps_lock); +ceph_put_snap_realm (mdsc, realm); + goto restart; + } ci =ceph_inode (in);spin_lock (&ci->i_ceph_lock);if (op ==QUOTA_CHECK_MAX_FILES_OP ) {
fs/ceph/super.h +2 -0
@@ -375,6 +375,8 @@ struct ceph_inode_info {struct list_head i_snap_realm_item;struct list_head i_snap_flush_item; +struct list_head i_quotarealms_inode_item; +struct work_struct i_wb_work;/* writeback work */ struct work_struct i_pg_inv_work;/* page invalidation work */
Re: [RFC PATCH 2/2] ceph: quota: fix quota subdir mounts
Quoted 20 lines · Luis Henriques
Quoted 17 lines · Zheng
Quoted 15 lines · Luis Henriques
The CephFS kernel client doesn't enforce quotas that are set in a directory that isn't visible in the mount point. For example, given the path '/dir1/dir2', if quotas are set in 'dir1' and the mount is done in with
mount -t ceph <server>:<port>:/dir1/ /mnt
then the client can't access the 'dir1' inode from the quota realm dir2 belongs to.
This patch fixes this by simply doing an MDS LOOKUPINO Op and grabbing a reference to it (so that it doesn't disappear again). This also requires an extra field in
ceph_snap_realmso that we know we have to release that reference when destroying the realm.
Quoted 9 lines
Quoted 3 lines
This may cause circle reference if somehow an inode owned by snaprealm get moved into mount subdir (other clients do rename). how about holding these inodes in
mds_client?Ok, before proceeded any further I wanted to make sure that what you were suggesting was something like the patch below. It simply keeps a list of inodes in
ceph_mds_clientuntil the filesystem is umounted, iput()ing them at that point.
yes,
Quoted 5 lines
I'm sure I'm missing another place where the reference should be dropped, but I couldn't figure it out yet. It can't be
ceph_destroy_inode;drop_inode_snap_realmis a possibility, but what if the inode becomes visible in the meantime? Well, I'll continue thinking about it.
why do you think we need to clean up the references at other place. what problem you encountered.
Regards
Yan, Zheng
Quoted 143 lines
Function
get_quota_realm()should have a similar construct to lookup inodes. But it's a bit more tricky, becauseceph_quota_is_same_realm()requiressnap_rwsemto be held for the 2 invocations ofget_quota_realm.Cheers,
--
LuisFrom
a429a4c167186781bd235a25d72be893baf9e029Mon Sep 17 00:00:00 2001
From: Luis Henriques <[email protected]>
Date: Wed, 6 Mar 2019 17:58:04 +0000
Subject: [PATCH] ceph: quota: fix quota subdir mounts (II)Signed-off-by Luis Henriques <[email protected]>--- fs/ceph/mds_client.c | 14 ++++++++++++++ fs/ceph/mds_client.h | 2 ++ fs/ceph/quota.c | 34 ++++++++++++++++++++++++++++++---- fs/ceph/super.h | 2 ++ 4 files changed, 48 insertions(+), 4 deletions(-)fs/ceph/mds_client.c +14
-0@@ -3656,6 +3656,8 @@ int ceph_mdsc_init(struct ceph_fs_client *fsc) mdsc->max_sessions =0 ; mdsc->stopping =0 ;atomic64_set (&mdsc->quotarealms_count,0 ); +INIT_LIST_HEAD (&mdsc->quotarealms_inodes_list); +spin_lock_init (&mdsc->quotarealms_inodes_lock); mdsc->last_snap_seq =0 ;init_rwsem (&mdsc->snap_rwsem); mdsc->snap_realms =RB_ROOT ;@@ -3726,9 +3728,21 @@ static void wait_requests(struct ceph_mds_client *mdsc)*/ void ceph_mdsc_pre_umount (struct ceph_mds_client *mdsc) { +struct ceph_inode_info *ci; +dout ("pre_umount\n" ); mdsc->stopping =1 ; +spin_lock (&mdsc->quotarealms_inodes_lock); +while (!list_empty (&mdsc->quotarealms_inodes_list)) { + ci =list_first_entry (&mdsc->quotarealms_inodes_list, + struct ceph_inode_info, + i_quotarealms_inode_item); +list_del (&ci->i_quotarealms_inode_item); +iput (&ci->vfs_inode); + } +spin_unlock (&mdsc->quotarealms_inodes_lock); +lock_unlock_sessions (mdsc);ceph_flush_dirty_caps (mdsc);wait_requests (mdsc);fs/ceph/mds_client.h +2
-0@@ -329,6 +329,8 @@ struct ceph_mds_client {int stopping;/* true if shutting down */ atomic64_t quotarealms_count;/* # realms with quota */ +struct list_head quotarealms_inodes_list; +spinlock_t quotarealms_inodes_lock;/* * snap_rwsem will cover cap linkage into snaprealms, and fs/ceph/quota.c +30
-4@@ -22,7 +22,16 @@ void ceph_adjust_quota_realms_count(struct inode *inode, bool inc)static inline bool ceph_has_realms_with_quotas (struct inode *inode) {struct ceph_mds_client *mdsc =ceph_inode_to_client (inode)->mdsc; -return atomic64_read (&mdsc->quotarealms_count) >0 ; +struct super_block *sb = mdsc->fsc->sb; + +if (atomic64_read (&mdsc->quotarealms_count) >0 ) +return true; +/* if root is the real CephFS root, we don't have quota realms */ +if (sb->s_root->d_inode && + (sb->s_root->d_inode->i_ino ==CEPH_INO_ROOT )) +return false; +/* otherwise, we can't know for sure */ +return true; } void ceph_handle_quota(struct ceph_mds_client *mdsc,@@ -166,6 +175,7 @@ static bool check_quota_exceeded(struct inode *inode, enum quota_check_op op,return false;down_read (&mdsc->snap_rwsem); +restart: realm =ceph_inode (inode)->i_snap_realm;if (realm)ceph_get_snap_realm (mdsc, realm);@@ -176,9 +186,25 @@ static bool check_quota_exceeded(struct inode *inode, enum quota_check_op op,spin_lock (&realm->inodes_with_caps_lock); in = realm->inode ?igrab (realm->inode) :NULL ;spin_unlock (&realm->inodes_with_caps_lock); -if (!in) -break ; - +if (!in) { +up_read (&mdsc->snap_rwsem); + in =ceph_lookup_inode (inode->i_sb, realm->ino); +down_read (&mdsc->snap_rwsem); +if (IS_ERR (in)) { +pr_warn ("Can't lookup inode %llx (err: %ld)\n" , + realm->ino,PTR_ERR (in)); +break ; + } +spin_lock (&mdsc->quotarealms_inodes_lock); +list_add (&ceph_inode (in)->i_quotarealms_inode_item, + &mdsc->quotarealms_inodes_list); +spin_unlock (&mdsc->quotarealms_inodes_lock); +spin_lock (&realm->inodes_with_caps_lock); + realm->inode = in; +spin_unlock (&realm->inodes_with_caps_lock); +ceph_put_snap_realm (mdsc, realm); + goto restart; + } ci =ceph_inode (in);spin_lock (&ci->i_ceph_lock);if (op ==QUOTA_CHECK_MAX_FILES_OP ) {fs/ceph/super.h +2
-0@@ -375,6 +375,8 @@ struct ceph_inode_info {struct list_head i_snap_realm_item;struct list_head i_snap_flush_item; +struct list_head i_quotarealms_inode_item; +struct work_struct i_wb_work;/* writeback work */ struct work_struct i_pg_inv_work;/* page invalidation work */
Re: [RFC PATCH 2/2] ceph: quota: fix quota subdir mounts
Quoted 32 lines · Zheng
Quoted 20 lines · Luis Henriques
Quoted 17 lines · Zheng
Quoted 15 lines · Luis Henriques
The CephFS kernel client doesn't enforce quotas that are set in a directory that isn't visible in the mount point. For example, given the path '/dir1/dir2', if quotas are set in 'dir1' and the mount is done in with
mount -t ceph <server>:<port>:/dir1/ /mnt
then the client can't access the 'dir1' inode from the quota realm dir2 belongs to.
This patch fixes this by simply doing an MDS LOOKUPINO Op and grabbing a reference to it (so that it doesn't disappear again). This also requires an extra field in
ceph_snap_realmso that we know we have to release that reference when destroying the realm.Quoted 9 lines
Quoted 3 lines
This may cause circle reference if somehow an inode owned by snaprealm get moved into mount subdir (other clients do rename). how about holding these inodes in
mds_client?Ok, before proceeded any further I wanted to make sure that what you were suggesting was something like the patch below. It simply keeps a list of inodes in
ceph_mds_clientuntil the filesystem is umounted, iput()ing them at that point.yes,
Quoted 8 lines
Quoted 5 lines
I'm sure I'm missing another place where the reference should be dropped, but I couldn't figure it out yet. It can't be
ceph_destroy_inode;drop_inode_snap_realmis a possibility, but what if the inode becomes visible in the meantime? Well, I'll continue thinking about it.why do you think we need to clean up the references at other place. what problem you encountered.
I'm not really seeing any issue, at least not at the moment. I believe that we could just be holding refs to inodes that may not exist anymore in the cluster. For example, in client 1:
mkdir -p /mnt/a/b
setfattr -n ceph.quota.max_files -v 5 /mnt/a
In client 2 we mount:
mount <addr>:/a/b /mnt
This client will access the realm and inode for 'a' (adding that inode to the ceph_mds_client list), because it has quotas. If client 1 then deletes 'a', client 2 will continue to have a reference to that inode in that list. That's why I thought we should be able to clean up that refs list in some other place, although that's probably a big deal, since we won't be able to a lot with this mount anyway.
Cheers,
Signature · 1 line
Luis
Re: [RFC PATCH 2/2] ceph: quota: fix quota subdir mounts
Quoted 53 lines · Luis Henriques
Quoted 32 lines · Zheng
Quoted 20 lines · Luis Henriques
Quoted 17 lines · Zheng
Quoted 15 lines · Luis Henriques
The CephFS kernel client doesn't enforce quotas that are set in a directory that isn't visible in the mount point. For example, given the path '/dir1/dir2', if quotas are set in 'dir1' and the mount is done in with
mount -t ceph <server>:<port>:/dir1/ /mnt
then the client can't access the 'dir1' inode from the quota realm dir2 belongs to.
This patch fixes this by simply doing an MDS LOOKUPINO Op and grabbing a reference to it (so that it doesn't disappear again). This also requires an extra field in
ceph_snap_realmso that we know we have to release that reference when destroying the realm.Quoted 9 lines
Quoted 3 lines
This may cause circle reference if somehow an inode owned by snaprealm get moved into mount subdir (other clients do rename). how about holding these inodes in
mds_client?Ok, before proceeded any further I wanted to make sure that what you were suggesting was something like the patch below. It simply keeps a list of inodes in
ceph_mds_clientuntil the filesystem is umounted, iput()ing them at that point.yes,
Quoted 8 lines
Quoted 5 lines
I'm sure I'm missing another place where the reference should be dropped, but I couldn't figure it out yet. It can't be
ceph_destroy_inode;drop_inode_snap_realmis a possibility, but what if the inode becomes visible in the meantime? Well, I'll continue thinking about it.why do you think we need to clean up the references at other place. what problem you encountered.
I'm not really seeing any issue, at least not at the moment. I believe that we could just be holding refs to inodes that may not exist anymore in the cluster. For example, in client 1:
mkdir -p
/mnt/a/b
setfattr -n ceph.quota.max_files -v 5/mnt/aIn client 2 we mount:
Quoted 9 lines
mount <addr>:/a/b /mnt
This client will access the realm and inode for 'a' (adding that inode to the
ceph_mds_clientlist), because it has quotas. If client 1 then deletes 'a', client 2 will continue to have a reference to that inode in that list. That's why I thought we should be able to clean up that refs list in some other place, although that's probably a big deal, since we won't be able to a lot with this mount anyway.
Agree, it's not big deal
Quoted 3 lines
Cheers,
--
Luis