ceph: fixes for directory fragment
cover ceph: fixes for directory fragment
From: "Yan, Zheng" <[email protected]>
This patch series is also in:
https://github.com/ceph/ceph-client.git wip-dirfrag
The MDS counterpart is in:
https://github.com/ceph/ceph.git wip-dirfrag
Regards
Yan, Zheng
PATCH 1/3 ceph: remove outdated frag information
From: "Yan, Zheng" <[email protected]>
If directory fragments change, fill_inode() inserts new frags into the fragtree, but it does not remove outdated frags from the fragtree. This patch fixes it.
--- fs/ceph/inode.c | 32 ++++++++++++++++++++++++++++---- 1 file changed, 28 insertions(+), 4 deletions(-)
fs/ceph/inode.c +28 -4
@@ -577,6 +577,8 @@ static int fill_inode(struct inode *inode,int issued =0 , implemented;struct timespec mtime, atime, ctime;u32 nsplits; +struct ceph_inode_frag *frag; +struct rb_node *rb_node;struct ceph_buffer *xattr_blob =NULL ;int err =0 ;int queue_trunc =0 ;
@@ -751,15 +753,37 @@ no_change:/* FIXME: move me up, if/when version reflects fragtree changes */ nsplits =le32_to_cpu (info->fragtree.nsplits);mutex_lock (&ci->i_fragtree_mutex); + rb_node =rb_first (&ci->i_fragtree);for (i =0 ; i < nsplits; i++) {u32 id =le32_to_cpu (info->fragtree.splits[i].frag); -struct ceph_inode_frag *frag =__get_or_create_frag (ci, id); - -if (IS_ERR (frag)) -continue ; + frag =NULL ; +while (rb_node) { + frag = rb_entry(rb_node, struct ceph_inode_frag, node); +if (ceph_frag_compare (frag->frag, id) >=0 ) { +if (frag->frag != id) + frag =NULL ; +else + rb_node =rb_next (rb_node); +break ; + } + rb_node =rb_next (rb_node); +rb_erase (&frag->node, &ci->i_fragtree); +kfree (frag); + } +if (!frag) { + frag =__get_or_create_frag (ci, id); +if (IS_ERR (frag)) +continue ; + } frag->split_by =le32_to_cpu (info->fragtree.splits[i].by);dout (" frag %x split by %d\n" , frag->frag, frag->split_by); } +while (rb_node) { + frag = rb_entry(rb_node, struct ceph_inode_frag, node); + rb_node =rb_next (rb_node); +rb_erase (&frag->node, &ci->i_fragtree); +kfree (frag); + }mutex_unlock (&ci->i_fragtree_mutex);/* were we issued a capability? */
Signature · 1 line
1.8.1.4
Re: [PATCH 1/3] ceph: remove outdated frag information
Quoted 70 lines · Zheng
From: "Yan, Zheng" <[email protected]>
If directory fragments change,
fill_inode()inserts new frags into the fragtree, but it does not remove outdated frags from the fragtree. This patch fixes it.Signed-off-by Yan, Zheng <[email protected]>--- fs/ceph/inode.c | 32 ++++++++++++++++++++++++++++---- 1 file changed, 28 insertions(+), 4 deletions(-)fs/ceph/inode.c +28
-4@@ -577,6 +577,8 @@ static int fill_inode(struct inode *inode,int issued =0 , implemented;struct timespec mtime, atime, ctime;u32 nsplits; +struct ceph_inode_frag *frag; +struct rb_node *rb_node;struct ceph_buffer *xattr_blob =NULL ;int err =0 ;int queue_trunc =0 ;@@ -751,15 +753,37 @@ no_change:/* FIXME: move me up, if/when version reflects fragtree changes */ nsplits =le32_to_cpu (info->fragtree.nsplits);mutex_lock (&ci->i_fragtree_mutex); + rb_node =rb_first (&ci->i_fragtree);for (i =0 ; i < nsplits; i++) {u32 id =le32_to_cpu (info->fragtree.splits[i].frag); -struct ceph_inode_frag *frag =__get_or_create_frag (ci, id); - -if (IS_ERR (frag)) -continue ; + frag =NULL ; +while (rb_node) { + frag = rb_entry(rb_node, struct ceph_inode_frag, node); +if (ceph_frag_compare (frag->frag, id) >=0 ) { +if (frag->frag != id) + frag =NULL ; +else + rb_node =rb_next (rb_node); +break ; + } + rb_node =rb_next (rb_node); +rb_erase (&frag->node, &ci->i_fragtree); +kfree (frag); + } +if (!frag) { + frag =__get_or_create_frag (ci, id); +if (IS_ERR (frag)) +continue ; + } frag->split_by =le32_to_cpu (info->fragtree.splits[i].by);dout (" frag %x split by %d\n" , frag->frag, frag->split_by); } +while (rb_node) { + frag = rb_entry(rb_node, struct ceph_inode_frag, node); + rb_node =rb_next (rb_node); +rb_erase (&frag->node, &ci->i_fragtree); +kfree (frag); + }mutex_unlock (&ci->i_fragtree_mutex);/* were we issued a capability? */ Signature · 1 line
1.8.1.4
PATCH 2/3 ceph: handle frag mismatch between readdir request and reply
In reply to Yan, ZhengFrom: "Yan, Zheng" <[email protected]>
If client has outdated directory fragments information, it may request readdir an non-existent directory fragment. In this case, the MDS finds an approximate directory fragment and sends its contents back to the client. When receiving a reply with fragment that is different than the requested one, the client need to reset the 'readdir offset'.
--- fs/ceph/dir.c | 11 ++++++++++- fs/ceph/inode.c | 15 +++++++++++++-- fs/ceph/mds_client.c | 3 +-- 3 files changed, 24 insertions(+), 5 deletions(-)
fs/ceph/dir.c +10 -1
@@ -352,8 +352,18 @@ more: }/* note next offset and last dentry name */ + rinfo = &req->r_reply_info; +if (le32_to_cpu (rinfo->dir_dir->frag) != frag) { + frag =le32_to_cpu (rinfo->dir_dir->frag); +if (ceph_frag_is_leftmost (frag)) + fi->next_offset =2 ; +else + fi->next_offset =0 ; + off = fi->next_offset; + } fi->offset = fi->next_offset; fi->last_readdir = req; + fi->frag = frag;if (req->r_reply_info.dir_end) {kfree (fi->last_name);
@@ -363,7 +373,6 @@ more:else fi->next_offset =0 ; }else { - rinfo = &req->r_reply_info; err =note_last_dentry (fi, rinfo->dir_dname[rinfo->dir_nr-1 ], rinfo->dir_dname_len[rinfo->dir_nr-1 ]);
fs/ceph/inode.c +13 -2
@@ -1274,8 +1274,19 @@ int ceph_readdir_prepopulate(struct ceph_mds_request *req,int err =0 , i;struct inode *snapdir =NULL ;struct ceph_mds_request_head *rhead = req->r_request->front.iov_base; -u64 frag =le32_to_cpu (rhead->args.readdir.frag);struct ceph_dentry_info *di; +u64 r_readdir_offset = req->r_readdir_offset; +u32 frag =le32_to_cpu (rhead->args.readdir.frag); + +if (le32_to_cpu (rinfo->dir_dir->frag) != frag) { +dout ("readdir_prepopulate got new frag %x -> %x\n" , + frag,le32_to_cpu (rinfo->dir_dir->frag)); + frag =le32_to_cpu (rinfo->dir_dir->frag); +if (ceph_frag_is_leftmost (frag)) + r_readdir_offset =2 ; +else + r_readdir_offset =0 ; + }if (req->r_aborted)return readdir_prepopulate_inodes_only (req, session);
@@ -1339,7 +1350,7 @@ retry_lookup: } di = dn->d_fsdata; - di->offset =ceph_make_fpos (frag, i + req->r_readdir_offset); + di->offset =ceph_make_fpos (frag, i + r_readdir_offset);/* inode */ if (dn->d_inode) {
fs/ceph/mds_client.c +1 -2
@@ -2238,8 +2238,7 @@ static void handle_reply(struct ceph_mds_session *session, struct ceph_msg *msg) err =ceph_fill_trace (mdsc->fsc->sb, req, req->r_session);if (err ==0 ) {if (result ==0 && (req->r_op ==CEPH_MDS_OP_READDIR || - req->r_op ==CEPH_MDS_OP_LSSNAP ) && - rinfo->dir_nr) + req->r_op ==CEPH_MDS_OP_LSSNAP ))ceph_readdir_prepopulate (req, req->r_session);ceph_unreserve_caps (mdsc, &req->r_caps_reservation); }
Signature · 1 line
1.8.1.4
Re: [PATCH 2/3] ceph: handle frag mismatch between readdir request and reply
Quoted 103 lines · Zheng
From: "Yan, Zheng" <[email protected]>
If client has outdated directory fragments information, it may request readdir an non-existent directory fragment. In this case, the MDS finds an approximate directory fragment and sends its contents back to the client. When receiving a reply with fragment that is different than the requested one, the client need to reset the 'readdir offset'.
Signed-off-by Yan, Zheng <[email protected]>--- fs/ceph/dir.c | 11 ++++++++++- fs/ceph/inode.c | 15 +++++++++++++-- fs/ceph/mds_client.c | 3 +-- 3 files changed, 24 insertions(+), 5 deletions(-)fs/ceph/dir.c +10
-1@@ -352,8 +352,18 @@ more: }/* note next offset and last dentry name */ + rinfo = &req->r_reply_info; +if (le32_to_cpu (rinfo->dir_dir->frag) != frag) { + frag =le32_to_cpu (rinfo->dir_dir->frag); +if (ceph_frag_is_leftmost (frag)) + fi->next_offset =2 ; +else + fi->next_offset =0 ; + off = fi->next_offset; + } fi->offset = fi->next_offset; fi->last_readdir = req; + fi->frag = frag;if (req->r_reply_info.dir_end) {kfree (fi->last_name);@@ -363,7 +373,6 @@ more:else fi->next_offset =0 ; }else { - rinfo = &req->r_reply_info; err =note_last_dentry (fi, rinfo->dir_dname[rinfo->dir_nr-1 ], rinfo->dir_dname_len[rinfo->dir_nr-1 ]);fs/ceph/inode.c +13
-2@@ -1274,8 +1274,19 @@ int ceph_readdir_prepopulate(struct ceph_mds_request *req,int err =0 , i;struct inode *snapdir =NULL ;struct ceph_mds_request_head *rhead = req->r_request->front.iov_base; -u64 frag =le32_to_cpu (rhead->args.readdir.frag);struct ceph_dentry_info *di; +u64 r_readdir_offset = req->r_readdir_offset; +u32 frag =le32_to_cpu (rhead->args.readdir.frag); + +if (le32_to_cpu (rinfo->dir_dir->frag) != frag) { +dout ("readdir_prepopulate got new frag %x -> %x\n" , + frag,le32_to_cpu (rinfo->dir_dir->frag)); + frag =le32_to_cpu (rinfo->dir_dir->frag); +if (ceph_frag_is_leftmost (frag)) + r_readdir_offset =2 ; +else + r_readdir_offset =0 ; + }if (req->r_aborted)return readdir_prepopulate_inodes_only (req, session);@@ -1339,7 +1350,7 @@ retry_lookup: } di = dn->d_fsdata; - di->offset =ceph_make_fpos (frag, i + req->r_readdir_offset); + di->offset =ceph_make_fpos (frag, i + r_readdir_offset);/* inode */ if (dn->d_inode) {fs/ceph/mds_client.c +1
-2@@ -2238,8 +2238,7 @@ static void handle_reply(struct ceph_mds_session *session, struct ceph_msg *msg) err =ceph_fill_trace (mdsc->fsc->sb, req, req->r_session);if (err ==0 ) {if (result ==0 && (req->r_op ==CEPH_MDS_OP_READDIR || - req->r_op ==CEPH_MDS_OP_LSSNAP ) && - rinfo->dir_nr) + req->r_op ==CEPH_MDS_OP_LSSNAP ))ceph_readdir_prepopulate (req, req->r_session);ceph_unreserve_caps (mdsc, &req->r_caps_reservation); }Signature · 6 lines
1.8.1.4 -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 3/3 ceph: drop unconnected inodes
In reply to Yan, ZhengFrom: "Yan, Zheng" <[email protected]>
Positve dentry and corresponding inode are always accompanied in MDS reply. So no need to keep inode in the cache after dropping all its aliases.
--- fs/ceph/inode.c | 10 ++++++++++ fs/ceph/super.c | 1 + fs/ceph/super.h | 1 + 3 files changed, 12 insertions(+)
fs/ceph/inode.c +10 -0
@@ -436,6 +436,16 @@ void ceph_destroy_inode(struct inode *inode)call_rcu (&inode->i_rcu, ceph_i_callback); } +int ceph_drop_inode (struct inode *inode) +{ +/* +* Positve dentry and corresponding inode are always accompanied +* in MDS reply. So no need to keep inode in the cache after +* dropping all its aliases. +*/ +return 1 ; +} +/* * Helpers to fill in size, ctime, mtime, and atime. We have to be * careful because either the client or MDS may have more up to date
fs/ceph/super.c +1 -0
@@ -686,6 +686,7 @@ static const struct super_operations ceph_super_ops = { .alloc_inode = ceph_alloc_inode, .destroy_inode = ceph_destroy_inode, .write_inode = ceph_write_inode, + .drop_inode = ceph_drop_inode, .sync_fs = ceph_sync_fs, .put_super = ceph_put_super, .show_options = ceph_show_options,
fs/ceph/super.h +1 -0
@@ -691,6 +691,7 @@ extern const struct inode_operations ceph_file_iops;extern struct inode *ceph_alloc_inode (struct super_block *sb);extern void ceph_destroy_inode (struct inode *inode); +extern int ceph_drop_inode (struct inode *inode);extern struct inode *ceph_get_inode (struct super_block *sb,struct ceph_vino vino);
Signature · 1 line
1.8.1.4
Re: [PATCH 3/3] ceph: drop unconnected inodes
Quoted 4 lines · Zheng
From: "Yan, Zheng" <[email protected]>
Positve dentry and corresponding inode are always accompanied in MDS reply. So no need to keep inode in the cache after dropping all its aliases.
Quoted 62 lines
Signed-off-by Yan, Zheng <[email protected]>--- fs/ceph/inode.c | 10 ++++++++++ fs/ceph/super.c | 1 + fs/ceph/super.h | 1 + 3 files changed, 12 insertions(+)fs/ceph/inode.c +10
-0@@ -436,6 +436,16 @@ void ceph_destroy_inode(struct inode *inode)call_rcu (&inode->i_rcu, ceph_i_callback); } +int ceph_drop_inode (struct inode *inode) +{ +/* +* Positve dentry and corresponding inode are always accompanied +* in MDS reply. So no need to keep inode in the cache after +* dropping all its aliases. +*/ +return 1 ; +} +/* * Helpers to fill in size, ctime, mtime, and atime. We have to be * careful because either the client or MDS may have more up to date fs/ceph/super.c +1
-0@@ -686,6 +686,7 @@ static const struct super_operations ceph_super_ops = { .alloc_inode = ceph_alloc_inode, .destroy_inode = ceph_destroy_inode, .write_inode = ceph_write_inode, + .drop_inode = ceph_drop_inode, .sync_fs = ceph_sync_fs, .put_super = ceph_put_super, .show_options = ceph_show_options,fs/ceph/super.h +1
-0@@ -691,6 +691,7 @@ extern const struct inode_operations ceph_file_iops;extern struct inode *ceph_alloc_inode (struct super_block *sb);extern void ceph_destroy_inode (struct inode *inode); +extern int ceph_drop_inode (struct inode *inode);extern struct inode *ceph_get_inode (struct super_block *sb,struct ceph_vino vino);Signature · 6 lines
1.8.1.4 -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
These look good.
I would *really* like to get some tests into ceph-qa-suite that cover multi-mds, multi-mds thrashing, and mds frag = true. Eventulaly also 'mds thrash frags = true' (is this what you're testing with?).
sage
Quoted 16 lines · Zheng
From: "Yan, Zheng" <[email protected]>
This patch series is also in:
https://github.com/ceph/ceph-client.git wip-dirfragThe MDS counterpart is in:
https://github.com/ceph/ceph.git wip-dirfragRegards
Yan, Zheng
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Sage Weil
These look good.
I would *really* like to get some tests into ceph-qa-suite that cover multi-mds, multi-mds thrashing, and mds frag = true. Eventulaly also 'mds thrash frags = true' (is this what you're testing with?).
sage
my conf for the directory fragment feaure
---
mds bal_frag = 1
mds bal_split_bits = 2
mds bal_split_size = 256
mds bal_merge_size = 32
I test these patches by running blogbench and a script that create/delete 500k file in a directory.
Besides the mds is restarted every two miniutes.
Regards
Yan, Zheng
Quoted 24 lines · Zheng
Quoted 9 lines
From: "Yan, Zheng" <[email protected]>
This patch series is also in:
https://github.com/ceph/ceph-client.git wip-dirfragThe MDS counterpart is in:
https://github.com/ceph/ceph.git wip-dirfragQuoted 8 lines
Regards
Yan, Zheng
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 16 lines · Zheng
Quoted 7 lines · Sage Weil
These look good.
I would *really* like to get some tests into ceph-qa-suite that cover multi-mds, multi-mds thrashing, and mds frag = true. Eventulaly also 'mds thrash frags = true' (is this what you're testing with?).
sage
my conf for the directory fragment feaure
---
mdsbal_frag= 1
mdsbal_split_bits= 2
mdsbal_split_size= 256
mdsbal_merge_size= 32
Quoted 2 lines
I test these patches by running blogbench and a script that create/delete 500k file in a directory.
Can you add the script to qa/workunits/ somewhere? Then we can add some tests to ceph-qa-suite.git/suites/fs/frag so that this gets nightly coverage.
Thanks!
sage
Quoted 31 lines
Besides the mds is restarted every two miniutes.
Regards
Yan, ZhengQuoted 24 lines · Zheng
Quoted 9 lines
From: "Yan, Zheng" <[email protected]>
This patch series is also in:
https://github.com/ceph/ceph-client.git wip-dirfragThe MDS counterpart is in:
https://github.com/ceph/ceph.git wip-dirfragQuoted 8 lines
Regards
Yan, Zheng
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html