ceph: add perf metrics support
Subsystems (2)
v5 cover ceph: add perf metrics support
From: Xiubo Li <[email protected]>
Changed in V2:
- add read/write/metadata latency metric support.
- add and send client provided metric flags in client metadata
- addressed the comments from Ilya and merged the 4/4 patch into 3/4.
- addressed all the other comments in v1 series.
Changed in V3:
- addressed Jeff's comments and let's the callers do the metric counting.
- with some small fixes for the read/write latency
- tested based on the latest testing branch
Changed in V4:
- fix the lock issue
Changed in V5:
- add r_end_stamp for the osdc request
- delete reset metric and move it to metric sysfs
- move ceph_osdc_{read,write}pages to ceph.ko
- use percpu counters instead for read/write/metadata latencies
It will send the metrics to the MDSs every second if sending_metrics is enabled, disable as default.
We can get the metrics from the debugfs:
$ cat /sys/kernel/debug/ceph/0c93a60d-5645-4c46-8568-4c8f63db4c7f.client4267/metrics item total sum_lat(us) avg_lat(us) ----------------------------------------------------- read 13 417000 32076 write 42 131205000 3123928 metadata 104 493000 4740
item total miss hit
-------------------------------------------------d_lease 204 0 918
session caps miss hit
-------------------------------------------------
0 204 213 368218
In the MDS side, we can get the metrics(NOTE: the latency is in nanosecond):
$ ./bin/ceph fs perf stats | python -m json.tool
{
"client_metadata": {
"client.4267": {
"IP": "v1:192.168.195.165",
"hostname": "fedora1",
"mount_point": "N/A",
"root": "/"
}
},
"counters": [
"cap_hit"
],
"global_counters": [
"read_latency",
"write_latency",
"metadata_latency",
"dentry_lease_hit"
],
"global_metrics": {
"client.4267": [
[
0,
32076923
],
[
3,
123928571
],
[
0,
4740384
],
[
918,
0
]
]
},
"metrics": {
"delayed_ranks": [],
"mds.0": {
"client.4267": [
[
368218,
213
]
]
}
}
}The provided metric flags in client metadata
$./bin/cephfs-journal-tool --rank=1:0 event get --type=SESSION json Wrote output to JSON file 'dump'
$ cat dump
[
{
"client instance": "client.4275 v1:192.168.195.165:0/461391971",
"open": "true",
"client map version": 1,
"inos": "[]",
"inotable version": 0,
"client_metadata": {
"client_features": {
"feature_bits": "0000000000001bff"
},
"metric_spec": {
"metric_flags": {
"feature_bits": "000000000000001f"
}
},
"entity_id": "",
"hostname": "fedora1",
"kernel_version": "5.5.0-rc2+",
"root": "/"
}
},
[...]*** BLURB HERE ***
Xiubo Li (10):
ceph: add caps perf metric for each session
ceph: move ceph_osdc_{read,write}pages to ceph.ko
ceph: add r_end_stamp for the osdc request
ceph: add global read latency metric support
ceph: add global write latency metric support
ceph: add global metadata perf metric support
ceph: periodically send perf metrics to MDS
ceph: add CEPH_DEFINE_RW_FUNC helper support
ceph: add reset metrics support
ceph: send client provided metric flags in client metadata
fs/ceph/acl.c | 2 + fs/ceph/addr.c | 106 ++++++++++- fs/ceph/caps.c | 74 ++++++++ fs/ceph/debugfs.c | 140 +++++++++++++- fs/ceph/dir.c | 9 +- fs/ceph/file.c | 26 +++ fs/ceph/mds_client.c | 327 +++++++++++++++++++++++++++++--- fs/ceph/mds_client.h | 15 +- fs/ceph/metric.h | 150 +++++++++++++++ fs/ceph/quota.c | 9 +- fs/ceph/super.h | 13 ++ fs/ceph/xattr.c | 17 +- include/linux/ceph/ceph_fs.h | 1 + include/linux/ceph/debugfs.h | 14 ++ include/linux/ceph/osd_client.h | 18 +- net/ceph/osd_client.c | 81 +------- 16 files changed, 862 insertions(+), 140 deletions(-) create mode 100644 fs/ceph/metric.h
Signature · 1 line
2.21.0
PATCH v5 1/10 ceph: add caps perf metric for each session
From: Xiubo Li <[email protected]>
This will fulfill the caps hit/miss metric for each session. When checking the "need" mask and if one cap has the subset of the "need" mask it means hit, or missed.
item total miss hit
-------------------------------------------------d_lease 295 0 993
session caps miss hit
-------------------------------------------------
0 295 107 4119 1 1 107 9
URL: https://tracker.ceph.com/issues/43215
--- fs/ceph/acl.c | 2 ++ fs/ceph/addr.c | 2 ++ fs/ceph/caps.c | 74 ++++++++++++++++++++++++++++++++++++++++++++ fs/ceph/debugfs.c | 20 ++++++++++++ fs/ceph/dir.c | 9 ++++-- fs/ceph/file.c | 3 ++ fs/ceph/mds_client.c | 16 +++++++++- fs/ceph/mds_client.h | 3 ++ fs/ceph/quota.c | 9 ++++-- fs/ceph/super.h | 11 +++++++ fs/ceph/xattr.c | 17 ++++++++-- 11 files changed, 158 insertions(+), 8 deletions(-)
fs/ceph/acl.c +2 -0
@@ -22,6 +22,8 @@ static inline void ceph_set_cached_acl(struct inode *inode,struct ceph_inode_info *ci =ceph_inode (inode);spin_lock (&ci->i_ceph_lock); +__ceph_caps_metric (ci,CEPH_CAP_XATTR_SHARED ); +if (__ceph_caps_issued_mask (ci,CEPH_CAP_XATTR_SHARED ,0 ))set_cached_acl (inode, type, acl);else
fs/ceph/addr.c +2 -0
@@ -1706,6 +1706,8 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) err = -ENOMEM ; goto out; } + +ceph_caps_metric (ci,CEPH_STAT_CAP_INLINE_DATA ); err =__ceph_do_getattr (inode, page,CEPH_STAT_CAP_INLINE_DATA , true);if (err <0 ) {
fs/ceph/caps.c +74 -0
104 lines not on this page · raw message
fs/ceph/debugfs.c +20 -0
@@ -128,6 +128,7 @@ static int metric_show(struct seq_file *s, void *p) {struct ceph_fs_client *fsc = s->private;struct ceph_mds_client *mdsc = fsc->mdsc; +int i;seq_printf (s,"item total miss hit\n" );seq_printf (s,"-------------------------------------------------\n" );
@@ -137,6 +138,25 @@ static int metric_show(struct seq_file *s, void *p)percpu_counter_sum (&mdsc->metric.d_lease_mis), percpu_counter_sum(&mdsc->metric.d_lease_hit)); +seq_printf (s,"\n" ); +seq_printf (s,"session caps miss hit\n" ); +seq_printf (s,"-------------------------------------------------\n" ); + +mutex_lock (&mdsc->mutex); +for (i =0 ; i < mdsc->max_sessions; i++) { +struct ceph_mds_session *session; + + session =__ceph_lookup_mds_session (mdsc, i); +if (!session) +continue ; +seq_printf (s,"%-14d%-16d%-16lld%lld\n" , i, + session->s_nr_caps, +percpu_counter_sum (&session->i_caps_mis), +percpu_counter_sum (&session->i_caps_hit)); +ceph_put_mds_session (session); + } +mutex_unlock (&mdsc->mutex); +return 0 ; }
fs/ceph/dir.c +7 -2
@@ -313,7 +313,7 @@ static int ceph_readdir(struct file *file, struct dir_context *ctx)struct ceph_fs_client *fsc =ceph_inode_to_client (inode);struct ceph_mds_client *mdsc = fsc->mdsc;int i; -int err; +int err, ret =-1 ;unsigned frag =-1 ;struct ceph_mds_reply_info_parsed *rinfo;
@@ -346,13 +346,16 @@ static int ceph_readdir(struct file *file, struct dir_context *ctx) !ceph_test_mount_opt(fsc, NOASYNCREADDIR) && ceph_snap(inode) != CEPH_SNAPDIR && __ceph_dir_is_complete_ordered(ci) && - __ceph_caps_issued_mask(ci, CEPH_CAP_FILE_SHARED, 1)) { + (ret = __ceph_caps_issued_mask(ci, CEPH_CAP_FILE_SHARED, 1))) {int shared_gen =atomic_read (&ci->i_shared_gen); +__ceph_caps_metric (ci,CEPH_CAP_XATTR_SHARED );spin_unlock (&ci->i_ceph_lock); err =__dcache_readdir (file, ctx, shared_gen);if (err != -EAGAIN )return err; } else { +if (ret !=-1 ) +__ceph_caps_metric (ci,CEPH_CAP_XATTR_SHARED );spin_unlock (&ci->i_ceph_lock); }
@@ -757,6 +760,8 @@ static struct dentry *ceph_lookup(struct inode *dir, struct dentry *dentry,struct ceph_dentry_info *di =ceph_dentry (dentry);spin_lock (&ci->i_ceph_lock); +__ceph_caps_metric (ci,CEPH_CAP_FILE_SHARED ); +dout (" dir %p flags are %d\n" , dir, ci->i_ceph_flags); if (strncmp(dentry->d_name.name, fsc->mount_options->snapdir_name,
fs/ceph/file.c +3 -0
@@ -384,6 +384,8 @@ int ceph_open(struct inode *inode, struct file *file)* asynchronously. */ spin_lock (&ci->i_ceph_lock); +__ceph_caps_metric (ci, wanted); +if (__ceph_is_any_real_caps (ci) && (((fmode &CEPH_FILE_MODE_WR ) ==0 ) || ci->i_auth_cap)) {int mds_wanted =__ceph_caps_mds_wanted (ci, true);
@@ -1340,6 +1342,7 @@ static ssize_t ceph_read_iter(struct kiocb *iocb, struct iov_iter *to)return -ENOMEM ; } +ceph_caps_metric (ci,CEPH_STAT_CAP_INLINE_DATA ); statret =__ceph_do_getattr (inode, page,CEPH_STAT_CAP_INLINE_DATA , !!page);if (statret <0 ) {
fs/ceph/mds_client.c +15 -1
@@ -558,6 +558,8 @@ void ceph_put_mds_session(struct ceph_mds_session *s)if (refcount_dec_and_test (&s->s_ref)) {if (s->s_auth.authorizer)ceph_auth_destroy_authorizer (s->s_auth.authorizer); +percpu_counter_destroy (&s->i_caps_hit); +percpu_counter_destroy (&s->i_caps_mis);kfree (s); } }
@@ -598,6 +600,7 @@ static struct ceph_mds_session *register_session(struct ceph_mds_client *mdsc, int mds) {struct ceph_mds_session *s; +int err;if (mds >= mdsc->mdsmap->possible_max_rank)return ERR_PTR (-EINVAL );
@@ -612,8 +615,10 @@ static struct ceph_mds_session *register_session(struct ceph_mds_client *mdsc,dout ("%s: realloc to %d\n" , __func__, newmax); sa =kcalloc (newmax,sizeof (void *),GFP_NOFS ); -if (!sa) +if (!sa) { + err = -ENOMEM ; goto fail_realloc; + }if (mdsc->sessions) {memcpy (sa, mdsc->sessions, mdsc->max_sessions *sizeof (void *));
@@ -653,6 +658,13 @@ static struct ceph_mds_session *register_session(struct ceph_mds_client *mdsc,INIT_LIST_HEAD (&s->s_cap_flushing); + err =percpu_counter_init (&s->i_caps_hit,0 ,GFP_NOFS ); +if (err) + goto fail_realloc; + err =percpu_counter_init (&s->i_caps_mis,0 ,GFP_NOFS ); +if (err) + goto fail_init; + mdsc->sessions[mds] = s;atomic_inc (&mdsc->num_sessions);refcount_inc (&s->s_ref);/* one ref to sessions[], one to caller */
@@ -662,6 +674,8 @@ static struct ceph_mds_session *register_session(struct ceph_mds_client *mdsc,return s; +fail_init: +percpu_counter_destroy (&s->i_caps_hit); fail_realloc:kfree (s);return ERR_PTR (-ENOMEM );
fs/ceph/mds_client.h +3 -0
@@ -201,6 +201,9 @@ struct ceph_mds_session {struct list_head s_waiting;/* waiting requests */ struct list_head s_unsafe;/* unsafe requests */ + +struct percpu_counter i_caps_hit; +struct percpu_counter i_caps_mis; };/*
fs/ceph/quota.c +7 -2
@@ -147,9 +147,14 @@ static struct inode *lookup_quotarealm_inode(struct ceph_mds_client *mdsc,return NULL ; }if (qri->inode) { +struct ceph_inode_info *ci =ceph_inode (qri->inode); +int ret; + +ceph_caps_metric (ci,CEPH_STAT_CAP_INODE ); +/* get caps */ -int ret =__ceph_do_getattr (qri->inode,NULL , -CEPH_STAT_CAP_INODE , true); + ret =__ceph_do_getattr (qri->inode,NULL , +CEPH_STAT_CAP_INODE , true);if (ret >=0 ) in = qri->inode;else
fs/ceph/super.h +11 -0
@@ -641,6 +641,14 @@ static inline bool __ceph_is_any_real_caps(struct ceph_inode_info *ci)return !RB_EMPTY_ROOT (&ci->i_caps); } +extern void __ceph_caps_metric (struct ceph_inode_info *ci,int mask); +static inline void ceph_caps_metric (struct ceph_inode_info *ci,int mask) +{ +spin_lock (&ci->i_ceph_lock); +__ceph_caps_metric (ci, mask); +spin_unlock (&ci->i_ceph_lock); +} +extern int __ceph_caps_issued (struct ceph_inode_info *ci,int *implemented);extern int __ceph_caps_issued_mask (struct ceph_inode_info *ci,int mask,int t); extern int __ceph_caps_issued_other(struct ceph_inode_info *ci,
@@ -927,6 +935,9 @@ extern int __ceph_do_getattr(struct inode *inode, struct page *locked_page, int mask, bool force);static inline int ceph_do_getattr (struct inode *inode,int mask,bool force) { +struct ceph_inode_info *ci =ceph_inode (inode); + +ceph_caps_metric (ci, mask);return __ceph_do_getattr (inode,NULL , mask, force); }extern int ceph_permission (struct inode *inode,int mask);
fs/ceph/xattr.c +14 -3
@@ -829,6 +829,7 @@ ssize_t __ceph_getxattr(struct inode *inode, const char *name, void *value,struct ceph_vxattr *vxattr =NULL ;int req_mask;ssize_t err; +int ret =-1 ;/* let's see if a virtual xattr was requested */ vxattr =ceph_match_vxattr (inode, name);
@@ -856,7 +857,9 @@ ssize_t __ceph_getxattr(struct inode *inode, const char *name, void *value,if (ci->i_xattrs.version ==0 || !((req_mask &CEPH_CAP_XATTR_SHARED ) || -__ceph_caps_issued_mask (ci,CEPH_CAP_XATTR_SHARED ,1 ))) { + (ret =__ceph_caps_issued_mask (ci,CEPH_CAP_XATTR_SHARED ,1 )))) { +if (ret !=-1 ) +__ceph_caps_metric (ci,CEPH_CAP_XATTR_SHARED );spin_unlock (&ci->i_ceph_lock);/* security module gets xattr while filling trace */
@@ -871,6 +874,9 @@ ssize_t __ceph_getxattr(struct inode *inode, const char *name, void *value,if (err)return err;spin_lock (&ci->i_ceph_lock); + }else { +if (ret !=-1 ) +__ceph_caps_metric (ci,CEPH_CAP_XATTR_SHARED ); } err =__build_xattrs (inode);
@@ -907,19 +913,24 @@ ssize_t ceph_listxattr(struct dentry *dentry, char *names, size_t size)struct ceph_inode_info *ci =ceph_inode (inode);bool len_only = (size ==0 );u32 namelen; -int err; +int err, ret =-1 ;spin_lock (&ci->i_ceph_lock);dout ("listxattr %p ver=%lld index_ver=%lld\n" , inode, ci->i_xattrs.version, ci->i_xattrs.index_version);if (ci->i_xattrs.version ==0 || - !__ceph_caps_issued_mask (ci,CEPH_CAP_XATTR_SHARED ,1 )) { + !(ret =__ceph_caps_issued_mask (ci,CEPH_CAP_XATTR_SHARED ,1 ))) { +if (ret !=-1 ) +__ceph_caps_metric (ci,CEPH_CAP_XATTR_SHARED );spin_unlock (&ci->i_ceph_lock); err =ceph_do_getattr (inode,CEPH_STAT_CAP_XATTR , true);if (err)return err;spin_lock (&ci->i_ceph_lock); + }else { +if (ret !=-1 ) +__ceph_caps_metric (ci,CEPH_CAP_XATTR_SHARED ); } err =__build_xattrs (inode);
Signature · 1 line
2.21.0
PATCH v5 2/10 ceph: move ceph_osdc_{read,write}pages to ceph.ko
In reply to [email protected]From: Xiubo Li <[email protected]>
Since this helpers are only used by cpeh.ko, let's move it to ceph.ko and rename to _sync_.
--- fs/ceph/addr.c | 86 ++++++++++++++++++++++++++++++++- include/linux/ceph/osd_client.h | 17 ------- net/ceph/osd_client.c | 79 ------------------------------ 3 files changed, 84 insertions(+), 98 deletions(-)
fs/ceph/addr.c +84 -2
@@ -182,6 +182,47 @@ static int ceph_releasepage(struct page *page, gfp_t g)return !PagePrivate (page); } +/* +* Read some contiguous pages. If we cross a stripe boundary, shorten +* *plen. Return number of bytes read, or error. +*/ +static int ceph_sync_readpages (struct ceph_fs_client *fsc, +struct ceph_vino vino, +struct ceph_file_layout *layout, +u64 off,u64 *plen, +u32 truncate_seq,u64 truncate_size, +struct page **pages,int num_pages, +int page_align) +{ +struct ceph_osd_client *osdc = &fsc->client->osdc; +struct ceph_osd_request *req; +int rc =0 ; + +dout ("readpages on ino %llx.%llx on %llu~%llu\n" , vino.ino, + vino.snap, off, *plen); + req =ceph_osdc_new_request (osdc, layout, vino, off, plen,0 ,1 , +CEPH_OSD_OP_READ ,CEPH_OSD_FLAG_READ , +NULL , truncate_seq, truncate_size, + false); +if (IS_ERR (req)) +return PTR_ERR (req); + +/* it may be a short read due to an object boundary */ +osd_req_op_extent_osd_data_pages (req,0 , + pages, *plen, page_align, false, false); + +dout ("readpages final extent is %llu~%llu (%llu bytes align %d)\n" , + off, *plen, *plen, page_align); + + rc =ceph_osdc_start_request (osdc, req, false); +if (!rc) + rc =ceph_osdc_wait_request (osdc, req); + +ceph_osdc_put_request (req); +dout ("readpages result %d\n" , rc); +return rc; +} +/* * read a single page, without unlocking it. */
@@ -218,7 +259,7 @@ static int ceph_do_readpage(struct file *filp, struct page *page)dout ("readpage inode %p file %p page %p index %lu\n" , inode, filp, page, page->index); - err =ceph_osdc_readpages (&fsc->client->osdc,ceph_vino (inode), + err =ceph_sync_readpages (fsc,ceph_vino (inode), &ci->i_layout, off, &len, ci->i_truncate_seq, ci->i_truncate_size, &page,1 ,0 );
@@ -570,6 +611,47 @@ static u64 get_writepages_data_length(struct inode *inode,return end > start ? end - start :0 ; } +/* +* do a synchronous write on N pages +*/ +static int ceph_sync_writepages (struct ceph_fs_client *fsc, +struct ceph_vino vino, +struct ceph_file_layout *layout, +struct ceph_snap_context *snapc, +u64 off,u64 len, +u32 truncate_seq,u64 truncate_size, +struct timespec64 *mtime, +struct page **pages,int num_pages) +{ +struct ceph_osd_client *osdc = &fsc->client->osdc; +struct ceph_osd_request *req; +int rc =0 ; +int page_align = off & ~PAGE_MASK ; + + req =ceph_osdc_new_request (osdc, layout, vino, off, &len,0 ,1 , +CEPH_OSD_OP_WRITE ,CEPH_OSD_FLAG_WRITE , + snapc, truncate_seq, truncate_size, + true); +if (IS_ERR (req)) +return PTR_ERR (req); + +/* it may be a short write due to an object boundary */ +osd_req_op_extent_osd_data_pages (req,0 , pages, len, page_align, + false, false); +dout ("writepages %llu~%llu (%llu bytes)\n" , off, len, len); + + req->r_mtime = *mtime; + rc =ceph_osdc_start_request (osdc, req, true); +if (!rc) + rc =ceph_osdc_wait_request (osdc, req); + +ceph_osdc_put_request (req); +if (rc ==0 ) + rc = len; +dout ("writepages result %d\n" , rc); +return rc; +} +/* * Write a single page, but leave the page locked. *
@@ -628,7 +710,7 @@ static int writepage_nounlock(struct page *page, struct writeback_control *wbc)set_bdi_congested (inode_to_bdi (inode),BLK_RW_ASYNC );set_page_writeback (page); - err = ceph_osdc_writepages(&fsc->client->osdc, ceph_vino(inode), + err = ceph_sync_writepages(fsc, ceph_vino(inode), &ci->i_layout, snapc, page_off, len, ceph_wbc.truncate_seq, ceph_wbc.truncate_size,
include/linux/ceph/osd_client.h +0 -17
@@ -509,23 +509,6 @@ int ceph_osdc_call(struct ceph_osd_client *osdc, struct page *req_page, size_t req_len, struct page **resp_pages, size_t *resp_len); -extern int ceph_osdc_readpages (struct ceph_osd_client *osdc, -struct ceph_vino vino, -struct ceph_file_layout *layout, -u64 off,u64 *plen, -u32 truncate_seq,u64 truncate_size, -struct page **pages,int nr_pages, -int page_align); - -extern int ceph_osdc_writepages (struct ceph_osd_client *osdc, -struct ceph_vino vino, -struct ceph_file_layout *layout, -struct ceph_snap_context *sc, -u64 off,u64 len, -u32 truncate_seq,u64 truncate_size, -struct timespec64 *mtime, -struct page **pages,int nr_pages); - int ceph_osdc_copy_from(struct ceph_osd_client *osdc, u64 src_snapid, u64 src_version, struct ceph_object_id *src_oid,
net/ceph/osd_client.c +0 -79
85 lines not on this page · raw message
Signature · 1 line
2.21.0
PATCH v5 3/10 ceph: add r_end_stamp for the osdc request
In reply to [email protected]From: Xiubo Li <[email protected]>
Grab the osdc requests' end time stamp.
--- include/linux/ceph/osd_client.h | 1 + net/ceph/osd_client.c | 2 ++ 2 files changed, 3 insertions(+)
include/linux/ceph/osd_client.h +1 -0
@@ -213,6 +213,7 @@ struct ceph_osd_request {/* internal */ unsigned long r_stamp;/* jiffies, send or check time */ unsigned long r_start_stamp;/* jiffies */ +unsigned long r_end_stamp;/* jiffies */ int r_attempts;u32 r_map_dne_bound;
net/ceph/osd_client.c +2 -0
@@ -2389,6 +2389,8 @@ static void finish_request(struct ceph_osd_request *req)WARN_ON (lookup_request_mc (&osdc->map_checks, req->r_tid));dout ("%s req %p tid %llu\n" , __func__, req, req->r_tid); + req->r_end_stamp = jiffies; +if (req->r_osd)unlink_request (req->r_osd, req);atomic_dec (&osdc->num_requests);
Signature · 1 line
2.21.0
PATCH v5 4/10 ceph: add global read latency metric support
In reply to [email protected]From: Xiubo Li <[email protected]>
item total sum_lat(us) avg_lat(us)
-----------------------------------------------------
read 73 3590000 49178082
URL: https://tracker.ceph.com/issues/43215
--- fs/ceph/addr.c | 8 ++++++++ fs/ceph/debugfs.c | 11 +++++++++++ fs/ceph/file.c | 15 +++++++++++++++ fs/ceph/mds_client.c | 29 +++++++++++++++++++++++------ fs/ceph/mds_client.h | 9 ++------- fs/ceph/metric.h | 30 ++++++++++++++++++++++++++++++ 6 files changed, 89 insertions(+), 13 deletions(-) create mode 100644 fs/ceph/metric.h
fs/ceph/addr.c +8 -0
@@ -195,6 +195,7 @@ static int ceph_sync_readpages(struct ceph_fs_client *fsc, int page_align) {struct ceph_osd_client *osdc = &fsc->client->osdc; +struct ceph_client_metric *metric = &fsc->mdsc->metric;struct ceph_osd_request *req;int rc =0 ;
@@ -218,6 +219,8 @@ static int ceph_sync_readpages(struct ceph_fs_client *fsc,if (!rc) rc =ceph_osdc_wait_request (osdc, req); +ceph_update_read_latency (metric, req, rc); +ceph_osdc_put_request (req);dout ("readpages result %d\n" , rc);return rc;
@@ -301,6 +304,8 @@ static int ceph_readpage(struct file *filp, struct page *page)static void finish_read (struct ceph_osd_request *req) {struct inode *inode = req->r_inode; +struct ceph_fs_client *fsc =ceph_inode_to_client (inode); +struct ceph_client_metric *metric = &fsc->mdsc->metric;struct ceph_osd_data *osd_data;int rc = req->r_result <=0 ? req->r_result :0 ;int bytes = req->r_result >=0 ? req->r_result :0 ;
@@ -338,6 +343,9 @@ static void finish_read(struct ceph_osd_request *req)put_page (page); bytes -=PAGE_SIZE ; } + +ceph_update_read_latency (metric, req, rc); +kfree (osd_data->pages); }
fs/ceph/debugfs.c +11 -0
@@ -128,8 +128,19 @@ static int metric_show(struct seq_file *s, void *p) {struct ceph_fs_client *fsc = s->private;struct ceph_mds_client *mdsc = fsc->mdsc; +s64 total, sum, avg =0 ;int i; +seq_printf (s,"item total sum_lat(us) avg_lat(us)\n" ); +seq_printf (s,"-----------------------------------------------------\n" ); + + total =percpu_counter_sum (&mdsc->metric.total_reads); + sum =percpu_counter_sum (&mdsc->metric.read_latency_sum); + avg = total ? sum / total :0 ; +seq_printf (s,"%-14s%-12lld%-16lld%lld\n" ,"read" , + total, sum /NSEC_PER_USEC , avg /NSEC_PER_USEC ); + +seq_printf (s,"\n" );seq_printf (s,"item total miss hit\n" );seq_printf (s,"-------------------------------------------------\n" );
fs/ceph/file.c +15 -0
@@ -588,6 +588,7 @@ static ssize_t ceph_sync_read(struct kiocb *iocb, struct iov_iter *to,struct inode *inode =file_inode (file);struct ceph_inode_info *ci =ceph_inode (inode);struct ceph_fs_client *fsc =ceph_inode_to_client (inode); +struct ceph_client_metric *metric = &fsc->mdsc->metric;struct ceph_osd_client *osdc = &fsc->client->osdc;ssize_t ret;u64 off = iocb->ki_pos;
@@ -660,6 +661,9 @@ static ssize_t ceph_sync_read(struct kiocb *iocb, struct iov_iter *to, ret =ceph_osdc_start_request (osdc, req, false);if (!ret) ret =ceph_osdc_wait_request (osdc, req); + +ceph_update_read_latency (metric, req, ret); +ceph_osdc_put_request (req); i_size =i_size_read (inode);
@@ -798,13 +802,20 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req)struct inode *inode = req->r_inode;struct ceph_aio_request *aio_req = req->r_priv;struct ceph_osd_data *osd_data =osd_req_op_extent_osd_data (req,0 ); +struct ceph_fs_client *fsc =ceph_inode_to_client (inode); +struct ceph_client_metric *metric = &fsc->mdsc->metric;BUG_ON (osd_data->type !=CEPH_OSD_DATA_TYPE_BVECS );BUG_ON (!osd_data->num_bvecs); +BUG_ON (!aio_req);dout ("ceph_aio_complete_req %p rc %d bytes %u\n" , inode, rc, osd_data->bvec_pos.iter.bi_size); +/* r_start_stamp == 0 means the request was not submitted */ +if (req->r_start_stamp && !aio_req->write) +ceph_update_read_latency (metric, req, rc); +if (rc == -EOLDSNAPC ) {struct ceph_aio_work *aio_work;BUG_ON (!aio_req->write);
@@ -933,6 +944,7 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter,struct inode *inode =file_inode (file);struct ceph_inode_info *ci =ceph_inode (inode);struct ceph_fs_client *fsc =ceph_inode_to_client (inode); +struct ceph_client_metric *metric = &fsc->mdsc->metric;struct ceph_vino vino;struct ceph_osd_request *req;struct bio_vec *bvecs;
@@ -1049,6 +1061,9 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter,if (!ret) ret =ceph_osdc_wait_request (&fsc->client->osdc, req); +if (!write) +ceph_update_read_latency (metric, req, ret); + size =i_size_read (inode);if (!write) {if (ret == -ENOENT )
fs/ceph/mds_client.c +23 -6
@@ -4182,14 +4182,29 @@ static int ceph_mdsc_metric_init(struct ceph_client_metric *metric)atomic64_set (&metric->total_dentries,0 ); ret =percpu_counter_init (&metric->d_lease_hit,0 ,GFP_KERNEL );if (ret) -return ret; +return ret;; ret =percpu_counter_init (&metric->d_lease_mis,0 ,GFP_KERNEL ); -if (ret) { -percpu_counter_destroy (&metric->d_lease_hit); -return ret; - } +if (ret) + goto err_dlease_mis; -return 0 ; + ret =percpu_counter_init (&metric->total_reads,0 ,GFP_KERNEL ); +if (ret) + goto err_total_reads; + + ret =percpu_counter_init (&metric->read_latency_sum,0 ,GFP_KERNEL ); +if (ret) + goto err_read_latency_sum; + +return ret; + +err_read_latency_sum: +percpu_counter_destroy (&metric->total_reads); +err_total_reads: +percpu_counter_destroy (&metric->d_lease_mis); +err_dlease_mis: +percpu_counter_destroy (&metric->d_lease_hit); + +return ret; }int ceph_mdsc_init (struct ceph_fs_client *fsc)
@@ -4529,6 +4544,8 @@ void ceph_mdsc_destroy(struct ceph_fs_client *fsc)ceph_mdsc_stop (mdsc); +percpu_counter_destroy (&mdsc->metric.read_latency_sum); +percpu_counter_destroy (&mdsc->metric.total_reads);percpu_counter_destroy (&mdsc->metric.d_lease_mis);percpu_counter_destroy (&mdsc->metric.d_lease_hit);
fs/ceph/mds_client.h +2 -7
@@ -16,6 +16,8 @@#include <linux/ceph/mdsmap.h> #include <linux/ceph/auth.h> +#include "metric.h" +/* The first 8 bits are reserved for old ceph releases */ enum ceph_feature_type {CEPHFS_FEATURE_MIMIC =8 ,
@@ -361,13 +363,6 @@ struct cap_wait {int want; }; -/* This is the global metrics */ -struct ceph_client_metric { -atomic64_t total_dentries; -struct percpu_counter d_lease_hit; -struct percpu_counter d_lease_mis; -}; -/* * mds client state */
fs/ceph/metric.h new file +30 -0
@@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _FS_CEPH_MDS_METRIC_H +#define _FS_CEPH_MDS_METRIC_H + +#include <linux/ceph/osd_client.h> + +/* This is the global metrics */ +struct ceph_client_metric { +atomic64_t total_dentries; +struct percpu_counter d_lease_hit; +struct percpu_counter d_lease_mis; + +struct percpu_counter total_reads; +struct percpu_counter read_latency_sum; +}; + +static inline void ceph_update_read_latency (struct ceph_client_metric *m, +struct ceph_osd_request *req, +int rc) +{ +if (!m || !req) +return ; + +if (rc >=0 || rc == -ENOENT || rc == -ETIMEDOUT ) { +s64 latency = req->r_end_stamp - req->r_start_stamp; +percpu_counter_inc (&m->total_reads); +percpu_counter_add (&m->read_latency_sum, latency); + } +} +#endif
Signature · 1 line
2.21.0
PATCH v5 5/10 ceph: add global write latency metric support
In reply to [email protected]From: Xiubo Li <[email protected]>
item total sum_lat(us) avg_lat(us)
-----------------------------------------------------
write 222 5287750000 23818693
URL: https://tracker.ceph.com/issues/43215
--- fs/ceph/addr.c | 10 ++++++++++ fs/ceph/debugfs.c | 6 ++++++ fs/ceph/file.c | 14 +++++++++++--- fs/ceph/mds_client.c | 15 ++++++++++++++- fs/ceph/metric.h | 17 +++++++++++++++++ 5 files changed, 58 insertions(+), 4 deletions(-)
fs/ceph/addr.c +10 -0
@@ -632,6 +632,7 @@ static int ceph_sync_writepages(struct ceph_fs_client *fsc, struct page **pages, int num_pages) {struct ceph_osd_client *osdc = &fsc->client->osdc; +struct ceph_client_metric *metric = &fsc->mdsc->metric;struct ceph_osd_request *req;int rc =0 ;int page_align = off & ~PAGE_MASK ;
@@ -653,6 +654,8 @@ static int ceph_sync_writepages(struct ceph_fs_client *fsc,if (!rc) rc =ceph_osdc_wait_request (osdc, req); +ceph_update_write_latency (metric, req, rc); +ceph_osdc_put_request (req);if (rc ==0 ) rc = len;
@@ -792,6 +795,7 @@ static void writepages_finish(struct ceph_osd_request *req)struct ceph_snap_context *snapc = req->r_snapc;struct address_space *mapping = inode->i_mapping;struct ceph_fs_client *fsc =ceph_inode_to_client (inode); +struct ceph_client_metric *metric = &fsc->mdsc->metric;bool remove_page;dout ("writepages_finish %p rc %d\n" , inode, rc);
@@ -804,6 +808,8 @@ static void writepages_finish(struct ceph_osd_request *req)ceph_clear_error_write (ci); } +ceph_update_write_latency (metric, req, rc); +/* * We lost the cache cap, need to truncate the page before * it is unlocked, otherwise we'd truncate it later in the
@@ -1752,6 +1758,7 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page)struct inode *inode =file_inode (filp);struct ceph_inode_info *ci =ceph_inode (inode);struct ceph_fs_client *fsc =ceph_inode_to_client (inode); +struct ceph_client_metric *metric = &fsc->mdsc->metric;struct ceph_osd_request *req;struct page *page =NULL ;u64 len, inline_version;
@@ -1864,6 +1871,9 @@ int ceph_uninline_data(struct file *filp, struct page *locked_page) err =ceph_osdc_start_request (&fsc->client->osdc, req, false);if (!err) err =ceph_osdc_wait_request (&fsc->client->osdc, req); + +ceph_update_write_latency (metric, req, err); + out_put:ceph_osdc_put_request (req);if (err == -ECANCELED )
fs/ceph/debugfs.c +6 -0
@@ -140,6 +140,12 @@ static int metric_show(struct seq_file *s, void *p)seq_printf (s,"%-14s%-12lld%-16lld%lld\n" ,"read" , total, sum /NSEC_PER_USEC , avg /NSEC_PER_USEC ); + total =percpu_counter_sum (&mdsc->metric.total_writes); + sum =percpu_counter_sum (&mdsc->metric.write_latency_sum); + avg = total ? sum / total :0 ; +seq_printf (s,"%-14s%-12lld%-16lld%lld\n" ,"write" , + total, sum /NSEC_PER_USEC , avg /NSEC_PER_USEC ); +seq_printf (s,"\n" );seq_printf (s,"item total miss hit\n" );seq_printf (s,"-------------------------------------------------\n" );
fs/ceph/file.c +11 -3
@@ -813,8 +813,12 @@ static void ceph_aio_complete_req(struct ceph_osd_request *req) inode, rc, osd_data->bvec_pos.iter.bi_size);/* r_start_stamp == 0 means the request was not submitted */ -if (req->r_start_stamp && !aio_req->write) -ceph_update_read_latency (metric, req, rc); +if (req->r_start_stamp) { +if (aio_req->write) +ceph_update_write_latency (metric, req, rc); +else +ceph_update_read_latency (metric, req, rc); + }if (rc == -EOLDSNAPC ) {struct ceph_aio_work *aio_work;
@@ -1061,7 +1065,9 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter,if (!ret) ret =ceph_osdc_wait_request (&fsc->client->osdc, req); -if (!write) +if (write) +ceph_update_write_latency (metric, req, ret); +else ceph_update_read_latency (metric, req, ret); size =i_size_read (inode);
@@ -1150,6 +1156,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,struct inode *inode =file_inode (file);struct ceph_inode_info *ci =ceph_inode (inode);struct ceph_fs_client *fsc =ceph_inode_to_client (inode); +struct ceph_client_metric *metric = &fsc->mdsc->metric;struct ceph_vino vino;struct ceph_osd_request *req;struct page **pages;
@@ -1235,6 +1242,7 @@ ceph_sync_write(struct kiocb *iocb, struct iov_iter *from, loff_t pos,if (!ret) ret =ceph_osdc_wait_request (&fsc->client->osdc, req); +ceph_update_write_latency (metric, req, ret); out:ceph_osdc_put_request (req);if (ret !=0 ) {
fs/ceph/mds_client.c +14 -1
@@ -4195,8 +4195,19 @@ static int ceph_mdsc_metric_init(struct ceph_client_metric *metric)if (ret) goto err_read_latency_sum; -return ret; + ret =percpu_counter_init (&metric->total_writes,0 ,GFP_KERNEL ); +if (ret) + goto err_total_writes; + ret =percpu_counter_init (&metric->write_latency_sum,0 ,GFP_KERNEL ); +if (ret) + goto err_write_latency_sum; + +return ret; +err_write_latency_sum: +percpu_counter_destroy (&metric->total_writes); +err_total_writes: +percpu_counter_destroy (&metric->read_latency_sum); err_read_latency_sum:percpu_counter_destroy (&metric->total_reads); err_total_reads:
@@ -4544,6 +4555,8 @@ void ceph_mdsc_destroy(struct ceph_fs_client *fsc)ceph_mdsc_stop (mdsc); +percpu_counter_destroy (&mdsc->metric.write_latency_sum); +percpu_counter_destroy (&mdsc->metric.total_writes);percpu_counter_destroy (&mdsc->metric.read_latency_sum);percpu_counter_destroy (&mdsc->metric.total_reads);percpu_counter_destroy (&mdsc->metric.d_lease_mis);
fs/ceph/metric.h +17 -0
@@ -12,6 +12,9 @@ struct ceph_client_metric {struct percpu_counter total_reads;struct percpu_counter read_latency_sum; + +struct percpu_counter total_writes; +struct percpu_counter write_latency_sum; }; static inline void ceph_update_read_latency(struct ceph_client_metric *m,
@@ -27,4 +30,18 @@ static inline void ceph_update_read_latency(struct ceph_client_metric *m, percpu_counter_add(&m->read_latency_sum, latency); } } + +static inline void ceph_update_write_latency (struct ceph_client_metric *m, +struct ceph_osd_request *req, +int rc) +{ +if (!m || !req) +return ; + +if (!rc || rc == -ETIMEDOUT ) { +s64 latency = req->r_end_stamp - req->r_start_stamp; +percpu_counter_inc (&m->total_writes); +percpu_counter_add (&m->write_latency_sum, latency); + } +}#endif
Signature · 1 line
2.21.0
PATCH v5 6/10 ceph: add global metadata perf metric support
In reply to [email protected]From: Xiubo Li <[email protected]>
item total sum_lat(us) avg_lat(us)
-----------------------------------------------------
metadata 1288 24506000 19026
URL: https://tracker.ceph.com/issues/43215
--- fs/ceph/debugfs.c | 6 ++++++ fs/ceph/mds_client.c | 25 +++++++++++++++++++++++++ fs/ceph/metric.h | 13 +++++++++++++ 3 files changed, 44 insertions(+)
fs/ceph/debugfs.c +6 -0
@@ -146,6 +146,12 @@ static int metric_show(struct seq_file *s, void *p)seq_printf (s,"%-14s%-12lld%-16lld%lld\n" ,"write" , total, sum /NSEC_PER_USEC , avg /NSEC_PER_USEC ); + total =percpu_counter_sum (&mdsc->metric.total_metadatas); + sum =percpu_counter_sum (&mdsc->metric.metadata_latency_sum); + avg = total ? sum / total :0 ; +seq_printf (s,"%-14s%-12lld%-16lld%lld\n" ,"metadata" , + total, sum /NSEC_PER_USEC , avg /NSEC_PER_USEC ); +seq_printf (s,"\n" );seq_printf (s,"item total miss hit\n" );seq_printf (s,"-------------------------------------------------\n" );
fs/ceph/mds_client.c +25 -0
@@ -2772,6 +2772,12 @@ static int ceph_mdsc_wait_request(struct ceph_mds_client *mdsc,else err = timeleft;/* killed */ } + +if (!err || err == -EIO ) { +s64 latency = jiffies - req->r_started; +ceph_update_metadata_latency (&mdsc->metric, latency); + } +dout ("do_request waited, got %d\n" , err);mutex_lock (&mdsc->mutex);
@@ -3033,6 +3039,11 @@ static void handle_reply(struct ceph_mds_session *session, struct ceph_msg *msg)/* kick calling process */ complete_request (mdsc, req); + +if (!result || result == -ENOENT ) { +s64 latency = jiffies - req->r_started; +ceph_update_metadata_latency (&mdsc->metric, latency); + } out:ceph_mdsc_put_request (req);return ;
@@ -4203,7 +4214,19 @@ static int ceph_mdsc_metric_init(struct ceph_client_metric *metric)if (ret) goto err_write_latency_sum; + ret =percpu_counter_init (&metric->total_metadatas,0 ,GFP_KERNEL ); +if (ret) + goto err_total_metadatas; + + ret =percpu_counter_init (&metric->metadata_latency_sum,0 ,GFP_KERNEL ); +if (ret) + goto err_metadata_latency_sum; +return ret; +err_metadata_latency_sum: +percpu_counter_destroy (&metric->total_metadatas); +err_total_metadatas: +percpu_counter_destroy (&metric->write_latency_sum); err_write_latency_sum:percpu_counter_destroy (&metric->total_writes); err_total_writes:
@@ -4555,6 +4578,8 @@ void ceph_mdsc_destroy(struct ceph_fs_client *fsc)ceph_mdsc_stop (mdsc); +percpu_counter_destroy (&mdsc->metric.metadata_latency_sum); +percpu_counter_destroy (&mdsc->metric.total_metadatas);percpu_counter_destroy (&mdsc->metric.write_latency_sum);percpu_counter_destroy (&mdsc->metric.total_writes);percpu_counter_destroy (&mdsc->metric.read_latency_sum);
fs/ceph/metric.h +13 -0
@@ -15,6 +15,9 @@ struct ceph_client_metric {struct percpu_counter total_writes;struct percpu_counter write_latency_sum; + +struct percpu_counter total_metadatas; +struct percpu_counter metadata_latency_sum; }; static inline void ceph_update_read_latency(struct ceph_client_metric *m,
@@ -44,4 +47,14 @@ static inline void ceph_update_write_latency(struct ceph_client_metric *m, percpu_counter_add(&m->write_latency_sum, latency); } } + +static inline void ceph_update_metadata_latency (struct ceph_client_metric *m, +s64 latency) +{ +if (!m) +return ; + +percpu_counter_inc (&m->total_metadatas); +percpu_counter_add (&m->metadata_latency_sum, latency); +}#endif
Signature · 1 line
2.21.0
PATCH v5 7/10 ceph: periodically send perf metrics to MDS
In reply to [email protected]From: Xiubo Li <[email protected]>
Add enable/disable sending metrics to MDS debugfs and disabled as default, if it's enabled the kclient will send metrics every second.
This will send global dentry lease hit/miss and read/write/metadata latency metrics and each session's caps hit/miss metric to MDS.
Every time only sends the global metrics once via any availible session.
URL: https://tracker.ceph.com/issues/43215
--- fs/ceph/debugfs.c | 44 +++++++- fs/ceph/mds_client.c | 201 ++++++++++++++++++++++++++++++++--- fs/ceph/mds_client.h | 3 + fs/ceph/metric.h | 76 +++++++++++++ fs/ceph/super.h | 1 + include/linux/ceph/ceph_fs.h | 1 + 6 files changed, 307 insertions(+), 19 deletions(-)
fs/ceph/debugfs.c +42 -2
@@ -124,6 +124,40 @@ static int mdsc_show(struct seq_file *s, void *p)return 0 ; } +/* +* metrics debugfs +*/ +static int sending_metrics_set (void *data,u64 val) +{ +struct ceph_fs_client *fsc = (struct ceph_fs_client *)data; +struct ceph_mds_client *mdsc = fsc->mdsc; + +if (val >1 ) { +pr_err ("Invalid sending metrics set value %llu\n" , val); +return -EINVAL ; + } + +mutex_lock (&mdsc->mutex); + mdsc->sending_metrics = (unsigned int )val; +mutex_unlock (&mdsc->mutex); + +return 0 ; +} + +static int sending_metrics_get (void *data,u64 *val) +{ +struct ceph_fs_client *fsc = (struct ceph_fs_client *)data; +struct ceph_mds_client *mdsc = fsc->mdsc; + +mutex_lock (&mdsc->mutex); + *val = (u64 )mdsc->sending_metrics; +mutex_unlock (&mdsc->mutex); + +return 0 ; +} +DEFINE_SIMPLE_ATTRIBUTE (sending_metrics_fops, sending_metrics_get, + sending_metrics_set,"%llu\n" ); +static int metric_show (struct seq_file *s,void *p) {struct ceph_fs_client *fsc = s->private;
@@ -302,11 +336,9 @@ static int congestion_kb_get(void *data, u64 *val) *val = (u64 )fsc->mount_options->congestion_kb;return 0 ; } -DEFINE_SIMPLE_ATTRIBUTE (congestion_kb_fops, congestion_kb_get, congestion_kb_set,"%llu\n" ); -void ceph_fs_debugfs_cleanup (struct ceph_fs_client *fsc) {dout ("ceph_fs_debugfs_cleanup\n" );
@@ -316,6 +348,7 @@ void ceph_fs_debugfs_cleanup(struct ceph_fs_client *fsc)debugfs_remove (fsc->debugfs_mds_sessions);debugfs_remove (fsc->debugfs_caps);debugfs_remove (fsc->debugfs_metric); +debugfs_remove (fsc->debugfs_sending_metrics);debugfs_remove (fsc->debugfs_mdsc); }
@@ -356,6 +389,13 @@ void ceph_fs_debugfs_init(struct ceph_fs_client *fsc) fsc, &mdsc_show_fops); + fsc->debugfs_sending_metrics = +debugfs_create_file ("sending_metrics" , +0600 , + fsc->client->debugfs_dir, + fsc, + &sending_metrics_fops); + fsc->debugfs_metric = debugfs_create_file("metrics", 0400, fsc->client->debugfs_dir,
fs/ceph/mds_client.c +184 -17
259 lines not on this page · raw message
fs/ceph/mds_client.h +3 -0
@@ -451,6 +451,9 @@ struct ceph_mds_client {struct list_head dentry_leases;/* fifo list */ struct list_head dentry_dir_leases;/* lru list */ +/* metrics */ +unsigned int sending_metrics; +unsigned int ticks;struct ceph_client_metric metric;spinlock_t snapid_map_lock;
fs/ceph/metric.h +76 -0
82 lines not on this page · raw message
fs/ceph/super.h +1 -0
@@ -128,6 +128,7 @@ struct ceph_fs_client {struct dentry *debugfs_congestion_kb;struct dentry *debugfs_bdi;struct dentry *debugfs_mdsc, *debugfs_mdsmap; +struct dentry *debugfs_sending_metrics;struct dentry *debugfs_metric;struct dentry *debugfs_mds_sessions;#endif
include/linux/ceph/ceph_fs.h +1 -0
@@ -130,6 +130,7 @@ struct ceph_dir_layout {#define CEPH_MSG_CLIENT_REQUEST 24#define CEPH_MSG_CLIENT_REQUEST_FORWARD 25#define CEPH_MSG_CLIENT_REPLY 26 +#define CEPH_MSG_CLIENT_METRICS 29#define CEPH_MSG_CLIENT_CAPS 0x310#define CEPH_MSG_CLIENT_LEASE 0x311#define CEPH_MSG_CLIENT_SNAP 0x312
Signature · 1 line
2.21.0
PATCH v5 8/10 ceph: add CEPH_DEFINE_RW_FUNC helper support
In reply to [email protected]From: Xiubo Li <[email protected]>
This will support the string store.
--- include/linux/ceph/debugfs.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
include/linux/ceph/debugfs.h +14 -0
@@ -18,6 +18,20 @@ static const struct file_operations name##_fops = { \ .release = single_release, \ }; +#define CEPH_DEFINE_RW_FUNC (name) \ +static int name##_open(struct inode *inode, struct file *file) \ +{ \ + return single_open(file, name##_show, inode->i_private); \ +} \ + \ +static const struct file_operations name##_fops = { \ + .open = name##_open, \ + .read = seq_read, \ + .write = name##_store, \ + .llseek = seq_lseek, \ + .release = single_release, \ +}; +/* debugfs.c */ extern void ceph_debugfs_init (void );extern void ceph_debugfs_cleanup (void );
Signature · 1 line
2.21.0
PATCH v5 9/10 ceph: add reset metrics support
In reply to [email protected]From: Xiubo Li <[email protected]>
This will reset the most metric counters, except the cap and dentry total numbers.
Sometimes we need to discard the old metrics and start to get new metrics.
URL: https://tracker.ceph.com/issues/43215
--- fs/ceph/debugfs.c | 53 +++++++++++++++++++++++++++++++++++++++++++++-- fs/ceph/super.h | 1 + 2 files changed, 52 insertions(+), 2 deletions(-)
fs/ceph/debugfs.c +51 -2
@@ -217,6 +217,55 @@ static int metric_show(struct seq_file *s, void *p)return 0 ; } +static ssize_t metric_store (struct file *file,const char __user *user_buf, +size_t count,loff_t *ppos) +{ +struct seq_file *s = file->private_data; +struct ceph_fs_client *fsc = s->private; +struct ceph_mds_client *mdsc = fsc->mdsc; +struct ceph_client_metric *metric = &mdsc->metric; +char buf[8 ]; +int i; + +if (copy_from_user (buf, user_buf,8 )) +return -EFAULT ; + +if (strcmp (buf,"reset" )) { +pr_err ("Invalid set value '%s', only 'reset' is valid\n" , buf); +return -EINVAL ; + } + +percpu_counter_set (&metric->d_lease_hit,0 ); +percpu_counter_set (&metric->d_lease_mis,0 ); + +percpu_counter_set (&metric->read_latency_sum,0 ); +percpu_counter_set (&metric->total_reads,0 ); + +percpu_counter_set (&metric->write_latency_sum,0 ); +percpu_counter_set (&metric->total_writes,0 ); + +percpu_counter_set (&metric->metadata_latency_sum,0 ); +percpu_counter_set (&metric->total_metadatas,0 ); + +mutex_lock (&mdsc->mutex); +for (i =0 ; i < mdsc->max_sessions; i++) { +struct ceph_mds_session *session; + + session =__ceph_lookup_mds_session (mdsc, i); +if (!session) +continue ; +percpu_counter_set (&session->i_caps_hit,0 ); +percpu_counter_set (&session->i_caps_mis,0 ); +ceph_put_mds_session (session); + } + +mutex_unlock (&mdsc->mutex); + +return 0 ; +} + +CEPH_DEFINE_RW_FUNC (metric) + static int caps_show_cb(struct inode *inode, struct ceph_cap *cap, void *p) {struct seq_file *s = p;
@@ -313,7 +362,6 @@ static int mds_sessions_show(struct seq_file *s, void *ptr) CEPH_DEFINE_SHOW_FUNC(mdsmap_show) CEPH_DEFINE_SHOW_FUNC(mdsc_show) -CEPH_DEFINE_SHOW_FUNC(metric_show) CEPH_DEFINE_SHOW_FUNC(caps_show) CEPH_DEFINE_SHOW_FUNC(mds_sessions_show)
@@ -349,6 +397,7 @@ void ceph_fs_debugfs_cleanup(struct ceph_fs_client *fsc)debugfs_remove (fsc->debugfs_caps);debugfs_remove (fsc->debugfs_metric);debugfs_remove (fsc->debugfs_sending_metrics); +debugfs_remove (fsc->debugfs_reset_metrics);debugfs_remove (fsc->debugfs_mdsc); }
@@ -400,7 +449,7 @@ void ceph_fs_debugfs_init(struct ceph_fs_client *fsc)0400 , fsc->client->debugfs_dir, fsc, - &metric_show_fops); + &metric_fops); fsc->debugfs_caps = debugfs_create_file("caps", 0400,
fs/ceph/super.h +1 -0
@@ -129,6 +129,7 @@ struct ceph_fs_client {struct dentry *debugfs_bdi;struct dentry *debugfs_mdsc, *debugfs_mdsmap;struct dentry *debugfs_sending_metrics; +struct dentry *debugfs_reset_metrics;struct dentry *debugfs_metric;struct dentry *debugfs_mds_sessions;#endif
Signature · 1 line
2.21.0
PATCH v5 10/10 ceph: send client provided metric flags in client metadata
In reply to [email protected]From: Xiubo Li <[email protected]>
Will send the metric flags to MDS, currently it supports the cap, dentry lease, read latency, write latency and metadata latency.
URL: https://tracker.ceph.com/issues/43435
--- fs/ceph/mds_client.c | 47 ++++++++++++++++++++++++++++++++++++++++++-- fs/ceph/metric.h | 14 +++++++++++++ 2 files changed, 59 insertions(+), 2 deletions(-)
fs/ceph/mds_client.c +45 -2
@@ -1096,6 +1096,41 @@ static void encode_supported_features(void **p, void *end) } } +static const unsigned char metric_bits[] =CEPHFS_METRIC_SPEC_CLIENT_SUPPORTED ; +#define METRIC_BYTES (cnt) (DIV_ROUND_UP((size_t)metric_bits[cnt - 1] + 1, 64) * 8) +static void encode_metric_spec (void **p,void *end) +{ +static const size_t count =ARRAY_SIZE (metric_bits); + +/* header */ +BUG_ON (*p +2 > end); +ceph_encode_8 (p,1 );/* version */ +ceph_encode_8 (p,1 );/* compat */ + +if (count >0 ) { +size_t i; +size_t size =METRIC_BYTES (count); + +BUG_ON (*p +4 +4 + size > end); + +/* metric spec info length */ +ceph_encode_32 (p,4 + size); + +/* metric spec */ +ceph_encode_32 (p, size); +memset (*p,0 , size); +for (i =0 ; i < count; i++) + ((unsigned char *)(*p))[i /8 ] |=BIT (metric_bits[i] %8 ); + *p += size; + }else { +BUG_ON (*p +4 +4 > end); +/* metric spec info length */ +ceph_encode_32 (p,4 ); +/* metric spec */ +ceph_encode_32 (p,0 ); + } +} +/* * session message, specialization for CEPH_SESSION_REQUEST_OPEN * to include additional client metadata fields.
@@ -1135,6 +1170,13 @@ static struct ceph_msg *create_session_open_msg(struct ceph_mds_client *mdsc, u6 size =FEATURE_BYTES (count); extra_bytes +=4 + size; +/* metric spec */ + size =0 ; + count =ARRAY_SIZE (metric_bits); +if (count >0 ) + size =METRIC_BYTES (count); + extra_bytes +=2 +4 +4 + size; +/* Allocate the message */ msg =ceph_msg_new (CEPH_MSG_CLIENT_SESSION ,sizeof (*h) + extra_bytes,GFP_NOFS , false);
@@ -1153,9 +1195,9 @@ static struct ceph_msg *create_session_open_msg(struct ceph_mds_client *mdsc, u6* Serialize client metadata into waiting buffer space, using * the format that userspace expects for map<string, string> * -* ClientSession messages with metadata are v3 +* ClientSession messages with metadata are v4 */ - msg->hdr.version =cpu_to_le16 (3 ); + msg->hdr.version =cpu_to_le16 (4 ); msg->hdr.compat_version =cpu_to_le16 (1 );/* The write pointer, following the session_head structure */
@@ -1178,6 +1220,7 @@ static struct ceph_msg *create_session_open_msg(struct ceph_mds_client *mdsc, u6 }encode_supported_features (&p, end); +encode_metric_spec (&p, end); msg->front.iov_len = p - msg->front.iov_base; msg->hdr.front_len =cpu_to_le32 (msg->front.iov_len);
fs/ceph/metric.h +14 -0
@@ -14,6 +14,20 @@ enum ceph_metric_type { CLIENT_METRIC_TYPE_MAX = CLIENT_METRIC_TYPE_DENTRY_LEASE, }; +/* +* This will always have the highest metric bit value +* as the last element of the array. +*/ +#define CEPHFS_METRIC_SPEC_CLIENT_SUPPORTED { \ + CLIENT_METRIC_TYPE_CAP_INFO, \ + CLIENT_METRIC_TYPE_READ_LATENCY, \ + CLIENT_METRIC_TYPE_WRITE_LATENCY, \ + CLIENT_METRIC_TYPE_METADATA_LATENCY, \ + CLIENT_METRIC_TYPE_DENTRY_LEASE, \ + \ + CLIENT_METRIC_TYPE_MAX, \ +} +/* metric caps header */ struct ceph_metric_cap {__le32 type;/* ceph metric type */
Signature · 1 line
2.21.0
Ilya Dryomov
maintainerQuoted 18 lines · On Tue, Jan 28, 2020 at 2:03 PM <[email protected]> wrote:
From: Xiubo Li <[email protected]>
Changed in V2:
- add read/write/metadata latency metric support.
- add and send client provided metric flags in client metadata
- addressed the comments from Ilya and merged the 4/4 patch into 3/4.
- addressed all the other comments in v1 series.Changed in V3:
- addressed Jeff's comments and let's the callers do the metric counting.
- with some small fixes for the read/write latency
- tested based on the latest testing branchChanged in V4:
- fix the lock issue
Quoted 7 lines
Changed in V5:
- addr_end_stampfor the osdc request
- delete reset metric and move it to metric sysfs
- move ceph_osdc_{read,write}pages to ceph.ko
- use percpu counters instead for read/write/metadata latenciesIt will send the metrics to the MDSs every second if
sending_metricsis enabled, disable as default.
Hi Xiubo,
What is this series based on? "[PATCH v5 01/10] ceph: add caps perf metric for each session" changes metric_show() in fs/ceph/debugfs.c, but there is no such function upstream or in the testing branch.
Thanks,
Ilya
Quoted 24 lines · Ilya Dryomov
Quoted 23 lines · On Tue, Jan 28, 2020 at 2:03 PM <[email protected]> wrote:
From: Xiubo Li <[email protected]>
Changed in V2:
- add read/write/metadata latency metric support.
- add and send client provided metric flags in client metadata
- addressed the comments from Ilya and merged the 4/4 patch into 3/4.
- addressed all the other comments in v1 series.Changed in V3:
- addressed Jeff's comments and let's the callers do the metric counting.
- with some small fixes for the read/write latency
- tested based on the latest testing branchChanged in V4:
- fix the lock issueChanged in V5:
- addr_end_stampfor the osdc request
- delete reset metric and move it to metric sysfs
- move ceph_osdc_{read,write}pages to ceph.ko
- use percpu counters instead for read/write/metadata latencies
Quoted 6 lines
Quoted 1 line
It will send the metrics to the MDSs every second if
sending_metricsis enabled, disable as default.Hi Xiubo,
What is this series based on? "[PATCH v5 01/10] ceph: add caps perf metric for each session" changes
metric_show()infs/ceph/debugfs.c, but there is no such function upstream or in the testing branch.
There actually 11 patches, I missed the first one, will resend it.
Thanks.
Quoted 4 lines
Thanks,
Ilya