Skip to contentSkip to conversation

ceph: add min/max/stdev latency support

PATCH In review [email protected] 3 participants 9 messages Mar 18, 2020 Ceph client

lore
Share & tools
lore
#0 lore raw

v4 cover ceph: add min/max/stdev latency support

From: Xiubo Li <[email protected]>

Changed in V4:
- fix the 32-bit arches div errors by using DIV64_U64_ROUND_CLOSEST instead. [1/4]
- rebase and combine the stdev patch series [3/4][4/4]
- remove the sum latency showing, which makes no sense for debugging, if it is really needed in some case then just do (avg * total) in userland. [4/4]
- switch {read/write/metadata}_latency_sum to atomic type since it will be readed very time when updating the latencies to calculate the stdev. [4/4]

Changed in V2:
- switch spin lock to cmpxchg [1/4]

Changed in V3:
- add the __update_min/max_latency helpers [1/4]

# cat /sys/kernel/debug/ceph/0f923fe5-00e6-4866-bf01-2027cb75e94b.client4150/metrics
item          total       avg_lat(us)     min_lat(us)     max_lat(us)     stdev(us)
-----------------------------------------------------------------------------------
read          2312        9000            1000            100000          607.4
write         21777       925000          2000            44551000        29700.3
metadata      6           4179000         1000            21414000        19590.8

item total miss hit
-------------------------------------------------

d_lease       2               0               11
caps          2               14              398418

Xiubo Li (4):
ceph: switch to DIV64_U64_ROUND_CLOSEST to support 32-bit arches
ceph: add min/max latency support for read/write/metadata metrics
ceph: move the metric helpers into one separate file
ceph: add standard deviation support for read/write/metadata perf
metric

 fs/ceph/Makefile     |   2 +-
 fs/ceph/debugfs.c    |  89 ++++++++++++++++++------
 fs/ceph/mds_client.c |  83 +---------------------
 fs/ceph/metric.c     | 190 +++++++++++++++++++++++++++++++++++++++++++++++++++
 fs/ceph/metric.h     |  79 +++++++++++----------
 5 files changed, 297 insertions(+), 146 deletions(-)
 create mode 100644 fs/ceph/metric.c
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH v4 1/4 ceph: switch to DIV64_U64_ROUND_CLOSEST to support 32-bit arches

From: Xiubo Li <[email protected]>

fs/ceph/debugfs.c:140: undefined reference to `__divdi3' Use math64 helpers to avoid 64-bit div on 32-bit arches.

---
 fs/ceph/debugfs.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
fs/ceph/debugfs.c +3 -3
index 60f3e307..95e8693 100644
@@ -137,19 +137,19 @@ static int metric_show(struct seq_file *s, void *p)
 	total = percpu_counter_sum(&mdsc->metric.total_reads);
 	sum = percpu_counter_sum(&mdsc->metric.read_latency_sum);
 	sum = jiffies_to_usecs(sum);
-	avg = total ? sum / total : 0;
+	avg = total ? DIV64_U64_ROUND_CLOSEST(sum, total) : 0;
 	seq_printf(s, "%-14s%-12lld%-16lld%lld\n", "read", total, sum, avg);
 
 	total = percpu_counter_sum(&mdsc->metric.total_writes);
 	sum = percpu_counter_sum(&mdsc->metric.write_latency_sum);
 	sum = jiffies_to_usecs(sum);
-	avg = total ? sum / total : 0;
+	avg = total ? DIV64_U64_ROUND_CLOSEST(sum, total) : 0;
 	seq_printf(s, "%-14s%-12lld%-16lld%lld\n", "write", total, sum, avg);
 
 	total = percpu_counter_sum(&mdsc->metric.total_metadatas);
 	sum = percpu_counter_sum(&mdsc->metric.metadata_latency_sum);
 	sum = jiffies_to_usecs(sum);
-	avg = total ? sum / total : 0;
+	avg = total ? DIV64_U64_ROUND_CLOSEST(sum, total) : 0;
 	seq_printf(s, "%-14s%-12lld%-16lld%lld\n", "metadata", total, sum, avg);
 
 	seq_printf(s, "\n");
Signature · 1 line
1.8.3.1
#2 lore raw

PATCH v4 2/4 ceph: add min/max latency support for read/write/metadata metrics

In reply to [email protected]

From: Xiubo Li <[email protected]>

These will be very useful help diagnose problems.

URL: https://tracker.ceph.com/issues/44533

---
 fs/ceph/debugfs.c    | 27 +++++++++++++++++++++------
 fs/ceph/mds_client.c |  9 +++++++++
 fs/ceph/metric.h     | 51 ++++++++++++++++++++++++++++++++++++++++++++++++---
 3 files changed, 78 insertions(+), 9 deletions(-)
fs/ceph/debugfs.c +21 -6
index 95e8693..01b95fe 100644
@@ -129,28 +129,43 @@ static int metric_show(struct seq_file *s, void *p)
 	struct ceph_fs_client *fsc = s->private;
 	struct ceph_mds_client *mdsc = fsc->mdsc;
 	int i, nr_caps = 0;
-	s64 total, sum, avg = 0;
+	s64 total, sum, avg = 0, min, max;
 
-	seq_printf(s, "item          total       sum_lat(us)     avg_lat(us)\n");
-	seq_printf(s, "-----------------------------------------------------\n");
+	seq_printf(s, "item          total       sum_lat(us)     avg_lat(us)     min_lat(us)     max_lat(us)\n");
+	seq_printf(s, "-------------------------------------------------------------------------------------\n");
 
 	total = percpu_counter_sum(&mdsc->metric.total_reads);
 	sum = percpu_counter_sum(&mdsc->metric.read_latency_sum);
 	sum = jiffies_to_usecs(sum);
 	avg = total ? DIV64_U64_ROUND_CLOSEST(sum, total) : 0;
-	seq_printf(s, "%-14s%-12lld%-16lld%lld\n", "read", total, sum, avg);
+	min = atomic64_read(&mdsc->metric.read_latency_min);
+	min = jiffies_to_usecs(min == S64_MAX ? 0 : min);
+	max = atomic64_read(&mdsc->metric.read_latency_max);
+	max = jiffies_to_usecs(max);
+	seq_printf(s, "%-14s%-12lld%-16lld%-16lld%-16lld%lld\n", "read",
+		   total, sum, avg, min, max);
 
 	total = percpu_counter_sum(&mdsc->metric.total_writes);
 	sum = percpu_counter_sum(&mdsc->metric.write_latency_sum);
 	sum = jiffies_to_usecs(sum);
 	avg = total ? DIV64_U64_ROUND_CLOSEST(sum, total) : 0;
-	seq_printf(s, "%-14s%-12lld%-16lld%lld\n", "write", total, sum, avg);
+	min = atomic64_read(&mdsc->metric.write_latency_min);
+	min = jiffies_to_usecs(min == S64_MAX ? 0 : min);
+	max = atomic64_read(&mdsc->metric.write_latency_max);
+	max = jiffies_to_usecs(max);
+	seq_printf(s, "%-14s%-12lld%-16lld%-16lld%-16lld%lld\n", "write",
+		   total, sum, avg, min, max);
 
 	total = percpu_counter_sum(&mdsc->metric.total_metadatas);
 	sum = percpu_counter_sum(&mdsc->metric.metadata_latency_sum);
 	sum = jiffies_to_usecs(sum);
 	avg = total ? DIV64_U64_ROUND_CLOSEST(sum, total) : 0;
-	seq_printf(s, "%-14s%-12lld%-16lld%lld\n", "metadata", total, sum, avg);
+	min = atomic64_read(&mdsc->metric.metadata_latency_min);
+	min = jiffies_to_usecs(min == S64_MAX ? 0 : min);
+	max = atomic64_read(&mdsc->metric.metadata_latency_max);
+	max = jiffies_to_usecs(max);
+	seq_printf(s, "%-14s%-12lld%-16lld%-16lld%-16lld%lld\n", "metadata",
+		   total, sum, avg, min, max);
 
 	seq_printf(s, "\n");
 	seq_printf(s, "item          total           miss            hit\n");
fs/ceph/mds_client.c +9 -0
index 5c03ed3..a3b2810 100644
@@ -4358,6 +4358,9 @@ static int ceph_mdsc_metric_init(struct ceph_client_metric *metric)
 	if (ret)
 		goto err_read_latency_sum;
 
+	atomic64_set(&metric->read_latency_min, S64_MAX);
+	atomic64_set(&metric->read_latency_max, 0);
+
 	ret = percpu_counter_init(&metric->total_writes, 0, GFP_KERNEL);
 	if (ret)
 		goto err_total_writes;
@@ -4366,6 +4369,9 @@ static int ceph_mdsc_metric_init(struct ceph_client_metric *metric)
 	if (ret)
 		goto err_write_latency_sum;
 
+	atomic64_set(&metric->write_latency_min, S64_MAX);
+	atomic64_set(&metric->write_latency_max, 0);
+
 	ret = percpu_counter_init(&metric->total_metadatas, 0, GFP_KERNEL);
 	if (ret)
 		goto err_total_metadatas;
@@ -4374,6 +4380,9 @@ static int ceph_mdsc_metric_init(struct ceph_client_metric *metric)
 	if (ret)
 		goto err_metadata_latency_sum;
 
+	atomic64_set(&metric->metadata_latency_min, S64_MAX);
+	atomic64_set(&metric->metadata_latency_max, 0);
+
 	return 0;
 
 err_metadata_latency_sum:
fs/ceph/metric.h +48 -3

103 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#3 lore raw

PATCH v4 3/4 ceph: move the metric helpers into one separate file

In reply to [email protected]

From: Xiubo Li <[email protected]>

The inline is not proper any more dues to the helpers becomes larger.

URL: https://tracker.ceph.com/issues/44534

---
 fs/ceph/Makefile     |   2 +-
 fs/ceph/mds_client.c |  92 +--------------------------
 fs/ceph/metric.c     | 173 +++++++++++++++++++++++++++++++++++++++++++++++++++
 fs/ceph/metric.h     |  87 +++++---------------------
 4 files changed, 190 insertions(+), 164 deletions(-)
 create mode 100644 fs/ceph/metric.c
fs/ceph/Makefile +1 -1
index 0a0823d..50c635d 100644
@@ -8,7 +8,7 @@ obj-$(CONFIG_CEPH_FS) += ceph.o
 ceph-y := super.o inode.o dir.o file.o locks.o addr.o ioctl.o \
 	export.o caps.o snap.o xattr.o quota.o io.o \
 	mds_client.o mdsmap.o strings.o ceph_frag.o \
-	debugfs.o util.o
+	debugfs.o util.o metric.o
 
 ceph-$(CONFIG_CEPH_FSCACHE) += cache.o
 ceph-$(CONFIG_CEPH_FS_POSIX_ACL) += acl.o
fs/ceph/mds_client.c +1 -91

104 lines not on this page · raw message

fs/ceph/metric.c new file +173 -0

173 lines not on this page · raw message

fs/ceph/metric.h +15 -72

97 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#4 lore raw

PATCH v4 4/4 ceph: add standard deviation support for read/write/metadata perf metric

In reply to [email protected]

From: Xiubo Li <[email protected]>

Switch {read/write/metadata}_latency_sum to atomic type and remove {read/write/metadata}_latency_sum showing in the debugfs, which makes no sense.

URL: https://tracker.ceph.com/issues/44534

---
 fs/ceph/debugfs.c | 88 +++++++++++++++++++++++++++++++++----------------
 fs/ceph/metric.c  | 99 ++++++++++++++++++++++++++++++++-----------------------
 fs/ceph/metric.h  | 18 ++++++----
 3 files changed, 129 insertions(+), 76 deletions(-)
fs/ceph/debugfs.c +59 -29
index 01b95fe..21f5663 100644
@@ -1,6 +1,7 @@
 // SPDX-License-Identifier: GPL-2.0
 #include <linux/ceph/ceph_debug.h>
 
+#include <linux/kernel.h>
 #include <linux/device.h>
 #include <linux/slab.h>
 #include <linux/module.h>
@@ -124,48 +125,77 @@ static int mdsc_show(struct seq_file *s, void *p)
 	return 0;
 }
 
+static s64 get_avg(atomic64_t *totalp, atomic64_t *sump, spinlock_t *lockp,
+		   s64 *total)
+{
+	s64 n, sum, avg = 0;
+
+	spin_lock(lockp);
+	n = atomic64_read(totalp);
+	sum = atomic64_read(sump);
+	spin_unlock(lockp);
+
+	if (likely(n))
+		avg = DIV64_U64_ROUND_CLOSEST(sum, n);
+
+	*total = n;
+	return avg;
+}
+
+#define METRIC(name, total, avg, min, max, sq)	{			\
+	s64 _total, _avg, _min, _max, _sq, _st, _re = 0;		\
+	_avg = jiffies_to_usecs(avg);					\
+	_min = jiffies_to_usecs(min == S64_MAX ? 0 : min);		\
+	_max = jiffies_to_usecs(max);					\
+	_total = total - 1;						\
+	_sq = _total > 0 ? DIV64_U64_ROUND_CLOSEST(sq, _total) : 0;	\
+	_sq = jiffies_to_usecs(_sq);					\
+	_st = int_sqrt64(_sq);						\
+	if (_st > 0) {							\
+		_re = 5 * (_sq - (_st * _st));				\
+		_re = _re > 0 ? _re - 1 : 0;				\
+		_re = _st > 0 ? div64_s64(_re, _st) : 0;		\
+	}								\
+	seq_printf(s, "%-14s%-12lld%-16lld%-16lld%-16lld%lld.%lld\n",	\
+		   name, total, _avg, _min, _max, _st, _re);		\
+}
+
 static int metric_show(struct seq_file *s, void *p)
 {
 	struct ceph_fs_client *fsc = s->private;
 	struct ceph_mds_client *mdsc = fsc->mdsc;
 	int i, nr_caps = 0;
-	s64 total, sum, avg = 0, min, max;
+	s64 total, avg, min, max, sq;
 
-	seq_printf(s, "item          total       sum_lat(us)     avg_lat(us)     min_lat(us)     max_lat(us)\n");
-	seq_printf(s, "-------------------------------------------------------------------------------------\n");
+	seq_printf(s, "item          total       avg_lat(us)     min_lat(us)     max_lat(us)     stdev(us)\n");
+	seq_printf(s, "-----------------------------------------------------------------------------------\n");
 
-	total = percpu_counter_sum(&mdsc->metric.total_reads);
-	sum = percpu_counter_sum(&mdsc->metric.read_latency_sum);
-	sum = jiffies_to_usecs(sum);
-	avg = total ? DIV64_U64_ROUND_CLOSEST(sum, total) : 0;
+	avg = get_avg(&mdsc->metric.total_reads,
+		      &mdsc->metric.read_latency_sum,
+		      &mdsc->metric.read_latency_lock,
+		      &total);
 	min = atomic64_read(&mdsc->metric.read_latency_min);
-	min = jiffies_to_usecs(min == S64_MAX ? 0 : min);
 	max = atomic64_read(&mdsc->metric.read_latency_max);
-	max = jiffies_to_usecs(max);
-	seq_printf(s, "%-14s%-12lld%-16lld%-16lld%-16lld%lld\n", "read",
-		   total, sum, avg, min, max);
-
-	total = percpu_counter_sum(&mdsc->metric.total_writes);
-	sum = percpu_counter_sum(&mdsc->metric.write_latency_sum);
-	sum = jiffies_to_usecs(sum);
-	avg = total ? DIV64_U64_ROUND_CLOSEST(sum, total) : 0;
+	sq = percpu_counter_sum(&mdsc->metric.read_latency_sq_sum);
+	METRIC("read", total, avg, min, max, sq);
+
+	avg = get_avg(&mdsc->metric.total_writes,
+		      &mdsc->metric.write_latency_sum,
+		      &mdsc->metric.write_latency_lock,
+		      &total);
 	min = atomic64_read(&mdsc->metric.write_latency_min);
-	min = jiffies_to_usecs(min == S64_MAX ? 0 : min);
 	max = atomic64_read(&mdsc->metric.write_latency_max);
-	max = jiffies_to_usecs(max);
-	seq_printf(s, "%-14s%-12lld%-16lld%-16lld%-16lld%lld\n", "write",
-		   total, sum, avg, min, max);
-
-	total = percpu_counter_sum(&mdsc->metric.total_metadatas);
-	sum = percpu_counter_sum(&mdsc->metric.metadata_latency_sum);
-	sum = jiffies_to_usecs(sum);
-	avg = total ? DIV64_U64_ROUND_CLOSEST(sum, total) : 0;
+	sq = percpu_counter_sum(&mdsc->metric.write_latency_sq_sum);
+	METRIC("write", total, avg, min, max, sq);
+
+	avg = get_avg(&mdsc->metric.total_metadatas,
+		      &mdsc->metric.metadata_latency_sum,
+		      &mdsc->metric.metadata_latency_lock,
+		      &total);
 	min = atomic64_read(&mdsc->metric.metadata_latency_min);
-	min = jiffies_to_usecs(min == S64_MAX ? 0 : min);
 	max = atomic64_read(&mdsc->metric.metadata_latency_max);
-	max = jiffies_to_usecs(max);
-	seq_printf(s, "%-14s%-12lld%-16lld%-16lld%-16lld%lld\n", "metadata",
-		   total, sum, avg, min, max);
+	sq = percpu_counter_sum(&mdsc->metric.metadata_latency_sq_sum);
+	METRIC("metadata", total, avg, min, max, sq);
 
 	seq_printf(s, "\n");
 	seq_printf(s, "item          total           miss            hit\n");
fs/ceph/metric.c +58 -41

161 lines not on this page · raw message

fs/ceph/metric.h +12 -6
index f139aff..d63b95e 100644
@@ -14,18 +14,24 @@ struct ceph_client_metric {
 	struct percpu_counter i_caps_hit;
 	struct percpu_counter i_caps_mis;
 
-	struct percpu_counter total_reads;
-	struct percpu_counter read_latency_sum;
+	struct percpu_counter read_latency_sq_sum;
+	spinlock_t read_latency_lock;
+	atomic64_t total_reads;
+	atomic64_t read_latency_sum;
 	atomic64_t read_latency_min;
 	atomic64_t read_latency_max;
 
-	struct percpu_counter total_writes;
-	struct percpu_counter write_latency_sum;
+	struct percpu_counter write_latency_sq_sum;
+	spinlock_t write_latency_lock;
+	atomic64_t total_writes;
+	atomic64_t write_latency_sum;
 	atomic64_t write_latency_min;
 	atomic64_t write_latency_max;
 
-	struct percpu_counter total_metadatas;
-	struct percpu_counter metadata_latency_sum;
+	struct percpu_counter metadata_latency_sq_sum;
+	spinlock_t metadata_latency_lock;
+	atomic64_t total_metadatas;
+	atomic64_t metadata_latency_sum;
 	atomic64_t metadata_latency_min;
 	atomic64_t metadata_latency_max;
 };
Signature · 1 line
1.8.3.1
maintainer
#5 lore raw
In reply to [email protected]
Quoted 40 lines · On Wed, Mar 18, 2020 at 6:46 AM <[email protected]> wrote:

From: Xiubo Li <[email protected]>

Changed in V4:
- fix the 32-bit arches div errors by using DIV64_U64_ROUND_CLOSEST instead. [1/4]
- rebase and combine the stdev patch series [3/4][4/4]
- remove the sum latency showing, which makes no sense for debugging, if it is really needed in some case then just do (avg * total) in userland. [4/4]
- switch {read/write/metadata}_latency_sum to atomic type since it will be readed very time when updating the latencies to calculate the stdev. [4/4]

Changed in V2:
- switch spin lock to cmpxchg [1/4]

Changed in V3:
- add the __update_min/max_latency helpers [1/4]

# cat /sys/kernel/debug/ceph/0f923fe5-00e6-4866-bf01-2027cb75e94b.client4150/metrics
item          total       avg_lat(us)     min_lat(us)     max_lat(us)     stdev(us)
-----------------------------------------------------------------------------------
read          2312        9000            1000            100000          607.4
write         21777       925000          2000            44551000        29700.3
metadata      6           4179000         1000            21414000        19590.8

item total miss hit
-------------------------------------------------

d_lease       2               0               11
caps          2               14              398418

Xiubo Li (4):
ceph: switch to DIV64_U64_ROUND_CLOSEST to support 32-bit arches
ceph: add min/max latency support for read/write/metadata metrics
ceph: move the metric helpers into one separate file
ceph: add standard deviation support for read/write/metadata perf
metric

Quoted 7 lines
 fs/ceph/Makefile     |   2 +-
 fs/ceph/debugfs.c    |  89 ++++++++++++++++++------
 fs/ceph/mds_client.c |  83 +---------------------
 fs/ceph/metric.c     | 190 +++++++++++++++++++++++++++++++++++++++++++++++++++
 fs/ceph/metric.h     |  79 +++++++++++----------
 5 files changed, 297 insertions(+), 146 deletions(-)
 create mode 100644 fs/ceph/metric.c

Hi Xiubo,

I think these additions need to be merged with your previous series, so that the history is clean. Ideally the whole thing would start with a single patch adding all of the metrics infrastructure to metric.[ch], followed by patches introducing new metrics and ceph_update_*() calls.

Related metrics and ceph_update_*() calls should be added together. No point in splitting read and write OSD latency in two patches as they touch the same functions in addr.c and file.c.

Thanks,

Ilya

author
#6 lore raw
Quoted 49 lines · Ilya Dryomov
Quoted 39 lines · On Wed, Mar 18, 2020 at 6:46 AM <[email protected]> wrote:

From: Xiubo Li <[email protected]>

Changed in V4:
- fix the 32-bit arches div errors by using DIV64_U64_ROUND_CLOSEST instead. [1/4]
- rebase and combine the stdev patch series [3/4][4/4]
- remove the sum latency showing, which makes no sense for debugging, if it is really needed in some case then just do (avg * total) in userland. [4/4]
- switch {read/write/metadata}_latency_sum to atomic type since it will be readed very time when updating the latencies to calculate the stdev. [4/4]

Changed in V2:
- switch spin lock to cmpxchg [1/4]

Changed in V3:
- add the __update_min/max_latency helpers [1/4]

# cat /sys/kernel/debug/ceph/0f923fe5-00e6-4866-bf01-2027cb75e94b.client4150/metrics
item          total       avg_lat(us)     min_lat(us)     max_lat(us)     stdev(us)
-----------------------------------------------------------------------------------
read          2312        9000            1000            100000          607.4
write         21777       925000          2000            44551000        29700.3
metadata      6           4179000         1000            21414000        19590.8

item total miss hit
-------------------------------------------------

d_lease       2               0               11
caps          2               14              398418

Xiubo Li (4):
ceph: switch to DIV64_U64_ROUND_CLOSEST to support 32-bit arches
ceph: add min/max latency support for read/write/metadata metrics
ceph: move the metric helpers into one separate file
ceph: add standard deviation support for read/write/metadata perf
metric

Quoted 7 lines
  fs/ceph/Makefile     |   2 +-
  fs/ceph/debugfs.c    |  89 ++++++++++++++++++------
  fs/ceph/mds_client.c |  83 +---------------------
  fs/ceph/metric.c     | 190 +++++++++++++++++++++++++++++++++++++++++++++++++++
  fs/ceph/metric.h     |  79 +++++++++++----------
  5 files changed, 297 insertions(+), 146 deletions(-)

create mode 100644 fs/ceph/metric.c

Hi Xiubo,

Quoted 8 lines

I think these additions need to be merged with your previous series, so that the history is clean. Ideally the whole thing would start with a single patch adding all of the metrics infrastructure to metric.[ch], followed by patches introducing new metrics and ceph_update_*() calls.

Related metrics and ceph_update_*() calls should be added together. No point in splitting read and write OSD latency in two patches as they touch the same functions in addr.c and file.c.

Hi Ilya,

Yeah, it makes sense and I will merge all the related patch series about the metric and post it again.

Thanks

BRs

Quoted 4 lines

Thanks,

Ilya

#7 lore raw
Quoted 55 lines · Xiubo Li
Quoted 54 lines · Ilya Dryomov
Quoted 39 lines · On Wed, Mar 18, 2020 at 6:46 AM <[email protected]> wrote:

From: Xiubo Li <[email protected]>

Changed in V4:
- fix the 32-bit arches div errors by using DIV64_U64_ROUND_CLOSEST instead. [1/4]
- rebase and combine the stdev patch series [3/4][4/4]
- remove the sum latency showing, which makes no sense for debugging, if it is really needed in some case then just do (avg * total) in userland. [4/4]
- switch {read/write/metadata}_latency_sum to atomic type since it will be readed very time when updating the latencies to calculate the stdev. [4/4]

Changed in V2:
- switch spin lock to cmpxchg [1/4]

Changed in V3:
- add the __update_min/max_latency helpers [1/4]

# cat /sys/kernel/debug/ceph/0f923fe5-00e6-4866-bf01-2027cb75e94b.client4150/metrics
item          total       avg_lat(us)     min_lat(us)     max_lat(us)     stdev(us)
-----------------------------------------------------------------------------------
read          2312        9000            1000            100000          607.4
write         21777       925000          2000            44551000        29700.3
metadata      6           4179000         1000            21414000        19590.8

item total miss hit
-------------------------------------------------

d_lease       2               0               11
caps          2               14              398418

Xiubo Li (4):
ceph: switch to DIV64_U64_ROUND_CLOSEST to support 32-bit arches
ceph: add min/max latency support for read/write/metadata metrics
ceph: move the metric helpers into one separate file
ceph: add standard deviation support for read/write/metadata perf
metric

Quoted 7 lines
  fs/ceph/Makefile     |   2 +-
  fs/ceph/debugfs.c    |  89 ++++++++++++++++++------
  fs/ceph/mds_client.c |  83 +---------------------
  fs/ceph/metric.c     | 190 +++++++++++++++++++++++++++++++++++++++++++++++++++
  fs/ceph/metric.h     |  79 +++++++++++----------
  5 files changed, 297 insertions(+), 146 deletions(-)

create mode 100644 fs/ceph/metric.c

Hi Xiubo,

I think these additions need to be merged with your previous series, so that the history is clean. Ideally the whole thing would start with a single patch adding all of the metrics infrastructure to metric.[ch], followed by patches introducing new metrics and ceph_update_*() calls.

Quoted 9 lines
Quoted 3 lines

Related metrics and ceph_update_*() calls should be added together. No point in splitting read and write OSD latency in two patches as they touch the same functions in addr.c and file.c.

Hi Ilya,

Yeah, it makes sense and I will merge all the related patch series about the metric and post it again.

Sounds good. I've gone ahead and dropped all of the metrics patches from the "testing" branch for now. Please resend the whole series and I'll re-merge them.

Thanks,

Signature · 1 line
Jeff Layton <[email protected]>
author
#8 lore raw
Quoted 59 lines · Jeff Layton
Quoted 58 lines · Xiubo Li
Quoted 57 lines · Ilya Dryomov
Quoted 39 lines · On Wed, Mar 18, 2020 at 6:46 AM <[email protected]> wrote:

From: Xiubo Li <[email protected]>

Changed in V4:
- fix the 32-bit arches div errors by using DIV64_U64_ROUND_CLOSEST instead. [1/4]
- rebase and combine the stdev patch series [3/4][4/4]
- remove the sum latency showing, which makes no sense for debugging, if it is really needed in some case then just do (avg * total) in userland. [4/4]
- switch {read/write/metadata}_latency_sum to atomic type since it will be readed very time when updating the latencies to calculate the stdev. [4/4]

Changed in V2:
- switch spin lock to cmpxchg [1/4]

Changed in V3:
- add the __update_min/max_latency helpers [1/4]

# cat /sys/kernel/debug/ceph/0f923fe5-00e6-4866-bf01-2027cb75e94b.client4150/metrics
item          total       avg_lat(us)     min_lat(us)     max_lat(us)     stdev(us)
-----------------------------------------------------------------------------------
read          2312        9000            1000            100000          607.4
write         21777       925000          2000            44551000        29700.3
metadata      6           4179000         1000            21414000        19590.8

item total miss hit
-------------------------------------------------

d_lease       2               0               11
caps          2               14              398418

Xiubo Li (4):
ceph: switch to DIV64_U64_ROUND_CLOSEST to support 32-bit arches
ceph: add min/max latency support for read/write/metadata metrics
ceph: move the metric helpers into one separate file
ceph: add standard deviation support for read/write/metadata perf
metric

Quoted 7 lines
   fs/ceph/Makefile     |   2 +-
   fs/ceph/debugfs.c    |  89 ++++++++++++++++++------
   fs/ceph/mds_client.c |  83 +---------------------
   fs/ceph/metric.c     | 190 +++++++++++++++++++++++++++++++++++++++++++++++++++
   fs/ceph/metric.h     |  79 +++++++++++----------
   5 files changed, 297 insertions(+), 146 deletions(-)

create mode 100644 fs/ceph/metric.c

Hi Xiubo,

I think these additions need to be merged with your previous series, so that the history is clean. Ideally the whole thing would start with a single patch adding all of the metrics infrastructure to metric.[ch], followed by patches introducing new metrics and ceph_update_*() calls.

Related metrics and ceph_update_*() calls should be added together. No point in splitting read and write OSD latency in two patches as they touch the same functions in addr.c and file.c.

Quoted 8 lines
Quoted 5 lines

Hi Ilya,

Yeah, it makes sense and I will merge all the related patch series about the metric and post it again.

Sounds good. I've gone ahead and dropped all of the metrics patches from the "testing" branch for now. Please resend the whole series and I'll re-merge them.

Sure, thanks Jeff.

BRs

Quoted 1 line

Thanks,