Skip to contentSkip to conversation

Time stats library (from bcache/bcachefs) & generic latency instrumentation

PATCH In review Kent Overstreet 3 participants 11 messages Aug 29, 2022 – Sep 1, 2022

Subsystems (5)
lore
Share & tools
lore

Kent Overstreet

author maintainer
#0 lore raw

Hey Coly, this is a fragment from a larger patch series - I wanted to send it out to you now for review because it changes the bcache sysfs interface for time_stats.

This series takes the time_stats code from bcachefs (which has some additional improvements), moves it to lib/ converts bcache to use it (otherwise we get clashes with the time_stats name) - and uses it for some really cool new latency instrumentation.

These patches won't apply/build, full patch series for any interested is here:

https://evilpiepirate.org/git/bcachefs.git/log/?h=alloc_tags

author maintainer
#1 lore raw

PATCH 1/3 lib/time_stats: New library for statistics on events

This adds a small new library for tracking statistics on events that have a duration, i.e. a start and end time.

- number of events
- rate/frequency
- average duration
- max duration
- duration quantiles

This code comes from bcachefs, and originally bcache: the next patch will be converting bcache to use this version, and a subsequent patch will be using code_tagging to instrument all wait_event() calls in the kernel.

---
 include/linux/time_stats.h |  44 +++++++
 lib/Kconfig.debug          |   3 +
 lib/Makefile               |   1 +
 lib/time_stats.c           | 236 +++++++++++++++++++++++++++++++++++++
 4 files changed, 284 insertions(+)
 create mode 100644 include/linux/time_stats.h
 create mode 100644 lib/time_stats.c
include/linux/time_stats.h new file +44 -0
new file mode 100644
index 0000000000..7ae929e6f8
@@ -0,0 +1,44 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef _LINUX_TIMESTATS_H
+#define _LINUX_TIMESTATS_H
+
+#include <linux/spinlock_types.h>
+#include <linux/types.h>
+
+#define NR_QUANTILES	15
+
+struct quantiles {
+	struct quantile_entry {
+		u64	m;
+		u64	step;
+	}		entries[NR_QUANTILES];
+};
+
+struct time_stat_buffer {
+	unsigned int	nr;
+	struct time_stat_buffer_entry {
+		u64	start;
+		u64	end;
+	}		entries[32];
+};
+
+struct time_stats {
+	spinlock_t	lock;
+	u64		count;
+	/* all fields are in nanoseconds */
+	u64		average_duration;
+	u64		average_frequency;
+	u64		max_duration;
+	u64		last_event;
+	struct quantiles quantiles;
+
+	struct time_stat_buffer __percpu *buffer;
+};
+
+struct seq_buf;
+void time_stats_update(struct time_stats *stats, u64 start);
+void time_stats_to_text(struct seq_buf *out, struct time_stats *stats);
+void time_stats_exit(struct time_stats *stats);
+
+#endif /* _LINUX_TIMESTATS_H */
lib/Kconfig.debug +3 -0
index bbe3ef939c..bfb49505c9 100644
@@ -1728,6 +1728,9 @@ config LATENCYTOP
 	  Enable this option if you want to use the LatencyTOP tool
 	  to find out which userspace is blocking on what kernel operations.
 
+config TIME_STATS
+	bool
+
 source "kernel/trace/Kconfig"
 
 config PROVIDE_OHCI1394_DMA_INIT
lib/Makefile +1 -0
index c69430213e..e09255c881 100644
@@ -232,6 +232,7 @@ obj-$(CONFIG_ALLOC_TAGGING) += alloc_tag.o
 obj-$(CONFIG_PAGE_ALLOC_TAGGING) += pgalloc_tag.o
 
 obj-$(CONFIG_CODETAG_FAULT_INJECTION) += dynamic_fault.o
+obj-$(CONFIG_TIME_STATS) += time_stats.o
 
 lib-$(CONFIG_GENERIC_BUG) += bug.o
 
lib/time_stats.c new file +236 -0

236 lines not on this page · raw message

Signature · 1 line
2.36.1
#2 lore raw

Re: [PATCH 1/3] lib/time_stats: New library for statistics on events

lib/Kconfig.debug config LATENCYTOP · line 1733

Quoted 11 lines · Kent Overstreet
lib/Kconfig.debug +3 -0
index bbe3ef939c..bfb49505c9 100644
@@ -1728,6 +1728,9 @@ config LATENCYTOP
 	  Enable this option if you want to use the LatencyTOP tool
 	  to find out which userspace is blocking on what kernel operations.
 
+config TIME_STATS
+	bool
+

Hi Kent,

Why not just in lib/Kconfig?

thanks.

Signature · 1 line
~Randy
author maintainer
#3 lore raw

Re: [PATCH 1/3] lib/time_stats: New library for statistics on events

Quoted 11 lines · Randy Dunlap
Quoted 8 lines · Kent Overstreet
lib/Kconfig.debug +0 -0
index bbe3ef939c..bfb49505c9 100644
@@ -1728,6 +1728,9 @@ config LATENCYTOP
 	  Enable this option if you want to use the LatencyTOP tool
 	  to find out which userspace is blocking on what kernel operations.
 
Quoted 7 lines

lib/Kconfig.debug config LATENCYTOP · line 1733

Quoted 3 lines
lib/Kconfig.debug +3 -0
+config TIME_STATS
+	bool
+

Hi Kent,

Why not just in lib/Kconfig?

Probably just lazyness, I'll move it there :)

author maintainer
#4 lore raw

PATCH 2/3 bcache: Convert to lib/time_stats

In reply to Kent Overstreet

This patch converts bcache to the new generic time_stats code lib/time_stats.c. The new code is from bcachefs, and has some changes from the version in bcache:

- we now use ktime_get_ns(), not local_clock(). When the code was originally written multi processor systems that lacked synchronized TSCs were still common, and so local_clock() was much cheaper than sched_clock() (though not necessarily fully accurate, due to TSC drift). ktime_get_ns() should be cheap enough on all common hardware now, and more standard/correct.

- time_stats are now exported in a single file in sysfs, which means we can improve the statistics we keep track of without changing all users. This also means we don't have to manually specify which units (ms, us, ns) a given time_stats should be printed in; that's handled dynamically.

- There's a lazily-allocated percpu buffer, which now needs to be freed with time_stats_exit().

---
 drivers/md/bcache/Kconfig  |  1 +
 drivers/md/bcache/bcache.h |  1 +
 drivers/md/bcache/bset.c   |  8 +++---
 drivers/md/bcache/bset.h   |  1 +
 drivers/md/bcache/btree.c  | 12 ++++----
 drivers/md/bcache/super.c  |  3 ++
 drivers/md/bcache/sysfs.c  | 43 ++++++++++++++++++++--------
 drivers/md/bcache/util.c   | 30 --------------------
 drivers/md/bcache/util.h   | 57 --------------------------------------
 9 files changed, 47 insertions(+), 109 deletions(-)
drivers/md/bcache/Kconfig +1 -0
index 529c9d04e9..8d165052e5 100644
@@ -4,6 +4,7 @@ config BCACHE
 	tristate "Block device as cache"
 	select BLOCK_HOLDER_DEPRECATED if SYSFS
 	select CRC64
+	select TIME_STATS
 	help
 	Allows a block device to be used as cache for other devices; uses
 	a btree for indexing and the layout is optimized for SSDs.
drivers/md/bcache/bcache.h +1 -0
index 2acda9cea0..5100010a38 100644
@@ -185,6 +185,7 @@
 #include <linux/rbtree.h>
 #include <linux/rwsem.h>
 #include <linux/refcount.h>
+#include <linux/time_stats.h>
 #include <linux/types.h>
 #include <linux/workqueue.h>
 #include <linux/kthread.h>
drivers/md/bcache/bset.c +4 -4
index 94d38e8a59..727e9b7aea 100644
@@ -1251,7 +1251,7 @@ static void __btree_sort(struct btree_keys *b, struct btree_iter *iter,
 		order = state->page_order;
 	}
 
-	start_time = local_clock();
+	start_time = ktime_get_ns();
 
 	btree_mergesort(b, out, iter, fixup, false);
 	b->nsets = start;
@@ -1286,7 +1286,7 @@ static void __btree_sort(struct btree_keys *b, struct btree_iter *iter,
 	bch_bset_build_written_tree(b);
 
 	if (!start)
-		bch_time_stats_update(&state->time, start_time);
+		time_stats_update(&state->time, start_time);
 }
 
 void bch_btree_sort_partial(struct btree_keys *b, unsigned int start,
@@ -1322,14 +1322,14 @@ void bch_btree_sort_and_fix_extents(struct btree_keys *b,
 void bch_btree_sort_into(struct btree_keys *b, struct btree_keys *new,
 			 struct bset_sort_state *state)
 {
-	uint64_t start_time = local_clock();
+	uint64_t start_time = ktime_get_ns();
 	struct btree_iter iter;
 
 	bch_btree_iter_init(b, &iter, NULL);
 
 	btree_mergesort(b, new->set->data, &iter, false, true);
 
-	bch_time_stats_update(&state->time, start_time);
+	time_stats_update(&state->time, start_time);
 
 	new->set->size = 0; // XXX: why?
 }
drivers/md/bcache/bset.h +1 -0
index d795c84246..13e524ad77 100644
@@ -3,6 +3,7 @@
 #define _BCACHE_BSET_H
 
 #include <linux/kernel.h>
+#include <linux/time_stats.h>
 #include <linux/types.h>
 
 #include "bcache_ondisk.h"
drivers/md/bcache/btree.c +6 -6
index 147c493a98..abf543bc75 100644
@@ -242,7 +242,7 @@ static void btree_node_read_endio(struct bio *bio)
 
 static void bch_btree_node_read(struct btree *b)
 {
-	uint64_t start_time = local_clock();
+	uint64_t start_time = ktime_get_ns();
 	struct closure cl;
 	struct bio *bio;
 
@@ -270,7 +270,7 @@ static void bch_btree_node_read(struct btree *b)
 		goto err;
 
 	bch_btree_node_read_done(b);
-	bch_time_stats_update(&b->c->btree_read_time, start_time);
+	time_stats_update(&b->c->btree_read_time, start_time);
 
 	return;
 err:
@@ -1789,7 +1789,7 @@ static void bch_btree_gc(struct cache_set *c)
 	struct gc_stat stats;
 	struct closure writes;
 	struct btree_op op;
-	uint64_t start_time = local_clock();
+	uint64_t start_time = ktime_get_ns();
 
 	trace_bcache_gc_start(c);
 
@@ -1815,7 +1815,7 @@ static void bch_btree_gc(struct cache_set *c)
 	bch_btree_gc_finish(c);
 	wake_up_allocators(c);
 
-	bch_time_stats_update(&c->btree_gc_time, start_time);
+	time_stats_update(&c->btree_gc_time, start_time);
 
 	stats.key_bytes *= sizeof(uint64_t);
 	stats.data	<<= 9;
@@ -2191,7 +2191,7 @@ static int btree_split(struct btree *b, struct btree_op *op,
 {
 	bool split;
 	struct btree *n1, *n2 = NULL, *n3 = NULL;
-	uint64_t start_time = local_clock();
+	uint64_t start_time = ktime_get_ns();
 	struct closure cl;
 	struct keylist parent_keys;
 
@@ -2297,7 +2297,7 @@ static int btree_split(struct btree *b, struct btree_op *op,
 	btree_node_free(b);
 	rw_unlock(true, n1);
 
-	bch_time_stats_update(&b->c->btree_split_time, start_time);
+	time_stats_update(&b->c->btree_split_time, start_time);
 
 	return 0;
 err_free2:
drivers/md/bcache/super.c +3 -0
index ba3909bb6b..26c8fa93b5 100644
@@ -1691,6 +1691,9 @@ static void cache_set_free(struct closure *cl)
 		kobject_put(&ca->kobj);
 	}
 
+	time_stats_exit(&c->btree_gc_time);
+	time_stats_exit(&c->btree_split_time);
+	time_stats_exit(&c->sort.time);
 
 	if (c->moving_gc_wq)
 		destroy_workqueue(c->moving_gc_wq);
drivers/md/bcache/sysfs.c +31 -12

73 lines not on this page · raw message

drivers/md/bcache/util.c +0 -30
index ae380bc399..95282bf0f9 100644
@@ -160,36 +160,6 @@ int bch_parse_uuid(const char *s, char *uuid)
 	return i;
 }
 
-void bch_time_stats_update(struct time_stats *stats, uint64_t start_time)
-{
-	uint64_t now, duration, last;
-
-	spin_lock(&stats->lock);
-
-	now		= local_clock();
-	duration	= time_after64(now, start_time)
-		? now - start_time : 0;
-	last		= time_after64(now, stats->last)
-		? now - stats->last : 0;
-
-	stats->max_duration = max(stats->max_duration, duration);
-
-	if (stats->last) {
-		ewma_add(stats->average_duration, duration, 8, 8);
-
-		if (stats->average_frequency)
-			ewma_add(stats->average_frequency, last, 8, 8);
-		else
-			stats->average_frequency  = last << 8;
-	} else {
-		stats->average_duration  = duration << 8;
-	}
-
-	stats->last = now ?: 1;
-
-	spin_unlock(&stats->lock);
-}
-
 /**
  * bch_next_delay() - update ratelimiting statistics and calculate next delay
  * @d: the struct bch_ratelimit to update
drivers/md/bcache/util.h +0 -57

68 lines not on this page · raw message

Signature · 1 line
2.36.1
Acked-by on v1 2/3
#5 lore raw

Re: [PATCH 2/3] bcache: Convert to lib/time_stats

Quoted 21 lines

2022年8月30日 00:53,Kent Overstreet <[email protected]> 写道:

This patch converts bcache to the new generic time_stats code lib/time_stats.c. The new code is from bcachefs, and has some changes from the version in bcache:

- we now use ktime_get_ns(), not local_clock(). When the code was originally written multi processor systems that lacked synchronized TSCs were still common, and so local_clock() was much cheaper than sched_clock() (though not necessarily fully accurate, due to TSC drift). ktime_get_ns() should be cheap enough on all common hardware now, and more standard/correct.

- time_stats are now exported in a single file in sysfs, which means we can improve the statistics we keep track of without changing all users. This also means we don't have to manually specify which units (ms, us, ns) a given time_stats should be printed in; that's handled dynamically.

Quoted 5 lines

- There's a lazily-allocated percpu buffer, which now needs to be freed with time_stats_exit().

Hi Kent,

Overall I am OK with the change to bcache code. You may add

in future version of this patch.

In bcache-tools, they don’t read the changed sysfs files (including bcache-status), IMHO changing the output format won’t be problem for upstream.

My only question is, how to understand the time_stats_to_text() output format,

count:          3
rate:           0/sec
frequency:      4 sec
avg duration:   4 sec
max duration:   4 sec
quantiles (ns): 0 4288669120 4288669120 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048

Fro the above output, what are “rate”, “frequence” and “quantiles” for?

Thanks.

Coly Li

Quoted 358 lines

---

drivers/md/bcache/Kconfig  |  1 +
drivers/md/bcache/bcache.h |  1 +
drivers/md/bcache/bset.c   |  8 +++---
drivers/md/bcache/bset.h   |  1 +
drivers/md/bcache/btree.c  | 12 ++++----
drivers/md/bcache/super.c  |  3 ++
drivers/md/bcache/sysfs.c  | 43 ++++++++++++++++++++--------
drivers/md/bcache/util.c   | 30 --------------------
drivers/md/bcache/util.h   | 57 --------------------------------------
9 files changed, 47 insertions(+), 109 deletions(-)
drivers/md/bcache/Kconfig +1 -0
index 529c9d04e9..8d165052e5 100644
@@ -4,6 +4,7 @@ config BCACHE
 	tristate "Block device as cache"
 	select BLOCK_HOLDER_DEPRECATED if SYSFS
 	select CRC64
+	select TIME_STATS
 	help
 	Allows a block device to be used as cache for other devices; uses
 	a btree for indexing and the layout is optimized for SSDs.
drivers/md/bcache/bcache.h +1 -0
index 2acda9cea0..5100010a38 100644
@@ -185,6 +185,7 @@
 #include <linux/rbtree.h>
 #include <linux/rwsem.h>
 #include <linux/refcount.h>
+#include <linux/time_stats.h>
 #include <linux/types.h>
 #include <linux/workqueue.h>
 #include <linux/kthread.h>
drivers/md/bcache/bset.c +4 -4
index 94d38e8a59..727e9b7aea 100644
@@ -1251,7 +1251,7 @@ static void __btree_sort(struct btree_keys *b, struct btree_iter *iter,
 		order = state->page_order;
 	}
 
-	start_time = local_clock();
+	start_time = ktime_get_ns();
 
 	btree_mergesort(b, out, iter, fixup, false);
 	b->nsets = start;
@@ -1286,7 +1286,7 @@ static void __btree_sort(struct btree_keys *b, struct btree_iter *iter,
 	bch_bset_build_written_tree(b);
 
 	if (!start)
-		bch_time_stats_update(&state->time, start_time);
+		time_stats_update(&state->time, start_time);
 }
 
 void bch_btree_sort_partial(struct btree_keys *b, unsigned int start,
@@ -1322,14 +1322,14 @@ void bch_btree_sort_and_fix_extents(struct btree_keys *b,
 void bch_btree_sort_into(struct btree_keys *b, struct btree_keys *new,
 			 struct bset_sort_state *state)
 {
-	uint64_t start_time = local_clock();
+	uint64_t start_time = ktime_get_ns();
 	struct btree_iter iter;
 
 	bch_btree_iter_init(b, &iter, NULL);
 
 	btree_mergesort(b, new->set->data, &iter, false, true);
 
-	bch_time_stats_update(&state->time, start_time);
+	time_stats_update(&state->time, start_time);
 
 	new->set->size = 0; // XXX: why?
 }
drivers/md/bcache/bset.h +1 -0
index d795c84246..13e524ad77 100644
@@ -3,6 +3,7 @@
 #define _BCACHE_BSET_H
 
 #include <linux/kernel.h>
+#include <linux/time_stats.h>
 #include <linux/types.h>
 
 #include "bcache_ondisk.h"
drivers/md/bcache/btree.c +2 -2
index 147c493a98..abf543bc75 100644
@@ -242,7 +242,7 @@ static void btree_node_read_endio(struct bio *bio)
 
 static void bch_btree_node_read(struct btree *b)
 {
-	uint64_t start_time = local_clock();
+	uint64_t start_time = ktime_get_ns();
 	struct closure cl;
 	struct bio *bio;
 
@@ -270,7 +270,7 @@ static void bch_btree_node_read(struct btree *b)
 		goto err;
 
 	bch_btree_node_read_done(b);
-	bch_time_stats_update(&b->c->btree_read_time, start_time);
+	time_stats_update(&b->c->btree_read_time, start_time);
 
 	return;

err:

drivers/md/bcache/btree.c +4 -4
@@ -1789,7 +1789,7 @@ static void bch_btree_gc(struct cache_set *c)
 	struct gc_stat stats;
 	struct closure writes;
 	struct btree_op op;
-	uint64_t start_time = local_clock();
+	uint64_t start_time = ktime_get_ns();
 
 	trace_bcache_gc_start(c);
 
@@ -1815,7 +1815,7 @@ static void bch_btree_gc(struct cache_set *c)
 	bch_btree_gc_finish(c);
 	wake_up_allocators(c);
 
-	bch_time_stats_update(&c->btree_gc_time, start_time);
+	time_stats_update(&c->btree_gc_time, start_time);
 
 	stats.key_bytes *= sizeof(uint64_t);
 	stats.data	<<= 9;
@@ -2191,7 +2191,7 @@ static int btree_split(struct btree *b, struct btree_op *op,
 {
 	bool split;
 	struct btree *n1, *n2 = NULL, *n3 = NULL;
-	uint64_t start_time = local_clock();
+	uint64_t start_time = ktime_get_ns();
 	struct closure cl;
 	struct keylist parent_keys;
 
@@ -2297,7 +2297,7 @@ static int btree_split(struct btree *b, struct btree_op *op,
 	btree_node_free(b);
 	rw_unlock(true, n1);
 
-	bch_time_stats_update(&b->c->btree_split_time, start_time);
+	time_stats_update(&b->c->btree_split_time, start_time);
 
 	return 0;

err_free2:

drivers/md/bcache/super.c +3 -0
index ba3909bb6b..26c8fa93b5 100644
@@ -1691,6 +1691,9 @@ static void cache_set_free(struct closure *cl)
 		kobject_put(&ca->kobj);
 	}
 
+	time_stats_exit(&c->btree_gc_time);
+	time_stats_exit(&c->btree_split_time);
+	time_stats_exit(&c->sort.time);
 
 	if (c->moving_gc_wq)
 		destroy_workqueue(c->moving_gc_wq);
drivers/md/bcache/sysfs.c +31 -12
index c6f6770592..01eec5877c 100644
@@ -16,6 +16,7 @@
 #include <linux/blkdev.h>
 #include <linux/sort.h>
 #include <linux/sched/clock.h>
+#include <linux/seq_buf.h>
 
 extern bool bcache_is_reboot;
 
@@ -79,10 +80,10 @@ read_attribute(active_journal_entries);
 read_attribute(backing_dev_name);
 read_attribute(backing_dev_uuid);
 
-sysfs_time_stats_attribute(btree_gc,	sec, ms);
-sysfs_time_stats_attribute(btree_split, sec, us);
-sysfs_time_stats_attribute(btree_sort,	ms,  us);
-sysfs_time_stats_attribute(btree_read,	ms,  us);
+read_attribute(btree_gc_time);
+read_attribute(btree_split_time);
+read_attribute(btree_sort_time);
+read_attribute(btree_read_time);
 
 read_attribute(btree_nodes);
 read_attribute(btree_used_percent);
@@ -731,6 +732,9 @@ static unsigned int bch_average_key_size(struct cache_set *c)
 SHOW(__bch_cache_set)
 {
 	struct cache_set *c = container_of(kobj, struct cache_set, kobj);
+	struct seq_buf s;
+
+	seq_buf_init(&s, buf, PAGE_SIZE);
 
 	sysfs_print(synchronous,		CACHE_SYNC(&c->cache->sb));
 	sysfs_print(journal_delay_ms,		c->journal_delay_ms);
@@ -743,10 +747,25 @@ SHOW(__bch_cache_set)
 	sysfs_print(btree_cache_max_chain,	bch_cache_max_chain(c));
 	sysfs_print(cache_available_percent,	100 - c->gc_stats.in_use);
 
-	sysfs_print_time_stats(&c->btree_gc_time,	btree_gc, sec, ms);
-	sysfs_print_time_stats(&c->btree_split_time,	btree_split, sec, us);
-	sysfs_print_time_stats(&c->sort.time,		btree_sort, ms, us);
-	sysfs_print_time_stats(&c->btree_read_time,	btree_read, ms, us);
+	if (attr == &sysfs_btree_gc_time) {
+		time_stats_to_text(&s, &c->btree_gc_time);
+		return s.len;
+	}
+
+	if (attr == &sysfs_btree_split_time) {
+		time_stats_to_text(&s, &c->btree_split_time);
+		return s.len;
+	}
+
+	if (attr == &sysfs_btree_sort_time) {
+		time_stats_to_text(&s, &c->sort.time);
+		return s.len;
+	}
+
+	if (attr == &sysfs_btree_read_time) {
+		time_stats_to_text(&s, &c->btree_read_time);
+		return s.len;
+	}
 
 	sysfs_print(btree_used_percent,	bch_btree_used(c));
 	sysfs_print(btree_nodes,	c->gc_stats.nodes);
@@ -988,10 +1007,10 @@ KTYPE(bch_cache_set);
 static struct attribute *bch_cache_set_internal_attrs[] = {
 	&sysfs_active_journal_entries,
 
-	sysfs_time_stats_attribute_list(btree_gc, sec, ms)
-	sysfs_time_stats_attribute_list(btree_split, sec, us)
-	sysfs_time_stats_attribute_list(btree_sort, ms, us)
-	sysfs_time_stats_attribute_list(btree_read, ms, us)
+	&sysfs_btree_gc_time,
+	&sysfs_btree_split_time,
+	&sysfs_btree_sort_time,
+	&sysfs_btree_read_time,
 
 	&sysfs_btree_nodes,
 	&sysfs_btree_used_percent,
drivers/md/bcache/util.c +0 -30
index ae380bc399..95282bf0f9 100644
@@ -160,36 +160,6 @@ int bch_parse_uuid(const char *s, char *uuid)
 	return i;
 }
 
-void bch_time_stats_update(struct time_stats *stats, uint64_t start_time)
-{
-	uint64_t now, duration, last;
-
-	spin_lock(&stats->lock);
-
-	now		= local_clock();
-	duration	= time_after64(now, start_time)
-		? now - start_time : 0;
-	last		= time_after64(now, stats->last)
-		? now - stats->last : 0;
-
-	stats->max_duration = max(stats->max_duration, duration);
-
-	if (stats->last) {
-		ewma_add(stats->average_duration, duration, 8, 8);
-
-		if (stats->average_frequency)
-			ewma_add(stats->average_frequency, last, 8, 8);
-		else
-			stats->average_frequency  = last << 8;
-	} else {
-		stats->average_duration  = duration << 8;
-	}
-
-	stats->last = now ?: 1;
-
-	spin_unlock(&stats->lock);
-}
-
 /**
 * bch_next_delay() - update ratelimiting statistics and calculate next delay
 * @d: the struct bch_ratelimit to update
drivers/md/bcache/util.h +0 -57
index 6f3cb7c921..1e1bdbae95 100644
@@ -345,68 +345,11 @@ ssize_t bch_hprint(char *buf, int64_t v);
 bool bch_is_zero(const char *p, size_t n);
 int bch_parse_uuid(const char *s, char *uuid);
 
-struct time_stats {
-	spinlock_t	lock;
-	/*
-	 * all fields are in nanoseconds, averages are ewmas stored left shifted
-	 * by 8
-	 */
-	uint64_t	max_duration;
-	uint64_t	average_duration;
-	uint64_t	average_frequency;
-	uint64_t	last;
-};
-
-void bch_time_stats_update(struct time_stats *stats, uint64_t time);
-
 static inline unsigned int local_clock_us(void)
 {
 	return local_clock() >> 10;
 }
 
-#define NSEC_PER_ns			1L
-#define NSEC_PER_us			NSEC_PER_USEC
-#define NSEC_PER_ms			NSEC_PER_MSEC
-#define NSEC_PER_sec			NSEC_PER_SEC
-
-#define __print_time_stat(stats, name, stat, units)			\
-	sysfs_print(name ## _ ## stat ## _ ## units,			\
-		    div_u64((stats)->stat >> 8, NSEC_PER_ ## units))
-
-#define sysfs_print_time_stats(stats, name,				\
-			       frequency_units,				\
-			       duration_units)				\
-do {									\
-	__print_time_stat(stats, name,					\
-			  average_frequency,	frequency_units);	\
-	__print_time_stat(stats, name,					\
-			  average_duration,	duration_units);	\
-	sysfs_print(name ## _ ##max_duration ## _ ## duration_units,	\
-			div_u64((stats)->max_duration,			\
-				NSEC_PER_ ## duration_units));		\
-									\
-	sysfs_print(name ## _last_ ## frequency_units, (stats)->last	\
-		    ? div_s64(local_clock() - (stats)->last,		\
-			      NSEC_PER_ ## frequency_units)		\
-		    : -1LL);						\
-} while (0)
-
-#define sysfs_time_stats_attribute(name,				\
-				   frequency_units,			\
-				   duration_units)			\
-read_attribute(name ## _average_frequency_ ## frequency_units);		\
-read_attribute(name ## _average_duration_ ## duration_units);		\
-read_attribute(name ## _max_duration_ ## duration_units);		\
-read_attribute(name ## _last_ ## frequency_units)
-
-#define sysfs_time_stats_attribute_list(name,				\
-					frequency_units,		\
-					duration_units)			\
-&sysfs_ ## name ## _average_frequency_ ## frequency_units,		\
-&sysfs_ ## name ## _average_duration_ ## duration_units,		\
-&sysfs_ ## name ## _max_duration_ ## duration_units,			\
-&sysfs_ ## name ## _last_ ## frequency_units,
-
 #define ewma_add(ewma, val, weight, factor)				\
 ({									\
 	(ewma) *= (weight) - 1;						\
Signature · 1 line
2.36.1
author maintainer
#6 lore raw

Re: [PATCH 2/3] bcache: Convert to lib/time_stats

Quoted 36 lines · Coly Li
Quoted 21 lines

2022年8月30日 00:53,Kent Overstreet <[email protected]> 写道:

This patch converts bcache to the new generic time_stats code lib/time_stats.c. The new code is from bcachefs, and has some changes from the version in bcache:

- we now use ktime_get_ns(), not local_clock(). When the code was originally written multi processor systems that lacked synchronized TSCs were still common, and so local_clock() was much cheaper than sched_clock() (though not necessarily fully accurate, due to TSC drift). ktime_get_ns() should be cheap enough on all common hardware now, and more standard/correct.

- time_stats are now exported in a single file in sysfs, which means we can improve the statistics we keep track of without changing all users. This also means we don't have to manually specify which units (ms, us, ns) a given time_stats should be printed in; that's handled dynamically.

Quoted 5 lines

- There's a lazily-allocated percpu buffer, which now needs to be freed with time_stats_exit().

Hi Kent,

Overall I am OK with the change to bcache code. You may add

in future version of this patch.

In bcache-tools, they don’t read the changed sysfs files (including bcache-status), IMHO changing the output format won’t be problem for upstream.

My only question is, how to understand the time_stats_to_text() output format,

Quoted 8 lines
count:          3
rate:           0/sec
frequency:      4 sec
avg duration:   4 sec
max duration:   4 sec
quantiles (ns): 0 4288669120 4288669120 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048

Fro the above output, what are “rate”, “frequence” and “quantiles” for?

Rate and frequency are inverses - in this example, we're seeing 4 events per second.

The quantiles are for the duration, they give you an idea of the statistical distribution, see https://en.wikipedia.org/wiki/Quantile

In the near future, the quantiles will be removed and replaced with standard deviation, true and weighted - the quantiles algorithm isn't super accurate, we can give more accurate numbers with standard deviation.

#7 lore raw

Re: [PATCH 2/3] bcache: Convert to lib/time_stats

Quoted 48 lines

2022年9月1日 00:54,Kent Overstreet <[email protected]> 写道:

Quoted 43 lines · Coly Li
Quoted 21 lines

2022年8月30日 00:53,Kent Overstreet <[email protected]> 写道:

This patch converts bcache to the new generic time_stats code lib/time_stats.c. The new code is from bcachefs, and has some changes from the version in bcache:

- we now use ktime_get_ns(), not local_clock(). When the code was originally written multi processor systems that lacked synchronized TSCs were still common, and so local_clock() was much cheaper than sched_clock() (though not necessarily fully accurate, due to TSC drift). ktime_get_ns() should be cheap enough on all common hardware now, and more standard/correct.

- time_stats are now exported in a single file in sysfs, which means we can improve the statistics we keep track of without changing all users. This also means we don't have to manually specify which units (ms, us, ns) a given time_stats should be printed in; that's handled dynamically.

Quoted 5 lines

- There's a lazily-allocated percpu buffer, which now needs to be freed with time_stats_exit().

Hi Kent,

Overall I am OK with the change to bcache code. You may add

in future version of this patch.

In bcache-tools, they don’t read the changed sysfs files (including bcache-status), IMHO changing the output format won’t be problem for upstream.

My only question is, how to understand the time_stats_to_text() output format,

count:          3
rate:           0/sec
frequency:      4 sec
avg duration:   4 sec
max duration:   4 sec
quantiles (ns): 0 4288669120 4288669120 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048
Quoted 5 lines
Quoted 1 line

Fro the above output, what are “rate”, “frequence” and “quantiles” for?

Rate and frequency are inverses - in this example, we're seeing 4 events per second.

Can I understand that rate = 1/frequency ? Then frequency 4 is around to rate 0.

Quoted 2 lines

The quantiles are for the duration, they give you an idea of the statistical distribution, see https://en.wikipedia.org/wiki/Quantile

I wanted to ask how to read the quantiles line. Does it mean that 1 ns is equally divided by 15 segments, and the counter values are for the divided 1/15 ns segments?

Quoted 4 lines

In the near future, the quantiles will be removed and replaced with standard deviation, true and weighted - the quantiles algorithm isn't super accurate, we can give more accurate numbers with standard deviation.

Quantiles are much humane, standard deviation is scared… Just FYI LOL.

Thanks.

Coly Li

author maintainer
#8 lore raw

Re: [PATCH 2/3] bcache: Convert to lib/time_stats

Quoted 1 line · Coly Li

Can I understand that rate = 1/frequency ? Then frequency 4 is around to rate 0.

Correct, rate = 1/frequency.

Quoted 6 lines
Quoted 2 lines

The quantiles are for the duration, they give you an idea of the statistical distribution, see https://en.wikipedia.org/wiki/Quantile

I wanted to ask how to read the quantiles line. Does it mean that 1 ns is equally divided by 15 segments, and the counter values are for the divided 1/15 ns segments?

It is the 15 boundaries between 16 equal segments, yes.

Quoted 1 line

Quantiles are much humane, standard deviation is scared… Just FYI LOL.

That's valuable feedback then, maybe we'll keep both. The more expensive part of time stats is just ingesting data points, but we've got a percpu buffer frontend for that - just doing a bit more moth doesn't cost much.

#9 lore raw

Re: [PATCH 2/3] bcache: Convert to lib/time_stats

Quoted 16 lines

2022年9月1日 01:36,Kent Overstreet <[email protected]> 写道:

Quoted 1 line · Coly Li

Can I understand that rate = 1/frequency ? Then frequency 4 is around to rate 0.

Correct, rate = 1/frequency.

Quoted 6 lines
Quoted 2 lines

The quantiles are for the duration, they give you an idea of the statistical distribution, see https://en.wikipedia.org/wiki/Quantile

I wanted to ask how to read the quantiles line. Does it mean that 1 ns is equally divided by 15 segments, and the counter values are for the divided 1/15 ns segments?

Quoted 7 lines

It is the 15 boundaries between 16 equal segments, yes.

Quoted 1 line

Quantiles are much humane, standard deviation is scared… Just FYI LOL.

That's valuable feedback then, maybe we'll keep both. The more expensive part of time stats is just ingesting data points, but we've got a percpu buffer frontend for that - just doing a bit more moth doesn't cost much.

I don’t have more question so far. Thanks for the detailed replay.

Coly Li

author maintainer
#10 lore raw

PATCH 3/3 Code tagging based latency tracking

In reply to Kent Overstreet

This adds the ability to easily instrument code for measuring latency. To use, add the following to calls to your code, at the start and end of the event you wish to measure:

  code_tag_time_stats_start(start_time);
  code_tag_time_stats_finish(start_time);

Stastistics will then show up in debugfs under /sys/kernel/debug/time_stats, listed by file and line number.

Stastics measured include weighted averages of frequency, duration, max duration, as well as quantiles.

This patch also instruments all calls to init_wait and finish_wait, which includes all calls to wait_event. Example debugfs output:

fs/xfs/xfs_trans_ail.c:746 module:xfs func:xfs_ail_push_all_sync

count:          17
rate:           0/sec
frequency:      2 sec
avg duration:   10 us
max duration:   232 us
quantiles (ns): 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128

lib/sbitmap.c:813 module:sbitmap func:sbitmap_finish_wait

count:          3
rate:           0/sec
frequency:      4 sec
avg duration:   4 sec
max duration:   4 sec
quantiles (ns): 0 4288669120 4288669120 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048 5360836048

net/core/datagram.c:122 module:datagram func:__skb_wait_for_more_packets

count:          10
rate:           1/sec
frequency:      859 ms
avg duration:   472 ms
max duration:   30 sec
quantiles (ns): 0 12279 12279 15669 15669 15669 15669 17217 17217 17217 17217 17217 17217 17217 17217
---
 include/asm-generic/codetag.lds.h  |   3 +-
 include/linux/codetag_time_stats.h |  54 +++++++++++
 include/linux/wait.h               |  22 ++++-
 kernel/sched/wait.c                |   6 +-
 lib/Kconfig.debug                  |   8 ++
 lib/Makefile                       |   1 +
 lib/codetag_time_stats.c           | 143 +++++++++++++++++++++++++++++
 7 files changed, 232 insertions(+), 5 deletions(-)
 create mode 100644 include/linux/codetag_time_stats.h
 create mode 100644 lib/codetag_time_stats.c
include/asm-generic/codetag.lds.h +2 -1
index 16fbf74edc..d799f4aced 100644
@@ -10,6 +10,7 @@
 
 #define CODETAG_SECTIONS()		\
 	SECTION_WITH_BOUNDARIES(alloc_tags)		\
-	SECTION_WITH_BOUNDARIES(dynamic_fault_tags)
+	SECTION_WITH_BOUNDARIES(dynamic_fault_tags)	\
+	SECTION_WITH_BOUNDARIES(time_stats_tags)
 
 #endif /* __ASM_GENERIC_CODETAG_LDS_H */
include/linux/codetag_time_stats.h new file +54 -0

54 lines not on this page · raw message

include/linux/wait.h +21 -1
index 91ced6a118..bab11b7ef1 100644
@@ -4,6 +4,7 @@
 /*
  * Linux wait queue related types and methods
  */
+#include <linux/codetag_time_stats.h>
 #include <linux/list.h>
 #include <linux/stddef.h>
 #include <linux/spinlock.h>
@@ -32,6 +33,9 @@ struct wait_queue_entry {
 	void			*private;
 	wait_queue_func_t	func;
 	struct list_head	entry;
+#ifdef CONFIG_CODETAG_TIME_STATS
+	u64			start_time;
+#endif
 };
 
 struct wait_queue_head {
@@ -79,10 +83,17 @@ extern void __init_waitqueue_head(struct wait_queue_head *wq_head, const char *n
 # define DECLARE_WAIT_QUEUE_HEAD_ONSTACK(name) DECLARE_WAIT_QUEUE_HEAD(name)
 #endif
 
+#ifdef CONFIG_CODETAG_TIME_STATS
+#define WAIT_QUEUE_ENTRY_START_TIME_INITIALIZER	.start_time = ktime_get_ns(),
+#else
+#define WAIT_QUEUE_ENTRY_START_TIME_INITIALIZER
+#endif
+
 #define WAIT_FUNC_INITIALIZER(name, function) {					\
 	.private	= current,						\
 	.func		= function,						\
 	.entry		= LIST_HEAD_INIT((name).entry),				\
+	WAIT_QUEUE_ENTRY_START_TIME_INITIALIZER					\
 }
 
 #define DEFINE_WAIT_FUNC(name, function)					\
@@ -98,6 +109,9 @@ __init_waitqueue_entry(struct wait_queue_entry *wq_entry, unsigned int flags,
 	wq_entry->private	= private;
 	wq_entry->func		= func;
 	INIT_LIST_HEAD(&wq_entry->entry);
+#ifdef CONFIG_CODETAG_TIME_STATS
+	wq_entry->start_time	= ktime_get_ns();
+#endif
 }
 
 #define init_waitqueue_func_entry(_wq_entry, _func)			\
@@ -1180,11 +1194,17 @@ do {										\
 void prepare_to_wait(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry, int state);
 bool prepare_to_wait_exclusive(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry, int state);
 long prepare_to_wait_event(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry, int state);
-void finish_wait(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry);
+void __finish_wait(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry);
 long wait_woken(struct wait_queue_entry *wq_entry, unsigned mode, long timeout);
 int woken_wake_function(struct wait_queue_entry *wq_entry, unsigned mode, int sync, void *key);
 int autoremove_wake_function(struct wait_queue_entry *wq_entry, unsigned mode, int sync, void *key);
 
+#define finish_wait(_wq_head, _wq_entry)					\
+do {										\
+	codetag_time_stats_finish((_wq_entry)->start_time);			\
+	__finish_wait(_wq_head, _wq_entry);					\
+} while (0)
+
 typedef int (*task_call_f)(struct task_struct *p, void *arg);
 extern int task_call_func(struct task_struct *p, task_call_f func, void *arg);
 
kernel/sched/wait.c +3 -3
index b992234607..e88de3f0c3 100644
@@ -367,7 +367,7 @@ int do_wait_intr_irq(wait_queue_head_t *wq, wait_queue_entry_t *wait)
 EXPORT_SYMBOL(do_wait_intr_irq);
 
 /**
- * finish_wait - clean up after waiting in a queue
+ * __finish_wait - clean up after waiting in a queue
  * @wq_head: waitqueue waited on
  * @wq_entry: wait descriptor
  *
@@ -375,7 +375,7 @@ EXPORT_SYMBOL(do_wait_intr_irq);
  * the wait descriptor from the given waitqueue if still
  * queued.
  */
-void finish_wait(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry)
+void __finish_wait(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_entry)
 {
 	unsigned long flags;
 
@@ -399,7 +399,7 @@ void finish_wait(struct wait_queue_head *wq_head, struct wait_queue_entry *wq_en
 		spin_unlock_irqrestore(&wq_head->lock, flags);
 	}
 }
-EXPORT_SYMBOL(finish_wait);
+EXPORT_SYMBOL(__finish_wait);
 
 int autoremove_wake_function(struct wait_queue_entry *wq_entry, unsigned mode, int sync, void *key)
 {
lib/Kconfig.debug +8 -0
index bfb49505c9..40584fe54a 100644
@@ -1731,6 +1731,14 @@ config LATENCYTOP
 config TIME_STATS
 	bool
 
+config CODETAG_TIME_STATS
+	bool "Code tagging based latency measuring"
+	depends on DEBUG_FS
+	select TIME_STATS
+	select CODE_TAGGING
+	help
+	  Enabling this option makes latency statistics available in debugfs
+
 source "kernel/trace/Kconfig"
 
 config PROVIDE_OHCI1394_DMA_INIT
lib/Makefile +1 -0
index e09255c881..7888bc8baa 100644
@@ -233,6 +233,7 @@ obj-$(CONFIG_PAGE_ALLOC_TAGGING) += pgalloc_tag.o
 
 obj-$(CONFIG_CODETAG_FAULT_INJECTION) += dynamic_fault.o
 obj-$(CONFIG_TIME_STATS) += time_stats.o
+obj-$(CONFIG_CODETAG_TIME_STATS) += codetag_time_stats.o
 
 lib-$(CONFIG_GENERIC_BUG) += bug.o
 
lib/codetag_time_stats.c new file +143 -0

143 lines not on this page · raw message

Signature · 1 line
2.36.1