Skip to contentSkip to conversation

bcache: dynamic incremental gc

PATCH New [email protected] 1 participant 1 message May 23, 2022 bcache (block layer cache)

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 bcache: dynamic incremental gc

From: ZouMingzhe <[email protected]>

Currently, GC wants no more than 100 times, with at least 100 nodes each time, the maximum number of nodes each time is not limited.

```
static size_t btree_gc_min_nodes(struct cache_set *c)
{
	......
	min_nodes = c->gc_stats.nodes / MAX_GC_TIMES;
	if (min_nodes < MIN_GC_NODES)
		min_nodes = MIN_GC_NODES;

	return min_nodes;
}
```

According to our test data, when nvme is used as the cache, it takes about 1ms for GC to handle each node (block 4k and bucket 512k). This means that the latency during GC is at least 100ms. During GC, IO performance would be reduced by half or more.

I want to optimize the IOPS and latency under high pressure. This patch hold the inflight peak. When IO depth up to maximum, GC only process very few(10) nodes, then sleep immediately and handle these requests.

bch_bucket_alloc() maybe wait for bch_allocator_thread() to wake up, and and bch_allocator_thread() needs to wait for gc to complete, in which case gc needs to end quickly. So, add bucket_alloc_inflight to cache_set in v3.

```
long bch_bucket_alloc(struct cache *ca, unsigned int reserve, bool wait)
{
	......
	do {
		prepare_to_wait(&ca->set->bucket_wait, &w,
				TASK_UNINTERRUPTIBLE);

		mutex_unlock(&ca->set->bucket_lock);
		schedule();
		mutex_lock(&ca->set->bucket_lock);
	} while (!fifo_pop(&ca->free[RESERVE_NONE], r) &&
		 !fifo_pop(&ca->free[reserve], r));
	......
}

static int bch_allocator_thread(void *arg)
{
	......
	allocator_wait(ca, bch_allocator_push(ca, bucket));
	wake_up(&ca->set->btree_cache_wait);
	wake_up(&ca->set->bucket_wait);
	......
}

static void bch_btree_gc(struct cache_set *c)
{
	......
	bch_btree_gc_finish(c);
	wake_up_allocators(c);
	......
}
```

Apply this patch, each GC maybe only process very few nodes, GC would last a long time if sleep 100ms each time. So, the sleep time should be calculated dynamically based on gc_cost.

At the same time, I added some cost statistics in gc_stat, hoping to provide useful information for future work.

changelog:
v2: fix auto build test ERROR
v3: add bucket_alloc_inflight to cache_set
improve commit comment
v4: improve dmesg print

---
 drivers/md/bcache/alloc.c  |   2 +
 drivers/md/bcache/bcache.h |   9 ++++
 drivers/md/bcache/btree.c  | 100 ++++++++++++++++++++++++++++++++-----
 3 files changed, 98 insertions(+), 13 deletions(-)
drivers/md/bcache/alloc.c +2 -0
index 097577ae3c47..bb8b16cc2189 100644
@@ -415,7 +415,9 @@ long bch_bucket_alloc(struct cache *ca, unsigned int reserve, bool wait)
 				TASK_UNINTERRUPTIBLE);
 
 		mutex_unlock(&ca->set->bucket_lock);
+		atomic_inc(&ca->set->bucket_alloc_inflight);
 		schedule();
+		atomic_dec(&ca->set->bucket_alloc_inflight);
 		mutex_lock(&ca->set->bucket_lock);
 	} while (!fifo_pop(&ca->free[RESERVE_NONE], r) &&
 		 !fifo_pop(&ca->free[reserve], r));
drivers/md/bcache/bcache.h +9 -0
index 9ed9c955add7..a113a3bc7356 100644
@@ -471,6 +471,14 @@ struct cache {
 };
 
 struct gc_stat {
+	uint64_t		gc_cost;
+	uint64_t		sleep_cost;
+	uint64_t		average_cost;
+	uint64_t		start_time;
+	uint64_t		finish_time;
+	size_t			max_inflight;
+
+	size_t			times;
 	size_t			nodes;
 	size_t			nodes_pre;
 	size_t			key_bytes;
@@ -595,6 +603,7 @@ struct cache_set {
 	 * written.
 	 */
 	atomic_t		prio_blocked;
+	atomic_t		bucket_alloc_inflight;
 	wait_queue_head_t	bucket_wait;
 
 	/*
drivers/md/bcache/btree.c +87 -13

160 lines not on this page · raw message

Signature · 1 line
2.17.1