bcache_gc: BUG: soft lockup - CPU#2 stuck for 23s!

DISCUSSION Joshua Schmid 1 participant 2 messages Feb 3, 2015 bcache (block layer cache)

lore

Share & tools

lore

Joshua Schmid

author

2015-02-03 10:37 UTC #0 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

I tested this patch for some time and it really helps to fix the gc issue we are running in. Since it might got lost, i will resend it.

Best Regards,
Joshua

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBAgAGBQJU0KTPAAoJEPUnwXO0u5uWMHYP/j18RLchiWUv2qD1RwNbRYdY 5nood3FekVhZTsTZTqw+0Js89XHLtCHgT9qKSsYKKng512jb8o0VHPWXPwviRiOg /wo3Pc3f7rq4AKd6Ek1hNeEoBovdkWDRpLlQeJWd4D73OJYl4gWHd/6l+vqVOKH4 RbealXcGaruPYgXCosGiFVXadfpYFFmfSO7sHFNJMJISC5ZezWpAfu51WCAIWiy4 bjwSuccIFtX2UISjm0TfyUX1q3jea+CRL4is8C3xkm/6zXQU8DGmWigK7Wui+czf OO35/+6qP/iwiZVjMGDac8nEPU4T8szrHigL6avvXwjbZLO05atAlfexm7Hg3UOl /ITDfhcEL1NSQUAgGfcR1cKn/t0ThFSDUaa+67AxURPMxpaz3MquE+FoDiUpn4fC kd0S5nGVIQj2Jd+n+E7+3/C8IDUSqq9nc2r9WzrOJk0G61ehnwxex1oH6rLvfaTG CO+NsEVBk+U3QpXHRXqwm17Pbxs50lXEB/PFixf+e0VN+pwg4w8bGsqWO2646o/y nxfj8+IyWRC3XW97l5qvMno+YMHH2qTnUgp5Z0QWWVWfbigFnIV8j1eHwc6+e67h AHHvb9YxA5unHl9b6Rf/nfZuac1G59+vaDUX5YABwim6jmJ/DVJUXwWunlphwTzc D4F1nLaaMob3cM1vCZPA
=pIKZ
-----END PGP SIGNATURE-----

Joshua Schmid

author Tested-by

2015-02-03 11:26 UTC #1 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 7 lines · Joshua Schmid

Hi,
I tested this patch for some time and it really helps to fix the gc issue we are running in. Since it might got lost, i will resend it.

patch attached*

Quoted 6 lines

Best Regards, Joshua

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBAgAGBQJU0LBlAAoJEPUnwXO0u5uWMVgP/iUGNroCzUSW0lGvV+awLyPX +p7O5tERuzs7Uq+yKBoE8Zvad76WuS1vnrLqjZgfI+3P4dGJPpf1DlLHlFU7dHkF /UBgqQFf6OT5MgXk+TXKLXitH+55Chl7WaVe3RMgXEMP3Q5R/rjvLM8Po5gLfRc8 uru7k0oFi+B7cnOmxXruD74baCXY+nwuXoE7PTyOzH+1Wf7SHsz8ygGNVbDhVWI4 A1zaTqoBMNXribBd3wN5uRV7XDHL5un+zF7sR8P9BiXHn1GKPwH7qLesHHTT2nvs joN7+xMx8In0cR4uloSDqgmwT/6OdeFCujKErwIkVIl57YqEUimByxcA4lGzXvXZ C5n760Ei/nw0X70qeA2Jkfg/Zk1HfwPcfkHVIi5KB4LelfVa1jbIEaSzdnclH2aD GlakovGJOkrcPhBgH0xc+7h2eH1kv6Vw8Al1AbTNho92vzVCKmcmGMUyuBpkmxnD zdxoXUKZLEuGWCt0ExYBU8tIw3twpatCA15sZ9kbgwkggqri7HHqjA8M/cMjPXKC VNStPPnscMbMjxDvZiJyNrCchi2kInJWLKJWNofk5RZs7f2eQcT1bXKYWxXPGIxn QZhKsrItZ/k0g2GrxWCRxH6fymHVKkokzt6R0FZcs/1Quf+CnwrhJ5ZPQivkdKLX BUIG+BeOeYwkLm27/VoH
=XQtd
-----END PGP SIGNATURE-----

From ab8c276a4997f394e252688f855e1b35374aedee Mon Sep 17 00:00:00 2001
From: Kent Overstreet <[email protected]>
Date: Sat, 1 Nov 2014 13:44:47 -0700
Subject: [PATCH] 3.17-rc6: bcache_gc: BUG: soft lockup - CPU#2 stuck for 23s!

Quoted 10 lines · Eric Wheeler

Hello Kent, Ross, all:
We're getting bcache_gc backtraces and soft lockups; the system continues to be responsive and eventually recovers. We are running 3.17-rc6. (This appears to be a continuation of the thread from 2014-09-15)
Please see the following two backtraces. The first shows up in btree_gc_count_keys(), the other is triggered somehow by rcu_sched. We will test with -rc7 this week, though I didn't see any bcache commits in rc7.

Quoted 8 lines

The server is quite busy:
dd in userspace from dm-thinp snapshots to another server two DRBD verify's active backed by dm-thinp volumes
note that, dd fills up the buffers so this could be operating with few pages free. (Though we have min-mem set to 256MB.)
I see we are hitting functions like bch_ptr_bad() and bch_extent_bad(). Could that indicate a cache corruption on our volume?

No - those are the normal "check the validity of medata" functions.

Quoted 2 lines

I'm happy to test patches if you have any suggestions or tests that I should run it through.

I think it might just be a missing cond_resched()... there's a check during garbage collection for need_resched() but it appears we might not actually be calling schedule() then.

Try this patch:

Hi, i tested this patch and it fixes our hangups. I'm afraid it got lost so i am resending it.

commit a64afc92e17e709bdd1618edd04bc608f6a44c55

Author: Kent Overstreet <[email protected]>
Date:   Sat Nov 1 13:44:13 2014 -0700

bcache: Add a cond_resched() call to gc

Change-Id: Id4f18c533b80ddb40df94ed0bb5e2a236a4bc325

Tested-by Joshua Schmid <[email protected]>

---
 drivers/md/bcache/btree.c | 1 +
 1 file changed, 1 insertion(+)

drivers/md/bcache/btree.c +1 -0

index 00cde40..218f21a 100644
@@ -1741,6 +1741,7 @@ static void bch_btree_gc(struct cache_set *c)
 	do {
 		ret = btree_root(gc_root, c, &op, &writes, &stats);
 		closure_sync(&writes);
+		cond_resched();
 
 		if (ret && ret != -EAGAIN)
 			pr_warn("gc failed!");

Signature · 1 line

2.1.2