Evil elevator to test cache flushes

Hi everyone,

We recently found some bugs in the btrfs cache flush code. The frustrating part was that I wasn't able to trigger these bugs by ripping the plug out of my test box, no matter how many times I tried.

The problem is that you have to rip out the cord at just the right time, and that's pretty hard to catch. I've been meaning to make an elevator or dm device or qemu hack to emulate a writeback cache for a while, and so here we go.

A big part of this elevator is the bit that reboots the box in the middle of servicing a cache flush (enabled via sysfs flag). With this on, I'm able to trigger the btrfs bug 100% of the time, and it doesn't trigger corruptions with the btrfs bug fixed. So far I haven't seen corruptions in the other filesystems unless I turn barriers off, but I've only tried a few runs.

I'm writing up a script to bash on things in a big reboot loop, so we should be able to get a lot of crashes in very little time.

I owe Jens at least one change to this patch. I made an elevator op to process cache flushes, and Jens doesn't like the part where I made it conditional (he wants a default operation given to the other elevators).

Other parts that should be unpopular include abusing elvpriv to make the elevator switching code wait for my elevator to exit, and my code to duplicate requests/bios.

I'm happy to include suggestions that make this elevator even more evil.

-chris

-------------------

block: add new barrier-test io scheduler

This elevator tries to simulate a drive with a non-battery backed writeback cache. Writes are completed immediately up to the caller and a duplicate of the request is sent down to the storage after the higher level has sent down a cache FLUSH operation.

This is only meant for debugging purposes, and it intentionally tries to send down the IO in an inefficient order in hopes of catching corner cases or bugs in the filesystem cache flush operations.

It also includes a sysfs file called random_reboot, which will call emergency_reboot() while it is processing a cache flush. It triggers the reboot only when a given flush is a fairly large operation, and it waits until after it has written about half the blocks.

When servicing a cache flush, it sends the IO down to disk in reverse order. This is assuming the FS writes the most important blocks last, so reversing the order is going to confuse the FS the most.

This doesn't maintain an index of things in the writeback cache, so any read operation triggers a full flush. This could be fixed by indexing the duplicate bios and servicing reads directly from the bios.

index 3199b76..1bc1925 100644
@@ -12,6 +12,14 @@ config IOSCHED_NOOP
 	  that do their own scheduling and require only minimal assistance from
 	  the kernel.
 
+config IOSCHED_BARRIER_TEST
+	tristate "Barrier testing I/O scheduler"
+	default n
+	---help---
+	  This is for testing purposes only.  It tries to simulate a worst case
+	  writeback cache that only properly flushes writes when barriers are
+	  done.  Say no here.
+
 config IOSCHED_DEADLINE
 	tristate "Deadline I/O scheduler"
 	default y

index 514c6e4..216a007 100644
@@ -14,6 +14,7 @@ obj-$(CONFIG_BLK_DEV_THROTTLING)	+= blk-throttle.o
 obj-$(CONFIG_IOSCHED_NOOP)	+= noop-iosched.o
 obj-$(CONFIG_IOSCHED_DEADLINE)	+= deadline-iosched.o
 obj-$(CONFIG_IOSCHED_CFQ)	+= cfq-iosched.o
+obj-$(CONFIG_IOSCHED_BARRIER_TEST)	+= barrier-test-iosched.o
 
 obj-$(CONFIG_BLOCK_COMPAT)	+= compat_ioctl.o
 obj-$(CONFIG_BLK_DEV_INTEGRITY)	+= blk-integrity.o

index 491eb30..16004ba 100644
@@ -346,6 +346,7 @@ void blk_insert_flush(struct request *rq)
 
 	blk_flush_complete_seq(rq, REQ_FSEQ_ACTIONS & ~policy, 0);
 }
+EXPORT_SYMBOL(blk_insert_flush);
 
 /**
  * blk_abort_flushes - @q is being aborted, abort flush requests

index a3b64bc..316719c 100644
@@ -716,7 +716,10 @@ void __elv_add_request(struct request_queue *q, struct request *rq, int where)
 
 	case ELEVATOR_INSERT_FLUSH:
 		rq->cmd_flags |= REQ_SOFTBARRIER;
-		blk_insert_flush(rq);
+		if (q->elevator->ops->elevator_add_flush_fn)
+			q->elevator->ops->elevator_add_flush_fn(q, rq);
+		else
+			blk_insert_flush(rq);
 		break;
 	default:
 		printk(KERN_ERR "%s: bad insertion point %d\n",

index d800d51..9c3f667 100644
@@ -42,6 +42,7 @@ struct elevator_ops
 
 	elevator_dispatch_fn *elevator_dispatch_fn;
 	elevator_add_req_fn *elevator_add_req_fn;
+	elevator_add_req_fn *elevator_add_flush_fn;
 	elevator_activate_req_fn *elevator_activate_req_fn;
 	elevator_deactivate_req_fn *elevator_deactivate_req_fn;