Skip to contentSkip to conversation

dm: log writes target

RFC PATCH In review Josef Bacik 2 participants 4 messages Dec 8, 2014 – Feb 5, 2015

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 dm: log writes target

This is my latest attempt at a target for testing power fail and fs consistency. This is based on the idea Zach Brown had where we could just walk through all the operations done to an fs in order to verify we're doing the correct thing. There is a userspace component as well that can be found here

https://github.com/josefbacik/log-writes

It is very rough as I just threw it together to test the various aspects of how you would want to replay a log to test it. Again I would love feedback on this, I really want to have something that we all think is usefull and eventually incorporate it into xfstests.

From: Josef Bacik <[email protected]>
Subject: [PATCH]dm: log writes target

This creates a new target that is meant for file system developers to test file system integrity at particular points in the life of a file system. We capture all write requests and the data and log the requests and the data to a separate device for later replay. There is a userspace utility to do this replay. The idea behind this is to give file system developers to verify that the file system is always consistent. Thanks,

---
 Documentation/device-mapper/dm-log-writes.txt | 136 +++++
 drivers/md/Kconfig                            |  16 +
 drivers/md/Makefile                           |   1 +
 drivers/md/dm-log-writes.c                    | 835 ++++++++++++++++++++++++++
 4 files changed, 988 insertions(+)
 create mode 100644 Documentation/device-mapper/dm-log-writes.txt
 create mode 100644 drivers/md/dm-log-writes.c
Documentation/device-mapper/dm-log-writes.txt new file +136 -0
new file mode 100644
index 0000000..f3a9fa2
@@ -0,0 +1,136 @@
+dm-log-writes
+=============
+
+This target takes 2 devices, one to pass all IO to normally, and one to log all
+of the write operations to.  This is intended for file system developers wishing
+to verify the integrity of metadata or data as the file system is written to.
+There is a log_writes_entry written for every WRITE request and the target is
+able to take arbitrary data from userspace to insert into the log.  The data
+that is in the WRITE requests is copied into the log to make the replay happen
+exactly as it happened originally.
+
+Log Ordering
+============
+
+We log things in order of completion once we are sure the write is no longer in
+cache.  This means that normal WRITE requests are not actually logged until the
+next REQ_FLUSH request.  This is to make it easier for userspace to replay the
+log in a way that correlates to what is on disk and not what is in cache, to
+make it easier to detect improper waiting/flushing.
+
+This works by attaching all WRITE requests to a list once the write completes.
+Once we see a REQ_FLUSH request we splice this list onto the request and once
+the FLUSH request completes we log all of the WRITE's and then the FLUSH.  Only
+completeled WRITEs at the time of the issue of the REQ_FLUSH are added in order
+to simulate the worst case scenario with regard to power failures.  Consider the
+following example (W means write, C means complete)
+
+W1,W2,W3,C3,C2,Wflush,C1,Cflush
+
+The log would show the following
+
+W3,W2,flush,W1....
+
+Again this is to simulate what is actually on disk, this allows us to detect
+cases where a power failure at a particular point in time would create an
+inconsistent file system.
+
+Any REQ_FUA requests bypass this flushing mechanism and are logged as soon as
+they complete as those requests will obviously bypass the device cache.
+
+Any REQ_DISCARD requests are treated like WRITE requests.  This is because
+otherwise we would have all the DISCARD requests, and then the WRITE requests
+and then the FLUSH request.  Consider the following example
+
+WRITE block 1, DISCARD block 1, FLUSH
+
+If we logged DISCARD when it completed, the replay would look like this
+
+DISCARD 1, WRITE 1, FLUSH
+
+which isn't quite what happened and wouldn't be caught during the log replay.
+
+Marks
+=====
+
+You can use dmsetup to set an arbitrary mark in a log.  For example say you want
+to fsck an file system after every write, but first you need to replay up to the
+mkfs to make sure we're fsck'ing something reasonable, you would do something
+like this
+
+mkfs.btrfs -f /dev/mapper/log
+dmsetup message log 0 mark mkfs
+<run test>
+
+This would allow you to replay the log up to the mkfs mark and then replay from
+that point on doing the fsck check in the interval that you want.
+
+Every log has a mark at the end labeled "log-writes-end".
+
+Userspace component
+===================
+
+There is a userspace tool that will replay the log for you in various ways.
+As of this writing the options are not well documented, they will be in the
+future.  It can be found here
+
+https://github.com/josefbacik/log-writes
+
+Example usage
+=============
+
+Say you want to test fsync on your file system.  You would do something like
+this
+
+TABLE="0 $(blockdev --getsz /dev/sdb) log-writes /dev/sdb /dev/sdc"
+dmsetup create log --table "$TABLE"
+mkfs.btrfs -f /dev/mapper/log
+dmsetup message log 0 mark mkfs
+
+mount /dev/mapper/log /mnt/btrfs-test
+<some test that does fsync at the end>
+dmsetup message log 0 mark fsync
+md5sum /mnt/btrfs-test/foo
+umount /mnt/btrfs-test
+
+dmsetup remove log
+replay-log --log /dev/sdc --replay /dev/sdb --end-mark fsync
+mount /dev/sdb /mnt/btrfs-test
+md5sum /mnt/btrfs-test/foo
+<verify md5sum's are correct>
+
+Another option is to do a complicated file system operation and verify the file
+system is consistent during the entire operation.  You could do this by doing
+
+
+TABLE="0 $(blockdev --getsz /dev/sdb) log-writes /dev/sdb /dev/sdc"
+dmsetup create log --table "$TABLE"
+mkfs.btrfs -f /dev/mapper/log
+dmsetup message log 0 mark mkfs
+
+mount /dev/mapper/log /mnt/btrfs-test
+<fsstress to dirty the fs>
+btrfs filesystem balance /mnt/btrfs-test
+umount /mnt/btrfs-test
+dmsetup remove log
+
+replay-log --log /dev/sdc --replay /dev/sdb --end-mark mkfs
+btrfsck /dev/sdb
+replay-log --log /dev/sdc --replay /dev/sdb --start-mark mkfs \
+	--fsck "btrfsck /dev/sdb" --check fua
+
+And that will replay the log until it sees a FUA request, run the fsck command
+and if the fsck passes it will replay to the next FUA, until it is completed or
+the fsck command exists abnormally.
+
+Table Parameters
+----------------
+  <dev path> <dev path for log>
+
+Mandatory parameters:
+  <dev path>: Full pathname to the underlying block-device, or a "major:minor"
+              device-number.  This device is the one that all of the IO will go
+              to normally, just think of it as a normal linear mapping.
+  <dev path for log>: Same format as <dev path>, this is the device where the
+                      log entries are written to.
+
drivers/md/Kconfig +16 -0
index 5bdedf6..ca2128d 100644
@@ -432,4 +432,20 @@ config DM_SWITCH
 
 	  If unsure, say N.
 
+config DM_LOG_WRITES
+	tristate "Log writes target support"
+	depends on BLK_DEV_DM
+	---help---
+	  This device-mapper target takes two devices, one device to use
+	  normally, one to log all write operations done to the first device.
+	  This is for use by file system developers wishing to verify that
+	  their fs is writing a consitent file system at all times by allowing
+	  them to replay the log in a variety of ways and to check the
+	  contents.
+
+	  To compile this code as a module, choose M here: the module will
+	  be called dm-log-writes.
+
+	  If unsure, say N.
+
 endif # MD
drivers/md/Makefile +1 -0
index a2da532..1863fea 100644
@@ -55,6 +55,7 @@ obj-$(CONFIG_DM_CACHE)		+= dm-cache.o
 obj-$(CONFIG_DM_CACHE_MQ)	+= dm-cache-mq.o
 obj-$(CONFIG_DM_CACHE_CLEANER)	+= dm-cache-cleaner.o
 obj-$(CONFIG_DM_ERA)		+= dm-era.o
+obj-$(CONFIG_DM_LOG_WRITES)	+= dm-log-writes.o
 
 ifeq ($(CONFIG_DM_UEVENT),y)
 dm-mod-objs			+= dm-uevent.o
drivers/md/dm-log-writes.c new file +835 -0

835 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#1 lore raw
Quoted 11 lines · Josef Bacik

This is my latest attempt at a target for testing power fail and fs consistency. This is based on the idea Zach Brown had where we could just walk through all the operations done to an fs in order to verify we're doing the correct thing. There is a userspace component as well that can be found here

https://github.com/josefbacik/log-writes

It is very rough as I just threw it together to test the various aspects of how you would want to replay a log to test it. Again I would love feedback on this, I really want to have something that we all think is usefull and eventually incorporate it into xfstests.

hey josef,

i finally took a quick look at your target. has this target proven useful to you (or others) since you posted? would you still like to see this land upstream?

i see no need to stack discard limits if discards are supported by the underlying device (you'll get that for free via blk_stack_limits).

here is a patch that applies ontop of your original submission. mostly whitespace (i'm not pedantic about 80 char limit, etc). but i did include some small improvements and FIXMEs/questions in the patch:

drivers/md/dm-log-writes.c +34 -57
index 4aa627a..3c84777 100644
@@ -137,7 +137,7 @@ static void log_end_io(struct bio *bio, int err)
 	if (err) {
 		unsigned long flags;
 
-		DMERR("Error writing log block %d\n", err);
+		DMERR("Error writing log block %d", err);
 		spin_lock_irqsave(&lc->blocks_lock, flags);
 		lc->logging_enabled = false;
 		spin_unlock_irqrestore(&lc->blocks_lock, flags);
@@ -335,8 +335,7 @@ static int log_writes_kthread(void *arg)
 		spin_lock_irq(&lc->blocks_lock);
 		if (!list_empty(&lc->logging_blocks)) {
 			block = list_first_entry(&lc->logging_blocks,
-						 struct pending_block,
-						 list);
+						 struct pending_block, list);
 			list_del_init(&block->list);
 			if (!lc->logging_enabled)
 				goto next;
@@ -362,8 +361,7 @@ static int log_writes_kthread(void *arg)
 			lc->logged_entries++;
 			atomic_inc(&lc->io_blocks);
 
-			super = (block->flags &
-				 (LOG_FUA_FLAG | LOG_MARK_FLAG));
+			super = (block->flags & (LOG_FUA_FLAG | LOG_MARK_FLAG));
 			if (super)
 				atomic_inc(&lc->io_blocks);
 		}
@@ -380,9 +378,8 @@ next:
 					lc->logging_enabled = false;
 					spin_unlock_irq(&lc->blocks_lock);
 				}
-			} else {
+			} else
 				free_pending_block(lc, block);
-			}
 			continue;
 		}
 
@@ -429,15 +426,13 @@ static int log_writes_ctr(struct dm_target *ti, unsigned int argc, char **argv)
 	atomic_set(&lc->pending_blocks, 0);
 
 	devname = dm_shift_arg(&as);
-	if (dm_get_device(ti, devname, dm_table_get_mode(ti->table),
-			  &lc->dev)) {
+	if (dm_get_device(ti, devname, dm_table_get_mode(ti->table), &lc->dev)) {
 		ti->error = "Device lookup failed";
 		goto bad;
 	}
 
 	logdevname = dm_shift_arg(&as);
-	if (dm_get_device(ti, logdevname, dm_table_get_mode(ti->table),
-			  &lc->logdev)) {
+	if (dm_get_device(ti, logdevname, dm_table_get_mode(ti->table), &lc->logdev)) {
 		ti->error = "Log device lookup failed";
 		dm_put_device(ti, lc->dev);
 		goto bad;
@@ -477,13 +472,13 @@ static int log_mark(struct log_writes_c *lc, char *data)
 
 	block = kzalloc(sizeof(struct pending_block), GFP_KERNEL);
 	if (!block) {
-		DMERR("Error allocating pending block\n");
+		DMERR("Error allocating pending block");
 		return -ENOMEM;
 	}
 
 	block->data = kstrndup(data, maxsize, GFP_KERNEL);
 	if (!block->data) {
-		DMERR("Error copying mark data\n");
+		DMERR("Error copying mark data");
 		kfree(block);
 		return -ENOMEM;
 	}
@@ -527,9 +522,10 @@ static void normal_map_bio(struct dm_target *ti, struct bio *bio)
 	struct log_writes_c *lc = ti->private;
 
 	bio->bi_bdev = lc->dev->bdev;
+	// FIXME: why would bi_sector ever need to be changed?
+	// if you just copied dm-linear then it is misplaced since there isn't an offset
 	if (bio_sectors(bio))
-		bio->bi_iter.bi_sector =
-			dm_target_offset(ti, bio->bi_iter.bi_sector);
+		bio->bi_iter.bi_sector = dm_target_offset(ti, bio->bi_iter.bi_sector);
 }
 
 static int log_writes_map(struct dm_target *ti, struct bio *bio)
@@ -568,8 +564,8 @@ static int log_writes_map(struct dm_target *ti, struct bio *bio)
 	if (discard_bio)
 		alloc_size = sizeof(struct pending_block);
 	else
-		alloc_size = sizeof(struct pending_block) +
-			sizeof(struct bio_vec) * bio_segments(bio);
+		alloc_size = sizeof(struct pending_block) + sizeof(struct bio_vec) * bio_segments(bio);
+
 	block = kzalloc(alloc_size, GFP_NOIO);
 	if (!block) {
 		DMERR("Error allocating pending block");
@@ -614,6 +610,9 @@ static int log_writes_map(struct dm_target *ti, struct bio *bio)
 	 * the actual contents into new pages so we know the data will always be
 	 * there.
 	 */
+	// FIXME: why not just hold onto the original bio until "way later"?
+	// would doing so compromise the target's function?
+	// seems it'd avoid all this duplication (of state and data) in pending_block
 	bio_for_each_segment(bv, bio, iter) {
 		struct page *page;
 		void *src, *dst;
@@ -661,16 +660,14 @@ static int normal_end_io(struct dm_target *ti, struct bio *bio, int error)
 
 		spin_lock_irqsave(&lc->blocks_lock, flags);
 		if (block->flags & LOG_FLUSH_FLAG) {
-			list_splice_tail_init(&block->list,
-					      &lc->logging_blocks);
+			list_splice_tail_init(&block->list, &lc->logging_blocks);
 			list_add_tail(&block->list, &lc->logging_blocks);
 			wake_up_process(lc->log_kthread);
 		} else if (block->flags & LOG_FUA_FLAG) {
 			list_add_tail(&block->list, &lc->logging_blocks);
 			wake_up_process(lc->log_kthread);
-		} else {
+		} else
 			list_add_tail(&block->list, &lc->unflushed_blocks);
-		}
 		spin_unlock_irqrestore(&lc->blocks_lock, flags);
 	}
 
@@ -692,11 +689,11 @@ static void log_writes_status(struct dm_target *ti, status_type_t type,
 		DMEMIT("%llu %llu", lc->logged_entries,
 		       (unsigned long long)lc->next_sector - 1);
 		if (!lc->logging_enabled)
-			DMEMIT(" logging disabled");
+			DMEMIT(" logging_disabled");
 		break;
 
 	case STATUSTYPE_TABLE:
-		DMEMIT("%s %s ", lc->dev->name, lc->logdev->name);
+		DMEMIT("%s %s", lc->dev->name, lc->logdev->name);
 		break;
 	}
 }
@@ -742,57 +739,37 @@ static int log_writes_iterate_devices(struct dm_target *ti,
 }
 
 /*
- * Valid messages
- *
- * mark <mark data> - specify the marked data.
+ * Messages supported:
+ *   mark <mark data> - specify the marked data.
  */
 static int log_writes_message(struct dm_target *ti, unsigned argc, char **argv)
 {
+	int r = -EINVAL;
 	struct log_writes_c *lc = ti->private;
 
 	if (argc != 2) {
-		DMWARN("Invalid log-writes message arguments, expect 2 "
-		       "argument, got %d", argc);
-		return -EINVAL;
+		DMWARN("Invalid log-writes message arguments, expect 2 arguments, got %d", argc);
+		return r;
 	}
 
-	if (!strcasecmp(argv[0], "mark")) {
-		int ret;
-
-		ret = log_mark(lc, argv[1]);
-		if (ret)
-			return ret;
-	} else {
-		DMWARN("Invalid argument %s", argv[0]);
-		return -EINVAL;
-	}
+	if (!strcasecmp(argv[0], "mark"))
+		r = log_mark(lc, argv[1]);
+	else
+		DMWARN("Unrecognised log writes target message received: %s", argv[0]);
 
-	return 0;
+	return r;
 }
 
-static void log_writes_io_hints(struct dm_target *ti,
-			       struct queue_limits *limits)
+static void log_writes_io_hints(struct dm_target *ti, struct queue_limits *limits)
 {
 	struct log_writes_c *lc = ti->private;
 	struct request_queue *q = bdev_get_queue(lc->dev->bdev);
-	sector_t max_discard = (UINT_MAX >> SECTOR_SHIFT);
 
-	if (!q || !blk_queue_discard(q))
+	if (!q || !blk_queue_discard(q)) {
 		lc->device_supports_discard = false;
-
-	if (lc->device_supports_discard) {
-		struct queue_limits *data_limits = &q->limits;
-		limits->max_discard_sectors =
-			min_t(unsigned int, data_limits->max_discard_sectors,
-			      max_discard);
-		limits->discard_granularity =
-			max_t(unsigned int, data_limits->discard_granularity,
-			      1 << SECTOR_SHIFT);
-	} else {
 		limits->discard_granularity = 1 << SECTOR_SHIFT;
-		limits->max_discard_sectors = max_discard;
+		limits->max_discard_sectors = (UINT_MAX >> SECTOR_SHIFT);
 	}
-	dump_stack();
 }
 
 static struct target_type log_writes_target = {
@@ -830,6 +807,6 @@ static void __exit dm_log_writes_exit(void)
 module_init(dm_log_writes_init);
 module_exit(dm_log_writes_exit);
 
-MODULE_DESCRIPTION(DM_NAME " write log target");
+MODULE_DESCRIPTION(DM_NAME " log writes target");
 MODULE_AUTHOR("Josef Bacik <[email protected]>");
 MODULE_LICENSE("GPL");
#2 lore raw

drivers/md/dm-log-writes.c static void normal_map_bio(struct dm_target *ti, struct bio *bio) · line 529

Quoted 11 lines · Mike Snitzer
drivers/md/dm-log-writes.c +3 -2
@@ -527,9 +522,10 @@ static void normal_map_bio(struct dm_target *ti, struct bio *bio)
 	struct log_writes_c *lc = ti->private;
 
 	bio->bi_bdev = lc->dev->bdev;
+	// FIXME: why would bi_sector ever need to be changed?
+	// if you just copied dm-linear then it is misplaced since there isn't an offset
 	if (bio_sectors(bio))
-		bio->bi_iter.bi_sector =
-			dm_target_offset(ti, bio->bi_iter.bi_sector);
+		bio->bi_iter.bi_sector = dm_target_offset(ti, bio->bi_iter.bi_sector);
 }

In above FIXME: s/misplaced/incomplete/

FYI sharing this quick private exchange I had with Alasdair just to make sure I'm clear on what the above FIXME was trying to convey:

Quoted 17 lines · Mike Snitzer
(quoted diff) +0 -0
 On Wed, Feb 04 2015 at  5:43pm -0500,
Quoted 5 lines · Alasdair G Kergon
Quoted 2 lines · Mike Snitzer

+ // FIXME: why would bi_sector ever need to be changed?
+ // if you just copied dm-linear then it is misplaced since there isn't an offset

Why shouldn't we use this target on the 2nd or later line of a

table?

Wasn't saying it shouldn't. Just was saying that Josef's target didn't go as far as say dm-linear with its 'start' offset. Without it then this dm-log-writes target implicitly assumes every logical address space, be it 2nd or later, identity maps to exact same offset on the backing the data disk.

My FIXME was too cryptic. I can follow-up on this point.

#3 lore raw
In reply to Mike Snitzer
Quoted 21 lines · Mike Snitzer
Quoted 11 lines · Josef Bacik

This is my latest attempt at a target for testing power fail and fs consistency. This is based on the idea Zach Brown had where we could just walk through all the operations done to an fs in order to verify we're doing the correct thing. There is a userspace component as well that can be found here

https://github.com/josefbacik/log-writes

It is very rough as I just threw it together to test the various aspects of how you would want to replay a log to test it. Again I would love feedback on this, I really want to have something that we all think is usefull and eventually incorporate it into xfstests.

hey josef,

i finally took a quick look at your target. has this target proven useful to you (or others) since you posted? would you still like to see this land upstream?

Quoted 7 lines

i see no need to stack discard limits if discards are supported by the underlying device (you'll get that for free via blk_stack_limits).

here is a patch that applies ontop of your original submission. mostly whitespace (i'm not pedantic about 80 char limit, etc). but i did include some small improvements and FIXMEs/questions in the patch:

Hello,

Yup I still want to land this upstream, I've been using it to find balance bugs and other things in btrfs. With the holidays and all my family medical things going on recently I haven't had as much time to put into this as I wanted but I've made some improvements to it recently. My plan was to wait until LSF so I could present my ideas to all the FS and DM guys at the same time to make sure we were happy with this approach and then submit shortly after that.

I will apply this and look through the FIXME's soon and send another update to make sure things still look good. Thanks for looking at this,

Josef