Skip to contentSkip to conversation

md: Introduction

PATCH In review NeilBrown 2 participants 7 messages Aug 24, 2006 – Aug 25, 2006

Subsystems (2)
Documentation MD RAID
lore
Share & tools
lore

NeilBrown

author
#0 lore raw

PATCH 1/1 md: Introduction

Following are 4 patches against 2.6.18-rc4-mm2

The first 2 are bug fixes which should go in 2.6.18, and apply equally well to that tree as to -mm.

The latter two should stay in -mm until after 2.6.18.

The second patch is maybe bigger than it absolutely needs to be as a bugfix. If you like I can stripe out all the rcu-extra-carefulness as a separate patch and just leave the important bit which involves moving the atomic_add down twenty-something lines.

Thanks,
NeilBrown

[PATCH 001 of 4] md: Fix recent breakage of md/raid1 array checking [PATCH 002 of 4] md: Fix issues with referencing rdev in md/raid1. [PATCH 003 of 4] md: new sysfs interface for setting bits in the write-intent-bitmap [PATCH 004 of 4] md: Remove unnecessary variable x in stripe_to_pdidx().

author
#1 lore raw

PATCH 1/1 md: Fix recent breakage of md/raid1 array checking

A recent patch broke the ability to do a user-request check of a raid1.
This patch fixes the breakage and also moves a comment that was dislocated by the same patch.

Signed-off-by Neil Brown <[email protected]>

### Diffstat output

 ./drivers/md/raid1.c |    7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
drivers/md/raid1.c +4 -3
diff .prev/drivers/md/raid1.c ./drivers/md/raid1.c
@@ -1644,15 +1644,16 @@ static sector_t sync_request(mddev_t *md
 		return 0;
 	}
 
-	/* before building a request, check if we can skip these blocks..
-	 * This call the bitmap_start_sync doesn't actually record anything
-	 */
 	if (mddev->bitmap == NULL &&
 	    mddev->recovery_cp == MaxSector &&
+	    !test_bit(MD_RECOVERY_REQUESTED, &mddev->recovery) &&
 	    conf->fullsync == 0) {
 		*skipped = 1;
 		return max_sector - sector_nr;
 	}
+	/* before building a request, check if we can skip these blocks..
+	 * This call the bitmap_start_sync doesn't actually record anything
+	 */
 	if (!bitmap_start_sync(mddev->bitmap, sector_nr, &sync_blocks, 1) &&
 	    !conf->fullsync && !test_bit(MD_RECOVERY_REQUESTED, &mddev->recovery)) {
 		/* We can skip this block, and probably several more */
author
#2 lore raw

PATCH 1/1 md: Fix issues with referencing rdev in md/raid1.

In reply to NeilBrown

We need to be careful when referencing mirrors[i].rdev. so it can disappear under us at various times.

So:
fix a couple of problem places.
comment a couple of non-problem places
move an 'atomic_add' which deferences rdev down a little
way to some where where it is sure to not be NULL.

Signed-off-by Neil Brown <[email protected]>

### Diffstat output

 ./drivers/md/raid1.c |   55 +++++++++++++++++++++++++++++++--------------------
 1 file changed, 34 insertions(+), 21 deletions(-)
drivers/md/raid1.c +34 -21
diff .prev/drivers/md/raid1.c ./drivers/md/raid1.c
@@ -930,10 +930,13 @@ static void status(struct seq_file *seq,
 
 	seq_printf(seq, " [%d/%d] [", conf->raid_disks,
 		   conf->raid_disks - mddev->degraded);
-	for (i = 0; i < conf->raid_disks; i++)
+	rcu_read_lock();
+	for (i = 0; i < conf->raid_disks; i++) {
+		mdk_rdev_t *rdev = rcu_dereference(conf->mirrors[i].rdev);
 		seq_printf(seq, "%s",
-			      conf->mirrors[i].rdev &&
-			      test_bit(In_sync, &conf->mirrors[i].rdev->flags) ? "U" : "_");
+			   rdev && test_bit(In_sync, &rdev->flags) ? "U" : "_");
+	}
+	rcu_read_unlock();
 	seq_printf(seq, "]");
 }
 

@@ -976,7 +979,6 @@ static void error(mddev_t *mddev, mdk_rd
 static void print_conf(conf_t *conf)
 {
 	int i;
-	mirror_info_t *tmp;
 
 	printk("RAID1 conf printout:\n");
 	if (!conf) {

@@ -986,14 +988,17 @@ static void print_conf(conf_t *conf)
 	printk(" --- wd:%d rd:%d\n", conf->raid_disks - conf->mddev->degraded,
 		conf->raid_disks);
 
+	rcu_read_lock();
 	for (i = 0; i < conf->raid_disks; i++) {
 		char b[BDEVNAME_SIZE];
-		tmp = conf->mirrors + i;
-		if (tmp->rdev)
+		mdk_rdev_t *rdev = rcu_dereference(conf->mirrors[i].rdev);
+		if (rdev)
 			printk(" disk %d, wo:%d, o:%d, dev:%s\n",
-				i, !test_bit(In_sync, &tmp->rdev->flags), !test_bit(Faulty, &tmp->rdev->flags),
-				bdevname(tmp->rdev->bdev,b));
+			       i, !test_bit(In_sync, &rdev->flags),
+			       !test_bit(Faulty, &rdev->flags),
+			       bdevname(rdev->bdev,b));
 	}
+	rcu_read_unlock();
 }
 
 static void close_sync(conf_t *conf)

@@ -1009,17 +1014,17 @@ static int raid1_spare_active(mddev_t *m
 {
 	int i;
 	conf_t *conf = mddev->private;
-	mirror_info_t *tmp;
 
 	/*
 	 * Find all failed disks within the RAID1 configuration 
-	 * and mark them readable
+	 * and mark them readable.
+	 * Called under mddev lock, so rcu protection not needed.
 	 */
 	for (i = 0; i < conf->raid_disks; i++) {
-		tmp = conf->mirrors + i;
-		if (tmp->rdev 
-		    && !test_bit(Faulty, &tmp->rdev->flags)
-		    && !test_and_set_bit(In_sync, &tmp->rdev->flags)) {
+		mdk_rdev_t *rdev = conf->mirrors[i].rdev;
+		if (rdev
+		    && !test_bit(Faulty, &rdev->flags)
+		    && !test_and_set_bit(In_sync, &rdev->flags)) {
 			unsigned long flags;
 			spin_lock_irqsave(&conf->device_lock, flags);
 			mddev->degraded--;

@@ -1239,7 +1244,7 @@ static void sync_request_write(mddev_t *
 		/* ouch - failed to read all of that.
 		 * Try some synchronous reads of other devices to get
 		 * good data, much like with normal read errors.  Only
-		 * read into the pages we already have so they we don't
+		 * read into the pages we already have so we don't
 		 * need to re-issue the read request.
 		 * We don't need to freeze the array, because being in an
 		 * active sync request, there is no normal IO, and

@@ -1259,6 +1264,10 @@ static void sync_request_write(mddev_t *
 				s = PAGE_SIZE >> 9;
 			do {
 				if (r1_bio->bios[d]->bi_end_io == end_sync_read) {
+					/* No rcu protection needed here devices
+					 * can only be removed when no resync is
+					 * active, and resync is currently active
+					 */
 					rdev = conf->mirrors[d].rdev;
 					if (sync_page_io(rdev->bdev,
 							 sect + rdev->data_offset,

@@ -1376,6 +1385,11 @@ static void fix_read_error(conf_t *conf,
 			s = PAGE_SIZE >> 9;
 
 		do {
+			/* Note: no rcu protection needed here
+			 * as this is synchronous in the raid1d thread
+			 * which is the thread that might remove
+			 * a device.  If raid1d ever becomes multi-threaded....
+			 */
 			rdev = conf->mirrors[d].rdev;
 			if (rdev &&
 			    test_bit(In_sync, &rdev->flags) &&

@@ -1403,7 +1417,6 @@ static void fix_read_error(conf_t *conf,
 				d = conf->raid_disks;
 			d--;
 			rdev = conf->mirrors[d].rdev;
-			atomic_add(s, &rdev->corrected_errors);
 			if (rdev &&
 			    test_bit(In_sync, &rdev->flags)) {
 				if (sync_page_io(rdev->bdev,

@@ -1429,7 +1442,8 @@ static void fix_read_error(conf_t *conf,
 				    == 0)
 					/* Well, this device is dead */
 					md_error(mddev, rdev);
-				else
+				else {
+					atomic_add(s, &rdev->corrected_errors);
 					printk(KERN_INFO
 					       "raid1:%s: read error corrected "
 					       "(%d sectors at %llu on %s)\n",

@@ -1437,6 +1451,7 @@ static void fix_read_error(conf_t *conf,
 					       (unsigned long long)sect +
 					           rdev->data_offset,
 					       bdevname(rdev->bdev, b));
+				}
 			}
 		}
 		sectors -= s;

@@ -1806,19 +1821,17 @@ static sector_t sync_request(mddev_t *md
 		for (i=0; i<conf->raid_disks; i++) {
 			bio = r1_bio->bios[i];
 			if (bio->bi_end_io == end_sync_read) {
-				md_sync_acct(conf->mirrors[i].rdev->bdev, nr_sectors);
+				md_sync_acct(bio->bi_bdev, nr_sectors);
 				generic_make_request(bio);
 			}
 		}
 	} else {
 		atomic_set(&r1_bio->remaining, 1);
 		bio = r1_bio->bios[r1_bio->read_disk];
-		md_sync_acct(conf->mirrors[r1_bio->read_disk].rdev->bdev,
-			     nr_sectors);
+		md_sync_acct(bio->bi_bdev, nr_sectors);
 		generic_make_request(bio);
 
 	}
-
 	return nr_sectors;
 }
author
#3 lore raw

PATCH 1/1 md: new sysfs interface for setting bits in the write-intent-bitmap

In reply to NeilBrown

From: Paul Clements <[email protected]>

This patch (tested against 2.6.18-rc1-mm1) adds a new sysfs interface that allows the bitmap of an array to be dirtied. The interface is write-only, and is used as follows:

echo "1000" > /sys/block/md2/md/bitmap

(dirty the bit for chunk 1000 [offset 0] in the in-memory and on-disk bitmaps of array md2)

echo "1000-2000" > /sys/block/md1/md/bitmap

(dirty the bits for chunks 1000-2000 in md1's bitmap)

This is useful, for example, in cluster environments where you may need to combine two disjoint bitmaps into one (following a server failure, after a secondary server has taken over the array). By combining the bitmaps on the two servers, a full resync can be avoided (This was discussed on the list back on March 18, 2005, "[PATCH 1/2] md bitmap bug fixes" thread).

Signed-off-by Neil Brown <[email protected]>

### Diffstat output

 ./Documentation/md.txt        |    9 +++++++++
 ./drivers/md/bitmap.c         |   14 ++++++++++++++
 ./drivers/md/md.c             |   31 +++++++++++++++++++++++++++++++
 ./include/linux/raid/bitmap.h |    2 ++
 4 files changed, 56 insertions(+)
Documentation/md.txt +9 -0
diff .prev/Documentation/md.txt ./Documentation/md.txt
@@ -410,6 +410,15 @@ also have
       than sectors, this my be larger than the number of actual errors
       by a factor of the number of sectors in a page.
 
+   bitmap_set_bits
+      If the array has a write-intent bitmap, then writing to this
+      attribute can set bits in the bitmap, indicating that a resync
+      would need to check the corresponding blocks. Either individual
+      numbers or start-end pairs can be written.  Multiple numbers
+      can be separated by a space.
+      Note that the numbers are 'bit' numbers, not 'block' numbers.
+      They should be scaled by the bitmap_chunksize.
+
 Each active md device may also have attributes specific to the
 personality module that manages it.
 These are specific to the implementation of the module and could
drivers/md/bitmap.c +14 -0
diff .prev/drivers/md/bitmap.c ./drivers/md/bitmap.c
@@ -613,6 +613,7 @@ static inline unsigned long file_page_of
 static inline struct page *filemap_get_page(struct bitmap *bitmap,
 					unsigned long chunk)
 {
+	if (file_page_index(chunk) >= bitmap->file_pages) return NULL;
 	return bitmap->filemap[file_page_index(chunk) - file_page_index(0)];
 }
 

@@ -739,6 +740,7 @@ static void bitmap_file_set_bit(struct b
 	}
 
 	page = filemap_get_page(bitmap, chunk);
+	if (!page) return;
 	bit = file_page_offset(chunk);
 
  	/* set the bit */

@@ -1322,6 +1324,18 @@ static void bitmap_set_memory_bits(struc
 
 }
 
+/* dirty the memory and file bits for bitmap chunks "s" to "e" */
+void bitmap_dirty_bits(struct bitmap *bitmap, unsigned long s, unsigned long e)
+{
+	unsigned long chunk;
+
+	for (chunk = s; chunk <= e; chunk++) {
+		sector_t sec = chunk << CHUNK_BLOCK_SHIFT(bitmap);
+		bitmap_set_memory_bits(bitmap, sec, 1);
+		bitmap_file_set_bit(bitmap, sec);
+	}
+}
+
 /*
  * flush out any pending updates
  */
drivers/md/md.c +31 -0
diff .prev/drivers/md/md.c ./drivers/md/md.c
@@ -2524,6 +2524,36 @@ static struct md_sysfs_entry md_new_devi
 __ATTR(new_dev, S_IWUSR, null_show, new_dev_store);
 
 static ssize_t
+bitmap_store(mddev_t *mddev, const char *buf, size_t len)
+{
+	char *end;
+	unsigned long chunk, end_chunk;
+
+	if (!mddev->bitmap)
+		goto out;
+	/* buf should be <chunk> <chunk> ... or <chunk>-<chunk> ... (range) */
+	while (*buf) {
+		chunk = end_chunk = simple_strtoul(buf, &end, 0);
+		if (buf == end) break;
+		if (*end == '-') { /* range */
+			buf = end + 1;
+			end_chunk = simple_strtoul(buf, &end, 0);
+			if (buf == end) break;
+		}
+		if (*end && !isspace(*end)) break;
+		bitmap_dirty_bits(mddev->bitmap, chunk, end_chunk);
+		buf = end;
+		while (isspace(*buf)) buf++;
+	}
+	bitmap_unplug(mddev->bitmap); /* flush the bits to disk */
+out:
+	return len;
+}
+
+static struct md_sysfs_entry md_bitmap =
+__ATTR(bitmap_set_bits, S_IWUSR, null_show, bitmap_store);
+
+static ssize_t
 size_show(mddev_t *mddev, char *page)
 {
 	return sprintf(page, "%llu\n", (unsigned long long)mddev->size);

@@ -2843,6 +2873,7 @@ static struct attribute *md_redundancy_a
 	&md_sync_completed.attr,
 	&md_suspend_lo.attr,
 	&md_suspend_hi.attr,
+	&md_bitmap.attr,
 	NULL,
 };
 static struct attribute_group md_redundancy_group = {
include/linux/raid/bitmap.h +2 -0
diff .prev/include/linux/raid/bitmap.h ./include/linux/raid/bitmap.h
@@ -265,6 +265,8 @@ int bitmap_update_sb(struct bitmap *bitm
 int  bitmap_setallbits(struct bitmap *bitmap);
 void bitmap_write_all(struct bitmap *bitmap);
 
+void bitmap_dirty_bits(struct bitmap *bitmap, unsigned long s, unsigned long e);
+
 /* these are exported */
 int bitmap_startwrite(struct bitmap *bitmap, sector_t offset,
 			unsigned long sectors, int behind);
author
#4 lore raw

PATCH 1/1 md: Remove unnecessary variable x in stripe_to_pdidx().

In reply to NeilBrown

From : Coywolf Qi Hunt <[email protected]>

Signed-off-by Coywolf Qi Hunt <[email protected]>
Signed-off-by Neil Brown <[email protected]>

### Diffstat output

 ./drivers/md/raid5.c |    5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/md/raid5.c +2 -3
diff .prev/drivers/md/raid5.c ./drivers/md/raid5.c
@@ -1350,10 +1350,9 @@ static int page_is_zero(struct page *p)
 static int stripe_to_pdidx(sector_t stripe, raid5_conf_t *conf, int disks)
 {
 	int sectors_per_chunk = conf->chunk_size >> 9;
-	sector_t x = stripe;
 	int pd_idx, dd_idx;
-	int chunk_offset = sector_div(x, sectors_per_chunk);
-	stripe = x;
+	int chunk_offset = sector_div(stripe, sectors_per_chunk);
+
 	raid5_compute_sector(stripe*(disks-1)*sectors_per_chunk
 			     + chunk_offset, disks, disks-1, &dd_idx, &pd_idx, conf);
 	return pd_idx;
#5 lore raw
In reply to NeilBrown
Quoted 13 lines · On Thu, 24 Aug 2006 17:40:56 +1000 NeilBrown <[email protected]> wrote:

Following are 4 patches against 2.6.18-rc4-mm2

The first 2 are bug fixes which should go in 2.6.18, and apply equally well to that tree as to -mm.

The latter two should stay in -mm until after 2.6.18.

The second patch is maybe bigger than it absolutely needs to be as a bugfix. If you like I can stripe out all the rcu-extra-carefulness as a separate patch and just leave the important bit which involves moving the atomic_add down twenty-something lines.

Quoted 7 lines

Thanks,
NeilBrown

[PATCH 001 of 4] md: Fix recent breakage of md/raid1 array checking [PATCH 002 of 4] md: Fix issues with referencing rdev in md/raid1. [PATCH 003 of 4] md: new sysfs interface for setting bits in the write-intent-bitmap [PATCH 004 of 4] md: Remove unnecessary variable x in stripe_to_pdidx().

The second patch is against -mm and doesn't come within a mile of applying to mainline.

author
#6 lore raw
Quoted 10 lines · On Thursday August 24, [email protected] wrote:
Quoted 5 lines · On Thu, 24 Aug 2006 17:40:56 +1000 NeilBrown <[email protected]> wrote:

[PATCH 001 of 4] md: Fix recent breakage of md/raid1 array checking [PATCH 002 of 4] md: Fix issues with referencing rdev in md/raid1. [PATCH 003 of 4] md: new sysfs interface for setting bits in the write-intent-bitmap [PATCH 004 of 4] md: Remove unnecessary variable x in stripe_to_pdidx().

The second patch is against -mm and doesn't come within a mile of applying to mainline.

Bother ...
I'll get you a really-truly good patch after the weekend :-(

NeilBrown