Regression- XFS won't mount on partitioned md array

I just attempted a kernel upgrade from 2.6.20.7 to 2.6.25.3 and it no longer mounts my xfs filesystem.

I bisected it to around a67d7c5f5d25d0b13a4dfb182697135b014fa478
[XFS] Move platform specific mount option parse out of core XFS code

I have a RAID5 array with partitions:

Partition Table for /dev/md_d0

First Last
# Type Sector Sector Offset Length Filesystem Type (ID) Flag
-- ------- ----------- ----------- ------ ----------- -------------------- ----

 1 Primary           0  2500288279      4  2500288280 Linux (83)           None
 2 Primary  2500288280  2500483583      0      195304 Non-FS data (DA)     None

when I attempt to mount /media:
/dev/md_d0p1 /media xfs rw,nobarrier,noatime,logdev=/dev/md_d0p2,allocsize=512m 0 0

I get:
md_d0: p1 p2
XFS mounting filesystem md_d0p1
attempt to access beyond end of device
md_d0p2: rw=0, want=195311, limit=195304
I/O error in filesystem ("md_d0p1") meta-data dev md_d0p2 block 0x2fae7 ("xlog_bread") error 5 buf count 512
XFS: empty log check failed
XFS: log mount/recovery failed: error 5
XFS: log mount failed

A repair:
xfs_repair /dev/md_d0p1 -l /dev/md_d0p2
gives no errors.

Phase 1 - find and verify superblock...
Phase 2 - using external log on /dev/md_d0p2
- zero log...
- scan filesystem freespace and inode maps...
- found root inode chunk
...

David

Ouch, still on 2.6.25.1 here, didn't reboot yet, but I do not use mdraid'ed partitions, just regular mdraid, if you boot back to 2.6.20.7 does it work again?

Yes, no probs.

It came in prior to 2.6.25-rc1
The machine has a root xfs filesystem with an internal log on a sata disk and a data filesystem on a partitioned array with an external log (also on the partitioned array).
Only the partitioned array/external-log filesystem is affected.

David

this[1] patch fixed?

this patch is for 2.6.25.5 kernel

1: http://git.kernel.org/?p=linux/kernel/git/stable/stable-queue.git;a=blob;f=queue-2.6.25/block-do_mounts-accept-root-non-existant-partition.patch;h=097cda9928b434994dbb157065b2ca38e7cec3a1;hb=8cc4c3b370d59deb16c2e92165a466c82e914020

Thanks,
Oliver

http://git.kernel.org/?p=linux/kernel/git/stable/stable-queue.git;a=blob;f=queue-2.6.25/block-do_mounts-accept-root-non-existant-partition.patch;h=097cda9928b434994dbb157065b2ca38e7cec3a1;hb=8cc4c3b370d59deb16c2e92165a466c82e914020

Looks like a possible candidate - thanks.

I think this patch is for mounting root on an md device when the partitions aren't yet initialised.

However:
* When I run cfdisk I can read the partition table.
* Subsequent attempts to mount the xfs when /proc/partitions is clearly present still fail.

? there isn't a 2.6.25.5
It doesn't apply to 2.6.25.4

I'll see if I can make it apply...

David

Sorry, I understand; it's queued for 2.6.25.5.

Yep - it was download corruption.

Applied but it didn't help.

David

around that... not exactly? That commit should have been largely a code move, which is not to say that it can't contain a bug... :)

mythbox? :)

Hm, so it's the external log size that it doesn't much like...

what does /proc/partitions say about md_d0p1 and p2? Is it different between the older & newer kernel?

What does xfs_info /mount/point say about the filesystem when you mount it under the older kernel? Or... if you can't mount it,

xfs_db -r -c "sb 0" -c p /dev/md_d0p1

-Eric

I got to within 4 on the bisect and my xfs partition containing the kernel src and the bisect history blew up telling me that files were directories and then exploding in a heap of lost+found/ fragments. Quite, erm, "interesting" really.

At that point I decided I was close enough to ask for advice, looked at the commits and took this one as the most likely to cause the bug :)

But, thinking about it, I can decode the kernel extraversion tags in /boot. From that I think my bounds were:
40ebd81d1a7635cf92a59c387a599fce4863206b
[XFS] Use kernel-supplied "roundup_pow_of_two" for simplicity
and:
3ed6526441053d79b85d206b14d75125e6f51cc2
[XFS] Implement fallocate.

so those bound:
[XFS] Remove the BPCSHIFT and NB* based macros from XFS. [XFS] Remove bogus assert
[XFS] optimize XFS_IS_REALTIME_INODE w/o realtime config [XFS] Move platform specific mount option parse out of core XFS code

and just glancing through the patches I didn't see any changes that looked likely in the others...

Hey - we test some interesting corner cases... :)
My *wife* just told *me* to buy, and I quote "No more than 10" 1Tb Samsung drives... I decided 5 would be plenty.

Yep - I noticed that - and ISTR that Neil has been fiddling in the md partitioning code over the last 6 months or so.
I wondered where it got the larger figure from and if, somehow, md was changing the partition size somehow...

2.6.20.7 (good)

 254     0 1250241792 md_d0
 254     1 1250144138 md_d0p1
 254     2      97652 md_d0p2

2.6.25.3 (bad)

 254     0 1250241792 md_d0
 254     1 1250144138 md_d0p1
 254     2      97652 md_d0p2

2.6.25.4 (bad)

 254     0 1250241792 md_d0
 254     1 1250144138 md_d0p1
 254     2      97652 md_d0p2

So nothing obvious there then...

teak:~# xfs_info /media/
meta-data=/dev/md_d0p1           isize=256    agcount=32, agsize=9766751 blks
         =                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=312536032, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096
log      =external               bsize=4096   blocks=24413, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=65536  blocks=0, rtextents=0

...

ok, and with:

So, xfs thinks the external log is 24413 4k blocks (from the sb geometry printed by xfs_info). This is 97652 1k units (matching your /proc/partitions output) and 195304 512-byte sectors (matching the partition table output). So that all looks consistent.

So if xfs is doing:

it surely does seem to be trying to read past the end of what even it thinks is the end of its log.

And, with your geometry I can reproduce this w/o md, partitioned or not.
So looks like xfs itself is busted:

loop5: rw=0, want=195311, limit=195304

I'll see if I have a little time today to track down the problem.

Thanks,

-Eric

I'm trying to figure out a mechanism to safely repair a stripe of data when I know a particular disk has a unrecoverable read error at a certain physical block (for 2.6 kernels)

My original plan was to figure out the range of blocks in md device that utilizes the known bad block and force a raw read on physical device that covers the entire chunk and let the md driver do all of the work.

Well, this didn't pan out. Problems include issues where if bad block maps to the parity block in a stripe then md won't necessarily read/verify parity, and in cases where you are running RAID1, then load balancing might result in the kernel reading the bad block from the good disk.

So the degree of difficulty is much higher than I expected. I prefer not to patch kernels due to maintenance issues as well as desire for the technique to work across numerous kernels and patch revisions, and frankly, the odds are I would screw it up. An application-level program that can be invoked as necessary would be ideal.

As such, anybody up to the challenge of writing the code? I want it enough to paypal somebody $500 who can write it, and will gladly open source the solution.

(And to clarify why, I know physical block x on disk y is bad before the O/S reads the block, and just want to rebuild the stripe, not the entire md device when this happens. I must not compromise any file system data, cached or non-cached that is built on the md device. I have system with

somewhere, then a full parity repair would never complete before another physical bad block is discovered.)

Contact me offline for the financial details, but I would certainly appreciate some thread discussion on an appropriate architecture. At least it is my opinion that such capability should eventually be native Linux, but as long as there is a program that can be run on demand that doesn't require rebuilding or patching kernels then that is all I need.

David @ santools.com

Damn, here goes $500 :) Unfortunately the only thing I can bring to the table is a thread[1] about a mechanism that would fit your request nicely. Hopefully someone will pick this stuff up and make it a reality.

Peter

[1] http://marc.info/?l=linux-raid&m=120605458309825

} -----Original Message-----
} From: [email protected] [mailto:linux-raid-
} [email protected]] On Behalf Of David Lethe
} Sent: Saturday, May 17, 2008 3:10 PM
} To: LinuxRaid; [email protected]
} Subject: Mechanism to safely force repair of single md stripe w/o hurting } data integrity of file system
}
} I'm trying to figure out a mechanism to safely repair a stripe of data } when I know a particular disk has a unrecoverable read error at a } certain physical block (for 2.6 kernels)
}
} My original plan was to figure out the range of blocks in md device that } utilizes the known bad block and force a raw read on physical device } that covers the entire chunk and let the md driver do all of the work. }
} Well, this didn't pan out. Problems include issues where if bad block } maps to the parity block in a stripe then md won't necessarily } read/verify parity, and in cases where you are running RAID1, then load } balancing might result in the kernel reading the bad block from the good } disk.
}
} So the degree of difficulty is much higher than I expected. I prefer } not to patch kernels due to maintenance issues as well as desire for the } technique to work across numerous kernels and patch revisions, and } frankly, the odds are I would screw it up. An application-level program } that can be invoked as necessary would be ideal.
}
} As such, anybody up to the challenge of writing the code? I want it } enough to paypal somebody $500 who can write it, and will gladly open } source the solution.
}
} (And to clarify why, I know physical block x on disk y is bad before the } O/S reads the block, and just want to rebuild the stripe, not the entire } md device when this happens. I must not compromise any file system data, } cached or non-cached that is built on the md device. I have system with } >100TB and if I did a rebuild every time I discovered a bad block } somewhere, then a full parity repair would never complete before another } physical bad block is discovered.)
}
} Contact me offline for the financial details, but I would certainly } appreciate some thread discussion on an appropriate architecture. At } least it is my opinion that such capability should eventually be native } Linux, but as long as there is a program that can be run on demand that } doesn't require rebuilding or patching kernels then that is all I need. }
} David @ santools.com

I thought this would cause md to read all blocks in an array:
echo repair > /sys/block/md0/md/sync_action

And rewrite any blocks that can't be read.

In the old days, md would kick out a disk on a read error. When you added it back, md would rewrite everything on that disk, which corrected read errors.

Guy

With a read bitmap (mdadm -G /dev/mdX -b internal, or during -C), it should resync less after an unwarranted kick.

This shouldn't be a problem.
You write a patch, submit it for review, it gets reviewed and eventually submitted to mainline.
Then it will work on all new kernels, and any screw ups that you make will be caught by someone else (me possibly).

It is largely done.
If you write a number to /sys/block/mdXX/md/sync_max, then recovery will stop when it gets there.
If you write 'check' to /sys/block/mdXX/md/sync_action, then it will read all blocks and auto-correct any unrecoverable read errors.

You just need some way to set the start point of the resync. Probably just create a sync_min attribute - see lightly tested patch below.

If this fits your needs, I'm sure www.compassion.com would be happy with your $500.

To use this:

1/ Write the end address (sectors) to sync_max 2/ Write the start address (sectors) to sync_min 3/ Write 'check' to sync_action
4/ Monitor sync_completed until it reaches sync_max 5/ Write 'idle' to sync_action

NeilBrown

### Diffstat output

 ./drivers/md/md.c           |   46 +++++++++++++++++++++++++++++++++++++++++---
 ./include/linux/raid/md_k.h |    2 +
 2 files changed, 45 insertions(+), 3 deletions(-)

diff .prev/drivers/md/md.c ./drivers/md/md.c
@@ -277,6 +277,7 @@ static mddev_t * mddev_find(dev_t unit)
 	spin_lock_init(&new->write_lock);
 	init_waitqueue_head(&new->sb_wait);
 	new->reshape_position = MaxSector;
+	new->resync_min = 0;
 	new->resync_max = MaxSector;
 	new->level = LEVEL_NONE;
 

@@ -3074,6 +3075,37 @@ sync_completed_show(mddev_t *mddev, char
 static struct md_sysfs_entry md_sync_completed = __ATTR_RO(sync_completed);
 
 static ssize_t
+min_sync_show(mddev_t *mddev, char *page)
+{
+	return sprintf(page, "%llu\n",
+		       (unsigned long long)mddev->resync_min);
+}
+static ssize_t
+min_sync_store(mddev_t *mddev, const char *buf, size_t len)
+{
+	char *ep;
+	unsigned long long min = simple_strtoull(buf, &ep, 10);
+	if (ep == buf || (*ep != 0 && *ep != '\n'))
+		return -EINVAL;
+	if (min > mddev->resync_max)
+		return -EINVAL;
+	if (test_bit(MD_RECOVERY_RUNNING, &mddev->recovery))
+		return -EBUSY;
+
+	/* Must be a multiple of chunk_size */
+	if (mddev->chunk_size) {
+		if (min & (sector_t)((mddev->chunk_size>>9)-1))
+			return -EINVAL;
+	}
+	mddev->resync_min = min;
+
+	return len;
+}
+
+static struct md_sysfs_entry md_min_sync =
+__ATTR(sync_min, S_IRUGO|S_IWUSR, min_sync_show, min_sync_store);
+
+static ssize_t
 max_sync_show(mddev_t *mddev, char *page)
 {
 	if (mddev->resync_max == MaxSector)

@@ -3092,6 +3124,9 @@ max_sync_store(mddev_t *mddev, const cha
 		unsigned long long max = simple_strtoull(buf, &ep, 10);
 		if (ep == buf || (*ep != 0 && *ep != '\n'))
 			return -EINVAL;
+		if (max < mddev->resync_min)
+			return -EINVAL;
+
 		if (max < mddev->resync_max &&
 		    test_bit(MD_RECOVERY_RUNNING, &mddev->recovery))
 			return -EBUSY;

@@ -3103,7 +3138,8 @@ max_sync_store(mddev_t *mddev, const cha
 		}
 		mddev->resync_max = max;
 	}
-	wake_up(&mddev->recovery_wait);
+	if (test_bit(MD_RECOVERY_RUNNING, &mddev->recovery))
+		wake_up(&mddev->recovery_wait);
 	return len;
 }
 

@@ -3221,6 +3257,7 @@ static struct attribute *md_redundancy_a
 	&md_sync_speed.attr,
 	&md_sync_force_parallel.attr,
 	&md_sync_completed.attr,
+	&md_min_sync.attr,
 	&md_max_sync.attr,
 	&md_suspend_lo.attr,
 	&md_suspend_hi.attr,

@@ -3776,6 +3813,7 @@ static int do_md_stop(mddev_t * mddev, i
 		mddev->size = 0;
 		mddev->raid_disks = 0;
 		mddev->recovery_cp = 0;
+		mddev->resync_min = 0;
 		mddev->resync_max = MaxSector;
 		mddev->reshape_position = MaxSector;
 		mddev->external = 0;

@@ -5622,9 +5660,11 @@ void md_do_sync(mddev_t *mddev)
 		max_sectors = mddev->resync_max_sectors;
 		mddev->resync_mismatches = 0;
 		/* we don't use the checkpoint if there's a bitmap */
-		if (!mddev->bitmap &&
-		    !test_bit(MD_RECOVERY_REQUESTED, &mddev->recovery))
+		if (test_bit(MD_RECOVERY_REQUESTED, &mddev->recovery))
+			j = mddev->resync_min;
+		else if (!mddev->bitmap)
 			j = mddev->recovery_cp;
+
 	} else if (test_bit(MD_RECOVERY_RESHAPE, &mddev->recovery))
 		max_sectors = mddev->size << 1;
 	else {

diff .prev/include/linux/raid/md_k.h ./include/linux/raid/md_k.h
@@ -227,6 +227,8 @@ struct mddev_s
 	atomic_t			recovery_active; /* blocks scheduled, but not written */
 	wait_queue_head_t		recovery_wait;
 	sector_t			recovery_cp;
+	sector_t			resync_min;	/* user request sync starts
+							 * here */
 	sector_t			resync_max;	/* resync should pause
 							 * when it gets here */
 

Does this patch fix it for you? Does for me though I can't yet explain why ;)

http://www.linux.sgi.com/archives/xfs/2008-05/msg00190.html

-Eric

So what's happening is that xfs is trying to read a page-sized IO from the last sector of the log... which goes off the end of the device. This looks like another regression introduced by a9759f2de38a3443d5107bddde03b4f3f550060e, but fixed by Christoph's patch in the URL above, which should be headed towards -stable.

(aside: it seems that this breaks any external log setup where the log consists of the entire device... but I'd have expected the xfsqa suite to catch this...?)

The patch avoids the problem by looking for some extra locking but it seems to me that the root cause is that the buffer being read at this point doesn't have it's b_offset, the offset in it's page, set. Might be another little buglet but harmless it seems.

-Eric

Yes, this fixes it for me - thanks :)

Damn, I guess I misread my bisect readings when things crashed then. Still, I said 'around' :)

mmmm
'little buglets' in the filesystem holding a few Tb of data... mmmm
Anything I can do to help find that? I suspect not if you can reproduce it.

Anyhow - thanks again.

David
PS I'll be back soon, back in 2.6.23 I was hitting a hibernate/xfs bug which I've been avoiding by powering down. Well, it's still there in 2.6.25.3...

Nah. I'll ask the sgi guys about it, it just seems a little inconsistent but maybe by design...

-Eric

Hi Greg
Perusing:
http://git.kernel.org/?p=linux/kernel/git/stable/stable-queue.git
doesn't show the patch referenced below as in the queue for 2.6.25.5

David

Hi David,

First, please avoid top-posting.

It would have helped to CC stable (fixed) and to give the mainline commit ID since the stable branch only holds already merged patches. Greg, the commit is 6ab455ee...

Willy

Yup I'll agree that this should probably go to -stable unless hch or dchinner disagree.

FWIW I've already put it in the Fedora kernels.

-Eric

The only way I can see that we'd catch this (by testing external logs) in the current qa setup is
if one sets up SCRATCH_LOGDEV and/or TEST_LOGDEV and USE_EXTERNAL. There are no specific tests to test out external logs (there is 044 but it requires the env vars set anyway) such as using a loop back device for the log. Perhaps we should do this.
I should check that our QA group are setting the vars in some runs.

--Tim