BUGS: internal bitmap during array create

After realizing my stupid error in specifying the bitmap during array creation, I've triggered a couple of 100% repeatable bugs with this scenario.

BUG 1)

When I create an array without a bitmap and add it after the array is synced, all works fine with any filesystem. If I create WITH the internal bitmap and use xfs, it chokes at mount time with:

mount: wrong fs type, bad option, bad superblock on /dev/md0, or too many mounted file systems

xfs_check also dies with:

[root@gtmp01 GTMP]# xfs_check /dev/md0
xfs_check: unexpected XFS SB magic number 0x00000000
xfs_check: read failed: Invalid argument
xfs_check: data size check failed /usr/sbin/xfs_check: line 28: 30580 Segmentation fault
xfs_db$DBOPTS -i -p xfs_check -c "check$OPTS" $1

Strangely, whatever the underlying cause is, ext3 seems immune (at least in brief testing) to this. I can create and mount an ext3 filesystem on top of the array that xfs dies trying to mount.

In the case where the array is created with bitmap at build time, if I wait until resync is completed, do a 'mdadm -Gb none' followed by 'mdadm -Gb internal', I can then safely create the XFS filesystem and mount it.

BUG 2)

Another bitmap failure during create time: MDADM dies with an error after creating the array, when it tries to assemble it, with an external-file bitmap (on ext3):

     [root@gtmp01 GTMP]# mdadm -C /dev/md0 -f --chunk=512 --level=10

-n14 -po2 -e1.2 -bESC[1P^M[root@gtmp01 GTMP]# mdadm -C /dev/md0 -f --chunk=512 --level=10 -n14 -po2 -e1.2 -b/var/tmp/bitmap /dev/mapper/mpath*
mdadm: RUN_ARRAY failed: Cannot allocate memory
mdadm: stopped /dev/md0

The array can be manually assembled, but it does not load with the bitmap, even when specifying it with 'mdadm -A /dev/md0 -b/var/tmp/bitmap'.

For reference, I'm running:

mdadm - v2.5.3 - 7 August 2006
mkfs.xfs version 2.8.11
kernel 2.6.18 (Opteron, x86_64, SMP)

I've attached typescript of the sessions where I run through all of these scenarios, as well as an strace of the "mdadm -C -b/var/tmp/bitmap" where it fails to assemble the array. Also is a file with the superblock detail on all the member devices.

Again, more than happy to help test patches and any scenarios.

Cheers,

/eli

After realizing my stupid error in specifying the bitmap during array creation, I've triggered a couple of 100% repeatable bugs with this scenario.

BUG 1)

Strangely, whatever the underlying cause is, ext3 seems immune (at least in brief testing) to this. I can create and mount an ext3 filesystem on top of the array that xfs dies trying to mount.

In the case where the array is created with bitmap at build time, if I wait until resync is completed, do a 'mdadm -Gb none' followed by 'mdadm -Gb internal', I can then safely create the XFS filesystem and mount it.

BUG 2)

Another bitmap failure during create time: MDADM dies with an error after creating the array, when it tries to assemble it, with an external-file bitmap (on ext3):

     [root@gtmp01 GTMP]# mdadm -C /dev/md0 -f --chunk=512 --level=10 

-n14 -po2 -e1.2 -bESC[1P^M[root@gtmp01 GTMP]# mdadm -C /dev/md0 -f --chunk=512 --level=10 -n14 -po2 -e1.2 -b/var/tmp/bitmap /dev/mapper/mpath*
mdadm: RUN_ARRAY failed: Cannot allocate memory
mdadm: stopped /dev/md0

Quoted 7 lines · On Wednesday October 11, [email protected] wrote:

After realizing my stupid error in specifying the bitmap during array creation, I've triggered a couple of 100% repeatable bugs with this scenario.

BUG 1)

....

Quoted 10 lines

Strangely, whatever the underlying cause is, ext3 seems immune (at least in brief testing) to this. I can create and mount an ext3 filesystem on top of the array that xfs dies trying to mount.

In the case where the array is created with bitmap at build time, if I wait until resync is completed, do a 'mdadm -Gb none' followed by 'mdadm -Gb internal', I can then safely create the XFS filesystem and mount it.

Can you get me the output of
mdadm -X some-component-device
both after the creation with a bitmap, and after the bitmap has been hot-removed and hot-added.
Just for good measure, include the "mdadm -E" output at the same times.

Quoted 11 lines

BUG 2)

Another bitmap failure during create time: MDADM dies with an error after creating the array, when it tries to assemble it, with an external-file bitmap (on ext3):

     [root@gtmp01 GTMP]# mdadm -C /dev/md0 -f --chunk=512 --level=10

-n14 -po2 -e1.2 -bESC[1P^M[root@gtmp01 GTMP]# mdadm -C /dev/md0 -f --chunk=512 --level=10 -n14 -po2 -e1.2 -b/var/tmp/bitmap

/dev/mapper/mpath*

Quoted 2 lines

mdadm: RUN_ARRAY failed: Cannot allocate memory
mdadm: stopped /dev/md0

I thought I had fixed this in 2.5, but on reflection that might not fixed it for 64bit hosts. Can you try explicitly setting the --bitmap-chunk size such that there will be fewer than 1,000,000 chunks?

NeilBrown
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

I've provided the requested info, attached as two files (typescript output):

Quoted 4 lines · On Wednesday October 18, [email protected] wrote:

I've provided the requested info, attached as two files (typescript output):

Thanks for persisting with this.

There is one bug in mdadm that is causing all of these problems. It only affect the 'offset' layout with raid10.

The fix is
http://neil.brown.name/git?p=mdadm;a=commitdiff;h=702b557b1c9

and is included below.
You might like to grab the latest source from
git://neil.brown.name/mdadm
and compile that, or just apply the patch.

Thanks again,
NeilBrown

-------------------------
Fix bugs related to raid10 and the new offset layout.

Need to mask of bits above the bottom 16 when calculating number of copies.

### Diffstat output

 ./ChangeLog |    1 +
 ./Create.c  |    2 +-
 ./util.c    |    2 +-
 3 files changed, 3 insertions(+), 2 deletions(-)

diff .prev/ChangeLog ./ChangeLog

ChangeLog 2006-10-19 16:38:24.000000000 +1000 renamed from ChangeLog 2006-10-19 16:38:07.000000000 +1000 +1 -0

@@ -13,6 +13,7 @@ Changes Prior to this release
        initramfs, but device doesn't yet exist in /dev.
     -   When --assemble --scan is run, if all arrays that could be found
        have already been started, don't report an error.
+    -   Fix a couple of bugs related to raid10 and the new 'offset' layout.
 
 Changes Prior to 2.5.4 release
     -   When creating devices in /dev/md/ create matching symlinks

diff .prev/Create.c ./Create.c

Create.c 2006-10-19 16:38:24.000000000 +1000 renamed from Create.c 2006-10-19 16:38:07.000000000 +1000 +1 -1

@@ -363,7 +363,7 @@ int Create(struct supertype *st, char *m
                 * which is array.size * raid_disks / ncopies;
                 * .. but convert to sectors.
                 */
-               int ncopies = (layout>>8) * (layout & 255);
+               int ncopies = ((layout>>8) & 255) * (layout & 255);
                bitmapsize = (unsigned long long)size * raiddisks / 
 ncopies * 2;
 /*             printf("bms=%llu as=%d rd=%d nc=%d\n", bitmapsize, size, 

raiddisks, ncopies);*/

        } else

diff .prev/util.c ./util.c

util.c 2006-10-19 16:38:24.000000000 +1000 renamed from util.c 2006-10-19 16:38:07.000000000 +1000 +1 -1

@@ -179,7 +179,7 @@ int enough(int level, int raid_disks, in
                /* This is the tricky one - we need to check
                 * which actual disks are present.
                 */
-               copies = (layout&255)* (layout>>8);
+               copies = (layout&255)* ((layout>>8) & 255);
                first=0;
                do {
                        /* there must be one of the 'copies' form 'first' */

Neil, thanks a ton. This issue appears resolved completely, array creation/assembly is clean, filesystem creation and consistency checking is clean, and I have not generated any filesystem or md errors in testing yet.