Skip to contentSkip to conversation

Shrinking number of devices on a RAID-10 (near 2) array

PATCH Applied Andy Smith 3 participants 20 messages Aug 23, 2014 – Sep 3, 2014 MD RAID

lore
Share & tools
lore

Andy Smith

author
#0 lore raw

Hi,

I am aware that for a very long time it was not possible to change the number of devices in an mdadm RAID-10. But then recently I am sure I saw threads here alluding to this being possible. e.g.:

    http://marc.info/?l=linux-raid&m=140768923829685&w=2

I have a 6 device RAID-10 near=2 array that I would like to shrink down to 4 devices.

I have compiled mdadm 3.3.2 and am using kernel 3.16.0.

$ sudo mdadm --detail /dev/md2
/dev/md2:
        Version : 0.90
  Creation Time : Sun Jun  4 08:18:58 2006
     Raid Level : raid10
     Array Size : 471859200 (450.00 GiB 483.18 GB)
  Used Dev Size : 309363520 (295.03 GiB 316.79 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 2
    Persistence : Superblock is persistent

Update Time : Sat Aug 23 15:23:38 2014
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

Layout : near=2
Chunk Size : 64K

UUID : 3905b303:ca604b72:be5949c4:ab051b7a
Events : 0.312149991

Number Major Minor RaidDevice State
0 8 51 0 active sync /dev/sdd3
1 8 67 1 active sync /dev/sde3
2 8 83 2 active sync /dev/sdf3
3 8 19 3 active sync /dev/sdb3
4 8 35 4 active sync /dev/sdc3
5 8 3 5 active sync /dev/sda3

$ sudo ./mdadm --grow -n4 /dev/md2 --backup-file /var/tmp/mdadm.backup
mdadm: Cannot set new_data_offset: RAID10 reshape not
       supported on this kernel

Is that not yet possible, then?

(Each device is 320GB so it should all fit with only four of them)

Cheers,
Andy

#1 lore raw
Quoted 46 lines · On Sat, 23 Aug 2014 16:31:10 +0000 Andy Smith <[email protected]> wrote:

Hi,

I am aware that for a very long time it was not possible to change the number of devices in an mdadm RAID-10. But then recently I am sure I saw threads here alluding to this being possible. e.g.:

    http://marc.info/?l=linux-raid&m=140768923829685&w=2

I have a 6 device RAID-10 near=2 array that I would like to shrink down to 4 devices.

I have compiled mdadm 3.3.2 and am using kernel 3.16.0.

$ sudo mdadm --detail /dev/md2
/dev/md2:
        Version : 0.90
  Creation Time : Sun Jun  4 08:18:58 2006
     Raid Level : raid10
     Array Size : 471859200 (450.00 GiB 483.18 GB)
  Used Dev Size : 309363520 (295.03 GiB 316.79 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 2
    Persistence : Superblock is persistent

Update Time : Sat Aug 23 15:23:38 2014
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

Layout : near=2
Chunk Size : 64K

UUID : 3905b303:ca604b72:be5949c4:ab051b7a
Events : 0.312149991

Number Major Minor RaidDevice State
0 8 51 0 active sync /dev/sdd3
1 8 67 1 active sync /dev/sde3
2 8 83 2 active sync /dev/sdf3
3 8 19 3 active sync /dev/sdb3
4 8 35 4 active sync /dev/sdc3
5 8 3 5 active sync /dev/sda3

Quoted 7 lines
$ sudo ./mdadm --grow -n4 /dev/md2 --backup-file /var/tmp/mdadm.backup
mdadm: Cannot set new_data_offset: RAID10 reshape not
       supported on this kernel

Is that not yet possible, then?

(Each device is 320GB so it should all fit with only four of them)

raid10 reshape is only supported for v1.x metadata, and you have 0.90. This is because it needs to move the data_offset and 0.90 cannot do that.

I see you have already adjusted the --array-size correctly - good.

You can achieve your result by:

1/ stop the array and assemble with --update=metadata. This will convert the
array to v1.0
2/ "mdadm /dev/md2 --grow --disks=4" That should perform the reshape.
You don't need a --backup-file with RAID10 ... or with other arrays if you have recent tools and enough space before or after.

I guess the error message could be better....

NeilBrown

#2 lore raw

Neil,

I assume you would need to shrink the filesystem first before this ? or does MDADM nicely do that as part of the process also ?

Craig

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of NeilBrown
Sent: Sunday, 24 August 2014 1:10 PM
To: Andy Smith
Cc: [email protected]
Subject: Re: Shrinking number of devices on a RAID-10 (near 2) array

Quoted 46 lines · On Sat, 23 Aug 2014 16:31:10 +0000 Andy Smith <[email protected]> wrote:

Hi,

I am aware that for a very long time it was not possible to change the number of devices in an mdadm RAID-10. But then recently I am sure I saw threads here alluding to this being possible. e.g.:

    http://marc.info/?l=linux-raid&m=140768923829685&w=2

I have a 6 device RAID-10 near=2 array that I would like to shrink down to 4 devices.

I have compiled mdadm 3.3.2 and am using kernel 3.16.0.

$ sudo mdadm --detail /dev/md2
/dev/md2:
        Version : 0.90
  Creation Time : Sun Jun  4 08:18:58 2006
     Raid Level : raid10
     Array Size : 471859200 (450.00 GiB 483.18 GB)
  Used Dev Size : 309363520 (295.03 GiB 316.79 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 2
    Persistence : Superblock is persistent

Update Time : Sat Aug 23 15:23:38 2014
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

Layout : near=2
Chunk Size : 64K

UUID : 3905b303:ca604b72:be5949c4:ab051b7a
Events : 0.312149991

Number Major Minor RaidDevice State
0 8 51 0 active sync /dev/sdd3
1 8 67 1 active sync /dev/sde3
2 8 83 2 active sync /dev/sdf3
3 8 19 3 active sync /dev/sdb3
4 8 35 4 active sync /dev/sdc3
5 8 3 5 active sync /dev/sda3

Quoted 7 lines
$ sudo ./mdadm --grow -n4 /dev/md2 --backup-file /var/tmp/mdadm.backup
mdadm: Cannot set new_data_offset: RAID10 reshape not
       supported on this kernel

Is that not yet possible, then?

(Each device is 320GB so it should all fit with only four of them)

raid10 reshape is only supported for v1.x metadata, and you have 0.90. This is because it needs to move the data_offset and 0.90 cannot do that.

I see you have already adjusted the --array-size correctly - good.

You can achieve your result by:

1/ stop the array and assemble with --update=metadata. This will convert the
array to v1.0
2/ "mdadm /dev/md2 --grow --disks=4" That should perform the reshape.
You don't need a --backup-file with RAID10 ... or with other arrays if you have recent tools and enough space before or after.

I guess the error message could be better....

NeilBrown

Disclaimer

CONFIDENTIAL

This message contains confidential information and is intended only for the intended recipients. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.

#3 lore raw
Quoted 3 lines · On Sun, 24 Aug 2014 06:28:25 +0000 Craig Curtin <[email protected]> wrote:

Neil,

I assume you would need to shrink the filesystem first before this ? or does MDADM nicely do that as part of the process also ?

mdadm is not a jack-of-all-trades. It won't touch your filesystem.

A careful look at Andy's "mdadm --detail" shows:

Quoted 2 lines
Quoted 2 lines
     Array Size : 471859200 (450.00 GiB 483.18 GB)

Used Dev Size : 309363520 (295.03 GiB 316.79 GB)

This is a near=2 6 drive RAID10, so Array Size should be 3 times Used Dev Size.
The fact that it isn't shows that he has already explicitly set the
--array-size to something appropriate. I'm sure he reduced the size of the
filesystem first as the man page says:

Before reducing the size of the array you should make sure that space isn't needed. If the device holds a filesystem, you would need to resize the filesystem to use less space.

If you try to resize an array to fewer drives and you haven't adjusted the --array-size already, mdadm will tell you what --array-size you need. This forces you into a two (at least) step process to minimize the chance of doing something silly.
The idea is that you "--grow --array-size" first, then use "fsck" to make sure the filesystem is still happy. Then use "--grow --raid-disks" to reshape the array without changing the size.
If you change the array size and get it wrong, you can always put it back. But once you start the reshape happening, you will have lost any data beyond "array-size".

NeilBrown

Quoted 88 lines

Craig

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of NeilBrown
Sent: Sunday, 24 August 2014 1:10 PM
To: Andy Smith
Cc: [email protected]
Subject: Re: Shrinking number of devices on a RAID-10 (near 2) array

Quoted 46 lines · On Sat, 23 Aug 2014 16:31:10 +0000 Andy Smith <[email protected]> wrote:

Hi,

I am aware that for a very long time it was not possible to change the number of devices in an mdadm RAID-10. But then recently I am sure I saw threads here alluding to this being possible. e.g.:

    http://marc.info/?l=linux-raid&m=140768923829685&w=2

I have a 6 device RAID-10 near=2 array that I would like to shrink down to 4 devices.

I have compiled mdadm 3.3.2 and am using kernel 3.16.0.

$ sudo mdadm --detail /dev/md2
/dev/md2:
        Version : 0.90
  Creation Time : Sun Jun  4 08:18:58 2006
     Raid Level : raid10
     Array Size : 471859200 (450.00 GiB 483.18 GB)
  Used Dev Size : 309363520 (295.03 GiB 316.79 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 2
    Persistence : Superblock is persistent

Update Time : Sat Aug 23 15:23:38 2014
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

Layout : near=2
Chunk Size : 64K

UUID : 3905b303:ca604b72:be5949c4:ab051b7a
Events : 0.312149991

Number Major Minor RaidDevice State
0 8 51 0 active sync /dev/sdd3
1 8 67 1 active sync /dev/sde3
2 8 83 2 active sync /dev/sdf3
3 8 19 3 active sync /dev/sdb3
4 8 35 4 active sync /dev/sdc3
5 8 3 5 active sync /dev/sda3

Quoted 7 lines
$ sudo ./mdadm --grow -n4 /dev/md2 --backup-file /var/tmp/mdadm.backup
mdadm: Cannot set new_data_offset: RAID10 reshape not
       supported on this kernel

Is that not yet possible, then?

(Each device is 320GB so it should all fit with only four of them)

raid10 reshape is only supported for v1.x metadata, and you have 0.90. This is because it needs to move the data_offset and 0.90 cannot do that.

I see you have already adjusted the --array-size correctly - good.

You can achieve your result by:

1/ stop the array and assemble with --update=metadata. This will convert the
array to v1.0
2/ "mdadm /dev/md2 --grow --disks=4" That should perform the reshape.
You don't need a --backup-file with RAID10 ... or with other arrays if you have recent tools and enough space before or after.

I guess the error message could be better....

NeilBrown

Disclaimer

CONFIDENTIAL

This message contains confidential information and is intended only for the intended recipients. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.

author
#4 lore raw
Quoted 8 lines · NeilBrown
Quoted 2 lines
Quoted 2 lines
     Array Size : 471859200 (450.00 GiB 483.18 GB)

Used Dev Size : 309363520 (295.03 GiB 316.79 GB)

This is a near=2 6 drive RAID10, so Array Size should be 3 times Used Dev Size.
The fact that it isn't shows that he has already explicitly set the
--array-size to something appropriate. I'm sure he reduced the size of the
filesystem first as the man page says:

Yup, it's an LVM PV so I did pvresize and mdadm --array-size first to some safe lower value, before I realised that mdadm would have told me the exact --array-size I needed anyway.

I should have mentioned that to reduce confusion, but it didn't seem pertinent to where I got stuck.

Cheers,
Andy

author
#5 lore raw
In reply to NeilBrown

Hi Neil,

Thanks for the quick response!

Quoted 9 lines · NeilBrown

raid10 reshape is only supported for v1.x metadata, and you have 0.90. This is because it needs to move the data_offset and 0.90 cannot do that.

I see you have already adjusted the --array-size correctly - good.

You can achieve your result by:

1/ stop the array and assemble with --update=metadata. This will convert the
array to v1.0

I've now done this, so:

$ sudo mdadm --detail /dev/md2
/dev/md2:
        Version : 1.0
  Creation Time : Sun Jun  4 08:18:59 2006
     Raid Level : raid10
     Array Size : 618726528 (590.06 GiB 633.58 GB)
  Used Dev Size : 309363264 (295.03 GiB 316.79 GB)
   Raid Devices : 6
  Total Devices : 6
    Persistence : Superblock is persistent

Update Time : Sun Aug 24 14:29:15 2014
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

Layout : near=2
Chunk Size : 64K

Name : 2
UUID : 3905b303:ca604b72:be5949c4:ab051b7a
Events : 1

Number Major Minor RaidDevice State
0 8 51 0 active sync /dev/sdd3
1 8 67 1 active sync /dev/sde3
2 8 83 2 active sync /dev/sdf3
3 8 19 3 active sync /dev/sdb3
4 8 35 4 active sync /dev/sdc3
5 8 3 5 active sync /dev/sda3

I noticed something: The assemble with update re-set the array size to 928089792 (885.10 GiB 950.36 GB).

(the above --detail is after all steps below)

Quoted 1 line

2/ "mdadm /dev/md2 --grow --disks=4" That should perform the reshape.

I assume that should have been --raid-devices; --disks seems to be a syntax error. :)

Firstly it told me the array was too big, as expected.

$ sudo ./mdadm /dev/md2 --grow --raid-devices=4
mdadm: this change will reduce the size of the array.
       use --grow --array-size first to truncate array.
       e.g. mdadm --grow /dev/md2 --array-size 618726528
unfreeze

Okay.

$ sudo ./mdadm --grow /dev/md2 --array-size 618726528

But:

$ sudo ./mdadm /dev/md2 --grow --raid-devices=4
mdadm: Cannot set new_data_offset: RAID10 reshape not
       supported on this kernel
Quoted 1 line

I guess the error message could be better....

Will be happy to send the trivial patch once we get to the bottom of this!

Cheers,
Andy

author
#6 lore raw
Quoted 3 lines · Andy Smith
$ sudo ./mdadm /dev/md2 --grow --raid-devices=4
mdadm: Cannot set new_data_offset: RAID10 reshape not
       supported on this kernel

Grow.c from latest git:

2397 err = sysfs_set_num(sra, sd, "new_offset",
2398 new_data_offset);

For the first device in the for loop (sda3), the above line 2397 is returning -1.

2399 }
2400 if (err < 0) {
2401 if (errno == E2BIG && data_offset != INVALID_SECTORS) {
2402 pr_err("data-offset is too big for %s\n",
2403 dn);
2404 goto release;
2405 }
2406 if (sd == sra->devs &&
2407 (errno == ENOENT || errno == E2BIG))
2408 /* Early kernel, no 'new_offset' file,

2409                                  * or kernel doesn't like us.
2410                                  * For RAID5/6 this is not fatal
2411                                  */

2412 return 1;

It reaches here line 2412 with errno == E2BIG.
/sys/block/md2/md/dev-sda3/new_offset exists.

From strace:

open("/sys/block/md2/md/dev-sda3/new_offset", O_WRONLY) = 4
write(4, "128", 3) = -1 E2BIG (Argument list too long)
close(4) = 0

Not sure where to debug next.

Cheers,
Andy

Quoted 1 line

The optimum programming team size is 1.

Has Jurassic Park taught us nothing?
— pfilandr
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#7 lore raw
Quoted 31 lines · On Mon, 25 Aug 2014 10:32:34 +0000 Andy Smith <[email protected]> wrote:
Quoted 3 lines · Andy Smith
$ sudo ./mdadm /dev/md2 --grow --raid-devices=4
mdadm: Cannot set new_data_offset: RAID10 reshape not
       supported on this kernel

Grow.c from latest git:

2397 err = sysfs_set_num(sra, sd, "new_offset",
2398 new_data_offset);

For the first device in the for loop (sda3), the above line 2397 is returning -1.

2399 }
2400 if (err < 0) {
2401 if (errno == E2BIG && data_offset != INVALID_SECTORS) {
2402 pr_err("data-offset is too big for %s\n",
2403 dn);
2404 goto release;
2405 }
2406 if (sd == sra->devs &&
2407 (errno == ENOENT || errno == E2BIG))
2408 /* Early kernel, no 'new_offset' file,

2409                                  * or kernel doesn't like us.
2410                                  * For RAID5/6 this is not fatal
2411                                  */

2412 return 1;

It reaches here line 2412 with errno == E2BIG.
/sys/block/md2/md/dev-sda3/new_offset exists.

Quoted 7 lines
Quoted 1 line

From strace:

open("/sys/block/md2/md/dev-sda3/new_offset", O_WRONLY) = 4
write(4, "128", 3) = -1 E2BIG (Argument list too long)
close(4) = 0

Not sure where to debug next.

That comes from

	else if (new_offset > rdev->data_offset) {
		/* must not push array size beyond rdev_sectors */
		if (new_offset - rdev->data_offset
		    + mddev->dev_sectors > rdev->sectors)
				return -E2BIG;

in drivers/md/md.c in the kernel...

What does "mdadm --examine" on one of the devices report now that it is v1.0?

Particularly interested in the "Unused Space :" line.

NeilBrown

author
#8 lore raw

Hi Neil,

Quoted 3 lines · NeilBrown

What does "mdadm --examine" on one of the devices report now that it is v1.0?

Particularly interested in the "Unused Space :" line.

/dev/sda3:

          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x0
     Array UUID : 3905b303:ca604b72:be5949c4:ab051b7a
           Name : 2

Creation Time : Sun Jun 4 08:18:59 2006
Raid Level : raid10
Raid Devices : 6

Avail Dev Size : 618726528 (295.03 GiB 316.79 GB)

     Array Size : 928089792 (885.10 GiB 950.36 GB)

Super Offset : 618727392 sectors
Unused Space : before=0 sectors, after=864 sectors
State : active
Device UUID : e30176be:81a57e84:1f2aa206:9515150f

Update Time : Sun Aug 24 14:29:15 2014
Checksum : bd599c72 - correct
Events : 1

Layout : near=2
Chunk Size : 64K

Device Role : Active device 5
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)

All other devices report same (apart from UUID/checksum/role).

Cheers,
Andy

author
#9 lore raw

Hi Neil,

Any ideas on the below? Not in a huge rush but at some point I might have to just re-create the array and copy data back.

Cheers,
Andy

Quoted 35 lines · Andy Smith

Hi Neil,

Quoted 3 lines · NeilBrown

What does "mdadm --examine" on one of the devices report now that it is v1.0?

Particularly interested in the "Unused Space :" line.

/dev/sda3:

          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x0
     Array UUID : 3905b303:ca604b72:be5949c4:ab051b7a
           Name : 2

Creation Time : Sun Jun 4 08:18:59 2006
Raid Level : raid10
Raid Devices : 6

Avail Dev Size : 618726528 (295.03 GiB 316.79 GB)

     Array Size : 928089792 (885.10 GiB 950.36 GB)

Super Offset : 618727392 sectors
Unused Space : before=0 sectors, after=864 sectors
State : active
Device UUID : e30176be:81a57e84:1f2aa206:9515150f

Update Time : Sun Aug 24 14:29:15 2014
Checksum : bd599c72 - correct
Events : 1

Layout : near=2
Chunk Size : 64K

Device Role : Active device 5
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)

All other devices report same (apart from UUID/checksum/role).

#10 lore raw
In reply to Andy Smith
Quoted 28 lines · On Mon, 25 Aug 2014 11:34:03 +0000 Andy Smith <[email protected]> wrote:

Hi Neil,

Quoted 3 lines · NeilBrown

What does "mdadm --examine" on one of the devices report now that it is v1.0?

Particularly interested in the "Unused Space :" line.

/dev/sda3:

          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x0
     Array UUID : 3905b303:ca604b72:be5949c4:ab051b7a
           Name : 2

Creation Time : Sun Jun 4 08:18:59 2006
Raid Level : raid10
Raid Devices : 6

Avail Dev Size : 618726528 (295.03 GiB 316.79 GB)

     Array Size : 928089792 (885.10 GiB 950.36 GB)

Super Offset : 618727392 sectors
Unused Space : before=0 sectors, after=864 sectors
State : active
Device UUID : e30176be:81a57e84:1f2aa206:9515150f

Update Time : Sun Aug 24 14:29:15 2014
Checksum : bd599c72 - correct
Events : 1

Quoted 7 lines

Layout : near=2
Chunk Size : 64K

Device Role : Active device 5
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)

All other devices report same (apart from UUID/checksum/role).

'after=864' should be plenty.

mdadm writes '128' to 'new_offset'
This is bigger than 'data_offset' (which must be zero, or it would have been reported), mddev->dev_sectors is 618726528 while rdev->sectors is 618727392 (difference is 864), so

		if (new_offset - rdev->data_offset
		    + mddev->dev_sectors > rdev->sectors)

cannot return 'true'. Yet it seems that it does.

Ahh.. somewhere else returns -E2BIG:

	if (mddev->pers && mddev->persistent &&
	    !super_types[mddev->major_version]
	    .allow_new_offset(rdev, new_offset))
		return -E2BIG;

But super_1_allow_new_offset always succeeds if new_offset > rdev->data_offset, so that cannot be the problem.

I know. rdev->sectors isn't "Super Offset", it is only "Avail Dev Size". You need to
echo 618727392 > /sys/block/md2/md/dev-sda3/size
and same for the other devices.

I looks like I need to get mdadm to be more clever when it gets -E2BIG. Can you check that the above works?

Thanks,
NeilBrown

author
#11 lore raw

Hi Neil,

Quoted 6 lines · NeilBrown

You need to
echo 618727392 > /sys/block/md2/md/dev-sda3/size
and same for the other devices.

I looks like I need to get mdadm to be more clever when it gets -E2BIG. Can you check that the above works?

Unfortunately it seems I cannot change that size:

# echo 618727392 > /sys/block/md2/md/dev-sda3/size
# cat /sys/block/md2/md/dev-sda3/size
309363264

Cheers,
Andy

#12 lore raw
Quoted 7 lines · On Fri, 29 Aug 2014 04:02:31 +0000 Andy Smith <[email protected]> wrote:

Hi Neil,

Quoted 4 lines · NeilBrown

You need to
echo 618727392 > /sys/block/md2/md/dev-sda3/size
and same for the other devices.

Quoted 9 lines
Quoted 2 lines

I looks like I need to get mdadm to be more clever when it gets -E2BIG. Can you check that the above works?

Unfortunately it seems I cannot change that size:

# echo 618727392 > /sys/block/md2/md/dev-sda3/size
# cat /sys/block/md2/md/dev-sda3/size
309363264

Of course 'size' is in kilobytes, not sectors (which moron did that !??!?!)

echo 309363696 > /sys/block/md2/md/dev-sda3/size

NeilBrown

author
#13 lore raw
Quoted 3 lines · NeilBrown

Of course 'size' is in kilobytes, not sectors (which moron did that !??!?!)

echo 309363696 > /sys/block/md2/md/dev-sda3/size

Sorry, still no joy:

# echo 309363696 > /sys/block/md2/md/dev-sda3/size
# cat /sys/block/md2/md/dev-sda3/size
309363264
#14 lore raw
Quoted 10 lines · On Fri, 29 Aug 2014 04:26:18 +0000 Andy Smith <[email protected]> wrote:
Quoted 3 lines · NeilBrown

Of course 'size' is in kilobytes, not sectors (which moron did that !??!?!)

echo 309363696 > /sys/block/md2/md/dev-sda3/size

Sorry, still no joy:

# echo 309363696 > /sys/block/md2/md/dev-sda3/size
# cat /sys/block/md2/md/dev-sda3/size
309363264

Try use "/bin/echo" so you get an error message.
Also try smaller numbers. Can you write anything larger than what is already there? Increasing by 64 (match the 128 sectors that mdadm wants for new_offset) should be enough.

NeilBrown

author
#15 lore raw

Hi Neil,

Quoted 11 lines · NeilBrown
Quoted 5 lines · On Fri, 29 Aug 2014 04:26:18 +0000 Andy Smith <[email protected]> wrote:

Sorry, still no joy:

# echo 309363696 > /sys/block/md2/md/dev-sda3/size
# cat /sys/block/md2/md/dev-sda3/size
309363264

Try use "/bin/echo" so you get an error message.
Also try smaller numbers. Can you write anything larger than what is already there? Increasing by 64 (match the 128 sectors that mdadm wants for new_offset) should be enough.

I can't even increase it by 64 from what it is now:

# /bin/echo 309363328 > /sys/block/md2/md/dev-sda3/size
# cat /sys/block/md2/md/dev-sda3/size
309363264

In fact I can't increase it by 1, and /bin/echo doesn't say why.

Cheers,
Andy

#16 lore raw
Quoted 15 lines · On Fri, 29 Aug 2014 04:42:26 +0000 Andy Smith <[email protected]> wrote:

Hi Neil,

Quoted 11 lines · NeilBrown
Quoted 5 lines · On Fri, 29 Aug 2014 04:26:18 +0000 Andy Smith <[email protected]> wrote:

Sorry, still no joy:

# echo 309363696 > /sys/block/md2/md/dev-sda3/size
# cat /sys/block/md2/md/dev-sda3/size
309363264

Try use "/bin/echo" so you get an error message.
Also try smaller numbers. Can you write anything larger than what is already there? Increasing by 64 (match the 128 sectors that mdadm wants for new_offset) should be enough.

Quoted 8 lines

I can't even increase it by 64 from what it is now:

# /bin/echo 309363328 > /sys/block/md2/md/dev-sda3/size
# cat /sys/block/md2/md/dev-sda3/size
309363264

In fact I can't increase it by 1, and /bin/echo doesn't say why.

Hmm.... I've obviously been assuming something that was wrong.

So I did some experiments myself....

If you apply the following patch to mdadm, stop the array, and then assemble with --update=devicesize, you should then be able to reshape the array to fewer devices.

This isn't a final solution, I need to think about the whole thing more carefully. But it will work for you.

Thanks for your patience.

NeilBrown

super1.c +11 -0
index 727a08a110e0..efa1042ab47a 100644
@@ -1251,6 +1251,17 @@ static int update_super1(struct supertype *st, struct mdinfo *info,
 			misc->device_size - __le64_to_cpu(sb->data_offset));
 		printf("Size is %llu\n", (unsigned long long)
 		       __le64_to_cpu(sb->data_size));
+	} else if (strcmp(update, "devicesize") == 0 &&
+		   !(__le32_to_cpu(sb->feature_map)&MD_FEATURE_BITMAP_OFFSET) &&
+		   __le64_to_cpu(sb->super_offset) >
+		   __le64_to_cpu(sb->data_offset)) {
+		/* set data_size to device size less data_offset */
+		printf("Size was %llu\n", (unsigned long long)
+		       __le64_to_cpu(sb->data_size));
+		sb->data_size = __cpu_to_le64(
+			__le64_to_cpu(sb->super_offset) - __le64_to_cpu(sb->data_offset));
+		printf("Size is %llu\n", (unsigned long long)
+		       __le64_to_cpu(sb->data_size));
 	} else if (strcmp(update, "revert-reshape") == 0) {
 		rv = -2;
 		if (!(sb->feature_map & __cpu_to_le32(MD_FEATURE_RESHAPE_ACTIVE)))
author
#17 lore raw

Hi Neil,

Quoted 11 lines · NeilBrown
Quoted 2 lines · On Fri, 29 Aug 2014 04:42:26 +0000 Andy Smith <[email protected]> wrote:

In fact I can't increase it by 1, and /bin/echo doesn't say why.

Hmm.... I've obviously been assuming something that was wrong.

So I did some experiments myself....

If you apply the following patch to mdadm, stop the array, and then assemble with --update=devicesize, you should then be able to reshape the array to fewer devices.

Thanks, I think we are getting somewhere:

# ./mdadm --stop /dev/md2
mdadm: stopped /dev/md2
# ./mdadm --assemble --update=devicesize /dev/md2
Size was 618726528
Size is 618727392
Size was 618726528
Size is 618727392
Size was 618726528
Size is 618727392
Size was 618726528
Size is 618727392
Size was 618726528
Size is 618727392
Size was 618726528
Size is 618727392
mdadm: /dev/md2 has been started with 6 drives.
# logout
$ head -5 /proc/mdstat
Personalities : [raid1] [raid10]
md2 : active raid10 sdd3[0] sda3[5] sdc3[4] sdb3[3] sdf3[2] sde3[1]
      928089792 blocks super 1.0 64K chunks 2 near-copies [6/6] [UUUUUU]
$ sudo ./mdadm /dev/md2 --grow --raid-devices=4
mdadm: this change will reduce the size of the array.
       use --grow --array-size first to truncate array.
       e.g. mdadm --grow /dev/md2 --array-size 618726528
unfreeze
$ sudo ./mdadm --grow /dev/md2 --array-size 618726528
$ sudo ./mdadm /dev/md2 --grow --raid-devices=4
$ head -5 /proc/mdstat
Personalities : [raid1] [raid10]
md2 : active raid10 sdd3[0] sda3[5] sdc3[4] sdb3[3] sdf3[2] sde3[1]
      618726528 blocks super 1.0 64K chunks 2 near-copies [4/4] [UUUU]
      [>....................]  reshape =  0.0% (211136/618726528) finish=244.1min speed=42227K/sec
Quoted 1 line

Thanks for your patience.

No problem, glad we got to the bottom of it!

Cheers,
Andy

author
#18 lore raw

PATCH 1/1 Grow: Report when grow needs metadata update

In reply to NeilBrown

Report when the array's metadata needs updating instead of just reporting the generic "kernel too old" message.

---
 Grow.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
Grow.c +3 -1
index a9c8589..76bb35a 100644
@@ -2255,7 +2255,9 @@ static int set_new_data_offset(struct mdinfo *sra, struct supertype *st,
 		if (info2.space_before == 0 &&
 		    info2.space_after == 0) {
 			/* Metadata doesn't support data_offset changes */
-			return 1;
+			pr_err("%s: Metadata version doesn't support"
+			       " data_offset changes\n", devname);
+			goto release;
 		}
 		if (before > info2.space_before)
 			before = info2.space_before;
Signature · 1 line
1.7.10.4
Applied
#19 lore raw

Re: [PATCH 1/1] Grow: Report when grow needs metadata update

Quoted 13 lines · On Fri, 29 Aug 2014 20:47:12 +0000 Andy Smith <[email protected]> wrote:

Report when the array's metadata needs updating instead of just reporting the generic "kernel too old" message.

---
 Grow.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
Grow.c +0 -0
index a9c8589..76bb35a 100644

Grow.c static int set_new_data_offset(struct mdinfo *sra, struct supertype *st, · line 2263

Quoted 10 lines
Grow.c +3 -1
@@ -2255,7 +2255,9 @@ static int set_new_data_offset(struct mdinfo *sra, struct supertype *st,
 		if (info2.space_before == 0 &&
 		    info2.space_after == 0) {
 			/* Metadata doesn't support data_offset changes */
-			return 1;
+			pr_err("%s: Metadata version doesn't support"
+			       " data_offset changes\n", devname);
+			goto release;
 		}
 		if (before > info2.space_before)
 			before = info2.space_before;

Applied, thanks.

NeilBrown