Shrinking number of devices on a RAID-10 (near 2) array
Hi,
I am aware that for a very long time it was not possible to change the number of devices in an mdadm RAID-10. But then recently I am sure I saw threads here alluding to this being possible. e.g.:
http://marc.info/?l=linux-raid&m=140768923829685&w=2
I have a 6 device RAID-10 near=2 array that I would like to shrink down to 4 devices.
I have compiled mdadm 3.3.2 and am using kernel 3.16.0.
$ sudo mdadm --detail /dev/md2
/dev/md2:
Version : 0.90
Creation Time : Sun Jun 4 08:18:58 2006
Raid Level : raid10
Array Size : 471859200 (450.00 GiB 483.18 GB)
Used Dev Size : 309363520 (295.03 GiB 316.79 GB)
Raid Devices : 6
Total Devices : 6
Preferred Minor : 2
Persistence : Superblock is persistent Update Time : Sat Aug 23 15:23:38 2014
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Layout : near=2
Chunk Size : 64K
UUID : 3905b303:ca604b72:be5949c4:ab051b7a
Events : 0.312149991
Number Major Minor RaidDevice State
0 8 51 0 active sync /dev/sdd3
1 8 67 1 active sync /dev/sde3
2 8 83 2 active sync /dev/sdf3
3 8 19 3 active sync /dev/sdb3
4 8 35 4 active sync /dev/sdc3
5 8 3 5 active sync /dev/sda3
$ sudo ./mdadm --grow -n4 /dev/md2 --backup-file /var/tmp/mdadm.backup
mdadm: Cannot set new_data_offset: RAID10 reshape not
supported on this kernelIs that not yet possible, then?
(Each device is 320GB so it should all fit with only four of them)
Cheers,
Andy
Quoted 46 lines · On Sat, 23 Aug 2014 16:31:10 +0000 Andy Smith <[email protected]> wrote:
Hi,
I am aware that for a very long time it was not possible to change the number of devices in an mdadm RAID-10. But then recently I am sure I saw threads here alluding to this being possible. e.g.:
http://marc.info/?l=linux-raid&m=140768923829685&w=2I have a 6 device RAID-10 near=2 array that I would like to shrink down to 4 devices.
I have compiled mdadm 3.3.2 and am using kernel 3.16.0.
$ sudo mdadm --detail /dev/md2 /dev/md2: Version : 0.90 Creation Time : Sun Jun 4 08:18:58 2006 Raid Level : raid10 Array Size : 471859200 (450.00 GiB 483.18 GB) Used Dev Size : 309363520 (295.03 GiB 316.79 GB) Raid Devices : 6 Total Devices : 6 Preferred Minor : 2 Persistence : Superblock is persistentUpdate Time : Sat Aug 23 15:23:38 2014
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0Layout : near=2
Chunk Size : 64KUUID : 3905b303:ca604b72:be5949c4:ab051b7a
Events : 0.312149991Number Major Minor RaidDevice State
0 8 51 0 active sync/dev/sdd3
1 8 67 1 active sync/dev/sde3
2 8 83 2 active sync/dev/sdf3
3 8 19 3 active sync/dev/sdb3
4 8 35 4 active sync/dev/sdc3
5 8 3 5 active sync/dev/sda3
Quoted 7 lines
$ sudo ./mdadm --grow -n4 /dev/md2 --backup-file /var/tmp/mdadm.backup mdadm: Cannot set new_data_offset: RAID10 reshape not supported on this kernelIs that not yet possible, then?
(Each device is 320GB so it should all fit with only four of them)
raid10 reshape is only supported for v1.x metadata, and you have 0.90. This is because it needs to move the data_offset and 0.90 cannot do that.
I see you have already adjusted the --array-size correctly - good.
You can achieve your result by:
1/ stop the array and assemble with --update=metadata. This will convert the
array to v1.0
2/ "mdadm /dev/md2 --grow --disks=4" That should perform the reshape.
You don't need a --backup-file with RAID10 ... or with other arrays if you have recent tools and enough space before or after.
I guess the error message could be better....
NeilBrown
Neil,
I assume you would need to shrink the filesystem first before this ? or does MDADM nicely do that as part of the process also ?
Craig
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of NeilBrown
Sent: Sunday, 24 August 2014 1:10 PM
To: Andy Smith
Cc: [email protected]
Subject: Re: Shrinking number of devices on a RAID-10 (near 2) array
Quoted 46 lines · On Sat, 23 Aug 2014 16:31:10 +0000 Andy Smith <[email protected]> wrote:
Hi,
I am aware that for a very long time it was not possible to change the number of devices in an mdadm RAID-10. But then recently I am sure I saw threads here alluding to this being possible. e.g.:
http://marc.info/?l=linux-raid&m=140768923829685&w=2I have a 6 device RAID-10 near=2 array that I would like to shrink down to 4 devices.
I have compiled mdadm 3.3.2 and am using kernel 3.16.0.
$ sudo mdadm --detail /dev/md2 /dev/md2: Version : 0.90 Creation Time : Sun Jun 4 08:18:58 2006 Raid Level : raid10 Array Size : 471859200 (450.00 GiB 483.18 GB) Used Dev Size : 309363520 (295.03 GiB 316.79 GB) Raid Devices : 6 Total Devices : 6 Preferred Minor : 2 Persistence : Superblock is persistentUpdate Time : Sat Aug 23 15:23:38 2014
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0Layout : near=2
Chunk Size : 64KUUID : 3905b303:ca604b72:be5949c4:ab051b7a
Events : 0.312149991Number Major Minor RaidDevice State
0 8 51 0 active sync/dev/sdd3
1 8 67 1 active sync/dev/sde3
2 8 83 2 active sync/dev/sdf3
3 8 19 3 active sync/dev/sdb3
4 8 35 4 active sync/dev/sdc3
5 8 3 5 active sync/dev/sda3
Quoted 7 lines
$ sudo ./mdadm --grow -n4 /dev/md2 --backup-file /var/tmp/mdadm.backup mdadm: Cannot set new_data_offset: RAID10 reshape not supported on this kernelIs that not yet possible, then?
(Each device is 320GB so it should all fit with only four of them)
raid10 reshape is only supported for v1.x metadata, and you have 0.90. This is because it needs to move the data_offset and 0.90 cannot do that.
I see you have already adjusted the --array-size correctly - good.
You can achieve your result by:
1/ stop the array and assemble with --update=metadata. This will convert the
array to v1.0
2/ "mdadm /dev/md2 --grow --disks=4" That should perform the reshape.
You don't need a --backup-file with RAID10 ... or with other arrays if you have recent tools and enough space before or after.
I guess the error message could be better....
NeilBrown
Disclaimer
CONFIDENTIAL
This message contains confidential information and is intended only for the intended recipients. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.
Quoted 3 lines · On Sun, 24 Aug 2014 06:28:25 +0000 Craig Curtin <[email protected]> wrote:
Neil,
I assume you would need to shrink the filesystem first before this ? or does MDADM nicely do that as part of the process also ?
mdadm is not a jack-of-all-trades. It won't touch your filesystem.
A careful look at Andy's "mdadm --detail" shows:
Quoted 2 lines
Quoted 2 lines
Array Size : 471859200 (450.00 GiB 483.18 GB)Used Dev Size : 309363520 (295.03 GiB 316.79 GB)
This is a near=2 6 drive RAID10, so Array Size should be 3 times Used Dev Size.
The fact that it isn't shows that he has already explicitly set the
--array-size to something appropriate. I'm sure he reduced the size of the
filesystem first as the man page says:
Before reducing the size of the array you should make sure that space isn't needed. If the device holds a filesystem, you would need to resize the filesystem to use less space.
If you try to resize an array to fewer drives and you haven't adjusted the --array-size already, mdadm will tell you what --array-size you need. This forces you into a two (at least) step process to minimize the chance of doing something silly.
The idea is that you "--grow --array-size" first, then use "fsck" to make sure the filesystem is still happy. Then use "--grow --raid-disks" to reshape the array without changing the size.
If you change the array size and get it wrong, you can always put it back. But once you start the reshape happening, you will have lost any data beyond "array-size".
NeilBrown
Quoted 88 lines
Craig
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of NeilBrown
Sent: Sunday, 24 August 2014 1:10 PM
To: Andy Smith
Cc: [email protected]
Subject: Re: Shrinking number of devices on a RAID-10 (near 2) arrayQuoted 46 lines · On Sat, 23 Aug 2014 16:31:10 +0000 Andy Smith <[email protected]> wrote:
Hi,
I am aware that for a very long time it was not possible to change the number of devices in an mdadm RAID-10. But then recently I am sure I saw threads here alluding to this being possible. e.g.:
http://marc.info/?l=linux-raid&m=140768923829685&w=2I have a 6 device RAID-10 near=2 array that I would like to shrink down to 4 devices.
I have compiled mdadm 3.3.2 and am using kernel 3.16.0.
$ sudo mdadm --detail /dev/md2 /dev/md2: Version : 0.90 Creation Time : Sun Jun 4 08:18:58 2006 Raid Level : raid10 Array Size : 471859200 (450.00 GiB 483.18 GB) Used Dev Size : 309363520 (295.03 GiB 316.79 GB) Raid Devices : 6 Total Devices : 6 Preferred Minor : 2 Persistence : Superblock is persistentUpdate Time : Sat Aug 23 15:23:38 2014
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0Layout : near=2
Chunk Size : 64KUUID : 3905b303:ca604b72:be5949c4:ab051b7a
Events : 0.312149991Number Major Minor RaidDevice State
0 8 51 0 active sync/dev/sdd3
1 8 67 1 active sync/dev/sde3
2 8 83 2 active sync/dev/sdf3
3 8 19 3 active sync/dev/sdb3
4 8 35 4 active sync/dev/sdc3
5 8 3 5 active sync/dev/sda3Quoted 7 lines
$ sudo ./mdadm --grow -n4 /dev/md2 --backup-file /var/tmp/mdadm.backup mdadm: Cannot set new_data_offset: RAID10 reshape not supported on this kernelIs that not yet possible, then?
(Each device is 320GB so it should all fit with only four of them)
raid10 reshape is only supported for v1.x metadata, and you have 0.90. This is because it needs to move the
data_offsetand 0.90 cannot do that.I see you have already adjusted the --array-size correctly - good.
You can achieve your result by:
1/ stop the array and assemble with --update=metadata. This will convert the
array to v1.0
2/ "mdadm/dev/md2--grow --disks=4" That should perform the reshape.
You don't need a --backup-file with RAID10 ... or with other arrays if you have recent tools and enough space before or after.I guess the error message could be better....
NeilBrown
Disclaimer
CONFIDENTIAL
This message contains confidential information and is intended only for the intended recipients. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.
Quoted 8 lines · NeilBrown
Quoted 2 lines
Quoted 2 lines
Array Size : 471859200 (450.00 GiB 483.18 GB)Used Dev Size : 309363520 (295.03 GiB 316.79 GB)
This is a near=2 6 drive RAID10, so Array Size should be 3 times Used Dev Size.
The fact that it isn't shows that he has already explicitly set the
--array-size to something appropriate. I'm sure he reduced the size of the
filesystem first as the man page says:
Yup, it's an LVM PV so I did pvresize and mdadm --array-size first to some safe lower value, before I realised that mdadm would have told me the exact --array-size I needed anyway.
I should have mentioned that to reduce confusion, but it didn't seem pertinent to where I got stuck.
Cheers,
Andy
Hi Neil,
Thanks for the quick response!
Quoted 9 lines · NeilBrown
raid10 reshape is only supported for v1.x metadata, and you have 0.90. This is because it needs to move the
data_offsetand 0.90 cannot do that.I see you have already adjusted the --array-size correctly - good.
You can achieve your result by:
1/ stop the array and assemble with --update=metadata. This will convert the
array to v1.0
I've now done this, so:
$ sudo mdadm --detail /dev/md2
/dev/md2:
Version : 1.0
Creation Time : Sun Jun 4 08:18:59 2006
Raid Level : raid10
Array Size : 618726528 (590.06 GiB 633.58 GB)
Used Dev Size : 309363264 (295.03 GiB 316.79 GB)
Raid Devices : 6
Total Devices : 6
Persistence : Superblock is persistent Update Time : Sun Aug 24 14:29:15 2014
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Layout : near=2
Chunk Size : 64K
Name : 2
UUID : 3905b303:ca604b72:be5949c4:ab051b7a
Events : 1
Number Major Minor RaidDevice State
0 8 51 0 active sync /dev/sdd3
1 8 67 1 active sync /dev/sde3
2 8 83 2 active sync /dev/sdf3
3 8 19 3 active sync /dev/sdb3
4 8 35 4 active sync /dev/sdc3
5 8 3 5 active sync /dev/sda3
I noticed something: The assemble with update re-set the array size to 928089792 (885.10 GiB 950.36 GB).
(the above --detail is after all steps below)
Quoted 1 line
2/ "mdadm
/dev/md2--grow --disks=4" That should perform the reshape.
I assume that should have been --raid-devices; --disks seems to be a syntax error. :)
Firstly it told me the array was too big, as expected.
$ sudo ./mdadm /dev/md2 --grow --raid-devices=4
mdadm: this change will reduce the size of the array.
use --grow --array-size first to truncate array.
e.g. mdadm --grow /dev/md2 --array-size 618726528
unfreezeOkay.
$ sudo ./mdadm --grow /dev/md2 --array-size 618726528
But:
$ sudo ./mdadm /dev/md2 --grow --raid-devices=4
mdadm: Cannot set new_data_offset: RAID10 reshape not
supported on this kernelQuoted 1 line
I guess the error message could be better....
Will be happy to send the trivial patch once we get to the bottom of this!
Cheers,
Andy
Quoted 3 lines · Andy Smith
$ sudo ./mdadm /dev/md2 --grow --raid-devices=4
mdadm: Cannot set new_data_offset: RAID10 reshape not
supported on this kernelGrow.c from latest git:
2397 err = sysfs_set_num(sra, sd, "new_offset",
2398 new_data_offset);
For the first device in the for loop (sda3), the above line 2397 is returning -1.
2399 }
2400 if (err < 0) {
2401 if (errno == E2BIG && data_offset != INVALID_SECTORS) {
2402 pr_err("data-offset is too big for %s\n",
2403 dn);
2404 goto release;
2405 }
2406 if (sd == sra->devs &&
2407 (errno == ENOENT || errno == E2BIG))
2408 /* Early kernel, no 'new_offset' file,
2409 * or kernel doesn't like us. 2410 * For RAID5/6 this is not fatal 2411 */
2412 return 1;
It reaches here line 2412 with errno == E2BIG./sys/block/md2/md/dev-sda3/new_offset exists.
From strace:
open("/sys/block/md2/md/dev-sda3/new_offset", O_WRONLY) = 4write(4, "128", 3) = -1 E2BIG (Argument list too long)
close(4) = 0
Not sure where to debug next.
Cheers,
Andy
Quoted 1 line
The optimum programming team size is 1.
Has Jurassic Park taught us nothing?
— pfilandr
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 31 lines · On Mon, 25 Aug 2014 10:32:34 +0000 Andy Smith <[email protected]> wrote:
Quoted 3 lines · Andy Smith
$ sudo ./mdadm /dev/md2 --grow --raid-devices=4 mdadm: Cannot set new_data_offset: RAID10 reshape not supported on this kernel
Grow.cfrom latest git:2397 err = sysfs_set_num(sra, sd, "
new_offset",
2398new_data_offset);For the first device in the for loop (sda3), the above line 2397 is returning -1.
2399 }
2400 if (err < 0) {
2401 if (errno == E2BIG &&data_offset!=INVALID_SECTORS) {
2402 pr_err("data-offset is too big for %s\n",
2403 dn);
2404 goto release;
2405 }
2406 if (sd ==sra->devs&&
2407 (errno == ENOENT || errno == E2BIG))
2408 /* Early kernel, no 'new_offset' file,2409 * or kernel doesn't like us. 2410 * For RAID5/6 this is not fatal 2411 */2412 return 1;
It reaches here line 2412 with errno == E2BIG.
/sys/block/md2/md/dev-sda3/new_offsetexists.
Quoted 7 lines
Quoted 1 line
From strace:
open("/sys/block/md2/md/dev-sda3/new_offset", O_WRONLY)= 4write(4, "128", 3)= -1 E2BIG (Argument list too long)
close(4) = 0Not sure where to debug next.
That comes from
else if (new_offset > rdev->data_offset) {
/* must not push array size beyond rdev_sectors */
if (new_offset - rdev->data_offset
+ mddev->dev_sectors > rdev->sectors)
return -E2BIG;in drivers/md/md.c in the kernel...
What does "mdadm --examine" on one of the devices report now that it is v1.0?
Particularly interested in the "Unused Space :" line.
NeilBrown
Hi Neil,
Quoted 3 lines · NeilBrown
What does "mdadm --examine" on one of the devices report now that it is v1.0?
Particularly interested in the "Unused Space :" line.
/dev/sda3:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x0
Array UUID : 3905b303:ca604b72:be5949c4:ab051b7a
Name : 2 Creation Time : Sun Jun 4 08:18:59 2006
Raid Level : raid10
Raid Devices : 6
Avail Dev Size : 618726528 (295.03 GiB 316.79 GB)
Array Size : 928089792 (885.10 GiB 950.36 GB)
Super Offset : 618727392 sectors
Unused Space : before=0 sectors, after=864 sectors
State : active
Device UUID : e30176be:81a57e84:1f2aa206:9515150f
Update Time : Sun Aug 24 14:29:15 2014
Checksum : bd599c72 - correct
Events : 1
Layout : near=2
Chunk Size : 64K
Device Role : Active device 5
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
All other devices report same (apart from UUID/checksum/role).
Cheers,
Andy
Hi Neil,
Any ideas on the below? Not in a huge rush but at some point I might have to just re-create the array and copy data back.
Cheers,
Andy
Quoted 35 lines · Andy Smith
Hi Neil,
Quoted 3 lines · NeilBrown
What does "mdadm --examine" on one of the devices report now that it is v1.0?
Particularly interested in the "Unused Space :" line.
/dev/sda3:Magic : a92b4efc Version : 1.0 Feature Map : 0x0 Array UUID : 3905b303:ca604b72:be5949c4:ab051b7a Name : 2Creation Time : Sun Jun 4 08:18:59 2006
Raid Level : raid10
Raid Devices : 6Avail Dev Size : 618726528 (295.03 GiB 316.79 GB)
Array Size : 928089792 (885.10 GiB 950.36 GB)Super Offset : 618727392 sectors
Unused Space : before=0 sectors, after=864 sectors
State : active
Device UUID : e30176be:81a57e84:1f2aa206:9515150fUpdate Time : Sun Aug 24 14:29:15 2014
Checksum :bd599c72- correct
Events : 1Layout : near=2
Chunk Size : 64KDevice Role : Active device 5
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)All other devices report same (apart from UUID/checksum/role).
Quoted 28 lines · On Mon, 25 Aug 2014 11:34:03 +0000 Andy Smith <[email protected]> wrote:
Hi Neil,
Quoted 3 lines · NeilBrown
What does "mdadm --examine" on one of the devices report now that it is v1.0?
Particularly interested in the "Unused Space :" line.
/dev/sda3:Magic : a92b4efc Version : 1.0 Feature Map : 0x0 Array UUID : 3905b303:ca604b72:be5949c4:ab051b7a Name : 2Creation Time : Sun Jun 4 08:18:59 2006
Raid Level : raid10
Raid Devices : 6Avail Dev Size : 618726528 (295.03 GiB 316.79 GB)
Array Size : 928089792 (885.10 GiB 950.36 GB)Super Offset : 618727392 sectors
Unused Space : before=0 sectors, after=864 sectors
State : active
Device UUID : e30176be:81a57e84:1f2aa206:9515150fUpdate Time : Sun Aug 24 14:29:15 2014
Checksum :bd599c72- correct
Events : 1
Quoted 7 lines
Layout : near=2
Chunk Size : 64KDevice Role : Active device 5
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)All other devices report same (apart from UUID/checksum/role).
'after=864' should be plenty.
mdadm writes '128' to 'new_offset'
This is bigger than 'data_offset' (which must be zero, or it would have been reported), mddev->dev_sectors is 618726528 while rdev->sectors is 618727392 (difference is 864), so
if (new_offset - rdev->data_offset + mddev->dev_sectors > rdev->sectors)
cannot return 'true'. Yet it seems that it does.
Ahh.. somewhere else returns -E2BIG:
if (mddev->pers && mddev->persistent && !super_types[mddev->major_version] .allow_new_offset(rdev, new_offset)) return -E2BIG;
But super_1_allow_new_offset always succeeds if new_offset > rdev->data_offset, so that cannot be the problem.
I know. rdev->sectors isn't "Super Offset", it is only "Avail Dev Size". You need to
echo 618727392 > /sys/block/md2/md/dev-sda3/size
and same for the other devices.
I looks like I need to get mdadm to be more clever when it gets -E2BIG. Can you check that the above works?
Thanks,
NeilBrown
Hi Neil,
Quoted 6 lines · NeilBrown
You need to
echo 618727392 >/sys/block/md2/md/dev-sda3/size
and same for the other devices.I looks like I need to get mdadm to be more clever when it gets
-E2BIG. Can you check that the above works?
Unfortunately it seems I cannot change that size:
# echo 618727392 > /sys/block/md2/md/dev-sda3/size # cat /sys/block/md2/md/dev-sda3/size 309363264
Cheers,
Andy
Quoted 7 lines · On Fri, 29 Aug 2014 04:02:31 +0000 Andy Smith <[email protected]> wrote:
Hi Neil,
Quoted 4 lines · NeilBrown
You need to
echo 618727392 >/sys/block/md2/md/dev-sda3/size
and same for the other devices.
Quoted 9 lines
Quoted 2 lines
I looks like I need to get mdadm to be more clever when it gets
-E2BIG. Can you check that the above works?Unfortunately it seems I cannot change that size:
# echo 618727392 > /sys/block/md2/md/dev-sda3/size # cat /sys/block/md2/md/dev-sda3/size 309363264
Of course 'size' is in kilobytes, not sectors (which moron did that !??!?!)
echo 309363696 > /sys/block/md2/md/dev-sda3/size
NeilBrown
Quoted 3 lines · NeilBrown
Of course 'size' is in kilobytes, not sectors (which moron did that !??!?!)
echo 309363696 >
/sys/block/md2/md/dev-sda3/size
Sorry, still no joy:
# echo 309363696 > /sys/block/md2/md/dev-sda3/size # cat /sys/block/md2/md/dev-sda3/size 309363264
Quoted 10 lines · On Fri, 29 Aug 2014 04:26:18 +0000 Andy Smith <[email protected]> wrote:
Quoted 3 lines · NeilBrown
Of course 'size' is in kilobytes, not sectors (which moron did that !??!?!)
echo 309363696 >
/sys/block/md2/md/dev-sda3/sizeSorry, still no joy:
# echo 309363696 > /sys/block/md2/md/dev-sda3/size # cat /sys/block/md2/md/dev-sda3/size 309363264
Try use "/bin/echo" so you get an error message.
Also try smaller numbers. Can you write anything larger than what is already there? Increasing by 64 (match the 128 sectors that mdadm wants for new_offset) should be enough.
NeilBrown
Hi Neil,
Quoted 11 lines · NeilBrown
Quoted 5 lines · On Fri, 29 Aug 2014 04:26:18 +0000 Andy Smith <[email protected]> wrote:
Sorry, still no joy:
# echo 309363696 > /sys/block/md2/md/dev-sda3/size # cat /sys/block/md2/md/dev-sda3/size 309363264Try use "
/bin/echo" so you get an error message.
Also try smaller numbers. Can you write anything larger than what is already there? Increasing by 64 (match the 128 sectors that mdadm wants fornew_offset) should be enough.
I can't even increase it by 64 from what it is now:
# /bin/echo 309363328 > /sys/block/md2/md/dev-sda3/size # cat /sys/block/md2/md/dev-sda3/size 309363264
In fact I can't increase it by 1, and /bin/echo doesn't say why.
Cheers,
Andy
Quoted 15 lines · On Fri, 29 Aug 2014 04:42:26 +0000 Andy Smith <[email protected]> wrote:
Hi Neil,
Quoted 11 lines · NeilBrown
Quoted 5 lines · On Fri, 29 Aug 2014 04:26:18 +0000 Andy Smith <[email protected]> wrote:
Sorry, still no joy:
# echo 309363696 > /sys/block/md2/md/dev-sda3/size # cat /sys/block/md2/md/dev-sda3/size 309363264Try use "
/bin/echo" so you get an error message.
Also try smaller numbers. Can you write anything larger than what is already there? Increasing by 64 (match the 128 sectors that mdadm wants fornew_offset) should be enough.
Quoted 8 lines
I can't even increase it by 64 from what it is now:
# /bin/echo 309363328 > /sys/block/md2/md/dev-sda3/size # cat /sys/block/md2/md/dev-sda3/size 309363264In fact I can't increase it by 1, and
/bin/echodoesn't say why.
Hmm.... I've obviously been assuming something that was wrong.
So I did some experiments myself....
If you apply the following patch to mdadm, stop the array, and then assemble with --update=devicesize, you should then be able to reshape the array to fewer devices.
This isn't a final solution, I need to think about the whole thing more carefully. But it will work for you.
Thanks for your patience.
NeilBrown
super1.c +11 -0
@@ -1251,6 +1251,17 @@ static int update_super1(struct supertype *st, struct mdinfo *info, misc->device_size - __le64_to_cpu(sb->data_offset));printf ("Size is %llu\n" , (unsigned long long )__le64_to_cpu (sb->data_size)); + }else if (strcmp (update,"devicesize" ) ==0 && + !(__le32_to_cpu (sb->feature_map)&MD_FEATURE_BITMAP_OFFSET ) && +__le64_to_cpu (sb->super_offset) > +__le64_to_cpu (sb->data_offset)) { +/* set data_size to device size less data_offset */ +printf ("Size was %llu\n" , (unsigned long long ) +__le64_to_cpu (sb->data_size)); + sb->data_size =__cpu_to_le64 ( +__le64_to_cpu (sb->super_offset) -__le64_to_cpu (sb->data_offset)); +printf ("Size is %llu\n" , (unsigned long long ) +__le64_to_cpu (sb->data_size)); }else if (strcmp (update,"revert-reshape" ) ==0 ) { rv =-2 ;if (!(sb->feature_map &__cpu_to_le32 (MD_FEATURE_RESHAPE_ACTIVE )))
Hi Neil,
Quoted 11 lines · NeilBrown
Quoted 2 lines · On Fri, 29 Aug 2014 04:42:26 +0000 Andy Smith <[email protected]> wrote:
In fact I can't increase it by 1, and
/bin/echodoesn't say why.Hmm.... I've obviously been assuming something that was wrong.
So I did some experiments myself....
If you apply the following patch to mdadm, stop the array, and then assemble with --update=devicesize, you should then be able to reshape the array to fewer devices.
Thanks, I think we are getting somewhere:
# ./mdadm --stop /dev/md2
mdadm: stopped /dev/md2
# ./mdadm --assemble --update=devicesize /dev/md2
Size was 618726528
Size is 618727392
Size was 618726528
Size is 618727392
Size was 618726528
Size is 618727392
Size was 618726528
Size is 618727392
Size was 618726528
Size is 618727392
Size was 618726528
Size is 618727392
mdadm: /dev/md2 has been started with 6 drives.
# logout
$ head -5 /proc/mdstat
Personalities : [raid1] [raid10]
md2 : active raid10 sdd3[0] sda3[5] sdc3[4] sdb3[3] sdf3[2] sde3[1]
928089792 blocks super 1.0 64K chunks 2 near-copies [6/6] [UUUUUU]$ sudo ./mdadm /dev/md2 --grow --raid-devices=4
mdadm: this change will reduce the size of the array.
use --grow --array-size first to truncate array.
e.g. mdadm --grow /dev/md2 --array-size 618726528
unfreeze
$ sudo ./mdadm --grow /dev/md2 --array-size 618726528
$ sudo ./mdadm /dev/md2 --grow --raid-devices=4
$ head -5 /proc/mdstat
Personalities : [raid1] [raid10]
md2 : active raid10 sdd3[0] sda3[5] sdc3[4] sdb3[3] sdf3[2] sde3[1]
618726528 blocks super 1.0 64K chunks 2 near-copies [4/4] [UUUU]
[>....................] reshape = 0.0% (211136/618726528) finish=244.1min speed=42227K/secQuoted 1 line
Thanks for your patience.
No problem, glad we got to the bottom of it!
Cheers,
Andy
PATCH 1/1 Grow: Report when grow needs metadata update
In reply to NeilBrownReport when the array's metadata needs updating instead of just reporting the generic "kernel too old" message.
--- Grow.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
Grow.c +3 -1
@@ -2255,7 +2255,9 @@ static int set_new_data_offset(struct mdinfo *sra, struct supertype *st,if (info2.space_before ==0 && info2.space_after ==0 ) {/* Metadata doesn't support data_offset changes */ -return 1 ; +pr_err ("%s: Metadata version doesn't support" +" data_offset changes\n" , devname); + goto release; }if (before > info2.space_before) before = info2.space_before;
Signature · 1 line
1.7.10.4
NeilBrown
AppliedRe: [PATCH 1/1] Grow: Report when grow needs metadata update
Quoted 13 lines · On Fri, 29 Aug 2014 20:47:12 +0000 Andy Smith <[email protected]> wrote:
Report when the array's metadata needs updating instead of just reporting the generic "kernel too old" message.
Signed-off-by Andy Smith <[email protected]>--- Grow.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)Grow.c +0
-0
Grow.c static int set_new_data_offset(struct mdinfo *sra, struct supertype *st, · line 2263
Quoted 10 lines
Grow.c +3
-1@@ -2255,7 +2255,9 @@ static int set_new_data_offset(struct mdinfo *sra, struct supertype *st,if (info2.space_before ==0 && info2.space_after ==0 ) {/* Metadata doesn't support data_offset changes */ -return 1 ; +pr_err ("%s: Metadata version doesn't support" +" data_offset changes\n" , devname); + goto release; }if (before > info2.space_before) before = info2.space_before;
Applied, thanks.
NeilBrown