messed up changing chunk size

I've been playing around with RAID config and may have finally messed
it up.

Initially, I created the array with 3 300G drives:

# mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sda4 /dev/sdb1 /dev/sdc1
# mkfs.ext4 -v -m 0.01 -E stride=16,stripe-width=32 /dev/md0

It's been working nicely so far, and I decided to add a 4th 300G drive:

# mdadm --grow --raid-devices=4 --backup-file=/root/grow_md0.bak /dev/md0

That finished overnight, while I looked around and found that chunk size of 512 should work better. I unmounted the FS and ran

# mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak /dev/md0
mdadm: component size 293033536K is not a multiple of chunksize 512K

so I sized it down a bit:

# mdadm --grow -z 293033472 --backup-file=/root/grow_md0_size.bak /dev/md0

and then back to resizing chunks:
mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak /dev/md0

It's running right now:

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1]
      293033472 blocks super 0.91 level 5, 64k chunk, algorithm 2 [4/4] 
[UUUU]
      [====>................]  reshape = 22.7% (66540032/293033472) 
finish=947.3min speed=3984K/sec

But just now I tried to mount the filesystem and it's failing:
EXT4-fs (md0): bad geometry: block count 146516768 exceeds size of device (73258368 blocks)

Here's the question, then: am I royally screwed or is my data still there? How do I recover?

Yes, in retrospect I should've probably checked around whether this process has been successful for others.. but that's why they say hindsight is 20/20

It seems that when you shrank the array a little bit that you chopped off the end of the ext4 filesystem...

The raid array is now smaller than the filesystem says it is - hence your error.

--
Steven Haigh

Email: [email protected]
Web: http://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
Fax: (03) 8338 0299

That's what I thought, too.
But, again, the question is whether or not I can recover the data...

I think this should work:
Put the chunk size back, resize the array to the original size, then run fsck and hope it did not need the data on the end that got trashed. Then shrink your file system first (by a really safe margin), then resize the array, then change chunk size. Then grow the file system to use the max allowed.

Maybe the data on the end did not get trashed, but I don't know.

} -----Original Message-----
} From: [email protected] [mailto:linux-raid-
} [email protected]] On Behalf Of Konstantin Svist
} Sent: Sunday, July 18, 2010 8:32 PM
} To: [email protected]
} Subject: messed up changing chunk size
}
} I've been playing around with RAID config and may have finally messed
} it up.
}
} Initially, I created the array with 3 300G drives:
} # mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sda4 } /dev/sdb1 /dev/sdc1
} # mkfs.ext4 -v -m 0.01 -E stride=16,stripe-width=32 /dev/md0
}
} It's been working nicely so far, and I decided to add a 4th 300G drive:
} # mdadm --grow --raid-devices=4 --backup-file=/root/grow_md0.bak /dev/md0 }
} That finished overnight, while I looked around and found that chunk size } of 512 should work better. I unmounted the FS and ran
} # mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak /dev/md0 } mdadm: component size 293033536K is not a multiple of chunksize 512K }
} so I sized it down a bit:
} # mdadm --grow -z 293033472 --backup-file=/root/grow_md0_size.bak /dev/md0 }
} and then back to resizing chunks:
} mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak /dev/md0 }
} It's running right now:
} # cat /proc/mdstat
} Personalities : [raid6] [raid5] [raid4]
} md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1]
} 293033472 blocks super 0.91 level 5, 64k chunk, algorithm 2 [4/4]
} [UUUU]
} [====>................] reshape = 22.7% (66540032/293033472)
} finish=947.3min speed=3984K/sec
}
}
} But just now I tried to mount the filesystem and it's failing:
} EXT4-fs (md0): bad geometry: block count 146516768 exceeds size of } device (73258368 blocks)
}
} Here's the question, then: am I royally screwed or is my data still } there? How do I recover?
}
}
}
} Yes, in retrospect I should've probably checked around whether this } process has been successful for others.. but that's why they say } hindsight is 20/20
}
} --
} To unsubscribe from this list: send the line "unsubscribe linux-raid" in } the body of a message to [email protected]
} More majordomo info at http://vger.kernel.org/majordomo-info.html

Right now the reshape is at "32.1% (94138368/293033472)
finish=861.4min speed=3847K/sec"
Is it safe to reshape back right now, will it pick up the command?

No idea, but I would not risk it. Let it finish, unless someone else has a useful tip.

} -----Original Message-----
} From: [email protected] [mailto:linux-raid-
} [email protected]] On Behalf Of Konstantin Svist
} Sent: Sunday, July 18, 2010 9:46 PM
} To: [email protected]
} Subject: Re: messed up changing chunk size
}
} Right now the reshape is at "32.1% (94138368/293033472)
} finish=861.4min speed=3847K/sec"
} Is it safe to reshape back right now, will it pick up the command? }
} On 07/18/2010 06:29 PM, Guy Watkins wrote:
} > I think this should work:
} > Put the chunk size back, resize the array to the original size, then run } > fsck and hope it did not need the data on the end that got trashed. } Then
} > shrink your file system first (by a really safe margin), then resize the } > array, then change chunk size. Then grow the file system to use the max } > allowed.
} >
} > Maybe the data on the end did not get trashed, but I don't know. } >
} >
} > } -----Original Message-----
} > } From: [email protected] [mailto:linux-raid- } > } [email protected]] On Behalf Of Konstantin Svist
} > } Sent: Sunday, July 18, 2010 8:32 PM
} > } To: [email protected]
} > } Subject: messed up changing chunk size
} > }
} > } I've been playing around with RAID config and may have finally
} messed
} > } it up.
} > }
} > } Initially, I created the array with 3 300G drives:
} > } # mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 } /dev/sda4
} > } /dev/sdb1 /dev/sdc1
} > } # mkfs.ext4 -v -m 0.01 -E stride=16,stripe-width=32 /dev/md0 } > }
} > } It's been working nicely so far, and I decided to add a 4th 300G } drive:
} > } # mdadm --grow --raid-devices=4 --backup-file=/root/grow_md0.bak } /dev/md0
} > }
} > } That finished overnight, while I looked around and found that chunk } size
} > } of 512 should work better. I unmounted the FS and ran
} > } # mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak } /dev/md0
} > } mdadm: component size 293033536K is not a multiple of chunksize 512K } > }
} > } so I sized it down a bit:
} > } # mdadm --grow -z 293033472 --backup-file=/root/grow_md0_size.bak } /dev/md0
} > }
} > } and then back to resizing chunks:
} > } mdadm --grow -c 512 --backup-file=/root/grow_md0_rechunk.bak /dev/md0 } > }
} > } It's running right now:
} > } # cat /proc/mdstat
} > } Personalities : [raid6] [raid5] [raid4]
} > } md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1]
} > } 293033472 blocks super 0.91 level 5, 64k chunk, algorithm 2
} [4/4]
} > } [UUUU]
} > } [====>................] reshape = 22.7% (66540032/293033472)
} > } finish=947.3min speed=3984K/sec
} > }
} > }
} > } But just now I tried to mount the filesystem and it's failing:
} > } EXT4-fs (md0): bad geometry: block count 146516768 exceeds size of } > } device (73258368 blocks)
} > }
} > } Here's the question, then: am I royally screwed or is my data still } > } there? How do I recover?
} > }
} > }
} > }
} > } Yes, in retrospect I should've probably checked around whether this } > } process has been successful for others.. but that's why they say } > } hindsight is 20/20
} > }
} > } --
} > } To unsubscribe from this list: send the line "unsubscribe linux-raid" } in
} > } the body of a message to [email protected]
} > } More majordomo info at http://vger.kernel.org/majordomo-info.html
} >
} >
}
} --
} To unsubscribe from this list: send the line "unsubscribe linux-raid" in } the body of a message to [email protected]
} More majordomo info at http://vger.kernel.org/majordomo-info.html

Not true, at least with RAID5/6 a chunk size of 64K performs faster, see
http://louwrentius.blogspot.com/2010/05/raid-level-and-chunk-size-benchmarks.html
http://alephnull.com/benchmarks/sata2009/chunksize.html

With respect,
Roman

There is no mention in this benchmark tests if he modified the stripe cache. With an increased stripe size a larger stripe cache would be needed for good performance (at least from my experience)

Best Regards

Jools

That's true, but increasing the stripe cache helps across all chunk sizes, even with relatively low 128K, see [1]. And I don't think that a large-chunk(512K) configuration will significantly, if at all, outperform a small-chunks(64K) one at any given stripe cache size (set to the same value in both cases).

There's also an opinion I heard from a couple of sources, that the stripe size better be chosen so that either one stripe, or a whole stride fits inside the CPU's L2 or L3 cache. Sounds logical, though I am not convinced that this is what causes the performance difference between various stripe sizes.

[1]
http://peterkieser.com/2009/11/29/raid-mdraid-stripe_cache_size-vs-write-transfer/

With respect,
Roman

Thanks for these benchmarks, I added them to our benchmarks page on the wiki.

It seems odd to me to use sata controllers on a 32-bit PCI bus,
this should limit performance to about 130 MB/s - which is also seen tin the graphs. I think people would be better off using the on-board SATA controller, this would give normally something like 4 SATA ports connected to the southbridge, which normally is much faster than a 32-bit PCI bus, and also faster than many PCI-E busses. So I think using all the on-board SATA ports first, and the using the SATA ports on a bus-connceted controller would give better performance.

In the louwrentius page, I miss info on which layout of RAID10 that is used. The 3 layouts (near, far, offset) has very different characteristics.

best regards
keld

Just noticed that I got this in /var/log/messages when I sized it down:

Jul 18 12:00:44 slon kernel: md0: detected capacity change from 900199022592 to 300066275328
Jul 18 12:00:44 slon kernel: md0: unknown partition table

And scraped from command line session:

[root@slon ~]# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1]

      879100608 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

[root@slon ~]# mdadm --grow -z 293033472 --backup-file=/root/grow_md0_size.bak /dev/md0
mdadm: component size of /dev/md0 has been set to 293033472K
[root@slon ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1]

      293033472 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

Looks like instead of setting the number of blocks per-drive (as per the man page), the -z option set the number of total blocks for the array, instead. Which means I'm very thoroughly screwed.

Right now I'm trying to reshape back to the old chunk size and will then try to resize the array back to the previous size. If the chunk size change and array resize are both fully reversible -- and I don't think I'm THAT lucky -- then I might be able to get some stuff back. Will see, I guess.

Can someone confirm that this command should work to revert the size? mdadm --grow -z 879100608 --backup-file=/root/grow_md0_size_revert.bak /dev/md0

Finally sat down and thought about it - created a temporary raid using /dev/ram## and copied some files to it, did the same operations on it. Amazingly, all the files came back. Will see if the real thing works too

For some reason, mdadm refused to resize back to max:

# mdadm --grow -z max --backup-file=/root/grow_md0_size_back.bak /dev/md0
mdadm: component size of /dev/md0 has been set to 293033536K
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[3] sda4[0] sdc1[2] sdb1[1]
      293033472 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

After stopping and re-assembling the array, though, it comes up with full size but with only 3 of 4 drives:

# mdadm --assemble --scan
mdadm: /dev/md0 has been started with 3 drives (out of 4).
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sda4[0] sdc1[2] sdb1[1]
      879100608 blocks level 5, 64k chunk, algorithm 2 [4/3] [UUU_]

The array is degraded but data is all there, so I'm backing it up right now. No idea why it doesn't like the 4th drive.

3 out of 4 was due to bad /etc/mdadm.conf -- I forgot the / in front of /dev/sdd1 It's now rebuilding and everything is fine so far