Dumb questions about mdadm #1 - replacing broken disks - "slot" reuse?
mdadm version 2.6.7.2-3 on Debian Lenny, kernel 2.6.26-2-xen-amd64
I'm new to mdadm, all my experience with software raid/volume management type systems has been with Vertas VxVM on UnixWare.
I'm replacing an existing UnixWare system by Linux and I'm trying to get a feel for how to perform some simple operations.
As I understand it to replace a failed disk (assuming no hot spares for the moment) I just do:
mdadm --manage /dev/md0 --remove /dev/failed-disk mdadm --manage /dev/md0 --add /dev/new-disk
This works, but when I look at the results it looks rather ugly, the new disk goes in a new "slot" in the raid superblock. Is it the case that every time I replace a disk I'm going to get a new slot? Doesn't that make the raid "superblock" grow without limit?
Before:
# cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid10 sdw[11] sdg[10] sdv[9] sdf[8] sdu[7] sde[6] sdt[5] sdd[4] sds[3] sdc[2] sdr[1] sdb[0]
426970368 blocks super 1.2 64K chunks 2 near-copies [12/12] [UUUUUUUUUUUU]
bitmap: 1/204 pages [4KB], 1024KB chunkunused devices: <none>
# mdadm --examine /dev/sdb:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 9477b121:204de4c4:a96d58e9:85746699
Name : caronia:testarray2 (local to host caronia)
Creation Time : Fri Sep 18 12:59:24 2009
Raid Level : raid10
Raid Devices : 12Avail Dev Size : 142323568 (67.87 GiB 72.87 GB)
Array Size : 853940736 (407.19 GiB 437.22 GB)
Used Dev Size : 142323456 (67.87 GiB 72.87 GB)
Data Offset : 144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 64949ade:b6d618a8:f45a3b07:29ddc35e
Internal Bitmap : 8 sectors from superblock
Update Time : Fri Sep 18 13:38:41 2009
Checksum : 256ea9d4 - correct
Events : 4
Layout : near=2, far=1
Chunk Size : 64K
Array Slot : 0 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11)
Array State : Uuuuuuuuuuuu
After:
# mdadm --manage /dev/md0 --fail /dev/sdc mdadm: set /dev/sdc faulty in /dev/md0 # mdadm --manage /dev/md0 --remove /dev/sdc mdadm: hot removed /dev/sdc # mdadm --manage /dev/md0 --add /dev/sdh mdadm: added /dev/sdh
[...]
# cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid10 sdh[12] sdw[11] sdg[10] sdv[9] sdf[8] sdu[7] sde[6] sdt[5] sdd[4] sds[3] sdr[1] sdb[0]
426970368 blocks super 1.2 64K chunks 2 near-copies [12/12] [UUUUUUUUUUUU]
bitmap: 0/204 pages [0KB], 1024KB chunkunused devices: <none>
Eeew, /dev/sdh is in slot 12, where is slot 2?
And:
# mdadm --examine /dev/sdh
/dev/sdh:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 9477b121:204de4c4:a96d58e9:85746699
Name : caronia:testarray2 (local to host caronia)
Creation Time : Fri Sep 18 12:59:24 2009
Raid Level : raid10
Raid Devices : 12Avail Dev Size : 142323568 (67.87 GiB 72.87 GB)
Array Size : 853940736 (407.19 GiB 437.22 GB)
Used Dev Size : 142323456 (67.87 GiB 72.87 GB)
Data Offset : 144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 31c4a4f5:7aef046d:8981c552:b63165c2
Internal Bitmap : 8 sectors from superblock
Update Time : Fri Sep 18 13:56:55 2009
Checksum : b09071e2 - correct
Events : 14
Layout : near=2, far=1
Chunk Size : 64K Array Slot : 12 (0, 1, failed, 3, 4, 5, 6, 7, 8, 9, 10, 11, 2)
Array State : uuUuuuuuuuuu 1 failed
Is it the case that every time I replace a disk I'm going to get a new slot? Doesn't that make the raid "superblock" grow without limit?
Quoted 2 lines · John Hughes
This works, but when I look at the results it looks rather ugly, the new disk goes in a new "slot" in the raid superblock. Is it the case that
This is not a dumb question. This is just unexpected behaviour :) I have the same here on a 1.1 superblock RAID1
Quoted 8 lines
Before:
md0 : active raid10 sdw[11] sdg[10] sdv[9] sdf[8] sdu[7] sde[6] sdt[5] sdd[4] sds[3] sdc[2] sdr[1] sdb[0]426970368 blocks super 1.2 64K chunks 2 near-copies [12/12] [UUUUUUUUUUUU] bitmap: 1/204 pages [4KB], 1024KB chunk After: md0 : active raid10 sdh[12] sdw[11] sdg[10] sdv[9] sdf[8] sdu[7] sde[6] sdt[5] sdd[4] sds[3] sdr[1] sdb[0] 426970368 blocks super 1.2 64K chunks 2 near-copies [12/12] [UUUUUUUUUUUU] bitmap: 0/204 pages [0KB], 1024KB chunk
In my case I created the array as:
# uname -a Linux darkside 2.6.30-1-686 #1 SMP Sat Aug 15 19:11:58 UTC 2009 i686 GNU/Linux # mdadm --create -e 1.1 -l raid1 -n 2 -b internal -a md /dev/md9 /dev/hdh missing
and subsequently added /dev/sdd.
/proc/mdstat:
md9 : active raid1 sdd[2] hdh[0]
312571092 blocks super 1.1 [2/2] [UU]
bitmap: 1/299 pages [4KB], 512KB chunk# mdadm -E /dev/hdh /dev/sdd
/dev/hdh:
Magic : a92b4efc
Version : 1.1
Feature Map : 0x1
...
Raid Level : raid1
Raid Devices : 2
...
Data Offset : 264 sectors
Super Offset : 0 sectors
State : clean
...
Internal Bitmap : 8 sectors from superblock
Update Time : Fri Sep 18 20:14:24 2009
Checksum : 754e56a3 - correct
Events : 12564
...
Device Role : Active device 0
Array State : AA ('A' == active, '.' == missing)
/dev/sdd:
Magic : a92b4efc
Version : 1.1
Feature Map : 0x1
...
Raid Level : raid1
Raid Devices : 2
...
Data Offset : 264 sectors
Super Offset : 0 sectors
State : clean
...
Internal Bitmap : 8 sectors from superblock
Update Time : Fri Sep 18 20:14:24 2009
Checksum : 8747d7b3 - correct
Events : 12564
...
Device Role : spare
Array State : AA ('A' == active, '.' == missing)mdadm -A --update=summaries' doesn't help (I didn't expect it to, tho) mdadm -A --update=resync' doesn't help
`--fail --remove --add' doesn't help--fail --remove --grow -n 1 --grow -n 2 --add' doesn't help --fail --remove --grow -n 1 --add --grow -n 2' doesn't help
The array resyncs the few bitmap-chunks and the component remains in slot 2, the array has no device missing, the bitmap stays at 1 page (1620 bits set). Writing to the array increases the bits set for a short time and then they go back to 1 page (1620 bits) and remain there. The components are in sync and remain in sync, I cmp'ed them.
regards
Mario
Signature · 7 lines
Damit das Mögliche entsteht, muß immer wieder das Unmögliche versucht werden.
-- Hermann Hesse
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlHello again,
the issue is btw. easily reproducible:
# dd if=/dev/zero of=/tmp/foo1 bs=1k count=100k
102400+0 records in
102400+0 records out
104857600 bytes (105 MB) copied, 0,18648 s, 562 MB/s
# losetup /dev/loop0 /tmp/foo1
# mdadm --create -e 1.1 -l raid1 -n 2 -a md /dev/md8 /dev/loop0 missing
mdadm: array /dev/md8 started.
# dd if=/dev/zero of=/tmp/foo2 bs=1k count=100k
102400+0 records in
102400+0 records out
104857600 bytes (105 MB) copied, 0,196626 s, 533 MB/s
# losetup /dev/loop1 /tmp/foo2
# mdadm --add /dev/md8 /dev/loop1
mdadm: added /dev/loop1
# grep -A1 ^md8 /proc/mdstat
md8 : active raid1 loop1[2] loop0[0]
102392 blocks super 1.1 [2/2] [UU]
# mdadm -E /dev/loop0 | tail -2
Device Role : Active device 0
Array State : AA ('A' == active, '.' == missing)
You have new mail in /var/mail/root
# mdadm -E /dev/loop1 | tail -2
Device Role : spare
Array State : AA ('A' == active, '.' == missing)
# regards
Mario
Signature · 4 lines
<jv> Oh well, config <jv> one actually wonders what force in the universe is holding it <jv> and makes it working <Beeth> chances and accidents :)
How to reclaim device slots on v1 superblock?
Hello,
md with v1 superblocks seem to somehow associate device slots with specific devices and doesn't re-use them with others (this was already reported by John Hughes a while ago, btw.).
So, my question is: how can I force md to finally give up a specific slot or re-use it with another device?
For example, creating a degraded array reserves slot 1 for the "missing" device and never re-uses it, but uses slot 2 instead:
# mdadm -C -n 2 -l raid1 -a md /dev/md9 /dev/loop0 missing # mdadm -a /dev/md9 /dev/loop1 md9 : active raid1 loop1[2] loop0[0]
Creating the array fully developed uses slot 0 and 1 as expected:
# mdadm -C -n 2 -l raid1 -a md /dev/md9 /dev/loop0 /dev/loop1 md9 : active raid1 loop1[1] loop0[0]
The only way I found to enforce using slot 0 and 1 while creating a degraded array is to grow the number of active devices later:
# mdadm -C -n 1 --force -l raid1 -a md /dev/md9 /dev/loop0 # mdadm -a /dev/md9 /dev/loop1 # mdadm -G -n 2 /dev/md9 md9 : active raid1 loop1[1] loop0[0]
Unfortunately, the reverse assumption: reduce the number of active devices to re-claim unused slots doesn't appear to work:
# mdadm -C -n 2 -l raid1 -a md /dev/md9 /dev/loop0 missing # mdadm -a /dev/md9 /dev/loop1 md9 : active raid1 loop1[2] loop0[0] # mdadm --fail /dev/md9 /dev/loop1 # mdadm --remove /dev/md9 /dev/loop1 # mdadm -G -n 1 --force /dev/md9 # mdadm -a /dev/md9 /dev/loop1 # mdadm -G -n 2 /dev/md9 md9 : active raid1 loop1[2] loop0[0]
I'm using Debians mdadm 3.1.2-2 which defaults to v1.2 superblocks.
regards
Mario
Signature · 3 lines
Tower: "Say fuelstate." Pilot: "Fuelstate." Tower: "Say again." Pilot: "Again." Tower: "Arghl, give me your fuel!" Pilot: "Sorry, need it by myself..."