RAID5 attempt rebuilding despite being incomplete
Hi,
I have the problem that my RAID5 (created with 4 drives) array is resyning despite the fact I removed one drive. Any Idea what it is doing?
To trigger this:
- Created RAID5 array with 4 partitions (no spares).
- Set one drive to fail with 'mdadm --fail /dev/md3 /dev/hdf4'
- Removed this drive with 'mdadm --remove /dev/md3 /dev/hdf4'
After this I get:
---------------------------------------------------
# mdadm --detail /dev/md3
/dev/md3:
Version : 00.90.01
Creation Time : Thu Jun 23 11:43:59 2005
Raid Level : raid5
Array Size : 228151872 (217.58 GiB 233.63 GB)
Device Size : 76050624 (72.53 GiB 77.88 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 3
Persistence : Superblock is persistent Update Time : Thu Jun 23 12:26:19 2005
State : active, degraded, resyncing
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Rebuild Status : 24% complete
UUID : b9e2b624:b9bef9df:dcdec561:0c3a8dfd
Events : 0.9
Number Major Minor RaidDevice State
0 3 4 0 active sync /dev/.static/dev/hda4
1 3 68 1 active sync /dev/.static/dev/hdb4
2 33 4 2 active sync /dev/.static/dev/hde4
3 0 0 - removed
--------------------------------------------------
and in /proc/mdstat is says:
--------------------------------------------------
md3 : active raid5 hde4[2] hdb4[1] hda4[0]
228151872 blocks level 5, 64k chunk, algorithm 2 [4/3] [UUU_]
[=====>...............] resync = 25.5% (19442692/76050624) finish=101.0min speed=9332K/sec--------------------------------------------------
Removing all the drives seem leads to the array resyncing with 0 drives in a loop, causing the /var/log/messages to grow at a rate of > 100MByte/sec :(
Isn't it a bug that the array starts resyning whithout checking if the drives needed for a synced state are present?
Bart
Quoted 4 lines · On Thursday June 23, [email protected] wrote:
Hi,
I have the problem that my RAID5 (created with 4 drives) array is resyning despite the fact I removed one drive. Any Idea what it is doing?
Sound's familar. Thought I had fixed it. What kernel are you running?
NeilBrown
Hi Neil,
Quoted 6 lines
Quoted 2 lines
I have the problem that my RAID5 (created with 4 drives) array is resyning despite the fact I removed one drive. Any Idea what it is doing?
Sound's familar. Thought I had fixed it. What kernel are you running?
I'm running on a 2.6.11 kernel, it should be pretty up to date.
I forgot to mention that it only occurs if the drive is removed from the RAID5 set before it is 'synced' for the first time.
Could it be the mddev->curr_resync or mddev->recovery_cp are handled wrong in this case? I saw the prints:
..
Jun 23 12:26:10 172 kernel: md: checkpointing recovery of md3. .. Jun 23 12:26:15 172 kernel: md: resuming recovery of md3 from checkpoint.
..
in the error situation, while there is no state to recover at all at that point (the array is incomplete/degraded).
Cheers,
Bart
Quoted 9 lines · On Thursday June 23, [email protected] wrote:
Hi Neil,
Quoted 6 lines
Quoted 2 lines
I have the problem that my RAID5 (created with 4 drives) array is resyning despite the fact I removed one drive. Any Idea what it is doing?
Sound's familar. Thought I had fixed it. What kernel are you running?
I'm running on a 2.6.11 kernel, it should be pretty up to date.
I cannot find the patch I was thinking of to check when it went in, but I have just tested various failure scenarios on 2.6.12-rc3-mm3 and it handles them all properly.
If you could try 2.6.12 and confirm, I would appreciate it.
Quoted 7 lines
I forgot to mention that it only occurs if the drive is removed from the RAID5 set before it is 'synced' for the first time.
Could it be the
mddev->curr_resyncormddev->recovery_cpare handled wrong in this case? I saw the prints:
Quoted 8 lines
..
Jun 23 12:26:10 172 kernel: md: checkpointing recovery of md3. .. Jun 23 12:26:15 172 kernel: md: resuming recovery of md3 from checkpoint...
in the error situation, while there is no state to recover at all at that point (the array is incomplete/degraded).
Yes, that shouldn't happen if there is an error, and it doesn't for me.
I noticed that the raid5 was resyncing rather than recovering. Normally when you create a raid5 with mdadm it will recover as this is faster than resync. Did you create the array with '-f' ??
NeilBrown
Hi Neil,
Quoted 6 lines
I cannot find the patch I was thinking of to check when it went in, but I have just tested various failure scenarios on 2.6.12-rc3-mm3 and it handles them all properly.
If you could try 2.6.12 and confirm, I would appreciate it.
Ok, will try with 2.6.12
Quoted 4 lines
I noticed that the raid5 was resyncing rather than recovering. Normally when you create a raid5 with mdadm it will recover as this is faster than resync. Did you create the array with '-f' ??
No, but the '--run' flag was set. I noticed that when ommiting the '--run' flag (like 'mdadm --create /dev/md3 --level=5 --raid-devices=4 /dev/hd[abef]4') the array goes into recovery straight away, so probably the '--run' flag forces a '-f'.
It also looks like the '--detail --test' flag of mdadm (to obtain the array status as return value) does not work (always returns 0). I'll start a different thread on that when I tested it further.
Cheers,
Bart