Skip to contentSkip to conversation

Fast RAID 1 Resync

DISCUSSION Philip Cameron 2 participants 2 messages Jan 3, 2003 – Jan 16, 2003 MD RAID

lore
Share & tools
lore
#0 lore raw

Hi Neil,

(Sorry if this is a repost. I had an error with returned mail)

Thanks for your comments.

I also don't see a need to synchronize disks at mkraid time. Its nice to have identical disks but not necessary as long as the result of reading a sector that has never been written is undefined. An option to do either approach can be done as long as there is a real need. Adding an option increases complexity especially during test.

I have been thinking of tracking writes to each chunk with a counter. The counters would be organized into a vector indexed by chunk number. The counter is incremented by the number of mirrors including any currently unavailable mirrors before the write starts. It is decremeted as each write completes. So when all writes in a chunk are complete the counter returns to zero. If there is a missing mirror, the counter will not return to zero (since one of the needed writes was not done).

When a disk is pulled, the counters increment but don't return to zero. When the disk is reinserted, the resync needs to copy chunks where the counter doesn't go to zero. When the resync of the chunk is complete, set the counter to zero.

To deal with a recovery after crash, I am thinking about using your approach. Use a bit per counter and set the bit when the counter is non-zero. When a bit goes from 0 to 1, the updated bit vector is written before starting the write to the chunk. On reboot after a crash, the bit vector from the selected mirror is used (the current mechanism is used to select the base disk). The counter is incremented for each chunk that has a bit that is set. After this, the resync in the above case can be performed. I don't see a need for timestamps beyond what is currently being done.

The transitions from 1 to 0 are not all that important since the worst case is syncing a chunk that is already mirrored. Overall performance can be improved by delaying the 1 to 0 transition for a few seconds. A lazy write of the bits can be done every 10 seconds or so if there are no 0 to 1 changes during that interval.

I have a third goal: minimize the resync time for a new (replacement) disk. In this case all of the chunks that have ever been written need to be copied. I have been thinking about controlling this through a second bit vector. When a chunk is written for the first time the bit is set and it is never reset. When the new disk is added, the counters corresponding to all of the bits that have been set in the vector are incremented and a resync (as above) is performed. As above the vector update needs to be done before the write to the chunk. The length of resync is proportional to how much of the disk has been used.

A hardware note: the system has two IO assemblies each of which contains a PCI bus, SCSI HBA and 3 hot plugable SCSI disk slots. We are using 72GB disks. Sets of 2 mirror RAID1 raid sets is the most practical configuration.

Phil Cameron

Quoted 48 lines
Quoted 46 lines

Hi,

You have two quite different, though admittedly similar, goal here.
1/ quick resync when a recently removed drive is re-added. 2/ quick resync after an unclean shutdown.

I would treat these quite separately.

For the latter I would have a bitmap which was written to disk whenever a bit was set and eventually after a bit was cleared. I would use a 16 bit counter for each 'chunk'.
If the high bit is clear, it stores the number of outstanding writes on that chunk.
If the high bit is set, it stores some sort of time stamp of when the number of outstanding writes hit zero.
Every time you write the bitmap, you increment this timestamp. So when you schedule a write, you only need to write out the bitmap first if the 16bit number of this chunk has the highbit set and has a timestamp which is different to the current one - which means that the bitmap has been written out with a zero in this slot. So:
On write, if highbit clear, increment counter
if highbit set and timestamp matches, set counter to 1
and set bit in bitmap
if highbit set and timestamp doesn't match, set
bit in bitmap, schedule write, set counter to 1
On write complete,
decrement counter. If it hits zero, set to timestamp with high bit set, clear the bitmap bit, and schudle a bitmap writout a few seconds hence.

For the former I would just hold a separate bitmap, one bit per chunk.
While all drives are working, this bitmap would be all zeros. Whenever a write fails to write to all drives, the relevant bit gets set.
When a recently failed drive comes back online, we resync all chunks that have that bit set.

I don't see a particular need to sync the drives are device creation time, but I would like to keep the option of doing so. I don't really care which behaviour is the default.

NeilBrown

#1 lore raw

Neil,

It is very interesting but I have a couple of questions on your suggestion as fellow;

1) Is a bit on the bitmap match with each chunk?
2) it has a counter( or timestamp) for each chunk on each mirror, right? then how do we find the timestamp matched or not? what do you compare
it with?
3) I think "a bit on the bitmap has the same meaning as the MSB(high bit) on the counter".
That means you need only a counter for each chunk? Am I wrong?

Bo

----- Original Message -----
From: "Philip Cameron" <[email protected]>
To: <[email protected]>; <[email protected]>
Sent: Thursday, January 02, 2003 5:22 PM
Subject: Re: Fast RAID 1 Resync

Quoted 7 lines

Hi Neil,

(Sorry if this is a repost. I had an error with returned mail)

Thanks for your comments.

I also don't see a need to synchronize disks at mkraid time. Its nice to

have

Quoted 1 line

identical disks but not necessary as long as the result of reading a

sector that

Quoted 1 line

has never been written is undefined. An option to do either approach can

be done

Quoted 5 lines

as long as there is a real need. Adding an option increases complexity especially during test.

I have been thinking of tracking writes to each chunk with a counter. The counters would be organized into a vector indexed by chunk number. The

counter

Quoted 1 line

is incremented by the number of mirrors including any currently

unavailable

Quoted 1 line

mirrors before the write starts. It is decremeted as each write completes.

So

Quoted 1 line

when all writes in a chunk are complete the counter returns to zero. If

there is

Quoted 1 line

a missing mirror, the counter will not return to zero (since one of the

needed

Quoted 3 lines

writes was not done).

When a disk is pulled, the counters increment but don't return to zero.

When the

Quoted 1 line

disk is reinserted, the resync needs to copy chunks where the counter

doesn't go

Quoted 1 line

to zero. When the resync of the chunk is complete, set the counter to

zero.

Quoted 2 lines

To deal with a recovery after crash, I am thinking about using your

approach.

Quoted 1 line

Use a bit per counter and set the bit when the counter is non-zero. When a

bit

Quoted 1 line

goes from 0 to 1, the updated bit vector is written before starting the

write to

Quoted 1 line

the chunk. On reboot after a crash, the bit vector from the selected

mirror is

Quoted 1 line

used (the current mechanism is used to select the base disk). The counter

is

Quoted 1 line

incremented for each chunk that has a bit that is set. After this, the

resync in

Quoted 1 line

the above case can be performed. I don't see a need for timestamps beyond

what

Quoted 3 lines

is currently being done.

The transitions from 1 to 0 are not all that important since the worst

case is

Quoted 1 line

syncing a chunk that is already mirrored. Overall performance can be

improved by

Quoted 1 line

delaying the 1 to 0 transition for a few seconds. A lazy write of the bits

can

Quoted 1 line

be done every 10 seconds or so if there are no 0 to 1 changes during that

interval.

Quoted 2 lines

I have a third goal: minimize the resync time for a new (replacement)

disk. In

Quoted 1 line

this case all of the chunks that have ever been written need to be copied.

I

Quoted 1 line

have been thinking about controlling this through a second bit vector.

When a

Quoted 1 line

chunk is written for the first time the bit is set and it is never reset.

When

Quoted 1 line

the new disk is added, the counters corresponding to all of the bits that

have

Quoted 1 line

been set in the vector are incremented and a resync (as above) is

performed. As

Quoted 1 line

above the vector update needs to be done before the write to the chunk.

The

Quoted 3 lines

length of resync is proportional to how much of the disk has been used.

A hardware note: the system has two IO assemblies each of which contains a

PCI

Quoted 1 line

bus, SCSI HBA and 3 hot plugable SCSI disk slots. We are using 72GB disks.

Sets

Quoted 58 lines

of 2 mirror RAID1 raid sets is the most practical configuration.

Phil Cameron

Quoted 41 lines
Quoted 41 lines

Hi,

You have two quite different, though admittedly similar, goal here.
1/ quick resync when a recently removed drive is re-added. 2/ quick resync after an unclean shutdown.

I would treat these quite separately.

For the latter I would have a bitmap which was written to disk whenever a bit was set and eventually after a bit was cleared. I would use a 16 bit counter for each 'chunk'.
If the high bit is clear, it stores the number of outstanding writes on that chunk.
If the high bit is set, it stores some sort of time stamp of when the number of outstanding writes hit zero.
Every time you write the bitmap, you increment this timestamp. So when you schedule a write, you only need to write out the bitmap first if the 16bit number of this chunk has the highbit set and has a timestamp which is different to the current one - which means that the bitmap has been written out with a zero in this slot. So:
On write, if highbit clear, increment counter
if highbit set and timestamp matches, set counter to 1
and set bit in bitmap
if highbit set and timestamp doesn't match, set
bit in bitmap, schedule write, set counter to 1
On write complete,
decrement counter. If it hits zero, set to timestamp with
high bit set, clear the bitmap bit, and schudle a bitmap
writout a few seconds hence.

For the former I would just hold a separate bitmap, one bit per chunk.
While all drives are working, this bitmap would be all zeros. Whenever a write fails to write to all drives, the relevant bit gets set.
When a recently failed drive comes back online, we resync all chunks that have that bit set.

Quoted 7 lines
Quoted 5 lines

I don't see a particular need to sync the drives are device creation time, but I would like to keep the option of doing so. I don't really care which behaviour is the default.

NeilBrown

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html