Fast RAID 1 Resync
Hi Neil,
(Sorry if this is a repost. I had an error with returned mail)
Thanks for your comments.
I also don't see a need to synchronize disks at mkraid time. Its nice to have identical disks but not necessary as long as the result of reading a sector that has never been written is undefined. An option to do either approach can be done as long as there is a real need. Adding an option increases complexity especially during test.
I have been thinking of tracking writes to each chunk with a counter. The counters would be organized into a vector indexed by chunk number. The counter is incremented by the number of mirrors including any currently unavailable mirrors before the write starts. It is decremeted as each write completes. So when all writes in a chunk are complete the counter returns to zero. If there is a missing mirror, the counter will not return to zero (since one of the needed writes was not done).
When a disk is pulled, the counters increment but don't return to zero. When the disk is reinserted, the resync needs to copy chunks where the counter doesn't go to zero. When the resync of the chunk is complete, set the counter to zero.
To deal with a recovery after crash, I am thinking about using your approach. Use a bit per counter and set the bit when the counter is non-zero. When a bit goes from 0 to 1, the updated bit vector is written before starting the write to the chunk. On reboot after a crash, the bit vector from the selected mirror is used (the current mechanism is used to select the base disk). The counter is incremented for each chunk that has a bit that is set. After this, the resync in the above case can be performed. I don't see a need for timestamps beyond what is currently being done.
The transitions from 1 to 0 are not all that important since the worst case is syncing a chunk that is already mirrored. Overall performance can be improved by delaying the 1 to 0 transition for a few seconds. A lazy write of the bits can be done every 10 seconds or so if there are no 0 to 1 changes during that interval.
I have a third goal: minimize the resync time for a new (replacement) disk. In this case all of the chunks that have ever been written need to be copied. I have been thinking about controlling this through a second bit vector. When a chunk is written for the first time the bit is set and it is never reset. When the new disk is added, the counters corresponding to all of the bits that have been set in the vector are incremented and a resync (as above) is performed. As above the vector update needs to be done before the write to the chunk. The length of resync is proportional to how much of the disk has been used.
A hardware note: the system has two IO assemblies each of which contains a PCI bus, SCSI HBA and 3 hot plugable SCSI disk slots. We are using 72GB disks. Sets of 2 mirror RAID1 raid sets is the most practical configuration.
Phil Cameron
Quoted 48 lines
Quoted 46 lines
Hi,
You have two quite different, though admittedly similar, goal here.
1/ quick resync when a recently removed drive is re-added. 2/ quick resync after an unclean shutdown.I would treat these quite separately.
For the latter I would have a bitmap which was written to disk whenever a bit was set and eventually after a bit was cleared. I would use a 16 bit counter for each 'chunk'.
If the high bit is clear, it stores the number of outstanding writes on that chunk.
If the high bit is set, it stores some sort of time stamp of when the number of outstanding writes hit zero.
Every time you write the bitmap, you increment this timestamp. So when you schedule a write, you only need to write out the bitmap first if the 16bit number of this chunk has the highbit set and has a timestamp which is different to the current one - which means that the bitmap has been written out with a zero in this slot. So:
On write, if highbit clear, increment counter
if highbit set and timestamp matches, set counter to 1
and set bit in bitmap
if highbit set and timestamp doesn't match, set
bit in bitmap, schedule write, set counter to 1
On write complete,
decrement counter. If it hits zero, set to timestamp with high bit set, clear the bitmap bit, and schudle a bitmap writout a few seconds hence.For the former I would just hold a separate bitmap, one bit per chunk.
While all drives are working, this bitmap would be all zeros. Whenever a write fails to write to all drives, the relevant bit gets set.
When a recently failed drive comes back online, we resync all chunks that have that bit set.I don't see a particular need to sync the drives are device creation time, but I would like to keep the option of doing so. I don't really care which behaviour is the default.
NeilBrown
Neil,
It is very interesting but I have a couple of questions on your suggestion as fellow;
1) Is a bit on the bitmap match with each chunk?
2) it has a counter( or timestamp) for each chunk on each mirror, right? then how do we find the timestamp matched or not? what do you compare
it with?
3) I think "a bit on the bitmap has the same meaning as the MSB(high bit) on the counter".
That means you need only a counter for each chunk? Am I wrong?
Bo
----- Original Message -----
From: "Philip Cameron" <[email protected]>
To: <[email protected]>; <[email protected]>
Sent: Thursday, January 02, 2003 5:22 PM
Subject: Re: Fast RAID 1 Resync
Quoted 7 lines
Hi Neil,
(Sorry if this is a repost. I had an error with returned mail)
Thanks for your comments.
I also don't see a need to synchronize disks at mkraid time. Its nice to
have
Quoted 1 line
identical disks but not necessary as long as the result of reading a
sector that
Quoted 1 line
has never been written is undefined. An option to do either approach can
be done
Quoted 5 lines
as long as there is a real need. Adding an option increases complexity especially during test.
I have been thinking of tracking writes to each chunk with a counter. The counters would be organized into a vector indexed by chunk number. The
counter
Quoted 1 line
is incremented by the number of mirrors including any currently
unavailable
Quoted 1 line
mirrors before the write starts. It is decremeted as each write completes.
So
Quoted 1 line
when all writes in a chunk are complete the counter returns to zero. If
there is
Quoted 1 line
a missing mirror, the counter will not return to zero (since one of the
needed
Quoted 3 lines
writes was not done).
When a disk is pulled, the counters increment but don't return to zero.
When the
Quoted 1 line
disk is reinserted, the resync needs to copy chunks where the counter
doesn't go
Quoted 1 line
to zero. When the resync of the chunk is complete, set the counter to
zero.
Quoted 2 lines
To deal with a recovery after crash, I am thinking about using your
approach.
Quoted 1 line
Use a bit per counter and set the bit when the counter is non-zero. When a
bit
Quoted 1 line
goes from 0 to 1, the updated bit vector is written before starting the
write to
Quoted 1 line
the chunk. On reboot after a crash, the bit vector from the selected
mirror is
Quoted 1 line
used (the current mechanism is used to select the base disk). The counter
is
Quoted 1 line
incremented for each chunk that has a bit that is set. After this, the
resync in
Quoted 1 line
the above case can be performed. I don't see a need for timestamps beyond
what
Quoted 3 lines
is currently being done.
The transitions from 1 to 0 are not all that important since the worst
case is
Quoted 1 line
syncing a chunk that is already mirrored. Overall performance can be
improved by
Quoted 1 line
delaying the 1 to 0 transition for a few seconds. A lazy write of the bits
can
Quoted 1 line
be done every 10 seconds or so if there are no 0 to 1 changes during that
interval.
Quoted 2 lines
I have a third goal: minimize the resync time for a new (replacement)
disk. In
Quoted 1 line
this case all of the chunks that have ever been written need to be copied.
I
Quoted 1 line
have been thinking about controlling this through a second bit vector.
When a
Quoted 1 line
chunk is written for the first time the bit is set and it is never reset.
When
Quoted 1 line
the new disk is added, the counters corresponding to all of the bits that
have
Quoted 1 line
been set in the vector are incremented and a resync (as above) is
performed. As
Quoted 1 line
above the vector update needs to be done before the write to the chunk.
The
Quoted 3 lines
length of resync is proportional to how much of the disk has been used.
A hardware note: the system has two IO assemblies each of which contains a
PCI
Quoted 1 line
bus, SCSI HBA and 3 hot plugable SCSI disk slots. We are using 72GB disks.
Sets
Quoted 58 lines
of 2 mirror RAID1 raid sets is the most practical configuration.
Phil Cameron
Quoted 41 lines
Quoted 41 lines
Hi,
You have two quite different, though admittedly similar, goal here.
1/ quick resync when a recently removed drive is re-added. 2/ quick resync after an unclean shutdown.I would treat these quite separately.
For the latter I would have a bitmap which was written to disk whenever a bit was set and eventually after a bit was cleared. I would use a 16 bit counter for each 'chunk'.
If the high bit is clear, it stores the number of outstanding writes on that chunk.
If the high bit is set, it stores some sort of time stamp of when the number of outstanding writes hit zero.
Every time you write the bitmap, you increment this timestamp. So when you schedule a write, you only need to write out the bitmap first if the 16bit number of this chunk has the highbit set and has a timestamp which is different to the current one - which means that the bitmap has been written out with a zero in this slot. So:
On write, if highbit clear, increment counter
if highbit set and timestamp matches, set counter to 1
and set bit in bitmap
if highbit set and timestamp doesn't match, set
bit in bitmap, schedule write, set counter to 1
On write complete,
decrement counter. If it hits zero, set to timestamp with
high bit set, clear the bitmap bit, and schudle a bitmap
writout a few seconds hence.For the former I would just hold a separate bitmap, one bit per chunk.
While all drives are working, this bitmap would be all zeros. Whenever a write fails to write to all drives, the relevant bit gets set.
When a recently failed drive comes back online, we resync all chunks that have that bit set.Quoted 7 lines
Quoted 5 lines
I don't see a particular need to sync the drives are device creation time, but I would like to keep the option of doing so. I don't really care which behaviour is the default.
NeilBrown
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html