RAID5 write hole?

Hi:

Recently I heard of the so called "write hole" problem of raid5 in Linux software raid. I use ext4 filesystem on my NAS, which assembles data disks using Linux software raid. So I wonder how safe my such system!

If the "write hole" is inevitable, will it result in the corruption of ext4 filesystem?

Shaochun Wang <[email protected]>

Jabber: [email protected]

RAID is never a replacement for backups, corruption can happen at multiple levels in your system for different reasons. Non-ECC memory can have bit flips which corrupts your data, write hole can cause data corruption, etc.

Generally, unless you have really really high demands on data integrity, this is not a major problem.

Ext4 has other potential software/fs interactions when it comes to data integrity, in that it write buffers for quite some time, so even if you think your file is saved, it might take many seconds before it's actually on disk if your software doesn't fsync() it. Most don't, because Ext3 took so long to do it.

So generally, don't worry too much, but make sure you have backups for your important data.

Mikael Abrahamsson    email: [email protected]

Maybe Sun's ZFS is the ultimate choice!

Shaochun Wang <[email protected]>

Jabber: [email protected]

The write hole occurs if your system crashes during a write operation, where one stripe gets updated but the other corresponding stripe does not. This could lead to parity information not matching the corresponding data.

If the raid 5 system atleast ensures that the data stripe is always written before parity, then the montly resync check that mdadm does should be able to detect this and write new parity information.

Atleast this way the bad parity does not lurk around forever on your raid system causing numerous problems when a disk finally fails.

The write hole is not inevitable, but would require some special measures at the raid level which could affect performance. And as with any corruption, it could definitely corrupt your filesystem.

--John

Correct.

This bit isn't so correct.
When the RAID5 is next assembled after the crash, if all devices are present (i.e. the array is not degraded) then it will check and correct all the parity blocks immediately. If you have a write-intent-bitmap configured, this will be quite quick. If not it could take hours.

Once the resync has completed you are safe again, any risk from the "write hole" will have disappeared.

If your array was degraded when the system crashed, or is degraded on restart, or degrades before the resync completes, then you could suffer from the "Write hole" ... if a write was interrupted by the crash.

In the first two cases (which are effectively the same case), mdadm will refuse to assemble the array because it knows it could be suffering from a write-hole problem. You need to reassemble with "--force" which means you acknowledge that there could be corruption due to the write hole.

If you lose a device during the resync you could still suffer from the write hole, but md doesn't alert you to this. That could be seen as a short-coming, but I'm not sure how it might be fixed. I wouldn't want the array to suddenly stop working because there is suddenly a risk of write-hold based corruption....

Yes, it certainly does not lurk forever - the resync fixes it.

The write hole can be "fixed" in two ways that I am aware of.
1/ log all writes (including parity updates) to some stable storage before
writing them to the RAID5. This is typically done in "hardware RAID" cards using NVRAM for the stable storage.
Once NVRAM is widely available on commodity server hardware I suspect md/raid5 will be enhanced to support this. I have thought about doing this using a RAID1 as the alternate stable storage, but the performance cost is unlikely to acceptable.
2/ use a filesystem which understands the layout of the RAID5 and which
somehow "knows" which stripes were written "recently" so that it can invalidate them (if it cannot verify them) after a crash. This would almost certainly require a copy-on-write disciple in the filesystem.

NeilBrown

How do I know whether my RAID5 has wirte-intent bitmap enabled?

Shaochun Wang <[email protected]>

Jabber: [email protected]

$ cat /proc/mdstat | grep bitmap
      bitmap: 0/8 pages [0KB], 131072KB chunk

If you don't get any bitmap information in there, it's not enabled.

Mikael Abrahamsson    email: [email protected]

It seems that I do not have write-intent bitmap enabled. If write-intent bitmap is useful, why is it not the default one?

Shaochun Wang <[email protected]>

Jabber: [email protected]