Skip to contentSkip to conversation

Avoiding BRTFS RAID5 write hole

DISCUSSION Hubert Tonneau 3 participants 7 messages Nov 13, 2019 – Nov 28, 2019 Btrfs file system

lore
Share & tools
lore
#0 lore raw
Quoted 3 lines · Goffredo Baroncelli
Quoted 2 lines

What I am suggesting is to write it as RAID1 instead of RAID5, so that if it's changed a lot of times, you pay only once.

I am not sure to understand what are you saying. Could you elaborate ?

The safety problem with RAID5 is that between the time you start to overwrite a stripe and the time you finish, disk safety is disabled because parity is broken. On the other hand, with RAID1, disk safety more or less remains all the time, so overwriting is no issue.

There are several possible strategies to keep RAID5 disk safety all the time:

1) Use a journal
This is the MDADM solution because it's the only resonable one if the RAID layer is separated from the filesystem (because you don't whan to add another sectors mapping layer). The problem is that it's IO expensive.
This is the solution implemented in Liu Bo 2017 patch, as far as I can understand it.

2) Never overwrite the RAID5 stripe
This is stripe COW. The new stripe is stored at different disks positions. The problem is that it's even more IO expensive.
This is the solution you are suggesting, as far as I can understand it.

What I'm suggesting is to use your COW solution, but also write the new (set of) stripe(s) as RAID1.
Let me call this operation stripe COW RAID5 to RAID1.
The key advantage is that if you have to overwrite it again a few seconds (or hours) later, then it can be fast, because it's already RAID1.

Morever, new stripes resulting from writing a new file, or appending, would be created as RAID1, even if the filesystem DATA is configured as RAID5, each time the stripe is not full or is likely to be modified soon. This will reduce the number of stripe COW RAID5 to RAID1 operations.

The final objective is to have few stripe COW operations, because they are IO expensive, and many RAID1 stripe overwrite operations.
The price to pay for the reduced number of stripe COW operations is consuming more disk space, because RAID1 stripes consumes more disk space than RAID5 ones, and that is why we would have a background process that does stripe COW from RAID1 to RAID5 in order to reclaim disk space, and we could make it more aggressive when we lack disk space.

What I'm trying to provide is the idea that seeing the DATA as RAID1 or RAID5 is not a good idea when we have BTRFS flexibility. We should rather see it as RAID1 and RAID5, RAID5 beeing just a way to reclaim disk space (same for RAID1C3 and RAID6). Having METADATA as RAID1 and DATA as RAID5 was a first step, but BTRFS flexibility probably allows to do more.

Please notice that I understand the BTRFS and RAID principles, but on the other hand, I have not read the code, so can hardly say what is easy to implement. Sorry about that. I've written a full new operating system (see www.fullpliant.org) but the kernel :-)

#1 lore raw

First of all, I am just a regular and a BTRFS enthusiast with no proper filesystem knowledge.

regarding the write hole.... I was just pondering (and I may be totally wrong about this , but it is worth a shot)

If raid5/6 needs to read-modify-write - would not the write hole be avoided if you first log a XOR cipher value in the metadata, then you modify an already existing stripe by XOR'ing whatever needs to be modified on the rewriten stripe. The journal would only need to know what stripes are being modified so it can be checked on a mount.

If you encounter the write hole, the parity would not match and since the XOR cipher is in the metadata you can roll back any failed update byte by byte until the checksum match and you will be good to go with the old data instead of the new one.

If on the other hand you can write a new stripe the problem goes away. I personally are willing to have increased disk IO and reduced performance for space. After all raid5/6 is not performance oriented , but primarily a space saver.

once (if ever) BTRFS supports per subvolume raid levels then the performance issues goes away as you can always raid1/0 some subvolume if you need to sacrifice space for performance.

- Waxhead

Quoted 36 lines · Hubert Tonneau
Quoted 3 lines · Goffredo Baroncelli
Quoted 2 lines

What I am suggesting is to write it as RAID1 instead of RAID5, so that if it's changed a lot of times, you pay only once.

I am not sure to understand what are you saying. Could you elaborate ?

The safety problem with RAID5 is that between the time you start to overwrite a stripe and the time you finish, disk safety is disabled because parity is broken. On the other hand, with RAID1, disk safety more or less remains all the time, so overwriting is no issue.

There are several possible strategies to keep RAID5 disk safety all the time:

1) Use a journal
This is the MDADM solution because it's the only resonable one if the RAID layer is separated from the filesystem (because you don't whan to add another sectors mapping layer). The problem is that it's IO expensive.
This is the solution implemented in Liu Bo 2017 patch, as far as I can understand it.

2) Never overwrite the RAID5 stripe
This is stripe COW. The new stripe is stored at different disks positions. The problem is that it's even more IO expensive.
This is the solution you are suggesting, as far as I can understand it.

What I'm suggesting is to use your COW solution, but also write the new (set of) stripe(s) as RAID1.
Let me call this operation stripe COW RAID5 to RAID1.
The key advantage is that if you have to overwrite it again a few seconds (or hours) later, then it can be fast, because it's already RAID1.

Morever, new stripes resulting from writing a new file, or appending, would be created as RAID1, even if the filesystem DATA is configured as RAID5, each time the stripe is not full or is likely to be modified soon. This will reduce the number of stripe COW RAID5 to RAID1 operations.

The final objective is to have few stripe COW operations, because they are IO expensive, and many RAID1 stripe overwrite operations.
The price to pay for the reduced number of stripe COW operations is consuming more disk space, because RAID1 stripes consumes more disk space than RAID5 ones, and that is why we would have a background process that does stripe COW from RAID1 to RAID5 in order to reclaim disk space, and we could make it more aggressive when we lack disk space.

What I'm trying to provide is the idea that seeing the DATA as RAID1 or RAID5 is not a good idea when we have BTRFS flexibility. We should rather see it as RAID1 and RAID5, RAID5 beeing just a way to reclaim disk space (same for RAID1C3 and RAID6). Having METADATA as RAID1 and DATA as RAID5 was a first step, but BTRFS flexibility probably allows to do more.

Please notice that I understand the BTRFS and RAID principles, but on the other hand, I have not read the code, so can hardly say what is easy to implement. Sorry about that. I've written a full new operating system (see www.fullpliant.org) but the kernel :-)

#2 lore raw
In reply to Hubert Tonneau
Quoted 10 lines · Hubert Tonneau
Quoted 3 lines · Goffredo Baroncelli
Quoted 2 lines

What I am suggesting is to write it as RAID1 instead of RAID5, so that if it's changed a lot of times, you pay only once.

I am not sure to understand what are you saying. Could you elaborate ?

The safety problem with RAID5 is that between the time you start to overwrite a stripe and the time you finish, disk safety is disabled because parity is broken. On the other hand, with RAID1, disk safety more or less remains all the time, so overwriting is no issue.

There are several possible strategies to keep RAID5 disk safety all the time:

Quoted 8 lines

1) Use a journal
This is the MDADM solution because it's the only resonable one if the RAID layer is separated from the filesystem (because you don't whan to add another sectors mapping layer). The problem is that it's IO expensive.
This is the solution implemented in Liu Bo 2017 patch, as far as I can understand it.

2) Never overwrite the RAID5 stripe
This is stripe COW. The new stripe is stored at different disks positions. The problem is that it's even more IO expensive.

Why do you think that this approach is more IO expensive ?

Supposing to have n+1 disks, configured as raid5 Raid5

The problem is only when you have to update a portion of stripe. So I consider of an amount of data in the range 0..n -> n/2 in average (I omit the size of the disk-stripe which is constant)

1) stripe update in place (current BTRFS behavior)
The data to write is ~ n/2 +1 (in average half of the stripe size + parity)

2) COW stripe (my idea)
The data to write is ~ n + 1 (the full stripe size + parity)

3) Your solution (cache in RAID1)
The data to write is
a) write data to raid1: ~ n/2 * 2 = n (data written on two disks)
b) update the RAID5 stripe: ~ n/2 + 1

Total: ~ 3 * n/2 + 1

(for the case 2 and 3 I don't consider the metadata update because it is several order of magnitude lower)

Quoted 5 lines

This is the solution you are suggesting, as far as I can understand it.

What I'm suggesting is to use your COW solution, but also write the new (set of) stripe(s) as RAID1.
Let me call this operation stripe COW RAID5 to RAID1.
The key advantage is that if you have to overwrite it again a few seconds (or hours) later, then it can be fast, because it's already RAID1.

On the basis of my simulation, I can't agree: COW-ing a stripe requires to write the full stripe; instead if you want to write the data in a RAID1 before updating the stripe, in average first you have to write an amount of data equal to 'n', then you have to update the raid5....

You can see this in another way: raid5 is more space friendly than raid1, this means that in raid5 you have to write less data....

Quoted 13 lines

Morever, new stripes resulting from writing a new file, or appending, would be created as RAID1, even if the filesystem DATA is configured as RAID5, each time the stripe is not full or is likely to be modified soon. This will reduce the number of stripe COW RAID5 to RAID1 operations.

The final objective is to have few stripe COW operations, because they are IO expensive, and many RAID1 stripe overwrite operations.
The price to pay for the reduced number of stripe COW operations is consuming more disk space, because RAID1 stripes consumes more disk space than RAID5 ones, and that is why we would have a background process that does stripe COW from RAID1 to RAID5 in order to reclaim disk space, and we could make it more aggressive when we lack disk space.

What I'm trying to provide is the idea that seeing the DATA as RAID1 or RAID5 is not a good idea when we have BTRFS flexibility. We should rather see it as RAID1 and RAID5, RAID5 beeing just a way to reclaim disk space (same for RAID1C3 and RAID6). Having METADATA as RAID1 and DATA as RAID5 was a first step, but BTRFS flexibility probably allows to do more.

Please notice that I understand the BTRFS and RAID principles, but on the other hand, I have not read the code, so can hardly say what is easy to implement. Sorry about that. I've written a full new operating system (see www.fullpliant.org) but the kernel :-)

Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
#3 lore raw
Quoted 6 lines · Goffredo Baroncelli

Why do you think that this approach is more IO expensive ?

Supposing to have n+1 disks, configured as raid5 Raid5

The problem is only when you have to update a portion of stripe. So I consider of an amount of data in the range 0..n -> n/2 in average (I omit the size of the disk-stripe which is constant)

Your approche is way too optimistic.
If the application is doing random writes on a nocow file, or doing append sync append sync append sync on a log, then it will rather be something like n/10 (10 is arbitrary number, but it will be much more than 2)

Quoted 2 lines

1) stripe update in place (current BTRFS behavior)
The data to write is ~ n/2 +1 (in average half of the stripe size + parity)

You have forgotten that you have to do a read of the parity before being abble to write, and this is what is going to take the server to a crawl if it does not write full RAID5 stripes.

Quoted 2 lines

2) COW stripe (my idea)
The data to write is ~ n + 1 (the full stripe size + parity)

Same, you have to do read before write, in order to get the unchanged part of the stripe.

Quoted 6 lines

3) Your solution (cache in RAID1)
The data to write is
a) write data to raid1: ~ n/2 * 2 = n (data written on two disks)
b) update the RAID5 stripe: ~ n/2 + 1

Total: ~ 3 * n/2 + 1

Your count shows that I did not express properly what I'm suggesting. Sorry about that.
What I'm suggesting, with something closer to your wording, is
a) copy the RAID5 stripe to a new RAID1 one at a different place: n * 2 (and need to read before write)
b) in place overwrite the new RAID1 stripe: n/2 * 2
Of course, you could optimize and do a) and b) in one step, so the overall cost would be n * 2 instead of n * 3
What is important is that what I'm suggesting is to NEVER update in place a RAID5 stripe. If you need to update a RAID5 stripe, first turn it to some RAID stripes at a different place of the disks. This is what a) does. Also, since all active parts are already RAID1, a) is a fairly rare operation.

---

Let me get back to the basic:
With RAID5, if you want good performances and data safety, you have to do only full new stripes writes (no in place overwrite), because it avoids the read before write need, and the write hole. It means that you need variable size stripes. This is ZFS way, as far as I understand it, and it brings a lot of complexity and makes it harder to deal with fragmentation, so this is what we would like to avoid it for BTRFS.

On the other hand, with RAID1, you can update in place, so everything is simple and fast.
This is why current BTRFS RAID1 is satisfying.
Only large file sequencial write on RAID5 is faster than RAID1, but we can assume that if people switch to RAID5, for all but very special use cases, it is not for faster sequencial write speed, but for more storage space available to the application.

As a summary, the only problem with current RAID1 BTRFS implementation, is that it provides a poor usable versus raw disk space ratio, just as any pure RAID1 implementation. So, I see RAID5 as just a way to improve this ratio (same for RAID6 and RAID1C3).
That is why you need a background process that will select some large files that don't have the nocow flag set, and convert (not in place) them from RAID1 to RAID5 in order to consume less disk space. The same process should convert (not in place) the file from RAID5 to RAID1 if the nocow flag has been set.
For any new files, use RAID1.
This is also why a) costy operation will be rare.
In the end, you get RAID1 performances and data safety, but a better usable versus raw disk space ratio, and this is just what most of us expect from the filesystem.

#4 lore raw
Quoted 10 lines · Hubert Tonneau
Quoted 6 lines · Goffredo Baroncelli

Why do you think that this approach is more IO expensive ?

Supposing to have n+1 disks, configured as raid5 Raid5

The problem is only when you have to update a portion of stripe. So I consider of an amount of data in the range 0..n -> n/2 in average (I omit the size of the disk-stripe which is constant)

Your approche is way too optimistic.
If the application is doing random writes on a nocow file, or doing append sync append sync append sync on a log, then it will rather be something like n/10 (10 is arbitrary number, but it will be much more than 2)

We need data to prove which is the best model: n/2 or n/10.... Otherwise we have only assumption. What it is true, is that in case of small repeated in same stripe writes your approach is better. In case the writes are all spread on the disks you have more writes.

Quoted 5 lines
Quoted 3 lines

1) stripe update in place (current BTRFS behavior)
The data to write is ~ n/2 +1 (in average half of the stripe size + parity)

You have forgotten that you have to do a read of the parity before being abble to write, and this is what is going to take the server to a crawl if it does not write full RAID5 stripes.

With exception of having a missing device, it is not needed to read the parity. However you have to read the *full* stripe (but this is true for every solution) to calculate the parity.

Quoted 11 lines
Quoted 3 lines

2) COW stripe (my idea)
The data to write is ~ n + 1 (the full stripe size + parity)

Same, you have to do read before write, in order to get the unchanged part of the stripe.

Quoted 5 lines

3) Your solution (cache in RAID1)
The data to write is
a) write data to raid1: ~ n/2 * 2 = n (data written on two disks)
b) update the RAID5 stripe: ~ n/2 + 1

Quoted 7 lines
Quoted 1 line

Total: ~ 3 * n/2 + 1

Your count shows that I did not express properly what I'm suggesting. Sorry about that.
What I'm suggesting, with something closer to your wording, is
a) copy the RAID5 stripe to a new RAID1 one at a different place: n * 2 (and need to read before write)
b) in place overwrite the new RAID1 stripe: n/2 * 2
Of course, you could optimize and do a) and b) in one step, so the overall cost would be n * 2 instead of n * 3

I wrote 3/2 *n not n*3... Pay attention that this includes also the updating of the RAID5...

Anyway I was wrong: you have to cache the full stripe, before updating the raid5 stripe. Why ? If you don't update the full stripe (so the parity is not aligned to the data) AND lose one disk (not involved int the update), you cannot rebuild the missing data.

So the corrected calculation is:
[...]
3) Your solution (cache in RAID1)
The data to write is
a) write data to raid1: ~ n * 2 (data written on two disks)
b) update the RAID5 stripe: ~ n/2 + 1

Total: ~ (2+1/2) * n + 1
[...]

Quoted 13 lines

What is important is that what I'm suggesting is to NEVER update in place a RAID5 stripe. If you need to update a RAID5 stripe, first turn it to some RAID stripes at a different place of the disks. This is what a) does. Also, since all active parts are already RAID1, a) is a fairly rare operation.

---

Let me get back to the basic:
With RAID5, if you want good performances and data safety, you have to do only full new stripes writes (no in place overwrite), because it avoids the read before write need, and the write hole. It means that you need variable size stripes. This is ZFS way, as far as I understand it, and it brings a lot of complexity and makes it harder to deal with fragmentation, so this is what we would like to avoid it for BTRFS.

On the other hand, with RAID1, you can update in place, so everything is simple and fast.
This is why current BTRFS RAID1 is satisfying.
Only large file sequencial write on RAID5 is faster than RAID1, but we can assume that if people switch to RAID5, for all but very special use cases, it is not for faster sequencial write speed, but for more storage space available to the application.

Quoted 7 lines

As a summary, the only problem with current RAID1 BTRFS implementation, is that it provides a poor usable versus raw disk space ratio, just as any pure RAID1 implementation. So, I see RAID5 as just a way to improve this ratio (same for RAID6 and RAID1C3).
That is why you need a background process that will select some large files that don't have the nocow flag set, and convert (not in place) them from RAID1 to RAID5 in order to consume less disk space. The same process should convert (not in place) the file from RAID5 to RAID1 if the nocow flag has been set.
For any new files, use RAID1.
This is also why a) costy operation will be rare.
In the end, you get RAID1 performances and data safety, but a better usable versus raw disk space ratio, and this is just what most of us expect from the filesystem.

If I understood correctly, ZFS acts like you describe: it has a cache called L2ARC, where the data are written on disks before updating the filesystem. You are suggesting to keep the "hot" data in a RAID1 block group, and write/update the RADID5 block group only when the data is "cold" in the RAID1 cache.

Quoted 1 line
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
#5 lore raw
Quoted 2 lines · Goffredo Baroncelli

You are suggesting to keep the "hot" data in a RAID1 block group, and write/update the RAID5 block group only when the data is "cold" in the RAID1 cache.

That's it.
Just remove the last word 'cache' because in some cases, the data might also remain as RAID1 forever, as an exemple if the nocow flag is set and the partition has enough free space.

Regards,
Hubert Tonneau

#6 lore raw
In reply to Goffredo Baroncelli
Quoted 2 lines · Goffredo Baroncelli

You are suggesting to keep the "hot" data in a RAID1 block group, and write/update the RADID5 block group only when the data is "cold" in the RAID1 cache.

Following this view, just adding a 'source data profile' (extent filter) and a 'target data profile' (overules filesystem data profile) option to 'btrfs filesystem defragment' utility would be a good starting point. It would allow, within a RAID1 btrfs filesystem, to manualy convert some subtree to RAID5, and increase free disk space as a result, so get the best of both worlds.