Skip to contentSkip to conversation

RAID6 write I/O amplification?

DISCUSSION Roman Mamedov 4 participants 4 messages Feb 23, 2015 – Feb 26, 2015 MD RAID

lore
Share & tools
lore
#0 lore raw

Hello,

Got a bit of a "how does it actually work" question...

Suppose I have an MD RAID6 of 8 drives, with 64KB chunk size.

I am rewriting a 4KB filesystem sector somewhere on that RAID (not crossing the stripe boundary).

What's the amount of disk I/O in total this will result in?

I assume the RAID will need to read data from all drives, recompute parity, then write to the data stripe where the updated piece happened to be, and also write to two parity stripes.

Is this done at a stripe granularity, so 6x64KB reads, 3x64KB writes? Or down to individual sectors (pages), i.e. 6x4KB reads, 3x4KB writes? Or am I describing this algorithm correctly at all?

Signature · 2 lines
With respect,
Roman
#1 lore raw
Quoted 16 lines

Von: [email protected] [[email protected]]" im Auftrag von "Roman Mamedov [[email protected]]
Gesendet: Dienstag, 24. Februar 2015 00:58
An: [email protected]
Betreff: RAID6 write I/O amplification?

Hello,

Got a bit of a "how does it actually work" question...

Suppose I have an MD RAID6 of 8 drives, with 64KB chunk size.

I am rewriting a 4KB filesystem sector somewhere on that RAID (not crossing the stripe boundary).

What's the amount of disk I/O in total this will result in?

Quoted 7 lines

I assume the RAID will need to read data from all drives, recompute parity, then write to the data stripe where the updated piece happened to be, and also write to two parity stripes.

Is this done at a stripe granularity, so 6x64KB reads, 3x64KB writes? Or down to individual sectors (pages), i.e. 6x4KB reads, 3x4KB writes? Or am I describing this algorithm correctly at all?

Implementation will work on "internal" stripe granularity and that is 4K So your case will be 6x4KB read + 3x4KB write. That said, you can only reduce the I/O overhead by writing data that is larger than your configured stripe size (e.g. 64K).

Looking at Neils development GIT you will find patches that allow read-modify-write cycles for RAID6. So we only need the old block, the old parities, recaluclate them and write the new block and the new parities. In your case that would reduce the I/Os to 3x4KB read + 3x4KB write. See http://git.neil.brown.name/?p=md.git;a=shortlog;h=refs/heads/devel

I posted them 6 months ago but they did not made their way into the stable tree. Additionally it conatins patches to batch adjacent writes to be processed in less & larger I/Os. Currently Linux Raid will break each operation into 4K I/Os.

Markus

****************************************************************************
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet.

Über das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung.

Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln

Vorstand:
Kadir Akin
Dr. Michael Höhnerbach

Vorsitzender des Aufsichtsrates:
Hans Kristian Langva

Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497

This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention.

Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln

executive board:
Kadir Akin
Dr. Michael Höhnerbach

President of the supervisory board:
Hans Kristian Langva

Registry office: district court Cologne
Register number: HRB 52 497

****************************************************************************

#2 lore raw
Quoted 20 lines · Markus Stockhausen
Quoted 20 lines

Von: [email protected] [[email protected]]" im Auftrag von "Roman Mamedov [[email protected]]
Gesendet: Dienstag, 24. Februar 2015 00:58
An: [email protected]
Betreff: RAID6 write I/O amplification?

Hello,

Got a bit of a "how does it actually work" question...

Suppose I have an MD RAID6 of 8 drives, with 64KB chunk size.

I am rewriting a 4KB filesystem sector somewhere on that RAID (not crossing the stripe boundary).

What's the amount of disk I/O in total this will result in?

I assume the RAID will need to read data from all drives, recompute parity, then write to the data stripe where the updated piece happened to be, and also write to two parity stripes.

Quoted 6 lines
Quoted 3 lines

Is this done at a stripe granularity, so 6x64KB reads, 3x64KB writes? Or down to individual sectors (pages), i.e. 6x4KB reads, 3x4KB writes? Or am I describing this algorithm correctly at all?

Implementation will work on "internal" stripe granularity and that is 4K So your case will be 6x4KB read + 3x4KB write.

Having said that, does it mean that following description of "chunk size" is wrong:
'[chunk size] is the smallest "atomic" mass of data that can be written to the devices'
since in this case chunk size is 64KB but 4KB is written atomically (?). I have find it in the kernel.org wiki page [1]

---
Alireza
1. https://raid.wiki.kernel.org/index.php/RAID_setup

#3 lore raw
Quoted 29 lines · On Wed, 25 Feb 2015 18:40:46 -0600 Alireza Haghdoost <[email protected]> wrote:
Quoted 20 lines · Markus Stockhausen
Quoted 20 lines

Von: [email protected] [[email protected]]&quot; im Auftrag von &quot;Roman Mamedov [[email protected]]
Gesendet: Dienstag, 24. Februar 2015 00:58
An: [email protected]
Betreff: RAID6 write I/O amplification?

Hello,

Got a bit of a "how does it actually work" question...

Suppose I have an MD RAID6 of 8 drives, with 64KB chunk size.

I am rewriting a 4KB filesystem sector somewhere on that RAID (not crossing the stripe boundary).

What's the amount of disk I/O in total this will result in?

I assume the RAID will need to read data from all drives, recompute parity, then write to the data stripe where the updated piece happened to be, and also write to two parity stripes.

Quoted 6 lines
Quoted 3 lines

Is this done at a stripe granularity, so 6x64KB reads, 3x64KB writes? Or down to individual sectors (pages), i.e. 6x4KB reads, 3x4KB writes? Or am I describing this algorithm correctly at all?

Implementation will work on "internal" stripe granularity and that is 4K So your case will be 6x4KB read + 3x4KB write.

Quoted 6 lines

Having said that, does it mean that following description of "chunk size" is wrong:
'[chunk size] is the smallest "atomic" mass of data that can be written to the devices'
since in this case chunk size is 64KB but 4KB is written atomically (?). I have find it in the kernel.org wiki page [1]

I think that when it says "atomic" it means in space, not time.

i.e. one (properly aligned) chunk of data will not be split up and written to different devices, it will all be written to one device. If you write more than a chunk, it will be split up and parts of if written to different devices.

You can still write less than a chunk.

So the intent is correct I think, but the word "atomic" doesn't really convey the right meaning. Probably it should be re-written to avoid that term and just spell out what is happening.

NeilBrown