Skip to contentSkip to conversation

Possible RAID6 regression with ASYNC_TX_DMA enabled in 4.1

DISCUSSION Maxime Ripard 3 participants 7 messages May 7, 2015 – May 13, 2015 MD RAID

lore
Share & tools
lore
#0 lore raw

Hi,

I'm currently trying to add support for the PQ operations on the marvell XOR engine, in dmaengine, obviously to be able to use async_tx to offload these operations.

I'm testing these patches with a RAID6 array with 4 disks.

However, since the commit 59fc630b8b5f ("RAID5: batch adjacent full stripe write", every write to that array fails with the following stacktrace.

http://code.bulix.org/eh8iew-88342?raw

It seems to be generated by that warning here:

http://lxr.free-electrons.com/source/crypto/async_tx/async_tx.c#L173

And indeed, if we dump the status of depend_tx here, it's already been acked.

That doesn't happen if ASYNC_TX_DMA is disabled, hence using the software version of it, instead of relying on our XOR engine. It doesn't happen on any commit prior to the one mentionned above, with the exact same changes applied. These changes are meant to be contributed, so I can definitely push them somewhere if needed.

I don't really know where to look for though, the change that is causing this is probably the change in ops_run_reconstruct6, but I'm not sure that this partial revert alone would work with regard to the rest of the patch.

Maxime

Signature · 3 lines
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#1 lore raw

Hi Maxime,

Quoted 12 lines

Von: [email protected] [[email protected]]" im Auftrag von "Maxime Ripard [[email protected]]
Gesendet: Donnerstag, 7. Mai 2015 14:57
An: Neil Brown; Shaohua Li
Cc: [email protected]; [email protected]; Lior Amsalem; Thomas Petazzoni; Gregory Clement; Boris Brezillon
Betreff: Possible RAID6 regression with ASYNC_TX_DMA enabled in 4.1

Hi,

I'm currently trying to add support for the PQ operations on the marvell XOR engine, in dmaengine, obviously to be able to use async_tx to offload these operations.

Quoted 7 lines

I'm testing these patches with a RAID6 array with 4 disks.

However, since the commit 59fc630b8b5f ("RAID5: batch adjacent full stripe write", every write to that array fails with the following stacktrace.

http://code.bulix.org/eh8iew-88342?raw

I don't know if it might be related. I added support for RAID6 Read-Modify-Write in software XOR with some patches. The following commit mangles some lines in async_pq.c:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?
id=584acdd49cd2472ca0f5a06adbe979db82d0b4af

I introduced a new flag ASYNC_TX_PQ_XOR_DST that notifies the async layer that we want to do a XOR syndrome operation instead of a full calculation. This will enforce the software path because I guessed that hardware does not support that case. Without hardware to check I might have missed some checks in the async layer.

In the upper layer ops_run_reconstruct6 will set the flag if we determined that rmw is faster than rcw.

Can you check if rmw_level=0 fixes the issue. See:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?
id=d06f191f8ecaef4d524e765fdb455f96392fbd42

Quoted 13 lines

It seems to be generated by that warning here:

http://lxr.free-electrons.com/source/crypto/async_tx/async_tx.c#L173

And indeed, if we dump the status of depend_tx here, it's already been acked.

That doesn't happen if ASYNC_TX_DMA is disabled, hence using the software version of it, instead of relying on our XOR engine. It doesn't happen on any commit prior to the one mentionned above, with the exact same changes applied. These changes are meant to be contributed, so I can definitely push them somewhere if needed.

Quoted 6 lines

I don't really know where to look for though, the change that is causing this is probably the change in ops_run_reconstruct6, but I'm not sure that this partial revert alone would work with regard to the rest of the patch.

Maxime

Markus

****************************************************************************
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet.

Über das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung.

Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln

Vorstand:
Kadir Akin
Dr. Michael Höhnerbach

Vorsitzender des Aufsichtsrates:
Hans Kristian Langva

Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497

This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention.

Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln

executive board:
Kadir Akin
Dr. Michael Höhnerbach

President of the supervisory board:
Hans Kristian Langva

Registry office: district court Cologne
Register number: HRB 52 497

****************************************************************************

#3 lore raw
In reply to Maxime Ripard
Quoted 27 lines · Maxime Ripard

Hi,

I'm currently trying to add support for the PQ operations on the marvell XOR engine, in dmaengine, obviously to be able to use async_tx to offload these operations.

I'm testing these patches with a RAID6 array with 4 disks.

However, since the commit 59fc630b8b5f ("RAID5: batch adjacent full stripe write", every write to that array fails with the following stacktrace.

http://code.bulix.org/eh8iew-88342?raw

It seems to be generated by that warning here:

http://lxr.free-electrons.com/source/crypto/async_tx/async_tx.c#L173

And indeed, if we dump the status of depend_tx here, it's already been acked.

That doesn't happen if ASYNC_TX_DMA is disabled, hence using the software version of it, instead of relying on our XOR engine. It doesn't happen on any commit prior to the one mentionned above, with the exact same changes applied. These changes are meant to be contributed, so I can definitely push them somewhere if needed.

Quoted 4 lines

I don't really know where to look for though, the change that is causing this is probably the change in ops_run_reconstruct6, but I'm not sure that this partial revert alone would work with regard to the rest of the patch.

I don't have a machine with dmaengine, it's likely there is error in this side. Could you please make stripe_can_batch() returns false always and check if the error disappear? This should narrow down if it's related to batch issue.

Thanks,
Shaohua

#2 lore raw
In reply to Markus Stockhausen

Hi Markus,

Quoted 35 lines · Markus Stockhausen

Hi Maxime,

Quoted 12 lines

Von: [email protected] [[email protected]]" im Auftrag von "Maxime Ripard [[email protected]]
Gesendet: Donnerstag, 7. Mai 2015 14:57
An: Neil Brown; Shaohua Li
Cc: [email protected]; [email protected]; Lior Amsalem; Thomas Petazzoni; Gregory Clement; Boris Brezillon
Betreff: Possible RAID6 regression with ASYNC_TX_DMA enabled in 4.1

Hi,

I'm currently trying to add support for the PQ operations on the marvell XOR engine, in dmaengine, obviously to be able to use async_tx to offload these operations.

Quoted 7 lines

I'm testing these patches with a RAID6 array with 4 disks.

However, since the commit 59fc630b8b5f ("RAID5: batch adjacent full stripe write", every write to that array fails with the following stacktrace.

http://code.bulix.org/eh8iew-88342?raw

I don't know if it might be related. I added support for RAID6 Read-Modify-Write in software XOR with some patches. The following commit mangles some lines in async_pq.c:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?
id=584acdd49cd2472ca0f5a06adbe979db82d0b4af

I introduced a new flag ASYNC_TX_PQ_XOR_DST that notifies the async layer that we want to do a XOR syndrome operation instead of a full calculation. This will enforce the software path because I guessed that hardware does not support that case. Without hardware to check I might have missed some checks in the async layer.

Quoted 7 lines

In the upper layer ops_run_reconstruct6 will set the flag if we determined that rmw is faster than rcw.

Can you check if rmw_level=0 fixes the issue. See:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?
id=d06f191f8ecaef4d524e765fdb455f96392fbd42

I just gave this a try, and it doesn't fix anything.

One thing I forgot to mention is that our hardware doesn't support the PQ multiplications and product sums, so one of the patches we have is to add a new ASYNC_TX flag to be able to identify and bail out of such transfers.

The patch is here:
https://github.com/MISL-EBU-System-SW/mainline-public/commit/9964fe4a79da10162f83bd527b3fe44da60d7e0f

There might be some interaction between your patch and this one, even though the async_tx code itself looks to be untouched by your patches.

Thanks!
Maxime

Signature · 3 lines
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#5 lore raw
In reply to Maxime Ripard
Quoted 36 lines · Maxime Ripard

Hi Shaohua,

Quoted 33 lines · Shaohua Li
Quoted 31 lines · Maxime Ripard

Hi,

I'm currently trying to add support for the PQ operations on the marvell XOR engine, in dmaengine, obviously to be able to use async_tx to offload these operations.

I'm testing these patches with a RAID6 array with 4 disks.

However, since the commit 59fc630b8b5f ("RAID5: batch adjacent full stripe write", every write to that array fails with the following stacktrace.

http://code.bulix.org/eh8iew-88342?raw

It seems to be generated by that warning here:

http://lxr.free-electrons.com/source/crypto/async_tx/async_tx.c#L173

And indeed, if we dump the status of depend_tx here, it's already been acked.

That doesn't happen if ASYNC_TX_DMA is disabled, hence using the software version of it, instead of relying on our XOR engine. It doesn't happen on any commit prior to the one mentionned above, with the exact same changes applied. These changes are meant to be contributed, so I can definitely push them somewhere if needed.

I don't really know where to look for though, the change that is causing this is probably the change in ops_run_reconstruct6, but I'm not sure that this partial revert alone would work with regard to the rest of the patch.

Quoted 5 lines
Quoted 3 lines

I don't have a machine with dmaengine, it's likely there is error in this side. Could you please make stripe_can_batch() returns false always and check if the error disappear? This should narrow down if it's related to batch issue.

The error indeed disappears if stripe_can_batch always returns false.

Does this fix it?

drivers/md/raid5.c +1 -1
index 77dfd72..5e820fc 100644
@@ -1825,7 +1825,7 @@ ops_run_reconstruct6(struct stripe_head *sh, struct raid5_percpu *percpu,
 	} else
 		init_async_submit(&submit, 0, tx, NULL, NULL,
 				  to_addr_conv(sh, percpu, j));
-	async_gen_syndrome(blocks, 0, count+2, STRIPE_SIZE,  &submit);
+	tx = async_gen_syndrome(blocks, 0, count+2, STRIPE_SIZE,  &submit);
 	if (!last_stripe) {
 		j++;
 		sh = list_first_entry(&sh->batch_list, struct stripe_head,
#4 lore raw
In reply to Shaohua Li

Hi Shaohua,

Quoted 28 lines · Shaohua Li
Quoted 27 lines · Maxime Ripard

Hi,

I'm currently trying to add support for the PQ operations on the marvell XOR engine, in dmaengine, obviously to be able to use async_tx to offload these operations.

I'm testing these patches with a RAID6 array with 4 disks.

However, since the commit 59fc630b8b5f ("RAID5: batch adjacent full stripe write", every write to that array fails with the following stacktrace.

http://code.bulix.org/eh8iew-88342?raw

It seems to be generated by that warning here:

http://lxr.free-electrons.com/source/crypto/async_tx/async_tx.c#L173

And indeed, if we dump the status of depend_tx here, it's already been acked.

That doesn't happen if ASYNC_TX_DMA is disabled, hence using the software version of it, instead of relying on our XOR engine. It doesn't happen on any commit prior to the one mentionned above, with the exact same changes applied. These changes are meant to be contributed, so I can definitely push them somewhere if needed.

Quoted 8 lines
Quoted 4 lines

I don't really know where to look for though, the change that is causing this is probably the change in ops_run_reconstruct6, but I'm not sure that this partial revert alone would work with regard to the rest of the patch.

I don't have a machine with dmaengine, it's likely there is error in this side. Could you please make stripe_can_batch() returns false always and check if the error disappear? This should narrow down if it's related to batch issue.

The error indeed disappears if stripe_can_batch always returns false.

Maxime

Signature · 3 lines
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#6 lore raw
In reply to Shaohua Li

Hi,

Quoted 51 lines · Shaohua Li
Quoted 36 lines · Maxime Ripard

Hi Shaohua,

Quoted 33 lines · Shaohua Li
Quoted 31 lines · Maxime Ripard

Hi,

I'm currently trying to add support for the PQ operations on the marvell XOR engine, in dmaengine, obviously to be able to use async_tx to offload these operations.

I'm testing these patches with a RAID6 array with 4 disks.

However, since the commit 59fc630b8b5f ("RAID5: batch adjacent full stripe write", every write to that array fails with the following stacktrace.

http://code.bulix.org/eh8iew-88342?raw

It seems to be generated by that warning here:

http://lxr.free-electrons.com/source/crypto/async_tx/async_tx.c#L173

And indeed, if we dump the status of depend_tx here, it's already been acked.

That doesn't happen if ASYNC_TX_DMA is disabled, hence using the software version of it, instead of relying on our XOR engine. It doesn't happen on any commit prior to the one mentionned above, with the exact same changes applied. These changes are meant to be contributed, so I can definitely push them somewhere if needed.

I don't really know where to look for though, the change that is causing this is probably the change in ops_run_reconstruct6, but I'm not sure that this partial revert alone would work with regard to the rest of the patch.

Quoted 5 lines
Quoted 3 lines

I don't have a machine with dmaengine, it's likely there is error in this side. Could you please make stripe_can_batch() returns false always and check if the error disappear? This should narrow down if it's related to batch issue.

The error indeed disappears if stripe_can_batch always returns false.

Does this fix it?

drivers/md/raid5.c +0 -0
index 77dfd72..5e820fc 100644

drivers/md/raid5.c ops_run_reconstruct6(struct stripe_head *sh, struct raid5_percpu *percpu, · line 1831

Quoted 8 lines
drivers/md/raid5.c +1 -1
@@ -1825,7 +1825,7 @@ ops_run_reconstruct6(struct stripe_head *sh, struct raid5_percpu *percpu,
 	} else
 		init_async_submit(&submit, 0, tx, NULL, NULL,
 				  to_addr_conv(sh, percpu, j));
-	async_gen_syndrome(blocks, 0, count+2, STRIPE_SIZE,  &submit);
+	tx = async_gen_syndrome(blocks, 0, count+2, STRIPE_SIZE,  &submit);
 	if (!last_stripe) {
 		j++;
 		sh = list_first_entry(&sh->batch_list, struct stripe_head,

It does, thanks!

Feel free to add my Tested-by if you submit this patch.

Maxime

Signature · 3 lines
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com