riscv: select DMA_DIRECT_REMAP by RISCV_ISA_SVPBMT and ERRATA_THEAD_MAE
PATCH 1/1 riscv: select DMA_DIRECT_REMAP by RISCV_ISA_SVPBMT and ERRATA_THEAD_MAE
Select DMA_DIRECT_REMAP for the RISC-V extensions that allow to set page-based memory types in PTEs according to the requested DMA attributes. This is the purpose of Svpbmt or XTheadMae extensions. Zicbom or XTheadCmo serve a different purpose, providing instructions to flush/invalidate cache blocks.
--- arch/riscv/Kconfig | 2 +- arch/riscv/Kconfig.errata | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
arch/riscv/Kconfig +1 -1
@@ -603,6 +603,7 @@ config RISCV_ISA_SVPBMT depends on 64BIT && MMU depends on RISCV_ALTERNATIVE default y + select DMA_DIRECT_REMAP help Adds support to dynamically detect the presence of the Svpbmt ISA-extension (Supervisor-mode: page-based memory types) and
@@ -787,7 +788,6 @@ config RISCV_ISA_ZICBOM depends on RISCV_ALTERNATIVE default y select RISCV_DMA_NONCOHERENT - select DMA_DIRECT_REMAP help Adds support to dynamically detect the presence of the ZICBOM extension (Cache Block Management Operations) and enable its
arch/riscv/Kconfig.errata +1 -1
@@ -86,6 +86,7 @@ config ERRATA_THEAD_MAE bool "Apply T-Head's memory attribute extension (XTheadMae) errata" depends on ERRATA_THEAD && 64BIT && MMU select RISCV_ALTERNATIVE_EARLY + select DMA_DIRECT_REMAP default y help This will apply the memory attribute extension errata to handle the
@@ -96,7 +97,6 @@ config ERRATA_THEAD_MAE config ERRATA_THEAD_CMO bool "Apply T-Head cache management errata" depends on ERRATA_THEAD && MMU - select DMA_DIRECT_REMAP select RISCV_DMA_NONCOHERENT select RISCV_NONSTANDARD_CACHE_OPS default y
Signature · 1 line
2.48.0
Quoted 5 lines · Sergey Matyukevich
Select
DMA_DIRECT_REMAPfor the RISC-V extensions that allow to set page-based memory types in PTEs according to the requested DMA attributes. This is the purpose of Svpbmt or XTheadMae extensions. Zicbom or XTheadCmo serve a different purpose, providing instructions to flush/invalidate cache blocks.
Please explain what this is supposed to solve, because the above explanation dosn't make any sense. DMA_DIRECT_REMAP is one of the implementations supporting dma coherent allocatiosn for non-coherent devices. So selecting it from something that just keyes off support for an extension, but not the dma implementation is wrong.
Quoted 7 lines · Christoph Hellwig
Quoted 5 lines · Sergey Matyukevich
Select
DMA_DIRECT_REMAPfor the RISC-V extensions that allow to set page-based memory types in PTEs according to the requested DMA attributes. This is the purpose of Svpbmt or XTheadMae extensions. Zicbom or XTheadCmo serve a different purpose, providing instructions to flush/invalidate cache blocks.
Quoted 6 lines
Please explain what this is supposed to solve, because the above explanation dosn't make any sense.
DMA_DIRECT_REMAPis one of the implementations supporting dma coherent allocatiosn for non-coherent devices. So selecting it from something that just keyes off support for an extension, but not the dma implementation is wrong.
Now DMA_DIRECT_REMAP is selected either by Zicbom (standard) or XTheadCmo (vendor) RISC-V extensions. However neither of them can help to implement DMA_DIRECT_REMAP on RISC-V. So selection of DMA_DIRECT_REMAP has been moved in Kconfigs to Svpbmt and XTheadMae extensions.
Quoted 9 lines · Sergey Matyukevich
Quoted 6 lines
Please explain what this is supposed to solve, because the above explanation dosn't make any sense.
DMA_DIRECT_REMAPis one of the implementations supporting dma coherent allocatiosn for non-coherent devices. So selecting it from something that just keyes off support for an extension, but not the dma implementation is wrong.Now
DMA_DIRECT_REMAPis selected either by Zicbom (standard) or XTheadCmo (vendor) RISC-V extensions.
Because they need DMA_DIRECT_REMAP to implement DMA coherent.
Quoted 3 lines
However neither of them can help to implement
DMA_DIRECT_REMAPon RISC-V. So selection ofDMA_DIRECT_REMAPhas been moved in Kconfigs to Svpbmt and XTheadMae extensions.
But Svpbmt does not imply that you even need DMA_DIRECT_REMAP.
Are you tying to solve a problem here? If so can you explain it?
Hi Christoph,
Quoted 13 lines · Christoph Hellwig
Quoted 9 lines · Sergey Matyukevich
Quoted 6 lines
Please explain what this is supposed to solve, because the above explanation dosn't make any sense.
DMA_DIRECT_REMAPis one of the implementations supporting dma coherent allocatiosn for non-coherent devices. So selecting it from something that just keyes off support for an extension, but not the dma implementation is wrong.Now
DMA_DIRECT_REMAPis selected either by Zicbom (standard) or XTheadCmo (vendor) RISC-V extensions.Because they need
DMA_DIRECT_REMAPto implement DMA coherent.
Quoted 7 lines
Quoted 3 lines
However neither of them can help to implement
DMA_DIRECT_REMAPon RISC-V. So selection ofDMA_DIRECT_REMAPhas been moved in Kconfigs to Svpbmt and XTheadMae extensions.But Svpbmt does not imply that you even need
DMA_DIRECT_REMAP.Are you tying to solve a problem here? If so can you explain it?
Sure. In brief, this about the choice between DMA_GLOBAL_POOL and DMA_DIRECT_REMAP. We can use either of them to work with non-coherent devices. But on RISC-V those two options require different extensions.
If we select DMA_GLOBAL_POOL, then we need only cache operations. So on RISC-V it is enough to have Zicbom (standard) or XTheadCmo (vendor) or any other vendor specific cache ops implemented using RISCV_ALTERNATIVE or RISCV_NONSTANDARD_CACHE_OPS.
Using DMA_DIRECT_REMAP requires not only cache management operations, but also a way to modify page attributes, e.g. to mark it non-cacheable. So on RISC-V in addition to CMO extensions we also need extensions for page-based memory types such as Svpbmt (standard) or XTheadMae (vendor).
Current RISC-V Kconfig files enable DMA_DIRECT_REMAP for Zicbom and XTheadCmo. According to the above comments, this is not good:
- it is wrong since Zicbom alone is not enough for DMA_DIRECT_REMAP
- it prevents using DMA_GLOBAL_POOL for platforms without support for page-based memory attributes
My suggestion was to move DMA_DIRECT_REMAP to the Kconfig entries for Svpbmt and XTheadMae. In this case platforms without Svpbmt support can disable it in kernel config and switch to DMA_GLOBAL_POOL instead.
IIUC one of your concerns was selecting DMA_DIRECT_REMAP under config option not related to DMA implementations. Does it make sense to add additional layer similar to RISCV_DMA_NONCOHERENT ?
Regards,
Sergey
Alexandre Ghiti
reviewerHi Sergey,
Quoted 19 lines · Sergey Matyukevich
Select
DMA_DIRECT_REMAPfor the RISC-V extensions that allow to set page-based memory types in PTEs according to the requested DMA attributes. This is the purpose of Svpbmt or XTheadMae extensions. Zicbom or XTheadCmo serve a different purpose, providing instructions to flush/invalidate cache blocks.Fixes 381cae169853 ("riscv: only select DMA_DIRECT_REMAP from RISCV_ISA_ZICBOM and ERRATA_THEAD_PBMT")Signed-off-by Sergey Matyukevich <[email protected]>--- arch/riscv/Kconfig | 2 +- arch/riscv/Kconfig.errata | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)arch/riscv/Kconfig +0
-0
arch/riscv/Kconfig config RISCV_ISA_SVPBMT · line 606
Quoted 4 lines
arch/riscv/Kconfig +1
-0@@ -603,6 +603,7 @@ config RISCV_ISA_SVPBMT depends on 64BIT && MMU depends on RISCV_ALTERNATIVE default y + select DMA_DIRECT_REMAP
From what I read, DMA_DIRECT_MAP relies on the ability to map pages uncached (pgprot_dmacoherent() here https://elixir.bootlin.com/linux/v6.13.2/source/kernel/dma/pool.c#L104). But CONFIG_RISCV_ISA_SVPBMT does not guarantee that the underlying platform supports svpbmt so to me it is wrong to select DMA_DIRECT_MAP, we would need some runtime check instead.
arch/riscv/Kconfig config RISCV_ISA_ZICBOM · line 790
Quoted 7 lines
arch/riscv/Kconfig +0
-0help Adds support to dynamically detect the presence of the Svpbmt ISA-extension (Supervisor-mode: page-based memory types) and@@ -787,7 +788,6 @@ config RISCV_ISA_ZICBOM depends on RISCV_ALTERNATIVE default y select RISCV_DMA_NONCOHERENT
And in the same way, we should not enable RISCV_DMA_NONCOHERENT since CONFIG_RISCV_ISA_ZICBOM does guarantee the presence of zicbom. Because then in mm/dma-noncoherent.c, the cache flush operations are nops.
Or am I missing something?
Thanks,
Alex
Quoted 24 lines
arch/riscv/Kconfig +0
-1- select DMA_DIRECT_REMAP help Adds support to dynamically detect the presence of the ZICBOM extension (Cache Block Management Operations) and enable itsarch/riscv/Kconfig.errata +1
-1@@ -86,6 +86,7 @@ config ERRATA_THEAD_MAE bool "Apply T-Head's memory attribute extension (XTheadMae) errata" depends on ERRATA_THEAD && 64BIT && MMU select RISCV_ALTERNATIVE_EARLY + select DMA_DIRECT_REMAP default y help This will apply the memory attribute extension errata to handle the@@ -96,7 +97,6 @@ config ERRATA_THEAD_MAE config ERRATA_THEAD_CMO bool "Apply T-Head cache management errata" depends on ERRATA_THEAD && MMU - select DMA_DIRECT_REMAP select RISCV_DMA_NONCOHERENT select RISCV_NONSTANDARD_CACHE_OPS default y
Hi Alexandre,
Quoted 28 lines · Alexandre Ghiti
Hi Sergey,
Quoted 19 lines · Sergey Matyukevich
Select
DMA_DIRECT_REMAPfor the RISC-V extensions that allow to set page-based memory types in PTEs according to the requested DMA attributes. This is the purpose of Svpbmt or XTheadMae extensions. Zicbom or XTheadCmo serve a different purpose, providing instructions to flush/invalidate cache blocks.Fixes 381cae169853 ("riscv: only select DMA_DIRECT_REMAP from RISCV_ISA_ZICBOM and ERRATA_THEAD_PBMT")Signed-off-by Sergey Matyukevich <[email protected]>--- arch/riscv/Kconfig | 2 +- arch/riscv/Kconfig.errata | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)arch/riscv/Kconfig +0
-0arch/riscv/Kconfig config RISCV_ISA_SVPBMT · line 606
Quoted 4 lines
arch/riscv/Kconfig +1
-0@@ -603,6 +603,7 @@ config RISCV_ISA_SVPBMT depends on 64BIT && MMU depends on RISCV_ALTERNATIVE default y + select DMA_DIRECT_REMAP
Quoted 6 lines
From what I read,
DMA_DIRECT_MAPrelies on the ability to map pages uncached (pgprot_dmacoherent()here
https://elixir.bootlin.com/linux/v6.13.2/source/kernel/dma/pool.c#L104). ButCONFIG_RISCV_ISA_SVPBMTdoes not guarantee that the underlying platform supports svpbmt so to me it is wrong to selectDMA_DIRECT_MAP, we would need some runtime check instead.
IIUC this function highlights coherent dma allocation options and their requirements even more clearly:
- https://elixir.bootlin.com/linux/v6.13.2/source/kernel/dma/direct.c#L222
Quoted 13 lines
arch/riscv/Kconfig config RISCV_ISA_ZICBOM · line 790
Quoted 7 lines
arch/riscv/Kconfig +0
-0help Adds support to dynamically detect the presence of the Svpbmt ISA-extension (Supervisor-mode: page-based memory types) and@@ -787,7 +788,6 @@ config RISCV_ISA_ZICBOM depends on RISCV_ALTERNATIVE default y select RISCV_DMA_NONCOHERENTAnd in the same way, we should not enable
RISCV_DMA_NONCOHERENTsinceCONFIG_RISCV_ISA_ZICBOMdoes guarantee the presence of zicbom. Because then inmm/dma-noncoherent.c, the cache flush operations are nops.
Quoted 1 line
Or am I missing something?
This is my understanding as well. In fact this patch is almost useless. It would only allow to enable DMA_GLOBAL_POOL for platforms that support Zicbom, but do not support Svpbmt.
The actual problem is that the RISC-V kernel image cannot have both DMA_DIRECT_REMAP and DMA_GLOBAL_POOL since they are now mutually exclusive in kernel/dma/Kconfig. This limitation is not convenient for RISC-V, where kernels can be built with support for multiple extensions and errata. But on boot only appropriate subset of them is enabled based on the chip's VENDOR_ID and selected dtb.
Currently a portable RISC-V kernel is suitable only for platforms with support for both Zicbom and Svpbmt or their vendor-specific alternatives. So it doesn't really matter where DMA_DIRECT_REMAP is selected. Platforms without Svpbmt need to build their own non-portable kernels anyway, enabling support for DMA_GLOBAL_POOL. For instance, Starfive and Andes in arch/riscv/Kconfig.errata and drivers/soc/renesas/Kconfig respectively.
Maybe one possible option would be to revert commit da323d464070 ("dma-direct: add dependencies to CONFIG_DMA_GLOBAL_POOL") and add runtime checks in dma_direct_alloc for dma_alloc_from_global_coherent.
In that case RISC-V kernels could be built with support for both DMA_GLOBAL_POOL and DMA_DIRECT_REMAP. Global pool code path could be enabled only for platforms that explicitly specify it in their dtbs.
Regards,
Sergey
Alexandre Ghiti
reviewer+cc Marek
Quoted 71 lines · Sergey Matyukevich
Hi Alexandre,
Quoted 17 lines · Alexandre Ghiti
Hi Sergey,
Quoted 14 lines · Sergey Matyukevich
Select
DMA_DIRECT_REMAPfor the RISC-V extensions that allow to set page-based memory types in PTEs according to the requested DMA attributes. This is the purpose of Svpbmt or XTheadMae extensions. Zicbom or XTheadCmo serve a different purpose, providing instructions to flush/invalidate cache blocks.Fixes 381cae169853 ("riscv: only select DMA_DIRECT_REMAP from RISCV_ISA_ZICBOM and ERRATA_THEAD_PBMT")Signed-off-by Sergey Matyukevich <[email protected]>--- arch/riscv/Kconfig | 2 +- arch/riscv/Kconfig.errata | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)Quoted 16 lines
arch/riscv/Kconfig config RISCV_ISA_SVPBMT · line 606
Quoted 9 lines
arch/riscv/Kconfig +1
-0@@ -603,6 +603,7 @@ config RISCV_ISA_SVPBMT depends on 64BIT && MMU depends on RISCV_ALTERNATIVE default y + select DMA_DIRECT_REMAPFrom what I read,
DMA_DIRECT_MAPrelies on the ability to map pages uncached
(pgprot_dmacoherent()here
https://elixir.bootlin.com/linux/v6.13.2/source/kernel/dma/pool.c#L104). ButCONFIG_RISCV_ISA_SVPBMTdoes not guarantee that the underlying platform supports svpbmt so to me it is wrong to selectDMA_DIRECT_MAP, we would need some runtime check instead.IIUC this function highlights coherent dma allocation options and their requirements even more clearly:
- https://elixir.bootlin.com/linux/v6.13.2/source/kernel/dma/direct.c#L222Quoted 12 lines
arch/riscv/Kconfig config RISCV_ISA_ZICBOM · line 790
Quoted 7 lines
arch/riscv/Kconfig +0
-0help Adds support to dynamically detect the presence of the Svpbmt ISA-extension (Supervisor-mode: page-based memory types) and@@ -787,7 +788,6 @@ config RISCV_ISA_ZICBOM depends on RISCV_ALTERNATIVE default y select RISCV_DMA_NONCOHERENTAnd in the same way, we should not enable
RISCV_DMA_NONCOHERENTsinceCONFIG_RISCV_ISA_ZICBOMdoes guarantee the presence of zicbom. Because then inmm/dma-noncoherent.c, the cache flush operations are nops.Quoted 1 line
Or am I missing something?
This is my understanding as well. In fact this patch is almost useless. It would only allow to enable
DMA_GLOBAL_POOLfor platforms that support Zicbom, but do not support Svpbmt.The actual problem is that the RISC-V kernel image cannot have both
DMA_DIRECT_REMAPandDMA_GLOBAL_POOLsince they are now mutually exclusive inkernel/dma/Kconfig. This limitation is not convenient for RISC-V, where kernels can be built with support for multiple extensions and errata. But on boot only appropriate subset of them is enabled based on the chip'sVENDOR_IDand selected dtb.Currently a portable RISC-V kernel is suitable only for platforms with support for both Zicbom and Svpbmt or their vendor-specific alternatives. So it doesn't really matter where
DMA_DIRECT_REMAPis selected. Platforms without Svpbmt need to build their own non-portable kernels anyway, enabling support forDMA_GLOBAL_POOL. For instance, Starfive and Andes inarch/riscv/Kconfig.errataanddrivers/soc/renesas/Kconfigrespectively.
Quoted 3 lines
Maybe one possible option would be to revert commit
da323d464070("dma-direct: add dependencies toCONFIG_DMA_GLOBAL_POOL") and add runtime checks indma_direct_allocfordma_alloc_from_global_coherent.
And dma_common_contiguous_remap(). Others? I guess the best would be to have a patch to propose, unless Christoph or Marek answer with ideas.
Can you prepare something Sergey?
Thanks,
Alex
Quoted 12 lines
In that case RISC-V kernels could be built with support for both
DMA_GLOBAL_POOLandDMA_DIRECT_REMAP. Global pool code path could be enabled only for platforms that explicitly specify it in their dtbs.Regards,
Sergey_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv