iommu/io-pgtable-arm-v7s: About pagetable 33bit PA
Hi Robin,
After the commit ad67f5a6545f ("arm64: replace ZONE_DMA with ZONE_DMA32"), we don't have ZONE_DMA in aarch64, but __arm_v7s_alloc_table[1] use the GFP_DMA to expect the physical address of pagetable should be 32bit.
Right now we meet a issue that the lvl1 pagetable PA is 0x1_3ab6_0000 in the 4GB broad. then the IOMMU initialize failed.(This issue can be fixed if we revert ad67f5a6545f.)
I planed to add GFP_DMA32 for pagetable allocation. the level1 was ok but level2 was failed. It looks that slab don't like GFP_DMA32[2]. this is the warning log:
=====
Unexpected gfp: 0x4 (GFP_DMA32). Fixing up to gfp: 0x488020 (GFP_ATOMIC| __GFP_ZERO). Fix your code!
=====
I don't know why slab don't allow GFP_DMA32, the gfp flags seems only be bypassed to alloc_page. Is it possible that let slab allow GFP_DMA32 for aarch64?
We notice that this has been discussed[3]. but if we use alloc_page for the level2 pagetable, It may waste lots of memory.
Any suggestion is appreciated.
[1]
https://elixir.bootlin.com/linux/v4.20-rc1/source/drivers/iommu/io-pgtable-arm-v7s.c#L200
[2] https://elixir.bootlin.com/linux/v4.20-rc1/source/mm/internal.h#L37
[3] https://patchwork.kernel.org/patch/10474289/
Quoted 10 lines · Yong Wu
Hi Robin,
After the commit
ad67f5a6545f("arm64: replaceZONE_DMAwithZONE_DMA32"), we don't haveZONE_DMAin aarch64, but __arm_v7s_alloc_table[1] use theGFP_DMAto expect the physical address of pagetable should be 32bit.Right now we meet a issue that the lvl1 pagetable PA is 0x1_3ab6_0000 in the 4GB broad. then the IOMMU initialize failed.(This issue can be fixed if we revert
ad67f5a6545f.)
But that shouldn't actually be failing on MTK platforms anyway, right, since your IOMMU is still capable of addressing that? Seems like we overlooked that check in __arm_v7s_alloc_table(), where the conversion by casting will want updating to something like "iopte_to_paddr(paddr_to_iopte(...))" in patch #4 of your series.
Quoted 10 lines
I planed to add
GFP_DMA32for pagetable allocation. the level1 was ok but level2 was failed. It looks that slab don't like GFP_DMA32[2]. this is the warning log:
=====
Unexpected gfp:0x4(GFP_DMA32). Fixing up to gfp:0x488020(GFP_ATOMIC|__GFP_ZERO). Fix your code!
=====
I don't know why slab don't allowGFP_DMA32, the gfp flags seems only be bypassed toalloc_page. Is it possible that let slab allowGFP_DMA32for aarch64?
I had a bit of a look into it some time ago, and I don't recall seeing any obvious reason that the kmem_cache API couldn't support ZONE_DMA32 in general (either via a separate SLAB_CACHE_DMA32, or just overloading SLAB_CACHE_DMA depending on which of CONFIG_ZONE_DMA/CONFIG_ZONE_DMA32 are enabled) - I guess that's just never been implemented since nobody needed it before.
Robin.
Quoted 13 lines
We notice that this has been discussed[3]. but if we use
alloc_pagefor the level2 pagetable, It may waste lots of memory.Any suggestion is appreciated.
Reported-by Nicolas Boichat <[email protected]>[1]
https://elixir.bootlin.com/linux/v4.20-rc1/source/drivers/iommu/io-pgtable-arm-v7s.c#L200
[2] https://elixir.bootlin.com/linux/v4.20-rc1/source/mm/internal.h#L37
[3] https://patchwork.kernel.org/patch/10474289/
A message that is not in the archive (sent off-list or not yet received): 1093be2a-f9e8-1889-fc9f-b44b6ecfd7da-5wv7dgnIgG8@public.gmane.org
Quoted 12 lines · Robin Murphy
Quoted 10 lines · Yong Wu
Hi Robin,
After the commit
ad67f5a6545f("arm64: replaceZONE_DMAwithZONE_DMA32"), we don't haveZONE_DMAin aarch64, but __arm_v7s_alloc_table[1] use theGFP_DMAto expect the physical address of pagetable should be 32bit.Right now we meet a issue that the lvl1 pagetable PA is 0x1_3ab6_0000 in the 4GB broad. then the IOMMU initialize failed.(This issue can be fixed if we revert
ad67f5a6545f.)
Quoted 5 lines
But that shouldn't actually be failing on MTK platforms anyway, right, since your IOMMU is still capable of addressing that? Seems like we overlooked that check in
__arm_v7s_alloc_table(), where the conversion by casting will want updating to something like "iopte_to_paddr(paddr_to_iopte(...))" in patch #4 of your series.
The current mt8183 IOMMU support the pagetable address over 4GB. Unfortunately the previous mt8173 don't support.(mt8173 IOMMU support mapping for the dram over 4GB while its pagetable still need locate below 4GB.)
In order to unify the code, we still expect lvl2 pagetable base don't over 4GB. thus I didn't change it in the current patchset.
Quoted 11 lines
Quoted 11 lines
I planed to add
GFP_DMA32for pagetable allocation. the level1 was ok but level2 was failed. It looks that slab don't like GFP_DMA32[2]. this is the warning log:
=====
Unexpected gfp:0x4(GFP_DMA32). Fixing up to gfp:0x488020(GFP_ATOMIC|__GFP_ZERO). Fix your code!
=====
I don't know why slab don't allowGFP_DMA32, the gfp flags seems only be bypassed toalloc_page. Is it possible that let slab allowGFP_DMA32for aarch64?
Quoted 6 lines
I had a bit of a look into it some time ago, and I don't recall seeing any obvious reason that the
kmem_cacheAPI couldn't supportZONE_DMA32in general (either via a separateSLAB_CACHE_DMA32, or just overloadingSLAB_CACHE_DMAdepending on which of CONFIG_ZONE_DMA/CONFIG_ZONE_DMA32 are enabled) - I guess that's just never been implemented since nobody needed it before.
Thanks for the comment. We could try a patch for this.
Quoted 16 lines
Robin.
Quoted 13 lines
We notice that this has been discussed[3]. but if we use
alloc_pagefor the level2 pagetable, It may waste lots of memory.Any suggestion is appreciated.
Reported-by Nicolas Boichat <[email protected]>[1]
https://elixir.bootlin.com/linux/v4.20-rc1/source/drivers/iommu/io-pgtable-arm-v7s.c#L200
[2] https://elixir.bootlin.com/linux/v4.20-rc1/source/mm/internal.h#L37
[3] https://patchwork.kernel.org/patch/10474289/
Hi Robin/Yong,
Quoted 39 lines · Yong Wu
Quoted 12 lines · Robin Murphy
Quoted 10 lines · Yong Wu
Hi Robin,
After the commit
ad67f5a6545f("arm64: replaceZONE_DMAwithZONE_DMA32"), we don't haveZONE_DMAin aarch64, but __arm_v7s_alloc_table[1] use theGFP_DMAto expect the physical address of pagetable should be 32bit.Right now we meet a issue that the lvl1 pagetable PA is 0x1_3ab6_0000 in the 4GB broad. then the IOMMU initialize failed.(This issue can be fixed if we revert
ad67f5a6545f.)Quoted 5 lines
But that shouldn't actually be failing on MTK platforms anyway, right, since your IOMMU is still capable of addressing that? Seems like we overlooked that check in
__arm_v7s_alloc_table(), where the conversion by casting will want updating to something like "iopte_to_paddr(paddr_to_iopte(...))" in patch #4 of your series.The current mt8183 IOMMU support the pagetable address over 4GB. Unfortunately the previous mt8173 don't support.(mt8173 IOMMU support mapping for the dram over 4GB while its pagetable still need locate below 4GB.)
In order to unify the code, we still expect lvl2 pagetable base don't over 4GB. thus I didn't change it in the current patchset.
Quoted 11 lines
Quoted 11 lines
I planed to add
GFP_DMA32for pagetable allocation. the level1 was ok but level2 was failed. It looks that slab don't like GFP_DMA32[2]. this is the warning log:
=====
Unexpected gfp:0x4(GFP_DMA32). Fixing up to gfp:0x488020(GFP_ATOMIC|__GFP_ZERO). Fix your code!
=====
I don't know why slab don't allowGFP_DMA32, the gfp flags seems only be bypassed toalloc_page. Is it possible that let slab allowGFP_DMA32for aarch64?
Quoted 8 lines
Quoted 6 lines
I had a bit of a look into it some time ago, and I don't recall seeing any obvious reason that the
kmem_cacheAPI couldn't supportZONE_DMA32in general (either via a separateSLAB_CACHE_DMA32, or just overloadingSLAB_CACHE_DMAdepending on which of CONFIG_ZONE_DMA/CONFIG_ZONE_DMA32 are enabled) - I guess that's just never been implemented since nobody needed it before.Thanks for the comment. We could try a patch for this.
I made an attempt here, if that's helpful:
https://lore.kernel.org/patchwork/cover/1009116/ .
I'm still a bit uncomfortable with GFP_DMA passed to kmem_cache_alloc suddenly becoming GFP_DMA32 in the underlying call, but I'm not sure if there is a better way (apart from duplicating the caches, and a lot of ifdefs in callers).
Thanks,
Quoted 18 lines
Quoted 16 lines
Robin.
Quoted 13 lines
We notice that this has been discussed[3]. but if we use
alloc_pagefor the level2 pagetable, It may waste lots of memory.Any suggestion is appreciated.
Reported-by Nicolas Boichat <[email protected]>[1]
https://elixir.bootlin.com/linux/v4.20-rc1/source/drivers/iommu/io-pgtable-arm-v7s.c#L200
[2] https://elixir.bootlin.com/linux/v4.20-rc1/source/mm/internal.h#L37
[3] https://patchwork.kernel.org/patch/10474289/