Skip to contentSkip to conversation

iommu/mediatek: Use totalram_pages to setup enable_4GB

PATCH In review Miles Chen 3 participants 7 messages Jun 4, 2020 – Jun 5, 2020

Subsystems (3)
lore
Share & tools
lore

Miles Chen

author
#0 lore raw

PATCH 1/1 iommu/mediatek: Use totalram_pages to setup enable_4GB

To build this driver as a kernel module, we cannot use the unexported symbol "max_pfn" to setup enable_4GB.

Use totalram_pages() instead to setup enable_4GB.

---
 drivers/iommu/mtk_iommu.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/iommu/mtk_iommu.c +2 -3
index 5f4d6df59cf6..c2798a6e0e38 100644
@@ -3,7 +3,6 @@
  * Copyright (c) 2015-2016 MediaTek Inc.
  * Author: Yong Wu <[email protected]>
  */
-#include <linux/memblock.h>
 #include <linux/bug.h>
 #include <linux/clk.h>
 #include <linux/component.h>
@@ -626,8 +625,8 @@ static int mtk_iommu_probe(struct platform_device *pdev)
 		return -ENOMEM;
 	data->protect_base = ALIGN(virt_to_phys(protect), MTK_PROTECT_PA_ALIGN);
 
-	/* Whether the current dram is over 4GB */
-	data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT));
+	/* Whether the current dram is over 4GB, note: DRAM start at 1GB  */
+	data->enable_4GB = !!(totalram_pages() > ((SZ_2G + SZ_1G) >> PAGE_SHIFT));
 	if (!data->plat_data->has_4gb_mode)
 		data->enable_4GB = false;
 
Signature · 1 line
2.18.0
#1 lore raw
Quoted 27 lines · Miles Chen

To build this driver as a kernel module, we cannot use the unexported symbol "max_pfn" to setup enable_4GB.

Use totalram_pages() instead to setup enable_4GB.

---
 drivers/iommu/mtk_iommu.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/iommu/mtk_iommu.c +0 -1
index 5f4d6df59cf6..c2798a6e0e38 100644
@@ -3,7 +3,6 @@
  * Copyright (c) 2015-2016 MediaTek Inc.
  * Author: Yong Wu <[email protected]>
  */
-#include <linux/memblock.h>
 #include <linux/bug.h>
 #include <linux/clk.h>
 #include <linux/component.h>

drivers/iommu/mtk_iommu.c static int mtk_iommu_probe(struct platform_device *pdev) · line 629

Quoted 7 lines
drivers/iommu/mtk_iommu.c +2 -2
@@ -626,8 +625,8 @@ static int mtk_iommu_probe(struct platform_device *pdev)
 		return -ENOMEM;
 	data->protect_base = ALIGN(virt_to_phys(protect), MTK_PROTECT_PA_ALIGN);
 
-	/* Whether the current dram is over 4GB */
-	data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT));
+	/* Whether the current dram is over 4GB, note: DRAM start at 1GB  */
+	data->enable_4GB = !!(totalram_pages() > ((SZ_2G + SZ_1G) >> PAGE_SHIFT));

A similar thing seems to be done by
drivers/media/platform/mtk-vpu/mtk_vpu.c:

	vpu->enable_4GB = !!(totalram_pages() > (SZ_2G >> PAGE_SHIFT));

I do wonder if some weird memory hotplug setups might give you false negatives.

E.g., start a VM with 1GB and hotplug 1GB - it will be hotplugged on x86-64 above 4GB, turning max_pfn into 5GB. totalram_pages() should return something < 2GB.

Same can happen when you have a VM and use ballooning to fake-unplug memory, making totalram_pages() return something < 4GB, but leaving usable pfns >= 4GB.

but
... I don't know if I understood what "enable_4GB" needs/implies ... I don't know if this is applicable to VMs at all (on real HW such
memory hotplug setups should not exist)
... I don't know how this code would react to memory hotplug, so if the
condition changes after the driver loaded and enable_4GB would suddenly apply. Again, most probably not relevant on real HW, only for VMs.

Signature · 3 lines
Thanks,

David / dhildenb
author
#2 lore raw
Quoted 43 lines · David Hildenbrand
Quoted 27 lines · Miles Chen

To build this driver as a kernel module, we cannot use the unexported symbol "max_pfn" to setup enable_4GB.

Use totalram_pages() instead to setup enable_4GB.

---
 drivers/iommu/mtk_iommu.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/iommu/mtk_iommu.c +0 -1
index 5f4d6df59cf6..c2798a6e0e38 100644
@@ -3,7 +3,6 @@
  * Copyright (c) 2015-2016 MediaTek Inc.
  * Author: Yong Wu <[email protected]>
  */
-#include <linux/memblock.h>
 #include <linux/bug.h>
 #include <linux/clk.h>
 #include <linux/component.h>

drivers/iommu/mtk_iommu.c static int mtk_iommu_probe(struct platform_device *pdev) · line 629

Quoted 7 lines
drivers/iommu/mtk_iommu.c +2 -2
@@ -626,8 +625,8 @@ static int mtk_iommu_probe(struct platform_device *pdev)
 		return -ENOMEM;
 	data->protect_base = ALIGN(virt_to_phys(protect), MTK_PROTECT_PA_ALIGN);
 
-	/* Whether the current dram is over 4GB */
-	data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT));
+	/* Whether the current dram is over 4GB, note: DRAM start at 1GB  */
+	data->enable_4GB = !!(totalram_pages() > ((SZ_2G + SZ_1G) >> PAGE_SHIFT));

A similar thing seems to be done by
drivers/media/platform/mtk-vpu/mtk_vpu.c:

	vpu->enable_4GB = !!(totalram_pages() > (SZ_2G >> PAGE_SHIFT));

I do wonder if some weird memory hotplug setups might give you false negatives.

Quoted 7 lines

E.g., start a VM with 1GB and hotplug 1GB - it will be hotplugged on x86-64 above 4GB, turning max_pfn into 5GB. totalram_pages() should return something < 2GB.

Same can happen when you have a VM and use ballooning to fake-unplug memory, making totalram_pages() return something < 4GB, but leaving usable pfns >= 4GB

Yes. Yingjoe also told me that this patch is not correct.

Thanks for pointing this out. totalram_pages() does not work for some cases:

e.g., DRAM start @0x4000_0000 and DRAM size is 0x1_0000_0000 but we reserve large amount of memory, which makes totalram_pages() < 3GB but it is possible to allocate a pfn >= 4GB.

I will discuss this internally.

Miles

Quoted 12 lines

.

but
... I don't know if I understood what "enable_4GB" needs/implies ... I don't know if this is applicable to VMs
at all (on real HW such
memory hotplug setups should not exist)
... I don't know how this code would react to memory hotplug, so if the
condition changes after the driver loaded and enable_4GB would suddenly apply. Again, most probably not relevant on real HW, only for VMs.

#3 lore raw
Quoted 48 lines · Miles Chen
Quoted 47 lines · David Hildenbrand
Quoted 27 lines · Miles Chen

To build this driver as a kernel module, we cannot use the unexported symbol "max_pfn" to setup enable_4GB.

Use totalram_pages() instead to setup enable_4GB.

---
 drivers/iommu/mtk_iommu.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/iommu/mtk_iommu.c +0 -1
index 5f4d6df59cf6..c2798a6e0e38 100644
@@ -3,7 +3,6 @@
  * Copyright (c) 2015-2016 MediaTek Inc.
  * Author: Yong Wu <[email protected]>
  */
-#include <linux/memblock.h>
 #include <linux/bug.h>
 #include <linux/clk.h>
 #include <linux/component.h>

drivers/iommu/mtk_iommu.c static int mtk_iommu_probe(struct platform_device *pdev) · line 629

Quoted 7 lines
drivers/iommu/mtk_iommu.c +2 -2
@@ -626,8 +625,8 @@ static int mtk_iommu_probe(struct platform_device *pdev)
 		return -ENOMEM;
 	data->protect_base = ALIGN(virt_to_phys(protect), MTK_PROTECT_PA_ALIGN);
 
-	/* Whether the current dram is over 4GB */
-	data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT));
+	/* Whether the current dram is over 4GB, note: DRAM start at 1GB  */
+	data->enable_4GB = !!(totalram_pages() > ((SZ_2G + SZ_1G) >> PAGE_SHIFT));

A similar thing seems to be done by
drivers/media/platform/mtk-vpu/mtk_vpu.c:

	vpu->enable_4GB = !!(totalram_pages() > (SZ_2G >> PAGE_SHIFT));

I do wonder if some weird memory hotplug setups might give you false negatives.

E.g., start a VM with 1GB and hotplug 1GB - it will be hotplugged on x86-64 above 4GB, turning max_pfn into 5GB. totalram_pages() should return something < 2GB.

Quoted 9 lines
Quoted 3 lines

Same can happen when you have a VM and use ballooning to fake-unplug memory, making totalram_pages() return something < 4GB, but leaving usable pfns >= 4GB

Yes. Yingjoe also told me that this patch is not correct.

Thanks for pointing this out. totalram_pages() does not work for some cases:

Just a thought: If memory hotplug is applicable as well, you might either want to always assume data->enable_4GB, or handle memory hotplug events from the memory notifier, when new memory gets onlined (not sure how tricky that is).

Signature · 9 lines
Thanks,

David / dhildenb


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
#4 lore raw
Quoted 4 lines · David Hildenbrand

Just a thought: If memory hotplug is applicable as well, you might either want to always assume data->enable_4GB, or handle memory hotplug events from the memory notifier, when new memory gets onlined (not sure how tricky that is).

We probably want a highest_pfn_possible() or similar API instead of having drivers poking into random VM internals.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#5 lore raw
Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · David Hildenbrand

Just a thought: If memory hotplug is applicable as well, you might either want to always assume data->enable_4GB, or handle memory hotplug events from the memory notifier, when new memory gets onlined (not sure how tricky that is).

We probably want a highest_pfn_possible() or similar API instead of having drivers poking into random VM internals.

Well, memory notifiers are a reasonable api used accross the kernel to get notified when new memory is onlined to the buddy that could be used for allocations.

highest_pfn_possible() would have to default to something linked to MAX_PHYSMEM_BITS whenever memory hotplug is configured, I am not sure how helpful that is (IOW, you can just default to enable_4GB=true in that case instead in most cases).

Signature · 3 lines
Thanks,

David / dhildenb
#6 lore raw
Quoted 10 lines · David Hildenbrand
Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · David Hildenbrand

Just a thought: If memory hotplug is applicable as well, you might either want to always assume data->enable_4GB, or handle memory hotplug events from the memory notifier, when new memory gets onlined (not sure how tricky that is).

We probably want a highest_pfn_possible() or similar API instead of having drivers poking into random VM internals.

Quoted 8 lines

Well, memory notifiers are a reasonable api used accross the kernel to get notified when new memory is onlined to the buddy that could be used for allocations.

highest_pfn_possible() would have to default to something linked to MAX_PHYSMEM_BITS whenever memory hotplug is configured, I am not sure how helpful that is (IOW, you can just default to enable_4GB=true in that case instead in most cases).

Correction: At least on x86-64 we have max_possible_pfn, which will consult the ACPI SRAT table to figure out the maximum possible PFN. (Without SRAT, max_possible_pfn will point at the end of initial boot memory and not consider hotplug memory - something that e.g., newer QEMU versions work around by creating SRAT tables if memory hotplug might be possible, even if there is no actual NUMA configuration).

pci-swiotlb.c similarly relies on that to figure out if there are any !DMA addresses to handle.

Signature · 8 lines
Thanks,

David / dhildenb

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu