Skip to contentSkip to conversation

arm: highmem: Add support for flushing kmap_atomic mappings

RFC PATCH New Laura Abbott 3 participants 5 messages Apr 5, 2013 – Apr 22, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 arm: highmem: Add support for flushing kmap_atomic mappings

The highmem code provides kmap_flush_unused to ensure all kmap mappings are really removed if they are used. This code does not handle kmap_atomic mappings since they are managed separately. This prevents an issue for any code which relies on having absolutely no mappings for a particular page. Rather than pay the penalty of having CONFIG_DEBUG_HIGHMEM on all the time, add functionality to remove the kmap_atomic mappings in a similar way to kmap_flush_unused.

This is intended to be an RFC to make sure this approach is reasonable. The goal is to have kmap_atomic_flush_unused be a public API.

---
 arch/arm/mm/highmem.c |   51 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 51 insertions(+), 0 deletions(-)
arch/arm/mm/highmem.c +51 -0
index 21b9e1b..f4c0466 100644
@@ -10,6 +10,7 @@
  * published by the Free Software Foundation.
  */
 
+#include <linux/cpu.h>
 #include <linux/module.h>
 #include <linux/highmem.h>
 #include <linux/interrupt.h>
@@ -135,3 +136,53 @@ struct page *kmap_atomic_to_page(const void *ptr)
 
 	return pte_page(get_top_pte(vaddr));
 }
+
+static void kmap_remove_unused_cpu(int cpu)
+{
+	int start_idx, idx, type;
+
+	pagefault_disable();
+	type = kmap_atomic_idx();
+	start_idx = type + 1 + KM_TYPE_NR * cpu;
+
+	for (idx = start_idx; idx < KM_TYPE_NR + KM_TYPE_NR * cpu; idx++) {
+		unsigned long vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
+		set_top_pte(vaddr, __pte(0));
+	}
+	pagefault_enable();
+}
+
+static void kmap_remove_unused(void *unused)
+{
+	kmap_remove_unused_cpu(smp_processor_id());
+}
+
+void kmap_atomic_flush_unused(void)
+{
+	on_each_cpu(kmap_remove_unused, NULL, 1);
+}
+
+static int hotplug_kmap_atomic_callback(struct notifier_block *nfb,
+                                unsigned long action, void *hcpu)
+{
+        switch (action & (~CPU_TASKS_FROZEN)) {
+        case CPU_DYING:
+		kmap_remove_unused_cpu((int)hcpu);
+                break;
+        default:
+                break;
+        }
+
+        return NOTIFY_OK;
+}
+
+static struct notifier_block hotplug_kmap_atomic_notifier = {
+        .notifier_call = hotplug_kmap_atomic_callback,
+};
+
+static int __init init_kmap_atomic(void)
+{
+
+        return register_hotcpu_notifier(&hotplug_kmap_atomic_notifier);
+}
+early_initcall(init_kmap_atomic);
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
hosted by The Linux Foundation
#1 lore raw

RFC PATCH 1/1 arm: highmem: Add support for flushing kmap_atomic mappings

Quoted 2 lines · Laura Abbott

The highmem code provides kmap_flush_unused to ensure all kmap mappings are really removed if they are used. This code does not

You meant "if they are *not* used", right?

Quoted 5 lines

handle kmap_atomic mappings since they are managed separately. This prevents an issue for any code which relies on having absolutely no mappings for a particular page. Rather than pay the penalty of having CONFIG_DEBUG_HIGHMEM on all the time, add functionality to remove the kmap_atomic mappings in a similar way to kmap_flush_unused.

Could you elaborate on that code that relies on having absolutely no mappings for a particular page please?

Quoted 3 lines

This is intended to be an RFC to make sure this approach is reasonable. The goal is to have kmap_atomic_flush_unused be a public API.

The clearing code is going to be costly since you do a set_top_pte(vaddr, __pte(0)) unconditionally on the whole range, regardless if the PTE is already set to 0.

Using it via an hotplug callback is rather strange, but I'm assuming that this was only for testing?

Nicolas

#2 lore raw

RFC PATCH 1/1 arm: highmem: Add support for flushing kmap_atomic mappings

Quoted 7 lines · Nicolas Pitre
Quoted 2 lines · Laura Abbott

The highmem code provides kmap_flush_unused to ensure all kmap mappings are really removed if they are used. This code does not

You meant "if they are *not* used", right?

Yes, missed a word there

Quoted 9 lines
Quoted 5 lines

handle kmap_atomic mappings since they are managed separately. This prevents an issue for any code which relies on having absolutely no mappings for a particular page. Rather than pay the penalty of having CONFIG_DEBUG_HIGHMEM on all the time, add functionality to remove the kmap_atomic mappings in a similar way to kmap_flush_unused.

Could you elaborate on that code that relies on having absolutely no mappings for a particular page please?

We have a use case where we pass memory to trustzone to have it protected such that the non-secure environment may not read or write that memory. The protecting/unprotecting can happen at runtime. If there are any valid mappings in the page tables, the CPU is free to speculatively access that memory. If the CPU speculates into a protected region while in the non-secure world, we get a fault violation. Essentially this means that even if we reserve the memory at bootup time with memblock_reserve, if the memory was ever previously mapped with kmap_atomic (to flush caches for example) we could still end up with stray mappings which can lead to faults.

In general, it seems like this is missing functionality from the intended behavior of kmap_flush_unused which is to get rid of all stray mappings.

Quoted 8 lines
Quoted 3 lines

This is intended to be an RFC to make sure this approach is reasonable. The goal is to have kmap_atomic_flush_unused be a public API.

The clearing code is going to be costly since you do a set_top_pte(vaddr, __pte(0)) unconditionally on the whole range, regardless if the PTE is already set to 0.

Good point. I'll add a check for that.

Quoted 3 lines

Using it via an hotplug callback is rather strange, but I'm assuming that this was only for testing?

The hotplug callback is needed because we clear the mappings per-CPU. If a CPU is hotplugged out with stray mappings they will not be cleared since on_each_cpu only works on online CPUs.

Quoted 3 lines

Nicolas

Thanks,
Laura

Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation
#3 lore raw

RFC PATCH 1/1 arm: highmem: Add support for flushing kmap_atomic mappings

In reply to Laura Abbott
Quoted 7 lines · Laura Abbott

The highmem code provides kmap_flush_unused to ensure all kmap mappings are really removed if they are used. This code does not handle kmap_atomic mappings since they are managed separately. This prevents an issue for any code which relies on having absolutely no mappings for a particular page. Rather than pay the penalty of having CONFIG_DEBUG_HIGHMEM on all the time, add functionality to remove the kmap_atomic mappings in a similar way to kmap_flush_unused.

I get the feeling that there's something behind this which is pushing for there only being no mappings for a particular page. In a SMP environment that sounds like a dodgy assumption. Maybe you could provide the background behind this patch?

Quoted 3 lines

This is intended to be an RFC to make sure this approach is reasonable. The goal is to have kmap_atomic_flush_unused be a public API.

That implies that there's some callers which need it, which aren't part of this patch - and a motivation for this outside of what we can see here. That needs to be explained. Plus, unused APIs in the mainline kernel don't tend ot stick around for long, so it really needs a use case to be demonstrated before it can be merged.

Quoted 26 lines
---
 arch/arm/mm/highmem.c |   51 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 51 insertions(+), 0 deletions(-)
arch/arm/mm/highmem.c +5 -0
index 21b9e1b..f4c0466 100644
@@ -10,6 +10,7 @@
  * published by the Free Software Foundation.
  */
 
+#include <linux/cpu.h>
 #include <linux/module.h>
 #include <linux/highmem.h>
 #include <linux/interrupt.h>
@@ -135,3 +136,53 @@ struct page *kmap_atomic_to_page(const void *ptr)
 
 	return pte_page(get_top_pte(vaddr));
 }
+
+static void kmap_remove_unused_cpu(int cpu)
+{
+	int start_idx, idx, type;

arch/arm/mm/highmem.c struct page *kmap_atomic_to_page(const void *ptr) · line 152

Quoted 10 lines
arch/arm/mm/highmem.c +10 -0
+
+	pagefault_disable();
+	type = kmap_atomic_idx();
+	start_idx = type + 1 + KM_TYPE_NR * cpu;
+
+	for (idx = start_idx; idx < KM_TYPE_NR + KM_TYPE_NR * cpu; idx++) {
+		unsigned long vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
+		set_top_pte(vaddr, __pte(0));
+	}
+	pagefault_enable();

Hmm. Why are you using pagefault_disable() around this? This code itself should not cause any page faults. Maybe you mean to disable preemption, but that should already be disabled (if not the above is buggy.) Maybe interrupts - that would make more sense, but as soon as interrupts are re-enabled, it's possible for drivers to start create kmap_atomic() mappings again.

So... the question is... why this... for what use case?

At the moment it just feels like it's rather buggy and racy.

arch/arm/mm/highmem.c struct page *kmap_atomic_to_page(const void *ptr) · line 168

Quoted 4 lines
arch/arm/mm/highmem.c +4 -0
+static int hotplug_kmap_atomic_callback(struct notifier_block *nfb,
+                                unsigned long action, void *hcpu)
+{
+        switch (action & (~CPU_TASKS_FROZEN)) {

Parens around ~CPU_TASKS_FROZEN is not required.

arch/arm/mm/highmem.c struct page *kmap_atomic_to_page(const void *ptr) · line 176

Quoted 8 lines
arch/arm/mm/highmem.c +8 -0
+        case CPU_DYING:
+		kmap_remove_unused_cpu((int)hcpu);
+                break;
+        default:
+                break;
+        }
+
+        return NOTIFY_OK;

Please watch your indentation...

arch/arm/mm/highmem.c struct page *kmap_atomic_to_page(const void *ptr) · line 185

Quoted 9 lines
arch/arm/mm/highmem.c +9 -0
+}
+
+static struct notifier_block hotplug_kmap_atomic_notifier = {
+        .notifier_call = hotplug_kmap_atomic_callback,
+};
+
+static int __init init_kmap_atomic(void)
+{
+

And this blank line isn't needed.

#4 lore raw

RFC PATCH 1/1 arm: highmem: Add support for flushing kmap_atomic mappings

Quoted 9 lines · Russell King - ARM Linux
Quoted 7 lines · Laura Abbott

The highmem code provides kmap_flush_unused to ensure all kmap mappings are really removed if they are used. This code does not handle kmap_atomic mappings since they are managed separately. This prevents an issue for any code which relies on having absolutely no mappings for a particular page. Rather than pay the penalty of having CONFIG_DEBUG_HIGHMEM on all the time, add functionality to remove the kmap_atomic mappings in a similar way to kmap_flush_unused.

Quoted 5 lines

I get the feeling that there's something behind this which is pushing for there only being no mappings for a particular page. In a SMP environment that sounds like a dodgy assumption. Maybe you could provide the background behind this patch?

Yes, I was asked this before and neglected to update the commit text.

We have a use case where we pass memory to trustzone to have it protected such that the non-secure environment may not read or write that memory. The protecting/unprotecting can happen at runtime. If there are any valid mappings in the page tables, the CPU is free to speculatively access that memory. If the CPU speculates into a protected region while in the non-secure world, we get a fault violation. Essentially this means that even if we reserve the memory at bootup time with memblock_reserve, if the memory was ever previously mapped with kmap_atomic (to flush caches for example) we could still end up with stray mappings which can lead to faults.

In general, it seems like this is missing functionality from the intended behavior of kmap_flush_unused which is to get rid of all stray mappings.

Quoted 54 lines
Quoted 3 lines

This is intended to be an RFC to make sure this approach is reasonable. The goal is to have kmap_atomic_flush_unused be a public API.

That implies that there's some callers which need it, which aren't part of this patch - and a motivation for this outside of what we can see here. That needs to be explained. Plus, unused APIs in the mainline kernel don't tend ot stick around for long, so it really needs a use case to be demonstrated before it can be merged.

Quoted 26 lines
---
  arch/arm/mm/highmem.c |   51 +++++++++++++++++++++++++++++++++++++++++++++++++
  1 files changed, 51 insertions(+), 0 deletions(-)
arch/arm/mm/highmem.c +5 -0
index 21b9e1b..f4c0466 100644
@@ -10,6 +10,7 @@
   * published by the Free Software Foundation.
   */
 
+#include <linux/cpu.h>
  #include <linux/module.h>
  #include <linux/highmem.h>
  #include <linux/interrupt.h>
@@ -135,3 +136,53 @@ struct page *kmap_atomic_to_page(const void *ptr)
 
  	return pte_page(get_top_pte(vaddr));
  }
+
+static void kmap_remove_unused_cpu(int cpu)
+{
+	int start_idx, idx, type;

arch/arm/mm/highmem.c struct page *kmap_atomic_to_page(const void *ptr) · line 152

Quoted 10 lines
arch/arm/mm/highmem.c +10 -0
+
+	pagefault_disable();
+	type = kmap_atomic_idx();
+	start_idx = type + 1 + KM_TYPE_NR * cpu;
+
+	for (idx = start_idx; idx < KM_TYPE_NR + KM_TYPE_NR * cpu; idx++) {
+		unsigned long vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
+		set_top_pte(vaddr, __pte(0));
+	}
+	pagefault_enable();

Hmm. Why are you using pagefault_disable() around this? This code itself should not cause any page faults. Maybe you mean to disable preemption, but that should already be disabled (if not the above is buggy.) Maybe interrupts - that would make more sense, but as soon as interrupts are re-enabled, it's possible for drivers to start create kmap_atomic() mappings again.

Quoted 4 lines

So... the question is... why this... for what use case?

At the moment it just feels like it's rather buggy and racy.

Yes, I think the pagefault disabling is overkill and preemption disabling is what I was going for. It is indeed possible for drivers to start creating kmap_atomic mappings again but I consider that a buggy driver design; any driver who intends for all mappings to really be gone needs to do proper locking to prevent against cases of getting new mappings.

Quoted 18 lines

arch/arm/mm/highmem.c struct page *kmap_atomic_to_page(const void *ptr) · line 168

Quoted 4 lines
arch/arm/mm/highmem.c +4 -0
+static int hotplug_kmap_atomic_callback(struct notifier_block *nfb,
+                                unsigned long action, void *hcpu)
+{
+        switch (action & (~CPU_TASKS_FROZEN)) {

Parens around ~CPU_TASKS_FROZEN is not required.

arch/arm/mm/highmem.c struct page *kmap_atomic_to_page(const void *ptr) · line 176

Quoted 8 lines
arch/arm/mm/highmem.c +8 -0
+        case CPU_DYING:
+		kmap_remove_unused_cpu((int)hcpu);
+                break;
+        default:
+                break;
+        }
+
+        return NOTIFY_OK;

Please watch your indentation...

Quoted 12 lines

arch/arm/mm/highmem.c struct page *kmap_atomic_to_page(const void *ptr) · line 185

Quoted 9 lines
arch/arm/mm/highmem.c +9 -0
+}
+
+static struct notifier_block hotplug_kmap_atomic_notifier = {
+        .notifier_call = hotplug_kmap_atomic_callback,
+};
+
+static int __init init_kmap_atomic(void)
+{
+

And this blank line isn't needed.

Yes, I missed the checkpatch cleanup.

Thanks,
Laura

Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation