Skip to contentSkip to conversation

Optimise cache-flushing system call and add iovec variant

PATCH New Will Deacon 3 participants 13 messages Mar 25, 2013 – May 23, 2013 Arm (32-bit)

lore
Share & tools
lore
#0 lore raw

cover Optimise cache-flushing system call and add iovec variant

Hello,

This patch series has been sitting on my box for some time, pending benchmark results and analysis.

The results are that, with the first two patches in this series, we see ~2% improvement in browser benchmark scores, which flush the cache fairly regularly in response to jitting JS code.

The final patch is more of an RFC, since I don't have any software that uses it outside of my toy programs.

All feedback welcome,

Will

Will Deacon (3):
ARM: cacheflush: don't round address range up to nearest page
ARM: cacheflush: don't bother rounding to nearest vma
ARM: cacheflush: add new iovec-based cache flushing system call

 arch/arm/include/asm/cacheflush.h  |  3 +--
 arch/arm/include/uapi/asm/unistd.h |  1 +
 arch/arm/kernel/traps.c            | 49 +++++++++++++++++++++++++++-----------
 3 files changed, 37 insertions(+), 16 deletions(-)
Signature · 1 line
1.8.0
#1 lore raw

PATCH 1/3 ARM: cacheflush: don't round address range up to nearest page

The flush_cache_user_range macro takes a pair of addresses describing the start and end of the virtual address range to flush. Due to an accidental oversight when flush_cache_range_user was introduced, the address range was rounded up so that the start and end addresses were page-aligned.

For historical reference, the interesting commits in history.git are:

10eacf1775e1 ("[ARM] Clean up ARM cache handling interfaces (part 1)") 71432e79b76b ("[ARM] Add flush_cache_user_page() for sys_cacheflush()")

This patch removes the alignment code, reducing the amount of flushing required for ranges that are not an exact multiple of PAGE_SIZE.

---
 arch/arm/include/asm/cacheflush.h | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/arm/include/asm/cacheflush.h +1 -2
index e1489c5..2965ad1 100644
@@ -268,8 +268,7 @@ extern void flush_cache_page(struct vm_area_struct *vma, unsigned long user_addr
  * Harvard caches are synchronised for the user space address range.
  * This is used for the ARM private sys_cacheflush system call.
  */
-#define flush_cache_user_range(start,end) \
-	__cpuc_coherent_user_range((start) & PAGE_MASK, PAGE_ALIGN(end))
+#define flush_cache_user_range(s,e)	__cpuc_coherent_user_range(s,e)
 
 /*
  * Perform necessary cache operations to ensure that data previously
Signature · 1 line
1.8.0
#2 lore raw

PATCH 1/3 ARM: cacheflush: don't round address range up to nearest page

Quoted 5 lines · Will Deacon

The flush_cache_user_range macro takes a pair of addresses describing the start and end of the virtual address range to flush. Due to an accidental oversight when flush_cache_range_user was introduced, the address range was rounded up so that the start and end addresses were page-aligned.

I don't think it was an accidental oversight but it was actually covering other issue where writing an instruction caused a full page CoW. We've fixed this issue since, so the patch is ok.

#3 lore raw

PATCH 2/3 ARM: cacheflush: don't bother rounding to nearest vma

In reply to Will Deacon

do_cache_op finds the lowest VMA contained in the specified address range and rounds the range to cover only the mapped addresses.

Since commit 4542b6a0fa6b ("ARM: 7365/1: drop unused parameter from flush_cache_user_range") the VMA is not used for anything else in this code and seeing as the low-level cache flushing routines return -EFAULT if the address is not valid, there is no need for this range truncation.

This patch removes the VMA handling code from the cacheflushing syscall.

---
 arch/arm/kernel/traps.c | 17 +----------------
 1 file changed, 1 insertion(+), 16 deletions(-)
arch/arm/kernel/traps.c +1 -16
index 1c08911..da5e268 100644
@@ -509,25 +509,10 @@ static int bad_syscall(int n, struct pt_regs *regs)
 static inline int
 do_cache_op(unsigned long start, unsigned long end, int flags)
 {
-	struct mm_struct *mm = current->active_mm;
-	struct vm_area_struct *vma;
-
 	if (end < start || flags)
 		return -EINVAL;
 
-	down_read(&mm->mmap_sem);
-	vma = find_vma(mm, start);
-	if (vma && vma->vm_start < end) {
-		if (start < vma->vm_start)
-			start = vma->vm_start;
-		if (end > vma->vm_end)
-			end = vma->vm_end;
-
-		up_read(&mm->mmap_sem);
-		return flush_cache_user_range(start, end);
-	}
-	up_read(&mm->mmap_sem);
-	return -EINVAL;
+	return flush_cache_user_range(start, end);
 }
 
 /*
Signature · 1 line
1.8.0
#4 lore raw

PATCH 2/3 ARM: cacheflush: don't bother rounding to nearest vma

Quoted 33 lines · Will Deacon

do_cache_op finds the lowest VMA contained in the specified address range and rounds the range to cover only the mapped addresses.

Since commit 4542b6a0fa6b ("ARM: 7365/1: drop unused parameter from flush_cache_user_range") the VMA is not used for anything else in this code and seeing as the low-level cache flushing routines return -EFAULT if the address is not valid, there is no need for this range truncation.

This patch removes the VMA handling code from the cacheflushing syscall.

---
 arch/arm/kernel/traps.c | 17 +----------------
 1 file changed, 1 insertion(+), 16 deletions(-)
arch/arm/kernel/traps.c +0 -7
index 1c08911..da5e268 100644
@@ -509,25 +509,10 @@ static int bad_syscall(int n, struct pt_regs *regs)
 static inline int
 do_cache_op(unsigned long start, unsigned long end, int flags)
 {
-	struct mm_struct *mm = current->active_mm;
-	struct vm_area_struct *vma;
-
 	if (end < start || flags)
 		return -EINVAL;
 
-	down_read(&mm->mmap_sem);
-	vma = find_vma(mm, start);
-	if (vma && vma->vm_start < end) {
-		if (start < vma->vm_start)

arch/arm/kernel/traps.c static int bad_syscall(int n, struct pt_regs *regs) · line 515

Quoted 10 lines
arch/arm/kernel/traps.c +1 -9
-			start = vma->vm_start;
-		if (end > vma->vm_end)
-			end = vma->vm_end;
-
-		up_read(&mm->mmap_sem);
-		return flush_cache_user_range(start, end);
-	}
-	up_read(&mm->mmap_sem);
-	return -EINVAL;
+	return flush_cache_user_range(start, end);

While this would work, it introduces a possibility of DoS where an application passes bigger valid range (kernel linear mapping) and the kernel code would not be preempted (CONFIG_PREEMPT disabled). IIRC, that's why Russell reject such patch a while back.

Signature · 1 line
Catalin
#5 lore raw

PATCH 2/3 ARM: cacheflush: don't bother rounding to nearest vma

Hello,

Quoted 26 lines · Catalin Marinas
Quoted 25 lines · Will Deacon
arch/arm/kernel/traps.c +0 -14
index 1c08911..da5e268 100644
@@ -509,25 +509,10 @@ static int bad_syscall(int n, struct pt_regs *regs)
 static inline int
 do_cache_op(unsigned long start, unsigned long end, int flags)
 {
-	struct mm_struct *mm = current->active_mm;
-	struct vm_area_struct *vma;
-
 	if (end < start || flags)
 		return -EINVAL;
 
-	down_read(&mm->mmap_sem);
-	vma = find_vma(mm, start);
-	if (vma && vma->vm_start < end) {
-		if (start < vma->vm_start)
-			start = vma->vm_start;
-		if (end > vma->vm_end)
-			end = vma->vm_end;
-
-		up_read(&mm->mmap_sem);
-		return flush_cache_user_range(start, end);
-	}
Quoted 8 lines

arch/arm/kernel/traps.c static int bad_syscall(int n, struct pt_regs *regs) · line 515

Quoted 3 lines
arch/arm/kernel/traps.c +1 -2
-	up_read(&mm->mmap_sem);
-	return -EINVAL;
+	return flush_cache_user_range(start, end);

While this would work, it introduces a possibility of DoS where an application passes bigger valid range (kernel linear mapping) and the kernel code would not be preempted (CONFIG_PREEMPT disabled). IIRC, that's why Russell reject such patch a while back.

Hmm, I'm not sure I buy that argument. Firstly, you can't just pass a kernel linear mapping address -- we'll fault straight away because it's not a userspace address. Secondly, what's to stop an application from mmaping a large area into a single VMA and giving rise to the same situation? Finally, interrupts are enabled during this operation, so I don't understand how you can trigger a DoS, irrespective of the preempt configuration.

Is there an old thread I can refer to with more details about this? It may be that some of the assumptions there no longer hold with subsequent changes to the fault handling on this path.

Will

#6 lore raw

PATCH 2/3 ARM: cacheflush: don't bother rounding to nearest vma

Quoted 36 lines · Will Deacon
Quoted 26 lines · Catalin Marinas
Quoted 25 lines · Will Deacon
arch/arm/kernel/traps.c +0 -14
index 1c08911..da5e268 100644
@@ -509,25 +509,10 @@ static int bad_syscall(int n, struct pt_regs *regs)
 static inline int
 do_cache_op(unsigned long start, unsigned long end, int flags)
 {
-	struct mm_struct *mm = current->active_mm;
-	struct vm_area_struct *vma;
-
 	if (end < start || flags)
 		return -EINVAL;
 
-	down_read(&mm->mmap_sem);
-	vma = find_vma(mm, start);
-	if (vma && vma->vm_start < end) {
-		if (start < vma->vm_start)
-			start = vma->vm_start;
-		if (end > vma->vm_end)
-			end = vma->vm_end;
-
-		up_read(&mm->mmap_sem);
-		return flush_cache_user_range(start, end);
-	}
Quoted 8 lines

arch/arm/kernel/traps.c static int bad_syscall(int n, struct pt_regs *regs) · line 515

Quoted 3 lines
arch/arm/kernel/traps.c +1 -2
-	up_read(&mm->mmap_sem);
-	return -EINVAL;
+	return flush_cache_user_range(start, end);

While this would work, it introduces a possibility of DoS where an application passes bigger valid range (kernel linear mapping) and the kernel code would not be preempted (CONFIG_PREEMPT disabled). IIRC, that's why Russell reject such patch a while back.

Quoted 3 lines

Hmm, I'm not sure I buy that argument. Firstly, you can't just pass a kernel linear mapping address -- we'll fault straight away because it's not a userspace address.

Fault where?

Quoted 4 lines

Secondly, what's to stop an application from mmaping a large area into a single VMA and giving rise to the same situation? Finally, interrupts are enabled during this operation, so I don't understand how you can trigger a DoS, irrespective of the preempt configuration.

You can prevent context switching to other threads. But I agree, with a large vma (which is already faulted in), you can get similar behaviour.

Quoted 3 lines

Is there an old thread I can refer to with more details about this? It may be that some of the assumptions there no longer hold with subsequent changes to the fault handling on this path.

You could search the list for "do_cache_op", I don't have any at hand.

Signature · 1 line
Catalin
#7 lore raw

PATCH 2/3 ARM: cacheflush: don't bother rounding to nearest vma

Quoted 7 lines · Catalin Marinas
Quoted 6 lines · Will Deacon
Quoted 4 lines · Catalin Marinas

While this would work, it introduces a possibility of DoS where an application passes bigger valid range (kernel linear mapping) and the kernel code would not be preempted (CONFIG_PREEMPT disabled). IIRC, that's why Russell reject such patch a while back.

Quoted 5 lines
Quoted 3 lines

Hmm, I'm not sure I buy that argument. Firstly, you can't just pass a kernel linear mapping address -- we'll fault straight away because it's not a userspace address.

Fault where?

I was expecting something like an access_ok check, but you're right, we don't have one (and I guess that's not strictly needed given that flushing isn't destructive). I still find it a bit scary that we allow userspace to pass kernel addresses through though -- especially if there's something like a DMA or CPU suspend operation running on another core.

Quoted 7 lines
Quoted 4 lines

Secondly, what's to stop an application from mmaping a large area into a single VMA and giving rise to the same situation? Finally, interrupts are enabled during this operation, so I don't understand how you can trigger a DoS, irrespective of the preempt configuration.

You can prevent context switching to other threads. But I agree, with a large vma (which is already faulted in), you can get similar behaviour.

So the easy fix is to split the range up into chunks and call cond_resched after processing each one.

Will

#8 lore raw

PATCH 2/3 ARM: cacheflush: don't bother rounding to nearest vma

Quoted 12 lines · Will Deacon
Quoted 11 lines · Catalin Marinas
Quoted 9 lines · Will Deacon
Quoted 4 lines · Catalin Marinas

While this would work, it introduces a possibility of DoS where an application passes bigger valid range (kernel linear mapping) and the kernel code would not be preempted (CONFIG_PREEMPT disabled). IIRC, that's why Russell reject such patch a while back.

Hmm, I'm not sure I buy that argument. Firstly, you can't just pass a kernel linear mapping address -- we'll fault straight away because it's not a userspace address.

Quoted 7 lines
Quoted 1 line

Fault where?

I was expecting something like an access_ok check, but you're right, we don't have one (and I guess that's not strictly needed given that flushing isn't destructive). I still find it a bit scary that we allow userspace to pass kernel addresses through though -- especially if there's something like a DMA or CPU suspend operation running on another core.

Currently we don't allow kernel addresses since we require a valid vma. If we drop the vma search, we should add an access_ok.

Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

Secondly, what's to stop an application from mmaping a large area into a single VMA and giving rise to the same situation? Finally, interrupts are enabled during this operation, so I don't understand how you can trigger a DoS, irrespective of the preempt configuration.

You can prevent context switching to other threads. But I agree, with a large vma (which is already faulted in), you can get similar behaviour.

So the easy fix is to split the range up into chunks and call cond_resched after processing each one.

This would work.

Signature · 1 line
Catalin
#9 lore raw

PATCH 3/3 ARM: cacheflush: add new iovec-based cache flushing system call

In reply to Will Deacon

The cacheflush system call flushes a contiguous range, specified by a pair of addresses. This means that user applications wishing to flush discrete ranges must issue multiple system calls, since attempting to flush past a vma results in range truncation.

This patch introduces a new private system call for ARM, cacheflush_iov, which processes a range of addresses described in an iovec structure.

---
 arch/arm/include/uapi/asm/unistd.h |  1 +
 arch/arm/kernel/traps.c            | 36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 37 insertions(+)
arch/arm/include/uapi/asm/unistd.h +1 -0
index af33b44..bcad38c 100644
@@ -421,6 +421,7 @@
 #define __ARM_NR_usr26			(__ARM_NR_BASE+3)
 #define __ARM_NR_usr32			(__ARM_NR_BASE+4)
 #define __ARM_NR_set_tls		(__ARM_NR_BASE+5)
+#define __ARM_NR_cacheflush_iov		(__ARM_NR_BASE+6)
 
 /*
  * *NOTE*: This is a ghost syscall private to the kernel.  Only the
arch/arm/kernel/traps.c +36 -0
index da5e268..f83eed6 100644
@@ -25,6 +25,7 @@
 #include <linux/delay.h>
 #include <linux/init.h>
 #include <linux/sched.h>
+#include <linux/slab.h>
 
 #include <linux/atomic.h>
 #include <asm/cacheflush.h>
@@ -515,6 +516,37 @@ do_cache_op(unsigned long start, unsigned long end, int flags)
 	return flush_cache_user_range(start, end);
 }
 
+static inline int
+do_cache_op_iov(const struct iovec __user *uiov, unsigned long cnt, int flags)
+{
+	int i, ret = 0;
+	unsigned long len = cnt * sizeof(struct iovec);
+	struct iovec *iov = kmalloc(len, GFP_KERNEL);
+
+	if (iov == NULL) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	if (copy_from_user(iov, uiov, len)) {
+		ret = -EFAULT;
+		goto out_free;
+	}
+
+	for (i = 0; i < cnt; ++i) {
+		unsigned long start = (unsigned long __force)iov[i].iov_base;
+		unsigned long end = start + iov[i].iov_len;
+		ret = do_cache_op(start, end, flags);
+		if (ret)
+			break;
+	}
+
+out_free:
+	kfree(iov);
+out:
+	return ret;
+}
+
 /*
  * Handle all unrecognised system calls.
  *  0x9f0000 - 0x9fffff are some more esoteric system calls
@@ -560,6 +592,10 @@ asmlinkage int arm_syscall(int no, struct pt_regs *regs)
 	case NR(cacheflush):
 		return do_cache_op(regs->ARM_r0, regs->ARM_r1, regs->ARM_r2);
 
+	case NR(cacheflush_iov):
+		return do_cache_op_iov((const struct iovec __user *)regs->ARM_r0,
+					regs->ARM_r1, regs->ARM_r2);
+
 	case NR(usr26):
 		if (!(elf_hwcap & HWCAP_26BIT))
 			break;
Signature · 1 line
1.8.0
#10 lore raw

PATCH 3/3 ARM: cacheflush: add new iovec-based cache flushing system call

Quoted 56 lines · Will Deacon

The cacheflush system call flushes a contiguous range, specified by a pair of addresses. This means that user applications wishing to flush discrete ranges must issue multiple system calls, since attempting to flush past a vma results in range truncation.

This patch introduces a new private system call for ARM, cacheflush_iov, which processes a range of addresses described in an iovec structure.

---
 arch/arm/include/uapi/asm/unistd.h |  1 +
 arch/arm/kernel/traps.c            | 36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 37 insertions(+)
arch/arm/include/uapi/asm/unistd.h +1 -0
index af33b44..bcad38c 100644
@@ -421,6 +421,7 @@
 #define __ARM_NR_usr26			(__ARM_NR_BASE+3)
 #define __ARM_NR_usr32			(__ARM_NR_BASE+4)
 #define __ARM_NR_set_tls		(__ARM_NR_BASE+5)
+#define __ARM_NR_cacheflush_iov		(__ARM_NR_BASE+6)
 
 /*
  * *NOTE*: This is a ghost syscall private to the kernel.  Only the
arch/arm/kernel/traps.c +15 -0
index da5e268..f83eed6 100644
@@ -25,6 +25,7 @@
 #include <linux/delay.h>
 #include <linux/init.h>
 #include <linux/sched.h>
+#include <linux/slab.h>
 
 #include <linux/atomic.h>
 #include <asm/cacheflush.h>
@@ -515,6 +516,37 @@ do_cache_op(unsigned long start, unsigned long end, int flags)
 	return flush_cache_user_range(start, end);
 }
 
+static inline int
+do_cache_op_iov(const struct iovec __user *uiov, unsigned long cnt, int flags)
+{
+	int i, ret = 0;
+	unsigned long len = cnt * sizeof(struct iovec);
+	struct iovec *iov = kmalloc(len, GFP_KERNEL);
+
+	if (iov == NULL) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	if (copy_from_user(iov, uiov, len)) {
+		ret = -EFAULT;

arch/arm/kernel/traps.c do_cache_op(unsigned long start, unsigned long end, int flags) · line 542

Quoted 10 lines
arch/arm/kernel/traps.c +10 -0
+		goto out_free;
+	}
+
+	for (i = 0; i < cnt; ++i) {
+		unsigned long start = (unsigned long __force)iov[i].iov_base;
+		unsigned long end = start + iov[i].iov_len;
+		ret = do_cache_op(start, end, flags);
+		if (ret)
+			break;
+	}

Could you use get_user() on struct iovec fields directly to avoid kmalloc? It would be slightly faster.

Signature · 1 line
Catalin
#11 lore raw

PATCH 3/3 ARM: cacheflush: add new iovec-based cache flushing system call

Quoted 20 lines · Catalin Marinas
Quoted 19 lines · Will Deacon
arch/arm/kernel/traps.c +19 -0
+static inline int
+do_cache_op_iov(const struct iovec __user *uiov, unsigned long cnt, int flags)
+{
+	int i, ret = 0;
+	unsigned long len = cnt * sizeof(struct iovec);
+	struct iovec *iov = kmalloc(len, GFP_KERNEL);
+
+	if (iov == NULL) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	if (copy_from_user(iov, uiov, len)) {
+		ret = -EFAULT;
+		goto out_free;
+	}
+
+	for (i = 0; i < cnt; ++i) {
+		unsigned long start = (unsigned long __force)iov[i].iov_base;
Quoted 8 lines

arch/arm/kernel/traps.c do_cache_op(unsigned long start, unsigned long end, int flags) · line 542

Quoted 5 lines
arch/arm/kernel/traps.c +5 -0
+		unsigned long end = start + iov[i].iov_len;
+		ret = do_cache_op(start, end, flags);
+		if (ret)
+			break;
+	}

Could you use get_user() on struct iovec fields directly to avoid kmalloc? It would be slightly faster.

I just had a look at this, but the problem is that we want to know the entire (combined) region size so that we may just decide to flush the entire cache, rather than iterate over each region line-by-line. This requires copying the whole iovec into kernel space up front, so that we can iterate over the regions with iov_length.

Will

#12 lore raw

cover Optimise cache-flushing system call and add iovec variant

In reply to Will Deacon

Hi Will,

Quoted 5 lines · Will Deacon

Hello,

This patch series has been sitting on my box for some time, pending benchmark results and analysis.

Quoted 9 lines

The results are that, with the first two patches in this series, we see ~2% improvement in browser benchmark scores, which flush the cache fairly regularly in response to jitting JS code.

The final patch is more of an RFC, since I don't have any software that uses it outside of my toy programs.

All feedback welcome,

For all three patches

The last one was tested with a modified jit that was hacked to use the new API as a direct replacement for the old one (IE no cleverness with delayed flushing/grouping of calls). The performance decrease from using the new call instead of the old one was well within the noise of the tests. We are, however, yet to test the improvement from actually *exploiting* the new mechanism. If there's a positive response to the idea then I can encourage the jit guys to have another look.

Jonny

Quoted 13 lines

Will

Will Deacon (3):
ARM: cacheflush: don't round address range up to nearest page
ARM: cacheflush: don't bother rounding to nearest vma
ARM: cacheflush: add new iovec-based cache flushing system call

  arch/arm/include/asm/cacheflush.h  |  3 +--
  arch/arm/include/uapi/asm/unistd.h |  1 +
  arch/arm/kernel/traps.c            | 49 +++++++++++++++++++++++++++-----------
  3 files changed, 37 insertions(+), 16 deletions(-)