Skip to contentSkip to conversation

ARM: copy_{from,to}_user() for vmsplit 4g/4g

RFC In review afzal mohammed 8 participants 27 messages Jun 12, 2020 – Jun 15, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover ARM: copy_{from,to}_user() for vmsplit 4g/4g

Hi,

copy_{from,to}_user() uaccess helpers are implemented by user page pinning, followed by temporary kernel mapping & then memcpy(). This helps to achieve user page copy when current virtual address mapping of the CPU excludes user pages.

Other uaccess routines are also planned to be modified to make use of pinning plus kmap_atomic() based on the feedback here.

This is done as one of the initial steps to achieve 4G virtual address mapping for user as well as Kernel on ARMv7 w/ LPAE.

Motive behind this is to enable Kernel access till 4GiB (almost) as lowmem, thus helping in removing highmem support for platforms having upto 4GiB RAM. In the case of platforms having >4GiB, highmem is still required for the Kernel to be able to access whole RAM.

Performance wise, results are not encouraging, 'dd' on tmpfs results,

ARM Cortex-A8, BeagleBone White (256MiB RAM):
w/o series - ~29.5 MB/s
w/ series - ~20.5 MB/s
w/ series & highmem disabled - ~21.2 MB/s

On Cortex-A15(2GiB RAM) in QEMU:
w/o series - ~4 MB/s
w/ series - ~2.6 MB/s

Roughly a one-third drop in performance. Disabling highmem improves performance only slightly.

'hackbench' also showed a similar pattern.

Ways to improve the performance has to be explored, if any one has thoughts on it, please share.

uaccess routines using page pinning & temporary kernel mapping is not something new, it has been done by Ingo long long ago [1] as part of 4G/4G user/kernel mapping implementation on x86, though not merged in mainline.

Arnd has outlined basic design for vmsplit 4g/4g, uaccess routines using user page pinning plus kmap_atomic() is one part of that.

[1] https://lore.kernel.org/lkml/[email protected]/

Last 2 patches are only meant for testing first patch.

Regards
afzal

afzal mohammed (3):
lib: copy_{from,to}_user using gup & kmap_atomic()
ARM: uaccess: let UACCESS_GUP_KMAP_MEMCPY enabling
ARM: provide CONFIG_VMSPLIT_4G_DEV for development

 arch/arm/Kconfig               |   9 ++
 arch/arm/include/asm/uaccess.h |  20 ++++
 arch/arm/kernel/armksyms.c     |   2 +
 arch/arm/lib/Makefile          |   7 +-
 lib/Kconfig                    |   4 +
 lib/Makefile                   |   3 +
 lib/uaccess_gup_kmap_memcpy.c  | 162 +++++++++++++++++++++++++++++++++
 7 files changed, 205 insertions(+), 2 deletions(-)
 create mode 100644 lib/uaccess_gup_kmap_memcpy.c
Signature · 1 line
2.26.2
#1 lore raw

RFC 1/3 lib: copy_{from,to}_user using gup & kmap_atomic()

copy_{from,to}_user() uaccess helpers are implemented by user page pinning, followed by temporary kernel mapping & then memcpy(). This helps to achieve user page copy when current virtual address mapping of the CPU excludes user pages.

Performance wise, results are not encouraging, 'dd' on tmpfs results,

ARM Cortex-A8, BeagleBone White (256MiB RAM):
w/o series - ~29.5 MB/s
w/ series - ~20.5 MB/s
w/ series & highmem disabled - ~21.2 MB/s

On Cortex-A15(2GiB RAM) in QEMU:
w/o series - ~4 MB/s
w/ series - ~2.6 MB/s

Roughly a one-third drop in performance. Disabling highmem improves performance only slightly.

'hackbench' also showed a similar pattern.

uaccess routines using page pinning & temporary kernel mapping is not something new, it has been done long long ago by Ingo [1] as part of 4G/4G user/kernel mapping implementation on x86, though not merged in mainline.

[1] https://lore.kernel.org/lkml/[email protected]/

---
 lib/Kconfig                   |   4 +
 lib/Makefile                  |   3 +
 lib/uaccess_gup_kmap_memcpy.c | 162 ++++++++++++++++++++++++++++++++++
 3 files changed, 169 insertions(+)
 create mode 100644 lib/uaccess_gup_kmap_memcpy.c
lib/Kconfig +4 -0
index 5d53f9609c252..dadf4f6cc391d 100644
@@ -622,6 +622,10 @@ config ARCH_HAS_MEMREMAP_COMPAT_ALIGN
 config UACCESS_MEMCPY
 	bool
 
+# pin page + kmap_atomic + memcpy for user copies, intended for vmsplit 4g/4g
+config UACCESS_GUP_KMAP_MEMCPY
+	bool
+
 config ARCH_HAS_UACCESS_FLUSHCACHE
 	bool
 
lib/Makefile +3 -0
index 685aee60de1d5..bc457f85e391a 100644
@@ -309,3 +309,6 @@ obj-$(CONFIG_OBJAGG) += objagg.o
 
 # KUnit tests
 obj-$(CONFIG_LIST_KUNIT_TEST) += list-test.o
+
+# uaccess
+obj-$(CONFIG_UACCESS_GUP_KMAP_MEMCPY) += uaccess_gup_kmap_memcpy.o
lib/uaccess_gup_kmap_memcpy.c new file +162 -0

162 lines not on this page · raw message

Signature · 1 line
2.26.2
#2 lore raw

Re: [RFC 1/3] lib: copy_{from,to}_user using gup & kmap_atomic()

Quoted 20 lines · afzal mohammed

copy_{from,to}_user() uaccess helpers are implemented by user page pinning, followed by temporary kernel mapping & then memcpy(). This helps to achieve user page copy when current virtual address mapping of the CPU excludes user pages.

Performance wise, results are not encouraging, 'dd' on tmpfs results,

ARM Cortex-A8, BeagleBone White (256MiB RAM):
w/o series - ~29.5 MB/s
w/ series - ~20.5 MB/s
w/ series & highmem disabled - ~21.2 MB/s

On Cortex-A15(2GiB RAM) in QEMU:
w/o series - ~4 MB/s
w/ series - ~2.6 MB/s

Roughly a one-third drop in performance. Disabling highmem improves performance only slightly.

Quoted 8 lines

'hackbench' also showed a similar pattern.

uaccess routines using page pinning & temporary kernel mapping is not something new, it has been done long long ago by Ingo [1] as part of 4G/4G user/kernel mapping implementation on x86, though not merged in mainline.

[1] https://lore.kernel.org/lkml/[email protected]/

Nice changelog text! I agree the performance drop is not great. There are probably some things that can be done to optimize it, but I guess most of the overhead is from the page table operations and cannot be avoided.

What was the exact 'dd' command you used, in particular the block size? Note that by default, 'dd' will request 512 bytes at a time, so you usually only access a single page. It would be interesting to see the overhead with other typical or extreme block sizes, e.g. '1', '64', '4K', '64K' or '1M'.

If you want to drill down into where exactly the overhead is (i.e. get_user_pages or kmap_atomic, or something different), using 'perf record dd ..', and 'perf report' may be helpful.

Quoted 7 lines
lib/uaccess_gup_kmap_memcpy.c new file +7 -0
+static int copy_chunk_from_user(unsigned long from, int len, void *arg)
+{
+       unsigned long *to_ptr = arg, to = *to_ptr;
+
+       memcpy((void *) to, (void *) from, len);
+       *to_ptr += len;
+       return 0;

lib/uaccess_gup_kmap_memcpy.c line 90

Quoted 10 lines
lib/uaccess_gup_kmap_memcpy.c new file +10 -0
+}
+
+static int copy_chunk_to_user(unsigned long to, int len, void *arg)
+{
+       unsigned long *from_ptr = arg, from = *from_ptr;
+
+       memcpy((void *) to, (void *) from, len);
+       *from_ptr += len;
+       return 0;
+}

Will gcc optimize away the indirect function call and inline everything? If not, that would be a small part of the overhead.

lib/uaccess_gup_kmap_memcpy.c line 103

Quoted 12 lines
lib/uaccess_gup_kmap_memcpy.c new file +12 -0
+unsigned long gup_kmap_copy_from_user(void *to, const void __user *from, unsigned long n)
+{
+       struct page **pages;
+       int num_pages, ret, i;
+
+       if (uaccess_kernel()) {
+               memcpy(to, (__force void *)from, n);
+               return 0;
+       }
+
+       num_pages = DIV_ROUND_UP((unsigned long)from + n, PAGE_SIZE) -
+                                (unsigned long)from / PAGE_SIZE;

Make sure this doesn't turn into actual division operations but uses shifts. It might even be clearer here to open-code the shift operation so readers can see what this is meant to compile into.

lib/uaccess_gup_kmap_memcpy.c line 106

Quoted 3 lines
lib/uaccess_gup_kmap_memcpy.c new file +3 -0
+       pages = kmalloc_array(num_pages, sizeof(*pages), GFP_KERNEL | __GFP_ZERO);
+       if (!pages)
+               goto end;

Another micro-optimization may be to avoid the kmalloc for the common case, e.g. anything with "num_pages <= 64", using an array on the stack.

lib/uaccess_gup_kmap_memcpy.c line 115

Quoted 8 lines
lib/uaccess_gup_kmap_memcpy.c new file +8 -0
+       ret = get_user_pages_fast((unsigned long)from, num_pages, 0, pages);
+       if (ret < 0)
+               goto free_pages;
+
+       if (ret != num_pages) {
+               num_pages = ret;
+               goto put_pages;
+       }

I think this is technically incorrect: if get_user_pages_fast() only gets some of the
pages, you should continue with the short buffer and return the number of remaining
bytes rather than not copying anything. I think you did that correctly for a failed
kmap_atomic(), but this has to use the same logic.

Arnd

#3 lore raw

Re: [RFC 1/3] lib: copy_{from,to}_user using gup & kmap_atomic()

Hi,

Quoted 8 lines · Arnd Bergmann
Quoted 2 lines · afzal mohammed

Roughly a one-third drop in performance. Disabling highmem improves performance only slightly.

There are probably some things that can be done to optimize it, but I guess most of the overhead is from the page table operations and cannot be avoided.

Ingo's series did a follow_page() first, then as a fallback did it invoke get_user_pages(), i will try that way as well.

Yes, i too feel get_user_pages_fast() path is the most time consuming, will instrument & check.

Quoted 4 lines

What was the exact 'dd' command you used, in particular the block size? Note that by default, 'dd' will request 512 bytes at a time, so you usually only access a single page. It would be interesting to see the overhead with other typical or extreme block sizes, e.g. '1', '64', '4K', '64K' or '1M'.

It was the default(512), more test results follows (in MB/s),

512 1K 4K 16K 32K 64K 1M

w/o series 30 46 89 95 90 85 65

w/ series 22 36 72 79 78 75 61

perf drop 26% 21% 19% 16% 13% 12% 6%

Hmm, results ain't that bad :)

Quoted 3 lines

If you want to drill down into where exactly the overhead is (i.e. get_user_pages or kmap_atomic, or something different), using 'perf record dd ..', and 'perf report' may be helpful.

Let me dig deeper & try to find out where the major overhead and try to figure out ways to reduce it.

One reason to disable highmem & test (results mentioned earlier) was to make kmap_atomic() very lightweight, that was not making much difference, around 3% only.

Quoted 12 lines
Quoted 12 lines
lib/uaccess_gup_kmap_memcpy.c new file +12 -0
+static int copy_chunk_from_user(unsigned long from, int len, void *arg)
+{
+       unsigned long *to_ptr = arg, to = *to_ptr;
+
+       memcpy((void *) to, (void *) from, len);
+       *to_ptr += len;
+       return 0;
+}
+
+static int copy_chunk_to_user(unsigned long to, int len, void *arg)
+{
+       unsigned long *from_ptr = arg, from = *from_ptr;
Quoted 8 lines

lib/uaccess_gup_kmap_memcpy.c line 90

Quoted 5 lines
lib/uaccess_gup_kmap_memcpy.c new file +5 -0
+
+       memcpy((void *) to, (void *) from, len);
+       *from_ptr += len;
+       return 0;
+}

Will gcc optimize away the indirect function call and inline everything? If not, that would be a small part of the overhead.

i think not, based on objdump, i will make these & wherever other places possible inline & see the difference.

Quoted 6 lines

lib/uaccess_gup_kmap_memcpy.c line 103

Quoted 2 lines
lib/uaccess_gup_kmap_memcpy.c new file +2 -0
+       num_pages = DIV_ROUND_UP((unsigned long)from + n, PAGE_SIZE) -
+                                (unsigned long)from / PAGE_SIZE;

Make sure this doesn't turn into actual division operations but uses shifts. It might even be clearer here to open-code the shift operation so readers can see what this is meant to compile into.

Okay

Quoted 7 lines

lib/uaccess_gup_kmap_memcpy.c line 106

Quoted 4 lines
lib/uaccess_gup_kmap_memcpy.c new file +3 -0
+       pages = kmalloc_array(num_pages, sizeof(*pages), GFP_KERNEL | __GFP_ZERO);
+       if (!pages)
+               goto end;

Another micro-optimization may be to avoid the kmalloc for the common case, e.g. anything with "num_pages <= 64", using an array on the stack.

Okay

Quoted 9 lines

lib/uaccess_gup_kmap_memcpy.c line 115

Quoted 8 lines
lib/uaccess_gup_kmap_memcpy.c new file +8 -0
+       ret = get_user_pages_fast((unsigned long)from, num_pages, 0, pages);
+       if (ret < 0)
+               goto free_pages;
+
+       if (ret != num_pages) {
+               num_pages = ret;
+               goto put_pages;
+       }
Quoted 7 lines

I think this is technically incorrect: if get_user_pages_fast() only gets some of the
pages, you should continue with the short buffer and return the number of remaining
bytes rather than not copying anything. I think you did that correctly for a failed
kmap_atomic(), but this has to use the same logic.

yes, will fix that.

Regards
afzal

#4 lore raw

Re: [RFC 1/3] lib: copy_{from,to}_user using gup & kmap_atomic()

Quoted 6 lines · afzal mohammed
Quoted 4 lines · Arnd Bergmann
Quoted 2 lines · afzal mohammed

Roughly a one-third drop in performance. Disabling highmem improves performance only slightly.

Quoted 6 lines
Quoted 3 lines

There are probably some things that can be done to optimize it, but I guess most of the overhead is from the page table operations and cannot be avoided.

Ingo's series did a follow_page() first, then as a fallback did it invoke get_user_pages(), i will try that way as well.

Right, that could help, in particular for the small copies. I think a lot of usercopy calls are only for a few bytes, though this is of course highly workload dependent and you might only care about the large ones.

Quoted 12 lines

Yes, i too feel get_user_pages_fast() path is the most time consuming, will instrument & check.

Quoted 4 lines

What was the exact 'dd' command you used, in particular the block size? Note that by default, 'dd' will request 512 bytes at a time, so you usually only access a single page. It would be interesting to see the overhead with other typical or extreme block sizes, e.g. '1', '64', '4K', '64K' or '1M'.

It was the default(512), more test results follows (in MB/s),

512 1K 4K 16K 32K 64K 1M

Quoted 7 lines

w/o series 30 46 89 95 90 85 65

w/ series 22 36 72 79 78 75 61

perf drop 26% 21% 19% 16% 13% 12% 6%

Hmm, results ain't that bad :)

There is also still hope of optimizing small aligned copies like

set_ttbr0(user_ttbr);
ldm();
set_ttbr0(kernel_ttbr);
stm();

which could do e.g. 32 bytes at a time, but with more overhead if you have to loop around it.

Arnd

#5 lore raw

Re: [RFC 1/3] lib: copy_{from,to}_user using gup & kmap_atomic()

Hi,

Quoted 4 lines · Arnd Bergmann

I think a lot
of usercopy calls are only for a few bytes, though this is of course highly workload dependent and you might only care about the large ones.

Observation is that max. pages reaching copy_{from,to}_user() is 2, observed maximum of n (number of bytes) being 1 page size. i think C library cuts any size read, write to page size (if it exceeds) & invokes the system call. Max. pages reaching 2, happens when 'n' crosses page boundary, this has been observed w/ small size request as well w/ ones of exact page size (but not page aligned).

Even w/ dd of various size >4K, never is the number of pages required to be mapped going greater than 2 (even w/ 'dd' 'bs=1M')

i have a worry (don't know whether it is an unnecessary one): even if we improve performance w/ large copy sizes, it might end up in a sluggishness w.r.t user experience due to most (hence a high amount) of user copy calls being few bytes & there the penalty being higher. And benchmark would not be able to detect anything abnormal since usercopy are being tested on large sizes.

Quickly comparing boot-time on Beagle Bone White, boot time increases by only 4%, perhaps this worry is irrelevant, but just thought will put it across.

Quoted 6 lines

There is also still hope of optimizing small aligned copies like

set_ttbr0(user_ttbr);
ldm();
set_ttbr0(kernel_ttbr);
stm();

Hmm, more needs to be done to be in a position to test it.

Regards
afzal

#6 lore raw

Re: [RFC 1/3] lib: copy_{from,to}_user using gup & kmap_atomic()

Quoted 17 lines · afzal mohammed

Observation is that max. pages reaching copy_{from,to}_user() is 2, observed maximum of n (number of bytes) being 1 page size. i think C library cuts any size read, write to page size (if it exceeds) & invokes the system call. Max. pages reaching 2, happens when 'n' crosses page boundary, this has been observed w/ small size request as well w/ ones of exact page size (but not page aligned).

Even w/ dd of various size >4K, never is the number of pages required to be mapped going greater than 2 (even w/ 'dd' 'bs=1M')

i have a worry (don't know whether it is an unnecessary one): even if we improve performance w/ large copy sizes, it might end up in a sluggishness w.r.t user experience due to most (hence a high amount) of user copy calls being few bytes & there the penalty being higher. And benchmark would not be able to detect anything abnormal since usercopy are being tested on large sizes.

Quoted 3 lines

Quickly comparing boot-time on Beagle Bone White, boot time increases by only 4%, perhaps this worry is irrelevant, but just thought will put it across.

Do stat(2) of the same tmpfs file in a loop (on tmpfs, to eliminate the filesystem playing silly buggers). And I wouldn't expect anything good there...

#7 lore raw

Re: [RFC 1/3] lib: copy_{from,to}_user using gup & kmap_atomic()

Quoted 19 lines · Al Viro
Quoted 17 lines · afzal mohammed

Observation is that max. pages reaching copy_{from,to}_user() is 2, observed maximum of n (number of bytes) being 1 page size. i think C library cuts any size read, write to page size (if it exceeds) & invokes the system call. Max. pages reaching 2, happens when 'n' crosses page boundary, this has been observed w/ small size request as well w/ ones of exact page size (but not page aligned).

Even w/ dd of various size >4K, never is the number of pages required to be mapped going greater than 2 (even w/ 'dd' 'bs=1M')

i have a worry (don't know whether it is an unnecessary one): even if we improve performance w/ large copy sizes, it might end up in a sluggishness w.r.t user experience due to most (hence a high amount) of user copy calls being few bytes & there the penalty being higher. And benchmark would not be able to detect anything abnormal since usercopy are being tested on large sizes.

Quoted 7 lines
Quoted 3 lines

Quickly comparing boot-time on Beagle Bone White, boot time increases by only 4%, perhaps this worry is irrelevant, but just thought will put it across.

Do stat(2) of the same tmpfs file in a loop (on tmpfs, to eliminate the filesystem playing silly buggers). And I wouldn't expect anything good there...

Incidentally, what about get_user()/put_user()? _That_ is where it's going to really hurt...

#8 lore raw

Re: [RFC 1/3] lib: copy_{from,to}_user using gup & kmap_atomic()

Hi,

Quoted 2 lines · Al Viro

Incidentally, what about get_user()/put_user()? _That_ is where it's going to really hurt...

All other uaccess routines are also planned to be added, posting only copy_{from,to}_user() was to get early feedback (mentioned in the cover letter)

Regards
afzal

#9 lore raw

Re: [RFC 1/3] lib: copy_{from,to}_user using gup & kmap_atomic()

Quoted 10 lines · afzal mohammed

Hi,

Quoted 2 lines · Al Viro

Incidentally, what about get_user()/put_user()? _That_ is where it's going to really hurt...

All other uaccess routines are also planned to be added, posting only copy_{from,to}_user() was to get early feedback (mentioned in the cover letter)

Sure, but what I mean is that I'd expect the performance loss to be dominated by that, not by copy_from_user/copy_to_user on large amounts of data. Especially on the loads like kernel builds - a lot of stat() and getdents() calls there.

#10 lore raw

Re: [RFC 1/3] lib: copy_{from,to}_user using gup & kmap_atomic()

Quoted 8 lines · Al Viro
Quoted 7 lines · afzal mohammed

Hi,

Quoted 2 lines · Al Viro

Incidentally, what about get_user()/put_user()? _That_ is where it's going to really hurt...

Quoted 8 lines
Quoted 3 lines

All other uaccess routines are also planned to be added, posting only copy_{from,to}_user() was to get early feedback (mentioned in the cover letter)

Sure, but what I mean is that I'd expect the performance loss to be dominated by that, not by copy_from_user/copy_to_user on large amounts of data. Especially on the loads like kernel builds - a lot of stat() and getdents() calls there.

To clarify: stat() means small copy_to_user(), getdents() - a mix of put_user() and small copy_to_user(). I would be very surprised if it does not hurt a lot.

#11 lore raw

Re: [RFC 1/3] lib: copy_{from,to}_user using gup & kmap_atomic()

Quoted 13 lines · Al Viro
Quoted 12 lines · Al Viro
Quoted 10 lines · afzal mohammed

Hi,

Quoted 2 lines · Al Viro

Incidentally, what about get_user()/put_user()? _That_ is where it's going to really hurt...

All other uaccess routines are also planned to be added, posting only copy_{from,to}_user() was to get early feedback (mentioned in the cover letter)

Quoted 8 lines
Quoted 4 lines

Sure, but what I mean is that I'd expect the performance loss to be dominated by that, not by copy_from_user/copy_to_user on large amounts of data. Especially on the loads like kernel builds - a lot of stat() and getdents() calls there.

To clarify: stat() means small copy_to_user(), getdents() - a mix of put_user() and small copy_to_user(). I would be very surprised if it does not hurt a lot.

PS: there's another fun issue here:

fill a file with zeroes
mmap that file in two areas, MAP_SHARED
thread 1:
munmap() the first area
fill the second one with 'X'
thread 2:
write() from the first area into pipe

One could expect that nothing by zeroes gets written into pipe - it might be a short write() (or -EFAULT), but finding any 'X' there would be a bug.

Your patches allow for a possibility of write() doing get_user_pages_fast(), getting the first page just as munmap() is about to remove it from page tables and bugger off. Then thread 1 proceeds with the store (via the second area). And then thread 2 does memcpy() from that thing via a kmap_atomic()-created alias, observing the effect of the store.

That might or might not be a POSIX violation, but it does look like a QoI issue...

#12 lore raw

Re: [RFC 1/3] lib: copy_{from,to}_user using gup & kmap_atomic()

Quoted 45 lines · Al Viro
Quoted 13 lines · Al Viro
Quoted 12 lines · Al Viro
Quoted 10 lines · afzal mohammed

Hi,

Quoted 2 lines · Al Viro

Incidentally, what about get_user()/put_user()? _That_ is where it's going to really hurt...

All other uaccess routines are also planned to be added, posting only copy_{from,to}_user() was to get early feedback (mentioned in the cover letter)

Quoted 8 lines
Quoted 4 lines

Sure, but what I mean is that I'd expect the performance loss to be dominated by that, not by copy_from_user/copy_to_user on large amounts of data. Especially on the loads like kernel builds - a lot of stat() and getdents() calls there.

To clarify: stat() means small copy_to_user(), getdents() - a mix of put_user() and small copy_to_user(). I would be very surprised if it does not hurt a lot.

PS: there's another fun issue here:

fill a file with zeroes
mmap that file in two areas, MAP_SHARED
thread 1:
munmap() the first area
fill the second one with 'X'
thread 2:
write() from the first area into pipe

One could expect that nothing by zeroes gets written into pipe - it might be a short write() (or -EFAULT), but finding any 'X' there would be a bug.

Your patches allow for a possibility of write() doing get_user_pages_fast(), getting the first page just as munmap() is about to remove it from page tables and bugger off. Then thread 1 proceeds with the store (via the second area). And then thread 2 does memcpy() from that thing via a kmap_atomic()-created alias, observing the effect of the store.

Quoted 2 lines

That might or might not be a POSIX violation, but it does look like a QoI issue...

I assume this problem exists in arch/um/kernel/skas/uaccess.c and in Ingo's old x86 VMSPLIT_4G_4G patch as well, right?

I guess holding mmap_read_lock() would prevent it but make it even more expensive.

Arnd

#13 lore raw

RE: [RFC 1/3] lib: copy_{from,to}_user using gup & kmap_atomic()

In reply to Al Viro

From: Al Viro

Quoted 9 lines

Sent: 13 June 2020 16:31

Quoted 7 lines · afzal mohammed

Hi,

Quoted 2 lines · Al Viro

Incidentally, what about get_user()/put_user()? _That_ is where it's going to really hurt...

Quoted 8 lines
Quoted 3 lines

All other uaccess routines are also planned to be added, posting only copy_{from,to}_user() was to get early feedback (mentioned in the cover letter)

Sure, but what I mean is that I'd expect the performance loss to be dominated by that, not by copy_from_user/copy_to_user on large amounts of data. Especially on the loads like kernel builds - a lot of stat() and getdents() calls there.

Or any network traffic where the number of usercopies involved in, for example, sendmsg() gives a measurable performance decrease when HARDENED_USERCOPY is enabled.

Do you have issues with cache aliasing?
(Is aliasing the right term?)
Where potentially the temporary kernel map doesn't use the same cache lines as the user processes map.

I'm not sure what problem you are trying to solve, but for 64bit systems it may be possible to map all of physical memory into the kernel address map, them you (loosely) only have to find the KVA that matches the user-VA to do the copy.

IIRC our SYSV kernels used to do that - until we had 384MB of physical memory and ran out of kernel address space.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#14 lore raw

Re: [RFC 1/3] lib: copy_{from,to}_user using gup & kmap_atomic()

In reply to afzal mohammed
Quoted 9 lines · afzal mohammed

Hi,

Quoted 4 lines · Arnd Bergmann

I think a lot
of usercopy calls are only for a few bytes, though this is of course highly workload dependent and you might only care about the large ones.

Quoted 6 lines

Observation is that max. pages reaching copy_{from,to}_user() is 2, observed maximum of n (number of bytes) being 1 page size. i think C library cuts any size read, write to page size (if it exceeds) & invokes the system call. Max. pages reaching 2, happens when 'n' crosses page boundary, this has been observed w/ small size request as well w/ ones of exact page size (but not page aligned).

You can't make that assumption about read(2). stdio in the C library may read a page size of data at a time, but programs are allowed to call read(2) directly, and the C library will pass such a call straight through to the kernel. So, if userspace requests a 16k read via read(2), then read(2) will be invoked covering 16k.

As an extreme case, for example:

$ strace -e read dd if=/dev/zero of=/dev/null bs=1048576 count=1
read(0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576) = 1048576
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
#15 lore raw

Re: [RFC 1/3] lib: copy_{from,to}_user using gup & kmap_atomic()

Hi,

Quoted 12 lines · Russell King - ARM Linux admin
Quoted 3 lines · afzal mohammed

i think C
library cuts any size read, write to page size (if it exceeds) & invokes the system call.

You can't make that assumption about read(2). stdio in the C library may read a page size of data at a time, but programs are allowed to call read(2) directly, and the C library will pass such a call straight through to the kernel. So, if userspace requests a 16k read via read(2), then read(2) will be invoked covering 16k.

Quoted 4 lines

As an extreme case, for example:

$ strace -e read dd if=/dev/zero of=/dev/null bs=1048576 count=1
read(0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576) = 1048576

Okay. Yes, observed that dd is passing whatever is the 'bs' to Kernel and from the 'dd' sources (of busybox), it is invoking read system call directly passing 'bs', so it is the tmpfs read that is splitting it to page size as mentioned by Arnd.

Regards
afzal

#16 lore raw

Re: [RFC 1/3] lib: copy_{from,to}_user using gup & kmap_atomic()

In reply to afzal mohammed
Quoted 7 lines · afzal mohammed
Quoted 4 lines · Arnd Bergmann

I think a lot
of usercopy calls are only for a few bytes, though this is of course highly workload dependent and you might only care about the large ones.

Quoted 6 lines

Observation is that max. pages reaching copy_{from,to}_user() is 2, observed maximum of n (number of bytes) being 1 page size. i think C library cuts any size read, write to page size (if it exceeds) & invokes the system call. Max. pages reaching 2, happens when 'n' crosses page boundary, this has been observed w/ small size request as well w/ ones of exact page size (but not page aligned).

Right, this is apparently because tmpfs uses shmem_file_read_iter() to copy the file pages one at a time. generic_file_buffered_read() seems similar, to copying between an aligned kernel page and address in user space that is not page aligned would be an important case to optimize for.

Quoted 3 lines

Quickly comparing boot-time on Beagle Bone White, boot time increases by only 4%, perhaps this worry is irrelevant, but just thought will put it across.

4% boot time increase sounds like a lot, especially if that is only for copy_from_user/copy_to_user. In the end it really depends on how well get_user()/put_user() and small copies can be optimized in the end.

Quoted 8 lines
Quoted 6 lines

There is also still hope of optimizing small aligned copies like

set_ttbr0(user_ttbr);
ldm();
set_ttbr0(kernel_ttbr);
stm();

Hmm, more needs to be done to be in a position to test it.

This is going to be highly microarchitecture specific, so anything you test on the Beaglebone's Cortex-A8 might not apply to A7/A15/A17 systems, but if you want to test what the overhead is, you could try changing /dev/zero (or a different chardev like it) to use a series of put_user(0, u32uptr++) in place of whatever it has, and then replace the 'str' instruction with dummy writes to ttbr0 using the value it already has, like:

      mcr     p15, 0, %0, c2, c0, 0  /* set_ttbr0() */
      isb  /* prevent speculative access to kernel table */
      str    %1, [%2],0 /* write 32 bit to user space */
      mcr     p15, 0, %0, c2, c0, 0  /* set_ttbr0() */
      isb  /* prevent speculative access to user table */

This is obviously going to be very slow compared to the simple store there is today but maybe cheaper than the CONFIG_ARM64_SW_TTBR0_PAN uaccess_en/disable()
on arm64 on a single get_user()/put_user().

It would be interesting to compare it to the overhead of a get_user_page_fast() based implementation. From the numbers you measured, it seems the beaglebone currently needs an extra ~6µs or 3µs per copy_to/from_user() call with your patch, depending on what your benchmark was (MB/s for just reading or writing vs MB/s for copying from one file to another through a user space buffer).

Arnd

#17 lore raw

Re: [RFC 1/3] lib: copy_{from,to}_user using gup & kmap_atomic()

Quoted 8 lines · Arnd Bergmann
Quoted 6 lines · afzal mohammed

Observation is that max. pages reaching copy_{from,to}_user() is 2, observed maximum of n (number of bytes) being 1 page size. i think C library cuts any size read, write to page size (if it exceeds) & invokes the system call. Max. pages reaching 2, happens when 'n' crosses page boundary, this has been observed w/ small size request as well w/ ones of exact page size (but not page aligned).

Quoted 5 lines

Right, this is apparently because tmpfs uses shmem_file_read_iter() to copy the file pages one at a time. generic_file_buffered_read() seems similar, to copying between an aligned kernel page and address in user space that is not page aligned would be an important case to optimize for.

This is kind of the nature of the page cache. The kernel doesn't necessarily have contiguous memory in the page cache, so it's going to be split on page boundaries. This is going to change with my THP series (I haven't actually changed generic_file_buffered_read(), but it'll come later).

Quoted 7 lines
Quoted 3 lines

Quickly comparing boot-time on Beagle Bone White, boot time increases by only 4%, perhaps this worry is irrelevant, but just thought will put it across.

4% boot time increase sounds like a lot, especially if that is only for copy_from_user/copy_to_user. In the end it really depends on how well get_user()/put_user() and small copies can be optimized in the end.

The write path should also be paid attention to. Look at generic_perform_write() which first calls iov_iter_fault_in_readable() (for the entire length of the write) and then actually does the copy later with iov_iter_copy_from_user_atomic(). So you're going to want to optimise the case where you access the same pages multiple times.

#18 lore raw

Re: [RFC 1/3] lib: copy_{from,to}_user using gup & kmap_atomic()

In reply to Arnd Bergmann

Hi,

Quoted 3 lines · Arnd Bergmann

4% boot time increase sounds like a lot, especially if that is only for copy_from_user/copy_to_user. In the end it really depends on how well get_user()/put_user() and small copies can be optimized in the end.

i mentioned the worst case(happened only once), normally it was in the range 2-3%

Quoted 5 lines

From the numbers you
measured, it seems the beaglebone currently needs an extra ~6µs or 3µs per copy_to/from_user() call with your patch, depending on what your benchmark was (MB/s for just reading or writing vs MB/s for copying from one file to another through a user space buffer).

It is MB/s for copying one file to another via user space buffer, i.e. the value coreutils 'dd' shows w/ status=progress (here it was busybox 'dd', so instead it was enabling a compile time option)

Quoted 6 lines

but if you want to test what the overhead is, you could try changing /dev/zero (or a different chardev like it) to use a series of put_user(0, u32uptr++) in place of whatever it has, and then replace the 'str' instruction with dummy writes to ttbr0 using the value it already has, like:

Quoted 8 lines
      mcr     p15, 0, %0, c2, c0, 0  /* set_ttbr0() */
      isb  /* prevent speculative access to kernel table */
      str    %1, [%2],0 /* write 32 bit to user space */
      mcr     p15, 0, %0, c2, c0, 0  /* set_ttbr0() */
      isb  /* prevent speculative access to user table */

It would be interesting to compare it to the overhead of a get_user_page_fast() based implementation.

i have to relocate & be on quarantine couple of weeks, so i will temporarily stop here, otherwise might end up in roadside.

Reading feedbacks from everyone, some of it i could grasp only bits & pieces, familiarizing more w/ mm & vfs would help me add value better to the goal/discussion. Linus Walleij, if you wish to explore things, feel free, right now don't know how my connectivity would be for next 3 weeks.

Regards
afzal

#19 lore raw

Re: [RFC 1/3] lib: copy_{from,to}_user using gup & kmap_atomic()

Hi,

Quoted 3 lines · afzal mohammed

It is MB/s for copying one file to another via user space buffer, i.e. the value coreutils 'dd' shows w/ status=progress (here it was busybox 'dd', so instead it was enabling a compile time option)

Just for correctness, status=progress is not required, it's there in the default 3rd line of coreutils 'dd' o/p

Regards
afzal

#20 lore raw

Re: [RFC 1/3] lib: copy_{from,to}_user using gup & kmap_atomic()

In reply to afzal mohammed
Quoted 20 lines · afzal mohammed

copy_{from,to}_user() uaccess helpers are implemented by user page pinning, followed by temporary kernel mapping & then memcpy(). This helps to achieve user page copy when current virtual address mapping of the CPU excludes user pages.

Performance wise, results are not encouraging, 'dd' on tmpfs results,

ARM Cortex-A8, BeagleBone White (256MiB RAM):
w/o series - ~29.5 MB/s
w/ series - ~20.5 MB/s
w/ series & highmem disabled - ~21.2 MB/s

On Cortex-A15(2GiB RAM) in QEMU:
w/o series - ~4 MB/s
w/ series - ~2.6 MB/s

Roughly a one-third drop in performance. Disabling highmem improves performance only slightly.

Quoted 8 lines

'hackbench' also showed a similar pattern.

uaccess routines using page pinning & temporary kernel mapping is not something new, it has been done long long ago by Ingo [1] as part of 4G/4G user/kernel mapping implementation on x86, though not merged in mainline.

[1] https://lore.kernel.org/lkml/[email protected]/

Some comments (more related to generic things).

...

Quoted 1 line

+// Started from arch/um/kernel/skas/uaccess.c

Does it mean you will deduplicate it there?

...

lib/uaccess_gup_kmap_memcpy.c line 7

Quoted 4 lines
lib/uaccess_gup_kmap_memcpy.c new file +4 -0
+#include <linux/err.h>
+#include <linux/slab.h>
+#include <linux/highmem.h>
+#include <linux/mm.h>

Perhaps ordered?

lib/uaccess_gup_kmap_memcpy.c line 14

Quoted 3 lines
lib/uaccess_gup_kmap_memcpy.c new file +3 -0
+static int do_op_one_page(unsigned long addr, int len,
+                int (*op)(unsigned long addr, int len, void *arg), void *arg,
+                struct page *page)

Maybe typedef for the func() ?

lib/uaccess_gup_kmap_memcpy.c line 18

Quoted 4 lines
lib/uaccess_gup_kmap_memcpy.c new file +4 -0
+{
+       int n;
+
+       addr = (unsigned long) kmap_atomic(page) + (addr & ~PAGE_MASK);

I don't remember about this one...

Quoted 4 lines
lib/uaccess_gup_kmap_memcpy.c new file +4 -0
+       n = (*op)(addr, len, arg);
+       kunmap_atomic((void *)addr);
+
+       return n;

lib/uaccess_gup_kmap_memcpy.c line 31

Quoted 10 lines
lib/uaccess_gup_kmap_memcpy.c new file +9 -0
+}
+
+static long buffer_op(unsigned long addr, int len,
+                     int (*op)(unsigned long, int, void *), void *arg,
+                     struct page **pages)
+{
+       long size, remain, n;
+
 
+       size = min(PAGE_ALIGN(addr) - addr, (unsigned long) len);

...but here seems to me you can use helpers (offset_in_page() or how it's called).

Also consider to use macros like PFN_DOWN(), PFN_UP(), etc in your code.

lib/uaccess_gup_kmap_memcpy.c line 38

Quoted 8 lines
lib/uaccess_gup_kmap_memcpy.c new file +7 -0
+       remain = len;
+       if (size == 0)
+               goto page_boundary;
+
+       n = do_op_one_page(addr, size, op, arg, *pages);
+       if (n != 0) {
 
+               remain = (n < 0 ? remain : 0);

Why duplicate three times (!) this line, if you can move it to under 'out'?

Quoted 11 lines
lib/uaccess_gup_kmap_memcpy.c new file +11 -0
+               goto out;
+       }
+
+       pages++;
+       addr += size;
+       remain -= size;
+
+page_boundary:
+       if (remain == 0)
+               goto out;
+       while (addr < ((addr + remain) & PAGE_MASK)) {

lib/uaccess_gup_kmap_memcpy.c line 59

Quoted 10 lines
lib/uaccess_gup_kmap_memcpy.c new file +10 -0
+               n = do_op_one_page(addr, PAGE_SIZE, op, arg, *pages);
+               if (n != 0) {
+                       remain = (n < 0 ? remain : 0);
+                       goto out;
+               }
+
+               pages++;
+               addr += PAGE_SIZE;
+               remain -= PAGE_SIZE;
+       }

Sounds like this can be refactored to iterate over pages rather than addresses.

Quoted 14 lines
lib/uaccess_gup_kmap_memcpy.c new file +13 -0
+       if (remain == 0)
+               goto out;
+
+       n = do_op_one_page(addr, remain, op, arg, *pages);
+       if (n != 0) {
+               remain = (n < 0 ? remain : 0);
+               goto out;
+       }
+
+       return 0;
 
+out:
+       return remain;
+}

...

lib/uaccess_gup_kmap_memcpy.c line 78

Quoted 5 lines
lib/uaccess_gup_kmap_memcpy.c new file +5 -0
+static int copy_chunk_from_user(unsigned long from, int len, void *arg)
+{
+       unsigned long *to_ptr = arg, to = *to_ptr;
+
+       memcpy((void *) to, (void *) from, len);

What is the point in the casting to void *?

lib/uaccess_gup_kmap_memcpy.c line 88

Quoted 10 lines
lib/uaccess_gup_kmap_memcpy.c new file +10 -0
+       *to_ptr += len;
+       return 0;
+}
+
+static int copy_chunk_to_user(unsigned long to, int len, void *arg)
+{
+       unsigned long *from_ptr = arg, from = *from_ptr;
+
+       memcpy((void *) to, (void *) from, len);
+       *from_ptr += len;

Ditto.

Quoted 6 lines
lib/uaccess_gup_kmap_memcpy.c new file +6 -0
+       return 0;
+}
+
+unsigned long gup_kmap_copy_from_user(void *to, const void __user *from, unsigned long n)
+{
+       struct page **pages;

lib/uaccess_gup_kmap_memcpy.c line 103

Quoted 10 lines
lib/uaccess_gup_kmap_memcpy.c new file +9 -0
+       int num_pages, ret, i;
+
+       if (uaccess_kernel()) {
+               memcpy(to, (__force void *)from, n);
+               return 0;
+       }
+
 
+       num_pages = DIV_ROUND_UP((unsigned long)from + n, PAGE_SIZE) -
+                                (unsigned long)from / PAGE_SIZE;

PFN_UP() ?

Quoted 13 lines
lib/uaccess_gup_kmap_memcpy.c new file +13 -0
+       pages = kmalloc_array(num_pages, sizeof(*pages), GFP_KERNEL | __GFP_ZERO);
+       if (!pages)
+               goto end;
+
+       ret = get_user_pages_fast((unsigned long)from, num_pages, 0, pages);
+       if (ret < 0)
+               goto free_pages;
+
+       if (ret != num_pages) {
+               num_pages = ret;
+               goto put_pages;
+       }
+

lib/uaccess_gup_kmap_memcpy.c line 126

Quoted 10 lines
lib/uaccess_gup_kmap_memcpy.c new file +10 -0
+       n = buffer_op((unsigned long) from, n, copy_chunk_from_user, &to, pages);
+
+put_pages:
+       for (i = 0; i < num_pages; i++)
+               put_page(pages[i]);
+free_pages:
+       kfree(pages);
+end:
+       return n;
+}

...

I think you can clean up the code a bit after you will get the main functionality working.

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#21 lore raw

Re: [RFC 1/3] lib: copy_{from,to}_user using gup & kmap_atomic()

Hi,

Quoted 5 lines · Andy Shevchenko
Quoted 1 line · afzal mohammed

+// Started from arch/um/kernel/skas/uaccess.c

Does it mean you will deduplicate it there?

What i meant was, that file was taken as a template & nothing more, at same time i wanted to give credit to that file, i will explicitly mention it next time.

It is not meant to deduplicate it. Functionality here is completely different.

In the case here, there would be different virtual address mapping that CPU will be see once in Kernel as compared to user mode.

Here a facility is provided to access the user page, when the current virtual address mapping of the CPU excludes it. This is for providing full 4G virtual address to both user & kernel on 32bit ARM to avoid using highmem or reduce the impact of highmem, i.e. so that Kernel can address till 4GiB (almost) as lowmem.

Here assumption is that user mapping is not a subset of virtual address mapped by CPU, but a separate one. Upon Kernel entry ttbr0 is changed to Kernel lowmem, while upon Kernel exit is changed back to user pages (ttbrx in ARM, iiuc, equivalent to cr3 in x86)

Now realize that i am unable to put coherently the problem being attempted to solve here to a person not familiar w/ the issue w/o taking considerable time. If above explanation is not enough, i will try to explain later in a better way.

Quoted 6 lines

lib/uaccess_gup_kmap_memcpy.c line 7

Quoted 4 lines
lib/uaccess_gup_kmap_memcpy.c new file +4 -0
+#include <linux/err.h>
+#include <linux/slab.h>
+#include <linux/highmem.h>
+#include <linux/mm.h>

Perhaps ordered?

will take care

Quoted 5 lines

lib/uaccess_gup_kmap_memcpy.c line 14

Quoted 3 lines
lib/uaccess_gup_kmap_memcpy.c new file +3 -0
+static int do_op_one_page(unsigned long addr, int len,
+                int (*op)(unsigned long addr, int len, void *arg), void *arg,
+                struct page *page)

Maybe typedef for the func() ?

will take care

Quoted 6 lines

lib/uaccess_gup_kmap_memcpy.c line 18

Quoted 4 lines
lib/uaccess_gup_kmap_memcpy.c new file +4 -0
+{
+       int n;
+
+       addr = (unsigned long) kmap_atomic(page) + (addr & ~PAGE_MASK);

I don't remember about this one...

i am not following you here, for my case !CONFIG_64BIT case in that file was required, hence only it was picked (or rather not deleted)

Quoted 4 lines
Quoted 1 line

+ size = min(PAGE_ALIGN(addr) - addr, (unsigned long) len);

...but here seems to me you can use helpers (offset_in_page() or how it's called).

i was not aware of it, will use it as required.

Quoted 2 lines

Also consider to use macros like PFN_DOWN(), PFN_UP(), etc in your code.

Okay

Quoted 11 lines

lib/uaccess_gup_kmap_memcpy.c line 38

Quoted 9 lines
lib/uaccess_gup_kmap_memcpy.c new file +7 -0
+       remain = len;
+       if (size == 0)
+               goto page_boundary;
+
+       n = do_op_one_page(addr, size, op, arg, *pages);
+       if (n != 0) {
 
+               remain = (n < 0 ? remain : 0);

Why duplicate three times (!) this line, if you can move it to under 'out'?

yes better to move there

Quoted 16 lines
Quoted 16 lines
lib/uaccess_gup_kmap_memcpy.c new file +15 -0
+               goto out;
+       }
+
+       pages++;
+       addr += size;
+       remain -= size;
+
+page_boundary:
+       if (remain == 0)
+               goto out;
+       while (addr < ((addr + remain) & PAGE_MASK)) {
+               n = do_op_one_page(addr, PAGE_SIZE, op, arg, *pages);
+               if (n != 0) {
+                       remain = (n < 0 ? remain : 0);
+                       goto out;
Quoted 8 lines

lib/uaccess_gup_kmap_memcpy.c line 59

Quoted 6 lines
lib/uaccess_gup_kmap_memcpy.c new file +6 -0
+               }
+
+               pages++;
+               addr += PAGE_SIZE;
+               remain -= PAGE_SIZE;
+       }

Sounds like this can be refactored to iterate over pages rather than addresses.

Okay, i will check

Quoted 7 lines

lib/uaccess_gup_kmap_memcpy.c line 78

Quoted 5 lines
lib/uaccess_gup_kmap_memcpy.c new file +5 -0
+static int copy_chunk_from_user(unsigned long from, int len, void *arg)
+{
+       unsigned long *to_ptr = arg, to = *to_ptr;
+
+       memcpy((void *) to, (void *) from, len);

What is the point in the casting to void *?

The reason it was there was because of copy-paste :), passing unsigned long as 'void *' or 'const void *' requires casting right ?, or you meant something else ?

now i checked removing the cast, compiler is abusing me :), says 'makes pointer from integer without a cast'

Quoted 4 lines

lib/uaccess_gup_kmap_memcpy.c line 103

Quoted 2 lines
lib/uaccess_gup_kmap_memcpy.c new file +2 -0
+       num_pages = DIV_ROUND_UP((unsigned long)from + n, PAGE_SIZE) -
+                                (unsigned long)from / PAGE_SIZE;

PFN_UP() ?

Okay

Quoted 2 lines

I think you can clean up the code a bit after you will get the main functionality working.

Yes, surely, intention was to post proof-of-concept ASAP, perhaps contents will change drastically in next version so that any resemblence of arch/um/kernel/skas/uaccess.c might not be there.

Regards
afzal

#22 lore raw

RFC 2/3 ARM: uaccess: let UACCESS_GUP_KMAP_MEMCPY enabling

In reply to afzal mohammed

Turn off existing raw_copy_{from,to}_user() using arm_copy_{from,to}_user() when CONFIG_UACCESS_GUP_KMAP_MEMCPY is enabled.

---
 arch/arm/include/asm/uaccess.h | 20 ++++++++++++++++++++
 arch/arm/kernel/armksyms.c     |  2 ++
 arch/arm/lib/Makefile          |  7 +++++--
 3 files changed, 27 insertions(+), 2 deletions(-)
arch/arm/include/asm/uaccess.h +20 -0
index 98c6b91be4a8a..4a16ae52d4978 100644
@@ -512,6 +512,15 @@ do {									\
 extern unsigned long __must_check
 arm_copy_from_user(void *to, const void __user *from, unsigned long n);
 
+#ifdef CONFIG_UACCESS_GUP_KMAP_MEMCPY
+extern unsigned long __must_check
+gup_kmap_copy_from_user(void *to, const void __user *from, unsigned long n);
+static inline __must_check unsigned long
+raw_copy_from_user(void *to, const void __user *from, unsigned long n)
+{
+	return gup_kmap_copy_from_user(to, from, n);
+}
+#else
 static inline unsigned long __must_check
 raw_copy_from_user(void *to, const void __user *from, unsigned long n)
 {
@@ -522,12 +531,22 @@ raw_copy_from_user(void *to, const void __user *from, unsigned long n)
 	uaccess_restore(__ua_flags);
 	return n;
 }
+#endif
 
 extern unsigned long __must_check
 arm_copy_to_user(void __user *to, const void *from, unsigned long n);
 extern unsigned long __must_check
 __copy_to_user_std(void __user *to, const void *from, unsigned long n);
 
+#ifdef CONFIG_UACCESS_GUP_KMAP_MEMCPY
+extern unsigned long __must_check
+gup_kmap_copy_to_user(void __user *to, const void *from, unsigned long n);
+static inline __must_check unsigned long
+raw_copy_to_user(void __user *to, const void *from, unsigned long n)
+{
+	return gup_kmap_copy_to_user(to, from, n);
+}
+#else
 static inline unsigned long __must_check
 raw_copy_to_user(void __user *to, const void *from, unsigned long n)
 {
@@ -541,6 +560,7 @@ raw_copy_to_user(void __user *to, const void *from, unsigned long n)
 	return arm_copy_to_user(to, from, n);
 #endif
 }
+#endif
 
 extern unsigned long __must_check
 arm_clear_user(void __user *addr, unsigned long n);
arch/arm/kernel/armksyms.c +2 -0
index 98bdea51089d5..8c92fe30d1559 100644
@@ -96,8 +96,10 @@ EXPORT_SYMBOL(mmiocpy);
 #ifdef CONFIG_MMU
 EXPORT_SYMBOL(copy_page);
 
+#ifndef CONFIG_UACCESS_GUP_KMAP_MEMCPY
 EXPORT_SYMBOL(arm_copy_from_user);
 EXPORT_SYMBOL(arm_copy_to_user);
+#endif
 EXPORT_SYMBOL(arm_clear_user);
 
 EXPORT_SYMBOL(__get_user_1);
arch/arm/lib/Makefile +5 -2
index 6d2ba454f25b6..1aeff2cd7b4b3 100644
@@ -16,8 +16,11 @@ lib-y		:= changebit.o csumipv6.o csumpartial.o               \
 		   io-readsb.o io-writesb.o io-readsl.o io-writesl.o  \
 		   call_with_stack.o bswapsdi2.o
 
-mmu-y		:= clear_user.o copy_page.o getuser.o putuser.o       \
-		   copy_from_user.o copy_to_user.o
+mmu-y		:= clear_user.o copy_page.o getuser.o putuser.o
+
+ifndef CONFIG_UACCESS_GUP_KMAP_MEMCPY
+  mmu-y		+= copy_from_user.o copy_to_user.o
+endif
 
 ifdef CONFIG_CC_IS_CLANG
   lib-y	+= backtrace-clang.o
Signature · 1 line
2.26.2
#23 lore raw

RFC 3/3 ARM: provide CONFIG_VMSPLIT_4G_DEV for development

In reply to afzal mohammed

Select UACCESS_GUP_KMAP_MEMCPY initially.

---
 arch/arm/Kconfig | 9 +++++++++
 1 file changed, 9 insertions(+)
arch/arm/Kconfig +9 -0
index c77c93c485a08..ae2687679d7c8 100644
@@ -1326,6 +1326,15 @@ config PAGE_OFFSET
 	default 0xB0000000 if VMSPLIT_3G_OPT
 	default 0xC0000000
 
+config VMSPLIT_4G_DEV
+	bool "Experimental changes for 4G/4G user/kernel split"
+	depends on ARM_LPAE
+	select UACCESS_GUP_KMAP_MEMCPY
+	help
+	  Experimental changes during 4G/4G user/kernel split development.
+	  Existing vmsplit config option is used, once development is done,
+	  this would be put as a new choice & _DEV suffix removed.
+
 config NR_CPUS
 	int "Maximum number of CPUs (2-32)"
 	range 2 32
Signature · 1 line
2.26.2
#24 lore raw
In reply to afzal mohammed
Quoted 7 lines · afzal mohammed

Performance wise, results are not encouraging, 'dd' on tmpfs results,

ARM Cortex-A8, BeagleBone White (256MiB RAM):
w/o series - ~29.5 MB/s
w/ series - ~20.5 MB/s
w/ series & highmem disabled - ~21.2 MB/s

Quoted 6 lines

On Cortex-A15(2GiB RAM) in QEMU:
w/o series - ~4 MB/s
w/ series - ~2.6 MB/s

Roughly a one-third drop in performance. Disabling highmem improves performance only slightly.

Could you compare with CONFIG_UACCESS_WITH_MEMCPY as well?

Nicolas

#25 lore raw

Hi,

Quoted 5 lines · Nicolas Pitre
Quoted 1 line · afzal mohammed

Performance wise, results are not encouraging, 'dd' on tmpfs results,

Could you compare with CONFIG_UACCESS_WITH_MEMCPY as well?

512 1K 4K 16K 32K 64K 1M

normal 30 46 89 95 90 85 65

uaccess_w_memcpy 28.5 45 85 92 91 85 65

w/ series 22 36 72 79 78 75 61

There are variations in the range +/-2 in some readings when repeated, not put above, to keep comparison simple.

Regards
afzal

#26 lore raw

Hi,

Quoted 7 lines · afzal mohammed

512 1K 4K 16K 32K 64K 1M

normal 30 46 89 95 90 85 65

uaccess_w_memcpy 28.5 45 85 92 91 85 65

w/ series 22 36 72 79 78 75 61

For the sake of completeness all in MB/s, w/ various 'dd' 'bs' sizes.

Regards
afzal