cache pollution aware __copy_from_user_ll()

RFC PATCH In review Hiro Yoshioka 7 participants 25 messages Aug 14, 2005 – Aug 18, 2005 1 message not in the archive

Subsystems (3)

Memory management Kbuild Page cache

lore

Share & tools

lore

Hiro Yoshioka

author

2005-08-14 09:16 UTC #0 lore raw

RFC PATCH 1/1 cache pollution aware __copy_from_user_ll()

Hi,

The following is a patch to reduce a cache pollution of __copy_from_user_ll().

When I run simple iozone benchmark to find a performance bottleneck of the linux kernel, I found that __copy_from_user_ll() spent CPU cycle most and it did many cache misses.

The following is profiled by oprofile.

Top 5 CPU cycle
CPU: P4 / Xeon, speed 2200.91 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (mandatory) count 100000

samples  %        app name                 symbol name
281538   15.2083  vmlinux                  __copy_from_user_ll
81069     4.3792  vmlinux                  _spin_lock
75523     4.0796  vmlinux                  journal_add_journal_head
63674     3.4396  vmlinux                  do_get_write_access
52634     2.8432  vmlinux                  journal_put_journal_head

(pattern9-0-cpu4-0-08141700/summary.out)

Top 5 Memory Access and Cache miss
CPU: P4 / Xeon, speed 2200.91 MHz (estimated)
Counted BSQ_CACHE_REFERENCE events (cache references seen by the bus unit) with a unit mask of 0x3f (multiple flags) count 3000 Counted BSQ_CACHE_REFERENCE events (cache references seen by the bus unit) with a unit mask of 0x200 (read 3rd level cache miss) count 3000

samples  %        samples  %        app name                 symbol name
120801    7.4379  37017    63.4603  vmlinux                  __copy_from_user_ll
84139     5.1806  885       1.5172  vmlinux                  _spin_lock
66027     4.0654  656       1.1246  vmlinux                 
journal_add_journal_head
60400     3.7189  250       0.4286  vmlinux                  __find_get_block
60032     3.6963  120       0.2057  vmlinux

journal_dirty_metadata

__copy_from_user_ll spent 63.4603% of L3 cache miss though it spent only 7.4379% of memory access.

In order to reduce the cache miss in the __copy_from_user_ll, I made the following patch and confirmed the reduction of the miss.

Top 5 CPU cycle
CPU: P4 / Xeon, speed 2200.93 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (mandatory) count 100000

samples  %        app name                 symbol name
120717    8.3454  vmlinux                  _mmx_memcpy_nt
65955     4.5596  vmlinux                  do_get_write_access
56088     3.8775  vmlinux                  journal_put_journal_head
52550     3.6329  vmlinux                  journal_dirty_metadata
38886     2.6883  vmlinux                  journal_add_journal_head

pattern9-0-cpu4-0-08141627/summary.out

_mmx_memcpy_nt is the new function which is called from __copy_from_user_ll and it spent only 42.88% of the original implementation. (120717/281538==42.88%)

Top 5 Memory Access
CPU: P4 / Xeon, speed 2200.93 MHz (estimated)
Counted BSQ_CACHE_REFERENCE events (cache references seen by the bus unit) with a unit mask of 0x3f (multiple flags) count 3000 Counted BSQ_CACHE_REFERENCE events (cache references seen by the bus unit) with a unit mask of 0x200 (read 3rd level cache miss) count 3000

samples  %        samples  %        app name                 symbol name
90918     6.3079  89        0.5673  vmlinux                  _mmx_memcpy_nt
83654     5.8039  177       1.1283  vmlinux                 
journal_dirty_metadata
57836     4.0127  348       2.2183  vmlinux                 
journal_put_journal_head
48236     3.3466  165       1.0518  vmlinux                  do_get_write_access
44546     3.0906  21        0.1339  vmlinux                  __getblk

The cache miss reduced from 37017 (63.4603%) to 89 (0.5673%). It is 0.24% of the original implementation.

The actual elapse time which five times run were 229.76 (sec) and 222.94 (sec). (229.76/222.94= 3.06% gain)

iozone -CMR -i 0 -+n -+u -s 8000MB -t 4

What do you think?

--- linux-2.6.12.4.orig/arch/i386/lib/usercopy.c 2005-08-05
16:04:37.000000000 +0900
+++ linux-2.6.12.4/arch/i386/lib/usercopy.c 2005-08-12 13:18:14.106916200 +0900
@@ -10,6 +10,7 @@
#include <linux/highmem.h>
#include <linux/blkdev.h>
#include <linux/module.h>
+#include <asm/i387.h>
#include <asm/uaccess.h>
#include <asm/mmx.h>

@@ -511,6 +512,108 @@
: "memory"); \
} while (0)

+/* Non Temporal Hint version of mmx_memcpy */
+/* It is cache aware */
+/* [email protected] */
+static unsigned long _mmx_memcpy_nt(void *to, const void *from, size_t len) +{
+ /* Note! gcc doesn't seem to align stack variables properly, so we
+ * need to make use of unaligned loads and stores.
+ */
+ void *p;
+ int i;
+
+ if (unlikely(in_interrupt())){
+ __copy_user_zeroing(to, from, len);
+ return len;
+ }
+
+ p = to;
+ i = len >> 6; /* len/64 */
+
+ kernel_fpu_begin();
+
+ __asm__ __volatile__ (
+ "1: prefetchnta (%0)\n" /* This set is 28 bytes */

+		"   prefetchnta 64(%0)\n"
+		"   prefetchnta 128(%0)\n"
+		"   prefetchnta 192(%0)\n"
+		"   prefetchnta 256(%0)\n"

+ "2: \n"
+ ".section .fixup, \"ax\"\n"
+ "3: movw $0x1AEB, 1b\n" /* jmp on 26 bytes */
+ " jmp 2b\n"
+ ".previous\n"
+ ".section __ex_table,\"a\"\n"
+ " .align 4\n"
+ " .long 1b, 3b\n"
+ ".previous"
+ : : "r" (from) );
+
+ for(; i>5; i--)
+ {
+ __asm__ __volatile__ (
+ "1: prefetchnta 320(%0)\n"
+ "2: movq (%0), %%mm0\n"

+		"  movq 8(%0), %%mm1\n"
+		"  movq 16(%0), %%mm2\n"
+		"  movq 24(%0), %%mm3\n"
+		"  movntq %%mm0, (%1)\n"
+		"  movntq %%mm1, 8(%1)\n"
+		"  movntq %%mm2, 16(%1)\n"
+		"  movntq %%mm3, 24(%1)\n"
+		"  movq 32(%0), %%mm0\n"
+		"  movq 40(%0), %%mm1\n"
+		"  movq 48(%0), %%mm2\n"
+		"  movq 56(%0), %%mm3\n"
+		"  movntq %%mm0, 32(%1)\n"
+		"  movntq %%mm1, 40(%1)\n"
+		"  movntq %%mm2, 48(%1)\n"
+		"  movntq %%mm3, 56(%1)\n"

+ ".section .fixup, \"ax\"\n"
+ "3: movw $0x05EB, 1b\n" /* jmp on 5 bytes */
+ " jmp 2b\n"
+ ".previous\n"
+ ".section __ex_table,\"a\"\n"
+ " .align 4\n"
+ " .long 1b, 3b\n"
+ ".previous"
+ : : "r" (from), "r" (to) : "memory");
+ from+=64;
+ to+=64;
+ }
+
+ for(; i>0; i--)
+ {
+ __asm__ __volatile__ (

+		"  movq (%0), %%mm0\n"
+		"  movq 8(%0), %%mm1\n"
+		"  movq 16(%0), %%mm2\n"
+		"  movq 24(%0), %%mm3\n"
+		"  movntq %%mm0, (%1)\n"
+		"  movntq %%mm1, 8(%1)\n"
+		"  movntq %%mm2, 16(%1)\n"
+		"  movntq %%mm3, 24(%1)\n"
+		"  movq 32(%0), %%mm0\n"
+		"  movq 40(%0), %%mm1\n"
+		"  movq 48(%0), %%mm2\n"
+		"  movq 56(%0), %%mm3\n"
+		"  movntq %%mm0, 32(%1)\n"
+		"  movntq %%mm1, 40(%1)\n"
+		"  movntq %%mm2, 48(%1)\n"
+		"  movntq %%mm3, 56(%1)\n"

+ : : "r" (from), "r" (to) : "memory");
+ from+=64;
+ to+=64;
+ }
+ /*
+ * Now do the tail of the block
+ */
+ kernel_fpu_end();
+ if(i=(len&63))
+ __copy_user_zeroing(to, from, i);
+ return i;
+}

unsigned long __copy_to_user_ll(void __user *to, const void *from,
unsigned long n)
{
@@ -575,10 +678,14 @@
__copy_from_user_ll(void *to, const void __user *from, unsigned long n)
{
BUG_ON((long)n < 0);
- if (movsl_is_ok(to, from, n))
+ if (n < 512) {
+ if (movsl_is_ok(to, from, n))
__copy_user_zeroing(to, from, n);
- else
+ else
n = __copy_user_zeroing_intel(to, from, n);
+ }
+ else
+ n = _mmx_memcpy_nt(to, from, n);
return n;
}

Thanks in advance,
Hiro
--
hyoshiok at miraclelinux.com

Arjan van de Ven

2005-08-14 09:41 UTC #1 lore raw

Quoted 8 lines · Hiro Yoshioka

Hi,
The following is a patch to reduce a cache pollution of __copy_from_user_ll().
When I run simple iozone benchmark to find a performance bottleneck of the linux kernel, I found that __copy_from_user_ll() spent CPU cycle most and it did many cache misses.

however... you copy something from userspace... aren't you going to USE it? The non-termoral versions actually throw the data out of the cache... so while this part might be nice, you pay BIG elsewhere....

Hiro Yoshioka

author

2005-08-14 10:22 UTC #2 lore raw

Thanks for your comments.

Quoted 6 lines · Arjan van de Ven

Quoted 5 lines · Hiro Yoshioka
Hi,
The following is a patch to reduce a cache pollution of __copy_from_user_ll().

Quoted 8 lines

Quoted 3 lines
When I run simple iozone benchmark to find a performance bottleneck of the linux kernel, I found that __copy_from_user_ll() spent CPU cycle most and it did many cache misses.
however... you copy something from userspace... aren't you going to USE it? The non-termoral versions actually throw the data out of the cache... so while this part might be nice, you pay BIG elsewhere....

The oprofile data does not give an evidence that we pay BIG elsewhere.

For examples, the original 2.6.12.4 Top 5 cache misses are the following,

37017 63.4603  vmlinux    __copy_from_user_ll
1049   1.7984  vmlinux    _spin_lock_irqsave
940    1.6115  vmlinux    blk_rq_map_sg
896    1.5361  vmlinux    generic_file_buffered_write
885    1.5172  vmlinux    _spin_lock

pattern9-0-cpu4-0-08141702

cache aware version Top 5 cache misses are

899 5.7305  vmlinux    blk_rq_map_sg
569 3.6270  vmlinux    journal_commit_transaction
531 3.3848  vmlinux    radix_tree_delete
514 3.2764  vmlinux    journal_add_journal_head
505 3.2190  vmlinux    release_pages

...
89 0.5673 vmlinux _mmx_memcpy_nt
pattern9-0-cpu4-0-08141625

What do you think?

Regards,
Hiro

Arjan van de Ven

2005-08-14 10:35 UTC #3 lore raw

Quoted 14 lines · Hiro Yoshioka

Thanks for your comments.
Quoted 11 lines · Arjan van de Ven
Quoted 8 lines · Hiro Yoshioka
Hi,
The following is a patch to reduce a cache pollution of __copy_from_user_ll().
When I run simple iozone benchmark to find a performance bottleneck of the linux kernel, I found that __copy_from_user_ll() spent CPU cycle most and it did many cache misses.

Quoted 5 lines

Quoted 3 lines
however... you copy something from userspace... aren't you going to USE it? The non-termoral versions actually throw the data out of the cache... so while this part might be nice, you pay BIG elsewhere....
The oprofile data does not give an evidence that we pay BIG elsewhere.

the problem is that the pay elsewhere is far more spread out, but not less. At least generally....

I can see the point of a copy_from_user_nocache() or something, for those cases where we *know* we are not going to use the copied data in the cpu (but say, only do DMA).
But that should be explicit, not implicit, since the general case will be that the kernel WILL use the data. And if that's the case your change is a loss.... (just harder to see because the cost is spread out)

Christoph Hellwig

2005-08-14 10:45 UTC #4 lore raw

Quoted 8 lines

the problem is that the pay elsewhere is far more spread out, but not less. At least generally....
I can see the point of a copy_from_user_nocache() or something, for those cases where we *know* we are not going to use the copied data in the cpu (but say, only do DMA).
But that should be explicit, not implicit, since the general case will be that the kernel WILL use the data.

Most of the callers probably want the normal one, but most of the copied data (buffered filesystem I/O) will want the non cache poluting one.

So yes, doing this explicit makes a lot of sense.

Hiro Yoshioka

author

2005-08-15 06:43 UTC #5 lore raw

In reply to Arjan van de Ven

Hi,

From: Arjan van de Ven <[email protected]>
Subject: Re: [RFC] [PATCH] cache pollution aware __copy_from_user_ll()
Date: Sun, 14 Aug 2005 12:35:43 +0200
Message-ID: <[email protected]>

Quoted 25 lines · Hiro Yoshioka

Quoted 14 lines
Thanks for your comments.
Quoted 11 lines · Arjan van de Ven
Quoted 8 lines · Hiro Yoshioka
Hi,
The following is a patch to reduce a cache pollution of __copy_from_user_ll().
When I run simple iozone benchmark to find a performance bottleneck of the linux kernel, I found that __copy_from_user_ll() spent CPU cycle most and it did many cache misses.
Quoted 5 lines
Quoted 3 lines
however... you copy something from userspace... aren't you going to USE it? The non-termoral versions actually throw the data out of the cache... so while this part might be nice, you pay BIG elsewhere....
The oprofile data does not give an evidence that we pay BIG elsewhere.
the problem is that the pay elsewhere is far more spread out, but not less. At least generally....

Quoted 6 lines

I can see the point of a copy_from_user_nocache() or something, for those cases where we *know* we are not going to use the copied data in the cpu (but say, only do DMA).
But that should be explicit, not implicit, since the general case will be that the kernel WILL use the data. And if that's the case your change is a loss.... (just harder to see because the cost is spread out)

I understand the iozone is not good benchmark nor reprsents any useful application so I did a kernel build as a simple benchmark.

What I did is

    cd /test/f1
    tar xjf ${baseDir}/src/linux-2.6.12.4.tar.bz2
    cd linux-2.6.12.4
    cp -p ${baseDir}/src/config .config
    make oldconfig
    time make -j $CPUS

The following is Top 5 of CPU cycle
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (mandatory) count 10
0000

samples  %        app name                 symbol name
7347544  72.8296  cc1                      (no symbols)
532307    5.2763  libbz2.so.1.0.2          (no symbols)
241853    2.3973  vmlinux                  buffered_rmqueue
128552    1.2742  libc-2.3.4.so            _int_malloc
107784    1.0684  vmlinux                  page_fault

...
10749 0.1065 vmlinux __copy_from_user_ll
pattern12-0-cpu4-0-08150920/summary.out

Since __copy_from_user_ll is not hot spot, so we didn't see any big performance difference. (the number is time (sec) of 5 runs)

original 2.6.12.4               real    user    system
No profiling                    532.27	1797.02	194.9
BSQ 0x200+0x3f                  620.15	2094.21	212.38

GLOBAL_POWER_EVENTS:100000: 586.01 1984.92 215.97

cache aware 2.6.12.4            real    user    system
No profiling                    526.65	1792.22	190.05
BSQ 0x200+0x3f                  615.51	2090.74	206.58
GLOBAL_POWER_EVENTS:100000:     587.69	1978.66	209.18

Now Top 5 of Memory Access (2.6.12.4)
Counted BSQ_CACHE_REFERENCE events (cache references seen by the bus unit) with a unit mask of 0x3f (multiple flags) count 3000 Counted BSQ_CACHE_REFERENCE events (cache references seen by the bus unit) with a unit mask of 0x200 (read 3rd level cache miss) count 3000

samples  %        samples  %        app name             symbol name
11439689 82.2135  33906    27.9328  cc1                  (no symbols)
277177    1.9920  347       0.2859  libc-2.3.4.so        _int_malloc
229593    1.6500  12946    10.6653  libbz2.so.1.0.2      (no symbols)
84348     0.6062  116       0.0956  libc-2.3.4.so        _int_free
83653     0.6012  438       0.3608  libc-2.3.4.so        calloc

...
8527 0.0613 1648 1.3577 vmlinux __copy_from_user_ll

Top 5 of Cache miss

33906   27.9328 cc1                     (no symbols)
30849   25.4144 vmlinux                 buffered_rmqueue
12946   10.6653 libbz2.so.1.0.2         (no symbols)
9178    7.5611  vmlinux                 __copy_to_user_ll
2934    2.4171  oprofiled               (no symbols)

...
1648 1.3577 vmlinux __copy_from_user_ll
pattern12-0-cpu4-0-08150917

Cache aware 2.6.12.4, Top 5 of Memory Access

samples  %        samples  %        app name             symbol name
11448487 82.8100  32786    28.1051  cc1                  (no symbols)
276812    2.0023  256       0.2195  libc-2.3.4.so        _int_malloc
230177    1.6649  12371    10.6048  libbz2.so.1.0.2      (no symbols)
84485     0.6111  120       0.1029  libc-2.3.4.so        _int_free
84043     0.6079  473       0.4055  libc-2.3.4.so        calloc

...
18282 0.1322 9060 7.7665 vmlinux __copy_from_user_ll

Top 5 of Cache miss

32786   28.1051 cc1                     (no symbols)
31175   26.7241 vmlinux                 buffered_rmqueue
12371   10.6048 libbz2.so.1.0.2         (no symbols)
9060    7.7665  vmlinux                 __copy_from_user_ll
2801    2.4011  oprofiled               (no symbols)

...
0 0 vmlinux __copy_to_user_ll
pattern12-0-cpu4-0-08151048

Cache miss of __copy_from_user_ll has been increased but __copy_to_user_ll has been decreased to 0. (oprofile could not get a sample.)

I don't know the reason why __copy_to_user_ll has been decreased.

Anyway we could not find the cache aware version of __copy_from_user_ll has a big regression yet.

What do you think?
Hiro

Arjan van de Ven

2005-08-15 07:16 UTC #6 lore raw

Quoted 3 lines

Anyway we could not find the cache aware version of __copy_from_user_ll has a big regression yet.

that is because you spread the cache misses out from one place to all over the place, so that no one single point sticks out anymore.

Do you agree that your copy is less optimal for the case where the kernel will (almost) immediately use the data?

I agree that your copy is really nice for places where the kernel will NOT use the data in the cpu, say for big write() system calls.

My suggestion is to realize there are basically 2 different use cases, and that in the code the first one is very common, while in your profiles the second one is very common. Based on that I suggest to make a special copy_from_user_nocache() API for the cases where the kernel will not use the data (and ignore software raid5 here) and use your excellent version for that API, while leaving the code for the cases where the kernel WILL use the data alone. Code wise the "will use" case is the vast majority, so only changing the few places that know they don't use the data will be very efficient, and will give immediate big improvement in your profile data, since those few places tend to get used a lot in the cases you benchmark.

Hiro Yoshioka

author

2005-08-15 08:44 UTC #7 lore raw

Hi,

I appreciate your suggestion.

Quoted 10 lines · Arjan van de Ven

Quoted 3 lines
Anyway we could not find the cache aware version of __copy_from_user_ll has a big regression yet.
that is because you spread the cache misses out from one place to all over the place, so that no one single point sticks out anymore.
Do you agree that your copy is less optimal for the case where the kernel will (almost) immediately use the data?

Yes, I do.

My server has 8KB of L1 cache. (512KB of L2/2MB of L3)

If you move more than 4KB of data using by __copy_from_user_ll(), the data will be spilled over L1 cache but in L2 (or L3)
When you move huge data (> 1MB), even L3 cache will not help you. (This is known as a cache pollution.)

Quoted 14 lines

I agree that your copy is really nice for places where the kernel will NOT use the data in the cpu, say for big write() system calls.
My suggestion is to realize there are basically 2 different use cases, and that in the code the first one is very common, while in your profiles the second one is very common. Based on that I suggest to make a special copy_from_user_nocache() API for the cases where the kernel will not use the data (and ignore software raid5 here) and use your excellent version for that API, while leaving the code for the cases where the kernel WILL use the data alone. Code wise the "will use" case is the vast majority, so only changing the few places that know they don't use the data will be very efficient, and will give immediate big improvement in your profile data, since those few places tend to get used a lot in the cases you benchmark.

copy_from_user_nocache() is fine.

But I don't know where I can use it. (I'm not so
familiar with the linux kernel file system yet.)

Regards,
Hiro

Arjan van de Ven

2005-08-15 08:53 UTC #8 lore raw

Quoted 16 lines · Hiro Yoshioka

Hi,
I appreciate your suggestion.
Quoted 10 lines · Arjan van de Ven
Quoted 3 lines
Anyway we could not find the cache aware version of __copy_from_user_ll has a big regression yet.
that is because you spread the cache misses out from one place to all over the place, so that no one single point sticks out anymore.
Do you agree that your copy is less optimal for the case where the kernel will (almost) immediately use the data?

Quoted 6 lines

Yes, I do.
My server has 8KB of L1 cache. (512KB of L2/2MB of L3)
If you move more than 4KB of data using by __copy_from_user_ll(), the data will be spilled over L1 cache but in L2 (or L3)

L2 access time isn't too bad. your code evicts the data even from L2 and L3 though (even if it was in there before)..

Quoted 2 lines

When you move huge data (> 1MB), even L3 cache will not help you. (This is known as a cache pollution.)

yes.

Quoted 4 lines

copy_from_user_nocache() is fine.
But I don't know where I can use it. (I'm not so
familiar with the linux kernel file system yet.)

I suspect the few cases where it will make the most difference will be in the VFS for the write() system call, and the AIO variants thereof.

generic_file_buffered_write() will be a good candidate to try first...

Hiro Yoshioka

author

2005-08-15 23:34 UTC #9 lore raw

Quoted 9 lines · Arjan van de Ven

Quoted 4 lines
copy_from_user_nocache() is fine.
But I don't know where I can use it. (I'm not so
familiar with the linux kernel file system yet.)
I suspect the few cases where it will make the most difference will be in the VFS for the write() system call, and the AIO variants thereof.
generic_file_buffered_write() will be a good candidate to try first...

Thanks.

filemap_copy_from_user() calls __copy_from_user_inatomic() calls __copy_from_user_ll().

I'll look at the code.

Hiro
--
Hiro Yoshioka
mailto:hyoshiok at miraclelinux.com

Hiro Yoshioka

2005-08-16 03:35 UTC #10 lore raw

From: Hiro Yoshioka <[email protected]>
Date: Tue, 16 Aug 2005 08:33:59 +0900

Quoted 6 lines

Thanks.
filemap_copy_from_user() calls __copy_from_user_inatomic() calls __copy_from_user_ll().
I'll look at the code.

The following is a quick hack of cache aware implementation of __copy_from_user_ll() and __copy_from_user_inatomic()

__copy_from_user_ll_nocache() and __copy_from_user_inatomic_nocache()

filemap_copy_from_user() calles __copy_from_user_inatomic_nocache() instead of __copy_from_user_inatomic() and reduced cashe miss.

The first column is the cache reference (memory access) and the third column is the 3rd level cache miss.

The following example shows the L3 cache miss is reduced from 37410 to 107.

2.6.12.4 nocache version
Counted BSQ_CACHE_REFERENCE events (cache references seen by the bus unit) with a unit mask of 0x3f (multiple flags) count 3000 Counted BSQ_CACHE_REFERENCE events (cache references seen by the bus unit) with a unit mask of 0x200 (read 3rd level cache miss) count 3000

samples  %        samples  %     app name       symbol name
120442    6.4106  107    0.5620  vmlinux        __copy_user_zeroing_nocache
80049     4.2606  578    3.0357  vmlinux        journal_add_journal_head
69194     3.6829  154    0.8088  vmlinux        journal_dirty_metadata
67059     3.5692  78     0.4097  vmlinux        __find_get_block
64145     3.4141  32     0.1681  vmlinux        journal_put_journal_head

pattern9-0-cpu4-0-08161154/summary.out

The 2.6.12.4 original version is
Counted BSQ_CACHE_REFERENCE events (cache references seen by the bus unit) with a unit mask of 0x3f (multiple flags) count 3000 Counted BSQ_CACHE_REFERENCE events (cache references seen by the bus unit) with a unit mask of 0x200 (read 3rd level cache miss) count 3000

samples  %        samples  %     app name       symbol name
120646    7.4680  37410 62.3355  vmlinux        __copy_from_user_ll
79508     4.9215  903    1.5046  vmlinux        _spin_lock
65526     4.0561  873    1.4547  vmlinux        journal_add_journal_head
59296     3.6704  129    0.2149  vmlinux        __find_get_block
58647     3.6302  215    0.3582  vmlinux        journal_dirty_metadata

What do you think?

Hiro

Makefile +1 -1

diff -ur linux-2.6.12.4.orig/Makefile linux-2.6.12.4.nocache/Makefile
@@ -1,7 +1,7 @@
 VERSION = 2
 PATCHLEVEL = 6
 SUBLEVEL = 12
-EXTRAVERSION = .4.orig
+EXTRAVERSION = .4.nocache
 NAME=Woozy Numbat
 
 # *DOCUMENTATION*

arch/i386/lib/usercopy.c +120 -0

138 lines not on this page · raw message

include/asm/uaccess.h +29 -0

diff -ur linux-2.6.12.4.orig/include/asm/uaccess.h linux-2.6.12.4.nocache/include/asm/uaccess.h
@@ -413,6 +413,8 @@
 				const void *from, unsigned long n);
 unsigned long __must_check __copy_from_user_ll(void *to,
 				const void __user *from, unsigned long n);
+unsigned long __must_check __copy_from_user_ll_nocache(void *to,
+				const void __user *from, unsigned long n);
 
 /*
  * Here we special-case 1, 2 and 4-byte copy_*_user invocations.  On a fault

@@ -502,11 +504,38 @@
 }
 
 static inline unsigned long
+__copy_from_user_inatomic_nocache(void *to, const void __user *from, unsigned long n)
+{
+	if (__builtin_constant_p(n)) {
+		unsigned long ret;
+
+		switch (n) {
+		case 1:
+			__get_user_size(*(u8 *)to, from, 1, ret, 1);
+			return ret;
+		case 2:
+			__get_user_size(*(u16 *)to, from, 2, ret, 2);
+			return ret;
+		case 4:
+			__get_user_size(*(u32 *)to, from, 4, ret, 4);
+			return ret;
+		}
+	}
+	return __copy_from_user_ll_nocache(to, from, n);
+}
+
+static inline unsigned long
 __copy_from_user(void *to, const void __user *from, unsigned long n)
 {
        might_sleep();
        return __copy_from_user_inatomic(to, from, n);
 }
+static inline unsigned long
+__copy_from_user_nocache(void *to, const void __user *from, unsigned long n)
+{
+       might_sleep();
+       return __copy_from_user_inatomic_nocache(to, from, n);
+}
 unsigned long __must_check copy_to_user(void __user *to,
 				const void *from, unsigned long n);
 unsigned long __must_check copy_from_user(void *to,

include/asm-i386/uaccess.h +29 -0

diff -ur linux-2.6.12.4.orig/include/asm-i386/uaccess.h linux-2.6.12.4.nocache/include/asm-i386/uaccess.h
@@ -413,6 +413,8 @@
 				const void *from, unsigned long n);
 unsigned long __must_check __copy_from_user_ll(void *to,
 				const void __user *from, unsigned long n);
+unsigned long __must_check __copy_from_user_ll_nocache(void *to,
+				const void __user *from, unsigned long n);
 
 /*
  * Here we special-case 1, 2 and 4-byte copy_*_user invocations.  On a fault

@@ -502,11 +504,38 @@
 }
 
 static inline unsigned long
+__copy_from_user_inatomic_nocache(void *to, const void __user *from, unsigned long n)
+{
+	if (__builtin_constant_p(n)) {
+		unsigned long ret;
+
+		switch (n) {
+		case 1:
+			__get_user_size(*(u8 *)to, from, 1, ret, 1);
+			return ret;
+		case 2:
+			__get_user_size(*(u16 *)to, from, 2, ret, 2);
+			return ret;
+		case 4:
+			__get_user_size(*(u32 *)to, from, 4, ret, 4);
+			return ret;
+		}
+	}
+	return __copy_from_user_ll_nocache(to, from, n);
+}
+
+static inline unsigned long
 __copy_from_user(void *to, const void __user *from, unsigned long n)
 {
        might_sleep();
        return __copy_from_user_inatomic(to, from, n);
 }
+static inline unsigned long
+__copy_from_user_nocache(void *to, const void __user *from, unsigned long n)
+{
+       might_sleep();
+       return __copy_from_user_inatomic_nocache(to, from, n);
+}
 unsigned long __must_check copy_to_user(void __user *to,
 				const void *from, unsigned long n);
 unsigned long __must_check copy_from_user(void *to,

include/linux/autoconf.h +2 -2

diff -ur linux-2.6.12.4.orig/include/linux/autoconf.h linux-2.6.12.4.nocache/include/linux/autoconf.h
@@ -1,7 +1,7 @@
 /*
  * Automatically generated C config: don't edit
- * Linux kernel version: 2.6.12.4.orig
- * Mon Aug 15 16:53:01 2005
+ * Linux kernel version: 2.6.12.4.nocache
+ * Tue Aug 16 10:32:33 2005
  */
 #define AUTOCONF_INCLUDED
 #define CONFIG_X86 1

mm/filemap.c +3 -3

diff -ur linux-2.6.12.4.orig/mm/filemap.c linux-2.6.12.4.nocache/mm/filemap.c
@@ -1727,13 +1727,13 @@
 	int left;
 
 	kaddr = kmap_atomic(page, KM_USER0);
-	left = __copy_from_user_inatomic(kaddr + offset, buf, bytes);
+	left = __copy_from_user_inatomic_nocache(kaddr + offset, buf, bytes);
 	kunmap_atomic(kaddr, KM_USER0);
 
 	if (left != 0) {
 		/* Do it the slow way */
 		kaddr = kmap(page);
-		left = __copy_from_user(kaddr + offset, buf, bytes);
+		left = __copy_from_user_nocache(kaddr + offset, buf, bytes);
 		kunmap(page);
 	}
 	return bytes - left;

@@ -1750,7 +1750,7 @@
 		int copy = min(bytes, iov->iov_len - base);
 
 		base = 0;
-		left = __copy_from_user_inatomic(vaddr, buf, copy);
+		left = __copy_from_user_inatomic_nocache(vaddr, buf, copy);
 		copied += copy;
 		bytes -= copy;
 		vaddr += copy;

Hirokazu Takahashi

2005-08-16 04:22 UTC #11 lore raw

Hi,

BTW, what are you going to do with the page-faults which may happen during __copy_user_zeroing_nocache()? The current process may be blocked in the handler for a while and get FPU registers polluted. kernel_fpu_begin() won't help the case. This is another issue, though.

Quoted 350 lines

Quoted 6 lines

Thanks.
filemap_copy_from_user() calls __copy_from_user_inatomic() calls __copy_from_user_ll().
I'll look at the code.

The following is a quick hack of cache aware implementation of __copy_from_user_ll() and __copy_from_user_inatomic()

__copy_from_user_ll_nocache() and __copy_from_user_inatomic_nocache()

filemap_copy_from_user() calles __copy_from_user_inatomic_nocache() instead of __copy_from_user_inatomic() and reduced cashe miss.

The first column is the cache reference (memory access) and the third column is the 3rd level cache miss.

The following example shows the L3 cache miss is reduced from 37410 to 107.

samples  %        samples  %     app name       symbol name
120442    6.4106  107    0.5620  vmlinux        __copy_user_zeroing_nocache
80049     4.2606  578    3.0357  vmlinux        journal_add_journal_head
69194     3.6829  154    0.8088  vmlinux        journal_dirty_metadata
67059     3.5692  78     0.4097  vmlinux        __find_get_block
64145     3.4141  32     0.1681  vmlinux        journal_put_journal_head

pattern9-0-cpu4-0-08161154/summary.out

samples  %        samples  %     app name       symbol name
120646    7.4680  37410 62.3355  vmlinux        __copy_from_user_ll
79508     4.9215  903    1.5046  vmlinux        _spin_lock
65526     4.0561  873    1.4547  vmlinux        journal_add_journal_head
59296     3.6704  129    0.2149  vmlinux        __find_get_block
58647     3.6302  215    0.3582  vmlinux        journal_dirty_metadata

What do you think?

Hiro

diff -ur linux-2.6.12.4.orig/Makefile linux-2.6.12.4.nocache/Makefile

Makefile +1 -1

@@ -1,7 +1,7 @@
 VERSION = 2
 PATCHLEVEL = 6
 SUBLEVEL = 12
-EXTRAVERSION = .4.orig
+EXTRAVERSION = .4.nocache
 NAME=Woozy Numbat
 
 # *DOCUMENTATION*

diff -ur linux-2.6.12.4.orig/arch/i386/lib/usercopy.c linux-2.6.12.4.nocache/arch/i386/lib/usercopy.c

arch/i386/lib/usercopy.c +120 -0

@@ -10,6 +10,7 @@
 #include <linux/highmem.h>
 #include <linux/blkdev.h>
 #include <linux/module.h>
+#include <asm/i387.h>
 #include <asm/uaccess.h>
 #include <asm/mmx.h>

@@ -511,6 +512,110 @@
 		: "memory");						\
 } while (0)
 
+/* Non Temporal Hint version of mmx_memcpy */
+/* It is cache aware                       */
+/* [email protected]               */
+static unsigned long 
+__copy_user_zeroing_nocache(void *to, const void *from, size_t len)
+{
+        /* Note! gcc doesn't seem to align stack variables properly, so we
+         * need to make use of unaligned loads and stores.
+         */
+	void *p;
+	int i;
+
+	if (unlikely(in_interrupt())){
+	        __copy_user_zeroing(to, from, len);
+		return len;
+	}
+
+	p = to;
+	i = len >> 6; /* len/64 */
+
+        kernel_fpu_begin();
+
+	__asm__ __volatile__ (
+		"1: prefetchnta (%0)\n"		/* This set is 28 bytes */
+		"   prefetchnta 64(%0)\n"
+		"   prefetchnta 128(%0)\n"
+		"   prefetchnta 192(%0)\n"
+		"   prefetchnta 256(%0)\n"
+		"2:  \n"
+		".section .fixup, \"ax\"\n"
+		"3: movw $0x1AEB, 1b\n"	/* jmp on 26 bytes */
+		"   jmp 2b\n"
+		".previous\n"
+		".section __ex_table,\"a\"\n"
+		"	.align 4\n"
+		"	.long 1b, 3b\n"
+		".previous"
+		: : "r" (from) );
+		
+	for(; i>5; i--)
+	{
+		__asm__ __volatile__ (
+		"1:  prefetchnta 320(%0)\n"
+		"2:  movq (%0), %%mm0\n"
+		"  movq 8(%0), %%mm1\n"
+		"  movq 16(%0), %%mm2\n"
+		"  movq 24(%0), %%mm3\n"
+		"  movntq %%mm0, (%1)\n"
+		"  movntq %%mm1, 8(%1)\n"
+		"  movntq %%mm2, 16(%1)\n"
+		"  movntq %%mm3, 24(%1)\n"
+		"  movq 32(%0), %%mm0\n"
+		"  movq 40(%0), %%mm1\n"
+		"  movq 48(%0), %%mm2\n"
+		"  movq 56(%0), %%mm3\n"
+		"  movntq %%mm0, 32(%1)\n"
+		"  movntq %%mm1, 40(%1)\n"
+		"  movntq %%mm2, 48(%1)\n"
+		"  movntq %%mm3, 56(%1)\n"
+		".section .fixup, \"ax\"\n"
+		"3: movw $0x05EB, 1b\n"	/* jmp on 5 bytes */
+		"   jmp 2b\n"
+		".previous\n"
+		".section __ex_table,\"a\"\n"
+		"	.align 4\n"
+		"	.long 1b, 3b\n"
+		".previous"
+		: : "r" (from), "r" (to) : "memory");
+		from+=64;
+		to+=64;
+	}
+
+	for(; i>0; i--)
+	{
+		__asm__ __volatile__ (
+		"  movq (%0), %%mm0\n"
+		"  movq 8(%0), %%mm1\n"
+		"  movq 16(%0), %%mm2\n"
+		"  movq 24(%0), %%mm3\n"
+		"  movntq %%mm0, (%1)\n"
+		"  movntq %%mm1, 8(%1)\n"
+		"  movntq %%mm2, 16(%1)\n"
+		"  movntq %%mm3, 24(%1)\n"
+		"  movq 32(%0), %%mm0\n"
+		"  movq 40(%0), %%mm1\n"
+		"  movq 48(%0), %%mm2\n"
+		"  movq 56(%0), %%mm3\n"
+		"  movntq %%mm0, 32(%1)\n"
+		"  movntq %%mm1, 40(%1)\n"
+		"  movntq %%mm2, 48(%1)\n"
+		"  movntq %%mm3, 56(%1)\n"
+		: : "r" (from), "r" (to) : "memory");
+		from+=64;
+		to+=64;
+	}
+	/*
+	 *	Now do the tail of the block
+	 */
+	kernel_fpu_end();
+	if(i=(len&63))
+	  __copy_user_zeroing(to, from, i);
+	return i;
+}
+
 
 unsigned long __copy_to_user_ll(void __user *to, const void *from, unsigned long n)
 {

@@ -582,6 +687,21 @@
 	return n;
 }
 
+unsigned long
+__copy_from_user_ll_nocache(void *to, const void __user *from, unsigned long n)
+{
+	BUG_ON((long)n < 0);
+        if (n < 512) {
+          if (movsl_is_ok(to, from, n))
+                __copy_user_zeroing(to, from, n);
+          else
+                n = __copy_user_zeroing_intel(to, from, n);
+        }
+        else
+          n = __copy_user_zeroing_nocache(to, from, n);
+	return n;
+}
+
 /**
  * copy_to_user: - Copy a block of data into user space.
  * @to:   Destination address, in user space.

diff -ur linux-2.6.12.4.orig/include/asm/uaccess.h linux-2.6.12.4.nocache/include/asm/uaccess.h

include/asm/uaccess.h +29 -0

@@ -413,6 +413,8 @@
 				const void *from, unsigned long n);
 unsigned long __must_check __copy_from_user_ll(void *to,
 				const void __user *from, unsigned long n);
+unsigned long __must_check __copy_from_user_ll_nocache(void *to,
+				const void __user *from, unsigned long n);
 
 /*
  * Here we special-case 1, 2 and 4-byte copy_*_user invocations.  On a fault

@@ -502,11 +504,38 @@
 }
 
 static inline unsigned long
+__copy_from_user_inatomic_nocache(void *to, const void __user *from, unsigned long n)
+{
+	if (__builtin_constant_p(n)) {
+		unsigned long ret;
+
+		switch (n) {
+		case 1:
+			__get_user_size(*(u8 *)to, from, 1, ret, 1);
+			return ret;
+		case 2:
+			__get_user_size(*(u16 *)to, from, 2, ret, 2);
+			return ret;
+		case 4:
+			__get_user_size(*(u32 *)to, from, 4, ret, 4);
+			return ret;
+		}
+	}
+	return __copy_from_user_ll_nocache(to, from, n);
+}
+
+static inline unsigned long
 __copy_from_user(void *to, const void __user *from, unsigned long n)
 {
        might_sleep();
        return __copy_from_user_inatomic(to, from, n);
 }
+static inline unsigned long
+__copy_from_user_nocache(void *to, const void __user *from, unsigned long n)
+{
+       might_sleep();
+       return __copy_from_user_inatomic_nocache(to, from, n);
+}
 unsigned long __must_check copy_to_user(void __user *to,
 				const void *from, unsigned long n);
 unsigned long __must_check copy_from_user(void *to,

diff -ur linux-2.6.12.4.orig/include/asm-i386/uaccess.h linux-2.6.12.4.nocache/include/asm-i386/uaccess.h

include/asm-i386/uaccess.h +29 -0

@@ -413,6 +413,8 @@
 				const void *from, unsigned long n);
 unsigned long __must_check __copy_from_user_ll(void *to,
 				const void __user *from, unsigned long n);
+unsigned long __must_check __copy_from_user_ll_nocache(void *to,
+				const void __user *from, unsigned long n);
 
 /*
  * Here we special-case 1, 2 and 4-byte copy_*_user invocations.  On a fault

@@ -502,11 +504,38 @@
 }
 
 static inline unsigned long
+__copy_from_user_inatomic_nocache(void *to, const void __user *from, unsigned long n)
+{
+	if (__builtin_constant_p(n)) {
+		unsigned long ret;
+
+		switch (n) {
+		case 1:
+			__get_user_size(*(u8 *)to, from, 1, ret, 1);
+			return ret;
+		case 2:
+			__get_user_size(*(u16 *)to, from, 2, ret, 2);
+			return ret;
+		case 4:
+			__get_user_size(*(u32 *)to, from, 4, ret, 4);
+			return ret;
+		}
+	}
+	return __copy_from_user_ll_nocache(to, from, n);
+}
+
+static inline unsigned long
 __copy_from_user(void *to, const void __user *from, unsigned long n)
 {
        might_sleep();
        return __copy_from_user_inatomic(to, from, n);
 }
+static inline unsigned long
+__copy_from_user_nocache(void *to, const void __user *from, unsigned long n)
+{
+       might_sleep();
+       return __copy_from_user_inatomic_nocache(to, from, n);
+}
 unsigned long __must_check copy_to_user(void __user *to,
 				const void *from, unsigned long n);
 unsigned long __must_check copy_from_user(void *to,

diff -ur linux-2.6.12.4.orig/include/linux/autoconf.h linux-2.6.12.4.nocache/include/linux/autoconf.h

include/linux/autoconf.h +2 -2

@@ -1,7 +1,7 @@
 /*
  * Automatically generated C config: don't edit
- * Linux kernel version: 2.6.12.4.orig
- * Mon Aug 15 16:53:01 2005
+ * Linux kernel version: 2.6.12.4.nocache
+ * Tue Aug 16 10:32:33 2005
  */
 #define AUTOCONF_INCLUDED
 #define CONFIG_X86 1

diff -ur linux-2.6.12.4.orig/mm/filemap.c linux-2.6.12.4.nocache/mm/filemap.c

mm/filemap.c +3 -4

@@ -1727,13 +1727,13 @@
 	int left;
 
 	kaddr = kmap_atomic(page, KM_USER0);
-	left = __copy_from_user_inatomic(kaddr + offset, buf, bytes);
+	left = __copy_from_user_inatomic_nocache(kaddr + offset, buf, bytes);
 	kunmap_atomic(kaddr, KM_USER0);
 
 	if (left != 0) {
 		/* Do it the slow way */
 		kaddr = kmap(page);
-		left = __copy_from_user(kaddr + offset, buf, bytes);
+		left = __copy_from_user_nocache(kaddr + offset, buf, bytes);
 		kunmap(page);
 	}
 	return bytes - left;

@@ -1750,7 +1750,7 @@
 		int copy = min(bytes, iov->iov_len - base);
 
 		base = 0;
-		left = __copy_from_user_inatomic(vaddr, buf, copy);
+		left = __copy_from_user_inatomic_nocache(vaddr, buf, copy);
 		copied += copy;
 		bytes -= copy;
 		vaddr += copy;
-

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Hiro Yoshioka

2005-08-16 04:59 UTC #12 lore raw

Takahashi san,

I appreciate your comments.

Quoted 6 lines

Hi,
BTW, what are you going to do with the page-faults which may happen during __copy_user_zeroing_nocache()? The current process may be blocked in the handler for a while and get FPU registers polluted. kernel_fpu_begin() won't help the case. This is another issue, though.

My code does nothing do it.

I need a volunteer to implement it.

Regards,
Hiro

Arjan van de Ven

2005-08-16 05:43 UTC #13 lore raw

Quoted 6 lines · Hiro Yoshioka

Takahashi san,
I appreciate your comments.
Quoted 2 lines
Hi,

Quoted 8 lines

Quoted 4 lines
BTW, what are you going to do with the page-faults which may happen during __copy_user_zeroing_nocache()? The current process may be blocked in the handler for a while and get FPU registers polluted. kernel_fpu_begin() won't help the case. This is another issue, though.
My code does nothing do it.
I need a volunteer to implement it.

it's actually not too hard; all you need is to use SSE and not MMX; and then just store sse register you're overwriting on the stack or so...

Hiro Yoshioka

2005-08-16 10:21 UTC #14 lore raw

From: Arjan van de Ven <[email protected]>

Quoted 6 lines

Quoted 3 lines
My code does nothing do it.
I need a volunteer to implement it.
it's actually not too hard; all you need is to use SSE and not MMX; and then just store sse register you're overwriting on the stack or so...

oh, really? Does the linux kernel take care of
SSE save/restore on a task switch?

Regards,
Hiro

Hirokazu Takahashi

2005-08-16 10:24 UTC #15 lore raw

Quoted 9 lines

Quoted 6 lines
Quoted 3 lines
My code does nothing do it.
I need a volunteer to implement it.
it's actually not too hard; all you need is to use SSE and not MMX; and then just store sse register you're overwriting on the stack or so...
oh, really? Does the linux kernel take care of
SSE save/restore on a task switch?

noop!

Arjan van de Ven

2005-08-16 10:25 UTC #16 lore raw

In reply to Hiro Yoshioka

Quoted 10 lines · Hiro Yoshioka

From: Arjan van de Ven <[email protected]>
Quoted 6 lines
Quoted 3 lines
My code does nothing do it.
I need a volunteer to implement it.
it's actually not too hard; all you need is to use SSE and not MMX; and then just store sse register you're overwriting on the stack or so...
oh, really? Does the linux kernel take care of
SSE save/restore on a task switch?

not on kernel entry afaik.
However just save the register on the stack and put it back at the end...

Hirokazu Takahashi

2005-08-16 10:29 UTC #17 lore raw

Hi,

Quoted 7 lines

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
My code does nothing do it.
I need a volunteer to implement it.
it's actually not too hard; all you need is to use SSE and not MMX; and then just store sse register you're overwriting on the stack or so...

Quoted 6 lines

Quoted 2 lines
oh, really? Does the linux kernel take care of
SSE save/restore on a task switch?
not on kernel entry afaik.
However just save the register on the stack and put it back at the end...

I think this have to be done in the pagefault handlers.

Thanks,
Hirokazu Takahashi.

Arjan van de Ven

2005-08-16 05:44 UTC #18 lore raw

In reply to Hirokazu Takahashi

Quoted 6 lines · Hirokazu Takahashi

Hi,
BTW, what are you going to do with the page-faults which may happen during __copy_user_zeroing_nocache()? The current process may be blocked in the handler for a while and get FPU registers polluted. kernel_fpu_begin() won't help the case. This is another issue, though.

__copy_from_user_inatomic

.. that implies it won't sleep actually ;)

Arjan van de Ven

2005-08-16 05:49 UTC #19 lore raw

In reply to Hiro Yoshioka

Quoted 1 line · Hiro Yoshioka

The following example shows the L3 cache miss is reduced from 37410 to 107.

most impressive; it seems the approach to do this selectively is paying off very well!

The only comment/question I have is about the use of prefetchnta; that might have cache-evicting properties as well (eg evict the cache of the original of the copy, eg the userspace memory). Is that really the right approach?
In addition, my measurements show that removing the prefetch from the main copy loop is a gain because the modern cpus have an autoprefetcher already in the hardware.

       "1: prefetchnta (%0)\n"         /* This set is 28 bytes */
+               "   prefetchnta 64(%0)\n"
+               "   prefetchnta 128(%0)\n"
+               "   prefetchnta 192(%0)\n"
+               "   prefetchnta 256(%0)\n"
+               "2:  \n"
+               ".section .fixup, \"ax\"\n"
+               "3: movw $0x1AEB, 1b\n" /* jmp on 26 bytes */
+               "   jmp 2b\n"
+               ".previous\n"

oh and prefetch(nta) is a non-faulting instruction so no need for the fixup handling...

But overall this is starting to look really interesting!

Greetings,
Arjan van de Ven

A message that is not in the archive (sent off-list or not yet received): [email protected]

Hiro Yoshioka

author

2005-08-17 05:10 UTC #21 lore raw

Akira,

Thanks for your suggestions.

Quoted 4 lines · Akira Tsukamoto

Anyway, going back to copy_user topic,
big remaining issues are
1)store/restore floating point register (80/64bytes) twice every time by
surrounding with kernel_fpu_begin()/kernel_fpu_end() is big penalty

I don't know. If nobody uses MMX/XMM, then there is no need to save and restore.

Quoted 1 line

2)after pagefault not always come back to copy function and corrupts fp register

I'm trying to understand this mechanism but I don't understand very well.

Quoted 6 lines

3)disabling long preemption
Please correct me if I am wrong.
I tried to implement fpsave inside pagefault handler once and here is my junk;
http://www.suna-asobi.com/~akira-t/linux/k7-copy-user/K7-copy_47_with_fpusave_not_finished.patch
never had a time to finish it. Hiro, does it help you?

Thanks. I'm reading your patch but could not understand very well.

I'll ask you.

Regards,
Hiro

Signature · 2 lines

Hiro Yoshioka
mailto:hyoshiok at miraclelinux.com

Akira Tsukamoto

2005-08-17 14:23 UTC #22 lore raw

On Wed, 17 Aug 2005 14:10:34 +0900
Hiro Yoshioka <[email protected]> mentioned:

Quoted 8 lines · Akira Tsukamoto

Quoted 4 lines
Anyway, going back to copy_user topic,
big remaining issues are
1)store/restore floating point register (80/64bytes) twice every time by
surrounding with kernel_fpu_begin()/kernel_fpu_end() is big penalty
I don't know. If nobody uses MMX/XMM, then there is no need to save and restore.

I think you are misunderstanding between
1)lazy fpu save handling for user space task 2)kernel_fpu_begin()/kernel_fpu_end() inside the kernel

Quoted 4 lines

Quoted 1 line
2)after pagefault not always come back to copy function and corrupts fp register
I'm trying to understand this mechanism but I don't understand very well.

My explanation was a bit ambiguous, see the code below. Where the fp register saved? It saves fp register *inside* task_struct,

static inline void kernel_fpu_begin(void)
+ if (tsk->flags & PF_USEDFPU) {
+ asm volatile("rex64 ; fxsave %0 ; fnclex"
+ : "=m" (tsk->thread.i387.fxsave));

static inline void save_init_fpu( struct task_struct *tsk )
+ if ( cpu_has_fxsr ) {
+ asm volatile( "fxrstor %0"
+ : : "m" (tsk->thread.i387.fxsave) );

What happens, during your copy function, if memory is not allocated and generates pagefualt and goto reclaim memories and go into task switch and change to other task.

Signature · 1 line

Akira Tsukamoto

Akira Tsukamoto

2005-08-17 15:20 UTC #23 lore raw

I am resubmitting this because it seems to be lost when I posted the before yesterday.

------------------------------------
Arjan van de Ven mentioned:

Quoted 7 lines

The only comment/question I have is about the use of prefetchnta; that might have cache-evicting properties as well (eg evict the cache of the original of the copy, eg the userspace memory). Is that really the right approach?
In addition, my measurements show that removing the prefetch from the main copy loop is a gain because the modern cpus have an autoprefetcher already in the hardware.

My computer with Athlon K7 was faster with manually prefetching, but I did not know it is already becoming obsolete.

It was pretty while ago, but I also made a similar copy_user function;
http://www.suna-asobi.com/~akira-t/linux/k7-copy-user/K7-copy-47.patch
I add comments on each item in the copy function. It was basically inspired from Takahashi's intel faster copy function.

I also have some explanation about the speedup for pipelined cpu.
http://www.suna-asobi.com/~akira-t/linux/k7-copy-user/copy_for_highlypipelined_cpu.txt

It was originally discussed in this thread,
http://marc.theaimsgroup.com/?l=linux-kernel&m=103742983924070&w=2

Lee Revell

2005-08-18 17:53 UTC #24 lore raw

Quoted 3 lines · Akira Tsukamoto

My computer with Athlon K7 was faster with manually prefetching, but I did not know it is already becoming obsolete.

Don't listen to people who tell you $FOO hardware is obsolete, they have a very narrow view. "Obsolete" is meaningless except in reference to some specific application. The 386 is obsolete on the desktop but still common on the embedded market.

Lee

Akira Tsukamoto

2005-08-18 02:30 UTC #25 lore raw

In reply to Akira Tsukamoto

On Wed, 17 Aug 2005 23:30:13 +0900
Akira Tsukamoto <[email protected]> mentioned:

Quoted 5 lines

Quoted 2 lines
I'm trying to understand this mechanism but I don't understand very well.
My explanation was a bit ambiguous, see the code below. Where the fp register saved? It saves fp register *inside* task_struct,

More clarification, to make fp_save generic,
after exception, such as pagefault, copy function might get nested, during page allocation.
First it has user space fp content, but nested copy needs to save kernel space fp content which came from the first copy function. So saving into task_struct is bit problem.

XMM_SAVE/XMM_RESTORE uses stack for it. Surrounding copy loop with XMM_SAVE/XMM_RESTORE should work.

Some might claim that, saving/restore every time might a big overhead,,, but i think it is better than having a lot of cache miss hit.

Isn't there some way to avoid long preemption disabling?

Signature · 1 line

Akira Tsukamoto <[email protected], [email protected]>