Why is get_user_pages so slow?
Hi all,
Why is get_user_pages much slower than taking the faults? (I would expect it to be faster).
Attached example program first mallocs a piece of memory (64MB in this case) then reads it "to take the faults". Afterwards, it uses mmap with MAP_POPULATE to "speed up" and not to have to take the faults, but have everything mapped in one go. I think mmap is using get_user_pages in this case.
$ ./memspeed malloc took 0 msecs read took 14 msecs write took 0 msecs free took 1 msecs mmap took 45 msecs munmap took 5 msecs
Using MAP_POPULATE is 3 times as slow as the 'stupid' implementation! I'm running a Core 2 duo e6300 system with linux 2.6.28.4.
Am I doing something wrong? MAP_POPULATE seems a bit of a joke to me.
Thanks,
Micha
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <sys/time.h>
#define SIZE (64 << 20)
unsigned tv_msecs(struct timeval *tvs, struct timeval *tve)
{
return (tve->tv_sec - tvs->tv_sec) * 1000 +
(tve->tv_usec - tvs->tv_usec) / 1000;
}int main(void)
{
struct timeval tvs, tve;
void *buf;
int *p, *e, i;
gettimeofday(&tvs, NULL);
buf = malloc(SIZE);
gettimeofday(&tve, NULL);
printf("malloc took %d msecs\n", tv_msecs(&tvs, &tve));
gettimeofday(&tvs, NULL);
for (p = buf, e = buf + SIZE; p < e; p += 0x1000)
i = *(volatile int*)p;
gettimeofday(&tve, NULL);
printf("read took %d msecs\n", tv_msecs(&tvs, &tve));
gettimeofday(&tvs, NULL);
for (p = buf, e = buf + SIZE; p < e; p += 0x1000)
*(volatile int*)p = 0xaa55aa55;
gettimeofday(&tve, NULL);
printf("write took %d msecs\n", tv_msecs(&tvs, &tve));
gettimeofday(&tvs, NULL);
free(buf);
gettimeofday(&tve, NULL);
printf("free took %d msecs\n", tv_msecs(&tvs, &tve));
gettimeofday(&tvs, NULL);
buf = mmap(NULL, SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE |
MAP_ANONYMOUS | MAP_POPULATE, -1, 0);
gettimeofday(&tve, NULL);
printf("mmap took %d msecs\n", tv_msecs(&tvs, &tve));
gettimeofday(&tvs, NULL);
munmap(buf, SIZE);
gettimeofday(&tve, NULL);
printf("munmap took %d msecs\n", tv_msecs(&tvs, &tve));
return 0;
}Quoting Micha Nelissen <[email protected]>:
Quoted 14 lines
Hi all,
Why is
get_user_pagesmuch slower than taking the faults? (I would expect it to be faster).Attached example program first mallocs a piece of memory (64MB in this case) then reads it "to take the faults". Afterwards, it uses mmap with
MAP_POPULATEto "speed up" and not to have to take the faults, but have everything mapped in one go. I think mmap is usingget_user_pagesin this case.$ ./memspeed malloc took 0 msecs read took 14 msecs write took 0 msecs free took 1 msecs mmap took 45 msecs munmap took 5 msecs
Quoted 3 lines
Using
MAP_POPULATEis 3 times as slow as the 'stupid' implementation! I'm running a Core 2 duo e6300 system with linux 2.6.28.4.Am I doing something wrong?
MAP_POPULATEseems a bit of a joke to me.
Hi Micha,
Yep, you are. Because your pointer 'p' is a pointer to int, when you increment it by 0x1000 in your loops you are actually incrementing it by 0x1000 * sizeof(int) - so you're only actually touching one page in four.
If you change the types of 'buf', 'p' and 'e' to 'char *' then it touches every page - and (and least on my test box) the MAP_POPULATE case pulls ahead.
- Kevin
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
Hi Kevin,
Quoted 1 line · Kevin Easton
Yep, you are. Because your pointer 'p' is a pointer to int, when you increment it by
0x1000in your loops you are actually incrementing it by0x1000* sizeof(int) - so you're only actually touching one page in four.
Oops sorry, thanks for catching my mistake.
I also discovered the following: if I read from all pages, then call get_user_pages, it is still quite slow (did I get a read-only page?). However, if I touch all pages by writing to them, then get_user_pages becomes a factor 40 times faster or so.
All is clear now, I think. Thanks.
Micha