[PATCH][RFC] add MAP_UNLOCKED mmap flag

Gleb Natapov

author

2009-10-06 09:51 UTC #0 lore raw

RFC PATCH 1/1 add MAP_UNLOCKED mmap flag

If application does mlockall(MCL_FUTURE) it is no longer possible to mmap file bigger than main memory or allocate big area of anonymous memory. Sometimes it is desirable to lock everything related to program execution into memory, but still be able to mmap big file or allocate huge amount of memory and allow OS to swap them on demand. MAP_UNLOCKED allows to do that.

Signed-off-by Gleb Natapov <[email protected]>

include/asm-generic/mman.h +1 -0

index 32c8bd6..0ab4c74 100644
@@ -12,6 +12,7 @@
 #define MAP_NONBLOCK	0x10000		/* do not block on IO */
 #define MAP_STACK	0x20000		/* give out an address that is best suited for process/thread stacks */
 #define MAP_HUGETLB	0x40000		/* create a huge page mapping */
+#define MAP_UNLOKED	0x80000         /* pages are unlocked */
 
 #define MCL_CURRENT	1		/* lock all current mappings */
 #define MCL_FUTURE	2		/* lock all future mappings */

mm/mmap.c +3 -0

index 73f5e4b..7c2abdb 100644
@@ -985,6 +985,9 @@ unsigned long do_mmap_pgoff(struct file *file, unsigned long addr,
 		if (!can_do_mlock())
 			return -EPERM;
 
+        if (flags & MAP_UNLOKED)
+                vm_flags &= ~VM_LOCKED;
+
 	/* mlock MCL_FUTURE? */
 	if (vm_flags & VM_LOCKED) {
 		unsigned long locked, lock_limit;

--
Gleb.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Frederik Deweerdt

2009-10-06 10:09 UTC #1 lore raw

Hello Gleb,

Quoted 13 lines · Gleb Natapov

If application does mlockall(MCL_FUTURE) it is no longer possible to mmap file bigger than main memory or allocate big area of anonymous memory. Sometimes it is desirable to lock everything related to program execution into memory, but still be able to mmap big file or allocate huge amount of memory and allow OS to swap them on demand. MAP_UNLOCKED allows to do that.
Signed-off-by Gleb Natapov <[email protected]>
include/asm-generic/mman.h +0 -0
index 32c8bd6..0ab4c74 100644

include/asm-generic/mman.h line 15

Quoted 4 lines

include/asm-generic/mman.h +1 -0

@@ -12,6 +12,7 @@
 #define MAP_NONBLOCK	0x10000		/* do not block on IO */
 #define MAP_STACK	0x20000		/* give out an address that is best suited for process/thread stacks */
 #define MAP_HUGETLB	0x40000		/* create a huge page mapping */
+#define MAP_UNLOKED	0x80000         /* pages are unlocked */

^^^

You're missing a 'C' here and below. Also '/* force page unlocking */' seems a better comment?

Regards,
Frederik

Quoted 18 lines

include/asm-generic/mman.h +0 -0

 #define MCL_CURRENT	1		/* lock all current mappings */
 #define MCL_FUTURE	2		/* lock all future mappings */

mm/mmap.c +3 -1

index 73f5e4b..7c2abdb 100644
@@ -985,6 +985,9 @@ unsigned long do_mmap_pgoff(struct file *file, unsigned long addr,
 		if (!can_do_mlock())
 			return -EPERM;
 
+        if (flags & MAP_UNLOKED)
+                vm_flags &= ~VM_LOCKED;
+
 	/* mlock MCL_FUTURE? */
 	if (vm_flags & VM_LOCKED) {
 		unsigned long locked, lock_limit;
--

Quoted 6 lines

Gleb.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

KOSAKI Motohiro

2009-10-06 10:11 UTC #2 lore raw

In reply to Gleb Natapov

Hi

Quoted 8 lines

If application does mlockall(MCL_FUTURE) it is no longer possible to mmap file bigger than main memory or allocate big area of anonymous memory. Sometimes it is desirable to lock everything related to program execution into memory, but still be able to mmap big file or allocate huge amount of memory and allow OS to swap them on demand. MAP_UNLOCKED allows to do that.
Signed-off-by Gleb Natapov <[email protected]>

Why don't you use explicit munlock()?
Plus, Can you please elabrate which workload nedd this feature?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Gleb Natapov

author

2009-10-06 10:21 UTC #3 lore raw

Quoted 9 lines · KOSAKI Motohiro

Hi
Quoted 7 lines
If application does mlockall(MCL_FUTURE) it is no longer possible to mmap file bigger than main memory or allocate big area of anonymous memory. Sometimes it is desirable to lock everything related to program execution into memory, but still be able to mmap big file or allocate huge amount of memory and allow OS to swap them on demand. MAP_UNLOCKED allows to do that.

Quoted 3 lines

Quoted 1 line
Signed-off-by Gleb Natapov <[email protected]>
Why don't you use explicit munlock()?

Because mmap will fail before I'll have a chance to run munlock on it. Actually when I run my process inside memory limited container host dies (I suppose trashing, but haven't checked).

Quoted 2 lines

Plus, Can you please elabrate which workload nedd this feature?

I wanted to run kvm with qemu process locked in memory, but guest memory unlocked. And guest memory is bigger then host memory in the case I am testing. I found out that it is impossible currently.

--
Gleb.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

KOSAKI Motohiro

2009-10-06 10:27 UTC #4 lore raw

Quoted 17 lines · KOSAKI Motohiro

Quoted 9 lines
Hi
Quoted 7 lines
If application does mlockall(MCL_FUTURE) it is no longer possible to mmap file bigger than main memory or allocate big area of anonymous memory. Sometimes it is desirable to lock everything related to program execution into memory, but still be able to mmap big file or allocate huge amount of memory and allow OS to swap them on demand. MAP_UNLOCKED allows to do that.
Quoted 3 lines
Quoted 1 line
Signed-off-by Gleb Natapov <[email protected]>
Why don't you use explicit munlock()?
Because mmap will fail before I'll have a chance to run munlock on it. Actually when I run my process inside memory limited container host dies (I suppose trashing, but haven't checked).

Quoted 5 lines

Quoted 2 lines
Plus, Can you please elabrate which workload nedd this feature?
I wanted to run kvm with qemu process locked in memory, but guest memory unlocked. And guest memory is bigger then host memory in the case I am testing. I found out that it is impossible currently.

1. process creation (qemu)
2. load all library
3. mlockall(MCL_CURRENT)
4. load guest OS

is impossible? why?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Gleb Natapov

author

2009-10-06 10:33 UTC #5 lore raw

Quoted 17 lines · KOSAKI Motohiro

Quoted 17 lines · KOSAKI Motohiro
Quoted 9 lines
Hi
Quoted 7 lines
If application does mlockall(MCL_FUTURE) it is no longer possible to mmap file bigger than main memory or allocate big area of anonymous memory. Sometimes it is desirable to lock everything related to program execution into memory, but still be able to mmap big file or allocate huge amount of memory and allow OS to swap them on demand. MAP_UNLOCKED allows to do that.
Quoted 3 lines
Quoted 1 line
Signed-off-by Gleb Natapov <[email protected]>
Why don't you use explicit munlock()?
Because mmap will fail before I'll have a chance to run munlock on it. Actually when I run my process inside memory limited container host dies (I suppose trashing, but haven't checked).

Quoted 8 lines

Quoted 5 lines
Quoted 2 lines
Plus, Can you please elabrate which workload nedd this feature?
I wanted to run kvm with qemu process locked in memory, but guest memory unlocked. And guest memory is bigger then host memory in the case I am testing. I found out that it is impossible currently.
1. process creation (qemu)
2. load all library

Can't control this if program has plugging. Not qemu case though.

Quoted 2 lines

3. mlockall(MCL_CURRENT)
4. load guest OS

And what about all other allocations qemu does during its life time? Not all of them will be small enough to be from brk area.

Quoted 3 lines

is impossible? why?

Because what you are proposing is not the same as mlockall(MCL_CURRENT|MCL_FUTURE); You essentially say that MCL_FUTURE is not needed.

--
Gleb.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

KOSAKI Motohiro

2009-10-06 12:10 UTC #6 lore raw

2009/10/6 Gleb Natapov <[email protected]>:

Quoted 31 lines · KOSAKI Motohiro

Quoted 17 lines
Quoted 17 lines · KOSAKI Motohiro
Quoted 9 lines
Hi
Quoted 7 lines
If application does mlockall(MCL_FUTURE) it is no longer possible to mmap file bigger than main memory or allocate big area of anonymous memory. Sometimes it is desirable to lock everything related to program execution into memory, but still be able to mmap big file or allocate huge amount of memory and allow OS to swap them on demand. MAP_UNLOCKED allows to do that.
Quoted 3 lines
Quoted 1 line
Signed-off-by Gleb Natapov <[email protected]>
Why don't you use explicit munlock()?
Because mmap will fail before I'll have a chance to run munlock on it. Actually when I run my process inside memory limited container host dies (I suppose trashing, but haven't checked).
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Plus, Can you please elabrate which workload nedd this feature?
I wanted to run kvm with qemu process locked in memory, but guest memory unlocked. And guest memory is bigger then host memory in the case I am testing. I found out that it is impossible currently.
1. process creation (qemu)
2. load all library
Can't control this if program has plugging. Not qemu case though.
Quoted 2 lines
3. mlockall(MCL_CURRENT)
4. load guest OS

Quoted 8 lines

And what about all other allocations qemu does during its life time? Not all of them will be small enough to be from brk area.
Quoted 2 lines
is impossible? why?
Because what you are proposing is not the same as mlockall(MCL_CURRENT|MCL_FUTURE);
You essentially say that MCL_FUTURE is not needed.

No, I only think your case doesn't fit MC_FUTURE.
I haven't find any real benefit in this patch.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Gleb Natapov

author

2009-10-06 12:16 UTC #7 lore raw

Quoted 35 lines · KOSAKI Motohiro

2009/10/6 Gleb Natapov <[email protected]>:
Quoted 34 lines · KOSAKI Motohiro
Quoted 17 lines
Quoted 17 lines · KOSAKI Motohiro
Quoted 9 lines
Hi
Quoted 7 lines
If application does mlockall(MCL_FUTURE) it is no longer possible to mmap file bigger than main memory or allocate big area of anonymous memory. Sometimes it is desirable to lock everything related to program execution into memory, but still be able to mmap big file or allocate huge amount of memory and allow OS to swap them on demand. MAP_UNLOCKED allows to do that.
Quoted 3 lines
Quoted 1 line
Signed-off-by Gleb Natapov <[email protected]>
Why don't you use explicit munlock()?
Because mmap will fail before I'll have a chance to run munlock on it. Actually when I run my process inside memory limited container host dies (I suppose trashing, but haven't checked).
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Plus, Can you please elabrate which workload nedd this feature?
I wanted to run kvm with qemu process locked in memory, but guest memory unlocked. And guest memory is bigger then host memory in the case I am testing. I found out that it is impossible currently.
1. process creation (qemu)
2. load all library
Can't control this if program has plugging. Not qemu case though.
Quoted 2 lines
3. mlockall(MCL_CURRENT)
4. load guest OS
And what about all other allocations qemu does during its life time? Not all of them will be small enough to be from brk area.

Quoted 8 lines

Quoted 5 lines
Quoted 2 lines
is impossible? why?
Because what you are proposing is not the same as mlockall(MCL_CURRENT|MCL_FUTURE);
You essentially say that MCL_FUTURE is not needed.
No, I only think your case doesn't fit MC_FUTURE.
I haven't find any real benefit in this patch.

I did. It allows me to achieve something I can't now. Steps you provide just don't fit my needs. I need all memory areas (current and feature) to be locked except one. Very big one. You propose to lock memory at some arbitrary point and from that point on all newly mapped memory areas will be unlocked. Don't you see it is different?

--
Gleb.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Peter Zijlstra

2009-10-06 13:49 UTC #8 lore raw

Quoted 8 lines · Gleb Natapov

Quoted 2 lines
No, I only think your case doesn't fit MC_FUTURE.
I haven't find any real benefit in this patch.
I did. It allows me to achieve something I can't now. Steps you provide just don't fit my needs. I need all memory areas (current and feature) to be locked except one. Very big one. You propose to lock memory at some arbitrary point and from that point on all newly mapped memory areas will be unlocked. Don't you see it is different?

While true, it does demonstrates very sloppy programming. The proper fix is to rework qemu to mlock what is needed.

I'm not sure encouraging mlockall() usage is a good thing. When using resource locks one had better know what he's doing. mlockall() doesn't promote caution.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Gleb Natapov

author

2009-10-06 14:06 UTC #9 lore raw

Quoted 4 lines · Peter Zijlstra

Quoted 2 lines · Gleb Natapov
Quoted 2 lines
No, I only think your case doesn't fit MC_FUTURE.
I haven't find any real benefit in this patch.

Quoted 9 lines

Quoted 5 lines
I did. It allows me to achieve something I can't now. Steps you provide just don't fit my needs. I need all memory areas (current and feature) to be locked except one. Very big one. You propose to lock memory at some arbitrary point and from that point on all newly mapped memory areas will be unlocked. Don't you see it is different?
While true, it does demonstrates very sloppy programming. The proper fix is to rework qemu to mlock what is needed.

So you are saying for application (any application forget about qemu) to lock everything except one memory region it needs to provide its own memory allocation routings and its own dynamic linker? BTW the interface is not symmetric currently. Application may mmap single memory area locked (MAP_LOCKED), but can't do reverse if mlockall(MC_FUTURE) was called.

Quoted 1 line

I'm not sure encouraging mlockall() usage is a good thing. When using

This is up to application programmer to decide whether he wants to use mlockall() or not. May be he has a good reason do so. As it stands the existing interface doesn't allow to do what I need without rewriting libc memory allocator and dynamic linking loader.

Quoted 2 lines

resource locks one had better know what he's doing. mlockall() doesn't promote caution.

No need to patronize userspace developers. Lets provide them with flexible interface and if they'll use it inappropriately we will not use their software.

--
Gleb.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Olivier Galibert

2009-10-07 18:50 UTC #10 lore raw

In reply to Gleb Natapov

Quoted 5 lines · Gleb Natapov

I did. It allows me to achieve something I can't now. Steps you provide just don't fit my needs. I need all memory areas (current and feature) to be locked except one. Very big one. You propose to lock memory at some arbitrary point and from that point on all newly mapped memory areas will be unlocked. Don't you see it is different?

What about mlockall(MCL_CURRENT); mmap(...); mlockall(MCL_FUTURE);? Or toggle MCL_FUTURE if a mlockall call can stop it?

OG.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Gleb Natapov

author

2009-10-07 18:59 UTC #11 lore raw

Quoted 10 lines · Olivier Galibert

Quoted 5 lines · Gleb Natapov
I did. It allows me to achieve something I can't now. Steps you provide just don't fit my needs. I need all memory areas (current and feature) to be locked except one. Very big one. You propose to lock memory at some arbitrary point and from that point on all newly mapped memory areas will be unlocked. Don't you see it is different?
What about mlockall(MCL_CURRENT); mmap(...); mlockall(MCL_FUTURE);? Or toggle MCL_FUTURE if a mlockall call can stop it?

This may work. And MCL_FUTURE can be toggled, but this is not thread safe.

--
Gleb.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Olivier Galibert

2009-10-07 20:10 UTC #12 lore raw

Quoted 8 lines · Gleb Natapov

Quoted 7 lines · Olivier Galibert
Quoted 5 lines · Gleb Natapov
I did. It allows me to achieve something I can't now. Steps you provide just don't fit my needs. I need all memory areas (current and feature) to be locked except one. Very big one. You propose to lock memory at some arbitrary point and from that point on all newly mapped memory areas will be unlocked. Don't you see it is different?

Quoted 5 lines

Quoted 3 lines
What about mlockall(MCL_CURRENT); mmap(...); mlockall(MCL_FUTURE);? Or toggle MCL_FUTURE if a mlockall call can stop it?
This may work. And MCL_FUTURE can be toggled, but this is not thread safe.

Just ensure that your one special mmap is done with the other threads not currently allocating stuff. It's probably a synchronization point for the whole process anyway.

OG.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Gleb Natapov

author

2009-10-07 20:47 UTC #13 lore raw

Quoted 12 lines · Olivier Galibert

Quoted 11 lines · Gleb Natapov
Quoted 10 lines · Olivier Galibert
Quoted 5 lines · Gleb Natapov
I did. It allows me to achieve something I can't now. Steps you provide just don't fit my needs. I need all memory areas (current and feature) to be locked except one. Very big one. You propose to lock memory at some arbitrary point and from that point on all newly mapped memory areas will be unlocked. Don't you see it is different?
What about mlockall(MCL_CURRENT); mmap(...); mlockall(MCL_FUTURE);? Or toggle MCL_FUTURE if a mlockall call can stop it?

Quoted 7 lines

Quoted 2 lines
This may work. And MCL_FUTURE can be toggled, but this is not thread safe.
Just ensure that your one special mmap is done with the other threads not currently allocating stuff. It's probably a synchronization point for the whole process anyway.

How can you stop other threads and libraries from calling malloc()? And if it is two special allocations? Or many mmap(big file)/munmap(big file)? This is the same issue as opening file CLOEXEC atomically. Why not prevent other thread from calling fork() instead of adding flags to bunch of system calls.

--
Gleb.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Arnd Bergmann

maintainer

2009-10-06 10:49 UTC #14 lore raw

In reply to Gleb Natapov

Quoted 13 lines · Gleb Natapov

If application does mlockall(MCL_FUTURE) it is no longer possible to mmap file bigger than main memory or allocate big area of anonymous memory. Sometimes it is desirable to lock everything related to program execution into memory, but still be able to mmap big file or allocate huge amount of memory and allow OS to swap them on demand. MAP_UNLOCKED allows to do that.
Signed-off-by Gleb Natapov <[email protected]>
include/asm-generic/mman.h +0 -0
index 32c8bd6..0ab4c74 100644

include/asm-generic/mman.h line 18

Quoted 7 lines

include/asm-generic/mman.h +1 -0

@@ -12,6 +12,7 @@
 #define MAP_NONBLOCK   0x10000         /* do not block on IO */
 #define MAP_STACK      0x20000         /* give out an address that is best suited for process/thread stacks */
 #define MAP_HUGETLB    0x40000         /* create a huge page mapping */
+#define MAP_UNLOKED    0x80000         /* pages are unlocked */
 
 #define MCL_CURRENT    1               /* lock all current mappings */
 #define MCL_FUTURE     2               /* lock all future mappings */

Not all architectures use asm-generic/mman.h, so you have to change the other architectures separately if you add a flag.

	Arnd <><

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Gleb Natapov

author

2009-10-06 11:00 UTC #15 lore raw

Quoted 19 lines · Arnd Bergmann

Quoted 18 lines · Gleb Natapov
If application does mlockall(MCL_FUTURE) it is no longer possible to mmap file bigger than main memory or allocate big area of anonymous memory. Sometimes it is desirable to lock everything related to program execution into memory, but still be able to mmap big file or allocate huge amount of memory and allow OS to swap them on demand. MAP_UNLOCKED allows to do that.
Signed-off-by Gleb Natapov <[email protected]>
include/asm-generic/mman.h +1 -0
index 32c8bd6..0ab4c74 100644
@@ -12,6 +12,7 @@
 #define MAP_NONBLOCK   0x10000         /* do not block on IO */
 #define MAP_STACK      0x20000         /* give out an address that is best suited for process/thread stacks */
 #define MAP_HUGETLB    0x40000         /* create a huge page mapping */
+#define MAP_UNLOKED    0x80000         /* pages are unlocked */
 

Quoted 6 lines

include/asm-generic/mman.h line 18
Quoted 2 lines
include/asm-generic/mman.h +0 -0
 #define MCL_CURRENT    1               /* lock all current mappings */
 #define MCL_FUTURE     2               /* lock all future mappings */
Not all architectures use asm-generic/mman.h, so you have to change the other architectures separately if you add a flag.

Ah, good to know. Thanks.

--
Gleb.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>