add MAP_UNLOCKED mmap flag

The current interaction between mlockall(MCL_FUTURE) and mmap has a deficiency. In 'normal' mode, without MCL_FUTURE in force, the default is that new memory mappings are not locked, but mmap provides MAP_LOCKED specifically to override that default. However, with MCL_FUTURE toggled to on, there is no analogous way to tell mmap to override the default. The proposed MAP_UNLOCKED flag would resolve this deficiency.

The benefit of the patch is that it makes it possible for an application which has previously called mlockall(MCL_FUTURE) to selectively exempt new memory mappings from memory locking, on a per-mmap-call basis. There is currently no thread-safe way for an application to do this as toggling MCL_FUTURE around calls to mmap is racy in a multi-threaded context. Other threads may manipulate the address space during the window where MCL_FUTURE is off, subverting the programmers intended memory locking semantics.

The ability to exempt specific memory mappings from memory locking is necessary when the region to be mapped is larger than physical memory. In such cases a call to mmap the region cannot succeed, unless MAP_UNLOCKED is available.

Hi Gleb,

Quoted 16 lines · Gleb Natapov

The current interaction between mlockall(MCL_FUTURE) and mmap has a deficiency. In 'normal' mode, without MCL_FUTURE in force, the default is that new memory mappings are not locked, but mmap provides MAP_LOCKED specifically to override that default. However, with MCL_FUTURE toggled to on, there is no analogous way to tell mmap to override the default. The proposed MAP_UNLOCKED flag would resolve this deficiency.

The benefit of the patch is that it makes it possible for an application which has previously called mlockall(MCL_FUTURE) to selectively exempt new memory mappings from memory locking, on a per-mmap-call basis. There is currently no thread-safe way for an application to do this as toggling MCL_FUTURE around calls to mmap is racy in a multi-threaded context. Other threads may manipulate the address space during the window where MCL_FUTURE is off, subverting the programmers intended memory locking semantics.

Quoted 4 lines

The ability to exempt specific memory mappings from memory locking is necessary when the region to be mapped is larger than physical memory. In such cases a call to mmap the region cannot succeed, unless MAP_UNLOCKED is available.

The changelog doesn't mention what kind of applications would want to use this. Are there some? Using mlockall(MCL_FUTURE) but then having some memory regions MAP_UNLOCKED sounds like a strange combination to me.

this kind of control. As of use of mlockall(MCL_FUTURE) how can I make sure that all memory allocated behind my application's back (by dynamic linker, libraries, stack) will be locked otherwise?

Quoted 3 lines

this kind of control. As of use of mlockall(MCL_FUTURE) how can I make sure that all memory allocated behind my application's back (by dynamic linker, libraries, stack) will be locked otherwise?

If you add this flag you can't do that anyway - some library will helpfully start up using it and then you are completely stuffed or will be back in two or three years adding MLOCKALL_ALWAYS.

Quoted 3 lines

this kind of control. As of use of mlockall(MCL_FUTURE) how can I make sure that all memory allocated behind my application's back (by dynamic linker, libraries, stack) will be locked otherwise?

If you add this flag you can't do that anyway - some library will helpfully start up using it and then you are completely stuffed or will be back in two or three years adding MLOCKALL_ALWAYS.

Quoted 4 lines · Alan Cox

Quoted 3 lines

this kind of control. As of use of mlockall(MCL_FUTURE) how can I make sure that all memory allocated behind my application's back (by dynamic linker, libraries, stack) will be locked otherwise?

Quoted 3 lines

If you add this flag you can't do that anyway - some library will helpfully start up using it and then you are completely stuffed or will be back in two or three years adding MLOCKALL_ALWAYS.

Agreed, mlockall() is a very bad interface and should not be used for a plethora of reasons, this being one of them.

The thing is, if you cant trust your library to do sane things, then don't use it.

There are valid uses for mlockall()

Quoted 2 lines · Gleb Natapov

There are valid uses for mlockall()

That's debatable.

Quoted 1 line · Gleb Natapov

There are valid uses for mlockall()

That's debatable.

Quoted 7 lines · Peter Zijlstra

Quoted 2 lines · Gleb Natapov

There are valid uses for mlockall()

That's debatable.

Real-time?

Quoted 8 lines · Avi Kivity

Quoted 6 lines · Peter Zijlstra

Quoted 3 lines · Gleb Natapov

There are valid uses for mlockall()

Quoted 6 lines

Quoted 4 lines

That's debatable.

Real-time?

I would not advice that, just mlock() the text and data you need for the real-time thread. mlockall() is a really blunt instrument.

Quoted 3 lines

I would not advice that, just mlock() the text and data you need for the real-time thread. mlockall() is a really blunt instrument.

May not be feasible due to libraries.

Quoted 5 lines · Avi Kivity

Quoted 3 lines

I would not advice that, just mlock() the text and data you need for the real-time thread. mlockall() is a really blunt instrument.

May not be feasible due to libraries.

Esp for the real-time case I could advise not to use those libraries then, since they're clearly not designed for that use case.

Quoted 9 lines · On Mon, 18 Jan 2010 16:24:07 +0100 Peter Zijlstra <[email protected]> wrote:

Quoted 5 lines · Avi Kivity

Quoted 3 lines

I would not advice that, just mlock() the text and data you need for the real-time thread. mlockall() is a really blunt instrument.

May not be feasible due to libraries.

Esp for the real-time case I could advise not to use those libraries then, since they're clearly not designed for that use case.

In "hard" real time cases an awful lot of libraries have things like memory allocations in them and don't care about stack growth which can cause faults and sleeps. The memory allocator if you are running threaded was not real time priority aware either last time I checked so the standard libraries are not going to give the behaviour you want unless you have a proper RT environment, and even then it may be a bit iffy here and there.

The specific use cases were discussed in the thread following previous version of the patch. I can describe my specific use case in a change log and I can copy what Andrew said about his case, but is it really needed in a commit message itself? It boils down to greater control over when and where application can get major fault. There are applications that need this kind of control. As of use of mlockall(MCL_FUTURE) how can I make sure that all memory allocated behind my application's back (by dynamic linker, libraries, stack) will be locked otherwise?

Quoted 8 lines · Gleb Natapov

The specific use cases were discussed in the thread following previous version of the patch. I can describe my specific use case in a change log and I can copy what Andrew said about his case, but is it really needed in a commit message itself? It boils down to greater control over when and where application can get major fault. There are applications that need this kind of control. As of use of mlockall(MCL_FUTURE) how can I make sure that all memory allocated behind my application's back (by dynamic linker, libraries, stack) will be locked otherwise?

Again, why do you want to MCL_FUTURE but then go and use MAP_UNLOCKED?

"Greater control" is not an argument for adding a new API that needs to be maintained forever, a real world use case is.

And yes, this stuff needs to be in the changelog. Whether you want to spell it out or post an URL to some previous discussion is up to you.

Quoted 5 lines · Avi Kivity

Quoted 4 lines · Peter Zijlstra

Quoted 2 lines · Gleb Natapov

There are valid uses for mlockall()

Quoted 5 lines

Quoted 3 lines

That's debatable.

Real-time?

I would not advice that, just mlock() the text and data you need for the real-time thread. mlockall() is a really blunt instrument.

Quoted 3 lines

"Greater control" is not an argument for adding a new API that needs to be maintained forever, a real world use case is.

If there is real world use case for mlockall() there is real use case for this too. People seems to be trying to convince me that I don't need mlockall() without proposing alternatives. The only alternative I see lock everything from userspace.

Quoted 2 lines

And yes, this stuff needs to be in the changelog. Whether you want to spell it out or post an URL to some previous discussion is up to you.

The discussion was here just a couple of days ago. Here is the link were I describe my use case: http://marc.info/?l=linux-mm&m=126345374125942&w=2
If you think it needs to be spelled out in commit log I'll do it.

Hi Gleb,

Quoted 8 lines · Gleb Natapov

Quoted 3 lines

"Greater control" is not an argument for adding a new API that needs to be maintained forever, a real world use case is.

If there is real world use case for mlockall() there is real use case for this too. People seems to be trying to convince me that I don't need mlockall() without proposing alternatives. The only alternative I see lock everything from userspace.

Quoted 5 lines

Quoted 2 lines

And yes, this stuff needs to be in the changelog. Whether you want to spell it out or post an URL to some previous discussion is up to you.

The discussion was here just a couple of days ago. Here is the link were I describe my use case: http://marc.info/?l=linux-mm&m=126345374125942&w=2
If you think it needs to be spelled out in commit log I'll do it.

So this is a performance thing? Btw, is there are reason you can't use plain mlock() for it as suggested by Peter earlier?

I can't realistically chase every address space mapping changes and mlock new areas. The only way other then mlockall() is to use custom memory allocator that allocates mlocked memory.

Quoted 5 lines · Gleb Natapov

Quoted 1 line

There are valid uses for mlockall()

That's debatable.

Well, I have use for it. You can look at previous thread were I described it and suggest alternatives.

Quoted 3 lines

I can't realistically chase every address space mapping changes and mlock new areas. The only way other then mlockall() is to use custom memory allocator that allocates mlocked memory.

Which keeps all the special cases in your app rather than in every single users kernel. That seems to be the right way up, especially as you can make a library of it !

The thread took a direction of bashing mlockall(). This is especially strange since proposed patch actually makes mlockall() more fine grained and thus more useful.

Hi Gleb,

Quoted 3 lines · Gleb Natapov

The thread took a direction of bashing mlockall(). This is especially strange since proposed patch actually makes mlockall() more fine grained and thus more useful.

No, the thread took a direction of you not being able to properly explain why we want MMAP_UNLOCKED in the kernel. It seems useless for

real-time and I've yet to figure out why you need _mlockall()_ if it's a performance thing.

It would be probably useful if you could point us to the application source code that actually wants this feature.

Quoted 3 lines

It would be probably useful if you could point us to the application source code that actually wants this feature.

This is two line patch to qemu that calls mlockall(MCL_CURRENT|MCL_FUTURE) at the beginning of the main() and changes guest memory allocation to use MAP_UNLOCKED flag. All alternative solutions in this thread suggest that I should rewrite qemu + all library it uses. You see why I can't take them seriously?

Hi Gleb,

Quoted 3 lines · Gleb Natapov

Quoted 3 lines

It would be probably useful if you could point us to the application source code that actually wants this feature.

Quoted 5 lines

This is two line patch to qemu that calls mlockall(MCL_CURRENT|MCL_FUTURE) at the beginning of the main() and changes guest memory allocation to use MAP_UNLOCKED flag. All alternative solutions in this thread suggest that I should rewrite qemu + all library it uses. You see why I can't take them seriously?

Well, that's not going to be portable, is it, so the application

design would still be broken, no? Did you try using (or extending) posix_madvise(MADV_DONTNEED) for the guest address space? It seems to

me that you're trying to use a big hammer (mlock) when a polite hint for the VM would probably be sufficient for it do its job.

Quoted 2 lines

design would still be broken, no? Did you try using (or extending) posix_madvise(MADV_DONTNEED) for the guest address space? It seems to

After mlockall() I can't even allocate guest address space. Or do you mean instead of mlockall()? Then how MADV_DONTNEED will help? It just drops page table for the address range (which is not what I need) and does not have any long time effect.

Quoted 3 lines

me that you're trying to use a big hammer (mlock) when a polite hint for the VM would probably be sufficient for it do its job.

I what to tell to VM "swap this, don't swap that" and as far as I see there is no other way to do it currently.

Quoted 5 lines · Gleb Natapov

Quoted 3 lines

me that you're trying to use a big hammer (mlock) when a polite hint for the VM would probably be sufficient for it do its job.

I what to tell to VM "swap this, don't swap that" and as far as I see there is no other way to do it currently.

Yeah, which is why I was suggesting that maybe posix_madvise() needs to be extended to have a MADV_NEED_BUT_LESS_IMPORTANT flag that can be used as a hint by mm/vmscan.c to first swap the guest address spaces.

In my case (virtualization) I want to test/profile guest under heavy swapping of a guests memory, so I intentionally create memory shortage by creating guest much large then host memory, but I want system to swap out only guest's memory.

Quoted 5 lines

It would be probably useful if you could point us to the application source code that actually wants this feature.

This is two line patch to qemu that calls mlockall(MCL_CURRENT|MCL_FUTURE) at the beginning of the main() and changes guest memory allocation to use MAP_UNLOCKED flag. All alternative solutions in this thread suggest that I should rewrite qemu + all library it uses. You see why I can't take them seriously?

Quoted 4 lines

In my case (virtualization) I want to test/profile guest under heavy swapping of a guests memory, so I intentionally create memory shortage by creating guest much large then host memory, but I want system to swap out only guest's memory.

So this isn't an API question this is an obscure corner case testing question.

Quoted 4 lines

Quoted 4 lines

It would be probably useful if you could point us to the application source code that actually wants this feature.

Quoted 5 lines

This is two line patch to qemu that calls mlockall(MCL_CURRENT|MCL_FUTURE) at the beginning of the main() and changes guest memory allocation to use MAP_UNLOCKED flag. All alternative solutions in this thread suggest that I should rewrite qemu + all library it uses. You see why I can't take them seriously?

And you want millions of users to have kernels with weird extra functions whole sole value is one test environment you wish to run

See why we can't take you seriously either ?

Hi Gleb,

Quoted 6 lines · Gleb Natapov

Quoted 2 lines

design would still be broken, no? Did you try using (or extending) posix_madvise(MADV_DONTNEED) for the guest address space? It seems to

After mlockall() I can't even allocate guest address space. Or do you mean instead of mlockall()? Then how MADV_DONTNEED will help? It just drops page table for the address range (which is not what I need) and does not have any long time effect.

Oh right, MADV_DONTNEED is no good.

Quoted 3 lines · Gleb Natapov

Quoted 3 lines

me that you're trying to use a big hammer (mlock) when a polite hint for the VM would probably be sufficient for it do its job.

Quoted 2 lines

I what to tell to VM "swap this, don't swap that" and as far as I see there is no other way to do it currently.

Yeah, which is why I was suggesting that maybe posix_madvise() needs to be extended to have a MADV_NEED_BUT_LESS_IMPORTANT flag that can be used as a hint by mm/vmscan.c to first swap the guest address spaces.

Pekka

Gleb. How about using MADV_SEQUENTIAL on guest memory?
It makes that pages of guest are moved into inactive reclaim list more fast. It means it is likely to swap out faster than other pages if it isn't hit during inactive list.

Quoted 3 lines

And you want millions of users to have kernels with weird extra functions whole sole value is one test environment you wish to run

We are talking about 4 lines of code that other people find useful too and they commented in this thread. This wouldn't be the first kernel feature not used by millions of people.

Quoted 4 lines · Minchan Kim

Gleb. How about using MADV_SEQUENTIAL on guest memory?
It makes that pages of guest are moved into inactive reclaim list more fast. It means it is likely to swap out faster than other pages if it isn't hit during inactive list.

Yeah, something like that but we don't want the readahead. OTOH, it's not clear what Gleb's real problem is. Are the guest address spaces anonymous or file backed?

Which parts of the emulator are swapped out
that are causing the problem?

Maybe it's a VM balancing issue that
mlock papers over?

In my case (virtualization) I want to test/profile guest under heavy swapping of a guests memory, so I intentionally create memory shortage by creating

guest much large then host memory, but I want system to swap out only guest's memory.

Quoted 2 lines · Gleb Natapov

In my case (virtualization) I want to test/profile guest under heavy swapping of a guests memory, so I intentionally create memory shortage by creating

You mean "guest memory" that is area emulated DRAM in qemu? It is anonymous vma.

Quoted 2 lines

guest much large then host memory, but I want system to swap out only guest's memory.

Couldn't you use MADV_SEQUENTIAL on only guest memory area? It doesn't make side effect about readahead since it's anon area. And it would make do best effort to swap out guest's memory.

Hi Gleb,

Quoted 6 lines · Gleb Natapov

Quoted 2 lines

design would still be broken, no? Did you try using (or extending) posix_madvise(MADV_DONTNEED) for the guest address space? It seems to

After mlockall() I can't even allocate guest address space. Or do you mean instead of mlockall()? Then how MADV_DONTNEED will help? It just drops page table for the address range (which is not what I need) and does not have any long time effect.

Oh right, MADV_DONTNEED is no good.