Some highmem pages still in use after shrink_all_memory()?
Hi!
For swsusp, I need to free as much memory as possible. Well, and it would be great if no highmem pages remained, so that I would not have to deal with that. Is that possible?
Pavel
Signature · 2 lines
When do you have a heart between your knees? [Johanka's followup: and *two* hearts?]
Quoted 6 lines · Pavel Machek
Hi!
For swsusp, I need to free as much memory as possible. Well, and it would be great if no highmem pages remained, so that I would not have to deal with that. Is that possible?
No, it isn't. There are pagetable pages and mlocked user pages which we cannot do anything with.
We could perhaps swap out the mlocked pages anyway if a suspend is in progress, but the highmem pagetable pages are not presently reclaimed by the VM.
Quoted 11 lines · Andrew Morton
Quoted 3 lines · Pavel Machek
For swsusp, I need to free as much memory as possible. Well, and it would be great if no highmem pages remained, so that I would not have to deal with that. Is that possible?
No, it isn't. There are pagetable pages and mlocked user pages which we cannot do anything with.
We could perhaps swap out the mlocked pages anyway if a suspend is in progress, but the highmem pagetable pages are not presently reclaimed by the VM.
Note that there are some applications for which it is a *bug* if an mlocked page gets written out to magnetic media. (gpg, for example.) I imagine that they'd rather lose the mapping and get a page fault on the next reference (which they can then fix up with a new mmap and mlock) than have precious key material written to disk.
... unless, of course, the swap device is securely encrypted a la OpenBSD's 'sysctl vm.swapencrypt.enable'.
http://www.openbsd.org/papers/swapencrypt.ps
However, I don't see how to implement a cryptographically secure swsusp.
(The importance of this behavior is obviously dependent on your threat model. Perhaps the Sufficiently Paranoid gpg users will simply need to avoid using swsusp.)
-andy
Hi.
Quoted 5 lines · Andy Isaacson
Note that there are some applications for which it is a *bug* if an mlocked page gets written out to magnetic media. (gpg, for example.) I imagine that they'd rather lose the mapping and get a page fault on the next reference (which they can then fix up with a new mmap and mlock) than have precious key material written to disk.
For such an application, we'd have to provide a mechanism to allow an application to set/clear a page's Nosave flag. We'd probably also want to be able to notify user space that a suspend cycle has just occurred and the page contents are invalid.
Quoted 1 line
However, I don't see how to implement a cryptographically secure swsusp.
It would be possible with Suspend2 - one could implement a backend (page transformer or writer) that implemented encryption and required the user to enter a passphrase at resume time.
Quoted 3 lines
(The importance of this behavior is obviously dependent on your threat model. Perhaps the Sufficiently Paranoid gpg users will simply need to avoid using swsusp.)
Yes. Or close all gpg apps before suspending?
Nigel
Signature · 7 lines
Nigel Cunningham C/- Westminster Presbyterian Church Belconnen 61 Templeton Street, Cook, ACT 2614. +61 (2) 6251 7727(wk); +61 (2) 6253 0250 (home) Evolution (n): A hypothetical process whereby infinitely improbable events occur with alarming frequency, order arises from chaos, and no one is given credit.
Quoted 9 lines · Andy Isaacson
Quoted 8 lines · Andrew Morton
Quoted 3 lines · Pavel Machek
For swsusp, I need to free as much memory as possible. Well, and it would be great if no highmem pages remained, so that I would not have to deal with that. Is that possible?
No, it isn't. There are pagetable pages and mlocked user pages which we cannot do anything with.
Quoted 6 lines
Quoted 3 lines
We could perhaps swap out the mlocked pages anyway if a suspend is in progress, but the highmem pagetable pages are not presently reclaimed by the VM.
Note that there are some applications for which it is a *bug* if an mlocked page gets written out to magnetic media. (gpg, for example.)
Well, but that can not be solved. During suspend-to-disk, data (by definition) go to magnetic media. We could block suspend, and we could kill such application....
Pavel
Signature · 2 lines
When do you have a heart between your knees? [Johanka's followup: and *two* hearts?]
Quoted 2 lines
Note that there are some applications for which it is a *bug* if an mlocked page gets written out to magnetic media. (gpg, for example.)
mlock() does not guarantee things not hitting magnetic media, just as mlock() doesn't guarantee that the physical address of a page doesn't change. mlock guarantees that you won't get hard pagefaults and that you have guaranteed memory for the task at hand (eg for realtime apps and oom-critical stuff)
Quoted 7 lines · Arjan van de Ven
Quoted 3 lines
Note that there are some applications for which it is a *bug* if an mlocked page gets written out to magnetic media. (gpg, for example.)
mlock()does not guarantee things not hitting magnetic media, just asmlock()doesn't guarantee that the physical address of a page doesn't change.
The mlock() man page sure seems to hint that they do, by explicitly describing its use by high-security data processing as a way to keep the information from getting to disk. There also seem to be a lot of references on the web about using mlock() in secure programming.
Something is not right...
Chris
Signature · 4 lines
Chris Friesen | MailStop: 043/33/F10 Nortel Networks | work: (613) 765-0557 3500 Carling Avenue | fax: (613) 765-2986 Nepean, ON K2H 8E9 Canada | email: [email protected]
Quoted 5 lines · Chris Friesen
Quoted 4 lines · Arjan van de Ven
Quoted 3 lines
Note that there are some applications for which it is a *bug* if an mlocked page gets written out to magnetic media. (gpg, for example.)
Quoted 7 lines
Quoted 3 lines
mlock()does not guarantee things not hitting magnetic media, just asmlock()doesn't guarantee that the physical address of a page doesn't change.The
mlock()man page sure seems to hint that they do, by explicitly describing its use by high-security data processing as a way to keep the information from getting to disk.
... and explicitly describing that this is not a 100% thing due to suspend etc etc.
----
mlock disables paging for the memory in the range starting at addr with length len bytes. All pages which contain a part of the specified memory range are guaranteed be resident in RAM when the mlock system call returns successfully and they are guaranteed to stay in RAM until the pages are unlocked by munlock or munlockall, until the pages are unmapped via munmap, or until the process terminates or starts another program with exec. Child processes do not inherit page locks across a fork.
-----
that is what it guarantees. it guarantees that you don't hard-fault. The rest of the manpage talks about potential usages but immediatly describes the crypto one as non-solid
Quoted 3 lines · Arjan van de Ven
that is what it guarantees. it guarantees that you don't hard-fault. The rest of the manpage talks about potential usages but immediatly describes the crypto one as non-solid
Guess I've got older manpages...mine don't have the caveats.
Chris
Signature · 4 lines
Chris Friesen | MailStop: 043/33/F10 Nortel Networks | work: (613) 765-0557 3500 Carling Avenue | fax: (613) 765-2986 Nepean, ON K2H 8E9 Canada | email: [email protected]
Quoted 7 lines · Chris Friesen
Quoted 3 lines · Arjan van de Ven
that is what it guarantees. it guarantees that you don't hard-fault. The rest of the manpage talks about potential usages but immediatly describes the crypto one as non-solid
Guess I've got older manpages...mine don't have the caveats.
POSIX don't mention sample usage or caveats either, it only states guaranteed memory residency.
Quoted 8 lines · Arjan van de Ven
Quoted 2 lines
Note that there are some applications for which it is a *bug* if an mlocked page gets written out to magnetic media. (gpg, for example.)
mlock()does not guarantee things not hitting magnetic media, just asmlock()doesn't guarantee that the physical address of a page doesn't change. mlock guarantees that you won't get hard pagefaults and that you have guaranteed memory for the task at hand (eg for realtime apps and oom-critical stuff)
Well, that's fine -- you can certainly define mlock to have whatever semantics you want. But the semantics that gpg depends on are reasonable, and if mlock is changed to have other semantics, there should be some way for apps to get the behavior that used to be implemented by mlock (and *documented* in the mlock man page).
It's a pity that mlock doesn't take a flags argument.
-andy
Hi!
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines
Note that there are some applications for which it is a *bug* if an mlocked page gets written out to magnetic media. (gpg, for example.)
mlock()does not guarantee things not hitting magnetic media, just asmlock()doesn't guarantee that the physical address of a page doesn't change. mlock guarantees that you won't get hard pagefaults and that you have guaranteed memory for the task at hand (eg for realtime apps and oom-critical stuff)
Quoted 7 lines
Well, that's fine -- you can certainly define mlock to have whatever semantics you want. But the semantics that gpg depends on are reasonable, and if mlock is changed to have other semantics, there should be some way for apps to get the behavior that used to be implemented by mlock (and *documented* in the mlock man page).
It's a pity that mlock doesn't take a flags argument.
How would it help?
Block system-wide suspend because 4K are mlocked?
Pavel
Signature · 2 lines
When do you have a heart between your knees? [Johanka's followup: and *two* hearts?]
Quoted 15 lines · Pavel Machek
Quoted 15 lines
Quoted 8 lines
Quoted 2 lines
Note that there are some applications for which it is a *bug* if an mlocked page gets written out to magnetic media. (gpg, for example.)
mlock()does not guarantee things not hitting magnetic media, just asmlock()doesn't guarantee that the physical address of a page doesn't change. mlock guarantees that you won't get hard pagefaults and that you have guaranteed memory for the task at hand (eg for realtime apps and oom-critical stuff)Well, that's fine -- you can certainly define mlock to have whatever semantics you want. But the semantics that gpg depends on are reasonable, and if mlock is changed to have other semantics, there should be some way for apps to get the behavior that used to be implemented by mlock (and *documented* in the mlock man page).
Quoted 5 lines
Quoted 1 line
It's a pity that mlock doesn't take a flags argument.
How would it help?
Block system-wide suspend because 4K are mlocked?
Sorry, I left too much of my train of thought implicit. I'm suggesting that it would be cool if there were an mlockf(addr, len, ML_NOSWAP) which would allow an app to say "do not write this page to disk or send it over the network." If the kernel decided to evict that page (due to doing a suspend, perhaps) it would just drop the mapping, and when the app next used it there would be a SEGV delivered.
Alternatively, you could define a protocol for suspend to notify apps with mlocked memory that they must clean up in preparation for a suspend. It doesn't have to be bulletproof; you can give them one chance, and if they don't do it just proceed with the suspend. (Unfortunately this does violate the "never write key material to magnetic store" semantic as well, but at least you've given the app a chance.) Perhaps just SIGUSR1 or something?
Perhaps a better API than mlockf(addr, len, flags) would be a mattr(addr, len, flags) with MA_NOFAULT, MA_FIXEDPHYSADDR, MA_NOSWAP... mlock() could then be defined as mattr(addr, len, MA_NOFAULT).
I agree that all of this is beyond the scope of what you're trying to do in swsusp. I just want to bring up the issues so that they're not ignored.
-andy