Fix leaking of kernel heap addresses via /proc
Subsystems (2)
Maintainers,
I am planning on submitting a series of patches to resolve the leakage of kernel heap addresses to userspace via network protocol /proc interfaces. Revealing this information is a bad idea from a security perspective for a number of reasons, the most obvious of which is it provides unprivileged users a mechanism by which to create a structure in the kernel heap containing function pointers, obtain the address of that structure, and overwrite those function pointers by leveraging other vulnerabilities. It is my hope that by eliminating this information leakage, in conjunction with making statically-declared function pointer tables read-only (to be done in a separate patch series), we can at least add a small hurdle for the exploitation of a subset of kernel vulnerabilities.
Here's the list of protocols that I've identified as leaking socket structure addresses via their /proc interfaces:
atm
ipv4
ipv6
key
netlink
packet
phonet
unix
There may be others, and I will continue looking for more examples, but this is at least a good start.
Clearly, in most cases we cannot just remove the field from the /proc output, as this would break a number of userspace programs that rely on consistency. However, I propose that we replace the address with a "0" rather than leaking this information.
I'm writing this email before submitting a patch to ask if anyone would have any objections to this change, and if anyone can foresee anything breaking as a result of this. Please let me know if you're on board, have some doubts, or are totally opposed, and I can get this patch ready for submission.
Thanks,
Dan Rosenberg
Quoted 5 lines · Dan Rosenberg
Clearly, in most cases we cannot just remove the field from the /proc output, as this would break a number of userspace programs that rely on consistency. However, I propose that we replace the address with a "0" rather than leaking this information.
I really think it would be much better to use the unidentified number or similar.
Just replacing with zeroes is annoying, and has the potential of losing actual information.
Linus
Quoted 11 lines · Linus Torvalds
Quoted 4 lines · Dan Rosenberg
Clearly, in most cases we cannot just remove the field from the /proc output, as this would break a number of userspace programs that rely on consistency. However, I propose that we replace the address with a "0" rather than leaking this information.
I really think it would be much better to use the unidentified number or similar.
Just replacing with zeroes is annoying, and has the potential of losing actual information.
Are there any userspace programs that might be reasonably expected to _use_ this information? If there is, we could just pick a random number at boot time, and then XOR the heap adddress with that random number.
- Ted
David Miller
maintainerFrom: Linus Torvalds <[email protected]>
Date: Sat, 6 Nov 2010 13:50:32 -0700
Quoted 7 lines · Dan Rosenberg
Quoted 5 lines
Clearly, in most cases we cannot just remove the field from the /proc output, as this would break a number of userspace programs that rely on consistency. However, I propose that we replace the address with a "0" rather than leaking this information.
Quoted 5 lines
I really think it would be much better to use the unidentified number or similar.
Just replacing with zeroes is annoying, and has the potential of losing actual information.
I would really like to see the specific examples of where this is happening, it sounds like something very silly to me.
Quoted 10 lines · David Miller
From: Linus Torvalds <[email protected]>
Date: Sat, 6 Nov 2010 13:50:32 -0700Quoted 7 lines · Dan Rosenberg
Quoted 5 lines
Clearly, in most cases we cannot just remove the field from the /proc output, as this would break a number of userspace programs that rely on consistency. However, I propose that we replace the address with a "0" rather than leaking this information.
Quoted 8 lines
Quoted 5 lines
I really think it would be much better to use the unidentified number or similar.
Just replacing with zeroes is annoying, and has the potential of losing actual information.
I would really like to see the specific examples of where this is happening, it sounds like something very silly to me.
Indeed Urs and me had a similar discussion before we decided to put different (user relevant) content into the procfs output and break the current layout:
http://patchwork.ozlabs.org/patch/70282/
The layout break was ok in this case as the people using the CAN procfs stuff do this only when facing problems (with their applications) at runtime.
A discussed approach that won't break the procfs layout was to set the values to "0" and only fill them with real content depending on CONFIG_DEBUG_INFO .
Would that fit here?
Or maybe a different config option CONFIG_DEBUG_KERNEL_ADDR would do the job, as i don't know which distros enable CONFIG_DEBUG_INFO by default ...
Regards,
Oliver
On Sun, 2010-11-07 at 11:28 +0100, Oliver Hartkopp wrote:
[...]
Quoted 10 lines
The layout break was ok in this case as the people using the CAN procfs stuff do this only when facing problems (with their applications) at runtime.
A discussed approach that won't break the procfs layout was to set the values to "0" and only fill them with real content depending on
CONFIG_DEBUG_INFO.Would that fit here?
Or maybe a different config option
CONFIG_DEBUG_KERNEL_ADDRwould do the job, as i don't know which distros enableCONFIG_DEBUG_INFOby default ...
I believe most distributions now enable CONFIG_DEBUG_INFO (though the debug information is then stripped and shipped in separate packages). I also dislike this idea of an implicit trade-off between debugging and security; such a trade-off may be necessary but then it should be explicit.
Ben.
Signature · 2 lines
Ben Hutchings Once a job is fouled up, anything done to improve it makes it worse.
Quoted 4 lines · Ted Ts'o
Are there any userspace programs that might be reasonably expected to _use_ this information? If there is, we could just pick a random number at boot time, and then XOR the heap adddress with that random number.
If any of the addresses can be guessed ever (and that is likely if it's allocated at boot) determining the random value will be trivial for everyone.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 10 lines · In message <[email protected]>,Andi Kleen writes:
Quoted 4 lines · Ted Ts'o
Are there any userspace programs that might be reasonably expected to _use_ this information? If there is, we could just pick a random number at boot time, and then XOR the heap adddress with that random number.
If any of the addresses can be guessed ever (and that is likely if it's allocated at boot) determining the random value will be trivial for everyone.
i suppose one could use idr to map the pointers to unique values. the infiniband code uses this technique>
Quoted 8 lines · Chas Williams (CONTRACTOR)
Quoted 7 lines · In message <[email protected]>,Andi Kleen writes:
Quoted 4 lines · Ted Ts'o
Are there any userspace programs that might be reasonably expected to _use_ this information? If there is, we could just pick a random number at boot time, and then XOR the heap adddress with that random number.
Quoted 6 lines
Quoted 3 lines
If any of the addresses can be guessed ever (and that is likely if it's allocated at boot) determining the random value will be trivial for everyone.
i suppose one could use idr to map the pointers to unique values. the infiniband code uses this technique>
idr requires allocating memory, and it's unclear you can do that in all the situations where debugging printks are used. And how would you get the idr table out of a broken kernel? And further the memory allocations would eventually fill up your memory if they go on. I don't think idr is a solution.
I don't really have a good solution either. Even if the the individual pointers were removed from printks (which probably doesn't make much difference anyways because those printks usually happen only in unlikely debug situations):
The information about the memory layout early on in dmesg is probably enough to make a good educated guess about the locations of standard slab caches on a known kernel image. For example the first N sockets opened are very likely easy to guess this way.
I guess one could make dmesg root only, although I personally use it often as non root.
Or maybe add some more randomization to the buddy allocator. The drawback of that is that they tend to make benchmarks unstable due to cache coloring differences.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 3 lines · Chas Williams (CONTRACTOR)
i suppose one could use idr to map the pointers to unique values. the infiniband code uses this technique>
We already _have_ the unique value that /proc uses - the inode number. It's what lsof and friends use to match things across different files anyway.
Why are people arguing about stupid things? If it's wrong to expose kernel pointers in /proc (and I do think it generally is something we should try to avoid), then we already do have the natural alternative. Which happens to be what the patch already does.
So the only question is whether kernel pointers are an information leak worth worrying about. Personally, I think it's just damn stupid to expose a kernel pointer unless you _have_ to. There is absolutely no reason to expose the address of a socket in /proc, perhaps unless you're in some super-duper-debugging mode that no sane person would ever use outside of specialized network debugging (and I bet that in that case you still shouldn't expose it in a _normal_ proc file, but in debugfs or something).
Linus
Based on the feedback so far, I think a few things need to be worked out. Firstly, I'm going to separate out printk leaks as an issue to be worked on at a later time - it seems there is some interest in evaluating whether dmesg should be restricted, but that is out of scope of my plans for this patch series.
That leaves the /proc leakage. I don't think XOR-ing is a viable option, since it would be relatively easy to infer the constant value. The criticism raised so far is that cutting out the pointers entirely results in the omission of potentially useful debugging information. I see two viable options to address this: either print out or omit addresses based on privileges (CAP_NET_ADMIN, for example), or have it controllable via sysctl. I'm leaning towards the sysctl option...thoughts?
-Dan
Quoted 3 lines
So the only question is whether kernel pointers are an information leak worth worrying about. Personally, I think it's just damn stupid to expose a kernel pointer unless you _have_ to. There is absolutely
Agreed.
Quoted 5 lines
no reason to expose the address of a socket in /proc, perhaps unless you're in some super-duper-debugging mode that no sane person would ever use outside of specialized network debugging (and I bet that in that case you still shouldn't expose it in a _normal_ proc file, but in debugfs or something).
In most cases you can get them by running gdb or crash on /proc/kcore anyways.
Still overall I suspect there may be a case that making dmesg root only is a good idea. At least optionally for the non kernel hackers out there.
The early memory map information can be likely used to guess a lot of locations and perhaps really make that heap overflow easier to write. And perhaps some more mapping randomization inside the kernel.
I'm not sure the case is very strong though.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 6 lines
The criticism raised so far is that cutting out the pointers entirely results in the omission of potentially useful debugging information. I see two viable options to address this: either print out or omit addresses based on privileges (
CAP_NET_ADMIN, for example), or have it controllable via sysctl. I'm leaning towards the sysctl option...thoughts?
I would just remove the pointers from /proc and supply gdb macros that extract the equivalent information from /proc/kcore. This is a bit racy, but for debugging it should be no problem to run them multiple times as needed.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 10 lines · David Miller
From: Linus Torvalds <[email protected]>
Date: Sat, 6 Nov 2010 13:50:32 -0700Quoted 7 lines · Dan Rosenberg
Quoted 5 lines
Clearly, in most cases we cannot just remove the field from the /proc output, as this would break a number of userspace programs that rely on consistency. However, I propose that we replace the address with a "0" rather than leaking this information.
Quoted 8 lines
Quoted 5 lines
I really think it would be much better to use the unidentified number or similar.
Just replacing with zeroes is annoying, and has the potential of losing actual information.
I would really like to see the specific examples of where this is happening, it sounds like something very silly to me.
It has happened to me several times to use an hex editor to check some socket's parameters (eg: backlog) based on the pointer. Sometimes I had even change some parameters at runtime as part of debugging sessions.
In fact we could consider than many places that return pointers could return 0 to normal users and the real value only to root (or any special capability). I find it important not to reduce the observability of the kernel for the sake of security.
Regards,
Willy
David Miller
maintainerFrom: Andi Kleen <[email protected]>
Date: Mon, 8 Nov 2010 00:56:10 +0100
Quoted 4 lines
I would just remove the pointers from /proc and supply gdb macros that extract the equivalent information from
/proc/kcore. This is a bit racy, but for debugging it should be no problem to run them multiple times as needed.
I do not think at all that this is tenable for the kind of things people use the socket pointers for when debugging problems.
I defeinitely prefer the inode number to this idea.
Quoted 8 lines · David Miller
From: Andi Kleen <[email protected]>
Date: Mon, 8 Nov 2010 00:56:10 +0100Quoted 4 lines
I would just remove the pointers from /proc and supply gdb macros that extract the equivalent information from
/proc/kcore. This is a bit racy, but for debugging it should be no problem to run them multiple times as needed.
Quoted 5 lines
I do not think at all that this is tenable for the kind of things people use the socket pointers for when debugging problems.
I defeinitely prefer the inode number to this idea.
We currently have no guarantee of sockets inode numbers unicity. I admit chances of clash are low.
When a printk() happens right before a BUG(), how are we going to check the dumped registers are possibly close the socket involved, if we dont have access to the machine, and only the crashlog ?
BTW, any local user can look at "dmesg", and crash reports. These reports are even published on a remote site (bugzilla) so that hostile hackers can be feeded.
I am OK to delete socket pointers from /proc files for non root users (after checking things like lsof continue to work correctly). I dont remember using them while doing debugging stuff.
BTW, rtnetlink also expose socket pointers to non root users :
$ ss -e dst 192.168.20.108 State Recv-Q Send-Q Local Address:Port Peer Address:Port ESTAB 0 0 10.150.51.210:46979 192.168.20.108:ssh timer:(keepalive,119min,0) ino:136919 sk:ffff88002129d7c0
Mixing in same patch /proc pointers removal and printk() pointers removal seems wrong to me. Very different problems.
Quoted 3 lines
When a
printk()happens right before aBUG(), how are we going to check the dumped registers are possibly close the socket involved, if we dont have access to the machine, and only the crashlog ?
Is that really something you do regularly? It seems highly obscure to me.
Besides if the kernel has timestamps enabled you can easily guess based on the timestamps if the printk and the oops are related.
-Andi
Quoted 6 lines · Andi Kleen
Quoted 3 lines
When a
printk()happens right before aBUG(), how are we going to check the dumped registers are possibly close the socket involved, if we dont have access to the machine, and only the crashlog ?Is that really something you do regularly? It seems highly obscure to me.
Yes, very regularly, I can find bugs thanks to every bit of information found in kernel logs, including code around the fault.
If people now say : "I have a kernel bug, but am not able to provide you a kernel stack trace and previous printk() messages because of security. You cannot have an access to this machine, and the bug happens once in a while. Kernel version is also hidden. Please help me."
Oh well, thats a challenge, maybe use this cristal ball I have somewhere in the attic ;)