Skip to contentSkip to conversation

Fix leaking of kernel heap addresses via /proc

SECURITY Dan Rosenberg 10 participants 18 messages Nov 6, 2010 – Nov 8, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Maintainers,

I am planning on submitting a series of patches to resolve the leakage of kernel heap addresses to userspace via network protocol /proc interfaces. Revealing this information is a bad idea from a security perspective for a number of reasons, the most obvious of which is it provides unprivileged users a mechanism by which to create a structure in the kernel heap containing function pointers, obtain the address of that structure, and overwrite those function pointers by leveraging other vulnerabilities. It is my hope that by eliminating this information leakage, in conjunction with making statically-declared function pointer tables read-only (to be done in a separate patch series), we can at least add a small hurdle for the exploitation of a subset of kernel vulnerabilities.

Here's the list of protocols that I've identified as leaking socket structure addresses via their /proc interfaces:

atm
ipv4
ipv6
key
netlink
packet
phonet
unix

There may be others, and I will continue looking for more examples, but this is at least a good start.

Clearly, in most cases we cannot just remove the field from the /proc output, as this would break a number of userspace programs that rely on consistency. However, I propose that we replace the address with a "0" rather than leaking this information.

I'm writing this email before submitting a patch to ask if anyone would have any objections to this change, and if anyone can foresee anything breaking as a result of this. Please let me know if you're on board, have some doubts, or are totally opposed, and I can get this patch ready for submission.

Thanks,
Dan Rosenberg

#1 lore raw
Quoted 5 lines · Dan Rosenberg

Clearly, in most cases we cannot just remove the field from the /proc output, as this would break a number of userspace programs that rely on consistency.  However, I propose that we replace the address with a "0" rather than leaking this information.

I really think it would be much better to use the unidentified number or similar.

Just replacing with zeroes is annoying, and has the potential of losing actual information.

Linus

#2 lore raw
Quoted 11 lines · Linus Torvalds
Quoted 4 lines · Dan Rosenberg

Clearly, in most cases we cannot just remove the field from the /proc output, as this would break a number of userspace programs that rely on consistency.  However, I propose that we replace the address with a "0" rather than leaking this information.

I really think it would be much better to use the unidentified number or similar.

Just replacing with zeroes is annoying, and has the potential of losing actual information.

Are there any userspace programs that might be reasonably expected to _use_ this information? If there is, we could just pick a random number at boot time, and then XOR the heap adddress with that random number.

- Ted

#3 lore raw
Quoted 4 lines · Ted Ts'o

Are there any userspace programs that might be reasonably expected to _use_ this information? If there is, we could just pick a random number at boot time, and then XOR the heap adddress with that random number.

If any of the addresses can be guessed ever (and that is likely if it's allocated at boot) determining the random value will be trivial for everyone.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#4 lore raw
Quoted 10 lines · In message <[email protected]>,Andi Kleen writes:
Quoted 4 lines · Ted Ts'o

Are there any userspace programs that might be reasonably expected to _use_ this information? If there is, we could just pick a random number at boot time, and then XOR the heap adddress with that random number.

If any of the addresses can be guessed ever (and that is likely if it's allocated at boot) determining the random value will be trivial for everyone.

i suppose one could use idr to map the pointers to unique values. the infiniband code uses this technique>

#5 lore raw
Quoted 8 lines · Chas Williams (CONTRACTOR)
Quoted 7 lines · In message <[email protected]>,Andi Kleen writes:
Quoted 4 lines · Ted Ts'o

Are there any userspace programs that might be reasonably expected to _use_ this information? If there is, we could just pick a random number at boot time, and then XOR the heap adddress with that random number.

Quoted 6 lines
Quoted 3 lines

If any of the addresses can be guessed ever (and that is likely if it's allocated at boot) determining the random value will be trivial for everyone.

i suppose one could use idr to map the pointers to unique values. the infiniband code uses this technique>

idr requires allocating memory, and it's unclear you can do that in all the situations where debugging printks are used. And how would you get the idr table out of a broken kernel? And further the memory allocations would eventually fill up your memory if they go on. I don't think idr is a solution.

I don't really have a good solution either. Even if the the individual pointers were removed from printks (which probably doesn't make much difference anyways because those printks usually happen only in unlikely debug situations):

The information about the memory layout early on in dmesg is probably enough to make a good educated guess about the locations of standard slab caches on a known kernel image. For example the first N sockets opened are very likely easy to guess this way.

I guess one could make dmesg root only, although I personally use it often as non root.

Or maybe add some more randomization to the buddy allocator. The drawback of that is that they tend to make benchmarks unstable due to cache coloring differences.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#6 lore raw
In reply to Chas Williams (CONTRACTOR)
Quoted 3 lines · Chas Williams (CONTRACTOR)

i suppose one could use idr to map the pointers to unique values.  the infiniband code uses this technique>

We already _have_ the unique value that /proc uses - the inode number. It's what lsof and friends use to match things across different files anyway.

Why are people arguing about stupid things? If it's wrong to expose kernel pointers in /proc (and I do think it generally is something we should try to avoid), then we already do have the natural alternative. Which happens to be what the patch already does.

So the only question is whether kernel pointers are an information leak worth worrying about. Personally, I think it's just damn stupid to expose a kernel pointer unless you _have_ to. There is absolutely no reason to expose the address of a socket in /proc, perhaps unless you're in some super-duper-debugging mode that no sane person would ever use outside of specialized network debugging (and I bet that in that case you still shouldn't expose it in a _normal_ proc file, but in debugfs or something).

Linus

#7 lore raw
Quoted 3 lines

So the only question is whether kernel pointers are an information leak worth worrying about. Personally, I think it's just damn stupid to expose a kernel pointer unless you _have_ to. There is absolutely

Agreed.

Quoted 5 lines

no reason to expose the address of a socket in /proc, perhaps unless you're in some super-duper-debugging mode that no sane person would ever use outside of specialized network debugging (and I bet that in that case you still shouldn't expose it in a _normal_ proc file, but in debugfs or something).

In most cases you can get them by running gdb or crash on /proc/kcore anyways.

Still overall I suspect there may be a case that making dmesg root only is a good idea. At least optionally for the non kernel hackers out there.

The early memory map information can be likely used to guess a lot of locations and perhaps really make that heap overflow easier to write. And perhaps some more mapping randomization inside the kernel.

I'm not sure the case is very strong though.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#8 lore raw
In reply to Chas Williams (CONTRACTOR)

Based on the feedback so far, I think a few things need to be worked out. Firstly, I'm going to separate out printk leaks as an issue to be worked on at a later time - it seems there is some interest in evaluating whether dmesg should be restricted, but that is out of scope of my plans for this patch series.

That leaves the /proc leakage. I don't think XOR-ing is a viable option, since it would be relatively easy to infer the constant value. The criticism raised so far is that cutting out the pointers entirely results in the omission of potentially useful debugging information. I see two viable options to address this: either print out or omit addresses based on privileges (CAP_NET_ADMIN, for example), or have it controllable via sysctl. I'm leaning towards the sysctl option...thoughts?

-Dan

#9 lore raw
Quoted 6 lines

The criticism raised so far is that cutting out the pointers entirely results in the omission of potentially useful debugging information. I see two viable options to address this: either print out or omit addresses based on privileges (CAP_NET_ADMIN, for example), or have it controllable via sysctl. I'm leaning towards the sysctl option...thoughts?

I would just remove the pointers from /proc and supply gdb macros that extract the equivalent information from /proc/kcore. This is a bit racy, but for debugging it should be no problem to run them multiple times as needed.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
maintainer
#10 lore raw

From: Andi Kleen <[email protected]>
Date: Mon, 8 Nov 2010 00:56:10 +0100

Quoted 4 lines

I would just remove the pointers from /proc and supply gdb macros that extract the equivalent information from /proc/kcore. This is a bit racy, but for debugging it should be no problem to run them multiple times as needed.

I do not think at all that this is tenable for the kind of things people use the socket pointers for when debugging problems.

I defeinitely prefer the inode number to this idea.

#11 lore raw
Quoted 8 lines · David Miller

From: Andi Kleen <[email protected]>
Date: Mon, 8 Nov 2010 00:56:10 +0100

Quoted 4 lines

I would just remove the pointers from /proc and supply gdb macros that extract the equivalent information from /proc/kcore. This is a bit racy, but for debugging it should be no problem to run them multiple times as needed.

Quoted 5 lines

I do not think at all that this is tenable for the kind of things people use the socket pointers for when debugging problems.

I defeinitely prefer the inode number to this idea.

We currently have no guarantee of sockets inode numbers unicity. I admit chances of clash are low.

When a printk() happens right before a BUG(), how are we going to check the dumped registers are possibly close the socket involved, if we dont have access to the machine, and only the crashlog ?

BTW, any local user can look at "dmesg", and crash reports. These reports are even published on a remote site (bugzilla) so that hostile hackers can be feeded.

I am OK to delete socket pointers from /proc files for non root users (after checking things like lsof continue to work correctly). I dont remember using them while doing debugging stuff.

BTW, rtnetlink also expose socket pointers to non root users :

$ ss -e dst 192.168.20.108
State      Recv-Q Send-Q    Local Address:Port    Peer Address:Port   
ESTAB      0      0         10.150.51.210:46979   192.168.20.108:ssh 
timer:(keepalive,119min,0) ino:136919 sk:ffff88002129d7c0

Mixing in same patch /proc pointers removal and printk() pointers removal seems wrong to me. Very different problems.

#12 lore raw
Quoted 3 lines

When a printk() happens right before a BUG(), how are we going to check the dumped registers are possibly close the socket involved, if we dont have access to the machine, and only the crashlog ?

Is that really something you do regularly? It seems highly obscure to me.

Besides if the kernel has timestamps enabled you can easily guess based on the timestamps if the printk and the oops are related.

-Andi

#13 lore raw
Quoted 6 lines · Andi Kleen
Quoted 3 lines

When a printk() happens right before a BUG(), how are we going to check the dumped registers are possibly close the socket involved, if we dont have access to the machine, and only the crashlog ?

Is that really something you do regularly? It seems highly obscure to me.

Yes, very regularly, I can find bugs thanks to every bit of information found in kernel logs, including code around the fault.

If people now say : "I have a kernel bug, but am not able to provide you a kernel stack trace and previous printk() messages because of security. You cannot have an access to this machine, and the bug happens once in a while. Kernel version is also hidden. Please help me."

Oh well, thats a challenge, maybe use this cristal ball I have somewhere in the attic ;)

maintainer
#14 lore raw
In reply to Linus Torvalds

From: Linus Torvalds <[email protected]>
Date: Sat, 6 Nov 2010 13:50:32 -0700

Quoted 7 lines · Dan Rosenberg
Quoted 5 lines

Clearly, in most cases we cannot just remove the field from the /proc output, as this would break a number of userspace programs that rely on consistency.  However, I propose that we replace the address with a "0" rather than leaking this information.

Quoted 5 lines

I really think it would be much better to use the unidentified number or similar.

Just replacing with zeroes is annoying, and has the potential of losing actual information.

I would really like to see the specific examples of where this is happening, it sounds like something very silly to me.

#15 lore raw
Quoted 10 lines · David Miller

From: Linus Torvalds <[email protected]>
Date: Sat, 6 Nov 2010 13:50:32 -0700

Quoted 7 lines · Dan Rosenberg
Quoted 5 lines

Clearly, in most cases we cannot just remove the field from the /proc output, as this would break a number of userspace programs that rely on consistency. However, I propose that we replace the address with a "0" rather than leaking this information.

Quoted 8 lines
Quoted 5 lines

I really think it would be much better to use the unidentified number or similar.

Just replacing with zeroes is annoying, and has the potential of losing actual information.

I would really like to see the specific examples of where this is happening, it sounds like something very silly to me.

Indeed Urs and me had a similar discussion before we decided to put different (user relevant) content into the procfs output and break the current layout:

http://patchwork.ozlabs.org/patch/70282/

The layout break was ok in this case as the people using the CAN procfs stuff do this only when facing problems (with their applications) at runtime.

A discussed approach that won't break the procfs layout was to set the values to "0" and only fill them with real content depending on CONFIG_DEBUG_INFO .

Would that fit here?

Or maybe a different config option CONFIG_DEBUG_KERNEL_ADDR would do the job, as i don't know which distros enable CONFIG_DEBUG_INFO by default ...

Regards,
Oliver

#16 lore raw

On Sun, 2010-11-07 at 11:28 +0100, Oliver Hartkopp wrote:
[...]

Quoted 10 lines

The layout break was ok in this case as the people using the CAN procfs stuff do this only when facing problems (with their applications) at runtime.

A discussed approach that won't break the procfs layout was to set the values to "0" and only fill them with real content depending on CONFIG_DEBUG_INFO .

Would that fit here?

Or maybe a different config option CONFIG_DEBUG_KERNEL_ADDR would do the job, as i don't know which distros enable CONFIG_DEBUG_INFO by default ...

I believe most distributions now enable CONFIG_DEBUG_INFO (though the debug information is then stripped and shipped in separate packages). I also dislike this idea of an implicit trade-off between debugging and security; such a trade-off may be necessary but then it should be explicit.

Ben.

Signature · 2 lines
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.
#17 lore raw
In reply to David Miller
Quoted 10 lines · David Miller

From: Linus Torvalds <[email protected]>
Date: Sat, 6 Nov 2010 13:50:32 -0700

Quoted 7 lines · Dan Rosenberg
Quoted 5 lines

Clearly, in most cases we cannot just remove the field from the /proc output, as this would break a number of userspace programs that rely on consistency.  However, I propose that we replace the address with a "0" rather than leaking this information.

Quoted 8 lines
Quoted 5 lines

I really think it would be much better to use the unidentified number or similar.

Just replacing with zeroes is annoying, and has the potential of losing actual information.

I would really like to see the specific examples of where this is happening, it sounds like something very silly to me.

It has happened to me several times to use an hex editor to check some socket's parameters (eg: backlog) based on the pointer. Sometimes I had even change some parameters at runtime as part of debugging sessions.

In fact we could consider than many places that return pointers could return 0 to normal users and the real value only to root (or any special capability). I find it important not to reduce the observability of the kernel for the sake of security.

Regards,
Willy