Use-after-free and management of reference counts
Hi,
Recently I studied the reported CVEs of last 2 years and there were fair number of use-after-free bugs. Usually we also see many reports of use-after-free bugs in the LKML [reported by one or other tools].
Also, at Kernel recipes Jonathan Corbet mentioned reference counts as a security issue. I believe if we have more kernel hardening patches then we can avoid such bugs. I was wondering if there is some ongoing work in the both [use-after-free and management of reference counts] of these areas?
If not, then I would like to work on the same. Any pointers on this would be appreciated.
Thank you.
Signature · 1 line
Vaishali
Hi,
Quoted 6 lines · Vaishali Thakkar
Hi,
Recently I studied the reported CVEs of last 2 years and there were fair number of use-after-free bugs. Usually we also see many reports of use-after-free bugs in the LKML [reported by one or other tools].
Quoted 6 lines
Also, at Kernel recipes Jonathan Corbet mentioned reference counts as a security issue. I believe if we have more kernel hardening patches then we can avoid such bugs. I was wondering if there is some ongoing work in the both [use-after-free and management of reference counts] of these areas?
There is a patchset currently on this mailing list (kernel-hardening) that directly addresses use-after free bugs. The feature is called HARDENED_ATOMIC and is based off of earlier work done by PaX
(https://forums.grsecurity.net/viewtopic.php?f=7&t=4173).
Here is a link to the feature's latest RFC patchset:
http://www.openwall.com/lists/kernel-hardening/2016/10/03/1
Here is the work we've done thus far in documenting the feature:
http://progbits.org
If you're interested in helping with this feature, we're always looking for people to help move the feature to other architectures. Someone else is already working on the arm64 port, so you may want to look at either arm (this is most preferable), mips, sparc or powerpc. You may want to talk to Takahiro Akashi, who is doing the arm64 work, to get his thoughts on the arm port. See this thread for more
information: http://www.openwall.com/lists/kernel-hardening/2016/10/12/2.
Quoted 3 lines
If not, then I would like to work on the same. Any pointers on this would be appreciated.
Great, thanks for offering to help! Let me know if you'd like any other pointers.
Quoted 4 lines
Thank you.
--
Vaishali
Hi Vaishali, David,
Quoted 4 lines
If you're interested in helping with this feature, we're always looking for people to help move the feature to other architectures. Someone else is already working on the arm64 port, so you may want to look at either arm (this is most preferable), mips, sparc or powerpc.
Actually, I've started to work on arm (see a little patch-set I've just send for review). But as beginning kernel developer, I am glad if others people can have a look/give help on that stuff :-)
Thanks,
Colin
Quoted 11 lines · Vaishali Thakkar
Hi,
Recently I studied the reported CVEs of last 2 years and there were fair number of use-after-free bugs. Usually we also see many reports of use-after-free bugs in the LKML [reported by one or other tools].
Also, at Kernel recipes Jonathan Corbet mentioned reference counts as a security issue. I believe if we have more kernel hardening patches then we can avoid such bugs. I was wondering if there is some ongoing work in the both [use-after-free and management of reference counts] of these areas?
Use-after-frees are really hard to deal with. I think I saw some patch semi-recently for randomizing kernel allocations, to make it harder to exploit memory safety bugs, but actually fixing use-after-free is a really hard problem.
For reference counting, refcount overdecrements are pretty much as hard to deal with as UAFs, but for refcount overincrements, you can take a look at Elena Reshetova's refcount hardening patch series (see
http://www.openwall.com/lists/kernel-hardening/2016/10/10/1).
Quoted 1 line · Jann Horn
Use-after-frees are really hard to deal with.
I want to ask a possibly naive questions -- why can't we avoid the whole class of bugs in the first place? I can think of only two main sources for use-after-free bugs, both avoidable.
One is allocating something within a function, then returning a pointer to it. I know I've done that on occasion, got a warning from lint(1) or the compiler, & fixed it; I have forgotten details. Surely current static analysis tools can catch nearly all of these. If not, writing one looks fairly straightforward. Sure, complex code with a lot of indirection might fool such tools, but in general such code should not be used anyway.
The other is misusing malloc()/free() or their kernel code analogs, & the obvious solution is to avoid using those wherever possible. There are some kernel data structures that need to grow dynamically, but I do not think there need to be a lot. Code the few very carefully and refuse to accept patches elsewhere that use dynamic allocation; the problem may not vanish, but it looks like this reduces it to manageable size.
I have the feeling I might be missing something here. Pointers welcome.
Quoted 8 lines · Sandy Harris
Quoted 1 line · Jann Horn
Use-after-frees are really hard to deal with.
I want to ask a possibly naive questions -- why can't we avoid the whole class of bugs in the first place? I can think of only two main sources for use-after-free bugs, both avoidable.
Quoted 7 lines
One is allocating something within a function, then returning a pointer to it. I know I've done that on occasion, got a warning from lint(1) or the compiler, & fixed it; I have forgotten details. Surely current static analysis tools can catch nearly all of these. If not, writing one looks fairly straightforward. Sure, complex code with a lot of indirection might fool such tools, but in general such code should not be used anyway.
I'm pretty sure the kernel does that in a lot of places, actually.
Quoted 4 lines
The other is misusing malloc()/free() or their kernel code analogs, & the obvious solution is to avoid using those wherever possible. There are some kernel data structures that need to grow dynamically, but I do not think there need to be a lot.
Actually, nearly everything in the kernel is allocated dynamically. Even data that you might normally put on the stack is allocated dynamically because kernel stacks are so small.
And thanks to namespaces, even things like the hostname can be stored in dynamically allocated memory.
Quoted 17 lines · Jann Horn
Quoted 8 lines · Sandy Harris
Quoted 1 line · Jann Horn
Use-after-frees are really hard to deal with.
I want to ask a possibly naive questions -- why can't we avoid the whole class of bugs in the first place? I can think of only two main sources for use-after-free bugs, both avoidable.
Quoted 7 lines
One is allocating something within a function, then returning a pointer to it. I know I've done that on occasion, got a warning from lint(1) or the compiler, & fixed it; I have forgotten details. Surely current static analysis tools can catch nearly all of these. If not, writing one looks fairly straightforward. Sure, complex code with a lot of indirection might fool such tools, but in general such code should not be used anyway.
Quoted 1 line
I'm pretty sure the kernel does that in a lot of places, actually.
I meant code like this:
char * something(void)
{
char array[SIZE], *p ;Quoted 12 lines
Quoted 6 lines
The other is misusing malloc()/free() or their kernel code analogs, & the obvious solution is to avoid using those wherever possible. There are some kernel data structures that need to grow dynamically, but I do not think there need to be a lot.
Actually, nearly everything in the kernel is allocated dynamically. Even data that you might normally put on the stack is allocated dynamically because kernel stacks are so small.
And thanks to namespaces, even things like the hostname can be stored in dynamically allocated memory.
Quoted 15 lines · Jann Horn
Quoted 6 lines · Sandy Harris
Quoted 1 line · Jann Horn
Use-after-frees are really hard to deal with.
...
Quoted 7 lines
One is allocating something within a function, then returning a pointer to it. I know I've done that on occasion, got a warning from lint(1) or the compiler, & fixed it; I have forgotten details. Surely current static analysis tools can catch nearly all of these. If not, writing one looks fairly straightforward. Sure, complex code with a lot of indirection might fool such tools, but in general such code should not be used anyway.
Quoted 1 line
I'm pretty sure the kernel does that in a lot of places, actually.
I meant code that allocates things on the stack & then returns a pointer to them. That memory is guaranteed invalid since the function's stack area is invalid after return. Surely such errors are rare & can be caught by static analysis. Their frequency should be near zero in any well-written code and zero once it has gone through the sort of testing kernel code gets.
I think you meant calling something from the malloc() family of functions, doing some error checking or initialisation, & then returning the pointer. That is valid code; it just requires great care to balance *alloc()s and free()s,
Quoted 8 lines · Sandy Harris
Quoted 1 line · Jann Horn
Use-after-frees are really hard to deal with.
I want to ask a possibly naive questions -- why can't we avoid the whole class of bugs in the first place? I can think of only two main sources for use-after-free bugs, both avoidable.
Quoted 8 lines
One is allocating something within a function, then returning a pointer to it. I know I've done that on occasion, got a warning from lint(1) or the compiler, & fixed it; I have forgotten details. Surely current static analysis tools can catch nearly all of these. If not, writing one looks fairly straightforward. Sure, complex code with a lot of indirection might fool such tools, but in general such code should not be used anyway.
The main use case in the kernel of reference counters is garbage
collection: the kernel manages a particular object that has multiple users. It cannot safely delete this object until all users of the
object have finished using it. Users acquire and release references
to this object via a get()/put() API. When all users have called put(), the reference counter becomes 0 (or -1, or some other sentinel), and the kernel frees the object.
Things go wrong when the reference counter gets overflowed. If the reference counter is represented with a signed integer type, overflowing the reference counter causes it to go from INT_MAX to INT_MIN, then approach 0. Depending on the logic, the transition to INT_MIN may be enough to trigger the bug, but when the reference counter becomes 0, the kernel will free the underlying object guarded by the reference counter while it still has valid users.
Quoted 9 lines
The other is misusing malloc()/free() or their kernel code analogs, & the obvious solution is to avoid using those wherever possible. There are some kernel data structures that need to grow dynamically, but I do not think there need to be a lot. Code the few very carefully and refuse to accept patches elsewhere that use dynamic allocation; the problem may not vanish, but it looks like this reduces it to manageable size.
I have the feeling I might be missing something here. Pointers welcome.
Quoted 26 lines · David Windsor
Quoted 8 lines · Sandy Harris
Quoted 1 line · Jann Horn
Use-after-frees are really hard to deal with.
I want to ask a possibly naive questions -- why can't we avoid the whole class of bugs in the first place? I can think of only two main sources for use-after-free bugs, both avoidable.
Quoted 8 lines
One is allocating something within a function, then returning a pointer to it. I know I've done that on occasion, got a warning from lint(1) or the compiler, & fixed it; I have forgotten details. Surely current static analysis tools can catch nearly all of these. If not, writing one looks fairly straightforward. Sure, complex code with a lot of indirection might fool such tools, but in general such code should not be used anyway.
The main use case in the kernel of reference counters is garbage
collection: the kernel manages a particular object that has multiple users. It cannot safely delete this object until all users of the
object have finished using it. Users acquire and release references
to this object via a get()/put() API. When all users have calledput(), the reference counter becomes 0 (or -1, or some other sentinel), and the kernel frees the object.
Quoted 7 lines
Things go wrong when the reference counter gets overflowed. If the reference counter is represented with a signed integer type, overflowing the reference counter causes it to go from
INT_MAXtoINT_MIN, then approach 0. Depending on the logic, the transition toINT_MINmay be enough to trigger the bug, but when the reference counter becomes 0, the kernel will free the underlying object guarded by the reference counter while it still has valid users.
One of the other case is when users forget to call put() functions. I think we have seen many such missing put() cases in the kernel. Sometimes in some cases it is also possible that users may call put() little early.
Quoted 10 lines
Quoted 9 lines
The other is misusing malloc()/free() or their kernel code analogs, & the obvious solution is to avoid using those wherever possible. There are some kernel data structures that need to grow dynamically, but I do not think there need to be a lot. Code the few very carefully and refuse to accept patches elsewhere that use dynamic allocation; the problem may not vanish, but it looks like this reduces it to manageable size.
I have the feeling I might be missing something here. Pointers welcome.
Signature · 1 line
Vaishali
Quoted 6 lines · Vaishali Thakkar
Hi,
Recently I studied the reported CVEs of last 2 years and there were fair number of use-after-free bugs. Usually we also see many reports of use-after-free bugs in the LKML [reported by one or other tools].
Quoted 8 lines
Also, at Kernel recipes Jonathan Corbet mentioned reference counts as a security issue. I believe if we have more kernel hardening patches then we can avoid such bugs. I was wondering if there is some ongoing work in the both [use-after-free and management of reference counts] of these areas?
If not, then I would like to work on the same. Any pointers on this would be appreciated.
Others have already mentioned the HARDENED_ATOMIC work that is underway, but one place I'd really like to take a little bit more time to examine is how to improve the zero-poisoning work that Laura Abbott did recently. Having this feature reduces the availability of a subset of kernel memory exposures and can frustrate a subset of use-after-free exploits. It's been in the kernel since v4.6, but it needs some performance tuning and likely some other adjustments. I talk about it a bit here:
https://outflux.net/blog/archives/2016/09/30/security-things-in-linux-v4-6/
I think to really dial in this protection so it is more like PAX_MEMORY_SANITIZE, we'd need to do the following:
- adjust the slab poisoning to do zeroing (upstream's slab poisoning uses a different value)
- consolidate the CONFIG and run-time cmdlines to give us a single "do the right thing" CONFIG
- identify frequently-freed slabs that are less security-sensitive and whitelist them from the poisoning (as done by PAX_MEMORY_SANITIZE)
- add kernel command-line argument to control poisoning (as done by PAX_MEMORY_SANITIZE)
-Kees
Signature · 2 lines
Kees Cook Nexus Security
Quoted 17 lines · Kees Cook
Quoted 6 lines · Vaishali Thakkar
Hi,
Recently I studied the reported CVEs of last 2 years and there were fair number of use-after-free bugs. Usually we also see many reports of use-after-free bugs in the LKML [reported by one or other tools].
Quoted 8 lines
Also, at Kernel recipes Jonathan Corbet mentioned reference counts as a security issue. I believe if we have more kernel hardening patches then we can avoid such bugs. I was wondering if there is some ongoing work in the both [use-after-free and management of reference counts] of these areas?
If not, then I would like to work on the same. Any pointers on this would be appreciated.
Quoted 8 lines
Others have already mentioned the
HARDENED_ATOMICwork that is underway, but one place I'd really like to take a little bit more time to examine is how to improve the zero-poisoning work that Laura Abbott did recently. Having this feature reduces the availability of a subset of kernel memory exposures and can frustrate a subset of use-after-free exploits. It's been in the kernel since v4.6, but it needs some performance tuning and likely some other adjustments. I talk about it a bit here:
Hi,
Sorry, I missed this as I was not CC'ed. Thanks for the pointers.
Quoted 5 lines
https://outflux.net/blog/archives/2016/09/30/security-things-in-linux-v4-6/
I think to really dial in this protection so it is more like
PAX_MEMORY_SANITIZE, we'd need to do the following:
Quoted 8 lines
- adjust the slab poisoning to do zeroing (upstream's slab poisoning uses a different value)
- consolidate the CONFIG and run-time cmdlines to give us a single "do the right thing" CONFIG
- identify frequently-freed slabs that are less security-sensitive and whitelist them from the poisoning (as done byPAX_MEMORY_SANITIZE)
- add kernel command-line argument to control poisoning (as done byPAX_MEMORY_SANITIZE)
I am looking in to the work done by Laura and PAX team. I'll get back to you on this soon.
Quoted 2 lines
-Kees
Signature · 1 line
Vaishali