perf: bogus correlation of kernel symbols
Hi all,
I would have appreciated a CC on this one, as the author of the feature that got disabled.
Quoted 7 lines · Dave Jones
Quoted 6 lines · Ingo Molnar
Quoted 3 lines
Dunno, i would not couple them necessarily - certain users might still have access to kernel symbols via some other channel - for example the System.map.
Quoted 8 lines
Quoted 4 lines
That always made this security by obscurity feature seem pointless for the bulk of users to me. Given the majority are going to be running distro kernels, anyone can find those addresses easily no matter how hard we hide them on the running system.
I certainly agree and made that argument as well, in the original thread(s) about
/proc/kallsyms.
I agree about the fact that kptr_restrict is an incomplete security feature. However, I disagree that it lacks usefulness entirely. Virtually every public kernel exploit in the past year leverages /proc/kallsyms or other kernel address leakage to target an attack. I'm not ignorant of the fact that it's trivial to fingerprint distribution kernels in the absence of this information, but the reality is, a huge portion of real life exploit attempts leverage pre-fabricated exploits and are conducted by people who lack the ability to adjust exploits to target a specific running kernel. Even though this is trivial to sidestep if you know what you're doing, this extra little step may mean some script kiddie can't root some poor sysadmin's machine, and that's a win. In addition, when more powerful randomization is hopefully introduced, blocking access to these pointers will be more essential in preserving the lack of knowledge of the location of kernel internals.
But this is all just for the record I suppose, since it seems that ship has sailed.
Quoted 46 lines
Quoted 3 lines
Unless we were somehow introduced randomness into where we unpack the kernel each boot, and using System.map as a table of offsets instead of absolute addresses.
Correct. This security feature is IMO only solving a tiny fraction of the problem and is thus in fact hindering the implementation of a *real* layer of protection of kernel absolute addresses:
The x86 kernel is relocatable, so slightly randomizing the position of the kernel would be feasible with no overhead on the vast majority of exising distro installs, with just an updated kernel.
When exposing randomized RIPs to user-space we could recalculate all RIPs back to the
0xffffffff80000000base, so oopses would have the usual non-randomized form:[ 32.946003] IP: [<ffffffff80222521>] get_cur_val+0xcc/0x106 [ 32.946003] PGD 0 [ 32.946003] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC [ 32.946003] last sysfs file: [ 32.946003] CPU 1 [ 32.946003] Pid: 1, comm: swapper Tainted: G W 2.6.29-rc1-00190-g37a76bd #10 [ 32.946003] RIP: 0010:[<ffffffff80222521>] [<ffffffff80222521>] get_cur_val+0xcc/0x106 [ 32.946003] RSP: 0018:ffff88003f977b80 EFLAGS: 00010202 [ 32.946003] RAX: 0000000000000001 RBX: ffff8800029c8c80 RCX: 0000000000000008 [ 32.946003] RDX: 0000000000000000 RSI: ffffffff80ce0100 RDI: 0000000000000000 [ 32.946003] RBP: ffff88003f977bd0 R08: 0000000000000004 R09: 0000000000000040 [ 32.946003] R10: 0000000000000060 R11: 0000000081363fa8 R12: ffffffff81c4f0e0 [ 32.946003] R13: ffffffff80ce0100 R14: ffff88003c888a00 R15: 0000000000000000 [ 32.946003] FS: 0000000000000000(0000) GS:ffff88003f802c00(0000) knlGS:0000000000000000 [ 32.946003] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [ 32.946003] CR2: 0000000000000000 CR3: 0000000000201000 CR4: 00000000000006e0 [ 32.946003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 32.946003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 32.946003] Process swapper (pid: 1, threadinfo ffff88003f976000, task ffff88003f978000) [ 32.946003] Stack:Likewise,
/proc/kallsymscould pass these addresses as well and the perf call-chain code and other places that sample RIPs could easily convert them to the constant address as well.We'd still leak some information like the relative position of symbols from each other (this can be useful to certain classes of attacks), but we could pretty effectively hide the absolute location of the kernel - which is the most valuable piece of information -.
Quoted 8 lines
Then the random base has to be protected: i.e. all information leaks of raw kernel RIPs have to be plugged. The nice thing is that this will happen as *bugfixes*: randomized RIPs will not be useful for anything, so any
tools/peoplewho rely on them will notice it immediately.I think *that* would be a maintainable and complete security feature to truly hide the exact location of the kernel image.
kptr_restrictis not.
I want this feature, as I think it is far more useful and important. This has been mentioned before, but no one has stepped up to actually do it. Unfortunately, I lack the necessary knowledge of the relevant code to do it properly. What's the best way to make this feature a reality?
Regards,
Dan
Quoted 3 lines
Thanks,
Ingo
Quoted 4 lines · Dan Rosenberg
Hi all,
I would have appreciated a CC on this one, as the author of the feature that got disabled.
That's true and sorry about it: i could have sworn the author was Cc:-ed but confused you with Kees ...
Quoted 15 lines
Quoted 7 lines · Dave Jones
Quoted 6 lines · Ingo Molnar
Quoted 3 lines
Dunno, i would not couple them necessarily - certain users might still have access to kernel symbols via some other channel - for example the System.map.
Quoted 7 lines
Quoted 4 lines
That always made this security by obscurity feature seem pointless for the bulk of users to me. Given the majority are going to be running distro kernels, anyone can find those addresses easily no matter how hard we hide them on the running system.
I certainly agree and made that argument as well, in the original thread(s) about
/proc/kallsyms.
Quoted 13 lines
I agree about the fact that
kptr_restrictis an incomplete security feature. However, I disagree that it lacks usefulness entirely. Virtually every public kernel exploit in the past year leverages/proc/kallsymsor other kernel address leakage to target an attack. I'm not ignorant of the fact that it's trivial to fingerprint distribution kernels in the absence of this information, but the reality is, a huge portion of real life exploit attempts leverage pre-fabricated exploits and are conducted by people who lack the ability to adjust exploits to target a specific running kernel. Even though this is trivial to sidestep if you know what you're doing, this extra little step may mean some script kiddie can't root some poor sysadmin's machine, and that's a win. In addition, when more powerful randomization is hopefully introduced, blocking access to these pointers will be more essential in preserving the lack of knowledge of the location of kernel internals.
Well, but lets think it through further: what happens when we do such a change?
- Script kiddies get thwarted for a few weeks.
- Script authors will laugh and will update their scripts to query rpmfind.net or other package servers for symbol info.
- After that transition all the exploits will continue to work. They might in fact be more robust because they can specifically target only package versions that are known to be exploitable.
- *Useful* tools that do not try to harm the system will stay less useful forever and that's permanent collateral damage.
I.e. we would have driven the development of *attack* tools to be even more harmful and will have hurt *useful* tools. Is this really what we want?
Quoted 2 lines
But this is all just for the record I suppose, since it seems that ship has sailed.
We can still revert the revert as well although indeed it is not very common.
Quoted 51 lines
Quoted 51 lines
Quoted 3 lines
Unless we were somehow introduced randomness into where we unpack the kernel each boot, and using System.map as a table of offsets instead of absolute addresses.
Correct. This security feature is IMO only solving a tiny fraction of the problem and is thus in fact hindering the implementation of a *real* layer of protection of kernel absolute addresses:
The x86 kernel is relocatable, so slightly randomizing the position of the kernel would be feasible with no overhead on the vast majority of exising distro installs, with just an updated kernel.
When exposing randomized RIPs to user-space we could recalculate all RIPs back to the
0xffffffff80000000base, so oopses would have the usual non-randomized form:[ 32.946003] IP: [<ffffffff80222521>] get_cur_val+0xcc/0x106 [ 32.946003] PGD 0 [ 32.946003] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC [ 32.946003] last sysfs file: [ 32.946003] CPU 1 [ 32.946003] Pid: 1, comm: swapper Tainted: G W 2.6.29-rc1-00190-g37a76bd #10 [ 32.946003] RIP: 0010:[<ffffffff80222521>] [<ffffffff80222521>] get_cur_val+0xcc/0x106 [ 32.946003] RSP: 0018:ffff88003f977b80 EFLAGS: 00010202 [ 32.946003] RAX: 0000000000000001 RBX: ffff8800029c8c80 RCX: 0000000000000008 [ 32.946003] RDX: 0000000000000000 RSI: ffffffff80ce0100 RDI: 0000000000000000 [ 32.946003] RBP: ffff88003f977bd0 R08: 0000000000000004 R09: 0000000000000040 [ 32.946003] R10: 0000000000000060 R11: 0000000081363fa8 R12: ffffffff81c4f0e0 [ 32.946003] R13: ffffffff80ce0100 R14: ffff88003c888a00 R15: 0000000000000000 [ 32.946003] FS: 0000000000000000(0000) GS:ffff88003f802c00(0000) knlGS:0000000000000000 [ 32.946003] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [ 32.946003] CR2: 0000000000000000 CR3: 0000000000201000 CR4: 00000000000006e0 [ 32.946003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 32.946003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 32.946003] Process swapper (pid: 1, threadinfo ffff88003f976000, task ffff88003f978000) [ 32.946003] Stack:Likewise,
/proc/kallsymscould pass these addresses as well and the perf call-chain code and other places that sample RIPs could easily convert them to the constant address as well.We'd still leak some information like the relative position of symbols from each other (this can be useful to certain classes of attacks), but we could pretty effectively hide the absolute location of the kernel - which is the most valuable piece of information -.
Then the random base has to be protected: i.e. all information leaks of raw kernel RIPs have to be plugged. The nice thing is that this will happen as *bugfixes*: randomized RIPs will not be useful for anything, so any
tools/peoplewho rely on them will notice it immediately.
Quoted 8 lines
Quoted 3 lines
I think *that* would be a maintainable and complete security feature to truly hide the exact location of the kernel image.
kptr_restrictis not.I want this feature, as I think it is far more useful and important. This has been mentioned before, but no one has stepped up to actually do it. Unfortunately, I lack the necessary knowledge of the relevant code to do it properly. What's the best way to make this feature a reality?
Agreed, it would be a very useful feature.
I'd suggest to implement it along the lines of:
- First check whether grsecurity or PAX has this implemented already via the relocation facility - they are pretty good at being paranoid so i'd be surprised if they didnt think of this already! :-)
- If not then have a look at CONFIG_RELOCATABLE and to relocate the kernel binary intentionally via a hardcoded parameter. Just see whether you can do it and whether it works as you expect it. Check /proc/kallsyms changing after your patch. Enjoy the kernel still working ;-)
- Then promote it to a boot parameter - this way you'll be able to tell whether there's any hidden build-time assumptions about relocation position. (there really shouldnt be any given that kexec works just fine - but i'd suggest this step just in case.)
- Then promote that hack to be a randomized parameter. Marvel at a different, randomized /proc/kallsyms output at every bootup and enjoy the still working kernel!
- Then look at all RIP outputs (thanks to your prior efforts they are now mostly concentrated in the vprints code!) and reverse apply the random offset before it's exported into user-space. wchan, etc. Marvel at the constant /proc/kallsyms output, fully knowing that the *real* addresses are randomized.
- Please do not forget to transfer perf RIPs and callchains and marvel at the well working 'perf top' output.
At that point the feature will be highly useful already IMO. Remaining work will be to think through and close down all remaining avenues of RIP leakage.
At this point kptr_restrict will be a lot less relevant - the symbols will expose offsets (so it's not totally unhelpful to attackers) but not the real absolute addresses.
Unless i'm missing some particularly difficult roadblock, which is possible.
If you try this then please keep us posted at every step above, even if your patches are not fully working and useful yet. Maybe some other details/ideas/suggestions will arise at that point.
Thanks,
Ingo
Quoted 38 lines · Ingo Molnar
Agreed, it would be a very useful feature.
I'd suggest to implement it along the lines of:
- First check whether grsecurity or PAX has this implemented already via the relocation facility - they are pretty good at being paranoid so i'd be surprised if they didnt think of this already! :-)
- If not then have a look at
CONFIG_RELOCATABLEand to relocate the kernel binary intentionally via a hardcoded parameter. Just see whether you can do it and whether it works as you expect it. Check/proc/kallsymschanging after your patch. Enjoy the kernel still working ;-)- Then promote it to a boot parameter - this way you'll be able to tell whether there's any hidden build-time assumptions about relocation position. (there really shouldnt be any given that kexec works just fine - but i'd suggest this step just in case.)
- Then promote that hack to be a randomized parameter. Marvel at a different, randomized
/proc/kallsymsoutput at every bootup and enjoy the still working kernel!- Then look at all RIP outputs (thanks to your prior efforts they are now mostly concentrated in the vprints code!) and reverse apply the random offset before it's exported into user-space. wchan, etc. Marvel at the constant
/proc/kallsymsoutput, fully knowing that the *real* addresses are randomized.- Please do not forget to transfer perf RIPs and callchains and marvel at the well working 'perf top' output.
At that point the feature will be highly useful already IMO. Remaining work will be to think through and close down all remaining avenues of RIP leakage.
At this point
kptr_restrictwill be a lot less relevant - the symbols will expose offsets (so it's not totally unhelpful to attackers) but not the real absolute addresses.
Quoted 6 lines
Unless i'm missing some particularly difficult roadblock, which is possible.
If you try this then please keep us posted at every step above, even if your patches are not fully working and useful yet. Maybe some other details/ideas/suggestions will arise at that point.
Thanks for the detailed response. I will attempt to go down this road, and will keep people posted with my progress.
-Dan
Quoted 3 lines
Thanks,
Ingo
Quoted 99 lines
Quoted 52 lines
Quoted 52 lines
Quoted 4 lines
Unless we were somehow introduced randomness into where we unpack the kernel each boot, and using System.map as a table of offsets instead of absolute addresses.
Correct. This security feature is IMO only solving a tiny fraction of the problem and is thus in fact hindering the implementation of a *real* layer of protection of kernel absolute addresses:
The x86 kernel is relocatable, so slightly randomizing the position of the kernel would be feasible with no overhead on the vast majority of exising distro installs, with just an updated kernel.
When exposing randomized RIPs to user-space we could recalculate all RIPs back to the
0xffffffff80000000base, so oopses would have the usual non-randomized form:[ 32.946003] IP: [<ffffffff80222521>] get_cur_val+0xcc/0x106 [ 32.946003] PGD 0 [ 32.946003] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC [ 32.946003] last sysfs file: [ 32.946003] CPU 1 [ 32.946003] Pid: 1, comm: swapper Tainted: G W 2.6.29-rc1-00190-g37a76bd #10 [ 32.946003] RIP: 0010:[<ffffffff80222521>] [<ffffffff80222521>] get_cur_val+0xcc/0x106 [ 32.946003] RSP: 0018:ffff88003f977b80 EFLAGS: 00010202 [ 32.946003] RAX: 0000000000000001 RBX: ffff8800029c8c80 RCX: 0000000000000008 [ 32.946003] RDX: 0000000000000000 RSI: ffffffff80ce0100 RDI: 0000000000000000 [ 32.946003] RBP: ffff88003f977bd0 R08: 0000000000000004 R09: 0000000000000040 [ 32.946003] R10: 0000000000000060 R11: 0000000081363fa8 R12: ffffffff81c4f0e0 [ 32.946003] R13: ffffffff80ce0100 R14: ffff88003c888a00 R15: 0000000000000000 [ 32.946003] FS: 0000000000000000(0000) GS:ffff88003f802c00(0000) knlGS:0000000000000000 [ 32.946003] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [ 32.946003] CR2: 0000000000000000 CR3: 0000000000201000 CR4: 00000000000006e0 [ 32.946003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 32.946003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 32.946003] Process swapper (pid: 1, threadinfo ffff88003f976000, task ffff88003f978000) [ 32.946003] Stack:Likewise,
/proc/kallsymscould pass these addresses as well and the perf call-chain code and other places that sample RIPs could easily convert them to the constant address as well.We'd still leak some information like the relative position of symbols from each other (this can be useful to certain classes of attacks), but we could pretty effectively hide the absolute location of the kernel - which is the most valuable piece of information -.
Then the random base has to be protected: i.e. all information leaks of raw kernel RIPs have to be plugged. The nice thing is that this will happen as *bugfixes*: randomized RIPs will not be useful for anything, so any
tools/peoplewho rely on them will notice it immediately.Quoted 8 lines
Quoted 3 lines
I think *that* would be a maintainable and complete security feature to truly hide the exact location of the kernel image.
kptr_restrictis not.I want this feature, as I think it is far more useful and important. This has been mentioned before, but no one has stepped up to actually do it. Unfortunately, I lack the necessary knowledge of the relevant code to do it properly. What's the best way to make this feature a reality?
Agreed, it would be a very useful feature.
I'd suggest to implement it along the lines of:
- First check whether grsecurity or PAX has this implemented already via the relocation facility - they are pretty good at being paranoid so i'd be surprised if they didnt think of this already! :-)
- If not then have a look at
CONFIG_RELOCATABLEand to relocate the kernel binary intentionally via a hardcoded parameter. Just see whether you can do it and whether it works as you expect it. Check/proc/kallsymschanging after your patch. Enjoy the kernel still working ;-)- Then promote it to a boot parameter - this way you'll be able to tell whether there's any hidden build-time assumptions about relocation position. (there really shouldnt be any given that kexec works just fine - but i'd suggest this step just in case.)
- Then promote that hack to be a randomized parameter. Marvel at a different, randomized
/proc/kallsymsoutput at every bootup and enjoy the still working kernel!- Then look at all RIP outputs (thanks to your prior efforts they are now mostly concentrated in the vprints code!) and reverse apply the random offset before it's exported into user-space. wchan, etc. Marvel at the constant
/proc/kallsymsoutput, fully knowing that the *real* addresses are randomized.- Please do not forget to transfer perf RIPs and callchains and marvel at the well working 'perf top' output.
At that point the feature will be highly useful already IMO. Remaining work will be to think through and close down all remaining avenues of RIP leakage.
At this point
kptr_restrictwill be a lot less relevant - the symbols will expose offsets (so it's not totally unhelpful to attackers) but not the real absolute addresses.
Quoted 6 lines
Unless i'm missing some particularly difficult roadblock, which is possible.
If you try this then please keep us posted at every step above, even if your patches are not fully working and useful yet. Maybe some other details/ideas/suggestions will arise at that point.
I was able to boot a relocatable kernel with the decompression location at a hard-coded offset without too much trouble. Everything seems to work fine.
However, it occurred to me that even if the kernel image's base address were randomized at boot, assuming a binary distro kernel it would still be possible to sidt the address of the IDT and calculate symbol offsets relative to that. Any thoughts on how to avoid that? Seems difficult. Another hurdle will be to find a reasonable source of entropy that early in the boot process.
-Dan
Quoted 3 lines
Thanks,
Ingo
Quoted 2 lines · Dan Rosenberg
I was able to boot a relocatable kernel with the decompression location at a hard-coded offset without too much trouble. Everything seems to work fine.
Nice!
Quoted 6 lines
However, it occurred to me that even if the kernel image's base address were randomized at boot, assuming a binary distro kernel it would still be possible to sidt the address of the IDT and calculate symbol offsets relative to that. Any thoughts on how to avoid that? Seems difficult. Another hurdle will be to find a reasonable source of entropy that early in the boot process.
I do not think it's an issue.
If an attacker can execute arbitrary privileged instructions like SIDT then it's game over. There's plenty of CPU state, the IDT, GDT, various MSRs that would tell roughly where the kernel is, etc.
The attack randomization protects against is when the attacker has a limited amount of control over a stack return address (due to a buffer overflow for example) and can redirect kernel execution to some 'interesting' place that allows more control. With SMEP and kernel image randomization this would be rather difficult to pull off: the kernel wont jump to a pre-prepared user-space shellcode buffer (due to SMEP) while the location of already existing, executable, supervisor-privileged pages is randomized.
So when you have implemented this i'd suggest enabling CONFIG_X86_PTDUMP=y to get access to a dump of all pagetables, in the /debug/kernel_page_tables file. There you can check every single executable, kernel-privileged mapping on a live system and make sure it's not easily discovered.
Thanks,
Ingo
Quoted 14 lines · Ingo Molnar
Quoted 2 lines · Dan Rosenberg
I was able to boot a relocatable kernel with the decompression location at a hard-coded offset without too much trouble. Everything seems to work fine.
Nice!
Quoted 6 lines
However, it occurred to me that even if the kernel image's base address were randomized at boot, assuming a binary distro kernel it would still be possible to sidt the address of the IDT and calculate symbol offsets relative to that. Any thoughts on how to avoid that? Seems difficult. Another hurdle will be to find a reasonable source of entropy that early in the boot process.
Quoted 6 lines
I do not think it's an issue.
If an attacker can execute arbitrary privileged instructions like SIDT then it's game over. There's plenty of CPU state, the IDT, GDT, various MSRs that would tell roughly where the kernel is, etc.
Except that SIDT isn't a privilege instruction, it's accessible as ring 3.
Quoted 8 lines
The attack randomization protects against is when the attacker has a limited amount of control over a stack return address (due to a buffer overflow for example) and can redirect kernel execution to some 'interesting' place that allows more control. With SMEP and kernel image randomization this would be rather difficult to pull off: the kernel wont jump to a pre-prepared user-space shellcode buffer (due to SMEP) while the location of already existing, executable, supervisor-privileged pages is randomized.
Yes, all true, except are you specifically considering remote-only attack vectors?
Quoted 5 lines
So when you have implemented this i'd suggest enabling
CONFIG_X86_PTDUMP=yto get access to a dump of all pagetables, in the /debug/kernel_page_tables file. There you can check every single executable, kernel-privileged mapping on a live system and make sure it's not easily discovered.
I'll do this too, but first I'd like to address the above.
Thanks,
Dan
Quoted 3 lines
Thanks,
Ingo
Quoted 15 lines · Dan Rosenberg
Quoted 14 lines · Ingo Molnar
Quoted 2 lines · Dan Rosenberg
I was able to boot a relocatable kernel with the decompression location at a hard-coded offset without too much trouble. Everything seems to work fine.
Nice!
Quoted 6 lines
However, it occurred to me that even if the kernel image's base address were randomized at boot, assuming a binary distro kernel it would still be possible to sidt the address of the IDT and calculate symbol offsets relative to that. Any thoughts on how to avoid that? Seems difficult. Another hurdle will be to find a reasonable source of entropy that early in the boot process.
Quoted 7 lines
Quoted 5 lines
I do not think it's an issue.
If an attacker can execute arbitrary privileged instructions like SIDT then it's game over. There's plenty of CPU state, the IDT, GDT, various MSRs that would tell roughly where the kernel is, etc.
Except that SIDT isn't a privilege instruction, it's accessible as ring 3.
Oops, stupid me :-/
We need to allocate the IDT dynamically: just kmalloc() it, update idt_descr and do a load_idt(). Double check places that modify idt_descr or use idt_table.
Note, you could do this as a side effect of a nice performance optimization:
would you be interested in allocating it in the percpu area, using percpu_alloc()? That way the IDT is distributed between CPUs - this has scalability advantages on NUMA systems and maybe even on SMP.
Quoted 11 lines
Quoted 8 lines
The attack randomization protects against is when the attacker has a limited amount of control over a stack return address (due to a buffer overflow for example) and can redirect kernel execution to some 'interesting' place that allows more control. With SMEP and kernel image randomization this would be rather difficult to pull off: the kernel wont jump to a pre-prepared user-space shellcode buffer (due to SMEP) while the location of already existing, executable, supervisor-privileged pages is randomized.
Yes, all true, except are you specifically considering remote-only attack vectors?
No, unprivileged local user, so yes, the IDT address has to be protected.
Thanks,
Ingo
Quoted 9 lines · Ingo Molnar
We need to allocate the IDT dynamically: just
kmalloc()it, updateidt_descrand do aload_idt(). Double check places that modifyidt_descror useidt_table.Note, you could do this as a side effect of a nice performance optimization:
would you be interested in allocating it in the percpu area, usingpercpu_alloc()? That way the IDT is distributed between CPUs - this has scalability advantages on NUMA systems and maybe even on SMP.
Any suggestions on when this allocation should take place? I'm hesitant to touch anything in arch/x86/kernel/head_32.S, where the IDT is setup and lidt idt_descr is called (on x86-32 anyway). That means at some point I'd have to copy the table into a region allocated with alloc_percpu() and set up a new descriptor. Seems like this should happen before IRQ is enabled, but I'm not sure about the best place.
Also, I'd still welcome suggestions on generating entropy so early in the boot process as to randomize the location at which the kernel is decompressed.
On a related note, would there be obstacles to marking the IDT as read-only?
Thanks,
Dan
Quoted 4 lines · Dan Rosenberg
Also, I'd still welcome suggestions on generating entropy so early in the boot process as to randomize the location at which the kernel is decompressed.
The fundamental problem with the whole kernel address randomization is sadly totally unrelated to any of the small details.
There's a *big* detail that makes it hard: there's only a few bits of randomness we can add to the address. The kernel base address ends up having various fundamental limitations (cacheline alignment for the code, and we have several segments that require page alignment), so you really can't realistically do more than something like 8-12 bits of address randomization.
Which means that once you have a vmlinux image (say, because it's a standard distro kernel), you only need to try your exploit a few hundred times. That can be done quickly enough that no MIS person will ever have time to react to the attack.
Sure, it will likely leave some hints around (oopses etc), but still..
Quoted 2 lines
On a related note, would there be obstacles to marking the IDT as read-only?
We do that for the F00F bug workaround. But while the linear address is read-only, the IDT can still be accessed read-write through the physical address through the normal 1:1 mapping.
Regardless, the virtual mapping trick (independently of whether it's read-only or not) can be used to avoid exposing the *actual* address of the IDT of the kernel, and would hide the kernel load address details. However, it does make traps slightly slower, if they cannot use the 1:1 mapping with large pages for the IDT access and thus cause more TLB pressure. Of course, in many situations we probably end up not having large pages for the kernel anyway, so..
As a result, we do that F00F bug workaround _only_ if we're actually running on a CPU with the FOOF bug.
Linus
Quoted 17 lines · Linus Torvalds
Quoted 4 lines · Dan Rosenberg
Also, I'd still welcome suggestions on generating entropy so early in the boot process as to randomize the location at which the kernel is decompressed.
The fundamental problem with the whole kernel address randomization is sadly totally unrelated to any of the small details.
There's a *big* detail that makes it hard: there's only a few bits of randomness we can add to the address. The kernel base address ends up having various fundamental limitations (cacheline alignment for the code, and we have several segments that require page alignment), so you really can't realistically do more than something like 8-12 bits of address randomization.
Quoted 6 lines
Which means that once you have a vmlinux image (say, because it's a standard distro kernel), you only need to try your exploit a few hundred times. That can be done quickly enough that no MIS person will ever have time to react to the attack.
Sure, it will likely leave some hints around (oopses etc), but still..
Certain flaws will present that way, yes. Others will take the entire system down on the first missed address guess. Many times, ASLR will give a statistical advantage to the defender. As a result it has value, even if it's not perfect.
-Kees
Signature · 2 lines
Kees Cook Ubuntu Security Team
Quoted 5 lines · Linus Torvalds
There's a *big* detail that makes it hard: there's only a few bits of randomness we can add to the address. The kernel base address ends up having various fundamental limitations (cacheline alignment for the code, and we have several segments that require page alignment), so you really can't realistically do more than something like 8-12 bits of address randomization.
Yeah, i tried to address this issue in my first mail: basically just a few bits would already make a big difference in practice: even a *single* bit of randomness makes an exploit crash 50% of the time - at which point the attack stops being stealth.
8 bits would be a lot.
So i think this is really realistic, even if a brute force, networked attack can successfully attack 1 out of 256, 512 or 1024 boxes. Even for the worm cas the networked attack would not scale very well.
Quoted 7 lines
Regardless, the virtual mapping trick (independently of whether it's read-only or not) can be used to avoid exposing the *actual* address of the IDT of the kernel, and would hide the kernel load address details. However, it does make traps slightly slower, if they cannot use the 1:1 mapping with large pages for the IDT access and thus cause more TLB pressure. Of course, in many situations we probably end up not having large pages for the kernel anyway, so..
We could put per CPU IDTs into the percpu area if that improves performance.
This might help on NUMA: on NUMA only one node has the IDT local, the others will take a remote DRAM access every time they miss the IDT - and the IDT could easily be missed if there are no IRQs or traps for a long time (say CPU-bound user-space processing).
There may also be cases where an implicit locked access is generated to the IDT?
Thanks,
Ingo
Quoted 23 lines · Kees Cook
Quoted 22 lines · Linus Torvalds
Quoted 4 lines · Dan Rosenberg
Also, I'd still welcome suggestions on generating entropy so early in the boot process as to randomize the location at which the kernel is decompressed.
The fundamental problem with the whole kernel address randomization is sadly totally unrelated to any of the small details.
There's a *big* detail that makes it hard: there's only a few bits of randomness we can add to the address. The kernel base address ends up having various fundamental limitations (cacheline alignment for the code, and we have several segments that require page alignment), so you really can't realistically do more than something like 8-12 bits of address randomization.
Which means that once you have a vmlinux image (say, because it's a standard distro kernel), you only need to try your exploit a few hundred times. That can be done quickly enough that no MIS person will ever have time to react to the attack.
Quoted 7 lines
Quoted 1 line
Sure, it will likely leave some hints around (oopses etc), but still..
Certain flaws will present that way, yes. Others will take the entire system down on the first missed address guess. Many times, ASLR will give a statistical advantage to the defender. As a result it has value, even if it's not perfect.
At least one distro (Red Hat) ships with panic_on_oops enabled by default, so attackers don't get more than one chance. Likewise, vulnerabilities in interrupt context will only have one chance, as will any issue where failed exploitation prevents subsequent attempts, as is frequently the case due to failures to clean up locking primitives on an OOPS.
-Dan
Quoted 5 lines
-Kees
Signature · 2 lines
Kees Cook Ubuntu Security Team
Quoted 12 lines · Dan Rosenberg
Quoted 9 lines · Ingo Molnar
We need to allocate the IDT dynamically: just
kmalloc()it, updateidt_descrand do aload_idt(). Double check places that modifyidt_descror useidt_table.Note, you could do this as a side effect of a nice performance optimization:
would you be interested in allocating it in the percpu area, usingpercpu_alloc()? That way the IDT is distributed between CPUs - this has scalability advantages on NUMA systems and maybe even on SMP.
Quoted 6 lines
Any suggestions on when this allocation should take place? I'm hesitant to touch anything in
arch/x86/kernel/head_32.S, where the IDT is setup and lidtidt_descris called (on x86-32 anyway). That means at some point I'd have to copy the table into a region allocated withalloc_percpu()and set up a new descriptor. Seems like this should happen before IRQ is enabled, but I'm not sure about the best place.
I think there's a static percpu area that can be used pretty early on.
The boot IDT can be marked __initdata so its space wont be wasted.
The thing is, until SMP is not initialized the boot IDT can be kept. So i'd suggest allocating per CPU IDTs after memory has initialized. For that a pretty good place is trap_init(): there we already have the page allocator initialized and probably the percpu allocator too. IDT allocation is also pretty naturally done in trap_init().
Quoted 5 lines
Also, I'd still welcome suggestions on generating entropy so early in the boot process as to randomize the location at which the kernel is decompressed.
On a related note, would there be obstacles to marking the IDT as read-only?
The cost is that its access TLB may change from a 2MB TB to a 4K TLB. We generally try to keep critical data structures in 2MB mapped areas.
But this is really hard to measure (you'd have to have a borderline workload where the loss of a single 4K TLB is measurable) so i'd suggest splitting this from the randomization step.
Thanks,
Ingo
Quoted 5 lines · Dan Rosenberg
At least one distro (Red Hat) ships with
panic_on_oopsenabled by default, so attackers don't get more than one chance. Likewise, vulnerabilities in interrupt context will only have one chance, as will any issue where failed exploitation prevents subsequent attempts, as is frequently the case due to failures to clean up locking primitives on an OOPS.
So it's basically a last line of defense: the attacker has to assume the risk of the attack being detected.
That has a chilling effect on some types of attacks: especially those where the attacker goes against a high value target with a zero day kernel exploit. Risking a crash does not just mean possibly alerting the target, but also means possibly losing the zero-day exploit - if that oops log gets to a kernel developer who starts wondering about the weird backtrace.
Thanks,
Ingo
Quoted 2 lines · Linus Torvalds
Sure, it will likely leave some hints around (oopses etc), but still..
as an admin running a large site, I'd love to have evidence around that strange things were happening, especially prior to the exploit running.
no, I probably can't take manual action fast enough to prevent the initial exploit, but I can alarm on the failed attacks and potentially react fast enough to prevent the attacker from going further into my network.
David Lang
Quoted 10 lines · Ingo Molnar
Quoted 8 lines · Dan Rosenberg
Quoted 7 lines · Ingo Molnar
Quoted 2 lines · Dan Rosenberg
I was able to boot a relocatable kernel with the decompression location at a hard-coded offset without too much trouble. Everything seems to work fine.
Nice!
A progress update, and a number of questions.
The randomization itself is working fine with the following hack in arch/x86/boot/compressed/head_32.S:
#ifdef CONFIG_RELOCATABLE movl %ebp, %ebx #ifdef CONFIG_RANDOMIZE_BASE rdtsc shll $0x8, %eax andl $0x3ffffff, %eax addl %eax, %ebx #endif movl BP_kernel_alignment(%esi), %eax decl %eax addl %eax, %ebx notl %eax andl %eax
#endif
That brings me to my first two questions:
1. Is it ok to assume the existence of rdtsc? If not, what are other ways of gathering entropy early in the boot process? If this is the approach that's going to be taken, system uptime potentially becomes useful for attackers. Any thoughts on how to address this?
2. The current default physical alignment is 16mb as a result of this patch:
Having 16mb alignment greatly restricts the amount of usable entropy for randomization. It seems an alternate solution to the problem the above patch addresses is reverting back to 1mb alignment (or 2/4 mb if that has performance benefits) and enforcing a 16mb minimum physical start for relocatable kernels by bumping it up in the boot code if necessary. Would this be possible? I'd like to avoid requiring distros to touch CONFIG_PHYSICAL_ALIGN (and risk breaking things) in order to have more useful randomization.
A few more questions arose during my efforts:
3. The current hack I'm using to determine the offset to reverse apply to symbol output looks something like this:
unsigned long kptr_adjust = ((unsigned long)_text & ~(CONFIG_PHYSICAL_ALIGN-1)) - (PAGE_OFFSET + CONFIG_PHYSICAL_START);
Is it safe to assume that kernel .text is the first thing in a decompressed kernel image? If not, any other suggestions? It seemed easier to compute this in the decompressed kernel at runtime rather than try to figure out a way to pass the actual decompression address from the boot stage to the main kernel.
4. What kind of behavior do people want with %pK and kptr_restrict? If possible, I'd like to find a way that perf users can have the benefits of this feature and still have usable symbol support. However, module symbols are a bit tricky, since they're not being relocated with the rest of the kernel, and it doesn't seem meaningful to reverse-apply the same offset in module symbol output. Perhaps a separate format specifier should be introduced to differentiate symbols that need to be offset?
Basically, we've got kernel .text symbols, module symbols, and dynamic kernel pointers, and I'm not sure with what granularity people are interested in hiding them. It seems perf at least wants more than "all or nothing".
5. I'd like some more opinions on moving the IDT. So far, the two options are using a fixed mapping similar to the F00F bug fix, and allocating it percpu at runtime.
Looking forward to feedback, criticism, disgust, etc.
Regards,
Dan
Quoted 12 lines · Tony Luck
On Sun, May 22, 2011 at 11:45 AM, Dan Rosenberg <[email protected]> wrote:
1. Is it ok to assume the existence of rdtsc? If not, what are other
ways of gathering entropy early in the boot process? If this is the
approach that's going to be taken, system uptime potentially becomes
useful for attackers. Any thoughts on how to address this?
Quoted 2 lines
There is a cpuid bit to tell you whether the processor supports rdtsc.
This might be worth checking, but it also might just be easier to clearly document in the Kconfig that the feature requires rdtsc. I'll play with it a bit.
Quoted 7 lines
In the cold boot case, I'd worry about whether rdtsc was all that random,
it counts from zero when the processors come out of cold reset, and things should be quite deterministic up to when the kernel loads; especially if you have a solid state boot drive rather than the old spinning kind.
The question really becomes whether it's "good enough". It's certainly not cryptographic randomness, but it will produce different results on different boots depending on a variety of factors, including out-of-order instruction execution, multi-CPU systems, and differences in CPU models. Even though a single machine may be slightly more likely to produce certain offsets more often, it seems to me that it would still accomplish the goal, because it's highly unlikely that an attacker would be able to perform the statistical analysis necessary to figure that out.
Quoted 3 lines
Sometime soon you'll have "rdrand" available (check a different cpuid bit).
This of course is highly preferable, but I'd rather implement a solution that's widely supported in hardware today. Perhaps further down the road it could be switched.
Quoted 3 lines
-Tony
Thanks for the feedback.
-Dan
Quoted 8 lines · Dan Rosenberg
Quoted 3 lines
Sometime soon you'll have "rdrand" available (check a different cpuid bit).
This of course is highly preferable, but I'd rather implement a solution that's widely supported in hardware today. Perhaps further down the road it could be switched.
Use the better one that is actually available.
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Quoted 13 lines · Dan Rosenberg
Quoted 12 lines · Tony Luck
On Sun, May 22, 2011 at 11:45 AM, Dan Rosenberg <[email protected]> wrote:
1. Is it ok to assume the existence of rdtsc? If not, what are other
ways of gathering entropy early in the boot process? If this is the
approach that's going to be taken, system uptime potentially becomes
useful for attackers. Any thoughts on how to address this?
Quoted 6 lines
Quoted 2 lines
There is a cpuid bit to tell you whether the processor supports rdtsc.
This might be worth checking, but it also might just be easier to clearly document in the
Kconfigthat the feature requires rdtsc. I'll play with it a bit.
All modern x86 boxes have an RDTSC so this not really a practical concern - other than not crashing old boxes that do not have an RDTSC.
In theory there's other sources of entropy: we could read out the RTC CMOS as well. We could also read the current CPU frequency and fan speeds - those have components of thermal noise as well. Obviously it's hard to read this out early during bootup, when system devices are not enumerated yet.
What would be very useful is to print out the bootup RDTSC value for several bootups. I've done this on a testbox here, with real full reboots inbetween, putting the RDTSC printout very close to where you'd sample it for kernel image randomization:
[ 0.000000] RDTSC: 26615467048 [ 0.000000] RDTSC: 26527108278 [ 0.000000] RDTSC: 26464738628 [ 0.000000] RDTSC: 26554778708 [ 0.000000] RDTSC: 26441165788 [ 0.000000] RDTSC: 26555252088 [ 0.000000] RDTSC: 26431986988 [ 0.000000] RDTSC: 26521303608 [ 0.000000] RDTSC: 26497878018 [ 0.000000] RDTSC: 26455546968 [ 0.000000] RDTSC: 26467673718 [ 0.000000] RDTSC: 26460404758 [ 0.000000] RDTSC: 26496175038
Even visually there's *plenty* of real randomness in the bootup value of the cycle counter (look at the above pattern of numbers and squint), even during early bootup, on real hardware.
While the last few bits are non-random there's at least 10-20 bits of randomness on this (very simple) testbox - possibly more.
I've done a few tests on virtual hardware as well, KVM based:
[ 0.000000] RDTSC: 208122033 [ 0.000000] RDTSC: 200455104 [ 0.000000] RDTSC: 207258945
As expected there's even more randmness on virtual hardware than on real
hardware: virtual hardware will boot in a complex host CPU state and is so exposed to the non-determinisic CPU cache state.
So the cycle counter is plenty good for this.
[ I'd only worry about the cycle counter if a hardware platform is so
minimalistic that it boots up very quickly without exposing itself to natural sources of entropy. Eventually we'll get such systems, but right now the cycle counter is good enough. ]
Btw, we already rely on the cycle counter for early bootup randomness:boot_init_stack_canary() relies on it.
Quoted 5 lines
Quoted 1 line
Sometime soon you'll have "rdrand" available (check a different cpuid bit).
This of course is highly preferable, but I'd rather implement a solution that's widely supported in hardware today. Perhaps further down the road it could be switched.
Well, since entropy does not get reduced on addition of independent variables the right sequence is (pseudocode):
rnd = entropy_cycles(); rnd += entropy_rdrand(); rnd += entropy_RTC(); rnd += entropy_system();
That way systems that do not have RDRAND will still have the other ones as fallback.
Of course in your prototype you can use RDTSC as a first step, just make it easy to add other sources.
Thanks,
Ingo
Quoted 7 lines · Ingo Molnar
Well, since entropy does not get reduced on addition of independent variables the right sequence is (pseudocode):
rnd =
entropy_cycles();
rnd +=entropy_rdrand();
rnd +=entropy_RTC();
rnd +=entropy_system();
I think you mean concatenation rather than addition? Or perhaps XOR, or a hash? It's pretty easy to show that the addition of n random variables evenly distributed between [0, 1] converges to 1/2 n +- 1/sqrt(n) (or numbers to that effect), which gives an attacker better chances than they would otherwise if they target the center of the distribution.
But none of this is to detract from your main point, which still holds. Structuring it such that other sources of randomness can be included as available keeps options open.
~r.
Quoted 11 lines · Ray Lee
Quoted 7 lines · Ingo Molnar
Well, since entropy does not get reduced on addition of independent variables the right sequence is (pseudocode):
rnd =
entropy_cycles();
rnd +=entropy_rdrand();
rnd +=entropy_RTC();
rnd +=entropy_system();I think you mean concatenation rather than addition? Or perhaps XOR, or a hash? [...]
Yeah.
In this special case probably concatenation works the best: the above 4 random variables have total randomness probably less than 32 bits, so we want to create 4 tight random numbers and concatenate them.
[ XOR would destroy some fair amount of entropy because most of these random
variables have their randomness in their low bits, and a hash would probably lose about 2 bits and would also be slower. A hash would probably be safer and more robust though, if we mis-identify any of the random variables. ]
Thanks,
Ingo
Quoted 7 lines · Ingo Molnar
Well, since entropy does not get reduced on addition of independent variables the right sequence is (pseudocode):
rnd = entropy_cycles(); rnd += entropy_rdrand(); rnd += entropy_RTC(); rnd += entropy_system();
I'm having trouble convincing myself that RTC and cycles are truly independent variables.... ;)
Consider the case of a fixed-frequency CPU - if you know the time since boot, and the current RTC, and the current cycle count, you can work backwards to find the RTC and cycle count at boot. I'm not sure that a variable clockspeed helps all that much - an attacker can perhaps find a way to force the highest/ lowest CPU speed - or the system may even helpfully do it for the attacker - I've seen plenty of misconfigured laptops that force lowest supported CPU clockspeed on battery rather than race-to-idle.
Having said that, the 13 bootup rdtsc values you list *seem* to have on the order of 24-28 bits of entropy, and only the lowest-order bit seems to be non-random (the low-order byte of the 13 values are 28, b6, 44, 54, dc, 78, 2c, 38, 02, 58, 76, 16, and be). So rdtsc appears to be good enough for what we want here...
Quoted 11 lines · * [email protected] <[email protected]> wrote:
Quoted 7 lines · Ingo Molnar
Well, since entropy does not get reduced on addition of independent variables the right sequence is (pseudocode):
rnd = entropy_cycles(); rnd += entropy_rdrand(); rnd += entropy_RTC(); rnd += entropy_system();I'm having trouble convincing myself that RTC and cycles are truly independent variables.... ;)
Generally the RTC stores absolute time in seconds (it stores the date), while cycles start new when the CPU is reset.
So they are independent.
The question i think you are asking is whether the fact that we can observe current values of them after bootup can be used to figure out their value:
Quoted 3 lines
Consider the case of a fixed-frequency CPU - if you know the time since boot, and the current RTC, and the current cycle count, you can work backwards to find the RTC and cycle count at boot. [...]
Yes, you are correct, if you are local then the guessing the RTC to the second is probably possible.
Guessing the cycle counter's value will be hard: see the natural noise it has at a fixed instruction after bootup in the same-bzImage test i performed - with no IRQs having executed at all yet ...
The RTC is still reasonably noisy to external attackers though.
Quoted 5 lines
[...] I'm not sure that a variable clockspeed helps all that much - an attacker can perhaps find a way to force the highest/ lowest CPU speed - or the system may even helpfully do it for the attacker - I've seen plenty of misconfigured laptops that force lowest supported CPU clockspeed on battery rather than race-to-idle.
The tests i performed were on a fixed frequency system - the cycle counter was still largely random during early bootup.
Others should try it too - i've attached a simple patch. Maybe my system has more bootup noise than others.
Quoted 5 lines
Having said that, the 13 bootup rdtsc values you list *seem* to have on the order of 24-28 bits of entropy, and only the lowest-order bit seems to be non-random (the low-order byte of the 13 values are 28, b6, 44, 54, dc, 78, 2c, 38, 02, 58, 76, 16, and be). So rdtsc appears to be good enough for what we want here...
Yeah. And for cases that the rdtsc might be predictable for some weird reason (say it would be 0 on an old system with no RDTSC), the RTC would give some minimal fallback seed to make the canary at least not remotely guessable.
Thanks,
Ingo
--- init/main.c | 6 ++++++ 1 file changed, 6 insertions(+)
init/main.c +6 -0
@@ -472,6 +472,12 @@ asmlinkage void __init start_kernel(void*/ boot_init_stack_canary (); + { +u64 cycles =get_cycles (); + +printk ("RDTSC: %Ld / %08Lx\n" , cycles, cycles); + } +cgroup_init_early ();local_irq_disable ();