Skip to contentSkip to conversation

randomized placement of x86_64 vdso

DISCUSSION Nathan Lynch 4 participants 9 messages Apr 21, 2014 – Apr 30, 2014 x86

lore
Share & tools
lore
#0 lore raw

Hi x86/vdso people,

I've been working on adding a vDSO to 32-bit ARM, and Kees suggested I look at x86_64's algorithm for placing the vDSO at a randomized offset above the stack VMA. I found that when the stack top occupies the last slot in the PTE (is that the right term?), the vdso_addr routine returns an address below mm->start_stack, equivalent to (mm->start_stack & PAGE_MASK). For instance if mm->start_stack is 0x7fff3ffffc96, vdso_addr returns 0x7fff3ffff000.

Since the address returned is always already occupied by the stack, get_unmapped_area detects the collision and falls back to vm_unmapped_area. This results in the vdso being placed in the address space next to libraries etc. While this is generally unnoticeable and doesn't break anything, it does mean that the vdso is placed below the stack when there is actually room above the stack. To me it also seems uncomfortably close to placing the vdso in the way of downward expansion of the stack.

I don't have a patch because I'm not sure what the algorithm should be, but thought I would bring it up as vdso_addr doesn't seem to be behaving as intended in all cases.

Thanks,
Nathan

#1 lore raw
Quoted 19 lines · Nathan Lynch

Hi x86/vdso people,

I've been working on adding a vDSO to 32-bit ARM, and Kees suggested I look at x86_64's algorithm for placing the vDSO at a randomized offset above the stack VMA. I found that when the stack top occupies the last slot in the PTE (is that the right term?), the vdso_addr routine returns an address below mm->start_stack, equivalent to (mm->start_stack & PAGE_MASK). For instance if mm->start_stack is 0x7fff3ffffc96, vdso_addr returns 0x7fff3ffff000.

Since the address returned is always already occupied by the stack, get_unmapped_area detects the collision and falls back to vm_unmapped_area. This results in the vdso being placed in the address space next to libraries etc. While this is generally unnoticeable and doesn't break anything, it does mean that the vdso is placed below the stack when there is actually room above the stack. To me it also seems uncomfortably close to placing the vdso in the way of downward expansion of the stack.

Quoted 4 lines

I don't have a patch because I'm not sure what the algorithm should be, but thought I would bring it up as vdso_addr doesn't seem to be behaving as intended in all cases.

If the stack occupies the last possible page, how can you say there is "space above the stack"?

-hpa

#2 lore raw
Quoted 20 lines · H. Peter Anvin
Quoted 19 lines · Nathan Lynch

Hi x86/vdso people,

I've been working on adding a vDSO to 32-bit ARM, and Kees suggested I look at x86_64's algorithm for placing the vDSO at a randomized offset above the stack VMA. I found that when the stack top occupies the last slot in the PTE (is that the right term?), the vdso_addr routine returns an address below mm->start_stack, equivalent to (mm->start_stack & PAGE_MASK). For instance if mm->start_stack is 0x7fff3ffffc96, vdso_addr returns 0x7fff3ffff000.

Since the address returned is always already occupied by the stack, get_unmapped_area detects the collision and falls back to vm_unmapped_area. This results in the vdso being placed in the address space next to libraries etc. While this is generally unnoticeable and doesn't break anything, it does mean that the vdso is placed below the stack when there is actually room above the stack. To me it also seems uncomfortably close to placing the vdso in the way of downward expansion of the stack.

Quoted 7 lines
Quoted 4 lines

I don't have a patch because I'm not sure what the algorithm should be, but thought I would bring it up as vdso_addr doesn't seem to be behaving as intended in all cases.

If the stack occupies the last possible page, how can you say there is "space above the stack"?

Sorry for being unclear. I probably am getting terminology wrong. What I'm trying to express is that if the stack top is in the last page of its last-level page table (which may be the last possible page, but that's not really the interesting case), vdso_addr returns an address below mm->start_stack.

If you do a lot of execs with the following debug patch applied, you should see occasional prints like:

got addr 0x7f9a2ba16000, asked 0x7fffa7bff000, start_stack=0x7fffa7bffc96 got addr 0x7f3877ff1000, asked 0x7fffd9bff000, start_stack=0x7fffd9bffc96 got addr 0x7f96e3637000, asked 0x7ffff39ff000, start_stack=0x7ffff39ffc96 got addr 0x7fb70588d000, asked 0x7fff271ff000, start_stack=0x7fff271ffc96 got addr 0x7f7957171000, asked 0x7fff71dff000, start_stack=0x7fff71dffc96

Hopefully this better illustrates.

arch/x86/vdso/vma.c +5 -3
index 1ad102613127..06c51329d1b3 100644
@@ -157,15 +157,17 @@ static int setup_additional_pages(struct linux_binprm *bprm,
 				  unsigned size)
 {
 	struct mm_struct *mm = current->mm;
-	unsigned long addr;
+	unsigned long addr, hint;
 	int ret;
 
 	if (!vdso_enabled)
 		return 0;
 
 	down_write(&mm->mmap_sem);
-	addr = vdso_addr(mm->start_stack, size);
-	addr = get_unmapped_area(NULL, addr, size, 0, 0);
+	hint = vdso_addr(mm->start_stack, size);
+	addr = get_unmapped_area(NULL, hint, size, 0, 0);
+	if (addr != hint)
+		pr_info("got addr 0x%lx, asked 0x%lx\n", addr, hint);
 	if (IS_ERR_VALUE(addr)) {
 		ret = addr;
 		goto up_fail;
#3 lore raw
Quoted 26 lines · Nathan Lynch
Quoted 25 lines · H. Peter Anvin
Quoted 23 lines · Nathan Lynch

Hi x86/vdso people,

I've been working on adding a vDSO to 32-bit ARM, and Kees suggested I look at x86_64's algorithm for placing the vDSO at a randomized offset above the stack VMA. I found that when the stack top occupies the last slot in the PTE (is that the right term?), the vdso_addr routine returns an address below mm->start_stack, equivalent to (mm->start_stack & PAGE_MASK). For instance if mm->start_stack is 0x7fff3ffffc96, vdso_addr returns 0x7fff3ffff000.

Since the address returned is always already occupied by the stack, get_unmapped_area detects the collision and falls back to vm_unmapped_area. This results in the vdso being placed in the address space next to libraries etc. While this is generally unnoticeable and doesn't break anything, it does mean that the vdso is placed below the stack when there is actually room above the stack. To me it also seems uncomfortably close to placing the vdso in the way of downward expansion of the stack.

I don't have a patch because I'm not sure what the algorithm should be, but thought I would bring it up as vdso_addr doesn't seem to be behaving as intended in all cases.

Quoted 8 lines
Quoted 2 lines

If the stack occupies the last possible page, how can you say there is "space above the stack"?

Sorry for being unclear. I probably am getting terminology wrong. What I'm trying to express is that if the stack top is in the last page of its last-level page table (which may be the last possible page, but that's not really the interesting case), vdso_addr returns an address below mm->start_stack.

It seems like this is avoidable, then? From your example, it seems like we lose the separated randomization in this case, but we don't need to? Do you have a suggestion for what could be done to fix this?

-Kees

Quoted 38 lines

If you do a lot of execs with the following debug patch applied, you should see occasional prints like:

got addr 0x7f9a2ba16000, asked 0x7fffa7bff000, start_stack=0x7fffa7bffc96 got addr 0x7f3877ff1000, asked 0x7fffd9bff000, start_stack=0x7fffd9bffc96 got addr 0x7f96e3637000, asked 0x7ffff39ff000, start_stack=0x7ffff39ffc96 got addr 0x7fb70588d000, asked 0x7fff271ff000, start_stack=0x7fff271ffc96 got addr 0x7f7957171000, asked 0x7fff71dff000, start_stack=0x7fff71dffc96

Hopefully this better illustrates.

arch/x86/vdso/vma.c +5 -3
index 1ad102613127..06c51329d1b3 100644
@@ -157,15 +157,17 @@ static int setup_additional_pages(struct linux_binprm *bprm,
                                  unsigned size)
 {
        struct mm_struct *mm = current->mm;
-       unsigned long addr;
+       unsigned long addr, hint;
        int ret;
 
        if (!vdso_enabled)
                return 0;
 
        down_write(&mm->mmap_sem);
-       addr = vdso_addr(mm->start_stack, size);
-       addr = get_unmapped_area(NULL, addr, size, 0, 0);
+       hint = vdso_addr(mm->start_stack, size);
+       addr = get_unmapped_area(NULL, hint, size, 0, 0);
+       if (addr != hint)
+               pr_info("got addr 0x%lx, asked 0x%lx\n", addr, hint);
        if (IS_ERR_VALUE(addr)) {
                ret = addr;
                goto up_fail;
Signature · 2 lines
Kees Cook
Chrome OS Security
#4 lore raw
Quoted 36 lines · Kees Cook
Quoted 26 lines · Nathan Lynch
Quoted 25 lines · H. Peter Anvin
Quoted 23 lines · Nathan Lynch

Hi x86/vdso people,

I've been working on adding a vDSO to 32-bit ARM, and Kees suggested I look at x86_64's algorithm for placing the vDSO at a randomized offset above the stack VMA. I found that when the stack top occupies the last slot in the PTE (is that the right term?), the vdso_addr routine returns an address below mm->start_stack, equivalent to (mm->start_stack & PAGE_MASK). For instance if mm->start_stack is 0x7fff3ffffc96, vdso_addr returns 0x7fff3ffff000.

Since the address returned is always already occupied by the stack, get_unmapped_area detects the collision and falls back to vm_unmapped_area. This results in the vdso being placed in the address space next to libraries etc. While this is generally unnoticeable and doesn't break anything, it does mean that the vdso is placed below the stack when there is actually room above the stack. To me it also seems uncomfortably close to placing the vdso in the way of downward expansion of the stack.

I don't have a patch because I'm not sure what the algorithm should be, but thought I would bring it up as vdso_addr doesn't seem to be behaving as intended in all cases.

Quoted 8 lines
Quoted 2 lines

If the stack occupies the last possible page, how can you say there is "space above the stack"?

Sorry for being unclear. I probably am getting terminology wrong. What I'm trying to express is that if the stack top is in the last page of its last-level page table (which may be the last possible page, but that's not really the interesting case), vdso_addr returns an address below mm->start_stack.

Quoted 3 lines

It seems like this is avoidable, then? From your example, it seems like we lose the separated randomization in this case, but we don't need to? Do you have a suggestion for what could be done to fix this?

I don't understand why the vDSO should be special here. Either the standard logic for randomizing the placement of DSOs is good, in which case it should be good for the vDSO too, or I think we should fix it for everything.

--Andy

#5 lore raw
Quoted 37 lines · Andy Lutomirski
Quoted 36 lines · Kees Cook
Quoted 34 lines · Nathan Lynch
Quoted 20 lines · H. Peter Anvin
Quoted 19 lines · Nathan Lynch

Hi x86/vdso people,

I've been working on adding a vDSO to 32-bit ARM, and Kees suggested I look at x86_64's algorithm for placing the vDSO at a randomized offset above the stack VMA. I found that when the stack top occupies the last slot in the PTE (is that the right term?), the vdso_addr routine returns an address below mm->start_stack, equivalent to (mm->start_stack & PAGE_MASK). For instance if mm->start_stack is 0x7fff3ffffc96, vdso_addr returns 0x7fff3ffff000.

Since the address returned is always already occupied by the stack, get_unmapped_area detects the collision and falls back to vm_unmapped_area. This results in the vdso being placed in the address space next to libraries etc. While this is generally unnoticeable and doesn't break anything, it does mean that the vdso is placed below the stack when there is actually room above the stack. To me it also seems uncomfortably close to placing the vdso in the way of downward expansion of the stack.

Quoted 7 lines
Quoted 4 lines

I don't have a patch because I'm not sure what the algorithm should be, but thought I would bring it up as vdso_addr doesn't seem to be behaving as intended in all cases.

If the stack occupies the last possible page, how can you say there is "space above the stack"?

Sorry for being unclear. I probably am getting terminology wrong. What I'm trying to express is that if the stack top is in the last page of its last-level page table (which may be the last possible page, but that's not really the interesting case), vdso_addr returns an address below mm->start_stack.

Quoted 8 lines
Quoted 3 lines

It seems like this is avoidable, then? From your example, it seems like we lose the separated randomization in this case, but we don't need to? Do you have a suggestion for what could be done to fix this?

I don't understand why the vDSO should be special here. Either the standard logic for randomizing the placement of DSOs is good, in which case it should be good for the vDSO too, or I think we should fix it for everything.

The issue is specific to the vdso randomizing-near-the-stack code; regular mmap randomization is operating correctly. The reason for randomizing stack, vdso, and mmap separately is to avoid correlation of leaked offsets in one to the other regions.

-Kees

Signature · 2 lines
Kees Cook
Chrome OS Security
#6 lore raw
Quoted 47 lines · Kees Cook
Quoted 37 lines · Andy Lutomirski
Quoted 36 lines · Kees Cook
Quoted 34 lines · Nathan Lynch
Quoted 20 lines · H. Peter Anvin
Quoted 19 lines · Nathan Lynch

Hi x86/vdso people,

I've been working on adding a vDSO to 32-bit ARM, and Kees suggested I look at x86_64's algorithm for placing the vDSO at a randomized offset above the stack VMA. I found that when the stack top occupies the last slot in the PTE (is that the right term?), the vdso_addr routine returns an address below mm->start_stack, equivalent to (mm->start_stack & PAGE_MASK). For instance if mm->start_stack is 0x7fff3ffffc96, vdso_addr returns 0x7fff3ffff000.

Since the address returned is always already occupied by the stack, get_unmapped_area detects the collision and falls back to vm_unmapped_area. This results in the vdso being placed in the address space next to libraries etc. While this is generally unnoticeable and doesn't break anything, it does mean that the vdso is placed below the stack when there is actually room above the stack. To me it also seems uncomfortably close to placing the vdso in the way of downward expansion of the stack.

Quoted 7 lines
Quoted 4 lines

I don't have a patch because I'm not sure what the algorithm should be, but thought I would bring it up as vdso_addr doesn't seem to be behaving as intended in all cases.

If the stack occupies the last possible page, how can you say there is "space above the stack"?

Sorry for being unclear. I probably am getting terminology wrong. What I'm trying to express is that if the stack top is in the last page of its last-level page table (which may be the last possible page, but that's not really the interesting case), vdso_addr returns an address below mm->start_stack.

Quoted 8 lines
Quoted 3 lines

It seems like this is avoidable, then? From your example, it seems like we lose the separated randomization in this case, but we don't need to? Do you have a suggestion for what could be done to fix this?

I don't understand why the vDSO should be special here. Either the standard logic for randomizing the placement of DSOs is good, in which case it should be good for the vDSO too, or I think we should fix it for everything.

Quoted 4 lines

The issue is specific to the vdso randomizing-near-the-stack code; regular mmap randomization is operating correctly. The reason for randomizing stack, vdso, and mmap separately is to avoid correlation of leaked offsets in one to the other regions.

I understand why the offset from the stack to the vDSO should be randomized, and why the offset from the stack to, say, glibc should be randomized. What I don't get is why the offset from glibc to the vDSO should be randomized but the offset from glibc to openssl should be deterministic. Or am I misunderstanding?

--Andy

#7 lore raw
Quoted 53 lines · Andy Lutomirski
Quoted 47 lines · Kees Cook
Quoted 37 lines · Andy Lutomirski
Quoted 36 lines · Kees Cook
Quoted 34 lines · Nathan Lynch
Quoted 20 lines · H. Peter Anvin
Quoted 19 lines · Nathan Lynch

Hi x86/vdso people,

I've been working on adding a vDSO to 32-bit ARM, and Kees suggested I look at x86_64's algorithm for placing the vDSO at a randomized offset above the stack VMA. I found that when the stack top occupies the last slot in the PTE (is that the right term?), the vdso_addr routine returns an address below mm->start_stack, equivalent to (mm->start_stack & PAGE_MASK). For instance if mm->start_stack is 0x7fff3ffffc96, vdso_addr returns 0x7fff3ffff000.

Since the address returned is always already occupied by the stack, get_unmapped_area detects the collision and falls back to vm_unmapped_area. This results in the vdso being placed in the address space next to libraries etc. While this is generally unnoticeable and doesn't break anything, it does mean that the vdso is placed below the stack when there is actually room above the stack. To me it also seems uncomfortably close to placing the vdso in the way of downward expansion of the stack.

Quoted 7 lines
Quoted 4 lines

I don't have a patch because I'm not sure what the algorithm should be, but thought I would bring it up as vdso_addr doesn't seem to be behaving as intended in all cases.

If the stack occupies the last possible page, how can you say there is "space above the stack"?

Sorry for being unclear. I probably am getting terminology wrong. What I'm trying to express is that if the stack top is in the last page of its last-level page table (which may be the last possible page, but that's not really the interesting case), vdso_addr returns an address below mm->start_stack.

Quoted 8 lines
Quoted 3 lines

It seems like this is avoidable, then? From your example, it seems like we lose the separated randomization in this case, but we don't need to? Do you have a suggestion for what could be done to fix this?

I don't understand why the vDSO should be special here. Either the standard logic for randomizing the placement of DSOs is good, in which case it should be good for the vDSO too, or I think we should fix it for everything.

Quoted 4 lines

The issue is specific to the vdso randomizing-near-the-stack code; regular mmap randomization is operating correctly. The reason for randomizing stack, vdso, and mmap separately is to avoid correlation of leaked offsets in one to the other regions.

Quoted 5 lines

I understand why the offset from the stack to the vDSO should be randomized, and why the offset from the stack to, say, glibc should be randomized. What I don't get is why the offset from glibc to the vDSO should be randomized but the offset from glibc to openssl should be deterministic. Or am I misunderstanding?

Sure, I agree. In a perfect world, every DSO would have unrelated offsets. Ignoring that for the moment, my take on this is that stack, vDSO, and mmap have their locations tracked/exposed separately (SP register, AUXV, and syscall return respectively). As such, they should have separate randomization to avoid having any one method lead to the discovery of all the offsets (i.e. the vDSO location is communicated to the process in a different way than other DSOs that come into the process via the mmap syscall).

As for per-mmap randomization, I would love this, but no one has proposed algorithms that aren't worse than single base offset randomization. This is especially true under 32-bit where attempting to balance the randomization against fragmentation just leads to non-random locations under certain situations/processes/DSOs, etc. It could be worth revisiting this for 64-bit, but I feel like it's not a high priority.

-Kees

Signature · 2 lines
Kees Cook
Chrome OS Security
#8 lore raw
Quoted 69 lines · Kees Cook
Quoted 53 lines · Andy Lutomirski
Quoted 47 lines · Kees Cook
Quoted 37 lines · Andy Lutomirski
Quoted 36 lines · Kees Cook
Quoted 34 lines · Nathan Lynch
Quoted 20 lines · H. Peter Anvin
Quoted 19 lines · Nathan Lynch

Hi x86/vdso people,

I've been working on adding a vDSO to 32-bit ARM, and Kees suggested I look at x86_64's algorithm for placing the vDSO at a randomized offset above the stack VMA. I found that when the stack top occupies the last slot in the PTE (is that the right term?), the vdso_addr routine returns an address below mm->start_stack, equivalent to (mm->start_stack & PAGE_MASK). For instance if mm->start_stack is 0x7fff3ffffc96, vdso_addr returns 0x7fff3ffff000.

Since the address returned is always already occupied by the stack, get_unmapped_area detects the collision and falls back to vm_unmapped_area. This results in the vdso being placed in the address space next to libraries etc. While this is generally unnoticeable and doesn't break anything, it does mean that the vdso is placed below the stack when there is actually room above the stack. To me it also seems uncomfortably close to placing the vdso in the way of downward expansion of the stack.

Quoted 7 lines
Quoted 4 lines

I don't have a patch because I'm not sure what the algorithm should be, but thought I would bring it up as vdso_addr doesn't seem to be behaving as intended in all cases.

If the stack occupies the last possible page, how can you say there is "space above the stack"?

Sorry for being unclear. I probably am getting terminology wrong. What I'm trying to express is that if the stack top is in the last page of its last-level page table (which may be the last possible page, but that's not really the interesting case), vdso_addr returns an address below mm->start_stack.

Quoted 8 lines
Quoted 3 lines

It seems like this is avoidable, then? From your example, it seems like we lose the separated randomization in this case, but we don't need to? Do you have a suggestion for what could be done to fix this?

I don't understand why the vDSO should be special here. Either the standard logic for randomizing the placement of DSOs is good, in which case it should be good for the vDSO too, or I think we should fix it for everything.

Quoted 4 lines

The issue is specific to the vdso randomizing-near-the-stack code; regular mmap randomization is operating correctly. The reason for randomizing stack, vdso, and mmap separately is to avoid correlation of leaked offsets in one to the other regions.

Quoted 5 lines

I understand why the offset from the stack to the vDSO should be randomized, and why the offset from the stack to, say, glibc should be randomized. What I don't get is why the offset from glibc to the vDSO should be randomized but the offset from glibc to openssl should be deterministic. Or am I misunderstanding?

Sure, I agree. In a perfect world, every DSO would have unrelated offsets. Ignoring that for the moment, my take on this is that stack, vDSO, and mmap have their locations tracked/exposed separately (SP register, AUXV, and syscall return respectively). As such, they should have separate randomization to avoid having any one method lead to the discovery of all the offsets (i.e. the vDSO location is communicated to the process in a different way than other DSOs that come into the process via the mmap syscall).

Quoted 7 lines

As for per-mmap randomization, I would love this, but no one has proposed algorithms that aren't worse than single base offset randomization. This is especially true under 32-bit where attempting to balance the randomization against fragmentation just leads to non-random locations under certain situations/processes/DSOs, etc. It could be worth revisiting this for 64-bit, but I feel like it's not a high priority.

I think that the offending vDSO randomization code doesn't even run on 32-bit.

If anyone wants to fix this up (do something intelligent on 32-bit and do something more intelligent on 64-bit), it might be easier if based on my vdso cleanups
(https://git.kernel.org/cgit/linux/kernel/git/luto/linux.git/log/?h=vdso/cleanups).
The code in question is a lot less twisted with those patches
applied.

--Andy

Signature · 2 lines
Andy Lutomirski
AMA Capital Management, LLC