brk() should check randomize_va_space rather then CONFIG_COMPAT_BRK

PATCH Frank Heckenbach 2 participants 4 messages Feb 27, 2011 – Mar 1, 2011

Subsystems (2)

Memory management Memory mapping

lore

Share & tools

lore

Frank Heckenbach

author

2011-02-27 12:09 UTC #0 lore raw

The brk() syscall checks CONFIG_COMPAT_BRK to decide whether the brk area starts just after the end of the code+bss.

Since CONFIG_COMPAT_BRK can be overridden by /proc/sys/kernel/randomize_va_space, it should rather check this variable (which is initialized in mm/memory.c from CONFIG_COMPAT_BRK), see the patch below.

I've tested it with an old libc5 binary: Without the patch, if the kernel was configured without CONFIG_COMPAT_BRK, the first malloc() fails with any setting of randomize_va_space. With the patch applied, the program runs correctly with randomize_va_space < 2, but not with randomize_va_space == 2, as it should be.

mm/mmap.c renamed from mm/mmap.c.orig +5 -5

@@ -252,11 +252,11 @@
 
 	down_write(&mm->mmap_sem);
 
-#ifdef CONFIG_COMPAT_BRK
-	min_brk = mm->end_code;
-#else
-	min_brk = mm->start_brk;
-#endif
+	if (randomize_va_space < 2)
+		min_brk = mm->end_code;
+	else
+		min_brk = mm->start_brk;
+
 	if (brk < min_brk)
 		goto out;

Jiri Kosina

2011-02-28 14:29 UTC #1 lore raw

Quoted 22 lines · Frank Heckenbach

The brk() syscall checks CONFIG_COMPAT_BRK to decide whether the brk area starts just after the end of the code+bss.
Since CONFIG_COMPAT_BRK can be overridden by /proc/sys/kernel/randomize_va_space, it should rather check this variable (which is initialized in mm/memory.c from CONFIG_COMPAT_BRK), see the patch below.
I've tested it with an old libc5 binary: Without the patch, if the kernel was configured without CONFIG_COMPAT_BRK, the first malloc() fails with any setting of randomize_va_space. With the patch applied, the program runs correctly with randomize_va_space < 2, but not with randomize_va_space == 2, as it should be.
mm/mmap.c renamed from mm/mmap.c.orig +0 -2
@@ -252,11 +252,11 @@
 
 	down_write(&mm->mmap_sem);
 
-#ifdef CONFIG_COMPAT_BRK
-	min_brk = mm->end_code;

mm/mmap.c line 261

Quoted 10 lines

mm/mmap.c renamed from mm/mmap.c.orig +5 -3

-#else
-	min_brk = mm->start_brk;
-#endif
+	if (randomize_va_space < 2)
+		min_brk = mm->end_code;
+	else
+		min_brk = mm->start_brk;
+
 	if (brk < min_brk)
 		goto out;

Hi Frank,

how do you avoid race here?

More precisely -- randomize_va_space can be changed in runtime, when already running processess have been started with different randomize_va_space value (and thus the shifting of mm->brk already happened in arch_randomize_brk())

Signature · 2 lines

Jiri Kosina
SUSE Labs, Novell Inc.

Jiri Kosina

2011-02-28 14:31 UTC #2 lore raw

Quoted 35 lines · Jiri Kosina

Quoted 22 lines
The brk() syscall checks CONFIG_COMPAT_BRK to decide whether the brk area starts just after the end of the code+bss.
Since CONFIG_COMPAT_BRK can be overridden by /proc/sys/kernel/randomize_va_space, it should rather check this variable (which is initialized in mm/memory.c from CONFIG_COMPAT_BRK), see the patch below.
I've tested it with an old libc5 binary: Without the patch, if the kernel was configured without CONFIG_COMPAT_BRK, the first malloc() fails with any setting of randomize_va_space. With the patch applied, the program runs correctly with randomize_va_space < 2, but not with randomize_va_space == 2, as it should be.
mm/mmap.c renamed from mm/mmap.c.orig +0 -2
@@ -252,11 +252,11 @@
 
 	down_write(&mm->mmap_sem);
 
-#ifdef CONFIG_COMPAT_BRK
-	min_brk = mm->end_code;
mm/mmap.c line 261
Quoted 10 lines
mm/mmap.c renamed from mm/mmap.c.orig +5 -3
-#else
-	min_brk = mm->start_brk;
-#endif
+	if (randomize_va_space < 2)
+		min_brk = mm->end_code;
+	else
+		min_brk = mm->start_brk;
+
 	if (brk < min_brk)
 		goto out;
Hi Frank,

Quoted 6 lines

how do you avoid race here?
More precisely -- randomize_va_space can be changed in runtime, when already running processess have been started with different randomize_va_space value (and thus the shifting of mm->brk already happened in arch_randomize_brk())

Oh, and also see the commit

	commit 5520e89485252c759ee60d313e9422447659947b
	Author: Jiri Kosina <[email protected]>
	Date:   Thu Jan 13 15:47:23 2011 -0800

	    brk: fix min_brk lower bound computation for COMPAT_BRK

which is quite relevant here as well (and you seem to be sending patch against code that doesn't have this patch applied).

Signature · 2 lines

Jiri Kosina
SUSE Labs, Novell Inc.

Frank Heckenbach

author

2011-03-01 05:33 UTC #3 lore raw

Quoted 8 lines · Jiri Kosina

Quoted 6 lines · Jiri Kosina
how do you avoid race here?
More precisely -- randomize_va_space can be changed in runtime, when already running processess have been started with different randomize_va_space value (and thus the shifting of mm->brk already happened in arch_randomize_brk())

I see. So would it help setting mm->brk = mm->start_brk = mm->end_code if randomize_va_space < 2 in load_elf_binary() (and elsewhere if needed -- not sure if other binformats are affected)?

Quoted 9 lines

Oh, and also see the commit

	commit 5520e89485252c759ee60d313e9422447659947b
	Author: Jiri Kosina <[email protected]>
	Date:   Thu Jan 13 15:47:23 2011 -0800

	    brk: fix min_brk lower bound computation for COMPAT_BRK

which is quite relevant here as well

AFAICS, this patch only changes the CONFIG_COMPAT_BRK case. I'm more interested in the !CONFIG_COMPAT_BRK case (since my old binaries work with CONFIG_COMPAT_BRK as is). Since e.g., Debian's default kernel doesn't set CONFIG_COMPAT_BRK, I had hoped I could get them to run by setting randomize_va_space to 0 or 1.

Quoted 2 lines

(and you seem to be sending patch
against code that doesn't have this patch applied).

Well, I used the last stable release; I hadn't noticed there was a more recent change, sorry.

Frank