Skip to contentSkip to conversation

: mm : mmap.c Arithmetic overflow in may_expand_vm()

PATCH New Earl Chew 1 participant 1 message Oct 16, 2009

Subsystems (2)
lore
Share & tools
lore

Earl Chew

author
#0 lore raw

PATCH 1/1 : mm : mmap.c Arithmetic overflow in may_expand_vm()

The function may_expand_vm() may return a false positive if the proposed increment (npages) is sufficient large to overflow the expression:

cur + npages

Assuming that cur < lim is an invariant, the proposed patch re-arranges the expression to avoid unsigned arithmetic overflow.

More robustly:

	if (cur > lim || npages > lim - cur)
		return 0;

might be preferred if it cannot be guaranteed that cur < lim.

mm/mmap.c renamed from mm/mmap.c.orig +1 -1
@@ -2303,7 +2303,7 @@ int may_expand_vm(struct mm_struct *mm,
 
 	lim = current->signal->rlim[RLIMIT_AS].rlim_cur >> PAGE_SHIFT;
 
-	if (cur + npages > lim)
+	if (npages > lim - cur)
 		return 0;
 	return 1;
 }