Skip to contentSkip to conversation

mm/mmap.c: make brk() check RLIMIT_AS before page-aligning requested amount

PATCH In review Kartavya Vashishtha 2 participants 2 messages Aug 19, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm/mmap.c: make brk() check RLIMIT_AS before page-aligning requested amount

Currently, brk() only checks against RLIMIT_DATA when validating whether the requested amount of memory is valid with respect to rlimit.

RLIMIT_AS is checked later in the may_expand_vm call in do_brk_flags, but that call occurs after aligning the new brk to a page boundary, making the following possible:

1. Allocate a non-page-sized amount of memory with brk()

2. brk() will internally page-align the requested amount, and allocate the necessary amount of pages.

3. Set RLIMIT_AS to 1 byte using setrlimit.

4. Calling brk() again with a small increment (such that it does not overflow to the next page) will succeed.

This violates setrlimit RLIMIT_AS, since the call succeeds despite a 1 byte limit.

The following code snippet reproduces this behavior:

```

int main() {
        void * mem = malloc(4096);
        sbrk(32);

	// set RLIMIT_AS for the processe's address space to 1 byte
        // This causes all future calls to sbrk to fail
        struct rlimit lim;
        getrlimit(RLIMIT_AS, &lim);
        lim.rlim_cur = 1;
        printf("lim.rlim_max: %ld\n", lim.rlim_max);
        setrlimit(RLIMIT_AS, &lim);

        printf("Mallocing an additional 8 bytes, which requires more
	       "memory from sbrk, but sbrk SHOULD fail\n");
        void * ptr = sbrk(8);
        printf("sbrk result: %p\n", ptr);
        if (ptr != -1) {
                printf("sbrk unexpectedly passed\n");
        } else {
                printf("sbrk expectedly failed\n");
        }
        free(mem);
}
```
---
 mm/mmap.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/mmap.c +2 -2
index d0dfc85b209b..5f7fc6591323 100644
@@ -253,8 +253,8 @@ SYSCALL_DEFINE1(brk, unsigned long, brk)
 	 * segment grow beyond its set limit the in case where the limit is
 	 * not page aligned -Ram Gupta
 	 */
-	if (check_data_rlimit(rlimit(RLIMIT_DATA), brk, mm->start_brk,
-			      mm->end_data, mm->start_data))
+	if (check_data_rlimit(min(rlimit(RLIMIT_AS), rlimit(RLIMIT_DATA)),
+			      brk, mm->start_brk, mm->end_data, mm->start_data))
 		goto out;
 
 	newbrk = PAGE_ALIGN(brk);
Signature · 1 line
2.34.1
#1 lore raw
Quoted 70 lines · Kartavya Vashishtha

Currently, brk() only checks against RLIMIT_DATA when validating whether the requested amount of memory is valid with respect to rlimit.

RLIMIT_AS is checked later in the may_expand_vm call in do_brk_flags, but that call occurs after aligning the new brk to a page boundary, making the following possible:

1. Allocate a non-page-sized amount of memory with brk()

2. brk() will internally page-align the requested amount, and allocate the necessary amount of pages.

3. Set RLIMIT_AS to 1 byte using setrlimit.

4. Calling brk() again with a small increment (such that it does not overflow to the next page) will succeed.

This violates setrlimit RLIMIT_AS, since the call succeeds despite a 1 byte limit.

The following code snippet reproduces this behavior:

```

int main() {
        void * mem = malloc(4096);
        sbrk(32);

	// set RLIMIT_AS for the processe's address space to 1 byte
        // This causes all future calls to sbrk to fail
        struct rlimit lim;
        getrlimit(RLIMIT_AS, &lim);
        lim.rlim_cur = 1;
        printf("lim.rlim_max: %ld\n", lim.rlim_max);
        setrlimit(RLIMIT_AS, &lim);

        printf("Mallocing an additional 8 bytes, which requires more
	       "memory from sbrk, but sbrk SHOULD fail\n");
        void * ptr = sbrk(8);
        printf("sbrk result: %p\n", ptr);
        if (ptr != -1) {
                printf("sbrk unexpectedly passed\n");
        } else {
                printf("sbrk expectedly failed\n");
        }
        free(mem);
}
```
---
 mm/mmap.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/mmap.c +2 -3
index d0dfc85b209b..5f7fc6591323 100644
@@ -253,8 +253,8 @@ SYSCALL_DEFINE1(brk, unsigned long, brk)
 	 * segment grow beyond its set limit the in case where the limit is
 	 * not page aligned -Ram Gupta
 	 */
-	if (check_data_rlimit(rlimit(RLIMIT_DATA), brk, mm->start_brk,
-			      mm->end_data, mm->start_data))
+	if (check_data_rlimit(min(rlimit(RLIMIT_AS), rlimit(RLIMIT_DATA)),
+			      brk, mm->start_brk, mm->end_data, mm->start_data))
 		goto out;
 
 	newbrk = PAGE_ALIGN(brk);
--
Quoted 3 lines

2.34.1

Thanks for your patch (I think your first? :), I don't think this is correct however. The address space usage is not changing between these calls, it's staying exactly the same (in fact, your efforts to avoid going over a page boundary ensure this).

All memory allocations in the kernel are performed at page granularity, as this is the only way to actually allocate and map memory.

As per the man page:

Lowering the soft limit for a resource below the process's current consumption of that resource will succeed (but will prevent the process from further increasing its consumption of the resource).

So the limit will not prevent usage of existing resource (i.e. the address space usage).

It seems that we make an effort with RLIMIT_DATA to actually proactively prevent usage beyond the limit, again as per the man page:

RLIMIT_DATA
This is the maximum size of the process's data segment (initialized data, uninitialized data, and heap). The limit is specified in bytes, and is rounded down to the system page size. This limit affects calls to brk(2), sbrk(2), and (since Linux 4.7) mmap(2), which fail with the error ENOMEM upon encountering the soft limit of this resource.

Note 'encountering' the soft limit of this resource. In the case of RLIMIT_AS:

RLIMIT_AS
...
This limit affects calls to brk(2), mmap(2), and mremap(2), which fail with the error ENOMEM upon exceeding this limit.
...

Note 'exceeding' this limit.