malloc problem to allocate very large blocks

Hello,

I am developing applications which requires more than 2GB memory. But I found that in my Linux system the malloc() cannot allocate more than 2GB memory. Here is the details of my system:

Quoted 5 lines · Tung-Han Hsieh

Hello,

I am developing applications which requires more than 2GB memory. But I found that in my Linux system the malloc() cannot allocate more than 2GB memory. Here is the details of my system:

Malloc is a library function, not a syscall.

That said i'm not surprised. Malloc front-ends sbrk and sometimes mmap. Sbrk uses a ptrdiff_t to indicate the size desired. ptrdiff_t is signed and on 32 bit platforms should be 32 bits. Therefore, the maximum you can allocate at one time with sbrk is going to be 2GB. That malloc would inherit this limitation is to be expected.

If you need to allocate that much memory in one chunk, don't use malloc.

Hello,

I am developing applications which requires more than 2GB memory. But I found that in my Linux system the malloc() cannot allocate more than 2GB memory. Here is the details of my system:

CPU:    Pentium 4 2.53 GHz
RAM:    2 GB
Swap:   512 MB

OS: Debian-3.0 stable
Kernel: 2.4.20
gcc: 2.95.4 20011002
glibc: 2.2.5-6

In theory, in a 32-bits machine the maximum allocatable memory is up to 4GB. But in the following very simple testing program:

=====================================================================
#include <stdio.h>
#include <stdlib.h>

main()

{
    size_t l;
    char *s1=NULL, *s2=NULL;

    l = 1024*1024*1024;

    s1 = malloc(l);
    s2 = malloc(l);
    if (! s1) printf("s1 malloc failed\n");
    if (! s2) printf("s2 malloc failed\n");
}

=====================================================================

only the block for s1 can be allocated. Further, if I change the program to

=====================================================================
#include <stdio.h>
#include <stdlib.h>

main()

{
    size_t l;
    char *s1=NULL;

    l = 2*1024*1024*1024;

    s1 = malloc(l);
    if (! s1) printf("s1 malloc failed\n");
}

=====================================================================

the gcc complier complain to me that "foo.c:9: warning: integer overflow in expression" during the compilation (I use: "gcc foo.c" to compile it),
and the block for s1 cannot be allocated at all. I am wondering if there is any way to overcome the 2GB limit.

Thank you very much for your reply in advance.

Best Regards,

T.H.Hsieh
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

AFAIK malloc will not return you memory more than the total virtual memory
(RAM+swap) in the system. So if you want more than 2GB allocations from malloc, make sure you have at least 2GB virtual mem, keeping aside some space for the kernel.

--tomar

Quoted 52 lines · Tung-Han Hsieh

Hello,

I am developing applications which requires more than 2GB memory. But I found that in my Linux system the malloc() cannot allocate more than 2GB memory. Here is the details of my system:

CPU:    Pentium 4 2.53 GHz
RAM:    2 GB
Swap:   512 MB

OS: Debian-3.0 stable
Kernel: 2.4.20
gcc: 2.95.4 20011002
glibc: 2.2.5-6

In theory, in a 32-bits machine the maximum allocatable memory is up to 4GB. But in the following very simple testing program:

=====================================================================
#include <stdio.h>
#include <stdlib.h>

main()

{
    size_t l;
    char *s1=NULL, *s2=NULL;

    l = 1024*1024*1024;

    s1 = malloc(l);
    s2 = malloc(l);
    if (! s1) printf("s1 malloc failed\n");
    if (! s2) printf("s2 malloc failed\n");
}

=====================================================================

only the block for s1 can be allocated. Further, if I change the program to

=====================================================================
#include <stdio.h>
#include <stdlib.h>

main()

{
    size_t l;
    char *s1=NULL;

    l = 2*1024*1024*1024;

    s1 = malloc(l);
    if (! s1) printf("s1 malloc failed\n");
}

Quoted 3 lines

=====================================================================

the gcc complier complain to me that "foo.c:9: warning: integer

overflow

Quoted 3 lines

in expression" during the compilation (I use: "gcc foo.c" to compile it),
and the block for s1 cannot be allocated at all. I am wondering if

there

Quoted 10 lines

is any way to overcome the 2GB limit.

Thank you very much for your reply in advance.

Best Regards,

T.H.Hsieh
-
To unsubscribe from this list: send the line "unsubscribe

linux-kernel"

Quoted 5 lines

in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

AFAIK malloc will not return you memory more than the total virtual memory (RAM+swap) in the system. So if you want more than 2GB allocations from malloc, make sure you have at least 2GB virtual mem, keeping aside some space for the kernel.