Linux 2.4: Allocation of >1GB in one chunk

Hello,

please forgive me to ask this (perhaps newbie?) question here on l-k, but I'm desperate. This is my problem:

I am running various kinds of EDA software on 32-bit Linux, and they need substantial amounts of memory. I am running 2.4.21 with with PAGE_OFFSET at 0xc0000000, so I can run processes just over 3GB. The machine (a dual Xeon) has 4GB memory and 4GB swap.

But there is this one program now that dies because it's out of memory. No surprise, as this happens frequently with tasks that would need 4GB or more.

But this one needs less than 3GB. But what it does need (I strace'ed this), is 1.3GB in one whole chunk.

I wrote a test program to mimic this:

The attached program allocates argv[1] MB in 1MB chunks and argv[2] MB in one big chunk. (The original version also touched every page, but this makes no difference here.)

[chm@trex7:~/C] ./foo 2500 500
Will allocate 2621440000 bytes in 1MB chunks...
Will allocate 524288000 bytes in one chunk...
Succeeded.

[chm@trex7:~/C] ./foo 1500 1000
Will allocate 1572864000 bytes in 1MB chunks...
Will allocate 1048576000 bytes in one chunk...
malloc: Cannot allocate memory
Out of memory.

The first call allocated 3GB and succeeds, the second one only 2.5GB but fails!

The thing that comes to my mind is memory fragmentation, but how could that be, with virtual memory?

rlimit is also unlimited (and it happens for root as well).

Skimming through the kernel sources shows too many places where memory allocation could fail, unfortunately I don't know _where_ it fails. The machine is used for production, I cannot simply take it down and run a debugging kernel on it.

I have played with /proc/sys/vm/overcommit_memory, to no avail.

I have watched /proc/slabinfo, and /proc/sys/vm/* while allocating. Still no idea.

Is there anything I can do to make this work?

Grateful for any help or pointers,
chm.

PS: Will the behaviour be different in 2.6?

----------------------------------------------------------------------
This is my test program (the sleep(60) is there to be able to peek around in /proc after the memory has been allocated):

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv)
{
unsigned int i;
unsigned long n1=0;
unsigned long n2=0;
char * p;

  if ( argc >= 2 )
    {
      n1=strtol(argv[1], 0, 10)*1024*1024;
    }

  if ( argc >= 3 )
    {
      n2=strtol(argv[2], 0, 10)*1024*1024;
    }

  fprintf(stderr, "Will allocate %lu bytes in 1MB chunks...\n", n1);

  for(i=0; i<n1; i+=1024*1024)
    {
      p=(char*)malloc(1024*1024);
      if ( p == 0 )
        {
          perror("malloc");
          fprintf(stderr, "Out of memory (%d).\n", i);
          sleep(60);
          exit(1);
        }
    }      

  fprintf(stderr, "Will allocate %lu bytes in one chunk...\n", n2);

  p=(char*)malloc(n2);
    
  if ( p == 0 )
    {
        perror("malloc");
        fprintf(stderr, "Out of memory.\n");
        sleep(60);
        exit(1);
    }
  
  fprintf(stderr, "Succeeded.\n");

  sleep(60);

return 0;
}

christian mautner -- chm bei istop punkt com -- ottawa, canada

Virtual memory fixes physical memory fragmentation only. I.e. you can `glue' multiple physical chunks together into one large virtual chunk.

Biut you're still limited to a 32-bit virtual address space (3 GB in user space). If this virtual 3 GB gets fragmented, you're still out of luck.

To check this, print all allocated virtual addresses, or look at /proc/<pid>/maps, and see why it fails.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

(since I hate googling for answers and always finding people with the same questions, but never with answers, I post the whole story:)

This was my problem:

This was a typo, I meant 0xe0000000

Matthew Wilcox and Geert Uytterhoeven pointed me to the /proc/*/maps file, which looked like this (last column trimmed):

08048000-08049000 r-xp 00000000 00:0b 1516708 foo 08049000-0804a000 rw-p 00000000 00:0b 1516708 foo 0804a000-2ed4a000 rwxp 00000000 00:00 0 42000000-42126000 r-xp 00000000 08:06 144004 libc-2.2.93.so 42126000-4212b000 rw-p 00126000 08:06 144004 libc-2.2.93.so 4212b000-4212f000 rw-p 00000000 00:00 0 4aaab000-4aabd000 r-xp 00000000 08:06 16004 ld-2.2.93.so 4aabd000-4aabe000 rw-p 00012000 08:06 16004 ld-2.2.93.so 4aac9000-dff15000 rw-p 00000000 00:00 0 dfffc000-e0000000 rwxp ffffd000 00:00 0

This shows that the memory space is split into two fragments, at around 0x40000000 (=1GB) I learned that the virtual memory space is usually laid out like this[1]:

0GB-1GB User space - Used for executable and brk/sbrk allocations
(malloc uses brk for small chunks).
1GB-2GB User space - Used for mmaps (shared memory), shared
libraries and malloc uses mmap (malloc uses mmap for large chunks).
2GB-3GB User space - Used for stack.
3GB-4GB Kernel Space - Used for the kernel itself.

I have moved the start of the kernel already up to 3.5GB by setting PAGE_OFFSET. To move the start of the mmap space, I set TASK_UNMAPPED_BASE in include/asm-i386/processor.h to 0x20000000 (=512MB).

But, to my surprise, this is, for example, a maps file with the new kernel:

08048000-0804c000 r-xp 00000000 08:06 112076 /bin/cat 0804c000-0804d000 rw-p 00003000 08:06 112076 /bin/cat 0804d000-0804e000 rwxp 00000000 00:00 0
20000000-20012000 r-xp 00000000 08:06 16004 /lib/ld-2.2.93.so 20012000-20013000 rw-p 00012000 08:06 16004 /lib/ld-2.2.93.so 2001e000-2001f000 rw-p 00000000 00:00 0
42000000-42126000 r-xp 00000000 08:06 144004 /lib/i686/libc-2.2.93.so 42126000-4212b000 rw-p 00126000 08:06 144004 /lib/i686/libc-2.2.93.so 4212b000-4212f000 rw-p 00000000 00:00 0
dfffe000-e0000000 rwxp fffff000 00:00 0

The program I had problems with now works fine, because the smaller chunks that are allocated first, fit in the hole between 0x20000000 and 0x42000000, leaving the big space for the 1.3GB chunk.

But why is the libc at 0x42000000, making my de-fragmentation efforts almost worthless? Turns out [2], "On RedHat 7.3 and 8.0 the C library is compiled for a fixed address rather than a dynamic address."

Great. The only solution is to get the libc RPM and recompile and re-package it. I might do this at a later point.

I had one more problem, that grub apparently cannot load the initial RAM disk if the PAGE_OFFSET is not 0xc0000000, but this does not belong here.

thanks everyone,
chm.

[1] http://www.puschitz.com/TuningLinuxForOracle.shtml
[2] http://www.daimi.au.dk/~kasperd/comp.os.linux.development.faq.html

PS: I did not choose RedHat voluntarily, but try running commercial (EDA) software on Linux that is not RedHat 7.3!

christian mautner -- chm bei istop punkt com -- ottawa, canada