Skip to contentSkip to conversation

a 15 GB file on tmpfs

DISCUSSION Bastiaan Naber 11 participants 16 messages Jul 20, 2005 – Jul 22, 2005

lore
Share & tools
lore
#0 lore raw

Hi,

I am not sure if I can ask this here but I could not find any other place where I could fine anyone with this knowledge.

I have a 15 GB file which I want to place in memory via tmpfs. I want to do this because I need to have this data accessible with a very low seek time.

I want to know if this is possible before spending 10,000 euros on a machine that has 16 GB of memory.

The machine we plan to buy is a HP Proliant Xeon machine and I want to run a 32 bit linux kernel on it (the xeon we want doesn't have the 64-bit stuff yet)

Thanks in advance,
Bastiaan

#1 lore raw
Quoted 2 lines · Bastiaan Naber

I have a 15 GB file which I want to place in memory via tmpfs. I want to do this because I need to have this data accessible with a very low seek time.

That should be no problem on a 64 bit architecture.

Quoted 2 lines

I want to know if this is possible before spending 10,000 euros on a machine that has 16 GB of memory.

If you want to spend that amount of money on memory anyway, the extra cost for an AMD64 machine isn't that large.

Quoted 3 lines

The machine we plan to buy is a HP Proliant Xeon machine and I want to run a 32 bit linux kernel on it (the xeon we want doesn't have the 64-bit stuff yet)

AFAIK you can't use a 15 GB tmpfs on i386 because large memory support is basically a hack to support multiple 4GB memory spaces (some VM guru correct me if I'm wrong). Just get an Athlon64 machine and run a 64 bit kernel on it. If compatibility is a problem, you can still run a 32 bit i386 userland on an x86_64 kernel.

Erik

Signature · 2 lines
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
#2 lore raw
Quoted 23 lines · Erik Mouw
Quoted 2 lines · Bastiaan Naber

I have a 15 GB file which I want to place in memory via tmpfs. I want to do this because I need to have this data accessible with a very low seek time.

That should be no problem on a 64 bit architecture.

Quoted 2 lines

I want to know if this is possible before spending 10,000 euros on a machine that has 16 GB of memory.

If you want to spend that amount of money on memory anyway, the extra cost for an AMD64 machine isn't that large.

Quoted 3 lines

The machine we plan to buy is a HP Proliant Xeon machine and I want to run a 32 bit linux kernel on it (the xeon we want doesn't have the 64-bit stuff yet)

AFAIK you can't use a 15 GB tmpfs on i386 because large memory support is basically a hack to support multiple 4GB memory spaces (some VM guru correct me if I'm wrong). Just get an Athlon64 machine and run a 64 bit kernel on it. If compatibility is a problem, you can still run a 32 bit i386 userland on an x86_64 kernel.

Quoted 6 lines

Erik

--

+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands

-

Bastian, Erik is dead-on on that one: go 64bit and forget all worries about your 15GB filesize. Just don't forget to look not only x86-64 (intel or amd) but also itanium, ppc64 and s390 machines, you never know about surprises!

Signature · 5 lines
Greetz, Antonio Vargas aka winden of network

http://wind.codepixel.com/

Las cosas no son lo que parecen, excepto cuando parecen lo que si son.
#3 lore raw
In reply to Erik Mouw

In article <[email protected]>,

Quoted 9 lines · Erik Mouw
Quoted 2 lines · Bastiaan Naber

I have a 15 GB file which I want to place in memory via tmpfs. I want to do this because I need to have this data accessible with a very low seek time.

That should be no problem on a 64 bit architecture.

AFAIK you can't use a 15 GB tmpfs on i386 because large memory support is basically a hack to support multiple 4GB memory spaces (some VM guru correct me if I'm wrong).

I'm no VM guru but I have a 32 bit machine here with 8 GB of memory and 8 GB of swap:

# mount -t tmpfs -o size=$((12*1024*1024*1024)) tmpfs /mnt
# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             19228276   1200132  17051396   7% /
tmpfs                 12582912         0  12582912   0% /mnt

There you go, a 12 GB tmpfs. I haven't tried to create a 12 GB file on it, though, since this is a production machine and it needs the memory ..

So yes that appears to work just fine.

Mike.

#4 lore raw
Quoted 16 lines · Miquel van Smoorenburg

In article <[email protected]>,

Quoted 4 lines · Erik Mouw

On Wed, Jul 20, 2005 at 02:16:36PM +0200, Bastiaan Naber wrote:
AFAIK you can't use a 15 GB tmpfs on i386 because large memory support is basically a hack to support multiple 4GB memory spaces (some VM guru correct me if I'm wrong).

I'm no VM guru but I have a 32 bit machine here with 8 GB of memory and 8 GB of swap:

# mount -t tmpfs -o size=$((12*1024*1024*1024)) tmpfs /mnt
# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             19228276   1200132  17051396   7% /
tmpfs                 12582912         0  12582912   0% /mnt
Quoted 3 lines

There you go, a 12 GB tmpfs. I haven't tried to create a 12 GB file on it, though, since this is a production machine and it needs the memory ..

I stand corrected.

Quoted 1 line

So yes that appears to work just fine.

The question is if it's a good idea to use a 15GB tmpfs on a 32 bit i386 class machine. I guess a real 64 bit machine will be much faster in handling suchs amounts of data simply because you don't have to go through the hurdles needed to address such memory on i386.

Erik

Signature · 2 lines
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
#5 lore raw
Quoted 25 lines · Erik Mouw
Quoted 16 lines · Miquel van Smoorenburg

In article <[email protected]>,

Quoted 4 lines · Erik Mouw

On Wed, Jul 20, 2005 at 02:16:36PM +0200, Bastiaan Naber wrote:
AFAIK you can't use a 15 GB tmpfs on i386 because large memory support is basically a hack to support multiple 4GB memory spaces (some VM guru correct me if I'm wrong).

I'm no VM guru but I have a 32 bit machine here with 8 GB of memory and 8 GB of swap:

# mount -t tmpfs -o size=$((12*1024*1024*1024)) tmpfs /mnt
# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             19228276   1200132  17051396   7% /
tmpfs                 12582912         0  12582912   0% /mnt
Quoted 3 lines

There you go, a 12 GB tmpfs. I haven't tried to create a 12 GB file on it, though, since this is a production machine and it needs the memory ..

I stand corrected.

Quoted 1 line

So yes that appears to work just fine.

Quoted 8 lines

The question is if it's a good idea to use a 15GB tmpfs on a 32 bit i386 class machine. I guess a real 64 bit machine will be much faster in handling suchs amounts of data simply because you don't have to go through the hurdles needed to address such memory on i386.

Erik

On 32bit: you would have to use read() and write() or mmap() munmap() mremap() to perform your own paging, since you can't fit 15GB on a 4GB address space.

On 64bit: you would simply mmap() the whole file and you are done.

Most probably the cost of programming and debugging the hand-made paging on 32bit machines will cost more than the difference for a 64bit machine.

Signature · 5 lines
Greetz, Antonio Vargas aka winden of network

http://wind.codepixel.com/

Las cosas no son lo que parecen, excepto cuando parecen lo que si son.
#6 lore raw
Quoted 3 lines · Antonio Vargas

Most probably the cost of programming and debugging the hand-made paging on 32bit machines will cost more than the difference for a 64bit machine.

I'll second that. There may not even be a price difference. I've had quotes for otherwise comprable 32 bit Xeon servers come in more expensive than 64 bit AMD machines.

One piece of advice I would offer is to make sure you buy good ECC memory. I've been working with large servers for the last few years, and trying to be cheap with memory has caused a lot of hard to track down problems.

- Jim

Signature · 2 lines
[email protected]
SDF Public Access UNIX System - http://sdf.lonestar.org
#7 lore raw
In reply to Bastiaan Naber
Quoted 3 lines

I want to know if this is possible before spending 10,000 euros on a machine that has 16 GB of memory.

I can get a Dual Opteron 242 with 16G for less than 10K euro. :)

Jan Engelhardt

#8 lore raw
In reply to Bastiaan Naber
Quoted 10 lines · Bastiaan Naber

Hi,

I am not sure if I can ask this here but I could not find any other place where I could fine anyone with this knowledge.

I have a 15 GB file which I want to place in memory via tmpfs. I want to do this because I need to have this data accessible with a very low seek time.

I want to know if this is possible before spending 10,000 euros on a machine that has 16 GB of memory.

Make large swap file and test it yourself. Not a problem. --
vda

#9 lore raw
In reply to Bastiaan Naber

On Wed, 2005-07-20 at 14:16 +0200, Bastiaan Naber wrote:
[...]

Quoted 2 lines

I have a 15 GB file which I want to place in memory via tmpfs. I want to do this because I need to have this data accessible with a very low seek time.

Apart fromn the 32-vs-64bit thing: Isn't it enough (and simpler and more flexible) to mmap(2) that file and mlock(2) it afterwards?

Bernd

Signature · 3 lines
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services
#10 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 11 lines · Bernd Petrovitsch

On Wed, 2005-07-20 at 14:16 +0200, Bastiaan Naber wrote:
[...]

Quoted 2 lines

I have a 15 GB file which I want to place in memory via tmpfs. I want to do this because I need to have this data accessible with a very low seek time.

Apart fromn the 32-vs-64bit thing: Isn't it enough (and simpler and more flexible) to mmap(2) that file and mlock(2) it afterwards?

Bernd

On 32bit arches, a pointer is 32bit large.
On 64bit arches, a pointer is 64bit large.

You can't mmap() the whole file if it's larger than 32bit on a 32bit arch.

void *mmap(void *start, size_t length, int prot, int flags,

       int fd, off_t offset);

test.c:
#include <sys/mman.h>
int main(void)
{
printf("sizeof(void *): %d\n", sizeof(void *));
printf("sizeof(size_t): %d\n", sizeof(size_t));
}

On a 64bit machine:

$ gcc test.c -o test64 ; ./test64; file ./test64
sizeof(void *): 8
sizeof(size_t): 8
test64: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for
GNU/Linux 2.4.0, dynamically linked (uses shared libs), not stripped

On a 32bit machine (or in this case, 32bit userland on a 64bit machine):

$ gcc -m32 test.c -o test32 ; ./test32; file ./test32
sizeof(void *): 4
sizeof(size_t): 4
test32: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for
GNU/Linux 2.2.5, dynamically linked (uses shared libs), not stripped

Meaning both the pointer and the size argument are only 32bit (4byte) on 32-bit arches and 64bit (8 byte) on 64bit arches.

// Stefan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (MingW32)

iD4DBQFC32caBrn2kJu9P78RAujmAJ9J3xYdbAwYdpGXuu8kiTwdcloaSQCY1TD1 SuJJ3Ylpsa+Cdo2uIQ/Prw==
=IFvd
-----END PGP SIGNATURE-----

#11 lore raw

On Thu, 2005-07-21 at 11:12 +0200, Stefan Smietanowski wrote:
[...]

Quoted 8 lines

On a 64bit machine:

$ gcc test.c -o test64 ; ./test64; file ./test64
sizeof(void *): 8
sizeof(size_t): 8
test64: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for
GNU/Linux 2.4.0, dynamically linked (uses shared libs), not stripped

On a 32bit machine (or in this case, 32bit userland on a 64bit machine):

Quoted 8 lines
$ gcc -m32 test.c -o test32 ; ./test32; file ./test32
sizeof(void *): 4
sizeof(size_t): 4
test32: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for
GNU/Linux 2.2.5, dynamically linked (uses shared libs), not stripped

Meaning both the pointer and the size argument are only 32bit (4byte) on 32-bit arches and 64bit (8 byte) on 64bit arches.

Which means that pure addressing of the 15GB on 32bit archs needs explicit handling (yes, you can manage the offset in a "long long" but you cannot index anything with it not addressable with a 32bit offset).

Well, 64bit is apparently the way to go ....

Bernd

Signature · 3 lines
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services
#12 lore raw
In reply to Bastiaan Naber
Quoted 3 lines · In article <[email protected]> you wrote:

The machine we plan to buy is a HP Proliant Xeon machine and I want to run a 32 bit linux kernel on it (the xeon we want doesn't have the 64-bit stuff yet)

You cant have 16GB of Memory with 32bit CPUs.

Bernd

#13 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 8 lines · Bernd Eckenfels
Quoted 3 lines · In article <[email protected]> you wrote:

The machine we plan to buy is a HP Proliant Xeon machine and I want to run a 32 bit linux kernel on it (the xeon we want doesn't have the 64-bit stuff yet)

You cant have 16GB of Memory with 32bit CPUs.

PAE

CONFIG_HIGMEM64G

Supports a 36bit address space, which Xeons do support.

That doesn't mean a program can access it, you still have the same old limitations per process.

// Stefan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (MingW32)

iD8DBQFC4NHCBrn2kJu9P78RApKpAKCEUzgKKRY8VG8XIwlCBrTzWpE7LwCdGUss bSM49ZnzRgFxJ4h0WF5Ulio=
=zwUX
-----END PGP SIGNATURE-----

#14 lore raw
Quoted 4 lines · Stefan Smietanowski
Quoted 1 line

You cant have 16GB of Memory with 32bit CPUs.

PAE
CONFIG_HIGMEM64G
Supports a 36bit address space, which Xeons do support.

Yes right, I was just not aware recent hardware (still) supports that. I mean even mit 2MB modules most of them are specified only to 8GB. I would consider buying such a system quite foolish. All of the HP servers with 12GB and more seem to support EM64T. Do you know vendors who ship non-EM64T servers with more than 16GB?

Gruss
Bernd

Signature · 4 lines
  (OO)     -- Bernd_Eckenfels@Mörscher_Strasse_8.76185Karlsruhe.de --
 ( .. )    ecki@{inka.de,linux.de,debian.org}  http://www.eckes.org/
  o--o   1024D/E383CD7E  eckes@IRCNet  v:+497211603874  f:+49721151516129
(O____O)  When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl!
#15 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 9 lines · Bernd Eckenfels
Quoted 5 lines · Stefan Smietanowski
Quoted 1 line

You cant have 16GB of Memory with 32bit CPUs.

PAE
CONFIG_HIGMEM64G
Supports a 36bit address space, which Xeons do support.

Quoted 5 lines

Yes right, I was just not aware recent hardware (still) supports that. I mean even mit 2MB modules most of them are specified only to 8GB. I would consider buying such a system quite foolish. All of the HP servers with 12GB and more seem to support EM64T. Do you know vendors who ship non-EM64T servers with more than 16GB?

http://www.intel.com/products/processor/xeon/index.htm

Shows that the lowest speed Xeon with EM64T is 2.83GHz.

http://commerce.euro.dell.com/dellstore/config/frameset.asp?c=607&n=3410&b=62417&m=gbp&cs=ukbsdt1&sbc=ukbsdpedge&v=d&cu=etstore&l=en&s=ukbsd&store=ukbsd

Shows a system taking 1-4 of those processors at 2.0, 2.2, 2.7 and 3.0GHz, making them 32bit only.

Memory you can configure is from 1GiB (4x256MiB) to 32GiB (16x2GiB).

If I'd buy a server I wouldn't buy one of those however, I'm just answering the question.

Disclaimer: I do not have anything to do with either of the above mentioned vendors. I just looked up the first vendor that came into my head.

// Stefan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (MingW32)

iD8DBQFC4WDjBrn2kJu9P78RAn9RAJ9wRffW5VB0WbRgRBNjfN9+k3XMvgCgtWAO mu8p8nj8iIpNIkuYiMkiTuI=
=9kQz
-----END PGP SIGNATURE-----