Skip to contentSkip to conversation

What is the limit size of tmpfs /dev/shm ?

DISCUSSION Tomasz Chmielewski 4 participants 9 messages Feb 6, 2008

lore
Share & tools
lore
#0 lore raw
Quoted 5 lines

Hello Kernel Users,

is there a size limit for tmpfs for the /dev/shm filesystem?
Normally its default size is set to 2 GB. Is it possible to create a 2 TB (Terrabyte) filesystem with tmpfs? Or is there a maximum size defined in the linux kernel?

Depends on your arch.

If it's 32 bit, it's limited to 16TB:

# mount -o size=16383G -t tmpfs tmpfs /mnt/2
# df -h
(...)
tmpfs                  16T     0   16T   0% /mnt/2
# umount /mnt/2
# mount -o size=16385G -t tmpfs tmpfs /mnt/2
# df -h
(...)
tmpfs                 1.0G     0  1.0G   0% /mnt/2

So 16384G would mean the same as 0.

If you're 64 bit, you need to have really loads of storage and/or RAM to accumulate 16EB:

# mount -t tmpfs -o size=171798691839G tmpfs /mnt/2
# df -h
(...)
tmpfs                  16E     0   16E   0% /mnt/2
Signature · 2 lines
Tomasz Chmielewski
http://lists.wpkg.org
#1 lore raw
Quoted 7 lines · Tomasz Chmielewski
Quoted 5 lines

Hello Kernel Users,

is there a size limit for tmpfs for the /dev/shm filesystem?
Normally its default size is set to 2 GB. Is it possible to create a 2 TB (Terrabyte) filesystem with tmpfs? Or is there a maximum size defined in the linux kernel?

Depends on your arch.

I have a 64 Bits architecture (amd64/x86_64).

Quoted 6 lines

If you're 64 bit, you need to have really loads of storage and/or RAM to accumulate 16EB:

# mount -t tmpfs -o size=171798691839G tmpfs /mnt/2
# df -h
(...)
tmpfs                  16E     0   16E   0% /mnt/2

Nice, I will try this out.
I have not the money for 16E of memory of RAM! :-) lol
Already to provide 1 TB it would cost at least for the memory about 20 000 EUR + the server costs which support this.

Signature · 1 line
Best regards
#2 lore raw
In reply to Tomasz Chmielewski
Quoted 3 lines · Tomasz Chmielewski
Quoted 3 lines

Hello Kernel Users,

is there a size limit for tmpfs for the /dev/shm filesystem?

There shouldn't be any artificial size limit on the /dev/shm filesystem, it's an "internal" mount, and those are unlimited by default. Which is not to say that you magically receive unlimited memory along with it!

The practical limit would be somewhere round about the total size of your swap plus, ooh, finger in the air, 75% of your RAM - though that all depends on what else you'd be wanting to use RAM+swap for.

Quoted 1 line
Quoted 1 line

Normally its default size is set to 2 GB.

Hmm, where did that number come from, maybe I'm forgetting something. The user-visible mounts are limited by default to half of ram, so if you've 4GB of ram, then 2GB would be the default for those tmpfs mounts.

Quoted 1 line
Quoted 1 line

Is it possible to create a 2 TB (Terrabyte) filesystem with tmpfs?

Yes.

Quoted 11 lines
Quoted 1 line

Or is there a maximum size defined in the linux kernel?

Depends on your arch.

If it's 32 bit, it's limited to 16TB:

# mount -o size=16383G -t tmpfs tmpfs /mnt/2
# df -h
(...)
tmpfs                  16T     0   16T   0% /mnt/2
Quoted 8 lines
# umount /mnt/2
# mount -o size=16385G -t tmpfs tmpfs /mnt/2
# df -h
(...)
tmpfs                 1.0G     0  1.0G   0% /mnt/2

So 16384G would mean the same as 0.

Hah! Nice investigation. That's more of a bug than anything, though not one I feel urgently compelled to fix.

Quoted 7 lines

If you're 64 bit, you need to have really loads of storage and/or RAM to accumulate 16EB:

# mount -t tmpfs -o size=171798691839G tmpfs /mnt/2
# df -h
(...)
tmpfs                  16E     0   16E   0% /mnt/2

Mika, you answered:

Quoted 4 lines

Nice, I will try this out.
I have not the money for 16E of memory of RAM! :-) lol
Already to provide 1 TB it would cost at least for the memory about 20 000 EUR + the server costs which support this.

Don't forget that tmpfs overflows into swap, so you could save money by adding adding more swap and cutting down on the RAM: though of course that will perform very poorly once it's actually using the swap, probably not the direction you want to go in.

Hugh

#3 lore raw
Quoted 6 lines · Hugh Dickins

Don't forget that tmpfs overflows into swap, so you could save money by adding adding more swap and cutting down on the RAM: though of course that will perform very poorly once it's actually using the swap, probably not the direction you want to go in.

It can still outperform conventional disk filesystems, however, mostly because it doesn't have to worry one iota about consistency.

-hpa

#4 lore raw
Quoted 9 lines · H. Peter Anvin
Quoted 5 lines · Hugh Dickins

Don't forget that tmpfs overflows into swap, so you could save money by adding adding more swap and cutting down on the RAM: though of course that will perform very poorly once it's actually using the swap, probably not the direction you want to go in.

It can still outperform conventional disk filesystems, however, mostly because it doesn't have to worry one iota about consistency.

In theory, yes, and should be true in practice before it hits swap. But I think you'll find our swap handling is too primitive for tmpfs to perform well once we hit swap. Most filesystems pay considerable attention to good performance within their constraints of correctness. Whereas with tmpfs we've just never worried about the performance once swapping. It's used so you don't lose your data, but if you're really expecting to be going to disk very much, better start with a filesystem really designed for that.

Hugh

#5 lore raw
Quoted 10 lines · Hugh Dickins

In theory, yes, and should be true in practice before it hits swap. But I think you'll find our swap handling is too primitive for tmpfs to perform well once we hit swap. Most filesystems pay considerable attention to good performance within their constraints of correctness. Whereas with tmpfs we've just never worried about the performance once swapping. It's used so you don't lose your data, but if you're really expecting to be going to disk very much, better start with a filesystem really designed for that.

That sounds like a problem in our overall swap handling, not specifically in tmpfs. Now, I can't say anything concrete about heavy swap conditions, but in light swap conditions I have measured a 20x performance improvement(!) over ext3 on real workloads.

-hpa

#6 lore raw
Quoted 5 lines · H. Peter Anvin

That sounds like a problem in our overall swap handling, not specifically in tmpfs. Now, I can't say anything concrete about heavy swap conditions, but in light swap conditions I have measured a 20x performance improvement(!) over ext3 on real workloads.

Wow, I'm surprised. I suppose I do jump to thinking of heavy swapping when light swapping won't be so bad; but even so, 20x ext3 astonishes me - ext3 wouldn't be anyone's choice for fastest, but even so...

I certainly guess too much and measure too little:
is there a useful test you could point me to? TIA

Hugh

#7 lore raw
Quoted 6 lines · Hugh Dickins
Quoted 4 lines · H. Peter Anvin

That sounds like a problem in our overall swap handling, not specifically in tmpfs. Now, I can't say anything concrete about heavy swap conditions, but in light swap conditions I have measured a 20x performance improvement(!) over ext3 on real workloads.

Quoted 7 lines

Wow, I'm surprised. I suppose I do jump to thinking of heavy swapping when light swapping won't be so bad; but even so, 20x ext3 astonishes me - ext3 wouldn't be anyone's choice for fastest, but even so...

I certainly guess too much and measure too little:
is there a useful test you could point me to? TIA

The specific application was this:

- extract a kernel tarball
- "make distclean"
- cp -al the resulting tree
- apply a patch to each tree
- do a diff between the trees
- delete all files

... repeat something like 20,000 times.

-hpa

#8 lore raw
Quoted 11 lines · H. Peter Anvin

The specific application was this:

- extract a kernel tarball
- "make distclean"
- cp -al the resulting tree
- apply a patch to each tree
- do a diff between the trees
- delete all files

... repeat something like 20,000 times.

Right, thanks.

Certainly extracting a tarball can be a surprising challenge for some filesystems.

And you remind me that of course, to the extent that a load is about metadata (lots of creates for example), tmpfs can hardly lose since its metadata never goes out to swap. But there is plenty of data in your test above.

Hugh