Skip to contentSkip to conversation

out of vmalloc space - but vmalloc parameter does not allow boot

DISCUSSION Ranko Zivojnovic 2 participants 4 messages Apr 4, 2005

lore
Share & tools
lore
#0 lore raw

Greetings,

(Please CC responses as I am not subscribed to the list. Thanks!)

I've recently started experiencing the following problem on one of my Linux servers:

allocation failed: out of vmalloc space - use vmalloc=<size> to increase size.
allocation failed: out of vmalloc space - use vmalloc=<size> to increase size.
XFS: possible memory allocation deadlock in kmem_alloc (mode:0x2d0)
XFS: possible memory allocation deadlock in kmem_alloc (mode:0x2d0)

...and so on until it completely locks up and needs reboot.

Quoted 1 line

From what I can tell from fs/xfs/linux-2.6/kmem.c, the XFS message is

just another confirmation that the machine has run out of vmalloc space.

The machine has 4GB of RAM and is running 2.6.11.5 kernel.

I have tried to specify vmalloc=256m to start with, but no luck - the machine does not even want to boot. It panics with:
EXT2-fs: unable to read superblock
isofs_fill_super: bread failed, dev=md0, iso_blknum=16, block=32
XFS: SB read failed
VFS: Cannot open root device "md0" or unknown-block(9,0) Please append a correct "root=" boot option

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-

block(9,0)

If I remove the "vmalloc" parameter, it boots just fine but then after some hours, when the load on the server goes up, I get the above "request" to increase vmalloc. Being desperate to find the way out, I have also tried increasing the hardcoded value in arch/i386/mm/init.c, but ended up with the same effect as with the parameter - panic on boot.

/proc/meminfo says (while the system is up and running):

MemTotal:      4073244 kB
MemFree:        144356 kB
Buffers:          1184 kB
Cached:        2735576 kB
SwapCached:          0 kB
Active:         921804 kB
Inactive:      2408800 kB
HighTotal:     3193792 kB
HighFree:          896 kB
LowTotal:       879452 kB
LowFree:        143460 kB
SwapTotal:     7341600 kB
SwapFree:      7341600 kB
Dirty:           50940 kB
Writeback:           0 kB
Mapped:         613172 kB
Slab:           498936 kB
CommitLimit:   9378220 kB
Committed_AS:   736392 kB
PageTables:       1760 kB
VmallocTotal:   114680 kB
VmallocUsed:     88996 kB
VmallocChunk:    20988 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     4096 kB

I have also tried changing the following parameters - but no luck either:
vm.lower_zone_protection = 900
vm.min_free_kbytes = 30000
vm.vfs_cache_pressure = 150

Please help! What am I doing wrong?

Also, if this question does not belong here - please point to the right direction :).

Best regards,

Ranko

#1 lore raw

UPDATE: out of vmalloc space - but vmalloc parameter does not allow boot

(please do CC replies as I am still not on the list)

As I am kind of pressured to resolve this issue, I've set up a test environment using VMWare in order to reproduce the problem and (un)fortunately the attempt was successful.

I have noticed a few points that relate to the size of the physical RAM and the behavior vmalloc. As I am not sure if this is by design or a bug, so please someone enlighten me:

The strange thing I have seen is that with the increase of the physical RAM, the VmallocTotal in the /proc/meminfo gets smaller! Is this how it is supposed to be?

Namely (tests done using VMWare, using the 2.6.11.5 kernel):

/proc/meminfo on a machine with 256M of physical RAM plus 512M swap:

MemTotal:       254580 kB
MemFree:        220504 kB
Buffers:          4928 kB
Cached:          18212 kB
SwapCached:          0 kB
Active:          13360 kB
Inactive:        12604 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       254580 kB
LowFree:        220504 kB
SwapTotal:      530128 kB
SwapFree:       530128 kB
Dirty:              12 kB
Writeback:           0 kB
Mapped:           6440 kB
Slab:             5812 kB
CommitLimit:    657416 kB
Committed_AS:     7272 kB
PageTables:        344 kB
VmallocTotal:   770040 kB  <----------
VmallocUsed:      1348 kB
VmallocChunk:   768504 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     4096 kB

/proc/meminfo on a machine with 512M of physical RAM plus 512M swap:

MemTotal:       514260 kB
MemFree:        479068 kB
Buffers:          4880 kB
Cached:          18000 kB
SwapCached:          0 kB
Active:          13264 kB
Inactive:        12556 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       514260 kB
LowFree:        479068 kB
SwapTotal:      530128 kB
SwapFree:       530128 kB
Dirty:               8 kB
Writeback:           0 kB
Mapped:           6452 kB
Slab:             5740 kB
CommitLimit:    787256 kB
Committed_AS:     7280 kB
PageTables:        344 kB
VmallocTotal:   507896 kB  <----------
VmallocUsed:      1348 kB
VmallocChunk:   506360 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     4096 kB

/proc/meminfo on a machine with 768M of physical RAM plus 512M swap:

MemTotal:       774348 kB
MemFree:        739132 kB
Buffers:          4888 kB
Cached:          17992 kB
SwapCached:          0 kB
Active:          13260 kB
Inactive:        12568 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       774348 kB
LowFree:        739132 kB
SwapTotal:      530128 kB
SwapFree:       530128 kB
Dirty:             228 kB
Writeback:           0 kB
Mapped:           6448 kB
Slab:             5736 kB
CommitLimit:    917300 kB
Committed_AS:     7272 kB
PageTables:        344 kB
VmallocTotal:   245752 kB  <----------
VmallocUsed:      1348 kB
VmallocChunk:   244216 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     4096 kB

/proc/meminfo on a machine with 1024M of physical RAM plus 512M swap:

MemTotal:      1034444 kB
MemFree:        997764 kB
Buffers:          4876 kB
Cached:          18004 kB
SwapCached:          0 kB
Active:          13260 kB
Inactive:        12544 kB
HighTotal:      131008 kB
HighFree:       108448 kB
LowTotal:       903436 kB
LowFree:        889316 kB
SwapTotal:      530128 kB
SwapFree:       530128 kB
Dirty:             612 kB
Writeback:           0 kB
Mapped:           6436 kB
Slab:             5824 kB
CommitLimit:   1047348 kB
Committed_AS:     7272 kB
PageTables:        344 kB
VmallocTotal:   114680 kB  <----------
VmallocUsed:      1348 kB
VmallocChunk:   113144 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     4096 kB

Or in summary:

 256M RAM --> VmallocTotal:   770040 kB
 512M RAM --> VmallocTotal:   507896 kB
 768M RAM --> VmallocTotal:   245752 kB
   1G RAM --> VmallocTotal:   114680 kB
   4G RAM --> VmallocTotal:   114680 kB
   6G RAM --> VmallocTotal:   114680 kB

(4G and 6G RAM VmallocTotal values taken off the real machines)

Now the question: Is this behavior normal? Should it not be in reverse - more RAM equals more space for vmalloc?

With regards to the 'vmalloc' kernel parameter, I was able to boot normally using kernel parameter vmalloc=192m with RAM sizes 256, 512, 768 but _not_ with 1024M of RAM and above.

With 1024M of RAM (and apparently everything above), it is unable to boot if vmalloc parameter is specified to a value lager than default 128m. It panics with the following:

EXT2-fs: unable to read superblock
isofs_fill_super: bread failed, dev=md0, iso_blknum=16, block=32
XFS: SB read failed
VFS: Cannot open root device "md0" or unknown-block(9,0) Please append a correct "root=" boot option

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(9,0)

Question: Is this inability to boot related to the fact that the system is unable to reserve enough space for vmalloc?

Thanks for your valuable time,

Ranko

#2 lore raw

Re: UPDATE: out of vmalloc space - but vmalloc parameter does not allow boot

Hi Ranko!

Quoted 6 lines · Ranko Zivojnovic

(please do CC replies as I am still not on the list)

As I am kind of pressured to resolve this issue, I've set up a test environment using VMWare in order to reproduce the problem and (un)fortunately the attempt was successful.

Quoted 8 lines

I have noticed a few points that relate to the size of the physical RAM and the behavior vmalloc. As I am not sure if this is by design or a bug, so please someone enlighten me:

The strange thing I have seen is that with the increase of the physical RAM, the VmallocTotal in the /proc/meminfo gets smaller! Is this how it is supposed to be?

Well, I'm by no means a VM expert (not even a regular kernel hacker), but it seems to me that the sum of LowTotal and VmallocTotal is rather constant for the different settings. Alas, I cannot offer an explanation why this should be, so hopefully a knowledgeable person will shed some light on this issue.

Ciao,
Roland

--
TU Muenchen, Physik-Department E18, James-Franck-Str. 85747 Garching Telefon 089/289-12592; Telefax 089/289-12570
--
When I am working on a problem I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong.
-- R. Buckminster Fuller
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GS/CS/M/MU d-(++) s:+ a-> C+++ UL++++ P-(+) L+++ E(+) W+ !N K- w--- M+ !V Y+ PGP++ t+(++) 5 R+ tv-- b+ DI++ e+++>++++ h---- x+++
------END GEEK CODE BLOCK------

#3 lore raw

Re: UPDATE: out of vmalloc space - but vmalloc parameter does not allow boot

In reply to Ranko Zivojnovic

Ok, I think I've figured it out so I will try and answer my own questions (the best part is at the end)...

Quoted 6 lines · Ranko Zivojnovic

(please do CC replies as I am still not on the list)

As I am kind of pressured to resolve this issue, I've set up a test environment using VMWare in order to reproduce the problem and (un)fortunately the attempt was successful.

Quoted 8 lines

I have noticed a few points that relate to the size of the physical RAM and the behavior vmalloc. As I am not sure if this is by design or a bug, so please someone enlighten me:

The strange thing I have seen is that with the increase of the physical RAM, the VmallocTotal in the /proc/meminfo gets smaller! Is this how it is supposed to be?

As the size of memory grows, more gets allocated to the low memory, less to the vmalloc memory - within first 1GB of RAM.

Quoted 1 line

Now the question: Is this behavior normal?

I guess it is (nobody said the oposite).

Quoted 3 lines

Should it not be in reverse -
more RAM equals more space for vmalloc?

It really depends on the setup and the workload - some reasonable defaults (i.e. 128M) have been defined - you can change them using vmalloc parameter - but with the _extreme_ care as it gets really tricky if your RAM is 1G and above - read on...

Quoted 8 lines

With regards to the 'vmalloc' kernel parameter, I was able to boot normally using kernel parameter vmalloc=192m with RAM sizes 256, 512, 768 but _not_ with 1024M of RAM and above.

With 1024M of RAM (and apparently everything above), it is unable to boot if vmalloc parameter is specified to a value lager than default 128m. It panics with the following:

Quoted 7 lines

EXT2-fs: unable to read superblock
isofs_fill_super: bread failed, dev=md0, iso_blknum=16, block=32
XFS: SB read failed
VFS: Cannot open root device "md0" or unknown-block(9,0) Please append a correct "root=" boot option

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(9,0)

And not just - I have just seen the actual culprit message (way up front):
initrd extends beyond end of memory (0x37fef33a > 0x34000000) disabling initrd

Quoted 3 lines

Question: Is this inability to boot related to the fact that the system is unable to reserve enough space for vmalloc?

The resolution (or rather workaround) to the above is to _trick_ the GRUB into loading the initrd image into the area below what is _going_ to be the calculated "end of memory" using the "uppermem" command.

Now:
1. I hope this is the right way around the problem.
2. I hope this is going to help someone.

Best regards,

Ranko