Skip to contentSkip to conversation

pre5 VM livelock

DISCUSSION Jeff Garzik 4 participants 12 messages Oct 30, 2001 – Oct 31, 2001

lore
Share & tools
lore
#0 lore raw

2.4.14-pre5 was looking very nice on my alpha, doing RPM builds. It seems to swap only when it needs to, and subjectively, performance appears better.

However at this very moment, the kernel is livelocked. I can type on console and do sysrq to your heart's content... I can even sysrq-s and sync successfully. But no processing occurs. I can ping, but two ssh sessions are frozen.

Key symptoms: Free swab 0Kb according to sysrq-m, and several processes in run state according to sysrq-t.

Let me know if I should poke at this alpha further before rebooting.

further info:
free pages: 2560 kb (0kb highmem)
( active 2422 inactive 38578 free 320 )
swap cache: add 850670 delete 850666 find 323063/440091 race 1+0 free swap: 0kb
49074 pages of ram
786 free pages
1299 reserved pages
2683 pages shared
4 pages swap cached
4 pages in page table cache
buffer memory: 168kb

This behavior is reproducible, I am pretty sure.

Signature · 3 lines
Jeff Garzik      | Only so many songs can be sung
Building 1024    | with two lips, two lungs, and one tongue.
MandrakeSoft     |         - nomeansno
#1 lore raw
Quoted 27 lines · Jeff Garzik

2.4.14-pre5 was looking very nice on my alpha, doing RPM builds. It seems to swap only when it needs to, and subjectively, performance appears better.

However at this very moment, the kernel is livelocked. I can type on console and do sysrq to your heart's content... I can even sysrq-s and sync successfully. But no processing occurs. I can ping, but two ssh sessions are frozen.

Key symptoms: Free swab 0Kb according to sysrq-m, and several processes in run state according to sysrq-t.

Let me know if I should poke at this alpha further before rebooting.

further info:
free pages: 2560 kb (0kb highmem)
( active 2422 inactive 38578 free 320 )
swap cache: add 850670 delete 850666 find 323063/440091 race 1+0 free swap: 0kb
49074 pages of ram
786 free pages
1299 reserved pages
2683 pages shared
4 pages swap cached
4 pages in page table cache
buffer memory: 168kb

Quoted 1 line

This behavior is reproducible, I am pretty sure.

can you reproduce with 2.4.14pre5aa1 too? The inactive list is pretty big so maybe it's something else but maybe it's really mlocked anon memory.

Andrea

#2 lore raw
In reply to Jeff Garzik
Quoted 3 lines · Jeff Garzik

Key symptoms: Free swab 0Kb according to sysrq-m, and several processes in run state according to sysrq-t.

What are the stack traces for the processes (ie "Ctrl-ScrollLock")

It actually _sounds_ like you're out-of-memory for real, you have no swap left, and you have only four pages in the swap cache which implies that the system has tried very very hard to get rid of the pages you _did_ write to swap.

That, in turn, sounds like a memory leak. You've got 38578 pages on the inactive list (300MB worth of memory).

The fact that they _are_ on the inactive list means that the kernel certainly sees them. They just aren't freeable for some reason - possibly because they are mapped and dirty in some process (and you're out of swap so the kernel cannot put them anywhere). And if the oom() killer doesn't react, you're sol.

Question: did you have some big process that you tried to test the VM with? Did you expect the oom killer to kill it?

OR there is a leak inside the kernel, where we forget to decrement the count for certain pages in certain circumstances. Not unlikely.

Linus

#3 lore raw
Quoted 2 lines · Linus Torvalds

Question: did you have some big process that you tried to test the VM with? Did you expect the oom killer to kill it?

AFAICT, yes. I am going to re-run again to make sure (both with pre5 and also pre5aa1).

Signature · 3 lines
Jeff Garzik      | Only so many songs can be sung
Building 1024    | with two lips, two lungs, and one tongue.
MandrakeSoft     |         - nomeansno
#4 lore raw
Quoted 6 lines · Jeff Garzik
Quoted 2 lines · Linus Torvalds

Question: did you have some big process that you tried to test the VM with? Did you expect the oom killer to kill it?

AFAICT, yes. I am going to re-run again to make sure (both with pre5 and also pre5aa1).

Ok. The oom-killer is something I didn't even bother worrying about in the pre-series, I'll give that another look.

Linus

#5 lore raw
Quoted 11 lines · Linus Torvalds
Quoted 6 lines · Jeff Garzik
Quoted 2 lines · Linus Torvalds

Question: did you have some big process that you tried to test the VM with? Did you expect the oom killer to kill it?

AFAICT, yes. I am going to re-run again to make sure (both with pre5 and also pre5aa1).

Ok. The oom-killer is something I didn't even bother worrying about in the pre-series, I'll give that another look.

Jeff,

Could you please show us the _full_ Alt+Sysrq+M output when the problem happens? I want to see the "free min low high" part.

Thanks

#6 lore raw
Quoted 4 lines · Marcelo Tosatti

Jeff,

Could you please show us the _full_ Alt+Sysrq+M output when the problem happens? I want to see the "free min low high" part.

under pre5 or pre6?

Signature · 3 lines
Jeff Garzik      | Only so many songs can be sung
Building 1024    | with two lips, two lungs, and one tongue.
MandrakeSoft     |         - nomeansno
#7 lore raw

pre6.

Quoted 13 lines · Jeff Garzik
Quoted 4 lines · Marcelo Tosatti

Jeff,

Could you please show us the _full_ Alt+Sysrq+M output when the problem happens? I want to see the "free min low high" part.

under pre5 or pre6?

Signature · 3 lines
Jeff Garzik      | Only so many songs can be sung
Building 1024    | with two lips, two lungs, and one tongue.
MandrakeSoft     |         - nomeansno
#8 lore raw
In reply to Jeff Garzik
Quoted 6 lines · Linus Torvalds
Quoted 3 lines · Jeff Garzik

Key symptoms: Free swab 0Kb according to sysrq-m, and several processes in run state according to sysrq-t.

Quoted 8 lines

What are the stack traces for the processes (ie "Ctrl-ScrollLock")

It actually _sounds_ like you're out-of-memory for real, you have no swap left, and you have only four pages in the swap cache which implies that the system has tried very very hard to get rid of the pages you _did_ write to swap.

That, in turn, sounds like a memory leak. You've got 38578 pages on the

I agree it's oom, I think it's the infinite loop, it probably thinks this memory is freeable but maybe it's all anonymous mlocked memory, or maybe there's no swap at all that is equivalent for the vm.

I think it's not reproducible in -aa.

Andrea

#9 lore raw
Quoted 4 lines · Andrea Arcangeli

I agree it's oom, I think it's the infinite loop, it probably thinks this memory is freeable but maybe it's all anonymous mlocked memory, or maybe there's no swap at all that is equivalent for the vm.

It doesn't have to be mlocked - Jeff has zero swap left.

Linus

#10 lore raw
In reply to Jeff Garzik

More detailed look at your numbers..

Quoted 2 lines · Jeff Garzik

free pages: 2560 kb (0kb highmem)

Ok, the above is just the "pages_low" for your machine, it refuses to use them except for atomic allocations and such (which is why "ping" still works).

Quoted 1 line

( active 2422 inactive 38578 free 320 )

As mentioned, you have about 300MB of pages in the inactive list, and about 20M in the active list.

Quoted 2 lines

swap cache: add 850670 delete 850666 find 323063/440091 race 1+0 free swap: 0kb

You obviously _do_ have a swapfile, but it's now gone. I suspect it's clearly smaller than your RAM (you seem to have 384MB in your machine, I have no idea what your swap size is)

Quoted 2 lines

49074 pages of ram
786 free pages

The free page calculation is wrong, as it doesn't understand about multi-page allocations. You really only have 320 free pages (see above), but because there are multi-page allocations the low-level stuff thinks the later pages are free.

Quoted 2 lines

1299 reserved pages
2683 pages shared

The "shared" count is the number of pages with page_count() > 1 (actually, the sum of the page_counts), so it's likely that these are the mapped pages that are shared due to fork(). Notably it's a much smaller number than your "inactive pages", which implies that the inactive pages mostly have a count of 1. Whcih is consistent with them being dirty and mapped, but nonfreeable due to being out of swap-space.

Quoted 1 line

4 pages swap cached

That's basically zero, you probably have a few pages on the active list that were swapped in and haven't been thrown away (or the livelock is continually throwing them away and re-loading them).

Quoted 2 lines

4 pages in page table cache
buffer memory: 168kb

That's 21 pages of buffer cache, most likely pinned by the filesystem (ext2 will pin down a number of buffers just to keep track of bitmaps etc).

In short, everything is very consistent with a out-of-memory condition. We'll need to tweak the oom killer to just kill whatever offending process it is that uses everything up.

I just want confirmation that you actually did something that could result in this, ie you were testing big processes?

Linus

#11 lore raw
Quoted 2 lines · Linus Torvalds

I just want confirmation that you actually did something that could result in this, ie you were testing big processes?

yes. here's meminfo, FWIW:

[jgarzik@brutus jgarzik]$ cat /proc/meminfo
total: used: free: shared: buffers: cached:

Mem:  391372800 217260032 174112768        0 13025280 163340288
Swap: 418996224        0 418996224
MemTotal:       382200 kB
MemFree:        170032 kB
MemShared:           0 kB
Buffers:         12720 kB
Cached:         159512 kB
SwapCached:          0 kB
Active:          39888 kB
Inactive:       137880 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       382200 kB
LowFree:        170032 kB
SwapTotal:      409176 kB
SwapFree:       409176 kB
Signature · 3 lines
Jeff Garzik      | Only so many songs can be sung
Building 1024    | with two lips, two lungs, and one tongue.
MandrakeSoft     |         - nomeansno