Skip to contentSkip to conversation

2.4.24 Paging Fault, Cache tries to swap with no swap partition

DISCUSSION Ross Dickson 2 participants 5 messages Feb 14, 2004 – Feb 16, 2004

lore
Share & tools
lore
#0 lore raw

Greetings,

I have an imaging system writing files to removable hard drives. Compact Flash boot with ram drives so I usually have no swap partition or file.

Recently I upgraded kernel from 2.4.20 to 2.4.24.

System has "mem=460M" (512M ram fitted) and starts with about 400M free. After recording for a while the Cached ram acquires all but about 4Mb MemFree.

On a hot 38C day it started Oops'ing re paging memory. It runs the same 2 programs all day gathering and compressing images.
Sorry I have no detail on the Oops at the moment, computer is in a vehicle and does not normally have a screen. From memory it couldn't allocate a virtual page.

I found if I put in a 16Mb ram drive as swap then it would grab roughly 1.4Mb of it on occasion and keep it until recording stopped for a while. SwapCached is either 0Kb or 1024Kb, not anything else.

Is this behaviour expected - to require a swap file?
Can the paging cache be tuned in /proc or somewhere to prevent it being so greedy as to want more memory than the machine has?

Is the quickest fix to give it more ram. I read on another posting that with greater than 512Mb the cache won't grab any more?

Regards
Ross.

#1 lore raw
Quoted 11 lines · Ross Dickson

I have an imaging system writing files to removable hard drives. Compact Flash boot with ram drives so I usually have no swap partition or file.

Recently I upgraded kernel from 2.4.20 to 2.4.24.

System has "mem=460M" (512M ram fitted) and starts with about 400M free. After recording for a while the Cached ram acquires all but about 4Mb MemFree.

On a hot 38C day it started Oops'ing re paging memory. It runs the

Too vague.
Do you have any logging? At least a circular buffer? Anything?

Quoted 8 lines

same 2 programs all day gathering and compressing images. Sorry I have no detail on the Oops at the moment, computer is in a vehicle and does not normally have a screen. From memory it couldn't allocate a virtual page.

I found if I put in a 16Mb ram drive as swap then it would grab roughly 1.4Mb of it on occasion and keep it until recording stopped for a while. SwapCached is either 0Kb or 1024Kb, not anything else.

If swap is active, some of it may be used even when box is not heavily loaded. That's normal.

Quoted 1 line

Is this behaviour expected - to require a swap file?

No.

Quoted 2 lines

Can the paging cache be tuned in /proc or somewhere to prevent it being so greedy as to want more memory than the machine has?

Maybe. But you should concentrate on finding where exactly it oopsed.

Quoted 2 lines

Is the quickest fix to give it more ram. I read on another posting that with greater than 512Mb the cache won't grab any more?

Please don't succumb to 'add more RAM' syndrome. 460 megs should be fine for you. I'd say better find the root of the problem.
--
vda

#2 lore raw
Quoted 7 lines · you
Quoted 6 lines · Ross Dickson

I have an imaging system writing files to removable hard drives. Compact Flash boot with ram drives so I usually have no swap partition or file.

Recently I upgraded kernel from 2.4.20 to 2.4.24.

Quoted 8 lines
Quoted 5 lines

System has "mem=460M" (512M ram fitted) and starts with about 400M free. After recording for a while the Cached ram acquires all but about 4Mb MemFree.

On a hot 38C day it started Oops'ing re paging memory. It runs the

Too vague.
Do you have any logging? At least a circular buffer? Anything?

Unfortunately not much, I was hot, tired, not at my best - should have grabbed it when I had the chance. All I grabbed was a partial code string at the bottom of the Oops which I doubt is of any benefit without the rest. 8b 5f 04 8d 77 08 83 eb 18 8b
Oh yeah, it killed init too.
System defaults to not logging to permanent storage, flash would die over time from writes and info would be meaningless to customer on their removable hard drive. Note to self - must change that - I'm sure customer could spare some space.

I assumed I could reproduce the fault today if but it wouldn't fault. First misbehaved Friday 13th, reproduced Sat 14th.
Self cured??? Sunday 15th. Doh!!!

Quoted 13 lines
Quoted 9 lines

same 2 programs all day gathering and compressing images. Sorry I have no detail on the Oops at the moment, computer is in a vehicle and does not normally have a screen. From memory it couldn't allocate a virtual page.

I found if I put in a 16Mb ram drive as swap then it would grab roughly 1.4Mb of it on occasion and keep it until recording stopped for a while. SwapCached is either 0Kb or 1024Kb, not anything else.

If swap is active, some of it may be used even when box is not heavily loaded. That's normal.

Quoted 8 lines
Quoted 1 line

Is this behaviour expected - to require a swap file?

No.

Quoted 2 lines

Can the paging cache be tuned in /proc or somewhere to prevent it being so greedy as to want more memory than the machine has?

Maybe. But you should concentrate on finding where exactly it oopsed.

I note memfree stabilises at around 4Mb when running OK, given it only wanted an extra 1Mb cache swap, can I cat something to /proc/sys/vm/????? to force it to stabilise at around 10Mb or 20Mb? Otherwise can I change a constant and recompile kernel to achieve same? It might help give more headroom when the event occurs.

Quoted 6 lines
Quoted 3 lines

Is the quickest fix to give it more ram. I read on another posting that with greater than 512Mb the cache won't grab any more?

Please don't succumb to 'add more RAM' syndrome. 460 megs should be fine for you. I'd say better find the root of the problem.

I admit it, I since tried the 'add more RAM' but a couple of capture card devices did not like more than about 800Mb so I pulled the stick back out. It ran quite well in 256Mb with the old kernel for about a year so it is puzzling.

Quoted 2 lines

vda

Thanks for the response
Regards
Ross.

#3 lore raw
Quoted 7 lines · Ross Dickson
Quoted 6 lines · you
Quoted 5 lines · Ross Dickson

I have an imaging system writing files to removable hard drives. Compact Flash boot with ram drives so I usually have no swap partition or file.

Recently I upgraded kernel from 2.4.20 to 2.4.24.

Is KM18G Pro (nforce2 dual memory mode), AMD 2400XP, Preempt, Low latency, 64Bit jiffies 1000Hz patched.

I found some articles about memory overcommitment, checked the source and saw strict in use for arm systems - no swap- so this time I thought I would try

echo 1 > /proc/sys/vm/overcommit_memory

I got another oops under equivalent circumstances to earlier (no swap).
I ran oops through ksymoops on another machine with same kernel , results below. At this point I think the trigger may be a slow (bad blocks?) 80Gb hard drive the files are being written to. The PCI bus is quite busy with imaging from 3 cameras on two capture cards (bttv and meteor II mc).

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

Can the paging cache be tuned in /proc or somewhere to prevent it being so greedy as to want more memory than the machine has?

Maybe. But you should concentrate on finding where exactly it oopsed.

...snip...

Quoted 13 lines

......I added a 16mb ram drive swap (see earlier posting)
I note memfree stabilises at around 4Mb when running OK, given it only wanted an extra 1Mb cache swap, can I cat something to /proc/sys/vm/????? to force it to stabilise at around 10Mb or 20Mb? Otherwise can I change a constant and recompile kernel to achieve same? It might help give more headroom when the event occurs.

Quoted 6 lines
Quoted 3 lines

Is the quickest fix to give it more ram. I read on another posting that with greater than 512Mb the cache won't grab any more?

Please don't succumb to 'add more RAM' syndrome. 460 megs should be fine for you. I'd say better find the root of the problem.

...snip...

Quoted 2 lines
Quoted 2 lines

vda

Thanks for the response
Regards
Ross.

Was run "mem=450" with this Oops

ksymoops 2.4.8 on i686 2.4.24-rd. Options used
-V (default)
-k /proc/ksyms (default)
-L (specified)
-o /lib/modules/2.4.24-rd/ (default)
-m /boot/System.map (specified)

Unable to handle kernel paging request at virtual address 6a65656a
c0133f20
*pde = 00000000
Oops: 0000
CPU:    0
EIP:    0010:[<c0133f20>]    Not tainted

Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010883
eax: 6a65656a ebx: 000000f5 ecx: c14dfdd4 edx: c14dfde4
esi: 00000000 edi: 00000008 ebp: c14dfe40 esp: dc16df38
ds: 0018 es: 0018 ss: 0018
Process kswapd (pid: 4, stackpage=dc16d000)

Stack: 000001d0 00000001 c14dfdd4 00000000 00000005 00000005 00000020 000001d0 
       c032c4d8 c032c4d8 c0134ebc dc16df84 000001d0 0000003c 00000020 c0134f58 
       dc16df84 dc16c000 00000000 00000000 c032c4d8 dc16c000 c032c400 00000000 
Call Trace:    [<c0134ebc>] [<c0134f58>] [<c01350f6>] [<c0135169>] [<c013529d>]
  [<c0135210>] [<c0105000>] [<c01057db>] [<c0135210>]
Code: 8b 00 43 39 d0 75 f9 8b 44 24 08 89 da 8b 70 24 8b 40 44 89 
Quoted 6 lines
Quoted 6 lines

EIP; c0133f20 <kmem_cache_reap+80/1f0> <=====

ecx; c14dfdd4 <_end+1115228/1e4db4b4>
edx; c14dfde4 <_end+1115238/1e4db4b4>
ebp; c14dfe40 <_end+1115294/1e4db4b4>
esp; dc16df38 <_end+1bda338c/1e4db4b4>

Trace; c0134ebc <shrink_caches+1c/60>
Trace; c0134f58 <try_to_free_pages_zone+58/e0>
Trace; c01350f6 <kswapd_balance_pgdat+56/b0>
Trace; c0135169 <kswapd_balance+19/30>
Trace; c013529d <kswapd+8d/b0>
Trace; c0135210 <kswapd+0/b0>
Trace; c0105000 <_stext+0/0>
Trace; c01057db <arch_kernel_thread+2b/40>
Trace; c0135210 <kswapd+0/b0>

Code; c0133f20 <kmem_cache_reap+80/1f0>
00000000 <_EIP>:
Code; c0133f20 <kmem_cache_reap+80/1f0> <=====
0: 8b 00 mov (%eax),%eax <=====
Code; c0133f22 <kmem_cache_reap+82/1f0>
2: 43 inc %ebx
Code; c0133f23 <kmem_cache_reap+83/1f0>
3: 39 d0 cmp %edx,%eax
Code; c0133f25 <kmem_cache_reap+85/1f0>
5: 75 f9 jne 0 <_EIP>
Code; c0133f27 <kmem_cache_reap+87/1f0>
7: 8b 44 24 08 mov 0x8(%esp,1),%eax
Code; c0133f2b <kmem_cache_reap+8b/1f0>
b: 89 da mov %ebx,%edx
Code; c0133f2d <kmem_cache_reap+8d/1f0>
d: 8b 70 24 mov 0x24(%eax),%esi
Code; c0133f30 <kmem_cache_reap+90/1f0>
10: 8b 40 44 mov 0x44(%eax),%eax
Code; c0133f33 <kmem_cache_reap+93/1f0>
13: 89 00 mov %eax,(%eax)

Mem starts like this when programs have been started. Was run "mem=460" for these mem readings.
total: used: free: shared: buffers: cached:

Mem:  473899008 22151168 451747840        0   327680  7737344
Swap:        0        0        0
MemTotal:       462792 kB
MemFree:        441160 kB
MemShared:           0 kB
Buffers:           320 kB
Cached:           7556 kB
SwapCached:          0 kB
Active:           2320 kB
Inactive:         7072 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       462792 kB
LowFree:        441160 kB
SwapTotal:           0 kB
SwapFree:            0 kB

And is like this near to Oops time
total: used: free: shared: buffers: cached:

Mem:  473899008 469348352  4550656        0   831488 420454400
Swap:        0        0        0
MemTotal:       462792 kB
MemFree:          4444 kB
MemShared:           0 kB
Buffers:           812 kB
Cached:         410600 kB
SwapCached:          0 kB
Active:           5064 kB
Inactive:       410968 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       462792 kB
LowFree:          4444 kB
SwapTotal:           0 kB
SwapFree:            0 kB
#4 lore raw

Re: 2.4.24 Paging Fault, Source Line located in slab.c, kmem_cache_reap()

Tracked down Oops to source line in kmem_cache_reap.
...............................................................
This Line is number 1784 in my file.........................

		full_free = 0;
		p = searchp->slabs_free.next;
		while (p != &searchp->slabs_free) {  
#if DEBUG
			slabp = list_entry(p, slab_t, list);

			if (slabp->inuse)
				BUG();
#endif
			full_free++;
			p = p->next; <======Oops Here
		}

............................
objdump for above code section:
Oops identified offending instruction at kmem_cache_reap+80 which is af0
............................
add: 8b 4c 24 08 mov 0x8(%esp,1),%ecx

     ae1:       31 db                   xor    %ebx,%ebx
     ae3:       8b 41 10                mov    0x10(%ecx),%eax
     ae6:       89 ca                   mov    %ecx,%edx
     ae8:       83 c2 10                add    $0x10,%edx
     aeb:       39 d0                   cmp    %edx,%eax
     aed:       74 08                   je     af7 <kmem_cache_reap+0x87>
     aef:       90                      nop
     af0:       8b 00                   mov    (%eax),%eax {<=====Oops}
     af2:       43                      inc    %ebx
     af3:       39 d0                   cmp    %edx,%eax
     af5:       75 f9                   jne    af0 <kmem_cache_reap+0x80>
     af7:       8b 44 24 08             mov    0x8(%esp,1),%eax

...............................
I do not know how this part of the kernel works. Have we walked off the end of a list or something? Can anyone help with a theory or still better a fix?

Hopefully thanks in advance,
Ross.

Quoted 154 lines · Ross Dickson
Quoted 7 lines · Ross Dickson
Quoted 6 lines · you
Quoted 5 lines · Ross Dickson

I have an imaging system writing files to removable hard drives. Compact Flash boot with ram drives so I usually have no swap partition or file.

Recently I upgraded kernel from 2.4.20 to 2.4.24.

Is KM18G Pro (nforce2 dual memory mode), AMD 2400XP, Preempt, Low latency, 64Bit jiffies 1000Hz patched.

I found some articles about memory overcommitment, checked the source and saw strict in use for arm systems - no swap- so this time I thought I would try

echo 1 > /proc/sys/vm/overcommit_memory

I got another oops under equivalent circumstances to earlier (no swap).
I ran oops through ksymoops on another machine with same kernel , results below. At this point I think the trigger may be a slow (bad blocks?) 80Gb hard drive the files are being written to. The PCI bus is quite busy with imaging from 3 cameras on two capture cards (bttv and meteor II mc).

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

Can the paging cache be tuned in /proc or somewhere to prevent it being so greedy as to want more memory than the machine has?

Maybe. But you should concentrate on finding where exactly it oopsed.

...snip...

Quoted 13 lines

......I added a 16mb ram drive swap (see earlier posting)
I note memfree stabilises at around 4Mb when running OK, given it only wanted an extra 1Mb cache swap, can I cat something to /proc/sys/vm/????? to force it to stabilise at around 10Mb or 20Mb? Otherwise can I change a constant and recompile kernel to achieve same? It might help give more headroom when the event occurs.

Quoted 6 lines
Quoted 3 lines

Is the quickest fix to give it more ram. I read on another posting that with greater than 512Mb the cache won't grab any more?

Please don't succumb to 'add more RAM' syndrome. 460 megs should be fine for you. I'd say better find the root of the problem.

...snip...

Quoted 2 lines
Quoted 2 lines

vda

Thanks for the response
Regards
Ross.

Was run "mem=450" with this Oops

ksymoops 2.4.8 on i686 2.4.24-rd. Options used
-V (default)
-k /proc/ksyms (default)
-L (specified)
-o /lib/modules/2.4.24-rd/ (default)
-m /boot/System.map (specified)

Unable to handle kernel paging request at virtual address 6a65656a
c0133f20
*pde = 00000000
Oops: 0000
CPU:    0
EIP:    0010:[<c0133f20>]    Not tainted

Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010883
eax: 6a65656a ebx: 000000f5 ecx: c14dfdd4 edx: c14dfde4
esi: 00000000 edi: 00000008 ebp: c14dfe40 esp: dc16df38
ds: 0018 es: 0018 ss: 0018
Process kswapd (pid: 4, stackpage=dc16d000)

Stack: 000001d0 00000001 c14dfdd4 00000000 00000005 00000005 00000020 000001d0 
       c032c4d8 c032c4d8 c0134ebc dc16df84 000001d0 0000003c 00000020 c0134f58 
       dc16df84 dc16c000 00000000 00000000 c032c4d8 dc16c000 c032c400 00000000 
Call Trace:    [<c0134ebc>] [<c0134f58>] [<c01350f6>] [<c0135169>] [<c013529d>]
  [<c0135210>] [<c0105000>] [<c01057db>] [<c0135210>]
Code: 8b 00 43 39 d0 75 f9 8b 44 24 08 89 da 8b 70 24 8b 40 44 89 
Quoted 6 lines
Quoted 6 lines

EIP; c0133f20 <kmem_cache_reap+80/1f0> <=====

ecx; c14dfdd4 <_end+1115228/1e4db4b4>
edx; c14dfde4 <_end+1115238/1e4db4b4>
ebp; c14dfe40 <_end+1115294/1e4db4b4>
esp; dc16df38 <_end+1bda338c/1e4db4b4>

Trace; c0134ebc <shrink_caches+1c/60>
Trace; c0134f58 <try_to_free_pages_zone+58/e0>
Trace; c01350f6 <kswapd_balance_pgdat+56/b0>
Trace; c0135169 <kswapd_balance+19/30>
Trace; c013529d <kswapd+8d/b0>
Trace; c0135210 <kswapd+0/b0>
Trace; c0105000 <_stext+0/0>
Trace; c01057db <arch_kernel_thread+2b/40>
Trace; c0135210 <kswapd+0/b0>

Code; c0133f20 <kmem_cache_reap+80/1f0>
00000000 <_EIP>:
Code; c0133f20 <kmem_cache_reap+80/1f0> <=====
0: 8b 00 mov (%eax),%eax <=====
Code; c0133f22 <kmem_cache_reap+82/1f0>
2: 43 inc %ebx
Code; c0133f23 <kmem_cache_reap+83/1f0>
3: 39 d0 cmp %edx,%eax
Code; c0133f25 <kmem_cache_reap+85/1f0>
5: 75 f9 jne 0 <_EIP>
Code; c0133f27 <kmem_cache_reap+87/1f0>
7: 8b 44 24 08 mov 0x8(%esp,1),%eax
Code; c0133f2b <kmem_cache_reap+8b/1f0>
b: 89 da mov %ebx,%edx
Code; c0133f2d <kmem_cache_reap+8d/1f0>
d: 8b 70 24 mov 0x24(%eax),%esi
Code; c0133f30 <kmem_cache_reap+90/1f0>
10: 8b 40 44 mov 0x44(%eax),%eax
Code; c0133f33 <kmem_cache_reap+93/1f0>
13: 89 00 mov %eax,(%eax)

Mem starts like this when programs have been started. Was run "mem=460" for these mem readings.
total: used: free: shared: buffers: cached:

Mem:  473899008 22151168 451747840        0   327680  7737344
Swap:        0        0        0
MemTotal:       462792 kB
MemFree:        441160 kB
MemShared:           0 kB
Buffers:           320 kB
Cached:           7556 kB
SwapCached:          0 kB
Active:           2320 kB
Inactive:         7072 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       462792 kB
LowFree:        441160 kB
SwapTotal:           0 kB
SwapFree:            0 kB

And is like this near to Oops time
total: used: free: shared: buffers: cached:

Mem:  473899008 469348352  4550656        0   831488 420454400
Swap:        0        0        0
MemTotal:       462792 kB
MemFree:          4444 kB
MemShared:           0 kB
Buffers:           812 kB
Cached:         410600 kB
SwapCached:          0 kB
Active:           5064 kB
Inactive:       410968 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       462792 kB
LowFree:          4444 kB
SwapTotal:           0 kB
SwapFree:            0 kB