Skip to contentSkip to conversation

oom-killer 2.6.8.1

DISCUSSION Anders Saaby 5 participants 12 messages Aug 18, 2004 – Aug 24, 2004

Subsystems (3)
Memory management MGLRU MM reclaim
lore
Share & tools
lore

Anders Saaby

author
#0 lore raw

Hi all,

This is a high-volume NFS server running almost no user-space applications. It serves a handfull of web server NFS clients from a ~700G XFS filesystem.

The machine has about 2.5 GB of RAM and 4G of swap (which is almost not in use
- i may use 5-10 MB total tops). CONFIG_HIGHMEM and CONFIG_HIGHMEM4G enabled, SMP enabled, preempt disabled.

Today the OOM killer kicked in - it seemed that swap was almost unused at the time (which is strange, as that should prevent the OOM killer from kicking in).

Relevant part of the syslog follows (syslogd was killed too eventually):

Aug 18 14:14:52 st1 kernel: oom-killer: gfp_mask=0xd0
Aug 18 14:14:52 st1 kernel: DMA per-cpu:
Aug 18 14:14:52 st1 kernel: cpu 0 hot: low 2, high 6, batch 1
Aug 18 14:14:52 st1 kernel: cpu 0 cold: low 0, high 2, batch 1
Aug 18 14:14:52 st1 kernel: cpu 1 hot: low 2, high 6, batch 1
Aug 18 14:14:52 st1 kernel: cpu 1 cold: low 0, high 2, batch 1
Aug 18 14:14:52 st1 kernel: Normal per-cpu:
Aug 18 14:14:52 st1 kernel: cpu 0 hot: low 32, high 96, batch 16
Aug 18 14:14:52 st1 kernel: cpu 0 cold: low 0, high 32, batch 16
Aug 18 14:14:52 st1 kernel: cpu 1 hot: low 32, high 96, batch 16
Aug 18 14:14:52 st1 kernel: cpu 1 cold: low 0, high 32, batch 16
Aug 18 14:14:52 st1 kernel: HighMem per-cpu:
Aug 18 14:14:52 st1 kernel: cpu 0 hot: low 32, high 96, batch 16
Aug 18 14:14:53 st1 kernel: cpu 0 cold: low 0, high 32, batch 16
Aug 18 14:14:54 st1 kernel: cpu 1 hot: low 32, high 96, batch 16
Aug 18 14:14:54 st1 kernel: cpu 1 cold: low 0, high 32, batch 16
Aug 18 14:14:54 st1 kernel:
Aug 18 14:14:54 st1 kernel: Free pages:      352376kB (348672kB HighMem)
Aug 18 14:14:54 st1 kernel: Active:18220 inactive:319911 dirty:12154
writeback:0 unstable:0 free:88094 slab:219644 mapped:3231 pagetables:121
Aug 18 14:14:54 st1 kernel: DMA free:1880kB min:16kB low:32kB high:48kB 
active:0kB inactive:0kB present:16384kB
Aug 18 14:14:54 st1 kernel: protections[]: 8 476 732
Aug 18 14:14:54 st1 kernel: Normal free:1824kB min:936kB low:1872kB 
high:2808kB active:84kB inactive:284kB present:901120kB
Aug 18 14:14:54 st1 kernel: protections[]: 0 468 724
Aug 18 14:14:54 st1 kernel: HighMem free:348672kB min:512kB low:1024kB 
high:1536kB active:72796kB inactive:1279360kB present:1703788kB
Aug 18 14:14:54 st1 kernel: protections[]: 0 0 256
Aug 18 14:14:54 st1 kernel: DMA: 6*4kB 2*8kB 1*16kB 1*32kB 0*64kB 0*128kB 
1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1880kB
Aug 18 14:14:54 st1 kernel: Normal: 88*4kB 42*8kB 3*16kB 6*32kB 2*64kB 2*128kB 
0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1824kB
Aug 18 14:14:55 st1 kernel: HighMem: 62500*4kB 11396*8kB 409*16kB 0*32kB 
1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 348672kB
Aug 18 14:14:55 st1 kernel: Swap cache: add 6433, delete 6394, find 
35305/35334, race 0+0
Aug 18 14:14:55 st1 kernel: Out of Memory: Killed process 1049 (rpc.statd).

Any ideas?

Try to disable swap? Tune magic VM settings? Any suggestions are highly welcome.

Thank you very much,

Signature · 9 lines
Med venlig hilsen - Best regards - Meilleures salutations

Anders Saaby
Systems Engineer
------------------------------------------------
Cohaesio A/S - Maglebjergvej 5D - DK-2800 Lyngby
Phone: +45 45 880 888 - Fax: +45 45 880 777
Mail: [email protected] - http://www.cohaesio.com
------------------------------------------------
#1 lore raw
Quoted 57 lines · Anders Saaby

Hi all,

This is a high-volume NFS server running almost no user-space applications. It serves a handfull of web server NFS clients from a ~700G XFS filesystem.

The machine has about 2.5 GB of RAM and 4G of swap (which is almost not in use - i may use 5-10 MB total tops). CONFIG_HIGHMEM and CONFIG_HIGHMEM4G enabled, SMP enabled, preempt disabled.

Today the OOM killer kicked in - it seemed that swap was almost unused at the time (which is strange, as that should prevent the OOM killer from kicking in).

Relevant part of the syslog follows (syslogd was killed too eventually):

Aug 18 14:14:52 st1 kernel: oom-killer: gfp_mask=0xd0
Aug 18 14:14:52 st1 kernel: DMA per-cpu:
Aug 18 14:14:52 st1 kernel: cpu 0 hot: low 2, high 6, batch 1
Aug 18 14:14:52 st1 kernel: cpu 0 cold: low 0, high 2, batch 1
Aug 18 14:14:52 st1 kernel: cpu 1 hot: low 2, high 6, batch 1
Aug 18 14:14:52 st1 kernel: cpu 1 cold: low 0, high 2, batch 1
Aug 18 14:14:52 st1 kernel: Normal per-cpu:
Aug 18 14:14:52 st1 kernel: cpu 0 hot: low 32, high 96, batch 16
Aug 18 14:14:52 st1 kernel: cpu 0 cold: low 0, high 32, batch 16
Aug 18 14:14:52 st1 kernel: cpu 1 hot: low 32, high 96, batch 16
Aug 18 14:14:52 st1 kernel: cpu 1 cold: low 0, high 32, batch 16
Aug 18 14:14:52 st1 kernel: HighMem per-cpu:
Aug 18 14:14:52 st1 kernel: cpu 0 hot: low 32, high 96, batch 16
Aug 18 14:14:53 st1 kernel: cpu 0 cold: low 0, high 32, batch 16
Aug 18 14:14:54 st1 kernel: cpu 1 hot: low 32, high 96, batch 16
Aug 18 14:14:54 st1 kernel: cpu 1 cold: low 0, high 32, batch 16
Aug 18 14:14:54 st1 kernel:
Aug 18 14:14:54 st1 kernel: Free pages:      352376kB (348672kB

HighMem) Aug 18 14:14:54 st1 kernel: Active:18220 inactive:319911 dirty:12154 writeback:0 unstable:0 free:88094 slab:219644 mapped:3231 pagetables:121 Aug 18 14:14:54 st1 kernel: DMA free:1880kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB

Aug 18 14:14:54 st1 kernel: protections[]: 8 476 732
Aug 18 14:14:54 st1 kernel: Normal free:1824kB min:936kB low:1872kB
high:2808kB active:84kB inactive:284kB present:901120kB
Aug 18 14:14:54 st1 kernel: protections[]: 0 468 724
Aug 18 14:14:54 st1 kernel: HighMem free:348672kB min:512kB
low:1024kB high:1536kB active:72796kB inactive:1279360kB
present:1703788kB Aug 18 14:14:54 st1 kernel: protections[]: 0 0
256
Aug 18 14:14:54 st1 kernel: DMA: 6*4kB 2*8kB 1*16kB 1*32kB 0*64kB
0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1880kB
Aug 18 14:14:54 st1 kernel: Normal: 88*4kB 42*8kB 3*16kB 6*32kB
2*64kB 2*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1824kB
Aug 18 14:14:55 st1 kernel: HighMem: 62500*4kB 11396*8kB 409*16kB
0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB =
348672kB Aug 18 14:14:55 st1 kernel: Swap cache: add 6433, delete
6394, find 35305/35334, race 0+0
Aug 18 14:14:55 st1 kernel: Out of Memory: Killed process 1049
Quoted 8 lines

(rpc.statd).

Any ideas?

Try to disable swap? Tune magic VM settings? Any suggestions are highly welcome.

Thank you very much,

You may be another candidate for the same thing thats troubleing me. Back up the log above that and look for an Oops and post that please.

Signature · 8 lines
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.24% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.
#2 lore raw

On Wed, Aug 18, 2004 at 09:57:03AM -0400, Gene Heskett wrote:
...

Quoted 9 lines

Any ideas?

Quoted 5 lines

Try to disable swap? Tune magic VM settings? Any suggestions are highly welcome.

Thank you very much,

You may be another candidate for the same thing thats troubleing me. Back up the log above that and look for an Oops and post that please.

Looking thru the swapfile.c and oom killer code, one thing that is making me scratch my head:

nr_swap_pages is a *signed* integer. This does not make sense. There are even tests in swapfile.c that explicitly test "nr_swap_pages <= 0" instead of simply "!nr_swap_pages" - this does not make sense at all either - or does it?

Stephen is that your code?

See, if nr_swap_pages can validly be negative and some meaning is attached to that (some meaning other than "we're out of swap"), the oom_killer surely misses that, as it tests "nr_swap_pages > 0".

I don't think that nr_swap_pages can be negative (unless one adds a *lot* of swap in which case this will unintentionally happen all by itself), but I felt I should chirp in with this comment in case someone's looking at it anyway :)

Signature · 1 line
 / jakob
#3 lore raw
Quoted 10 lines · Jakob Oestergaard

Looking thru the swapfile.c and oom killer code, one thing that is making me scratch my head:

nr_swap_pages is a *signed* integer. This does not make sense. There are even tests in swapfile.c that explicitly test "nr_swap_pages <= 0" instead of simply "!nr_swap_pages" - this does not make sense at all either - or does it?

Stephen is that your code?

I'm not Stephen, and it wasn't originally my code, but I do remember tidying this up to stop /proc/meminfo showing negative SwapFree (see nr_to_be_unused).

nr_swap_pages _may_ legitimately be negative, during sys_swapoff:
that does "nr_swap_pages -= p->pages", which is liable to send it negative, before going on to "try_to_unuse", which slowly increments nr_swap_pages back up to its final value (0 if no other swap areas), page by page via delete_from_swap_cache's swap_free.

Surprising, I agree, but it allows swap_free to increment nr_swap_pages without any special casing for swapoff.

Hugh

Quoted 8 lines

See, if nr_swap_pages can validly be negative and some meaning is attached to that (some meaning other than "we're out of swap"), the oom_killer surely misses that, as it tests "nr_swap_pages > 0".

I don't think that nr_swap_pages can be negative (unless one adds a *lot* of swap in which case this will unintentionally happen all by itself), but I felt I should chirp in with this comment in case someone's looking at it anyway :)

author
#4 lore raw
In reply to Gene Heskett
Quoted 62 lines · Gene Heskett
Quoted 61 lines · Anders Saaby

Hi all,

This is a high-volume NFS server running almost no user-space applications. It serves a handfull of web server NFS clients from a ~700G XFS filesystem.

The machine has about 2.5 GB of RAM and 4G of swap (which is almost not in use - i may use 5-10 MB total tops). CONFIG_HIGHMEM and CONFIG_HIGHMEM4G enabled, SMP enabled, preempt disabled.

Today the OOM killer kicked in - it seemed that swap was almost unused at the time (which is strange, as that should prevent the OOM killer from kicking in).

Relevant part of the syslog follows (syslogd was killed too eventually):

Aug 18 14:14:52 st1 kernel: oom-killer: gfp_mask=0xd0
Aug 18 14:14:52 st1 kernel: DMA per-cpu:
Aug 18 14:14:52 st1 kernel: cpu 0 hot: low 2, high 6, batch 1
Aug 18 14:14:52 st1 kernel: cpu 0 cold: low 0, high 2, batch 1
Aug 18 14:14:52 st1 kernel: cpu 1 hot: low 2, high 6, batch 1
Aug 18 14:14:52 st1 kernel: cpu 1 cold: low 0, high 2, batch 1
Aug 18 14:14:52 st1 kernel: Normal per-cpu:
Aug 18 14:14:52 st1 kernel: cpu 0 hot: low 32, high 96, batch 16
Aug 18 14:14:52 st1 kernel: cpu 0 cold: low 0, high 32, batch 16
Aug 18 14:14:52 st1 kernel: cpu 1 hot: low 32, high 96, batch 16
Aug 18 14:14:52 st1 kernel: cpu 1 cold: low 0, high 32, batch 16
Aug 18 14:14:52 st1 kernel: HighMem per-cpu:
Aug 18 14:14:52 st1 kernel: cpu 0 hot: low 32, high 96, batch 16
Aug 18 14:14:53 st1 kernel: cpu 0 cold: low 0, high 32, batch 16
Aug 18 14:14:54 st1 kernel: cpu 1 hot: low 32, high 96, batch 16
Aug 18 14:14:54 st1 kernel: cpu 1 cold: low 0, high 32, batch 16
Aug 18 14:14:54 st1 kernel:
Aug 18 14:14:54 st1 kernel: Free pages:      352376kB (348672kB

HighMem) Aug 18 14:14:54 st1 kernel: Active:18220 inactive:319911 dirty:12154 writeback:0 unstable:0 free:88094 slab:219644 mapped:3231 pagetables:121 Aug 18 14:14:54 st1 kernel: DMA free:1880kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB

Aug 18 14:14:54 st1 kernel: protections[]: 8 476 732
Aug 18 14:14:54 st1 kernel: Normal free:1824kB min:936kB low:1872kB
high:2808kB active:84kB inactive:284kB present:901120kB
Aug 18 14:14:54 st1 kernel: protections[]: 0 468 724
Aug 18 14:14:54 st1 kernel: HighMem free:348672kB min:512kB
low:1024kB high:1536kB active:72796kB inactive:1279360kB
present:1703788kB Aug 18 14:14:54 st1 kernel: protections[]: 0 0
256
Aug 18 14:14:54 st1 kernel: DMA: 6*4kB 2*8kB 1*16kB 1*32kB 0*64kB
0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1880kB
Aug 18 14:14:54 st1 kernel: Normal: 88*4kB 42*8kB 3*16kB 6*32kB
2*64kB 2*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1824kB
Aug 18 14:14:55 st1 kernel: HighMem: 62500*4kB 11396*8kB 409*16kB
0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB =
348672kB Aug 18 14:14:55 st1 kernel: Swap cache: add 6433, delete
6394, find 35305/35334, race 0+0
Aug 18 14:14:55 st1 kernel: Out of Memory: Killed process 1049

(rpc.statd).

Any ideas?

Quoted 7 lines
Quoted 4 lines

Try to disable swap? Tune magic VM settings? Any suggestions are highly welcome.

Thank you very much,

You may be another candidate for the same thing thats troubleing me. Back up the log above that and look for an Oops and post that please.

No Oops :-)

/Saaby

#5 lore raw
In reply to Anders Saaby
Quoted 9 lines · Anders Saaby

This is a high-volume NFS server running almost no user-space applications. It serves a handfull of web server NFS clients from a ~700G XFS filesystem. The machine has about 2.5 GB of RAM and 4G of swap (which is almost not in use - i may use 5-10 MB total tops). CONFIG_HIGHMEM and CONFIG_HIGHMEM4G enabled, SMP enabled, preempt disabled. Today the OOM killer kicked in - it seemed that swap was almost unused at the time (which is strange, as that should prevent the OOM killer from kicking in).
Relevant part of the syslog follows (syslogd was killed too eventually):

This seems to have been meant to resolve laptop_mode issues, but looks like it didn't get applied. I'm not convinced it will help given that you appear to have a vanilla ZONE_NORMAL slab OOM (858MB slab), but you never know. Capturing /proc/slabinfo data may be more helpful.

mm/vmscan.c +2 -1
Index: oom-2.6.8-rc1/mm/vmscan.c
===================================================================
@@ -417,7 +417,8 @@
 				goto keep_locked;
 			if (!may_enter_fs)
 				goto keep_locked;
-			if (laptop_mode && !sc->may_writepage)
+			if (laptop_mode && !sc->may_writepage &&
+							!PageSwapCache(page))
 				goto keep_locked;
 
 			/* Page is dirty, try to write it out here */
author
#6 lore raw
Quoted 22 lines · William Lee Irwin III
Quoted 9 lines · Anders Saaby

This is a high-volume NFS server running almost no user-space applications. It serves a handfull of web server NFS clients from a ~700G XFS filesystem. The machine has about 2.5 GB of RAM and 4G of swap (which is almost not in use - i may use 5-10 MB total tops). CONFIG_HIGHMEM and CONFIG_HIGHMEM4G enabled, SMP enabled, preempt disabled. Today the OOM killer kicked in - it seemed that swap was almost unused at the time (which is strange, as that should prevent the OOM killer from kicking in).
Relevant part of the syslog follows (syslogd was killed too eventually):

This seems to have been meant to resolve laptop_mode issues, but looks like it didn't get applied. I'm not convinced it will help given that you appear to have a vanilla ZONE_NORMAL slab OOM (858MB slab), but you never know. Capturing /proc/slabinfo data may be more helpful.

Index: oom-2.6.8-rc1/mm/vmscan.c
===================================================================

mm/vmscan.c line 424

Quoted 9 lines
mm/vmscan.c +2 -1
@@ -417,7 +417,8 @@
 				goto keep_locked;
 			if (!may_enter_fs)
 				goto keep_locked;
-			if (laptop_mode && !sc->may_writepage)
+			if (laptop_mode && !sc->may_writepage &&
+							!PageSwapCache(page))
 				goto keep_locked;
 
 			/* Page is dirty, try to write it out here */

laptop_mode is not set on this server <- :-)

- So I guess this is not relevant for my setup?

/Saaby

#7 lore raw
Quoted 14 lines · William Lee Irwin III
Quoted 14 lines

Index: oom-2.6.8-rc1/mm/vmscan.c
===================================================================

mm/vmscan.c +2 -1
@@ -417,7 +417,8 @@
 				goto keep_locked;
 			if (!may_enter_fs)
 				goto keep_locked;
-			if (laptop_mode && !sc->may_writepage)
+			if (laptop_mode && !sc->may_writepage &&
+							!PageSwapCache(page))
 				goto keep_locked;
 
 			/* Page is dirty, try to write it out here */
Quoted 2 lines · Anders Saaby

laptop_mode is not set on this server <- :-)
- So I guess this is not relevant for my setup?

Probably not. Please try to collect /proc/slabinfo snapshots while the system is still functional as it degrades.

-- wli

author
#8 lore raw
Quoted 14 lines · William Lee Irwin III
Quoted 13 lines · William Lee Irwin III
Quoted 13 lines

Index: oom-2.6.8-rc1/mm/vmscan.c
===================================================================

mm/vmscan.c +2 -1
@@ -417,7 +417,8 @@
 				goto keep_locked;
 			if (!may_enter_fs)
 				goto keep_locked;
-			if (laptop_mode && !sc->may_writepage)
+			if (laptop_mode && !sc->may_writepage &&
+							!PageSwapCache(page))
 				goto keep_locked;
Quoted 9 lines
Quoted 1 line
Quoted 1 line
mm/vmscan.c +0 -0
 			/* Page is dirty, try to write it out here */
Quoted 2 lines · Anders Saaby

laptop_mode is not set on this server <- :-)
- So I guess this is not relevant for my setup?

Probably not. Please try to collect /proc/slabinfo snapshots while the system is still functional as it degrades.

OK, I am now collecting /proc/slabinfo every hour to some logfiles - I will send you the results when I have some interesting data.

/Saaby

Quoted 2 lines

-- wli

author
#9 lore raw
In reply to William Lee Irwin III

OK - I now have some additional info regarding the slapinfo/oom-killer issue.

As I wrote earlier this server is a storage server providing NFS storage to a number of webservers - ondisk filesystem is xfs, kernel = 2.6.8.1. At 03:00 some logrotate scripts runs throug a lot of files. It appears that this is what is using the slabs (see this graph, K = M, so max used slab is approx. 700M)

http://saaby.com/slabused.gif (values are from /proc/meminfo)

These are the values (from slabinfo, active_objs), which changed remarkably from 03:00 to 06:00:

03:00: 06:00:

xfs_chashlist  91297    xfs_chashlist     151994
xfs_inode     243791    xfs_inode         586780
linvfs_icache 243791    linvfs_icache     586807
dentry_cache  196033    dentry_cache      430609

The server crashed every night at approx. 03:00 to 04:00 - until last night where we changed:

vm.min_free_kbytes from default (approx. 900K) to vm.min_free_kbytes=32768 (32M)

This seems to solve the problem - Does this make any sense to you? - Or just pure luck?

/Saaby

Quoted 25 lines · William Lee Irwin III
Quoted 14 lines · William Lee Irwin III
Quoted 14 lines

Index: oom-2.6.8-rc1/mm/vmscan.c
===================================================================

mm/vmscan.c 2004-07-14 06:22:15.986416200 -0700 renamed from mm/vmscan.c 2004-07-14 06:17:13.876343912 -0700 +2 -1
@@ -417,7 +417,8 @@
     goto keep_locked;
    if (!may_enter_fs)
     goto keep_locked;
-   if (laptop_mode && !sc->may_writepage)
+   if (laptop_mode && !sc->may_writepage &&
+       !PageSwapCache(page))
     goto keep_locked;
 
    /* Page is dirty, try to write it out here */
Quoted 2 lines · Anders Saaby

laptop_mode is not set on this server <- :-)
- So I guess this is not relevant for my setup?

Probably not. Please try to collect /proc/slabinfo snapshots while the system is still functional as it degrades.

-- wli

#10 lore raw
Quoted 11 lines · Anders Saaby

OK - I now have some additional info regarding the slapinfo/oom-killer issue.
As I wrote earlier this server is a storage server providing NFS storage to a number of webservers - ondisk filesystem is xfs, kernel = 2.6.8.1. At 03:00 some logrotate scripts runs throug a lot of files. It appears that this is what is using the slabs (see this graph, K = M, so max used slab is approx. 700M)
http://saaby.com/slabused.gif (values are from /proc/meminfo) These are the values (from slabinfo, active_objs), which changed remarkably from 03:00 to 06:00:
03:00: 06:00:

Quoted 4 lines
xfs_chashlist  91297    xfs_chashlist     151994
xfs_inode     243791    xfs_inode         586780
linvfs_icache 243791    linvfs_icache     586807
dentry_cache  196033    dentry_cache      430609

xfs has some known bad slab behavior. I think punting this in the direction of xfs mailing lists may be useful.

Quoted 6 lines · Anders Saaby

The server crashed every night at approx. 03:00 to 04:00 - until last night where we changed:
vm.min_free_kbytes from default (approx. 900K) to vm.min_free_kbytes=32768 (32M)
This seems to solve the problem - Does this make any sense to you? - Or just pure luck?

I guess it makes some sense since it refuses to let slab cut into the very last bits of RAM. If you're getting temporarily heavily fragmented with active references it may mean the difference between the box livelocking/deadlocking and making forward progress.

-- wli

author
#11 lore raw
Quoted 17 lines · William Lee Irwin III
Quoted 11 lines · Anders Saaby

OK - I now have some additional info regarding the slapinfo/oom-killer issue.
As I wrote earlier this server is a storage server providing NFS storage to a number of webservers - ondisk filesystem is xfs, kernel = 2.6.8.1. At 03:00 some logrotate scripts runs throug a lot of files. It appears that this is what is using the slabs (see this graph, K = M, so max used slab is approx. 700M)
http://saaby.com/slabused.gif (values are from /proc/meminfo) These are the values (from slabinfo, active_objs), which changed remarkably from 03:00 to 06:00:
03:00: 06:00:

Quoted 4 lines
xfs_chashlist  91297    xfs_chashlist     151994
xfs_inode     243791    xfs_inode         586780
linvfs_icache 243791    linvfs_icache     586807
dentry_cache  196033    dentry_cache      430609
Quoted 3 lines

xfs has some known bad slab behavior. I think punting this in the direction of xfs mailing lists may be useful.

OK, interesting! - I will do that.

Quoted 8 lines · Anders Saaby
Quoted 6 lines

The server crashed every night at approx. 03:00 to 04:00 - until last night where we changed:
vm.min_free_kbytes from default (approx. 900K) to vm.min_free_kbytes=32768 (32M)
This seems to solve the problem - Does this make any sense to you? - Or just pure luck?

Quoted 5 lines

I guess it makes some sense since it refuses to let slab cut into the very last bits of RAM. If you're getting temporarily heavily fragmented with active references it may mean the difference between the box livelocking/deadlocking and making forward progress.

Sounds right.

Thanks for your help! :-)

/Saaby

Quoted 2 lines

-- wli