Skip to contentSkip to conversation

Large memory application exhuasts buffers during write.

DISCUSSION Robin Holt 4 participants 5 messages Mar 26, 2004

lore
Share & tools
lore

Robin Holt

author
#0 lore raw

We have a large memory application which is being killed by the OOM killer.

This is a 2.4 based kernel with many of the redhat patches applied. Before the application is started, there is approx 350GB of memory free according to top. When the app starts, it mallocs a 300GB buffer, initializes it, does computations into it, and then starts to write it to a disk file.

What we see happen is the first approx 30GBs gets written and then swap starts getting utilized. Once swap has been heavily utilized, the OOM killer kicks in and kills the job.

The application is a vendor provided app and probably cannot be modified. Does anybody have any suggestion on possible changes to make to the kernel to eliminate or significantly reduce the likelihood that the job gets terminated.

Thanks,
Robin Holt

#1 lore raw
Quoted 9 lines · Robin Holt

We have a large memory application which is being killed by the OOM killer.

This is a 2.4 based kernel with many of the redhat patches applied. Before the application is started, there is approx 350GB of memory free according to top. When the app starts, it mallocs a 300GB buffer, initializes it, does computations into it, and then starts to write it to a disk file.

Quoted 3 lines

What we see happen is the first approx 30GBs gets written and then swap starts getting utilized. Once swap has been heavily utilized, the OOM killer kicks in and kills the job.

How many swap do you have? What do you see in top?

Quoted 4 lines

The application is a vendor provided app and probably cannot be modified. Does anybody have any suggestion on possible changes to make to the kernel to eliminate or significantly reduce the likelihood that the job gets terminated.

Signature · 1 line
vda
author
#2 lore raw
Quoted 10 lines · Denis Vlasenko
Quoted 9 lines · Robin Holt

We have a large memory application which is being killed by the OOM killer.

This is a 2.4 based kernel with many of the redhat patches applied. Before the application is started, there is approx 350GB of memory free according to top. When the app starts, it mallocs a 300GB buffer, initializes it, does computations into it, and then starts to write it to a disk file.

Quoted 5 lines
Quoted 3 lines

What we see happen is the first approx 30GBs gets written and then swap starts getting utilized. Once swap has been heavily utilized, the OOM killer kicks in and kills the job.

How many swap do you have? What do you see in top?

I am not sure how much swap is configured or available, but I highly doubt that there is 350GB of swap.

Robin

#3 lore raw
In reply to Robin Holt
Quoted 9 lines · Robin Holt

This is a 2.4 based kernel with many of the redhat patches applied. Before the application is started, there is approx 350GB of memory free according to top. When the app starts, it mallocs a 300GB buffer, initializes it, does computations into it, and then starts to write it to a disk file.

What we see happen is the first approx 30GBs gets written and then swap starts getting utilized. Once swap has been heavily utilized, the OOM killer kicks in and kills the job.

Buffered writes or O_DIRECT? I guess you're doing the former and actually want the latter. Try preloading a tiny library stub that adds O_DIRECT to open for the interesting fds.

#4 lore raw
In reply to Robin Holt

Robin> What we see happen is the first approx 30GBs gets written and then Robin> swap starts getting utilized. Once swap has been heavily utilized, Robin> the OOM killer kicks in and kills the job.

Can you post the output of 'vmstat 1' starting just before your app is
fired up, and until OOM kicks in an kills it off? And more details
on exactly which version of Linux kernel you're running and how the system is configured as well.

John