Large memory application exhuasts buffers during write.
We have a large memory application which is being killed by the OOM killer.
This is a 2.4 based kernel with many of the redhat patches applied. Before the application is started, there is approx 350GB of memory free according to top. When the app starts, it mallocs a 300GB buffer, initializes it, does computations into it, and then starts to write it to a disk file.
What we see happen is the first approx 30GBs gets written and then swap starts getting utilized. Once swap has been heavily utilized, the OOM killer kicks in and kills the job.
The application is a vendor provided app and probably cannot be modified. Does anybody have any suggestion on possible changes to make to the kernel to eliminate or significantly reduce the likelihood that the job gets terminated.
Thanks,
Robin Holt
Quoted 9 lines · Robin Holt
We have a large memory application which is being killed by the OOM killer.
This is a 2.4 based kernel with many of the redhat patches applied. Before the application is started, there is approx 350GB of memory free according to top. When the app starts, it mallocs a 300GB buffer, initializes it, does computations into it, and then starts to write it to a disk file.
Quoted 3 lines
What we see happen is the first approx 30GBs gets written and then swap starts getting utilized. Once swap has been heavily utilized, the OOM killer kicks in and kills the job.
How many swap do you have? What do you see in top?
Quoted 4 lines
The application is a vendor provided app and probably cannot be modified. Does anybody have any suggestion on possible changes to make to the kernel to eliminate or significantly reduce the likelihood that the job gets terminated.
Signature · 1 line
vda
Quoted 10 lines · Denis Vlasenko
Quoted 9 lines · Robin Holt
We have a large memory application which is being killed by the OOM killer.
This is a 2.4 based kernel with many of the redhat patches applied. Before the application is started, there is approx 350GB of memory free according to top. When the app starts, it mallocs a 300GB buffer, initializes it, does computations into it, and then starts to write it to a disk file.
Quoted 5 lines
Quoted 3 lines
What we see happen is the first approx 30GBs gets written and then swap starts getting utilized. Once swap has been heavily utilized, the OOM killer kicks in and kills the job.
How many swap do you have? What do you see in top?
I am not sure how much swap is configured or available, but I highly doubt that there is 350GB of swap.
Robin
Quoted 9 lines · Robin Holt
This is a 2.4 based kernel with many of the redhat patches applied. Before the application is started, there is approx 350GB of memory free according to top. When the app starts, it mallocs a 300GB buffer, initializes it, does computations into it, and then starts to write it to a disk file.
What we see happen is the first approx 30GBs gets written and then swap starts getting utilized. Once swap has been heavily utilized, the OOM killer kicks in and kills the job.
Buffered writes or O_DIRECT? I guess you're doing the former and actually want the latter. Try preloading a tiny library stub that adds O_DIRECT to open for the interesting fds.
Robin> What we see happen is the first approx 30GBs gets written and then Robin> swap starts getting utilized. Once swap has been heavily utilized, Robin> the OOM killer kicks in and kills the job.
Can you post the output of 'vmstat 1' starting just before your app is
fired up, and until OOM kicks in an kills it off? And more details
on exactly which version of Linux kernel you're running and how the system is configured as well.
John