Need assistance in determining memory usage
Quoted 8 lines · Thomas Schenk
We are building an online game system. On some of the systems, there are simulator processes running that each service a player. There may be up to 200 or more of these processes running at any given time and each uses a fairly large amount of memory ... When the simulator processes start swapping, the systems are becoming unstable, performance goes all to hell... It would
Quoted 8 lines
be useful for us to be able to monitor as closely as possible the amount of memory each processes is using and especially to be notified when these processes start using significant amounts of swap, so that we can be prepared to react before the situation gets out of hand. The other reason why we want to collect this data is so that the developers can analyze the process when it starts to swap ...
A few things you might try:
1. Set RLIMIT_DATA or RLIMIT_AS for your processes using ulimit. That should cause malloc() and the like to return NULL or throw an exception if you go over the limit. (RLIMIT_AS doesn't work too well on heavily multithreaded programs, though, because of the many stacks.)
2. Try the no-overcommit patch. That will fail allocation requests that might conceivably cause swapping later. Harsh, but you did want early notification :-)
3. Start reading the kernel source to see how it calculates memory use and enforces the above limits. That helped clarify things for me a bit.
4. If you're feeling really nasty, you could try applying my patch that implements RLIMIT_RSS by killing the process. That's pretty immediate feedback, too :-)
- Dan