Need assistance in determining memory usage
Greetings all.
I have been asked a question by some of the developers in my organization and after searching Google, scouring the Linux newsgroups, and searching as many mailing list archives and howtos as I could find, I still cannot find a satisfactory answer to the following question:
Q. How can you determine how much memory a process is using at a given point in time? Specifically, I want to know of a method or tool that will tell me how much total memory a process is using, how much of that total is shared with other processes, how much is resident, and how much is swapped out.
Please don't say to just use ps or top, because if either of these tools was adequate, I wouldn't be asking here and every reference I could find indicates that this is not a trivial problem. There were also indications I found while searching that these tools do not always report memory numbers accurately. If there is a way to determine this information using /proc, this would be ideal, since I could then conceivably create a script or simple program that could determine the answer given the process ID, which is what the developers here really want.
If you can assist me in determining the answer to this question, I would greatly appreciate it. Please note that I am far from being a kernel expert (being just a lowly systems administrator), so please keep make your explanations as detailed as possible.
Tom S.
Signature · 5 lines
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ | Tom Schenk | A positive attitude may not solve all your | | Online Ops | problems, but it will annoy enough people to | | [email protected] | make it worth the effort. -- Herm Albright | +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
Quoted 2 lines · Thomas Schenk
Please don't say to just use ps or top, because if either of these tools was adequate, I wouldn't be asking here and every reference I could find
(...)
Quoted 2 lines
expert (being just a lowly systems administrator), so please keep make your explanations as detailed as possible.
If you could state your question as detailed as possible. Please explain how 'ps' and 'top' are inadequate.
Regards,
bert
Signature · 2 lines
http://www.PowerDNS.com Versatile DNS Software & Services http://lartc.org Linux Advanced Routing & Traffic Control HOWTO
Quoted 5 lines · Thomas Schenk
Q. How can you determine how much memory a process is using at a given point in time? Specifically, I want to know of a method or tool that will tell me how much total memory a process is using, how much of that total is shared with other processes, how much is resident, and how much is swapped out.
/proc/<pid>/{stat,statm,status} are probably helpful. There's some info on this in Documentation/filesystems/proc.txt
Ketil
In /proc there are a bunch of directories that are numbered. These numbers correspond to the PID of each process. In each of these directories is a file called "status".
This file contains most all the information you need just in case you don't like ps or top.
Name: init
State: S (sleeping)
Tgid: 1
Pid: 1
PPid: 0
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 32
Groups:
VmSize: 224 kB Total virtual memory accessed so far.
VmLck: 0 kB Locks held
VmRSS: 148 kB Resident virtual memory (not swapped)
VmData: 28 kB Data virtual memory
VmStk: 4 kB Stack virtual memory VmExe: 184 kB Executable virtual memory VmLib: 0 kB Shared library
SigPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000057f0defc
SigCgt: 00000000280b2003
CapInh: 0000000000000000
CapPrm: 00000000ffffffff
CapEff: 00000000fffffeff
Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
Bush : The Fourth Reich of America
Quoted 8 lines · Thomas Schenk
was adequate, I wouldn't be asking here and every reference I could find indicates that this is not a trivial problem. There were also indications I found while searching that these tools do not always report memory numbers accurately. If there is a way to determine this information using /proc, this would be ideal, since I could then conceivably create a script or simple program that could determine the answer given the process ID, which is what the developers here really want.
Neither the question nor the answer are trivial. What are you trying to do with the data may be the most relevant question
Quoted 10 lines · Alan Cox
Quoted 8 lines · Thomas Schenk
was adequate, I wouldn't be asking here and every reference I could find indicates that this is not a trivial problem. There were also indications I found while searching that these tools do not always report memory numbers accurately. If there is a way to determine this information using /proc, this would be ideal, since I could then conceivably create a script or simple program that could determine the answer given the process ID, which is what the developers here really want.
Quoted 2 lines
Neither the question nor the answer are trivial. What are you trying to do with the data may be the most relevant question
This situation is this:
We are building an online game system. On some of the systems, there are simulator processes running that each service a player. There may be up to 200 or more of these processes running at any given time and each uses a fairly large amount of memory (as reported by ps). Part of this is due to the fact that the processes have not been optimized to make the most efficient use of memory. When the simulator processes start swapping, then the systems are becoming unstable, performance goes all to hell and sometimes the systems totally hang. It would be useful for us to be able to monitor as closely as possible the amount of memory each processes is using and especially to be notified when these processes start using significant amounts of swap, so that we can be prepared to react before the situation gets out of hand. The other reason why we want to collect this data is so that the developers can analyze the process when it starts to swap and determine if there optimizations that they can make that will improve the memory utilization of these processes so that more processes can be run on the same box and that swap usage is minimized.
A couple of points that may be useful (or not):
1. These systems are based on RedHat 7.2, but are running a kernel built from the kernel source tree for Advanced Server 2.1 (as obtained from the kernel source RPM for the 2.4.9.e-3 kernel). Originally, they were running on the 2.4.18 kernel from RedHat 7.3, but in our particular situation, the 2.4.9 Advanced Server kernel was found to have better performance characteristics.
2. Each of the systems running the simulator processes consist of the following:
Dual P4 Xeon 2.5 GHz processors (hyperthreading is enabled in the
motherboard BIOS setup)
4 Gigs of RAM
3Ware RAID controller
2 x 40 GB disks in RAID 1 configuration
2 x E1000 NICS
3. The only modifications that were made to the 2.4.9 AS kernel was to update to the latest version of the E1000 driver from Intel, since the one in the AS 2.1 kernel source tree didn't work with the systems. The reason why the kernel was compiled locally was to remove unwanted options, such as USB and sound support, and to eliminate the need for an initrd.
If further data is required, I can provide it. Thanks for all who have responded thus far.
Tom S.
Signature · 5 lines
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ | Tom Schenk | A positive attitude may not solve all your | | Online Ops | problems, but it will annoy enough people to | | [email protected] | make it worth the effort. -- Herm Albright | +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
Quoted 16 lines · Thomas Schenk
Quoted 10 lines · Alan Cox
Quoted 8 lines · Thomas Schenk
was adequate, I wouldn't be asking here and every reference I could find indicates that this is not a trivial problem. There were also indications I found while searching that these tools do not always report memory numbers accurately. If there is a way to determine this information using /proc, this would be ideal, since I could then conceivably create a script or simple program that could determine the answer given the process ID, which is what the developers here really want.
Quoted 2 lines
Neither the question nor the answer are trivial. What are you trying to do with the data may be the most relevant question
This situation is this:
Quoted 12 lines
We are building an online game system. On some of the systems, there are simulator processes running that each service a player. There may be up to 200 or more of these processes running at any given time and each uses a fairly large amount of memory (as reported by ps). Part of this is due to the fact that the processes have not been optimized to make the most efficient use of memory. When the simulator processes start swapping, then the systems are becoming unstable, performance goes all to hell and sometimes the systems totally hang. It would be useful for us to be able to monitor as closely as possible the amount of memory each processes is using and especially to be notified when these processes start using significant amounts of swap, so that we can be prepared to react before the situation gets out of hand.
I do not believe that the kernel exports the information "what processes are using swap?". You can answer some of your questions by using my pmap program; it's in at least some recent procps packages, or download the source:
http://web.hexapodia.org/~adi/pmap.c
-andy