Skip to contentSkip to conversation

How do you accurately determine a process' RAM usage?

DISCUSSION [email protected] 4 participants 10 messages Jul 6, 2005 – Jul 20, 2005

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I wrote a tool to report how much RAM a
particular program (apache for e.g.) was using:
http://www.pixelbeat.org/scripts/ps_mem.py

I was then pointed at the following:
http://wiki.apache.org/spamassassin/TopSharedMemoryBug
which describes how copy-on-write pages are
not counted as shared since 2.6.

So how can one determine how much RAM a process is using? Seems like a fundamental requirement to me.
Could we add a "SharedTotal" column to /proc/$$/statm for e.g. ?

Here's a little python script to demo the problem:

#!/usr/bin/python
import os, sys, time
a="\x00"*(32*1024*1024) #alloc 32MiB
pid=os.fork()
if(not pid):

    time.sleep(1)
    sys.exit()

sizel=map(float,open("/proc/"+str(pid)+"/statm").readlines()[0].split()[1:3]) print "only %.1f%% reported as shared" % ((sizel[1]/sizel[0])*100) os.wait()

Signature · 2 lines
Pádraig Brady - http://www.pixelbeat.org
--
#1 lore raw
Quoted 5 lines · [email protected] wrote:

I wrote a tool to report how much RAM a
particular program (apache for e.g.) was using:
http://www.pixelbeat.org/scripts/ps_mem.py

Quoted 8 lines

I was then pointed at the following:
http://wiki.apache.org/spamassassin/TopSharedMemoryBug
which describes how copy-on-write pages are
not counted as shared since 2.6.

So how can one determine how much RAM a process is using? Seems like a fundamental requirement to me.
Could we add a "SharedTotal" column to /proc/$$/statm for e.g. ?

Calculating this stuff accurately is very expensive. You'll get a better answer using proc-pid-smaps.patch from -mm, but even that won't tell you things about sharing levels of the pages.

#2 lore raw
Quoted 3 lines · Andrew Morton

Calculating this stuff accurately is very expensive. You'll get a better answer using proc-pid-smaps.patch from -mm, but even that won't tell you things about sharing levels of the pages.

Great, thanks! I'll play around with this:
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.13-rc1/2.6.13-rc1-mm1/broken-out/proc-pid-smaps.patch
Looks like it's been stable for 4 months?

Given that it's an independent /proc/$pid/smaps file, it only needs to be queried when required and so
I wouldn't worry too much about cost. top wouldn't use it for e.g., but specialised tools like mine would.

Signature · 2 lines
Pádraig Brady - http://www.pixelbeat.org
--
#3 lore raw
Quoted 8 lines · [email protected] wrote:
Quoted 3 lines · Andrew Morton

Calculating this stuff accurately is very expensive. You'll get a better answer using proc-pid-smaps.patch from -mm, but even that won't tell you things about sharing levels of the pages.

Great, thanks! I'll play around with this:
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.13-rc1/2.6.13-rc1-mm1/broken-out/proc-pid-smaps.patch

OK, please let us know how it goes.

Quoted 1 line

Looks like it's been stable for 4 months?

yup, although I don't think it's been used much.

Quoted 4 lines

Given that it's an independent /proc/$pid/smaps file, it only needs to be queried when required and so
I wouldn't worry too much about cost. top wouldn't use it for e.g., but specialised tools like mine would.

I agree, but people get upset ;)

Plus some userspace tool developer might see it and start using it without knowing the cost on big iron.

#4 lore raw
Quoted 5 lines · Andrew Morton
Quoted 1 line · [email protected] wrote:

Looks like it's been stable for 4 months?

yup, although I don't think it's been used much.

Just a sidenote to say I should be sending you an update to it (the /proc/$pid/smaps code) in the next couple of days, but merely cosmetic ("map" -> "vma", cond_resched_lock, p?d_addr_next ptwalking).

I'm still pretty sceptical about it, but it's probably a useful framework for people to hack on, to report whatever kind of page numbers they're interested in for this or that investigation.

Hugh

#5 lore raw
In reply to Andrew Morton
Quoted 1 line · Andrew Morton

OK, please let us know how it goes.

It went very well. I could find no problems at all.
I've updated my script to use the new method, so please merge smaps :)
http://www.pixelbeat.org/scripts/ps_mem.py

Usually the shared mem reported by /proc/$$/statm
is the same as summing all the shared values in in /proc/$$/smaps but there can be large discrepancies.
In the real world you can see this with a newly started apache. On my system statm reported that apache was using 35MB, whereas smaps reported the correct amount of 11MB. These numbers will converge over time as the unused pages of the apache processes were swapped out.
Attached is a mem.py script that triggers and reports the memory discrepancy on an smaps enabled system.

As for the overhead of calling smaps, well comparing to /proc/$$/maps which does essentially the same thing only not walking mem, on my 1.6GHz P4, 384MB laptop, gives:

time read_10K_smaps

real    0m3.323s
user    0m0.636s
sys     0m2.660s

time read_10K_maps

real    0m1.742s
user    0m0.300s
sys     0m1.428s

I've also attached a patch to make smaps output match the maps output, as smaps was not showing info like which map was heap and stack etc. The code is smaller also, as show_smap() now just calls show_map(). Note this patch is to be applied _after_ Hugh's changes.

Signature · 2 lines
Pádraig Brady - http://www.pixelbeat.org
--

#!/usr/bin/python
import os, sys, time
PAGESIZE=os.sysconf("SC_PAGE_SIZE")/1024 #KiB

#stat = {Shared,Rss}
#file = {smaps,statm}
def getMemStat(pid,stat,file):

    if file=="smaps":
        shared_lines=[line
                      for line in open("/proc/"+str(pid)+"/smaps").readlines()
                      if line.find(stat)!=-1]
        return sum([int(line.split()[1]) for line in shared_lines])
    else:
        if stat == "Shared": stat_index=2
        elif stat == "Rss": stat_index=1
        return int(open("/proc/"+str(pid)+"/statm").readline().split()[stat_index])*PAGESIZE

a="\x00"*(32*1024*1024) #alloc 32MiB
pid=os.fork()
if(not pid):

     time.sleep(1)
     sys.exit()

for file in ["statm", "smaps"]:

    rss=float(getMemStat(pid,"Rss",file))
    shr=float(getMemStat(pid,"Shared",file))
    print "/proc/%d/%s reports %.1f%% shared" % (pid,file,((shr/rss)*100))

os.wait()

fs/proc/task_mmu.c +5 -16
@@ -121,9 +121,9 @@
 	 * Print the dentry name for named mappings, and a
 	 * special [heap] marker for the heap:
 	 */
-	if (vma->vm_file) {
+	if (file) {
 		pad_len_spaces(m, len);
-		seq_path(m, file->f_vfsmnt, file->f_dentry, "");
+		seq_path(m, file->f_vfsmnt, file->f_dentry, "\n");
 	} else {
 		if (mm) {
 			if (vma->vm_start <= mm->start_brk &&
@@ -245,32 +245,21 @@
 static int show_smap(struct seq_file *m, void *v)
 {
 	struct vm_area_struct *vma = v;
-	struct file *file = vma->vm_file;
-	int flags = vma->vm_flags;
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long vma_len = (vma->vm_end - vma->vm_start);
 	struct mem_size_stats mss;
 
 	memset(&mss, 0, sizeof mss);
 
+	show_map(m, v);
+
 	if (mm) {
 		spin_lock(&mm->page_table_lock);
 		smaps_pgd_range(vma, vma->vm_start, vma->vm_end, &mss);
 		spin_unlock(&mm->page_table_lock);
 	}
 
-	seq_printf(m, "%08lx-%08lx %c%c%c%c ",
-		   vma->vm_start,
-		   vma->vm_end,
-		   flags & VM_READ ? 'r' : '-',
-		   flags & VM_WRITE ? 'w' : '-',
-		   flags & VM_EXEC ? 'x' : '-',
-		   flags & VM_MAYSHARE ? 's' : 'p');
-
-	if (vma->vm_file)
-		seq_path(m, file->f_vfsmnt, file->f_dentry, " \t\n\\");
-
-	seq_printf(m, "\n"
+	seq_printf(m,
 		   "Size:          %8lu KB\n"
 		   "Rss:           %8lu KB\n"
 		   "Shared_Clean:  %8lu KB\n"
#6 lore raw

Hi,

Quoted 10 lines · On 7/12/05, [email protected] <[email protected]> wrote:
Quoted 1 line · Andrew Morton

OK, please let us know how it goes.

It went very well. I could find no problems at all.
I've updated my script to use the new method, so please merge smaps :)
http://www.pixelbeat.org/scripts/ps_mem.py

Usually the shared mem reported by /proc/$$/statm
is the same as summing all the shared values in in /proc/$$/smaps but there can be large discrepancies.

Have you checked how the statm shared is calculated? I guess it does something like:
shared = mm->rss - mm->anon_rss

But in smaps output you can have anonymous area like:

b6e0e000-b6e13000 rw-p

Size:                20 KB
Rss:                  4 KB
Shared_Clean:         0 KB
Shared_Dirty:         4 KB
Private_Clean:        0 KB
Private_Dirty:        0 KB

Look that it presents 4 KB of shared value in area considered anonymous.

ANDREW: anon_rss is the rss for anonymous area, right?

Quoted 3 lines

In the real world you can see this with a newly started apache. On my system statm reported that apache was using 35MB, whereas smaps reported the correct amount of 11MB.

How dou you know that 11MB is the correct shared value and the 35MB is the wrong value?

BR,

Mauricio Lin.

#7 lore raw
Quoted 12 lines · Mauricio Lin

Hi,

Quoted 8 lines · On 7/12/05, [email protected] <[email protected]> wrote:
Quoted 1 line · Andrew Morton

OK, please let us know how it goes.

It went very well. I could find no problems at all.
I've updated my script to use the new method, so please merge smaps :)
http://www.pixelbeat.org/scripts/ps_mem.py

Quoted 8 lines
Quoted 3 lines

Usually the shared mem reported by /proc/$$/statm
is the same as summing all the shared values in in /proc/$$/smaps but there can be large discrepancies.

Have you checked how the statm shared is calculated? I guess it does something like:
shared = mm->rss - mm->anon_rss

yes

Quoted 10 lines

But in smaps output you can have anonymous area like:

b6e0e000-b6e13000 rw-p

Size:                20 KB
Rss:                  4 KB
Shared_Clean:         0 KB
Shared_Dirty:         4 KB
Private_Clean:        0 KB
Private_Dirty:        0 KB
Quoted 3 lines

Look that it presents 4 KB of shared value in area considered anonymous.

ANDREW: anon_rss is the rss for anonymous area, right?

I see your point and I'm not sure.
The following shell gets the shared values for the first httpd process:

FIRST_HTTPD=ps -C httpd -o pid= | head -1 | tr -d ' '
HTTPD_STATM_SHARED=$(expr 4 '*' cut -f3 -d' ' /proc/$FIRST_HTTPD/statm) HTTPD_SMAPS_SHARED=$(grep Shared /proc/$FIRST_HTTPD/smaps | tr -s ' ' | cut -f2 -d' ' | ( tr '\n' +; echo 0 ) | bc)

This shows that "smaps" reports 3060 KB more shared mem than "statm". However adding up all the anon sections in smaps only gives 2456 KB?

When doing this I also noticed that there are duplicate entries in smaps. Any ideas why?

grep -F - /proc/$FIRST_HTTPD/smaps | sort | uniq -d -c

      2 b7f7d000-b7f7e000 r-xp 00000000 03:05 246646 
/usr/lib/httpd/modules/mod_auth_anon.so
      2 b7f7e000-b7f7f000 rwxp 00000000 03:05 246646 
/usr/lib/httpd/modules/mod_auth_anon.so
      2 b7f7f000-b7f81000 r-xp 00000000 03:05 246645 
/usr/lib/httpd/modules/mod_auth.so
      2 b7f81000-b7f82000 rwxp 00001000 03:05 246645 
/usr/lib/httpd/modules/mod_auth.so
      2 b7f82000-b7f84000 r-xp 00000000 03:05 246641 
/usr/lib/httpd/modules/mod_access.so
      2 b7f84000-b7f85000 rwxp 00001000 03:05 246641 
/usr/lib/httpd/modules/mod_access.so
      2 b7f85000-b7f9a000 r-xp 00000000 03:05 361234     /lib/ld-2.3.3.so
      2 b7f9a000-b7f9b000 r-xp 00014000 03:05 361234     /lib/ld-2.3.3.so
      2 b7f9b000-b7f9c000 rwxp 00015000 03:05 361234     /lib/ld-2.3.3.so
      2 bfb85000-bfb9a000 rw-p bfb85000 00:00 0          [stack]
      2 ffffe000-fffff000 ---p 00000000 00:00 0          [vdso]

If you factor that in, it means that smaps will report 156 KB too much shared mem in this example.

Quoted 7 lines
Quoted 3 lines

In the real world you can see this with a newly started apache. On my system statm reported that apache was using 35MB, whereas smaps reported the correct amount of 11MB.

How dou you know that 11MB is the correct shared value and the 35MB is the wrong value?

Well I'm quite sure that COW pages ar not accounted for in the statm shared value, which can be easily seen with my previously posted test program. Also putting the machine into swap, and then using httpd again causes the value reported to be very close to 11MB (i.e. the unused pages are not swapped back in).

Signature · 2 lines
Pádraig Brady - http://www.pixelbeat.org
--
#8 lore raw

Hi Brady,

Quoted 42 lines · On 7/20/05, [email protected] <[email protected]> wrote:
Quoted 12 lines · Mauricio Lin

Hi,

Quoted 8 lines · On 7/12/05, [email protected] <[email protected]> wrote:
Quoted 1 line · Andrew Morton

OK, please let us know how it goes.

It went very well. I could find no problems at all.
I've updated my script to use the new method, so please merge smaps :)
http://www.pixelbeat.org/scripts/ps_mem.py

Quoted 8 lines
Quoted 3 lines

Usually the shared mem reported by /proc/$$/statm
is the same as summing all the shared values in in /proc/$$/smaps but there can be large discrepancies.

Have you checked how the statm shared is calculated? I guess it does something like:
shared = mm->rss - mm->anon_rss

yes

Quoted 10 lines

But in smaps output you can have anonymous area like:

b6e0e000-b6e13000 rw-p

Size:                20 KB
Rss:                  4 KB
Shared_Clean:         0 KB
Shared_Dirty:         4 KB
Private_Clean:        0 KB
Private_Dirty:        0 KB
Quoted 3 lines

Look that it presents 4 KB of shared value in area considered anonymous.

ANDREW: anon_rss is the rss for anonymous area, right?

I see your point and I'm not sure.
The following shell gets the shared values for the first httpd process:

Quoted 8 lines

FIRST_HTTPD=ps -C httpd -o pid= | head -1 | tr -d ' ' HTTPD_STATM_SHARED=$(expr 4 '*' cut -f3 -d' ' /proc/$FIRST_HTTPD/statm) HTTPD_SMAPS_SHARED=$(grep Shared /proc/$FIRST_HTTPD/smaps | tr -s ' ' | cut -f2 -d' ' | ( tr '\n' +; echo 0 ) | bc)

This shows that "smaps" reports 3060 KB more shared mem than "statm". However adding up all the anon sections in smaps only gives 2456 KB?

You are adding up all Shared_Clean and Shared_Dirty as Shared, right?

Quoted 3 lines

When doing this I also noticed that there are duplicate entries in smaps. Any ideas why?

Each pair of address per line indicates the start and end address of a memory area (VMA) such as:

b7f7d000-b7f7e000

This means that an specific memory area start on virtual address b7f7d000 and end on b7f7e000 .

An mapped file like /lib/ld-2.3.3.so is organized in different memory areas. The memory area can be a text section, data section or bss. So it is normal you find the same filename mapped in more than one memory area.

You can find more information about VMA on Linux Kernel Development book (chapter 14) written by Robert Love.

For instance:

Quoted 4 lines

grep -F - /proc/$FIRST_HTTPD/smaps | sort | uniq -d -c

       2 b7f7d000-b7f7e000 r-xp 00000000 03:05 246646

/usr/lib/httpd/modules/mod_auth_anon.so

This is a text section.

Quoted 2 lines
       2 b7f7e000-b7f7f000 rwxp 00000000 03:05 246646

/usr/lib/httpd/modules/mod_auth_anon.so

This should be a data section.

IMHO, bss section corresponds to the anonymous area where the mapping is not backed by a file.

BR,

Mauricio Lin.

#9 lore raw
Quoted 13 lines · Mauricio Lin

Hi Brady,

Quoted 9 lines · On 7/20/05, [email protected] <[email protected]> wrote:

The following shell gets the shared values for the first httpd process:

FIRST_HTTPD=ps -C httpd -o pid= | head -1 | tr -d ' ' HTTPD_STATM_SHARED=$(expr 4 '*' cut -f3 -d' ' /proc/$FIRST_HTTPD/statm) HTTPD_SMAPS_SHARED=$(grep Shared /proc/$FIRST_HTTPD/smaps | tr -s ' ' | cut -f2 -d' ' | ( tr '\n' +; echo 0 ) | bc)

Quoted 5 lines
Quoted 2 lines

This shows that "smaps" reports 3060 KB more shared mem than "statm". However adding up all the anon sections in smaps only gives 2456 KB?

You are adding up all Shared_Clean and Shared_Dirty as Shared, right?

yes. Look at the command I posted.

Quoted 7 lines
Quoted 2 lines

When doing this I also noticed that there are duplicate entries in smaps. Any ideas why?

Each pair of address per line indicates the start and end address of a memory area (VMA) such as:

Quoted 8 lines

b7f7d000-b7f7e000

This means that an specific memory area start on virtual address b7f7d000 and end on b7f7e000 .

An mapped file like /lib/ld-2.3.3.so is organized in different memory areas. The memory area can be a text section, data section or bss. So it is normal you find the same filename mapped in more than one memory area.

yes. Look at the command I posted.
There are multiple entries with the _same addresses_

Quoted 1 line
Quoted 1 line

grep -F - /proc/$FIRST_HTTPD/smaps | sort | uniq -d -c

Signature · 2 lines
Pádraig Brady - http://www.pixelbeat.org
--