Accounting for "missing RAM"

I wrote a script to parse /proc/zoneinfo to figure out how memory on my server was getting used.

On many machines this is able to account for RAM +/- 10MB (which I consider within the margin of error).

But on many machines, there is 2-3GB of RAM that's unaccounted for and mysteriously missing from zoneinfo as well as /proc/meminfo.

When I parse /proc/kpageflags, pages with flags==0 correspond to nr_free_pages and pages with flags=KPF_BUDDY correspond to the number of "missing ram pages".

As far as I understand, KPF_BUDDY is set only on the first page of a higher order page. So if for some reason, we were counting an order=3 page as 7 pages instead of 8, it'd explain what I am seeing.

-Arun

#!/usr/bin/env python

import sys, re, string, os

class NestedDict(dict):
"""Implementation of perl's autovivification feature.""" def __getitem__(self, item):
try:
return dict.__getitem__(self, item)
except KeyError:
value = self[item] = type(self)() return value

sep = re.compile('^Node (.*)')
pages = re.compile('.*nr.*pages.*')
slab = re.compile('.*nr.*slab.*')
present = re.compile('.*present.*')
pcpu_pageset_count = re.compile('.*count:.*')
zone = None
page_size = os.sysconf('SC_PAGESIZE')

zones = NestedDict()
for line in open('/proc/zoneinfo').readlines():
m = sep.match(line)
if m:
zone = m.group(1) zones[zone]['pcpu_pageset'] = 0
# Old kernels don't have this in /proc/zoneinfo zones[zone]['nr_free_pages'] = 0
m1 = pages.match(line)
m2 = slab.match(line)
m3 = present.match(line)
if m1 or m2 or m3:
name, val = string.split(line)
val = int(val)
zones[zone][name] = val
continue
m4 = pcpu_pageset_count.match(line)
if m4:
name, val = string.split(line)
val = int(val) zones[zone]['pcpu_pageset'] += val

global_diff = 0L
for z in zones.keys():

    allocated = zones[z]['present'] - zones[z]['nr_free_pages']
    if allocated < 0: continue
    print '##############'
    print z
    accounted = 0
    for k in ('nr_file_pages', 'nr_anon_pages',
              'nr_slab_reclaimable', 'nr_slab_unreclaimable',
              'nr_page_table_pages', 'pcpu_pageset'):
        val = zones[z][k]
        print k, val
        accounted += val
    print "allocated", "accounted", "diff"
    print allocated, accounted, allocated - accounted
    global_diff += (allocated - accounted)

vmalloc = 0L
vmalloc_re = re.compile('.*VmallocUsed:\s+(\d+).*') for line in open('/proc/meminfo').readlines():
m = vmalloc_re.match(line)
if m:
vmalloc_kb = int(m.group(1))
vmalloc = vmalloc_kb * 1024/page_size

print '##############'
print 'vmalloc', vmalloc
print "missing ram: ", (global_diff - vmalloc) * page_size

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>