perf script: add script to profile and resolve physical mem type
PATCH 1/1 perf script: add script to profile and resolve physical mem type
From: Kan Liang <[email protected]>
There could be different types of memory in the system. E.g normal System Memory, Persistent Memory. To understand how the workload maps to those memories, it's important to know the I/O statistics on different type of memorys. Perf can collect address maps with physical addresses, but those are raw data. It still needs extra work to resolve the physical addresses.
Providing a script to facilitate the physical addresses resolving and I/O statistics.
Profiling with mem-loads and mem-stores if they are available. Looking up the physical address samples in /proc/iomem Providing memory type summary
Here is an example
#perf script record mem-phys-addr -- pmem_test_kernel
[ perf record: Woken up 32 times to write data ]
[ perf record: Captured and wrote 7.797 MB perf.data (101995 samples) ] #perf script report mem-phys-addr
Memory type summary
Event: mem-loads
Memory type count percentage
---------------------------------------- ----------- -----------
Persistent Memory 43740 60.6% System RAM 27179 37.7%
N/A 1268 1.8%
Event: mem-stores
Memory type count percentage
---------------------------------------- ----------- -----------
System RAM 24508 82.2%
N/A 5140 17.2%
Persistent Memory 160 0.5%
--- tools/perf/scripts/python/bin/mem-phys-addr-record | 30 ++++++ tools/perf/scripts/python/bin/mem-phys-addr-report | 3 + tools/perf/scripts/python/mem-phys-addr.py | 109 +++++++++++++++++++++ .../util/scripting-engines/trace-event-python.c | 2 + 4 files changed, 144 insertions(+) create mode 100644 tools/perf/scripts/python/bin/mem-phys-addr-record create mode 100644 tools/perf/scripts/python/bin/mem-phys-addr-report create mode 100644 tools/perf/scripts/python/mem-phys-addr.py
tools/perf/scripts/python/bin/mem-phys-addr-record new file +30 -0
@@ -0,0 +1,30 @@ +#!/bin/bash + +# +# Profiling physical memory accesses +# + +load=`perf list pmu | grep mem-loads` +store=`perf list pmu | grep mem-stores` +if [ -z "$load" ] && [ -z "$store" ] ; then + echo "There is no mem-loads or mem-stores support" + exit 1 +fi + +arg="-e" +if [ ! -z "$store" ] ; then + arg="$arg mem-stores:P" +fi + +if [ ! -z "$load" ] ; then + if [ ! -z "$store" ] ; then + arg="$arg,mem-loads:P" + else + arg="$arg mem-loads:P" + fi + arg="$arg -W" +fi + +arg="$arg -d --phys-data" + +perf record $arg $@
tools/perf/scripts/python/bin/mem-phys-addr-report new file +3 -0
@@ -0,0 +1,3 @@ +#!/bin/bash +# description: resolve physical address samples +perf script $@ -s "$PERF_EXEC_PATH"/scripts/python/mem-phys-addr.py
tools/perf/scripts/python/mem-phys-addr.py new file +109 -0
109 lines not on this page · raw message
tools/perf/util/scripting-engines/trace-event-python.c +2 -0
@@ -500,6 +500,8 @@ static PyObject *get_perf_sample_dict(struct perf_sample *sample, PyLong_FromUnsignedLongLong(sample->time));pydict_set_item_string_decref (dict_sample,"period" ,PyLong_FromUnsignedLongLong (sample->period)); +pydict_set_item_string_decref (dict_sample,"phys_addr" , +PyLong_FromUnsignedLongLong (sample->phys_addr));set_sample_read_in_dict (dict_sample, sample, evsel);pydict_set_item_string_decref (dict,"sample" , dict_sample);
Signature · 1 line
2.7.4
Quoted 116 lines · On Mon, Oct 16, 2017 at 3:26 PM, <[email protected]> wrote:
From: Kan Liang <[email protected]>
There could be different types of memory in the system. E.g normal System Memory, Persistent Memory. To understand how the workload maps to those memories, it's important to know the I/O statistics on different type of memorys. Perf can collect address maps with physical addresses, but those are raw data. It still needs extra work to resolve the physical addresses.
Providing a script to facilitate the physical addresses resolving and I/O statistics.Profiling with mem-loads and mem-stores if they are available. Looking up the physical address samples in
/proc/iomemProviding memory type summaryHere is an example
#perf script record mem-phys-addr --pmem_test_kernel
[ perf record: Woken up 32 times to write data ]
[ perf record: Captured and wrote 7.797 MB perf.data (101995 samples) ] #perf script report mem-phys-addr
Memory type summaryEvent: mem-loads
Memory type count percentage
---------------------------------------- ----------- -----------Persistent Memory 43740 60.6% System RAM 27179 37.7%N/A 1268 1.8%
Event: mem-stores
Memory type count percentage
---------------------------------------- ----------- -----------
System RAM 24508 82.2%
N/A 5140 17.2%
Persistent Memory 160 0.5%Signed-off-by Kan Liang <[email protected]>--- tools/perf/scripts/python/bin/mem-phys-addr-record | 30 ++++++ tools/perf/scripts/python/bin/mem-phys-addr-report | 3 + tools/perf/scripts/python/mem-phys-addr.py | 109 +++++++++++++++++++++ .../util/scripting-engines/trace-event-python.c | 2 + 4 files changed, 144 insertions(+) create mode 100644 tools/perf/scripts/python/bin/mem-phys-addr-record create mode 100644 tools/perf/scripts/python/bin/mem-phys-addr-report create mode 100644 tools/perf/scripts/python/mem-phys-addr.pytools/perf/scripts/python/bin/mem-phys-addr-record new file +30
-0@@ -0,0 +1,30 @@ +#!/bin/bash + +# +# Profiling physical memory accesses +# + +load=`perf list pmu | grep mem-loads` +store=`perf list pmu | grep mem-stores` +if [ -z "$load" ] && [ -z "$store" ] ; then + echo "There is no mem-loads or mem-stores support" + exit 1 +fi + +arg="-e" +if [ ! -z "$store" ] ; then + arg="$arg mem-stores:P" +fi + +if [ ! -z "$load" ] ; then + if [ ! -z "$store" ] ; then + arg="$arg,mem-loads:P" + else + arg="$arg mem-loads:P" + fi + arg="$arg -W" +fi + +arg="$arg -d --phys-data" + +perf record $arg $@tools/perf/scripts/python/bin/mem-phys-addr-report new file +3
-0@@ -0,0 +1,3 @@ +#!/bin/bash +# description: resolve physical address samples +perf script $@ -s "$PERF_EXEC_PATH"/scripts/python/mem-phys-addr.pytools/perf/scripts/python/mem-phys-addr.py new file +18
-0@@ -0,0 +1,109 @@ +# mem-phys-addr.py: Resolve physical address samples +# Copyright (c) 2017, Intel Corporation. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms and conditions of the GNU General Public License, +# version 2, as published by the Free Software Foundation. +# +# This program is distributed in the hope it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +# more details. + +from __future__ import division +import os +import sys +import struct +import re +import bisect
tools/perf/scripts/python/mem-phys-addr.py line 28
Quoted 10 lines
tools/perf/scripts/python/mem-phys-addr.py new file +10
-0+import collections + +sys.path.append(os.environ['PERF_EXEC_PATH'] + \ + '/scripts/python/Perf-Trace-Util/lib/Perf/Trace') + +system_ram = [] +pmem = [] +f = None +load_event = ('mem-loads', '0x1cd') +store_event = ('mem-stores', '0x82d0');
I don't like the fact that you are mixing DLA and Load Latency. The two mechanism do not operate the same way and any attempt at comparing the loads and stores lead to misinterpretations.
Quoted 97 lines
tools/perf/scripts/python/mem-phys-addr.py new file +81
-0+load_mem_type_cnt = collections.Counter() +store_mem_type_cnt = collections.Counter() + +def parse_iomem(): + global f + f = open('/proc/iomem', 'r') + for i, j in enumerate(f): + m = re.split('-|:',j,2) + if m[2].strip() == 'System RAM': + system_ram.append(long(m[0], 16)) + system_ram.append(long(m[1], 16)) + if m[2].strip() == 'Persistent Memory': + pmem.append(long(m[0], 16)) + pmem.append(long(m[1], 16)) + +def print_memory_type(): + print "Memory type summary\n" + print "Event: mem-loads" + print "%-40s %10s %10s\n" % ("Memory type", "count", "percentage"), + print "%-40s %10s %10s\n" % ("----------------------------------------", \ + "-----------", "-----------"), + total = sum(load_mem_type_cnt.values()) + for mem_type, count in sorted(load_mem_type_cnt.most_common(), \ + key = lambda(k, v): (v, k), reverse = True): + print "%-40s %10d %10.1f%%\n" % (mem_type, count, 100 * count / total), + print "\n\n" + print "Event: mem-stores" + print "%-40s %10s %10s\n" % ("Memory type", "count", "percentage"), + print "%-40s %10s %10s\n" % ("----------------------------------------", \ + "-----------", "-----------"), + total = sum(store_mem_type_cnt.values()) + for mem_type, count in sorted(store_mem_type_cnt.most_common(), \ + key = lambda(k, v): (v, k), reverse = True): + print "%-40s %10d %10.1f%%\n" % (mem_type, count, 100 * count / total), + +def trace_begin(): + parse_iomem() + +def trace_end(): + print_memory_type() + f.close() + +def is_system_ram(phys_addr): + #/proc/iomem is sorted + position = bisect.bisect(system_ram, phys_addr) + if position % 2 == 0: + return False + return True + +def is_persistent_mem(phys_addr): + position = bisect.bisect(pmem, phys_addr) + if position % 2 == 0: + return False + return True + +def find_memory_type(phys_addr): + if phys_addr == 0: + return "N/A" + if is_system_ram(phys_addr): + return "System RAM" + + if is_persistent_mem(phys_addr): + return "Persistent Memory" + + #slow path, search all + f.seek(0, 0) + for j in f: + m = re.split('-|:',j,2) + if long(m[0], 16) <= phys_addr <= long(m[1], 16): + return m[2] + return "N/A" + +def process_event(param_dict): + name = param_dict["ev_name"] + sample = param_dict["sample"] + phys_addr = sample["phys_addr"] + + if any(x in name for x in load_event): + load_mem_type_cnt[find_memory_type(phys_addr)] += 1 + if any(x in name for x in store_event): + store_mem_type_cnt[find_memory_type(phys_addr)] += 1tools/perf/util/scripting-engines/trace-event-python.c +2
-1@@ -500,6 +500,8 @@ static PyObject *get_perf_sample_dict(struct perf_sample *sample, PyLong_FromUnsignedLongLong(sample->time));pydict_set_item_string_decref (dict_sample,"period" ,PyLong_FromUnsignedLongLong (sample->period)); +pydict_set_item_string_decref (dict_sample,"phys_addr" , +PyLong_FromUnsignedLongLong (sample->phys_addr));set_sample_read_in_dict (dict_sample, sample, evsel);pydict_set_item_string_decref (dict,"sample" , dict_sample); --2.7.4
Quoted 131 lines · On Mon, Oct 16, 2017 at 3:26 PM, <[email protected]> wrote:
Quoted 4 lines
From: Kan Liang <[email protected]>
There could be different types of memory in the system. E.g normal System Memory, Persistent Memory. To understand how the workload
maps to
Quoted 32 lines
those memories, it's important to know the I/O statistics on different type of memorys. Perf can collect address maps with physical addresses, but those are raw data. It still needs extra work to resolve the physical addresses.
Providing a script to facilitate the physical addresses resolving and I/O statistics.Profiling with mem-loads and mem-stores if they are available. Looking up the physical address samples in
/proc/iomemProviding memory type summaryHere is an example
#perf script record mem-phys-addr --pmem_test_kernel
[ perf record: Woken up 32 times to write data ]
[ perf record: Captured and wrote 7.797 MB perf.data (101995 samples) ] #perf script report mem-phys-addr
Memory type summaryEvent: mem-loads
Memory type count percentage
---------------------------------------- ----------- -----------Persistent Memory 43740 60.6% System RAM 27179 37.7%N/A 1268 1.8%
Event: mem-stores
Memory type count percentage
---------------------------------------- ----------- -----------
System RAM 24508 82.2%
N/A 5140 17.2%
Persistent Memory 160 0.5%Quoted 5 lines
Signed-off-by Kan Liang <[email protected]>--- tools/perf/scripts/python/bin/mem-phys-addr-record | 30 ++++++ tools/perf/scripts/python/bin/mem-phys-addr-report | 3 + tools/perf/scripts/python/mem-phys-addr.py | 109+++++++++++++++++++++
Quoted 3 lines
.../util/scripting-engines/trace-event-python.c | 2 + 4 files changed, 144 insertions(+) create mode 100644 tools/perf/scripts/python/bin/mem-phys-addr-record
Review
Quoted 4 lines
create mode 100644 tools/perf/scripts/python/bin/mem-phys-addr-report create mode 100644 tools/perf/scripts/python/mem-phys-addr.py(quoted diff) +0
-0b/tools/perf/scripts/python/bin/mem-phys-addr-record
Review
Quoted 36 lines
new file mode 100644
index 0000000..395b256tools/perf/scripts/python/bin/mem-phys-addr-record new file +30
-0@@ -0,0 +1,30 @@ +#!/bin/bash + +# +# Profiling physical memory accesses +# + +load=`perf list pmu | grep mem-loads` +store=`perf list pmu | grep mem-stores` +if [ -z "$load" ] && [ -z "$store" ] ; then + echo "There is no mem-loads or mem-stores support" + exit 1 +fi + +arg="-e" +if [ ! -z "$store" ] ; then + arg="$arg mem-stores:P" +fi + +if [ ! -z "$load" ] ; then + if [ ! -z "$store" ] ; then + arg="$arg,mem-loads:P" + else + arg="$arg mem-loads:P" + fi + arg="$arg -W" +fi + +arg="$arg -d --phys-data" + +perf record $arg $@(quoted diff) +0
-0b/tools/perf/scripts/python/bin/mem-phys-addr-report
Review
Quoted 9 lines
new file mode 100644
index 0000000..3f2b847tools/perf/scripts/python/bin/mem-phys-addr-report new file +3
-0@@ -0,0 +1,3 @@ +#!/bin/bash +# description: resolve physical address samples +perf script $@ -s "$PERF_EXEC_PATH"/scripts/python/mem-phys-addr.py(quoted diff) +0
-0
b/tools/perf/scripts/python/mem-phys-addr.pyQuoted 5 lines
new file mode 100644
index 0000000..73b3a63tools/perf/scripts/python/mem-phys-addr.py line 9
Quoted 9 lines
tools/perf/scripts/python/mem-phys-addr.py new file +9
-0@@ -0,0 +1,109 @@ +# mem-phys-addr.py: Resolve physical address samples +# Copyright (c) 2017, Intel Corporation. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms and conditions of the GNU General Public License, +# version 2, as published by the Free Software Foundation. +# +# This program is distributed in the hope it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty ofMERCHANTABILITY or
tools/perf/scripts/python/mem-phys-addr.py line 10
Quoted 1 line
tools/perf/scripts/python/mem-phys-addr.py new file +1
-0+# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General PublicLicense for
Quoted 14 lines
tools/perf/scripts/python/mem-phys-addr.py new file +14
-0+# more details. + +from __future__ import division +import os +import sys +import struct +import re +import bisect +import collections + +sys.path.append(os.environ['PERF_EXEC_PATH'] + \ + '/scripts/python/Perf-Trace-Util/lib/Perf/Trace') + +system_ram = []
Quoted 9 lines
tools/perf/scripts/python/mem-phys-addr.py line 28
Quoted 4 lines
tools/perf/scripts/python/mem-phys-addr.py new file +4
-0+pmem = [] +f = None +load_event = ('mem-loads', '0x1cd') +store_event = ('mem-stores', '0x82d0');I don't like the fact that you are mixing DLA and Load Latency. The two mechanism do not operate the same way and any attempt at comparing the loads and stores lead to misinterpretations.
OK. I will only keep "-d --phys-data" options as default. For the event, is only mem-loads good enough as default event? Or should we just let the user fill up the event?
Thanks,
Kan
Quoted 105 lines
Quoted 9 lines
tools/perf/scripts/python/mem-phys-addr.py new file +9
-0+load_mem_type_cnt = collections.Counter() +store_mem_type_cnt = collections.Counter() + +def parse_iomem(): + global f + f = open('/proc/iomem', 'r') + for i, j in enumerate(f): + m = re.split('-|:',j,2) + if m[2].strip() == 'System RAM':tools/perf/scripts/python/mem-phys-addr.py
Quoted 10 lines
tools/perf/scripts/python/mem-phys-addr.py new file +10
-0+ system_ram.append(long(m[0], 16)) + system_ram.append(long(m[1], 16)) + if m[2].strip() == 'Persistent Memory': + pmem.append(long(m[0], 16)) + pmem.append(long(m[1], 16)) + +def print_memory_type(): + print "Memory type summary\n" + print "Event: mem-loads" + print "%-40s %10s %10s\n" % ("Memory type", "count","percentage"),
tools/perf/scripts/python/mem-phys-addr.py
Quoted 4 lines
tools/perf/scripts/python/mem-phys-addr.py new file +4
-0+ print "%-40s %10s %10s\n" % ("----------------------------------------", \ + "-----------", "-----------"), + total = sum(load_mem_type_cnt.values()) + for mem_type, count in sorted(load_mem_type_cnt.most_common(),\
tools/perf/scripts/python/mem-phys-addr.py line 53
Quoted 2 lines
tools/perf/scripts/python/mem-phys-addr.py new file +2
-0+ key = lambda(k, v): (v, k), reverse = True): + print "%-40s %10d %10.1f%%\n" % (mem_type, count, 100 *count / total),
tools/perf/scripts/python/mem-phys-addr.py
Quoted 3 lines
tools/perf/scripts/python/mem-phys-addr.py new file +3
-0+ print "\n\n" + print "Event: mem-stores" + print "%-40s %10s %10s\n" % ("Memory type", "count","percentage"),
tools/perf/scripts/python/mem-phys-addr.py
Quoted 4 lines
tools/perf/scripts/python/mem-phys-addr.py new file +4
-0+ print "%-40s %10s %10s\n" % ("----------------------------------------", \ + "-----------", "-----------"), + total = sum(store_mem_type_cnt.values()) + for mem_type, count in sorted(store_mem_type_cnt.most_common(),\
tools/perf/scripts/python/mem-phys-addr.py line 62
Quoted 2 lines
tools/perf/scripts/python/mem-phys-addr.py new file +2
-0+ key = lambda(k, v): (v, k), reverse = True): + print "%-40s %10d %10.1f%%\n" % (mem_type, count, 100 *count / total),
Review
Quoted 48 lines
tools/perf/scripts/python/mem-phys-addr.py new file +47
-0+ +def trace_begin(): + parse_iomem() + +def trace_end(): + print_memory_type() + f.close() + +def is_system_ram(phys_addr): + #/proc/iomem is sorted + position = bisect.bisect(system_ram, phys_addr) + if position % 2 == 0: + return False + return True + +def is_persistent_mem(phys_addr): + position = bisect.bisect(pmem, phys_addr) + if position % 2 == 0: + return False + return True + +def find_memory_type(phys_addr): + if phys_addr == 0: + return "N/A" + if is_system_ram(phys_addr): + return "System RAM" + + if is_persistent_mem(phys_addr): + return "Persistent Memory" + + #slow path, search all + f.seek(0, 0) + for j in f: + m = re.split('-|:',j,2) + if long(m[0], 16) <= phys_addr <= long(m[1], 16): + return m[2] + return "N/A" + +def process_event(param_dict): + name = param_dict["ev_name"] + sample = param_dict["sample"] + phys_addr = sample["phys_addr"] + + if any(x in name for x in load_event): + load_mem_type_cnt[find_memory_type(phys_addr)] += 1 + if any(x in name for x in store_event): + store_mem_type_cnt[find_memory_type(phys_addr)] += 1(quoted diff) +0
-0
b/tools/perf/util/scripting-engines/trace-event-python.ctools/perf/util/scripting-engines/trace-event-python.c static PyObject *get_perf_sample_dict(struct
Quoted 4 lines
index c7187f0..8cd6317 100644
tools/perf/util/scripting-engines/trace-event-python.c +0
-0@@ -500,6 +500,8 @@ static PyObject *get_perf_sample_dict(struct
perf_sample*sample,Quoted 11 lines
tools/perf/util/scripting-engines/trace-event-python.c +2
-1PyLong_FromUnsignedLongLong(sample->time));pydict_set_item_string_decref (dict_sample,"period" ,PyLong_FromUnsignedLongLong (sample->period)); +pydict_set_item_string_decref (dict_sample,"phys_addr" , +PyLong_FromUnsignedLongLong (sample->phys_addr));set_sample_read_in_dict (dict_sample, sample, evsel);pydict_set_item_string_decref (dict,"sample" , dict_sample); --2.7.4
Quoted 141 lines · Kan
Quoted 131 lines · On Mon, Oct 16, 2017 at 3:26 PM, <[email protected]> wrote:
Quoted 4 lines
From: Kan Liang <[email protected]>
There could be different types of memory in the system. E.g normal System Memory, Persistent Memory. To understand how the workload
maps to
Quoted 32 lines
those memories, it's important to know the I/O statistics on different type of memorys. Perf can collect address maps with physical addresses, but those are raw data. It still needs extra work to resolve the physical addresses.
Providing a script to facilitate the physical addresses resolving and I/O statistics.Profiling with mem-loads and mem-stores if they are available. Looking up the physical address samples in
/proc/iomemProviding memory type summaryHere is an example
#perf script record mem-phys-addr --pmem_test_kernel
[ perf record: Woken up 32 times to write data ]
[ perf record: Captured and wrote 7.797 MB perf.data (101995 samples) ] #perf script report mem-phys-addr
Memory type summaryEvent: mem-loads
Memory type count percentage
---------------------------------------- ----------- -----------Persistent Memory 43740 60.6% System RAM 27179 37.7%N/A 1268 1.8%
Event: mem-stores
Memory type count percentage
---------------------------------------- ----------- -----------
System RAM 24508 82.2%
N/A 5140 17.2%
Persistent Memory 160 0.5%Quoted 5 lines
Signed-off-by Kan Liang <[email protected]>--- tools/perf/scripts/python/bin/mem-phys-addr-record | 30 ++++++ tools/perf/scripts/python/bin/mem-phys-addr-report | 3 + tools/perf/scripts/python/mem-phys-addr.py | 109+++++++++++++++++++++
Quoted 3 lines
.../util/scripting-engines/trace-event-python.c | 2 + 4 files changed, 144 insertions(+) create mode 100644 tools/perf/scripts/python/bin/mem-phys-addr-record
Review
Quoted 4 lines
create mode 100644 tools/perf/scripts/python/bin/mem-phys-addr-report create mode 100644 tools/perf/scripts/python/mem-phys-addr.py(quoted diff) +0
-0b/tools/perf/scripts/python/bin/mem-phys-addr-record
Review
Quoted 36 lines
new file mode 100644
index 0000000..395b256tools/perf/scripts/python/bin/mem-phys-addr-record new file +30
-0@@ -0,0 +1,30 @@ +#!/bin/bash + +# +# Profiling physical memory accesses +# + +load=`perf list pmu | grep mem-loads` +store=`perf list pmu | grep mem-stores` +if [ -z "$load" ] && [ -z "$store" ] ; then + echo "There is no mem-loads or mem-stores support" + exit 1 +fi + +arg="-e" +if [ ! -z "$store" ] ; then + arg="$arg mem-stores:P" +fi + +if [ ! -z "$load" ] ; then + if [ ! -z "$store" ] ; then + arg="$arg,mem-loads:P" + else + arg="$arg mem-loads:P" + fi + arg="$arg -W" +fi + +arg="$arg -d --phys-data" + +perf record $arg $@(quoted diff) +0
-0b/tools/perf/scripts/python/bin/mem-phys-addr-report
Review
Quoted 9 lines
new file mode 100644
index 0000000..3f2b847tools/perf/scripts/python/bin/mem-phys-addr-report new file +3
-0@@ -0,0 +1,3 @@ +#!/bin/bash +# description: resolve physical address samples +perf script $@ -s "$PERF_EXEC_PATH"/scripts/python/mem-phys-addr.py(quoted diff) +0
-0
b/tools/perf/scripts/python/mem-phys-addr.pyQuoted 5 lines
new file mode 100644
index 0000000..73b3a63tools/perf/scripts/python/mem-phys-addr.py line 9
Quoted 9 lines
tools/perf/scripts/python/mem-phys-addr.py new file +9
-0@@ -0,0 +1,109 @@ +# mem-phys-addr.py: Resolve physical address samples +# Copyright (c) 2017, Intel Corporation. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms and conditions of the GNU General Public License, +# version 2, as published by the Free Software Foundation. +# +# This program is distributed in the hope it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty ofMERCHANTABILITY or
tools/perf/scripts/python/mem-phys-addr.py line 10
Quoted 1 line
tools/perf/scripts/python/mem-phys-addr.py new file +1
-0+# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General PublicLicense for
Quoted 14 lines
tools/perf/scripts/python/mem-phys-addr.py new file +14
-0+# more details. + +from __future__ import division +import os +import sys +import struct +import re +import bisect +import collections + +sys.path.append(os.environ['PERF_EXEC_PATH'] + \ + '/scripts/python/Perf-Trace-Util/lib/Perf/Trace') + +system_ram = []Quoted 9 lines
tools/perf/scripts/python/mem-phys-addr.py line 28
Quoted 4 lines
tools/perf/scripts/python/mem-phys-addr.py new file +4
-0+pmem = [] +f = None +load_event = ('mem-loads', '0x1cd') +store_event = ('mem-stores', '0x82d0');I don't like the fact that you are mixing DLA and Load Latency. The two mechanism do not operate the same way and any attempt at comparing the loads and stores lead to misinterpretations.
Quoted 4 lines
OK. I will only keep "-d --phys-data" options as default. For the event, is only mem-loads good enough as default event? Or should we just let the user fill up the event?
Why not use 0x81d0 to be symmetrical with the stores?
Quoted 108 lines
Thanks,
KanQuoted 105 lines
Quoted 9 lines
tools/perf/scripts/python/mem-phys-addr.py new file +9
-0+load_mem_type_cnt = collections.Counter() +store_mem_type_cnt = collections.Counter() + +def parse_iomem(): + global f + f = open('/proc/iomem', 'r') + for i, j in enumerate(f): + m = re.split('-|:',j,2) + if m[2].strip() == 'System RAM':tools/perf/scripts/python/mem-phys-addr.py
Quoted 10 lines
tools/perf/scripts/python/mem-phys-addr.py new file +10
-0+ system_ram.append(long(m[0], 16)) + system_ram.append(long(m[1], 16)) + if m[2].strip() == 'Persistent Memory': + pmem.append(long(m[0], 16)) + pmem.append(long(m[1], 16)) + +def print_memory_type(): + print "Memory type summary\n" + print "Event: mem-loads" + print "%-40s %10s %10s\n" % ("Memory type", "count","percentage"),
tools/perf/scripts/python/mem-phys-addr.py
Quoted 4 lines
tools/perf/scripts/python/mem-phys-addr.py new file +4
-0+ print "%-40s %10s %10s\n" % ("----------------------------------------", \ + "-----------", "-----------"), + total = sum(load_mem_type_cnt.values()) + for mem_type, count in sorted(load_mem_type_cnt.most_common(),\
tools/perf/scripts/python/mem-phys-addr.py line 53
Quoted 2 lines
tools/perf/scripts/python/mem-phys-addr.py new file +2
-0+ key = lambda(k, v): (v, k), reverse = True): + print "%-40s %10d %10.1f%%\n" % (mem_type, count, 100 *count / total),
tools/perf/scripts/python/mem-phys-addr.py
Quoted 3 lines
tools/perf/scripts/python/mem-phys-addr.py new file +3
-0+ print "\n\n" + print "Event: mem-stores" + print "%-40s %10s %10s\n" % ("Memory type", "count","percentage"),
tools/perf/scripts/python/mem-phys-addr.py
Quoted 4 lines
tools/perf/scripts/python/mem-phys-addr.py new file +4
-0+ print "%-40s %10s %10s\n" % ("----------------------------------------", \ + "-----------", "-----------"), + total = sum(store_mem_type_cnt.values()) + for mem_type, count in sorted(store_mem_type_cnt.most_common(),\
tools/perf/scripts/python/mem-phys-addr.py line 62
Quoted 2 lines
tools/perf/scripts/python/mem-phys-addr.py new file +2
-0+ key = lambda(k, v): (v, k), reverse = True): + print "%-40s %10d %10.1f%%\n" % (mem_type, count, 100 *count / total),
Review
Quoted 48 lines
tools/perf/scripts/python/mem-phys-addr.py new file +47
-0+ +def trace_begin(): + parse_iomem() + +def trace_end(): + print_memory_type() + f.close() + +def is_system_ram(phys_addr): + #/proc/iomem is sorted + position = bisect.bisect(system_ram, phys_addr) + if position % 2 == 0: + return False + return True + +def is_persistent_mem(phys_addr): + position = bisect.bisect(pmem, phys_addr) + if position % 2 == 0: + return False + return True + +def find_memory_type(phys_addr): + if phys_addr == 0: + return "N/A" + if is_system_ram(phys_addr): + return "System RAM" + + if is_persistent_mem(phys_addr): + return "Persistent Memory" + + #slow path, search all + f.seek(0, 0) + for j in f: + m = re.split('-|:',j,2) + if long(m[0], 16) <= phys_addr <= long(m[1], 16): + return m[2] + return "N/A" + +def process_event(param_dict): + name = param_dict["ev_name"] + sample = param_dict["sample"] + phys_addr = sample["phys_addr"] + + if any(x in name for x in load_event): + load_mem_type_cnt[find_memory_type(phys_addr)] += 1 + if any(x in name for x in store_event): + store_mem_type_cnt[find_memory_type(phys_addr)] += 1(quoted diff) +0
-0
b/tools/perf/util/scripting-engines/trace-event-python.ctools/perf/util/scripting-engines/trace-event-python.c static PyObject *get_perf_sample_dict(struct
Quoted 4 lines
index c7187f0..8cd6317 100644
tools/perf/util/scripting-engines/trace-event-python.c +0
-0@@ -500,6 +500,8 @@ static PyObject *get_perf_sample_dict(struct
perf_sample*sample,Quoted 11 lines
tools/perf/util/scripting-engines/trace-event-python.c +2
-1PyLong_FromUnsignedLongLong(sample->time));pydict_set_item_string_decref (dict_sample,"period" ,PyLong_FromUnsignedLongLong (sample->period)); +pydict_set_item_string_decref (dict_sample,"phys_addr" , +PyLong_FromUnsignedLongLong (sample->phys_addr));set_sample_read_in_dict (dict_sample, sample, evsel);pydict_set_item_string_decref (dict,"sample" , dict_sample); --2.7.4
Quoted 141 lines · Kan
Quoted 98 lines
Quoted 98 lines · On Mon, Oct 16, 2017 at 3:26 PM, <[email protected]> wrote:
Quoted 4 lines
From: Kan Liang <[email protected]>
There could be different types of memory in the system. E.g normal System Memory, Persistent Memory. To understand how the workload
maps to
Quoted 31 lines
those memories, it's important to know the I/O statistics on different type of memorys. Perf can collect address maps with physical addresses, but those are raw data. It still needs extra work to resolve the physical addresses.
Providing a script to facilitate the physical addresses resolving and I/O statistics.Profiling with mem-loads and mem-stores if they are available. Looking up the physical address samples in
/proc/iomemProviding memory type summaryHere is an example
#perf script record mem-phys-addr --pmem_test_kernel[ perf
record: Woken up 32 times to write data ] [ perf record: Captured and wrote 7.797 MB perf.data (101995 samples) ] #perf script report mem-phys-addr Memory type summaryEvent: mem-loads
Memory type count percentage
---------------------------------------- ----------- -----------Persistent Memory 43740 60.6% System RAM 27179 37.7%N/A 1268 1.8%
Event: mem-stores
Memory type count percentage
---------------------------------------- ----------- -----------
System RAM 24508 82.2%
N/A 5140 17.2%
Persistent Memory 160 0.5%Quoted 5 lines
Signed-off-by Kan Liang <[email protected]>--- tools/perf/scripts/python/bin/mem-phys-addr-record | 30 ++++++ tools/perf/scripts/python/bin/mem-phys-addr-report | 3 + tools/perf/scripts/python/mem-phys-addr.py | 109+++++++++++++++++++++
Quoted 3 lines
.../util/scripting-engines/trace-event-python.c | 2 + 4 files changed, 144 insertions(+) create mode 100644
tools/perf/scripts/python/bin/mem-phys-addr-record
Review
Quoted 5 lines
create mode 100644 tools/perf/scripts/python/bin/mem-phys-addr-report create mode 100644 tools/perf/scripts/python/mem-phys-addr.py(quoted diff) +0
-0b/tools/perf/scripts/python/bin/mem-phys-addr-record
Review
Quoted 34 lines
new file mode 100644
index 0000000..395b256tools/perf/scripts/python/bin/mem-phys-addr-record new file +28
-0@@ -0,0 +1,30 @@ +#!/bin/bash + +# +# Profiling physical memory accesses # + +load=`perf list pmu | grep mem-loads` store=`perf list pmu | grep +mem-stores` if [ -z "$load" ] && [ -z "$store" ] ; then + echo "There is no mem-loads or mem-stores support" + exit 1 +fi + +arg="-e" +if [ ! -z "$store" ] ; then + arg="$arg mem-stores:P" +fi + +if [ ! -z "$load" ] ; then + if [ ! -z "$store" ] ; then + arg="$arg,mem-loads:P" + else + arg="$arg mem-loads:P" + fi + arg="$arg -W" +fi + +arg="$arg -d --phys-data" + +perf record $arg $@(quoted diff) +0
-0b/tools/perf/scripts/python/bin/mem-phys-addr-report
Review
Quoted 9 lines
new file mode 100644
index 0000000..3f2b847tools/perf/scripts/python/bin/mem-phys-addr-report new file +3
-0@@ -0,0 +1,3 @@ +#!/bin/bash +# description: resolve physical address samples perf script $@ -s +"$PERF_EXEC_PATH"/scripts/python/mem-phys-addr.py(quoted diff) +0
-0
b/tools/perf/scripts/python/mem-phys-addr.pyQuoted 11 lines
Quoted 11 lines
tools/perf/scripts/python/mem-phys-addr.py line 6
Quoted 11 lines
new file mode 100644
index 0000000..73b3a63tools/perf/scripts/python/mem-phys-addr.py new file +6
-0@@ -0,0 +1,109 @@ +# mem-phys-addr.py: Resolve physical address samples # Copyright +(c) 2017, Intel Corporation. +# +# This program is free software; you can redistribute it and/or +modify it # under the terms and conditions of the GNU General +Public License, # version 2, as published by the Free SoftwareFoundation.
Quoted 30 lines
Quoted 29 lines
tools/perf/scripts/python/mem-phys-addr.py line 9
Quoted 3 lines
tools/perf/scripts/python/mem-phys-addr.py new file +3
-0+# +# This program is distributed in the hope it will be useful, but +WITHOUT # ANY WARRANTY; without even the implied warranty ofMERCHANTABILITY or
tools/perf/scripts/python/mem-phys-addr.py line 10
Quoted 1 line
tools/perf/scripts/python/mem-phys-addr.py new file +1
-0+# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General PublicLicense for
Quoted 8 lines
tools/perf/scripts/python/mem-phys-addr.py new file +8
-0+# more details. + +from __future__ import division +import os +import sys +import struct +import re +import bisecttools/perf/scripts/python/mem-phys-addr.py
Quoted 10 lines
tools/perf/scripts/python/mem-phys-addr.py new file +10
-0+import collections + +sys.path.append(os.environ['PERF_EXEC_PATH'] + \ + '/scripts/python/Perf-Trace-Util/lib/Perf/Trace') + +system_ram = [] +pmem = [] +f = None +load_event = ('mem-loads', '0x1cd') store_event = ('mem-stores', +'0x82d0');I don't like the fact that you are mixing DLA and Load Latency. The two mechanism do not operate the same way and any attempt at comparing the loads and stores lead to misinterpretations.
Quoted 6 lines
Quoted 4 lines
OK. I will only keep "-d --phys-data" options as default. For the event, is only mem-loads good enough as default event? Or should we just let the user fill up the event?
Why not use
0x81d0to be symmetrical with the stores?
Right, it doesn’t need load latency. 0x81d0 should be a better choice. I will use 0x81d0 and 0x82d0 as default event for V2.
Thanks,
Kan
Quoted 113 lines
Quoted 113 lines
Quoted 113 lines
Quoted 10 lines
tools/perf/scripts/python/mem-phys-addr.py new file +9
-0+load_mem_type_cnt = collections.Counter() store_mem_type_cnt = +collections.Counter() + +def parse_iomem(): + global f + f = open('/proc/iomem', 'r') + for i, j in enumerate(f): + m = re.split('-|:',j,2) + if m[2].strip() == 'System RAM':tools/perf/scripts/python/mem-phys-addr.py
Quoted 10 lines
tools/perf/scripts/python/mem-phys-addr.py new file +10
-0+ system_ram.append(long(m[0], 16)) + system_ram.append(long(m[1], 16)) + if m[2].strip() == 'Persistent Memory': + pmem.append(long(m[0], 16)) + pmem.append(long(m[1], 16)) + +def print_memory_type(): + print "Memory type summary\n" + print "Event: mem-loads" + print "%-40s %10s %10s\n" % ("Memory type", "count","percentage"),
tools/perf/scripts/python/mem-phys-addr.py
Quoted 5 lines
tools/perf/scripts/python/mem-phys-addr.py new file +5
-0+ print "%-40s %10s %10s\n" % ("----------------------------------------", \ + "-----------", "-----------"), + total = sum(load_mem_type_cnt.values()) + for mem_type, count in + sorted(load_mem_type_cnt.most_common(),\
tools/perf/scripts/python/mem-phys-addr.py line 55
Quoted 3 lines
tools/perf/scripts/python/mem-phys-addr.py new file +3
-0+ key = lambda(k, v): (v, k), reverse = True): + print "%-40s %10d %10.1f%%\n" % (mem_type, count, + 100 *count / total),
tools/perf/scripts/python/mem-phys-addr.py
Quoted 3 lines
tools/perf/scripts/python/mem-phys-addr.py new file +3
-0+ print "\n\n" + print "Event: mem-stores" + print "%-40s %10s %10s\n" % ("Memory type", "count","percentage"),
tools/perf/scripts/python/mem-phys-addr.py
Quoted 5 lines
tools/perf/scripts/python/mem-phys-addr.py new file +5
-0+ print "%-40s %10s %10s\n" % ("----------------------------------------", \ + "-----------", "-----------"), + total = sum(store_mem_type_cnt.values()) + for mem_type, count in + sorted(store_mem_type_cnt.most_common(),\
tools/perf/scripts/python/mem-phys-addr.py line 66
Quoted 3 lines
tools/perf/scripts/python/mem-phys-addr.py new file +3
-0+ key = lambda(k, v): (v, k), reverse = True): + print "%-40s %10d %10.1f%%\n" % (mem_type, count, + 100 *count / total),
Review
Quoted 49 lines
tools/perf/scripts/python/mem-phys-addr.py new file +48
-0+ +def trace_begin(): + parse_iomem() + +def trace_end(): + print_memory_type() + f.close() + +def is_system_ram(phys_addr): + #/proc/iomem is sorted + position = bisect.bisect(system_ram, phys_addr) + if position % 2 == 0: + return False + return True + +def is_persistent_mem(phys_addr): + position = bisect.bisect(pmem, phys_addr) + if position % 2 == 0: + return False + return True + +def find_memory_type(phys_addr): + if phys_addr == 0: + return "N/A" + if is_system_ram(phys_addr): + return "System RAM" + + if is_persistent_mem(phys_addr): + return "Persistent Memory" + + #slow path, search all + f.seek(0, 0) + for j in f: + m = re.split('-|:',j,2) + if long(m[0], 16) <= phys_addr <= long(m[1], 16): + return m[2] + return "N/A" + +def process_event(param_dict): + name = param_dict["ev_name"] + sample = param_dict["sample"] + phys_addr = sample["phys_addr"] + + if any(x in name for x in load_event): + load_mem_type_cnt[find_memory_type(phys_addr)] += 1 + if any(x in name for x in store_event): + store_mem_type_cnt[find_memory_type(phys_addr)] += + 1(quoted diff) +0
-0
b/tools/perf/util/scripting-engines/trace-event-python.ctools/perf/util/scripting-engines/trace-event-python.c static PyObject *get_perf_sample_dict(struct
Quoted 4 lines
index c7187f0..8cd6317 100644
tools/perf/util/scripting-engines/trace-event-python.c +0
-0@@ -500,6 +500,8 @@ static PyObject *get_perf_sample_dict(struct
perf_sample*sample,Quoted 13 lines
tools/perf/util/scripting-engines/trace-event-python.c +3
-1PyLong_FromUnsignedLongLong(sample->time));pydict_set_item_string_decref (dict_sample,"period" ,PyLong_FromUnsignedLongLong (sample->period)); +pydict_set_item_string_decref (dict_sample,"phys_addr" , + +PyLong_FromUnsignedLongLong (sample->phys_addr));set_sample_read_in_dict (dict_sample, sample, evsel);pydict_set_item_string_decref (dict,"sample" , dict_sample); --2.7.4
Quoted 3 lines
Right, it doesn’t need load latency.
0x81d0should be a better choice. I will use0x81d0and0x82d0as default event for V2.
That's model specific. You would need to check the model number if you do that.
Also with modern perf you can use the correct event names of course.
-Andi
Quoted 8 lines
Quoted 3 lines
Right, it doesn’t need load latency.
0x81d0should be a better choice. I will use0x81d0and0x82d0as default event for V2.That's model specific. You would need to check the model number if you do that.
Also with modern perf you can use the correct event names of course.
The event names are model specific as well.
For Mem load events,
MEM_INST_RETIRED.ALL_LOADS is the event name for skylake. MEM_UOPS_RETIRED.ALL_LOADS is the event name for the rest of platforms. I think it still need to check the model number.
Thanks,
Kan
Quoted 7 lines · Kan
Quoted 7 lines
Quoted 3 lines
Right, it doesn’t need load latency.
0x81d0should be a better choice. I will use0x81d0and0x82d0as default event for V2.That's model specific. You would need to check the model number if you do that.
Quoted 7 lines
Quoted 1 line
Also with modern perf you can use the correct event names of course.
The event names are model specific as well.
For Mem load events,
MEM_INST_RETIRED.ALL_LOADS is the event name for skylake. MEM_UOPS_RETIRED.ALL_LOADS is the event name for the rest of platforms. I think it still need to check the model number.
You can just try the common variations, or grep them in perf list output.
-Andi