powerpc/vpa_pmu: Add interface to expose vpa counters via perf
Subsystems (2)
PATCH 1/2 powerpc/vpa_pmu: Add interface to expose vpa counters via perf
The pseries Shared Processor Logical Partition(SPLPAR) machines can retrieve a log of dispatch and preempt events from the hypervisor using data from Disptach Trace Log(DTL) buffer. With this information, user can retrieve when and why each dispatch & preempt has occurred. Added an interface to expose the Virtual Processor Area(VPA) DTL counters via perf.
The following events are available and exposed in sysfs:
vpa_dtl/dtl_cede/ - Trace voluntary (OS initiated) virtual processor waits vpa_dtl/dtl_preempt/ - Trace time slice preempts vpa_dtl/dtl_fault/ - Trace virtual partition memory page faults. vpa_dtl/dtl_all/ - Trace all (dtl_cede/dtl_preempt/dtl_fault)
Added interface defines supported event list, config fields for the event attributes and their corresponding bit values which are exported via sysfs. User could use the standard perf tool to access perf events exposed via vpa-dtl pmu.
The VPA DTL PMU counters do not interrupt on overflow or generate any PMI interrupts. Therefore, the kernel needs to poll the counters, added hrtimer code to do that. The timer interval can be provided by user via sample_period field in nano seconds.
Result on power10 SPLPAR system with 656 cpu threads.
In the below perf record command with vpa_dtl pmu, -c option is used to provide sample_period whch corresponding to 1000000000ns i.e; 1sec and the workload time is also 1 second, hence we are getting 656 samples:
[command] perf record -a -R -e vpa_dtl/dtl_all/ -c 1000000000 sleep 1 [ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.828 MB perf.data (656 samples) ]
There is one hrtimer added per vpa-dtl pmu thread. Code added to handle addition of dtl buffer data in the raw sample. Since DTL does not provide IP address for a sample and it just have traces on reason of dispatch/preempt, we directly saving DTL buffer data to perf.data file as raw sample. For each hrtimer restart call, interface will dump all the new dtl entries added to dtl buffer as a raw sample.
To ensure there are no other conflicting dtl users (example: debugfs dtl or /proc/powerpc/vcpudispatch_stats), interface added code to use "down_write_trylock" call to take the dtl_access_lock. The dtl_access_lock is defined in dtl.h file. Also added global reference count variable called "dtl_global_refc", to ensure dtl data can be captured per-cpu. Code also added global lock called "dtl_global_lock" to avoid race condition.
---
Notes:
- Made code changes on top of recent fix sent by Michael Ellerman.
Link to the patch: https://patchwork.ozlabs.org/project/linuxppc-dev/patch/[email protected]/
arch/powerpc/perf/Makefile | 2 +- arch/powerpc/perf/vpa-pmu.c | 469 ++++++++++++++++++++++++++++++++++++ include/linux/cpuhotplug.h | 1 + 3 files changed, 471 insertions(+), 1 deletion(-) create mode 100644 arch/powerpc/perf/vpa-pmu.c
arch/powerpc/perf/Makefile +1 -1
@@ -14,7 +14,7 @@ obj-$(CONFIG_PPC_POWERNV) += imc-pmu.o obj-$(CONFIG_FSL_EMB_PERF_EVENT) += core-fsl-emb.o obj-$(CONFIG_FSL_EMB_PERF_EVENT_E500) += e500-pmu.o e6500-pmu.o -obj-$(CONFIG_HV_PERF_CTRS) += hv-24x7.o hv-gpci.o hv-common.o +obj-$(CONFIG_HV_PERF_CTRS) += hv-24x7.o hv-gpci.o hv-common.o vpa-pmu.o obj-$(CONFIG_PPC_8xx) += 8xx-pmu.o
arch/powerpc/perf/vpa-pmu.c new file +469 -0
469 lines not on this page · raw message
include/linux/cpuhotplug.h +1 -0
@@ -234,6 +234,7 @@ enum cpuhp_state {CPUHP_AP_PERF_POWERPC_TRACE_IMC_ONLINE , CPUHP_AP_PERF_POWERPC_HV_24x7_ONLINE,CPUHP_AP_PERF_POWERPC_HV_GPCI_ONLINE , +CPUHP_AP_PERF_POWERPC_VPA_DTL_ONLINE ,CPUHP_AP_PERF_CSKY_ONLINE ,CPUHP_AP_TMIGR_ONLINE , CPUHP_AP_WATCHDOG_ONLINE,
Signature · 1 line
2.43.5
PATCH 2/2 docs: ABI: sysfs-bus-event_source-devices-vpa-dtl: Document sysfs event format entries for vpa_dtl pmu
Details are added for the vpa_dtl pmu event and format attributes in the ABI documentation.
--- .../sysfs-bus-event_source-devices-vpa-dtl | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 Documentation/ABI/testing/sysfs-bus-event_source-devices-vpa-dtl
Documentation/ABI/testing/sysfs-bus-event_source-devices-vpa-dtl new file +25 -0
@@ -0,0 +1,25 @@ +What: /sys/bus/event_source/devices/vpa_dtl/format +Date: August 2024 +Contact: Linux on PowerPC Developer List <[email protected]> +Description: Read-only. Attribute group to describe the magic bits + that go into perf_event_attr.config for a particular pmu. + (See ABI/testing/sysfs-bus-event_source-devices-format). + + Each attribute under this group defines a bit range of the + perf_event_attr.config. Supported attribute are listed + below:: + + event = "config:0-7" - event ID + + For example:: + + dtl_cede = "event=0x1" + +What: /sys/bus/event_source/devices/vpa_dtl/events +Date: August 2024 +Contact: Linux on PowerPC Developer List <[email protected]> +Description: (RO) Attribute group to describe performance monitoring events + for the Virtual Processor Dispatch Trace Log. Each attribute in + this group describes a single performance monitoring event + supported by vpa_dtl pmu. The name of the file is the name of + the event (See ABI/testing/sysfs-bus-event_source-devices-events).
Signature · 1 line
2.43.5
Le 28/08/2024 à 12:21, Kajol Jain a écrit :
Quoted 51 lines
The pseries Shared Processor Logical Partition(SPLPAR) machines can retrieve a log of dispatch and preempt events from the hypervisor using data from Disptach Trace Log(DTL) buffer. With this information, user can retrieve when and why each dispatch & preempt has occurred. Added an interface to expose the Virtual Processor Area(VPA) DTL counters via perf.
The following events are available and exposed in sysfs:
vpa_dtl/dtl_cede/ - Trace voluntary (OS initiated) virtual processor waits vpa_dtl/dtl_preempt/ - Trace time slice preempts vpa_dtl/dtl_fault/ - Trace virtual partition memory page faults. vpa_dtl/dtl_all/ - Trace all (dtl_cede/dtl_preempt/dtl_fault)
Added interface defines supported event list, config fields for the event attributes and their corresponding bit values which are exported via sysfs. User could use the standard perf tool to access perf events exposed via vpa-dtl pmu.
The VPA DTL PMU counters do not interrupt on overflow or generate any PMI interrupts. Therefore, the kernel needs to poll the counters, added hrtimer code to do that. The timer interval can be provided by user via
sample_periodfield in nano seconds.Result on power10 SPLPAR system with 656 cpu threads.
In the below perf record command withvpa_dtlpmu, -c option is used to providesample_periodwhch corresponding to 1000000000ns i.e; 1sec and the workload time is also 1 second, hence we are getting 656 samples:[command] perf record -a -R -e vpa_dtl/dtl_all/ -c 1000000000 sleep 1 [ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.828 MB perf.data (656 samples) ]There is one hrtimer added per vpa-dtl pmu thread. Code added to handle addition of dtl buffer data in the raw sample. Since DTL does not provide IP address for a sample and it just have traces on reason of dispatch/preempt, we directly saving DTL buffer data to perf.data file as raw sample. For each hrtimer restart call, interface will dump all the new dtl entries added to dtl buffer as a raw sample.
To ensure there are no other conflicting dtl users (example: debugfs dtl or
/proc/powerpc/vcpudispatch_stats), interface added code to use "down_write_trylock" call to take thedtl_access_lock. Thedtl_access_lockis defined indtl.hfile. Also added global reference count variable called "dtl_global_refc", to ensure dtl data can be captured per-cpu. Code also added global lock called "dtl_global_lock" to avoid race condition.Signed-off-by Kajol Jain <[email protected]>---
Notes:
Quoted 8 lines
- Made code changes on top of recent fix sent by Michael Ellerman.
Link to the patch: https://patchwork.ozlabs.org/project/linuxppc-dev/patch/[email protected]/arch/powerpc/perf/Makefile | 2 +- arch/powerpc/perf/vpa-pmu.c | 469 ++++++++++++++++++++++++++++++++++++ include/linux/cpuhotplug.h | 1 + 3 files changed, 471 insertions(+), 1 deletion(-)create mode 100644
arch/powerpc/perf/vpa-pmu.c
Seems like it doesn't build on PPC64:
arch/powerpc/perf/vpa-pmu.c#L212
passing argument 1 of 'up_write' from incompatible pointer type [-Wincompatible-pointer-types]
arch/powerpc/perf/vpa-pmu.c#L261
passing argument 1 of 'down_write_trylock' from incompatible pointer type [-Wincompatible-pointer-types]
arch/powerpc/perf/vpa-pmu.c#L402
passing argument 1 of 'up_write' from incompatible pointer type [-Wincompatible-pointer-types]
Quoted 74 lines · Christophe Leroy
Le 28/08/2024 à 12:21, Kajol Jain a écrit :
Quoted 58 lines
The pseries Shared Processor Logical Partition(SPLPAR) machines can retrieve a log of dispatch and preempt events from the hypervisor using data from Disptach Trace Log(DTL) buffer. With this information, user can retrieve when and why each dispatch & preempt has occurred. Added an interface to expose the Virtual Processor Area(VPA) DTL counters via perf.
The following events are available and exposed in sysfs:
vpa_dtl/dtl_cede/ - Trace voluntary (OS initiated) virtual processor waits
vpa_dtl/dtl_preempt/ - Trace time slice preempts vpa_dtl/dtl_fault/ - Trace virtual partition memory page faults. vpa_dtl/dtl_all/ - Trace all (dtl_cede/dtl_preempt/dtl_fault)Added interface defines supported event list, config fields for the event attributes and their corresponding bit values which are exported via sysfs. User could use the standard perf tool to access perf events exposed via vpa-dtl pmu.
The VPA DTL PMU counters do not interrupt on overflow or generate any PMI interrupts. Therefore, the kernel needs to poll the counters, added hrtimer code to do that. The timer interval can be provided by user via
sample_periodfield in nano seconds.Result on power10 SPLPAR system with 656 cpu threads.
In the below perf record command withvpa_dtlpmu, -c option is used to providesample_periodwhch corresponding to 1000000000ns i.e; 1sec and the workload time is also 1 second, hence we are getting 656 samples:[command] perf record -a -R -e vpa_dtl/dtl_all/ -c 1000000000 sleep 1 [ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.828 MB perf.data (656 samples) ]There is one hrtimer added per vpa-dtl pmu thread. Code added to handle addition of dtl buffer data in the raw sample. Since DTL does not provide IP address for a sample and it just have traces on reason of dispatch/preempt, we directly saving DTL buffer data to perf.data file as raw sample. For each hrtimer restart call, interface will dump all the new dtl entries added to dtl buffer as a raw sample.
To ensure there are no other conflicting dtl users (example: debugfs dtl or
/proc/powerpc/vcpudispatch_stats), interface added code to use "down_write_trylock" call to take thedtl_access_lock. Thedtl_access_lock
is defined indtl.hfile. Also added global reference count variable called
"dtl_global_refc", to ensure dtl data can be captured per-cpu. Code also added global lock called "dtl_global_lock" to avoid race condition.Signed-off-by Kajol Jain <[email protected]>---
Notes:- Made code changes on top of recent fix sent by Michael Ellerman. Link to the patch:
https://patchwork.ozlabs.org/project/linuxppc-dev/patch/[email protected]/Quoted 5 lines
arch/powerpc/perf/Makefile | 2 +-
arch/powerpc/perf/vpa-pmu.c| 469 ++++++++++++++++++++++++++++++++++++ include/linux/cpuhotplug.h | 1 +3 files changed, 471 insertions(+), 1 deletion(-)create mode 100644
arch/powerpc/perf/vpa-pmu.cSeems like it doesn't build on PPC64:
arch/powerpc/perf/vpa-pmu.c#L212
passing argument 1 of 'up_write' from incompatible pointer type [-Wincompatible-pointer-types]
Quoted 7 lines
arch/powerpc/perf/vpa-pmu.c#L261
passing argument 1 of 'down_write_trylock' from incompatible pointer type [-Wincompatible-pointer-types]arch/powerpc/perf/vpa-pmu.c#L402
passing argument 1 of 'up_write' from incompatible pointer type [-Wincompatible-pointer-types]
Hi Christophe,
Thanks for checking the patch. These changes are on top of fix patch
sent by Michael Ellerman
https://patchwork.ozlabs.org/project/linuxppc-dev/patch/[email protected]/
Since he changed the dtl_access_lock to be a rw_semaphore.
Are you trying with Michael patch changes?
Thanks,
Kajol Jain
Le 13/09/2024 à 10:35, kajoljain a écrit :
Quoted 85 lines · Christophe Leroy
Quoted 74 lines
Le 28/08/2024 à 12:21, Kajol Jain a écrit :
Quoted 58 lines
The pseries Shared Processor Logical Partition(SPLPAR) machines can retrieve a log of dispatch and preempt events from the hypervisor using data from Disptach Trace Log(DTL) buffer. With this information, user can retrieve when and why each dispatch & preempt has occurred. Added an interface to expose the Virtual Processor Area(VPA) DTL counters via perf.
The following events are available and exposed in sysfs:
vpa_dtl/dtl_cede/ - Trace voluntary (OS initiated) virtual processor
waits
vpa_dtl/dtl_preempt/ - Trace time slice preempts vpa_dtl/dtl_fault/ - Trace virtual partition memory page faults. vpa_dtl/dtl_all/ - Trace all (dtl_cede/dtl_preempt/dtl_fault)Added interface defines supported event list, config fields for the event attributes and their corresponding bit values which are exported via sysfs. User could use the standard perf tool to access perf events exposed via vpa-dtl pmu.
The VPA DTL PMU counters do not interrupt on overflow or generate any PMI interrupts. Therefore, the kernel needs to poll the counters, added hrtimer code to do that. The timer interval can be provided by user via
sample_periodfield in nano seconds.Result on power10 SPLPAR system with 656 cpu threads.
In the below perf record command withvpa_dtlpmu, -c option is used to providesample_periodwhch corresponding to 1000000000ns i.e; 1sec and the workload time is also 1 second, hence we are getting 656 samples:[command] perf record -a -R -e vpa_dtl/dtl_all/ -c 1000000000 sleep 1 [ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.828 MB perf.data (656 samples) ]There is one hrtimer added per vpa-dtl pmu thread. Code added to handle addition of dtl buffer data in the raw sample. Since DTL does not provide IP address for a sample and it just have traces on reason of dispatch/preempt, we directly saving DTL buffer data to perf.data file as raw sample. For each hrtimer restart call, interface will dump all the new dtl entries added to dtl buffer as a raw sample.
To ensure there are no other conflicting dtl users (example: debugfs dtl or
/proc/powerpc/vcpudispatch_stats), interface added code to use "down_write_trylock" call to take thedtl_access_lock. Thedtl_access_lock
is defined indtl.hfile. Also added global reference count variable called
"dtl_global_refc", to ensure dtl data can be captured per-cpu. Code also added global lock called "dtl_global_lock" to avoid race condition.Signed-off-by Kajol Jain <[email protected]>---
Notes:- Made code changes on top of recent fix sent by Michael Ellerman. Link to the patch:
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatchwork.ozlabs.org%2Fproject%2Flinuxppc-dev%2Fpatch%2F20240819122401.513203-1-mpe%40ellerman.id.au%2F&data=05%7C02%7Cchristophe.leroy%40csgroup.eu%7C95cfb2842b2a44907c9108dcd3cf0b12%7C8b87af7d86474dc78df45f69a2011bb5%7C0%7C0%7C638618133431151306%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=IqFjcvT9G0HYGIbuUWhCrnXkFr9yMtSC1mGFFKZ66MI%3D&reserved=0Quoted 5 lines
arch/powerpc/perf/Makefile | 2 +- arch/powerpc/perf/vpa-pmu.c | 469 ++++++++++++++++++++++++++++++++++++ include/linux/cpuhotplug.h | 1 + 3 files changed, 471 insertions(+), 1 deletion(-)create mode 100644
arch/powerpc/perf/vpa-pmu.cSeems like it doesn't build on PPC64:
arch/powerpc/perf/vpa-pmu.c#L212
passing argument 1 of 'up_write' from incompatible pointer type [-Wincompatible-pointer-types]Quoted 7 lines
arch/powerpc/perf/vpa-pmu.c#L261
passing argument 1 of 'down_write_trylock' from incompatible pointer type [-Wincompatible-pointer-types]arch/powerpc/perf/vpa-pmu.c#L402
passing argument 1 of 'up_write' from incompatible pointer type [-Wincompatible-pointer-types]
Quoted 8 lines
Hi Christophe,
Thanks for checking the patch. These changes are on top of fix patch
sent by Michael Ellerman
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatchwork.ozlabs.org%2Fproject%2Flinuxppc-dev%2Fpatch%2F20240819122401.513203-1-mpe%40ellerman.id.au%2F&data=05%7C02%7Cchristophe.leroy%40csgroup.eu%7C95cfb2842b2a44907c9108dcd3cf0b12%7C8b87af7d86474dc78df45f69a2011bb5%7C0%7C0%7C638618133431160525%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=r1vQxb%2F8IXBRffDtFMwiqbKnFZ4iiqer6cLykT00%2Bew%3D&reserved=0Since he changed the
dtl_access_lockto be arw_semaphore.Are you trying with Michael patch changes?
No, I only saw CI test failure here : https://github.com/linuxppc/linux-snowpatch/actions/runs/10594868105
Sorry I didn't see you mentioned it in a note.
Christophe