Non perf based Gen Graphics OA unit driver
Subsystems (4)
cover Non perf based Gen Graphics OA unit driver
After some recent progress enabling the Observation Architecture unit for Gen8+, we can hopefully paint a fairly complete picture of the requirements for supporting the unit from Haswell to Skylake and so I'm looking again at the challenges in upstreaming this work.
Considering this, it looked like it could be worthwhile experimenting with a non-perf based driver for the OA unit and I'm hoping to explain why and how it went as well as request some feedback on whether we should aim to move forward without perf.
Besides the patches forwarded here, a branch can be found for reference here:
https://github.com/rib/linux - wip/rib/oa-without-perf branch
I created corresponding branches for Mesa and GPU Top to test this here (same branch names):
https://github.com/rib/mesa
https://github.com/rib/gputop
Here I've only included the patches up to an initial Haswell driver, although the wip/rib/oa-without-perf branch on github includes support for Gen8+. Please let me know if it would be helpful to forward more.
At this point I have two drivers at feature parity; one based on perf, one not. Technically they're very similar and the patches are split to hopefully be quite comparable. My latest perf-based work is under wip/rib/oa-next branches in the above repos.
So, these are the concerns I have a.t.m about upstreaming this work:
- We're bridging two complex architectures
To review this work I think it will be relevant to have a good general familiarity with Gen graphics (e.g. thinking about the OA unit's interaction with the command streamer and execlist scheduling) as well as our userspace architecture and how we're consuming OA data within Mesa to implement the INTEL_performance_query extension.
On the flip side here, its necessary to understand the perf userspace interface (for most this is hidden by tools so the details aren't common knowledge) as well as the internal design, considering that the PMU we're looking at seems to break several current design assumptions. I can only claim a limited familiarity with perf's design, just as a result of this work.
- Limited documentation for the OA unit:
Not unique to the OA unit but I think having a driver that extends outside of the graphics stack, into the core perf infrastructure probably requires more comprehensive HW + graphics stack documentation for non drm/i915 developers. Earlier RFC discussions were hampered somewhat by limited documentation. Improved documentation is always desirable, but of course it can also take a significant amount of time and effort while some key aspects (notably the PRMs) aren't directly under my control.
- The current OA PMU driver breaks some significant design assumptions.
Existing perf pmus are used for profiling work on a cpu and we're introducing the idea of _IS_DEVICE pmus with different security implications, the need to fake cpu-related data (such as user/kernel registers) to fit with perf's current design, and adding _DEVICE records as a way to forward device-specific status records.
The OA unit writes reports of counters into a circular buffer, without involvement from the CPU, making our PMU driver the first of a kind.
Given the way we periodically forward data from the OA buffer to perf's buffer, these bursts of sample writes look to perf like we're sampling too fast and so it throttles us.
Perf supports groups of counters and allows those to be read via transactions internally but transactions currently seem designed to be explicitly initiated from the cpu (say in response to a userspace read()) and while we could pull a report out of the OA buffer we can't trigger a report from the cpu on demand.
Related to being report based; the OA counters are configured in HW as a set while perf generally expects counter configurations to be orthogonal. Although counters can be associated with a group leader as they are opened, there's no clear precedent for being able to provide group-wide configuration attributes and no obvious solution as yet that's expected to be acceptable to upstream and meets our userspace needs. We currently avoid using perf's grouping feature and forward OA reports to userspace via perf's 'raw' sample field. This suits our userspace well considering how coupled the counters are when dealing with normalizing. It would be inconvenient to split counters up into separate events, only to require userspace to recombine them. For Mesa it's also convenient to be forwarded raw, periodic reports for combining with the raw reports it captures using MI_REPORT_PERF_COUNT commands.
Related to counter orthogonality; we can't time share the OA unit, while event scheduling is a central design idea within perf for allowing userspace to open + enable more events than can be configured in HW at any one time. The OA unit is not designed to allow re-configuration while in use. We can't reconfigure the OA unit without loosing internal OA unit state which we can't access explicitly to save and restore. Reconfiguring the OA unit is also relatively slow, involving ~100 register writes. From userspace Mesa also depends on a stable OA configuration when emitting MI_REPORT_PERF_COUNT commands and importantly the OA unit can't be disabled while there are outstanding MI_RPC commands lest we hang the command streamer.
- We may be making some technical compromises a.t.m for the sake of using perf.
perf_event_open() requires events to either relate to a pid or a specific cpu core, while our device pmu relates to neither. Events opened with a pid will be automatically enabled/disabled according to the scheduling of that process - so not appropriate for us. When an event is related to a cpu id, perf ensures pmu methods will be invoked via an inter process interrupt on that core. To avoid invasive changes our userspace opens OA perf events for a specific cpu. This is workable but it means the majority of the OA driver now runs in atomic context, including all OA report forwarding, which isn't really necessary in our case and seems to make our locking requirements somewhat complex as we handle the interaction with the rest of the i915 driver.
- I'm not confident our use case benefits much from building on perf:
We aren't using existing perf based tooling with our PMU. Existing tools typically assume you're profiling work running on a cpu, e.g. expecting samples to be associated with instruction pointers and user/kernel registers and aiming to represent metrics in relation to application source code. We're forwarding fake register values and userspace needs needs to know how to decode the raw OA reports before anything can be reported to a user.
With the buffering done by the OA unit I don't think we currently benefit from perf's mmapped circular buffer interface. We already have a decoupled producer and consumer and since we have to copy out of the OA buffer, it would work well for us to hide that copy in a simpler read() based interface.
- Logistically it might be more practical to contain this to the graphics stack.
It seems fair to consider that if we can't see a very compelling benefit to building on perf, then containing this work to drivers/gpu/drm/i915 may simplify the review process as well as future maintenance and development.
About the initial non-perf driver:
Structurally it's very similar to the perf based implementation. The userspace interface is inspired by perf and adds a DRM_IOCTL_I915_PERF_OPEN ioctl that's conceptually comparable to perf_event_open() returning an fd that userspace can poll() and read() samples from. The fds also supports I915_PERF_IOCTL_ENABLE/DISABLE ioctls much like perf.
When opening an event; users specify an event type (enum based a.t.m like perf's built in event types) and a pointer to an event-specific attributes structure which is extensible in the same way as struct i915_perf_event_attr. Metrics can optionally be opened for a specific GPU context (comparable to passing a pid to perf_event_open()) and the contents of samples can be controlled via a sample_flags member as with perf.
Userspace collects samples via read() which writes (only complete) records to the user's given buffer. Records have a type + size header equivalent to struct i915_perf_event_header.
Updating Mesa and GPU Top to experiment with this was straightforward given the similarity to the perf interface. The main difference is that it only supports forwarding metrics via read()s instead of an mmaped circular buffer. As mentioned above, I think that suits this well, and requires no additional copying of data. I think the userspace code has ended up being a little simpler too.
Overall the driver currently isn't much more code than with perf (~200 lines).
Personally my gut feeling a.t.m, is that we should aim to move forward independent from perf.
I'd really appreciate some feedback from others on this though.
Daniel and Chris; although I think it made sense at the outset to try and use perf, in light of the above would you be open to a non-perf based driver for the OA unit?
Peter; I wonder if you would tend to agree too that it could make sense for us to go with our own interface here?
Kind Regards,
Robert
Robert Bragg (6):
drm/i915: Add i915 perf infrastructure drm/i915: rename OACONTROL GEN7_OACONTROL drm/i915: Add static '3D' Haswell OA unit config drm/i915: Add i915 perf event for Haswell OA unit drm/i915: Add dev.i915.perf_event_paranoid sysctl option drm/i915: add oa_event_min_timer_exponent sysctl
drivers/gpu/drm/i915/Makefile | 4 + drivers/gpu/drm/i915/i915_cmd_parser.c | 4 +- drivers/gpu/drm/i915/i915_dma.c | 7 + drivers/gpu/drm/i915/i915_drv.h | 136 ++++ drivers/gpu/drm/i915/i915_gem_context.c | 23 +- drivers/gpu/drm/i915/i915_oa_hsw.c | 98 +++ drivers/gpu/drm/i915/i915_oa_hsw.h | 36 + drivers/gpu/drm/i915/i915_perf.c | 1201 +++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_reg.h | 340 ++++++++- include/uapi/drm/i915_drm.h | 125 ++++ 10 files changed, 1967 insertions(+), 7 deletions(-) create mode 100644 drivers/gpu/drm/i915/i915_oa_hsw.c create mode 100644 drivers/gpu/drm/i915/i915_oa_hsw.h create mode 100644 drivers/gpu/drm/i915/i915_perf.c
Signature · 1 line
2.5.2
RFC 1/6 drm/i915: Add i915 perf infrastructure
This adds a DRM_IOCTL_I915_PERF_OPEN ioctl comparable to perf_event_open that opens a file descriptor for an event source.
Based on our initial experience aiming to use the core perf infrastructure, this interface is inspired by perf, but focused on exposing metrics about work running on Gen graphics instead a CPU.
One notable difference is that it doesn't support mmaping a circular buffer of samples into userspace. The currently planned use cases require an internal buffering that forces at least one copy of data which can be neatly hidden in a read() based interface.
No specific event types are supported yet so perf_event_open can currently only get as far as returning EINVAL for an unknown event type.
--- drivers/gpu/drm/i915/Makefile | 3 + drivers/gpu/drm/i915/i915_dma.c | 7 + drivers/gpu/drm/i915/i915_drv.h | 74 +++++++ drivers/gpu/drm/i915/i915_perf.c | 447 +++++++++++++++++++++++++++++++++++++++ include/uapi/drm/i915_drm.h | 62 ++++++ 5 files changed, 593 insertions(+) create mode 100644 drivers/gpu/drm/i915/i915_perf.c
drivers/gpu/drm/i915/Makefile +3 -0
@@ -93,6 +93,9 @@ i915-y += dvo_ch7017.o \ # virtual gpu code i915-y += i915_vgpu.o +# perf code +i915-y += i915_perf.o + # legacy horrors i915-y += i915_dma.o
drivers/gpu/drm/i915/i915_dma.c +7 -0
@@ -841,6 +841,11 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags)mutex_init (&dev_priv->modeset_restore_lock);mutex_init (&dev_priv->csr_lock); +/* Must at least be initialized before trying to pin any context +* which i915_perf hooks into. +*/ +i915_perf_init (dev); +intel_pm_setup (dev);intel_display_crc_init (dev);
@@ -1090,6 +1095,7 @@ int i915_driver_unload(struct drm_device *dev)return ret; } +i915_perf_fini (dev);intel_power_domains_fini (dev_priv);intel_gpu_ips_teardown ();
@@ -1280,6 +1286,7 @@ const struct drm_ioctl_desc i915_ioctls[] = {DRM_IOCTL_DEF_DRV (I915_GEM_USERPTR , i915_gem_userptr_ioctl,DRM_UNLOCKED |DRM_RENDER_ALLOW ),DRM_IOCTL_DEF_DRV (I915_GEM_CONTEXT_GETPARAM , i915_gem_context_getparam_ioctl,DRM_UNLOCKED |DRM_RENDER_ALLOW ),DRM_IOCTL_DEF_DRV (I915_GEM_CONTEXT_SETPARAM , i915_gem_context_setparam_ioctl,DRM_UNLOCKED |DRM_RENDER_ALLOW ), + DRM_IOCTL_DEF_DRV(I915_PERF_OPEN, i915_perf_open_ioctl, DRM_UNLOCKED|DRM_RENDER_ALLOW), };int i915_max_ioctl =ARRAY_SIZE (i915_ioctls);
drivers/gpu/drm/i915/i915_drv.h +74 -0
98 lines not on this page · raw message
drivers/gpu/drm/i915/i915_perf.c new file +447 -0
447 lines not on this page · raw message
include/uapi/drm/i915_drm.h +62 -0
78 lines not on this page · raw message
Signature · 6 lines
2.5.2 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
RFC 2/6 drm/i915: rename OACONTROL GEN7_OACONTROL
In reply to Robert BraggOACONTROL changes quite a bit for gen8, with some bits split out into a per-context OACTXCONTROL register
--- drivers/gpu/drm/i915/i915_cmd_parser.c | 4 ++-- drivers/gpu/drm/i915/i915_reg.h | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-)
drivers/gpu/drm/i915/i915_cmd_parser.c +2 -2
@@ -439,7 +439,7 @@ static const struct drm_i915_reg_descriptor gen7_render_regs[] = {REG64 (CL_PRIMITIVES_COUNT ),REG64 (PS_INVOCATION_COUNT ),REG64 (PS_DEPTH_COUNT ), -REG32 (OACONTROL ),/* Only allowed for LRI and SRM. See below. */ +REG32 (GEN7_OACONTROL ),/* Only allowed for LRI and SRM. See below. */ REG64 (MI_PREDICATE_SRC0 ),REG64 (MI_PREDICATE_SRC1 ), REG32(GEN7_3DPRIM_END_OFFSET),
@@ -1020,7 +1020,7 @@ static bool check_cmd(const struct intel_engine_cs *ring,* to the register. Hence, limit OACONTROL writes to * only MI_LOAD_REGISTER_IMM commands. */ -if (reg_addr ==OACONTROL ) { +if (reg_addr ==GEN7_OACONTROL ) {if (desc->cmd.value ==MI_LOAD_REGISTER_MEM (1 )) {DRM_DEBUG_DRIVER ("CMD: Rejected LRM to OACONTROL\n" );return false;
drivers/gpu/drm/i915/i915_reg.h +1 -1
@@ -536,7 +536,7 @@#define GEN7_3DPRIM_START_INSTANCE 0x243C#define GEN7_3DPRIM_BASE_VERTEX 0x2440 -#define OACONTROL 0x2360 +#define GEN7_OACONTROL 0x2360#define _GEN7_PIPEA_DE_LOAD_SL 0x70068#define _GEN7_PIPEB_DE_LOAD_SL 0x71068
Signature · 6 lines
2.5.2 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
RFC 3/6 drm/i915: Add static '3D' Haswell OA unit config
In reply to Robert BraggAdds a static OA unit, MUX + B Counter configuration for basic '3D' metrics on Haswell. This is autogenerated from an internal XML description of metric sets.
--- drivers/gpu/drm/i915/Makefile | 3 +- drivers/gpu/drm/i915/i915_drv.h | 5 ++ drivers/gpu/drm/i915/i915_oa_hsw.c | 98 ++++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_oa_hsw.h | 36 ++++++++++++++ 4 files changed, 141 insertions(+), 1 deletion(-) create mode 100644 drivers/gpu/drm/i915/i915_oa_hsw.c create mode 100644 drivers/gpu/drm/i915/i915_oa_hsw.h
drivers/gpu/drm/i915/Makefile +2 -1
@@ -94,7 +94,8 @@ i915-y += dvo_ch7017.o \ i915-y += i915_vgpu.o # perf code -i915-y += i915_perf.o +i915-y += i915_perf.o \ + i915_oa_hsw.o # legacy horrors i915-y += i915_dma.o
drivers/gpu/drm/i915/i915_drv.h +5 -0
@@ -1694,6 +1694,11 @@ struct i915_execbuffer_params {struct drm_i915_gem_request *request; }; +struct i915_oa_reg { +u32 addr; +u32 value; +}; +struct i915_perf_read_state {int count;ssize_t read;
drivers/gpu/drm/i915/i915_oa_hsw.c new file +98 -0
98 lines not on this page · raw message
drivers/gpu/drm/i915/i915_oa_hsw.h new file +36 -0
@@ -0,0 +1,36 @@ +/* +* Autogenerated file, DO NOT EDIT manually! +* +* Copyright (c) 2015 Intel Corporation +* +* Permission is hereby granted, free of charge, to any person obtaining a +* copy of this software and associated documentation files (the "Software"), +* to deal in the Software without restriction, including without limitation +* the rights to use, copy, modify, merge, publish, distribute, sublicense, +* and/or sell copies of the Software, and to permit persons to whom the +* Software is furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice (including the next +* paragraph) shall be included in all copies or substantial portions of the +* Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +* IN THE SOFTWARE. +* +*/ + +#ifndef __I915_OA_HSW_H__ +#define __I915_OA_HSW_H__ + +/* HSW Render Metrics Basic Gen7.5 */ +extern const struct i915_oa_reg i915_oa_3d_b_counter_config_hsw[]; +extern const int i915_oa_3d_b_counter_config_hsw_len; +extern const struct i915_oa_reg i915_oa_3d_mux_config_hsw[]; +extern const int i915_oa_3d_mux_config_hsw_len; + +#endif
Signature · 6 lines
2.5.2 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
RFC 4/6 drm/i915: Add i915 perf event for Haswell OA unit
In reply to Robert BraggGen graphics hardware can be set up to periodically write snapshots of performance counters into a circular buffer via its Observation Architecture and this patch exposes that capability to userspace via the i915 perf interface.
--- drivers/gpu/drm/i915/i915_drv.h | 57 +++ drivers/gpu/drm/i915/i915_gem_context.c | 23 +- drivers/gpu/drm/i915/i915_perf.c | 697 +++++++++++++++++++++++++++++++- drivers/gpu/drm/i915/i915_reg.h | 338 ++++++++++++++++ include/uapi/drm/i915_drm.h | 63 +++ 5 files changed, 1171 insertions(+), 7 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +57 -0
@@ -1694,6 +1694,11 @@ struct i915_execbuffer_params {struct drm_i915_gem_request *request; }; +struct i915_oa_format { +u32 format; +int size; +}; +struct i915_oa_reg {u32 addr;u32 value;
@@ -1760,6 +1765,20 @@ struct i915_perf_event {void (*destroy)(struct i915_perf_event *event); }; +struct i915_oa_ops { +void (*init_oa_buffer)(struct drm_i915_private *dev_priv); +void (*enable_metric_set)(struct drm_i915_private *dev_priv); +void (*disable_metric_set)(struct drm_i915_private *dev_priv); +void (*oa_enable)(struct drm_i915_private *dev_priv); +void (*oa_disable)(struct drm_i915_private *dev_priv); +void (*update_oacontrol)(struct drm_i915_private *dev_priv); +void (*update_specific_hw_ctx_id)(struct drm_i915_private *dev_priv, +u32 ctx_id); +void (*read)(struct i915_perf_event *event, +struct i915_perf_read_state *read_state); +bool (*oa_buffer_is_empty)(struct drm_i915_private *dev_priv); +}; +struct drm_i915_private {struct drm_device *dev;struct kmem_cache *objects;
@@ -1996,7 +2015,43 @@ struct drm_i915_private {struct {bool initialized; +struct mutex lock; + +struct ctl_table_header *sysctl_header; + +struct { +struct i915_perf_event *exclusive_event; + +u32 specific_ctx_id; + +struct hrtimer poll_check_timer; +wait_queue_head_t poll_wq; + +bool periodic; +u32 period_exponent; + +u32 metrics_set; + +const struct i915_oa_reg *mux_regs; +int mux_regs_len; +const struct i915_oa_reg *b_counter_regs; +int b_counter_regs_len; + +struct { +struct drm_i915_gem_object *obj; +u32 gtt_offset; +u8 *addr; +u32 head; +u32 tail; +int format; +int format_size; + } oa_buffer; + +struct i915_oa_ops ops; +const struct i915_oa_format *oa_formats; + } oa; +struct list_head events; } perf;
@@ -3204,6 +3259,8 @@ int i915_gem_context_setparam_ioctl(struct drm_device *dev, void *data,int i915_perf_open_ioctl (struct drm_device *dev,void *data,struct drm_file *file); +void i915_oa_context_pin_notify (struct drm_i915_private *dev_priv, +struct intel_context *context);/* i915_gem_evict.c */ int __must_check i915_gem_evict_something(struct drm_device *dev,
drivers/gpu/drm/i915/i915_gem_context.c +19 -4
@@ -133,6 +133,23 @@ static int get_context_size(struct drm_device *dev)return ret; } +static int i915_gem_context_pin_state (struct drm_device *dev, +struct intel_context *ctx) +{ +int ret; + +BUG_ON (!mutex_is_locked (&dev->struct_mutex)); + + ret =i915_gem_obj_ggtt_pin (ctx->legacy_hw_ctx.rcs_state, +get_context_alignment (dev),0 ); +if (ret) +return ret; + +i915_oa_context_pin_notify (dev->dev_private, ctx); + +return 0 ; +} +void i915_gem_context_free (struct kref *ctx_ref) {struct intel_context *ctx =container_of (ctx_ref,typeof (*ctx), ref);
@@ -258,8 +275,7 @@ i915_gem_create_context(struct drm_device *dev,* be available. To avoid this we always pin the default * context. */ - ret =i915_gem_obj_ggtt_pin (ctx->legacy_hw_ctx.rcs_state, -get_context_alignment (dev),0 ); + ret =i915_gem_context_pin_state (dev, ctx);if (ret) {DRM_DEBUG_DRIVER ("Couldn't pin %d\n" , ret); goto err_destroy;
@@ -634,8 +650,7 @@ static int do_switch(struct drm_i915_gem_request *req)/* Trying to pin first makes error handling easier. */ if (ring == &dev_priv->ring[RCS ]) { - ret =i915_gem_obj_ggtt_pin (to->legacy_hw_ctx.rcs_state, -get_context_alignment (ring->dev),0 ); + ret =i915_gem_context_pin_state (ring->dev, to);if (ret)return ret; }
drivers/gpu/drm/i915/i915_perf.c +694 -3
733 lines not on this page · raw message
drivers/gpu/drm/i915/i915_reg.h +338 -0
350 lines not on this page · raw message
include/uapi/drm/i915_drm.h +63 -0
84 lines not on this page · raw message
Signature · 6 lines
2.5.2 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
kbuild test robot
build botRe: [Intel-gfx] [RFC 4/6] drm/i915: Add i915 perf event for Haswell OA unit
config: i386-defconfig (attached as .config) reproduce: git checkout a1d59679ae8f3e7e7659e9723ae3fc69af2532e6
Hi Robert,
[auto build test results on v4.3-rc3 -- if it's inappropriate base, please ignore]
config: i386-defconfig (attached as .config) reproduce:
git checkout a1d59679ae8f3e7e7659e9723ae3fc69af2532e6
# save the attached .config to linux build tree make ARCH=i386
All warnings (new ones prefixed by >>):
drivers/gpu/drm/i915/i915_perf.c: In function 'append_oa_sample':
>> drivers/gpu/drm/i915/i915_perf.c:199:2: warning: #warning "fixme: extract context ID from OA reports" [-Wcpp]
#warning "fixme: extract context ID from OA reports"
^
>> drivers/gpu/drm/i915/i915_perf.c:205:2: warning: #warning "fixme: extract timestamp from OA reports" [-Wcpp]
#warning "fixme: extract timestamp from OA reports"
^
drivers/gpu/drm/i915/i915_perf.c: In function 'append_oa_status':
>> drivers/gpu/drm/i915/i915_perf.c:155:2: warning: ignoring return value of 'copy_to_user', declared with attribute warn_unused_result [-Wunused-result]
copy_to_user(read_state->buf, &header, sizeof(header));
^
drivers/gpu/drm/i915/i915_perf.c: In function 'append_oa_sample':
drivers/gpu/drm/i915/i915_perf.c:195:2: warning: ignoring return value of 'copy_to_user', declared with attribute warn_unused_result [-Wunused-result]
copy_to_user(read_state->buf, &header, sizeof(header));
^
drivers/gpu/drm/i915/i915_perf.c:200:3: warning: ignoring return value of 'copy_to_user', declared with attribute warn_unused_result [-Wunused-result]
copy_to_user(read_state->buf, &dummy_ctx_id, 4);
^
drivers/gpu/drm/i915/i915_perf.c:206:3: warning: ignoring return value of 'copy_to_user', declared with attribute warn_unused_result [-Wunused-result]
copy_to_user(read_state->buf, &dummy_timestamp, 4);
^
drivers/gpu/drm/i915/i915_perf.c:211:3: warning: ignoring return value of 'copy_to_user', declared with attribute warn_unused_result [-Wunused-result]
copy_to_user(read_state->buf, report, report_size);
^vim +199 drivers/gpu/drm/i915/i915_perf.c
149 {
150 struct drm_i915_perf_event_header header = { type, 0, sizeof(header) };
151
152 if ((read_state->count - read_state->read) < header.size)
153 return false;
154
> 155 copy_to_user(read_state->buf, &header, sizeof(header));
156
157 read_state->buf += sizeof(header);
158 read_state->read += header.size;
159
160 return true;
161 }
162
163 static bool append_oa_sample(struct i915_perf_event *event,
164 struct i915_perf_read_state *read_state,
165 const u8 *report)
166 {
167 struct drm_i915_private *dev_priv = event->dev_priv;
168 int report_size = dev_priv->perf.oa.oa_buffer.format_size;
169 struct drm_i915_perf_event_header header;
170 u32 sample_flags = event->sample_flags;
171 u32 dummy_ctx_id = 0;
172 u32 dummy_timestamp = 0;
173
174 header.type = DRM_I915_PERF_RECORD_SAMPLE;
175 header.misc = 0;
176 header.size = sizeof(header);
177
178
179 /* XXX: could pre-compute this when opening the event... */
180
181 if (sample_flags & I915_PERF_SAMPLE_CTXID)
182 header.size += 4;
183
184 if (sample_flags & I915_PERF_SAMPLE_TIMESTAMP)
185 header.size += 4;
186
187 if (sample_flags & I915_PERF_SAMPLE_OA_REPORT)
188 header.size += report_size;
189
190
191 if ((read_state->count - read_state->read) < header.size)
192 return false;
193
194
195 copy_to_user(read_state->buf, &header, sizeof(header));
196 read_state->buf += sizeof(header);
197
198 if (sample_flags & I915_PERF_SAMPLE_CTXID) {
> 199 #warning "fixme: extract context ID from OA reports"
200 copy_to_user(read_state->buf, &dummy_ctx_id, 4);
201 read_state->buf += 4;
202 }
203
204 if (sample_flags & I915_PERF_SAMPLE_TIMESTAMP) {
> 205 #warning "fixme: extract timestamp from OA reports" 206 copy_to_user(read_state->buf, &dummy_timestamp, 4);
207 read_state->buf += 4;
208 }
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
_______________________________________________
dri-devel mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [Intel-gfx] [RFC 4/6] drm/i915: Add i915 perf event for Haswell OA unit
Quoted 4 lines · kbuild test robot
Hi Robert,
[auto build test results on v4.3-rc3 -- if it's inappropriate base, please ignore]
Quoted 8 lines
config: i386-defconfig (attached as .config) reproduce:
git checkouta1d59679ae8f3e7e7659e9723ae3fc69af2532e6# save the attached .config to linux build tree make ARCH=i386All warnings (new ones prefixed by >>):
@Wu, hehe, another series pattern to match ;-)
_______________________________________________
dri-devel mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/dri-devel
Fengguang Wu
build botRe: [kbuild-all] [Intel-gfx] [RFC 4/6] drm/i915: Add i915 perf event for Haswell OA unit
Thanks! I'm now matching ^([...])? ?[... ii/NN] as patch series. :-)
On Tue, Sep 29, 2015 at 05:18:45PM +0200, Peter Zijlstra wrote:
> On Tue, Sep 29, 2015 at 10:55:39PM +0800, kbuild test robot wrote: > > Hi Robert, > > > > [auto build test results on v4.3-rc3 -- if it's inappropriate base, please ignore] > > > > config: i386-defconfig (attached as .config) > > reproduce: > > git checkout a1d59679ae8f3e7e7659e9723ae3fc69af2532e6 > > # save the attached .config to linux build tree > > make ARCH=i386 > > > > All warnings (new ones prefixed by >>): > > > > @Wu, hehe, another series pattern to match ;-)
Thanks! I'm now matching ^([...])? ?[... ii/NN] as patch series. :-)
Regards,
Fengguang
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
RFC 5/6 drm/i915: Add dev.i915.perf_event_paranoid sysctl option
In reply to Robert BraggConsistent with the kernel.perf_event_paranoid sysctl option that can allow non-root users to access system wide cpu metrics, this can optionally allow non-root users to access system wide OA counter metrics from Gen graphics hardware.
--- drivers/gpu/drm/i915/i915_perf.c | 46 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_perf.c +45 -1
@@ -38,6 +38,8 @@#define POLL_FREQUENCY 200#define POLL_PERIOD max_t(u64, 10000, NSEC_PER_SEC / POLL_FREQUENCY) +static u32 i915_perf_event_paranoid = true; +#define OA_EXPONENT_MAX 0x3fstatic struct i915_oa_format hsw_oa_formats[I915_OA_FORMAT_MAX ] = {
@@ -1016,7 +1018,13 @@ int i915_perf_open_ioctl_locked(struct drm_device *dev, void *data, } } -if (!specific_ctx && !capable (CAP_SYS_ADMIN )) { +/* Similar to perf's kernel.perf_paranoid_cpu sysctl option +* we check a dev.i915.perf_event_paranoid sysctl option +* to determine if it's ok to access system wide OA counters +* without CAP_SYS_ADMIN privileges. +*/ +if (!specific_ctx && + i915_perf_event_paranoid && !capable (CAP_SYS_ADMIN )) {DRM_ERROR ("Insufficient privileges to open perf event\n" ); ret = -EACCES ; goto err_ctx;
@@ -1096,6 +1104,38 @@ int i915_perf_open_ioctl(struct drm_device *dev, void *data,return ret; } + +static struct ctl_table oa_table[] = { + { + .procname ="perf_event_paranoid" , + .data = &i915_perf_event_paranoid, + .maxlen =sizeof (i915_perf_event_paranoid), + .mode =0644 , + .proc_handler = proc_dointvec, + }, + {} +}; + +static struct ctl_table i915_root[] = { + { + .procname ="i915" , + .maxlen =0 , + .mode =0555 , + .child = oa_table, + }, + {} +}; + +static struct ctl_table dev_root[] = { + { + .procname ="dev" , + .maxlen =0 , + .mode =0555 , + .child = i915_root, + }, + {} +}; +void i915_perf_init (struct drm_device *dev) {struct drm_i915_private *dev_priv =to_i915 (dev);
@@ -1103,6 +1143,8 @@ void i915_perf_init(struct drm_device *dev)if (!IS_HASWELL (dev))return ; + dev_priv->perf.sysctl_header =register_sysctl_table (dev_root); +hrtimer_init (&dev_priv->perf.oa.poll_check_timer,CLOCK_MONOTONIC ,HRTIMER_MODE_REL ); dev_priv->perf.oa.poll_check_timer.function = poll_check_timer_cb;
@@ -1132,6 +1174,8 @@ void i915_perf_fini(struct drm_device *dev)if (!dev_priv->perf.initialized)return ; +unregister_sysctl_table (dev_priv->perf.sysctl_header); + dev_priv->perf.oa.ops.init_oa_buffer =NULL ; dev_priv->perf.initialized = false;
Signature · 1 line
2.5.2
RFC 6/6 drm/i915: add oa_event_min_timer_exponent sysctl
In reply to Robert BraggThe minimal sampling period is now configurable via a dev.i915.oa_event_min_timer_exponent sysctl parameter.
Following the precedent set by perf, the default is the minimum that won't (on its own) exceed the default kernel.perf_event_max_sample_rate default of 100000 samples/s.
--- drivers/gpu/drm/i915/i915_perf.c | 37 ++++++++++++++++++++++++++++--------- 1 file changed, 28 insertions(+), 9 deletions(-)
drivers/gpu/drm/i915/i915_perf.c +28 -9
@@ -42,6 +42,23 @@ static u32 i915_perf_event_paranoid = true;#define OA_EXPONENT_MAX 0x3f +/* for sysctl proc_dointvec_minmax of i915_oa_event_min_timer_exponent */ +static int zero; +static int oa_exponent_max =OA_EXPONENT_MAX ; + +/* Theoretically we can program the OA unit to sample every 160ns but don't +* allow that by default unless root... +* +* The period is derived from the exponent as: +* +* period = 80ns * 2^(exponent + 1) +* +* Referring to perf's kernel.perf_event_max_sample_rate for a precedent +* (100000 by default); with an OA exponent of 6 we get a period of 10.240 +* microseconds - just under 100000Hz +*/ +static u32 i915_oa_event_min_timer_exponent =6 ; +static struct i915_oa_format hsw_oa_formats[I915_OA_FORMAT_MAX ] = { [I915_OA_FORMAT_A13 ] = {0 ,64 }, [I915_OA_FORMAT_A29 ] = {1 ,128 },
@@ -674,15 +691,8 @@ static int i915_oa_event_init(struct i915_perf_event *event,if (period_exponent >OA_EXPONENT_MAX )return -EINVAL ; -/* Theoretically we can program the OA unit to sample every -* 160ns but don't allow that by default unless root... -* -* Referring to perf's kernel.perf_event_max_sample_rate for -* a precedent (100000 by default); with an OA exponent of -* 6 we get a period of 10.240 microseconds -just under -* 100000Hz -*/ -if (period_exponent <6 && !capable (CAP_SYS_ADMIN )) { +if (period_exponent < i915_oa_event_min_timer_exponent && + !capable (CAP_SYS_ADMIN )) {DRM_ERROR ("Sampling period too high without root privileges\n" );return -EACCES ; }
@@ -1113,6 +1123,15 @@ static struct ctl_table oa_table[] = { .mode = 0644, .proc_handler = proc_dointvec, }, + { + .procname = "oa_event_min_timer_exponent", + .data = &i915_oa_event_min_timer_exponent, + .maxlen = sizeof(i915_oa_event_min_timer_exponent), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + .extra2 = &oa_exponent_max, + }, {} };
Signature · 6 lines
2.5.2 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 9 lines · Robert Bragg
- Logistically it might be more practical to contain this to the graphics stack.
It seems fair to consider that if we can't see a very compelling benefit to building on perf, then containing this work to
drivers/gpu/drm/i915may simplify the review process as well as future maintenance and development.
I think even we all initially like to go with perf but it appears later that we might need to stick this more close with i915 driver. Also think about to enable global profiling for all graphics clients, extending or enabling it within i915 specific interface seems more feasible instead of trying to create another PMU driver like previous implementation attempt to suit the need for different gfx perf data definition.
Robert, thanks for send and elaborate on this.
Signature · 3 lines
Open Source Technology Center, Intel ltd. $gpg --keyserver wwwkeys.pgp.net --recv-keys 4D781827
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 6 lines · Robert Bragg
Updating Mesa and GPU Top to experiment with this was straightforward given the similarity to the perf interface. The main difference is that it only supports forwarding metrics via read()s instead of an mmaped circular buffer. As mentioned above, I think that suits this well, and requires no additional copying of data. I think the userspace code has ended up being a little simpler too.
Did you try updating the existing perf based overlay?
Quoted 11 lines
Overall the driver currently isn't much more code than with perf (~200 lines).
Personally my gut feeling a.t.m, is that we should aim to move forward independent from perf.
I'd really appreciate some feedback from others on this though.
Daniel and Chris; although I think it made sense at the outset to try and use perf, in light of the above would you be open to a non-perf based driver for the OA unit?
No. I strongly dislike that they will be multiple incompatibile perf interfaces and strongly like the coupling with other profiling that comes with perf - i.e. we very much want to simultaneously sample CPU and GPU workloads along with other devices, that information is much more useful to me for the purposes of scheduling work and maximising concurrency than optimising shaders.
-Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
Quoted 10 lines · Chris Wilson
Quoted 6 lines · Robert Bragg
Updating Mesa and GPU Top to experiment with this was straightforward given the similarity to the perf interface. The main difference is that it only supports forwarding metrics via read()s instead of an mmaped circular buffer. As mentioned above, I think that suits this well, and requires no additional copying of data. I think the userspace code has ended up being a little simpler too.
Did you try updating the existing perf based overlay?
I don't recall the overlay attempting to read OA counters, but potentially it could be quite nice to add support - sorry I hadn't considered that so far.
I don't believe being perf based or not will affect the effort to do this though. The perf based driver doesn't handle OA counter normalization in the kernel so userspace needs to be able to handle that - which is probably the bigger effort.
Something to note here about your early pmu driver, is that it was notably for counters that were explicitly sampled from the cpu using a hrtimer via mmio. I think they were a better fit for the existing perf design than the OA unit, primarily because they were explicitly read from the cpu and each counter was very independent.
Quoted 13 lines
Quoted 4 lines
Overall the driver currently isn't much more code than with perf (~200 lines).
Quoted 8 lines
Personally my gut feeling a.t.m, is that we should aim to move forward independent from perf.
I'd really appreciate some feedback from others on this though.
Daniel and Chris; although I think it made sense at the outset to try and use perf, in light of the above would you be open to a non-perf based driver for the OA unit?
Quoted 7 lines
No. I strongly dislike that they will be multiple incompatibile perf interfaces and strongly like the coupling with other profiling that comes with perf - i.e. we very much want to simultaneously sample CPU and GPU workloads along with other devices, that information is much more useful to me for the purposes of scheduling work and maximising concurrency than optimising shaders.
In this case I don't think there's inherently any more compatibility that comes from using perf or not - no existing userspace will Just Work™ with the perf based OA driver.
I think some of the cases you're referring to may be ok to expose via the existing perf infrastructure, but I'm currently enabling the OA unit which poses some unique difficulties I've tried to explain.
A guiding differentiator may be whether or not the counter is orthogonal (in terms of configuration and normalization) and explicitly readable from the cpu, as to whether the existing perf pmu infrastructure is a good fit.
'i915 perf' shows my lack of imagination naming this and maybe another name could imply a more limited scope. I.e. on a case by case basis, when looking to expose a new counters we can still evaluate whether it makes sense to expose via the existing perf infrastructure or this.
- Robert
Quoted 10 lines
-Chris
--
Chris Wilson, Intel Open Source Technology Centre
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 9 lines · Robert Bragg
- We're bridging two complex architectures
To review this work I think it will be relevant to have a good general familiarity with Gen graphics (e.g. thinking about the OA unit's interaction with the command streamer and execlist scheduling) as well as our userspace architecture and how we're consuming OA data within Mesa to implement the
INTEL_performance_queryextension.
Quoted 6 lines
On the flip side here, its necessary to understand the perf userspace interface (for most this is hidden by tools so the details aren't common knowledge) as well as the internal design, considering that the PMU we're looking at seems to break several current design assumptions. I can only claim a limited familiarity with perf's design, just as a result of this work.
Right; but a little effort and patience on both sides should get us there I think. At worst we'll both learn something new ;-)
Quoted 7 lines
- The current OA PMU driver breaks some significant design assumptions.
Existing perf pmus are used for profiling work on a cpu and we're introducing the idea of
_IS_DEVICEpmus with different security implications, the need to fake cpu-related data (such as user/kernel registers) to fit with perf's current design, and adding_DEVICErecords as a way to forward device-specific status records.
There are more devices with counters on than GPUs, so I think it might make sense to look at extending perf to better deal with this.
Quoted 3 lines
The OA unit writes reports of counters into a circular buffer, without involvement from the CPU, making our PMU driver the first of a kind.
Agreed, this is somewhat 'odd' from where we are today.
Quoted 6 lines
Perf supports groups of counters and allows those to be read via transactions internally but transactions currently seem designed to be explicitly initiated from the cpu (say in response to a userspace
read()) and while we could pull a report out of the OA buffer we can't trigger a report from the cpu on demand.
Quoted 7 lines
Related to being report based; the OA counters are configured in HW as a set while perf generally expects counter configurations to be orthogonal. Although counters can be associated with a group leader as they are opened, there's no clear precedent for being able to provide group-wide configuration attributes and no obvious solution as yet that's expected to be acceptable to upstream and meets our userspace needs.
I'm not entirely sure what you mean with group-wide configuration attributes; could you elaborate?
Quoted 6 lines
We currently avoid using perf's grouping feature
and forward OA reports to userspace via perf's 'raw' sample field. This suits our userspace well considering how coupled the counters are when dealing with normalizing. It would be inconvenient to split counters up into separate events, only to require userspace to recombine them.
So IF you were using a group, a single read from the leader can return you a vector of all values (PERF_FORMAT_GROUP), this avoids having to do that recombine.
Another option would be to view the arrival of an OA vector in the datastream as an 'event' and generate a PERF_RECORD_READ in the perf buffer (which again can use the GROUP vector format).
Quoted 4 lines
Related to counter orthogonality; we can't time share the OA unit, while event scheduling is a central design idea within perf for allowing userspace to open + enable more events than can be configured in HW at any one time.
So we have other PMUs that cannot do this; Gen OA would not be unique in this. Intel PT for example only allows a single active event.
That said; earlier today I saw:
https://www.youtube.com/watch?v=9J3BQcAeHpI&list=PLe6I3NKr-I4J2oLGXhGOeBMEjh8h10jT3&index=7
where exactly this feature was mentioned as not fitting well into the existing GPU performance interfaces (GL_AMD_performance_monitor / GL_INTEL_performance_query).
So there is hardware (Nvidia) out there that does support this. Also mentioned was that this hardware has global and local counters, where the local ones are specific to a rendering context. That is not unlike the per-cpu / per-task stuff perf does.
Quoted 9 lines
The OA unit is not designed to
allow re-configuration while in use. We can't reconfigure the OA unit without loosing internal OA unit state which we can't access explicitly to save and restore. Reconfiguring the OA unit is also relatively slow, involving ~100 register writes. From userspace Mesa also depends on a stable OA configuration when emittingMI_REPORT_PERF_COUNTcommands and importantly the OA unit can't be disabled while there are outstandingMI_RPCcommands lest we hang the command streamer.
Right; see the PERF_PMU_CAP_EXCLUSIVE stuff.
Quoted 7 lines
- We may be making some technical compromises a.t.m for the sake of using perf.
perf_event_open()requires events to either relate to a pid or a specific cpu core, while our device pmu relates to neither. Events opened with a pid will be automatically enabled/disabled according to the scheduling of that process - so not appropriate for us.
Right; the traditional cpu/pid mapping doesn't work well for devices; but maybe, with some work, we can create something like that global/local render context from it; although I've no clue what form that would need at this time.
Quoted 5 lines
When
an event is related to a cpu id, perf ensures pmu methods will be invoked via an inter process interrupt on that core. To avoid invasive changes our userspace opens OA perf events for a specific cpu.
Some of that might still make sense in the sense that GPUs are subject to the NUMA topology of machines. I would think you would want most such things to be done on the node the device is attached to.
Granted, this might not be a concern for Intel graphics, but it might be relevant for some of the discrete GPUs.
Quoted 20 lines
- I'm not confident our use case benefits much from building on perf:
We aren't using existing perf based tooling with our PMU. Existing tools typically assume you're profiling work running on a cpu, e.g. expecting samples to be associated with instruction pointers and user/kernel registers and aiming to represent metrics in relation to application source code. We're forwarding fake register values and userspace needs needs to know how to decode the raw OA reports before anything can be reported to a user.
With the buffering done by the OA unit I don't think we currently benefit from perf's mmapped circular buffer interface. We already have a decoupled producer and consumer and since we have to copy out of the OA buffer, it would work well for us to hide that copy in a simpler
read()based interface.- Logistically it might be more practical to contain this to the graphics stack.
Quoted 7 lines
It seems fair to consider that if we can't see a very compelling benefit to building on perf, then containing this work to
drivers/gpu/drm/i915may simplify the review process as well as future maintenance and development.Peter; I wonder if you would tend to agree too that it could make sense for us to go with our own interface here?
Sorry this took so long; this wanted a well considered response and those tend to get delayed in light of 'urgent' stuff.
While I can certainly see the pain points and why you would rather not deal with them. I think it would make Linux a better place if we could manage to come up with a generic interface that would work for 'all' GPUs (and possibly more devices).
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 8 lines · Peter Zijlstra
Quoted 7 lines
- We may be making some technical compromises a.t.m for the sake of using perf.
perf_event_open()requires events to either relate to a pid or a specific cpu core, while our device pmu relates to neither. Events opened with a pid will be automatically enabled/disabled according to the scheduling of that process - so not appropriate for us.
Quoted 4 lines
Right; the traditional cpu/pid mapping doesn't work well for devices; but maybe, with some work, we can create something like that global/local render context from it; although I've no clue what form that would need at this time.
Could someone please help with some very basic questions, such as what the hardware model of the 'OA' unit model is? How are OA registers set up, how are their values made accessible to the host side, etc.
I see some references to 'OA' registers in:
https://01.org/sites/default/files/documentation/intel-gfx-prm-osrc-bdw-vol03-gpu_overview_1.pdf
and I tried to find a more high level description in:
but couldn't find it. (Maybe it's my fault!)
Thanks,
Ingo
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 11 lines · Ingo Molnar
Quoted 8 lines · Peter Zijlstra
Quoted 7 lines
- We may be making some technical compromises a.t.m for the sake of using perf.
perf_event_open()requires events to either relate to a pid or a specific cpu core, while our device pmu relates to neither. Events opened with a pid will be automatically enabled/disabled according to the scheduling of that process - so not appropriate for us.
Quoted 8 lines
Quoted 4 lines
Right; the traditional cpu/pid mapping doesn't work well for devices; but maybe, with some work, we can create something like that global/local render context from it; although I've no clue what form that would need at this time.
Could someone please help with some very basic questions, such as what the hardware model of the 'OA' unit model is? How are OA registers set up, how are their values made accessible to the host side, etc.
Robert linked to:
https://01.org/sites/default/files/documentation/observability_performance_counters_haswell.pdf
In a previous posting. It has some info, but full documentation, is as per the initial post, 'pending'.
_______________________________________________
dri-devel mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 21 lines · Peter Zijlstra
Quoted 11 lines · Ingo Molnar
Quoted 8 lines · Peter Zijlstra
Quoted 7 lines
- We may be making some technical compromises a.t.m for the sake of using perf.
perf_event_open()requires events to either relate to a pid or a specific cpu core, while our device pmu relates to neither. Events opened with a pid will be automatically enabled/disabled according to the scheduling of that process - so not appropriate for us.Quoted 8 lines
Quoted 4 lines
Right; the traditional cpu/pid mapping doesn't work well for devices; but maybe, with some work, we can create something like that global/local render context from it; although I've no clue what form that would need at this time.
Could someone please help with some very basic questions, such as what the hardware model of the 'OA' unit model is? How are OA registers set up, how are their values made accessible to the host side, etc.
Quoted 6 lines
Robert linked to:
https://01.org/sites/default/files/documentation/observability_performance_counters_haswell.pdf
In a previous posting. It has some info, but full documentation, is as per the initial post, 'pending'.
There is now also some Broadwell documentation here:
https://01.org/sites/default/files/documentation/intel-gfx-prm-osrc-bdw-vol14-observability.pdf
Unfortunately though a mistake was made by the documentation team when generating the PDF which unintentionally stripped out a lot of information so it's not very helpful a.t.m. I've let them know about some of the issues, but I'm not sure a.t.m when it may be updated.
I tried to fill in the gaps in some of our earlier conversations, so maybe also go over those for more details too.
Otherwise the best reference is probably my code currently, either the RFC patches I sent most recently which at least cover up to Haswell, or the wip/rib/oa-next branch here: https://github.com/rib/linux. The lastest perf based driver is currently in the archive/rib/oa-core-perf branch for reference too.
- Robert
Quoted 17 lines · Peter Zijlstra
Quoted 9 lines · Robert Bragg
- We're bridging two complex architectures
To review this work I think it will be relevant to have a good general familiarity with Gen graphics (e.g. thinking about the OA unit's interaction with the command streamer and execlist scheduling) as well as our userspace architecture and how we're consuming OA data within Mesa to implement the
INTEL_performance_queryextension.Quoted 6 lines
On the flip side here, its necessary to understand the perf userspace interface (for most this is hidden by tools so the details aren't common knowledge) as well as the internal design, considering that the PMU we're looking at seems to break several current design assumptions. I can only claim a limited familiarity with perf's design, just as a result of this work.
Quoted 2 lines
Right; but a little effort and patience on both sides should get us there I think. At worst we'll both learn something new ;-)
I suppose I'm also concerned time is an important factor too. When it comes to the OA metrics; we already have userspace tools that could be more widely used by developers once we have an upstream interface. Today perf isn't very well suited to our OA unit use case, and although we may be able to change that - and I can try to help with that - at this point I think I'd prefer not to block moving forward in the mean time with the alternative i915 interface.
Although code-wise it didn't require any big changes to events/core to get an initial perf based driver working for our use case, we have raised a number of quite significant design questions and arguably cut some corners, which could take a long time to resolve properly. I also tend to think it's an open question at this stage whether it would really be in everyone's interest to take perf in this direction without a clear sense of the benefits it brings in comparison to the complexity it may add.
It's also a bit awkward I had already started to move ahead with this idea of upstreaming a non-perf based driver for the OA unit after asking Daniel Vetter about this on IRC. There are some knock on effects here too; Sourab Gupta is looking at building on this OA driver and has now started adapting his work for this non-perf approach.
Quoted 11 lines
Quoted 8 lines
- The current OA PMU driver breaks some significant design assumptions.
Existing perf pmus are used for profiling work on a cpu and we're introducing the idea of
_IS_DEVICEpmus with different security implications, the need to fake cpu-related data (such as user/kernel registers) to fit with perf's current design, and adding_DEVICErecords as a way to forward device-specific status records.There are more devices with counters on than GPUs, so I think it might make sense to look at extending perf to better deal with this.
I wonder if it could be good to look at exposing some of the mmio accessible Gen graphics counters before tackling a more complex case like the OA unit. We have a number of counters that could be interesting to sample periodically via a hrtimer, that require no configuration, are global (so no need to specify a gpu context) but as they relate to the GPU an _IS_DEVICE pmu would still be appropriate. Some of these seem like they could be better suited to being exposed via perf than OA unit counters so they might be a helpful stepping stone.
Quoted 21 lines
Quoted 4 lines
The OA unit writes reports of counters into a circular buffer, without involvement from the CPU, making our PMU driver the first of a kind.
Agreed, this is somewhat 'odd' from where we are today.
Quoted 6 lines
Perf supports groups of counters and allows those to be read via transactions internally but transactions currently seem designed to be explicitly initiated from the cpu (say in response to a userspace
read()) and while we could pull a report out of the OA buffer we can't trigger a report from the cpu on demand.Quoted 7 lines
Related to being report based; the OA counters are configured in HW as a set while perf generally expects counter configurations to be orthogonal. Although counters can be associated with a group leader as they are opened, there's no clear precedent for being able to provide group-wide configuration attributes and no obvious solution as yet that's expected to be acceptable to upstream and meets our userspace needs.
Quoted 2 lines
I'm not entirely sure what you mean with group-wide configuration attributes; could you elaborate?
Here I'm thinking of configuration details that conceptually relate to a set of OA unit counters, not individual events/counters:
- The choice of 'metric set' which represents a MUX configuration + boolean logic configuration for a set of counters that will be included in the reports written by the OA unit.
- The OA unit exponent for periodic sampling applies to the whole group.
- The choice of report layout which the OA unit writes all the counters in.
- The choice to profile a single context or system-wide applies to the group, as well as the specification of a file descriptor + context ID in the single-context case.
Quoted 11 lines
Quoted 7 lines
We currently avoid using perf's grouping feature
and forward OA reports to userspace via perf's 'raw' sample field. This suits our userspace well considering how coupled the counters are when dealing with normalizing. It would be inconvenient to split counters up into separate events, only to require userspace to recombine them.So IF you were using a group, a single read from the leader can return you a vector of all values (
PERF_FORMAT_GROUP), this avoids having to do that recombine.
Although recombining isn't necessary with _FORMAT_GROUP, this vector layout could be similarly inconvenient for userspace...
afict we couldn't avoid also requesting the _ID to be included in the vector, so instead of the 32/40bits per counter we would now have 16bytes per counter which would seem to loose some benefit from a compact HW layout to minimize our use of memory bandwidth.
Userspace has to be aware that we're placing 32bit or 40bit values within a 64bit field and the values will overflow as such.
As userspace receives reports it calculates deltas between sequential reports to accumulate 64bit counters. For each event in the vector it needs to lookup the index of the accumulated value, and a lookup based on a u64 event id for each counter isn't as direct as with a rigid report layout. (I don't know what assumptions can be made about the vector ordering from one sample to the next, but maybe a fixed mapping could be made after the first sample) When calculating the delta userspace needs to be careful not to treat the vector values as 64bit but as 32/40bit to account for overflow.
Mesa will still have to be able to handle the raw OA report layouts to process reports collected via the command stream using MI_REPORT_PERF_COUNT commands, so any alternative at least represents some amount of extra layout handling code and needing to handle different combinations of raw or vector layouts when calculating a single delta.
Quoted 4 lines
Another option would be to view the arrival of an OA vector in the datastream as an 'event' and generate a
PERF_RECORD_READin the perf buffer (which again can use the GROUP vector format).
Tbh I couldn't really figure out what PERF_RECORD_READ is intended for
- the userspace code I found currently ignores these.
The extensible sample design seems more appropriate, but it could be good if samples were extensible by device drivers as an alternative to packing data into the raw field. Sourab who's been building on my base OA driver is exposing more data as part of samples, including a context ID, and we were extending what we included in the raw field (we wouldn't get this extra data via _RECORD_READ).
Conceptually the aim was to expose event specific sample flags, to extend samples (mirroring the existing design for pre-defined sample flags, except driver extensible). We just used the raw field as the most convenient extension point to start with.
Extending the raw field has some difficulties though because events/core currently only lets a pmu give a single raw data pointer plus len which will be copied into the ring buffer, so to include more than the OA report we'd have to copy the report into an intermediate larger buffer. I'd been considering allowing a vector of data+len values to be specified for copying the raw data.
Quoted 9 lines
Quoted 5 lines
Related to counter orthogonality; we can't time share the OA unit, while event scheduling is a central design idea within perf for allowing userspace to open + enable more events than can be configured in HW at any one time.
So we have other PMUs that cannot do this; Gen OA would not be unique in this. Intel PT for example only allows a single active event.
Quoted 7 lines
That said; earlier today I saw:
https://www.youtube.com/watch?v=9J3BQcAeHpI&list=PLe6I3NKr-I4J2oLGXhGOeBMEjh8h10jT3&index=7
where exactly this feature was mentioned as not fitting well into the existing GPU performance interfaces (
GL_AMD_performance_monitor/GL_INTEL_performance_query).
I think Samuel was generally commenting that these Intel/AMD GL extensions aren't a good fit for Nvidia hw since it's awkward to abstract the need to update the MUX configuration in-flight so they can derive higher level counters from more inputs than the HW can expose at the same time. Unlike Intel and AMD, it looks like instead of defining a GL extension for accessing performance counters Nvidia has a perfkit api which can be used in conjuction with OpenGL, CUDA or Direct3D which Samuel has also been implementing.
I'm not sure what exactly about the AMD/Intel GL extensions makes it tricky to abstract round-robin updates of the MUXs internally vs what perfkit offers but since we wouldn't expect to round-robin the OA counter configuration the INTEL_performance_query spec authors wouldn't have given that any consideration.
Quoted 5 lines
So there is hardware (Nvidia) out there that does support this. Also mentioned was that this hardware has global and local counters, where the local ones are specific to a rendering context. That is not unlike the per-cpu / per-task stuff perf does.
For reference, OA counters can be viewed as global or local counters. When opening an event an application can request a system-wide view vs single-context view.
Quoted 11 lines
Quoted 10 lines
The OA unit is not designed to
allow re-configuration while in use. We can't reconfigure the OA unit without loosing internal OA unit state which we can't access explicitly to save and restore. Reconfiguring the OA unit is also relatively slow, involving ~100 register writes. From userspace Mesa also depends on a stable OA configuration when emittingMI_REPORT_PERF_COUNTcommands and importantly the OA unit can't be disabled while there are outstandingMI_RPCcommands lest we hang the command streamer.
Quoted 1 line
Right; see the
PERF_PMU_CAP_EXCLUSIVEstuff.
This looks like it's incompatible with the group mechanism re: the other question of exposing OA reports via the grouping mechanims vs raw sample data.
Quoted 9 lines
Quoted 8 lines
- We may be making some technical compromises a.t.m for the sake of using perf.
perf_event_open()requires events to either relate to a pid or a specific cpu core, while our device pmu relates to neither. Events opened with a pid will be automatically enabled/disabled according to the scheduling of that process - so not appropriate for us.
Quoted 4 lines
Right; the traditional cpu/pid mapping doesn't work well for devices; but maybe, with some work, we can create something like that global/local render context from it; although I've no clue what form that would need at this time.
Currently the way we identify a context is with the combination of a file descriptor and a u32 context handle (only unique to that fd). The use of a drm file descriptor here also relates to the security model for accessing single context metrics, in that we don't require root privileges to profile a context associated with a file descriptor that the process has open.
Recently we've also been working with a globally unique context ID (not upstream yet) and it could be interesting to also allow single context profiling given a global ID, without any fd (comparable to passing a pid to perf_event_open). This form of selection would probably require root privileges by default.
Quoted 7 lines
Quoted 6 lines
When
an event is related to a cpu id, perf ensures pmu methods will be invoked via an inter process interrupt on that core. To avoid invasive changes our userspace opens OA perf events for a specific cpu.
Quoted 6 lines
Some of that might still make sense in the sense that GPUs are subject to the NUMA topology of machines. I would think you would want most such things to be done on the node the device is attached to.
Granted, this might not be a concern for Intel graphics, but it might be relevant for some of the discrete GPUs.
I'm not sure whether Nouveau takes this kind of idea into consideration or not.
Quoted 29 lines
Quoted 21 lines
- I'm not confident our use case benefits much from building on perf:
We aren't using existing perf based tooling with our PMU. Existing tools typically assume you're profiling work running on a cpu, e.g. expecting samples to be associated with instruction pointers and user/kernel registers and aiming to represent metrics in relation to application source code. We're forwarding fake register values and userspace needs needs to know how to decode the raw OA reports before anything can be reported to a user.
With the buffering done by the OA unit I don't think we currently benefit from perf's mmapped circular buffer interface. We already have a decoupled producer and consumer and since we have to copy out of the OA buffer, it would work well for us to hide that copy in a simpler
read()based interface.- Logistically it might be more practical to contain this to the graphics stack.
Quoted 7 lines
It seems fair to consider that if we can't see a very compelling benefit to building on perf, then containing this work to
drivers/gpu/drm/i915may simplify the review process as well as future maintenance and development.Peter; I wonder if you would tend to agree too that it could make sense for us to go with our own interface here?
Quoted 7 lines
Sorry this took so long; this wanted a well considered response and those tend to get delayed in light of 'urgent' stuff.
While I can certainly see the pain points and why you would rather not deal with them. I think it would make Linux a better place if we could manage to come up with a generic interface that would work for 'all' GPUs (and possibly more devices).
I'm not sure a generic multi-vendor interface for gpu metrics is what's at stake or really needed here. The perf driver I worked on didn't attempt to provide a generic interface: E.g. counter normalizing is relatively complex and HW specific but makes sense to leave to userspace. Normalizing may combine periodic OA reports (from perf) with MI_REPORT_PERF_COUNT reports got via a command stream (not involving perf). Normalizing might also be done offline or remotely for a reduced runtime overhead).
Although it could be good to work more closely with Samuel on this, I did raise the idea of using perf with him last year but I think his priority is still with accessing metrics synchronized with the command stream. We were looking at a perf-like interface to help support the periodic sampling feature of the OA unit, but I'm not aware that Nvidia HW has the same kind of feature.
I think it's more the norm that GPU drivers don't share a lot of kernel interfaces given the significant architectural differences between vendors. A large proportion of a GPU driver is typically in userspace and interface standardisation is done at the OpenGL/CL level more so than with kernel interfaces. While perf may be adaptable to help access some gpu/device metrics it's not in a good position to be involved with capturing metrics via command streams in sync with specific commands submitted in userspace so I wouldn't expect to aim for perf to be the sole interface for gpu metrics.
Okey, sorry to still be inclined to prioritize the non-perf interface at this stage, at least for OA unit metrics. That said; I think it may still be practical to look at exposing other mmio accessible counters of Gen graphics via perf and maybe these could serve as a stepping stone before attempting supporting the OA unit via perf.
I'm still open to trying to help with adapting perf in this direction if you feel it could be worthwhile, but would like to decouple the effort for now.
Regards,
- Robert
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx