drm/xe/guc: Improve quality and robustness of GuC log dumping
Subsystems (3)
v4 cover drm/xe/guc: Improve quality and robustness of GuC log dumping
From: John Harrison <[email protected]>
drm/xe/guc: Improve GuC log dumping and add dump on CT failures
There is a debug mechanism for dumping the GuC log as an ASCII hex stream via dmesg. This is extremely useful for situations where it is not possibe to query the log from debugfs (self tests, bugs that cause the driver to fail to load, system hangs, etc.). However, dumping via dmesg is not the most reliable. The dmesg buffer is limited in size, can be rate limited and a simple hex stream is hard to parse by tools.
So add extra information to the dump to make it more robust and parsable. This includes adding start and end tags to delimit the dump, using longer lines to reduce the per line overhead, adding a rolling count to check for missing lines and interleaved concurrent dumps and adding other important information such as the GuC version number and timestamp offset.
There are various internal error states that the CTB code can check for. These should never happen but when they do (driver bug, firmware bug or even hardware bug), they can be a nightmare to debug. So add in a capture of the GuC log and CT state at the point of error and subsequent dump from a worker thread.
Note that the ultimate aim is to add the GuC log to the devcoredump capture. And then to provide a mechanism for generating a devcoredump at an arbitrary point (such as dead CTB or failed selftest) and dumping that to dmesg. Unfortunately, there are still issues with doing that. But this is a good first step along the way.
v2: Remove pm get/put as unnecessary (review feedback from Matthew B).
v3: Add firmware filename and 'wanted' version number.
v4: Use DRM level line printer wrapper from Michal W. Add 'dead CTB' dump support. Lots of restructuring of capture vs dump for both GuC log and CTB capture for both the dead CTB dump and for future inclusion in devcoredump.
John Harrison (6):
drm/xe/guc: Remove spurious line feed in debug print drm/xe/guc: Copy GuC log prior to dumping
drm/xe/guc: Use a two stage dump for GuC logs and add more info drm/xe/guc: Add a helper function for dumping GuC log to dmesg drm/xe/guc: Dead CT helper
drm/xe/guc: Dump entire CTB on errors
Michal Wajdeczko (1):
drm/print: Introduce drm_line_printer
drivers/gpu/drm/drm_print.c | 14 + .../drm/xe/abi/guc_communication_ctb_abi.h | 1 + drivers/gpu/drm/xe/regs/xe_guc_regs.h | 1 + drivers/gpu/drm/xe/xe_devcoredump.c | 2 +- drivers/gpu/drm/xe/xe_guc_ct.c | 355 ++++++++++++++---- drivers/gpu/drm/xe/xe_guc_ct.h | 9 +- drivers/gpu/drm/xe/xe_guc_ct_types.h | 24 ++ drivers/gpu/drm/xe/xe_guc_debugfs.c | 2 +- drivers/gpu/drm/xe/xe_guc_log.c | 227 ++++++++++- drivers/gpu/drm/xe/xe_guc_log.h | 11 +- drivers/gpu/drm/xe/xe_guc_log_types.h | 29 ++ include/drm/drm_print.h | 64 ++++ 12 files changed, 636 insertions(+), 103 deletions(-)
Signature · 1 line
2.43.2
PATCH v4 1/7 drm/xe/guc: Remove spurious line feed in debug print
From: John Harrison <[email protected]>
Including line feeds at the start of a debug print messes up the output when sent to dmesg. The break actually appears between all the usefu prefix information and the actual string being printed. In this case, each block of data has a very clear start line and an extra delimeter is really not necessary. So don't do it.
--- drivers/gpu/drm/xe/xe_guc_ct.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/xe/xe_guc_ct.c +1 -1
@@ -1439,7 +1439,7 @@ void xe_guc_ct_snapshot_print(struct xe_guc_ct_snapshot *snapshot,drm_puts (p,"H2G CTB (all sizes in DW):\n" );guc_ctb_snapshot_print (&snapshot->h2g, p); -drm_puts (p,"\nG2H CTB (all sizes in DW):\n" ); +drm_puts (p,"G2H CTB (all sizes in DW):\n" );guc_ctb_snapshot_print (&snapshot->g2h, p); drm_printf(p, "\tg2h outstanding: %d\n",
Signature · 1 line
2.43.2
PATCH v4 2/7 drm/xe/guc: Copy GuC log prior to dumping
In reply to [email protected]From: John Harrison <[email protected]>
Refactor the hexdump code into a separate function ready to be used for dumps of other objects. Also change to dumping a host memory copy rather than the live GPU buffer object. Doing so helps prevent inconsistencies due to the log being updated as it is being dumped. It also paves the way for decoupling the save from the print to allow inclusion in error reports such as the devcoredump.
Switch to use the dedicated kernel hexdump helper rather than printf. The helper makes it easier to print out much wider lines which can dramatically reduce the total line count of the dump (useful when dumping to dmesg).
Another issue with dumping such a large buffer is that it can be slow, especially if dumping to dmesg over a serial port. So add a yield to prevent the 'task has been stuck for 120s' kernel hang check feature from firing.
--- drivers/gpu/drm/xe/xe_guc_debugfs.c | 2 +- drivers/gpu/drm/xe/xe_guc_log.c | 119 ++++++++++++++++++++++++---- drivers/gpu/drm/xe/xe_guc_log.h | 2 +- 3 files changed, 106 insertions(+), 17 deletions(-)
drivers/gpu/drm/xe/xe_guc_debugfs.c +1 -1
@@ -41,7 +41,7 @@ static int guc_log(struct seq_file *m, void *data)struct drm_printer p =drm_seq_file_printer (m);xe_pm_runtime_get (xe); -xe_guc_log_print (&guc->log, &p); +xe_guc_log_print (&guc->log, &p, false);xe_pm_runtime_put (xe);return 0 ;
drivers/gpu/drm/xe/xe_guc_log.c +104 -15
142 lines not on this page · raw message
drivers/gpu/drm/xe/xe_guc_log.h +1 -1
@@ -37,7 +37,7 @@ struct drm_printer;#define GUC_LOG_LEVEL_MAX GUC_VERBOSITY_TO_LOG_LEVEL(GUC_LOG_VERBOSITY_MAX)int xe_guc_log_init (struct xe_guc_log *log); -void xe_guc_log_print (struct xe_guc_log *log,struct drm_printer *p); +void xe_guc_log_print (struct xe_guc_log *log,struct drm_printer *p,bool atomic);static inline u32 xe_guc_log_get_level (struct xe_guc_log *log)
Signature · 1 line
2.43.2
PATCH v4 3/7 drm/xe/guc: Use a two stage dump for GuC logs and add more info
In reply to [email protected]From: John Harrison <[email protected]>
Split the GuC log dump into a two stage snapshot and print mechanism. This allows the log to be captured at the point of an error (which may be in a resticted context) and then dump it out later (from a regular context such as a worker function or a sysfs file handler).
Also add a bunch of other useful pieces of information that can help (or are fundamentally required!) to decode and parse the log.
--- drivers/gpu/drm/xe/regs/xe_guc_regs.h | 1 + drivers/gpu/drm/xe/xe_guc_log.c | 150 ++++++++++++++++++++------ drivers/gpu/drm/xe/xe_guc_log.h | 6 ++ drivers/gpu/drm/xe/xe_guc_log_types.h | 29 +++++ 4 files changed, 155 insertions(+), 31 deletions(-)
drivers/gpu/drm/xe/regs/xe_guc_regs.h +1 -0
@@ -84,6 +84,7 @@#define HUC_LOADING_AGENT_GUC REG_BIT(1)#define GUC_WOPCM_OFFSET_VALID REG_BIT(0)#define GUC_MAX_IDLE_COUNT XE_REG(0xc3e4) +#define GUC_PMTIMESTAMP XE_REG(0xc3e8)#define GUC_SEND_INTERRUPT XE_REG(0xc4c8)#define GUC_SEND_TRIGGER REG_BIT(0)
drivers/gpu/drm/xe/xe_guc_log.c +119 -31
193 lines not on this page · raw message
drivers/gpu/drm/xe/xe_guc_log.h +6 -0
@@ -9,6 +9,7 @@#include "xe_guc_log_types.h" struct drm_printer ; +struct xe_device ;#if IS_ENABLED (CONFIG_DRM_XE_LARGE_GUC_BUFFER )#define CRASH_BUFFER_SIZE SZ_1M
@@ -38,6 +39,11 @@ struct drm_printer;int xe_guc_log_init (struct xe_guc_log *log);void xe_guc_log_print (struct xe_guc_log *log,struct drm_printer *p,bool atomic); +struct xe_guc_log_snapshot *xe_guc_log_snapshot_alloc (struct xe_guc_log *log,bool atomic); +struct xe_guc_log_snapshot *xe_guc_log_snapshot_capture (struct xe_guc_log *log,bool atomic); +void xe_guc_log_snapshot_print (struct xe_device *xe,struct xe_guc_log_snapshot *snapshot, +struct drm_printer *p,bool atomic); +void xe_guc_log_snapshot_free (struct xe_guc_log_snapshot *snapshot);static inline u32 xe_guc_log_get_level (struct xe_guc_log *log)
drivers/gpu/drm/xe/xe_guc_log_types.h +29 -0
@@ -8,8 +8,37 @@#include <linux/types.h> +#include "xe_uc_fw_types.h" +struct xe_bo ; +/** +* struct xe_guc_log_snapshot: +* Capture of the GuC log plus various state useful for decoding the log +*/ +struct xe_guc_log_snapshot { +/** @size: Size in bytes of the @copy allocation */ +size_t size; +/** @copy: Host memory copy of the log buffer for later dumping, split into chunks */ +void **copy; +/** @num_chunks: Number of chunks withint @copy */ +int num_chunks; +/** @ktime: Kernel time the snapshot was taken */ +u64 ktime; +/** @stamp: GuC timestamp at which the snapshot was taken */ +u32 stamp; +/** @ref_clk: GuC timestamp frequency */ +u32 ref_clk; +/** @level: GuC log verbosity level */ +u32 level; +/** @ver_found: GuC firmware version */ +struct xe_uc_fw_version ver_found; +/** @ver_want: GuC firmware version that driver expected */ +struct xe_uc_fw_version ver_want; +/** @path: Path of GuC firmware blob */ +const char *path; +}; +/** * struct xe_guc_log - GuC log */
Signature · 1 line
2.43.2
PATCH v4 4/7 drm/print: Introduce drm_line_printer
In reply to [email protected]From: Michal Wajdeczko <[email protected]>
This drm printer wrapper can be used to increase the robustness of the captured output generated by any other drm_printer to make sure we didn't lost any intermediate lines of the output by adding line numbers to each output line. Helpful for capturing some crash data.
--- drivers/gpu/drm/drm_print.c | 14 ++++++++ include/drm/drm_print.h | 64 +++++++++++++++++++++++++++++++++++++ 2 files changed, 78 insertions(+)
drivers/gpu/drm/drm_print.c +14 -0
@@ -214,6 +214,20 @@ void __drm_printfn_err(struct drm_printer *p, struct va_format *vaf) }EXPORT_SYMBOL (__drm_printfn_err); +void __drm_printfn_line (struct drm_printer *p,struct va_format *vaf) +{ +unsigned int counter = ++p->line.counter; +const char *prefix = p->prefix ?:"" ; +const char *pad = p->prefix ?" " :"" ; + +if (p->line.series) +drm_printf (p->arg,"%s%s%u.%u: %pV" , + prefix, pad, p->line.series, counter, vaf); +else +drm_printf (p->arg,"%s%s%u: %pV" , prefix, pad, counter, vaf); +} +EXPORT_SYMBOL (__drm_printfn_line); +/** * drm_puts - print a const string to a &drm_printer stream * @p: the &drm printer
include/drm/drm_print.h +64 -0
82 lines not on this page · raw message
Signature · 1 line
2.43.2
PATCH v4 5/7 drm/xe/guc: Add a helper function for dumping GuC log to dmesg
In reply to [email protected]From: John Harrison <[email protected]>
Create a helper function that can be used to dump the GuC log to dmesg in a manner that is reliable for extraction and decode. The intention is that calls to this can be added by developers when debugging specific issues that require a GuC log but do not allow easy capture of the log - e.g. failures in selftests and failues that lead to kernel hangs.
--- drivers/gpu/drm/xe/xe_guc_log.c | 14 ++++++++++++++ drivers/gpu/drm/xe/xe_guc_log.h | 1 + 2 files changed, 15 insertions(+)
drivers/gpu/drm/xe/xe_guc_log.c +14 -0
@@ -245,6 +245,20 @@ void xe_guc_log_snapshot_print(struct xe_device *xe, struct xe_guc_log_snapshot } } +void xe_guc_log_print_dmesg (struct xe_guc_log *log,bool atomic) +{ +struct xe_gt *gt =log_to_gt (log); +static int g_count; +struct drm_printer ip =xe_gt_info_printer (gt); +struct drm_printer lp =drm_line_printer (&ip,"Capture" , ++g_count); + +drm_printf (&lp,"Dumping GuC log for %ps...\n" ,__builtin_return_address (0 )); + +xe_guc_log_print (log, &lp, atomic); + +drm_printf (&lp,"Done.\n" ); +} +void xe_guc_log_print (struct xe_guc_log *log,struct drm_printer *p,bool atomic) {struct xe_guc_log_snapshot *snapshot;
drivers/gpu/drm/xe/xe_guc_log.h +1 -0
@@ -39,6 +39,7 @@ struct xe_device;int xe_guc_log_init (struct xe_guc_log *log);void xe_guc_log_print (struct xe_guc_log *log,struct drm_printer *p,bool atomic); +void xe_guc_log_print_dmesg (struct xe_guc_log *log,bool atomic);struct xe_guc_log_snapshot *xe_guc_log_snapshot_alloc (struct xe_guc_log *log,bool atomic);struct xe_guc_log_snapshot *xe_guc_log_snapshot_capture (struct xe_guc_log *log,bool atomic); void xe_guc_log_snapshot_print(struct xe_device *xe, struct xe_guc_log_snapshot *snapshot,
Signature · 1 line
2.43.2
PATCH v4 6/7 drm/xe/guc: Dead CT helper
In reply to [email protected]From: John Harrison <[email protected]>
Add a worker function helper for asynchronously dumping state when an internal/fatal error is detected in CT processing. Being asynchronous is required to avoid deadlocks and scheduling-while-atomic or process-stalled-for-too-long issues. Also check for a bunch more error conditions and improve the handling of some existing checks.
--- .../drm/xe/abi/guc_communication_ctb_abi.h | 1 + drivers/gpu/drm/xe/xe_guc_ct.c | 256 ++++++++++++++++-- drivers/gpu/drm/xe/xe_guc_ct_types.h | 22 ++ 3 files changed, 258 insertions(+), 21 deletions(-)
drivers/gpu/drm/xe/abi/guc_communication_ctb_abi.h +1 -0
@@ -52,6 +52,7 @@ struct guc_ct_buffer_desc {#define GUC_CTB_STATUS_OVERFLOW (1 << 0)#define GUC_CTB_STATUS_UNDERFLOW (1 << 1)#define GUC_CTB_STATUS_MISMATCH (1 << 2) +#define GUC_CTB_STATUS_DISABLED (1 << 3)u32 reserved[13 ]; } __packed;static_assert (sizeof (struct guc_ct_buffer_desc ) ==64 );
drivers/gpu/drm/xe/xe_guc_ct.c +235 -21
418 lines not on this page · raw message
drivers/gpu/drm/xe/xe_guc_ct_types.h +22 -0
@@ -86,6 +86,24 @@ enum xe_guc_ct_state { XE_GUC_CT_STATE_ENABLED, }; +#if IS_ENABLED (CONFIG_DRM_XE_DEBUG ) +/** struct xe_dead_ct - Information for debugging a dead CT */ +struct xe_dead_ct { +/** @lock: protects memory allocation/free operations, and @reason updates */ +spinlock_t lock; +/** @reason: bit mask of CT_DEAD_* reason codes */ +int reason; +/** @reported: for preventing multiple dumps per error sequence */ +bool reported; +/** @worker: worker thread to get out of interrupt context before dumping */ +struct work_struct worker; +/** snapshot_ct: copy of CT state and CTB content at point of error */ +struct xe_guc_ct_snapshot *snapshot_ct; +/** snapshot_log: copy of GuC log at point of error */ +struct xe_guc_log_snapshot *snapshot_log; +}; +#endif +/** * struct xe_guc_ct - GuC command transport (CT) layer *
@@ -126,6 +144,10 @@ struct xe_guc_ct {u32 msg[GUC_CTB_MSG_MAX_LEN ];/** @fast_msg: Message buffer */ u32 fast_msg[GUC_CTB_MSG_MAX_LEN ]; + +#if IS_ENABLED (CONFIG_DRM_XE_DEBUG ) +struct xe_dead_ct dead; +#endif };#endif
Signature · 1 line
2.43.2
PATCH v4 7/7 drm/xe/guc: Dump entire CTB on errors
In reply to [email protected]From: John Harrison <[email protected]>
The dump of the CT buffers was only showing the unprocessed data which is not generally useful for saying why a hang occurred - because it was probably caused by the commands that were just processed. So save and dump the entire buffer but in a more compact dump format. Also zero fill it on allocation to avoid confusion over uninitialised data in the dump.
--- drivers/gpu/drm/xe/xe_devcoredump.c | 2 +- drivers/gpu/drm/xe/xe_guc_ct.c | 101 ++++++++++++--------------- drivers/gpu/drm/xe/xe_guc_ct.h | 9 +-- drivers/gpu/drm/xe/xe_guc_ct_types.h | 2 + drivers/gpu/drm/xe/xe_guc_log.c | 4 +- drivers/gpu/drm/xe/xe_guc_log.h | 2 + 6 files changed, 55 insertions(+), 65 deletions(-)
drivers/gpu/drm/xe/xe_devcoredump.c +1 -1
@@ -114,7 +114,7 @@ static ssize_t xe_devcoredump_read(char *buffer, loff_t offset,xe_device_snapshot_print (xe, &p);drm_printf (&p,"\n**** GuC CT ****\n" ); -xe_guc_ct_snapshot_print (coredump->snapshot.ct, &p); +xe_guc_ct_snapshot_print (xe, coredump->snapshot.ct, &p, false);xe_guc_exec_queue_snapshot_print (coredump->snapshot.ge, &p);drm_printf (&p,"\n**** Job ****\n" );
drivers/gpu/drm/xe/xe_guc_ct.c +43 -58
168 lines not on this page · raw message
drivers/gpu/drm/xe/xe_guc_ct.h +5 -4
@@ -9,6 +9,7 @@#include "xe_guc_ct_types.h" struct drm_printer ; +struct xe_device ;int xe_guc_ct_init (struct xe_guc_ct *ct);int xe_guc_ct_enable (struct xe_guc_ct *ct);
@@ -16,10 +17,10 @@ void xe_guc_ct_disable(struct xe_guc_ct *ct);void xe_guc_ct_stop (struct xe_guc_ct *ct);void xe_guc_ct_fast_path (struct xe_guc_ct *ct); -struct xe_guc_ct_snapshot * -xe_guc_ct_snapshot_capture (struct xe_guc_ct *ct,bool atomic); -void xe_guc_ct_snapshot_print (struct xe_guc_ct_snapshot *snapshot, -struct drm_printer *p); +struct xe_guc_ct_snapshot *xe_guc_ct_snapshot_alloc (struct xe_guc_ct *ct,bool atomic); +struct xe_guc_ct_snapshot *xe_guc_ct_snapshot_capture (struct xe_guc_ct *ct,bool atomic); +void xe_guc_ct_snapshot_print (struct xe_device *xe,struct xe_guc_ct_snapshot *snapshot, +struct drm_printer *p,bool atomic);void xe_guc_ct_snapshot_free (struct xe_guc_ct_snapshot *snapshot);void xe_guc_ct_print (struct xe_guc_ct *ct,struct drm_printer *p,bool atomic);
drivers/gpu/drm/xe/xe_guc_ct_types.h +2 -0
@@ -52,6 +52,8 @@ struct guc_ctb {struct guc_ctb_snapshot {/** @desc: snapshot of the CTB descriptor */ struct guc_ct_buffer_desc desc; +/** @cmds: size of the snapshot of the CTB commands */ +size_t size;/** @cmds: snapshot of the CTB commands */ u32 *cmds;/** @info: snapshot of the CTB info */
drivers/gpu/drm/xe/xe_guc_log.c +2 -2
@@ -64,8 +64,8 @@ static size_t guc_log_size(void)#define LINES_PER_READ 4#define WORDS_PER_READ (WORDS_PER_DUMP * DUMPS_PER_LINE * LINES_PER_READ) -static void xe_hexdump_blob (struct xe_device *xe,const void *blob,size_t size, -struct drm_printer *p,bool atomic) +void xe_hexdump_blob (struct xe_device *xe,const void *blob,size_t size, +struct drm_printer *p,bool atomic) {char line_buff[DUMPS_PER_LINE *WORDS_PER_DUMP *9 +1 ];const u32 *blob32 = (const u32 *)blob;
drivers/gpu/drm/xe/xe_guc_log.h +2 -0
@@ -45,6 +45,8 @@ struct xe_guc_log_snapshot *xe_guc_log_snapshot_capture(struct xe_guc_log *log,void xe_guc_log_snapshot_print (struct xe_device *xe,struct xe_guc_log_snapshot *snapshot,struct drm_printer *p,bool atomic);void xe_guc_log_snapshot_free (struct xe_guc_log_snapshot *snapshot); +void xe_hexdump_blob (struct xe_device *xe,const void *blob,size_t size, +struct drm_printer *p,bool atomic);static inline u32 xe_guc_log_get_level (struct xe_guc_log *log)
Signature · 1 line
2.43.2
Patchwork
CI✗ CI.Patch_applied: failure for drm/xe/guc: Improve quality and robustness of GuC log dumping
In reply to [email protected]== Series Details ==
== Series Details ==
Series: drm/xe/guc: Improve quality and robustness of GuC log dumping
URL : https://patchwork.freedesktop.org/series/134695/
State : failure
== Summary ==
=== Applying kernel patches on branch 'drm-tip' with base: ===
Base commit: bf9d891af55d drm-tip: 2024y-06m-10d-19h-36m-52s UTC integration manifest === git am output follows ===
error: patch failed: include/drm/drm_print.h:176
error: include/drm/drm_print.h: patch does not apply
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Applying: drm/xe/guc: Remove spurious line feed in debug print
Applying: drm/xe/guc: Copy GuC log prior to dumping
Applying: drm/xe/guc: Use a two stage dump for GuC logs and add more info
Applying: drm/print: Introduce drm_line_printer
Patch failed at 0004 drm/print: Introduce drm_line_printer
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".