Skip to contentSkip to conversation

scsi: ufs-debugfs: Add UFS Exception Event reporting

PATCH Applied Adrian Hunter 4 participants 10 messages Feb 9, 2021 – Mar 4, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover scsi: ufs-debugfs: Add UFS Exception Event reporting

Hi

Here are V2 patches to add a tracepoint for UFS Exception Events and to allow users to enable specific exception events without affecting the driver's use of exception events.

Changes in V2:

scsi: ufs: Add exception event tracepoint
Change status field from "exception event status" to "status"

scsi: ufs: Add exception event definitions
Add Reviewed-by: Bean Huo

Adrian Hunter (4):
scsi: ufs: Add exception event tracepoint
scsi: ufs: Add exception event definitions
scsi: ufs-debugfs: Add user-defined exception_event_mask
scsi: ufs-debugfs: Add user-defined exception event rate limiting

 drivers/scsi/ufs/ufs-debugfs.c | 90 ++++++++++++++++++++++++++++++++++++++++++
 drivers/scsi/ufs/ufs-debugfs.h |  2 +
 drivers/scsi/ufs/ufs.h         | 10 ++++-
 drivers/scsi/ufs/ufshcd.c      | 87 +++++++++++++++++++++++++---------------
 drivers/scsi/ufs/ufshcd.h      | 26 +++++++++++-
 include/trace/events/ufs.h     | 21 ++++++++++
 6 files changed, 201 insertions(+), 35 deletions(-)

Regards
Adrian

#1 lore raw

PATCH v2 1/4 scsi: ufs: Add exception event tracepoint

Currently, exception event status can be read from wExceptionEventStatus attribute (sysfs file attributes/exception_event_status under the UFS host controller device directory). Polling that attribute to track UFS exception events is impractical, so add a tracepoint to track exception events for testing and debugging purposes.

Note, by the time the exception event status is read, the exception event may have cleared, so the value can be zero - see example below.

Note also, only enabled exception events can be reported. A subsequent patch adds the ability for users to enable selected exception events via debugfs.

Example with driver instrumented to enable all exception events:

  # echo 1 > /sys/kernel/debug/tracing/events/ufs/ufshcd_exception_event/enable

... do some I/O ...

  # cat /sys/kernel/debug/tracing/trace
  # tracer: nop
  #
  # entries-in-buffer/entries-written: 3/3   #P:5
  #
  #                                _-----=> irqs-off
  #                               / _----=> need-resched
  #                              | / _---=> hardirq/softirq
  #                              || / _--=> preempt-depth
  #                              ||| /     delay
  #           TASK-PID     CPU#  ||||   TIMESTAMP  FUNCTION
  #              | |         |   ||||      |         |
       kworker/2:2-173     [002] ....   731.486419: ufshcd_exception_event: 0000:00:12.5: status 0x0
       kworker/2:2-173     [002] ....   732.608918: ufshcd_exception_event: 0000:00:12.5: status 0x4
       kworker/2:2-173     [002] ....   732.609312: ufshcd_exception_event: 0000:00:12.5: status 0x4
---
 drivers/scsi/ufs/ufshcd.c  |  2 ++
 include/trace/events/ufs.h | 21 +++++++++++++++++++++
 2 files changed, 23 insertions(+)
drivers/scsi/ufs/ufshcd.c +2 -0
index 721f55db181f..d6fdce655388 100644
@@ -5616,6 +5616,8 @@ static void ufshcd_exception_event_handler(struct work_struct *work)
 		goto out;
 	}
 
+	trace_ufshcd_exception_event(dev_name(hba->dev), status);
+
 	status &= hba->ee_ctrl_mask;
 
 	if (status & MASK_EE_URGENT_BKOPS)
include/trace/events/ufs.h +21 -0
index e151477d645c..1cb6f1afba0e 100644
@@ -349,6 +349,27 @@ TRACE_EVENT(ufshcd_upiu,
 	)
 );
 
+TRACE_EVENT(ufshcd_exception_event,
+
+	TP_PROTO(const char *dev_name, u16 status),
+
+	TP_ARGS(dev_name, status),
+
+	TP_STRUCT__entry(
+		__string(dev_name, dev_name)
+		__field(u16, status)
+	),
+
+	TP_fast_assign(
+		__assign_str(dev_name, dev_name);
+		__entry->status = status;
+	),
+
+	TP_printk("%s: status 0x%x",
+		__get_str(dev_name), __entry->status
+	)
+);
+
 #endif /* if !defined(_TRACE_UFS_H) || defined(TRACE_HEADER_MULTI_READ) */
 
 /* This part must be outside protection */
Signature · 1 line
2.17.1
Reviewed-by on v2 1/4
#2 lore raw

RE: [PATCH V2 1/4] scsi: ufs: Add exception event tracepoint

Quoted 40 lines

Currently, exception event status can be read from wExceptionEventStatus attribute (sysfs file attributes/exception_event_status under the UFS host controller device directory). Polling that attribute to track UFS exception events is impractical, so add a tracepoint to track exception events for testing and debugging purposes.

Note, by the time the exception event status is read, the exception event may have cleared, so the value can be zero - see example below.

Note also, only enabled exception events can be reported. A subsequent patch adds the ability for users to enable selected exception events via debugfs.

Example with driver instrumented to enable all exception events:

  # echo 1 >
/sys/kernel/debug/tracing/events/ufs/ufshcd_exception_event/enable

... do some I/O ...

  # cat /sys/kernel/debug/tracing/trace
  # tracer: nop
  #
  # entries-in-buffer/entries-written: 3/3   #P:5
  #
  #                                _-----=> irqs-off
  #                               / _----=> need-resched
  #                              | / _---=> hardirq/softirq
  #                              || / _--=> preempt-depth
  #                              ||| /     delay
  #           TASK-PID     CPU#  ||||   TIMESTAMP  FUNCTION
  #              | |         |   ||||      |         |
       kworker/2:2-173     [002] ....   731.486419: ufshcd_exception_event:
0000:00:12.5: status 0x0
       kworker/2:2-173     [002] ....   732.608918: ufshcd_exception_event:
0000:00:12.5: status 0x4
       kworker/2:2-173     [002] ....   732.609312: ufshcd_exception_event:
0000:00:12.5: status 0x4
Quoted 1 line
Reviewed-by on v2 1/4
#3 lore raw

Re: [PATCH V2 1/4] scsi: ufs: Add exception event tracepoint

In reply to Adrian Hunter
Quoted 19 lines · Adrian Hunter
  # cat /sys/kernel/debug/tracing/trace
  # tracer: nop
  #
  # entries-in-buffer/entries-written: 3/3   #P:5
  #
  #                                _-----=> irqs-off
  #                               / _----=> need-resched
  #                              | / _---=> hardirq/softirq
  #                              || / _--=> preempt-depth
  #                              ||| /     delay
  #           TASK-PID     CPU#  ||||   TIMESTAMP  FUNCTION
  #              | |         |   ||||      |         |
       kworker/2:2-173     [002] ....   731.486419:
ufshcd_exception_event: 0000:00:12.5: status 0x0
       kworker/2:2-173     [002] ....   732.608918:
ufshcd_exception_event: 0000:00:12.5: status 0x4
       kworker/2:2-173     [002] ....   732.609312:
ufshcd_exception_event: 0000:00:12.5: status 0x4
Quoted 1 line
#4 lore raw

PATCH v2 2/4 scsi: ufs: Add exception event definitions

In reply to Adrian Hunter

For readability and completeness, add exception event definitions.

---
 drivers/scsi/ufs/ufs.h | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
drivers/scsi/ufs/ufs.h +8 -2
index bf1897a72532..cb80b9670bfe 100644
@@ -348,8 +348,14 @@ enum power_desc_param_offset {
 
 /* Exception event mask values */
 enum {
-	MASK_EE_STATUS		= 0xFFFF,
-	MASK_EE_URGENT_BKOPS	= (1 << 2),
+	MASK_EE_STATUS			= 0xFFFF,
+	MASK_EE_DYNCAP_EVENT		= BIT(0),
+	MASK_EE_SYSPOOL_EVENT		= BIT(1),
+	MASK_EE_URGENT_BKOPS		= BIT(2),
+	MASK_EE_TOO_HIGH_TEMP		= BIT(3),
+	MASK_EE_TOO_LOW_TEMP		= BIT(4),
+	MASK_EE_WRITEBOOSTER_EVENT	= BIT(5),
+	MASK_EE_PERFORMANCE_THROTTLING	= BIT(6),
 };
 
 /* Background operation status */
Signature · 1 line
2.17.1
#5 lore raw

PATCH v2 3/4 scsi: ufs-debugfs: Add user-defined exception_event_mask

In reply to Adrian Hunter

Allow users to enable specific exception events via debugfs.

The bits enabled by the driver ee_drv_ctrl are separated from the bits enabled by the user ee_usr_ctrl. The control mask ee_mask_ctrl is the logical-or of those two. A mutex is needed to ensure that the masks match what was written to the device.

---
 drivers/scsi/ufs/ufs-debugfs.c | 46 ++++++++++++++++++
 drivers/scsi/ufs/ufshcd.c      | 86 +++++++++++++++++++++-------------
 drivers/scsi/ufs/ufshcd.h      | 22 ++++++++-
 3 files changed, 120 insertions(+), 34 deletions(-)
drivers/scsi/ufs/ufs-debugfs.c +46 -0
index dee98dc72d29..59729073b569 100644
@@ -44,10 +44,56 @@ static int ufs_debugfs_stats_show(struct seq_file *s, void *data)
 }
 DEFINE_SHOW_ATTRIBUTE(ufs_debugfs_stats);
 
+static int ee_usr_mask_get(void *data, u64 *val)
+{
+	struct ufs_hba *hba = data;
+
+	*val = hba->ee_usr_mask;
+	return 0;
+}
+
+static int ufs_debugfs_get_user_access(struct ufs_hba *hba)
+__acquires(&hba->host_sem)
+{
+	down(&hba->host_sem);
+	if (!ufshcd_is_user_access_allowed(hba)) {
+		up(&hba->host_sem);
+		return -EBUSY;
+	}
+	pm_runtime_get_sync(hba->dev);
+	return 0;
+}
+
+static void ufs_debugfs_put_user_access(struct ufs_hba *hba)
+__releases(&hba->host_sem)
+{
+	pm_runtime_put_sync(hba->dev);
+	up(&hba->host_sem);
+}
+
+static int ee_usr_mask_set(void *data, u64 val)
+{
+	struct ufs_hba *hba = data;
+	int err;
+
+	if (val & ~(u64)MASK_EE_STATUS)
+		return -EINVAL;
+	err = ufs_debugfs_get_user_access(hba);
+	if (err)
+		return err;
+	err = ufshcd_update_ee_usr_mask(hba, val, MASK_EE_STATUS);
+	ufs_debugfs_put_user_access(hba);
+	return err;
+}
+
+DEFINE_DEBUGFS_ATTRIBUTE(ee_usr_mask_fops, ee_usr_mask_get, ee_usr_mask_set, "%#llx\n");
+
 void ufs_debugfs_hba_init(struct ufs_hba *hba)
 {
 	hba->debugfs_root = debugfs_create_dir(dev_name(hba->dev), ufs_debugfs_root);
 	debugfs_create_file("stats", 0400, hba->debugfs_root, hba, &ufs_debugfs_stats_fops);
+	debugfs_create_file("exception_event_mask", 0600, hba->debugfs_root,
+			    hba, &ee_usr_mask_fops);
 }
 
 void ufs_debugfs_hba_exit(struct ufs_hba *hba)
drivers/scsi/ufs/ufshcd.c +53 -33

130 lines not on this page · raw message

drivers/scsi/ufs/ufshcd.h +21 -1
index ee61f821f75d..9b7413f35def 100644
@@ -773,7 +773,10 @@ struct ufs_hba {
 	u32 ufshcd_state;
 	u32 eh_flags;
 	u32 intr_mask;
-	u16 ee_ctrl_mask;
+	u16 ee_ctrl_mask; /* Exception event mask */
+	u16 ee_drv_mask;  /* Exception event mask for driver */
+	u16 ee_usr_mask;  /* Exception event mask for user (via debugfs) */
+	struct mutex ee_ctrl_mutex;
 	bool is_powered;
 	bool shutting_down;
 	struct semaphore host_sem;
@@ -1285,4 +1288,21 @@ static inline u8 ufshcd_scsi_to_upiu_lun(unsigned int scsi_lun)
 int ufshcd_dump_regs(struct ufs_hba *hba, size_t offset, size_t len,
 		     const char *prefix);
 
+int ufshcd_update_ee_control(struct ufs_hba *hba, u16 *mask, u16 *other_mask,
+			     u16 set, u16 clr);
+
+static inline int ufshcd_update_ee_drv_mask(struct ufs_hba *hba,
+					    u16 set, u16 clr)
+{
+	return ufshcd_update_ee_control(hba, &hba->ee_drv_mask,
+					&hba->ee_usr_mask, set, clr);
+}
+
+static inline int ufshcd_update_ee_usr_mask(struct ufs_hba *hba,
+					    u16 set, u16 clr)
+{
+	return ufshcd_update_ee_control(hba, &hba->ee_usr_mask,
+					&hba->ee_drv_mask, set, clr);
+}
+
 #endif /* End of Header */
Signature · 1 line
2.17.1
Acked-by on v2 3/4
#6 lore raw

Re: [PATCH V2 3/4] scsi: ufs-debugfs: Add user-defined exception_event_mask

Quoted 10 lines · Adrian Hunter

Allow users to enable specific exception events via debugfs.

The bits enabled by the driver ee_drv_ctrl are separated from the bits
enabled by the user ee_usr_ctrl. The control mask ee_mask_ctrl is the logical-or of those two. A mutex is needed to ensure that the masks match
what was written to the device.

Hi Adrian,
I tested this series patch on my platform, and you can add:

Bean

#7 lore raw

PATCH v2 4/4 scsi: ufs-debugfs: Add user-defined exception event rate limiting

In reply to Adrian Hunter

An enabled user-specified exception event that does not clear quickly will repeatedly cause the handler to run. That could unduly disturb the driver behaviour being tested or debugged. To prevent that add debugfs file exception_event_rate_limit_ms. When a exception event happens, it is disabled, and then after a period of time (default 20ms) the exception event is enabled again.

Note that if the driver also has that exception event enabled, it will not be disabled.

---
 drivers/scsi/ufs/ufs-debugfs.c | 44 ++++++++++++++++++++++++++++++++++
 drivers/scsi/ufs/ufs-debugfs.h |  2 ++
 drivers/scsi/ufs/ufshcd.c      |  5 ++--
 drivers/scsi/ufs/ufshcd.h      |  4 ++++
 4 files changed, 53 insertions(+), 2 deletions(-)
drivers/scsi/ufs/ufs-debugfs.c +44 -0
index 59729073b569..ced9ef4d7c78 100644
@@ -88,15 +88,59 @@ static int ee_usr_mask_set(void *data, u64 val)
 
 DEFINE_DEBUGFS_ATTRIBUTE(ee_usr_mask_fops, ee_usr_mask_get, ee_usr_mask_set, "%#llx\n");
 
+void ufs_debugfs_exception_event(struct ufs_hba *hba, u16 status)
+{
+	bool chgd = false;
+	u16 ee_ctrl_mask;
+	int err = 0;
+
+	if (!hba->debugfs_ee_rate_limit_ms || !status)
+		return;
+
+	mutex_lock(&hba->ee_ctrl_mutex);
+	ee_ctrl_mask = hba->ee_drv_mask | (hba->ee_usr_mask & ~status);
+	chgd = ee_ctrl_mask != hba->ee_ctrl_mask;
+	if (chgd) {
+		err = __ufshcd_write_ee_control(hba, ee_ctrl_mask);
+		if (err)
+			dev_err(hba->dev, "%s: failed to write ee control %d\n",
+				__func__, err);
+	}
+	mutex_unlock(&hba->ee_ctrl_mutex);
+
+	if (chgd && !err) {
+		unsigned long delay = msecs_to_jiffies(hba->debugfs_ee_rate_limit_ms);
+
+		queue_delayed_work(system_freezable_wq, &hba->debugfs_ee_work, delay);
+	}
+}
+
+static void ufs_debugfs_restart_ee(struct work_struct *work)
+{
+	struct ufs_hba *hba = container_of(work, struct ufs_hba, debugfs_ee_work.work);
+
+	if (!hba->ee_usr_mask || pm_runtime_suspended(hba->dev) ||
+	    ufs_debugfs_get_user_access(hba))
+		return;
+	ufshcd_write_ee_control(hba);
+	ufs_debugfs_put_user_access(hba);
+}
+
 void ufs_debugfs_hba_init(struct ufs_hba *hba)
 {
+	/* Set default exception event rate limit period to 20ms */
+	hba->debugfs_ee_rate_limit_ms = 20;
+	INIT_DELAYED_WORK(&hba->debugfs_ee_work, ufs_debugfs_restart_ee);
 	hba->debugfs_root = debugfs_create_dir(dev_name(hba->dev), ufs_debugfs_root);
 	debugfs_create_file("stats", 0400, hba->debugfs_root, hba, &ufs_debugfs_stats_fops);
 	debugfs_create_file("exception_event_mask", 0600, hba->debugfs_root,
 			    hba, &ee_usr_mask_fops);
+	debugfs_create_u32("exception_event_rate_limit_ms", 0600, hba->debugfs_root,
+			   &hba->debugfs_ee_rate_limit_ms);
 }
 
 void ufs_debugfs_hba_exit(struct ufs_hba *hba)
 {
 	debugfs_remove_recursive(hba->debugfs_root);
+	cancel_delayed_work_sync(&hba->debugfs_ee_work);
 }
drivers/scsi/ufs/ufs-debugfs.h +2 -0
index f35b39c4b4f5..3ca29d30460a 100644
@@ -12,11 +12,13 @@ void __init ufs_debugfs_init(void);
 void __exit ufs_debugfs_exit(void);
 void ufs_debugfs_hba_init(struct ufs_hba *hba);
 void ufs_debugfs_hba_exit(struct ufs_hba *hba);
+void ufs_debugfs_exception_event(struct ufs_hba *hba, u16 status);
 #else
 static inline void ufs_debugfs_init(void) {}
 static inline void ufs_debugfs_exit(void) {}
 static inline void ufs_debugfs_hba_init(struct ufs_hba *hba) {}
 static inline void ufs_debugfs_hba_exit(struct ufs_hba *hba) {}
+static inline void ufs_debugfs_exception_event(struct ufs_hba *hba, u16 status) {}
 #endif
 
 #endif
drivers/scsi/ufs/ufshcd.c +3 -2
index 065a662e7886..42d9a96a5721 100644
@@ -5160,14 +5160,14 @@ static irqreturn_t ufshcd_transfer_req_compl(struct ufs_hba *hba)
 	}
 }
 
-static int __ufshcd_write_ee_control(struct ufs_hba *hba, u32 ee_ctrl_mask)
+int __ufshcd_write_ee_control(struct ufs_hba *hba, u32 ee_ctrl_mask)
 {
 	return ufshcd_query_attr_retry(hba, UPIU_QUERY_OPCODE_WRITE_ATTR,
 				       QUERY_ATTR_IDN_EE_CONTROL, 0, 0,
 				       &ee_ctrl_mask);
 }
 
-static int ufshcd_write_ee_control(struct ufs_hba *hba)
+int ufshcd_write_ee_control(struct ufs_hba *hba)
 {
 	int err;
 
@@ -5635,6 +5635,7 @@ static void ufshcd_exception_event_handler(struct work_struct *work)
 	if (status & hba->ee_drv_mask & MASK_EE_URGENT_BKOPS)
 		ufshcd_bkops_exception_event_handler(hba);
 
+	ufs_debugfs_exception_event(hba, status);
 out:
 	ufshcd_scsi_unblock_requests(hba);
 	/*
drivers/scsi/ufs/ufshcd.h +4 -0
index 9b7413f35def..8a52b63ad4e6 100644
@@ -843,6 +843,8 @@ struct ufs_hba {
 #endif
 #ifdef CONFIG_DEBUG_FS
 	struct dentry *debugfs_root;
+	struct delayed_work debugfs_ee_work;
+	u32 debugfs_ee_rate_limit_ms;
 #endif
 };
 
@@ -1288,6 +1290,8 @@ static inline u8 ufshcd_scsi_to_upiu_lun(unsigned int scsi_lun)
 int ufshcd_dump_regs(struct ufs_hba *hba, size_t offset, size_t len,
 		     const char *prefix);
 
+int __ufshcd_write_ee_control(struct ufs_hba *hba, u32 ee_ctrl_mask);
+int ufshcd_write_ee_control(struct ufs_hba *hba);
 int ufshcd_update_ee_control(struct ufs_hba *hba, u16 *mask, u16 *other_mask,
 			     u16 set, u16 clr);
 
Signature · 1 line
2.17.1
Acked-by on v2 4/4
#8 lore raw

Re: [PATCH V2 4/4] scsi: ufs-debugfs: Add user-defined exception event rate limiting

Quoted 10 lines · Adrian Hunter

An enabled user-specified exception event that does not clear quickly will
repeatedly cause the handler to run. That could unduly disturb the driver
behaviour being tested or debugged. To prevent that add debugfs file exception_event_rate_limit_ms. When a exception event happens, it is disabled, and then after a period of time (default 20ms) the exception
event is enabled again.

Quoted 5 lines

Note that if the driver also has that exception event enabled, it will not
be disabled.

#9 lore raw
In reply to Adrian Hunter
Quoted 8 lines · Adrian Hunter

Here are V2 patches to add a tracepoint for UFS Exception Events and to allow users to enable specific exception events without affecting the driver's use of exception events.

Changes in V2:

[...]

Applied to 5.13/scsi-queue, thanks!

[1/4] scsi: ufs: Add exception event tracepoint

      https://git.kernel.org/mkp/scsi/c/26b01633fac8
[2/4] scsi: ufs: Add exception event definitions
      https://git.kernel.org/mkp/scsi/c/8fd31fc2b84f
[3/4] scsi: ufs-debugfs: Add user-defined exception_event_mask
      https://git.kernel.org/mkp/scsi/c/e1c8b528dd23
[4/4] scsi: ufs-debugfs: Add user-defined exception event rate limiting
      https://git.kernel.org/mkp/scsi/c/c3f04083d653
Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering