Skip to contentSkip to conversation

drm: introduce KMS recovery mechanism

PATCH Superseded Hamza Mahfooz 5 participants 16 messages Feb 12 – Apr 17

Subsystems (4)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/2 drm: introduce KMS recovery mechanism

There should be a mechanism for drivers to respond to flip_done timeouts. Since, as it stands it is possible for the display to stall indefinitely, necessitating a hard reset. So, introduce a new mechanism that tries various methods of recovery with increasing aggression, in the following order:

1. Force a full modeset (have the compositor reprogram the state from scratch).
2. As a last resort, have the driver attempt a vendor specific reset (which they can do by reading the return value of drm_atomic_helper_wait_for_flip_done()).

---
v2: new to the series
v3: get rid of page_flip_timeout() and have
drm_atomic_helper_wait_for_flip_done() return a error.

---
 drivers/gpu/drm/drm_atomic_helper.c | 45 +++++++++++++++++++++++++----
 include/drm/drm_atomic_helper.h     |  4 +--
 include/drm/drm_device.h            | 24 +++++++++++++++
 3 files changed, 66 insertions(+), 7 deletions(-)
drivers/gpu/drm/drm_atomic_helper.c +40 -5
index 5840e9cc6f66..6ae1234b9e20 100644
@@ -42,6 +42,7 @@
 #include <drm/drm_gem_atomic_helper.h>
 #include <drm/drm_panic.h>
 #include <drm/drm_print.h>
+#include <drm/drm_probe_helper.h>
 #include <drm/drm_self_refresh_helper.h>
 #include <drm/drm_vblank.h>
 #include <drm/drm_writeback.h>
@@ -1864,11 +1865,15 @@ EXPORT_SYMBOL(drm_atomic_helper_wait_for_vblanks);
  *
  * This requires that drivers use the nonblocking commit tracking support
  * initialized using drm_atomic_helper_setup_commit().
+ *
+ * Returns:
+ * -ETIMEDOUT to indicate that drivers can attempt a vendor reset, 0 otherwise.
  */
-void drm_atomic_helper_wait_for_flip_done(struct drm_device *dev,
-					  struct drm_atomic_state *state)
+int drm_atomic_helper_wait_for_flip_done(struct drm_device *dev,
+					 struct drm_atomic_state *state)
 {
 	struct drm_crtc *crtc;
+	int ret = 0;
 	int i;
 
 	for (i = 0; i < dev->mode_config.num_crtc; i++) {
@@ -1881,13 +1886,43 @@ void drm_atomic_helper_wait_for_flip_done(struct drm_device *dev,
 			continue;
 
 		ret = wait_for_completion_timeout(&commit->flip_done, 10 * HZ);
-		if (ret == 0)
-			drm_err(dev, "[CRTC:%d:%s] flip_done timed out\n",
-				crtc->base.id, crtc->name);
+		if (!ret) {
+			switch (dev->reset_phase) {
+			case DRM_KMS_RESET_NONE:
+				drm_err(dev, "[CRTC:%d:%s] flip_done timed out\n",
+					crtc->base.id, crtc->name);
+				dev->reset_phase = DRM_KMS_RESET_FORCE_MODESET;
+				drm_kms_helper_hotplug_event(dev);
+				break;
+			case DRM_KMS_RESET_FORCE_MODESET:
+				drm_err(dev, "[CRTC:%d:%s] force full modeset failed\n",
+					crtc->base.id, crtc->name);
+				dev->reset_phase = DRM_KMS_RESET_VENDOR;
+				ret = -ETIMEDOUT;
+				break;
+			case DRM_KMS_RESET_VENDOR:
+				drm_err(dev, "[CRTC:%d:%s] KMS recovery failed!\n",
+					crtc->base.id, crtc->name);
+				dev->reset_phase = DRM_KMS_RESET_GIVE_UP;
+				break;
+			default:
+				break;
+			}
+
+			goto exit;
+		}
+	}
+
+	if (dev->reset_phase) {
+		drm_info(dev, "KMS recovery succeeded!\n");
+		dev->reset_phase = DRM_KMS_RESET_NONE;
 	}
 
+exit:
 	if (state->fake_commit)
 		complete_all(&state->fake_commit->flip_done);
+
+	return ret;
 }
 EXPORT_SYMBOL(drm_atomic_helper_wait_for_flip_done);
 
include/drm/drm_atomic_helper.h +2 -2
index 53382fe93537..298c8dff3993 100644
@@ -79,8 +79,8 @@ int drm_atomic_helper_wait_for_fences(struct drm_device *dev,
 void drm_atomic_helper_wait_for_vblanks(struct drm_device *dev,
 					struct drm_atomic_state *old_state);
 
-void drm_atomic_helper_wait_for_flip_done(struct drm_device *dev,
-					  struct drm_atomic_state *old_state);
+int drm_atomic_helper_wait_for_flip_done(struct drm_device *dev,
+					 struct drm_atomic_state *old_state);
 
 void
 drm_atomic_helper_update_legacy_modeset_state(struct drm_device *dev,
include/drm/drm_device.h +24 -0
index bc78fb77cc27..1244d7527e7b 100644
@@ -66,6 +66,23 @@ enum switch_power_state {
 	DRM_SWITCH_POWER_DYNAMIC_OFF = 3,
 };
 
+/**
+ * enum drm_kms_reset_phase - reset phase of drm device
+ */
+enum drm_kms_reset_phase {
+	/** @DRM_KMS_RESET_NONE: Not currently attempting recovery */
+	DRM_KMS_RESET_NONE,
+
+	/** @DRM_KMS_RESET_FORCE_MODESET: Force a full modeset */
+	DRM_KMS_RESET_FORCE_MODESET,
+
+	/** @DRM_KMS_RESET_VENDOR: Attempt a vendor reset */
+	DRM_KMS_RESET_VENDOR,
+
+	/** @DRM_KMS_RESET_GIVE_UP: All recovery methods failed */
+	DRM_KMS_RESET_GIVE_UP,
+};
+
 /**
  * struct drm_device - DRM device structure
  *
@@ -375,6 +392,13 @@ struct drm_device {
 	 * Root directory for debugfs files.
 	 */
 	struct dentry *debugfs_root;
+
+	/**
+	 * @reset_phase:
+	 *
+	 * Reset phase that the device is in.
+	 */
+	enum drm_kms_reset_phase reset_phase;
 };
 
 void drm_dev_set_dma_dev(struct drm_device *dev, struct device *dma_dev);
Signature · 1 line
2.53.0
#1 lore raw

PATCH v3 2/2 drm/amd/display: add vendor specific reset

We now have a means to respond to page flip timeouts. So, hook up support by sending out a wedged event if drm_atomic_helper_wait_for_flip_done() fails.

---
v2: send a wedged event instead of attempting a GPU reset.
v3: read return value of drm_atomic_helper_wait_for_flip_done().

---
 drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c +7 -2
index 7c51d8d7e73c..06d8353cb616 100644
@@ -87,6 +87,7 @@
 #include <drm/drm_atomic_uapi.h>
 #include <drm/drm_atomic_helper.h>
 #include <drm/drm_blend.h>
+#include <drm/drm_drv.h>
 #include <drm/drm_fixed.h>
 #include <drm/drm_fourcc.h>
 #include <drm/drm_edid.h>
@@ -11085,8 +11086,12 @@ static void amdgpu_dm_atomic_commit_tail(struct drm_atomic_state *state)
 	/* Signal HW programming completion */
 	drm_atomic_helper_commit_hw_done(state);
 
-	if (wait_for_vblank)
-		drm_atomic_helper_wait_for_flip_done(dev, state);
+	if (wait_for_vblank) {
+		if (drm_atomic_helper_wait_for_flip_done(dev, state))
+			drm_dev_wedged_event(dev, DRM_WEDGE_RECOVERY_REBIND |
+					     DRM_WEDGE_RECOVERY_BUS_RESET,
+					     NULL);
+	}
 
 	drm_atomic_helper_cleanup_planes(dev, state);
 
Signature · 1 line
2.53.0
#3 lore raw
In reply to Hamza Mahfooz
Quoted 11 lines · Hamza Mahfooz

There should be a mechanism for drivers to respond to flip_done timeouts. Since, as it stands it is possible for the display to stall indefinitely, necessitating a hard reset. So, introduce a new mechanism that tries various methods of recovery with increasing aggression, in the following order:

1. Force a full modeset (have the compositor reprogram the state from scratch).
2. As a last resort, have the driver attempt a vendor specific reset (which they can do by reading the return value of drm_atomic_helper_wait_for_flip_done()).

Since you were able to (relatively) reliably reproduce a problem in amdgpu, how far in your iterative flow did you get? Did you manage to need the vendor specific handling? And presumably that helped?

Quoted 47 lines

---
v2: new to the series
v3: get rid of page_flip_timeout() and have
drm_atomic_helper_wait_for_flip_done() return a error.

---
  drivers/gpu/drm/drm_atomic_helper.c | 45 +++++++++++++++++++++++++----
  include/drm/drm_atomic_helper.h     |  4 +--
  include/drm/drm_device.h            | 24 +++++++++++++++
  3 files changed, 66 insertions(+), 7 deletions(-)
drivers/gpu/drm/drm_atomic_helper.c +7 -3
index 5840e9cc6f66..6ae1234b9e20 100644
@@ -42,6 +42,7 @@
  #include <drm/drm_gem_atomic_helper.h>
  #include <drm/drm_panic.h>
  #include <drm/drm_print.h>
+#include <drm/drm_probe_helper.h>
  #include <drm/drm_self_refresh_helper.h>
  #include <drm/drm_vblank.h>
  #include <drm/drm_writeback.h>
@@ -1864,11 +1865,15 @@ EXPORT_SYMBOL(drm_atomic_helper_wait_for_vblanks);
   *
   * This requires that drivers use the nonblocking commit tracking support
   * initialized using drm_atomic_helper_setup_commit().
+ *
+ * Returns:
+ * -ETIMEDOUT to indicate that drivers can attempt a vendor reset, 0 otherwise.
   */
-void drm_atomic_helper_wait_for_flip_done(struct drm_device *dev,
-					  struct drm_atomic_state *state)
+int drm_atomic_helper_wait_for_flip_done(struct drm_device *dev,
+					 struct drm_atomic_state *state)
  {
  	struct drm_crtc *crtc;
+	int ret = 0;
  	int i;
 
  	for (i = 0; i < dev->mode_config.num_crtc; i++) {
@@ -1881,13 +1886,43 @@ void drm_atomic_helper_wait_for_flip_done(struct drm_device *dev,
  			continue;
 
  		ret = wait_for_completion_timeout(&commit->flip_done, 10 * HZ);
-		if (ret == 0)

drivers/gpu/drm/drm_atomic_helper.c void drm_atomic_helper_wait_for_flip_done(struct drm_device *dev, · line 1896

Quoted 10 lines
drivers/gpu/drm/drm_atomic_helper.c +8 -2
-			drm_err(dev, "[CRTC:%d:%s] flip_done timed out\n",
-				crtc->base.id, crtc->name);
+		if (!ret) {
+			switch (dev->reset_phase) {
+			case DRM_KMS_RESET_NONE:
+				drm_err(dev, "[CRTC:%d:%s] flip_done timed out\n",
+					crtc->base.id, crtc->name);
+				dev->reset_phase = DRM_KMS_RESET_FORCE_MODESET;
+				drm_kms_helper_hotplug_event(dev);
+				break;

Since you're iterating multiple CRTCs if you manage to recover from one with this call shouldn't you keep iterating the rest?

Quoted 89 lines
drivers/gpu/drm/drm_atomic_helper.c +25 -0
+			case DRM_KMS_RESET_FORCE_MODESET:
+				drm_err(dev, "[CRTC:%d:%s] force full modeset failed\n",
+					crtc->base.id, crtc->name);
+				dev->reset_phase = DRM_KMS_RESET_VENDOR;
+				ret = -ETIMEDOUT;
+				break;
+			case DRM_KMS_RESET_VENDOR:
+				drm_err(dev, "[CRTC:%d:%s] KMS recovery failed!\n",
+					crtc->base.id, crtc->name);
+				dev->reset_phase = DRM_KMS_RESET_GIVE_UP;
+				break;
+			default:
+				break;
+			}
+
+			goto exit;
+		}
+	}
+
+	if (dev->reset_phase) {
+		drm_info(dev, "KMS recovery succeeded!\n");
+		dev->reset_phase = DRM_KMS_RESET_NONE;
  	}
 
+exit:
  	if (state->fake_commit)
  		complete_all(&state->fake_commit->flip_done);
+
+	return ret;
  }
  EXPORT_SYMBOL(drm_atomic_helper_wait_for_flip_done);
 
include/drm/drm_atomic_helper.h +2 -2
index 53382fe93537..298c8dff3993 100644
@@ -79,8 +79,8 @@ int drm_atomic_helper_wait_for_fences(struct drm_device *dev,
  void drm_atomic_helper_wait_for_vblanks(struct drm_device *dev,
  					struct drm_atomic_state *old_state);
 
-void drm_atomic_helper_wait_for_flip_done(struct drm_device *dev,
-					  struct drm_atomic_state *old_state);
+int drm_atomic_helper_wait_for_flip_done(struct drm_device *dev,
+					 struct drm_atomic_state *old_state);
 
  void
  drm_atomic_helper_update_legacy_modeset_state(struct drm_device *dev,
include/drm/drm_device.h +24 -0
index bc78fb77cc27..1244d7527e7b 100644
@@ -66,6 +66,23 @@ enum switch_power_state {
  	DRM_SWITCH_POWER_DYNAMIC_OFF = 3,
  };
 
+/**
+ * enum drm_kms_reset_phase - reset phase of drm device
+ */
+enum drm_kms_reset_phase {
+	/** @DRM_KMS_RESET_NONE: Not currently attempting recovery */
+	DRM_KMS_RESET_NONE,
+
+	/** @DRM_KMS_RESET_FORCE_MODESET: Force a full modeset */
+	DRM_KMS_RESET_FORCE_MODESET,
+
+	/** @DRM_KMS_RESET_VENDOR: Attempt a vendor reset */
+	DRM_KMS_RESET_VENDOR,
+
+	/** @DRM_KMS_RESET_GIVE_UP: All recovery methods failed */
+	DRM_KMS_RESET_GIVE_UP,
+};
+
  /**
   * struct drm_device - DRM device structure
   *
@@ -375,6 +392,13 @@ struct drm_device {
  	 * Root directory for debugfs files.
  	 */
  	struct dentry *debugfs_root;
+
+	/**
+	 * @reset_phase:
+	 *
+	 * Reset phase that the device is in.
+	 */
+	enum drm_kms_reset_phase reset_phase;
  };
 
  void drm_dev_set_dma_dev(struct drm_device *dev, struct device *dma_dev);
#4 lore raw
Quoted 4 lines · Mario Limonciello

Since you were able to (relatively) reliably reproduce a problem in amdgpu, how far in your iterative flow did you get? Did you manage to need the vendor specific handling? And presumably that helped?

Every time I've tested it (with my repro) the full modeset has failed and it was able to recover with the vendor specific handling. Though it's worth noting that I strongly suspect a firmware hang in my case[1].

[1] https://lore.kernel.org/r/aYplYyf6Pp20lOAD@hal-station/

Quoted 9 lines
Quoted 9 lines
drivers/gpu/drm/drm_atomic_helper.c +3 -3
@@ -1881,13 +1886,43 @@ void drm_atomic_helper_wait_for_flip_done(struct drm_device *dev,
  			continue;
  		ret = wait_for_completion_timeout(&commit->flip_done, 10 * HZ);
-		if (ret == 0)
-			drm_err(dev, "[CRTC:%d:%s] flip_done timed out\n",
-				crtc->base.id, crtc->name);
+		if (!ret) {
+			switch (dev->reset_phase) {
+			case DRM_KMS_RESET_NONE:
Quoted 9 lines

drivers/gpu/drm/drm_atomic_helper.c void drm_atomic_helper_wait_for_flip_done(struct drm_device *dev, · line 1896

Quoted 5 lines
drivers/gpu/drm/drm_atomic_helper.c +5 -0
+				drm_err(dev, "[CRTC:%d:%s] flip_done timed out\n",
+					crtc->base.id, crtc->name);
+				dev->reset_phase = DRM_KMS_RESET_FORCE_MODESET;
+				drm_kms_helper_hotplug_event(dev);
+				break;

Since you're iterating multiple CRTCs if you manage to recover from one with this call shouldn't you keep iterating the rest?

Most measures that the can be implemented at the kernel level (including forcing a full modeset), can't save the the current commit. So, in all likelihood we will just end up waiting an extra 10 seconds per CRTC (assuming they haven't completed already, unrelated to the forced modeset).

#5 lore raw
Quoted 20 lines · Hamza Mahfooz
Quoted 9 lines · Mario Limonciello
Quoted 9 lines
drivers/gpu/drm/drm_atomic_helper.c +3 -3
@@ -1881,13 +1886,43 @@ void drm_atomic_helper_wait_for_flip_done(struct drm_device *dev,
  			continue;
  		ret = wait_for_completion_timeout(&commit->flip_done, 10 * HZ);
-		if (ret == 0)
-			drm_err(dev, "[CRTC:%d:%s] flip_done timed out\n",
-				crtc->base.id, crtc->name);
+		if (!ret) {
+			switch (dev->reset_phase) {
+			case DRM_KMS_RESET_NONE:
Quoted 8 lines

drivers/gpu/drm/drm_atomic_helper.c void drm_atomic_helper_wait_for_flip_done(struct drm_device *dev, · line 1896

Quoted 5 lines
drivers/gpu/drm/drm_atomic_helper.c +5 -0
+				drm_err(dev, "[CRTC:%d:%s] flip_done timed out\n",
+					crtc->base.id, crtc->name);
+				dev->reset_phase = DRM_KMS_RESET_FORCE_MODESET;
+				drm_kms_helper_hotplug_event(dev);
+				break;

Since you're iterating multiple CRTCs if you manage to recover from one with this call shouldn't you keep iterating the rest?

Quoted 2 lines

Most measures that the can be implemented at the kernel level (including forcing a full modeset), can't save the the current commit.

Why couldn't a full modeset?

Quoted 3 lines

So, in all likelihood we will just end up waiting an extra 10 seconds per CRTC (assuming they haven't completed already, unrelated to the forced modeset).

In principle it's possible to do (the equivalent of) a modeset with the current state for all CRTCs, no need to do it separately per CRTC.

Signature · 2 lines
Earthling Michel Dänzer       \        GNOME / Xwayland / Mesa developer
https://redhat.com             \               Libre software enthusiast
#6 lore raw
Quoted 2 lines · Michel Dänzer

Why couldn't a full modeset?

As far as I see it the only reasons why we should be timing out is either an interrupt went missing (perhaps due to a race condition in driver code) or hung hardware. In either case, the interrupt associated with the page flip for the current commit is long gone.

Quoted 1 line

In principle it's possible to do (the equivalent of) a modeset with the current state for all CRTCs, no need to do it separately per CRTC.

AFAIK that is what the uevent is already doing (unless I'm mistaken).

#7 lore raw
Quoted 7 lines · Hamza Mahfooz
Quoted 1 line · Michel Dänzer

Why couldn't a full modeset?

As far as I see it the only reasons why we should be timing out is either an interrupt went missing (perhaps due to a race condition in driver code) or hung hardware. In either case, the interrupt associated with the page flip for the current commit is long gone.

That's a matter of bookkeeping, the interrupt isn't required to keep track of the commit and complete it.

Quoted 3 lines
Quoted 1 line

In principle it's possible to do (the equivalent of) a modeset with the current state for all CRTCs, no need to do it separately per CRTC.

AFAIK that is what the uevent is already doing (unless I'm mistaken).

This is about just doing a full modeset, which isn't something user space can do in response to a wedged event.

Signature · 2 lines
Earthling Michel Dänzer       \        GNOME / Xwayland / Mesa developer
https://redhat.com             \               Libre software enthusiast
#8 lore raw
Quoted 10 lines · Michel Dänzer
Quoted 7 lines · Hamza Mahfooz
Quoted 1 line · Michel Dänzer

Why couldn't a full modeset?

As far as I see it the only reasons why we should be timing out is either an interrupt went missing (perhaps due to a race condition in driver code) or hung hardware. In either case, the interrupt associated with the page flip for the current commit is long gone.

That's a matter of bookkeeping, the interrupt isn't required to keep track of the commit and complete it.

Oh, if you're talking about sending out a "fake" drm_crtc_send_vblank_event(), I had considered that. Though, drivers that have hardware vblank counters take ownership of the relevant struct drm_pending_vblank_event [1] pretty early on. So, there wouldn't be a way to ensure that they send that out and that would mean we never get a chance to force a full modeset.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/drm/drm_crtc.h?h=v6.19#n386

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

In principle it's possible to do (the equivalent of) a modeset with the current state for all CRTCs, no need to do it separately per CRTC.

AFAIK that is what the uevent is already doing (unless I'm mistaken).

This is about just doing a full modeset, which isn't something user space can do in response to a wedged event.

I was referring to what compositors are doing in response to drm_kms_helper_hotplug_event(). Perhaps, the enum constants should be renamed, since the forced modeset is actually sent when the current reset phase is DRM_KMS_RESET_NONE (the phase is updated before sending out the event though).

#9 lore raw
Quoted 10 lines · Hamza Mahfooz
Quoted 9 lines · Michel Dänzer
Quoted 7 lines · Hamza Mahfooz
Quoted 1 line · Michel Dänzer

Why couldn't a full modeset?

As far as I see it the only reasons why we should be timing out is either an interrupt went missing (perhaps due to a race condition in driver code) or hung hardware. In either case, the interrupt associated with the page flip for the current commit is long gone.

Quoted 8 lines
Quoted 1 line

That's a matter of bookkeeping, the interrupt isn't required to keep track of the commit and complete it.

Oh, if you're talking about sending out a "fake" drm_crtc_send_vblank_event(), I had considered that. Though, drivers that have hardware vblank counters take ownership of the relevant struct drm_pending_vblank_event [1] pretty early on. So, there wouldn't be a way to ensure that they send that out and that would mean we never get a chance to force a full modeset.

"never get a chance" why? The driver can do a full modeset and then send out the event(s) with the current vblank sequence number(s) and timestamp(s) at that time.

Quoted 11 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

In principle it's possible to do (the equivalent of) a modeset with the current state for all CRTCs, no need to do it separately per CRTC.

AFAIK that is what the uevent is already doing (unless I'm mistaken).

This is about just doing a full modeset, which isn't something user space can do in response to a wedged event.

I was referring to what compositors are doing in response to drm_kms_helper_hotplug_event(). Perhaps, the enum constants should be renamed, since the forced modeset is actually sent when the current reset phase is DRM_KMS_RESET_NONE (the phase is updated before sending out the event though).

Ah, you're talking about the drm_kms_helper_hotplug_event call in drm_atomic_helper_wait_for_flip_done (I thought "uevent" referred to drm_dev_wedged_event in patch 2).

I don't know about other compositors, but I don't think mutter will do a modeset in response to a hotplug event if no KMS state changed (because some monitors can generate spurious hotplug events).

(Just one more reason why kicking the full modeset can down to user space is questionable)

Signature · 2 lines
Earthling Michel Dänzer       \        GNOME / Xwayland / Mesa developer
https://redhat.com             \               Libre software enthusiast
maintainer
#15 lore raw
In reply to Hamza Mahfooz
Quoted 54 lines · Hamza Mahfooz

There should be a mechanism for drivers to respond to flip_done timeouts. Since, as it stands it is possible for the display to stall indefinitely, necessitating a hard reset. So, introduce a new mechanism that tries various methods of recovery with increasing aggression, in the following order:

1. Force a full modeset (have the compositor reprogram the state from scratch).
2. As a last resort, have the driver attempt a vendor specific reset (which they can do by reading the return value of drm_atomic_helper_wait_for_flip_done()).

---
v2: new to the series
v3: get rid of page_flip_timeout() and have
drm_atomic_helper_wait_for_flip_done() return a error.

---
 drivers/gpu/drm/drm_atomic_helper.c | 45 +++++++++++++++++++++++++----
 include/drm/drm_atomic_helper.h     |  4 +--
 include/drm/drm_device.h            | 24 +++++++++++++++
 3 files changed, 66 insertions(+), 7 deletions(-)
drivers/gpu/drm/drm_atomic_helper.c +7 -2
index 5840e9cc6f66..6ae1234b9e20 100644
@@ -42,6 +42,7 @@
 #include <drm/drm_gem_atomic_helper.h>
 #include <drm/drm_panic.h>
 #include <drm/drm_print.h>
+#include <drm/drm_probe_helper.h>
 #include <drm/drm_self_refresh_helper.h>
 #include <drm/drm_vblank.h>
 #include <drm/drm_writeback.h>
@@ -1864,11 +1865,15 @@ EXPORT_SYMBOL(drm_atomic_helper_wait_for_vblanks);
  *
  * This requires that drivers use the nonblocking commit tracking support
  * initialized using drm_atomic_helper_setup_commit().
+ *
+ * Returns:
+ * -ETIMEDOUT to indicate that drivers can attempt a vendor reset, 0 otherwise.
  */
-void drm_atomic_helper_wait_for_flip_done(struct drm_device *dev,
-					  struct drm_atomic_state *state)
+int drm_atomic_helper_wait_for_flip_done(struct drm_device *dev,
+					 struct drm_atomic_state *state)
 {
 	struct drm_crtc *crtc;
+	int ret = 0;
 	int i;
 
 	for (i = 0; i < dev->mode_config.num_crtc; i++) {

drivers/gpu/drm/drm_atomic_helper.c void drm_atomic_helper_wait_for_flip_done(struct drm_device *dev, · line 1891

Quoted 9 lines
drivers/gpu/drm/drm_atomic_helper.c +3 -3
@@ -1881,13 +1886,43 @@ void drm_atomic_helper_wait_for_flip_done(struct drm_device *dev,
 			continue;
 
 		ret = wait_for_completion_timeout(&commit->flip_done, 10 * HZ);
-		if (ret == 0)
-			drm_err(dev, "[CRTC:%d:%s] flip_done timed out\n",
-				crtc->base.id, crtc->name);
+		if (!ret) {
+			switch (dev->reset_phase) {
+			case DRM_KMS_RESET_NONE:

That state machine is driver specific and so doesn't belong into DRM.

So please completely nuke that and just return the error code to the driver.

Regards,
Christian.

Quoted 94 lines
drivers/gpu/drm/drm_atomic_helper.c +30 -0
+				drm_err(dev, "[CRTC:%d:%s] flip_done timed out\n",
+					crtc->base.id, crtc->name);
+				dev->reset_phase = DRM_KMS_RESET_FORCE_MODESET;
+				drm_kms_helper_hotplug_event(dev);
+				break;
+			case DRM_KMS_RESET_FORCE_MODESET:
+				drm_err(dev, "[CRTC:%d:%s] force full modeset failed\n",
+					crtc->base.id, crtc->name);
+				dev->reset_phase = DRM_KMS_RESET_VENDOR;
+				ret = -ETIMEDOUT;
+				break;
+			case DRM_KMS_RESET_VENDOR:
+				drm_err(dev, "[CRTC:%d:%s] KMS recovery failed!\n",
+					crtc->base.id, crtc->name);
+				dev->reset_phase = DRM_KMS_RESET_GIVE_UP;
+				break;
+			default:
+				break;
+			}
+
+			goto exit;
+		}
+	}
+
+	if (dev->reset_phase) {
+		drm_info(dev, "KMS recovery succeeded!\n");
+		dev->reset_phase = DRM_KMS_RESET_NONE;
 	}
 
+exit:
 	if (state->fake_commit)
 		complete_all(&state->fake_commit->flip_done);
+
+	return ret;
 }
 EXPORT_SYMBOL(drm_atomic_helper_wait_for_flip_done);
 
include/drm/drm_atomic_helper.h +2 -2
index 53382fe93537..298c8dff3993 100644
@@ -79,8 +79,8 @@ int drm_atomic_helper_wait_for_fences(struct drm_device *dev,
 void drm_atomic_helper_wait_for_vblanks(struct drm_device *dev,
 					struct drm_atomic_state *old_state);
 
-void drm_atomic_helper_wait_for_flip_done(struct drm_device *dev,
-					  struct drm_atomic_state *old_state);
+int drm_atomic_helper_wait_for_flip_done(struct drm_device *dev,
+					 struct drm_atomic_state *old_state);
 
 void
 drm_atomic_helper_update_legacy_modeset_state(struct drm_device *dev,
include/drm/drm_device.h +24 -0
index bc78fb77cc27..1244d7527e7b 100644
@@ -66,6 +66,23 @@ enum switch_power_state {
 	DRM_SWITCH_POWER_DYNAMIC_OFF = 3,
 };
 
+/**
+ * enum drm_kms_reset_phase - reset phase of drm device
+ */
+enum drm_kms_reset_phase {
+	/** @DRM_KMS_RESET_NONE: Not currently attempting recovery */
+	DRM_KMS_RESET_NONE,
+
+	/** @DRM_KMS_RESET_FORCE_MODESET: Force a full modeset */
+	DRM_KMS_RESET_FORCE_MODESET,
+
+	/** @DRM_KMS_RESET_VENDOR: Attempt a vendor reset */
+	DRM_KMS_RESET_VENDOR,
+
+	/** @DRM_KMS_RESET_GIVE_UP: All recovery methods failed */
+	DRM_KMS_RESET_GIVE_UP,
+};
+
 /**
  * struct drm_device - DRM device structure
  *
@@ -375,6 +392,13 @@ struct drm_device {
 	 * Root directory for debugfs files.
 	 */
 	struct dentry *debugfs_root;
+
+	/**
+	 * @reset_phase:
+	 *
+	 * Reset phase that the device is in.
+	 */
+	enum drm_kms_reset_phase reset_phase;
 };
 
 void drm_dev_set_dma_dev(struct drm_device *dev, struct device *dma_dev);
maintainer
#2 lore raw

Re: [PATCH v3 2/2] drm/amd/display: add vendor specific reset

In reply to Hamza Mahfooz
Quoted 25 lines · Hamza Mahfooz

We now have a means to respond to page flip timeouts. So, hook up support by sending out a wedged event if drm_atomic_helper_wait_for_flip_done() fails.

---
v2: send a wedged event instead of attempting a GPU reset.
v3: read return value of drm_atomic_helper_wait_for_flip_done().

---
 drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c +1 -0
index 7c51d8d7e73c..06d8353cb616 100644
@@ -87,6 +87,7 @@
 #include <drm/drm_atomic_uapi.h>
 #include <drm/drm_atomic_helper.h>
 #include <drm/drm_blend.h>
+#include <drm/drm_drv.h>
 #include <drm/drm_fixed.h>
 #include <drm/drm_fourcc.h>
 #include <drm/drm_edid.h>

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c static void amdgpu_dm_atomic_commit_tail(struct drm_atomic_state *state) · line 11093

Quoted 10 lines
drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c +5 -2
@@ -11085,8 +11086,12 @@ static void amdgpu_dm_atomic_commit_tail(struct drm_atomic_state *state)
 	/* Signal HW programming completion */
 	drm_atomic_helper_commit_hw_done(state);
 
-	if (wait_for_vblank)
-		drm_atomic_helper_wait_for_flip_done(dev, state);
+	if (wait_for_vblank) {
+		if (drm_atomic_helper_wait_for_flip_done(dev, state))
+			drm_dev_wedged_event(dev, DRM_WEDGE_RECOVERY_REBIND |
+					     DRM_WEDGE_RECOVERY_BUS_RESET,
+					     NULL);

Well that is the worst of all options since it will completely kill userspace.

We need to figure out what exactly is necessary to get the CRTC going again.

Regards,
Christian.

Quoted 4 lines
drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c +1 -0
+	}
 
 	drm_atomic_helper_cleanup_planes(dev, state);
 
#10 lore raw
In reply to Michel Dänzer
Quoted 17 lines · Michel Dänzer
Quoted 8 lines · Hamza Mahfooz
Quoted 7 lines · Michel Dänzer
Quoted 5 lines · Hamza Mahfooz
Quoted 1 line · Michel Dänzer

In principle it's possible to do (the equivalent of) a modeset with the current state for all CRTCs, no need to do it separately per CRTC.

AFAIK that is what the uevent is already doing (unless I'm mistaken).

Quoted 7 lines
Quoted 1 line

This is about just doing a full modeset, which isn't something user space can do in response to a wedged event.

I was referring to what compositors are doing in response to drm_kms_helper_hotplug_event(). Perhaps, the enum constants should be renamed, since the forced modeset is actually sent when the current reset phase is DRM_KMS_RESET_NONE (the phase is updated before sending out the event though).

Quoted 3 lines

Ah, you're talking about the drm_kms_helper_hotplug_event call in drm_atomic_helper_wait_for_flip_done (I thought "uevent" referred to drm_dev_wedged_event in patch 2).

I don't know about other compositors, but I don't think mutter will do a modeset in response to a hotplug event if no KMS state changed (because some monitors can generate spurious hotplug events).

FWIW, if it really turns out impossible for the kernel to do a modeset itself (which I remain unconvinced of), one way to require a modeset from user space is to set the "link-status" connector property to "Bad".

Signature · 2 lines
Earthling Michel Dänzer       \        GNOME / Xwayland / Mesa developer
https://redhat.com             \               Libre software enthusiast
#11 lore raw
Quoted 18 lines · Michel Dänzer
Quoted 17 lines · Michel Dänzer
Quoted 15 lines · Hamza Mahfooz
Quoted 8 lines · Michel Dänzer
Quoted 5 lines · Hamza Mahfooz
Quoted 1 line · Michel Dänzer

In principle it's possible to do (the equivalent of) a modeset with the current state for all CRTCs, no need to do it separately per CRTC.

AFAIK that is what the uevent is already doing (unless I'm mistaken).

This is about just doing a full modeset, which isn't something user space can do in response to a wedged event.

I was referring to what compositors are doing in response to drm_kms_helper_hotplug_event(). Perhaps, the enum constants should be renamed, since the forced modeset is actually sent when the current reset phase is DRM_KMS_RESET_NONE (the phase is updated before sending out the event though).

Quoted 5 lines
Quoted 3 lines

Ah, you're talking about the drm_kms_helper_hotplug_event call in drm_atomic_helper_wait_for_flip_done (I thought "uevent" referred to drm_dev_wedged_event in patch 2).

I don't know about other compositors, but I don't think mutter will do a modeset in response to a hotplug event if no KMS state changed (because some monitors can generate spurious hotplug events).

FWIW, if it really turns out impossible for the kernel to do a modeset itself (which I remain unconvinced of), one way to require a modeset from user space is to set the "link-status" connector property to "Bad".

Having written that down, a reason why the kernel can't just do a modeset itself occurred to me: A modeset might affect other CRTCs than the ones affected by the timed-out commit, which could interact badly with other pending commits affecting those other CRTCs.

The "link-status" property seems like the best solution so far.

(Note that not all compositors are paying attention to it yet, they'll need to be fixed)

Signature · 2 lines
Earthling Michel Dänzer       \        GNOME / Xwayland / Mesa developer
https://redhat.com             \               Libre software enthusiast
#12 lore raw
Quoted 21 lines · Michel Dänzer
Quoted 20 lines · Michel Dänzer
Quoted 19 lines · Michel Dänzer
Quoted 8 lines · Hamza Mahfooz
Quoted 7 lines · Michel Dänzer
Quoted 5 lines · Hamza Mahfooz
Quoted 1 line · Michel Dänzer

In principle it's possible to do (the equivalent of) a modeset with the current state for all CRTCs, no need to do it separately per CRTC.

AFAIK that is what the uevent is already doing (unless I'm mistaken).

Quoted 7 lines
Quoted 1 line

This is about just doing a full modeset, which isn't something user space can do in response to a wedged event.

I was referring to what compositors are doing in response to drm_kms_helper_hotplug_event(). Perhaps, the enum constants should be renamed, since the forced modeset is actually sent when the current reset phase is DRM_KMS_RESET_NONE (the phase is updated before sending out the event though).

Ah, you're talking about the drm_kms_helper_hotplug_event call in drm_atomic_helper_wait_for_flip_done (I thought "uevent" referred to drm_dev_wedged_event in patch 2).

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

I don't know about other compositors, but I don't think mutter will do a modeset in response to a hotplug event if no KMS state changed (because some monitors can generate spurious hotplug events).

FWIW, if it really turns out impossible for the kernel to do a modeset itself (which I remain unconvinced of), one way to require a modeset from user space is to set the "link-status" connector property to "Bad".

Having written that down, a reason why the kernel can't just do a modeset itself occurred to me: A modeset might affect other CRTCs than the ones affected by the timed-out commit, which could interact badly with other pending commits affecting those other CRTCs.

The "link-status" property seems like the best solution so far.

Discussing the "link-status" property, it turns out I slightly misremembered its purpose, it's not really suitable for this.

Maybe the hotplug event is the only way after all.

Signature · 2 lines
Earthling Michel Dänzer       \        GNOME / Xwayland / Mesa developer
https://redhat.com             \               Libre software enthusiast
#13 lore raw
In reply to Michel Dänzer
Quoted 19 lines · Michel Dänzer
Quoted 18 lines · Michel Dänzer
Quoted 17 lines · Michel Dänzer
Quoted 15 lines · Hamza Mahfooz
Quoted 8 lines · Michel Dänzer
Quoted 5 lines · Hamza Mahfooz
Quoted 1 line · Michel Dänzer

In principle it's possible to do (the equivalent of) a modeset with the current state for all CRTCs, no need to do it separately per CRTC.

AFAIK that is what the uevent is already doing (unless I'm mistaken).

This is about just doing a full modeset, which isn't something user space can do in response to a wedged event.

I was referring to what compositors are doing in response to drm_kms_helper_hotplug_event(). Perhaps, the enum constants should be renamed, since the forced modeset is actually sent when the current reset phase is DRM_KMS_RESET_NONE (the phase is updated before sending out the event though).

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Ah, you're talking about the drm_kms_helper_hotplug_event call in drm_atomic_helper_wait_for_flip_done (I thought "uevent" referred to drm_dev_wedged_event in patch 2).

I don't know about other compositors, but I don't think mutter will do a modeset in response to a hotplug event if no KMS state changed (because some monitors can generate spurious hotplug events).

[...]

[...], a reason why the kernel can't just do a modeset itself occurred to me: A modeset might affect other CRTCs than the ones affected by the timed-out commit, which could interact badly with other pending commits affecting those other CRTCs.

In a different thread, Ville (added to Cc) said this shouldn't be an issue.

In which case this would seem like the simplest solution.

Signature · 2 lines
Earthling Michel Dänzer       \        GNOME / Xwayland / Mesa developer
https://redhat.com             \               Libre software enthusiast
#14 lore raw
Quoted 24 lines · Michel Dänzer
Quoted 23 lines · Michel Dänzer
Quoted 22 lines · Michel Dänzer
Quoted 20 lines · Michel Dänzer
Quoted 8 lines · Hamza Mahfooz
Quoted 7 lines · Michel Dänzer
Quoted 5 lines · Hamza Mahfooz
Quoted 1 line · Michel Dänzer

In principle it's possible to do (the equivalent of) a modeset with the current state for all CRTCs, no need to do it separately per CRTC.

AFAIK that is what the uevent is already doing (unless I'm mistaken).

Quoted 7 lines
Quoted 1 line

This is about just doing a full modeset, which isn't something user space can do in response to a wedged event.

I was referring to what compositors are doing in response to drm_kms_helper_hotplug_event(). Perhaps, the enum constants should be renamed, since the forced modeset is actually sent when the current reset phase is DRM_KMS_RESET_NONE (the phase is updated before sending out the event though).

Ah, you're talking about the drm_kms_helper_hotplug_event call in drm_atomic_helper_wait_for_flip_done (I thought "uevent" referred to drm_dev_wedged_event in patch 2).

I don't know about other compositors, but I don't think mutter will do a modeset in response to a hotplug event if no KMS state changed (because some monitors can generate spurious hotplug events).

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

[...]

[...], a reason why the kernel can't just do a modeset itself occurred to me: A modeset might affect other CRTCs than the ones affected by the timed-out commit, which could interact badly with other pending commits affecting those other CRTCs.

In a different thread, Ville (added to Cc) said this shouldn't be an issue.

In which case this would seem like the simplest solution.

Yeah, so extra blocking commits will just make everything (including subsequent non-blocking commits) block on the modeset mutexes before the -EBUSY check even happens.

On a somewhat relatd note, some time ago I proposed making blocking commits also do the commit_tail part locklessly. But realizing that something might depend on the current not-actually-nonblocking commit semantics I was careful to preserve that even while dropping the locks:
https://lore.kernel.org/dri-devel/[email protected]/

Signature · 2 lines
Ville Syrjälä
Intel