Skip to contentSkip to conversation

media: i2c: ov8865: Program the mode on stream start if needed

PATCH In review Jurison Murati 3 participants 13 messages Jun 9 – Jun 10

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/2 media: i2c: ov8865: Program the mode on stream start if needed

The sensor registers are only written in the runtime PM resume handler; ov8865_set_fmt() merely stores the requested mode, relying on the sensor being runtime suspended between uses so that the next resume applies it.

That assumption breaks when something keeps the sensor powered. On IPU3 platforms, ipu_bridge instantiates the VCM device with a DL_FLAG_PM_RUNTIME device link to the sensor, so a userspace process holding the VCM subdev open (e.g. wireplumber's camera monitor) pins the sensor runtime-active. A subsequent set_fmt() then never reaches the hardware: the sensor keeps streaming the mode programmed on the last resume while the CSI-2 receiver expects the newly negotiated format.

On a Surface Book 2 (IPU3, ov8865 + dw9719 VCM), requesting the 3264x2448 mode while the hardware was left programmed for the 1632x1224 binned mode makes ipu3-cio2 report "frame sync error" and "payload length is 10340352, received 2585088" (exactly one binned frame) for every frame, and the inverse case stalls the stream after a single frame. Camera applications end up displaying one bogus frame forever.

Track the mode actually programmed into the hardware, invalidate it when the sensor is powered off, and reprogram the sensor on stream start whenever it does not match the negotiated mode, re-applying the control values afterwards.

---

drivers/media/i2c/ov8865.c +28 -0
@@ -670,6 +670,10 @@
 	const struct ov8865_mode *mode;
 	u32 mbus_code;
 
+	/* Mode currently programmed into the hardware, NULL when unpowered. */
+	const struct ov8865_mode *hw_mode;
+	u32 hw_mbus_code;
+
 	bool streaming;
 };
 
@@ -2383,6 +2387,9 @@
 		return ret;
 	}
 
+	sensor->state.hw_mode = sensor->state.mode;
+	sensor->state.hw_mbus_code = sensor->state.mbus_code;
+
 	return 0;
 }
 
@@ -2618,6 +2625,25 @@
 	}
 
 	mutex_lock(&sensor->mutex);
+
+	/*
+	 * If something else kept the sensor powered (e.g. the VCM's PM
+	 * device link holding it active), runtime resume did not run when
+	 * streaming was requested and the hardware may still be programmed
+	 * for a previous mode. Reprogram it to match the current state.
+	 */
+	if (enable && (state->hw_mode != state->mode ||
+		       state->hw_mbus_code != state->mbus_code)) {
+		ret = ov8865_sensor_init(sensor);
+		if (!ret)
+			ret = __v4l2_ctrl_handler_setup(&sensor->ctrls.handler);
+		if (ret) {
+			mutex_unlock(&sensor->mutex);
+			pm_runtime_put(sensor->dev);
+			return ret;
+		}
+	}
+
 	ret = ov8865_sw_standby(sensor, !enable);
 	mutex_unlock(&sensor->mutex);
 
@@ -2895,6 +2921,8 @@
 	ret = ov8865_sensor_power(sensor, false);
 	if (ret)
 		ov8865_sw_standby(sensor, false);
+	else
+		state->hw_mode = NULL;
 
 complete:
 	mutex_unlock(&sensor->mutex);
Signature · 1 line
2.49.0
#1 lore raw

PATCH 2/2 media: i2c: ov8865: Drop PM runtime reference if stream start fails

ov8865_s_stream() takes a runtime PM reference when enabling the stream, but returns without releasing it if ov8865_sw_standby() fails, leaving the reference unbalanced and the sensor powered indefinitely.

---

drivers/media/i2c/ov8865.c +4 -1
@@ -2647,8 +2647,11 @@
 	ret = ov8865_sw_standby(sensor, !enable);
 	mutex_unlock(&sensor->mutex);
 
-	if (ret)
+	if (ret) {
+		if (enable)
+			pm_runtime_put(sensor->dev);
 		return ret;
+	}
 
 	state->streaming = !!enable;
 
Signature · 1 line
2.49.0
Reviewed-by on v1 2/2
#2 lore raw

Re: [PATCH 2/2] media: i2c: ov8865: Drop PM runtime reference if stream start fails

Hi Jurison

Quoted 7 lines · Jurison Murati

ov8865_s_stream() takes a runtime PM reference when enabling the stream, but returns without releasing it if ov8865_sw_standby() fails, leaving the reference unbalanced and the sensor powered indefinitely.

---

Good spot, thanks:

Quoted 15 lines
drivers/media/i2c/ov8865.c +4 -1
@@ -2647,8 +2647,11 @@
  	ret = ov8865_sw_standby(sensor, !enable);
  	mutex_unlock(&sensor->mutex);
 
-	if (ret)
+	if (ret) {
+		if (enable)
+			pm_runtime_put(sensor->dev);
  		return ret;
+	}
 
  	state->streaming = !!enable;
 
#3 lore raw

Re: [PATCH 2/2] media: i2c: ov8865: Drop PM runtime reference if stream start fails

Hi Dan,

Quoted 3 lines

Good spot, thanks:

Thanks! Note that in v2 I restructured this per Sakari's suggestion (the runtime PM reference is now also dropped if disabling the stream fails), so I didn't carry your Reviewed-by over - please take another look when you get a chance.

Regards,
Jurison

maintainer
#4 lore raw

Re: [PATCH 2/2] media: i2c: ov8865: Drop PM runtime reference if stream start fails

In reply to Jurison Murati

Hi Jurison,

Quoted 11 lines · Jurison Murati

ov8865_s_stream() takes a runtime PM reference when enabling the stream, but returns without releasing it if ov8865_sw_standby() fails, leaving the reference unbalanced and the sensor powered indefinitely.

---

drivers/media/i2c/ov8865.c +0 -0
@@ -2647,8 +2647,11 @@
 	ret = ov8865_sw_standby(sensor, !enable);

drivers/media/i2c/ov8865.c line 2656

Quoted 10 lines
drivers/media/i2c/ov8865.c +4 -1
 	mutex_unlock(&sensor->mutex);
 
-	if (ret)
+	if (ret) {
+		if (enable)
+			pm_runtime_put(sensor->dev);
 		return ret;
+	}
 
 	state->streaming = !!enable;

How about:

	if (ret || !enable)
		pm_runtime_put(sensor->dev);

	if (!ret)
		state->streaming = enable;

	return ret;
Quoted 1 line

The driver should be switched to {enable,disable}_streams but that's another issue altogether.

Signature · 3 lines
Regards,

Sakari Ailus
#5 lore raw

Re: [PATCH 2/2] media: i2c: ov8865: Drop PM runtime reference if stream start fails

Hi Sakari,

Thanks for the review.

Quoted 9 lines

How about:

	if (ret || !enable)
		pm_runtime_put(sensor->dev);

	if (!ret)
		state->streaming = enable;

	return ret;

Done in v2 - that also covers the case where disabling the stream fails, which v1 missed. Since the logic changed I didn't carry Dan's Reviewed-by over. I also moved this patch to 1/2 so the stream start patch can rely on the unified error path.

Quoted 2 lines

The driver should be switched to {enable,disable}_streams but that's another issue altogether.

Noted - I'll leave that for a separate series.

Regards,
Jurison

maintainer
#6 lore raw
In reply to Jurison Murati

Hi Jurison,

Quoted 32 lines · Jurison Murati

The sensor registers are only written in the runtime PM resume handler; ov8865_set_fmt() merely stores the requested mode, relying on the sensor being runtime suspended between uses so that the next resume applies it.

That assumption breaks when something keeps the sensor powered. On IPU3 platforms, ipu_bridge instantiates the VCM device with a DL_FLAG_PM_RUNTIME device link to the sensor, so a userspace process holding the VCM subdev open (e.g. wireplumber's camera monitor) pins the sensor runtime-active. A subsequent set_fmt() then never reaches the hardware: the sensor keeps streaming the mode programmed on the last resume while the CSI-2 receiver expects the newly negotiated format.

On a Surface Book 2 (IPU3, ov8865 + dw9719 VCM), requesting the 3264x2448 mode while the hardware was left programmed for the 1632x1224 binned mode makes ipu3-cio2 report "frame sync error" and "payload length is 10340352, received 2585088" (exactly one binned frame) for every frame, and the inverse case stalls the stream after a single frame. Camera applications end up displaying one bogus frame forever.

Track the mode actually programmed into the hardware, invalidate it when the sensor is powered off, and reprogram the sensor on stream start whenever it does not match the negotiated mode, re-applying the control values afterwards.

---

drivers/media/i2c/ov8865.c line 674

Quoted 5 lines
drivers/media/i2c/ov8865.c +2 -0
@@ -670,6 +670,10 @@
 	const struct ov8865_mode *mode;
 	u32 mbus_code;
 
+	/* Mode currently programmed into the hardware, NULL when unpowered. */
+	const struct ov8865_mode *hw_mode;

We'd rather like to get rid of state outside sub-device state, not add more of it.

drivers/media/i2c/ov8865.c line 675

Quoted 1 line
drivers/media/i2c/ov8865.c +1 -0
+	u32 hw_mbus_code;

Same for this one.

Could you simply write the configuration to the sensor when it changes?

Quoted 49 lines
drivers/media/i2c/ov8865.c +25 -0
+
 	bool streaming;
 };
 
@@ -2383,6 +2387,9 @@
 		return ret;
 	}
 
+	sensor->state.hw_mode = sensor->state.mode;
+	sensor->state.hw_mbus_code = sensor->state.mbus_code;
+
 	return 0;
 }
 
@@ -2618,6 +2625,25 @@
 	}
 
 	mutex_lock(&sensor->mutex);
+
+	/*
+	 * If something else kept the sensor powered (e.g. the VCM's PM
+	 * device link holding it active), runtime resume did not run when
+	 * streaming was requested and the hardware may still be programmed
+	 * for a previous mode. Reprogram it to match the current state.
+	 */
+	if (enable && (state->hw_mode != state->mode ||
+		       state->hw_mbus_code != state->mbus_code)) {
+		ret = ov8865_sensor_init(sensor);
+		if (!ret)
+			ret = __v4l2_ctrl_handler_setup(&sensor->ctrls.handler);
+		if (ret) {
+			mutex_unlock(&sensor->mutex);
+			pm_runtime_put(sensor->dev);
+			return ret;
+		}
+	}
+
 	ret = ov8865_sw_standby(sensor, !enable);
 	mutex_unlock(&sensor->mutex);
 
@@ -2895,6 +2921,8 @@
 	ret = ov8865_sensor_power(sensor, false);
 	if (ret)
 		ov8865_sw_standby(sensor, false);
+	else
+		state->hw_mode = NULL;
 
 complete:
 	mutex_unlock(&sensor->mutex);
Signature · 3 lines
Regards,

Sakari Ailus
#7 lore raw

Hi Sakari,

Thanks for the review.

Quoted 4 lines

We'd rather like to get rid of state outside sub-device state, not add more of it.

Could you simply write the configuration to the sensor when it changes?

Right, that's nicer. I've done it the way Dan suggested in his reply:
the configuration is now written unconditionally at stream start, and the runtime PM resume handler only writes it when resuming with the stream already started. No new driver state needed.

Sent as v2, retested on the Surface Book 2 in the original failing configuration.

Regards,
Jurison

#8 lore raw
In reply to Jurison Murati

Hi Jurison, thanks for the patch

Quoted 64 lines · Jurison Murati

The sensor registers are only written in the runtime PM resume handler; ov8865_set_fmt() merely stores the requested mode, relying on the sensor being runtime suspended between uses so that the next resume applies it.

That assumption breaks when something keeps the sensor powered. On IPU3 platforms, ipu_bridge instantiates the VCM device with a DL_FLAG_PM_RUNTIME device link to the sensor, so a userspace process holding the VCM subdev open (e.g. wireplumber's camera monitor) pins the sensor runtime-active. A subsequent set_fmt() then never reaches the hardware: the sensor keeps streaming the mode programmed on the last resume while the CSI-2 receiver expects the newly negotiated format.

On a Surface Book 2 (IPU3, ov8865 + dw9719 VCM), requesting the 3264x2448 mode while the hardware was left programmed for the 1632x1224 binned mode makes ipu3-cio2 report "frame sync error" and "payload length is 10340352, received 2585088" (exactly one binned frame) for every frame, and the inverse case stalls the stream after a single frame. Camera applications end up displaying one bogus frame forever.

Track the mode actually programmed into the hardware, invalidate it when the sensor is powered off, and reprogram the sensor on stream start whenever it does not match the negotiated mode, re-applying the control values afterwards.

---

drivers/media/i2c/ov8865.c +15 -0
@@ -670,6 +670,10 @@
  	const struct ov8865_mode *mode;
  	u32 mbus_code;
 
+	/* Mode currently programmed into the hardware, NULL when unpowered. */
+	const struct ov8865_mode *hw_mode;
+	u32 hw_mbus_code;
+
  	bool streaming;
  };
 
@@ -2383,6 +2387,9 @@
  		return ret;
  	}
 
+	sensor->state.hw_mode = sensor->state.mode;
+	sensor->state.hw_mbus_code = sensor->state.mbus_code;
+
  	return 0;
  }
 
@@ -2618,6 +2625,25 @@
  	}
 
  	mutex_lock(&sensor->mutex);
+
+	/*
+	 * If something else kept the sensor powered (e.g. the VCM's PM
+	 * device link holding it active), runtime resume did not run when
+	 * streaming was requested and the hardware may still be programmed
+	 * for a previous mode. Reprogram it to match the current state.
+	 */
+	if (enable && (state->hw_mode != state->mode ||

drivers/media/i2c/ov8865.c line 2645

Quoted 10 lines
drivers/media/i2c/ov8865.c +10 -0
+		       state->hw_mbus_code != state->mbus_code)) {
+		ret = ov8865_sensor_init(sensor);
+		if (!ret)
+			ret = __v4l2_ctrl_handler_setup(&sensor->ctrls.handler);
+		if (ret) {
+			mutex_unlock(&sensor->mutex);
+			pm_runtime_put(sensor->dev);
+			return ret;
+		}
+	}

ov8865_resume() writes the sensor configuration probably to support mid-stream suspend. If we moved the ov8865_sensor_init() and __v4l2_ctrl_handler_setup() calls there behind the state->streaming guard along with ov8865_sw_standby() then I think they could just be called unconditionally in the enable path in ov886_s_stream() without having to store a mode in struct ov8865_state

Thanks
Dan

Quoted 12 lines
drivers/media/i2c/ov8865.c +2 -0
  	ret = ov8865_sw_standby(sensor, !enable);
  	mutex_unlock(&sensor->mutex);
 
@@ -2895,6 +2921,8 @@
  	ret = ov8865_sensor_power(sensor, false);
  	if (ret)
  		ov8865_sw_standby(sensor, false);
+	else
+		state->hw_mode = NULL;
 
  complete:
  	mutex_unlock(&sensor->mutex);
#9 lore raw

Hi Dan,

Thanks for the review.

Quoted 5 lines

ov8865_resume() writes the sensor configuration probably to support mid-stream suspend. If we moved the ov8865_sensor_init() and __v4l2_ctrl_handler_setup() calls there behind the state->streaming guard along with ov8865_sw_standby() then I think they could just be called unconditionally in the enable path in ov886_s_stream() without having to store a mode in struct ov8865_state

That works nicely - done in v2, and it addresses Sakari's concern about the extra state as well. Retested on the Surface Book 2: both modes stream fine with the sensor held runtime-active, and the resume handler still reprograms the sensor when resuming mid-stream.

Regards,
Jurison

#10 lore raw

PATCH v2 1/2 media: i2c: ov8865: Drop the runtime PM usage count on stream failure

In reply to Jurison Murati

ov8865_s_stream() takes a runtime PM reference when enabling the stream, but returns without releasing it if ov8865_sw_standby() fails, leaving the reference unbalanced and the sensor powered indefinitely. The same applies to a failure while disabling the stream, in which case the reference acquired at stream start is never dropped.

Drop the reference in a single place, both when disabling the stream and on failure, and only update the streaming state on success.

---
Changes in v2:
- Drop the runtime PM usage count in a single place as suggested by Sakari, which now also covers a failure while disabling the stream.
- Not carrying Dan's Reviewed-by from v1 over since the logic changed.
- Reordered to 1/2 so the next patch can rely on the unified error handling. This was 2/2 in v1.

 drivers/media/i2c/ov8865.c | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)
drivers/media/i2c/ov8865.c +5 -7
index a8586df..5b909a8 100644
@@ -2621,15 +2621,13 @@ static int ov8865_s_stream(struct v4l2_subdev *subdev, int enable)
 	ret = ov8865_sw_standby(sensor, !enable);
 	mutex_unlock(&sensor->mutex);
 
-	if (ret)
-		return ret;
-
-	state->streaming = !!enable;
-
-	if (!enable)
+	if (ret || !enable)
 		pm_runtime_put(sensor->dev);
 
-	return 0;
+	if (!ret)
+		state->streaming = enable;
+
+	return ret;
 }
 
 static const struct v4l2_subdev_video_ops ov8865_subdev_video_ops = {
Signature · 1 line
2.54.0
#11 lore raw

PATCH v2 2/2 media: i2c: ov8865: Program the sensor on stream start

The sensor registers are only written in the runtime PM resume handler; ov8865_set_fmt() merely stores the requested mode, relying on the sensor being runtime suspended between uses so that the next resume applies it.

That assumption breaks when something keeps the sensor powered. On IPU3 platforms, ipu_bridge instantiates the VCM device with a DL_FLAG_PM_RUNTIME device link to the sensor, so a userspace process holding the VCM subdev open (e.g. wireplumber's camera monitor) pins the sensor runtime-active. A subsequent set_fmt() then never reaches the hardware: the sensor keeps streaming the mode programmed on the last resume while the CSI-2 receiver expects the newly negotiated format.

On a Surface Book 2 (IPU3, ov8865 + dw9719 VCM), requesting the 3264x2448 mode while the hardware was left programmed for the 1632x1224 binned mode makes ipu3-cio2 report "frame sync error" and "payload length is 10340352, received 2585088" (exactly one binned frame) for every frame, and the inverse case stalls the stream after a single frame. Camera applications end up displaying one bogus frame forever.

Program the sensor configuration and apply the control values on stream start instead, where the negotiated mode is always current, and only write the configuration in the runtime PM resume handler when resuming with the stream already started.

---
Changes in v2:
- Drop the hw_mode/hw_mbus_code state tracking. Instead, program the sensor unconditionally on stream start and move the configuration writes in the runtime PM resume handler behind the streaming check, as suggested by Dan, keeping mid-stream suspend working while not adding state outside the sub-device state, as requested by Sakari.
- Rebased on top of the runtime PM usage count fix (now 1/2), reusing its unified error path. This was 1/2 in v1.

 drivers/media/i2c/ov8865.c | 34 +++++++++++++++++++++++++---------
 1 file changed, 25 insertions(+), 9 deletions(-)
drivers/media/i2c/ov8865.c +25 -9
index 5b909a8..c6d53c3 100644
@@ -2609,7 +2609,7 @@ static int ov8865_s_stream(struct v4l2_subdev *subdev, int enable)
 {
 	struct ov8865_sensor *sensor = ov8865_subdev_sensor(subdev);
 	struct ov8865_state *state = &sensor->state;
-	int ret;
+	int ret = 0;
 
 	if (enable) {
 		ret = pm_runtime_resume_and_get(sensor->dev);
@@ -2618,7 +2618,23 @@ static int ov8865_s_stream(struct v4l2_subdev *subdev, int enable)
 	}
 
 	mutex_lock(&sensor->mutex);
-	ret = ov8865_sw_standby(sensor, !enable);
+
+	/*
+	 * The sensor may have been kept powered by something else (e.g. the
+	 * VCM's runtime PM device link on IPU3 platforms), in which case
+	 * runtime resume did not run and the hardware may still be
+	 * configured for a previous mode. Always program the negotiated
+	 * configuration on stream start.
+	 */
+	if (enable) {
+		ret = ov8865_sensor_init(sensor);
+		if (!ret)
+			ret = __v4l2_ctrl_handler_setup(&sensor->ctrls.handler);
+	}
+
+	if (!ret)
+		ret = ov8865_sw_standby(sensor, !enable);
+
 	mutex_unlock(&sensor->mutex);
 
 	if (ret || !enable)
@@ -2914,15 +2930,15 @@ static int ov8865_resume(struct device *dev)
 	if (ret)
 		goto complete;
 
-	ret = ov8865_sensor_init(sensor);
-	if (ret)
-		goto error_power;
+	if (state->streaming) {
+		ret = ov8865_sensor_init(sensor);
+		if (ret)
+			goto error_power;
 
-	ret = __v4l2_ctrl_handler_setup(&sensor->ctrls.handler);
-	if (ret)
-		goto error_power;
+		ret = __v4l2_ctrl_handler_setup(&sensor->ctrls.handler);
+		if (ret)
+			goto error_power;
 
-	if (state->streaming) {
 		ret = ov8865_sw_standby(sensor, false);
 		if (ret)
 			goto error_power;
Signature · 1 line
2.54.0
Reviewed-by on v2 1/2
#12 lore raw

Re: [PATCH v2 1/2] media: i2c: ov8865: Drop the runtime PM usage count on stream failure

In reply to Jurison Murati

Hi Jurison, thanks for the revision

Quoted 33 lines · Jurison Murati

ov8865_s_stream() takes a runtime PM reference when enabling the stream, but returns without releasing it if ov8865_sw_standby() fails, leaving the reference unbalanced and the sensor powered indefinitely. The same applies to a failure while disabling the stream, in which case the reference acquired at stream start is never dropped.

Drop the reference in a single place, both when disabling the stream and on failure, and only update the streaming state on success.

---
Changes in v2:
- Drop the runtime PM usage count in a single place as suggested by Sakari, which now also covers a failure while disabling the stream.
- Not carrying Dan's Reviewed-by from v1 over since the logic changed.
- Reordered to 1/2 so the next patch can rely on the unified error handling. This was 2/2 in v1.

  drivers/media/i2c/ov8865.c | 12 +++++-------
  1 file changed, 5 insertions(+), 7 deletions(-)
drivers/media/i2c/ov8865.c +0 -2
index a8586df..5b909a8 100644
@@ -2621,15 +2621,13 @@ static int ov8865_s_stream(struct v4l2_subdev *subdev, int enable)
  	ret = ov8865_sw_standby(sensor, !enable);
  	mutex_unlock(&sensor->mutex);
 
-	if (ret)
-		return ret;

drivers/media/i2c/ov8865.c static int ov8865_s_stream(struct v4l2_subdev *subdev, int enable) · line 2628

Quoted 10 lines
drivers/media/i2c/ov8865.c +3 -5
-
-	state->streaming = !!enable;
-
-	if (!enable)
+	if (ret || !enable)
  		pm_runtime_put(sensor->dev);
 
-	return 0;
+	if (!ret)
+		state->streaming = enable;

I think this is ok, so you can have that tag back:

Quoted 5 lines
drivers/media/i2c/ov8865.c +2 -0
+
+	return ret;
  }
 
  static const struct v4l2_subdev_video_ops ov8865_subdev_video_ops = {