media: i2c: ov8865: Program the mode on stream start if needed
Subsystems (2)
PATCH 1/2 media: i2c: ov8865: Program the mode on stream start if needed
The sensor registers are only written in the runtime PM resume handler; ov8865_set_fmt() merely stores the requested mode, relying on the sensor being runtime suspended between uses so that the next resume applies it.
That assumption breaks when something keeps the sensor powered. On IPU3 platforms, ipu_bridge instantiates the VCM device with a DL_FLAG_PM_RUNTIME device link to the sensor, so a userspace process holding the VCM subdev open (e.g. wireplumber's camera monitor) pins the sensor runtime-active. A subsequent set_fmt() then never reaches the hardware: the sensor keeps streaming the mode programmed on the last resume while the CSI-2 receiver expects the newly negotiated format.
On a Surface Book 2 (IPU3, ov8865 + dw9719 VCM), requesting the 3264x2448 mode while the hardware was left programmed for the 1632x1224 binned mode makes ipu3-cio2 report "frame sync error" and "payload length is 10340352, received 2585088" (exactly one binned frame) for every frame, and the inverse case stalls the stream after a single frame. Camera applications end up displaying one bogus frame forever.
Track the mode actually programmed into the hardware, invalidate it when the sensor is powered off, and reprogram the sensor on stream start whenever it does not match the negotiated mode, re-applying the control values afterwards.
---
drivers/media/i2c/ov8865.c +28 -0
@@ -670,6 +670,10 @@const struct ov8865_mode *mode;u32 mbus_code; +/* Mode currently programmed into the hardware, NULL when unpowered. */ +const struct ov8865_mode *hw_mode; +u32 hw_mbus_code; +bool streaming; };
@@ -2383,6 +2387,9 @@return ret; } + sensor->state.hw_mode = sensor->state.mode; + sensor->state.hw_mbus_code = sensor->state.mbus_code; +return 0 ; }
@@ -2618,6 +2625,25 @@ }mutex_lock (&sensor->mutex); + +/* +* If something else kept the sensor powered (e.g. the VCM's PM +* device link holding it active), runtime resume did not run when +* streaming was requested and the hardware may still be programmed +* for a previous mode. Reprogram it to match the current state. +*/ +if (enable && (state->hw_mode != state->mode || + state->hw_mbus_code != state->mbus_code)) { + ret =ov8865_sensor_init (sensor); +if (!ret) + ret =__v4l2_ctrl_handler_setup (&sensor->ctrls.handler); +if (ret) { +mutex_unlock (&sensor->mutex); +pm_runtime_put (sensor->dev); +return ret; + } + } + ret =ov8865_sw_standby (sensor, !enable);mutex_unlock (&sensor->mutex);
@@ -2895,6 +2921,8 @@ ret =ov8865_sensor_power (sensor, false);if (ret)ov8865_sw_standby (sensor, false); +else + state->hw_mode =NULL ; complete:mutex_unlock (&sensor->mutex);
Signature · 1 line
2.49.0
PATCH 2/2 media: i2c: ov8865: Drop PM runtime reference if stream start fails
ov8865_s_stream() takes a runtime PM reference when enabling the stream, but returns without releasing it if ov8865_sw_standby() fails, leaving the reference unbalanced and the sensor powered indefinitely.
---
drivers/media/i2c/ov8865.c +4 -1
@@ -2647,8 +2647,11 @@ ret =ov8865_sw_standby (sensor, !enable);mutex_unlock (&sensor->mutex); -if (ret) +if (ret) { +if (enable) +pm_runtime_put (sensor->dev);return ret; + } state->streaming = !!enable;
Signature · 1 line
2.49.0
Re: [PATCH 2/2] media: i2c: ov8865: Drop PM runtime reference if stream start fails
Hi Jurison
Quoted 7 lines · Jurison Murati
ov8865_s_stream()takes a runtime PM reference when enabling the stream, but returns without releasing it ifov8865_sw_standby()fails, leaving the reference unbalanced and the sensor powered indefinitely.Signed-off-by Jurison Murati <[email protected]>---
Good spot, thanks:
Quoted 15 lines
drivers/media/i2c/ov8865.c +4
-1@@ -2647,8 +2647,11 @@ ret =ov8865_sw_standby (sensor, !enable);mutex_unlock (&sensor->mutex); -if (ret) +if (ret) { +if (enable) +pm_runtime_put (sensor->dev);return ret; + } state->streaming = !!enable;
Re: [PATCH 2/2] media: i2c: ov8865: Drop PM runtime reference if stream start fails
Hi Dan,
Quoted 3 lines
Good spot, thanks:
Reviewed-by Daniel Scally <[email protected]>
Thanks! Note that in v2 I restructured this per Sakari's suggestion (the runtime PM reference is now also dropped if disabling the stream fails), so I didn't carry your Reviewed-by over - please take another look when you get a chance.
Regards,
Jurison
Sakari Ailus
maintainerRe: [PATCH 2/2] media: i2c: ov8865: Drop PM runtime reference if stream start fails
In reply to Jurison MuratiHi Jurison,
Quoted 11 lines · Jurison Murati
ov8865_s_stream()takes a runtime PM reference when enabling the stream, but returns without releasing it ifov8865_sw_standby()fails, leaving the reference unbalanced and the sensor powered indefinitely.Signed-off-by Jurison Murati <[email protected]>---
drivers/media/i2c/ov8865.c +0
-0@@ -2647,8 +2647,11 @@ ret =ov8865_sw_standby (sensor, !enable);
drivers/media/i2c/ov8865.c line 2656
Quoted 10 lines
drivers/media/i2c/ov8865.c +4
-1mutex_unlock (&sensor->mutex); -if (ret) +if (ret) { +if (enable) +pm_runtime_put (sensor->dev);return ret; + } state->streaming = !!enable;
How about:
if (ret || !enable) pm_runtime_put(sensor->dev); if (!ret) state->streaming = enable; return ret;
Quoted 1 line
The driver should be switched to {enable,disable}_streams but that's another issue altogether.
Signature · 3 lines
Regards, Sakari Ailus
Re: [PATCH 2/2] media: i2c: ov8865: Drop PM runtime reference if stream start fails
Hi Sakari,
Thanks for the review.
Quoted 9 lines
How about:
if (ret || !enable) pm_runtime_put(sensor->dev); if (!ret) state->streaming = enable; return ret;
Done in v2 - that also covers the case where disabling the stream fails, which v1 missed. Since the logic changed I didn't carry Dan's Reviewed-by over. I also moved this patch to 1/2 so the stream start patch can rely on the unified error path.
Quoted 2 lines
The driver should be switched to {enable,disable}_streams but that's another issue altogether.
Noted - I'll leave that for a separate series.
Regards,
Jurison
Sakari Ailus
maintainerHi Jurison,
Quoted 32 lines · Jurison Murati
The sensor registers are only written in the runtime PM resume handler;
ov8865_set_fmt()merely stores the requested mode, relying on the sensor being runtime suspended between uses so that the next resume applies it.That assumption breaks when something keeps the sensor powered. On IPU3 platforms,
ipu_bridgeinstantiates the VCM device with aDL_FLAG_PM_RUNTIMEdevice link to the sensor, so a userspace process holding the VCM subdev open (e.g. wireplumber's camera monitor) pins the sensor runtime-active. A subsequentset_fmt()then never reaches the hardware: the sensor keeps streaming the mode programmed on the last resume while the CSI-2 receiver expects the newly negotiated format.On a Surface Book 2 (IPU3, ov8865 + dw9719 VCM), requesting the 3264x2448 mode while the hardware was left programmed for the 1632x1224 binned mode makes ipu3-cio2 report "frame sync error" and "payload length is 10340352, received 2585088" (exactly one binned frame) for every frame, and the inverse case stalls the stream after a single frame. Camera applications end up displaying one bogus frame forever.
Track the mode actually programmed into the hardware, invalidate it when the sensor is powered off, and reprogram the sensor on stream start whenever it does not match the negotiated mode, re-applying the control values afterwards.
Signed-off-by Jurison Murati <[email protected]>---
drivers/media/i2c/ov8865.c line 674
Quoted 5 lines
drivers/media/i2c/ov8865.c +2
-0@@ -670,6 +670,10 @@const struct ov8865_mode *mode;u32 mbus_code; +/* Mode currently programmed into the hardware, NULL when unpowered. */ +const struct ov8865_mode *hw_mode;
We'd rather like to get rid of state outside sub-device state, not add more of it.
drivers/media/i2c/ov8865.c line 675
Quoted 1 line
drivers/media/i2c/ov8865.c +1
-0+u32 hw_mbus_code;
Same for this one.
Could you simply write the configuration to the sensor when it changes?
Quoted 49 lines
drivers/media/i2c/ov8865.c +25
-0+bool streaming; };@@ -2383,6 +2387,9 @@return ret; } + sensor->state.hw_mode = sensor->state.mode; + sensor->state.hw_mbus_code = sensor->state.mbus_code; +return 0 ; }@@ -2618,6 +2625,25 @@ }mutex_lock (&sensor->mutex); + +/* +* If something else kept the sensor powered (e.g. the VCM's PM +* device link holding it active), runtime resume did not run when +* streaming was requested and the hardware may still be programmed +* for a previous mode. Reprogram it to match the current state. +*/ +if (enable && (state->hw_mode != state->mode || + state->hw_mbus_code != state->mbus_code)) { + ret =ov8865_sensor_init (sensor); +if (!ret) + ret =__v4l2_ctrl_handler_setup (&sensor->ctrls.handler); +if (ret) { +mutex_unlock (&sensor->mutex); +pm_runtime_put (sensor->dev); +return ret; + } + } + ret =ov8865_sw_standby (sensor, !enable);mutex_unlock (&sensor->mutex);@@ -2895,6 +2921,8 @@ ret =ov8865_sensor_power (sensor, false);if (ret)ov8865_sw_standby (sensor, false); +else + state->hw_mode =NULL ; complete:mutex_unlock (&sensor->mutex);
Signature · 3 lines
Regards, Sakari Ailus
Hi Sakari,
Thanks for the review.
Quoted 4 lines
We'd rather like to get rid of state outside sub-device state, not add more of it.
Could you simply write the configuration to the sensor when it changes?
Right, that's nicer. I've done it the way Dan suggested in his reply:
the configuration is now written unconditionally at stream start, and the runtime PM resume handler only writes it when resuming with the stream already started. No new driver state needed.
Sent as v2, retested on the Surface Book 2 in the original failing configuration.
Regards,
Jurison
Hi Jurison, thanks for the patch
Quoted 64 lines · Jurison Murati
The sensor registers are only written in the runtime PM resume handler;
ov8865_set_fmt()merely stores the requested mode, relying on the sensor being runtime suspended between uses so that the next resume applies it.That assumption breaks when something keeps the sensor powered. On IPU3 platforms,
ipu_bridgeinstantiates the VCM device with aDL_FLAG_PM_RUNTIMEdevice link to the sensor, so a userspace process holding the VCM subdev open (e.g. wireplumber's camera monitor) pins the sensor runtime-active. A subsequentset_fmt()then never reaches the hardware: the sensor keeps streaming the mode programmed on the last resume while the CSI-2 receiver expects the newly negotiated format.On a Surface Book 2 (IPU3, ov8865 + dw9719 VCM), requesting the 3264x2448 mode while the hardware was left programmed for the 1632x1224 binned mode makes ipu3-cio2 report "frame sync error" and "payload length is 10340352, received 2585088" (exactly one binned frame) for every frame, and the inverse case stalls the stream after a single frame. Camera applications end up displaying one bogus frame forever.
Track the mode actually programmed into the hardware, invalidate it when the sensor is powered off, and reprogram the sensor on stream start whenever it does not match the negotiated mode, re-applying the control values afterwards.
Signed-off-by Jurison Murati <[email protected]>---
drivers/media/i2c/ov8865.c +15
-0@@ -670,6 +670,10 @@const struct ov8865_mode *mode;u32 mbus_code; +/* Mode currently programmed into the hardware, NULL when unpowered. */ +const struct ov8865_mode *hw_mode; +u32 hw_mbus_code; +bool streaming; };@@ -2383,6 +2387,9 @@return ret; } + sensor->state.hw_mode = sensor->state.mode; + sensor->state.hw_mbus_code = sensor->state.mbus_code; +return 0 ; }@@ -2618,6 +2625,25 @@ }mutex_lock (&sensor->mutex); + +/* +* If something else kept the sensor powered (e.g. the VCM's PM +* device link holding it active), runtime resume did not run when +* streaming was requested and the hardware may still be programmed +* for a previous mode. Reprogram it to match the current state. +*/ + if (enable && (state->hw_mode != state->mode ||
drivers/media/i2c/ov8865.c line 2645
Quoted 10 lines
drivers/media/i2c/ov8865.c +10
-0+ state->hw_mbus_code != state->mbus_code)) { + ret =ov8865_sensor_init (sensor); +if (!ret) + ret =__v4l2_ctrl_handler_setup (&sensor->ctrls.handler); +if (ret) { +mutex_unlock (&sensor->mutex); +pm_runtime_put (sensor->dev); +return ret; + } + }
ov8865_resume() writes the sensor configuration probably to support mid-stream suspend. If we moved the ov8865_sensor_init() and __v4l2_ctrl_handler_setup() calls there behind the state->streaming guard along with ov8865_sw_standby() then I think they could just be called unconditionally in the enable path in ov886_s_stream() without having to store a mode in struct ov8865_state
Thanks
Dan
Quoted 12 lines
drivers/media/i2c/ov8865.c +2
-0ret =ov8865_sw_standby (sensor, !enable);mutex_unlock (&sensor->mutex);@@ -2895,6 +2921,8 @@ ret =ov8865_sensor_power (sensor, false);if (ret)ov8865_sw_standby (sensor, false); +else + state->hw_mode =NULL ; complete:mutex_unlock (&sensor->mutex);
Hi Dan,
Thanks for the review.
Quoted 5 lines
ov8865_resume()writes the sensor configuration probably to support mid-stream suspend. If we moved theov8865_sensor_init()and__v4l2_ctrl_handler_setup()calls there behind thestate->streamingguard along withov8865_sw_standby()then I think they could just be called unconditionally in the enable path inov886_s_stream()without having to store a mode in structov8865_state
That works nicely - done in v2, and it addresses Sakari's concern about the extra state as well. Retested on the Surface Book 2: both modes stream fine with the sensor held runtime-active, and the resume handler still reprograms the sensor when resuming mid-stream.
Regards,
Jurison
PATCH v2 1/2 media: i2c: ov8865: Drop the runtime PM usage count on stream failure
In reply to Jurison Muratiov8865_s_stream() takes a runtime PM reference when enabling the stream, but returns without releasing it if ov8865_sw_standby() fails, leaving the reference unbalanced and the sensor powered indefinitely. The same applies to a failure while disabling the stream, in which case the reference acquired at stream start is never dropped.
Drop the reference in a single place, both when disabling the stream and on failure, and only update the streaming state on success.
---
Changes in v2:
- Drop the runtime PM usage count in a single place as suggested by Sakari, which now also covers a failure while disabling the stream.
- Not carrying Dan's Reviewed-by from v1 over since the logic changed.
- Reordered to 1/2 so the next patch can rely on the unified error handling. This was 2/2 in v1.
drivers/media/i2c/ov8865.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-)
drivers/media/i2c/ov8865.c +5 -7
@@ -2621,15 +2621,13 @@ static int ov8865_s_stream(struct v4l2_subdev *subdev, int enable) ret =ov8865_sw_standby (sensor, !enable);mutex_unlock (&sensor->mutex); -if (ret) -return ret; - - state->streaming = !!enable; - -if (!enable) +if (ret || !enable)pm_runtime_put (sensor->dev); -return 0 ; +if (!ret) + state->streaming = enable; + +return ret; }static const struct v4l2_subdev_video_ops ov8865_subdev_video_ops = {
Signature · 1 line
2.54.0
PATCH v2 2/2 media: i2c: ov8865: Program the sensor on stream start
The sensor registers are only written in the runtime PM resume handler; ov8865_set_fmt() merely stores the requested mode, relying on the sensor being runtime suspended between uses so that the next resume applies it.
That assumption breaks when something keeps the sensor powered. On IPU3 platforms, ipu_bridge instantiates the VCM device with a DL_FLAG_PM_RUNTIME device link to the sensor, so a userspace process holding the VCM subdev open (e.g. wireplumber's camera monitor) pins the sensor runtime-active. A subsequent set_fmt() then never reaches the hardware: the sensor keeps streaming the mode programmed on the last resume while the CSI-2 receiver expects the newly negotiated format.
On a Surface Book 2 (IPU3, ov8865 + dw9719 VCM), requesting the 3264x2448 mode while the hardware was left programmed for the 1632x1224 binned mode makes ipu3-cio2 report "frame sync error" and "payload length is 10340352, received 2585088" (exactly one binned frame) for every frame, and the inverse case stalls the stream after a single frame. Camera applications end up displaying one bogus frame forever.
Program the sensor configuration and apply the control values on stream start instead, where the negotiated mode is always current, and only write the configuration in the runtime PM resume handler when resuming with the stream already started.
---
Changes in v2:
- Drop the hw_mode/hw_mbus_code state tracking. Instead, program the sensor unconditionally on stream start and move the configuration writes in the runtime PM resume handler behind the streaming check, as suggested by Dan, keeping mid-stream suspend working while not adding state outside the sub-device state, as requested by Sakari.
- Rebased on top of the runtime PM usage count fix (now 1/2), reusing its unified error path. This was 1/2 in v1.
drivers/media/i2c/ov8865.c | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-)
drivers/media/i2c/ov8865.c +25 -9
@@ -2609,7 +2609,7 @@ static int ov8865_s_stream(struct v4l2_subdev *subdev, int enable) {struct ov8865_sensor *sensor =ov8865_subdev_sensor (subdev);struct ov8865_state *state = &sensor->state; -int ret; +int ret =0 ;if (enable) { ret =pm_runtime_resume_and_get (sensor->dev);
@@ -2618,7 +2618,23 @@ static int ov8865_s_stream(struct v4l2_subdev *subdev, int enable) }mutex_lock (&sensor->mutex); - ret =ov8865_sw_standby (sensor, !enable); + +/* +* The sensor may have been kept powered by something else (e.g. the +* VCM's runtime PM device link on IPU3 platforms), in which case +* runtime resume did not run and the hardware may still be +* configured for a previous mode. Always program the negotiated +* configuration on stream start. +*/ +if (enable) { + ret =ov8865_sensor_init (sensor); +if (!ret) + ret =__v4l2_ctrl_handler_setup (&sensor->ctrls.handler); + } + +if (!ret) + ret =ov8865_sw_standby (sensor, !enable); +mutex_unlock (&sensor->mutex);if (ret || !enable)
@@ -2914,15 +2930,15 @@ static int ov8865_resume(struct device *dev)if (ret) goto complete; - ret =ov8865_sensor_init (sensor); -if (ret) - goto error_power; +if (state->streaming) { + ret =ov8865_sensor_init (sensor); +if (ret) + goto error_power; - ret =__v4l2_ctrl_handler_setup (&sensor->ctrls.handler); -if (ret) - goto error_power; + ret =__v4l2_ctrl_handler_setup (&sensor->ctrls.handler); +if (ret) + goto error_power; -if (state->streaming) { ret =ov8865_sw_standby (sensor, false);if (ret) goto error_power;
Signature · 1 line
2.54.0
Re: [PATCH v2 1/2] media: i2c: ov8865: Drop the runtime PM usage count on stream failure
In reply to Jurison MuratiHi Jurison, thanks for the revision
Quoted 33 lines · Jurison Murati
ov8865_s_stream()takes a runtime PM reference when enabling the stream, but returns without releasing it ifov8865_sw_standby()fails, leaving the reference unbalanced and the sensor powered indefinitely. The same applies to a failure while disabling the stream, in which case the reference acquired at stream start is never dropped.Drop the reference in a single place, both when disabling the stream and on failure, and only update the streaming state on success.
Signed-off-by Jurison Murati <[email protected]>---
Changes in v2:
- Drop the runtime PM usage count in a single place as suggested by Sakari, which now also covers a failure while disabling the stream.
- Not carrying Dan's Reviewed-by from v1 over since the logic changed.
- Reordered to 1/2 so the next patch can rely on the unified error handling. This was 2/2 in v1.drivers/media/i2c/ov8865.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-)drivers/media/i2c/ov8865.c +0
-2@@ -2621,15 +2621,13 @@ static int ov8865_s_stream(struct v4l2_subdev *subdev, int enable) ret =ov8865_sw_standby (sensor, !enable);mutex_unlock (&sensor->mutex); -if (ret) -return ret;
drivers/media/i2c/ov8865.c static int ov8865_s_stream(struct v4l2_subdev *subdev, int enable) · line 2628
Quoted 10 lines
drivers/media/i2c/ov8865.c +3
-5- - state->streaming = !!enable; - -if (!enable) +if (ret || !enable)pm_runtime_put (sensor->dev); -return 0 ; +if (!ret) + state->streaming = enable;
I think this is ok, so you can have that tag back:
Quoted 5 lines
drivers/media/i2c/ov8865.c +2
-0+ +return ret; }static const struct v4l2_subdev_video_ops ov8865_subdev_video_ops = {