Ensure sensor drivers set V4L2_CTRL_FLAG_MODIFY_LAYOUT for flips
Subsystems (7)
Dave Stevenson
maintainerv2 cover Ensure sensor drivers set V4L2_CTRL_FLAG_MODIFY_LAYOUT for flips
Hi.
I was doing a basic sweep of drivers and noted that these 5 drivers change the Bayer order based on HFLIP and VFLIP. However they don't set the V4L2_CTRL_FLAG_MODIFY_LAYOUT flag on the controls, so userspace has no notion that it needs to check for a changed format.
Add the flag to all the controls.
Dave
Changes from v1:
- Corrected typo in imx355
- Moved setting the flags for ov2680 at Rui's request.
drivers/media/i2c/imx208.c | 4 ++++ drivers/media/i2c/imx319.c | 4 ++++ drivers/media/i2c/imx355.c | 4 ++++ drivers/media/i2c/ov08d10.c | 5 +++++ drivers/media/i2c/ov2680.c | 2 ++ 5 files changed, 19 insertions(+)
Signature · 1 line
2.34.1
Dave Stevenson
maintainerPATCH v2 1/5 media: i2c: ov2680: Set V4L2_CTRL_FLAG_MODIFY_LAYOUT on flips
The driver changes the Bayer order based on the flips, but does not define the control correctly with the V4L2_CTRL_FLAG_MODIFY_LAYOUT flag.
Add the V4L2_CTRL_FLAG_MODIFY_LAYOUT flag.
--- drivers/media/i2c/ov2680.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/media/i2c/ov2680.c +2 -0
@@ -967,6 +967,8 @@ static int ov2680_v4l2_register(struct ov2680_dev *sensor) ctrls->gain->flags |=V4L2_CTRL_FLAG_VOLATILE ; ctrls->exposure->flags |=V4L2_CTRL_FLAG_VOLATILE ; + ctrls->vflip->flags |=V4L2_CTRL_FLAG_MODIFY_LAYOUT ; + ctrls->hflip->flags |=V4L2_CTRL_FLAG_MODIFY_LAYOUT ;v4l2_ctrl_auto_cluster (2 , &ctrls->auto_gain,0 , true);v4l2_ctrl_auto_cluster (2 , &ctrls->auto_exp,1 , true);
Signature · 1 line
2.34.1
Dave Stevenson
maintainerPATCH v2 2/5 media: i2c: imx208: Set V4L2_CTRL_FLAG_MODIFY_LAYOUT on flips
In reply to Dave StevensonThe driver changes the Bayer order based on the flips, but does not define the control correctly with the V4L2_CTRL_FLAG_MODIFY_LAYOUT flag.
Add the V4L2_CTRL_FLAG_MODIFY_LAYOUT flag.
--- drivers/media/i2c/imx208.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/media/i2c/imx208.c +4 -0
@@ -937,8 +937,12 @@ static int imx208_init_controls(struct imx208 *imx208) imx208->hflip =v4l2_ctrl_new_std (ctrl_hdlr, &imx208_ctrl_ops,V4L2_CID_HFLIP ,0 ,1 ,1 ,0 ); +if (imx208->hflip) + imx208->hflip->flags |=V4L2_CTRL_FLAG_MODIFY_LAYOUT ; imx208->vflip =v4l2_ctrl_new_std (ctrl_hdlr, &imx208_ctrl_ops,V4L2_CID_VFLIP ,0 ,1 ,1 ,0 ); +if (imx208->vflip) + imx208->vflip->flags |=V4L2_CTRL_FLAG_MODIFY_LAYOUT ; v4l2_ctrl_new_std(ctrl_hdlr, &imx208_ctrl_ops, V4L2_CID_ANALOGUE_GAIN, IMX208_ANA_GAIN_MIN, IMX208_ANA_GAIN_MAX,
Signature · 1 line
2.34.1
Dave Stevenson
maintainerPATCH v2 3/5 media: i2c: imx319: Set V4L2_CTRL_FLAG_MODIFY_LAYOUT on flips
In reply to Dave StevensonThe driver changes the Bayer order based on the flips, but does not define the control correctly with the V4L2_CTRL_FLAG_MODIFY_LAYOUT flag.
Add the V4L2_CTRL_FLAG_MODIFY_LAYOUT flag.
--- drivers/media/i2c/imx319.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/media/i2c/imx319.c +4 -0
@@ -2328,8 +2328,12 @@ static int imx319_init_controls(struct imx319 *imx319) imx319->hflip =v4l2_ctrl_new_std (ctrl_hdlr, &imx319_ctrl_ops,V4L2_CID_HFLIP ,0 ,1 ,1 ,0 ); +if (imx319->hflip) + imx319->hflip->flags |=V4L2_CTRL_FLAG_MODIFY_LAYOUT ; imx319->vflip =v4l2_ctrl_new_std (ctrl_hdlr, &imx319_ctrl_ops,V4L2_CID_VFLIP ,0 ,1 ,1 ,0 ); +if (imx319->vflip) + imx319->vflip->flags |=V4L2_CTRL_FLAG_MODIFY_LAYOUT ; v4l2_ctrl_new_std(ctrl_hdlr, &imx319_ctrl_ops, V4L2_CID_ANALOGUE_GAIN, IMX319_ANA_GAIN_MIN, IMX319_ANA_GAIN_MAX,
Signature · 1 line
2.34.1
RE: [PATCH v2 3/5] media: i2c: imx319: Set V4L2_CTRL_FLAG_MODIFY_LAYOUT on flips
Stevenson,
Thanks for your patch.
I am wondering how V4L2_CTRL_FLAG_MODIFY_LAYOUT flag was used in current v4l2 ctrl framework, literally it means the v4l2 ctrl will change the buffer layout. From my understanding, such as 90 degrees rotation apparently change the layout. But I am not sure this is also the case for vflip/hflip, user can notice the bayer order update from get_fmt.
Sakari, what do you think?
________________________
BRs,
VTG - Linux&Chrome IPU SW
Bingbu Cao
Quoted 40 lines
-----Original Message-----
From: Dave Stevenson <[email protected]>
Sent: Monday, December 5, 2022 23:22
To: Rui Miguel Silva <[email protected]>; Sakari Ailus <[email protected]>; Cao, Bingbu <[email protected]>; Qiu, Tian Shu <[email protected]>; Su, Jimmy <[email protected]>; linux- [email protected]
Cc: Dave Stevenson <[email protected]>
Subject: [PATCH v2 3/5] media: i2c: imx319: SetV4L2_CTRL_FLAG_MODIFY_LAYOUTon flipsThe driver changes the Bayer order based on the flips, but does not define the control correctly with the
V4L2_CTRL_FLAG_MODIFY_LAYOUTflag.Add the
V4L2_CTRL_FLAG_MODIFY_LAYOUTflag.Signed-off-by Dave Stevenson <[email protected]>--- drivers/media/i2c/imx319.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/media/i2c/imx319.c index renamed from drivers/media/i2c/imx319.c +0
-0
245a18fb40ad..45b1b61b2880100644drivers/media/i2c/imx319.c +4
-1@@ -2328,8 +2328,12 @@ static int imx319_init_controls(struct imx319 *imx319) imx319->hflip =v4l2_ctrl_new_std (ctrl_hdlr, &imx319_ctrl_ops,V4L2_CID_HFLIP ,0 ,1 ,1 ,0 ); +if (imx319->hflip) + imx319->hflip->flags |=V4L2_CTRL_FLAG_MODIFY_LAYOUT ; imx319->vflip =v4l2_ctrl_new_std (ctrl_hdlr, &imx319_ctrl_ops,V4L2_CID_VFLIP ,0 ,1 ,1 ,0 ); +if (imx319->vflip) + imx319->vflip->flags |=V4L2_CTRL_FLAG_MODIFY_LAYOUT ; v4l2_ctrl_new_std(ctrl_hdlr, &imx319_ctrl_ops, V4L2_CID_ANALOGUE_GAIN, IMX319_ANA_GAIN_MIN, IMX319_ANA_GAIN_MAX, --2.34.1
Dave Stevenson
maintainerRe: [PATCH v2 3/5] media: i2c: imx319: Set V4L2_CTRL_FLAG_MODIFY_LAYOUT on flips
Hi
Quoted 10 lines · Bingbu
Stevenson,
Thanks for your patch.
I am wondering how
V4L2_CTRL_FLAG_MODIFY_LAYOUTflag was used in current v4l2 ctrl framework, literally it means the v4l2 ctrl will change the buffer layout. From my understanding, such as 90 degrees rotation apparently change the layout. But I am not sure this is also the case for vflip/hflip, user can notice the bayer order update fromget_fmt.
Documentation at [1]
"3.5.1. Interactions between formats, controls and buffers
...
The set of information needed to interpret the content of a buffer (e.g. the pixel format, the line stride, the tiling orientation or the rotation) is collectively referred to in the rest of this section as the buffer layout."pixel_format is part of the buffer layout.
V4L2_CTRL_FLAG_MODIFY_LAYOUT is telling the userspace that it must call get_fmt after changing the control in order to find out how the format has changed. Without it there is no obligation to call get_fmt, and userspace can legitimately expect the format/layout to be the same.
Not all sensors change the Bayer order with flips (OnSemi sensors in particular tend not to), so you can't make assumptions over the behaviour.
Quoted 1 line
Sakari, what do you think?
Seeing as Sakari has accepted the patches and created a pull request to Mauro including them suggests that this is indeed the correct thing to do.
There is now a unified behaviour across all sensor drivers that change Bayer order due to flips.
libcamera is relying on correct behaviour in order to be able to work out the native Bayer order of the sensor, and that is why I was checking that all mainline drivers were doing the right thing.
Thanks
Dave
Quoted 46 lines
________________________
BRs,
VTG - Linux&Chrome IPU SW
Bingbu CaoQuoted 40 lines
-----Original Message-----
From: Dave Stevenson <[email protected]>
Sent: Monday, December 5, 2022 23:22
To: Rui Miguel Silva <[email protected]>; Sakari Ailus <[email protected]>; Cao, Bingbu <[email protected]>; Qiu, Tian Shu <[email protected]>; Su, Jimmy <[email protected]>; linux- [email protected]
Cc: Dave Stevenson <[email protected]>
Subject: [PATCH v2 3/5] media: i2c: imx319: SetV4L2_CTRL_FLAG_MODIFY_LAYOUTon flipsThe driver changes the Bayer order based on the flips, but does not define the control correctly with the
V4L2_CTRL_FLAG_MODIFY_LAYOUTflag.Add the
V4L2_CTRL_FLAG_MODIFY_LAYOUTflag.Signed-off-by Dave Stevenson <[email protected]>--- drivers/media/i2c/imx319.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/media/i2c/imx319.c index renamed from drivers/media/i2c/imx319.c +0
-0
245a18fb40ad..45b1b61b2880100644drivers/media/i2c/imx319.c +4
-1@@ -2328,8 +2328,12 @@ static int imx319_init_controls(struct imx319 *imx319) imx319->hflip =v4l2_ctrl_new_std (ctrl_hdlr, &imx319_ctrl_ops,V4L2_CID_HFLIP ,0 ,1 ,1 ,0 ); +if (imx319->hflip) + imx319->hflip->flags |=V4L2_CTRL_FLAG_MODIFY_LAYOUT ; imx319->vflip =v4l2_ctrl_new_std (ctrl_hdlr, &imx319_ctrl_ops,V4L2_CID_VFLIP ,0 ,1 ,1 ,0 ); +if (imx319->vflip) + imx319->vflip->flags |=V4L2_CTRL_FLAG_MODIFY_LAYOUT ; v4l2_ctrl_new_std(ctrl_hdlr, &imx319_ctrl_ops, V4L2_CID_ANALOGUE_GAIN, IMX319_ANA_GAIN_MIN, IMX319_ANA_GAIN_MAX, --2.34.1
Cao, Bingbu
Reviewed-byRE: [PATCH v2 3/5] media: i2c: imx319: Set V4L2_CTRL_FLAG_MODIFY_LAYOUT on flips
Stevenson,
Thanks for your explanation.
________________________
BRs,
Bingbu Cao
Quoted 106 lines
-----Original Message-----
From: Dave Stevenson <[email protected]>
Sent: Tuesday, December 6, 2022 19:40
To: Cao, Bingbu <[email protected]>
Cc: Rui Miguel Silva <[email protected]>; Sakari Ailus <[email protected]>; Qiu, Tian Shu <[email protected]>; Su, Jimmy <[email protected]>; [email protected]
Subject: Re: [PATCH v2 3/5] media: i2c: imx319: SetV4L2_CTRL_FLAG_MODIFY_LAYOUTon flipsHi
Quoted 11 lines · Bingbu
Stevenson,
Thanks for your patch.
I am wondering how
V4L2_CTRL_FLAG_MODIFY_LAYOUTflag was used in current
v4l2 ctrl framework, literally it means the v4l2 ctrl will change the buffer layout. From my understanding, such as 90 degrees rotation apparently change the layout. But I am not sure this is also the case for vflip/hflip, user can notice the bayer order update fromget_fmt.Documentation at [1]
"3.5.1. Interactions between formats, controls and buffers ... The set of information needed to interpret the content of a buffer (e.g. the pixel format, the line stride, the tiling orientation or the rotation) is collectively referred to in the rest of this section as the buffer layout."pixel_formatis part of the buffer layout.
V4L2_CTRL_FLAG_MODIFY_LAYOUTis telling the userspace that it must callget_fmtafter changing the control in order to find out how the format has changed. Without it there is no obligation to callget_fmt, and userspace can legitimately expect the format/layout to be the same.
Not all sensors change the Bayer order with flips (OnSemi sensors in particular tend not to), so you can't make assumptions over the behaviour.Quoted 1 line
Sakari, what do you think?
Seeing as Sakari has accepted the patches and created a pull request to Mauro including them suggests that this is indeed the correct thing to do.
There is now a unified behaviour across all sensor drivers that change Bayer order due to flips.
libcamera is relying on correct behaviour in order to be able to work out the native Bayer order of the sensor, and that is why I was checking that all mainline drivers were doing the right thing.Thanks
Dave[1] https://www.kernel.org/doc/html/latest/userspace-
api/media/v4l/buffer.html#interactions-between-formats-controls-and-buffersQuoted 43 lines
________________________
BRs,
VTG - Linux&Chrome IPU SW
Bingbu CaoQuoted 38 lines
-----Original Message-----
From: Dave Stevenson <[email protected]>
Sent: Monday, December 5, 2022 23:22
To: Rui Miguel Silva <[email protected]>; Sakari Ailus <[email protected]>; Cao, Bingbu <[email protected]>; Qiu, Tian Shu <[email protected]>; Su, Jimmy <[email protected]>; linux- [email protected]
Cc: Dave Stevenson <[email protected]>
Subject: [PATCH v2 3/5] media: i2c: imx319: SetV4L2_CTRL_FLAG_MODIFY_LAYOUTon flipsThe driver changes the Bayer order based on the flips, but does not define the control correctly with the
V4L2_CTRL_FLAG_MODIFY_LAYOUTflag.Add the
V4L2_CTRL_FLAG_MODIFY_LAYOUTflag.Signed-off-by Dave Stevenson <[email protected]>--- drivers/media/i2c/imx319.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/media/i2c/imx319.c +0
-0index
245a18fb40ad..45b1b61b2880100644drivers/media/i2c/imx319.c +4
-0@@ -2328,8 +2328,12 @@ static int imx319_init_controls(struct imx319 *imx319) imx319->hflip =v4l2_ctrl_new_std (ctrl_hdlr, &imx319_ctrl_ops,V4L2_CID_HFLIP ,0 ,1 ,1 ,0 ); +if (imx319->hflip) + imx319->hflip->flags |=V4L2_CTRL_FLAG_MODIFY_LAYOUT ; imx319->vflip =v4l2_ctrl_new_std (ctrl_hdlr, &imx319_ctrl_ops,V4L2_CID_VFLIP ,0 ,1 ,1 ,0 ); +if (imx319->vflip) + imx319->vflip->flags |=V4L2_CTRL_FLAG_MODIFY_LAYOUT ;Quoted 1 line
drivers/media/i2c/imx319.c static int imx319_init_controls(struct imx319 · line 2339
Quoted 1 line
drivers/media/i2c/imx319.c +0
-0v4l2_ctrl_new_std(ctrl_hdlr, &imx319_ctrl_ops,
V4L2_CID_ANALOGUE_GAIN,Quoted 4 lines
Quoted 3 lines
IMX319_ANA_GAIN_MIN, IMX319_ANA_GAIN_MAX,--
2.34.1
Dave Stevenson
maintainerPATCH v2 4/5 media: i2c: imx355: Set V4L2_CTRL_FLAG_MODIFY_LAYOUT on flips
In reply to Dave StevensonThe driver changes the Bayer order based on the flips, but does not define the control correctly with the V4L2_CTRL_FLAG_MODIFY_LAYOUT flag.
Add the V4L2_CTRL_FLAG_MODIFY_LAYOUT flag.
--- drivers/media/i2c/imx355.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/media/i2c/imx355.c +4 -0
@@ -1617,8 +1617,12 @@ static int imx355_init_controls(struct imx355 *imx355) imx355->hflip =v4l2_ctrl_new_std (ctrl_hdlr, &imx355_ctrl_ops,V4L2_CID_HFLIP ,0 ,1 ,1 ,0 ); +if (imx355->hflip) + imx355->hflip->flags |=V4L2_CTRL_FLAG_MODIFY_LAYOUT ; imx355->vflip =v4l2_ctrl_new_std (ctrl_hdlr, &imx355_ctrl_ops,V4L2_CID_VFLIP ,0 ,1 ,1 ,0 ); +if (imx355->vflip) + imx355->vflip->flags |=V4L2_CTRL_FLAG_MODIFY_LAYOUT ; v4l2_ctrl_new_std(ctrl_hdlr, &imx355_ctrl_ops, V4L2_CID_ANALOGUE_GAIN, IMX355_ANA_GAIN_MIN, IMX355_ANA_GAIN_MAX,
Signature · 1 line
2.34.1
RE: [PATCH v2 4/5] media: i2c: imx355: Set V4L2_CTRL_FLAG_MODIFY_LAYOUT on flips
________________________
BRs,
VTG - Linux&Chrome IPU SW
Bingbu Cao
Quoted 40 lines
-----Original Message-----
From: Dave Stevenson <[email protected]>
Sent: Monday, December 5, 2022 23:22
To: Rui Miguel Silva <[email protected]>; Sakari Ailus <[email protected]>; Cao, Bingbu <[email protected]>; Qiu, Tian Shu <[email protected]>; Su, Jimmy <[email protected]>; linux- [email protected]
Cc: Dave Stevenson <[email protected]>
Subject: [PATCH v2 4/5] media: i2c: imx355: SetV4L2_CTRL_FLAG_MODIFY_LAYOUTon flipsThe driver changes the Bayer order based on the flips, but does not define the control correctly with the
V4L2_CTRL_FLAG_MODIFY_LAYOUTflag.Add the
V4L2_CTRL_FLAG_MODIFY_LAYOUTflag.Signed-off-by Dave Stevenson <[email protected]>--- drivers/media/i2c/imx355.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/media/i2c/imx355.c index renamed from drivers/media/i2c/imx355.c +0
-0
b46178681c05..25d4dbb6041e100644drivers/media/i2c/imx355.c +4
-1@@ -1617,8 +1617,12 @@ static int imx355_init_controls(struct imx355 *imx355) imx355->hflip =v4l2_ctrl_new_std (ctrl_hdlr, &imx355_ctrl_ops,V4L2_CID_HFLIP ,0 ,1 ,1 ,0 ); +if (imx355->hflip) + imx355->hflip->flags |=V4L2_CTRL_FLAG_MODIFY_LAYOUT ; imx355->vflip =v4l2_ctrl_new_std (ctrl_hdlr, &imx355_ctrl_ops,V4L2_CID_VFLIP ,0 ,1 ,1 ,0 ); +if (imx355->vflip) + imx355->vflip->flags |=V4L2_CTRL_FLAG_MODIFY_LAYOUT ; v4l2_ctrl_new_std(ctrl_hdlr, &imx355_ctrl_ops, V4L2_CID_ANALOGUE_GAIN, IMX355_ANA_GAIN_MIN, IMX355_ANA_GAIN_MAX, --2.34.1
Dave Stevenson
maintainerPATCH v2 5/5 media: i2c: ov08d10: Set V4L2_CTRL_FLAG_MODIFY_LAYOUT on flips
In reply to Dave StevensonThe driver changes the Bayer order based on the flips, but does not define the control correctly with the V4L2_CTRL_FLAG_MODIFY_LAYOUT flag.
Add the V4L2_CTRL_FLAG_MODIFY_LAYOUT flag.
--- drivers/media/i2c/ov08d10.c | 5 +++++ 1 file changed, 5 insertions(+)
drivers/media/i2c/ov08d10.c +5 -0
@@ -990,8 +990,13 @@ static int ov08d10_init_controls(struct ov08d10 *ov08d10) ov08d10->hflip =v4l2_ctrl_new_std (ctrl_hdlr, &ov08d10_ctrl_ops,V4L2_CID_HFLIP ,0 ,1 ,1 ,0 ); +if (ov08d10->hflip) + ov08d10->hflip->flags |=V4L2_CTRL_FLAG_MODIFY_LAYOUT ; ov08d10->vflip =v4l2_ctrl_new_std (ctrl_hdlr, &ov08d10_ctrl_ops,V4L2_CID_VFLIP ,0 ,1 ,1 ,0 ); +if (ov08d10->vflip) + ov08d10->vflip->flags |=V4L2_CTRL_FLAG_MODIFY_LAYOUT ; +if (ctrl_hdlr->error)return ctrl_hdlr->error;
Signature · 1 line
2.34.1