Skip to contentSkip to conversation

Generic line based metadata support, internal pads

PATCH In review Sakari Ailus 5 participants 103 messages Apr 16, 2024 – Sep 22, 2025 Page 2 of 2

Subsystems (5)
lore
Share & tools
lore
author maintainer
#79 lore raw

PATCH v9 36/46 media: uapi: Add media bus code for ov2740 embedded data

In reply to Sakari Ailus

Add a media bus code for ov2740 camera sensor embedded data and document it.

---
 .../media/v4l/subdev-formats.rst              | 70 +++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |  3 +-
 2 files changed, 72 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/v4l/subdev-formats.rst +70 -0
index a04756092238..c99b58cb8c7b 100644
@@ -8596,3 +8596,73 @@ This mbus code are only used for "2-byte simplified tagged data format" (code
 embedded data format codes.
 
 Also see :ref:`CCS driver documentation <media-ccs-routes>`.
+
+Omnivision OV2740 Embedded Data Format
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The Omnivision OV2740 camera sensor produces the following embedded data format,
+indicated by mbus code MEDIA_BUS_FMT_OV2740_EMBEDDED. The format conforms to
+:ref:`CCS embedded data format <MEDIA-BUS-FMT-CCS-EMBEDDED>` up to level 1.
+
+.. flat-table:: Omnivision OV2740 Embedded Data Format. Octets at indices marked
+                reserved or unused have been omitted from the table. The values
+                are big endian byte order.
+    :header-rows: 1
+
+    * - Offset
+      - Size in bits (active bits if not the same as size)
+      - Content description
+    * - 4
+      - 16 (10--0)
+      - Analogue gain
+    * - 6
+      - 16
+      - Coarse integration time
+    * - 10
+      - 8
+      - Dpc correction threshold
+    * - 15
+      - 16
+      - Output image width
+    * - 17
+      - 16
+      - Output image height
+    * - 23
+      - 8
+      - MIPI header revision number (2)
+    * - 31
+      - 8
+      - Vertical (bit 1) and horizontal flip (bit 0)
+    * - 32
+      - 8
+      - Frame duration A
+    * - 33
+      - 8
+      - Frame duration B
+    * - 34
+      - 8
+      - Context count (2)
+    * - 35
+      - 8
+      - Context select
+    * - 54
+      - 8
+      - Data pedestal bits 9--2
+    * - 63
+      - 8
+      - Frame average bits 9--2
+    * - 64
+      - 16
+      - Digital gain red
+    * - 66
+      - 16
+      - Digital gain greenr
+    * - 68
+      - 16
+      - Digital gain blue
+    * - 70
+      - 16
+      - Digital gain greenb
+    * - 89
+      - 8
+      - Frame counter (starts at 1, wraps to 0 after 255)
include/uapi/linux/media-bus-format.h +2 -1
index 03f7e9ab517b..13e68c2ccb61 100644
@@ -183,7 +183,8 @@
 #define MEDIA_BUS_FMT_META_20			0x8006
 #define MEDIA_BUS_FMT_META_24			0x8007
 
-/* Specific metadata formats. Next is 0x9002. */
+/* Specific metadata formats. Next is 0x9003. */
 #define MEDIA_BUS_FMT_CCS_EMBEDDED		0x9001
+#define MEDIA_BUS_FMT_OV2740_EMBEDDED		0x9002
 
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
Signature · 1 line
2.39.2
maintainer
#80 lore raw

Re: [PATCH v9 36/46] media: uapi: Add media bus code for ov2740 embedded data

Hi Sakari,

Thank you for the patch.

Quoted 19 lines · Sakari Ailus

Add a media bus code for ov2740 camera sensor embedded data and document it.

---
 .../media/v4l/subdev-formats.rst              | 70 +++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |  3 +-
 2 files changed, 72 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/v4l/subdev-formats.rst +1 -0
index a04756092238..c99b58cb8c7b 100644
@@ -8596,3 +8596,73 @@ This mbus code are only used for "2-byte simplified tagged data format" (code
 embedded data format codes.
 
 Also see :ref:`CCS driver documentation <media-ccs-routes>`.
+

Documentation/userspace-api/media/v4l/subdev-formats.rst This mbus code are only used for "2-byte simplified tagged data format" (code · line 8609

Quoted 10 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +10 -0
+Omnivision OV2740 Embedded Data Format
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The Omnivision OV2740 camera sensor produces the following embedded data format,
+indicated by mbus code MEDIA_BUS_FMT_OV2740_EMBEDDED. The format conforms to
+:ref:`CCS embedded data format <MEDIA-BUS-FMT-CCS-EMBEDDED>` up to level 1.
+
+.. flat-table:: Omnivision OV2740 Embedded Data Format. Octets at indices marked
+                reserved or unused have been omitted from the table. The values
+                are big endian byte order.

s/are big endian/are in big endian/

Quoted 4 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +4 -0
+    :header-rows: 1
+
+    * - Offset
+      - Size in bits (active bits if not the same as size)

Documentation/userspace-api/media/v4l/subdev-formats.rst This mbus code are only used for "2-byte simplified tagged data format" (code · line 8623

Quoted 10 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +10 -0
+      - Content description
+    * - 4
+      - 16 (10--0)
+      - Analogue gain
+    * - 6
+      - 16
+      - Coarse integration time
+    * - 10
+      - 8
+      - Dpc correction threshold

v8 indicated this contained bits 9:2 of the value, was that wrong, or did it get dropped by mistake ?

Quoted 14 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +14 -0
+    * - 15
+      - 16
+      - Output image width
+    * - 17
+      - 16
+      - Output image height
+    * - 23
+      - 8
+      - MIPI header revision number (2)
+    * - 31
+      - 8
+      - Vertical (bit 1) and horizontal flip (bit 0)
+    * - 32
+      - 8

Documentation/userspace-api/media/v4l/subdev-formats.rst This mbus code are only used for "2-byte simplified tagged data format" (code · line 8647

Quoted 10 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +10 -0
+      - Frame duration A
+    * - 33
+      - 8
+      - Frame duration B
+    * - 34
+      - 8
+      - Context count (2)
+    * - 35
+      - 8
+      - Context select

I'm not sure to understand what those four values are.

Quoted 35 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +21 -0
+    * - 54
+      - 8
+      - Data pedestal bits 9--2
+    * - 63
+      - 8
+      - Frame average bits 9--2
+    * - 64
+      - 16
+      - Digital gain red
+    * - 66
+      - 16
+      - Digital gain greenr
+    * - 68
+      - 16
+      - Digital gain blue
+    * - 70
+      - 16
+      - Digital gain greenb
+    * - 89
+      - 8
+      - Frame counter (starts at 1, wraps to 0 after 255)
include/uapi/linux/media-bus-format.h +2 -1
index 03f7e9ab517b..13e68c2ccb61 100644
@@ -183,7 +183,8 @@
 #define MEDIA_BUS_FMT_META_20			0x8006
 #define MEDIA_BUS_FMT_META_24			0x8007
 
-/* Specific metadata formats. Next is 0x9002. */
+/* Specific metadata formats. Next is 0x9003. */
 #define MEDIA_BUS_FMT_CCS_EMBEDDED		0x9001
+#define MEDIA_BUS_FMT_OV2740_EMBEDDED		0x9002
 
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#81 lore raw

Re: [PATCH v9 36/46] media: uapi: Add media bus code for ov2740 embedded data

Hi Laurent,

Quoted 22 lines · Laurent Pinchart

Hi Sakari,

Thank you for the patch.

Quoted 17 lines · Sakari Ailus

Add a media bus code for ov2740 camera sensor embedded data and document it.

---
 .../media/v4l/subdev-formats.rst              | 70 +++++++++++++++++++
 include/uapi/linux/media-bus-format.h         |  3 +-
 2 files changed, 72 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/v4l/subdev-formats.rst +0 -0
index a04756092238..c99b58cb8c7b 100644
@@ -8596,3 +8596,73 @@ This mbus code are only used for "2-byte simplified tagged data format" (code
 embedded data format codes.
 
Quoted 14 lines

Documentation/userspace-api/media/v4l/subdev-formats.rst This mbus code are only used for "2-byte simplified tagged data format" (code · line 8609

Quoted 12 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +11 -0
 Also see :ref:`CCS driver documentation <media-ccs-routes>`.
+
+Omnivision OV2740 Embedded Data Format
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The Omnivision OV2740 camera sensor produces the following embedded data format,
+indicated by mbus code MEDIA_BUS_FMT_OV2740_EMBEDDED. The format conforms to
+:ref:`CCS embedded data format <MEDIA-BUS-FMT-CCS-EMBEDDED>` up to level 1.
+
+.. flat-table:: Omnivision OV2740 Embedded Data Format. Octets at indices marked
+                reserved or unused have been omitted from the table. The values
+                are big endian byte order.

s/are big endian/are in big endian/

Yes.

Quoted 16 lines
Quoted 5 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +4 -0
+    :header-rows: 1
+
+    * - Offset
+      - Size in bits (active bits if not the same as size)

Documentation/userspace-api/media/v4l/subdev-formats.rst This mbus code are only used for "2-byte simplified tagged data format" (code · line 8623

Quoted 10 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +10 -0
+      - Content description
+    * - 4
+      - 16 (10--0)
+      - Analogue gain
+    * - 6
+      - 16
+      - Coarse integration time
+    * - 10
+      - 8
+      - Dpc correction threshold
Quoted 2 lines

v8 indicated this contained bits 9:2 of the value, was that wrong, or did it get dropped by mistake ?

I'll add it back, it was accidentally lost in register hi/lo part merging.

Quoted 26 lines
Quoted 15 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +14 -0
+    * - 15
+      - 16
+      - Output image width
+    * - 17
+      - 16
+      - Output image height
+    * - 23
+      - 8
+      - MIPI header revision number (2)
+    * - 31
+      - 8
+      - Vertical (bit 1) and horizontal flip (bit 0)
+    * - 32
+      - 8

Documentation/userspace-api/media/v4l/subdev-formats.rst This mbus code are only used for "2-byte simplified tagged data format" (code · line 8647

Quoted 10 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +10 -0
+      - Frame duration A
+    * - 33
+      - 8
+      - Frame duration B
+    * - 34
+      - 8
+      - Context count (2)
+    * - 35
+      - 8
+      - Context select
Quoted 1 line

I'm not sure to understand what those four values are.

This comes from documentation, I'm not sure about them either. I could drop them as well, I don't think they carry any significance anyway.

Quoted 37 lines
Quoted 36 lines
Documentation/userspace-api/media/v4l/subdev-formats.rst +21 -0
+    * - 54
+      - 8
+      - Data pedestal bits 9--2
+    * - 63
+      - 8
+      - Frame average bits 9--2
+    * - 64
+      - 16
+      - Digital gain red
+    * - 66
+      - 16
+      - Digital gain greenr
+    * - 68
+      - 16
+      - Digital gain blue
+    * - 70
+      - 16
+      - Digital gain greenb
+    * - 89
+      - 8
+      - Frame counter (starts at 1, wraps to 0 after 255)
include/uapi/linux/media-bus-format.h +2 -1
index 03f7e9ab517b..13e68c2ccb61 100644
@@ -183,7 +183,8 @@
 #define MEDIA_BUS_FMT_META_20			0x8006
 #define MEDIA_BUS_FMT_META_24			0x8007
 
-/* Specific metadata formats. Next is 0x9002. */
+/* Specific metadata formats. Next is 0x9003. */
 #define MEDIA_BUS_FMT_CCS_EMBEDDED		0x9001
+#define MEDIA_BUS_FMT_OV2740_EMBEDDED		0x9002
 
 #endif /* __LINUX_MEDIA_BUS_FORMAT_H */
Signature · 3 lines
Regards,

Sakari Ailus
author maintainer
#82 lore raw

PATCH v9 37/46 media: ov2740: Fix LINK_FREQ and PIXEL_RATE control value reporting

In reply to Sakari Ailus

The driver dug the supported link frequency up from the V4L2 fwnode endpoint and used it internally, but failed to report this in the LINK_FREQ and PIXEL_RATE controls. Fix this.

Fixes 0677a2d9b735 ("media: ov2740: Add support for 180 MHz link frequency")
Cc [email protected] # for v6.8 and later
---
 drivers/media/i2c/ov2740.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
drivers/media/i2c/ov2740.c +6 -5
index 552935ccb4a9..57906df7be4e 100644
@@ -768,14 +768,15 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
 	cur_mode = ov2740->cur_mode;
 	size = ARRAY_SIZE(link_freq_menu_items);
 
-	ov2740->link_freq = v4l2_ctrl_new_int_menu(ctrl_hdlr, &ov2740_ctrl_ops,
-						   V4L2_CID_LINK_FREQ,
-						   size - 1, 0,
-						   link_freq_menu_items);
+	ov2740->link_freq =
+		v4l2_ctrl_new_int_menu(ctrl_hdlr, &ov2740_ctrl_ops,
+				       V4L2_CID_LINK_FREQ, size - 1,
+				       ov2740->supported_modes->link_freq_index,
+				       link_freq_menu_items);
 	if (ov2740->link_freq)
 		ov2740->link_freq->flags |= V4L2_CTRL_FLAG_READ_ONLY;
 
-	pixel_rate = to_pixel_rate(OV2740_LINK_FREQ_360MHZ_INDEX);
+	pixel_rate = to_pixel_rate(ov2740->supported_modes->link_freq_index);
 	ov2740->pixel_rate = v4l2_ctrl_new_std(ctrl_hdlr, &ov2740_ctrl_ops,
 					       V4L2_CID_PIXEL_RATE, 0,
 					       pixel_rate, 1, pixel_rate);
Signature · 1 line
2.39.2
maintainer Reviewed-by on v9 37/46
#83 lore raw

Re: [PATCH v9 37/46] media: ov2740: Fix LINK_FREQ and PIXEL_RATE control value reporting

Hi Sakari,

Thank you for the patch.

Quoted 1 line · Sakari Ailus

The driver dug the supported link frequency up from the V4L2 fwnode

s/dug/digs/

Quoted 1 line

endpoint and used it internally, but failed to report this in the

s/used/uses/
s/failed/fails/

Quoted 5 lines

LINK_FREQ and PIXEL_RATE controls. Fix this.

Fixes 0677a2d9b735 ("media: ov2740: Add support for 180 MHz link frequency")
Cc [email protected] # for v6.8 and later

You're missing the tags given by Hans and Bingbu. As this patch is unrelated to the rest of the series, it should be split off and merged in v6.10.

Quoted 29 lines
---
 drivers/media/i2c/ov2740.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
drivers/media/i2c/ov2740.c +6 -5
index 552935ccb4a9..57906df7be4e 100644
@@ -768,14 +768,15 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
 	cur_mode = ov2740->cur_mode;
 	size = ARRAY_SIZE(link_freq_menu_items);
 
-	ov2740->link_freq = v4l2_ctrl_new_int_menu(ctrl_hdlr, &ov2740_ctrl_ops,
-						   V4L2_CID_LINK_FREQ,
-						   size - 1, 0,
-						   link_freq_menu_items);
+	ov2740->link_freq =
+		v4l2_ctrl_new_int_menu(ctrl_hdlr, &ov2740_ctrl_ops,
+				       V4L2_CID_LINK_FREQ, size - 1,
+				       ov2740->supported_modes->link_freq_index,
+				       link_freq_menu_items);
 	if (ov2740->link_freq)
 		ov2740->link_freq->flags |= V4L2_CTRL_FLAG_READ_ONLY;
 
-	pixel_rate = to_pixel_rate(OV2740_LINK_FREQ_360MHZ_INDEX);
+	pixel_rate = to_pixel_rate(ov2740->supported_modes->link_freq_index);
 	ov2740->pixel_rate = v4l2_ctrl_new_std(ctrl_hdlr, &ov2740_ctrl_ops,
 					       V4L2_CID_PIXEL_RATE, 0,
 					       pixel_rate, 1, pixel_rate);
Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#84 lore raw

Re: [PATCH v9 37/46] media: ov2740: Fix LINK_FREQ and PIXEL_RATE control value reporting

Hi Laurent,

Quoted 20 lines · Laurent Pinchart

Hi Sakari,

Thank you for the patch.

Quoted 1 line · Sakari Ailus

The driver dug the supported link frequency up from the V4L2 fwnode

s/dug/digs/

Quoted 1 line

endpoint and used it internally, but failed to report this in the

s/used/uses/
s/failed/fails/

Quoted 5 lines

LINK_FREQ and PIXEL_RATE controls. Fix this.

Fixes 0677a2d9b735 ("media: ov2740: Add support for 180 MHz link frequency")
Cc [email protected] # for v6.8 and later
Quoted 5 lines

You're missing the tags given by Hans and Bingbu. As this patch is unrelated to the rest of the series, it should be split off and merged in v6.10.

Thanks. I think I forgot the patch to this branch as well. It's been merged already so all is well.

Signature · 3 lines
Regards,

Sakari Ailus
author maintainer
#85 lore raw

PATCH v9 38/46 media: ov2740: Remove shorthand variables

In reply to Sakari Ailus

Remove two variables in ov2740_init_control() that are used as a shorthand for where the information is really located. Make the code more readable by removing them.

---
 drivers/media/i2c/ov2740.c | 18 +++++++-----------
 1 file changed, 7 insertions(+), 11 deletions(-)
drivers/media/i2c/ov2740.c +7 -11
index 57906df7be4e..196a111516b0 100644
@@ -754,10 +754,8 @@ static const struct v4l2_ctrl_ops ov2740_ctrl_ops = {
 static int ov2740_init_controls(struct ov2740 *ov2740)
 {
 	struct v4l2_ctrl_handler *ctrl_hdlr;
-	const struct ov2740_mode *cur_mode;
 	s64 exposure_max, h_blank, pixel_rate;
 	u32 vblank_min, vblank_max, vblank_default;
-	int size;
 	int ret;
 
 	ctrl_hdlr = &ov2740->ctrl_handler;
@@ -765,12 +763,10 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
 	if (ret)
 		return ret;
 
-	cur_mode = ov2740->cur_mode;
-	size = ARRAY_SIZE(link_freq_menu_items);
-
 	ov2740->link_freq =
 		v4l2_ctrl_new_int_menu(ctrl_hdlr, &ov2740_ctrl_ops,
-				       V4L2_CID_LINK_FREQ, size - 1,
+				       V4L2_CID_LINK_FREQ,
+				       ARRAY_SIZE(link_freq_menu_items) - 1,
 				       ov2740->supported_modes->link_freq_index,
 				       link_freq_menu_items);
 	if (ov2740->link_freq)
@@ -781,14 +777,14 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
 					       V4L2_CID_PIXEL_RATE, 0,
 					       pixel_rate, 1, pixel_rate);
 
-	vblank_min = cur_mode->vts_min - cur_mode->height;
-	vblank_max = cur_mode->vts_max - cur_mode->height;
-	vblank_default = cur_mode->vts_def - cur_mode->height;
+	vblank_min = ov2740->cur_mode->vts_min - ov2740->cur_mode->height;
+	vblank_max = ov2740->cur_mode->vts_max - ov2740->cur_mode->height;
+	vblank_default = ov2740->cur_mode->vts_def - ov2740->cur_mode->height;
 	ov2740->vblank = v4l2_ctrl_new_std(ctrl_hdlr, &ov2740_ctrl_ops,
 					   V4L2_CID_VBLANK, vblank_min,
 					   vblank_max, 1, vblank_default);
 
-	h_blank = cur_mode->hts - cur_mode->width;
+	h_blank = ov2740->cur_mode->hts - ov2740->cur_mode->width;
 	ov2740->hblank = v4l2_ctrl_new_std(ctrl_hdlr, &ov2740_ctrl_ops,
 					   V4L2_CID_HBLANK, h_blank, h_blank, 1,
 					   h_blank);
@@ -801,7 +797,7 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
 	v4l2_ctrl_new_std(ctrl_hdlr, &ov2740_ctrl_ops, V4L2_CID_DIGITAL_GAIN,
 			  OV2740_DGTL_GAIN_MIN, OV2740_DGTL_GAIN_MAX,
 			  OV2740_DGTL_GAIN_STEP, OV2740_DGTL_GAIN_DEFAULT);
-	exposure_max = cur_mode->vts_def - OV2740_EXPOSURE_MAX_MARGIN;
+	exposure_max = ov2740->cur_mode->vts_def - OV2740_EXPOSURE_MAX_MARGIN;
 	ov2740->exposure = v4l2_ctrl_new_std(ctrl_hdlr, &ov2740_ctrl_ops,
 					     V4L2_CID_EXPOSURE,
 					     OV2740_EXPOSURE_MIN, exposure_max,
Signature · 1 line
2.39.2
maintainer Reviewed-by on v9 38/46
#86 lore raw

Re: [PATCH v9 38/46] media: ov2740: Remove shorthand variables

Hi Sakari,

Thank you for the patch.

Quoted 3 lines · Sakari Ailus

Remove two variables in ov2740_init_control() that are used as a shorthand for where the information is really located. Make the code more readable by removing them.

Dropping size is nice. I don't know if removing cur_mode makes the code more readable, but if it does for you,

Quoted 63 lines
---
 drivers/media/i2c/ov2740.c | 18 +++++++-----------
 1 file changed, 7 insertions(+), 11 deletions(-)
drivers/media/i2c/ov2740.c +7 -11
index 57906df7be4e..196a111516b0 100644
@@ -754,10 +754,8 @@ static const struct v4l2_ctrl_ops ov2740_ctrl_ops = {
 static int ov2740_init_controls(struct ov2740 *ov2740)
 {
 	struct v4l2_ctrl_handler *ctrl_hdlr;
-	const struct ov2740_mode *cur_mode;
 	s64 exposure_max, h_blank, pixel_rate;
 	u32 vblank_min, vblank_max, vblank_default;
-	int size;
 	int ret;
 
 	ctrl_hdlr = &ov2740->ctrl_handler;
@@ -765,12 +763,10 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
 	if (ret)
 		return ret;
 
-	cur_mode = ov2740->cur_mode;
-	size = ARRAY_SIZE(link_freq_menu_items);
-
 	ov2740->link_freq =
 		v4l2_ctrl_new_int_menu(ctrl_hdlr, &ov2740_ctrl_ops,
-				       V4L2_CID_LINK_FREQ, size - 1,
+				       V4L2_CID_LINK_FREQ,
+				       ARRAY_SIZE(link_freq_menu_items) - 1,
 				       ov2740->supported_modes->link_freq_index,
 				       link_freq_menu_items);
 	if (ov2740->link_freq)
@@ -781,14 +777,14 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
 					       V4L2_CID_PIXEL_RATE, 0,
 					       pixel_rate, 1, pixel_rate);
 
-	vblank_min = cur_mode->vts_min - cur_mode->height;
-	vblank_max = cur_mode->vts_max - cur_mode->height;
-	vblank_default = cur_mode->vts_def - cur_mode->height;
+	vblank_min = ov2740->cur_mode->vts_min - ov2740->cur_mode->height;
+	vblank_max = ov2740->cur_mode->vts_max - ov2740->cur_mode->height;
+	vblank_default = ov2740->cur_mode->vts_def - ov2740->cur_mode->height;
 	ov2740->vblank = v4l2_ctrl_new_std(ctrl_hdlr, &ov2740_ctrl_ops,
 					   V4L2_CID_VBLANK, vblank_min,
 					   vblank_max, 1, vblank_default);
 
-	h_blank = cur_mode->hts - cur_mode->width;
+	h_blank = ov2740->cur_mode->hts - ov2740->cur_mode->width;
 	ov2740->hblank = v4l2_ctrl_new_std(ctrl_hdlr, &ov2740_ctrl_ops,
 					   V4L2_CID_HBLANK, h_blank, h_blank, 1,
 					   h_blank);
@@ -801,7 +797,7 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
 	v4l2_ctrl_new_std(ctrl_hdlr, &ov2740_ctrl_ops, V4L2_CID_DIGITAL_GAIN,
 			  OV2740_DGTL_GAIN_MIN, OV2740_DGTL_GAIN_MAX,
 			  OV2740_DGTL_GAIN_STEP, OV2740_DGTL_GAIN_DEFAULT);
-	exposure_max = cur_mode->vts_def - OV2740_EXPOSURE_MAX_MARGIN;
+	exposure_max = ov2740->cur_mode->vts_def - OV2740_EXPOSURE_MAX_MARGIN;
 	ov2740->exposure = v4l2_ctrl_new_std(ctrl_hdlr, &ov2740_ctrl_ops,
 					     V4L2_CID_EXPOSURE,
 					     OV2740_EXPOSURE_MIN, exposure_max,
Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#87 lore raw

PATCH v9 39/46 media: ov2740: Switch to {enable,disable}_streams

In reply to Sakari Ailus

Switch from s_stream to enable_streams and disable_streams callbacks.

---
 drivers/media/i2c/ov2740.c | 72 +++++++++++++++++---------------------
 1 file changed, 32 insertions(+), 40 deletions(-)
drivers/media/i2c/ov2740.c +32 -40
index 196a111516b0..a12dc58fa0c0 100644
@@ -916,16 +916,23 @@ static int ov2740_load_otp_data(struct nvm_data *nvm)
 	return ret;
 }
 
-static int ov2740_start_streaming(struct ov2740 *ov2740)
+static int ov2740_enable_streams(struct v4l2_subdev *sd,
+				 struct v4l2_subdev_state *state, u32 pad,
+				 u64 streams_mask)
 {
-	struct i2c_client *client = v4l2_get_subdevdata(&ov2740->sd);
+	struct i2c_client *client = v4l2_get_subdevdata(sd);
+	struct ov2740 *ov2740 = to_ov2740(sd);
 	const struct ov2740_reg_list *reg_list;
 	int link_freq_index;
 	int ret;
 
+	ret = pm_runtime_resume_and_get(&client->dev);
+	if (ret < 0)
+		return ret;
+
 	ret = ov2740_identify_module(ov2740);
 	if (ret)
-		return ret;
+		goto out_pm_put;
 
 	if (ov2740->nvm)
 		ov2740_load_otp_data(ov2740->nvm);
@@ -934,7 +941,7 @@ static int ov2740_start_streaming(struct ov2740 *ov2740)
 	ret = ov2740_write_reg(ov2740, 0x0103, 1, 0x01);
 	if (ret) {
 		dev_err(&client->dev, "failed to reset\n");
-		return ret;
+		goto out_pm_put;
 	}
 
 	usleep_range(10000, 15000);
@@ -944,64 +951,47 @@ static int ov2740_start_streaming(struct ov2740 *ov2740)
 	ret = ov2740_write_reg_list(ov2740, reg_list);
 	if (ret) {
 		dev_err(&client->dev, "failed to set plls\n");
-		return ret;
+		goto out_pm_put;
 	}
 
 	reg_list = &ov2740->cur_mode->reg_list;
 	ret = ov2740_write_reg_list(ov2740, reg_list);
 	if (ret) {
 		dev_err(&client->dev, "failed to set mode\n");
-		return ret;
+		goto out_pm_put;
 	}
 
 	ret = __v4l2_ctrl_handler_setup(ov2740->sd.ctrl_handler);
 	if (ret)
-		return ret;
+		goto out_pm_put;
 
 	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
 			       OV2740_MODE_STREAMING);
-	if (ret)
+	if (ret) {
 		dev_err(&client->dev, "failed to start streaming\n");
+		goto out_pm_put;
+	}
 
-	return ret;
-}
+	return 0;
 
-static void ov2740_stop_streaming(struct ov2740 *ov2740)
-{
-	struct i2c_client *client = v4l2_get_subdevdata(&ov2740->sd);
+out_pm_put:
+	pm_runtime_put(&client->dev);
 
-	if (ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
-			     OV2740_MODE_STANDBY))
-		dev_err(&client->dev, "failed to stop streaming\n");
+	return ret;
 }
 
-static int ov2740_set_stream(struct v4l2_subdev *sd, int enable)
+static int ov2740_disable_streams(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_state *state, u32 pad,
+				  u64 streams_mask)
 {
-	struct ov2740 *ov2740 = to_ov2740(sd);
 	struct i2c_client *client = v4l2_get_subdevdata(sd);
-	struct v4l2_subdev_state *sd_state;
-	int ret = 0;
-
-	sd_state = v4l2_subdev_lock_and_get_active_state(&ov2740->sd);
-
-	if (enable) {
-		ret = pm_runtime_resume_and_get(&client->dev);
-		if (ret < 0)
-			goto out_unlock;
+	struct ov2740 *ov2740 = to_ov2740(sd);
+	int ret;
 
-		ret = ov2740_start_streaming(ov2740);
-		if (ret) {
-			enable = 0;
-			ov2740_stop_streaming(ov2740);
-			pm_runtime_put(&client->dev);
-		}
-	} else {
-		ov2740_stop_streaming(ov2740);
-		pm_runtime_put(&client->dev);
-	}
+	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
+			       OV2740_MODE_STANDBY);
 
-out_unlock:
-	v4l2_subdev_unlock_state(sd_state);
+	pm_runtime_put(&client->dev);
 
 	return ret;
 }
@@ -1086,7 +1076,7 @@ static int ov2740_init_state(struct v4l2_subdev *sd,
 }
 
 static const struct v4l2_subdev_video_ops ov2740_video_ops = {
-	.s_stream = ov2740_set_stream,
+	.s_stream = v4l2_subdev_s_stream_helper,
 };
 
 static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
@@ -1094,6 +1084,8 @@ static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
 	.set_fmt = ov2740_set_format,
 	.enum_mbus_code = ov2740_enum_mbus_code,
 	.enum_frame_size = ov2740_enum_frame_size,
+	.enable_streams = ov2740_enable_streams,
+	.disable_streams = ov2740_disable_streams,
 };
 
 static const struct v4l2_subdev_ops ov2740_subdev_ops = {
Signature · 1 line
2.39.2
author maintainer
#88 lore raw

PATCH v9 40/46 media: ov2740: Track streaming state

In reply to Sakari Ailus

With enable_streams and disable_streams, the driver for a device where streams are not independently started and stopped needs to maintain state information on streams that have been requested to be started. Do that now.

In the future, a helper function in the framework is a desirable way to do this instead.

---
 drivers/media/i2c/ov2740.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
drivers/media/i2c/ov2740.c +14 -0
index a12dc58fa0c0..c29e9b8cde15 100644
@@ -539,6 +539,9 @@ struct ov2740 {
 
 	/* True if the device has been identified */
 	bool identified;
+
+	/* Track streaming state */
+	u8 streaming;
 };
 
 static inline struct ov2740 *to_ov2740(struct v4l2_subdev *subdev)
@@ -926,6 +929,11 @@ static int ov2740_enable_streams(struct v4l2_subdev *sd,
 	int link_freq_index;
 	int ret;
 
+	if (ov2740->streaming) {
+		ov2740->streaming |= streams_mask;
+		return 0;
+	}
+
 	ret = pm_runtime_resume_and_get(&client->dev);
 	if (ret < 0)
 		return ret;
@@ -972,6 +980,8 @@ static int ov2740_enable_streams(struct v4l2_subdev *sd,
 		goto out_pm_put;
 	}
 
+	ov2740->streaming |= streams_mask;
+
 	return 0;
 
 out_pm_put:
@@ -988,6 +998,10 @@ static int ov2740_disable_streams(struct v4l2_subdev *sd,
 	struct ov2740 *ov2740 = to_ov2740(sd);
 	int ret;
 
+	ov2740->streaming &= ~streams_mask;
+	if (ov2740->streaming)
+		return 0;
+
 	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
 			       OV2740_MODE_STANDBY);
 
Signature · 1 line
2.39.2
author maintainer
#89 lore raw

PATCH v9 41/46 media: ov2740: Add support for embedded data

In reply to Sakari Ailus

Add support for embedded data. This introduces two internal pads for pixel and embedded data streams. As the driver supports a single mode only, there's no need for backward compatibility in mode selection.

The embedded data is configured to be placed before the image data whereas after the image data is the default.

---
 drivers/media/i2c/ov2740.c | 156 +++++++++++++++++++++++++++++++++----
 1 file changed, 141 insertions(+), 15 deletions(-)
drivers/media/i2c/ov2740.c +141 -15
index c29e9b8cde15..dc0931308053 100644
@@ -11,6 +11,7 @@
 #include <linux/pm_runtime.h>
 #include <linux/nvmem-provider.h>
 #include <linux/regmap.h>
+#include <media/mipi-csi2.h>
 #include <media/v4l2-ctrls.h>
 #include <media/v4l2-device.h>
 #include <media/v4l2-fwnode.h>
@@ -71,11 +72,31 @@
 #define OV2740_REG_ISP_CTRL00		0x5000
 /* ISP CTRL01 */
 #define OV2740_REG_ISP_CTRL01		0x5001
+
+/* Embedded data line location control */
+#define OV2740_REG_EMBEDDED_FLAG	0x5a08
+#define OV2740_EMBEDDED_FLAG_FOOTER	BIT(2) /* otherwise it's in header */
+#define OV2740_EMBEDDED_FLAG_MYSTERY	BIT(1)
 /* Customer Addresses: 0x7010 - 0x710F */
 #define CUSTOMER_USE_OTP_SIZE		0x100
 /* OTP registers from sensor */
 #define OV2740_REG_OTP_CUSTOMER		0x7010
 
+enum {
+	OV2740_PAD_SOURCE,
+	OV2740_PAD_PIXEL,
+	OV2740_PAD_META,
+	OV2740_NUM_PADS,
+};
+
+enum {
+	OV2740_STREAM_PIXEL,
+	OV2740_STREAM_META,
+};
+
+#define OV2740_META_WIDTH		100U
+#define OV2740_META_HEIGHT		1U
+
 struct nvm_data {
 	struct nvmem_device *nvmem;
 	struct regmap *regmap;
@@ -149,6 +170,7 @@ static const struct ov2740_reg mode_1932x1092_regs_360mhz[] = {
 	{0x3000, 0x00},
 	{0x3018, 0x32},
 	{0x3031, 0x0a},
+	{0x3036, 0x12},
 	{0x3080, 0x08},
 	{0x3083, 0xB4},
 	{0x3103, 0x00},
@@ -253,7 +275,7 @@ static const struct ov2740_reg mode_1932x1092_regs_360mhz[] = {
 	{0x4017, 0x10},
 	{0x4044, 0x02},
 	{0x4304, 0x08},
-	{0x4307, 0x30},
+	{0x4307, 0x31},
 	{0x4320, 0x80},
 	{0x4322, 0x00},
 	{0x4323, 0x00},
@@ -302,6 +324,7 @@ static const struct ov2740_reg mode_1932x1092_regs_180mhz[] = {
 	{0x3000, 0x00},
 	{0x3018, 0x32},	/* 0x32 for 2 lanes, 0x12 for 1 lane */
 	{0x3031, 0x0a},
+	{0x3036, 0x12},
 	{0x3080, 0x08},
 	{0x3083, 0xB4},
 	{0x3103, 0x00},
@@ -406,7 +429,7 @@ static const struct ov2740_reg mode_1932x1092_regs_180mhz[] = {
 	{0x4017, 0x10},
 	{0x4044, 0x02},
 	{0x4304, 0x08},
-	{0x4307, 0x30},
+	{0x4307, 0x31},
 	{0x4320, 0x80},
 	{0x4322, 0x00},
 	{0x4323, 0x00},
@@ -513,7 +536,7 @@ static const struct ov2740_mode supported_modes_180mhz[] = {
 
 struct ov2740 {
 	struct v4l2_subdev sd;
-	struct media_pad pad;
+	struct media_pad pads[OV2740_NUM_PADS];
 	struct v4l2_ctrl_handler ctrl_handler;
 
 	/* V4L2 Controls */
@@ -973,6 +996,11 @@ static int ov2740_enable_streams(struct v4l2_subdev *sd,
 	if (ret)
 		goto out_pm_put;
 
+	ret = ov2740_write_reg(ov2740, OV2740_REG_EMBEDDED_FLAG, 1,
+			       OV2740_EMBEDDED_FLAG_MYSTERY);
+	if (ret)
+		goto out_pm_put;
+
 	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
 			       OV2740_MODE_STREAMING);
 	if (ret) {
@@ -1010,23 +1038,49 @@ static int ov2740_disable_streams(struct v4l2_subdev *sd,
 	return ret;
 }
 
-static int ov2740_set_format(struct v4l2_subdev *sd,
-			     struct v4l2_subdev_state *sd_state,
-			     struct v4l2_subdev_format *fmt)
+static int __ov2740_set_format(struct v4l2_subdev *sd,
+			       struct v4l2_subdev_state *sd_state,
+			       struct v4l2_mbus_framefmt *format,
+			       enum v4l2_subdev_format_whence which,
+			       unsigned int pad, unsigned int stream)
 {
+	struct v4l2_mbus_framefmt *src_pix_fmt, *src_meta_fmt, *pix_fmt,
+		*meta_fmt;
 	struct ov2740 *ov2740 = to_ov2740(sd);
 	const struct ov2740_mode *mode;
 	s32 vblank_def, h_blank;
 
+	/*
+	 * Allow setting format on internal pixel pad as well as the source
+	 * pad's pixel stream (for compatibility).
+	 */
+	if ((pad == OV2740_PAD_SOURCE && stream == OV2740_STREAM_META) ||
+	    pad == OV2740_PAD_META) {
+		*format = *v4l2_subdev_state_get_format(sd_state, pad, stream);
+		return 0;
+	}
+
+	pix_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_PIXEL, 0);
+	meta_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_META, 0);
+	src_pix_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
+						   OV2740_STREAM_PIXEL);
+	src_meta_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
+						    OV2740_STREAM_META);
+
 	mode = v4l2_find_nearest_size(ov2740->supported_modes,
 				      ov2740->supported_modes_count,
 				      width, height,
-				      fmt->format.width, fmt->format.height);
+				      format->width, format->height);
+	ov2740_update_pad_format(mode, pix_fmt);
+	*format = *src_pix_fmt = *pix_fmt;
 
-	ov2740_update_pad_format(mode, &fmt->format);
-	*v4l2_subdev_state_get_format(sd_state, fmt->pad) = fmt->format;
+	meta_fmt->code = MEDIA_BUS_FMT_OV2740_EMBEDDED;
+	meta_fmt->width = OV2740_META_WIDTH;
+	meta_fmt->height = OV2740_META_HEIGHT;
+	*src_meta_fmt = *meta_fmt;
+	src_meta_fmt->code = MEDIA_BUS_FMT_META_10;
 
-	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY)
+	if (which == V4L2_SUBDEV_FORMAT_TRY)
 		return 0;
 
 	ov2740->cur_mode = mode;
@@ -1046,6 +1100,14 @@ static int ov2740_set_format(struct v4l2_subdev *sd,
 	return 0;
 }
 
+static int ov2740_set_format(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_state *sd_state,
+			     struct v4l2_subdev_format *fmt)
+{
+	return __ov2740_set_format(sd, sd_state, &fmt->format, fmt->which,
+				   fmt->pad, fmt->stream);
+}
+
 static int ov2740_enum_mbus_code(struct v4l2_subdev *sd,
 				 struct v4l2_subdev_state *sd_state,
 				 struct v4l2_subdev_mbus_code_enum *code)
@@ -1082,10 +1144,68 @@ static int ov2740_enum_frame_size(struct v4l2_subdev *sd,
 static int ov2740_init_state(struct v4l2_subdev *sd,
 			     struct v4l2_subdev_state *sd_state)
 {
+	struct v4l2_subdev_route routes[] = {
+		{
+			.sink_pad = OV2740_PAD_PIXEL,
+			.source_pad = OV2740_PAD_SOURCE,
+			.source_stream = OV2740_STREAM_PIXEL,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+		}, {
+			.sink_pad = OV2740_PAD_META,
+			.source_pad = OV2740_PAD_SOURCE,
+			.source_stream = OV2740_STREAM_META,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+		},
+	};
+	struct v4l2_subdev_krouting routing = {
+		.routes = routes,
+		.num_routes = ARRAY_SIZE(routes),
+	};
+	struct v4l2_subdev_state *active_state;
+	struct v4l2_mbus_framefmt format = { 0 };
 	struct ov2740 *ov2740 = to_ov2740(sd);
+	int ret;
+
+	ret = v4l2_subdev_set_routing(sd, sd_state, &routing);
+	if (ret)
+		return ret;
+
+	active_state = v4l2_subdev_get_locked_active_state(sd);
+
+	ov2740_update_pad_format(&ov2740->supported_modes[0], &format);
+
+	return __ov2740_set_format(sd, sd_state, &format,
+				   active_state == sd_state ?
+				   V4L2_SUBDEV_FORMAT_ACTIVE :
+				   V4L2_SUBDEV_FORMAT_TRY, OV2740_PAD_PIXEL, 0);
+}
+
+static int ov2740_get_frame_desc(struct v4l2_subdev *sd, unsigned int pad,
+				 struct v4l2_mbus_frame_desc *desc)
+{
+	struct v4l2_mbus_frame_desc_entry *entry = desc->entry;
+	struct v4l2_subdev_state *sd_state;
+	struct v4l2_mbus_framefmt *fmt;
+
+	desc->type = V4L2_MBUS_FRAME_DESC_TYPE_CSI2;
+
+	sd_state = v4l2_subdev_lock_and_get_active_state(sd);
+	fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
+					   OV2740_STREAM_PIXEL);
+	entry->pixelcode = fmt->code;
+	v4l2_subdev_unlock_state(sd_state);
+
+	entry->stream = OV2740_STREAM_PIXEL;
+	entry->bus.csi2.dt = MIPI_CSI2_DT_RAW10;
+	entry++;
+	desc->num_entries++;
+
+	entry->pixelcode = MEDIA_BUS_FMT_META_10;
+	entry->stream = OV2740_STREAM_META;
+	entry->bus.csi2.dt = MIPI_CSI2_DT_EMBEDDED_8B;
+	entry++;
+	desc->num_entries++;
 
-	ov2740_update_pad_format(&ov2740->supported_modes[0],
-				 v4l2_subdev_state_get_format(sd_state, 0));
 	return 0;
 }
 
@@ -1100,6 +1220,7 @@ static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
 	.enum_frame_size = ov2740_enum_frame_size,
 	.enable_streams = ov2740_enable_streams,
 	.disable_streams = ov2740_disable_streams,
+	.get_frame_desc = ov2740_get_frame_desc,
 };
 
 static const struct v4l2_subdev_ops ov2740_subdev_ops = {
@@ -1366,11 +1487,16 @@ static int ov2740_probe(struct i2c_client *client)
 	}
 
 	ov2740->sd.state_lock = ov2740->ctrl_handler.lock;
-	ov2740->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
+	ov2740->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_STREAMS;
 	ov2740->sd.entity.ops = &ov2740_subdev_entity_ops;
 	ov2740->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;
-	ov2740->pad.flags = MEDIA_PAD_FL_SOURCE;
-	ret = media_entity_pads_init(&ov2740->sd.entity, 1, &ov2740->pad);
+	ov2740->pads[OV2740_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
+	ov2740->pads[OV2740_PAD_PIXEL].flags =
+		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
+	ov2740->pads[OV2740_PAD_META].flags =
+		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
+	ret = media_entity_pads_init(&ov2740->sd.entity,
+				     ARRAY_SIZE(ov2740->pads), ov2740->pads);
 	if (ret) {
 		dev_err_probe(dev, ret, "failed to init entity pads\n");
 		goto probe_error_v4l2_ctrl_handler_free;
Signature · 1 line
2.39.2
maintainer
#90 lore raw

Re: [PATCH v9 41/46] media: ov2740: Add support for embedded data

Hi Sakari,

Thank you for the patch.

Quoted 57 lines · Sakari Ailus

Add support for embedded data. This introduces two internal pads for pixel and embedded data streams. As the driver supports a single mode only, there's no need for backward compatibility in mode selection.

The embedded data is configured to be placed before the image data whereas after the image data is the default.

---
 drivers/media/i2c/ov2740.c | 156 +++++++++++++++++++++++++++++++++----
 1 file changed, 141 insertions(+), 15 deletions(-)
drivers/media/i2c/ov2740.c +21 -0
index c29e9b8cde15..dc0931308053 100644
@@ -11,6 +11,7 @@
 #include <linux/pm_runtime.h>
 #include <linux/nvmem-provider.h>
 #include <linux/regmap.h>
+#include <media/mipi-csi2.h>
 #include <media/v4l2-ctrls.h>
 #include <media/v4l2-device.h>
 #include <media/v4l2-fwnode.h>
@@ -71,11 +72,31 @@
 #define OV2740_REG_ISP_CTRL00		0x5000
 /* ISP CTRL01 */
 #define OV2740_REG_ISP_CTRL01		0x5001
+
+/* Embedded data line location control */
+#define OV2740_REG_EMBEDDED_FLAG	0x5a08
+#define OV2740_EMBEDDED_FLAG_FOOTER	BIT(2) /* otherwise it's in header */
+#define OV2740_EMBEDDED_FLAG_MYSTERY	BIT(1)
 /* Customer Addresses: 0x7010 - 0x710F */
 #define CUSTOMER_USE_OTP_SIZE		0x100
 /* OTP registers from sensor */
 #define OV2740_REG_OTP_CUSTOMER		0x7010
 
+enum {
+	OV2740_PAD_SOURCE,
+	OV2740_PAD_PIXEL,
+	OV2740_PAD_META,
+	OV2740_NUM_PADS,
+};
+
+enum {
+	OV2740_STREAM_PIXEL,
+	OV2740_STREAM_META,
+};
+
+#define OV2740_META_WIDTH		100U
+#define OV2740_META_HEIGHT		1U
+
 struct nvm_data {
 	struct nvmem_device *nvmem;
 	struct regmap *regmap;

drivers/media/i2c/ov2740.c static const struct ov2740_reg mode_1932x1092_regs_360mhz[] = { · line 173

Quoted 4 lines
drivers/media/i2c/ov2740.c +1 -0
@@ -149,6 +170,7 @@ static const struct ov2740_reg mode_1932x1092_regs_360mhz[] = {
 	{0x3000, 0x00},
 	{0x3018, 0x32},
 	{0x3031, 0x0a},
+	{0x3036, 0x12},

Please name this register and document the value.

drivers/media/i2c/ov2740.c static const struct ov2740_reg mode_1932x1092_regs_360mhz[] = { · line 278

Quoted 9 lines
drivers/media/i2c/ov2740.c +1 -1
 	{0x3080, 0x08},
 	{0x3083, 0xB4},
 	{0x3103, 0x00},
@@ -253,7 +275,7 @@ static const struct ov2740_reg mode_1932x1092_regs_360mhz[] = {
 	{0x4017, 0x10},
 	{0x4044, 0x02},
 	{0x4304, 0x08},
-	{0x4307, 0x30},
+	{0x4307, 0x31},

Same here.

Quoted 54 lines
drivers/media/i2c/ov2740.c +13 -5
 	{0x4320, 0x80},
 	{0x4322, 0x00},
 	{0x4323, 0x00},
@@ -302,6 +324,7 @@ static const struct ov2740_reg mode_1932x1092_regs_180mhz[] = {
 	{0x3000, 0x00},
 	{0x3018, 0x32},	/* 0x32 for 2 lanes, 0x12 for 1 lane */
 	{0x3031, 0x0a},
+	{0x3036, 0x12},
 	{0x3080, 0x08},
 	{0x3083, 0xB4},
 	{0x3103, 0x00},
@@ -406,7 +429,7 @@ static const struct ov2740_reg mode_1932x1092_regs_180mhz[] = {
 	{0x4017, 0x10},
 	{0x4044, 0x02},
 	{0x4304, 0x08},
-	{0x4307, 0x30},
+	{0x4307, 0x31},
 	{0x4320, 0x80},
 	{0x4322, 0x00},
 	{0x4323, 0x00},
@@ -513,7 +536,7 @@ static const struct ov2740_mode supported_modes_180mhz[] = {
 
 struct ov2740 {
 	struct v4l2_subdev sd;
-	struct media_pad pad;
+	struct media_pad pads[OV2740_NUM_PADS];
 	struct v4l2_ctrl_handler ctrl_handler;
 
 	/* V4L2 Controls */
@@ -973,6 +996,11 @@ static int ov2740_enable_streams(struct v4l2_subdev *sd,
 	if (ret)
 		goto out_pm_put;
 
+	ret = ov2740_write_reg(ov2740, OV2740_REG_EMBEDDED_FLAG, 1,
+			       OV2740_EMBEDDED_FLAG_MYSTERY);
+	if (ret)
+		goto out_pm_put;
+
 	ret = ov2740_write_reg(ov2740, OV2740_REG_MODE_SELECT, 1,
 			       OV2740_MODE_STREAMING);
 	if (ret) {
@@ -1010,23 +1038,49 @@ static int ov2740_disable_streams(struct v4l2_subdev *sd,
 	return ret;
 }
 
-static int ov2740_set_format(struct v4l2_subdev *sd,
-			     struct v4l2_subdev_state *sd_state,
-			     struct v4l2_subdev_format *fmt)
+static int __ov2740_set_format(struct v4l2_subdev *sd,
+			       struct v4l2_subdev_state *sd_state,
+			       struct v4l2_mbus_framefmt *format,
+			       enum v4l2_subdev_format_whence which,
+			       unsigned int pad, unsigned int stream)
 {

drivers/media/i2c/ov2740.c static int ov2740_disable_streams(struct v4l2_subdev *sd, · line 1056

Quoted 10 lines
drivers/media/i2c/ov2740.c +6 -0
+	struct v4l2_mbus_framefmt *src_pix_fmt, *src_meta_fmt, *pix_fmt,
+		*meta_fmt;
 	struct ov2740 *ov2740 = to_ov2740(sd);
 	const struct ov2740_mode *mode;
 	s32 vblank_def, h_blank;
 
+	/*
+	 * Allow setting format on internal pixel pad as well as the source
+	 * pad's pixel stream (for compatibility).
+	 */

As mentioned in the review of v8, the internal image pad represents the pixel array, it should thus report the full pixel array size, and be unaffected by the selected mode. We need to discuss how to expose raw sensor parameters to userspace, which will affect this patch. I don't think it's a good idea to expose a non-standard behaviour to userspace before we agree on how to standardize it.

Quoted 70 lines
drivers/media/i2c/ov2740.c +47 -4
+	if ((pad == OV2740_PAD_SOURCE && stream == OV2740_STREAM_META) ||
+	    pad == OV2740_PAD_META) {
+		*format = *v4l2_subdev_state_get_format(sd_state, pad, stream);
+		return 0;
+	}
+
+	pix_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_PIXEL, 0);
+	meta_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_META, 0);
+	src_pix_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
+						   OV2740_STREAM_PIXEL);
+	src_meta_fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
+						    OV2740_STREAM_META);
+
 	mode = v4l2_find_nearest_size(ov2740->supported_modes,
 				      ov2740->supported_modes_count,
 				      width, height,
-				      fmt->format.width, fmt->format.height);
+				      format->width, format->height);
+	ov2740_update_pad_format(mode, pix_fmt);
+	*format = *src_pix_fmt = *pix_fmt;
 
-	ov2740_update_pad_format(mode, &fmt->format);
-	*v4l2_subdev_state_get_format(sd_state, fmt->pad) = fmt->format;
+	meta_fmt->code = MEDIA_BUS_FMT_OV2740_EMBEDDED;
+	meta_fmt->width = OV2740_META_WIDTH;
+	meta_fmt->height = OV2740_META_HEIGHT;
+	*src_meta_fmt = *meta_fmt;
+	src_meta_fmt->code = MEDIA_BUS_FMT_META_10;
 
-	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY)
+	if (which == V4L2_SUBDEV_FORMAT_TRY)
 		return 0;
 
 	ov2740->cur_mode = mode;
@@ -1046,6 +1100,14 @@ static int ov2740_set_format(struct v4l2_subdev *sd,
 	return 0;
 }
 
+static int ov2740_set_format(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_state *sd_state,
+			     struct v4l2_subdev_format *fmt)
+{
+	return __ov2740_set_format(sd, sd_state, &fmt->format, fmt->which,
+				   fmt->pad, fmt->stream);
+}
+
 static int ov2740_enum_mbus_code(struct v4l2_subdev *sd,
 				 struct v4l2_subdev_state *sd_state,
 				 struct v4l2_subdev_mbus_code_enum *code)
@@ -1082,10 +1144,68 @@ static int ov2740_enum_frame_size(struct v4l2_subdev *sd,
 static int ov2740_init_state(struct v4l2_subdev *sd,
 			     struct v4l2_subdev_state *sd_state)
 {
+	struct v4l2_subdev_route routes[] = {
+		{
+			.sink_pad = OV2740_PAD_PIXEL,
+			.source_pad = OV2740_PAD_SOURCE,
+			.source_stream = OV2740_STREAM_PIXEL,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+		}, {
+			.sink_pad = OV2740_PAD_META,
+			.source_pad = OV2740_PAD_SOURCE,
+			.source_stream = OV2740_STREAM_META,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+		},
+	};
+	struct v4l2_subdev_krouting routing = {
+		.routes = routes,
+		.num_routes = ARRAY_SIZE(routes),
+	};

drivers/media/i2c/ov2740.c static int ov2740_enum_frame_size(struct v4l2_subdev *sd, · line 1173

Quoted 10 lines
drivers/media/i2c/ov2740.c +9 -0
+	struct v4l2_subdev_state *active_state;
+	struct v4l2_mbus_framefmt format = { 0 };
 	struct ov2740 *ov2740 = to_ov2740(sd);
+	int ret;
+
+	ret = v4l2_subdev_set_routing(sd, sd_state, &routing);
+	if (ret)
+		return ret;
+
+	active_state = v4l2_subdev_get_locked_active_state(sd);

There's a lockdep assertion that will trip when initializing any try state.

I'll stop reviewing this as comments on v8 haven't been taken into account. Please revisit them for v10.

Quoted 68 lines
drivers/media/i2c/ov2740.c +43 -5
+
+	ov2740_update_pad_format(&ov2740->supported_modes[0], &format);
+
+	return __ov2740_set_format(sd, sd_state, &format,
+				   active_state == sd_state ?
+				   V4L2_SUBDEV_FORMAT_ACTIVE :
+				   V4L2_SUBDEV_FORMAT_TRY, OV2740_PAD_PIXEL, 0);
+}
+
+static int ov2740_get_frame_desc(struct v4l2_subdev *sd, unsigned int pad,
+				 struct v4l2_mbus_frame_desc *desc)
+{
+	struct v4l2_mbus_frame_desc_entry *entry = desc->entry;
+	struct v4l2_subdev_state *sd_state;
+	struct v4l2_mbus_framefmt *fmt;
+
+	desc->type = V4L2_MBUS_FRAME_DESC_TYPE_CSI2;
+
+	sd_state = v4l2_subdev_lock_and_get_active_state(sd);
+	fmt = v4l2_subdev_state_get_format(sd_state, OV2740_PAD_SOURCE,
+					   OV2740_STREAM_PIXEL);
+	entry->pixelcode = fmt->code;
+	v4l2_subdev_unlock_state(sd_state);
+
+	entry->stream = OV2740_STREAM_PIXEL;
+	entry->bus.csi2.dt = MIPI_CSI2_DT_RAW10;
+	entry++;
+	desc->num_entries++;
+
+	entry->pixelcode = MEDIA_BUS_FMT_META_10;
+	entry->stream = OV2740_STREAM_META;
+	entry->bus.csi2.dt = MIPI_CSI2_DT_EMBEDDED_8B;
+	entry++;
+	desc->num_entries++;
 
-	ov2740_update_pad_format(&ov2740->supported_modes[0],
-				 v4l2_subdev_state_get_format(sd_state, 0));
 	return 0;
 }
 
@@ -1100,6 +1220,7 @@ static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
 	.enum_frame_size = ov2740_enum_frame_size,
 	.enable_streams = ov2740_enable_streams,
 	.disable_streams = ov2740_disable_streams,
+	.get_frame_desc = ov2740_get_frame_desc,
 };
 
 static const struct v4l2_subdev_ops ov2740_subdev_ops = {
@@ -1366,11 +1487,16 @@ static int ov2740_probe(struct i2c_client *client)
 	}
 
 	ov2740->sd.state_lock = ov2740->ctrl_handler.lock;
-	ov2740->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
+	ov2740->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_STREAMS;
 	ov2740->sd.entity.ops = &ov2740_subdev_entity_ops;
 	ov2740->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;
-	ov2740->pad.flags = MEDIA_PAD_FL_SOURCE;
-	ret = media_entity_pads_init(&ov2740->sd.entity, 1, &ov2740->pad);
+	ov2740->pads[OV2740_PAD_SOURCE].flags = MEDIA_PAD_FL_SOURCE;
+	ov2740->pads[OV2740_PAD_PIXEL].flags =
+		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
+	ov2740->pads[OV2740_PAD_META].flags =
+		MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_INTERNAL;
+	ret = media_entity_pads_init(&ov2740->sd.entity,
+				     ARRAY_SIZE(ov2740->pads), ov2740->pads);
 	if (ret) {
 		dev_err_probe(dev, ret, "failed to init entity pads\n");
 		goto probe_error_v4l2_ctrl_handler_free;
Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#91 lore raw

PATCH v9 42/46 media: ov2740: Add generic sensor fwnode properties as controls

In reply to Sakari Ailus

Add generic sensor property information as controĺs by using v4l2_fwnode_device_parse() and v4l2_ctrl_new_fwnode_properties().

---
 drivers/media/i2c/ov2740.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/media/i2c/ov2740.c +6 -0
index dc0931308053..e37d824291fe 100644
@@ -779,6 +779,8 @@ static const struct v4l2_ctrl_ops ov2740_ctrl_ops = {
 
 static int ov2740_init_controls(struct ov2740 *ov2740)
 {
+	struct i2c_client *client = v4l2_get_subdevdata(&ov2740->sd);
+	struct v4l2_fwnode_device_properties props;
 	struct v4l2_ctrl_handler *ctrl_hdlr;
 	s64 exposure_max, h_blank, pixel_rate;
 	u32 vblank_min, vblank_max, vblank_default;
@@ -789,6 +791,10 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
 	if (ret)
 		return ret;
 
+	if (!v4l2_fwnode_device_parse(&client->dev, &props))
+		v4l2_ctrl_new_fwnode_properties(ctrl_hdlr, &ov2740_ctrl_ops,
+						&props);
+
 	ov2740->link_freq =
 		v4l2_ctrl_new_int_menu(ctrl_hdlr, &ov2740_ctrl_ops,
 				       V4L2_CID_LINK_FREQ,
Signature · 1 line
2.39.2
maintainer Reviewed-by on v9 42/46
#92 lore raw

Re: [PATCH v9 42/46] media: ov2740: Add generic sensor fwnode properties as controls

Hi Sakari,

Thank you for the patch.

Quoted 22 lines · Sakari Ailus

Add generic sensor property information as controĺs by using v4l2_fwnode_device_parse() and v4l2_ctrl_new_fwnode_properties().

---
 drivers/media/i2c/ov2740.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/media/i2c/ov2740.c +2 -0
index dc0931308053..e37d824291fe 100644
@@ -779,6 +779,8 @@ static const struct v4l2_ctrl_ops ov2740_ctrl_ops = {
 
 static int ov2740_init_controls(struct ov2740 *ov2740)
 {
+	struct i2c_client *client = v4l2_get_subdevdata(&ov2740->sd);
+	struct v4l2_fwnode_device_properties props;
 	struct v4l2_ctrl_handler *ctrl_hdlr;
 	s64 exposure_max, h_blank, pixel_rate;
 	u32 vblank_min, vblank_max, vblank_default;

drivers/media/i2c/ov2740.c static int ov2740_init_controls(struct ov2740 *ov2740) · line 794

Quoted 4 lines
drivers/media/i2c/ov2740.c +1 -0
@@ -789,6 +791,10 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
 	if (ret)
 		return ret;
 
+	if (!v4l2_fwnode_device_parse(&client->dev, &props))

If you moved the parsing earlier, you could set the right number of controls when initializing the handler. This being said, maybe we should instead try to get rid of the controls count hint to the handler initialization function.

Quoted 6 lines
drivers/media/i2c/ov2740.c +3 -0
+		v4l2_ctrl_new_fwnode_properties(ctrl_hdlr, &ov2740_ctrl_ops,
+						&props);
+
 	ov2740->link_freq =
 		v4l2_ctrl_new_int_menu(ctrl_hdlr, &ov2740_ctrl_ops,
 				       V4L2_CID_LINK_FREQ,
Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#93 lore raw

Re: [PATCH v9 42/46] media: ov2740: Add generic sensor fwnode properties as controls

Hi Laurent,

Quoted 30 lines · Laurent Pinchart

Hi Sakari,

Thank you for the patch.

Quoted 25 lines · Sakari Ailus

Add generic sensor property information as controĺs by using v4l2_fwnode_device_parse() and v4l2_ctrl_new_fwnode_properties().

---
 drivers/media/i2c/ov2740.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/media/i2c/ov2740.c +2 -0
index dc0931308053..e37d824291fe 100644
@@ -779,6 +779,8 @@ static const struct v4l2_ctrl_ops ov2740_ctrl_ops = {
 
 static int ov2740_init_controls(struct ov2740 *ov2740)
 {
+	struct i2c_client *client = v4l2_get_subdevdata(&ov2740->sd);
+	struct v4l2_fwnode_device_properties props;
 	struct v4l2_ctrl_handler *ctrl_hdlr;
 	s64 exposure_max, h_blank, pixel_rate;
 	u32 vblank_min, vblank_max, vblank_default;
@@ -789,6 +791,10 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
 	if (ret)
 		return ret;
 
Quoted 6 lines

drivers/media/i2c/ov2740.c static int ov2740_init_controls(struct ov2740 *ov2740) · line 794

Quoted 1 line
drivers/media/i2c/ov2740.c +1 -0
+	if (!v4l2_fwnode_device_parse(&client->dev, &props))

If you moved the parsing earlier, you could set the right number of controls when initializing the handler. This being said, maybe we should instead try to get rid of the controls count hint to the handler initialization function.

I'm not quite sure how that's related.

But I'll update the number.

Quoted 2 lines

Thanks!

Quoted 8 lines
Quoted 7 lines
drivers/media/i2c/ov2740.c +3 -0
+		v4l2_ctrl_new_fwnode_properties(ctrl_hdlr, &ov2740_ctrl_ops,
+						&props);
+
 	ov2740->link_freq =
 		v4l2_ctrl_new_int_menu(ctrl_hdlr, &ov2740_ctrl_ops,
 				       V4L2_CID_LINK_FREQ,
Signature · 3 lines
Regards,

Sakari Ailus
maintainer
#94 lore raw

Re: [PATCH v9 42/46] media: ov2740: Add generic sensor fwnode properties as controls

Quoted 27 lines · Sakari Ailus
Quoted 26 lines · Laurent Pinchart
Quoted 25 lines · Sakari Ailus

Add generic sensor property information as controĺs by using v4l2_fwnode_device_parse() and v4l2_ctrl_new_fwnode_properties().

---
 drivers/media/i2c/ov2740.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/media/i2c/ov2740.c +2 -0
index dc0931308053..e37d824291fe 100644
@@ -779,6 +779,8 @@ static const struct v4l2_ctrl_ops ov2740_ctrl_ops = {
 
 static int ov2740_init_controls(struct ov2740 *ov2740)
 {
+	struct i2c_client *client = v4l2_get_subdevdata(&ov2740->sd);
+	struct v4l2_fwnode_device_properties props;
 	struct v4l2_ctrl_handler *ctrl_hdlr;
 	s64 exposure_max, h_blank, pixel_rate;
 	u32 vblank_min, vblank_max, vblank_default;
@@ -789,6 +791,10 @@ static int ov2740_init_controls(struct ov2740 *ov2740)
 	if (ret)
 		return ret;
 
Quoted 8 lines
Quoted 6 lines

drivers/media/i2c/ov2740.c static int ov2740_init_controls(struct ov2740 *ov2740) · line 794

Quoted 1 line
drivers/media/i2c/ov2740.c +1 -0
+	if (!v4l2_fwnode_device_parse(&client->dev, &props))

If you moved the parsing earlier, you could set the right number of controls when initializing the handler. This being said, maybe we should instead try to get rid of the controls count hint to the handler initialization function.

I'm not quite sure how that's related.

The move is related because you need to know if v4l2_fwnode_device_parse() succeeded to know how many controls v4l2_ctrl_new_fwnode_properties() will add.

Quoted 12 lines

But I'll update the number.

Quoted 1 line

Thanks!

Quoted 6 lines
Quoted 6 lines
drivers/media/i2c/ov2740.c +3 -0
+		v4l2_ctrl_new_fwnode_properties(ctrl_hdlr, &ov2740_ctrl_ops,
+						&props);
+
 	ov2740->link_freq =
 		v4l2_ctrl_new_int_menu(ctrl_hdlr, &ov2740_ctrl_ops,
 				       V4L2_CID_LINK_FREQ,
Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#95 lore raw

PATCH v9 43/46 media: ov2740: Add support for G_SELECTION IOCTL

In reply to Sakari Ailus

Add support for the G_SELECTION IOCTL in the ov2740 driver.

---
 drivers/media/i2c/ov2740.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
drivers/media/i2c/ov2740.c +31 -0
index e37d824291fe..6e355e986b88 100644
@@ -23,6 +23,11 @@
 #define OV2740_DATA_LANES		2
 #define OV2740_RGB_DEPTH		10
 
+#define OV2740_DUMMY_LINES_PRE		8U
+#define OV2740_DUMMY_LINES_POST		8U
+#define OV2740_ACTIVE_WIDTH		1932U
+#define OV2740_ACTIVE_HEIGHT		1092U
+
 #define OV2740_REG_CHIP_ID		0x300a
 #define OV2740_CHIP_ID			0x2740
 
@@ -1114,6 +1119,31 @@ static int ov2740_set_format(struct v4l2_subdev *sd,
 				   fmt->pad, fmt->stream);
 }
 
+static int ov2740_get_selection(struct v4l2_subdev *sd,
+				struct v4l2_subdev_state *state,
+				struct v4l2_subdev_selection *sel)
+{
+	switch (sel->target) {
+	case V4L2_SEL_TGT_NATIVE_SIZE:
+	case V4L2_SEL_TGT_CROP_BOUNDS:
+		sel->r.top = 0;
+		sel->r.left = 0;
+		sel->r.width = OV2740_ACTIVE_WIDTH;
+		sel->r.height = OV2740_DUMMY_LINES_PRE + OV2740_ACTIVE_HEIGHT +
+			OV2740_DUMMY_LINES_POST;
+		return 0;
+	case V4L2_SEL_TGT_CROP:
+	case V4L2_SEL_TGT_CROP_DEFAULT:
+		sel->r.top = OV2740_DUMMY_LINES_PRE;
+		sel->r.left = 0;
+		sel->r.width = OV2740_ACTIVE_WIDTH;
+		sel->r.height = OV2740_ACTIVE_HEIGHT;
+		return 0;
+	}
+
+	return -EINVAL;
+}
+
 static int ov2740_enum_mbus_code(struct v4l2_subdev *sd,
 				 struct v4l2_subdev_state *sd_state,
 				 struct v4l2_subdev_mbus_code_enum *code)
@@ -1222,6 +1252,7 @@ static const struct v4l2_subdev_video_ops ov2740_video_ops = {
 static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
 	.get_fmt = v4l2_subdev_get_fmt,
 	.set_fmt = ov2740_set_format,
+	.get_selection = ov2740_get_selection,
 	.enum_mbus_code = ov2740_enum_mbus_code,
 	.enum_frame_size = ov2740_enum_frame_size,
 	.enable_streams = ov2740_enable_streams,
Signature · 1 line
2.39.2
maintainer
#96 lore raw

Re: [PATCH v9 43/46] media: ov2740: Add support for G_SELECTION IOCTL

Hi Sakari,

Thank you for the patch.

Quoted 1 line · Sakari Ailus

Add support for the G_SELECTION IOCTL in the ov2740 driver.

We need to first define and document how G_SELECTION should behave for raw sensors.

Quoted 61 lines
---
 drivers/media/i2c/ov2740.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
drivers/media/i2c/ov2740.c +31 -0
index e37d824291fe..6e355e986b88 100644
@@ -23,6 +23,11 @@
 #define OV2740_DATA_LANES		2
 #define OV2740_RGB_DEPTH		10
 
+#define OV2740_DUMMY_LINES_PRE		8U
+#define OV2740_DUMMY_LINES_POST		8U
+#define OV2740_ACTIVE_WIDTH		1932U
+#define OV2740_ACTIVE_HEIGHT		1092U
+
 #define OV2740_REG_CHIP_ID		0x300a
 #define OV2740_CHIP_ID			0x2740
 
@@ -1114,6 +1119,31 @@ static int ov2740_set_format(struct v4l2_subdev *sd,
 				   fmt->pad, fmt->stream);
 }
 
+static int ov2740_get_selection(struct v4l2_subdev *sd,
+				struct v4l2_subdev_state *state,
+				struct v4l2_subdev_selection *sel)
+{
+	switch (sel->target) {
+	case V4L2_SEL_TGT_NATIVE_SIZE:
+	case V4L2_SEL_TGT_CROP_BOUNDS:
+		sel->r.top = 0;
+		sel->r.left = 0;
+		sel->r.width = OV2740_ACTIVE_WIDTH;
+		sel->r.height = OV2740_DUMMY_LINES_PRE + OV2740_ACTIVE_HEIGHT +
+			OV2740_DUMMY_LINES_POST;
+		return 0;
+	case V4L2_SEL_TGT_CROP:
+	case V4L2_SEL_TGT_CROP_DEFAULT:
+		sel->r.top = OV2740_DUMMY_LINES_PRE;
+		sel->r.left = 0;
+		sel->r.width = OV2740_ACTIVE_WIDTH;
+		sel->r.height = OV2740_ACTIVE_HEIGHT;
+		return 0;
+	}
+
+	return -EINVAL;
+}
+
 static int ov2740_enum_mbus_code(struct v4l2_subdev *sd,
 				 struct v4l2_subdev_state *sd_state,
 				 struct v4l2_subdev_mbus_code_enum *code)
@@ -1222,6 +1252,7 @@ static const struct v4l2_subdev_video_ops ov2740_video_ops = {
 static const struct v4l2_subdev_pad_ops ov2740_pad_ops = {
 	.get_fmt = v4l2_subdev_get_fmt,
 	.set_fmt = ov2740_set_format,
+	.get_selection = ov2740_get_selection,
 	.enum_mbus_code = ov2740_enum_mbus_code,
 	.enum_frame_size = ov2740_enum_frame_size,
 	.enable_streams = ov2740_enable_streams,
Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#97 lore raw

PATCH v9 44/46 media: v4l: Add V4L2_SUBDEV_ROUTE_FL_IMMUTABLE sub-device routing flag

In reply to Sakari Ailus

Add a flag to denote immutable routes, V4L2_SUBDEV_ROUTE_FL_IMMUTABLE. Such routes cannot be changed and they're always active.

---
 Documentation/userspace-api/media/v4l/dev-subdev.rst         | 4 +++-
 .../userspace-api/media/v4l/vidioc-subdev-g-routing.rst      | 5 +++++
 include/uapi/linux/v4l2-subdev.h                             | 5 +++++
 3 files changed, 13 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/v4l/dev-subdev.rst +3 -1
index de8dfd4f11a5..5c63c8c24108 100644
@@ -572,7 +572,9 @@ internal pad always has a single stream only (0).
 Routes from an internal sink pad to an external source pad are typically not
 modifiable but they can be activated and deactivated using the
 :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
-on driver capabilities.
+on driver capabilities. The :ref:`V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
+<v4l2-subdev-routing-flags>` flag indicates that the
+``V4L2_SUBDEV_ROUTE_FLAG_ACTIVE`` of the route may not be unset.
 
 Interaction between routes, streams, formats and selections
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst +5 -0
index 88df7bf80b00..c0f6d49cac6d 100644
@@ -150,6 +150,11 @@ Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in
     * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
       - 0x0001
       - The route is enabled. Set by applications.
+    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
+      - 0x0002
+      - The route is immutable. Set by the driver. Indicates that the
+	``V4L2_SUBDEV_ROUTE_FL_ACTIVE`` flag of an immutable route may not be
+	unset.
 
 Return Value
 ============
include/uapi/linux/v4l2-subdev.h +5 -0
index 6a39128d0606..c6f1228d43b1 100644
@@ -204,6 +204,11 @@ struct v4l2_subdev_capability {
  * on a video node.
  */
 #define V4L2_SUBDEV_ROUTE_FL_ACTIVE		(1U << 0)
+/*
+ * Is the route immutable. The ACTIVE flag of an immutable route may not be
+ * changed.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		(1U << 1)
 
 /**
  * struct v4l2_subdev_route - A route inside a subdev
Signature · 1 line
2.39.2
maintainer Reviewed-by on v9 44/46
#98 lore raw

Re: [PATCH v9 44/46] media: v4l: Add V4L2_SUBDEV_ROUTE_FL_IMMUTABLE sub-device routing flag

Hi Sakari,

Thank you for the patch.

Quoted 46 lines · Sakari Ailus

Add a flag to denote immutable routes, V4L2_SUBDEV_ROUTE_FL_IMMUTABLE. Such routes cannot be changed and they're always active.

---
 Documentation/userspace-api/media/v4l/dev-subdev.rst         | 4 +++-
 .../userspace-api/media/v4l/vidioc-subdev-g-routing.rst      | 5 +++++
 include/uapi/linux/v4l2-subdev.h                             | 5 +++++
 3 files changed, 13 insertions(+), 1 deletion(-)
Documentation/userspace-api/media/v4l/dev-subdev.rst +3 -1
index de8dfd4f11a5..5c63c8c24108 100644
@@ -572,7 +572,9 @@ internal pad always has a single stream only (0).
 Routes from an internal sink pad to an external source pad are typically not
 modifiable but they can be activated and deactivated using the
 :ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending
-on driver capabilities.
+on driver capabilities. The :ref:`V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
+<v4l2-subdev-routing-flags>` flag indicates that the
+``V4L2_SUBDEV_ROUTE_FLAG_ACTIVE`` of the route may not be unset.
 
 Interaction between routes, streams, formats and selections
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst +5 -0
index 88df7bf80b00..c0f6d49cac6d 100644
@@ -150,6 +150,11 @@ Also ``VIDIOC_SUBDEV_S_ROUTING`` may return more route than the user provided in
     * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
       - 0x0001
       - The route is enabled. Set by applications.
+    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
+      - 0x0002
+      - The route is immutable. Set by the driver. Indicates that the
+	``V4L2_SUBDEV_ROUTE_FL_ACTIVE`` flag of an immutable route may not be
+	unset.
 
 Return Value
 ============
include/uapi/linux/v4l2-subdev.h +0 -0
index 6a39128d0606..c6f1228d43b1 100644

include/uapi/linux/v4l2-subdev.h struct v4l2_subdev_capability { · line 209

Quoted 6 lines
include/uapi/linux/v4l2-subdev.h +3 -0
@@ -204,6 +204,11 @@ struct v4l2_subdev_capability {
  * on a video node.
  */
 #define V4L2_SUBDEV_ROUTE_FL_ACTIVE		(1U << 0)
+/*
+ * Is the route immutable. The ACTIVE flag of an immutable route may not be
+ * changed.

s/changed/unset/

Quoted 5 lines
include/uapi/linux/v4l2-subdev.h +2 -0
+ */
+#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		(1U << 1)
 
 /**
  * struct v4l2_subdev_route - A route inside a subdev
Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#99 lore raw

PATCH v9 45/46 media: ccs: Add IMMUTABLE route flag

In reply to Sakari Ailus

Add immutable route flag to the routing table. The embedded data stream is always there.

---
 drivers/media/i2c/ccs/ccs-core.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/media/i2c/ccs/ccs-core.c +4 -2
index 0f68e64a4042..785e5f2dc107 100644
@@ -3345,12 +3345,14 @@ static int ccs_src_init_state(struct v4l2_subdev *sd,
 			.sink_pad = CCS_PAD_SINK,
 			.source_pad = CCS_PAD_SRC,
 			.source_stream = CCS_STREAM_PIXEL,
-			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
+				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
 		}, {
 			.sink_pad = CCS_PAD_META,
 			.source_pad = CCS_PAD_SRC,
 			.source_stream = CCS_STREAM_META,
-			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
+				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
 		}
 	};
 	struct v4l2_subdev_krouting routing = {
Signature · 1 line
2.39.2
author maintainer
#100 lore raw

PATCH v9 46/46 media: ov2740: Add IMMUTABLE route flag

In reply to Sakari Ailus

Add immutable route flag to the routing table. The driver does not support disabling the embedded data whereas the sensor itself does.

---
 drivers/media/i2c/ov2740.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/media/i2c/ov2740.c +4 -2
index 6e355e986b88..d6b4533a834a 100644
@@ -1185,12 +1185,14 @@ static int ov2740_init_state(struct v4l2_subdev *sd,
 			.sink_pad = OV2740_PAD_PIXEL,
 			.source_pad = OV2740_PAD_SOURCE,
 			.source_stream = OV2740_STREAM_PIXEL,
-			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
+				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
 		}, {
 			.sink_pad = OV2740_PAD_META,
 			.source_pad = OV2740_PAD_SOURCE,
 			.source_stream = OV2740_STREAM_META,
-			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
+			.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE |
+				 V4L2_SUBDEV_ROUTE_FL_IMMUTABLE,
 		},
 	};
 	struct v4l2_subdev_krouting routing = {
Signature · 1 line
2.39.2
maintainer
#101 lore raw
In reply to Sakari Ailus

Hi Sakari,

Thank you for the patches. This is progressing nicely, and I think we can merge part of this series for v6.10. Patches from 01/46 to 08/46 and from 11/46 to 17/46 are nearly there, just a few of them will need a new (and hopefully final) version. You could additionally merge the CCS-specific patches 18/46 to 24/46 if desired. Some ov2740 rework could go in too.

The rest of the patches (and in particular 09/46 and 10/46) depend on internal pads support, which still needs discussions and additional documentation for raw sensors.

When posting v10, please move 09/46 and 10/46 after 24/46.

Quoted 395 lines · Sakari Ailus

Hi folks,

Here are a few patches to add support generic, line based metadata as well as internal pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.

Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.

Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.

Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.

Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).

The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).

These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.

The internal sink pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.

Effectively internal sink pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable and disable routes from internal sink pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal sink pad.

I've also pushed these here and I'll keep updating the branch, I've also included untested OV2740 patches:

<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>

Questions and comments are most welcome.

Preliminary media-ctl and yavta patches can be found here:

<URL:https://git.retiisi.eu/?p=~sailus/yavta.git;a=shortlog;h=refs/heads/metadata> <URL:https://git.retiisi.eu/?p=~sailus/v4l-utils.git;a=shortlog;h=refs/heads/metadata>

I have used IMX219 as an example on routing in a sensor driver in this version.

The patches are on my master branch <URL:https://git.linuxtv.org/sailus/media_tree.git/log/>.

[1] <URL:https://lore.kernel.org/linux-media/[email protected]/>

since v8:

- Move the patch adding internal pad flag past the routing API reworks, as well as a few other patches, in order to separate the patches to those that could still be merged for v6.10 (routing changes) and those that couldn't (sensor API related). The patch on the edge is "media: uapi:
v4l: subdev: Enable streams API".

- Include Laurent's two patches to address crop API issues wrt. streams.

- Add two patches to prepare for CCS driver rework (media: ccs: Move ccs_pm_get_init function up and media: ccs: Rename out label of ccs_start_streaming).

- Address issues in the ov2740 driver patches (as well as the driver itself), 4 more patches towards the end of the set.

- Improved generic metadata format names, align with other existing formats.

- Improved ov2740 embedded data documentation.

- Reworked streams and camera sensor documentation based on Laurent's comments mainly. In particular, the contradictory concept of internal source pads no longer should exist in the patches.

- Fixed pad numbering in the CCS example.

- Fixed S_ROUTING behaviour when len_routes is too small and when S_ROUTING isn't implemented by the driver.

- Reorder sections in meta-formats.rst alphabetically.

- Add a note per struct fields that certain struct v4l2_subdev_format are zero for metadata mbus codes.

- CCS driver patch cleanups.

- CCS driver metadata width fix for space-efficient embedded data at 16 bpp and over.

- Postpone CCS frame descriptor quirk for now.

- Use MIPI_CSI2_DT_USER_DEFINED(0) instead of a numerical value for compressed data datatype.

since v7:

- Add embedded data support for the ov2740 driver.

- Add three patches on top, to add an IMMUTABLE flag to source streams when they cannot be disabled.

- Improved documentation of len_routes and num_routes arguments of [GS]_ROUTING.

- Remove one inclusion of twice-included media/v4l2-fwnode.h in drivers/media/i2c/ccs/ccs-core.c .

- Add missing forward declaration of ccs_internal_ops in drivers/media/i2c/ccs/ccs-core.c .

since v6:

- Improve embedded data UAPI documentation on camera sensors.

- Improve wording of stream glossary entry.

- Improve internal pad flag documentation.

- Fix definition of "data unit" and remove an extra "only" in INTERNAL pad flag description (1st patch).

- Use IMX219 driver in examples consistently.

- Remove the CSI-2 to parallel bridge from the example to simplify the example.

- Minor rewording of some parts of the routing examples.

- Rebase on unified sub-device state information access functions:
<URL:https://lore.kernel.org/linux-media/[email protected]/T/#t>

- In CCS driver, do not maintain current active configuration in driver's device context struct (apart from mbus codes). Rely on sub-device state locking and clean up the code. (Multiple patches towards the end of the set.)

- Arrange the CCS patches early in the set towards the end of the set.

- Move the patch enabling streams API to the end of the set.

- Rework IOCTL argument copying condition for [GS]_ROUTING).

- Handle copying back routes in S_ROUTING, do not rely on G_ROUTING IOCTL implementation.

- Rebase on metadata preparation patchset v6:
<URL:https://lore.kernel.org/linux-media/[email protected]/T/#t>.

since v5:

- Rebase on new set of preparation patches.

- Switch CCS driver from s_stream to enable_streams/disable_streams. Keep streaming state information --- the sensor remains in streaming state if any of the streams is enabled.

- Fix setting mbus code on embedded data in get_frame_desc() op in the CCS driver.

since v4:

- Add a patch to acquire two sub-device states that may use the same lock.

- Add a patch for CCS driver to remove ccs_get_crop_compose() helper.

- Add a patch for CCS driver moving acquiring and releasing the mutex to the s_stream callback.

- Add a patch for CCS driver to rely on sub-device state locking using a single driver-provided lock.

- Fixed calculating minimum number of routes in copying the routes (thanks, Laurent).

- Moved a label in S_ROUTING handling to make Clang happy (hopefully).

- Fixed setting emb_data_ctrl register for CCS embedded data support.

- Rebase on Laurent's cleanup patches.

- Wrap a few long lines.

- Write in embedded data documentation sensor drivers generally don't allow configuring it.

since v3:

- Separate preparation patches from this set.

- Add a definition for "Data unit", a pixel that is not image data and use it instead in format documentation.

- Fix more numbered lists in dev-subdev.rst.

- Remove a redundant definition for V4L2_META_FMT_GENERIC_CSI2_2_24 --- V4L2_META_FMT_GENERIC_CSI2_12 can be used instead.

- Use "X" instead of "p" to denote padding in format documentation.

- Use IMX219 in examples instead of CCS.

- Document that the generic V4L2 CSI-2 metadata formats use padding defined in CSI-2 spec and packing defined in CCS spec.

- Add patches to align [GS]_ROUTING behaviour with V4L2. This means mainly returning configured routes as part of S_ROUTING as well. "len_routes" field is added to denote the length of the array and having more routes than fits in the array is no longer an error. Also added more reserved fields.

- Added trivial support for S_ROUTING (via G_ROUTING implementation) for use in drivers with static-only routes.

- Added helper functions to obtain mbus format as well as crop and compose rectangles that are streams-independent.

- Added a patch to define generic CSI-2 long packet types.

- Removed MEDIA_BUS_FMT_IS_META() macro. It didn't seem useful in the end.

- Use a single CCS embedded data format. The bit depth can be selected using the meta stream on the source pad.

- Fix mbus code numbers (there were holes due to removed redundant formats).

- Fix generic mbus code documentation (byte was being used instead of bit).

- Fix spelling of "length".

- Added a patch to remove v4l2_subdev_enable_streams_api that disables streams API. This should be merged once libcamera support for streams works nicely.

- Don't use strings in printing frame descriptor flags.

- Warn on string truncation in printing frame descriptor.

since v2:

- Add a better example, with formats.

- Add CCS static data media bus codes.

- Added an example demonstrating the use of internal pads. --- Is the level of detail enough for the purpose?

- Improved documentation.

- Added a macro to tell whether a format is a metadata format. (Documentation could be added.)

- A small ReST syntax fix in the same section.

- Drop leftovers of a patch checking for the INTERNAL_SOURCE flag.

since v1:

- Make the new pad flag just "INTERNAL", requiring either SINK or SOURCE pad flag to accompany it. Removed the union in struct v4l2_subdev_route.

- Add the term "stream" to MC glossary.

- Improved and fixed documentation (according to comments).

- Note these formats are little endian.

- Remove 1X8 from the names of the mbus codes. These formats have generally 8 bits per pixel.

- Fix mbus code numbering (had holes in RFC).

- Add new metadata fields to debug prints.

- Fix a minor documentation build issue.

Laurent Pinchart (2):
media: v4l2-subdev: Fix stream handling for crop API
media: v4l2-subdev: Clearly document that the crop API won't be
extended

Sakari Ailus (44):
media: Documentation: Add "stream" into glossary
media: uapi: Add generic serial metadata mbus formats
media: uapi: Document which mbus format fields are valid for metadata
media: uapi: v4l: Add generic 8-bit metadata format definitions
media: v4l: Support line-based metadata capture
media: Documentation: Additional streams generally don't harm capture
media: Documentation: Document embedded data guidelines for camera
sensors
media: Documentation: v4l: Document internal sink pads
media: Documentation: Document S_ROUTING behaviour
media: v4l: subdev: Add a function to lock two sub-device states, use
it
media: v4l: subdev: Move G_ROUTING handling below S_ROUTING
media: v4l: subdev: Copy argument back to user also for S_ROUTING
media: v4l: subdev: Add len_routes field to struct v4l2_subdev_routing
media: v4l: subdev: Return routes set using S_ROUTING
media: v4l: subdev: Add trivial set_routing support
media: ccs: No need to set streaming to false in power off
media: ccs: Move ccs_pm_get_init function up
media: ccs: Rename out label of ccs_start_streaming
media: ccs: Use {enable,disable}_streams operations
media: ccs: Track streaming state
media: ccs: Move ccs_validate_csi_data_format up
media: ccs: Support frame descriptors
media: uapi: v4l: subdev: Enable streams API
media: mc: Add INTERNAL pad flag
media: uapi: ccs: Add media bus code for MIPI CCS embedded data
media: Documentation: Document non-CCS use of CCS embedded data format
media: Documentation: ccs: Document routing
media: ccs: Add support for embedded data stream
media: ccs: Remove ccs_get_crop_compose helper
media: ccs: Rely on sub-device state locking
media: ccs: Compute binning configuration from sub-device state
media: ccs: Compute scaling configuration from sub-device state
media: ccs: Remove which parameter from ccs_propagate
media: uapi: Add media bus code for ov2740 embedded data
media: ov2740: Fix LINK_FREQ and PIXEL_RATE control value reporting
media: ov2740: Remove shorthand variables
media: ov2740: Switch to {enable,disable}_streams
media: ov2740: Track streaming state
media: ov2740: Add support for embedded data
media: ov2740: Add generic sensor fwnode properties as controls
media: ov2740: Add support for G_SELECTION IOCTL
media: v4l: Add V4L2_SUBDEV_ROUTE_FL_IMMUTABLE sub-device routing flag
media: ccs: Add IMMUTABLE route flag
media: ov2740: Add IMMUTABLE route flag

 .../media/drivers/camera-sensor.rst           |   32 +
 .../userspace-api/media/drivers/ccs.rst       |   38 +-
 .../userspace-api/media/glossary.rst          |   15 +
 .../media/mediactl/media-types.rst            |    9 +
 .../userspace-api/media/v4l/dev-meta.rst      |   21 +
 .../userspace-api/media/v4l/dev-subdev.rst    |  179 ++-
 .../userspace-api/media/v4l/meta-formats.rst  |    3 +-
 .../media/v4l/metafmt-generic.rst             |  328 +++++
 .../media/v4l/subdev-formats.rst              |  374 +++++-
 .../media/v4l/vidioc-enum-fmt.rst             |    7 +
 .../media/v4l/vidioc-subdev-g-crop.rst        |    6 +-
 .../media/v4l/vidioc-subdev-g-routing.rst     |   60 +-
 .../media/videodev2.h.rst.exceptions          |    1 +
 drivers/media/i2c/ccs/ccs-core.c              | 1050 +++++++++++------
 drivers/media/i2c/ccs/ccs.h                   |   27 +-
 drivers/media/i2c/ov2740.c                    |  304 +++--
 drivers/media/mc/mc-entity.c                  |   15 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |   25 +-
 drivers/media/v4l2-core/v4l2-subdev.c         |  118 +-
 include/media/v4l2-subdev.h                   |   42 +
 include/uapi/linux/media-bus-format.h         |   13 +
 include/uapi/linux/media.h                    |    1 +
 include/uapi/linux/v4l2-mediabus.h            |   18 +-
 include/uapi/linux/v4l2-subdev.h              |   18 +-
 include/uapi/linux/videodev2.h                |   18 +
 25 files changed, 2183 insertions(+), 539 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst
Signature · 3 lines
Regards,

Laurent Pinchart
#102 lore raw
In reply to Sakari Ailus

Hi Sakari & all,

Quoted 59 lines · Sakari Ailus

Hi folks,

Here are a few patches to add support generic, line based metadata as well as internal pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining.

Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors.

Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run.

Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach.

Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats).

The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be
beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing).

These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1.

The internal sink pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality.

Quoted 6 lines

Effectively internal sink pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable and disable routes from internal sink pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal sink pad.

I wanted to ask what is the current status and intentions regarding this patch-set?

I am particularly interested in the internal sink pads, as I hope that is what I need for a sensor driver that I am currently working on. It is for an HDR sensor which is sending the long exposure on one virtual channel, and the short exposure on another virtual channel. While adding streams & routes to the sensor driver, which only has one source pad, the routes are invalid on the sink side (still, I can capture on 2 streams, with the routes active by default).

One way to solve that would be to add an internal sink pad. Another would have been to have a route flag, V4L2_SUBDEV_ROUTE_FL_SOURCE was proposed here, but it seems it was not approved:

https://patchwork.linuxtv.org/project/linux-media/patch/[email protected]/#129080

Thanks,

Mirela

Quoted 331 lines

I've also pushed these here and I'll keep updating the branch, I've also included untested OV2740 patches:

<URL:https://git.linuxtv.org/sailus/media_tree.git/log/?h=metadata>

Questions and comments are most welcome.

Preliminary media-ctl and yavta patches can be found here:

<URL:https://git.retiisi.eu/?p=~sailus/yavta.git;a=shortlog;h=refs/heads/metadata> <URL:https://git.retiisi.eu/?p=~sailus/v4l-utils.git;a=shortlog;h=refs/heads/metadata>

I have used IMX219 as an example on routing in a sensor driver in this version.

The patches are on my master branch <URL:https://git.linuxtv.org/sailus/media_tree.git/log/>.

[1] <URL:https://lore.kernel.org/linux-media/[email protected]/>

since v8:

- Move the patch adding internal pad flag past the routing API reworks, as well as a few other patches, in order to separate the patches to those that could still be merged for v6.10 (routing changes) and those that couldn't (sensor API related). The patch on the edge is "media: uapi:
v4l: subdev: Enable streams API".

- Include Laurent's two patches to address crop API issues wrt. streams.

- Add two patches to prepare for CCS driver rework (media: ccs: Move ccs_pm_get_init function up and media: ccs: Rename out label of ccs_start_streaming).

- Address issues in the ov2740 driver patches (as well as the driver itself), 4 more patches towards the end of the set.

- Improved generic metadata format names, align with other existing formats.

- Improved ov2740 embedded data documentation.

- Reworked streams and camera sensor documentation based on Laurent's comments mainly. In particular, the contradictory concept of internal source pads no longer should exist in the patches.

- Fixed pad numbering in the CCS example.

- Fixed S_ROUTING behaviour when len_routes is too small and when S_ROUTING isn't implemented by the driver.

- Reorder sections in meta-formats.rst alphabetically.

- Add a note per struct fields that certain struct v4l2_subdev_format are zero for metadata mbus codes.

- CCS driver patch cleanups.

- CCS driver metadata width fix for space-efficient embedded data at 16 bpp and over.

- Postpone CCS frame descriptor quirk for now.

- Use MIPI_CSI2_DT_USER_DEFINED(0) instead of a numerical value for compressed data datatype.

since v7:

- Add embedded data support for the ov2740 driver.

- Add three patches on top, to add an IMMUTABLE flag to source streams when they cannot be disabled.

- Improved documentation of len_routes and num_routes arguments of [GS]_ROUTING.

- Remove one inclusion of twice-included media/v4l2-fwnode.h in drivers/media/i2c/ccs/ccs-core.c .

- Add missing forward declaration of ccs_internal_ops in drivers/media/i2c/ccs/ccs-core.c .

since v6:

- Improve embedded data UAPI documentation on camera sensors.

- Improve wording of stream glossary entry.

- Improve internal pad flag documentation.

- Fix definition of "data unit" and remove an extra "only" in INTERNAL pad flag description (1st patch).

- Use IMX219 driver in examples consistently.

- Remove the CSI-2 to parallel bridge from the example to simplify the example.

- Minor rewording of some parts of the routing examples.

- Rebase on unified sub-device state information access functions:

   <URL:https://lore.kernel.org/linux-media/[email protected]/T/#t>

- In CCS driver, do not maintain current active configuration in driver's device context struct (apart from mbus codes). Rely on sub-device state locking and clean up the code. (Multiple patches towards the end of the set.)

- Arrange the CCS patches early in the set towards the end of the set.

- Move the patch enabling streams API to the end of the set.

- Rework IOCTL argument copying condition for [GS]_ROUTING).

- Handle copying back routes in S_ROUTING, do not rely on G_ROUTING IOCTL implementation.

- Rebase on metadata preparation patchset v6:

   <URL:https://lore.kernel.org/linux-media/[email protected]/T/#t>.

since v5:

- Rebase on new set of preparation patches.

- Switch CCS driver from s_stream to enable_streams/disable_streams. Keep streaming state information --- the sensor remains in streaming state if any of the streams is enabled.

- Fix setting mbus code on embedded data in get_frame_desc() op in the CCS driver.

since v4:

- Add a patch to acquire two sub-device states that may use the same lock.

- Add a patch for CCS driver to remove ccs_get_crop_compose() helper.

- Add a patch for CCS driver moving acquiring and releasing the mutex to the s_stream callback.

- Add a patch for CCS driver to rely on sub-device state locking using a single driver-provided lock.

- Fixed calculating minimum number of routes in copying the routes (thanks, Laurent).

- Moved a label in S_ROUTING handling to make Clang happy (hopefully).

- Fixed setting emb_data_ctrl register for CCS embedded data support.

- Rebase on Laurent's cleanup patches.

- Wrap a few long lines.

- Write in embedded data documentation sensor drivers generally don't allow configuring it.

since v3:

- Separate preparation patches from this set.

- Add a definition for "Data unit", a pixel that is not image data and use it instead in format documentation.

- Fix more numbered lists in dev-subdev.rst.

- Remove a redundant definition for V4L2_META_FMT_GENERIC_CSI2_2_24 --- V4L2_META_FMT_GENERIC_CSI2_12 can be used instead.

- Use "X" instead of "p" to denote padding in format documentation.

- Use IMX219 in examples instead of CCS.

- Document that the generic V4L2 CSI-2 metadata formats use padding defined in CSI-2 spec and packing defined in CCS spec.

- Add patches to align [GS]_ROUTING behaviour with V4L2. This means mainly returning configured routes as part of S_ROUTING as well. "len_routes" field is added to denote the length of the array and having more routes than fits in the array is no longer an error. Also added more reserved fields.

- Added trivial support for S_ROUTING (via G_ROUTING implementation) for use in drivers with static-only routes.

- Added helper functions to obtain mbus format as well as crop and compose rectangles that are streams-independent.

- Added a patch to define generic CSI-2 long packet types.

- Removed MEDIA_BUS_FMT_IS_META() macro. It didn't seem useful in the end.

- Use a single CCS embedded data format. The bit depth can be selected using the meta stream on the source pad.

- Fix mbus code numbers (there were holes due to removed redundant formats).

- Fix generic mbus code documentation (byte was being used instead of bit).

- Fix spelling of "length".

- Added a patch to remove v4l2_subdev_enable_streams_api that disables streams API. This should be merged once libcamera support for streams works nicely.

- Don't use strings in printing frame descriptor flags.

- Warn on string truncation in printing frame descriptor.

since v2:

- Add a better example, with formats.

- Add CCS static data media bus codes.

- Added an example demonstrating the use of internal pads. --- Is the level of detail enough for the purpose?

- Improved documentation.

- Added a macro to tell whether a format is a metadata format. (Documentation could be added.)

- A small ReST syntax fix in the same section.

- Drop leftovers of a patch checking for the INTERNAL_SOURCE flag.

since v1:

- Make the new pad flag just "INTERNAL", requiring either SINK or SOURCE pad flag to accompany it. Removed the union in struct v4l2_subdev_route.

- Add the term "stream" to MC glossary.

- Improved and fixed documentation (according to comments).

- Note these formats are little endian.

- Remove 1X8 from the names of the mbus codes. These formats have generally 8 bits per pixel.

- Fix mbus code numbering (had holes in RFC).

- Add new metadata fields to debug prints.

- Fix a minor documentation build issue.

Laurent Pinchart (2):
media: v4l2-subdev: Fix stream handling for crop API
media: v4l2-subdev: Clearly document that the crop API won't be
extended

Sakari Ailus (44):
media: Documentation: Add "stream" into glossary
media: uapi: Add generic serial metadata mbus formats
media: uapi: Document which mbus format fields are valid for metadata
media: uapi: v4l: Add generic 8-bit metadata format definitions
media: v4l: Support line-based metadata capture
media: Documentation: Additional streams generally don't harm capture
media: Documentation: Document embedded data guidelines for camera
sensors
media: Documentation: v4l: Document internal sink pads
media: Documentation: Document S_ROUTING behaviour
media: v4l: subdev: Add a function to lock two sub-device states, use
it
media: v4l: subdev: Move G_ROUTING handling below S_ROUTING
media: v4l: subdev: Copy argument back to user also for S_ROUTING
media: v4l: subdev: Add len_routes field to struct v4l2_subdev_routing
media: v4l: subdev: Return routes set using S_ROUTING
media: v4l: subdev: Add trivial set_routing support
media: ccs: No need to set streaming to false in power off
media: ccs: Move ccs_pm_get_init function up
media: ccs: Rename out label of ccs_start_streaming
media: ccs: Use {enable,disable}_streams operations
media: ccs: Track streaming state
media: ccs: Move ccs_validate_csi_data_format up
media: ccs: Support frame descriptors
media: uapi: v4l: subdev: Enable streams API
media: mc: Add INTERNAL pad flag
media: uapi: ccs: Add media bus code for MIPI CCS embedded data
media: Documentation: Document non-CCS use of CCS embedded data format
media: Documentation: ccs: Document routing
media: ccs: Add support for embedded data stream
media: ccs: Remove ccs_get_crop_compose helper
media: ccs: Rely on sub-device state locking
media: ccs: Compute binning configuration from sub-device state
media: ccs: Compute scaling configuration from sub-device state
media: ccs: Remove which parameter from ccs_propagate
media: uapi: Add media bus code for ov2740 embedded data
media: ov2740: Fix LINK_FREQ and PIXEL_RATE control value reporting
media: ov2740: Remove shorthand variables
media: ov2740: Switch to {enable,disable}_streams
media: ov2740: Track streaming state
media: ov2740: Add support for embedded data
media: ov2740: Add generic sensor fwnode properties as controls
media: ov2740: Add support for G_SELECTION IOCTL
media: v4l: Add V4L2_SUBDEV_ROUTE_FL_IMMUTABLE sub-device routing flag
media: ccs: Add IMMUTABLE route flag
media: ov2740: Add IMMUTABLE route flag

  .../media/drivers/camera-sensor.rst           |   32 +
  .../userspace-api/media/drivers/ccs.rst       |   38 +-
  .../userspace-api/media/glossary.rst          |   15 +
  .../media/mediactl/media-types.rst            |    9 +
  .../userspace-api/media/v4l/dev-meta.rst      |   21 +
  .../userspace-api/media/v4l/dev-subdev.rst    |  179 ++-
  .../userspace-api/media/v4l/meta-formats.rst  |    3 +-
  .../media/v4l/metafmt-generic.rst             |  328 +++++
  .../media/v4l/subdev-formats.rst              |  374 +++++-
  .../media/v4l/vidioc-enum-fmt.rst             |    7 +
  .../media/v4l/vidioc-subdev-g-crop.rst        |    6 +-
  .../media/v4l/vidioc-subdev-g-routing.rst     |   60 +-
  .../media/videodev2.h.rst.exceptions          |    1 +
  drivers/media/i2c/ccs/ccs-core.c              | 1050 +++++++++++------
  drivers/media/i2c/ccs/ccs.h                   |   27 +-
  drivers/media/i2c/ov2740.c                    |  304 +++--
  drivers/media/mc/mc-entity.c                  |   15 +-
  drivers/media/v4l2-core/v4l2-ioctl.c          |   25 +-
  drivers/media/v4l2-core/v4l2-subdev.c         |  118 +-
  include/media/v4l2-subdev.h                   |   42 +
  include/uapi/linux/media-bus-format.h         |   13 +
  include/uapi/linux/media.h                    |    1 +
  include/uapi/linux/v4l2-mediabus.h            |   18 +-
  include/uapi/linux/v4l2-subdev.h              |   18 +-
  include/uapi/linux/videodev2.h                |   18 +
  25 files changed, 2183 insertions(+), 539 deletions(-)

create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst