i.MX6: Support 16-bit BT.1120 video input

PATCH Superseded Krzysztof Hałasa 3 participants 5 messages Oct 6, 2021 – Oct 8, 2021

Subsystems (6)

DRM drivers Media (V4L/DVB) Arm i.MX and Layerscape Staging +2 more subsystems

lore

Share & tools

lore

Krzysztof Hałasa

author

2021-10-06 06:13 UTC #0 lore raw

PATCH 1/1 i.MX6: Support 16-bit BT.1120 video input

Confirmed to work with ADV7610 HDMI receiver.

Signed-off-by Krzysztof Hałasa <[email protected]>

drivers/gpu/ipu-v3/ipu-csi.c +26 -5

index 658c173bebdf..2893b68f1f7a 100644
@@ -261,10 +261,24 @@ static int mbus_code_to_bus_cfg(struct ipu_csi_bus_config *cfg, u32 mbus_code,
 		cfg->data_width = IPU_CSI_DATA_WIDTH_8;
 		break;
 	case MEDIA_BUS_FMT_UYVY8_1X16:
+		if (mbus_type == V4L2_MBUS_BT656) {
+			cfg->data_fmt = CSI_SENS_CONF_DATA_FMT_YUV422_UYVY;
+			cfg->data_width = IPU_CSI_DATA_WIDTH_8;
+		} else {
+			cfg->data_fmt = CSI_SENS_CONF_DATA_FMT_BAYER;
+			cfg->data_width = IPU_CSI_DATA_WIDTH_16;
+		}
+		cfg->mipi_dt = MIPI_DT_YUV422;
+		break;
 	case MEDIA_BUS_FMT_YUYV8_1X16:
-		cfg->data_fmt = CSI_SENS_CONF_DATA_FMT_BAYER;
+		if (mbus_type == V4L2_MBUS_BT656) {
+			cfg->data_fmt = CSI_SENS_CONF_DATA_FMT_YUV422_YUYV;
+			cfg->data_width = IPU_CSI_DATA_WIDTH_8;
+		} else {
+			cfg->data_fmt = CSI_SENS_CONF_DATA_FMT_BAYER;
+			cfg->data_width = IPU_CSI_DATA_WIDTH_16;
+		}
 		cfg->mipi_dt = MIPI_DT_YUV422;
-		cfg->data_width = IPU_CSI_DATA_WIDTH_16;
 		break;
 	case MEDIA_BUS_FMT_SBGGR8_1X8:
 	case MEDIA_BUS_FMT_SGBRG8_1X8:

@@ -352,7 +366,7 @@ static int fill_csi_bus_cfg(struct ipu_csi_bus_config *csicfg,
 			    const struct v4l2_mbus_config *mbus_cfg,
 			    const struct v4l2_mbus_framefmt *mbus_fmt)
 {
-	int ret;
+	int ret, is_bt1120;
 
 	memset(csicfg, 0, sizeof(*csicfg));

@@ -373,11 +387,18 @@ static int fill_csi_bus_cfg(struct ipu_csi_bus_config *csicfg,
 		break;
 	case V4L2_MBUS_BT656:
 		csicfg->ext_vsync = 0;
+		/* UYVY10_1X20 etc. should be supported as well */
+		is_bt1120 = mbus_fmt->code == MEDIA_BUS_FMT_UYVY8_1X16 ||
+			mbus_fmt->code == MEDIA_BUS_FMT_YUYV8_1X16;
 		if (V4L2_FIELD_HAS_BOTH(mbus_fmt->field) ||
 		    mbus_fmt->field == V4L2_FIELD_ALTERNATE)
-			csicfg->clk_mode = IPU_CSI_CLK_MODE_CCIR656_INTERLACED;
+			csicfg->clk_mode = is_bt1120 ?
+				IPU_CSI_CLK_MODE_CCIR1120_INTERLACED_SDR :
+				IPU_CSI_CLK_MODE_CCIR656_INTERLACED;
 		else
-			csicfg->clk_mode = IPU_CSI_CLK_MODE_CCIR656_PROGRESSIVE;
+			csicfg->clk_mode = is_bt1120 ?
+				IPU_CSI_CLK_MODE_CCIR1120_PROGRESSIVE_SDR :
+				IPU_CSI_CLK_MODE_CCIR656_PROGRESSIVE;
 		break;
 	case V4L2_MBUS_CSI2_DPHY:
 		/*

drivers/staging/media/imx/imx-media-csi.c +5 -0

index 45f9d797b9da..ba93512f8c71 100644
@@ -139,6 +139,8 @@ static inline bool is_parallel_16bit_bus(struct v4l2_fwnode_endpoint *ep)
  * Check for conditions that require the IPU to handle the
  * data internally as generic data, aka passthrough mode:
  * - raw bayer media bus formats, or
+ * - BT.656 and BT.1120 (8/10-bit YUV422) data can always be processed
+ *   on-the-fly (converted to YUV420)
  * - the CSI is receiving from a 16-bit parallel bus, or
  * - the CSI is receiving from an 8-bit parallel bus and the incoming
  *   media bus format is other than UYVY8_2X8/YUYV8_2X8.

@@ -147,6 +149,9 @@ static inline bool requires_passthrough(struct v4l2_fwnode_endpoint *ep,
 					struct v4l2_mbus_framefmt *infmt,
 					const struct imx_media_pixfmt *incc)
 {
+	if (ep->bus_type == V4L2_MBUS_BT656) // including BT.1120
+		return 0;
+
 	return incc->bayer || is_parallel_16bit_bus(ep) ||
 		(is_parallel_bus(ep) &&
 		 infmt->code != MEDIA_BUS_FMT_UYVY8_2X8 &&

Signature · 5 lines

Krzysztof "Chris" Hałasa

Sieć Badawcza Łukasiewicz
Przemysłowy Instytut Automatyki i Pomiarów PIAP
Al. Jerozolimskie 202, 02-486 Warszawa

Dan Carpenter

2021-10-06 08:45 UTC #1 lore raw

drivers/gpu/ipu-v3/ipu-csi.c static int fill_csi_bus_cfg(struct ipu_csi_bus_config *csicfg, · line 392

Quoted 7 lines · Krzysztof Hałasa

drivers/gpu/ipu-v3/ipu-csi.c +3 -0

@@ -373,11 +387,18 @@ static int fill_csi_bus_cfg(struct ipu_csi_bus_config *csicfg,
 		break;
 	case V4L2_MBUS_BT656:
 		csicfg->ext_vsync = 0;
+		/* UYVY10_1X20 etc. should be supported as well */
+		is_bt1120 = mbus_fmt->code == MEDIA_BUS_FMT_UYVY8_1X16 ||
+			mbus_fmt->code == MEDIA_BUS_FMT_YUYV8_1X16;

Could you align better for readability, otherwise it's harder to spot the two characters which are swapped between those to macros. (Also it's basically normal for both sides of a || to be aligned...)

		is_bt1120 = (mbus_fmt->code == MEDIA_BUS_FMT_UYVY8_1X16 ||
			     mbus_fmt->code == MEDIA_BUS_FMT_YUYV8_1X16);

Quoted 4 lines

drivers/gpu/ipu-v3/ipu-csi.c +1 -1

 		if (V4L2_FIELD_HAS_BOTH(mbus_fmt->field) ||
 		    mbus_fmt->field == V4L2_FIELD_ALTERNATE)
-			csicfg->clk_mode = IPU_CSI_CLK_MODE_CCIR656_INTERLACED;
+			csicfg->clk_mode = is_bt1120 ?

drivers/gpu/ipu-v3/ipu-csi.c static int fill_csi_bus_cfg(struct ipu_csi_bus_config *csicfg, · line 404

Quoted 10 lines

drivers/gpu/ipu-v3/ipu-csi.c +5 -1

+				IPU_CSI_CLK_MODE_CCIR1120_INTERLACED_SDR :
+				IPU_CSI_CLK_MODE_CCIR656_INTERLACED;
 		else
-			csicfg->clk_mode = IPU_CSI_CLK_MODE_CCIR656_PROGRESSIVE;
+			csicfg->clk_mode = is_bt1120 ?
+				IPU_CSI_CLK_MODE_CCIR1120_PROGRESSIVE_SDR :
+				IPU_CSI_CLK_MODE_CCIR656_PROGRESSIVE;
 		break;
 	case V4L2_MBUS_CSI2_DPHY:
 		/*

regards,
dan carpenter

Philipp Zabel

maintainer

2021-10-06 09:36 UTC #2 lore raw

In reply to Krzysztof Hałasa

Hi Krzysztof,

Quoted 70 lines · Krzysztof Hałasa

Confirmed to work with ADV7610 HDMI receiver.

Signed-off-by Krzysztof Hałasa <[email protected]>

drivers/gpu/ipu-v3/ipu-csi.c +26 -5

index 658c173bebdf..2893b68f1f7a 100644
@@ -261,10 +261,24 @@ static int mbus_code_to_bus_cfg(struct ipu_csi_bus_config *cfg, u32 mbus_code,
 		cfg->data_width = IPU_CSI_DATA_WIDTH_8;
 		break;
 	case MEDIA_BUS_FMT_UYVY8_1X16:
+		if (mbus_type == V4L2_MBUS_BT656) {
+			cfg->data_fmt = CSI_SENS_CONF_DATA_FMT_YUV422_UYVY;
+			cfg->data_width = IPU_CSI_DATA_WIDTH_8;
+		} else {
+			cfg->data_fmt = CSI_SENS_CONF_DATA_FMT_BAYER;
+			cfg->data_width = IPU_CSI_DATA_WIDTH_16;
+		}
+		cfg->mipi_dt = MIPI_DT_YUV422;
+		break;
 	case MEDIA_BUS_FMT_YUYV8_1X16:
-		cfg->data_fmt = CSI_SENS_CONF_DATA_FMT_BAYER;
+		if (mbus_type == V4L2_MBUS_BT656) {
+			cfg->data_fmt = CSI_SENS_CONF_DATA_FMT_YUV422_YUYV;
+			cfg->data_width = IPU_CSI_DATA_WIDTH_8;
+		} else {
+			cfg->data_fmt = CSI_SENS_CONF_DATA_FMT_BAYER;
+			cfg->data_width = IPU_CSI_DATA_WIDTH_16;
+		}
 		cfg->mipi_dt = MIPI_DT_YUV422;
-		cfg->data_width = IPU_CSI_DATA_WIDTH_16;
 		break;
 	case MEDIA_BUS_FMT_SBGGR8_1X8:
 	case MEDIA_BUS_FMT_SGBRG8_1X8:

@@ -352,7 +366,7 @@ static int fill_csi_bus_cfg(struct ipu_csi_bus_config *csicfg,
 			    const struct v4l2_mbus_config *mbus_cfg,
 			    const struct v4l2_mbus_framefmt *mbus_fmt)
 {
-	int ret;
+	int ret, is_bt1120;
 
 	memset(csicfg, 0, sizeof(*csicfg));

@@ -373,11 +387,18 @@ static int fill_csi_bus_cfg(struct ipu_csi_bus_config *csicfg,
 		break;
 	case V4L2_MBUS_BT656:
 		csicfg->ext_vsync = 0;
+		/* UYVY10_1X20 etc. should be supported as well */
+		is_bt1120 = mbus_fmt->code == MEDIA_BUS_FMT_UYVY8_1X16 ||
+			mbus_fmt->code == MEDIA_BUS_FMT_YUYV8_1X16;
 		if (V4L2_FIELD_HAS_BOTH(mbus_fmt->field) ||
 		    mbus_fmt->field == V4L2_FIELD_ALTERNATE)
-			csicfg->clk_mode = IPU_CSI_CLK_MODE_CCIR656_INTERLACED;
+			csicfg->clk_mode = is_bt1120 ?
+				IPU_CSI_CLK_MODE_CCIR1120_INTERLACED_SDR :
+				IPU_CSI_CLK_MODE_CCIR656_INTERLACED;
 		else
-			csicfg->clk_mode = IPU_CSI_CLK_MODE_CCIR656_PROGRESSIVE;
+			csicfg->clk_mode = is_bt1120 ?
+				IPU_CSI_CLK_MODE_CCIR1120_PROGRESSIVE_SDR :
+				IPU_CSI_CLK_MODE_CCIR656_PROGRESSIVE;
 		break;
 	case V4L2_MBUS_CSI2_DPHY:
 		/*

drivers/staging/media/imx/imx-media-csi.c +0 -0

index 45f9d797b9da..ba93512f8c71 100644

drivers/staging/media/imx/imx-media-csi.c static inline bool is_parallel_16bit_bus(struct v4l2_fwnode_endpoint *ep) · line 143

Quoted 5 lines

drivers/staging/media/imx/imx-media-csi.c +2 -0

@@ -139,6 +139,8 @@ static inline bool is_parallel_16bit_bus(struct v4l2_fwnode_endpoint *ep)
  * Check for conditions that require the IPU to handle the
  * data internally as generic data, aka passthrough mode:
  * - raw bayer media bus formats, or
+ * - BT.656 and BT.1120 (8/10-bit YUV422) data can always be processed
+ *   on-the-fly (converted to YUV420)

This comment seems misleading. The CSI converts to YUV 4:4:4 internally.

drivers/staging/media/imx/imx-media-csi.c static inline bool requires_passthrough(struct v4l2_fwnode_endpoint *ep, · line 157

Quoted 13 lines

drivers/staging/media/imx/imx-media-csi.c +3 -0

  * - the CSI is receiving from a 16-bit parallel bus, or
  * - the CSI is receiving from an 8-bit parallel bus and the incoming
  *   media bus format is other than UYVY8_2X8/YUYV8_2X8.

@@ -147,6 +149,9 @@ static inline bool requires_passthrough(struct v4l2_fwnode_endpoint *ep,
 					struct v4l2_mbus_framefmt *infmt,
 					const struct imx_media_pixfmt *incc)
 {
+	if (ep->bus_type == V4L2_MBUS_BT656) // including BT.1120
+		return 0;
+
 	return incc->bayer || is_parallel_16bit_bus(ep) ||
 		(is_parallel_bus(ep) &&
 		 infmt->code != MEDIA_BUS_FMT_UYVY8_2X8 &&

regards
Philipp

Krzysztof Hałasa

author

2021-10-06 10:52 UTC #3 lore raw

Hi Philipp,

Quoted 4 lines · Philipp Zabel

Quoted 2 lines
+ * - BT.656 and BT.1120 (8/10-bit YUV422) data can always be processed
+ * on-the-fly (converted to YUV420)
This comment seems misleading. The CSI converts to YUV 4:4:4 internally.

Well... this is surprising. You mean "on the internal bus", don't you?

Please correct me if the following is wrong:

I always though that the "on-the-fly processing", in case of YUV422, means in practice I can get YUV420 out of the IPU, without a need to do e.g. NEON conversion. I know I can get the original YUV422 as well, using the "generic data" mode, but it's incompatible with the CODA H.264 encoder.

Ok, the DQRM (37.4.3.2.1) states that for parallel YUV the output from CSI is always YUV444.

Then 37.4.3.9 says that the only YUV422 way is to use 16-bit "generic data". This doesn't seem to be very true, however I'm not exactly sure about the "on-the-fly" thing.
The fact is the patch works.
Also, the CSIx_SENS_DATA_FORMAT field in IPUx_CSIx_SENS_CONF register shows YUV422 YUYV and UYVY input data formats, clearly separate from "Bayer of Generic data".

DQIEC, 4.12.10.1, isn't very clear either:
8) YCbCr 20-bit (10-bit Y + 10-bit U/V) is supported with BT.1120 only
7) YCbCr 16-bit is supported under the same conditions as 8)
6) YCbCr 16-bit (= YUV422) is also supported as "generic-data" (no on-the-fly processing). This seems to imply 8) and 7) are supported WITH o-t-f-p (and obviously I have tested it, 16-bit only).

I think I will just remove the comment :-)

Signature · 5 lines

Krzysztof "Chris" Hałasa

Sieć Badawcza Łukasiewicz
Przemysłowy Instytut Automatyki i Pomiarów PIAP
Al. Jerozolimskie 202, 02-486 Warszawa

Philipp Zabel

maintainer

2021-10-08 11:53 UTC #4 lore raw

Hi Krzysztof,

Quoted 10 lines · Krzysztof Hałasa

Hi Philipp,
Quoted 4 lines · Philipp Zabel
Quoted 2 lines
+ * - BT.656 and BT.1120 (8/10-bit YUV422) data can always be processed
+ * on-the-fly (converted to YUV420)
This comment seems misleading. The CSI converts to YUV 4:4:4 internally.
Well... this is surprising. You mean "on the internal bus", don't you?

Yes, anything apart from the bayer/generic data modes, internally everything is converted into 32-bit YUVA/RGBA pixels (according to the Reference Manual, 37.4.2.3 FCW & FCR). That is represented by the MEDIA_BUS_FMT_AYUV8_1X32 media bus format at the CSI source pads.

Quoted 5 lines

Please correct me if the following is wrong:
I always though that the "on-the-fly processing", in case of YUV422, means in practice I can get YUV420 out of the IPU, without a need to do e.g. NEON conversion.

That is done in the IDMAC, which can write any supported YUV format from the internal YUV pixels (if not in bayer/generic data mode).

Quoted 3 lines

I know I can get the original YUV422 as well,
using the "generic data" mode, but it's incompatible with the CODA H.264 encoder.

You should also be able to store the YUV formatted pixels as NV12, NV16, YUYV, etc.

Quoted 2 lines

Ok, the DQRM (37.4.3.2.1) states that for parallel YUV the output from CSI is always YUV444.

Ack.

Quoted 3 lines

Then 37.4.3.9 says that the only YUV422 way is to use 16-bit "generic data". This doesn't seem to be very true, however I'm not exactly sure about the "on-the-fly" thing.

I think that statement is limited to the parallel 16-bit interface in hsync/vsync mode, whereas in bt.656 / bt.1120 mode the interface operates as if the two components were clocked in as two separate 8-bit (or 10-bit) values.

Quoted 5 lines

The fact is the patch works.
Also, the CSIx_SENS_DATA_FORMAT field in IPUx_CSIx_SENS_CONF register shows YUV422 YUYV and UYVY input data formats, clearly separate from "Bayer of Generic data".

Quoted 8 lines

DQIEC, 4.12.10.1, isn't very clear either:
8) YCbCr 20-bit (10-bit Y + 10-bit U/V) is supported with BT.1120 only
7) YCbCr 16-bit is supported under the same conditions as 8)
6) YCbCr 16-bit (= YUV422) is also supported as "generic-data" (no on-the-fly processing). This seems to imply 8) and 7) are supported WITH o-t-f-p (and obviously I have tested it, 16-bit only).
I think I will just remove the comment :-)

That sounds good to me.

regards
Philipp