i.MX6 video capture support in mainline

Hello,
we have an BD-SL-i.MX6 platform (compatible with the Nitrogen6X) where we are currently running the BSP from Freescale, which is based on kernel 3.10 if I recall properly.

We are aware that those drivers have some issues, specially when it comes to compliance with the V4L2 frameworks like the media controller API, stability, etc...

Furthermore, we need to use the mainline kernel because some of the drivers that we need to use are not available in the Freescale kernel.

The biggest problem that we have found so far for switching to the mainline kernel is the video capture support in the I.MX6 IPU. I've been following some old e-mail threads (from 2014) and I eventually found Philipp Zabel's repository branch 'nitrogen6x-ipu-media' which has what seems to be an early version of an i.MX6 IPU capture driver via the CSI interface.

We've got here the same setup with an ov5642 sensor connected to the CSI interface and we have been giving a try to the driver.

This is what we have tried so far:

cat /dev/video0 # This is needed so that open gets called and the csi links are created media-ctl -l '"ov5642 1-003c":0->"mipi_ipu1_mux":1[1], "/soc/ipu@02400000/port@0":1->"IPU0 SMFC0":0[1]' media-ctl -l '"IPU0 SMFC0":1->"imx-ipuv3-camera.2":0[1]'

The last command will fail like this:

imx-ipuv3 2400000.ipu: invalid link 'IPU0 SMFC0'(5):1 -> 'imx-ipuv3-camera.2'(2):0 Unable to parse link: Invalid argument (22)

The reason it fails, apparently, is that the links that have been created when opening /dev/video0 are not included in the "ipu_links[]" static table defined in "drivers/gpio/ipu-v3/ipu-media.c" which is where the "ipu_smfc_link_setup()" function tries to find a valid link.

I've got some questions regarding this driver and iMX6 video capture support in general that someone here may gladly answer:

a) Is anyone currently working on mainlining iMX6 video capture support? I know about Steve's and Philipp's work but I haven't seen any progress since September 2014.

b) Does anyone know whether it's possible to capture YUV420P video using the driver in Philipp's repository? If so could you please provide the pipeline setup that you used with media-ctl?

c) If we were willing to help with mainline submission of this driver what issues should we focus on?

Regards,
Javier.

[1] git://git.pengutronix.de/git/pza/linux.git

Hi Javier,

Am Dienstag, den 23.06.2015, 12:20 +0200 schrieb Javier Martin:
[...]

Those are the IPU internal links, not handling the video device node entities is a bug in ipu-media. Try something like this:

-----8<-----

index fda80c7..920806c 100644
@@ -341,14 +341,16 @@ static int ipu_smfc_link_setup(struct media_entity *entity,
 			       const struct media_pad *remote, u32 flags)
 {
 	struct ipu_soc *ipu = to_ipu(entity);
-	const struct ipu_link *link;
+	const struct ipu_link *link = NULL;
 	const struct ipu_pad *pad;
 	u32 mask, sel = FS_SEL_ARM;
 	int csi, smfc;
 
-	link = ipu_find_link(ipu, local, remote);
-	if (!link)
-		return ipu_invalid_link(ipu, local, remote);
+	if (remote->entity->type != MEDIA_ENT_T_DEVNODE_V4L) {
+		link = ipu_find_link(ipu, local, remote);
+		if (!link)
+			return ipu_invalid_link(ipu, local, remote);
+	}
 
 	if (local->flags & MEDIA_PAD_FL_SINK) {
 		/* SMFC_MAP_CHx */

@@ -371,7 +373,7 @@ static int ipu_smfc_link_setup(struct media_entity *entity,
 		/* SMFCx_DEST_SEL */
 		pad = &ipu_entities[ipu_entity(ipu, entity)].pads[local->index];
 		mask = pad->mask << pad->offset;
-		if (flags & MEDIA_LNK_FL_ENABLED)
+		if (link && (flags & MEDIA_LNK_FL_ENABLED))
 			sel = link->dest_sel << pad->offset;
 
 		ipu_cm_update_bits(ipu, IPU_FS_PROC_FLOW3, mask, sel);

----->8-----

I am working on this on and off, but haven't found the time to fit in the media entity rework.

What do you mean by YUV420P? The capture driver supports V4L2_PIX_FMT_YUV420, but not the V4L2_PIX_FMT_YUV420M multiplanar format, because the hardware U/V planes are just fixed offsets from a base address. Memory bandwidth consumption wise, the most efficient formats are NV12 and YUYV/UYVY.

In my opinion, the media entities should be reworked and reduced to just two CSI entities, a single IC(PRP) entity, and four video device nodes (that correspond to the SMFC[0-3] FIFOs / CSI[0-3] IDMAC channels) with their entities for each IPU.
Describing details like the SMFC FIFOs or different IC stages as media entities and describing the IDMAC channel linking via buffers in system RAM as links between media entities turned out to be the wrong abstraction.
One thing I'm not yet clear about is how to handle MIPI CSI-2 streams with multiple virtual channels properly, but I haven't seen hardware that produces such streams yet.

regards
Philipp