DRM: add i.MX IPUv3 base driver

Hi Sascha,

What is the status of this series, and especially of this patch, as to its integration into upstream? At some point, you were talking about pushing it to staging, but since things don't seem to have evolved.

Do you have a schedule for that? Are you waiting for more customer projects at Pengutronix to do that?

Would your customers_pengutronix_mx5_master-20120612 tag be the starting point for these new files? Can it be used as a working solution in the meantime, or does it have limitations?

That would be great to finally have full IPU/VPU/GPU support for i.MX5 into upstream. As you already know, Linaro also have some video support for i.MX5 in this head:
http://git.linaro.org/gitweb?p=landing-teams/working/freescale/kernel.git;a=shortlog;h=refs/heads/lt-3.2-imx5

Matt, I think you also have video support for efikamx, don't you?

Is there a plan to merge all these developments into upstream?

Best regards,
Beno?t

It's my understanding that Richard is looking at the VPU for i.MX6, based on Sascha's IPU patches (this thread) and Javier Martin's i.MX27 coda patches.

Best regards

Dirk

P.S.: Sascha might be on holiday, just in case there is no immediate answer from him ;)

Hi Dirk,

The VPU has no dependencies on the IPU, it's completely orthogonal.

Sascha

Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Beno?t,

Things are making progress. The intention is to get the base patches into staging; however, the holiday season made the progress a bit slow recently.

We don't have a fixed schedule; things make progress according to what our projects make possible. The more customers we get behind this effort, the more time we can spent on it.

- CSI is running on MX6 here
- Deinterlacer is WIP on MX53
- VPU was posted today, based on the Vista Silicon patches; we can do encoding and decoding, but not in all possible formats
- scaler works here
- We have a m2m plugin for GStreamer (which needs polishing as well, but works)

That's quite an old tree... especially the port to 3.5 made a lot of headache until all the oftree and clock issues were solved. However, it's still good for reference.

it's in a state that we think we can show it to you folks.

rsc

Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

DRM guys "don't have time to review it", apparently.

I can't find the patch series in any patchwork (I probably am not looking hard enough) but the base driver should be an MFD or something or placed somewhere other than in DRM directories as it has been in the past. It might make it quicker to review the core driver.

Then there is a major problem here; Sascha's DRM helper functions are also basically rejected again (which is a shame..) and nobody really has a cohesive device tree binding that would let us individually initialize and specify IPU units. It's not something easy to "tree-ize". I thought about it and walked away as it made my head hurt. Those two make it INCREDIBLY difficult to engineer a solution whereby a modular, independent IPU subsystem can support modular, independent encoder/connector pairs (which are usually 1:1 the same chip) without making the encoder/connector itself entirely dependent on the DRM driver underneath. In the case of the SII9022 which is on the Efika MX, this driver would need to be written for the Efika MX, have specific support for MX53 Quickstart, debouncing of hotplug or not depending on the board, some Beagle clones and TI's OMAP3 EVK have the chip too, and many others floating around. Nobody wants to write the same driver 5 times, and while the Silicon Image controllers are wonderfully generic with a common API ("TPI") and there could be a generic MFD solution with shim drivers on top for DRM, nobody else's are quite as nice.

DRM just doesn't support the passing of information that makes this generic, even if you had the binding. I was thinking of hijacking the "slave encoder" subsystem which is designed to help PCI/AGP cards register i2c buses local to the card (i.e. after PCI probe is done, while the DRM subsystem is handling things) and access i2c data like EDID or just talk to external encoder chips on these cards which change every 5 minutes. It would need to be told which i2c bus it needs to talk on (maybe a phandle to a ddc@ node for generic access, or a phandle
to a specific encoder if the DDC is bounded by some kind of locking or access restriction during normal use like on SII9022 - each slave encoder would need to supply it's DDC access functions or push in good default ones from another driver, but at the time of init, it cannot know the driver information to get this..). There are also several known flaws in the way DRM works (you cannot validate mode ids at the level you need to - it is a "connector" task, but it needs to be a crtc task too especially for IPU where bandwidth is defined by the IDMAC FIFO settings and units in use) which might require more than a rudimentary rework of parts of DRM (more like, a new API entirely..)

It's horrible, hideous even, and could require several years of development for something which nobody had the foresight to see, even though the configurations required have existed since the early 90's - a SoC with parallel or serial display bus, with transmitters to the panels or connectors connected over i2c for configuration and EDID data gathering. DRM just doesn't really support it in a generic way.

Both Pengutronix and Genesi have slightly differing implementations of these (especially csi, ic) which might need to be merged together and coordinated. I am not sure what the schedule for this is, Genesi's customer doesn't in the final result need actual Linux support (it's just a better case for proof of concept development and feature comparison/development than FSL's BSP code which is a spaghetti factory).

Well, here is the problem;

IPU, VPU and GPU are basically completely seperate. They do not need to depend on each other for anything, but the most EXCITING uses would coordinate these three units together through some userspace API (DRM for IPU-DI and GPU access, dma_buf for sharing, v4l2 mem2mem for VPU and IPU processing tasks..) such that you could send H.264 for example to decode, deinterlace, rotate and colour convert it in IPU then use it as a GPU texture (if only because the GPU doesn't support some common YUV formats without expensive shader) without copying anything around and just signalling your intent to the kernel.

That kind of coordination is simply missing from pretty much everyone's SoCs right now except perhaps Samsung's Exynos5.

VPU would be better serviced by improving the CODA driver someone submitted a month ago. The CODA driver ported uses the same v4l2 mem2mem API as Samsung's MFC video codec, but that codec is a different core than CODA. This is Chips&Media IP core, roughly compatible with it's various versions and included in i.MX series for a long time.. it is also shared by *older* Exynos (I think Exynos3) designs and probably exists in some lower-end ARM processor cores too from other manufacturers). It may by that legacy be in the original iPhone :)

GPU couldn't even get into mainline when developed by Qualcomm (same GPU for i.MX51 and i.MX53 as their "Adreno 200") - some significant coordination between companies here is required. Qualcomm uses a custom GPU access API called "GSL" which does somewhat of the same job as DRM could/should, but it needs porting, then there is a significant problem here after that in that the userspace is completely closed. Genesi has access to most of the source (some parts are still binaries even to us), some other companies do too, but the terms of the NDA are extremely restrictive. I cannot recommend Freedreno because I am technically not even allowed to look at it, but it might be your best chance for opensource here.

At some point. I would love to do it, but there's so much to actually do. It would benefit far more here than just Freescale chips, except the IPU, which would benefit far more than just a single Freescale SoC (i.MX51, 53, 6 and onwards would be supported) anyway due to a pretty much 99% shared API implementation with a few subtle but almost irrelevant differences.

Matt Sealey <[email protected]>
Product Development Analyst, Genesi USA, Inc.

I got here and immediately thought "oh god not more layers to DRM". DRM does _not_ need any more bloody layers on top of the existing layers. It needs a library of helper functions to provide common functionality so that folk don't have to write the same bloody stuff time and time again.

A library of helpers, not a shim layer to convert it to something else.