Skip to contentSkip to conversation

i.MX Media Driver

PATCH Superseded Steve Longerbeam 12 participants 193 messages Jan 7, 2017 – Feb 15, 2017 Page 1 of 3

Subsystems (8)

v3 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Steve Longerbeam

author maintainer
#0 lore raw

v3 cover i.MX Media Driver

In version 3:

Changes suggested by Rob Herring <[email protected]>:

- prepended FIM node properties with vendor prefix "fsl,".

- make mipi csi-2 receiver compatible string SoC specific:

    "fsl,imx6-mipi-csi2" instead of "fsl,imx-mipi-csi2".

- redundant "_clk" removed from mipi csi-2 receiver clock-names property.

- removed board-specific info from the media driver binding doc. These were all related to sensor bindings, which already are (adv7180) or will be (ov564x) covered in separate binding docs. All reference board info not related to DT bindings has been moved to Documentation/media/v4l-drivers/imx.rst.

- removed "_mipi" from the OV5640 compatible string.

Changes suggested by Vladimir Zapolskiy <[email protected]>:

Mostly cosmetic/non-functional changes which I won't list here, except for the following:

- spin_lock_irqsave() changed to spin_lock() in a couple interrupt handlers.

- fixed some unnecessary of_node_put()'s in for_each_child_of_node() loops.

- check/handle return code from required reg property of CSI port nodes.

- check/handle return code from clk_prepare_enable().

Changes suggested by Fabio Estevam <[email protected]>:

- switch to VGEN3 Analog Vdd supply assuming rev. C SabreSD boards.

- finally got around to passing valid IOMUX pin config values to the pin groups.

Other changes:

- removed the FIM properties that overrided the v4l2 FIM control defaults values. This was left-over from a requirement of a customer and is not necessary here.

- The FIM must be explicitly enabled in the fim child node under the CSI port nodes, using the status property. If not enabled, FIM v4l2 controls will not appear in the video capture driver.

- brought in additional media types patch from Philipp Zabel. Use new MEDIA_ENT_F_VID_IF_BRIDGE in mipi csi-2 receiver subdev.

- brought in latest platform generic video multiplexer subdevice driver from Philipp Zabel (squashed with patch that uses new MEDIA_ENT_F_MUX).

- removed imx-media-of.h, moved those prototypes into imx-media.h.

Philipp Zabel (3):
ARM: dts: imx6qdl: Add mipi_ipu1/2 multiplexers, mipi_csi, and their
connections
add mux and video interface bridge entity functions
platform: add video-multiplexer subdevice driver

Steve Longerbeam (21):
[media] dt-bindings: Add bindings for i.MX media driver
ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node
ARM: dts: imx6qdl: add media device
ARM: dts: imx6qdl-sabrelite: remove erratum ERR006687 workaround
ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors
ARM: dts: imx6-sabresd: add OV5642 and OV5640 camera sensors
ARM: dts: imx6-sabreauto: create i2cmux for i2c3
ARM: dts: imx6-sabreauto: add reset-gpios property for max7310_b
ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture
ARM: dts: imx6-sabreauto: add the ADV7180 video decoder
UAPI: Add media UAPI Kbuild file
media: Add userspace header file for i.MX
media: Add i.MX media core driver
media: imx: Add CSI subdev driver
media: imx: Add SMFC subdev driver
media: imx: Add IC subdev drivers
media: imx: Add Camera Interface subdev driver
media: imx: Add MIPI CSI-2 Receiver subdev driver
media: imx: Add MIPI CSI-2 OV5640 sensor subdev driver
media: imx: Add Parallel OV5642 sensor subdev driver
ARM: imx_v6_v7_defconfig: Enable staging video4linux drivers

 Documentation/devicetree/bindings/media/imx.txt    |   57 +
 .../bindings/media/video-multiplexer.txt           |   59 +
 Documentation/media/uapi/mediactl/media-types.rst  |   22 +
 Documentation/media/v4l-drivers/imx.rst            |  443 ++
 arch/arm/boot/dts/imx6dl-sabrelite.dts             |    5 +
 arch/arm/boot/dts/imx6dl-sabresd.dts               |    5 +
 arch/arm/boot/dts/imx6dl.dtsi                      |  187 +
 arch/arm/boot/dts/imx6q-sabrelite.dts              |    6 +
 arch/arm/boot/dts/imx6q-sabresd.dts                |    5 +
 arch/arm/boot/dts/imx6q.dtsi                       |  127 +
 arch/arm/boot/dts/imx6qdl-sabreauto.dtsi           |  147 +-
 arch/arm/boot/dts/imx6qdl-sabrelite.dtsi           |  122 +-
 arch/arm/boot/dts/imx6qdl-sabresd.dtsi             |  114 +-
 arch/arm/boot/dts/imx6qdl.dtsi                     |   25 +-
 arch/arm/configs/imx_v6_v7_defconfig               |   12 +-
 drivers/media/platform/Kconfig                     |    8 +
 drivers/media/platform/Makefile                    |    2 +
 drivers/media/platform/video-multiplexer.c         |  472 +++
 drivers/staging/media/Kconfig                      |    2 +
 drivers/staging/media/Makefile                     |    1 +
 drivers/staging/media/imx/Kconfig                  |   36 +
 drivers/staging/media/imx/Makefile                 |   15 +
 drivers/staging/media/imx/TODO                     |   22 +
 drivers/staging/media/imx/imx-camif.c              | 1000 +++++
 drivers/staging/media/imx/imx-csi.c                |  644 +++
 drivers/staging/media/imx/imx-ic-common.c          |  109 +
 drivers/staging/media/imx/imx-ic-pp.c              |  636 +++
 drivers/staging/media/imx/imx-ic-prpenc.c          | 1033 +++++
 drivers/staging/media/imx/imx-ic-prpvf.c           | 1179 ++++++
 drivers/staging/media/imx/imx-ic.h                 |   38 +
 drivers/staging/media/imx/imx-media-common.c       |  981 +++++
 drivers/staging/media/imx/imx-media-dev.c          |  486 +++
 drivers/staging/media/imx/imx-media-fim.c          |  471 +++
 drivers/staging/media/imx/imx-media-internal-sd.c  |  457 ++
 drivers/staging/media/imx/imx-media-of.c           |  289 ++
 drivers/staging/media/imx/imx-media.h              |  310 ++
 drivers/staging/media/imx/imx-mipi-csi2.c          |  501 +++
 drivers/staging/media/imx/imx-smfc.c               |  737 ++++
 drivers/staging/media/imx/ov5640-mipi.c            | 2348 +++++++++++
 drivers/staging/media/imx/ov5642.c                 | 4363 ++++++++++++++++++++
 include/media/imx.h                                |   15 +
 include/uapi/Kbuild                                |    1 +
 include/uapi/linux/media.h                         |    6 +
 include/uapi/linux/v4l2-controls.h                 |    4 +
 include/uapi/media/Kbuild                          |    2 +
 include/uapi/media/imx.h                           |   30 +
 46 files changed, 17505 insertions(+), 29 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/media/imx.txt
 create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
 create mode 100644 Documentation/media/v4l-drivers/imx.rst
 create mode 100644 drivers/media/platform/video-multiplexer.c
 create mode 100644 drivers/staging/media/imx/Kconfig
 create mode 100644 drivers/staging/media/imx/Makefile
 create mode 100644 drivers/staging/media/imx/TODO
 create mode 100644 drivers/staging/media/imx/imx-camif.c
 create mode 100644 drivers/staging/media/imx/imx-csi.c
 create mode 100644 drivers/staging/media/imx/imx-ic-common.c
 create mode 100644 drivers/staging/media/imx/imx-ic-pp.c
 create mode 100644 drivers/staging/media/imx/imx-ic-prpenc.c
 create mode 100644 drivers/staging/media/imx/imx-ic-prpvf.c
 create mode 100644 drivers/staging/media/imx/imx-ic.h
 create mode 100644 drivers/staging/media/imx/imx-media-common.c
 create mode 100644 drivers/staging/media/imx/imx-media-dev.c
 create mode 100644 drivers/staging/media/imx/imx-media-fim.c
 create mode 100644 drivers/staging/media/imx/imx-media-internal-sd.c
 create mode 100644 drivers/staging/media/imx/imx-media-of.c
 create mode 100644 drivers/staging/media/imx/imx-media.h
 create mode 100644 drivers/staging/media/imx/imx-mipi-csi2.c
 create mode 100644 drivers/staging/media/imx/imx-smfc.c
 create mode 100644 drivers/staging/media/imx/ov5640-mipi.c
 create mode 100644 drivers/staging/media/imx/ov5642.c
 create mode 100644 include/media/imx.h
 create mode 100644 include/uapi/media/Kbuild
 create mode 100644 include/uapi/media/imx.h
Signature · 1 line
2.7.4
author maintainer
#1 lore raw

PATCH v3 1/24 dt-bindings: Add bindings for i.MX media driver

Add bindings documentation for the i.MX media driver.

---
 Documentation/devicetree/bindings/media/imx.txt | 57 +++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/imx.txt
Documentation/devicetree/bindings/media/imx.txt new file +57 -0
new file mode 100644
index 0000000..254b64a
@@ -0,0 +1,57 @@
+Freescale i.MX Media Video Devices
+
+Video Media Controller node
+---------------------------
+
+This is the parent media controller node for video capture support.
+
+Required properties:
+- compatible : "fsl,imx-media";
+- ports      : Should contain a list of phandles pointing to camera
+  	       sensor interface ports of IPU devices
+
+
+fim child node
+--------------
+
+This is an optional child node of the ipu_csi port nodes. If present and
+available, it enables the Frame Interval Monitor. Its properties can be
+used to modify the method in which the FIM measures frame intervals.
+Refer to Documentation/media/v4l-drivers/imx.rst for more info on the
+Frame Interval Monitor.
+
+Optional properties:
+- fsl,input-capture-channel: an input capture channel and channel flags,
+			     specified as <chan flags>. The channel number
+			     must be 0 or 1. The flags can be
+			     IRQ_TYPE_EDGE_RISING, IRQ_TYPE_EDGE_FALLING, or
+			     IRQ_TYPE_EDGE_BOTH, and specify which input
+			     capture signal edge will trigger the input
+			     capture event. If an input capture channel is
+			     specified, the FIM will use this method to
+			     measure frame intervals instead of via the EOF
+			     interrupt. The input capture method is much
+			     preferred over EOF as it is not subject to
+			     interrupt latency errors. However it requires
+			     routing the VSYNC or FIELD output signals of
+			     the camera sensor to one of the i.MX input
+			     capture pads (SD1_DAT0, SD1_DAT1), which also
+			     gives up support for SD1.
+
+
+mipi_csi2 node
+--------------
+
+This is the device node for the MIPI CSI-2 Receiver, required for MIPI
+CSI-2 sensors.
+
+Required properties:
+- compatible	: "fsl,imx6-mipi-csi2";
+- reg           : physical base address and length of the register set;
+- clocks	: the MIPI CSI-2 receiver requires three clocks: hsi_tx
+                  (the DPHY clock), video_27m, and eim_sel;
+- clock-names	: must contain "dphy", "cfg", "pix";
+
+Optional properties:
+- interrupts	: must contain two level-triggered interrupts,
+                  in order: 100 and 101;
Signature · 1 line
2.7.4
maintainer
#2 lore raw

Re: [PATCH v3 01/24] [media] dt-bindings: Add bindings for i.MX media driver

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:

Quoted 14 lines

Add bindings documentation for the i.MX media driver.

---
 Documentation/devicetree/bindings/media/imx.txt | 57 +++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/imx.txt
Documentation/devicetree/bindings/media/imx.txt new file +0 -0
new file mode 100644
index 0000000..254b64a

Documentation/devicetree/bindings/media/imx.txt line 9

Quoted 9 lines
Documentation/devicetree/bindings/media/imx.txt new file +9 -0
@@ -0,0 +1,57 @@
+Freescale i.MX Media Video Devices
+
+Video Media Controller node
+---------------------------
+
+This is the parent media controller node for video capture support.
+
+Required properties:
+- compatible : "fsl,imx-media";

Would you be opposed to calling this "capture-subsystem" instead of "imx-media"? We already use "fsl,imx-display-subsystem" and "fsl,imx-gpu-subsystem" for the display and GPU compound devices.

Quoted 20 lines
Documentation/devicetree/bindings/media/imx.txt new file +20 -0
+- ports      : Should contain a list of phandles pointing to camera
+  	       sensor interface ports of IPU devices
+
+
+fim child node
+--------------
+
+This is an optional child node of the ipu_csi port nodes. If present and
+available, it enables the Frame Interval Monitor. Its properties can be
+used to modify the method in which the FIM measures frame intervals.
+Refer to Documentation/media/v4l-drivers/imx.rst for more info on the
+Frame Interval Monitor.
+
+Optional properties:
+- fsl,input-capture-channel: an input capture channel and channel flags,
+			     specified as <chan flags>. The channel number
+			     must be 0 or 1. The flags can be
+			     IRQ_TYPE_EDGE_RISING, IRQ_TYPE_EDGE_FALLING, or
+			     IRQ_TYPE_EDGE_BOTH, and specify which input
+			     capture signal edge will trigger the input

Documentation/devicetree/bindings/media/imx.txt line 39

Quoted 10 lines
Documentation/devicetree/bindings/media/imx.txt new file +10 -0
+			     capture event. If an input capture channel is
+			     specified, the FIM will use this method to
+			     measure frame intervals instead of via the EOF
+			     interrupt. The input capture method is much
+			     preferred over EOF as it is not subject to
+			     interrupt latency errors. However it requires
+			     routing the VSYNC or FIELD output signals of
+			     the camera sensor to one of the i.MX input
+			     capture pads (SD1_DAT0, SD1_DAT1), which also
+			     gives up support for SD1.

This is a clever method to get better frame timestamps. Too bad about the routing requirements. Can this be used on Nitrogen6X?

Documentation/devicetree/bindings/media/imx.txt line 49

Quoted 9 lines
Documentation/devicetree/bindings/media/imx.txt new file +9 -0
+
+mipi_csi2 node
+--------------
+
+This is the device node for the MIPI CSI-2 Receiver, required for MIPI
+CSI-2 sensors.
+
+Required properties:
+- compatible	: "fsl,imx6-mipi-csi2";

I think this should get an additional "snps,dw-mipi-csi2" compatible, since the only i.MX6 specific part is the bolted-on IPU2CSI gasket.

Documentation/devicetree/bindings/media/imx.txt line 52

Quoted 3 lines
Documentation/devicetree/bindings/media/imx.txt new file +3 -0
+- reg           : physical base address and length of the register set;
+- clocks	: the MIPI CSI-2 receiver requires three clocks: hsi_tx
+                  (the DPHY clock), video_27m, and eim_sel;

Note that hsi_tx is incorrectly named. CCGR3[CG8] just happens to be the shared gate bit that gates the HSI clocks as well as the MIPI "ac_clk_125m", "cfg_clk", "ips_clk", and "pll_refclk" inputs to the mipi csi-2 core, but we are missing shared gate clocks in the clock tree for these.
Both cfg_clk and pll_refclk are sourced from video_27m, so "cfg" -> video_27m seems fine.
But I don't get "dphy". Which input clock would that correspond to? "pll_refclk?"
Also the pixel clock input is a gate after aclk_podf (which we call eim_podf), not aclk_sel (eim_sel).

Documentation/devicetree/bindings/media/imx.txt line 57

Quoted 5 lines
Documentation/devicetree/bindings/media/imx.txt new file +5 -0
+- clock-names	: must contain "dphy", "cfg", "pix";
+
+Optional properties:
+- interrupts	: must contain two level-triggered interrupts,
+                  in order: 100 and 101;

regards
Philipp

author maintainer
#3 lore raw

Re: [PATCH v3 01/24] [media] dt-bindings: Add bindings for i.MX media driver

Quoted 19 lines · Philipp Zabel

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:

Quoted 18 lines

Add bindings documentation for the i.MX media driver.

---
  Documentation/devicetree/bindings/media/imx.txt | 57 +++++++++++++++++++++++++
  1 file changed, 57 insertions(+)

create mode 100644 Documentation/devicetree/bindings/media/imx.txt

Documentation/devicetree/bindings/media/imx.txt new file +4 -0
new file mode 100644
index 0000000..254b64a
@@ -0,0 +1,57 @@
+Freescale i.MX Media Video Devices
+
+Video Media Controller node
+---------------------------
Quoted 8 lines

Documentation/devicetree/bindings/media/imx.txt line 9

Quoted 5 lines
Documentation/devicetree/bindings/media/imx.txt new file +5 -0
+
+This is the parent media controller node for video capture support.
+
+Required properties:
+- compatible : "fsl,imx-media";

Would you be opposed to calling this "capture-subsystem" instead of "imx-media"? We already use "fsl,imx-display-subsystem" and "fsl,imx-gpu-subsystem" for the display and GPU compound devices.

sure. Some pie-in-the-sky day when DRM and media are unified, there could be a single device that handles them all, but for now I'm fine with "fsl,capture-subsystem".

Quoted 24 lines
Quoted 24 lines
Documentation/devicetree/bindings/media/imx.txt new file +24 -0
+- ports      : Should contain a list of phandles pointing to camera
+  	       sensor interface ports of IPU devices
+
+
+fim child node
+--------------
+
+This is an optional child node of the ipu_csi port nodes. If present and
+available, it enables the Frame Interval Monitor. Its properties can be
+used to modify the method in which the FIM measures frame intervals.
+Refer to Documentation/media/v4l-drivers/imx.rst for more info on the
+Frame Interval Monitor.
+
+Optional properties:
+- fsl,input-capture-channel: an input capture channel and channel flags,
+			     specified as <chan flags>. The channel number
+			     must be 0 or 1. The flags can be
+			     IRQ_TYPE_EDGE_RISING, IRQ_TYPE_EDGE_FALLING, or
+			     IRQ_TYPE_EDGE_BOTH, and specify which input
+			     capture signal edge will trigger the input
+			     capture event. If an input capture channel is
+			     specified, the FIM will use this method to
+			     measure frame intervals instead of via the EOF
+			     interrupt. The input capture method is much
Quoted 8 lines

Documentation/devicetree/bindings/media/imx.txt line 39

Quoted 6 lines
Documentation/devicetree/bindings/media/imx.txt new file +6 -0
+			     preferred over EOF as it is not subject to
+			     interrupt latency errors. However it requires
+			     routing the VSYNC or FIELD output signals of
+			     the camera sensor to one of the i.MX input
+			     capture pads (SD1_DAT0, SD1_DAT1), which also
+			     gives up support for SD1.

This is a clever method to get better frame timestamps. Too bad about the routing requirements. Can this be used on Nitrogen6X?

Absolutely, this support just needs use of the input-capture channels in the imx GPT. I still need to submit the patch to the imx-gpt driver that adds an input capture API, so at this point fsl,input-capture-channel has no effect, but it does work (tested on SabreAuto).

Quoted 12 lines

Documentation/devicetree/bindings/media/imx.txt line 49

Quoted 10 lines
Documentation/devicetree/bindings/media/imx.txt new file +9 -0
+
+mipi_csi2 node
+--------------
+
+This is the device node for the MIPI CSI-2 Receiver, required for MIPI
+CSI-2 sensors.
+
+Required properties:
+- compatible	: "fsl,imx6-mipi-csi2";

I think this should get an additional "snps,dw-mipi-csi2" compatible, since the only i.MX6 specific part is the bolted-on IPU2CSI gasket.

right, minus the gasket it's a Synopsys core. I'll add that compatible flag. Or should wait until the day this subdev is exported for general use, after pulling out the gasket specifics?

Quoted 9 lines

Documentation/devicetree/bindings/media/imx.txt line 52

Quoted 4 lines
Documentation/devicetree/bindings/media/imx.txt new file +3 -0
+- reg           : physical base address and length of the register set;
+- clocks	: the MIPI CSI-2 receiver requires three clocks: hsi_tx
+                  (the DPHY clock), video_27m, and eim_sel;

Note that hsi_tx is incorrectly named. CCGR3[CG8] just happens to be the shared gate bit that gates the HSI clocks as well as the MIPI "ac_clk_125m", "cfg_clk", "ips_clk", and "pll_refclk" inputs to the mipi csi-2 core, but we are missing shared gate clocks in the clock tree for these.

Yes, so many clocks for the MIPI core. Why so many? I would think there would need to be at most three: a clock for the MIPI CSI-2 core and HSI core, and a clock for the D-PHY (oh and maybe a clock for an M-PHY if there is one). I have no clue what all these other clocks are. But anyway, a single gating bit, CCGR3[CG8], seems to enable them all.

Quoted 3 lines

Both cfg_clk and pll_refclk are sourced from video_27m, so "cfg" -> video_27m seems fine.
But I don't get "dphy".

I presume it's the clock for the D-PHY.

Quoted 2 lines

Which input clock would that correspond to?
"pll_refclk?"

the mux at CDCDR says it comes from PLL3_120M, or PLL2_PFD2.

Quoted 2 lines

Also the pixel clock input is a gate after aclk_podf (which we call eim_podf), not aclk_sel (eim_sel).

ok, I'll switch to eim_podf for the pixel clock.

Steve

Quoted 9 lines

Documentation/devicetree/bindings/media/imx.txt line 57

Quoted 6 lines
Documentation/devicetree/bindings/media/imx.txt new file +5 -0
+- clock-names	: must contain "dphy", "cfg", "pix";
+
+Optional properties:
+- interrupts	: must contain two level-triggered interrupts,
+                  in order: 100 and 101;

regards
Philipp

maintainer
#4 lore raw

Re: [PATCH v3 01/24] [media] dt-bindings: Add bindings for i.MX media driver

Quoted 30 lines · Steve Longerbeam
Quoted 19 lines · Philipp Zabel

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:

Quoted 18 lines

Add bindings documentation for the i.MX media driver.

---
  Documentation/devicetree/bindings/media/imx.txt | 57 +++++++++++++++++++++++++
  1 file changed, 57 insertions(+)

create mode 100644 Documentation/devicetree/bindings/media/imx.txt

Documentation/devicetree/bindings/media/imx.txt new file +4 -0
new file mode 100644
index 0000000..254b64a
@@ -0,0 +1,57 @@
+Freescale i.MX Media Video Devices
+
+Video Media Controller node
+---------------------------
Quoted 8 lines

Documentation/devicetree/bindings/media/imx.txt line 9

Quoted 5 lines
Documentation/devicetree/bindings/media/imx.txt new file +5 -0
+
+This is the parent media controller node for video capture support.
+
+Required properties:
+- compatible : "fsl,imx-media";

Would you be opposed to calling this "capture-subsystem" instead of "imx-media"? We already use "fsl,imx-display-subsystem" and "fsl,imx-gpu-subsystem" for the display and GPU compound devices.

Quoted 2 lines

sure. Some pie-in-the-sky day when DRM and media are unified, there could be a single device that handles them all,

Indeed :)

Quoted 2 lines

but for now
I'm fine with "fsl,capture-subsystem".

Actually, I meant fsl,imx-capture-subsystem. fsl,imx-media-subsystem would be fine, too. Either way, I'll be happy if it looks similar to the other two.

[...]

Quoted 7 lines
Quoted 2 lines

This is a clever method to get better frame timestamps. Too bad about the routing requirements. Can this be used on Nitrogen6X?

Absolutely, this support just needs use of the input-capture channels in the imx GPT. I still need to submit the patch to the imx-gpt driver that adds an input capture API, so at this point fsl,input-capture-channel has no effect, but it does work (tested on SabreAuto).

Nice.

[...]

Quoted 8 lines
Quoted 4 lines

Documentation/devicetree/bindings/media/imx.txt line 11

Quoted 2 lines
Documentation/devicetree/bindings/media/imx.txt new file +2 -0
+Required properties:
+- compatible	: "fsl,imx6-mipi-csi2";

I think this should get an additional "snps,dw-mipi-csi2" compatible, since the only i.MX6 specific part is the bolted-on IPU2CSI gasket.

right, minus the gasket it's a Synopsys core. I'll add that compatible flag. Or should wait until the day this subdev is exported for general use, after pulling out the gasket specifics?

It can be added right away.

Quoted 9 lines
Quoted 8 lines

Documentation/devicetree/bindings/media/imx.txt line 52

Quoted 3 lines
Documentation/devicetree/bindings/media/imx.txt new file +3 -0
+- reg           : physical base address and length of the register set;
+- clocks	: the MIPI CSI-2 receiver requires three clocks: hsi_tx
+                  (the DPHY clock), video_27m, and eim_sel;

Note that hsi_tx is incorrectly named. CCGR3[CG8] just happens to be the shared gate bit that gates the HSI clocks as well as the MIPI "ac_clk_125m", "cfg_clk", "ips_clk", and "pll_refclk" inputs to the mipi csi-2 core, but we are missing shared gate clocks in the clock tree for these.

Quoted 5 lines

Yes, so many clocks for the MIPI core. Why so many? I would think there would need to be at most three: a clock for the MIPI CSI-2 core and HSI core, and a clock for the D-PHY (oh and maybe a clock for an M-PHY if there is one). I have no clue what all these other clocks are. But anyway, a single gating bit, CCGR3[CG8], seems to enable them all.

I would imagine the CSI-2 core has a high-speed clock input from the D-PHY for serial input, an APB clock for register access (ips_clk), and a pixel clock input for the parallel output (pixel_clk), at least. The D-PHY will have a PLL reference input (pll_refclk?) and probably its own register clock (cfg_clk?).

I've looked at the MIPI DSI chapter, and it looks like ac_clk_125m is used for DSI only.

Quoted 10 lines
Quoted 3 lines

Both cfg_clk and pll_refclk are sourced from video_27m, so "cfg" -> video_27m seems fine.
But I don't get "dphy".

I presume it's the clock for the D-PHY.

Quoted 2 lines

Which input clock would that correspond to?
"pll_refclk?"

the mux at CDCDR says it comes from PLL3_120M, or PLL2_PFD2.

I think that makes sense.

regards
Philipp

author maintainer
#5 lore raw

Re: [PATCH v3 01/24] [media] dt-bindings: Add bindings for i.MX media driver

Quoted 29 lines · Philipp Zabel
Quoted 28 lines · Steve Longerbeam
Quoted 27 lines · Philipp Zabel

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:

Quoted 14 lines

Add bindings documentation for the i.MX media driver.

---
   Documentation/devicetree/bindings/media/imx.txt | 57 +++++++++++++++++++++++++
   1 file changed, 57 insertions(+)

create mode 100644 Documentation/devicetree/bindings/media/imx.txt

Documentation/devicetree/bindings/media/imx.txt new file +0 -0
new file mode 100644
index 0000000..254b64a

Documentation/devicetree/bindings/media/imx.txt line 9

Quoted 9 lines
Documentation/devicetree/bindings/media/imx.txt new file +9 -0
@@ -0,0 +1,57 @@
+Freescale i.MX Media Video Devices
+
+Video Media Controller node
+---------------------------
+
+This is the parent media controller node for video capture support.
+
+Required properties:
+- compatible : "fsl,imx-media";

Would you be opposed to calling this "capture-subsystem" instead of "imx-media"? We already use "fsl,imx-display-subsystem" and "fsl,imx-gpu-subsystem" for the display and GPU compound devices.

Quoted 7 lines
Quoted 2 lines

sure. Some pie-in-the-sky day when DRM and media are unified, there could be a single device that handles them all,

Indeed :)

Quoted 2 lines

but for now
I'm fine with "fsl,capture-subsystem".

Actually, I meant fsl,imx-capture-subsystem.

right, I caught my error and that is the name chosen.

Quoted 14 lines

fsl,imx-media-subsystem
would be fine, too. Either way, I'll be happy if it looks similar to the other two.

[...]

Quoted 6 lines
Quoted 2 lines

This is a clever method to get better frame timestamps. Too bad about the routing requirements. Can this be used on Nitrogen6X?

Absolutely, this support just needs use of the input-capture channels in the imx GPT. I still need to submit the patch to the imx-gpt driver that adds an input capture API, so at this point fsl,input-capture-channel has no effect, but it does work (tested on SabreAuto).

Nice.

[...]

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

Documentation/devicetree/bindings/media/imx.txt line 11

Quoted 2 lines
Documentation/devicetree/bindings/media/imx.txt new file +2 -0
+Required properties:
+- compatible	: "fsl,imx6-mipi-csi2";

I think this should get an additional "snps,dw-mipi-csi2" compatible, since the only i.MX6 specific part is the bolted-on IPU2CSI gasket.

right, minus the gasket it's a Synopsys core. I'll add that compatible flag. Or should wait until the day this subdev is exported for general use, after pulling out the gasket specifics?

It can be added right away.

ok, I will add.

Steve

Quoted 36 lines
Quoted 9 lines
Quoted 9 lines

Documentation/devicetree/bindings/media/imx.txt line 52

Quoted 4 lines
Documentation/devicetree/bindings/media/imx.txt new file +3 -0
+- reg           : physical base address and length of the register set;
+- clocks	: the MIPI CSI-2 receiver requires three clocks: hsi_tx
+                  (the DPHY clock), video_27m, and eim_sel;

Note that hsi_tx is incorrectly named. CCGR3[CG8] just happens to be the shared gate bit that gates the HSI clocks as well as the MIPI "ac_clk_125m", "cfg_clk", "ips_clk", and "pll_refclk" inputs to the mipi csi-2 core, but we are missing shared gate clocks in the clock tree for these.

Quoted 5 lines

Yes, so many clocks for the MIPI core. Why so many? I would think there would need to be at most three: a clock for the MIPI CSI-2 core and HSI core, and a clock for the D-PHY (oh and maybe a clock for an M-PHY if there is one). I have no clue what all these other clocks are. But anyway, a single gating bit, CCGR3[CG8], seems to enable them all.

I would imagine the CSI-2 core has a high-speed clock input from the D-PHY for serial input, an APB clock for register access (ips_clk), and a pixel clock input for the parallel output (pixel_clk), at least. The D-PHY will have a PLL reference input (pll_refclk?) and probably its own register clock (cfg_clk?).

I've looked at the MIPI DSI chapter, and it looks like ac_clk_125m is used for DSI only.

Quoted 8 lines
Quoted 3 lines

Both cfg_clk and pll_refclk are sourced from video_27m, so "cfg" -> video_27m seems fine.
But I don't get "dphy".

I presume it's the clock for the D-PHY.

Quoted 2 lines

Which input clock would that correspond to?
"pll_refclk?"

the mux at CDCDR says it comes from PLL3_120M, or PLL2_PFD2.

I think that makes sense.

regards
Philipp

author maintainer
#6 lore raw

PATCH v3 2/24 ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node

In reply to Steve Longerbeam

Add to the MIPI CSI2 receiver node: compatible string, interrupt sources, clocks.

---
 arch/arm/boot/dts/imx6qdl.dtsi | 7 +++++++
 1 file changed, 7 insertions(+)
arch/arm/boot/dts/imx6qdl.dtsi +7 -0
index 53e6e63..42926e9 100644
@@ -1125,7 +1125,14 @@
 			};
 
 			mipi_csi: mipi at 021dc000 {
+				compatible = "fsl,imx6-mipi-csi2";
 				reg = <0x021dc000 0x4000>;
+				interrupts = <0 100 0x04>, <0 101 0x04>;
+				clocks = <&clks IMX6QDL_CLK_HSI_TX>,
+					 <&clks IMX6QDL_CLK_VIDEO_27M>,
+					 <&clks IMX6QDL_CLK_EIM_SEL>;
+				clock-names = "dphy", "cfg", "pix";
+				status = "disabled";
 			};
 
 			mipi_dsi: mipi at 021e0000 {
Signature · 1 line
2.7.4
maintainer
#7 lore raw

Re: [PATCH v3 02/24] ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:

Quoted 13 lines

Add to the MIPI CSI2 receiver node: compatible string, interrupt sources, clocks.

---
 arch/arm/boot/dts/imx6qdl.dtsi | 7 +++++++
 1 file changed, 7 insertions(+)
arch/arm/boot/dts/imx6qdl.dtsi +0 -0
index 53e6e63..42926e9 100644

arch/arm/boot/dts/imx6qdl.dtsi line 1133

Quoted 9 lines
arch/arm/boot/dts/imx6qdl.dtsi +5 -0
@@ -1125,7 +1125,14 @@
 			};
 
 			mipi_csi: mipi at 021dc000 {
+				compatible = "fsl,imx6-mipi-csi2";
 				reg = <0x021dc000 0x4000>;
+				interrupts = <0 100 0x04>, <0 101 0x04>;
+				clocks = <&clks IMX6QDL_CLK_HSI_TX>,
+					 <&clks IMX6QDL_CLK_VIDEO_27M>,
+					 <&clks IMX6QDL_CLK_EIM_SEL>;

I think the latter should be EIM_PODF

arch/arm/boot/dts/imx6qdl.dtsi line 1134

Quoted 1 line
arch/arm/boot/dts/imx6qdl.dtsi +1 -0
+				clock-names = "dphy", "cfg", "pix";

and I'm not sure dphy is the right name for this one. Is that the pll ref input?

arch/arm/boot/dts/imx6qdl.dtsi line 1138

Quoted 4 lines
arch/arm/boot/dts/imx6qdl.dtsi +1 -0
+				status = "disabled";
 			};
 
 			mipi_dsi: mipi at 021e0000 {

regards
Philipp

author maintainer
#8 lore raw

Re: [PATCH v3 02/24] ARM: dts: imx6qdl: Add compatible, clocks, irqs to MIPI CSI-2 node

Quoted 16 lines · Philipp Zabel

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:

Quoted 15 lines

Add to the MIPI CSI2 receiver node: compatible string, interrupt sources, clocks.

---
  arch/arm/boot/dts/imx6qdl.dtsi | 7 +++++++
  1 file changed, 7 insertions(+)
arch/arm/boot/dts/imx6qdl.dtsi +0 -0
index 53e6e63..42926e9 100644
@@ -1125,7 +1125,14 @@
  			};
 
Quoted 8 lines

arch/arm/boot/dts/imx6qdl.dtsi line 1133

Quoted 7 lines
arch/arm/boot/dts/imx6qdl.dtsi +5 -0
  			mipi_csi: mipi at 021dc000 {
+				compatible = "fsl,imx6-mipi-csi2";
  				reg = <0x021dc000 0x4000>;
+				interrupts = <0 100 0x04>, <0 101 0x04>;
+				clocks = <&clks IMX6QDL_CLK_HSI_TX>,
+					 <&clks IMX6QDL_CLK_VIDEO_27M>,
+					 <&clks IMX6QDL_CLK_EIM_SEL>;

I think the latter should be EIM_PODF

done.

Quoted 4 lines

arch/arm/boot/dts/imx6qdl.dtsi line 1134

Quoted 2 lines
arch/arm/boot/dts/imx6qdl.dtsi +1 -0
+				clock-names = "dphy", "cfg", "pix";

and I'm not sure dphy is the right name for this one. Is that the pll ref input?

I believe this naming came from FSL's mipi csi-2 driver. It is the "hsi_tx" clock (presumably for the MIPI HSI controller) whose parents are selected by the CDCDR register as PLL3_120M or PLL2_PFD2. I have no clue whether this is indeed also used as the clock for the MIPI CSI-2 D-PHY, but according
to FSL naming convention it might be.

Steve

author maintainer
#9 lore raw

PATCH v3 3/24 ARM: dts: imx6qdl: Add mipi_ipu1/2 multiplexers, mipi_csi, and their connections

In reply to Steve Longerbeam

From: Philipp Zabel <[email protected]>

This patch adds the device tree graph connecting the input multiplexers to the IPU CSIs and the MIPI-CSI2 gasket on i.MX6. The MIPI_IPU multiplexers are added as children of the iomuxc-gpr syscon device node. On i.MX6Q/D two two-input multiplexers in front of IPU1 CSI0 and IPU2 CSI1 allow to select between CSI0/1 parallel input pads and the MIPI CSI-2 virtual channels 0/3.
On i.MX6DL/S two five-input multiplexers in front of IPU1 CSI0 and IPU1 CSI1 allow to select between CSI0/1 parallel input pads and any of the four MIPI CSI-2 virtual channels.

--

- Removed some dangling/unused endpoints (ipu2_csi0_from_csi2ipu)
- Renamed the mipi virtual channel endpoint labels, from "mipi_csiX_..." to "mipi_vcX...".
- Added input endpoint anchors to the video muxes for the connections from parallel sensors.
- Added input endpoint anchors to the mipi_csi for the connections from mipi csi-2 sensors.

---
 arch/arm/boot/dts/imx6dl.dtsi  | 187 +++++++++++++++++++++++++++++++++++++++++
 arch/arm/boot/dts/imx6q.dtsi   | 123 +++++++++++++++++++++++++++
 arch/arm/boot/dts/imx6qdl.dtsi |  10 ++-
 3 files changed, 319 insertions(+), 1 deletion(-)
arch/arm/boot/dts/imx6dl.dtsi +187 -0
index 1ade195..0849e85 100644
@@ -181,6 +181,193 @@
 		      "di0", "di1";
 };
 
+&gpr {
+	ipu1_csi0_mux: ipu1_csi0_mux at 34 {
+		compatible = "video-multiplexer";
+		reg = <0x34>;
+		bit-mask = <0x7>;
+		bit-shift = <0>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+		status = "okay";
+
+		port at 0 {
+			reg = <0>;
+
+			ipu1_csi0_mux_from_mipi_vc0: endpoint {
+				remote-endpoint = <&mipi_vc0_to_ipu1_csi0_mux>;
+			};
+		};
+
+		port at 1 {
+			reg = <1>;
+
+			ipu1_csi0_mux_from_mipi_vc1: endpoint {
+				remote-endpoint = <&mipi_vc1_to_ipu1_csi0_mux>;
+			};
+		};
+
+		port at 2 {
+			reg = <2>;
+
+			ipu1_csi0_mux_from_mipi_vc2: endpoint {
+				remote-endpoint = <&mipi_vc2_to_ipu1_csi0_mux>;
+			};
+		};
+
+		port at 3 {
+			reg = <3>;
+
+			ipu1_csi0_mux_from_mipi_vc3: endpoint {
+				remote-endpoint = <&mipi_vc3_to_ipu1_csi0_mux>;
+			};
+		};
+
+		port at 4 {
+			reg = <4>;
+
+			ipu1_csi0_mux_from_parallel_sensor: endpoint {
+			};
+		};
+
+		port at 5 {
+			reg = <5>;
+
+			ipu1_csi0_mux_to_ipu1_csi0: endpoint {
+				remote-endpoint = <&ipu1_csi0_from_ipu1_csi0_mux>;
+			};
+		};
+	};
+
+	ipu1_csi1_mux: ipu1_csi1_mux at 34 {
+		compatible = "video-multiplexer";
+		reg = <0x34>;
+		bit-mask = <0x7>;
+		bit-shift = <3>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+		status = "okay";
+
+		port at 0 {
+			reg = <0>;
+
+			ipu1_csi1_mux_from_mipi_vc0: endpoint {
+				remote-endpoint = <&mipi_vc0_to_ipu1_csi1_mux>;
+			};
+		};
+
+		port at 1 {
+			reg = <1>;
+
+			ipu1_csi1_mux_from_mipi_vc1: endpoint {
+				remote-endpoint = <&mipi_vc1_to_ipu1_csi1_mux>;
+			};
+		};
+
+		port at 2 {
+			reg = <2>;
+
+			ipu1_csi1_mux_from_mipi_vc2: endpoint {
+				remote-endpoint = <&mipi_vc2_to_ipu1_csi1_mux>;
+			};
+		};
+
+		port at 3 {
+			reg = <3>;
+
+			ipu1_csi1_mux_from_mipi_vc3: endpoint {
+				remote-endpoint = <&mipi_vc3_to_ipu1_csi1_mux>;
+			};
+		};
+
+		port at 4 {
+			reg = <4>;
+
+			ipu1_csi1_mux_from_parallel_sensor: endpoint {
+			};
+		};
+
+		port at 5 {
+			reg = <5>;
+
+			ipu1_csi1_mux_to_ipu1_csi1: endpoint {
+				remote-endpoint = <&ipu1_csi1_from_ipu1_csi1_mux>;
+			};
+		};
+	};
+};
+
+&ipu1_csi1 {
+	ipu1_csi1_from_ipu1_csi1_mux: endpoint {
+		remote-endpoint = <&ipu1_csi1_mux_to_ipu1_csi1>;
+	};
+};
+
+&mipi_csi {
+	port at 0 {
+		reg = <0>;
+
+		mipi_csi_from_mipi_sensor: endpoint {
+		};
+	};
+
+	port at 1 {
+		reg = <1>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		mipi_vc0_to_ipu1_csi0_mux: endpoint at 0 {
+			remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc0>;
+		};
+
+		mipi_vc0_to_ipu1_csi1_mux: endpoint at 1 {
+			remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc0>;
+		};
+	};
+
+	port at 2 {
+		reg = <2>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		mipi_vc1_to_ipu1_csi0_mux: endpoint at 0 {
+			remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc1>;
+		};
+
+		mipi_vc1_to_ipu1_csi1_mux: endpoint at 1 {
+			remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc1>;
+		};
+	};
+
+	port at 3 {
+		reg = <3>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		mipi_vc2_to_ipu1_csi0_mux: endpoint at 0 {
+			remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc2>;
+		};
+
+		mipi_vc2_to_ipu1_csi1_mux: endpoint at 1 {
+			remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc2>;
+		};
+	};
+
+	port at 4 {
+		reg = <4>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		mipi_vc3_to_ipu1_csi0_mux: endpoint at 0 {
+			remote-endpoint = <&ipu1_csi0_mux_from_mipi_vc3>;
+		};
+
+		mipi_vc3_to_ipu1_csi1_mux: endpoint at 1 {
+			remote-endpoint = <&ipu1_csi1_mux_from_mipi_vc3>;
+		};
+	};
+};
+
 &vpu {
 	compatible = "fsl,imx6dl-vpu", "cnm,coda960";
 };
arch/arm/boot/dts/imx6q.dtsi +123 -0

145 lines not on this page · raw message

arch/arm/boot/dts/imx6qdl.dtsi +9 -1
index 42926e9..010388c 100644
@@ -799,8 +799,10 @@
 			};
 
 			gpr: iomuxc-gpr at 020e0000 {
-				compatible = "fsl,imx6q-iomuxc-gpr", "syscon";
+				compatible = "fsl,imx6q-iomuxc-gpr", "syscon", "simple-mfd";
 				reg = <0x020e0000 0x38>;
+				#address-cells = <1>;
+				#size-cells = <0>;
 			};
 
 			iomuxc: iomuxc at 020e0000 {
@@ -1127,6 +1129,8 @@
 			mipi_csi: mipi at 021dc000 {
 				compatible = "fsl,imx6-mipi-csi2";
 				reg = <0x021dc000 0x4000>;
+				#address-cells = <1>;
+				#size-cells = <0>;
 				interrupts = <0 100 0x04>, <0 101 0x04>;
 				clocks = <&clks IMX6QDL_CLK_HSI_TX>,
 					 <&clks IMX6QDL_CLK_VIDEO_27M>,
@@ -1232,6 +1236,10 @@
 
 			ipu1_csi0: port at 0 {
 				reg = <0>;
+
+				ipu1_csi0_from_ipu1_csi0_mux: endpoint {
+					remote-endpoint = <&ipu1_csi0_mux_to_ipu1_csi0>;
+				};
 			};
 
 			ipu1_csi1: port at 1 {
Signature · 1 line
2.7.4
author maintainer
#10 lore raw

PATCH v3 4/24 ARM: dts: imx6qdl: add media device

In reply to Steve Longerbeam
---
 arch/arm/boot/dts/imx6q.dtsi   | 4 ++++
 arch/arm/boot/dts/imx6qdl.dtsi | 8 ++++++++
 2 files changed, 12 insertions(+)
arch/arm/boot/dts/imx6q.dtsi +4 -0
index 9b2ca32..8867e78 100644
@@ -458,3 +458,7 @@
 &vpu {
 	compatible = "fsl,imx6q-vpu", "cnm,coda960";
 };
+
+&media0 {
+	ports = <&ipu1_csi0>, <&ipu1_csi1>, <&ipu2_csi0>, <&ipu2_csi1>;
+};
arch/arm/boot/dts/imx6qdl.dtsi +8 -0
index 010388c..cbb42ec 100644
@@ -1296,5 +1296,13 @@
 				};
 			};
 		};
+
+		media0: media at 0 {
+			compatible = "fsl,imx-media";
+			ports = <&ipu1_csi0>, <&ipu1_csi1>;
+			#address-cells = <1>;
+			#size-cells = <0>;
+			status = "okay";
+		};
 	};
 };
Signature · 1 line
2.7.4
author maintainer
#11 lore raw

PATCH v3 5/24 ARM: dts: imx6qdl-sabrelite: remove erratum ERR006687 workaround

In reply to Steve Longerbeam

There is a pin conflict with GPIO_6. This pin functions as a power input pin to the OV5642 camera sensor, but ENET uses it as the h/w workaround for erratum ERR006687, to wake-up the ARM cores on normal RX and TX packet done events. So we need to remove the h/w workaround to support the OV5642. The result is that the CPUidle driver will no longer allow entering the deep idle states on the sabrelite.

This is a partial revert of

commit 6261c4c8f13e ("ARM: dts: imx6qdl-sabrelite: use GPIO_6 for FEC
interrupt.")
commit a28eeb43ee57 ("ARM: dts: imx6: tag boards that have the HW workaround

			for ERR006687")
---
 arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 4 ----
 1 file changed, 4 deletions(-)
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi +0 -4
index 1f9076e..795b5a5 100644
@@ -271,9 +271,6 @@
 	txd1-skew-ps = <0>;
 	txd2-skew-ps = <0>;
 	txd3-skew-ps = <0>;
-	interrupts-extended = <&gpio1 6 IRQ_TYPE_LEVEL_HIGH>,
-			      <&intc 0 119 IRQ_TYPE_LEVEL_HIGH>;
-	fsl,err006687-workaround-present;
 	status = "okay";
 };
 
@@ -374,7 +371,6 @@
 				MX6QDL_PAD_RGMII_RX_CTL__RGMII_RX_CTL	0x1b030
 				/* Phy reset */
 				MX6QDL_PAD_EIM_D23__GPIO3_IO23		0x000b0
-				MX6QDL_PAD_GPIO_6__ENET_IRQ		0x000b1
 			>;
 		};
 
Signature · 1 line
2.7.4
maintainer
#12 lore raw

Re: [PATCH v3 05/24] ARM: dts: imx6qdl-sabrelite: remove erratum ERR006687 workaround

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:

Quoted 7 lines

There is a pin conflict with GPIO_6. This pin functions as a power input pin to the OV5642 camera sensor, but ENET uses it as the h/w workaround for erratum ERR006687, to wake-up the ARM cores on normal RX and TX packet done events. So we need to remove the h/w workaround to support the OV5642. The result is that the CPUidle driver will no longer allow entering the deep idle states on the sabrelite.

Quoted 8 lines

This is a partial revert of

commit 6261c4c8f13e ("ARM: dts: imx6qdl-sabrelite: use GPIO_6 for FEC
interrupt.")
commit a28eeb43ee57 ("ARM: dts: imx6: tag boards that have the HW workaround

			for ERR006687")

Pity this has to be removed for all, Nitrogen6X should really use DT overlays for the add-on boards / cameras.

regards
Philipp

author maintainer
#13 lore raw

PATCH v3 6/24 ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors

In reply to Steve Longerbeam

Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor. Both hang off the same i2c2 bus, so they require different (and non- default) i2c slave addresses.

The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.

The OV5640 connects to the input port on the MIPI CSI-2 receiver on mipi_csi. It is set to transmit over MIPI virtual channel 1.

---
 arch/arm/boot/dts/imx6dl-sabrelite.dts   |   5 ++
 arch/arm/boot/dts/imx6q-sabrelite.dts    |   6 ++
 arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 +++++++++++++++++++++++++++++++
 3 files changed, 129 insertions(+)
arch/arm/boot/dts/imx6dl-sabrelite.dts +5 -0
index 0f06ca5..fec2524 100644
@@ -48,3 +48,8 @@
 	model = "Freescale i.MX6 DualLite SABRE Lite Board";
 	compatible = "fsl,imx6dl-sabrelite", "fsl,imx6dl";
 };
+
+&ipu1_csi1_from_ipu1_csi1_mux {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
arch/arm/boot/dts/imx6q-sabrelite.dts +6 -0
index 66d10d8..9e2d26d 100644
@@ -52,3 +52,9 @@
 &sata {
 	status = "okay";
 };
+
+&ipu1_csi1_from_mipi_vc1 {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
+
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi +118 -0

157 lines not on this page · raw message

Signature · 1 line
2.7.4
maintainer
#14 lore raw

Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:

Quoted 98 lines

Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor. Both hang off the same i2c2 bus, so they require different (and non- default) i2c slave addresses.

The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.

The OV5640 connects to the input port on the MIPI CSI-2 receiver on mipi_csi. It is set to transmit over MIPI virtual channel 1.

---
 arch/arm/boot/dts/imx6dl-sabrelite.dts   |   5 ++
 arch/arm/boot/dts/imx6q-sabrelite.dts    |   6 ++
 arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 +++++++++++++++++++++++++++++++
 3 files changed, 129 insertions(+)
arch/arm/boot/dts/imx6dl-sabrelite.dts +5 -0
index 0f06ca5..fec2524 100644
@@ -48,3 +48,8 @@
 	model = "Freescale i.MX6 DualLite SABRE Lite Board";
 	compatible = "fsl,imx6dl-sabrelite", "fsl,imx6dl";
 };
+
+&ipu1_csi1_from_ipu1_csi1_mux {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
arch/arm/boot/dts/imx6q-sabrelite.dts +6 -0
index 66d10d8..9e2d26d 100644
@@ -52,3 +52,9 @@
 &sata {
 	status = "okay";
 };
+
+&ipu1_csi1_from_mipi_vc1 {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
+
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi +27 -0
index 795b5a5..bca9fed 100644
@@ -39,6 +39,8 @@
  *     FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
  *     OTHER DEALINGS IN THE SOFTWARE.
  */
+
+#include <dt-bindings/clock/imx6qdl-clock.h>
 #include <dt-bindings/gpio/gpio.h>
 #include <dt-bindings/input/input.h>
 
@@ -96,6 +98,15 @@
 		};
 	};
 
+	mipi_xclk: mipi_xclk {
+		compatible = "pwm-clock";
+		#clock-cells = <0>;
+		clock-frequency = <22000000>;
+		clock-output-names = "mipi_pwm3";
+		pwms = <&pwm3 0 45>; /* 1 / 45 ns = 22 MHz */
+		status = "okay";
+	};
+
 	gpio-keys {
 		compatible = "gpio-keys";
 		pinctrl-names = "default";
@@ -220,6 +231,22 @@
 	};
 };
 
+&ipu1_csi0_from_ipu1_csi0_mux {
+	bus-width = <8>;
+	data-shift = <12>; /* Lines 19:12 used */
+	hsync-active = <1>;
+	vync-active = <1>;
+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+	remote-endpoint = <&ov5642_to_ipu1_csi0_mux>;
+};
+
+&ipu1_csi0 {
+	pinctrl-names = "default";
+	pinctrl-0 = <&pinctrl_ipu1_csi0>;
+};
+
 &audmux {
 	pinctrl-names = "default";
 	pinctrl-0 = <&pinctrl_audmux>;
@@ -299,6 +326,52 @@
 	pinctrl-names = "default";
 	pinctrl-0 = <&pinctrl_i2c2>;

arch/arm/boot/dts/imx6qdl-sabrelite.dtsi line 337

Quoted 10 lines
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi +9 -0
 	status = "okay";
+
+	ov5640: camera at 40 {
+		compatible = "ovti,ov5640";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_ov5640>;
+		clocks = <&mipi_xclk>;
+		clock-names = "xclk";
+		reg = <0x40>;
+		xclk = <22000000>;

This is superfluous, you can use clk_get_rate on mipi_xclk.

Quoted 14 lines
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi +14 -0
+		reset-gpios = <&gpio2 5 GPIO_ACTIVE_LOW>; /* NANDF_D5 */
+		pwdn-gpios = <&gpio6 9 GPIO_ACTIVE_HIGH>; /* NANDF_WP_B */
+
+		port {
+			#address-cells = <1>;
+			#size-cells = <0>;
+
+			ov5640_to_mipi_csi: endpoint at 1 {
+				reg = <1>;
+				remote-endpoint = <&mipi_csi_from_mipi_sensor>;
+				data-lanes = <0 1>;
+				clock-lanes = <2>;
+			};
+		};

arch/arm/boot/dts/imx6qdl-sabrelite.dtsi line 361

Quoted 10 lines
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi +10 -0
+	};
+
+	ov5642: camera at 42 {
+		compatible = "ovti,ov5642";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_ov5642>;
+		clocks = <&clks IMX6QDL_CLK_CKO2>;
+		clock-names = "xclk";
+		reg = <0x42>;
+		xclk = <24000000>;

Same here, use assigned-clock-rates on IMX6QDL_CLK_CKO2 if necessary.

Quoted 69 lines
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi +48 -0
+		reset-gpios = <&gpio1 8 GPIO_ACTIVE_LOW>;
+		pwdn-gpios = <&gpio1 6 GPIO_ACTIVE_HIGH>;
+		gp-gpios = <&gpio1 16 GPIO_ACTIVE_HIGH>;
+
+		port {
+			ov5642_to_ipu1_csi0_mux: endpoint {
+				remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
+				bus-width = <8>;
+				hsync-active = <1>;
+				vsync-active = <1>;
+			};
+		};
+	};
 };
 
 &i2c3 {
@@ -412,6 +485,23 @@
 			>;
 		};
 
+		pinctrl_ipu1_csi0: ipu1csi0grp {
+			fsl,pins = <
+				MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18    0x1b0b0
+				MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19    0x1b0b0
+				MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK   0x1b0b0
+				MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC      0x1b0b0
+				MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC     0x1b0b0
+				MX6QDL_PAD_CSI0_DATA_EN__IPU1_CSI0_DATA_EN 0x1b0b0
+			>;
+		};
+
 		pinctrl_j15: j15grp {
 			fsl,pins = <
 				MX6QDL_PAD_DI0_DISP_CLK__IPU1_DI0_DISP_CLK 0x10
@@ -445,6 +535,22 @@
 			>;
 		};
 
+		pinctrl_ov5640: ov5640grp {
+			fsl,pins = <
+				MX6QDL_PAD_NANDF_D5__GPIO2_IO05   0x000b0
+				MX6QDL_PAD_NANDF_WP_B__GPIO6_IO09 0x0b0b0
+			>;
+		};
+
+		pinctrl_ov5642: ov5642grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD1_DAT0__GPIO1_IO16 0x1b0b0
+				MX6QDL_PAD_GPIO_6__GPIO1_IO06   0x1b0b0
+				MX6QDL_PAD_GPIO_8__GPIO1_IO08   0x130b0
+				MX6QDL_PAD_GPIO_3__CCM_CLKO2    0x000b0
+			>;
+		};
+
 		pinctrl_pwm1: pwm1grp {
 			fsl,pins = <
 				MX6QDL_PAD_SD1_DAT3__PWM1_OUT 0x1b0b1
@@ -601,3 +707,15 @@
 	vmmc-supply = <&reg_3p3v>;
 	status = "okay";
 };
+
+&mipi_csi {

arch/arm/boot/dts/imx6qdl-sabrelite.dtsi line 721

Quoted 10 lines
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi +10 -0
+        status = "okay";
+};
+
+/* Incoming port from sensor */
+&mipi_csi_from_mipi_sensor {
+        remote-endpoint = <&ov5640_to_mipi_csi>;
+        data-lanes = <0 1>;
+        clock-lanes = <2>;
+};
+

regards
Philipp

author maintainer
#15 lore raw

Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors

Quoted 102 lines · Philipp Zabel

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:

Quoted 101 lines

Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor. Both hang off the same i2c2 bus, so they require different (and non- default) i2c slave addresses.

The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.

The OV5640 connects to the input port on the MIPI CSI-2 receiver on mipi_csi. It is set to transmit over MIPI virtual channel 1.

---
  arch/arm/boot/dts/imx6dl-sabrelite.dts   |   5 ++
  arch/arm/boot/dts/imx6q-sabrelite.dts    |   6 ++
  arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 +++++++++++++++++++++++++++++++
  3 files changed, 129 insertions(+)
arch/arm/boot/dts/imx6dl-sabrelite.dts +5 -0
index 0f06ca5..fec2524 100644
@@ -48,3 +48,8 @@
  	model = "Freescale i.MX6 DualLite SABRE Lite Board";
  	compatible = "fsl,imx6dl-sabrelite", "fsl,imx6dl";
  };
+
+&ipu1_csi1_from_ipu1_csi1_mux {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
arch/arm/boot/dts/imx6q-sabrelite.dts +6 -0
index 66d10d8..9e2d26d 100644
@@ -52,3 +52,9 @@
  &sata {
  	status = "okay";
  };
+
+&ipu1_csi1_from_mipi_vc1 {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
+
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi +29 -0
index 795b5a5..bca9fed 100644
@@ -39,6 +39,8 @@
   *     FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
   *     OTHER DEALINGS IN THE SOFTWARE.
   */
+
+#include <dt-bindings/clock/imx6qdl-clock.h>
  #include <dt-bindings/gpio/gpio.h>
  #include <dt-bindings/input/input.h>
 
@@ -96,6 +98,15 @@
  		};
  	};
 
+	mipi_xclk: mipi_xclk {
+		compatible = "pwm-clock";
+		#clock-cells = <0>;
+		clock-frequency = <22000000>;
+		clock-output-names = "mipi_pwm3";
+		pwms = <&pwm3 0 45>; /* 1 / 45 ns = 22 MHz */
+		status = "okay";
+	};
+
  	gpio-keys {
  		compatible = "gpio-keys";
  		pinctrl-names = "default";
@@ -220,6 +231,22 @@
  	};
  };
 
+&ipu1_csi0_from_ipu1_csi0_mux {
+	bus-width = <8>;
+	data-shift = <12>; /* Lines 19:12 used */
+	hsync-active = <1>;
+	vync-active = <1>;
+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+	remote-endpoint = <&ov5642_to_ipu1_csi0_mux>;
+};
+
+&ipu1_csi0 {
+	pinctrl-names = "default";
+	pinctrl-0 = <&pinctrl_ipu1_csi0>;
+};
+
  &audmux {
  	pinctrl-names = "default";
  	pinctrl-0 = <&pinctrl_audmux>;
@@ -299,6 +326,52 @@
  	pinctrl-names = "default";
  	pinctrl-0 = <&pinctrl_i2c2>;
  	status = "okay";
+
+	ov5640: camera at 40 {
Quoted 8 lines

arch/arm/boot/dts/imx6qdl-sabrelite.dtsi line 337

Quoted 7 lines
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi +7 -0
+		compatible = "ovti,ov5640";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_ov5640>;
+		clocks = <&mipi_xclk>;
+		clock-names = "xclk";
+		reg = <0x40>;
+		xclk = <22000000>;

This is superfluous, you can use clk_get_rate on mipi_xclk.

This property is actually there to tell the driver what to set the rate to, with clk_set_rate(). So you are saying it would be better to set the rate in the device tree and the driver should only retrieve the rate?

Steve

maintainer
#16 lore raw

Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors

Quoted 23 lines · Steve Longerbeam
Quoted 21 lines · Philipp Zabel

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:

Quoted 20 lines

Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor. Both hang off the same i2c2 bus, so they require different (and non- default) i2c slave addresses.

The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.

The OV5640 connects to the input port on the MIPI CSI-2 receiver on mipi_csi. It is set to transmit over MIPI virtual channel 1.

---
  arch/arm/boot/dts/imx6dl-sabrelite.dts   |   5 ++
  arch/arm/boot/dts/imx6q-sabrelite.dts    |   6 ++
  arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 +++++++++++++++++++++++++++++++
  3 files changed, 129 insertions(+)
arch/arm/boot/dts/imx6dl-sabrelite.dts +0 -0
index 0f06ca5..fec2524 100644

[...]

Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
arch/arm/boot/dts/imx6dl-sabrelite.dts +2 -0
@@ -299,6 +326,52 @@
  	pinctrl-names = "default";
  	pinctrl-0 = <&pinctrl_i2c2>;
  	status = "okay";
+
+	ov5640: camera at 40 {
Quoted 8 lines

arch/arm/boot/dts/imx6dl-sabrelite.dts line 337

Quoted 7 lines
arch/arm/boot/dts/imx6dl-sabrelite.dts +7 -0
+		compatible = "ovti,ov5640";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_ov5640>;
+		clocks = <&mipi_xclk>;
+		clock-names = "xclk";
+		reg = <0x40>;
+		xclk = <22000000>;

This is superfluous, you can use clk_get_rate on mipi_xclk.

Quoted 4 lines

This property is actually there to tell the driver what to set the rate to, with clk_set_rate(). So you are saying it would be better to set the rate in the device tree and the driver should only retrieve the rate?

Yes. Given that this is a reference clock input that is constant on a given board and never changes during runtime, I think this is the correct way. The clock will be fixed rate on most boards, I assume.

regards
Philipp

author maintainer
#17 lore raw

Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors

Quoted 16 lines · Philipp Zabel
Quoted 14 lines · Steve Longerbeam
Quoted 14 lines

arch/arm/boot/dts/imx6qdl-sabrelite.dtsi line 337

Quoted 13 lines
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi +9 -0
@@ -299,6 +326,52 @@
   	pinctrl-names = "default";
   	pinctrl-0 = <&pinctrl_i2c2>;
   	status = "okay";
+
+	ov5640: camera at 40 {
+		compatible = "ovti,ov5640";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_ov5640>;
+		clocks = <&mipi_xclk>;
+		clock-names = "xclk";
+		reg = <0x40>;
+		xclk = <22000000>;

This is superfluous, you can use clk_get_rate on mipi_xclk.

Quoted 7 lines
Quoted 4 lines

This property is actually there to tell the driver what to set the rate to, with clk_set_rate(). So you are saying it would be better to set the rate in the device tree and the driver should only retrieve the rate?

Yes. Given that this is a reference clock input that is constant on a given board and never changes during runtime, I think this is the correct way. The clock will be fixed rate on most boards, I assume.

Ok, that makes sense, I'll make that change.

Steve

maintainer
#18 lore raw

Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors

In reply to Philipp Zabel
Quoted 47 lines · On Monday 16 Jan 2017 13:55:23 Philipp Zabel wrote:
Quoted 25 lines · Steve Longerbeam
Quoted 24 lines · Philipp Zabel

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:

Quoted 23 lines

Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.
Both hang off the same i2c2 bus, so they require different (and non- default) i2c slave addresses.

The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.

The OV5640 connects to the input port on the MIPI CSI-2 receiver on mipi_csi. It is set to transmit over MIPI virtual channel 1.

---

  arch/arm/boot/dts/imx6dl-sabrelite.dts   |   5 ++
  arch/arm/boot/dts/imx6q-sabrelite.dts    |   6 ++
  arch/arm/boot/dts/imx6qdl-sabrelite.dtsi | 118 ++++++++++++++++++++++
  3 files changed, 129 insertions(+)
(quoted diff) +0 -0

b/arch/arm/boot/dts/imx6dl-sabrelite.dts index 0f06ca5..fec2524 100644

arch/arm/boot/dts/imx6dl-sabrelite.dts +0 -0

[...]

Quoted 18 lines
Quoted 17 lines
Quoted 5 lines
arch/arm/boot/dts/imx6dl-sabrelite.dts +0 -0
@@ -299,6 +326,52 @@
 
  	pinctrl-names = "default";
  	pinctrl-0 = <&pinctrl_i2c2>;
  	status = "okay";

arch/arm/boot/dts/imx6dl-sabrelite.dts line 339

Quoted 10 lines
arch/arm/boot/dts/imx6dl-sabrelite.dts +9 -0
 
+
+	ov5640: camera@40 {
+		compatible = "ovti,ov5640";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_ov5640>;
+		clocks = <&mipi_xclk>;
+		clock-names = "xclk";
+		reg = <0x40>;
+		xclk = <22000000>;

This is superfluous, you can use clk_get_rate on mipi_xclk.

Quoted 8 lines
Quoted 4 lines

This property is actually there to tell the driver what to set the rate to, with clk_set_rate(). So you are saying it would be better to set the rate in the device tree and the driver should only retrieve the rate?

Yes. Given that this is a reference clock input that is constant on a given board and never changes during runtime, I think this is the correct way. The clock will be fixed rate on most boards, I assume.

I think it's a bit worse than that. The ov5640 and ov5642 drivers should retrieve the clock rate and compute register values accordingly (PLL configuration parameters for instance, but most probably other values as well). Unfortunately, as usual with Omnivision, the lack of public and even non-public information results in drivers hardcoding large lists of register/value pairs that have been computed for a specific clock frequency. The drivers will thus not operate correctly if the clock is running at a different rate. Until that can be fixed, the best option is probably to assign the rate in the device tree as Philipp proposed, and to use clk_get_rate() in the driver to reject any rate other than 22MHz.

Signature · 3 lines
Regards,

Laurent Pinchart
#19 lore raw

Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors

In reply to Steve Longerbeam
Quoted 18 lines · Steve Longerbeam
arch/arm/boot/dts/imx6q-sabrelite.dts +6 -0
index 66d10d8..9e2d26d 100644
@@ -52,3 +52,9 @@
 &sata {
 	status = "okay";
 };
+
+&ipu1_csi1_from_mipi_vc1 {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
+
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi +0 -0
index 795b5a5..bca9fed 100644

...

arch/arm/boot/dts/imx6qdl-sabrelite.dtsi line 721

Quoted 7 lines
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi +7 -0
+/* Incoming port from sensor */
+&mipi_csi_from_mipi_sensor {
+        remote-endpoint = <&ov5640_to_mipi_csi>;
+        data-lanes = <0 1>;
+        clock-lanes = <2>;
+};
+

Applying: ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors .git/rebase-apply/patch:33: new blank line at EOF.
+
.git/rebase-apply/patch:201: new blank line at EOF.
+
warning: 2 lines add whitespace errors.

Signature · 3 lines
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
#20 lore raw

Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors

In reply to Steve Longerbeam
Quoted 10 lines · Steve Longerbeam
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi +10 -0
+	ov5640: camera at 40 {
+		compatible = "ovti,ov5640";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_ov5640>;
+		clocks = <&mipi_xclk>;
+		clock-names = "xclk";
+		reg = <0x40>;
+		xclk = <22000000>;
+		reset-gpios = <&gpio2 5 GPIO_ACTIVE_LOW>; /* NANDF_D5 */
+		pwdn-gpios = <&gpio6 9 GPIO_ACTIVE_HIGH>; /* NANDF_WP_B */

arch/arm/boot/dts/imx6qdl-sabrelite.dtsi line 349

Quoted 10 lines
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi +10 -0
+
+		port {
+			#address-cells = <1>;
+			#size-cells = <0>;
+
+			ov5640_to_mipi_csi: endpoint at 1 {
+				reg = <1>;
+				remote-endpoint = <&mipi_csi_from_mipi_sensor>;
+				data-lanes = <0 1>;
+				clock-lanes = <2>;

How do you envision a four-lane sensor being described?

	data-lanes = <0 1 3 4>;
	clock-lanes = <2>;

?

The binding document for video-interfaces.txt says:

- clock-lanes: an array of physical clock lane indexes. Position of an entry determines the logical lane number, while the value of an entry indicates physical lane, e.g. for a MIPI CSI-2 bus we could have "clock-lanes = <0>;", which places the clock lane on hardware lane 0. This property is valid for serial busses only (e.g. MIPI CSI-2). Note that for the MIPI CSI-2 bus this array contains only one entry.

So I think you need to have a good reason to make the clock lane non-zero.

Signature · 3 lines
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
maintainer
#21 lore raw

Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors

Hi Russell,

Quoted 11 lines · On Monday 30 Jan 2017 22:51:33 Russell King - ARM Linux wrote:
Quoted 10 lines · Steve Longerbeam
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi +10 -0
+	ov5640: camera@40 {
+		compatible = "ovti,ov5640";
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_ov5640>;
+		clocks = <&mipi_xclk>;
+		clock-names = "xclk";
+		reg = <0x40>;
+		xclk = <22000000>;
+		reset-gpios = <&gpio2 5 GPIO_ACTIVE_LOW>; /* NANDF_D5 */
+		pwdn-gpios = <&gpio6 9 GPIO_ACTIVE_HIGH>; /* NANDF_WP_B */
Quoted 8 lines

arch/arm/boot/dts/imx6qdl-sabrelite.dtsi

Quoted 8 lines
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi +8 -0
+
+		port {
+			#address-cells = <1>;
+			#size-cells = <0>;
+
+			ov5640_to_mipi_csi: endpoint@1 {
+				reg = <1>;
+				remote-endpoint = 

<&mipi_csi_from_mipi_sensor>;

Quoted 12 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+				data-lanes = <0 1>;
+				clock-lanes = <2>;

How do you envision a four-lane sensor being described?

	data-lanes = <0 1 3 4>;
	clock-lanes = <2>;

?

The binding document for video-interfaces.txt says:

Quoted 8 lines

- clock-lanes: an array of physical clock lane indexes. Position of an entry determines the logical lane number, while the value of an entry indicates physical lane, e.g. for a MIPI CSI-2 bus we could have "clock-lanes = <0>;", which places the clock lane on hardware lane 0. This property is valid for serial busses only (e.g. MIPI CSI-2). Note that for the MIPI CSI-2 bus this array contains only one entry.

So I think you need to have a good reason to make the clock lane non-zero.

The purpose of the data-lanes and clock-lanes properties is to describe lane assignment for hardware that supports lane routing. As far as I know the OV5640 doesn't support lane routing and has dedicated pins for the clock and data lanes. The data-lanes and clock-lanes properties should probably not be specified at all.

Signature · 3 lines
Regards,

Laurent Pinchart
#22 lore raw

Re: [PATCH v3 06/24] ARM: dts: imx6-sabrelite: add OV5642 and OV5640 camera sensors

Quoted 26 lines · Laurent Pinchart

Hi Russell,

Quoted 3 lines · On Monday 30 Jan 2017 22:51:33 Russell King - ARM Linux wrote:

arch/arm/boot/dts/imx6qdl-sabrelite.dtsi

Quoted 3 lines
arch/arm/boot/dts/imx6qdl-sabrelite.dtsi +3 -0
+			ov5640_to_mipi_csi: endpoint at 1 {
+				reg = <1>;
+				remote-endpoint = 

<&mipi_csi_from_mipi_sensor>;

Quoted 19 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+				data-lanes = <0 1>;
+				clock-lanes = <2>;

How do you envision a four-lane sensor being described?

	data-lanes = <0 1 3 4>;
	clock-lanes = <2>;

?

The binding document for video-interfaces.txt says:

- clock-lanes: an array of physical clock lane indexes. Position of an entry determines the logical lane number, while the value of an entry indicates physical lane, e.g. for a MIPI CSI-2 bus we could have "clock-lanes = <0>;", which places the clock lane on hardware lane 0. This property is valid for serial busses only (e.g. MIPI CSI-2). Note that for the MIPI CSI-2 bus this array contains only one entry.

Quoted 7 lines
Quoted 1 line

So I think you need to have a good reason to make the clock lane non-zero.

The purpose of the data-lanes and clock-lanes properties is to describe lane assignment for hardware that supports lane routing. As far as I know the OV5640 doesn't support lane routing and has dedicated pins for the clock and data lanes. The data-lanes and clock-lanes properties should probably not be specified at all.

You need at least data-lanes so you know how many data lanes are wired between the camera and the mipi csi2 receiver. Just because a camera has (eg) four lanes does not mean you need to wire all four, and in some cases less than four will be wired.

If data-lanes does not describe that, then all existing users of the binding are abusing it:

$ grep data_lanes drivers/media/i2c -r
drivers/media/i2c/s5k5baf.c:                state->nlanes = ep.bus.mipi_csi2.num_data_lanes;
drivers/media/i2c/s5c73m3/s5c73m3-core.c:   if (ep.bus.mipi_csi2.num_data_lanes != S5C73M3_MIPI_DATA_LANES)
drivers/media/i2c/tc358743.c:           endpoint->bus.mipi_csi2.num_data_lanes == 0 ||
drivers/media/i2c/smiapp/smiapp-core.c:     hwcfg->lanes = bus_cfg->bus.mipi_csi2.num_data_lanes;

So all those drivers are using it for the _number_ of CSI2 lanes, and are not touching the mapping in any way (not even checking that it is an identity mapping.) You could specify any mapping to these drivers, as long as num_data_lanes came out right.

And... there's no point having a property in a binding if no one is using it... and even more silly not to have a property that everyone needs...

Signature · 3 lines
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
author maintainer
#23 lore raw

PATCH v3 7/24 ARM: dts: imx6-sabresd: add OV5642 and OV5640 camera sensors

In reply to Steve Longerbeam

Enables the OV5642 parallel-bus sensor, and the OV5640 MIPI CSI-2 sensor.

The OV5642 connects to the parallel-bus mux input port on ipu1_csi0_mux.

The OV5640 connects to the input port on the MIPI CSI-2 receiver on mipi_csi. It is set to transmit over MIPI virtual channel 1.

Until the OV5652 sensor module compatible with the SabreSD becomes available for testing, the ov5642 node is currently disabled.

---
 arch/arm/boot/dts/imx6dl-sabresd.dts   |   5 ++
 arch/arm/boot/dts/imx6q-sabresd.dts    |   5 ++
 arch/arm/boot/dts/imx6qdl-sabresd.dtsi | 114 ++++++++++++++++++++++++++++++++-
 3 files changed, 123 insertions(+), 1 deletion(-)
arch/arm/boot/dts/imx6dl-sabresd.dts +5 -0
index 1e45f2f..6cf7a50 100644
@@ -15,3 +15,8 @@
 	model = "Freescale i.MX6 DualLite SABRE Smart Device Board";
 	compatible = "fsl,imx6dl-sabresd", "fsl,imx6dl";
 };
+
+&ipu1_csi1_from_ipu1_csi1_mux {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
arch/arm/boot/dts/imx6q-sabresd.dts +5 -0
index 9cbdfe7..8c1d7ad 100644
@@ -23,3 +23,8 @@
 &sata {
 	status = "okay";
 };
+
+&ipu1_csi1_from_mipi_vc1 {
+	data-lanes = <0 1>;
+	clock-lanes = <2>;
+};
arch/arm/boot/dts/imx6qdl-sabresd.dtsi +113 -1

144 lines not on this page · raw message

Signature · 1 line
2.7.4
author maintainer
#24 lore raw

PATCH v3 8/24 ARM: dts: imx6-sabreauto: create i2cmux for i2c3

In reply to Steve Longerbeam

The sabreauto uses a steering pin to select between the SDA signal on i2c3 bus, and a data-in pin for an SPI NOR chip. Use i2cmux to control this steering pin. Idle state of the i2cmux selects SPI NOR. This is not a classic way to use i2cmux, since one side of the mux selects something other than an i2c bus, but it works and is probably the cleanest solution. Note that if one thread is attempting to access SPI NOR while another thread is accessing i2c3, the SPI NOR access will fail since the i2cmux has selected the SDA pin rather than SPI NOR data-in. This couldn't be avoided in any case, the board is not designed to allow concurrent i2c3 and SPI NOR functions (and the default device-tree does not enable SPI NOR anyway).

Devices hanging off i2c3 should now be defined under i2cmux, so that the steering pin can be properly controlled to access those devices. The port expanders (MAX7310) are thus moved into i2cmux.

---
 arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 65 +++++++++++++++++++++-----------
 1 file changed, 44 insertions(+), 21 deletions(-)
arch/arm/boot/dts/imx6qdl-sabreauto.dtsi +44 -21
index 52390ba..cace88c 100644
@@ -108,6 +108,44 @@
 		default-brightness-level = <7>;
 		status = "okay";
 	};
+
+	i2cmux {
+		compatible = "i2c-mux-gpio";
+		#address-cells = <1>;
+		#size-cells = <0>;
+		pinctrl-names = "default";
+		pinctrl-0 = <&pinctrl_i2c3mux>;
+		mux-gpios = <&gpio5 4 0>;
+		i2c-parent = <&i2c3>;
+		idle-state = <0>;
+
+		i2c at 1 {
+			#address-cells = <1>;
+			#size-cells = <0>;
+			reg = <1>;
+
+			max7310_a: gpio at 30 {
+				compatible = "maxim,max7310";
+				reg = <0x30>;
+				gpio-controller;
+				#gpio-cells = <2>;
+			};
+
+			max7310_b: gpio at 32 {
+				compatible = "maxim,max7310";
+				reg = <0x32>;
+				gpio-controller;
+				#gpio-cells = <2>;
+			};
+
+			max7310_c: gpio at 34 {
+				compatible = "maxim,max7310";
+				reg = <0x34>;
+				gpio-controller;
+				#gpio-cells = <2>;
+			};
+		};
+	};
 };
 
 &clks {
@@ -291,27 +329,6 @@
 	pinctrl-names = "default";
 	pinctrl-0 = <&pinctrl_i2c3>;
 	status = "okay";
-
-	max7310_a: gpio at 30 {
-		compatible = "maxim,max7310";
-		reg = <0x30>;
-		gpio-controller;
-		#gpio-cells = <2>;
-	};
-
-	max7310_b: gpio at 32 {
-		compatible = "maxim,max7310";
-		reg = <0x32>;
-		gpio-controller;
-		#gpio-cells = <2>;
-	};
-
-	max7310_c: gpio at 34 {
-		compatible = "maxim,max7310";
-		reg = <0x34>;
-		gpio-controller;
-		#gpio-cells = <2>;
-	};
 };
 
 &iomuxc {
@@ -419,6 +436,12 @@
 			>;
 		};
 
+		pinctrl_i2c3mux: i2c3muxgrp {
+			fsl,pins = <
+				MX6QDL_PAD_EIM_A24__GPIO5_IO04 0x0b0b1
+			>;
+		};
+
 		pinctrl_pwm3: pwm1grp {
 			fsl,pins = <
 				MX6QDL_PAD_SD4_DAT1__PWM3_OUT		0x1b0b1
Signature · 1 line
2.7.4
author maintainer
#25 lore raw

PATCH v3 9/24 ARM: dts: imx6-sabreauto: add reset-gpios property for max7310_b

In reply to Steve Longerbeam

The reset pin to the port expander chip (MAX7310) is controlled by a gpio, so define a reset-gpios property to control it. There are three MAX7310's on the SabreAuto CPU card (max7310_[abc]), but all use the same pin for their reset. Since all can't acquire the same pin, assign it to max7310_b, that chip is needed by more functions (usb and adv7180).

---
 arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 9 +++++++++
 1 file changed, 9 insertions(+)
arch/arm/boot/dts/imx6qdl-sabreauto.dtsi +9 -0
index cace88c..967c3b8 100644
@@ -136,6 +136,9 @@
 				reg = <0x32>;
 				gpio-controller;
 				#gpio-cells = <2>;
+				pinctrl-names = "default";
+				pinctrl-0 = <&pinctrl_max7310>;
+				reset-gpios = <&gpio1 15 GPIO_ACTIVE_LOW>;
 			};
 
 			max7310_c: gpio at 34 {
@@ -442,6 +445,12 @@
 			>;
 		};
 
+		pinctrl_max7310: max7310grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD2_DAT0__GPIO1_IO15 0x1b0b0
+			>;
+		};
+
 		pinctrl_pwm3: pwm1grp {
 			fsl,pins = <
 				MX6QDL_PAD_SD4_DAT1__PWM3_OUT		0x1b0b1
Signature · 1 line
2.7.4
author maintainer
#26 lore raw

PATCH v3 10/24 ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture

In reply to Steve Longerbeam

Add pinctrl groups for both GPT input capture channels.

---
 arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 12 ++++++++++++
 1 file changed, 12 insertions(+)
arch/arm/boot/dts/imx6qdl-sabreauto.dtsi +12 -0
index 967c3b8..495709f 100644
@@ -457,6 +457,18 @@
 			>;
 		};
 
+		pinctrl_gpt_input_capture0: gptinputcapture0grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD1_DAT0__GPT_CAPTURE1	0x1b0b0
+			>;
+		};
+
+		pinctrl_gpt_input_capture1: gptinputcapture1grp {
+			fsl,pins = <
+				MX6QDL_PAD_SD1_DAT1__GPT_CAPTURE2	0x1b0b0
+			>;
+		};
+
 		pinctrl_spdif: spdifgrp {
 			fsl,pins = <
 				MX6QDL_PAD_KEY_COL3__SPDIF_IN 0x1b0b0
Signature · 1 line
2.7.4
#27 lore raw

Re: [PATCH v3 10/24] ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture

Quoted 21 lines · Steve Longerbeam

Add pinctrl groups for both GPT input capture channels.

---
 arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 12 ++++++++++++
 1 file changed, 12 insertions(+)
arch/arm/boot/dts/imx6qdl-sabreauto.dtsi +6 -0
index 967c3b8..495709f 100644
@@ -457,6 +457,18 @@
                        >;
                };
 
+               pinctrl_gpt_input_capture0: gptinputcapture0grp {
+                       fsl,pins = <
+                               MX6QDL_PAD_SD1_DAT0__GPT_CAPTURE1       0x1b0b0
+                       >;
+               };
+

arch/arm/boot/dts/imx6qdl-sabreauto.dtsi

Quoted 10 lines
arch/arm/boot/dts/imx6qdl-sabreauto.dtsi +6 -1
+               pinctrl_gpt_input_capture1: gptinputcapture1grp {
+                       fsl,pins = <
+                               MX6QDL_PAD_SD1_DAT1__GPT_CAPTURE2       0x1b0b0
+                       >;
+               };
+
                pinctrl_spdif: spdifgrp {
                        fsl,pins = <
                                MX6QDL_PAD_KEY_COL3__SPDIF_IN 0x1b0b0
--

Steve,

These are not used anywhere.

Tim

#28 lore raw

Re: [PATCH v3 10/24] ARM: dts: imx6-sabreauto: add pinctrl for gpt input capture

Quoted 34 lines · Tim Harvey
Quoted 21 lines · Steve Longerbeam

Add pinctrl groups for both GPT input capture channels.

---
  arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 12 ++++++++++++
  1 file changed, 12 insertions(+)
arch/arm/boot/dts/imx6qdl-sabreauto.dtsi +6 -0
index 967c3b8..495709f 100644
@@ -457,6 +457,18 @@
                         >;
                 };
 
+               pinctrl_gpt_input_capture0: gptinputcapture0grp {
+                       fsl,pins = <
+                               MX6QDL_PAD_SD1_DAT0__GPT_CAPTURE1       0x1b0b0
+                       >;
+               };
+

arch/arm/boot/dts/imx6qdl-sabreauto.dtsi

Quoted 10 lines
arch/arm/boot/dts/imx6qdl-sabreauto.dtsi +6 -1
+               pinctrl_gpt_input_capture1: gptinputcapture1grp {
+                       fsl,pins = <
+                               MX6QDL_PAD_SD1_DAT1__GPT_CAPTURE2       0x1b0b0
+                       >;
+               };
+
                 pinctrl_spdif: spdifgrp {
                         fsl,pins = <
                                 MX6QDL_PAD_KEY_COL3__SPDIF_IN 0x1b0b0
--

Steve,

Quoted 1 line

These are not used anywhere.

Yes, maybe I should just remove this patch for now. I'm only keeping it because eventually it will be needed to support i.MX6 input capture.

Steve

author maintainer
#29 lore raw

PATCH v3 11/24 ARM: dts: imx6-sabreauto: add the ADV7180 video decoder

In reply to Steve Longerbeam

Enables the ADV7180 decoder sensor. The ADV7180 connects to the parallel-bus mux input on ipu1_csi0_mux.

On the sabreauto, two analog video inputs are routed to the ADV7180, composite on Ain1, and composite on Ain3. Those inputs are defined via inputs and input-names under the ADV7180 node. The ADV7180 power pin is via max7310_b port expander.

---
 arch/arm/boot/dts/imx6qdl-sabreauto.dtsi | 61 ++++++++++++++++++++++++++++++++
 1 file changed, 61 insertions(+)
arch/arm/boot/dts/imx6qdl-sabreauto.dtsi +61 -0
index 495709f..7999857 100644
@@ -124,6 +124,24 @@
 			#size-cells = <0>;
 			reg = <1>;
 
+			adv7180: camera at 21 {
+				compatible = "adi,adv7180";
+				reg = <0x21>;
+				powerdown-gpios = <&max7310_b 2 GPIO_ACTIVE_LOW>;
+				interrupt-parent = <&gpio1>;
+				interrupts = <27 0x8>;
+				inputs = <0x00 0x02>;
+				input-names = "ADV7180 Composite on Ain1",
+						"ADV7180 Composite on Ain3";
+
+				port {
+					adv7180_to_ipu1_csi0_mux: endpoint {
+						remote-endpoint = <&ipu1_csi0_mux_from_parallel_sensor>;
+						bus-width = <8>;
+					};
+				};
+			};
+
 			max7310_a: gpio at 30 {
 				compatible = "maxim,max7310";
 				reg = <0x30>;
@@ -151,6 +169,25 @@
 	};
 };
 
+&ipu1_csi0_from_ipu1_csi0_mux {
+	bus-width = <8>;
+};
+
+&ipu1_csi0_mux_from_parallel_sensor {
+	remote-endpoint = <&adv7180_to_ipu1_csi0_mux>;
+	bus-width = <8>;
+};
+
+&ipu1_csi0 {
+	pinctrl-names = "default";
+	pinctrl-0 = <&pinctrl_ipu1_csi0>;
+
+	/* enable frame interval monitor on this port */
+	fim {
+		status = "okay";
+	};
+};
+
 &clks {
 	assigned-clocks = <&clks IMX6QDL_PLL4_BYPASS_SRC>,
 			  <&clks IMX6QDL_PLL4_BYPASS>,
@@ -445,6 +482,30 @@
 			>;
 		};
 
+		pinctrl_ipu1_csi0: ipu1csi0grp {
+			fsl,pins = <
+				MX6QDL_PAD_CSI0_DAT4__IPU1_CSI0_DATA04   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT5__IPU1_CSI0_DATA05   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT6__IPU1_CSI0_DATA06   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT7__IPU1_CSI0_DATA07   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT8__IPU1_CSI0_DATA08   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT9__IPU1_CSI0_DATA09   0x1b0b0
+				MX6QDL_PAD_CSI0_DAT10__IPU1_CSI0_DATA10  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT11__IPU1_CSI0_DATA11  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT12__IPU1_CSI0_DATA12  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT13__IPU1_CSI0_DATA13  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT14__IPU1_CSI0_DATA14  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT15__IPU1_CSI0_DATA15  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT16__IPU1_CSI0_DATA16  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT17__IPU1_CSI0_DATA17  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT18__IPU1_CSI0_DATA18  0x1b0b0
+				MX6QDL_PAD_CSI0_DAT19__IPU1_CSI0_DATA19  0x1b0b0
+				MX6QDL_PAD_CSI0_PIXCLK__IPU1_CSI0_PIXCLK 0x1b0b0
+				MX6QDL_PAD_CSI0_MCLK__IPU1_CSI0_HSYNC    0x1b0b0
+				MX6QDL_PAD_CSI0_VSYNC__IPU1_CSI0_VSYNC   0x1b0b0
+			>;
+		};
+
 		pinctrl_max7310: max7310grp {
 			fsl,pins = <
 				MX6QDL_PAD_SD2_DAT0__GPIO1_IO15 0x1b0b0
Signature · 1 line
2.7.4
author maintainer
#30 lore raw

PATCH v3 12/24 add mux and video interface bridge entity functions

In reply to Steve Longerbeam

From: Philipp Zabel <[email protected]>

---
 Documentation/media/uapi/mediactl/media-types.rst | 22 ++++++++++++++++++++++
 include/uapi/linux/media.h                        |  6 ++++++
 2 files changed, 28 insertions(+)
Documentation/media/uapi/mediactl/media-types.rst +22 -0
index 3e03dc2..023be29 100644
@@ -298,6 +298,28 @@ Types and flags used to represent the media graph elements
 	  received on its sink pad and outputs the statistics data on
 	  its source pad.
 
+    -  ..  row 29
+
+       ..  _MEDIA-ENT-F-MUX:
+
+       -  ``MEDIA_ENT_F_MUX``
+
+       - Video multiplexer. An entity capable of multiplexing must have at
+         least two sink pads and one source pad, and must pass the video
+         frame(s) received from the active sink pad to the source pad. Video
+         frame(s) from the inactive sink pads are discarded.
+
+    -  ..  row 30
+
+       ..  _MEDIA-ENT-F-VID-IF-BRIDGE:
+
+       -  ``MEDIA_ENT_F_VID_IF_BRIDGE``
+
+       - Video interface bridge. A video interface bridge entity must have at
+         least one sink pad and one source pad. It receives video frame(s) on
+         its sink pad in one bus format (HDMI, eDP, MIPI CSI-2, ...) and
+         converts them and outputs them on its source pad in another bus format
+         (eDP, MIPI CSI-2, parallel, ...).
 
 ..  tabularcolumns:: |p{5.5cm}|p{12.0cm}|
 
include/uapi/linux/media.h +6 -0
index 4890787..08a8bfa 100644
@@ -105,6 +105,12 @@ struct media_device_info {
 #define MEDIA_ENT_F_PROC_VIDEO_STATISTICS	(MEDIA_ENT_F_BASE + 0x4006)
 
 /*
+ * Switch and bridge entitites
+ */
+#define MEDIA_ENT_F_MUX				(MEDIA_ENT_F_BASE + 0x5001)
+#define MEDIA_ENT_F_VID_IF_BRIDGE		(MEDIA_ENT_F_BASE + 0x5002)
+
+/*
  * Connectors
  */
 /* It is a responsibility of the entity drivers to add connectors and links */
Signature · 1 line
2.7.4
#31 lore raw

Re: [PATCH v3 12/24] add mux and video interface bridge entity functions

Quoted 13 lines · Steve Longerbeam

From: Philipp Zabel <[email protected]>

---
 Documentation/media/uapi/mediactl/media-types.rst | 22 ++++++++++++++++++++++
 include/uapi/linux/media.h                        |  6 ++++++
 2 files changed, 28 insertions(+)
Documentation/media/uapi/mediactl/media-types.rst +0 -0
index 3e03dc2..023be29 100644

Documentation/media/uapi/mediactl/media-types.rst Types and flags used to represent the media graph elements · line 305

Quoted 8 lines
Documentation/media/uapi/mediactl/media-types.rst +5 -0
@@ -298,6 +298,28 @@ Types and flags used to represent the media graph elements
 	  received on its sink pad and outputs the statistics data on
 	  its source pad.
 
+    -  ..  row 29
+
+       ..  _MEDIA-ENT-F-MUX:
+
+       -  ``MEDIA_ENT_F_MUX``

I would prefer MEDIA_ENT_F_VID_MUX since this is video specific.

Regards,

Hans

Quoted 38 lines
Documentation/media/uapi/mediactl/media-types.rst +17 -0
+
+       - Video multiplexer. An entity capable of multiplexing must have at
+         least two sink pads and one source pad, and must pass the video
+         frame(s) received from the active sink pad to the source pad. Video
+         frame(s) from the inactive sink pads are discarded.
+
+    -  ..  row 30
+
+       ..  _MEDIA-ENT-F-VID-IF-BRIDGE:
+
+       -  ``MEDIA_ENT_F_VID_IF_BRIDGE``
+
+       - Video interface bridge. A video interface bridge entity must have at
+         least one sink pad and one source pad. It receives video frame(s) on
+         its sink pad in one bus format (HDMI, eDP, MIPI CSI-2, ...) and
+         converts them and outputs them on its source pad in another bus format
+         (eDP, MIPI CSI-2, parallel, ...).
 
 ..  tabularcolumns:: |p{5.5cm}|p{12.0cm}|
 
include/uapi/linux/media.h +6 -0
index 4890787..08a8bfa 100644
@@ -105,6 +105,12 @@ struct media_device_info {
 #define MEDIA_ENT_F_PROC_VIDEO_STATISTICS	(MEDIA_ENT_F_BASE + 0x4006)
 
 /*
+ * Switch and bridge entitites
+ */
+#define MEDIA_ENT_F_MUX				(MEDIA_ENT_F_BASE + 0x5001)
+#define MEDIA_ENT_F_VID_IF_BRIDGE		(MEDIA_ENT_F_BASE + 0x5002)
+
+/*
  * Connectors
  */
 /* It is a responsibility of the entity drivers to add connectors and links */
maintainer
#32 lore raw

Re: [PATCH v3 12/24] add mux and video interface bridge entity functions

In reply to Steve Longerbeam

Hi Steve,

Thank you for the patch

Quoted 26 lines · On Friday 06 Jan 2017 18:11:30 Steve Longerbeam wrote:

From: Philipp Zabel <[email protected]>

---
 Documentation/media/uapi/mediactl/media-types.rst | 22 ++++++++++++++++++++
 include/uapi/linux/media.h                        |  6 ++++++
 2 files changed, 28 insertions(+)
(quoted diff) +0 -0

b/Documentation/media/uapi/mediactl/media-types.rst index 3e03dc2..023be29 100644

Documentation/media/uapi/mediactl/media-types.rst +0 -0
@@ -298,6 +298,28 @@ Types and flags used to represent the media graph

elements received on its sink pad and outputs the statistics data on

Documentation/media/uapi/mediactl/media-types.rst +9 -0
 	  its source pad.
 
+    -  ..  row 29
+
+       ..  _MEDIA-ENT-F-MUX:
+
+       -  ``MEDIA_ENT_F_MUX``
+
+       - Video multiplexer. An entity capable of multiplexing must have at
+         least two sink pads and one source pad, and must pass the video
+         frame(s) received from the active sink pad to the source pad.
Quoted 2 lines

Video
+ frame(s) from the inactive sink pads are discarded.

Apart from the comment made by Hans regarding the macro name, this looks good to me.

Quoted 12 lines
Documentation/media/uapi/mediactl/media-types.rst +7 -0
+    -  ..  row 30
+
+       ..  _MEDIA-ENT-F-VID-IF-BRIDGE:
+
+       -  ``MEDIA_ENT_F_VID_IF_BRIDGE``
+
+       - Video interface bridge. A video interface bridge entity must have

at
+ least one sink pad and one source pad. It receives video frame(s)
on

Documentation/media/uapi/mediactl/media-types.rst +2 -0
+         its sink pad in one bus format (HDMI, eDP, MIPI CSI-2, ...) and
+         converts them and outputs them on its source pad in another bus
Quoted 2 lines

format
+ (eDP, MIPI CSI-2, parallel, ...).

The first sentence mentions *at least* one sink pad and one source pad, and the second one refers to "its sink pad" and "its source pad". This is a bit confusing. An easy option would be to require a single sink and a single source pad, but that would exclude bridges that combine a multiplexer.

I also wonder whether "bridge" is the appropriate word here. Transceiver might be a better choice, to insist on the fact that one of the two pads corresponds to a physical interface that has special electrical properties (such as HDMI, eDP, or CSI-2 that all require PHYs).

include/uapi/linux/media.h struct media_device_info { · line 110

Quoted 13 lines

.. tabularcolumns:: |p{5.5cm}|p{12.0cm}|

include/uapi/linux/media.h +3 -0
index 4890787..08a8bfa 100644
@@ -105,6 +105,12 @@ struct media_device_info {
 #define MEDIA_ENT_F_PROC_VIDEO_STATISTICS	(MEDIA_ENT_F_BASE + 0x4006)
 
 /*
+ * Switch and bridge entitites
+ */
+#define MEDIA_ENT_F_MUX				(MEDIA_ENT_F_BASE + 

0x5001)

Quoted 7 lines
include/uapi/linux/media.h +3 -0
+#define MEDIA_ENT_F_VID_IF_BRIDGE		(MEDIA_ENT_F_BASE + 0x5002)
+
+/*
  * Connectors
  */
 /* It is a responsibility of the entity drivers to add connectors and links

*/

Signature · 3 lines
Regards,

Laurent Pinchart
maintainer
#33 lore raw

Re: [PATCH v3 12/24] add mux and video interface bridge entity functions

Quoted 49 lines · Laurent Pinchart

Hi Steve,

Thank you for the patch

Quoted 26 lines · On Friday 06 Jan 2017 18:11:30 Steve Longerbeam wrote:

From: Philipp Zabel <[email protected]>

---
 Documentation/media/uapi/mediactl/media-types.rst | 22 ++++++++++++++++++++
 include/uapi/linux/media.h                        |  6 ++++++
 2 files changed, 28 insertions(+)
(quoted diff) +0 -0

b/Documentation/media/uapi/mediactl/media-types.rst index 3e03dc2..023be29 100644

Documentation/media/uapi/mediactl/media-types.rst +0 -0
@@ -298,6 +298,28 @@ Types and flags used to represent the media graph

elements received on its sink pad and outputs the statistics data on

Documentation/media/uapi/mediactl/media-types.rst +9 -0
 	  its source pad.
 
+    -  ..  row 29
+
+       ..  _MEDIA-ENT-F-MUX:
+
+       -  ``MEDIA_ENT_F_MUX``
+
+       - Video multiplexer. An entity capable of multiplexing must have at
+         least two sink pads and one source pad, and must pass the video
+         frame(s) received from the active sink pad to the source pad.
Quoted 2 lines

Video
+ frame(s) from the inactive sink pads are discarded.

Apart from the comment made by Hans regarding the macro name, this looks good to me.

Quoted 12 lines
Documentation/media/uapi/mediactl/media-types.rst +7 -0
+    -  ..  row 30
+
+       ..  _MEDIA-ENT-F-VID-IF-BRIDGE:
+
+       -  ``MEDIA_ENT_F_VID_IF_BRIDGE``
+
+       - Video interface bridge. A video interface bridge entity must have

at
+ least one sink pad and one source pad. It receives video frame(s)
on

Documentation/media/uapi/mediactl/media-types.rst +2 -0
+         its sink pad in one bus format (HDMI, eDP, MIPI CSI-2, ...) and
+         converts them and outputs them on its source pad in another bus
Quoted 8 lines
Quoted 2 lines

format
+ (eDP, MIPI CSI-2, parallel, ...).

The first sentence mentions *at least* one sink pad and one source pad, and the second one refers to "its sink pad" and "its source pad". This is a bit confusing. An easy option would be to require a single sink and a single source pad, but that would exclude bridges that combine a multiplexer.

Would it be enough to just switch to plural?

"It receives video frame(s) on its sink pads in one bus format and converts them and outputs them on its source pads in another bus format"?

I fear if this is made too specific to single-input single-output devices, a whole lot of multi-input devices will have to be split up unnecessarily. Although in cases of freely configurable multiplexers, it might also make sense to describe them as multiple entities. Especially if they can run in parallel with different configurations.

Quoted 4 lines

I also wonder whether "bridge" is the appropriate word here. Transceiver might be a better choice, to insist on the fact that one of the two pads corresponds to a physical interface that has special electrical properties (such as HDMI, eDP, or CSI-2 that all require PHYs).

What media entity function would you suggest to use for the IPU CSI, which basically is
- a video interface bridge, because it converts media bus formats from an external (up to) 20-bit parallel bus to an internal 128-bit bus, with the option to either expand/compand/pack >8-bit per component pixel formats (so parts of a write pixel formatter)
- also a video scaler, because it can crop and reduce width and/or height to 1/2 the original size

I wouldn't call it a transceiver, as it is completely contained inside the SoC.

regards
Philipp

author maintainer
#34 lore raw

PATCH v3 13/24 platform: add video-multiplexer subdevice driver

In reply to Steve Longerbeam

From: Philipp Zabel <[email protected]>

This driver can handle SoC internal and external video bus multiplexers, controlled either by register bit fields or by a GPIO. The subdevice passes through frame interval and mbus configuration of the active input to the output side.

--

- fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev() should be unregister.

- added media_entity_cleanup() and v4l2_device_unregister_subdev() to vidsw_remove().

- there was a line left over from a previous iteration that negated the new way of determining the pad count just before it which has been removed (num_pads = of_get_child_count(np)).

- Philipp Zabel has developed a set of patches that allow adding to the subdev async notifier waiting list using a chaining method from the async registered callbacks (v4l2_of_subdev_registered() and the prep patches for that). For now, I've removed the use of v4l2_of_subdev_registered() for the vidmux driver's registered callback. This doesn't affect the functionality of this driver, but allows for it to be merged now, before adding the chaining support.

---
 .../bindings/media/video-multiplexer.txt           |  59 +++
 drivers/media/platform/Kconfig                     |   8 +
 drivers/media/platform/Makefile                    |   2 +
 drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
 4 files changed, 541 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
 create mode 100644 drivers/media/platform/video-multiplexer.c
Documentation/devicetree/bindings/media/video-multiplexer.txt new file +59 -0
new file mode 100644
index 0000000..9d133d9
@@ -0,0 +1,59 @@
+Video Multiplexer
+=================
+
+Video multiplexers allow to select between multiple input ports. Video received
+on the active input port is passed through to the output port. Muxes described
+by this binding may be controlled by a syscon register bitfield or by a GPIO.
+
+Required properties:
+- compatible : should be "video-multiplexer"
+- reg: should be register base of the register containing the control bitfield
+- bit-mask: bitmask of the control bitfield in the control register
+- bit-shift: bit offset of the control bitfield in the control register
+- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
+  may be given to switch between two inputs
+- #address-cells: should be <1>
+- #size-cells: should be <0>
+- port@*: at least three port nodes containing endpoints connecting to the
+  source and sink devices according to of_graph bindings. The last port is
+  the output port, all others are inputs.
+
+Example:
+
+syscon {
+	compatible = "syscon", "simple-mfd";
+
+	mux {
+		compatible = "video-multiplexer";
+		/* Single bit (1 << 19) in syscon register 0x04: */
+		reg = <0x04>;
+		bit-mask = <1>;
+		bit-shift = <19>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		port at 0 {
+			reg = <0>;
+
+			mux_in0: endpoint {
+				remote-endpoint = <&video_source0_out>;
+			};
+		};
+
+		port at 1 {
+			reg = <1>;
+
+			mux_in1: endpoint {
+				remote-endpoint = <&video_source1_out>;
+			};
+		};
+
+		port at 2 {
+			reg = <2>;
+
+			mux_out: endpoint {
+				remote-endpoint = <&capture_interface_in>;
+			};
+		};
+	};
+};
drivers/media/platform/Kconfig +8 -0
index d944421..65614b5 100644
@@ -74,6 +74,14 @@ config VIDEO_M32R_AR_M64278
 	  To compile this driver as a module, choose M here: the
 	  module will be called arv.
 
+config VIDEO_MULTIPLEXER
+	tristate "Video Multiplexer"
+	depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER
+	help
+	  This driver provides support for SoC internal N:1 video bus
+	  multiplexers controlled by register bitfields as well as external
+	  2:1 video multiplexers controlled by a single GPIO.
+
 config VIDEO_OMAP3
 	tristate "OMAP 3 Camera support"
 	depends on VIDEO_V4L2 && I2C && VIDEO_V4L2_SUBDEV_API && ARCH_OMAP3
drivers/media/platform/Makefile +2 -0
index 5b3cb27..7cf0ee5 100644
@@ -27,6 +27,8 @@ obj-$(CONFIG_VIDEO_SH_VEU)		+= sh_veu.o
 
 obj-$(CONFIG_VIDEO_MEM2MEM_DEINTERLACE)	+= m2m-deinterlace.o
 
+obj-$(CONFIG_VIDEO_MULTIPLEXER)		+= video-multiplexer.o
+
 obj-$(CONFIG_VIDEO_S3C_CAMIF) 		+= s3c-camif/
 obj-$(CONFIG_VIDEO_SAMSUNG_EXYNOS4_IS) 	+= exynos4-is/
 obj-$(CONFIG_VIDEO_SAMSUNG_S5P_JPEG)	+= s5p-jpeg/
drivers/media/platform/video-multiplexer.c new file +472 -0

472 lines not on this page · raw message

Signature · 1 line
2.7.4
maintainer
#35 lore raw

Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver

Quoted 47 lines · Steve Longerbeam

From: Philipp Zabel <[email protected]>

This driver can handle SoC internal and external video bus multiplexers, controlled either by register bit fields or by a GPIO. The subdevice passes through frame interval and mbus configuration of the active input to the output side.

--

- fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev() should be unregister.

- added media_entity_cleanup() and v4l2_device_unregister_subdev() to vidsw_remove().

- there was a line left over from a previous iteration that negated the new way of determining the pad count just before it which has been removed (num_pads = of_get_child_count(np)).

- Philipp Zabel has developed a set of patches that allow adding to the subdev async notifier waiting list using a chaining method from the async registered callbacks (v4l2_of_subdev_registered() and the prep patches for that). For now, I've removed the use of v4l2_of_subdev_registered() for the vidmux driver's registered callback. This doesn't affect the functionality of this driver, but allows for it to be merged now, before adding the chaining support.

---
 .../bindings/media/video-multiplexer.txt           |  59 +++
 drivers/media/platform/Kconfig                     |   8 +
 drivers/media/platform/Makefile                    |   2 +
 drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
 4 files changed, 541 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
 create mode 100644 drivers/media/platform/video-multiplexer.c
Documentation/devicetree/bindings/media/video-multiplexer.txt new file +0 -0
new file mode 100644
index 0000000..9d133d9

Documentation/devicetree/bindings/media/video-multiplexer.txt line 9

Quoted 9 lines
Documentation/devicetree/bindings/media/video-multiplexer.txt new file +9 -0
@@ -0,0 +1,59 @@
+Video Multiplexer
+=================
+
+Video multiplexers allow to select between multiple input ports. Video received
+on the active input port is passed through to the output port. Muxes described
+by this binding may be controlled by a syscon register bitfield or by a GPIO.
+
+Required properties:
+- compatible : should be "video-multiplexer"

This should have an SoC/chip specific compatible string additionally. Just need a note to that effect here, but i.MX will need a compat string.

Quoted 10 lines
Documentation/devicetree/bindings/media/video-multiplexer.txt new file +10 -0
+- reg: should be register base of the register containing the control bitfield
+- bit-mask: bitmask of the control bitfield in the control register
+- bit-shift: bit offset of the control bitfield in the control register
+- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
+  may be given to switch between two inputs
+- #address-cells: should be <1>
+- #size-cells: should be <0>
+- port@*: at least three port nodes containing endpoints connecting to the
+  source and sink devices according to of_graph bindings. The last port is
+  the output port, all others are inputs.
#36 lore raw

Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver

In reply to Steve Longerbeam
Quoted 353 lines · Steve Longerbeam

From: Philipp Zabel <[email protected]>

This driver can handle SoC internal and external video bus multiplexers, controlled either by register bit fields or by a GPIO. The subdevice passes through frame interval and mbus configuration of the active input to the output side.

--

- fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev() should be unregister.

- added media_entity_cleanup() and v4l2_device_unregister_subdev() to vidsw_remove().

- there was a line left over from a previous iteration that negated the new way of determining the pad count just before it which has been removed (num_pads = of_get_child_count(np)).

- Philipp Zabel has developed a set of patches that allow adding to the subdev async notifier waiting list using a chaining method from the async registered callbacks (v4l2_of_subdev_registered() and the prep patches for that). For now, I've removed the use of v4l2_of_subdev_registered() for the vidmux driver's registered callback. This doesn't affect the functionality of this driver, but allows for it to be merged now, before adding the chaining support.

---
 .../bindings/media/video-multiplexer.txt           |  59 +++
 drivers/media/platform/Kconfig                     |   8 +
 drivers/media/platform/Makefile                    |   2 +
 drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
 4 files changed, 541 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
 create mode 100644 drivers/media/platform/video-multiplexer.c
Documentation/devicetree/bindings/media/video-multiplexer.txt new file +59 -0
new file mode 100644
index 0000000..9d133d9
@@ -0,0 +1,59 @@
+Video Multiplexer
+=================
+
+Video multiplexers allow to select between multiple input ports. Video received
+on the active input port is passed through to the output port. Muxes described
+by this binding may be controlled by a syscon register bitfield or by a GPIO.
+
+Required properties:
+- compatible : should be "video-multiplexer"
+- reg: should be register base of the register containing the control bitfield
+- bit-mask: bitmask of the control bitfield in the control register
+- bit-shift: bit offset of the control bitfield in the control register
+- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
+  may be given to switch between two inputs
+- #address-cells: should be <1>
+- #size-cells: should be <0>
+- port@*: at least three port nodes containing endpoints connecting to the
+  source and sink devices according to of_graph bindings. The last port is
+  the output port, all others are inputs.
+
+Example:
+
+syscon {
+	compatible = "syscon", "simple-mfd";
+
+	mux {
+		compatible = "video-multiplexer";
+		/* Single bit (1 << 19) in syscon register 0x04: */
+		reg = <0x04>;
+		bit-mask = <1>;
+		bit-shift = <19>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		port at 0 {
+			reg = <0>;
+
+			mux_in0: endpoint {
+				remote-endpoint = <&video_source0_out>;
+			};
+		};
+
+		port at 1 {
+			reg = <1>;
+
+			mux_in1: endpoint {
+				remote-endpoint = <&video_source1_out>;
+			};
+		};
+
+		port at 2 {
+			reg = <2>;
+
+			mux_out: endpoint {
+				remote-endpoint = <&capture_interface_in>;
+			};
+		};
+	};
+};
drivers/media/platform/Kconfig +8 -0
index d944421..65614b5 100644
@@ -74,6 +74,14 @@ config VIDEO_M32R_AR_M64278
 	  To compile this driver as a module, choose M here: the
 	  module will be called arv.
 
+config VIDEO_MULTIPLEXER
+	tristate "Video Multiplexer"
+	depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER
+	help
+	  This driver provides support for SoC internal N:1 video bus
+	  multiplexers controlled by register bitfields as well as external
+	  2:1 video multiplexers controlled by a single GPIO.
+
 config VIDEO_OMAP3
 	tristate "OMAP 3 Camera support"
 	depends on VIDEO_V4L2 && I2C && VIDEO_V4L2_SUBDEV_API && ARCH_OMAP3
drivers/media/platform/Makefile +2 -0
index 5b3cb27..7cf0ee5 100644
@@ -27,6 +27,8 @@ obj-$(CONFIG_VIDEO_SH_VEU)		+= sh_veu.o
 
 obj-$(CONFIG_VIDEO_MEM2MEM_DEINTERLACE)	+= m2m-deinterlace.o
 
+obj-$(CONFIG_VIDEO_MULTIPLEXER)		+= video-multiplexer.o
+
 obj-$(CONFIG_VIDEO_S3C_CAMIF) 		+= s3c-camif/
 obj-$(CONFIG_VIDEO_SAMSUNG_EXYNOS4_IS) 	+= exynos4-is/
 obj-$(CONFIG_VIDEO_SAMSUNG_S5P_JPEG)	+= s5p-jpeg/
drivers/media/platform/video-multiplexer.c new file +209 -0
new file mode 100644
index 0000000..48980c4
@@ -0,0 +1,472 @@
+/*
+ * video stream multiplexer controlled via gpio or syscon
+ *
+ * Copyright (C) 2013 Pengutronix, Sascha Hauer <[email protected]>
+ * Copyright (C) 2016 Pengutronix, Philipp Zabel <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/err.h>
+#include <linux/gpio/consumer.h>
+#include <linux/mfd/syscon.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_graph.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+#include <media/v4l2-async.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-of.h>
+
+struct vidsw {
+	struct v4l2_subdev subdev;
+	unsigned int num_pads;
+	struct media_pad *pads;
+	struct v4l2_mbus_framefmt *format_mbus;
+	struct v4l2_fract timeperframe;
+	struct v4l2_of_endpoint *endpoint;
+	struct regmap_field *field;
+	struct gpio_desc *gpio;
+	int active;
+};
+
+static inline struct vidsw *v4l2_subdev_to_vidsw(struct v4l2_subdev *sd)
+{
+	return container_of(sd, struct vidsw, subdev);
+}
+
+static void vidsw_set_active(struct vidsw *vidsw, int active)
+{
+	vidsw->active = active;
+	if (active < 0)
+		return;
+
+	dev_dbg(vidsw->subdev.dev, "setting %d active\n", active);
+
+	if (vidsw->field)
+		regmap_field_write(vidsw->field, active);
+	else if (vidsw->gpio)
+		gpiod_set_value(vidsw->gpio, active);
+}
+
+static int vidsw_link_setup(struct media_entity *entity,
+			    const struct media_pad *local,
+			    const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	/* We have no limitations on enabling or disabling our output link */
+	if (local->index == vidsw->num_pads - 1)
+		return 0;
+
+	dev_dbg(sd->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	if (!(flags & MEDIA_LNK_FL_ENABLED)) {
+		if (local->index == vidsw->active) {
+			dev_dbg(sd->dev, "going inactive\n");
+			vidsw->active = -1;
+		}
+		return 0;
+	}
+
+	if (vidsw->active >= 0) {
+		struct media_pad *pad;
+
+		if (vidsw->active == local->index)
+			return 0;
+
+		pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
+		if (pad) {
+			struct media_link *link;
+			int ret;
+
+			link = media_entity_find_link(pad,
+						&vidsw->pads[vidsw->active]);
+			if (link) {
+				ret = __media_entity_setup_link(link, 0);
+				if (ret)
+					return ret;
+			}
+		}
+	}
+
+	vidsw_set_active(vidsw, local->index);
+
+	return 0;
+}
+
+static struct media_entity_operations vidsw_ops = {
+	.link_setup = vidsw_link_setup,
+};
+
+static bool vidsw_endpoint_disabled(struct device_node *ep)
+{
+	struct device_node *rpp;
+
+	if (!of_device_is_available(ep))
+		return true;
+
+	rpp = of_graph_get_remote_port_parent(ep);
+	if (!rpp)
+		return true;
+
+	return !of_device_is_available(rpp);
+}
+
+static int vidsw_async_init(struct vidsw *vidsw, struct device_node *node)
+{
+	struct device_node *ep;
+	u32 portno;
+	int numports;
+	int ret;
+	int i;
+	bool active_link = false;
+
+	numports = vidsw->num_pads;
+
+	for (i = 0; i < numports - 1; i++)
+		vidsw->pads[i].flags = MEDIA_PAD_FL_SINK;
+	vidsw->pads[numports - 1].flags = MEDIA_PAD_FL_SOURCE;
+
+	vidsw->subdev.entity.function = MEDIA_ENT_F_MUX;
+	ret = media_entity_pads_init(&vidsw->subdev.entity, numports,
+				     vidsw->pads);
+	if (ret < 0)
+		return ret;
+
+	vidsw->subdev.entity.ops = &vidsw_ops;
+
+	for_each_endpoint_of_node(node, ep) {
+		struct v4l2_of_endpoint endpoint;
+
+		v4l2_of_parse_endpoint(ep, &endpoint);
+
+		portno = endpoint.base.port;
+		if (portno >= numports - 1)
+			continue;
+
+		if (vidsw_endpoint_disabled(ep)) {
+			dev_dbg(vidsw->subdev.dev, "port %d disabled\n", portno);
+			continue;
+		}
+
+		vidsw->endpoint[portno] = endpoint;
+
+		if (portno == vidsw->active)
+			active_link = true;
+	}
+
+	for (portno = 0; portno < numports - 1; portno++) {
+		if (!vidsw->endpoint[portno].base.local_node)
+			continue;
+
+		/* If the active input is not connected, use another */
+		if (!active_link) {
+			vidsw_set_active(vidsw, portno);
+			active_link = true;
+		}
+	}
+
+	return v4l2_async_register_subdev(&vidsw->subdev);
+}
+
+int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config *cfg)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+	struct media_pad *pad;
+	int ret;
+
+	if (vidsw->active == -1) {
+		dev_err(sd->dev, "no configuration for inactive mux\n");
+		return -EINVAL;
+	}
+
+	/*
+	 * Retrieve media bus configuration from the entity connected to the
+	 * active input
+	 */
+	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
+	if (pad) {
+		sd = media_entity_to_v4l2_subdev(pad->entity);
+		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
+		if (ret == -ENOIOCTLCMD)
+			pad = NULL;
+		else if (ret < 0) {
+			dev_err(sd->dev, "failed to get source configuration\n");
+			return ret;
+		}
+	}

drivers/media/platform/video-multiplexer.c line 219

Quoted 10 lines
drivers/media/platform/video-multiplexer.c new file +10 -0
+	if (!pad) {
+		/* Mirror the input side on the output side */
+		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
+		if (cfg->type == V4L2_MBUS_PARALLEL ||
+		    cfg->type == V4L2_MBUS_BT656)
+			cfg->flags = vidsw->endpoint[vidsw->active].bus.parallel.flags;
+	}
+
+	return 0;
+}

I am not certain this op is needed at all. In the current kernel this op is only used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this information should come from the device tree and there should be no need for this op.

My (tentative) long-term plan was to get rid of this op.

If you don't need it, then I recommend it is removed.

Quoted 244 lines
drivers/media/platform/video-multiplexer.c new file +243 -0
+
+static int vidsw_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+	struct v4l2_subdev *upstream_sd;
+	struct media_pad *pad;
+
+	if (vidsw->active == -1) {
+		dev_err(sd->dev, "Can not start streaming on inactive mux\n");
+		return -EINVAL;
+	}
+
+	pad = media_entity_remote_pad(&sd->entity.pads[vidsw->active]);
+	if (!pad) {
+		dev_err(sd->dev, "Failed to find remote source pad\n");
+		return -ENOLINK;
+	}
+
+	if (!is_media_entity_v4l2_subdev(pad->entity)) {
+		dev_err(sd->dev, "Upstream entity is not a v4l2 subdev\n");
+		return -ENODEV;
+	}
+
+	upstream_sd = media_entity_to_v4l2_subdev(pad->entity);
+
+	return v4l2_subdev_call(upstream_sd, video, s_stream, enable);
+}
+
+static int vidsw_g_frame_interval(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_frame_interval *fi)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	fi->interval = vidsw->timeperframe;
+
+	return 0;
+}
+
+static int vidsw_s_frame_interval(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_frame_interval *fi)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	vidsw->timeperframe = fi->interval;
+
+	return 0;
+}
+
+static const struct v4l2_subdev_video_ops vidsw_subdev_video_ops = {
+	.g_mbus_config = vidsw_g_mbus_config,
+	.s_stream = vidsw_s_stream,
+	.g_frame_interval = vidsw_g_frame_interval,
+	.s_frame_interval = vidsw_s_frame_interval,
+};
+
+static struct v4l2_mbus_framefmt *
+__vidsw_get_pad_format(struct v4l2_subdev *sd,
+		       struct v4l2_subdev_pad_config *cfg,
+		       unsigned int pad, u32 which)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	switch (which) {
+	case V4L2_SUBDEV_FORMAT_TRY:
+		return v4l2_subdev_get_try_format(sd, cfg, pad);
+	case V4L2_SUBDEV_FORMAT_ACTIVE:
+		return &vidsw->format_mbus[pad];
+	default:
+		return NULL;
+	}
+}
+
+static int vidsw_get_format(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_pad_config *cfg,
+			    struct v4l2_subdev_format *sdformat)
+{
+	sdformat->format = *__vidsw_get_pad_format(sd, cfg, sdformat->pad,
+						   sdformat->which);
+	return 0;
+}
+
+static int vidsw_set_format(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_pad_config *cfg,
+			    struct v4l2_subdev_format *sdformat)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+	struct v4l2_mbus_framefmt *mbusformat;
+
+	if (sdformat->pad >= vidsw->num_pads)
+		return -EINVAL;
+
+	mbusformat = __vidsw_get_pad_format(sd, cfg, sdformat->pad,
+					    sdformat->which);
+	if (!mbusformat)
+		return -EINVAL;
+
+	/* Output pad mirrors active input pad, no limitations on input pads */
+	if (sdformat->pad == (vidsw->num_pads - 1) && vidsw->active >= 0)
+		sdformat->format = vidsw->format_mbus[vidsw->active];
+
+	*mbusformat = sdformat->format;
+
+	return 0;
+}
+
+static struct v4l2_subdev_pad_ops vidsw_pad_ops = {
+	.get_fmt = vidsw_get_format,
+	.set_fmt = vidsw_set_format,
+};
+
+static struct v4l2_subdev_ops vidsw_subdev_ops = {
+	.pad = &vidsw_pad_ops,
+	.video = &vidsw_subdev_video_ops,
+};
+
+static int of_get_reg_field(struct device_node *node, struct reg_field *field)
+{
+	u32 bit_mask;
+	int ret;
+
+	ret = of_property_read_u32(node, "reg", &field->reg);
+	if (ret < 0)
+		return ret;
+
+	ret = of_property_read_u32(node, "bit-mask", &bit_mask);
+	if (ret < 0)
+		return ret;
+
+	ret = of_property_read_u32(node, "bit-shift", &field->lsb);
+	if (ret < 0)
+		return ret;
+
+	field->msb = field->lsb + fls(bit_mask) - 1;
+
+	return 0;
+}
+
+static int vidsw_probe(struct platform_device *pdev)
+{
+	struct device_node *np = pdev->dev.of_node;
+	struct of_endpoint endpoint;
+	struct device_node *ep;
+	struct reg_field field;
+	struct vidsw *vidsw;
+	struct regmap *map;
+	unsigned int num_pads;
+	int ret;
+
+	vidsw = devm_kzalloc(&pdev->dev, sizeof(*vidsw), GFP_KERNEL);
+	if (!vidsw)
+		return -ENOMEM;
+
+	platform_set_drvdata(pdev, vidsw);
+
+	v4l2_subdev_init(&vidsw->subdev, &vidsw_subdev_ops);
+	snprintf(vidsw->subdev.name, sizeof(vidsw->subdev.name), "%s",
+			np->name);
+	vidsw->subdev.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
+	vidsw->subdev.dev = &pdev->dev;
+
+	/*
+	 * The largest numbered port is the output port. It determines
+	 * total number of pads
+	 */
+	num_pads = 0;
+	for_each_endpoint_of_node(np, ep) {
+		of_graph_parse_endpoint(ep, &endpoint);
+		num_pads = max(num_pads, endpoint.port + 1);
+	}
+
+	if (num_pads < 2) {
+		dev_err(&pdev->dev, "Not enough ports %d\n", num_pads);
+		return -EINVAL;
+	}
+
+	ret = of_get_reg_field(np, &field);
+	if (ret == 0) {
+		map = syscon_node_to_regmap(np->parent);
+		if (!map) {
+			dev_err(&pdev->dev, "Failed to get syscon register map\n");
+			return PTR_ERR(map);
+		}
+
+		vidsw->field = devm_regmap_field_alloc(&pdev->dev, map, field);
+		if (IS_ERR(vidsw->field)) {
+			dev_err(&pdev->dev, "Failed to allocate regmap field\n");
+			return PTR_ERR(vidsw->field);
+		}
+
+		regmap_field_read(vidsw->field, &vidsw->active);
+	} else {
+		if (num_pads > 3) {
+			dev_err(&pdev->dev, "Too many ports %d\n", num_pads);
+			return -EINVAL;
+		}
+
+		vidsw->gpio = devm_gpiod_get(&pdev->dev, NULL, GPIOD_OUT_LOW);
+		if (IS_ERR(vidsw->gpio)) {
+			dev_warn(&pdev->dev,
+				 "could not request control gpio: %d\n", ret);
+			vidsw->gpio = NULL;
+		}
+
+		vidsw->active = gpiod_get_value(vidsw->gpio) ? 1 : 0;
+	}
+
+	vidsw->num_pads = num_pads;
+	vidsw->pads = devm_kzalloc(&pdev->dev, sizeof(*vidsw->pads) * num_pads,
+			GFP_KERNEL);
+	vidsw->format_mbus = devm_kzalloc(&pdev->dev,
+			sizeof(*vidsw->format_mbus) * num_pads, GFP_KERNEL);
+	vidsw->endpoint = devm_kzalloc(&pdev->dev,
+			sizeof(*vidsw->endpoint) * (num_pads - 1), GFP_KERNEL);
+
+	ret = vidsw_async_init(vidsw, np);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static int vidsw_remove(struct platform_device *pdev)
+{
+	struct vidsw *vidsw = platform_get_drvdata(pdev);
+	struct v4l2_subdev *sd = &vidsw->subdev;
+
+	v4l2_async_unregister_subdev(sd);
+	media_entity_cleanup(&sd->entity);
+	v4l2_device_unregister_subdev(sd);
+
+	return 0;
+}
+
+static const struct of_device_id vidsw_dt_ids[] = {
+	{ .compatible = "video-multiplexer", },
+	{ /* sentinel */ }
+};
+
+static struct platform_driver vidsw_driver = {
+	.probe		= vidsw_probe,
+	.remove		= vidsw_remove,
+	.driver		= {
+		.of_match_table = vidsw_dt_ids,

drivers/media/platform/video-multiplexer.c line 472

Quoted 10 lines
drivers/media/platform/video-multiplexer.c new file +10 -0
+		.name = "video-multiplexer",
+	},
+};
+
+module_platform_driver(vidsw_driver);
+
+MODULE_DESCRIPTION("video stream multiplexer");
+MODULE_AUTHOR("Sascha Hauer, Pengutronix");
+MODULE_AUTHOR("Philipp Zabel, Pengutronix");
+MODULE_LICENSE("GPL");

Regards,

Hans

maintainer
#37 lore raw

Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver

Hi Hans,

Quoted 369 lines · Hans Verkuil
Quoted 353 lines · Steve Longerbeam

From: Philipp Zabel <[email protected]>

This driver can handle SoC internal and external video bus multiplexers, controlled either by register bit fields or by a GPIO. The subdevice passes through frame interval and mbus configuration of the active input to the output side.

--

- fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev() should be unregister.

- added media_entity_cleanup() and v4l2_device_unregister_subdev() to vidsw_remove().

- there was a line left over from a previous iteration that negated the new way of determining the pad count just before it which has been removed (num_pads = of_get_child_count(np)).

- Philipp Zabel has developed a set of patches that allow adding to the subdev async notifier waiting list using a chaining method from the async registered callbacks (v4l2_of_subdev_registered() and the prep patches for that). For now, I've removed the use of v4l2_of_subdev_registered() for the vidmux driver's registered callback. This doesn't affect the functionality of this driver, but allows for it to be merged now, before adding the chaining support.

---
 .../bindings/media/video-multiplexer.txt           |  59 +++
 drivers/media/platform/Kconfig                     |   8 +
 drivers/media/platform/Makefile                    |   2 +
 drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
 4 files changed, 541 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
 create mode 100644 drivers/media/platform/video-multiplexer.c
Documentation/devicetree/bindings/media/video-multiplexer.txt new file +59 -0
new file mode 100644
index 0000000..9d133d9
@@ -0,0 +1,59 @@
+Video Multiplexer
+=================
+
+Video multiplexers allow to select between multiple input ports. Video received
+on the active input port is passed through to the output port. Muxes described
+by this binding may be controlled by a syscon register bitfield or by a GPIO.
+
+Required properties:
+- compatible : should be "video-multiplexer"
+- reg: should be register base of the register containing the control bitfield
+- bit-mask: bitmask of the control bitfield in the control register
+- bit-shift: bit offset of the control bitfield in the control register
+- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
+  may be given to switch between two inputs
+- #address-cells: should be <1>
+- #size-cells: should be <0>
+- port@*: at least three port nodes containing endpoints connecting to the
+  source and sink devices according to of_graph bindings. The last port is
+  the output port, all others are inputs.
+
+Example:
+
+syscon {
+	compatible = "syscon", "simple-mfd";
+
+	mux {
+		compatible = "video-multiplexer";
+		/* Single bit (1 << 19) in syscon register 0x04: */
+		reg = <0x04>;
+		bit-mask = <1>;
+		bit-shift = <19>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		port at 0 {
+			reg = <0>;
+
+			mux_in0: endpoint {
+				remote-endpoint = <&video_source0_out>;
+			};
+		};
+
+		port at 1 {
+			reg = <1>;
+
+			mux_in1: endpoint {
+				remote-endpoint = <&video_source1_out>;
+			};
+		};
+
+		port at 2 {
+			reg = <2>;
+
+			mux_out: endpoint {
+				remote-endpoint = <&capture_interface_in>;
+			};
+		};
+	};
+};
drivers/media/platform/Kconfig +8 -0
index d944421..65614b5 100644
@@ -74,6 +74,14 @@ config VIDEO_M32R_AR_M64278
 	  To compile this driver as a module, choose M here: the
 	  module will be called arv.
 
+config VIDEO_MULTIPLEXER
+	tristate "Video Multiplexer"
+	depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER
+	help
+	  This driver provides support for SoC internal N:1 video bus
+	  multiplexers controlled by register bitfields as well as external
+	  2:1 video multiplexers controlled by a single GPIO.
+
 config VIDEO_OMAP3
 	tristate "OMAP 3 Camera support"
 	depends on VIDEO_V4L2 && I2C && VIDEO_V4L2_SUBDEV_API && ARCH_OMAP3
drivers/media/platform/Makefile +2 -0
index 5b3cb27..7cf0ee5 100644
@@ -27,6 +27,8 @@ obj-$(CONFIG_VIDEO_SH_VEU)		+= sh_veu.o
 
 obj-$(CONFIG_VIDEO_MEM2MEM_DEINTERLACE)	+= m2m-deinterlace.o
 
+obj-$(CONFIG_VIDEO_MULTIPLEXER)		+= video-multiplexer.o
+
 obj-$(CONFIG_VIDEO_S3C_CAMIF) 		+= s3c-camif/
 obj-$(CONFIG_VIDEO_SAMSUNG_EXYNOS4_IS) 	+= exynos4-is/
 obj-$(CONFIG_VIDEO_SAMSUNG_S5P_JPEG)	+= s5p-jpeg/
drivers/media/platform/video-multiplexer.c new file +209 -0
new file mode 100644
index 0000000..48980c4
@@ -0,0 +1,472 @@
+/*
+ * video stream multiplexer controlled via gpio or syscon
+ *
+ * Copyright (C) 2013 Pengutronix, Sascha Hauer <[email protected]>
+ * Copyright (C) 2016 Pengutronix, Philipp Zabel <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/err.h>
+#include <linux/gpio/consumer.h>
+#include <linux/mfd/syscon.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_graph.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+#include <media/v4l2-async.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-of.h>
+
+struct vidsw {
+	struct v4l2_subdev subdev;
+	unsigned int num_pads;
+	struct media_pad *pads;
+	struct v4l2_mbus_framefmt *format_mbus;
+	struct v4l2_fract timeperframe;
+	struct v4l2_of_endpoint *endpoint;
+	struct regmap_field *field;
+	struct gpio_desc *gpio;
+	int active;
+};
+
+static inline struct vidsw *v4l2_subdev_to_vidsw(struct v4l2_subdev *sd)
+{
+	return container_of(sd, struct vidsw, subdev);
+}
+
+static void vidsw_set_active(struct vidsw *vidsw, int active)
+{
+	vidsw->active = active;
+	if (active < 0)
+		return;
+
+	dev_dbg(vidsw->subdev.dev, "setting %d active\n", active);
+
+	if (vidsw->field)
+		regmap_field_write(vidsw->field, active);
+	else if (vidsw->gpio)
+		gpiod_set_value(vidsw->gpio, active);
+}
+
+static int vidsw_link_setup(struct media_entity *entity,
+			    const struct media_pad *local,
+			    const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+
+	/* We have no limitations on enabling or disabling our output link */
+	if (local->index == vidsw->num_pads - 1)
+		return 0;
+
+	dev_dbg(sd->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	if (!(flags & MEDIA_LNK_FL_ENABLED)) {
+		if (local->index == vidsw->active) {
+			dev_dbg(sd->dev, "going inactive\n");
+			vidsw->active = -1;
+		}
+		return 0;
+	}
+
+	if (vidsw->active >= 0) {
+		struct media_pad *pad;
+
+		if (vidsw->active == local->index)
+			return 0;
+
+		pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
+		if (pad) {
+			struct media_link *link;
+			int ret;
+
+			link = media_entity_find_link(pad,
+						&vidsw->pads[vidsw->active]);
+			if (link) {
+				ret = __media_entity_setup_link(link, 0);
+				if (ret)
+					return ret;
+			}
+		}
+	}
+
+	vidsw_set_active(vidsw, local->index);
+
+	return 0;
+}
+
+static struct media_entity_operations vidsw_ops = {
+	.link_setup = vidsw_link_setup,
+};
+
+static bool vidsw_endpoint_disabled(struct device_node *ep)
+{
+	struct device_node *rpp;
+
+	if (!of_device_is_available(ep))
+		return true;
+
+	rpp = of_graph_get_remote_port_parent(ep);
+	if (!rpp)
+		return true;
+
+	return !of_device_is_available(rpp);
+}
+
+static int vidsw_async_init(struct vidsw *vidsw, struct device_node *node)
+{
+	struct device_node *ep;
+	u32 portno;
+	int numports;
+	int ret;
+	int i;
+	bool active_link = false;
+
+	numports = vidsw->num_pads;
+
+	for (i = 0; i < numports - 1; i++)
+		vidsw->pads[i].flags = MEDIA_PAD_FL_SINK;
+	vidsw->pads[numports - 1].flags = MEDIA_PAD_FL_SOURCE;
+
+	vidsw->subdev.entity.function = MEDIA_ENT_F_MUX;
+	ret = media_entity_pads_init(&vidsw->subdev.entity, numports,
+				     vidsw->pads);
+	if (ret < 0)
+		return ret;
+
+	vidsw->subdev.entity.ops = &vidsw_ops;
+
+	for_each_endpoint_of_node(node, ep) {
+		struct v4l2_of_endpoint endpoint;
+
+		v4l2_of_parse_endpoint(ep, &endpoint);
+
+		portno = endpoint.base.port;
+		if (portno >= numports - 1)
+			continue;
+
+		if (vidsw_endpoint_disabled(ep)) {
+			dev_dbg(vidsw->subdev.dev, "port %d disabled\n", portno);
+			continue;
+		}
+
+		vidsw->endpoint[portno] = endpoint;
+
+		if (portno == vidsw->active)
+			active_link = true;
+	}
+
+	for (portno = 0; portno < numports - 1; portno++) {
+		if (!vidsw->endpoint[portno].base.local_node)
+			continue;
+
+		/* If the active input is not connected, use another */
+		if (!active_link) {
+			vidsw_set_active(vidsw, portno);
+			active_link = true;
+		}
+	}
+
+	return v4l2_async_register_subdev(&vidsw->subdev);
+}
+
+int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config *cfg)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+	struct media_pad *pad;
+	int ret;
+
+	if (vidsw->active == -1) {
+		dev_err(sd->dev, "no configuration for inactive mux\n");
+		return -EINVAL;
+	}
+
+	/*
+	 * Retrieve media bus configuration from the entity connected to the
+	 * active input
+	 */
+	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
+	if (pad) {
+		sd = media_entity_to_v4l2_subdev(pad->entity);
+		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
+		if (ret == -ENOIOCTLCMD)
+			pad = NULL;
+		else if (ret < 0) {
+			dev_err(sd->dev, "failed to get source configuration\n");
+			return ret;
+		}
+	}

drivers/media/platform/video-multiplexer.c line 219

Quoted 10 lines
drivers/media/platform/video-multiplexer.c new file +10 -0
+	if (!pad) {
+		/* Mirror the input side on the output side */
+		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
+		if (cfg->type == V4L2_MBUS_PARALLEL ||
+		    cfg->type == V4L2_MBUS_BT656)
+			cfg->flags = vidsw->endpoint[vidsw->active].bus.parallel.flags;
+	}
+
+	return 0;
+}

I am not certain this op is needed at all. In the current kernel this op is only used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this information should come from the device tree and there should be no need for this op.

Quoted 3 lines

My (tentative) long-term plan was to get rid of this op.

If you don't need it, then I recommend it is removed.

We currently use this to make the CSI capture interface understand whether its source from the MIPI CSI-2 or from the parallel bus. That is probably something that should be fixed, but I'm not quite sure how.

The Synopsys DesignWare MIPI CSI-2 reciever turns the incoming MIPI CSI-2 signal into a 32-bit parallel pixel bus plus some signals for the MIPI specific metadata (virtual channel, data type).

Then the CSI2IPU gasket turns this input bus into four separate parallel 16-bit pixel buses plus an 8-bit "mct_di" bus for each of them, that carries the MIPI metadata. The incoming data is split into the four outputs according to the MIPI virtual channel.

Two of these 16-bit + 8-bit parallel buses are routed through a multiplexer before finally arriving at the CSI on the other side.

We need to configure the CSI to either use or ignore the data from the 8-bit mct_di bus depending on whether the source of the mux is configured to the MIPI CSI-2 receiver / CSI2IPU gasket, or to a parallel input.

Currently we let g_mbus_config pretend that even the internal 32-bit + metadata and 16-bit + 8-bit metadata parallel buses are of type V4L2_MBUS_CSI so that the CSI can ask the mux, which propagates to the CSI-2 receiver, if connected.

Without g_mbus_config we'd need to get that information from somewhere else. One possibility would be to extend MEDIA_BUS formats to describe these "parallelized MIPI data" buses separately.

regards
Philipp

author maintainer
#38 lore raw

Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver

Quoted 48 lines · Philipp Zabel

Hi Hans,

Quoted 45 lines · Hans Verkuil

<snip>

Quoted 27 lines
drivers/media/platform/video-multiplexer.c new file +27 -0
+
+int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config *cfg)
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+	struct media_pad *pad;
+	int ret;
+
+	if (vidsw->active == -1) {
+		dev_err(sd->dev, "no configuration for inactive mux\n");
+		return -EINVAL;
+	}
+
+	/*
+	 * Retrieve media bus configuration from the entity connected to the
+	 * active input
+	 */
+	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
+	if (pad) {
+		sd = media_entity_to_v4l2_subdev(pad->entity);
+		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
+		if (ret == -ENOIOCTLCMD)
+			pad = NULL;
+		else if (ret < 0) {
+			dev_err(sd->dev, "failed to get source configuration\n");
+			return ret;
+		}
+	}

drivers/media/platform/video-multiplexer.c line 219

Quoted 10 lines
drivers/media/platform/video-multiplexer.c new file +10 -0
+	if (!pad) {
+		/* Mirror the input side on the output side */
+		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
+		if (cfg->type == V4L2_MBUS_PARALLEL ||
+		    cfg->type == V4L2_MBUS_BT656)
+			cfg->flags = vidsw->endpoint[vidsw->active].bus.parallel.flags;
+	}
+
+	return 0;
+}

I am not certain this op is needed at all. In the current kernel this op is only used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this information should come from the device tree and there should be no need for this op.

My (tentative) long-term plan was to get rid of this op.

If you don't need it, then I recommend it is removed.

Hi Hans, the imx-media driver was only calling g_mbus_config to the camera sensor, and it was doing that to determine the sensor's bus type. This info was already available from parsing a v4l2_of_endpoint from the sensor node. So it was simple to remove the g_mbus_config calls, and instead rely on the parsed sensor v4l2_of_endpoint.

Quoted 13 lines

We currently use this to make the CSI capture interface understand whether its source from the MIPI CSI-2 or from the parallel bus. That is probably something that should be fixed, but I'm not quite sure how.

The Synopsys DesignWare MIPI CSI-2 reciever turns the incoming MIPI CSI-2 signal into a 32-bit parallel pixel bus plus some signals for the MIPI specific metadata (virtual channel, data type).

Then the CSI2IPU gasket turns this input bus into four separate parallel 16-bit pixel buses plus an 8-bit "mct_di" bus for each of them, that carries the MIPI metadata. The incoming data is split into the four outputs according to the MIPI virtual channel.

Quoted 7 lines

Two of these 16-bit + 8-bit parallel buses are routed through a multiplexer before finally arriving at the CSI on the other side.

We need to configure the CSI to either use or ignore the data from the 8-bit mct_di bus depending on whether the source of the mux is configured to the MIPI CSI-2 receiver / CSI2IPU gasket, or to a parallel input.

Philipp, from my experience, the CSI_MIPI_DI register (configured by ipu_csi_set_mipi_datatype()) can only be given a virtual channel 0, otherwise no data is received from the MIPI CSI-2 sensor, regardless of the virtual channel the sensor is transmitting over.

So it seems the info on the 8-bit mct_di buses generated by the CSI2IPU gasket are ignored by the CSI's, at least the virtual channel number is ignored.

For example, if the sensor is transmitting on vc 1, the gasket routes the sensor data to the parallel bus to CSI1. But if CSI_MIPI_DI on CSI1 is written with vc 1, no data is received.

Steve

Quoted 13 lines

Currently we let g_mbus_config pretend that even the internal 32-bit + metadata and 16-bit + 8-bit metadata parallel buses are of type V4L2_MBUS_CSI so that the CSI can ask the mux, which propagates to the CSI-2 receiver, if connected.

Without g_mbus_config we'd need to get that information from somewhere else. One possibility would be to extend MEDIA_BUS formats to describe these "parallelized MIPI data" buses separately.

regards
Philipp

maintainer
#39 lore raw

Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver

Hi Steve,

Quoted 19 lines · On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
Quoted 18 lines · Philipp Zabel
Quoted 17 lines · Hans Verkuil
Quoted 17 lines
(quoted diff) +2 -0
+
+int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config

*cfg)

drivers/media/platform/video-multiplexer.c new file +14 -0
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+	struct media_pad *pad;
+	int ret;
+
+	if (vidsw->active == -1) {
+		dev_err(sd->dev, "no configuration for inactive mux\n");
+		return -EINVAL;
+	}
+
+	/*
+	 * Retrieve media bus configuration from the entity connected to the
+	 * active input
+	 */
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/media/platform/video-multiplexer.c

Quoted 8 lines
drivers/media/platform/video-multiplexer.c new file +8 -0
+	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
+	if (pad) {
+		sd = media_entity_to_v4l2_subdev(pad->entity);
+		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
+		if (ret == -ENOIOCTLCMD)
+			pad = NULL;
+		else if (ret < 0) {
+			dev_err(sd->dev, "failed to get source 

configuration\n");

Quoted 22 lines
Quoted 9 lines
Quoted 9 lines

drivers/media/platform/video-multiplexer.c

Quoted 9 lines
drivers/media/platform/video-multiplexer.c new file +9 -0
+			return ret;
+		}
+	}
+	if (!pad) {
+		/* Mirror the input side on the output side */
+		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
+		if (cfg->type == V4L2_MBUS_PARALLEL ||
+		    cfg->type == V4L2_MBUS_BT656)
+			cfg->flags = vidsw->endpoint[vidsw-

active].bus.parallel.flags;

Quoted 12 lines
Quoted 12 lines

drivers/media/platform/video-multiplexer.c line 79

Quoted 4 lines
drivers/media/platform/video-multiplexer.c new file +4 -0
+	}
+
+	return 0;
+}

I am not certain this op is needed at all. In the current kernel this op is only used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this information should come from the device tree and there should be no need for this op.

My (tentative) long-term plan was to get rid of this op.

Quoted 7 lines
Quoted 1 line
Quoted 1 line

If you don't need it, then I recommend it is removed.

Hi Hans, the imx-media driver was only calling g_mbus_config to the camera sensor, and it was doing that to determine the sensor's bus type. This info was already available from parsing a v4l2_of_endpoint from the sensor node. So it was simple to remove the g_mbus_config calls, and instead rely on the parsed sensor v4l2_of_endpoint.

That's not a good point. The imx-media driver must not parse the sensor DT node as it is not aware of what bindings the sensor is compatible with. Information must instead be queried from the sensor subdev at runtime, through the g_mbus_config() operation.

Of course, if you can get the information from the imx-media DT node, that's certainly an option. It's only information provided by the sensor driver that you have no choice but query using a subdev operation.

Quoted 44 lines
Quoted 13 lines

We currently use this to make the CSI capture interface understand whether its source from the MIPI CSI-2 or from the parallel bus. That is probably something that should be fixed, but I'm not quite sure how.

The Synopsys DesignWare MIPI CSI-2 reciever turns the incoming MIPI CSI-2 signal into a 32-bit parallel pixel bus plus some signals for the MIPI specific metadata (virtual channel, data type).

Then the CSI2IPU gasket turns this input bus into four separate parallel 16-bit pixel buses plus an 8-bit "mct_di" bus for each of them, that carries the MIPI metadata. The incoming data is split into the four outputs according to the MIPI virtual channel.

Quoted 7 lines

Two of these 16-bit + 8-bit parallel buses are routed through a multiplexer before finally arriving at the CSI on the other side.

We need to configure the CSI to either use or ignore the data from the 8-bit mct_di bus depending on whether the source of the mux is configured to the MIPI CSI-2 receiver / CSI2IPU gasket, or to a parallel input.

Philipp, from my experience, the CSI_MIPI_DI register (configured by ipu_csi_set_mipi_datatype()) can only be given a virtual channel 0, otherwise no data is received from the MIPI CSI-2 sensor, regardless of the virtual channel the sensor is transmitting over.

So it seems the info on the 8-bit mct_di buses generated by the CSI2IPU gasket are ignored by the CSI's, at least the virtual channel number is ignored.

For example, if the sensor is transmitting on vc 1, the gasket routes the sensor data to the parallel bus to CSI1. But if CSI_MIPI_DI on CSI1 is written with vc 1, no data is received.

Steve

Quoted 8 lines

Currently we let g_mbus_config pretend that even the internal 32-bit + metadata and 16-bit + 8-bit metadata parallel buses are of type V4L2_MBUS_CSI so that the CSI can ask the mux, which propagates to the CSI-2 receiver, if connected.

Without g_mbus_config we'd need to get that information from somewhere else. One possibility would be to extend MEDIA_BUS formats to describe these "parallelized MIPI data" buses separately.

Signature · 3 lines
Regards,

Laurent Pinchart
#40 lore raw

Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver

Quoted 61 lines · Laurent Pinchart

Hi Steve,

Quoted 19 lines · On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
Quoted 18 lines · Philipp Zabel
Quoted 17 lines · Hans Verkuil
Quoted 17 lines
(quoted diff) +2 -0
+
+int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct v4l2_mbus_config

*cfg)

drivers/media/platform/video-multiplexer.c new file +14 -0
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+	struct media_pad *pad;
+	int ret;
+
+	if (vidsw->active == -1) {
+		dev_err(sd->dev, "no configuration for inactive mux\n");
+		return -EINVAL;
+	}
+
+	/*
+	 * Retrieve media bus configuration from the entity connected to the
+	 * active input
+	 */
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/media/platform/video-multiplexer.c

Quoted 8 lines
drivers/media/platform/video-multiplexer.c new file +8 -0
+	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
+	if (pad) {
+		sd = media_entity_to_v4l2_subdev(pad->entity);
+		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
+		if (ret == -ENOIOCTLCMD)
+			pad = NULL;
+		else if (ret < 0) {
+			dev_err(sd->dev, "failed to get source 

configuration\n");

Quoted 22 lines
Quoted 9 lines
Quoted 9 lines

drivers/media/platform/video-multiplexer.c

Quoted 9 lines
drivers/media/platform/video-multiplexer.c new file +9 -0
+			return ret;
+		}
+	}
+	if (!pad) {
+		/* Mirror the input side on the output side */
+		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
+		if (cfg->type == V4L2_MBUS_PARALLEL ||
+		    cfg->type == V4L2_MBUS_BT656)
+			cfg->flags = vidsw->endpoint[vidsw-

active].bus.parallel.flags;

Quoted 12 lines
Quoted 12 lines

drivers/media/platform/video-multiplexer.c line 79

Quoted 4 lines
drivers/media/platform/video-multiplexer.c new file +4 -0
+	}
+
+	return 0;
+}

I am not certain this op is needed at all. In the current kernel this op is only used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this information should come from the device tree and there should be no need for this op.

My (tentative) long-term plan was to get rid of this op.

Quoted 7 lines
Quoted 1 line
Quoted 1 line

If you don't need it, then I recommend it is removed.

Hi Hans, the imx-media driver was only calling g_mbus_config to the camera sensor, and it was doing that to determine the sensor's bus type. This info was already available from parsing a v4l2_of_endpoint from the sensor node. So it was simple to remove the g_mbus_config calls, and instead rely on the parsed sensor v4l2_of_endpoint.

Quoted 8 lines

That's not a good point. The imx-media driver must not parse the sensor DT node as it is not aware of what bindings the sensor is compatible with. Information must instead be queried from the sensor subdev at runtime, through the g_mbus_config() operation.

Of course, if you can get the information from the imx-media DT node, that's certainly an option. It's only information provided by the sensor driver that you have no choice but query using a subdev operation.

Shouldn't this come from the imx-media DT node? BTW, why is omap3isp using this?

The reason I am suspicious about this op is that it came from soc-camera and predates the DT. The contents of v4l2_mbus_config seems very much like a HW description to me, i.e. something that belongs in the DT.

Regards,

Hans

maintainer
#41 lore raw

Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver

Hi Hans,

(CC'ing Sakari)

Quoted 4 lines · On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
Quoted 3 lines · Laurent Pinchart
Quoted 2 lines · On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
Quoted 1 line · Philipp Zabel

On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

drivers/media/platform/video-multiplexer.c

Quoted 11 lines
(quoted diff) +2 -0
+
+int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct

v4l2_mbus_config
*cfg)

drivers/media/platform/video-multiplexer.c new file +7 -0
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+	struct media_pad *pad;
+	int ret;
+
+	if (vidsw->active == -1) {
+		dev_err(sd->dev, "no configuration for inactive 

mux\n");

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drivers/media/platform/video-multiplexer.c

Quoted 5 lines
drivers/media/platform/video-multiplexer.c new file +5 -0
+		return -EINVAL;
+	}
+
+	/*
+	 * Retrieve media bus configuration from the entity connected 

to the

Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 34 lines
Quoted 34 lines
Quoted 21 lines
drivers/media/platform/video-multiplexer.c new file +10 -0
+	 * active input
+	 */
+	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
+	if (pad) {
+		sd = media_entity_to_v4l2_subdev(pad->entity);
+		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
+		if (ret == -ENOIOCTLCMD)
+			pad = NULL;
+		else if (ret < 0) {
+			dev_err(sd->dev, "failed to get source

configuration\n");

drivers/media/platform/video-multiplexer.c new file +9 -0
+			return ret;
+		}
+	}
+	if (!pad) {
+		/* Mirror the input side on the output side */
+		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
+		if (cfg->type == V4L2_MBUS_PARALLEL ||
+		    cfg->type == V4L2_MBUS_BT656)
+			cfg->flags = vidsw->endpoint[vidsw-

active].bus.parallel.flags;

drivers/media/platform/video-multiplexer.c line 79

Quoted 4 lines
drivers/media/platform/video-multiplexer.c new file +4 -0
+	}
+
+	return 0;
+}

I am not certain this op is needed at all. In the current kernel this op is only used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this information should come from the device tree and there should be no need for this op.

My (tentative) long-term plan was to get rid of this op.

If you don't need it, then I recommend it is removed.

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

Hi Hans, the imx-media driver was only calling g_mbus_config to the camera sensor, and it was doing that to determine the sensor's bus type. This info was already available from parsing a v4l2_of_endpoint from the sensor node. So it was simple to remove the g_mbus_config calls, and instead rely on the parsed sensor v4l2_of_endpoint.

That's not a good point.

(mea culpa, s/point/idea/)

Quoted 11 lines
Quoted 8 lines

The imx-media driver must not parse the sensor DT node as it is not aware of what bindings the sensor is compatible with. Information must instead be queried from the sensor subdev at runtime, through the g_mbus_config() operation.

Of course, if you can get the information from the imx-media DT node, that's certainly an option. It's only information provided by the sensor driver that you have no choice but query using a subdev operation.

Shouldn't this come from the imx-media DT node? BTW, why is omap3isp using this?

It all depends on what type of information needs to be retrieved, and whether it can change at runtime or is fixed. Adding properties to the imx-media DT node is certainly fine as long as those properties describe the i.MX side.

In the omap3isp case, we use the operation to query whether parallel data contains embedded sync (BT.656) or uses separate h/v sync signals.

Quoted 3 lines

The reason I am suspicious about this op is that it came from soc-camera and predates the DT. The contents of v4l2_mbus_config seems very much like a HW description to me, i.e. something that belongs in the DT.

Part of it is possibly outdated, but for buses that support multiple modes of operation (such as the parallel bus case described above) we need to make that information discoverable at runtime. Maybe this should be considered as related to Sakari's efforts to support VC/DT for CSI-2, and supported through the API he is working on.

Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#42 lore raw

Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver

Quoted 59 lines · Laurent Pinchart

Hi Hans,

(CC'ing Sakari)

Quoted 4 lines · On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
Quoted 3 lines · Laurent Pinchart
Quoted 2 lines · On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
Quoted 1 line · Philipp Zabel

On Fri, 2017-01-20 at 15:03 +0100, Hans Verkuil wrote:

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

Review

Quoted 11 lines
(quoted diff) +2 -0
+
+int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct

v4l2_mbus_config
*cfg)

(quoted diff) +7 -0
+{
+	struct vidsw *vidsw = v4l2_subdev_to_vidsw(sd);
+	struct media_pad *pad;
+	int ret;
+
+	if (vidsw->active == -1) {
+		dev_err(sd->dev, "no configuration for inactive

mux\n");

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+		return -EINVAL;
+	}
+
+	/*
+	 * Retrieve media bus configuration from the entity connected

to the

Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 21 lines
(quoted diff) +10 -0
+	 * active input
+	 */
+	pad = media_entity_remote_pad(&vidsw->pads[vidsw->active]);
+	if (pad) {
+		sd = media_entity_to_v4l2_subdev(pad->entity);
+		ret = v4l2_subdev_call(sd, video, g_mbus_config, cfg);
+		if (ret == -ENOIOCTLCMD)
+			pad = NULL;
+		else if (ret < 0) {
+			dev_err(sd->dev, "failed to get source

configuration\n");

(quoted diff) +9 -0
+			return ret;
+		}
+	}
+	if (!pad) {
+		/* Mirror the input side on the output side */
+		cfg->type = vidsw->endpoint[vidsw->active].bus_type;
+		if (cfg->type == V4L2_MBUS_PARALLEL ||
+		    cfg->type == V4L2_MBUS_BT656)
+			cfg->flags = vidsw->endpoint[vidsw-

active].bus.parallel.flags;

Review

Quoted 4 lines
(quoted diff) +4 -0
+	}
+
+	return 0;
+}

I am not certain this op is needed at all. In the current kernel this op is only used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this information should come from the device tree and there should be no need for this op.

My (tentative) long-term plan was to get rid of this op.

Quoted 11 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line

If you don't need it, then I recommend it is removed.

Hi Hans, the imx-media driver was only calling g_mbus_config to the camera sensor, and it was doing that to determine the sensor's bus type. This info was already available from parsing a v4l2_of_endpoint from the sensor node. So it was simple to remove the g_mbus_config calls, and instead rely on the parsed sensor v4l2_of_endpoint.

That's not a good point.

(mea culpa, s/point/idea/)

Quoted 2 lines
Quoted 2 lines

The imx-media driver must not parse the sensor DT node as it is not aware of what bindings the sensor is compatible with.

Hi Laurent,

I don't really understand this argument. The sensor node has been found by parsing the OF graph, so it is known to be a camera sensor node at that point.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Information must instead
be queried from the sensor subdev at runtime, through the g_mbus_config() operation.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Of course, if you can get the information from the imx-media DT node, that's certainly an option. It's only information provided by the sensor driver that you have no choice but query using a subdev operation.

Shouldn't this come from the imx-media DT node? BTW, why is omap3isp using this?

It all depends on what type of information needs to be retrieved, and whether it can change at runtime or is fixed. Adding properties to the imx-media DT node is certainly fine as long as those properties describe the i.MX side.

In this case the info needed is the media bus type. That info is most easily available by calling v4l2_of_parse_endpoint() on the sensor's endpoint node.

The media bus type is not something that can be added to the imx-media node since it contains no endpoint nodes.

Quoted 11 lines

In the omap3isp case, we use the operation to query whether parallel data contains embedded sync (BT.656) or uses separate h/v sync signals.

Quoted 3 lines

The reason I am suspicious about this op is that it came from soc-camera and predates the DT. The contents of v4l2_mbus_config seems very much like a HW description to me, i.e. something that belongs in the DT.

Part of it is possibly outdated, but for buses that support multiple modes of operation (such as the parallel bus case described above) we need to make that information discoverable at runtime. Maybe this should be considered as related to Sakari's efforts to support VC/DT for CSI-2, and supported through the API he is working on.

That sounds interesting, can you point me to some info on this effort? I've been thinking the DT should contain virtual channel info for CSI-2 buses.

Steve

Quoted 1 line
maintainer
#43 lore raw

Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver

Hi Steve,

Quoted 9 lines · On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
Quoted 8 lines · Laurent Pinchart
Quoted 7 lines · On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
Quoted 6 lines · Laurent Pinchart
Quoted 5 lines · On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
Quoted 4 lines · Philipp Zabel
Quoted 3 lines · Hans Verkuil
Quoted 3 lines
(quoted diff) +2 -0
+
+int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct

v4l2_mbus_config *cfg)

[snip]

Quoted 19 lines
Quoted 19 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines

I am not certain this op is needed at all. In the current kernel this op is only used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this information should come from the device tree and there should be no need for this op.

My (tentative) long-term plan was to get rid of this op.

If you don't need it, then I recommend it is removed.

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

Hi Hans, the imx-media driver was only calling g_mbus_config to the camera sensor, and it was doing that to determine the sensor's bus type. This info was already available from parsing a v4l2_of_endpoint from the sensor node. So it was simple to remove the g_mbus_config calls, and instead rely on the parsed sensor v4l2_of_endpoint.

That's not a good point.

(mea culpa, s/point/idea/)

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

The imx-media driver must not parse the sensor DT node as it is not aware of what bindings the sensor is compatible with.

Hi Laurent,

I don't really understand this argument. The sensor node has been found by parsing the OF graph, so it is known to be a camera sensor node at that point.

All you know in the i.MX6 driver is that the remote node is a video source. You can rely on the fact that it implements the OF graph bindings to locate other ports in that DT node, but that's more or less it.

DT properties are defined by DT bindings and thus qualified by a compatible string. Unless you match on sensor compat strings in the i.MX6 driver (which you shouldn't do, to keep the driver generic) you can't know for certain how to parse the sensor node DT properties. For all you know, the video source could be a bridge such as an HDMI to CSI-2 converter for instance, so you can't even rely on the fact that it's a sensor.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines

Information must instead be queried from the sensor subdev at runtime, through the g_mbus_config() operation.

Of course, if you can get the information from the imx-media DT node, that's certainly an option. It's only information provided by the sensor driver that you have no choice but query using a subdev operation.

Shouldn't this come from the imx-media DT node? BTW, why is omap3isp using this?

Quoted 8 lines
Quoted 4 lines

It all depends on what type of information needs to be retrieved, and whether it can change at runtime or is fixed. Adding properties to the imx-media DT node is certainly fine as long as those properties describe the i.MX side.

In this case the info needed is the media bus type. That info is most easily available by calling v4l2_of_parse_endpoint() on the sensor's endpoint node.

I haven't had time to check the code in details yet, so I can't really comment on what you need and how it should be implemented exactly.

Quoted 2 lines

The media bus type is not something that can be added to the imx-media node since it contains no endpoint nodes.

Agreed. You have endpoints in the CSI nodes though.

Quoted 7 lines
Quoted 7 lines

In the omap3isp case, we use the operation to query whether parallel data contains embedded sync (BT.656) or uses separate h/v sync signals.

Quoted 3 lines

The reason I am suspicious about this op is that it came from soc-camera and predates the DT. The contents of v4l2_mbus_config seems very much like a HW description to me, i.e. something that belongs in the DT.

Quoted 7 lines
Quoted 5 lines

Part of it is possibly outdated, but for buses that support multiple modes of operation (such as the parallel bus case described above) we need to make that information discoverable at runtime. Maybe this should be considered as related to Sakari's efforts to support VC/DT for CSI-2, and supported through the API he is working on.

That sounds interesting, can you point me to some info on this effort?

Sure.

http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc

Quoted 2 lines

I've been thinking the DT should contain virtual channel info for CSI-2 buses.

I don't think it should. CSI-2 virtual channels and data types should be handled as a software concept, and thus supported through driver code without involving DT.

Signature · 3 lines
Regards,

Laurent Pinchart
maintainer
#44 lore raw

Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver

Quoted 98 lines · Laurent Pinchart

Hi Steve,

Quoted 9 lines · On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
Quoted 8 lines · Laurent Pinchart
Quoted 7 lines · On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
Quoted 6 lines · Laurent Pinchart
Quoted 5 lines · On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
Quoted 4 lines · Philipp Zabel
Quoted 3 lines · Hans Verkuil
Quoted 3 lines
(quoted diff) +2 -0
+
+int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct

v4l2_mbus_config *cfg)

[snip]

Quoted 19 lines
Quoted 19 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines

I am not certain this op is needed at all. In the current kernel this op is only used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this information should come from the device tree and there should be no need for this op.

My (tentative) long-term plan was to get rid of this op.

If you don't need it, then I recommend it is removed.

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

Hi Hans, the imx-media driver was only calling g_mbus_config to the camera sensor, and it was doing that to determine the sensor's bus type. This info was already available from parsing a v4l2_of_endpoint from the sensor node. So it was simple to remove the g_mbus_config calls, and instead rely on the parsed sensor v4l2_of_endpoint.

That's not a good point.

(mea culpa, s/point/idea/)

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

The imx-media driver must not parse the sensor DT node as it is not aware of what bindings the sensor is compatible with.

Hi Laurent,

I don't really understand this argument. The sensor node has been found by parsing the OF graph, so it is known to be a camera sensor node at that point.

All you know in the i.MX6 driver is that the remote node is a video source. You can rely on the fact that it implements the OF graph bindings to locate other ports in that DT node, but that's more or less it.

DT properties are defined by DT bindings and thus qualified by a compatible string. Unless you match on sensor compat strings in the i.MX6 driver (which you shouldn't do, to keep the driver generic) you can't know for certain how to parse the sensor node DT properties. For all you know, the video source could be a bridge such as an HDMI to CSI-2 converter for instance, so you can't even rely on the fact that it's a sensor.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines

Information must instead be queried from the sensor subdev at runtime, through the g_mbus_config() operation.

Of course, if you can get the information from the imx-media DT node, that's certainly an option. It's only information provided by the sensor driver that you have no choice but query using a subdev operation.

Shouldn't this come from the imx-media DT node? BTW, why is omap3isp using this?

Quoted 8 lines
Quoted 4 lines

It all depends on what type of information needs to be retrieved, and whether it can change at runtime or is fixed. Adding properties to the imx-media DT node is certainly fine as long as those properties describe the i.MX side.

In this case the info needed is the media bus type. That info is most easily available by calling v4l2_of_parse_endpoint() on the sensor's endpoint node.

I haven't had time to check the code in details yet, so I can't really comment on what you need and how it should be implemented exactly.

Quoted 2 lines

The media bus type is not something that can be added to the imx-media node since it contains no endpoint nodes.

Agreed. You have endpoints in the CSI nodes though.

Quoted 7 lines
Quoted 7 lines

In the omap3isp case, we use the operation to query whether parallel data contains embedded sync (BT.656) or uses separate h/v sync signals.

Quoted 3 lines

The reason I am suspicious about this op is that it came from soc-camera and predates the DT. The contents of v4l2_mbus_config seems very much like a HW description to me, i.e. something that belongs in the DT.

Quoted 7 lines
Quoted 5 lines

Part of it is possibly outdated, but for buses that support multiple modes of operation (such as the parallel bus case described above) we need to make that information discoverable at runtime. Maybe this should be considered as related to Sakari's efforts to support VC/DT for CSI-2, and supported through the API he is working on.

That sounds interesting, can you point me to some info on this effort?

Sure.

Quoted 8 lines

http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc

Quoted 2 lines

I've been thinking the DT should contain virtual channel info for CSI-2 buses.

I don't think it should. CSI-2 virtual channels and data types should be handled as a software concept, and thus supported through driver code without involving DT.

I agree. The CSI2IPU gasket is a bit special in that it distributes its input data to four different parallel buses depending on the input's VC, but upstream of the MIPI CSI-2 receiver, any virtual channel information is purely a matter of the data sent over the CSI-2 link, and not board specific hardware description.

regards
Philipp

maintainer
#45 lore raw

Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver

Hi Philipp,

Quoted 109 lines · On Tuesday 07 Feb 2017 11:41:30 Philipp Zabel wrote:
Quoted 99 lines · Laurent Pinchart
Quoted 9 lines · On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
Quoted 8 lines · Laurent Pinchart
Quoted 7 lines · On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
Quoted 6 lines · Laurent Pinchart
Quoted 5 lines · On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
Quoted 4 lines · Philipp Zabel
Quoted 3 lines · Hans Verkuil
Quoted 3 lines
(quoted diff) +2 -0
+
+int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct

v4l2_mbus_config *cfg)

[snip]

Quoted 20 lines
Quoted 20 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines

I am not certain this op is needed at all. In the current kernel this op is only used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this information should come from the device tree and there should be no need for this op.

My (tentative) long-term plan was to get rid of this op.

If you don't need it, then I recommend it is removed.

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

Hi Hans, the imx-media driver was only calling g_mbus_config to the camera sensor, and it was doing that to determine the sensor's bus type. This info was already available from parsing a v4l2_of_endpoint from the sensor node. So it was simple to remove the g_mbus_config calls, and instead rely on the parsed sensor v4l2_of_endpoint.

That's not a good point.

(mea culpa, s/point/idea/)

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

The imx-media driver must not parse the sensor DT node as it is not aware of what bindings the sensor is compatible with.

Hi Laurent,

I don't really understand this argument. The sensor node has been found by parsing the OF graph, so it is known to be a camera sensor node at that point.

All you know in the i.MX6 driver is that the remote node is a video source. You can rely on the fact that it implements the OF graph bindings to locate other ports in that DT node, but that's more or less it.

DT properties are defined by DT bindings and thus qualified by a compatible string. Unless you match on sensor compat strings in the i.MX6 driver (which you shouldn't do, to keep the driver generic) you can't know for certain how to parse the sensor node DT properties. For all you know, the video source could be a bridge such as an HDMI to CSI-2 converter for instance, so you can't even rely on the fact that it's a sensor.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines

Information must instead be queried from the sensor subdev at runtime, through the g_mbus_config() operation.

Of course, if you can get the information from the imx-media DT node, that's certainly an option. It's only information provided by the sensor driver that you have no choice but query using a subdev operation.

Shouldn't this come from the imx-media DT node? BTW, why is omap3isp using this?

Quoted 8 lines
Quoted 4 lines

It all depends on what type of information needs to be retrieved, and whether it can change at runtime or is fixed. Adding properties to the imx-media DT node is certainly fine as long as those properties describe the i.MX side.

In this case the info needed is the media bus type. That info is most easily available by calling v4l2_of_parse_endpoint() on the sensor's endpoint node.

I haven't had time to check the code in details yet, so I can't really comment on what you need and how it should be implemented exactly.

Quoted 2 lines

The media bus type is not something that can be added to the imx-media node since it contains no endpoint nodes.

Agreed. You have endpoints in the CSI nodes though.

Quoted 8 lines
Quoted 8 lines

In the omap3isp case, we use the operation to query whether parallel data contains embedded sync (BT.656) or uses separate h/v sync signals.

Quoted 4 lines

The reason I am suspicious about this op is that it came from soc-camera and predates the DT. The contents of v4l2_mbus_config seems very much like a HW description to me, i.e. something that belongs in the DT.

Quoted 7 lines
Quoted 5 lines

Part of it is possibly outdated, but for buses that support multiple modes of operation (such as the parallel bus case described above) we need to make that information discoverable at runtime. Maybe this should be considered as related to Sakari's efforts to support VC/DT for CSI-2, and supported through the API he is working on.

That sounds interesting, can you point me to some info on this effort?

Sure.

Quoted 8 lines

http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc

Quoted 2 lines

I've been thinking the DT should contain virtual channel info for CSI-2 buses.

I don't think it should. CSI-2 virtual channels and data types should be handled as a software concept, and thus supported through driver code without involving DT.

Quoted 5 lines

I agree. The CSI2IPU gasket is a bit special in that it distributes its input data to four different parallel buses depending on the input's VC, but upstream of the MIPI CSI-2 receiver, any virtual channel information is purely a matter of the data sent over the CSI-2 link, and not board specific hardware description.

If the CSI2IPU gasket has four physical output buses then it can be modelled as an entity with four source pads. Only when VC/DT are multiplexed on the same physical bus do I think they should be handled without involving the device tree and the media controller graph.

Signature · 3 lines
Regards,

Laurent Pinchart
#46 lore raw

Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver

In reply to Laurent Pinchart

Laurent Pinchart <[email protected]> wrote on Tue [2017-Feb-07 12:26:32 +0200]:

Quoted 98 lines

Hi Steve,

Quoted 9 lines · On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
Quoted 8 lines · Laurent Pinchart
Quoted 7 lines · On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
Quoted 6 lines · Laurent Pinchart
Quoted 5 lines · On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
Quoted 4 lines · Philipp Zabel
Quoted 3 lines · Hans Verkuil
Quoted 3 lines
(quoted diff) +2 -0
+
+int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct

v4l2_mbus_config *cfg)

[snip]

Quoted 19 lines
Quoted 19 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines

I am not certain this op is needed at all. In the current kernel this op is only used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this information should come from the device tree and there should be no need for this op.

My (tentative) long-term plan was to get rid of this op.

If you don't need it, then I recommend it is removed.

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

Hi Hans, the imx-media driver was only calling g_mbus_config to the camera sensor, and it was doing that to determine the sensor's bus type. This info was already available from parsing a v4l2_of_endpoint from the sensor node. So it was simple to remove the g_mbus_config calls, and instead rely on the parsed sensor v4l2_of_endpoint.

That's not a good point.

(mea culpa, s/point/idea/)

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

The imx-media driver must not parse the sensor DT node as it is not aware of what bindings the sensor is compatible with.

Hi Laurent,

I don't really understand this argument. The sensor node has been found by parsing the OF graph, so it is known to be a camera sensor node at that point.

All you know in the i.MX6 driver is that the remote node is a video source. You can rely on the fact that it implements the OF graph bindings to locate other ports in that DT node, but that's more or less it.

DT properties are defined by DT bindings and thus qualified by a compatible string. Unless you match on sensor compat strings in the i.MX6 driver (which you shouldn't do, to keep the driver generic) you can't know for certain how to parse the sensor node DT properties. For all you know, the video source could be a bridge such as an HDMI to CSI-2 converter for instance, so you can't even rely on the fact that it's a sensor.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines

Information must instead be queried from the sensor subdev at runtime, through the g_mbus_config() operation.

Of course, if you can get the information from the imx-media DT node, that's certainly an option. It's only information provided by the sensor driver that you have no choice but query using a subdev operation.

Shouldn't this come from the imx-media DT node? BTW, why is omap3isp using this?

Quoted 8 lines
Quoted 4 lines

It all depends on what type of information needs to be retrieved, and whether it can change at runtime or is fixed. Adding properties to the imx-media DT node is certainly fine as long as those properties describe the i.MX side.

In this case the info needed is the media bus type. That info is most easily available by calling v4l2_of_parse_endpoint() on the sensor's endpoint node.

I haven't had time to check the code in details yet, so I can't really comment on what you need and how it should be implemented exactly.

Quoted 2 lines

The media bus type is not something that can be added to the imx-media node since it contains no endpoint nodes.

Agreed. You have endpoints in the CSI nodes though.

Quoted 7 lines
Quoted 7 lines

In the omap3isp case, we use the operation to query whether parallel data contains embedded sync (BT.656) or uses separate h/v sync signals.

Quoted 3 lines

The reason I am suspicious about this op is that it came from soc-camera and predates the DT. The contents of v4l2_mbus_config seems very much like a HW description to me, i.e. something that belongs in the DT.

Quoted 7 lines
Quoted 5 lines

Part of it is possibly outdated, but for buses that support multiple modes of operation (such as the parallel bus case described above) we need to make that information discoverable at runtime. Maybe this should be considered as related to Sakari's efforts to support VC/DT for CSI-2, and supported through the API he is working on.

That sounds interesting, can you point me to some info on this effort?

Sure.

Quoted 8 lines

http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc

Quoted 2 lines

I've been thinking the DT should contain virtual channel info for CSI-2 buses.

I don't think it should. CSI-2 virtual channels and data types should be handled as a software concept, and thus supported through driver code without involving DT.

Laurent,

So when you have a CSI2 port aggregator for instance where traffic from up to 4 CSI2 sources where each source is now assigned its own VC by the aggregator and interleaved into a single CSI2 Receiver. I was hoping that in this case the VC would be DT discoverable as a specicic source identifier. So the CSI-RX side could associate a specific source and create its own video device. I am guessing that no such thing exist today?

Benoit

Quoted 6 lines
Signature · 4 lines
-- 
Regards,

Laurent Pinchart
#47 lore raw

Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver

Hi Benoit,

Quoted 108 lines · Benoit Parrot

Laurent Pinchart <[email protected]> wrote on Tue [2017-Feb-07 12:26:32 +0200]:

Quoted 98 lines

Hi Steve,

Quoted 9 lines · On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
Quoted 8 lines · Laurent Pinchart
Quoted 7 lines · On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
Quoted 6 lines · Laurent Pinchart
Quoted 5 lines · On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
Quoted 4 lines · Philipp Zabel
Quoted 3 lines · Hans Verkuil
Quoted 3 lines
(quoted diff) +2 -0
+
+int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct

v4l2_mbus_config *cfg)

[snip]

Quoted 19 lines
Quoted 19 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines

I am not certain this op is needed at all. In the current kernel this op is only used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this information should come from the device tree and there should be no need for this op.

My (tentative) long-term plan was to get rid of this op.

If you don't need it, then I recommend it is removed.

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

Hi Hans, the imx-media driver was only calling g_mbus_config to the camera sensor, and it was doing that to determine the sensor's bus type. This info was already available from parsing a v4l2_of_endpoint from the sensor node. So it was simple to remove the g_mbus_config calls, and instead rely on the parsed sensor v4l2_of_endpoint.

That's not a good point.

(mea culpa, s/point/idea/)

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

The imx-media driver must not parse the sensor DT node as it is not aware of what bindings the sensor is compatible with.

Hi Laurent,

I don't really understand this argument. The sensor node has been found by parsing the OF graph, so it is known to be a camera sensor node at that point.

All you know in the i.MX6 driver is that the remote node is a video source. You can rely on the fact that it implements the OF graph bindings to locate other ports in that DT node, but that's more or less it.

DT properties are defined by DT bindings and thus qualified by a compatible string. Unless you match on sensor compat strings in the i.MX6 driver (which you shouldn't do, to keep the driver generic) you can't know for certain how to parse the sensor node DT properties. For all you know, the video source could be a bridge such as an HDMI to CSI-2 converter for instance, so you can't even rely on the fact that it's a sensor.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines

Information must instead be queried from the sensor subdev at runtime, through the g_mbus_config() operation.

Of course, if you can get the information from the imx-media DT node, that's certainly an option. It's only information provided by the sensor driver that you have no choice but query using a subdev operation.

Shouldn't this come from the imx-media DT node? BTW, why is omap3isp using this?

Quoted 8 lines
Quoted 4 lines

It all depends on what type of information needs to be retrieved, and whether it can change at runtime or is fixed. Adding properties to the imx-media DT node is certainly fine as long as those properties describe the i.MX side.

In this case the info needed is the media bus type. That info is most easily available by calling v4l2_of_parse_endpoint() on the sensor's endpoint node.

I haven't had time to check the code in details yet, so I can't really comment on what you need and how it should be implemented exactly.

Quoted 2 lines

The media bus type is not something that can be added to the imx-media node since it contains no endpoint nodes.

Agreed. You have endpoints in the CSI nodes though.

Quoted 7 lines
Quoted 7 lines

In the omap3isp case, we use the operation to query whether parallel data contains embedded sync (BT.656) or uses separate h/v sync signals.

Quoted 3 lines

The reason I am suspicious about this op is that it came from soc-camera and predates the DT. The contents of v4l2_mbus_config seems very much like a HW description to me, i.e. something that belongs in the DT.

Quoted 7 lines
Quoted 5 lines

Part of it is possibly outdated, but for buses that support multiple modes of operation (such as the parallel bus case described above) we need to make that information discoverable at runtime. Maybe this should be considered as related to Sakari's efforts to support VC/DT for CSI-2, and supported through the API he is working on.

That sounds interesting, can you point me to some info on this effort?

Sure.

Quoted 8 lines

http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc

Quoted 2 lines

I've been thinking the DT should contain virtual channel info for CSI-2 buses.

I don't think it should. CSI-2 virtual channels and data types should be handled as a software concept, and thus supported through driver code without involving DT.

Quoted 8 lines

Laurent,

So when you have a CSI2 port aggregator for instance where traffic from up to 4 CSI2 sources where each source is now assigned its own VC by the aggregator and interleaved into a single CSI2 Receiver. I was hoping that in this case the VC would be DT discoverable as a specicic source identifier. So the CSI-RX side could associate a specific source and create its own video device. I am guessing that no such thing exist today?

This should be configurable in software: the sensors connected to the aggregator may also output multiple streams. The number of streams may also depend on the user specified configuration over the MC / V4L2 sub-device interfaces. Thus, this needs to be configurable in software.

We do need additional patches on top of the current mediatree.git master branch though, some of which are not yet written...

Signature · 4 lines
Kind regards,

Sakari Ailus
e-mail: sakari.ailus at iki.fi	XMPP: sailus at retiisi.org.uk
maintainer
#48 lore raw

Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver

In reply to Benoit Parrot

Hi Benoit,

Quoted 109 lines · On Tuesday 07 Feb 2017 07:36:48 Benoit Parrot wrote:

Laurent Pinchart wrote on Tue [2017-Feb-07 12:26:32 +0200]:

Quoted 99 lines · On Monday 06 Feb 2017 15:10:46 Steve Longerbeam wrote:
Quoted 9 lines · Laurent Pinchart
Quoted 8 lines · On Monday 06 Feb 2017 10:50:22 Hans Verkuil wrote:
Quoted 7 lines · Laurent Pinchart
Quoted 6 lines · On Tuesday 24 Jan 2017 18:07:55 Steve Longerbeam wrote:
Quoted 5 lines · Philipp Zabel
Quoted 4 lines · Hans Verkuil
Quoted 3 lines
Quoted 3 lines
(quoted diff) +2 -0
+
+int vidsw_g_mbus_config(struct v4l2_subdev *sd, struct

v4l2_mbus_config *cfg)

[snip]

Quoted 20 lines
Quoted 20 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines

I am not certain this op is needed at all. In the current kernel this op is only used by soc_camera, pxa_camera and omap3isp (somewhat dubious). Normally this information should come from the device tree and there should be no need for this op.

My (tentative) long-term plan was to get rid of this op.

If you don't need it, then I recommend it is removed.

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

Hi Hans, the imx-media driver was only calling g_mbus_config to the camera sensor, and it was doing that to determine the sensor's bus type. This info was already available from parsing a v4l2_of_endpoint from the sensor node. So it was simple to remove the g_mbus_config calls, and instead rely on the parsed sensor v4l2_of_endpoint.

That's not a good point.

(mea culpa, s/point/idea/)

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

The imx-media driver must not parse the sensor DT node as it is not aware of what bindings the sensor is compatible with.

Hi Laurent,

I don't really understand this argument. The sensor node has been found by parsing the OF graph, so it is known to be a camera sensor node at that point.

All you know in the i.MX6 driver is that the remote node is a video source. You can rely on the fact that it implements the OF graph bindings to locate other ports in that DT node, but that's more or less it.

DT properties are defined by DT bindings and thus qualified by a compatible string. Unless you match on sensor compat strings in the i.MX6 driver (which you shouldn't do, to keep the driver generic) you can't know for certain how to parse the sensor node DT properties. For all you know, the video source could be a bridge such as an HDMI to CSI-2 converter for instance, so you can't even rely on the fact that it's a sensor.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines

Information must instead be queried from the sensor subdev at runtime, through the g_mbus_config() operation.

Of course, if you can get the information from the imx-media DT node, that's certainly an option. It's only information provided by the sensor driver that you have no choice but query using a subdev operation.

Shouldn't this come from the imx-media DT node? BTW, why is omap3isp using this?

Quoted 8 lines
Quoted 4 lines

It all depends on what type of information needs to be retrieved, and whether it can change at runtime or is fixed. Adding properties to the imx-media DT node is certainly fine as long as those properties describe the i.MX side.

In this case the info needed is the media bus type. That info is most easily available by calling v4l2_of_parse_endpoint() on the sensor's endpoint node.

I haven't had time to check the code in details yet, so I can't really comment on what you need and how it should be implemented exactly.

Quoted 2 lines

The media bus type is not something that can be added to the imx-media node since it contains no endpoint nodes.

Agreed. You have endpoints in the CSI nodes though.

Quoted 8 lines
Quoted 8 lines

In the omap3isp case, we use the operation to query whether parallel data contains embedded sync (BT.656) or uses separate h/v sync signals.

Quoted 4 lines

The reason I am suspicious about this op is that it came from soc-camera and predates the DT. The contents of v4l2_mbus_config seems very much like a HW description to me, i.e. something that belongs in the DT.

Quoted 7 lines
Quoted 5 lines

Part of it is possibly outdated, but for buses that support multiple modes of operation (such as the parallel bus case described above) we need to make that information discoverable at runtime. Maybe this should be considered as related to Sakari's efforts to support VC/DT for CSI-2, and supported through the API he is working on.

That sounds interesting, can you point me to some info on this effort?

Sure.

Quoted 8 lines

http://git.retiisi.org.uk/?p=~sailus/linux.git;a=shortlog;h=refs/heads/vc

Quoted 2 lines

I've been thinking the DT should contain virtual channel info for CSI-2 buses.

I don't think it should. CSI-2 virtual channels and data types should be handled as a software concept, and thus supported through driver code without involving DT.

Quoted 8 lines

Laurent,

So when you have a CSI2 port aggregator for instance where traffic from up to 4 CSI2 sources where each source is now assigned its own VC by the aggregator and interleaved into a single CSI2 Receiver. I was hoping that in this case the VC would be DT discoverable as a specicic source identifier. So the CSI-RX side could associate a specific source and create its own video device.

In this specific example, I believe the aggregator should be modelled with 4 input ports and one output port in DT, and with 4 sink pads and one source pad in MC. Information about the VCs multiplexed over the aggregator's source link should not be part of the device tree, but should be discoverable at runtime through V4L2 subdev operations. This would include information about how the 4 input streams are routed to VCs inside the aggregator.

Quoted 1 line

I am guessing that no such thing exist today?

There's very little (to not say nothing) in terms of VC support in V4L2 today.

Signature · 3 lines
Regards,

Laurent Pinchart
#49 lore raw

Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver

In reply to Steve Longerbeam

Hello Steve,

Quoted 1 line · Steve Longerbeam

From: Philipp Zabel <[email protected]>

[snip]

drivers/media/platform/Kconfig config VIDEO_M32R_AR_M64278 · line 79

Quoted 4 lines
drivers/media/platform/Kconfig +3 -0
+config VIDEO_MULTIPLEXER
+       tristate "Video Multiplexer"
+       depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER

The driver can be build as a module...

drivers/media/platform/video-multiplexer.c line 457

Quoted 6 lines
drivers/media/platform/video-multiplexer.c new file +6 -0
+
+static const struct of_device_id vidsw_dt_ids[] = {
+       { .compatible = "video-multiplexer", },
+       { /* sentinel */ }
+};
+

... so you need a MODULE_DEVICE_TABLE(of, vidsw_dt_ids) here or otherwise module autoloading won't work.

Best regards,
Javier

author maintainer
#50 lore raw

Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver

Quoted 11 lines · Javier Martinez Canillas

Hello Steve,

Quoted 1 line · Steve Longerbeam

From: Philipp Zabel <[email protected]>

[snip]

drivers/media/platform/Kconfig config VIDEO_M32R_AR_M64278 · line 79

Quoted 3 lines
drivers/media/platform/Kconfig +3 -0
+config VIDEO_MULTIPLEXER
+       tristate "Video Multiplexer"
+       depends on VIDEO_V4L2_SUBDEV_API && MEDIA_CONTROLLER

The driver can be build as a module...

Quoted 8 lines

drivers/media/platform/video-multiplexer.c line 457

Quoted 6 lines
drivers/media/platform/video-multiplexer.c new file +6 -0
+
+static const struct of_device_id vidsw_dt_ids[] = {
+       { .compatible = "video-multiplexer", },
+       { /* sentinel */ }
+};
+

... so you need a MODULE_DEVICE_TABLE(of, vidsw_dt_ids) here or otherwise module autoloading won't work.

Hi Javier, thanks for catching, done.

Steve

#51 lore raw

Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver

In reply to Steve Longerbeam

Hi Steve,

Quoted 63 lines · Steve Longerbeam

From: Philipp Zabel <[email protected]>

This driver can handle SoC internal and external video bus multiplexers, controlled either by register bit fields or by a GPIO. The subdevice passes through frame interval and mbus configuration of the active input to the output side.

--

- fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev() should be unregister.

- added media_entity_cleanup() and v4l2_device_unregister_subdev() to vidsw_remove().

- there was a line left over from a previous iteration that negated the new way of determining the pad count just before it which has been removed (num_pads = of_get_child_count(np)).

- Philipp Zabel has developed a set of patches that allow adding to the subdev async notifier waiting list using a chaining method from the async registered callbacks (v4l2_of_subdev_registered() and the prep patches for that). For now, I've removed the use of v4l2_of_subdev_registered() for the vidmux driver's registered callback. This doesn't affect the functionality of this driver, but allows for it to be merged now, before adding the chaining support.

---
 .../bindings/media/video-multiplexer.txt           |  59 +++
 drivers/media/platform/Kconfig                     |   8 +
 drivers/media/platform/Makefile                    |   2 +
 drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
 4 files changed, 541 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
 create mode 100644 drivers/media/platform/video-multiplexer.c
Documentation/devicetree/bindings/media/video-multiplexer.txt new file +16 -0
new file mode 100644
index 0000000..9d133d9
@@ -0,0 +1,59 @@
+Video Multiplexer
+=================
+
+Video multiplexers allow to select between multiple input ports. Video received
+on the active input port is passed through to the output port. Muxes described
+by this binding may be controlled by a syscon register bitfield or by a GPIO.
+
+Required properties:
+- compatible : should be "video-multiplexer"
+- reg: should be register base of the register containing the control bitfield
+- bit-mask: bitmask of the control bitfield in the control register
+- bit-shift: bit offset of the control bitfield in the control register
+- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
+  may be given to switch between two inputs
+- #address-cells: should be <1>
+- #size-cells: should be <0>

Documentation/devicetree/bindings/media/video-multiplexer.txt line 26

Quoted 10 lines
Documentation/devicetree/bindings/media/video-multiplexer.txt new file +10 -0
+- port@*: at least three port nodes containing endpoints connecting to the
+  source and sink devices according to of_graph bindings. The last port is
+  the output port, all others are inputs.
+
+Example:
+
+syscon {
+	compatible = "syscon", "simple-mfd";
+
+	mux {

Could you use standardised properties for this, i.e. ones defined in Documentation/devicetree/bindings/media/video-interfaces.txt ?

This is very similar to another patch "[PATCH] devicetree: Add video bus switch" posted by Pavel Machek recently. The problem with that is also similar than with this one: how to pass the CSI-2 bus configuration to the receiver.

There's some discussion here:

<URL:http://www.spinics.net/lists/linux-media/msg109493.html>

As Laurent already suggested, I think we should have a common solution for the problem that, besides conveying the bus parameters to the receiver, also encompasses CSI-2 virtual channels and data types.

That would mean finishing the series of patches in the branch I believe Laurent already quoted here.

Quoted 33 lines
Documentation/devicetree/bindings/media/video-multiplexer.txt new file +33 -0
+		compatible = "video-multiplexer";
+		/* Single bit (1 << 19) in syscon register 0x04: */
+		reg = <0x04>;
+		bit-mask = <1>;
+		bit-shift = <19>;
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		port at 0 {
+			reg = <0>;
+
+			mux_in0: endpoint {
+				remote-endpoint = <&video_source0_out>;
+			};
+		};
+
+		port at 1 {
+			reg = <1>;
+
+			mux_in1: endpoint {
+				remote-endpoint = <&video_source1_out>;
+			};
+		};
+
+		port at 2 {
+			reg = <2>;
+
+			mux_out: endpoint {
+				remote-endpoint = <&capture_interface_in>;
+			};
+		};
+	};
+};
Signature · 4 lines
Kind regards,

Sakari Ailus
e-mail: sakari.ailus at iki.fi	XMPP: sailus at retiisi.org.uk
maintainer
#52 lore raw

Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver

Hi Sakari,

Quoted 71 lines · Sakari Ailus

Hi Steve,

Quoted 68 lines · Steve Longerbeam

From: Philipp Zabel <[email protected]>

This driver can handle SoC internal and external video bus multiplexers, controlled either by register bit fields or by a GPIO. The subdevice passes through frame interval and mbus configuration of the active input to the output side.

--

- fixed a cut&paste error in vidsw_remove(): v4l2_async_register_subdev() should be unregister.

- added media_entity_cleanup() and v4l2_device_unregister_subdev() to vidsw_remove().

- there was a line left over from a previous iteration that negated the new way of determining the pad count just before it which has been removed (num_pads = of_get_child_count(np)).

- Philipp Zabel has developed a set of patches that allow adding to the subdev async notifier waiting list using a chaining method from the async registered callbacks (v4l2_of_subdev_registered() and the prep patches for that). For now, I've removed the use of v4l2_of_subdev_registered() for the vidmux driver's registered callback. This doesn't affect the functionality of this driver, but allows for it to be merged now, before adding the chaining support.

---
 .../bindings/media/video-multiplexer.txt           |  59 +++
 drivers/media/platform/Kconfig                     |   8 +
 drivers/media/platform/Makefile                    |   2 +
 drivers/media/platform/video-multiplexer.c         | 472 +++++++++++++++++++++
 4 files changed, 541 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/video-multiplexer.txt
 create mode 100644 drivers/media/platform/video-multiplexer.c
Documentation/devicetree/bindings/media/video-multiplexer.txt new file +21 -0
new file mode 100644
index 0000000..9d133d9
@@ -0,0 +1,59 @@
+Video Multiplexer
+=================
+
+Video multiplexers allow to select between multiple input ports. Video received
+on the active input port is passed through to the output port. Muxes described
+by this binding may be controlled by a syscon register bitfield or by a GPIO.
+
+Required properties:
+- compatible : should be "video-multiplexer"
+- reg: should be register base of the register containing the control bitfield
+- bit-mask: bitmask of the control bitfield in the control register
+- bit-shift: bit offset of the control bitfield in the control register
+- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO phandle
+  may be given to switch between two inputs
+- #address-cells: should be <1>
+- #size-cells: should be <0>
+- port@*: at least three port nodes containing endpoints connecting to the
+  source and sink devices according to of_graph bindings. The last port is
+  the output port, all others are inputs.
+
+Example:
Quoted 8 lines

Documentation/devicetree/bindings/media/video-multiplexer.txt line 26

Quoted 5 lines
Documentation/devicetree/bindings/media/video-multiplexer.txt new file +5 -0
+
+syscon {
+	compatible = "syscon", "simple-mfd";
+
+	mux {

Could you use standardised properties for this, i.e. ones defined in Documentation/devicetree/bindings/media/video-interfaces.txt ?

What do you mean? Should we add the optional bus-width property to describe the bit width of the parallel bus even though the driver doesn't care about any of the properties?

Quoted 8 lines

This is very similar to another patch "[PATCH] devicetree: Add video bus switch" posted by Pavel Machek recently. The problem with that is also similar than with this one: how to pass the CSI-2 bus configuration to the receiver.

There's some discussion here:

<URL:http://www.spinics.net/lists/linux-media/msg109493.html>

[Added Sebastian do Cc:]

Yes, this is essentially the same driver, except that this driver also handles MMIO-bitfield controlled muxes, and that the actual physical bus (which the drivers currently don't care about) is MIPI CSI-2 in the other case, and parallel this one.
They should probably be combined, or maybe split into two separate drivers (MMIO controlled mux, GPIO controlled switch), possibly using the same v4l2_subdev boilerplate.

Quoted 3 lines

As Laurent already suggested, I think we should have a common solution for the problem that, besides conveying the bus parameters to the receiver, also encompasses CSI-2 virtual channels and data types.

Yes, that seems to be necessary, as certainly we can't configure the mux output bus parameters in DT to a fixed setting.

Quoted 2 lines

That would mean finishing the series of patches in the branch I believe Laurent already quoted here.

regards
Philipp

maintainer
#53 lore raw

Re: [PATCH v3 13/24] platform: add video-multiplexer subdevice driver

Hi Philipp,

Quoted 106 lines · On Wednesday 08 Feb 2017 10:47:15 Philipp Zabel wrote:
Quoted 81 lines · Sakari Ailus
Quoted 80 lines · Steve Longerbeam

From: Philipp Zabel <[email protected]>

This driver can handle SoC internal and external video bus multiplexers, controlled either by register bit fields or by a GPIO. The subdevice passes through frame interval and mbus configuration of the active input to the output side.

--

- fixed a cut&paste error in vidsw_remove():
v4l2_async_register_subdev()
should be unregister.

- added media_entity_cleanup() and v4l2_device_unregister_subdev() to vidsw_remove().

- there was a line left over from a previous iteration that negated the new way of determining the pad count just before it which has been removed (num_pads = of_get_child_count(np)).

- Philipp Zabel has developed a set of patches that allow adding to the subdev async notifier waiting list using a chaining method from the async registered callbacks (v4l2_of_subdev_registered() and the prep patches for that). For now, I've removed the use of v4l2_of_subdev_registered() for the vidmux driver's registered callback. This doesn't affect the functionality of this driver, but allows for it to be merged now, before adding the chaining support.

---

 .../bindings/media/video-multiplexer.txt           |  59 +++
 drivers/media/platform/Kconfig                     |   8 +
 drivers/media/platform/Makefile                    |   2 +
 drivers/media/platform/video-multiplexer.c         | 472 ++++++++++++++
 4 files changed, 541 insertions(+)
 create mode 100644

Documentation/devicetree/bindings/media/video-multiplexer.txt create mode 100644 drivers/media/platform/video-multiplexer.c

diff --git a/Documentation/devicetree/bindings/media/video-multiplexer.txt b/Documentation/devicetree/bindings/media/video-multiplexer.txt new file mode 100644
index 0000000..9d133d9

Documentation/devicetree/bindings/media/video-multiplexer.txt new file +4 -0
@@ -0,0 +1,59 @@
+Video Multiplexer
+=================
+
+Video multiplexers allow to select between multiple input ports. Video

received
+on the active input port is passed through to the output port. Muxes described
+by this binding may be controlled by a syscon register bitfield or by a GPIO.

Documentation/devicetree/bindings/media/video-multiplexer.txt new file +4 -0
+
+Required properties:
+- compatible : should be "video-multiplexer"
+- reg: should be register base of the register containing the control

bitfield

+- bit-mask: bitmask of the control bitfield in the control register
+- bit-shift: bit offset of the control bitfield in the control register
+- gpios: alternatively to reg, bit-mask, and bit-shift, a single GPIO
phandle
+  may be given to switch between two inputs
+- #address-cells: should be <1>
+- #size-cells: should be <0>
+- port@*: at least three port nodes containing endpoints connecting to

the
+ source and sink devices according to of_graph bindings. The last port is

Documentation/devicetree/bindings/media/video-multiplexer.txt new file +3 -0
+  the output port, all others are inputs.
+
+Example:
Quoted 8 lines

Documentation/devicetree/bindings/media/video-multiplexer.txt line 26

Quoted 5 lines
Documentation/devicetree/bindings/media/video-multiplexer.txt new file +5 -0
+
+syscon {
+	compatible = "syscon", "simple-mfd";
+
+	mux {

Could you use standardised properties for this, i.e. ones defined in Documentation/devicetree/bindings/media/video-interfaces.txt ?

What do you mean? Should we add the optional bus-width property to describe the bit width of the parallel bus even though the driver doesn't care about any of the properties?

Quoted 8 lines

This is very similar to another patch "[PATCH] devicetree: Add video bus switch" posted by Pavel Machek recently. The problem with that is also similar than with this one: how to pass the CSI-2 bus configuration to the receiver.

There's some discussion here:

<URL:http://www.spinics.net/lists/linux-media/msg109493.html>

[Added Sebastian do Cc:]

Quoted 7 lines

Yes, this is essentially the same driver, except that this driver also handles MMIO-bitfield controlled muxes, and that the actual physical bus (which the drivers currently don't care about) is MIPI CSI-2 in the other case, and parallel this one.
They should probably be combined, or maybe split into two separate drivers (MMIO controlled mux, GPIO controlled switch), possibly using the same v4l2_subdev boilerplate.

I'd split the bindings in two with two separate compat strings, but we can handle both in a single driver as most of the code would be identical otherwise.

Quoted 9 lines
Quoted 3 lines

As Laurent already suggested, I think we should have a common solution for the problem that, besides conveying the bus parameters to the receiver, also encompasses CSI-2 virtual channels and data types.

Yes, that seems to be necessary, as certainly we can't configure the mux output bus parameters in DT to a fixed setting.

Quoted 2 lines

That would mean finishing the series of patches in the branch I believe Laurent already quoted here.

Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#54 lore raw

PATCH v3 14/24 UAPI: Add media UAPI Kbuild file

In reply to Steve Longerbeam

Add an empty UAPI Kbuild file for media UAPI headers.

---
 include/uapi/Kbuild       | 1 +
 include/uapi/media/Kbuild | 1 +
 2 files changed, 2 insertions(+)
 create mode 100644 include/uapi/media/Kbuild
include/uapi/Kbuild +1 -0
index 245aa6e..9a51957 100644
@@ -6,6 +6,7 @@
 header-y += asm-generic/
 header-y += linux/
 header-y += sound/
+header-y += media/
 header-y += mtd/
 header-y += rdma/
 header-y += video/
include/uapi/media/Kbuild new file +1 -0
new file mode 100644
index 0000000..aafaa5a
@@ -0,0 +1 @@
+# UAPI Header export list
Signature · 1 line
2.7.4
author maintainer
#55 lore raw

PATCH v3 15/24 media: Add userspace header file for i.MX

In reply to Steve Longerbeam

This adds a header file for use by userspace programs wanting to interact with the i.MX media driver. It defines custom v4l2 controls and events generated by the i.MX v4l2 subdevices.

---
 include/uapi/media/Kbuild |  1 +
 include/uapi/media/imx.h  | 30 ++++++++++++++++++++++++++++++
 2 files changed, 31 insertions(+)
 create mode 100644 include/uapi/media/imx.h
include/uapi/media/Kbuild +1 -0
index aafaa5a..fa78958 100644
@@ -1 +1,2 @@
 # UAPI Header export list
+header-y += imx.h
include/uapi/media/imx.h new file +30 -0
new file mode 100644
index 0000000..2421d9c
@@ -0,0 +1,30 @@
+/*
+ * Copyright (c) 2014-2015 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the
+ * License, or (at your option) any later version
+ */
+
+#ifndef __UAPI_MEDIA_IMX_H__
+#define __UAPI_MEDIA_IMX_H__
+
+/*
+ * events from the subdevs
+ */
+#define V4L2_EVENT_IMX_CLASS          V4L2_EVENT_PRIVATE_START
+#define V4L2_EVENT_IMX_NFB4EOF        (V4L2_EVENT_IMX_CLASS + 1)
+#define V4L2_EVENT_IMX_EOF_TIMEOUT    (V4L2_EVENT_IMX_CLASS + 2)
+#define V4L2_EVENT_IMX_FRAME_INTERVAL (V4L2_EVENT_IMX_CLASS + 3)
+
+enum imx_ctrl_id {
+	V4L2_CID_IMX_MOTION = (V4L2_CID_USER_IMX_BASE + 0),
+	V4L2_CID_IMX_FIM_ENABLE,
+	V4L2_CID_IMX_FIM_NUM,
+	V4L2_CID_IMX_FIM_TOLERANCE_MIN,
+	V4L2_CID_IMX_FIM_TOLERANCE_MAX,
+	V4L2_CID_IMX_FIM_NUM_SKIP,
+};
+
+#endif
Signature · 1 line
2.7.4
maintainer
#56 lore raw

Re: [PATCH v3 15/24] media: Add userspace header file for i.MX

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:

Quoted 33 lines

This adds a header file for use by userspace programs wanting to interact with the i.MX media driver. It defines custom v4l2 controls and events generated by the i.MX v4l2 subdevices.

---
 include/uapi/media/Kbuild |  1 +
 include/uapi/media/imx.h  | 30 ++++++++++++++++++++++++++++++
 2 files changed, 31 insertions(+)
 create mode 100644 include/uapi/media/imx.h
include/uapi/media/Kbuild +1 -0
index aafaa5a..fa78958 100644
@@ -1 +1,2 @@
 # UAPI Header export list
+header-y += imx.h
include/uapi/media/imx.h new file +9 -0
new file mode 100644
index 0000000..2421d9c
@@ -0,0 +1,30 @@
+/*
+ * Copyright (c) 2014-2015 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the
+ * License, or (at your option) any later version
+ */
+

include/uapi/media/imx.h line 19

Quoted 10 lines
include/uapi/media/imx.h new file +10 -0
+#ifndef __UAPI_MEDIA_IMX_H__
+#define __UAPI_MEDIA_IMX_H__
+
+/*
+ * events from the subdevs
+ */
+#define V4L2_EVENT_IMX_CLASS          V4L2_EVENT_PRIVATE_START
+#define V4L2_EVENT_IMX_NFB4EOF        (V4L2_EVENT_IMX_CLASS + 1)
+#define V4L2_EVENT_IMX_EOF_TIMEOUT    (V4L2_EVENT_IMX_CLASS + 2)
+#define V4L2_EVENT_IMX_FRAME_INTERVAL (V4L2_EVENT_IMX_CLASS + 3)

Aren't these generic enough to warrant common events? I would think there have to be other capture IP cores that can signal aborted frames or frame timeouts.

include/uapi/media/imx.h line 30

Quoted 11 lines
include/uapi/media/imx.h new file +11 -0
+
+enum imx_ctrl_id {
+	V4L2_CID_IMX_MOTION = (V4L2_CID_USER_IMX_BASE + 0),
+	V4L2_CID_IMX_FIM_ENABLE,
+	V4L2_CID_IMX_FIM_NUM,
+	V4L2_CID_IMX_FIM_TOLERANCE_MIN,
+	V4L2_CID_IMX_FIM_TOLERANCE_MAX,
+	V4L2_CID_IMX_FIM_NUM_SKIP,
+};
+
+#endif

regards
Philipp

author maintainer
#57 lore raw

Re: [PATCH v3 15/24] media: Add userspace header file for i.MX

Quoted 39 lines · Philipp Zabel

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:

Quoted 38 lines

This adds a header file for use by userspace programs wanting to interact with the i.MX media driver. It defines custom v4l2 controls and events generated by the i.MX v4l2 subdevices.

---
  include/uapi/media/Kbuild |  1 +
  include/uapi/media/imx.h  | 30 ++++++++++++++++++++++++++++++
  2 files changed, 31 insertions(+)

create mode 100644 include/uapi/media/imx.h

include/uapi/media/Kbuild +1 -0
index aafaa5a..fa78958 100644
@@ -1 +1,2 @@
  # UAPI Header export list
+header-y += imx.h
include/uapi/media/imx.h new file +14 -0
new file mode 100644
index 0000000..2421d9c
@@ -0,0 +1,30 @@
+/*
+ * Copyright (c) 2014-2015 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the
+ * License, or (at your option) any later version
+ */
+
+#ifndef __UAPI_MEDIA_IMX_H__
+#define __UAPI_MEDIA_IMX_H__
+
+/*
+ * events from the subdevs
Quoted 8 lines

include/uapi/media/imx.h line 19

Quoted 5 lines
include/uapi/media/imx.h new file +5 -0
+ */
+#define V4L2_EVENT_IMX_CLASS          V4L2_EVENT_PRIVATE_START
+#define V4L2_EVENT_IMX_NFB4EOF        (V4L2_EVENT_IMX_CLASS + 1)
+#define V4L2_EVENT_IMX_EOF_TIMEOUT    (V4L2_EVENT_IMX_CLASS + 2)
+#define V4L2_EVENT_IMX_FRAME_INTERVAL (V4L2_EVENT_IMX_CLASS + 3)

Aren't these generic enough to warrant common events? I would think there have to be other capture IP cores that can signal aborted frames or frame timeouts.

Yes, agreed. A frame capture timeout, or frame interval error, are both generic concepts. At some point it would be great to make the Frame Interval Monitor generally available under v4l2-core. As for the EOF timeout event, I'll look into moving that into a generic V4L2 event.

Steve

maintainer
#58 lore raw

Re: [PATCH v3 15/24] media: Add userspace header file for i.MX

Hi Steve,

Quoted 53 lines · On Friday 13 Jan 2017 15:13:33 Steve Longerbeam wrote:
Quoted 43 lines · Philipp Zabel

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:

Quoted 42 lines

This adds a header file for use by userspace programs wanting to interact with the i.MX media driver. It defines custom v4l2 controls and events generated by the i.MX v4l2 subdevices.

---

  include/uapi/media/Kbuild |  1 +
  include/uapi/media/imx.h  | 30 ++++++++++++++++++++++++++++++
  2 files changed, 31 insertions(+)

create mode 100644 include/uapi/media/imx.h

include/uapi/media/Kbuild +1 -0
index aafaa5a..fa78958 100644
@@ -1 +1,2 @@
 
  # UAPI Header export list
 
+header-y += imx.h
include/uapi/media/imx.h new file +5 -0
new file mode 100644
index 0000000..2421d9c
@@ -0,0 +1,30 @@
+/*
+ * Copyright (c) 2014-2015 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by

the + * Free Software Foundation; either version 2 of the
+ * License, or (at your option) any later version
+ */

include/uapi/media/imx.h new file +7 -0
+
+#ifndef __UAPI_MEDIA_IMX_H__
+#define __UAPI_MEDIA_IMX_H__
+
+/*
+ * events from the subdevs
+ */
Quoted 8 lines

include/uapi/media/imx.h line 19

Quoted 4 lines
include/uapi/media/imx.h new file +4 -0
+#define V4L2_EVENT_IMX_CLASS          V4L2_EVENT_PRIVATE_START
+#define V4L2_EVENT_IMX_NFB4EOF        (V4L2_EVENT_IMX_CLASS + 1)
+#define V4L2_EVENT_IMX_EOF_TIMEOUT    (V4L2_EVENT_IMX_CLASS + 2)
+#define V4L2_EVENT_IMX_FRAME_INTERVAL (V4L2_EVENT_IMX_CLASS + 3)

Aren't these generic enough to warrant common events? I would think there have to be other capture IP cores that can signal aborted frames or frame timeouts.

Quoted 4 lines

Yes, agreed. A frame capture timeout, or frame interval error, are both generic concepts. At some point it would be great to make the Frame Interval Monitor generally available under v4l2-core. As for the EOF timeout event, I'll look into moving that into a generic V4L2 event.

I'd prefer generic events if possible, but regardless of whether that's possible, the events must be documented in the V4L2 specification.

Signature · 3 lines
Regards,

Laurent Pinchart
author maintainer
#59 lore raw

PATCH v3 16/24 media: Add i.MX media core driver

In reply to Steve Longerbeam

Add the core media driver for i.MX SOC.

---
 Documentation/media/v4l-drivers/imx.rst           | 443 ++++++++++
 drivers/staging/media/Kconfig                     |   2 +
 drivers/staging/media/Makefile                    |   1 +
 drivers/staging/media/imx/Kconfig                 |   8 +
 drivers/staging/media/imx/Makefile                |   6 +
 drivers/staging/media/imx/TODO                    |  22 +
 drivers/staging/media/imx/imx-media-common.c      | 981 ++++++++++++++++++++++
 drivers/staging/media/imx/imx-media-dev.c         | 486 +++++++++++
 drivers/staging/media/imx/imx-media-fim.c         | 471 +++++++++++
 drivers/staging/media/imx/imx-media-internal-sd.c | 457 ++++++++++
 drivers/staging/media/imx/imx-media-of.c          | 289 +++++++
 drivers/staging/media/imx/imx-media.h             | 310 +++++++
 include/media/imx.h                               |  15 +
 include/uapi/linux/v4l2-controls.h                |   4 +
 14 files changed, 3495 insertions(+)
 create mode 100644 Documentation/media/v4l-drivers/imx.rst
 create mode 100644 drivers/staging/media/imx/Kconfig
 create mode 100644 drivers/staging/media/imx/Makefile
 create mode 100644 drivers/staging/media/imx/TODO
 create mode 100644 drivers/staging/media/imx/imx-media-common.c
 create mode 100644 drivers/staging/media/imx/imx-media-dev.c
 create mode 100644 drivers/staging/media/imx/imx-media-fim.c
 create mode 100644 drivers/staging/media/imx/imx-media-internal-sd.c
 create mode 100644 drivers/staging/media/imx/imx-media-of.c
 create mode 100644 drivers/staging/media/imx/imx-media.h
 create mode 100644 include/media/imx.h
Documentation/media/v4l-drivers/imx.rst new file +443 -0
new file mode 100644
index 0000000..87b37b5
@@ -0,0 +1,443 @@
+i.MX Video Capture Driver
+=========================
+
+Introduction
+------------
+
+The Freescale i.MX5/6 contains an Image Processing Unit (IPU), which
+handles the flow of image frames to and from capture devices and
+display devices.
+
+For image capture, the IPU contains the following internal subunits:
+
+- Image DMA Controller (IDMAC)
+- Camera Serial Interface (CSI)
+- Image Converter (IC)
+- Sensor Multi-FIFO Controller (SMFC)
+- Image Rotator (IRT)
+- Video De-Interlace Controller (VDIC)
+
+The IDMAC is the DMA controller for transfer of image frames to and from
+memory. Various dedicated DMA channels exist for both video capture and
+display paths.
+
+The CSI is the frontend capture unit that interfaces directly with
+capture sensors over Parallel, BT.656/1120, and MIPI CSI-2 busses.
+
+The IC handles color-space conversion, resizing, and rotation
+operations. There are three independent "tasks" within the IC that can
+carry out conversions concurrently: pre-processing encoding,
+pre-processing preview, and post-processing.
+
+The SMFC is composed of four independent channels that each can transfer
+captured frames from sensors directly to memory concurrently.
+
+The IRT carries out 90 and 270 degree image rotation operations.
+
+The VDIC handles the conversion of interlaced video to progressive, with
+support for different motion compensation modes (low, medium, and high
+motion). The deinterlaced output frames from the VDIC can be sent to the
+IC pre-process preview task for further conversions.
+
+In addition to the IPU internal subunits, there are also two units
+outside the IPU that are also involved in video capture on i.MX:
+
+- MIPI CSI-2 Receiver for camera sensors with the MIPI CSI-2 bus
+  interface. This is a Synopsys DesignWare core.
+- A video multiplexer for selecting among multiple sensor inputs to
+  send to a CSI.
+
+For more info, refer to the latest versions of the i.MX5/6 reference
+manuals listed under References.
+
+
+Features
+--------
+
+Some of the features of this driver include:
+
+- Many different pipelines can be configured via media controller API,
+  that correspond to the hardware video capture pipelines supported in
+  the i.MX.
+
+- Supports parallel, BT.565, and MIPI CSI-2 interfaces.
+
+- Up to four concurrent sensor acquisitions, by configuring each
+  sensor's pipeline using independent entities. This is currently
+  demonstrated with the SabreSD and SabreLite reference boards with
+  independent OV5642 and MIPI CSI-2 OV5640 sensor modules.
+
+- Scaling, color-space conversion, and image rotation via IC task
+  subdevs.
+
+- Many pixel formats supported (RGB, packed and planar YUV, partial
+  planar YUV).
+
+- The IC pre-process preview subdev supports motion compensated
+  de-interlacing using the VDIC, with three motion compensation modes:
+  low, medium, and high motion. The mode is specified with a custom
+  control. Pipelines are defined that allow sending frames to the
+  preview subdev directly from the CSI or from the SMFC.
+
+- Includes a Frame Interval Monitor (FIM) that can correct vertical sync
+  problems with the ADV718x video decoders. See below for a description
+  of the FIM.
+
+
+Capture Pipelines
+-----------------
+
+The following describe the various use-cases supported by the pipelines.
+
+The links shown do not include the frontend sensor, video mux, or mipi
+csi-2 receiver links. This depends on the type of sensor interface
+(parallel or mipi csi-2). So in all cases, these pipelines begin with:
+
+sensor -> ipu_csi_mux -> ipu_csi -> ...
+
+for parallel sensors, or:
+
+sensor -> imx-mipi-csi2 -> (ipu_csi_mux) -> ipu_csi -> ...
+
+for mipi csi-2 sensors. The imx-mipi-csi2 receiver may need to route
+to the video mux (ipu_csi_mux) before sending to the CSI, depending
+on the mipi csi-2 virtual channel, hence ipu_csi_mux is shown in
+parenthesis.
+
+Unprocessed Video Capture:
+--------------------------
+
+Send frames directly from sensor to camera interface, with no
+conversions:
+
+-> ipu_smfc -> camif
+
+Note the ipu_smfc can do pixel reordering within the same colorspace.
+For example, its sink pad can take UYVY2X8, but its source pad can
+output YUYV2X8.
+
+IC Direct Conversions:
+----------------------
+
+This pipeline uses the preprocess encode entity to route frames directly
+from the CSI to the IC (bypassing the SMFC), to carry out scaling up to
+1024x1024 resolution, CSC, and image rotation:
+
+-> ipu_ic_prpenc -> camif
+
+This can be a useful capture pipeline for heavily loaded memory bus
+traffic environments, since it has minimal IDMAC channel usage.
+
+Post-Processing Conversions:
+----------------------------
+
+This pipeline routes frames from the SMFC to the post-processing
+entity. In addition to CSC and rotation, this entity supports tiling
+which allows scaled output beyond the 1024x1024 limitation of the IC
+(up to 4096x4096 scaling output is supported):
+
+-> ipu_smfc -> ipu_ic_pp -> camif
+
+Motion Compensated De-interlace:
+--------------------------------
+
+This pipeline routes frames from the SMFC to the preprocess preview
+entity to support motion-compensated de-interlacing using the VDIC,
+scaling up to 1024x1024, and CSC:
+
+-> ipu_smfc -> ipu_ic_prpvf -> camif
+
+This pipeline also carries out the same conversions as above, but routes
+frames directly from the CSI to the IC preprocess preview entity for
+minimal memory bandwidth usage (note: this pipeline only works in
+"high motion" mode):
+
+-> ipu_ic_prpvf -> camif
+
+This pipeline takes the motion-compensated de-interlaced frames and
+sends them to the post-processor, to support motion-compensated
+de-interlacing, scaling up to 4096x4096, CSC, and rotation:
+
+-> (ipu_smfc) -> ipu_ic_prpvf -> ipu_ic_pp -> camif
+
+
+Usage Notes
+-----------
+
+Many of the subdevs require information from the active sensor in the
+current pipeline when configuring pad formats. Therefore the media links
+should be established before configuring the media pad formats.
+
+Similarly, the capture v4l2 interface subdev inherits controls from the
+active subdevs in the current pipeline at link-setup time. Therefore the
+capture links should be the last links established in order for capture
+to "see" and inherit all possible controls.
+
+The following are usage notes for Sabre- reference platforms:
+
+
+SabreLite with OV5642 and OV5640
+--------------------------------
+
+This platform requires the OmniVision OV5642 module with a parallel
+camera interface, and the OV5640 module with a MIPI CSI-2
+interface. Both modules are available from Boundary Devices:
+
+https://boundarydevices.com/products/nit6x_5mp
+https://boundarydevices.com/product/nit6x_5mp_mipi
+
+Note that if only one camera module is available, the other sensor
+node can be disabled in the device tree.
+
+The OV5642 module is connected to the parallel bus input on the i.MX
+internal video mux to IPU1 CSI0. It's i2c bus connects to i2c bus 2.
+
+The MIPI CSI-2 OV5640 module is connected to the i.MX internal MIPI CSI-2
+receiver, and the four virtual channel outputs from the receiver are
+routed as follows: vc0 to the IPU1 CSI0 mux, vc1 directly to IPU1 CSI1,
+vc2 directly to IPU2 CSI0, and vc3 to the IPU2 CSI1 mux. The OV5640 is
+also connected to i2c bus 2 on the SabreLite, therefore the OV5642 and
+OV5640 must not share the same i2c slave address.
+
+The following basic example configures unprocessed video capture
+pipelines for both sensors. The OV5642 is routed to camif0
+(usually /dev/video0), and the OV5640 (transmitting on mipi csi-2
+virtual channel 1) is routed to camif1 (usually /dev/video1). Both
+sensors are configured to output 640x480, UYVY (not shown: all pad
+field types should be set to "NONE"):
+
+.. code-block:: none
+
+   # Setup links for OV5642
+   media-ctl -l '"ov5642 1-0042":0 -> "ipu1_csi0_mux":1[1]'
+   media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
+   media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
+   media-ctl -l '"ipu1_smfc0":1 -> "camif0":0[1]'
+   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
+   # Setup links for OV5640
+   media-ctl -l '"ov5640_mipi 1-0040":0 -> "imx-mipi-csi2":0[1]'
+   media-ctl -l '"imx-mipi-csi2":2 -> "ipu1_csi1":0[1]'
+   media-ctl -l '"ipu1_csi1":1 -> "ipu1_smfc1":0[1]'
+   media-ctl -l '"ipu1_smfc1":1 -> "camif1":0[1]'
+   media-ctl -l '"camif1":1 -> "camif1 devnode":0[1]'
+   # Configure pads for OV5642 pipeline
+   media-ctl -V "\"ov5642 1-0042\":0 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_csi0\":0 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_csi0\":1 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_smfc0\":0 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480]"
+   # Configure pads for OV5640 pipeline
+   media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"ipu1_smfc1\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"ipu1_smfc1\":1 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"camif1\":1 [fmt:UYVY2X8/640x480]"
+
+Streaming can then begin independently on device nodes /dev/video0
+and /dev/video1.
+
+SabreAuto with ADV7180 decoder
+------------------------------
+
+On the SabreAuto, an on-board ADV7180 SD decoder is connected to the
+parallel bus input on the internal video mux to IPU1 CSI0.
+
+The following example configures a pipeline to capture from the ADV7180
+video decoder, assuming NTSC 720x480 input signals, with Motion
+Compensated de-interlacing (not shown: all pad field types should be set
+as indicated). $outputfmt can be any format supported by the
+ipu1_ic_prpvf entity at its output pad:
+
+.. code-block:: none
+
+   # Setup links
+   media-ctl -l '"adv7180 3-0021":0 -> "ipu1_csi0_mux":1[1]'
+   media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
+   media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
+   media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]'
+   media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]'
+   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
+   # Configure pads
+   # pad field types for below pads must be an interlaced type
+   # such as "ALTERNATE"
+   media-ctl -V "\"adv7180 3-0021\":0 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_csi0\":0 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_csi0\":1 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_smfc0\":0 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_ic_prpvf\":0 [fmt:UYVY2X8/720x480]"
+   # pad field types for below pads must be "NONE"
+   media-ctl -V "\"ipu1_ic_prpvf\":1 [fmt:$outputfmt]"
+   media-ctl -V "\"camif0\":0 [fmt:$outputfmt]"
+   media-ctl -V "\"camif0\":1 [fmt:$outputfmt]"
+
+Streaming can then begin on /dev/video0.
+
+This platform accepts Composite Video analog inputs to the ADV7180 on
+Ain1 (connector J42) and Ain3 (connector J43).
+
+To switch to Ain1:
+
+.. code-block:: none
+
+   # v4l2-ctl -i0
+
+To switch to Ain3:
+
+.. code-block:: none
+
+   # v4l2-ctl -i1
+
+
+Frame Interval Monitor
+----------------------
+
+The adv718x decoders can occasionally send corrupt fields during
+NTSC/PAL signal re-sync (too little or too many video lines). When
+this happens, the IPU triggers a mechanism to re-establish vertical
+sync by adding 1 dummy line every frame, which causes a rolling effect
+from image to image, and can last a long time before a stable image is
+recovered. Or sometimes the mechanism doesn't work at all, causing a
+permanent split image (one frame contains lines from two consecutive
+captured images).
+
+From experiment it was found that during image rolling, the frame
+intervals (elapsed time between two EOF's) drop below the nominal
+value for the current standard, by about one frame time (60 usec),
+and remain at that value until rolling stops.
+
+While the reason for this observation isn't known (the IPU dummy
+line mechanism should show an increase in the intervals by 1 line
+time every frame, not a fixed value), we can use it to detect the
+corrupt fields using a frame interval monitor. If the FIM detects a
+bad frame interval, a subdev event is sent. In response, userland can
+issue a streaming restart to correct the rolling/split image.
+
+The FIM is implemented in the imx-csi entity, and the entities that have
+direct connections to the CSI call into the FIM to monitor the frame
+intervals: ipu_smfc, ipu_ic_prpenc, and ipu_prpvf (when configured with
+a direct link from ipu_csi). Userland can register with the FIM event
+notifications on the imx-csi subdev device node
+(V4L2_EVENT_IMX_FRAME_INTERVAL).
+
+The imx-csi entity includes custom controls to tweak some dials for FIM.
+If one of these controls is changed during streaming, the FIM will be
+reset and will continue at the new settings.
+
+- V4L2_CID_IMX_FIM_ENABLE
+
+Enable/disable the FIM.
+
+- V4L2_CID_IMX_FIM_NUM
+
+How many frame interval errors to average before comparing against the
+nominal frame interval reported by the sensor. This can reduce noise
+from interrupt latency.
+
+- V4L2_CID_IMX_FIM_TOLERANCE_MIN
+
+If the averaged intervals fall outside nominal by this amount, in
+microseconds, streaming will be restarted.
+
+- V4L2_CID_IMX_FIM_TOLERANCE_MAX
+
+If any interval errors are higher than this value, those error samples
+are discarded and do not enter into the average. This can be used to
+discard really high interval errors that might be due to very high
+system load, causing excessive interrupt latencies.
+
+- V4L2_CID_IMX_FIM_NUM_SKIP
+
+How many frames to skip after a FIM reset or stream restart before
+FIM begins to average intervals. It has been found that there can
+be a few bad frame intervals after stream restart which are not
+attributed to adv718x sending a corrupt field, so this is used to
+skip those frames to prevent unnecessary restarts.
+
+
+SabreSD with MIPI CSI-2 OV5640
+------------------------------
+
+Similarly to SabreLite, the SabreSD supports a parallel interface
+OV5642 module on IPU1 CSI0, and a MIPI CSI-2 OV5640 module. The OV5642
+connects to i2c bus 1 and the OV5640 to i2c bus 2.
+
+The device tree for SabreSD includes OF graphs for both the parallel
+OV5642 and the MIPI CSI-2 OV5640, but as of this writing only the MIPI
+CSI-2 OV5640 has been tested, so the OV5642 node is currently disabled.
+The OV5640 module connects to MIPI connector J5 (sorry I don't have the
+compatible module part number or URL).
+
+The following example configures a post-processing pipeline to capture
+from the OV5640 (not shown: all pad field types should be set to
+"NONE"). $sensorfmt can be any format supported by the
+OV5640. $outputfmt can be any format supported by the ipu1_ic_pp1
+entity at its output pad:
+
+
+.. code-block:: none
+
+   # Setup links
+   media-ctl -l '"ov5640_mipi 1-003c":0 -> "imx-mipi-csi2":0[1]'
+   media-ctl -l '"imx-mipi-csi2":2 -> "ipu1_csi1":0[1]'
+   media-ctl -l '"ipu1_csi1":1 -> "ipu1_smfc1":0[1]'
+   media-ctl -l '"ipu1_smfc1":1 -> "ipu1_ic_pp1":0[1]'
+   media-ctl -l '"ipu1_ic_pp1":1 -> "camif0":0[1]'
+   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
+   # Configure pads
+   media-ctl -V "\"ov5640_mipi 1-003c\":0 [fmt:$sensorfmt]"
+   media-ctl -V "\"imx-mipi-csi2\":0 [fmt:$sensorfmt]"
+   media-ctl -V "\"imx-mipi-csi2\":2 [fmt:$sensorfmt]"
+   media-ctl -V "\"ipu1_csi1\":0 [fmt:$sensorfmt]"
+   media-ctl -V "\"ipu1_csi1\":1 [fmt:$sensorfmt]"
+   media-ctl -V "\"ipu1_smfc1\":0 [fmt:$sensorfmt]"
+   media-ctl -V "\"ipu1_smfc1\":1 [fmt:$sensorfmt]"
+   media-ctl -V "\"ipu1_ic_pp1\":0 [fmt:$sensorfmt]"
+   media-ctl -V "\"ipu1_ic_pp1\":1 [fmt:$outputfmt]"
+   media-ctl -V "\"camif0\":0 [fmt:$outputfmt]"
+   media-ctl -V "\"camif0\":1 [fmt:$outputfmt]"
+
+Streaming can then begin on /dev/video0.
+
+
+
+Known Issues
+------------
+
+1. When using 90 or 270 degree rotation control at capture resolutions
+   near the IC resizer limit of 1024x1024, and combined with planar
+   pixel formats (YUV420, YUV422p), frame capture will often fail with
+   no end-of-frame interrupts from the IDMAC channel. To work around
+   this, use lower resolution and/or packed formats (YUYV, RGB3, etc.)
+   when 90 or 270 rotations are needed.
+
+
+File list
+---------
+
+drivers/staging/media/imx/
+include/media/imx.h
+include/uapi/media/imx.h
+
+References
+----------
+
+[1] "i.MX 6Dual/6Quad Applications Processor Reference Manual"
+[2] "i.MX 6Solo/6DualLite Applications Processor Reference Manual"
+
+
+Author
+------
+Steve Longerbeam <[email protected]>
+
+Copyright (C) 2012-2016 Mentor Graphics Inc.
drivers/staging/media/Kconfig +2 -0
index ffb8fa7..05b55a8 100644
@@ -25,6 +25,8 @@ source "drivers/staging/media/cxd2099/Kconfig"
 
 source "drivers/staging/media/davinci_vpfe/Kconfig"
 
+source "drivers/staging/media/imx/Kconfig"
+
 source "drivers/staging/media/omap4iss/Kconfig"
 
 source "drivers/staging/media/s5p-cec/Kconfig"
drivers/staging/media/Makefile +1 -0
index a28e82c..6f50ddd 100644
@@ -1,6 +1,7 @@
 obj-$(CONFIG_I2C_BCM2048)	+= bcm2048/
 obj-$(CONFIG_VIDEO_SAMSUNG_S5P_CEC) += s5p-cec/
 obj-$(CONFIG_DVB_CXD2099)	+= cxd2099/
+obj-$(CONFIG_VIDEO_IMX_MEDIA)	+= imx/
 obj-$(CONFIG_LIRC_STAGING)	+= lirc/
 obj-$(CONFIG_VIDEO_DM365_VPFE)	+= davinci_vpfe/
 obj-$(CONFIG_VIDEO_OMAP4)	+= omap4iss/
drivers/staging/media/imx/Kconfig new file +8 -0
new file mode 100644
index 0000000..bfde58d
@@ -0,0 +1,8 @@
+config VIDEO_IMX_MEDIA
+	tristate "i.MX5/6 V4L2 media core driver"
+	depends on MEDIA_CONTROLLER && VIDEO_V4L2 && ARCH_MXC && IMX_IPUV3_CORE
+	default y
+	---help---
+	  Say yes here to enable support for video4linux media controller
+	  driver for the i.MX5/6 SOC.
+
drivers/staging/media/imx/Makefile new file +6 -0
new file mode 100644
index 0000000..ef9f11b
@@ -0,0 +1,6 @@
+imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \
+	imx-media-of.o
+
+obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
+obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
+
drivers/staging/media/imx/TODO new file +22 -0
new file mode 100644
index 0000000..1f42381
@@ -0,0 +1,22 @@
+
+- v4l2-compliance
+
+- imx-csi subdev is not being autoloaded as a kernel module, probably
+  because ipu_add_client_devices() does not register the IPU client
+  platform devices, but only allocates those devices.
+
+- Verify driver remove paths.
+
+- Currently registering with notifications from subdevs are only
+  available through the subdev device nodes and not through the main
+  capture device node. Need to come up with a way to find the camif in
+  the current pipeline that owns the subdev that sent the notify.
+
+- Need to decide whether a mem2mem device should be incorporated into
+  the media graph, or whether it should be a separate device that does
+  not link with any other entities.
+
+- Combine, clean up, and move the ov5640/ov5642 subdevs to
+  drivers/media/i2c. Once that is done the binding docs for ov564x
+  can be created under Documentation/devicetree/bindings/media/i2c.
+
drivers/staging/media/imx/imx-media-common.c new file +981 -0

981 lines not on this page · raw message

drivers/staging/media/imx/imx-media-dev.c new file +486 -0

486 lines not on this page · raw message

drivers/staging/media/imx/imx-media-fim.c new file +471 -0

471 lines not on this page · raw message

drivers/staging/media/imx/imx-media-internal-sd.c new file +457 -0

457 lines not on this page · raw message

drivers/staging/media/imx/imx-media-of.c new file +289 -0

111 lines not on this page · raw message

drivers/staging/media/imx/imx-media.h new file +310 -0

310 lines not on this page · raw message

include/media/imx.h new file +15 -0
new file mode 100644
index 0000000..5025a72

Lines of this file are not shown. View the raw message.

include/uapi/linux/v4l2-controls.h +4 -0
index 0d2e1e0..6c29f42 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.7.4

Part of the diff is not shown here. View the raw message.

maintainer
#60 lore raw

Re: [PATCH v3 16/24] media: Add i.MX media core driver

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:

Quoted 45 lines

Add the core media driver for i.MX SOC.

---
 Documentation/media/v4l-drivers/imx.rst           | 443 ++++++++++
 drivers/staging/media/Kconfig                     |   2 +
 drivers/staging/media/Makefile                    |   1 +
 drivers/staging/media/imx/Kconfig                 |   8 +
 drivers/staging/media/imx/Makefile                |   6 +
 drivers/staging/media/imx/TODO                    |  22 +
 drivers/staging/media/imx/imx-media-common.c      | 981 ++++++++++++++++++++++
 drivers/staging/media/imx/imx-media-dev.c         | 486 +++++++++++
 drivers/staging/media/imx/imx-media-fim.c         | 471 +++++++++++
 drivers/staging/media/imx/imx-media-internal-sd.c | 457 ++++++++++
 drivers/staging/media/imx/imx-media-of.c          | 289 +++++++
 drivers/staging/media/imx/imx-media.h             | 310 +++++++
 include/media/imx.h                               |  15 +
 include/uapi/linux/v4l2-controls.h                |   4 +
 14 files changed, 3495 insertions(+)
 create mode 100644 Documentation/media/v4l-drivers/imx.rst
 create mode 100644 drivers/staging/media/imx/Kconfig
 create mode 100644 drivers/staging/media/imx/Makefile
 create mode 100644 drivers/staging/media/imx/TODO
 create mode 100644 drivers/staging/media/imx/imx-media-common.c
 create mode 100644 drivers/staging/media/imx/imx-media-dev.c
 create mode 100644 drivers/staging/media/imx/imx-media-fim.c
 create mode 100644 drivers/staging/media/imx/imx-media-internal-sd.c
 create mode 100644 drivers/staging/media/imx/imx-media-of.c
 create mode 100644 drivers/staging/media/imx/imx-media.h
 create mode 100644 include/media/imx.h
Documentation/media/v4l-drivers/imx.rst new file +8 -0
new file mode 100644
index 0000000..87b37b5
@@ -0,0 +1,443 @@
+i.MX Video Capture Driver
+=========================
+
+Introduction
+------------
+
+The Freescale i.MX5/6 contains an Image Processing Unit (IPU), which
+handles the flow of image frames to and from capture devices and

Documentation/media/v4l-drivers/imx.rst line 18

Quoted 10 lines
Documentation/media/v4l-drivers/imx.rst new file +10 -0
+display devices.
+
+For image capture, the IPU contains the following internal subunits:
+
+- Image DMA Controller (IDMAC)
+- Camera Serial Interface (CSI)
+- Image Converter (IC)
+- Sensor Multi-FIFO Controller (SMFC)
+- Image Rotator (IRT)
+- Video De-Interlace Controller (VDIC)

Nitpick: Video De-Interlacing or Combining Block (VDIC)

Documentation/media/v4l-drivers/imx.rst

Quoted 10 lines
Documentation/media/v4l-drivers/imx.rst new file +10 -0
+
+The IDMAC is the DMA controller for transfer of image frames to and from
+memory. Various dedicated DMA channels exist for both video capture and
+display paths.
+
+The CSI is the frontend capture unit that interfaces directly with
+capture sensors over Parallel, BT.656/1120, and MIPI CSI-2 busses.
+
+The IC handles color-space conversion, resizing, and rotation
+operations. 

And horizontal flipping.

Documentation/media/v4l-drivers/imx.rst line 30

Quoted 3 lines

There are three independent "tasks" within the IC that can

Documentation/media/v4l-drivers/imx.rst new file +2 -0
+carry out conversions concurrently: pre-processing encoding,
+pre-processing preview, and post-processing.

s/preview/viewfinder/ seems to be the commonly used name.

This paragraph could mention that a single hardware unit is used transparently time multiplexed by the three tasks at different granularity for the downsizing, main processing, and rotation sections. The downscale unit switches between tasks at 8-pixel burst granularity, the main processing unit at line granularity. The rotation units switch only at frame granularity.

Documentation/media/v4l-drivers/imx.rst line 35

Quoted 4 lines
Documentation/media/v4l-drivers/imx.rst new file +4 -0
+The SMFC is composed of four independent channels that each can transfer
+captured frames from sensors directly to memory concurrently.
+
+The IRT carries out 90 and 270 degree image rotation operations.

... on 8x8 pixel blocks, supported by the IDMAC which handles block transfers, block reordering, and vertical flipping.

Documentation/media/v4l-drivers/imx.rst line 48

Quoted 12 lines
Documentation/media/v4l-drivers/imx.rst new file +12 -0
+The VDIC handles the conversion of interlaced video to progressive, with
+support for different motion compensation modes (low, medium, and high
+motion). The deinterlaced output frames from the VDIC can be sent to the
+IC pre-process preview task for further conversions.
+
+In addition to the IPU internal subunits, there are also two units
+outside the IPU that are also involved in video capture on i.MX:
+
+- MIPI CSI-2 Receiver for camera sensors with the MIPI CSI-2 bus
+  interface. This is a Synopsys DesignWare core.
+- A video multiplexer for selecting among multiple sensor inputs to
+  send to a CSI.

Two of them, actually.

Quoted 25 lines
Documentation/media/v4l-drivers/imx.rst new file +25 -0
+For more info, refer to the latest versions of the i.MX5/6 reference
+manuals listed under References.
+
+
+Features
+--------
+
+Some of the features of this driver include:
+
+- Many different pipelines can be configured via media controller API,
+  that correspond to the hardware video capture pipelines supported in
+  the i.MX.
+
+- Supports parallel, BT.565, and MIPI CSI-2 interfaces.
+
+- Up to four concurrent sensor acquisitions, by configuring each
+  sensor's pipeline using independent entities. This is currently
+  demonstrated with the SabreSD and SabreLite reference boards with
+  independent OV5642 and MIPI CSI-2 OV5640 sensor modules.
+
+- Scaling, color-space conversion, and image rotation via IC task
+  subdevs.
+
+- Many pixel formats supported (RGB, packed and planar YUV, partial
+  planar YUV).

Documentation/media/v4l-drivers/imx.rst line 84

Quoted 10 lines
Documentation/media/v4l-drivers/imx.rst new file +10 -0
+
+- The IC pre-process preview subdev supports motion compensated
+  de-interlacing using the VDIC, with three motion compensation modes:
+  low, medium, and high motion. The mode is specified with a custom
+  control. Pipelines are defined that allow sending frames to the
+  preview subdev directly from the CSI or from the SMFC.
+
+- Includes a Frame Interval Monitor (FIM) that can correct vertical sync
+  problems with the ADV718x video decoders. See below for a description
+  of the FIM.

Could this also be used to calculate more precise capture timestamps?

Quoted 17 lines
Documentation/media/v4l-drivers/imx.rst new file +17 -0
+Capture Pipelines
+-----------------
+
+The following describe the various use-cases supported by the pipelines.
+
+The links shown do not include the frontend sensor, video mux, or mipi
+csi-2 receiver links. This depends on the type of sensor interface
+(parallel or mipi csi-2). So in all cases, these pipelines begin with:
+
+sensor -> ipu_csi_mux -> ipu_csi -> ...
+
+for parallel sensors, or:
+
+sensor -> imx-mipi-csi2 -> (ipu_csi_mux) -> ipu_csi -> ...
+
+for mipi csi-2 sensors. The imx-mipi-csi2 receiver may need to route
+to the video mux (ipu_csi_mux) before sending to the CSI, depending

Documentation/media/v4l-drivers/imx.rst line 113

Quoted 10 lines
Documentation/media/v4l-drivers/imx.rst new file +10 -0
+on the mipi csi-2 virtual channel, hence ipu_csi_mux is shown in
+parenthesis.
+
+Unprocessed Video Capture:
+--------------------------
+
+Send frames directly from sensor to camera interface, with no
+conversions:
+
+-> ipu_smfc -> camif

I'd call this capture interface, this is not just for cameras. Or maybe idmac if you want to mirror hardware names?

Quoted 1 line

+Note the ipu_smfc can do pixel reordering within the same colorspace.

That isn't a feature of the SMFC, but of the IDMAC (FCW & FCR).

Documentation/media/v4l-drivers/imx.rst line 117

Quoted 2 lines
Documentation/media/v4l-drivers/imx.rst new file +2 -0
+For example, its sink pad can take UYVY2X8, but its source pad can
+output YUYV2X8.

I don't think this is correct. Re-reading "37.4.3.7 Packing to memory" in the CSI chapter, for 8-bit per component data, the internal format between CSI, SMFC, and IDMAC is always some 32-bit RGBx/YUVx variant (or "bayer/generic data"). In either case, the internal format does not change along the way.

Documentation/media/v4l-drivers/imx.rst line 129

Quoted 11 lines
Documentation/media/v4l-drivers/imx.rst new file +11 -0
+IC Direct Conversions:
+----------------------
+
+This pipeline uses the preprocess encode entity to route frames directly
+from the CSI to the IC (bypassing the SMFC), to carry out scaling up to
+1024x1024 resolution, CSC, and image rotation:
+
+-> ipu_ic_prpenc -> camif
+
+This can be a useful capture pipeline for heavily loaded memory bus
+traffic environments, since it has minimal IDMAC channel usage.

Note that if rotation is enabled, transfers between IC processing and rotation still have to go through memory once.

Documentation/media/v4l-drivers/imx.rst

Quoted 5 lines
Documentation/media/v4l-drivers/imx.rst new file +5 -0
+Post-Processing Conversions:
+----------------------------
+
+This pipeline routes frames from the SMFC to the post-processing
+entity.

No, frames written by the CSI -> SMFC -> IDMAC path are read back into the post-processing entity.

Quoted 4 lines

In addition to CSC and rotation, this entity supports tiling

Documentation/media/v4l-drivers/imx.rst new file +3 -0
+which allows scaled output beyond the 1024x1024 limitation of the IC
+(up to 4096x4096 scaling output is supported):
+

Documentation/media/v4l-drivers/imx.rst line 148

Quoted 10 lines
Documentation/media/v4l-drivers/imx.rst new file +10 -0
+-> ipu_smfc -> ipu_ic_pp -> camif
+
+Motion Compensated De-interlace:
+--------------------------------
+
+This pipeline routes frames from the SMFC to the preprocess preview
+entity to support motion-compensated de-interlacing using the VDIC,
+scaling up to 1024x1024, and CSC:
+
+-> ipu_smfc -> ipu_ic_prpvf -> camif

Same as above.

Quoted 16 lines
Documentation/media/v4l-drivers/imx.rst new file +16 -0
+This pipeline also carries out the same conversions as above, but routes
+frames directly from the CSI to the IC preprocess preview entity for
+minimal memory bandwidth usage (note: this pipeline only works in
+"high motion" mode):
+
+-> ipu_ic_prpvf -> camif
+
+This pipeline takes the motion-compensated de-interlaced frames and
+sends them to the post-processor, to support motion-compensated
+de-interlacing, scaling up to 4096x4096, CSC, and rotation:
+
+-> (ipu_smfc) -> ipu_ic_prpvf -> ipu_ic_pp -> camif
+
+
+Usage Notes
+-----------

[...]

Quoted 42 lines
Documentation/media/v4l-drivers/imx.rst new file +42 -0
+SabreLite with OV5642 and OV5640
+--------------------------------
+
+This platform requires the OmniVision OV5642 module with a parallel
+camera interface, and the OV5640 module with a MIPI CSI-2
+interface. Both modules are available from Boundary Devices:
+
+https://boundarydevices.com/products/nit6x_5mp
+https://boundarydevices.com/product/nit6x_5mp_mipi
+
+Note that if only one camera module is available, the other sensor
+node can be disabled in the device tree.
+
+The OV5642 module is connected to the parallel bus input on the i.MX
+internal video mux to IPU1 CSI0. It's i2c bus connects to i2c bus 2.
+
+The MIPI CSI-2 OV5640 module is connected to the i.MX internal MIPI CSI-2
+receiver, and the four virtual channel outputs from the receiver are
+routed as follows: vc0 to the IPU1 CSI0 mux, vc1 directly to IPU1 CSI1,
+vc2 directly to IPU2 CSI0, and vc3 to the IPU2 CSI1 mux. The OV5640 is
+also connected to i2c bus 2 on the SabreLite, therefore the OV5642 and
+OV5640 must not share the same i2c slave address.
+
+The following basic example configures unprocessed video capture
+pipelines for both sensors. The OV5642 is routed to camif0
+(usually /dev/video0), and the OV5640 (transmitting on mipi csi-2
+virtual channel 1) is routed to camif1 (usually /dev/video1). Both
+sensors are configured to output 640x480, UYVY (not shown: all pad
+field types should be set to "NONE"):
+
+.. code-block:: none
+
+   # Setup links for OV5642
+   media-ctl -l '"ov5642 1-0042":0 -> "ipu1_csi0_mux":1[1]'
+   media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
+   media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
+   media-ctl -l '"ipu1_smfc0":1 -> "camif0":0[1]'
+   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
+   # Setup links for OV5640
+   media-ctl -l '"ov5640_mipi 1-0040":0 -> "imx-mipi-csi2":0[1]'
+   media-ctl -l '"imx-mipi-csi2":2 -> "ipu1_csi1":0[1]'
+   media-ctl -l '"ipu1_csi1":1 -> "ipu1_smfc1":0[1]'

Documentation/media/v4l-drivers/imx.rst line 230

Quoted 10 lines
Documentation/media/v4l-drivers/imx.rst new file +10 -0
+   media-ctl -l '"ipu1_smfc1":1 -> "camif1":0[1]'
+   media-ctl -l '"camif1":1 -> "camif1 devnode":0[1]'
+   # Configure pads for OV5642 pipeline
+   media-ctl -V "\"ov5642 1-0042\":0 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_csi0\":0 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_csi0\":1 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_smfc0\":0 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/640x480]"

I think the smfc entities should be dropped.

Quoted 8 lines
Documentation/media/v4l-drivers/imx.rst new file +8 -0
+   media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480]"
+   # Configure pads for OV5640 pipeline
+   media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480]"

[...]

Quoted 1 line

+ media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"

I agree this looks very intuitive, but technically correct for the csi1:1 and camif1:0 pads would be a 32-bit YUV format. (MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).

I think it would be better to use the correct format as that will allow to chose the regular vs. companded packings in the future for formats with more than 8 bits per component.

Quoted 19 lines
Documentation/media/v4l-drivers/imx.rst new file +19 -0
+   media-ctl -V "\"camif1\":1 [fmt:UYVY2X8/640x480]"
+
+Streaming can then begin independently on device nodes /dev/video0
+and /dev/video1.
+
+SabreAuto with ADV7180 decoder
+------------------------------
+
+On the SabreAuto, an on-board ADV7180 SD decoder is connected to the
+parallel bus input on the internal video mux to IPU1 CSI0.
+
+The following example configures a pipeline to capture from the ADV7180
+video decoder, assuming NTSC 720x480 input signals, with Motion
+Compensated de-interlacing (not shown: all pad field types should be set
+as indicated). $outputfmt can be any format supported by the
+ipu1_ic_prpvf entity at its output pad:
+
+.. code-block:: none
+

Documentation/media/v4l-drivers/imx.rst line 270

Quoted 10 lines
Documentation/media/v4l-drivers/imx.rst new file +10 -0
+   # Setup links
+   media-ctl -l '"adv7180 3-0021":0 -> "ipu1_csi0_mux":1[1]'
+   media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
+   media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
+   media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]'
+   media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]'
+   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
+   # Configure pads
+   # pad field types for below pads must be an interlaced type
+   # such as "ALTERNATE"

I think alternate should only extend as far as the CSI, since the CSI can only capture NTSC/PAL fields in a fixed order.

Quoted 4 lines
Documentation/media/v4l-drivers/imx.rst new file +2 -0
+   media-ctl -V "\"adv7180 3-0021\":0 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480]"

From here the interlaced field type should be sequential in the correct

order depending on NTSC/PAL.

Quoted 30 lines
Documentation/media/v4l-drivers/imx.rst new file +30 -0
+   media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_csi0\":0 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_csi0\":1 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_smfc0\":0 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_ic_prpvf\":0 [fmt:UYVY2X8/720x480]"
+   # pad field types for below pads must be "NONE"
+   media-ctl -V "\"ipu1_ic_prpvf\":1 [fmt:$outputfmt]"
+   media-ctl -V "\"camif0\":0 [fmt:$outputfmt]"
+   media-ctl -V "\"camif0\":1 [fmt:$outputfmt]"
+
+Streaming can then begin on /dev/video0.
+
+This platform accepts Composite Video analog inputs to the ADV7180 on
+Ain1 (connector J42) and Ain3 (connector J43).
+
+To switch to Ain1:
+
+.. code-block:: none
+
+   # v4l2-ctl -i0
+
+To switch to Ain3:
+
+.. code-block:: none
+
+   # v4l2-ctl -i1
+
+
+Frame Interval Monitor

Documentation/media/v4l-drivers/imx.rst line 312

Quoted 10 lines
Documentation/media/v4l-drivers/imx.rst new file +10 -0
+----------------------
+
+The adv718x decoders can occasionally send corrupt fields during
+NTSC/PAL signal re-sync (too little or too many video lines). When
+this happens, the IPU triggers a mechanism to re-establish vertical
+sync by adding 1 dummy line every frame, which causes a rolling effect
+from image to image, and can last a long time before a stable image is
+recovered. Or sometimes the mechanism doesn't work at all, causing a
+permanent split image (one frame contains lines from two consecutive
+captured images).

Is it only SabreAuto on which the FIM mechanism can be used due to the pad routing?

[...]

Quoted 11 lines
Documentation/media/v4l-drivers/imx.rst new file +11 -0
+/*
+ * DMA buffer ring handling
+ */
+struct imx_media_dma_buf_ring {
+	struct imx_media_dev *imxmd;
+
+	/* the ring */
+	struct imx_media_dma_buf buf[IMX_MEDIA_MAX_RING_BUFS];
+	/* the scratch buffer for underruns */
+	struct imx_media_dma_buf scratch;
+

Documentation/media/v4l-drivers/imx.rst

Quoted 10 lines
Documentation/media/v4l-drivers/imx.rst new file +10 -0
+	/* buffer generator */
+	struct media_entity *src;
+	/* buffer receiver */
+	struct media_entity *sink;
+
+	spinlock_t lock;
+
+	int num_bufs;
+	unsigned long last_seq;
+};

I don't think this belongs in the capture driver at all. Memory-to-memory transfers should be handled at the videobuf2 level.

[...]

Quoted 7 lines
Documentation/media/v4l-drivers/imx.rst new file +7 -0
+static struct imx_media_dma_buf *
+__dma_buf_queue(struct imx_media_dma_buf_ring *ring, int index)
+{
+	struct imx_media_dma_buf *buf;
+
+	if (index >= ring->num_bufs)
+		return ERR_PTR(-EINVAL);

Documentation/media/v4l-drivers/imx.rst

Quoted 10 lines
Documentation/media/v4l-drivers/imx.rst new file +10 -0
+
+	buf = &ring->buf[index];
+	if (WARN_ON(buf->state != IMX_MEDIA_BUF_STATUS_PREPARED))
+		return ERR_PTR(-EINVAL);
+
+	buf->state = IMX_MEDIA_BUF_STATUS_QUEUED;
+	buf->seq = ring->last_seq++;
+
+	return buf;
+}

Is this a whole software buffer queue implementation? I thought the whole point of putting the custom mem2mem framework into the capture driver was to use the hardware FSU channel linking?

Quoted 66 lines
Documentation/media/v4l-drivers/imx.rst new file +66 -0
+int imx_media_dma_buf_queue(struct imx_media_dma_buf_ring *ring, int index)
+{
+	struct imx_media_dma_buf *buf;
+	unsigned long flags;
+
+	spin_lock_irqsave(&ring->lock, flags);
+	buf = __dma_buf_queue(ring, index);
+	spin_unlock_irqrestore(&ring->lock, flags);
+
+	if (IS_ERR(buf))
+		return PTR_ERR(buf);
+
+	dev_dbg(ring->imxmd->dev, "buf%d [%s -> %s] queued\n",
+		index, ring->src->name, ring->sink->name);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(imx_media_dma_buf_queue);
+
+int imx_media_dma_buf_queue_from_vb(struct imx_media_dma_buf_ring *ring,
+				    struct vb2_buffer *vb)
+{
+	struct imx_media_dma_buf *buf;
+	unsigned long flags;
+	dma_addr_t phys;
+	void *virt;
+
+	if (vb->index >= ring->num_bufs)
+		return -EINVAL;
+
+	virt = vb2_plane_vaddr(vb, 0);
+	phys = vb2_dma_contig_plane_dma_addr(vb, 0);
+
+	spin_lock_irqsave(&ring->lock, flags);
+	buf = __dma_buf_queue(ring, vb->index);
+	if (IS_ERR(buf))
+		goto err_unlock;
+
+	buf->virt = virt;
+	buf->phys = phys;
+	buf->vb = vb;
+	spin_unlock_irqrestore(&ring->lock, flags);
+
+	dev_dbg(ring->imxmd->dev, "buf%d [%s -> %s] queued from vb\n",
+		buf->index, ring->src->name, ring->sink->name);
+
+	return 0;
+err_unlock:
+	spin_unlock_irqrestore(&ring->lock, flags);
+	return PTR_ERR(buf);
+}
+EXPORT_SYMBOL_GPL(imx_media_dma_buf_queue_from_vb);
+
+void imx_media_dma_buf_done(struct imx_media_dma_buf *buf,
+			    enum imx_media_dma_buf_status status)
+{
+	struct imx_media_dma_buf_ring *ring = buf->ring;
+	unsigned long flags;
+
+	spin_lock_irqsave(&ring->lock, flags);
+	WARN_ON(buf->state != IMX_MEDIA_BUF_STATUS_ACTIVE);
+	buf->state = buf->status = status;
+	spin_unlock_irqrestore(&ring->lock, flags);
+
+	if (buf == &ring->scratch)
+		dev_dbg(ring->imxmd->dev, "buf-scratch [%s -> %s] done\n",

Documentation/media/v4l-drivers/imx.rst

Quoted 10 lines
Documentation/media/v4l-drivers/imx.rst new file +10 -0
+			ring->src->name, ring->sink->name);
+	else
+		dev_dbg(ring->imxmd->dev, "buf%d [%s -> %s] done\n",
+			buf->index, ring->src->name, ring->sink->name);
+
+	/* if the sink is a subdev, inform it that new buffers are available */
+	if (is_media_entity_v4l2_subdev(ring->sink)) {
+		struct v4l2_subdev *sd =
+			media_entity_to_v4l2_subdev(ring->sink);
+		v4l2_subdev_call(sd, core, ioctl, IMX_MEDIA_NEW_DMA_BUF, NULL);

What is the purpose of this if the sink should be triggered by the FSU?

[...]

Quoted 39 lines
Documentation/media/v4l-drivers/imx.rst new file +39 -0
+/*
+ * The subdevs have to be powered on/off, and streaming
+ * enabled/disabled, in a specific sequence.
+ */
+static const u32 stream_on_seq[] = {
+	IMX_MEDIA_GRP_ID_IC_PP,
+	IMX_MEDIA_GRP_ID_IC_PRPVF,
+	IMX_MEDIA_GRP_ID_IC_PRPENC,
+	IMX_MEDIA_GRP_ID_SMFC,
+	IMX_MEDIA_GRP_ID_SENSOR,
+	IMX_MEDIA_GRP_ID_CSI2,
+	IMX_MEDIA_GRP_ID_VIDMUX,
+	IMX_MEDIA_GRP_ID_CSI,
+};
+
+static const u32 stream_off_seq[] = {
+	IMX_MEDIA_GRP_ID_IC_PP,
+	IMX_MEDIA_GRP_ID_IC_PRPVF,
+	IMX_MEDIA_GRP_ID_IC_PRPENC,
+	IMX_MEDIA_GRP_ID_SMFC,
+	IMX_MEDIA_GRP_ID_CSI,
+	IMX_MEDIA_GRP_ID_VIDMUX,
+	IMX_MEDIA_GRP_ID_CSI2,
+	IMX_MEDIA_GRP_ID_SENSOR,
+};
+
+#define NUM_STREAM_ENTITIES ARRAY_SIZE(stream_on_seq)
+
+static const u32 power_on_seq[] = {
+	IMX_MEDIA_GRP_ID_CSI2,
+	IMX_MEDIA_GRP_ID_SENSOR,
+	IMX_MEDIA_GRP_ID_VIDMUX,
+	IMX_MEDIA_GRP_ID_CSI,
+	IMX_MEDIA_GRP_ID_SMFC,
+	IMX_MEDIA_GRP_ID_IC_PRPENC,
+	IMX_MEDIA_GRP_ID_IC_PRPVF,
+	IMX_MEDIA_GRP_ID_IC_PP,
+};
+

Documentation/media/v4l-drivers/imx.rst

Quoted 10 lines
Documentation/media/v4l-drivers/imx.rst new file +10 -0
+static const u32 power_off_seq[] = {
+	IMX_MEDIA_GRP_ID_IC_PP,
+	IMX_MEDIA_GRP_ID_IC_PRPVF,
+	IMX_MEDIA_GRP_ID_IC_PRPENC,
+	IMX_MEDIA_GRP_ID_SMFC,
+	IMX_MEDIA_GRP_ID_CSI,
+	IMX_MEDIA_GRP_ID_VIDMUX,
+	IMX_MEDIA_GRP_ID_SENSOR,
+	IMX_MEDIA_GRP_ID_CSI2,
+};

This seems somewhat arbitrary. Why is a power sequence needed?

[...]

Quoted 19 lines
Documentation/media/v4l-drivers/imx.rst new file +19 -0
+/*
+ * Turn current pipeline power on/off starting from start_entity.
+ * Must be called with mdev->graph_mutex held.
+ */
+int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
+				 struct media_entity_graph *graph,
+				 struct media_entity *start_entity, bool on)
+{
+	struct media_entity *entity;
+	struct v4l2_subdev *sd;
+	int i, ret = 0;
+	u32 id;
+
+	for (i = 0; i < NUM_POWER_ENTITIES; i++) {
+		id = on ? power_on_seq[i] : power_off_seq[i];
+		entity = find_pipeline_entity(imxmd, graph, start_entity, id);
+		if (!entity)
+			continue;
+

Documentation/media/v4l-drivers/imx.rst

Quoted 10 lines
Documentation/media/v4l-drivers/imx.rst new file +10 -0
+		sd = media_entity_to_v4l2_subdev(entity);
+
+		ret = v4l2_subdev_call(sd, core, s_power, on);
+		if (ret && ret != -ENOIOCTLCMD)
+			break;
+	}
+
+	return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
+}
+EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);

This should really be handled by v4l2_pipeline_pm_use.

drivers/staging/media/imx/imx-media-dev.c

Quoted 49 lines
Documentation/media/v4l-drivers/imx.rst new file +44 -0
+/*
+ * Inherit the v4l2 controls from all entities in a pipeline
+ * to the given video device.
+ * Must be called with mdev->graph_mutex held.
+ */
+int imx_media_inherit_controls(struct imx_media_dev *imxmd,
+			       struct video_device *vfd,
+			       struct media_entity *start_entity)
+{
+	struct media_entity_graph graph;
+	struct media_entity *entity;
+	struct v4l2_subdev *sd;
+	int ret;
+
+	ret = media_entity_graph_walk_init(&graph, &imxmd->md);
+	if (ret)
+		return ret;
+
+	media_entity_graph_walk_start(&graph, start_entity);
+
+	while ((entity = media_entity_graph_walk_next(&graph))) {
+		if (is_media_entity_v4l2_video_device(entity))
+			continue;
+
+		sd = media_entity_to_v4l2_subdev(entity);
+
+		dev_dbg(imxmd->dev, "%s: adding controls from %s\n",
+			__func__, sd->name);
+
+		ret = v4l2_ctrl_add_handler(vfd->ctrl_handler,
+					    sd->ctrl_handler,
+					    NULL);
+		if (ret)
+			break;
+	}
+
+	media_entity_graph_walk_cleanup(&graph);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(imx_media_inherit_controls);
+
+MODULE_DESCRIPTION("i.MX5/6 v4l2 media controller driver");
+MODULE_AUTHOR("Steve Longerbeam <[email protected]>");
+MODULE_LICENSE("GPL");
drivers/staging/media/imx/imx-media-dev.c new file +0 -0
new file mode 100644
index 0000000..357654d

This file is full of code that should live in the v4l2 core.

[...]

Quoted 2 lines
drivers/staging/media/imx/imx-media-internal-sd.c new file +0 -0

[...]

drivers/staging/media/imx/imx-media-internal-sd.c line 445

Quoted 6 lines
drivers/staging/media/imx/imx-media-internal-sd.c new file +6 -0
+int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
+				   struct imx_media_subdev *csi[4])
+{
+	int ret;
+
+	/* there must be at least one CSI in first IPU */

Why?

Quoted 79 lines
drivers/staging/media/imx/imx-media-internal-sd.c new file +12 -0
+	if (!(csi[0] || csi[1]))
+		return -EINVAL;
+
+	ret = add_ipu_internal_subdevs(imxmd, csi[0], csi[1], 0);
+	if (ret)
+		return ret;
+
+	if (csi[2] || csi[3])
+		ret = add_ipu_internal_subdevs(imxmd, csi[2], csi[3], 1);
+
+	return ret;
+}
drivers/staging/media/imx/imx-media-of.c new file +61 -0
new file mode 100644
index 0000000..a939c34
@@ -0,0 +1,289 @@
+/*
+ * Media driver for Freescale i.MX5/6 SOC
+ *
+ * Open Firmware parsing.
+ *
+ * Copyright (c) 2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/of_platform.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <video/imx-ipu-v3.h>
+#include "imx-media.h"
+
+static int of_add_pad_link(struct imx_media_dev *imxmd,
+			   struct imx_media_pad *pad,
+			   struct device_node *local_sd_node,
+			   struct device_node *remote_sd_node,
+			   int local_pad, int remote_pad)
+{
+	dev_dbg(imxmd->dev, "%s: adding %s:%d -> %s:%d\n", __func__,
+		local_sd_node->name, local_pad,
+		remote_sd_node->name, remote_pad);
+
+	return imx_media_add_pad_link(imxmd, pad, remote_sd_node, NULL,
+				      local_pad, remote_pad);
+}
+
+/* parse inputs property from a sensor node */
+static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
+				   struct imx_media_subdev *sensor,
+				   struct device_node *sensor_np)
+{
+	struct imx_media_sensor_input *sinput = &sensor->input;
+	int ret, i;
+
+	for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
+		const char *input_name;
+		u32 val;
+
+		ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
+		if (ret)
+			break;
+
+		sinput->value[i] = val;
+
+		ret = of_property_read_string_index(sensor_np, "input-names",
+						    i, &input_name);
+		/*
+		 * if input-names not provided, they will be set using
+		 * the subdev name once the sensor is known during
+		 * async bind
+		 */
+		if (!ret)

drivers/staging/media/imx/imx-media-of.c line 71

Quoted 10 lines
drivers/staging/media/imx/imx-media-of.c new file +10 -0
+			strncpy(sinput->name[i], input_name,
+				sizeof(sinput->name[i]));
+	}
+
+	sinput->num = i;
+
+	/* if no inputs provided just assume a single input */
+	if (sinput->num == 0)
+		sinput->num = 1;
+}

This should be parsed by the sensor driver, not imx-media.

Quoted 20 lines
drivers/staging/media/imx/imx-media-of.c new file +20 -0
+static void of_parse_sensor(struct imx_media_dev *imxmd,
+			    struct imx_media_subdev *sensor,
+			    struct device_node *sensor_np)
+{
+	struct device_node *endpoint;
+
+	of_parse_sensor_inputs(imxmd, sensor, sensor_np);
+
+	endpoint = of_graph_get_next_endpoint(sensor_np, NULL);
+	if (endpoint) {
+		v4l2_of_parse_endpoint(endpoint, &sensor->sensor_ep);
+		of_node_put(endpoint);
+	}
+}
+
+static int of_get_port_count(const struct device_node *np)
+{
+	struct device_node *child;
+	int num = 0;
+

drivers/staging/media/imx/imx-media-of.c line 102

Quoted 10 lines
drivers/staging/media/imx/imx-media-of.c new file +10 -0
+	/* if this node is itself a port, return 1 */
+	if (of_node_cmp(np->name, "port") == 0)
+		return 1;
+
+	for_each_child_of_node(np, child)
+		if (of_node_cmp(child->name, "port") == 0)
+			num++;
+
+	return num;
+}

If this is extended to handle the ports subnode properly, it could be moved into drivers/of/base.c.

regards
Philipp

author maintainer
#61 lore raw

Re: [PATCH v3 16/24] media: Add i.MX media core driver

Quoted 10 lines · Philipp Zabel

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:

Documentation/media/v4l-drivers/imx.rst line 18

Quoted 8 lines
Documentation/media/v4l-drivers/imx.rst new file +8 -0
+For image capture, the IPU contains the following internal subunits:
+
+- Image DMA Controller (IDMAC)
+- Camera Serial Interface (CSI)
+- Image Converter (IC)
+- Sensor Multi-FIFO Controller (SMFC)
+- Image Rotator (IRT)
+- Video De-Interlace Controller (VDIC)

Nitpick: Video De-Interlacing or Combining Block (VDIC)

done.

Quoted 11 lines

Documentation/media/v4l-drivers/imx.rst

Quoted 10 lines
Documentation/media/v4l-drivers/imx.rst new file +10 -0
+
+The IDMAC is the DMA controller for transfer of image frames to and from
+memory. Various dedicated DMA channels exist for both video capture and
+display paths.
+
+The CSI is the frontend capture unit that interfaces directly with
+capture sensors over Parallel, BT.656/1120, and MIPI CSI-2 busses.
+
+The IC handles color-space conversion, resizing, and rotation
+operations.

And horizontal flipping.

done.

Quoted 4 lines

Documentation/media/v4l-drivers/imx.rst line 30

Quoted 3 lines

There are three independent "tasks" within the IC that can

Documentation/media/v4l-drivers/imx.rst new file +2 -0
+carry out conversions concurrently: pre-processing encoding,
+pre-processing preview, and post-processing.

s/preview/viewfinder/ seems to be the commonly used name.

replaced everywhere in the doc.

Quoted 6 lines

This paragraph could mention that a single hardware unit is used transparently time multiplexed by the three tasks at different granularity for the downsizing, main processing, and rotation sections. The downscale unit switches between tasks at 8-pixel burst granularity, the main processing unit at line granularity. The rotation units switch only at frame granularity.

I've added that info.

Quoted 6 lines

Documentation/media/v4l-drivers/imx.rst line 35

Quoted 4 lines
Documentation/media/v4l-drivers/imx.rst new file +4 -0
+The SMFC is composed of four independent channels that each can transfer
+captured frames from sensors directly to memory concurrently.
+
+The IRT carries out 90 and 270 degree image rotation operations.

... on 8x8 pixel blocks, supported by the IDMAC which handles block transfers, block reordering, and vertical flipping.

done.

Quoted 5 lines
Quoted 5 lines
Documentation/media/v4l-drivers/imx.rst new file +5 -0
+The VDIC handles the conversion of interlaced video to progressive, with
+support for different motion compensation modes (low, medium, and high
+motion). The deinterlaced output frames from the VDIC can be sent to the
+IC pre-process preview task for further conversions.
+
Quoted 8 lines

Documentation/media/v4l-drivers/imx.rst line 48

Quoted 7 lines
Documentation/media/v4l-drivers/imx.rst new file +7 -0
+In addition to the IPU internal subunits, there are also two units
+outside the IPU that are also involved in video capture on i.MX:
+
+- MIPI CSI-2 Receiver for camera sensors with the MIPI CSI-2 bus
+  interface. This is a Synopsys DesignWare core.
+- A video multiplexer for selecting among multiple sensor inputs to
+  send to a CSI.

Two of them, actually.

done.

Quoted 5 lines

Documentation/media/v4l-drivers/imx.rst line 84

Quoted 4 lines
Documentation/media/v4l-drivers/imx.rst new file +4 -0
+
+- Includes a Frame Interval Monitor (FIM) that can correct vertical sync
+  problems with the ADV718x video decoders. See below for a description
+  of the FIM.

Could this also be used to calculate more precise capture timestamps?

An input capture function could do that, triggered off a VSYNC or FIELD signal such as on the ADV718x. The FIM is only used to calculate frame intervals at this point, but its input capture method could be used to also record more accurate timestamps.

Quoted 21 lines
Quoted 21 lines
Documentation/media/v4l-drivers/imx.rst new file +21 -0
+Capture Pipelines
+-----------------
+
+The following describe the various use-cases supported by the pipelines.
+
+The links shown do not include the frontend sensor, video mux, or mipi
+csi-2 receiver links. This depends on the type of sensor interface
+(parallel or mipi csi-2). So in all cases, these pipelines begin with:
+
+sensor -> ipu_csi_mux -> ipu_csi -> ...
+
+for parallel sensors, or:
+
+sensor -> imx-mipi-csi2 -> (ipu_csi_mux) -> ipu_csi -> ...
+
+for mipi csi-2 sensors. The imx-mipi-csi2 receiver may need to route
+to the video mux (ipu_csi_mux) before sending to the CSI, depending
+on the mipi csi-2 virtual channel, hence ipu_csi_mux is shown in
+parenthesis.
+
+Unprocessed Video Capture:
Quoted 8 lines

Documentation/media/v4l-drivers/imx.rst line 113

Quoted 6 lines

+--------------------------

Documentation/media/v4l-drivers/imx.rst new file +5 -0
+
+Send frames directly from sensor to camera interface, with no
+conversions:
+
+-> ipu_smfc -> camif

I'd call this capture interface, this is not just for cameras. Or maybe idmac if you want to mirror hardware names?

Camif is so named because it is the V4L2 user interface for video capture. I suppose it could be named "capif", but that doesn't role off the tongue quite as well.

Quoted 2 lines
Quoted 1 line

+Note the ipu_smfc can do pixel reordering within the same colorspace.

That isn't a feature of the SMFC, but of the IDMAC (FCW & FCR).

yes, the doc is re-worded to make that more clear.

Quoted 7 lines

Documentation/media/v4l-drivers/imx.rst line 117

Quoted 2 lines
Documentation/media/v4l-drivers/imx.rst new file +2 -0
+For example, its sink pad can take UYVY2X8, but its source pad can
+output YUYV2X8.

I don't think this is correct. Re-reading "37.4.3.7 Packing to memory" in the CSI chapter, for 8-bit per component data, the internal format between CSI, SMFC, and IDMAC is always some 32-bit RGBx/YUVx variant (or "bayer/generic data"). In either case, the internal format does not change along the way.

these are pixels in memory buffers, not the IPU internal formats.

Quoted 5 lines
Quoted 5 lines
Documentation/media/v4l-drivers/imx.rst new file +5 -0
+IC Direct Conversions:
+----------------------
+
+This pipeline uses the preprocess encode entity to route frames directly
+from the CSI to the IC (bypassing the SMFC), to carry out scaling up to
Quoted 8 lines

Documentation/media/v4l-drivers/imx.rst line 129

Quoted 6 lines

+1024x1024 resolution, CSC, and image rotation:

Documentation/media/v4l-drivers/imx.rst new file +5 -0
+
+-> ipu_ic_prpenc -> camif
+
+This can be a useful capture pipeline for heavily loaded memory bus
+traffic environments, since it has minimal IDMAC channel usage.

Note that if rotation is enabled, transfers between IC processing and rotation still have to go through memory once.

yep.

Quoted 7 lines

Documentation/media/v4l-drivers/imx.rst

Quoted 5 lines
Documentation/media/v4l-drivers/imx.rst new file +5 -0
+Post-Processing Conversions:
+----------------------------
+
+This pipeline routes frames from the SMFC to the post-processing
+entity.

No, frames written by the CSI -> SMFC -> IDMAC path are read back into the post-processing entity.

that's true. The post-processing entity kicks off its read channels to transfer those frames into the post-processor. Anyway this wording will change after doing away with the SMFC entity.

Quoted 4 lines

Documentation/media/v4l-drivers/imx.rst line 230

Quoted 3 lines
Documentation/media/v4l-drivers/imx.rst new file +3 -0
+   media-ctl -V "\"ipu1_csi0\":1 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_smfc0\":0 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/640x480]"

I think the smfc entities should be dropped.

yes working on that.

Quoted 8 lines

Documentation/media/v4l-drivers/imx.rst line 238

Quoted 8 lines
Documentation/media/v4l-drivers/imx.rst new file +8 -0
+   media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480]"
+   # Configure pads for OV5640 pipeline
+   media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480]"
Quoted 7 lines

[...]

Quoted 1 line

+ media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"

I agree this looks very intuitive, but technically correct for the csi1:1 and camif1:0 pads would be a 32-bit YUV format. (MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).

I think it would be better to use the correct format

I'm not sure I follow you here.

Quoted 27 lines

as that will allow
to chose the regular vs. companded packings in the future for formats with more than 8 bits per component.

Quoted 23 lines
Documentation/media/v4l-drivers/imx.rst new file +23 -0
+   media-ctl -V "\"camif1\":1 [fmt:UYVY2X8/640x480]"
+
+Streaming can then begin independently on device nodes /dev/video0
+and /dev/video1.
+
+SabreAuto with ADV7180 decoder
+------------------------------
+
+On the SabreAuto, an on-board ADV7180 SD decoder is connected to the
+parallel bus input on the internal video mux to IPU1 CSI0.
+
+The following example configures a pipeline to capture from the ADV7180
+video decoder, assuming NTSC 720x480 input signals, with Motion
+Compensated de-interlacing (not shown: all pad field types should be set
+as indicated). $outputfmt can be any format supported by the
+ipu1_ic_prpvf entity at its output pad:
+
+.. code-block:: none
+
+   # Setup links
+   media-ctl -l '"adv7180 3-0021":0 -> "ipu1_csi0_mux":1[1]'
+   media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
+   media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
Quoted 8 lines

Documentation/media/v4l-drivers/imx.rst line 270

Quoted 6 lines
Documentation/media/v4l-drivers/imx.rst new file +6 -0
+   media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]'
+   media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]'
+   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
+   # Configure pads
+   # pad field types for below pads must be an interlaced type
+   # such as "ALTERNATE"

I think alternate should only extend as far as the CSI, since the CSI can only capture NTSC/PAL fields in a fixed order.

Agreed, I'm doing the translation from alternate to seq_bt/seq_tb depending on sensor std in imx-vdic.c, but it should move upstream. Will fix.

Quoted 4 lines

Documentation/media/v4l-drivers/imx.rst line 272

Quoted 2 lines
Documentation/media/v4l-drivers/imx.rst new file +2 -0
+   media-ctl -V "\"adv7180 3-0021\":0 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480]"

From here the interlaced field type should be sequential in the correct
order depending on NTSC/PAL.

right.

Quoted 6 lines
Quoted 6 lines
Documentation/media/v4l-drivers/imx.rst new file +6 -0
+
+Frame Interval Monitor
+----------------------
+
+The adv718x decoders can occasionally send corrupt fields during
+NTSC/PAL signal re-sync (too little or too many video lines). When
Quoted 8 lines

Documentation/media/v4l-drivers/imx.rst line 312

Quoted 6 lines
Documentation/media/v4l-drivers/imx.rst new file +6 -0
+this happens, the IPU triggers a mechanism to re-establish vertical
+sync by adding 1 dummy line every frame, which causes a rolling effect
+from image to image, and can last a long time before a stable image is
+recovered. Or sometimes the mechanism doesn't work at all, causing a
+permanent split image (one frame contains lines from two consecutive
+captured images).

Is it only SabreAuto on which the FIM mechanism can be used due to the pad routing?

No, FIM can be used on any target, the fim child node of ipu_csi just needs to be enabled via status property.

Quoted 16 lines

[...]

Quoted 15 lines
drivers/staging/media/imx/imx-media-common.c new file +15 -0
+/*
+ * DMA buffer ring handling
+ */
+struct imx_media_dma_buf_ring {
+	struct imx_media_dev *imxmd;
+
+	/* the ring */
+	struct imx_media_dma_buf buf[IMX_MEDIA_MAX_RING_BUFS];
+	/* the scratch buffer for underruns */
+	struct imx_media_dma_buf scratch;
+
+	/* buffer generator */
+	struct media_entity *src;
+	/* buffer receiver */
+	struct media_entity *sink;
Quoted 8 lines

drivers/staging/media/imx/imx-media-common.c line 232

Quoted 6 lines
drivers/staging/media/imx/imx-media-common.c new file +6 -0
+
+	spinlock_t lock;
+
+	int num_bufs;
+	unsigned long last_seq;
+};

I don't think this belongs in the capture driver at all. Memory-to-memory transfers should be handled at the videobuf2 level.

see below.

Quoted 13 lines

[...]

Quoted 12 lines
drivers/staging/media/imx/imx-media-common.c new file +12 -0
+static struct imx_media_dma_buf *
+__dma_buf_queue(struct imx_media_dma_buf_ring *ring, int index)
+{
+	struct imx_media_dma_buf *buf;
+
+	if (index >= ring->num_bufs)
+		return ERR_PTR(-EINVAL);
+
+	buf = &ring->buf[index];
+	if (WARN_ON(buf->state != IMX_MEDIA_BUF_STATUS_PREPARED))
+		return ERR_PTR(-EINVAL);
+
Quoted 8 lines

drivers/staging/media/imx/imx-media-common.c line 359

Quoted 5 lines
drivers/staging/media/imx/imx-media-common.c new file +5 -0
+	buf->state = IMX_MEDIA_BUF_STATUS_QUEUED;
+	buf->seq = ring->last_seq++;
+
+	return buf;
+}

Is this a whole software buffer queue implementation? I thought the whole point of putting the custom mem2mem framework into the capture driver was to use the hardware FSU channel linking?

see below.

Quoted 1 line

What is the purpose of this if the sink should be triggered by the FSU?

Ok, here is where I need to make an admission.

The only FSU links I have attempted (and which currently have entries in the fsu_link_info[] table), are the enc/vf/pp --> IRT links for rotation.

There does not appear to be support in the FSU for linking a write channel to the VDIC read channels (8, 9, 10) according to VDI_SRC_SEL field. There is support for the direct link from CSI (which I am using), but that's not an IDMAC channel link.

There is a PRP_SRC_SEL field, with linking from IDMAC (SMFC) channels 0..2 (and 3? it's not clear, and not clear whether this includes channel 1). But I think this links to channel 12, and not to channels 8,9,10 to the VDIC. Or will it? It's worth experimenting. It would have helped if FSL listed which IDMAC channels these FSU links correspond to, instead of making us guess at it.

In any event, the docs are not clear enough to implement a real FSU link to the VDIC read channels, if it's even possible. And trying to get programming help from FSL can be difficult, and no coding examples for this link AFAIK.

So I ended resorted to linking to VDIC channels 8,9,10 with a software approach, instead of attempting a hardware FSU link.

The EOF interrupt handler for the SMFC channels informs the VDIC entity via a v4l2_subdev_ioctl() call that a buffer is available. The VDIC then manually kicks off its read channels to bring that buffer (and a previous buffer for F(n-1) field) into the VDIC.

There is a small amount of extra overhead going this route compared to a FSU hardware link: there is the EOF irq latency (a few usec), and the CPU overhead for the VDIC to manually start the read channels, which is also a few usec at most (see prepare_vdi_in_buffers() in imx-vdic.c). So in total at most ~10 usec of extra overhead (CPU use plus irq latency) under normal system load.

Of course, in order to implement this software link, I had to implement a straightforward FIFO dma buffer ring. The sink (VDIC) allocates the ring at stream on, and the source requests a pointer to this ring in its own stream on. Passing buffers from source to sink then follows a straightforward FIFO queue/done/dequeue/queue model: sink queues buffers to src, src grabs queued buffers and makes them active, src signals completed buffers to sink, sink dequeues buffers in response, and sink queues buffers back when it is finished with them.

Quoted 43 lines

[...]

Quoted 42 lines
drivers/staging/media/imx/imx-media-common.c new file +42 -0
+/*
+ * The subdevs have to be powered on/off, and streaming
+ * enabled/disabled, in a specific sequence.
+ */
+static const u32 stream_on_seq[] = {
+	IMX_MEDIA_GRP_ID_IC_PP,
+	IMX_MEDIA_GRP_ID_IC_PRPVF,
+	IMX_MEDIA_GRP_ID_IC_PRPENC,
+	IMX_MEDIA_GRP_ID_SMFC,
+	IMX_MEDIA_GRP_ID_SENSOR,
+	IMX_MEDIA_GRP_ID_CSI2,
+	IMX_MEDIA_GRP_ID_VIDMUX,
+	IMX_MEDIA_GRP_ID_CSI,
+};
+
+static const u32 stream_off_seq[] = {
+	IMX_MEDIA_GRP_ID_IC_PP,
+	IMX_MEDIA_GRP_ID_IC_PRPVF,
+	IMX_MEDIA_GRP_ID_IC_PRPENC,
+	IMX_MEDIA_GRP_ID_SMFC,
+	IMX_MEDIA_GRP_ID_CSI,
+	IMX_MEDIA_GRP_ID_VIDMUX,
+	IMX_MEDIA_GRP_ID_CSI2,
+	IMX_MEDIA_GRP_ID_SENSOR,
+};
+
+#define NUM_STREAM_ENTITIES ARRAY_SIZE(stream_on_seq)
+
+static const u32 power_on_seq[] = {
+	IMX_MEDIA_GRP_ID_CSI2,
+	IMX_MEDIA_GRP_ID_SENSOR,
+	IMX_MEDIA_GRP_ID_VIDMUX,
+	IMX_MEDIA_GRP_ID_CSI,
+	IMX_MEDIA_GRP_ID_SMFC,
+	IMX_MEDIA_GRP_ID_IC_PRPENC,
+	IMX_MEDIA_GRP_ID_IC_PRPVF,
+	IMX_MEDIA_GRP_ID_IC_PP,
+};
+
+static const u32 power_off_seq[] = {
+	IMX_MEDIA_GRP_ID_IC_PP,
+	IMX_MEDIA_GRP_ID_IC_PRPVF,
Quoted 8 lines

drivers/staging/media/imx/imx-media-common.c line 829

Quoted 7 lines
drivers/staging/media/imx/imx-media-common.c new file +7 -0
+	IMX_MEDIA_GRP_ID_IC_PRPENC,
+	IMX_MEDIA_GRP_ID_SMFC,
+	IMX_MEDIA_GRP_ID_CSI,
+	IMX_MEDIA_GRP_ID_VIDMUX,
+	IMX_MEDIA_GRP_ID_SENSOR,
+	IMX_MEDIA_GRP_ID_CSI2,
+};

This seems somewhat arbitrary. Why is a power sequence needed?

The CSI-2 receiver must be powered up before the sensor, that's the only requirement IIRC. The others have no s_power requirement. So I can probably change this to power up in the frontend -> backend order (IC_PP to sensor). And vice-versa for power off.

Quoted 23 lines

[...]

Quoted 22 lines
drivers/staging/media/imx/imx-media-common.c new file +22 -0
+/*
+ * Turn current pipeline power on/off starting from start_entity.
+ * Must be called with mdev->graph_mutex held.
+ */
+int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
+				 struct media_entity_graph *graph,
+				 struct media_entity *start_entity, bool on)
+{
+	struct media_entity *entity;
+	struct v4l2_subdev *sd;
+	int i, ret = 0;
+	u32 id;
+
+	for (i = 0; i < NUM_POWER_ENTITIES; i++) {
+		id = on ? power_on_seq[i] : power_off_seq[i];
+		entity = find_pipeline_entity(imxmd, graph, start_entity, id);
+		if (!entity)
+			continue;
+
+		sd = media_entity_to_v4l2_subdev(entity);
+
+		ret = v4l2_subdev_call(sd, core, s_power, on);
Quoted 8 lines

drivers/staging/media/imx/imx-media-common.c line 936

Quoted 7 lines
drivers/staging/media/imx/imx-media-common.c new file +7 -0
+		if (ret && ret != -ENOIOCTLCMD)
+			break;
+	}
+
+	return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
+}
+EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);

This should really be handled by v4l2_pipeline_pm_use.

I thought about this earlier, but v4l2_pipeline_pm_use() seems to be doing some other stuff that bothered me, at least that's what I remember. I will revisit this.

Quoted 42 lines
Quoted 42 lines
drivers/staging/media/imx/imx-media-common.c new file +42 -0
+/*
+ * Inherit the v4l2 controls from all entities in a pipeline
+ * to the given video device.
+ * Must be called with mdev->graph_mutex held.
+ */
+int imx_media_inherit_controls(struct imx_media_dev *imxmd,
+			       struct video_device *vfd,
+			       struct media_entity *start_entity)
+{
+	struct media_entity_graph graph;
+	struct media_entity *entity;
+	struct v4l2_subdev *sd;
+	int ret;
+
+	ret = media_entity_graph_walk_init(&graph, &imxmd->md);
+	if (ret)
+		return ret;
+
+	media_entity_graph_walk_start(&graph, start_entity);
+
+	while ((entity = media_entity_graph_walk_next(&graph))) {
+		if (is_media_entity_v4l2_video_device(entity))
+			continue;
+
+		sd = media_entity_to_v4l2_subdev(entity);
+
+		dev_dbg(imxmd->dev, "%s: adding controls from %s\n",
+			__func__, sd->name);
+
+		ret = v4l2_ctrl_add_handler(vfd->ctrl_handler,
+					    sd->ctrl_handler,
+					    NULL);
+		if (ret)
+			break;
+	}
+
+	media_entity_graph_walk_cleanup(&graph);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(imx_media_inherit_controls);
+
+MODULE_DESCRIPTION("i.MX5/6 v4l2 media controller driver");
Quoted 8 lines

drivers/staging/media/imx/imx-media-dev.c

Quoted 7 lines
(quoted diff) +2 -0
+MODULE_AUTHOR("Steve Longerbeam<[email protected]>");
+MODULE_LICENSE("GPL");
drivers/staging/media/imx/imx-media-dev.c new file +0 -0
new file mode 100644
index 0000000..357654d

This file is full of code that should live in the v4l2 core.

Agreed. Stuff like imx_media_inherit_controls() really should be widely available.

Quoted 7 lines

drivers/staging/media/imx/imx-media-dev.c

Quoted 6 lines
drivers/staging/media/imx/imx-media-dev.c new file +6 -0
+int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
+				   struct imx_media_subdev *csi[4])
+{
+	int ret;
+
+	/* there must be at least one CSI in first IPU */

Why?

Well yeah, imx-media doesn't necessarily need a CSI if things like the VDIC or post-processor are being used by an output overlay pipeline, for example. I'll fix this.

Quoted 30 lines
Quoted 30 lines
drivers/staging/media/imx/imx-media-dev.c new file +30 -0
+
+/* parse inputs property from a sensor node */
+static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
+				   struct imx_media_subdev *sensor,
+				   struct device_node *sensor_np)
+{
+	struct imx_media_sensor_input *sinput = &sensor->input;
+	int ret, i;
+
+	for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
+		const char *input_name;
+		u32 val;
+
+		ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
+		if (ret)
+			break;
+
+		sinput->value[i] = val;
+
+		ret = of_property_read_string_index(sensor_np, "input-names",
+						    i, &input_name);
+		/*
+		 * if input-names not provided, they will be set using
+		 * the subdev name once the sensor is known during
+		 * async bind
+		 */
+		if (!ret)
+			strncpy(sinput->name[i], input_name,
+				sizeof(sinput->name[i]));
+	}
Quoted 8 lines

drivers/staging/media/imx/imx-media-dev.c

Quoted 7 lines
drivers/staging/media/imx/imx-media-dev.c new file +7 -0
+
+	sinput->num = i;
+
+	/* if no inputs provided just assume a single input */
+	if (sinput->num == 0)
+		sinput->num = 1;
+}

This should be parsed by the sensor driver, not imx-media.

you're probably right. I'll submit a patch for adv7180.c.

Quoted 24 lines
Quoted 24 lines
drivers/staging/media/imx/imx-media-dev.c new file +24 -0
+static void of_parse_sensor(struct imx_media_dev *imxmd,
+			    struct imx_media_subdev *sensor,
+			    struct device_node *sensor_np)
+{
+	struct device_node *endpoint;
+
+	of_parse_sensor_inputs(imxmd, sensor, sensor_np);
+
+	endpoint = of_graph_get_next_endpoint(sensor_np, NULL);
+	if (endpoint) {
+		v4l2_of_parse_endpoint(endpoint, &sensor->sensor_ep);
+		of_node_put(endpoint);
+	}
+}
+
+static int of_get_port_count(const struct device_node *np)
+{
+	struct device_node *child;
+	int num = 0;
+
+	/* if this node is itself a port, return 1 */
+	if (of_node_cmp(np->name, "port") == 0)
+		return 1;
+
Quoted 8 lines

drivers/staging/media/imx/imx-media-dev.c

Quoted 6 lines
drivers/staging/media/imx/imx-media-dev.c new file +6 -0
+	for_each_child_of_node(np, child)
+		if (of_node_cmp(child->name, "port") == 0)
+			num++;
+
+	return num;
+}

If this is extended to handle the ports subnode properly, it could be moved into drivers/of/base.c.

More that eventually needs exporting, agreed.

Steve

author maintainer
#62 lore raw

Re: [PATCH v3 16/24] media: Add i.MX media core driver

Quoted 39 lines · Steve Longerbeam
Quoted 30 lines
Quoted 30 lines
drivers/staging/media/imx/imx-media-of.c new file +29 -0
+/* parse inputs property from a sensor node */
+static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
+				   struct imx_media_subdev *sensor,
+				   struct device_node *sensor_np)
+{
+	struct imx_media_sensor_input *sinput = &sensor->input;
+	int ret, i;
+
+	for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
+		const char *input_name;
+		u32 val;
+
+		ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
+		if (ret)
+			break;
+
+		sinput->value[i] = val;
+
+		ret = of_property_read_string_index(sensor_np, "input-names",
+						    i, &input_name);
+		/*
+		 * if input-names not provided, they will be set using
+		 * the subdev name once the sensor is known during
+		 * async bind
+		 */
+		if (!ret)
+			strncpy(sinput->name[i], input_name,
+				sizeof(sinput->name[i]));
+	}
Quoted 8 lines

drivers/staging/media/imx/imx-media-of.c line 71

Quoted 7 lines
drivers/staging/media/imx/imx-media-of.c new file +7 -0
+
+	sinput->num = i;
+
+	/* if no inputs provided just assume a single input */
+	if (sinput->num == 0)
+		sinput->num = 1;
+}

This should be parsed by the sensor driver, not imx-media.

Quoted 1 line

you're probably right. I'll submit a patch for adv7180.c.

Actually, the problem here is that this parses an input routing value to pass to s_routing, and an input name string. There would need to be another subdev callback, maybe enum_imput, that would return this information for the bridge driver, if this info were to be parsed and maintained by the sensor.

But this info should really be known and parsed by the bridge anyway, because as the header for s_routing states,

"An i2c device shouldn't know about whether an input pin is connected
to a Composite connector, because on another board or platform it might be connected to something else entirely. The calling driver is responsible for mapping a user-level input to the right pins on the i2c device."

Steve

maintainer
#63 lore raw

Re: [PATCH v3 16/24] media: Add i.MX media core driver

Quoted 49 lines · Steve Longerbeam
Quoted 39 lines · Steve Longerbeam
Quoted 30 lines
Quoted 30 lines
drivers/staging/media/imx/imx-media-of.c new file +29 -0
+/* parse inputs property from a sensor node */
+static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
+				   struct imx_media_subdev *sensor,
+				   struct device_node *sensor_np)
+{
+	struct imx_media_sensor_input *sinput = &sensor->input;
+	int ret, i;
+
+	for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
+		const char *input_name;
+		u32 val;
+
+		ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
+		if (ret)
+			break;
+
+		sinput->value[i] = val;
+
+		ret = of_property_read_string_index(sensor_np, "input-names",
+						    i, &input_name);
+		/*
+		 * if input-names not provided, they will be set using
+		 * the subdev name once the sensor is known during
+		 * async bind
+		 */
+		if (!ret)
+			strncpy(sinput->name[i], input_name,
+				sizeof(sinput->name[i]));
+	}
Quoted 8 lines

drivers/staging/media/imx/imx-media-of.c line 71

Quoted 7 lines
drivers/staging/media/imx/imx-media-of.c new file +7 -0
+
+	sinput->num = i;
+
+	/* if no inputs provided just assume a single input */
+	if (sinput->num == 0)
+		sinput->num = 1;
+}

This should be parsed by the sensor driver, not imx-media.

Quoted 1 line

you're probably right. I'll submit a patch for adv7180.c.

Actually, the problem here is that this parses an input routing value to pass to s_routing, and an input name string. There would need to be another subdev callback, maybe enum_imput, that would return this information for the bridge driver, if this info were to be parsed and maintained by the sensor.

Quoted 8 lines

But this info should really be known and parsed by the bridge anyway, because as the header for s_routing states,

"An i2c device shouldn't know about whether an input pin is connected
to a Composite connector, because on another board or platform it might be connected to something else entirely. The calling driver is responsible for mapping a user-level input to the right pins on the i2c device."

I think this description is for non-DT only, as parsing the DT bindings is the obligation of the bound driver, and with that it information it can very well know its connections.

regards
Philipp

author maintainer
#64 lore raw

Re: [PATCH v3 16/24] media: Add i.MX media core driver

In reply to Philipp Zabel

(sorry sending again w/o html)

Quoted 10 lines · Philipp Zabel

Am Freitag, den 06.01.2017, 18:11 -0800 schrieb Steve Longerbeam:

Documentation/media/v4l-drivers/imx.rst line 18

Quoted 8 lines
Documentation/media/v4l-drivers/imx.rst new file +8 -0
+For image capture, the IPU contains the following internal subunits:
+
+- Image DMA Controller (IDMAC)
+- Camera Serial Interface (CSI)
+- Image Converter (IC)
+- Sensor Multi-FIFO Controller (SMFC)
+- Image Rotator (IRT)
+- Video De-Interlace Controller (VDIC)

Nitpick: Video De-Interlacing or Combining Block (VDIC)

done.

Quoted 11 lines

Documentation/media/v4l-drivers/imx.rst

Quoted 10 lines
Documentation/media/v4l-drivers/imx.rst new file +10 -0
+
+The IDMAC is the DMA controller for transfer of image frames to and from
+memory. Various dedicated DMA channels exist for both video capture and
+display paths.
+
+The CSI is the frontend capture unit that interfaces directly with
+capture sensors over Parallel, BT.656/1120, and MIPI CSI-2 busses.
+
+The IC handles color-space conversion, resizing, and rotation
+operations.

And horizontal flipping.

done.

Quoted 4 lines

Documentation/media/v4l-drivers/imx.rst line 30

Quoted 3 lines

There are three independent "tasks" within the IC that can

Documentation/media/v4l-drivers/imx.rst new file +2 -0
+carry out conversions concurrently: pre-processing encoding,
+pre-processing preview, and post-processing.

s/preview/viewfinder/ seems to be the commonly used name.

replaced everywhere in the doc.

Quoted 6 lines

This paragraph could mention that a single hardware unit is used transparently time multiplexed by the three tasks at different granularity for the downsizing, main processing, and rotation sections. The downscale unit switches between tasks at 8-pixel burst granularity, the main processing unit at line granularity. The rotation units switch only at frame granularity.

I've added that info.

Quoted 6 lines

Documentation/media/v4l-drivers/imx.rst line 35

Quoted 4 lines
Documentation/media/v4l-drivers/imx.rst new file +4 -0
+The SMFC is composed of four independent channels that each can transfer
+captured frames from sensors directly to memory concurrently.
+
+The IRT carries out 90 and 270 degree image rotation operations.

... on 8x8 pixel blocks, supported by the IDMAC which handles block transfers, block reordering, and vertical flipping.

done.

Quoted 5 lines
Quoted 5 lines
Documentation/media/v4l-drivers/imx.rst new file +5 -0
+The VDIC handles the conversion of interlaced video to progressive, with
+support for different motion compensation modes (low, medium, and high
+motion). The deinterlaced output frames from the VDIC can be sent to the
+IC pre-process preview task for further conversions.
+
Quoted 8 lines

Documentation/media/v4l-drivers/imx.rst line 48

Quoted 7 lines
Documentation/media/v4l-drivers/imx.rst new file +7 -0
+In addition to the IPU internal subunits, there are also two units
+outside the IPU that are also involved in video capture on i.MX:
+
+- MIPI CSI-2 Receiver for camera sensors with the MIPI CSI-2 bus
+  interface. This is a Synopsys DesignWare core.
+- A video multiplexer for selecting among multiple sensor inputs to
+  send to a CSI.

Two of them, actually.

done.

Quoted 5 lines

Documentation/media/v4l-drivers/imx.rst line 84

Quoted 4 lines
Documentation/media/v4l-drivers/imx.rst new file +4 -0
+
+- Includes a Frame Interval Monitor (FIM) that can correct vertical sync
+  problems with the ADV718x video decoders. See below for a description
+  of the FIM.

Could this also be used to calculate more precise capture timestamps?

An input capture function could do that, triggered off a VSYNC or FIELD signal such as on the ADV718x. The FIM is only used to calculate frame intervals at this point, but its input capture method could be used to also record more accurate timestamps.

Quoted 21 lines
Quoted 21 lines
Documentation/media/v4l-drivers/imx.rst new file +21 -0
+Capture Pipelines
+-----------------
+
+The following describe the various use-cases supported by the pipelines.
+
+The links shown do not include the frontend sensor, video mux, or mipi
+csi-2 receiver links. This depends on the type of sensor interface
+(parallel or mipi csi-2). So in all cases, these pipelines begin with:
+
+sensor -> ipu_csi_mux -> ipu_csi -> ...
+
+for parallel sensors, or:
+
+sensor -> imx-mipi-csi2 -> (ipu_csi_mux) -> ipu_csi -> ...
+
+for mipi csi-2 sensors. The imx-mipi-csi2 receiver may need to route
+to the video mux (ipu_csi_mux) before sending to the CSI, depending
+on the mipi csi-2 virtual channel, hence ipu_csi_mux is shown in
+parenthesis.
+
+Unprocessed Video Capture:
Quoted 8 lines

Documentation/media/v4l-drivers/imx.rst line 113

Quoted 6 lines

+--------------------------

Documentation/media/v4l-drivers/imx.rst new file +5 -0
+
+Send frames directly from sensor to camera interface, with no
+conversions:
+
+-> ipu_smfc -> camif

I'd call this capture interface, this is not just for cameras. Or maybe idmac if you want to mirror hardware names?

Camif is so named because it is the V4L2 user interface for video capture. I suppose it could be named "capif", but that doesn't role off the tongue quite as well.

Quoted 2 lines
Quoted 1 line

+Note the ipu_smfc can do pixel reordering within the same colorspace.

That isn't a feature of the SMFC, but of the IDMAC (FCW & FCR).

yes, the doc is re-worded to make that more clear.

Quoted 7 lines

Documentation/media/v4l-drivers/imx.rst line 117

Quoted 2 lines
Documentation/media/v4l-drivers/imx.rst new file +2 -0
+For example, its sink pad can take UYVY2X8, but its source pad can
+output YUYV2X8.

I don't think this is correct. Re-reading "37.4.3.7 Packing to memory" in the CSI chapter, for 8-bit per component data, the internal format between CSI, SMFC, and IDMAC is always some 32-bit RGBx/YUVx variant (or "bayer/generic data"). In either case, the internal format does not change along the way.

these are pixels in memory buffers, not the IPU internal formats.

Quoted 5 lines
Quoted 5 lines
Documentation/media/v4l-drivers/imx.rst new file +5 -0
+IC Direct Conversions:
+----------------------
+
+This pipeline uses the preprocess encode entity to route frames directly
+from the CSI to the IC (bypassing the SMFC), to carry out scaling up to
Quoted 8 lines

Documentation/media/v4l-drivers/imx.rst line 129

Quoted 6 lines

+1024x1024 resolution, CSC, and image rotation:

Documentation/media/v4l-drivers/imx.rst new file +5 -0
+
+-> ipu_ic_prpenc -> camif
+
+This can be a useful capture pipeline for heavily loaded memory bus
+traffic environments, since it has minimal IDMAC channel usage.

Note that if rotation is enabled, transfers between IC processing and rotation still have to go through memory once.

yep.

Quoted 7 lines

Documentation/media/v4l-drivers/imx.rst

Quoted 5 lines
Documentation/media/v4l-drivers/imx.rst new file +5 -0
+Post-Processing Conversions:
+----------------------------
+
+This pipeline routes frames from the SMFC to the post-processing
+entity.

No, frames written by the CSI -> SMFC -> IDMAC path are read back into the post-processing entity.

that's true. The post-processing entity kicks off its read channels to transfer those frames into the post-processor. Anyway this wording will change after doing away with the SMFC entity.

Quoted 4 lines

Documentation/media/v4l-drivers/imx.rst line 230

Quoted 3 lines
Documentation/media/v4l-drivers/imx.rst new file +3 -0
+   media-ctl -V "\"ipu1_csi0\":1 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_smfc0\":0 [fmt:YUYV2X8/640x480]"
+   media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/640x480]"

I think the smfc entities should be dropped.

yes working on that.

Quoted 8 lines

Documentation/media/v4l-drivers/imx.rst line 238

Quoted 8 lines
Documentation/media/v4l-drivers/imx.rst new file +8 -0
+   media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480]"
+   # Configure pads for OV5640 pipeline
+   media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480]"
Quoted 7 lines

[...]

Quoted 1 line

+ media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"

I agree this looks very intuitive, but technically correct for the csi1:1 and camif1:0 pads would be a 32-bit YUV format. (MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).

I think it would be better to use the correct format

I'm not sure I follow you here.

Quoted 27 lines

as that will allow
to chose the regular vs. companded packings in the future for formats with more than 8 bits per component.

Quoted 23 lines
Documentation/media/v4l-drivers/imx.rst new file +23 -0
+   media-ctl -V "\"camif1\":1 [fmt:UYVY2X8/640x480]"
+
+Streaming can then begin independently on device nodes /dev/video0
+and /dev/video1.
+
+SabreAuto with ADV7180 decoder
+------------------------------
+
+On the SabreAuto, an on-board ADV7180 SD decoder is connected to the
+parallel bus input on the internal video mux to IPU1 CSI0.
+
+The following example configures a pipeline to capture from the ADV7180
+video decoder, assuming NTSC 720x480 input signals, with Motion
+Compensated de-interlacing (not shown: all pad field types should be set
+as indicated). $outputfmt can be any format supported by the
+ipu1_ic_prpvf entity at its output pad:
+
+.. code-block:: none
+
+   # Setup links
+   media-ctl -l '"adv7180 3-0021":0 -> "ipu1_csi0_mux":1[1]'
+   media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]'
+   media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]'
Quoted 8 lines

Documentation/media/v4l-drivers/imx.rst line 270

Quoted 6 lines
Documentation/media/v4l-drivers/imx.rst new file +6 -0
+   media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]'
+   media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]'
+   media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]'
+   # Configure pads
+   # pad field types for below pads must be an interlaced type
+   # such as "ALTERNATE"

I think alternate should only extend as far as the CSI, since the CSI can only capture NTSC/PAL fields in a fixed order.

Agreed, I'm doing the translation from alternate to seq_bt/seq_tb depending on sensor std in imx-vdic.c, but it should move upstream. Will fix.

Quoted 4 lines

Documentation/media/v4l-drivers/imx.rst line 272

Quoted 2 lines
Documentation/media/v4l-drivers/imx.rst new file +2 -0
+   media-ctl -V "\"adv7180 3-0021\":0 [fmt:UYVY2X8/720x480]"
+   media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480]"

From here the interlaced field type should be sequential in the correct
order depending on NTSC/PAL.

right.

Quoted 6 lines
Quoted 6 lines
Documentation/media/v4l-drivers/imx.rst new file +6 -0
+
+Frame Interval Monitor
+----------------------
+
+The adv718x decoders can occasionally send corrupt fields during
+NTSC/PAL signal re-sync (too little or too many video lines). When
Quoted 8 lines

Documentation/media/v4l-drivers/imx.rst line 312

Quoted 6 lines
Documentation/media/v4l-drivers/imx.rst new file +6 -0
+this happens, the IPU triggers a mechanism to re-establish vertical
+sync by adding 1 dummy line every frame, which causes a rolling effect
+from image to image, and can last a long time before a stable image is
+recovered. Or sometimes the mechanism doesn't work at all, causing a
+permanent split image (one frame contains lines from two consecutive
+captured images).

Is it only SabreAuto on which the FIM mechanism can be used due to the pad routing?

No, FIM can be used on any target, the fim child node of ipu_csi just needs to be enabled via status property.

Quoted 16 lines

[...]

Quoted 15 lines
drivers/staging/media/imx/imx-media-common.c new file +15 -0
+/*
+ * DMA buffer ring handling
+ */
+struct imx_media_dma_buf_ring {
+	struct imx_media_dev *imxmd;
+
+	/* the ring */
+	struct imx_media_dma_buf buf[IMX_MEDIA_MAX_RING_BUFS];
+	/* the scratch buffer for underruns */
+	struct imx_media_dma_buf scratch;
+
+	/* buffer generator */
+	struct media_entity *src;
+	/* buffer receiver */
+	struct media_entity *sink;
Quoted 8 lines

drivers/staging/media/imx/imx-media-common.c line 232

Quoted 6 lines
drivers/staging/media/imx/imx-media-common.c new file +6 -0
+
+	spinlock_t lock;
+
+	int num_bufs;
+	unsigned long last_seq;
+};

I don't think this belongs in the capture driver at all. Memory-to-memory transfers should be handled at the videobuf2 level.

see below.

Quoted 13 lines

[...]

Quoted 12 lines
drivers/staging/media/imx/imx-media-common.c new file +12 -0
+static struct imx_media_dma_buf *
+__dma_buf_queue(struct imx_media_dma_buf_ring *ring, int index)
+{
+	struct imx_media_dma_buf *buf;
+
+	if (index >= ring->num_bufs)
+		return ERR_PTR(-EINVAL);
+
+	buf = &ring->buf[index];
+	if (WARN_ON(buf->state != IMX_MEDIA_BUF_STATUS_PREPARED))
+		return ERR_PTR(-EINVAL);
+
Quoted 8 lines

drivers/staging/media/imx/imx-media-common.c line 359

Quoted 5 lines
drivers/staging/media/imx/imx-media-common.c new file +5 -0
+	buf->state = IMX_MEDIA_BUF_STATUS_QUEUED;
+	buf->seq = ring->last_seq++;
+
+	return buf;
+}

Is this a whole software buffer queue implementation? I thought the whole point of putting the custom mem2mem framework into the capture driver was to use the hardware FSU channel linking?

see below.

Quoted 1 line

What is the purpose of this if the sink should be triggered by the FSU?

Ok, here is where I need to make an admission.

The only FSU links I have attempted (and which currently have entries in the fsu_link_info[] table), are the enc/vf/pp --> IRT links for rotation.

There does not appear to be support in the FSU for linking a write channel to the VDIC read channels (8, 9, 10) according to VDI_SRC_SEL field. There is support for the direct link from CSI (which I am using), but that's not an
IDMAC channel link.

There is a PRP_SRC_SEL field, with linking from IDMAC (SMFC) channels 0..2 (and 3? it's not clear, and not clear whether this includes channel 1). But I think this links to channel 12, and not to channels 8,9,10 to the VDIC.
Or will it? It's worth experimenting. It would have helped if FSL listed which
IDMAC channels these FSU links correspond to, instead of making us guess at it.

In any event, the docs are not clear enough to implement a real FSU link to the VDIC read channels, if it's even possible. And trying to get programming help from FSL can be difficult, and no coding examples for this link AFAIK.

So I ended resorted to linking to VDIC channels 8,9,10 with a software approach, instead of attempting a hardware FSU link.

The EOF interrupt handler for the SMFC channels informs the VDIC entity via a v4l2_subdev_ioctl() call that a buffer is available. The VDIC then manually kicks off its read channels to bring that buffer (and a previous buffer for F(n-1) field) into the VDIC.

There is a small amount of extra overhead going this route compared to a FSU hardware link: there is the EOF irq latency (a few usec), and the CPU overhead for the VDIC to manually start the read channels, which is also a few usec at most (see prepare_vdi_in_buffers() in imx-vdic.c). So in total at most ~10 usec of extra overhead (CPU use plus irq latency) under normal system load.

Of course, in order to implement this software link, I had to implement a straightforward FIFO dma buffer ring. The sink (VDIC) allocates the ring at stream on, and the source requests a pointer to this ring in its own stream on. Passing buffers from source to sink then follows a straightforward
FIFO queue/done/dequeue/queue model: sink queues buffers to src, src grabs queued buffers and makes them active, src signals completed buffers to sink, sink dequeues buffers in response, and sink queues buffers back when it is finished with them.

Quoted 43 lines

[...]

Quoted 42 lines
drivers/staging/media/imx/imx-media-common.c new file +42 -0
+/*
+ * The subdevs have to be powered on/off, and streaming
+ * enabled/disabled, in a specific sequence.
+ */
+static const u32 stream_on_seq[] = {
+	IMX_MEDIA_GRP_ID_IC_PP,
+	IMX_MEDIA_GRP_ID_IC_PRPVF,
+	IMX_MEDIA_GRP_ID_IC_PRPENC,
+	IMX_MEDIA_GRP_ID_SMFC,
+	IMX_MEDIA_GRP_ID_SENSOR,
+	IMX_MEDIA_GRP_ID_CSI2,
+	IMX_MEDIA_GRP_ID_VIDMUX,
+	IMX_MEDIA_GRP_ID_CSI,
+};
+
+static const u32 stream_off_seq[] = {
+	IMX_MEDIA_GRP_ID_IC_PP,
+	IMX_MEDIA_GRP_ID_IC_PRPVF,
+	IMX_MEDIA_GRP_ID_IC_PRPENC,
+	IMX_MEDIA_GRP_ID_SMFC,
+	IMX_MEDIA_GRP_ID_CSI,
+	IMX_MEDIA_GRP_ID_VIDMUX,
+	IMX_MEDIA_GRP_ID_CSI2,
+	IMX_MEDIA_GRP_ID_SENSOR,
+};
+
+#define NUM_STREAM_ENTITIES ARRAY_SIZE(stream_on_seq)
+
+static const u32 power_on_seq[] = {
+	IMX_MEDIA_GRP_ID_CSI2,
+	IMX_MEDIA_GRP_ID_SENSOR,
+	IMX_MEDIA_GRP_ID_VIDMUX,
+	IMX_MEDIA_GRP_ID_CSI,
+	IMX_MEDIA_GRP_ID_SMFC,
+	IMX_MEDIA_GRP_ID_IC_PRPENC,
+	IMX_MEDIA_GRP_ID_IC_PRPVF,
+	IMX_MEDIA_GRP_ID_IC_PP,
+};
+
+static const u32 power_off_seq[] = {
+	IMX_MEDIA_GRP_ID_IC_PP,
+	IMX_MEDIA_GRP_ID_IC_PRPVF,
Quoted 8 lines

drivers/staging/media/imx/imx-media-common.c line 829

Quoted 7 lines
drivers/staging/media/imx/imx-media-common.c new file +7 -0
+	IMX_MEDIA_GRP_ID_IC_PRPENC,
+	IMX_MEDIA_GRP_ID_SMFC,
+	IMX_MEDIA_GRP_ID_CSI,
+	IMX_MEDIA_GRP_ID_VIDMUX,
+	IMX_MEDIA_GRP_ID_SENSOR,
+	IMX_MEDIA_GRP_ID_CSI2,
+};

This seems somewhat arbitrary. Why is a power sequence needed?

The CSI-2 receiver must be powered up before the sensor, that's the only requirement IIRC. The others have no s_power requirement. So I can probably change this to power up in the frontend -> backend order (IC_PP to sensor). And vice-versa for power off.

Quoted 23 lines

[...]

Quoted 22 lines
drivers/staging/media/imx/imx-media-common.c new file +22 -0
+/*
+ * Turn current pipeline power on/off starting from start_entity.
+ * Must be called with mdev->graph_mutex held.
+ */
+int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
+				 struct media_entity_graph *graph,
+				 struct media_entity *start_entity, bool on)
+{
+	struct media_entity *entity;
+	struct v4l2_subdev *sd;
+	int i, ret = 0;
+	u32 id;
+
+	for (i = 0; i < NUM_POWER_ENTITIES; i++) {
+		id = on ? power_on_seq[i] : power_off_seq[i];
+		entity = find_pipeline_entity(imxmd, graph, start_entity, id);
+		if (!entity)
+			continue;
+
+		sd = media_entity_to_v4l2_subdev(entity);
+
+		ret = v4l2_subdev_call(sd, core, s_power, on);
Quoted 8 lines

drivers/staging/media/imx/imx-media-common.c line 936

Quoted 7 lines
drivers/staging/media/imx/imx-media-common.c new file +7 -0
+		if (ret && ret != -ENOIOCTLCMD)
+			break;
+	}
+
+	return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
+}
+EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);

This should really be handled by v4l2_pipeline_pm_use.

I thought about this earlier, but v4l2_pipeline_pm_use() seems to be doing some other stuff that bothered me, at least that's what I remember. I will revisit this.

Quoted 42 lines
Quoted 42 lines
drivers/staging/media/imx/imx-media-common.c new file +42 -0
+/*
+ * Inherit the v4l2 controls from all entities in a pipeline
+ * to the given video device.
+ * Must be called with mdev->graph_mutex held.
+ */
+int imx_media_inherit_controls(struct imx_media_dev *imxmd,
+			       struct video_device *vfd,
+			       struct media_entity *start_entity)
+{
+	struct media_entity_graph graph;
+	struct media_entity *entity;
+	struct v4l2_subdev *sd;
+	int ret;
+
+	ret = media_entity_graph_walk_init(&graph, &imxmd->md);
+	if (ret)
+		return ret;
+
+	media_entity_graph_walk_start(&graph, start_entity);
+
+	while ((entity = media_entity_graph_walk_next(&graph))) {
+		if (is_media_entity_v4l2_video_device(entity))
+			continue;
+
+		sd = media_entity_to_v4l2_subdev(entity);
+
+		dev_dbg(imxmd->dev, "%s: adding controls from %s\n",
+			__func__, sd->name);
+
+		ret = v4l2_ctrl_add_handler(vfd->ctrl_handler,
+					    sd->ctrl_handler,
+					    NULL);
+		if (ret)
+			break;
+	}
+
+	media_entity_graph_walk_cleanup(&graph);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(imx_media_inherit_controls);
+
+MODULE_DESCRIPTION("i.MX5/6 v4l2 media controller driver");
Quoted 8 lines

drivers/staging/media/imx/imx-media-dev.c

Quoted 7 lines
(quoted diff) +2 -0
+MODULE_AUTHOR("Steve Longerbeam<[email protected]>");
+MODULE_LICENSE("GPL");
drivers/staging/media/imx/imx-media-dev.c new file +0 -0
new file mode 100644
index 0000000..357654d

This file is full of code that should live in the v4l2 core.

Agreed. Stuff like imx_media_inherit_controls() really should be widely available.

Quoted 7 lines

drivers/staging/media/imx/imx-media-dev.c

Quoted 6 lines
drivers/staging/media/imx/imx-media-dev.c new file +6 -0
+int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
+				   struct imx_media_subdev *csi[4])
+{
+	int ret;
+
+	/* there must be at least one CSI in first IPU */

Why?

Well yeah, imx-media doesn't necessarily need a CSI if things like the VDIC or post-processor are being used by an output overlay pipeline, for example. I'll fix this.

Quoted 30 lines
Quoted 30 lines
drivers/staging/media/imx/imx-media-dev.c new file +30 -0
+
+/* parse inputs property from a sensor node */
+static void of_parse_sensor_inputs(struct imx_media_dev *imxmd,
+				   struct imx_media_subdev *sensor,
+				   struct device_node *sensor_np)
+{
+	struct imx_media_sensor_input *sinput = &sensor->input;
+	int ret, i;
+
+	for (i = 0; i < IMX_MEDIA_MAX_SENSOR_INPUTS; i++) {
+		const char *input_name;
+		u32 val;
+
+		ret = of_property_read_u32_index(sensor_np, "inputs", i, &val);
+		if (ret)
+			break;
+
+		sinput->value[i] = val;
+
+		ret = of_property_read_string_index(sensor_np, "input-names",
+						    i, &input_name);
+		/*
+		 * if input-names not provided, they will be set using
+		 * the subdev name once the sensor is known during
+		 * async bind
+		 */
+		if (!ret)
+			strncpy(sinput->name[i], input_name,
+				sizeof(sinput->name[i]));
+	}
Quoted 8 lines

drivers/staging/media/imx/imx-media-dev.c

Quoted 7 lines
drivers/staging/media/imx/imx-media-dev.c new file +7 -0
+
+	sinput->num = i;
+
+	/* if no inputs provided just assume a single input */
+	if (sinput->num == 0)
+		sinput->num = 1;
+}

This should be parsed by the sensor driver, not imx-media.

you're probably right. I'll submit a patch for adv7180.c.

Quoted 24 lines
Quoted 24 lines
drivers/staging/media/imx/imx-media-dev.c new file +24 -0
+static void of_parse_sensor(struct imx_media_dev *imxmd,
+			    struct imx_media_subdev *sensor,
+			    struct device_node *sensor_np)
+{
+	struct device_node *endpoint;
+
+	of_parse_sensor_inputs(imxmd, sensor, sensor_np);
+
+	endpoint = of_graph_get_next_endpoint(sensor_np, NULL);
+	if (endpoint) {
+		v4l2_of_parse_endpoint(endpoint, &sensor->sensor_ep);
+		of_node_put(endpoint);
+	}
+}
+
+static int of_get_port_count(const struct device_node *np)
+{
+	struct device_node *child;
+	int num = 0;
+
+	/* if this node is itself a port, return 1 */
+	if (of_node_cmp(np->name, "port") == 0)
+		return 1;
+
Quoted 8 lines

drivers/staging/media/imx/imx-media-dev.c

Quoted 6 lines
drivers/staging/media/imx/imx-media-dev.c new file +6 -0
+	for_each_child_of_node(np, child)
+		if (of_node_cmp(child->name, "port") == 0)
+			num++;
+
+	return num;
+}

If this is extended to handle the ports subnode properly, it could be moved into drivers/of/base.c.

More that eventually needs exporting, agreed.

Steve

maintainer
#65 lore raw

Re: [PATCH v3 16/24] media: Add i.MX media core driver

On Sat, 2017-01-14 at 14:46 -0800, Steve Longerbeam wrote:
[...]

Quoted 10 lines
Quoted 9 lines

Documentation/media/v4l-drivers/imx.rst line 113

Quoted 7 lines
Documentation/media/v4l-drivers/imx.rst new file +7 -0
+Unprocessed Video Capture:
+--------------------------
+
+Send frames directly from sensor to camera interface, with no
+conversions:
+
+-> ipu_smfc -> camif

I'd call this capture interface, this is not just for cameras. Or maybe idmac if you want to mirror hardware names?

Quoted 3 lines

Camif is so named because it is the V4L2 user interface for video capture. I suppose it could be named "capif", but that doesn't role off the tongue quite as well.

Agreed, capif sounds weird. I find camif a bit confusing though, because Samsung S3C has a camera interface that is actually called "CAMIF".

Quoted 13 lines
Quoted 2 lines
Quoted 1 line

+Note the ipu_smfc can do pixel reordering within the same colorspace.

That isn't a feature of the SMFC, but of the IDMAC (FCW & FCR).

yes, the doc is re-worded to make that more clear.

Quoted 7 lines

Documentation/media/v4l-drivers/imx.rst line 117

Quoted 2 lines
Documentation/media/v4l-drivers/imx.rst new file +2 -0
+For example, its sink pad can take UYVY2X8, but its source pad can
+output YUYV2X8.

I don't think this is correct. Re-reading "37.4.3.7 Packing to memory" in the CSI chapter, for 8-bit per component data, the internal format between CSI, SMFC, and IDMAC is always some 32-bit RGBx/YUVx variant (or "bayer/generic data"). In either case, the internal format does not change along the way.

Quoted 1 line

these are pixels in memory buffers, not the IPU internal formats.

As long as we are talking about the CSI -> SMFC -> IDMAC path, these should be IPU internal formats. How else would one choose between 8-bit companded RGB, and 16-bit expanded RGB for a 10-bit per component input signal? This is the same issue as in the next comment.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Documentation/media/v4l-drivers/imx.rst new file +8 -0
+   media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480]"
+   # Configure pads for OV5640 pipeline
+   media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480]"
+   media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480]"

[...]

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

+ media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"

I agree this looks very intuitive, but technically correct for the csi1:1 and camif1:0 pads would be a 32-bit YUV format. (MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).

I think it would be better to use the correct format

I'm not sure I follow you here.

The ov5640 sends UYVY2X8 on the wire, so pads "ov5640_mipi 1-0040":0 up to "ipu1_csi1":0 are correct. But the CSI writes 32-bit YUV values into the SMFC, so the CSI output pad and the IDMAC input pad should have a YUV8_1X32 format.

Chapter 37.4.2.3 "FCW & FCR - Format converter write and read" in the IDMAC chapter states that all internal submodules only work on 8-bit per component formats with four components: YUVA or RGBA.

Quoted 5 lines
Quoted 4 lines

[...]
Is this a whole software buffer queue implementation? I thought the whole point of putting the custom mem2mem framework into the capture driver was to use the hardware FSU channel linking?

Quoted 8 lines

see below.

Quoted 1 line

What is the purpose of this if the sink should be triggered by the FSU?

Ok, here is where I need to make an admission.

The only FSU links I have attempted (and which currently have entries in the fsu_link_info[] table), are the enc/vf/pp --> IRT links for rotation.

Which are not described as media entity links because the rotation units do not have separate media entities. So me arguing against handling mem2mem chaining via media entity links doesn't concern these implicit links.

Quoted 8 lines

There does not appear to be support in the FSU for linking a write channel to the VDIC read channels (8, 9, 10) according to VDI_SRC_SEL field. There is support for the direct link from CSI (which I am using), but that's not an
IDMAC channel link.

There is a PRP_SRC_SEL field, with linking from IDMAC (SMFC) channels 0..2 (and 3? it's not clear, and not clear whether this includes channel 1).

As I read it, that is 0 and 2 only, no idea why. But since there are only 2 CSIs, that shouldn't be a problem.

Quoted 6 lines

But I think this links to channel 12, and not to channels 8,9,10 to the VDIC.
Or will it? It's worth experimenting. It would have helped if FSL listed which
IDMAC channels these FSU links correspond to, instead of making us guess at it.

I would have assumed that the FSU triggering only works on 1:1 channels and the VDIC with its three input channels is different. But then there's the alleged VDOA link to ch8/9/10 ro ch9, depending on VDI_MOT_SEL.

This makes me more convinced that the CSI -> VDIC link should only describe the direct path (real-time mode, single field).

Quoted 13 lines

In any event, the docs are not clear enough to implement a real FSU link to the VDIC read channels, if it's even possible. And trying to get programming help from FSL can be difficult, and no coding examples for this link AFAIK.

So I ended resorted to linking to VDIC channels 8,9,10 with a software approach, instead of attempting a hardware FSU link.

The EOF interrupt handler for the SMFC channels informs the VDIC entity via a v4l2_subdev_ioctl() call that a buffer is available. The VDIC then manually kicks off its read channels to bring that buffer (and a previous buffer for F(n-1) field) into the VDIC.

Quoted 6 lines

There is a small amount of extra overhead going this route compared to a FSU hardware link: there is the EOF irq latency (a few usec), and the CPU overhead for the VDIC to manually start the read channels, which is also a few usec at most (see prepare_vdi_in_buffers() in imx-vdic.c). So in total at most ~10 usec of extra overhead (CPU use plus irq latency) under normal system load.

That the same low overhead could be reached by linking videobuf2 queues of different video devices, that would be a lot more flexible.

Quoted 9 lines

Of course, in order to implement this software link, I had to implement a straightforward FIFO dma buffer ring. The sink (VDIC) allocates the ring at stream on, and the source requests a pointer to this ring in its own stream on. Passing buffers from source to sink then follows a straightforward
FIFO queue/done/dequeue/queue model: sink queues buffers to src, src grabs queued buffers and makes them active, src signals completed buffers to sink, sink dequeues buffers in response, and sink queues buffers back when it is finished with them.

Thank you for the explanation.

[...]

Quoted 12 lines
Quoted 11 lines

drivers/staging/media/imx/imx-media-common.c line 840

Quoted 10 lines
drivers/staging/media/imx/imx-media-common.c new file +10 -0
+static const u32 power_off_seq[] = {
+	IMX_MEDIA_GRP_ID_IC_PP,
+	IMX_MEDIA_GRP_ID_IC_PRPVF,
+	IMX_MEDIA_GRP_ID_IC_PRPENC,
+	IMX_MEDIA_GRP_ID_SMFC,
+	IMX_MEDIA_GRP_ID_CSI,
+	IMX_MEDIA_GRP_ID_VIDMUX,
+	IMX_MEDIA_GRP_ID_SENSOR,
+	IMX_MEDIA_GRP_ID_CSI2,
+};

This seems somewhat arbitrary. Why is a power sequence needed?

Quoted 4 lines

The CSI-2 receiver must be powered up before the sensor, that's the only requirement IIRC. The others have no s_power requirement. So I can probably change this to power up in the frontend -> backend order (IC_PP to sensor). And vice-versa for power off.

Yes, I think that should work (see below).

Quoted 32 lines
Quoted 23 lines

[...]

Quoted 22 lines
drivers/staging/media/imx/imx-media-common.c new file +22 -0
+/*
+ * Turn current pipeline power on/off starting from start_entity.
+ * Must be called with mdev->graph_mutex held.
+ */
+int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
+				 struct media_entity_graph *graph,
+				 struct media_entity *start_entity, bool on)
+{
+	struct media_entity *entity;
+	struct v4l2_subdev *sd;
+	int i, ret = 0;
+	u32 id;
+
+	for (i = 0; i < NUM_POWER_ENTITIES; i++) {
+		id = on ? power_on_seq[i] : power_off_seq[i];
+		entity = find_pipeline_entity(imxmd, graph, start_entity, id);
+		if (!entity)
+			continue;
+
+		sd = media_entity_to_v4l2_subdev(entity);
+
+		ret = v4l2_subdev_call(sd, core, s_power, on);
Quoted 8 lines

drivers/staging/media/imx/imx-media-common.c line 936

Quoted 7 lines
drivers/staging/media/imx/imx-media-common.c new file +7 -0
+		if (ret && ret != -ENOIOCTLCMD)
+			break;
+	}
+
+	return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
+}
+EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);

This should really be handled by v4l2_pipeline_pm_use.

Quoted 3 lines

I thought about this earlier, but v4l2_pipeline_pm_use() seems to be doing some other stuff that bothered me, at least that's what I remember. I will revisit this.

I have used it with a tc358743 -> mipi-csi2 pipeline, it didn't cause any problems. It would be better to reuse and, if necessary, fix the existing infrastructure where available.

Quoted 11 lines
Quoted 7 lines

drivers/staging/media/imx/imx-media-internal-sd.c line 445

Quoted 6 lines
drivers/staging/media/imx/imx-media-internal-sd.c new file +6 -0
+int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
+				   struct imx_media_subdev *csi[4])
+{
+	int ret;
+
+	/* there must be at least one CSI in first IPU */

Why?

Well yeah, imx-media doesn't necessarily need a CSI if things like the VDIC or post-processor are being used by an output overlay pipeline, for example. I'll fix this.

I haven't even thought that far, but there could be boards with only a parallel sensor connected to IPU2 CSI1 and IPU1 disabled for power saving reasons.

regards
Philipp

author maintainer
#66 lore raw

Re: [PATCH v3 16/24] media: Add i.MX media core driver

Quoted 11 lines · Philipp Zabel

On Sat, 2017-01-14 at 14:46 -0800, Steve Longerbeam wrote:
[...]

Quoted 9 lines
Quoted 9 lines

Documentation/media/v4l-drivers/imx.rst line 113

Quoted 7 lines
Documentation/media/v4l-drivers/imx.rst new file +7 -0
+Unprocessed Video Capture:
+--------------------------
+
+Send frames directly from sensor to camera interface, with no
+conversions:
+
+-> ipu_smfc -> camif

I'd call this capture interface, this is not just for cameras. Or maybe idmac if you want to mirror hardware names?

Quoted 5 lines
Quoted 3 lines

Camif is so named because it is the V4L2 user interface for video capture. I suppose it could be named "capif", but that doesn't role off the tongue quite as well.

Agreed, capif sounds weird. I find camif a bit confusing though, because Samsung S3C has a camera interface that is actually called "CAMIF".

how about simply "capture" ?

<snip>

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

+ media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480]"

I agree this looks very intuitive, but technically correct for the csi1:1 and camif1:0 pads would be a 32-bit YUV format. (MEDIA_BUS_FMT_YUV8_1X32_PADLO doesn't exist yet).

I think it would be better to use the correct format I'm not sure I follow you here.

Quoted 8 lines

The ov5640 sends UYVY2X8 on the wire, so pads "ov5640_mipi 1-0040":0 up to "ipu1_csi1":0 are correct. But the CSI writes 32-bit YUV values into the SMFC, so the CSI output pad and the IDMAC input pad should have a YUV8_1X32 format.

Chapter 37.4.2.3 "FCW & FCR - Format converter write and read" in the IDMAC chapter states that all internal submodules only work on 8-bit per component formats with four components: YUVA or RGBA.

Right, the "direct" IPU internal (that do not transfer buffers to/from memory via IDMAC channels) should only allow the IPU internal formats YUVA and RGBA. I get you now.

The "direct" pads now only accept MEDIA_BUS_FMT_AYUV8_1X32 and MEDIA_BUS_FMT_ARGB8888_1X32.

Those pads are:

ipu_csi:1
ipu_vdic:1
ipu_ic_prp:0
ipu_ic_prp:1
ipu_ic_prpenc:0
ipu_ic_prpenc:1
ipu_ic_prpvf:0
ipu_ic_prpvf:1

<snip>

Quoted 11 lines
Quoted 9 lines

There does not appear to be support in the FSU for linking a write channel to the VDIC read channels (8, 9, 10) according to VDI_SRC_SEL field. There is support for the direct link from CSI (which I am using), but that's not an
IDMAC channel link.

There is a PRP_SRC_SEL field, with linking from IDMAC (SMFC) channels 0..2 (and 3? it's not clear, and not clear whether this includes channel 1).

As I read it, that is 0 and 2 only, no idea why. But since there are only 2 CSIs, that shouldn't be a problem.

ipu_csi1 is now transferring on IDMAC/SMFC channel 2 (ipu_csi0 still at channel 0).

<snip>

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

drivers/staging/media/imx/imx-media-common.c line 840

Quoted 10 lines
drivers/staging/media/imx/imx-media-common.c new file +10 -0
+static const u32 power_off_seq[] = {
+	IMX_MEDIA_GRP_ID_IC_PP,
+	IMX_MEDIA_GRP_ID_IC_PRPVF,
+	IMX_MEDIA_GRP_ID_IC_PRPENC,
+	IMX_MEDIA_GRP_ID_SMFC,
+	IMX_MEDIA_GRP_ID_CSI,
+	IMX_MEDIA_GRP_ID_VIDMUX,
+	IMX_MEDIA_GRP_ID_SENSOR,
+	IMX_MEDIA_GRP_ID_CSI2,
+};

This seems somewhat arbitrary. Why is a power sequence needed?

Quoted 5 lines
Quoted 4 lines

The CSI-2 receiver must be powered up before the sensor, that's the only requirement IIRC. The others have no s_power requirement. So I can probably change this to power up in the frontend -> backend order (IC_PP to sensor). And vice-versa for power off.

Yes, I think that should work (see below).

Actually there are problems using this, see below.

Quoted 31 lines
Quoted 31 lines
Quoted 31 lines

[...]

Quoted 19 lines
drivers/staging/media/imx/imx-media-common.c new file +19 -0
+/*
+ * Turn current pipeline power on/off starting from start_entity.
+ * Must be called with mdev->graph_mutex held.
+ */
+int imx_media_pipeline_set_power(struct imx_media_dev *imxmd,
+				 struct media_entity_graph *graph,
+				 struct media_entity *start_entity, bool on)
+{
+	struct media_entity *entity;
+	struct v4l2_subdev *sd;
+	int i, ret = 0;
+	u32 id;
+
+	for (i = 0; i < NUM_POWER_ENTITIES; i++) {
+		id = on ? power_on_seq[i] : power_off_seq[i];
+		entity = find_pipeline_entity(imxmd, graph, start_entity, id);
+		if (!entity)
+			continue;
+

drivers/staging/media/imx/imx-media-common.c line 936

Quoted 10 lines
drivers/staging/media/imx/imx-media-common.c new file +10 -0
+		sd = media_entity_to_v4l2_subdev(entity);
+
+		ret = v4l2_subdev_call(sd, core, s_power, on);
+		if (ret && ret != -ENOIOCTLCMD)
+			break;
+	}
+
+	return (ret && ret != -ENOIOCTLCMD) ? ret : 0;
+}
+EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);

This should really be handled by v4l2_pipeline_pm_use.

Quoted 6 lines
Quoted 3 lines

I thought about this earlier, but v4l2_pipeline_pm_use() seems to be doing some other stuff that bothered me, at least that's what I remember. I will revisit this.

I have used it with a tc358743 -> mipi-csi2 pipeline, it didn't cause any problems. It would be better to reuse and, if necessary, fix the existing infrastructure where available.

I tried this API, by switching to v4l2_pipeline_pm_use() in camif open/release,
and switched to v4l2_pipeline_link_notify() instead of imx_media_link_notify()
in the media driver's media_device_ops.

This API assumes the video device has an open file handle while the media links are being established. This doesn't work for me, I want to be able to establish the links using 'media-ctl -l', and that won't work unless there is an
open file handle on the video capture device node.

Also, I looked into calling v4l2_pipeline_pm_use() during imx_media_link_notify(),
instead of imx_media_pipeline_set_power(). Again there are problems with that.

First, v4l2_pipeline_pm_use() acquires the graph mutex, so it can't be called inside
link_notify which already acquires that lock. The header for this function also
clearly states it should only be called in open/release.

Second, ignoring the above locking issue for a moment, v4l2_pipeline_pm_use()
will call s_power on the sensor _first_, then the mipi csi-2 s_power, when executing
media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the wrong order.
In my version which enforces the correct power on order, the mipi csi-2 s_power
is called first in that link setup, followed by the sensor.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/staging/media/imx/imx-media-internal-sd.c line 445

Quoted 6 lines
drivers/staging/media/imx/imx-media-internal-sd.c new file +6 -0
+int imx_media_add_internal_subdevs(struct imx_media_dev *imxmd,
+				   struct imx_media_subdev *csi[4])
+{
+	int ret;
+
+	/* there must be at least one CSI in first IPU */

Why?

Quoted 6 lines
Quoted 3 lines

Well yeah, imx-media doesn't necessarily need a CSI if things like the VDIC or post-processor are being used by an output overlay pipeline, for example. I'll fix this.

I haven't even thought that far, but there could be boards with only a parallel sensor connected to IPU2 CSI1 and IPU1 disabled for power saving reasons.

done! A very simple change to imx_media_add_internal_subdevs(), and now no CSI's are necessary, but the remaining IPU internal entities are loaded and linked just fine without them, so for example with no CSI's in IPU2, the VDIC entity in IPU2 is still present in the graph and could still be used in the future by a mem2mem device for instance.

Steve

maintainer
#67 lore raw

Re: [PATCH v3 16/24] media: Add i.MX media core driver

Hi Steve,

Quoted 13 lines · Steve Longerbeam
Quoted 11 lines · Philipp Zabel

On Sat, 2017-01-14 at 14:46 -0800, Steve Longerbeam wrote:
[...]

Quoted 9 lines
Quoted 9 lines

Documentation/media/v4l-drivers/imx.rst line 113

Quoted 7 lines
Documentation/media/v4l-drivers/imx.rst new file +7 -0
+Unprocessed Video Capture:
+--------------------------
+
+Send frames directly from sensor to camera interface, with no
+conversions:
+
+-> ipu_smfc -> camif

I'd call this capture interface, this is not just for cameras. Or maybe idmac if you want to mirror hardware names?

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Camif is so named because it is the V4L2 user interface for video capture. I suppose it could be named "capif", but that doesn't role off the tongue quite as well.

Agreed, capif sounds weird. I find camif a bit confusing though, because Samsung S3C has a camera interface that is actually called "CAMIF".

how about simply "capture" ?

That sounds good to me.

[...]

Quoted 13 lines
Quoted 3 lines

Chapter 37.4.2.3 "FCW & FCR - Format converter write and read" in the IDMAC chapter states that all internal submodules only work on 8-bit per component formats with four components: YUVA or RGBA.

Right, the "direct" IPU internal (that do not transfer buffers to/from memory via IDMAC channels) should only allow the IPU internal formats YUVA and RGBA. I get you now.

The "direct" pads now only accept MEDIA_BUS_FMT_AYUV8_1X32 and MEDIA_BUS_FMT_ARGB8888_1X32.

Those pads are:

Quoted 8 lines

ipu_csi:1
ipu_vdic:1
ipu_ic_prp:0
ipu_ic_prp:1
ipu_ic_prpenc:0
ipu_ic_prpenc:1
ipu_ic_prpvf:0
ipu_ic_prpvf:1

Yes, that is what I meant. The csi:1 can then be extended to support additional expanded/packed/raw formats for the SMFC->memory path.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

There does not appear to be support in the FSU for linking a write channel to the VDIC read channels (8, 9, 10) according to VDI_SRC_SEL field. There is support for the direct link from CSI (which I am using), but that's not an
IDMAC channel link.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

There is a PRP_SRC_SEL field, with linking from IDMAC (SMFC) channels 0..2 (and 3? it's not clear, and not clear whether this includes channel 1).

As I read it, that is 0 and 2 only, no idea why. But since there are only 2 CSIs, that shouldn't be a problem.

ipu_csi1 is now transferring on IDMAC/SMFC channel 2 (ipu_csi0 still at channel 0).

Ok.

Quoted 18 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

drivers/staging/media/imx/imx-media-common.c line 840

Quoted 10 lines
drivers/staging/media/imx/imx-media-common.c new file +10 -0
+static const u32 power_off_seq[] = {
+	IMX_MEDIA_GRP_ID_IC_PP,
+	IMX_MEDIA_GRP_ID_IC_PRPVF,
+	IMX_MEDIA_GRP_ID_IC_PRPENC,
+	IMX_MEDIA_GRP_ID_SMFC,
+	IMX_MEDIA_GRP_ID_CSI,
+	IMX_MEDIA_GRP_ID_VIDMUX,
+	IMX_MEDIA_GRP_ID_SENSOR,
+	IMX_MEDIA_GRP_ID_CSI2,
+};

This seems somewhat arbitrary. Why is a power sequence needed?

Quoted 5 lines
Quoted 4 lines

The CSI-2 receiver must be powered up before the sensor, that's the only requirement IIRC. The others have no s_power requirement. So I can probably change this to power up in the frontend -> backend order (IC_PP to sensor). And vice-versa for power off.

Yes, I think that should work (see below).

Actually there are problems using this, see below.

[...]

Quoted 26 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Quoted 1 line

+EXPORT_SYMBOL_GPL(imx_media_pipeline_set_power);

This should really be handled by v4l2_pipeline_pm_use.

I thought about this earlier, but v4l2_pipeline_pm_use() seems to be doing some other stuff that bothered me, at least that's what I remember. I will revisit this.

I have used it with a tc358743 -> mipi-csi2 pipeline, it didn't cause any problems. It would be better to reuse and, if necessary, fix the existing infrastructure where available.

I tried this API, by switching to v4l2_pipeline_pm_use() in camif open/release,
and switched to v4l2_pipeline_link_notify() instead of imx_media_link_notify()
in the media driver's media_device_ops.

This API assumes the video device has an open file handle while the media links are being established. This doesn't work for me, I want to be able to establish the links using 'media-ctl -l', and that won't work unless there is an
open file handle on the video capture device node.

Also, I looked into calling v4l2_pipeline_pm_use() during imx_media_link_notify(),
instead of imx_media_pipeline_set_power(). Again there are problems with that.

Quoted 5 lines

First, v4l2_pipeline_pm_use() acquires the graph mutex, so it can't be called inside
link_notify which already acquires that lock. The header for this function also
clearly states it should only be called in open/release.

So why not call it in open/release then?

Quoted 9 lines

Second, ignoring the above locking issue for a moment, v4l2_pipeline_pm_use()
will call s_power on the sensor _first_, then the mipi csi-2 s_power, when executing
media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the wrong order.
In my version which enforces the correct power on order, the mipi csi-2 s_power
is called first in that link setup, followed by the sensor.

I don't understand why you want to power up subdevs as soon as the links are established. Shouldn't that rather be done for all subdevices in the pipeline when the corresponding capture device is opened?
It seems to me that powering up the pipeline should be the last step before userspace actually starts the capture.

[...]

Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

drivers/staging/media/imx/imx-media-internal-sd.c line 445

Quoted 1 line
drivers/staging/media/imx/imx-media-internal-sd.c new file +1 -0
+	/* there must be at least one CSI in first IPU */

Why?

Well yeah, imx-media doesn't necessarily need a CSI if things like the VDIC or post-processor are being used by an output overlay pipeline, for example. I'll fix this.

I haven't even thought that far, but there could be boards with only a parallel sensor connected to IPU2 CSI1 and IPU1 disabled for power saving reasons.

Quoted 5 lines

done! A very simple change to imx_media_add_internal_subdevs(), and now no CSI's are necessary, but the remaining IPU internal entities are loaded and linked just fine without them, so for example with no CSI's in IPU2, the VDIC entity in IPU2 is still present in the graph and could still be used in the future by a mem2mem device for instance.

Excellent, thanks.

regards
Philipp

author maintainer
#68 lore raw

Re: [PATCH v3 16/24] media: Add i.MX media core driver

Quoted 15 lines · Philipp Zabel

Hi Steve,

Quoted 12 lines · Steve Longerbeam
Quoted 11 lines · Philipp Zabel

On Sat, 2017-01-14 at 14:46 -0800, Steve Longerbeam wrote:
[...]

Quoted 9 lines
Quoted 9 lines

Documentation/media/v4l-drivers/imx.rst line 113

Quoted 7 lines
Documentation/media/v4l-drivers/imx.rst new file +7 -0
+Unprocessed Video Capture:
+--------------------------
+
+Send frames directly from sensor to camera interface, with no
+conversions:
+
+-> ipu_smfc -> camif

I'd call this capture interface, this is not just for cameras. Or maybe idmac if you want to mirror hardware names?

Quoted 7 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

Camif is so named because it is the V4L2 user interface for video capture. I suppose it could be named "capif", but that doesn't role off the tongue quite as well.

Agreed, capif sounds weird. I find camif a bit confusing though, because Samsung S3C has a camera interface that is actually called "CAMIF".

how about simply "capture" ?

That sounds good to me.

done.

Quoted 25 lines
Quoted 25 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

This should really be handled by v4l2_pipeline_pm_use.

I thought about this earlier, but v4l2_pipeline_pm_use() seems to be doing some other stuff that bothered me, at least that's what I remember. I will revisit this.

I have used it with a tc358743 -> mipi-csi2 pipeline, it didn't cause any problems. It would be better to reuse and, if necessary, fix the existing infrastructure where available.

I tried this API, by switching to v4l2_pipeline_pm_use() in camif open/release,
and switched to v4l2_pipeline_link_notify() instead of imx_media_link_notify()
in the media driver's media_device_ops.

This API assumes the video device has an open file handle while the media links are being established. This doesn't work for me, I want to be able to establish the links using 'media-ctl -l', and that won't work unless there is an
open file handle on the video capture device node.

Also, I looked into calling v4l2_pipeline_pm_use() during imx_media_link_notify(),
instead of imx_media_pipeline_set_power(). Again there are problems with that.

Quoted 6 lines
Quoted 5 lines

First, v4l2_pipeline_pm_use() acquires the graph mutex, so it can't be called inside
link_notify which already acquires that lock. The header for this function also
clearly states it should only be called in open/release.

So why not call it in open/release then?

er, see above (?)

Quoted 10 lines
Quoted 10 lines

Second, ignoring the above locking issue for a moment, v4l2_pipeline_pm_use()
will call s_power on the sensor _first_, then the mipi csi-2 s_power, when executing
media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the wrong order.
In my version which enforces the correct power on order, the mipi csi-2 s_power
is called first in that link setup, followed by the sensor.

Quoted 2 lines

I don't understand why you want to power up subdevs as soon as the links are established.

Because that is the precedence, all other media drivers do pipeline power on/off at link_notify. And v4l2_pipeline_link_notify() was written as a link_notify method.

Quoted 2 lines

Shouldn't that rather be done for all subdevices in the
pipeline when the corresponding capture device is opened?

that won't work. There's no guarantee the links will be established at capture device open time.

Quoted 2 lines

It seems to me that powering up the pipeline should be the last step before userspace actually starts the capture.

Well, I'm ok with moving pipeline power on/off to start/stop streaming. I would actually prefer to do it then, I only chose at link_notify because of precedence. I'll look into it.

Steve

#69 lore raw

Re: [PATCH v3 16/24] media: Add i.MX media core driver

Quoted 18 lines · Steve Longerbeam
Quoted 11 lines
Quoted 11 lines

Second, ignoring the above locking issue for a moment, v4l2_pipeline_pm_use()
will call s_power on the sensor _first_, then the mipi csi-2 s_power, when executing
media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the wrong order.
In my version which enforces the correct power on order, the mipi csi-2 s_power
is called first in that link setup, followed by the sensor.

Quoted 2 lines

I don't understand why you want to power up subdevs as soon as the links are established.

Because that is the precedence, all other media drivers do pipeline power on/off at link_notify. And v4l2_pipeline_link_notify() was written as a link_notify method.

Quoted 5 lines
Quoted 2 lines

Shouldn't that rather be done for all subdevices in the
pipeline when the corresponding capture device is opened?

that won't work. There's no guarantee the links will be established at capture device open time.

ugh, maybe v4l2_pipeline_pm_use() would work at open/release. If there are no links yet, it would basically be a no-op. And stream on requires opening the
device, and the pipeline links should be established by then, so this might be
fine, looking into this too.

Steve

Quoted 8 lines
Quoted 3 lines

It seems to me that powering up the pipeline should be the last step before userspace actually starts the capture.

Well, I'm ok with moving pipeline power on/off to start/stop streaming. I would actually prefer to do it then, I only chose at link_notify because
of precedence. I'll look into it.

maintainer
#70 lore raw

Re: [PATCH v3 16/24] media: Add i.MX media core driver

Hi Steve,

Quoted 20 lines · Steve Longerbeam
Quoted 18 lines · Steve Longerbeam
Quoted 11 lines
Quoted 11 lines

Second, ignoring the above locking issue for a moment, v4l2_pipeline_pm_use()
will call s_power on the sensor _first_, then the mipi csi-2 s_power, when executing
media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the wrong order.
In my version which enforces the correct power on order, the mipi csi-2 s_power
is called first in that link setup, followed by the sensor.

Quoted 2 lines

I don't understand why you want to power up subdevs as soon as the links are established.

Because that is the precedence, all other media drivers do pipeline power on/off at link_notify. And v4l2_pipeline_link_notify() was written as a link_notify method.

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

Shouldn't that rather be done for all subdevices in the
pipeline when the corresponding capture device is opened?

that won't work. There's no guarantee the links will be established at capture device open time.

If the device is opened before the links are established, it won't be usable anyway. And I think the connected pipeline should be locked in place while the video device is opened. Is there any reason to ever open the video device and only then start linking entities?

Quoted 6 lines

ugh, maybe v4l2_pipeline_pm_use() would work at open/release. If there are no links yet, it would basically be a no-op. And stream on requires opening the
device, and the pipeline links should be established by then, so this might be
fine, looking into this too.

Thanks for looking into it, at least I had that working for the TC358743->MIPI-CSI2 link in my driver.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

It seems to me that powering up the pipeline should be the last step before userspace actually starts the capture.

Well, I'm ok with moving pipeline power on/off to start/stop streaming. I would actually prefer to do it then, I only chose at link_notify because of precedence. I'll look into it.

That might be too late, though. I would expect STREAMON/STREAMOFF to be a rather fast operation as all the slow preparation could be at open / REQBUFS time. Also, there might be sensors that need to be powered on to handle the v4l2_ctrl passthrough?

regards
Philipp

#71 lore raw

Re: [PATCH v3 16/24] media: Add i.MX media core driver

In reply to Philipp Zabel
Quoted 13 lines · Philipp Zabel

Hi Steve,

Quoted 9 lines · Steve Longerbeam

Second, ignoring the above locking issue for a moment, v4l2_pipeline_pm_use()
will call s_power on the sensor _first_, then the mipi csi-2 s_power, when executing
media-ctl -l '"ov5640 1-003c":0 -> "imx6-mipi-csi2":0[1]'. Which is the wrong order.
In my version which enforces the correct power on order, the mipi csi-2 s_power
is called first in that link setup, followed by the sensor.

Quoted 5 lines

I don't understand why you want to power up subdevs as soon as the links are established. Shouldn't that rather be done for all subdevices in the pipeline when the corresponding capture device is opened?
It seems to me that powering up the pipeline should be the last step before userspace actually starts the capture.

I agree with Philipp here - configuration of the software pipeline shouldn't result in hardware being forced to be powered up. That's more of a decision for the individual sub-driver than for core.

Executing media-ctl to enable a link between two sub-device endpoints should really be a matter of setting the software state, and when the video device is opened for streaming, surely that's when the hardware in the chain between the source and the capture device should be powered up and programmed.

Signature · 3 lines
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
#72 lore raw

Re: [PATCH v3 16/24] media: Add i.MX media core driver

In reply to Steve Longerbeam
Quoted 3 lines · Steve Longerbeam

Add the core media driver for i.MX SOC.

Applying: media: Add i.MX media core driver .git/rebase-apply/patch:516: new blank line at EOF. +
.git/rebase-apply/patch:528: new blank line at EOF. +
.git/rebase-apply/patch:556: new blank line at EOF. +
warning: 3 lines add whitespace errors.

Signature · 3 lines
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
#73 lore raw

Re: [PATCH v3 16/24] media: Add i.MX media core driver

In reply to Steve Longerbeam

drivers/staging/media/imx/imx-media.h line 110

Quoted 3 lines · Steve Longerbeam
drivers/staging/media/imx/imx-media.h new file +3 -0
+struct imx_media_dev {
+	struct media_device md;
+	struct v4l2_device  v4l2_dev;

This is similarly buggy.

struct v4l2_device {
        struct device *dev;
#if defined(CONFIG_MEDIA_CONTROLLER)
        struct media_device *mdev;
#endif
        struct list_head subdevs;
        spinlock_t lock;
        char name[V4L2_DEVICE_NAME_SIZE];
        void (*notify)(struct v4l2_subdev *sd,
                        unsigned int notification, void *arg);
        struct v4l2_ctrl_handler *ctrl_handler;
        struct v4l2_prio_state prio;
        struct kref ref;
        void (*release)(struct v4l2_device *v4l2_dev);
};

Notice the kref and release function. This is the only way the memory backing "struct v4l2_device" may be released. If you wish to embed this structure into another structure, then the lifetime of that other structure is determined by this one. IOW, when this release function is called, only then may you kfree() the memory backing struct imx_media_dev.

drivers/staging/media/imx/imx-media.h line 111

Quoted 1 line
drivers/staging/media/imx/imx-media.h new file +1 -0
+	struct device *dev;

And... do you need all these struct device pointers?

        imxmd->dev = dev;
        imxmd->md.dev = dev;

As media_device already contains a pointer, can't you re-use that?

Signature · 3 lines
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
#74 lore raw

Re: [PATCH v3 16/24] media: Add i.MX media core driver

In reply to Steve Longerbeam
Quoted 38 lines · Steve Longerbeam
drivers/staging/media/imx/imx-media-internal-sd.c new file +38 -0
+/* register an internal subdev as a platform device */
+static struct imx_media_subdev *
+add_internal_subdev(struct imx_media_dev *imxmd,
+		    const struct internal_subdev *isd,
+		    int ipu_id)
+{
+	struct imx_media_internal_sd_platformdata pdata;
+	struct platform_device_info pdevinfo = {0};
+	struct imx_media_subdev *imxsd;
+	struct platform_device *pdev;
+
+	switch (isd->id->grp_id) {
+	case IMX_MEDIA_GRP_ID_CAMIF0...IMX_MEDIA_GRP_ID_CAMIF1:
+		pdata.grp_id = isd->id->grp_id +
+			((2 * ipu_id) << IMX_MEDIA_GRP_ID_CAMIF_BIT);
+		break;
+	default:
+		pdata.grp_id = isd->id->grp_id;
+		break;
+	}
+
+	/* the id of IPU this subdev will control */
+	pdata.ipu_id = ipu_id;
+
+	/* create subdev name */
+	imx_media_grp_id_to_sd_name(pdata.sd_name, sizeof(pdata.sd_name),
+				    pdata.grp_id, ipu_id);
+
+	pdevinfo.name = isd->id->name;
+	pdevinfo.id = ipu_id * num_isd + isd->id->index;
+	pdevinfo.parent = imxmd->dev;
+	pdevinfo.data = &pdata;
+	pdevinfo.size_data = sizeof(pdata);
+	pdevinfo.dma_mask = DMA_BIT_MASK(32);
+
+	pdev = platform_device_register_full(&pdevinfo);
+	if (IS_ERR(pdev))
+		return ERR_CAST(pdev);

drivers/staging/media/imx/imx-media-internal-sd.c line 393

Quoted 10 lines
drivers/staging/media/imx/imx-media-internal-sd.c new file +10 -0
+
+	imxsd = imx_media_add_async_subdev(imxmd, NULL, dev_name(&pdev->dev));
+	if (IS_ERR(imxsd))
+		return imxsd;
+
+	imxsd->num_sink_pads = isd->num_sink_pads;
+	imxsd->num_src_pads = isd->num_src_pads;
+
+	return imxsd;
+}

You seem to create platform devices here, but I see nowhere that you ever remove them - so if you get to the lucky point of being able to rmmod imx-media and then try to re-insert it, you end up with a load of kernel warnings, one for each device created this way, and platform_device_register_full() fails:

WARNING: CPU: 0 PID: 2143 at /home/rmk/git/linux-rmk/fs/sysfs/dir.c:31 sysfs_warn_dup+0x64/0x80
sysfs: cannot create duplicate filename '/devices/soc0/soc/soc:media at 0/imx-ipuv3-smfc.2'
Modules linked in: imx_media(C+) rfcomm bnep bluetooth nfsd imx_camif(C) imx_ic(C) imx_smfc(C) caam_jr snd_soc_imx_sgtl5000 snd_soc_fsl_asoc_card uvcvideo snd_soc_imx_spdif imx_mipi_csi2(C) imx_media_common(C) snd_soc_imx_audmux imx219 snd_soc_sgtl5000 video_multiplexer imx_sdma caam imx2_wdt rc_cec coda v4l2_mem2mem videobuf2_v4l2 snd_soc_fsl_ssi snd_soc_fsl_spdif videobuf2_dma_contig imx_pcm_dma videobuf2_core videobuf2_vmalloc videobuf2_memops imx_thermal dw_hdmi_cec dw_hdmi_ahb_audio etnaviv fuse rc_pinnacle_pctv_hd [last unloaded: imx_media]
CPU: 0 PID: 2143 Comm: modprobe Tainted: G         C      4.10.0-rc6+ #2103
Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
Backtrace:
[<c0013ba4>] (dump_backtrace) from [<c0013de4>] (show_stack+0x18/0x1c)
 r6:60000013 r5:ffffffff r4:00000000 r3:00000000
[<c0013dcc>] (show_stack) from [<c03334e8>] (dump_stack+0xa4/0xdc)
[<c0333444>] (dump_stack) from [<c0033210>] (__warn+0xdc/0x108)
 r6:c08ad998 r5:00000000 r4:cf4e9a78 r3:ec984980
[<c0033134>] (__warn) from [<c00332f4>] (warn_slowpath_fmt+0x40/0x48)
 r10:e5800010 r8:ef1fa818 r7:ef1fa810 r6:ef202300 r5:e9809000 r4:ee868000
[<c00332b8>] (warn_slowpath_fmt) from [<c01f5918>] (sysfs_warn_dup+0x64/0x80)
 r3:ee868000 r2:c08ad9c0
[<c01f58b4>] (sysfs_warn_dup) from [<c01f5a10>] (sysfs_create_dir_ns+0x90/0x98)
 r6:ffffffef r5:ef202300 r4:eb5e3418
[<c01f5980>] (sysfs_create_dir_ns) from [<c03361a4>] (kobject_add_internal+0xa4/0x2d8)
 r6:ef1fa818 r5:00000000 r4:eb5e3418
[<c0336100>] (kobject_add_internal) from [<c033657c>] (kobject_add+0x50/0x98)
 r8:bf1f9018 r7:ef1fa810 r6:ef1fa818 r5:00000000 r4:eb5e3418
[<c0336530>] (kobject_add) from [<c0415fd8>] (device_add+0xc8/0x538)
 r3:ef1fa834 r2:00000000
 r6:eb5e3418 r5:00000000 r4:eb5e3410
[<c0415f10>] (device_add) from [<c041ae24>] (platform_device_add+0xb0/0x214)
 r10:e5800010 r9:00000000 r8:bf1f9018 r7:e5806fd4 r6:eb5e3410 r5:eb5e3400
 r4:00000000
[<c041ad74>] (platform_device_add) from [<c041b7f4>] (platform_device_register_full+0xe8/0x110)
 r7:e5806fd4 r6:00000002 r5:eb5e3400 r4:cf4e9c18
[<c041b70c>] (platform_device_register_full) from [<bf1f7f54>] (add_ipu_internal_subdevs+0x128/0x2c8 [imx_media])
 r5:bf1f9000 r4:00000918
[<bf1f7e2c>] (add_ipu_internal_subdevs [imx_media]) from [<bf1f8120>] (imx_media_add_internal_subdevs+0x2c/0x70 [imx_media])
 r10:00000048 r9:f31ceef8 r8:ef7f1d94 r7:e5800230 r6:e5800200 r5:e5800010
 r4:cf4e9c90
[<bf1f80f4>] (imx_media_add_internal_subdevs [imx_media]) from [<bf1f71ec>] (imx_media_probe+0xc4/0x1c0 [imx_media])
 r5:00000000 r4:e5800010
[<bf1f7128>] (imx_media_probe [imx_media]) from [<c041ac80>] (platform_drv_probe+0x58/0xb8)
 r8:00000000 r7:bf1fbd48 r6:fffffdfb r5:ef1fa810 r4:fffffffe
[<c041ac28>] (platform_drv_probe) from [<c0418c90>] (driver_probe_device+0x204/0x2c8)
 r7:bf1fbd48 r6:00000000 r5:c1419de8 r4:ef1fa810
[<c0418a8c>] (driver_probe_device) from [<c0418e10>] (__driver_attach+0xbc/0xc0)
 r10:00000124 r8:00000001 r7:00000000 r6:ef1fa844 r5:bf1fbd48 r4:ef1fa810
[<c0418d54>] (__driver_attach) from [<c04170b0>] (bus_for_each_dev+0x5c/0x90)
 r6:c0418d54 r5:bf1fbd48 r4:00000000 r3:00000000
[<c0417054>] (bus_for_each_dev) from [<c04184f4>] (driver_attach+0x24/0x28)
 r6:c0a45e90 r5:ed5a8980 r4:bf1fbd48
[<c04184d0>] (driver_attach) from [<c04181f4>] (bus_add_driver+0xf4/0x200)
[<c0418100>] (bus_add_driver) from [<c0419c90>] (driver_register+0x80/0xfc)
 r7:00000000 r6:bf1fe000 r5:c0a70528 r4:bf1fbd48
[<c0419c10>] (driver_register) from [<c041ab54>] (__platform_driver_register+0x48/0x4c)
 r5:c0a70528 r4:bf1fbdc0
[<c041ab0c>] (__platform_driver_register) from [<bf1fe018>] (imx_media_pdrv_init+0x18/0x24 [imx_media])
[<bf1fe000>] (imx_media_pdrv_init [imx_media]) from [<c00098ac>] (do_one_initcall+0x44/0x170)
[<c0009868>] (do_one_initcall) from [<c011b090>] (do_init_module+0x68/0x1d8)
 r8:00000001 r7:bf1fbdc0 r6:e9809540 r5:c0a70528 r4:bf1fbdc0
[<c011b028>] (do_init_module) from [<c00d2d44>] (load_module+0x195c/0x2080)
 r7:bf1fbdc0 r6:c09e04ec r5:c0a70528 r4:c09f5f27
[<c00d13e8>] (load_module) from [<c00d3640>] (SyS_finit_module+0x94/0xa0)
 r10:00000000 r9:cf4e8000 r8:7f6b2398 r7:00000003 r6:00000000 r5:00000000
 r4:7fffffff
[<c00d35ac>] (SyS_finit_module) from [<c000fd60>] (ret_fast_syscall+0x0/0x1c)
 r8:c000ff04 r7:0000017b r6:80c95148 r5:80c95b10 r4:80c95370
---[ end trace 05abce0bbb26bc34 ]---

imx-media: add_internal_subdevs failed with -17
imx-media: probe of soc:media at 0 failed with error -17

Signature · 3 lines
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
author maintainer
#75 lore raw

Re: [PATCH v3 16/24] media: Add i.MX media core driver

Quoted 46 lines · Russell King - ARM Linux
Quoted 45 lines · Steve Longerbeam
drivers/staging/media/imx/imx-media-internal-sd.c new file +45 -0
+/* register an internal subdev as a platform device */
+static struct imx_media_subdev *
+add_internal_subdev(struct imx_media_dev *imxmd,
+		    const struct internal_subdev *isd,
+		    int ipu_id)
+{
+	struct imx_media_internal_sd_platformdata pdata;
+	struct platform_device_info pdevinfo = {0};
+	struct imx_media_subdev *imxsd;
+	struct platform_device *pdev;
+
+	switch (isd->id->grp_id) {
+	case IMX_MEDIA_GRP_ID_CAMIF0...IMX_MEDIA_GRP_ID_CAMIF1:
+		pdata.grp_id = isd->id->grp_id +
+			((2 * ipu_id) << IMX_MEDIA_GRP_ID_CAMIF_BIT);
+		break;
+	default:
+		pdata.grp_id = isd->id->grp_id;
+		break;
+	}
+
+	/* the id of IPU this subdev will control */
+	pdata.ipu_id = ipu_id;
+
+	/* create subdev name */
+	imx_media_grp_id_to_sd_name(pdata.sd_name, sizeof(pdata.sd_name),
+				    pdata.grp_id, ipu_id);
+
+	pdevinfo.name = isd->id->name;
+	pdevinfo.id = ipu_id * num_isd + isd->id->index;
+	pdevinfo.parent = imxmd->dev;
+	pdevinfo.data = &pdata;
+	pdevinfo.size_data = sizeof(pdata);
+	pdevinfo.dma_mask = DMA_BIT_MASK(32);
+
+	pdev = platform_device_register_full(&pdevinfo);
+	if (IS_ERR(pdev))
+		return ERR_CAST(pdev);
+
+	imxsd = imx_media_add_async_subdev(imxmd, NULL, dev_name(&pdev->dev));
+	if (IS_ERR(imxsd))
+		return imxsd;
+
+	imxsd->num_sink_pads = isd->num_sink_pads;
+	imxsd->num_src_pads = isd->num_src_pads;
Quoted 8 lines

drivers/staging/media/imx/imx-media-common.c line 610

Quoted 3 lines
drivers/staging/media/imx/imx-media-common.c new file +3 -0
+
+	return imxsd;
+}

You seem to create platform devices here, but I see nowhere that you ever remove them - so if you get to the lucky point of being able to rmmod imx-media and then try to re-insert it, you end up with a load of kernel warnings, one for each device created this way, and platform_device_register_full() fails:

Right, I never free the platform devices for the internal subdevs. Fixed.

Steve

author maintainer
#76 lore raw

PATCH v3 17/24 media: imx: Add CSI subdev driver

In reply to Steve Longerbeam

This is a media entity subdevice for the i.MX Camera Serial Interface module.

---
 drivers/staging/media/imx/Kconfig   |  13 +
 drivers/staging/media/imx/Makefile  |   2 +
 drivers/staging/media/imx/imx-csi.c | 644 ++++++++++++++++++++++++++++++++++++
 3 files changed, 659 insertions(+)
 create mode 100644 drivers/staging/media/imx/imx-csi.c
drivers/staging/media/imx/Kconfig +13 -0
index bfde58d..ce2d2c8 100644
@@ -6,3 +6,16 @@ config VIDEO_IMX_MEDIA
 	  Say yes here to enable support for video4linux media controller
 	  driver for the i.MX5/6 SOC.
 
+if VIDEO_IMX_MEDIA
+menu "i.MX5/6 Media Sub devices"
+
+config VIDEO_IMX_CAMERA
+	tristate "i.MX5/6 Camera driver"
+	depends on VIDEO_IMX_MEDIA && VIDEO_DEV && I2C
+	select VIDEOBUF2_DMA_CONTIG
+	default y
+	---help---
+	  A video4linux camera capture driver for i.MX5/6.
+
+endmenu
+endif
drivers/staging/media/imx/Makefile +2 -0
index ef9f11b..133672a 100644
@@ -4,3 +4,5 @@ imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
 
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
+
drivers/staging/media/imx/imx-csi.c new file +644 -0

644 lines not on this page · raw message

Signature · 1 line
2.7.4
maintainer
#77 lore raw

Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver

Quoted 2 lines · Steve Longerbeam

This is a media entity subdevice for the i.MX Camera Serial Interface module.

s/Serial/Sensor/

Quoted 13 lines
---
 drivers/staging/media/imx/Kconfig   |  13 +
 drivers/staging/media/imx/Makefile  |   2 +
 drivers/staging/media/imx/imx-csi.c | 644 ++++++++++++++++++++++++++++++++++++
 3 files changed, 659 insertions(+)
 create mode 100644 drivers/staging/media/imx/imx-csi.c
drivers/staging/media/imx/Kconfig +0 -0
index bfde58d..ce2d2c8 100644

drivers/staging/media/imx/Kconfig config VIDEO_IMX_MEDIA · line 12

Quoted 7 lines
drivers/staging/media/imx/Kconfig +4 -0
@@ -6,3 +6,16 @@ config VIDEO_IMX_MEDIA
 	  Say yes here to enable support for video4linux media controller
 	  driver for the i.MX5/6 SOC.
 
+if VIDEO_IMX_MEDIA
+menu "i.MX5/6 Media Sub devices"
+
+config VIDEO_IMX_CAMERA

s/CAMERA/CSI/ ?

drivers/staging/media/imx/Kconfig config VIDEO_IMX_MEDIA · line 13

Quoted 1 line
drivers/staging/media/imx/Kconfig +1 -0
+	tristate "i.MX5/6 Camera driver"

i.MX5/6 Camera Sensor Interface driver

Quoted 56 lines
drivers/staging/media/imx/Kconfig +8 -0
+	depends on VIDEO_IMX_MEDIA && VIDEO_DEV && I2C
+	select VIDEOBUF2_DMA_CONTIG
+	default y
+	---help---
+	  A video4linux camera capture driver for i.MX5/6.
+
+endmenu
+endif
drivers/staging/media/imx/Makefile +2 -0
index ef9f11b..133672a 100644
@@ -4,3 +4,5 @@ imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
 
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
+
drivers/staging/media/imx/imx-csi.c new file +32 -0
new file mode 100644
index 0000000..64ef862
@@ -0,0 +1,644 @@
+/*
+ * V4L2 Capture CSI Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <video/imx-ipu-v3.h>
+#include "imx-media.h"
+
+#define CSI_NUM_PADS 2
+
+struct csi_priv {
+	struct device *dev;
+	struct ipu_soc *ipu;
+	struct imx_media_dev *md;
+	struct v4l2_subdev sd;
+	struct media_pad pad[CSI_NUM_PADS];
+	struct v4l2_mbus_framefmt format_mbus[CSI_NUM_PADS];
+	struct v4l2_mbus_config sensor_mbus_cfg;
+	struct v4l2_rect crop;
+	struct ipu_csi *csi;

drivers/staging/media/imx/imx-csi.c line 42

Quoted 10 lines
drivers/staging/media/imx/imx-csi.c new file +10 -0
+	int csi_id;
+	int input_pad;
+	int output_pad;
+	bool power_on;  /* power is on */
+	bool stream_on; /* streaming is on */
+
+	/* the sink for the captured frames */
+	struct v4l2_subdev *sink_sd;
+	enum ipu_csi_dest dest;
+	struct v4l2_subdev *src_sd;

src_sd is not used except that its presence marks an enabled input link. -> could be changed to bool.

Quoted 17 lines
drivers/staging/media/imx/imx-csi.c new file +17 -0
+	struct v4l2_ctrl_handler ctrl_hdlr;
+	struct imx_media_fim *fim;
+
+	/* the attached sensor at stream on */
+	struct imx_media_subdev *sensor;
+};
+
+static inline struct csi_priv *sd_to_dev(struct v4l2_subdev *sdev)
+{
+	return container_of(sdev, struct csi_priv, sd);
+}
+
+/* Update the CSI whole sensor and active windows */
+static int csi_setup(struct csi_priv *priv)
+{
+	struct v4l2_mbus_framefmt infmt;
+

drivers/staging/media/imx/imx-csi.c line 70

Quoted 10 lines
drivers/staging/media/imx/imx-csi.c new file +10 -0
+	ipu_csi_set_window(priv->csi, &priv->crop);
+
+	/*
+	 * the ipu-csi doesn't understand ALTERNATE, but it only
+	 * needs to know whether the stream is interlaced, so set
+	 * to INTERLACED if infmt field is ALTERNATE.
+	 */
+	infmt = priv->format_mbus[priv->input_pad];
+	if (infmt.field == V4L2_FIELD_ALTERNATE)
+		infmt.field = V4L2_FIELD_INTERLACED;

That should be SEQ_TB/BT depending on video standard.

Quoted 51 lines
drivers/staging/media/imx/imx-csi.c new file +51 -0
+	ipu_csi_init_interface(priv->csi, &priv->sensor_mbus_cfg, &infmt);
+
+	ipu_csi_set_dest(priv->csi, priv->dest);
+
+	ipu_csi_dump(priv->csi);
+
+	return 0;
+}
+
+static int csi_start(struct csi_priv *priv)
+{
+	int ret;
+
+	if (!priv->sensor) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return -EINVAL;
+	}
+
+	ret = csi_setup(priv);
+	if (ret)
+		return ret;
+
+	/* start the frame interval monitor */
+	if (priv->fim) {
+		ret = imx_media_fim_set_stream(priv->fim, priv->sensor, true);
+		if (ret)
+			return ret;
+	}
+
+	ret = ipu_csi_enable(priv->csi);
+	if (ret) {
+		v4l2_err(&priv->sd, "CSI enable error: %d\n", ret);
+		goto fim_off;
+	}
+
+	return 0;
+
+fim_off:
+	if (priv->fim)
+		imx_media_fim_set_stream(priv->fim, priv->sensor, false);
+	return ret;
+}
+
+static void csi_stop(struct csi_priv *priv)
+{
+	/* stop the frame interval monitor */
+	if (priv->fim)
+		imx_media_fim_set_stream(priv->fim, priv->sensor, false);
+
+	ipu_csi_disable(priv->csi);
+}

drivers/staging/media/imx/imx-csi.c line 132

Quoted 10 lines
drivers/staging/media/imx/imx-csi.c new file +10 -0
+
+static int csi_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	int ret = 0;
+
+	if (!priv->src_sd || !priv->sink_sd)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");

These could be silenced a bit.

[...]

Quoted 4 lines
drivers/staging/media/imx/imx-csi.c new file +4 -0
+static int csi_s_power(struct v4l2_subdev *sd, int on)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	int ret = 0;

drivers/staging/media/imx/imx-csi.c line 157

Quoted 10 lines
drivers/staging/media/imx/imx-csi.c new file +10 -0
+
+	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
+
+	if (priv->fim && on != priv->power_on)
+		ret = imx_media_fim_set_power(priv->fim, on);
+
+	if (!ret)
+		priv->power_on = on;
+	return ret;
+}

Is this called multiple times? I'd expect a poweron during open and a poweroff during close, so no need for priv->power_on.

Quoted 30 lines
drivers/staging/media/imx/imx-csi.c new file +30 -0
+static int csi_link_setup(struct media_entity *entity,
+			  const struct media_pad *local,
+			  const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_subdev *remote_sd;
+
+	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SINK) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->src_sd)
+				return -EBUSY;
+			priv->src_sd = remote_sd;
+		} else {
+			priv->src_sd = NULL;
+		}
+
+		return 0;
+	}
+
+	if (flags & MEDIA_LNK_FL_ENABLED) {
+		if (priv->sink_sd)
+			return -EBUSY;
+		priv->sink_sd = remote_sd;
+	} else {

drivers/staging/media/imx/imx-csi.c line 198

Quoted 10 lines
drivers/staging/media/imx/imx-csi.c new file +10 -0
+		priv->sink_sd = NULL;
+		return 0;
+	}
+
+	/* set CSI destination */
+	switch (remote_sd->grp_id) {
+	case IMX_MEDIA_GRP_ID_SMFC0:
+	case IMX_MEDIA_GRP_ID_SMFC1:
+	case IMX_MEDIA_GRP_ID_SMFC2:
+	case IMX_MEDIA_GRP_ID_SMFC3:

With removal of the SMFC entities, CSI0 could be fixed to SMFC0 and CSI1 to the SMFC2 channel.

[...]

Quoted 23 lines
drivers/staging/media/imx/imx-csi.c new file +23 -0
+static int csi_set_fmt(struct v4l2_subdev *sd,
+		       struct v4l2_subdev_pad_config *cfg,
+		       struct v4l2_subdev_format *sdformat)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	struct v4l2_rect crop;
+	int ret;
+
+	if (sdformat->pad >= CSI_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	if (sdformat->pad == priv->output_pad) {
+		sdformat->format.code = infmt->code;
+		sdformat->format.field = infmt->field;
+		crop.left = priv->crop.left;
+		crop.top = priv->crop.top;

drivers/staging/media/imx/imx-csi.c line 381

Quoted 10 lines
drivers/staging/media/imx/imx-csi.c new file +10 -0
+		crop.width = sdformat->format.width;
+		crop.height = sdformat->format.height;
+		ret = csi_try_crop(priv, &crop);
+		if (ret)
+			return ret;
+		sdformat->format.width = crop.width;
+		sdformat->format.height = crop.height;
+	}
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {

Should there be some limitations on the format here?

regards
Philipp

author maintainer
#78 lore raw

Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver

Quoted 4 lines · Philipp Zabel
Quoted 2 lines · Steve Longerbeam

This is a media entity subdevice for the i.MX Camera Serial Interface module.

s/Serial/Sensor/

done.

Quoted 16 lines
Quoted 16 lines
---
  drivers/staging/media/imx/Kconfig   |  13 +
  drivers/staging/media/imx/Makefile  |   2 +
  drivers/staging/media/imx/imx-csi.c | 644 ++++++++++++++++++++++++++++++++++++
  3 files changed, 659 insertions(+)

create mode 100644 drivers/staging/media/imx/imx-csi.c

drivers/staging/media/imx/Kconfig +0 -0
index bfde58d..ce2d2c8 100644
@@ -6,3 +6,16 @@ config VIDEO_IMX_MEDIA
  	  Say yes here to enable support for video4linux media controller
  	  driver for the i.MX5/6 SOC.
 
Quoted 5 lines

drivers/staging/media/imx/Kconfig config VIDEO_IMX_MEDIA · line 12

Quoted 4 lines
drivers/staging/media/imx/Kconfig +4 -0
+if VIDEO_IMX_MEDIA
+menu "i.MX5/6 Media Sub devices"
+
+config VIDEO_IMX_CAMERA

s/CAMERA/CSI/ ?

done.

Quoted 2 lines

drivers/staging/media/imx/Kconfig config VIDEO_IMX_MEDIA · line 13

Quoted 1 line
drivers/staging/media/imx/Kconfig +1 -0
+	tristate "i.MX5/6 Camera driver"

i.MX5/6 Camera Sensor Interface driver

done.

Quoted 16 lines
Quoted 16 lines
drivers/staging/media/imx/Kconfig +15 -0
+
+struct csi_priv {
+	struct device *dev;
+	struct ipu_soc *ipu;
+	struct imx_media_dev *md;
+	struct v4l2_subdev sd;
+	struct media_pad pad[CSI_NUM_PADS];
+	struct v4l2_mbus_framefmt format_mbus[CSI_NUM_PADS];
+	struct v4l2_mbus_config sensor_mbus_cfg;
+	struct v4l2_rect crop;
+	struct ipu_csi *csi;
+	int csi_id;
+	int input_pad;
+	int output_pad;
+	bool power_on;  /* power is on */
Quoted 8 lines

drivers/staging/media/imx/Kconfig

Quoted 6 lines
drivers/staging/media/imx/Kconfig +6 -0
+	bool stream_on; /* streaming is on */
+
+	/* the sink for the captured frames */
+	struct v4l2_subdev *sink_sd;
+	enum ipu_csi_dest dest;
+	struct v4l2_subdev *src_sd;

src_sd is not used except that its presence marks an enabled input link. -> could be changed to bool.

For now I prefer to keep it a pointer to the src/sink subdevs. At some point the CSI may have some reason to know the identity of the source.

Quoted 21 lines
Quoted 21 lines
drivers/staging/media/imx/Kconfig +20 -0
+	struct v4l2_ctrl_handler ctrl_hdlr;
+	struct imx_media_fim *fim;
+
+	/* the attached sensor at stream on */
+	struct imx_media_subdev *sensor;
+};
+
+static inline struct csi_priv *sd_to_dev(struct v4l2_subdev *sdev)
+{
+	return container_of(sdev, struct csi_priv, sd);
+}
+
+/* Update the CSI whole sensor and active windows */
+static int csi_setup(struct csi_priv *priv)
+{
+	struct v4l2_mbus_framefmt infmt;
+
+	ipu_csi_set_window(priv->csi, &priv->crop);
+
+	/*
Quoted 8 lines

drivers/staging/media/imx/Kconfig

Quoted 7 lines
drivers/staging/media/imx/Kconfig +7 -0
+	 * the ipu-csi doesn't understand ALTERNATE, but it only
+	 * needs to know whether the stream is interlaced, so set
+	 * to INTERLACED if infmt field is ALTERNATE.
+	 */
+	infmt = priv->format_mbus[priv->input_pad];
+	if (infmt.field == V4L2_FIELD_ALTERNATE)
+		infmt.field = V4L2_FIELD_INTERLACED;

That should be SEQ_TB/BT depending on video standard.

fixed.

Quoted 11 lines

drivers/staging/media/imx/Kconfig

Quoted 10 lines
drivers/staging/media/imx/Kconfig +10 -0
+
+static int csi_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	int ret = 0;
+
+	if (!priv->src_sd || !priv->sink_sd)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");

These could be silenced a bit.

yeah, I think it is time for that. I've silenced all the v4l2_info()'s for stream on/off, power on/off, as well as some others.

Quoted 17 lines
Quoted 5 lines
drivers/staging/media/imx/Kconfig +4 -0
+static int csi_s_power(struct v4l2_subdev *sd, int on)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	int ret = 0;

drivers/staging/media/imx/Kconfig

Quoted 10 lines
drivers/staging/media/imx/Kconfig +10 -0
+
+	v4l2_info(sd, "power %s\n", on ? "ON" : "OFF");
+
+	if (priv->fim && on != priv->power_on)
+		ret = imx_media_fim_set_power(priv->fim, on);
+
+	if (!ret)
+		priv->power_on = on;
+	return ret;
+}

Is this called multiple times? I'd expect a poweron during open and a poweroff during close, so no need for priv->power_on.

It is actually called multiple times. The s_power subdev callbacks are made every time there is a new link established, in the imx-media core's link_notify(), in order to re-establish power to the active subdevs in the new pipeline.

This might change after I look into using v4l2_pipeline_pm_use().

Quoted 35 lines
Quoted 35 lines
drivers/staging/media/imx/Kconfig +34 -0
+static int csi_link_setup(struct media_entity *entity,
+			  const struct media_pad *local,
+			  const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_subdev *remote_sd;
+
+	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SINK) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->src_sd)
+				return -EBUSY;
+			priv->src_sd = remote_sd;
+		} else {
+			priv->src_sd = NULL;
+		}
+
+		return 0;
+	}
+
+	if (flags & MEDIA_LNK_FL_ENABLED) {
+		if (priv->sink_sd)
+			return -EBUSY;
+		priv->sink_sd = remote_sd;
+	} else {
+		priv->sink_sd = NULL;
+		return 0;
+	}
+
Quoted 8 lines

drivers/staging/media/imx/Kconfig

Quoted 6 lines
drivers/staging/media/imx/Kconfig +6 -0
+	/* set CSI destination */
+	switch (remote_sd->grp_id) {
+	case IMX_MEDIA_GRP_ID_SMFC0:
+	case IMX_MEDIA_GRP_ID_SMFC1:
+	case IMX_MEDIA_GRP_ID_SMFC2:
+	case IMX_MEDIA_GRP_ID_SMFC3:

With removal of the SMFC entities, CSI0 could be fixed to SMFC0 and CSI1 to the SMFC2 channel.

right, I'll do that.

Quoted 28 lines

[...]

Quoted 26 lines
drivers/staging/media/imx/Kconfig +26 -0
+static int csi_set_fmt(struct v4l2_subdev *sd,
+		       struct v4l2_subdev_pad_config *cfg,
+		       struct v4l2_subdev_format *sdformat)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	struct v4l2_rect crop;
+	int ret;
+
+	if (sdformat->pad >= CSI_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	if (sdformat->pad == priv->output_pad) {
+		sdformat->format.code = infmt->code;
+		sdformat->format.field = infmt->field;
+		crop.left = priv->crop.left;
+		crop.top = priv->crop.top;
+		crop.width = sdformat->format.width;
+		crop.height = sdformat->format.height;
+		ret = csi_try_crop(priv, &crop);
Quoted 8 lines

drivers/staging/media/imx/Kconfig

Quoted 7 lines
drivers/staging/media/imx/Kconfig +7 -0
+		if (ret)
+			return ret;
+		sdformat->format.width = crop.width;
+		sdformat->format.height = crop.height;
+	}
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {

Should there be some limitations on the format here?

done, I've added call to v4l_bound_align_image(), passing it the min/max frame sizes. CSI's sensor/active frame size register fields are 12 bits, so max is 4096 for both width and height.

Steve

#79 lore raw

Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver

In reply to Steve Longerbeam
Quoted 4 lines · Steve Longerbeam

This is a media entity subdevice for the i.MX Camera Serial Interface module.

warning: 3 lines add whitespace errors.
Applying: media: imx: Add CSI subdev driver .git/rebase-apply/patch:38: new blank line at EOF. +
warning: 1 line adds whitespace errors.

Signature · 3 lines
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
#80 lore raw

Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver

In reply to Steve Longerbeam
Quoted 5 lines · Steve Longerbeam

This is a media entity subdevice for the i.MX Camera Serial Interface module.

---

The lack of s_frame_interval/g_frame_interval in this driver means:

media-ctl -v -d /dev/media1 --set-v4l2 '"imx6-mipi-csi2":1[fmt:SGBRG8/512x512 at 1/30]'

Opening media device /dev/media1
Enumerating entities
Found 29 entities
Enumerating pads and links
Setting up format SGBRG8 512x512 on pad imx6-mipi-csi2/1
Format set: SGBRG8 512x512
Setting up frame interval 1/30 on entity imx6-mipi-csi2
Unable to set frame interval: Inappropriate ioctl for device (-25)Unable to setup formats: Inappropriate ioctl for device (25)

which causes the setup of the next element in the chain to also fail (because the above media-ctl command doesn't set the sink on the ipu1 csi0 mux.)

It seems to me that without the frame interval supported throughout the chain, there's no way for an application to properly negotiate the capture parameters via the "try" mechanism, since it has no idea whether the frame rate it wants is supported throughout the subdev chain. Eg, the sensor may be able to do 100fps but there could be something in the pipeline that restricts it due to bandwidth.

Signature · 3 lines
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
#81 lore raw

Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver

In reply to Steve Longerbeam
Quoted 35 lines · Steve Longerbeam
drivers/staging/media/imx/imx-csi.c new file +35 -0
+static int csi_link_validate(struct v4l2_subdev *sd,
+			     struct media_link *link,
+			     struct v4l2_subdev_format *source_fmt,
+			     struct v4l2_subdev_format *sink_fmt)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	bool is_csi2;
+	int ret;
+
+	ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);
+	if (ret)
+		return ret;
+
+	priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(priv->sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		ret = PTR_ERR(priv->sensor);
+		priv->sensor = NULL;
+		return ret;
+	}
+
+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
+			       &priv->sensor_mbus_cfg);
+	if (ret)
+		return ret;
+
+	is_csi2 = (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2);
+
+	if (is_csi2) {
+		int vc_num = 0;
+		/*
+		 * NOTE! It seems the virtual channels from the mipi csi-2
+		 * receiver are used only for routing by the video mux's,
+		 * or for hard-wired routing to the CSI's. Once the stream
+		 * enters the CSI's however, they are treated internally

drivers/staging/media/imx/imx-csi.c line 258

Quoted 10 lines
drivers/staging/media/imx/imx-csi.c new file +10 -0
+		 * in the IPU as virtual channel 0.
+		 */
+#if 0
+		vc_num = imx_media_find_mipi_csi2_channel(priv->md,
+							  &priv->sd.entity);
+		if (vc_num < 0)
+			return vc_num;
+#endif
+		ipu_csi_set_mipi_datatype(priv->csi, vc_num,
+					  &priv->format_mbus[priv->input_pad]);

This seems (at least to me) to go against the spirit of the subdev negotiation. Here, you seem to bypass the negotiation performed between the CSI and the attached device, wanting to grab the format from the CSI2 sensor directly. That bypasses negotiation performed at the CSI2 subdev and video muxes.

The same goes for the frame rate monitoring code - imho, the frame rate should be something that results from negotiation with the neighbouring element, not by poking at some entity that is several entities away.

Signature · 3 lines
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
author maintainer
#82 lore raw

Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver

Quoted 43 lines · Russell King - ARM Linux
Quoted 42 lines · Steve Longerbeam
drivers/staging/media/imx/imx-csi.c new file +42 -0
+static int csi_link_validate(struct v4l2_subdev *sd,
+			     struct media_link *link,
+			     struct v4l2_subdev_format *source_fmt,
+			     struct v4l2_subdev_format *sink_fmt)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	bool is_csi2;
+	int ret;
+
+	ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);
+	if (ret)
+		return ret;
+
+	priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(priv->sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		ret = PTR_ERR(priv->sensor);
+		priv->sensor = NULL;
+		return ret;
+	}
+
+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
+			       &priv->sensor_mbus_cfg);
+	if (ret)
+		return ret;
+
+	is_csi2 = (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2);
+
+	if (is_csi2) {
+		int vc_num = 0;
+		/*
+		 * NOTE! It seems the virtual channels from the mipi csi-2
+		 * receiver are used only for routing by the video mux's,
+		 * or for hard-wired routing to the CSI's. Once the stream
+		 * enters the CSI's however, they are treated internally
+		 * in the IPU as virtual channel 0.
+		 */
+#if 0
+		vc_num = imx_media_find_mipi_csi2_channel(priv->md,
+							  &priv->sd.entity);
+		if (vc_num < 0)
+			return vc_num;
Quoted 8 lines

drivers/staging/media/imx/imx-csi.c line 258

Quoted 3 lines
drivers/staging/media/imx/imx-csi.c new file +3 -0
+#endif
+		ipu_csi_set_mipi_datatype(priv->csi, vc_num,
+					  &priv->format_mbus[priv->input_pad]);

This seems (at least to me) to go against the spirit of the subdev negotiation. Here, you seem to bypass the negotiation performed between the CSI and the attached device, wanting to grab the format from the CSI2 sensor directly. That bypasses negotiation performed at the CSI2 subdev and video muxes.

This isn't so much grabbing a pad format, it is determining which source pad at the imx6-mipi-csi2 receiver subdev is reached from this CSI (which determines the virtual channel the CSI is receiving).

If there was a way to determine the vc# via a status register in the CSI, that would be perfect, but there isn't. In fact, the CSIs seem to be ignoring the meta-data bus sent by the CSI2IPU gasket which contains this info, or that bus is not being routed to the CSIs. As the note says, the CSIs only accept vc0 at the CSI_MIPI_DI register. Any other value programmed there results in no data from the CSI.

And even the presence of CSI_MIPI_DI register makes no sense to me, the CSIs are given the vc# and MIPI datatype from the CSI2IPU meta-data bus, so I don't understand why it needs to be told.

Anyway I can just yank this disabled code, it's only there for documentation purposes.

Quoted 5 lines

The same goes for the frame rate monitoring code - imho, the frame rate should be something that results from negotiation with the neighbouring element, not by poking at some entity that is several entities away.

I will fix this once the g_frame_interval op is implemented in every subdev. I believe you mentioned this already, but g_frame_interval will need to be chained until it reaches the originating sensor.

Steve

#83 lore raw

Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver

Quoted 67 lines · Steve Longerbeam
Quoted 43 lines · Russell King - ARM Linux
Quoted 42 lines · Steve Longerbeam
drivers/staging/media/imx/imx-csi.c new file +42 -0
+static int csi_link_validate(struct v4l2_subdev *sd,
+			     struct media_link *link,
+			     struct v4l2_subdev_format *source_fmt,
+			     struct v4l2_subdev_format *sink_fmt)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	bool is_csi2;
+	int ret;
+
+	ret = v4l2_subdev_link_validate_default(sd, link, source_fmt, sink_fmt);
+	if (ret)
+		return ret;
+
+	priv->sensor = __imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(priv->sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		ret = PTR_ERR(priv->sensor);
+		priv->sensor = NULL;
+		return ret;
+	}
+
+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
+			       &priv->sensor_mbus_cfg);
+	if (ret)
+		return ret;
+
+	is_csi2 = (priv->sensor_mbus_cfg.type == V4L2_MBUS_CSI2);
+
+	if (is_csi2) {
+		int vc_num = 0;
+		/*
+		 * NOTE! It seems the virtual channels from the mipi csi-2
+		 * receiver are used only for routing by the video mux's,
+		 * or for hard-wired routing to the CSI's. Once the stream
+		 * enters the CSI's however, they are treated internally
+		 * in the IPU as virtual channel 0.
+		 */
+#if 0
+		vc_num = imx_media_find_mipi_csi2_channel(priv->md,
+							  &priv->sd.entity);
+		if (vc_num < 0)
+			return vc_num;
Quoted 8 lines

drivers/staging/media/imx/imx-csi.c line 258

Quoted 3 lines
drivers/staging/media/imx/imx-csi.c new file +3 -0
+#endif
+		ipu_csi_set_mipi_datatype(priv->csi, vc_num,
+					  &priv->format_mbus[priv->input_pad]);

This seems (at least to me) to go against the spirit of the subdev negotiation. Here, you seem to bypass the negotiation performed between the CSI and the attached device, wanting to grab the format from the CSI2 sensor directly. That bypasses negotiation performed at the CSI2 subdev and video muxes.

This isn't so much grabbing a pad format, it is determining which source pad at the imx6-mipi-csi2 receiver subdev is reached from this CSI (which determines the virtual channel the CSI is receiving).

If there was a way to determine the vc# via a status register in the CSI, that would be perfect, but there isn't. In fact, the CSIs seem to be ignoring the meta-data bus sent by the CSI2IPU gasket which contains this info, or that bus is not being routed to the CSIs. As the note says, the CSIs only accept vc0 at the CSI_MIPI_DI register. Any other value programmed there results in no data from the CSI.

Quoted 3 lines

And even the presence of CSI_MIPI_DI register makes no sense to me, the CSIs are given the vc# and MIPI datatype from the CSI2IPU meta-data bus, so I don't understand why it needs to be told.

The CSI_MIPI_DI register selects the data stream we want from the CSI2 camera.

CSI2 cameras can produce many different data streams - for example, a CSI2 camera can, for the same image, produce a YUV encoded frame and a jpeg-encoded frame. These are sent over the CSI2 serial bus using two different data types.

The CSI2IPU converts the serial data streams into a parallel data stream, feeding that into the CSI layer. The CSI layer, in conjunction with the CSI_MIPI_DI register, selects one of these streams to pass on to the SMFC and other blocks.

Quoted 1 line

From what I've read, the CSI can also be programmed to pass other

streams on as well, but I've never tried that.

In my particular case, the IMX219 camera produces a complete frame using the RAW8 or RAW10 MIPI data type, and also produces two lines of register data per frame, encoded using another data type. I think it should be possible to program the CSI to pass this other data on as "generic data" through a different FIFO and have it written to memory, which makes it possible to know the camera settings for each frame. (This isn't something I'm interested in though, I'm just using it as an example of why, possibly, there's that register in the CSI block.)

Signature · 3 lines
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
maintainer
#84 lore raw

Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver

In reply to Steve Longerbeam

On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
[...]

Quoted 16 lines
drivers/staging/media/imx/imx-csi.c new file +16 -0
+static int csi_set_fmt(struct v4l2_subdev *sd,
+		       struct v4l2_subdev_pad_config *cfg,
+		       struct v4l2_subdev_format *sdformat)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	struct v4l2_rect crop;
+	int ret;
+
+	if (sdformat->pad >= CSI_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];

drivers/staging/media/imx/imx-csi.c line 374

Quoted 10 lines
drivers/staging/media/imx/imx-csi.c new file +10 -0
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	if (sdformat->pad == priv->output_pad) {
+		sdformat->format.code = infmt->code;
+		sdformat->format.field = infmt->field;
+		crop.left = priv->crop.left;
+		crop.top = priv->crop.top;
+		crop.width = sdformat->format.width;
+		crop.height = sdformat->format.height;
+		ret = csi_try_crop(priv, &crop);

This is the wrong way around, see also below.

Here the the output sdformat->format.width/height should be set to the priv->crop.width/height (or priv->crop.width/height / 2, to enable downscaling). The crop rectangle should not be changed by an output pad set_fmt.

drivers/staging/media/imx/imx-csi.c line 387

Quoted 13 lines
drivers/staging/media/imx/imx-csi.c new file +13 -0
+		if (ret)
+			return ret;
+		sdformat->format.width = crop.width;
+		sdformat->format.height = crop.height;
+	}
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		priv->format_mbus[sdformat->pad] = sdformat->format;
+		/* Update the crop window if this is output pad  */
+		if (sdformat->pad == priv->output_pad)
+			priv->crop = crop;

The crop rectangle instead should be reset to the full input frame when the input pad format is set.

regards
Philipp

maintainer
#85 lore raw

Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver

Quoted 35 lines · Philipp Zabel

On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
[...]

Quoted 16 lines
(quoted diff) +16 -0
+static int csi_set_fmt(struct v4l2_subdev *sd,
+		       struct v4l2_subdev_pad_config *cfg,
+		       struct v4l2_subdev_format *sdformat)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	struct v4l2_rect crop;
+	int ret;
+
+	if (sdformat->pad >= CSI_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];

Review

Quoted 10 lines
(quoted diff) +10 -0
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	if (sdformat->pad == priv->output_pad) {
+		sdformat->format.code = infmt->code;
+		sdformat->format.field = infmt->field;
+		crop.left = priv->crop.left;
+		crop.top = priv->crop.top;
+		crop.width = sdformat->format.width;
+		crop.height = sdformat->format.height;
+		ret = csi_try_crop(priv, &crop);

This is the wrong way around, see also below.

Here the the output sdformat->format.width/height should be set to the priv->crop.width/height (or priv->crop.width/height / 2, to enable downscaling). The crop rectangle should not be changed by an output pad set_fmt.

[...]

Quoted 2 lines

The crop rectangle instead should be reset to the full input frame when the input pad format is set.

How about this:

----------8<----------

drivers/staging/media/imx/imx-media-csi.c +20 -12
index 5e80a0871b139..8220e4204a125 100644
@@ -484,6 +484,8 @@ static int csi_setup(struct csi_priv *priv)
 	if_fmt.field = outfmt->field;
 
 	ipu_csi_set_window(priv->csi, &priv->crop);
+	ipu_csi_set_downsize(priv->csi, priv->crop.width == 2 * outfmt->width,
+					priv->crop.height == 2 * outfmt->height);
 
 	ipu_csi_init_interface(priv->csi, &sensor_mbus_cfg, &if_fmt);
 
@@ -830,15 +832,14 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
 	switch (sdformat->pad) {
 	case CSI_SRC_PAD_DIRECT:
 	case CSI_SRC_PAD_IDMAC:
-		crop.left = priv->crop.left;
-		crop.top = priv->crop.top;
-		crop.width = sdformat->format.width;
-		crop.height = sdformat->format.height;
-		ret = csi_try_crop(priv, &crop, sensor);
-		if (ret)
-			return ret;
-		sdformat->format.width = crop.width;
-		sdformat->format.height = crop.height;
+		if (sdformat->format.width < priv->crop.width * 3 / 4)
+			sdformat->format.width = priv->crop.width / 2;
+		else
+			sdformat->format.width = priv->crop.width;
+		if (sdformat->format.height < priv->crop.height * 3 / 4)
+			sdformat->format.height = priv->crop.height / 2;
+		else
+			sdformat->format.height = priv->crop.height;
 
 		if (sdformat->pad == CSI_SRC_PAD_IDMAC) {
 			cc = imx_media_find_format(0, sdformat->format.code,
@@ -887,6 +888,14 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
 		}
 		break;
 	case CSI_SINK_PAD:
+		crop.left = 0;
+		crop.top = 0;
+		crop.width = sdformat->format.width;
+		crop.height = sdformat->format.height;
+		ret = csi_try_crop(priv, &crop, sensor);
+		if (ret)
+			return ret;
+
 		cc = imx_media_find_format(0, sdformat->format.code,
 					   true, false);
 		if (!cc) {
@@ -904,9 +913,8 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
 	} else {
 		priv->format_mbus[sdformat->pad] = sdformat->format;
 		priv->cc[sdformat->pad] = cc;
-		/* Update the crop window if this is an output pad  */
-		if (sdformat->pad == CSI_SRC_PAD_DIRECT ||
-		    sdformat->pad == CSI_SRC_PAD_IDMAC)
+		/* Reset the crop window if this is the input pad  */
+		if (sdformat->pad == CSI_SINK_PAD)
 			priv->crop = crop;
 	}
 

---------->8----------

regards
Philipp

author maintainer
#86 lore raw

Re: [PATCH v3 17/24] media: imx: Add CSI subdev driver

Quoted 31 lines · Philipp Zabel
Quoted 30 lines · Philipp Zabel

On Fri, 2017-01-06 at 18:11 -0800, Steve Longerbeam wrote:
[...]

Quoted 16 lines
(quoted diff) +16 -0
+static int csi_set_fmt(struct v4l2_subdev *sd,
+		       struct v4l2_subdev_pad_config *cfg,
+		       struct v4l2_subdev_format *sdformat)
+{
+	struct csi_priv *priv = v4l2_get_subdevdata(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	struct v4l2_rect crop;
+	int ret;
+
+	if (sdformat->pad >= CSI_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];

Review

Quoted 10 lines
(quoted diff) +10 -0
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	if (sdformat->pad == priv->output_pad) {
+		sdformat->format.code = infmt->code;
+		sdformat->format.field = infmt->field;
+		crop.left = priv->crop.left;
+		crop.top = priv->crop.top;
+		crop.width = sdformat->format.width;
+		crop.height = sdformat->format.height;
+		ret = csi_try_crop(priv, &crop);

This is the wrong way around, see also below.

Quoted 8 lines
Quoted 4 lines

Here the the output sdformat->format.width/height should be set to the priv->crop.width/height (or priv->crop.width/height / 2, to enable downscaling). The crop rectangle should not be changed by an output pad set_fmt.

[...]

Quoted 2 lines

The crop rectangle instead should be reset to the full input frame when the input pad format is set.

How about this:

Thanks for the patch, I'll try it tomorrow.

Steve

Quoted 70 lines

----------8<----------

drivers/staging/media/imx/imx-media-csi.c +20 -13
index 5e80a0871b139..8220e4204a125 100644
@@ -484,6 +484,8 @@ static int csi_setup(struct csi_priv *priv)
  	if_fmt.field = outfmt->field;
 
  	ipu_csi_set_window(priv->csi, &priv->crop);
+	ipu_csi_set_downsize(priv->csi, priv->crop.width == 2 * outfmt->width,
+					priv->crop.height == 2 * outfmt->height);
 
  	ipu_csi_init_interface(priv->csi, &sensor_mbus_cfg, &if_fmt);
 
@@ -830,15 +832,14 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
  	switch (sdformat->pad) {
  	case CSI_SRC_PAD_DIRECT:
  	case CSI_SRC_PAD_IDMAC:
-		crop.left = priv->crop.left;
-		crop.top = priv->crop.top;
-		crop.width = sdformat->format.width;
-		crop.height = sdformat->format.height;
-		ret = csi_try_crop(priv, &crop, sensor);
-		if (ret)
-			return ret;
-		sdformat->format.width = crop.width;
-		sdformat->format.height = crop.height;
+		if (sdformat->format.width < priv->crop.width * 3 / 4)
+			sdformat->format.width = priv->crop.width / 2;
+		else
+			sdformat->format.width = priv->crop.width;
+		if (sdformat->format.height < priv->crop.height * 3 / 4)
+			sdformat->format.height = priv->crop.height / 2;
+		else
+			sdformat->format.height = priv->crop.height;
 
  		if (sdformat->pad == CSI_SRC_PAD_IDMAC) {
  			cc = imx_media_find_format(0, sdformat->format.code,
@@ -887,6 +888,14 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
  		}
  		break;
  	case CSI_SINK_PAD:
+		crop.left = 0;
+		crop.top = 0;
+		crop.width = sdformat->format.width;
+		crop.height = sdformat->format.height;
+		ret = csi_try_crop(priv, &crop, sensor);
+		if (ret)
+			return ret;
+
  		cc = imx_media_find_format(0, sdformat->format.code,
  					   true, false);
  		if (!cc) {
@@ -904,9 +913,8 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
  	} else {
  		priv->format_mbus[sdformat->pad] = sdformat->format;
  		priv->cc[sdformat->pad] = cc;
-		/* Update the crop window if this is an output pad  */
-		if (sdformat->pad == CSI_SRC_PAD_DIRECT ||
-		    sdformat->pad == CSI_SRC_PAD_IDMAC)
+		/* Reset the crop window if this is the input pad  */
+		if (sdformat->pad == CSI_SINK_PAD)
  			priv->crop = crop;
  	}
 
---------->8----------

regards
Philipp

maintainer
#87 lore raw

PATCH 1/1 media: imx: csi: fix crop rectangle reset in sink set_fmt

In reply to Steve Longerbeam

The csi_try_crop call in set_fmt should compare the cropping rectangle to the currently set input format, not to the previous input format.

---
This is a patch against the current imx-media-staging-md-wip branch. S_FMT wouldn't update the cropping rectangle during the first time it is called, since the csi_try_crop call would compare to the input frame format returned by __csi_get_fmt, which still contains the old value.

---
 drivers/staging/media/imx/imx-media-csi.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
(quoted diff) +0 -0

b/drivers/staging/media/imx/imx-media-csi.c
index 637d0f137938a..7c590bf94fad5 100644

drivers/staging/media/imx/imx-media-csi.c +0 -0
@@ -820,15 +820,13 @@ __csi_get_fmt(struct csi_priv *priv, struct

v4l2_subdev_pad_config *cfg,
static int csi_try_crop(struct csi_priv *priv,
struct v4l2_rect *crop, struct v4l2_subdev_pad_config *cfg,
- enum v4l2_subdev_format_whence which,
+ struct v4l2_mbus_framefmt *infmt,
struct imx_media_subdev *sensor)
{
struct v4l2_of_endpoint *sensor_ep;
- struct v4l2_mbus_framefmt *infmt;
v4l2_std_id std;
int ret;

- infmt = __csi_get_fmt(priv, cfg, CSI_SINK_PAD, which);
sensor_ep = &sensor->sensor_ep;

crop->width = min_t(__u32, infmt->width, crop->width);
@@ -1023,8 +1021,7 @@ static int csi_set_fmt(struct v4l2_subdev *sd,
crop.top = 0;
crop.width = sdformat->format.width; crop.height = sdformat->format.height;
- ret = csi_try_crop(priv, &crop, cfg,
- sdformat->which, sensor);
+ ret = csi_try_crop(priv, &crop, cfg, &sdformat->format, sensor);
if (ret)
return ret;

@@ -1104,6 +1101,7 @@ static int csi_set_selection(struct v4l2_subdev *sd,
struct v4l2_subdev_selection *sel)
{
struct csi_priv *priv = v4l2_get_subdevdata(sd);
+ struct v4l2_mbus_framefmt *infmt;
struct imx_media_subdev *sensor;
int ret;

@@ -1133,7 +1131,8 @@ static int csi_set_selection(struct v4l2_subdev *sd,
return 0;
}

- ret = csi_try_crop(priv, &sel->r, cfg, sel->which, sensor);
+ infmt = __csi_get_fmt(priv, cfg, CSI_SINK_PAD, sel->which);
+ ret = csi_try_crop(priv, &sel->r, cfg, infmt, sensor);
if (ret)
return ret;

Signature · 1 line
2.11.0
author maintainer
#88 lore raw

PATCH v3 18/24 media: imx: Add SMFC subdev driver

In reply to Steve Longerbeam

This is a media entity subdevice driver for the i.MX Sensor Multi-FIFO Controller module. Video frames are received from the CSI and can be routed to various sinks including the i.MX Image Converter for scaling, color-space conversion, motion compensated deinterlacing, and image rotation.

---
 drivers/staging/media/imx/Makefile   |   1 +
 drivers/staging/media/imx/imx-smfc.c | 737 +++++++++++++++++++++++++++++++++++
 2 files changed, 738 insertions(+)
 create mode 100644 drivers/staging/media/imx/imx-smfc.c
drivers/staging/media/imx/Makefile +1 -0
index 133672a..3559d7b 100644
@@ -5,4 +5,5 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
 
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
 
drivers/staging/media/imx/imx-smfc.c new file +737 -0

737 lines not on this page · raw message

Signature · 1 line
2.7.4
#89 lore raw

Re: [PATCH v3 18/24] media: imx: Add SMFC subdev driver

Hi Steve,

drivers/staging/media/imx/imx-smfc.c line 44

Quoted 12 lines · Steve Longerbeam
drivers/staging/media/imx/imx-smfc.c new file +12 -0
+/*
+ * Min/Max supported width and heights.
+ *
+ * We allow planar output from the SMFC, so we have to align
+ * output width by 16 pixels to meet IDMAC alignment requirements,
+ * which also means input width must have the same alignment.
+ */
+#define MIN_W       176
+#define MIN_H       144
+#define MAX_W      8192
+#define MAX_H      4096
+#define W_ALIGN    4 /* multiple of 16 pixels */

Does this only apply to planar formats?

I notice Philipp's driver allows 8 pixel alignment. If it's only for planar formats, it ought to determine the alignment based on the requested format rather than hard-coding it to the maximum alignment of all the supported formats.

Thanks.

Signature · 3 lines
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
author maintainer
#90 lore raw

Re: [PATCH v3 18/24] media: imx: Add SMFC subdev driver

Quoted 17 lines · Russell King - ARM Linux

Hi Steve,

drivers/staging/media/imx/imx-smfc.c line 44

Quoted 12 lines · Steve Longerbeam
drivers/staging/media/imx/imx-smfc.c new file +12 -0
+/*
+ * Min/Max supported width and heights.
+ *
+ * We allow planar output from the SMFC, so we have to align
+ * output width by 16 pixels to meet IDMAC alignment requirements,
+ * which also means input width must have the same alignment.
+ */
+#define MIN_W       176
+#define MIN_H       144
+#define MAX_W      8192
+#define MAX_H      4096
+#define W_ALIGN    4 /* multiple of 16 pixels */

Does this only apply to planar formats?

Quoted 4 lines

I notice Philipp's driver allows 8 pixel alignment. If it's only for planar formats, it ought to determine the alignment based on the requested format rather than hard-coding it to the maximum alignment of all the supported formats.

yeah, I got lazy/tired there. I will fix this.

Steve

author maintainer
#91 lore raw

PATCH v3 19/24 media: imx: Add IC subdev drivers

In reply to Steve Longerbeam

This is a set of three media entity subdevice drivers for the i.MX Image Converter. The i.MX IC module contains three independent "tasks":

- Pre-processing Encode task: video frames are routed directly from the CSI and can be scaled, color-space converted, and rotated. Scaled output is limited to 1024x1024 resolution. Output frames are routed to the camera interface entities (camif).

- Pre-processing Viewfinder task: this task can perform the same conversions as the pre-process encode task, but in addition can be used for hardware motion compensated deinterlacing. Frames can come either directly from the CSI or from the SMFC entities (memory buffers via IDMAC channels). Scaled output is limited to 1024x1024 resolution. Output frames can be routed to various sinks including the post-processing task entities.

- Post-processing task: same conversions as pre-process encode. However this entity sends frames to the i.MX IPU image converter which supports image tiling, which allows scaled output up to 4096x4096 resolution. Output frames can be routed to the camera interfaces.

---
 drivers/staging/media/imx/Makefile        |    2 +
 drivers/staging/media/imx/imx-ic-common.c |  109 +++
 drivers/staging/media/imx/imx-ic-pp.c     |  636 ++++++++++++++++
 drivers/staging/media/imx/imx-ic-prpenc.c | 1033 +++++++++++++++++++++++++
 drivers/staging/media/imx/imx-ic-prpvf.c  | 1179 +++++++++++++++++++++++++++++
 drivers/staging/media/imx/imx-ic.h        |   38 +
 6 files changed, 2997 insertions(+)
 create mode 100644 drivers/staging/media/imx/imx-ic-common.c
 create mode 100644 drivers/staging/media/imx/imx-ic-pp.c
 create mode 100644 drivers/staging/media/imx/imx-ic-prpenc.c
 create mode 100644 drivers/staging/media/imx/imx-ic-prpvf.c
 create mode 100644 drivers/staging/media/imx/imx-ic.h
drivers/staging/media/imx/Makefile +2 -0
index 3559d7b..d2a962c 100644
@@ -1,8 +1,10 @@
 imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \
 	imx-media-of.o
+imx-ic-objs := imx-ic-common.o imx-ic-prpenc.o imx-ic-prpvf.o imx-ic-pp.o
 
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
+obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
 
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
drivers/staging/media/imx/imx-ic-common.c new file +109 -0

109 lines not on this page · raw message

drivers/staging/media/imx/imx-ic-pp.c new file +636 -0

636 lines not on this page · raw message

drivers/staging/media/imx/imx-ic-prpenc.c new file +1033 -0

1,033 lines not on this page · raw message

drivers/staging/media/imx/imx-ic-prpvf.c new file +1179 -0

1,179 lines not on this page · raw message

drivers/staging/media/imx/imx-ic.h new file +38 -0
new file mode 100644
index 0000000..1ab222b
@@ -0,0 +1,38 @@
+/*
+ * V4L2 Image Converter Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#ifndef _IMX_IC_H
+#define _IMX_IC_H
+
+#include <media/v4l2-subdev.h>
+
+struct imx_ic_priv {
+	struct device *dev;
+	struct v4l2_subdev sd;
+	int    ipu_id;
+	int    task_id;
+	void   *task_priv;
+};
+
+struct imx_ic_ops {
+	struct v4l2_subdev_ops *subdev_ops;
+	struct v4l2_subdev_internal_ops *internal_ops;
+	struct media_entity_operations *entity_ops;
+
+	int (*init)(struct imx_ic_priv *ic_priv);
+	void (*remove)(struct imx_ic_priv *ic_priv);
+};
+
+extern struct imx_ic_ops imx_ic_prpenc_ops;
[… more lines in the raw message]

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.7.4

Part of the diff is not shown here. View the raw message.

#92 lore raw

Re: [PATCH v3 19/24] media: imx: Add IC subdev drivers

Quoted 1,680 lines · Steve Longerbeam

This is a set of three media entity subdevice drivers for the i.MX Image Converter. The i.MX IC module contains three independent "tasks":

- Pre-processing Encode task: video frames are routed directly from the CSI and can be scaled, color-space converted, and rotated. Scaled output is limited to 1024x1024 resolution. Output frames are routed to the camera interface entities (camif).

- Pre-processing Viewfinder task: this task can perform the same conversions as the pre-process encode task, but in addition can be used for hardware motion compensated deinterlacing. Frames can come either directly from the CSI or from the SMFC entities (memory buffers via IDMAC channels). Scaled output is limited to 1024x1024 resolution. Output frames can be routed to various sinks including the post-processing task entities.

- Post-processing task: same conversions as pre-process encode. However this entity sends frames to the i.MX IPU image converter which supports image tiling, which allows scaled output up to 4096x4096 resolution. Output frames can be routed to the camera interfaces.

---
 drivers/staging/media/imx/Makefile        |    2 +
 drivers/staging/media/imx/imx-ic-common.c |  109 +++
 drivers/staging/media/imx/imx-ic-pp.c     |  636 ++++++++++++++++
 drivers/staging/media/imx/imx-ic-prpenc.c | 1033 +++++++++++++++++++++++++
 drivers/staging/media/imx/imx-ic-prpvf.c  | 1179 +++++++++++++++++++++++++++++
 drivers/staging/media/imx/imx-ic.h        |   38 +
 6 files changed, 2997 insertions(+)
 create mode 100644 drivers/staging/media/imx/imx-ic-common.c
 create mode 100644 drivers/staging/media/imx/imx-ic-pp.c
 create mode 100644 drivers/staging/media/imx/imx-ic-prpenc.c
 create mode 100644 drivers/staging/media/imx/imx-ic-prpvf.c
 create mode 100644 drivers/staging/media/imx/imx-ic.h
drivers/staging/media/imx/Makefile +2 -0
index 3559d7b..d2a962c 100644
@@ -1,8 +1,10 @@
 imx-media-objs := imx-media-dev.o imx-media-fim.o imx-media-internal-sd.o \
 	imx-media-of.o
+imx-ic-objs := imx-ic-common.o imx-ic-prpenc.o imx-ic-prpvf.o imx-ic-pp.o
 
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media.o
 obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-media-common.o
+obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
 
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
drivers/staging/media/imx/imx-ic-common.c new file +109 -0
new file mode 100644
index 0000000..45706ca
@@ -0,0 +1,109 @@
+/*
+ * V4L2 Image Converter Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+static struct imx_ic_ops *ic_ops[IC_NUM_TASKS] = {
+	[IC_TASK_ENCODER]        = &imx_ic_prpenc_ops,
+	[IC_TASK_VIEWFINDER]     = &imx_ic_prpvf_ops,
+	[IC_TASK_POST_PROCESSOR] = &imx_ic_pp_ops,
+};
+
+static int imx_ic_probe(struct platform_device *pdev)
+{
+	struct imx_media_internal_sd_platformdata *pdata;
+	struct imx_ic_priv *priv;
+	int ret;
+
+	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	platform_set_drvdata(pdev, &priv->sd);
+	priv->dev = &pdev->dev;
+
+	/* get our ipu_id, grp_id and IC task id */
+	pdata = priv->dev->platform_data;
+	priv->ipu_id = pdata->ipu_id;
+	switch (pdata->grp_id) {
+	case IMX_MEDIA_GRP_ID_IC_PRPENC:
+		priv->task_id = IC_TASK_ENCODER;
+		break;
+	case IMX_MEDIA_GRP_ID_IC_PRPVF:
+		priv->task_id = IC_TASK_VIEWFINDER;
+		break;
+	case IMX_MEDIA_GRP_ID_IC_PP0...IMX_MEDIA_GRP_ID_IC_PP3:
+		priv->task_id = IC_TASK_POST_PROCESSOR;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	v4l2_subdev_init(&priv->sd, ic_ops[priv->task_id]->subdev_ops);
+	v4l2_set_subdevdata(&priv->sd, priv);
+	priv->sd.internal_ops = ic_ops[priv->task_id]->internal_ops;
+	priv->sd.entity.ops = ic_ops[priv->task_id]->entity_ops;
+	priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
+	priv->sd.dev = &pdev->dev;
+	priv->sd.owner = THIS_MODULE;
+	priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
+	priv->sd.grp_id = pdata->grp_id;
+	strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
+
+	ret = ic_ops[priv->task_id]->init(priv);
+	if (ret)
+		return ret;
+
+	ret = v4l2_async_register_subdev(&priv->sd);
+	if (ret)
+		ic_ops[priv->task_id]->remove(priv);
+
+	return ret;
+}
+
+static int imx_ic_remove(struct platform_device *pdev)
+{
+	struct v4l2_subdev *sd = platform_get_drvdata(pdev);
+	struct imx_ic_priv *priv = container_of(sd, struct imx_ic_priv, sd);
+
+	ic_ops[priv->task_id]->remove(priv);
+
+	v4l2_async_unregister_subdev(&priv->sd);
+	media_entity_cleanup(&priv->sd.entity);
+	v4l2_device_unregister_subdev(sd);
+
+	return 0;
+}
+
+static const struct platform_device_id imx_ic_ids[] = {
+	{ .name = "imx-ipuv3-ic" },
+	{ },
+};
+MODULE_DEVICE_TABLE(platform, imx_ic_ids);
+
+static struct platform_driver imx_ic_driver = {
+	.probe = imx_ic_probe,
+	.remove = imx_ic_remove,
+	.id_table = imx_ic_ids,
+	.driver = {
+		.name = "imx-ipuv3-ic",
+	},
+};
+module_platform_driver(imx_ic_driver);
+
+MODULE_DESCRIPTION("i.MX IC subdev driver");
+MODULE_AUTHOR("Steve Longerbeam <[email protected]>");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("platform:imx-ipuv3-ic");
drivers/staging/media/imx/imx-ic-pp.c new file +636 -0
new file mode 100644
index 0000000..1f75616
@@ -0,0 +1,636 @@
+/*
+ * V4L2 IC Post-Processor Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2014-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <video/imx-ipu-image-convert.h>
+#include <media/imx.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+#define PP_NUM_PADS 2
+
+struct pp_priv {
+	struct imx_media_dev *md;
+	struct imx_ic_priv *ic_priv;
+	int pp_id;
+
+	struct ipu_soc *ipu;
+	struct ipu_image_convert_ctx *ic_ctx;
+
+	struct media_pad pad[PP_NUM_PADS];
+	int input_pad;
+	int output_pad;
+
+	/* our dma buffer sink ring */
+	struct imx_media_dma_buf_ring *in_ring;
+	/* the dma buffer ring we send to sink */
+	struct imx_media_dma_buf_ring *out_ring;
+	struct ipu_image_convert_run *out_run;
+
+	struct imx_media_dma_buf *inbuf; /* last input buffer */
+
+	bool stream_on;    /* streaming is on */
+	bool stop;         /* streaming is stopping */
+	spinlock_t irqlock;
+
+	struct v4l2_subdev *src_sd;
+	struct v4l2_subdev *sink_sd;
+
+	struct v4l2_mbus_framefmt format_mbus[PP_NUM_PADS];
+	const struct imx_media_pixfmt *cc[PP_NUM_PADS];
+
+	/* motion select control */
+	struct v4l2_ctrl_handler ctrl_hdlr;
+	int  rotation; /* degrees */
+	bool hflip;
+	bool vflip;
+
+	/* derived from rotation, hflip, vflip controls */
+	enum ipu_rotate_mode rot_mode;
+};
+
+static inline struct pp_priv *sd_to_priv(struct v4l2_subdev *sd)
+{
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+
+	return ic_priv->task_priv;
+}
+
+static void pp_convert_complete(struct ipu_image_convert_run *run,
+				void *data)
+{
+	struct pp_priv *priv = data;
+	struct imx_media_dma_buf *done;
+	unsigned long flags;
+
+	spin_lock_irqsave(&priv->irqlock, flags);
+
+	done = imx_media_dma_buf_get_active(priv->out_ring);
+	/* give the completed buffer to the sink */
+	if (!WARN_ON(!done))
+		imx_media_dma_buf_done(done, run->status ?
+				       IMX_MEDIA_BUF_STATUS_ERROR :
+				       IMX_MEDIA_BUF_STATUS_DONE);
+
+	/* we're done with the inbuf, queue it back */
+	imx_media_dma_buf_queue(priv->in_ring, priv->inbuf->index);
+
+	spin_unlock_irqrestore(&priv->irqlock, flags);
+}
+
+static void pp_queue_conversion(struct pp_priv *priv,
+				struct imx_media_dma_buf *inbuf)
+{
+	struct ipu_image_convert_run *run;
+	struct imx_media_dma_buf *outbuf;
+
+	/* get next queued buffer and make it active */
+	outbuf = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	imx_media_dma_buf_set_active(outbuf);
+	priv->inbuf = inbuf;
+
+	run = &priv->out_run[outbuf->index];
+	run->ctx = priv->ic_ctx;
+	run->in_phys = inbuf->phys;
+	run->out_phys = outbuf->phys;
+	ipu_image_convert_queue(run);
+}
+
+static long pp_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
+{
+	struct pp_priv *priv = sd_to_priv(sd);
+	struct imx_media_dma_buf_ring **ring;
+	struct imx_media_dma_buf *buf;
+	unsigned long flags;
+
+	switch (cmd) {
+	case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
+		/* src asks for a buffer ring */
+		if (!priv->in_ring)
+			return -EINVAL;
+		ring = (struct imx_media_dma_buf_ring **)arg;
+		*ring = priv->in_ring;
+		break;
+	case IMX_MEDIA_NEW_DMA_BUF:
+		/* src hands us a new buffer */
+		spin_lock_irqsave(&priv->irqlock, flags);
+		if (!priv->stop &&
+		    !imx_media_dma_buf_get_active(priv->out_ring)) {
+			buf = imx_media_dma_buf_dequeue(priv->in_ring);
+			if (buf)
+				pp_queue_conversion(priv, buf);
+		}
+		spin_unlock_irqrestore(&priv->irqlock, flags);
+		break;
+	case IMX_MEDIA_REL_DMA_BUF_SINK_RING:
+		/* src indicates sink buffer ring can be freed */
+		if (!priv->in_ring)
+			return 0;
+		v4l2_info(sd, "%s: freeing sink ring\n", __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+		priv->in_ring = NULL;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int pp_start(struct pp_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct ipu_image image_in, image_out;
+	const struct imx_media_pixfmt *incc;
+	struct v4l2_mbus_framefmt *infmt;
+	int i, in_size, ret;
+
+	/* ask the sink for the buffer ring */
+	ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			       IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+			       &priv->out_ring);
+	if (ret)
+		return ret;
+
+	imx_media_mbus_fmt_to_ipu_image(&image_in,
+					&priv->format_mbus[priv->input_pad]);
+	imx_media_mbus_fmt_to_ipu_image(&image_out,
+					&priv->format_mbus[priv->output_pad]);
+
+	priv->ipu = priv->md->ipu[ic_priv->ipu_id];
+	priv->ic_ctx = ipu_image_convert_prepare(priv->ipu,
+						 IC_TASK_POST_PROCESSOR,
+						 &image_in, &image_out,
+						 priv->rot_mode,
+						 pp_convert_complete, priv);
+	if (IS_ERR(priv->ic_ctx))
+		return PTR_ERR(priv->ic_ctx);
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	incc = priv->cc[priv->input_pad];
+	in_size = (infmt->width * incc->bpp * infmt->height) >> 3;
+
+	if (priv->in_ring) {
+		v4l2_warn(&ic_priv->sd, "%s: dma-buf ring was not freed\n",
+			  __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+	}
+
+	priv->in_ring = imx_media_alloc_dma_buf_ring(priv->md,
+						     &priv->src_sd->entity,
+						     &ic_priv->sd.entity,
+						     in_size,
+						     IMX_MEDIA_MIN_RING_BUFS,
+						     true);
+	if (IS_ERR(priv->in_ring)) {
+		v4l2_err(&ic_priv->sd,
+			 "failed to alloc dma-buf ring\n");
+		ret = PTR_ERR(priv->in_ring);
+		priv->in_ring = NULL;
+		goto out_unprep;
+	}
+
+	for (i = 0; i < IMX_MEDIA_MIN_RING_BUFS; i++)
+		imx_media_dma_buf_queue(priv->in_ring, i);
+
+	priv->out_run = kzalloc(IMX_MEDIA_MAX_RING_BUFS *
+				sizeof(*priv->out_run), GFP_KERNEL);
+	if (!priv->out_run) {
+		v4l2_err(&ic_priv->sd, "failed to alloc src ring runs\n");
+		ret = -ENOMEM;
+		goto out_free_ring;
+	}
+
+	priv->stop = false;
+
+	return 0;
+
+out_free_ring:
+	imx_media_free_dma_buf_ring(priv->in_ring);
+	priv->in_ring = NULL;
+out_unprep:
+	ipu_image_convert_unprepare(priv->ic_ctx);
+	return ret;
+}
+
+static void pp_stop(struct pp_priv *priv)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&priv->irqlock, flags);
+	priv->stop = true;
+	spin_unlock_irqrestore(&priv->irqlock, flags);
+
+	ipu_image_convert_unprepare(priv->ic_ctx);
+	kfree(priv->out_run);
+
+	priv->out_ring = NULL;
+
+	/* inform sink that its sink buffer ring can now be freed */
+	v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			 IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
+}
+
+static int pp_s_stream(struct v4l2_subdev *sd, int enable)
+{
+	struct pp_priv *priv = sd_to_priv(sd);
+	int ret = 0;
+
+	if (!priv->src_sd || !priv->sink_sd)
+		return -EPIPE;
+
+	v4l2_info(sd, "stream %s\n", enable ? "ON" : "OFF");
+
+	if (enable && !priv->stream_on)
+		ret = pp_start(priv);
+	else if (!enable && priv->stream_on)
+		pp_stop(priv);
+
+	if (!ret)
+		priv->stream_on = enable;
+	return ret;
+}
+
+static int pp_enum_mbus_code(struct v4l2_subdev *sd,
+			     struct v4l2_subdev_pad_config *cfg,
+			     struct v4l2_subdev_mbus_code_enum *code)
+{
+	const struct imx_media_pixfmt *cc;
+	u32 fourcc;
+	int ret;
+
+	if (code->pad >= PP_NUM_PADS)
+		return -EINVAL;
+
+	ret = ipu_image_convert_enum_format(code->index, &fourcc);
+	if (ret)
+		return ret;
+
+	/* convert returned fourcc to mbus code */
+	cc = imx_media_find_format(fourcc, 0, true, true);
+	if (WARN_ON(!cc))
+		return -EINVAL;
+
+	code->code = cc->codes[0];
+	return 0;
+}
+
+static int pp_get_fmt(struct v4l2_subdev *sd,
+		      struct v4l2_subdev_pad_config *cfg,
+		      struct v4l2_subdev_format *sdformat)
+{
+	struct pp_priv *priv = sd_to_priv(sd);
+
+	if (sdformat->pad >= PP_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int pp_set_fmt(struct v4l2_subdev *sd,
+		      struct v4l2_subdev_pad_config *cfg,
+		      struct v4l2_subdev_format *sdformat)
+{
+	struct pp_priv *priv = sd_to_priv(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *cc;
+	struct ipu_image test_in, test_out;
+	u32 code;
+
+	if (sdformat->pad >= PP_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	cc = imx_media_find_format(0, sdformat->format.code, true, true);
+	if (!cc) {
+		imx_media_enum_format(&code, 0, true, true);
+		cc = imx_media_find_format(0, code, true, true);
+		sdformat->format.code = cc->codes[0];
+	}
+
+	if (sdformat->pad == priv->output_pad) {
+		imx_media_mbus_fmt_to_ipu_image(&test_out, &sdformat->format);
+		imx_media_mbus_fmt_to_ipu_image(&test_in, infmt);
+		ipu_image_convert_adjust(&test_in, &test_out, priv->rot_mode);
+		imx_media_ipu_image_to_mbus_fmt(&sdformat->format, &test_out);
+	} else {
+		imx_media_mbus_fmt_to_ipu_image(&test_in, &sdformat->format);
+		imx_media_mbus_fmt_to_ipu_image(&test_out, outfmt);
+		ipu_image_convert_adjust(&test_in, &test_out, priv->rot_mode);
+		imx_media_ipu_image_to_mbus_fmt(&sdformat->format, &test_in);
+	}
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		if (sdformat->pad == priv->output_pad) {
+			*outfmt = sdformat->format;
+			imx_media_ipu_image_to_mbus_fmt(infmt, &test_in);
+		} else {
+			*infmt = sdformat->format;
+			imx_media_ipu_image_to_mbus_fmt(outfmt, &test_out);
+		}
+		priv->cc[sdformat->pad] = cc;
+	}
+
+	return 0;
+}
+
+static int pp_link_setup(struct media_entity *entity,
+			 const struct media_pad *local,
+			 const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+	struct pp_priv *priv = ic_priv->task_priv;
+	struct v4l2_subdev *remote_sd;
+
+	dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SOURCE) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->sink_sd)
+				return -EBUSY;
+			priv->sink_sd = remote_sd;
+		} else {
+			priv->sink_sd = NULL;
+		}
+	} else {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->src_sd)
+				return -EBUSY;
+			priv->src_sd = remote_sd;
+		} else {
+			priv->src_sd = NULL;
+		}
+	}
+
+	return 0;
+}
+
+static int pp_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	struct pp_priv *priv = container_of(ctrl->handler,
+					       struct pp_priv, ctrl_hdlr);
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	enum ipu_rotate_mode rot_mode;
+	bool hflip, vflip;
+	int rotation, ret;
+
+	rotation = priv->rotation;
+	hflip = priv->hflip;
+	vflip = priv->vflip;
+
+	switch (ctrl->id) {
+	case V4L2_CID_HFLIP:
+		hflip = (ctrl->val == 1);
+		break;
+	case V4L2_CID_VFLIP:
+		vflip = (ctrl->val == 1);
+		break;
+	case V4L2_CID_ROTATE:
+		rotation = ctrl->val;
+		break;
+	default:
+		v4l2_err(&ic_priv->sd, "Invalid control\n");
+		return -EINVAL;
+	}
+
+	ret = ipu_degrees_to_rot_mode(&rot_mode, rotation, hflip, vflip);
+	if (ret)
+		return ret;
+
+	if (rot_mode != priv->rot_mode) {
+		struct v4l2_mbus_framefmt *infmt, *outfmt;
+		struct ipu_image test_in, test_out;
+
+		/* can't change rotation mid-streaming */
+		if (priv->stream_on)
+			return -EBUSY;
+
+		/*
+		 * make sure this rotation will work with current input/output
+		 * formats before setting
+		 */
+		infmt = &priv->format_mbus[priv->input_pad];
+		outfmt = &priv->format_mbus[priv->output_pad];
+		imx_media_mbus_fmt_to_ipu_image(&test_in, infmt);
+		imx_media_mbus_fmt_to_ipu_image(&test_out, outfmt);
+
+		ret = ipu_image_convert_verify(&test_in, &test_out, rot_mode);
+		if (ret)
+			return ret;
+
+		priv->rot_mode = rot_mode;
+		priv->rotation = rotation;
+		priv->hflip = hflip;
+		priv->vflip = vflip;
+	}
+
+	return 0;
+}
+
+static const struct v4l2_ctrl_ops pp_ctrl_ops = {
+	.s_ctrl = pp_s_ctrl,
+};
+
+static const struct v4l2_ctrl_config pp_std_ctrl[] = {
+	{
+		.id = V4L2_CID_HFLIP,
+		.name = "Horizontal Flip",
+		.type = V4L2_CTRL_TYPE_BOOLEAN,
+		.def =  0,
+		.min =  0,
+		.max =  1,
+		.step = 1,
+	}, {
+		.id = V4L2_CID_VFLIP,
+		.name = "Vertical Flip",
+		.type = V4L2_CTRL_TYPE_BOOLEAN,
+		.def =  0,
+		.min =  0,
+		.max =  1,
+		.step = 1,
+	}, {
+		.id = V4L2_CID_ROTATE,
+		.name = "Rotation",
+		.type = V4L2_CTRL_TYPE_INTEGER,
+		.def =   0,
+		.min =   0,
+		.max = 270,
+		.step = 90,
+	},
+};
+
+#define PP_NUM_CONTROLS ARRAY_SIZE(pp_std_ctrl)
+
+static int pp_init_controls(struct pp_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_ctrl_handler *hdlr = &priv->ctrl_hdlr;
+	const struct v4l2_ctrl_config *c;
+	int i, ret;
+
+	v4l2_ctrl_handler_init(hdlr, PP_NUM_CONTROLS);
+
+	for (i = 0; i < PP_NUM_CONTROLS; i++) {
+		c = &pp_std_ctrl[i];
+		v4l2_ctrl_new_std(hdlr, &pp_ctrl_ops,
+				  c->id, c->min, c->max, c->step, c->def);
+	}
+
+	ic_priv->sd.ctrl_handler = hdlr;
+
+	if (hdlr->error) {
+		ret = hdlr->error;
+		v4l2_ctrl_handler_free(hdlr);
+		return ret;
+	}
+
+	v4l2_ctrl_handler_setup(hdlr);
+
+	return 0;
+}
+
+/*
+ * retrieve our pads parsed from the OF graph by the media device
+ */
+static int pp_registered(struct v4l2_subdev *sd)
+{
+	struct pp_priv *priv = sd_to_priv(sd);
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i, ret;
+
+	/* get media device */
+	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	imxsd = imx_media_find_subdev_by_sd(priv->md, sd);
+	if (IS_ERR(imxsd))
+		return PTR_ERR(imxsd);
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1)
+		return -EINVAL;
+
+	for (i = 0; i < PP_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		priv->pad[i] = pad->pad;
+		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+			priv->input_pad = i;
+		else
+			priv->output_pad = i;
+
+		/* set a default mbus format  */
+		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
+					      640, 480, 0, V4L2_FIELD_NONE,
+					      &priv->cc[i]);
+		if (ret)
+			return ret;
+	}
+
+	ret = pp_init_controls(priv);
+	if (ret)
+		return ret;
+
+	ret = media_entity_pads_init(&sd->entity, PP_NUM_PADS, priv->pad);
+	if (ret)
+		goto free_ctrls;
+
+	return 0;
+free_ctrls:
+	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+	return ret;
+}
+
+static struct v4l2_subdev_pad_ops pp_pad_ops = {
+	.enum_mbus_code = pp_enum_mbus_code,
+	.get_fmt = pp_get_fmt,
+	.set_fmt = pp_set_fmt,
+};
+
+static struct v4l2_subdev_video_ops pp_video_ops = {
+	.s_stream = pp_s_stream,
+};
+
+static struct v4l2_subdev_core_ops pp_core_ops = {
+	.ioctl = pp_ioctl,
+};
+
+static struct media_entity_operations pp_entity_ops = {
+	.link_setup = pp_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_ops pp_subdev_ops = {
+	.video = &pp_video_ops,
+	.pad = &pp_pad_ops,
+	.core = &pp_core_ops,
+};
+
+static struct v4l2_subdev_internal_ops pp_internal_ops = {
+	.registered = pp_registered,
+};
+
+static int pp_init(struct imx_ic_priv *ic_priv)
+{
+	struct pp_priv *priv;
+
+	priv = devm_kzalloc(ic_priv->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	ic_priv->task_priv = priv;
+	priv->ic_priv = ic_priv;
+	spin_lock_init(&priv->irqlock);
+
+	/* get our PP id */
+	priv->pp_id = (ic_priv->sd.grp_id >> IMX_MEDIA_GRP_ID_IC_PP_BIT) - 1;
+
+	return 0;
+}
+
+static void pp_remove(struct imx_ic_priv *ic_priv)
+{
+	struct pp_priv *priv = ic_priv->task_priv;
+
+	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+}
+
+struct imx_ic_ops imx_ic_pp_ops = {
+	.subdev_ops = &pp_subdev_ops,
+	.internal_ops = &pp_internal_ops,
+	.entity_ops = &pp_entity_ops,
+	.init = pp_init,
+	.remove = pp_remove,
+};
drivers/staging/media/imx/imx-ic-prpenc.c new file +865 -0
new file mode 100644
index 0000000..3d85a82
@@ -0,0 +1,1033 @@
+/*
+ * V4L2 Capture IC Encoder Subdev for Freescale i.MX5/6 SOC
+ *
+ * This subdevice handles capture of video frames from the CSI, which
+ * are routed directly to the Image Converter preprocess encode task,
+ * for resizing, colorspace conversion, and rotation.
+ *
+ * Copyright (c) 2012-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/imx.h>
+#include "imx-media.h"
+#include "imx-ic.h"
+
+#define PRPENC_NUM_PADS 2
+
+#define MAX_W_IC   1024
+#define MAX_H_IC   1024
+#define MAX_W_SINK 4096
+#define MAX_H_SINK 4096
+
+struct prpenc_priv {
+	struct imx_media_dev *md;
+	struct imx_ic_priv *ic_priv;
+
+	/* IPU units we require */
+	struct ipu_soc *ipu;
+	struct ipu_ic *ic_enc;
+
+	struct media_pad pad[PRPENC_NUM_PADS];
+	int input_pad;
+	int output_pad;
+
+	struct ipuv3_channel *enc_ch;
+	struct ipuv3_channel *enc_rot_in_ch;
+	struct ipuv3_channel *enc_rot_out_ch;
+
+	/* the dma buffer ring to send to sink */
+	struct imx_media_dma_buf_ring *out_ring;
+	struct imx_media_dma_buf *next;
+
+	int ipu_buf_num;  /* ipu double buffer index: 0-1 */
+
+	struct v4l2_subdev *src_sd;
+	struct v4l2_subdev *sink_sd;
+
+	/* the CSI id at link validate */
+	int csi_id;
+
+	/* the attached sensor at stream on */
+	struct imx_media_subdev *sensor;
+
+	struct v4l2_mbus_framefmt format_mbus[PRPENC_NUM_PADS];
+	const struct imx_media_pixfmt *cc[PRPENC_NUM_PADS];
+
+	struct imx_media_dma_buf rot_buf[2];
+
+	/* controls */
+	struct v4l2_ctrl_handler ctrl_hdlr;
+	int  rotation; /* degrees */
+	bool hflip;
+	bool vflip;
+
+	/* derived from rotation, hflip, vflip controls */
+	enum ipu_rotate_mode rot_mode;
+
+	spinlock_t irqlock;
+
+	struct timer_list eof_timeout_timer;
+	int eof_irq;
+	int nfb4eof_irq;
+
+	bool stream_on; /* streaming is on */
+	bool last_eof;  /* waiting for last EOF at stream off */
+	struct completion last_eof_comp;
+};
+
+static inline struct prpenc_priv *sd_to_priv(struct v4l2_subdev *sd)
+{
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+
+	return ic_priv->task_priv;
+}
+
+static void prpenc_put_ipu_resources(struct prpenc_priv *priv)
+{
+	if (!IS_ERR_OR_NULL(priv->ic_enc))
+		ipu_ic_put(priv->ic_enc);
+	priv->ic_enc = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->enc_ch))
+		ipu_idmac_put(priv->enc_ch);
+	priv->enc_ch = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->enc_rot_in_ch))
+		ipu_idmac_put(priv->enc_rot_in_ch);
+	priv->enc_rot_in_ch = NULL;
+
+	if (!IS_ERR_OR_NULL(priv->enc_rot_out_ch))
+		ipu_idmac_put(priv->enc_rot_out_ch);
+	priv->enc_rot_out_ch = NULL;
+}
+
+static int prpenc_get_ipu_resources(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	int ret;
+
+	priv->ipu = priv->md->ipu[ic_priv->ipu_id];
+
+	priv->ic_enc = ipu_ic_get(priv->ipu, IC_TASK_ENCODER);
+	if (IS_ERR(priv->ic_enc)) {
+		v4l2_err(&ic_priv->sd, "failed to get IC ENC\n");
+		ret = PTR_ERR(priv->ic_enc);
+		goto out;
+	}
+
+	priv->enc_ch = ipu_idmac_get(priv->ipu,
+				     IPUV3_CHANNEL_IC_PRP_ENC_MEM);
+	if (IS_ERR(priv->enc_ch)) {
+		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
+			 IPUV3_CHANNEL_IC_PRP_ENC_MEM);
+		ret = PTR_ERR(priv->enc_ch);
+		goto out;
+	}
+
+	priv->enc_rot_in_ch = ipu_idmac_get(priv->ipu,
+					    IPUV3_CHANNEL_MEM_ROT_ENC);
+	if (IS_ERR(priv->enc_rot_in_ch)) {
+		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
+			 IPUV3_CHANNEL_MEM_ROT_ENC);
+		ret = PTR_ERR(priv->enc_rot_in_ch);
+		goto out;
+	}
+
+	priv->enc_rot_out_ch = ipu_idmac_get(priv->ipu,
+					     IPUV3_CHANNEL_ROT_ENC_MEM);
+	if (IS_ERR(priv->enc_rot_out_ch)) {
+		v4l2_err(&ic_priv->sd, "could not get IDMAC channel %u\n",
+			 IPUV3_CHANNEL_ROT_ENC_MEM);
+		ret = PTR_ERR(priv->enc_rot_out_ch);
+		goto out;
+	}
+
+	return 0;
+out:
+	prpenc_put_ipu_resources(priv);
+	return ret;
+}
+
+static irqreturn_t prpenc_eof_interrupt(int irq, void *dev_id)
+{
+	struct prpenc_priv *priv = dev_id;
+	struct imx_media_dma_buf *done, *next;
+	struct ipuv3_channel *channel;
+
+	spin_lock(&priv->irqlock);
+
+	if (priv->last_eof) {
+		complete(&priv->last_eof_comp);
+		priv->last_eof = false;
+		goto unlock;
+	}
+
+	/* inform CSI of this EOF so it can monitor frame intervals */
+	v4l2_subdev_call(priv->src_sd, core, interrupt_service_routine,
+			 0, NULL);
+
+	channel = (ipu_rot_mode_is_irt(priv->rot_mode)) ?
+		priv->enc_rot_out_ch : priv->enc_ch;
+
+	done = imx_media_dma_buf_get_active(priv->out_ring);
+	/* give the completed buffer to the sink  */
+	if (!WARN_ON(!done))
+		imx_media_dma_buf_done(done, IMX_MEDIA_BUF_STATUS_DONE);
+
+	/* priv->next buffer is now the active one */
+	imx_media_dma_buf_set_active(priv->next);
+
+	/* bump the EOF timeout timer */
+	mod_timer(&priv->eof_timeout_timer,
+		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+	if (ipu_idmac_buffer_is_ready(channel, priv->ipu_buf_num))
+		ipu_idmac_clear_buffer(channel, priv->ipu_buf_num);
+
+	/* get next queued buffer */
+	next = imx_media_dma_buf_get_next_queued(priv->out_ring);
+
+	ipu_cpmem_set_buffer(channel, priv->ipu_buf_num, next->phys);
+	ipu_idmac_select_buffer(channel, priv->ipu_buf_num);
+
+	/* toggle IPU double-buffer index */
+	priv->ipu_buf_num ^= 1;
+	priv->next = next;
+
+unlock:
+	spin_unlock(&priv->irqlock);
+	return IRQ_HANDLED;
+}
+
+static irqreturn_t prpenc_nfb4eof_interrupt(int irq, void *dev_id)
+{
+	struct prpenc_priv *priv = dev_id;
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_NFB4EOF,
+	};
+
+	v4l2_err(&ic_priv->sd, "NFB4EOF\n");
+
+	v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+
+	return IRQ_HANDLED;
+}
+
+/*
+ * EOF timeout timer function.
+ */
+static void prpenc_eof_timeout(unsigned long data)
+{
+	struct prpenc_priv *priv = (struct prpenc_priv *)data;
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	static const struct v4l2_event ev = {
+		.type = V4L2_EVENT_IMX_EOF_TIMEOUT,
+	};
+
+	v4l2_err(&ic_priv->sd, "EOF timeout\n");
+
+	v4l2_subdev_notify_event(&ic_priv->sd, &ev);
+}
+
+static void prpenc_setup_channel(struct prpenc_priv *priv,
+				 struct ipuv3_channel *channel,
+				 enum ipu_rotate_mode rot_mode,
+				 dma_addr_t addr0, dma_addr_t addr1,
+				 bool rot_swap_width_height)
+{
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	unsigned int burst_size;
+	struct ipu_image image;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+
+	if (rot_swap_width_height)
+		swap(outfmt->width, outfmt->height);
+
+	ipu_cpmem_zero(channel);
+
+	imx_media_mbus_fmt_to_ipu_image(&image, outfmt);
+
+	image.phys0 = addr0;
+	image.phys1 = addr1;
+	ipu_cpmem_set_image(channel, &image);
+
+	if (channel == priv->enc_rot_in_ch ||
+	    channel == priv->enc_rot_out_ch) {
+		burst_size = 8;
+		ipu_cpmem_set_block_mode(channel);
+	} else {
+		burst_size = (outfmt->width & 0xf) ? 8 : 16;
+	}
+
+	ipu_cpmem_set_burstsize(channel, burst_size);
+
+	if (rot_mode)
+		ipu_cpmem_set_rotation(channel, rot_mode);
+
+	if (outfmt->field == V4L2_FIELD_NONE &&
+	    (V4L2_FIELD_HAS_BOTH(infmt->field) ||
+	     infmt->field == V4L2_FIELD_ALTERNATE) &&
+	    channel == priv->enc_ch)
+		ipu_cpmem_interlaced_scan(channel, image.pix.bytesperline);
+
+	ipu_ic_task_idma_init(priv->ic_enc, channel,
+			      outfmt->width, outfmt->height,
+			      burst_size, rot_mode);
+	ipu_cpmem_set_axi_id(channel, 1);
+
+	ipu_idmac_set_double_buffer(channel, true);
+
+	if (rot_swap_width_height)
+		swap(outfmt->width, outfmt->height);
+}
+
+static int prpenc_setup_rotation(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *outcc, *incc;
+	struct imx_media_dma_buf *buf0, *buf1;
+	int out_size, ret;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	incc = priv->cc[priv->input_pad];
+	outcc = priv->cc[priv->output_pad];
+
+	out_size = (outfmt->width * outcc->bpp * outfmt->height) >> 3;
+
+	ret = imx_media_alloc_dma_buf(priv->md, &priv->rot_buf[0], out_size);
+	if (ret) {
+		v4l2_err(&ic_priv->sd, "failed to alloc rot_buf[0], %d\n", ret);
+		return ret;
+	}
+	ret = imx_media_alloc_dma_buf(priv->md, &priv->rot_buf[1], out_size);
+	if (ret) {
+		v4l2_err(&ic_priv->sd, "failed to alloc rot_buf[1], %d\n", ret);
+		goto free_rot0;
+	}
+
+	ret = ipu_ic_task_init(priv->ic_enc,
+			       infmt->width, infmt->height,
+			       outfmt->height, outfmt->width,
+			       incc->cs, outcc->cs);
+	if (ret) {
+		v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
+		goto free_rot1;
+	}
+
+	/* init the IC ENC-->MEM IDMAC channel */
+	prpenc_setup_channel(priv, priv->enc_ch,
+			     IPU_ROTATE_NONE,
+			     priv->rot_buf[0].phys,
+			     priv->rot_buf[1].phys,
+			     true);
+
+	/* init the MEM-->IC ENC ROT IDMAC channel */
+	prpenc_setup_channel(priv, priv->enc_rot_in_ch,
+			     priv->rot_mode,
+			     priv->rot_buf[0].phys,
+			     priv->rot_buf[1].phys,
+			     true);
+
+	buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	imx_media_dma_buf_set_active(buf0);
+	buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	priv->next = buf1;
+
+	/* init the destination IC ENC ROT-->MEM IDMAC channel */
+	prpenc_setup_channel(priv, priv->enc_rot_out_ch,
+			     IPU_ROTATE_NONE,
+			     buf0->phys, buf1->phys,
+			     false);
+
+	/* now link IC ENC-->MEM to MEM-->IC ENC ROT */
+	ipu_idmac_link(priv->enc_ch, priv->enc_rot_in_ch);
+
+	/* enable the IC */
+	ipu_ic_enable(priv->ic_enc);
+
+	/* set buffers ready */
+	ipu_idmac_select_buffer(priv->enc_ch, 0);
+	ipu_idmac_select_buffer(priv->enc_ch, 1);
+	ipu_idmac_select_buffer(priv->enc_rot_out_ch, 0);
+	ipu_idmac_select_buffer(priv->enc_rot_out_ch, 1);
+
+	/* enable the channels */
+	ipu_idmac_enable_channel(priv->enc_ch);
+	ipu_idmac_enable_channel(priv->enc_rot_in_ch);
+	ipu_idmac_enable_channel(priv->enc_rot_out_ch);
+
+	/* and finally enable the IC PRPENC task */
+	ipu_ic_task_enable(priv->ic_enc);
+
+	return 0;
+
+free_rot1:
+	imx_media_free_dma_buf(priv->md, &priv->rot_buf[1]);
+free_rot0:
+	imx_media_free_dma_buf(priv->md, &priv->rot_buf[0]);
+	return ret;
+}
+
+static void prpenc_unsetup_rotation(struct prpenc_priv *priv)
+{
+	ipu_ic_task_disable(priv->ic_enc);
+
+	ipu_idmac_disable_channel(priv->enc_ch);
+	ipu_idmac_disable_channel(priv->enc_rot_in_ch);
+	ipu_idmac_disable_channel(priv->enc_rot_out_ch);
+
+	ipu_idmac_unlink(priv->enc_ch, priv->enc_rot_in_ch);
+
+	ipu_ic_disable(priv->ic_enc);
+
+	imx_media_free_dma_buf(priv->md, &priv->rot_buf[0]);
+	imx_media_free_dma_buf(priv->md, &priv->rot_buf[1]);
+}
+
+static int prpenc_setup_norotation(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *outcc, *incc;
+	struct imx_media_dma_buf *buf0, *buf1;
+	int ret;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	incc = priv->cc[priv->input_pad];
+	outcc = priv->cc[priv->output_pad];
+
+	ret = ipu_ic_task_init(priv->ic_enc,
+			       infmt->width, infmt->height,
+			       outfmt->width, outfmt->height,
+			       incc->cs, outcc->cs);
+	if (ret) {
+		v4l2_err(&ic_priv->sd, "ipu_ic_task_init failed, %d\n", ret);
+		return ret;
+	}
+
+	buf0 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	imx_media_dma_buf_set_active(buf0);
+	buf1 = imx_media_dma_buf_get_next_queued(priv->out_ring);
+	priv->next = buf1;
+
+	/* init the IC PRP-->MEM IDMAC channel */
+	prpenc_setup_channel(priv, priv->enc_ch, priv->rot_mode,
+			     buf0->phys, buf1->phys,
+			     false);
+
+	ipu_cpmem_dump(priv->enc_ch);
+	ipu_ic_dump(priv->ic_enc);
+	ipu_dump(priv->ipu);
+
+	ipu_ic_enable(priv->ic_enc);
+
+	/* set buffers ready */
+	ipu_idmac_select_buffer(priv->enc_ch, 0);
+	ipu_idmac_select_buffer(priv->enc_ch, 1);
+
+	/* enable the channels */
+	ipu_idmac_enable_channel(priv->enc_ch);
+
+	/* enable the IC ENCODE task */
+	ipu_ic_task_enable(priv->ic_enc);
+
+	return 0;
+}
+
+static void prpenc_unsetup_norotation(struct prpenc_priv *priv)
+{
+	ipu_ic_task_disable(priv->ic_enc);
+	ipu_idmac_disable_channel(priv->enc_ch);
+	ipu_ic_disable(priv->ic_enc);
+}
+
+static int prpenc_start(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	int ret;
+
+	if (!priv->sensor) {
+		v4l2_err(&ic_priv->sd, "no sensor attached\n");
+		return -EINVAL;
+	}
+
+	ret = prpenc_get_ipu_resources(priv);
+	if (ret)
+		return ret;
+
+	/* set IC to receive from CSI */
+	ipu_set_ic_src_mux(priv->ipu, priv->csi_id, false);
+
+	/* ask the sink for the buffer ring */
+	ret = v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			       IMX_MEDIA_REQ_DMA_BUF_SINK_RING,
+			       &priv->out_ring);
+	if (ret)
+		goto out_put_ipu;
+
+	priv->ipu_buf_num = 0;
+
+	/* init EOF completion waitq */
+	init_completion(&priv->last_eof_comp);
+	priv->last_eof = false;
+
+	if (ipu_rot_mode_is_irt(priv->rot_mode))
+		ret = prpenc_setup_rotation(priv);
+	else
+		ret = prpenc_setup_norotation(priv);
+	if (ret)
+		goto out_put_ipu;
+
+	priv->nfb4eof_irq = ipu_idmac_channel_irq(priv->ipu,
+						  priv->enc_ch,
+						  IPU_IRQ_NFB4EOF);
+	ret = devm_request_irq(ic_priv->dev, priv->nfb4eof_irq,
+			       prpenc_nfb4eof_interrupt, 0,
+			       "imx-ic-prpenc-nfb4eof", priv);
+	if (ret) {
+		v4l2_err(&ic_priv->sd,
+			 "Error registering NFB4EOF irq: %d\n", ret);
+		goto out_unsetup;
+	}
+
+	if (ipu_rot_mode_is_irt(priv->rot_mode))
+		priv->eof_irq = ipu_idmac_channel_irq(
+			priv->ipu, priv->enc_rot_out_ch, IPU_IRQ_EOF);
+	else
+		priv->eof_irq = ipu_idmac_channel_irq(
+			priv->ipu, priv->enc_ch, IPU_IRQ_EOF);
+
+	ret = devm_request_irq(ic_priv->dev, priv->eof_irq,
+			       prpenc_eof_interrupt, 0,
+			       "imx-ic-prpenc-eof", priv);
+	if (ret) {
+		v4l2_err(&ic_priv->sd,
+			 "Error registering eof irq: %d\n", ret);
+		goto out_free_nfb4eof_irq;
+	}
+
+	/* start the EOF timeout timer */
+	mod_timer(&priv->eof_timeout_timer,
+		  jiffies + msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+
+	return 0;
+
+out_free_nfb4eof_irq:
+	devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+out_unsetup:
+	if (ipu_rot_mode_is_irt(priv->rot_mode))
+		prpenc_unsetup_rotation(priv);
+	else
+		prpenc_unsetup_norotation(priv);
+out_put_ipu:
+	prpenc_put_ipu_resources(priv);
+	return ret;
+}
+
+static void prpenc_stop(struct prpenc_priv *priv)
+{
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	unsigned long flags;
+	int ret;
+
+	/* mark next EOF interrupt as the last before stream off */
+	spin_lock_irqsave(&priv->irqlock, flags);
+	priv->last_eof = true;
+	spin_unlock_irqrestore(&priv->irqlock, flags);
+
+	/*
+	 * and then wait for interrupt handler to mark completion.
+	 */
+	ret = wait_for_completion_timeout(
+		&priv->last_eof_comp,
+		msecs_to_jiffies(IMX_MEDIA_EOF_TIMEOUT));
+	if (ret == 0)
+		v4l2_warn(&ic_priv->sd, "wait last EOF timeout\n");
+
+	devm_free_irq(ic_priv->dev, priv->eof_irq, priv);
+	devm_free_irq(ic_priv->dev, priv->nfb4eof_irq, priv);
+
+	if (ipu_rot_mode_is_irt(priv->rot_mode))
+		prpenc_unsetup_rotation(priv);
+	else
+		prpenc_unsetup_norotation(priv);
+
+	prpenc_put_ipu_resources(priv);
+
+	/* cancel the EOF timeout timer */
+	del_timer_sync(&priv->eof_timeout_timer);
+
+	priv->out_ring = NULL;
+
+	/* inform sink that the buffer ring can now be freed */
+	v4l2_subdev_call(priv->sink_sd, core, ioctl,
+			 IMX_MEDIA_REL_DMA_BUF_SINK_RING, 0);
+}
+
+static int prpenc_enum_mbus_code(struct v4l2_subdev *sd,
+				 struct v4l2_subdev_pad_config *cfg,
+				 struct v4l2_subdev_mbus_code_enum *code)
+{
+	struct prpenc_priv *priv = sd_to_priv(sd);
+	bool allow_planar;
+
+	if (code->pad >= PRPENC_NUM_PADS)
+		return -EINVAL;
+
+	allow_planar = (code->pad == priv->output_pad);
+
+	return imx_media_enum_format(&code->code, code->index,
+				     true, allow_planar);
+}
+
+static int prpenc_get_fmt(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_pad_config *cfg,
+			  struct v4l2_subdev_format *sdformat)
+{
+	struct prpenc_priv *priv = sd_to_priv(sd);
+
+	if (sdformat->pad >= PRPENC_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int prpenc_set_fmt(struct v4l2_subdev *sd,
+			  struct v4l2_subdev_pad_config *cfg,
+			  struct v4l2_subdev_format *sdformat)
+{
+	struct prpenc_priv *priv = sd_to_priv(sd);
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	const struct imx_media_pixfmt *cc;
+	bool allow_planar;
+	u32 code;
+
+	if (sdformat->pad >= PRPENC_NUM_PADS)
+		return -EINVAL;
+
+	if (priv->stream_on)
+		return -EBUSY;
+
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	allow_planar = (sdformat->pad == priv->output_pad);
+
+	cc = imx_media_find_format(0, sdformat->format.code,
+				   true, allow_planar);
+	if (!cc) {
+		imx_media_enum_format(&code, 0, true, false);
+		cc = imx_media_find_format(0, code, true, false);
+		sdformat->format.code = cc->codes[0];
+	}
+
+	if (sdformat->pad == priv->output_pad) {
+		sdformat->format.width = min_t(__u32,
+					       sdformat->format.width,
+					       MAX_W_IC);
+		sdformat->format.height = min_t(__u32,
+						sdformat->format.height,
+						MAX_H_IC);
+
+		if (sdformat->format.field != V4L2_FIELD_NONE)
+			sdformat->format.field = infmt->field;
+
+		/* IC resizer cannot downsize more than 4:1 */
+		if (ipu_rot_mode_is_irt(priv->rot_mode)) {
+			sdformat->format.width = max_t(__u32,
+						       sdformat->format.width,
+						       infmt->height / 4);
+			sdformat->format.height = max_t(__u32,
+							sdformat->format.height,
+							infmt->width / 4);
+		} else {
+			sdformat->format.width = max_t(__u32,
+						       sdformat->format.width,
+						       infmt->width / 4);
+			sdformat->format.height = max_t(__u32,
+							sdformat->format.height,
+							infmt->height / 4);
+		}
+	} else {
+		sdformat->format.width = min_t(__u32,
+					       sdformat->format.width,
+					       MAX_W_SINK);
+		sdformat->format.height = min_t(__u32,
+						sdformat->format.height,
+						MAX_H_SINK);
+	}
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		priv->format_mbus[sdformat->pad] = sdformat->format;
+		priv->cc[sdformat->pad] = cc;
+	}
+
+	return 0;
+}
+
+static int prpenc_link_setup(struct media_entity *entity,
+			     const struct media_pad *local,
+			     const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+	struct prpenc_priv *priv = ic_priv->task_priv;
+	struct v4l2_subdev *remote_sd;
+
+	dev_dbg(ic_priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	if (local->flags & MEDIA_PAD_FL_SOURCE) {
+		if (flags & MEDIA_LNK_FL_ENABLED) {
+			if (priv->sink_sd)
+				return -EBUSY;
+			priv->sink_sd = remote_sd;
+		} else {
+			priv->sink_sd = NULL;
+		}
+
+		return 0;
+	}
+
+	/* this is sink pad */
+	if (flags & MEDIA_LNK_FL_ENABLED) {
+		if (priv->src_sd)
+			return -EBUSY;
+		priv->src_sd = remote_sd;
+	} else {
+		priv->src_sd = NULL;
+		return 0;
+	}
+
+	switch (remote_sd->grp_id) {
+	case IMX_MEDIA_GRP_ID_CSI0:
+		priv->csi_id = 0;
+		break;
+	case IMX_MEDIA_GRP_ID_CSI1:
+		priv->csi_id = 1;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int prpenc_link_validate(struct v4l2_subdev *sd,
+				struct media_link *link,
+				struct v4l2_subdev_format *source_fmt,
+				struct v4l2_subdev_format *sink_fmt)
+{
+	struct imx_ic_priv *ic_priv = v4l2_get_subdevdata(sd);
+	struct prpenc_priv *priv = ic_priv->task_priv;
+	struct v4l2_mbus_config sensor_mbus_cfg;
+	int ret;
+
+	ret = v4l2_subdev_link_validate_default(sd, link,
+						source_fmt, sink_fmt);
+	if (ret)
+		return ret;
+
+	priv->sensor = __imx_media_find_sensor(priv->md, &ic_priv->sd.entity);
+	if (IS_ERR(priv->sensor)) {
+		v4l2_err(&ic_priv->sd, "no sensor attached\n");
+		ret = PTR_ERR(priv->sensor);
+		priv->sensor = NULL;
+		return ret;
+	}
+
+	ret = v4l2_subdev_call(priv->sensor->sd, video, g_mbus_config,
+			       &sensor_mbus_cfg);
+	if (ret)
+		return ret;
+
+	if (sensor_mbus_cfg.type == V4L2_MBUS_CSI2) {
+		int vc_num = 0;
+		/* see NOTE in imx-csi.c */
+#if 0
+		vc_num = imx_media_find_mipi_csi2_channel(
+			priv->md, &ic_priv->sd.entity);
+		if (vc_num < 0)
+			return vc_num;
+#endif
+		/* only virtual channel 0 can be sent to IC */
+		if (vc_num != 0)
+			return -EINVAL;
+	} else {
+		/*
+		 * only 8-bit pixels can be sent to IC for parallel
+		 * busses
+		 */
+		if (priv->sensor->sensor_ep.bus.parallel.bus_width >= 16)
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int prpenc_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+	struct prpenc_priv *priv = container_of(ctrl->handler,
+					       struct prpenc_priv, ctrl_hdlr);
+	struct imx_ic_priv *ic_priv = priv->ic_priv;
+	enum ipu_rotate_mode rot_mode;
+	bool hflip, vflip;
+	int rotation, ret;
+
+	rotation = priv->rotation;
+	hflip = priv->hflip;
+	vflip = priv->vflip;
+
+	switch (ctrl->id) {
+	case V4L2_CID_HFLIP:
+		hflip = (ctrl->val == 1);
+		break;
+	case V4L2_CID_VFLIP:
+		vflip = (ctrl->val == 1);
+		break;
+	case V4L2_CID_ROTATE:
+		rotation = ctrl->val;
+		break;
+	default:
+		v4l2_err(&ic_priv->sd, "Invalid control\n");
+		return -EINVAL;
+	}
+
+	ret = ipu_degrees_to_rot_mode(&rot_mode, rotation, hflip, vflip);
+	if (ret)
+		return ret;
+
+	if (rot_mode != priv->rot_mode) {
+		/* can't change rotation mid-streaming */
+		if (priv->stream_on)
+			return -EBUSY;
+
+		priv->rot_mode = rot_mode;
+		priv->rotation = rotation;
+		priv->hflip = hflip;
+		priv->vflip = vflip;
+	}
+
+	return 0;
+}
+
+static const struct v4l2_ctrl_ops prpenc_ctrl_ops = {
+	.s_ctrl = prpenc_s_ctrl,
+};
+
+static const struct v4l2_ctrl_config prpenc_std_ctrl[] = {
+	{
+		.id = V4L2_CID_HFLIP,
+		.name = "Horizontal Flip",
+		.type = V4L2_CTRL_TYPE_BOOLEAN,
+		.def =  0,
+		.min =  0,
+		.max =  1,
+		.step = 1,
+	}, {
+		.id = V4L2_CID_VFLIP,
+		.name = "Vertical Flip",
+		.type = V4L2_CTRL_TYPE_BOOLEAN,
+		.def =  0,
+		.min =  0,
+		.max =  1,
+		.step = 1,

drivers/staging/media/imx/imx-ic-prpenc.c line 875

Quoted 10 lines
drivers/staging/media/imx/imx-ic-prpenc.c new file +10 -0
+	}, {
+		.id = V4L2_CID_ROTATE,
+		.name = "Rotation",
+		.type = V4L2_CTRL_TYPE_INTEGER,
+		.def =   0,
+		.min =   0,
+		.max = 270,
+		.step = 90,
+	},
+};

Use v4l2_ctrl_new_std() instead of this array: this avoids duplicating information like the name and type.

If this is also done elsewhere, then it should be changed there as well.

Regards,

Hans

author maintainer
#93 lore raw

Re: [PATCH v3 19/24] media: imx: Add IC subdev drivers

Quoted 32 lines · Hans Verkuil
Quoted 18 lines · Steve Longerbeam
drivers/staging/media/imx/imx-ic-prpenc.c new file +18 -0
+
+static const struct v4l2_ctrl_config prpenc_std_ctrl[] = {
+	{
+		.id = V4L2_CID_HFLIP,
+		.name = "Horizontal Flip",
+		.type = V4L2_CTRL_TYPE_BOOLEAN,
+		.def =  0,
+		.min =  0,
+		.max =  1,
+		.step = 1,
+	}, {
+		.id = V4L2_CID_VFLIP,
+		.name = "Vertical Flip",
+		.type = V4L2_CTRL_TYPE_BOOLEAN,
+		.def =  0,
+		.min =  0,
+		.max =  1,
+		.step = 1,

drivers/staging/media/imx/imx-ic-prpenc.c line 875

Quoted 10 lines
drivers/staging/media/imx/imx-ic-prpenc.c new file +10 -0
+	}, {
+		.id = V4L2_CID_ROTATE,
+		.name = "Rotation",
+		.type = V4L2_CTRL_TYPE_INTEGER,
+		.def =   0,
+		.min =   0,
+		.max = 270,
+		.step = 90,
+	},
+};

Use v4l2_ctrl_new_std() instead of this array: this avoids duplicating information like the name and type.

Quoted 1 line

If this is also done elsewhere, then it should be changed there as well.

done.

Steve

#94 lore raw

Re: [PATCH v3 19/24] media: imx: Add IC subdev drivers

In reply to Steve Longerbeam
Quoted 17 lines · Steve Longerbeam

This is a set of three media entity subdevice drivers for the i.MX Image Converter. The i.MX IC module contains three independent "tasks":

- Pre-processing Encode task: video frames are routed directly from the CSI and can be scaled, color-space converted, and rotated. Scaled output is limited to 1024x1024 resolution. Output frames are routed to the camera interface entities (camif).

- Pre-processing Viewfinder task: this task can perform the same conversions as the pre-process encode task, but in addition can be used for hardware motion compensated deinterlacing. Frames can come either directly from the CSI or from the SMFC entities (memory buffers via IDMAC channels). Scaled output is limited to 1024x1024 resolution. Output frames can be routed to various sinks including the post-processing task entities.

Quoted 6 lines

- Post-processing task: same conversions as pre-process encode. However this entity sends frames to the i.MX IPU image converter which supports image tiling, which allows scaled output up to 4096x4096 resolution. Output frames can be routed to the camera interfaces.

Applying: media: imx: Add IC subdev drivers .git/rebase-apply/patch:3054: new blank line at EOF. +
warning: 1 line adds whitespace errors.

Signature · 3 lines
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
author maintainer
#95 lore raw

PATCH v3 20/24 media: imx: Add Camera Interface subdev driver

In reply to Steve Longerbeam

This is the camera interface driver that provides the v4l2 user interface. Frames can be received from various sources:

- directly from SMFC for capturing unconverted images directly from camera sensors.

- from the IC pre-process encode task.

- from the IC pre-process viewfinder task.

- from the IC post-process task.

---
 drivers/staging/media/imx/Makefile    |    2 +-
 drivers/staging/media/imx/imx-camif.c | 1000 +++++++++++++++++++++++++++++++++
 2 files changed, 1001 insertions(+), 1 deletion(-)
 create mode 100644 drivers/staging/media/imx/imx-camif.c
drivers/staging/media/imx/Makefile +1 -1
index d2a962c..fe9e992 100644
@@ -8,4 +8,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
 
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
-
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
drivers/staging/media/imx/imx-camif.c new file +1000 -0

1,000 lines not on this page · raw message

Signature · 1 line
2.7.4
#96 lore raw

Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver

Quoted 110 lines · Steve Longerbeam

This is the camera interface driver that provides the v4l2 user interface. Frames can be received from various sources:

- directly from SMFC for capturing unconverted images directly from camera sensors.

- from the IC pre-process encode task.

- from the IC pre-process viewfinder task.

- from the IC post-process task.

---
 drivers/staging/media/imx/Makefile    |    2 +-
 drivers/staging/media/imx/imx-camif.c | 1000 +++++++++++++++++++++++++++++++++
 2 files changed, 1001 insertions(+), 1 deletion(-)
 create mode 100644 drivers/staging/media/imx/imx-camif.c
drivers/staging/media/imx/Makefile +1 -1
index d2a962c..fe9e992 100644
@@ -8,4 +8,4 @@ obj-$(CONFIG_VIDEO_IMX_MEDIA) += imx-ic.o
 
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-csi.o
 obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-smfc.o
-
+obj-$(CONFIG_VIDEO_IMX_CAMERA) += imx-camif.o
drivers/staging/media/imx/imx-camif.c new file +75 -0
new file mode 100644
index 0000000..404f724
@@ -0,0 +1,1000 @@
+/*
+ * Video Camera Capture Subdev for Freescale i.MX5/6 SOC
+ *
+ * Copyright (c) 2012-2016 Mentor Graphics Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/module.h>
+#include <linux/of_platform.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/timer.h>
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-ioctl.h>
+#include <media/v4l2-of.h>
+#include <media/v4l2-subdev.h>
+#include <media/videobuf2-dma-contig.h>
+#include <video/imx-ipu-v3.h>
+#include <media/imx.h>
+#include "imx-media.h"
+
+#define CAMIF_NUM_PADS 2
+
+struct camif_priv {
+	struct device         *dev;
+	struct video_device    vfd;
+	struct media_pipeline  mp;
+	struct imx_media_dev  *md;
+	struct v4l2_subdev     sd;
+	struct media_pad       pad[CAMIF_NUM_PADS];
+	struct media_pad       vd_pad;
+	int id;
+	int input_pad;
+	int output_pad;
+
+	struct v4l2_mbus_framefmt format_mbus[CAMIF_NUM_PADS];
+	const struct imx_media_pixfmt *cc[CAMIF_NUM_PADS];
+
+	/* dma buffer ring */
+	struct imx_media_dma_buf_ring *in_ring;
+	struct v4l2_subdev     *src_sd;
+
+	struct mutex           mutex;       /* capture device mutex */
+	spinlock_t             q_lock;      /* protect ready_q */
+
+	/* buffer queue used in videobuf2 */
+	struct vb2_queue       buffer_queue;
+
+	/* streaming buffer queue */
+	struct list_head       ready_q;
+
+	/* controls inherited from subdevs */
+	struct v4l2_ctrl_handler ctrl_hdlr;
+
+	/* misc status */
+	int                    current_input; /* the current input */
+	v4l2_std_id            current_std;   /* current standard */
+	bool                   stop;          /* streaming is stopping */
+};
+
+/* In bytes, per queue */
+#define VID_MEM_LIMIT	SZ_64M
+
+static struct vb2_ops camif_qops;
+

drivers/staging/media/imx/imx-camif.c line 85

Quoted 10 lines
drivers/staging/media/imx/imx-camif.c new file +10 -0
+/*
+ * Video ioctls follow
+ */
+
+static int vidioc_querycap(struct file *file, void *fh,
+			   struct v4l2_capability *cap)
+{
+	strncpy(cap->driver, "imx-media-camif", sizeof(cap->driver) - 1);
+	strncpy(cap->card, "imx-media-camif", sizeof(cap->card) - 1);
+	cap->bus_info[0] = 0;

Should be set to something like 'platform:imx-media-camif'. v4l2-compliance should complain about this.

drivers/staging/media/imx/imx-camif.c line 87

Quoted 2 lines
drivers/staging/media/imx/imx-camif.c new file +2 -0
+	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;

Set device_caps in struct video_device, then drop these two lines since the core will set these up based on the device_caps field in struct video_device.

Quoted 111 lines
drivers/staging/media/imx/imx-camif.c new file +111 -0
+
+	return 0;
+}
+
+static int camif_enum_fmt_vid_cap(struct file *file, void *fh,
+				  struct v4l2_fmtdesc *f)
+{
+	const struct imx_media_pixfmt *cc;
+	u32 code;
+	int ret;
+
+	ret = imx_media_enum_format(&code, f->index, true, true);
+	if (ret)
+		return ret;
+	cc = imx_media_find_format(0, code, true, true);
+	if (!cc)
+		return -EINVAL;
+
+	f->pixelformat = cc->fourcc;
+
+	return 0;
+}
+
+static int camif_g_fmt_vid_cap(struct file *file, void *fh,
+			       struct v4l2_format *f)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct v4l2_mbus_framefmt *outfmt;
+
+	/* user format is the same as the format from output pad */
+	outfmt = &priv->format_mbus[priv->output_pad];
+	return imx_media_mbus_fmt_to_pix_fmt(&f->fmt.pix, outfmt);
+}
+
+static int camif_try_fmt_vid_cap(struct file *file, void *fh,
+				 struct v4l2_format *f)
+{
+	return camif_g_fmt_vid_cap(file, fh, f);
+}
+
+static int camif_s_fmt_vid_cap(struct file *file, void *fh,
+			       struct v4l2_format *f)
+{
+	struct camif_priv *priv = video_drvdata(file);
+
+	if (vb2_is_busy(&priv->buffer_queue)) {
+		v4l2_err(&priv->sd, "%s queue busy\n", __func__);
+		return -EBUSY;
+	}
+
+	return camif_try_fmt_vid_cap(file, priv, f);
+}
+
+static int camif_querystd(struct file *file, void *fh, v4l2_std_id *std)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	return v4l2_subdev_call(sensor->sd, video, querystd, std);
+}
+
+static int camif_g_std(struct file *file, void *fh, v4l2_std_id *std)
+{
+	struct camif_priv *priv = video_drvdata(file);
+
+	*std = priv->current_std;
+	return 0;
+}
+
+static int camif_s_std(struct file *file, void *fh, v4l2_std_id std)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+	int ret;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	if (vb2_is_busy(&priv->buffer_queue))
+		return -EBUSY;
+
+	ret = v4l2_subdev_call(sensor->sd, video, s_std, std);
+	if (ret < 0)
+		return ret;
+
+	priv->current_std = std;
+	return 0;
+}
+
+static int camif_enum_input(struct file *file, void *fh,
+			    struct v4l2_input *input)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+	int index = input->index;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+

drivers/staging/media/imx/imx-camif.c line 208

Quoted 10 lines
drivers/staging/media/imx/imx-camif.c new file +10 -0
+	if (index >= sensor->input.num)
+		return -EINVAL;
+
+	input->type = V4L2_INPUT_TYPE_CAMERA;
+	strncpy(input->name, sensor->input.name[index], sizeof(input->name));
+
+	if (index == priv->current_input) {
+		v4l2_subdev_call(sensor->sd, video, g_input_status,
+				 &input->status);
+		v4l2_subdev_call(sensor->sd, video, querystd, &input->std);

Wrong op, use g_tvnorms instead.

Quoted 783 lines
drivers/staging/media/imx/imx-camif.c new file +782 -0
+	} else {
+		input->status = V4L2_IN_ST_NO_SIGNAL;
+		input->std = V4L2_STD_UNKNOWN;
+	}
+
+	return 0;
+}
+
+static int camif_g_input(struct file *file, void *fh, unsigned int *index)
+{
+	struct camif_priv *priv = video_drvdata(file);
+
+	*index = priv->current_input;
+	return 0;
+}
+
+static int camif_s_input(struct file *file, void *fh, unsigned int index)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+	int ret;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	if (index >= sensor->input.num)
+		return -EINVAL;
+
+	if (index == priv->current_input)
+		return 0;
+
+	/* select the sensor's input */
+	ret = v4l2_subdev_call(sensor->sd, video, s_routing,
+			       sensor->input.value[index], 0, 0);
+	if (!ret)
+		priv->current_input = index;
+
+	return ret;
+}
+
+static int camif_g_parm(struct file *file, void *fh,
+			struct v4l2_streamparm *a)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	return v4l2_subdev_call(sensor->sd, video, g_parm, a);
+}
+
+static int camif_s_parm(struct file *file, void *fh,
+			struct v4l2_streamparm *a)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	if (a->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	return v4l2_subdev_call(sensor->sd, video, s_parm, a);
+}
+
+static const struct v4l2_ioctl_ops camif_ioctl_ops = {
+	.vidioc_querycap	= vidioc_querycap,
+
+	.vidioc_enum_fmt_vid_cap        = camif_enum_fmt_vid_cap,
+	.vidioc_g_fmt_vid_cap           = camif_g_fmt_vid_cap,
+	.vidioc_try_fmt_vid_cap         = camif_try_fmt_vid_cap,
+	.vidioc_s_fmt_vid_cap           = camif_s_fmt_vid_cap,
+
+	.vidioc_querystd        = camif_querystd,
+	.vidioc_g_std           = camif_g_std,
+	.vidioc_s_std           = camif_s_std,
+
+	.vidioc_enum_input      = camif_enum_input,
+	.vidioc_g_input         = camif_g_input,
+	.vidioc_s_input         = camif_s_input,
+
+	.vidioc_g_parm          = camif_g_parm,
+	.vidioc_s_parm          = camif_s_parm,
+
+	.vidioc_reqbufs		= vb2_ioctl_reqbufs,
+	.vidioc_create_bufs     = vb2_ioctl_create_bufs,
+	.vidioc_prepare_buf     = vb2_ioctl_prepare_buf,
+	.vidioc_querybuf	= vb2_ioctl_querybuf,
+	.vidioc_qbuf		= vb2_ioctl_qbuf,
+	.vidioc_dqbuf		= vb2_ioctl_dqbuf,
+	.vidioc_expbuf		= vb2_ioctl_expbuf,
+	.vidioc_streamon	= vb2_ioctl_streamon,
+	.vidioc_streamoff	= vb2_ioctl_streamoff,
+};
+
+/*
+ * Queue operations
+ */
+
+static u32 camif_get_sizeimage(struct camif_priv *priv)
+{
+	struct v4l2_mbus_framefmt *outfmt;
+	const struct imx_media_pixfmt *outcc;
+
+	outfmt = &priv->format_mbus[priv->output_pad];
+	outcc = priv->cc[priv->output_pad];
+	return (outfmt->width * outfmt->height * outcc->bpp) >> 3;
+}
+
+static int camif_queue_setup(struct vb2_queue *vq,
+			     unsigned int *nbuffers, unsigned int *nplanes,
+			     unsigned int sizes[], struct device *alloc_devs[])
+{
+	struct camif_priv *priv = vb2_get_drv_priv(vq);
+	unsigned int count = *nbuffers;
+	u32 sizeimage;
+
+	if (!priv->src_sd)
+		return -EPIPE;
+
+	if (vq->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		return -EINVAL;
+
+	sizeimage = camif_get_sizeimage(priv);
+
+	count = min_t(unsigned int, count, IMX_MEDIA_MAX_RING_BUFS);
+	count = max_t(unsigned int, count, IMX_MEDIA_MIN_RING_BUFS);
+
+	while (sizeimage * count > VID_MEM_LIMIT)
+		count--;
+
+	if (count < IMX_MEDIA_MIN_RING_BUFS)
+		return -EINVAL;
+
+	*nplanes = 1;
+	*nbuffers = count;
+	sizes[0] = sizeimage;
+
+	return 0;
+}
+
+static int camif_buf_init(struct vb2_buffer *vb)
+{
+	struct vb2_queue *vq = vb->vb2_queue;
+	struct camif_priv *priv = vb2_get_drv_priv(vq);
+	struct imx_media_buffer *buf = to_imx_media_vb(vb);
+
+	if (!priv->src_sd)
+		return -EPIPE;
+
+	INIT_LIST_HEAD(&buf->list);
+
+	return 0;
+}
+
+static int camif_buf_prepare(struct vb2_buffer *vb)
+{
+	struct vb2_queue *vq = vb->vb2_queue;
+	struct camif_priv *priv = vb2_get_drv_priv(vq);
+	u32 sizeimage = camif_get_sizeimage(priv);
+	int ret;
+
+	if (!priv->src_sd)
+		return -EPIPE;
+
+	if (vb2_plane_size(vb, 0) < sizeimage) {
+		v4l2_err(&priv->sd,
+			 "data will not fit into plane (%lu < %lu)\n",
+			 vb2_plane_size(vb, 0), (long)sizeimage);
+		return -EINVAL;
+	}
+
+	vb2_set_plane_payload(vb, 0, sizeimage);
+
+	if (!priv->in_ring) {
+		priv->in_ring = imx_media_alloc_dma_buf_ring(
+			priv->md, &priv->src_sd->entity, &priv->sd.entity,
+			sizeimage, vq->num_buffers, false);
+		if (IS_ERR(priv->in_ring)) {
+			v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
+			ret = PTR_ERR(priv->in_ring);
+			priv->in_ring = NULL;
+			return ret;
+		}
+	}
+
+	ret = imx_media_dma_buf_queue_from_vb(priv->in_ring, vb);
+	if (ret)
+		goto free_ring;
+
+	return 0;
+
+free_ring:
+	imx_media_free_dma_buf_ring(priv->in_ring);
+	priv->in_ring = NULL;
+	return ret;
+}
+
+static void camif_buf_queue(struct vb2_buffer *vb)
+{
+	struct camif_priv *priv = vb2_get_drv_priv(vb->vb2_queue);
+	struct imx_media_buffer *buf = to_imx_media_vb(vb);
+	unsigned long flags;
+
+	spin_lock_irqsave(&priv->q_lock, flags);
+
+	list_add_tail(&buf->list, &priv->ready_q);
+
+	spin_unlock_irqrestore(&priv->q_lock, flags);
+}
+
+static int camif_start_streaming(struct vb2_queue *vq, unsigned int count)
+{
+	struct camif_priv *priv = vb2_get_drv_priv(vq);
+	u32 sizeimage = camif_get_sizeimage(priv);
+	struct imx_media_buffer *buf, *tmp;
+	unsigned long flags;
+	int ret;
+
+	if (vb2_is_streaming(vq))
+		return 0;
+
+	if (!priv->src_sd)
+		return -EPIPE;
+
+	if (!priv->in_ring) {
+		priv->in_ring = imx_media_alloc_dma_buf_ring(
+			priv->md, &priv->src_sd->entity, &priv->sd.entity,
+			sizeimage, vq->num_buffers, false);
+		if (IS_ERR(priv->in_ring)) {
+			v4l2_err(&priv->sd, "failed to alloc dma-buf ring\n");
+			ret = PTR_ERR(priv->in_ring);
+			priv->in_ring = NULL;
+			goto return_bufs;
+		}
+	}
+
+	ret = imx_media_pipeline_set_stream(priv->md, &priv->sd.entity,
+					    &priv->mp, true);
+	if (ret) {
+		v4l2_err(&priv->sd, "pipeline_set_stream failed with %d\n",
+			 ret);
+		goto free_ring;
+	}
+
+	priv->stop = false;
+
+	return 0;
+
+free_ring:
+	imx_media_free_dma_buf_ring(priv->in_ring);
+	priv->in_ring = NULL;
+return_bufs:
+	spin_lock_irqsave(&priv->q_lock, flags);
+	list_for_each_entry_safe(buf, tmp, &priv->ready_q, list) {
+		list_del(&buf->list);
+		vb2_buffer_done(&buf->vbuf.vb2_buf, VB2_BUF_STATE_QUEUED);
+	}
+	spin_unlock_irqrestore(&priv->q_lock, flags);
+	return ret;
+}
+
+static void camif_stop_streaming(struct vb2_queue *vq)
+{
+	struct camif_priv *priv = vb2_get_drv_priv(vq);
+	struct imx_media_buffer *frame;
+	unsigned long flags;
+	int ret;
+
+	if (!vb2_is_streaming(vq))
+		return;
+
+	spin_lock_irqsave(&priv->q_lock, flags);
+	priv->stop = true;
+	spin_unlock_irqrestore(&priv->q_lock, flags);
+
+	ret = imx_media_pipeline_set_stream(priv->md, &priv->sd.entity,
+					    &priv->mp, false);
+	if (ret)
+		v4l2_warn(&priv->sd, "pipeline_set_stream failed with %d\n",
+			  ret);
+
+	if (priv->in_ring) {
+		v4l2_warn(&priv->sd, "%s: in_ring was not freed\n",
+			  __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+		priv->in_ring = NULL;
+	}
+
+	/* release all active buffers */
+	spin_lock_irqsave(&priv->q_lock, flags);
+	while (!list_empty(&priv->ready_q)) {
+		frame = list_entry(priv->ready_q.next,
+				   struct imx_media_buffer, list);
+		list_del(&frame->list);
+		vb2_buffer_done(&frame->vbuf.vb2_buf, VB2_BUF_STATE_ERROR);
+	}
+	spin_unlock_irqrestore(&priv->q_lock, flags);
+}
+
+static struct vb2_ops camif_qops = {
+	.queue_setup	 = camif_queue_setup,
+	.buf_init        = camif_buf_init,
+	.buf_prepare	 = camif_buf_prepare,
+	.buf_queue	 = camif_buf_queue,
+	.wait_prepare	 = vb2_ops_wait_prepare,
+	.wait_finish	 = vb2_ops_wait_finish,
+	.start_streaming = camif_start_streaming,
+	.stop_streaming  = camif_stop_streaming,
+};
+
+/*
+ * File operations
+ */
+static int camif_open(struct file *file)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	int ret;
+
+	if (mutex_lock_interruptible(&priv->mutex))
+		return -ERESTARTSYS;
+
+	ret = v4l2_fh_open(file);
+	if (ret)
+		v4l2_err(&priv->sd, "v4l2_fh_open failed\n");
+
+	mutex_unlock(&priv->mutex);
+	return ret;
+}
+
+static int camif_release(struct file *file)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct vb2_queue *vq = &priv->buffer_queue;
+	int ret = 0;
+
+	mutex_lock(&priv->mutex);
+
+	if (file->private_data == vq->owner) {
+		vb2_queue_release(vq);
+		vq->owner = NULL;
+	}
+
+	v4l2_fh_release(file);
+	mutex_unlock(&priv->mutex);
+	return ret;
+}
+
+static const struct v4l2_file_operations camif_fops = {
+	.owner		= THIS_MODULE,
+	.open		= camif_open,
+	.release	= camif_release,
+	.poll		= vb2_fop_poll,
+	.unlocked_ioctl	= video_ioctl2,
+	.mmap		= vb2_fop_mmap,
+};
+
+static struct video_device camif_videodev = {
+	.fops		= &camif_fops,
+	.ioctl_ops	= &camif_ioctl_ops,
+	.minor		= -1,
+	.release	= video_device_release,
+	.vfl_dir	= VFL_DIR_RX,
+	.tvnorms	= V4L2_STD_NTSC | V4L2_STD_PAL | V4L2_STD_SECAM,
+};
+
+/*
+ * Subdev and media entity operations
+ */
+
+static void camif_new_dma_buf(struct camif_priv *priv)
+{
+	struct imx_media_dma_buf *dmabuf;
+	struct imx_media_buffer *buf;
+	enum vb2_buffer_state state;
+	struct vb2_buffer *vb;
+	unsigned long flags;
+
+	spin_lock_irqsave(&priv->q_lock, flags);
+
+	if (priv->stop || list_empty(&priv->ready_q))
+		goto unlock;
+
+	dmabuf = imx_media_dma_buf_dequeue(priv->in_ring);
+	if (!dmabuf)
+		goto unlock;
+
+	vb = dmabuf->vb;
+	buf = to_imx_media_vb(vb);
+	if (list_empty(&buf->list)) {
+		dev_dbg(priv->dev, "%s: buf%d not queued\n", __func__,
+			vb->index);
+		goto unlock;
+	}
+
+	dev_dbg(priv->dev, "%s: new buf%d\n", __func__, vb->index);
+	vb->timestamp = ktime_get_ns();
+	list_del_init(&buf->list);
+	state = dmabuf->status == IMX_MEDIA_BUF_STATUS_DONE ?
+		VB2_BUF_STATE_DONE : VB2_BUF_STATE_ERROR;
+	vb2_buffer_done(vb, state);
+unlock:
+	spin_unlock_irqrestore(&priv->q_lock, flags);
+}
+
+static long camif_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)
+{
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+	struct imx_media_dma_buf_ring **ring;
+
+	switch (cmd) {
+	case IMX_MEDIA_REQ_DMA_BUF_SINK_RING:
+		if (!priv->in_ring)
+			return -EINVAL;
+		ring = (struct imx_media_dma_buf_ring **)arg;
+		*ring = priv->in_ring;
+		break;
+	case IMX_MEDIA_NEW_DMA_BUF:
+		camif_new_dma_buf(priv);
+		break;
+	case IMX_MEDIA_REL_DMA_BUF_SINK_RING:
+		/* src indicates sink buffer ring can be freed */
+		if (!priv->in_ring)
+			return 0;
+		v4l2_info(sd, "%s: freeing sink ring\n", __func__);
+		imx_media_free_dma_buf_ring(priv->in_ring);
+		priv->in_ring = NULL;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static int camif_link_setup(struct media_entity *entity,
+			    const struct media_pad *local,
+			    const struct media_pad *remote, u32 flags)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+	struct video_device *vfd = &priv->vfd;
+	struct v4l2_subdev *remote_sd;
+	int ret = 0;
+
+	dev_dbg(priv->dev, "link setup %s -> %s", remote->entity->name,
+		local->entity->name);
+
+	if (is_media_entity_v4l2_video_device(remote->entity))
+		return 0;
+
+	WARN_ON(local->flags & MEDIA_PAD_FL_SOURCE);
+
+	remote_sd = media_entity_to_v4l2_subdev(remote->entity);
+
+	/* reset controls to refresh with inherited from subdevs */
+	v4l2_ctrl_handler_free(vfd->ctrl_handler);
+	v4l2_ctrl_handler_init(vfd->ctrl_handler, 0);
+
+	if (flags & MEDIA_LNK_FL_ENABLED) {
+		if (priv->src_sd)
+			return -EBUSY;
+		priv->src_sd = remote_sd;
+		ret = imx_media_inherit_controls(priv->md, vfd,
+						 &priv->src_sd->entity);
+	} else {
+		priv->src_sd = NULL;
+	}
+
+	return ret;
+}
+
+static int camif_enum_mbus_code(struct v4l2_subdev *sd,
+				struct v4l2_subdev_pad_config *cfg,
+				struct v4l2_subdev_mbus_code_enum *code)
+{
+	if (code->pad >= CAMIF_NUM_PADS)
+		return -EINVAL;
+
+	return imx_media_enum_format(&code->code, code->index, true, true);
+}
+
+static int camif_get_fmt(struct v4l2_subdev *sd,
+			 struct v4l2_subdev_pad_config *cfg,
+			 struct v4l2_subdev_format *sdformat)
+{
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+
+	if (sdformat->pad >= CAMIF_NUM_PADS)
+		return -EINVAL;
+
+	sdformat->format = priv->format_mbus[sdformat->pad];
+
+	return 0;
+}
+
+static int camif_set_fmt(struct v4l2_subdev *sd,
+			 struct v4l2_subdev_pad_config *cfg,
+			 struct v4l2_subdev_format *sdformat)
+{
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+	const struct imx_media_pixfmt *cc;
+	u32 code;
+
+	if (sdformat->pad >= CAMIF_NUM_PADS)
+		return -EINVAL;
+
+	if (vb2_is_busy(&priv->buffer_queue)) {
+		v4l2_err(&priv->sd, "%s queue busy\n", __func__);
+		return -EBUSY;
+	}
+
+	cc = imx_media_find_format(0, sdformat->format.code, true, true);
+	if (!cc) {
+		imx_media_enum_format(&code, 0, true, true);
+		cc = imx_media_find_format(0, code, true, true);
+		sdformat->format.code = cc->codes[0];
+	}
+
+	/* Output pad mirrors input pad, no limitations on input pads */
+	if (sdformat->pad == priv->output_pad)
+		sdformat->format = priv->format_mbus[priv->input_pad];
+
+	if (sdformat->which == V4L2_SUBDEV_FORMAT_TRY) {
+		cfg->try_fmt = sdformat->format;
+	} else {
+		priv->format_mbus[sdformat->pad] = sdformat->format;
+		priv->cc[sdformat->pad] = cc;
+	}
+
+	return 0;
+}
+
+static int camif_init_pads(struct camif_priv *priv)
+{
+	struct video_device *vfd = &priv->vfd;
+	struct imx_media_subdev *imxsd;
+	struct imx_media_pad *pad;
+	int i, ret;
+
+	imxsd = imx_media_find_subdev_by_sd(priv->md, &priv->sd);
+	if (IS_ERR(imxsd))
+		return PTR_ERR(imxsd);
+
+	if (imxsd->num_sink_pads != 1 || imxsd->num_src_pads != 1) {
+		v4l2_err(&priv->sd, "invalid num pads %d/%d\n",
+			 imxsd->num_sink_pads, imxsd->num_src_pads);
+		return -EINVAL;
+	}
+
+	priv->vd_pad.flags = MEDIA_PAD_FL_SINK;
+	ret = media_entity_pads_init(&vfd->entity, 1, &priv->vd_pad);
+	if (ret) {
+		v4l2_err(&priv->sd, "failed to init device node pad\n");
+		return ret;
+	}
+
+	for (i = 0; i < CAMIF_NUM_PADS; i++) {
+		pad = &imxsd->pad[i];
+		priv->pad[i] = pad->pad;
+		if (priv->pad[i].flags & MEDIA_PAD_FL_SINK)
+			priv->input_pad = i;
+		else
+			priv->output_pad = i;
+
+		/* set a default mbus format  */
+		ret = imx_media_init_mbus_fmt(&priv->format_mbus[i],
+					      640, 480, 0, V4L2_FIELD_NONE,
+					      &priv->cc[i]);
+		if (ret)
+			return ret;
+	}
+
+	return media_entity_pads_init(&priv->sd.entity, CAMIF_NUM_PADS,
+				      priv->pad);
+}
+
+static int camif_registered(struct v4l2_subdev *sd)
+{
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+	struct vb2_queue *vq = &priv->buffer_queue;
+	struct video_device *vfd = &priv->vfd;
+	struct v4l2_mbus_framefmt *infmt, *outfmt;
+	int ret;
+
+	/* get media device */
+	priv->md = dev_get_drvdata(sd->v4l2_dev->dev);
+
+	vfd->v4l2_dev = sd->v4l2_dev;
+
+	ret = video_register_device(vfd, VFL_TYPE_GRABBER, 0);
+	if (ret) {
+		v4l2_err(sd, "Failed to register video device\n");
+		return ret;
+	}
+
+	vq->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+	vq->io_modes = VB2_MMAP | VB2_DMABUF;
+	vq->drv_priv = priv;
+	vq->buf_struct_size = sizeof(struct imx_media_buffer);
+	vq->ops = &camif_qops;
+	vq->mem_ops = &vb2_dma_contig_memops;
+	vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
+	vq->lock = &priv->mutex;
+	vq->min_buffers_needed = 2;
+	vq->dev = priv->dev;
+
+	ret = vb2_queue_init(vq);
+	if (ret) {
+		v4l2_err(sd, "vb2_queue_init failed\n");
+		goto unreg;
+	}
+
+	INIT_LIST_HEAD(&priv->ready_q);
+
+	ret = camif_init_pads(priv);
+	if (ret) {
+		v4l2_err(sd, "camif_init_pads failed\n");
+		goto unreg;
+	}
+
+	/* create the link to our device node */
+	ret = media_create_pad_link(&sd->entity, priv->output_pad,
+				    &vfd->entity, 0,
+				    MEDIA_LNK_FL_IMMUTABLE |
+				    MEDIA_LNK_FL_ENABLED);
+	if (ret) {
+		v4l2_err(sd, "failed to create link to device node\n");
+		goto unreg;
+	}
+
+	/* setup default pad formats */
+	infmt = &priv->format_mbus[priv->input_pad];
+	outfmt = &priv->format_mbus[priv->output_pad];
+	ret = imx_media_init_mbus_fmt(outfmt, 640, 480, 0, V4L2_FIELD_NONE,
+				      &priv->cc[priv->output_pad]);
+	if (ret)
+		goto unreg;
+
+	*infmt = *outfmt;
+	priv->cc[priv->input_pad] = priv->cc[priv->output_pad];
+
+	priv->current_std = V4L2_STD_UNKNOWN;
+
+	v4l2_info(sd, "Registered %s as /dev/%s\n", vfd->name,
+		  video_device_node_name(vfd));
+
+	vfd->ctrl_handler = &priv->ctrl_hdlr;
+
+	return 0;
+unreg:
+	video_unregister_device(vfd);
+	return ret;
+}
+
+static void camif_unregistered(struct v4l2_subdev *sd)
+{
+	struct camif_priv *priv = v4l2_get_subdevdata(sd);
+	struct video_device *vfd = &priv->vfd;
+
+	mutex_lock(&priv->mutex);
+
+	if (video_is_registered(vfd)) {
+		video_unregister_device(vfd);
+		media_entity_cleanup(&vfd->entity);
+	}
+
+	mutex_unlock(&priv->mutex);
+}
+
+static const struct v4l2_subdev_internal_ops camif_internal_ops = {
+	.registered = camif_registered,
+	.unregistered = camif_unregistered,
+};
+
+static struct v4l2_subdev_core_ops camif_core_ops = {
+	.ioctl = camif_ioctl,
+};
+
+static struct media_entity_operations camif_entity_ops = {
+	.link_setup = camif_link_setup,
+	.link_validate = v4l2_subdev_link_validate,
+};
+
+static struct v4l2_subdev_pad_ops camif_pad_ops = {
+	.enum_mbus_code = camif_enum_mbus_code,
+	.get_fmt = camif_get_fmt,
+	.set_fmt = camif_set_fmt,
+};
+
+static struct v4l2_subdev_ops camif_subdev_ops = {
+	.pad = &camif_pad_ops,
+	.core = &camif_core_ops,
+};
+
+static int camif_probe(struct platform_device *pdev)
+{
+	struct imx_media_internal_sd_platformdata *pdata;
+	struct camif_priv *priv;
+	struct video_device *vfd;
+	int ret;
+
+	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return -ENOMEM;
+
+	platform_set_drvdata(pdev, priv);
+	priv->dev = &pdev->dev;
+
+	pdata = priv->dev->platform_data;
+
+	mutex_init(&priv->mutex);
+	spin_lock_init(&priv->q_lock);
+
+	v4l2_subdev_init(&priv->sd, &camif_subdev_ops);
+	v4l2_set_subdevdata(&priv->sd, priv);
+	priv->sd.internal_ops = &camif_internal_ops;
+	priv->sd.entity.ops = &camif_entity_ops;
+	priv->sd.entity.function = MEDIA_ENT_F_PROC_VIDEO_PIXEL_FORMATTER;
+	priv->sd.dev = &pdev->dev;
+	priv->sd.owner = THIS_MODULE;
+	/* get our group id and camif id */
+	priv->sd.grp_id = pdata->grp_id;
+	priv->id = (pdata->grp_id >> IMX_MEDIA_GRP_ID_CAMIF_BIT) - 1;
+	strncpy(priv->sd.name, pdata->sd_name, sizeof(priv->sd.name));
+	snprintf(camif_videodev.name, sizeof(camif_videodev.name),
+		 "%s devnode", pdata->sd_name);
+
+	priv->sd.flags = V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
+
+	vfd = &priv->vfd;
+	*vfd = camif_videodev;
+	vfd->lock = &priv->mutex;
+	vfd->queue = &priv->buffer_queue;
+
+	video_set_drvdata(vfd, priv);
+
+	v4l2_ctrl_handler_init(&priv->ctrl_hdlr, 0);
+
+	ret = v4l2_async_register_subdev(&priv->sd);
+	if (ret)
+		v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+
+	return ret;
+}
+
+static int camif_remove(struct platform_device *pdev)
+{
+	struct camif_priv *priv =
+		(struct camif_priv *)platform_get_drvdata(pdev);
+
+	v4l2_ctrl_handler_free(&priv->ctrl_hdlr);
+	v4l2_async_unregister_subdev(&priv->sd);
+	media_entity_cleanup(&priv->sd.entity);
+	v4l2_device_unregister_subdev(&priv->sd);
+
+	return 0;
+}
+
+static const struct platform_device_id camif_ids[] = {
+	{ .name = "imx-media-camif" },
+	{ },
+};
+MODULE_DEVICE_TABLE(platform, camif_ids);
+
+static struct platform_driver imx_camif_driver = {
+	.probe		= camif_probe,
+	.remove		= camif_remove,

drivers/staging/media/imx/imx-camif.c line 1000

Quoted 10 lines
drivers/staging/media/imx/imx-camif.c new file +10 -0
+	.driver		= {
+		.name	= "imx-media-camif",
+	},
+};
+
+module_platform_driver(imx_camif_driver);
+
+MODULE_DESCRIPTION("i.MX camera interface subdev driver");
+MODULE_AUTHOR("Steve Longerbeam <[email protected]>");
+MODULE_LICENSE("GPL");

Regards,

Hans

author maintainer
#97 lore raw

Re: [PATCH v3 20/24] media: imx: Add Camera Interface subdev driver

Quoted 9 lines · Hans Verkuil

drivers/staging/media/imx/imx-camif.c line 85

Quoted 6 lines · Steve Longerbeam
drivers/staging/media/imx/imx-camif.c new file +6 -0
+static int vidioc_querycap(struct file *file, void *fh,
+			   struct v4l2_capability *cap)
+{
+	strncpy(cap->driver, "imx-media-camif", sizeof(cap->driver) - 1);
+	strncpy(cap->card, "imx-media-camif", sizeof(cap->card) - 1);
+	cap->bus_info[0] = 0;

Should be set to something like 'platform:imx-media-camif'. v4l2-compliance should complain about this.

Right, I've fixed this already as part of v4l2-compliance testing.

Quoted 5 lines

drivers/staging/media/imx/imx-camif.c line 87

Quoted 3 lines
drivers/staging/media/imx/imx-camif.c new file +2 -0
+	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;

Set device_caps in struct video_device, then drop these two lines since the core will set these up based on the device_caps field in struct video_device.

done.

Quoted 18 lines
Quoted 18 lines
drivers/staging/media/imx/imx-camif.c new file +17 -0
+
+static int camif_enum_input(struct file *file, void *fh,
+			    struct v4l2_input *input)
+{
+	struct camif_priv *priv = video_drvdata(file);
+	struct imx_media_subdev *sensor;
+	int index = input->index;
+
+	sensor = imx_media_find_sensor(priv->md, &priv->sd.entity);
+	if (IS_ERR(sensor)) {
+		v4l2_err(&priv->sd, "no sensor attached\n");
+		return PTR_ERR(sensor);
+	}
+
+	if (index >= sensor->input.num)
+		return -EINVAL;
+
Quoted 8 lines

drivers/staging/media/imx/imx-camif.c line 208

Quoted 7 lines
drivers/staging/media/imx/imx-camif.c new file +7 -0
+	input->type = V4L2_INPUT_TYPE_CAMERA;
+	strncpy(input->name, sensor->input.name[index], sizeof(input->name));
+
+	if (index == priv->current_input) {
+		v4l2_subdev_call(sensor->sd, video, g_input_status,
+				 &input->status);
+		v4l2_subdev_call(sensor->sd, video, querystd, &input->std);

Wrong op, use g_tvnorms instead.

done.

Steve