Skip to contentSkip to conversation

media: platform: Add support for ISP Pass 1 on mt8183 SoC

RFC PATCH In review Frederic Chen 3 participants 25 messages Feb 5, 2019 – Mar 25, 2019

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover media: platform: Add support for ISP Pass 1 on mt8183 SoC

Hello,

This is a first version of the RFC patch series adding the driver for Pass 1 (P1) unit in Mediatek's camera ISP system on mt8183 SoC, which will be used in camera features of CrOS. It's the first time Mediatek develops ISP kernel drivers based on V4L2 and media controller framework. I posted the main part of the ISP Pass 1 driver as RFC to discuss first and would like some review comments on the overall architecture of the driver.

Pass 1 unit processes image signal from sensor devices and accepts the tuning parameters to adjust the image quality. It performs optical black correction, defect pixel correction, W/IR imbalance correction and lens shading correction for RAW processing.

The driver is implemented with V4L2 and media controller framework so we have the following entities to describe the ISP pass 1 path. (The current metadata interface used in meta input and partial meta nodes is only a temporary solution to kick off the driver development and is not ready to be reviewed yet):

1. meta input (output video device): connects to ISP P1 sub device. It accepts the tuning buffer from user.

2. ISP P1 (sub device): connects to partial meta 0, partial meta 1, main stream and packed out video devices. When processing an image, Pass 1 hardware supports multiple output images with different sizes and formats so it needs two capture video devices ("main stream" and "packed out") to return the image data to the user.

3. partial meta 0 (capture video device): return the statistics metadata.

4. partial meta 1 (capture video device): return the statistics metadata.

5. main stream (capture video device): return the processed image data which is used in capture scenario.

6. packed out (capture video device): return the processed image data which is used in preview scenario.

The overall file structure of the ISP Pass 1 driver is as following:

* mtk_cam.c: Controls the hardware dependent flow and configuration.
* mtk_cam-v4l2.c: High-level software context configuration.
* mtk_cam-v4l2-util.c: Implements V4L2 and vb2 ops.
* mtk_cam-dev-ctx-core.c: Common software flow of the driver.
* mtk_cam-dev.c: Implements context independent flow.
* mtk_cam-vpu.c: Communicates with the co-processor on the SoC through the VPU driver.
* mtk_cam-smem-drv.c: Provides the shared memory management required operations. We reserved a memory region for the co-processor and Pass 1 unit to exchange the tuning and configuration data.

Frederic Chen (2):
[media] dt-bindings: mt8183: Add binding for ISP Pass 1 shared memory
media: platform: Add Mediatek ISP Pass 1 driver KConfig

Jungo Lin (5):
dts: arm64: mt8183: Add ISP Pass 1 shared memory node [media] dt-bindings: mt8183: Added CAM-SMEM dt-bindings [media] dt-bindings: mt8183: Added camera ISP Pass 1 dt-bindings
dts: arm64: mt8183: Add ISP Pass 1 nodes
[media] platform: mtk-isp: Add Mediatek ISP Pass 1 driver

 .../bindings/media/mediatek,cam_smem.txt           |   32 +
 .../bindings/media/mediatek,mt8183-camisp.txt      |   59 +
 .../mediatek,reserve-memory-cam_smem.txt           |   44 +
 arch/arm64/boot/dts/mediatek/mt8183.dtsi           |   54 +
 drivers/media/platform/Kconfig                     |    2 +
 drivers/media/platform/Makefile                    |    2 +
 drivers/media/platform/mtk-isp/Kconfig             |   21 +
 drivers/media/platform/mtk-isp/Makefile            |   14 +
 drivers/media/platform/mtk-isp/isp_50/Makefile     |   17 +
 drivers/media/platform/mtk-isp/isp_50/cam/Makefile |   35 +
 .../platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h      |  327 ++++
 .../mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c      |  986 +++++++++++++
 .../platform/mtk-isp/isp_50/cam/mtk_cam-dev.c      |  381 +++++
 .../platform/mtk-isp/isp_50/cam/mtk_cam-dev.h      |  204 +++
 .../platform/mtk-isp/isp_50/cam/mtk_cam-regs.h     |  146 ++
 .../platform/mtk-isp/isp_50/cam/mtk_cam-smem-drv.c |  452 ++++++
 .../platform/mtk-isp/isp_50/cam/mtk_cam-smem.h     |   27 +
 .../mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c         | 1555 ++++++++++++++++++++
 .../mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h         |   49 +
 .../platform/mtk-isp/isp_50/cam/mtk_cam-v4l2.c     |  288 ++++
 .../platform/mtk-isp/isp_50/cam/mtk_cam-v4l2.h     |   40 +
 .../platform/mtk-isp/isp_50/cam/mtk_cam-vpu.c      |  466 ++++++
 .../platform/mtk-isp/isp_50/cam/mtk_cam-vpu.h      |  158 ++
 .../media/platform/mtk-isp/isp_50/cam/mtk_cam.c    | 1235 ++++++++++++++++
 .../media/platform/mtk-isp/isp_50/cam/mtk_cam.h    |  347 +++++
 25 files changed, 6941 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/mediatek,cam_smem.txt
 create mode 100644 Documentation/devicetree/bindings/media/mediatek,mt8183-camisp.txt
 create mode 100644 Documentation/devicetree/bindings/reserved-memory/mediatek,reserve-memory-cam_smem.txt
 create mode 100644 drivers/media/platform/mtk-isp/Kconfig
 create mode 100644 drivers/media/platform/mtk-isp/Makefile
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/Makefile
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/Makefile
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem-drv.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.h
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2.h
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-vpu.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-vpu.h
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.h
Signature · 1 line
1.9.1
#1 lore raw

RFC PATCH 1/7 dt-bindings: mt8183: Add binding for ISP Pass 1 shared memory

This patch adds the binding for describing the shared memory used to exchange configuration and tuning data between the co-processor and Pass 1 (P1) unit of the camera ISP system on Mediatek SoCs.

---
 .../mediatek,reserve-memory-cam_smem.txt           | 44 ++++++++++++++++++++++
 1 file changed, 44 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/reserved-memory/mediatek,reserve-memory-cam_smem.txt
Documentation/devicetree/bindings/reserved-memory/mediatek,reserve-memory-cam_smem.txt new file +44 -0
new file mode 100644
index 0000000..05c1bf1
@@ -0,0 +1,44 @@
+Mediatek ISP Pass 1 Shared Memory binding
+
+This binding describes the shared memory, which serves the purpose of
+describing the shared memory region used to exchange data between Pass 1
+unit of Image Signal Processor (ISP) and the co-processor in Mediatek
+SoCs.
+
+The co-processor doesn't have the iommu so we need to use the physical
+address to access the shared buffer in the firmware.
+
+The Pass 1 unit of ISP can access memory through the iommu so it
+uses the dma address to access the memory region.
+(See iommu/mediatek,iommu.txt for the detailed description of Mediatek IOMMU)
+
+
+Required properties:
+
+- compatible: must be "mediatek,reserve-memory-cam_smem"
+
+- reg: required for static allocation (see reserved-memory.txt for
+  the detailed usage)
+
+- alloc-range: required for dynamic allocation. The range must
+  between 0x00000400 and 0x100000000 due to the co-processer's
+  addressing limitation
+
+- size: required for dynamic allocation. The unit is bytes.
+
+
+Example:
+
+The following example shows the ISP Pass1 shared memory setup for MT8183.
+
+	reserved-memory {
+		#address-cells = <2>;
+		#size-cells = <2>;
+		ranges;
+		reserve-memory-cam_smem {
+			compatible = "mediatek,reserve-memory-cam_smem";
+			size = <0 0x1400000>;
+			alignment = <0 0x1000>;
+			alloc-ranges = <0 0x40000000 0 0x50000000>;
+		};
+	};
Signature · 1 line
1.9.1
#2 lore raw

RFC PATCH 2/7 dts: arm64: mt8183: Add ISP Pass 1 shared memory node

In reply to Frederic Chen

From: Jungo Lin <[email protected]>

This patch adds a shared memory region used on mt8183 for exchanging tuning data between co-processor and the Pass 1 unit of the camera ISP system

---
 arch/arm64/boot/dts/mediatek/mt8183.dtsi | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
arch/arm64/boot/dts/mediatek/mt8183.dtsi +14 -0
index 63db9cc..3b85a6c 100644
@@ -109,6 +109,19 @@
 		};
 	};
 
+	reserved-memory {
+		#address-cells = <2>;
+		#size-cells = <2>;
+		ranges;
+		reserve-memory-cam_smem {
+			compatible = "mediatek,reserve-memory-cam_smem";
+			no-map;
+			size = <0 0x01400000>; /*20 MB share mem size */
+			alignment = <0 0x1000>;
+			alloc-ranges = <0 0x40000000 0 0x10000000>;
+		};
+	};
+
 	pmu-a53 {
 		compatible = "arm,cortex-a53-pmu";
 		interrupt-parent = <&gic>;
@@ -405,4 +418,5 @@
 		reg = <0 0x1a000000 0 0x1000>;
 		#clock-cells = <1>;
 	};
+
 };
Signature · 1 line
1.9.1
#3 lore raw

RFC PATCH 3/7 dt-bindings: mt8183: Added CAM-SMEM dt-bindings

In reply to Frederic Chen

From: Jungo Lin <[email protected]>

This patch adds the DT binding documentation for the shared memory between Pass 1 unit of the camera ISP system and the co-processor in Mediatek SoCs.

---
 .../bindings/media/mediatek,cam_smem.txt           | 32 ++++++++++++++++++++++
 1 file changed, 32 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/mediatek,cam_smem.txt
Documentation/devicetree/bindings/media/mediatek,cam_smem.txt new file +32 -0
new file mode 100644
index 0000000..d0d9c17
@@ -0,0 +1,32 @@
+Mediatek Camera ISP Pass 1 Shared Memory Device
+
+Mediatek Camera ISP Pass 1 Shared Memory Device is used to manage shared
+memory among CPU, Camera ISP Pass 1 hardware and coprocessor. The Camera
+ISP Pass 1 is a hardware unit for processing image signal from the image
+sensor. Camera ISP Pass 1 is responsible for RAW processing and 3A tuning.
+
+It is associated with a reserved memory region
+(Please see Documentation\devicetree\bindings\reserved-memory\mediatek,
+reserve-memory-cam_smem.txt) and and provides the context to
+allocate memory with dma addresses.
+
+Required properties:
+- compatible: Should be "mediatek,cam_smem"
+
+- iommus: should point to the respective IOMMU block with master port
+  as argument, see Documentation/devicetree/bindings/iommu/mediatek,iommu.txt
+  for details.
+
+- mediatek,larb: must contain the local arbiters in the current SOCs, see
+  Documentation/devicetree/bindings/memory-controllers/mediatek,smi-larb.txt
+  for details.
+
+Example:
+	cam_smem: cam_smem {
+		compatible = "mediatek,cam_smem";
+		mediatek,larb = <&larb3>,
+				<&larb6>;
+		iommus = <&iommu M4U_PORT_CAM_LSCI0>,
+				<&iommu M4U_PORT_CAM_LSCI1>,
+				<&iommu M4U_PORT_CAM_BPCI>;
+	};
Signature · 1 line
1.9.1
#4 lore raw

RFC PATCH 4/7 dt-bindings: mt8183: Added camera ISP Pass 1 dt-bindings

In reply to Frederic Chen

From: Jungo Lin <[email protected]>

This patch adds DT binding document for the Pass 1 (P1) unit in Mediatek's camera ISP system. The Pass 1 unit grabs the sensor data out from the sensor interface, applies ISP effects and writes the image data to DRAM.

---
 .../bindings/media/mediatek,mt8183-camisp.txt      | 59 ++++++++++++++++++++++
 1 file changed, 59 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/mediatek,mt8183-camisp.txt
Documentation/devicetree/bindings/media/mediatek,mt8183-camisp.txt new file +59 -0
new file mode 100644
index 0000000..ba16b4e
@@ -0,0 +1,59 @@
+* Mediatek Image Signal Processor Pass 1 (ISP P1)
+
+The Pass 1 unit of Mediatek's camera ISP system grabs the sensor data out
+from the sensor interface, applies ISP effects and writes the image data
+to DRAM. Furthermore, Pass 1 unit has the ability to output two different
+resolutions frames at the same time to increase the performance of the
+camera application.
+
+Required properties:
+- compatible: "mediatek,mt8183-camisp" for MT8183
+- reg: Must contain an entry for each entry in reg-names.
+- interrupts: interrupt number to the cpu.
+- iommus: should point to the respective IOMMU block with master port
+  as argument, see Documentation/devicetree/bindings/iommu/mediatek,iommu.txt
+  for details.
+- power-domains : a phandle to the power domain of this local arbiter.
+- mediatek,smi : a phandle to the smi_common node.
+- clocks: device clocks, see
+  Documentation/devicetree/bindings/clock/clock-bindings.txt for details.
+- clock-names: must be "CAMSYS_CAM_CGPDN" and "CAMSYS_CAMTG_CGPDN".
+- mediatek,larb: must contain the local arbiters in the current SOCs, see
+  Documentation/devicetree/bindings/memory-controllers/mediatek,smi-larb.txt
+  for details.
+- mediatek,vpu : the node of video processor unit
+- smem_device : the shared memory device managing the shared memory between
+  Pass 1 unit and the video processor unit
+
+Example:
+	camisp: camisp@1a000000 {
+		compatible = "mediatek,mt8183-camisp", "syscon";
+		reg = <0 0x1a000000 0 0x1000>,
+		      <0 0x1a003000 0 0x1000>,
+		      <0 0x1a004000 0 0x2000>,
+		      <0 0x1a006000 0 0x2000>;
+		reg-names = "camisp",
+		            "cam1",
+		            "cam2",
+		            "cam3";
+		interrupts = <GIC_SPI 253 IRQ_TYPE_LEVEL_LOW>,
+				<GIC_SPI 254 IRQ_TYPE_LEVEL_LOW>,
+				<GIC_SPI 255 IRQ_TYPE_LEVEL_LOW>;
+		interrupt-names = "cam1",
+		            "cam2",
+		            "cam3";
+		iommus = <&iommu M4U_PORT_CAM_LSCI0>,
+				<&iommu M4U_PORT_CAM_LSCI1>,
+				<&iommu M4U_PORT_CAM_BPCI>;
+		#clock-cells = <1>;
+		power-domains = <&scpsys MT8183_POWER_DOMAIN_CAM>;
+		/* Camera CCF */
+		clocks = <&camsys CLK_CAM_CAM>,
+				<&camsys CLK_CAM_CAMTG>;
+		clock-names = "CAMSYS_CAM_CGPDN",
+				"CAMSYS_CAMTG_CGPDN";
+		mediatek,larb = <&larb3>,
+				<&larb6>;
+		mediatek,vpu = <&vpu>;
+		smem_device = <&cam_smem>;
+	};
Signature · 1 line
1.9.1
#5 lore raw

RFC PATCH 5/7 dts: arm64: mt8183: Add ISP Pass 1 nodes

In reply to Frederic Chen

From: Jungo Lin <[email protected]>

Add nodes for Pass 1 unit of Mediatek's camera ISP system. Pass 1 unit embedded in Mediatek SOCs, works with the co-processor to process image signal from the image sensor and output RAW data.

---
 arch/arm64/boot/dts/mediatek/mt8183.dtsi | 40 ++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
arch/arm64/boot/dts/mediatek/mt8183.dtsi +40 -0
index 3b85a6c..780d2c5 100644
@@ -419,4 +419,44 @@
 		#clock-cells = <1>;
 	};
 
+	cam_smem: cam_smem {
+		compatible = "mediatek,cam_smem";
+		mediatek,larb = <&larb3>,
+				<&larb6>;
+		iommus = <&iommu M4U_PORT_CAM_LSCI0>,
+				<&iommu M4U_PORT_CAM_LSCI1>,
+				<&iommu M4U_PORT_CAM_BPCI>;
+	};
+
+	camisp: camisp@1a000000 {
+		compatible = "mediatek,mt8183-camisp", "syscon";
+		reg = <0 0x1a000000 0 0x1000>,
+			<0 0x1a003000 0 0x1000>,
+			<0 0x1a004000 0 0x2000>,
+			<0 0x1a006000 0 0x2000>;
+		reg-names = "camisp",
+					"cam1",
+					"cam2",
+					"cam3";
+		interrupts = <GIC_SPI 253 IRQ_TYPE_LEVEL_LOW>,
+				<GIC_SPI 254 IRQ_TYPE_LEVEL_LOW>,
+				<GIC_SPI 255 IRQ_TYPE_LEVEL_LOW>;
+		interrupt-names = "cam1",
+					"cam2",
+					"cam3";
+		iommus = <&iommu M4U_PORT_CAM_LSCI0>,
+				<&iommu M4U_PORT_CAM_LSCI1>,
+				<&iommu M4U_PORT_CAM_BPCI>;
+		#clock-cells = <1>;
+		power-domains = <&scpsys MT8183_POWER_DOMAIN_CAM>;
+		/* Camera CCF */
+		clocks = <&camsys CLK_CAM_CAM>,
+				<&camsys CLK_CAM_CAMTG>;
+		clock-names = "CAMSYS_CAM_CGPDN",
+				"CAMSYS_CAMTG_CGPDN";
+		mediatek,larb = <&larb3>,
+				<&larb6>;
+		mediatek,vpu = <&vpu>;
+		smem_device = <&cam_smem>;
+	};
 };
Signature · 1 line
1.9.1
#6 lore raw

RFC PATCH 6/7 media: platform: Add Mediatek ISP Pass 1 driver KConfig

In reply to Frederic Chen

This patch adds KConfig for Pass 1 (P1) unit driver of Mediatek's camera ISP system. ISP P1 unit is embedded in Mediatek SOCs. It provides RAW processing which includes optical black correction, defect pixel correction, W/IR imbalance correction and lens shading correction.

---
 drivers/media/platform/Kconfig         |  2 ++
 drivers/media/platform/mtk-isp/Kconfig | 21 +++++++++++++++++++++
 2 files changed, 23 insertions(+)
 create mode 100644 drivers/media/platform/mtk-isp/Kconfig
drivers/media/platform/Kconfig +2 -0
index a505e9f..ef08d48 100644
@@ -32,6 +32,8 @@ source "drivers/media/platform/davinci/Kconfig"
 
 source "drivers/media/platform/omap/Kconfig"
 
+source "drivers/media/platform/mtk-isp/Kconfig"
+
 config VIDEO_ASPEED
 	tristate "Aspeed AST2400 and AST2500 Video Engine driver"
 	depends on VIDEO_V4L2
drivers/media/platform/mtk-isp/Kconfig new file +21 -0
new file mode 100644
index 0000000..999eb24
@@ -0,0 +1,21 @@
+config VIDEO_MEDIATEK_ISP_PASS1_SUPPORT
+	bool "Mediatek pass 1 image processing function"
+
+	select DMA_SHARED_BUFFER
+	select VIDEO_V4L2_SUBDEV_API
+	select VIDEOBUF2_DMA_CONTIG
+	select VIDEOBUF2_CORE
+	select VIDEOBUF2_V4L2
+	select VIDEOBUF2_MEMOPS
+	select VIDEOBUF2_VMALLOC
+	select MEDIA_CONTROLLER
+
+	default n
+	help
+		Pass 1 driver controls 3A (autofocus, exposure,
+		and white balance) with tuning feature and output
+		the first capture image buffer in Mediatek's camera system.
+
+		Choose y if you want to use Mediatek SoCs to create image
+		vcapture application such as video recording and still image
+		capture.
Signature · 1 line
1.9.1
#7 lore raw

RFC PATCH 7/7 platform: mtk-isp: Add Mediatek ISP Pass 1 driver

In reply to Frederic Chen

From: Jungo Lin <jungo.lin-NuS5LvNUpcJWk0Htik3J/[email protected]>

This patch adds the driver for Pass unit in Mediatek's camera ISP system. Pass 1 unit is embedded in Mediatek SOCs. It provides RAW processing which includes optical black correction, defect pixel correction, W/IR imbalance correction and lens shading correction.

The mtk-isp directory will contain drivers for multiple IP blocks found in Mediatek ISP system. It will include ISP Pass 1 driver, sensor interface driver, DIP driver and face detection driver.

---
 drivers/media/platform/Makefile                    |    2 +
 drivers/media/platform/mtk-isp/Makefile            |   14 +
 drivers/media/platform/mtk-isp/isp_50/Makefile     |   17 +
 drivers/media/platform/mtk-isp/isp_50/cam/Makefile |   35 +
 .../platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h      |  327 ++++
 .../mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c      |  986 +++++++++++++
 .../platform/mtk-isp/isp_50/cam/mtk_cam-dev.c      |  381 +++++
 .../platform/mtk-isp/isp_50/cam/mtk_cam-dev.h      |  204 +++
 .../platform/mtk-isp/isp_50/cam/mtk_cam-regs.h     |  146 ++
 .../platform/mtk-isp/isp_50/cam/mtk_cam-smem-drv.c |  452 ++++++
 .../platform/mtk-isp/isp_50/cam/mtk_cam-smem.h     |   27 +
 .../mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c         | 1555 ++++++++++++++++++++
 .../mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h         |   49 +
 .../platform/mtk-isp/isp_50/cam/mtk_cam-v4l2.c     |  288 ++++
 .../platform/mtk-isp/isp_50/cam/mtk_cam-v4l2.h     |   40 +
 .../platform/mtk-isp/isp_50/cam/mtk_cam-vpu.c      |  466 ++++++
 .../platform/mtk-isp/isp_50/cam/mtk_cam-vpu.h      |  158 ++
 .../media/platform/mtk-isp/isp_50/cam/mtk_cam.c    | 1235 ++++++++++++++++
 .../media/platform/mtk-isp/isp_50/cam/mtk_cam.h    |  347 +++++
 19 files changed, 6729 insertions(+)
 create mode 100644 drivers/media/platform/mtk-isp/Makefile
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/Makefile
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/Makefile
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem-drv.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.h
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2.h
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-vpu.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-vpu.h
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c
 create mode 100644 drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.h
drivers/media/platform/Makefile +2 -0
index e6deb25..9773b3a 100644
@@ -103,3 +103,5 @@ obj-y					+= meson/
 obj-y					+= cros-ec-cec/
 
 obj-$(CONFIG_VIDEO_SUN6I_CSI)		+= sunxi/sun6i-csi/
+
+obj-y	+= mtk-isp/
drivers/media/platform/mtk-isp/Makefile new file +14 -0
new file mode 100644
index 0000000..84d575a
@@ -0,0 +1,14 @@
+#
+# Copyright (C) 2018 MediaTek Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+
+obj-y += isp_50/
drivers/media/platform/mtk-isp/isp_50/Makefile new file +17 -0
new file mode 100644
index 0000000..b4718a2
@@ -0,0 +1,17 @@
+#
+# Copyright (C) 2018 MediaTek Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+
+ifeq ($(CONFIG_VIDEO_MEDIATEK_ISP_PASS1_SUPPORT),y)
+obj-y += cam/
+endif
+
drivers/media/platform/mtk-isp/isp_50/cam/Makefile new file +35 -0
new file mode 100644
index 0000000..12dea40
@@ -0,0 +1,35 @@
+#
+# Copyright (C) 2018 MediaTek Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+
+ccflags-y += -I$(srctree)/drivers/media/platform/mtk-vpu
+ccflags-y += -I$(srctree)/drivers/media/platform/mtk-isp/isp_50/cam
+
+obj-y += mtk_cam-vpu.o
+obj-y += mtk_cam.o
+obj-y += mtk_cam-v4l2.o
+
+# To provide alloc context managing memory shared
+# between CPU and ISP coprocessor
+mtk_cam_smem-objs := \
+mtk_cam-smem-drv.o
+
+obj-y += mtk_cam_smem.o
+
+# Utilits to provide frame-based streaming model
+# with v4l2 user interfaces
+mtk_cam_util-objs := \
+mtk_cam-dev.o \
+mtk_cam-v4l2-util.o \
+mtk_cam-dev-ctx-core.o \
+
+obj-y += mtk_cam_util.o
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +327 -0

327 lines not on this page · raw message

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +986 -0

986 lines not on this page · raw message

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.c new file +381 -0

381 lines not on this page · raw message

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +204 -0

204 lines not on this page · raw message

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h new file +146 -0

146 lines not on this page · raw message

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem-drv.c new file +452 -0

452 lines not on this page · raw message

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.h new file +27 -0
new file mode 100644
index 0000000..4e1cf20
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2018 MediaTek Inc.
+ * Author: Frederic Chen <frederic.chen-NuS5LvNUpcJWk0Htik3J/[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __MTK_CAM_SMEM_H__
+#define __MTK_CAM_SMEM_H__
+
+#include <linux/dma-mapping.h>
+
+phys_addr_t mtk_cam_smem_iova_to_phys(struct device *smem_dev,
+				      dma_addr_t iova);
+
+void mtk_cam_smem_enable_mpu(struct device *smem_dev);
+
+#endif /*__MTK_CAM_SMEM_H__*/
+
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +1555 -0

406 lines not on this page · raw message

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.h new file +49 -0

49 lines not on this page · raw message

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2.c new file +288 -0

288 lines not on this page · raw message

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2.h new file +40 -0
new file mode 100644
index 0000000..649c175

Lines of this file are not shown. View the raw message.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-vpu.c new file +466 -0

466 lines not on this page · raw message

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-vpu.h new file +158 -0

158 lines not on this page · raw message

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.c new file +1235 -0

1,235 lines not on this page · raw message

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam.h new file +347 -0

347 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
1.9.1

Part of the diff is not shown here. View the raw message.

#8 lore raw

Re: [RFC PATCH V0 7/7] [media] platform: mtk-isp: Add Mediatek ISP Pass 1 driver

(() . ( strHi Frederic, Jungo,

Quoted 9 lines · Frederic Chen

From: Jungo Lin <[email protected]>

This patch adds the driver for Pass unit in Mediatek's camera ISP system. Pass 1 unit is embedded in Mediatek SOCs. It provides RAW processing which includes optical black correction, defect pixel correction, W/IR imbalance correction and lens shading correction.

Quoted 4 lines

The mtk-isp directory will contain drivers for multiple IP blocks found in Mediatek ISP system. It will include ISP Pass 1 driver, sensor interface driver, DIP driver and face detection driver.

Thanks for the patches! Please see my comments inline.

[snip]

Quoted 28 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +22 -0
new file mode 100644
index 0000000..11a60a6
@@ -0,0 +1,327 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2018 MediaTek Inc.
+ * Author: Frederic Chen <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __MTK_CAM_CTX_H__
+#define __MTK_CAM_CTX_H__
+
+#include <linux/types.h>
+#include <linux/videodev2.h>
+#include <media/v4l2-ctrls.h>
+#include <media/videobuf2-core.h>

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 32

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +10 -0
+#include <media/v4l2-subdev.h>
+#include "mtk_cam-v4l2-util.h"
+
+#define MTK_CAM_CTX_QUEUES (16)
+#define MTK_CAM_CTX_FRAME_BUNDLE_BUFFER_MAX (MTK_CAM_CTX_QUEUES)
+#define MTK_CAM_CTX_DESC_MAX (MTK_CAM_CTX_QUEUES)
+
+#define MTK_CAM_CTX_MODE_DEBUG_OFF (0)
+#define MTK_CAM_CTX_MODE_DEBUG_BYPASS_JOB_TRIGGER (1)
+#define MTK_CAM_CTX_MODE_DEBUG_BYPASS_ALL (2)

nit: No need for parentheses for simple values. Also please align macro values together using tabs.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 35

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +3 -0
+
+#define MTK_CAM_GET_CTX_ID_FROM_SEQUENCE(sequence) \
+       ((sequence) >> 16 & 0x0000FFFF)

Sounds like a job for a static inline function, with appropriate argument and return types enforced by the compiler.

Quoted 17 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +17 -0
+
+#define MTK_CAM_CTX_META_BUF_DEFAULT_SIZE (1110 * 1024)
+
+struct mtk_cam_ctx;
+struct mtk_cam_ctx_open_param;
+struct mtk_cam_ctx_release_param;
+struct mtk_cam_ctx_streamon_param;
+struct mtk_cam_ctx_streamoff_param;
+struct mtk_cam_ctx_start_param;
+struct mtk_cam_ctx_finish_param;
+
+/* struct mtk_cam_ctx_ops - background hardware driving ops */
+/* sdefines background driver specific callback APIs  */
+struct mtk_cam_ctx_ops {
+       int (*open)(struct mtk_cam_ctx *dev_ctx,
+                   struct mtk_cam_ctx_open_param *param);
+       int (*release)(struct mtk_cam_ctx *dev_ctx,

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 62

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +10 -0
+                      struct mtk_cam_ctx_release_param *param);
+       int (*start)(struct mtk_cam_ctx *dev_ctx,
+                    struct mtk_cam_ctx_start_param *param);
+       int (*finish)(struct mtk_cam_ctx *dev_ctx,
+                     struct mtk_cam_ctx_finish_param *param);
+       int (*streamon)(struct mtk_cam_ctx *dev_ctx,
+                       struct mtk_cam_ctx_streamon_param *param);
+       int (*streamoff)(struct mtk_cam_ctx *dev_ctx,
+                        struct mtk_cam_ctx_streamoff_param *param);
+};

We only have one driver here and these ops look really close to what vb2 already provides. Why do we need this indirection?

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 66

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +4 -0
+
+/* Attributes setup by device context owner */
+struct mtk_cam_ctx_queue_desc {
+       int id; /* id of the context queue */

Please use kerneldoc syntax for documenting the structures.
(https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txtjjj

Quoted 9 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +9 -0
+       char *name;
+       /* Will be exported to media entity name */
+       int capture;
+       /* true for capture queue (device to user), false for output queue */
+       /* (from user to device) */
+       int image;
+       /* Using the cam_smem_drv as alloc ctx or not */
+       int smem_alloc;
+       /* true for image, false for meta data */

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 85

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +10 -0
+       unsigned int dma_port; /*The dma port associated to the buffer*/
+       /* Supported format */
+       struct mtk_cam_ctx_format *fmts;
+       int num_fmts;
+       /* Default format of this queue */
+       int default_fmt_idx;
+};
+
+/* Supported format and the information used for */
+/* size calculation */

CodingStyle:
/*
* This is the kernel style for multi-line
* comments.
*/

Quoted 15 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +15 -0
+struct mtk_cam_ctx_meta_format {
+       u32 dataformat;
+       u32 max_buffer_size;
+       u8 flags;
+};
+
+struct mtk_cam_ctx_img_format {
+       u32     pixelformat;
+       u8      depth[VIDEO_MAX_PLANES];
+       u8      row_depth[VIDEO_MAX_PLANES];
+       u8      num_planes;
+       u32     flags;
+};
+
+struct mtk_cam_ctx_format {

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 110

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +10 -0
+       union {
+               struct mtk_cam_ctx_meta_format meta;
+               struct mtk_cam_ctx_img_format img;
+       } fmt;
+};
+
+union mtk_v4l2_fmt {
+       struct v4l2_pix_format_mplane pix_mp;
+       struct v4l2_meta_format meta;
+};

Why not just use the standard v4l2_format?

Quoted 17 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +17 -0
+
+/* Attributes setup by device context owner */
+struct mtk_cam_ctx_queues_setting {
+       int master;
+       /* The master input node to trigger the frame data enqueue */
+       struct mtk_cam_ctx_queue_desc *output_queue_descs;
+       int total_output_queues;
+       struct mtk_cam_ctx_queue_desc *capture_queue_descs;
+       int total_capture_queues;
+};
+
+struct mtk_cam_ctx_queue_attr {
+       int master;
+       int input_offset;
+       int total_num;
+};
+

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 137

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +10 -0
+/* Video node context. Since we use */
+/* mtk_cam_ctx_frame_bundle to manage enqueued */
+/* buffers by frame now, we don't use bufs filed of */
+/* mtk_cam_ctx_queue now */
+struct mtk_cam_ctx_queue {
+       union mtk_v4l2_fmt fmt;
+       struct mtk_cam_ctx_format *ctx_fmt;
+       /* Currently we used in standard v4l2 image format */
+       /* in the device context */
+       unsigned int width_pad; /* bytesperline, reserved */

What's the meaning of this field?

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 146

Quoted 9 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +9 -0
+       struct mtk_cam_ctx_queue_desc desc;
+       struct list_head bufs; /* Reserved, not used now */
+};
+
+enum mtk_cam_ctx_frame_bundle_state {
+       MTK_CAM_CTX_FRAME_NEW,  /* Not allocated */
+       MTK_CAM_CTX_FRAME_PREPARED, /* Allocated but has not be processed */
+       MTK_CAM_CTX_FRAME_PROCESSING,   /* Queued, waiting to be filled */
+};

Feels like duplicating the media_request::state. (and eliminating the benefit of the state machine being fully managed by the Request API core).

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 155

Quoted 9 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +9 -0
+
+/* The definiation is compatible with DIP driver's state definiation */
+/* currently and will be decoupled after further integration */
+enum mtk_cam_ctx_frame_data_state {
+       MTK_CAM_CTX_FRAME_DATA_EMPTY = 0, /* FRAME_STATE_INIT */
+       MTK_CAM_CTX_FRAME_DATA_DONE = 3, /* FRAME_STATE_DONE */
+       MTK_CAM_CTX_FRAME_DATA_STREAMOFF_DONE = 4, /*FRAME_STATE_STREAMOFF*/
+       MTK_CAM_CTX_FRAME_DATA_ERROR = 5, /*FRAME_STATE_ERROR*/
+};

What are these states for?

Quoted 7 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +7 -0
+
+struct mtk_cam_ctx_frame_bundle {
+       struct mtk_cam_ctx_buffer*
+               buffers[MTK_CAM_CTX_FRAME_BUNDLE_BUFFER_MAX];
+       int id;
+       int num_img_capture_bufs;
+       int num_img_output_bufs;

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 172

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +10 -0
+       int num_meta_capture_bufs;
+       int num_meta_output_bufs;
+       int last_index;
+       int state;
+       struct list_head list;
+};
+
+struct mtk_cam_ctx_frame_bundle_list {
+       struct list_head list;
+};

This sounds like duplicating the request API core functionality, which aggregates all the buffers in the request.

Quoted 22 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +22 -0
+
+struct mtk_cam_ctx {
+       struct platform_device *pdev;
+       struct platform_device *smem_device;
+       /* buffer queues will be added later */
+       unsigned short ctx_id;
+       char *device_name;
+       const struct mtk_cam_ctx_ops *ops;
+       struct mtk_cam_dev_node_mapping *mtk_cam_dev_node_map;
+       unsigned int dev_node_num;
+       /* mtk_cam_ctx_queue is the context for the video nodes */
+       struct mtk_cam_ctx_queue queue[MTK_CAM_CTX_QUEUES];
+       struct mtk_cam_ctx_queue_attr queues_attr;
+       atomic_t frame_param_sequence;
+       int streaming;
+       void *default_vb2_alloc_ctx;
+       void *smem_vb2_alloc_ctx;
+       struct v4l2_subdev_fh *fh;
+       struct mtk_cam_ctx_frame_bundle frame_bundles[VB2_MAX_FRAME];
+       struct mtk_cam_ctx_frame_bundle_list processing_frames;
+       struct mtk_cam_ctx_frame_bundle_list free_frames;
+       int enabled_dma_ports;

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 204

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +10 -0
+       int num_frame_bundle;
+       spinlock_t qlock; /* frame queues protection */
+};
+
+enum mtk_cam_ctx_buffer_state {
+       MTK_CAM_CTX_BUFFER_NEW,
+       MTK_CAM_CTX_BUFFER_PROCESSING,
+       MTK_CAM_CTX_BUFFER_DONE,
+       MTK_CAM_CTX_BUFFER_FAILED,
+};

Why these custom states? VB2 already manages buffer states for you and the driver should just obtain available buffers in the .buf_queue callback and return them with appropriate state using vb2_buffer_done() when they are no longer in use.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 212

Quoted 8 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +8 -0
+
+struct mtk_cam_ctx_buffer {
+       union mtk_v4l2_fmt fmt;
+       struct mtk_cam_ctx_format *ctx_fmt;
+       int capture;
+       int image;
+       int frame_id;
+       int user_sequence; /* Sequence number assigned by user */

What's this user_sequence? There shouldn't be any sequence coming from the user, since the requests are used to aggregate buffers.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 214

Quoted 2 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +2 -0
+       dma_addr_t daddr;
+       void *vaddr;

We shouldn't need to touch the buffers from the kernel driver.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 215

Quoted 1 line
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +1 -0
+       phys_addr_t paddr;

We shouldn't need physical address either. I suppose this is for the SCP, but then it should be a DMA address obtained from dma_map_*() with struct device pointer of the SCP.

Quoted 24 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +24 -0
+       unsigned int queue;
+       enum mtk_cam_ctx_buffer_state state;
+       struct list_head list;
+};
+
+struct mtk_cam_ctx_setting {
+       struct mtk_cam_ctx_ops *ops;
+       char *device_name;
+};
+
+struct mtk_cam_ctx_desc {
+       char *proc_dev_phandle;
+       /* The context device's compatble string name in device tree*/
+       int (*init)(struct mtk_cam_ctx *ctx);
+       /* configure the core functions of the device context */
+};
+
+struct mtk_cam_ctx_init_table {
+       int total_dev_ctx;
+       struct mtk_cam_ctx_desc *ctx_desc_tbl;
+};
+
+struct mtk_cam_ctx_open_param {
+       /* Bitmask used to notify that the DMA port is enabled or not */

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 249

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +10 -0
+       unsigned int enabled_dma_ports;
+};
+
+struct mtk_cam_ctx_streamon_param {
+       unsigned int enabled_dma_ports;
+};
+
+struct mtk_cam_ctx_streamoff_param {
+       unsigned int enabled_dma_ports;
+};

Hmm, the 3 structs above are identical. Is there a need to have all 3 of them?

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 258

Quoted 9 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +9 -0
+
+struct mtk_cam_ctx_start_param {
+       /* carry buffer information of the frame */
+       struct mtk_cam_ctx_frame_bundle *frame_bundle;
+};
+
+struct mtk_cam_ctx_release_param {
+       unsigned int enabled_dma_ports;
+};

One more identical struct.

[snip]

Quoted 48 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +42 -0
new file mode 100644
index 0000000..7d0197b
@@ -0,0 +1,986 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2018 MediaTek Inc.
+ * Author: Frederic Chen <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/device.h>
+#include <linux/platform_device.h>
+#include <media/videobuf2-dma-contig.h>
+#include <linux/dma-mapping.h>
+#include <media/v4l2-event.h>
+
+#include "mtk_cam-dev.h"
+#include "mtk_cam-v4l2-util.h"
+#include "mtk_cam-v4l2.h"
+#include "mtk_cam-smem.h"
+
+static struct mtk_cam_ctx_format *mtk_cam_ctx_find_fmt
+       (struct mtk_cam_ctx_queue *queue,
+        u32 format);
+
+static int mtk_cam_ctx_process_frame(struct mtk_cam_ctx *dev_ctx,
+                                    struct mtk_cam_ctx_frame_bundle
+                                    *frame_bundle);
+
+static int mtk_cam_ctx_free_frame(struct mtk_cam_ctx *dev_ctx,
+                                 struct mtk_cam_ctx_frame_bundle
+                                 *frame_bundle);
+
+static struct mtk_cam_ctx_frame_bundle *mtk_cam_ctx_get_free_frame
+       (struct mtk_cam_ctx *dev_ctx);
+
+static struct mtk_cam_ctx_frame_bundle *mtk_cam_ctx_get_processing_frame

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 52

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +10 -0
+(struct mtk_cam_ctx *dev_ctx, int frame_id);
+
+static int mtk_cam_ctx_init_frame_bundles(struct mtk_cam_ctx *dev_ctx);
+
+static void mtk_cam_ctx_queue_event_frame_done
+       (struct mtk_cam_ctx *dev_ctx,
+       struct mtk_cam_dev_frame_done_event_data *fdone);
+
+static void debug_bundle(struct mtk_cam_ctx  *dev_ctx,
+                        struct mtk_cam_ctx_frame_bundle *bundle_data);

Could you reorder the functions in this file so that there is no need for forward declarations? In general, a kernel source file should be ordered in such way that the reader can understand what's happening by reading from the top to bottom, without jumping between functions.

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +5 -0
+
+struct vb2_v4l2_buffer *mtk_cam_ctx_buffer_get_vb2_v4l2_buffer
+(struct mtk_cam_ctx_buffer *ctx_buf)
+{
+       struct mtk_cam_dev_buffer *dev_buf = NULL;

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 67

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +10 -0
+
+       if (!ctx_buf) {
+               pr_err("Failed to convert ctx_buf to dev_buf: Null pointer\n");
+               return NULL;
+       }
+
+       dev_buf = mtk_cam_ctx_buf_to_dev_buf(ctx_buf);
+
+       return &dev_buf->m2m2_buf.vbb;
+}

I also added a comment to where mtk_cam_dev_buffer struct is defined, but let me repeat here. I don't think we need to split these structs so much. We just need one driver-specific struct that encapsulates the vb2_v4l2_buffer struct, e.g. mtk_cam_buffer.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 80

Quoted 13 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +13 -0
+
+/* The helper to configure the device context */
+int mtk_cam_ctx_core_steup(struct mtk_cam_ctx *ctx,
+                          struct mtk_cam_ctx_setting *ctx_setting)
+{
+       if (!ctx || !ctx_setting)
+               return -EINVAL;
+
+       ctx->ops = ctx_setting->ops;
+       ctx->device_name = ctx_setting->device_name;
+
+       return 0;
+}

I don't understand what's the point of this. We only have one device, P1, and only one context ops, for this P1 device. Do we need this abstraction?

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 91

Quoted 11 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +11 -0
+
+int mtk_cam_ctx_core_queue_setup(struct mtk_cam_ctx *ctx,
+                                struct mtk_cam_ctx_queues_setting
+                                *queues_setting)
+{
+       int queue_idx = 0;
+       int i = 0;
+
+       for (i = 0; i < queues_setting->total_output_queues; i++) {
+               struct mtk_cam_ctx_queue_desc *queue_desc =
+                       queues_setting->output_queue_descs + i;

The typical kernel convention would be &queues_setting->output_queue_descs[i].

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 94

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +3 -0
+
+               if (!queue_desc)
+                       return -EINVAL;

I don't think this can ever happen, especially for i > 0.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 103

Quoted 9 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +9 -0
+
+               /* Since the *ctx->queue has been initialized to 0 */
+               /* when it is allocated with mtk_cam_dev , */
+               /* I don't initialize the struct here */
+               ctx->queue[queue_idx].desc = *queue_desc;
+               queue_idx++;
+       }
+
+       ctx->queues_attr.input_offset = queue_idx;

This input_offset doesn't seem to be used in the driver in any meaningful way. The only place it's read is in mtk_cam_dev_mem2mem2_init() and that could be easily replaced with checking for (!isp_dev->ctx.queue[i].capture).

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 111

Quoted 8 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +8 -0
+
+       /* Setup the capture queue */
+       for (i = 0; i < queues_setting->total_capture_queues; i++) {
+               struct mtk_cam_ctx_queue_desc *queue_desc =
+                       queues_setting->capture_queue_descs + i;
+
+               if (!queue_desc)
+                       return -EINVAL;

Ditto.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 124

Quoted 13 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +13 -0
+
+               /* Since the *ctx->queue has been initialized to 0 */
+               /* when allocating the memory, I don't */
+               /* reinitialied the struct here */
+               ctx->queue[queue_idx].desc = *queue_desc;
+               queue_idx++;
+       }
+
+       ctx->queues_attr.master = queues_setting->master;
+       ctx->queues_attr.total_num = queue_idx;
+       ctx->dev_node_num = ctx->queues_attr.total_num;
+       return 0;
+}

This function just copies some compile-time constant data at runtime. Why couldn't we just use the compile-time constants directly?

Also, I don't see vb2 queues handled here. What a V4L2 driver would normally do is:
- have a driver-private struct to represent the queue (e.g. mtk_cam_queue) and encapsulating vb2_queue,
- set vb2_queue::drv_priv to point to the mtk_cam_queue struct encapsulating it,
- both the vb2_queue and internal data fields would be initialized in the same function, to keep queue-initialization code in the same part of the driver.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 134

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +10 -0
+
+/* Mediatek ISP context core initialization */
+int mtk_cam_ctx_core_init(struct mtk_cam_ctx *ctx,
+                         struct platform_device *pdev, int ctx_id,
+       struct mtk_cam_ctx_desc *ctx_desc,
+       struct platform_device *proc_pdev,
+       struct platform_device *smem_pdev)
+{
+       /* Initialize main data structure */
+       int r = 0;

Please don't initialize return value variables, unless really necessary. It prevents the compiler from detecting forgotten assignments further in the function.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 143

Quoted 9 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +9 -0
+
+       ctx->smem_vb2_alloc_ctx = &smem_pdev->dev;
+       ctx->default_vb2_alloc_ctx = &pdev->dev;
+
+       if (IS_ERR((__force void *)ctx->smem_vb2_alloc_ctx))
+               pr_err("Failed to alloc vb2 dma context: smem_vb2_alloc_ctx");
+
+       if (IS_ERR((__force void *)ctx->default_vb2_alloc_ctx))
+               pr_err("Failed to alloc vb2 dma context: default_vb2_alloc_ctx");

It's impossible for the two conditions above to happen.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 148

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +5 -0
+
+       ctx->pdev = pdev;
+       ctx->ctx_id = ctx_id;
+       /* keep th smem pdev to use related iommu functions */
+       ctx->smem_device = smem_pdev;

nit: Perhaps the field could be renamed to smem_pdev?

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 156

Quoted 8 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +8 -0
+
+       /* initialized the global frame index of the device context */
+       atomic_set(&ctx->frame_param_sequence, 0);
+       spin_lock_init(&ctx->qlock);
+
+       /* setup the core operation of the device context */
+       if (ctx_desc && ctx_desc->init)
+               r = ctx_desc->init(ctx);

There is only one possible descriptor in this driver - mtk_cam_ctx_desc_p1. Please remove this unnecessary abstraction and just call the right functions directly.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 165

Quoted 9 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +9 -0
+
+       return r;
+}
+EXPORT_SYMBOL_GPL(mtk_cam_ctx_core_init);
+
+int mtk_cam_ctx_core_exit(struct mtk_cam_ctx *ctx)
+{
+       ctx->smem_vb2_alloc_ctx = NULL;
+       ctx->default_vb2_alloc_ctx = NULL;

ctx is going to be freed soon after this function returns, so there is no need to set these pointers to NULL. (And so, no need for this function at all.)

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 171

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +6 -0
+
+       return 0;
+}
+EXPORT_SYMBOL_GPL(mtk_cam_ctx_core_exit);
+
+/* Get the corrospnd FH of a specific buffer */

typo: corresponding
What's FH?
This function doesn't seem to accept a buffer pointer, so not sure what you mean by "a specific buffer".
Also, the function increments ctx->frame_param_sequence, so it's not just a "get".

Quoted 8 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +8 -0
+int mtk_cam_ctx_next_global_frame_sequence(struct mtk_cam_ctx *ctx,
+                                          int locked)
+{
+       int global_frame_sequence =
+               atomic_inc_return(&ctx->frame_param_sequence);
+
+       if (!locked)
+               spin_lock(&ctx->qlock);

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 189

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +10 -0
+
+       global_frame_sequence =
+               (global_frame_sequence & 0x0000FFFF) | (ctx->ctx_id << 16);
+
+       if (!locked)
+               spin_unlock(&ctx->qlock);
+
+       return global_frame_sequence;
+}
+EXPORT_SYMBOL_GPL(mtk_cam_ctx_next_global_frame_sequence);

Please don't make a function change locking behavior based on arguments - it's super error prone and hard to debug. Instead, please provide 2 variants:

__mtk_cam_ctx_next_global_frame_sequence_locked() which does everything except locking

and

mtk_cam_ctx_next_global_frame_sequence() which simply grabs the lock and calls __mtk_cam_ctx_next_global_frame_sequence_locked().

But, do you really need a _locked() variant here? I can see this function being called only once in mtk_cam_ctx_trigger_job(), with dev_ctx->ctx_id as the second argument, which sounds like a bug. Also, why do you even need to acquire ctx->qlock here, if ctx->frame_param_sequence is atomic?

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 200

Quoted 11 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +11 -0
+
+static void mtk_cam_ctx_buffer_done
+       (struct mtk_cam_ctx_buffer *ctx_buf, int state)
+{
+               if (!ctx_buf ||
+                   state != MTK_CAM_CTX_BUFFER_DONE ||
+                       state != MTK_CAM_CTX_BUFFER_FAILED)
+                       return;
+
+               ctx_buf->state = state;
+}

I don't see why this function could even be useful. VB2 already tracks buffer states, as I mentioned in my earlier comment.

[snip]

Quoted 27 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +27 -0
+#define file_to_mtk_cam_node(__file) \
+       container_of(video_devdata(__file),\
+       struct mtk_cam_dev_video_device, vdev)
+
+#define mtk_cam_ctx_to_dev(__ctx) \
+       container_of(__ctx,\
+       struct mtk_cam_dev, ctx)
+
+#define mtk_cam_m2m_to_dev(__m2m) \
+       container_of(__m2m,\
+       struct mtk_cam_dev, mem2mem2)
+
+#define mtk_cam_subdev_to_dev(__sd) \
+       container_of(__sd, \
+       struct mtk_cam_dev, mem2mem2.subdev)
+
+#define mtk_cam_vbq_to_isp_node(__vq) \
+       container_of(__vq, \
+       struct mtk_cam_dev_video_device, vbq)
+
+#define mtk_cam_ctx_buf_to_dev_buf(__ctx_buf) \
+       container_of(__ctx_buf, \
+       struct mtk_cam_dev_buffer, ctx_buf)
+
+#define mtk_cam_vb2_buf_to_dev_buf(__vb) \
+       container_of(vb, \
+       struct mtk_cam_dev_buffer, \

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 237

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +10 -0
+       m2m2_buf.vbb.vb2_buf)
+
+#define mtk_cam_vb2_buf_to_m2m_buf(__vb) \
+       container_of(__vb, \
+       struct mtk_cam_mem2mem2_buffer, \
+       vbb.vb2_buf)
+
+#define mtk_cam_subdev_to_m2m(__sd) \
+       container_of(__sd, \
+       struct mtk_cam_mem2mem2_device, subdev)

All the macros above need to be replaced with static inline functions to guarantee the right argument and return types at compilation time.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 249

Quoted 12 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +12 -0
+
+struct mtk_cam_mem2mem2_device;
+
+struct mtk_cam_mem2mem2_buffer {
+       struct vb2_v4l2_buffer vbb;
+       struct list_head list;
+};
+
+struct mtk_cam_dev_buffer {
+       struct mtk_cam_mem2mem2_buffer m2m2_buf;
+       /* Intenal part */
+       struct mtk_cam_ctx_buffer ctx_buf;

Why do we need to separate this? All the data here belongs to this driver.

The reply is quite long already, so let me send it. Second part to follow.

Best regards,
Tomasz

#9 lore raw

Re: [RFC PATCH V0 7/7] [media] platform: mtk-isp: Add Mediatek ISP Pass 1 driver

Dear Thomas:

Thanks for your comments.

We will revise the source codes based on your comments. Since there are many comments to fix/revise, we will categorize & prioritize these with below list:

1. Coding style issues.
2. Coding defects, including unused codes.
3. Driver architecture refactoring, such as removing mtk_cam_ctx, unnecessary abstraction layer, etc.

Quoted 53 lines · Tomasz Figa

(() . ( strHi Frederic, Jungo,

Quoted 9 lines · Frederic Chen

From: Jungo Lin <[email protected]>

This patch adds the driver for Pass unit in Mediatek's camera ISP system. Pass 1 unit is embedded in Mediatek SOCs. It provides RAW processing which includes optical black correction, defect pixel correction, W/IR imbalance correction and lens shading correction.

Quoted 4 lines

The mtk-isp directory will contain drivers for multiple IP blocks found in Mediatek ISP system. It will include ISP Pass 1 driver, sensor interface driver, DIP driver and face detection driver.

Thanks for the patches! Please see my comments inline.

[snip]

Quoted 33 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +27 -0
new file mode 100644
index 0000000..11a60a6
@@ -0,0 +1,327 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2018 MediaTek Inc.
+ * Author: Frederic Chen <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __MTK_CAM_CTX_H__
+#define __MTK_CAM_CTX_H__
+
+#include <linux/types.h>
+#include <linux/videodev2.h>
+#include <media/v4l2-ctrls.h>
+#include <media/videobuf2-core.h>
+#include <media/v4l2-subdev.h>
+#include "mtk_cam-v4l2-util.h"
+
+#define MTK_CAM_CTX_QUEUES (16)
+#define MTK_CAM_CTX_FRAME_BUNDLE_BUFFER_MAX (MTK_CAM_CTX_QUEUES)
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 32

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +5 -0
+#define MTK_CAM_CTX_DESC_MAX (MTK_CAM_CTX_QUEUES)
+
+#define MTK_CAM_CTX_MODE_DEBUG_OFF (0)
+#define MTK_CAM_CTX_MODE_DEBUG_BYPASS_JOB_TRIGGER (1)
+#define MTK_CAM_CTX_MODE_DEBUG_BYPASS_ALL (2)

nit: No need for parentheses for simple values. Also please align macro values together using tabs.

1. Coding style issues
- Fix in next release.

Quoted 7 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 35

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +3 -0
+
+#define MTK_CAM_GET_CTX_ID_FROM_SEQUENCE(sequence) \
+       ((sequence) >> 16 & 0x0000FFFF)

Sounds like a job for a static inline function, with appropriate argument and return types enforced by the compiler.

2. Coding defects
- Plan to fix.

Quoted 22 lines
Quoted 22 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +22 -0
+
+#define MTK_CAM_CTX_META_BUF_DEFAULT_SIZE (1110 * 1024)
+
+struct mtk_cam_ctx;
+struct mtk_cam_ctx_open_param;
+struct mtk_cam_ctx_release_param;
+struct mtk_cam_ctx_streamon_param;
+struct mtk_cam_ctx_streamoff_param;
+struct mtk_cam_ctx_start_param;
+struct mtk_cam_ctx_finish_param;
+
+/* struct mtk_cam_ctx_ops - background hardware driving ops */
+/* sdefines background driver specific callback APIs  */
+struct mtk_cam_ctx_ops {
+       int (*open)(struct mtk_cam_ctx *dev_ctx,
+                   struct mtk_cam_ctx_open_param *param);
+       int (*release)(struct mtk_cam_ctx *dev_ctx,
+                      struct mtk_cam_ctx_release_param *param);
+       int (*start)(struct mtk_cam_ctx *dev_ctx,
+                    struct mtk_cam_ctx_start_param *param);
+       int (*finish)(struct mtk_cam_ctx *dev_ctx,
+                     struct mtk_cam_ctx_finish_param *param);
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 62

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +5 -0
+       int (*streamon)(struct mtk_cam_ctx *dev_ctx,
+                       struct mtk_cam_ctx_streamon_param *param);
+       int (*streamoff)(struct mtk_cam_ctx *dev_ctx,
+                        struct mtk_cam_ctx_streamoff_param *param);
+};

We only have one driver here and these ops look really close to what vb2 already provides. Why do we need this indirection?

3. Driver architecture refactoring
- Plan to fix.

Quoted 8 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 66

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +4 -0
+
+/* Attributes setup by device context owner */
+struct mtk_cam_ctx_queue_desc {
+       int id; /* id of the context queue */

Please use kerneldoc syntax for documenting the structures.
(https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txtjjj

1. Coding style issues
- Fix in next release.

Quoted 17 lines
Quoted 17 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +17 -0
+       char *name;
+       /* Will be exported to media entity name */
+       int capture;
+       /* true for capture queue (device to user), false for output queue */
+       /* (from user to device) */
+       int image;
+       /* Using the cam_smem_drv as alloc ctx or not */
+       int smem_alloc;
+       /* true for image, false for meta data */
+       unsigned int dma_port; /*The dma port associated to the buffer*/
+       /* Supported format */
+       struct mtk_cam_ctx_format *fmts;
+       int num_fmts;
+       /* Default format of this queue */
+       int default_fmt_idx;
+};
+
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 85

Quoted 2 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +2 -0
+/* Supported format and the information used for */
+/* size calculation */

CodingStyle:
/*
* This is the kernel style for multi-line
* comments.
*/

1. Coding style issues
- Fix in next release.

Quoted 19 lines
Quoted 19 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +19 -0
+struct mtk_cam_ctx_meta_format {
+       u32 dataformat;
+       u32 max_buffer_size;
+       u8 flags;
+};
+
+struct mtk_cam_ctx_img_format {
+       u32     pixelformat;
+       u8      depth[VIDEO_MAX_PLANES];
+       u8      row_depth[VIDEO_MAX_PLANES];
+       u8      num_planes;
+       u32     flags;
+};
+
+struct mtk_cam_ctx_format {
+       union {
+               struct mtk_cam_ctx_meta_format meta;
+               struct mtk_cam_ctx_img_format img;
+       } fmt;
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 110

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +6 -0
+};
+
+union mtk_v4l2_fmt {
+       struct v4l2_pix_format_mplane pix_mp;
+       struct v4l2_meta_format meta;
+};

Why not just use the standard v4l2_format?

3. Driver architecture refactoring
- yes, it is possible to use v4l2_format and will depended on how to remove mtk_cam_ctx structure.

Quoted 21 lines
Quoted 21 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +21 -0
+
+/* Attributes setup by device context owner */
+struct mtk_cam_ctx_queues_setting {
+       int master;
+       /* The master input node to trigger the frame data enqueue */
+       struct mtk_cam_ctx_queue_desc *output_queue_descs;
+       int total_output_queues;
+       struct mtk_cam_ctx_queue_desc *capture_queue_descs;
+       int total_capture_queues;
+};
+
+struct mtk_cam_ctx_queue_attr {
+       int master;
+       int input_offset;
+       int total_num;
+};
+
+/* Video node context. Since we use */
+/* mtk_cam_ctx_frame_bundle to manage enqueued */
+/* buffers by frame now, we don't use bufs filed of */
+/* mtk_cam_ctx_queue now */
Quoted 8 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 137

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +6 -0
+struct mtk_cam_ctx_queue {
+       union mtk_v4l2_fmt fmt;
+       struct mtk_cam_ctx_format *ctx_fmt;
+       /* Currently we used in standard v4l2 image format */
+       /* in the device context */
+       unsigned int width_pad; /* bytesperline, reserved */

What's the meaning of this field?

2. Coding defects
- Unused field. Should be removed in next release.

Quoted 15 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 146

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +9 -0
+       struct mtk_cam_ctx_queue_desc desc;
+       struct list_head bufs; /* Reserved, not used now */
+};
+
+enum mtk_cam_ctx_frame_bundle_state {
+       MTK_CAM_CTX_FRAME_NEW,  /* Not allocated */
+       MTK_CAM_CTX_FRAME_PREPARED, /* Allocated but has not be processed */
+       MTK_CAM_CTX_FRAME_PROCESSING,   /* Queued, waiting to be filled */
+};

Feels like duplicating the media_request::state. (and eliminating the benefit of the state machine being fully managed by the Request API core).

3. Driver architecture refactoring
- Plan to remove based on Request API mechanism.

Quoted 12 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 155

Quoted 9 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +9 -0
+
+/* The definiation is compatible with DIP driver's state definiation */
+/* currently and will be decoupled after further integration */
+enum mtk_cam_ctx_frame_data_state {
+       MTK_CAM_CTX_FRAME_DATA_EMPTY = 0, /* FRAME_STATE_INIT */
+       MTK_CAM_CTX_FRAME_DATA_DONE = 3, /* FRAME_STATE_DONE */
+       MTK_CAM_CTX_FRAME_DATA_STREAMOFF_DONE = 4, /*FRAME_STATE_STREAMOFF*/
+       MTK_CAM_CTX_FRAME_DATA_ERROR = 5, /*FRAME_STATE_ERROR*/
+};

What are these states for?

3. Driver architecture refactoring
- Plan to replace with VB2 state.

Quoted 12 lines
Quoted 12 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +12 -0
+
+struct mtk_cam_ctx_frame_bundle {
+       struct mtk_cam_ctx_buffer*
+               buffers[MTK_CAM_CTX_FRAME_BUNDLE_BUFFER_MAX];
+       int id;
+       int num_img_capture_bufs;
+       int num_img_output_bufs;
+       int num_meta_capture_bufs;
+       int num_meta_output_bufs;
+       int last_index;
+       int state;
+       struct list_head list;
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 172

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +5 -0
+};
+
+struct mtk_cam_ctx_frame_bundle_list {
+       struct list_head list;
+};

This sounds like duplicating the request API core functionality, which aggregates all the buffers in the request.

3. Driver architecture refactoring
- Plan to remove based on Request API mechanism.

Quoted 29 lines
Quoted 29 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +29 -0
+
+struct mtk_cam_ctx {
+       struct platform_device *pdev;
+       struct platform_device *smem_device;
+       /* buffer queues will be added later */
+       unsigned short ctx_id;
+       char *device_name;
+       const struct mtk_cam_ctx_ops *ops;
+       struct mtk_cam_dev_node_mapping *mtk_cam_dev_node_map;
+       unsigned int dev_node_num;
+       /* mtk_cam_ctx_queue is the context for the video nodes */
+       struct mtk_cam_ctx_queue queue[MTK_CAM_CTX_QUEUES];
+       struct mtk_cam_ctx_queue_attr queues_attr;
+       atomic_t frame_param_sequence;
+       int streaming;
+       void *default_vb2_alloc_ctx;
+       void *smem_vb2_alloc_ctx;
+       struct v4l2_subdev_fh *fh;
+       struct mtk_cam_ctx_frame_bundle frame_bundles[VB2_MAX_FRAME];
+       struct mtk_cam_ctx_frame_bundle_list processing_frames;
+       struct mtk_cam_ctx_frame_bundle_list free_frames;
+       int enabled_dma_ports;
+       int num_frame_bundle;
+       spinlock_t qlock; /* frame queues protection */
+};
+
+enum mtk_cam_ctx_buffer_state {
+       MTK_CAM_CTX_BUFFER_NEW,
+       MTK_CAM_CTX_BUFFER_PROCESSING,
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 204

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +3 -0
+       MTK_CAM_CTX_BUFFER_DONE,
+       MTK_CAM_CTX_BUFFER_FAILED,
+};

Why these custom states? VB2 already manages buffer states for you and the driver should just obtain available buffers in the .buf_queue callback and return them with appropriate state using vb2_buffer_done() when they are no longer in use.

3. Driver architecture refactoring
- Plan to replace with VB2 state.

Quoted 12 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 212

Quoted 8 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +8 -0
+
+struct mtk_cam_ctx_buffer {
+       union mtk_v4l2_fmt fmt;
+       struct mtk_cam_ctx_format *ctx_fmt;
+       int capture;
+       int image;
+       int frame_id;
+       int user_sequence; /* Sequence number assigned by user */

What's this user_sequence? There shouldn't be any sequence coming from the user, since the requests are used to aggregate buffers.

2. Coding defects
- It is used without Request API support.
In current design, it should be removed in next release.

Quoted 5 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 214

Quoted 2 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +2 -0
+       dma_addr_t daddr;
+       void *vaddr;

We shouldn't need to touch the buffers from the kernel driver.

2. Coding defects
- Will remove in next release.

Quoted 6 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 215

Quoted 1 line
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +1 -0
+       phys_addr_t paddr;

We shouldn't need physical address either. I suppose this is for the SCP, but then it should be a DMA address obtained from dma_map_*() with struct device pointer of the SCP.

Yes, this physical address is designed for SCP.
For tuning buffer, it will be touched by SCP and
SCP can't get the physical address by itself. So we need to get this physical address in the kernel space via mtk_cam_smem_iova_to_phys function call and pass it to the SCP. At the same time, DMA address (daddr) is used for ISP HW.

Quoted 28 lines
Quoted 28 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +28 -0
+       unsigned int queue;
+       enum mtk_cam_ctx_buffer_state state;
+       struct list_head list;
+};
+
+struct mtk_cam_ctx_setting {
+       struct mtk_cam_ctx_ops *ops;
+       char *device_name;
+};
+
+struct mtk_cam_ctx_desc {
+       char *proc_dev_phandle;
+       /* The context device's compatble string name in device tree*/
+       int (*init)(struct mtk_cam_ctx *ctx);
+       /* configure the core functions of the device context */
+};
+
+struct mtk_cam_ctx_init_table {
+       int total_dev_ctx;
+       struct mtk_cam_ctx_desc *ctx_desc_tbl;
+};
+
+struct mtk_cam_ctx_open_param {
+       /* Bitmask used to notify that the DMA port is enabled or not */
+       unsigned int enabled_dma_ports;
+};
+
+struct mtk_cam_ctx_streamon_param {
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 249

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +6 -0
+       unsigned int enabled_dma_ports;
+};
+
+struct mtk_cam_ctx_streamoff_param {
+       unsigned int enabled_dma_ports;
+};

Hmm, the 3 structs above are identical. Is there a need to have all 3 of them?

3. Driver architecture refactoring
- Plan to revise.

Quoted 12 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h line 258

Quoted 9 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-ctx.h new file +9 -0
+
+struct mtk_cam_ctx_start_param {
+       /* carry buffer information of the frame */
+       struct mtk_cam_ctx_frame_bundle *frame_bundle;
+};
+
+struct mtk_cam_ctx_release_param {
+       unsigned int enabled_dma_ports;
+};

One more identical struct.

Same as above.

Quoted 56 lines

[snip]

Quoted 55 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +49 -0
new file mode 100644
index 0000000..7d0197b
@@ -0,0 +1,986 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2018 MediaTek Inc.
+ * Author: Frederic Chen <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/device.h>
+#include <linux/platform_device.h>
+#include <media/videobuf2-dma-contig.h>
+#include <linux/dma-mapping.h>
+#include <media/v4l2-event.h>
+
+#include "mtk_cam-dev.h"
+#include "mtk_cam-v4l2-util.h"
+#include "mtk_cam-v4l2.h"
+#include "mtk_cam-smem.h"
+
+static struct mtk_cam_ctx_format *mtk_cam_ctx_find_fmt
+       (struct mtk_cam_ctx_queue *queue,
+        u32 format);
+
+static int mtk_cam_ctx_process_frame(struct mtk_cam_ctx *dev_ctx,
+                                    struct mtk_cam_ctx_frame_bundle
+                                    *frame_bundle);
+
+static int mtk_cam_ctx_free_frame(struct mtk_cam_ctx *dev_ctx,
+                                 struct mtk_cam_ctx_frame_bundle
+                                 *frame_bundle);
+
+static struct mtk_cam_ctx_frame_bundle *mtk_cam_ctx_get_free_frame
+       (struct mtk_cam_ctx *dev_ctx);
+
+static struct mtk_cam_ctx_frame_bundle *mtk_cam_ctx_get_processing_frame
+(struct mtk_cam_ctx *dev_ctx, int frame_id);
+
+static int mtk_cam_ctx_init_frame_bundles(struct mtk_cam_ctx *dev_ctx);
+
+static void mtk_cam_ctx_queue_event_frame_done
+       (struct mtk_cam_ctx *dev_ctx,
+       struct mtk_cam_dev_frame_done_event_data *fdone);
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 52

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +3 -0
+
+static void debug_bundle(struct mtk_cam_ctx  *dev_ctx,
+                        struct mtk_cam_ctx_frame_bundle *bundle_data);

Could you reorder the functions in this file so that there is no need for forward declarations? In general, a kernel source file should be ordered in such way that the reader can understand what's happening by reading from the top to bottom, without jumping between functions.

2. Coding defects
- Will fix in next release.

Quoted 12 lines
Quoted 12 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +12 -0
+
+struct vb2_v4l2_buffer *mtk_cam_ctx_buffer_get_vb2_v4l2_buffer
+(struct mtk_cam_ctx_buffer *ctx_buf)
+{
+       struct mtk_cam_dev_buffer *dev_buf = NULL;
+
+       if (!ctx_buf) {
+               pr_err("Failed to convert ctx_buf to dev_buf: Null pointer\n");
+               return NULL;
+       }
+
+       dev_buf = mtk_cam_ctx_buf_to_dev_buf(ctx_buf);
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 67

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +3 -0
+
+       return &dev_buf->m2m2_buf.vbb;
+}

I also added a comment to where mtk_cam_dev_buffer struct is defined, but let me repeat here. I don't think we need to split these structs so much. We just need one driver-specific struct that encapsulates the vb2_v4l2_buffer struct, e.g. mtk_cam_buffer.

3. Driver architecture refactoring
- Yes, we will plan to refactoring current implementation.

Quoted 9 lines
Quoted 9 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +9 -0
+
+/* The helper to configure the device context */
+int mtk_cam_ctx_core_steup(struct mtk_cam_ctx *ctx,
+                          struct mtk_cam_ctx_setting *ctx_setting)
+{
+       if (!ctx || !ctx_setting)
+               return -EINVAL;
+
+       ctx->ops = ctx_setting->ops;
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 80

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +4 -0
+       ctx->device_name = ctx_setting->device_name;
+
+       return 0;
+}

I don't understand what's the point of this. We only have one device, P1, and only one context ops, for this P1 device. Do we need this abstraction?

3. Driver architecture refactoring
- Yes, we will plan to remove this abstraction.

Quoted 5 lines
Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +5 -0
+
+int mtk_cam_ctx_core_queue_setup(struct mtk_cam_ctx *ctx,
+                                struct mtk_cam_ctx_queues_setting
+                                *queues_setting)
+{
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 91

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +6 -0
+       int queue_idx = 0;
+       int i = 0;
+
+       for (i = 0; i < queues_setting->total_output_queues; i++) {
+               struct mtk_cam_ctx_queue_desc *queue_desc =
+                       queues_setting->output_queue_descs + i;

The typical kernel convention would be &queues_setting->output_queue_descs[i].

2. Coding defects
- Fix in next release.

Quoted 6 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 94

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +3 -0
+
+               if (!queue_desc)
+                       return -EINVAL;

I don't think this can ever happen, especially for i > 0.

2. Coding defects
- Remove in next release.

Quoted 6 lines
Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +6 -0
+
+               /* Since the *ctx->queue has been initialized to 0 */
+               /* when it is allocated with mtk_cam_dev , */
+               /* I don't initialize the struct here */
+               ctx->queue[queue_idx].desc = *queue_desc;
+               queue_idx++;
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 103

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +3 -0
+       }
+
+       ctx->queues_attr.input_offset = queue_idx;

This input_offset doesn't seem to be used in the driver in any meaningful way. The only place it's read is in mtk_cam_dev_mem2mem2_init() and that could be easily replaced with checking for (!isp_dev->ctx.queue[i].capture).

2. Coding defects
- Plan to revise in next release.

Quoted 11 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 111

Quoted 8 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +8 -0
+
+       /* Setup the capture queue */
+       for (i = 0; i < queues_setting->total_capture_queues; i++) {
+               struct mtk_cam_ctx_queue_desc *queue_desc =
+                       queues_setting->capture_queue_descs + i;
+
+               if (!queue_desc)
+                       return -EINVAL;

Ditto.

Same as above.

Quoted 17 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 124

Quoted 13 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +13 -0
+
+               /* Since the *ctx->queue has been initialized to 0 */
+               /* when allocating the memory, I don't */
+               /* reinitialied the struct here */
+               ctx->queue[queue_idx].desc = *queue_desc;
+               queue_idx++;
+       }
+
+       ctx->queues_attr.master = queues_setting->master;
+       ctx->queues_attr.total_num = queue_idx;
+       ctx->dev_node_num = ctx->queues_attr.total_num;
+       return 0;
+}

This function just copies some compile-time constant data at runtime. Why couldn't we just use the compile-time constants directly?

Quoted 9 lines

Also, I don't see vb2 queues handled here. What a V4L2 driver would normally do is:
- have a driver-private struct to represent the queue (e.g. mtk_cam_queue) and encapsulating vb2_queue,
- set vb2_queue::drv_priv to point to the mtk_cam_queue struct encapsulating it,
- both the vb2_queue and internal data fields would be initialized in the same function, to keep queue-initialization code in the same part of the driver.

3. Driver architecture refactoring
- Yes, we will plan to refactoring current implementation.

Quoted 6 lines
Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +6 -0
+
+/* Mediatek ISP context core initialization */
+int mtk_cam_ctx_core_init(struct mtk_cam_ctx *ctx,
+                         struct platform_device *pdev, int ctx_id,
+       struct mtk_cam_ctx_desc *ctx_desc,
+       struct platform_device *proc_pdev,
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 134

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +4 -0
+       struct platform_device *smem_pdev)
+{
+       /* Initialize main data structure */
+       int r = 0;

Please don't initialize return value variables, unless really necessary. It prevents the compiler from detecting forgotten assignments further in the function.

2. Coding defects
- Fix in next release.

Quoted 12 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 143

Quoted 9 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +9 -0
+
+       ctx->smem_vb2_alloc_ctx = &smem_pdev->dev;
+       ctx->default_vb2_alloc_ctx = &pdev->dev;
+
+       if (IS_ERR((__force void *)ctx->smem_vb2_alloc_ctx))
+               pr_err("Failed to alloc vb2 dma context: smem_vb2_alloc_ctx");
+
+       if (IS_ERR((__force void *)ctx->default_vb2_alloc_ctx))
+               pr_err("Failed to alloc vb2 dma context: default_vb2_alloc_ctx");

It's impossible for the two conditions above to happen.

2. Coding defects
- Remove in next release.

Quoted 8 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 148

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +5 -0
+
+       ctx->pdev = pdev;
+       ctx->ctx_id = ctx_id;
+       /* keep th smem pdev to use related iommu functions */
+       ctx->smem_device = smem_pdev;

nit: Perhaps the field could be renamed to smem_pdev?

2. Coding defects
- Fix in next release.

Quoted 4 lines
Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +4 -0
+
+       /* initialized the global frame index of the device context */
+       atomic_set(&ctx->frame_param_sequence, 0);
+       spin_lock_init(&ctx->qlock);
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 156

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +4 -0
+
+       /* setup the core operation of the device context */
+       if (ctx_desc && ctx_desc->init)
+               r = ctx_desc->init(ctx);

There is only one possible descriptor in this driver - mtk_cam_ctx_desc_p1. Please remove this unnecessary abstraction and just call the right functions directly.

3. Driver architecture refactoring
- Yes, we will plan to remove this abstraction.

Quoted 5 lines
Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +5 -0
+
+       return r;
+}
+EXPORT_SYMBOL_GPL(mtk_cam_ctx_core_init);
+
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 165

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +4 -0
+int mtk_cam_ctx_core_exit(struct mtk_cam_ctx *ctx)
+{
+       ctx->smem_vb2_alloc_ctx = NULL;
+       ctx->default_vb2_alloc_ctx = NULL;

ctx is going to be freed soon after this function returns, so there is no need to set these pointers to NULL. (And so, no need for this function at all.)

2. Coding defects
- Remove in next release.

Quoted 5 lines
Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +5 -0
+
+       return 0;
+}
+EXPORT_SYMBOL_GPL(mtk_cam_ctx_core_exit);
+
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 171

Quoted 1 line
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +1 -0
+/* Get the corrospnd FH of a specific buffer */

typo: corresponding
What's FH?
This function doesn't seem to accept a buffer pointer, so not sure what you mean by "a specific buffer".
Also, the function increments ctx->frame_param_sequence, so it's not just a "get".

2. Coding defects
- The purpose of this function is to generate one unique frame sequence. We will correct the comment of this function.

Quoted 28 lines
Quoted 8 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +8 -0
+int mtk_cam_ctx_next_global_frame_sequence(struct mtk_cam_ctx *ctx,
+                                          int locked)
+{
+       int global_frame_sequence =
+               atomic_inc_return(&ctx->frame_param_sequence);
+
+       if (!locked)
+               spin_lock(&ctx->qlock);

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 189

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +10 -0
+
+       global_frame_sequence =
+               (global_frame_sequence & 0x0000FFFF) | (ctx->ctx_id << 16);
+
+       if (!locked)
+               spin_unlock(&ctx->qlock);
+
+       return global_frame_sequence;
+}
+EXPORT_SYMBOL_GPL(mtk_cam_ctx_next_global_frame_sequence);

Please don't make a function change locking behavior based on arguments - it's super error prone and hard to debug. Instead, please provide 2 variants:

__mtk_cam_ctx_next_global_frame_sequence_locked() which does everything except locking

and

Quoted 9 lines

mtk_cam_ctx_next_global_frame_sequence() which simply grabs the lock and calls __mtk_cam_ctx_next_global_frame_sequence_locked().

But, do you really need a _locked() variant here? I can see this function being called only once in mtk_cam_ctx_trigger_job(), with dev_ctx->ctx_id as the second argument, which sounds like a bug. Also, why do you even need to acquire ctx->qlock here, if ctx->frame_param_sequence is atomic?

3. Driver architecture refactoring
- We will plan to remove context implementation in the P1 driver. So this function may be removed in future.
If we still keep this function, we will revise based on your suggestion.

Quoted 15 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 200

Quoted 11 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +11 -0
+
+static void mtk_cam_ctx_buffer_done
+       (struct mtk_cam_ctx_buffer *ctx_buf, int state)
+{
+               if (!ctx_buf ||
+                   state != MTK_CAM_CTX_BUFFER_DONE ||
+                       state != MTK_CAM_CTX_BUFFER_FAILED)
+                       return;
+
+               ctx_buf->state = state;
+}

I don't see why this function could even be useful. VB2 already tracks buffer states, as I mentioned in my earlier comment.

Quoted 2 lines

[snip]

3. Driver architecture refactoring
- Plan to remove during driver refactoring.

Quoted 32 lines
Quoted 32 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +32 -0
+#define file_to_mtk_cam_node(__file) \
+       container_of(video_devdata(__file),\
+       struct mtk_cam_dev_video_device, vdev)
+
+#define mtk_cam_ctx_to_dev(__ctx) \
+       container_of(__ctx,\
+       struct mtk_cam_dev, ctx)
+
+#define mtk_cam_m2m_to_dev(__m2m) \
+       container_of(__m2m,\
+       struct mtk_cam_dev, mem2mem2)
+
+#define mtk_cam_subdev_to_dev(__sd) \
+       container_of(__sd, \
+       struct mtk_cam_dev, mem2mem2.subdev)
+
+#define mtk_cam_vbq_to_isp_node(__vq) \
+       container_of(__vq, \
+       struct mtk_cam_dev_video_device, vbq)
+
+#define mtk_cam_ctx_buf_to_dev_buf(__ctx_buf) \
+       container_of(__ctx_buf, \
+       struct mtk_cam_dev_buffer, ctx_buf)
+
+#define mtk_cam_vb2_buf_to_dev_buf(__vb) \
+       container_of(vb, \
+       struct mtk_cam_dev_buffer, \
+       m2m2_buf.vbb.vb2_buf)
+
+#define mtk_cam_vb2_buf_to_m2m_buf(__vb) \
+       container_of(__vb, \
+       struct mtk_cam_mem2mem2_buffer, \
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +5 -0
+       vbb.vb2_buf)
+
+#define mtk_cam_subdev_to_m2m(__sd) \
+       container_of(__sd, \
+       struct mtk_cam_mem2mem2_device, subdev)

All the macros above need to be replaced with static inline functions to guarantee the right argument and return types at compilation time.

3. Driver architecture refactoring
- Plan to revise during driver refactoring.

Quoted 6 lines
Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +6 -0
+
+struct mtk_cam_mem2mem2_device;
+
+struct mtk_cam_mem2mem2_buffer {
+       struct vb2_v4l2_buffer vbb;
+       struct list_head list;
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +6 -0
+};
+
+struct mtk_cam_dev_buffer {
+       struct mtk_cam_mem2mem2_buffer m2m2_buf;
+       /* Intenal part */
+       struct mtk_cam_ctx_buffer ctx_buf;

Why do we need to separate this? All the data here belongs to this driver.

3. Driver architecture refactoring
- Plan to revise during driver refactoring.

Quoted 4 lines

The reply is quite long already, so let me send it. Second part to follow.

Best regards,
Tomasz

Appreciate your patience to provide these helpful comments. We look forward to further comments.

Thanks,

Jungo

#10 lore raw

Re: [RFC PATCH V0 7/7] [media] platform: mtk-isp: Add Mediatek ISP Pass 1 driver

Hi Jungo,

Quoted 26 lines · Jungo Lin
Quoted 12 lines · Tomasz Figa

(() . ( strHi Frederic, Jungo,

Quoted 9 lines · Frederic Chen

From: Jungo Lin <[email protected]>

This patch adds the driver for Pass unit in Mediatek's camera ISP system. Pass 1 unit is embedded in Mediatek SOCs. It provides RAW processing which includes optical black correction, defect pixel correction, W/IR imbalance correction and lens shading correction.

Quoted 7 lines
Quoted 4 lines

The mtk-isp directory will contain drivers for multiple IP blocks found in Mediatek ISP system. It will include ISP Pass 1 driver, sensor interface driver, DIP driver and face detection driver.

Thanks for the patches! Please see my comments inline.

Dear Thomas:

Thanks for your comments.

Quoted 9 lines

We will revise the source codes based on your comments. Since there are many comments to fix/revise, we will categorize & prioritize these with below list:

1. Coding style issues.
2. Coding defects, including unused codes.
3. Driver architecture refactoring, such as removing mtk_cam_ctx, unnecessary abstraction layer, etc.

Thanks for replying to the comments!

Just to clarify, there is no need to hurry with resending a next patch set with only a subset of the changes. Please take your time to address all the comments before sending the next revision. This prevents forgetting about some remaining comments and also lets other reviewers come with new comments or some alternative ideas for already existing comments. Second part of my review is coming too.

P.S. Please avoid top-posting on mailing lists. If replying to a message, please reply below the related part of that message. (I've moved your reply to the place in the email where it should be.)

[snip]

Quoted 7 lines
Quoted 6 lines
Quoted 1 line

+ phys_addr_t paddr;

We shouldn't need physical address either. I suppose this is for the SCP, but then it should be a DMA address obtained from dma_map_*() with struct device pointer of the SCP.

Quoted 7 lines

Yes, this physical address is designed for SCP.
For tuning buffer, it will be touched by SCP and
SCP can't get the physical address by itself. So we need to get this physical address in the kernel space via mtk_cam_smem_iova_to_phys function call and pass it to the SCP. At the same time, DMA address (daddr) is used for ISP HW.

Right, but my point is that in the kernel phys_addr_t is the physical address from the CPU point of view. Any address from device point of view is dma_addr_t and should be obtained from the DMA mapping API (even if it's numerically the same as physical address).

Best regards,
Tomasz

#11 lore raw

Re: [RFC PATCH V0 7/7] [media] platform: mtk-isp: Add Mediatek ISP Pass 1 driver

Quoted 48 lines · Tomasz Figa

Hi Jungo,

Quoted 26 lines · Jungo Lin
Quoted 12 lines · Tomasz Figa

(() . ( strHi Frederic, Jungo,

Quoted 9 lines · Frederic Chen

From: Jungo Lin <[email protected]>

This patch adds the driver for Pass unit in Mediatek's camera ISP system. Pass 1 unit is embedded in Mediatek SOCs. It provides RAW processing which includes optical black correction, defect pixel correction, W/IR imbalance correction and lens shading correction.

Quoted 7 lines
Quoted 4 lines

The mtk-isp directory will contain drivers for multiple IP blocks found in Mediatek ISP system. It will include ISP Pass 1 driver, sensor interface driver, DIP driver and face detection driver.

Thanks for the patches! Please see my comments inline.

Dear Thomas:

Thanks for your comments.

Quoted 9 lines

We will revise the source codes based on your comments. Since there are many comments to fix/revise, we will categorize & prioritize these with below list:

1. Coding style issues.
2. Coding defects, including unused codes.
3. Driver architecture refactoring, such as removing mtk_cam_ctx, unnecessary abstraction layer, etc.

Thanks for replying to the comments!

Just to clarify, there is no need to hurry with resending a next patch set with only a subset of the changes. Please take your time to address all the comments before sending the next revision. This prevents forgetting about some remaining comments and also lets other reviewers come with new comments or some alternative ideas for already existing comments. Second part of my review is coming too.

Quoted 5 lines

P.S. Please avoid top-posting on mailing lists. If replying to a message, please reply below the related part of that message. (I've moved your reply to the place in the email where it should be.)

[snip]

Hi, Tomasz,

Thanks for your advice.
We will prepare the next patch set after all comments are revised.

Quoted 15 lines
Quoted 7 lines
Quoted 6 lines
Quoted 1 line

+ phys_addr_t paddr;

We shouldn't need physical address either. I suppose this is for the SCP, but then it should be a DMA address obtained from dma_map_*() with struct device pointer of the SCP.

Quoted 7 lines

Yes, this physical address is designed for SCP.
For tuning buffer, it will be touched by SCP and
SCP can't get the physical address by itself. So we need to get this physical address in the kernel space via mtk_cam_smem_iova_to_phys function call and pass it to the SCP. At the same time, DMA address (daddr) is used for ISP HW.

Quoted 5 lines

Right, but my point is that in the kernel phys_addr_t is the physical address from the CPU point of view. Any address from device point of view is dma_addr_t and should be obtained from the DMA mapping API (even if it's numerically the same as physical address).

OK.
In order to clarify the address usage, is it ok to rename "dma_addr_t scp_addr"?
Moreover, below function will be also renamed.
dma_addr_t mtk_cam_smem_iova_to_scp_phys(struct device *dev,
dma_addr_t iova)
Could you help us to review?
If you have any better suggestion, please kindly let us know.

Best regards,

Jungo

Quoted 7 lines

Best regards,
Tomasz

_______________________________________________
Linux-mediatek mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-mediatek

#12 lore raw

Re: [RFC PATCH V0 7/7] [media] platform: mtk-isp: Add Mediatek ISP Pass 1 driver

Quoted 83 lines · Jungo Lin
Quoted 48 lines · Tomasz Figa

Hi Jungo,

Quoted 26 lines · Jungo Lin
Quoted 12 lines · Tomasz Figa

(() . ( strHi Frederic, Jungo,

Quoted 9 lines · Frederic Chen

From: Jungo Lin <[email protected]>

This patch adds the driver for Pass unit in Mediatek's camera ISP system. Pass 1 unit is embedded in Mediatek SOCs. It provides RAW processing which includes optical black correction, defect pixel correction, W/IR imbalance correction and lens shading correction.

Quoted 7 lines
Quoted 4 lines

The mtk-isp directory will contain drivers for multiple IP blocks found in Mediatek ISP system. It will include ISP Pass 1 driver, sensor interface driver, DIP driver and face detection driver.

Thanks for the patches! Please see my comments inline.

Dear Thomas:

Thanks for your comments.

Quoted 9 lines

We will revise the source codes based on your comments. Since there are many comments to fix/revise, we will categorize & prioritize these with below list:

1. Coding style issues.
2. Coding defects, including unused codes.
3. Driver architecture refactoring, such as removing mtk_cam_ctx, unnecessary abstraction layer, etc.

Thanks for replying to the comments!

Just to clarify, there is no need to hurry with resending a next patch set with only a subset of the changes. Please take your time to address all the comments before sending the next revision. This prevents forgetting about some remaining comments and also lets other reviewers come with new comments or some alternative ideas for already existing comments. Second part of my review is coming too.

Quoted 5 lines

P.S. Please avoid top-posting on mailing lists. If replying to a message, please reply below the related part of that message. (I've moved your reply to the place in the email where it should be.)

[snip]

Hi, Tomasz,

Thanks for your advice.
We will prepare the next patch set after all comments are revised.

Quoted 15 lines
Quoted 7 lines
Quoted 6 lines
Quoted 1 line

+ phys_addr_t paddr;

We shouldn't need physical address either. I suppose this is for the SCP, but then it should be a DMA address obtained from dma_map_*() with struct device pointer of the SCP.

Quoted 7 lines

Yes, this physical address is designed for SCP.
For tuning buffer, it will be touched by SCP and
SCP can't get the physical address by itself. So we need to get this physical address in the kernel space via mtk_cam_smem_iova_to_phys function call and pass it to the SCP. At the same time, DMA address (daddr) is used for ISP HW.

Quoted 5 lines

Right, but my point is that in the kernel phys_addr_t is the physical address from the CPU point of view. Any address from device point of view is dma_addr_t and should be obtained from the DMA mapping API (even if it's numerically the same as physical address).

Quoted 3 lines

OK.
In order to clarify the address usage, is it ok to rename "dma_addr_t scp_addr"?

Sounds good to me.

Quoted 3 lines

Moreover, below function will be also renamed. dma_addr_t mtk_cam_smem_iova_to_scp_phys(struct device *dev,
dma_addr_t iova)

Perhaps mtk_cam_smem_iova_to_scp_addr() for consistency with the struct field above?

#13 lore raw

Re: [RFC PATCH V0 7/7] [media] platform: mtk-isp: Add Mediatek ISP Pass 1 driver

Quoted 88 lines · Tomasz Figa
Quoted 83 lines · Jungo Lin
Quoted 48 lines · Tomasz Figa

Hi Jungo,

Quoted 26 lines · Jungo Lin
Quoted 12 lines · Tomasz Figa

(() . ( strHi Frederic, Jungo,

Quoted 9 lines · Frederic Chen

From: Jungo Lin <[email protected]>

This patch adds the driver for Pass unit in Mediatek's camera ISP system. Pass 1 unit is embedded in Mediatek SOCs. It provides RAW processing which includes optical black correction, defect pixel correction, W/IR imbalance correction and lens shading correction.

Quoted 7 lines
Quoted 4 lines

The mtk-isp directory will contain drivers for multiple IP blocks found in Mediatek ISP system. It will include ISP Pass 1 driver, sensor interface driver, DIP driver and face detection driver.

Thanks for the patches! Please see my comments inline.

Dear Thomas:

Thanks for your comments.

Quoted 9 lines

We will revise the source codes based on your comments. Since there are many comments to fix/revise, we will categorize & prioritize these with below list:

1. Coding style issues.
2. Coding defects, including unused codes.
3. Driver architecture refactoring, such as removing mtk_cam_ctx, unnecessary abstraction layer, etc.

Thanks for replying to the comments!

Just to clarify, there is no need to hurry with resending a next patch set with only a subset of the changes. Please take your time to address all the comments before sending the next revision. This prevents forgetting about some remaining comments and also lets other reviewers come with new comments or some alternative ideas for already existing comments. Second part of my review is coming too.

Quoted 5 lines

P.S. Please avoid top-posting on mailing lists. If replying to a message, please reply below the related part of that message. (I've moved your reply to the place in the email where it should be.)

[snip]

Hi, Tomasz,

Thanks for your advice.
We will prepare the next patch set after all comments are revised.

Quoted 15 lines
Quoted 7 lines
Quoted 6 lines
Quoted 1 line

+ phys_addr_t paddr;

We shouldn't need physical address either. I suppose this is for the SCP, but then it should be a DMA address obtained from dma_map_*() with struct device pointer of the SCP.

Quoted 7 lines

Yes, this physical address is designed for SCP.
For tuning buffer, it will be touched by SCP and
SCP can't get the physical address by itself. So we need to get this physical address in the kernel space via mtk_cam_smem_iova_to_phys function call and pass it to the SCP. At the same time, DMA address (daddr) is used for ISP HW.

Quoted 5 lines

Right, but my point is that in the kernel phys_addr_t is the physical address from the CPU point of view. Any address from device point of view is dma_addr_t and should be obtained from the DMA mapping API (even if it's numerically the same as physical address).

Quoted 3 lines

OK.
In order to clarify the address usage, is it ok to rename "dma_addr_t scp_addr"?

Quoted 8 lines

Sounds good to me.

Quoted 3 lines

Moreover, below function will be also renamed. dma_addr_t mtk_cam_smem_iova_to_scp_phys(struct device *dev,
dma_addr_t iova)

Perhaps mtk_cam_smem_iova_to_scp_addr() for consistency with the struct field above?

Ok, we will align the function name to struct field. This fix will be included in v1 version.

Thanks for your comments.

Best regards,

Jungo

#14 lore raw

Re: [RFC PATCH V0 7/7] [media] platform: mtk-isp: Add Mediatek ISP Pass 1 driver

In reply to Frederic Chen

Hi Frederic, Jungo,

Sorry for making you wait. Here comes the second part. Please see my comments inline.

[snip]

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 93

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +5 -0
+
+#define MTK_CAM_IO_CON_PADS (1)
+
+/* mtk_cam_io_connection --> sensor IF --> sensor 1 */
+/*                                     --> sensor 2 */

nit: Comment style.

Also, isn't the direction opposite?

Review

Quoted 2 lines
(quoted diff) +2 -0
+struct mtk_cam_io_connection {
+       const char *name;

Do we need this name? Could we just use subdev.name instead?

Quoted 1 line

+ struct platform_device *pdev;

This is assigned to mtk_cam_dev::pdev in the code and mtk_cam_dev is a superclass of this struct, so we always have access to it and can get pdev from it. No need to store it here as well. (And not sure why one would normally need to access pdev. Perhaps storing the struct device pointer inside mtk_cam_dev would be more practical?)

Quoted 1 line

+ struct v4l2_subdev subdev;

Do we need a subdev for an IO connection? Normally the processing part of the ISP would be represented as a subdev and then it would have its pads linked to other entities.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 105

Quoted 8 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +8 -0
+       int enable;
+       /* sensor connected */
+       struct v4l2_subdev *sensor;
+       /* sensor interface connected */
+       struct v4l2_subdev *sensor_if;
+       struct media_pad subdev_pads[MTK_CAM_IO_CON_PADS];
+       /* Current sensor input format*/
+       struct v4l2_mbus_framefmt subdev_fmt;

To summarize, it sounds like only "enable", "sensor", "sensor_if" and "subdev_fmt" would be appropriate attributes for an "IO connection". All the rest would normally be considered to be a part of the ISP device (and "pdev" and "name" are redundant).

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 108

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +3 -0
+};
+
+struct mtk_cam_dev_video_device {

nit: Why not just mtk_cam_video_device?

Quoted 1 line

+ const char *name;

Why not use vdev.name?

Quoted 1 line

+ int output;

Already available in vdev.vfl_dir, although not sure why it would be normally needed.

Quoted 1 line

+ int immutable;

Only master queue seems to be immutable, so maybe just check if the queue is master, wherever this is used currently?

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 113

Quoted 2 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +2 -0
+       int enabled;
+       int queued;

Looks unused.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 120

Quoted 7 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +7 -0
+       struct v4l2_format vdev_fmt;
+       struct video_device vdev;
+       struct media_pad vdev_pad;
+       struct v4l2_mbus_framefmt pad_fmt;
+       struct vb2_queue vbq;
+       struct list_head buffers;
+       struct mutex lock; /* vb2 queue and video device data protection */

It's actually about operations, not data. I'd write it "Serializes vb2 queue and video device operations.".

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 126

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +6 -0
+       atomic_t sequence;
+};
+
+struct mtk_cam_mem2mem2_device {
+       const char *name;
+       const char *model;

For both of the fields above, they seem to be always MTK_CAM_DEV_P1_NAME, so we can just use the macro directly whenever needed. No need for this indirection.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 130

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +4 -0
+       struct device *dev;
+       int num_nodes;
+       struct mtk_cam_dev_video_device *nodes;
+       const struct vb2_mem_ops *vb2_mem_ops;

This is always "vb2_dma_contig_memops", so it can be used directly.

Quoted 1 line

+ unsigned int buf_struct_size;

This is always sizeof(struct mtk_cam_dev_buffer), so no need to save it in the struct.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 134

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +3 -0
+       int streaming;
+       struct v4l2_device *v4l2_dev;
+       struct media_device *media_dev;

These 2 fields are already in mtk_cam_dev which is a superclass of this struct. One can just access them from there directly.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 136

Quoted 2 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +2 -0
+       struct media_pipeline pipeline;
+       struct v4l2_subdev subdev;

Could you remind me what was the media topology exposed by this driver? This is already the second subdev I spotted in this patch, which looks strange.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 140

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +4 -0
+       struct media_pad *subdev_pads;
+       struct v4l2_file_operations v4l2_file_ops;
+       const struct file_operations fops;
+};

Given most of the comments above, it looks like the remaining useful fields in this struct could be just moved to mtk_cam_dev, without the need for this separate struct.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 145

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +5 -0
+
+struct mtk_cam_dev {
+       struct platform_device *pdev;
+       struct mtk_cam_dev_video_device mem2mem2_nodes[MTK_CAM_DEV_NODE_MAX];
+       int queue_enabled[MTK_CAM_DEV_NODE_MAX];

Isn't this redundant with struct mtk_cam_dev_video_device::enabled?

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 152

Quoted 7 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +7 -0
+       struct mtk_cam_mem2mem2_device mem2mem2;
+       struct mtk_cam_io_connection cio;
+       struct v4l2_device v4l2_dev;
+       struct media_device media_dev;
+       struct mtk_cam_ctx ctx;
+       struct v4l2_async_notifier notifier;
+       struct mutex lock; /* device level data protection */

Just a side note: As a rule of thumb, mutex is normally used to protect operations not data (with some exceptions). For simple data protection, where accesses would normally take very short time, spinlock would be used. Of course a mutex would technically work too, so let's keep it for now and we can optimize later.

Quoted 14 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +14 -0
+};
+
+int mtk_cam_media_register(struct device *dev,
+                          struct media_device *media_dev,
+       const char *model);
+int mtk_cam_v4l2_register(struct device *dev,
+                         struct media_device *media_dev,
+       struct v4l2_device *v4l2_dev,
+       struct v4l2_ctrl_handler *ctrl_handler);
+int mtk_cam_v4l2_unregister(struct mtk_cam_dev *dev);
+int mtk_cam_mem2mem2_v4l2_register(struct mtk_cam_dev *dev,
+                                  struct media_device *media_dev,
+       struct v4l2_device *v4l2_dev);
+

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 176

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +10 -0
+int mtk_cam_v4l2_async_register(struct mtk_cam_dev *isp_dev);
+
+void mtk_cam_v4l2_async_unregister(struct mtk_cam_dev *isp_dev);
+
+int mtk_cam_v4l2_discover_sensor(struct mtk_cam_dev *isp_dev);
+
+void mtk_cam_v4l2_buffer_done(struct vb2_buffer *vb,
+                             enum vb2_buffer_state state);
+extern int mtk_cam_dev_queue_buffers
+       (struct mtk_cam_dev *dev, int initial);

Remove extern (here and other places too).

Quoted 16 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +16 -0
+extern int mtk_cam_dev_get_total_node
+       (struct mtk_cam_dev *mtk_cam_dev);
+extern char *mtk_cam_dev_get_node_name
+       (struct mtk_cam_dev *mtk_cam_dev_obj, int node);
+int mtk_cam_dev_init(struct mtk_cam_dev *isp_dev,
+                    struct platform_device *pdev,
+                    struct media_device *media_dev,
+                    struct v4l2_device *v4l2_dev);
+extern void mtk_cam_dev_mem2mem2_exit
+       (struct mtk_cam_dev *mtk_cam_dev_obj);
+int mtk_cam_dev_mem2mem2_init(struct mtk_cam_dev *isp_dev,
+                             struct media_device *media_dev,
+       struct v4l2_device *v4l2_dev);
+int mtk_cam_dev_get_queue_id_of_dev_node
+       (struct mtk_cam_dev *mtk_cam_dev_obj,
+        struct mtk_cam_dev_video_device *node);

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 202

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +10 -0
+int mtk_cam_dev_core_init(struct platform_device *pdev,
+                         struct mtk_cam_dev *isp_dev,
+       struct mtk_cam_ctx_desc *ctx_desc);
+int mtk_cam_dev_core_init_ext(struct platform_device *pdev,
+                             struct mtk_cam_dev *isp_dev,
+       struct mtk_cam_ctx_desc *ctx_desc,
+       struct media_device *media_dev,
+       struct v4l2_device *v4l2_dev);
+extern int mtk_cam_dev_core_release
+(struct platform_device *pdev, struct mtk_cam_dev *isp_dev);

nit: Strange line break. Some more usual examples:

int mtk_cam_dev_core_release(struct platform_device *pdev,

                             struct mtk_cam_dev *isp_dev);

struct long_struct_name *mtk_cam_dev_core_release(
struct platform_device *pdev, struct mtk_cam_dev *isp_dev);

struct very_very_very_long_struct_name *
my_function(int a, int b);

Quoted 112 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +2 -0
+
+#endif /* __MTK_CAM_DEV_H__ */
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h new file +104 -0
new file mode 100644
index 0000000..b5067d6
@@ -0,0 +1,146 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2018 MediaTek Inc.
+ * Author: Ryan Yu <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef _CAM_REGS_H
+#define _CAM_REGS_H
+
+/* TG Bit Mask */
+#define VFDATA_EN_BIT BIT(0)
+#define CMOS_EN_BIT BIT(0)
+
+/* normal signal bit */
+#define VS_INT_ST      BIT(0)
+#define HW_PASS1_DON_ST        BIT(11)
+#define SOF_INT_ST     BIT(12)
+#define SW_PASS1_DON_ST        BIT(30)
+
+/* err status bit */
+#define TG_ERR_ST      BIT(4)
+#define TG_GBERR_ST    BIT(5)
+#define CQ_CODE_ERR_ST BIT(6)
+#define CQ_APB_ERR_ST  BIT(7)
+#define CQ_VS_ERR_ST   BIT(8)
+#define AMX_ERR_ST     BIT(15)
+#define RMX_ERR_ST     BIT(16)
+#define BMX_ERR_ST     BIT(17)
+#define RRZO_ERR_ST    BIT(18)
+#define AFO_ERR_ST     BIT(19)
+#define IMGO_ERR_ST    BIT(20)
+#define AAO_ERR_ST     BIT(21)
+#define PSO_ERR_ST     BIT(22)
+#define LCSO_ERR_ST    BIT(23)
+#define BNR_ERR_ST     BIT(24)
+#define LSCI_ERR_ST    BIT(25)
+#define DMA_ERR_ST     BIT(29)
+
+/* CAM DMA done status */
+#define FLKO_DONE_ST   BIT(4)
+#define AFO_DONE_ST    BIT(5)
+#define AAO_DONE_ST    BIT(7)
+#define PSO_DONE_ST    BIT(14)
+
+/* IRQ signal mask */
+#define INT_ST_MASK_CAM        ( \
+                       VS_INT_ST |\
+                       HW_PASS1_DON_ST |\
+                       SOF_INT_ST |\
+                       SW_PASS1_DON_ST)
+
+/* IRQ Warning Mask */
+#define INT_ST_MASK_CAM_WARN   (\
+                               RRZO_ERR_ST |\
+                               AFO_ERR_ST |\
+                               IMGO_ERR_ST |\
+                               AAO_ERR_ST |\
+                               PSO_ERR_ST | \
+                               LCSO_ERR_ST |\
+                               BNR_ERR_ST |\
+                               LSCI_ERR_ST)
+
+/* IRQ Error Mask */
+#define INT_ST_MASK_CAM_ERR    (\
+                               TG_ERR_ST |\
+                               TG_GBERR_ST |\
+                               CQ_CODE_ERR_ST |\
+                               CQ_APB_ERR_ST |\
+                               CQ_VS_ERR_ST |\
+                               BNR_ERR_ST |\
+                               RMX_ERR_ST |\
+                               BMX_ERR_ST |\
+                               BNR_ERR_ST |\
+                               LSCI_ERR_ST |\
+                               DMA_ERR_ST)
+
+/* IRQ Signal Log Mask */
+#define INT_ST_LOG_MASK_CAM    (\
+                               SOF_INT_ST |\
+                               SW_PASS1_DON_ST |\
+                               VS_INT_ST |\
+                               TG_ERR_ST |\
+                               TG_GBERR_ST |\
+                               RRZO_ERR_ST |\
+                               AFO_ERR_ST |\
+                               IMGO_ERR_ST |\
+                               AAO_ERR_ST |\
+                               DMA_ERR_ST)
+
+/* DMA Event Notification Mask */
+#define DMA_ST_MASK_CAM        (\
+                       AFO_DONE_ST |\
+                       AAO_DONE_ST |\
+                       PSO_DONE_ST |\
+                       FLKO_DONE_ST)

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h line 114

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h new file +10 -0
+
+/* Status check */
+#define REG_CTL_EN                     0x0004
+#define REG_CTL_DMA_EN                 0x0008
+#define REG_CTL_FMT_SEL                0x0010
+#define REG_CTL_EN2                    0x0018
+#define REG_CTL_RAW_INT_EN             0x0020
+#define REG_CTL_RAW_INT_STAT           0x0024
+#define REG_CTL_RAW_INT2_STAT          0x0034
+#define REG_CTL_RAW_INT3_STAT          0x00C4

nit: Please use lowercase hex literals.

[snip]

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem-drv.c

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem-drv.c new file +0 -0
new file mode 100644
index 0000000..020c38c

I don't think we need any of the code that is in this file. We should just use the DMA API. We should be able to create appropriate reserved memory pools in DT and properly assign them to the right allocating devices.

Skipping review of this file for the time being.

[snip]

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.h

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.h new file +0 -0
new file mode 100644
index 0000000..4e1cf20

Ditto.

[snip]

Quoted 11 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
new file mode 100644
index 0000000..7da312d
@@ -0,0 +1,1555 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2018 Mediatek Corporation.
+ * Copyright (c) 2017 Intel Corporation.
+ *

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 15

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * MTK_CAM-v4l2 is highly based on Intel IPU 3 chrome driver

To be precise, it's not a "chrome driver". I think you could just say "IPU3 ImgU driver". (The driver was merged in Linux 5.0.)

Quoted 15 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +15 -0
+ *
+ */
+
+#include <linux/module.h>
+#include <linux/pm_runtime.h>
+#include <linux/videodev2.h>
+#include <media/v4l2-ioctl.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-fwnode.h>
+#include <linux/device.h>
+#include <linux/platform_device.h>
+#include <linux/of.h>
+#include <linux/of_graph.h>

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 40

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+#include <media/v4l2-common.h>
+#include <media/media-entity.h>
+#include <media/v4l2-async.h>
+
+#include "mtk_cam.h"
+#include "mtk_cam-dev.h"
+#include "mtk_cam-v4l2.h"
+#include "mtk_cam-v4l2-util.h"
+
+#define CONFIG_MEDIATEK_MEDIA_REQUEST

I guess this is for testing, but I'd suggest switching to the target setup sooner than later and dropping this and related #ifdefs.

Quoted 29 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +29 -0
+
+#define MTK_CAM_SENSOR_MAIN_PAD_SRC 0
+#define MTK_CAM_SENSOR_SUB_PAD_SRC 0
+#define MTK_CAM_SENSOR_IF_PAD_MAIN_SINK 0
+#define MTK_CAM_SENSOR_IF_PAD_SUB_SINK 1
+#define MTK_CAM_SENSOR_IF_PAD_SRC 4
+#define MTK_CAM_CIO_PAD_SINK 0
+
+static u32 mtk_cam_node_get_v4l2_cap
+       (struct mtk_cam_ctx_queue *node_ctx);
+
+static int mtk_cam_videoc_s_meta_fmt(struct file *file,
+                                    void *fh, struct v4l2_format *f);
+
+static int mtk_cam_subdev_open(struct v4l2_subdev *sd,
+                              struct v4l2_subdev_fh *fh)
+{
+       struct mtk_cam_dev *isp_dev = mtk_cam_subdev_to_dev(sd);
+
+       isp_dev->ctx.fh = fh;
+       return mtk_cam_ctx_open(&isp_dev->ctx);
+}
+
+static int mtk_cam_subdev_close(struct v4l2_subdev *sd,
+                               struct v4l2_subdev_fh *fh)
+{
+       struct mtk_cam_dev *isp_dev = mtk_cam_subdev_to_dev(sd);
+
+       return mtk_cam_ctx_release(&isp_dev->ctx);

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 79

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+}
+
+static int mtk_cam_subdev_s_stream(struct v4l2_subdev *sd,
+                                  int enable)
+{
+       int ret = 0;
+       struct mtk_cam_dev *isp_dev = mtk_cam_subdev_to_dev(sd);
+       struct mtk_cam_io_connection *cio = &isp_dev->cio;
+
+       if (enable) {

Please extract enable and disable cases into separate functions.

Quoted 19 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +19 -0
+               /* Get sensor interace and sensor sub device */
+               /* If the call succeeds, sensor if and sensor are filled */
+               /* in isp_dev->cio->sensor_if and isp_dev->cio->sensor */
+               ret = mtk_cam_v4l2_discover_sensor(isp_dev);
+               if (ret) {
+                       dev_err(&isp_dev->pdev->dev,
+                               "no sensor or sensor if connected (%d)\n",
+                               ret);
+                       return -EPERM;
+               }
+
+               /* seninf must stream on first */
+               ret = v4l2_subdev_call(cio->sensor_if, video, s_stream, 1);
+               if (ret) {
+                       dev_err(&isp_dev->pdev->dev,
+                               "sensor-if(%s) stream on failed (%d)\n",
+                               cio->sensor_if->entity.name, ret);
+                       return -EPERM;
+               }

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 108

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+
+               dev_dbg(&isp_dev->pdev->dev, "streamed on sensor-if(%s)\n",
+                       cio->sensor_if->entity.name);
+
+               ret = v4l2_subdev_call(cio->sensor, video, s_stream, 1);
+               if (ret) {
+                       dev_err(&isp_dev->pdev->dev,
+                               "sensor(%s) stream on failed (%d)\n",
+                               cio->sensor->entity.name, ret);
+                       return -EPERM;

We should undo any previous operations.

Quoted 12 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +12 -0
+               }
+
+               dev_dbg(&isp_dev->pdev->dev, "streamed on sensor(%s)\n",
+                       cio->sensor->entity.name);
+
+               ret = mtk_cam_ctx_streamon(&isp_dev->ctx);
+               if (ret) {
+                       dev_err(&isp_dev->pdev->dev,
+                               "Pass 1 stream on failed (%d)\n", ret);
+                       return -EPERM;
+               }
+

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 130

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+               isp_dev->mem2mem2.streaming = enable;
+
+               ret = mtk_cam_dev_queue_buffers(isp_dev, true);
+               if (ret)
+                       dev_err(&isp_dev->pdev->dev,
+                               "failed to queue initial buffers (%d)", ret);
+
+               dev_dbg(&isp_dev->pdev->dev, "streamed on Pass 1\n");
+       } else {
+               if (cio->sensor) {

Is it possible to have cio->sensor NULL here? This function would have failed if it wasn't found when enabling.

Quoted 7 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +7 -0
+                       ret = v4l2_subdev_call(cio->sensor, video, s_stream, 0);
+                       if (ret) {
+                               dev_err(&isp_dev->pdev->dev,
+                                       "sensor(%s) stream off failed (%d)\n",
+                                       cio->sensor->entity.name, ret);
+                       } else {
+                               dev_dbg(&isp_dev->pdev->dev,

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 147

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+                                       "streamed off sensor(%s)\n",
+                                       cio->sensor->entity.name);
+                               cio->sensor = NULL;
+                       }
+               } else {
+                       dev_err(&isp_dev->pdev->dev,
+                               "Can't find sensor connected\n");
+               }
+
+               if (cio->sensor_if) {

Ditto.

Quoted 49 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +49 -0
+                       ret = v4l2_subdev_call(cio->sensor_if, video,
+                                              s_stream, 0);
+                       if (ret) {
+                               dev_err(&isp_dev->pdev->dev,
+                                       "sensor if(%s) stream off failed (%d)\n",
+                                       cio->sensor_if->entity.name, ret);
+                       } else {
+                               dev_dbg(&isp_dev->pdev->dev,
+                                       "streamed off sensor-if(%s)\n",
+                                       cio->sensor_if->entity.name);
+                               cio->sensor_if = NULL;
+                       }
+               } else {
+                       dev_err(&isp_dev->pdev->dev,
+                               "Can't find sensor-if connected\n");
+               }
+
+               ret = mtk_cam_ctx_streamoff(&isp_dev->ctx);
+               if (ret) {
+                       dev_err(&isp_dev->pdev->dev,
+                               "Pass 1 stream off failed (%d)\n", ret);
+                       return -EPERM;
+               }
+
+               isp_dev->mem2mem2.streaming = false;
+
+               dev_dbg(&isp_dev->pdev->dev, "streamed off Pass 1\n");
+       }
+
+       return 0;
+}
+
+static void v4l2_event_merge(const struct v4l2_event *old,
+                            struct v4l2_event *new)
+{
+       struct mtk_cam_dev_stat_event_data *old_evt_stat_data =
+               (void *)old->u.data;
+       struct mtk_cam_dev_stat_event_data *new_evt_stat_data =
+               (void *)new->u.data;
+
+       if (old->type == V4L2_EVENT_MTK_CAM_ENGINE_STATE &&
+           new->type == V4L2_EVENT_MTK_CAM_ENGINE_STATE) {
+               pr_debug("%s, merge IRQ, old(type(0x%x) frame no(%d) IRQ(0x%x) DMA(0x%x)), new(type(0x%x) frame_number(%d) IRQ(0x%x) DMA(0x%x))",
+                        __func__,
+                       old->type,
+                       old_evt_stat_data->frame_number,
+                       old_evt_stat_data->irq_status_mask,
+                       old_evt_stat_data->dma_status_mask,
+                       new->type,

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 206

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+                       new_evt_stat_data->frame_number,
+                       new_evt_stat_data->irq_status_mask,
+                       new_evt_stat_data->dma_status_mask);
+
+               /* merge IRQ event */
+               new_evt_stat_data->irq_status_mask |=
+                       old_evt_stat_data->irq_status_mask;
+               new_evt_stat_data->dma_status_mask |=
+                       old_evt_stat_data->dma_status_mask;
+       }

What are these custom events for? Userspace shouldn't need to know anything about IRQ and DMA.

[snip]

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 294

Quoted 8 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +8 -0
+static void mtk_cam_vb2_buf_queue(struct vb2_buffer *vb)
+{
+       struct mtk_cam_mem2mem2_device *m2m2 = vb2_get_drv_priv(vb->vb2_queue);
+
+       struct mtk_cam_dev *mtk_cam_dev = mtk_cam_m2m_to_dev(m2m2);
+
+       struct device *dev = &mtk_cam_dev->pdev->dev;
+

nit: Please remove these unnecessary blank lines between variables.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 297

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +3 -0
+       struct mtk_cam_dev_buffer *buf = NULL;
+
+       struct vb2_v4l2_buffer *v4l2_buf = NULL;

Please default-initialize local variables only if really needed. Otherwise it prevents the compiler from catching missing assignments for you.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 302

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+
+       struct mtk_cam_dev_video_device *node =
+               mtk_cam_vbq_to_isp_node(vb->vb2_queue);
+
+       int queue = mtk_cam_dev_get_queue_id_of_dev_node(mtk_cam_dev, node);

Why not just add queue_id field to mtk_cam_dev_video_device?

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 312

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+
+       dev_dbg(dev,
+               "queue vb2_buf: Node(%s) queue id(%d)\n",
+               node->name,
+               queue);
+
+       if (queue < 0) {
+               dev_err(m2m2->dev, "Invalid mtk_cam_dev node.\n");
+               return;
+       }

I don't think this is possible.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 315

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +3 -0
+
+       if (!vb)
+               pr_err("VB can't be null\n");

Ditto.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 320

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+
+       buf = mtk_cam_vb2_buf_to_dev_buf(vb);
+
+       if (!buf)
+               pr_err("buf can't be null\n");

Ditto.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 325

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+
+       v4l2_buf = to_vb2_v4l2_buffer(vb);
+
+       if (!v4l2_buf)
+               pr_err("v4l2_buf can't be null\n");

Ditto.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 334

Quoted 9 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +9 -0
+
+       mutex_lock(&mtk_cam_dev->lock);
+
+       pr_debug("init  mtk_cam_ctx_buf, sequence(%d)\n", v4l2_buf->sequence);
+
+       /* the dma address will be filled in later frame buffer handling*/
+       mtk_cam_ctx_buf_init(&buf->ctx_buf, queue, (dma_addr_t)0);
+       pr_debug("set mtk_cam_ctx_buf_init: user seq=%d\n",
+                buf->ctx_buf.user_sequence);

This should normally happen in the buf_init vb2 operation.

By the way, what's the point of having the DMA address argument to mtk_cam_ctx_buf_init() if it's always 0?

Given the above, is there even a reason for mtk_cam_ctx_buf_init() to exist? It sounds like the structure could be just initialize directly in buf_init.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 339

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+
+       /* Added the buffer into the tracking list */
+       list_add_tail(&buf->m2m2_buf.list,
+                     &m2m2->nodes[node - m2m2->nodes].buffers);
+       mutex_unlock(&mtk_cam_dev->lock);

If this mutex is only for the buffer list, I'd consider replacing it with a spinlock.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 345

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+
+       /* Enqueue the buffer */
+       if (mtk_cam_dev->mem2mem2.streaming) {
+               pr_debug("%s: mtk_cam_dev_queue_buffers\n",
+                        node->name);
+               mtk_cam_dev_queue_buffers(mtk_cam_dev, false);

Doesn't mtk_cam_dev_queue_buffers() already check if streaming is active?

Okay, let me send the comments above and continue third part of the review in another email.

Best regards,
Tomasz

#15 lore raw

Re: [RFC PATCH V0 7/7] [media] platform: mtk-isp: Add Mediatek ISP Pass 1 driver

Quoted 14 lines · Tomasz Figa

Hi Frederic, Jungo,

Sorry for making you wait. Here comes the second part. Please see my comments inline.

[snip]

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 93

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +5 -0
+
+#define MTK_CAM_IO_CON_PADS (1)
+
+/* mtk_cam_io_connection --> sensor IF --> sensor 1 */
+/*                                     --> sensor 2 */

nit: Comment style.

Quoted 2 lines

Also, isn't the direction opposite?

Hi Tomasz:

Thanks for your part 2's comment.
Below is our feedback.

For coding style issue, we will fix in next patch.

Quoted 5 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+struct mtk_cam_io_connection {
+       const char *name;

Do we need this name? Could we just use subdev.name instead?

Quoted 8 lines
Quoted 1 line

+ struct platform_device *pdev;

This is assigned to mtk_cam_dev::pdev in the code and mtk_cam_dev is a superclass of this struct, so we always have access to it and can get pdev from it. No need to store it here as well. (And not sure why one would normally need to access pdev. Perhaps storing the struct device pointer inside mtk_cam_dev would be more practical?)

Yes, you are right.
We will remove these two fields in next patch.

Quoted 6 lines
Quoted 1 line

+ struct v4l2_subdev subdev;

Do we need a subdev for an IO connection? Normally the processing part of the ISP would be represented as a subdev and then it would have its pads linked to other entities.

After internal discussion, we will revise the current CIO design. We will use the same ISP sub-dev to connect with sensor-IF without creating one new sub-device.

Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 105

Quoted 8 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +8 -0
+       int enable;
+       /* sensor connected */
+       struct v4l2_subdev *sensor;
+       /* sensor interface connected */
+       struct v4l2_subdev *sensor_if;
+       struct media_pad subdev_pads[MTK_CAM_IO_CON_PADS];
+       /* Current sensor input format*/
+       struct v4l2_mbus_framefmt subdev_fmt;
Quoted 5 lines

To summarize, it sounds like only "enable", "sensor", "sensor_if" and "subdev_fmt" would be appropriate attributes for an "IO connection". All the rest would normally be considered to be a part of the ISP device (and "pdev" and "name" are redundant).

After simplifying, we will only keep "enable", "sensor" and "sensor_f" for an "IO connection" as below:

/*
* struct mtk_cam_io_connection - CIO (Camera IO) information. *
* @enabled: indicate stream on or off
* @sensor: sensor sub-device
* @sensor_if: sensor_if sub-device
*
* Below is the graph topology.
* sensor 1 (main) --> sensor IF --> P1 sub-device
* sensor 2 (sub) -->
*
*/

struct mtk_cam_io_connection {
	int enabled;
	struct v4l2_subdev *sensor;
	struct v4l2_subdev *sensor_if;
};
Quoted 6 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 108

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +3 -0
+};
+
+struct mtk_cam_dev_video_device {

nit: Why not just mtk_cam_video_device?

OK, rename in next patch.

Quoted 4 lines
Quoted 1 line

+ const char *name;

Why not use vdev.name?

OK, remove this field and change to use vdev.name in next patch.

Quoted 5 lines
Quoted 1 line

+ int output;

Already available in vdev.vfl_dir, although not sure why it would be normally needed.

OK, remove this field in next patch.

Quoted 2 lines
Quoted 1 line

+ int immutable;

It is removed based on comment part 1's change.

Quoted 8 lines

Only master queue seems to be immutable, so maybe just check if the queue is master, wherever this is used currently?

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 113

Quoted 2 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +2 -0
+       int enabled;
+       int queued;

Looks unused.

For "enabled", it indicates the video device is enabled or not during the media link setup phase. Based on this information, we could know how many video devices (DMAs) are enabled. So we will keep this. For "queued", we will remove in next patch.

Quoted 11 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 120

Quoted 7 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +7 -0
+       struct v4l2_format vdev_fmt;
+       struct video_device vdev;
+       struct media_pad vdev_pad;
+       struct v4l2_mbus_framefmt pad_fmt;
+       struct vb2_queue vbq;
+       struct list_head buffers;
+       struct mutex lock; /* vb2 queue and video device data protection */

It's actually about operations, not data. I'd write it "Serializes vb2 queue and video device operations.".

Ok, we will fix the comment wording to clarify the correct usage. Below is the revised structure.

/*
* struct mtk_cam_video_device - Mediatek video device structure. *
* @enabled: indicate stream on or off
* @queue_id: queue id for mtk_cam_ctx_queue
* @pending_list: list for pending buffer
* @lock: serializes vb2 queue and video device operations.
* @slock: protect for pending_list.
*
*/

struct mtk_cam_video_device {
	int enabled;
	int queue_id;
	struct v4l2_format vdev_fmt;
	struct video_device vdev;
	struct media_pad vdev_pad;
	struct vb2_queue vbq;
	struct list_head pending_list;
	/* Used for vbq & vdev */
	struct mutex lock;
	/* protect for pending_list */
	spinlock_t slock;
};
Quoted 11 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 126

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +6 -0
+       atomic_t sequence;
+};
+
+struct mtk_cam_mem2mem2_device {
+       const char *name;
+       const char *model;

For both of the fields above, they seem to be always MTK_CAM_DEV_P1_NAME, so we can just use the macro directly whenever needed. No need for this indirection.

OK. These two fields will be removed in next patch.

Quoted 7 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 130

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +4 -0
+       struct device *dev;
+       int num_nodes;
+       struct mtk_cam_dev_video_device *nodes;
+       const struct vb2_mem_ops *vb2_mem_ops;

This is always "vb2_dma_contig_memops", so it can be used directly.

Ditto.

Quoted 5 lines
Quoted 1 line

+ unsigned int buf_struct_size;

This is always sizeof(struct mtk_cam_dev_buffer), so no need to save it in the struct.

Ditto.

Quoted 7 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 134

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +3 -0
+       int streaming;
+       struct v4l2_device *v4l2_dev;
+       struct media_device *media_dev;

These 2 fields are already in mtk_cam_dev which is a superclass of this struct. One can just access them from there directly.

Ditto.

Quoted 7 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 136

Quoted 2 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +2 -0
+       struct media_pipeline pipeline;
+       struct v4l2_subdev subdev;

Could you remind me what was the media topology exposed by this driver? This is already the second subdev I spotted in this patch, which looks strange.

For sub-device design, we will remove the sub-device for CIO and keep only one sub-device for ISP driver in next patch. We will also provide the media topology in RFC v1 patch to clarify.

Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 140

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +4 -0
+       struct media_pad *subdev_pads;
+       struct v4l2_file_operations v4l2_file_ops;
+       const struct file_operations fops;
+};

Given most of the comments above, it looks like the remaining useful fields in this struct could be just moved to mtk_cam_dev, without the need for this separate struct.

This is the final revision for these two structures. Do you suggest to merge it to simplify?

struct mtk_cam_mem2mem2_device {
	struct mtk_cam_video_device *nodes;
	struct media_pipeline pipeline;
	struct v4l2_subdev subdev;
	struct media_pad *subdev_pads;
};
struct mtk_cam_dev {
	struct platform_device *pdev;
	struct mtk_cam_video_device     mem2mem2_nodes[MTK_CAM_DEV_NODE_MAX];
	struct mtk_cam_mem2mem2_device mem2mem2;
	struct mtk_cam_io_connection cio;
	struct v4l2_device v4l2_dev;
	struct media_device media_dev;
	struct mtk_cam_ctx ctx;
	struct v4l2_async_notifier notifier;
};
Quoted 8 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 145

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +5 -0
+
+struct mtk_cam_dev {
+       struct platform_device *pdev;
+       struct mtk_cam_dev_video_device mem2mem2_nodes[MTK_CAM_DEV_NODE_MAX];
+       int queue_enabled[MTK_CAM_DEV_NODE_MAX];

Isn't this redundant with struct mtk_cam_dev_video_device::enabled?

Yes, we have removed this.

Quoted 8 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 152

Quoted 7 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +7 -0
+       struct mtk_cam_mem2mem2_device mem2mem2;
+       struct mtk_cam_io_connection cio;
+       struct v4l2_device v4l2_dev;
+       struct media_device media_dev;
+       struct mtk_cam_ctx ctx;
+       struct v4l2_async_notifier notifier;
+       struct mutex lock; /* device level data protection */
Quoted 6 lines

Just a side note: As a rule of thumb, mutex is normally used to protect operations not data (with some exceptions). For simple data protection, where accesses would normally take very short time, spinlock would be used. Of course a mutex would technically work too, so let's keep it for now and we can optimize later.

After review, the mutex is useless and will remove it in next patch. Btw, thanks for your explanation the difference between mutex & spinlock.

Quoted 18 lines
Quoted 18 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +18 -0
+};
+
+int mtk_cam_media_register(struct device *dev,
+                          struct media_device *media_dev,
+       const char *model);
+int mtk_cam_v4l2_register(struct device *dev,
+                         struct media_device *media_dev,
+       struct v4l2_device *v4l2_dev,
+       struct v4l2_ctrl_handler *ctrl_handler);
+int mtk_cam_v4l2_unregister(struct mtk_cam_dev *dev);
+int mtk_cam_mem2mem2_v4l2_register(struct mtk_cam_dev *dev,
+                                  struct media_device *media_dev,
+       struct v4l2_device *v4l2_dev);
+
+int mtk_cam_v4l2_async_register(struct mtk_cam_dev *isp_dev);
+
+void mtk_cam_v4l2_async_unregister(struct mtk_cam_dev *isp_dev);
+
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 176

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +6 -0
+int mtk_cam_v4l2_discover_sensor(struct mtk_cam_dev *isp_dev);
+
+void mtk_cam_v4l2_buffer_done(struct vb2_buffer *vb,
+                             enum vb2_buffer_state state);
+extern int mtk_cam_dev_queue_buffers
+       (struct mtk_cam_dev *dev, int initial);

Remove extern (here and other places too).

This coding style issue has been fixed in comment part 1.

Quoted 30 lines
Quoted 16 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +16 -0
+extern int mtk_cam_dev_get_total_node
+       (struct mtk_cam_dev *mtk_cam_dev);
+extern char *mtk_cam_dev_get_node_name
+       (struct mtk_cam_dev *mtk_cam_dev_obj, int node);
+int mtk_cam_dev_init(struct mtk_cam_dev *isp_dev,
+                    struct platform_device *pdev,
+                    struct media_device *media_dev,
+                    struct v4l2_device *v4l2_dev);
+extern void mtk_cam_dev_mem2mem2_exit
+       (struct mtk_cam_dev *mtk_cam_dev_obj);
+int mtk_cam_dev_mem2mem2_init(struct mtk_cam_dev *isp_dev,
+                             struct media_device *media_dev,
+       struct v4l2_device *v4l2_dev);
+int mtk_cam_dev_get_queue_id_of_dev_node
+       (struct mtk_cam_dev *mtk_cam_dev_obj,
+        struct mtk_cam_dev_video_device *node);

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 202

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +10 -0
+int mtk_cam_dev_core_init(struct platform_device *pdev,
+                         struct mtk_cam_dev *isp_dev,
+       struct mtk_cam_ctx_desc *ctx_desc);
+int mtk_cam_dev_core_init_ext(struct platform_device *pdev,
+                             struct mtk_cam_dev *isp_dev,
+       struct mtk_cam_ctx_desc *ctx_desc,
+       struct media_device *media_dev,
+       struct v4l2_device *v4l2_dev);
+extern int mtk_cam_dev_core_release
+(struct platform_device *pdev, struct mtk_cam_dev *isp_dev);

nit: Strange line break. Some more usual examples:

int mtk_cam_dev_core_release(struct platform_device *pdev,

Quoted 9 lines
                             struct mtk_cam_dev *isp_dev);

struct long_struct_name *mtk_cam_dev_core_release(
struct platform_device *pdev, struct mtk_cam_dev *isp_dev);

struct very_very_very_long_struct_name *
my_function(int a, int b);

Thanks for your comment.
We will revise our coding style based on your hint.

Quoted 125 lines
Quoted 112 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +2 -0
+
+#endif /* __MTK_CAM_DEV_H__ */
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h new file +104 -0
new file mode 100644
index 0000000..b5067d6
@@ -0,0 +1,146 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2018 MediaTek Inc.
+ * Author: Ryan Yu <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef _CAM_REGS_H
+#define _CAM_REGS_H
+
+/* TG Bit Mask */
+#define VFDATA_EN_BIT BIT(0)
+#define CMOS_EN_BIT BIT(0)
+
+/* normal signal bit */
+#define VS_INT_ST      BIT(0)
+#define HW_PASS1_DON_ST        BIT(11)
+#define SOF_INT_ST     BIT(12)
+#define SW_PASS1_DON_ST        BIT(30)
+
+/* err status bit */
+#define TG_ERR_ST      BIT(4)
+#define TG_GBERR_ST    BIT(5)
+#define CQ_CODE_ERR_ST BIT(6)
+#define CQ_APB_ERR_ST  BIT(7)
+#define CQ_VS_ERR_ST   BIT(8)
+#define AMX_ERR_ST     BIT(15)
+#define RMX_ERR_ST     BIT(16)
+#define BMX_ERR_ST     BIT(17)
+#define RRZO_ERR_ST    BIT(18)
+#define AFO_ERR_ST     BIT(19)
+#define IMGO_ERR_ST    BIT(20)
+#define AAO_ERR_ST     BIT(21)
+#define PSO_ERR_ST     BIT(22)
+#define LCSO_ERR_ST    BIT(23)
+#define BNR_ERR_ST     BIT(24)
+#define LSCI_ERR_ST    BIT(25)
+#define DMA_ERR_ST     BIT(29)
+
+/* CAM DMA done status */
+#define FLKO_DONE_ST   BIT(4)
+#define AFO_DONE_ST    BIT(5)
+#define AAO_DONE_ST    BIT(7)
+#define PSO_DONE_ST    BIT(14)
+
+/* IRQ signal mask */
+#define INT_ST_MASK_CAM        ( \
+                       VS_INT_ST |\
+                       HW_PASS1_DON_ST |\
+                       SOF_INT_ST |\
+                       SW_PASS1_DON_ST)
+
+/* IRQ Warning Mask */
+#define INT_ST_MASK_CAM_WARN   (\
+                               RRZO_ERR_ST |\
+                               AFO_ERR_ST |\
+                               IMGO_ERR_ST |\
+                               AAO_ERR_ST |\
+                               PSO_ERR_ST | \
+                               LCSO_ERR_ST |\
+                               BNR_ERR_ST |\
+                               LSCI_ERR_ST)
+
+/* IRQ Error Mask */
+#define INT_ST_MASK_CAM_ERR    (\
+                               TG_ERR_ST |\
+                               TG_GBERR_ST |\
+                               CQ_CODE_ERR_ST |\
+                               CQ_APB_ERR_ST |\
+                               CQ_VS_ERR_ST |\
+                               BNR_ERR_ST |\
+                               RMX_ERR_ST |\
+                               BMX_ERR_ST |\
+                               BNR_ERR_ST |\
+                               LSCI_ERR_ST |\
+                               DMA_ERR_ST)
+
+/* IRQ Signal Log Mask */
+#define INT_ST_LOG_MASK_CAM    (\
+                               SOF_INT_ST |\
+                               SW_PASS1_DON_ST |\
+                               VS_INT_ST |\
+                               TG_ERR_ST |\
+                               TG_GBERR_ST |\
+                               RRZO_ERR_ST |\
+                               AFO_ERR_ST |\
+                               IMGO_ERR_ST |\
+                               AAO_ERR_ST |\
+                               DMA_ERR_ST)
+
+/* DMA Event Notification Mask */
+#define DMA_ST_MASK_CAM        (\
+                       AFO_DONE_ST |\
+                       AAO_DONE_ST |\
+                       PSO_DONE_ST |\
+                       FLKO_DONE_ST)

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h line 114

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-regs.h new file +10 -0
+
+/* Status check */
+#define REG_CTL_EN                     0x0004
+#define REG_CTL_DMA_EN                 0x0008
+#define REG_CTL_FMT_SEL                0x0010
+#define REG_CTL_EN2                    0x0018
+#define REG_CTL_RAW_INT_EN             0x0020
+#define REG_CTL_RAW_INT_STAT           0x0024
+#define REG_CTL_RAW_INT2_STAT          0x0034
+#define REG_CTL_RAW_INT3_STAT          0x00C4

nit: Please use lowercase hex literals.

Quoted 1 line

[snip]

Ok, we will fix this coding style issue in next patch.

Quoted 11 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem-drv.c

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem-drv.c new file +0 -0
new file mode 100644
index 0000000..020c38c

I don't think we need any of the code that is in this file. We should just use the DMA API. We should be able to create appropriate reserved memory pools in DT and properly assign them to the right allocating devices.

Quoted 2 lines

Skipping review of this file for the time being.

For this file, we may need your help.
Its purpose is same as DIP SMEM driver.
It is used for creating the ISP P1 specific vb2 buffer allocation context with reserved memory. Unfortunately, the implementation of mtk_cam-smem-drive.c is our best solution now.

Could you give us more hints how to implement?
Or do you think we could leverage the implementation from "Samsung S5P Multi Format Codec driver"? drivers/media/platform/s5p-mfc/s5p_mfc.c
- s5p_mfc_configure_dma_memory function
- s5p_mfc_configure_2port_memory
- s5p_mfc_alloc_memdev

Quoted 26 lines

[snip]

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.h

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem.h new file +0 -0
new file mode 100644
index 0000000..4e1cf20

Ditto.

[snip]

Quoted 16 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
new file mode 100644
index 0000000..7da312d
@@ -0,0 +1,1555 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2018 Mediatek Corporation.
+ * Copyright (c) 2017 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 15

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * MTK_CAM-v4l2 is highly based on Intel IPU 3 chrome driver

To be precise, it's not a "chrome driver". I think you could just say "IPU3 ImgU driver". (The driver was merged in Linux 5.0.)

OK, we will revise our wording.

Quoted 20 lines
Quoted 20 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +20 -0
+ *
+ */
+
+#include <linux/module.h>
+#include <linux/pm_runtime.h>
+#include <linux/videodev2.h>
+#include <media/v4l2-ioctl.h>
+#include <media/videobuf2-dma-contig.h>
+#include <media/v4l2-subdev.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-fwnode.h>
+#include <linux/device.h>
+#include <linux/platform_device.h>
+#include <linux/of.h>
+#include <linux/of_graph.h>
+#include <media/v4l2-common.h>
+#include <media/media-entity.h>
+#include <media/v4l2-async.h>
+
+#include "mtk_cam.h"
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 40

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+#include "mtk_cam-dev.h"
+#include "mtk_cam-v4l2.h"
+#include "mtk_cam-v4l2-util.h"
+
+#define CONFIG_MEDIATEK_MEDIA_REQUEST

I guess this is for testing, but I'd suggest switching to the target setup sooner than later and dropping this and related #ifdefs.

Yes, it is testing purpose and has been removed in our latest codebase.

Quoted 33 lines
Quoted 33 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +33 -0
+
+#define MTK_CAM_SENSOR_MAIN_PAD_SRC 0
+#define MTK_CAM_SENSOR_SUB_PAD_SRC 0
+#define MTK_CAM_SENSOR_IF_PAD_MAIN_SINK 0
+#define MTK_CAM_SENSOR_IF_PAD_SUB_SINK 1
+#define MTK_CAM_SENSOR_IF_PAD_SRC 4
+#define MTK_CAM_CIO_PAD_SINK 0
+
+static u32 mtk_cam_node_get_v4l2_cap
+       (struct mtk_cam_ctx_queue *node_ctx);
+
+static int mtk_cam_videoc_s_meta_fmt(struct file *file,
+                                    void *fh, struct v4l2_format *f);
+
+static int mtk_cam_subdev_open(struct v4l2_subdev *sd,
+                              struct v4l2_subdev_fh *fh)
+{
+       struct mtk_cam_dev *isp_dev = mtk_cam_subdev_to_dev(sd);
+
+       isp_dev->ctx.fh = fh;
+       return mtk_cam_ctx_open(&isp_dev->ctx);
+}
+
+static int mtk_cam_subdev_close(struct v4l2_subdev *sd,
+                               struct v4l2_subdev_fh *fh)
+{
+       struct mtk_cam_dev *isp_dev = mtk_cam_subdev_to_dev(sd);
+
+       return mtk_cam_ctx_release(&isp_dev->ctx);
+}
+
+static int mtk_cam_subdev_s_stream(struct v4l2_subdev *sd,
+                                  int enable)
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 79

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+{
+       int ret = 0;
+       struct mtk_cam_dev *isp_dev = mtk_cam_subdev_to_dev(sd);
+       struct mtk_cam_io_connection *cio = &isp_dev->cio;
+
+       if (enable) {

Please extract enable and disable cases into separate functions.

Ok, we will revise current implementation as below:

static int mtk_cam_subdev_s_stream(struct v4l2_subdev *sd,
int enable)

{
	struct mtk_cam_dev *isp_dev = mtk_cam_subdev_to_dev(sd);
	struct mtk_cam_io_connection *cio = &isp_dev->cio;

	if (enable)
		return mtk_cam_cio_stream_on(isp_dev, cio);
	else
		return mtk_cam_cio_stream_off(isp_dev, cio);
}
Quoted 23 lines
Quoted 23 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +23 -0
+               /* Get sensor interace and sensor sub device */
+               /* If the call succeeds, sensor if and sensor are filled */
+               /* in isp_dev->cio->sensor_if and isp_dev->cio->sensor */
+               ret = mtk_cam_v4l2_discover_sensor(isp_dev);
+               if (ret) {
+                       dev_err(&isp_dev->pdev->dev,
+                               "no sensor or sensor if connected (%d)\n",
+                               ret);
+                       return -EPERM;
+               }
+
+               /* seninf must stream on first */
+               ret = v4l2_subdev_call(cio->sensor_if, video, s_stream, 1);
+               if (ret) {
+                       dev_err(&isp_dev->pdev->dev,
+                               "sensor-if(%s) stream on failed (%d)\n",
+                               cio->sensor_if->entity.name, ret);
+                       return -EPERM;
+               }
+
+               dev_dbg(&isp_dev->pdev->dev, "streamed on sensor-if(%s)\n",
+                       cio->sensor_if->entity.name);
+
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 108

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+               ret = v4l2_subdev_call(cio->sensor, video, s_stream, 1);
+               if (ret) {
+                       dev_err(&isp_dev->pdev->dev,
+                               "sensor(%s) stream on failed (%d)\n",
+                               cio->sensor->entity.name, ret);
+                       return -EPERM;

We should undo any previous operations.

OK, we will add "undo procedure" in the new function.

Quoted 17 lines
Quoted 17 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +17 -0
+               }
+
+               dev_dbg(&isp_dev->pdev->dev, "streamed on sensor(%s)\n",
+                       cio->sensor->entity.name);
+
+               ret = mtk_cam_ctx_streamon(&isp_dev->ctx);
+               if (ret) {
+                       dev_err(&isp_dev->pdev->dev,
+                               "Pass 1 stream on failed (%d)\n", ret);
+                       return -EPERM;
+               }
+
+               isp_dev->mem2mem2.streaming = enable;
+
+               ret = mtk_cam_dev_queue_buffers(isp_dev, true);
+               if (ret)
+                       dev_err(&isp_dev->pdev->dev,
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 130

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+                               "failed to queue initial buffers (%d)", ret);
+
+               dev_dbg(&isp_dev->pdev->dev, "streamed on Pass 1\n");
+       } else {
+               if (cio->sensor) {

Is it possible to have cio->sensor NULL here? This function would have failed if it wasn't found when enabling.

In the original design, it is protected to avoid abnormal double stream off (s_stream) call from upper layer. For stability reason, it is better to check.

Quoted 74 lines
Quoted 7 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +7 -0
+                       ret = v4l2_subdev_call(cio->sensor, video, s_stream, 0);
+                       if (ret) {
+                               dev_err(&isp_dev->pdev->dev,
+                                       "sensor(%s) stream off failed (%d)\n",
+                                       cio->sensor->entity.name, ret);
+                       } else {
+                               dev_dbg(&isp_dev->pdev->dev,

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 147

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+                                       "streamed off sensor(%s)\n",
+                                       cio->sensor->entity.name);
+                               cio->sensor = NULL;
+                       }
+               } else {
+                       dev_err(&isp_dev->pdev->dev,
+                               "Can't find sensor connected\n");
+               }
+
+               if (cio->sensor_if) {

Ditto.

Quoted 54 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +54 -0
+                       ret = v4l2_subdev_call(cio->sensor_if, video,
+                                              s_stream, 0);
+                       if (ret) {
+                               dev_err(&isp_dev->pdev->dev,
+                                       "sensor if(%s) stream off failed (%d)\n",
+                                       cio->sensor_if->entity.name, ret);
+                       } else {
+                               dev_dbg(&isp_dev->pdev->dev,
+                                       "streamed off sensor-if(%s)\n",
+                                       cio->sensor_if->entity.name);
+                               cio->sensor_if = NULL;
+                       }
+               } else {
+                       dev_err(&isp_dev->pdev->dev,
+                               "Can't find sensor-if connected\n");
+               }
+
+               ret = mtk_cam_ctx_streamoff(&isp_dev->ctx);
+               if (ret) {
+                       dev_err(&isp_dev->pdev->dev,
+                               "Pass 1 stream off failed (%d)\n", ret);
+                       return -EPERM;
+               }
+
+               isp_dev->mem2mem2.streaming = false;
+
+               dev_dbg(&isp_dev->pdev->dev, "streamed off Pass 1\n");
+       }
+
+       return 0;
+}
+
+static void v4l2_event_merge(const struct v4l2_event *old,
+                            struct v4l2_event *new)
+{
+       struct mtk_cam_dev_stat_event_data *old_evt_stat_data =
+               (void *)old->u.data;
+       struct mtk_cam_dev_stat_event_data *new_evt_stat_data =
+               (void *)new->u.data;
+
+       if (old->type == V4L2_EVENT_MTK_CAM_ENGINE_STATE &&
+           new->type == V4L2_EVENT_MTK_CAM_ENGINE_STATE) {
+               pr_debug("%s, merge IRQ, old(type(0x%x) frame no(%d) IRQ(0x%x) DMA(0x%x)), new(type(0x%x) frame_number(%d) IRQ(0x%x) DMA(0x%x))",
+                        __func__,
+                       old->type,
+                       old_evt_stat_data->frame_number,
+                       old_evt_stat_data->irq_status_mask,
+                       old_evt_stat_data->dma_status_mask,
+                       new->type,
+                       new_evt_stat_data->frame_number,
+                       new_evt_stat_data->irq_status_mask,
+                       new_evt_stat_data->dma_status_mask);
+
+               /* merge IRQ event */
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 206

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+               new_evt_stat_data->irq_status_mask |=
+                       old_evt_stat_data->irq_status_mask;
+               new_evt_stat_data->dma_status_mask |=
+                       old_evt_stat_data->dma_status_mask;
+       }

What are these custom events for? Userspace shouldn't need to know anything about IRQ and DMA.

We will remove custom event and change to use V4L2_EVENT_FRAME_SYNC event.

Quoted 12 lines

[snip]

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 294

Quoted 8 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +8 -0
+static void mtk_cam_vb2_buf_queue(struct vb2_buffer *vb)
+{
+       struct mtk_cam_mem2mem2_device *m2m2 = vb2_get_drv_priv(vb->vb2_queue);
+
+       struct mtk_cam_dev *mtk_cam_dev = mtk_cam_m2m_to_dev(m2m2);
+
+       struct device *dev = &mtk_cam_dev->pdev->dev;
+

nit: Please remove these unnecessary blank lines between variables.

Ok, we will fix this kind of coding style issue in next patch.

Quoted 8 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 297

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +3 -0
+       struct mtk_cam_dev_buffer *buf = NULL;
+
+       struct vb2_v4l2_buffer *v4l2_buf = NULL;

Please default-initialize local variables only if really needed. Otherwise it prevents the compiler from catching missing assignments for you.

Ditto.

Quoted 8 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 302

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+
+       struct mtk_cam_dev_video_device *node =
+               mtk_cam_vbq_to_isp_node(vb->vb2_queue);
+
+       int queue = mtk_cam_dev_get_queue_id_of_dev_node(mtk_cam_dev, node);

Why not just add queue_id field to mtk_cam_dev_video_device?

Yes, it is a good idea. We will add this field for mtk_cam_dev_video_device.

Quoted 4 lines
Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +4 -0
+
+       dev_dbg(dev,
+               "queue vb2_buf: Node(%s) queue id(%d)\n",
+               node->name,
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 312

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+               queue);
+
+       if (queue < 0) {
+               dev_err(m2m2->dev, "Invalid mtk_cam_dev node.\n");
+               return;
+       }

I don't think this is possible.

Ok, we will remove this kind of logic check.

Quoted 34 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 315

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +3 -0
+
+       if (!vb)
+               pr_err("VB can't be null\n");

Ditto.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 320

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+
+       buf = mtk_cam_vb2_buf_to_dev_buf(vb);
+
+       if (!buf)
+               pr_err("buf can't be null\n");

Ditto.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 325

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+
+       v4l2_buf = to_vb2_v4l2_buffer(vb);
+
+       if (!v4l2_buf)
+               pr_err("v4l2_buf can't be null\n");

Ditto.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 334

Quoted 9 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +9 -0
+
+       mutex_lock(&mtk_cam_dev->lock);
+
+       pr_debug("init  mtk_cam_ctx_buf, sequence(%d)\n", v4l2_buf->sequence);
+
+       /* the dma address will be filled in later frame buffer handling*/
+       mtk_cam_ctx_buf_init(&buf->ctx_buf, queue, (dma_addr_t)0);
+       pr_debug("set mtk_cam_ctx_buf_init: user seq=%d\n",
+                buf->ctx_buf.user_sequence);

This should normally happen in the buf_init vb2 operation.

Quoted 7 lines

By the way, what's the point of having the DMA address argument to mtk_cam_ctx_buf_init() if it's always 0?

Given the above, is there even a reason for mtk_cam_ctx_buf_init() to exist? It sounds like the structure could be just initialize directly in buf_init.

Based on comment part 1, we have revised this implementation. There is no need to perform buffer initialization in this function.

Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 339

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+
+       /* Added the buffer into the tracking list */
+       list_add_tail(&buf->m2m2_buf.list,
+                     &m2m2->nodes[node - m2m2->nodes].buffers);
+       mutex_unlock(&mtk_cam_dev->lock);

If this mutex is only for the buffer list, I'd consider replacing it with a spinlock.

Yes, in the new version, we change to use spinlock to protect list data, not mutex lock.

e.g.

	buf = mtk_cam_vb2_buf_to_dev_buf(vb);

	/* Added the buffer into the tracking list */
	spin_lock(&node->slock);
	list_add_tail(&buf->list, &node->pending_list);
	spin_unlock(&node->slock);
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 345

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+
+       /* Enqueue the buffer */
+       if (mtk_cam_dev->mem2mem2.streaming) {
+               pr_debug("%s: mtk_cam_dev_queue_buffers\n",
+                        node->name);
+               mtk_cam_dev_queue_buffers(mtk_cam_dev, false);

Doesn't mtk_cam_dev_queue_buffers() already check if streaming is active?

Yes, we could remove the logic check in this function and count on the checking of mtk_cam_dev_queue_buffers() function.

Quoted 5 lines

Okay, let me send the comments above and continue third part of the review in another email.

Best regards,
Tomasz

Thanks for your valued comments on part 2.
It is helpful for us to make our driver implementation better.

We'd like to know your opinion about the schedule for RFC V1. Do you suggest us to send RFC V1 patch set after revising all comments on part 1 & 2 or wait for part 3 review?

Best regards,

Jungo

#16 lore raw

Re: [RFC PATCH V0 7/7] [media] platform: mtk-isp: Add Mediatek ISP Pass 1 driver

Quoted 2 lines · Jungo Lin

On Thu, 2019-03-07 at 19:04 +0900, Tomasz Figa wrote:

[snip]

Quoted 20 lines
Quoted 11 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem-drv.c

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem-drv.c new file +0 -0
new file mode 100644
index 0000000..020c38c

I don't think we need any of the code that is in this file. We should just use the DMA API. We should be able to create appropriate reserved memory pools in DT and properly assign them to the right allocating devices.

Quoted 2 lines

Skipping review of this file for the time being.

For this file, we may need your help.
Its purpose is same as DIP SMEM driver.
It is used for creating the ISP P1 specific vb2 buffer allocation context with reserved memory. Unfortunately, the implementation of mtk_cam-smem-drive.c is our best solution now.

Quoted 7 lines

Could you give us more hints how to implement?
Or do you think we could leverage the implementation from "Samsung S5P Multi Format Codec driver"? drivers/media/platform/s5p-mfc/s5p_mfc.c
- s5p_mfc_configure_dma_memory function
- s5p_mfc_configure_2port_memory
- s5p_mfc_alloc_memdev

I think we can indeed take some ideas from there. I need some time to check this and give you more details.

[snip]

Quoted 27 lines
Quoted 17 lines
Quoted 17 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem-drv.c new file +17 -0
+               }
+
+               dev_dbg(&isp_dev->pdev->dev, "streamed on sensor(%s)\n",
+                       cio->sensor->entity.name);
+
+               ret = mtk_cam_ctx_streamon(&isp_dev->ctx);
+               if (ret) {
+                       dev_err(&isp_dev->pdev->dev,
+                               "Pass 1 stream on failed (%d)\n", ret);
+                       return -EPERM;
+               }
+
+               isp_dev->mem2mem2.streaming = enable;
+
+               ret = mtk_cam_dev_queue_buffers(isp_dev, true);
+               if (ret)
+                       dev_err(&isp_dev->pdev->dev,
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem-drv.c

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem-drv.c new file +5 -0
+                               "failed to queue initial buffers (%d)", ret);
+
+               dev_dbg(&isp_dev->pdev->dev, "streamed on Pass 1\n");
+       } else {
+               if (cio->sensor) {

Is it possible to have cio->sensor NULL here? This function would have failed if it wasn't found when enabling.

Quoted 3 lines

In the original design, it is protected to avoid abnormal double stream off (s_stream) call from upper layer. For stability reason, it is better to check.

If so, having some state (e.g. field in a struct) for tracking the streaming state would make the code much easier to understand. Also, the error message on the else case is totally misleading, because it complains about a missing sensor, rather than double s_stream.

[snip]

Quoted 6 lines

Thanks for your valued comments on part 2.
It is helpful for us to make our driver implementation better.

We'd like to know your opinion about the schedule for RFC V1. Do you suggest us to send RFC V1 patch set after revising all comments on part 1 & 2 or wait for part 3 review?

I'm going to be a bit busy for the next few days, so it may be a good idea to address the comments for parts 1, 2 and 3 and send RFC V1. Also, for the more general comments, please check if they don't apply to the other drivers too (DIP, FD, Seninf, MDP). Thanks in advance!

Best regards,
Tomasz

#17 lore raw

Re: [RFC PATCH V0 7/7] [media] platform: mtk-isp: Add Mediatek ISP Pass 1 driver

Quoted 32 lines · Tomasz Figa
Quoted 2 lines · Jungo Lin

On Thu, 2019-03-07 at 19:04 +0900, Tomasz Figa wrote:

[snip]

Quoted 20 lines
Quoted 11 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem-drv.c

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem-drv.c new file +0 -0
new file mode 100644
index 0000000..020c38c

I don't think we need any of the code that is in this file. We should just use the DMA API. We should be able to create appropriate reserved memory pools in DT and properly assign them to the right allocating devices.

Quoted 2 lines

Skipping review of this file for the time being.

For this file, we may need your help.
Its purpose is same as DIP SMEM driver.
It is used for creating the ISP P1 specific vb2 buffer allocation context with reserved memory. Unfortunately, the implementation of mtk_cam-smem-drive.c is our best solution now.

Quoted 7 lines

Could you give us more hints how to implement?
Or do you think we could leverage the implementation from "Samsung S5P Multi Format Codec driver"? drivers/media/platform/s5p-mfc/s5p_mfc.c
- s5p_mfc_configure_dma_memory function
- s5p_mfc_configure_2port_memory
- s5p_mfc_alloc_memdev

Quoted 4 lines

I think we can indeed take some ideas from there. I need some time to check this and give you more details.

[snip]

Thanks for your support.
If you have any input, please kindly let us know.
We will list this revision in the to-do list of V1 version. At the same time, we will also continue to investigate how to implement based on current information.

Quoted 31 lines
Quoted 27 lines
Quoted 17 lines
Quoted 17 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem-drv.c new file +17 -0
+               }
+
+               dev_dbg(&isp_dev->pdev->dev, "streamed on sensor(%s)\n",
+                       cio->sensor->entity.name);
+
+               ret = mtk_cam_ctx_streamon(&isp_dev->ctx);
+               if (ret) {
+                       dev_err(&isp_dev->pdev->dev,
+                               "Pass 1 stream on failed (%d)\n", ret);
+                       return -EPERM;
+               }
+
+               isp_dev->mem2mem2.streaming = enable;
+
+               ret = mtk_cam_dev_queue_buffers(isp_dev, true);
+               if (ret)
+                       dev_err(&isp_dev->pdev->dev,
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem-drv.c

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-smem-drv.c new file +5 -0
+                               "failed to queue initial buffers (%d)", ret);
+
+               dev_dbg(&isp_dev->pdev->dev, "streamed on Pass 1\n");
+       } else {
+               if (cio->sensor) {

Is it possible to have cio->sensor NULL here? This function would have failed if it wasn't found when enabling.

Quoted 3 lines

In the original design, it is protected to avoid abnormal double stream off (s_stream) call from upper layer. For stability reason, it is better to check.

Quoted 7 lines

If so, having some state (e.g. field in a struct) for tracking the streaming state would make the code much easier to understand. Also, the error message on the else case is totally misleading, because it complains about a missing sensor, rather than double s_stream.

[snip]

Yes, your suggestion is helpful.
We will correct our implementation to make it more clear in next version.

Quoted 7 lines
Quoted 6 lines

Thanks for your valued comments on part 2.
It is helpful for us to make our driver implementation better.

We'd like to know your opinion about the schedule for RFC V1. Do you suggest us to send RFC V1 patch set after revising all comments on part 1 & 2 or wait for part 3 review?

Quoted 8 lines

I'm going to be a bit busy for the next few days, so it may be a good idea to address the comments for parts 1, 2 and 3 and send RFC V1. Also, for the more general comments, please check if they don't apply to the other drivers too (DIP, FD, Seninf, MDP). Thanks in advance!

Best regards,
Tomasz

Ok, we plan to send our RFC V1 for ISP P1 driver next week after revising current comments.
For DIP & FD drivers, they also have common implementations with P1 driver. We will sync current comments to them. For Seninf & MDP drivers. we will share our coding style & coding defect issues to them.

Moreover, we will provide our V4L2_Compliance testing report from RFC V1.

Best regards,

Jungo

Quoted 4 lines

_______________________________________________
Linux-mediatek mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-mediatek

#18 lore raw

Re: [RFC PATCH V0 7/7] [media] platform: mtk-isp: Add Mediatek ISP Pass 1 driver

In reply to Jungo Lin
Quoted 2 lines · Jungo Lin

On Thu, 2019-03-07 at 19:04 +0900, Tomasz Figa wrote:

[snip]

Quoted 72 lines
Quoted 8 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 126

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +3 -0
+struct mtk_cam_mem2mem2_device {
+       const char *name;
+       const char *model;

For both of the fields above, they seem to be always MTK_CAM_DEV_P1_NAME, so we can just use the macro directly whenever needed. No need for this indirection.

OK. These two fields will be removed in next patch.

Quoted 7 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 130

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +4 -0
+       struct device *dev;
+       int num_nodes;
+       struct mtk_cam_dev_video_device *nodes;
+       const struct vb2_mem_ops *vb2_mem_ops;

This is always "vb2_dma_contig_memops", so it can be used directly.

Ditto.

Quoted 5 lines
Quoted 1 line

+ unsigned int buf_struct_size;

This is always sizeof(struct mtk_cam_dev_buffer), so no need to save it in the struct.

Ditto.

Quoted 7 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 134

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +3 -0
+       int streaming;
+       struct v4l2_device *v4l2_dev;
+       struct media_device *media_dev;

These 2 fields are already in mtk_cam_dev which is a superclass of this struct. One can just access them from there directly.

Ditto.

Quoted 7 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 136

Quoted 2 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +2 -0
+       struct media_pipeline pipeline;
+       struct v4l2_subdev subdev;

Could you remind me what was the media topology exposed by this driver? This is already the second subdev I spotted in this patch, which looks strange.

For sub-device design, we will remove the sub-device for CIO and keep only one sub-device for ISP driver in next patch. We will also provide the media topology in RFC v1 patch to clarify.

Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 140

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +4 -0
+       struct media_pad *subdev_pads;
+       struct v4l2_file_operations v4l2_file_ops;
+       const struct file_operations fops;
+};

Given most of the comments above, it looks like the remaining useful fields in this struct could be just moved to mtk_cam_dev, without the need for this separate struct.

This is the final revision for these two structures. Do you suggest to merge it to simplify?

struct mtk_cam_mem2mem2_device {
        struct mtk_cam_video_device *nodes;
        struct media_pipeline pipeline;
        struct v4l2_subdev subdev;
        struct media_pad *subdev_pads;
};
Quoted 11 lines
struct mtk_cam_dev {
        struct platform_device *pdev;
        struct mtk_cam_video_device     mem2mem2_nodes[MTK_CAM_DEV_NODE_MAX];
        struct mtk_cam_mem2mem2_device mem2mem2;
        struct mtk_cam_io_connection cio;
        struct v4l2_device v4l2_dev;
        struct media_device media_dev;
        struct mtk_cam_ctx ctx;
        struct v4l2_async_notifier notifier;
};

I feel like there is not much benefit in having this split. Similarly, I'm not sure if there is a reason to have separate structs for mtk_cam_io_connection and mtk_cam_ctx.

(Sorry, missed this one in previous reply.)

Best regards,
Tomasz

#19 lore raw

Re: [RFC PATCH V0 7/7] [media] platform: mtk-isp: Add Mediatek ISP Pass 1 driver

Quoted 88 lines · Tomasz Figa
Quoted 2 lines · Jungo Lin

On Thu, 2019-03-07 at 19:04 +0900, Tomasz Figa wrote:

[snip]

Quoted 72 lines
Quoted 8 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 126

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +3 -0
+struct mtk_cam_mem2mem2_device {
+       const char *name;
+       const char *model;

For both of the fields above, they seem to be always MTK_CAM_DEV_P1_NAME, so we can just use the macro directly whenever needed. No need for this indirection.

OK. These two fields will be removed in next patch.

Quoted 7 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 130

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +4 -0
+       struct device *dev;
+       int num_nodes;
+       struct mtk_cam_dev_video_device *nodes;
+       const struct vb2_mem_ops *vb2_mem_ops;

This is always "vb2_dma_contig_memops", so it can be used directly.

Ditto.

Quoted 5 lines
Quoted 1 line

+ unsigned int buf_struct_size;

This is always sizeof(struct mtk_cam_dev_buffer), so no need to save it in the struct.

Ditto.

Quoted 7 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 134

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +3 -0
+       int streaming;
+       struct v4l2_device *v4l2_dev;
+       struct media_device *media_dev;

These 2 fields are already in mtk_cam_dev which is a superclass of this struct. One can just access them from there directly.

Ditto.

Quoted 7 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 136

Quoted 2 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +2 -0
+       struct media_pipeline pipeline;
+       struct v4l2_subdev subdev;

Could you remind me what was the media topology exposed by this driver? This is already the second subdev I spotted in this patch, which looks strange.

For sub-device design, we will remove the sub-device for CIO and keep only one sub-device for ISP driver in next patch. We will also provide the media topology in RFC v1 patch to clarify.

Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h line 140

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev.h new file +4 -0
+       struct media_pad *subdev_pads;
+       struct v4l2_file_operations v4l2_file_ops;
+       const struct file_operations fops;
+};

Given most of the comments above, it looks like the remaining useful fields in this struct could be just moved to mtk_cam_dev, without the need for this separate struct.

This is the final revision for these two structures. Do you suggest to merge it to simplify?

struct mtk_cam_mem2mem2_device {
        struct mtk_cam_video_device *nodes;
        struct media_pipeline pipeline;
        struct v4l2_subdev subdev;
        struct media_pad *subdev_pads;
};
Quoted 11 lines
struct mtk_cam_dev {
        struct platform_device *pdev;
        struct mtk_cam_video_device     mem2mem2_nodes[MTK_CAM_DEV_NODE_MAX];
        struct mtk_cam_mem2mem2_device mem2mem2;
        struct mtk_cam_io_connection cio;
        struct v4l2_device v4l2_dev;
        struct media_device media_dev;
        struct mtk_cam_ctx ctx;
        struct v4l2_async_notifier notifier;
};
Quoted 8 lines

I feel like there is not much benefit in having this split. Similarly, I'm not sure if there is a reason to have separate structs for mtk_cam_io_connection and mtk_cam_ctx.

(Sorry, missed this one in previous reply.)

Best regards,
Tomasz

Ok, agree your comment.
We will remove both mtk_cam_io_connection and mtk_cam_ctx and merge those fields into mtk_cam_dev.

Thanks for your suggestion.

Best regards,

Jungo

#20 lore raw

Re: [RFC PATCH V0 7/7] [media] platform: mtk-isp: Add Mediatek ISP Pass 1 driver

In reply to Frederic Chen

Hi Frederic, Jungo,

Please see more comments inline.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 360

Quoted 12 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +12 -0
+static int mtk_cam_vb2_queue_setup(struct vb2_queue *vq,
+                                  unsigned int *num_buffers,
+                               unsigned int *num_planes,
+                               unsigned int sizes[],
+                               struct device *alloc_devs[])
+{
+       struct mtk_cam_mem2mem2_device *m2m2 = vb2_get_drv_priv(vq);
+       struct mtk_cam_dev_video_device *node =
+               mtk_cam_vbq_to_isp_node(vq);
+       struct mtk_cam_dev *isp_dev = mtk_cam_m2m_to_dev(m2m2);
+       struct device *dev = &isp_dev->pdev->dev;
+       void *buf_alloc_ctx = NULL;

Don't initialize by default, if not strictly necessary.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 366

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+       int queue_id = mtk_cam_dev_get_queue_id_of_dev_node(isp_dev, node);
+
+       /* Get V4L2 format with the following method */
+       const struct v4l2_format *fmt = &node->vdev_fmt;
+
+       *num_planes = 1;

This doesn't handle VIDIOC_CREATE_BUFS, which triggers a .queue_setup() call with *num_planes > 0 and sizes[] already initialized. The driver needs to validate that the sizes and number of planes are valid in that case and return -EINVAL otherwise. Perhaps you should try running v4l2-compliance
(https://git.linuxtv.org/v4l-utils.git/tree/utils/v4l2-compliance) on this driver, which should catch issues like this.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 372

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+       *num_buffers = clamp_val(*num_buffers, 1, VB2_MAX_FRAME);
+
+       if (isp_dev->ctx.queue[queue_id].desc.smem_alloc) {
+               buf_alloc_ctx = isp_dev->ctx.smem_vb2_alloc_ctx;
+               dev_dbg(dev, "Select smem_vb2_alloc_ctx(%llx)\n",
+                       (unsigned long long)buf_alloc_ctx);

Use %p for printing pointers.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 380

Quoted 8 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +8 -0
+       } else {
+               buf_alloc_ctx = isp_dev->ctx.default_vb2_alloc_ctx;
+               dev_dbg(dev, "Select default_vb2_alloc_ctx(%llx)\n",
+                       (unsigned long long)buf_alloc_ctx);
+       }
+
+       vq->dma_attrs |= DMA_ATTR_NON_CONSISTENT;
+       dev_dbg(dev, "queue(%d): cached mmap enabled\n", queue_id);

This isn't supported in upstream. (By the way, neither it is in Chrome OS 4.19 kernel. If we really need cached mmap for some reason, we should propose a proper solution upstream. I'd still first investigate why write-combine mmap is slow for operations that should be simple one-time writes or reads.)

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 388

Quoted 8 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +8 -0
+
+       if (vq->type == V4L2_BUF_TYPE_META_CAPTURE ||
+           vq->type == V4L2_BUF_TYPE_META_OUTPUT)
+               sizes[0] = fmt->fmt.meta.buffersize;
+       else
+               sizes[0] = fmt->fmt.pix_mp.plane_fmt[0].sizeimage;
+
+       alloc_devs[0] = (struct device *)buf_alloc_ctx;

Please don't add random type casts. If the compiler gives a type error, that normally means that there is something wrong elsewhere in the code (i.e. the type of buf_alloc_ctx variable is wrong here - it should be struct device *) and casting just masks the original problem.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 395

Quoted 7 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +7 -0
+
+       dev_dbg(dev, "queue(%d):type(%d),size(%d),alloc_ctx(%llx)\n",
+               queue_id, vq->type, sizes[0],
+               (unsigned long long)buf_alloc_ctx);
+
+       /* Initialize buffer queue */
+       INIT_LIST_HEAD(&node->buffers);

This is incorrect. .queue_setup() can be also called on VIDIOC_CREATE_BUFS, which must preserve the other buffers in the queue.

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +3 -0
+
+       return 0;
+}

[snip]

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 439

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+static int mtk_cam_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
+{
+       struct mtk_cam_mem2mem2_device *m2m2 = vb2_get_drv_priv(vq);
+       struct mtk_cam_dev_video_device *node =
+               mtk_cam_vbq_to_isp_node(vq);
+       int r;

nit: "ret" is more common and already used by some other functions in this patch.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 444

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+
+       if (m2m2->streaming) {
+               r = -EBUSY;
+               goto fail_return_bufs;
+       }

We shouldn't need to check this ourselves. It's not possible to have this call on an already streaming vb2 queue. Since we start streaming the m2m2 subdev only when all video nodes start streaming, this should never happen.

Quoted 55 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +55 -0
+
+       if (!node->enabled) {
+               pr_err("Node (%ld) is not enable\n", node - m2m2->nodes);
+               r = -EINVAL;
+               goto fail_return_bufs;
+       }
+
+       r = media_pipeline_start(&node->vdev.entity, &m2m2->pipeline);
+       if (r < 0) {
+               pr_err("Node (%ld) media_pipeline_start failed\n",
+                      node - m2m2->nodes);
+               goto fail_return_bufs;
+       }
+
+       if (!mtk_cam_all_nodes_streaming(m2m2, node))
+               return 0;
+
+       /* Start streaming of the whole pipeline now */
+
+       r = v4l2_subdev_call(&m2m2->subdev, video, s_stream, 1);
+       if (r < 0) {
+               pr_err("Node (%ld) v4l2_subdev_call s_stream failed\n",
+                      node - m2m2->nodes);
+               goto fail_stop_pipeline;
+       }
+       return 0;
+
+fail_stop_pipeline:
+       media_pipeline_stop(&node->vdev.entity);
+fail_return_bufs:
+       mtk_cam_return_all_buffers(m2m2, node, VB2_BUF_STATE_QUEUED);
+
+       return r;
+}
+
+static void mtk_cam_vb2_stop_streaming(struct vb2_queue *vq)
+{
+       struct mtk_cam_mem2mem2_device *m2m2 = vb2_get_drv_priv(vq);
+       struct mtk_cam_dev_video_device *node =
+               mtk_cam_vbq_to_isp_node(vq);
+       int r;
+
+       WARN_ON(!node->enabled);
+
+       /* Was this the first node with streaming disabled? */
+       if (mtk_cam_all_nodes_streaming(m2m2, node)) {
+               /* Yes, really stop streaming now */
+               r = v4l2_subdev_call(&m2m2->subdev, video, s_stream, 0);
+               if (r)
+                       dev_err(m2m2->dev, "failed to stop streaming\n");
+       }
+
+       mtk_cam_return_all_buffers(m2m2, node, VB2_BUF_STATE_ERROR);
+       media_pipeline_stop(&node->vdev.entity);
+}

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 509

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+
+static int mtk_cam_videoc_querycap(struct file *file, void *fh,
+                                  struct v4l2_capability *cap)
+{
+       struct mtk_cam_mem2mem2_device *m2m2 = video_drvdata(file);
+       struct mtk_cam_dev_video_device *node = file_to_mtk_cam_node(file);
+       struct mtk_cam_dev *isp_dev = mtk_cam_m2m_to_dev(m2m2);
+       int queue_id =
+               mtk_cam_dev_get_queue_id_of_dev_node(isp_dev, node);
+       struct mtk_cam_ctx_queue *node_ctx = &isp_dev->ctx.queue[queue_id];

It feels like this could be just stored as node->ctx.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 518

Quoted 9 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +9 -0
+
+       strlcpy(cap->driver, m2m2->name, sizeof(cap->driver));
+       strlcpy(cap->card, m2m2->model, sizeof(cap->card));
+       snprintf(cap->bus_info, sizeof(cap->bus_info), "platform:%s",
+                node->name);
+
+       cap->device_caps =
+               mtk_cam_node_get_v4l2_cap(node_ctx) | V4L2_CAP_STREAMING;
+       cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;

No need to set these 2 fields manually. They are filled in automatically from struct video_device::device_caps.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 523

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+
+       return 0;
+}
+
+/* Propagate forward always the format from the mtk_cam_dev subdev */

It doesn't seem to match what the function is doing, i.e. returning the format structure of the node itself. I'd just drop this comment. The code should be written in a self-explaining way anyway.

Quoted 15 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +15 -0
+static int mtk_cam_videoc_g_fmt(struct file *file, void *fh,
+                               struct v4l2_format *f)
+{
+       struct mtk_cam_dev_video_device *node = file_to_mtk_cam_node(file);
+
+       f->fmt = node->vdev_fmt.fmt;
+
+       return 0;
+}
+
+static int mtk_cam_videoc_try_fmt(struct file *file,
+                                 void *fh,
+        struct v4l2_format *f)
+{
+       struct mtk_cam_mem2mem2_device *m2m2 = video_drvdata(file);

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 548

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+       struct mtk_cam_dev *isp_dev = mtk_cam_m2m_to_dev(m2m2);
+       struct mtk_cam_ctx *dev_ctx = &isp_dev->ctx;
+       struct mtk_cam_dev_video_device *node = file_to_mtk_cam_node(file);
+       int queue_id =
+               mtk_cam_dev_get_queue_id_of_dev_node(isp_dev, node);
+       int ret = 0;
+
+       ret = mtk_cam_ctx_fmt_set_img(dev_ctx, queue_id,
+                                     &f->fmt.pix_mp,
+               &node->vdev_fmt.fmt.pix_mp);

Doesn't this actually change the current node format? VIDIOC_TRY_FMT must not have any side effects on current driver state.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 553

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+
+       /* Simply set the format to the node context in the initial version */
+       if (ret) {
+               pr_warn("Fmt(%d) not support for queue(%d), will load default fmt\n",
+                       f->fmt.pix_mp.pixelformat, queue_id);

No need for this warning.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 556

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +3 -0
+
+               ret =   mtk_cam_ctx_format_load_default_fmt
+                       (&dev_ctx->queue[queue_id], f);

Wouldn't this also change the current node state?

Also, something wrong with the number of spaces after "ret =".

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 561

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+       }
+
+       if (!ret) {
+               node->vdev_fmt.fmt.pix_mp = f->fmt.pix_mp;
+               dev_ctx->queue[queue_id].fmt.pix_mp = node->vdev_fmt.fmt.pix_mp;

Ditto.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 904

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +3 -0
+       }
+
+       return ret;

VIDIOC_TRY_FMT must not return an error unless for the cases described
in https://www.kernel.org/doc/html/latest/media/uapi/v4l/vidioc-g-fmt.html#return-value
.

Quoted 12 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +12 -0
+}
+
+static int mtk_cam_videoc_s_fmt(struct file *file, void *fh,
+                               struct v4l2_format *f)
+{
+       struct mtk_cam_mem2mem2_device *m2m2 = video_drvdata(file);
+       struct mtk_cam_dev *isp_dev = mtk_cam_m2m_to_dev(m2m2);
+       struct mtk_cam_ctx *dev_ctx = &isp_dev->ctx;
+       struct mtk_cam_dev_video_device *node = file_to_mtk_cam_node(file);
+       int queue_id = mtk_cam_dev_get_queue_id_of_dev_node(isp_dev, node);
+       int ret = 0;
+

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 586

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+       ret = mtk_cam_ctx_fmt_set_img(dev_ctx, queue_id,
+                                     &f->fmt.pix_mp,
+               &node->vdev_fmt.fmt.pix_mp);
+
+       /* Simply set the format to the node context in the initial version */
+       if (!ret)
+               dev_ctx->queue[queue_id].fmt.pix_mp = node->vdev_fmt.fmt.pix_mp;
+       else
+               dev_warn(&isp_dev->pdev->dev,
+                        "s_fmt, format not support\n");

No need for error messages for userspace errors.

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+       return ret;

Instead of opencoding most of this function, one would normally call mtk_cam_videoc_try_fmt() first to adjust the format struct and then just update the driver state with the adjusted format.

Also, similarly to VIDIOC_TRY_FMT, VIDIOC_SET_FMT doesn't fail unless in the very specific cases, as described in
https://www.kernel.org/doc/html/latest/media/uapi/v4l/vidioc-g-fmt.html#return-value
.

Quoted 32 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +32 -0
+}
+
+static int mtk_cam_enum_framesizes(struct file *filp, void *priv,
+                                  struct v4l2_frmsizeenum *sizes)
+{
+       struct mtk_cam_mem2mem2_device *m2m2 = video_drvdata(filp);
+       struct mtk_cam_dev *isp_dev = mtk_cam_m2m_to_dev(m2m2);
+       struct mtk_cam_dev_video_device *node =
+               file_to_mtk_cam_node(filp);
+       int queue_id = mtk_cam_dev_get_queue_id_of_dev_node(isp_dev, node);
+
+       if (sizes->index != 0)
+               return -EINVAL;
+
+       if (queue_id == MTK_CAM_CTX_P1_MAIN_STREAM_OUT) {
+               sizes->type = V4L2_FRMSIZE_TYPE_CONTINUOUS;
+               sizes->stepwise.max_width = CAM_B_MAX_WIDTH;
+               sizes->stepwise.min_width = CAM_MIN_WIDTH;
+               sizes->stepwise.max_height = CAM_B_MAX_HEIGHT;
+               sizes->stepwise.min_height = CAM_MIN_HEIGHT;
+               sizes->stepwise.step_height = 1;
+               sizes->stepwise.step_width = 1;
+       } else if (queue_id == MTK_CAM_CTX_P1_PACKED_BIN_OUT) {
+               sizes->type = V4L2_FRMSIZE_TYPE_CONTINUOUS;
+               sizes->stepwise.max_width = RRZ_MAX_WIDTH;
+               sizes->stepwise.min_width = RRZ_MIN_WIDTH;
+               sizes->stepwise.max_height = RRZ_MAX_HEIGHT;
+               sizes->stepwise.min_height = RRZ_MIN_HEIGHT;
+               sizes->stepwise.step_height = 1;
+               sizes->stepwise.step_width = 1;
+       }
+

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 630

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+       return 0;
+}
+
+static int mtk_cam_meta_enum_format(struct file *file,
+                                   void *fh, struct v4l2_fmtdesc *f)
+{
+       struct mtk_cam_dev_video_device *node = file_to_mtk_cam_node(file);
+
+       if (f->index > 0 || f->type != node->vbq.type)
+               return -EINVAL;

f->type is already checked by the V4L2 core. See v4l_enum_fmt().

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+
+       f->pixelformat = node->vdev_fmt.fmt.meta.dataformat;
+
+       return 0;
+}

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 645

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+
+static int mtk_cam_videoc_s_meta_fmt(struct file *file,
+                                    void *fh, struct v4l2_format *f)
+{
+       struct mtk_cam_mem2mem2_device *m2m2 = video_drvdata(file);
+       struct mtk_cam_dev *isp_dev = mtk_cam_m2m_to_dev(m2m2);
+       struct mtk_cam_ctx *dev_ctx = &isp_dev->ctx;
+       struct mtk_cam_dev_video_device *node = file_to_mtk_cam_node(file);
+       int queue_id = mtk_cam_dev_get_queue_id_of_dev_node(isp_dev, node);
+

No need for this blank line.

Quoted 1 line

+ int ret = 0;

Please don't default-initialize without a good reason.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 649

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +3 -0
+
+       if (f->type != node->vbq.type)
+               return -EINVAL;

Ditto.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 652

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +3 -0
+
+       ret = mtk_cam_ctx_format_load_default_fmt(&dev_ctx->queue[queue_id], f);
+

No need for this blank line.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 658

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+       if (!ret) {
+               node->vdev_fmt.fmt.meta = f->fmt.meta;
+               dev_ctx->queue[queue_id].fmt.meta = node->vdev_fmt.fmt.meta;
+       } else {
+               dev_warn(&isp_dev->pdev->dev,
+                        "s_meta_fm failed, format not support\n");

No need for this warning.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 905

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +4 -0
+       }
+
+       return ret;
+}

Actually, why do we even need to do all the things? Do we support multiple different meta formats on the same video node? If not, we can just have all the TRY_FMT/S_FMT/G_FMT return the hardcoded format.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 670

Quoted 8 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +8 -0
+
+static int mtk_cam_videoc_g_meta_fmt(struct file *file,
+                                    void *fh, struct v4l2_format *f)
+{
+       struct mtk_cam_dev_video_device *node = file_to_mtk_cam_node(file);
+
+       if (f->type != node->vbq.type)
+               return -EINVAL;

Not needed.

Quoted 7 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +7 -0
+
+       f->fmt = node->vdev_fmt.fmt;
+
+       return 0;
+}
+
+int mtk_cam_videoc_qbuf(struct file *file, void *priv, struct v4l2_buffer *p)

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 687

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+{
+       struct video_device *vdev = video_devdata(file);
+       struct vb2_buffer *vb;
+       struct mtk_cam_dev_buffer *db;
+       int r = 0;
+
+       /* check if vb2 queue is busy */
+       if (vdev->queue->owner &&
+           vdev->queue->owner != file->private_data)
+               return -EBUSY;

This should be already handled by the core.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 691

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +4 -0
+
+       /* Keep the value of sequence in v4l2_buffer */
+       /* in ctx buf since vb2_qbuf will set it to 0 */
+       vb = vdev->queue->bufs[p->index];

Why do you need a sequence number for buffers on queue time? The field is not specified to be set by the userspace and should be ignored by the driver. The driver should rely on the Request API to match any buffers together anyway.

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 701

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+
+       if (vb) {
+               db = mtk_cam_vb2_buf_to_dev_buf(vb);
+               pr_debug("qbuf: p:%llx,vb:%llx, db:%llx\n",
+                        (unsigned long long)p,
+                       (unsigned long long)vb,
+                       (unsigned long long)db);
+               db->ctx_buf.user_sequence = p->sequence;
+       }
+

Generally this driver shouldn't need to implement this callback itself. Instead it can just use the vb2_ioctl_qbuf() helper.

Quoted 107 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +107 -0
+       r = vb2_qbuf(vdev->queue, vdev->v4l2_dev->mdev, p);
+
+       if (r)
+               pr_err("vb2_qbuf failed(err=%d): buf idx(%d)\n",
+                      r, p->index);
+
+       return r;
+}
+EXPORT_SYMBOL_GPL(mtk_cam_videoc_qbuf);
+
+/******************** function pointers ********************/
+
+/* subdev internal operations */
+static const struct v4l2_subdev_internal_ops mtk_cam_subdev_internal_ops = {
+       .open = mtk_cam_subdev_open,
+       .close = mtk_cam_subdev_close,
+};
+
+static const struct v4l2_subdev_core_ops mtk_cam_subdev_core_ops = {
+       .subscribe_event = mtk_cam_subdev_subscribe_event,
+       .unsubscribe_event = mtk_cam_subdev_unsubscribe_event,
+};
+
+static const struct v4l2_subdev_video_ops mtk_cam_subdev_video_ops = {
+       .s_stream = mtk_cam_subdev_s_stream,
+};
+
+static const struct v4l2_subdev_ops mtk_cam_subdev_ops = {
+       .core = &mtk_cam_subdev_core_ops,
+       .video = &mtk_cam_subdev_video_ops,
+};
+
+static const struct media_entity_operations mtk_cam_media_ops = {
+       .link_setup = mtk_cam_link_setup,
+       .link_validate = v4l2_subdev_link_validate,
+};
+
+#ifdef CONFIG_MEDIATEK_MEDIA_REQUEST
+static void mtk_cam_vb2_buf_request_complete(struct vb2_buffer *vb)
+{
+       struct mtk_cam_mem2mem2_device *m2m2 = vb2_get_drv_priv(vb->vb2_queue);
+
+       v4l2_ctrl_request_complete(vb->req_obj.req,
+                                  m2m2->v4l2_dev->ctrl_handler);
+}
+#endif
+
+static const struct vb2_ops mtk_cam_vb2_ops = {
+       .buf_queue = mtk_cam_vb2_buf_queue,
+       .queue_setup = mtk_cam_vb2_queue_setup,
+       .start_streaming = mtk_cam_vb2_start_streaming,
+       .stop_streaming = mtk_cam_vb2_stop_streaming,
+       .wait_prepare = vb2_ops_wait_prepare,
+       .wait_finish = vb2_ops_wait_finish,
+#ifdef CONFIG_MEDIATEK_MEDIA_REQUEST
+       .buf_request_complete = mtk_cam_vb2_buf_request_complete,
+#endif
+};
+
+static const struct v4l2_file_operations mtk_cam_v4l2_fops = {
+       .unlocked_ioctl = video_ioctl2,
+       .open = v4l2_fh_open,
+       .release = vb2_fop_release,
+       .poll = vb2_fop_poll,
+       .mmap = vb2_fop_mmap,
+#ifdef CONFIG_COMPAT
+       .compat_ioctl32 = v4l2_compat_ioctl32,
+#endif
+};
+
+static const struct v4l2_ioctl_ops mtk_cam_v4l2_ioctl_ops = {
+       .vidioc_querycap = mtk_cam_videoc_querycap,
+       .vidioc_enum_framesizes = mtk_cam_enum_framesizes,
+
+       .vidioc_g_fmt_vid_cap_mplane = mtk_cam_videoc_g_fmt,
+       .vidioc_s_fmt_vid_cap_mplane = mtk_cam_videoc_s_fmt,
+       .vidioc_try_fmt_vid_cap_mplane = mtk_cam_videoc_try_fmt,
+
+       .vidioc_g_fmt_vid_out_mplane = mtk_cam_videoc_g_fmt,
+       .vidioc_s_fmt_vid_out_mplane = mtk_cam_videoc_s_fmt,
+       .vidioc_try_fmt_vid_out_mplane = mtk_cam_videoc_try_fmt,
+
+       /* buffer queue management */
+       .vidioc_reqbufs = vb2_ioctl_reqbufs,
+       .vidioc_create_bufs = vb2_ioctl_create_bufs,
+       .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+       .vidioc_querybuf = vb2_ioctl_querybuf,
+       .vidioc_qbuf = mtk_cam_videoc_qbuf,
+       .vidioc_dqbuf = vb2_ioctl_dqbuf,
+       .vidioc_streamon = vb2_ioctl_streamon,
+       .vidioc_streamoff = vb2_ioctl_streamoff,
+       .vidioc_expbuf = vb2_ioctl_expbuf,
+};
+
+static const struct v4l2_ioctl_ops mtk_cam_v4l2_meta_ioctl_ops = {
+       .vidioc_querycap = mtk_cam_videoc_querycap,
+
+       .vidioc_enum_fmt_meta_cap = mtk_cam_meta_enum_format,
+       .vidioc_g_fmt_meta_cap = mtk_cam_videoc_g_meta_fmt,
+       .vidioc_s_fmt_meta_cap = mtk_cam_videoc_s_meta_fmt,
+       .vidioc_try_fmt_meta_cap = mtk_cam_videoc_g_meta_fmt,
+
+       .vidioc_enum_fmt_meta_out = mtk_cam_meta_enum_format,
+       .vidioc_g_fmt_meta_out = mtk_cam_videoc_g_meta_fmt,
+       .vidioc_s_fmt_meta_out = mtk_cam_videoc_s_meta_fmt,
+       .vidioc_try_fmt_meta_out = mtk_cam_videoc_g_meta_fmt,
+

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 818

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+       .vidioc_reqbufs = vb2_ioctl_reqbufs,
+       .vidioc_create_bufs = vb2_ioctl_create_bufs,
+       .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+       .vidioc_querybuf = vb2_ioctl_querybuf,
+       .vidioc_qbuf = mtk_cam_videoc_qbuf,
+       .vidioc_dqbuf = vb2_ioctl_dqbuf,
+       .vidioc_streamon = vb2_ioctl_streamon,
+       .vidioc_streamoff = vb2_ioctl_streamoff,
+       .vidioc_expbuf = vb2_ioctl_expbuf,
+};

The ops should be split for each node type, {VIDEO, META} x {OUTPUT, CAPTURE}. Then the core would validate the type given to all the ioctls automatically.

Quoted 8 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +8 -0
+
+static u32 mtk_cam_node_get_v4l2_cap(struct mtk_cam_ctx_queue *node_ctx)
+{
+       u32 cap = 0;
+
+       if (node_ctx->desc.capture)
+               if (node_ctx->desc.image)
+                       cap = V4L2_CAP_VIDEO_CAPTURE_MPLANE;

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 836

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+               else
+                       cap = V4L2_CAP_META_CAPTURE;
+       else
+               if (node_ctx->desc.image)
+                       cap = V4L2_CAP_VIDEO_OUTPUT_MPLANE;
+               else
+                       cap = V4L2_CAP_META_OUTPUT;
+
+       return cap;
+}

Why not just have this defined statically as node_ctx->desc.cap?

Quoted 8 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +8 -0
+
+static u32 mtk_cam_node_get_format_type(struct mtk_cam_ctx_queue *node_ctx)
+{
+       u32 type;
+
+       if (node_ctx->desc.capture)
+               if (node_ctx->desc.image)
+                       type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 854

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+               else
+                       type = V4L2_BUF_TYPE_META_CAPTURE;
+       else
+               if (node_ctx->desc.image)
+                       type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
+               else
+                       type = V4L2_BUF_TYPE_META_OUTPUT;
+
+       return type;
+}

Why not just have this defined statically as node_ctx->desc.buf_type?

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 866

Quoted 12 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +12 -0
+
+static const struct v4l2_ioctl_ops *mtk_cam_node_get_ioctl_ops
+       (struct mtk_cam_ctx_queue *node_ctx)
+{
+       const struct v4l2_ioctl_ops *ops = NULL;
+
+       if (node_ctx->desc.image)
+               ops = &mtk_cam_v4l2_ioctl_ops;
+       else
+               ops = &mtk_cam_v4l2_meta_ioctl_ops;
+       return ops;
+}

It's also preferable to just put this inside some structure rather than have getter functions. (node_ctx->desc.ioctl_ops?)

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 879

Quoted 13 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +13 -0
+
+/* Config node's video properties */
+/* according to the device context requirement */
+static void mtk_cam_node_to_v4l2(struct mtk_cam_dev *isp_dev, u32 node,
+                                struct video_device *vdev,
+                                struct v4l2_format *f)
+{
+       u32 cap;
+       struct mtk_cam_ctx *device_ctx = &isp_dev->ctx;
+       struct mtk_cam_ctx_queue *node_ctx = &device_ctx->queue[node];
+
+       WARN_ON(node >= mtk_cam_dev_get_total_node(isp_dev));
+       WARN_ON(!node_ctx);

How is this possible?

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 889

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+
+       /* set cap of the node */
+       cap = mtk_cam_node_get_v4l2_cap(node_ctx);
+       f->type = mtk_cam_node_get_format_type(node_ctx);
+       vdev->ioctl_ops = mtk_cam_node_get_ioctl_ops(node_ctx);
+
+       if (mtk_cam_ctx_format_load_default_fmt(&device_ctx->queue[node], f)) {
+               dev_err(&isp_dev->pdev->dev,
+                       "Can't load default for node (%d): (%s)",
+               node, device_ctx->queue[node].desc.name);

mtk_cam_ctx_format_load_default_fmt() doesn't fail (and that's right). It should be actually made void.

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+       }       else {
+               if (device_ctx->queue[node].desc.image) {
+                       dev_dbg(&isp_dev->pdev->dev,
+                               "Node (%d): (%s), dfmt (f:0x%x w:%d: h:%d s:%d)\n",
+                       node, device_ctx->queue[node].desc.name,
+                       f->fmt.pix_mp.pixelformat,
+                       f->fmt.pix_mp.width,
+                       f->fmt.pix_mp.height,
+                       f->fmt.pix_mp.plane_fmt[0].sizeimage
+                       );

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 909

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+                       node_ctx->fmt.pix_mp = f->fmt.pix_mp;
+               } else {
+                       dev_dbg(&isp_dev->pdev->dev,
+                               "Node (%d): (%s), dfmt (f:0x%x s:%u)\n",
+                       node, device_ctx->queue[node].desc.name,
+                       f->fmt.meta.dataformat,
+                       f->fmt.meta.buffersize
+                       );
+                       node_ctx->fmt.meta = f->fmt.meta;
+               }

Drop the debugging messages and just replace the whole if/else block with:

node_ctx->fmt = f->fmt;

Sorry, ran out of time for today. Fourth part will come. :)

Best regards,
Tomasz

#21 lore raw

Re: [RFC PATCH V0 7/7] [media] platform: mtk-isp: Add Mediatek ISP Pass 1 driver

Quoted 4 lines · Tomasz Figa

Hi Frederic, Jungo,

Please see more comments inline.

Hi Tomasz:

Thanks for your part 3 comments.
Below is our feedback.

Quoted 6 lines
Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+static int mtk_cam_vb2_queue_setup(struct vb2_queue *vq,
+                                  unsigned int *num_buffers,
+                               unsigned int *num_planes,
+                               unsigned int sizes[],
+                               struct device *alloc_devs[])
+{
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 360

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+       struct mtk_cam_mem2mem2_device *m2m2 = vb2_get_drv_priv(vq);
+       struct mtk_cam_dev_video_device *node =
+               mtk_cam_vbq_to_isp_node(vq);
+       struct mtk_cam_dev *isp_dev = mtk_cam_m2m_to_dev(m2m2);
+       struct device *dev = &isp_dev->pdev->dev;
+       void *buf_alloc_ctx = NULL;

Don't initialize by default, if not strictly necessary.

Ok, fixed in next patch.

Quoted 7 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 366

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+       int queue_id = mtk_cam_dev_get_queue_id_of_dev_node(isp_dev, node);
+
+       /* Get V4L2 format with the following method */
+       const struct v4l2_format *fmt = &node->vdev_fmt;
+
+       *num_planes = 1;
Quoted 8 lines

This doesn't handle VIDIOC_CREATE_BUFS, which triggers a .queue_setup() call with *num_planes > 0 and sizes[] already initialized. The driver needs to validate that the sizes and number of planes are valid in that case and return -EINVAL otherwise. Perhaps you should try running v4l2-compliance
(https://git.linuxtv.org/v4l-utils.git/tree/utils/v4l2-compliance) on this driver, which should catch issues like this.

Ok, we will add code check logic for this user case. The function will be similar to below:
https://elixir.bootlin.com/linux/latest/source/drivers/media/platform/vivid/vivid-vid-cap.c#L87
https://elixir.bootlin.com/linux/latest/source/drivers/staging/media/ipu3/ipu3-v4l2.c#L388

Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 372

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+       *num_buffers = clamp_val(*num_buffers, 1, VB2_MAX_FRAME);
+
+       if (isp_dev->ctx.queue[queue_id].desc.smem_alloc) {
+               buf_alloc_ctx = isp_dev->ctx.smem_vb2_alloc_ctx;
+               dev_dbg(dev, "Select smem_vb2_alloc_ctx(%llx)\n",
+                       (unsigned long long)buf_alloc_ctx);

Use %p for printing pointers.

For security reason, we will change to use "%pK" for printing kernel pointers.

Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 380

Quoted 8 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +8 -0
+       } else {
+               buf_alloc_ctx = isp_dev->ctx.default_vb2_alloc_ctx;
+               dev_dbg(dev, "Select default_vb2_alloc_ctx(%llx)\n",
+                       (unsigned long long)buf_alloc_ctx);
+       }
+
+       vq->dma_attrs |= DMA_ATTR_NON_CONSISTENT;
+       dev_dbg(dev, "queue(%d): cached mmap enabled\n", queue_id);
Quoted 6 lines

This isn't supported in upstream. (By the way, neither it is in Chrome OS 4.19 kernel. If we really need cached mmap for some reason, we should propose a proper solution upstream. I'd still first investigate why write-combine mmap is slow for operations that should be simple one-time writes or reads.)

Ok, we will remove this in upstream version and follow your suggestion to find out better solution.

Quoted 6 lines
Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+
+       if (vq->type == V4L2_BUF_TYPE_META_CAPTURE ||
+           vq->type == V4L2_BUF_TYPE_META_OUTPUT)
+               sizes[0] = fmt->fmt.meta.buffersize;
+       else
+               sizes[0] = fmt->fmt.pix_mp.plane_fmt[0].sizeimage;
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 388

Quoted 2 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +2 -0
+
+       alloc_devs[0] = (struct device *)buf_alloc_ctx;

Please don't add random type casts. If the compiler gives a type error, that normally means that there is something wrong elsewhere in the code (i.e. the type of buf_alloc_ctx variable is wrong here - it should be struct device *) and casting just masks the original problem.

Ok, we will fix this coding defect.

Quoted 12 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 395

Quoted 7 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +7 -0
+
+       dev_dbg(dev, "queue(%d):type(%d),size(%d),alloc_ctx(%llx)\n",
+               queue_id, vq->type, sizes[0],
+               (unsigned long long)buf_alloc_ctx);
+
+       /* Initialize buffer queue */
+       INIT_LIST_HEAD(&node->buffers);

This is incorrect. .queue_setup() can be also called on VIDIOC_CREATE_BUFS, which must preserve the other buffers in the queue.

In our new version, we have removed this for request-API new design.

Quoted 4 lines
Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +3 -0
+
+       return 0;
+}

[snip]

Quoted 10 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 439

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+static int mtk_cam_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
+{
+       struct mtk_cam_mem2mem2_device *m2m2 = vb2_get_drv_priv(vq);
+       struct mtk_cam_dev_video_device *node =
+               mtk_cam_vbq_to_isp_node(vq);
+       int r;

nit: "ret" is more common and already used by some other functions in this patch.

We will rename this variable in next patch and try to unify the variable naming in our driver.

Quoted 11 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 444

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+
+       if (m2m2->streaming) {
+               r = -EBUSY;
+               goto fail_return_bufs;
+       }

We shouldn't need to check this ourselves. It's not possible to have this call on an already streaming vb2 queue. Since we start streaming the m2m2 subdev only when all video nodes start streaming, this should never happen.

Ok, we will remove this redundant checking in next patch.

Quoted 59 lines
Quoted 59 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +59 -0
+
+       if (!node->enabled) {
+               pr_err("Node (%ld) is not enable\n", node - m2m2->nodes);
+               r = -EINVAL;
+               goto fail_return_bufs;
+       }
+
+       r = media_pipeline_start(&node->vdev.entity, &m2m2->pipeline);
+       if (r < 0) {
+               pr_err("Node (%ld) media_pipeline_start failed\n",
+                      node - m2m2->nodes);
+               goto fail_return_bufs;
+       }
+
+       if (!mtk_cam_all_nodes_streaming(m2m2, node))
+               return 0;
+
+       /* Start streaming of the whole pipeline now */
+
+       r = v4l2_subdev_call(&m2m2->subdev, video, s_stream, 1);
+       if (r < 0) {
+               pr_err("Node (%ld) v4l2_subdev_call s_stream failed\n",
+                      node - m2m2->nodes);
+               goto fail_stop_pipeline;
+       }
+       return 0;
+
+fail_stop_pipeline:
+       media_pipeline_stop(&node->vdev.entity);
+fail_return_bufs:
+       mtk_cam_return_all_buffers(m2m2, node, VB2_BUF_STATE_QUEUED);
+
+       return r;
+}
+
+static void mtk_cam_vb2_stop_streaming(struct vb2_queue *vq)
+{
+       struct mtk_cam_mem2mem2_device *m2m2 = vb2_get_drv_priv(vq);
+       struct mtk_cam_dev_video_device *node =
+               mtk_cam_vbq_to_isp_node(vq);
+       int r;
+
+       WARN_ON(!node->enabled);
+
+       /* Was this the first node with streaming disabled? */
+       if (mtk_cam_all_nodes_streaming(m2m2, node)) {
+               /* Yes, really stop streaming now */
+               r = v4l2_subdev_call(&m2m2->subdev, video, s_stream, 0);
+               if (r)
+                       dev_err(m2m2->dev, "failed to stop streaming\n");
+       }
+
+       mtk_cam_return_all_buffers(m2m2, node, VB2_BUF_STATE_ERROR);
+       media_pipeline_stop(&node->vdev.entity);
+}
+
+static int mtk_cam_videoc_querycap(struct file *file, void *fh,
+                                  struct v4l2_capability *cap)
+{
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 509

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+       struct mtk_cam_mem2mem2_device *m2m2 = video_drvdata(file);
+       struct mtk_cam_dev_video_device *node = file_to_mtk_cam_node(file);
+       struct mtk_cam_dev *isp_dev = mtk_cam_m2m_to_dev(m2m2);
+       int queue_id =
+               mtk_cam_dev_get_queue_id_of_dev_node(isp_dev, node);
+       struct mtk_cam_ctx_queue *node_ctx = &isp_dev->ctx.queue[queue_id];

It feels like this could be just stored as node->ctx.

After refactoring this function, the node_ctx is removed. Moreover, we will follow your suggestion to store mtk_cam_ctx_queue pointer in mtk_cam_dev_video_device for better access.

Quoted 4 lines
Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +4 -0
+
+       strlcpy(cap->driver, m2m2->name, sizeof(cap->driver));
+       strlcpy(cap->card, m2m2->model, sizeof(cap->card));
+       snprintf(cap->bus_info, sizeof(cap->bus_info), "platform:%s",
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 518

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+                node->name);
+
+       cap->device_caps =
+               mtk_cam_node_get_v4l2_cap(node_ctx) | V4L2_CAP_STREAMING;
+       cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;

No need to set these 2 fields manually. They are filled in automatically from struct video_device::device_caps.

We will remove this redundant code.

Quoted 10 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 523

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+
+       return 0;
+}
+
+/* Propagate forward always the format from the mtk_cam_dev subdev */

It doesn't seem to match what the function is doing, i.e. returning the format structure of the node itself. I'd just drop this comment. The code should be written in a self-explaining way anyway.

Ok, we will remove this comment to avoid misunderstanding.

Quoted 20 lines
Quoted 20 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +20 -0
+static int mtk_cam_videoc_g_fmt(struct file *file, void *fh,
+                               struct v4l2_format *f)
+{
+       struct mtk_cam_dev_video_device *node = file_to_mtk_cam_node(file);
+
+       f->fmt = node->vdev_fmt.fmt;
+
+       return 0;
+}
+
+static int mtk_cam_videoc_try_fmt(struct file *file,
+                                 void *fh,
+        struct v4l2_format *f)
+{
+       struct mtk_cam_mem2mem2_device *m2m2 = video_drvdata(file);
+       struct mtk_cam_dev *isp_dev = mtk_cam_m2m_to_dev(m2m2);
+       struct mtk_cam_ctx *dev_ctx = &isp_dev->ctx;
+       struct mtk_cam_dev_video_device *node = file_to_mtk_cam_node(file);
+       int queue_id =
+               mtk_cam_dev_get_queue_id_of_dev_node(isp_dev, node);
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 548

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+       int ret = 0;
+
+       ret = mtk_cam_ctx_fmt_set_img(dev_ctx, queue_id,
+                                     &f->fmt.pix_mp,
+               &node->vdev_fmt.fmt.pix_mp);

Doesn't this actually change the current node format? VIDIOC_TRY_FMT must not have any side effects on current driver state.

This is a bug in our implementation. We will fix it in next patch.

Quoted 8 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 553

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+
+       /* Simply set the format to the node context in the initial version */
+       if (ret) {
+               pr_warn("Fmt(%d) not support for queue(%d), will load default fmt\n",
+                       f->fmt.pix_mp.pixelformat, queue_id);

No need for this warning.

Ok, we will remove this in next patch.

Quoted 8 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 556

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +3 -0
+
+               ret =   mtk_cam_ctx_format_load_default_fmt
+                       (&dev_ctx->queue[queue_id], f);

Wouldn't this also change the current node state?

Also, something wrong with the number of spaces after "ret =".

Ditto.

Quoted 8 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 561

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+       }
+
+       if (!ret) {
+               node->vdev_fmt.fmt.pix_mp = f->fmt.pix_mp;
+               dev_ctx->queue[queue_id].fmt.pix_mp = node->vdev_fmt.fmt.pix_mp;

Ditto.

Quoted 8 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 904

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +3 -0
+       }
+
+       return ret;

VIDIOC_TRY_FMT must not return an error unless for the cases described
in https://www.kernel.org/doc/html/latest/media/uapi/v4l/vidioc-g-fmt.html#return-value
.

Ok, we will try to follow the VIDIOC_TYPE_FMT API description of V4L2 manual.

Quoted 16 lines
Quoted 16 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +16 -0
+}
+
+static int mtk_cam_videoc_s_fmt(struct file *file, void *fh,
+                               struct v4l2_format *f)
+{
+       struct mtk_cam_mem2mem2_device *m2m2 = video_drvdata(file);
+       struct mtk_cam_dev *isp_dev = mtk_cam_m2m_to_dev(m2m2);
+       struct mtk_cam_ctx *dev_ctx = &isp_dev->ctx;
+       struct mtk_cam_dev_video_device *node = file_to_mtk_cam_node(file);
+       int queue_id = mtk_cam_dev_get_queue_id_of_dev_node(isp_dev, node);
+       int ret = 0;
+
+       ret = mtk_cam_ctx_fmt_set_img(dev_ctx, queue_id,
+                                     &f->fmt.pix_mp,
+               &node->vdev_fmt.fmt.pix_mp);
+
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 586

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+       /* Simply set the format to the node context in the initial version */
+       if (!ret)
+               dev_ctx->queue[queue_id].fmt.pix_mp = node->vdev_fmt.fmt.pix_mp;
+       else
+               dev_warn(&isp_dev->pdev->dev,
+                        "s_fmt, format not support\n");

No need for error messages for userspace errors.

Ok, we will remove this check.

Quoted 12 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+       return ret;

Instead of opencoding most of this function, one would normally call mtk_cam_videoc_try_fmt() first to adjust the format struct and then just update the driver state with the adjusted format.

Also, similarly to VIDIOC_TRY_FMT, VIDIOC_SET_FMT doesn't fail unless in the very specific cases, as described in
https://www.kernel.org/doc/html/latest/media/uapi/v4l/vidioc-g-fmt.html#return-value
.

Ok, below is our revised version of this function.

int mtk_cam_videoc_s_fmt(struct file *file, void *fh,
struct v4l2_format *f)

{
	struct mtk_cam_mem2mem2_device *m2m2 = video_drvdata(file);
	struct mtk_cam_dev *cam_dev = mtk_cam_m2m_to_dev(m2m2);
	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);

	/* Get the valid format*/
	mtk_cam_videoc_try_fmt(file, fh, f);
	/* Configure to video device */
	mtk_cam_ctx_fmt_set_img(&cam_dev->pdev->dev,
				&node->vdev_fmt.fmt.pix_mp,
				&f->fmt.pix_mp,
				node->queue_id);

	return 0;
}
Quoted 36 lines
Quoted 36 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +36 -0
+}
+
+static int mtk_cam_enum_framesizes(struct file *filp, void *priv,
+                                  struct v4l2_frmsizeenum *sizes)
+{
+       struct mtk_cam_mem2mem2_device *m2m2 = video_drvdata(filp);
+       struct mtk_cam_dev *isp_dev = mtk_cam_m2m_to_dev(m2m2);
+       struct mtk_cam_dev_video_device *node =
+               file_to_mtk_cam_node(filp);
+       int queue_id = mtk_cam_dev_get_queue_id_of_dev_node(isp_dev, node);
+
+       if (sizes->index != 0)
+               return -EINVAL;
+
+       if (queue_id == MTK_CAM_CTX_P1_MAIN_STREAM_OUT) {
+               sizes->type = V4L2_FRMSIZE_TYPE_CONTINUOUS;
+               sizes->stepwise.max_width = CAM_B_MAX_WIDTH;
+               sizes->stepwise.min_width = CAM_MIN_WIDTH;
+               sizes->stepwise.max_height = CAM_B_MAX_HEIGHT;
+               sizes->stepwise.min_height = CAM_MIN_HEIGHT;
+               sizes->stepwise.step_height = 1;
+               sizes->stepwise.step_width = 1;
+       } else if (queue_id == MTK_CAM_CTX_P1_PACKED_BIN_OUT) {
+               sizes->type = V4L2_FRMSIZE_TYPE_CONTINUOUS;
+               sizes->stepwise.max_width = RRZ_MAX_WIDTH;
+               sizes->stepwise.min_width = RRZ_MIN_WIDTH;
+               sizes->stepwise.max_height = RRZ_MAX_HEIGHT;
+               sizes->stepwise.min_height = RRZ_MIN_HEIGHT;
+               sizes->stepwise.step_height = 1;
+               sizes->stepwise.step_width = 1;
+       }
+
+       return 0;
+}
+
+static int mtk_cam_meta_enum_format(struct file *file,
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 630

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+                                   void *fh, struct v4l2_fmtdesc *f)
+{
+       struct mtk_cam_dev_video_device *node = file_to_mtk_cam_node(file);
+
+       if (f->index > 0 || f->type != node->vbq.type)
+               return -EINVAL;

f->type is already checked by the V4L2 core. See v4l_enum_fmt().

We will remove the redundant checking in next patch.

Quoted 9 lines
Quoted 9 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +9 -0
+
+       f->pixelformat = node->vdev_fmt.fmt.meta.dataformat;
+
+       return 0;
+}
+
+static int mtk_cam_videoc_s_meta_fmt(struct file *file,
+                                    void *fh, struct v4l2_format *f)
+{
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 576

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+       struct mtk_cam_mem2mem2_device *m2m2 = video_drvdata(file);
+       struct mtk_cam_dev *isp_dev = mtk_cam_m2m_to_dev(m2m2);
+       struct mtk_cam_ctx *dev_ctx = &isp_dev->ctx;
+       struct mtk_cam_dev_video_device *node = file_to_mtk_cam_node(file);
+       int queue_id = mtk_cam_dev_get_queue_id_of_dev_node(isp_dev, node);
+

No need for this blank line.

We will fix this coding style in next patch.

Quoted 4 lines
Quoted 1 line

+ int ret = 0;

Please don't default-initialize without a good reason.

Ok, fix in next patch.

Quoted 6 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 649

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +3 -0
+
+       if (f->type != node->vbq.type)
+               return -EINVAL;

Ditto.

Ok, fix in next patch.

Quoted 6 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 652

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +3 -0
+
+       ret = mtk_cam_ctx_format_load_default_fmt(&dev_ctx->queue[queue_id], f);
+

No need for this blank line.

Ok, fix in next patch.

Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 658

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+       if (!ret) {
+               node->vdev_fmt.fmt.meta = f->fmt.meta;
+               dev_ctx->queue[queue_id].fmt.meta = node->vdev_fmt.fmt.meta;
+       } else {
+               dev_warn(&isp_dev->pdev->dev,
+                        "s_meta_fm failed, format not support\n");

No need for this warning.

Ok, fix in next patch.

Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 905

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +4 -0
+       }
+
+       return ret;
+}

Actually, why do we even need to do all the things? Do we support multiple different meta formats on the same video node? If not, we can just have all the TRY_FMT/S_FMT/G_FMT return the hardcoded format.

Ok, it is a good idea. We will return the hardcode format for meta video devices.
Below is the revision version.

int mtk_cam_meta_enum_format(struct file *file,
void *fh, struct v4l2_fmtdesc *f)

{
	struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);

	f->pixelformat = node->vdev_fmt.fmt.meta.dataformat;

	return 0;
}

const struct v4l2_ioctl_ops mtk_cam_v4l2_meta_cap_ioctl_ops = {
.vidioc_querycap = mtk_cam_videoc_querycap, .vidioc_enum_fmt_meta_cap = mtk_cam_meta_enum_format, .vidioc_g_fmt_meta_cap = mtk_cam_videoc_g_meta_fmt, .vidioc_s_fmt_meta_cap = mtk_cam_videoc_g_meta_fmt, .vidioc_try_fmt_meta_cap = mtk_cam_videoc_g_meta_fmt,
...

Quoted 11 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 670

Quoted 8 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +8 -0
+
+static int mtk_cam_videoc_g_meta_fmt(struct file *file,
+                                    void *fh, struct v4l2_format *f)
+{
+       struct mtk_cam_dev_video_device *node = file_to_mtk_cam_node(file);
+
+       if (f->type != node->vbq.type)
+               return -EINVAL;

Not needed.

Ok, remove it in next patch.

Quoted 11 lines
Quoted 11 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +11 -0
+
+       f->fmt = node->vdev_fmt.fmt;
+
+       return 0;
+}
+
+int mtk_cam_videoc_qbuf(struct file *file, void *priv, struct v4l2_buffer *p)
+{
+       struct video_device *vdev = video_devdata(file);
+       struct vb2_buffer *vb;
+       struct mtk_cam_dev_buffer *db;
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 687

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+       int r = 0;
+
+       /* check if vb2 queue is busy */
+       if (vdev->queue->owner &&
+           vdev->queue->owner != file->private_data)
+               return -EBUSY;

This should be already handled by the core.

Ok, remove it in next patch.

Quoted 10 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 691

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +4 -0
+
+       /* Keep the value of sequence in v4l2_buffer */
+       /* in ctx buf since vb2_qbuf will set it to 0 */
+       vb = vdev->queue->bufs[p->index];

Why do you need a sequence number for buffers on queue time? The field is not specified to be set by the userspace and should be ignored by the driver. The driver should rely on the Request API to match any buffers together anyway.

It is our old design for frame-sync mechanism. However, we change to use "Request API" design and this function is removed in new version.

Quoted 5 lines
Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+
+       if (vb) {
+               db = mtk_cam_vb2_buf_to_dev_buf(vb);
+               pr_debug("qbuf: p:%llx,vb:%llx, db:%llx\n",
+                        (unsigned long long)p,
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 701

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+                       (unsigned long long)vb,
+                       (unsigned long long)db);
+               db->ctx_buf.user_sequence = p->sequence;
+       }
+

Generally this driver shouldn't need to implement this callback itself. Instead it can just use the vb2_ioctl_qbuf() helper.

Same as above. This function is removed in new version.

Quoted 113 lines
Quoted 113 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +113 -0
+       r = vb2_qbuf(vdev->queue, vdev->v4l2_dev->mdev, p);
+
+       if (r)
+               pr_err("vb2_qbuf failed(err=%d): buf idx(%d)\n",
+                      r, p->index);
+
+       return r;
+}
+EXPORT_SYMBOL_GPL(mtk_cam_videoc_qbuf);
+
+/******************** function pointers ********************/
+
+/* subdev internal operations */
+static const struct v4l2_subdev_internal_ops mtk_cam_subdev_internal_ops = {
+       .open = mtk_cam_subdev_open,
+       .close = mtk_cam_subdev_close,
+};
+
+static const struct v4l2_subdev_core_ops mtk_cam_subdev_core_ops = {
+       .subscribe_event = mtk_cam_subdev_subscribe_event,
+       .unsubscribe_event = mtk_cam_subdev_unsubscribe_event,
+};
+
+static const struct v4l2_subdev_video_ops mtk_cam_subdev_video_ops = {
+       .s_stream = mtk_cam_subdev_s_stream,
+};
+
+static const struct v4l2_subdev_ops mtk_cam_subdev_ops = {
+       .core = &mtk_cam_subdev_core_ops,
+       .video = &mtk_cam_subdev_video_ops,
+};
+
+static const struct media_entity_operations mtk_cam_media_ops = {
+       .link_setup = mtk_cam_link_setup,
+       .link_validate = v4l2_subdev_link_validate,
+};
+
+#ifdef CONFIG_MEDIATEK_MEDIA_REQUEST
+static void mtk_cam_vb2_buf_request_complete(struct vb2_buffer *vb)
+{
+       struct mtk_cam_mem2mem2_device *m2m2 = vb2_get_drv_priv(vb->vb2_queue);
+
+       v4l2_ctrl_request_complete(vb->req_obj.req,
+                                  m2m2->v4l2_dev->ctrl_handler);
+}
+#endif
+
+static const struct vb2_ops mtk_cam_vb2_ops = {
+       .buf_queue = mtk_cam_vb2_buf_queue,
+       .queue_setup = mtk_cam_vb2_queue_setup,
+       .start_streaming = mtk_cam_vb2_start_streaming,
+       .stop_streaming = mtk_cam_vb2_stop_streaming,
+       .wait_prepare = vb2_ops_wait_prepare,
+       .wait_finish = vb2_ops_wait_finish,
+#ifdef CONFIG_MEDIATEK_MEDIA_REQUEST
+       .buf_request_complete = mtk_cam_vb2_buf_request_complete,
+#endif
+};
+
+static const struct v4l2_file_operations mtk_cam_v4l2_fops = {
+       .unlocked_ioctl = video_ioctl2,
+       .open = v4l2_fh_open,
+       .release = vb2_fop_release,
+       .poll = vb2_fop_poll,
+       .mmap = vb2_fop_mmap,
+#ifdef CONFIG_COMPAT
+       .compat_ioctl32 = v4l2_compat_ioctl32,
+#endif
+};
+
+static const struct v4l2_ioctl_ops mtk_cam_v4l2_ioctl_ops = {
+       .vidioc_querycap = mtk_cam_videoc_querycap,
+       .vidioc_enum_framesizes = mtk_cam_enum_framesizes,
+
+       .vidioc_g_fmt_vid_cap_mplane = mtk_cam_videoc_g_fmt,
+       .vidioc_s_fmt_vid_cap_mplane = mtk_cam_videoc_s_fmt,
+       .vidioc_try_fmt_vid_cap_mplane = mtk_cam_videoc_try_fmt,
+
+       .vidioc_g_fmt_vid_out_mplane = mtk_cam_videoc_g_fmt,
+       .vidioc_s_fmt_vid_out_mplane = mtk_cam_videoc_s_fmt,
+       .vidioc_try_fmt_vid_out_mplane = mtk_cam_videoc_try_fmt,
+
+       /* buffer queue management */
+       .vidioc_reqbufs = vb2_ioctl_reqbufs,
+       .vidioc_create_bufs = vb2_ioctl_create_bufs,
+       .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+       .vidioc_querybuf = vb2_ioctl_querybuf,
+       .vidioc_qbuf = mtk_cam_videoc_qbuf,
+       .vidioc_dqbuf = vb2_ioctl_dqbuf,
+       .vidioc_streamon = vb2_ioctl_streamon,
+       .vidioc_streamoff = vb2_ioctl_streamoff,
+       .vidioc_expbuf = vb2_ioctl_expbuf,
+};
+
+static const struct v4l2_ioctl_ops mtk_cam_v4l2_meta_ioctl_ops = {
+       .vidioc_querycap = mtk_cam_videoc_querycap,
+
+       .vidioc_enum_fmt_meta_cap = mtk_cam_meta_enum_format,
+       .vidioc_g_fmt_meta_cap = mtk_cam_videoc_g_meta_fmt,
+       .vidioc_s_fmt_meta_cap = mtk_cam_videoc_s_meta_fmt,
+       .vidioc_try_fmt_meta_cap = mtk_cam_videoc_g_meta_fmt,
+
+       .vidioc_enum_fmt_meta_out = mtk_cam_meta_enum_format,
+       .vidioc_g_fmt_meta_out = mtk_cam_videoc_g_meta_fmt,
+       .vidioc_s_fmt_meta_out = mtk_cam_videoc_s_meta_fmt,
+       .vidioc_try_fmt_meta_out = mtk_cam_videoc_g_meta_fmt,
+
+       .vidioc_reqbufs = vb2_ioctl_reqbufs,
+       .vidioc_create_bufs = vb2_ioctl_create_bufs,
+       .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+       .vidioc_querybuf = vb2_ioctl_querybuf,
+       .vidioc_qbuf = mtk_cam_videoc_qbuf,
+       .vidioc_dqbuf = vb2_ioctl_dqbuf,
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 794

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +4 -0
+       .vidioc_streamon = vb2_ioctl_streamon,
+       .vidioc_streamoff = vb2_ioctl_streamoff,
+       .vidioc_expbuf = vb2_ioctl_expbuf,
+};

The ops should be split for each node type, {VIDEO, META} x {OUTPUT, CAPTURE}. Then the core would validate the type given to all the ioctls automatically.

Ok, below is new implementation.

static const struct v4l2_ioctl_ops mtk_cam_v4l2_vcap_ioctl_ops static const struct v4l2_ioctl_ops mtk_cam_v4l2_vout_ioctl_ops static const struct v4l2_ioctl_ops mtk_cam_v4l2_meta_cap_ioctl_ops static const struct v4l2_ioctl_ops mtk_cam_v4l2_meta_out_ioctl_ops

Quoted 12 lines
Quoted 12 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +12 -0
+
+static u32 mtk_cam_node_get_v4l2_cap(struct mtk_cam_ctx_queue *node_ctx)
+{
+       u32 cap = 0;
+
+       if (node_ctx->desc.capture)
+               if (node_ctx->desc.image)
+                       cap = V4L2_CAP_VIDEO_CAPTURE_MPLANE;
+               else
+                       cap = V4L2_CAP_META_CAPTURE;
+       else
+               if (node_ctx->desc.image)
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 836

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+                       cap = V4L2_CAP_VIDEO_OUTPUT_MPLANE;
+               else
+                       cap = V4L2_CAP_META_OUTPUT;
+
+       return cap;
+}

Why not just have this defined statically as node_ctx->desc.cap?

Ok, it is refactoring done.

Quoted 12 lines
Quoted 12 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +12 -0
+
+static u32 mtk_cam_node_get_format_type(struct mtk_cam_ctx_queue *node_ctx)
+{
+       u32 type;
+
+       if (node_ctx->desc.capture)
+               if (node_ctx->desc.image)
+                       type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
+               else
+                       type = V4L2_BUF_TYPE_META_CAPTURE;
+       else
+               if (node_ctx->desc.image)
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 854

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+                       type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
+               else
+                       type = V4L2_BUF_TYPE_META_OUTPUT;
+
+       return type;
+}

Why not just have this defined statically as node_ctx->desc.buf_type?

Same as above.

Quoted 7 lines
Quoted 7 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +7 -0
+
+static const struct v4l2_ioctl_ops *mtk_cam_node_get_ioctl_ops
+       (struct mtk_cam_ctx_queue *node_ctx)
+{
+       const struct v4l2_ioctl_ops *ops = NULL;
+
+       if (node_ctx->desc.image)
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 866

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+               ops = &mtk_cam_v4l2_ioctl_ops;
+       else
+               ops = &mtk_cam_v4l2_meta_ioctl_ops;
+       return ops;
+}

It's also preferable to just put this inside some structure rather than have getter functions. (node_ctx->desc.ioctl_ops?)

Same as above.
Below is the new version for struct mtk_cam_ctx_queue_desc

/*
* struct mtk_cam_ctx_queue_desc - queue attributes
* setup by device context owner
* @id: id of the context queue
* @name: media entity name
* @cap: mapped to V4L2 capabilities
* @buf_type: mapped to V4L2 buffer type
* @capture: true for capture queue (device to user)
* false for output queue (from user to device)
* @image: true for image, false for meta data
* @smem_alloc: Using the cam_smem_drv as alloc ctx or not
* @dma_port: the dma port associated to the buffer
* @fmts: supported format
* @num_fmts: the number of supported formats
* @default_fmt_idx: default format of this queue
* @max_buf_count: maximum V4L2 buffer count
* @max_buf_count: mapped to v4l2_ioctl_ops */

struct mtk_cam_ctx_queue_desc {
	u8 id;
	char *name;
	u32 cap;
	u32 buf_type;
	u32 dma_port;
	u32 smem_alloc:1;
	u8 capture:1;
	u8 image:1;
	u8 num_fmts;
	u8 default_fmt_idx;
	u8 max_buf_count;
	const struct v4l2_ioctl_ops *ioctl_ops;
	struct v4l2_format *fmts;
};
Quoted 7 lines
Quoted 7 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +7 -0
+
+/* Config node's video properties */
+/* according to the device context requirement */
+static void mtk_cam_node_to_v4l2(struct mtk_cam_dev *isp_dev, u32 node,
+                                struct video_device *vdev,
+                                struct v4l2_format *f)
+{
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 879

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+       u32 cap;
+       struct mtk_cam_ctx *device_ctx = &isp_dev->ctx;
+       struct mtk_cam_ctx_queue *node_ctx = &device_ctx->queue[node];
+
+       WARN_ON(node >= mtk_cam_dev_get_total_node(isp_dev));
+       WARN_ON(!node_ctx);

How is this possible?

Ok, we will remove this in next version.

Quoted 5 lines
Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+
+       /* set cap of the node */
+       cap = mtk_cam_node_get_v4l2_cap(node_ctx);
+       f->type = mtk_cam_node_get_format_type(node_ctx);
+       vdev->ioctl_ops = mtk_cam_node_get_ioctl_ops(node_ctx);
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 889

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+
+       if (mtk_cam_ctx_format_load_default_fmt(&device_ctx->queue[node], f)) {
+               dev_err(&isp_dev->pdev->dev,
+                       "Can't load default for node (%d): (%s)",
+               node, device_ctx->queue[node].desc.name);

mtk_cam_ctx_format_load_default_fmt() doesn't fail (and that's right). It should be actually made void.

Ok, we will revise this in next version.

Quoted 21 lines
Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+       }       else {
+               if (device_ctx->queue[node].desc.image) {
+                       dev_dbg(&isp_dev->pdev->dev,
+                               "Node (%d): (%s), dfmt (f:0x%x w:%d: h:%d s:%d)\n",
+                       node, device_ctx->queue[node].desc.name,
+                       f->fmt.pix_mp.pixelformat,
+                       f->fmt.pix_mp.width,
+                       f->fmt.pix_mp.height,
+                       f->fmt.pix_mp.plane_fmt[0].sizeimage
+                       );

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 909

Quoted 10 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +10 -0
+                       node_ctx->fmt.pix_mp = f->fmt.pix_mp;
+               } else {
+                       dev_dbg(&isp_dev->pdev->dev,
+                               "Node (%d): (%s), dfmt (f:0x%x s:%u)\n",
+                       node, device_ctx->queue[node].desc.name,
+                       f->fmt.meta.dataformat,
+                       f->fmt.meta.buffersize
+                       );
+                       node_ctx->fmt.meta = f->fmt.meta;
+               }
Quoted 4 lines

Drop the debugging messages and just replace the whole if/else block with:

node_ctx->fmt = f->fmt;

Same as above.

Quoted 5 lines

Sorry, ran out of time for today. Fourth part will come. :)

Best regards,
Tomasz

Appreciate your support and hard working on this review. We will look forward your part 4 review.

Best regards,

Jungo

Quoted 4 lines

_______________________________________________
Linux-mediatek mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-mediatek

#22 lore raw

Re: [RFC PATCH V0 7/7] [media] platform: mtk-isp: Add Mediatek ISP Pass 1 driver

Quoted 2 lines · Jungo Lin

On Tue, 2019-03-12 at 19:04 +0900, Tomasz Figa wrote:

[snip]

Quoted 14 lines
Quoted 9 lines

Instead of opencoding most of this function, one would normally call mtk_cam_videoc_try_fmt() first to adjust the format struct and then just update the driver state with the adjusted format.

Also, similarly to VIDIOC_TRY_FMT, VIDIOC_SET_FMT doesn't fail unless in the very specific cases, as described in
https://www.kernel.org/doc/html/latest/media/uapi/v4l/vidioc-g-fmt.html#return-value
.

Ok, below is our revised version of this function.

int mtk_cam_videoc_s_fmt(struct file *file, void *fh,
struct v4l2_format *f)

Quoted 16 lines
{
        struct mtk_cam_mem2mem2_device *m2m2 = video_drvdata(file);
        struct mtk_cam_dev *cam_dev = mtk_cam_m2m_to_dev(m2m2);
        struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);

        /* Get the valid format*/
        mtk_cam_videoc_try_fmt(file, fh, f);
        /* Configure to video device */
        mtk_cam_ctx_fmt_set_img(&cam_dev->pdev->dev,
                                &node->vdev_fmt.fmt.pix_mp,
                                &f->fmt.pix_mp,
                                node->queue_id);

        return 0;
}

Looks almost good. We still need to signal the -EBUSY error condition if an attempt to change the format is made while streaming is active.

[snip]

Quoted 68 lines
Quoted 12 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 645

Quoted 9 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +9 -0
+static int mtk_cam_videoc_s_meta_fmt(struct file *file,
+                                    void *fh, struct v4l2_format *f)
+{
+       struct mtk_cam_mem2mem2_device *m2m2 = video_drvdata(file);
+       struct mtk_cam_dev *isp_dev = mtk_cam_m2m_to_dev(m2m2);
+       struct mtk_cam_ctx *dev_ctx = &isp_dev->ctx;
+       struct mtk_cam_dev_video_device *node = file_to_mtk_cam_node(file);
+       int queue_id = mtk_cam_dev_get_queue_id_of_dev_node(isp_dev, node);
+

No need for this blank line.

We will fix this coding style in next patch.

Quoted 4 lines
Quoted 1 line

+ int ret = 0;

Please don't default-initialize without a good reason.

Ok, fix in next patch.

Quoted 6 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 649

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +3 -0
+
+       if (f->type != node->vbq.type)
+               return -EINVAL;

Ditto.

Ok, fix in next patch.

Quoted 6 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 652

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +3 -0
+
+       ret = mtk_cam_ctx_format_load_default_fmt(&dev_ctx->queue[queue_id], f);
+

No need for this blank line.

Ok, fix in next patch.

Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 658

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+       if (!ret) {
+               node->vdev_fmt.fmt.meta = f->fmt.meta;
+               dev_ctx->queue[queue_id].fmt.meta = node->vdev_fmt.fmt.meta;
+       } else {
+               dev_warn(&isp_dev->pdev->dev,
+                        "s_meta_fm failed, format not support\n");

No need for this warning.

Ok, fix in next patch.

Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 905

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +4 -0
+       }
+
+       return ret;
+}

Actually, why do we even need to do all the things? Do we support multiple different meta formats on the same video node? If not, we can just have all the TRY_FMT/S_FMT/G_FMT return the hardcoded format.

Ok, it is a good idea. We will return the hardcode format for meta video devices.
Below is the revision version.

int mtk_cam_meta_enum_format(struct file *file,
void *fh, struct v4l2_fmtdesc *f)

Quoted 7 lines
{
        struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);

        f->pixelformat = node->vdev_fmt.fmt.meta.dataformat;

        return 0;
}

Need to error out if f->index > 0. Also need to initialize the other output fields - flags and description.

[snip]

Quoted 92 lines
Quoted 11 lines
Quoted 11 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +11 -0
+static u32 mtk_cam_node_get_v4l2_cap(struct mtk_cam_ctx_queue *node_ctx)
+{
+       u32 cap = 0;
+
+       if (node_ctx->desc.capture)
+               if (node_ctx->desc.image)
+                       cap = V4L2_CAP_VIDEO_CAPTURE_MPLANE;
+               else
+                       cap = V4L2_CAP_META_CAPTURE;
+       else
+               if (node_ctx->desc.image)
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 836

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+                       cap = V4L2_CAP_VIDEO_OUTPUT_MPLANE;
+               else
+                       cap = V4L2_CAP_META_OUTPUT;
+
+       return cap;
+}

Why not just have this defined statically as node_ctx->desc.cap?

Ok, it is refactoring done.

Quoted 12 lines
Quoted 12 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +12 -0
+
+static u32 mtk_cam_node_get_format_type(struct mtk_cam_ctx_queue *node_ctx)
+{
+       u32 type;
+
+       if (node_ctx->desc.capture)
+               if (node_ctx->desc.image)
+                       type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
+               else
+                       type = V4L2_BUF_TYPE_META_CAPTURE;
+       else
+               if (node_ctx->desc.image)
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 854

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+                       type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
+               else
+                       type = V4L2_BUF_TYPE_META_OUTPUT;
+
+       return type;
+}

Why not just have this defined statically as node_ctx->desc.buf_type?

Same as above.

Quoted 7 lines
Quoted 7 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +7 -0
+
+static const struct v4l2_ioctl_ops *mtk_cam_node_get_ioctl_ops
+       (struct mtk_cam_ctx_queue *node_ctx)
+{
+       const struct v4l2_ioctl_ops *ops = NULL;
+
+       if (node_ctx->desc.image)
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 866

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+               ops = &mtk_cam_v4l2_ioctl_ops;
+       else
+               ops = &mtk_cam_v4l2_meta_ioctl_ops;
+       return ops;
+}

It's also preferable to just put this inside some structure rather than have getter functions. (node_ctx->desc.ioctl_ops?)

Same as above.
Below is the new version for struct mtk_cam_ctx_queue_desc

/*
* struct mtk_cam_ctx_queue_desc - queue attributes
* setup by device context owner
* @id: id of the context queue
* @name: media entity name
* @cap: mapped to V4L2 capabilities
* @buf_type: mapped to V4L2 buffer type
* @capture: true for capture queue (device to user)
* false for output queue (from user to device)
* @image: true for image, false for meta data
* @smem_alloc: Using the cam_smem_drv as alloc ctx or not

 * @dma_port:   the dma port associated to the buffer
 * @fmts:       supported format
 * @num_fmts:   the number of supported formats
 * @default_fmt_idx: default format of this queue
 * @max_buf_count: maximum V4L2 buffer count
 * @max_buf_count: mapped to v4l2_ioctl_ops
 */
struct mtk_cam_ctx_queue_desc {
        u8 id;
        char *name;
        u32 cap;
        u32 buf_type;
        u32 dma_port;
        u32 smem_alloc:1;
Quoted 8 lines
        u8 capture:1;
        u8 image:1;
        u8 num_fmts;
        u8 default_fmt_idx;
        u8 max_buf_count;
        const struct v4l2_ioctl_ops *ioctl_ops;
        struct v4l2_format *fmts;
};

SGTM +/- the missing kerneldoc for the new fields.

[snip]

Quoted 8 lines
Quoted 5 lines

Sorry, ran out of time for today. Fourth part will come. :)

Best regards,
Tomasz

Appreciate your support and hard working on this review. We will look forward your part 4 review.

Thanks for replying to all the comments, it's very helpful.

As I mentioned in another reply, I'm going to be busy for the next few days, so I'd suggest addressing the existing comments, fixing any v4l2-compliance issues and also checking if any changes could be applied to the other drivers (DIP, FD, Seninf, MDP) too and then sending RFC V1.

Best regards,
Tomasz

#23 lore raw

Re: [RFC PATCH V0 7/7] [media] platform: mtk-isp: Add Mediatek ISP Pass 1 driver

Quoted 31 lines · Tomasz Figa
Quoted 2 lines · Jungo Lin

On Tue, 2019-03-12 at 19:04 +0900, Tomasz Figa wrote:

[snip]

Quoted 27 lines
Quoted 9 lines

Instead of opencoding most of this function, one would normally call mtk_cam_videoc_try_fmt() first to adjust the format struct and then just update the driver state with the adjusted format.

Also, similarly to VIDIOC_TRY_FMT, VIDIOC_SET_FMT doesn't fail unless in the very specific cases, as described in
https://www.kernel.org/doc/html/latest/media/uapi/v4l/vidioc-g-fmt.html#return-value
.

Ok, below is our revised version of this function.

int mtk_cam_videoc_s_fmt(struct file *file, void *fh,
struct v4l2_format *f)

{
        struct mtk_cam_mem2mem2_device *m2m2 = video_drvdata(file);
        struct mtk_cam_dev *cam_dev = mtk_cam_m2m_to_dev(m2m2);
        struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);

        /* Get the valid format*/
        mtk_cam_videoc_try_fmt(file, fh, f);
        /* Configure to video device */
        mtk_cam_ctx_fmt_set_img(&cam_dev->pdev->dev,
                                &node->vdev_fmt.fmt.pix_mp,
                                &f->fmt.pix_mp,
                                node->queue_id);
Quoted 8 lines
Quoted 3 lines
        return 0;
}

Looks almost good. We still need to signal the -EBUSY error condition if an attempt to change the format is made while streaming is active.

[snip]

Ok, we will add streaming status checking in this function.

Quoted 73 lines
Quoted 73 lines
Quoted 12 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 645

Quoted 9 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +9 -0
+static int mtk_cam_videoc_s_meta_fmt(struct file *file,
+                                    void *fh, struct v4l2_format *f)
+{
+       struct mtk_cam_mem2mem2_device *m2m2 = video_drvdata(file);
+       struct mtk_cam_dev *isp_dev = mtk_cam_m2m_to_dev(m2m2);
+       struct mtk_cam_ctx *dev_ctx = &isp_dev->ctx;
+       struct mtk_cam_dev_video_device *node = file_to_mtk_cam_node(file);
+       int queue_id = mtk_cam_dev_get_queue_id_of_dev_node(isp_dev, node);
+

No need for this blank line.

We will fix this coding style in next patch.

Quoted 4 lines
Quoted 1 line

+ int ret = 0;

Please don't default-initialize without a good reason.

Ok, fix in next patch.

Quoted 6 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 649

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +3 -0
+
+       if (f->type != node->vbq.type)
+               return -EINVAL;

Ditto.

Ok, fix in next patch.

Quoted 6 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 652

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +3 -0
+
+       ret = mtk_cam_ctx_format_load_default_fmt(&dev_ctx->queue[queue_id], f);
+

No need for this blank line.

Ok, fix in next patch.

Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 658

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+       if (!ret) {
+               node->vdev_fmt.fmt.meta = f->fmt.meta;
+               dev_ctx->queue[queue_id].fmt.meta = node->vdev_fmt.fmt.meta;
+       } else {
+               dev_warn(&isp_dev->pdev->dev,
+                        "s_meta_fm failed, format not support\n");

No need for this warning.

Ok, fix in next patch.

Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 905

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +4 -0
+       }
+
+       return ret;
+}

Actually, why do we even need to do all the things? Do we support multiple different meta formats on the same video node? If not, we can just have all the TRY_FMT/S_FMT/G_FMT return the hardcoded format.

Ok, it is a good idea. We will return the hardcode format for meta video devices.
Below is the revision version.

int mtk_cam_meta_enum_format(struct file *file,
void *fh, struct v4l2_fmtdesc *f)

{
        struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);

        f->pixelformat = node->vdev_fmt.fmt.meta.dataformat;
Quoted 7 lines
Quoted 2 lines
        return 0;
}

Need to error out if f->index > 0. Also need to initialize the other output fields - flags and description.

[snip]

Ok, we will add format index & type checking in the beginning of this function. Moreover, we will initialize flags & description based on current meta format value.

Quoted 101 lines
Quoted 92 lines
Quoted 11 lines
Quoted 11 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +11 -0
+static u32 mtk_cam_node_get_v4l2_cap(struct mtk_cam_ctx_queue *node_ctx)
+{
+       u32 cap = 0;
+
+       if (node_ctx->desc.capture)
+               if (node_ctx->desc.image)
+                       cap = V4L2_CAP_VIDEO_CAPTURE_MPLANE;
+               else
+                       cap = V4L2_CAP_META_CAPTURE;
+       else
+               if (node_ctx->desc.image)
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 836

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+                       cap = V4L2_CAP_VIDEO_OUTPUT_MPLANE;
+               else
+                       cap = V4L2_CAP_META_OUTPUT;
+
+       return cap;
+}

Why not just have this defined statically as node_ctx->desc.cap?

Ok, it is refactoring done.

Quoted 12 lines
Quoted 12 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +12 -0
+
+static u32 mtk_cam_node_get_format_type(struct mtk_cam_ctx_queue *node_ctx)
+{
+       u32 type;
+
+       if (node_ctx->desc.capture)
+               if (node_ctx->desc.image)
+                       type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
+               else
+                       type = V4L2_BUF_TYPE_META_CAPTURE;
+       else
+               if (node_ctx->desc.image)
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 854

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+                       type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
+               else
+                       type = V4L2_BUF_TYPE_META_OUTPUT;
+
+       return type;
+}

Why not just have this defined statically as node_ctx->desc.buf_type?

Same as above.

Quoted 7 lines
Quoted 7 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +7 -0
+
+static const struct v4l2_ioctl_ops *mtk_cam_node_get_ioctl_ops
+       (struct mtk_cam_ctx_queue *node_ctx)
+{
+       const struct v4l2_ioctl_ops *ops = NULL;
+
+       if (node_ctx->desc.image)
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 866

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+               ops = &mtk_cam_v4l2_ioctl_ops;
+       else
+               ops = &mtk_cam_v4l2_meta_ioctl_ops;
+       return ops;
+}

It's also preferable to just put this inside some structure rather than have getter functions. (node_ctx->desc.ioctl_ops?)

Same as above.
Below is the new version for struct mtk_cam_ctx_queue_desc

/*
* struct mtk_cam_ctx_queue_desc - queue attributes
* setup by device context owner
* @id: id of the context queue
* @name: media entity name
* @cap: mapped to V4L2 capabilities
* @buf_type: mapped to V4L2 buffer type
* @capture: true for capture queue (device to user)
* false for output queue (from user to device)
* @image: true for image, false for meta data
* @smem_alloc: Using the cam_smem_drv as alloc ctx or not

 * @dma_port:   the dma port associated to the buffer
 * @fmts:       supported format
 * @num_fmts:   the number of supported formats
 * @default_fmt_idx: default format of this queue
 * @max_buf_count: maximum V4L2 buffer count
 * @max_buf_count: mapped to v4l2_ioctl_ops
 */
struct mtk_cam_ctx_queue_desc {
        u8 id;
        char *name;
        u32 cap;
        u32 buf_type;
        u32 dma_port;
        u32 smem_alloc:1;
Quoted 8 lines
        u8 capture:1;
        u8 image:1;
        u8 num_fmts;
        u8 default_fmt_idx;
        u8 max_buf_count;
        const struct v4l2_ioctl_ops *ioctl_ops;
        struct v4l2_format *fmts;
};
Quoted 3 lines

SGTM +/- the missing kerneldoc for the new fields.

[snip]

Ok, we will fix the missing part.

Quoted 11 lines
Quoted 8 lines
Quoted 5 lines

Sorry, ran out of time for today. Fourth part will come. :)

Best regards,
Tomasz

Appreciate your support and hard working on this review. We will look forward your part 4 review.

Thanks for replying to all the comments, it's very helpful.

Quoted 8 lines

As I mentioned in another reply, I'm going to be busy for the next few days, so I'd suggest addressing the existing comments, fixing any v4l2-compliance issues and also checking if any changes could be applied to the other drivers (DIP, FD, Seninf, MDP) too and then sending RFC V1.

Best regards,
Tomasz

Ok, we will deliver the RFC V1 next week.
Btw, for v4l2-compliance testing, we have some questions about this. We list these two issues below.
If you have time, please kindly provide your comments.
Or we could discuss these two issues in RFC V1.

1. Kernel minor issue for 32 bit test program.
=> fail: v4l2-test-media.cpp(368): check_0(links.reserved, sizeof(links.reserved))
This issue is similar to below patch.
https://patchwork.kernel.org/patch/8578421/
The reserved fields of media_links_enum is missing copy from kernel space to user space in media_device_enum_links32 function. So it makes the above failed.
Do we provide our patch to fix it?

2. Some v4l2-compliance testing results are failed in the sensor drivers's sub devices nodes. which are implemented in ov5695 & ov2685 driver. Do you suggest us to fix them?

Best regards,

Jungo

#24 lore raw

Re: [RFC PATCH V0 7/7] [media] platform: mtk-isp: Add Mediatek ISP Pass 1 driver

Quoted 263 lines · Jungo Lin
Quoted 31 lines · Tomasz Figa
Quoted 2 lines · Jungo Lin

On Tue, 2019-03-12 at 19:04 +0900, Tomasz Figa wrote:

[snip]

Quoted 27 lines
Quoted 9 lines

Instead of opencoding most of this function, one would normally call mtk_cam_videoc_try_fmt() first to adjust the format struct and then just update the driver state with the adjusted format.

Also, similarly to VIDIOC_TRY_FMT, VIDIOC_SET_FMT doesn't fail unless in the very specific cases, as described in
https://www.kernel.org/doc/html/latest/media/uapi/v4l/vidioc-g-fmt.html#return-value
.

Ok, below is our revised version of this function.

int mtk_cam_videoc_s_fmt(struct file *file, void *fh,
struct v4l2_format *f)

{
        struct mtk_cam_mem2mem2_device *m2m2 = video_drvdata(file);
        struct mtk_cam_dev *cam_dev = mtk_cam_m2m_to_dev(m2m2);
        struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);

        /* Get the valid format*/
        mtk_cam_videoc_try_fmt(file, fh, f);
        /* Configure to video device */
        mtk_cam_ctx_fmt_set_img(&cam_dev->pdev->dev,
                                &node->vdev_fmt.fmt.pix_mp,
                                &f->fmt.pix_mp,
                                node->queue_id);
Quoted 8 lines
Quoted 3 lines
        return 0;
}

Looks almost good. We still need to signal the -EBUSY error condition if an attempt to change the format is made while streaming is active.

[snip]

Ok, we will add streaming status checking in this function.

Quoted 73 lines
Quoted 73 lines
Quoted 12 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 645

Quoted 9 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +9 -0
+static int mtk_cam_videoc_s_meta_fmt(struct file *file,
+                                    void *fh, struct v4l2_format *f)
+{
+       struct mtk_cam_mem2mem2_device *m2m2 = video_drvdata(file);
+       struct mtk_cam_dev *isp_dev = mtk_cam_m2m_to_dev(m2m2);
+       struct mtk_cam_ctx *dev_ctx = &isp_dev->ctx;
+       struct mtk_cam_dev_video_device *node = file_to_mtk_cam_node(file);
+       int queue_id = mtk_cam_dev_get_queue_id_of_dev_node(isp_dev, node);
+

No need for this blank line.

We will fix this coding style in next patch.

Quoted 4 lines
Quoted 1 line

+ int ret = 0;

Please don't default-initialize without a good reason.

Ok, fix in next patch.

Quoted 6 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 649

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +3 -0
+
+       if (f->type != node->vbq.type)
+               return -EINVAL;

Ditto.

Ok, fix in next patch.

Quoted 6 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 652

Quoted 3 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +3 -0
+
+       ret = mtk_cam_ctx_format_load_default_fmt(&dev_ctx->queue[queue_id], f);
+

No need for this blank line.

Ok, fix in next patch.

Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 658

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+       if (!ret) {
+               node->vdev_fmt.fmt.meta = f->fmt.meta;
+               dev_ctx->queue[queue_id].fmt.meta = node->vdev_fmt.fmt.meta;
+       } else {
+               dev_warn(&isp_dev->pdev->dev,
+                        "s_meta_fm failed, format not support\n");

No need for this warning.

Ok, fix in next patch.

Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c line 905

Quoted 4 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-dev-ctx-core.c new file +4 -0
+       }
+
+       return ret;
+}

Actually, why do we even need to do all the things? Do we support multiple different meta formats on the same video node? If not, we can just have all the TRY_FMT/S_FMT/G_FMT return the hardcoded format.

Ok, it is a good idea. We will return the hardcode format for meta video devices.
Below is the revision version.

int mtk_cam_meta_enum_format(struct file *file,
void *fh, struct v4l2_fmtdesc *f)

{
        struct mtk_cam_video_device *node = file_to_mtk_cam_node(file);

        f->pixelformat = node->vdev_fmt.fmt.meta.dataformat;
Quoted 7 lines
Quoted 2 lines
        return 0;
}

Need to error out if f->index > 0. Also need to initialize the other output fields - flags and description.

[snip]

Ok, we will add format index & type checking in the beginning of this function. Moreover, we will initialize flags & description based on current meta format value.

Quoted 101 lines
Quoted 92 lines
Quoted 11 lines
Quoted 11 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +11 -0
+static u32 mtk_cam_node_get_v4l2_cap(struct mtk_cam_ctx_queue *node_ctx)
+{
+       u32 cap = 0;
+
+       if (node_ctx->desc.capture)
+               if (node_ctx->desc.image)
+                       cap = V4L2_CAP_VIDEO_CAPTURE_MPLANE;
+               else
+                       cap = V4L2_CAP_META_CAPTURE;
+       else
+               if (node_ctx->desc.image)
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 836

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+                       cap = V4L2_CAP_VIDEO_OUTPUT_MPLANE;
+               else
+                       cap = V4L2_CAP_META_OUTPUT;
+
+       return cap;
+}

Why not just have this defined statically as node_ctx->desc.cap?

Ok, it is refactoring done.

Quoted 12 lines
Quoted 12 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +12 -0
+
+static u32 mtk_cam_node_get_format_type(struct mtk_cam_ctx_queue *node_ctx)
+{
+       u32 type;
+
+       if (node_ctx->desc.capture)
+               if (node_ctx->desc.image)
+                       type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
+               else
+                       type = V4L2_BUF_TYPE_META_CAPTURE;
+       else
+               if (node_ctx->desc.image)
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 854

Quoted 6 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +6 -0
+                       type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
+               else
+                       type = V4L2_BUF_TYPE_META_OUTPUT;
+
+       return type;
+}

Why not just have this defined statically as node_ctx->desc.buf_type?

Same as above.

Quoted 7 lines
Quoted 7 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +7 -0
+
+static const struct v4l2_ioctl_ops *mtk_cam_node_get_ioctl_ops
+       (struct mtk_cam_ctx_queue *node_ctx)
+{
+       const struct v4l2_ioctl_ops *ops = NULL;
+
+       if (node_ctx->desc.image)
Quoted 9 lines

drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c line 866

Quoted 5 lines
drivers/media/platform/mtk-isp/isp_50/cam/mtk_cam-v4l2-util.c new file +5 -0
+               ops = &mtk_cam_v4l2_ioctl_ops;
+       else
+               ops = &mtk_cam_v4l2_meta_ioctl_ops;
+       return ops;
+}

It's also preferable to just put this inside some structure rather than have getter functions. (node_ctx->desc.ioctl_ops?)

Same as above.
Below is the new version for struct mtk_cam_ctx_queue_desc

/*
* struct mtk_cam_ctx_queue_desc - queue attributes
* setup by device context owner
* @id: id of the context queue
* @name: media entity name
* @cap: mapped to V4L2 capabilities
* @buf_type: mapped to V4L2 buffer type
* @capture: true for capture queue (device to user)
* false for output queue (from user to device)
* @image: true for image, false for meta data
* @smem_alloc: Using the cam_smem_drv as alloc ctx or not

 * @dma_port:   the dma port associated to the buffer
 * @fmts:       supported format
 * @num_fmts:   the number of supported formats
 * @default_fmt_idx: default format of this queue
 * @max_buf_count: maximum V4L2 buffer count
 * @max_buf_count: mapped to v4l2_ioctl_ops
 */
struct mtk_cam_ctx_queue_desc {
        u8 id;
        char *name;
        u32 cap;
        u32 buf_type;
        u32 dma_port;
        u32 smem_alloc:1;
Quoted 8 lines
        u8 capture:1;
        u8 image:1;
        u8 num_fmts;
        u8 default_fmt_idx;
        u8 max_buf_count;
        const struct v4l2_ioctl_ops *ioctl_ops;
        struct v4l2_format *fmts;
};
Quoted 3 lines

SGTM +/- the missing kerneldoc for the new fields.

[snip]

Ok, we will fix the missing part.

Quoted 11 lines
Quoted 8 lines
Quoted 5 lines

Sorry, ran out of time for today. Fourth part will come. :)

Best regards,
Tomasz

Appreciate your support and hard working on this review. We will look forward your part 4 review.

Thanks for replying to all the comments, it's very helpful.

Quoted 8 lines

As I mentioned in another reply, I'm going to be busy for the next few days, so I'd suggest addressing the existing comments, fixing any v4l2-compliance issues and also checking if any changes could be applied to the other drivers (DIP, FD, Seninf, MDP) too and then sending RFC V1.

Best regards,
Tomasz

Ok, we will deliver the RFC V1 next week.
Btw, for v4l2-compliance testing, we have some questions about this. We list these two issues below.
If you have time, please kindly provide your comments.
Or we could discuss these two issues in RFC V1.

Quoted 9 lines

1. Kernel minor issue for 32 bit test program.
=> fail: v4l2-test-media.cpp(368): check_0(links.reserved, sizeof(links.reserved))
This issue is similar to below patch.
https://patchwork.kernel.org/patch/8578421/
The reserved fields of media_links_enum is missing copy from kernel space to user space in media_device_enum_links32 function. So it makes the above failed.
Do we provide our patch to fix it?

Yes, it would be nice if you could send a patch fixing it.

Quoted 4 lines

2. Some v4l2-compliance testing results are failed in the sensor drivers's sub devices nodes. which are implemented in ov5695 & ov2685 driver. Do you suggest us to fix them?

Let's look at this off the list and figure out. Please send me an email with the results.

Other than that, just to make sure we're all on the same page, please try to fix any remaining v4l2-compliance issues of Mediatek drivers for RFC V1.

Best regards,
Tomasz