Skip to contentSkip to conversation

media: rockchip: Add the IEP2 deinterlacer of RK3568/RK3588

PATCH New Arun Iyengar 2 participants 5 messages Nov 20

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v3 cover media: rockchip: Add the IEP2 deinterlacer of RK3568/RK3588

Add a V4L2 mem2mem driver for the IEP2 deinterlacer of the RK3568 and RK3588, with its binding and the RK3588 DT node. The V4L2 side follows Alex Bee's IEP driver.

Tested on an Orange Pi 5 (RK3588S) with a 7.3-based kernel (KASAN, lockdep); build-tested on media next. v4l2-compliance 59/59, all four modes at 1080i without IOMMU faults, s2idle and deep suspend while streaming, unbind while streaming, forced watchdog and IOMMU fault resets. Not tested: RK3568. DMABUF imports of exactly the image size are refused, because of the 4 KiB OUTPUT padding (see 2/3).

One case is still open: a reset during a transfer, seen only with forced watchdog resets, can leave the IOMMU busy. Jobs then fail without DMA and the driver suspends at once so that PD_VDPU powers off, which restores it. While another device of the domain, e.g. a video decoder, keeps it on, the jobs keep failing.

Developed with Claude (Anthropic), which wrote the code and the commit messages. I reviewed every change.

---
Changes in v3:
- Drop the vendor kernel notes from the commit messages and cover (Krzysztof)
- 1/3: drop the linux-media list from the MAINTAINERS entry (Krzysztof)
- 1/3: add Krzysztof's Reviewed-by
- 2/3: start the job and its watchdog under irq_lock (Sashiko)
- 2/3: soft-reset the core with the AXI side, CRU reset if the IOMMU stays lost, shared IRQ off during a reset; no DMA until the IOMMU is restored
- 2/3: return OUTPUT buffers before CAPTURE buffers
- 2/3: mask and clear all interrupts on completion and before a job
- 2/3: suspend at once when the IOMMU is lost
- 2/3: take the mutex in system suspend
- Link to v2: https://patch.msgid.link/[email protected]

---
Arun Iyengar (3):
media: dt-bindings: Add Rockchip IEP2 deinterlacer
media: rockchip: Add IEP2 deinterlacer driver
arm64: dts: rockchip: Add IEP2 deinterlacer to RK3588

 Documentation/admin-guide/media/rockchip-iep2.rst  |   32 +
 Documentation/admin-guide/media/v4l-drivers.rst    |    1 +
 .../bindings/media/rockchip,rk3568-iep2.yaml       |   95 ++
 MAINTAINERS                                        |    8 +
 arch/arm64/boot/dts/rockchip/rk3588-base.dtsi      |   22 +
 drivers/media/platform/rockchip/Kconfig            |    1 +
 drivers/media/platform/rockchip/Makefile           |    1 +
 drivers/media/platform/rockchip/iep2/Kconfig       |   16 +
 drivers/media/platform/rockchip/iep2/Makefile      |    3 +
 drivers/media/platform/rockchip/iep2/iep2-regs.h   |  191 +++
 drivers/media/platform/rockchip/iep2/iep2.c        | 1320 ++++++++++++++++++++
 drivers/media/platform/rockchip/iep2/iep2.h        |  108 ++
 12 files changed, 1798 insertions(+)

---
base-commit: ad6549a46748d00818483b535e5cd2a704685c98
change-id: 20261110-iep2-b9a9c6e77c44

Best regards,

Signature · 1 line
Arun Iyengar <[email protected]>
#1 lore raw

PATCH v3 1/3 media: dt-bindings: Add Rockchip IEP2 deinterlacer

Add a binding for the IEP2 deinterlacer of the RK3568 and RK3588 SoCs. It is a different block from the IEP of RK3228/RK3288/RK3328/RK3399, hence "iep2". The RK3588 block falls back to the RK3568 compatible.

Clock and reset names follow the RGA binding. The resets are optional:
the block has its own soft reset.

---
 .../bindings/media/rockchip,rk3568-iep2.yaml       | 95 ++++++++++++++++++++++
 MAINTAINERS                                        |  6 ++
 2 files changed, 101 insertions(+)
Documentation/devicetree/bindings/media/rockchip,rk3568-iep2.yaml new file +95 -0
new file mode 100644
index 000000000000..0c7666de19f8
@@ -0,0 +1,95 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/media/rockchip,rk3568-iep2.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Rockchip IEP2 image enhancement processor
+
+maintainers:
+  - Arun Iyengar <[email protected]>
+
+description:
+  Memory to memory video deinterlacer of the Rockchip RK3568 and RK3588
+  SoCs. It has its own IOMMU, with which it shares its interrupt line.
+
+properties:
+  compatible:
+    oneOf:
+      - const: rockchip,rk3568-iep2
+      - items:
+          - const: rockchip,rk3588-iep2
+          - const: rockchip,rk3568-iep2
+
+  reg:
+    maxItems: 1
+
+  interrupts:
+    maxItems: 1
+
+  clocks:
+    items:
+      - description: AXI bus clock
+      - description: AHB bus clock
+      - description: Core clock
+
+  clock-names:
+    items:
+      - const: aclk
+      - const: hclk
+      - const: sclk
+
+  resets:
+    items:
+      - description: AXI reset
+      - description: AHB reset
+      - description: Core reset
+
+  reset-names:
+    items:
+      - const: axi
+      - const: ahb
+      - const: core
+
+  power-domains:
+    maxItems: 1
+
+  iommus:
+    maxItems: 1
+
+required:
+  - compatible
+  - reg
+  - interrupts
+  - clocks
+  - clock-names
+  - power-domains
+  - iommus
+
+additionalProperties: false
+
+examples:
+  - |
+    #include <dt-bindings/clock/rockchip,rk3588-cru.h>
+    #include <dt-bindings/interrupt-controller/arm-gic.h>
+    #include <dt-bindings/power/rk3588-power.h>
+    #include <dt-bindings/reset/rockchip,rk3588-cru.h>
+
+    bus {
+      #address-cells = <2>;
+      #size-cells = <2>;
+
+      deinterlace@fdbb0000 {
+        compatible = "rockchip,rk3588-iep2", "rockchip,rk3568-iep2";
+        reg = <0x0 0xfdbb0000 0x0 0x500>;
+        interrupts = <GIC_SPI 117 IRQ_TYPE_LEVEL_HIGH>;
+        clocks = <&cru ACLK_IEP2P0>, <&cru HCLK_IEP2P0>,
+                 <&cru CLK_IEP2P0_CORE>;
+        clock-names = "aclk", "hclk", "sclk";
+        resets = <&cru SRST_A_IEP2P0>, <&cru SRST_H_IEP2P0>,
+                 <&cru SRST_IEP2P0_CORE>;
+        reset-names = "axi", "ahb", "core";
+        power-domains = <&power RK3588_PD_VDPU>;
+        iommus = <&iep_mmu>;
+      };
+    };
MAINTAINERS +6 -0
index f4456f4589a1..74a85e1653f4 100644
@@ -24285,6 +24285,12 @@ S:	Maintained
 F:	Documentation/devicetree/bindings/sound/rockchip,i2s-tdm.yaml
 F:	sound/soc/rockchip/rockchip_i2s_tdm.*
 
+ROCKCHIP IEP2 DEINTERLACER DRIVER
+M:	Arun Iyengar <[email protected]>
+L:	[email protected]
+S:	Maintained
+F:	Documentation/devicetree/bindings/media/rockchip,rk3568-iep2.yaml
+
 ROCKCHIP ISP V1 DRIVER
 M:	Dafna Hirschfeld <[email protected]>
 M:	Laurent Pinchart <[email protected]>
Signature · 1 line
2.55.0
#2 lore raw

PATCH v3 2/3 media: rockchip: Add IEP2 deinterlacer driver

In reply to Arun Iyengar

Add a V4L2 mem2mem driver for the IEP2 deinterlacer of the RK3568 and RK3588. The V4L2 part follows sun8i-di and Alex Bee's IEP driver.

The hardware reads past the end of the source, so OUTPUT buffers have 4 KiB of tail padding.

---
 Documentation/admin-guide/media/rockchip-iep2.rst |   32 +
 Documentation/admin-guide/media/v4l-drivers.rst   |    1 +
 MAINTAINERS                                       |    2 +
 drivers/media/platform/rockchip/Kconfig           |    1 +
 drivers/media/platform/rockchip/Makefile          |    1 +
 drivers/media/platform/rockchip/iep2/Kconfig      |   16 +
 drivers/media/platform/rockchip/iep2/Makefile     |    3 +
 drivers/media/platform/rockchip/iep2/iep2-regs.h  |  191 +++
 drivers/media/platform/rockchip/iep2/iep2.c       | 1320 +++++++++++++++++++++
 drivers/media/platform/rockchip/iep2/iep2.h       |  108 ++
 10 files changed, 1675 insertions(+)
Documentation/admin-guide/media/rockchip-iep2.rst new file +32 -0
new file mode 100644
index 000000000000..ee0ffe23dec2
@@ -0,0 +1,32 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+Rockchip IEP2 deinterlacer driver
+=================================
+
+The rockchip-iep2 driver supports the IEP2 deinterlacer of the RK3568 and
+RK3588 as a V4L2 mem2mem device. OUTPUT takes interlaced frames, CAPTURE
+returns progressive frames in the same format. For
+``V4L2_FIELD_INTERLACED`` the field order follows the height: bottom field
+first below 576 lines, top field first otherwise.
+
+``V4L2_CID_DEINTERLACING_MODE (menu)``
+    The rate cannot change while a queue is streaming (``EBUSY``).
+
+.. flat-table::
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 4
+
+    * - ``"Bob, Frame Rate" (0)``
+      - One CAPTURE buffer per OUTPUT buffer. Default.
+    * - ``"Bob, Field Rate" (1)``
+      - Two CAPTURE buffers per OUTPUT buffer.
+    * - ``"Motion Adaptive, Frame Rate" (2)``
+      - Uses the previous OUTPUT buffer as reference, so at least two
+        OUTPUT buffers are needed. One CAPTURE buffer per OUTPUT buffer.
+    * - ``"Motion Adaptive, Field Rate" (3)``
+      - As above, two CAPTURE buffers per OUTPUT buffer.
+
+The hardware reads past the end of the source image, so the OUTPUT
+``sizeimage`` is 4 KiB larger than the image and OUTPUT buffers must be at
+least that size.
Documentation/admin-guide/media/v4l-drivers.rst +1 -0
index 4621eae9fa1e..3d59d5d73fd0 100644
@@ -29,6 +29,7 @@ Video4Linux (V4L) driver-specific documentation
 	rcar-fdp1
 	rkcif
 	rkisp1
+	rockchip-iep2
 	raspberrypi-rp1-cfe
 	saa7134
 	si470x
MAINTAINERS +2 -0
index 74a85e1653f4..ce603dcec801 100644
@@ -24289,7 +24289,9 @@ ROCKCHIP IEP2 DEINTERLACER DRIVER
 M:	Arun Iyengar <[email protected]>
 L:	[email protected]
 S:	Maintained
+F:	Documentation/admin-guide/media/rockchip-iep2.rst
 F:	Documentation/devicetree/bindings/media/rockchip,rk3568-iep2.yaml
+F:	drivers/media/platform/rockchip/iep2/
 
 ROCKCHIP ISP V1 DRIVER
 M:	Dafna Hirschfeld <[email protected]>
drivers/media/platform/rockchip/Kconfig +1 -0
index ba401d32f01b..4f65bd0126de 100644
@@ -2,6 +2,7 @@
 
 comment "Rockchip media platform drivers"
 
+source "drivers/media/platform/rockchip/iep2/Kconfig"
 source "drivers/media/platform/rockchip/rga/Kconfig"
 source "drivers/media/platform/rockchip/rkcif/Kconfig"
 source "drivers/media/platform/rockchip/rkisp1/Kconfig"
drivers/media/platform/rockchip/Makefile +1 -0
index 0e0b2cbbd4bd..c385048ba347 100644
@@ -1,4 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0-only
+obj-y += iep2/
 obj-y += rga/
 obj-y += rkcif/
 obj-y += rkisp1/
drivers/media/platform/rockchip/iep2/Kconfig new file +16 -0
new file mode 100644
index 000000000000..de1e445b3fd8
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config VIDEO_ROCKCHIP_IEP2
+	tristate "Rockchip IEP2 deinterlacer"
+	depends on V4L_MEM2MEM_DRIVERS
+	depends on VIDEO_DEV
+	depends on ARCH_ROCKCHIP || COMPILE_TEST
+	select VIDEOBUF2_DMA_CONTIG
+	select V4L2_MEM2MEM_DEV
+	help
+	  Support for the IEP2 (image enhancement processor) found in the
+	  Rockchip RK3568 and RK3588 SoCs. It is exposed as a V4L2 memory to
+	  memory device that deinterlaces NV12/NV21/NV16/NV61 frames, using
+	  either intra-field (bob) or motion-adaptive interpolation.
+
+	  To compile this driver as a module, choose M here: the module
+	  will be called rockchip-iep2.
drivers/media/platform/rockchip/iep2/Makefile new file +3 -0
new file mode 100644
index 000000000000..f13bb799de54
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_VIDEO_ROCKCHIP_IEP2) += rockchip-iep2.o
+rockchip-iep2-y := iep2.o
drivers/media/platform/rockchip/iep2/iep2-regs.h new file +191 -0

191 lines not on this page · raw message

drivers/media/platform/rockchip/iep2/iep2.c new file +1320 -0

1,320 lines not on this page · raw message

drivers/media/platform/rockchip/iep2/iep2.h new file +108 -0

108 lines not on this page · raw message

Signature · 1 line
2.55.0
#3 lore raw

Re: [PATCH v3 2/3] media: rockchip: Add IEP2 deinterlacer driver

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] media: rockchip: iep2: unhandled empty_domain alloc failure in iep_probe() --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] media: rockchip: iep2: unhandled empty_domain alloc failure in iep_probe()
--

commit 412e46dad64f85e912b35379802af12bdb8fee8c
Author: Arun Iyengar <[email protected]>

media: rockchip: Add IEP2 deinterlacer driver

This commit adds a V4L2 mem2mem driver for the Rockchip IEP2 deinterlacer on the RK3568 and RK3588 SoCs.

Quoted 6 lines
drivers/media/platform/rockchip/iep2/iep2.c new file +0 -0
new file mode 100644
index 0000000000000..11b349dda7196
@@ -0,0 +1,1320 @@

[ ... ]

Quoted 19 lines
drivers/media/platform/rockchip/iep2/iep2.c new file +19 -0
+static int iep_iommu_restore(struct iep_dev *iep)
+{
+	int ret;
+
+	if (!iep->empty_domain)
+		return 0;
+
+	/* A reset leaves the IOMMU in bypass; reattaching reprograms it */
+	ret = iommu_attach_device(iep->empty_domain, iep->dev);
+	if (!ret)
+		iommu_detach_device(iep->empty_domain, iep->dev);
+
+	return ret;
+}
+
+static void iep_reset(struct iep_dev *iep)
+{
+	u32 status;
+	int ret;

[ ... ]

Quoted 6 lines
drivers/media/platform/rockchip/iep2/iep2.c new file +6 -0
+	ret = iep_iommu_restore(iep);
+	enable_irq(iep->irq);
+	if (ret)
+		dev_err_ratelimited(iep->dev, "reset failed: %d\n", ret);
+	iep->iommu_lost = ret;
+}

[ ... ]

drivers/media/platform/rockchip/iep2/iep2.c line 1213

Quoted 7 lines
drivers/media/platform/rockchip/iep2/iep2.c new file +7 -0
+	if (iommu_get_domain_for_dev(dev)) {
+		iep->empty_domain = iommu_paging_domain_alloc(dev);
+		if (IS_ERR(iep->empty_domain)) {
+			dev_warn(dev, "cannot allocate empty IOMMU domain\n");
+			iep->empty_domain = NULL;
+		}
+	}

[Severity: Medium]
Should iep_probe() fail when iommu_paging_domain_alloc() returns an error?

Setting iep->empty_domain to NULL causes iep_iommu_restore() to return 0 after a hardware reset even though an IOMMU is attached and left in bypass mode. iep_reset() then clears iep->iommu_lost, skipping the synchronous runtime PM suspend in iep_pm_put() and the iep->iommu_lost check in iep_device_run(), so later jobs run DMA with IOVA addresses while the IOMMU is in bypass.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#4 lore raw

PATCH v3 3/3 arm64: dts: rockchip: Add IEP2 deinterlacer to RK3588

In reply to Arun Iyengar

Add the IEP2 deinterlacer and its IOMMU to the RK3588 base dtsi. Both share GIC SPI 117 and sit in the VDPU power domain.

---
 arch/arm64/boot/dts/rockchip/rk3588-base.dtsi | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
arch/arm64/boot/dts/rockchip/rk3588-base.dtsi +22 -0
index aa69fd3793fd..5297519bd251 100644
@@ -1397,6 +1397,28 @@ vepu121_3_mmu: iommu@fdbac800 {
 		#iommu-cells = <0>;
 	};
 
+	iep: deinterlace@fdbb0000 {
+		compatible = "rockchip,rk3588-iep2", "rockchip,rk3568-iep2";
+		reg = <0x0 0xfdbb0000 0x0 0x500>;
+		interrupts = <GIC_SPI 117 IRQ_TYPE_LEVEL_HIGH 0>;
+		clocks = <&cru ACLK_IEP2P0>, <&cru HCLK_IEP2P0>, <&cru CLK_IEP2P0_CORE>;
+		clock-names = "aclk", "hclk", "sclk";
+		resets = <&cru SRST_A_IEP2P0>, <&cru SRST_H_IEP2P0>, <&cru SRST_IEP2P0_CORE>;
+		reset-names = "axi", "ahb", "core";
+		power-domains = <&power RK3588_PD_VDPU>;
+		iommus = <&iep_mmu>;
+	};
+
+	iep_mmu: iommu@fdbb0800 {
+		compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
+		reg = <0x0 0xfdbb0800 0x0 0x100>;
+		interrupts = <GIC_SPI 117 IRQ_TYPE_LEVEL_HIGH 0>;
+		clocks = <&cru ACLK_IEP2P0>, <&cru HCLK_IEP2P0>;
+		clock-names = "aclk", "iface";
+		power-domains = <&power RK3588_PD_VDPU>;
+		#iommu-cells = <0>;
+	};
+
 	vdec0: video-codec@fdc38000 {
 		compatible = "rockchip,rk3588-vdec";
 		reg = <0x0 0xfdc38000 0x0 0x100>,
Signature · 1 line
2.55.0