media: rockchip: Add the IEP2 deinterlacer of RK3568/RK3588
Subsystems (3)
v3 cover media: rockchip: Add the IEP2 deinterlacer of RK3568/RK3588
Add a V4L2 mem2mem driver for the IEP2 deinterlacer of the RK3568 and RK3588, with its binding and the RK3588 DT node. The V4L2 side follows Alex Bee's IEP driver.
Tested on an Orange Pi 5 (RK3588S) with a 7.3-based kernel (KASAN, lockdep); build-tested on media next. v4l2-compliance 59/59, all four modes at 1080i without IOMMU faults, s2idle and deep suspend while streaming, unbind while streaming, forced watchdog and IOMMU fault resets. Not tested: RK3568. DMABUF imports of exactly the image size are refused, because of the 4 KiB OUTPUT padding (see 2/3).
One case is still open: a reset during a transfer, seen only with forced watchdog resets, can leave the IOMMU busy. Jobs then fail without DMA and the driver suspends at once so that PD_VDPU powers off, which restores it. While another device of the domain, e.g. a video decoder, keeps it on, the jobs keep failing.
Developed with Claude (Anthropic), which wrote the code and the commit messages. I reviewed every change.
---
Changes in v3:
- Drop the vendor kernel notes from the commit messages and cover (Krzysztof)
- 1/3: drop the linux-media list from the MAINTAINERS entry (Krzysztof)
- 1/3: add Krzysztof's Reviewed-by
- 2/3: start the job and its watchdog under irq_lock (Sashiko)
- 2/3: soft-reset the core with the AXI side, CRU reset if the IOMMU stays lost, shared IRQ off during a reset; no DMA until the IOMMU is restored
- 2/3: return OUTPUT buffers before CAPTURE buffers
- 2/3: mask and clear all interrupts on completion and before a job
- 2/3: suspend at once when the IOMMU is lost
- 2/3: take the mutex in system suspend
- Link to v2: https://patch.msgid.link/[email protected]
---
Arun Iyengar (3):
media: dt-bindings: Add Rockchip IEP2 deinterlacer
media: rockchip: Add IEP2 deinterlacer driver
arm64: dts: rockchip: Add IEP2 deinterlacer to RK3588
Documentation/admin-guide/media/rockchip-iep2.rst | 32 + Documentation/admin-guide/media/v4l-drivers.rst | 1 + .../bindings/media/rockchip,rk3568-iep2.yaml | 95 ++ MAINTAINERS | 8 + arch/arm64/boot/dts/rockchip/rk3588-base.dtsi | 22 + drivers/media/platform/rockchip/Kconfig | 1 + drivers/media/platform/rockchip/Makefile | 1 + drivers/media/platform/rockchip/iep2/Kconfig | 16 + drivers/media/platform/rockchip/iep2/Makefile | 3 + drivers/media/platform/rockchip/iep2/iep2-regs.h | 191 +++ drivers/media/platform/rockchip/iep2/iep2.c | 1320 ++++++++++++++++++++ drivers/media/platform/rockchip/iep2/iep2.h | 108 ++ 12 files changed, 1798 insertions(+)
---
base-commit: ad6549a46748d00818483b535e5cd2a704685c98
change-id: 20261110-iep2-b9a9c6e77c44
Best regards,
Signature · 1 line
Arun Iyengar <[email protected]>
PATCH v3 1/3 media: dt-bindings: Add Rockchip IEP2 deinterlacer
Add a binding for the IEP2 deinterlacer of the RK3568 and RK3588 SoCs. It is a different block from the IEP of RK3228/RK3288/RK3328/RK3399, hence "iep2". The RK3588 block falls back to the RK3568 compatible.
Clock and reset names follow the RGA binding. The resets are optional:
the block has its own soft reset.
--- .../bindings/media/rockchip,rk3568-iep2.yaml | 95 ++++++++++++++++++++++ MAINTAINERS | 6 ++ 2 files changed, 101 insertions(+)
Documentation/devicetree/bindings/media/rockchip,rk3568-iep2.yaml new file +95 -0
@@ -0,0 +1,95 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/media/rockchip,rk3568-iep2.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Rockchip IEP2 image enhancement processor + +maintainers: + - Arun Iyengar <[email protected]> + +description: + Memory to memory video deinterlacer of the Rockchip RK3568 and RK3588 + SoCs. It has its own IOMMU, with which it shares its interrupt line. + +properties: + compatible: + oneOf: + - const: rockchip,rk3568-iep2 + - items: + - const: rockchip,rk3588-iep2 + - const: rockchip,rk3568-iep2 + + reg: + maxItems: 1 + + interrupts: + maxItems: 1 + + clocks: + items: + - description: AXI bus clock + - description: AHB bus clock + - description: Core clock + + clock-names: + items: + - const: aclk + - const: hclk + - const: sclk + + resets: + items: + - description: AXI reset + - description: AHB reset + - description: Core reset + + reset-names: + items: + - const: axi + - const: ahb + - const: core + + power-domains: + maxItems: 1 + + iommus: + maxItems: 1 + +required: + - compatible + - reg + - interrupts + - clocks + - clock-names + - power-domains + - iommus + +additionalProperties: false + +examples: + - | + #include <dt-bindings/clock/rockchip,rk3588-cru.h> + #include <dt-bindings/interrupt-controller/arm-gic.h> + #include <dt-bindings/power/rk3588-power.h> + #include <dt-bindings/reset/rockchip,rk3588-cru.h> + + bus { + #address-cells = <2>; + #size-cells = <2>; + + deinterlace@fdbb0000 { + compatible = "rockchip,rk3588-iep2", "rockchip,rk3568-iep2"; + reg = <0x0 0xfdbb0000 0x0 0x500>; + interrupts = <GIC_SPI 117 IRQ_TYPE_LEVEL_HIGH>; + clocks = <&cru ACLK_IEP2P0>, <&cru HCLK_IEP2P0>, + <&cru CLK_IEP2P0_CORE>; + clock-names = "aclk", "hclk", "sclk"; + resets = <&cru SRST_A_IEP2P0>, <&cru SRST_H_IEP2P0>, + <&cru SRST_IEP2P0_CORE>; + reset-names = "axi", "ahb", "core"; + power-domains = <&power RK3588_PD_VDPU>; + iommus = <&iep_mmu>; + }; + };
MAINTAINERS +6 -0
@@ -24285,6 +24285,12 @@ S: Maintained F: Documentation/devicetree/bindings/sound/rockchip,i2s-tdm.yaml F: sound/soc/rockchip/rockchip_i2s_tdm.* +ROCKCHIP IEP2 DEINTERLACER DRIVER +M: Arun Iyengar <[email protected]> +L: [email protected] +S: Maintained +F: Documentation/devicetree/bindings/media/rockchip,rk3568-iep2.yaml + ROCKCHIP ISP V1 DRIVER M: Dafna Hirschfeld <[email protected]> M: Laurent Pinchart <[email protected]>
Signature · 1 line
2.55.0
PATCH v3 2/3 media: rockchip: Add IEP2 deinterlacer driver
In reply to Arun IyengarAdd a V4L2 mem2mem driver for the IEP2 deinterlacer of the RK3568 and RK3588. The V4L2 part follows sun8i-di and Alex Bee's IEP driver.
The hardware reads past the end of the source, so OUTPUT buffers have 4 KiB of tail padding.
--- Documentation/admin-guide/media/rockchip-iep2.rst | 32 + Documentation/admin-guide/media/v4l-drivers.rst | 1 + MAINTAINERS | 2 + drivers/media/platform/rockchip/Kconfig | 1 + drivers/media/platform/rockchip/Makefile | 1 + drivers/media/platform/rockchip/iep2/Kconfig | 16 + drivers/media/platform/rockchip/iep2/Makefile | 3 + drivers/media/platform/rockchip/iep2/iep2-regs.h | 191 +++ drivers/media/platform/rockchip/iep2/iep2.c | 1320 +++++++++++++++++++++ drivers/media/platform/rockchip/iep2/iep2.h | 108 ++ 10 files changed, 1675 insertions(+)
Documentation/admin-guide/media/rockchip-iep2.rst new file +32 -0
@@ -0,0 +1,32 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Rockchip IEP2 deinterlacer driver +================================= + +The rockchip-iep2 driver supports the IEP2 deinterlacer of the RK3568 and +RK3588 as a V4L2 mem2mem device. OUTPUT takes interlaced frames, CAPTURE +returns progressive frames in the same format. For +``V4L2_FIELD_INTERLACED`` the field order follows the height: bottom field +first below 576 lines, top field first otherwise. + +``V4L2_CID_DEINTERLACING_MODE (menu)`` + The rate cannot change while a queue is streaming (``EBUSY``). + +.. flat-table:: + :header-rows: 0 + :stub-columns: 0 + :widths: 1 4 + + * - ``"Bob, Frame Rate" (0)`` + - One CAPTURE buffer per OUTPUT buffer. Default. + * - ``"Bob, Field Rate" (1)`` + - Two CAPTURE buffers per OUTPUT buffer. + * - ``"Motion Adaptive, Frame Rate" (2)`` + - Uses the previous OUTPUT buffer as reference, so at least two + OUTPUT buffers are needed. One CAPTURE buffer per OUTPUT buffer. + * - ``"Motion Adaptive, Field Rate" (3)`` + - As above, two CAPTURE buffers per OUTPUT buffer. + +The hardware reads past the end of the source image, so the OUTPUT +``sizeimage`` is 4 KiB larger than the image and OUTPUT buffers must be at +least that size.
Documentation/admin-guide/media/v4l-drivers.rst +1 -0
@@ -29,6 +29,7 @@ Video4Linux (V4L) driver-specific documentation rcar-fdp1 rkcif rkisp1 + rockchip-iep2 raspberrypi-rp1-cfe saa7134 si470x
MAINTAINERS +2 -0
@@ -24289,7 +24289,9 @@ ROCKCHIP IEP2 DEINTERLACER DRIVER M: Arun Iyengar <[email protected]> L: [email protected] S: Maintained +F: Documentation/admin-guide/media/rockchip-iep2.rst F: Documentation/devicetree/bindings/media/rockchip,rk3568-iep2.yaml +F: drivers/media/platform/rockchip/iep2/ ROCKCHIP ISP V1 DRIVER M: Dafna Hirschfeld <[email protected]>
drivers/media/platform/rockchip/Kconfig +1 -0
@@ -2,6 +2,7 @@ comment "Rockchip media platform drivers" +source "drivers/media/platform/rockchip/iep2/Kconfig" source "drivers/media/platform/rockchip/rga/Kconfig" source "drivers/media/platform/rockchip/rkcif/Kconfig" source "drivers/media/platform/rockchip/rkisp1/Kconfig"
drivers/media/platform/rockchip/Makefile +1 -0
@@ -1,4 +1,5 @@ # SPDX-License-Identifier: GPL-2.0-only +obj-y += iep2/ obj-y += rga/ obj-y += rkcif/ obj-y += rkisp1/
drivers/media/platform/rockchip/iep2/Kconfig new file +16 -0
@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: GPL-2.0-only +config VIDEO_ROCKCHIP_IEP2 + tristate "Rockchip IEP2 deinterlacer" + depends on V4L_MEM2MEM_DRIVERS + depends on VIDEO_DEV + depends on ARCH_ROCKCHIP || COMPILE_TEST + select VIDEOBUF2_DMA_CONTIG + select V4L2_MEM2MEM_DEV + help + Support for the IEP2 (image enhancement processor) found in the + Rockchip RK3568 and RK3588 SoCs. It is exposed as a V4L2 memory to + memory device that deinterlaces NV12/NV21/NV16/NV61 frames, using + either intra-field (bob) or motion-adaptive interpolation. + + To compile this driver as a module, choose M here: the module + will be called rockchip-iep2.
drivers/media/platform/rockchip/iep2/Makefile new file +3 -0
@@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0-only +obj-$(CONFIG_VIDEO_ROCKCHIP_IEP2) += rockchip-iep2.o +rockchip-iep2-y := iep2.o
drivers/media/platform/rockchip/iep2/iep2-regs.h new file +191 -0
191 lines not on this page · raw message
drivers/media/platform/rockchip/iep2/iep2.c new file +1320 -0
1,320 lines not on this page · raw message
drivers/media/platform/rockchip/iep2/iep2.h new file +108 -0
108 lines not on this page · raw message
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH v3 2/3] media: rockchip: Add IEP2 deinterlacer driver
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] media: rockchip: iep2: unhandled empty_domain alloc failure in iep_probe() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] media: rockchip: iep2: unhandled empty_domain alloc failure in iep_probe()
--
commit 412e46dad64f85e912b35379802af12bdb8fee8c
Author: Arun Iyengar <[email protected]>
media: rockchip: Add IEP2 deinterlacer driver
This commit adds a V4L2 mem2mem driver for the Rockchip IEP2 deinterlacer on the RK3568 and RK3588 SoCs.
Quoted 6 lines
drivers/media/platform/rockchip/iep2/iep2.c new file +0
-0@@ -0,0 +1,1320 @@
[ ... ]
Quoted 19 lines
drivers/media/platform/rockchip/iep2/iep2.c new file +19
-0+static int iep_iommu_restore (struct iep_dev *iep) +{ +int ret; + +if (!iep->empty_domain) +return 0 ; + +/* A reset leaves the IOMMU in bypass; reattaching reprograms it */ + ret =iommu_attach_device (iep->empty_domain, iep->dev); +if (!ret) +iommu_detach_device (iep->empty_domain, iep->dev); + +return ret; +} + +static void iep_reset (struct iep_dev *iep) +{ +u32 status; +int ret;
[ ... ]
Quoted 6 lines
drivers/media/platform/rockchip/iep2/iep2.c new file +6
-0+ ret =iep_iommu_restore (iep); +enable_irq (iep->irq); +if (ret) +dev_err_ratelimited (iep->dev,"reset failed: %d\n" , ret); + iep->iommu_lost = ret; +}
[ ... ]
drivers/media/platform/rockchip/iep2/iep2.c line 1213
Quoted 7 lines
drivers/media/platform/rockchip/iep2/iep2.c new file +7
-0+if (iommu_get_domain_for_dev (dev)) { + iep->empty_domain =iommu_paging_domain_alloc (dev); +if (IS_ERR (iep->empty_domain)) { +dev_warn (dev,"cannot allocate empty IOMMU domain\n" ); + iep->empty_domain =NULL ; + } + }
[Severity: Medium]
Should iep_probe() fail when iommu_paging_domain_alloc() returns an error?
Setting iep->empty_domain to NULL causes iep_iommu_restore() to return 0 after a hardware reset even though an IOMMU is attached and left in bypass mode. iep_reset() then clears iep->iommu_lost, skipping the synchronous runtime PM suspend in iep_pm_put() and the iep->iommu_lost check in iep_device_run(), so later jobs run DMA with IOVA addresses while the IOMMU is in bypass.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
PATCH v3 3/3 arm64: dts: rockchip: Add IEP2 deinterlacer to RK3588
In reply to Arun IyengarAdd the IEP2 deinterlacer and its IOMMU to the RK3588 base dtsi. Both share GIC SPI 117 and sit in the VDPU power domain.
--- arch/arm64/boot/dts/rockchip/rk3588-base.dtsi | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
arch/arm64/boot/dts/rockchip/rk3588-base.dtsi +22 -0
@@ -1397,6 +1397,28 @@ vepu121_3_mmu: iommu@fdbac800 { #iommu-cells = <0>; }; + iep: deinterlace@fdbb0000 { + compatible = "rockchip,rk3588-iep2", "rockchip,rk3568-iep2"; + reg = <0x0 0xfdbb0000 0x0 0x500>; + interrupts = <GIC_SPI 117 IRQ_TYPE_LEVEL_HIGH 0>; + clocks = <&cru ACLK_IEP2P0>, <&cru HCLK_IEP2P0>, <&cru CLK_IEP2P0_CORE>; + clock-names = "aclk", "hclk", "sclk"; + resets = <&cru SRST_A_IEP2P0>, <&cru SRST_H_IEP2P0>, <&cru SRST_IEP2P0_CORE>; + reset-names = "axi", "ahb", "core"; + power-domains = <&power RK3588_PD_VDPU>; + iommus = <&iep_mmu>; + }; + + iep_mmu: iommu@fdbb0800 { + compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu"; + reg = <0x0 0xfdbb0800 0x0 0x100>; + interrupts = <GIC_SPI 117 IRQ_TYPE_LEVEL_HIGH 0>; + clocks = <&cru ACLK_IEP2P0>, <&cru HCLK_IEP2P0>; + clock-names = "aclk", "iface"; + power-domains = <&power RK3588_PD_VDPU>; + #iommu-cells = <0>; + }; + vdec0: video-codec@fdc38000 { compatible = "rockchip,rk3588-vdec"; reg = <0x0 0xfdc38000 0x0 0x100>,
Signature · 1 line
2.55.0