media: rockchip: RKVENC v1 H.264/HEVC encoder for the RV1126
Subsystems (4)
cover media: rockchip: RKVENC v1 H.264/HEVC encoder for the RV1126
This RFC adds a driver for RKVENC v1, the H.264 and HEVC video encoder of the Rockchip RV1109 and RV1126 (Rockchip calls this generation "vepu541"; the RK3566 and RK3568 carry the "vepu540" variant). It is a stateful V4L2 memory-to-memory encoder: NV12 or NV12M in, H.264 or HEVC Annex B out, low-delay IDR plus P pictures with a single reference, no B frames. The driver writes the parameter sets (SPS, PPS and VPS), the hardware writes the slice headers and data. Rate control is frame level, in the driver: CBR with one QP per picture, the hardware row rate control stays off. The core clock follows the load through devfreq and the encoder registers as a cooling device. No new uAPI is added; the driver uses the existing stateful encoder controls and formats only.
The series:
1 binding (with MAINTAINERS entry) 2 RV1126 clock tree of the VEPU power domain 3 the driver 4 frequency scaling and cooling 5 RV1126 device tree node, IOMMU, power domain and QoS nodes 6 RV1126 operating points
It is an RFC because of the questions below, which decide where and in which shape this can go.
Hardware and testing:
I have tested H.264 and H.265 encoding extensively with this driver on Linux 6.18 in an RV1126 IP camera (Herospeed board, Sony IMX415 sensor) behind the RV1126 ISP, including 3856x2192 and 1280x720 H.265 at 30 fps streamed continuously over RTSP. The driver has been part of the OpenWrt rockchip/armv7 target since 2026-10-17 (target/linux/rockchip/patches-6.18/183-*), so that testing can be reproduced by anyone with the hardware.
The port to linux-next in this series is compile-tested only: gcc 16.2 and clang 23, both with W=1, ARM multi_v7_defconfig plus the RKVENC options; dt_binding_check on the binding and dtbs_check on the RV1109 and RV1126 boards in tree, all without warnings (dtschema 2026.9). Against the 6.18 code the driver differs in the cooling device registration signature of the base, kzalloc_obj(), one min_t(), a few named constants and its comments.
Design questions:
There is no stateless encoder framework upstream yet. The driver implements a stateful front end only, and keeps the hardware back end apart from it so that it can move to stateless infrastructure once that exists:
- Hardware back end: rkvenc.c (job start, interrupt, recovery, watchdog, power), rkvenc-vepu541*.c (the register image, built from struct rkvenc_frame, which carries no V4L2 types), the tuning tables, rkvenc-ref.c (reconstruction and reference buffers) and rkvenc-dvfs.c.
- Codec front end: rkvenc-gop.c (IDR cadence, forced key frames, which slot is the reference), rkvenc-h264.c and rkvenc-hevc.c (frame_num, POC, idr_pic_id and NAL type of each picture), rkvenc-nal.c (the parameter set writer) and rkvenc-rc.c (the picture QP).
- V4L2 binding: rkvenc-v4l2.c.
Under the H.264 stateless encode series Paul posted in May, userspace decides the frame type, frame_num, POC and references and passes them in V4L2_CID_STATELESS_H264_ENCODE_PARAMS, while header writing and rate control stay in the kernel as shared helpers. Applied to this driver, rkvenc-gop.c and the picture fields of rkvenc-h264.c become userspace input, rkvenc-nal.c and rkvenc-rc.c give way to the shared helpers, and the back end stays as it is: struct rkvenc_frame is already the per-picture state such a core hands to a driver. The single reference with sliding window marking in that series is what the two-slot reference handling here implements. HEVC has no posted stateless encode uAPI at the moment. Still to be done for a conversion: the GOP, reference and rate control code works on struct rkvenc_ctx, which also carries the V4L2 file handle and control state, and the parameter set code reads V4L2 control enums; both need a codec state of their own.
Questions:
1. Conformance of a stateful encoder.
The maintainer entry profile requires v4l2-compliance to pass, and dev-encoder.rst defines the interface, but there is no fluster equivalent for encoders, and Nicolas noted in January 2025 that we have no metric of how good or bad the in-tree encoders are. The plan for the next version is: v4l2-compliance -s with streaming from a file, plus every stream decoded by FFmpeg and by the H.264 JM and HEVC HM reference decoders, the decoded pictures required to be bit-identical between the decoders, PSNR reported per picture against the source, a static-scene sequence to catch encoder/decoder drift, and the bitrate of CBR runs reported against the target. Is that what you expect to see for a new encoder, is there a test corpus or tool you would rather have used, and should the results go into the cover letter or somewhere else?
2. Staging or drivers/media/platform/rockchip.
hantro and rkvdec were staged because the stateless uAPI was not stable yet, see commit fbb6c848dd89 ("media: destage Hantro VPU driver"), and allegro left staging once the stateful encoder API was final, see commit d74d4e2359ec ("media: allegro: move driver out of staging"). This driver uses only the stable stateful encoder uAPI and adds no controls, formats or ioctls of its own, so it is posted for drivers/media/platform/rockchip/rkvenc. The in-kernel rate control and the driver-written parameter sets are the parts that differ from the firmware-based stateful encoders in tree. Do either of them, or anything else, argue for staging in your view?
3. Stateful now, and the back end for the stateless work?
dev-encoder.rst strongly discourages header generation in the driver for a stateful encoder and advises the stateless encoder interface instead, and in the VEPU510 thread Nicolas asked not to offer two interfaces for the same hardware. RKVENC v1 is the generation before VEPU510/VEPU580, its main users are 32-bit camera SoCs whose userspace is GStreamer and v4l2 RTSP servers, and no other interface for it has been posted. Nicolas also asked for a second implementation of the stateless encoder spec, from another vendor than the VC8000E. Would you take the stateful driver now, with its back end offered as that second implementation and the driver converted once the spec is merged, or should RKVENC v1 only come as a stateless driver?
4. Naming. The binding is rockchip,rv1126-rkvenc.yaml with the compatibles rockchip,rv1126-rkvenc and rockchip,rk3568-rkvenc, and the driver directory rockchip/rkvenc. The RK3576 and RK3588 RFCs below used rockchip/rkvenc for a different generation of the block. Should RKVENC v1 keep that directory and those names, or take generation specific ones (for example vepu541)?
5. RK3566/RK3568. The driver matches rockchip,rk3568-rkvenc with the vepu540 variant data, but I don't have hardware to actuall test that. Keep the compatible for a later RK356x series, or drop it until someone with the hardware has done some testing?
Related postings:
- Jiaxing Hu, "[RFC PATCH 0/3] media: rockchip: VEPU510 H.264 encoder for RK3576", stateful, I frames only; the author moved to the Vulkan Video direction in the thread:
https://lore.kernel.org/all/[email protected]/
- Arun Iyengar, "[PATCH RFC 00/12] media: rockchip: Add a VEPU580 stateless H.264/HEVC encoder driver for the RK3588", withdrawn by the author on 2026-11-13:
https://lore.kernel.org/all/[email protected]/
- Paul Kocialkowski, "[PATCH 00/14] media: Add V4L2 H.264 stateless encode and VC8000E support":
https://lore.kernel.org/all/[email protected]/
Not in this series:
- Binding the encoder to the RV1126 NPU thermal zone, the zone that covers the VEPU power domain. It needs RV1126 TSADC support, which is not in mainline yet, and follows with it.
- venc-supply on the RV1126 boards in tree. Their encoder shares the vdd_npu_vepu rail with the NPU.
Provenance:
There is no public documentation of this block. The driver was written by reverse engineering the hardware, with the Rockchip MPP userspace library as a behavioural reference. A spec was derived by an LLM agent, and that spec was then used to create the driver
Tool use:
Most of this series was written with multiple LLM coding assistants:
The spec was derived by Claude Sonnet 5.5, handed to a locally run Qwen3.8-37B which carried out the initial implementation. Claude Opus 5.5 then did the initial testing and bug fixing, and pre-submission polishment under my micro-management.
The full test coverage, and long-term testing on the off-the-shelf camera was done by me.
Daniel Golle (6):
dt-bindings: media: Add Rockchip RKVENC v1 video encoder
clk: rockchip: rv1126: Add PD_VEPU clock tree
media: rockchip: Add RKVENC v1 video encoder driver
media: rockchip: rkvenc: Add frequency scaling and cooling
ARM: dts: rockchip: Add RKVENC encoder to RV1126
ARM: dts: rockchip: Add RKVENC operating points to RV1126
.../media/rockchip,rv1126-rkvenc.yaml | 104 ++ MAINTAINERS | 8 + arch/arm/boot/dts/rockchip/rv1126.dtsi | 80 ++ drivers/clk/rockchip/clk-rv1126.c | 21 + drivers/media/platform/rockchip/Kconfig | 1 + drivers/media/platform/rockchip/Makefile | 1 + .../media/platform/rockchip/rkvenc/Kconfig | 20 + .../media/platform/rockchip/rkvenc/Makefile | 16 + .../platform/rockchip/rkvenc/rkvenc-dvfs.c | 364 ++++++ .../platform/rockchip/rkvenc/rkvenc-gop.c | 72 ++ .../platform/rockchip/rkvenc/rkvenc-h264.c | 82 ++ .../platform/rockchip/rkvenc/rkvenc-hevc.c | 63 + .../platform/rockchip/rkvenc/rkvenc-nal.c | 351 ++++++ .../platform/rockchip/rkvenc/rkvenc-nal.h | 12 + .../platform/rockchip/rkvenc/rkvenc-rc.c | 214 ++++ .../platform/rockchip/rkvenc/rkvenc-ref.c | 114 ++ .../platform/rockchip/rkvenc/rkvenc-v4l2.c | 1085 +++++++++++++++++ .../rockchip/rkvenc/rkvenc-vepu541-h264.c | 206 ++++ .../rockchip/rkvenc/rkvenc-vepu541-hevc.c | 182 +++ .../rockchip/rkvenc/rkvenc-vepu541-regs.h | 283 +++++ .../rockchip/rkvenc/rkvenc-vepu541-tables.c | 256 ++++ .../platform/rockchip/rkvenc/rkvenc-vepu541.c | 215 ++++ .../platform/rockchip/rkvenc/rkvenc-vepu541.h | 201 +++ .../media/platform/rockchip/rkvenc/rkvenc.c | 498 ++++++++ .../media/platform/rockchip/rkvenc/rkvenc.h | 263 ++++ 25 files changed, 4712 insertions(+) create mode 100644 Documentation/devicetree/bindings/media/rockchip,rv1126-rkvenc.yaml create mode 100644 drivers/media/platform/rockchip/rkvenc/Kconfig create mode 100644 drivers/media/platform/rockchip/rkvenc/Makefile create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-gop.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-h264.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-hevc.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-nal.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-nal.h create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-rc.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-ref.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-vepu541-h264.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-vepu541-hevc.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-vepu541-regs.h create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-vepu541-tables.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-vepu541.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-vepu541.h create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc.h
base-commit: 84abb955b1015477aa11a96e2f3ac1b938b77f48
Signature · 1 line
2.56.0
RFC PATCH 1/6 dt-bindings: media: Add Rockchip RKVENC v1 video encoder
Add the binding of the RKVENC v1 video encoder found in the RV1109 and RV1126 ("vepu541") and in the RK3566 and RK3568 ("vepu540"). The block encodes H.264 and HEVC from raw frames and reaches memory through an IOMMU.
Describe the register space, the interrupt, the AXI, AHB and core clocks and resets, the power domain and the IOMMU. The core clock is scaled between operating points with an optional supply, and the encoder is a cooling device of the thermal zones.
--- .../media/rockchip,rv1126-rkvenc.yaml | 104 ++++++++++++++++++ MAINTAINERS | 7 ++ 2 files changed, 111 insertions(+) create mode 100644 Documentation/devicetree/bindings/media/rockchip,rv1126-rkvenc.yaml
Documentation/devicetree/bindings/media/rockchip,rv1126-rkvenc.yaml new file +104 -0
@@ -0,0 +1,104 @@ +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/media/rockchip,rv1126-rkvenc.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Rockchip RKVENC v1 H.264/HEVC video encoder + +maintainers: + - Daniel Golle <[email protected]> + +description: + The RKVENC v1 encoder found in the RV1109, RV1126, RK3566 and RK3568 + encodes H.264 and HEVC from raw YUV or RGB frames. It reads and writes + memory through an IOMMU with two MMU instances. + +properties: + compatible: + enum: + - rockchip,rk3568-rkvenc + - rockchip,rv1126-rkvenc + + reg: + maxItems: 1 + + interrupts: + maxItems: 1 + + clocks: + items: + - description: AXI master clock + - description: AHB register interface clock + - description: Encoder core clock + + clock-names: + items: + - const: axi + - const: ahb + - const: core + + assigned-clocks: true + + assigned-clock-rates: true + + operating-points-v2: true + + venc-supply: + description: Supply of the encoder core, scaled with the operating points. + + "#cooling-cells": + const: 2 + + resets: + items: + - description: AXI reset + - description: AHB reset + - description: Encoder core reset + + reset-names: + items: + - const: axi + - const: ahb + - const: core + + power-domains: + maxItems: 1 + + iommus: + maxItems: 1 + +required: + - compatible + - reg + - interrupts + - clocks + - clock-names + - resets + - reset-names + - power-domains + - iommus + +additionalProperties: false + +examples: + - | + #include <dt-bindings/clock/rockchip,rv1126-cru.h> + #include <dt-bindings/interrupt-controller/arm-gic.h> + #include <dt-bindings/power/rockchip,rv1126-power.h> + + video-codec@ffbb0000 { + compatible = "rockchip,rv1126-rkvenc"; + reg = <0xffbb0000 0xf00>; + interrupts = <GIC_SPI 68 IRQ_TYPE_LEVEL_HIGH>; + clocks = <&cru ACLK_VENC>, <&cru HCLK_VENC>, <&cru CLK_VENC_CORE>; + clock-names = "axi", "ahb", "core"; + assigned-clocks = <&cru ACLK_VENC>, <&cru CLK_VENC_CORE>; + assigned-clock-rates = <297000000>, <396000000>; + resets = <&cru SRST_VENC_A>, <&cru SRST_VENC_H>, + <&cru SRST_VENC_CORE>; + reset-names = "axi", "ahb", "core"; + iommus = <&venc_mmu>; + power-domains = <&power RV1126_PD_VEPU>; + }; +...
MAINTAINERS +7 -0
@@ -24544,6 +24544,13 @@ S: Maintained F: Documentation/devicetree/bindings/media/rockchip,vdec.yaml F: drivers/media/platform/rockchip/rkvdec/ +ROCKCHIP RKVENC VIDEO ENCODER DRIVER +M: Daniel Golle <[email protected]> +L: [email protected] +L: [email protected] +S: Maintained +F: Documentation/devicetree/bindings/media/rockchip,rv1126-rkvenc.yaml + ROCKCHIP RK3308 INTERNAL AUDIO CODEC M: Luca Ceresoli <[email protected]> S: Maintained
Signature · 1 line
2.56.0
RFC PATCH 2/6 clk: rockchip: rv1126: Add PD_VEPU clock tree
In reply to Daniel GolleThe video encoder power domain of the RV1126 contains the RKVENC encoder. Add its AXI and AHB bus clocks, the AXI and AHB clocks of the encoder and the encoder core clock, together with the two interconnect gates that stay enabled and the two parent selections of the bus and core clocks.
--- drivers/clk/rockchip/clk-rv1126.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+)
drivers/clk/rockchip/clk-rv1126.c +21 -0
@@ -176,6 +176,8 @@ PNAME(mux_i2s2_out2io_p) = { "mclk_i2s2", "xin12m" };PNAME (mux_gpll_cpll_xin24m_p) = {"gpll" ,"cpll" ,"xin24m" };PNAME (mux_audpwm_p) = {"sclk_audpwm_div" ,"sclk_audpwm_fracdiv" ,"xin24m" };PNAME (mux_dclk_vop_p) = {"dclk_vop_div" ,"dclk_vop_fracdiv" ,"xin24m" }; +PNAME (mux_cpll_gpll_hpll_p) = {"cpll" ,"gpll" ,"hpll" }; +PNAME (mux_cpll_hpll_gpll_p) = {"cpll" ,"hpll" ,"gpll" };PNAME (mux_usb480m_gpll_p) = {"usb480m" ,"gpll" };PNAME (clk_gmac_src_m0_p) = {"clk_gmac_div" ,"clk_gmac_rgmii_m0" };PNAME (clk_gmac_src_m1_p) = {"clk_gmac_div" ,"clk_gmac_rgmii_m1" };
@@ -719,6 +721,25 @@ static struct rockchip_clk_branch rv1126_clk_branches[] __initdata = { RV1126_CLKSEL_CON(72), 8, 1, MFLAGS, 0, 7, DFLAGS, RV1126_CLKGATE_CON(11), 1, GFLAGS), +/* PD_VEPU */ +COMPOSITE (ACLK_PDVEPU ,"aclk_pdvepu" , mux_cpll_hpll_gpll_p,0 , +RV1126_CLKSEL_CON (40 ),6 ,2 ,MFLAGS ,0 ,5 ,DFLAGS , +RV1126_CLKGATE_CON (12 ),0 ,GFLAGS ), +COMPOSITE_NOMUX (HCLK_PDVEPU ,"hclk_pdvepu" ,"aclk_pdvepu" ,0 , +RV1126_CLKSEL_CON (41 ),0 ,5 ,DFLAGS , +RV1126_CLKGATE_CON (12 ),2 ,GFLAGS ), +GATE (ACLK_VENC ,"aclk_venc" ,"aclk_pdvepu" ,0 , +RV1126_CLKGATE_CON (12 ),5 ,GFLAGS ), +GATE (HCLK_VENC ,"hclk_venc" ,"hclk_pdvepu" ,0 , +RV1126_CLKGATE_CON (12 ),6 ,GFLAGS ), +COMPOSITE (CLK_VENC_CORE ,"clk_venc_core" , mux_cpll_gpll_hpll_p,0 , +RV1126_CLKSEL_CON (40 ),14 ,2 ,MFLAGS ,8 ,5 ,DFLAGS , +RV1126_CLKGATE_CON (12 ),1 ,GFLAGS ), +GATE (0 ,"aclk_pdvepu_niu" ,"aclk_pdvepu" ,CLK_IGNORE_UNUSED , +RV1126_CLKGATE_CON (12 ),3 ,GFLAGS ), +GATE (0 ,"hclk_pdvepu_niu" ,"hclk_pdvepu" ,CLK_IGNORE_UNUSED , +RV1126_CLKGATE_CON (12 ),4 ,GFLAGS ), +/* * Clock-Architecture Diagram 9 */
Signature · 1 line
2.56.0
RFC PATCH 3/6 media: rockchip: Add RKVENC v1 video encoder driver
In reply to Daniel GolleAdd a stateful V4L2 memory-to-memory encoder driver for the RKVENC v1 H.264 and HEVC encoder of the RV1109/RV1126 ("vepu541") and the RK3566/RK3568 ("vepu540").
The driver takes NV12 or NV12M and produces low-delay streams of IDR and P pictures with one short-term reference, POC type 0 and no B frames. It writes the SPS, PPS and VPS itself, the hardware writes the slice headers and data. The SEPARATE and JOINED_WITH_1ST_FRAME header modes are supported, the latter also with PREPEND_SPSPPS_TO_IDR. Pictures too small for P frames are coded as IDR pictures only.
The register image builder takes a struct rkvenc_frame with no V4L2 types, so that a stateless front end can drive the same hardware back end; rkvenc-v4l2.c is the stateful front end on the mem2mem framework.
Rate control is frame level CBR: the hardware row rate control stays off and every picture is coded at one QP. An I picture takes the mean P QP of the previous GOP lowered by 4 for HEVC and 2 for H.264, the P QP moves by at most one per picture, and the deviation from the plan is taken at GOP boundaries and spread over the next GOP. A bitstream overflow is fed to the rate control and the picture is encoded once more. A failed job is stopped with a force clear, and with a safe clear and a block reset only when that does not stop it.
Source frames are imported without a kernel mapping and a capture buffer is sized to half the pixel count, so that 4K streams fit the vmalloc space of 32-bit systems.
--- MAINTAINERS | 1 + drivers/media/platform/rockchip/Kconfig | 1 + drivers/media/platform/rockchip/Makefile | 1 + .../media/platform/rockchip/rkvenc/Kconfig | 17 + .../media/platform/rockchip/rkvenc/Makefile | 15 + .../platform/rockchip/rkvenc/rkvenc-gop.c | 72 ++ .../platform/rockchip/rkvenc/rkvenc-h264.c | 82 ++ .../platform/rockchip/rkvenc/rkvenc-hevc.c | 63 + .../platform/rockchip/rkvenc/rkvenc-nal.c | 351 ++++++ .../platform/rockchip/rkvenc/rkvenc-nal.h | 12 + .../platform/rockchip/rkvenc/rkvenc-rc.c | 214 ++++ .../platform/rockchip/rkvenc/rkvenc-ref.c | 114 ++ .../platform/rockchip/rkvenc/rkvenc-v4l2.c | 1069 +++++++++++++++++ .../rockchip/rkvenc/rkvenc-vepu541-h264.c | 206 ++++ .../rockchip/rkvenc/rkvenc-vepu541-hevc.c | 182 +++ .../rockchip/rkvenc/rkvenc-vepu541-regs.h | 283 +++++ .../rockchip/rkvenc/rkvenc-vepu541-tables.c | 256 ++++ .../platform/rockchip/rkvenc/rkvenc-vepu541.c | 215 ++++ .../platform/rockchip/rkvenc/rkvenc-vepu541.h | 201 ++++ .../media/platform/rockchip/rkvenc/rkvenc.c | 489 ++++++++ .../media/platform/rockchip/rkvenc/rkvenc.h | 216 ++++ 21 files changed, 4060 insertions(+) create mode 100644 drivers/media/platform/rockchip/rkvenc/Kconfig create mode 100644 drivers/media/platform/rockchip/rkvenc/Makefile create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-gop.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-h264.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-hevc.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-nal.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-nal.h create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-rc.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-ref.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-vepu541-h264.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-vepu541-hevc.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-vepu541-regs.h create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-vepu541-tables.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-vepu541.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-vepu541.h create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc.c create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc.h
MAINTAINERS +1 -0
@@ -24550,6 +24550,7 @@ L: [email protected] L: [email protected] S: Maintained F: Documentation/devicetree/bindings/media/rockchip,rv1126-rkvenc.yaml +F: drivers/media/platform/rockchip/rkvenc/ ROCKCHIP RK3308 INTERNAL AUDIO CODEC M: Luca Ceresoli <[email protected]>
drivers/media/platform/rockchip/Kconfig +1 -0
@@ -6,3 +6,4 @@ source "drivers/media/platform/rockchip/rga/Kconfig" source "drivers/media/platform/rockchip/rkcif/Kconfig" source "drivers/media/platform/rockchip/rkisp1/Kconfig" source "drivers/media/platform/rockchip/rkvdec/Kconfig" +source "drivers/media/platform/rockchip/rkvenc/Kconfig"
drivers/media/platform/rockchip/Makefile +1 -0
@@ -3,3 +3,4 @@ obj-y += rga/ obj-y += rkcif/ obj-y += rkisp1/ obj-y += rkvdec/ +obj-y += rkvenc/
drivers/media/platform/rockchip/rkvenc/Kconfig new file +17 -0
@@ -0,0 +1,17 @@ +# SPDX-License-Identifier: GPL-2.0-only +config VIDEO_ROCKCHIP_RKVENC + tristate "Rockchip RKVENC (vepu540/vepu541) H.264/HEVC encoder" + depends on ARCH_ROCKCHIP || COMPILE_TEST + depends on VIDEO_DEV + select MEDIA_CONTROLLER + select VIDEOBUF2_DMA_CONTIG + select V4L2_MEM2MEM_DEV + help + Support for the Rockchip RKVENC v1 hardware video encoder found + in the RV1109/RV1126 (vepu541) and RK3566/RK3568 (vepu540) SoCs. + + It encodes NV12 raw frames into H.264 (AVC) or HEVC (H.265) + bitstreams using a stateful memory-to-memory V4L2 interface. + + To compile this driver as a module, choose M here: the module + will be called rockchip-rkvenc.
drivers/media/platform/rockchip/rkvenc/Makefile new file +15 -0
@@ -0,0 +1,15 @@ +# SPDX-License-Identifier: GPL-2.0-only +rockchip-rkvenc-y += rkvenc.o \ + rkvenc-v4l2.o \ + rkvenc-gop.o \ + rkvenc-ref.o \ + rkvenc-rc.o \ + rkvenc-h264.o \ + rkvenc-hevc.o \ + rkvenc-nal.o \ + rkvenc-vepu541.o \ + rkvenc-vepu541-h264.o \ + rkvenc-vepu541-hevc.o \ + rkvenc-vepu541-tables.o + +obj-$(CONFIG_VIDEO_ROCKCHIP_RKVENC) += rockchip-rkvenc.o
drivers/media/platform/rockchip/rkvenc/rkvenc-gop.c new file +72 -0
72 lines not on this page · raw message
drivers/media/platform/rockchip/rkvenc/rkvenc-h264.c new file +82 -0
82 lines not on this page · raw message
drivers/media/platform/rockchip/rkvenc/rkvenc-hevc.c new file +63 -0
63 lines not on this page · raw message
drivers/media/platform/rockchip/rkvenc/rkvenc-nal.c new file +351 -0
351 lines not on this page · raw message
drivers/media/platform/rockchip/rkvenc/rkvenc-nal.h new file +12 -0
@@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef RKVENC_NAL_H +#define RKVENC_NAL_H + +#include "rkvenc-vepu541.h" + +int rkvenc_h264_headers_write (const struct rkvenc_h264_seq *seq,u8 *buf, +u32 size); +int rkvenc_hevc_headers_write (const struct rkvenc_hevc_seq *seq,u8 *buf, +u32 size); + +#endif
drivers/media/platform/rockchip/rkvenc/rkvenc-rc.c new file +214 -0
214 lines not on this page · raw message
drivers/media/platform/rockchip/rkvenc/rkvenc-ref.c new file +114 -0
114 lines not on this page · raw message
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +1069 -0
1,069 lines not on this page · raw message
drivers/media/platform/rockchip/rkvenc/rkvenc-vepu541-h264.c new file +206 -0
206 lines not on this page · raw message
drivers/media/platform/rockchip/rkvenc/rkvenc-vepu541-hevc.c new file +182 -0
182 lines not on this page · raw message
drivers/media/platform/rockchip/rkvenc/rkvenc-vepu541-regs.h new file +283 -0
283 lines not on this page · raw message
drivers/media/platform/rockchip/rkvenc/rkvenc-vepu541-tables.c new file +256 -0
256 lines not on this page · raw message
drivers/media/platform/rockchip/rkvenc/rkvenc-vepu541.c new file +215 -0
49 lines not on this page · raw message
drivers/media/platform/rockchip/rkvenc/rkvenc-vepu541.h new file +201 -0
201 lines not on this page · raw message
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +489 -0
489 lines not on this page · raw message
drivers/media/platform/rockchip/rkvenc/rkvenc.h new file +216 -0
216 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.56.0
Part of the diff is not shown here. View the raw message.
RFC PATCH 4/6 media: rockchip: rkvenc: Add frequency scaling and cooling
In reply to Daniel GolleThe encoder core runs at the fixed 396 MHz of the clock tree setup, which is a little short of a 4K stream at 30 frames per second.
Register the core clock with devfreq. The simple_ondemand governor takes the busy time of the jobs as load and runs once per round of the streaming contexts: a job end runs it when the context that opened the round completes again, so the load covers one job of every context and the clock settles at the lowest point that sustains all of them together. No polling interval is involved. The clock changes only while no job is on the hardware: a job start waits for a clock change in progress and a clock change waits for the running job. A venc-supply in the node is scaled with the operating points.
Register a cooling device on top. Its first states lower the highest allowed operating point through a PM QoS frequency limit, one point per state. The remaining states stretch the period between jobs so that the busy time stays at 75, 50 and 25 percent of it, which limits the frame rate once the lowest point is reached. The cooling device, the QoS request and the devfreq device are removed from remove(), before the driver data goes away, since their callbacks use it.
--- .../media/platform/rockchip/rkvenc/Kconfig | 3 + .../media/platform/rockchip/rkvenc/Makefile | 1 + .../platform/rockchip/rkvenc/rkvenc-dvfs.c | 364 ++++++++++++++++++ .../platform/rockchip/rkvenc/rkvenc-v4l2.c | 20 +- .../media/platform/rockchip/rkvenc/rkvenc.c | 11 +- .../media/platform/rockchip/rkvenc/rkvenc.h | 47 +++ 6 files changed, 443 insertions(+), 3 deletions(-) create mode 100644 drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c
drivers/media/platform/rockchip/rkvenc/Kconfig +3 -0
@@ -3,7 +3,10 @@ config VIDEO_ROCKCHIP_RKVENC tristate "Rockchip RKVENC (vepu540/vepu541) H.264/HEVC encoder" depends on ARCH_ROCKCHIP || COMPILE_TEST depends on VIDEO_DEV + select DEVFREQ_GOV_SIMPLE_ONDEMAND select MEDIA_CONTROLLER + select PM_DEVFREQ + select PM_OPP select VIDEOBUF2_DMA_CONTIG select V4L2_MEM2MEM_DEV help
drivers/media/platform/rockchip/rkvenc/Makefile +1 -0
@@ -1,5 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only rockchip-rkvenc-y += rkvenc.o \ + rkvenc-dvfs.o \ rkvenc-v4l2.o \ rkvenc-gop.o \ rkvenc-ref.o \
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c new file +364 -0
364 lines not on this page · raw message
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c +18 -2
@@ -351,6 +351,7 @@ static void rkvenc_stop_streaming(struct vb2_queue *q)struct vb2_v4l2_buffer *vbuf;v4l2_m2m_update_stop_streaming_state (ctx->m2m_ctx, q); +rkvenc_dvfs_ctx_idle (ctx->dev, ctx);if (V4L2_TYPE_IS_OUTPUT (q->type)) {while ((vbuf =v4l2_m2m_src_buf_remove (ctx->m2m_ctx)))
@@ -457,17 +458,22 @@ static void rkvenc_device_run(void *priv)else rkvenc_hevc_build_pic (ctx, &f.hevc, idr); +rkvenc_dvfs_job_begin (dev); + ret =pm_runtime_resume_and_get (dev->dev);if (ret) - goto finish_error; + goto finish_error_idle; ret =rkvenc_hw_run (dev, &f);if (ret) {pm_runtime_put (dev->dev); - goto finish_error; + goto finish_error_idle; }return ; +finish_error_idle: +rkvenc_dvfs_job_end (dev, ctx); + finish_error: dev->job_running = false; dev->job_ctx =NULL ;
@@ -478,8 +484,16 @@ static void rkvenc_device_run(void *priv)v4l2_m2m_job_finish (dev->m2m_dev, ctx->m2m_ctx); } +static int rkvenc_job_ready (void *priv) +{ +struct rkvenc_ctx *ctx = priv; + +return rkvenc_dvfs_start_allowed (ctx->dev); +} +const struct v4l2_m2m_ops rkvenc_m2m_ops = { .device_run = rkvenc_device_run, + .job_ready = rkvenc_job_ready, };static int rkvenc_s_ctrl (struct v4l2_ctrl *ctrl)
@@ -944,6 +958,7 @@ static int rkvenc_open(struct file *filp)if (ret) goto err_m2m; +rkvenc_dvfs_ctx_add (dev, ctx);v4l2_fh_add (&ctx->fh, filp);return 0 ;
@@ -959,6 +974,7 @@ static int rkvenc_release(struct file *filp) {struct rkvenc_ctx *ctx =fh_to_rkvenc_ctx (filp->private_data); +rkvenc_dvfs_ctx_del (ctx->dev, ctx);v4l2_m2m_ctx_release (ctx->m2m_ctx);v4l2_ctrl_handler_free (&ctx->ctrl_hdl);v4l2_fh_del (&ctx->fh, filp);
drivers/media/platform/rockchip/rkvenc/rkvenc.c +10 -1
@@ -187,6 +187,7 @@ static void rkvenc_finish_job(struct rkvenc_dev *dev, struct rkvenc_ctx *ctx, ctx->joined_header_done = false; dev->job_ctx =NULL ;pm_runtime_put (dev->dev); +rkvenc_dvfs_job_end (dev, ctx); rkvenc_m2m_ops.device_run (ctx);return ; }else {
@@ -210,6 +211,7 @@ static void rkvenc_finish_job(struct rkvenc_dev *dev, struct rkvenc_ctx *ctx, dev->job_ctx =NULL ;pm_runtime_put (dev->dev); +rkvenc_dvfs_job_end (dev, ctx);v4l2_m2m_buf_done_and_job_finish (dev->m2m_dev, ctx->m2m_ctx, state); }
@@ -422,12 +424,18 @@ static int rkvenc_probe(struct platform_device *pdev) goto err_unreg_v4l2; } - ret =rkvenc_v4l2_init (rkvenc); + ret =rkvenc_dvfs_init (rkvenc);if (ret) goto err_release_m2m; + ret =rkvenc_v4l2_init (rkvenc); +if (ret) + goto err_dvfs; +return 0 ; +err_dvfs: +rkvenc_dvfs_exit (rkvenc); err_release_m2m:v4l2_m2m_release (rkvenc->m2m_dev); err_unreg_v4l2:
@@ -442,6 +450,7 @@ static void rkvenc_remove(struct platform_device *pdev)struct rkvenc_dev *rkvenc =platform_get_drvdata (pdev);cancel_delayed_work_sync (&rkvenc->watchdog); +rkvenc_dvfs_exit (rkvenc);rkvenc_v4l2_cleanup (rkvenc);v4l2_m2m_release (rkvenc->m2m_dev);v4l2_device_unregister (&rkvenc->v4l2_dev);
drivers/media/platform/rockchip/rkvenc/rkvenc.h +47 -0
84 lines not on this page · raw message
Signature · 1 line
2.56.0
RFC PATCH 5/6 ARM: dts: rockchip: Add RKVENC encoder to RV1126
In reply to Daniel GolleAdd the encoder with its IOMMU, the VEPU power domain and the three QoS nodes of the encoder ports. The encoder AXI clock is set to 297 MHz and the core clock to 396 MHz.
--- arch/arm/boot/dts/rockchip/rv1126.dtsi | 54 ++++++++++++++++++++++++++ 1 file changed, 54 insertions(+)
arch/arm/boot/dts/rockchip/rv1126.dtsi +54 -0
@@ -139,6 +139,21 @@ qos_sdio: qos@fe86c000 { reg = <0xfe86c000 0x20>; }; + qos_venc_rd_m0: qos@fe870000 { + compatible = "rockchip,rv1126-qos", "syscon"; + reg = <0xfe870000 0x20>; + }; + + qos_venc_rd_m1: qos@fe870080 { + compatible = "rockchip,rv1126-qos", "syscon"; + reg = <0xfe870080 0x20>; + }; + + qos_venc_wr: qos@fe870100 { + compatible = "rockchip,rv1126-qos", "syscon"; + reg = <0xfe870100 0x20>; + }; + qos_iep: qos@fe8a0000 { compatible = "rockchip,rv1126-qos", "syscon"; reg = <0xfe8a0000 0x20>;
@@ -223,6 +238,17 @@ power-domain@RV1126_PD_VO { <&qos_iep>; #power-domain-cells = <0>; }; + + power-domain@RV1126_PD_VEPU { + reg = <RV1126_PD_VEPU>; + clocks = <&cru ACLK_VENC>, + <&cru HCLK_VENC>, + <&cru CLK_VENC_CORE>; + pm_qos = <&qos_venc_rd_m0>, + <&qos_venc_rd_m1>, + <&qos_venc_wr>; + #power-domain-cells = <0>; + }; }; };
@@ -630,6 +656,34 @@ vop_mmu: iommu@ffb00f00 { status = "disabled"; }; + venc: video-codec@ffbb0000 { + compatible = "rockchip,rv1126-rkvenc"; + reg = <0xffbb0000 0xf00>; + interrupts = <GIC_SPI 68 IRQ_TYPE_LEVEL_HIGH>; + clocks = <&cru ACLK_VENC>, <&cru HCLK_VENC>, + <&cru CLK_VENC_CORE>; + clock-names = "axi", "ahb", "core"; + assigned-clocks = <&cru ACLK_VENC>, <&cru CLK_VENC_CORE>; + assigned-clock-rates = <297000000>, <396000000>; + resets = <&cru SRST_VENC_A>, <&cru SRST_VENC_H>, + <&cru SRST_VENC_CORE>; + reset-names = "axi", "ahb", "core"; + iommus = <&venc_mmu>; + power-domains = <&power RV1126_PD_VEPU>; + }; + + venc_mmu: iommu@ffbb0f00 { + compatible = "rockchip,iommu"; + reg = <0xffbb0f00 0x40>, <0xffbb0f40 0x40>; + interrupts = <GIC_SPI 69 IRQ_TYPE_LEVEL_HIGH>, + <GIC_SPI 70 IRQ_TYPE_LEVEL_HIGH>; + clocks = <&cru ACLK_VENC>, <&cru HCLK_VENC>; + clock-names = "aclk", "iface"; + #iommu-cells = <0>; + power-domains = <&power RV1126_PD_VEPU>; + rockchip,disable-mmu-reset; + }; + gmac: ethernet@ffc40000 { compatible = "rockchip,rv1126-gmac", "snps,dwmac-4.20a"; reg = <0xffc40000 0x4000>;
Signature · 1 line
2.56.0
RFC PATCH 6/6 ARM: dts: rockchip: Add RKVENC operating points to RV1126
In reply to Daniel GolleAdd the four core clock rates of the encoder, 297, 396, 500 and 594 MHz, which the clock tree provides as GPLL/4, GPLL/3, CPLL and GPLL/2, with the voltages of 725, 725, 750 and 825 mV that the vendor device tree gives them. The voltages apply on boards that describe the supply of the encoder as venc-supply. Make the encoder a cooling device so that thermal zones can use it.
--- arch/arm/boot/dts/rockchip/rv1126.dtsi | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
arch/arm/boot/dts/rockchip/rv1126.dtsi +26 -0
@@ -104,6 +104,30 @@ xin24m: oscillator { #clock-cells = <0>; }; + venc_opp_table: opp-table-venc { + compatible = "operating-points-v2"; + + opp-297000000 { + opp-hz = /bits/ 64 <297000000>; + opp-microvolt = <725000 725000 1000000>; + }; + + opp-396000000 { + opp-hz = /bits/ 64 <396000000>; + opp-microvolt = <725000 725000 1000000>; + }; + + opp-500000000 { + opp-hz = /bits/ 64 <500000000>; + opp-microvolt = <750000 750000 1000000>; + }; + + opp-594000000 { + opp-hz = /bits/ 64 <594000000>; + opp-microvolt = <825000 825000 1000000>; + }; + }; + grf: syscon@fe000000 { compatible = "rockchip,rv1126-grf", "syscon", "simple-mfd"; reg = <0xfe000000 0x20000>;
@@ -665,6 +689,8 @@ venc: video-codec@ffbb0000 { clock-names = "axi", "ahb", "core"; assigned-clocks = <&cru ACLK_VENC>, <&cru CLK_VENC_CORE>; assigned-clock-rates = <297000000>, <396000000>; + operating-points-v2 = <&venc_opp_table>; + #cooling-cells = <2>; resets = <&cru SRST_VENC_A>, <&cru SRST_VENC_H>, <&cru SRST_VENC_CORE>; reset-names = "axi", "ahb", "core";
Signature · 1 line
2.56.0
[email protected]
AI reviewRe: [RFC PATCH 6/6] ARM: dts: rockchip: Add RKVENC operating points to RV1126
Thank you for your contribution! Sashiko AI review found 6 potential issues to consider: - [High] media: rkvenc: probe fails in rkvenc_dvfs_cooling_init() without THERMAL_OF - [High] media: rkvenc…
Thank you for your contribution! Sashiko AI review found 6 potential issues to consider:
- [High] media: rkvenc: probe fails in rkvenc_dvfs_cooling_init() without THERMAL_OF
- [High] media: rkvenc: teardown race in rkvenc_remove() and UAF in rkvenc_release()
- [High] media: rkvenc: SCLR_DONE IRQ and unlocked job_running race in rkvenc
- [High] media: rkvenc: slot DMA buffer overflow via rkvenc_s_fmt_vid_out()
- [Medium] media: rkvenc: missing initial OPP and PM lifecycle for venc-supply
- [Medium] media: rkvenc: unbalanced pm_runtime_enable() in rkvenc_probe()
--
commit f41b6ddf3d36774f6ef1b61d948fc7c27e1896c8
Author: Daniel Golle <[email protected]>
ARM: dts: rockchip: Add RKVENC operating points to RV1126
This commit adds the operating points table for the RKVENC video encoder on RV1126 and registers the encoder node as a cooling device.
Quoted 5 lines
arch/arm/boot/dts/rockchip/rv1126.dtsi +0
-0
arch/arm/boot/dts/rockchip/rv1126.dtsi xin24m: oscillator { · line 113
Quoted 10 lines
arch/arm/boot/dts/rockchip/rv1126.dtsi +7
-0@@ -104,6 +104,30 @@ xin24m: oscillator { #clock-cells = <0>; }; + venc_opp_table: opp-table-venc { + compatible = "operating-points-v2"; + + opp-297000000 { + opp-hz = /bits/ 64 <297000000>; + opp-microvolt = <725000 725000 1000000>; + };
[Severity: Medium]
On boards that define venc-supply, does rkvenc_dvfs_init() ever apply the initial OPP voltage or disable venc-supply across runtime suspend?
In commit "media: rockchip: rkvenc: Add frequency scaling and cooling", devm_pm_opp_set_config() acquires venc-supply without enabling it or setting its voltage, and devfreq_add_device() does not run an initial update_devfreq() because polling_ms is 0:
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c:rkvenc_dvfs_init() {
...
if (of_property_present(dev->of_node, "venc-supply"))
opp_config.regulator_names = supply_names;
ret = devm_pm_opp_set_config(dev, &opp_config);
if (ret)
return ret;
ret = devm_pm_opp_of_add_table(dev);
...
}As a result, the first batch of encode jobs runs at the 396 MHz assigned clock rate before dev_pm_opp_set_rate() has ever configured or enabled venc-supply. Once rkvenc_dvfs_job_end() later triggers dev_pm_opp_set_rate(), rkvenc_runtime_suspend() only disables clocks without calling dev_pm_opp_set_rate(dev, 0), leaving venc-supply enabled at the last active OPP voltage (up to 825 mV) while idle.
[ ... ]
arch/arm/boot/dts/rockchip/rv1126.dtsi venc: video-codec@ffbb0000 { · line 693
Quoted 6 lines
arch/arm/boot/dts/rockchip/rv1126.dtsi +2
-0@@ -665,6 +689,8 @@ venc: video-codec@ffbb0000 { clock-names = "axi", "ahb", "core"; assigned-clocks = <&cru ACLK_VENC>, <&cru CLK_VENC_CORE>; assigned-clock-rates = <297000000>, <396000000>; + operating-points-v2 = <&venc_opp_table>; + #cooling-cells = <2>;
[Severity: High]
Does adding #cooling-cells = <2> cause rkvenc_probe() to fail with -ENODEV when CONFIG_THERMAL_OF is disabled?
VIDEO_ROCKCHIP_RKVENC does not depend on or select THERMAL or THERMAL_OF, and thermal_of_cooling_device_register() returns ERR_PTR(-ENODEV) when CONFIG_THERMAL_OF is disabled, which rkvenc_dvfs_cooling_init() treats as a fatal probe error:
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c:rkvenc_dvfs_cooling_init() {
if (!of_property_present(dev->of_node, "#cooling-cells"))
return 0;
...
dvfs->cdev = thermal_of_cooling_device_register(
dev->of_node, 0, "rkvenc", dvfs, &rkvenc_cdev_ops);
if (IS_ERR(dvfs->cdev)) {
ret = PTR_ERR(dvfs->cdev);
dvfs->cdev = NULL;
return dev_err_probe(dev, ret,
"could not add the cooling device\n");
}
}[Severity: Medium]
In commit "media: rockchip: Add RKVENC v1 video encoder driver", does rkvenc_probe() leave runtime PM enabled if pm_runtime_resume_and_get() fails?
drivers/media/platform/rockchip/rkvenc/rkvenc.c:rkvenc_probe() {
...
pm_runtime_enable(dev);
ret = pm_runtime_resume_and_get(dev);
if (ret)
return ret;
...
err_pm_disable:
pm_runtime_disable(dev);
return ret;
}Returning directly instead of jumping to err_pm_disable leaves dev->power.disable_depth unbalanced.
[Severity: High]
In commits "media: rockchip: Add RKVENC v1 video encoder driver" and "media: rockchip: rkvenc: Add frequency scaling and cooling", can rkvenc_remove() and rkvenc_release() trigger a use-after-free on dvfs->devfreq, rkvenc_dev, and rkvenc->m2m_dev?
drivers/media/platform/rockchip/rkvenc/rkvenc.c:rkvenc_remove() {
struct rkvenc_dev *rkvenc = platform_get_drvdata(pdev);
cancel_delayed_work_sync(&rkvenc->watchdog);
rkvenc_dvfs_exit(rkvenc);
rkvenc_v4l2_cleanup(rkvenc);
v4l2_m2m_release(rkvenc->m2m_dev);
v4l2_device_unregister(&rkvenc->v4l2_dev);
pm_runtime_disable(&pdev->dev);
}rkvenc_remove() cancels the watchdog and tears down DVFS before stopping V4L2/M2M:
CPU0 CPU1rkvenc_remove()
rkvenc_dvfs_exit()
rkvenc_irq_thread()
rkvenc_finish_job()
rkvenc_dvfs_job_end()
devfreq = dvfs->devfreq
devfreq_remove_device(devfreq)
mutex_lock(&devfreq->lock)
<-- use-after-freeIn addition, active jobs during rkvenc_dvfs_exit() can re-queue dvfs->throttle_work and rkvenc->watchdog after cancel_delayed_work_sync(). Finally, because struct rkvenc_dev is allocated with devm_kzalloc() and rkvenc->m2m_dev is freed in rkvenc_remove(), closing an open V4L2 file descriptor after unbind runs rkvenc_release() -> rkvenc_dvfs_ctx_del(ctx->dev, ctx) and v4l2_m2m_ctx_release(ctx->m2m_ctx) on freed ctx->dev and rkvenc->m2m_dev memory.
[Severity: High]
In commit "media: rockchip: Add RKVENC v1 video encoder driver", can rkvenc_hw_recover() and unlocked updates to dev->job_running race with rkvenc_irq_thread() and rkvenc_watchdog_work()?
First, the IRQ is registered with IRQF_ONESHOT, which keeps the interrupt line masked while rkvenc_irq_thread() is running. When rkvenc_finish_job() -> rkvenc_hw_recover() waits on dev->sclr_done from inside rkvenc_irq_thread(), RKVENC_INT_SCLR_DONE cannot be delivered until rkvenc_irq_thread() returns, so wait_for_completion_timeout() always times out and the subsequent RKVENC_INT_SCLR_DONE interrupt wakes a spurious rkvenc_irq_thread().
Second, rkvenc_device_run() sets dev->job_ctx and dev->job_running = true without holding dev->irq_lock and before sleeping in rkvenc_dvfs_job_begin(), calling pm_runtime_resume_and_get(), and starting the hardware in rkvenc_hw_run():
CPU0 CPU1rkvenc_device_run()
dev->job_ctx = ctx
dev->job_running = true
rkvenc_dvfs_job_begin()
rkvenc_irq_thread() /
rkvenc_watchdog_work()
spin_lock_irqsave(&dev->irq_lock)
if (dev->job_running)
dev->job_running = false
spin_unlock_irqrestore(&dev->irq_lock)
rkvenc_finish_job()
pm_runtime_put()
v4l2_m2m_buf_done_and_job_finish() pm_runtime_resume_and_get()
rkvenc_hw_run() <-- starts DMA on completed vb2 buffers
When the spurious rkvenc_irq_thread() or a racing rkvenc_watchdog_work() (since rkvenc_irq_thread() uses non-sync cancel_delayed_work()) runs in that window, rkvenc_finish_job() reads MMIO while the device may be suspended, underflows the runtime PM usage count, and returns the vb2 buffers to userspace before rkvenc_hw_run() starts DMA on them.
[Severity: High]
In commit "media: rockchip: Add RKVENC v1 video encoder driver", can changing the OUTPUT format while CAPTURE is streaming overflow the reference frame DMA buffers?
rkvenc_start_streaming() only calls rkvenc_ref_alloc() when starting the CAPTURE queue, sizing ctx->slot[] from ctx->width and ctx->height once while ctx->slots_alloc is false. However, rkvenc_s_fmt_vid_out() only checks whether the OUTPUT queue is busy before updating ctx->width and ctx->height:
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c:rkvenc_s_fmt_vid_out() {
struct rkvenc_ctx *ctx = fh_to_rkvenc_ctx(file->private_data);
struct vb2_queue *vq = v4l2_m2m_get_vq(ctx->m2m_ctx, f->type);
if (vb2_is_busy(vq))
return -EBUSY;
rkvenc_try_fmt_vid_out(ctx, f);
...
ctx->width = f->fmt.pix_mp.width;
ctx->height = f->fmt.pix_mp.height;
}If userspace starts streaming on CAPTURE first and then increases the OUTPUT dimensions with VIDIOC_S_FMT before starting OUTPUT, rkvenc_device_run() programs the encoder with the larger resolution while DMA-writing reconstructed frames into the smaller ctx->slot[] buffers.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
[email protected]
AI reviewRe: [RFC PATCH 4/6] media: rockchip: rkvenc: Add frequency scaling and cooling
In reply to Daniel GolleThank you for your contribution! Sashiko AI review found 7 potential issues to consider: - [High] media: rockchip: rkvenc: rkvenc_dvfs_init() leaves initial OPP and venc-supply unconfigured - [High]…
Thank you for your contribution! Sashiko AI review found 7 potential issues to consider:
- [High] media: rockchip: rkvenc: rkvenc_dvfs_init() leaves initial OPP and venc-supply unconfigured
- [High] media: rkvenc: UAF and teardown race in rkvenc_remove() and rkvenc_dvfs_exit()
- [Medium] media: rockchip: rkvenc: probe fails in rkvenc_dvfs_cooling_init() without THERMAL_OF
- [Medium] media: rockchip: rkvenc: thermal duty-cycle throttling bypassed in rkvenc_dvfs_start_allowed()
- [Medium] media: rockchip: rkvenc: devfreq updates stall in rkvenc_dvfs_job_end() when round_ctx idles
- [Medium] media: rockchip: rkvenc: unreset last_eval in rkvenc_dvfs_job_end() skews first round load
- [Low] media: rockchip: rkvenc: freq / HZ_PER_KHZ truncation in rkvenc_cdev_set_state()
--
commit f718f2271f482c079204fa578dffff0091dcfc44
Author: Daniel Golle <[email protected]>
media: rockchip: rkvenc: Add frequency scaling and cooling
This commit adds devfreq-based frequency scaling for the RKVENC core clock andvenc-supply regulator, along with a thermal cooling device that caps the maximum frequency via PM QoS and throttles job execution duty cycle.
Quoted 5 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c new file +0
-0
[ ... ]
Quoted 1 line
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c new file +0
-0@@ -0,0 +1,364 @@
[ ... ]
Quoted 7 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c new file +7
-0+bool rkvenc_dvfs_start_allowed (struct rkvenc_dev *rkvenc) +{ +struct rkvenc_dvfs *dvfs = &rkvenc->dvfs; +unsigned long flags, delay; +s64 delay_us; +bool allowed; +
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c line 113
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c new file +10
-0+spin_lock_irqsave (&dvfs->lock, flags); + delay_us =ktime_to_us (ktime_sub (dvfs->next_start,ktime_get ())); + allowed = dvfs->duty_pct >=100 || delay_us <=0 ; + delay =usecs_to_jiffies (clamp_t (s64, delay_us,0 ,UINT_MAX )); +if (!allowed) +queue_delayed_work (system_wq, &dvfs->throttle_work, delay); +spin_unlock_irqrestore (&dvfs->lock, flags); + +return allowed; +}
[Severity: Medium]
Can multiple contexts bypass duty-cycle throttling when dvfs->duty_pct < 100?
v4l2_m2m only checks .job_ready() in __v4l2_m2m_try_queue() before placing a context onto m2m_dev->job_queue; once queued, v4l2_m2m_try_run() pops and runs contexts via .device_run() without calling .job_ready() again. Because dvfs->next_start is only updated when a job finishes in rkvenc_dvfs_job_end() and rkvenc_dvfs_start_allowed() does not check dvfs->busy, every context scheduled by rkvenc_dvfs_throttle_work() (or via VIDIOC_QBUF while the first job is running) sees delay_us <= 0 and enters m2m_dev->job_queue to run back-to-back without throttling.
In addition, when rkvenc_finish_job() retries a frame on bitstream overflow, it calls rkvenc_dvfs_job_end() followed directly by rkvenc_m2m_ops.device_run(ctx), bypassing rkvenc_dvfs_start_allowed() completely.
Quoted 12 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c new file +12
-0+ +void rkvenc_dvfs_job_end (struct rkvenc_dev *rkvenc,struct rkvenc_ctx *ctx) +{ +struct rkvenc_dvfs *dvfs = &rkvenc->dvfs; +struct devfreq *devfreq = dvfs->devfreq; +unsigned long flags; +bool evaluate; +ktime_t now; + +spin_lock_irqsave (&dvfs->lock, flags); + now =ktime_get (); + dvfs->busy_total =
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c line 135
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c new file +10
-0+ktime_add (dvfs->busy_total,ktime_sub (now, dvfs->busy_since)); + dvfs->next_start =ktime_add_ns ( + dvfs->job_start, +div_u64 (ktime_to_ns (ktime_sub (now, dvfs->job_start)) *100 , + dvfs->duty_pct)); + dvfs->busy = false; +/* A round ends with the context that opened it, covering every other one */ + evaluate = dvfs->round_ctx == ctx; +if (!dvfs->round_ctx) + dvfs->round_ctx = ctx;
[Severity: Medium]
Does opening a new evaluation round when dvfs->round_ctx is NULL leave a stale dvfs->last_eval from before the encoder was idle?
dvfs->last_eval is only initialized in rkvenc_dvfs_init() and updated in rkvenc_dvfs_status(). When the first job after boot or after VIDIOC_STREAMOFF sets dvfs->round_ctx = ctx, dvfs->last_eval and dvfs->busy_total are not reset, so on the next job completion rkvenc_dvfs_status() computes stat->total_time across the entire preceding idle period and drops the clock to the lowest OPP.
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c line 145
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c new file +10
-0+spin_unlock_irqrestore (&dvfs->lock, flags); +wake_up (&dvfs->idle_wq); + +if (!devfreq || !evaluate) +return ; + +mutex_lock (&devfreq->lock); +update_devfreq (devfreq); +mutex_unlock (&devfreq->lock); +}
[Severity: Medium]
What happens to frequency scaling if the context stored in dvfs->round_ctx stops submitting jobs without calling VIDIOC_STREAMOFF or closing the fd?
dvfs->round_ctx is only cleared in rkvenc_dvfs_ctx_idle() from rkvenc_stop_streaming() and rkvenc_release(). If that context finishes draining via v4l2_m2m_mark_stopped() or hits vb2_queue_error() in rkvenc_finish_job() (which then calls rkvenc_dvfs_job_end() and keeps or assigns dvfs->round_ctx = ctx), or if userspace simply pauses queueing frames on that context while other contexts keep streaming, evaluate will never be true again and update_devfreq() stops running.
[ ... ]
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c new file +10
-0+static int rkvenc_cdev_set_state (struct thermal_cooling_device *cdev, +unsigned long state) +{ +struct rkvenc_dvfs *dvfs = cdev->devdata; + unsigned long idx = min_t(unsigned long, state, dvfs->num_freqs - 1); +unsigned long freq = dvfs->freqs[dvfs->num_freqs -1 - idx]; +unsigned long flags; +int ret; + + ret =dev_pm_qos_update_request (&dvfs->max_freq_req, freq /HZ_PER_KHZ );
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c line 212
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c new file +10
-0+if (ret <0 ) +return ret; + +spin_lock_irqsave (&dvfs->lock, flags); + dvfs->duty_pct =100 -RKVENC_THROTTLE_STEP_PCT * (state - idx); +spin_unlock_irqrestore (&dvfs->lock, flags); + dvfs->cooling_state = state; + +return 0 ; +}
[Severity: Low]
Should freq / HZ_PER_KHZ use DIV_ROUND_UP(freq, HZ_PER_KHZ) here?
devfreq converts the PM QoS maximum frequency back to Hz via HZ_PER_KHZ * qos_max_freq and looks up the target OPP with DEVFREQ_FLAG_LEAST_UPPER_BOUND (floor), so truncating an OPP frequency that is not an exact multiple of 1000 Hz drops the limit below that OPP and selects the next lower one. Also, state is not bounds-checked against dvfs->num_freqs - 1 + RKVENC_THROTTLE_STEPS before computing dvfs->duty_pct.
[ ... ]
Quoted 17 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c new file +17
-0+static int rkvenc_dvfs_cooling_init (struct rkvenc_dev *rkvenc) +{ +struct rkvenc_dvfs *dvfs = &rkvenc->dvfs; +struct device *dev = rkvenc->dev; +int ret; + +if (!of_property_present (dev->of_node,"#cooling-cells" )) +return 0 ; + + ret =dev_pm_qos_add_request (dev, &dvfs->max_freq_req, +DEV_PM_QOS_MAX_FREQUENCY , +PM_QOS_MAX_FREQUENCY_DEFAULT_VALUE ); +if (ret <0 ) +return ret; + dvfs->qos_added = true; + + dvfs->cdev = thermal_of_cooling_device_register(
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c line 306
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c new file +10
-0+ dev->of_node, 0, "rkvenc", dvfs, &rkvenc_cdev_ops); +if (IS_ERR (dvfs->cdev)) { + ret =PTR_ERR (dvfs->cdev); + dvfs->cdev =NULL ; +return dev_err_probe (dev, ret, +"could not add the cooling device\n" ); + } + +return 0 ; +}
[Severity: Medium]
Does rkvenc_probe() fail with -ENODEV when CONFIG_THERMAL_OF is disabled?
VIDEO_ROCKCHIP_RKVENC does not depend on or select THERMAL or THERMAL_OF in Kconfig. When CONFIG_THERMAL_OF is disabled, the thermal_of_cooling_device_register() stub returns ERR_PTR(-ENODEV), which aborts probe on any board whose device tree node has #cooling-cells.
Quoted 3 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c new file +3
-0+ +int rkvenc_dvfs_init (struct rkvenc_dev *rkvenc) +{
[ ... ]
Quoted 14 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c new file +14
-0+if (of_property_present (dev->of_node,"venc-supply" )) + opp_config.regulator_names = supply_names; + + ret =devm_pm_opp_set_config (dev, &opp_config); +if (ret) +return ret; + + ret =devm_pm_opp_of_add_table (dev); +if (ret) +return ret; + + ret =rkvenc_dvfs_freqs_get (rkvenc); +if (ret) +return ret;
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c line 353
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-dvfs.c new file +10
-0+ + dvfs->profile.target = rkvenc_dvfs_target; + dvfs->profile.get_dev_status = rkvenc_dvfs_status; + dvfs->profile.get_cur_freq = rkvenc_dvfs_cur_freq; + dvfs->profile.freq_table = dvfs->freqs; + dvfs->profile.max_state = dvfs->num_freqs; +rkvenc_dvfs_cur_freq (dev, &dvfs->profile.initial_freq); + + devfreq =devfreq_add_device (dev, &dvfs->profile, +DEVFREQ_GOV_SIMPLE_ONDEMAND ,NULL );
[Severity: High]
Does anything enable and configure venc-supply or set an initial OPP before the first round of encoding jobs completes?
devm_pm_opp_set_config() only acquires venc-supply via regulator_get_optional(); the OPP core does not enable the regulator or configure its voltage until dev_pm_opp_set_rate() or dev_pm_opp_set_opp() is called. Because polling is disabled, devfreq_add_device() does not invoke rkvenc_dvfs_target() at registration time, so RKVENC_VERSION is read in rkvenc_probe() and the first round of jobs runs on the hardware before dev_pm_opp_set_rate() is ever called (and regulator_init_complete() can disable a boot-on venc-supply as unused). In addition, setting dvfs->profile.initial_freq directly from clk_get_rate() without rounding it to an OPP entry causes devfreq_update_status() to fail with -EINVAL on the first transition if the boot clock rate does not exactly match an OPP.
Quoted 4 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c +0
-0
[ ... ]
drivers/media/platform/rockchip/rkvenc/rkvenc.c static void rkvenc_remove(struct platform_device *pdev) · line 456
Quoted 8 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c +1
-0@@ -442,6 +450,7 @@ static void rkvenc_remove(struct platform_device *pdev)struct rkvenc_dev *rkvenc =platform_get_drvdata (pdev);cancel_delayed_work_sync (&rkvenc->watchdog); +rkvenc_dvfs_exit (rkvenc);rkvenc_v4l2_cleanup (rkvenc);v4l2_m2m_release (rkvenc->m2m_dev);v4l2_device_unregister (&rkvenc->v4l2_dev);
[Severity: High]
Can calling rkvenc_dvfs_exit() before rkvenc_v4l2_cleanup() and v4l2_m2m_release() cause a use-after-free during driver teardown?
While V4L2 contexts and an in-flight job are still active, rkvenc_dvfs_exit() frees dvfs->devfreq before clearing the pointer and cancels dvfs->throttle_work while dvfs->duty_pct can still be below 100:
CPU0 CPU1rkvenc_remove()
rkvenc_dvfs_exit()
devfreq_remove_device(dvfs->devfreq)
rkvenc_irq_thread()
rkvenc_finish_job()
rkvenc_dvfs_job_end()
mutex_lock(&devfreq->lock)
<-- use-after-free
cancel_delayed_work_sync(&dvfs->throttle_work)
v4l2_m2m_buf_done_and_job_finish()
...
rkvenc_dvfs_start_allowed()
queue_delayed_work(...,
&dvfs->throttle_work, ...) ...
(devm frees rkvenc)
rkvenc_dvfs_throttle_work()
<-- use-after-free on rkvenc
Also, in rkvenc_dvfs_exit(), calling dev_pm_qos_remove_request() before devfreq_remove_device() fires devfreq's max-frequency QoS notifier and runs update_devfreq() -> rkvenc_dvfs_target() during teardown right before devfreq is removed.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
[email protected]
AI reviewRe: [RFC PATCH 3/6] media: rockchip: Add RKVENC v1 video encoder driver
In reply to Daniel GolleThank you for your contribution! Sashiko AI review found 22 potential issues to consider: - [High] media: rkvenc: DMA buffer overflow via rkvenc_s_fmt_vid_out() while streaming - [High] media…
Thank you for your contribution! Sashiko AI review found 22 potential issues to consider:
- [High] media: rkvenc: DMA buffer overflow via rkvenc_s_fmt_vid_out() while streaming
- [High] media: rkvenc: missing NV12M plane 1 and data_offset check in rkvenc_buf_prepare()
- [High] media: rkvenc: divide-by-zero in rkvenc_rc_pick() and 32-bit RC overflows
- [High] media: rkvenc: missing plane 1 and data_offset checks in rkvenc_buf_prepare()
- [High] media: rkvenc: premature watchdog cancel in rkvenc_remove() before cleanup
- [High] media: rkvenc: SCLR_DONE masked by IRQF_ONESHOT in rkvenc_hw_recover()
- [High] media: rkvenc: watchdog cancel race in rkvenc_irq_thread() and device_run()
- [High] media: rockchip: rkvenc: job_running races in IRQ, watchdog, and device_run()
- [Medium] media: rkvenc: unreachable I-frame RC model and broken IDR overflow retry
- [Medium] media: rkvenc: VB2_BUF_STATE_DONE in rkvenc_buf_queue() and stale header_mode
- [Medium] media: rkvenc: rkvenc_gop_commit() drops concurrent force_keyframe requests
- [Medium] media: rkvenc: missing CREATE_BUFS validation in rkvenc_queue_setup()
- [Medium] media: rkvenc: rkvenc_apply_ctrls() prevents setting gop_size below 30
- [Medium] media: rkvenc: control handler leak on error in rkvenc_init_ctrls()
- [Medium] media: rkvenc: missing pm_runtime_disable() on error in rkvenc_probe()
- [Medium] media: rkvenc: rkvenc_try_fmt_vid_cap() ignores active OUTPUT dimensions
- [Medium] media: rockchip: rkvenc: userspace control/format values trigger WARN_ON()
- [Medium] media: rockchip: rkvenc: broken control updates and IDR header refresh in v4l2
- [Medium] media: rockchip: rkvenc: missing pm_runtime_disable() on probe error path
- [Medium] media: rockchip: rkvenc: watchdog canceled before V4L2/M2M teardown in remove()
- [Low] media: rkvenc: HEVC merge candidate 0 overridden in rkvenc_hevc_setup()
- [Low] media: rkvenc: CAPTURE format mismatch and round_up() overflow in try_fmt
--
commit aafc130d19efda3692073c7ce74ebbd170ec8aca
Author: Daniel Golle <[email protected]>
media: rockchip: Add RKVENC v1 video encoder driver
This commit adds a stateful V4L2 memory-to-memory H.264 and HEVC video encoder driver for the Rockchip RKVENC v1 IP on RV1109/RV1126 and RK3566/RK3568 SoCs.
Quoted 6 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-gop.c new file +0
-0@@ -0,0 +1,72 @@
[ ... ]
Quoted 16 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-gop.c new file +16
-0+void rkvenc_gop_commit (struct rkvenc_ctx *ctx) +{ +struct rkvenc_gop *g = &ctx->gop; + +if (ctx->cur_intra) { + g->idr_pic_id++; + ctx->last_good = ctx->cur; + ctx->last_good_valid = true; + g->frames_since_idr =1 ; + }else { + g->frames_since_idr++; + } + ctx->last = ctx->cur; + ctx->cur ^=1 ; + ctx->c.force_keyframe = false; +}
drivers/media/platform/rockchip/rkvenc/rkvenc-gop.c line 72
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-gop.c new file +10
-0+ +void rkvenc_gop_rollback (struct rkvenc_ctx *ctx) +{ + ctx->c.force_keyframe = false; +} + +void rkvenc_gop_force_keyframe (struct rkvenc_ctx *ctx) +{ + ctx->c.force_keyframe = true; +}
[Severity: Medium]
Does clearing ctx->c.force_keyframe unconditionally in rkvenc_gop_commit() and rkvenc_gop_rollback() drop keyframe requests that arrive while a frame is in flight?
CPU0 CPU1rkvenc_device_run()
idr = rkvenc_gop_is_idr(ctx)
/* false: starts a P frame */
rkvenc_s_ctrl()
rkvenc_gop_force_keyframe(ctx)
ctx->c.force_keyframe = true
rkvenc_irq_thread()
rkvenc_finish_job()
rkvenc_gop_commit(ctx)
ctx->c.force_keyframe = false <-- request lost before next frameConsuming ctx->c.force_keyframe when rkvenc_device_run() decides idr = true (or only clearing it when committing an intra frame) would preserve requests made while a P frame is encoding.
Quoted 6 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-hevc.c new file +0
-0@@ -0,0 +1,63 @@
[ ... ]
drivers/media/platform/rockchip/rkvenc/rkvenc-hevc.c line 37
Quoted 11 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-hevc.c new file +11
-0+int rkvenc_hevc_setup (struct rkvenc_ctx *ctx) +{ +struct rkvenc_hevc_seq *seq = &ctx->hevc_seq; + +memset (seq,0 ,sizeof (*seq)); + seq->width = ctx->width; + seq->height = ctx->height; + seq->level_idc =rkvenc_hevc_level (ctx->c.level); + seq->log2_max_poc_lsb_m4 =4 ; + seq->pic_init_qp = ctx->c.p_frame_qp; + seq->max_merge_cand = ctx->c.hevc_max_merge ?:2 ;
[Severity: Low]
Does using ?: 2 here prevent setting V4L2_CID_MPEG_VIDEO_HEVC_MAX_NUM_MERGE_MV_MINUS1 to 0? rkvenc_init_ctrls() registers that control with a minimum of 0, so a valid control value of 0 is overwritten with 2.
Also, V4L2_CID_MPEG_VIDEO_HEVC_TMV_PREDICTION is registered and stored into ctx->c.hevc_tmv in rkvenc_apply_ctrls(), and ctx->vlock is initialized in rkvenc_open(), but neither is used anywhere in the driver.
[ ... ]
Quoted 6 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-rc.c new file +0
-0@@ -0,0 +1,214 @@
[ ... ]
Quoted 6 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-rc.c new file +6
-0+static u32 rkvenc_rc_capacity (struct rkvenc_ctx *ctx) +{ +if (ctx->c.vbv_size) +return ctx->c.vbv_size; +return ctx->rc_tavg *max_t (u32, ctx->c.gop_size,1 ); +}
[ ... ]
Quoted 22 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-rc.c new file +22
-0+if (ctx->rc_x[0 ] && ctx->rc_x[1 ]) { +u64 w =div_u64 ((u64 )ctx->rc_x[0 ] *256 , + ctx->rc_x[1 ] ?:1 ); +u32 n =max_t (u32, ctx->c.gop_size,1 ); +u64 t_p, t_t; + + w =min_t (u64, w,RKVENC_RC_W_CLAMP ); + t_p =div64_u64 ((u64 )ctx->rc_tavg * n *256 , + w +256 * (n -1 )); + t_t = t ? t_p : t_p * w /256 ; + t_t =rkvenc_rc_correct (ctx, t_t, n); + + target = (u32 )div_u64 ((u64 )ctx->rc_x[t] *256 , t_t); + +for (q =0 ; q <=RKVENC_QP_MAX ; q++) +if (rkvenc_qs[q] >= target) +break ; +if (q >RKVENC_QP_MAX ) + q =RKVENC_QP_MAX ; + }else { + q = prev; + }
[ ... ]
Quoted 7 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-rc.c new file +7
-0+void rkvenc_rc_update (struct rkvenc_ctx *ctx,u32 bytes,u32 qp_avg, +u8 frame_type) +{ +unsigned int t = frame_type ?1 :0 ; +u64 s = (u64 )bytes *8 ; +s64 cap =max_t (u64,rkvenc_rc_capacity (ctx),1 ); +
drivers/media/platform/rockchip/rkvenc/rkvenc-rc.c line 203
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-rc.c new file +10
-0+if (!ctx->rc_active) +return ; + + ctx->rc_x[t] =rkvenc_rc_complexity (ctx, t, s, qp_avg); + + ctx->rc_buf += (s32 )s - (s32 )ctx->rc_tavg; + ctx->rc_buf =clamp_t (s64, ctx->rc_buf, -cap, cap); + + ctx->rc_first[t] = false; +}
[Severity: High]
Can div64_u64() in rkvenc_rc_pick() divide by zero when ctx->c.gop_size is 0?
When ctx->c.gop_size is 0 (allowed by the 0..300 control range), n is 1. If a low-complexity, low-QP I frame is followed by a high-complexity, high-QP P frame such that ctx->rc_x[1] > (u64)ctx->rc_x[0] * 256, w truncates to 0 and the divisor w + 256 * (n - 1) becomes 0.
In addition, ctx->rc_tavg * max_t(u32, ctx->c.gop_size, 1) in rkvenc_rc_capacity() can overflow u32 at high bitrates and low frame rates (e.g. 40 Mbps at 1 fps with gop_size = 300), and ctx->rc_buf += (s32)s - (s32)ctx->rc_tavg in rkvenc_rc_update() can overflow 32-bit signed arithmetic before being clamped to [-cap, cap].
drivers/media/platform/rockchip/rkvenc/rkvenc-rc.c line 214
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-rc.c new file +10
-0+/* The step limit alone would need frames that the overflow keeps dropping */ +void rkvenc_rc_overflow (struct rkvenc_ctx *ctx,u32 bytes,u8 frame_type) +{ +s8 start = frame_type ? ctx->c.p_frame_qp : ctx->c.i_frame_qp; +unsigned int t = frame_type ?1 :0 ; + +rkvenc_rc_update (ctx, bytes, ctx->cur_qp, frame_type); +if (ctx->rc_qp[t] < start) + ctx->rc_qp[t] = start; +}
[Severity: Medium]
Does the I-frame rate control and overflow retry logic in rkvenc_rc_pick() and rkvenc_rc_overflow() ever adjust QP for IDR frames?
In rkvenc_rc_pick(), when t == 0:
- if !ctx->rc_first[1] is true, it always takes the rkvenc_rc_p_qp_mean() branch and returns early;
- if ctx->rc_first[1] is true, ctx->rc_x[1] is still 0, so
if (ctx->rc_x[0] && ctx->rc_x[1]) is false and it falls back to q = prev.
As a result, the t == 0 complexity branch in rkvenc_rc_pick() is unreachable, all-Intra streams (gop_size == 1 or intra-only resolutions) stay at the initial i_frame_qp forever, and when an IDR frame overflows the bitstream buffer, the retry in rkvenc_finish_job() ignores the complexity update from rkvenc_rc_overflow() and re-encodes at the same failing QP (or an unadjusted P-frame mean QP). Also, ctx->rc_tavg is only computed in rkvenc_rc_init(), so runtime changes to V4L2_CID_MPEG_VIDEO_BITRATE have no effect.
Quoted 6 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +0
-0@@ -0,0 +1,1069 @@
[ ... ]
Quoted 18 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +18
-0+/* Runs for every frame, so that control changes apply to the next one */ +static void rkvenc_apply_ctrls (struct rkvenc_ctx *ctx) +{ +struct rkvenc_ctrls *c = &ctx->c; +bool h264 = ctx->codec ==RKVENC_CODEC_H264 ; + + c->bitrate_mode =rkvenc_ctrl_val (ctx,V4L2_CID_MPEG_VIDEO_BITRATE_MODE ); + c->bitrate =rkvenc_ctrl_val (ctx,V4L2_CID_MPEG_VIDEO_BITRATE ); + c->vbv_size =rkvenc_ctrl_val (ctx,V4L2_CID_MPEG_VIDEO_VBV_SIZE ); + c->frame_rc_enable = +rkvenc_ctrl_val (ctx,V4L2_CID_MPEG_VIDEO_FRAME_RC_ENABLE ); + c->gop_size =rkvenc_ctrl_val (ctx,V4L2_CID_MPEG_VIDEO_GOP_SIZE ); + c->i_period =rkvenc_ctrl_val (ctx,V4L2_CID_MPEG_VIDEO_H264_I_PERIOD ); + c->header_mode =rkvenc_ctrl_val (ctx,V4L2_CID_MPEG_VIDEO_HEADER_MODE ); + c->prepend_spspps = +rkvenc_ctrl_val (ctx,V4L2_CID_MPEG_VIDEO_PREPEND_SPSPPS_TO_IDR ); + +if (h264) {
[ ... ]
Quoted 12 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +12
-0+ c->min_qp =rkvenc_ctrl_val (ctx,V4L2_CID_MPEG_VIDEO_H264_MIN_QP ); + c->max_qp =rkvenc_ctrl_val (ctx,V4L2_CID_MPEG_VIDEO_H264_MAX_QP ); + c->i_frame_qp = +rkvenc_ctrl_val (ctx,V4L2_CID_MPEG_VIDEO_H264_I_FRAME_QP ); + c->p_frame_qp = +rkvenc_ctrl_val (ctx,V4L2_CID_MPEG_VIDEO_H264_P_FRAME_QP ); +if (rkvenc_ctrl_val (ctx,V4L2_CID_MPEG_VIDEO_H264_I_PERIOD ) > + c->gop_size) + c->gop_size = +rkvenc_ctrl_val (ctx, +V4L2_CID_MPEG_VIDEO_H264_I_PERIOD ); + }else {
[ ... ]
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c line 144
Quoted 6 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +6
-0+if (rkvenc_ctrl_val (ctx,V4L2_CID_MPEG_VIDEO_HEVC_REFRESH_PERIOD ) > + c->gop_size) + c->gop_size = +rkvenc_ctrl_val (ctx, +V4L2_CID_MPEG_VIDEO_HEVC_REFRESH_PERIOD ); + }
[Severity: Medium]
Does taking the maximum of V4L2_CID_MPEG_VIDEO_GOP_SIZE and V4L2_CID_MPEG_VIDEO_H264_I_PERIOD (or V4L2_CID_MPEG_VIDEO_HEVC_REFRESH_PERIOD) prevent setting a GOP size below 30?
All three controls default to 30 in rkvenc_init_ctrls(), so if an application lowers only V4L2_CID_MPEG_VIDEO_GOP_SIZE (for example to 1 for all-Intra or 0 for infinite GOP), the untouched codec-specific period control stays at 30 and overrides c->gop_size back to 30.
[ ... ]
Quoted 24 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +24
-0+static int rkvenc_queue_setup (struct vb2_queue *vq,unsigned int *num_buffers, +unsigned int *num_planes,unsigned int sizes[], +struct device *alloc_devs[]) +{ +struct rkvenc_ctx *ctx =vb2_get_drv_priv (vq); +const struct rkvenc_fmt *fmt; +unsigned int i; +u32 size; + +if (V4L2_TYPE_IS_OUTPUT (vq->type)) { + fmt =rkvenc_find_fmt (rkvenc_output_fmts, +ARRAY_SIZE (rkvenc_output_fmts), + ctx->src_fourcc); +if (!fmt) +return -EINVAL ; + *num_planes = fmt->num_planes; +for (i =0 ; i < *num_planes; i++) + sizes[i] =rkvenc_output_plane_size (ctx, i); +return 0 ; + } + + fmt =rkvenc_find_fmt (rkvenc_capture_fmts, +ARRAY_SIZE (rkvenc_capture_fmts), + ctx->coded_fourcc);
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c line 239
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +10
-0+if (!fmt) +return -EINVAL ; + size =rkvenc_capture_size (ctx); + *num_planes =1 ; +if (*num_buffers) + sizes[0 ] =max (sizes[0 ], size); +else + sizes[0 ] = size; +return 0 ; +}
[Severity: Medium]
Does rkvenc_queue_setup() validate *num_planes and sizes[] when called from VIDIOC_CREATE_BUFS?
When *num_planes != 0, the vb2 queue_setup callback is expected to return -EINVAL if *num_planes does not match the format or if any sizes[i] is smaller than the minimum plane size. Here, the OUTPUT branch unconditionally overwrites *num_planes and sizes[i], and the CAPTURE branch checks *num_buffers instead of *num_planes and clamps sizes[0] instead of rejecting invalid plane counts or undersized buffers.
Quoted 4 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +4
-0+static int rkvenc_buf_prepare (struct vb2_buffer *vb) +{ +struct vb2_queue *vq = vb->vb2_queue; +struct rkvenc_ctx *ctx =vb2_get_drv_priv (vq);
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c line 254
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +10
-0+ +if (V4L2_TYPE_IS_OUTPUT (vq->type)) { +if (vb2_plane_size (vb,0 ) <rkvenc_output_plane_size (ctx,0 )) +return -EINVAL ; + }else { +if (vb2_plane_size (vb,0 ) <rkvenc_capture_size (ctx)) +return -EINVAL ; + } +return 0 ; +}
[Severity: High]
When V4L2_PIX_FMT_NV12M (ctx->src_num_planes == 2) is used, can a queued buffer with an undersized plane 1 or a large planes[1].data_offset cause an out-of-bounds DMA read?
rkvenc_buf_prepare() only checks vb2_plane_size(vb, 0) and does not check plane 1 or account for planes[1].data_offset before rkvenc_device_run() adds data_offset to the plane 1 DMA address.
Quoted 30 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +30
-0+static void rkvenc_buf_queue (struct vb2_buffer *vb) +{ +struct rkvenc_ctx *ctx =vb2_get_drv_priv (vb->vb2_queue); +struct vb2_v4l2_buffer *vbuf =to_vb2_v4l2_buffer (vb); + +if (V4L2_TYPE_IS_CAPTURE (vb->vb2_queue->type)) { +if (vb2_is_streaming (vb->vb2_queue) && +v4l2_m2m_dst_buf_is_last (ctx->m2m_ctx)) { +vb2_set_plane_payload (vb,0 ,0 ); + vbuf->field =V4L2_FIELD_NONE ; +v4l2_m2m_last_buffer_done (ctx->m2m_ctx, vbuf); +return ; + } + +/* The parameter sets take a CAPTURE buffer of their own */ +if (ctx->header_pending && + ctx->c.header_mode == +V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE ) { +u8 *vaddr =vb2_plane_vaddr (vb,0 ); + +if (vaddr) { +rkvenc_ensure_header (ctx); +if (rkvenc_copy_headers (ctx, vaddr, +vb2_plane_size (vb,0 )) <0 ) +vb2_set_plane_payload (vb,0 ,0 ); +else +vb2_set_plane_payload (vb,0 , + ctx->hdr_len); + }else { +vb2_set_plane_payload (vb,0 ,0 );
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c line 295
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +10
-0+ } + vbuf->field =V4L2_FIELD_NONE ; +v4l2_m2m_buf_done (vbuf,VB2_BUF_STATE_DONE ); + ctx->header_pending = false; +return ; + } + } + +v4l2_m2m_buf_queue (ctx->m2m_ctx, vbuf); +}
[Severity: Medium]
Can completing the separate-header buffer with VB2_BUF_STATE_DONE inside rkvenc_buf_queue() trigger a vb2 warning if rkvenc_start_streaming() subsequently fails?
For buffers queued before VIDIOC_STREAMON, vb2_start_streaming() invokes .buf_queue() before .start_streaming(). If rkvenc_buf_queue() finishes a buffer onto q->done_list and rkvenc_start_streaming() then returns an error, rkvenc_return_dst_bufs() cannot reclaim that buffer and vb2_start_streaming() hits WARN_ON(!list_empty(&q->done_list)).
In addition, rkvenc_buf_queue() tests ctx->c.header_mode, which is only updated in rkvenc_apply_ctrls() (and starts at 0, i.e. SEPARATE, before the first stream), while rkvenc_s_ctrl() sets ctx->header_pending = true on SEPARATE without updating ctx->c.header_mode or clearing ctx->header_pending when switching back to JOINED_WITH_1ST_FRAME, and rkvenc_start_streaming() does not re-arm ctx->header_pending on stream restart.
[ ... ]
Quoted 4 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +4
-0+static void rkvenc_device_run (void *priv) +{ +struct rkvenc_ctx *ctx = priv; +struct rkvenc_dev *dev = ctx->dev;
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c line 388
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +10
-0+struct vb2_v4l2_buffer *src, *dst; +struct rkvenc_frame f = {}; +u8 qp; +bool idr; +u32 hdr_len =0 ; +u8 *vaddr; +int ret; + + dev->job_ctx = ctx; + dev->job_running = true;
[Severity: High]
Can setting dev->job_ctx and dev->job_running = true here without holding dev->irq_lock race with an already-dispatched rkvenc_watchdog_work()?
CPU0 (rkvenc_irq_thread) CPU1 (kworker)spin_lock(&dev->irq_lock)dev->job_running = false spin_unlock(&dev->irq_lock) cancel_delayed_work(&watchdog)
rkvenc_watchdog_work()rkvenc_finish_job()
rkvenc_device_run()
dev->job_ctx = ctx
dev->job_running = true
spin_lock(&dev->irq_lock)
/* sees job_running == true! */
dev->job_running = false
spin_unlock(&dev->irq_lock)
rkvenc_finish_job() <-- aborts new job
pm_runtime_resume_and_get()Because rkvenc_irq_thread() uses non-synchronous cancel_delayed_work(), rkvenc_watchdog_work() can already be running on another CPU and mistake the newly starting job for the timed-out one before pm_runtime_resume_and_get() and rkvenc_hw_run() have even run, accessing MMIO registers while clocks are disabled and underflowing the runtime PM count.
Quoted 7 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +7
-0+ src =v4l2_m2m_next_src_buf (ctx->m2m_ctx); + dst =v4l2_m2m_next_dst_buf (ctx->m2m_ctx); +if (!src || !dst) + goto finish_error; + + idr =rkvenc_gop_is_idr (ctx); +
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c line 406
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +10
-0+if (idr) + ctx->gop.frames_since_idr =0 ; + +rkvenc_apply_ctrls (ctx); +rkvenc_rc_pick (ctx, &qp, idr ?0 :1 ); + ctx->cur_qp = qp; + ctx->cur_intra = idr; + +if (idr) +rkvenc_refresh_header (ctx);
[Severity: Medium]
Does calling rkvenc_refresh_header(ctx) on every IDR frame desynchronize the slice syntax registers from the bitstream parameter sets when new headers are not emitted?
rkvenc_refresh_header() overwrites ctx->h264_seq or ctx->hevc_seq (which feed the hardware slice registers) from the current V4L2 controls on every IDR picture, even when V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE is used or when V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME is used without V4L2_CID_MPEG_VIDEO_PREPEND_SPSPPS_TO_IDR. Also, as noted earlier, rkvenc_apply_ctrls() overrides gop_size < 30 with the default 30 of H264_I_PERIOD / HEVC_REFRESH_PERIOD, does not update ctx->rc_tavg when V4L2_CID_MPEG_VIDEO_BITRATE changes, and rkvenc_hevc_setup() replaces a hevc_max_merge value of 0 with 2.
[ ... ]
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c line 439
Quoted 8 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +8
-0+ f.src.plane[0 ] =lower_32_bits ( +vb2_dma_contig_plane_dma_addr (&src->vb2_buf,0 )); +if (ctx->src_num_planes ==1 ) + f.src.plane[1 ] = f.src.plane[0 ] + ctx->bytesperline * ctx->height; +else + f.src.plane[1 ] =lower_32_bits ( +vb2_dma_contig_plane_dma_addr (&src->vb2_buf,1 ) + + src->vb2_buf.planes[1 ].data_offset);
[Severity: High]
Can src->vb2_buf.planes[1].data_offset push f.src.plane[1] past the end of the plane 1 DMA buffer?
Because rkvenc_buf_prepare() never checks that vb2_plane_size(vb, 1) - vb->planes[1].data_offset is at least rkvenc_output_plane_size(ctx, 1), queueing a 2-plane V4L2_PIX_FMT_NV12M buffer with a non-zero data_offset or an undersized second plane programs RKVENC_ADR_SRC1 to DMA-read up to ctx->bytesperline * ctx->height / 2 bytes out of bounds.
[ ... ]
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c line 637
Quoted 5 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +5
-0+v4l2_ctrl_new_std (hdl, &rkvenc_ctrl_ops, +V4L2_CID_MPEG_VIDEO_AVERAGE_QP ,0 ,51 ,1 ,0 ); + +if (hdl->error) +return hdl->error;
[Severity: Medium]
Does returning hdl->error here without calling v4l2_ctrl_handler_free(hdl) leak the control handler and any controls allocated before the failure? rkvenc_open()'s err_m2m path also does not free ctx->ctrl_hdl when rkvenc_init_ctrls() fails.
[ ... ]
Quoted 7 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +7
-0+static void rkvenc_try_fmt_vid_out (struct rkvenc_ctx *ctx,struct v4l2_format *f) +{ +struct v4l2_pix_format_mplane *pix = &f->fmt.pix_mp; +const struct rkvenc_fmt *fmt; +u32 bpl; + + fmt = rkvenc_find_fmt(rkvenc_output_fmts,
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c line 706
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +10
-0+ARRAY_SIZE (rkvenc_output_fmts), + pix->pixelformat); +if (!fmt) { + fmt = &rkvenc_output_fmts[0 ]; + pix->pixelformat = fmt->fourcc; + } + + pix->field =V4L2_FIELD_NONE ; + pix->width =clamp (round_up (pix->width,2 ),16 , ctx->dev->max_width); + pix->height =clamp (round_up (pix->height,2 ),16 , ctx->dev->max_height);
[Severity: Low]
Can round_up(pix->width, 2) overflow u32 before clamp() if userspace passes 0xffffffff? Wrapping to 0 clamps the dimension to 16 instead of ctx->dev->max_width (and likewise in rkvenc_try_fmt_vid_cap()). Also, rkvenc_start_streaming() does not call v4l2_m2m_update_start_streaming_state() to pair with v4l2_m2m_update_stop_streaming_state() in rkvenc_stop_streaming().
[ ... ]
Quoted 21 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +21
-0+static void rkvenc_try_fmt_vid_cap (struct rkvenc_ctx *ctx,struct v4l2_format *f) +{ +struct v4l2_pix_format_mplane *pix = &f->fmt.pix_mp; +const struct rkvenc_fmt *fmt; + + fmt =rkvenc_find_fmt (rkvenc_capture_fmts, +ARRAY_SIZE (rkvenc_capture_fmts), + pix->pixelformat); +if (!fmt) { + fmt = &rkvenc_capture_fmts[0 ]; + pix->pixelformat = fmt->fourcc; + } + + pix->field =V4L2_FIELD_NONE ; + pix->width =clamp (round_up (pix->width,2 ),16 , ctx->dev->max_width); + pix->height =clamp (round_up (pix->height,2 ),16 , ctx->dev->max_height); + pix->num_planes =1 ; + pix->plane_fmt[0 ].bytesperline =0 ; + +if (fmt->fourcc ==V4L2_PIX_FMT_H264 ) { +u32 mbw =ALIGN (pix->width,16 ) /16 ;
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c line 750
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +10
-0+u32 mbh =ALIGN (pix->height,16 ) /16 ; + + pix->plane_fmt[0 ].sizeimage = +min (mbw * mbh *400 , pix->width * pix->height /2 ) + +RKVENC_HDR_MAX ; + }else { + pix->plane_fmt[0 ].sizeimage = + pix->width * pix->height /2 +RKVENC_HDR_MAX ; + } +}
[Severity: Medium]
Should rkvenc_try_fmt_vid_cap() derive pix->width, pix->height, and pix->plane_fmt[0].sizeimage from ctx->width and ctx->height instead of the caller-supplied f->fmt.pix_mp?
Because rkvenc_s_fmt_vid_cap() does not update ctx->width or ctx->height, calling VIDIOC_TRY_FMT or VIDIOC_S_FMT on the CAPTURE queue with different dimensions (such as 0x0 when selecting the coded format) returns a width, height, and sizeimage that disagree with VIDIOC_G_FMT(CAPTURE), rkvenc_queue_setup(), and rkvenc_buf_prepare().
[ ... ]
Quoted 7 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +7
-0+static int rkvenc_s_fmt_vid_out (struct file *file,void *priv, +struct v4l2_format *f) +{ +struct rkvenc_ctx *ctx =fh_to_rkvenc_ctx (file->private_data); +struct vb2_queue *vq =v4l2_m2m_get_vq (ctx->m2m_ctx, f->type); + +if (vb2_is_busy (vq))
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c line 786
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-v4l2.c new file +10
-0+return -EBUSY ; + +rkvenc_try_fmt_vid_out (ctx, f); + ctx->src_fourcc = f->fmt.pix_mp.pixelformat; + ctx->src_num_planes = f->fmt.pix_mp.num_planes; + ctx->width = f->fmt.pix_mp.width; + ctx->height = f->fmt.pix_mp.height; + ctx->bytesperline = f->fmt.pix_mp.plane_fmt[0 ].bytesperline; +return 0 ; +}
[Severity: High]
Can changing the OUTPUT format while the CAPTURE queue is already streaming overflow the reconstruction and reference DMA buffers in ctx->slot[]?
rkvenc_start_streaming() only allocates ctx->slot[] (via rkvenc_ref_alloc()) and validates dimensions when the CAPTURE queue starts streaming, while returning 0 immediately for the OUTPUT queue. Because rkvenc_s_fmt_vid_out() only checks vb2_is_busy() on the OUTPUT queue, userspace can start CAPTURE streaming at a small resolution (e.g. 16x16), then call VIDIOC_S_FMT(OUTPUT) with a larger resolution (up to 4096x2304) and start OUTPUT streaming, causing the hardware encoder to DMA-write past the end of the undersized ctx->slot[] buffers.
[ ... ]
Quoted 6 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +0
-0@@ -0,0 +1,489 @@
[ ... ]
drivers/media/platform/rockchip/rkvenc/rkvenc.c line 45
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +10
-0+int rkvenc_hw_run (struct rkvenc_dev *dev,const struct rkvenc_frame *f) +{ +struct rkvenc_img *img = &dev->img; +void __iomem *regs = dev->regs; +unsigned int a; +int ret; + + ret =rkvenc_vepu541_img_build (f, img); +if (WARN_ON (ret)) +return ret;
[Severity: Medium]
Can userspace trigger this WARN_ON(ret) via standard V4L2 controls or format changes?
V4L2_CID_MPEG_VIDEO_H264_MIN_QP / MAX_QP (and the HEVC equivalents) are independent controls with range 1..51 and no check enforcing min_qp <= max_qp. Setting min_qp > max_qp passes inverted bounds to clamp_t() in rkvenc_rc_pick() and causes rkvenc_frame_check() inside rkvenc_vepu541_img_build() to return -EINVAL and hit WARN_ON(ret) on every frame (as does setting a narrow H.264 resolution with width <= 32 and height > 16 on OUTPUT after CAPTURE is already streaming).
[ ... ]
Quoted 35 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +35
-0+void rkvenc_hw_recover (struct rkvenc_dev *dev) +{ +void __iomem *regs = dev->regs; + +if (rkvenc_hw_force_clear (dev)) +return ; + +dev_err (dev->dev,"job did not stop on force clear, resetting\n" ); + +writel (RKVENC_INT_ALL & ~RKVENC_INT_SCLR_DONE , regs +RKVENC_INT_MSK ); +writel (RKVENC_INT_ALL , regs +RKVENC_INT_CLR ); + +reinit_completion (&dev->sclr_done); +writel (RKVENC_ENC_CLR_SAFE , regs +RKVENC_ENC_CLR ); +wait_for_completion_timeout (&dev->sclr_done, +msecs_to_jiffies (RKVENC_SAFE_CLEAR_MS )); + +if (dev->num_resets) { +reset_control_bulk_assert (dev->num_resets, dev->resets); +udelay (RKVENC_RESET_PULSE_US ); +reset_control_bulk_deassert (dev->num_resets, dev->resets); + } + +writel (0 , regs +RKVENC_INT_MSK ); +} + +static irqreturn_t rkvenc_irq (int irq,void *priv) +{ +struct rkvenc_dev *dev = priv; +u32 sta; + + sta =readl (dev->regs +RKVENC_INT_STA ); +if (!sta) +return IRQ_NONE ; +
drivers/media/platform/rockchip/rkvenc/rkvenc.c line 137
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +10
-0+writel (sta, dev->regs +RKVENC_INT_CLR ); +/* Flush the posted clear before the interrupt line is sampled again */ +readl (dev->regs +RKVENC_INT_STA ); + +if (sta &RKVENC_INT_SCLR_DONE ) +complete (&dev->sclr_done); + + dev->irq_sta = sta; +return IRQ_WAKE_THREAD ; +}
[Severity: High]
Can dev->sclr_done ever be completed by rkvenc_irq() when rkvenc_hw_recover() is called from rkvenc_irq_thread()?
Because the IRQ is requested with IRQF_ONESHOT, the interrupt line remains masked at the interrupt controller for the entire duration of rkvenc_irq_thread(). When rkvenc_irq_thread() -> rkvenc_finish_job() -> rkvenc_hw_recover() waits on dev->sclr_done, rkvenc_irq() cannot run until rkvenc_irq_thread() exits, so wait_for_completion_timeout() always stalls for the full 100 ms timeout.
Furthermore, when RKVENC_INT_SCLR_DONE does fire in rkvenc_irq(), it overwrites dev->irq_sta = sta and returns IRQ_WAKE_THREAD instead of IRQ_HANDLED, queuing a spurious run of rkvenc_irq_thread().
[ ... ]
Quoted 9 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +9
-0+static irqreturn_t rkvenc_irq_thread (int irq,void *priv) +{ +struct rkvenc_dev *dev = priv; +struct rkvenc_ctx *ctx; +enum vb2_buffer_state state; +u32 int_sta, bytes, qp_sum; +unsigned long flags; + +/* Whoever clears job_running first, IRQ or watchdog, finishes the job */
drivers/media/platform/rockchip/rkvenc/rkvenc.c line 235
Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +10
-0+spin_lock_irqsave (&dev->irq_lock, flags); +if (!dev->job_running) { +spin_unlock_irqrestore (&dev->irq_lock, flags); +return IRQ_HANDLED ; + } + ctx = dev->job_ctx; + dev->job_running = false; +spin_unlock_irqrestore (&dev->irq_lock, flags); + +cancel_delayed_work (&dev->watchdog);
[Severity: High]
Can a spurious IRQ thread wake-up from RKVENC_INT_SCLR_DONE or a fast IRQ before schedule_delayed_work() corrupt the next job's state?
<hardirq> rkvenc_irq_threadrkvenc_irq() (SCLR_DONE)
dev->irq_sta = SCLR_DONE
return IRQ_WAKE_THREAD
rkvenc_finish_job()
rkvenc_device_run() (retry / next job)
dev->job_running = true
/* next threaded IRQ run starts */
rkvenc_irq_thread()
dev->job_running = false
/* ENC_DONE not set in irq_sta */
rkvenc_finish_job(..., ERROR) <-- abortsIn addition to the unlocked dev->job_running assignment in rkvenc_device_run() and the non-synchronous cancel_delayed_work() above, rkvenc_hw_run() writes RKVENC_ENC_STRT before calling schedule_delayed_work(&dev->watchdog, ...). If a small frame finishes and rkvenc_irq_thread() calls cancel_delayed_work() before rkvenc_hw_run() calls schedule_delayed_work(), the watchdog timer remains armed after the job completes.
[ ... ]
drivers/media/platform/rockchip/rkvenc/rkvenc.c line 379
Quoted 7 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +7
-0+ ret =devm_request_threaded_irq (dev, irq, rkvenc_irq, + rkvenc_irq_thread,IRQF_ONESHOT , +dev_name (dev), rkvenc); +if (ret) { +dev_err (dev,"Could not request encoder IRQ\n" ); +return ret; + }
[Severity: Medium]
Is registering the IRQ with devm_request_threaded_irq() before pm_runtime_enable(dev) safe?
Because devm_request_threaded_irq() is called before pm_runtime_enable(dev), devres will also free the IRQ after rkvenc_remove() has already called pm_runtime_disable(&pdev->dev), and an early error return after pm_runtime_enable(dev) leaves runtime PM enabled.
drivers/media/platform/rockchip/rkvenc/rkvenc.c line 385
Quoted 6 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +6
-0+ +pm_runtime_enable (dev); + + ret =pm_runtime_resume_and_get (dev); +if (ret) +return ret;
[Severity: Medium]
Does returning directly when pm_runtime_resume_and_get(dev) fails leave runtime PM enabled on the device? Jumping to err_pm_disable would balance pm_runtime_enable(dev).
[ ... ]
drivers/media/platform/rockchip/rkvenc/rkvenc.c line 444
Quoted 5 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +5
-0+static void rkvenc_remove (struct platform_device *pdev) +{ +struct rkvenc_dev *rkvenc =platform_get_drvdata (pdev); + +cancel_delayed_work_sync (&rkvenc->watchdog);
[Severity: High]
Can rkvenc->watchdog be re-armed after cancel_delayed_work_sync() returns in rkvenc_remove()?
CPU0 CPU1rkvenc_remove()
cancel_delayed_work_sync(&watchdog)
rkvenc_device_run() / retry
rkvenc_hw_run()
schedule_delayed_work(&watchdog, ...)
rkvenc_v4l2_cleanup()
...
/* devm frees rkvenc */
rkvenc_watchdog_work() <-- use-after-free
Because cancel_delayed_work_sync() runs before rkvenc_v4l2_cleanup() and v4l2_m2m_release(), a new job or an in-flight overflow retry in rkvenc_finish_job() can call rkvenc_hw_run() and re-schedule rkvenc->watchdog before the V4L2 and M2M devices are torn down.
drivers/media/platform/rockchip/rkvenc/rkvenc.c line 449
Quoted 5 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +5
-0+rkvenc_v4l2_cleanup (rkvenc); +v4l2_m2m_release (rkvenc->m2m_dev); +v4l2_device_unregister (&rkvenc->v4l2_dev); +pm_runtime_disable (&pdev->dev); +}
[Severity: Medium]
Should cancel_delayed_work_sync(&rkvenc->watchdog) (and stopping any active IRQ/retry activity) happen after rkvenc_v4l2_cleanup() and v4l2_m2m_release() have stopped new M2M jobs from starting, so that teardown cannot leave rkvenc->watchdog queued when devres frees struct rkvenc_dev?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3