Skip to contentSkip to conversation

media: rockchip: VEPU510 H.264 encoder for RK3576

RFC PATCH In review Jiaxing Hu 5 participants 15 messages Jul 22 – Jul 23

Subsystems (3)
lore
Share & tools
lore

Jiaxing Hu

author
#0 lore raw

cover media: rockchip: VEPU510 H.264 encoder for RK3576

This is an RFC for a from-scratch mainline V4L2 driver for the H.264 hardware video encoder (VEPU510) on the Rockchip RK3576. It is a stateful mem2mem encoder (NV12 in, H.264 Annex-B out) modelled on the verisilicon/hantro driver, not on the downstream MPP-service model.

I'm posting it as an RFC because intra frames work but inter frames do not, and I'd like a second pair of eyes on the inter-frame problem before this is worth a real submission.

What works
----------

Intra-only (I-frame) encoding is confirmed on real hardware (Radxa ROCK 4D). With GOP size 1 the driver produces a valid H.264 stream the reference decoder accepts. This already exercises the full path: V4L2 m2m, the register programming, the software SPS/PPS prepend, and the hardware slice output.

There is no upstream userspace involved (an encoder needs no request API); I drive it with a small ioctl test program that feeds NV12 frames and writes the CAPTURE buffers to a .h264 file.

The open problem: inter (P-frame) frames hang
---------------------------------------------

Every P-frame stalls the encoder's own hardware watchdog (INT_STA bit 8, ~20 ms after the kick) and produces essentially no bitstream. I have spent a lot of time narrowing this; it is sharply localized but I cannot close it:

- The P-frame register writes match a real register-write trace of the vendor stack encoding the same content, byte for byte (I traced the vendor kernel's writes and diffed against this driver's).

- The reconstruction the previous frame writes is *valid*: dumping the recon buffer after the I-frame shows correct reconstructed pixels.

- If the P-frame reads its own (older, settled) recon slot instead of the immediately-preceding frame's, it completes. Reading the immediately-preceding frame's *fresh* reconstruction as the reference is what hangs.

- It is not FBC: storing the reconstruction uncompressed (enc_pic.rec_fbc_dis = 1) still hangs.

- The first frame of a session always works; the first P-frame hangs; after a couple of failures + core resets, later frames sometimes start completing. It behaves like a warm-up / settling problem on the reference-read path, not a wrong register value.

So the encoder programs identically to the vendor and reads a valid reference, but stalls fetching the previous frame's reconstruction as the inter reference on the first inter frame of a session. My best guess is that the vendor does something between consecutive frame submissions -- a completion/drain wait, a cache/coherency step, or a per-frame re-arm -- that I am missing, but I have not found it. If anyone recognizes this on VEPU5xx, or knows what the reference-read path needs between frames, a pointer would be very welcome.

The shape of this -- the first operation of a power session works, the next stalls, and a reset/warm-up sometimes helps -- is the same one I ran into bringing up this SoC's NPU (accel/rocket RKNN), where the second/chained submit in a session would not fire [1]. I can't claim they share a root cause, but if this is a known RK3576-wide submit / re-arm quirk rather than an encoder-specific bug, that would be good to know.

[1] https://lore.kernel.org/all/[email protected]/

Notes for review
----------------

- The PARAM/SQI register classes are programmed from mpp's constant "default tuning" tables (not derived per-frame), as noted in the code. They are required: leaving them unwritten stalls even the I-frame.

- Rate control is fixed-QP only for now (the bitrate control is advisory).

- H.264 baseline/main, single slice, 4:2:0 only.

- Tested at 176x144; other resolutions are not yet validated.

Jiaxing Hu (3):
dt-bindings: media: add Rockchip RK3576 VEPU H.264 encoder
media: rockchip: add VEPU510 H.264 encoder driver for RK3576
arm64: dts: rockchip: rk3576: add VEPU H.264 encoder nodes

 .../bindings/media/rockchip,rk3576-vepu.yaml       |   94 ++
 arch/arm64/boot/dts/rockchip/rk3576.dtsi           |   50 +
 drivers/media/platform/rockchip/Kconfig            |    1 +
 drivers/media/platform/rockchip/Makefile           |    1 +
 drivers/media/platform/rockchip/rkvenc/Kconfig     |   14 +
 drivers/media/platform/rockchip/rkvenc/Makefile    |    6 +
 .../media/platform/rockchip/rkvenc/rkvenc-h264.c   | 1095 ++++++++++++++++++++
 .../media/platform/rockchip/rkvenc/rkvenc-regs.h   |  929 +++++++++++++++++
 drivers/media/platform/rockchip/rkvenc/rkvenc.c    |  892 ++++++++++++++++
 drivers/media/platform/rockchip/rkvenc/rkvenc.h    |  212 ++++
 10 files changed, 3294 insertions(+)

--
2.43.0

author
#1 lore raw

RFC PATCH 1/3 dt-bindings: media: add Rockchip RK3576 VEPU H.264 encoder

Add the binding for the VEPU510 H.264 hardware video encoder found on the Rockchip RK3576. Each SoC has two independent encoder cores, each behind its own IOMMU.

---
 .../bindings/media/rockchip,rk3576-vepu.yaml       | 94 ++++++++++++++++++++++
 1 file changed, 94 insertions(+)
Documentation/devicetree/bindings/media/rockchip,rk3576-vepu.yaml new file +94 -0
new file mode 100644
index 000000000..942118cb3
@@ -0,0 +1,94 @@
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/media/rockchip,rk3576-vepu.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Rockchip VEPU510 Video Encoder (RK3576)
+
+maintainers:
+  - Jiaxing Hu <[email protected]>
+
+description: |-
+  RK3576 has two identical VEPU510 hardware H.264/H.265 video encoder
+  cores. Each core is a standalone V4L2 mem2mem device in this binding;
+  there is no shared hardware descriptor/link-list engine tying the two
+  cores together (unlike the RK3576 video decoder's CCU), so each core
+  node is independent and self-contained.
+
+properties:
+  compatible:
+    const: rockchip,rk3576-vepu
+
+  reg:
+    maxItems: 1
+
+  interrupts:
+    maxItems: 1
+
+  clocks:
+    items:
+      - description: The Video Encoder AXI interface clock
+      - description: The Video Encoder AHB interface clock
+      - description: The Video Encoder core clock
+
+  clock-names:
+    items:
+      - const: aclk_vcodec
+      - const: hclk_vcodec
+      - const: clk_core
+
+  assigned-clocks: true
+
+  assigned-clock-rates: true
+
+  resets:
+    items:
+      - description: The Video Encoder AXI interface reset
+      - description: The Video Encoder AHB interface reset
+      - description: The Video Encoder core reset
+
+  reset-names:
+    items:
+      - const: video_a
+      - const: video_h
+      - const: video_core
+
+  power-domains:
+    maxItems: 1
+
+  iommus:
+    maxItems: 1
+
+required:
+  - compatible
+  - reg
+  - interrupts
+  - clocks
+  - clock-names
+  - power-domains
+  - iommus
+
+additionalProperties: false
+
+examples:
+  - |
+    #include <dt-bindings/interrupt-controller/arm-gic.h>
+    #include <dt-bindings/clock/rockchip,rk3576-cru.h>
+    #include <dt-bindings/power/rockchip,rk3576-power.h>
+
+    vepu0: video-codec@27a00000 {
+        compatible = "rockchip,rk3576-vepu";
+        reg = <0x27a00000 0x6000>;
+        interrupts = <GIC_SPI 310 IRQ_TYPE_LEVEL_HIGH>;
+        clocks = <&cru ACLK_VEPU0>, <&cru HCLK_VEPU0>, <&cru CLK_VEPU0_CORE>;
+        clock-names = "aclk_vcodec", "hclk_vcodec", "clk_core";
+        assigned-clocks = <&cru ACLK_VEPU0>, <&cru CLK_VEPU0_CORE>;
+        assigned-clock-rates = <400000000>, <702000000>;
+        resets = <&cru SRST_A_VEPU0>, <&cru SRST_H_VEPU0>, <&cru SRST_VEPU0_CORE>;
+        reset-names = "video_a", "video_h", "video_core";
+        power-domains = <&power RK3576_PD_VEPU0>;
+        iommus = <&vepu0_mmu>;
+    };
+
+...
Signature · 1 line
2.43.0
author
#2 lore raw

RFC PATCH 2/3 media: rockchip: add VEPU510 H.264 encoder driver for RK3576

In reply to Jiaxing Hu

Add a from-scratch stateful V4L2 mem2mem driver for the Rockchip RK3576 VEPU510 H.264 hardware video encoder (raw NV12 in, H.264 Annex-B out), modelled on the verisilicon/hantro device_run()/codec_ops split rather than the downstream MPP-service/task-queue model.

The two encoder cores (rkvenc0/rkvenc1) are exposed as two independent V4L2 M2M device nodes, each driving one physical core standalone; the downstream vendor CCU cross-core load-balancing is not implemented.

Register field semantics were worked out by trial-and-error against Rockchip's own open-source userspace codec library (rockchip-linux/mpp) and cross-checked against a real register-write trace of the downstream vendor stack running the same encode.

Intra (I-frame) encoding is confirmed working on real hardware (Radxa ROCK 4D): it produces valid H.264 that the reference decoders accept. Inter (P-frame) encoding still hits a hardware-watchdog stall in the reference-read datapath -- this is the main open question for this RFC; see the cover letter for the full symptom analysis.

---
 drivers/media/platform/rockchip/Kconfig            |    1 +
 drivers/media/platform/rockchip/Makefile           |    1 +
 drivers/media/platform/rockchip/rkvenc/Kconfig     |   14 +
 drivers/media/platform/rockchip/rkvenc/Makefile    |    6 +
 .../media/platform/rockchip/rkvenc/rkvenc-h264.c   | 1095 ++++++++++++++++++++
 .../media/platform/rockchip/rkvenc/rkvenc-regs.h   |  929 +++++++++++++++++
 drivers/media/platform/rockchip/rkvenc/rkvenc.c    |  892 ++++++++++++++++
 drivers/media/platform/rockchip/rkvenc/rkvenc.h    |  212 ++++
 8 files changed, 3150 insertions(+)
drivers/media/platform/rockchip/Kconfig +1 -0
index ba401d32f..e6afa6092 100644
@@ -6,3 +6,4 @@ source "drivers/media/platform/rockchip/rga/Kconfig"
 source "drivers/media/platform/rockchip/rkcif/Kconfig"
 source "drivers/media/platform/rockchip/rkisp1/Kconfig"
 source "drivers/media/platform/rockchip/rkvdec/Kconfig"
+source "drivers/media/platform/rockchip/rkvenc/Kconfig"
drivers/media/platform/rockchip/Makefile +1 -0
index 0e0b2cbbd..6a1d847a1 100644
@@ -3,3 +3,4 @@ obj-y += rga/
 obj-y += rkcif/
 obj-y += rkisp1/
 obj-y += rkvdec/
+obj-y += rkvenc/
drivers/media/platform/rockchip/rkvenc/Kconfig new file +14 -0
new file mode 100644
index 000000000..cd2a9c2eb
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: GPL-2.0
+config VIDEO_ROCKCHIP_VEPU510
+	tristate "Rockchip VEPU510 Video Encoder driver"
+	depends on ARCH_ROCKCHIP || COMPILE_TEST
+	depends on VIDEO_DEV
+	select VIDEOBUF2_DMA_CONTIG
+	select V4L2_MEM2MEM_DEV
+	help
+	  Support for the Rockchip VEPU510 hardware video encoder IP
+	  present on the RK3576 SoC, which accelerates H.264 video
+	  encoding.
+
+	  To compile this driver as a module, choose M here: the module
+	  will be called rockchip-vepu510.
drivers/media/platform/rockchip/rkvenc/Makefile new file +6 -0
new file mode 100644
index 000000000..322bf10af
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_VIDEO_ROCKCHIP_VEPU510) += rockchip-vepu510.o
+
+rockchip-vepu510-y += \
+		      rkvenc.o \
+		      rkvenc-h264.o
drivers/media/platform/rockchip/rkvenc/rkvenc-h264.c new file +1095 -0

1,095 lines not on this page · raw message

drivers/media/platform/rockchip/rkvenc/rkvenc-regs.h new file +929 -0

929 lines not on this page · raw message

drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +892 -0

892 lines not on this page · raw message

drivers/media/platform/rockchip/rkvenc/rkvenc.h new file +212 -0

56 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.43.0

Part of the diff is not shown here. View the raw message.

#3 lore raw

Re: [RFC PATCH 2/3] media: rockchip: add VEPU510 H.264 encoder driver for RK3576

Thank you for your contribution! Sashiko AI review found 13 potential issue(s) to consider: - [High] DMA Use-After-Free during `VIDIOC_STREAMOFF` - [High] V4L2 Object Lifetime / Use-After-Free on…

Thank you for your contribution! Sashiko AI review found 13 potential issue(s) to consider:
- [High] DMA Use-After-Free during VIDIOC_STREAMOFF
- [High] V4L2 Object Lifetime / Use-After-Free on Unbind
- [High] Reference Buffer Corruption from frame_num Wrap
- [High] Lifecycle and Resource Leak in remove
- [High] State Desync / Bitstream Corruption from Cached SPS/PPS
- [High] Compilation Failure from Non-existent Macro
- [High] NULL Pointer Dereference and DMA Cache Coherency
- [High] Race Condition / TOCTOU across Watchdog and IRQ
- [Medium] Resource Leak in probe Error Path
- [Medium] clk_bulk_enable without prepare
- [Low] PM Subsystem Violation without pm_ptr wrapper
- [Low] Logic Flaw in NV12 Size Calculation
- [Low] C bitfields for MMIO registers
--

--- Patch [2]: [RFC PATCH 2/3] media: rockchip: add VEPU510 H.264 encoder driver for RK3576 --- commit a8783bc02129bde5b3f6c151d6c2930f8cb47a0a

Author: Jiaxing Hu <[email protected]>
Date:   Wed Jul 22 07:34:16 2026 +0000

media: rockchip: add VEPU510 H.264 encoder driver for RK3576

Add a from-scratch stateful V4L2 mem2mem driver for the Rockchip RK3576 VEPU510 H.264 hardware video encoder (raw NV12 in, H.264 Annex-B out), modelled on the verisilicon/hantro device_run()/codec_ops split rather than the downstream MPP-service/task-queue model.

Quoted 5 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-regs.h new file +0 -0
new file mode 100644
index 0000000000000..353ba3ebafada

[ ... ]

drivers/media/platform/rockchip/rkvenc/rkvenc-regs.h line 199

Quoted 4 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-regs.h new file +4 -0
+union rkvenc_reg_enc_pic {
+	struct {
+		u32 enc_stnd:2;			/* 0 = H.264 */
+		u32 cur_frm_ref:1;

[Severity: Low]
Can using C bitfields for MMIO registers cause endianness issues? On big-endian systems, the compiler packs bitfields MSB-to-LSB, which could assign configuration bits to the wrong offsets. Explicit bitwise shifts and masks are generally preferred for hardware register layouts.

Quoted 5 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-h264.c new file +0 -0
new file mode 100644
index 0000000000000..58b4fb65b4991

[ ... ]

drivers/media/platform/rockchip/rkvenc/rkvenc-h264.c line 415

Quoted 12 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-h264.c new file +12 -0
+static void rkvenc_h264_stop(struct rkvenc_ctx *ctx)
+{
+	struct rkvenc_dev *dev = ctx->dev;
+	struct rkvenc_h264_ctx *h264 = &ctx->h264;
+	size_t total_size = h264->pixel_buf_size + h264->thumb_buf_size +
+			     h264->smear_buf_size;
+	int i;
+
+	for (i = 0; i < 2; i++) {
+		if (h264->recn_buf_cpu[i])
+			dma_free_coherent(dev->dev, total_size,
+					  h264->recn_buf_cpu[i], h264->recn_buf_dma[i]);

[Severity: High]
If userspace invokes VIDIOC_STREAMOFF while an active encoding job is running, does the stop callback free the recn_buf and meiw_buf DMA memory before the hardware is stopped?

Since the driver makes no attempt to wait for the hardware to idle, could it continue executing and perform DMA writes into the now-freed memory and unmapped V4L2 buffers?

[ ... ]

Quoted 2 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-h264.c new file +2 -0
+static void rkvenc_h264_run(struct rkvenc_ctx *ctx)
+{

[ ... ]

drivers/media/platform/rockchip/rkvenc/rkvenc-h264.c line 469

Quoted 7 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-h264.c new file +7 -0
+	unsigned int write_idx = h264->frame_num & 1;
+	/* Frame 0 has no previously-written reference at all -- alias read to
+	 * write (real hardware's refr_idx == curr_idx for frame 0, see the
+	 * rkvenc_h264_ctx comment in rkvenc.h). From frame 1 on, buf[write_idx
+	 * ^ 1] genuinely holds the previous frame's real reconstruction.
+	 */
+	unsigned int read_idx = h264->frame_num == 0 ? write_idx : (write_idx ^ 1);

[Severity: High]
When h264->frame_num wraps around 255 back to 0 in rkvenc_h264_done(), does the aliasing logic here incorrectly assume it is an IDR frame?

Since frame 256 is typically a P-frame, would this cause it to read its reference from the same buffer it is actively writing its reconstruction to, potentially corrupting the encode and stalling the hardware? Should this logic use the is_idr flag instead?

[ ... ]

drivers/media/platform/rockchip/rkvenc/rkvenc-h264.c line 491

Quoted 2 lines
drivers/media/platform/rockchip/rkvenc/rkvenc-h264.c new file +2 -0
+	if (header_len)
+		memcpy(vb2_plane_vaddr(&dst_buf->vb2_buf, 0), h264->sps_pps_nal, header_len);

[Severity: High]
Is it possible for vb2_plane_vaddr() to return NULL here? If the destination buffer is a DMABUF, V4L2 does not automatically vmap DMABUFs unless requested, which could lead to a NULL pointer dereference in memcpy().

Also, does this CPU write require a dma_sync_single_for_device() call afterwards to flush the CPU cache before the hardware DMA writes the subsequent slice NAL unit?

[ ... ]

drivers/media/platform/rockchip/rkvenc/rkvenc-h264.c line 914

Quoted 1 line
drivers/media/platform/rockchip/rkvenc/rkvenc-h264.c new file +1 -0
+	synt_pps.pic_init_qp = h264->i_qp->val;

[Severity: High]
If userspace dynamically changes the V4L2_CID_MPEG_VIDEO_H264_I_FRAME_QP control during streaming, could the hardware slice QP desync from the statically cached SPS/PPS NAL?

Since rkvenc_h264_gen_sps_pps() is only called once during start_streaming, the hardware receives the dynamically updated QP but the bitstream still gets the stale cached PPS. Could this break decoder image reconstruction?

Quoted 5 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +0 -0
new file mode 100644
index 0000000000000..d70bcc5fb96d4

[ ... ]

drivers/media/platform/rockchip/rkvenc/rkvenc.c line 127

Quoted 4 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +4 -0
+static u32 rkvenc_nv12_sizeimage(unsigned int width, unsigned int height)
+{
+	return width * height + 2 * ALIGN(width, 2) / 2 * ALIGN(height, 2) / 2 * 2;
+}

[Severity: Low]
Does the chroma size calculation evaluate to width * height due to C operator precedence? The expression has 2 * ALIGN(...) / 2 * ALIGN(...) / 2 * 2. It appears this computes 2 * W * H instead of the expected 1.5 * W * H for NV12, which could result in allocating oversized buffers.

[ ... ]

Quoted 4 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +4 -0
+static void rkvenc_device_run(void *priv)
+{
+	struct rkvenc_ctx *ctx = priv;
+	struct rkvenc_dev *dev = ctx->dev;

drivers/media/platform/rockchip/rkvenc/rkvenc.c line 448

Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +10 -0
+	int ret;
+
+	ret = pm_runtime_resume_and_get(dev->dev);
+	if (ret < 0) {
+		v4l2_m2m_buf_done_and_job_finish(dev->m2m_dev, ctx->fh.m2m_ctx,
+						  VB2_BUF_STATE_ERROR);
+		return;
+	}
+
+	ret = clk_bulk_enable(dev->num_clocks, dev->clocks);

[Severity: Medium]
If the kernel is built with CONFIG_PM disabled, pm_runtime_resume_and_get() does nothing. Could calling clk_bulk_enable() here on an unprepared clock violate the clock API and trigger a warning?

[ ... ]

drivers/media/platform/rockchip/rkvenc/rkvenc.c line 571

Quoted 11 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +11 -0
+static irqreturn_t rkvenc_irq_thread(int irq, void *priv)
+{
+	struct rkvenc_dev *dev = priv;
+	struct rkvenc_ctx *ctx = dev->cur_ctx;
+	enum vb2_buffer_state state;
+
+	if (!ctx)
+		return IRQ_HANDLED;
+
+	if (!cancel_delayed_work(&dev->watchdog_work))
+		return IRQ_HANDLED;

[Severity: High]
Can a TOCTOU race occur here if a hardware watchdog timeout coincides with the scheduling of the next job?

If a watchdog fires, it completes the hung job via rkvenc_job_finish(). The M2M core then pulls the next job, updating dev->cur_ctx and re-arming the watchdog.

If the hard IRQ from the hung job then schedules rkvenc_irq_thread(), could it read the new context locklessly, cancel the new job's watchdog, and erroneously abort the new job while the hardware is actively processing it?

[ ... ]

drivers/media/platform/rockchip/rkvenc/rkvenc.c line 650

Quoted 7 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +7 -0
+static int rkvenc_open(struct file *file)
+{
+	struct rkvenc_dev *dev = video_drvdata(file);
+	struct rkvenc_ctx *ctx;
+	int ret;
+
+	ctx = kzalloc_obj(*ctx);

[Severity: High]
Does the kzalloc_obj() macro exist in the kernel tree? Would this cause a build failure? Should this be kzalloc(sizeof(*ctx), GFP_KERNEL) instead?

[ ... ]

Quoted 2 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +2 -0
+static int rkvenc_probe(struct platform_device *pdev)
+{

[ ... ]

drivers/media/platform/rockchip/rkvenc/rkvenc.c line 805

Quoted 5 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +5 -0
+	rkvenc->vfd = (struct video_device){
+		.fops		= &rkvenc_fops,
+		.ioctl_ops	= &rkvenc_ioctl_ops,
+		.minor		= -1,
+		.release	= video_device_release_empty,

[Severity: High]
If the driver is unbound via sysfs while userspace still holds a file descriptor open, does devres free the rkvenc_dev structure (which embeds the video_device) when rkvenc_remove() returns?

Since video_unregister_device() does not forcefully close existing open file descriptors, would rkvenc_release() later dereference the freed dev and m2m_ctx when the file descriptor is finally closed?

[ ... ]

drivers/media/platform/rockchip/rkvenc/rkvenc.c line 839

Quoted 3 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +3 -0
+	ret = pm_runtime_resume_and_get(dev);
+	if (ret)
+		return dev_err_probe(dev, ret, "failed to power up for initial reset\n");

[Severity: Medium]
If pm_runtime_resume_and_get() fails during probe, does this early return bypass the err_release_m2m and err_unreg_v4l2 cleanup labels? Could this leak the allocated v4l2_dev and m2m_dev structures?

[ ... ]

drivers/media/platform/rockchip/rkvenc/rkvenc.c line 870

Quoted 7 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +7 -0
+static void rkvenc_remove(struct platform_device *pdev)
+{
+	struct rkvenc_dev *rkvenc = platform_get_drvdata(pdev);
+
+	cancel_delayed_work_sync(&rkvenc->watchdog_work);
+	video_unregister_device(&rkvenc->vfd);
+	v4l2_m2m_release(rkvenc->m2m_dev);

[Severity: High]
Does rkvenc_remove() tear down the M2M device and cancel the watchdog without stopping active hardware encodes or waiting for them to finish?

Because the IRQ handler remains active until after remove() returns, could a completing hardware job fire an interrupt and access the destroyed m2m_dev via v4l2_m2m_buf_done_and_job_finish(), causing memory corruption?

[ ... ]

drivers/media/platform/rockchip/rkvenc/rkvenc.c line 886

Quoted 7 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +7 -0
+static struct platform_driver rkvenc_driver = {
+	.probe = rkvenc_probe,
+	.remove = rkvenc_remove,
+	.driver = {
+		.name = "rkvenc",
+		.of_match_table = rkvenc_of_match,
+		.pm = &rkvenc_pm_ops,

[Severity: Low]
Should this assignment use the pm_ptr() wrapper? Without it, building the kernel with CONFIG_PM disabled could leave the static rkvenc_pm_ops structure and its callbacks linked as dead code, potentially triggering compiler warnings.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
maintainer
#4 lore raw

Re: [RFC PATCH 2/3] media: rockchip: add VEPU510 H.264 encoder driver for RK3576

In reply to Jiaxing Hu

Am Mittwoch, 22. Juli 2026, 09:34:16 Mitteleuropäische Sommerzeit schrieb Jiaxing Hu:

Quoted 21 lines

Add a from-scratch stateful V4L2 mem2mem driver for the Rockchip RK3576 VEPU510 H.264 hardware video encoder (raw NV12 in, H.264 Annex-B out), modelled on the verisilicon/hantro device_run()/codec_ops split rather than the downstream MPP-service/task-queue model.

The two encoder cores (rkvenc0/rkvenc1) are exposed as two independent V4L2 M2M device nodes, each driving one physical core standalone; the downstream vendor CCU cross-core load-balancing is not implemented.

Register field semantics were worked out by trial-and-error against Rockchip's own open-source userspace codec library (rockchip-linux/mpp) and cross-checked against a real register-write trace of the downstream vendor stack running the same encode.

Intra (I-frame) encoding is confirmed working on real hardware (Radxa ROCK 4D): it produces valid H.264 that the reference decoders accept. Inter (P-frame) encoding still hits a hardware-watchdog stall in the reference-read datapath -- this is the main open question for this RFC; see the cover letter for the full symptom analysis.

[...]

Quoted 23 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +17 -0
new file mode 100644
index 000000000..d70bcc5fb
@@ -0,0 +1,892 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Rockchip VEPU510 (RK3576) hardware video encoder driver.
+ *
+ * Copyright (C) 2026 Jiaxing Hu <[email protected]>
+ *
+ * Architecture notes (see also rkvenc-regs.h):
+ *
+ *  - This is a *stateful* V4L2 mem2mem encoder (raw NV12 in on OUTPUT,
+ *    H.264 Annex-B out on CAPTURE) modelled on
+ *    drivers/media/platform/verisilicon's hantro_drv.c device_run()/
+ *    codec_ops{run,done} split, NOT on rkvdec's stateless request-API
+ *    decoder pattern — an encoder has no per-frame bitstream to parse,
+ *    so there is nothing analogous to rkvdec_run_preamble/postamble here.
+ *
+ *  - The vendor downstream driver (rockchip-linux/kernel,
+ *    drivers/video/rockchip/mpp/mpp_rkvenc2.c) groups rkvenc0/rkvenc1

drivers/media/platform/rockchip/rkvenc/rkvenc.c line 27

Quoted 10 lines
drivers/media/platform/rockchip/rkvenc/rkvenc.c new file +10 -0
+ *    under a "CCU" (rockchip,rkv-encoder-rk3576-ccu) that does pure
+ *    software task-queue load balancing across both cores, plus an
+ *    optional DCHS (dual-core-handshake) register protocol used only
+ *    when *deliberately* splitting one frame's rows across both cores.
+ *    There is no hardware descriptor/link-list engine behind it (unlike
+ *    the decoder's CCU). v1 of this driver does not implement either:
+ *    rkvenc0 and rkvenc1 are exposed as two independent V4L2 M2M device
+ *    nodes, each driving one physical core standalone — the same
+ *    simplification rkvdec itself makes for multi-core VDPU hardware
+ *    (see rkvdec_disable_multicore()).

At least the comment and from a casual look also the code get this backwards. The idea is to explicitly _not_ expose multiple video devices.

See
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/media/platform/rockchip/rkvdec/rkvdec.c#n1613

Any (future) scheduling should happen inside the driver, and not get offloaded onto _every_ userspace application individually.

Heiko

author
#5 lore raw

Re: [RFC PATCH 2/3] media: rockchip: add VEPU510 H.264 encoder driver for RK3576

Quoted 6 lines · Heiko Stübner

At least the comment and from a casual look also the code get this backwards. The idea is to explicitly _not_ expose multiple video devices.
[...]
Any (future) scheduling should happen inside the driver, and not get offloaded onto _every_ userspace application individually.

You're right, thanks. Exposing rkvenc0/rkvenc1 as two nodes and pushing the core choice onto userspace is backwards; rkvdec_disable_multicore() is the pattern to follow -- probe only the first core, keep any multi-core scheduling inside the driver. I'll fix that.

Depending on where the interface discussion in the 0/3 thread lands (Nicolas is pointing at a lower-level stateless / Vulkan-Video direction that Detlev is already building for RK3588) this driver's shape may change quite a bit, but the "don't offload core selection to userspace" point holds either way.

Thanks,
Jiaxing

author
#6 lore raw

RFC PATCH 3/3 arm64: dts: rockchip: rk3576: add VEPU H.264 encoder nodes

In reply to Jiaxing Hu

Add the two VEPU510 encoder cores at 0x27a00000 / 0x27a10000 and their IOMMUs.

---
 arch/arm64/boot/dts/rockchip/rk3576.dtsi | 50 ++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)
arch/arm64/boot/dts/rockchip/rk3576.dtsi +50 -0
index b622c6fee..9bf4f2c41 100644
@@ -1320,6 +1320,56 @@ vdec_mmu: iommu@27b00800 {
 			#iommu-cells = <0>;
 		};
 
+		vepu0: video-codec@27a00000 {
+			compatible = "rockchip,rk3576-vepu";
+			reg = <0x0 0x27a00000 0x0 0x6000>;
+			interrupts = <GIC_SPI 310 IRQ_TYPE_LEVEL_HIGH>;
+			clocks = <&cru ACLK_VEPU0>, <&cru HCLK_VEPU0>, <&cru CLK_VEPU0_CORE>;
+			clock-names = "aclk_vcodec", "hclk_vcodec", "clk_core";
+			assigned-clocks = <&cru ACLK_VEPU0>, <&cru CLK_VEPU0_CORE>;
+			assigned-clock-rates = <400000000>, <702000000>;
+			resets = <&cru SRST_A_VEPU0>, <&cru SRST_H_VEPU0>, <&cru SRST_VEPU0_CORE>;
+			reset-names = "video_a", "video_h", "video_core";
+			power-domains = <&power RK3576_PD_VEPU0>;
+			iommus = <&vepu0_mmu>;
+		};
+
+		vepu0_mmu: iommu@27a0f000 {
+			compatible = "rockchip,rk3576-iommu", "rockchip,rk3568-iommu";
+			reg = <0x0 0x27a0f000 0x0 0x40>;
+			interrupts = <GIC_SPI 311 IRQ_TYPE_LEVEL_HIGH>;
+			clocks = <&cru ACLK_VEPU0>, <&cru HCLK_VEPU0>;
+			clock-names = "aclk", "iface";
+			power-domains = <&power RK3576_PD_VEPU0>;
+			rockchip,disable-mmu-reset;
+			#iommu-cells = <0>;
+		};
+
+		vepu1: video-codec@27a10000 {
+			compatible = "rockchip,rk3576-vepu";
+			reg = <0x0 0x27a10000 0x0 0x6000>;
+			interrupts = <GIC_SPI 387 IRQ_TYPE_LEVEL_HIGH>;
+			clocks = <&cru ACLK_VEPU1>, <&cru HCLK_VEPU1>, <&cru CLK_VEPU1_CORE>;
+			clock-names = "aclk_vcodec", "hclk_vcodec", "clk_core";
+			assigned-clocks = <&cru ACLK_VEPU1>, <&cru CLK_VEPU1_CORE>;
+			assigned-clock-rates = <400000000>, <702000000>;
+			resets = <&cru SRST_A_VEPU1>, <&cru SRST_H_VEPU1>, <&cru SRST_VEPU1_CORE>;
+			reset-names = "video_a", "video_h", "video_core";
+			power-domains = <&power RK3576_PD_VEPU1>;
+			iommus = <&vepu1_mmu>;
+		};
+
+		vepu1_mmu: iommu@27a1f000 {
+			compatible = "rockchip,rk3576-iommu", "rockchip,rk3568-iommu";
+			reg = <0x0 0x27a1f000 0x0 0x40>;
+			interrupts = <GIC_SPI 388 IRQ_TYPE_LEVEL_HIGH>;
+			clocks = <&cru ACLK_VEPU1>, <&cru HCLK_VEPU1>;
+			clock-names = "aclk", "iface";
+			power-domains = <&power RK3576_PD_VEPU1>;
+			rockchip,disable-mmu-reset;
+			#iommu-cells = <0>;
+		};
+
 		vop: vop@27d00000 {
 			compatible = "rockchip,rk3576-vop";
 			reg = <0x0 0x27d00000 0x0 0x3000>, <0x0 0x27d05000 0x0 0x1000>;
Signature · 1 line
2.43.0
#7 lore raw

Re: [RFC PATCH 3/3] arm64: dts: rockchip: rk3576: add VEPU H.264 encoder nodes

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Device tree nodes are not sorted by unit-address in ascending order. --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Device tree nodes are not sorted by unit-address in ascending order. --

--- Patch [3]: [RFC PATCH 3/3] arm64: dts: rockchip: rk3576: add VEPU H.264 encoder nodes --- commit 6e5cf8fb6138851fdbc5bc797d5562505948a1ae
Author: Jiaxing Hu <[email protected]>

arm64: dts: rockchip: rk3576: add VEPU H.264 encoder nodes

This commit adds the two VEPU510 encoder cores at 0x27a00000 and 0x27a10000 along with their corresponding IOMMUs to the rk3576 device tree.

arch/arm64/boot/dts/rockchip/rk3576.dtsi vdec_mmu: iommu@27b00800 { · line 1317

Quoted 9 lines
arch/arm64/boot/dts/rockchip/rk3576.dtsi +1 -0
index e12a2a0cfb891..58d706d563290 100644
@@ -1314,6 +1314,56 @@ vdec_mmu: iommu@27b00800 {
 			#iommu-cells = <0>;
 		};
 
+		vepu0: video-codec@27a00000 {

[Severity: Low]
This isn't a bug, but should these new nodes be placed before vdec_mmu?

The device tree coding style generally expects nodes to be sorted by their unit-address in ascending order. Here, the vepu0 node at 0x27a00000 and subsequent vepu1 node at 0x27a10000 are being placed after vdec_mmu at 0x27b00800.

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#8 lore raw
In reply to Jiaxing Hu

Hi,

Le mercredi 22 juillet 2026 à 19:34 +1200, Jiaxing Hu a écrit :

Quoted 8 lines

This is an RFC for a from-scratch mainline V4L2 driver for the H.264 hardware video encoder (VEPU510) on the Rockchip RK3576.  It is a stateful mem2mem encoder (NV12 in, H.264 Annex-B out) modelled on the verisilicon/hantro driver, not on the downstream MPP-service model.

I'm posting it as an RFC because intra frames work but inter frames do not, and I'd like a second pair of eyes on the inter-frame problem before this is worth a real submission.

This is really nice to see interest in enabling encoder for this chips set. I'm adding Detlev in CC here, as he's actively working on RK3588 encoder, which is quite a similar chip. But there is quite a twist his approach which I'll explain shortly.

What I believe I've seen walking through your RFC is an encoder based on V4L2 Stateful encoder interface:

https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-encoder.html

While this isn't invalid, specially for encoders, for this type of hardware, the community agreed direction was to introduce V4L2 Stateless Encoder specification, using the media request and compound controls to maintain a lower level interface for this HW. See Paul's proposal for IMX8MP Hantro VC8000E chips (H.264 only for now):

https://lore.kernel.org/all/[email protected]/

The downside is that we need a new spec document for this class of driver to reach upstream (not part of Paul's series). Plus, for each codec, appropriate encode parameter controls needs to be designed, and they must be proven to work across multiple hardware (generally at least 2). I still think there is a need for V4L2 drivers like this. Being high level, they offer a great interface to protect against HW actions that would be harmful to the Linux operating system. Specially when you don't have an IOMMU like the IMX8M Plus. But at the same time, V4L2 memory model often clash with modern graphic stack, making the integration quite painful.

Here's the twist to Detlev (and myself) approach. In order to make the kernel module even thinner, and to avoid having to specify new V4L2 interfaces, we decided to look into making drivers similar to how we do GPU drivers today. This is possible today thanks to the Vulkan Video standard. So Detlev driver, which finally got P-Frames support few days ago (he'll explain why this didn't work initially for him), is split in two part:

- The kernel module, which expose hardware specific interface (he decided to implement it in Rust)
- The userspace driver, which he's implementing in Mesa project

What I really like of this approach is that the kernel driver is much more stable, as it expose the HW at a lower level in a codec agnostic way. For the final application, the interface is the same regardless if its a GPU attached codec or a standalone codec. Its even the same interface if you are on Windows, except for the low level zero-copy interop. The rest of the development can happen in userspace, which is a lot faster to develop and easier to debug.

One thing we notice, is that for performance reason, keeping rate control as close to the IRQ as possible is important. Its impossible to batch any work if you have to constantly wait for the previous work to be done in order to compute the next set of QP values. For that reason, we plan to eventually share these in-kernel implementation with V4L2. In our imagination, eBPF could also be a usable option.

I will leave to Detlev to share some early code once he's ready, but considering you both are working on the same family of encoder, it would certainly be a lot nicer if both endup with the same type of drivers. Offering two interface for the same hardware would impose a lot of extra effort that I would rather avoid.

regards,
Nicolas

Quoted 98 lines

What works
----------

Intra-only (I-frame) encoding is confirmed on real hardware (Radxa ROCK 4D).  With GOP size 1 the driver produces a valid H.264 stream the reference decoder accepts.  This already exercises the full path: V4L2 m2m, the register programming, the software SPS/PPS prepend, and the hardware slice output.

There is no upstream userspace involved (an encoder needs no request API); I drive it with a small ioctl test program that feeds NV12 frames and writes the CAPTURE buffers to a .h264 file.

The open problem: inter (P-frame) frames hang
---------------------------------------------

Every P-frame stalls the encoder's own hardware watchdog (INT_STA bit 8, ~20 ms after the kick) and produces essentially no bitstream.  I have spent a lot of time narrowing this; it is sharply localized but I cannot close it:

  - The P-frame register writes match a real register-write trace of the vendor stack encoding the same content, byte for byte (I traced the vendor kernel's writes and diffed against this driver's).

  - The reconstruction the previous frame writes is *valid*: dumping the recon buffer after the I-frame shows correct reconstructed pixels.

  - If the P-frame reads its own (older, settled) recon slot instead of the immediately-preceding frame's, it completes.  Reading the immediately-preceding frame's *fresh* reconstruction as the reference is what hangs.

  - It is not FBC: storing the reconstruction uncompressed (enc_pic.rec_fbc_dis = 1) still hangs.

  - The first frame of a session always works; the first P-frame hangs; after a couple of failures + core resets, later frames sometimes start completing.  It behaves like a warm-up / settling problem on the reference-read path, not a wrong register value.

So the encoder programs identically to the vendor and reads a valid reference, but stalls fetching the previous frame's reconstruction as the inter reference on the first inter frame of a session.  My best guess is that the vendor does something between consecutive frame submissions -- a completion/drain wait, a cache/coherency step, or a per-frame re-arm -- that I am missing, but I have not found it.  If anyone recognizes this on VEPU5xx, or knows what the reference-read path needs between frames, a pointer would be very welcome.

The shape of this -- the first operation of a power session works, the next stalls, and a reset/warm-up sometimes helps -- is the same one I ran into bringing up this SoC's NPU (accel/rocket RKNN), where the second/chained submit in a session would not fire [1].  I can't claim they share a root cause, but if this is a known RK3576-wide submit / re-arm quirk rather than an encoder-specific bug, that would be good to know.

[1] https://lore.kernel.org/all/[email protected]/

Notes for review
----------------

  - The PARAM/SQI register classes are programmed from mpp's constant "default tuning" tables (not derived per-frame), as noted in the code.  They are required: leaving them unwritten stalls even the I-frame.

  - Rate control is fixed-QP only for now (the bitrate control is advisory).

  - H.264 baseline/main, single slice, 4:2:0 only.

  - Tested at 176x144; other resolutions are not yet validated.

Jiaxing Hu (3):
  dt-bindings: media: add Rockchip RK3576 VEPU H.264 encoder
  media: rockchip: add VEPU510 H.264 encoder driver for RK3576
  arm64: dts: rockchip: rk3576: add VEPU H.264 encoder nodes

 .../bindings/media/rockchip,rk3576-vepu.yaml       |   94 ++ arch/arm64/boot/dts/rockchip/rk3576.dtsi           |   50 + drivers/media/platform/rockchip/Kconfig            |    1 + drivers/media/platform/rockchip/Makefile           |    1 + drivers/media/platform/rockchip/rkvenc/Kconfig     |   14 + drivers/media/platform/rockchip/rkvenc/Makefile    |    6 + .../media/platform/rockchip/rkvenc/rkvenc-h264.c   | 1095
++++++++++++++++++++
 .../media/platform/rockchip/rkvenc/rkvenc-regs.h   |  929 +++++++++++++++++ drivers/media/platform/rockchip/rkvenc/rkvenc.c    |  892 ++++++++++++++++ drivers/media/platform/rockchip/rkvenc/rkvenc.h    |  212 ++++

 10 files changed, 3294 insertions(+)

--
2.43.0

author
#9 lore raw
Quoted 3 lines · Nicolas Dufresne

This is really nice to see interest in enabling encoder for this chips set. I'm adding Detlev in CC here, as he's actively working on RK3588 encoder, which is quite a similar chip.

Thanks, and thanks for looping Detlev in.

Quoted 2 lines

So Detlev driver, which finally got P-Frames support few days ago (he'll explain why this didn't work initially for him) [...]

This is the most useful sentence in the thread for me. RK3576's VEPU510 and RK3588's VEPU580 are the same VEPU5xx family -- same vendor HAL flow, identical reconstruction/reference-read (recn_refr) path -- and that reference read is exactly where every P-frame stalls for me: the first inter frame of a session hangs the encoder watchdog fetching the previous frame's reconstruction, even though the reconstruction is valid and my register writes match the vendor byte for byte. So whatever Detlev hit is very likely the same bug.

Detlev -- whatever you can share about why P-frames didn't work at first would help me enormously.

Quoted 5 lines

While this isn't invalid [...] the community agreed direction was to introduce V4L2 Stateless Encoder specification [...]
[...] we decided to look into making drivers similar to how we do GPU drivers today. This is possible today thanks to the Vulkan Video standard.

I'm not attached to the stateful interface -- I used it to get something running on real silicon, not out of conviction. Before I commit to reworking, I'd like to be sure I understand what "converge" means here, because I see two fairly different directions in the thread: Paul's V4L2 stateless H.264 encoder (kernel-side reflist/rbsp/RC core), and the Vulkan-Video split you describe for Detlev (thin kernel module + Mesa userspace). Those differ a lot in kind and in effort. Which is the intended target for the Rockchip encoders, or is that still open? Is there a branch, early code or spec draft I should read beyond Paul's series?

Either way the RK3576 hardware enablement and the reference-read fix are shared with RK3588, so I'd much rather contribute the VEPU510 side to a common base than maintain a parallel one -- I'd just like to know which base that is before rebuilding on it.

Thanks,
Jiaxing

#10 lore raw

Hi,

Le jeudi 23 juillet 2026 à 12:46 +1200, Jiaxing Hu a écrit :

Quoted 10 lines

I'm not attached to the stateful interface -- I used it to get something running on real silicon, not out of conviction. Before I commit to reworking, I'd like to be sure I understand what "converge" means here, because I see two fairly different directions in the thread: Paul's V4L2 stateless H.264 encoder (kernel-side reflist/rbsp/RC core), and the Vulkan-Video split you describe for Detlev (thin kernel module + Mesa userspace). Those differ a lot in kind and in effort. Which is the intended target for the Rockchip encoders, or is that still open? Is there a branch, early code or spec draft I should read beyond Paul's series?

This is a fair point, and something that will certainly bring some level of confusion in the short term. My personal goal, and what Detlev is actively working on, is to move all codecs drivers toward Vulkan Video, using the most meaningful driver interface for the purpose. The drivers needs to stay safe though, so it matters that the HW can protect memory access.

Detlev and I picked the RKVENC because there was no upstream driver for it, so a good place for a clean transition. It has an IOMMU and can ensure per process memory access protection. If a process miss-program the encoder and make it corrupt on other encoding session it owns, its all right, its not different then writing a C program with multiple thread and corrupting another thread. But corruption other process session, or the system is not acceptable, and something Linux driver must protect against.

Paul does not have this hardware protection, so I think he needs a higher level interface for the driver, and at the point, why not base it on existing kernel interface. Though, V4l2 is massively larger (probably around 100x) interface then what we are drafting currently for RKVENC in a drm style driver.

The common part is the rate control. Typically, stateful encoder are firmware based, and the rate control is in the firmware. We need an in kernel rate control for performance reason. The programming latency is going to be too high otherwise. The point of convergence, is that we can share helpers for that purpose, and Paul's work is really clean, and already in the shape of helpers not tied to v4l2.

Quoted 4 lines

Either way the RK3576 hardware enablement and the reference-read fix are shared with RK3588, so I'd much rather contribute the VEPU510 side to a common base than maintain a parallel one -- I'd just like to know which base that is before rebuilding on it.

This is our experience with decoder too, its relatively easy to support both, differences being minor. I will ping Detlev today and see if he can help here. I know in his case he was trying to avoid reconstructed frame compression, and it only started working when he enable that compression. Though, he had mmu faults prior to that.

cheers,
Nicolas

#11 lore raw

Hi,

Quoted 11 lines · Nicolas Dufresne

Le jeudi 23 juillet 2026 à 12:46 +1200, Jiaxing Hu a écrit :

Quoted 10 lines

I'm not attached to the stateful interface -- I used it to get something running on real silicon, not out of conviction. Before I commit to reworking, I'd like to be sure I understand what "converge" means here, because I see two fairly different directions in the thread: Paul's V4L2 stateless H.264 encoder (kernel-side reflist/rbsp/RC core), and the Vulkan-Video split you describe for Detlev (thin kernel module + Mesa userspace). Those differ a lot in kind and in effort. Which is the intended target for the Rockchip encoders, or is that still open? Is there a branch, early code or spec draft I should read beyond Paul's series?

I would recommend taking a look at the slides of the talks I gave at the Linux Media Summit last year and this year, which should give you more context and insight about the design choices.

- https://paulk.fr/talks/2025-linux-media-summit/2025-linux-media-summit-v4l2-stateless-video-encoding-uapi.pdf
- https://paulk.fr/talks/2026-linux-media-summit/2026-linux-media-summit-v4l2-stateless-video-encoding-uapi-progress-update.pdf

Generally speaking the idea of this proposal is to stick with the same design idea that we had for stateless decoders, including V4L2 M2M queues, dedicated controls and media requests.

Like Nicolas explained, some hardware can be a good fit for both approaches. I think we agreed that it wouldn't be a problem to have two drivers (one v4l2, one drm-ish) for the same hardware, so the question is really about what you prefer to do and get involved with.

My V4L2-based proposal is already rather usable, although it has not yet reached its final form. I have a GStreamer implementation ready that has been used extensively too, but it will also need to be reworked before it can reach upstream.

The DRM-ish proposal is at an earlier stage and will likely require a complete redesign of your driver, moving most things in userspace.

Quoted 11 lines

Paul does not have this hardware protection, so I think he needs a higher level interface for the driver, and at the point, why not base it on existing kernel interface. Though, V4l2 is massively larger (probably around 100x) interface then what we are drafting currently for RKVENC in a drm style driver.

The common part is the rate control. Typically, stateful encoder are firmware based, and the rate control is in the firmware. We need an in kernel rate control for performance reason. The programming latency is going to be too high otherwise. The point of convergence, is that we can share helpers for that purpose, and Paul's work is really clean, and already in the shape of helpers not tied to v4l2.

Well there is some level of dependency to v4l2 in some places, but I'm sure we can extract relevant parts and make them common so that a drm-ish subsystem can reuse them easily.

Quoted 10 lines
Quoted 4 lines

Either way the RK3576 hardware enablement and the reference-read fix are shared with RK3588, so I'd much rather contribute the VEPU510 side to a common base than maintain a parallel one -- I'd just like to know which base that is before rebuilding on it.

This is our experience with decoder too, its relatively easy to support both, differences being minor. I will ping Detlev today and see if he can help here. I know in his case he was trying to avoid reconstructed frame compression, and it only started working when he enable that compression. Though, he had mmu faults prior to that.

In my opinion it would be fine to have two drivers, but it's really up to you. Since you already have a v4l2 base for it, I would encourage you to move it to my proposal, which will probably be finalized in a shorter time frame than the other proposal and lets you reuse the code you've already written. Then you could also add RK3576 support to the drm-ish Vulkan Video proposal without too much work when it is ready, reusing the RK3588 work.

But again, it's really your decision. And it's hard to say how long things will take to finalize. But I think it would be beneficial to have two drivers for this hardware and not trash the work you've done.

All the best,

Paul

Signature · 6 lines
Paul Kocialkowski,

Independent contractor - sys-base - https://www.sys-base.io/
Free software developer - https://www.paulk.fr/

Expert in multimedia, graphics and embedded hardware support with Linux.
#12 lore raw

Hi Paul.

Le jeudi 23 juillet 2026 à 20:41 +0200, Paul Kocialkowski a écrit :

Quoted 7 lines

In my opinion it would be fine to have two drivers, but it's really up to you. Since you already have a v4l2 base for it, I would encourage you to move it to my proposal, which will probably be finalized in a shorter time frame than the other proposal and lets you reuse the code you've already written. Then you could also add RK3576 support to the drm-ish Vulkan Video proposal without too much work when it is ready, reusing the RK3588 work.

My main concern is that once you exposed an API, the transition path is near impossible unless you accept to support both APIs concurrently. So I may raise a slight objection on the above proposal. Once we have the code shared (Detlev is working on it), you'll see that Detlev (and Daniel Almeida) implementation is just a step ahead, and adding RK3576 to an RK3588 is just a minor update (its the same chip, different minor version). On top of which, once the kernel driver will have settled, adding more codecs will mostly (or entirely) happen in userspace.

With no offense, the RFC here requires significant work before it can be upstreamed. A lot of the code is hex tables generated from inspecting a running driver. So there is a lot of work to decipher this into clean code.

On that aspect, if you are missing any reference code or technical documentation, let us know, though I believe most of this is public information for this chip.

regards,
Nicolas

#13 lore raw

Hi Nicolas,

Quoted 11 lines · Nicolas Dufresne

Hi Paul.

Le jeudi 23 juillet 2026 à 20:41 +0200, Paul Kocialkowski a écrit :

Quoted 7 lines

In my opinion it would be fine to have two drivers, but it's really up to you. Since you already have a v4l2 base for it, I would encourage you to move it to my proposal, which will probably be finalized in a shorter time frame than the other proposal and lets you reuse the code you've already written. Then you could also add RK3576 support to the drm-ish Vulkan Video proposal without too much work when it is ready, reusing the RK3588 work.

Quoted 3 lines

My main concern is that once you exposed an API, the transition path is near impossible unless you accept to support both APIs concurrently. So I may raise a slight objection on the above proposal.

Well there could be a recommended implementation using your design and a more experimental v4l2 implementation that is discouraged to use but still available.

But that means another driver to maintain and more long-term involvement from Jiaxing Hu. I am not sure it is worth it, but I don't think we should entirely close the door on the idea. I also understand that there can be a bit of frustration with trashing code and a desire to continue working on it (especially now that it can benefit from the technical knowledge acquired from your work). But I'm just speculating here.

Still I would be interested to eventually be able to compare performance between the two approaches on the same hardware :)

Quoted 10 lines

Once we have the code shared (Detlev is
working on it), you'll see that Detlev (and Daniel Almeida) implementation is just a step ahead, and adding RK3576 to an RK3588 is just a minor update (its the same chip, different minor version). On top of which, once the kernel driver will have settled, adding more codecs will mostly (or entirely) happen in userspace.

With no offense, the RFC here requires significant work before it can be upstreamed. A lot of the code is hex tables generated from inspecting a running driver. So there is a lot of work to decipher this into clean code.

Yes I'm sure there is still a lot of work to do.

All the best,

Paul

Quoted 6 lines

On that aspect, if you are missing any reference code or technical documentation, let us know, though I believe most of this is public information for this chip.

regards,
Nicolas

Signature · 6 lines
Paul Kocialkowski,

Independent contractor - sys-base - https://www.sys-base.io/
Free software developer - https://www.paulk.fr/

Expert in multimedia, graphics and embedded hardware support with Linux.
author
#14 lore raw
In reply to Nicolas Dufresne

Hi Nicolas, Paul, Detlev,

Thanks both for laying out the trade-offs so clearly.

Decision from my side: I'll converge on the DRM-ish / Vulkan Video direction and sync with Detlev, rather than push the stateful V4L2 driver further. Nicolas' point about UAPI lock-in is the deciding one for me -- I don't want to expose an encoder UAPI that then has to be supported forever -- and if RK3576 is just a minor delta on Detlev's RK3588 base, that's exactly where my hardware work is worth the most.

Paul -- thank you for the generous offer. I read both your Media Summit decks and your posted series ([PATCH 00/14], the generic in-kernel h264-enc core + rbsp + rate control on VC8000E); it's clearly further along than I'd assumed. I'm not closing the door on a V4L2 version later, but I'd rather not commit to maintaining two drivers up front, so I'll treat that as a possible follow-up, not the primary path.

Where I think I can be useful right now is the hardware, since that part is shared with RK3588 regardless of interface:

Quoted 3 lines

I know in his case he was trying to avoid reconstructed frame compression, and it only started working when he enable that compression. Though, he had mmu faults prior to that.

Useful data point, but let me be precise about where I already am, so I don't send Detlev chasing something I've done. This driver already runs with reconstruction compression enabled (enc_pic.rec_fbc_dis = 0) -- the state Detlev needed -- and the first inter frame still hangs. Disabling it (rec_fbc_dis = 1) was one of my experiments and also hung, so FBC state alone doesn't explain my stall.

The part that does line up is the mmu fault. I still have a residual rk_iommu write fault on this path; I traced it from a boot-varying garbage IOVA down to a benign IOVA 0, and I'd concluded it was a separate AXI transaction from the recon write, independent of the P-frame hang. Detlev's report -- that his mmu faults and his missing P-frames cleared together -- is a direct reason to distrust that "independent" conclusion and re-check whether the fault and the stall share a root cause on RK3576 too.

Detlev -- when you have a moment: on RK3588, were the mmu fault and the P-frame failure the same underlying problem? And did enabling reconstruction compression clear the fault on its own, or did the reference-read mapping (how the previous frame's reconstruction is mapped for the encoder to fetch) need a separate change as well? Whatever you can share, I'm happy to test on RK3576 and feed back a Tested-by.

One point on the current code, since it came up:

Quoted 2 lines

A lot of the code is hex tables generated from inspecting a running driver.

Fair, and I won't pretend otherwise -- the PARAM/SQI classes are currently shipped as fixed tables taken from a captured encode rather than computed. The one thing I'd add is that those particular values are mpp's public default-tuning tables (the RDO lambda/cost and subjective-quality tables in Rockchip's open-source mpp HAL, Apache-2.0), so they're constants I can cite to mpp source rather than opaque state. But I fully agree the driver needs significant cleanup before it's upstream-worthy, and in the DRM-ish model most of that tuning moves to userspace anyway, so a lot of it won't survive the transition in its current form.

So: please point me at Detlev's shared branch / early code whenever it's in a shape to build against, and I'll start porting the RK3576 hardware bring-up onto it.

Thanks,
Jiaxing