Skip to contentSkip to conversation

: rockchip mpp for v4l2 video deocder

PATCH In review Randy Li 3 participants 7 messages Mar 7, 2019 – Mar 8, 2019

Subsystems (4)
lore
Share & tools
lore

Randy Li

author
#0 lore raw

v2 cover : rockchip mpp for v4l2 video deocder

Although I really want to push those work after I added more codec supports, but I found it is more urge to do those in v4l2 core framework and userspace.

I would use this driver to present the current problems, write down a summary here and I would reply to those threads later to push forward.

1. Slice construction is a bad idea. I think I have said my reason in the IRC and mail before, vp9 is always good example.

And it would request the driver to update QP table/CABAC table every slice.

I would make something to describe a buffer with some addtional meta data.

But the current request API limit a buffer with associated with an request buffer, which prevent sharing some sequence data, but it still can solve some problems.

2. Advantage DMA memory control.
I think I need to do some work at v4l2 core.

2.1 The DMA address of each planes. I have sent a mail before talked about why multiple planes is necessary for the rockchip platform. And it maybe required by the other platforms.

2.2 IOMMU resume
The most effective way to restore the decoder from critical error is doing a restting by reset controller.
Which would leading its slave IOMMU reset at the same time. Then none of those v4l2 buffers are mapping in the IOMMU.

3. H.264 and HEVC header
I still think those structure have some not necessary fileds in dpb or reference part, which I don't think hardware decoder would care about that or can be predict from the other information.

I would join to talk later.

4. The work flow of V4L2
I need a method to prepare the register set before the device acutally begin the transaction. Which is necessary for those high frame rate usecase.

Also it is useful for those device would share some hardware resources with the other device and it can save more power.

I think I need to do some work at v4l2 core.

Randy Li (6):
arm64: dts: rockchip: add power domain to iommu
staging: video: rockchip: add v4l2 decoder [TEST]: rockchip: mpp: support qptable
staging: video: rockchip: add video codec
arm64: dts: rockchip: boost clocks for rk3328
arm64: dts: rockchip: add video codec for rk3328

 .../arm64/boot/dts/rockchip/rk3328-rock64.dts |   32 +
 arch/arm64/boot/dts/rockchip/rk3328.dtsi      |  116 +-
 arch/arm64/boot/dts/rockchip/rk3399.dtsi      |    2 +
 drivers/staging/Kconfig                       |    2 +
 drivers/staging/Makefile                      |    1 +
 drivers/staging/rockchip-mpp/Kconfig          |   34 +
 drivers/staging/rockchip-mpp/Makefile         |   10 +
 drivers/staging/rockchip-mpp/mpp_debug.h      |   87 ++
 drivers/staging/rockchip-mpp/mpp_dev_common.c | 1367 +++++++++++++++++
 drivers/staging/rockchip-mpp/mpp_dev_common.h |  212 +++
 drivers/staging/rockchip-mpp/mpp_dev_rkvdec.c |  997 ++++++++++++
 drivers/staging/rockchip-mpp/mpp_dev_vdpu2.c  |  619 ++++++++
 drivers/staging/rockchip-mpp/mpp_service.c    |  197 +++
 drivers/staging/rockchip-mpp/mpp_service.h    |   38 +
 drivers/staging/rockchip-mpp/rkvdec/hal.h     |   63 +
 drivers/staging/rockchip-mpp/rkvdec/hevc.c    |  166 ++
 drivers/staging/rockchip-mpp/rkvdec/regs.h    |  608 ++++++++
 drivers/staging/rockchip-mpp/vdpu2/hal.h      |   52 +
 drivers/staging/rockchip-mpp/vdpu2/mpeg2.c    |  277 ++++
 drivers/staging/rockchip-mpp/vdpu2/regs.h     |  699 +++++++++
 20 files changed, 5575 insertions(+), 4 deletions(-)
 create mode 100644 drivers/staging/rockchip-mpp/Kconfig
 create mode 100644 drivers/staging/rockchip-mpp/Makefile
 create mode 100644 drivers/staging/rockchip-mpp/mpp_debug.h
 create mode 100644 drivers/staging/rockchip-mpp/mpp_dev_common.c
 create mode 100644 drivers/staging/rockchip-mpp/mpp_dev_common.h
 create mode 100644 drivers/staging/rockchip-mpp/mpp_dev_rkvdec.c
 create mode 100644 drivers/staging/rockchip-mpp/mpp_dev_vdpu2.c
 create mode 100644 drivers/staging/rockchip-mpp/mpp_service.c
 create mode 100644 drivers/staging/rockchip-mpp/mpp_service.h
 create mode 100644 drivers/staging/rockchip-mpp/rkvdec/hal.h
 create mode 100644 drivers/staging/rockchip-mpp/rkvdec/hevc.c
 create mode 100644 drivers/staging/rockchip-mpp/rkvdec/regs.h
 create mode 100644 drivers/staging/rockchip-mpp/vdpu2/hal.h
 create mode 100644 drivers/staging/rockchip-mpp/vdpu2/mpeg2.c
 create mode 100644 drivers/staging/rockchip-mpp/vdpu2/regs.h
Signature · 1 line
2.20.1
author
#1 lore raw

PATCH v2 1/6 arm64: dts: rockchip: add power domain to iommu

From: Randy Li <[email protected]>

---
 arch/arm64/boot/dts/rockchip/rk3328.dtsi | 4 ++++
 arch/arm64/boot/dts/rockchip/rk3399.dtsi | 2 ++
 2 files changed, 6 insertions(+)
arch/arm64/boot/dts/rockchip/rk3328.dtsi +4 -0
index 84f14b132e8f..117d02e21932 100644
@@ -581,6 +581,7 @@
 		clocks = <&cru ACLK_H265>, <&cru PCLK_H265>;
 		clock-names = "aclk", "iface";
 		#iommu-cells = <0>;
+		power-domains = <&power RK3328_PD_HEVC>;
 		status = "disabled";
 	};
 
@@ -592,6 +593,7 @@
 		clocks = <&cru ACLK_VPU>, <&cru HCLK_VPU>;
 		clock-names = "aclk", "iface";
 		#iommu-cells = <0>;
+		power-domains = <&power RK3328_PD_HEVC>;
 		status = "disabled";
 	};
 
@@ -603,6 +605,7 @@
 		clocks = <&cru ACLK_VPU>, <&cru HCLK_VPU>;
 		clock-names = "aclk", "iface";
 		#iommu-cells = <0>;
+		power-domains = <&power RK3328_PD_VPU>;
 		status = "disabled";
 	};
 
@@ -614,6 +617,7 @@
 		clocks = <&cru ACLK_RKVDEC>, <&cru HCLK_RKVDEC>;
 		clock-names = "aclk", "iface";
 		#iommu-cells = <0>;
+		power-domains = <&power RK3328_PD_VIDEO>;
 		status = "disabled";
 	};
 
arch/arm64/boot/dts/rockchip/rk3399.dtsi +2 -0
index db9d948c0b03..23d061ab6fa6 100644
@@ -1263,6 +1263,7 @@
 		clock-names = "aclk", "iface";
 		#iommu-cells = <0>;
 		power-domains = <&power RK3399_PD_VCODEC>;
+		status = "disabled";
 	};
 
 	vdec_mmu: iommu@ff660480 {
@@ -1273,6 +1274,7 @@
 		clocks = <&cru ACLK_VDU>, <&cru HCLK_VDU>;
 		clock-names = "aclk", "iface";
 		#iommu-cells = <0>;
+		power-domains = <&power RK3399_PD_VDU>;
 		status = "disabled";
 	};
 
Signature · 1 line
2.20.1
author
#2 lore raw

PATCH v2 2/6 staging: video: rockchip: add v4l2 decoder

In reply to Randy Li

From: Randy Li <ayaka-xPW3/0Ywev/[email protected]>

It is based on the vendor driver sent to mail list before.

Only MPEG2 video for VDPU2 works. And it need a patch to fill the QP table.

I have finished the register table of the rkvdec and rk hevc decoder. But I can't feed its proper input stream, I will talk the reason below.

The slice header reconstruction is not acceptable, even I don't care about VP9 now, the reference picture set for HEVC is very complex.

---
 drivers/staging/rockchip-mpp/Kconfig          |   34 +
 drivers/staging/rockchip-mpp/Makefile         |   10 +
 drivers/staging/rockchip-mpp/mpp_debug.h      |   87 ++
 drivers/staging/rockchip-mpp/mpp_dev_common.c | 1368 +++++++++++++++++
 drivers/staging/rockchip-mpp/mpp_dev_common.h |  212 +++
 drivers/staging/rockchip-mpp/mpp_dev_rkvdec.c |  997 ++++++++++++
 drivers/staging/rockchip-mpp/mpp_dev_vdpu2.c  |  616 ++++++++
 drivers/staging/rockchip-mpp/mpp_service.c    |  197 +++
 drivers/staging/rockchip-mpp/mpp_service.h    |   38 +
 drivers/staging/rockchip-mpp/rkvdec/hal.h     |   63 +
 drivers/staging/rockchip-mpp/rkvdec/hevc.c    |  166 ++
 drivers/staging/rockchip-mpp/rkvdec/regs.h    |  608 ++++++++
 drivers/staging/rockchip-mpp/vdpu2/hal.h      |   52 +
 drivers/staging/rockchip-mpp/vdpu2/mpeg2.c    |  270 ++++
 drivers/staging/rockchip-mpp/vdpu2/regs.h     |  699 +++++++++
 15 files changed, 5417 insertions(+)
 create mode 100644 drivers/staging/rockchip-mpp/Kconfig
 create mode 100644 drivers/staging/rockchip-mpp/Makefile
 create mode 100644 drivers/staging/rockchip-mpp/mpp_debug.h
 create mode 100644 drivers/staging/rockchip-mpp/mpp_dev_common.c
 create mode 100644 drivers/staging/rockchip-mpp/mpp_dev_common.h
 create mode 100644 drivers/staging/rockchip-mpp/mpp_dev_rkvdec.c
 create mode 100644 drivers/staging/rockchip-mpp/mpp_dev_vdpu2.c
 create mode 100644 drivers/staging/rockchip-mpp/mpp_service.c
 create mode 100644 drivers/staging/rockchip-mpp/mpp_service.h
 create mode 100644 drivers/staging/rockchip-mpp/rkvdec/hal.h
 create mode 100644 drivers/staging/rockchip-mpp/rkvdec/hevc.c
 create mode 100644 drivers/staging/rockchip-mpp/rkvdec/regs.h
 create mode 100644 drivers/staging/rockchip-mpp/vdpu2/hal.h
 create mode 100644 drivers/staging/rockchip-mpp/vdpu2/mpeg2.c
 create mode 100644 drivers/staging/rockchip-mpp/vdpu2/regs.h
drivers/staging/rockchip-mpp/Kconfig new file +34 -0
new file mode 100644
index 000000000000..75eaa6a2541a
@@ -0,0 +1,34 @@
+# SPDX-License-Identifier: GPL-2.0
+menu "ROCKCHIP_MPP"
+	depends on ARCH_ROCKCHIP
+
+config ROCKCHIP_MPP_SERVICE
+	tristate "mpp service scheduler"
+	default n
+	help
+	  rockchip mpp service.
+
+config ROCKCHIP_MPP_DEVICE
+	tristate "mpp device framework"
+	depends on ROCKCHIP_MPP_SERVICE
+	select V4L2_MEM2MEM_DEV
+	select VIDEOBUF2_DMA_CONTIG
+	default n
+	help
+	  rockchip mpp device framework.
+
+config ROCKCHIP_MPP_VDEC_DEVICE
+	tristate "video decoder device driver"
+	depends on ROCKCHIP_MPP_DEVICE
+	default n
+	help
+	  rockchip mpp video decoder and hevc decoder.
+
+config ROCKCHIP_MPP_VDPU2_DEVICE
+	tristate "VPU decoder v2 device driver"
+	depends on ROCKCHIP_MPP_DEVICE
+	default n
+	help
+	  rockchip mpp vpu decoder v2.
+
+endmenu
drivers/staging/rockchip-mpp/Makefile new file +10 -0
new file mode 100644
index 000000000000..c35ea3a13461
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: GPL-2.0
+rk-mpp-service-objs := mpp_service.o
+rk-mpp-device-objs := mpp_dev_common.o
+rk-mpp-vdec-objs := mpp_dev_rkvdec.o rkvdec/hevc.o
+rk-mpp-vdpu2-objs := mpp_dev_vdpu2.o vdpu2/mpeg2.o
+
+obj-$(CONFIG_ROCKCHIP_MPP_SERVICE) += rk-mpp-service.o
+obj-$(CONFIG_ROCKCHIP_MPP_DEVICE) += rk-mpp-device.o
+obj-$(CONFIG_ROCKCHIP_MPP_VDEC_DEVICE) += rk-mpp-vdec.o
+obj-$(CONFIG_ROCKCHIP_MPP_VDPU2_DEVICE) += rk-mpp-vdpu2.o
drivers/staging/rockchip-mpp/mpp_debug.h new file +87 -0

87 lines not on this page · raw message

drivers/staging/rockchip-mpp/mpp_dev_common.c new file +1368 -0

1,368 lines not on this page · raw message

drivers/staging/rockchip-mpp/mpp_dev_common.h new file +212 -0

212 lines not on this page · raw message

drivers/staging/rockchip-mpp/mpp_dev_rkvdec.c new file +997 -0

997 lines not on this page · raw message

drivers/staging/rockchip-mpp/mpp_dev_vdpu2.c new file +616 -0

292 lines not on this page · raw message

drivers/staging/rockchip-mpp/mpp_service.c new file +197 -0

197 lines not on this page · raw message

drivers/staging/rockchip-mpp/mpp_service.h new file +38 -0
new file mode 100644
index 000000000000..a77cff7b02df

Lines of this file are not shown. View the raw message.

drivers/staging/rockchip-mpp/rkvdec/hal.h new file +63 -0

63 lines not on this page · raw message

drivers/staging/rockchip-mpp/rkvdec/hevc.c new file +166 -0

166 lines not on this page · raw message

drivers/staging/rockchip-mpp/rkvdec/regs.h new file +608 -0

608 lines not on this page · raw message

drivers/staging/rockchip-mpp/vdpu2/hal.h new file +52 -0

52 lines not on this page · raw message

drivers/staging/rockchip-mpp/vdpu2/mpeg2.c new file +270 -0

270 lines not on this page · raw message

drivers/staging/rockchip-mpp/vdpu2/regs.h new file +699 -0

699 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.20.1

Part of the diff is not shown here. View the raw message.

#3 lore raw

Re: [PATCH v2 2/6] staging: video: rockchip: add v4l2 decoder

Quoted 1 line · Randy Li

It is based on the vendor driver sent to mail list before.

trivial notes:

drivers/staging/rockchip-mpp/mpp_debug.h

Quoted 1 line
drivers/staging/rockchip-mpp/mpp_debug.h +0 -0

[]

Quoted 4 lines
drivers/staging/rockchip-mpp/mpp_debug.h +4 -0
+#define mpp_debug_func(type, fmt, args...)			\
+	do {							\
+		if (unlikely(debug & type)) {			\
+			pr_info("%s:%d: " fmt,			\

drivers/staging/rockchip-mpp/mpp_debug.h line 67

Quoted 10 lines
drivers/staging/rockchip-mpp/mpp_debug.h +10 -0
+				 __func__, __LINE__, ##args);	\
+		}						\
+	} while (0)
+#define mpp_debug(type, fmt, args...)				\
+	do {							\
+		if (unlikely(debug & type)) {			\
+			pr_info(fmt, ##args);			\
+		}						\
+	} while (0)
+

It's generally better to emit debug messages at KERN_DEBUG

Quoted 5 lines
drivers/staging/rockchip-mpp/mpp_debug.h +5 -0
+#define mpp_debug_enter()					\
+	do {							\
+		if (unlikely(debug & DEBUG_FUNCTION)) {		\
+			pr_info("%s:%d: enter\n",		\
+				 __func__, __LINE__);		\

drivers/staging/rockchip-mpp/mpp_debug.h line 82

Quoted 10 lines
drivers/staging/rockchip-mpp/mpp_debug.h +10 -0
+		}						\
+	} while (0)
+
+#define mpp_debug_leave()					\
+	do {							\
+		if (unlikely(debug & DEBUG_FUNCTION)) {		\
+			pr_info("%s:%d: leave\n",		\
+				 __func__, __LINE__);		\
+		}						\
+	} while (0)

I suggest removal of these macros and uses.

There's not much value in enter/leave markings as
the generic ftrace facility does this already.

drivers/staging/rockchip-mpp/mpp_debug.h line 85

Quoted 3 lines
drivers/staging/rockchip-mpp/mpp_debug.h +3 -0
+
+#define mpp_err(fmt, args...)					\
+		pr_err("%s:%d: " fmt, __func__, __LINE__, ##args)

__func__, __LINE__ markings generally have little value.

author
#4 lore raw

PATCH v2 3/6 : rockchip: mpp: support qptable

In reply to Randy Li

I don't care, I don't want to store buffers for a session. I just want to use it to verify the FFmpeg.

I want the memory region !!!
It can save more time if those data are prepared in userspace.

---
 drivers/staging/rockchip-mpp/mpp_dev_common.c |  3 +--
 drivers/staging/rockchip-mpp/mpp_dev_vdpu2.c  |  3 +++
 drivers/staging/rockchip-mpp/vdpu2/mpeg2.c    | 19 +++++++++++++------
 3 files changed, 17 insertions(+), 8 deletions(-)
drivers/staging/rockchip-mpp/mpp_dev_common.c +1 -2
index c43304c3e7b8..1291b642179e 100644
@@ -1193,8 +1193,7 @@ static int rockchip_mpp_queue_init(void *priv, struct vb2_queue *src_vq,
 	src_vq->io_modes = VB2_MMAP | VB2_DMABUF;
 	src_vq->drv_priv = session;
 	src_vq->mem_ops = &vb2_dma_contig_memops;
-	src_vq->dma_attrs = DMA_ATTR_ALLOC_SINGLE_PAGES |
-			    DMA_ATTR_NO_KERNEL_MAPPING;
+	src_vq->dma_attrs = DMA_ATTR_ALLOC_SINGLE_PAGES;
 	src_vq->buf_struct_size = sizeof(struct v4l2_m2m_buffer);
 	src_vq->min_buffers_needed = 1;
 	src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
drivers/staging/rockchip-mpp/mpp_dev_vdpu2.c +3 -0
index ce98aa15025e..00413936623e 100644
@@ -147,6 +147,9 @@ static int rkvdpu_s_fmt_vid_out_mplane(struct file *filp, void *priv,
 	if (sizes >= SZ_16M)
 		return -EINVAL;
 
+	/* For those slice header data */
+	pix_mp->plane_fmt[pix_mp->num_planes - 1].sizeimage += SZ_1M;
+
 	session->fmt_out = *pix_mp;
 
 	/* Copy the pixel format information from OUTPUT to CAPUTRE */
drivers/staging/rockchip-mpp/vdpu2/mpeg2.c +13 -6
index d32958c4cb20..c12d1a8ef2da 100644
@@ -52,7 +52,7 @@ static const u8 intra_default_q_matrix[64] = {
 static void mpeg2_dec_copy_qtable(u8 * qtable, const struct v4l2_ctrl_mpeg2_quantization
 				  *ctrl)
 {
-	int i, n;
+	int i;
 
 	if (!qtable || !ctrl)
 		return;
@@ -111,21 +111,22 @@ int rkvdpu_mpeg2_gen_reg(struct mpp_session *session, void *regs,
 			 struct vb2_v4l2_buffer *src_buf)
 {
 	const struct v4l2_ctrl_mpeg2_slice_params *params;
-	const struct v4l2_ctrl_mpeg2_quantization *quantization;
 	const struct v4l2_mpeg2_sequence *sequence;
 	const struct v4l2_mpeg2_picture *picture;
+	const struct v4l2_ctrl_mpeg2_quantization *quantization;
 	struct vdpu2_regs *p_regs = regs;
+	void *qtable = NULL;
+	size_t stream_len = 0;
 
 	params = rockchip_mpp_get_cur_ctrl(session,
 					   V4L2_CID_MPEG_VIDEO_MPEG2_SLICE_PARAMS);
-	quantization = rockchip_mpp_get_cur_ctrl(session,
-						 V4L2_CID_MPEG_VIDEO_MPEG2_QUANTIZATION);
-
 	if (!params)
 		return -EINVAL;
 
 	sequence = &params->sequence;
 	picture = &params->picture;
+	quantization = rockchip_mpp_get_cur_ctrl(session,
+			V4L2_CID_MPEG_VIDEO_MPEG2_QUANTIZATION);
 
 	init_hw_cfg(p_regs);
 
@@ -202,7 +203,13 @@ int rkvdpu_mpeg2_gen_reg(struct mpp_session *session, void *regs,
 	p_regs->sw64.rlc_vlc_base =
 	    vb2_dma_contig_plane_dma_addr(&src_buf->vb2_buf, 0);
 	p_regs->sw122.strm_start_bit = params->data_bit_offset;
-	p_regs->sw51.stream_len = vb2_get_plane_payload(&src_buf->vb2_buf, 0);
+	stream_len = vb2_get_plane_payload(&src_buf->vb2_buf, 0);
+	p_regs->sw51.stream_len = stream_len;
+
+	qtable = vb2_plane_vaddr(&src_buf->vb2_buf, 0) + ALIGN(stream_len, 8);
+	mpeg2_dec_copy_qtable(qtable, quantization);
+        p_regs->sw61.qtable_base = p_regs->sw64.rlc_vlc_base
+		+ ALIGN(stream_len, 8);
 
 	return 0;
 }
Signature · 1 line
2.20.1
author
#5 lore raw

PATCH v2 4/6 staging: video: rockchip: add video codec

In reply to Randy Li

From: Randy Li <[email protected]>

---
 drivers/staging/Kconfig  | 2 ++
 drivers/staging/Makefile | 1 +
 2 files changed, 3 insertions(+)
drivers/staging/Kconfig +2 -0
index c0901b96cfe4..5f84035e2a89 100644
@@ -124,4 +124,6 @@ source "drivers/staging/axis-fifo/Kconfig"
 
 source "drivers/staging/erofs/Kconfig"
 
+source "drivers/staging/rockchip-mpp/Kconfig"
+
 endif # STAGING
drivers/staging/Makefile +1 -0
index 57c6bce13ff4..fc3ed97a0eab 100644
@@ -52,3 +52,4 @@ obj-$(CONFIG_SOC_MT7621)	+= mt7621-dts/
 obj-$(CONFIG_STAGING_GASKET_FRAMEWORK)	+= gasket/
 obj-$(CONFIG_XIL_AXIS_FIFO)	+= axis-fifo/
 obj-$(CONFIG_EROFS_FS)		+= erofs/
+obj-$(CONFIG_ROCKCHIP_MPP_SERVICE)	+= rockchip-mpp/
Signature · 1 line
2.20.1
#6 lore raw
In reply to Randy Li
Quoted 13 lines · Randy Li

Although I really want to push those work after I added more codec supports, but I found it is more urge to do those in v4l2 core framework and userspace.

I would use this driver to present the current problems, write down a summary here and I would reply to those threads later to push forward.

1. Slice construction is a bad idea. I think I have said my reason in the IRC and mail before, vp9 is always good example.

And it would request the driver to update QP table/CABAC table every slice.

Quoted 6 lines

I would make something to describe a buffer with some addtional meta data.

But the current request API limit a buffer with associated with an request buffer, which prevent sharing some sequence data, but it still can solve some problems.

I guess you are trying to make some argument here, but I don't really understand what you are referring to. Right now there is two concurrent drivers for the Rockchip, yours and Ezequiel. Ezequiel does not seem to have raised any blockers around any of this (yet).

Quoted 8 lines

2. Advantage DMA memory control.
I think I need to do some work at v4l2 core.

2.1 The DMA address of each planes. I have sent a mail before talked about why multiple planes is necessary for the rockchip platform. And it maybe required by the other platforms.

Quoted 5 lines

2.2 IOMMU resume
The most effective way to restore the decoder from critical error is doing a restting by reset controller.
Which would leading its slave IOMMU reset at the same time. Then none of those v4l2 buffers are mapping in the IOMMU.

You can't invalidate application memory mapping in V4L2, this is generic to V4L2 interface. If you still need to do this, you'll have to tell user-space through the SRC_CHANGE event, forcing a reconfiguration hence a reallocation of the buffers. Then your driver would be responsible for caching the allocation in order to not introduce delays. The framework does not prevent you from doing so, but yet, this is likely difficult.

Quoted 5 lines

3. H.264 and HEVC header
I still think those structure have some not necessary fileds in dpb or reference part, which I don't think hardware decoder would care about that or can be predict from the other information.

This was discussed during the review, but all the information in there exist in the in the slice headers bitstream. There is no reason some information should not be made available to the driver, used or not.

This is the only way we can guaranty that we won't prevent other HW to be integrated in the future. This is specially needed for H264_SLICE_RAW format used by Allwinner. In Ezequiel H264 decoder patchset for Rockchip HW, the format is H264_SLICE_ANNEX_B, of course in this format one could always do a bit of parsing in the kernel, but we don't want that really.

That being said, I'd like to remind that we don't expose the control publicly yet, these are still unstable. We will freeze these after drivers get added to mainline and are considered stable.

Quoted 6 lines

I would join to talk later.

4. The work flow of V4L2
I need a method to prepare the register set before the device acutally begin the transaction. Which is necessary for those high frame rate usecase.

Can't you just increase the buffer queue size ? Then prepare the buffers on application thread, and process on another or something. This seems like driver specific, not an API thing.

Quoted 5 lines

Also it is useful for those device would share some hardware resources with the other device and it can save more power.

I think I need to do some work at v4l2 core.

Seems all possible to optimize, imho, we should aim at getting a working driver into mainline first and then progressively tweak it to gain best performance. The driver and userspace need complete re- implementation, and it won't happen all in one pass. I'm sure there is multiple years of effort, and multiple iterations on the vendor kernel and userspace. We can learn from that, but my point is that we are not yet at a stage where we should focus on driver specific optimization.

From what you have said so far, I haven't found anything for which the kernel interface that is being merged would prevent doing the suggested optimization. Instead I read you as some of the interface decision will require a bit more work for this specific driver. This is often the tradeoff we have to do to make sure we can expose generically usable interface. And that's likely why mainline drivers are often a tad more complex then their vendor equivalent.

Quoted 46 lines

Randy Li (6):
arm64: dts: rockchip: add power domain to iommu
staging: video: rockchip: add v4l2 decoder [TEST]: rockchip: mpp: support qptable
staging: video: rockchip: add video codec
arm64: dts: rockchip: boost clocks for rk3328
arm64: dts: rockchip: add video codec for rk3328

 .../arm64/boot/dts/rockchip/rk3328-rock64.dts |   32 +
 arch/arm64/boot/dts/rockchip/rk3328.dtsi      |  116 +-
 arch/arm64/boot/dts/rockchip/rk3399.dtsi      |    2 +
 drivers/staging/Kconfig                       |    2 +
 drivers/staging/Makefile                      |    1 +
 drivers/staging/rockchip-mpp/Kconfig          |   34 +
 drivers/staging/rockchip-mpp/Makefile         |   10 +
 drivers/staging/rockchip-mpp/mpp_debug.h      |   87 ++
 drivers/staging/rockchip-mpp/mpp_dev_common.c | 1367 +++++++++++++++++
 drivers/staging/rockchip-mpp/mpp_dev_common.h |  212 +++
 drivers/staging/rockchip-mpp/mpp_dev_rkvdec.c |  997 ++++++++++++
 drivers/staging/rockchip-mpp/mpp_dev_vdpu2.c  |  619 ++++++++
 drivers/staging/rockchip-mpp/mpp_service.c    |  197 +++
 drivers/staging/rockchip-mpp/mpp_service.h    |   38 +
 drivers/staging/rockchip-mpp/rkvdec/hal.h     |   63 +
 drivers/staging/rockchip-mpp/rkvdec/hevc.c    |  166 ++
 drivers/staging/rockchip-mpp/rkvdec/regs.h    |  608 ++++++++
 drivers/staging/rockchip-mpp/vdpu2/hal.h      |   52 +
 drivers/staging/rockchip-mpp/vdpu2/mpeg2.c    |  277 ++++
 drivers/staging/rockchip-mpp/vdpu2/regs.h     |  699 +++++++++
 20 files changed, 5575 insertions(+), 4 deletions(-)
 create mode 100644 drivers/staging/rockchip-mpp/Kconfig
 create mode 100644 drivers/staging/rockchip-mpp/Makefile
 create mode 100644 drivers/staging/rockchip-mpp/mpp_debug.h
 create mode 100644 drivers/staging/rockchip-mpp/mpp_dev_common.c
 create mode 100644 drivers/staging/rockchip-mpp/mpp_dev_common.h
 create mode 100644 drivers/staging/rockchip-mpp/mpp_dev_rkvdec.c
 create mode 100644 drivers/staging/rockchip-mpp/mpp_dev_vdpu2.c
 create mode 100644 drivers/staging/rockchip-mpp/mpp_service.c
 create mode 100644 drivers/staging/rockchip-mpp/mpp_service.h
 create mode 100644 drivers/staging/rockchip-mpp/rkvdec/hal.h
 create mode 100644 drivers/staging/rockchip-mpp/rkvdec/hevc.c
 create mode 100644 drivers/staging/rockchip-mpp/rkvdec/regs.h
 create mode 100644 drivers/staging/rockchip-mpp/vdpu2/hal.h
 create mode 100644 drivers/staging/rockchip-mpp/vdpu2/mpeg2.c
 create mode 100644 drivers/staging/rockchip-mpp/vdpu2/regs.h