Skip to contentSkip to conversation

support mt8188 h264 encoder

PATCH In review Irui Wang 3 participants 10 messages Jul 20, 2022 – Aug 26, 2022

Subsystems (4)
lore
Share & tools
lore

Irui Wang

author
#0 lore raw

v2 cover support mt8188 h264 encoder

mt8188 encoder hardware has a change compared to the previous chipsets, it supports 34-bit iova access. For 34-bit iova, the encoder driver needs to write all 34 bits to hardware, otherwise encoder will encode fail.

Patch 1 add the 34-bit iova supporting in driver.
Patch 2~3 add dt-bindings and driver compatible and private data. Patch 4~6 some modification for in driver.

---
compare with v1:
- split vpu_enc_encode function into 32bits and 34bits.
- some review comments modification.
---

Irui Wang (6):
media: mediatek: vcodec: Add encoder driver support for 34-bit iova
dt-bindings: media: mediatek: vcodec: Add encoder dt-bindings for

    mt8188

media: mediatek: vcodec: Add mt8188 encoder driver
media: mediatek: vcodec: Remove encoder driver get IRQ resource
media: mediatek: vcodec: Fix bitstream crop information error
media: mediatek: vcodec: Use ctx vb2_queue mutex instead of device
mutex

 .../media/mediatek,vcodec-encoder.yaml        |   1 +
 .../platform/mediatek/vcodec/mtk_vcodec_drv.h |   6 +
 .../platform/mediatek/vcodec/mtk_vcodec_enc.c |  14 +-
 .../mediatek/vcodec/mtk_vcodec_enc_drv.c      |  22 +-
 .../mediatek/vcodec/venc/venc_h264_if.c       | 200 +++++++++++++++---
 .../platform/mediatek/vcodec/venc_ipi_msg.h   |  24 +++
 .../platform/mediatek/vcodec/venc_vpu_if.c    |  76 ++++++-
 7 files changed, 291 insertions(+), 52 deletions(-)
Signature · 1 line
2.18.0
author
#1 lore raw

PATCH v2 1/6 media: mediatek: vcodec: Add encoder driver support for 34-bit iova

Encoder driver got iova from IOMMU is 34-bit, for example:

Here is the sample code:
encoder input frame buffer dma address is:
frm_buf =

    vb2_dma_contig_plane_dma_addr(&vb2_v4l2_buffer->vb2_buf, 0);

the value of frm_buf is 0x1_ff30_0000.

encoder driver got the frm_buf and send the iova to SCP firmware through SCP IPI message, then write to encoder hardware in SCP. The iova is stored in IPI message as uint32_t data type, so the value will be truncated from *0x1_ff30_0000* to *0xff30_0000*, and then *0xff30_0000* will be written to encoder hardware, but IOMMU will help to add the high *0x1_* bit back, so IOMMU can translate the iova to PA correctly, encoder hardware can access the correct memory for encoding.
Another reason to do this is the encoder hardware can't access the 34-bit iova, IOMMU will help to add the remaining high bits of iova. But for mt8188, encoder hardware can access 34-bit iova directly, and encoder driver need write all 34 bits because IOMMU can't help driver do this if the hardware support access 34-bit iova.
For the reasons above, this patch is added to support transfer 34-bit iova between kernel and SCP encoder driver. Use uint64_t data type to store the iova, for compatibility with old chipsets, add some new struct definitions for 34-bit.

---
 .../platform/mediatek/vcodec/mtk_vcodec_drv.h |   3 +
 .../mediatek/vcodec/venc/venc_h264_if.c       | 200 +++++++++++++++---
 .../platform/mediatek/vcodec/venc_ipi_msg.h   |  24 +++
 .../platform/mediatek/vcodec/venc_vpu_if.c    |  76 ++++++-
 4 files changed, 266 insertions(+), 37 deletions(-)
drivers/media/platform/mediatek/vcodec/mtk_vcodec_drv.h +3 -0
index ef4584a46417..fe8ed6a456e0 100644
@@ -401,6 +401,7 @@ struct mtk_vcodec_dec_pdata {
  * @output_formats: array of supported output formats
  * @num_output_formats: number of entries in output_formats
  * @core_id: stand for h264 or vp8 encode index
+ * @uses_34bit: whether the encoder uses 34-bit iova
  */
 struct mtk_vcodec_enc_pdata {
 	bool uses_ext;
@@ -411,9 +412,11 @@ struct mtk_vcodec_enc_pdata {
 	const struct mtk_video_fmt *output_formats;
 	size_t num_output_formats;
 	int core_id;
+	bool uses_34bit;
 };
 
 #define MTK_ENC_CTX_IS_EXT(ctx) ((ctx)->dev->venc_pdata->uses_ext)
+#define MTK_ENC_IOVA_IS_34BIT(ctx) ((ctx)->dev->venc_pdata->uses_34bit)
 
 /**
  * struct mtk_vcodec_dev - driver data
drivers/media/platform/mediatek/vcodec/venc/venc_h264_if.c +167 -33

309 lines not on this page · raw message

drivers/media/platform/mediatek/vcodec/venc_ipi_msg.h +24 -0
index 587a2cf15b76..bb16d96a7f57 100644
@@ -100,6 +100,30 @@ struct venc_ap_ipi_msg_enc_ext {
 	uint32_t data[32];
 };
 
+/**
+ * struct venc_ap_ipi_msg_enc_ext_34 - AP to SCP extended enc cmd structure
+ * @msg_id:		message id (AP_IPIMSG_XXX_ENC_ENCODE)
+ * @vpu_inst_addr:	VPU encoder instance addr
+ * @bs_mode:		bitstream mode for h264
+ * @reserved:		for struct padding
+ * @input_addr:		input frame buffer 34 bit address
+ * @bs_addr:		output bitstream buffer 34 bit address
+ * @bs_size:		bitstream buffer size
+ * @data_item:		number of items in the data array
+ * @data:		data array to store the set parameters
+ */
+struct venc_ap_ipi_msg_enc_ext_34 {
+	u32 msg_id;
+	u32 vpu_inst_addr;
+	u32 bs_mode;
+	u32 reserved;
+	u64 input_addr[3];
+	u64 bs_addr;
+	u32 bs_size;
+	u32 data_item;
+	u32 data[32];
+};
+
 /**
  * struct venc_ap_ipi_msg_deinit - AP to VPU deinit cmd structure
  * @msg_id:	message id (AP_IPIMSG_XXX_ENC_DEINIT)
drivers/media/platform/mediatek/vcodec/venc_vpu_if.c +72 -4

88 lines not on this page · raw message

Signature · 1 line
2.18.0
author
#2 lore raw

PATCH v2 2/6 dt-bindings: media: mediatek: vcodec: Add encoder dt-bindings for mt8188

In reply to Irui Wang

Add encoder dt-bindings for mt8188.

---
 .../devicetree/bindings/media/mediatek,vcodec-encoder.yaml       | 1 +
 1 file changed, 1 insertion(+)
Documentation/devicetree/bindings/media/mediatek,vcodec-encoder.yaml +1 -0
index d36fcca04cbc..32aee09aea33 100644
@@ -20,6 +20,7 @@ properties:
       - mediatek,mt8173-vcodec-enc-vp8
       - mediatek,mt8173-vcodec-enc
       - mediatek,mt8183-vcodec-enc
+      - mediatek,mt8188-vcodec-enc
       - mediatek,mt8192-vcodec-enc
       - mediatek,mt8195-vcodec-enc
 
Signature · 1 line
2.18.0
maintainer Acked-by on v2 2/6
#3 lore raw

Re: [PATCH v2, 2/6] dt-bindings: media: mediatek: vcodec: Add encoder dt-bindings for mt8188

Quoted 7 lines · Irui Wang

Add encoder dt-bindings for mt8188.

---
 .../devicetree/bindings/media/mediatek,vcodec-encoder.yaml       | 1 +
 1 file changed, 1 insertion(+)
author
#4 lore raw

PATCH v2 3/6 media: mediatek: vcodec: Add mt8188 encoder driver

In reply to Irui Wang

Add mt8188's compatible "mediatek,mt8188-vcodec-enc". Add mt8188's device private data "mt8188_pdata".

---
 .../platform/mediatek/vcodec/mtk_vcodec_enc_drv.c   | 13 +++++++++++++
 1 file changed, 13 insertions(+)
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_drv.c +13 -0
index 95e8c29ccc65..ea667b867b56 100644
@@ -406,6 +406,18 @@ static const struct mtk_vcodec_enc_pdata mt8183_pdata = {
 	.core_id = VENC_SYS,
 };
 
+static const struct mtk_vcodec_enc_pdata mt8188_pdata = {
+	.uses_ext = true,
+	.capture_formats = mtk_video_formats_capture_h264,
+	.num_capture_formats = ARRAY_SIZE(mtk_video_formats_capture_h264),
+	.output_formats = mtk_video_formats_output,
+	.num_output_formats = ARRAY_SIZE(mtk_video_formats_output),
+	.min_bitrate = 64,
+	.max_bitrate = 50000000,
+	.core_id = VENC_SYS,
+	.uses_34bit = true,
+};
+
 static const struct mtk_vcodec_enc_pdata mt8192_pdata = {
 	.uses_ext = true,
 	.capture_formats = mtk_video_formats_capture_h264,
@@ -434,6 +446,7 @@ static const struct of_device_id mtk_vcodec_enc_match[] = {
 	{.compatible = "mediatek,mt8173-vcodec-enc-vp8",
 			.data = &mt8173_vp8_pdata},
 	{.compatible = "mediatek,mt8183-vcodec-enc", .data = &mt8183_pdata},
+	{.compatible = "mediatek,mt8188-vcodec-enc", .data = &mt8188_pdata},
 	{.compatible = "mediatek,mt8192-vcodec-enc", .data = &mt8192_pdata},
 	{.compatible = "mediatek,mt8195-vcodec-enc", .data = &mt8195_pdata},
 	{},
Signature · 1 line
2.18.0
author
#5 lore raw

PATCH v2 4/6 media: mediatek: vcodec: Remove encoder driver get IRQ resource

In reply to Irui Wang

The "platform_get_resource(pdev, IORESOURCE_IRQ, 0)" is no longer used after commit a1a2b7125e107("of/platform: Drop static setup of IRQ resource from DT core"), so just remove the function in encoder driver to avoid driver probe failed.

---
 .../media/platform/mediatek/vcodec/mtk_vcodec_enc_drv.c   | 8 --------
 1 file changed, 8 deletions(-)
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_drv.c +0 -8
index ea667b867b56..6d8964fb4fa2 100644
@@ -228,7 +228,6 @@ static int mtk_vcodec_probe(struct platform_device *pdev)
 {
 	struct mtk_vcodec_dev *dev;
 	struct video_device *vfd_enc;
-	struct resource *res;
 	phandle rproc_phandle;
 	enum mtk_vcodec_fw_type fw_type;
 	int ret;
@@ -272,13 +271,6 @@ static int mtk_vcodec_probe(struct platform_device *pdev)
 		goto err_res;
 	}
 
-	res = platform_get_resource(pdev, IORESOURCE_IRQ, 0);
-	if (res == NULL) {
-		dev_err(&pdev->dev, "failed to get irq resource");
-		ret = -ENOENT;
-		goto err_res;
-	}
-
 	dev->enc_irq = platform_get_irq(pdev, 0);
 	irq_set_status_flags(dev->enc_irq, IRQ_NOAUTOEN);
 	ret = devm_request_irq(&pdev->dev, dev->enc_irq,
Signature · 1 line
2.18.0
#6 lore raw

Re: [PATCH v2, 4/6] media: mediatek: vcodec: Remove encoder driver get IRQ resource

Hi,

Quoted 25 lines · Irui Wang

The "platform_get_resource(pdev, IORESOURCE_IRQ, 0)" is no longer used after commit a1a2b7125e107("of/platform: Drop static setup of IRQ resource from DT core"), so just remove the function in encoder driver to avoid driver probe failed.

---
 .../media/platform/mediatek/vcodec/mtk_vcodec_enc_drv.c   | 8 --------
 1 file changed, 8 deletions(-)
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_drv.c +0 -1
index ea667b867b56..6d8964fb4fa2 100644
@@ -228,7 +228,6 @@ static int mtk_vcodec_probe(struct platform_device *pdev)
 {
        struct mtk_vcodec_dev *dev;
        struct video_device *vfd_enc;
-       struct resource *res;
        phandle rproc_phandle;
        enum mtk_vcodec_fw_type fw_type;
        int ret;
@@ -272,13 +271,6 @@ static int mtk_vcodec_probe(struct platform_device *pdev)
                goto err_res;

drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_drv.c static int mtk_vcodec_probe(struct platform_device *pdev) · line 274

Quoted 10 lines
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_drv.c +0 -7
        }
 
-       res = platform_get_resource(pdev, IORESOURCE_IRQ, 0);
-       if (res == NULL) {
-               dev_err(&pdev->dev, "failed to get irq resource");
-               ret = -ENOENT;
-               goto err_res;
-       }
-
        dev->enc_irq = platform_get_irq(pdev, 0);

platform_get_irq() returns a negative number on failure. You should check for errors here as the previous check was removed.

The description of platform_get_irq() explicitly mentions:

Device drivers should check the return value for errors so as to not pass a negative integer value to the request_irq() APIs.

Hans has queued up this patch, so maybe you could provide a follow-up patch to fix this?

Regards
ChenYu

Quoted 5 lines
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_drv.c +0 -1
        irq_set_status_flags(dev->enc_irq, IRQ_NOAUTOEN);
        ret = devm_request_irq(&pdev->dev, dev->enc_irq,
--

2.18.0

author
#7 lore raw

Re: [PATCH v2, 4/6] media: mediatek: vcodec: Remove encoder driver get IRQ resource

Dear Chen-Yu,

many thanks for your comments.

Quoted 48 lines · Chen-Yu Tsai

Hi,

Quoted 30 lines · Irui Wang

The "platform_get_resource(pdev, IORESOURCE_IRQ, 0)" is no longer used after commit a1a2b7125e107("of/platform: Drop static setup of IRQ resource from DT core"), so just remove the function in encoder driver to avoid driver probe failed.

---
 .../media/platform/mediatek/vcodec/mtk_vcodec_enc_drv.c   | 8 ----
----
 1 file changed, 8 deletions(-)

diff --git a/drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_drv.c b/drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_drv.c index ea667b867b56..6d8964fb4fa2 100644

drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_drv.c +0 -1
@@ -228,7 +228,6 @@ static int mtk_vcodec_probe(struct
 platform_device *pdev)
 {
        struct mtk_vcodec_dev *dev;
        struct video_device *vfd_enc;
-       struct resource *res;
        phandle rproc_phandle;
        enum mtk_vcodec_fw_type fw_type;
        int ret;
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_drv.c +0 -0
@@ -272,13 +271,6 @@ static int mtk_vcodec_probe(struct
 platform_device *pdev)
                goto err_res;

drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_drv.c static int mtk_vcodec_probe(struct · line 275

Quoted 10 lines
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_drv.c +0 -7
        }
 
-       res = platform_get_resource(pdev, IORESOURCE_IRQ, 0);
-       if (res == NULL) {
-               dev_err(&pdev->dev, "failed to get irq resource");
-               ret = -ENOENT;
-               goto err_res;
-       }
-
        dev->enc_irq = platform_get_irq(pdev, 0);

platform_get_irq() returns a negative number on failure. You should check for errors here as the previous check was removed.

Quoted 8 lines

The description of platform_get_irq() explicitly mentions:

Device drivers should check the return value for errors so as to not pass a negative integer value to the request_irq() APIs.

Hans has queued up this patch, so maybe you could provide a follow-up patch to fix this?

Another same patch has been already merged into linux-next tree, so this current patch can be dropped.

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_drv.c?id=750a914adb7ded3669b47daed6d9d749110b4107

Thanks
Best Regards

Quoted 10 lines

Regards
ChenYu

Quoted 5 lines
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_drv.c +0 -0
        irq_set_status_flags(dev->enc_irq, IRQ_NOAUTOEN);
        ret = devm_request_irq(&pdev->dev, dev->enc_irq,

--
2.18.0

author
#8 lore raw

PATCH v2 5/6 media: mediatek: vcodec: Fix bitstream crop information error

In reply to Irui Wang

Usually, the real bitstream width and height will set to driver by vidioc_s_fmt, and vidioc_try_fmt() does align to get the buffer width and height, driver calculate the encoded bitstream crop information through them. The aligned resolution will be set as real resolution now if user didn't set crop info by V4L2_SEL_TGT_CROP, and the encoded bitstream may exist green line because of crop information error.

Fixs: 'b6c57d313f5f8 ("media: mtk-vcodec: venc: remove redundant code")'

---
 drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c +4 -4
index 25e816863597..c310bb1dbbcf 100644
@@ -503,13 +503,13 @@ static int vidioc_venc_s_fmt_out(struct file *file, void *priv,
 		f->fmt.pix.pixelformat = fmt->fourcc;
 	}
 
-	ret = vidioc_try_fmt_out(ctx, f, fmt);
+	q_data->visible_width = f->fmt.pix_mp.width;
+	q_data->visible_height = f->fmt.pix_mp.height;
+	q_data->fmt = fmt;
+	ret = vidioc_try_fmt_out(ctx, f, q_data->fmt);
 	if (ret)
 		return ret;
 
-	q_data->fmt = fmt;
-	q_data->visible_width = f->fmt.pix_mp.width;
-	q_data->visible_height = f->fmt.pix_mp.height;
 	q_data->coded_width = f->fmt.pix_mp.width;
 	q_data->coded_height = f->fmt.pix_mp.height;
 
Signature · 1 line
2.18.0
author
#9 lore raw

PATCH v2 6/6 media: mediatek: vcodec: Use ctx vb2_queue mutex instead of device mutex

In reply to Irui Wang

There is only one device mutex to lock vb2_queue when running multi-instance encoding, it can be set by each encoder context.

---
 drivers/media/platform/mediatek/vcodec/mtk_vcodec_drv.h     | 3 +++
 drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c     | 6 +++---
 drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_drv.c | 1 +
 3 files changed, 7 insertions(+), 3 deletions(-)
drivers/media/platform/mediatek/vcodec/mtk_vcodec_drv.h +3 -0
index fe8ed6a456e0..2a12f13557ac 100644
@@ -324,6 +324,9 @@ struct mtk_vcodec_ctx {
 	int hw_id;
 
 	struct vdec_msg_queue msg_queue;
+
+	/*q_mutex: vb2_queue mutex*/
+	struct mutex q_mutex;
 };
 
 /*
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c +3 -3
index c310bb1dbbcf..63e7fe958406 100644
@@ -1300,7 +1300,7 @@ void mtk_vcodec_enc_set_default_params(struct mtk_vcodec_ctx *ctx)
 {
 	struct mtk_q_data *q_data;
 
-	ctx->m2m_ctx->q_lock = &ctx->dev->dev_mutex;
+	ctx->m2m_ctx->q_lock = &ctx->q_mutex;
 	ctx->fh.m2m_ctx = ctx->m2m_ctx;
 	ctx->fh.ctrl_handler = &ctx->ctrl_hdl;
 	INIT_WORK(&ctx->encode_work, mtk_venc_worker);
@@ -1435,7 +1435,7 @@ int mtk_vcodec_enc_queue_init(void *priv, struct vb2_queue *src_vq,
 	src_vq->ops		= &mtk_venc_vb2_ops;
 	src_vq->mem_ops		= &vb2_dma_contig_memops;
 	src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
-	src_vq->lock		= &ctx->dev->dev_mutex;
+	src_vq->lock		= &ctx->q_mutex;
 	src_vq->dev		= &ctx->dev->plat_dev->dev;
 
 	ret = vb2_queue_init(src_vq);
@@ -1449,7 +1449,7 @@ int mtk_vcodec_enc_queue_init(void *priv, struct vb2_queue *src_vq,
 	dst_vq->ops		= &mtk_venc_vb2_ops;
 	dst_vq->mem_ops		= &vb2_dma_contig_memops;
 	dst_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
-	dst_vq->lock		= &ctx->dev->dev_mutex;
+	dst_vq->lock		= &ctx->q_mutex;
 	dst_vq->dev		= &ctx->dev->plat_dev->dev;
 
 	return vb2_queue_init(dst_vq);
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_drv.c +1 -0
index 6d8964fb4fa2..0abe1dac75b3 100644
@@ -130,6 +130,7 @@ static int fops_vcodec_open(struct file *file)
 	INIT_LIST_HEAD(&ctx->list);
 	ctx->dev = dev;
 	init_waitqueue_head(&ctx->queue[0]);
+	mutex_init(&ctx->q_mutex);
 
 	ctx->type = MTK_INST_ENCODER;
 	ret = mtk_vcodec_enc_ctrls_setup(ctx);
Signature · 1 line
2.18.0