media: amphion: decoder add support to RealMedia

PATCH Superseded Ming Qian 2 participants 5 messages Dec 20, 2022 – Jan 12, 2023

Subsystems (2)

Media (V4L/DVB) Amphion VPU codec V4L2 driver

lore

Share & tools

lore

Ming Qian

author maintainer

2022-12-20 08:39 UTC #0 lore raw

cover media: amphion: decoder add support to RealMedia

RealMedia is a proprietary multimedia container format created by RealNetworks with the filename extension .rm.

amphion decoder can support decoding this video format.

Ming Qian (2):
media: add RealMedia format
media: amphion: support to decode RealMedia video

 .../media/v4l/pixfmt-compressed.rst           | 11 +++++++++++
 drivers/media/platform/amphion/vdec.c         |  7 +++++++
 drivers/media/platform/amphion/vpu_malone.c   | 19 +++++++++++++++++++
 drivers/media/platform/amphion/vpu_malone.h   |  1 +
 drivers/media/v4l2-core/v4l2-ioctl.c          |  1 +
 include/uapi/linux/videodev2.h                |  1 +
 6 files changed, 40 insertions(+)

Signature · 1 line

2.38.1

Ming Qian

author maintainer

2022-12-20 08:40 UTC #1 lore raw

PATCH 1/2 media: add RealMedia format

RealMedia is a proprietary multimedia container format created by RealNetworks with the filename extension .rm. RealMedia is generally used in conjunction with RealVideo and RealAudio, while also being used for streaming content over the Internet.

Signed-off-by Ming Qian <[email protected]>

---
 .../userspace-api/media/v4l/pixfmt-compressed.rst     | 11 +++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c                  |  1 +
 include/uapi/linux/videodev2.h                        |  1 +
 3 files changed, 13 insertions(+)

Documentation/userspace-api/media/v4l/pixfmt-compressed.rst +11 -0

index 8794b92fde36..31ba2c00091e 100644
@@ -237,6 +237,17 @@ Compressed Formats
         Metadata associated with the frame to decode is required to be passed
         through the ``V4L2_CID_STATELESS_FWHT_PARAMS`` control.
 	See the :ref:`associated Codec Control ID <codec-stateless-fwht>`.
+    * .. _V4L2-PIX-FMT-RV:
+
+      - ``V4L2_PIX_FMT_RV``
+      - 'RealMedia'
+      - RealMedia is a proprietary multimedia container format
+        created by RealNetworks with the filename extension .rm.
+        RealMedia is generally used in conjunction with RealVideo and RealAudio,
+        while also being used for streaming content over the Internet.
+        Typically these streams are in CBR (constant bitrate),
+        but a container for VBR (variable bitrate) streams
+        named RMVB (RealMedia variable bitrate) has been developed.
 
 .. raw:: latex

drivers/media/v4l2-core/v4l2-ioctl.c +1 -0

index 9b5b04b8aa69..a4d12eea7fc5 100644
@@ -1473,6 +1473,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 		case V4L2_PIX_FMT_FWHT:		descr = "FWHT"; break; /* used in vicodec */
 		case V4L2_PIX_FMT_FWHT_STATELESS:	descr = "FWHT Stateless"; break; /* used in vicodec */
 		case V4L2_PIX_FMT_SPK:		descr = "Sorenson Spark"; break;
+		case V4L2_PIX_FMT_RV:		descr = "RealMedia"; break;
 		case V4L2_PIX_FMT_CPIA1:	descr = "GSPCA CPiA YUV"; break;
 		case V4L2_PIX_FMT_WNVA:		descr = "WNVA"; break;
 		case V4L2_PIX_FMT_SN9C10X:	descr = "GSPCA SN9C10X"; break;

include/uapi/linux/videodev2.h +1 -0

index 262ef10cfa02..a7a7969ae4f8 100644
@@ -740,6 +740,7 @@ struct v4l2_pix_format {
 #define V4L2_PIX_FMT_H264_SLICE v4l2_fourcc('S', '2', '6', '4') /* H264 parsed slices */
 #define V4L2_PIX_FMT_HEVC_SLICE v4l2_fourcc('S', '2', '6', '5') /* HEVC parsed slices */
 #define V4L2_PIX_FMT_SPK      v4l2_fourcc('S', 'P', 'K', '0') /* Sorenson Spark */
+#define V4L2_PIX_FMT_RV       v4l2_fourcc('R', 'V', '0', '0') /* RealMedia */
 
 /*  Vendor-specific formats   */
 #define V4L2_PIX_FMT_CPIA1    v4l2_fourcc('C', 'P', 'I', 'A') /* cpia1 YUV */

Signature · 1 line

2.38.1

Nicolas Dufresne

2023-01-11 14:29 UTC #2 lore raw

Re: [PATCH 1/2] media: add RealMedia format

Hi Ming,

sorry for the late reply ...

Le mardi 20 décembre 2022 à 16:39 +0800, Ming Qian a écrit :

Quoted 21 lines

RealMedia is a proprietary multimedia container format created by RealNetworks with the filename extension .rm. RealMedia is generally used in conjunction with RealVideo and RealAudio, while also being used for streaming content over the Internet.
Signed-off-by Ming Qian <[email protected]>
---
 .../userspace-api/media/v4l/pixfmt-compressed.rst     | 11 +++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c                  |  1 +
 include/uapi/linux/videodev2.h                        |  1 +
 3 files changed, 13 insertions(+)
Documentation/userspace-api/media/v4l/pixfmt-compressed.rst +1 -0
index 8794b92fde36..31ba2c00091e 100644
@@ -237,6 +237,17 @@ Compressed Formats
         Metadata associated with the frame to decode is required to be passed
         through the ``V4L2_CID_STATELESS_FWHT_PARAMS`` control.
 	See the :ref:`associated Codec Control ID <codec-stateless-fwht>`.
+    * .. _V4L2-PIX-FMT-RV:

Documentation/userspace-api/media/v4l/pixfmt-compressed.rst Compressed Formats · line 250

Quoted 10 lines

Documentation/userspace-api/media/v4l/pixfmt-compressed.rst +10 -0

+
+      - ``V4L2_PIX_FMT_RV``
+      - 'RealMedia'
+      - RealMedia is a proprietary multimedia container format
+        created by RealNetworks with the filename extension .rm.
+        RealMedia is generally used in conjunction with RealVideo and RealAudio,
+        while also being used for streaming content over the Internet.
+        Typically these streams are in CBR (constant bitrate),
+        but a container for VBR (variable bitrate) streams
+        named RMVB (RealMedia variable bitrate) has been developed.

I'm a bit confused with the description here. It describes the RealMedia (RM) container format, but what you wanted is a RealVideo encoding (RV).

A quick search into FFMpeg (the only Open Source software decoder I could find), the RealVideo comes in 4 incompatible flavours, RV10/RV20/RV30/RV40 also known as RealVideo 1 / G2 / 8 / 9+10. I think the format here should be a RealVideo format, and it should specify the flavours you want to support (probably only RV40).

regards,
Nicolas

Quoted 20 lines

Documentation/userspace-api/media/v4l/pixfmt-compressed.rst +0 -0

 .. raw:: latex

drivers/media/v4l2-core/v4l2-ioctl.c +1 -0

index 9b5b04b8aa69..a4d12eea7fc5 100644
@@ -1473,6 +1473,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 		case V4L2_PIX_FMT_FWHT:		descr = "FWHT"; break; /* used in vicodec */
 		case V4L2_PIX_FMT_FWHT_STATELESS:	descr = "FWHT Stateless"; break; /* used in vicodec */
 		case V4L2_PIX_FMT_SPK:		descr = "Sorenson Spark"; break;
+		case V4L2_PIX_FMT_RV:		descr = "RealMedia"; break;
 		case V4L2_PIX_FMT_CPIA1:	descr = "GSPCA CPiA YUV"; break;
 		case V4L2_PIX_FMT_WNVA:		descr = "WNVA"; break;
 		case V4L2_PIX_FMT_SN9C10X:	descr = "GSPCA SN9C10X"; break;

include/uapi/linux/videodev2.h +0 -0

index 262ef10cfa02..a7a7969ae4f8 100644

include/uapi/linux/videodev2.h struct v4l2_pix_format { · line 746

Quoted 7 lines

include/uapi/linux/videodev2.h +1 -0

@@ -740,6 +740,7 @@ struct v4l2_pix_format {
 #define V4L2_PIX_FMT_H264_SLICE v4l2_fourcc('S', '2', '6', '4') /* H264 parsed slices */
 #define V4L2_PIX_FMT_HEVC_SLICE v4l2_fourcc('S', '2', '6', '5') /* HEVC parsed slices */
 #define V4L2_PIX_FMT_SPK      v4l2_fourcc('S', 'P', 'K', '0') /* Sorenson Spark */
+#define V4L2_PIX_FMT_RV       v4l2_fourcc('R', 'V', '0', '0') /* RealMedia */
 
 /*  Vendor-specific formats   */
 #define V4L2_PIX_FMT_CPIA1    v4l2_fourcc('C', 'P', 'I', 'A') /* cpia1 YUV */

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Ming Qian

author maintainer

2023-01-12 01:17 UTC #3 lore raw

RE: [EXT] Re: [PATCH 1/2] media: add RealMedia format

Quoted 56 lines

From: Nicolas Dufresne <[email protected]>
Sent: 2023年1月11日 22:28
To: Ming Qian <[email protected]>; [email protected]; hverkuil- [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; dl-linux-imx <linux- [email protected]>; X.H. Bao <[email protected]>; Ming Zhou <[email protected]>; [email protected]; linux- [email protected]; [email protected]
Subject: [EXT] Re: [PATCH 1/2] media: add RealMedia format
Caution: EXT Email
Hi Ming,
sorry for the late reply ...
Quoted 19 lines · Ming Qian
RealMedia is a proprietary multimedia container format created by RealNetworks with the filename extension .rm.
RealMedia is generally used in conjunction with RealVideo and RealAudio, while also being used for streaming content over the Internet.
Signed-off-by Ming Qian <[email protected]>
---
 .../userspace-api/media/v4l/pixfmt-compressed.rst     | 11 +++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c                  |  1 +
 include/uapi/linux/videodev2.h                        |  1 +
 3 files changed, 13 insertions(+)
diff --git a/Documentation/userspace-api/media/v4l/pixfmt-compressed.rst b/Documentation/userspace-api/media/v4l/pixfmt-compressed.rst index 8794b92fde36..31ba2c00091e 100644
Documentation/userspace-api/media/v4l/pixfmt-compressed.rst Compressed Formats · line 246
Quoted 10 lines
Documentation/userspace-api/media/v4l/pixfmt-compressed.rst +7 -0
@@ -237,6 +237,17 @@ Compressed Formats
         Metadata associated with the frame to decode is required to be passed
         through the ``V4L2_CID_STATELESS_FWHT_PARAMS`` control.
      See the :ref:`associated Codec Control ID <codec-stateless-fwht>`.
+    * .. _V4L2-PIX-FMT-RV:
+
+      - ``V4L2_PIX_FMT_RV``
+      - 'RealMedia'
+      - RealMedia is a proprietary multimedia container format
+        created by RealNetworks with the filename extension .rm.
+        RealMedia is generally used in conjunction with RealVideo and
RealAudio,
Documentation/userspace-api/media/v4l/pixfmt-compressed.rst Compressed Formats · line 250
Quoted 4 lines
Documentation/userspace-api/media/v4l/pixfmt-compressed.rst +4 -0
+        while also being used for streaming content over the Internet.
+        Typically these streams are in CBR (constant bitrate),
+        but a container for VBR (variable bitrate) streams
+        named RMVB (RealMedia variable bitrate) has been developed.
I'm a bit confused with the description here. It describes the RealMedia (RM) container format, but what you wanted is a RealVideo encoding (RV).

Quoted 8 lines

A quick search into FFMpeg (the only Open Source software decoder I could find), the RealVideo comes in 4 incompatible flavours, RV10/RV20/RV30/RV40 also known as RealVideo 1 / G2 / 8 / 9+10. I think the format here should be a RealVideo format, and it should specify the flavours you want to support (probably only RV40).
regards,
Nicolas

Hi Nicolas,
Yes, you're right, I confused the encoding format with the container format,
I'll check and correct it.
Thanks very much

Ming

Quoted 35 lines

drivers/media/v4l2-core/v4l2-ioctl.c static void v4l_fill_fmtdesc(struct v4l2_fmtdesc
Quoted 10 lines
Documentation/userspace-api/media/v4l/pixfmt-compressed.rst +0 -0
 .. raw:: latex
 
drivers/media/v4l2-core/v4l2-ioctl.c +0 -0
index 9b5b04b8aa69..a4d12eea7fc5 100644
@@ -1473,6 +1473,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc
*fmt)
drivers/media/v4l2-core/v4l2-ioctl.c static void v4l_fill_fmtdesc(struct v4l2_fmtdesc · line 1473
Quoted 1 line
drivers/media/v4l2-core/v4l2-ioctl.c +0 -0
              case V4L2_PIX_FMT_FWHT:         descr = "FWHT"; break; /* used in
vicodec */
drivers/media/v4l2-core/v4l2-ioctl.c static void v4l_fill_fmtdesc(struct v4l2_fmtdesc · line 1474
Quoted 1 line
drivers/media/v4l2-core/v4l2-ioctl.c +0 -0
              case V4L2_PIX_FMT_FWHT_STATELESS:       descr = "FWHT Stateless";
break; /* used in vicodec */
Quoted 11 lines
drivers/media/v4l2-core/v4l2-ioctl.c +1 -0
              case V4L2_PIX_FMT_SPK:          descr = "Sorenson Spark"; break;
+             case V4L2_PIX_FMT_RV:           descr = "RealMedia"; break;
              case V4L2_PIX_FMT_CPIA1:        descr = "GSPCA CPiA YUV"; break;
              case V4L2_PIX_FMT_WNVA:         descr = "WNVA"; break;
              case V4L2_PIX_FMT_SN9C10X:      descr = "GSPCA SN9C10X"; break;
(quoted diff) +0 -0
b/include/uapi/linux/videodev2.h index 262ef10cfa02..a7a7969ae4f8 100644
include/uapi/linux/videodev2.h +0 -0
@@ -740,6 +740,7 @@ struct v4l2_pix_format {  #define
Quoted 2 lines
V4L2_PIX_FMT_H264_SLICE v4l2_fourcc('S', '2', '6', '4') /* H264 parsed slices */ #define V4L2_PIX_FMT_HEVC_SLICE v4l2_fourcc('S', '2', '6', '5') /*
HEVC parsed slices */
Quoted 1 line
#define V4L2_PIX_FMT_SPK v4l2_fourcc('S', 'P', 'K', '0') /* Sorenson Spark
*/
Quoted 4 lines
+#define V4L2_PIX_FMT_RV v4l2_fourcc('R', 'V', '0', '0') /* RealMedia */
/* Vendor-specific formats */
#define V4L2_PIX_FMT_CPIA1 v4l2_fourcc('C', 'P', 'I', 'A') /* cpia1 YUV */

Ming Qian

author maintainer

2022-12-20 08:40 UTC #4 lore raw

PATCH 2/2 media: amphion: support to decode RealMedia video

In reply to Ming Qian

amphion decoder add support for the RealMedia video

Signed-off-by Ming Qian <[email protected]>

---
 drivers/media/platform/amphion/vdec.c       |  7 +++++++
 drivers/media/platform/amphion/vpu_malone.c | 19 +++++++++++++++++++
 drivers/media/platform/amphion/vpu_malone.h |  1 +
 3 files changed, 27 insertions(+)

drivers/media/platform/amphion/vdec.c +7 -0

index 09304b96f40d..7e47465ee749 100644
@@ -172,6 +172,13 @@ static const struct vpu_format vdec_formats[] = {
 		.type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE,
 		.flags = V4L2_FMT_FLAG_DYN_RESOLUTION | V4L2_FMT_FLAG_COMPRESSED
 	},
+	{
+		.pixfmt = V4L2_PIX_FMT_RV,
+		.mem_planes = 1,
+		.comp_planes = 1,
+		.type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE,
+		.flags = V4L2_FMT_FLAG_DYN_RESOLUTION | V4L2_FMT_FLAG_COMPRESSED
+	},
 	{0, 0, 0, 0},
 };

drivers/media/platform/amphion/vpu_malone.c +19 -0

index 67ba637c4c7f..6e3626b74c7a 100644
@@ -68,6 +68,8 @@
 #define STREAM_CONFIG_NUM_DBE_SET(x, y)		CONFIG_SET(x, y, 30, 0x40000000)
 #define STREAM_CONFIG_FS_CTRL_MODE_SET(x, y)	CONFIG_SET(x, y, 31, 0x80000000)
 
+#define MALONE_DEC_FMT_RV_MASK			BIT(21)
+
 enum vpu_malone_stream_input_mode {
 	INVALID_MODE = 0,
 	FRAME_LVL,

@@ -478,6 +480,8 @@ u32 vpu_malone_get_version(struct vpu_shared_addr *shared)
 {
 	struct malone_iface *iface = shared->iface;
 
+	vpu_malone_enable_format(V4L2_PIX_FMT_RV, iface->fw_version & MALONE_DEC_FMT_RV_MASK);
+
 	return iface->fw_version;
 }

@@ -563,8 +567,21 @@ static struct malone_fmt_mapping fmt_mappings[] = {
 	{V4L2_PIX_FMT_JPEG,        MALONE_FMT_JPG},
 	{V4L2_PIX_FMT_VP8,         MALONE_FMT_VP8},
 	{V4L2_PIX_FMT_SPK,         MALONE_FMT_SPK},
+	{V4L2_PIX_FMT_RV,          MALONE_FMT_RV},
 };
 
+void vpu_malone_enable_format(u32 pixelformat, int enable)
+{
+	u32 i;
+
+	for (i = 0; i < ARRAY_SIZE(fmt_mappings); i++) {
+		if (pixelformat == fmt_mappings[i].pixelformat) {
+			fmt_mappings[i].is_disabled = enable ? 0 : 1;
+			return;
+		}
+	}
+}
+
 static enum vpu_malone_format vpu_malone_format_remap(u32 pixelformat)
 {
 	u32 i;

@@ -989,6 +1006,7 @@ static const struct malone_padding_scode padding_scodes[] = {
 	{SCODE_PADDING_EOS,      V4L2_PIX_FMT_H263,        {0xb1010000, 0x0}},
 	{SCODE_PADDING_EOS,      V4L2_PIX_FMT_VP8,         {0x34010000, 0x0}},
 	{SCODE_PADDING_EOS,      V4L2_PIX_FMT_SPK,         {0x34010000, 0x0}},
+	{SCODE_PADDING_EOS,      V4L2_PIX_FMT_RV,          {0x34010000, 0x0}},
 	{SCODE_PADDING_EOS,      V4L2_PIX_FMT_JPEG,        {0xefff0000, 0x0}},
 	{SCODE_PADDING_ABORT,    V4L2_PIX_FMT_H264,        {0x0B010000, 0}},
 	{SCODE_PADDING_ABORT,    V4L2_PIX_FMT_H264_MVC,    {0x0B010000, 0}},

@@ -1001,6 +1019,7 @@ static const struct malone_padding_scode padding_scodes[] = {
 	{SCODE_PADDING_ABORT,    V4L2_PIX_FMT_H263,        {0xb1010000, 0x0}},
 	{SCODE_PADDING_ABORT,    V4L2_PIX_FMT_VP8,         {0x34010000, 0x0}},
 	{SCODE_PADDING_ABORT,    V4L2_PIX_FMT_SPK,         {0x34010000, 0x0}},
+	{SCODE_PADDING_ABORT,    V4L2_PIX_FMT_RV,          {0x34010000, 0x0}},
 	{SCODE_PADDING_EOS,      V4L2_PIX_FMT_JPEG,        {0x0, 0x0}},
 	{SCODE_PADDING_BUFFLUSH, V4L2_PIX_FMT_H264,        {0x15010000, 0x0}},
 	{SCODE_PADDING_BUFFLUSH, V4L2_PIX_FMT_H264_MVC,    {0x15010000, 0x0}},

drivers/media/platform/amphion/vpu_malone.h +1 -0

index 02a9d9530970..c95b53629199 100644
@@ -41,5 +41,6 @@ int vpu_malone_post_cmd(struct vpu_shared_addr *shared, u32 instance);
 int vpu_malone_init_instance(struct vpu_shared_addr *shared, u32 instance);
 u32 vpu_malone_get_max_instance_count(struct vpu_shared_addr *shared);
 bool vpu_malone_check_fmt(enum vpu_core_type type, u32 pixelfmt);
+void vpu_malone_enable_format(u32 pixelformat, int enable);
 
 #endif

Signature · 1 line

2.38.1