Skip to contentSkip to conversation

how to use stateless-mpeg2-vp8-h264-v4 on imx8m with vpu?

DISCUSSION Martin Kepplinger 2 participants 4 messages Jun 30, 2020 Media (V4L/DVB)

lore
Share & tools
lore
#0 lore raw

Hi Ezequiel,

On the Librem 5 (imx8mq) I try to decode a (h264) full hd video file using the VPU.

I'm running the following tree that doesn't change much, but adds the VPU dts description:
https://source.puri.sm/martin.kepplinger/linux-next/-/commits/5.8-rc3/librem5___vpu
(output of "v4l-ctl --all" at the bottom).

I now run your version of ffmpeg:
https://gitlab.collabora.com/ezequiel/ffmpeg/-/commits/stateless-mpeg2-vp8-h264-v4
(with minor build fixes) but simply doing "ffplay <file>" seems like not using /dev/video0

Is this supposed to work? If so, do I need to know anything about ffmpeg to be able to test this? How do you verify that the v4l2 mem2mem device (vpu) is being used?

thanks a lot,
martin

purism@pureos:~/ffmpeg_sources/ffmpeg$ v4l2-ctl --all
Driver Info:
	Driver name      : hantro-vpu
	Card type        : nxp,imx8mq-vpu-dec
	Bus info         : platform: hantro-vpu
	Driver version   : 5.8.0
	Capabilities     : 0x84204000
		Video Memory-to-Memory Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04204000
		Video Memory-to-Memory Multiplanar
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : hantro-vpu
	Model            : hantro-vpu
	Serial           :
	Bus info         : platform: hantro-vpu
	Media version    : 5.8.0
	Hardware revision: 0x00000000 (0)
	Driver version   : 5.8.0
Interface Info:
	ID               : 0x0300000c
	Type             : V4L Video
Entity Info:
	ID               : 0x00000001 (1)
	Name             : nxp,imx8mq-vpu-dec-source
	Function         : V4L2 I/O
	Pad 0x01000002   : 0: Source
	  Link 0x02000008: to remote pad 0x1000004 of entity
'nxp,imx8mq-vpu-dec-proc': Data, Enabled, Immutable
Priority: 2
Format Video Capture Multiplanar:
	Width/Height      : 48/48
	Pixel Format      : 'NV12' (Y/CbCr 4:2:0)
	Field             : None
	Number of planes  : 1
	Flags             :
	Colorspace        : JPEG
	Transfer Function : Default
	YCbCr/HSV Encoding: Default
	Quantization      : Default
	Plane 0           :
	   Bytes per Line : 48
	   Size Image     : 3456
Format Video Output Multiplanar:
	Width/Height      : 48/48
	Pixel Format      : 'MG2S' (MPEG-2 Parsed Slice Data)
	Field             : None
	Number of planes  : 1
	Flags             :
	Colorspace        : JPEG
	Transfer Function : Default
	YCbCr/HSV Encoding: Default
	Quantization      : Default
	Plane 0           :
	   Bytes per Line : 0
	   Size Image     : 4608

Codec Controls

        mpeg_2_slice_parameters 0x009909fa (unknown): type=103
flags=has-payload
   mpeg_2_quantization_matrices 0x009909fb (unknown): type=104
flags=has-payload
                   h264_profile 0x00990a6b (menu)   : min=0 max=4
default=2 value=2
    h264_sequence_parameter_set 0x00990ce8 (unknown): type=110
flags=has-payload
     h264_picture_parameter_set 0x00990ce9 (unknown): type=111
flags=has-payload
            h264_scaling_matrix 0x00990cea (unknown): type=112
flags=has-payload
          h264_slice_parameters 0x00990ceb (unknown): type=113
flags=has-payload
         h264_decode_parameters 0x00990cec (unknown): type=114
flags=has-payload
               h264_decode_mode 0x00990ced (menu)   : min=1 max=1
default=1 value=1
                h264_start_code 0x00990cee (menu)   : min=1 max=1
default=1 value=1
               vp8_frame_header 0x009910d0 (unknown): type=301

flags=has-payload

#1 lore raw

Hi Martin,

Quoted 10 lines · Martin Kepplinger

Hi Ezequiel,

On the Librem 5 (imx8mq) I try to decode a (h264) full hd video file using the VPU.

I'm running the following tree that doesn't change much, but adds the VPU dts description:
https://source.puri.sm/martin.kepplinger/linux-next/-/commits/5.8-rc3/librem5___vpu
(output of "v4l-ctl --all" at the bottom).

OK, this means the VPU is enabled with the Hantro driver, so far so good.

Quoted 10 lines

I now run your version of ffmpeg:
https://gitlab.collabora.com/ezequiel/ffmpeg/-/commits/stateless-mpeg2-vp8-h264-v4
(with minor build fixes) but simply doing "ffplay <file>" seems like not using /dev/video0

Is this supposed to work? If so, do I need to know anything about ffmpeg to be able to test this? How do you verify that the v4l2 mem2mem device (vpu) is being used?

Not sure what state the branch is on, and also IIRC you should use that together with mpv for proper zero-copy operation.

I think the libre elec community can help you with that, as they maintain proper ffmpeg/mpv branches.

Do you need this to work with ffmpeg, or is GStreamer also OK?

Thanks,
Ezequiel

Quoted 90 lines

thanks a lot,
martin

purism@pureos:~/ffmpeg_sources/ffmpeg$ v4l2-ctl --all
Driver Info:
	Driver name      : hantro-vpu
	Card type        : nxp,imx8mq-vpu-dec
	Bus info         : platform: hantro-vpu
	Driver version   : 5.8.0
	Capabilities     : 0x84204000
		Video Memory-to-Memory Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04204000
		Video Memory-to-Memory Multiplanar
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : hantro-vpu
	Model            : hantro-vpu
	Serial           :
	Bus info         : platform: hantro-vpu
	Media version    : 5.8.0
	Hardware revision: 0x00000000 (0)
	Driver version   : 5.8.0
Interface Info:
	ID               : 0x0300000c
	Type             : V4L Video
Entity Info:
	ID               : 0x00000001 (1)
	Name             : nxp,imx8mq-vpu-dec-source
	Function         : V4L2 I/O
	Pad 0x01000002   : 0: Source
	  Link 0x02000008: to remote pad 0x1000004 of entity
'nxp,imx8mq-vpu-dec-proc': Data, Enabled, Immutable
Priority: 2
Format Video Capture Multiplanar:
	Width/Height      : 48/48
	Pixel Format      : 'NV12' (Y/CbCr 4:2:0)
	Field             : None
	Number of planes  : 1
	Flags             :
	Colorspace        : JPEG
	Transfer Function : Default
	YCbCr/HSV Encoding: Default
	Quantization      : Default
	Plane 0           :
	   Bytes per Line : 48
	   Size Image     : 3456
Format Video Output Multiplanar:
	Width/Height      : 48/48
	Pixel Format      : 'MG2S' (MPEG-2 Parsed Slice Data)
	Field             : None
	Number of planes  : 1
	Flags             :
	Colorspace        : JPEG
	Transfer Function : Default
	YCbCr/HSV Encoding: Default
	Quantization      : Default
	Plane 0           :
	   Bytes per Line : 0
	   Size Image     : 4608

Codec Controls

        mpeg_2_slice_parameters 0x009909fa (unknown): type=103
flags=has-payload
   mpeg_2_quantization_matrices 0x009909fb (unknown): type=104
flags=has-payload
                   h264_profile 0x00990a6b (menu)   : min=0 max=4
default=2 value=2
    h264_sequence_parameter_set 0x00990ce8 (unknown): type=110
flags=has-payload
     h264_picture_parameter_set 0x00990ce9 (unknown): type=111
flags=has-payload
            h264_scaling_matrix 0x00990cea (unknown): type=112
flags=has-payload
          h264_slice_parameters 0x00990ceb (unknown): type=113
flags=has-payload
         h264_decode_parameters 0x00990cec (unknown): type=114
flags=has-payload
               h264_decode_mode 0x00990ced (menu)   : min=1 max=1
default=1 value=1
                h264_start_code 0x00990cee (menu)   : min=1 max=1
default=1 value=1
               vp8_frame_header 0x009910d0 (unknown): type=301

flags=has-payload

#2 lore raw
Quoted 27 lines · Ezequiel Garcia

Hi Martin,

Quoted 10 lines · Martin Kepplinger

Hi Ezequiel,

On the Librem 5 (imx8mq) I try to decode a (h264) full hd video file using the VPU.

I'm running the following tree that doesn't change much, but adds the VPU dts description:
https://source.puri.sm/martin.kepplinger/linux-next/-/commits/5.8-rc3/librem5___vpu
(output of "v4l-ctl --all" at the bottom).

OK, this means the VPU is enabled with the Hantro driver, so far so good.

Quoted 10 lines

I now run your version of ffmpeg:
https://gitlab.collabora.com/ezequiel/ffmpeg/-/commits/stateless-mpeg2-vp8-h264-v4
(with minor build fixes) but simply doing "ffplay <file>" seems like not using /dev/video0

Is this supposed to work? If so, do I need to know anything about ffmpeg to be able to test this? How do you verify that the v4l2 mem2mem device (vpu) is being used?

Quoted 5 lines

Not sure what state the branch is on, and also IIRC you should use that together with mpv for proper zero-copy operation.

I think the libre elec community can help you with that, as they maintain proper ffmpeg/mpv branches.

thanks for the hint.

Quoted 2 lines

Do you need this to work with ffmpeg, or is GStreamer also OK?

gstreamer would be equally fine (or easier as I've used that before) for me. Last time I checked I think I'd have to build many more libraries from source in order to test though. That's the only reason why I've tried ffmpeg.

Quoted 96 lines

Thanks,
Ezequiel

Quoted 90 lines

thanks a lot,
martin

purism@pureos:~/ffmpeg_sources/ffmpeg$ v4l2-ctl --all
Driver Info:
	Driver name      : hantro-vpu
	Card type        : nxp,imx8mq-vpu-dec
	Bus info         : platform: hantro-vpu
	Driver version   : 5.8.0
	Capabilities     : 0x84204000
		Video Memory-to-Memory Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04204000
		Video Memory-to-Memory Multiplanar
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : hantro-vpu
	Model            : hantro-vpu
	Serial           :
	Bus info         : platform: hantro-vpu
	Media version    : 5.8.0
	Hardware revision: 0x00000000 (0)
	Driver version   : 5.8.0
Interface Info:
	ID               : 0x0300000c
	Type             : V4L Video
Entity Info:
	ID               : 0x00000001 (1)
	Name             : nxp,imx8mq-vpu-dec-source
	Function         : V4L2 I/O
	Pad 0x01000002   : 0: Source
	  Link 0x02000008: to remote pad 0x1000004 of entity
'nxp,imx8mq-vpu-dec-proc': Data, Enabled, Immutable
Priority: 2
Format Video Capture Multiplanar:
	Width/Height      : 48/48
	Pixel Format      : 'NV12' (Y/CbCr 4:2:0)
	Field             : None
	Number of planes  : 1
	Flags             :
	Colorspace        : JPEG
	Transfer Function : Default
	YCbCr/HSV Encoding: Default
	Quantization      : Default
	Plane 0           :
	   Bytes per Line : 48
	   Size Image     : 3456
Format Video Output Multiplanar:
	Width/Height      : 48/48
	Pixel Format      : 'MG2S' (MPEG-2 Parsed Slice Data)
	Field             : None
	Number of planes  : 1
	Flags             :
	Colorspace        : JPEG
	Transfer Function : Default
	YCbCr/HSV Encoding: Default
	Quantization      : Default
	Plane 0           :
	   Bytes per Line : 0
	   Size Image     : 4608

Codec Controls

        mpeg_2_slice_parameters 0x009909fa (unknown): type=103
flags=has-payload
   mpeg_2_quantization_matrices 0x009909fb (unknown): type=104
flags=has-payload
                   h264_profile 0x00990a6b (menu)   : min=0 max=4
default=2 value=2
    h264_sequence_parameter_set 0x00990ce8 (unknown): type=110
flags=has-payload
     h264_picture_parameter_set 0x00990ce9 (unknown): type=111
flags=has-payload
            h264_scaling_matrix 0x00990cea (unknown): type=112
flags=has-payload
          h264_slice_parameters 0x00990ceb (unknown): type=113
flags=has-payload
         h264_decode_parameters 0x00990cec (unknown): type=114
flags=has-payload
               h264_decode_mode 0x00990ced (menu)   : min=1 max=1
default=1 value=1
                h264_start_code 0x00990cee (menu)   : min=1 max=1
default=1 value=1
               vp8_frame_header 0x009910d0 (unknown): type=301

flags=has-payload

#3 lore raw
Quoted 33 lines · Martin Kepplinger
Quoted 26 lines · Ezequiel Garcia

Hi Martin,

Quoted 10 lines · Martin Kepplinger

Hi Ezequiel,

On the Librem 5 (imx8mq) I try to decode a (h264) full hd video file using the VPU.

I'm running the following tree that doesn't change much, but adds the VPU dts description:
https://source.puri.sm/martin.kepplinger/linux-next/-/commits/5.8-rc3/librem5___vpu
(output of "v4l-ctl --all" at the bottom).

OK, this means the VPU is enabled with the Hantro driver, so far so good.

Quoted 9 lines

I now run your version of ffmpeg:
https://gitlab.collabora.com/ezequiel/ffmpeg/-/commits/stateless-mpeg2-vp8-h264-v4
(with minor build fixes) but simply doing "ffplay <file>" seems like not using /dev/video0

Is this supposed to work? If so, do I need to know anything about ffmpeg to be able to test this? How do you verify that the v4l2 mem2mem device (vpu) is being used?

Quoted 5 lines

Not sure what state the branch is on, and also IIRC you should use that together with mpv for proper zero-copy operation.

I think the libre elec community can help you with that, as they maintain proper ffmpeg/mpv branches.

Quoted 9 lines

thanks for the hint.

Quoted 1 line

Do you need this to work with ffmpeg, or is GStreamer also OK?

gstreamer would be equally fine (or easier as I've used that before) for me. Last time I checked I think I'd have to build many more libraries from source in order to test though. That's the only reason why I've tried ffmpeg.

In my experience, using gst-build is not too hard; combined with meson, it's really fast to cross build.

There are a couple articles:

https://www.collabora.com/news-and-blog/blog/2020/03/19/getting-started-with-gstreamer-gst-build/
https://www.collabora.com/news-and-blog/blog/2020/05/15/cross-compiling-with-gst-build-and-gstreamer/

.. or tl;dr off the top of my head, hope it works:

(host) cd whatever_your_nfs_rootfs_is/gst-build
(host) meson --cross-file /home/user/meson-cross/arm64.txt build (host) ninja -C build

(The second article explains how to generate a cross-file)

Then on the target:

(target) cd gst-build/
(target) ./gst-uninstalled.py

Few examples that have worked out of the box. The GL ones might or might not work, depending on your support.

gst-launch-1.0 filesrc location=/$1 ! parsebin ! decodebin3 ! fakevideosink
gst-launch-1.0 -ev filesrc location=$1 ! decodebin3 ! videoconvert ! video/x-raw,format=YUY2 ! glupload ! glfiltercube ! glimagesink gst-launch-1.0 -ev filesrc location=$1 ! decodebin3 ! videoconvert ! video/x-raw,format=NV12 ! kmssink

You'll have VPU interrupts and should see nice CPU usage. The GStreamer filter element that decodebin3 should pick up is "v4l2slh264dec".

Let us know if it works :-)

Thanks,
Ezequiel