how to use stateless-mpeg2-vp8-h264-v4 on imx8m with vpu?
Hi Ezequiel,
On the Librem 5 (imx8mq) I try to decode a (h264) full hd video file using the VPU.
I'm running the following tree that doesn't change much, but adds the VPU dts description:
https://source.puri.sm/martin.kepplinger/linux-next/-/commits/5.8-rc3/librem5___vpu
(output of "v4l-ctl --all" at the bottom).
I now run your version of ffmpeg:
https://gitlab.collabora.com/ezequiel/ffmpeg/-/commits/stateless-mpeg2-vp8-h264-v4
(with minor build fixes) but simply doing "ffplay <file>" seems like not using /dev/video0
Is this supposed to work? If so, do I need to know anything about ffmpeg to be able to test this? How do you verify that the v4l2 mem2mem device (vpu) is being used?
thanks a lot,
martin
purism@pureos:~/ffmpeg_sources/ffmpeg$ v4l2-ctl --all Driver Info: Driver name : hantro-vpu Card type : nxp,imx8mq-vpu-dec Bus info : platform: hantro-vpu Driver version : 5.8.0 Capabilities : 0x84204000 Video Memory-to-Memory Multiplanar Streaming Extended Pix Format Device Capabilities Device Caps : 0x04204000 Video Memory-to-Memory Multiplanar Streaming Extended Pix Format Media Driver Info: Driver name : hantro-vpu Model : hantro-vpu Serial : Bus info : platform: hantro-vpu Media version : 5.8.0 Hardware revision: 0x00000000 (0) Driver version : 5.8.0 Interface Info: ID : 0x0300000c Type : V4L Video Entity Info: ID : 0x00000001 (1) Name : nxp,imx8mq-vpu-dec-source Function : V4L2 I/O Pad 0x01000002 : 0: Source Link 0x02000008: to remote pad 0x1000004 of entity 'nxp,imx8mq-vpu-dec-proc': Data, Enabled, Immutable Priority: 2 Format Video Capture Multiplanar: Width/Height : 48/48 Pixel Format : 'NV12' (Y/CbCr 4:2:0) Field : None Number of planes : 1 Flags : Colorspace : JPEG Transfer Function : Default YCbCr/HSV Encoding: Default Quantization : Default Plane 0 : Bytes per Line : 48 Size Image : 3456 Format Video Output Multiplanar: Width/Height : 48/48 Pixel Format : 'MG2S' (MPEG-2 Parsed Slice Data) Field : None Number of planes : 1 Flags : Colorspace : JPEG Transfer Function : Default YCbCr/HSV Encoding: Default Quantization : Default Plane 0 : Bytes per Line : 0 Size Image : 4608
Codec Controls
mpeg_2_slice_parameters 0x009909fa (unknown): type=103
flags=has-payload
mpeg_2_quantization_matrices 0x009909fb (unknown): type=104
flags=has-payload
h264_profile 0x00990a6b (menu) : min=0 max=4
default=2 value=2
h264_sequence_parameter_set 0x00990ce8 (unknown): type=110
flags=has-payload
h264_picture_parameter_set 0x00990ce9 (unknown): type=111
flags=has-payload
h264_scaling_matrix 0x00990cea (unknown): type=112
flags=has-payload
h264_slice_parameters 0x00990ceb (unknown): type=113
flags=has-payload
h264_decode_parameters 0x00990cec (unknown): type=114
flags=has-payload
h264_decode_mode 0x00990ced (menu) : min=1 max=1
default=1 value=1
h264_start_code 0x00990cee (menu) : min=1 max=1
default=1 value=1
vp8_frame_header 0x009910d0 (unknown): type=301flags=has-payload
Hi Martin,
Quoted 10 lines · Martin Kepplinger
Hi Ezequiel,
On the Librem 5 (imx8mq) I try to decode a (h264) full hd video file using the VPU.
I'm running the following tree that doesn't change much, but adds the VPU dts description:
https://source.puri.sm/martin.kepplinger/linux-next/-/commits/5.8-rc3/librem5___vpu
(output of "v4l-ctl --all" at the bottom).
OK, this means the VPU is enabled with the Hantro driver, so far so good.
Quoted 10 lines
I now run your version of ffmpeg:
https://gitlab.collabora.com/ezequiel/ffmpeg/-/commits/stateless-mpeg2-vp8-h264-v4
(with minor build fixes) but simply doing "ffplay <file>" seems like not using/dev/video0Is this supposed to work? If so, do I need to know anything about ffmpeg to be able to test this? How do you verify that the v4l2 mem2mem device (vpu) is being used?
Not sure what state the branch is on, and also IIRC you should use that together with mpv for proper zero-copy operation.
I think the libre elec community can help you with that, as they maintain proper ffmpeg/mpv branches.
Do you need this to work with ffmpeg, or is GStreamer also OK?
Thanks,
Ezequiel
Quoted 90 lines
thanks a lot,
martinpurism@pureos:~/ffmpeg_sources/ffmpeg$ v4l2-ctl --all Driver Info: Driver name : hantro-vpu Card type : nxp,imx8mq-vpu-dec Bus info : platform: hantro-vpu Driver version : 5.8.0 Capabilities : 0x84204000 Video Memory-to-Memory Multiplanar Streaming Extended Pix Format Device Capabilities Device Caps : 0x04204000 Video Memory-to-Memory Multiplanar Streaming Extended Pix Format Media Driver Info: Driver name : hantro-vpu Model : hantro-vpu Serial : Bus info : platform: hantro-vpu Media version : 5.8.0 Hardware revision: 0x00000000 (0) Driver version : 5.8.0 Interface Info: ID : 0x0300000c Type : V4L Video Entity Info: ID : 0x00000001 (1) Name : nxp,imx8mq-vpu-dec-source Function : V4L2 I/O Pad 0x01000002 : 0: Source Link 0x02000008: to remote pad 0x1000004 of entity 'nxp,imx8mq-vpu-dec-proc': Data, Enabled, Immutable Priority: 2 Format Video Capture Multiplanar: Width/Height : 48/48 Pixel Format : 'NV12' (Y/CbCr 4:2:0) Field : None Number of planes : 1 Flags : Colorspace : JPEG Transfer Function : Default YCbCr/HSV Encoding: Default Quantization : Default Plane 0 : Bytes per Line : 48 Size Image : 3456 Format Video Output Multiplanar: Width/Height : 48/48 Pixel Format : 'MG2S' (MPEG-2 Parsed Slice Data) Field : None Number of planes : 1 Flags : Colorspace : JPEG Transfer Function : Default YCbCr/HSV Encoding: Default Quantization : Default Plane 0 : Bytes per Line : 0 Size Image : 4608Codec Controls
mpeg_2_slice_parameters 0x009909fa (unknown): type=103 flags=has-payload mpeg_2_quantization_matrices 0x009909fb (unknown): type=104 flags=has-payload h264_profile 0x00990a6b (menu) : min=0 max=4 default=2 value=2 h264_sequence_parameter_set 0x00990ce8 (unknown): type=110 flags=has-payload h264_picture_parameter_set 0x00990ce9 (unknown): type=111 flags=has-payload h264_scaling_matrix 0x00990cea (unknown): type=112 flags=has-payload h264_slice_parameters 0x00990ceb (unknown): type=113 flags=has-payload h264_decode_parameters 0x00990cec (unknown): type=114 flags=has-payload h264_decode_mode 0x00990ced (menu) : min=1 max=1 default=1 value=1 h264_start_code 0x00990cee (menu) : min=1 max=1 default=1 value=1 vp8_frame_header 0x009910d0 (unknown): type=301flags=has-payload
Quoted 27 lines · Ezequiel Garcia
Hi Martin,
Quoted 10 lines · Martin Kepplinger
Hi Ezequiel,
On the Librem 5 (imx8mq) I try to decode a (h264) full hd video file using the VPU.
I'm running the following tree that doesn't change much, but adds the VPU dts description:
https://source.puri.sm/martin.kepplinger/linux-next/-/commits/5.8-rc3/librem5___vpu
(output of "v4l-ctl --all" at the bottom).OK, this means the VPU is enabled with the Hantro driver, so far so good.
Quoted 10 lines
I now run your version of ffmpeg:
https://gitlab.collabora.com/ezequiel/ffmpeg/-/commits/stateless-mpeg2-vp8-h264-v4
(with minor build fixes) but simply doing "ffplay <file>" seems like not using/dev/video0Is this supposed to work? If so, do I need to know anything about ffmpeg to be able to test this? How do you verify that the v4l2 mem2mem device (vpu) is being used?
Quoted 5 lines
Not sure what state the branch is on, and also IIRC you should use that together with mpv for proper zero-copy operation.
I think the libre elec community can help you with that, as they maintain proper ffmpeg/mpv branches.
thanks for the hint.
Quoted 2 lines
Do you need this to work with ffmpeg, or is GStreamer also OK?
gstreamer would be equally fine (or easier as I've used that before) for me. Last time I checked I think I'd have to build many more libraries from source in order to test though. That's the only reason why I've tried ffmpeg.
Quoted 96 lines
Thanks,
EzequielQuoted 90 lines
thanks a lot,
martinpurism@pureos:~/ffmpeg_sources/ffmpeg$ v4l2-ctl --all Driver Info: Driver name : hantro-vpu Card type : nxp,imx8mq-vpu-dec Bus info : platform: hantro-vpu Driver version : 5.8.0 Capabilities : 0x84204000 Video Memory-to-Memory Multiplanar Streaming Extended Pix Format Device Capabilities Device Caps : 0x04204000 Video Memory-to-Memory Multiplanar Streaming Extended Pix Format Media Driver Info: Driver name : hantro-vpu Model : hantro-vpu Serial : Bus info : platform: hantro-vpu Media version : 5.8.0 Hardware revision: 0x00000000 (0) Driver version : 5.8.0 Interface Info: ID : 0x0300000c Type : V4L Video Entity Info: ID : 0x00000001 (1) Name : nxp,imx8mq-vpu-dec-source Function : V4L2 I/O Pad 0x01000002 : 0: Source Link 0x02000008: to remote pad 0x1000004 of entity 'nxp,imx8mq-vpu-dec-proc': Data, Enabled, Immutable Priority: 2 Format Video Capture Multiplanar: Width/Height : 48/48 Pixel Format : 'NV12' (Y/CbCr 4:2:0) Field : None Number of planes : 1 Flags : Colorspace : JPEG Transfer Function : Default YCbCr/HSV Encoding: Default Quantization : Default Plane 0 : Bytes per Line : 48 Size Image : 3456 Format Video Output Multiplanar: Width/Height : 48/48 Pixel Format : 'MG2S' (MPEG-2 Parsed Slice Data) Field : None Number of planes : 1 Flags : Colorspace : JPEG Transfer Function : Default YCbCr/HSV Encoding: Default Quantization : Default Plane 0 : Bytes per Line : 0 Size Image : 4608Codec Controls
mpeg_2_slice_parameters 0x009909fa (unknown): type=103 flags=has-payload mpeg_2_quantization_matrices 0x009909fb (unknown): type=104 flags=has-payload h264_profile 0x00990a6b (menu) : min=0 max=4 default=2 value=2 h264_sequence_parameter_set 0x00990ce8 (unknown): type=110 flags=has-payload h264_picture_parameter_set 0x00990ce9 (unknown): type=111 flags=has-payload h264_scaling_matrix 0x00990cea (unknown): type=112 flags=has-payload h264_slice_parameters 0x00990ceb (unknown): type=113 flags=has-payload h264_decode_parameters 0x00990cec (unknown): type=114 flags=has-payload h264_decode_mode 0x00990ced (menu) : min=1 max=1 default=1 value=1 h264_start_code 0x00990cee (menu) : min=1 max=1 default=1 value=1 vp8_frame_header 0x009910d0 (unknown): type=301flags=has-payload
Quoted 33 lines · Martin Kepplinger
Quoted 26 lines · Ezequiel Garcia
Hi Martin,
Quoted 10 lines · Martin Kepplinger
Hi Ezequiel,
On the Librem 5 (imx8mq) I try to decode a (h264) full hd video file using the VPU.
I'm running the following tree that doesn't change much, but adds the VPU dts description:
https://source.puri.sm/martin.kepplinger/linux-next/-/commits/5.8-rc3/librem5___vpu
(output of "v4l-ctl --all" at the bottom).OK, this means the VPU is enabled with the Hantro driver, so far so good.
Quoted 9 lines
I now run your version of ffmpeg:
https://gitlab.collabora.com/ezequiel/ffmpeg/-/commits/stateless-mpeg2-vp8-h264-v4
(with minor build fixes) but simply doing "ffplay <file>" seems like not using/dev/video0Is this supposed to work? If so, do I need to know anything about ffmpeg to be able to test this? How do you verify that the v4l2 mem2mem device (vpu) is being used?
Quoted 5 lines
Not sure what state the branch is on, and also IIRC you should use that together with mpv for proper zero-copy operation.
I think the libre elec community can help you with that, as they maintain proper ffmpeg/mpv branches.
Quoted 9 lines
thanks for the hint.
Quoted 1 line
Do you need this to work with ffmpeg, or is GStreamer also OK?
gstreamer would be equally fine (or easier as I've used that before) for me. Last time I checked I think I'd have to build many more libraries from source in order to test though. That's the only reason why I've tried ffmpeg.
In my experience, using gst-build is not too hard; combined with meson, it's really fast to cross build.
There are a couple articles:
https://www.collabora.com/news-and-blog/blog/2020/03/19/getting-started-with-gstreamer-gst-build/
https://www.collabora.com/news-and-blog/blog/2020/05/15/cross-compiling-with-gst-build-and-gstreamer/
.. or tl;dr off the top of my head, hope it works:
(host) cd whatever_your_nfs_rootfs_is/gst-build
(host) meson --cross-file /home/user/meson-cross/arm64.txt build (host) ninja -C build
(The second article explains how to generate a cross-file)
Then on the target:
(target) cd gst-build/
(target) ./gst-uninstalled.py
Few examples that have worked out of the box. The GL ones might or might not work, depending on your support.
gst-launch-1.0 filesrc location=/$1 ! parsebin ! decodebin3 ! fakevideosink
gst-launch-1.0 -ev filesrc location=$1 ! decodebin3 ! videoconvert ! video/x-raw,format=YUY2 ! glupload ! glfiltercube ! glimagesink gst-launch-1.0 -ev filesrc location=$1 ! decodebin3 ! videoconvert ! video/x-raw,format=NV12 ! kmssink
You'll have VPU interrupts and should see nice CPU usage. The GStreamer filter element that decodebin3 should pick up is "v4l2slh264dec".
Let us know if it works :-)
Thanks,
Ezequiel