Add Rockchip VPU JPEG encoder

This series adds support for JPEG encoding via the VPU block present in Rockchip platforms. Currently, support for RK3288 and RK3399 is included.

The hardware produces a Raw JPEG format (i.e. works as a JPEG accelerator). It requires quantization tables provided by the application, and uses standard huffman tables, as recommended by the JPEG specification.

In order to support this, the series introduces a new pixel format, and a new pair of controls, V4L2_CID_JPEG_{LUMA,CHROMA}_QUANTIZATION allowing userspace to specify the quantization tables.

Userspace is then responsible to add the required headers and tables to the produced raw payload, to produce a JPEG image.

Compliance
==========

v4l2-compliance SHA: d0f4ea7ddab6d0244c4fe1e960bb2aaeefb911b9, 64 bits

Compliance test for device /dev/video0:

Driver Info:

	Driver name      : rockchip-vpu
	Card type        : rockchip,rk3399-vpu-enc
	Bus info         : platform: rockchip-vpu
	Driver version   : 4.18.0
	Capabilities     : 0x84204000

Video Memory-to-Memory Multiplanar Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04204000
Video Memory-to-Memory Multiplanar Streaming
Extended Pix Format
Media Driver Info:

	Driver name      : rockchip-vpu
	Model            : rockchip-vpu
	Serial           : 
	Bus info         : 
	Media version    : 4.18.0
	Hardware revision: 0x00000000 (0)
	Driver version   : 4.18.0
Interface Info:
	ID               : 0x0300000c
	Type             : V4L Video
Entity Info:
	ID               : 0x00000001 (1)
	Name             : rockchip,rk3399-vpu-enc-source
	Function         : V4L2 I/O
	Pad 0x01000002   : Source
	  Link 0x02000008: to remote pad 0x1000005 of entity 'rockchip,rk3399-vpu-enc-proc': Data, Enabled, Immutable

Required ioctls:
test MC information (see 'Media Driver Info' above): OK test VIDIOC_QUERYCAP: OK

Allow for multiple opens:
test second /dev/video0 open: OK
test VIDIOC_QUERYCAP: OK
test VIDIOC_G/S_PRIORITY: OK
test for unlimited opens: OK

Debug ioctls:
test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK (Not Supported) test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 0 Audio Inputs: 0 Tuners: 0

Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_ENUMAUDOUT: OK (Not Supported) test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls:
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK test VIDIOC_QUERYCTRL: OK
test VIDIOC_G/S_CTRL: OK
test VIDIOC_G/S/TRY_EXT_CTRLS: OK test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) Standard Controls: 2 Private Controls: 0

Format ioctls:
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK test VIDIOC_G/S_PARM: OK (Not Supported) test VIDIOC_G_FBUF: OK (Not Supported) test VIDIOC_G_FMT: OK
test VIDIOC_TRY_FMT: OK
test VIDIOC_S_FMT: OK
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported) test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK

Codec ioctls:
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported) test VIDIOC_G_ENC_INDEX: OK (Not Supported) test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls:
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK test VIDIOC_EXPBUF: OK

Test input 0:

Streaming ioctls:
test read/write: OK (Not Supported)
test blocking wait: OK
test MMAP: OK
test USERPTR: OK (Not Supported)
test DMABUF: Cannot test, specify --expbuf-device

Total: 48, Succeeded: 48, Failed: 0, Warnings: 0

v3:
- Refactor driver to allow a more elegant integration with other codec modes (h264 decoding, jpeg decoding, etc). Each variant can now be encoders, decoders or both.
- Register driver in the media controller framework,
in preparation for the Request API.
- Set values for JPEG quantization controls in the core, as suggested by Tomasz and Hans.
- Move pm_runtime_get/put to run/done, reducing power consumption. This was possible thanks to Miouyouyou, who pointed out the power domains missing [1].
- Use bulk clock API for simpler code.
v2:
- Add devicetree binding documentation and devicetree changes
- Add documentation to added pixel format and controls
- Address Hans' review comments
- Get rid of unused running_ctx field
- Fix wrong planar pixel format depths
- Other minor changes for v4l2-compliance
- Drop .crop support, we will add support for the
selector API later, if needed.

[1] https://github.com/Miouyouyou/RockMyy/blob/master/patches/kernel/v4.18/DTS/0026-ARM-DTSI-rk3288-Set-the-VPU-MMU-power-domains.patch

Ezequiel Garcia (5):
dt-bindings: Document the Rockchip VPU bindings
ARM: dts: rockchip: add VPU device node for RK3288
arm64: dts: rockchip: add VPU device node for RK3399
v4l2-ctrls: Support dimensions override for standard controls
media: add Rockchip VPU JPEG encoder driver

Shunqian Zheng (2):
media: Add JPEG_RAW format
media: Add controls for JPEG quantization tables

 .../bindings/media/rockchip-vpu.txt           |  30 +
 .../media/uapi/v4l/extended-controls.rst      |  13 +
 .../media/uapi/v4l/pixfmt-compressed.rst      |   9 +
 MAINTAINERS                                   |   7 +
 arch/arm/boot/dts/rk3288.dtsi                 |  14 +-
 arch/arm64/boot/dts/rockchip/rk3399.dtsi      |  14 +-
 drivers/media/platform/Kconfig                |  13 +
 drivers/media/platform/Makefile               |   1 +
 drivers/media/platform/rockchip/vpu/Makefile  |   9 +
 .../platform/rockchip/vpu/rk3288_vpu_hw.c     | 123 ++++
 .../rockchip/vpu/rk3288_vpu_hw_jpege.c        | 123 ++++
 .../platform/rockchip/vpu/rk3288_vpu_regs.h   | 442 +++++++++++++
 .../platform/rockchip/vpu/rk3399_vpu_hw.c     | 124 ++++
 .../rockchip/vpu/rk3399_vpu_hw_jpege.c        | 151 +++++
 .../platform/rockchip/vpu/rk3399_vpu_regs.h   | 601 +++++++++++++++++
 .../platform/rockchip/vpu/rockchip_vpu.h      | 362 +++++++++++
 .../rockchip/vpu/rockchip_vpu_common.h        |  37 ++
 .../platform/rockchip/vpu/rockchip_vpu_drv.c  | 549 ++++++++++++++++
 .../platform/rockchip/vpu/rockchip_vpu_enc.c  | 607 ++++++++++++++++++
 .../platform/rockchip/vpu/rockchip_vpu_hw.h   |  65 ++
 drivers/media/v4l2-core/v4l2-common.c         |   2 +-
 drivers/media/v4l2-core/v4l2-ctrls.c          |  30 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |   1 +
 include/media/v4l2-ctrls.h                    |   2 +-
 include/uapi/linux/v4l2-controls.h            |   3 +
 include/uapi/linux/videodev2.h                |   1 +
 26 files changed, 3322 insertions(+), 11 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/media/rockchip-vpu.txt
 create mode 100644 drivers/media/platform/rockchip/vpu/Makefile
 create mode 100644 drivers/media/platform/rockchip/vpu/rk3288_vpu_hw.c
 create mode 100644 drivers/media/platform/rockchip/vpu/rk3288_vpu_hw_jpege.c
 create mode 100644 drivers/media/platform/rockchip/vpu/rk3288_vpu_regs.h
 create mode 100644 drivers/media/platform/rockchip/vpu/rk3399_vpu_hw.c
 create mode 100644 drivers/media/platform/rockchip/vpu/rk3399_vpu_hw_jpege.c
 create mode 100644 drivers/media/platform/rockchip/vpu/rk3399_vpu_regs.h
 create mode 100644 drivers/media/platform/rockchip/vpu/rockchip_vpu.h
 create mode 100644 drivers/media/platform/rockchip/vpu/rockchip_vpu_common.h
 create mode 100644 drivers/media/platform/rockchip/vpu/rockchip_vpu_drv.c
 create mode 100644 drivers/media/platform/rockchip/vpu/rockchip_vpu_enc.c
 create mode 100644 drivers/media/platform/rockchip/vpu/rockchip_vpu_hw.h

2.18.0

Add devicetree binding documentation for Rockchip Video Processing Unit IP.

---
 .../bindings/media/rockchip-vpu.txt           | 30 +++++++++++++++++++
 1 file changed, 30 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/media/rockchip-vpu.txt

new file mode 100644
index 000000000000..5e0d421301ca
@@ -0,0 +1,30 @@
+device-tree bindings for rockchip VPU codec
+
+Rockchip (Video Processing Unit) present in various Rockchip platforms,
+such as RK3288 and RK3399.
+
+Required properties:
+- compatible: value should be one of the following
+		"rockchip,rk3288-vpu";
+		"rockchip,rk3399-vpu";
+- interrupts: encoding and decoding interrupt specifiers
+- interrupt-names: should be "vepu" and "vdpu"
+- clocks: phandle to VPU aclk, hclk clocks
+- clock-names: should be "aclk" and "hclk"
+- power-domains: phandle to power domain node
+- iommus: phandle to a iommu node
+
+Example:
+SoC-specific DT entry:
+	vpu: video-codec@ff9a0000 {
+		compatible = "rockchip,rk3288-vpu";
+		reg = <0x0 0xff9a0000 0x0 0x800>;
+		interrupts = <GIC_SPI 9 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 10 IRQ_TYPE_LEVEL_HIGH>;
+		interrupt-names = "vepu", "vdpu";
+		clocks = <&cru ACLK_VCODEC>, <&cru HCLK_VCODEC>;
+		clock-names = "aclk", "hclk";
+		power-domains = <&power RK3288_PD_VIDEO>;
+		iommus = <&vpu_mmu>;
+	};
+

2.18.0

Add the Video Processing Unit node for RK3288 SoC.

Fix the VPU IOMMU node, which was disabled and lacking its power domain property.

---
 arch/arm/boot/dts/rk3288.dtsi | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

index d7e49d29ace5..093f941ae273 100644
@@ -1209,6 +1209,18 @@
 		};
 	};
 
+	vpu: video-codec@ff9a0000 {
+		compatible = "rockchip,rk3288-vpu";
+		reg = <0x0 0xff9a0000 0x0 0x800>;
+		interrupts = <GIC_SPI 9 IRQ_TYPE_LEVEL_HIGH>,
+			     <GIC_SPI 10 IRQ_TYPE_LEVEL_HIGH>;
+		interrupt-names = "vepu", "vdpu";
+		clocks = <&cru ACLK_VCODEC>, <&cru HCLK_VCODEC>;
+		clock-names = "aclk", "hclk";
+		power-domains = <&power RK3288_PD_VIDEO>;
+		iommus = <&vpu_mmu>;
+	};
+
 	vpu_mmu: iommu@ff9a0800 {
 		compatible = "rockchip,iommu";
 		reg = <0x0 0xff9a0800 0x0 0x100>;

@@ -1216,8 +1228,8 @@
 		interrupt-names = "vpu_mmu";
 		clocks = <&cru ACLK_VCODEC>, <&cru HCLK_VCODEC>;
 		clock-names = "aclk", "iface";
+		power-domains = <&power RK3288_PD_VIDEO>;
 		#iommu-cells = <0>;
-		status = "disabled";
 	};
 
 	hevc_mmu: iommu@ff9c0440 {

2.18.0

Add the Video Processing Unit node for the RK3399 SoC.

Also, fix the VPU IOMMU node, which was disabled and lacking its power domain property.

---
 arch/arm64/boot/dts/rockchip/rk3399.dtsi | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

index 6ba438427515..5dd840b3deb1 100644
@@ -1231,6 +1231,18 @@
 		status = "disabled";
 	};
 
+	vpu: video-codec@ff650000 {
+		compatible = "rockchip,rk3399-vpu";
+		reg = <0x0 0xff650000 0x0 0x800>;
+		interrupts = <GIC_SPI 114 IRQ_TYPE_LEVEL_HIGH 0>,
+			     <GIC_SPI 113 IRQ_TYPE_LEVEL_HIGH 0>;
+		interrupt-names = "vepu", "vdpu";
+		clocks = <&cru ACLK_VCODEC>, <&cru HCLK_VCODEC>;
+		clock-names = "aclk", "hclk";
+		power-domains = <&power RK3399_PD_VCODEC>;
+		iommus = <&vpu_mmu>;
+	};
+
 	vpu_mmu: iommu@ff650800 {
 		compatible = "rockchip,iommu";
 		reg = <0x0 0xff650800 0x0 0x40>;

@@ -1238,8 +1250,8 @@
 		interrupt-names = "vpu_mmu";
 		clocks = <&cru ACLK_VCODEC>, <&cru HCLK_VCODEC>;
 		clock-names = "aclk", "iface";
+		power-domains = <&power RK3399_PD_VCODEC>;
 		#iommu-cells = <0>;
-		status = "disabled";
 	};
 
 	vdec_mmu: iommu@ff660480 {

2.18.0

Pass the v4l2_ctrl_config->dims array to v4l2_ctrl_fill, so the dimensions can be overridden for standard controls. This will be used to fill V4L2_CID_JPEG_LUMA_QUANTIZATION and V4L2_CID_JPEG_CHROMA_QUANTIZATION.

---
 drivers/media/v4l2-core/v4l2-common.c |  2 +-
 drivers/media/v4l2-core/v4l2-ctrls.c  | 17 ++++++++++-------
 include/media/v4l2-ctrls.h            |  2 +-
 3 files changed, 12 insertions(+), 9 deletions(-)

index b518b92d6d96..7b6353dab8a5 100644
@@ -90,7 +90,7 @@ int v4l2_ctrl_query_fill(struct v4l2_queryctrl *qctrl, s32 _min, s32 _max, s32 _
 	s64 def = _def;
 
 	v4l2_ctrl_fill(qctrl->id, &name, &qctrl->type,
-		       &min, &max, &step, &def, &qctrl->flags);
+		       &min, &max, &step, &def, NULL, &qctrl->flags);
 
 	if (name == NULL)
 		return -EINVAL;

index 599c1cbff3b9..6ab15f4a0fea 100644
@@ -1068,7 +1068,7 @@ const char *v4l2_ctrl_get_name(u32 id)
 EXPORT_SYMBOL(v4l2_ctrl_get_name);
 
 void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,
-		    s64 *min, s64 *max, u64 *step, s64 *def, u32 *flags)
+		    s64 *min, s64 *max, u64 *step, s64 *def, u32 *dims, u32 *flags)
 {
 	*name = v4l2_ctrl_get_name(id);
 	*flags = 0;

@@ -2244,10 +2244,13 @@ struct v4l2_ctrl *v4l2_ctrl_new_custom(struct v4l2_ctrl_handler *hdl,
 	s64 max = cfg->max;
 	u64 step = cfg->step;
 	s64 def = cfg->def;
+	u32 dims[V4L2_CTRL_MAX_DIMS];
+
+	memcpy(dims, cfg->dims, sizeof(dims));
 
 	if (name == NULL)
 		v4l2_ctrl_fill(cfg->id, &name, &type, &min, &max, &step,
-								&def, &flags);
+			       &def, dims, &flags);
 
 	is_menu = (cfg->type == V4L2_CTRL_TYPE_MENU ||
 		   cfg->type == V4L2_CTRL_TYPE_INTEGER_MENU);

@@ -2266,7 +2269,7 @@ struct v4l2_ctrl *v4l2_ctrl_new_custom(struct v4l2_ctrl_handler *hdl,
 	ctrl = v4l2_ctrl_new(hdl, cfg->ops, cfg->type_ops, cfg->id, name,
 			type, min, max,
 			is_menu ? cfg->menu_skip_mask : step, def,
-			cfg->dims, cfg->elem_size,
+			dims, cfg->elem_size,
 			flags, qmenu, qmenu_int, priv);
 	if (ctrl)
 		ctrl->is_private = cfg->is_private;

@@ -2283,7 +2286,7 @@ struct v4l2_ctrl *v4l2_ctrl_new_std(struct v4l2_ctrl_handler *hdl,
 	enum v4l2_ctrl_type type;
 	u32 flags;
 
-	v4l2_ctrl_fill(id, &name, &type, &min, &max, &step, &def, &flags);
+	v4l2_ctrl_fill(id, &name, &type, &min, &max, &step, &def, NULL, &flags);
 	if (type == V4L2_CTRL_TYPE_MENU ||
 	    type == V4L2_CTRL_TYPE_INTEGER_MENU ||
 	    type >= V4L2_CTRL_COMPOUND_TYPES) {

@@ -2312,7 +2315,7 @@ struct v4l2_ctrl *v4l2_ctrl_new_std_menu(struct v4l2_ctrl_handler *hdl,
 	u64 step;
 	u32 flags;
 
-	v4l2_ctrl_fill(id, &name, &type, &min, &max, &step, &def, &flags);
+	v4l2_ctrl_fill(id, &name, &type, &min, &max, &step, &def, NULL, &flags);
 
 	if (type == V4L2_CTRL_TYPE_MENU)
 		qmenu = v4l2_ctrl_get_menu(id);

@@ -2350,7 +2353,7 @@ struct v4l2_ctrl *v4l2_ctrl_new_std_menu_items(struct v4l2_ctrl_handler *hdl,
 		return NULL;
 	}
 
-	v4l2_ctrl_fill(id, &name, &type, &min, &max, &step, &def, &flags);
+	v4l2_ctrl_fill(id, &name, &type, &min, &max, &step, &def, NULL, &flags);
 	if (type != V4L2_CTRL_TYPE_MENU || qmenu == NULL) {
 		handler_set_err(hdl, -EINVAL);
 		return NULL;

@@ -2375,7 +2378,7 @@ struct v4l2_ctrl *v4l2_ctrl_new_int_menu(struct v4l2_ctrl_handler *hdl,
 	s64 def = _def;
 	u32 flags;
 
-	v4l2_ctrl_fill(id, &name, &type, &min, &max, &step, &def, &flags);
+	v4l2_ctrl_fill(id, &name, &type, &min, &max, &step, &def, NULL, &flags);
 	if (type != V4L2_CTRL_TYPE_INTEGER_MENU) {
 		handler_set_err(hdl, -EINVAL);
 		return NULL;

index f615ba1b29dd..55ab01d97c12 100644
@@ -370,7 +370,7 @@ struct v4l2_ctrl_config {
  *    control framework this function will no longer be exported.
  */
 void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,
-		    s64 *min, s64 *max, u64 *step, s64 *def, u32 *flags);
+		    s64 *min, s64 *max, u64 *step, s64 *def, u32 *dims, u32 *flags);
 
 
 /**

2.18.0

From: Shunqian Zheng <[email protected]>

Add V4L2_PIX_FMT_JPEG_RAW format that does not contain JPEG header in the output frame.

---
 Documentation/media/uapi/v4l/pixfmt-compressed.rst | 9 +++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c               | 1 +
 include/uapi/linux/videodev2.h                     | 1 +
 3 files changed, 11 insertions(+)

index d382e7a5c38e..4dffe40097f2 100644
@@ -23,6 +23,15 @@ Compressed Formats
       - 'JPEG'
       - TBD. See also :ref:`VIDIOC_G_JPEGCOMP <VIDIOC_G_JPEGCOMP>`,
 	:ref:`VIDIOC_S_JPEGCOMP <VIDIOC_G_JPEGCOMP>`.
+    * .. _V4L2-PIX-FMT-JPEG-RAW:
+
+      - ``V4L2_PIX_FMT_JPEG_RAW``
+      - 'Raw JPEG'
+      - Raw JPEG bitstream, containing a compressed payload. This format
+        contains an image scan, i.e. without any metadata or headers.
+        The user is expected to set the needed metadata such as
+        quantization and entropy encoding tables, via ``V4L2_CID_JPEG``
+        controls, see :ref:`jpeg-control-id`.
     * .. _V4L2-PIX-FMT-MPEG:
 
       - ``V4L2_PIX_FMT_MPEG``

index 54afc9c7ee6e..0dcd95f4bdf1 100644
@@ -1301,6 +1301,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 		/* Max description length mask:	descr = "0123456789012345678901234567890" */
 		case V4L2_PIX_FMT_MJPEG:	descr = "Motion-JPEG"; break;
 		case V4L2_PIX_FMT_JPEG:		descr = "JFIF JPEG"; break;
+		case V4L2_PIX_FMT_JPEG_RAW:	descr = "Raw JPEG"; break;
 		case V4L2_PIX_FMT_DV:		descr = "1394"; break;
 		case V4L2_PIX_FMT_MPEG:		descr = "MPEG-1/2/4"; break;
 		case V4L2_PIX_FMT_H264:		descr = "H.264"; break;

index 5d1a3685bea9..f271048c89c4 100644
@@ -627,6 +627,7 @@ struct v4l2_pix_format {
 /* compressed formats */
 #define V4L2_PIX_FMT_MJPEG    v4l2_fourcc('M', 'J', 'P', 'G') /* Motion-JPEG   */
 #define V4L2_PIX_FMT_JPEG     v4l2_fourcc('J', 'P', 'E', 'G') /* JFIF JPEG     */
+#define V4L2_PIX_FMT_JPEG_RAW v4l2_fourcc('J', 'P', 'G', 'R') /* JFIF JPEG RAW without headers */
 #define V4L2_PIX_FMT_DV       v4l2_fourcc('d', 'v', 's', 'd') /* 1394          */
 #define V4L2_PIX_FMT_MPEG     v4l2_fourcc('M', 'P', 'E', 'G') /* MPEG-1/2/4 Multiplexed */
 #define V4L2_PIX_FMT_H264     v4l2_fourcc('H', '2', '6', '4') /* H264 with start codes */

2.18.0

From: Shunqian Zheng <[email protected]>

Add V4L2_CID_JPEG_LUMA/CHROMA_QUANTIZATION controls to allow userspace configure the JPEG quantization tables.

---
 Documentation/media/uapi/v4l/extended-controls.rst | 13 +++++++++++++
 drivers/media/v4l2-core/v4l2-ctrls.c               | 13 +++++++++++++
 include/uapi/linux/v4l2-controls.h                 |  3 +++
 3 files changed, 29 insertions(+)

index 9f7312bf3365..1189750a022c 100644
@@ -3354,6 +3354,19 @@ JPEG Control IDs
     Specify which JPEG markers are included in compressed stream. This
     control is valid only for encoders.
 
+.. _jpeg-quant-tables-control:
+
+``V4L2_CID_JPEG_LUMA_QUANTIZATION (__u8 matrix)``
+    Sets the luma quantization table to be used for encoding
+    or decoding a V4L2_PIX_FMT_JPEG_RAW format buffer. This table is
+    a 8x8-byte matrix, and it's expected to be in JPEG zigzag order,
+    as per the JPEG specification.
+
+``V4L2_CID_JPEG_CHROMA_QUANTIZATION (__u8 matrix)``
+    Sets the chroma quantization table to be used for encoding
+    or decoding a V4L2_PIX_FMT_JPEG_RAW format buffer. This table is
+    a 8x8-byte matrix, and it's expected to be in JPEG zigzag order,
+    as per the JPEG specification.
 
 
 .. flat-table::

index 6ab15f4a0fea..677af8069bfc 100644
@@ -999,6 +999,8 @@ const char *v4l2_ctrl_get_name(u32 id)
 	case V4L2_CID_JPEG_RESTART_INTERVAL:	return "Restart Interval";
 	case V4L2_CID_JPEG_COMPRESSION_QUALITY:	return "Compression Quality";
 	case V4L2_CID_JPEG_ACTIVE_MARKER:	return "Active Markers";
+	case V4L2_CID_JPEG_LUMA_QUANTIZATION:	return "Luminance Quantization Matrix";
+	case V4L2_CID_JPEG_CHROMA_QUANTIZATION:	return "Chrominance Quantization Matrix";
 
 	/* Image source controls */
 	/* Keep the order of the 'case's the same as in v4l2-controls.h! */

@@ -1286,6 +1288,17 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,
 	case V4L2_CID_DETECT_MD_REGION_GRID:
 		*type = V4L2_CTRL_TYPE_U8;
 		break;
+	case V4L2_CID_JPEG_LUMA_QUANTIZATION:
+	case V4L2_CID_JPEG_CHROMA_QUANTIZATION:
+		*min = *def = 0;
+		*max = 0xff;
+		*step = 1;
+		*type = V4L2_CTRL_TYPE_U8;
+		if (dims) {
+			memset(dims, 0, V4L2_CTRL_MAX_DIMS * sizeof(dims[0]));
+			dims[0] = dims[1] = 8;
+		}
+		break;
 	case V4L2_CID_DETECT_MD_THRESHOLD_GRID:
 		*type = V4L2_CTRL_TYPE_U16;
 		break;

index e4ee10ee917d..a466acf40625 100644
@@ -987,6 +987,9 @@ enum v4l2_jpeg_chroma_subsampling {
 #define	V4L2_JPEG_ACTIVE_MARKER_DQT		(1 << 17)
 #define	V4L2_JPEG_ACTIVE_MARKER_DHT		(1 << 18)
 
+#define V4L2_CID_JPEG_LUMA_QUANTIZATION		(V4L2_CID_JPEG_CLASS_BASE + 5)
+#define V4L2_CID_JPEG_CHROMA_QUANTIZATION	(V4L2_CID_JPEG_CLASS_BASE + 6)
+
 
 /* Image source controls */
 #define V4L2_CID_IMAGE_SOURCE_CLASS_BASE	(V4L2_CTRL_CLASS_IMAGE_SOURCE | 0x900)

2.18.0

Hi,

How about having a single control for quantization?

In MPEG-2/H.264/H.265, we have a single control exposed as a structure, which contains the tables for both luma and chroma. In the case of JPEG, it's not that big a deal, but for advanced video formats, it would be too much hassle to have one control per table.

In order to keep the interface consistent, I think it'd be best to merge both matrices into a single control.

What do you think?

Paul

Paul Kocialkowski, Bootlin (formerly Free Electrons)
Embedded Linux and kernel engineering
https://bootlin.com

_______________________________________________
Linux-rockchip mailing list [email protected]
http://lists.infradead.org/mailman/listinfo/linux-rockchip

I think it makes a lot of sense. I don't see the benefit in having luma and chroma separated, and consistency is good.

I guess the more consistent solution would be to expose a compound control, similar to the video quantization one.

struct v4l2_ctrl_jpeg_quantization {
       __u8    luma_quantization_matrix[64];
       __u8    chroma_quantization_matrix[64];
};

Thanks!
Eze

Makes sense indeed. It also lets us avoid the hassle of setting .min/.max/.dims and other array control stuff, since everything is already defined by the C struct itself.

Best regards,
Tomasz

Add a mem2mem driver for the VPU available on Rockchip SoCs. Currently only JPEG encoding is supported, for RK3399 and RK3288 platforms.

---
 MAINTAINERS                                   |   7 +
 drivers/media/platform/Kconfig                |  13 +
 drivers/media/platform/Makefile               |   1 +
 drivers/media/platform/rockchip/vpu/Makefile  |   9 +
 .../platform/rockchip/vpu/rk3288_vpu_hw.c     | 123 ++++
 .../rockchip/vpu/rk3288_vpu_hw_jpege.c        | 123 ++++
 .../platform/rockchip/vpu/rk3288_vpu_regs.h   | 442 +++++++++++++
 .../platform/rockchip/vpu/rk3399_vpu_hw.c     | 124 ++++
 .../rockchip/vpu/rk3399_vpu_hw_jpege.c        | 151 +++++
 .../platform/rockchip/vpu/rk3399_vpu_regs.h   | 601 +++++++++++++++++
 .../platform/rockchip/vpu/rockchip_vpu.h      | 362 +++++++++++
 .../rockchip/vpu/rockchip_vpu_common.h        |  37 ++
 .../platform/rockchip/vpu/rockchip_vpu_drv.c  | 549 ++++++++++++++++
 .../platform/rockchip/vpu/rockchip_vpu_enc.c  | 607 ++++++++++++++++++
 .../platform/rockchip/vpu/rockchip_vpu_hw.h   |  65 ++
 15 files changed, 3214 insertions(+)
 create mode 100644 drivers/media/platform/rockchip/vpu/Makefile
 create mode 100644 drivers/media/platform/rockchip/vpu/rk3288_vpu_hw.c
 create mode 100644 drivers/media/platform/rockchip/vpu/rk3288_vpu_hw_jpege.c
 create mode 100644 drivers/media/platform/rockchip/vpu/rk3288_vpu_regs.h
 create mode 100644 drivers/media/platform/rockchip/vpu/rk3399_vpu_hw.c
 create mode 100644 drivers/media/platform/rockchip/vpu/rk3399_vpu_hw_jpege.c
 create mode 100644 drivers/media/platform/rockchip/vpu/rk3399_vpu_regs.h
 create mode 100644 drivers/media/platform/rockchip/vpu/rockchip_vpu.h
 create mode 100644 drivers/media/platform/rockchip/vpu/rockchip_vpu_common.h
 create mode 100644 drivers/media/platform/rockchip/vpu/rockchip_vpu_drv.c
 create mode 100644 drivers/media/platform/rockchip/vpu/rockchip_vpu_enc.c
 create mode 100644 drivers/media/platform/rockchip/vpu/rockchip_vpu_hw.h

index da68e6da9981..e99b49c8dcf2 100644
@@ -12272,6 +12272,13 @@ S:	Maintained
 F:	drivers/media/platform/rockchip/rga/
 F:	Documentation/devicetree/bindings/media/rockchip-rga.txt
 
+ROCKCHIP VPU CODEC DRIVER
+M:	Ezequiel Garcia <ezequiel-ZGY8ohtN/[email protected]>
+L:	[email protected]
+S:	Maintained
+F:	drivers/media/platform/rockchip/vpu/
+F:	Documentation/devicetree/bindings/media/rockchip-vpu.txt
+
 ROCKER DRIVER
 M:	Jiri Pirko <[email protected]>
 L:	[email protected]

index b25c8d3c1c31..87eb854cf7cb 100644
@@ -448,6 +448,19 @@ config VIDEO_ROCKCHIP_RGA
 
 	  To compile this driver as a module choose m here.
 
+config VIDEO_ROCKCHIP_VPU
+	tristate "Rockchip VPU driver"
+	depends on ARCH_ROCKCHIP || COMPILE_TEST
+	depends on VIDEO_DEV && VIDEO_V4L2 && MEDIA_CONTROLLER
+	select VIDEOBUF2_DMA_CONTIG
+	select V4L2_MEM2MEM_DEV
+	default n
+	help
+	  Support for the Video Processing Unit present on Rockchip SoC,
+	  which accelerates video and image encoding and decoding.
+	  To compile this driver as a module, choose M here: the module
+	  will be called rockchip-vpu.
+
 config VIDEO_TI_VPE
 	tristate "TI VPE (Video Processing Engine) driver"
 	depends on VIDEO_DEV && VIDEO_V4L2

index 08640ba87fc2..9b93f6a6b6e2 100644
@@ -67,6 +67,7 @@ obj-$(CONFIG_VIDEO_RENESAS_JPU)		+= rcar_jpu.o
 obj-$(CONFIG_VIDEO_RENESAS_VSP1)	+= vsp1/
 
 obj-$(CONFIG_VIDEO_ROCKCHIP_RGA)	+= rockchip/rga/
+obj-$(CONFIG_VIDEO_ROCKCHIP_VPU)        += rockchip/vpu/
 
 obj-y	+= omap/
 

new file mode 100644
index 000000000000..f717dfda1d42
@@ -0,0 +1,9 @@
+obj-$(CONFIG_VIDEO_ROCKCHIP_VPU) += rockchip-vpu.o
+
+rockchip-vpu-y += \
+		rockchip_vpu_drv.o \
+		rockchip_vpu_enc.o \
+		rk3288_vpu_hw.o \
+		rk3288_vpu_hw_jpege.o \
+		rk3399_vpu_hw.o \
+		rk3399_vpu_hw_jpege.o

new file mode 100644
index 000000000000..1d7c7bf8f323
@@ -0,0 +1,37 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Rockchip VPU codec driver
+ *
+ * Copyright (C) 2018 Rockchip Electronics Co., Ltd.
+ *	Alpha Lin <[email protected]>
+ *	Jeffy Chen <[email protected]>
+ *
+ * Copyright (C) 2018 Google, Inc.
+ *	Tomasz Figa <[email protected]>
+ *
+ * Based on s5p-mfc driver by Samsung Electronics Co., Ltd.
+ * Copyright (C) 2011 Samsung Electronics Co., Ltd.
+ */
+
+#ifndef ROCKCHIP_VPU_COMMON_H_
+#define ROCKCHIP_VPU_COMMON_H_
+
+#include "rockchip_vpu.h"
+
+extern const struct v4l2_ioctl_ops rockchip_vpu_enc_ioctl_ops;
+extern const struct v4l2_ioctl_ops rockchip_vpu_dec_ioctl_ops;
+extern const struct vb2_ops rockchip_vpu_enc_queue_ops;
+extern const struct vb2_ops rockchip_vpu_dec_queue_ops;
+
+void *rockchip_vpu_find_control_data(struct rockchip_vpu_ctx *ctx,
+				unsigned int id);
+void rockchip_vpu_enc_reset_src_fmt(struct rockchip_vpu_dev *vpu,
+				struct rockchip_vpu_ctx *ctx);
+void rockchip_vpu_enc_reset_dst_fmt(struct rockchip_vpu_dev *vpu,
+				struct rockchip_vpu_ctx *ctx);
+void rockchip_vpu_dec_reset_src_fmt(struct rockchip_vpu_dev *vpu,
+				struct rockchip_vpu_ctx *ctx);
+void rockchip_vpu_dec_reset_dst_fmt(struct rockchip_vpu_dev *vpu,
+				struct rockchip_vpu_ctx *ctx);
+
+#endif /* ROCKCHIP_VPU_COMMON_H_ */

2.18.0

Part of the diff is not shown here. View the raw message.