Skip to contentSkip to conversation

media: platform: Renesas IMR driver

PATCH In review Sergei Shtylyov 4 participants 7 messages Aug 4, 2017 – Apr 16, 2018 1 message not in the archive

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH v7 1/1 media: platform: Renesas IMR driver

The image renderer, or the distortion correction engine, is a drawing processor with a simple instruction system capable of referencing video capture data or data in an external memory as the 2D texture data and performing texture mapping and drawing with respect to any shape that is split into triangular objects.

This V4L2 memory-to-memory device driver only supports image renderer light extended 4 (IMR-LX4) found in the R-Car gen3 SoCs; the R-Car gen2 support can be added later...

Based on the original patch by Konstantin Kozhevnikov.

---
The patch is against the 'media_tree.git' repo's 'master' branch.

Changes in version 7:
- switched to using 'v4l2_fh::m2m_ctx' which permitted using v4l2_m2m_fop_{poll| mmap} instead of imr_{poll|mmap}();
- moved the configuration check from imr_qbuf() to imr_buf_prepare(), replacing the former with v4l2_m2m_ioctl_qbuf();
- renamed imr_cfg_[un]ref() to imr_cfg_{get|put}();
- removed VB2_USERPTR from imr_queue_init();
- removed V4L2_CAP_VIDEO_{CAPTURE|OUTPUT} device capabilities;
- replaced "luminance" with "luma" and "chrominance" with "chroma" in the driver and UAPI header comments;
- fixed typo in the email address in MODULE_AUTHOR();
- added the vertex-related structures to the UAPI header;
- replaced the '__packed' annotations with explicit __attribute__((packed)) in the UAPI header;
- fixed/clarified the comments in the UAPI header;
- added the dependence table for the vertex object variants in the driver document;
- added the description of the luma/chroma correction in the driver document;
- added the mesh construction code example to the driver document;
- removed the fragments effectively duplicating the UAPI hearder comments in the driver document, referring a reader to that header instead;
- clarified the use of the auto-generated source/destination coordinates in the driver document;
- marked up the references to the code in the driver document;
- fixed typo/wording in the driver document;
- removed the main text indentation in the driver document;
- changed the patch authorship to mine;
- added Rob Herring's ACK.

Changes in version 6:
- fixed the bug where if imr_cfg_create() fails, 'ctx->cfg' wasn't set to NULL and so wouldn't fail the validity checks;
- fixed the height minimum/alignment passed to v4l_bound_align_image();
- removed the buggy !V4L2_TYPE_IS_OUTPUT() check from imr_qbuf();
- added the driver UAPI documentation;
- replaced 'imr_ctx::crop' array with the 'imr_ctx::rect' structure;
- replaced imr_{g|s}_crop() with imr_{g|s}_selection();
- removed 'imr_format_info::name' and the related code;
- completely rewrote imr_queue_setup();
- moved the applicable code from imr_buf_queue() to imr_buf_prepare(), moved the rest of imr_buf_queue() after imr_buf_finish();
- removed imr_start_streaming();
- assigned 'src_vq->dev' and 'dst_vq->dev' in imr_queue_init();
- clarified the math in imt_tri_type_{a|b|c}_length();
- removed useless local variables, added useful variables, avoided casts to 'void *', and clarified the pointer math in imr_tri_set_type_{a|b|c}();
- replaced kmalloc()/copy_from_user() calls with memdup_user() call;
- moved the 'type' variable assignment in imr_ioctl_map() after the following comment;
- merged the matrix size checks for the cases of the automatic generation pattern and the absolute coordinates in imr_ioctl_map();
- swapped the operands in the VBO size check in imr_ioctl_map();
- moved setting device capabilities from imr_querycap() to imr_probe();
- replaced imr_{reqbufs|querybuf|dqbuf|expbuf|streamon|streamoff}() with the generic helpers, implemented vidioc_{create_bufs|prepare_buf}() methods;
- set the valid default queue pixel format in imr_probe();
- used tabs for indentation where possible in imr_probe();
- removed the rest of the *inline* keywords;
- declared 'imr_map_desc::data' as '__u64' instead of 'void *';
- switched to '__u{16|32}' instead of 'u{16|32}' in the UAPI header;
- spelled out the VBO abbreviation and removed redundancy in the UAPI header comments.

Changes in version 5:
- used ALIGN() macro in imr_ioctl_map();
- moved the display list #define's after the register #define's, removing the redundant comment;
- uppercased the "tbd" abbreviation in the comments;
- made the TRI instruction types A/B/C named consistently;
- avoided quotes around the coordinate's names;
- avoided some hyphens in the comments;
- removed spaces around / and before ? in the comments;
- reworded some comments;
- reformatted some multiline comments;
- fixed typos in the comments.

Changes in version 4:
- added/used the SUSR fields/shifts.

Changes in version 3:
- added/used the {UV|CP}DPOR fields/shifts;
- removed unsupported LINE instruction;
- replaced '*' with 'x' in the string passed to v4l2_dbg() in imr_dl_program_setup();
- switched to prepending the SoC model to "imr-lx4" in the "compatible" prop strings.

Changes in version 2:
- renamed the ICR bits to match the manual;
- added/used the IMR bits;
- changed the prefixes of the CMRCR[2]/TRI{M|C}R bits/fields to match the manual;
- renamed the CMRCR.DY1{0|2} bits to match the manual;
- removed non-existent TRIMR.D{Y|U}D{X|V}M bits;
- used the SR bits instead of a bare number;
- sorted the instruction macros by opcode, removing redundant parens;
- masked the register address in WTL[2]/WTS instruction macros;
- avoided setting reserved bits when writing to CMRCCR[2]/TRIMCR;
- mentioned the video capture data as a texture source in the binding and the patch description;
- documented the SoC specific "compatible" values;
- clarified the "interrupts" and "clocks" property text;
- updated the patch description.

 Documentation/devicetree/bindings/media/rcar_imr.txt |   27 
 Documentation/media/v4l-drivers/index.rst            |    1 
 Documentation/media/v4l-drivers/rcar_imr.rst         |  372 +++
 drivers/media/platform/Kconfig                       |   13 
 drivers/media/platform/Makefile                      |    1 
 drivers/media/platform/rcar_imr.c                    | 1832 +++++++++++++++++++
 include/uapi/linux/rcar_imr.h                        |  182 +
 7 files changed, 2428 insertions(+)
Documentation/devicetree/bindings/media/rcar_imr.txt new file +27 -0
Index: media_tree/Documentation/devicetree/bindings/media/rcar_imr.txt
===================================================================
@@ -0,0 +1,27 @@
+Renesas R-Car Image Renderer (Distortion Correction Engine)
+-----------------------------------------------------------
+
+The image renderer, or the distortion correction engine, is a drawing processor
+with a simple instruction system capable of referencing video capture data or
+data in an external memory as 2D texture data and performing texture mapping
+and drawing with respect to any shape that is split into triangular objects.
+
+Required properties:
+
+- compatible: "renesas,<soctype>-imr-lx4", "renesas,imr-lx4" as a fallback for
+  the image renderer light extended 4 (IMR-LX4) found in the R-Car gen3 SoCs,
+  where the examples with <soctype> are:
+  - "renesas,r8a7795-imr-lx4" for R-Car H3,
+  - "renesas,r8a7796-imr-lx4" for R-Car M3-W.
+- reg: offset and length of the register block;
+- interrupts: single interrupt specifier;
+- clocks: single clock phandle/specifier pair.
+
+Example:
+
+	imr-lx4@fe860000 {
+		compatible = "renesas,r8a7795-imr-lx4", "renesas,imr-lx4";
+		reg = <0 0xfe860000 0 0x2000>;
+		interrupts = <GIC_SPI 192 IRQ_TYPE_LEVEL_HIGH>;
+		clocks = <&cpg CPG_MOD 823>;
+	};
Documentation/media/v4l-drivers/index.rst +1 -0
Index: media_tree/Documentation/media/v4l-drivers/index.rst
===================================================================
@@ -52,6 +52,7 @@ For more details see the file COPYING in
 	pxa_camera
 	radiotrack
 	rcar-fdp1
+	rcar_imr
 	saa7134
 	sh_mobile_ceu_camera
 	si470x
Documentation/media/v4l-drivers/rcar_imr.rst new file +372 -0

372 lines not on this page · raw message

drivers/media/platform/Kconfig +13 -0
Index: media_tree/drivers/media/platform/Kconfig
===================================================================
@@ -438,6 +438,19 @@ config VIDEO_RENESAS_FCP
 	  To compile this driver as a module, choose M here: the module
 	  will be called rcar-fcp.
 
+config VIDEO_RENESAS_IMR
+	tristate "Renesas Image Renderer (Distortion Correction Engine)"
+	depends on VIDEO_DEV && VIDEO_V4L2
+	depends on ARCH_RENESAS || COMPILE_TEST
+	select VIDEOBUF2_DMA_CONTIG
+	select V4L2_MEM2MEM_DEV
+	---help---
+	  This is a V4L2 driver for the Renesas Image Renderer Light Extended 4
+	  (IMR-LX4).
+
+	  To compile this driver as a module, choose M here: the module
+	  will be called rcar_imr.
+
 config VIDEO_RENESAS_VSP1
 	tristate "Renesas VSP1 Video Processing Engine"
 	depends on VIDEO_V4L2 && VIDEO_V4L2_SUBDEV_API && HAS_DMA
drivers/media/platform/Makefile +1 -0
Index: media_tree/drivers/media/platform/Makefile
===================================================================
@@ -60,6 +60,7 @@ obj-$(CONFIG_VIDEO_RCAR_DRIF)		+= rcar_d
 obj-$(CONFIG_VIDEO_RENESAS_FCP) 	+= rcar-fcp.o
 obj-$(CONFIG_VIDEO_RENESAS_FDP1)	+= rcar_fdp1.o
 obj-$(CONFIG_VIDEO_RENESAS_JPU) 	+= rcar_jpu.o
+obj-$(CONFIG_VIDEO_RENESAS_IMR) 	+= rcar_imr.o
 obj-$(CONFIG_VIDEO_RENESAS_VSP1)	+= vsp1/
 
 obj-y	+= omap/
drivers/media/platform/rcar_imr.c new file +1832 -0

1,832 lines not on this page · raw message

include/uapi/linux/rcar_imr.h new file +182 -0

182 lines not on this page · raw message

A message that is not in the archive (sent off-list or not yet received): 20170804180402.795437602-M4DtvfQ/[email protected]

#2 lore raw

Hi Sergei,

A few high level comments (I'll look at the patch itself later):

- There is no MAINTAINERS entry, please add one.
- Don't attach the patch, post it inline (ideally with 'git send-email')
- Split up the patch into 4 separate patches: bindings, doc changes, driver and MAINTAINERS patch. This will make it easier to review.
- Please give the v4l2-compliance output in the cover letter of the v8 patch series. I can't merge this driver without being certain there are no compliance issues.
- You also have IMR-LSX3 and IMR-LX3 patches, why not add them to the patch series? I can review the set as a single unit. Up to you, though.

Regards,

Hans
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
In reply to Sergei Shtylyov

Hi Sergei,

A quick review. I'm concentrating on the mesh ioctl, since that's what sets this driver apart.

On 08/04/2017 08:03 PM, Sergei Shtylyov wrote:

<snip>

Documentation/media/v4l-drivers/rcar_imr.rst line 5

Quoted 10 lines

Index: media_tree/Documentation/media/v4l-drivers/rcar_imr.rst
===================================================================

Documentation/media/v4l-drivers/rcar_imr.rst new file +5 -0
@@ -0,0 +1,372 @@
+Renesas R-Car Image Renderer (IMR) Driver
+=========================================
+
+This file documents some driver-specific aspects of the IMR driver, such as
+the driver-specific ioctl.

Just drop the part ', such as...'.

Can you add a high-level description of the functionality here? Similar to what you did in the bindings document.

Quoted 7 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +7 -0
+
+The ioctl reference
+~~~~~~~~~~~~~~~~~~~
+
+See ``include/uapi/linux/rcar_imr.h`` for the data structures used.
+
+VIDIOC_IMR_MESH - Set mapping data

Documentation/media/v4l-drivers/rcar_imr.rst line 22

Quoted 10 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +10 -0
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Argument: ``struct imr_map_desc``
+
+**Description**:
+
+This ioctl sets up the mesh through which the input frames will be transformed
+into the output frames. The mesh can be strictly rectangular (when
+``IMR_MAP_MESH`` bit is set in ``imr_map_desc::type``) or arbitrary (when that
+bit is not set).

Wouldn't something like 'IMR_MAP_RECTANGULAR' be much more descriptive than _MESH? There is nothing in the name _MESH to indicate that this switches the data to rectangles.

Documentation/media/v4l-drivers/rcar_imr.rst line 29

Quoted 7 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +7 -0
+
+A rectangular mesh consists of the imr_mesh structure followed by M*N vertex
+objects (where M is ``imr_mesh::rows`` and N is ``imr_mesh::columns``).
+In case either ``IMR_MAP_AUTOSG`` or ``IMR_MAP_AUTODG`` (not both) bits are
+set in ``imr_map_desc::type``, ``imr_mesh::{x|y}0`` specify the coordinates
+of the top left corner of the auto-generated mesh and ``imr_mesh::d{x|y}``
+specify the mesh's X/Y steps.

So if the auto bits are set, then there are no vertex objects? Since it's auto generated by the hardware?

I believe we discussed in the past whether 'type' should be split in a 'type' and 'flags' field.

Documentation/media/v4l-drivers/rcar_imr.rst line 40

Quoted 11 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +11 -0
+
+An arbitrary mesh consists of the imr_vbo structure followed by N triangle
+objects (where N is ``imr_vbo::num``), consisting of 3 vertex objects each.
+Setting ``IMR_MAP_AUTODG`` and ``IMR_MAP_AUTOSG`` bits in
+``imr_map_desc::type``) isn't allowed for this type of mesh.
+
+The vertex object has a complex structure depending on some of the bits in
+``imr_map_desc::type``:
+
+============  ============  ==============  ==============  ===========================
+IMR_MAP_CLCE  IMR_MAP_LUCE  IMR_MAP_AUTODG  IMR_MAP_AUTOSG  Vertex structure variant

You should explain the meaning of these bits in this section. I.e., what does CLCE or AUTODG stand for?

Documentation/media/v4l-drivers/rcar_imr.rst line 47

Quoted 7 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +7 -0
+============  ============  ==============  ==============  ===========================
+\                                                           ``imr_full_coord``
+\                                           X               ``imr_dst_coord``
+\                           X                               ``imr_src_coord``
+\             X                                             ``imr_full_coord_any_correct``
+\             X                             X               ``imr_auto_coord_any_correct``
+\             X             X                               ``imr_auto_coord_any_crrect``

crrect -> correct

Quoted 25 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +25 -0
+X                                                           ``imr_full_coord_any_correct``
+X                                           X               ``imr_auto_coord_any_correct``
+X                           X                               ``imr_auto_coord_any_correct``
+X             X                                             ``imr_full_coord_both_correct``
+X             X                             X               ``imr_auto_coord_both_correct``
+X             X             X                               ``imr_auto_coord_both_correct``
+============  ============  ==============  ==============  ===========================
+
+The luma correction is calculated according to the following formula (where
+``Y`` is the luma value after texture mapping, ``Y'`` is the luma value after
+luma correction, ``lscal`` and ``lofst`` are the luma correction scale and
+offset taken from ``struct imr_luma_correct``, ``YLDPO`` is a luma correction
+scale decimal point position specified by ``IMR_MAP_YLDPO(n)``): ::
+
+	Y' = ((Y * lscal) >> YLDPO) + lofst
+
+The chroma correction is calculated according to the following formula (where
+``U/V`` are the chroma values after texture mapping, ``U'/V'`` are the chroma
+values after chroma correction, ``ubscl/vrscl`` and ``ubofs/vrofs`` are the
+U/V value chroma correction scales and offsets taken from
+``struct imr_chroma_correct``, ``UBDPO/VRDPO`` are the chroma correction scale
+decimal point positions specified by ``IMR_MAP_{UBDPO|VRDPO}(n)``): ::
+
+	U' = ((U + ubofs) * ubscl) >> UBDPO
+	V' = ((V + vrofs) * vrscl) >> VRDPO

Documentation/media/v4l-drivers/rcar_imr.rst line 82

Quoted 10 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +10 -0
+
+**Return value**:
+
+On success 0 is returned. On error -1 is returned and ``errno`` is set
+appropriately.
+
+**Example code**:
+
+Below is an example code for constructing the meshes: ``imr_map_create()``
+constructs an arbitraty mesh, ``imr_map_mesh_src()`` function constructs

arbitrary

Documentation/media/v4l-drivers/rcar_imr.rst line 93

Quoted 11 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +11 -0
+a rectangular mesh with the auto-generated destination coordinates.
+
+.. code-block:: C
+
+ #include <malloc.h>
+ #include <math.h>
+ #include <linux/rcar_imr.h>
+
+ /* IMR device data */
+ struct imr_device {
+ 	/* V4L2 file decriptor */

descriptor

Quoted 279 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +279 -0
+ 	int			vfd;
+
+ 	/* input/output buffers dimensions */
+ 	int			w, h, W, H;
+ };
+
+ #define IMR_SRC_SUBSAMPLE	5
+ #define IMR_DST_SUBSAMPLE	2
+ #define IMR_COORD_THRESHOLD	(128 * 128 * (1 << 2 * IMR_DST_SUBSAMPLE))
+
+ /* find the longest side (index) */
+ static void find_longest_side(int n, __s16 xy0, __s16 xy1, int *max, int *side)
+ {
+ 	int t = xy1 - xy0;
+
+ 	t *= t;
+ 	if (*max < t) {
+ 		*max  = t;
+ 		*side = n;
+ 	}
+ }
+
+ /* recursively split a triangle until it can be passed to IMR */
+ static int split_triangle(struct imr_full_coord *coord,
+ 			   __s16 *xy0, __s16 *xy1, __s16 *xy2,
+ 			   __u16 *uv0, __u16 *uv1, __u16 *uv2, int avail)
+ {
+ 	int	max = 0, k = 0;
+
+ 	/* we need to have at least one available triangle */
+ 	if (avail < 1)
+ 		return 0;
+
+ 	find_longest_side(0, xy0[0], xy1[0], &max, &k);
+ 	find_longest_side(0, xy0[1], xy1[1], &max, &k);
+ 	find_longest_side(1, xy1[0], xy2[0], &max, &k);
+ 	find_longest_side(1, xy1[1], xy2[1], &max, &k);
+ 	find_longest_side(2, xy2[0], xy0[0], &max, &k);
+ 	find_longest_side(2, xy2[1], xy0[1], &max, &k);
+
+ 	/* if value exceeds the threshold, do splitting */
+ 	if (max >= IMR_COORD_THRESHOLD) {
+ 		__s16	XY[2];
+ 		__u16	UV[2];
+ 		int	n;
+
+ 		switch (k) {
+ 		case 0:
+ 			/* split triangle along edge 0 - 1 */
+ 			XY[0] = (xy0[0] + xy1[0]) / 2;
+ 			XY[1] = (xy0[1] + xy1[1]) / 2;
+ 			UV[0] = (uv0[0] + uv1[0]) / 2;
+ 			UV[1] = (uv0[1] + uv1[1]) / 2;
+ 			n = split_triangle(coord, xy0, XY, xy2, uv0, UV, uv2,
+ 					   avail);
+ 			n += split_triangle(coord + 3 * n, XY, xy1, xy2,
+ 					    UV, uv1, uv2, avail - n);
+ 			break;
+ 		case 1:
+ 			/* split triangle along edge 1 - 2 */
+ 			XY[0] = (xy1[0] + xy2[0]) / 2;
+ 			XY[1] = (xy1[1] + xy2[1]) / 2;
+ 			UV[0] = (uv1[0] + uv2[0]) / 2;
+ 			UV[1] = (uv1[1] + uv2[1]) / 2;
+ 			n = split_triangle(coord, xy1, XY, xy0, uv1, UV, uv0,
+ 					   avail);
+ 			n += split_triangle(coord + 3 * n, XY, xy2, xy0,
+ 					    UV, uv2, uv0, avail - n);
+ 			break;
+ 		default:
+ 			/* split triangle along edge 2 - 0 */
+ 			XY[0] = (xy2[0] + xy0[0]) / 2;
+ 			XY[1] = (xy2[1] + xy0[1]) / 2;
+ 			UV[0] = (uv2[0] + uv0[0]) / 2;
+ 			UV[1] = (uv2[1] + uv0[1]) / 2;
+ 			n = split_triangle(coord, xy2, XY, xy1, uv2, UV, uv1,
+ 					   avail);
+ 			n += split_triangle(coord + 3 * n, XY, xy0, xy1,
+ 					    UV, uv0, uv1, avail - n);
+ 		}
+
+ 		/* return number of triangles added */
+ 		return n;
+ 	} else {
+ 		/* no need to split a rectangle; save coordinates */
+ 		coord[0].src.u = uv0[0];
+ 		coord[0].src.v = uv0[1];
+ 		coord[0].dst.x = xy0[0];
+ 		coord[0].dst.y = xy0[1];
+ 		coord[1].src.u = uv1[0];
+ 		coord[1].src.v = uv1[1];
+ 		coord[1].dst.x = xy1[0];
+ 		coord[1].dst.y = xy1[1];
+ 		coord[2].src.u = uv2[0];
+ 		coord[2].src.v = uv2[1];
+ 		coord[2].dst.x = xy2[0];
+ 		coord[2].dst.y = xy2[1];
+
+ 		/* single triangle is created */
+ 		return 1;
+ 	}
+ }
+
+ /* process single triangle */
+ static int process_triangle(struct imr_full_coord *coord, __u16 *uv, __s16 *xy,
+ 			     int avail)
+ {
+ 	/* cull invisible triangle first */
+ 	if ((xy[2] - xy[0]) * (xy[5] - xy[3]) >=
+ 	    (xy[3] - xy[1]) * (xy[4] - xy[2])) {
+ 		return 0;
+ 	} else {
+ 		/* recursively split triangle into smaller ones */
+ 		return split_triangle(coord, xy + 0, xy + 2, xy + 4,
+ 				      uv + 0, uv + 2, uv + 4, avail);
+ 	}
+ }
+
+ /* clamp texture coordinates to not exceed input dimensions */
+ static void clamp_texture(__u16 *uv, float *UV, int w, int h)
+ {
+ 	float t;
+
+ 	t = UV[0];
+ 	if (t < 0)
+ 		uv[0] = 0;
+ 	t *= w;
+ 	if (t > w - 1)
+ 		uv[0] = w - 1;
+ 	else
+ 		uv[0] = round(t);
+
+ 	t = UV[1];
+ 	if (t < 0)
+ 		uv[1] = 0;
+ 	t *= h;
+ 	if (t > h - 1)
+ 		uv[1] = h - 1;
+ 	else
+ 		uv[1] = round(t);
+ }
+
+ /* clamp vertex coordinates */
+ static int clamp_vertex(__s16 *xy, float *XY, int W, int H)
+ {
+ 	float x = round(XY[0] * W), y = round(XY[1] * H), z = XY[2];
+
+ 	if (z < 0.1)
+ 		return 0;
+ 	if (x < -(256 << IMR_DST_SUBSAMPLE) || x >= W + (256 << IMR_DST_SUBSAMPLE))
+ 		return 0;
+ 	if (y < -(256 << IMR_DST_SUBSAMPLE) || y >= H + (256 << IMR_DST_SUBSAMPLE))
+ 		return 0;
+
+ 	xy[0] = (__s16)x;
+ 	xy[1] = (__s16)y;
+
+ 	return 1;
+ }
+
+ /* create arbitrary mesh */
+ struct imr_map_desc *imr_map_create(struct imr_device *dev,
+ 				     float *uv, float *xy, int n)
+ {
+ 	struct imr_map_desc	*desc;
+ 	struct imr_vbo		*vbo;
+ 	struct imr_full_coord	*coord;
+ 	int			j, m, w, h, W, H;
+
+ 	/* create a configuration structure */
+ 	desc = malloc(sizeof(*desc) + sizeof(*vbo) + 3 * n * sizeof(*coord));
+ 	if (!desc)
+ 		return NULL;
+
+ 	/* fill-in VBO coordinates */
+ 	vbo = (void *)(desc + 1);
+ 	coord = (void *)(vbo + 1);
+
+ 	/* calculate source/destination dimensions in subpixel coordinates */
+ 	w = dev->w << IMR_SRC_SUBSAMPLE;
+ 	h = dev->h << IMR_SRC_SUBSAMPLE;
+ 	W = dev->W << IMR_DST_SUBSAMPLE;
+ 	H = dev->H << IMR_DST_SUBSAMPLE;
+
+ 	/* put at most N triangles into mesh descriptor */
+ 	for (j = 0, m = 0; j < n && m < n; j++, xy += 9, uv += 6) {
+ 		__u16	UV[6];
+ 		__s16	XY[6];
+ 		int	k;
+
+ 		/* translate model coordinates to fixed-point */
+ 		if (!clamp_vertex(XY + 0, xy + 0, W, H))
+ 			continue;
+ 		if (!clamp_vertex(XY + 2, xy + 3, W, H))
+ 			continue;
+ 		if (!clamp_vertex(XY + 4, xy + 6, W, H))
+ 			continue;
+
+ 		/* translate source coordinates */
+ 		clamp_texture(UV + 0, uv + 0, w, h);
+ 		clamp_texture(UV + 2, uv + 2, w, h);
+ 		clamp_texture(UV + 4, uv + 4, w, h);
+
+ 		/* process single triangle */
+ 		k = process_triangle(coord, UV, XY, n - m);
+ 		if (k != 0) {
+ 			/* advance vertex coordinates pointer */
+ 			coord += 3 * k;
+ 			m += k;
+ 		}
+ 	}
+
+ 	/* put number of triangles in VBO */
+ 	vbo->num = m;
+
+ 	/* fill-in descriptor */
+ 	desc->type = IMR_MAP_UVDPOR(IMR_SRC_SUBSAMPLE) |
+ 		     (IMR_DST_SUBSAMPLE ? IMR_MAP_DDP : 0);
+ 	desc->size = (void *)coord - (void *)vbo;
+ 	desc->data = (__u64)vbo;
+
+ 	return desc;
+ }
+
+ /* create rectangular mesh */
+ struct imr_map_desc *imr_map_mesh_src(struct imr_device *dev, float *uv,
+ 				       int rows, int columns,
+				       float x0, float y0, float dx, float dy)
+ {
+ 	struct imr_map_desc	*desc;
+ 	struct imr_mesh		*mesh;
+ 	struct imr_src_coord	*coord;
+ 	int			k, w, h, W, H;
+
+ 	/* create a configuration structure */
+ 	desc = malloc(sizeof(*desc) + sizeof(*mesh) + rows * columns *
+   		      sizeof(*coord));
+ 	if (!desc)
+ 		return NULL;
+
+ 	/* fill-in rectangular mesh coordinates */
+ 	mesh = (void *)(desc + 1);
+ 	coord = (void *)(mesh + 1);
+
+ 	/* calculate source/destination dimensions in subpixel coordinates */
+ 	w = dev->w << IMR_SRC_SUBSAMPLE;
+ 	h = dev->h << IMR_SRC_SUBSAMPLE;
+ 	W = dev->W << IMR_DST_SUBSAMPLE;
+ 	H = dev->H << IMR_DST_SUBSAMPLE;
+
+ 	/* set mesh parameters */
+ 	mesh->rows = rows;
+ 	mesh->columns = columns;
+ 	mesh->x0 = (__u16)round(x0 * W);
+ 	mesh->y0 = (__u16)round(y0 * H);
+ 	mesh->dx = (__u16)round(dx * W);
+ 	mesh->dy = (__u16)round(dy * H);
+
+ 	/* put mesh coordinates */
+ 	for (k = 0; k < rows * columns; k++, uv += 2, coord++) {
+ 		__u16	UV[2];
+
+ 		/* transform point into texture coordinates */
+ 		clamp_texture(UV, uv, w, h);
+
+ 		/* fill the mesh */
+ 		coord->u = UV[0];
+ 		coord->v = UV[1];
+ 	}
+
+ 	/* fill-in descriptor */
+ 	desc->type = IMR_MAP_MESH | IMR_MAP_AUTODG |
+ 		     IMR_MAP_UVDPOR(IMR_SRC_SUBSAMPLE) |
+ 		     (IMR_DST_SUBSAMPLE ? IMR_MAP_DDP : 0);
+ 	desc->size = (void *)coord - (void *)mesh;
+ 	desc->data = (__u64)mesh;
+
+ 	return desc;
+ }

<snip>

Quoted 66 lines

Index: media_tree/include/uapi/linux/rcar_imr.h
===================================================================

include/uapi/linux/rcar_imr.h new file +61 -0
@@ -0,0 +1,182 @@
+/*
+ * rcar_imr.h -- R-Car IMR-LX4 Driver UAPI
+ *
+ * Copyright (C) 2016-2017 Cogent Embedded, Inc. <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#ifndef __RCAR_IMR_H
+#define __RCAR_IMR_H
+
+#include <linux/videodev2.h>
+
+/*******************************************************************************
+ * Mapping specification descriptor
+ ******************************************************************************/
+
+struct imr_map_desc {
+	/* bitmask of the mapping type (see below) */
+	__u32			type;
+
+	/* total data size */
+	__u32			size;
+
+	/* data user-pointer */
+	__u64			data;
+} __attribute__((packed));
+
+/* regular mesh specification */
+#define IMR_MAP_MESH		(1 << 0)
+
+/* auto-generated source coordinates */
+#define IMR_MAP_AUTOSG		(1 << 1)
+
+/* auto-generated destination coordinates */
+#define IMR_MAP_AUTODG		(1 << 2)
+
+/* luma correction flag */
+#define IMR_MAP_LUCE		(1 << 3)
+
+/* chroma correction flag */
+#define IMR_MAP_CLCE		(1 << 4)
+
+/* vertex clockwise-mode order */
+#define IMR_MAP_TCM		(1 << 5)
+
+/* source coordinate decimal point position */
+#define __IMR_MAP_UVDPOR_SHIFT	8
+#define __IMR_MAP_UVDPOR(v)	(((v) >> __IMR_MAP_UVDPOR_SHIFT) & 0x7)
+#define IMR_MAP_UVDPOR(n)	(((n) & 0x7) << __IMR_MAP_UVDPOR_SHIFT)
+
+/* destination coordinate sub-pixel mode */
+#define IMR_MAP_DDP		(1 << 11)
+
+/* luminance correction scale decimal point position */
+#define __IMR_MAP_YLDPO_SHIFT	12
+#define __IMR_MAP_YLDPO(v)	(((v) >> __IMR_MAP_YLDPO_SHIFT) & 0x7)
+#define IMR_MAP_YLDPO(n)	(((n) & 0x7) << __IMR_MAP_YLDPO_SHIFT)

include/uapi/linux/rcar_imr.h line 71

Quoted 10 lines
include/uapi/linux/rcar_imr.h new file +10 -0
+
+/* chroma (U) correction scale decimal point position */
+#define __IMR_MAP_UBDPO_SHIFT	15
+#define __IMR_MAP_UBDPO(v)	(((v) >> __IMR_MAP_UBDPO_SHIFT) & 0x7)
+#define IMR_MAP_UBDPO(n)	(((n) & 0x7) << __IMR_MAP_UBDPO_SHIFT)
+
+/* chroma (V) correction scale decimal point position */
+#define __IMR_MAP_VRDPO_SHIFT	18
+#define __IMR_MAP_VRDPO(v)	(((v) >> __IMR_MAP_VRDPO_SHIFT) & 0x7)
+#define IMR_MAP_VRDPO(n)	(((n) & 0x7) << __IMR_MAP_VRDPO_SHIFT)

You need to document all these bits in the documentation. I.e. TCM and DDP are not currently documented.

It helps a lot to understand these defines if the documentation and the comments explain what the abbreviations mean. E.g. what does DDP stand for, or YLDPO? It's an alphabet soup right now.

include/uapi/linux/rcar_imr.h line 74

Quoted 3 lines
include/uapi/linux/rcar_imr.h new file +3 -0
+
+/* regular mesh specification */
+struct imr_mesh {

How about imr_rectangles? There is no indication in the struct name that this describes a rectangle.

Quoted 11 lines
include/uapi/linux/rcar_imr.h new file +11 -0
+	/* rectangular mesh size */
+	__u16			rows, columns;
+
+	/* auto-generated mesh parameters */
+	__u16			x0, y0, dx, dy;
+} __attribute__((packed));
+
+/* vertex-buffer-object (VBO) descriptor */
+struct imr_vbo {
+	/* number of triangles */
+	__u16			num;

include/uapi/linux/rcar_imr.h line 95

Quoted 10 lines
include/uapi/linux/rcar_imr.h new file +10 -0
+} __attribute__((packed));
+
+/*******************************************************************************
+ * Vertex-related structures
+ ******************************************************************************/
+
+/* source coordinates */
+struct imr_src_coord {
+	/* vertical, horizontal */
+	__u16			v, u;

Confusing: why isn't this 'v, h;' given the comment? Or does this refer to chroma (U and V)? And what does 'vertical, horizontal' mean anyway? Vertical what?

include/uapi/linux/rcar_imr.h line 100

Quoted 5 lines
include/uapi/linux/rcar_imr.h new file +5 -0
+} __attribute__((packed));
+
+/* destination coordinates */
+struct imr_dst_coord {
+	/* vertical, horizontal */

Confusing comment as well. I assume y and x are simply coordinates of a vertex? Just say so. This comment doesn't mean anything.

include/uapi/linux/rcar_imr.h line 112

Quoted 12 lines
include/uapi/linux/rcar_imr.h new file +12 -0
+	__u16			y, x;
+} __attribute__((packed));
+
+/* luma correction parameters */
+struct imr_luma_correct {
+	/* offset */
+	__s8			lofst;
+
+	/* scale */
+	__u8			lscal;
+
+	__u16			reserved;

Why is this reserved? Is that for padding? If so, then add a comment that mentions that.

Quoted 18 lines
include/uapi/linux/rcar_imr.h new file +18 -0
+} __attribute__((packed));
+
+/* chroma correction parameters */
+struct imr_chroma_correct {
+	/* V value offset */
+	__s8			vrofs;
+
+	/* V value scale */
+	__u8			vrscl;
+
+	/* U value offset */
+	__s8			ubofs;
+
+	/* V value scale */
+	__u8			ubscl;
+} __attribute__((packed));
+
+/* fully specified source/destination coordinates */

include/uapi/linux/rcar_imr.h line 140

Quoted 10 lines
include/uapi/linux/rcar_imr.h new file +10 -0
+struct imr_full_coord {
+	struct imr_src_coord	src;
+	struct imr_dst_coord	dst;
+} __attribute__((packed));
+
+/* auto-generated coordinates with luma or chroma correction */
+struct imr_auto_coord_any_correct {
+	union {
+		struct imr_src_coord src;
+		struct imr_dst_coord dst;

Why have separate imr_src_coord and imr_dst_coord structs? Why not just call it imr_coord? I think that is part of the reason of my confusion regarding understanding those structs.

The field name here indicates whether it is a source or destination, the coordinate information is the same for both.

Quoted 33 lines
include/uapi/linux/rcar_imr.h new file +32 -0
+	};
+	union {
+		struct imr_luma_correct luma;
+		struct imr_chroma_correct chroma;
+	};
+} __attribute__((packed));
+
+/* auto-generated coordinates with both luma and chroma correction */
+struct imr_auto_coord_both_correct {
+	union {
+		struct imr_src_coord src;
+		struct imr_dst_coord dst;
+	};
+	struct imr_luma_correct luma;
+	struct imr_chroma_correct chroma;
+} __attribute__((packed));
+
+/* fully specified coordinates with luma or chroma correction */
+struct imr_full_coord_any_correct {
+	struct imr_src_coord src;
+	struct imr_dst_coord dst;
+	union {
+		struct imr_luma_correct luma;
+		struct imr_chroma_correct chroma;
+	};
+} __attribute__((packed));
+
+/* fully specified coordinates with both luma and chroma correction */
+struct imr_full_coord_both_correct {
+	struct imr_src_coord src;
+	struct imr_dst_coord dst;
+	struct imr_luma_correct luma;

include/uapi/linux/rcar_imr.h line 182

Quoted 10 lines
include/uapi/linux/rcar_imr.h new file +10 -0
+	struct imr_chroma_correct chroma;
+} __attribute__((packed));
+
+/*******************************************************************************
+ * Private IOCTL codes
+ ******************************************************************************/
+
+#define VIDIOC_IMR_MESH _IOW('V', BASE_VIDIOC_PRIVATE + 0, struct imr_map_desc)
+
+#endif /* __RCAR_IMR_H */

Do you know the typical number of rectangles or triangles that are passed to this function? Is there an upper hardware limit?

I ask, because I wonder whether using a fixed vertex struct like imr_full_coord_both_correct for all variations isn't much simpler. The driver just ignores the fields it doesn't need in that case.

Yes, you get some memory overhead, but the code for both userspace and kernelspace will be much simpler.

Regards,

Hans

#4 lore raw
Quoted 645 lines · Hans Verkuil

Hi Sergei,

A quick review. I'm concentrating on the mesh ioctl, since that's what sets this driver apart.

On 08/04/2017 08:03 PM, Sergei Shtylyov wrote:

<snip>

Documentation/media/v4l-drivers/rcar_imr.rst line 5

Quoted 10 lines

Index: media_tree/Documentation/media/v4l-drivers/rcar_imr.rst
===================================================================

Documentation/media/v4l-drivers/rcar_imr.rst new file +5 -0
@@ -0,0 +1,372 @@
+Renesas R-Car Image Renderer (IMR) Driver
+=========================================
+
+This file documents some driver-specific aspects of the IMR driver, such as
+the driver-specific ioctl.

Just drop the part ', such as...'.

Can you add a high-level description of the functionality here? Similar to what you did in the bindings document.

Quoted 7 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +7 -0
+
+The ioctl reference
+~~~~~~~~~~~~~~~~~~~
+
+See ``include/uapi/linux/rcar_imr.h`` for the data structures used.
+
+VIDIOC_IMR_MESH - Set mapping data

Documentation/media/v4l-drivers/rcar_imr.rst line 22

Quoted 10 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +10 -0
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Argument: ``struct imr_map_desc``
+
+**Description**:
+
+This ioctl sets up the mesh through which the input frames will be transformed
+into the output frames. The mesh can be strictly rectangular (when
+``IMR_MAP_MESH`` bit is set in ``imr_map_desc::type``) or arbitrary (when that
+bit is not set).

Wouldn't something like 'IMR_MAP_RECTANGULAR' be much more descriptive than _MESH? There is nothing in the name _MESH to indicate that this switches the data to rectangles.

Documentation/media/v4l-drivers/rcar_imr.rst line 29

Quoted 7 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +7 -0
+
+A rectangular mesh consists of the imr_mesh structure followed by M*N vertex
+objects (where M is ``imr_mesh::rows`` and N is ``imr_mesh::columns``).
+In case either ``IMR_MAP_AUTOSG`` or ``IMR_MAP_AUTODG`` (not both) bits are
+set in ``imr_map_desc::type``, ``imr_mesh::{x|y}0`` specify the coordinates
+of the top left corner of the auto-generated mesh and ``imr_mesh::d{x|y}``
+specify the mesh's X/Y steps.

So if the auto bits are set, then there are no vertex objects? Since it's auto generated by the hardware?

I believe we discussed in the past whether 'type' should be split in a 'type' and 'flags' field.

Documentation/media/v4l-drivers/rcar_imr.rst line 40

Quoted 11 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +11 -0
+
+An arbitrary mesh consists of the imr_vbo structure followed by N triangle
+objects (where N is ``imr_vbo::num``), consisting of 3 vertex objects each.
+Setting ``IMR_MAP_AUTODG`` and ``IMR_MAP_AUTOSG`` bits in
+``imr_map_desc::type``) isn't allowed for this type of mesh.
+
+The vertex object has a complex structure depending on some of the bits in
+``imr_map_desc::type``:
+
+============  ============  ==============  ==============  ===========================
+IMR_MAP_CLCE  IMR_MAP_LUCE  IMR_MAP_AUTODG  IMR_MAP_AUTOSG  Vertex structure variant

You should explain the meaning of these bits in this section. I.e., what does CLCE or AUTODG stand for?

Documentation/media/v4l-drivers/rcar_imr.rst line 47

Quoted 7 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +7 -0
+============  ============  ==============  ==============  ===========================
+\                                                           ``imr_full_coord``
+\                                           X               ``imr_dst_coord``
+\                           X                               ``imr_src_coord``
+\             X                                             ``imr_full_coord_any_correct``
+\             X                             X               ``imr_auto_coord_any_correct``
+\             X             X                               ``imr_auto_coord_any_crrect``

crrect -> correct

Quoted 25 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +25 -0
+X                                                           ``imr_full_coord_any_correct``
+X                                           X               ``imr_auto_coord_any_correct``
+X                           X                               ``imr_auto_coord_any_correct``
+X             X                                             ``imr_full_coord_both_correct``
+X             X                             X               ``imr_auto_coord_both_correct``
+X             X             X                               ``imr_auto_coord_both_correct``
+============  ============  ==============  ==============  ===========================
+
+The luma correction is calculated according to the following formula (where
+``Y`` is the luma value after texture mapping, ``Y'`` is the luma value after
+luma correction, ``lscal`` and ``lofst`` are the luma correction scale and
+offset taken from ``struct imr_luma_correct``, ``YLDPO`` is a luma correction
+scale decimal point position specified by ``IMR_MAP_YLDPO(n)``): ::
+
+	Y' = ((Y * lscal) >> YLDPO) + lofst
+
+The chroma correction is calculated according to the following formula (where
+``U/V`` are the chroma values after texture mapping, ``U'/V'`` are the chroma
+values after chroma correction, ``ubscl/vrscl`` and ``ubofs/vrofs`` are the
+U/V value chroma correction scales and offsets taken from
+``struct imr_chroma_correct``, ``UBDPO/VRDPO`` are the chroma correction scale
+decimal point positions specified by ``IMR_MAP_{UBDPO|VRDPO}(n)``): ::
+
+	U' = ((U + ubofs) * ubscl) >> UBDPO
+	V' = ((V + vrofs) * vrscl) >> VRDPO

Documentation/media/v4l-drivers/rcar_imr.rst line 82

Quoted 10 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +10 -0
+
+**Return value**:
+
+On success 0 is returned. On error -1 is returned and ``errno`` is set
+appropriately.
+
+**Example code**:
+
+Below is an example code for constructing the meshes: ``imr_map_create()``
+constructs an arbitraty mesh, ``imr_map_mesh_src()`` function constructs

arbitrary

Documentation/media/v4l-drivers/rcar_imr.rst line 93

Quoted 11 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +11 -0
+a rectangular mesh with the auto-generated destination coordinates.
+
+.. code-block:: C
+
+ #include <malloc.h>
+ #include <math.h>
+ #include <linux/rcar_imr.h>
+
+ /* IMR device data */
+ struct imr_device {
+ 	/* V4L2 file decriptor */

descriptor

Quoted 279 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +279 -0
+ 	int			vfd;
+
+ 	/* input/output buffers dimensions */
+ 	int			w, h, W, H;
+ };
+
+ #define IMR_SRC_SUBSAMPLE	5
+ #define IMR_DST_SUBSAMPLE	2
+ #define IMR_COORD_THRESHOLD	(128 * 128 * (1 << 2 * IMR_DST_SUBSAMPLE))
+
+ /* find the longest side (index) */
+ static void find_longest_side(int n, __s16 xy0, __s16 xy1, int *max, int *side)
+ {
+ 	int t = xy1 - xy0;
+
+ 	t *= t;
+ 	if (*max < t) {
+ 		*max  = t;
+ 		*side = n;
+ 	}
+ }
+
+ /* recursively split a triangle until it can be passed to IMR */
+ static int split_triangle(struct imr_full_coord *coord,
+ 			   __s16 *xy0, __s16 *xy1, __s16 *xy2,
+ 			   __u16 *uv0, __u16 *uv1, __u16 *uv2, int avail)
+ {
+ 	int	max = 0, k = 0;
+
+ 	/* we need to have at least one available triangle */
+ 	if (avail < 1)
+ 		return 0;
+
+ 	find_longest_side(0, xy0[0], xy1[0], &max, &k);
+ 	find_longest_side(0, xy0[1], xy1[1], &max, &k);
+ 	find_longest_side(1, xy1[0], xy2[0], &max, &k);
+ 	find_longest_side(1, xy1[1], xy2[1], &max, &k);
+ 	find_longest_side(2, xy2[0], xy0[0], &max, &k);
+ 	find_longest_side(2, xy2[1], xy0[1], &max, &k);
+
+ 	/* if value exceeds the threshold, do splitting */
+ 	if (max >= IMR_COORD_THRESHOLD) {
+ 		__s16	XY[2];
+ 		__u16	UV[2];
+ 		int	n;
+
+ 		switch (k) {
+ 		case 0:
+ 			/* split triangle along edge 0 - 1 */
+ 			XY[0] = (xy0[0] + xy1[0]) / 2;
+ 			XY[1] = (xy0[1] + xy1[1]) / 2;
+ 			UV[0] = (uv0[0] + uv1[0]) / 2;
+ 			UV[1] = (uv0[1] + uv1[1]) / 2;
+ 			n = split_triangle(coord, xy0, XY, xy2, uv0, UV, uv2,
+ 					   avail);
+ 			n += split_triangle(coord + 3 * n, XY, xy1, xy2,
+ 					    UV, uv1, uv2, avail - n);
+ 			break;
+ 		case 1:
+ 			/* split triangle along edge 1 - 2 */
+ 			XY[0] = (xy1[0] + xy2[0]) / 2;
+ 			XY[1] = (xy1[1] + xy2[1]) / 2;
+ 			UV[0] = (uv1[0] + uv2[0]) / 2;
+ 			UV[1] = (uv1[1] + uv2[1]) / 2;
+ 			n = split_triangle(coord, xy1, XY, xy0, uv1, UV, uv0,
+ 					   avail);
+ 			n += split_triangle(coord + 3 * n, XY, xy2, xy0,
+ 					    UV, uv2, uv0, avail - n);
+ 			break;
+ 		default:
+ 			/* split triangle along edge 2 - 0 */
+ 			XY[0] = (xy2[0] + xy0[0]) / 2;
+ 			XY[1] = (xy2[1] + xy0[1]) / 2;
+ 			UV[0] = (uv2[0] + uv0[0]) / 2;
+ 			UV[1] = (uv2[1] + uv0[1]) / 2;
+ 			n = split_triangle(coord, xy2, XY, xy1, uv2, UV, uv1,
+ 					   avail);
+ 			n += split_triangle(coord + 3 * n, XY, xy0, xy1,
+ 					    UV, uv0, uv1, avail - n);
+ 		}
+
+ 		/* return number of triangles added */
+ 		return n;
+ 	} else {
+ 		/* no need to split a rectangle; save coordinates */
+ 		coord[0].src.u = uv0[0];
+ 		coord[0].src.v = uv0[1];
+ 		coord[0].dst.x = xy0[0];
+ 		coord[0].dst.y = xy0[1];
+ 		coord[1].src.u = uv1[0];
+ 		coord[1].src.v = uv1[1];
+ 		coord[1].dst.x = xy1[0];
+ 		coord[1].dst.y = xy1[1];
+ 		coord[2].src.u = uv2[0];
+ 		coord[2].src.v = uv2[1];
+ 		coord[2].dst.x = xy2[0];
+ 		coord[2].dst.y = xy2[1];
+
+ 		/* single triangle is created */
+ 		return 1;
+ 	}
+ }
+
+ /* process single triangle */
+ static int process_triangle(struct imr_full_coord *coord, __u16 *uv, __s16 *xy,
+ 			     int avail)
+ {
+ 	/* cull invisible triangle first */
+ 	if ((xy[2] - xy[0]) * (xy[5] - xy[3]) >=
+ 	    (xy[3] - xy[1]) * (xy[4] - xy[2])) {
+ 		return 0;
+ 	} else {
+ 		/* recursively split triangle into smaller ones */
+ 		return split_triangle(coord, xy + 0, xy + 2, xy + 4,
+ 				      uv + 0, uv + 2, uv + 4, avail);
+ 	}
+ }
+
+ /* clamp texture coordinates to not exceed input dimensions */
+ static void clamp_texture(__u16 *uv, float *UV, int w, int h)
+ {
+ 	float t;
+
+ 	t = UV[0];
+ 	if (t < 0)
+ 		uv[0] = 0;
+ 	t *= w;
+ 	if (t > w - 1)
+ 		uv[0] = w - 1;
+ 	else
+ 		uv[0] = round(t);
+
+ 	t = UV[1];
+ 	if (t < 0)
+ 		uv[1] = 0;
+ 	t *= h;
+ 	if (t > h - 1)
+ 		uv[1] = h - 1;
+ 	else
+ 		uv[1] = round(t);
+ }
+
+ /* clamp vertex coordinates */
+ static int clamp_vertex(__s16 *xy, float *XY, int W, int H)
+ {
+ 	float x = round(XY[0] * W), y = round(XY[1] * H), z = XY[2];
+
+ 	if (z < 0.1)
+ 		return 0;
+ 	if (x < -(256 << IMR_DST_SUBSAMPLE) || x >= W + (256 << IMR_DST_SUBSAMPLE))
+ 		return 0;
+ 	if (y < -(256 << IMR_DST_SUBSAMPLE) || y >= H + (256 << IMR_DST_SUBSAMPLE))
+ 		return 0;
+
+ 	xy[0] = (__s16)x;
+ 	xy[1] = (__s16)y;
+
+ 	return 1;
+ }
+
+ /* create arbitrary mesh */
+ struct imr_map_desc *imr_map_create(struct imr_device *dev,
+ 				     float *uv, float *xy, int n)
+ {
+ 	struct imr_map_desc	*desc;
+ 	struct imr_vbo		*vbo;
+ 	struct imr_full_coord	*coord;
+ 	int			j, m, w, h, W, H;
+
+ 	/* create a configuration structure */
+ 	desc = malloc(sizeof(*desc) + sizeof(*vbo) + 3 * n * sizeof(*coord));
+ 	if (!desc)
+ 		return NULL;
+
+ 	/* fill-in VBO coordinates */
+ 	vbo = (void *)(desc + 1);
+ 	coord = (void *)(vbo + 1);
+
+ 	/* calculate source/destination dimensions in subpixel coordinates */
+ 	w = dev->w << IMR_SRC_SUBSAMPLE;
+ 	h = dev->h << IMR_SRC_SUBSAMPLE;
+ 	W = dev->W << IMR_DST_SUBSAMPLE;
+ 	H = dev->H << IMR_DST_SUBSAMPLE;
+
+ 	/* put at most N triangles into mesh descriptor */
+ 	for (j = 0, m = 0; j < n && m < n; j++, xy += 9, uv += 6) {
+ 		__u16	UV[6];
+ 		__s16	XY[6];
+ 		int	k;
+
+ 		/* translate model coordinates to fixed-point */
+ 		if (!clamp_vertex(XY + 0, xy + 0, W, H))
+ 			continue;
+ 		if (!clamp_vertex(XY + 2, xy + 3, W, H))
+ 			continue;
+ 		if (!clamp_vertex(XY + 4, xy + 6, W, H))
+ 			continue;
+
+ 		/* translate source coordinates */
+ 		clamp_texture(UV + 0, uv + 0, w, h);
+ 		clamp_texture(UV + 2, uv + 2, w, h);
+ 		clamp_texture(UV + 4, uv + 4, w, h);
+
+ 		/* process single triangle */
+ 		k = process_triangle(coord, UV, XY, n - m);
+ 		if (k != 0) {
+ 			/* advance vertex coordinates pointer */
+ 			coord += 3 * k;
+ 			m += k;
+ 		}
+ 	}
+
+ 	/* put number of triangles in VBO */
+ 	vbo->num = m;
+
+ 	/* fill-in descriptor */
+ 	desc->type = IMR_MAP_UVDPOR(IMR_SRC_SUBSAMPLE) |
+ 		     (IMR_DST_SUBSAMPLE ? IMR_MAP_DDP : 0);
+ 	desc->size = (void *)coord - (void *)vbo;
+ 	desc->data = (__u64)vbo;
+
+ 	return desc;
+ }
+
+ /* create rectangular mesh */
+ struct imr_map_desc *imr_map_mesh_src(struct imr_device *dev, float *uv,
+ 				       int rows, int columns,
+				       float x0, float y0, float dx, float dy)
+ {
+ 	struct imr_map_desc	*desc;
+ 	struct imr_mesh		*mesh;
+ 	struct imr_src_coord	*coord;
+ 	int			k, w, h, W, H;
+
+ 	/* create a configuration structure */
+ 	desc = malloc(sizeof(*desc) + sizeof(*mesh) + rows * columns *
+   		      sizeof(*coord));
+ 	if (!desc)
+ 		return NULL;
+
+ 	/* fill-in rectangular mesh coordinates */
+ 	mesh = (void *)(desc + 1);
+ 	coord = (void *)(mesh + 1);
+
+ 	/* calculate source/destination dimensions in subpixel coordinates */
+ 	w = dev->w << IMR_SRC_SUBSAMPLE;
+ 	h = dev->h << IMR_SRC_SUBSAMPLE;
+ 	W = dev->W << IMR_DST_SUBSAMPLE;
+ 	H = dev->H << IMR_DST_SUBSAMPLE;
+
+ 	/* set mesh parameters */
+ 	mesh->rows = rows;
+ 	mesh->columns = columns;
+ 	mesh->x0 = (__u16)round(x0 * W);
+ 	mesh->y0 = (__u16)round(y0 * H);
+ 	mesh->dx = (__u16)round(dx * W);
+ 	mesh->dy = (__u16)round(dy * H);
+
+ 	/* put mesh coordinates */
+ 	for (k = 0; k < rows * columns; k++, uv += 2, coord++) {
+ 		__u16	UV[2];
+
+ 		/* transform point into texture coordinates */
+ 		clamp_texture(UV, uv, w, h);
+
+ 		/* fill the mesh */
+ 		coord->u = UV[0];
+ 		coord->v = UV[1];
+ 	}
+
+ 	/* fill-in descriptor */
+ 	desc->type = IMR_MAP_MESH | IMR_MAP_AUTODG |
+ 		     IMR_MAP_UVDPOR(IMR_SRC_SUBSAMPLE) |
+ 		     (IMR_DST_SUBSAMPLE ? IMR_MAP_DDP : 0);
+ 	desc->size = (void *)coord - (void *)mesh;
+ 	desc->data = (__u64)mesh;
+
+ 	return desc;
+ }

<snip>

Quoted 66 lines

Index: media_tree/include/uapi/linux/rcar_imr.h
===================================================================

include/uapi/linux/rcar_imr.h new file +61 -0
@@ -0,0 +1,182 @@
+/*
+ * rcar_imr.h -- R-Car IMR-LX4 Driver UAPI
+ *
+ * Copyright (C) 2016-2017 Cogent Embedded, Inc. <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#ifndef __RCAR_IMR_H
+#define __RCAR_IMR_H
+
+#include <linux/videodev2.h>
+
+/*******************************************************************************
+ * Mapping specification descriptor
+ ******************************************************************************/
+
+struct imr_map_desc {
+	/* bitmask of the mapping type (see below) */
+	__u32			type;
+
+	/* total data size */
+	__u32			size;
+
+	/* data user-pointer */
+	__u64			data;
+} __attribute__((packed));
+
+/* regular mesh specification */
+#define IMR_MAP_MESH		(1 << 0)
+
+/* auto-generated source coordinates */
+#define IMR_MAP_AUTOSG		(1 << 1)
+
+/* auto-generated destination coordinates */
+#define IMR_MAP_AUTODG		(1 << 2)
+
+/* luma correction flag */
+#define IMR_MAP_LUCE		(1 << 3)
+
+/* chroma correction flag */
+#define IMR_MAP_CLCE		(1 << 4)
+
+/* vertex clockwise-mode order */
+#define IMR_MAP_TCM		(1 << 5)
+
+/* source coordinate decimal point position */
+#define __IMR_MAP_UVDPOR_SHIFT	8
+#define __IMR_MAP_UVDPOR(v)	(((v) >> __IMR_MAP_UVDPOR_SHIFT) & 0x7)
+#define IMR_MAP_UVDPOR(n)	(((n) & 0x7) << __IMR_MAP_UVDPOR_SHIFT)
+
+/* destination coordinate sub-pixel mode */
+#define IMR_MAP_DDP		(1 << 11)
+
+/* luminance correction scale decimal point position */
+#define __IMR_MAP_YLDPO_SHIFT	12
+#define __IMR_MAP_YLDPO(v)	(((v) >> __IMR_MAP_YLDPO_SHIFT) & 0x7)
+#define IMR_MAP_YLDPO(n)	(((n) & 0x7) << __IMR_MAP_YLDPO_SHIFT)

include/uapi/linux/rcar_imr.h line 71

Quoted 10 lines
include/uapi/linux/rcar_imr.h new file +10 -0
+
+/* chroma (U) correction scale decimal point position */
+#define __IMR_MAP_UBDPO_SHIFT	15
+#define __IMR_MAP_UBDPO(v)	(((v) >> __IMR_MAP_UBDPO_SHIFT) & 0x7)
+#define IMR_MAP_UBDPO(n)	(((n) & 0x7) << __IMR_MAP_UBDPO_SHIFT)
+
+/* chroma (V) correction scale decimal point position */
+#define __IMR_MAP_VRDPO_SHIFT	18
+#define __IMR_MAP_VRDPO(v)	(((v) >> __IMR_MAP_VRDPO_SHIFT) & 0x7)
+#define IMR_MAP_VRDPO(n)	(((n) & 0x7) << __IMR_MAP_VRDPO_SHIFT)

You need to document all these bits in the documentation. I.e. TCM and DDP are not currently documented.

It helps a lot to understand these defines if the documentation and the comments explain what the abbreviations mean. E.g. what does DDP stand for, or YLDPO? It's an alphabet soup right now.

include/uapi/linux/rcar_imr.h line 74

Quoted 3 lines
include/uapi/linux/rcar_imr.h new file +3 -0
+
+/* regular mesh specification */
+struct imr_mesh {

How about imr_rectangles? There is no indication in the struct name that this describes a rectangle.

Quoted 11 lines
include/uapi/linux/rcar_imr.h new file +11 -0
+	/* rectangular mesh size */
+	__u16			rows, columns;
+
+	/* auto-generated mesh parameters */
+	__u16			x0, y0, dx, dy;
+} __attribute__((packed));
+
+/* vertex-buffer-object (VBO) descriptor */
+struct imr_vbo {
+	/* number of triangles */
+	__u16			num;

include/uapi/linux/rcar_imr.h line 95

Quoted 10 lines
include/uapi/linux/rcar_imr.h new file +10 -0
+} __attribute__((packed));
+
+/*******************************************************************************
+ * Vertex-related structures
+ ******************************************************************************/
+
+/* source coordinates */
+struct imr_src_coord {
+	/* vertical, horizontal */
+	__u16			v, u;

Confusing: why isn't this 'v, h;' given the comment? Or does this refer to chroma (U and V)? And what does 'vertical, horizontal' mean anyway? Vertical what?

include/uapi/linux/rcar_imr.h line 100

Quoted 5 lines
include/uapi/linux/rcar_imr.h new file +5 -0
+} __attribute__((packed));
+
+/* destination coordinates */
+struct imr_dst_coord {
+	/* vertical, horizontal */

Confusing comment as well. I assume y and x are simply coordinates of a vertex? Just say so. This comment doesn't mean anything.

include/uapi/linux/rcar_imr.h line 112

Quoted 12 lines
include/uapi/linux/rcar_imr.h new file +12 -0
+	__u16			y, x;
+} __attribute__((packed));
+
+/* luma correction parameters */
+struct imr_luma_correct {
+	/* offset */
+	__s8			lofst;
+
+	/* scale */
+	__u8			lscal;
+
+	__u16			reserved;

Why is this reserved? Is that for padding? If so, then add a comment that mentions that.

Quoted 18 lines
include/uapi/linux/rcar_imr.h new file +18 -0
+} __attribute__((packed));
+
+/* chroma correction parameters */
+struct imr_chroma_correct {
+	/* V value offset */
+	__s8			vrofs;
+
+	/* V value scale */
+	__u8			vrscl;
+
+	/* U value offset */
+	__s8			ubofs;
+
+	/* V value scale */
+	__u8			ubscl;
+} __attribute__((packed));
+
+/* fully specified source/destination coordinates */

include/uapi/linux/rcar_imr.h line 140

Quoted 10 lines
include/uapi/linux/rcar_imr.h new file +10 -0
+struct imr_full_coord {
+	struct imr_src_coord	src;
+	struct imr_dst_coord	dst;
+} __attribute__((packed));
+
+/* auto-generated coordinates with luma or chroma correction */
+struct imr_auto_coord_any_correct {
+	union {
+		struct imr_src_coord src;
+		struct imr_dst_coord dst;

Why have separate imr_src_coord and imr_dst_coord structs? Why not just call it imr_coord? I think that is part of the reason of my confusion regarding understanding those structs.

The field name here indicates whether it is a source or destination, the coordinate information is the same for both.

Quoted 33 lines
include/uapi/linux/rcar_imr.h new file +32 -0
+	};
+	union {
+		struct imr_luma_correct luma;
+		struct imr_chroma_correct chroma;
+	};
+} __attribute__((packed));
+
+/* auto-generated coordinates with both luma and chroma correction */
+struct imr_auto_coord_both_correct {
+	union {
+		struct imr_src_coord src;
+		struct imr_dst_coord dst;
+	};
+	struct imr_luma_correct luma;
+	struct imr_chroma_correct chroma;
+} __attribute__((packed));
+
+/* fully specified coordinates with luma or chroma correction */
+struct imr_full_coord_any_correct {
+	struct imr_src_coord src;
+	struct imr_dst_coord dst;
+	union {
+		struct imr_luma_correct luma;
+		struct imr_chroma_correct chroma;
+	};
+} __attribute__((packed));
+
+/* fully specified coordinates with both luma and chroma correction */
+struct imr_full_coord_both_correct {
+	struct imr_src_coord src;
+	struct imr_dst_coord dst;
+	struct imr_luma_correct luma;

include/uapi/linux/rcar_imr.h line 182

Quoted 10 lines
include/uapi/linux/rcar_imr.h new file +10 -0
+	struct imr_chroma_correct chroma;
+} __attribute__((packed));
+
+/*******************************************************************************
+ * Private IOCTL codes
+ ******************************************************************************/
+
+#define VIDIOC_IMR_MESH _IOW('V', BASE_VIDIOC_PRIVATE + 0, struct imr_map_desc)
+
+#endif /* __RCAR_IMR_H */

Do you know the typical number of rectangles or triangles that are passed to this function? Is there an upper hardware limit?

Quoted 6 lines

I ask, because I wonder whether using a fixed vertex struct like imr_full_coord_both_correct for all variations isn't much simpler. The driver just ignores the fields it doesn't need in that case.

Yes, you get some memory overhead, but the code for both userspace and kernelspace will be much simpler.

I just had a brainwave: rather than making these complicated structure variants, why not just do this (just thrown together, I didn't look at alignment etc.):

struct imr_map_desc {
	/* bitmask of the mapping type (see below) */
	__u32			type;

	/* total data size */
	__u32			size;

	/* number of vertices */
	__u32			vertices;

	/* rectangular mesh size (ignored for triangles) */
	__u16			rows, columns;

	/* auto-generated mesh parameters (ignored if not auto-generated) */
	__u16			x0, y0, dx, dy;

	/* data user-pointer */
	__u64			src_coords;
	__u64			dst_coords;
	__u64			luma_corr;
	__u64			chroma_corr;
} __attribute__((packed));

Just leave the corresponding data pointer 0 when not needed, and otherwise these data pointers point to a struct imr_coord, imr_luma_corr or imr_chroma_corr arrays. Much simpler and still memory efficient.

BTW, I prefer either _corr or _correction over _correct. 'correct' is confusing since it is also a normal english word and it is not obvious that it is an abbreviation for 'correction'.

Regards,

Hans

#5 lore raw
In reply to Hans Verkuil

Hello!

Quoted 2 lines · Hans Verkuil

A quick review. I'm concentrating on the mesh ioctl, since that's what sets this driver apart.

OK, waiting for the detailed review...

Quoted 13 lines

Documentation/media/v4l-drivers/rcar_imr.rst line 5

Quoted 10 lines

Index: media_tree/Documentation/media/v4l-drivers/rcar_imr.rst
===================================================================

Documentation/media/v4l-drivers/rcar_imr.rst new file +5 -0
@@ -0,0 +1,372 @@
+Renesas R-Car Image Renderer (IMR) Driver
+=========================================
+
+This file documents some driver-specific aspects of the IMR driver, such as
+the driver-specific ioctl.

Just drop the part ', such as...'.

Quoted 2 lines

Can you add a high-level description of the functionality here? Similar to what you did in the bindings document.

Sure, I can.

Quoted 13 lines
Quoted 13 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +13 -0
+
+The ioctl reference
+~~~~~~~~~~~~~~~~~~~
+
+See ``include/uapi/linux/rcar_imr.h`` for the data structures used.
+
+VIDIOC_IMR_MESH - Set mapping data
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Argument: ``struct imr_map_desc``
+
+**Description**:
+
Quoted 8 lines

Documentation/media/v4l-drivers/rcar_imr.rst line 22

Quoted 4 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +4 -0
+This ioctl sets up the mesh through which the input frames will be transformed
+into the output frames. The mesh can be strictly rectangular (when
+``IMR_MAP_MESH`` bit is set in ``imr_map_desc::type``) or arbitrary (when that
+bit is not set).

Wouldn't something like 'IMR_MAP_RECTANGULAR' be much more descriptive than _MESH? There is nothing in the name _MESH to indicate that this switches the data to rectangles.

Well, in my Russian mind, mesh consists of the rectangles. :-)

Quoted 9 lines

Documentation/media/v4l-drivers/rcar_imr.rst line 29

Quoted 7 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +7 -0
+
+A rectangular mesh consists of the imr_mesh structure followed by M*N vertex
+objects (where M is ``imr_mesh::rows`` and N is ``imr_mesh::columns``).
+In case either ``IMR_MAP_AUTOSG`` or ``IMR_MAP_AUTODG`` (not both) bits are
+set in ``imr_map_desc::type``, ``imr_mesh::{x|y}0`` specify the coordinates
+of the top left corner of the auto-generated mesh and ``imr_mesh::d{x|y}``
+specify the mesh's X/Y steps.

So if the auto bits are set, then there are no vertex objects?

No, there are no source or destination (not both) coordinate structures
present in the vertex object; at least one of them must always be there.

Quoted 1 line

Since it's auto generated by the hardware?

Yes.

Quoted 2 lines

I believe we discussed in the past whether 'type' should be split in a 'type' and 'flags' field.

In this version, I still tried Cogent's original userland working, so the
structures were left intact (aside of renaming).

Quoted 6 lines
Quoted 6 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +6 -0
+
+An arbitrary mesh consists of the imr_vbo structure followed by N triangle
+objects (where N is ``imr_vbo::num``), consisting of 3 vertex objects each.
+Setting ``IMR_MAP_AUTODG`` and ``IMR_MAP_AUTOSG`` bits in
+``imr_map_desc::type``) isn't allowed for this type of mesh.
+
Quoted 8 lines

Documentation/media/v4l-drivers/rcar_imr.rst line 40

Quoted 5 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +5 -0
+The vertex object has a complex structure depending on some of the bits in
+``imr_map_desc::type``:
+
+============  ============  ==============  ==============  ===========================
+IMR_MAP_CLCE  IMR_MAP_LUCE  IMR_MAP_AUTODG  IMR_MAP_AUTOSG  Vertex structure variant

You should explain the meaning of these bits in this section. I.e., what does CLCE or AUTODG stand for?

I think I have explained the IRM_MAPO_AUTO* bits above.

Quoted 39 lines

Documentation/media/v4l-drivers/rcar_imr.rst line 47

Quoted 7 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +7 -0
+============  ============  ==============  ==============  ===========================
+\                                                           ``imr_full_coord``
+\                                           X               ``imr_dst_coord``
+\                           X                               ``imr_src_coord``
+\             X                                             ``imr_full_coord_any_correct``
+\             X                             X               ``imr_auto_coord_any_correct``
+\             X             X                               ``imr_auto_coord_any_crrect``

crrect -> correct

Quoted 29 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +29 -0
+X                                                           ``imr_full_coord_any_correct``
+X                                           X               ``imr_auto_coord_any_correct``
+X                           X                               ``imr_auto_coord_any_correct``
+X             X                                             ``imr_full_coord_both_correct``
+X             X                             X               ``imr_auto_coord_both_correct``
+X             X             X                               ``imr_auto_coord_both_correct``
+============  ============  ==============  ==============  ===========================
+
+The luma correction is calculated according to the following formula (where
+``Y`` is the luma value after texture mapping, ``Y'`` is the luma value after
+luma correction, ``lscal`` and ``lofst`` are the luma correction scale and
+offset taken from ``struct imr_luma_correct``, ``YLDPO`` is a luma correction
+scale decimal point position specified by ``IMR_MAP_YLDPO(n)``): ::
+
+	Y' = ((Y * lscal) >> YLDPO) + lofst
+
+The chroma correction is calculated according to the following formula (where
+``U/V`` are the chroma values after texture mapping, ``U'/V'`` are the chroma
+values after chroma correction, ``ubscl/vrscl`` and ``ubofs/vrofs`` are the
+U/V value chroma correction scales and offsets taken from
+``struct imr_chroma_correct``, ``UBDPO/VRDPO`` are the chroma correction scale
+decimal point positions specified by ``IMR_MAP_{UBDPO|VRDPO}(n)``): ::
+
+	U' = ((U + ubofs) * ubscl) >> UBDPO
+	V' = ((V + vrofs) * vrscl) >> VRDPO
+
+**Return value**:
+
+On success 0 is returned. On error -1 is returned and ``errno`` is set
Quoted 8 lines

Documentation/media/v4l-drivers/rcar_imr.rst line 82

Quoted 6 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +6 -0
+appropriately.
+
+**Example code**:
+
+Below is an example code for constructing the meshes: ``imr_map_create()``
+constructs an arbitraty mesh, ``imr_map_mesh_src()`` function constructs

arbitrary

TY, will fix.

Quoted 5 lines
Quoted 5 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +5 -0
+a rectangular mesh with the auto-generated destination coordinates.
+
+.. code-block:: C
+
+ #include <malloc.h>
Quoted 8 lines

Documentation/media/v4l-drivers/rcar_imr.rst line 93

Quoted 6 lines
Documentation/media/v4l-drivers/rcar_imr.rst new file +6 -0
+ #include <math.h>
+ #include <linux/rcar_imr.h>
+
+ /* IMR device data */
+ struct imr_device {
+ 	/* V4L2 file decriptor */

descriptor

Will fix too.

Quoted 1 line

<snip>

Not enough of a <snip> it seems -- I had to cut out the whole code
example. :-)

Quoted 71 lines
Quoted 71 lines

Index: media_tree/include/uapi/linux/rcar_imr.h
===================================================================

include/uapi/linux/rcar_imr.h new file +66 -0
@@ -0,0 +1,182 @@
+/*
+ * rcar_imr.h -- R-Car IMR-LX4 Driver UAPI
+ *
+ * Copyright (C) 2016-2017 Cogent Embedded, Inc. <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#ifndef __RCAR_IMR_H
+#define __RCAR_IMR_H
+
+#include <linux/videodev2.h>
+
+/*******************************************************************************
+ * Mapping specification descriptor
+ ******************************************************************************/
+
+struct imr_map_desc {
+	/* bitmask of the mapping type (see below) */
+	__u32			type;
+
+	/* total data size */
+	__u32			size;
+
+	/* data user-pointer */
+	__u64			data;
+} __attribute__((packed));
+
+/* regular mesh specification */
+#define IMR_MAP_MESH		(1 << 0)
+
+/* auto-generated source coordinates */
+#define IMR_MAP_AUTOSG		(1 << 1)
+
+/* auto-generated destination coordinates */
+#define IMR_MAP_AUTODG		(1 << 2)
+
+/* luma correction flag */
+#define IMR_MAP_LUCE		(1 << 3)
+
+/* chroma correction flag */
+#define IMR_MAP_CLCE		(1 << 4)
+
+/* vertex clockwise-mode order */
+#define IMR_MAP_TCM		(1 << 5)
+
+/* source coordinate decimal point position */
+#define __IMR_MAP_UVDPOR_SHIFT	8
+#define __IMR_MAP_UVDPOR(v)	(((v) >> __IMR_MAP_UVDPOR_SHIFT) & 0x7)
+#define IMR_MAP_UVDPOR(n)	(((n) & 0x7) << __IMR_MAP_UVDPOR_SHIFT)
+
+/* destination coordinate sub-pixel mode */
+#define IMR_MAP_DDP		(1 << 11)
+
+/* luminance correction scale decimal point position */
+#define __IMR_MAP_YLDPO_SHIFT	12
+#define __IMR_MAP_YLDPO(v)	(((v) >> __IMR_MAP_YLDPO_SHIFT) & 0x7)
+#define IMR_MAP_YLDPO(n)	(((n) & 0x7) << __IMR_MAP_YLDPO_SHIFT)
+
+/* chroma (U) correction scale decimal point position */
+#define __IMR_MAP_UBDPO_SHIFT	15
+#define __IMR_MAP_UBDPO(v)	(((v) >> __IMR_MAP_UBDPO_SHIFT) & 0x7)
+#define IMR_MAP_UBDPO(n)	(((n) & 0x7) << __IMR_MAP_UBDPO_SHIFT)
Quoted 8 lines

include/uapi/linux/rcar_imr.h line 71

Quoted 5 lines
include/uapi/linux/rcar_imr.h new file +5 -0
+
+/* chroma (V) correction scale decimal point position */
+#define __IMR_MAP_VRDPO_SHIFT	18
+#define __IMR_MAP_VRDPO(v)	(((v) >> __IMR_MAP_VRDPO_SHIFT) & 0x7)
+#define IMR_MAP_VRDPO(n)	(((n) & 0x7) << __IMR_MAP_VRDPO_SHIFT)

You need to document all these bits in the documentation. I.e. TCM and DDP are not currently documented.

I thought the comments above the #define's were enough.

Quoted 3 lines

It helps a lot to understand these defines if the documentation and the comments explain what the abbreviations mean. E.g. what does DDP stand for, or YLDPO? It's an alphabet soup right now.

I think I explained YLDPO where I was describing the luma correction.

Quoted 6 lines

include/uapi/linux/rcar_imr.h line 74

Quoted 3 lines
include/uapi/linux/rcar_imr.h new file +3 -0
+
+/* regular mesh specification */
+struct imr_mesh {

How about imr_rectangles? There is no indication in the struct name that this describes a rectangle.

OK, if you think it's not obvious.

Quoted 15 lines
Quoted 15 lines
include/uapi/linux/rcar_imr.h new file +15 -0
+	/* rectangular mesh size */
+	__u16			rows, columns;
+
+	/* auto-generated mesh parameters */
+	__u16			x0, y0, dx, dy;
+} __attribute__((packed));
+
+/* vertex-buffer-object (VBO) descriptor */
+struct imr_vbo {
+	/* number of triangles */
+	__u16			num;
+} __attribute__((packed));
+
+/*******************************************************************************
+ * Vertex-related structures
Quoted 8 lines

include/uapi/linux/rcar_imr.h line 95

Quoted 6 lines
include/uapi/linux/rcar_imr.h new file +6 -0
+ ******************************************************************************/
+
+/* source coordinates */
+struct imr_src_coord {
+	/* vertical, horizontal */
+	__u16			v, u;

Confusing: why isn't this 'v, h;' given the comment?

Because the source coordinate spaces are called this way in the manual: u
and v. And the vertical coordinate goes first in the display lists.

Quoted 1 line

Or does this refer to chroma (U and V)?

No! :-)

Quoted 2 lines

And what does 'vertical, horizontal' mean anyway? Vertical what?

I thought that was obvious given the comment above the structure...

Quoted 8 lines

include/uapi/linux/rcar_imr.h line 100

Quoted 5 lines
include/uapi/linux/rcar_imr.h new file +5 -0
+} __attribute__((packed));
+
+/* destination coordinates */
+struct imr_dst_coord {
+	/* vertical, horizontal */

Confusing comment as well. I assume y and x are simply coordinates of a vertex? Just say so. This comment doesn't mean anything.

Please see the comment to the whole structure again. :-)

Quoted 7 lines
Quoted 7 lines
include/uapi/linux/rcar_imr.h new file +7 -0
+	__u16			y, x;
+} __attribute__((packed));
+
+/* luma correction parameters */
+struct imr_luma_correct {
+	/* offset */
+	__s8			lofst;
Quoted 8 lines

include/uapi/linux/rcar_imr.h line 112

Quoted 5 lines
include/uapi/linux/rcar_imr.h new file +5 -0
+
+	/* scale */
+	__u8			lscal;
+
+	__u16			reserved;

Why is this reserved? Is that for padding? If so, then add a comment that mentions that.

Yes, padding in the display list.

Quoted 33 lines
Quoted 18 lines
include/uapi/linux/rcar_imr.h new file +18 -0
+} __attribute__((packed));
+
+/* chroma correction parameters */
+struct imr_chroma_correct {
+	/* V value offset */
+	__s8			vrofs;
+
+	/* V value scale */
+	__u8			vrscl;
+
+	/* U value offset */
+	__s8			ubofs;
+
+	/* V value scale */
+	__u8			ubscl;
+} __attribute__((packed));
+
+/* fully specified source/destination coordinates */

include/uapi/linux/rcar_imr.h line 140

Quoted 10 lines
include/uapi/linux/rcar_imr.h new file +10 -0
+struct imr_full_coord {
+	struct imr_src_coord	src;
+	struct imr_dst_coord	dst;
+} __attribute__((packed));
+
+/* auto-generated coordinates with luma or chroma correction */
+struct imr_auto_coord_any_correct {
+	union {
+		struct imr_src_coord src;
+		struct imr_dst_coord dst;

Why have separate imr_src_coord and imr_dst_coord structs? Why not just call it imr_coord? I think that is part of the reason of my confusion regarding understanding those structs.

Quoted 2 lines

The field name here indicates whether it is a source or destination, the coordinate information is the same for both.

The manual keeps talking about the different coordinate spaces... can fix,
I guess.

Quoted 38 lines
Quoted 38 lines
include/uapi/linux/rcar_imr.h new file +38 -0
+	};
+	union {
+		struct imr_luma_correct luma;
+		struct imr_chroma_correct chroma;
+	};
+} __attribute__((packed));
+
+/* auto-generated coordinates with both luma and chroma correction */
+struct imr_auto_coord_both_correct {
+	union {
+		struct imr_src_coord src;
+		struct imr_dst_coord dst;
+	};
+	struct imr_luma_correct luma;
+	struct imr_chroma_correct chroma;
+} __attribute__((packed));
+
+/* fully specified coordinates with luma or chroma correction */
+struct imr_full_coord_any_correct {
+	struct imr_src_coord src;
+	struct imr_dst_coord dst;
+	union {
+		struct imr_luma_correct luma;
+		struct imr_chroma_correct chroma;
+	};
+} __attribute__((packed));
+
+/* fully specified coordinates with both luma and chroma correction */
+struct imr_full_coord_both_correct {
+	struct imr_src_coord src;
+	struct imr_dst_coord dst;
+	struct imr_luma_correct luma;
+	struct imr_chroma_correct chroma;
+} __attribute__((packed));
+
+/*******************************************************************************
+ * Private IOCTL codes
+ ******************************************************************************/
Quoted 8 lines

include/uapi/linux/rcar_imr.h line 182

Quoted 5 lines
include/uapi/linux/rcar_imr.h new file +4 -0
+
+#define VIDIOC_IMR_MESH _IOW('V', BASE_VIDIOC_PRIVATE + 0, struct imr_map_desc)
+
+#endif /* __RCAR_IMR_H */

Do you know the typical number of rectangles or triangles that are passed to this function? Is there an upper hardware limit?

There's a limit on the number of vertices for each TRI instruction, 65535.
The coordinate limits are 2047 max, so 16 bits are enough for them.

Quoted 6 lines

I ask, because I wonder whether using a fixed vertex struct like imr_full_coord_both_correct for all variations isn't much simpler. The driver just ignores the fields it doesn't need in that case.

Yes, you get some memory overhead, but the code for both userspace and kernelspace will be much simpler.

Well, I need to think about it...

Quoted 3 lines

Regards,

Hans

MBR, Sergei

#6 lore raw
In reply to Sergei Shtylyov

Hi Sergei,

Quoted 16 lines · Sergei Shtylyov

The image renderer, or the distortion correction engine, is a drawing processor with a simple instruction system capable of referencing video capture data or data in an external memory as the 2D texture data and performing texture mapping and drawing with respect to any shape that is split into triangular objects.

This V4L2 memory-to-memory device driver only supports image renderer light extended 4 (IMR-LX4) found in the R-Car gen3 SoCs; the R-Car gen2 support can be added later...

Based on the original patch by Konstantin Kozhevnikov.

Quoted 8 lines
 Documentation/devicetree/bindings/media/rcar_imr.txt |   27
 Documentation/media/v4l-drivers/index.rst            |    1
 Documentation/media/v4l-drivers/rcar_imr.rst         |  372 +++
 drivers/media/platform/Kconfig                       |   13
 drivers/media/platform/Makefile                      |    1
 drivers/media/platform/rcar_imr.c                    | 1832 +++++++++++++++++++
 include/uapi/linux/rcar_imr.h                        |  182 +
 7 files changed, 2428 insertions(+)

What's the status of this patch?
The compatible value "renesas,r8a7796-imr-lx4" has been in use since v4.14.

Thanks!

Gr{oetje,eeting}s,

Geert

Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
#7 lore raw
Quoted 25 lines · Geert Uytterhoeven
Quoted 16 lines

The image renderer, or the distortion correction engine, is a drawing processor with a simple instruction system capable of referencing video capture data or data in an external memory as the 2D texture data and performing texture mapping and drawing with respect to any shape that is split into triangular objects.

This V4L2 memory-to-memory device driver only supports image renderer light extended 4 (IMR-LX4) found in the R-Car gen3 SoCs; the R-Car gen2 support can be added later...

Based on the original patch by Konstantin Kozhevnikov.

Quoted 8 lines
 Documentation/devicetree/bindings/media/rcar_imr.txt |   27
 Documentation/media/v4l-drivers/index.rst            |    1
 Documentation/media/v4l-drivers/rcar_imr.rst         |  372 +++
 drivers/media/platform/Kconfig                       |   13
 drivers/media/platform/Makefile                      |    1
 drivers/media/platform/rcar_imr.c                    | 1832 +++++++++++++++++++
 include/uapi/linux/rcar_imr.h                        |  182 +
 7 files changed, 2428 insertions(+)
Quoted 1 line

What's the status of this patch?

Changes requested, and I'm still having no bandwidth to make them...

Quoted 1 line

The compatible value "renesas,r8a7796-imr-lx4" has been in use since v4.14.

That's because the SoC bindings are unlikely to change...

Quoted 5 lines

Thanks!

Gr{oetje,eeting}s,

Geert

MBR, Sergei