Skip to contentSkip to conversation

media: vim2m: add multiplanar API support

PATCH In review Matthew Majewski 2 participants 7 messages Mar 4, 2025 – Apr 29, 2025 Media (V4L/DVB)

lore
Share & tools
lore
#0 lore raw

cover media: vim2m: add multiplanar API support

Hi everyone,

This patch series adds multiplane API support for the virtual M2M driver, along with some minor driver refactoring/improvements.

I followed the lead of the vivid driver and made multiplanar support selectable with a module parameter, and the default is to use the single planar api.

Although there are not yet any pixelformats in the driver that make use of multiple memory planes, adding support for these should be easier now with the API level changes taken care of.

v4l2-compliance reports the following with multiplane support disabled:

Total for vim2m device /dev/video0: 48, Succeeded: 48, Failed: 0, Warnings: 0

and the same with multiplane support enabled:

Total for vim2m device /dev/video0: 48, Succeeded: 48, Failed: 0, Warnings: 0

Patches need to be applied in increasing numerical order (Patch [3/3] depends on [1/3] and [2/3]).

Since the multi-plane changes had to touch a lot of the driver, I did a basic regression test with the following script which generates a test input image with vivid and an output image from vim2m for each supported format. I confirmed all outputs visually and verified they were identical to the outputs before the change. Testing was done on an x86_64 qemu image.

#!/bin/sh

# tested with HDMI vivid emulation
# modprobe vivid num_inputs=1 input_types=3

vim2m=/dev/video0
vivid=/dev/video1

width=640
height=480
out_width=320
out_height=240

capture_formats=$(v4l2-ctl -d $vim2m --list-formats | awk '/\]:/ {print $2}' | sed "s/'//g") output_formats=$(v4l2-ctl -d $vim2m --list-formats-out | awk '/\]:/ {print $2}' | sed "s/'//g")

# Turn off text mode so that images will be identical v4l2-ctl -d $vivid -c osd_text_mode=2

for ofmt in ${output_formats}; do

    # generate input image
    inname="${width}x${height}.${ofmt}"
    v4l2-ctl -d $vivid -v pixelformat=$ofmt,width=$width,height=$height,field=none \
             --stream-mmap --stream-count=1 --stream-to=$inname
    for cfmt in ${capture_formats}; do
        outname="${out_width}x${out_height}-out.${cfmt}"
        v4l2-ctl -d $vim2m -x pixelformat=$ofmt,width=$width,height=$height \
                 -v pixelformat=$cfmt,width=$out_width,height=$out_height \
                 --stream-from=$inname --stream-to=$outname --stream-mmap --stream-out-mmap \
                 --stream-count=1
    done
done

Matthew Majewski (3):
media: v4l2-common: Add RGBR format info
media: vim2m: Simplify try_fmt
media: vim2m: Add parametized support for multiplanar API

 drivers/media/test-drivers/vim2m.c    | 327 +++++++++++++++++++++-----
 drivers/media/v4l2-core/v4l2-common.c |   1 +
 2 files changed, 274 insertions(+), 54 deletions(-)
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/3 media: v4l2-common: Add RGBR format info

Add missing RGBR entry in the v4l2_format_info[] table.

RGBR has identical format information to RGBP, as it is a big endian variant of RGB-5-6-5 pixel encoding according to the description in videodev2.h.

---
 drivers/media/v4l2-core/v4l2-common.c | 1 +
 1 file changed, 1 insertion(+)
drivers/media/v4l2-core/v4l2-common.c +1 -0
index 0a2f4f0d0a07..b33035760e12 100644
@@ -250,6 +250,7 @@ const struct v4l2_format_info *v4l2_format_info(u32 format)
 		{ .format = V4L2_PIX_FMT_ABGR32,  .pixel_enc = V4L2_PIXEL_ENC_RGB, .mem_planes = 1, .comp_planes = 1, .bpp = { 4, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },
 		{ .format = V4L2_PIX_FMT_BGRA32,  .pixel_enc = V4L2_PIXEL_ENC_RGB, .mem_planes = 1, .comp_planes = 1, .bpp = { 4, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },
 		{ .format = V4L2_PIX_FMT_RGB565,  .pixel_enc = V4L2_PIXEL_ENC_RGB, .mem_planes = 1, .comp_planes = 1, .bpp = { 2, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },
+		{ .format = V4L2_PIX_FMT_RGB565X, .pixel_enc = V4L2_PIXEL_ENC_RGB, .mem_planes = 1, .comp_planes = 1, .bpp = { 2, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },
 		{ .format = V4L2_PIX_FMT_RGB555,  .pixel_enc = V4L2_PIXEL_ENC_RGB, .mem_planes = 1, .comp_planes = 1, .bpp = { 2, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },
 		{ .format = V4L2_PIX_FMT_BGR666,  .pixel_enc = V4L2_PIXEL_ENC_RGB, .mem_planes = 1, .comp_planes = 1, .bpp = { 4, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },
 		{ .format = V4L2_PIX_FMT_BGR48_12, .pixel_enc = V4L2_PIXEL_ENC_RGB, .mem_planes = 1, .comp_planes = 1, .bpp = { 6, 0, 0, 0 }, .bpp_div = { 1, 1, 1, 1 }, .hdiv = 1, .vdiv = 1 },
Signature · 1 line
2.25.1
#2 lore raw

PATCH 2/3 media: vim2m: Simplify try_fmt

In reply to Matthew Majewski

Clean up vidioc_try_fmt with the following changes:

1. remove unsused vim2m_fmt parameter
2. use clamp() macro to restrain width/height bounds
3. use ALIGN() macro to align width/height
4. use v4l2_fill_pixfmt to set bytesperline/sizeimage

---
 drivers/media/test-drivers/vim2m.c | 33 +++++++++++-------------------
 1 file changed, 12 insertions(+), 21 deletions(-)
drivers/media/test-drivers/vim2m.c +12 -21
index 6c24dcf27eb0..a22b61793a52 100644
@@ -26,6 +26,7 @@
 #include <media/v4l2-ctrls.h>
 #include <media/v4l2-event.h>
 #include <media/videobuf2-vmalloc.h>
+#include <media/v4l2-common.h>
 
 MODULE_DESCRIPTION("Virtual device for mem2mem framework testing");
 MODULE_AUTHOR("Pawel Osciak, <[email protected]>");
@@ -755,31 +756,21 @@ static int vidioc_g_fmt_vid_cap(struct file *file, void *priv,
 	return vidioc_g_fmt(file2ctx(file), f);
 }
 
-static int vidioc_try_fmt(struct v4l2_format *f, struct vim2m_fmt *fmt)
+static int vidioc_try_fmt(struct v4l2_format *f)
 {
-	int walign, halign;
-	/*
-	 * V4L2 specification specifies the driver corrects the
-	 * format struct if any of the dimensions is unsupported
-	 */
-	if (f->fmt.pix.height < MIN_H)
-		f->fmt.pix.height = MIN_H;
-	else if (f->fmt.pix.height > MAX_H)
-		f->fmt.pix.height = MAX_H;
+	int width, height, walign, halign;
 
-	if (f->fmt.pix.width < MIN_W)
-		f->fmt.pix.width = MIN_W;
-	else if (f->fmt.pix.width > MAX_W)
-		f->fmt.pix.width = MAX_W;
+	width = clamp(f->fmt.pix.width, MIN_W, MAX_W);
+	height = clamp(f->fmt.pix.width, MIN_H, MAX_H);
 
 	get_alignment(f->fmt.pix.pixelformat, &walign, &halign);
-	f->fmt.pix.width &= ~(walign - 1);
-	f->fmt.pix.height &= ~(halign - 1);
-	f->fmt.pix.bytesperline = (f->fmt.pix.width * fmt->depth) >> 3;
-	f->fmt.pix.sizeimage = f->fmt.pix.height * f->fmt.pix.bytesperline;
+	width = ALIGN(width, walign);
+	height = ALIGN(height, halign);
+
 	f->fmt.pix.field = V4L2_FIELD_NONE;
 
-	return 0;
+	return v4l2_fill_pixfmt(&f->fmt.pix, f->fmt.pix.pixelformat,
+			       width, height);
 }
 
 static int vidioc_try_fmt_vid_cap(struct file *file, void *priv,
@@ -804,7 +795,7 @@ static int vidioc_try_fmt_vid_cap(struct file *file, void *priv,
 	f->fmt.pix.ycbcr_enc = ctx->ycbcr_enc;
 	f->fmt.pix.quantization = ctx->quant;
 
-	return vidioc_try_fmt(f, fmt);
+	return vidioc_try_fmt(f);
 }
 
 static int vidioc_try_fmt_vid_out(struct file *file, void *priv,
@@ -827,7 +818,7 @@ static int vidioc_try_fmt_vid_out(struct file *file, void *priv,
 	if (!f->fmt.pix.colorspace)
 		f->fmt.pix.colorspace = V4L2_COLORSPACE_REC709;
 
-	return vidioc_try_fmt(f, fmt);
+	return vidioc_try_fmt(f);
 }
 
 static int vidioc_s_fmt(struct vim2m_ctx *ctx, struct v4l2_format *f)
Signature · 1 line
2.25.1
#3 lore raw

PATCH 3/3 media: vim2m: Add parametized support for multiplanar API

In reply to Matthew Majewski

Add support for the mulitiplaner API. The device can now act as either a multi-planar or a single-planar device depending on a module parameter, similar to the way vivid behaves.

Multiplanar support was added by implementing the appropate try/get/set mplane functions, and by modifying the queue_setup() and buf_prepare() functions to handle multiple planes. Implementation was inspired by vivid.

---
 drivers/media/test-drivers/vim2m.c | 306 +++++++++++++++++++++++++----
 1 file changed, 267 insertions(+), 39 deletions(-)
drivers/media/test-drivers/vim2m.c +267 -39
index a22b61793a52..91123ec4cd22 100644
@@ -43,6 +43,10 @@ static unsigned int default_transtime = 40; /* Max 25 fps */
 module_param(default_transtime, uint, 0644);
 MODULE_PARM_DESC(default_transtime, "default transaction time in ms");
 
+static unsigned int multiplanar = 1;
+module_param(multiplanar, uint, 0644);
+MODULE_PARM_DESC(multiplanar, "1 (default) creates a single planar device, 2 creates multiplanar device.");
+
 #define MIN_W 32
 #define MIN_H 32
 #define MAX_W 640
@@ -135,7 +139,8 @@ static struct vim2m_fmt formats[] = {
 struct vim2m_q_data {
 	unsigned int		width;
 	unsigned int		height;
-	unsigned int		sizeimage;
+	unsigned int            num_mem_planes;
+	unsigned int		sizeimage[VIDEO_MAX_PLANES];
 	unsigned int		sequence;
 	struct vim2m_fmt	*fmt;
 };
@@ -194,6 +199,7 @@ struct vim2m_dev {
 	struct mutex		dev_mutex;
 
 	struct v4l2_m2m_dev	*m2m_dev;
+	bool			multiplanar;
 };
 
 struct vim2m_ctx {
@@ -238,8 +244,10 @@ static struct vim2m_q_data *get_q_data(struct vim2m_ctx *ctx,
 {
 	switch (type) {
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
 		return &ctx->q_data[V4L2_M2M_SRC];
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
 		return &ctx->q_data[V4L2_M2M_DST];
 	default:
 		return NULL;
@@ -250,8 +258,10 @@ static const char *type_name(enum v4l2_buf_type type)
 {
 	switch (type) {
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
 		return "Output";
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
 		return "Capture";
 	default:
 		return "Invalid";
@@ -721,6 +731,7 @@ static int vidioc_g_fmt(struct vim2m_ctx *ctx, struct v4l2_format *f)
 {
 	struct vb2_queue *vq;
 	struct vim2m_q_data *q_data;
+	int ret;
 
 	vq = v4l2_m2m_get_vq(ctx->fh.m2m_ctx, f->type);
 	if (!vq)
@@ -730,12 +741,12 @@ static int vidioc_g_fmt(struct vim2m_ctx *ctx, struct v4l2_format *f)
 	if (!q_data)
 		return -EINVAL;
 
-	f->fmt.pix.width	= q_data->width;
-	f->fmt.pix.height	= q_data->height;
+	ret = v4l2_fill_pixfmt(&f->fmt.pix, q_data->fmt->fourcc,
+			       q_data->width, q_data->height);
+	if (ret)
+		return ret;
+
 	f->fmt.pix.field	= V4L2_FIELD_NONE;
-	f->fmt.pix.pixelformat	= q_data->fmt->fourcc;
-	f->fmt.pix.bytesperline	= (q_data->width * q_data->fmt->depth) >> 3;
-	f->fmt.pix.sizeimage	= q_data->sizeimage;
 	f->fmt.pix.colorspace	= ctx->colorspace;
 	f->fmt.pix.xfer_func	= ctx->xfer_func;
 	f->fmt.pix.ycbcr_enc	= ctx->ycbcr_enc;
@@ -744,33 +755,102 @@ static int vidioc_g_fmt(struct vim2m_ctx *ctx, struct v4l2_format *f)
 	return 0;
 }
 
+static int vidioc_g_fmt_mplane(struct vim2m_ctx *ctx, struct v4l2_format *f)
+{
+	struct vb2_queue *vq;
+	struct vim2m_q_data *q_data;
+	int ret;
+
+	vq = v4l2_m2m_get_vq(ctx->fh.m2m_ctx, f->type);
+	if (!vq)
+		return -EINVAL;
+
+	q_data = get_q_data(ctx, f->type);
+	if (!q_data)
+		return -EINVAL;
+
+	ret = v4l2_fill_pixfmt_mp(&f->fmt.pix_mp, q_data->fmt->fourcc,
+				  q_data->width, q_data->height);
+	if (ret)
+		return ret;
+
+	f->fmt.pix_mp.field	   = V4L2_FIELD_NONE;
+	f->fmt.pix_mp.colorspace   = ctx->colorspace;
+	f->fmt.pix_mp.xfer_func	   = ctx->xfer_func;
+	f->fmt.pix_mp.ycbcr_enc	   = ctx->ycbcr_enc;
+	f->fmt.pix_mp.quantization = ctx->quant;
+
+	return 0;
+}
+
 static int vidioc_g_fmt_vid_out(struct file *file, void *priv,
 				struct v4l2_format *f)
 {
+	struct vim2m_dev *dev = video_drvdata(file);
+
+	if (dev->multiplanar)
+		return -ENOTTY;
+
 	return vidioc_g_fmt(file2ctx(file), f);
 }
 
 static int vidioc_g_fmt_vid_cap(struct file *file, void *priv,
 				struct v4l2_format *f)
 {
+	struct vim2m_dev *dev = video_drvdata(file);
+
+	if (dev->multiplanar)
+		return -ENOTTY;
+
 	return vidioc_g_fmt(file2ctx(file), f);
 }
 
-static int vidioc_try_fmt(struct v4l2_format *f)
+static int vidioc_g_fmt_vid_out_mplane(struct file *file, void *priv,
+				       struct v4l2_format *f)
 {
-	int width, height, walign, halign;
+	struct vim2m_dev *dev = video_drvdata(file);
+
+	if (!dev->multiplanar)
+		return -ENOTTY;
+
+	return vidioc_g_fmt_mplane(file2ctx(file), f);
+}
+
+static int vidioc_g_fmt_vid_cap_mplane(struct file *file, void *priv,
+				       struct v4l2_format *f)
+{
+	struct vim2m_dev *dev = video_drvdata(file);
+
+	if (!dev->multiplanar)
+		return -ENOTTY;
 
-	width = clamp(f->fmt.pix.width, MIN_W, MAX_W);
-	height = clamp(f->fmt.pix.width, MIN_H, MAX_H);
+	return vidioc_g_fmt_mplane(file2ctx(file), f);
+}
+
+static int vidioc_try_fmt(struct v4l2_format *f, bool is_mplane)
+{
+	int walign, halign, ret;
+	int width = (is_mplane) ? f->fmt.pix_mp.width : f->fmt.pix.width;
+	int height = (is_mplane) ? f->fmt.pix_mp.height : f->fmt.pix.height;
+	u32 pixfmt = (is_mplane) ? f->fmt.pix_mp.pixelformat :
+		f->fmt.pix.pixelformat;
+
+	width = clamp(width, MIN_W, MAX_W);
+	height = clamp(height, MIN_H, MAX_H);
 
-	get_alignment(f->fmt.pix.pixelformat, &walign, &halign);
+	get_alignment(pixfmt, &walign, &halign);
 	width = ALIGN(width, walign);
 	height = ALIGN(height, halign);
 
 	f->fmt.pix.field = V4L2_FIELD_NONE;
 
-	return v4l2_fill_pixfmt(&f->fmt.pix, f->fmt.pix.pixelformat,
-			       width, height);
+	if (is_mplane) {
+		ret = v4l2_fill_pixfmt_mp(&f->fmt.pix_mp, pixfmt, width,
+					  height);
+	} else {
+		ret = v4l2_fill_pixfmt(&f->fmt.pix, pixfmt,  width, height);
+	}
+	return ret;
 }
 
 static int vidioc_try_fmt_vid_cap(struct file *file, void *priv,
@@ -778,6 +858,10 @@ static int vidioc_try_fmt_vid_cap(struct file *file, void *priv,
 {
 	struct vim2m_fmt *fmt;
 	struct vim2m_ctx *ctx = file2ctx(file);
+	struct vim2m_dev *dev = video_drvdata(file);
+
+	if (dev->multiplanar)
+		return -ENOTTY;
 
 	fmt = find_format(f->fmt.pix.pixelformat);
 	if (!fmt) {
@@ -795,7 +879,36 @@ static int vidioc_try_fmt_vid_cap(struct file *file, void *priv,
 	f->fmt.pix.ycbcr_enc = ctx->ycbcr_enc;
 	f->fmt.pix.quantization = ctx->quant;
 
-	return vidioc_try_fmt(f);
+	return vidioc_try_fmt(f, false);
+}
+
+static int vidioc_try_fmt_vid_cap_mplane(struct file *file, void *priv,
+					 struct v4l2_format *f)
+{
+	struct vim2m_fmt *fmt;
+	struct vim2m_ctx *ctx = file2ctx(file);
+	struct vim2m_dev *dev = video_drvdata(file);
+
+	if (!dev->multiplanar)
+		return -ENOTTY;
+
+	fmt = find_format(f->fmt.pix_mp.pixelformat);
+	if (!fmt) {
+		f->fmt.pix_mp.pixelformat = formats[0].fourcc;
+		fmt = find_format(f->fmt.pix_mp.pixelformat);
+	}
+	if (!(fmt->types & MEM2MEM_CAPTURE)) {
+		v4l2_err(&ctx->dev->v4l2_dev,
+			 "Fourcc format (0x%08x) invalid.\n",
+			 f->fmt.pix.pixelformat);
+		return -EINVAL;
+	}
+	f->fmt.pix_mp.colorspace = ctx->colorspace;
+	f->fmt.pix_mp.xfer_func = ctx->xfer_func;
+	f->fmt.pix_mp.ycbcr_enc = ctx->ycbcr_enc;
+	f->fmt.pix_mp.quantization = ctx->quant;
+
+	return vidioc_try_fmt(f, true);
 }
 
 static int vidioc_try_fmt_vid_out(struct file *file, void *priv,
@@ -803,6 +916,10 @@ static int vidioc_try_fmt_vid_out(struct file *file, void *priv,
 {
 	struct vim2m_fmt *fmt;
 	struct vim2m_ctx *ctx = file2ctx(file);
+	struct vim2m_dev *dev = video_drvdata(file);
+
+	if (dev->multiplanar)
+		return -ENOTTY;
 
 	fmt = find_format(f->fmt.pix.pixelformat);
 	if (!fmt) {
@@ -818,13 +935,45 @@ static int vidioc_try_fmt_vid_out(struct file *file, void *priv,
 	if (!f->fmt.pix.colorspace)
 		f->fmt.pix.colorspace = V4L2_COLORSPACE_REC709;
 
-	return vidioc_try_fmt(f);
+	return vidioc_try_fmt(f, false);
+}
+
+static int vidioc_try_fmt_vid_out_mplane(struct file *file, void *priv,
+					 struct v4l2_format *f)
+{
+	struct vim2m_fmt *fmt;
+	struct vim2m_ctx *ctx = file2ctx(file);
+	struct vim2m_dev *dev = video_drvdata(file);
+
+	if (!dev->multiplanar)
+		return -ENOTTY;
+
+	fmt = find_format(f->fmt.pix_mp.pixelformat);
+	if (!fmt) {
+		f->fmt.pix_mp.pixelformat = formats[0].fourcc;
+		fmt = find_format(f->fmt.pix_mp.pixelformat);
+	}
+	if (!(fmt->types & MEM2MEM_OUTPUT)) {
+		v4l2_err(&ctx->dev->v4l2_dev,
+			 "Fourcc format (0x%08x) invalid.\n",
+			 f->fmt.pix_mp.pixelformat);
+		return -EINVAL;
+	}
+	if (!f->fmt.pix_mp.colorspace)
+		f->fmt.pix_mp.colorspace = V4L2_COLORSPACE_REC709;
+
+	return vidioc_try_fmt(f, true);
 }
 
 static int vidioc_s_fmt(struct vim2m_ctx *ctx, struct v4l2_format *f)
 {
 	struct vim2m_q_data *q_data;
 	struct vb2_queue *vq;
+	unsigned int i;
+	bool is_mplane = ctx->dev->multiplanar;
+	u32 pixfmt = (is_mplane) ? f->fmt.pix_mp.pixelformat : f->fmt.pix.pixelformat;
+	u32 width = (is_mplane) ? f->fmt.pix_mp.width : f->fmt.pix.width;
+	u32 height = (is_mplane) ? f->fmt.pix_mp.height : f->fmt.pix.height;
 
 	vq = v4l2_m2m_get_vq(ctx->fh.m2m_ctx, f->type);
 	if (!vq)
@@ -839,11 +988,17 @@ static int vidioc_s_fmt(struct vim2m_ctx *ctx, struct v4l2_format *f)
 		return -EBUSY;
 	}
 
-	q_data->fmt		= find_format(f->fmt.pix.pixelformat);
-	q_data->width		= f->fmt.pix.width;
-	q_data->height		= f->fmt.pix.height;
-	q_data->sizeimage	= q_data->width * q_data->height
-				* q_data->fmt->depth >> 3;
+	q_data->fmt		= find_format(pixfmt);
+	q_data->width		= width;
+	q_data->height		= height;
+	if (is_mplane) {
+		q_data->num_mem_planes = f->fmt.pix_mp.num_planes;
+		for (i = 0; i < f->fmt.pix_mp.num_planes; i++)
+			q_data->sizeimage[i] = f->fmt.pix_mp.plane_fmt[i].sizeimage;
+	} else {
+		q_data->sizeimage[0] = f->fmt.pix.sizeimage;
+		q_data->num_mem_planes = 1;
+	}
 
 	dprintk(ctx->dev, 1,
 		"Format for type %s: %dx%d (%d bpp), fmt: %c%c%c%c\n",
@@ -861,6 +1016,10 @@ static int vidioc_s_fmt_vid_cap(struct file *file, void *priv,
 				struct v4l2_format *f)
 {
 	int ret;
+	struct vim2m_dev *dev = video_drvdata(file);
+
+	if (dev->multiplanar)
+		return -ENOTTY;
 
 	ret = vidioc_try_fmt_vid_cap(file, priv, f);
 	if (ret)
@@ -869,12 +1028,32 @@ static int vidioc_s_fmt_vid_cap(struct file *file, void *priv,
 	return vidioc_s_fmt(file2ctx(file), f);
 }
 
+static int vidioc_s_fmt_vid_cap_mplane(struct file *file, void *priv,
+				       struct v4l2_format *f)
+{
+	int ret;
+	struct vim2m_dev *dev = video_drvdata(file);
+
+	if (!dev->multiplanar)
+		return -ENOTTY;
+
+	ret = vidioc_try_fmt_vid_cap_mplane(file, priv, f);
+	if (ret)
+		return ret;
+
+	return vidioc_s_fmt(file2ctx(file), f);
+}
+
 static int vidioc_s_fmt_vid_out(struct file *file, void *priv,
 				struct v4l2_format *f)
 {
 	struct vim2m_ctx *ctx = file2ctx(file);
+	struct vim2m_dev *dev = video_drvdata(file);
 	int ret;
 
+	if (dev->multiplanar)
+		return -ENOTTY;
+
 	ret = vidioc_try_fmt_vid_out(file, priv, f);
 	if (ret)
 		return ret;
@@ -889,6 +1068,30 @@ static int vidioc_s_fmt_vid_out(struct file *file, void *priv,
 	return ret;
 }
 
+static int vidioc_s_fmt_vid_out_mplane(struct file *file, void *priv,
+				       struct v4l2_format *f)
+{
+	struct vim2m_ctx *ctx = file2ctx(file);
+	struct vim2m_dev *dev = video_drvdata(file);
+	int ret;
+
+	if (!dev->multiplanar)
+		return -ENOTTY;
+
+	ret = vidioc_try_fmt_vid_out_mplane(file, priv, f);
+	if (ret)
+		return ret;
+
+	ret = vidioc_s_fmt(file2ctx(file), f);
+	if (!ret) {
+		ctx->colorspace = f->fmt.pix_mp.colorspace;
+		ctx->xfer_func = f->fmt.pix_mp.xfer_func;
+		ctx->ycbcr_enc = f->fmt.pix_mp.ycbcr_enc;
+		ctx->quant = f->fmt.pix_mp.quantization;
+	}
+	return ret;
+}
+
 static int vim2m_s_ctrl(struct v4l2_ctrl *ctrl)
 {
 	struct vim2m_ctx *ctx =
@@ -939,11 +1142,17 @@ static const struct v4l2_ioctl_ops vim2m_ioctl_ops = {
 	.vidioc_g_fmt_vid_cap	= vidioc_g_fmt_vid_cap,
 	.vidioc_try_fmt_vid_cap	= vidioc_try_fmt_vid_cap,
 	.vidioc_s_fmt_vid_cap	= vidioc_s_fmt_vid_cap,
+	.vidioc_g_fmt_vid_cap_mplane = vidioc_g_fmt_vid_cap_mplane,
+	.vidioc_try_fmt_vid_cap_mplane = vidioc_try_fmt_vid_cap_mplane,
+	.vidioc_s_fmt_vid_cap_mplane	= vidioc_s_fmt_vid_cap_mplane,
 
 	.vidioc_enum_fmt_vid_out = vidioc_enum_fmt_vid_out,
 	.vidioc_g_fmt_vid_out	= vidioc_g_fmt_vid_out,
 	.vidioc_try_fmt_vid_out	= vidioc_try_fmt_vid_out,
 	.vidioc_s_fmt_vid_out	= vidioc_s_fmt_vid_out,
+	.vidioc_g_fmt_vid_out_mplane = vidioc_g_fmt_vid_out_mplane,
+	.vidioc_try_fmt_vid_out_mplane = vidioc_try_fmt_vid_out_mplane,
+	.vidioc_s_fmt_vid_out_mplane	= vidioc_s_fmt_vid_out_mplane,
 
 	.vidioc_reqbufs		= v4l2_m2m_ioctl_reqbufs,
 	.vidioc_querybuf	= v4l2_m2m_ioctl_querybuf,
@@ -972,23 +1181,32 @@ static int vim2m_queue_setup(struct vb2_queue *vq,
 {
 	struct vim2m_ctx *ctx = vb2_get_drv_priv(vq);
 	struct vim2m_q_data *q_data;
-	unsigned int size, count = *nbuffers;
+	unsigned int size, p, count = *nbuffers;
 
 	q_data = get_q_data(ctx, vq->type);
 	if (!q_data)
 		return -EINVAL;
 
-	size = q_data->width * q_data->height * q_data->fmt->depth >> 3;
+	size = 0;
+	for (p = 0; p < q_data->num_mem_planes; p++)
+		size += q_data->sizeimage[p];
 
 	while (size * count > MEM2MEM_VID_MEM_LIMIT)
 		(count)--;
 	*nbuffers = count;
 
-	if (*nplanes)
-		return sizes[0] < size ? -EINVAL : 0;
-
-	*nplanes = 1;
-	sizes[0] = size;
+	if (*nplanes) {
+		if (*nplanes != q_data->num_mem_planes)
+			return -EINVAL;
+		for (p = 0; p < q_data->num_mem_planes; p++) {
+			if (sizes[p] < q_data->sizeimage[p])
+				return -EINVAL;
+		}
+	} else {
+		*nplanes = q_data->num_mem_planes;
+		for (p = 0; p < q_data->num_mem_planes; p++)
+			sizes[p] = q_data->sizeimage[p];
+	}
 
 	dprintk(ctx->dev, 1, "%s: get %d buffer(s) of size %d each.\n",
 		type_name(vq->type), count, size);
@@ -1015,21 +1233,24 @@ static int vim2m_buf_prepare(struct vb2_buffer *vb)
 {
 	struct vim2m_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue);
 	struct vim2m_q_data *q_data;
+	unsigned int p;
 
 	dprintk(ctx->dev, 2, "type: %s\n", type_name(vb->vb2_queue->type));
 
 	q_data = get_q_data(ctx, vb->vb2_queue->type);
 	if (!q_data)
 		return -EINVAL;
-	if (vb2_plane_size(vb, 0) < q_data->sizeimage) {
-		dprintk(ctx->dev, 1,
-			"%s data will not fit into plane (%lu < %lu)\n",
-			__func__, vb2_plane_size(vb, 0),
-			(long)q_data->sizeimage);
-		return -EINVAL;
-	}
 
-	vb2_set_plane_payload(vb, 0, q_data->sizeimage);
+	for (p = 0; p < q_data->num_mem_planes; p++) {
+		if (vb2_plane_size(vb, p) < q_data->sizeimage[p]) {
+			dprintk(ctx->dev, 1,
+				"%s data will not fit into plane (%lu < %lu)\n",
+				__func__, vb2_plane_size(vb, p),
+				(long)q_data->sizeimage[p]);
+			return -EINVAL;
+		}
+		vb2_set_plane_payload(vb, p, q_data->sizeimage[p]);
+	}
 
 	return 0;
 }
@@ -1100,7 +1321,8 @@ static int queue_init(void *priv, struct vb2_queue *src_vq,
 	struct vim2m_ctx *ctx = priv;
 	int ret;
 
-	src_vq->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
+	src_vq->type = (ctx->dev->multiplanar) ? V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE :
+		V4L2_BUF_TYPE_VIDEO_OUTPUT;
 	src_vq->io_modes = VB2_MMAP | VB2_USERPTR | VB2_DMABUF;
 	src_vq->drv_priv = ctx;
 	src_vq->buf_struct_size = sizeof(struct v4l2_m2m_buffer);
@@ -1114,7 +1336,8 @@ static int queue_init(void *priv, struct vb2_queue *src_vq,
 	if (ret)
 		return ret;
 
-	dst_vq->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+	dst_vq->type = (ctx->dev->multiplanar) ? V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE :
+		V4L2_BUF_TYPE_VIDEO_CAPTURE;
 	dst_vq->io_modes = VB2_MMAP | VB2_USERPTR | VB2_DMABUF;
 	dst_vq->drv_priv = ctx;
 	dst_vq->buf_struct_size = sizeof(struct v4l2_m2m_buffer);
@@ -1188,10 +1411,11 @@ static int vim2m_open(struct file *file)
 	ctx->q_data[V4L2_M2M_SRC].fmt = &formats[0];
 	ctx->q_data[V4L2_M2M_SRC].width = 640;
 	ctx->q_data[V4L2_M2M_SRC].height = 480;
-	ctx->q_data[V4L2_M2M_SRC].sizeimage =
+	ctx->q_data[V4L2_M2M_SRC].sizeimage[0] =
 		ctx->q_data[V4L2_M2M_SRC].width *
 		ctx->q_data[V4L2_M2M_SRC].height *
 		(ctx->q_data[V4L2_M2M_SRC].fmt->depth >> 3);
+	ctx->q_data[V4L2_M2M_SRC].num_mem_planes = 1;
 	ctx->q_data[V4L2_M2M_DST] = ctx->q_data[V4L2_M2M_SRC];
 	ctx->colorspace = V4L2_COLORSPACE_REC709;
 
@@ -1268,7 +1492,7 @@ static const struct video_device vim2m_videodev = {
 	.ioctl_ops	= &vim2m_ioctl_ops,
 	.minor		= -1,
 	.release	= vim2m_device_release,
-	.device_caps	= V4L2_CAP_VIDEO_M2M | V4L2_CAP_STREAMING,
+	.device_caps	= V4L2_CAP_STREAMING,
 };
 
 static const struct v4l2_m2m_ops m2m_ops = {
@@ -1299,10 +1523,14 @@ static int vim2m_probe(struct platform_device *pdev)
 	atomic_set(&dev->num_inst, 0);
 	mutex_init(&dev->dev_mutex);
 
+	dev->multiplanar = (multiplanar == 2);
+
 	dev->vfd = vim2m_videodev;
 	vfd = &dev->vfd;
 	vfd->lock = &dev->dev_mutex;
 	vfd->v4l2_dev = &dev->v4l2_dev;
+	vfd->device_caps |= (dev->multiplanar) ? V4L2_CAP_VIDEO_M2M_MPLANE :
+		V4L2_CAP_VIDEO_M2M;
 
 	video_set_drvdata(vfd, dev);
 	v4l2_info(&dev->v4l2_dev,
Signature · 1 line
2.25.1
#4 lore raw
In reply to Matthew Majewski

Hi Matthew,

Quoted 5 lines · Matthew Majewski

Hi everyone,

This patch series adds multiplane API support for the virtual M2M driver, along with some minor driver refactoring/improvements.

Quoted 7 lines

I followed the lead of the vivid driver and made multiplanar support selectable with a module parameter, and the default is to use the single planar api.

Although there are not yet any pixelformats in the driver that make use of multiple memory planes, adding support for these should be easier now with the API level changes taken care of.

Are you planning follow-up patches adding support for at least one multiplanar format? That would be really nice.

Regards,

Hans

Quoted 61 lines

v4l2-compliance reports the following with multiplane support disabled:

Total for vim2m device /dev/video0: 48, Succeeded: 48, Failed: 0, Warnings: 0

and the same with multiplane support enabled:

Total for vim2m device /dev/video0: 48, Succeeded: 48, Failed: 0, Warnings: 0

Patches need to be applied in increasing numerical order (Patch [3/3] depends on [1/3] and [2/3]).

Since the multi-plane changes had to touch a lot of the driver, I did a basic regression test with the following script which generates a test input image with vivid and an output image from vim2m for each supported format. I confirmed all outputs visually and verified they were identical to the outputs before the change. Testing was done on an x86_64 qemu image.

#!/bin/sh

# tested with HDMI vivid emulation
# modprobe vivid num_inputs=1 input_types=3

vim2m=/dev/video0
vivid=/dev/video1

width=640
height=480
out_width=320
out_height=240

capture_formats=$(v4l2-ctl -d $vim2m --list-formats | awk '/\]:/ {print $2}' | sed "s/'//g") output_formats=$(v4l2-ctl -d $vim2m --list-formats-out | awk '/\]:/ {print $2}' | sed "s/'//g")

# Turn off text mode so that images will be identical v4l2-ctl -d $vivid -c osd_text_mode=2

for ofmt in ${output_formats}; do

    # generate input image
    inname="${width}x${height}.${ofmt}"
    v4l2-ctl -d $vivid -v pixelformat=$ofmt,width=$width,height=$height,field=none \
             --stream-mmap --stream-count=1 --stream-to=$inname
    for cfmt in ${capture_formats}; do
        outname="${out_width}x${out_height}-out.${cfmt}"
        v4l2-ctl -d $vim2m -x pixelformat=$ofmt,width=$width,height=$height \
                 -v pixelformat=$cfmt,width=$out_width,height=$out_height \
                 --stream-from=$inname --stream-to=$outname --stream-mmap --stream-out-mmap \
                 --stream-count=1
    done
done

Matthew Majewski (3):
media: v4l2-common: Add RGBR format info
media: vim2m: Simplify try_fmt
media: vim2m: Add parametized support for multiplanar API

 drivers/media/test-drivers/vim2m.c    | 327 +++++++++++++++++++++-----
 drivers/media/v4l2-core/v4l2-common.c |   1 +
 2 files changed, 274 insertions(+), 54 deletions(-)
#5 lore raw

Hello Hans,

Quoted 17 lines · Hans Verkuil

Hi Matthew,

Quoted 5 lines · Matthew Majewski

Hi everyone,

This patch series adds multiplane API support for the virtual M2M driver, along with some minor driver refactoring/improvements.

Quoted 8 lines

I followed the lead of the vivid driver and made multiplanar support
selectable with a module parameter, and the default is to use the single planar api.

Although there are not yet any pixelformats in the driver that make use of multiple memory planes, adding support for these should be easier now with the API level changes taken care of.

Quoted 7 lines

Are you planning follow-up patches adding support for at least one multiplanar format? That would be really nice.

Regards,

Hans

Yes, I currently have a WIP patch series to add support for all of the formats that vivid currently supports to vim2m. It involves a huge refactor though, so I split the groundwork for supporting the multi- planar formats into a separate patch series first.

Best,

Matthew

Quoted 76 lines
Quoted 75 lines

v4l2-compliance reports the following with multiplane support disabled:

  Total for vim2m device /dev/video0: 48, Succeeded: 48, Failed: 0,
Warnings: 0

and the same with multiplane support enabled:

  Total for vim2m device /dev/video0: 48, Succeeded: 48, Failed: 0,
Warnings: 0

Patches need to be applied in increasing numerical order (Patch [3/3]
depends on [1/3] and [2/3]).

Since the multi-plane changes had to touch a lot of the driver, I did
a basic regression test with the following script which generates a test input image with vivid and an output image from vim2m for each supported format. I confirmed all outputs visually and verified they
were identical to the outputs before the change. Testing was done on
an x86_64 qemu image.

#!/bin/sh

# tested with HDMI vivid emulation
# modprobe vivid num_inputs=1 input_types=3

vim2m=/dev/video0
vivid=/dev/video1

width=640
height=480
out_width=320
out_height=240

capture_formats=$(v4l2-ctl -d $vim2m --list-formats | awk '/\]:/ {print $2}' | sed "s/'//g")
output_formats=$(v4l2-ctl -d $vim2m --list-formats-out | awk '/\]:/ {print $2}' | sed "s/'//g")

# Turn off text mode so that images will be identical v4l2-ctl -d $vivid -c osd_text_mode=2

for ofmt in ${output_formats}; do

    # generate input image
    inname="${width}x${height}.${ofmt}"
    v4l2-ctl -d $vivid -v
pixelformat=$ofmt,width=$width,height=$height,field=none \
             --stream-mmap --stream-count=1 --stream-to=$inname
    for cfmt in ${capture_formats}; do
        outname="${out_width}x${out_height}-out.${cfmt}"
        v4l2-ctl -d $vim2m -x
pixelformat=$ofmt,width=$width,height=$height \
                 -v
pixelformat=$cfmt,width=$out_width,height=$out_height \
                 --stream-from=$inname --stream-to=$outname --
stream-mmap --stream-out-mmap \
                 --stream-count=1
    done
done

Matthew Majewski (3):
  media: v4l2-common: Add RGBR format info
  media: vim2m: Simplify try_fmt
  media: vim2m: Add parametized support for multiplanar API

 drivers/media/test-drivers/vim2m.c    | 327 +++++++++++++++++++++-
----
 drivers/media/v4l2-core/v4l2-common.c |   1 +

 2 files changed, 274 insertions(+), 54 deletions(-)
#6 lore raw
Quoted 25 lines · Matthew Majewski

Hello Hans,

Quoted 22 lines · Hans Verkuil

Hi Matthew,

Quoted 5 lines · Matthew Majewski

Hi everyone,

This patch series adds multiplane API support for the virtual M2M driver, along with some minor driver refactoring/improvements.

Quoted 8 lines

I followed the lead of the vivid driver and made multiplanar support
selectable with a module parameter, and the default is to use the single planar api.

Although there are not yet any pixelformats in the driver that make use of multiple memory planes, adding support for these should be easier now with the API level changes taken care of.

Are you planning follow-up patches adding support for at least one multiplanar format? That would be really nice.

Regards,

Quoted 7 lines
Quoted 2 lines

Hans

Yes, I currently have a WIP patch series to add support for all of the formats that vivid currently supports to vim2m. It involves a huge refactor though, so I split the groundwork for supporting the multi- planar formats into a separate patch series first.

Ah, great news! Looking forward to that.

FYI: I've added vim2m multiplanar tests to contrib/test/test-media in v4l-utils. That script is part of our regression testing, so this helps testing multiplanar support.

Regards,

Hans

Quoted 82 lines

Best,

Matthew

Quoted 76 lines
Quoted 75 lines

v4l2-compliance reports the following with multiplane support disabled:

  Total for vim2m device /dev/video0: 48, Succeeded: 48, Failed: 0,
Warnings: 0

and the same with multiplane support enabled:

  Total for vim2m device /dev/video0: 48, Succeeded: 48, Failed: 0,
Warnings: 0

Patches need to be applied in increasing numerical order (Patch [3/3]
depends on [1/3] and [2/3]).

Since the multi-plane changes had to touch a lot of the driver, I did
a basic regression test with the following script which generates a test input image with vivid and an output image from vim2m for each supported format. I confirmed all outputs visually and verified they
were identical to the outputs before the change. Testing was done on
an x86_64 qemu image.

#!/bin/sh

# tested with HDMI vivid emulation
# modprobe vivid num_inputs=1 input_types=3

vim2m=/dev/video0
vivid=/dev/video1

width=640
height=480
out_width=320
out_height=240

capture_formats=$(v4l2-ctl -d $vim2m --list-formats | awk '/\]:/ {print $2}' | sed "s/'//g")
output_formats=$(v4l2-ctl -d $vim2m --list-formats-out | awk '/\]:/ {print $2}' | sed "s/'//g")

# Turn off text mode so that images will be identical v4l2-ctl -d $vivid -c osd_text_mode=2

for ofmt in ${output_formats}; do

    # generate input image
    inname="${width}x${height}.${ofmt}"
    v4l2-ctl -d $vivid -v
pixelformat=$ofmt,width=$width,height=$height,field=none \
             --stream-mmap --stream-count=1 --stream-to=$inname
    for cfmt in ${capture_formats}; do
        outname="${out_width}x${out_height}-out.${cfmt}"
        v4l2-ctl -d $vim2m -x
pixelformat=$ofmt,width=$width,height=$height \
                 -v
pixelformat=$cfmt,width=$out_width,height=$out_height \
                 --stream-from=$inname --stream-to=$outname --
stream-mmap --stream-out-mmap \
                 --stream-count=1
    done
done

Matthew Majewski (3):
  media: v4l2-common: Add RGBR format info
  media: vim2m: Simplify try_fmt
  media: vim2m: Add parametized support for multiplanar API

 drivers/media/test-drivers/vim2m.c    | 327 +++++++++++++++++++++-
----
 drivers/media/v4l2-core/v4l2-common.c |   1 +

 2 files changed, 274 insertions(+), 54 deletions(-)