Skip to contentSkip to conversation

media: vcodec: Fix potential array out-of-bounds in encoder queue_setup

PATCH In review Wei Chen 6 participants 6 messages Aug 10, 2023 – Aug 16, 2023

Subsystems (4)
lore
Share & tools
lore

Wei Chen

author
#0 lore raw

PATCH v2 1/1 media: vcodec: Fix potential array out-of-bounds in encoder queue_setup

variable *nplanes is provided by user via system call argument. The possible value of q_data->fmt->num_planes is 1-3, while the value of *nplanes can be 1-8. The array access by index i can cause array out-of-bounds.

Fix this bug by checking *nplanes against the array size.

Fixes 4e855a6efa54 ("[media] vcodec: mediatek: Add Mediatek V4L2 Video Encoder Driver")

---
Changes in v2:
- Add Fixes tag and CC stable email address
- Change the title to be more expressive

 drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c +2 -0
index 9ff439a50f53..9e8817863cb8 100644
@@ -821,6 +821,8 @@ static int vb2ops_venc_queue_setup(struct vb2_queue *vq,
 		return -EINVAL;
 
 	if (*nplanes) {
+		if (*nplanes != q_data->fmt->num_planes)
+			return -EINVAL;
 		for (i = 0; i < *nplanes; i++)
 			if (sizes[i] < q_data->sizeimage[i])
 				return -EINVAL;
Signature · 1 line
2.25.1
Reviewed-by on v2 1/1
#1 lore raw
Quoted 11 lines · Wei Chen

variable *nplanes is provided by user via system call argument. The possible value of q_data->fmt->num_planes is 1-3, while the value of *nplanes can be 1-8. The array access by index i can cause array out-of-bounds.

Fix this bug by checking *nplanes against the array size.

Fixes 4e855a6efa54 ("[media] vcodec: mediatek: Add Mediatek V4L2 Video Encoder Driver")
#2 lore raw
In reply to Wei Chen

Hi Wei,

Quoted 5 lines · Wei Chen

variable *nplanes is provided by user via system call argument. The possible value of q_data->fmt->num_planes is 1-3, while the value of *nplanes can be 1-8. The array access by index i can cause array out-of-bounds.

Quoted 8 lines

Fix this bug by checking *nplanes against the array size.

Fixes 4e855a6efa54 ("[media] vcodec: mediatek: Add Mediatek V4L2 Video Encoder Driver")

---
Changes in v2:
- Add Fixes tag and CC stable email address

I guess stable needs to be added by the maintainer, not by the submitter

Quoted 1 line

- Change the title to be more expressive

Subject message should include mediatek I believe, as media: vcodec: does not reference the mediatek vcodec.

Quoted 9 lines
  drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c | 2 ++
  1 file changed, 2 insertions(+)
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c +0 -0
index 9ff439a50f53..9e8817863cb8 100644

drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c static int vb2ops_venc_queue_setup(struct vb2_queue *vq, · line 825

Quoted 5 lines
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c +2 -0
@@ -821,6 +821,8 @@ static int vb2ops_venc_queue_setup(struct vb2_queue *vq,
  		return -EINVAL;
 
  	if (*nplanes) {
+		if (*nplanes != q_data->fmt->num_planes)
+			return -EINVAL;

Have you run v4l2-compliance on the device to make sure nothing is broken ?

drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c static int vb2ops_venc_queue_setup(struct vb2_queue *vq, · line 828

Quoted 3 lines
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c +0 -0
  		for (i = 0; i < *nplanes; i++)
  			if (sizes[i] < q_data->sizeimage[i])
  				return -EINVAL;

Greetings,

maintainer
#3 lore raw
Quoted 10 lines · Eugen Hristev

Hi Wei,

Quoted 7 lines · Wei Chen

variable *nplanes is provided by user via system call argument. The possible value of q_data->fmt->num_planes is 1-3, while the value of *nplanes can be 1-8. The array access by index i can cause array out-of-bounds.

Fix this bug by checking *nplanes against the array size.

Quoted 7 lines
Quoted 6 lines
Fixes 4e855a6efa54 ("[media] vcodec: mediatek: Add Mediatek V4L2 Video Encoder Driver")

---
Changes in v2:
- Add Fixes tag and CC stable email address

I guess stable needs to be added by the maintainer, not by the submitter

It's easiest if it is added by the submitter, makes the maintainer's job much easier.

thanks,

greg k-h

#4 lore raw
In reply to Wei Chen

Hi,

Le jeudi 10 août 2023 à 08:23 +0000, Wei Chen a écrit :

Quoted 23 lines

variable *nplanes is provided by user via system call argument. The possible value of q_data->fmt->num_planes is 1-3, while the value of *nplanes can be 1-8. The array access by index i can cause array out-of-bounds.

Fix this bug by checking *nplanes against the array size.

Fixes 4e855a6efa54 ("[media] vcodec: mediatek: Add Mediatek V4L2 Video Encoder Driver")

---
Changes in v2:
- Add Fixes tag and CC stable email address
- Change the title to be more expressive

 drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c +0 -0
index 9ff439a50f53..9e8817863cb8 100644

drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c static int vb2ops_venc_queue_setup(struct vb2_queue *vq, · line 825

Quoted 5 lines
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c +2 -0
@@ -821,6 +821,8 @@ static int vb2ops_venc_queue_setup(struct vb2_queue *vq,
 		return -EINVAL;
 
 	if (*nplanes) {
+		if (*nplanes != q_data->fmt->num_planes)
+			return -EINVAL;

I don't think the claim really exists. For this driver, when *nplane is set,
it will be:

        case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
                requested_planes = f->fmt.pix_mp.num_planes;
                if (requested_planes == 0 ||
                    requested_planes > VIDEO_MAX_PLANES)
                        return -EINVAL;
                for (i = 0; i < requested_planes; i++)
                        requested_sizes[i] =
                                f->fmt.pix_mp.plane_fmt[i].sizeimage;
                break;

Or the value the driver have set it in the previous call with *nplane == 0. So unless there is a bug, this should not happen, and more importantly, the core should not let that happen, meaning it should not be driver jobs to validate this.

my 2 cents,
Nicolas

Quoted 3 lines
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c +0 -0
 		for (i = 0; i < *nplanes; i++)
 			if (sizes[i] < q_data->sizeimage[i])
 				return -EINVAL;
#5 lore raw
Quoted 46 lines · Nicolas Dufresne

Hi,

Le jeudi 10 août 2023 à 08:23 +0000, Wei Chen a écrit :

Quoted 23 lines

variable *nplanes is provided by user via system call argument. The possible value of q_data->fmt->num_planes is 1-3, while the value of *nplanes can be 1-8. The array access by index i can cause array out-of-bounds.

Fix this bug by checking *nplanes against the array size.

Fixes 4e855a6efa54 ("[media] vcodec: mediatek: Add Mediatek V4L2 Video Encoder Driver")

---
Changes in v2:
- Add Fixes tag and CC stable email address
- Change the title to be more expressive

 drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c +0 -0
index 9ff439a50f53..9e8817863cb8 100644

drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c static int vb2ops_venc_queue_setup(struct vb2_queue *vq, · line 825

Quoted 5 lines
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c +2 -0
@@ -821,6 +821,8 @@ static int vb2ops_venc_queue_setup(struct vb2_queue *vq,
 		return -EINVAL;
 
 	if (*nplanes) {
+		if (*nplanes != q_data->fmt->num_planes)
+			return -EINVAL;

I don't think the claim really exists. For this driver, when *nplane is set,
it will be:

        case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
                requested_planes = f->fmt.pix_mp.num_planes;
                if (requested_planes == 0 ||
                    requested_planes > VIDEO_MAX_PLANES)
                        return -EINVAL;
                for (i = 0; i < requested_planes; i++)
                        requested_sizes[i] =
                                f->fmt.pix_mp.plane_fmt[i].sizeimage;
                break;
Quoted 11 lines

Or the value the driver have set it in the previous call with *nplane == 0. So unless there is a bug, this should not happen, and more importantly, the core should not let that happen, meaning it should not be driver jobs to validate this.

my 2 cents,
Nicolas

Quoted 2 lines
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c +0 -0
 		for (i = 0; i < *nplanes; i++)
 			if (sizes[i] < q_data->sizeimage[i])

It's q_data->sizeimage that has only 3 elements, so if *nplanes is 4, then this will fail with an OOB.

However, I think the check should really happen in the vb2 core. If no buffers have been allocated yet, then the queue_setup callback will set the number of planes based on the current format.

If CREATE_BUFS is called afterwards, then *nplanes is set to the number of planes that is specified in the format field of struct v4l2_create_buffers.

The core clips that value to the range [1..VIDEO_MAX_PLANES], but otherwise it places no restrictions.

I was afraid that this would cause serious problems if fewer planes are requested than the current format needs, but in that case the size checks in the buf_prepare callback of the driver will fail (since one or more planes will have size 0).

The idea behind allowing userspace to allocate different number of planes in CREATE_BUFS really makes little sense and there are no drivers that rely on this. It would be much better if the vb2 core would return -EINVAL if the requested number of planes does not match that of what is used for already allocated buffers.

If we ever get drivers that for some reason want more flexibility, then those drivers can set a special flag indicating that they want to check this themselves.

I'll post a patch adding the check to the vb2 core.

Regards,

Hans

Quoted 2 lines
Quoted 1 line
drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc.c +0 -0
 				return -EINVAL;