media: Two oopses and a hang when unbinding a streaming sensor
Subsystems (2)
v2 cover media: Two oopses and a hang when unbinding a streaming sensor
This is the second version of a series first posted on 12 August:
https://lore.kernel.org/linux-media/[email protected]/
What changed:
- Rebased and re-verified on v7.3-rc2. No code changes: none of the three files has changed in mainline since the first posting, so all three patches apply unchanged.
- Cover letter: the paragraph on patch 1 named ipu6_isys_csi2_get_remote_desc(), which does not exist. The functions are ipu6_isys_csi2_enable_streams() and ipu6_isys_csi2_disable_streams(), as the patch itself says.
- Patch 3 now carries a note on why the check and the marking are not
done under q->lock, and on what the proper fix would look like.
- Dropped [email protected] and [email protected] from Cc:
both bounced with 550 #5.1.0 on the last message, and neither is in the MAINTAINERS entry for this driver any more.
There have been no review comments so far. I am resending because the series has been sitting for a month and because patch 1 now overlaps with the IPU7 work; there is a question about that at the end.
Unbinding a camera sensor while a capture is running is a scenario that nothing in the IPU6 path handles: the kernel oopses twice, corrupts memory once, and leaves the application blocked forever. None of this is caused by the sensor drivers themselves, and all four failures are present in mainline today.
They were found while testing two new sensor drivers on a CHUWI Hi10 X1 (Intel N100, Alder Lake-N, IPU6) on a kernel built with KASAN, UBSAN, KMEMLEAK, PROVE_LOCKING and DETECT_HUNG_TASK. Three of them are fixed here; the fourth, a use-after-free in the media controller, is sent separately because it belongs to a different subsystem.
Patch 1 is a NULL pointer dereference in ipu6_isys_csi2_enable_streams() and ipu6_isys_csi2_disable_streams(). The remote pad is dereferenced without being checked, and unbinding the sensor mid-stream makes it NULL. Present since May 2024.
Patch 2 is a second NULL pointer dereference, in subdev_open(). v4l2_device_unregister_subdev() clears sd->v4l2_dev before the device node goes away, so anything opening /dev/v4l-subdevN in that window oopses. This one was not provoked deliberately: udev's v4l_id walked into it on its own. The window has been open since 2011.
Patch 3 is the hang. isys_async_ops has no .unbind() callback, so nothing tells the video nodes that the sensor is gone, and a DQBUF already waiting in vb2_core_dqbuf() never returns. The sleep is interruptible, so DETECT_HUNG_TASK stays quiet and the process is simply stuck until it is killed. Reproduced 10 times out of 10 on both sensors of the machine; with the patch, all 10 return -EIO and exit.
How each one was verified, since the three differ:
- patch 1: the oops was provoked deliberately on the unpatched kernel before the fix was written
- patch 2: reproduced itself, unprompted, with udev alone; after the fix, 150 cycles of a reproducer with four concurrent openers left no oops and no leaked minors
- patch 3: rebuilt both ways, same kernel and same test. Without it, 3 attempts out of 3 hang; with it, 10 out of 10 wake up and return -EIO
The full test cycle with the three fixes in place is clean: no KASAN or UBSAN reports, no KMEMLEAK findings, and lockdep still enabled at the end of the run. That last detail matters: lockdep disables itself on its first complaint and silently invalidates everything measured afterwards.
All of the above was measured on the first posting. The machine no longer has a kernel tree on it, so this one has not been rebuilt; it is the same code, and the rebase is a no-op verified with git apply.
The reproducer is a shell script that streams with v4l2-ctl, unbinds the sensor after two seconds, then rebinds it, in a loop. I am happy to post it if that would be useful.
One open question, on patch 1. The IPU7 series reworks both functions it
touches: "media: ipu6: Split ipu6 csi2 stream enable/disable" is in the ipu6 branch of the media tree and in [PATCH v4 00/45]. The rework is a refactor and carries the bug along -- remote_pad is still dereferenced unchecked in both functions -- so the fix is still needed there, in a different shape. Patch 1 as posted here applies to mainline and not to that branch; a version rebased on the branch is in the v1 thread:
https://lore.kernel.org/linux-media/[email protected]/
Which base would you prefer? I am happy to resend against either, or to split the difference: patches 2 and 3 are unaffected and apply to both.
Nicola Fiorillo (3):
media: ipu6: Check the remote pad before dereferencing it
media: v4l2-subdev: Check v4l2_dev before dereferencing it in open()
media: ipu6: Signal the video queues when a sensor is unbound
drivers/media/pci/intel/ipu6/ipu6-isys-csi2.c | 14 ++++++- drivers/media/pci/intel/ipu6/ipu6-isys.c | 41 +++++++++++++++++++ drivers/media/v4l2-core/v4l2-subdev.c | 31 +++++++++++--- 3 files changed, 78 insertions(+), 8 deletions(-)
base-commit: df2908090cda368b01ff43709f51890076c56157
Signature · 1 line
2.47.3
PATCH v2 1/3 media: ipu6: Check the remote pad before dereferencing it
Unbinding a sensor driver while a capture is running oopses the kernel:
BUG: kernel NULL pointer dereference, address: 0000000000000020 RIP: 0010:ipu6_isys_csi2_disable_streams+0x3c/0x70 [intel_ipu6_isys] Call Trace: v4l2_subdev_disable_streams+0x1b7/0x370 [videodev] ipu6_isys_video_set_streaming+0x20f/0x930 [intel_ipu6_isys] stop_streaming+0x102/0x110 [intel_ipu6_isys] __vb2_queue_cancel+0x2a/0x2d0 [videobuf2_common] vb2_core_queue_release+0x22/0x80 [videobuf2_common] _vb2_fop_release+0x58/0xb0 [videobuf2_v4l2] v4l2_release+0xbd/0xd0 [videodev] __fput+0xde/0x2a0
media_pad_remote_pad_first() returns NULL once the sensor is gone and the link with it, but both the enable and the disable path dereference the result unconditionally. The faulting address is the offset of the entity member in struct media_pad.
Check it. On enable there is nothing to stream from, so refuse with -ENOLINK. On disable the receiver still has to be stopped, so stop it and skip only the call towards the sensor that is no longer there.
Reproduced on a CHUWI Hi10 X1 (Alder Lake-N, IPU6) running 6.12.86, with the CSI-2 port of a sensor being unbound mid capture. The code is unchanged in v7.3-rc2.
---
Unchanged since v1.
This one collides with the IPU7 work; see the cover letter. A version of it rebased on the ipu6 branch of the media tree was posted in the v1 thread:
https://lore.kernel.org/linux-media/[email protected]/
drivers/media/pci/intel/ipu6/ipu6-isys-csi2.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/media/pci/intel/ipu6/ipu6-isys-csi2.c +12 -2
@@ -356,6 +356,9 @@ static int ipu6_isys_csi2_enable_streams(struct v4l2_subdev *sd,int ret; remote_pad =media_pad_remote_pad_first (&sd->entity.pads[CSI2_PAD_SINK ]); +if (!remote_pad) +return -ENOLINK ; + remote_sd =media_entity_to_v4l2_subdev (remote_pad->entity); sink_streams =
@@ -392,10 +395,17 @@ static int ipu6_isys_csi2_disable_streams(struct v4l2_subdev *sd,v4l2_subdev_state_xlate_streams (state, pad,CSI2_PAD_SINK , &streams_mask); +ipu6_isys_csi2_set_stream (sd,NULL ,0 , false); + +/* +* The link is gone if the sensor driver was unbound while streaming. +* Stop the receiver anyway, there is just no one left to tell. +*/ remote_pad =media_pad_remote_pad_first (&sd->entity.pads[CSI2_PAD_SINK ]); - remote_sd =media_entity_to_v4l2_subdev (remote_pad->entity); +if (!remote_pad) +return 0 ; -ipu6_isys_csi2_set_stream (sd,NULL ,0 , false); + remote_sd =media_entity_to_v4l2_subdev (remote_pad->entity);v4l2_subdev_disable_streams (remote_sd, remote_pad->index, sink_streams);
Signature · 1 line
2.47.3
PATCH v2 2/3 media: v4l2-subdev: Check v4l2_dev before dereferencing it in open()
In reply to Nicola FiorilloUnbinding a sensor driver while something opens its /dev/v4l-subdevN node oopses the kernel:
BUG: kernel NULL pointer dereference, address: 0000000000000008 RIP: 0010:subdev_open+0x8a/0x190 [videodev] Call Trace: v4l2_open+0xa9/0x100 [videodev] chrdev_open+0xb2/0x230 do_dentry_open+0x14c/0x440 vfs_open+0x2e/0xe0 path_openat+0x82e/0x12d0 do_filp_open+0xc4/0x170 do_sys_openat2+0xae/0xe0 __x64_sys_openat+0x55/0xa0
v4l2_device_unregister_subdev() clears sd->v4l2_dev, then unregisters the media entity, and only then unregisters the device node. Until the node is gone userspace can still open it, and subdev_open() dereferences sd->v4l2_dev unconditionally. The faulting address is the offset of the mdev member in struct v4l2_device.
The window is not a narrow one: media_device_unregister_entity() sleeps, and the first oops seen here was not provoked at all, it was hit by v4l_id, run by udev on the very node that was appearing and disappearing.
The same window leaves sd->entity.graph_obj.mdev NULL while sd->v4l2_dev->mdev is not, and the second dereference on that line goes through it. That one was found by reading the teardown path, not by crashing on it; it arrived later, with commit 218bf10e39ed ("media:
v4l2-subdev: handle module refcounting here").
Unregistering the device node before clearing the pointers would narrow the window but not close it, because v4l2_open() drops videodev_lock before it calls fops->open() and the whole of v4l2_device_unregister_subdev() can run in between. Check the pointers in subdev_open() instead.
Reproduced on a CHUWI Hi10 X1 (Alder Lake-N, IPU6) running 6.12.86, at cycle 7 of a loop unbinding and rebinding a sensor while four processes opened every /dev/v4l-subdev*. The code is unchanged in v7.3-rc2.
---
Unchanged since v1.
drivers/media/v4l2-core/v4l2-subdev.c | 31 +++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-)
drivers/media/v4l2-core/v4l2-subdev.c +25 -6
@@ -97,8 +97,19 @@ static int subdev_open(struct file *file)struct video_device *vdev =video_devdata (file);struct v4l2_subdev *sd =vdev_to_v4l2_subdev (vdev);struct v4l2_subdev_fh *subdev_fh; +struct v4l2_device *v4l2_dev;int ret; +/* +* v4l2_device_unregister_subdev() clears sd->v4l2_dev and unregisters +* the entity before it unregisters the device node, so an open() that +* races with the sub-device going away lands here with those pointers +* already gone. +*/ + v4l2_dev =READ_ONCE (sd->v4l2_dev); +if (!v4l2_dev) +return -ENODEV ; + subdev_fh =kzalloc_obj (*subdev_fh);if (subdev_fh ==NULL )return -ENOMEM ;
@@ -112,15 +123,23 @@ static int subdev_open(struct file *file)v4l2_fh_init (&subdev_fh->vfh, vdev);v4l2_fh_add (&subdev_fh->vfh, file); -if (sd->v4l2_dev->mdev && sd->entity.graph_obj.mdev->dev) { -struct module *owner; +if (v4l2_dev->mdev) { +struct media_device *mdev =READ_ONCE (sd->entity.graph_obj.mdev); - owner = sd->entity.graph_obj.mdev->dev->driver->owner; -if (!try_module_get (owner)) { - ret = -EBUSY ; +if (!mdev) { + ret = -ENODEV ; goto err; } - subdev_fh->owner = owner; + +if (mdev->dev) { +struct module *owner = mdev->dev->driver->owner; + +if (!try_module_get (owner)) { + ret = -EBUSY ; + goto err; + } + subdev_fh->owner = owner; + } }if (sd->internal_ops && sd->internal_ops->open) {
Signature · 1 line
2.47.3
PATCH v2 3/3 media: ipu6: Signal the video queues when a sensor is unbound
In reply to Nicola Fiorilloisys_async_ops implements .bound() and .complete() but not .unbind(), so nothing tells the ISYS video nodes that the sensor feeding them has gone away. A capture that is streaming when the sensor is unbound stays blocked in vb2_core_dqbuf() forever, waiting for a frame that can no longer arrive:
[<0>] vb2_core_dqbuf+0x362/0x1190 [videobuf2_common] [<0>] vb2_dqbuf+0xb4/0x210 [videobuf2_v4l2] [<0>] __video_do_ioctl+0x894/0xb30 [<0>] video_usercopy+0x479/0xde0 [<0>] v4l2_ioctl+0x198/0x220 [<0>] __x64_sys_ioctl+0x134/0x1c0
The wait in __vb2_wait_for_done_vb() ends on a new buffer, on !q->streaming, or on q->error. Tearing the sensor down sets none of the three. The sleep is interruptible, so DETECT_HUNG_TASK stays quiet as well and the process is simply stuck until something kills it.
Add the missing .unbind() and mark the queues of the CSI-2 receiver the departing sensor was attached to, which is enough for DQBUF to return -EIO. Only streaming queues are flagged: q->error is cleared by __vb2_queue_cancel(), so flagging an idle queue would leave it in error until the next VIDIOC_STREAMOFF.
Reproduced on a CHUWI Hi10 X1 (Alder Lake-N, IPU6) by unbinding the sensor while v4l2-ctl was streaming, with both sensors of the machine. Without this patch 3 attempts out of 3 hang; with it, 10 out of 10 wake up, report "VIDIOC_DQBUF: failed: Input/output error" and exit. The same run under KASAN reports nothing.
---
Unchanged since v1; the note below is new.
The check and the marking are deliberately not done under q->lock, and I would rather say so than have it look like an oversight.
The lock of these queues is av->mutex (aq->vbq.lock, ipu6-isys-queue.c), and taking it here is not possible as the teardown stands: isys_remove() calls isys_unregister_devices() before isys_notifier_cleanup(), and the former ends in ipu6_isys_video_cleanup() -> mutex_destroy(&av->mutex). So on driver removal this callback runs after that mutex has been destroyed, and taking it would be an OOPS with CONFIG_DEBUG_MUTEXES.
Without the lock there is a narrow race with a concurrent STREAMOFF (an idle queue left flagged until the next STREAMOFF) or STREAMON (the hang comes back). The unlocked read is safe in the removal path itself, because vb2_video_unregister_device() has already released the queue under the lock, so vb2_is_streaming() is false there and the loop does nothing.
The proper fix looks like unregistering the notifier before the video devices, which would also make teardown the mirror of setup -- isys_register_devices() registers the video devices first and inits the notifier last, and its own error path unwinds in that order. I did not put that in this series because I cannot build or test a kernel at the moment, and changing the removal path untested seemed worse than leaving this documented. I am happy to write it as a follow-up if you agree with the direction.
drivers/media/pci/intel/ipu6/ipu6-isys.c | 41 ++++++++++++++++++++++++ 1 file changed, 41 insertions(+)
drivers/media/pci/intel/ipu6/ipu6-isys.c +41 -0
@@ -31,6 +31,7 @@#include <media/v4l2-async.h> #include <media/v4l2-device.h> #include <media/v4l2-fwnode.h> +#include <media/videobuf2-core.h> #include "ipu6-bus.h" #include "ipu6-cpd.h"
@@ -700,6 +701,45 @@ static int isys_notifier_bound(struct v4l2_async_notifier *notifier,return v4l2_device_register_subdev_nodes (&isys->v4l2_dev); } +/* The .unbind() notifier callback when a sub-device goes away */ +static void isys_notifier_unbind (struct v4l2_async_notifier *notifier, +struct v4l2_subdev *sd, +struct v4l2_async_connection *asc) +{ +struct ipu6_isys *isys = + container_of(notifier, struct ipu6_isys, notifier); +struct sensor_async_sd *s_asd = + container_of(asc, struct sensor_async_sd, asc); +struct ipu6_isys_csi2 *csi2; +unsigned int i; + +if (s_asd->csi2.port >= isys->pdata->ipdata->csi2.nports) +return ; + +/* +* The sensor is gone, so no more frames will ever arrive on the video +* nodes fed by it. Tell videobuf2, or a DQBUF already blocked in +* vb2_core_dqbuf() would sleep forever: nothing else in the teardown +* path wakes that queue up. +* +* Only queues that are actually streaming are marked. The error flag +* is only cleared by __vb2_queue_cancel(), so flagging an idle queue +* would leave it poisoned until the next STREAMOFF. +*/ + csi2 = &isys->csi2[s_asd->csi2.port]; +for (i =0 ; i <NR_OF_CSI2_SRC_PADS ; i++) { +struct vb2_queue *q = &csi2->av[i].aq.vbq; + +if (!vb2_is_streaming (q)) +continue ; + +dev_dbg (&isys->adev->auxdev.dev, +"%s went away while streaming on %s\n" , sd->name, + csi2->av[i].vdev.name); +vb2_queue_error (q); + } +} +static int isys_notifier_complete (struct v4l2_async_notifier *notifier) { struct ipu6_isys *isys =
@@ -710,6 +750,7 @@ static int isys_notifier_complete(struct v4l2_async_notifier *notifier)static const struct v4l2_async_notifier_operations isys_async_ops = { .bound = isys_notifier_bound, + .unbind = isys_notifier_unbind, .complete = isys_notifier_complete, };
Signature · 1 line
2.47.3
Sakari Ailus
maintainerHi Nicola,
Quoted 5 lines · Nicola Fiorillo
Unbinding a camera sensor while a capture is running is a scenario that nothing in the IPU6 path handles: the kernel oopses twice, corrupts memory once, and leaves the application blocked forever. None of this is caused by the sensor drivers themselves, and all four failures are present in mainline today.
Thanks for the patchset. These are known issues and unfortunately unbinding drivers while streaming isn't supported on MC-enabled drivers currently. This is a MC/V4L2 framework limitation and cannot be meaningfully worked around in drivers.
Signature · 3 lines
Kind regards, Sakari Ailus
Hi Sakari,
Quoted 4 lines · Sakari Ailus
Thanks for the patchset. These are known issues and unfortunately unbinding drivers while streaming isn't supported on MC-enabled drivers currently. This is a MC/V4L2 framework limitation and cannot be meaningfully worked around in drivers.
Understood, and I won't argue the point for 1/3 and 3/3: both are IPU6 driver changes for exactly the unsupported scenario you describe. I'll drop them.
Could I ask you to look at 2/3 on its own, though? I believe it is a different bug, and the cover letter framed it badly -- that is my fault.
2/3 is not a driver change and does not involve streaming at all. It is a NULL dereference in subdev_open() in the framework itself, caused by the ordering inside v4l2_device_unregister_subdev():
sd->v4l2_dev = NULL; /* ... */ media_device_unregister_entity(&sd->entity); /* clears sd->entity.graph_obj.mdev via media_gobj_destroy() */ /* ... */ video_unregister_device(sd->devnode); /* the node goes away last */
Because the device node is removed last, there is a window in which /dev/v4l-subdevN can still be opened while both sd->v4l2_dev and sd->entity.graph_obj.mdev are already NULL. subdev_open() then runs
if (sd->v4l2_dev->mdev && sd->entity.graph_obj.mdev->dev)
on those pointers and oopses.
No capture has to be running for this, and it is not specific to IPU6:
any sub-device with a device node can hit it on any removal path -- unbind, module unload, or a hot-removed PCI/USB device. Unbinding during streaming is simply how I ran into it.
If you would rather have it resent on its own, with a cover letter that doesn't mention unbinding a streaming sensor, I'm happy to do that instead.
Thanks for taking the time to look.
Signature · 1 line
Nicola Fiorillo
Hi Sakari,
On 18 September you mentioned you would reply to the framework patch (2/3 of this series) separately. Before you spend time on it: I have reworked it, and I would rather you look at the new version than at this one.
The v2 patch only checked the two pointers in subdev_open(), which narrows the race but does not close it. The rework stops reading the sub-device's registration state in the file operations altogether and uses the node's own vdev->v4l2_dev, which unregistration never clears. Reading the code further, the same sd->v4l2_dev dereference is also in the EXT_CTRLS ioctls, so it is now a two-patch series touching only v4l2-subdev.c.
I tested it in QEMU with KASAN and vimc, unbinding and rebinding vimc in a loop while 16 threads open the sub-device nodes or issue EXT_CTRLS ioctls. On media next every run oopsed in subdev_open(); with the first patch only, every EXT_CTRLS run oopsed in subdev_do_ioctl(); with both, there was no oops or KASAN report at all. It does not address the lifetime limitation you described on v2, and the commit messages say so.
Shall I post it as v3, or would you prefer to comment on the v2 patch first?
Thanks,
Nicola