Skip to contentSkip to conversation

media: dvb-core: add upper bound check in DMX_SET_BUFFER_SIZE ioctl

PATCH Superseded Cen Zhang (Microsoft) 2 participants 4 messages Jul 22 – Jul 31 Media (V4L/DVB)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 media: dvb-core: add upper bound check in DMX_SET_BUFFER_SIZE ioctl

dvb_dvr_set_buffer_size() and dvb_dmxdev_set_buffer_size() pass the user-supplied size argument directly to vmalloc() without any upper bound check. This allows excessive kernel memory allocation via the DMX_SET_BUFFER_SIZE ioctl, which can lead to system-wide OOM conditions.

  Kernel panic - not syncing: System is deadlocked on memory

  Call Trace:
   out_of_memory+0x12fd/0x1370
   __alloc_frozen_pages_noprof+0x2620/0x2fa0
   __vmalloc_node_range_noprof+0x7fa/0x1490
   dvb_dvr_do_ioctl+0x11e/0x260 (drivers/media/dvb-core/dmxdev.c:296)
   dvb_usercopy+0x15b/0x360

Fix by cap both functions at 64 MB and return -EINVAL for oversized requests.

Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")
Fixes a095be4b030c ("V4L/DVB (7659): dvb-core: Implement DMX_SET_BUFFER_SIZE for dvr")
Reported-by [email protected]
---
 drivers/media/dvb-core/dmxdev.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/media/dvb-core/dmxdev.c +7 -0
index 3c8bc75e4d6c..b4dc87945be1 100644
@@ -20,6 +20,9 @@
 #include <media/dmxdev.h>
 #include <media/dvb_vb2.h>
 
+/* 64 MB upper bound for DVB ring buffer allocations */
+#define DVB_BUFFER_SIZE_MAX (64 * 1024 * 1024)
+
 static int debug;
 
 module_param(debug, int, 0644);
@@ -292,6 +295,8 @@ static int dvb_dvr_set_buffer_size(struct dmxdev *dmxdev,
 		return 0;
 	if (!size)
 		return -EINVAL;
+	if (size > DVB_BUFFER_SIZE_MAX)
+		return -EINVAL;
 
 	newmem = vmalloc(size);
 	if (!newmem)
@@ -333,6 +338,8 @@ static int dvb_dmxdev_set_buffer_size(struct dmxdev_filter *dmxdevfilter,
 		return -EINVAL;
 	if (dmxdevfilter->state >= DMXDEV_STATE_GO)
 		return -EBUSY;
+	if (size > DVB_BUFFER_SIZE_MAX)
+		return -EINVAL;
 
 	newmem = vmalloc(size);
 	if (!newmem)
Signature · 1 line
2.53.0
#1 lore raw
Quoted 30 lines · On Wed, 22 Jul 2026 15:37:19 -0400 "Cen Zhang (Microsoft)" <[email protected]>…

dvb_dvr_set_buffer_size() and dvb_dmxdev_set_buffer_size() pass the user-supplied size argument directly to vmalloc() without any upper bound check. This allows excessive kernel memory allocation via the DMX_SET_BUFFER_SIZE ioctl, which can lead to system-wide OOM conditions.

  Kernel panic - not syncing: System is deadlocked on memory

  Call Trace:
   out_of_memory+0x12fd/0x1370
   __alloc_frozen_pages_noprof+0x2620/0x2fa0
   __vmalloc_node_range_noprof+0x7fa/0x1490
   dvb_dvr_do_ioctl+0x11e/0x260 (drivers/media/dvb-core/dmxdev.c:296)
   dvb_usercopy+0x15b/0x360

Fix by cap both functions at 64 MB and return -EINVAL for oversized requests.

Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")
Fixes a095be4b030c ("V4L/DVB (7659): dvb-core: Implement DMX_SET_BUFFER_SIZE for dvr")
Reported-by [email protected]
---
 drivers/media/dvb-core/dmxdev.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/media/dvb-core/dmxdev.c +0 -0
index 3c8bc75e4d6c..b4dc87945be1 100644

drivers/media/dvb-core/dmxdev.c line 24

Quoted 5 lines
drivers/media/dvb-core/dmxdev.c +2 -0
@@ -20,6 +20,9 @@
 #include <media/dmxdev.h>
 #include <media/dvb_vb2.h>
 
+/* 64 MB upper bound for DVB ring buffer allocations */
+#define DVB_BUFFER_SIZE_MAX (64 * 1024 * 1024)

Having a limit is good, but why 64MB?

Can you provide me more details about the test scenario: e.g. with what TV standards had you test it, and such.

Regards,
Mauro

Thanks,
Mauro

#2 lore raw
Quoted 36 lines · On Wed, 29 Jul 2026 08:30:26 +0200 Mauro Carvalho Chehab…
Quoted 33 lines · On Wed, 22 Jul 2026 15:37:19 -0400 "Cen Zhang (Microsoft)" <[email protected]>…

dvb_dvr_set_buffer_size() and dvb_dmxdev_set_buffer_size() pass the user-supplied size argument directly to vmalloc() without any upper bound check. This allows excessive kernel memory allocation via the DMX_SET_BUFFER_SIZE ioctl, which can lead to system-wide OOM conditions.

  Kernel panic - not syncing: System is deadlocked on memory

  Call Trace:
   out_of_memory+0x12fd/0x1370
   __alloc_frozen_pages_noprof+0x2620/0x2fa0
   __vmalloc_node_range_noprof+0x7fa/0x1490
   dvb_dvr_do_ioctl+0x11e/0x260 (drivers/media/dvb-core/dmxdev.c:296)
   dvb_usercopy+0x15b/0x360

Fix by cap both functions at 64 MB and return -EINVAL for oversized requests.

Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")
Fixes a095be4b030c ("V4L/DVB (7659): dvb-core: Implement DMX_SET_BUFFER_SIZE for dvr")
Reported-by [email protected]
---
 drivers/media/dvb-core/dmxdev.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/media/dvb-core/dmxdev.c +0 -0
index 3c8bc75e4d6c..b4dc87945be1 100644
@@ -20,6 +20,9 @@
 #include <media/dmxdev.h>
 #include <media/dvb_vb2.h>
 
Quoted 7 lines

drivers/media/dvb-core/dmxdev.c line 24

Quoted 2 lines
drivers/media/dvb-core/dmxdev.c +2 -0
+/* 64 MB upper bound for DVB ring buffer allocations */
+#define DVB_BUFFER_SIZE_MAX (64 * 1024 * 1024)

Having a limit is good, but why 64MB?

Can you provide me more details about the test scenario: e.g. with what TV standards had you test it, and such.

Forgot to mention, but instead of returning -EINVAL, probably the best would be to setup the ringbuffer size to the maximum value, as otherwise this would break existing apps.

Btw, what apps did you use to test it? Had you check what's the current limit on what apps?

Regards,
Mauro

#3 lore raw

Hi Mauro,

Quoted 3 lines

Forgot to mention, but instead of returning -EINVAL, probably the best would be to setup the ringbuffer size to the maximum value, as otherwise this would break existing apps.

Thanks for pointing this out. I will revise this in v3.

Quoted 4 lines

Having a limit is good, but why 64MB?

Btw, what apps did you use to test it? Had you check what's the current limit on what apps?

I chose 64 MiB as a conservative upper bound based on a brief source survey of DVB applications:

- The kernel DVR buffer defaults to approximately 1.84 MiB. In the upstream revisions I checked, TVheadend, MythTV and MuMuDVB do not actively call DMX_SET_BUFFER_SIZE and therefore retain this default:

https://github.com/torvalds/linux/blob/acb7500801e98639f6d8c2d796ed9f64cba83d3a/include/media/dmxdev.h#L189

- SATPI allows approximately 30 MiB:

https://github.com/Barracuda09/SATPI/blob/09df7402870b1dff2c7b2d7f4a085002a333045f/src/input/dvb/Frontend.cpp#L57

- DVBlast defaults to approximately 7.34 MiB:

https://github.com/videolan/dvblast/blob/ae6b24ae8cdf6eaa1ada8534a257258f3c906a77/dvb.c#L71

- dvbv5-zap uses approximately 5.88 MiB:

https://github.com/gjasny/v4l-utils/blob/95ad25f6a77a0a6650f5f657ac2c5046efcd04a0/utils/dvb/dvbv5-zap.c#L32

- minisatip defaults to approximately 5.51 MiB:

https://github.com/catalinii/minisatip/blob/d20136e23d47bd7f5de2967b34463f14bce6f988/src/adapter.h#L12

I have limited experience with DVB hardware and production workloads. Do you have a recommended limit based on known hardware or application requirements? I am happy to use the value you recommend in v3.

Quoted 2 lines

Can you provide me more details about the test scenario: e.g. with what TV standards had you test it, and such.

The reproducer used the vidtv virtual DVB adapter (dvb-vidtv-bridge) in a QEMU VM with approximately 3 GiB of RAM. The root-only setup loaded the driver, made dvr0 accessible and set oom_score_adj=-1000 for the process lineage to make the panic deterministic. The ioctl itself was issued by an unprivileged process.

The test did not exercise a particular TV standard or transport stream, since the failure occurs directly in the buffer-allocation ioctl path before stream processing. I have not tested this on physical DVB hardware.

I can share the reproducer package privately if that would be useful.

Best regards,
Cen