imx8-isi: Bug fixes and format support enhancements
v2 cover imx8-isi: Bug fixes and format support enhancements
This series addresses critical bugs in the imx8-isi driver and extends format support for high-end sensors and Android requirements.
Patch 1 fixes a critical stream ID validation bug in the crossbar routing where the validation loop iterates over the wrong routing table, allowing userspace to bypass validation entirely and configure invalid routes.
Patch 2 adds additional stream ID validation to enforce hardware constraints (SOURCE stream must be 0).
Patch 3 fixes a stream reference counting bug in the crossbar that prevents multiple streams from different virtual channels on the same input pad from being enabled correctly. Only the first stream gets enabled in hardware, subsequent streams are silently ignored.
Patch 4 adds support for 16-bit raw Bayer formats (SBGGR16, SGBRG16, SGRBG16, SRGGB16) commonly used by high-end image sensors.
Patch 5 fixes incorrect color mapping for XBGR32 format in memory-to-memory mode (marked for stable backport).
Patch 6 extends RGB format support by adding BGRA32, RGBA32, BGRX32, RGBX32, and ARGB2101010 formats with full M2M capabilities to meet Android requirements.
---
Changes in v2:
- [1/6] Split v1 patch 1/5 into two patches: this patch fixes the core for_each_active_route() bug, next patch adds additional stream validation (Frank Li)
- [2/6] New patch split from v1 1/5: adds stream ID validation on top of for_each_active_route() fix (Frank Li)
- [2/6] Remove incorrect sink_stream validation
- [3/6] Use fixed-size array for enabled_count instead of dynamic allocation
- [3/6] Use BIT_ULL() macro for u64 bitmask operations
- [3/6] Use MXC_ISI_MAX_STREAMS (64) as loop boundary instead of num_sources
- [3/6] Remove mxc_isi_stream_counters_alloc/free functions
- [4/6] Add Reviewed-by tag from Frank Li
- [5/6] Reword commit description for clarity (Frank Li)
- [6/6] Add Reviewed-by tag from Frank Li
- Link to v1: https://lore.kernel.org/r/[email protected]
---
Guoniu Zhou (5):
media: nxp: imx8-isi: Fix stream ID validation bypass in crossbar routing
media: nxp: imx8-isi: Add stream ID validation for crossbar routing
media: nxp: imx8-isi: Fix per-stream reference counting for multiplexed streams
media: nxp: imx8-isi: Correct color map between V4L2 and ISI
media: nxp: imx8-isi: Add additional 32-bit RGB format support
Laurentiu Palcu (1):
media: nxp: imx8-isi: Add 16-bit raw Bayer format support
.../media/platform/nxp/imx8-isi/imx8-isi-core.h | 7 +- .../platform/nxp/imx8-isi/imx8-isi-crossbar.c | 92 +++++++++++++++----- .../media/platform/nxp/imx8-isi/imx8-isi-pipe.c | 24 ++++++ .../media/platform/nxp/imx8-isi/imx8-isi-video.c | 97 +++++++++++++++++++++- 4 files changed, 196 insertions(+), 24 deletions(-)
---
base-commit: 06cb687a5132fcffe624c0070576ab852ac6b568
change-id: 20260626-isi-00f05b044ac9
Best regards,
Signature · 1 line
Guoniu Zhou <[email protected]>
PATCH v2 1/6 media: nxp: imx8-isi: Fix stream ID validation bypass in crossbar routing
The crossbar routing validation has a critical bug where it validates the wrong routing table, allowing userspace to bypass validation entirely.
The __mxc_isi_crossbar_set_routing() function is called to validate and apply a new routing table from userspace. However, the validation loop iterates over state->routing (the currently active routing table) instead of the routing parameter (the new table being validated):
for_each_active_route(&state->routing, route) {This means userspace can submit any invalid routing configuration and it will pass validation as long as the currently active routing is valid. This is a security issue as it allows userspace to configure routes that violate hardware constraints, potentially causing undefined hardware behavior.
Fix by validating the routing table that will actually be applied.
---
Changes in v2:
- Split v1 patch 1/5 into two patches: this patch fixes the core for_each_active_route() bug, next patch adds additional stream validation (Frank Li)
--- drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +1 -1
@@ -107,7 +107,7 @@ static int __mxc_isi_crossbar_set_routing(struct v4l2_subdev *sd,return ret;/* The memory input can be routed to the first pipeline only. */ -for_each_active_route (&state->routing, route) { +for_each_active_route (routing, route) {if (route->sink_pad == xbar->num_sinks -1 && route->source_pad != xbar->num_sinks) { dev_dbg(xbar->isi->dev,
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v2 1/6] media: nxp: imx8-isi: Fix stream ID validation bypass in crossbar routing
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Pre-existing issues:
- [High] Undefined behavior from bit shift overflow when using BIT() macro on user-controlled stream IDs on 32-bit architectures.
- [High] Missing V4L2_SUBDEV_ROUTING_NO_SOURCE_STREAM_MIX flag in routing validation allows stream mixing, violating driver assumptions and causing pipeline state corruption. --
--- Patch [1]: [PATCH v2 1/6] media: nxp: imx8-isi: Fix stream ID validation bypass in crossbar routing --- commit c48b83ab851ce7535e8d8249fb85d35c57bb63e1
Author: Guoniu Zhou <[email protected]>
media: nxp: imx8-isi: Fix stream ID validation bypass in crossbar routing
The crossbar routing validation has a bug where it validates the wrong routing table, allowing userspace to bypass validation entirely. This commit fixes it by validating the routing table that will actually be applied.
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int __mxc_isi_crossbar_set_routing(struct v4l2_subdev *sd, · line 107
Quoted 6 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +0
-0@@ -107,7 +107,7 @@ static int __mxc_isi_crossbar_set_routing(struct v4l2_subdev *sd,return ret;
[Severity: High]
This isn't a bug introduced by this patch, but does the routing validation here miss the V4L2_SUBDEV_ROUTING_NO_SOURCE_STREAM_MIX flag?
The validation in __mxc_isi_crossbar_set_routing() does not pass this flag:
imx8-isi-crossbar.c:__mxc_isi_crossbar_set_routing() {
...
ret = v4l2_subdev_routing_validate(sd, routing,
V4L2_SUBDEV_ROUTING_NO_N_TO_1);
...
}This allows stream mixing, violating an explicit assumption in mxc_isi_crossbar_xlate_streams(), where the driver silently overwrites sink_pad for matching routes:
imx8-isi-crossbar.c:mxc_isi_crossbar_xlate_streams() {
...
/*
* Translate the source pad and streams to the sink side. The routing
* validation forbids stream merging, so all matching entries in the
* routing table are guaranteed to have the same sink pad.
*/
...
sink_streams |= BIT(route->sink_stream);
sink_pad = route->sink_pad;
...
}If userspace configures a routing table via VIDIOC_SUBDEV_S_ROUTING with multiple routes from different sink pads ending at the same source pad, mxc_isi_crossbar_enable_streams() will only enable streams on the last matched remote subdevice, leaving others disabled and leading to hardware pipeline hangs.
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int __mxc_isi_crossbar_set_routing(struct v4l2_subdev *sd, · line 110
Quoted 4 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +1
-1/* The memory input can be routed to the first pipeline only. */ -for_each_active_route (&state->routing, route) { +for_each_active_route (routing, route) {
[Severity: High]
This is a pre-existing issue, but can using the BIT() macro on user-controlled stream IDs cause undefined behavior on 32-bit architectures?
The core framework limits stream IDs to 63 (V4L2_SUBDEV_MAX_STREAM_ID), but the driver uses BIT() on these IDs in mxc_isi_crossbar_xlate_streams():
imx8-isi-crossbar.c:mxc_isi_crossbar_xlate_streams() {
...
if (route->source_pad != source_pad ||
!(source_streams & BIT(route->source_stream)))
continue;
sink_streams |= BIT(route->sink_stream);
...
}On 32-bit architectures, unsigned long is 32 bits. If userspace provides a routing table via VIDIOC_SUBDEV_S_ROUTING with stream IDs >= 32, this causes a shift out of bounds. This can result in incorrect bitmasks, causing the driver to start or stop the wrong hardware streams and hang the pipeline.
Quoted 3 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +0
-0if (route->sink_pad == xbar->num_sinks -1 && route->source_pad != xbar->num_sinks) { dev_dbg(xbar->isi->dev,
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Re: [PATCH v2 1/6] media: nxp: imx8-isi: Fix stream ID validation bypass in crossbar routing
In reply to Guoniu ZhouQuoted 16 lines · Guoniu Zhou
The crossbar routing validation has a critical bug where it validates the wrong routing table, allowing userspace to bypass validation entirely.
The
__mxc_isi_crossbar_set_routing()function is called to validate and apply a new routing table from userspace. However, the validation loop iterates overstate->routing(the currently active routing table) instead of the routing parameter (the new table being validated):for_each_active_route(&state->routing, route) {This means userspace can submit any invalid routing configuration and it will pass validation as long as the currently active routing is valid. This is a security issue as it allows userspace to configure routes that violate hardware constraints, potentially causing undefined hardware behavior.
Quoted 6 lines
Fix by validating the routing table that will actually be applied.
Fixes cf21f328fcaf ("media: nxp: Add i.MX8 ISI driver")Signed-off-by Guoniu Zhou <[email protected]>---
Quoted 26 lines
Changes in v2:
- Split v1 patch 1/5 into two patches: this patch fixes the corefor_each_active_route()bug, next patch adds additional stream validation (Frank Li)--- drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +1
-2@@ -107,7 +107,7 @@ static int __mxc_isi_crossbar_set_routing(struct v4l2_subdev *sd,return ret;/* The memory input can be routed to the first pipeline only. */ -for_each_active_route (&state->routing, route) { +for_each_active_route (routing, route) {if (route->sink_pad == xbar->num_sinks -1 && route->source_pad != xbar->num_sinks) { dev_dbg(xbar->isi->dev, --2.34.1
Re: [PATCH v2 1/6] media: nxp: imx8-isi: Fix stream ID validation bypass in crossbar routing
In reply to Guoniu ZhouHi Guoniu,
Thank you for the patch.
Quoted 16 lines · Guoniu Zhou
The crossbar routing validation has a critical bug where it validates the wrong routing table, allowing userspace to bypass validation entirely.
The
__mxc_isi_crossbar_set_routing()function is called to validate and apply a new routing table from userspace. However, the validation loop iterates overstate->routing(the currently active routing table) instead of the routing parameter (the new table being validated):for_each_active_route(&state->routing, route) {This means userspace can submit any invalid routing configuration and it will pass validation as long as the currently active routing is valid. This is a security issue as it allows userspace to configure routes that violate hardware constraints, potentially causing undefined hardware behavior.
Quoted 5 lines
Fix by validating the routing table that will actually be applied.
Fixes cf21f328fcaf ("media: nxp: Add i.MX8 ISI driver")Signed-off-by Guoniu Zhou <[email protected]>
Quoted 23 lines
---
Changes in v2:
- Split v1 patch 1/5 into two patches: this patch fixes the corefor_each_active_route()bug, next patch adds additional stream validation (Frank Li)--- drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +1
-1@@ -107,7 +107,7 @@ static int __mxc_isi_crossbar_set_routing(struct v4l2_subdev *sd,return ret;/* The memory input can be routed to the first pipeline only. */ -for_each_active_route (&state->routing, route) { +for_each_active_route (routing, route) {if (route->sink_pad == xbar->num_sinks -1 && route->source_pad != xbar->num_sinks) { dev_dbg(xbar->isi->dev,
Signature · 3 lines
Regards, Laurent Pinchart
PATCH v2 2/6 media: nxp: imx8-isi: Add stream ID validation for crossbar routing
In reply to Guoniu ZhouAdd validation to enforce hardware constraints that were previously missing in the crossbar routing configuration:
- SOURCE stream must be 0 (ISI pipes are hardcoded to stream 0)
This check complements the existing memory input validation and ensures that all routing configurations respect hardware limitations.
---
Changes in v2:
- New patch split from v1 1/5: adds stream ID validation on top of for_each_active_route() fix (Frank Li)
- Remove incorrect sink_stream validation
--- drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +13 -1
@@ -106,8 +106,20 @@ static int __mxc_isi_crossbar_set_routing(struct v4l2_subdev *sd,if (ret)return ret; -/* The memory input can be routed to the first pipeline only. */ +/* +* Validate routes against hardware constraints: +* - SOURCE stream must be 0 (pipes are hardcoded to stream 0) +* - Memory input can only route to the first pipeline +*/ for_each_active_route (routing, route) { +if (route->source_stream !=0 ) { +dev_dbg (xbar->isi->dev, +"route to pipe %u must use source_stream=0, got %u\n" , + route->source_pad - xbar->num_sinks, + route->source_stream); +return -ENXIO ; + } +if (route->sink_pad == xbar->num_sinks -1 && route->source_pad != xbar->num_sinks) { dev_dbg(xbar->isi->dev,
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v2 2/6] media: nxp: imx8-isi: Add stream ID validation for crossbar routing
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] The driver uses the BIT() macro on 64-bit stream masks (source_streams and sink_streams), which will cause undefined behavior on 32-bit platforms if the stream ID exceeds 31. --
--- Patch [2]: [PATCH v2 2/6] media: nxp: imx8-isi: Add stream ID validation for crossbar routing --- commit d756aed8e216cd683009378d45743e6351a90389
Author: Guoniu Zhou <[email protected]>
media: nxp: imx8-isi: Add stream ID validation for crossbar routing
This commit adds validation to enforce hardware constraints in the crossbar routing configuration. It ensures that the SOURCE stream is strictly set to 0, as ISI pipes are hardcoded to stream 0. It also ensures memory input respects routing limitations to the first pipeline.
Quoted 6 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +0
-0@@ -106,8 +106,20 @@ static int __mxc_isi_crossbar_set_routing(struct v4l2_subdev *sd,if (ret)
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int __mxc_isi_crossbar_set_routing(struct v4l2_subdev *sd, · line 115
Quoted 10 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +6
-1return ret; -/* The memory input can be routed to the first pipeline only. */ +/* +* Validate routes against hardware constraints: +* - SOURCE stream must be 0 (pipes are hardcoded to stream 0) +* - Memory input can only route to the first pipeline +*/ for_each_active_route (routing, route) { +if (route->source_stream !=0 ) {
[Severity: High]
This is a pre-existing issue, but while reviewing this stream ID validation, I noticed a potential bit shift overflow regarding stream IDs in the related mxc_isi_crossbar_xlate_streams() function:
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c:mxc_isi_crossbar_xlate_streams() {
...
if (!(source_streams & BIT(route->source_stream)))
...
sink_streams |= BIT(route->sink_stream);
...
}Does this code cause undefined behavior on 32-bit platforms if userspace provides a V4L2 routing configuration with a sink_stream or source_stream greater than or equal to 32?
Since the BIT() macro evaluates to a 32-bit value on 32-bit architectures, shifting it by 32 or more results in a bit shift overflow. This could allow the compiler to make unsafe optimization assumptions, potentially corrupting the 64-bit stream mask logic.
Should this use BIT_ULL() instead?
Quoted 10 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +7
-0+dev_dbg (xbar->isi->dev, +"route to pipe %u must use source_stream=0, got %u\n" , + route->source_pad - xbar->num_sinks, + route->source_stream); +return -ENXIO ; + } +if (route->sink_pad == xbar->num_sinks -1 && route->source_pad != xbar->num_sinks) { dev_dbg(xbar->isi->dev,
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Re: [PATCH v2 2/6] media: nxp: imx8-isi: Add stream ID validation for crossbar routing
In reply to Guoniu ZhouQuoted 25 lines · Guoniu Zhou
Add validation to enforce hardware constraints that were previously missing in the crossbar routing configuration:
- SOURCE stream must be 0 (ISI pipes are hardcoded to stream 0)
This check complements the existing memory input validation and ensures that all routing configurations respect hardware limitations.
Signed-off-by Guoniu Zhou <[email protected]>---
Changes in v2:
- New patch split from v1 1/5: adds stream ID validation on top offor_each_active_route()fix (Frank Li)
- Remove incorrectsink_streamvalidation--- drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +0
-0@@ -106,8 +106,20 @@ static int __mxc_isi_crossbar_set_routing(struct v4l2_subdev *sd,if (ret)return ret;
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int __mxc_isi_crossbar_set_routing(struct v4l2_subdev *sd, · line 116
Quoted 10 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +7
-1-/* The memory input can be routed to the first pipeline only. */ +/* +* Validate routes against hardware constraints: +* - SOURCE stream must be 0 (pipes are hardcoded to stream 0) +* - Memory input can only route to the first pipeline +*/ for_each_active_route (routing, route) { +if (route->source_stream !=0 ) { + dev_dbg(xbar->isi->dev,
it is one error, should be dev_err()?
Frank
Quoted 14 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +6
-1+"route to pipe %u must use source_stream=0, got %u\n" , + route->source_pad - xbar->num_sinks, + route->source_stream); +return -ENXIO ; + } +if (route->sink_pad == xbar->num_sinks -1 && route->source_pad != xbar->num_sinks) { dev_dbg(xbar->isi->dev, --2.34.1
Re: [PATCH v2 2/6] media: nxp: imx8-isi: Add stream ID validation for crossbar routing
Quoted 27 lines · Frank Li
Quoted 26 lines · Guoniu Zhou
Add validation to enforce hardware constraints that were previously missing in the crossbar routing configuration:
- SOURCE stream must be 0 (ISI pipes are hardcoded to stream 0)
This check complements the existing memory input validation and ensures that all routing configurations respect hardware limitations.
Signed-off-by Guoniu Zhou <[email protected]>---
Changes in v2:
- New patch split from v1 1/5: adds stream ID validation on top offor_each_active_route()fix (Frank Li)
- Remove incorrectsink_streamvalidation--- drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +0
-0@@ -106,8 +106,20 @@ static int __mxc_isi_crossbar_set_routing(struct v4l2_subdev *sd,if (ret)return ret;
Quoted 11 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int __mxc_isi_crossbar_set_routing(struct v4l2_subdev *sd, · line 116
Quoted 9 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +7
-1-/* The memory input can be routed to the first pipeline only. */ +/* +* Validate routes against hardware constraints: +* - SOURCE stream must be 0 (pipes are hardcoded to stream 0) +* - Memory input can only route to the first pipeline +*/ for_each_active_route (routing, route) { +if (route->source_stream !=0 ) { + dev_dbg(xbar->isi->dev,it is one error, should be
dev_err()?
As this is an error that can be triggered by userspace, we prefer dev_dbg() to avoid giving unpriviledge userspace a way to flood the kernel log.
Quoted 9 lines
Quoted 9 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +6
-0+"route to pipe %u must use source_stream=0, got %u\n" , + route->source_pad - xbar->num_sinks, + route->source_stream); +return -ENXIO ; + } +if (route->sink_pad == xbar->num_sinks -1 && route->source_pad != xbar->num_sinks) { dev_dbg(xbar->isi->dev,
Signature · 3 lines
Regards, Laurent Pinchart
PATCH v2 3/6 media: nxp: imx8-isi: Fix per-stream reference counting for multiplexed streams
In reply to Guoniu ZhouThe ISI crossbar fails to properly enable multiple streams from different virtual channels on the same input pad. Only the first stream gets enabled in hardware, subsequent streams are silently ignored.
The driver uses a single enable_count per input to track the input state. When enable_count is non-zero, the code assumes the input is already active and skips calling v4l2_subdev_enable_streams() for additional streams:
Call 1: enable_streams(stream 0)
-> enable_count == 0, enable gasket and stream 0 in hardware
-> enable_count = 1 Call 2: enable_streams(stream 1)
-> enable_count == 1, skip hardware enable (BUG!)
-> enable_count = 2
-> stream 1 never gets enabledSimilarly on disable, when enable_count reaches zero, ALL streams are disabled regardless of which streams are actually still active.
Fix this by tracking per-stream state using:
- enabled_streams (u64 bitmask): tracks which streams are currently enabled
- enabled_count[] (array): per-stream reference counter to support the same stream being enabled/disabled multiple times
Now each stream is independently enabled/disabled in hardware based on the enabled_streams bitmask, while enabled_count[] provides reference counting for scenarios where the same stream is enabled multiple times, such as duplicate cases in the ISI stream.
---
Changes in v2:
- Use fixed-size array for enabled_count instead of dynamic allocation
- Use BIT_ULL() macro for u64 bitmask operations
- Use MXC_ISI_MAX_STREAMS (64) as loop boundary instead of num_sources
- Remove mxc_isi_stream_counters_alloc/free functions
--- .../media/platform/nxp/imx8-isi/imx8-isi-core.h | 7 +- .../platform/nxp/imx8-isi/imx8-isi-crossbar.c | 76 ++++++++++++++++------ 2 files changed, 63 insertions(+), 20 deletions(-)
drivers/media/platform/nxp/imx8-isi/imx8-isi-core.h +6 -1
@@ -184,8 +184,13 @@ struct mxc_isi_dma_buffer {dma_addr_t dma; }; +/* V4L2 subdev max stream ID is 63, need 64 counters (0-63) */ +#define MXC_ISI_MAX_STREAMS 64 +struct mxc_isi_input { -unsigned int enable_count; +u64 enabled_streams; +/* Per-stream reference counter */ +unsigned int enabled_count[MXC_ISI_MAX_STREAMS ]; };struct mxc_isi_crossbar {
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +57 -19
118 lines not on this page · raw message
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v2 3/6] media: nxp: imx8-isi: Fix per-stream reference counting for multiplexed streams
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Pre-existing issues:
- [High] Unchecked return value from media_pad_remote_pad_first() causes a NULL pointer dereference if the sink pad has no enabled link.
- [Medium] Use of BIT() macro for 64-bit stream IDs causes undefined behavior and bitwise truncation on 32-bit platforms. --
--- Patch [3]: [PATCH v2 3/6] media: nxp: imx8-isi: Fix per-stream reference counting for multiplexed streams --- commit 976517cc78165ba12439fb004220b58271a0beea
Author: Guoniu Zhou <[email protected]>
media: nxp: imx8-isi: Fix per-stream reference counting for multiplexed streams
The driver uses a single enable_count per input to track the input state. This fails to properly enable multiple streams from different virtual channels on the same input pad. This patch tracks per-stream state using enabled_streams bitmask and enabled_count[] array.
Quoted 14 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +2
-0@@ -337,6 +337,8 @@ static int mxc_isi_crossbar_enable_streams(struct v4l2_subdev *sd,struct mxc_isi_crossbar *xbar =to_isi_crossbar (sd);struct v4l2_subdev *remote_sd;struct mxc_isi_input *input; +u64 streams_to_enable; +unsigned long stream;u64 sink_streams;u32 sink_pad;u32 remote_pad;
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int mxc_isi_crossbar_enable_streams(struct v4l2_subdev *sd, · line 353
Quoted 2 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +0
-0@@ -350,30 +352,47 @@ static int mxc_isi_crossbar_enable_streams(struct v4l2_subdev *sd, input = &xbar->inputs[sink_pad];
[Severity: High]
This is a pre-existing issue, but can this lead to a NULL pointer dereference if the remote link is disabled?
When userspace configures a V4L2 subdev route connecting a source pad to a sink pad on the crossbar, but leaves the incoming media link to that sink pad disabled, media_pad_remote_pad_first() will return NULL.
In drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c:
mxc_isi_crossbar_xlate_streams() {
...
pad = media_pad_remote_pad_first(&xbar->pads[sink_pad]);
sd = media_entity_to_v4l2_subdev(pad->entity);
if (!sd) {
...
}The media_entity_to_v4l2_subdev() macro immediately dereferences pad to access pad->entity before any NULL check is performed.
[Severity: Medium]
This is also a pre-existing issue, but might there be bitwise truncation and undefined behavior on 32-bit platforms when computing sink_streams?
When userspace configures a routing table using VIDIOC_SUBDEV_S_ROUTING with a sink_stream ID greater than or equal to 32, using BIT() will cause issues on 32-bit systems where BIT() operates on a 32-bit unsigned long.
In drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c:
mxc_isi_crossbar_xlate_streams() {
...
for_each_active_route(&state->routing, route) {
...
sink_streams |= BIT(route->sink_stream);
sink_pad = route->sink_pad;
}
...
}Could BIT_ULL() be used instead to safely construct the u64 sink_streams bitmask?
Quoted 4 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +0
-4-/* -* TODO: Track per-stream enable counts to support multiplexed -* streams. -*/
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Re: [PATCH v2 3/6] media: nxp: imx8-isi: Fix per-stream reference counting for multiplexed streams
In reply to Guoniu ZhouQuoted 49 lines · Guoniu Zhou
The ISI crossbar fails to properly enable multiple streams from different virtual channels on the same input pad. Only the first stream gets enabled in hardware, subsequent streams are silently ignored.
The driver uses a single
enable_countper input to track the input state. Whenenable_countis non-zero, the code assumes the input is already active and skips callingv4l2_subdev_enable_streams()for additional streams:Call 1:
enable_streams(stream 0)-> enable_count == 0, enable gasket and stream 0 in hardware -> enable_count = 1Call 2:
enable_streams(stream 1)-> enable_count == 1, skip hardware enable (BUG!) -> enable_count = 2 -> stream 1 never gets enabledSimilarly on disable, when
enable_countreaches zero, ALL streams are disabled regardless of which streams are actually still active.Fix this by tracking per-stream state using:
-enabled_streams(u64 bitmask): tracks which streams are currently enabled
- enabled_count[] (array): per-stream reference counter to support the same stream being enabled/disabled multiple timesNow each stream is independently enabled/disabled in hardware based on the
enabled_streamsbitmask, while enabled_count[] provides reference counting for scenarios where the same stream is enabled multiple times, such as duplicate cases in the ISI stream.Fixes cf21f328fcaf ("media: nxp: Add i.MX8 ISI driver")Signed-off-by Guoniu Zhou <[email protected]>---
Changes in v2:
- Use fixed-size array forenabled_countinstead of dynamic allocation
- UseBIT_ULL()macro for u64 bitmask operations
- UseMXC_ISI_MAX_STREAMS(64) as loop boundary instead ofnum_sources
- Remove mxc_isi_stream_counters_alloc/free functions--- .../media/platform/nxp/imx8-isi/imx8-isi-core.h | 7 +- .../platform/nxp/imx8-isi/imx8-isi-crossbar.c | 76 ++++++++++++++++------ 2 files changed, 63 insertions(+), 20 deletions(-)drivers/media/platform/nxp/imx8-isi/imx8-isi-core.h +0
-0
drivers/media/platform/nxp/imx8-isi/imx8-isi-core.h struct mxc_isi_dma_buffer { · line 188
Quoted 5 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-core.h +2
-0@@ -184,8 +184,13 @@ struct mxc_isi_dma_buffer {dma_addr_t dma; }; +/* V4L2 subdev max stream ID is 63, need 64 counters (0-63) */ +#define MXC_ISI_MAX_STREAMS 64
Any existing Macro define max stream ID as 63? it'd better to use such macro
Quoted 8 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-core.h +4
-1+struct mxc_isi_input { -unsigned int enable_count; +u64 enabled_streams; +/* Per-stream reference counter */ +unsigned int enabled_count[MXC_ISI_MAX_STREAMS ]; };
...
drivers/media/platform/nxp/imx8-isi/imx8-isi-core.h static int mxc_isi_crossbar_disable_streams(struct v4l2_subdev *sd, · line 426
Quoted 12 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-core.h +7
-1@@ -396,19 +417,36 @@ static int mxc_isi_crossbar_disable_streams(struct v4l2_subdev *sd, input = &xbar->inputs[sink_pad]; - input->enable_count--; +/* +* Decrease the enable count for each stream. Only disable streams +* whose count reaches zero. +*/ +for (stream =0 ; stream <MXC_ISI_MAX_STREAMS ; stream++) { +if (!(sink_streams &BIT_ULL (stream))) +continue ;
for_each_set_bit() ?
Frank
Quoted 39 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-core.h +19
-9-if (!input->enable_count) { - ret =v4l2_subdev_disable_streams (remote_sd, remote_pad, - sink_streams); -if (ret) -dev_err (xbar->isi->dev, -"failed to disable streams 0x%llx on '%s':%u: %d\n" , - sink_streams, remote_sd->name, remote_pad, ret); +if (!(input->enabled_streams &BIT_ULL (stream))) +continue ; -mxc_isi_crossbar_gasket_disable (xbar, sink_pad); +if (--input->enabled_count[stream] ==0 ) + streams_to_disable |=BIT_ULL (stream); } +if (!streams_to_disable) +return 0 ; + + ret =v4l2_subdev_disable_streams (remote_sd, remote_pad, + streams_to_disable); +if (ret) +dev_err (xbar->isi->dev, +"failed to disable streams 0x%llx on '%s':%u: %d\n" , + streams_to_disable, remote_sd->name, remote_pad, ret); + + input->enabled_streams &= ~streams_to_disable; + +if (!input->enabled_streams) +mxc_isi_crossbar_gasket_disable (xbar, sink_pad); +return ret; } --2.34.1
Laurent Pinchart
maintainerRe: [PATCH v2 3/6] media: nxp: imx8-isi: Fix per-stream reference counting for multiplexed streams
In reply to Guoniu ZhouQuoted 25 lines · Guoniu Zhou
The ISI crossbar fails to properly enable multiple streams from different virtual channels on the same input pad. Only the first stream gets enabled in hardware, subsequent streams are silently ignored.
The driver uses a single
enable_countper input to track the input state. Whenenable_countis non-zero, the code assumes the input is already active and skips callingv4l2_subdev_enable_streams()for additional streams:Call 1:
enable_streams(stream 0)-> enable_count == 0, enable gasket and stream 0 in hardware -> enable_count = 1Call 2:
enable_streams(stream 1)-> enable_count == 1, skip hardware enable (BUG!) -> enable_count = 2 -> stream 1 never gets enabledSimilarly on disable, when
enable_countreaches zero, ALL streams are disabled regardless of which streams are actually still active.Fix this by tracking per-stream state using:
-enabled_streams(u64 bitmask): tracks which streams are currently enabled
- enabled_count[] (array): per-stream reference counter to support the same stream being enabled/disabled multiple times
Quoted 7 lines
Now each stream is independently enabled/disabled in hardware based on the
enabled_streamsbitmask, while enabled_count[] provides reference counting for scenarios where the same stream is enabled multiple times, such as duplicate cases in the ISI stream.Fixes cf21f328fcaf ("media: nxp: Add i.MX8 ISI driver")
As far as I understand, only newer SoCs support multi-stream for ISI inputs (please tell me if that's incorrect). Those SoCs were not supported when the ISI driver was merged, so I don't think this should be backported to stable kernels. The subject line should then read "Implement per-stream ..." instead of "Fix per-stream ..." and the commit message adapted accordingly.
Quoted 18 lines
Signed-off-by Guoniu Zhou <[email protected]>---
Changes in v2:
- Use fixed-size array forenabled_countinstead of dynamic allocation
- UseBIT_ULL()macro for u64 bitmask operations
- UseMXC_ISI_MAX_STREAMS(64) as loop boundary instead ofnum_sources
- Remove mxc_isi_stream_counters_alloc/free functions--- .../media/platform/nxp/imx8-isi/imx8-isi-core.h | 7 +- .../platform/nxp/imx8-isi/imx8-isi-crossbar.c | 76 ++++++++++++++++------ 2 files changed, 63 insertions(+), 20 deletions(-)drivers/media/platform/nxp/imx8-isi/imx8-isi-core.h +0
-0@@ -184,8 +184,13 @@ struct mxc_isi_dma_buffer {dma_addr_t dma;
drivers/media/platform/nxp/imx8-isi/imx8-isi-core.h struct mxc_isi_dma_buffer { · line 193
Quoted 10 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-core.h +6
-1}; +/* V4L2 subdev max stream ID is 63, need 64 counters (0-63) */ +#define MXC_ISI_MAX_STREAMS 64 +struct mxc_isi_input { -unsigned int enable_count; +u64 enabled_streams; +/* Per-stream reference counter */ +unsigned int enabled_count[MXC_ISI_MAX_STREAMS ];
You can save some memory here by replacing unsigned int with u8. The counter can never go above the number of ISI outputs, as streams can be enabled once only.
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int mxc_isi_crossbar_enable_streams(struct v4l2_subdev *sd, · line 341
Quoted 13 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-core.h +0
-0};struct mxc_isi_crossbar {drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +2
-0@@ -337,6 +337,8 @@ static int mxc_isi_crossbar_enable_streams(struct v4l2_subdev *sd,struct mxc_isi_crossbar *xbar =to_isi_crossbar (sd);struct v4l2_subdev *remote_sd;struct mxc_isi_input *input; +u64 streams_to_enable; +unsigned long stream;
unsigned int is enough, this is a counter, not a bitmask.
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int mxc_isi_crossbar_enable_streams(struct v4l2_subdev *sd, · line 357
Quoted 12 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +0
-5u64 sink_streams;u32 sink_pad;u32 remote_pad;@@ -350,30 +352,47 @@ static int mxc_isi_crossbar_enable_streams(struct v4l2_subdev *sd, input = &xbar->inputs[sink_pad]; -/* -* TODO: Track per-stream enable counts to support multiplexed -* streams. -*/ -if (!input->enable_count) {
Let's keep a short comment here:
/* Enable the gasket when the first stream is enabled for this input. */
Quoted 23 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +10
-4+if (!input->enabled_streams) { ret =mxc_isi_crossbar_gasket_enable (xbar, state, remote_sd, remote_pad, sink_pad);if (ret)return ret; + } +/* +* Track per-stream enable counts to support multiplexed streams. +* Only enable streams that are not already enabled. +*/ + streams_to_enable = sink_streams & ~input->enabled_streams; + +if (streams_to_enable) { ret =v4l2_subdev_enable_streams (remote_sd, remote_pad, - sink_streams); + streams_to_enable);if (ret) { dev_err(xbar->isi->dev, "failed to enable streams 0x%llx on '%s':%u: %d\n", - sink_streams, remote_sd->name, remote_pad, ret); -mxc_isi_crossbar_gasket_disable (xbar, sink_pad); -return ret;
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int mxc_isi_crossbar_enable_streams(struct v4l2_subdev *sd, · line 382
Quoted 10 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +6
-1+ streams_to_enable, remote_sd->name, remote_pad, ret); + goto err_gasket_disable; } + + input->enabled_streams |= streams_to_enable; } - input->enable_count++; +/* Increment reference count for all requested streams */ +for (stream =0 ; stream <MXC_ISI_MAX_STREAMS ; stream++) {
for (unsigned int stream = 0; stream < MXC_ISI_MAX_STREAMS; stream++) {and drop the local variable above.
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int mxc_isi_crossbar_enable_streams(struct v4l2_subdev *sd, · line 387
Quoted 5 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +5
-0+if (!(sink_streams &BIT_ULL (stream))) +continue ; + + input->enabled_count[stream]++; + }
Each ISI pipe processes a single stream (the driver uses the V4L2_SUBDEV_ROUTING_NO_N_TO_1 validation flag, and patch 2/6 ensures that only stream 0 is accepted on the source side of the crossbar). This means that that, after translating streams_mask to the sink side, sink_streams should have a single bit set, and streams_to_enable will have either 0 or 1 bit set.
If that explanation is correct (please let me know if you disagree), then this loop can be replaced with
input->enabled_count[__ffs64(sink_streams)]++;
Quoted 12 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +6
-0return 0 ; + +err_gasket_disable: +if (!input->enabled_streams) +mxc_isi_crossbar_gasket_disable (xbar, sink_pad); + +return ret; } static int mxc_isi_crossbar_disable_streams(struct v4l2_subdev *sd,
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int mxc_isi_crossbar_disable_streams(struct v4l2_subdev *sd, · line 406
Quoted 5 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +2
-0@@ -383,6 +402,8 @@ static int mxc_isi_crossbar_disable_streams(struct v4l2_subdev *sd,struct mxc_isi_crossbar *xbar =to_isi_crossbar (sd);struct v4l2_subdev *remote_sd;struct mxc_isi_input *input; +u64 streams_to_disable =0 ; +unsigned long stream;
unsigned int here too.
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int mxc_isi_crossbar_disable_streams(struct v4l2_subdev *sd, · line 424
Quoted 13 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +5
-1u64 sink_streams;u32 sink_pad;u32 remote_pad;@@ -396,19 +417,36 @@ static int mxc_isi_crossbar_disable_streams(struct v4l2_subdev *sd, input = &xbar->inputs[sink_pad]; - input->enable_count--; +/* +* Decrease the enable count for each stream. Only disable streams +* whose count reaches zero. +*/ +for (stream =0 ; stream <MXC_ISI_MAX_STREAMS ; stream++) {
for (unsigned int stream = 0; stream < MXC_ISI_MAX_STREAMS; stream++) {Quoted 7 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +2
-4+if (!(sink_streams &BIT_ULL (stream))) +continue ; -if (!input->enable_count) { - ret =v4l2_subdev_disable_streams (remote_sd, remote_pad, - sink_streams); -if (ret)
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int mxc_isi_crossbar_disable_streams(struct v4l2_subdev *sd, · line 433
Quoted 10 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +4
-4-dev_err (xbar->isi->dev, -"failed to disable streams 0x%llx on '%s':%u: %d\n" , - sink_streams, remote_sd->name, remote_pad, ret); +if (!(input->enabled_streams &BIT_ULL (stream))) +continue ; -mxc_isi_crossbar_gasket_disable (xbar, sink_pad); +if (--input->enabled_count[stream] ==0 ) + streams_to_disable |=BIT_ULL (stream); }
Similarly here, the whole loop could be replaced by
if (--input->enabled_count[__ffs64(stream)] == 0) streams_to_disable |= BIT_ULL(stream);
Another option is to store the information in the mxc_isi_pipe structure in the form of two new fields:
unsigned int input; unsigned int input_stream;
The fields would be set to the input index and the sink stream mask in mxc_isi_crossbar_enable_streams():
struct mxc_isi_pipe *pipe = &xbar->isi->pipes[pad - xbar->num_sources]; ... pipe->input = sink_pad; pipe->input_stream = sink_streams;
In this function you would then do (completely untested)
struct mxc_isi_pipe *pipe = &xbar->isi->pipes[pad - xbar->num_sources];
...
pipe->input = 0;
pipe->input_stream = 0;
/*
* Check if any other pipe receives the same input stream. If so we
* can't disable it yet, so return immediately.
*/
for (i = 0; i < xbar->isi->pdata->num_channels; ++i) {
struct mxc_isi_pipe *pipe = &xbar->isi->pipes[i];
if (pipe->input == sink_pad &&
pipe->input_stream == sink_streams)
return 0;
}
input->enabled_streams &= ~sink_streams;The advantage of this is that we won't have to add a 64 entries array to each input. I'm not too concerned about the memory usage (even if reducing memory usage is always nice, more for the improvement in cache locality than for the memory saving itself), but the 64 entries feel a bit arbitrary.
What do you think is best ?
Quoted 18 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +15
-0+if (!streams_to_disable) +return 0 ; + + ret =v4l2_subdev_disable_streams (remote_sd, remote_pad, + streams_to_disable); +if (ret) +dev_err (xbar->isi->dev, +"failed to disable streams 0x%llx on '%s':%u: %d\n" , + streams_to_disable, remote_sd->name, remote_pad, ret); + + input->enabled_streams &= ~streams_to_disable; + +if (!input->enabled_streams) +mxc_isi_crossbar_gasket_disable (xbar, sink_pad); +return ret; }
Signature · 3 lines
Regards, Laurent Pinchart
RE: [PATCH v2 3/6] media: nxp: imx8-isi: Fix per-stream reference counting for multiplexed streams
Hi Laurent,
Thanks for the review.
Quoted 171 lines
-----Original Message-----
From: Laurent Pinchart <[email protected]>
Sent: Tuesday, July 21, 2026 5:04 AM
To: G.N. Zhou (OSS) <[email protected]>
Cc: Mauro Carvalho Chehab <[email protected]>; Frank Li <[email protected]>; Sascha Hauer <[email protected]>; Pengutronix Kernel Team <[email protected]>; Fabio Estevam <[email protected]>; [email protected]; Stefan Riedmüller <[email protected]>; Jacopo Mondi <[email protected]>; Loic Poulain <[email protected]>; Bryan O'Donoghue <[email protected]>; Aisheng Dong <[email protected]>; G.N. Zhou <[email protected]>; [email protected]; [email protected]; linux-arm- [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH v2 3/6] media: nxp: imx8-isi: Fix per-stream reference counting for multiplexed streamsQuoted 26 lines · Guoniu Zhou
The ISI crossbar fails to properly enable multiple streams from different virtual channels on the same input pad. Only the first stream gets enabled in hardware, subsequent streams are silently ignored.
The driver uses a single
enable_countper input to track the input state. Whenenable_countis non-zero, the code assumes the input is already active and skips callingv4l2_subdev_enable_streams()for additional streams:Call 1:
enable_streams(stream 0)-> enable_count == 0, enable gasket and stream 0 in hardware -> enable_count = 1Call 2:
enable_streams(stream 1)-> enable_count == 1, skip hardware enable (BUG!) -> enable_count = 2 -> stream 1 never gets enabledSimilarly on disable, when
enable_countreaches zero, ALL streams are disabled regardless of which streams are actually still active.Fix this by tracking per-stream state using:
-enabled_streams(u64 bitmask): tracks which streams are currently enabled
- enabled_count[] (array): per-stream reference counter to support the same stream being enabled/disabled multiple timesQuoted 7 lines
Now each stream is independently enabled/disabled in hardware based on the
enabled_streamsbitmask, while enabled_count[] provides reference counting for scenarios where the same stream is enabled multiple times, such as duplicate cases in the ISI stream.Fixes cf21f328fcaf ("media: nxp: Add i.MX8 ISI driver")As far as I understand, only newer SoCs support multi-stream for ISI inputs (please tell me if that's incorrect). Those SoCs were not supported when the ISI driver was merged, so I don't think this should be backported to stable kernels. The subject line should then read "Implement per-stream ..." instead of "Fix per-stream ..." and the commit message adapted accordingly.
Quoted 19 lines
Signed-off-by Guoniu Zhou <[email protected]>---
Changes in v2:
- Use fixed-size array forenabled_countinstead of dynamic allocation
- UseBIT_ULL()macro for u64 bitmask operations
- UseMXC_ISI_MAX_STREAMS(64) as loop boundary instead ofnum_sources
- Remove mxc_isi_stream_counters_alloc/free functions--- .../media/platform/nxp/imx8-isi/imx8-isi-core.h | 7 +- .../platform/nxp/imx8-isi/imx8-isi-crossbar.c | 76 ++++++++++++++++------ 2 files changed, 63 insertions(+), 20 deletions(-)drivers/media/platform/nxp/imx8-isi/imx8-isi-core.h +0
-0@@ -184,8 +184,13 @@ struct mxc_isi_dma_buffer {dma_addr_t dma;drivers/media/platform/nxp/imx8-isi/imx8-isi-core.h struct mxc_isi_dma_buffer { · line 193
Quoted 10 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-core.h +6
-1}; +/* V4L2 subdev max stream ID is 63, need 64 counters (0-63) */ +#define MXC_ISI_MAX_STREAMS 64 +struct mxc_isi_input { -unsigned int enable_count; +u64 enabled_streams; +/* Per-stream reference counter */ +unsigned int enabled_count[MXC_ISI_MAX_STREAMS];You can save some memory here by replacing unsigned int with u8. The counter can never go above the number of ISI outputs, as streams can be enabled once only.
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int mxc_isi_crossbar_enable_streams(struct
Quoted 9 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-core.h +0
-0};struct mxc_isi_crossbar {drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +0
-0@@ -337,6 +337,8 @@ static int mxc_isi_crossbar_enable_streams(struct
v4l2_subdev*sd,drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int mxc_isi_crossbar_enable_streams(struct v4l2_subdev *sd, · line 341
Quoted 5 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +2
-0struct mxc_isi_crossbar *xbar =to_isi_crossbar (sd);struct v4l2_subdev *remote_sd;struct mxc_isi_input *input; +u64 streams_to_enable; +unsigned long stream;unsigned int is enough, this is a counter, not a bitmask.
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int · line 358
Quoted 13 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +0
-5u64 sink_streams;u32 sink_pad;u32 remote_pad;@@ -350,30 +352,47 @@ static int mxc_isi_crossbar_enable_streams(struct v4l2_subdev *sd, input = &xbar->inputs[sink_pad]; -/* -* TODO: Track per-stream enable counts to support multiplexed -* streams. -*/ -if (!input->enable_count) {Let's keep a short comment here:
/* Enable the gasket when the first stream is enabled for this input. */Quoted 10 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +5
-0+if (!input->enabled_streams) { ret =mxc_isi_crossbar_gasket_enable (xbar, state, remote_sd, remote_pad, sink_pad);if (ret)return ret; + } +/* +* Track per-stream enable counts to support multiplexed streams. +* Only enable streams that are not already enabled. drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int mxc_isi_crossbar_enable_streams(struct v4l2_subdev *sd,
Quoted 10 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +5
-1+*/ + streams_to_enable = sink_streams & ~input->enabled_streams; + +if (streams_to_enable) { ret =v4l2_subdev_enable_streams (remote_sd, remote_pad, - sink_streams); + streams_to_enable);if (ret) { dev_err(xbar->isi->dev, "failed to enable streams 0x%llx on'%s':%u: %d\n",
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int · line 369
Quoted 1 line
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +0
-1- sink_streams, remote_sd->name, remote_pad,ret);
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int mxc_isi_crossbar_enable_streams(struct v4l2_subdev *sd,
Quoted 3 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +1
-2-mxc_isi_crossbar_gasket_disable (xbar, sink_pad); -return ret; + streams_to_enable, remote_sd->name,
remote_pad, ret);drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int mxc_isi_crossbar_enable_streams(struct v4l2_subdev *sd, · line 382
Quoted 9 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +5
-1+ goto err_gasket_disable; } + + input->enabled_streams |= streams_to_enable; } - input->enable_count++; +/* Increment reference count for all requested streams */ +for (stream =0 ; stream <MXC_ISI_MAX_STREAMS ; stream++) {for (unsigned int stream = 0; stream < MXC_ISI_MAX_STREAMS;stream++) {
and drop the local variable above.
Quoted 11 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int mxc_isi_crossbar_enable_streams(struct v4l2_subdev *sd, · line 387
Quoted 5 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +5
-0+if (!(sink_streams &BIT_ULL (stream))) +continue ; + + input->enabled_count[stream]++; + }Each ISI pipe processes a single stream (the driver uses the
V4L2_SUBDEV_ROUTING_NO_N_TO_1validation flag, and patch 2/6 ensures that only stream 0 is accepted on the source side of the crossbar). This means that that, after translatingstreams_maskto the sink side,sink_streamsshould have a single bit set, andstreams_to_enablewill have either 0 or 1 bit set.
Yes, you are absolutely correct.
Quoted 112 lines
If that explanation is correct (please let me know if you disagree), then this loop can be replaced with
input->enabled_count[__ffs64(sink_streams)]++;drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int mxc_isi_crossbar_disable_streams(struct
Quoted 12 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +6
-0return 0 ; + +err_gasket_disable: +if (!input->enabled_streams) +mxc_isi_crossbar_gasket_disable (xbar, sink_pad); + +return ret; } static int mxc_isi_crossbar_disable_streams(struct v4l2_subdev *sd,@@ -383,6 +402,8 @@ static int mxc_isi_crossbar_disable_streams(struct
v4l2_subdev*sd,drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int mxc_isi_crossbar_disable_streams(struct v4l2_subdev *sd, · line 406
Quoted 5 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +2
-0struct mxc_isi_crossbar *xbar =to_isi_crossbar (sd);struct v4l2_subdev *remote_sd;struct mxc_isi_input *input; +u64 streams_to_disable =0 ; +unsigned long stream;unsigned int here too.
Quoted 4 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +0
-0u64 sink_streams;u32 sink_pad;u32 remote_pad;drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int · line 425
Quoted 10 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +5
-1@@ -396,19 +417,36 @@ static int mxc_isi_crossbar_disable_streams(struct v4l2_subdev *sd, input = &xbar->inputs[sink_pad]; - input->enable_count--; +/* +* Decrease the enable count for each stream. Only disable streams +* whose count reaches zero. +*/ +for (stream =0 ; stream <MXC_ISI_MAX_STREAMS ; stream++) {for (unsigned int stream = 0; stream < MXC_ISI_MAX_STREAMS;stream++) {
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int mxc_isi_crossbar_disable_streams(struct v4l2_subdev *sd,
Quoted 9 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +2
-6+if (!(sink_streams &BIT_ULL (stream))) +continue ; -if (!input->enable_count) { - ret =v4l2_subdev_disable_streams (remote_sd, remote_pad, - sink_streams); -if (ret) - dev_err(xbar->isi->dev, - "failed to disable streams 0x%llx on'%s':%u: %d\n",
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int · line 408
Quoted 1 line
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +0
-1- sink_streams, remote_sd->name, remote_pad,ret);
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int mxc_isi_crossbar_disable_streams(struct v4l2_subdev *sd, · line 433
Quoted 7 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +4
-1+if (!(input->enabled_streams &BIT_ULL (stream))) +continue ; -mxc_isi_crossbar_gasket_disable (xbar, sink_pad); +if (--input->enabled_count[stream] ==0 ) + streams_to_disable |=BIT_ULL (stream); }Similarly here, the whole loop could be replaced by
if (--input->enabled_count[__ffs64(stream)] == 0) streams_to_disable |= BIT_ULL(stream);Another option is to store the information in the
mxc_isi_pipestructure in the form of two new fields:unsigned int input; unsigned int input_stream;The fields would be set to the input index and the sink stream mask in
mxc_isi_crossbar_enable_streams():struct mxc_isi_pipe *pipe = &xbar->isi->pipes[pad - xbar-drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int · line 435
Quoted 1 line
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +0
-0num_sources];... pipe->input = sink_pad; pipe->input_stream = sink_streams;In this function you would then do (completely untested)
struct mxc_isi_pipe *pipe = &xbar->isi->pipes[pad - xbar-drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int · line 436
Quoted 1 line
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +0
-0num_sources];... pipe->input = 0; pipe->input_stream = 0; /* * Check if any other pipe receives the same input stream. If so we * can't disable it yet, so return immediately. */ for (i = 0; i < xbar->isi->pdata->num_channels; ++i) { struct mxc_isi_pipe *pipe = &xbar->isi->pipes[i]; if (pipe->input == sink_pad && pipe->input_stream == sink_streams) return 0; } input->enabled_streams &= ~sink_streams;
Quoted 6 lines
The advantage of this is that we won't have to add a 64 entries array to each input. I'm not too concerned about the memory usage (even if reducing memory usage is always nice, more for the improvement in cache locality than for the memory saving itself), but the 64 entries feel a bit arbitrary.
What do you think is best ?
Thank you for the detailed review and suggestions!
I agree that the second approach is better. Storing the input and input_stream directly in the mxc_isi_pipe structure is cleaner and avoids the arbitrary 64 entries array. The improved cache locality is also a nice benefit.
I will adopt this approach in the next version.
Thanks again for the suggestion!
Quoted 25 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c static int mxc_isi_crossbar_disable_streams(struct v4l2_subdev *sd,
Quoted 10 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +9
-0+if (!streams_to_disable) +return 0 ; + + ret =v4l2_subdev_disable_streams (remote_sd, remote_pad, + streams_to_disable); +if (ret) + dev_err(xbar->isi->dev, + "failed to disable streams 0x%llx on '%s':%u: %d\n", + streams_to_disable, remote_sd->name, remote_pad,ret);
Quoted 9 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-crossbar.c +6
-0+ + input->enabled_streams &= ~streams_to_disable; + +if (!input->enabled_streams) +mxc_isi_crossbar_gasket_disable (xbar, sink_pad); +return ret;}
--
Regards,Laurent Pinchart
PATCH v2 4/6 media: nxp: imx8-isi: Add 16-bit raw Bayer format support
In reply to Guoniu ZhouFrom: Laurentiu Palcu <[email protected]>
Add support for 16-bit raw Bayer formats (SBGGR16, SGBRG16, SGRBG16, SRGGB16) to both the pipeline subdev and video capture interface.
These formats are commonly used by high-end image sensors that output 16-bit raw data, enabling the ISI to process and capture full dynamic range from such sensors.
---
Changes in v2:
- Add Reviewed-by tag from Frank Li
--- .../media/platform/nxp/imx8-isi/imx8-isi-pipe.c | 24 +++++++++++++++ .../media/platform/nxp/imx8-isi/imx8-isi-video.c | 36 ++++++++++++++++++++++ 2 files changed, 60 insertions(+)
drivers/media/platform/nxp/imx8-isi/imx8-isi-pipe.c +24 -0
@@ -179,6 +179,30 @@ static const struct mxc_isi_bus_format_info mxc_isi_bus_formats[] = { .pads = BIT(MXC_ISI_PIPE_PAD_SINK) | BIT(MXC_ISI_PIPE_PAD_SOURCE), .encoding = MXC_ISI_ENC_RAW, + }, { + .mbus_code = MEDIA_BUS_FMT_SBGGR16_1X16, + .output = MEDIA_BUS_FMT_SBGGR16_1X16, + .pads = BIT(MXC_ISI_PIPE_PAD_SINK) + | BIT(MXC_ISI_PIPE_PAD_SOURCE), + .encoding = MXC_ISI_ENC_RAW, + }, { + .mbus_code = MEDIA_BUS_FMT_SGBRG16_1X16, + .output = MEDIA_BUS_FMT_SGBRG16_1X16, + .pads = BIT(MXC_ISI_PIPE_PAD_SINK) + | BIT(MXC_ISI_PIPE_PAD_SOURCE), + .encoding = MXC_ISI_ENC_RAW, + }, { + .mbus_code = MEDIA_BUS_FMT_SGRBG16_1X16, + .output = MEDIA_BUS_FMT_SGRBG16_1X16, + .pads = BIT(MXC_ISI_PIPE_PAD_SINK) + | BIT(MXC_ISI_PIPE_PAD_SOURCE), + .encoding = MXC_ISI_ENC_RAW, + }, { + .mbus_code = MEDIA_BUS_FMT_SRGGB16_1X16, + .output = MEDIA_BUS_FMT_SRGGB16_1X16, + .pads = BIT(MXC_ISI_PIPE_PAD_SINK) + | BIT(MXC_ISI_PIPE_PAD_SOURCE), + .encoding = MXC_ISI_ENC_RAW, }, /* JPEG */ {
drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c +36 -0
@@ -356,6 +356,42 @@ static const struct mxc_isi_format_info mxc_isi_formats[] = { .color_planes = 1, .depth = { 16 }, .encoding = MXC_ISI_ENC_RAW, + }, { + .mbus_code = MEDIA_BUS_FMT_SBGGR16_1X16, + .fourcc = V4L2_PIX_FMT_SBGGR16, + .type = MXC_ISI_VIDEO_CAP, + .isi_out_format = CHNL_IMG_CTRL_FORMAT_RAW16, + .mem_planes = 1, + .color_planes = 1, + .depth = { 16 }, + .encoding = MXC_ISI_ENC_RAW, + }, { + .mbus_code = MEDIA_BUS_FMT_SGBRG16_1X16, + .fourcc = V4L2_PIX_FMT_SGBRG16, + .type = MXC_ISI_VIDEO_CAP, + .isi_out_format = CHNL_IMG_CTRL_FORMAT_RAW16, + .mem_planes = 1, + .color_planes = 1, + .depth = { 16 }, + .encoding = MXC_ISI_ENC_RAW, + }, { + .mbus_code = MEDIA_BUS_FMT_SGRBG16_1X16, + .fourcc = V4L2_PIX_FMT_SGRBG16, + .type = MXC_ISI_VIDEO_CAP, + .isi_out_format = CHNL_IMG_CTRL_FORMAT_RAW16, + .mem_planes = 1, + .color_planes = 1, + .depth = { 16 }, + .encoding = MXC_ISI_ENC_RAW, + }, { + .mbus_code = MEDIA_BUS_FMT_SRGGB16_1X16, + .fourcc = V4L2_PIX_FMT_SRGGB16, + .type = MXC_ISI_VIDEO_CAP, + .isi_out_format = CHNL_IMG_CTRL_FORMAT_RAW16, + .mem_planes = 1, + .color_planes = 1, + .depth = { 16 }, + .encoding = MXC_ISI_ENC_RAW, }, /* JPEG */ {
Signature · 1 line
2.34.1
Re: [PATCH v2 4/6] media: nxp: imx8-isi: Add 16-bit raw Bayer format support
Hello Guoniu, Laurentiu,
Thank you for the patch.
Quoted 8 lines · On Mon, Jul 20, 2026 at 11:34:06AM +0800, [email protected] wrote:
From: Laurentiu Palcu <[email protected]>
Add support for 16-bit raw Bayer formats (SBGGR16, SGBRG16, SGRBG16, SRGGB16) to both the pipeline subdev and video capture interface.
These formats are commonly used by high-end image sensors that output 16-bit raw data, enabling the ISI to process and capture full dynamic range from such sensors.
It's not really high-end any more :-) Lots of sensors can output RAW16 today.
Quoted 16 lines
Signed-off-by Laurentiu Palcu <[email protected]>Reviewed-by Frank Li <[email protected]>Signed-off-by Guoniu Zhou <[email protected]>---
Changes in v2:
- Add Reviewed-by tag from Frank Li--- .../media/platform/nxp/imx8-isi/imx8-isi-pipe.c | 24 +++++++++++++++ .../media/platform/nxp/imx8-isi/imx8-isi-video.c | 36 ++++++++++++++++++++++ 2 files changed, 60 insertions(+)drivers/media/platform/nxp/imx8-isi/imx8-isi-pipe.c +0
-0
drivers/media/platform/nxp/imx8-isi/imx8-isi-pipe.c static const struct mxc_isi_bus_format_info mxc_isi_bus_formats[] = { · line 181
Quoted 3 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-pipe.c +0
-0@@ -179,6 +179,30 @@ static const struct mxc_isi_bus_format_info mxc_isi_bus_formats[] = { .pads = BIT(MXC_ISI_PIPE_PAD_SINK) | BIT(MXC_ISI_PIPE_PAD_SOURCE), .encoding = MXC_ISI_ENC_RAW,
While at it, could you please also add support for MEDIA_BUS_FMT_Y16_1X16, for 16-bit monochrome sensors (here and in mxc_isi_formats) ? With that,
Quoted 74 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-pipe.c +24
-0+ }, { + .mbus_code = MEDIA_BUS_FMT_SBGGR16_1X16, + .output = MEDIA_BUS_FMT_SBGGR16_1X16, + .pads = BIT(MXC_ISI_PIPE_PAD_SINK) + | BIT(MXC_ISI_PIPE_PAD_SOURCE), + .encoding = MXC_ISI_ENC_RAW, + }, { + .mbus_code = MEDIA_BUS_FMT_SGBRG16_1X16, + .output = MEDIA_BUS_FMT_SGBRG16_1X16, + .pads = BIT(MXC_ISI_PIPE_PAD_SINK) + | BIT(MXC_ISI_PIPE_PAD_SOURCE), + .encoding = MXC_ISI_ENC_RAW, + }, { + .mbus_code = MEDIA_BUS_FMT_SGRBG16_1X16, + .output = MEDIA_BUS_FMT_SGRBG16_1X16, + .pads = BIT(MXC_ISI_PIPE_PAD_SINK) + | BIT(MXC_ISI_PIPE_PAD_SOURCE), + .encoding = MXC_ISI_ENC_RAW, + }, { + .mbus_code = MEDIA_BUS_FMT_SRGGB16_1X16, + .output = MEDIA_BUS_FMT_SRGGB16_1X16, + .pads = BIT(MXC_ISI_PIPE_PAD_SINK) + | BIT(MXC_ISI_PIPE_PAD_SOURCE), + .encoding = MXC_ISI_ENC_RAW, }, /* JPEG */ {drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c +36
-0@@ -356,6 +356,42 @@ static const struct mxc_isi_format_info mxc_isi_formats[] = { .color_planes = 1, .depth = { 16 }, .encoding = MXC_ISI_ENC_RAW, + }, { + .mbus_code = MEDIA_BUS_FMT_SBGGR16_1X16, + .fourcc = V4L2_PIX_FMT_SBGGR16, + .type = MXC_ISI_VIDEO_CAP, + .isi_out_format = CHNL_IMG_CTRL_FORMAT_RAW16, + .mem_planes = 1, + .color_planes = 1, + .depth = { 16 }, + .encoding = MXC_ISI_ENC_RAW, + }, { + .mbus_code = MEDIA_BUS_FMT_SGBRG16_1X16, + .fourcc = V4L2_PIX_FMT_SGBRG16, + .type = MXC_ISI_VIDEO_CAP, + .isi_out_format = CHNL_IMG_CTRL_FORMAT_RAW16, + .mem_planes = 1, + .color_planes = 1, + .depth = { 16 }, + .encoding = MXC_ISI_ENC_RAW, + }, { + .mbus_code = MEDIA_BUS_FMT_SGRBG16_1X16, + .fourcc = V4L2_PIX_FMT_SGRBG16, + .type = MXC_ISI_VIDEO_CAP, + .isi_out_format = CHNL_IMG_CTRL_FORMAT_RAW16, + .mem_planes = 1, + .color_planes = 1, + .depth = { 16 }, + .encoding = MXC_ISI_ENC_RAW, + }, { + .mbus_code = MEDIA_BUS_FMT_SRGGB16_1X16, + .fourcc = V4L2_PIX_FMT_SRGGB16, + .type = MXC_ISI_VIDEO_CAP, + .isi_out_format = CHNL_IMG_CTRL_FORMAT_RAW16, + .mem_planes = 1, + .color_planes = 1, + .depth = { 16 }, + .encoding = MXC_ISI_ENC_RAW, }, /* JPEG */ {
Signature · 3 lines
Regards, Laurent Pinchart
PATCH v2 5/6 media: nxp: imx8-isi: Correct color map between V4L2 and ISI
In reply to Guoniu ZhouFix the ISI input format for the color map V4L2_PIX_FMT_XBGR32 in memory-to-memory mode.
---
Changes in v2:
- Reword commit description for clarity (Frank Li)
--- drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c +1 -1
@@ -151,7 +151,7 @@ static const struct mxc_isi_format_info mxc_isi_formats[] = { .fourcc = V4L2_PIX_FMT_XBGR32, .type =MXC_ISI_VIDEO_CAP |MXC_ISI_VIDEO_M2M_OUT | MXC_ISI_VIDEO_M2M_CAP, - .isi_in_format = CHNL_MEM_RD_CTRL_IMG_TYPE_XBGR8, + .isi_in_format = CHNL_MEM_RD_CTRL_IMG_TYPE_XRGB8, .isi_out_format = CHNL_IMG_CTRL_FORMAT_XRGB888, .mem_planes = 1, .color_planes = 1,
Signature · 1 line
2.34.1
Re: [PATCH v2 5/6] media: nxp: imx8-isi: Correct color map between V4L2 and ISI
Quoted 7 lines · Guoniu Zhou
Fix the ISI input format for the color map
V4L2_PIX_FMT_XBGR32in memory-to-memory mode.Fixes cf21f328fcaf ("media: nxp: Add i.MX8 ISI driver")Signed-off-by Guoniu Zhou <[email protected]>---
Quoted 24 lines
Changes in v2:
- Reword commit description for clarity (Frank Li)--- drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c +1
-2@@ -151,7 +151,7 @@ static const struct mxc_isi_format_info mxc_isi_formats[] = { .fourcc = V4L2_PIX_FMT_XBGR32, .type =MXC_ISI_VIDEO_CAP |MXC_ISI_VIDEO_M2M_OUT | MXC_ISI_VIDEO_M2M_CAP, - .isi_in_format = CHNL_MEM_RD_CTRL_IMG_TYPE_XBGR8, + .isi_in_format = CHNL_MEM_RD_CTRL_IMG_TYPE_XRGB8, .isi_out_format = CHNL_IMG_CTRL_FORMAT_XRGB888, .mem_planes = 1, .color_planes = 1, --2.34.1
Re: [PATCH v2 5/6] media: nxp: imx8-isi: Correct color map between V4L2 and ISI
In reply to Guoniu ZhouHi Guoniu,
Quoted 6 lines · Guoniu Zhou
Fix the ISI input format for the color map
V4L2_PIX_FMT_XBGR32in memory-to-memory mode.Fixes cf21f328fcaf ("media: nxp: Add i.MX8 ISI driver")Signed-off-by Guoniu Zhou <[email protected]>
Tentatively,
I'm running a build to test this change.
Quoted 20 lines
---
Changes in v2:
- Reword commit description for clarity (Frank Li)--- drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c +1
-1@@ -151,7 +151,7 @@ static const struct mxc_isi_format_info mxc_isi_formats[] = { .fourcc = V4L2_PIX_FMT_XBGR32, .type =MXC_ISI_VIDEO_CAP |MXC_ISI_VIDEO_M2M_OUT | MXC_ISI_VIDEO_M2M_CAP, - .isi_in_format = CHNL_MEM_RD_CTRL_IMG_TYPE_XBGR8, + .isi_in_format = CHNL_MEM_RD_CTRL_IMG_TYPE_XRGB8, .isi_out_format = CHNL_IMG_CTRL_FORMAT_XRGB888, .mem_planes = 1, .color_planes = 1,
Signature · 3 lines
Regards, Laurent Pinchart
Re: [PATCH v2 5/6] media: nxp: imx8-isi: Correct color map between V4L2 and ISI
Quoted 8 lines · Laurent Pinchart
Quoted 6 lines · Guoniu Zhou
Fix the ISI input format for the color map
V4L2_PIX_FMT_XBGR32in memory-to-memory mode.Fixes cf21f328fcaf ("media: nxp: Add i.MX8 ISI driver")Signed-off-by Guoniu Zhou <[email protected]>
Quoted 5 lines
Tentatively,
Reviewed-by Laurent Pinchart <[email protected]>I'm running a build to test this change.
Quoted 20 lines
Quoted 20 lines
---
Changes in v2:
- Reword commit description for clarity (Frank Li)--- drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c +1
-1@@ -151,7 +151,7 @@ static const struct mxc_isi_format_info mxc_isi_formats[] = { .fourcc = V4L2_PIX_FMT_XBGR32, .type =MXC_ISI_VIDEO_CAP |MXC_ISI_VIDEO_M2M_OUT | MXC_ISI_VIDEO_M2M_CAP, - .isi_in_format = CHNL_MEM_RD_CTRL_IMG_TYPE_XBGR8, + .isi_in_format = CHNL_MEM_RD_CTRL_IMG_TYPE_XRGB8, .isi_out_format = CHNL_IMG_CTRL_FORMAT_XRGB888, .mem_planes = 1, .color_planes = 1,
Signature · 3 lines
Regards, Laurent Pinchart
PATCH v2 6/6 media: nxp: imx8-isi: Add additional 32-bit RGB format support
In reply to Guoniu ZhouAdd support for additional 32-bit RGB pixel formats (BGRA32, RGBA32, BGRX32, RGBX32, ARGB2101010) and extend existing ABGR32 format with full memory-to-memory capabilities to meet Android requirements.
All formats support capture, M2M input, and M2M output operations, enabling complete format conversion pipelines.
---
Changes in v2:
- Add Reviewed-by tag from Frank Li
--- .../media/platform/nxp/imx8-isi/imx8-isi-video.c | 59 +++++++++++++++++++++- 1 file changed, 58 insertions(+), 1 deletion(-)
drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c +58 -1
@@ -160,12 +160,69 @@ static const struct mxc_isi_format_info mxc_isi_formats[] = { }, { .mbus_code = MEDIA_BUS_FMT_RGB888_1X24, .fourcc = V4L2_PIX_FMT_ABGR32, - .type = MXC_ISI_VIDEO_CAP | MXC_ISI_VIDEO_M2M_CAP, + .type =MXC_ISI_VIDEO_CAP |MXC_ISI_VIDEO_M2M_OUT + | MXC_ISI_VIDEO_M2M_CAP, + .isi_in_format = CHNL_MEM_RD_CTRL_IMG_TYPE_XRGB8, .isi_out_format = CHNL_IMG_CTRL_FORMAT_ARGB8888, .mem_planes = 1, .color_planes = 1, .depth = { 32 }, .encoding = MXC_ISI_ENC_RGB, + }, { + .mbus_code = MEDIA_BUS_FMT_RGB888_1X24, + .fourcc = V4L2_PIX_FMT_BGRA32, + .type =MXC_ISI_VIDEO_CAP |MXC_ISI_VIDEO_M2M_OUT + | MXC_ISI_VIDEO_M2M_CAP, + .isi_in_format = CHNL_MEM_RD_CTRL_IMG_TYPE_RGBX8, + .isi_out_format = CHNL_IMG_CTRL_FORMAT_RGBA8888, + .mem_planes = 1, + .color_planes = 1, + .depth = { 32 }, + .encoding = MXC_ISI_ENC_RGB, + }, { + .mbus_code = MEDIA_BUS_FMT_RGB888_1X24, + .fourcc = V4L2_PIX_FMT_RGBA32, + .type =MXC_ISI_VIDEO_CAP |MXC_ISI_VIDEO_M2M_OUT + | MXC_ISI_VIDEO_M2M_CAP, + .isi_in_format = CHNL_MEM_RD_CTRL_IMG_TYPE_XBGR8, + .isi_out_format = CHNL_IMG_CTRL_FORMAT_ABGR8888, + .mem_planes = 1, + .color_planes = 1, + .depth = { 32 }, + .encoding = MXC_ISI_ENC_RGB, + }, { + .mbus_code = MEDIA_BUS_FMT_RGB888_1X24, + .fourcc = V4L2_PIX_FMT_BGRX32, + .type =MXC_ISI_VIDEO_CAP |MXC_ISI_VIDEO_M2M_OUT + | MXC_ISI_VIDEO_M2M_CAP, + .isi_in_format = CHNL_MEM_RD_CTRL_IMG_TYPE_RGBX8, + .isi_out_format = CHNL_IMG_CTRL_FORMAT_RGBX888, + .mem_planes = 1, + .color_planes = 1, + .depth = { 32 }, + .encoding = MXC_ISI_ENC_RGB, + }, { + .mbus_code = MEDIA_BUS_FMT_RGB888_1X24, + .fourcc = V4L2_PIX_FMT_RGBX32, + .type =MXC_ISI_VIDEO_CAP |MXC_ISI_VIDEO_M2M_OUT + | MXC_ISI_VIDEO_M2M_CAP, + .isi_in_format = CHNL_MEM_RD_CTRL_IMG_TYPE_XBGR8, + .isi_out_format = CHNL_IMG_CTRL_FORMAT_XBGR888, + .mem_planes = 1, + .color_planes = 1, + .depth = { 32 }, + .encoding = MXC_ISI_ENC_RGB, + }, { + .mbus_code = MEDIA_BUS_FMT_RGB888_1X24, + .fourcc = V4L2_PIX_FMT_ARGB2101010, + .type =MXC_ISI_VIDEO_CAP |MXC_ISI_VIDEO_M2M_OUT + | MXC_ISI_VIDEO_M2M_CAP, + .isi_in_format = CHNL_MEM_RD_CTRL_IMG_TYPE_A2RGB10, + .isi_out_format = CHNL_IMG_CTRL_FORMAT_A2RGB10, + .mem_planes = 1, + .color_planes = 1, + .depth = { 32 }, + .encoding = MXC_ISI_ENC_RGB, },/* * RAW formats
Signature · 1 line
2.34.1
Re: [PATCH v2 6/6] media: nxp: imx8-isi: Add additional 32-bit RGB format support
Hi Guoniu, Robert,
Thank you for the patch.
Quoted 3 lines · Guoniu Zhou
Add support for additional 32-bit RGB pixel formats (BGRA32, RGBA32, BGRX32, RGBX32, ARGB2101010) and extend existing ABGR32 format with full memory-to-memory capabilities to meet Android requirements.
What are those Android requirements ?
Quoted 18 lines
All formats support capture, M2M input, and M2M output operations, enabling complete format conversion pipelines.
Signed-off-by Robert Chiras <[email protected]>Reviewed-by Frank Li <[email protected]>Signed-off-by Guoniu Zhou <[email protected]>---
Changes in v2:
- Add Reviewed-by tag from Frank Li--- .../media/platform/nxp/imx8-isi/imx8-isi-video.c | 59 +++++++++++++++++++++- 1 file changed, 58 insertions(+), 1 deletion(-)drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c +0
-0
drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c static const struct mxc_isi_format_info mxc_isi_formats[] = { · line 165
Quoted 7 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c +3
-1@@ -160,12 +160,69 @@ static const struct mxc_isi_format_info mxc_isi_formats[] = { }, { .mbus_code = MEDIA_BUS_FMT_RGB888_1X24, .fourcc = V4L2_PIX_FMT_ABGR32, - .type = MXC_ISI_VIDEO_CAP | MXC_ISI_VIDEO_M2M_CAP, + .type =MXC_ISI_VIDEO_CAP |MXC_ISI_VIDEO_M2M_OUT + | MXC_ISI_VIDEO_M2M_CAP, + .isi_in_format = CHNL_MEM_RD_CTRL_IMG_TYPE_XRGB8,
This doesn't seem right. The value is the same as for V4L2_PIX_FMT_XBGR32. As far as I understand, the ISI will ignore the alpha bits (MSBs in the 32-bit data).
This is why the V4L2_PIX_FMT_ABGR32 format doesn't setMXC_ISI_VIDEO_M2M_OUT: to be supported as an output format (input to the ISI), we would need a hardware mode where the alpha bits are read from memory and used by the ISI. I recommend dropping this part of the patch.
drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c static const struct mxc_isi_format_info mxc_isi_formats[] = { · line 176
Quoted 11 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c +6
-0.isi_out_format = CHNL_IMG_CTRL_FORMAT_ARGB8888, .mem_planes = 1, .color_planes = 1, .depth = { 32 }, .encoding = MXC_ISI_ENC_RGB, + }, { + .mbus_code = MEDIA_BUS_FMT_RGB888_1X24, + .fourcc = V4L2_PIX_FMT_BGRA32, + .type =MXC_ISI_VIDEO_CAP |MXC_ISI_VIDEO_M2M_OUT + | MXC_ISI_VIDEO_M2M_CAP, + .isi_in_format = CHNL_MEM_RD_CTRL_IMG_TYPE_RGBX8,
Same comment here, this format should not have MXC_ISI_VIDEO_M2M_OUT set.
drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c static const struct mxc_isi_format_info mxc_isi_formats[] = { · line 187
Quoted 11 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c +11
-0+ .isi_out_format = CHNL_IMG_CTRL_FORMAT_RGBA8888, + .mem_planes = 1, + .color_planes = 1, + .depth = { 32 }, + .encoding = MXC_ISI_ENC_RGB, + }, { + .mbus_code = MEDIA_BUS_FMT_RGB888_1X24, + .fourcc = V4L2_PIX_FMT_RGBA32, + .type =MXC_ISI_VIDEO_CAP |MXC_ISI_VIDEO_M2M_OUT + | MXC_ISI_VIDEO_M2M_CAP, + .isi_in_format = CHNL_MEM_RD_CTRL_IMG_TYPE_XBGR8,
Same here too.
Quoted 28 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c +28
-0+ .isi_out_format = CHNL_IMG_CTRL_FORMAT_ABGR8888, + .mem_planes = 1, + .color_planes = 1, + .depth = { 32 }, + .encoding = MXC_ISI_ENC_RGB, + }, { + .mbus_code = MEDIA_BUS_FMT_RGB888_1X24, + .fourcc = V4L2_PIX_FMT_BGRX32, + .type =MXC_ISI_VIDEO_CAP |MXC_ISI_VIDEO_M2M_OUT + | MXC_ISI_VIDEO_M2M_CAP, + .isi_in_format = CHNL_MEM_RD_CTRL_IMG_TYPE_RGBX8, + .isi_out_format = CHNL_IMG_CTRL_FORMAT_RGBX888, + .mem_planes = 1, + .color_planes = 1, + .depth = { 32 }, + .encoding = MXC_ISI_ENC_RGB, + }, { + .mbus_code = MEDIA_BUS_FMT_RGB888_1X24, + .fourcc = V4L2_PIX_FMT_RGBX32, + .type =MXC_ISI_VIDEO_CAP |MXC_ISI_VIDEO_M2M_OUT + | MXC_ISI_VIDEO_M2M_CAP, + .isi_in_format = CHNL_MEM_RD_CTRL_IMG_TYPE_XBGR8, + .isi_out_format = CHNL_IMG_CTRL_FORMAT_XBGR888, + .mem_planes = 1, + .color_planes = 1, + .depth = { 32 }, + .encoding = MXC_ISI_ENC_RGB, + }, {
drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c static const struct mxc_isi_format_info mxc_isi_formats[] = { · line 225
Quoted 10 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c +10
-0+ .mbus_code = MEDIA_BUS_FMT_RGB888_1X24, + .fourcc = V4L2_PIX_FMT_ARGB2101010, + .type =MXC_ISI_VIDEO_CAP |MXC_ISI_VIDEO_M2M_OUT + | MXC_ISI_VIDEO_M2M_CAP, + .isi_in_format = CHNL_MEM_RD_CTRL_IMG_TYPE_A2RGB10, + .isi_out_format = CHNL_IMG_CTRL_FORMAT_A2RGB10, + .mem_planes = 1, + .color_planes = 1, + .depth = { 32 }, + .encoding = MXC_ISI_ENC_RGB,
The rest looks good. I have tested all the new formats, both on the input and output side, and have not noticed any issue.
Quoted 3 lines
drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c +0
-0},/* * RAW formats
Signature · 3 lines
Regards, Laurent Pinchart
Laurent Pinchart
maintainerQuoted 18 lines · Guoniu Zhou
This series addresses critical bugs in the imx8-isi driver and extends format support for high-end sensors and Android requirements.
Patch 1 fixes a critical stream ID validation bug in the crossbar routing where the validation loop iterates over the wrong routing table, allowing userspace to bypass validation entirely and configure invalid routes.
Patch 2 adds additional stream ID validation to enforce hardware constraints (SOURCE stream must be 0).
Patch 3 fixes a stream reference counting bug in the crossbar that prevents multiple streams from different virtual channels on the same input pad from being enabled correctly. Only the first stream gets enabled in hardware, subsequent streams are silently ignored.
Patch 4 adds support for 16-bit raw Bayer formats (SBGGR16, SGBRG16, SGRBG16, SRGGB16) commonly used by high-end image sensors.
Quoted 8 lines
Patch 5 fixes incorrect color mapping for XBGR32 format in memory-to-memory mode (marked for stable backport).
Patch 6 extends RGB format support by adding BGRA32, RGBA32, BGRX32, RGBX32, and ARGB2101010 formats with full M2M capabilities to meet Android requirements.
Signed-off-by Guoniu Zhou <[email protected]>
I have taken patches 1, 2 and 5 in my tree already ([1]). The CI pipeline is running ([2]). Once it completes successfully, I will send a pull request. You can base the next version of this series (patches 3, 4 and 6) on top of [1].
[1] https://gitlab.freedesktop.org/linux-media/users/pinchartl/-/commit/db30b841e534f6b97008583f6e63c93473f14afc
[2] https://gitlab.freedesktop.org/linux-media/users/pinchartl/-/pipelines/1710408
Quoted 36 lines
---
Changes in v2:
- [1/6] Split v1 patch 1/5 into two patches: this patch fixes the corefor_each_active_route()bug, next patch adds additional stream validation (Frank Li)
- [2/6] New patch split from v1 1/5: adds stream ID validation on top offor_each_active_route()fix (Frank Li)
- [2/6] Remove incorrectsink_streamvalidation
- [3/6] Use fixed-size array forenabled_countinstead of dynamic allocation
- [3/6] UseBIT_ULL()macro for u64 bitmask operations
- [3/6] UseMXC_ISI_MAX_STREAMS(64) as loop boundary instead ofnum_sources
- [3/6] Remove mxc_isi_stream_counters_alloc/free functions
- [4/6] Add Reviewed-by tag from Frank Li
- [5/6] Reword commit description for clarity (Frank Li)
- [6/6] Add Reviewed-by tag from Frank Li
- Link to v1: https://lore.kernel.org/r/[email protected]---
Guoniu Zhou (5):
media: nxp: imx8-isi: Fix stream ID validation bypass in crossbar routing
media: nxp: imx8-isi: Add stream ID validation for crossbar routing
media: nxp: imx8-isi: Fix per-stream reference counting for multiplexed streams
media: nxp: imx8-isi: Correct color map between V4L2 and ISI
media: nxp: imx8-isi: Add additional 32-bit RGB format supportLaurentiu Palcu (1):
media: nxp: imx8-isi: Add 16-bit raw Bayer format support.../media/platform/nxp/imx8-isi/imx8-isi-core.h | 7 +- .../platform/nxp/imx8-isi/imx8-isi-crossbar.c | 92 +++++++++++++++----- .../media/platform/nxp/imx8-isi/imx8-isi-pipe.c | 24 ++++++ .../media/platform/nxp/imx8-isi/imx8-isi-video.c | 97 +++++++++++++++++++++- 4 files changed, 196 insertions(+), 24 deletions(-)---
base-commit:06cb687a5132fcffe624c0070576ab852ac6b568
change-id: 20260626-isi-00f05b044ac9
Signature · 3 lines
Regards, Laurent Pinchart
Laurent Pinchart
maintainerQuoted 28 lines · Laurent Pinchart
Quoted 18 lines · Guoniu Zhou
This series addresses critical bugs in the imx8-isi driver and extends format support for high-end sensors and Android requirements.
Patch 1 fixes a critical stream ID validation bug in the crossbar routing where the validation loop iterates over the wrong routing table, allowing userspace to bypass validation entirely and configure invalid routes.
Patch 2 adds additional stream ID validation to enforce hardware constraints (SOURCE stream must be 0).
Patch 3 fixes a stream reference counting bug in the crossbar that prevents multiple streams from different virtual channels on the same input pad from being enabled correctly. Only the first stream gets enabled in hardware, subsequent streams are silently ignored.
Patch 4 adds support for 16-bit raw Bayer formats (SBGGR16, SGBRG16, SGRBG16, SRGGB16) commonly used by high-end image sensors.
Quoted 8 lines
Patch 5 fixes incorrect color mapping for XBGR32 format in memory-to-memory mode (marked for stable backport).
Patch 6 extends RGB format support by adding BGRA32, RGBA32, BGRX32, RGBX32, and ARGB2101010 formats with full M2M capabilities to meet Android requirements.
Signed-off-by Guoniu Zhou <[email protected]>
Quoted 7 lines
I have taken patches 1, 2 and 5 in my tree already ([1]). The CI pipeline is running ([2]). Once it completes successfully, I will send a pull request. You can base the next version of this series (patches 3, 4 and 6) on top of [1].
[1] https://gitlab.freedesktop.org/linux-media/users/pinchartl/-/commit/db30b841e534f6b97008583f6e63c93473f14afc
[2] https://gitlab.freedesktop.org/linux-media/users/pinchartl/-/pipelines/1710408
This failed due to a mistake on my side. I've submitted a new pipeline:
https://gitlab.freedesktop.org/linux-media/users/pinchartl/-/pipelines/1710412
Quoted 36 lines
Quoted 36 lines
---
Changes in v2:
- [1/6] Split v1 patch 1/5 into two patches: this patch fixes the corefor_each_active_route()bug, next patch adds additional stream validation (Frank Li)
- [2/6] New patch split from v1 1/5: adds stream ID validation on top offor_each_active_route()fix (Frank Li)
- [2/6] Remove incorrectsink_streamvalidation
- [3/6] Use fixed-size array forenabled_countinstead of dynamic allocation
- [3/6] UseBIT_ULL()macro for u64 bitmask operations
- [3/6] UseMXC_ISI_MAX_STREAMS(64) as loop boundary instead ofnum_sources
- [3/6] Remove mxc_isi_stream_counters_alloc/free functions
- [4/6] Add Reviewed-by tag from Frank Li
- [5/6] Reword commit description for clarity (Frank Li)
- [6/6] Add Reviewed-by tag from Frank Li
- Link to v1: https://lore.kernel.org/r/[email protected]---
Guoniu Zhou (5):
media: nxp: imx8-isi: Fix stream ID validation bypass in crossbar routing
media: nxp: imx8-isi: Add stream ID validation for crossbar routing
media: nxp: imx8-isi: Fix per-stream reference counting for multiplexed streams
media: nxp: imx8-isi: Correct color map between V4L2 and ISI
media: nxp: imx8-isi: Add additional 32-bit RGB format supportLaurentiu Palcu (1):
media: nxp: imx8-isi: Add 16-bit raw Bayer format support.../media/platform/nxp/imx8-isi/imx8-isi-core.h | 7 +- .../platform/nxp/imx8-isi/imx8-isi-crossbar.c | 92 +++++++++++++++----- .../media/platform/nxp/imx8-isi/imx8-isi-pipe.c | 24 ++++++ .../media/platform/nxp/imx8-isi/imx8-isi-video.c | 97 +++++++++++++++++++++- 4 files changed, 196 insertions(+), 24 deletions(-)---
base-commit:06cb687a5132fcffe624c0070576ab852ac6b568
change-id: 20260626-isi-00f05b044ac9
Signature · 3 lines
Regards, Laurent Pinchart