media: rockchip: rkisp2: Add driver for ISP on Rk3588
v3 cover media: rockchip: rkisp2: Add driver for ISP on Rk3588
Hello everyone!
This patch series adds a new rkisp2 driver that aims to support all Rockchip 2.x and 3.x series ISPs. Here we add support for the version on the RK3588, which as far as I'm aware is a 3.0.
Patches 1 and 2 add dt bindings and dt nodes; patches 3~5 add the actual driver.
Notable changes from the RFC version:
- fixed a couple potential page fault storms
- use extensible stats (previously only params were extensible)
- no more subdev pad crops; all cropping has been moved to parameter buffers so that they can be configured synchronously
- the DMA RX sink pads and the main/self path source pads on the ISP subdev have been split into individual pads
- an unused sink pad has been allocated on the ISP subdev for future inline mode support (to prevent uapi from changing)
Changes in v3:
- add explicit padding to uapi structs
- reorder patches so that dt goes last
- move uapi header to a nicer location
This series is based on the latest version (v7.2-rc1) (at the time of posting) of media-commiters/next [0], with some patches on top for testing on the Orange Pi CM5 Base with an imx708. I have a branch [1] that integrates this series on top of said patches and media-committers/next.
Corresponding libcamera patches+branch will be coming soon (within minutes), but since I cannot send and link these simultanously, you will have to search the libcamera mailing list :) I'll be sending a v3 to match the driver v3.
[0] https://gitlab.freedesktop.org/linux-media/media-committers/-/tree/4900cad020c0580dfb1be27776ff10a4ef110cfa
[1] https://git.ideasonboard.com/epaul/linux/src/branch/epaul/v7.2-rc1/rk3588/rkisp2/upstream-v3
To: Mauro Carvalho Chehab <[email protected]>
To: Heiko Stuebner <[email protected]>
To: Dafna Hirschfeld <[email protected]>
To: Laurent Pinchart <[email protected]>
To: Rob Herring <[email protected]>
To: Krzysztof Kozlowski <[email protected]>
To: Conor Dooley <[email protected]>
To: [email protected]
To: [email protected]
To: [email protected]
To: [email protected]
To: [email protected]
To: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
---
Paul Elder (3):
media: rkisp2: Add parameters output video node
media: rkisp2: Add statistics capture video node
dt-bindings: media: Add rockchip rkisp2
Xu Hongfei (2):
media: rockchip: rkisp2: Add rkisp2 driver
arm64: dts: rockchip: add ISP nodes to rk3588
.../devicetree/bindings/media/rockchip-isp2.yaml | 125 + .../userspace-api/media/v4l/meta-formats.rst | 1 + .../userspace-api/media/v4l/metafmt-rkisp2.rst | 112 + MAINTAINERS | 10 + arch/arm64/boot/dts/rockchip/rk3588-base.dtsi | 57 + drivers/media/platform/rockchip/Kconfig | 1 + drivers/media/platform/rockchip/Makefile | 1 + drivers/media/platform/rockchip/rkisp2/Kconfig | 19 + drivers/media/platform/rockchip/rkisp2/Makefile | 14 + .../platform/rockchip/rkisp2/rkisp2-capture.c | 1540 +++++++++ .../media/platform/rockchip/rkisp2/rkisp2-common.c | 180 + .../media/platform/rockchip/rkisp2/rkisp2-common.h | 620 ++++ .../media/platform/rockchip/rkisp2/rkisp2-debug.c | 234 ++ .../media/platform/rockchip/rkisp2/rkisp2-dev.c | 372 +++ .../media/platform/rockchip/rkisp2/rkisp2-dmarx.c | 706 ++++ .../media/platform/rockchip/rkisp2/rkisp2-isp.c | 839 +++++ .../media/platform/rockchip/rkisp2/rkisp2-params.c | 1167 +++++++ .../platform/rockchip/rkisp2/rkisp2-regs-v2x.h | 2625 +++++++++++++++ .../platform/rockchip/rkisp2/rkisp2-regs-v3x.h | 3428 ++++++++++++++++++++ .../media/platform/rockchip/rkisp2/rkisp2-regs.h | 1736 ++++++++++ .../platform/rockchip/rkisp2/rkisp2-resizer.c | 407 +++ .../media/platform/rockchip/rkisp2/rkisp2-stats.c | 509 +++ drivers/media/v4l2-core/v4l2-ioctl.c | 2 + include/uapi/linux/media/rockchip/rkisp2-config.h | 643 ++++ include/uapi/linux/rkisp1-config.h | 1 + include/uapi/linux/videodev2.h | 4 + 26 files changed, 15353 insertions(+)
---
base-commit: 7517ffce3b70c999f6038fc87d4811004d994d25
change-id: 20260826-rkisp2-cd295f7eae4a
Best regards,
Signature · 1 line
Paul Elder <[email protected]>
PATCH v3 1/5 media: rockchip: rkisp2: Add rkisp2 driver
From: Xu Hongfei <[email protected]>
This patch adds the rkisp2 driver, which aims to support all Rockchip ISP 2.x and 3.x versions. This initial version only supports the version of the ISP on the RK3588, which is a 3.0.
The current version of the driver has the following features and limitations:
Supported features:
- Single frame input/output
- mem-to-mem input via DMARX
- Main path and self path capture (scaling and cropping will be added in a subsequent patch)
- Output formats: UYVY, NV12, NV16, NV61, GREY
- YUYV doesn't seem to be supported by hardware
Limitations:
- No inline mode (a sink pad has been reserved to support it in the future without breaking uapi)
- No HDR
- No virtual pipeline (TDM)
- No unite mode (using both ISPs together for one big image)
---
Misc notes:
- NV21 and YUV422 and YVU422 need some debugging
- I haven't tested raw output yet since VICAP can be used on its own for raw output
- Self path is untested
- The features flag isn't really necessary at this stage (since we only support one ISP version) but I wanted to include it already since as soon as we add support for another ISP version we will need it. I've used dual crop as the first feature flag because that is one known difference between rk3588 and rk3568 afaia.
---
Changes in v3:
- mv include/uapi/linux/rkisp2-config.h include/uapi/linux/media/rockchip/rkisp2-config.h
- add MAINTAINERS
Changes in v2:
- fixed 8-bit raw input
- disable DMA input on stop stream (thus fixing a page fault storm)
- split DMA RX pads
- split main/self path pads
- remove cropping and scaling (it has been moved to params)
--- MAINTAINERS | 10 + drivers/media/platform/rockchip/Kconfig | 1 + drivers/media/platform/rockchip/Makefile | 1 + drivers/media/platform/rockchip/rkisp2/Kconfig | 19 + drivers/media/platform/rockchip/rkisp2/Makefile | 11 + .../platform/rockchip/rkisp2/rkisp2-capture.c | 1540 +++++++++ .../media/platform/rockchip/rkisp2/rkisp2-common.c | 180 ++ .../media/platform/rockchip/rkisp2/rkisp2-common.h | 465 +++ .../media/platform/rockchip/rkisp2/rkisp2-debug.c | 220 ++ .../media/platform/rockchip/rkisp2/rkisp2-dev.c | 342 ++ .../media/platform/rockchip/rkisp2/rkisp2-dmarx.c | 706 ++++ .../media/platform/rockchip/rkisp2/rkisp2-isp.c | 826 +++++ .../platform/rockchip/rkisp2/rkisp2-regs-v2x.h | 2588 +++++++++++++++ .../platform/rockchip/rkisp2/rkisp2-regs-v3x.h | 3383 ++++++++++++++++++++ .../media/platform/rockchip/rkisp2/rkisp2-regs.h | 1736 ++++++++++ include/uapi/linux/media/rockchip/rkisp2-config.h | 20 + include/uapi/linux/rkisp1-config.h | 1 + 17 files changed, 12049 insertions(+)
MAINTAINERS +10 -0
@@ -23426,6 +23426,16 @@ F: Documentation/userspace-api/media/v4l/metafmt-rkisp1.rst F: drivers/media/platform/rockchip/rkisp1 F: include/uapi/linux/rkisp1-config.h +ROCKCHIP ISP V2 DRIVER +M: Paul Elder <[email protected]> +L: [email protected] +L: [email protected] +S: Maintained +F: Documentation/devicetree/bindings/media/rockchip-isp2.yaml +F: Documentation/userspace-api/media/v4l/metafmt-rkisp2.rst +F: drivers/media/platform/rockchip/rkisp2 +F: include/uapi/linux/media/rockchip/rkisp2-config.h + ROCKCHIP RK3568 RANDOM NUMBER GENERATOR SUPPORT M: Daniel Golle <[email protected]> M: Aurelien Jarno <[email protected]>
drivers/media/platform/rockchip/Kconfig +1 -0
@@ -6,3 +6,4 @@ source "drivers/media/platform/rockchip/rga/Kconfig" source "drivers/media/platform/rockchip/rkcif/Kconfig" source "drivers/media/platform/rockchip/rkisp1/Kconfig" source "drivers/media/platform/rockchip/rkvdec/Kconfig" +source "drivers/media/platform/rockchip/rkisp2/Kconfig"
drivers/media/platform/rockchip/Makefile +1 -0
@@ -3,3 +3,4 @@ obj-y += rga/ obj-y += rkcif/ obj-y += rkisp1/ obj-y += rkvdec/ +obj-y += rkisp2/
drivers/media/platform/rockchip/rkisp2/Kconfig new file +19 -0
@@ -0,0 +1,19 @@ +# SPDX-License-Identifier: GPL-2.0-only +config VIDEO_ROCKCHIP_ISP2 + tristate "Rockchip Image Signal Processing v2 Unit driver" + depends on V4L_PLATFORM_DRIVERS + depends on VIDEO_DEV && OF + depends on ARCH_ROCKCHIP || COMPILE_TEST + select MEDIA_CONTROLLER + select VIDEO_V4L2_SUBDEV_API + select VIDEOBUF2_DMA_CONTIG + select VIDEOBUF2_VMALLOC + select V4L2_FWNODE + select GENERIC_PHY_MIPI_DPHY + default n + help + Enable this to support the Image Signal Processing (ISP) module + present in RK3588 SoCs. + + To compile this driver as a module, choose M here: the module + will be called rockchip-isp2.
drivers/media/platform/rockchip/rkisp2/Makefile new file +11 -0
@@ -0,0 +1,11 @@ +# SPDX-License-Identifier: GPL-2.0-only + +rockchip-isp2-y := rkisp2-capture.o \ + rkisp2-common.o \ + rkisp2-dev.o \ + rkisp2-dmarx.o \ + rkisp2-isp.o + +rockchip-isp2-$(CONFIG_DEBUG_FS) += rkisp2-debug.o + +obj-$(CONFIG_VIDEO_ROCKCHIP_ISP2) += rockchip-isp2.o
drivers/media/platform/rockchip/rkisp2/rkisp2-capture.c new file +1540 -0
1,540 lines not on this page · raw message
drivers/media/platform/rockchip/rkisp2/rkisp2-common.c new file +180 -0
180 lines not on this page · raw message
drivers/media/platform/rockchip/rkisp2/rkisp2-common.h new file +465 -0
465 lines not on this page · raw message
drivers/media/platform/rockchip/rkisp2/rkisp2-debug.c new file +220 -0
220 lines not on this page · raw message
drivers/media/platform/rockchip/rkisp2/rkisp2-dev.c new file +342 -0
342 lines not on this page · raw message
drivers/media/platform/rockchip/rkisp2/rkisp2-dmarx.c new file +706 -0
199 lines not on this page · raw message
drivers/media/platform/rockchip/rkisp2/rkisp2-isp.c new file +826 -0
826 lines not on this page · raw message
drivers/media/platform/rockchip/rkisp2/rkisp2-regs-v2x.h new file +2588 -0
2,588 lines not on this page · raw message
drivers/media/platform/rockchip/rkisp2/rkisp2-regs-v3x.h new file +3383 -0
3,383 lines not on this page · raw message
drivers/media/platform/rockchip/rkisp2/rkisp2-regs.h new file +1736 -0
1,736 lines not on this page · raw message
include/uapi/linux/media/rockchip/rkisp2-config.h new file +20 -0
Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.47.2
Part of the diff is not shown here. View the raw message.
Re: [PATCH v3 1/5] media: rockchip: rkisp2: Add rkisp2 driver
Quoted 5 lines · Paul Elder
From: Xu Hongfei <[email protected]>
This patch adds the rkisp2 driver, which aims to support all Rockchip ISP 2.x and 3.x versions. This initial version only supports the version of the ISP on the RK3588, which is a 3.0.
[...]
Hi Paul,
I have been running this on the RV1126 (ISP 2.0) for a while; the comments below are the ones that apply to the code as posted and not only to the 2.0. Everything about the RK3588 is by code inspection, I do not have one.
[...]
Quoted 1 line
+++ b/drivers/media/platform/rockchip/rkisp2/Kconfig
[...]
drivers/media/platform/rockchip/rkisp2/Kconfig line 12
Quoted 11 lines
drivers/media/platform/rockchip/rkisp2/Kconfig new file +11
-0+config VIDEO_ROCKCHIP_ISP2 + tristate "Rockchip Image Signal Processing v2 Unit driver" + depends on V4L_PLATFORM_DRIVERS + depends on VIDEO_DEV && OF + depends on ARCH_ROCKCHIP || COMPILE_TEST + select MEDIA_CONTROLLER + select VIDEO_V4L2_SUBDEV_API + select VIDEOBUF2_DMA_CONTIG + select VIDEOBUF2_VMALLOC + select V4L2_FWNODE + select GENERIC_PHY_MIPI_DPHY
Patches 2 and 3 call the v4l2-isp helpers (v4l2_isp_params_validate_buffer(), v4l2_isp_stats_init_block()), and V4L2_ISP is a hidden symbol, so this needs "select V4L2_ISP" like rkisp1 has; in a configuration where this driver is the only user the module does not link. V4L2_FWNODE and GENERIC_PHY_MIPI_DPHY are selected, but nothing in the driver uses either.
[...]
Quoted 1 line
+++
b/drivers/media/platform/rockchip/rkisp2/rkisp2-capture.c
[...]
Quoted 4 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-capture.c new file +4
-0+/* yuv420 */ + { + .fourcc = V4L2_PIX_FMT_NV21, + .uv_swap = 1,
drivers/media/platform/rockchip/rkisp2/rkisp2-capture.c line 154
Quoted 10 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-capture.c new file +10
-0+ .write_format = RKISP2_MI_CTRL_MP_WRITE_YUV_SPLA, + .output_format = ISP32_MI_OUTPUT_YUV420, + .mbus = MEDIA_BUS_FMT_YUYV8_1_5X8, + }, { + .fourcc = V4L2_PIX_FMT_NV12, + .uv_swap = 0, + .write_format = RKISP2_MI_CTRL_MP_WRITE_YUV_SPLA, + .output_format = ISP32_MI_OUTPUT_YUV420, + .mbus = MEDIA_BUS_FMT_YUYV8_2X8, + }, {
The ISP source pads only offer YUYV8_2X8 (rkisp2_formats[] has no YUYV8_1_5X8 entry) and rkisp2_capture_link_validate() compares the pad code with fmt->mbus, so NV21, NV21M, NV12M, YUV420 and YVU420 fail link validation on both paths; NV12 passes only because its entry carries the code of the pad. The ISP produces the same 4:2:2 data for every YUV output and the resizer and MI do the conversion, so I used YUYV8_2X8 for all YUV entries of both tables.
[...]
Quoted 1 line
+static void
rkisp2_mp_config(struct rkisp2_capture *cap)
[...]
drivers/media/platform/rockchip/rkisp2/rkisp2-capture.c line 447
Quoted 11 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-capture.c new file +11
-0+u32 mask =ISP3X_MPFBC_FORCE_UPD |ISP3X_MP_YUV_MODE ; + + reg =rkisp2_read (rkisp2,ISP3X_MPFBC_CTRL ) & ~mask; +if (pixm->pixelformat ==V4L2_PIX_FMT_NV21 || + pixm->pixelformat ==V4L2_PIX_FMT_NV12 || + pixm->pixelformat ==V4L2_PIX_FMT_NV21M || + pixm->pixelformat ==V4L2_PIX_FMT_NV21M || + pixm->pixelformat ==V4L2_PIX_FMT_YVU420 ) + reg |=ISP3X_SEPERATE_YUV_CFG ; +else + reg |=ISP3X_SEPERATE_YUV_CFG |ISP3X_MP_YUV_MODE ;
NV21M is listed twice and NV12M and YUV420 are missing, so those two get ISP3X_MP_YUV_MODE set unlike the other 4:2:0 formats. The list in rkisp2_sp_config() has the same slip.
[...]
Quoted 1 line
+static void
rkisp2_set_next_buf(struct rkisp2_capture *cap)
[...]
Quoted 16 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-capture.c new file +16
-0+rkisp2_write (cap->rkisp2, cap->config->mi.y_base_ad_init, + buff_addr[RKISP2_PLANE_Y ]); +/* +* In order to support grey format we capture +* YUV422 planar format from the camera and +* set the U and V planes to the dummy buffer +*/ +if (cap->pix.cfg->fourcc ==V4L2_PIX_FMT_GREY ) { +rkisp2_write (cap->rkisp2, + cap->config->mi.cb_base_ad_init, + cap->buf.dummy.dma_addr); + }else { +rkisp2_write (cap->rkisp2, + cap->config->mi.cb_base_ad_init, + buff_addr[RKISP2_PLANE_CB ]); + }
drivers/media/platform/rockchip/rkisp2/rkisp2-capture.c line 686
Quoted 10 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-capture.c new file +10
-0+ }else { +/* +* Use the dummy space allocated by dma_alloc_coherent to +* throw data if there is no available buffer. +*/ +rkisp2_write (cap->rkisp2, cap->config->mi.y_base_ad_init, + cap->buf.dummy.dma_addr); +rkisp2_write (cap->rkisp2, cap->config->mi.cb_base_ad_init, + cap->buf.dummy.dma_addr); + }
mi.cr_base_ad_init is never written, in neither branch, although rkisp2_vb2_buf_init() computes (and for uv_swap swaps) buff_addr[RKISP2_PLANE_CR]. For the three-plane formats (YUV422P, YVU422M, YUV420, YVU420, and GREY on the self path, which writes YUV422 planar) the MI writes the CR plane to the reset value of the register or to whatever the previous stream left there. By code inspection this is what the "YUV422 and YVU422 need some debugging" note refers to. rkisp1_set_next_buf() writes all three (and the dummy for CR on GREY).
[...]
Quoted 1 line
+static int
rkisp2_pipeline_stream_enable(struct rkisp2_capture *cap)
[...]
drivers/media/platform/rockchip/rkisp2/rkisp2-capture.c line 998
Quoted 8 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-capture.c new file +8
-0+/* +* If the other capture is streaming, isp and sensor nodes are already +* enabled, skip them. +*/ +if (rkisp2->pipe.start_count >1 ) +return 0 ; + + ret =v4l2_subdev_call (&rkisp2->isp.sd, video, s_stream, true);
The raw readback nodes call video_device_pipeline_start() on the same rkisp2->pipe and never start the ISP themselves, so start_count counts them too. Starting rawrd0 first and then the main path gives start_count 2 here, the ISP is never configured or enabled, no frame is produced, and the stop waits out the 1 s timeout in rkisp2_cap_stream_disable(). I ran into this on the RV1126 when libcamera started the readback node first; by the code it holds for every model. rkisp2_pipeline_stream_disable() makes the opposite assumption (start_count < (source ? 2 : 3)) and does count the readback node. Tracking whether the ISP is running, or counting only the capture nodes, would make this independent of the start order.
[...]
drivers/media/platform/rockchip/rkisp2/rkisp2-capture.c line 1515
Quoted 2 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-capture.c new file +2
-0+memset (&pixm,0 ,sizeof (pixm)); + pixm.pixelformat =V4L2_PIX_FMT_YUYV ;
YUYV is in neither format table, so rkisp2_try_fmt() silently turns the default into UYVY. The first table entry would say what happens.
[...]
Quoted 1 line
+++
b/drivers/media/platform/rockchip/rkisp2/rkisp2-common.h
[...]
Quoted 1 line
+irqreturn_t
rkisp2_mipi_isr(int irq, void *ctx);
Declared, but neither defined nor used.
[...]
Quoted 1 line
+++
b/drivers/media/platform/rockchip/rkisp2/rkisp2-debug.c
[...]
drivers/media/platform/rockchip/rkisp2/rkisp2-debug.c line 113
Quoted 2 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-debug.c new file +2
-0+RKISP2_DEBUG_REG (MI_MP_Y_SIZE_INIT ), + RKISP2_DEBUG_REG(MI_MP_Y_SIZE_INIT),
Listed twice.
[...]
Quoted 1 line
+++
b/drivers/media/platform/rockchip/rkisp2/rkisp2-dev.c
[...]
drivers/media/platform/rockchip/rkisp2/rkisp2-dev.c line 317
Quoted 2 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-dev.c new file +2
-0+v4l2_async_nf_unregister (&rkisp2->notifier); +v4l2_async_nf_cleanup (&rkisp2->notifier);
The notifier is never initialised or registered in probe; both helpers tolerate the zeroed structure, so this is dead code until the inline mode arrives.
[...]
Quoted 1 line
+++
b/drivers/media/platform/rockchip/rkisp2/rkisp2-dmarx.c
[...]
drivers/media/platform/rockchip/rkisp2/rkisp2-dmarx.c line 332
Quoted 5 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-dmarx.c new file +5
-0+if (done) +vb2_buffer_done (&done->vb.vb2_buf,VB2_BUF_STATE_DONE ); + +dev_dbg (rkisp2->dev,"%s: %s buffer %pad done\n" , + __func__, chan->cfg->name, &done->buff_addr[0 ]);
done can be NULL here; %pad then prints "(efault)". This belongs under the if.
[...]
Quoted 1 line
+static void
rkisp2_rawrd_stop_streaming(struct vb2_queue *queue)
[...]
drivers/media/platform/rockchip/rkisp2/rkisp2-dmarx.c line 413
Quoted 11 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-dmarx.c new file +11
-0+mutex_lock (&rkisp2->stream_lock); + chan->streaming = false; +rkisp2_rawrd_return_all_buffers (chan,VB2_BUF_STATE_ERROR ); + +/* +* TODO Use stream counter to stop/start all video nodes when all are +* started / one is stopped +* We only need to clear BIT(30) but we write just BIT(30) anyway so +* writing 0 is ok for now +*/ +rkisp2_write (rkisp2,MI_RD_CTRL2 ,0 );
When the readback node is stopped while the main path keeps streaming, curr_buf (the frame the engine may be reading) is returned and the vb2 buffers are freed right after; the CSI2RX stays enabled and nothing waits for the read in progress, while the capture side waits for the shadow disable at the next frame end. Is there something on the 3.0 that guarantees the read has finished at this point? On the RV1126 I had to wait for the ISP frame completion before releasing the buffers.
[...]
Quoted 1 line
+++
b/drivers/media/platform/rockchip/rkisp2/rkisp2-isp.c
[...]
drivers/media/platform/rockchip/rkisp2/rkisp2-isp.c line 262
Quoted 2 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-isp.c new file +2
-0+ val =RKISP2_CIF_SWS_MIPI_DROP_FRM_DIS |RKISP2_CIF_SWS_ACK_FRM_PRO_DIS ; +rkisp2_write (rkisp2,ISP3X_SWS_CFG , val);
This and rkisp2_runtime_resume() replace the whole register. On the 2.0 the reset value has further bits set (the DDR read enables of the TMO, WDR and DHAZ blocks) and ACK_FRM_PRO_DIS does not exist, so I set the drop bit on top of the current value there. Does the 3.0 hold anything in SWS_CFG at reset that this write should keep?
[...]
drivers/media/platform/rockchip/rkisp2/rkisp2-isp.c line 496
Quoted 2 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-isp.c new file +2
-0+if (format->quantization !=V4L2_QUANTIZATION_DEFAULT ) + src_fmt->quantization = format->quantization;
A full range request is accepted here but nothing acts on it: the CSM_Y_FULL_ENA and CSM_C_FULL_ENA bits of ISP_CTRL are never set and the CSM parameters block has no range field, so with full range coefficients in the CSM block the picture stays limited range (seen on the RV1126 with the sYCC configuration of the libcamera IPA). rkisp1_csm_config() sets the two bits from the pad quantization; doing the same in rkisp2_config_isp() fixed it for me.
[...]
include/uapi/linux/rkisp1-config.h enum rkisp1_cif_isp_version { · line 211
Quoted 7 lines
+++
b/include/uapi/linux/rkisp1-config.hinclude/uapi/linux/rkisp1-config.h +1
-0@@ -207,6 +207,7 @@ enum rkisp1_cif_isp_version {RKISP1_V12 ,RKISP1_V13 ,RKISP1_V_IMX8MP , + RKISP1_V30, };
Nothing uses this; rkisp2 has its own enum rkisp2_isp_version with RKISP3_V0. It changes the uapi of another driver for no reason.
Best regards,
Daniel
PATCH v3 2/5 media: rkisp2: Add parameters output video node
In reply to Paul ElderImplement support for setting parameters on the ISP by queueing parameter buffers to rkisp2.
Support for the following parameters is added:
- BLS (black level subtraction)
- AWB gains (color gains)
- CSM (color space conversion)
- CCM (color correction matrix)
- GOC (gamma out correction)
- LSC (lens shading correction)
- Crop
---
Changes in v3:
- add padding to uapi structs
- fix metafmt documentation
Changes in v2:
- enable bls
- add crop
--- .../userspace-api/media/v4l/meta-formats.rst | 1 + .../userspace-api/media/v4l/metafmt-rkisp2.rst | 70 ++ drivers/media/platform/rockchip/rkisp2/Makefile | 4 +- .../media/platform/rockchip/rkisp2/rkisp2-common.h | 99 +++ .../media/platform/rockchip/rkisp2/rkisp2-dev.c | 17 + .../media/platform/rockchip/rkisp2/rkisp2-isp.c | 13 +- .../media/platform/rockchip/rkisp2/rkisp2-params.c | 929 +++++++++++++++++++++ .../platform/rockchip/rkisp2/rkisp2-regs-v3x.h | 31 + .../platform/rockchip/rkisp2/rkisp2-resizer.c | 407 +++++++++ drivers/media/v4l2-core/v4l2-ioctl.c | 1 + include/uapi/linux/media/rockchip/rkisp2-config.h | 361 ++++++++ include/uapi/linux/videodev2.h | 3 + 12 files changed, 1934 insertions(+), 2 deletions(-)
Documentation/userspace-api/media/v4l/meta-formats.rst +1 -0
@@ -21,6 +21,7 @@ These formats are used for the :ref:`metadata` interface only. metafmt-pisp-be metafmt-pisp-fe metafmt-rkisp1 + metafmt-rkisp2 metafmt-rppx1 metafmt-uvc metafmt-uvc-msxu-1-5
Documentation/userspace-api/media/v4l/metafmt-rkisp2.rst new file +70 -0
70 lines not on this page · raw message
drivers/media/platform/rockchip/rkisp2/Makefile +3 -1
@@ -4,7 +4,9 @@ rockchip-isp2-y := rkisp2-capture.o \ rkisp2-common.o \ rkisp2-dev.o \ rkisp2-dmarx.o \ - rkisp2-isp.o + rkisp2-isp.o \ + rkisp2-params.o \ + rkisp2-resizer.o rockchip-isp2-$(CONFIG_DEBUG_FS) += rkisp2-debug.o
drivers/media/platform/rockchip/rkisp2/rkisp2-common.h +99 -0
149 lines not on this page · raw message
drivers/media/platform/rockchip/rkisp2/rkisp2-dev.c +17 -0
@@ -123,11 +123,22 @@ static int rkisp2_create_links(struct rkisp2_device *rkisp2)return ret; } +/* params links */ + ret =media_create_pad_link (&rkisp2->params.vnode.vdev.entity,0 , + &rkisp2->isp.sd.entity, +RKISP2_ISP_PAD_SINK_PARAMS , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); +if (ret) +return ret; + +return 0 ; }static void rkisp2_entities_unregister (struct rkisp2_device *rkisp2) { +rkisp2_params_unregister (rkisp2);rkisp2_dmarx_unregister (rkisp2);rkisp2_capture_devs_unregister (rkisp2);rkisp2_isp_unregister (rkisp2);
@@ -141,6 +152,8 @@ static int rkisp2_entities_register(struct rkisp2_device *rkisp2)if (ret) goto error; +rkisp2_resizer_devs_init (rkisp2); + ret =rkisp2_capture_devs_register (rkisp2);if (ret) goto error;
@@ -149,6 +162,10 @@ static int rkisp2_entities_register(struct rkisp2_device *rkisp2)if (ret) goto error; + ret =rkisp2_params_register (rkisp2); +if (ret) + goto error; + ret =rkisp2_create_links (rkisp2);if (ret) goto error;
drivers/media/platform/rockchip/rkisp2/rkisp2-isp.c +12 -1
@@ -93,6 +93,7 @@ static int rkisp2_config_isp(struct rkisp2_isp *isp,const struct rkisp2_mbus_info *sink_fmt;const struct rkisp2_mbus_info *mp_src_fmt;const struct v4l2_mbus_framefmt *mp_src_frm; +const struct v4l2_mbus_framefmt *sp_src_frm;const struct v4l2_mbus_framefmt *sink_frm;/* All the video sink formats and crops are kept equal */
@@ -150,6 +151,11 @@ static int rkisp2_config_isp(struct rkisp2_isp *isp,RKISP2_CIF_ISP_PIC_SIZE_ERROR ;rkisp2_write (rkisp2,RKISP2_CIF_ISP_IMSC , irq_mask); + sp_src_frm =v4l2_subdev_state_get_format (sd_state, +RKISP2_ISP_PAD_SOURCE_VIDEO_SELF ); +rkisp2_params_pre_configure (&rkisp2->params, sink_fmt->bayer_pat, + sink_frm, mp_src_frm, sp_src_frm); + isp->sink_fmt = sink_fmt;return 0 ;
@@ -278,6 +284,8 @@ static int rkisp2_isp_start(struct rkisp2_isp *isp,RKISP2_CIF_ISP_CTRL_ISP_INFORM_ENABLE |RKISP2_CIF_ISP_CTRL_ISP_CFG_UPD_PERMANENT ;rkisp2_write (rkisp2,RKISP2_CIF_ISP_CTRL , val); +rkisp2_params_post_configure (&rkisp2->params); +return 0 ; }
@@ -727,6 +735,7 @@ int rkisp2_isp_register(struct rkisp2_device *rkisp2) pads[RKISP2_ISP_PAD_SINK_VIDEO_DMA_1 ].flags =MEDIA_PAD_FL_SINK ; pads[RKISP2_ISP_PAD_SINK_VIDEO_DMA_2 ].flags =MEDIA_PAD_FL_SINK ; pads[RKISP2_ISP_PAD_SINK_VIDEO_CIF ].flags =MEDIA_PAD_FL_SINK ; + pads[RKISP2_ISP_PAD_SINK_PARAMS ].flags =MEDIA_PAD_FL_SINK ; pads[RKISP2_ISP_PAD_SOURCE_VIDEO_MAIN ].flags =MEDIA_PAD_FL_SOURCE ; pads[RKISP2_ISP_PAD_SOURCE_VIDEO_SELF ].flags =MEDIA_PAD_FL_SOURCE ;
@@ -817,8 +826,10 @@ irqreturn_t rkisp2_isp_isr(int irq, void *ctx) rkisp2->debug.data_loss++; } -if (status &RKISP2_CIF_ISP_FRAME ) +if (status &RKISP2_CIF_ISP_FRAME ) { rkisp2->debug.complete_frames++; +rkisp2_params_isr (&rkisp2->params); + }rkisp2_write (rkisp2,RKISP2_CIF_ISP_ICR , status);
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c new file +929 -0
929 lines not on this page · raw message
drivers/media/platform/rockchip/rkisp2/rkisp2-regs-v3x.h +31 -0
@@ -881,6 +881,15 @@#define ISP3X_LSC_YSIZE_CD (ISP3X_LSC_BASE + 0x000A8)#define ISP3X_LSC_YSIZE_EF (ISP3X_LSC_BASE + 0x000AC) +#define ISP3X_LSC_XGRAD (n) (n < 4 ? (ISP3X_LSC_XGRAD_01 + (n) * 4) : \ + (ISP3X_LSC_XGRAD_89 + ((n) - 4) * 4)) +#define ISP3X_LSC_YGRAD (n) (n < 4 ? (ISP3X_LSC_YGRAD_01 + (n) * 4) : \ + (ISP3X_LSC_YGRAD_89 + ((n) - 4) * 4)) +#define ISP3X_LSC_XSIZE (n) (n < 4 ? (ISP3X_LSC_XSIZE_01 + (n) * 4) : \ + (ISP3X_LSC_XSIZE_89 + ((n) - 4) * 4)) +#define ISP3X_LSC_YSIZE (n) (n < 4 ? (ISP3X_LSC_YSIZE_01 + (n) * 4) : \ + (ISP3X_LSC_YSIZE_89 + ((n) - 4) * 4)) +#define ISP3X_DEBAYER_BASE 0x00002500#define ISP3X_DEBAYER_CONTROL (ISP3X_DEBAYER_BASE + 0x00000)#define ISP3X_DEBAYER_G_INTERP (ISP3X_DEBAYER_BASE + 0x00004)
@@ -3235,6 +3244,12 @@#define ISP33_IIR_WR_ID (x) (((x) & 0x3f) << 16)#define ISP33_IIR_WR_CLEAR BIT(24) +/* GAMMA_OUT */ +#define ISP3X_GAMMA_OUT_CTRL_EN BIT(0) +#define ISP3X_GAMMA_OUT_CTRL_MODE_EQUIDISTANT BIT(1) +#define ISP3X_GAMMA_OUT_CTRL_SEGMENTS_48 BIT(2) +#define ISP3X_GAMMA_OUT_SAMPLE (a, b) (((a) & 0xfff) | (((b) & 0xfff) << 16)) +/* HDRTMO */ /* HDRDRC */
@@ -3264,6 +3279,7 @@#define ISP3X_DPCC_WORKING BIT(30)/* CCM */ +#define ISP3X_CCM_COEFF (a, b) ((a & 0x7ff) | (b << 16))#define ISP3X_CCM_HIGHY_ADJ_DIS BIT(1)#define ISP32_CCM_ENH_ADJ_EN BIT(2)#define ISP32_CCM_ASYM_ADJ_EN BIT(3)
@@ -3288,6 +3304,21 @@#define ISP3X_LSC_TABLE_ADDRESS_0 0#define ISP3X_LSC_TABLE_ADDRESS_153 153 +#define ISP3X_LSC_CTRL_EN BIT(0) + +#define ISP3X_LSC_ACTIVE_TABLE BIT(1) + +#define ISP3X_LSC_LUT_EN BIT(1) +#define ISP3X_LSC_SECTOR_16X16 BIT(2) +#define ISP3X_LSC_PRE_RD_ST_MODE BIT(4) + +#define ISP3X_LSC_TABLE_DATA (v0, v1) \ + (((v0) & 0x1FFF) | (((v1) & 0x1FFF) << 16)) +#define ISP3X_LSC_SECT_SIZE (v0, v1) \ + (((v0) & 0xFFF) | (((v1) & 0xFFF) << 16)) +#define ISP3X_LSC_GRAD_SIZE (v0, v1) \ + (((v0) & 0xFFF) | (((v1) & 0xFFF) << 16)) +#define ISP3X_LSC_LUT_EN BIT(1)#define ISP3X_LSC_SECTOR_16X16 BIT(2)#define ISP3X_LSC_PRE_RD_ST_MODE BIT(4)
drivers/media/platform/rockchip/rkisp2/rkisp2-resizer.c new file +407 -0
407 lines not on this page · raw message
drivers/media/v4l2-core/v4l2-ioctl.c +1 -0
@@ -1467,6 +1467,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)case V4L2_META_FMT_RK_ISP1_PARAMS : descr ="Rockchip ISP1 3A Parameters" ;break ;case V4L2_META_FMT_RK_ISP1_STAT_3A : descr ="Rockchip ISP1 3A Statistics" ;break ;case V4L2_META_FMT_RK_ISP1_EXT_PARAMS : descr ="Rockchip ISP1 Ext 3A Params" ;break ; +case V4L2_META_FMT_RKISP2_PARAMS : descr ="Rockchip ISP2 3A Parameters" ;break ;case V4L2_META_FMT_C3ISP_PARAMS : descr ="Amlogic C3 ISP Parameters" ;break ;case V4L2_META_FMT_C3ISP_STATS : descr ="Amlogic C3 ISP Statistics" ;break ;case V4L2_META_FMT_MALI_C55_PARAMS : descr ="ARM Mali-C55 ISP Parameters" ;break ;
include/uapi/linux/media/rockchip/rkisp2-config.h +361 -0
371 lines not on this page · raw message
include/uapi/linux/videodev2.h +3 -0
@@ -876,6 +876,9 @@ struct v4l2_pix_format {#define V4L2_META_FMT_RK_ISP1_STAT_3A v4l2_fourcc('R', 'K', '1', 'S')/* Rockchip ISP1 3A Statistics */ #define V4L2_META_FMT_RK_ISP1_EXT_PARAMS v4l2_fourcc('R', 'K', '1', 'E')/* Rockchip ISP1 3a Extensible Parameters */ +/* Vendor specific - used for RKISP2 camera sub-system */ +#define V4L2_META_FMT_RKISP2_PARAMS v4l2_fourcc('R', 'K', '2', 'P')/* Rockchip ISP2 Parameters */ +/* Vendor specific - used for C3_ISP */ #define V4L2_META_FMT_C3ISP_PARAMS v4l2_fourcc('C', '3', 'P', 'M')/* Amlogic C3 ISP Parameters */ #define V4L2_META_FMT_C3ISP_STATS v4l2_fourcc('C', '3', 'S', 'T')/* Amlogic C3 ISP Statistics */
Signature · 1 line
2.47.2
[email protected]
AI reviewRe: [PATCH v3 2/5] media: rkisp2: Add parameters output video node
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Critical] The parameter buffer validation checks the userspace-mapped memory instead of the internal…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] The parameter buffer validation checks the userspace-mapped memory instead of the internal kernel copy, creating a Time-Of-Check to Time-Of-Use (TOCTOU) vulnerability that can lead to out-of-bounds memory access.
- [High] The first queued parameter buffer is permanently leaked every time streaming starts because it is dequeued but never completed.
- [Medium] Mutex and media entity resources are leaked on the error path if video_register_device fails.
--
commit 29257cb1b743c9d1fdc2e61085bac5b0b1b5d99d
Author: Paul Elder <[email protected]>
media: rkisp2: Add parameters output video node
Implement support for setting parameters on the ISP by queueing parameter buffers to rkisp2.
Quoted 4 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c new file +0
-0
[ ... ]
Quoted 14 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c new file +14
-0+static int rkisp2_params_vb2_buf_prepare (struct vb2_buffer *vb) +{ +struct rkisp2_params *params = vb->vb2_queue->drv_priv; +struct vb2_v4l2_buffer *vbuf =to_vb2_v4l2_buffer (vb); +struct rkisp2_params_buffer *params_buf =to_rkisp2_params_buffer (vbuf); +struct v4l2_isp_params_buffer *cfg =vb2_plane_vaddr (&vbuf->vb2_buf,0 ); +size_t payload_size =vb2_get_plane_payload (vb,0 ); +int ret; + + ret =v4l2_isp_params_validate_buffer_size (params->rkisp2->dev, vb, +v4l2_isp_buffer_size (RKISP2_PARAMS_MAX_SIZE )); +if (ret) +return ret; +
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c line 796
Quoted 10 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c new file +10
-0+/* +* Copy the parameters buffer to the internal scratch buffer to avoid +* userspace modifying the buffer content while the driver processes it. +*/ +memcpy (params_buf->cfg, cfg, payload_size); + +return v4l2_isp_params_validate_buffer (params->rkisp2->dev, vb, cfg, + rkisp2_params_block_types_info, +ARRAY_SIZE (rkisp2_params_block_types_info)); +}
[Severity: Critical]
Can this introduce a TOCTOU (Time-Of-Check to Time-Of-Use) regression?
The code correctly copies the userspace buffer to an internal scratch buffer (params_buf->cfg) to prevent concurrent modification. However, v4l2_isp_params_validate_buffer() is then called on the mutable userspace pointer (cfg) rather than the safe internal copy (params_buf->cfg).
A malicious userspace program could submit invalid data, wait for the memcpy() to finish, and then overwrite the shared memory with valid data before the validation step completes.
Since the driver later uses the internal copy, it might end up consuming an unvalidated parameter block type and using it to index rkisp2_params_handlers[], which can lead to out-of-bounds memory access.
[ ... ]
Quoted 6 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c new file +6
-0+void rkisp2_params_post_configure (struct rkisp2_params *params) +{ +struct rkisp2_params_buffer *buf; + +spin_lock_irq (¶ms->buf_lock); + buf = list_first_entry_or_null(¶ms->params,
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c line 662
Quoted 10 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c new file +10
-0+ struct rkisp2_params_buffer, queue); +if (buf) +list_del (&buf->queue); +spin_unlock_irq (¶ms->buf_lock); + +if (!buf) +return ; + +rkisp2_params_configure (params, buf,RKISP2_PARAMS_CONFIG_PRIO_POST ); +}
[Severity: High]
Is a buffer permanently leaked here when streaming starts?
When the ISP starts, the first queued parameter buffer is removed from the list via list_del(&buf->queue) and passed to rkisp2_params_configure() with RKISP2_PARAMS_CONFIG_PRIO_POST.
However, rkisp2_params_configure() explicitly checks for RKISP2_PARAMS_CONFIG_PRIO_NONE before calling vb2_buffer_done():
static void rkisp2_params_configure(struct rkisp2_params *params,
...
if (prio == RKISP2_PARAMS_CONFIG_PRIO_NONE) vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_DONE); }
Since the buffer is no longer on the params->params list and vb2_buffer_done() is skipped, it seems the buffer will never be returned to vb2. This could freeze the pipeline or result in a WARN_ON during stop_streaming() due to left behind active buffers.
[ ... ]
Quoted 2 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c new file +2
-0+int rkisp2_params_register (struct rkisp2_device *rkisp2) +{
[ ... ]
Quoted 9 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c new file +9
-0+ node->pad.flags =MEDIA_PAD_FL_SOURCE ; + ret =media_entity_pads_init (&vdev->entity,1 , &node->pad); +if (ret) + goto err_media; + + ret =video_register_device (vdev,VFL_TYPE_VIDEO ,-1 ); +if (ret) { +dev_err (rkisp2->dev, +"failed to register %s, ret=%d\n" , vdev->name, ret);
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c line 915
Quoted 10 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c new file +10
-0+return ret; + } + +return 0 ; + +err_media: +media_entity_cleanup (&vdev->entity); +mutex_destroy (&node->vlock); +return ret; +}
[Severity: Medium]
Are resources leaked on this error path if video_register_device() fails?
If video_register_device() returns an error, the code returns the error code directly instead of jumping to the err_media cleanup block. This leaves the media entity and node->vlock uncleaned, leaking these resources.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Re: [PATCH v3 2/5] media: rkisp2: Add parameters output video node
In reply to Paul ElderQuoted 2 lines · Paul Elder
Implement support for setting parameters on the ISP by queueing parameter buffers to rkisp2.
[...]
Hi Paul,
A few comments from running this on the RV1126; the RK3588 statements are by code inspection.
[...]
Quoted 1 line
+++
b/Documentation/userspace-api/media/v4l/metafmt-rkisp2.rst
[...]
Quoted 2 lines
+The configuration parameters are passed to the :ref:
rkisp2_params +<rkisp2_params>metadata output video node, using the
There is no rkisp2_params label anywhere under Documentation/ (rkisp1 defines its labels in admin-guide/media/rkisp1.rst), so Sphinx warns about an undefined label. The same holds for rkisp2_stats in patch 3.
[...]
Quoted 1 line
+++
b/drivers/media/platform/rockchip/rkisp2/rkisp2-params.c
[...]
Quoted 1 line
+static void rkisp2_params_goc(struct
rkisp2_params*params,
[...]
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c line 311
Quoted 4 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c new file +4
-0+if (arg->mode ==RKISP2_ISP_GOC_MODE_EQUIDISTANT ) + control =ISP3X_GAMMA_OUT_CTRL_MODE_EQUIDISTANT ; +if (arg->mode ==RKISP2_ISP_GOC_SEGMENTS_48 ) + control |=ISP3X_GAMMA_OUT_CTRL_SEGMENTS_48 ;
The second test is meant for arg->segments. As RKISP2_ISP_GOC_SEGMENTS_48 and RKISP2_ISP_GOC_MODE_EQUIDISTANT are both 1, equidistant mode always gets the 48 segment layout and the segments member of the uapi is dead.
[...]
Quoted 5 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c new file +5
-0+static void rkisp2_params_crop_init (struct rkisp2_params *params) +{ +struct rkisp2_device *rkisp2 = params->rkisp2; + +/* Create an initial crop configuration to configure */
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c line 543
Quoted 10 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c new file +10
-0+struct rkisp2_params_crop arg = { + .header.type =RKISP2_PARAMS_BLOCK_CROP , + .header.flags =0 , + .header.size =sizeof (struct rkisp2_params_crop ), + .crop_en =RKISP2_ISP_CROP_ENABLE_MAIN |RKISP2_ISP_CROP_ENABLE_SELF , + + .mp_crop.h_offs =0 , + .mp_crop.v_offs =0 , + .mp_crop.h_size = rkisp2->resizer_devs[0 ].source_size.width, + .mp_crop.v_size = rkisp2->resizer_devs[0 ].source_size.height,
Without a params buffer at stream start this crops a window of the output size at the top left corner, so a 1920x1080 capture from a 3864x2192 sensor shows that corner only. The sink size (what the CROP block uses when it is disabled), scaled by the resizer, is what rkisp1 does and what a plain v4l2-ctl capture expects.
[...]
Quoted 1 line
+static void rkisp2_params_configure(struct
rkisp2_params*params,
[...]
Quoted 7 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c new file +7
-0+/* update shadow register immediately */ +rkisp2_param_set_bits (params,RKISP2_CIF_ISP_CTRL , +RKISP2_CIF_ISP_CTRL_ISP_CFG_UPD ); + +if (prio ==RKISP2_PARAMS_CONFIG_PRIO_NONE ) +vb2_buffer_done (&buf->vb.vb2_buf,VB2_BUF_STATE_DONE ); +}
[...]
Quoted 1 line
+void rkisp2_params_pre_configure(struct
rkisp2_params*params,
[...]
Quoted 5 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c new file +5
-0+spin_lock_irq (¶ms->buf_lock); + buf =list_first_entry_or_null (¶ms->params, + struct rkisp2_params_buffer, queue); +spin_unlock_irq (¶ms->buf_lock); +
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c line 644
Quoted 10 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c new file +10
-0+if (!buf) { +/* +* Initialize crop in the event that are no pre-queued +* parameter buffers +*/ +rkisp2_params_crop_init (params); +return ; + } + +rkisp2_params_configure (params, buf,RKISP2_PARAMS_CONFIG_PRIO_PRE );
When a buffer is queued but carries no CROP block, neither path programs the dual crop and the resizer, so they keep the state of the previous stream (the IRCL pulse in rkisp2_isp_stop() resets the ISP and MIPI blocks only) or the reset state. With a capture size that differs from the sink size the MI is then fed frames of the wrong size. Running rkisp2_params_crop_init() first and letting a CROP block override it covers both cases.
Quoted 8 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c new file +8
-0+} + +void rkisp2_params_post_configure (struct rkisp2_params *params) +{ +struct rkisp2_params_buffer *buf; + +spin_lock_irq (¶ms->buf_lock); + buf = list_first_entry_or_null(¶ms->params,
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c line 662
Quoted 10 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c new file +10
-0+ struct rkisp2_params_buffer, queue); +if (buf) +list_del (&buf->queue); +spin_unlock_irq (¶ms->buf_lock); + +if (!buf) +return ; + +rkisp2_params_configure (params, buf,RKISP2_PARAMS_CONFIG_PRIO_POST ); +}
The buffer is taken off the list here and configured with PRIO_POST, which does not complete it (see above), so a params buffer queued before STREAMON never comes back from DQBUF; at STREAMOFF vb2 warns that stop_streaming left a buffer in active state and releases it itself. rkisp1_params_post_configure() completes the buffer after the LSC group; vb2_buffer_done() belongs here after the POST pass.
[...]
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c line 914
Quoted 13 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c new file +13
-0+ ret =video_register_device (vdev,VFL_TYPE_VIDEO ,-1 ); +if (ret) { +dev_err (rkisp2->dev, +"failed to register %s, ret=%d\n" , vdev->name, ret); +return ret; + } + +return 0 ; + +err_media: +media_entity_cleanup (&vdev->entity); +mutex_destroy (&node->vlock); +return ret;
goto err_media here as well. Neither path releases the vb2 queue that was initialised above.
[...]
Quoted 1 line
+++
b/drivers/media/platform/rockchip/rkisp2/rkisp2-resizer.c
[...]
Quoted 1 line
+void rkisp2_resizer_configure(struct
rkisp2_resizer*rsz,
[...]
drivers/media/platform/rockchip/rkisp2/rkisp2-resizer.c line 364
Quoted 6 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-resizer.c new file +6
-0+if (crop->width == rsz->source_size.width && crop->height == rsz->source_size.height) +rkisp2_resizer_disable_rsz (rsz); +else +rkisp2_resizer_config_rsz (rsz, crop, + &rsz->source_size, + &rsz->source_format);
The bypass test compares the luma size only. The ISP delivers 4:2:2, and rkisp2_resizer_config_rsz() handles the chroma subsampling when it runs, but an unscaled 4:2:0 stream (crop equal to the output size) disables the resizer and the MI gets one chroma row per luma row. On the RV1126 main path the chroma plane offset wraps halfway through the frame and the chroma of the lower half lands on the upper half (displaced colour ghosts and neutral halos). rkisp1_rsz_config() bypasses only when the size and the subsampling are both unchanged; the same test here fixed it for me. I do not know whether the 3.0 MI subsamples on its own in the ISP32_MI_OUTPUT_YUV420 mode; if it does not, this applies to the RK3588 too.
Best regards,
Daniel
PATCH v3 3/5 media: rkisp2: Add statistics capture video node
In reply to Paul ElderImplement support for getting statistics out of the ISP by dequeueing statistics buffers from rkisp2.
Support for configuring the statistics engines in the ISP is added as well, implemented as configurations in parameter buffers.
Support for the following statistics engines is added:
- AE (auto exposure) lite
- Histogram lite
- Histogram big
- AWB (auto white balance)
---
I've kept the debug counters that helped me debug interrupts for the stats engines.
---
Changes in v3:
- add padding to uapi structs
- fix metafmt documentation
Changes in v2:
- redesign stats accumulation
- rfc/v1 checked imsc at v_start to see what stats we expect and send them to userspace only after we receive all of them (or at next v_start if that comes first)
- stats interrupt*s* come between v_start and input formatter
- v2 we just send whatever stats came at end-of-frame time
- this is guaranteed to be after all stats interrupts have come
- fix memops
- use extensible stats
--- .../userspace-api/media/v4l/metafmt-rkisp2.rst | 48 +- drivers/media/platform/rockchip/rkisp2/Makefile | 3 +- .../media/platform/rockchip/rkisp2/rkisp2-common.h | 56 +++ .../media/platform/rockchip/rkisp2/rkisp2-debug.c | 14 + .../media/platform/rockchip/rkisp2/rkisp2-dev.c | 13 + .../media/platform/rockchip/rkisp2/rkisp2-isp.c | 2 + .../media/platform/rockchip/rkisp2/rkisp2-params.c | 238 ++++++++++ .../platform/rockchip/rkisp2/rkisp2-regs-v2x.h | 39 +- .../platform/rockchip/rkisp2/rkisp2-regs-v3x.h | 14 + .../media/platform/rockchip/rkisp2/rkisp2-stats.c | 509 +++++++++++++++++++++ drivers/media/v4l2-core/v4l2-ioctl.c | 1 + include/uapi/linux/media/rockchip/rkisp2-config.h | 264 ++++++++++- include/uapi/linux/videodev2.h | 1 + 13 files changed, 1196 insertions(+), 6 deletions(-)
Documentation/userspace-api/media/v4l/metafmt-rkisp2.rst +45 -3
@@ -1,10 +1,11 @@ .. SPDX-License-Identifier: GPL-2.0 .. _v4l2-meta-fmt-rkisp2-params: +.. _v4l2-meta-fmt-rkisp2-stats: -************************************ -V4L2_META_FMT_RKISP2_PARAMS ('RK2P') -************************************ +************************************************************************* +V4L2_META_FMT_RKISP2_PARAMS ('RK2P'), V4L2_META_FMT_RKISP2_STATS ('RK2S') +************************************************************************* Configuration Parameters ========================
@@ -64,6 +65,47 @@ first member and userspace must populate the type member with a value from params->data_size += sizeof(struct rkisp2_params_bls); +3A Statistics +============= + +The ISP device collects different statistics over an input bayer frame. Those +statistics can be obtained by userspace from the +:ref:`rkisp2_stats <rkisp2_stats>` metadata capture video node, using +the :c:type:`v4l2_meta_format` interface. Rather than a single struct containing +sub-structs for each statistics area of the ISP, statistics for the RkISP2 use +the v4l2-isp statistics system, through which groups of statistics are defined +as distinct members or "blocks" which may be added to the data member of type +:c:type:`v4l2_isp_buffer`. Userspace is responsible for parsing the buffer and +extracting the blocks of statistics. Each block-specific struct embeds +:c:type:`v4l2_isp_block_header` as its first member and userspace must +interpret the type member with a value from :c:type:`rkisp2_stats_block_type`. + +.. code-block:: C + + const struct v4l2_isp_buffer *stats = + (struct v4l2_isp_buffer *)buf; + size_t block_offset = 0; + + while (block_offset < stats->data_size) { + const struct v4l2_isp_stats_block_header *block = + (void*)(stats->data + block_offset); + + block_offset += block->size; + + switch (block->type) { + case RKISP2_STATS_BLOCK_AE_LITE: + const struct rkisp2_stats_ae_lite *ae_lite = + (struct rkisp2_stats_ae_lite *)block; + for (unsigned int i = 0; i < RKISP2_ISP_AE_MEAN_MAX_LITE; i++) + printf("ae_lite.exp_mean_r[%u] = 0x%08x\n", + i, ae_lite.exp_mean_r[%i]); + break; + default: + printf("Unknown block type 0x%04x", block->type); + break; + } + } + rkisp2 uAPI data types ======================
drivers/media/platform/rockchip/rkisp2/Makefile +2 -1
@@ -6,7 +6,8 @@ rockchip-isp2-y := rkisp2-capture.o \ rkisp2-dmarx.o \ rkisp2-isp.o \ rkisp2-params.o \ - rkisp2-resizer.o + rkisp2-resizer.o \ + rkisp2-stats.o rockchip-isp2-$(CONFIG_DEBUG_FS) += rkisp2-debug.o
drivers/media/platform/rockchip/rkisp2/rkisp2-common.h +56 -0
101 lines not on this page · raw message
drivers/media/platform/rockchip/rkisp2/rkisp2-debug.c +14 -0
@@ -191,6 +191,8 @@ void rkisp2_debug_init(struct rkisp2_device *rkisp2) &debug->mipi_error);debugfs_create_ulong ("stats_error" ,0444 , debug->debugfs_dir, &debug->stats_error); +debugfs_create_ulong ("stats3a_irq" ,0444 , debug->debugfs_dir, + &debug->stats3a_irq);debugfs_create_ulong ("mp_stop_timeout" ,0444 , debug->debugfs_dir, &debug->stop_timeout[RKISP2_MAINPATH ]); debugfs_create_ulong("sp_stop_timeout", 0444, debug->debugfs_dir,
@@ -201,6 +203,18 @@ void rkisp2_debug_init(struct rkisp2_device *rkisp2) &debug->frame_drop[RKISP2_SELFPATH]);debugfs_create_ulong ("complete_frames" ,0444 , debug->debugfs_dir, &debug->complete_frames); +debugfs_create_ulong ("stats3a_hist_ch0_count" ,0444 , debug->debugfs_dir, + &debug->stats3a_hist_ch0_count); +debugfs_create_ulong ("stats3a_hist_ch1_count" ,0444 , debug->debugfs_dir, + &debug->stats3a_hist_ch1_count); +debugfs_create_ulong ("stats3a_hist_ch2_count" ,0444 , debug->debugfs_dir, + &debug->stats3a_hist_ch2_count); +debugfs_create_ulong ("stats3a_hist_big_count" ,0444 , debug->debugfs_dir, + &debug->stats3a_hist_big_count); +debugfs_create_ulong ("stats3a_awb_count" ,0444 , debug->debugfs_dir, + &debug->stats3a_awb_count); +debugfs_create_ulong ("stats3a_awb_done_count" ,0444 , debug->debugfs_dir, + &debug->stats3a_awb_done_count);debugfs_create_file ("input_status" ,0444 , debug->debugfs_dir, rkisp2, &rkisp2_debug_input_status_fops);
drivers/media/platform/rockchip/rkisp2/rkisp2-dev.c +13 -0
@@ -132,12 +132,21 @@ static int rkisp2_create_links(struct rkisp2_device *rkisp2)if (ret)return ret; +/* stats links */ + ret =media_create_pad_link (&rkisp2->isp.sd.entity, +RKISP2_ISP_PAD_SOURCE_STATS , + &rkisp2->stats.vnode.vdev.entity,0 , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); +if (ret) +return ret;return 0 ; }static void rkisp2_entities_unregister (struct rkisp2_device *rkisp2) { +rkisp2_stats_unregister (rkisp2);rkisp2_params_unregister (rkisp2);rkisp2_dmarx_unregister (rkisp2);rkisp2_capture_devs_unregister (rkisp2);
@@ -166,6 +175,10 @@ static int rkisp2_entities_register(struct rkisp2_device *rkisp2)if (ret) goto error; + ret =rkisp2_stats_register (rkisp2); +if (ret) + goto error; + ret =rkisp2_create_links (rkisp2);if (ret) goto error;
drivers/media/platform/rockchip/rkisp2/rkisp2-isp.c +2 -0
@@ -738,6 +738,7 @@ int rkisp2_isp_register(struct rkisp2_device *rkisp2) pads[RKISP2_ISP_PAD_SINK_PARAMS ].flags =MEDIA_PAD_FL_SINK ; pads[RKISP2_ISP_PAD_SOURCE_VIDEO_MAIN ].flags =MEDIA_PAD_FL_SOURCE ; pads[RKISP2_ISP_PAD_SOURCE_VIDEO_SELF ].flags =MEDIA_PAD_FL_SOURCE ; + pads[RKISP2_ISP_PAD_SOURCE_STATS ].flags =MEDIA_PAD_FL_SOURCE ; ret =media_entity_pads_init (&sd->entity,RKISP2_ISP_PAD_MAX , pads);if (ret)
@@ -829,6 +830,7 @@ irqreturn_t rkisp2_isp_isr(int irq, void *ctx)if (status &RKISP2_CIF_ISP_FRAME ) { rkisp2->debug.complete_frames++;rkisp2_params_isr (&rkisp2->params); +rkisp2_stats_isr_eof (&rkisp2->stats); }rkisp2_write (rkisp2,RKISP2_CIF_ISP_ICR , status);
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c +238 -0
274 lines not on this page · raw message
drivers/media/platform/rockchip/rkisp2/rkisp2-regs-v2x.h +38 -1
@@ -1429,7 +1429,7 @@#define ISP_RAWAE_LITE_BLK_SIZ (ISP_RAWAE_LITE_BASE + 0x00004)#define ISP_RAWAE_LITE_OFFSET (ISP_RAWAE_LITE_BASE + 0x00008)#define ISP_RAWAE_LITE_R2Y_CC (ISP_RAWAE_LITE_BASE + 0x0000c) -#define ISP_RAWAE_LITE_RO_MEAN (ISP_RAWAE_LITE_BASE + 0x00010) +#define ISP_RAWAE_LITE_RO_MEAN (i) (ISP_RAWAE_LITE_BASE + 0x00010 + ((i) * 4))#define ISP_RAWAE_LITE_RO_DBG1 (ISP_RAWAE_LITE_BASE + 0x00074)#define ISP_RAWAE_LITE_RO_DBG2 (ISP_RAWAE_LITE_BASE + 0x00078)
@@ -2585,4 +2585,41 @@/* ISP21 DHAZ/DRC/BAY3D */ #define ISP21_SELF_FORCE_UPD BIT(31) +/* RAWHIST */ +#define ISP_RAWHIST_CTRL_EN BIT(0) +#define ISP_RAWHIST_CTRL_STEPSIZE (a) (((a) & 0x7) << 1) +#define ISP_RAWHIST_CTRL_MODE (a) (((a) & 0x7) << 8) +#define ISP_RAWHIST_CTRL_WATERLINE (a) (((a) & 0xfff) << 12) +#define ISP_RAWHIST_CTRL_DATA_SEL (a) (((a) & 0x7) << 24) +#define ISP_RAWHIST_CTRL_MEAS_DONE BIT(31) +#define ISP_RAWHIST_BIG_CTRL_WND_5X5_1 BIT(28) +#define ISP_RAWHIST_BIG_CTRL_WND_15X15_0 BIT(29) +#define ISP_RAWHIST_BIG_CTRL_WND_15X15_1 (0x3 << 28) +#define ISP_RAWHIST_BIG_CTRL_WND_MASK GENMASK(29, 28) + +#define ISP_RAWHIST_H_SIZE (a) ((a) & 0x7ff) +#define ISP_RAWHIST_V_SIZE (a) (((a) & 0x7ff) << 16) +#define ISP_RAWHIST_H_OFFS (a) ((a) & 0x1fff) +#define ISP_RAWHIST_V_OFFS (a) (((a) & 0x1fff) << 16) + +#define ISP_RAWHIST_RAW2Y_CC_RCC (a) ((a) & 0xff) +#define ISP_RAWHIST_RAW2Y_CC_GCC (a) (((a) & 0xff) << 8) +#define ISP_RAWHIST_RAW2Y_CC_BCC (a) (((a) & 0xff) << 16) +#define ISP_RAWHIST_RAW2Y_CC_OFF (a) (((a) & 0xff) << 24) + +#define ISP_RAWHIST_LITE_WEIGHT_WND0 (a) ((a) & 0x3f) +#define ISP_RAWHIST_LITE_WEIGHT_WND1 (a) (((a) & 0x3f) << 8) +#define ISP_RAWHIST_LITE_WEIGHT_WND2 (a) (((a) & 0x3f) << 16) +#define ISP_RAWHIST_LITE_WEIGHT_WND3 (a) (((a) & 0x3f) << 24) + +#define ISP_RAWHIST_BIG_WEIGHT_WND0 (a) ((a) & 0x3f) +#define ISP_RAWHIST_BIG_WEIGHT_WND1 (a) (((a) & 0x3f) << 6) +#define ISP_RAWHIST_BIG_WEIGHT_WND2 (a) (((a) & 0x3f) << 12) +#define ISP_RAWHIST_BIG_WEIGHT_WND3 (a) (((a) & 0x3f) << 18) +#define ISP_RAWHIST_BIG_WEIGHT_WND4 (a) (((a) & 0x3f) << 24) + +#define ISP_RAWHIST_BIN_MASK GENMASK(27, 0) + +#define ISP_RAWHIST_RAM_OFFSET (a) ((a) & 0xff) +#endif /* _RKISP_REGS_V2X_H */
drivers/media/platform/rockchip/rkisp2/rkisp2-regs-v3x.h +14 -0
@@ -3359,6 +3359,7 @@/* RAWAE */ #define ISP3X_RAWAE_LITE_EN BIT(0)#define ISP3X_RAWAE_LITE_WNDNUM BIT(1) +#define ISP3X_RAWAE_LITE_DONE BIT(31)#define ISP3X_RAWAE_BIG_EN BIT(0)#define ISP3X_RAWAE_BIG_WND0_NUM (x) (((x) & 0x3) << 1)
@@ -3367,6 +3368,13 @@#define ISP3X_RAWAE_BIG_WND3_EN BIT(6)#define ISP3X_RAWAE_BIG_WND4_EN BIT(7) +#define ISP3X_RAWAE_LITE_H (a) ((a) & 0x1fff) +#define ISP3X_RAWAE_LITE_V (a) (((a) & 0x1fff) << 16) + +#define ISP3X_RAWAE_LITE_RO_MEAN_R (a) (((a) >> 22) & 0x3ff) +#define ISP3X_RAWAE_LITE_RO_MEAN_B (a) (((a) >> 12) & 0x3ff) +#define ISP3X_RAWAE_LITE_RO_MEAN_G (a) ((a) & 0xfff) +/* RAWHIST */ #define ISP3X_RAWHIST_EN BIT(0)#define ISP3X_RAWHIST_STEPSIZE (x) (((x) & 0x7) << 1)
@@ -3397,6 +3405,12 @@#define ISP3X_RAWAF_INTLINE0_EN BIT(27)/* RAWAWB */ +#define ISP3X_RAWAWB_CTRL_EN BIT(0) +#define ISP3X_RAWAWB_CTRL_MEAS_DONE BIT(31) + +#define ISP3X_RAWAWB_WIN (h, v) (((h) & 0x1fff) | (((v) & 0x1fff) << 16)) +#define ISP3X_RAWAWB_LIMITS (a, b) (((a) & 0xff) | (((b) & 0xff) << 16)) +#define ISP32_RAWAWB_2DDR_PATH_EN BIT(23)#define ISP32_RAWAWB_2DDR_PATH_DS BIT(27)#define ISP32_RAWAWB_2DDR_PATH_ERR BIT(29)
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c new file +509 -0
509 lines not on this page · raw message
drivers/media/v4l2-core/v4l2-ioctl.c +1 -0
@@ -1468,6 +1468,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)case V4L2_META_FMT_RK_ISP1_STAT_3A : descr ="Rockchip ISP1 3A Statistics" ;break ;case V4L2_META_FMT_RK_ISP1_EXT_PARAMS : descr ="Rockchip ISP1 Ext 3A Params" ;break ;case V4L2_META_FMT_RKISP2_PARAMS : descr ="Rockchip ISP2 3A Parameters" ;break ; +case V4L2_META_FMT_RKISP2_STATS : descr ="Rockchip ISP2 3A Statistics" ;break ;case V4L2_META_FMT_C3ISP_PARAMS : descr ="Amlogic C3 ISP Parameters" ;break ;case V4L2_META_FMT_C3ISP_STATS : descr ="Amlogic C3 ISP Statistics" ;break ;case V4L2_META_FMT_MALI_C55_PARAMS : descr ="ARM Mali-C55 ISP Parameters" ;break ;
include/uapi/linux/media/rockchip/rkisp2-config.h +263 -1
305 lines not on this page · raw message
include/uapi/linux/videodev2.h +1 -0
@@ -878,6 +878,7 @@ struct v4l2_pix_format {/* Vendor specific - used for RKISP2 camera sub-system */ #define V4L2_META_FMT_RKISP2_PARAMS v4l2_fourcc('R', 'K', '2', 'P')/* Rockchip ISP2 Parameters */ +#define V4L2_META_FMT_RKISP2_STATS v4l2_fourcc('R', 'K', '2', 'S')/* Rockchip ISP2 3A Statistics */ /* Vendor specific - used for C3_ISP */ #define V4L2_META_FMT_C3ISP_PARAMS v4l2_fourcc('C', '3', 'P', 'M')/* Amlogic C3 ISP Parameters */
Signature · 1 line
2.47.2
[email protected]
AI reviewRe: [PATCH v3 3/5] media: rkisp2: Add statistics capture video node
Thank you for your contribution! Sashiko AI review found 10 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 10 potential issue(s) to consider:
New issues:
- [High] Missing V4L2 block type info for big histogram 2, causing an infinite loop in userspace parsers.
- [High] Unacknowledged hardware 3A interrupts cause an IRQ storm when no stats buffers are queued.
- [High] Return value of rkisp2_stats_init_vb2_queue() is ignored.
- [High] mutex_destroy() is called while file descriptors might still be open, leading to Use-After-Free.
- [High] Unprotected initialization of stats->icr in start_streaming races with the ISR.
- [High] Measurement window dimensions are not clamped in the statistics configuration handlers, which can cause ISP hardware hangs.
- [Medium] Uninitialized stats block exposes stale memory to userspace when hardware status bits are unset.
- [Medium] Truncated programming of big histogram weights.
- [Medium] Missing SRAM/WRAM address pointer reset before reading or writing AWB statistics and parameters.
Pre-existing issues:
- [Critical] Time-of-check to time-of-use (TOCTOU) vulnerability in parameters buffer validation. --
commit a35565a8e85636bb5290bad9e33a1d9d66a1f24a
Author: Paul Elder <[email protected]>
media: rkisp2: Add statistics capture video node
This commit adds support for capturing statistics out of the ISP by dequeueing statistics buffers from rkisp2. Support for configuring the statistics engines in the ISP is also added, implemented as configurations in parameter buffers.
Quoted 3 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c +0
-0
[ ... ]
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c rkisp2_params_handlers[] = { · line 109
Quoted 11 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c +6
-0@@ -84,6 +100,12 @@ rkisp2_params_handlers[] = {RKISP2_PARAMS_BLOCK_HANDLER_INFO (GOC , goc,PRE ),RKISP2_PARAMS_BLOCK_HANDLER_INFO (LSC , lsc,POST ),RKISP2_PARAMS_BLOCK_HANDLER_INFO (CROP , crop,PRE ), +RKISP2_PARAMS_BLOCK_HANDLER_INFO (AE_LITE , ae_lite,PRE ), +RKISP2_PARAMS_BLOCK_HANDLER_INFO (HIST_LITE , hist_lite,PRE ), +RKISP2_PARAMS_BLOCK_HANDLER_INFO (HIST_BIG0 , hist_big,PRE ), +RKISP2_PARAMS_BLOCK_HANDLER_INFO (HIST_BIG1 , hist_big,PRE ), +RKISP2_PARAMS_BLOCK_HANDLER_INFO (HIST_BIG2 , hist_big,PRE ), + RKISP2_PARAMS_BLOCK_HANDLER_INFO(AWB_MEAS, awb_meas, PRE), };
[Severity: Critical]
This is a pre-existing issue, but can malicious userspace bypass block validation to trigger an out-of-bounds read on the rkisp2_params_handlers array?
In rkisp2_params_vb2_buf_prepare(), the buffer is copied to params_buf->cfg, but v4l2_isp_params_validate_buffer() is mistakenly called on the userspace-accessible buffer (cfg) instead of the snapshot.
If userspace concurrently changes block.header->type in the shared memory after memcpy() completes, rkisp2_params_configure() will read the unvalidated type from the snapshot, resulting in an out-of-bounds index when accessing info = &rkisp2_params_handlers[block.header->type] and potentially executing an arbitrary function pointer.
[ ... ]
Quoted 5 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c +5
-0+static void rkisp2_params_ae_lite (struct rkisp2_params *params, +union rkisp2_params_block block) +{ +const struct rkisp2_params_ae_lite *arg = block.ae_lite; +u32 control;
[ ... ]
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c static void rkisp2_params_crop_init(struct rkisp2_params *params) · line 608
Quoted 3 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c +3
-0+rkisp2_write (params->rkisp2,ISP_RAWAE_LITE_BLK_SIZ , +ISP3X_RAWAE_LITE_H (arg->meas_window.h_size) | +ISP3X_RAWAE_LITE_V (arg->meas_window.v_size));
[Severity: High]
Is it safe to write these user-controlled window sizes directly to the ISP registers without bounds checking? If a userspace program provides extremely large dimensions that exceed the sensor frame size, could it cause the ISP hardware to hang indefinitely while waiting for lines that will never arrive?
[ ... ]
Quoted 3 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c +3
-0+static void rkisp2_params_hist_big (struct rkisp2_params *params, +union rkisp2_params_block block) +{
[ ... ]
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c static void rkisp2_params_crop_init(struct rkisp2_params *params) · line 730
Quoted 8 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c +8
-0+for (i =0 ; i <16 ; i++) { +rkisp2_write (params->rkisp2, reg_base +ISP_RAWHIST_BIG_WEIGHT_BASE +4 * i, +ISP_RAWHIST_BIG_WEIGHT_WND0 (arg->weights[5 * i +0 ]) | +ISP_RAWHIST_BIG_WEIGHT_WND1 (arg->weights[5 * i +1 ]) | +ISP_RAWHIST_BIG_WEIGHT_WND2 (arg->weights[5 * i +2 ]) | +ISP_RAWHIST_BIG_WEIGHT_WND3 (arg->weights[5 * i +3 ]) | +ISP_RAWHIST_BIG_WEIGHT_WND4 (arg->weights[5 * i +4 ])); + }
[Severity: Medium]
Is this loop intentionally truncated? The rkisp2_params_hist_big struct defines the weights array size as RKISP2_ISP_HIST_WEIGHT_GRIDS_SIZE_BIG (which is 225), but this loop only runs 16 times, writing 80 weights. Could this leave 145 weights unprogrammed and corrupt the histogram output?
[ ... ]
Quoted 3 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c +3
-0+static void rkisp2_params_awb_meas (struct rkisp2_params *params, +union rkisp2_params_block block) +{
[ ... ]
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c static void rkisp2_params_crop_init(struct rkisp2_params *params) · line 786
Quoted 8 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c +8
-0+for (i =0 ; i <RKISP2_ISP_AWB_COUNTS_SIZE /5 ; i++) { + val = (arg->weights[5 * i] &0x3f ) + | ((arg->weights[5 * i +1 ] &0x3f ) <<6 ) + | ((arg->weights[5 * i +2 ] &0x3f ) <<12 ) + | ((arg->weights[5 * i +3 ] &0x3f ) <<18 ) + | ((arg->weights[5 * i +4 ] &0x3f ) <<24 ); +rkisp2_write (params->rkisp2,ISP21_RAWAWB_WRAM_DATA_BASE , val); + }
[Severity: Medium]
Should the internal WRAM address pointer be reset before writing these values? Since data is written sequentially to ISP21_RAWAWB_WRAM_DATA_BASE, the pointer might start at whatever leftover value it had from previous operations if ISP21_RAWAWB_RAM_CTRL isn't cleared first, potentially misaligning the parameters.
Quoted 4 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c new file +0
-0@@ -0,0 +1,509 @@
[ ... ]
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c line 33
Quoted 9 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c new file +9
-0+static const struct v4l2_isp_stats_block_type_info +rkisp2_stats_blocks_info[] = { +RKISP2_STATS_BLOCK_INFO (AE_LITE , ae_lite), +RKISP2_STATS_BLOCK_INFO (HIST_LITE , hist), +RKISP2_STATS_BLOCK_INFO (HIST_BIG0 , hist), +RKISP2_STATS_BLOCK_INFO (HIST_BIG1 , hist), +RKISP2_STATS_BLOCK_INFO (HIST_BIG1 , hist), +RKISP2_STATS_BLOCK_INFO (AWB , awb), +};
[Severity: High]
Is the duplicate HIST_BIG1 entry here a typo? It looks like HIST_BIG2 is missing, which leaves its block type uninitialized and implicitly sized at 0.
If rkisp2_stats_hist_read() processes RKISP2_STATS_BLOCK_HIST_BIG2, would adding a block with a size of 0 cause userspace parsers to spin in an infinite loop since their block offset logic will never advance?
[ ... ]
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c line 162
Quoted 10 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c new file +10
-0+static int rkisp2_stats_vb2_start_streaming (struct vb2_queue *vq,unsigned int count) +{ +struct rkisp2_stats *stats = vq->drv_priv; + + stats->icr =0 ; + stats->cur_buf =NULL ; + stats->awb_window_offset =0 ; + +return 0 ; +}
[Severity: High]
Does initializing stats->icr to 0 here need to be protected by stats->lock?
If userspace calls VIDIOC_STREAMON while the ISP is already running and generating EOF interrupts, could this race with rkisp2_stats_isr_eof()? This might clobber the bits accumulated by the ISR, causing it to write 0 to ISP_ISP3A_ICR and fail to clear the pending hardware interrupts, leading to an IRQ storm.
[ ... ]
Quoted 4 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c new file +4
-0+static void rkisp2_stats_get_rawae_lite (struct rkisp2_stats *stats, +u32 status, +struct v4l2_isp_buffer *pbuf) +{
[ ... ]
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c line 230
Quoted 10 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c new file +10
-0+ block =rkisp2_init_stats_block (rkisp2, pbuf, +RKISP2_STATS_BLOCK_AE_LITE ); +if (IS_ERR (block)) +return ; + + ae_lite = &block->ae_lite; + +/* TODO figure out what the other channels are for */ +if (!(status &ISP2X_3A_RAWAE_CH0 )) +return ;
[Severity: Medium]
Could this early return expose uninitialized kernel memory to userspace?
Since rkisp2_init_stats_block() allocates the block and increments buf->data_size, returning here without filling the payload or initializing the done flag might cause userspace to consume stale or poisoned statistics data from the buffer. This same pattern appears to be present in rkisp2_stats_get_rawawb() below.
[ ... ]
Quoted 4 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c new file +4
-0+static void rkisp2_stats_get_rawawb (struct rkisp2_stats *stats, +u32 status, +struct v4l2_isp_buffer *pbuf) +{
[ ... ]
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c line 360
Quoted 8 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c new file +8
-0+for (i =0 ; i <RKISP2_ISP_AWB_COUNTS_SIZE ; i++) { + val1 =rkisp2_read (rkisp2,ISP21_RAWAWB_RAM_DATA_BASE ); + val2 =rkisp2_read (rkisp2,ISP21_RAWAWB_RAM_DATA_BASE ); + awb->counts_r[i] = (val2 >>4 ) &0x3ffff ; + awb->counts_g[i] = (val1 >>18 ) | ((val2 &0xf ) <<14 ); + awb->counts_b[i] = val1 &0x3ffff ; + awb->counts_w[i] = val2 >>22 ; + }
[Severity: Medium]
Should the internal address pointer be reset before reading these AWB counts? It appears this reads from the data port sequentially without first writing 0 to ISP21_RAWAWB_RAM_CTRL, which might cause the read pointer to start at a leftover value from previous operations and corrupt the stats output.
[ ... ]
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c line 410
Quoted 8 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c new file +8
-0+static void rkisp2_stats_send_measurement_3a (struct rkisp2_stats *stats,u32 status) +{ +struct v4l2_isp_buffer *cur_stat_buf; +struct rkisp2_buffer *cur_buf; + + cur_buf =rkisp2_stats_get_buf (stats); +if (!cur_buf) +return ;
[Severity: High]
If no V4L2 stats buffers are queued, does this early return bypass the hardware interrupt acknowledgment routine?
If the ISP is running and asserting 3A interrupts on EOF, skipping the write to ISP_ISP3A_ICR could leave the IRQ line asserted indefinitely, resulting in an infinite IRQ storm and system hang.
[ ... ]
Quoted 2 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c new file +2
-0+int rkisp2_stats_register (struct rkisp2_device *rkisp2) +{
[ ... ]
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c line 474
Quoted 5 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c new file +5
-0+ vdev->device_caps =V4L2_CAP_META_CAPTURE |V4L2_CAP_STREAMING ; + vdev->vfl_dir =VFL_DIR_RX ; +rkisp2_stats_init_vb2_queue (vdev->queue, stats); +rkisp2_init_stats (stats); +video_set_drvdata (vdev, stats);
[Severity: High]
Is it safe to ignore the return value of rkisp2_stats_init_vb2_queue() here? If the queue initialization fails due to memory pressure, allowing the registration to proceed could register the video node with a broken vb2_queue, potentially leading to crashes when queue ioctls are invoked.
[ ... ]
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c line 509
Quoted 13 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c new file +13
-0+void rkisp2_stats_unregister (struct rkisp2_device *rkisp2) +{ +struct rkisp2_stats *stats = &rkisp2->stats; +struct rkisp2_vdev_node *node = &stats->vnode; +struct video_device *vdev = &node->vdev; + +if (!stats->rkisp2) +return ; + +vb2_video_unregister_device (vdev); +media_entity_cleanup (&vdev->entity); +mutex_destroy (&node->vlock); +}
[Severity: High]
Can destroying the mutex here lead to a use-after-free?
The mutex_destroy() is being called synchronously during device unregistration. If a userspace process still holds an open file descriptor to the stats node, it could later call vb2_fop_release() upon closing the file descriptor, which will attempt to acquire this destroyed mutex.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Re: [PATCH v3 3/5] media: rkisp2: Add statistics capture video node
In reply to Paul ElderQuoted 2 lines · Paul Elder
Implement support for getting statistics out of the ISP by dequeueing statistics buffers from rkisp2.
[...]
Hi Paul,
Comments from running the statistics on the RV1126; the RK3588 statements are by code inspection.
[...]
Quoted 1 line
+++
b/Documentation/userspace-api/media/v4l/metafmt-rkisp2.rst
[...]
Quoted 1 line
+:ref:
rkisp2_stats <rkisp2_stats>metadata capture video node, using
Undefined label, as rkisp2_params in patch 2.
[...]
Quoted 2 lines
Documentation/userspace-api/media/v4l/metafmt-rkisp2.rst +2
-0+ const struct v4l2_isp_stats_block_header *block = + (void*)(stats->data + block_offset);
[...]
Documentation/userspace-api/media/v4l/metafmt-rkisp2.rst first member and userspace must populate the type member with a value from · line 101
Quoted 2 lines
Documentation/userspace-api/media/v4l/metafmt-rkisp2.rst +2
-0+ printf("ae_lite.exp_mean_r[%u] = 0x%08x\n", + i, ae_lite.exp_mean_r[%i]);
struct v4l2_isp_stats_block_header does not exist, the header is struct v4l2_isp_block_header; and ae_lite is a pointer and %i is not an index (ae_lite->exp_mean_r[i]).
[...]
Quoted 1 line
+++
b/drivers/media/platform/rockchip/rkisp2/rkisp2-params.c
[...]
Quoted 1 line
+static void rkisp2_params_hist_big(struct
rkisp2_params*params,
[...]
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c static void rkisp2_params_crop_init(struct rkisp2_params *params) · line 692
Quoted 3 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-params.c +3
-0+default : +WARN_ONCE (1 ,"invalid big histogram base %x\n" , reg_base); +return ;
reg_base is uninitialised in the default branch (clang warns); block.header->type is what you want to print.
[...]
Quoted 1 line
+++
b/drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c
[...]
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c line 33
Quoted 9 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c new file +9
-0+static const struct v4l2_isp_stats_block_type_info +rkisp2_stats_blocks_info[] = { +RKISP2_STATS_BLOCK_INFO (AE_LITE , ae_lite), +RKISP2_STATS_BLOCK_INFO (HIST_LITE , hist), +RKISP2_STATS_BLOCK_INFO (HIST_BIG0 , hist), +RKISP2_STATS_BLOCK_INFO (HIST_BIG1 , hist), +RKISP2_STATS_BLOCK_INFO (HIST_BIG1 , hist), +RKISP2_STATS_BLOCK_INFO (AWB , awb), +};
HIST_BIG1 twice, and HIST_BIG2 is left with size 0. v4l2_isp_stats_init_block() then writes a zero-sized header for HIST_BIG2 without advancing data_size, the 256 bins are written after it and the AWB block overwrites both, so user space never sees a HIST_BIG2 block. With the entry fixed the block shows up on the RV1126.
[...]
Quoted 1 line
+static void rkisp2_stats_get_rawae_lite(struct
rkisp2_stats*stats,
[...]
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c line 230
Quoted 10 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c new file +10
-0+ block =rkisp2_init_stats_block (rkisp2, pbuf, +RKISP2_STATS_BLOCK_AE_LITE ); +if (IS_ERR (block)) +return ; + + ae_lite = &block->ae_lite; + +/* TODO figure out what the other channels are for */ +if (!(status &ISP2X_3A_RAWAE_CH0 )) +return ;
The block is reserved and its header written before the status check, and v4l2_isp_stats_init_block() does not clear the payload, so when the bit is not set the block goes out with whatever the buffer held from its previous use, done flag included (every frame after AE lite was disabled through the params node, for instance). Reserve the block after the check, or clear done and the means when the measurement is missing. rkisp2_stats_get_rawawb() has the same pattern.
[...]
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c line 432
Quoted 4 lines
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c new file +4
-0+irqreturn_t rkisp2_stats_isr_eof (struct rkisp2_stats *stats) +{ +struct rkisp2_device *rkisp2 = stats->rkisp2; +u32 status =rkisp2_read (rkisp2,ISP_ISP3A_MIS );
Is the 3A interrupt on its own line on the RK3588? On the RV1126 it shares the ISP line: the measurements complete early in the frame, the ISP handler returns IRQ_NONE for the rest of the frame as ISP_MIS is clear, and the line stays asserted until this clears it, about 1400 unhandled interrupts per frame until the kernel disables the line. Leaving ISP3A_IMSC clear (the params handlers set it) and reading ISP3A_RIS here instead works on both.
[...]
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c line 472
Quoted 1 line
drivers/media/platform/rockchip/rkisp2/rkisp2-stats.c new file +1
-0+rkisp2_stats_init_vb2_queue (vdev->queue, stats);
The return value is ignored.
[...]
Quoted 1 line
+++
b/include/uapi/linux/media/rockchip/rkisp2-config.h
[...]
include/uapi/linux/media/rockchip/rkisp2-config.h struct rkisp2_params_crop { · line 596
Quoted 9 lines
include/uapi/linux/media/rockchip/rkisp2-config.h +9
-0+/** +* Same as struct rkisp2_params_hist_lite but for big channel +* +* RkISP2 parameters histogram configuration block. +* Identified by :c:type:`RKISP2_PARAMS_BLOCK_HIST_BIG{0,1,2}`. +* +* @window_num: 0 or 1 for 5x5, 2 or 3 for 15x15 +*/ +struct rkisp2_params_hist_big {
scripts/kernel-doc warns that this is not a kernel-doc comment; it needs the "struct rkisp2_params_hist_big - ..." title and the other members, since the header is pulled into the documentation.
Best regards,
Daniel
PATCH v3 4/5 dt-bindings: media: Add rockchip rkisp2
In reply to Paul ElderAdd documentation for the Rockchip rkisp2 bindings. This is meant to support multiple versions of Rockchip ISPs going forward, including the 2.x series and 3.x series. The current version only adds the compatible for the RK3588, which is a 3.0.
---
Changes in v3:
- reorder patches
No change in v2
--- .../devicetree/bindings/media/rockchip-isp2.yaml | 125 +++++++++++++++++++++ 1 file changed, 125 insertions(+)
Documentation/devicetree/bindings/media/rockchip-isp2.yaml new file +125 -0
@@ -0,0 +1,125 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/media/rockchip-isp2.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Rockchip SoC Image Signal Processing unit v2 + +maintainers: + - Paul Elder <[email protected]> + +description: | + Rockchip ISP2 is the Camera interface for the Rockchip series of SoCs which + contains image processing, scaling, and compression functions. + +properties: + compatible: + enum: + - rockchip,rk3588-isp + + reg: + maxItems: 1 + + interrupts: + minItems: 2 + maxItems: 2 + + interrupt-names: + items: + - const: isp_irq + - const: mi_irq + + clocks: + minItems: 3 + items: + - description: ISP AXI clock (aclk) + - description: ISP AHB clock (hclk) + - description: ISP core clock (isp) + # for rk3588 + - description: ISP core clock (marvin) + - description: ISP core clock (vicap) + + clock-names: + minItems: 3 + items: + - const: aclk + - const: hclk + - const: clk_core + # for rk3588 + - const: clk_core_marvin + - const: clk_core_vicap + + iommus: + maxItems: 1 + + power-domains: + items: + - description: ISP power domain + + ports: + $ref: /schemas/graph.yaml#/properties/ports + + properties: + port@0: + $ref: /schemas/graph.yaml#/$defs/port-base + unevaluatedProperties: false + description: connection point for VICAP in inline mode + + properties: + endpoint: + $ref: video-interfaces.yaml# + unevaluatedProperties: false + + required: + - port@0 + +required: + - compatible + - reg + - interrupts + - clocks + - clock-names + - power-domains + +allOf: + - if: + properties: + compatible: + contains: + const: rockchip,rk3588-isp + then: + properties: + clocks: + minItems: 5 + clock-names: + minItems: 5 + +additionalProperties: false + +examples: + - | + + #include <dt-bindings/clock/rockchip,rk3588-cru.h> + #include <dt-bindings/interrupt-controller/arm-gic.h> + #include <dt-bindings/power/rk3588-power.h> + + parent0: parent { + #address-cells = <2>; + #size-cells = <2>; + + isp0: isp@fdcb0000 { + compatible = "rockchip,rk3588-isp"; + reg = <0x0 0xfdcb0000 0x0 0x7f00>; + interrupts = <GIC_SPI 131 IRQ_TYPE_LEVEL_HIGH 0>, + <GIC_SPI 133 IRQ_TYPE_LEVEL_HIGH 0>; + interrupt-names = "isp_irq", "mi_irq"; + clocks = <&cru ACLK_ISP0>, <&cru HCLK_ISP0>, + <&cru CLK_ISP0_CORE>, <&cru CLK_ISP0_CORE_MARVIN>, + <&cru CLK_ISP0_CORE_VICAP>; + clock-names = "aclk", "hclk", "clk_core", + "clk_core_marvin", "clk_core_vicap"; + power-domains = <&power RK3588_PD_VI>; + iommus = <&isp0_mmu>; + }; + };
Signature · 1 line
2.47.2
[email protected]
AI reviewRe: [PATCH v3 4/5] dt-bindings: media: Add rockchip rkisp2
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Missing `interrupt-names` property in the `required` list allows incomplete device trees to pass…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Missing interrupt-names property in the required list allows incomplete device trees to pass schema validation but fail driver probing. --
commit 6781ced6c9930c8c3220edc52b5934581e2961ae
Author: Paul Elder <[email protected]>
dt-bindings: media: Add rockchip rkisp2
This commit adds device tree binding documentation for the Rockchip ISP2, specifically introducing support for the rk3588 compatible string. It defines the required properties, clocks, interrupts, and ports for the image signal processing unit.
Quoted 4 lines
Documentation/devicetree/bindings/media/rockchip-isp2.yaml new file +0
-0
[ ... ]
Documentation/devicetree/bindings/media/rockchip-isp2.yaml line 83
Quoted 10 lines
Documentation/devicetree/bindings/media/rockchip-isp2.yaml new file +10
-0+ required: + - port@0 + +required: + - compatible + - reg + - interrupts + - clocks + - clock-names + - power-domains
[Severity: Medium]
Should interrupt-names be included in this required list to ensure incomplete device trees do not pass schema validation?
The Rockchip ISP2 driver in drivers/media/platform/rockchip/rkisp2/rkisp2-dev.c iterates over its statically defined rk3588_isp_isrs array, which sets the name field for both interrupts (isp_irq and mi_irq). The driver then unconditionally executes:
info->isrs[i].name ? platform_get_irq_byname(...) : ...
This will fail and return less than 0 if interrupt-names is absent in the device tree, aborting the probe. Adding interrupt-names to the required properties would enforce an accurate hardware description that matches the strict probing requirements of the driver.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Re: [PATCH v3 4/5] dt-bindings: media: Add rockchip rkisp2
In reply to Paul ElderDocumentation/devicetree/bindings/media/rockchip-isp2.yaml
Quoted 21 lines · Paul Elder
Add documentation for the Rockchip rkisp2 bindings. This is meant to support multiple versions of Rockchip ISPs going forward, including the 2.x series and 3.x series. The current version only adds the compatible for the RK3588, which is a 3.0.
Signed-off-by Paul Elder <[email protected]>---
Changes in v3:
- reorder patchesNo change in v2
--- .../devicetree/bindings/media/rockchip-isp2.yaml | 125 +++++++++++++++++++++ 1 file changed, 125 insertions(+)Documentation/devicetree/bindings/media/rockchip-isp2.yaml new file +0
-0
Filename matching compatible please.
Quoted 17 lines
Documentation/devicetree/bindings/media/rockchip-isp2.yaml new file +16
-0@@ -0,0 +1,125 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/media/rockchip-isp2.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Rockchip SoC Image Signal Processing unit v2 + +maintainers: + - Paul Elder <[email protected]> + +description: | + Rockchip ISP2 is the Camera interface for the Rockchip series of SoCs which + contains image processing, scaling, and compression functions. + +properties:
Documentation/devicetree/bindings/media/rockchip-isp2.yaml line 26
Quoted 10 lines
Documentation/devicetree/bindings/media/rockchip-isp2.yaml new file +10
-0+ compatible: + enum: + - rockchip,rk3588-isp + + reg: + maxItems: 1 + + interrupts: + minItems: 2 + maxItems: 2
When they match, you only need one.
An items list however might be nice here to explain what the interrupts actually are.
Documentation/devicetree/bindings/media/rockchip-isp2.yaml line 31
Quoted 5 lines
Documentation/devicetree/bindings/media/rockchip-isp2.yaml new file +5
-0+ + interrupt-names: + items: + - const: isp_irq + - const: mi_irq
They're interrupts by definition, having _irq is redundant.
Documentation/devicetree/bindings/media/rockchip-isp2.yaml line 34
Quoted 3 lines
Documentation/devicetree/bindings/media/rockchip-isp2.yaml new file +3
-0+ + clocks: + minItems: 3
You support one device, which has all 5. Drop this.
Documentation/devicetree/bindings/media/rockchip-isp2.yaml line 39
Quoted 5 lines
Documentation/devicetree/bindings/media/rockchip-isp2.yaml new file +5
-0+ items: + - description: ISP AXI clock (aclk) + - description: ISP AHB clock (hclk) + - description: ISP core clock (isp) + # for rk3588
Commentary like this is redundant, you have the explicit conditional below, and this will become a mess with many devices.
Documentation/devicetree/bindings/media/rockchip-isp2.yaml line 51
Quoted 12 lines
Documentation/devicetree/bindings/media/rockchip-isp2.yaml new file +12
-0+ - description: ISP core clock (marvin) + - description: ISP core clock (vicap) + + clock-names: + minItems: 3 + items: + - const: aclk + - const: hclk + - const: clk_core + # for rk3588 + - const: clk_core_marvin + - const: clk_core_vicap
isp, marvin, vicap.
Quoted 19 lines
Documentation/devicetree/bindings/media/rockchip-isp2.yaml new file +19
-0+ + iommus: + maxItems: 1 + + power-domains: + items: + - description: ISP power domain + + ports: + $ref: /schemas/graph.yaml#/properties/ports + + properties: + port@0: + $ref: /schemas/graph.yaml#/$defs/port-base + unevaluatedProperties: false + description: connection point for VICAP in inline mode + + properties: + endpoint:
Documentation/devicetree/bindings/media/rockchip-isp2.yaml line 80
Quoted 10 lines
Documentation/devicetree/bindings/media/rockchip-isp2.yaml new file +10
-0+ $ref: video-interfaces.yaml# + unevaluatedProperties: false + + required: + - port@0 + +required: + - compatible + - reg + - interrupts
If you want to ever be able to actually use interrupt-names, make it required too.
Quoted 21 lines
Documentation/devicetree/bindings/media/rockchip-isp2.yaml new file +21
-0+ - clocks + - clock-names + - power-domains + +allOf: + - if: + properties: + compatible: + contains: + const: rockchip,rk3588-isp + then: + properties: + clocks: + minItems: 5 + clock-names: + minItems: 5 + +additionalProperties: false + +examples: + - |
Documentation/devicetree/bindings/media/rockchip-isp2.yaml line 111
Quoted 10 lines
Documentation/devicetree/bindings/media/rockchip-isp2.yaml new file +10
-0+ + #include <dt-bindings/clock/rockchip,rk3588-cru.h> + #include <dt-bindings/interrupt-controller/arm-gic.h> + #include <dt-bindings/power/rk3588-power.h> + + parent0: parent { + #address-cells = <2>; + #size-cells = <2>; + + isp0: isp@fdcb0000 {
Drop the unused label ehre please.
pw-bot: changes-requested
Thanks,
Conor.
Quoted 18 lines
Documentation/devicetree/bindings/media/rockchip-isp2.yaml new file +14
-0+ compatible = "rockchip,rk3588-isp"; + reg = <0x0 0xfdcb0000 0x0 0x7f00>; + interrupts = <GIC_SPI 131 IRQ_TYPE_LEVEL_HIGH 0>, + <GIC_SPI 133 IRQ_TYPE_LEVEL_HIGH 0>; + interrupt-names = "isp_irq", "mi_irq"; + clocks = <&cru ACLK_ISP0>, <&cru HCLK_ISP0>, + <&cru CLK_ISP0_CORE>, <&cru CLK_ISP0_CORE_MARVIN>, + <&cru CLK_ISP0_CORE_VICAP>; + clock-names = "aclk", "hclk", "clk_core", + "clk_core_marvin", "clk_core_vicap"; + power-domains = <&power RK3588_PD_VI>; + iommus = <&isp0_mmu>; + }; + };Signature · 2 lines
-- 2.47.2
PATCH v3 5/5 arm64: dts: rockchip: add ISP nodes to rk3588
In reply to Paul ElderFrom: Xu Hongfei <[email protected]>
Add device tree nodes for the ISP and their iommus on the RK3588.
---
Changes in v3:
- reorder patches
No change in v2
--- arch/arm64/boot/dts/rockchip/rk3588-base.dtsi | 57 +++++++++++++++++++++++++++ 1 file changed, 57 insertions(+)
arch/arm64/boot/dts/rockchip/rk3588-base.dtsi +57 -0
@@ -3590,6 +3590,63 @@ gpio4: gpio@fec50000 { #interrupt-cells = <2>; }; }; + + isp0: isp@fdcb0000 { + compatible = "rockchip,rk3588-isp"; + reg = <0x0 0xfdcb0000 0x0 0x7f00>; + interrupts = <GIC_SPI 131 IRQ_TYPE_LEVEL_HIGH 0>, + <GIC_SPI 133 IRQ_TYPE_LEVEL_HIGH 0>, + <GIC_SPI 134 IRQ_TYPE_LEVEL_HIGH 0>; + interrupt-names = "isp_irq", "mi_irq"; + clocks = <&cru ACLK_ISP0>, <&cru HCLK_ISP0>, + <&cru CLK_ISP0_CORE>, <&cru CLK_ISP0_CORE_MARVIN>, + <&cru CLK_ISP0_CORE_VICAP>; + clock-names = "aclk", "hclk", "clk_core", + "clk_core_marvin", "clk_core_vicap"; + power-domains = <&power RK3588_PD_VI>; + iommus = <&isp0_mmu>; + status = "disabled"; + }; + + isp0_mmu: iommu@fdcb7f00 { + compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu"; + reg = <0x0 0xfdcb7f00 0x0 0x100>; + interrupts = <GIC_SPI 132 IRQ_TYPE_LEVEL_HIGH 0>; + clocks = <&cru ACLK_ISP0>, <&cru HCLK_ISP0>; + clock-names = "aclk", "iface"; + power-domains = <&power RK3588_PD_VI>; + #iommu-cells = <0>; + rockchip,disable-mmu-reset; + status = "disabled"; + }; + + isp1: isp@fdcc0000 { + compatible = "rockchip,rk3588-isp"; + reg = <0x0 0xfdcc0000 0x0 0x7f00>; + interrupts = <GIC_SPI 135 IRQ_TYPE_LEVEL_HIGH 0>, + <GIC_SPI 137 IRQ_TYPE_LEVEL_HIGH 0>; + interrupt-names = "isp_irq", "mi_irq"; + clocks = <&cru ACLK_ISP1>, <&cru HCLK_ISP1>, + <&cru CLK_ISP1_CORE>, <&cru CLK_ISP1_CORE_MARVIN>, + <&cru CLK_ISP1_CORE_VICAP>; + clock-names = "aclk", "hclk", "clk_core", + "clk_core_marvin", "clk_core_vicap"; + power-domains = <&power RK3588_PD_ISP1>; + iommus = <&isp1_mmu>; + status = "disabled"; + }; + + isp1_mmu: iommu@fdcc7f00 { + compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu"; + reg = <0x0 0xfdcc7f00 0x0 0x100>; + interrupts = <GIC_SPI 136 IRQ_TYPE_LEVEL_HIGH 0>; + clocks = <&cru ACLK_ISP1>, <&cru HCLK_ISP1>; + clock-names = "aclk", "iface"; + power-domains = <&power RK3588_PD_ISP1>; + #iommu-cells = <0>; + rockchip,disable-mmu-reset; + status = "disabled"; + }; }; #include "rk3588-base-pinctrl.dtsi"
Signature · 1 line
2.47.2
[email protected]
AI reviewRe: [PATCH v3 5/5] arm64: dts: rockchip: add ISP nodes to rk3588
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Low] Mismatch between the number of interrupts and interrupt-names in the isp0 node, violating the…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Low] Mismatch between the number of interrupts and interrupt-names in the isp0 node, violating the dt-binding schema.
- [Low] The newly added ISP and IOMMU nodes are not sorted by their unit address in the .dtsi file. --
commit 9cc7cdafa66ffa2298585c4888f169256dd48661
Author: Xu Hongfei <[email protected]>
arm64: dts: rockchip: add ISP nodes to rk3588
Add device tree nodes for the ISP and their iommus on the RK3588.
arch/arm64/boot/dts/rockchip/rk3588-base.dtsi gpio4: gpio@fec50000 { · line 3595
Quoted 11 lines
arch/arm64/boot/dts/rockchip/rk3588-base.dtsi +3
-0@@ -3590,6 +3590,63 @@ gpio4: gpio@fec50000 { #interrupt-cells = <2>; }; }; + + isp0: isp@fdcb0000 { + compatible = "rockchip,rk3588-isp";
[Severity: Low]
Are these new ISP and IOMMU nodes sorted by their unit address in the .dtsi file?
The standard convention is to order sibling nodes strictly by their unit address, but placing isp@fdcb0000 after gpio@fec50000 breaks this chronological ordering.
arch/arm64/boot/dts/rockchip/rk3588-base.dtsi gpio4: gpio@fec50000 { · line 3600
Quoted 5 lines
arch/arm64/boot/dts/rockchip/rk3588-base.dtsi +5
-0+ reg = <0x0 0xfdcb0000 0x0 0x7f00>; + interrupts = <GIC_SPI 131 IRQ_TYPE_LEVEL_HIGH 0>, + <GIC_SPI 133 IRQ_TYPE_LEVEL_HIGH 0>, + <GIC_SPI 134 IRQ_TYPE_LEVEL_HIGH 0>; + interrupt-names = "isp_irq", "mi_irq";
[Severity: Low]
Is there a mismatch here between the number of items in the interrupts property and the interrupt-names property?
The rockchip,rk3588-isp dt-binding schema strictly enforces a maximum of 2 items for the interrupts property. Including three interrupts will result in a schema validation error when building the devicetree with dtbs_check.
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Re: [PATCH v3 5/5] arm64: dts: rockchip: add ISP nodes to rk3588
In reply to Paul ElderQuoted 3 lines · Paul Elder
From: Xu Hongfei <[email protected]>
Add device tree nodes for the ISP and their iommus on the RK3588.
[...]
arch/arm64/boot/dts/rockchip/rk3588-base.dtsi gpio4: gpio@fec50000 { · line 3600
Quoted 7 lines
arch/arm64/boot/dts/rockchip/rk3588-base.dtsi +7
-0+ isp0: isp@fdcb0000 { + compatible = "rockchip,rk3588-isp"; + reg = <0x0 0xfdcb0000 0x0 0x7f00>; + interrupts = <GIC_SPI 131 IRQ_TYPE_LEVEL_HIGH 0>, + <GIC_SPI 133 IRQ_TYPE_LEVEL_HIGH 0>, + <GIC_SPI 134 IRQ_TYPE_LEVEL_HIGH 0>; + interrupt-names = "isp_irq", "mi_irq";
Three interrupts with two names, while the binding in patch 4 has
maxItems: 2 and isp1 below lists two, so dtbs_check rejects isp0. If 134 is the MIPI line it needs a name and an entry in the binding (the RV1126 has that third line and the driver can use it for the CSI-2 receiver errors); otherwise drop it.
Best regards,
Daniel
Quoted 5 lines · Paul Elder
Hello everyone!
This patch series adds a new rkisp2 driver that aims to support all Rockchip 2.x and 3.x series ISPs. Here we add support for the version on the RK3588, which as far as I'm aware is a 3.0.
[...]
Hi Paul,
I have been running this series on the RV1126, whose ISP is a 2.0, with an IMX415 at 1920x1080 and 30 fps, through libcamera (your rkisp2 pipeline handler with a few changes) and through the raw readback path fed from the VICAP driver. Most of the series works there as it is. Below is a summary of the comments I sent on the individual patches, followed by the list of changes the 2.0 needed, so that you can see which of them are differences between the ISP versions and which may affect the RK3588 as well. I have no RK3588, so every statement about the 3.0 is by code inspection.
Summary of the per-patch comments
---------------------------------
Patch 1, driver:
- The ISP is not started when a raw readback node is started before the main or self path: pipe.start_count counts the readback nodes.
- The CR plane base address is never written, so the three-plane formats write their third plane to an unprogrammed address.
- The 4:2:0 capture formats name YUYV8_1_5X8 as media bus code, which the ISP source pads do not offer, so link validation fails for them.
- The MPFBC_CTRL format list has NV21M twice and lacks NV12M and YUV420.
- V4L2_ISP is not selected although patches 2 and 3 need it; V4L2_FWNODE and GENERIC_PHY_MIPI_DPHY are selected without a user.
- A full quantization range accepted on the source pad is never applied to the hardware (CSM_Y_FULL_ENA and CSM_C_FULL_ENA).
- The raw readback stop path releases the buffers without waiting for the read in progress.
- RKISP1_V30 is added to the rkisp1 uapi without a user, plus a few small items.
Patch 2, params:
- The params buffer consumed at stream start is never returned to user space.
- The resizer bypass test compares the luma size only, so an unscaled 4:2:0 output gets one chroma row per luma row.
- The GOC handler tests mode against the segments enum.
- The resizer is left unprogrammed when the first params buffer has no CROP block, and the default crop without a params buffer is the top-left window of the frame.
- A missing cleanup on a registration failure and an undefined label in the documentation.
Patch 3, stats:
- The stats block table lists HIST_BIG1 twice, so HIST_BIG2 is lost.
- The AE lite and AWB blocks go out with stale payload and done flag when their status bit is not set.
- Smaller items, and a question about the 3A interrupt line.
Patch 5, dts:
- The isp0 node has three interrupts while the binding allows two.
Changes needed for the RV1126
-----------------------------
Full (draft-quality) tree at
https://github.com/dangowrt/linux/tree/rk-isp2-rv1126
These are commits on top of your v3. "2.0" below means what I observed on the RV1126; where I know whether the 3.0 shares it, I say so.
1. Model description for the RV1126: three clocks, a third interrupt line for the CSI-2 receiver and raw read status (the handler only counts the receiver errors), 4416x3312 as maximum input size, plus the binding and dts entries. RV1126 specific.
2. SWS_CFG: keep the reset value and add only the frame drop disable bit; ACK_FRM_PRO_DIS is not defined on the 2.0. Whether the 3.0 holds anything in that register that the current write clears I do not know (asked in patch 1).
3. Extra readback lines: the 2.0 does not finish the last lines of a frame it converts from Bayer to YUV out of memory unless the readback presents 24 lines more than the frame has. The lines are added to the readback buffer and picture size and to the ISP acquisition and output height, and cut off by the dual crop, which stays enabled for the YUV paths when the crop covers the frame. The vendor driver does this for the 2.0 only.
4. No image stabiliser on the 2.0: rkisp2_config_ism() is skipped by a model flag. The RK3588 keeps it, as your comment there says.
5. Bus priority and hurry bits in the upper part of ISP_CTRL, set on the 2.0 as the vendor driver does; without them the readback competes with the camera interface on equal terms. 2.0 specific as far as I know.
6. The word swap of the unpacked readback samples (LE_ALIGN) is made a model property: the RV1126 camera interface writes the samples in the order the 2.0 readback expects, and with the swap the chroma comes out as noise. Strictly this depends on what produced the buffer rather than on the ISP version.
7. VI_ISP_CLK_CTRL: rkisp2_config_clk() writes the register with the ISP1 bit names (ISP_3A | ISP_RAW). The 2.0 layout differs, and its bits 21 and 22 gate the memory read and raw read engines; left clear, the readback delivers exactly one frame after a reset and nothing afterwards. The vendor 2.0 driver writes 0x7fe009. I do not know the 3.0 layout.
8. The readback buffer is handed back from the ISP frame complete interrupt instead of the main or self path frame end: on the 2.0 the main path frame end arrives about 10 ms before the readback engine has finished the frame, so the buffer went back to user space and the next base address was written while the frame was still being read. I do not know where the 3.0 finishes the read relative to the MI frame end.
9. Readback trigger ordering: on the 2.0 the CSI2RX_CTRL0 write that switches SW_IBUF_OP_MODE away from the reset value and sets SW_CSI2RX_EN at the same time loses its trigger in about 7 % of the stream starts (the mode is latched, the enable is dropped, no V_START follows). Writing the op mode with the enable clear before the MI configuration update that rkisp2_cap_stream_enable() issues, and starting the first buffer like every later one (base write, then the enable write), fixed it: 120 of 120 starts. I do not know whether the 3.0 shares this.
10. Resizer chroma subsampling for 4:2:0 output (patch 2 comment).
11. AWB measurement on the 2.0: its RAWAWB block does not have the weight RAM and the count RAM at the addresses the driver uses (the RAM read zeros in every mode I tried), so the AWB stats block stays empty. The 2.0 still has the white balance measurement of the ISP core, the rkisp1 style block, so I program that from the AWB_MEAS parameters (window spanning the 15x15 grid, RGB mode, the maximum limits as thresholds) and report its count and means in a new AWB_MEAN stats block. 2.0 specific.
12. CSM output range from the pad quantization (patch 1 comment).
13. CSC defaults: when user space sets only the colorspace with SET_CSC (libcamera does that for sYCC), I filled the other fields from the colorspace defaults. Re-reading subdev-formats.rst, a DEFAULT field keeps the received value by definition, so your code is right here and this is a libcamera matter.
14. 3A status at frame end (patch 3 question): leaving ISP3A_IMSC clear and reading ISP3A_RIS in rkisp2_stats_isr_eof() brought the ISP interrupt count from about 1400 per frame down to two.
15. Default crop covering the whole frame (patch 2 comment).
16. vb2_dma_contig_set_max_seg_size(dev, DMA_BIT_MASK(32)) in probe:
the arm32 IOMMU DMA ops map an imported dmabuf in segments of the device's maximum segment size with separate IOVA allocations, so importing the readback buffers into the main path failed as not contiguous. Harmless on arm64; rkvdec does the same.
17. ISP reset on stop: on the 2.0 the IRCL ISP and MIPI reset pulse in rkisp2_isp_stop() does not clear the raw readback front end, and the next stream starts with the previous CSI2RX state and misses its first frame. The "isp" and "isp-rx-p" reset lines are asserted instead, as the vendor driver does in its stop path. RV1126 specific.
18. HIST_BIG2 stats table entry (patch 3 comment).
19. VICAP (rkcif) side: the unpacked raw output is written MSB aligned as the readback expects, the MIPI interrupts are masked while the receiver is powered, and a larger sizeimage is accepted so that the buffers carry the extra lines of item 3. The frames are handed to the readback inside the kernel through a small consumer interface of the rkcif driver, selected when the readback node's link to the ISP is disabled; this replaces the user space copy of every raw frame into the readback node that I started with. It is the RV1126 counterpart of the inline mode pad you reserved, except that the frames still go through memory.
Tested configuration: RV1126 with an IMX415 at 1920x1080 and 30 fps. libcamera with cam and with GStreamer (libcamerasrc into the hantro JPEG encoder), AE and AWB converging, two minutes at a constant 30 fps; the readback path alone, 1974 frames in 66 s; 120 of 120 stream starts delivering frames. The self path, 4K and raw output I have not tested.
Once the series has settled I can post the remainder of the above as follow-up patches on top of it, if that suits you. The libcamera side (VICAP entities taken from the media graph, start order, colour space propagation, IMX415 tuning) is on a branch here as well and I can send it to libcamera-devel.
Best regards,
Daniel