Skip to contentSkip to conversation

Add AMD ISP4 driver

PATCH Superseded Bin Du 6 participants 13 messages Aug 28, 2025 – Sep 10, 2025 Media (V4L/DVB)

v3 of 11 Superseded Latest is v11 →

lore
Share & tools
lore

Bin Du

author
#0 lore raw

v3 cover Add AMD ISP4 driver

Hello,

AMD ISP4 is the AMD image processing gen 4 which can be found in HP ZBook Ultra G1a 14 inch Mobile Workstation PC ( Ryzen AI Max 385)
(https://ubuntu.com/certified/202411-36043)
This patch series introduces the initial driver support for the AMD ISP4.

Patch summary:
- Powers up/off and initializes ISP HW
- Configures and kicks off ISP FW
- Interacts with APP using standard V4l2 interface by video node
- Controls ISP HW and interacts with ISP FW to do image processing
- Support enum/set output image format and resolution
- Support queueing buffer from app and dequeueing ISP filled buffer to App
- It supports libcamera ver0.2 SimplePipeline
- It is verified on qv4l2, cheese and qcam
- It is verified together with following patches

	platform/x86: Add AMD ISP platform config (https://lore.kernel.org/all/[email protected]/)
	pinctrl: amd: isp411: Add amdisp GPIO pinctrl (https://github.com/torvalds/linux/commit/e97435ab09f3ad7b6a588dd7c4e45a96699bbb4a)
	drm/amd/amdgpu: Add GPIO resources required for amdisp (https://gitlab.freedesktop.org/agd5f/linux/-/commit/ad0f5966ed8297aa47b3184192b00b7379ae0758)

AMD ISP4 Key features:
- Processes bayer raw data from the connected sensor and output them to different YUV formats
- Downscale input image to different output image resolution
- Pipeline to do image processing on the input image including demosaic, denoise, 3A, etc

----------

Changes v2 -> v3:

- All the dependent patches in other modules (drm/amd/amdgpu, platform/x86, pinctrl/amd) merged on upstream mainline kernel (https://github.com/torvalds/linux) v6.17.
- Removed usage of amdgpu structs in ISP driver. Added helper functions in amdgpu accepting opaque params from ISP driver to allocate and release ISP GART buffers.
- Moved sensor and MIPI phy control entirely into ISP FW instead of the previous hybrid approach controlling sensor from both FW and x86 (sensor driver).
- Removed phy configuration and sensor binding as x86 (sensor driver) had relinquished the sensor control for ISP FW. With this approach the driver will be exposed as web camera like interface.
- New FW with built-in sensor driver is submitted on upstream linux-firmware repo (https://gitlab.com/kernel-firmware/linux-firmware/).
- Please note the new FW submitted is not directly compatible with OEM Kernel ISP4.0 (https://github.com/amd/Linux_ISP_Kernel/tree/4.0) and the previous ISP V2 patch series.
- If intend to use the new FW, please rebuild OEM ISP4.0 Kernel with CONFIG_VIDEO_OV05C10=N and CONFIG_PINCTRL_AMDISP=Y.
- Included critical fixes from Sultan Alsawaf branch (https://github.com/kerneltoast/kernel_x86_laptop.git) related to managing lifetime of isp buffers.
media: amd: isp4: Add missing refcount tracking to mmap memop
media: amd: isp4: Don't put or unmap the dmabuf when detaching
media: amd: isp4: Don't increment refcount when dmabuf export fails
media: amd: isp4: Fix possible use-after-free in isp4vid_vb2_put()
media: amd: isp4: Always export a new dmabuf from get_dmabuf memop
media: amd: isp4: Fix implicit dmabuf lifetime tracking
media: amd: isp4: Fix possible use-after-free when putting implicit dmabuf
media: amd: isp4: Simplify isp4vid_get_dmabuf() arguments
media: amd: isp4: Move up buf->vaddr check in isp4vid_get_dmabuf()
media: amd: isp4: Remove unused userptr memops
media: amd: isp4: Add missing cleanup on error in isp4vid_vb2_alloc()
media: amd: isp4: Release queued buffers on error in start_streaming
- Addressed all code related upstream comments
- Fix typo errors and other cosmetic issue.

Changes v1 -> v2:

- Fix media CI test errors and valid warnings
- Reduce patch number in the series from 9 to 8 by merging MAINTAINERS adding patch to the first patch
- In patch 5
- do modification to use remote endpoint instead of local endpoint
- use link frequency and port number as start phy parameter instead of extra added phy-id and phy-bit-rate property of endpoint

----------

It passes v4l2 compliance test, the test reports for:

(a) amd_isp_capture device /dev/video0

Compliance test for amd_isp_capture device /dev/video0:
-------------------------------------------------------

atg@atg-HP-PV:~/bin$ ./v4l2-compliance -d /dev/video0
v4l2-compliance 1.29.0-5348, 64 bits, 64-bit time_t
v4l2-compliance SHA: 75e3f0e2c2cb 2025-03-17 18:12:17

Compliance test for amd_isp_capture device /dev/video0:

Driver Info:

        Driver name      : amd_isp_capture
        Card type        : amd_isp_capture
        Bus info         : platform:amd_isp_capture
        Driver version   : 6.14.0
        Capabilities     : 0xa4200001

Video Capture
I/O MC
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x24200001
Video Capture
I/O MC
Streaming
Extended Pix Format
Media Driver Info:

        Driver name      : amd_isp_capture
        Model            : amd_isp41_mdev
        Serial           :
        Bus info         : platform:amd_isp_capture
        Media version    : 6.14.0
        Hardware revision: 0x00000000 (0)
        Driver version   : 6.14.0

Interface Info:
ID : 0x03000005
Type : V4L Video
Entity Info:

        ID               : 0x00000003 (3)
        Name             : Preview
        Function         : V4L2 I/O
        Pad 0x01000004   : 0: Sink
          Link 0x02000007: from remote pad 0x1000002 of entity 'amd isp4' (Image Signal Processor): Data, Enabled, Immutable

Required ioctls:
test MC information (see 'Media Driver Info' above): OK test VIDIOC_QUERYCAP: OK
test invalid ioctls: OK

Allow for multiple opens:
test second /dev/video0 open: OK
test VIDIOC_QUERYCAP: OK
test VIDIOC_G/S_PRIORITY: OK
test for unlimited opens: OK

Debug ioctls:
test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK
test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 1 Audio Inputs: 0 Tuners: 0

Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_ENUMAUDOUT: OK (Not Supported) test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls (Input 0):
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported) test VIDIOC_QUERYCTRL: OK (Not Supported)
test VIDIOC_G/S_CTRL: OK (Not Supported)
test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported) test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported) test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) Standard Controls: 0 Private Controls: 0

Format ioctls (Input 0):
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK test VIDIOC_G/S_PARM: OK
test VIDIOC_G_FBUF: OK (Not Supported) test VIDIOC_G_FMT: OK
test VIDIOC_TRY_FMT: OK
test VIDIOC_S_FMT: OK
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported) test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK (Not Supported)

Codec ioctls (Input 0):
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported) test VIDIOC_G_ENC_INDEX: OK (Not Supported) test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls (Input 0):
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK test CREATE_BUFS maximum buffers: OK test VIDIOC_REMOVE_BUFS: OK
test VIDIOC_EXPBUF: OK
test Requests: OK (Not Supported) test blocking wait: OK

Total for amd_isp_capture device /dev/video0: 49, Succeeded: 49, Failed: 0, Warnings: 0

Please review and provide feedback.

Many thanks,

Bin Du (7):
media: platform: amd: Introduce amd isp4 capture driver
media: platform: amd: low level support for isp4 firmware
media: platform: amd: Add isp4 fw and hw interface
media: platform: amd: isp4 subdev and firmware loading handling added
media: platform: amd: isp4 video node and buffers handling added
media: platform: amd: isp4 debug fs logging and more descriptive
errors
Documentation: add documentation of AMD isp 4 driver

 Documentation/admin-guide/media/amdisp4-1.rst |   66 +
 Documentation/admin-guide/media/amdisp4.dot   |    8 +
 .../admin-guide/media/v4l-drivers.rst         |    1 +
 MAINTAINERS                                   |   25 +
 drivers/media/platform/Kconfig                |    1 +
 drivers/media/platform/Makefile               |    1 +
 drivers/media/platform/amd/Kconfig            |    3 +
 drivers/media/platform/amd/Makefile           |    3 +
 drivers/media/platform/amd/isp4/Kconfig       |   13 +
 drivers/media/platform/amd/isp4/Makefile      |   10 +
 drivers/media/platform/amd/isp4/isp4.c        |  237 ++++
 drivers/media/platform/amd/isp4/isp4.h        |   26 +
 drivers/media/platform/amd/isp4/isp4_debug.c  |  272 ++++
 drivers/media/platform/amd/isp4/isp4_debug.h  |   41 +
 .../platform/amd/isp4/isp4_fw_cmd_resp.h      |  314 +++++
 drivers/media/platform/amd/isp4/isp4_hw_reg.h |  125 ++
 .../media/platform/amd/isp4/isp4_interface.c  |  972 +++++++++++++
 .../media/platform/amd/isp4/isp4_interface.h  |  149 ++
 drivers/media/platform/amd/isp4/isp4_subdev.c | 1198 ++++++++++++++++
 drivers/media/platform/amd/isp4/isp4_subdev.h |  133 ++
 drivers/media/platform/amd/isp4/isp4_video.c  | 1213 +++++++++++++++++
 drivers/media/platform/amd/isp4/isp4_video.h  |   87 ++
 22 files changed, 4898 insertions(+)
 create mode 100644 Documentation/admin-guide/media/amdisp4-1.rst
 create mode 100644 Documentation/admin-guide/media/amdisp4.dot
 create mode 100644 drivers/media/platform/amd/Kconfig
 create mode 100644 drivers/media/platform/amd/Makefile
 create mode 100644 drivers/media/platform/amd/isp4/Kconfig
 create mode 100644 drivers/media/platform/amd/isp4/Makefile
 create mode 100644 drivers/media/platform/amd/isp4/isp4.c
 create mode 100644 drivers/media/platform/amd/isp4/isp4.h
 create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.c
 create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.h
 create mode 100644 drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
 create mode 100644 drivers/media/platform/amd/isp4/isp4_hw_reg.h
 create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.c
 create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.h
 create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c
 create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h
 create mode 100644 drivers/media/platform/amd/isp4/isp4_video.c
 create mode 100644 drivers/media/platform/amd/isp4/isp4_video.h
Signature · 1 line
2.34.1
#1 lore raw

Hi Bin Bu,

Have you sent out the cover letter only ? I haven't received the rest of the series, and it's not found on lore.kernel.org either.

Quoted 239 lines · Bin Du

Hello,

AMD ISP4 is the AMD image processing gen 4 which can be found in HP ZBook Ultra G1a 14 inch Mobile Workstation PC ( Ryzen AI Max 385)
(https://ubuntu.com/certified/202411-36043)
This patch series introduces the initial driver support for the AMD ISP4.

Patch summary:
- Powers up/off and initializes ISP HW
- Configures and kicks off ISP FW
- Interacts with APP using standard V4l2 interface by video node
- Controls ISP HW and interacts with ISP FW to do image processing
- Support enum/set output image format and resolution
- Support queueing buffer from app and dequeueing ISP filled buffer to App
- It supports libcamera ver0.2 SimplePipeline
- It is verified on qv4l2, cheese and qcam
- It is verified together with following patches

	platform/x86: Add AMD ISP platform config (https://lore.kernel.org/all/[email protected]/)
	pinctrl: amd: isp411: Add amdisp GPIO pinctrl (https://github.com/torvalds/linux/commit/e97435ab09f3ad7b6a588dd7c4e45a96699bbb4a)
	drm/amd/amdgpu: Add GPIO resources required for amdisp (https://gitlab.freedesktop.org/agd5f/linux/-/commit/ad0f5966ed8297aa47b3184192b00b7379ae0758)

AMD ISP4 Key features:
- Processes bayer raw data from the connected sensor and output them to different YUV formats
- Downscale input image to different output image resolution
- Pipeline to do image processing on the input image including demosaic, denoise, 3A, etc

----------

Changes v2 -> v3:

- All the dependent patches in other modules (drm/amd/amdgpu, platform/x86, pinctrl/amd) merged on upstream mainline kernel (https://github.com/torvalds/linux) v6.17.
- Removed usage of amdgpu structs in ISP driver. Added helper functions in amdgpu accepting opaque params from ISP driver to allocate and release ISP GART buffers.
- Moved sensor and MIPI phy control entirely into ISP FW instead of the previous hybrid approach controlling sensor from both FW and x86 (sensor driver).
- Removed phy configuration and sensor binding as x86 (sensor driver) had relinquished the sensor control for ISP FW. With this approach the driver will be exposed as web camera like interface.
- New FW with built-in sensor driver is submitted on upstream linux-firmware repo (https://gitlab.com/kernel-firmware/linux-firmware/).
- Please note the new FW submitted is not directly compatible with OEM Kernel ISP4.0 (https://github.com/amd/Linux_ISP_Kernel/tree/4.0) and the previous ISP V2 patch series.
- If intend to use the new FW, please rebuild OEM ISP4.0 Kernel with CONFIG_VIDEO_OV05C10=N and CONFIG_PINCTRL_AMDISP=Y.
- Included critical fixes from Sultan Alsawaf branch (https://github.com/kerneltoast/kernel_x86_laptop.git) related to managing lifetime of isp buffers.
media: amd: isp4: Add missing refcount tracking to mmap memop
media: amd: isp4: Don't put or unmap the dmabuf when detaching
media: amd: isp4: Don't increment refcount when dmabuf export fails
media: amd: isp4: Fix possible use-after-free in isp4vid_vb2_put()
media: amd: isp4: Always export a new dmabuf from get_dmabuf memop
media: amd: isp4: Fix implicit dmabuf lifetime tracking
media: amd: isp4: Fix possible use-after-free when putting implicit dmabuf
media: amd: isp4: Simplify isp4vid_get_dmabuf() arguments
media: amd: isp4: Move up buf->vaddr check in isp4vid_get_dmabuf()
media: amd: isp4: Remove unused userptr memops
media: amd: isp4: Add missing cleanup on error in isp4vid_vb2_alloc()
media: amd: isp4: Release queued buffers on error in start_streaming
- Addressed all code related upstream comments
- Fix typo errors and other cosmetic issue.

Changes v1 -> v2:

- Fix media CI test errors and valid warnings
- Reduce patch number in the series from 9 to 8 by merging MAINTAINERS adding patch to the first patch
- In patch 5
- do modification to use remote endpoint instead of local endpoint
- use link frequency and port number as start phy parameter instead of extra added phy-id and phy-bit-rate property of endpoint

----------

It passes v4l2 compliance test, the test reports for:

(a) amd_isp_capture device /dev/video0

Compliance test for amd_isp_capture device /dev/video0:
-------------------------------------------------------

atg@atg-HP-PV:~/bin$ ./v4l2-compliance -d /dev/video0
v4l2-compliance 1.29.0-5348, 64 bits, 64-bit time_t
v4l2-compliance SHA: 75e3f0e2c2cb 2025-03-17 18:12:17

Compliance test for amd_isp_capture device /dev/video0:

Driver Info:

        Driver name      : amd_isp_capture
        Card type        : amd_isp_capture
        Bus info         : platform:amd_isp_capture
        Driver version   : 6.14.0
        Capabilities     : 0xa4200001

Video Capture
I/O MC
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x24200001
Video Capture
I/O MC
Streaming
Extended Pix Format
Media Driver Info:

        Driver name      : amd_isp_capture
        Model            : amd_isp41_mdev
        Serial           :
        Bus info         : platform:amd_isp_capture
        Media version    : 6.14.0
        Hardware revision: 0x00000000 (0)
        Driver version   : 6.14.0

Interface Info:
ID : 0x03000005
Type : V4L Video
Entity Info:

        ID               : 0x00000003 (3)
        Name             : Preview
        Function         : V4L2 I/O
        Pad 0x01000004   : 0: Sink
          Link 0x02000007: from remote pad 0x1000002 of entity 'amd isp4' (Image Signal Processor): Data, Enabled, Immutable

Required ioctls:
test MC information (see 'Media Driver Info' above): OK test VIDIOC_QUERYCAP: OK
test invalid ioctls: OK

Allow for multiple opens:
test second /dev/video0 open: OK
test VIDIOC_QUERYCAP: OK
test VIDIOC_G/S_PRIORITY: OK
test for unlimited opens: OK

Debug ioctls:
test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK
test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 1 Audio Inputs: 0 Tuners: 0

Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_ENUMAUDOUT: OK (Not Supported) test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls (Input 0):
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported) test VIDIOC_QUERYCTRL: OK (Not Supported)
test VIDIOC_G/S_CTRL: OK (Not Supported)
test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported) test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported) test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) Standard Controls: 0 Private Controls: 0

Format ioctls (Input 0):
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK test VIDIOC_G/S_PARM: OK
test VIDIOC_G_FBUF: OK (Not Supported) test VIDIOC_G_FMT: OK
test VIDIOC_TRY_FMT: OK
test VIDIOC_S_FMT: OK
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported) test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK (Not Supported)

Codec ioctls (Input 0):
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported) test VIDIOC_G_ENC_INDEX: OK (Not Supported) test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls (Input 0):
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK test CREATE_BUFS maximum buffers: OK test VIDIOC_REMOVE_BUFS: OK
test VIDIOC_EXPBUF: OK
test Requests: OK (Not Supported) test blocking wait: OK

Total for amd_isp_capture device /dev/video0: 49, Succeeded: 49, Failed: 0, Warnings: 0

Please review and provide feedback.

Many thanks,

Bin Du (7):
media: platform: amd: Introduce amd isp4 capture driver
media: platform: amd: low level support for isp4 firmware
media: platform: amd: Add isp4 fw and hw interface
media: platform: amd: isp4 subdev and firmware loading handling added
media: platform: amd: isp4 video node and buffers handling added
media: platform: amd: isp4 debug fs logging and more descriptive
errors
Documentation: add documentation of AMD isp 4 driver

 Documentation/admin-guide/media/amdisp4-1.rst |   66 +
 Documentation/admin-guide/media/amdisp4.dot   |    8 +
 .../admin-guide/media/v4l-drivers.rst         |    1 +
 MAINTAINERS                                   |   25 +
 drivers/media/platform/Kconfig                |    1 +
 drivers/media/platform/Makefile               |    1 +
 drivers/media/platform/amd/Kconfig            |    3 +
 drivers/media/platform/amd/Makefile           |    3 +
 drivers/media/platform/amd/isp4/Kconfig       |   13 +
 drivers/media/platform/amd/isp4/Makefile      |   10 +
 drivers/media/platform/amd/isp4/isp4.c        |  237 ++++
 drivers/media/platform/amd/isp4/isp4.h        |   26 +
 drivers/media/platform/amd/isp4/isp4_debug.c  |  272 ++++
 drivers/media/platform/amd/isp4/isp4_debug.h  |   41 +
 .../platform/amd/isp4/isp4_fw_cmd_resp.h      |  314 +++++
 drivers/media/platform/amd/isp4/isp4_hw_reg.h |  125 ++
 .../media/platform/amd/isp4/isp4_interface.c  |  972 +++++++++++++
 .../media/platform/amd/isp4/isp4_interface.h  |  149 ++
 drivers/media/platform/amd/isp4/isp4_subdev.c | 1198 ++++++++++++++++
 drivers/media/platform/amd/isp4/isp4_subdev.h |  133 ++
 drivers/media/platform/amd/isp4/isp4_video.c  | 1213 +++++++++++++++++
 drivers/media/platform/amd/isp4/isp4_video.h  |   87 ++
 22 files changed, 4898 insertions(+)
 create mode 100644 Documentation/admin-guide/media/amdisp4-1.rst
 create mode 100644 Documentation/admin-guide/media/amdisp4.dot
 create mode 100644 drivers/media/platform/amd/Kconfig
 create mode 100644 drivers/media/platform/amd/Makefile
 create mode 100644 drivers/media/platform/amd/isp4/Kconfig
 create mode 100644 drivers/media/platform/amd/isp4/Makefile
 create mode 100644 drivers/media/platform/amd/isp4/isp4.c
 create mode 100644 drivers/media/platform/amd/isp4/isp4.h
 create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.c
 create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.h
 create mode 100644 drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
 create mode 100644 drivers/media/platform/amd/isp4/isp4_hw_reg.h
 create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.c
 create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.h
 create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c
 create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h
 create mode 100644 drivers/media/platform/amd/isp4/isp4_video.c
 create mode 100644 drivers/media/platform/amd/isp4/isp4_video.h
Signature · 3 lines
Regards,

Laurent Pinchart
#2 lore raw
Quoted 4 lines · Laurent Pinchart

Hi Bin Bu,

Have you sent out the cover letter only ? I haven't received the rest of the series, and it's not found on lore.kernel.org either.

Something seems wrong with the cover letter threading to the series. Here's the rest of the series (which I did receive too).

https://lore.kernel.org/linux-media/[email protected]/

Quoted 242 lines · Bin Du
Quoted 239 lines

Hello,

AMD ISP4 is the AMD image processing gen 4 which can be found in HP ZBook Ultra G1a 14 inch Mobile Workstation PC ( Ryzen AI Max 385)
(https://ubuntu.com/certified/202411-36043)
This patch series introduces the initial driver support for the AMD ISP4.

Patch summary:
- Powers up/off and initializes ISP HW
- Configures and kicks off ISP FW
- Interacts with APP using standard V4l2 interface by video node
- Controls ISP HW and interacts with ISP FW to do image processing
- Support enum/set output image format and resolution
- Support queueing buffer from app and dequeueing ISP filled buffer to App
- It supports libcamera ver0.2 SimplePipeline
- It is verified on qv4l2, cheese and qcam
- It is verified together with following patches

	platform/x86: Add AMD ISP platform config (https://lore.kernel.org/all/[email protected]/)
	pinctrl: amd: isp411: Add amdisp GPIO pinctrl (https://github.com/torvalds/linux/commit/e97435ab09f3ad7b6a588dd7c4e45a96699bbb4a)
	drm/amd/amdgpu: Add GPIO resources required for amdisp (https://gitlab.freedesktop.org/agd5f/linux/-/commit/ad0f5966ed8297aa47b3184192b00b7379ae0758)

AMD ISP4 Key features:
- Processes bayer raw data from the connected sensor and output them to different YUV formats
- Downscale input image to different output image resolution
- Pipeline to do image processing on the input image including demosaic, denoise, 3A, etc

----------

Changes v2 -> v3:

- All the dependent patches in other modules (drm/amd/amdgpu, platform/x86, pinctrl/amd) merged on upstream mainline kernel (https://github.com/torvalds/linux) v6.17.
- Removed usage of amdgpu structs in ISP driver. Added helper functions in amdgpu accepting opaque params from ISP driver to allocate and release ISP GART buffers.
- Moved sensor and MIPI phy control entirely into ISP FW instead of the previous hybrid approach controlling sensor from both FW and x86 (sensor driver).
- Removed phy configuration and sensor binding as x86 (sensor driver) had relinquished the sensor control for ISP FW. With this approach the driver will be exposed as web camera like interface.
- New FW with built-in sensor driver is submitted on upstream linux-firmware repo (https://gitlab.com/kernel-firmware/linux-firmware/).
- Please note the new FW submitted is not directly compatible with OEM Kernel ISP4.0 (https://github.com/amd/Linux_ISP_Kernel/tree/4.0) and the previous ISP V2 patch series.
- If intend to use the new FW, please rebuild OEM ISP4.0 Kernel with CONFIG_VIDEO_OV05C10=N and CONFIG_PINCTRL_AMDISP=Y.
- Included critical fixes from Sultan Alsawaf branch (https://github.com/kerneltoast/kernel_x86_laptop.git) related to managing lifetime of isp buffers.
media: amd: isp4: Add missing refcount tracking to mmap memop
media: amd: isp4: Don't put or unmap the dmabuf when detaching
media: amd: isp4: Don't increment refcount when dmabuf export fails
media: amd: isp4: Fix possible use-after-free in isp4vid_vb2_put()
media: amd: isp4: Always export a new dmabuf from get_dmabuf memop
media: amd: isp4: Fix implicit dmabuf lifetime tracking
media: amd: isp4: Fix possible use-after-free when putting implicit dmabuf
media: amd: isp4: Simplify isp4vid_get_dmabuf() arguments
media: amd: isp4: Move up buf->vaddr check in isp4vid_get_dmabuf()
media: amd: isp4: Remove unused userptr memops
media: amd: isp4: Add missing cleanup on error in isp4vid_vb2_alloc()
media: amd: isp4: Release queued buffers on error in start_streaming
- Addressed all code related upstream comments
- Fix typo errors and other cosmetic issue.

Changes v1 -> v2:

- Fix media CI test errors and valid warnings
- Reduce patch number in the series from 9 to 8 by merging MAINTAINERS adding patch to the first patch
- In patch 5
- do modification to use remote endpoint instead of local endpoint
- use link frequency and port number as start phy parameter instead of extra added phy-id and phy-bit-rate property of endpoint

----------

It passes v4l2 compliance test, the test reports for:

(a) amd_isp_capture device /dev/video0

Compliance test for amd_isp_capture device /dev/video0:
-------------------------------------------------------

atg@atg-HP-PV:~/bin$ ./v4l2-compliance -d /dev/video0
v4l2-compliance 1.29.0-5348, 64 bits, 64-bit time_t
v4l2-compliance SHA: 75e3f0e2c2cb 2025-03-17 18:12:17

Compliance test for amd_isp_capture device /dev/video0:

Driver Info:

         Driver name      : amd_isp_capture
         Card type        : amd_isp_capture
         Bus info         : platform:amd_isp_capture
         Driver version   : 6.14.0
         Capabilities     : 0xa4200001

Video Capture
I/O MC
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x24200001
Video Capture
I/O MC
Streaming
Extended Pix Format
Media Driver Info:

         Driver name      : amd_isp_capture
         Model            : amd_isp41_mdev
         Serial           :
         Bus info         : platform:amd_isp_capture
         Media version    : 6.14.0
         Hardware revision: 0x00000000 (0)
         Driver version   : 6.14.0

Interface Info:
ID : 0x03000005
Type : V4L Video
Entity Info:

         ID               : 0x00000003 (3)
         Name             : Preview
         Function         : V4L2 I/O
         Pad 0x01000004   : 0: Sink
           Link 0x02000007: from remote pad 0x1000002 of entity 'amd isp4' (Image Signal Processor): Data, Enabled, Immutable

Required ioctls:
test MC information (see 'Media Driver Info' above): OK test VIDIOC_QUERYCAP: OK
test invalid ioctls: OK

Allow for multiple opens:
test second /dev/video0 open: OK
test VIDIOC_QUERYCAP: OK
test VIDIOC_G/S_PRIORITY: OK
test for unlimited opens: OK

Debug ioctls:
test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK
test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 1 Audio Inputs: 0 Tuners: 0

Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_ENUMAUDOUT: OK (Not Supported) test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls (Input 0):
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported) test VIDIOC_QUERYCTRL: OK (Not Supported)
test VIDIOC_G/S_CTRL: OK (Not Supported)
test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported) test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported) test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) Standard Controls: 0 Private Controls: 0

Format ioctls (Input 0):
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK test VIDIOC_G/S_PARM: OK
test VIDIOC_G_FBUF: OK (Not Supported) test VIDIOC_G_FMT: OK
test VIDIOC_TRY_FMT: OK
test VIDIOC_S_FMT: OK
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported) test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK (Not Supported)

Codec ioctls (Input 0):
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported) test VIDIOC_G_ENC_INDEX: OK (Not Supported) test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls (Input 0):
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK test CREATE_BUFS maximum buffers: OK test VIDIOC_REMOVE_BUFS: OK
test VIDIOC_EXPBUF: OK
test Requests: OK (Not Supported) test blocking wait: OK

Total for amd_isp_capture device /dev/video0: 49, Succeeded: 49, Failed: 0, Warnings: 0

Please review and provide feedback.

Many thanks,

Bin Du (7):
media: platform: amd: Introduce amd isp4 capture driver
media: platform: amd: low level support for isp4 firmware
media: platform: amd: Add isp4 fw and hw interface
media: platform: amd: isp4 subdev and firmware loading handling added
media: platform: amd: isp4 video node and buffers handling added
media: platform: amd: isp4 debug fs logging and more descriptive
errors
Documentation: add documentation of AMD isp 4 driver

  Documentation/admin-guide/media/amdisp4-1.rst |   66 +
  Documentation/admin-guide/media/amdisp4.dot   |    8 +
  .../admin-guide/media/v4l-drivers.rst         |    1 +
  MAINTAINERS                                   |   25 +
  drivers/media/platform/Kconfig                |    1 +
  drivers/media/platform/Makefile               |    1 +
  drivers/media/platform/amd/Kconfig            |    3 +
  drivers/media/platform/amd/Makefile           |    3 +
  drivers/media/platform/amd/isp4/Kconfig       |   13 +
  drivers/media/platform/amd/isp4/Makefile      |   10 +
  drivers/media/platform/amd/isp4/isp4.c        |  237 ++++
  drivers/media/platform/amd/isp4/isp4.h        |   26 +
  drivers/media/platform/amd/isp4/isp4_debug.c  |  272 ++++
  drivers/media/platform/amd/isp4/isp4_debug.h  |   41 +
  .../platform/amd/isp4/isp4_fw_cmd_resp.h      |  314 +++++
  drivers/media/platform/amd/isp4/isp4_hw_reg.h |  125 ++
  .../media/platform/amd/isp4/isp4_interface.c  |  972 +++++++++++++
  .../media/platform/amd/isp4/isp4_interface.h  |  149 ++
  drivers/media/platform/amd/isp4/isp4_subdev.c | 1198 ++++++++++++++++
  drivers/media/platform/amd/isp4/isp4_subdev.h |  133 ++
  drivers/media/platform/amd/isp4/isp4_video.c  | 1213 +++++++++++++++++
  drivers/media/platform/amd/isp4/isp4_video.h  |   87 ++
  22 files changed, 4898 insertions(+)

create mode 100644 Documentation/admin-guide/media/amdisp4-1.rst create mode 100644 Documentation/admin-guide/media/amdisp4.dot create mode 100644 drivers/media/platform/amd/Kconfig create mode 100644 drivers/media/platform/amd/Makefile create mode 100644 drivers/media/platform/amd/isp4/Kconfig create mode 100644 drivers/media/platform/amd/isp4/Makefile create mode 100644 drivers/media/platform/amd/isp4/isp4.c create mode 100644 drivers/media/platform/amd/isp4/isp4.h create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.c create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.h create mode 100644 drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h create mode 100644 drivers/media/platform/amd/isp4/isp4_hw_reg.h create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.c create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.h create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h create mode 100644 drivers/media/platform/amd/isp4/isp4_video.c create mode 100644 drivers/media/platform/amd/isp4/isp4_video.h

author
#3 lore raw

Thanks Laurent, and Mario for the clarification.

Quoted 254 lines · Mario
Quoted 4 lines · Laurent Pinchart

Hi Bin Bu,

Have you sent out the cover letter only ? I haven't received the rest of the series, and it's not found on lore.kernel.org either.

Something seems wrong with the cover letter threading to the series. Here's the rest of the series (which I did receive too).

https://lore.kernel.org/linux-media/[email protected]/

Quoted 242 lines · Bin Du
Quoted 239 lines

Hello,

AMD ISP4 is the AMD image processing gen 4 which can be found in HP ZBook Ultra G1a 14 inch Mobile Workstation PC ( Ryzen AI Max 385)
(https://ubuntu.com/certified/202411-36043)
This patch series introduces the initial driver support for the AMD ISP4.

Patch summary:
- Powers up/off and initializes ISP HW
- Configures and kicks off ISP FW
- Interacts with APP using standard V4l2 interface by video node
- Controls ISP HW and interacts with ISP FW to do image processing
- Support enum/set output image format and resolution
- Support queueing buffer from app and dequeueing ISP filled buffer to App
- It supports libcamera ver0.2 SimplePipeline
- It is verified on qv4l2, cheese and qcam
- It is verified together with following patches

	platform/x86: Add AMD ISP platform config (https://lore.kernel.org/all/[email protected]/)
	pinctrl: amd: isp411: Add amdisp GPIO pinctrl (https://github.com/torvalds/linux/commit/e97435ab09f3ad7b6a588dd7c4e45a96699bbb4a)
	drm/amd/amdgpu: Add GPIO resources required for amdisp (https://gitlab.freedesktop.org/agd5f/linux/-/commit/ad0f5966ed8297aa47b3184192b00b7379ae0758)

AMD ISP4 Key features:
- Processes bayer raw data from the connected sensor and output them to different YUV formats
- Downscale input image to different output image resolution
- Pipeline to do image processing on the input image including demosaic, denoise, 3A, etc

----------

Changes v2 -> v3:

- All the dependent patches in other modules (drm/amd/amdgpu, platform/x86, pinctrl/amd) merged on upstream mainline kernel (https://github.com/torvalds/linux) v6.17.
- Removed usage of amdgpu structs in ISP driver. Added helper functions in amdgpu accepting opaque params from ISP driver to allocate and release ISP GART buffers.
- Moved sensor and MIPI phy control entirely into ISP FW instead of the previous hybrid approach controlling sensor from both FW and x86 (sensor driver).
- Removed phy configuration and sensor binding as x86 (sensor driver) had relinquished the sensor control for ISP FW. With this approach the driver will be exposed as web camera like interface.
- New FW with built-in sensor driver is submitted on upstream linux-firmware repo (https://gitlab.com/kernel-firmware/linux-firmware/).
- Please note the new FW submitted is not directly compatible with OEM Kernel ISP4.0 (https://github.com/amd/Linux_ISP_Kernel/tree/4.0) and the previous ISP V2 patch series.
- If intend to use the new FW, please rebuild OEM ISP4.0 Kernel with CONFIG_VIDEO_OV05C10=N and CONFIG_PINCTRL_AMDISP=Y.
- Included critical fixes from Sultan Alsawaf branch (https://github.com/kerneltoast/kernel_x86_laptop.git) related to managing lifetime of isp buffers.
media: amd: isp4: Add missing refcount tracking to mmap memop
media: amd: isp4: Don't put or unmap the dmabuf when detaching
media: amd: isp4: Don't increment refcount when dmabuf export fails
media: amd: isp4: Fix possible use-after-free in isp4vid_vb2_put()
media: amd: isp4: Always export a new dmabuf from get_dmabuf memop
media: amd: isp4: Fix implicit dmabuf lifetime tracking
media: amd: isp4: Fix possible use-after-free when putting implicit dmabuf
media: amd: isp4: Simplify isp4vid_get_dmabuf() arguments
media: amd: isp4: Move up buf->vaddr check in isp4vid_get_dmabuf()
media: amd: isp4: Remove unused userptr memops
media: amd: isp4: Add missing cleanup on error in isp4vid_vb2_alloc()
media: amd: isp4: Release queued buffers on error in start_streaming
- Addressed all code related upstream comments
- Fix typo errors and other cosmetic issue.

Changes v1 -> v2:

- Fix media CI test errors and valid warnings
- Reduce patch number in the series from 9 to 8 by merging MAINTAINERS adding patch to the first patch
- In patch 5
- do modification to use remote endpoint instead of local endpoint
- use link frequency and port number as start phy parameter instead of extra added phy-id and phy-bit-rate property of endpoint

----------

It passes v4l2 compliance test, the test reports for:

(a) amd_isp_capture device /dev/video0

Compliance test for amd_isp_capture device /dev/video0:
-------------------------------------------------------

atg@atg-HP-PV:~/bin$ ./v4l2-compliance -d /dev/video0
v4l2-compliance 1.29.0-5348, 64 bits, 64-bit time_t
v4l2-compliance SHA: 75e3f0e2c2cb 2025-03-17 18:12:17

Compliance test for amd_isp_capture device /dev/video0:

Driver Info:

          Driver name      : amd_isp_capture
          Card type        : amd_isp_capture
          Bus info         : platform:amd_isp_capture
          Driver version   : 6.14.0
          Capabilities     : 0xa4200001

Video Capture
I/O MC
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x24200001
Video Capture
I/O MC
Streaming
Extended Pix Format
Media Driver Info:

          Driver name      : amd_isp_capture
          Model            : amd_isp41_mdev
          Serial           :
          Bus info         : platform:amd_isp_capture
          Media version    : 6.14.0
          Hardware revision: 0x00000000 (0)
          Driver version   : 6.14.0

Interface Info:
ID : 0x03000005
Type : V4L Video
Entity Info:

          ID               : 0x00000003 (3)
          Name             : Preview
          Function         : V4L2 I/O
          Pad 0x01000004   : 0: Sink
            Link 0x02000007: from remote pad 0x1000002 of entity 'amd isp4' (Image Signal Processor): Data, Enabled, Immutable

Required ioctls:
test MC information (see 'Media Driver Info' above): OK test VIDIOC_QUERYCAP: OK
test invalid ioctls: OK

Allow for multiple opens:
test second /dev/video0 open: OK
test VIDIOC_QUERYCAP: OK
test VIDIOC_G/S_PRIORITY: OK
test for unlimited opens: OK

Debug ioctls:
test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK
test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 1 Audio Inputs: 0 Tuners: 0

Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_ENUMAUDOUT: OK (Not Supported) test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls (Input 0):
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported) test VIDIOC_QUERYCTRL: OK (Not Supported)
test VIDIOC_G/S_CTRL: OK (Not Supported)
test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported) test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported) test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) Standard Controls: 0 Private Controls: 0

Format ioctls (Input 0):
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK test VIDIOC_G/S_PARM: OK
test VIDIOC_G_FBUF: OK (Not Supported) test VIDIOC_G_FMT: OK
test VIDIOC_TRY_FMT: OK
test VIDIOC_S_FMT: OK
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported) test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK (Not Supported)

Codec ioctls (Input 0):
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported) test VIDIOC_G_ENC_INDEX: OK (Not Supported) test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls (Input 0):
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK test CREATE_BUFS maximum buffers: OK test VIDIOC_REMOVE_BUFS: OK
test VIDIOC_EXPBUF: OK
test Requests: OK (Not Supported) test blocking wait: OK

Total for amd_isp_capture device /dev/video0: 49, Succeeded: 49, Failed: 0, Warnings: 0

Please review and provide feedback.

Many thanks,

Bin Du (7):
media: platform: amd: Introduce amd isp4 capture driver
media: platform: amd: low level support for isp4 firmware
media: platform: amd: Add isp4 fw and hw interface
media: platform: amd: isp4 subdev and firmware loading handling added
media: platform: amd: isp4 video node and buffers handling added
media: platform: amd: isp4 debug fs logging and more descriptive
errors
Documentation: add documentation of AMD isp 4 driver

   Documentation/admin-guide/media/amdisp4-1.rst |   66 +
   Documentation/admin-guide/media/amdisp4.dot   |    8 +
   .../admin-guide/media/v4l-drivers.rst         |    1 +
   MAINTAINERS                                   |   25 +
   drivers/media/platform/Kconfig                |    1 +
   drivers/media/platform/Makefile               |    1 +
   drivers/media/platform/amd/Kconfig            |    3 +
   drivers/media/platform/amd/Makefile           |    3 +
   drivers/media/platform/amd/isp4/Kconfig       |   13 +
   drivers/media/platform/amd/isp4/Makefile      |   10 +
   drivers/media/platform/amd/isp4/isp4.c        |  237 ++++
   drivers/media/platform/amd/isp4/isp4.h        |   26 +
   drivers/media/platform/amd/isp4/isp4_debug.c  |  272 ++++
   drivers/media/platform/amd/isp4/isp4_debug.h  |   41 +
   .../platform/amd/isp4/isp4_fw_cmd_resp.h      |  314 +++++
   drivers/media/platform/amd/isp4/isp4_hw_reg.h |  125 ++
   .../media/platform/amd/isp4/isp4_interface.c  |  972 +++++++++++++
   .../media/platform/amd/isp4/isp4_interface.h  |  149 ++
   drivers/media/platform/amd/isp4/isp4_subdev.c | 1198 ++++++++++++++++
   drivers/media/platform/amd/isp4/isp4_subdev.h |  133 ++
   drivers/media/platform/amd/isp4/isp4_video.c  | 1213 +++++++++++++++++
   drivers/media/platform/amd/isp4/isp4_video.h  |   87 ++
   22 files changed, 4898 insertions(+)

create mode 100644 Documentation/admin-guide/media/amdisp4-1.rst create mode 100644 Documentation/admin-guide/media/amdisp4.dot create mode 100644 drivers/media/platform/amd/Kconfig create mode 100644 drivers/media/platform/amd/Makefile create mode 100644 drivers/media/platform/amd/isp4/Kconfig create mode 100644 drivers/media/platform/amd/isp4/Makefile create mode 100644 drivers/media/platform/amd/isp4/isp4.c create mode 100644 drivers/media/platform/amd/isp4/isp4.h create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.c create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.h create mode 100644 drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h create mode 100644 drivers/media/platform/amd/isp4/isp4_hw_reg.h create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.c create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.h create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h create mode 100644 drivers/media/platform/amd/isp4/isp4_video.c create mode 100644 drivers/media/platform/amd/isp4/isp4_video.h

Signature · 2 lines
Regards,
Bin
#4 lore raw
In reply to Laurent Pinchart
Quoted 4 lines · Laurent Pinchart

Hi Bin Bu,

Have you sent out the cover letter only ? I haven't received the rest of the series, and it's not found on lore.kernel.org either.

I've just noticed you sent the rest later and separately, as
https://lore.kernel.org/all/[email protected]/.

Quoted 240 lines · Bin Du
Quoted 239 lines

Hello,

AMD ISP4 is the AMD image processing gen 4 which can be found in HP ZBook Ultra G1a 14 inch Mobile Workstation PC ( Ryzen AI Max 385)
(https://ubuntu.com/certified/202411-36043)
This patch series introduces the initial driver support for the AMD ISP4.

Patch summary:
- Powers up/off and initializes ISP HW
- Configures and kicks off ISP FW
- Interacts with APP using standard V4l2 interface by video node
- Controls ISP HW and interacts with ISP FW to do image processing
- Support enum/set output image format and resolution
- Support queueing buffer from app and dequeueing ISP filled buffer to App
- It supports libcamera ver0.2 SimplePipeline
- It is verified on qv4l2, cheese and qcam
- It is verified together with following patches

	platform/x86: Add AMD ISP platform config (https://lore.kernel.org/all/[email protected]/)
	pinctrl: amd: isp411: Add amdisp GPIO pinctrl (https://github.com/torvalds/linux/commit/e97435ab09f3ad7b6a588dd7c4e45a96699bbb4a)
	drm/amd/amdgpu: Add GPIO resources required for amdisp (https://gitlab.freedesktop.org/agd5f/linux/-/commit/ad0f5966ed8297aa47b3184192b00b7379ae0758)

AMD ISP4 Key features:
- Processes bayer raw data from the connected sensor and output them to different YUV formats
- Downscale input image to different output image resolution
- Pipeline to do image processing on the input image including demosaic, denoise, 3A, etc

----------

Changes v2 -> v3:

- All the dependent patches in other modules (drm/amd/amdgpu, platform/x86, pinctrl/amd) merged on upstream mainline kernel (https://github.com/torvalds/linux) v6.17.
- Removed usage of amdgpu structs in ISP driver. Added helper functions in amdgpu accepting opaque params from ISP driver to allocate and release ISP GART buffers.
- Moved sensor and MIPI phy control entirely into ISP FW instead of the previous hybrid approach controlling sensor from both FW and x86 (sensor driver).
- Removed phy configuration and sensor binding as x86 (sensor driver) had relinquished the sensor control for ISP FW. With this approach the driver will be exposed as web camera like interface.
- New FW with built-in sensor driver is submitted on upstream linux-firmware repo (https://gitlab.com/kernel-firmware/linux-firmware/).
- Please note the new FW submitted is not directly compatible with OEM Kernel ISP4.0 (https://github.com/amd/Linux_ISP_Kernel/tree/4.0) and the previous ISP V2 patch series.
- If intend to use the new FW, please rebuild OEM ISP4.0 Kernel with CONFIG_VIDEO_OV05C10=N and CONFIG_PINCTRL_AMDISP=Y.
- Included critical fixes from Sultan Alsawaf branch (https://github.com/kerneltoast/kernel_x86_laptop.git) related to managing lifetime of isp buffers.
media: amd: isp4: Add missing refcount tracking to mmap memop
media: amd: isp4: Don't put or unmap the dmabuf when detaching
media: amd: isp4: Don't increment refcount when dmabuf export fails
media: amd: isp4: Fix possible use-after-free in isp4vid_vb2_put()
media: amd: isp4: Always export a new dmabuf from get_dmabuf memop
media: amd: isp4: Fix implicit dmabuf lifetime tracking
media: amd: isp4: Fix possible use-after-free when putting implicit dmabuf
media: amd: isp4: Simplify isp4vid_get_dmabuf() arguments
media: amd: isp4: Move up buf->vaddr check in isp4vid_get_dmabuf()
media: amd: isp4: Remove unused userptr memops
media: amd: isp4: Add missing cleanup on error in isp4vid_vb2_alloc()
media: amd: isp4: Release queued buffers on error in start_streaming
- Addressed all code related upstream comments
- Fix typo errors and other cosmetic issue.

Changes v1 -> v2:

- Fix media CI test errors and valid warnings
- Reduce patch number in the series from 9 to 8 by merging MAINTAINERS adding patch to the first patch
- In patch 5
- do modification to use remote endpoint instead of local endpoint
- use link frequency and port number as start phy parameter instead of extra added phy-id and phy-bit-rate property of endpoint

----------

It passes v4l2 compliance test, the test reports for:

(a) amd_isp_capture device /dev/video0

Compliance test for amd_isp_capture device /dev/video0:
-------------------------------------------------------

atg@atg-HP-PV:~/bin$ ./v4l2-compliance -d /dev/video0
v4l2-compliance 1.29.0-5348, 64 bits, 64-bit time_t
v4l2-compliance SHA: 75e3f0e2c2cb 2025-03-17 18:12:17

Compliance test for amd_isp_capture device /dev/video0:

Driver Info:

        Driver name      : amd_isp_capture
        Card type        : amd_isp_capture
        Bus info         : platform:amd_isp_capture
        Driver version   : 6.14.0
        Capabilities     : 0xa4200001

Video Capture
I/O MC
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x24200001
Video Capture
I/O MC
Streaming
Extended Pix Format
Media Driver Info:

        Driver name      : amd_isp_capture
        Model            : amd_isp41_mdev
        Serial           :
        Bus info         : platform:amd_isp_capture
        Media version    : 6.14.0
        Hardware revision: 0x00000000 (0)
        Driver version   : 6.14.0

Interface Info:
ID : 0x03000005
Type : V4L Video
Entity Info:

        ID               : 0x00000003 (3)
        Name             : Preview
        Function         : V4L2 I/O
        Pad 0x01000004   : 0: Sink
          Link 0x02000007: from remote pad 0x1000002 of entity 'amd isp4' (Image Signal Processor): Data, Enabled, Immutable

Required ioctls:
test MC information (see 'Media Driver Info' above): OK test VIDIOC_QUERYCAP: OK
test invalid ioctls: OK

Allow for multiple opens:
test second /dev/video0 open: OK
test VIDIOC_QUERYCAP: OK
test VIDIOC_G/S_PRIORITY: OK
test for unlimited opens: OK

Debug ioctls:
test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK
test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 1 Audio Inputs: 0 Tuners: 0

Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_ENUMAUDOUT: OK (Not Supported) test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls (Input 0):
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported) test VIDIOC_QUERYCTRL: OK (Not Supported)
test VIDIOC_G/S_CTRL: OK (Not Supported)
test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported) test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported) test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) Standard Controls: 0 Private Controls: 0

Format ioctls (Input 0):
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK test VIDIOC_G/S_PARM: OK
test VIDIOC_G_FBUF: OK (Not Supported) test VIDIOC_G_FMT: OK
test VIDIOC_TRY_FMT: OK
test VIDIOC_S_FMT: OK
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported) test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK (Not Supported)

Codec ioctls (Input 0):
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported) test VIDIOC_G_ENC_INDEX: OK (Not Supported) test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls (Input 0):
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK test CREATE_BUFS maximum buffers: OK test VIDIOC_REMOVE_BUFS: OK
test VIDIOC_EXPBUF: OK
test Requests: OK (Not Supported) test blocking wait: OK

Total for amd_isp_capture device /dev/video0: 49, Succeeded: 49, Failed: 0, Warnings: 0

Please review and provide feedback.

Many thanks,

Bin Du (7):
media: platform: amd: Introduce amd isp4 capture driver
media: platform: amd: low level support for isp4 firmware
media: platform: amd: Add isp4 fw and hw interface
media: platform: amd: isp4 subdev and firmware loading handling added
media: platform: amd: isp4 video node and buffers handling added
media: platform: amd: isp4 debug fs logging and more descriptive
errors
Documentation: add documentation of AMD isp 4 driver

 Documentation/admin-guide/media/amdisp4-1.rst |   66 +
 Documentation/admin-guide/media/amdisp4.dot   |    8 +
 .../admin-guide/media/v4l-drivers.rst         |    1 +
 MAINTAINERS                                   |   25 +
 drivers/media/platform/Kconfig                |    1 +
 drivers/media/platform/Makefile               |    1 +
 drivers/media/platform/amd/Kconfig            |    3 +
 drivers/media/platform/amd/Makefile           |    3 +
 drivers/media/platform/amd/isp4/Kconfig       |   13 +
 drivers/media/platform/amd/isp4/Makefile      |   10 +
 drivers/media/platform/amd/isp4/isp4.c        |  237 ++++
 drivers/media/platform/amd/isp4/isp4.h        |   26 +
 drivers/media/platform/amd/isp4/isp4_debug.c  |  272 ++++
 drivers/media/platform/amd/isp4/isp4_debug.h  |   41 +
 .../platform/amd/isp4/isp4_fw_cmd_resp.h      |  314 +++++
 drivers/media/platform/amd/isp4/isp4_hw_reg.h |  125 ++
 .../media/platform/amd/isp4/isp4_interface.c  |  972 +++++++++++++
 .../media/platform/amd/isp4/isp4_interface.h  |  149 ++
 drivers/media/platform/amd/isp4/isp4_subdev.c | 1198 ++++++++++++++++
 drivers/media/platform/amd/isp4/isp4_subdev.h |  133 ++
 drivers/media/platform/amd/isp4/isp4_video.c  | 1213 +++++++++++++++++
 drivers/media/platform/amd/isp4/isp4_video.h  |   87 ++
 22 files changed, 4898 insertions(+)
 create mode 100644 Documentation/admin-guide/media/amdisp4-1.rst
 create mode 100644 Documentation/admin-guide/media/amdisp4.dot
 create mode 100644 drivers/media/platform/amd/Kconfig
 create mode 100644 drivers/media/platform/amd/Makefile
 create mode 100644 drivers/media/platform/amd/isp4/Kconfig
 create mode 100644 drivers/media/platform/amd/isp4/Makefile
 create mode 100644 drivers/media/platform/amd/isp4/isp4.c
 create mode 100644 drivers/media/platform/amd/isp4/isp4.h
 create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.c
 create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.h
 create mode 100644 drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h
 create mode 100644 drivers/media/platform/amd/isp4/isp4_hw_reg.h
 create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.c
 create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.h
 create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c
 create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h
 create mode 100644 drivers/media/platform/amd/isp4/isp4_video.c
 create mode 100644 drivers/media/platform/amd/isp4/isp4_video.h
Signature · 3 lines
Regards,

Laurent Pinchart
author
#5 lore raw

Hi Laurent Pinchart, sorry for the confusion.

Quoted 9 lines · Laurent Pinchart
Quoted 4 lines · Laurent Pinchart

Hi Bin Bu,

Have you sent out the cover letter only ? I haven't received the rest of the series, and it's not found on lore.kernel.org either.

I've just noticed you sent the rest later and separately, as
https://lore.kernel.org/all/[email protected]/.

Before sending the formal ones, i did some internal test, after the cover letter, when i tried to send the rest, i encountered this error, 4.4.2 Message submission rate for this client has exceeded the configured limit. So i had to wait for some time so i could send again.

Quoted 241 lines
Quoted 240 lines · Bin Du
Quoted 239 lines

Hello,

AMD ISP4 is the AMD image processing gen 4 which can be found in HP ZBook Ultra G1a 14 inch Mobile Workstation PC ( Ryzen AI Max 385)
(https://ubuntu.com/certified/202411-36043)
This patch series introduces the initial driver support for the AMD ISP4.

Patch summary:
- Powers up/off and initializes ISP HW
- Configures and kicks off ISP FW
- Interacts with APP using standard V4l2 interface by video node
- Controls ISP HW and interacts with ISP FW to do image processing
- Support enum/set output image format and resolution
- Support queueing buffer from app and dequeueing ISP filled buffer to App
- It supports libcamera ver0.2 SimplePipeline
- It is verified on qv4l2, cheese and qcam
- It is verified together with following patches

	platform/x86: Add AMD ISP platform config (https://lore.kernel.org/all/[email protected]/)
	pinctrl: amd: isp411: Add amdisp GPIO pinctrl (https://github.com/torvalds/linux/commit/e97435ab09f3ad7b6a588dd7c4e45a96699bbb4a)
	drm/amd/amdgpu: Add GPIO resources required for amdisp (https://gitlab.freedesktop.org/agd5f/linux/-/commit/ad0f5966ed8297aa47b3184192b00b7379ae0758)

AMD ISP4 Key features:
- Processes bayer raw data from the connected sensor and output them to different YUV formats
- Downscale input image to different output image resolution
- Pipeline to do image processing on the input image including demosaic, denoise, 3A, etc

----------

Changes v2 -> v3:

- All the dependent patches in other modules (drm/amd/amdgpu, platform/x86, pinctrl/amd) merged on upstream mainline kernel (https://github.com/torvalds/linux) v6.17.
- Removed usage of amdgpu structs in ISP driver. Added helper functions in amdgpu accepting opaque params from ISP driver to allocate and release ISP GART buffers.
- Moved sensor and MIPI phy control entirely into ISP FW instead of the previous hybrid approach controlling sensor from both FW and x86 (sensor driver).
- Removed phy configuration and sensor binding as x86 (sensor driver) had relinquished the sensor control for ISP FW. With this approach the driver will be exposed as web camera like interface.
- New FW with built-in sensor driver is submitted on upstream linux-firmware repo (https://gitlab.com/kernel-firmware/linux-firmware/).
- Please note the new FW submitted is not directly compatible with OEM Kernel ISP4.0 (https://github.com/amd/Linux_ISP_Kernel/tree/4.0) and the previous ISP V2 patch series.
- If intend to use the new FW, please rebuild OEM ISP4.0 Kernel with CONFIG_VIDEO_OV05C10=N and CONFIG_PINCTRL_AMDISP=Y.
- Included critical fixes from Sultan Alsawaf branch (https://github.com/kerneltoast/kernel_x86_laptop.git) related to managing lifetime of isp buffers.
media: amd: isp4: Add missing refcount tracking to mmap memop
media: amd: isp4: Don't put or unmap the dmabuf when detaching
media: amd: isp4: Don't increment refcount when dmabuf export fails
media: amd: isp4: Fix possible use-after-free in isp4vid_vb2_put()
media: amd: isp4: Always export a new dmabuf from get_dmabuf memop
media: amd: isp4: Fix implicit dmabuf lifetime tracking
media: amd: isp4: Fix possible use-after-free when putting implicit dmabuf
media: amd: isp4: Simplify isp4vid_get_dmabuf() arguments
media: amd: isp4: Move up buf->vaddr check in isp4vid_get_dmabuf()
media: amd: isp4: Remove unused userptr memops
media: amd: isp4: Add missing cleanup on error in isp4vid_vb2_alloc()
media: amd: isp4: Release queued buffers on error in start_streaming
- Addressed all code related upstream comments
- Fix typo errors and other cosmetic issue.

Changes v1 -> v2:

- Fix media CI test errors and valid warnings
- Reduce patch number in the series from 9 to 8 by merging MAINTAINERS adding patch to the first patch
- In patch 5
- do modification to use remote endpoint instead of local endpoint
- use link frequency and port number as start phy parameter instead of extra added phy-id and phy-bit-rate property of endpoint

----------

It passes v4l2 compliance test, the test reports for:

(a) amd_isp_capture device /dev/video0

Compliance test for amd_isp_capture device /dev/video0:
-------------------------------------------------------

atg@atg-HP-PV:~/bin$ ./v4l2-compliance -d /dev/video0
v4l2-compliance 1.29.0-5348, 64 bits, 64-bit time_t
v4l2-compliance SHA: 75e3f0e2c2cb 2025-03-17 18:12:17

Compliance test for amd_isp_capture device /dev/video0:

Driver Info:

         Driver name      : amd_isp_capture
         Card type        : amd_isp_capture
         Bus info         : platform:amd_isp_capture
         Driver version   : 6.14.0
         Capabilities     : 0xa4200001

Video Capture
I/O MC
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x24200001
Video Capture
I/O MC
Streaming
Extended Pix Format
Media Driver Info:

         Driver name      : amd_isp_capture
         Model            : amd_isp41_mdev
         Serial           :
         Bus info         : platform:amd_isp_capture
         Media version    : 6.14.0
         Hardware revision: 0x00000000 (0)
         Driver version   : 6.14.0

Interface Info:
ID : 0x03000005
Type : V4L Video
Entity Info:

         ID               : 0x00000003 (3)
         Name             : Preview
         Function         : V4L2 I/O
         Pad 0x01000004   : 0: Sink
           Link 0x02000007: from remote pad 0x1000002 of entity 'amd isp4' (Image Signal Processor): Data, Enabled, Immutable

Required ioctls:
test MC information (see 'Media Driver Info' above): OK test VIDIOC_QUERYCAP: OK
test invalid ioctls: OK

Allow for multiple opens:
test second /dev/video0 open: OK
test VIDIOC_QUERYCAP: OK
test VIDIOC_G/S_PRIORITY: OK
test for unlimited opens: OK

Debug ioctls:
test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK
test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 1 Audio Inputs: 0 Tuners: 0

Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_ENUMAUDOUT: OK (Not Supported) test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls (Input 0):
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported) test VIDIOC_QUERYCTRL: OK (Not Supported)
test VIDIOC_G/S_CTRL: OK (Not Supported)
test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported) test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported) test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) Standard Controls: 0 Private Controls: 0

Format ioctls (Input 0):
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK test VIDIOC_G/S_PARM: OK
test VIDIOC_G_FBUF: OK (Not Supported) test VIDIOC_G_FMT: OK
test VIDIOC_TRY_FMT: OK
test VIDIOC_S_FMT: OK
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported) test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK (Not Supported)

Codec ioctls (Input 0):
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported) test VIDIOC_G_ENC_INDEX: OK (Not Supported) test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls (Input 0):
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK test CREATE_BUFS maximum buffers: OK test VIDIOC_REMOVE_BUFS: OK
test VIDIOC_EXPBUF: OK
test Requests: OK (Not Supported) test blocking wait: OK

Total for amd_isp_capture device /dev/video0: 49, Succeeded: 49, Failed: 0, Warnings: 0

Please review and provide feedback.

Many thanks,

Bin Du (7):
media: platform: amd: Introduce amd isp4 capture driver
media: platform: amd: low level support for isp4 firmware
media: platform: amd: Add isp4 fw and hw interface
media: platform: amd: isp4 subdev and firmware loading handling added
media: platform: amd: isp4 video node and buffers handling added
media: platform: amd: isp4 debug fs logging and more descriptive
errors
Documentation: add documentation of AMD isp 4 driver

  Documentation/admin-guide/media/amdisp4-1.rst |   66 +
  Documentation/admin-guide/media/amdisp4.dot   |    8 +
  .../admin-guide/media/v4l-drivers.rst         |    1 +
  MAINTAINERS                                   |   25 +
  drivers/media/platform/Kconfig                |    1 +
  drivers/media/platform/Makefile               |    1 +
  drivers/media/platform/amd/Kconfig            |    3 +
  drivers/media/platform/amd/Makefile           |    3 +
  drivers/media/platform/amd/isp4/Kconfig       |   13 +
  drivers/media/platform/amd/isp4/Makefile      |   10 +
  drivers/media/platform/amd/isp4/isp4.c        |  237 ++++
  drivers/media/platform/amd/isp4/isp4.h        |   26 +
  drivers/media/platform/amd/isp4/isp4_debug.c  |  272 ++++
  drivers/media/platform/amd/isp4/isp4_debug.h  |   41 +
  .../platform/amd/isp4/isp4_fw_cmd_resp.h      |  314 +++++
  drivers/media/platform/amd/isp4/isp4_hw_reg.h |  125 ++
  .../media/platform/amd/isp4/isp4_interface.c  |  972 +++++++++++++
  .../media/platform/amd/isp4/isp4_interface.h  |  149 ++
  drivers/media/platform/amd/isp4/isp4_subdev.c | 1198 ++++++++++++++++
  drivers/media/platform/amd/isp4/isp4_subdev.h |  133 ++
  drivers/media/platform/amd/isp4/isp4_video.c  | 1213 +++++++++++++++++
  drivers/media/platform/amd/isp4/isp4_video.h  |   87 ++
  22 files changed, 4898 insertions(+)

create mode 100644 Documentation/admin-guide/media/amdisp4-1.rst create mode 100644 Documentation/admin-guide/media/amdisp4.dot create mode 100644 drivers/media/platform/amd/Kconfig create mode 100644 drivers/media/platform/amd/Makefile create mode 100644 drivers/media/platform/amd/isp4/Kconfig create mode 100644 drivers/media/platform/amd/isp4/Makefile create mode 100644 drivers/media/platform/amd/isp4/isp4.c create mode 100644 drivers/media/platform/amd/isp4/isp4.h create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.c create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.h create mode 100644 drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h create mode 100644 drivers/media/platform/amd/isp4/isp4_hw_reg.h create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.c create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.h create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h create mode 100644 drivers/media/platform/amd/isp4/isp4_video.c create mode 100644 drivers/media/platform/amd/isp4/isp4_video.h

Signature · 2 lines
Regards,
Bin
#6 lore raw
Quoted 6 lines · Bin

Hi Laurent Pinchart, sorry for the confusion.

Quoted 3 lines · Laurent Pinchart
Quoted 2 lines · Laurent Pinchart

Hi Bin Bu,

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Have you sent out the cover letter only ? I haven't received the rest of the series, and it's not found on lore.kernel.org either.

I've just noticed you sent the rest later and separately, as
https://lore.kernel.org/all/[email protected]/.

Before sending the formal ones, i did some internal test, after the cover letter, when i tried to send the rest, i encountered this error, 4.4.2 Message submission rate for this client has exceeded the configured limit. So i had to wait for some time so i could send again.

FYI - this was an AMD I/T SMTP server problem with git send-email.

I understand some other teams were hitting it too and worked with I/T and it should be fixed now. If you have problems the next go around raise it I/T again.

Quoted 256 lines
Quoted 255 lines
Quoted 254 lines · Bin Du
Quoted 252 lines

Hello,

AMD ISP4 is the AMD image processing gen 4 which can be found in HP ZBook Ultra G1a 14 inch Mobile Workstation PC ( Ryzen AI Max 385)
(https://ubuntu.com/certified/202411-36043)
This patch series introduces the initial driver support for the AMD ISP4.

Patch summary:
- Powers up/off and initializes ISP HW
- Configures and kicks off ISP FW
- Interacts with APP using standard V4l2 interface by video node
- Controls ISP HW and interacts with ISP FW to do image processing
- Support enum/set output image format and resolution
- Support queueing buffer from app and dequeueing ISP filled buffer
to App
- It supports libcamera ver0.2 SimplePipeline
- It is verified on qv4l2, cheese and qcam
- It is verified together with following patches
    platform/x86: Add AMD ISP platform config (https:// lore.kernel.org/all/[email protected]/)
    pinctrl: amd: isp411: Add amdisp GPIO pinctrl (https:// github.com/torvalds/linux/commit/ e97435ab09f3ad7b6a588dd7c4e45a96699bbb4a) drm/amd/amdgpu: Add GPIO resources required for amdisp (https:// gitlab.freedesktop.org/agd5f/linux/-/commit/ ad0f5966ed8297aa47b3184192b00b7379ae0758)

AMD ISP4 Key features:
- Processes bayer raw data from the connected sensor and output them to different YUV formats
- Downscale input image to different output image resolution
- Pipeline to do image processing on the input image including demosaic, denoise, 3A, etc

----------

Changes v2 -> v3:

- All the dependent patches in other modules (drm/amd/amdgpu,
platform/x86, pinctrl/amd) merged on upstream mainline kernel (https://github.com/torvalds/linux) v6.17.
- Removed usage of amdgpu structs in ISP driver. Added helper
functions in amdgpu accepting opaque params from ISP driver to allocate and release ISP GART buffers.
- Moved sensor and MIPI phy control entirely into ISP FW instead of
the previous hybrid approach controlling sensor from both FW and x86 (sensor driver).
- Removed phy configuration and sensor binding as x86 (sensor
driver) had relinquished the sensor control for ISP FW. With this approach the driver will be exposed as web camera like interface.
- New FW with built-in sensor driver is submitted on upstream linux-
firmware repo (https://gitlab.com/kernel-firmware/linux-firmware/).
- Please note the new FW submitted is not directly compatible with
OEM Kernel ISP4.0 (https://github.com/amd/Linux_ISP_Kernel/tree/4.0) and the previous ISP V2 patch series.
- If intend to use the new FW, please rebuild OEM ISP4.0 Kernel with
CONFIG_VIDEO_OV05C10=N and CONFIG_PINCTRL_AMDISP=Y.
- Included critical fixes from Sultan Alsawaf branch (https://
github.com/kerneltoast/kernel_x86_laptop.git) related to managing lifetime of isp buffers.
       media: amd: isp4: Add missing refcount tracking to mmap memop
       media: amd: isp4: Don't put or unmap the dmabuf when detaching
       media: amd: isp4: Don't increment refcount when dmabuf export fails
       media: amd: isp4: Fix possible use-after-free in isp4vid_vb2_put()
       media: amd: isp4: Always export a new dmabuf from get_dmabuf memop
       media: amd: isp4: Fix implicit dmabuf lifetime tracking
       media: amd: isp4: Fix possible use-after-free when putting implicit dmabuf
       media: amd: isp4: Simplify isp4vid_get_dmabuf() arguments
       media: amd: isp4: Move up buf->vaddr check in isp4vid_get_dmabuf()
       media: amd: isp4: Remove unused userptr memops
       media: amd: isp4: Add missing cleanup on error in isp4vid_vb2_alloc()
       media: amd: isp4: Release queued buffers on error in start_streaming
- Addressed all code related upstream comments
- Fix typo errors and other cosmetic issue.

Changes v1 -> v2:

- Fix media CI test errors and valid warnings
- Reduce patch number in the series from 9 to 8 by merging
MAINTAINERS adding patch to the first patch
- In patch 5
    - do modification to use remote endpoint instead of local endpoint
    - use link frequency and port number as start phy parameter instead of extra added phy-id and phy-bit-rate property of endpoint

----------

It passes v4l2 compliance test, the test reports for:

(a) amd_isp_capture device /dev/video0

Compliance test for amd_isp_capture device /dev/video0:
-------------------------------------------------------

atg@atg-HP-PV:~/bin$ ./v4l2-compliance -d /dev/video0
v4l2-compliance 1.29.0-5348, 64 bits, 64-bit time_t
v4l2-compliance SHA: 75e3f0e2c2cb 2025-03-17 18:12:17

Compliance test for amd_isp_capture device /dev/video0:

Driver Info:
         Driver name      : amd_isp_capture
         Card type        : amd_isp_capture
         Bus info         : platform:amd_isp_capture
         Driver version   : 6.14.0
         Capabilities     : 0xa4200001
                 Video Capture
                 I/O MC
                 Streaming
                 Extended Pix Format
                 Device Capabilities
         Device Caps      : 0x24200001
                 Video Capture
                 I/O MC
                 Streaming
                 Extended Pix Format
Media Driver Info:
         Driver name      : amd_isp_capture
         Model            : amd_isp41_mdev
         Serial           :
         Bus info         : platform:amd_isp_capture
         Media version    : 6.14.0
         Hardware revision: 0x00000000 (0)
         Driver version   : 6.14.0
Interface Info:
         ID               : 0x03000005
         Type             : V4L Video
Entity Info:
         ID               : 0x00000003 (3)
         Name             : Preview
         Function         : V4L2 I/O
         Pad 0x01000004   : 0: Sink
           Link 0x02000007: from remote pad 0x1000002 of entity 'amd isp4' (Image Signal Processor): Data, Enabled, Immutable

Required ioctls:
         test MC information (see 'Media Driver Info' above): OK test VIDIOC_QUERYCAP: OK
         test invalid ioctls: OK

Allow for multiple opens:
         test second /dev/video0 open: OK
         test VIDIOC_QUERYCAP: OK
         test VIDIOC_G/S_PRIORITY: OK
         test for unlimited opens: OK

Debug ioctls:
         test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
         test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) test VIDIOC_ENUMAUDIO: OK (Not Supported)
         test VIDIOC_G/S/ENUMINPUT: OK
         test VIDIOC_G/S_AUDIO: OK (Not Supported)
         Inputs: 1 Audio Inputs: 0 Tuners: 0

Output ioctls:
         test VIDIOC_G/S_MODULATOR: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_ENUMAUDOUT: OK (Not Supported) test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) test VIDIOC_G/S_AUDOUT: OK (Not Supported)
         Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
         test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls (Input 0):
         test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported) test VIDIOC_QUERYCTRL: OK (Not Supported)
         test VIDIOC_G/S_CTRL: OK (Not Supported)
         test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported) test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported) test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) Standard Controls: 0 Private Controls: 0

Format ioctls (Input 0):
         test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK test VIDIOC_G/S_PARM: OK
         test VIDIOC_G_FBUF: OK (Not Supported) test VIDIOC_G_FMT: OK
         test VIDIOC_TRY_FMT: OK
         test VIDIOC_S_FMT: OK
         test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported) test Cropping: OK (Not Supported)
         test Composing: OK (Not Supported)
         test Scaling: OK (Not Supported)

Codec ioctls (Input 0):
         test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported) test VIDIOC_G_ENC_INDEX: OK (Not Supported) test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls (Input 0):
         test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK test CREATE_BUFS maximum buffers: OK test VIDIOC_REMOVE_BUFS: OK
         test VIDIOC_EXPBUF: OK
         test Requests: OK (Not Supported) test blocking wait: OK

Total for amd_isp_capture device /dev/video0: 49, Succeeded: 49, Failed: 0, Warnings: 0

Please review and provide feedback.

Many thanks,

Bin Du (7):
   media: platform: amd: Introduce amd isp4 capture driver
   media: platform: amd: low level support for isp4 firmware
   media: platform: amd: Add isp4 fw and hw interface
   media: platform: amd: isp4 subdev and firmware loading handling added
   media: platform: amd: isp4 video node and buffers handling added
   media: platform: amd: isp4 debug fs logging and  more descriptive errors
   Documentation: add documentation of AMD isp 4 driver

  Documentation/admin-guide/media/amdisp4-1.rst |   66 + Documentation/admin-guide/media/amdisp4.dot   |    8 + .../admin-guide/media/v4l-drivers.rst         |    1 + MAINTAINERS                                   |   25 + drivers/media/platform/Kconfig                |    1 + drivers/media/platform/Makefile               |    1 + drivers/media/platform/amd/Kconfig            |    3 + drivers/media/platform/amd/Makefile           |    3 + drivers/media/platform/amd/isp4/Kconfig       |   13 + drivers/media/platform/amd/isp4/Makefile      |   10 + drivers/media/platform/amd/isp4/isp4.c        |  237 ++++ drivers/media/platform/amd/isp4/isp4.h        |   26 + drivers/media/platform/amd/isp4/isp4_debug.c  |  272 ++++ drivers/media/platform/amd/isp4/isp4_debug.h  |   41 + .../platform/amd/isp4/isp4_fw_cmd_resp.h      |  314 +++++ drivers/media/platform/amd/isp4/isp4_hw_reg.h |  125 ++ .../media/platform/amd/isp4/isp4_interface.c  |  972 +++++++++++++ .../media/platform/amd/isp4/isp4_interface.h  |  149 ++ drivers/media/platform/amd/isp4/isp4_subdev.c | 1198 ++++++++++++++++ drivers/media/platform/amd/isp4/isp4_subdev.h |  133 ++ drivers/media/platform/amd/isp4/isp4_video.c  | 1213 +++++++++++++
++++
  drivers/media/platform/amd/isp4/isp4_video.h  |   87 ++

  22 files changed, 4898 insertions(+)

  create mode 100644 Documentation/admin-guide/media/amdisp4-1.rst create mode 100644 Documentation/admin-guide/media/amdisp4.dot create mode 100644 drivers/media/platform/amd/Kconfig create mode 100644 drivers/media/platform/amd/Makefile create mode 100644 drivers/media/platform/amd/isp4/Kconfig create mode 100644 drivers/media/platform/amd/isp4/Makefile create mode 100644 drivers/media/platform/amd/isp4/isp4.c create mode 100644 drivers/media/platform/amd/isp4/isp4.h create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.c create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.h create mode 100644 drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h create mode 100644 drivers/media/platform/amd/isp4/isp4_hw_reg.h create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.c create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.h create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h create mode 100644 drivers/media/platform/amd/isp4/isp4_video.c create mode 100644 drivers/media/platform/amd/isp4/isp4_video.h

author
#7 lore raw

Thanks Mario

Quoted 15 lines · Mario Limonciello
Quoted 13 lines · Bin

Hi Laurent Pinchart, sorry for the confusion.

Quoted 9 lines · Laurent Pinchart
Quoted 4 lines · Laurent Pinchart

Hi Bin Bu,

Have you sent out the cover letter only ? I haven't received the rest of the series, and it's not found on lore.kernel.org either.

I've just noticed you sent the rest later and separately, as
https://lore.kernel.org/all/[email protected]/.

Quoted 6 lines
Quoted 1 line

Before sending the formal ones, i did some internal test, after the cover letter, when i tried to send the rest, i encountered this error, 4.4.2 Message submission rate for this client has exceeded the configured limit. So i had to wait for some time so i could send again.

FYI - this was an AMD I/T SMTP server problem with git send-email.

I understand some other teams were hitting it too and worked with I/T and it should be fixed now.  If you have problems the next go around raise it I/T again.

Good to know, sure, will raise ticket and complain to I/T if run into the same issue next time. :)

Quoted 258 lines
Quoted 257 lines
Quoted 256 lines
Quoted 255 lines · Bin Du
Quoted 253 lines

Hello,

AMD ISP4 is the AMD image processing gen 4 which can be found in HP ZBook Ultra G1a 14 inch Mobile Workstation PC ( Ryzen AI Max 385)
(https://ubuntu.com/certified/202411-36043)
This patch series introduces the initial driver support for the AMD ISP4.

Patch summary:
- Powers up/off and initializes ISP HW
- Configures and kicks off ISP FW
- Interacts with APP using standard V4l2 interface by video node
- Controls ISP HW and interacts with ISP FW to do image processing
- Support enum/set output image format and resolution
- Support queueing buffer from app and dequeueing ISP filled buffer
to App
- It supports libcamera ver0.2 SimplePipeline
- It is verified on qv4l2, cheese and qcam
- It is verified together with following patches
    platform/x86: Add AMD ISP platform config (https:// lore.kernel.org/all/[email protected]/)
    pinctrl: amd: isp411: Add amdisp GPIO pinctrl (https:// github.com/torvalds/linux/commit/ e97435ab09f3ad7b6a588dd7c4e45a96699bbb4a) drm/amd/amdgpu: Add GPIO resources required for amdisp (https:// gitlab.freedesktop.org/agd5f/linux/-/commit/ ad0f5966ed8297aa47b3184192b00b7379ae0758)

AMD ISP4 Key features:
- Processes bayer raw data from the connected sensor and output them to different YUV formats
- Downscale input image to different output image resolution
- Pipeline to do image processing on the input image including demosaic, denoise, 3A, etc

----------

Changes v2 -> v3:

- All the dependent patches in other modules (drm/amd/amdgpu,
platform/x86, pinctrl/amd) merged on upstream mainline kernel (https://github.com/torvalds/linux) v6.17.
- Removed usage of amdgpu structs in ISP driver. Added helper
functions in amdgpu accepting opaque params from ISP driver to allocate and release ISP GART buffers.
- Moved sensor and MIPI phy control entirely into ISP FW instead of
the previous hybrid approach controlling sensor from both FW and x86 (sensor driver).
- Removed phy configuration and sensor binding as x86 (sensor
driver) had relinquished the sensor control for ISP FW. With this approach the driver will be exposed as web camera like interface.
- New FW with built-in sensor driver is submitted on upstream
linux- firmware repo (https://gitlab.com/kernel-firmware/linux- firmware/).
- Please note the new FW submitted is not directly compatible with
OEM Kernel ISP4.0 (https://github.com/amd/Linux_ISP_Kernel/ tree/4.0) and the previous ISP V2 patch series.
- If intend to use the new FW, please rebuild OEM ISP4.0 Kernel
with CONFIG_VIDEO_OV05C10=N and CONFIG_PINCTRL_AMDISP=Y.
- Included critical fixes from Sultan Alsawaf branch (https://
github.com/kerneltoast/kernel_x86_laptop.git) related to managing lifetime of isp buffers.
       media: amd: isp4: Add missing refcount tracking to mmap memop
       media: amd: isp4: Don't put or unmap the dmabuf when detaching
       media: amd: isp4: Don't increment refcount when dmabuf export fails
       media: amd: isp4: Fix possible use-after-free in isp4vid_vb2_put()
       media: amd: isp4: Always export a new dmabuf from get_dmabuf memop
       media: amd: isp4: Fix implicit dmabuf lifetime tracking
       media: amd: isp4: Fix possible use-after-free when putting implicit dmabuf
       media: amd: isp4: Simplify isp4vid_get_dmabuf() arguments
       media: amd: isp4: Move up buf->vaddr check in isp4vid_get_dmabuf()
       media: amd: isp4: Remove unused userptr memops
       media: amd: isp4: Add missing cleanup on error in isp4vid_vb2_alloc()
       media: amd: isp4: Release queued buffers on error in start_streaming
- Addressed all code related upstream comments
- Fix typo errors and other cosmetic issue.

Changes v1 -> v2:

- Fix media CI test errors and valid warnings
- Reduce patch number in the series from 9 to 8 by merging
MAINTAINERS adding patch to the first patch
- In patch 5
    - do modification to use remote endpoint instead of local endpoint
    - use link frequency and port number as start phy parameter instead of extra added phy-id and phy-bit-rate property of endpoint

----------

It passes v4l2 compliance test, the test reports for:

(a) amd_isp_capture device /dev/video0

Compliance test for amd_isp_capture device /dev/video0:
-------------------------------------------------------

atg@atg-HP-PV:~/bin$ ./v4l2-compliance -d /dev/video0
v4l2-compliance 1.29.0-5348, 64 bits, 64-bit time_t
v4l2-compliance SHA: 75e3f0e2c2cb 2025-03-17 18:12:17

Compliance test for amd_isp_capture device /dev/video0:

Driver Info:
         Driver name      : amd_isp_capture
         Card type        : amd_isp_capture
         Bus info         : platform:amd_isp_capture
         Driver version   : 6.14.0
         Capabilities     : 0xa4200001
                 Video Capture
                 I/O MC
                 Streaming
                 Extended Pix Format
                 Device Capabilities
         Device Caps      : 0x24200001
                 Video Capture
                 I/O MC
                 Streaming
                 Extended Pix Format
Media Driver Info:
         Driver name      : amd_isp_capture
         Model            : amd_isp41_mdev
         Serial           :
         Bus info         : platform:amd_isp_capture
         Media version    : 6.14.0
         Hardware revision: 0x00000000 (0)
         Driver version   : 6.14.0
Interface Info:
         ID               : 0x03000005
         Type             : V4L Video
Entity Info:
         ID               : 0x00000003 (3)
         Name             : Preview
         Function         : V4L2 I/O
         Pad 0x01000004   : 0: Sink
           Link 0x02000007: from remote pad 0x1000002 of entity 'amd isp4' (Image Signal Processor): Data, Enabled, Immutable

Required ioctls:
         test MC information (see 'Media Driver Info' above): OK test VIDIOC_QUERYCAP: OK
         test invalid ioctls: OK

Allow for multiple opens:
         test second /dev/video0 open: OK
         test VIDIOC_QUERYCAP: OK
         test VIDIOC_G/S_PRIORITY: OK
         test for unlimited opens: OK

Debug ioctls:
         test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
         test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) test VIDIOC_ENUMAUDIO: OK (Not Supported)
         test VIDIOC_G/S/ENUMINPUT: OK
         test VIDIOC_G/S_AUDIO: OK (Not Supported)
         Inputs: 1 Audio Inputs: 0 Tuners: 0

Output ioctls:
         test VIDIOC_G/S_MODULATOR: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_ENUMAUDOUT: OK (Not Supported) test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) test VIDIOC_G/S_AUDOUT: OK (Not Supported)
         Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
         test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls (Input 0):
         test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported) test VIDIOC_QUERYCTRL: OK (Not Supported)
         test VIDIOC_G/S_CTRL: OK (Not Supported)
         test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported) test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported) test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) Standard Controls: 0 Private Controls: 0

Format ioctls (Input 0):
         test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK test VIDIOC_G/S_PARM: OK
         test VIDIOC_G_FBUF: OK (Not Supported) test VIDIOC_G_FMT: OK
         test VIDIOC_TRY_FMT: OK
         test VIDIOC_S_FMT: OK
         test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported) test Cropping: OK (Not Supported)
         test Composing: OK (Not Supported)
         test Scaling: OK (Not Supported)

Codec ioctls (Input 0):
         test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported) test VIDIOC_G_ENC_INDEX: OK (Not Supported) test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls (Input 0):
         test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK test CREATE_BUFS maximum buffers: OK test VIDIOC_REMOVE_BUFS: OK
         test VIDIOC_EXPBUF: OK
         test Requests: OK (Not Supported) test blocking wait: OK

Total for amd_isp_capture device /dev/video0: 49, Succeeded: 49, Failed: 0, Warnings: 0

Please review and provide feedback.

Many thanks,

Bin Du (7):
   media: platform: amd: Introduce amd isp4 capture driver
   media: platform: amd: low level support for isp4 firmware
   media: platform: amd: Add isp4 fw and hw interface
   media: platform: amd: isp4 subdev and firmware loading handling added
   media: platform: amd: isp4 video node and buffers handling added
   media: platform: amd: isp4 debug fs logging and  more descriptive errors
   Documentation: add documentation of AMD isp 4 driver

  Documentation/admin-guide/media/amdisp4-1.rst |   66 + Documentation/admin-guide/media/amdisp4.dot   |    8 + .../admin-guide/media/v4l-drivers.rst         |    1 + MAINTAINERS                                   |   25 + drivers/media/platform/Kconfig                |    1 + drivers/media/platform/Makefile               |    1 + drivers/media/platform/amd/Kconfig            |    3 + drivers/media/platform/amd/Makefile           |    3 + drivers/media/platform/amd/isp4/Kconfig       |   13 + drivers/media/platform/amd/isp4/Makefile      |   10 + drivers/media/platform/amd/isp4/isp4.c        |  237 ++++ drivers/media/platform/amd/isp4/isp4.h        |   26 + drivers/media/platform/amd/isp4/isp4_debug.c  |  272 ++++ drivers/media/platform/amd/isp4/isp4_debug.h  |   41 + .../platform/amd/isp4/isp4_fw_cmd_resp.h      |  314 +++++ drivers/media/platform/amd/isp4/isp4_hw_reg.h |  125 ++ .../media/platform/amd/isp4/isp4_interface.c  |  972 +++++++++++++ .../media/platform/amd/isp4/isp4_interface.h  |  149 ++ drivers/media/platform/amd/isp4/isp4_subdev.c | 1198 ++++++++++++
++++
  drivers/media/platform/amd/isp4/isp4_subdev.h |  133 ++ drivers/media/platform/amd/isp4/isp4_video.c  | 1213 ++++++++++++
+ ++++
  drivers/media/platform/amd/isp4/isp4_video.h  |   87 ++

  22 files changed, 4898 insertions(+)

  create mode 100644 Documentation/admin-guide/media/amdisp4-1.rst create mode 100644 Documentation/admin-guide/media/amdisp4.dot create mode 100644 drivers/media/platform/amd/Kconfig create mode 100644 drivers/media/platform/amd/Makefile create mode 100644 drivers/media/platform/amd/isp4/Kconfig create mode 100644 drivers/media/platform/amd/isp4/Makefile create mode 100644 drivers/media/platform/amd/isp4/isp4.c create mode 100644 drivers/media/platform/amd/isp4/isp4.h create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.c create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.h create mode 100644 drivers/media/platform/amd/isp4/ isp4_fw_cmd_resp.h create mode 100644 drivers/media/platform/amd/isp4/isp4_hw_reg.h create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.c create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.h create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h create mode 100644 drivers/media/platform/amd/isp4/isp4_video.c create mode 100644 drivers/media/platform/amd/isp4/isp4_video.h

Signature · 2 lines
Regards,
Bin
#8 lore raw
In reply to Bin Du

Hello,

I have tested this patch on my HP ZBook Ultra G1a, and webcam now works great, thank you!

One trick that I noticed, mkinitcpio does not include isp_4_1_1.bin firmware inside the image, however, other amdgpu firmware files are present. So I need to manually add via config.

Should be "MODULE_FIRMWARE("amdgpu/isp_4_1_1.bin");" added in isp_v4_1_1.c like other firmware, for work right out of the box?

Quoted 240 lines · Bin Du

Hello,

AMD ISP4 is the AMD image processing gen 4 which can be found in HP ZBook Ultra G1a 14 inch Mobile Workstation PC ( Ryzen AI Max 385)
(https://ubuntu.com/certified/202411-36043)
This patch series introduces the initial driver support for the AMD ISP4.

Patch summary:
- Powers up/off and initializes ISP HW
- Configures and kicks off ISP FW
- Interacts with APP using standard V4l2 interface by video node
- Controls ISP HW and interacts with ISP FW to do image processing
- Support enum/set output image format and resolution
- Support queueing buffer from app and dequeueing ISP filled buffer to App
- It supports libcamera ver0.2 SimplePipeline
- It is verified on qv4l2, cheese and qcam
- It is verified together with following patches

	platform/x86: Add AMD ISP platform config (https://lore.kernel.org/all/[email protected]/)
	pinctrl: amd: isp411: Add amdisp GPIO pinctrl (https://github.com/torvalds/linux/commit/e97435ab09f3ad7b6a588dd7c4e45a96699bbb4a)
	drm/amd/amdgpu: Add GPIO resources required for amdisp (https://gitlab.freedesktop.org/agd5f/linux/-/commit/ad0f5966ed8297aa47b3184192b00b7379ae0758)

AMD ISP4 Key features:
- Processes bayer raw data from the connected sensor and output them to different YUV formats
- Downscale input image to different output image resolution
- Pipeline to do image processing on the input image including demosaic, denoise, 3A, etc

----------

Changes v2 -> v3:

- All the dependent patches in other modules (drm/amd/amdgpu, platform/x86, pinctrl/amd) merged on upstream mainline kernel (https://github.com/torvalds/linux) v6.17.
- Removed usage of amdgpu structs in ISP driver. Added helper functions in amdgpu accepting opaque params from ISP driver to allocate and release ISP GART buffers.
- Moved sensor and MIPI phy control entirely into ISP FW instead of the previous hybrid approach controlling sensor from both FW and x86 (sensor driver).
- Removed phy configuration and sensor binding as x86 (sensor driver) had relinquished the sensor control for ISP FW. With this approach the driver will be exposed as web camera like interface.
- New FW with built-in sensor driver is submitted on upstream linux-firmware repo (https://gitlab.com/kernel-firmware/linux-firmware/).
- Please note the new FW submitted is not directly compatible with OEM Kernel ISP4.0 (https://github.com/amd/Linux_ISP_Kernel/tree/4.0) and the previous ISP V2 patch series.
- If intend to use the new FW, please rebuild OEM ISP4.0 Kernel with CONFIG_VIDEO_OV05C10=N and CONFIG_PINCTRL_AMDISP=Y.
- Included critical fixes from Sultan Alsawaf branch (https://github.com/kerneltoast/kernel_x86_laptop.git) related to managing lifetime of isp buffers.
media: amd: isp4: Add missing refcount tracking to mmap memop
media: amd: isp4: Don't put or unmap the dmabuf when detaching
media: amd: isp4: Don't increment refcount when dmabuf export fails
media: amd: isp4: Fix possible use-after-free in isp4vid_vb2_put()
media: amd: isp4: Always export a new dmabuf from get_dmabuf memop
media: amd: isp4: Fix implicit dmabuf lifetime tracking
media: amd: isp4: Fix possible use-after-free when putting implicit dmabuf
media: amd: isp4: Simplify isp4vid_get_dmabuf() arguments
media: amd: isp4: Move up buf->vaddr check in isp4vid_get_dmabuf()
media: amd: isp4: Remove unused userptr memops
media: amd: isp4: Add missing cleanup on error in isp4vid_vb2_alloc()
media: amd: isp4: Release queued buffers on error in start_streaming
- Addressed all code related upstream comments
- Fix typo errors and other cosmetic issue.

Changes v1 -> v2:

- Fix media CI test errors and valid warnings
- Reduce patch number in the series from 9 to 8 by merging MAINTAINERS adding patch to the first patch
- In patch 5
- do modification to use remote endpoint instead of local endpoint
- use link frequency and port number as start phy parameter instead of extra added phy-id and phy-bit-rate property of endpoint

----------

It passes v4l2 compliance test, the test reports for:

(a) amd_isp_capture device /dev/video0

Compliance test for amd_isp_capture device /dev/video0:
-------------------------------------------------------

atg@atg-HP-PV:~/bin$ ./v4l2-compliance -d /dev/video0
v4l2-compliance 1.29.0-5348, 64 bits, 64-bit time_t
v4l2-compliance SHA: 75e3f0e2c2cb 2025-03-17 18:12:17

Compliance test for amd_isp_capture device /dev/video0:

Driver Info:

         Driver name      : amd_isp_capture
         Card type        : amd_isp_capture
         Bus info         : platform:amd_isp_capture
         Driver version   : 6.14.0
         Capabilities     : 0xa4200001

Video Capture
I/O MC
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x24200001
Video Capture
I/O MC
Streaming
Extended Pix Format
Media Driver Info:

         Driver name      : amd_isp_capture
         Model            : amd_isp41_mdev
         Serial           :
         Bus info         : platform:amd_isp_capture
         Media version    : 6.14.0
         Hardware revision: 0x00000000 (0)
         Driver version   : 6.14.0

Interface Info:
ID : 0x03000005
Type : V4L Video
Entity Info:

         ID               : 0x00000003 (3)
         Name             : Preview
         Function         : V4L2 I/O
         Pad 0x01000004   : 0: Sink
           Link 0x02000007: from remote pad 0x1000002 of entity 'amd isp4' (Image Signal Processor): Data, Enabled, Immutable

Required ioctls:
test MC information (see 'Media Driver Info' above): OK test VIDIOC_QUERYCAP: OK
test invalid ioctls: OK

Allow for multiple opens:
test second /dev/video0 open: OK
test VIDIOC_QUERYCAP: OK
test VIDIOC_G/S_PRIORITY: OK
test for unlimited opens: OK

Debug ioctls:
test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK
test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 1 Audio Inputs: 0 Tuners: 0

Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_ENUMAUDOUT: OK (Not Supported) test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls (Input 0):
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported) test VIDIOC_QUERYCTRL: OK (Not Supported)
test VIDIOC_G/S_CTRL: OK (Not Supported)
test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported) test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported) test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) Standard Controls: 0 Private Controls: 0

Format ioctls (Input 0):
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK test VIDIOC_G/S_PARM: OK
test VIDIOC_G_FBUF: OK (Not Supported) test VIDIOC_G_FMT: OK
test VIDIOC_TRY_FMT: OK
test VIDIOC_S_FMT: OK
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported) test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK (Not Supported)

Codec ioctls (Input 0):
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported) test VIDIOC_G_ENC_INDEX: OK (Not Supported) test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls (Input 0):
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK test CREATE_BUFS maximum buffers: OK test VIDIOC_REMOVE_BUFS: OK
test VIDIOC_EXPBUF: OK
test Requests: OK (Not Supported) test blocking wait: OK

Total for amd_isp_capture device /dev/video0: 49, Succeeded: 49, Failed: 0, Warnings: 0

Please review and provide feedback.

Many thanks,

Bin Du (7):
media: platform: amd: Introduce amd isp4 capture driver
media: platform: amd: low level support for isp4 firmware
media: platform: amd: Add isp4 fw and hw interface
media: platform: amd: isp4 subdev and firmware loading handling added
media: platform: amd: isp4 video node and buffers handling added
media: platform: amd: isp4 debug fs logging and more descriptive
errors
Documentation: add documentation of AMD isp 4 driver

  Documentation/admin-guide/media/amdisp4-1.rst |   66 +
  Documentation/admin-guide/media/amdisp4.dot   |    8 +
  .../admin-guide/media/v4l-drivers.rst         |    1 +
  MAINTAINERS                                   |   25 +
  drivers/media/platform/Kconfig                |    1 +
  drivers/media/platform/Makefile               |    1 +
  drivers/media/platform/amd/Kconfig            |    3 +
  drivers/media/platform/amd/Makefile           |    3 +
  drivers/media/platform/amd/isp4/Kconfig       |   13 +
  drivers/media/platform/amd/isp4/Makefile      |   10 +
  drivers/media/platform/amd/isp4/isp4.c        |  237 ++++
  drivers/media/platform/amd/isp4/isp4.h        |   26 +
  drivers/media/platform/amd/isp4/isp4_debug.c  |  272 ++++
  drivers/media/platform/amd/isp4/isp4_debug.h  |   41 +
  .../platform/amd/isp4/isp4_fw_cmd_resp.h      |  314 +++++
  drivers/media/platform/amd/isp4/isp4_hw_reg.h |  125 ++
  .../media/platform/amd/isp4/isp4_interface.c  |  972 +++++++++++++
  .../media/platform/amd/isp4/isp4_interface.h  |  149 ++
  drivers/media/platform/amd/isp4/isp4_subdev.c | 1198 ++++++++++++++++
  drivers/media/platform/amd/isp4/isp4_subdev.h |  133 ++
  drivers/media/platform/amd/isp4/isp4_video.c  | 1213 +++++++++++++++++
  drivers/media/platform/amd/isp4/isp4_video.h  |   87 ++
  22 files changed, 4898 insertions(+)

create mode 100644 Documentation/admin-guide/media/amdisp4-1.rst create mode 100644 Documentation/admin-guide/media/amdisp4.dot create mode 100644 drivers/media/platform/amd/Kconfig create mode 100644 drivers/media/platform/amd/Makefile create mode 100644 drivers/media/platform/amd/isp4/Kconfig create mode 100644 drivers/media/platform/amd/isp4/Makefile create mode 100644 drivers/media/platform/amd/isp4/isp4.c create mode 100644 drivers/media/platform/amd/isp4/isp4.h create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.c create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.h create mode 100644 drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h create mode 100644 drivers/media/platform/amd/isp4/isp4_hw_reg.h create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.c create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.h create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h create mode 100644 drivers/media/platform/amd/isp4/isp4_video.c create mode 100644 drivers/media/platform/amd/isp4/isp4_video.h

author
#9 lore raw

Hi Alexey Zagorodnikov.

Quoted 4 lines · Alexey Zagorodnikov

Hello,

I have tested this patch on my HP ZBook Ultra G1a, and webcam now works great, thank you!

Thanks for the verification, really glad it works.

Quoted 4 lines

One trick that I noticed, mkinitcpio does not include isp_4_1_1.bin firmware inside the image, however, other amdgpu firmware files are present. So I need to manually add via config.

Should be "MODULE_FIRMWARE("amdgpu/isp_4_1_1.bin");" added in isp_v4_1_1.c like other firmware, for work right out of the box?

Many thank for the fix, yes MODULE_FIRMWARE("amdgpu/isp_4_1_1.bin") should be added, maybe amdgpu_isp.c is a better place. Will follow up with separated amdgpu patch.

Quoted 253 lines · Bin Du
Quoted 252 lines

Hello,

AMD ISP4 is the AMD image processing gen 4 which can be found in HP ZBook Ultra G1a 14 inch Mobile Workstation PC ( Ryzen AI Max 385)
(https://ubuntu.com/certified/202411-36043)
This patch series introduces the initial driver support for the AMD ISP4.

Patch summary:
- Powers up/off and initializes ISP HW
- Configures and kicks off ISP FW
- Interacts with APP using standard V4l2 interface by video node
- Controls ISP HW and interacts with ISP FW to do image processing
- Support enum/set output image format and resolution
- Support queueing buffer from app and dequeueing ISP filled buffer to
App
- It supports libcamera ver0.2 SimplePipeline
- It is verified on qv4l2, cheese and qcam
- It is verified together with following patches
    platform/x86: Add AMD ISP platform config (https:// lore.kernel.org/all/[email protected]/)
    pinctrl: amd: isp411: Add amdisp GPIO pinctrl (https://github.com/ torvalds/linux/commit/e97435ab09f3ad7b6a588dd7c4e45a96699bbb4a) drm/amd/amdgpu: Add GPIO resources required for amdisp (https:// gitlab.freedesktop.org/agd5f/linux/-/commit/ ad0f5966ed8297aa47b3184192b00b7379ae0758)

AMD ISP4 Key features:
- Processes bayer raw data from the connected sensor and output them to different YUV formats
- Downscale input image to different output image resolution
- Pipeline to do image processing on the input image including demosaic, denoise, 3A, etc

----------

Changes v2 -> v3:

- All the dependent patches in other modules (drm/amd/amdgpu,
platform/x86, pinctrl/amd) merged on upstream mainline kernel (https://github.com/torvalds/linux) v6.17.
- Removed usage of amdgpu structs in ISP driver. Added helper
functions in amdgpu accepting opaque params from ISP driver to allocate and release ISP GART buffers.
- Moved sensor and MIPI phy control entirely into ISP FW instead of
the previous hybrid approach controlling sensor from both FW and x86 (sensor driver).
- Removed phy configuration and sensor binding as x86 (sensor driver)
had relinquished the sensor control for ISP FW. With this approach the driver will be exposed as web camera like interface.
- New FW with built-in sensor driver is submitted on upstream linux-
firmware repo (https://gitlab.com/kernel-firmware/linux-firmware/).
- Please note the new FW submitted is not directly compatible with OEM
Kernel ISP4.0 (https://github.com/amd/Linux_ISP_Kernel/tree/4.0) and the previous ISP V2 patch series.
- If intend to use the new FW, please rebuild OEM ISP4.0 Kernel with
CONFIG_VIDEO_OV05C10=N and CONFIG_PINCTRL_AMDISP=Y.
- Included critical fixes from Sultan Alsawaf branch (https://
github.com/kerneltoast/kernel_x86_laptop.git) related to managing lifetime of isp buffers.
       media: amd: isp4: Add missing refcount tracking to mmap memop
       media: amd: isp4: Don't put or unmap the dmabuf when detaching
       media: amd: isp4: Don't increment refcount when dmabuf export fails
       media: amd: isp4: Fix possible use-after-free in isp4vid_vb2_put()
       media: amd: isp4: Always export a new dmabuf from get_dmabuf memop
       media: amd: isp4: Fix implicit dmabuf lifetime tracking
       media: amd: isp4: Fix possible use-after-free when putting implicit dmabuf
       media: amd: isp4: Simplify isp4vid_get_dmabuf() arguments
       media: amd: isp4: Move up buf->vaddr check in isp4vid_get_dmabuf()
       media: amd: isp4: Remove unused userptr memops
       media: amd: isp4: Add missing cleanup on error in isp4vid_vb2_alloc()
       media: amd: isp4: Release queued buffers on error in start_streaming
- Addressed all code related upstream comments
- Fix typo errors and other cosmetic issue.

Changes v1 -> v2:

- Fix media CI test errors and valid warnings
- Reduce patch number in the series from 9 to 8 by merging MAINTAINERS
adding patch to the first patch
- In patch 5
    - do modification to use remote endpoint instead of local endpoint
    - use link frequency and port number as start phy parameter instead of extra added phy-id and phy-bit-rate property of endpoint

----------

It passes v4l2 compliance test, the test reports for:

(a) amd_isp_capture device /dev/video0

Compliance test for amd_isp_capture device /dev/video0:
-------------------------------------------------------

atg@atg-HP-PV:~/bin$ ./v4l2-compliance -d /dev/video0
v4l2-compliance 1.29.0-5348, 64 bits, 64-bit time_t
v4l2-compliance SHA: 75e3f0e2c2cb 2025-03-17 18:12:17

Compliance test for amd_isp_capture device /dev/video0:

Driver Info:
         Driver name      : amd_isp_capture
         Card type        : amd_isp_capture
         Bus info         : platform:amd_isp_capture
         Driver version   : 6.14.0
         Capabilities     : 0xa4200001
                 Video Capture
                 I/O MC
                 Streaming
                 Extended Pix Format
                 Device Capabilities
         Device Caps      : 0x24200001
                 Video Capture
                 I/O MC
                 Streaming
                 Extended Pix Format
Media Driver Info:
         Driver name      : amd_isp_capture
         Model            : amd_isp41_mdev
         Serial           :
         Bus info         : platform:amd_isp_capture
         Media version    : 6.14.0
         Hardware revision: 0x00000000 (0)
         Driver version   : 6.14.0
Interface Info:
         ID               : 0x03000005
         Type             : V4L Video
Entity Info:
         ID               : 0x00000003 (3)
         Name             : Preview
         Function         : V4L2 I/O
         Pad 0x01000004   : 0: Sink
           Link 0x02000007: from remote pad 0x1000002 of entity 'amd isp4' (Image Signal Processor): Data, Enabled, Immutable

Required ioctls:
         test MC information (see 'Media Driver Info' above): OK test VIDIOC_QUERYCAP: OK
         test invalid ioctls: OK

Allow for multiple opens:
         test second /dev/video0 open: OK
         test VIDIOC_QUERYCAP: OK
         test VIDIOC_G/S_PRIORITY: OK
         test for unlimited opens: OK

Debug ioctls:
         test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported) test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
         test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported) test VIDIOC_ENUMAUDIO: OK (Not Supported)
         test VIDIOC_G/S/ENUMINPUT: OK
         test VIDIOC_G/S_AUDIO: OK (Not Supported)
         Inputs: 1 Audio Inputs: 0 Tuners: 0

Output ioctls:
         test VIDIOC_G/S_MODULATOR: OK (Not Supported) test VIDIOC_G/S_FREQUENCY: OK (Not Supported) test VIDIOC_ENUMAUDOUT: OK (Not Supported) test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported) test VIDIOC_G/S_AUDOUT: OK (Not Supported)
         Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
         test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported) test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported) test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported) test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls (Input 0):
         test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK (Not Supported) test VIDIOC_QUERYCTRL: OK (Not Supported)
         test VIDIOC_G/S_CTRL: OK (Not Supported)
         test VIDIOC_G/S/TRY_EXT_CTRLS: OK (Not Supported) test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK (Not Supported) test VIDIOC_G/S_JPEGCOMP: OK (Not Supported) Standard Controls: 0 Private Controls: 0

Format ioctls (Input 0):
         test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK test VIDIOC_G/S_PARM: OK
         test VIDIOC_G_FBUF: OK (Not Supported) test VIDIOC_G_FMT: OK
         test VIDIOC_TRY_FMT: OK
         test VIDIOC_S_FMT: OK
         test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported) test Cropping: OK (Not Supported)
         test Composing: OK (Not Supported)
         test Scaling: OK (Not Supported)

Codec ioctls (Input 0):
         test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported) test VIDIOC_G_ENC_INDEX: OK (Not Supported) test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls (Input 0):
         test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK test CREATE_BUFS maximum buffers: OK test VIDIOC_REMOVE_BUFS: OK
         test VIDIOC_EXPBUF: OK
         test Requests: OK (Not Supported) test blocking wait: OK

Total for amd_isp_capture device /dev/video0: 49, Succeeded: 49, Failed: 0, Warnings: 0

Please review and provide feedback.

Many thanks,

Bin Du (7):
   media: platform: amd: Introduce amd isp4 capture driver
   media: platform: amd: low level support for isp4 firmware
   media: platform: amd: Add isp4 fw and hw interface
   media: platform: amd: isp4 subdev and firmware loading handling added
   media: platform: amd: isp4 video node and buffers handling added
   media: platform: amd: isp4 debug fs logging and  more descriptive errors
   Documentation: add documentation of AMD isp 4 driver

  Documentation/admin-guide/media/amdisp4-1.rst |   66 + Documentation/admin-guide/media/amdisp4.dot   |    8 + .../admin-guide/media/v4l-drivers.rst         |    1 + MAINTAINERS                                   |   25 + drivers/media/platform/Kconfig                |    1 + drivers/media/platform/Makefile               |    1 + drivers/media/platform/amd/Kconfig            |    3 + drivers/media/platform/amd/Makefile           |    3 + drivers/media/platform/amd/isp4/Kconfig       |   13 + drivers/media/platform/amd/isp4/Makefile      |   10 + drivers/media/platform/amd/isp4/isp4.c        |  237 ++++ drivers/media/platform/amd/isp4/isp4.h        |   26 + drivers/media/platform/amd/isp4/isp4_debug.c  |  272 ++++ drivers/media/platform/amd/isp4/isp4_debug.h  |   41 + .../platform/amd/isp4/isp4_fw_cmd_resp.h      |  314 +++++ drivers/media/platform/amd/isp4/isp4_hw_reg.h |  125 ++ .../media/platform/amd/isp4/isp4_interface.c  |  972 +++++++++++++ .../media/platform/amd/isp4/isp4_interface.h  |  149 ++ drivers/media/platform/amd/isp4/isp4_subdev.c | 1198 ++++++++++++++++ drivers/media/platform/amd/isp4/isp4_subdev.h |  133 ++ drivers/media/platform/amd/isp4/isp4_video.c  | 1213 +++++++++++++++++ drivers/media/platform/amd/isp4/isp4_video.h  |   87 ++

  22 files changed, 4898 insertions(+)

  create mode 100644 Documentation/admin-guide/media/amdisp4-1.rst create mode 100644 Documentation/admin-guide/media/amdisp4.dot create mode 100644 drivers/media/platform/amd/Kconfig create mode 100644 drivers/media/platform/amd/Makefile create mode 100644 drivers/media/platform/amd/isp4/Kconfig create mode 100644 drivers/media/platform/amd/isp4/Makefile create mode 100644 drivers/media/platform/amd/isp4/isp4.c create mode 100644 drivers/media/platform/amd/isp4/isp4.h create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.c create mode 100644 drivers/media/platform/amd/isp4/isp4_debug.h create mode 100644 drivers/media/platform/amd/isp4/isp4_fw_cmd_resp.h create mode 100644 drivers/media/platform/amd/isp4/isp4_hw_reg.h create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.c create mode 100644 drivers/media/platform/amd/isp4/isp4_interface.h create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.c create mode 100644 drivers/media/platform/amd/isp4/isp4_subdev.h create mode 100644 drivers/media/platform/amd/isp4/isp4_video.c create mode 100644 drivers/media/platform/amd/isp4/isp4_video.h

Signature · 2 lines
Regards,
Bin
#10 lore raw
In reply to Bin Du

Hi,

Quoted 7 lines

Changes v2 -> v3:

- All the dependent patches in other modules (drm/amd/amdgpu, platform/x86, pinctrl/amd) merged on upstream mainline kernel (https://github.com/torvalds/linux) v6.17.
- Removed usage of amdgpu structs in ISP driver. Added helper functions in amdgpu accepting opaque params from ISP driver to allocate and release ISP GART buffers.
- Moved sensor and MIPI phy control entirely into ISP FW instead of the previous hybrid approach controlling sensor from both FW and x86 (sensor driver).
- Removed phy configuration and sensor binding as x86 (sensor driver) had relinquished the sensor control for ISP FW. With this approach the driver will be exposed as web camera like interface.
- New FW with built-in sensor driver is submitted on upstream linux-firmware repo (https://gitlab.com/kernel-firmware/linux-firmware/).

The sensor is MIPI CSI and controlled by I2C, isn't it? Then why we put all control logic into FW rather than driver? Any existing CSI camera ISP support implementation are not doing this by FW.

Cheers,
Shengyu

author
#11 lore raw

Hi Shengyu,

Quoted 17 lines · Shengyu Qu

Hi,

Quoted 12 lines

Changes v2 -> v3:

- All the dependent patches in other modules (drm/amd/amdgpu,
platform/x86, pinctrl/amd) merged on upstream mainline kernel (https://github.com/torvalds/linux) v6.17.
- Removed usage of amdgpu structs in ISP driver. Added helper
functions in amdgpu accepting opaque params from ISP driver to allocate and release ISP GART buffers.
- Moved sensor and MIPI phy control entirely into ISP FW instead of
the previous hybrid approach controlling sensor from both FW and x86 (sensor driver).
- Removed phy configuration and sensor binding as x86 (sensor driver)
had relinquished the sensor control for ISP FW. With this approach the driver will be exposed as web camera like interface.
- New FW with built-in sensor driver is submitted on upstream linux-
firmware repo (https://gitlab.com/kernel-firmware/linux-firmware/).

Quoted 2 lines

The sensor is MIPI CSI and controlled by I2C, isn't it? Then why we put all control logic into FW rather than driver? Any existing CSI camera ISP support implementation are not doing this by FW.

Yes, it's MIPI CSI sensor and controlled by I2C. In AMD ISP 3A algo architecture, part of the sensor control resides in the FW and in previous version, sensor init control is put in the sensor driver. With discussion with the V4L2 maintainer, the current approach starting from V3 is to move all sensor control into FW and remove the sensor driver dependency. Please refer to https://lore.kernel.org/all/[email protected]/#t which provides the context and background on how we arrived at the current architectural change in v3.

Quoted 2 lines

Cheers,
Shengyu

Signature · 2 lines
Regards,
Bin
author
#12 lore raw

Hi,

Quoted 20 lines · Bin

Hi Shengyu,

Quoted 17 lines · Shengyu Qu

Hi,

Quoted 12 lines

Changes v2 -> v3:

- All the dependent patches in other modules (drm/amd/amdgpu,
platform/x86, pinctrl/amd) merged on upstream mainline kernel (https://github.com/torvalds/linux) v6.17.
- Removed usage of amdgpu structs in ISP driver. Added helper
functions in amdgpu accepting opaque params from ISP driver to allocate and release ISP GART buffers.
- Moved sensor and MIPI phy control entirely into ISP FW instead of
the previous hybrid approach controlling sensor from both FW and x86 (sensor driver).
- Removed phy configuration and sensor binding as x86 (sensor driver)
had relinquished the sensor control for ISP FW. With this approach the driver will be exposed as web camera like interface.
- New FW with built-in sensor driver is submitted on upstream linux-
firmware repo (https://gitlab.com/kernel-firmware/linux-firmware/).

Quoted 5 lines
Quoted 2 lines

The sensor is MIPI CSI and controlled by I2C, isn't it? Then why we put all control logic into FW rather than driver? Any existing CSI camera ISP support implementation are not doing this by FW.

Yes, it's MIPI CSI sensor and controlled by I2C. In AMD ISP 3A algo architecture, part of the sensor control resides in the FW and in previous version, sensor init control is put in the sensor driver. With discussion with the V4L2 maintainer, the current approach starting from V3 is to move all sensor control into FW and remove the sensor driver dependency. Please refer to https://lore.kernel.org/all/63949b3b- [email protected]/#t which provides the context and background on how we arrived at the current architectural change in v3.

Also quote the comments from Laurent Pinchart in https://lore.kernel.org/all/[email protected]/ "Exposing the camera sensor to both the firmware and the host concurrently is asking for trouble. If you really want to abstract the camera behind a firmware and only expose a webcam-like API (or not even that in this version, as the driver exposes no control as far as I can see), then you should push the whole sensor handling to the firmware too. In my opinion that would not be a good solution compared to exposing the ISP to the host, but it would be better than this hybrid model." Many thanks Laurent Pinchart and Sakari Ailus for the arch change guide.

Quoted 3 lines
Quoted 2 lines

Cheers,
Shengyu

Signature · 2 lines
Regards,
Bin