Skip to content

Separate links and async sub-devices

Sakari Ailus linux-media 7 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v6 Latest38 patches 2 repliesIn review
  2. v538 patches 0 repliesSuperseded
  3. v438 patches 5 repliesSuperseded
Earlier versions · 4
VersionPostedPatchesRepliesStatus
v3 posting 2/232 patches47 repliesSuperseded
v3 posting 1/232 patches2 repliesSuperseded
v231 patches7 repliesSuperseded
v118 patches54 repliesSuperseded

Changes in v6 from the cover letter

- Convert new V4L2 async API usage in ds9ub9xx and atomisp bridge drivers.

- Rebased on the current media stage tree.

Patches in v6 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/38media: v4l: async: Drop v4l2_async_nf_parse_fwnode_endpoints()drivers/media/v4l2-core/v4l2-async.c drivers/media/v4l2-core/v4l2-fwnode.c include/media/v4l2-async.h +1 more+7 -1980 replies
2/38media: v4l: fwnode: Remove unneeded forward declarationinclude/media/v4l2-fwnode.h+0 -20 replies
3/38media: atmel-isi: Remote unneeeded forward declarationdrivers/media/platform/atmel/atmel-isi.h+0 -20 replies
4/38media: xilinx-vipp: Clean up bound async notifier callbackdrivers/media/platform/xilinx/xilinx-vipp.c+5 -270 replies
5/38media: omap3isp: Move link creation to bound callbackdrivers/media/platform/ti/omap3isp/isp.c+19 -190 replies
6/38media: Documentation: v4l: Fix async notifier registration exampleDocumentation/driver-api/media/v4l2-subdev.rst+2 -20 replies
7/38media: Documentation: v4l: Add section titles for asyncDocumentation/driver-api/media/v4l2-subdev.rst+18 -10 replies
8/38media: Documentation: v4l: Fix async sensor subdev helper documentationDocumentation/driver-api/media/v4l2-subdev.rst+5 -50 replies
9/38media: Documentation: v4l: Document missing async subdev functionDocumentation/driver-api/media/v4l2-subdev.rst+5 -90 replies
10/38media: Documentation: v4l: Document v4l2_async_nf_cleanupDocumentation/driver-api/media/v4l2-subdev.rst+5 -60 replies
11/38media: v4l: async: Add some debug printsdrivers/media/v4l2-core/v4l2-async.c+63 -80 replies
12/38media: v4l: async: Clean up testing for duplicate async subdevsdrivers/media/v4l2-core/v4l2-async.c+12 -120 replies
13/38media: v4l: async: Don't check whether asd is NULL in validity checkdrivers/media/v4l2-core/v4l2-async.c+0 -30 replies
14/38media: v4l: async: Make V4L2 async match information a structdrivers/media/v4l2-core/v4l2-async.c include/media/v4l2-async.h+32 -270 replies
15/38media: v4l: async: Rename V4L2_ASYNC_MATCH_ macros, add TYPE_drivers/media/v4l2-core/v4l2-async.c include/media/v4l2-async.h+19 -190 replies
16/38media: v4l: async: Only pass match information for async subdev validationdrivers/media/v4l2-core/v4l2-async.c+56 -530 replies
17/38media: v4l: async: Clean up list heads and entriesdrivers/media/pci/intel/ipu3/ipu3-cio2.c drivers/media/platform/xilinx/xilinx-vipp.c drivers/media/v4l2-core/v4l2-async.c +3 more+46 -460 replies
18/38media: v4l: async: Simplify async sub-device fwnode matchingdrivers/media/i2c/adv748x/adv748x-csi2.c drivers/media/i2c/max9286.c drivers/media/i2c/rdacm20.c +4 more+26 -1240 replies
19/38media: v4l: async: Rename v4l2_async_subdev as v4l2_async_connectionDocumentation/driver-api/media/v4l2-subdev.rst drivers/media/i2c/ds90ub913.c drivers/media/i2c/ds90ub953.c +67 more+368 -3610 replies
20/38media: v4l: async: Clean up error handling in v4l2_async_match_notifydrivers/media/v4l2-core/v4l2-async.c+12 -90 replies
21/38media: v4l: async: Drop duplicate handling when adding connectionsdrivers/media/v4l2-core/v4l2-async.c+8 -190 replies
22/38media: v4l: async: Rework internal listsdrivers/media/pci/intel/ipu3/ipu3-cio2.c drivers/media/platform/renesas/rcar-vin/rcar-core.c drivers/media/platform/renesas/rzg2l-cru/rzg2l-core.c +5 more+68 -650 replies
23/38media: v4l: async: Obtain async connection based on sub-devicedrivers/media/platform/rockchip/rkisp1/rkisp1-csi.c drivers/media/platform/rockchip/rkisp1/rkisp1-isp.c drivers/media/platform/ti/omap3isp/isp.h +6 more+68 -100 replies
24/38media: v4l: async: Allow multiple connections between entitiesdrivers/media/v4l2-core/v4l2-async.c include/media/v4l2-async.h include/media/v4l2-subdev.h+90 -830 replies
25/38media: v4l: async: Drop unneeded list entry initialisationdrivers/media/v4l2-core/v4l2-async.c+0 -20 replies
26/38media: v4l: async: Try more connectionsdrivers/media/v4l2-core/v4l2-async.c+2 -21 reply
27/38media: v4l: async: Support fwnode endpoint list matching for subdevsdrivers/media/v4l2-core/v4l2-async.c drivers/media/v4l2-core/v4l2-subdev.c include/media/v4l2-async.h +1 more+90 -20 replies
28/38media: adv748x: Return to endpoint matchingdrivers/media/i2c/adv748x/adv748x-csi2.c+9 -30 replies
29/38media: pxa_camera: Fix probe error handlingdrivers/media/platform/intel/pxa_camera.c+25 -230 replies
30/38media: pxa_camera: Register V4L2 device earlydrivers/media/platform/intel/pxa_camera.c+10 -90 replies
31/38media: marvell: cafe: Register V4L2 device earlierdrivers/media/platform/marvell/cafe-driver.c drivers/media/platform/marvell/mcam-core.c+9 -110 replies
32/38media: am437x-vpfe: Register V4L2 device earlydrivers/media/platform/ti/am437x/am437x-vpfe.c+13 -140 replies
33/38media: omap3isp: Initialise V4L2 async notifier laterdrivers/media/platform/ti/omap3isp/isp.c+8 -70 replies
34/38media: xilinx-vipp: Init async notifier after registering V4L2 devicedrivers/media/platform/xilinx/xilinx-vipp.c+2 -10 replies
35/38media: davinci: Init async notifier after registering V4L2 devicedrivers/media/platform/ti/davinci/vpif_capture.c+9 -100 replies
36/38media: qcom: Initialise V4L2 async notifier laterdrivers/media/platform/qcom/camss/camss.c+10 -110 replies
37/38media: v4l: async: Set v4l2_device and subdev in async notifier initdrivers/media/i2c/ds90ub913.c drivers/media/i2c/ds90ub953.c drivers/media/i2c/ds90ub960.c +50 more+142 -1540 replies
38/38media: Documentation: v4l: Document sub-device notifiersDocumentation/driver-api/media/v4l2-subdev.rst+11 -01 reply

Reviews trailers per patch

9 of 38 patches have a Reviewed-by or Acked-by.

PatchPhilipp ZabelNiklas SöderlundLaurent PinchartAishwarya KothariLad Prabhakar
1/38T-b T-b R-b T-b T-b
2/38T-b
3/38T-b
4/38T-b T-b R-b T-b T-b
5/38T-b T-b T-b T-b
6/38T-b
7/38T-b
8/38T-b
9/38T-b T-b R-b T-b T-b
10/38T-b
11/38T-b T-b T-b T-b
12/38T-b T-b T-b T-b
13/38T-b T-b R-b T-b T-b
14/38T-b T-b R-b T-b T-b
15/38T-b T-b R-b T-b T-b
16/38T-b T-b R-b T-b T-b
17/38T-b T-b T-b T-b
18/38T-b T-b T-b T-b
19/38T-b T-b T-b T-b
20/38T-b T-b R-b T-b T-b
21/38T-b T-b T-b T-b
22/38T-b T-b T-b T-b
23/38T-b T-b T-b T-b
24/38T-b T-b T-b T-b
25/38T-b T-b T-b T-b
26/38T-b T-b T-b T-b
27/38T-b T-b T-b T-b
28/38T-b T-b T-b T-b
29/38T-b T-b T-b T-b
30/38T-b T-b T-b T-b
31/38T-b T-b T-b T-b
32/38T-b T-b T-b T-b
33/38T-b T-b T-b T-b
34/38T-b T-b R-b T-b T-b
35/38T-b T-b T-b T-b
36/38T-b T-b T-b T-b
37/38T-b T-b T-b T-b
38/38T-b T-b T-b T-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 5 reviewers.

Read the v6 thread

Cover letter [PATCH v6 00/38] Separate links and async sub-devices

Hi all,

This set adds support for multiple downstream links in an async sub-device, by separating the sub-device registration from the link creation.

A new concept, V4L2 async connection is added. A connection later on translates to an MC ancillary or data link. Generally async notifiers have a number of connections but at that level there is no knowledge of how many sub-devices they will connect to. The bound and unbound callbacks now work on connections. For the existing drivers there's only one connection so I do not expect regressions because of that.

Async sub-device fwnode matching will now take place between the device (the dev field of struct v4l2_subdev) and a struct v4l2_async_connection (an endpoint for devices that have endpoints or the device for those that do not). This is because the graph data structure only describes point-to-point connections so therefore defining one end of the connection defines the entire connection.

This set is unlikely to address all needs people have related to the async framework but I think that beyond what it does, it paves some way for addressing more of those additional needs.

To be frank, I'd like to get rid of the entire V4L2 async framework, but it would require allowing much more dynamic driver initialisation, including sub-devices and device nodes popping up in the system in the order and extent there is successfully probed hardware. Until that, and this may well be the entire foreseeable future, we have at least some of this complexity.

This has been tested on ipu3-cio2, i.MX6Q + tc358743, rcar-vin + adv746x, omap3isp + CCS and Renesas RZ/G2L SMARC, two of which (i.MX6 and adv746x) have never worked at the same time without kernel changes for one or the other.

The code also can be found in my async-multi branch, on media tree master.

since v5:

- Convert new V4L2 async API usage in ds9ub9xx and atomisp bridge drivers.

- Rebased on the current media stage tree.

since v4:

- Fixed Tested-by: tags (thanks to Prabhakar).

- Fixed platform_data pointer use/assignment order in davinci driver.

since v3:

- Improve V4L2 async documentation (a few new patches):

- Add subsections

- Document sub-device notifiers and sensor driver async helper

- Some overall improvements

- Document the need for calling v4l2_async_nf_cleanup() to release notifier resources

- Fix bugs in the example (it wouldn't compile)

- Remove redundant forward declarations in drivers/media/platform/atmel/atmel-isi.h and include/media/v4l2-fwnode.h (two new patches).

- Register IRQ just before registering the async notifier in pxa_camera_probe().

- Drop now-obsolete comment in drivers/media/platform/marvell/mcam-core.c.

- Fixes in V4L2 async kerneldoc (wrong field names and changes from subdevices to connections).

- Fix async notifier initialisation in drivers/media/platform/sunxi/sun6i-csi/sun6i_csi_bridge.c . The driver can act as both a master or a sub-notifier.

- Move link creation from complete to bound in omap3isp. Also addresses issues in having sub-notifiers introduced earlier version of this set.

- Small comment changes based on Laurent's review.

- Move patch dropping initialising sub-device's async list later as it cannot safely be applied as early as it was.

- Search the entire async match list for duplicate entries.

since v2:

- Drop the sub-devices (struct v4l2_async_subdev) earlier re-introduced in this patchset. They aren't necessary, instead we can use struct v4l2_subdev to store the information. This simplifies the code quite a bit, including removal of one of the global lists: only the sub-device and notifier lists are left now. The sub-device is assigned to the connection at the bound time: information on which sub-devices are connected via async connections is only actually available at the time of binding.

- Address issues related to sub-device handling in the rcar-vin driver. Also other fixes to the rcar-vin driver (bugs introduced in v2).

- Remove sub-device's notifier field, including an omap3isp patch removing an unnecessary test.

- Drop support for async sub-device side endpoint matching. Convert NXP imx-mipi-csis.

since v1:

- Fixed object relation issues. The set has now been tested on an async sub-device with two pads connected by data links to sub-devices on a notifier (struct) device. (Multiple notifiers should work, too, but has not been tested.)

- Add a function to obtain an async connection based on the sub-device. This is useful for drivers for accessing their own link specific data.

- Improved documentation. Include a patch documenting v4l2_async_nf_add_fwnode().

- Return endpoint matching and address adv748x driver breakage in v1. It's a special case so other drivers can remain simpler.

- Swap notifier initialisation arguments, by making the notifier the first argument.

- Remove extra fwnode_handle_put() in max9286_v4l2_unregister().

- Make struct device available before notifier initialisation for consistent debug messages.

- Simplify notifier and async sub-device linked lists. Consistent list head and list entry naming.

- Drop leftovers from an early experimenation work in rkisp1 and omap3isp drivers.

- Simplify xilinx-vipp sub-device binding.

- Use if()s in notifier_dev() of v4l2-async.c.

- Improved debug messages in v4l2-async.c, use v4l2-async prefix and generally with notifier device.

- Call match types with macros V4L2_ASYNC_MATCH_TYPE_* (was V4L2_ASYNC_MATCH_*).

- Create ancillary links only when the sub-device is registered, not when a connection is bound (which can take place more than once for a sub-device).

- Rename struct v4l2_async_match as v4l2_async_match_desc.

- Perform list initialisation in notifier init rather than registration.

- Get rid of the "readd" parameter for v4l2_async_nf_unbind_all_subdevs().

- Check async sub-device validity on a notifier only when the notifier is registered. This removes extra list traversal and simplifies the code.

- Remove extra list initialisation in v4l2_async_register_subdev().

- Drop v4l2_async_cleanup(). It was no longer useful, called from a single place.

- Lots of kerneldoc fixes (mostly changed argument names).

since RFC v1:

- Address missing API usage changes in a lot of drivers.

- Fix compilation problems in intermediate patches.

- Move V4L2 device registration earlier or move notifier initialisation and fwnode endpoint parsing past the current V4L2 device registration (patches 11--16).

Jacopo Mondi (1):
media: v4l: async: Drop v4l2_async_nf_parse_fwnode_endpoints()

Sakari Ailus (37):
media: v4l: fwnode: Remove unneeded forward declaration
media: atmel-isi: Remote unneeeded forward declaration
media: xilinx-vipp: Clean up bound async notifier callback
media: omap3isp: Move link creation to bound callback
media: Documentation: v4l: Fix async notifier registration example
media: Documentation: v4l: Add section titles for async
media: Documentation: v4l: Fix async sensor subdev helper
documentation
media: Documentation: v4l: Document missing async subdev function
media: Documentation: v4l: Document v4l2_async_nf_cleanup
media: v4l: async: Add some debug prints
media: v4l: async: Clean up testing for duplicate async subdevs
media: v4l: async: Don't check whether asd is NULL in validity check
media: v4l: async: Make V4L2 async match information a struct
media: v4l: async: Rename V4L2_ASYNC_MATCH_ macros, add TYPE_
media: v4l: async: Only pass match information for async subdev
validation
media: v4l: async: Clean up list heads and entries
media: v4l: async: Simplify async sub-device fwnode matching
media: v4l: async: Rename v4l2_async_subdev as v4l2_async_connection
media: v4l: async: Clean up error handling in v4l2_async_match_notify
media: v4l: async: Drop duplicate handling when adding connections
media: v4l: async: Rework internal lists
media: v4l: async: Obtain async connection based on sub-device
media: v4l: async: Allow multiple connections between entities
media: v4l: async: Drop unneeded list entry initialisation
media: v4l: async: Try more connections
media: v4l: async: Support fwnode endpoint list matching for subdevs
media: adv748x: Return to endpoint matching
media: pxa_camera: Fix probe error handling
media: pxa_camera: Register V4L2 device early
media: marvell: cafe: Register V4L2 device earlier
media: am437x-vpfe: Register V4L2 device early
media: omap3isp: Initialise V4L2 async notifier later
media: xilinx-vipp: Init async notifier after registering V4L2 device
media: davinci: Init async notifier after registering V4L2 device
media: qcom: Initialise V4L2 async notifier later
media: v4l: async: Set v4l2_device and subdev in async notifier init
media: Documentation: v4l: Document sub-device notifiers

 .../driver-api/media/v4l2-subdev.rst          | 110 ++-
 drivers/media/i2c/adv748x/adv748x-csi2.c      |  13 +-
 drivers/media/i2c/ds90ub913.c                 |  10 +-
 drivers/media/i2c/ds90ub953.c                 |  10 +-
 drivers/media/i2c/ds90ub960.c                 |  12 +-
 drivers/media/i2c/max9286.c                   |  27 +-
 drivers/media/i2c/rdacm20.c                   |  16 +-
 drivers/media/i2c/rdacm21.c                   |  15 +-
 drivers/media/i2c/st-mipid02.c                |  12 +-
 drivers/media/i2c/tc358746.c                  |  15 +-
 drivers/media/pci/intel/ipu3/ipu3-cio2.c      |  14 +-
 drivers/media/platform/atmel/atmel-isi.c      |  12 +-
 drivers/media/platform/atmel/atmel-isi.h      |   2 -
 drivers/media/platform/cadence/cdns-csi2rx.c  |  10 +-
 drivers/media/platform/intel/pxa_camera.c     |  77 +-
 drivers/media/platform/marvell/cafe-driver.c  |  18 +-
 drivers/media/platform/marvell/mcam-core.c    |  15 +-
 drivers/media/platform/marvell/mmp-driver.c   |   6 +-
 .../platform/microchip/microchip-csi2dc.c     |  11 +-
 .../platform/microchip/microchip-isc-base.c   |   4 +-
 .../media/platform/microchip/microchip-isc.h  |   2 +-
 .../microchip/microchip-sama5d2-isc.c         |   9 +-
 .../microchip/microchip-sama7g5-isc.c         |   9 +-
 drivers/media/platform/nxp/imx-mipi-csis.c    |  17 +-
 drivers/media/platform/nxp/imx7-media-csi.c   |  10 +-
 .../platform/nxp/imx8-isi/imx8-isi-core.c     |  12 +-
 drivers/media/platform/nxp/imx8mq-mipi-csi2.c |  10 +-
 drivers/media/platform/qcom/camss/camss.c     |  26 +-
 drivers/media/platform/qcom/camss/camss.h     |   2 +-
 drivers/media/platform/renesas/rcar-isp.c     |  12 +-
 .../platform/renesas/rcar-vin/rcar-core.c     |  52 +-
 .../platform/renesas/rcar-vin/rcar-csi2.c     |  20 +-
 .../platform/renesas/rcar-vin/rcar-vin.h      |  10 +-
 drivers/media/platform/renesas/rcar_drif.c    |  12 +-
 drivers/media/platform/renesas/renesas-ceu.c  |  10 +-
 .../platform/renesas/rzg2l-cru/rzg2l-core.c   |  14 +-
 .../platform/renesas/rzg2l-cru/rzg2l-cru.h    |   2 +-
 .../platform/renesas/rzg2l-cru/rzg2l-csi2.c   |  12 +-
 .../platform/rockchip/rkisp1/rkisp1-common.h  |   2 +-
 .../platform/rockchip/rkisp1/rkisp1-csi.c     |   7 +-
 .../platform/rockchip/rkisp1/rkisp1-dev.c     |  12 +-
 .../platform/rockchip/rkisp1/rkisp1-isp.c     |   8 +-
 .../platform/samsung/exynos4-is/media-dev.c   |  11 +-
 .../platform/samsung/exynos4-is/media-dev.h   |   2 +-
 drivers/media/platform/st/stm32/stm32-dcmi.c  |  12 +-
 .../platform/sunxi/sun4i-csi/sun4i_csi.c      |  10 +-
 .../sunxi/sun6i-csi/sun6i_csi_bridge.c        |  12 +-
 .../sunxi/sun6i-csi/sun6i_csi_bridge.h        |   2 +-
 .../sunxi/sun6i-mipi-csi2/sun6i_mipi_csi2.c   |  10 +-
 .../sun8i_a83t_mipi_csi2.c                    |  10 +-
 .../media/platform/ti/am437x/am437x-vpfe.c    |  36 +-
 .../media/platform/ti/am437x/am437x-vpfe.h    |   2 +-
 drivers/media/platform/ti/cal/cal.c           |  10 +-
 .../media/platform/ti/davinci/vpif_capture.c  |  35 +-
 drivers/media/platform/ti/omap3isp/isp.c      |  55 +-
 drivers/media/platform/ti/omap3isp/isp.h      |  15 +-
 drivers/media/platform/ti/omap3isp/ispccdc.c  |  13 +-
 drivers/media/platform/ti/omap3isp/ispccp2.c  |   2 +
 drivers/media/platform/ti/omap3isp/ispcsi2.c  |   2 +
 .../media/platform/ti/omap3isp/ispcsiphy.c    |  15 +-
 drivers/media/platform/video-mux.c            |  10 +-
 drivers/media/platform/xilinx/xilinx-vipp.c   |  55 +-
 drivers/media/v4l2-core/v4l2-async.c          | 689 ++++++++++--------
 drivers/media/v4l2-core/v4l2-fwnode.c         | 109 +--
 drivers/media/v4l2-core/v4l2-subdev.c         |  13 +
 .../media/atomisp/pci/atomisp_csi2_bridge.c   |   8 +-
 .../staging/media/atomisp/pci/atomisp_v4l2.c  |   2 +-
 .../media/deprecated/atmel/atmel-isc-base.c   |   4 +-
 .../media/deprecated/atmel/atmel-isc.h        |   2 +-
 .../deprecated/atmel/atmel-sama5d2-isc.c      |   9 +-
 .../deprecated/atmel/atmel-sama7g5-isc.c      |   9 +-
 drivers/staging/media/imx/imx-media-csi.c     |  10 +-
 .../staging/media/imx/imx-media-dev-common.c  |   6 +-
 drivers/staging/media/imx/imx-media-dev.c     |   2 +-
 drivers/staging/media/imx/imx-media-of.c      |   4 +-
 drivers/staging/media/imx/imx6-mipi-csi2.c    |  12 +-
 .../media/sunxi/sun6i-isp/sun6i_isp_proc.c    |   6 +-
 .../media/sunxi/sun6i-isp/sun6i_isp_proc.h    |   2 +-
 drivers/staging/media/tegra-video/vi.c        |  21 +-
 include/media/davinci/vpif_types.h            |   2 +-
 include/media/v4l2-async.h                    | 238 +++---
 include/media/v4l2-fwnode.h                   |  70 --
 include/media/v4l2-subdev.h                   |  19 +-
 83 files changed, 1075 insertions(+), 1160 deletions(-)
Signature · 1 line
2.39.2