Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1619A1DED53 for ; Fri, 22 Nov 2024 10:06:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.9 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732270018; cv=none; b=boRoysFEpfDepx53j2yJZ+N6heJecQfVXfx3ngD4lGa5yR6+NDcJA0Yg1u2N2FpC2bjORULNoNJ6Z+7tqF96Uq7TaKS6d29bee5xOqNc0yEqDrXNLltjfkhTCNHKCTcqkzLJgpDY2+0fiCxsKFQTDq8kZmi2jvbDDlFIYByC9JM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732270018; c=relaxed/simple; bh=tJsom9Lw1f0xcgHsaOpi52x91chLoU1mAB23TRtdTlM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=pv9xbAu1vxaRGTwZCN8ijrCb437e3G7oYpU9ssMRpDdydHmmEIJBhT6IRIvjxE7ULJ/iJMFFrvgMfW2HFODMvjDj6KGmY0Ef56VmepnnkE0uFhm1KvWPGz5kr2k2aYM6u1lxv+LcXZMgwjHzfFwO8YyOwjVOpmLxG8X77419ItU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=himuXO2Z; arc=none smtp.client-ip=198.175.65.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="himuXO2Z" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1732270015; x=1763806015; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=tJsom9Lw1f0xcgHsaOpi52x91chLoU1mAB23TRtdTlM=; b=himuXO2ZxBqzCq33lEkKwPLc84XkoqS6euFN6/2R8UYI/hRwH78k2IKb 6XByq2y781pjaT4H0QV3gR0kbyI6n/+3omkLf4TG2SU3EQH+RI7s2eUQj /oye/3lllVFH2Jq1l2UmRvVP8/3Iw2UpAeTpzVsU0fulh/HaY2l+IgkDj xsvpQVmn/ox+NEbfVH1oQqCi40j8UVWdBFsHkEdhcMm8/UdV0Uw8OonEP rmNi7XCuhVA1kcbVVtS/47lShRNbBHOyj795clOOB+9YF6afsBor0Odcx nzvd7bhvTvEKYKoxKUkYE7TWEBbAIMQZODUQENQ3oyET80x6g95ZD+EaH Q==; X-CSE-ConnectionGUID: wBlF9Rt8RFazzH/y4zZoPg== X-CSE-MsgGUID: wffmJhMmRsmb/5m7Cb9BZQ== X-IronPort-AV: E=McAfee;i="6700,10204,11263"; a="54927599" X-IronPort-AV: E=Sophos;i="6.12,175,1728975600"; d="scan'208";a="54927599" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by orvoesa101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Nov 2024 02:06:53 -0800 X-CSE-ConnectionGUID: e8VSYI2XTEGdo2Nod1rJ2A== X-CSE-MsgGUID: 1QPbZuR9SN+vWl+vMjlY7A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,175,1728975600"; d="scan'208";a="121403064" Received: from turnipsi.fi.intel.com (HELO kekkonen.fi.intel.com) ([10.237.72.44]) by orviesa002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Nov 2024 02:06:47 -0800 Received: from punajuuri.localdomain (punajuuri.localdomain [192.168.240.130]) by kekkonen.fi.intel.com (Postfix) with ESMTP id 9039F11FBCC; Fri, 22 Nov 2024 12:06:43 +0200 (EET) Received: from sailus by punajuuri.localdomain with local (Exim 4.96) (envelope-from ) id 1tEQYl-0002LR-1r; Fri, 22 Nov 2024 12:06:43 +0200 Organization: Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo From: Sakari Ailus To: linux-media@vger.kernel.org Cc: hverkuil@xs4all.nl, laurent.pinchart@ideasonboard.com, Prabhakar , Kate Hsuan , Alexander Shiyan , Mikhail Rudenko , Dave Stevenson , Tommaso Merciai , Umang Jain , Benjamin Mugnier , Sylvain Petinot , Christophe JAILLET , Julien Massot , Naushir Patuck , "Yan, Dongcheng" , "Cao, Bingbu" , "Qiu, Tian Shu" , "Wang, Hongju" , Stefan Klug , Mirela Rabulea , =?UTF-8?q?Andr=C3=A9=20Apitzsch?= , Heimir Thor Sverrisson , Kieran Bingham , Stanislaw Gruszka , Mehdi Djait Subject: [PATCH 4/8] media: Documentation: Add subdev configuration models, raw sensor model Date: Fri, 22 Nov 2024 12:06:29 +0200 Message-Id: <20241122100633.8971-5-sakari.ailus@linux.intel.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241122100633.8971-1-sakari.ailus@linux.intel.com> References: <20241122100633.8971-1-sakari.ailus@linux.intel.com> Precedence: bulk X-Mailing-List: linux-media@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sub-device configuration models define what V4L2 API elements are available on a compliant sub-device and how do they behave. The patch also adds a model for common raw sensors. Signed-off-by: Sakari Ailus --- .../media/drivers/camera-sensor.rst | 4 + .../media/v4l/common-raw-sensor.dia | 441 ++++++++++++++++++ .../media/v4l/common-raw-sensor.svg | 134 ++++++ .../userspace-api/media/v4l/dev-subdev.rst | 2 + .../media/v4l/subdev-config-model.rst | 203 ++++++++ 5 files changed, 784 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/common-raw-sensor= .dia create mode 100644 Documentation/userspace-api/media/v4l/common-raw-sensor= .svg create mode 100644 Documentation/userspace-api/media/v4l/subdev-config-mod= el.rst diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/= Documentation/userspace-api/media/drivers/camera-sensor.rst index bc55c861fb69..5bc4c79d230c 100644 --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst @@ -18,6 +18,8 @@ binning functionality. The sensor drivers belong to two d= istinct classes, freely configurable and register list-based drivers, depending on how the driver configures this functionality. =20 +Also see :ref:`media_subdev_config_model_common_raw_sensor`. + Freely configurable camera sensor drivers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 @@ -105,6 +107,8 @@ values programmed by the register sequences. The defaul= t values of these controls shall be 0 (disabled). Especially these controls shall not be inv= erted, independently of the sensor's mounting rotation. =20 +.. _media_using_camera_sensor_drivers_embedded_data: + Embedded data ------------- =20 diff --git a/Documentation/userspace-api/media/v4l/common-raw-sensor.dia b/= Documentation/userspace-api/media/v4l/common-raw-sensor.dia new file mode 100644 index 000000000000..aa927527eae3 --- /dev/null +++ b/Documentation/userspace-api/media/v4l/common-raw-sensor.dia @@ -0,0 +1,441 @@ + + + + + + + + + + + + + #A4# + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + #image data (1)# + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + #embedded data (2)# + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + #source pad (0)# + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/Documentation/userspace-api/media/v4l/common-raw-sensor.svg b/= Documentation/userspace-api/media/v4l/common-raw-sensor.svg new file mode 100644 index 000000000000..1d6055da2519 --- /dev/null +++ b/Documentation/userspace-api/media/v4l/common-raw-sensor.svg @@ -0,0 +1,134 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documen= tation/userspace-api/media/v4l/dev-subdev.rst index dcfcbd52490d..4d145bd3bd09 100644 --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst @@ -838,3 +838,5 @@ stream while it may be possible to enable and disable t= he embedded data stream. =20 The embedded data format does not need to be configured on the sensor's pa= ds as the format is dictated by the pixel data format in this case. + +.. include:: subdev-config-model.rst diff --git a/Documentation/userspace-api/media/v4l/subdev-config-model.rst = b/Documentation/userspace-api/media/v4l/subdev-config-model.rst new file mode 100644 index 000000000000..25030b80734f --- /dev/null +++ b/Documentation/userspace-api/media/v4l/subdev-config-model.rst @@ -0,0 +1,203 @@ +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later + +.. _media_subdev_config_model: + +Sub-device configuration models +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D + +A sub-device configuration model specifies in detail what the user space c= an +expect from a sub-device in terms of V4L2 sub-device interface support, +including IOCTL (including selection targets and controls) semantics. + +A sub-device may implement more than one configuration model at the same +time. The implemented configuration models can be obtained from the sub-de= vice's +``V4L2_CID_CONFIG_MODEL`` control. + +.. _media_subdev_config_model_common_raw_sensor: + +Common raw camera sensor model +------------------------------ + +The common raw camera sensor model defines a set of enumeration and +configuration interfaces (formats, selections etc.) that cover the vast ma= jority +of funcitionality of raw camera sensors. Not all of the interfaces are +necessarily offered by all drivers. + +A sub-device complies with the common raw sensor model if the +``V4L2_CONFIG_MODEL_COMMON_RAW`` bit is set in the ``V4L2_CID_CONFIG_MODEL= `` +control of the sub-device. + +The common raw camera sensor model is aligned with +:ref:`media_using_camera_sensor_drivers`. Please refer to that regarding a= spects +not specified here. + +Each camera sensor implementing the common raw sensor model exposes a sing= le +V4L2 sub-device. The sub-device contains a single source pad (0) and two o= r more +internal pads: an image data internal pad (1) and optionally an embedded d= ata +pad (2). Additionally, further internal pads may be supported for other +features, in which case they are documented separately for the given devic= e. + +This is shown in :ref:`media_subdev_config_model_common_raw_sensor_subdev`. + +.. _media_subdev_config_model_common_raw_sensor_subdev: + +.. kernel-figure:: common-raw-sensor.svg + :alt: common-raw-sensor.svg + :align: center + + **Common raw sensor sub-device** + +Routes +^^^^^^ + +A sub-device conforming to common raw camera sensor model implements the +following routes. + +.. flat-table:: Routes + :header-rows: 1 + + * - Sink pad/stream + - Source pad/stream + - Static (X/M(aybe)/-) + - Mandatory (X/-) + - Synopsis + * - 1/0 + - 0/0 + - X + - X + - Image data + * - 2/0 + - 0/1 + - M + - \- + - Embedded data + +Some devices do not support the embedded data stream, others do support it= and +in some of the latter, it can be turned on and off before streaming is sta= rted. + +Sensor pixel array size, cropping and binning +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The sensor's pixel array is divided into one or more areas. The areas arou= nd the +edge of the pixel array, usually one or more sides, may contain optical bl= ack +pixels, dummy pixels and other non-image pixels. + +A rectangle within the pixel array contains the visible pixels. Capturing = the +non-visible pixels may be supported by the sensor. + +The sensor can perform three operations that affect the output image size.= First +comes analogue crop. This configuration limits parts of the pixel array wh= ich +the sensor will read, affecting sensor timing as well. The granularity of = the +analogue crop configuration varies greatly across sensors: some sensors su= pport +a few different analogue crop configurations whereas others may support an= ything +divisible by a given number of pixels. + +The default analogue crop rectangle corresponds to the visible pixel area. + +In the next step, binning is performed on the image data read from camera +sensor's pixel array. This will effectively result in an image smaller tha= n the +original by given binning factors horizontally and vertically. Typical val= ues +are 1/2 and 1/3 but others may well be supported by the hardware as well. + +Sub-sampling follows binning. Sub-sampling, like binning, reduces the size= of +the image by including only a subset of samples read from the sensor's pix= el +matrix, typically every n'th pixel horizontally and vertically, taking the +sensor's colour pattern into account. Sub-sampling is generally configurab= le +separately horizontally and vertically. + +Scaling is further done after sub-sampling. Scaling generally is fine grai= ner +than binning or sub-sampling. + +The binning, sub-sampling and scaling operations are configured using the +compose rectangle. It depends on the driver which of these operations are = being +used to achieve the resulting size. + +The combination of the analogue crop and binning, sub-sampling and scaling +operations may result in an image size that may be larger than desirable. = For +this purpose, a digital crop operation may be performed on the image after= these +operations. The resulting image size is further output by the sensor. + +Drivers may only support some of even none of these configurations, in whi= ch +case they do not expose the corresponding selection rectangles. + +Also refer to :ref:`Selection targets `. + +.. flat-table:: Selection targets on pads + :header-rows: 1 + + * - Pad/Stream + - Selection target/format + - Mandatory (X/-) + - Modifiable (X/-) + - Synopsis + * - 1/0 + - Format + - X + - \- + - Image data format. The width and height fields of this format are = the + same than those for the V4L2_SEL_TGT_CROP_BOUNDS rectangle. The me= dia + bus code of this format reflects the native pixel depth of the sen= sor. + * - 1/0 + - V4L2_SEL_TGT_CROP_BOUNDS + - X + - \- + - The full pixel area, including non-visible pixels. No pixels outsi= de + this area can be captured. This rectangle is relative to the forma= t on + the same (pad, stream). + * - 1/0 + - V4L2_SEL_TGT_CROP_DEFAULT + - X + - \ + - The visible pixel area. This rectangle is relative to the format o= n the + same (pad, stream). + * - 1/0 + - V4L2_SEL_TGT_CROP + - \- + - X + - Analogue crop. Analogue crop typically has a coarse granularity. T= his + rectangle is relative to the format on the same (pad, stream). + * - 1/0 + - V4L2_SEL_TGT_COMPOSE + - \- + - X + - Binning, sub-sampling and scaling. This rectangle is relative to t= he + V4L2_SEL_TGT_CROP rectangle on the same (pad, stream). The combina= tion + of binning, sub-sampling and scaling is configured using this sele= ction + target. + * - 2/0 + - Format + - X + - \- + - Embedded data format. + * - 0/0 + - V4L2_SEL_TGT_CROP + - \- + - X + - Digital crop. This rectangle is relative to the V4L2_SEL_TGT_COMPO= SE + rectangle on (pad, stream) pair 1/0. + * - 0/0 + - Format + - X + - X + - Image data source format. The width and height fields of the forma= t are + the same than for the V4L2_SEL_TGT_CROP rectangle on (pad, stream)= pair + 0/0 and the media bus code reflects the pixel data output of the s= ensor. + * - 0/1 + - Format + - X + - \- + - Embedded data source format. + +Embedded data +^^^^^^^^^^^^^ + +The embedded data stream is produced by the sensor when the corresponding = route +is enabled. The embedded data route may also be immutable or not exist at = all, +in case the sensor (or the driver) does not support it. + +Generally the sensor embedded data width is determined by the width of the= image +data whereas the number of lines are constant for the embedded data. The u= ser +space may obtain the size of the embedded data once the image data size on= the +source pad has been configured. + +Also see :ref:`media_using_camera_sensor_drivers_embedded_data`. --=20 2.39.5