Skip to contentSkip to conversation

media: add imx319 camera sensor driver

PATCH In review [email protected] 4 participants 10 messages Sep 26, 2018 – Oct 21, 2018

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH v7 1/1 media: add imx319 camera sensor driver

From: Bingbu Cao <[email protected]>

Add a v4l2 sub-device driver for the Sony imx319 image sensor. This is a camera sensor using the i2c bus for control and the csi-2 bus for data.

This driver supports following features:
- manual exposure and analog/digital gain control support
- vblank/hblank control support
- 4 test patterns control support
- vflip/hflip control support (will impact the output bayer order)
- support following resolutions:
- 3264x2448, 3280x2464 @ 30fps
- 1936x1096, 1920x1080 @ 60fps
- 1640x1232, 1640x922, 1296x736, 1280x720 @ 120fps
- support 4 bayer orders output (via change v/hflip)
- SRGGB10(default), SGRBG10, SGBRG10, SBGGR10

---

This patch is based on sakari's media-tree git:
https://git.linuxtv.org/sailus/media_tree.git/log/?h=for-4.20-1

Changes from v5:
- add some comments for gain calculation
- use lock to protect the format
- fix some style issues

Changes from v4 to v5:
- use single PLL for all internal clocks
- change link frequency to 482.4MHz
- adjust frame timing for 2x2 binning modes and enlarge frame readout time
- get CSI-2 link frequencies and external clock from firmware
- use unlocked __v4l2_ctrl_grab() with change from:
https://git.linuxtv.org/sailus/media_tree.git/commit/?h=unlocked-ctrl-grab

Changes since v1:
- fix some coding style issues - line breaks
- add v4l2_ctrl_grab() to prevent v/hflip change
during streaming
- add v4l2 ctrl event (un)subscribe support
- add more info into commit message

---

---
 MAINTAINERS                |    7 +
 drivers/media/i2c/Kconfig  |   11 +
 drivers/media/i2c/Makefile |    1 +
 drivers/media/i2c/imx319.c | 2556 ++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 2575 insertions(+)
 create mode 100644 drivers/media/i2c/imx319.c
MAINTAINERS +7 -0
index a5b256b25905..abc4abb6f83c 100644
@@ -13530,6 +13530,13 @@ S:	Maintained
 F:	drivers/media/i2c/imx274.c
 F:	Documentation/devicetree/bindings/media/i2c/imx274.txt
 
+SONY IMX319 SENSOR DRIVER
+M:	Bingbu Cao <[email protected]>
+L:	[email protected]
+T:	git git://linuxtv.org/media_tree.git
+S:	Maintained
+F:	drivers/media/i2c/imx319.c
+
 SONY MEMORYSTICK CARD SUPPORT
 M:	Alex Dubov <[email protected]>
 W:	http://tifmxx.berlios.de/
drivers/media/i2c/Kconfig +11 -0
index bfdb494686bf..603ac087975b 100644
@@ -614,6 +614,17 @@ config VIDEO_IMX274
 	  This is a V4L2 sensor driver for the Sony IMX274
 	  CMOS image sensor.
 
+config VIDEO_IMX319
+	tristate "Sony IMX319 sensor support"
+	depends on I2C && VIDEO_V4L2 && VIDEO_V4L2_SUBDEV_API
+	depends on MEDIA_CAMERA_SUPPORT
+	help
+	  This is a Video4Linux2 sensor driver for the Sony
+	  IMX319 camera.
+
+	  To compile this driver as a module, choose M here: the
+	  module will be called imx319.
+
 config VIDEO_OV2640
 	tristate "OmniVision OV2640 sensor support"
 	depends on VIDEO_V4L2 && I2C
drivers/media/i2c/Makefile +1 -0
index a94eb03d10d4..d10b438577be 100644
@@ -108,5 +108,6 @@ obj-$(CONFIG_VIDEO_OV2659)	+= ov2659.o
 obj-$(CONFIG_VIDEO_TC358743)	+= tc358743.o
 obj-$(CONFIG_VIDEO_IMX258)	+= imx258.o
 obj-$(CONFIG_VIDEO_IMX274)	+= imx274.o
+obj-$(CONFIG_VIDEO_IMX319)	+= imx319.o
 
 obj-$(CONFIG_SDR_MAX2175) += max2175.o
drivers/media/i2c/imx319.c new file +2556 -0

2,556 lines not on this page · raw message

Signature · 1 line
1.9.1
maintainer
#1 lore raw

Hi Bingbu,

Quoted 25 lines · On Wed, Sep 26, 2018 at 10:42:18AM +0800, [email protected] wrote:

From: Bingbu Cao <[email protected]>

Add a v4l2 sub-device driver for the Sony imx319 image sensor. This is a camera sensor using the i2c bus for control and the csi-2 bus for data.

This driver supports following features:
- manual exposure and analog/digital gain control support
- vblank/hblank control support
- 4 test patterns control support
- vflip/hflip control support (will impact the output bayer order)
- support following resolutions:
- 3264x2448, 3280x2464 @ 30fps
- 1936x1096, 1920x1080 @ 60fps
- 1640x1232, 1640x922, 1296x736, 1280x720 @ 120fps
- support 4 bayer orders output (via change v/hflip)
- SRGGB10(default), SGRBG10, SGBRG10, SBGGR10

---

Quoted 7 lines

This patch is based on sakari's media-tree git:
https://git.linuxtv.org/sailus/media_tree.git/log/?h=for-4.20-1

Changes from v5:
- add some comments for gain calculation
- use lock to protect the format
- fix some style issues

Thanks for the update!

I've applied the patch with the following diff. Dividing a 64-bit number generally requires do_div() which was missed in the review:

drivers/media/i2c/imx319.c +6 -4
index e10d60f500dd..37c31d17ecf0 100644
@@ -2038,7 +2038,7 @@ imx319_set_pad_format(struct v4l2_subdev *sd,
 	s32 vblank_def;
 	s32 vblank_min;
 	s64 h_blank;
-	s64 pixel_rate;
+	u64 pixel_rate;
 	u32 height;
 
 	mutex_lock(&imx319->mutex);
@@ -2059,7 +2059,8 @@ imx319_set_pad_format(struct v4l2_subdev *sd,
 		*framefmt = fmt->format;
 	} else {
 		imx319->cur_mode = mode;
-		pixel_rate = (imx319->link_def_freq * 2 * 4) / 10;
+		pixel_rate = imx319->link_def_freq * 2 * 4;
+		do_div(pixel_rate, 10);
 		__v4l2_ctrl_s_ctrl_int64(imx319->pixel_rate, pixel_rate);
 		/* Update limits and set FPS to default */
 		height = imx319->cur_mode->height;
@@ -2268,7 +2269,7 @@ static int imx319_init_controls(struct imx319 *imx319)
 	s64 vblank_def;
 	s64 vblank_min;
 	s64 hblank;
-	s64 pixel_rate;
+	u64 pixel_rate;
 	const struct imx319_mode *mode;
 	u32 max;
 	int ret;
@@ -2287,7 +2288,8 @@ static int imx319_init_controls(struct imx319 *imx319)
 		imx319->link_freq->flags |= V4L2_CTRL_FLAG_READ_ONLY;
 
 	/* pixel_rate = link_freq * 2 * nr_of_lanes / bits_per_sample */
-	pixel_rate = (imx319->link_def_freq * 2 * 4) / 10;
+	pixel_rate = imx319->link_def_freq * 2 * 4;
+	do_div(pixel_rate, 10);
 	/* By default, PIXEL_RATE is read only */
 	imx319->pixel_rate = v4l2_ctrl_new_std(ctrl_hdlr, &imx319_ctrl_ops,
 					       V4L2_CID_PIXEL_RATE, pixel_rate,
Signature · 4 lines
Kind regards,

Sakari Ailus
[email protected]
#2 lore raw
Quoted 31 lines · Sakari Ailus

Hi Bingbu,

Quoted 28 lines · On Wed, Sep 26, 2018 at 10:42:18AM +0800, [email protected] wrote:

From: Bingbu Cao <[email protected]>

Add a v4l2 sub-device driver for the Sony imx319 image sensor. This is a camera sensor using the i2c bus for control and the csi-2 bus for data.

This driver supports following features:
- manual exposure and analog/digital gain control support
- vblank/hblank control support
- 4 test patterns control support
- vflip/hflip control support (will impact the output bayer order)
- support following resolutions:
- 3264x2448, 3280x2464 @ 30fps
- 1936x1096, 1920x1080 @ 60fps
- 1640x1232, 1640x922, 1296x736, 1280x720 @ 120fps
- support 4 bayer orders output (via change v/hflip)
- SRGGB10(default), SGRBG10, SGBRG10, SBGGR10

---

This patch is based on sakari's media-tree git:
https://git.linuxtv.org/sailus/media_tree.git/log/?h=for-4.20-1

Quoted 8 lines
Quoted 4 lines

Changes from v5:
- add some comments for gain calculation
- use lock to protect the format
- fix some style issues

Thanks for the update!

I've applied the patch with the following diff. Dividing a 64-bit number generally requires do_div() which was missed in the review:

Thanks!

Quoted 44 lines
drivers/media/i2c/imx319.c +6 -4
index e10d60f500dd..37c31d17ecf0 100644
@@ -2038,7 +2038,7 @@ imx319_set_pad_format(struct v4l2_subdev *sd,
 	s32 vblank_def;
 	s32 vblank_min;
 	s64 h_blank;
-	s64 pixel_rate;
+	u64 pixel_rate;
 	u32 height;
 
 	mutex_lock(&imx319->mutex);
@@ -2059,7 +2059,8 @@ imx319_set_pad_format(struct v4l2_subdev *sd,
 		*framefmt = fmt->format;
 	} else {
 		imx319->cur_mode = mode;
-		pixel_rate = (imx319->link_def_freq * 2 * 4) / 10;
+		pixel_rate = imx319->link_def_freq * 2 * 4;
+		do_div(pixel_rate, 10);
 		__v4l2_ctrl_s_ctrl_int64(imx319->pixel_rate, pixel_rate);
 		/* Update limits and set FPS to default */
 		height = imx319->cur_mode->height;
@@ -2268,7 +2269,7 @@ static int imx319_init_controls(struct imx319 *imx319)
 	s64 vblank_def;
 	s64 vblank_min;
 	s64 hblank;
-	s64 pixel_rate;
+	u64 pixel_rate;
 	const struct imx319_mode *mode;
 	u32 max;
 	int ret;
@@ -2287,7 +2288,8 @@ static int imx319_init_controls(struct imx319 *imx319)
 		imx319->link_freq->flags |= V4L2_CTRL_FLAG_READ_ONLY;
 
 	/* pixel_rate = link_freq * 2 * nr_of_lanes / bits_per_sample */
-	pixel_rate = (imx319->link_def_freq * 2 * 4) / 10;
+	pixel_rate = imx319->link_def_freq * 2 * 4;
+	do_div(pixel_rate, 10);
 	/* By default, PIXEL_RATE is read only */
 	imx319->pixel_rate = v4l2_ctrl_new_std(ctrl_hdlr, &imx319_ctrl_ops,
 					       V4L2_CID_PIXEL_RATE, pixel_rate,
#3 lore raw
In reply to [email protected]

Hi Sakari,

Quoted 34 lines · On Wed, Sep 26, 2018 at 11:38 AM <[email protected]> wrote:

From: Bingbu Cao <[email protected]>

Add a v4l2 sub-device driver for the Sony imx319 image sensor. This is a camera sensor using the i2c bus for control and the csi-2 bus for data.

This driver supports following features:
- manual exposure and analog/digital gain control support
- vblank/hblank control support
- 4 test patterns control support
- vflip/hflip control support (will impact the output bayer order)
- support following resolutions:
- 3264x2448, 3280x2464 @ 30fps
- 1936x1096, 1920x1080 @ 60fps
- 1640x1232, 1640x922, 1296x736, 1280x720 @ 120fps
- support 4 bayer orders output (via change v/hflip)
- SRGGB10(default), SGRBG10, SGBRG10, SBGGR10

---

This patch is based on sakari's media-tree git:
https://git.linuxtv.org/sailus/media_tree.git/log/?h=for-4.20-1

Changes from v5:
- add some comments for gain calculation
- use lock to protect the format
- fix some style issues

Quoted 7 lines

Changes from v4 to v5:
- use single PLL for all internal clocks
- change link frequency to 482.4MHz
- adjust frame timing for 2x2 binning modes
and enlarge frame readout time
- get CSI-2 link frequencies and external clock
from firmware

If I remember correctly, that was suggested by you. Why do we need to specify link frequency in firmware if it's fully configured by the driver, with the only external dependency being the external clock?

We're having problems with firmware listing the link frequency from v4 and we can't easily change it anymore to report the new one. I feel like this dependency on the firmware here is unnecessary, as long as the external clock frequency matches.

Best regards,
Tomasz

maintainer
#4 lore raw

Hi Tomasz,

Quoted 45 lines · Tomasz Figa

Hi Sakari,

Quoted 34 lines · On Wed, Sep 26, 2018 at 11:38 AM <[email protected]> wrote:

From: Bingbu Cao <[email protected]>

Add a v4l2 sub-device driver for the Sony imx319 image sensor. This is a camera sensor using the i2c bus for control and the csi-2 bus for data.

This driver supports following features:
- manual exposure and analog/digital gain control support
- vblank/hblank control support
- 4 test patterns control support
- vflip/hflip control support (will impact the output bayer order)
- support following resolutions:
- 3264x2448, 3280x2464 @ 30fps
- 1936x1096, 1920x1080 @ 60fps
- 1640x1232, 1640x922, 1296x736, 1280x720 @ 120fps
- support 4 bayer orders output (via change v/hflip)
- SRGGB10(default), SGRBG10, SGBRG10, SBGGR10

---

This patch is based on sakari's media-tree git:
https://git.linuxtv.org/sailus/media_tree.git/log/?h=for-4.20-1

Changes from v5:
- add some comments for gain calculation
- use lock to protect the format
- fix some style issues

Quoted 7 lines

Changes from v4 to v5:
- use single PLL for all internal clocks
- change link frequency to 482.4MHz
- adjust frame timing for 2x2 binning modes
and enlarge frame readout time
- get CSI-2 link frequencies and external clock
from firmware

Quoted 3 lines

If I remember correctly, that was suggested by you. Why do we need to specify link frequency in firmware if it's fully configured by the driver, with the only external dependency being the external clock?

The driver that's now in upstream supports, for now, a very limited set of configurations from what the sensor supports. These are more or less tailored to the particular system where it is being used right now (output image size, external clock frequency, frame rates, link frequencies etc.). If the same sensor is needed elsewhere (quite likely), the configuration needed elsewhere is very likely to be different from what you're using now.

The link frequency in particular is important as using a different link frequency (which could be fine elsewhere) could cause EMI issues, e.g. rendering your GPS receiver inoperable during the time the camera sensor is streaming images.

Should new configurations be added to this driver to support a different system, the link frequencies used by those configurations may be problematic to your system, and after a software update the driver could as well use those new frequencies. That's a big no-no.

Quoted 5 lines

We're having problems with firmware listing the link frequency from v4 and we can't easily change it anymore to report the new one. I feel like this dependency on the firmware here is unnecessary, as long as the external clock frequency matches.

This is information you really need to know.

A number of older drivers do not use the link frequency information from the firmware but that comes with a risk. Really, it's better to change the frequency now to something you can choose, rather than have it changed later on to something someone else chose for you.

Signature · 4 lines
Kind regards,

Sakari Ailus
[email protected]
#5 lore raw
Quoted 49 lines · Sakari Ailus

Hi Tomasz,

Quoted 45 lines · Tomasz Figa

Hi Sakari,

Quoted 41 lines · On Wed, Sep 26, 2018 at 11:38 AM <[email protected]> wrote:

From: Bingbu Cao <[email protected]>

Add a v4l2 sub-device driver for the Sony imx319 image sensor. This is a camera sensor using the i2c bus for control and the csi-2 bus for data.

This driver supports following features:
- manual exposure and analog/digital gain control support
- vblank/hblank control support
- 4 test patterns control support
- vflip/hflip control support (will impact the output bayer order)
- support following resolutions:
- 3264x2448, 3280x2464 @ 30fps
- 1936x1096, 1920x1080 @ 60fps
- 1640x1232, 1640x922, 1296x736, 1280x720 @ 120fps
- support 4 bayer orders output (via change v/hflip)
- SRGGB10(default), SGRBG10, SGBRG10, SBGGR10

---

This patch is based on sakari's media-tree git:
https://git.linuxtv.org/sailus/media_tree.git/log/?h=for-4.20-1

Changes from v5:
- add some comments for gain calculation
- use lock to protect the format
- fix some style issues

Changes from v4 to v5:
- use single PLL for all internal clocks
- change link frequency to 482.4MHz
- adjust frame timing for 2x2 binning modes
and enlarge frame readout time
- get CSI-2 link frequencies and external clock
from firmware

Quoted 8 lines
Quoted 3 lines

If I remember correctly, that was suggested by you. Why do we need to specify link frequency in firmware if it's fully configured by the driver, with the only external dependency being the external clock?

The driver that's now in upstream supports, for now, a very limited set of configurations from what the sensor supports. These are more or less tailored to the particular system where it is being used right now (output image size, external clock frequency, frame rates, link frequencies etc.).

As a side note, they're tailored to exactly the system I mentioned, with different link frequency hardcoded in the firmware, coming from earlier stage of development.

Quoted 8 lines

If the same sensor is needed elsewhere (quite likely), the configuration needed elsewhere is very likely to be different from what you're using now.

The link frequency in particular is important as using a different link frequency (which could be fine elsewhere) could cause EMI issues, e.g. rendering your GPS receiver inoperable during the time the camera sensor is streaming images.

Quoted 5 lines

Should new configurations be added to this driver to support a different system, the link frequencies used by those configurations may be problematic to your system, and after a software update the driver could as well use those new frequencies. That's a big no-no.

Okay, those are some valid points indeed, thanks for clarifying.

Quoted 6 lines
Quoted 5 lines

We're having problems with firmware listing the link frequency from v4 and we can't easily change it anymore to report the new one. I feel like this dependency on the firmware here is unnecessary, as long as the external clock frequency matches.

Quoted 6 lines

This is information you really need to know.

A number of older drivers do not use the link frequency information from the firmware but that comes with a risk. Really, it's better to change the frequency now to something you can choose, rather than have it changed later on to something someone else chose for you.

I guess it means that we have to carry a local downstream patch that bypasses this check, since we cannot easily change the firmware anymore.

An alternative would be to make the driver try to select a frequency that matches what's in the firmware, but issue a warning and fall back to a default one if a matching is not found. It might be actually better than nothing for some early testing on new systems, since it wouldn't require firmware changes.

Best regards,
Tomasz

maintainer
#6 lore raw

Hi Tomasz,

Quoted 88 lines · Tomasz Figa
Quoted 49 lines · Sakari Ailus

Hi Tomasz,

Quoted 45 lines · Tomasz Figa

Hi Sakari,

Quoted 41 lines · On Wed, Sep 26, 2018 at 11:38 AM <[email protected]> wrote:

From: Bingbu Cao <[email protected]>

Add a v4l2 sub-device driver for the Sony imx319 image sensor. This is a camera sensor using the i2c bus for control and the csi-2 bus for data.

This driver supports following features:
- manual exposure and analog/digital gain control support
- vblank/hblank control support
- 4 test patterns control support
- vflip/hflip control support (will impact the output bayer order)
- support following resolutions:
- 3264x2448, 3280x2464 @ 30fps
- 1936x1096, 1920x1080 @ 60fps
- 1640x1232, 1640x922, 1296x736, 1280x720 @ 120fps
- support 4 bayer orders output (via change v/hflip)
- SRGGB10(default), SGRBG10, SGBRG10, SBGGR10

---

This patch is based on sakari's media-tree git:
https://git.linuxtv.org/sailus/media_tree.git/log/?h=for-4.20-1

Changes from v5:
- add some comments for gain calculation
- use lock to protect the format
- fix some style issues

Changes from v4 to v5:
- use single PLL for all internal clocks
- change link frequency to 482.4MHz
- adjust frame timing for 2x2 binning modes
and enlarge frame readout time
- get CSI-2 link frequencies and external clock
from firmware

Quoted 8 lines
Quoted 3 lines

If I remember correctly, that was suggested by you. Why do we need to specify link frequency in firmware if it's fully configured by the driver, with the only external dependency being the external clock?

The driver that's now in upstream supports, for now, a very limited set of configurations from what the sensor supports. These are more or less tailored to the particular system where it is being used right now (output image size, external clock frequency, frame rates, link frequencies etc.).

As a side note, they're tailored to exactly the system I mentioned, with different link frequency hardcoded in the firmware, coming from earlier stage of development.

Quoted 8 lines

If the same sensor is needed elsewhere (quite likely), the configuration needed elsewhere is very likely to be different from what you're using now.

The link frequency in particular is important as using a different link frequency (which could be fine elsewhere) could cause EMI issues, e.g. rendering your GPS receiver inoperable during the time the camera sensor is streaming images.

Quoted 5 lines

Should new configurations be added to this driver to support a different system, the link frequencies used by those configurations may be problematic to your system, and after a software update the driver could as well use those new frequencies. That's a big no-no.

Okay, those are some valid points indeed, thanks for clarifying.

Quoted 8 lines
Quoted 5 lines

We're having problems with firmware listing the link frequency from v4 and we can't easily change it anymore to report the new one. I feel like this dependency on the firmware here is unnecessary, as long as the external clock frequency matches.

This is information you really need to know.

Quoted 8 lines
Quoted 4 lines

A number of older drivers do not use the link frequency information from the firmware but that comes with a risk. Really, it's better to change the frequency now to something you can choose, rather than have it changed later on to something someone else chose for you.

I guess it means that we have to carry a local downstream patch that bypasses this check, since we cannot easily change the firmware anymore.

Is there a possibility update the firmware, or carry an SSDT overlay as part of the software? The options are laid out in Documentation/acpi/ssdt-overlays.txt . Do remember to pay attention to the revision field --- also in future Coreboot updates.

Quoted 6 lines

An alternative would be to make the driver try to select a frequency that matches what's in the firmware, but issue a warning and fall back to a default one if a matching is not found. It might be actually better than nothing for some early testing on new systems, since it wouldn't require firmware changes.

You don't need firmware changes per se; see above. Allowing that will very, very easily lead this being unaddressed during developement and changing later on inadvertly.

Signature · 4 lines
Kind regards,

Sakari Ailus
[email protected]
#7 lore raw
Quoted 97 lines · Sakari Ailus

Hi Tomasz,

Quoted 93 lines · Tomasz Figa
Quoted 49 lines · Sakari Ailus

Hi Tomasz,

Quoted 45 lines · Tomasz Figa

Hi Sakari,

Quoted 41 lines · On Wed, Sep 26, 2018 at 11:38 AM <[email protected]> wrote:

From: Bingbu Cao <[email protected]>

Add a v4l2 sub-device driver for the Sony imx319 image sensor. This is a camera sensor using the i2c bus for control and the csi-2 bus for data.

This driver supports following features:
- manual exposure and analog/digital gain control support
- vblank/hblank control support
- 4 test patterns control support
- vflip/hflip control support (will impact the output bayer order)
- support following resolutions:
- 3264x2448, 3280x2464 @ 30fps
- 1936x1096, 1920x1080 @ 60fps
- 1640x1232, 1640x922, 1296x736, 1280x720 @ 120fps
- support 4 bayer orders output (via change v/hflip)
- SRGGB10(default), SGRBG10, SGBRG10, SBGGR10

---

This patch is based on sakari's media-tree git:
https://git.linuxtv.org/sailus/media_tree.git/log/?h=for-4.20-1

Changes from v5:
- add some comments for gain calculation
- use lock to protect the format
- fix some style issues

Changes from v4 to v5:
- use single PLL for all internal clocks
- change link frequency to 482.4MHz
- adjust frame timing for 2x2 binning modes
and enlarge frame readout time
- get CSI-2 link frequencies and external clock
from firmware

Quoted 8 lines
Quoted 3 lines

If I remember correctly, that was suggested by you. Why do we need to specify link frequency in firmware if it's fully configured by the driver, with the only external dependency being the external clock?

The driver that's now in upstream supports, for now, a very limited set of configurations from what the sensor supports. These are more or less tailored to the particular system where it is being used right now (output image size, external clock frequency, frame rates, link frequencies etc.).

As a side note, they're tailored to exactly the system I mentioned, with different link frequency hardcoded in the firmware, coming from earlier stage of development.

Quoted 8 lines

If the same sensor is needed elsewhere (quite likely), the configuration needed elsewhere is very likely to be different from what you're using now.

The link frequency in particular is important as using a different link frequency (which could be fine elsewhere) could cause EMI issues, e.g. rendering your GPS receiver inoperable during the time the camera sensor is streaming images.

Quoted 5 lines

Should new configurations be added to this driver to support a different system, the link frequencies used by those configurations may be problematic to your system, and after a software update the driver could as well use those new frequencies. That's a big no-no.

Okay, those are some valid points indeed, thanks for clarifying.

Quoted 12 lines
Quoted 5 lines

We're having problems with firmware listing the link frequency from v4 and we can't easily change it anymore to report the new one. I feel like this dependency on the firmware here is unnecessary, as long as the external clock frequency matches.

This is information you really need to know.

A number of older drivers do not use the link frequency information from the firmware but that comes with a risk. Really, it's better to change the frequency now to something you can choose, rather than have it changed later on to something someone else chose for you.

Quoted 9 lines
Quoted 3 lines

I guess it means that we have to carry a local downstream patch that bypasses this check, since we cannot easily change the firmware anymore.

Is there a possibility update the firmware, or carry an SSDT overlay as part of the software? The options are laid out in Documentation/acpi/ssdt-overlays.txt . Do remember to pay attention to the revision field --- also in future Coreboot updates.

Generally we try to avoid updating the firmware in the field, but most of the time there is a reason to do it anyway, so that might eventually happen. Let me try to figure out.

Quoted 10 lines
Quoted 6 lines

An alternative would be to make the driver try to select a frequency that matches what's in the firmware, but issue a warning and fall back to a default one if a matching is not found. It might be actually better than nothing for some early testing on new systems, since it wouldn't require firmware changes.

You don't need firmware changes per se; see above. Allowing that will very, very easily lead this being unaddressed during developement and changing later on inadvertly.

Still, requiring the user to create an SSDT overlay sounds like an overkill, for something that is not really a fatal error. We handle ACPI firmware bugs in many parts of the kernel by issuing a warning and using some reasonably safe fallback and I don't know why we couldn't do the same here.

Best regards,
Tomasz

maintainer
#8 lore raw

Hi Tomasz,

Quoted 124 lines · Tomasz Figa
Quoted 97 lines · Sakari Ailus

Hi Tomasz,

Quoted 93 lines · Tomasz Figa
Quoted 49 lines · Sakari Ailus

Hi Tomasz,

Quoted 45 lines · Tomasz Figa

Hi Sakari,

Quoted 41 lines · On Wed, Sep 26, 2018 at 11:38 AM <[email protected]> wrote:

From: Bingbu Cao <[email protected]>

Add a v4l2 sub-device driver for the Sony imx319 image sensor. This is a camera sensor using the i2c bus for control and the csi-2 bus for data.

This driver supports following features:
- manual exposure and analog/digital gain control support
- vblank/hblank control support
- 4 test patterns control support
- vflip/hflip control support (will impact the output bayer order)
- support following resolutions:
- 3264x2448, 3280x2464 @ 30fps
- 1936x1096, 1920x1080 @ 60fps
- 1640x1232, 1640x922, 1296x736, 1280x720 @ 120fps
- support 4 bayer orders output (via change v/hflip)
- SRGGB10(default), SGRBG10, SGBRG10, SBGGR10

---

This patch is based on sakari's media-tree git:
https://git.linuxtv.org/sailus/media_tree.git/log/?h=for-4.20-1

Changes from v5:
- add some comments for gain calculation
- use lock to protect the format
- fix some style issues

Changes from v4 to v5:
- use single PLL for all internal clocks
- change link frequency to 482.4MHz
- adjust frame timing for 2x2 binning modes
and enlarge frame readout time
- get CSI-2 link frequencies and external clock
from firmware

Quoted 8 lines
Quoted 3 lines

If I remember correctly, that was suggested by you. Why do we need to specify link frequency in firmware if it's fully configured by the driver, with the only external dependency being the external clock?

The driver that's now in upstream supports, for now, a very limited set of configurations from what the sensor supports. These are more or less tailored to the particular system where it is being used right now (output image size, external clock frequency, frame rates, link frequencies etc.).

As a side note, they're tailored to exactly the system I mentioned, with different link frequency hardcoded in the firmware, coming from earlier stage of development.

Quoted 8 lines

If the same sensor is needed elsewhere (quite likely), the configuration needed elsewhere is very likely to be different from what you're using now.

The link frequency in particular is important as using a different link frequency (which could be fine elsewhere) could cause EMI issues, e.g. rendering your GPS receiver inoperable during the time the camera sensor is streaming images.

Quoted 5 lines

Should new configurations be added to this driver to support a different system, the link frequencies used by those configurations may be problematic to your system, and after a software update the driver could as well use those new frequencies. That's a big no-no.

Okay, those are some valid points indeed, thanks for clarifying.

Quoted 12 lines
Quoted 5 lines

We're having problems with firmware listing the link frequency from v4 and we can't easily change it anymore to report the new one. I feel like this dependency on the firmware here is unnecessary, as long as the external clock frequency matches.

This is information you really need to know.

A number of older drivers do not use the link frequency information from the firmware but that comes with a risk. Really, it's better to change the frequency now to something you can choose, rather than have it changed later on to something someone else chose for you.

Quoted 9 lines
Quoted 3 lines

I guess it means that we have to carry a local downstream patch that bypasses this check, since we cannot easily change the firmware anymore.

Is there a possibility update the firmware, or carry an SSDT overlay as part of the software? The options are laid out in Documentation/acpi/ssdt-overlays.txt . Do remember to pay attention to the revision field --- also in future Coreboot updates.

Generally we try to avoid updating the firmware in the field, but most of the time there is a reason to do it anyway, so that might eventually happen. Let me try to figure out.

Quoted 10 lines
Quoted 6 lines

An alternative would be to make the driver try to select a frequency that matches what's in the firmware, but issue a warning and fall back to a default one if a matching is not found. It might be actually better than nothing for some early testing on new systems, since it wouldn't require firmware changes.

You don't need firmware changes per se; see above. Allowing that will very, very easily lead this being unaddressed during developement and changing later on inadvertly.

Quoted 5 lines

Still, requiring the user to create an SSDT overlay sounds like an overkill, for something that is not really a fatal error. We handle ACPI firmware bugs in many parts of the kernel by issuing a warning and using some reasonably safe fallback and I don't know why we couldn't do the same here.

To address this in kernel only, I presume one would have to plant that snippet in the ACPI table based on recognising the machine name and the firmware version. I don't think this is supported right now.

OTOH, an alternative option could be, if you use initrd for other reasons, to bake the SSDT overlay into the initrd based on the machine identification when creating the initrd image. That would require no kernel changes and compared to making kernel changes, this way also there would be no adverse effect on the size of the kernel.

Signature · 4 lines
Regards,

Sakari Ailus
[email protected]
#9 lore raw
Quoted 134 lines · Sakari Ailus

Hi Tomasz,

Quoted 124 lines · Tomasz Figa
Quoted 97 lines · Sakari Ailus

Hi Tomasz,

Quoted 93 lines · Tomasz Figa
Quoted 49 lines · Sakari Ailus

Hi Tomasz,

Quoted 45 lines · Tomasz Figa

Hi Sakari,

Quoted 41 lines · On Wed, Sep 26, 2018 at 11:38 AM <[email protected]> wrote:

From: Bingbu Cao <[email protected]>

Add a v4l2 sub-device driver for the Sony imx319 image sensor. This is a camera sensor using the i2c bus for control and the csi-2 bus for data.

This driver supports following features:
- manual exposure and analog/digital gain control support
- vblank/hblank control support
- 4 test patterns control support
- vflip/hflip control support (will impact the output bayer order)
- support following resolutions:
- 3264x2448, 3280x2464 @ 30fps
- 1936x1096, 1920x1080 @ 60fps
- 1640x1232, 1640x922, 1296x736, 1280x720 @ 120fps
- support 4 bayer orders output (via change v/hflip)
- SRGGB10(default), SGRBG10, SGBRG10, SBGGR10

---

This patch is based on sakari's media-tree git:
https://git.linuxtv.org/sailus/media_tree.git/log/?h=for-4.20-1

Changes from v5:
- add some comments for gain calculation
- use lock to protect the format
- fix some style issues

Changes from v4 to v5:
- use single PLL for all internal clocks
- change link frequency to 482.4MHz
- adjust frame timing for 2x2 binning modes
and enlarge frame readout time
- get CSI-2 link frequencies and external clock
from firmware

Quoted 8 lines
Quoted 3 lines

If I remember correctly, that was suggested by you. Why do we need to specify link frequency in firmware if it's fully configured by the driver, with the only external dependency being the external clock?

The driver that's now in upstream supports, for now, a very limited set of configurations from what the sensor supports. These are more or less tailored to the particular system where it is being used right now (output image size, external clock frequency, frame rates, link frequencies etc.).

As a side note, they're tailored to exactly the system I mentioned, with different link frequency hardcoded in the firmware, coming from earlier stage of development.

Quoted 8 lines

If the same sensor is needed elsewhere (quite likely), the configuration needed elsewhere is very likely to be different from what you're using now.

The link frequency in particular is important as using a different link frequency (which could be fine elsewhere) could cause EMI issues, e.g. rendering your GPS receiver inoperable during the time the camera sensor is streaming images.

Quoted 5 lines

Should new configurations be added to this driver to support a different system, the link frequencies used by those configurations may be problematic to your system, and after a software update the driver could as well use those new frequencies. That's a big no-no.

Okay, those are some valid points indeed, thanks for clarifying.

Quoted 12 lines
Quoted 5 lines

We're having problems with firmware listing the link frequency from v4 and we can't easily change it anymore to report the new one. I feel like this dependency on the firmware here is unnecessary, as long as the external clock frequency matches.

This is information you really need to know.

A number of older drivers do not use the link frequency information from the firmware but that comes with a risk. Really, it's better to change the frequency now to something you can choose, rather than have it changed later on to something someone else chose for you.

Quoted 9 lines
Quoted 3 lines

I guess it means that we have to carry a local downstream patch that bypasses this check, since we cannot easily change the firmware anymore.

Is there a possibility update the firmware, or carry an SSDT overlay as part of the software? The options are laid out in Documentation/acpi/ssdt-overlays.txt . Do remember to pay attention to the revision field --- also in future Coreboot updates.

Generally we try to avoid updating the firmware in the field, but most of the time there is a reason to do it anyway, so that might eventually happen. Let me try to figure out.

Quoted 10 lines
Quoted 6 lines

An alternative would be to make the driver try to select a frequency that matches what's in the firmware, but issue a warning and fall back to a default one if a matching is not found. It might be actually better than nothing for some early testing on new systems, since it wouldn't require firmware changes.

You don't need firmware changes per se; see above. Allowing that will very, very easily lead this being unaddressed during developement and changing later on inadvertly.

Quoted 5 lines

Still, requiring the user to create an SSDT overlay sounds like an overkill, for something that is not really a fatal error. We handle ACPI firmware bugs in many parts of the kernel by issuing a warning and using some reasonably safe fallback and I don't know why we couldn't do the same here.

Quoted 3 lines

To address this in kernel only, I presume one would have to plant that snippet in the ACPI table based on recognising the machine name and the firmware version. I don't think this is supported right now.

My point is that we normally handle such mismatches in the kernel by just printing a warning and forcing whatever the kernel needs for the device to work. For the cases where such override is not sensible by default (due to potential side-effects or so), a command line argument is often provided to force ignoring the firmware data.

Quoted 6 lines

OTOH, an alternative option could be, if you use initrd for other reasons, to bake the SSDT overlay into the initrd based on the machine identification when creating the initrd image. That would require no kernel changes and compared to making kernel changes, this way also there would be no adverse effect on the size of the kernel.

We don't use initrd.

Generally it looks like the simplest way forward is to carry a downstream patch turning the driver error into a warning. It's probably the simplest thing to do, if we can't have it handled upstream.

Best regards,
Tomasz