Soc-Camera architecture and nomenclature, and I2C in V4L

Hello all,

I am developing a synchronous camera array based on i.MX31 architecture and using Guennadi's current patches, as posted on this list (thanks!). Hardware is based on on a phyCORE-i.MX31, to whose camera bus (CMOS Sensor Interface) we connect logic which interleaves data from several cameras.

First of all I want to thank you guys and specially Guennadi because this V4L work is helping me getting my project a good head-start. Hopefully I will contribute something back, for now I am trying to understand the existing system.

I have a couple comments to date, this is the first:

Comment on nomenclature: "camera host [bus]" and "xxx_camera" drivers.

Regarding Soc-Camera Architecture as described in Documentation/video4linux/soc-camera.txt, the following two 'terms' are defined:

a.- "Camera host - an interface to which a camera is connected." and:
b.- "Camera host bus - a connection between a camera host and a camera".

What's the idea behind defining term 'b'? I guess it has something to do with an scenario where several cameras are attached to the same host interface, but in that case the interface itself, that is term 'a', must be some kind of bus, thus the name 'bus' in term 'b' is misleading...

Later on in the document, the sentence "[...] specify to which camera host bus the sensor is connected" mixes the terms definitely, while it is the sole appearance the 'bus'.

I would suggest removing the term definition for "camera host bus", since it does not help the reader, at least myself!

Then, while examining drivers for this subsystem, I find several called "pxa_camera", "mx3_camera", and so on. What I have understood after more detailed examination is that they are actually the drivers for the host bus in different SoC architectures. What I would suggest, if it is not too late, is that their names reflect that, instead of suggesting they are camera drivers.

Please don't take me wrong, this SoC-Camera thing is a great job! V4L seems awfully complex and it must have been hard to integrate the SoC-Camera subsystem. I am just commenting from the point of view of a newcomer who has the privilege of analyzing the architecture at a stage where things are pretty developed.

And I have a question about the use of "platform device" kernel concept within SoC-camera drivers. As far as I understand, it is just used to be able to share some drivers among different 'platforms'. Is that all?

Because I am finding it easier to start off writing a (simple and) specific driver which later I could "platform-ize", if I ever want to enable "the same code in several platforms".

Finally, while caring of the I2C stuff in my design I have found some I2C 'magic' within V4L API, but I am not quite sure yet. I personally prefer to keep them apart V4L and I2C, let my camera driver care of the way it communicates with its hardware... How is that a sin against V4L bible? :-)

Again, thank you guys for the great job in V4L!

Hi Agustin,

Quoted 8 lines · Agustin Ferrin Pozuelo

Hello all,

I am developing a synchronous camera array based on i.MX31 architecture and using Guennadi's current patches, as posted on this list (thanks!).
Hardware is based on on a phyCORE-i.MX31, to whose camera bus (CMOS Sensor Interface) we connect logic which interleaves data from several cameras.

Would be interesting to know what cameras you're going to connect and in which mode - Bayer / RGB / YUV, how many bits?

(a) refers to the controller itself, to its registers if you will. (b) refers to the signals connecting sensors to the controller - syncs, data, clocks...

Quoted 3 lines

Later on in the document, the sentence "[...] specify to which camera host bus the sensor is connected" mixes the terms definitely, while it is the sole appearance the 'bus'.

No it doesn't. There can be several sensors on a bus, but there can also be several busses on a system, at least theoretically. That's what is meant there. And those multiple busses are numbered and camera platform data specify to which of the busses it is connected.

Quoted 1 line

I would suggest removing the term [...]

I hope the above explanation helped to clarify the terms.

Quoted 2 lines

Then, while examining drivers for this subsystem, I find several called "pxa_camera", "mx3_camera", and so on. [...]

Well, for pxa_camera and sh_mobile_ceu_camera it is already too late. And mx3_camera just follows that convention. And personally I don't find it much worse than any other, certainly not bad enough to rename them :-)

Quoted 5 lines

[...]
And I have a question about the use of "platform device" kernel concept within SoC-camera drivers. As far as I understand, it is just used to be able to share some drivers among different 'platforms'. Is that all?

It is there to be able to configure a driver for a specific platform. For example, on one PXA270 board a sensor can be connected to the SoC using all 10 data lines, on another board only 8 lines can be used. That's the sort of things that you specify in platform data.

Quoted 2 lines

Because I am finding it easier to start off writing a (simple and) specific driver [...]

Well, it is both I guess. [...] it is best to do everything right from the first time, of course :-)

Quoted 5 lines

Finally, while caring of the I2C stuff in my design I have found some I2C 'magic' within V4L API, but I am not quite sure yet. I personally prefer to keep them apart V4L and I2C, let my camera driver care of the way it communicates with its hardware... How is that a sin against V4L bible? :-)

What exactly do you mean here?

Quoted 1 line

Again, thank you guys for the great job in V4L!

Thanks for using it. As usual, patches are welcome. Especially patches to the documentation :-) It will have to be updated now as we extended the API a bit, but improvements to the style or the contents are gladly accepted as well:-)

Absolutely. Right now I am connecting 6 MT9P031, the monochrome type, with 12 bits ADC so I think the right format would be 'Y16'.

Later on I plan to use the complete bus width in the i.MX31 (16 or 15 bit?) as 'raw' data, in order to pack pixels and get higher data rate at the same clock.

For multiplexing I2C I have a 4-channel I2C switch 'PCA9546A'. This is required as the MT9P031 can only choose among 2 I2C address (an improvement over previous models, insufficient for me though.

Quoted 3 lines

(a) refers to the controller itself, to its registers if you will. (b) refers to the signals connecting sensors to the controller - syncs, data, clocks...

I see the use and meaning of (a), regarding (b) I have the feeling that it is something internal to SoC-Camera implementation, as I have not recognized it during the (unfinished) development of my SoC-Camera driver.

Quoted 7 lines

Quoted 5 lines

Finally, while caring of the I2C stuff in my design I have found some I2C 'magic' within V4L API, but I am not quite sure yet. I personally prefer to keep them apart V4L and I2C, let my camera driver care of the way it communicates with its hardware... How is that a sin against V4L bible? :-)

What exactly do you mean here?

I mean things like this I see in mt9m001 driver:

static int mt9m001_get_chip_id(struct soc_camera_device *icd,
struct v4l2_chip_ident *id)

{
	if (id->match_type != V4L2_CHIP_MATCH_I2C_ADDR)
		return -EINVAL;

[...]
}

Where V4L seems to be caring about I2C somehow behind the scenes. And in my case I need full control over what's going on in I2C, as I have something there in the middle (an I2C switch!).

I know there is a better way to integrate the I2C switch, I just don't have time now to get to implement a transparent I2C switch driver, since I don't even know where to start from!

Quoted 2 lines · Agustin

Absolutely. Right now I am connecting 6 MT9P031, the monochrome type, with 12 bits ADC so I think the right format would be 'Y16'.

Hm, actually I think it is not. As you might have seen in the current soc-camera sources, we now handle two formats:
one from the sensor to the host controller driver, and one from the host driver to the user. i.MX31 can handle 15, 10, 8 and 4 bits, so, you will either have to extend your 12 bits to 15, or truncate them to 10 or 8. Respectively you will choose a suitable format. But - it will have to be "15-bit monochrome", "10-bit monochrome", or "8-bit monochrome."
Currently I only see 8 and 16 bits defined in v4l, so, if you use anything different you will have to define it. Yes, I know I used V4L2_PIX_FMT_Y16 in mt9m001 for "monochrome 10 bit" - this is wrong, as well as using V4L2_PIX_FMT_SBGGR16 for "Bayer 10 bit,"
I will have to fix this some time.

Quoted 3 lines

Later on I plan to use the complete bus width in the i.MX31 (16 or 15 bit?) as 'raw' data, in order to pack pixels and get higher data rate at the same clock.

Maximum 15, but you already use raw data for monochrome, i.MX31 doesn't have any special support for it.

(Changing subject)

Quoted 4 lines · Guennadi Liakhovetski

Quoted 2 lines · Agustin

Absolutely. Right now I am connecting 6 MT9P031, the monochrome type, with 12 bits ADC so I think the right format would be 'Y16'.

Quoted 15 lines

Hm, actually I think it is not. As you might have seen in the current soc-camera sources, we now handle two formats:
one from the sensor to the host controller driver, and one from the host driver to the user. i.MX31 can handle 15, 10, 8 and 4 bits, so, you will either have to extend your 12 bits to 15, or truncate them to 10 or 8. Respectively you will choose a suitable format. But - it will have to be "15-bit monochrome", "10-bit monochrome", or "8-bit monochrome."
Currently I only see 8 and 16 bits defined in v4l, so, if you use anything different you will have to define it. Yes, I know I used V4L2_PIX_FMT_Y16 in mt9m001 for "monochrome 10 bit" - this is wrong, as well as using V4L2_PIX_FMT_SBGGR16 for "Bayer 10 bit,"
I will have to fix this some time.

According to V4L2 API (http://v4l2spec.bytesex.org/spec/r4246.htm),
V4L2_PIX_FMT_Y16 ('Y16 ') describes "a grey-scale image with a depth of 16 bits per pixel". And it is also stated that "the actual sampling precision may be lower than 16 bits, for example 10 bits per pixel".

So I think V4L2_PIX_FMT_Y16 is valid for all the monochrome widths in i.MX31, isn't it?

Regarding CSI bus width, I understand it is 'negotiated' through soc_camera_ops.query/set_bus_params(). I still don't know who makes the choice here, so I am just announcing 10 bit cause is what I need at the moment.

(Changing subject)

On Fri, 12/12/08, Guennadi Liakhovetski <[email protected]>

Quoted 3 lines · Agustin

Quoted 2 lines

Absolutely. Right now I am connecting 6 MT9P031, the monochrome type, with 12 bits ADC so I think the right format would be

Quoted 9 lines

Hm, actually I think it is not. As you might have seen in the current soc-camera sources, we now handle two formats:
one from the sensor to the host controller driver, and one from the host driver to the user. i.MX31 can handle 15, 10, 8 and 4 bits, so, you will either have to extend your 12 bits to 15, or truncate them to 10 or 8. Respectively you will choose a suitable format. But - it will have to be "15-bit monochrome", "10-bit monochrome", or

Quoted 7 lines

monochrome."
Currently I only see 8 and 16 bits defined in v4l, so, if you use anything different you will have to define it. Yes, I know I used V4L2_PIX_FMT_Y16 in mt9m001 for "monochrome 10 bit" - this is wrong, as well as using V4L2_PIX_FMT_SBGGR16 for "Bayer 10 bit,"
I will have to fix this some time.

According to V4L2 API (http://v4l2spec.bytesex.org/spec/r4246.htm),
V4L2_PIX_FMT_Y16 ('Y16 ') describes "a grey-scale image with

16 bits per pixel". And it is also stated that "the actual

precision may be lower than 16 bits, for example 10 bits per pixel".

So I think V4L2_PIX_FMT_Y16 is valid for all the monochrome widths in i.MX31, isn't it?

Regarding CSI bus width, I understand it is 'negotiated' through soc_camera_ops.query/set_bus_params(). I still don't know who makes

choice here, so I am just announcing 10 bit cause is what I need at the moment.

Hi Guennadi,

According to manual, i.MX31's CSI can handle up to 16 bits wide, not just 15. I have been checking my hardware limitations and IOMUX options and 16 bits seems possible.

However, that's far from my current working status. I am using such a big frame format (~7700x970x10bits) that it requires a 16MB buffer for a single frame. I did the required mods to kernel to allow a big enough coherent DMA to be reserved, and I am trying to work with one single buffer, though I guess I will use two, sooner or later.

I have come as far to having the image as to successfully requesting a video capture from userspace through mmap, then get some message from mx3_camera that some IRQ is reserved, then select() expires (10 seconds) and the application stays "zombie" on exit.

Today I am troubleshooting this. I will see if the frame size or bus 'params' are wrong, and what is the IPU/CSI seeing in the bus, as my logic analyzer sees a pretty usable thing. Any tips?

Regards,
--

Agustin Ferrin Pozuelo

Embedded Systems Consultant

http://embedded.ferrin.org/

Tel. +34 610502587

--- El vie, 12/12/08, Guennadi Liakhovetski <[email protected]> escribió:
De: Guennadi Liakhovetski <[email protected]>
Asunto: Re: SoC-Camera bus width, and "V4LV4L2_PIX_FMT_Y16" for "monochrome 10 bit"
Para: "Agustin" <[email protected]>
CC: "video4linux list" <[email protected]>
Fecha: viernes, 12 diciembre, 2008 5:52

Quoted 3 lines · Agustin

(Changing subject)

On Fri, 12/12/08, Guennadi Liakhovetski <[email protected]>

Quoted 3 lines · wrote:

Quoted 3 lines · Agustin

Quoted 2 lines

Absolutely. Right now I am connecting 6 MT9P031, the monochrome type, with 12 bits ADC so I think the right format would be

'Y16'.

Quoted 9 lines

Quoted 9 lines

Hm, actually I think it is not. As you might have seen in the current soc-camera sources, we now handle two formats:
one from the sensor to the host controller driver, and one from the host driver to the user. i.MX31 can handle 15, 10, 8 and 4 bits, so, you will either have to extend your 12 bits to 15, or truncate them to 10 or 8. Respectively you will choose a suitable format. But - it will have to be "15-bit monochrome", "10-bit monochrome", or

"8-bit

Quoted 10 lines

Quoted 7 lines

monochrome."
Currently I only see 8 and 16 bits defined in v4l, so, if you use anything different you will have to define it. Yes, I know I used V4L2_PIX_FMT_Y16 in mt9m001 for "monochrome 10 bit" - this is wrong, as well as using V4L2_PIX_FMT_SBGGR16 for "Bayer 10 bit,"
I will have to fix this some time.

According to V4L2 API (http://v4l2spec.bytesex.org/spec/r4246.htm),
V4L2_PIX_FMT_Y16 ('Y16 ') describes "a grey-scale image with

a depth of

Quoted 1 line

16 bits per pixel". And it is also stated that "the actual

sampling

Quoted 4 lines

precision may be lower than 16 bits, for example 10 bits per pixel".

So I think V4L2_PIX_FMT_Y16 is valid for all the monochrome widths in i.MX31, isn't it?

Hm, ok, in case of i.MX31 it is the case - if you connect a sensor over 15 bit to it, you get 16 bit (see 44.1.1.3) - 15 _most_ significant bits of a 16-bit word will be used! This is not the case with pxa270 - it uses _least_ significant bits, so, if you get a 10-bit monochrome format. Now with your 12 bits - to get 16 bits out of them you will have to connect them to high 12 bits of imx31: D11 of a sensor to D14 of imx31 ... D0 of a sensor to D3 of imx31, then you get 16 bits too. I think, if you connect 10 bits and configure imx31 for 10 bits, you get 16 bits too, but I haven't found anything in the datasheet about it.

So, if you connect your sensors correctly _from_ imx31 to the user you get 8 or 16 bits. But, to describe the format the sensor sends to the imx31 you have to specify 15 bits, so that your sensor works correctly also when connected to other hosts, e.g., pxa.

Quoted 2 lines

Regarding CSI bus width, I understand it is 'negotiated' through soc_camera_ops.query/set_bus_params(). I still don't know who makes

the

Quoted 2 lines

choice here, so I am just announcing 10 bit cause is what I need at the moment.

The host decides. You will see soon enough whether what you announce is right:-) Even more so after updates that I hope to push out early next week, where flags comparison is made even more strict.

Thanks
Guennadi
---
Guennadi Liakhovetski