Skip to contentSkip to conversation

How to pass camera Orientation to userspace

RFC New Adam Baker 8 participants 38 messages Feb 18, 2009 – Mar 1, 2009

Subsystems (2)
lore
Share & tools
lore

Adam Baker

author
#0 lore raw

(linux-omap included in distribution as lots of omap systems include cameras so this could be relevant there.)

Background

A number of the webcams now supported by v4l have sensors that are mounted upside down. Up to now this has been handled by having a table in libv4l of the USB IDs of affected cameras. This approach however fails to address two known cases (and probably more as yet unknown ones) where the USB ID is insufficient to determine the sensor orientation.

In one of those cases (SQ-905) USB commands must be issued to the camera at probe time) to determine what sensor is fitted and in the other case (Genesys gl860) the camera can be pointed towards or away from the user and it swaps orientation when it is changed.

It is possible that there are cameras that can use gravity sensors or similar to report how they are being held but such user driven orientation which may be intended for creative effect should probably be separated from this hardware related issue.

Most if not all of the cameras affected by this problem produce video formats that are not widely supported by applications. They are therefore all likely to be normally used in conjunction with libv4l to perform format conversion and so it makes sense to retain the actual flipping operation in libv4l. libv4l provides a capability via LD_PRELOAD to attach itself to unmodified binaries.

It is likely that whatever solution is chosen there will end up being cases that have to be handled in libv4l as external information (such as laptop model ID) is the only mechanism to distinguish otherwise identical cameras.

So far libv4l only supports the case of data needing a 180 degree rotation but there is known to exist hardware which requires a VFLIP and it is possible some hardware requires just HFLIP so all of those cases should be supported.

There have been a number of inconclusive discussions on this subject on the v4l / linux-media mailing lists over the last few months offering many options for how to pass this information across which I will list below and hopefully a preferred solution can be selected from them

1) Reuse the existing HFLIP and VFLIP controls, marking them as read-only Pros : No change needed to videodev2.h
Cons: It is confusing to have controls that have a subtly different meaning if they are read only. Existing apps that support those controls might get confused. Would require polling to support the case of a camera being turned toward / away from the user while streaming.

2) Introduce a new orientation control (possibly in a new CAMERA_PROPERTIES class)
Pros: libv4l can easily tell if the driver supports the control.
Cons: It is really a property, not a control so calling it a control is wrong. Controls add lots of overhead in terms of driver code. Would require polling to support the case of a camera being turned toward / away from the user while streaming.

3) Use an extra couple of bits in V4L2_BUF_FLAGS
Pros: Simple to implement. Can change per frame without needing polling.
Cons: Doesn't work for non libv4l apps that try to use the read() interface. Can't easily identify drivers that don't support it (does 0 mean not rotated or just not implemented). Can only be read when streaming (does that matter?)

4) Use some reserved bits from the v4l2_capability structure
Pros: Less overhead than controls.
Cons: Would require polling to support the case of a camera being turned toward / away from the user while streaming. Can't easily identify drivers that don't support it.

5) Use some reserved bits from the v4l2_input structure (or possibly the status word but that is normally only valid for current input)
Pros: Less overhead than controls. Could support multiple sensors in one camera if such a beast exists.
Cons: Would require polling to support the case of a camera being turned toward / away from the user while streaming. Can't easily identify drivers that don't support it.

The interest in detecting if a driver provides this informnation is to allow libv4l to know when it should use the driver provided information and when it should use its internal table (which needs to be retained for backward compatibility). With no detection capability the driver provided info should be ignored for USB IDs in the built in table.

Thoughts please

Adam Baker

#1 lore raw

Hello Adam,

I've been thinking exactly the same issue not usb but SoC based camera. I have no idea about how usb cameras work but I am quite curious about is it really possible to make proper orientation with only querying camera driver.
Because in case of SoC based camera device, many of camera ISPs are supporting VFLIP, HFLIP register on their own, and we can read current orientation by reading those registers.

But the problem is ISP's registers are set as not flipped at all but it physically mounted upside down, because the H/W vendor has packed the camera module upside down. (it sounds ridiculous but happens sometimes)
So in that case when we query orientation of camera, it returns not flipped vertically or horizontally at all but actually it turns out to be upside down. Actually we are setting camera device to be flipped for default in that case.

I guess in USB camera case that kind of situation is not happening, but for SoC camera it should be a tough job to find out exact way of determine the orientation of camera automatically.

Cheers,

Nate

Quoted 90 lines · Adam Baker

(linux-omap included in distribution as lots of omap systems include cameras so this could be relevant there.)

Background

A number of the webcams now supported by v4l have sensors that are mounted upside down. Up to now this has been handled by having a table in libv4l of the USB IDs of affected cameras. This approach however fails to address two known cases (and probably more as yet unknown ones) where the USB ID is insufficient to determine the sensor orientation.

In one of those cases (SQ-905) USB commands must be issued to the camera at probe time) to determine what sensor is fitted and in the other case (Genesys gl860) the camera can be pointed towards or away from the user and it swaps orientation when it is changed.

It is possible that there are cameras that can use gravity sensors or similar to report how they are being held but such user driven orientation which may be intended for creative effect should probably be separated from this hardware related issue.

Most if not all of the cameras affected by this problem produce video formats that are not widely supported by applications. They are therefore all likely to be normally used in conjunction with libv4l to perform format conversion and so it makes sense to retain the actual flipping operation in libv4l. libv4l provides a capability via LD_PRELOAD to attach itself to unmodified binaries.

It is likely that whatever solution is chosen there will end up being cases that have to be handled in libv4l as external information (such as laptop model ID) is the only mechanism to distinguish otherwise identical cameras.

So far libv4l only supports the case of data needing a 180 degree rotation but there is known to exist hardware which requires a VFLIP and it is possible some hardware requires just HFLIP so all of those cases should be supported.

There have been a number of inconclusive discussions on this subject on the v4l / linux-media mailing lists over the last few months offering many options for how to pass this information across which I will list below and hopefully a preferred solution can be selected from them

1) Reuse the existing HFLIP and VFLIP controls, marking them as read-only Pros : No change needed to videodev2.h
Cons: It is confusing to have controls that have a subtly different meaning if they are read only. Existing apps that support those controls might get confused. Would require polling to support the case of a camera being turned toward / away from the user while streaming.

2) Introduce a new orientation control (possibly in a new CAMERA_PROPERTIES class)
Pros: libv4l can easily tell if the driver supports the control.
Cons: It is really a property, not a control so calling it a control is wrong. Controls add lots of overhead in terms of driver code. Would require polling to support the case of a camera being turned toward / away from the user while streaming.

3) Use an extra couple of bits in V4L2_BUF_FLAGS
Pros: Simple to implement. Can change per frame without needing polling.
Cons: Doesn't work for non libv4l apps that try to use the read() interface. Can't easily identify drivers that don't support it (does 0 mean not rotated or just not implemented). Can only be read when streaming (does that matter?)

4) Use some reserved bits from the v4l2_capability structure
Pros: Less overhead than controls.
Cons: Would require polling to support the case of a camera being turned toward / away from the user while streaming. Can't easily identify drivers that don't support it.

5) Use some reserved bits from the v4l2_input structure (or possibly the status word but that is normally only valid for current input)
Pros: Less overhead than controls. Could support multiple sensors in one camera if such a beast exists.
Cons: Would require polling to support the case of a camera being turned toward / away from the user while streaming. Can't easily identify drivers that don't support it.

The interest in detecting if a driver provides this informnation is to allow libv4l to know when it should use the driver provided information and when it should use its internal table (which needs to be retained for backward compatibility). With no detection capability the driver provided info should be ignored for USB IDs in the built in table.

Thoughts please

Adam Baker
--
To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 9 lines
========================================================
DongSoo(Nathaniel), Kim
Engineer
Mobile S/W Platform Lab. S/W centre
Telecommunication R&D Centre
Samsung Electronics CO., LTD.
e-mail : [email protected]
          [email protected]
========================================================
#2 lore raw
Quoted 10 lines · DongSoo(Nathaniel) Kim

Hello Adam,

I've been thinking exactly the same issue not usb but SoC based camera. I have no idea about how usb cameras work but I am quite curious about is it really possible to make proper orientation with only querying camera driver.
Because in case of SoC based camera device, many of camera ISPs are supporting VFLIP, HFLIP register on their own, and we can read current orientation by reading those registers.

Quoted 4 lines

But the problem is ISP's registers are set as not flipped at all but it physically mounted upside down, because the H/W vendor has packed the camera module upside down. (it sounds ridiculous but happens sometimes)

That happens a lot with webcams too. Given that these SoC systems will come with some board specific config anyways, all that is needed is to pass some boardconfig in to the camera driver (through platform data for example) which tells the camera driver that on this board the sensor is mounted upside down.

Quoted 4 lines

So in that case when we query orientation of camera, it returns not flipped vertically or horizontally at all but actually it turns out to be upside down. Actually we are setting camera device to be flipped for default in that case.

Ack, but the right thing to do is not to set the vflip and hflip video4linux2 controls on by default, but to invert their meaning, so when the sensor is upside down, the hflip and vflip controls as seen by the application through the v4l2 API will report not flipping, but the hwcontrols will actually be set to flipping, and when an app enables flipping at the v4l2 API level it will actually gets disables at the HW level, this way the upside downness is 100% hidden from userspace. So your problem does not need any of the new API we are working on. The new API is for when the hardware cannot flip and we need to tell userspace to correct for this in software.

Regards,

Hans

#3 lore raw

Hello Hans.

I went too far.
Just forgot what we can do through porting job. You are right. That should be board specific item. Cheers,

Nate

2009. 02. 18, 오후 11:36, Hans de Goede 작성:

Quoted 28 lines
Quoted 4 lines · DongSoo(Nathaniel) Kim

Hello Adam,
I've been thinking exactly the same issue not usb but SoC based camera. I have no idea about how usb cameras work but I am quite curious about is it really possible to make proper orientation with only querying

Quoted 8 lines

camera driver.
Because in case of SoC based camera device, many of camera ISPs are supporting VFLIP, HFLIP register on their own, and we can read current orientation by reading those registers.
But the problem is ISP's registers are set as not flipped at all but it physically mounted upside down, because the H/W vendor has packed the camera module upside down. (it sounds ridiculous but happens sometimes)

That happens a lot with webcams too. Given that these SoC systems will come with some board specific config anyways, all that is needed is to pass some boardconfig in to the camera driver (through platform data for example) which tells the camera driver that on this board the sensor is mounted upside down.

Quoted 4 lines

So in that case when we query orientation of camera, it returns not flipped vertically or horizontally at all but actually it turns out to be upside down. Actually we are setting camera device to be flipped for default in that case.

Ack, but the right thing to do is not to set the vflip and hflip video4linux2 controls on by default, but to invert their meaning, so when the sensor is upside down, the hflip and vflip controls as seen by the application through the v4l2 API will report not flipping, but the hwcontrols will actually be set to flipping, and when an app enables flipping at the v4l2 API level it will actually gets disables at the HW level, this way the upside downness is 100% hidden from userspace. So your problem does not need any of the new API we are working on. The new API is for when the hardware cannot flip and we need to tell userspace to correct for this in software.

Regards,

Hans

#4 lore raw
In reply to Adam Baker

Hi Adam,

Sorry for the late reply, it's been very busy.

Quoted 16 lines · On Wednesday 18 February 2009 01:30:52 Adam Baker wrote:

(linux-omap included in distribution as lots of omap systems include cameras so this could be relevant there.)

Background

A number of the webcams now supported by v4l have sensors that are mounted upside down. Up to now this has been handled by having a table in libv4l of the USB IDs of affected cameras. This approach however fails to address two known cases (and probably more as yet unknown ones) where the USB ID is insufficient to determine the sensor orientation.

In one of those cases (SQ-905) USB commands must be issued to the camera at probe time) to determine what sensor is fitted and in the other case (Genesys gl860) the camera can be pointed towards or away from the user and it swaps orientation when it is changed.

Quoted 4 lines

It is possible that there are cameras that can use gravity sensors or similar to report how they are being held but such user driven orientation which may be intended for creative effect should probably be separated from this hardware related issue.

Yes, I strongly agree with this.

Quoted 12 lines

Most if not all of the cameras affected by this problem produce video formats that are not widely supported by applications. They are therefore all likely to be normally used in conjunction with libv4l to perform format conversion and so it makes sense to retain the actual flipping operation in libv4l. libv4l provides a capability via LD_PRELOAD to attach itself to unmodified binaries.

It is likely that whatever solution is chosen there will end up being cases that have to be handled in libv4l as external information (such as laptop model ID) is the only mechanism to distinguish otherwise identical cameras.

Quoted 4 lines

So far libv4l only supports the case of data needing a 180 degree rotation but there is known to exist hardware which requires a VFLIP and it is possible some hardware requires just HFLIP so all of those cases should be supported.

While libv4l will be the main user of this information, it should not matter for the chosen API whether it is libv4l or another application, possibly a custom application running on an embedded system.

Quoted 20 lines

There have been a number of inconclusive discussions on this subject on the v4l / linux-media mailing lists over the last few months offering many options for how to pass this information across which I will list below and hopefully a preferred solution can be selected from them

1) Reuse the existing HFLIP and VFLIP controls, marking them as read-only Pros : No change needed to videodev2.h
Cons: It is confusing to have controls that have a subtly different meaning if they are read only. Existing apps that support those controls might get confused. Would require polling to support the case of a camera being turned toward / away from the user while streaming.

2) Introduce a new orientation control (possibly in a new CAMERA_PROPERTIES class)
Pros: libv4l can easily tell if the driver supports the control.
Cons: It is really a property, not a control so calling it a control is wrong. Controls add lots of overhead in terms of driver code. Would require polling to support the case of a camera being turned toward / away from the user while streaming.

Quoted 6 lines

3) Use an extra couple of bits in V4L2_BUF_FLAGS
Pros: Simple to implement. Can change per frame without needing polling.
Cons: Doesn't work for non libv4l apps that try to use the read() interface. Can't easily identify drivers that don't support it (does 0 mean not rotated or just not implemented). Can only be read when streaming (does that matter?)

I think that matters, yes.

Quoted 10 lines

4) Use some reserved bits from the v4l2_capability structure
Pros: Less overhead than controls.
Cons: Would require polling to support the case of a camera being turned toward / away from the user while streaming. Can't easily identify drivers that don't support it.

5) Use some reserved bits from the v4l2_input structure (or possibly the status word but that is normally only valid for current input)
Pros: Less overhead than controls. Could support multiple sensors in one camera if such a beast exists.

What does exist is devices with a video input (e.g. composite) and a camera
input: each input will have different flags. Since these vflip/hflip properties do not change they can be enumerated in advance and you know what each input supports.

Quoted 2 lines

Cons: Would require polling to support the case of a camera being turned toward / away from the user while streaming.

Polling applies only to the bits that tell the orientation of the camera. See below for a discussion of this.

Quoted 1 line

Can't easily identify drivers that don't support it.

Not too difficult to add through the use of a capability bit. Either in v4l2_input or (perhaps) v4l2_capability.

Another Pro is that this approach will also work for v4l2_output in the case of, say, rotated LCD displays. Using camera orientation bits in v4l2_buffer while capturing will work, but using similar bits for output will fail since the data is going in the wrong direction.

Quoted 7 lines

The interest in detecting if a driver provides this informnation is to allow libv4l to know when it should use the driver provided information and when it should use its internal table (which needs to be retained for backward compatibility). With no detection capability the driver provided info should be ignored for USB IDs in the built in table.

Thoughts please

Is polling bad in this case? It is not something that needs immediate attention IMHO. The overhead for checking once every X seconds is quite low. Furthermore, it is only needed on devices that cannot do v/hflipping in hardware.

An alternative is to put some effort in a proper event interface. There is one implemented in include/linux/dvb/video.h and used by ivtv for video decoding. The idea is that the application registers events it wants to receive, and whenever such an event arrives the select() call will exit with a high-prio event (exception). The application then checks what happened.

The video.h implementation is pretty crappy, but we can make something more useful. Alternatively, I believe there are other event mechanisms as well in the kernel. I know some work was done on that, but I don't know what the end result was.

Note that this does not apply to sensor mount information. I think that v4l2_input is the perfect place for that. It is after all an input-specific property and static property, and it maps 1-on-1 to the output case as well, should we need it there as well.

Regards,

Hans

Signature · 1 line
Hans Verkuil - video4linux developer - sponsored by TANDBERG
#5 lore raw
Quoted 4 lines · Hans Verkuil

Hi Adam,

Sorry for the late reply, it's been very busy.

Same here :)

Quoted 16 lines · On Wednesday 18 February 2009 01:30:52 Adam Baker wrote:
Quoted 15 lines

(linux-omap included in distribution as lots of omap systems include cameras so this could be relevant there.)

Background

A number of the webcams now supported by v4l have sensors that are mounted upside down. Up to now this has been handled by having a table in libv4l of the USB IDs of affected cameras. This approach however fails to address two known cases (and probably more as yet unknown ones) where the USB ID is insufficient to determine the sensor orientation.

In one of those cases (SQ-905) USB commands must be issued to the camera at probe time) to determine what sensor is fitted and in the other case (Genesys gl860) the camera can be pointed towards or away from the user
and it swaps orientation when it is changed.

Quoted 7 lines
Quoted 4 lines

It is possible that there are cameras that can use gravity sensors or similar to report how they are being held but such user driven orientation which may be intended for creative effect should probably be separated from this hardware related issue.

Yes, I strongly agree with this.

+1

Quoted 9 lines
Quoted 6 lines

3) Use an extra couple of bits in V4L2_BUF_FLAGS
Pros: Simple to implement. Can change per frame without needing polling.
Cons: Doesn't work for non libv4l apps that try to use the read() interface. Can't easily identify drivers that don't support it (does 0 mean not rotated or just not implemented). Can only be read when streaming (does that matter?)

I think that matters, yes.

I too can see it being usefull to get the orientation when not streaming.

Quoted 11 lines
Quoted 10 lines

4) Use some reserved bits from the v4l2_capability structure
Pros: Less overhead than controls.
Cons: Would require polling to support the case of a camera being turned toward / away from the user while streaming. Can't easily identify drivers that don't support it.

5) Use some reserved bits from the v4l2_input structure (or possibly the status word but that is normally only valid for current input)
Pros: Less overhead than controls. Could support multiple sensors in one camera if such a beast exists.

Quoted 7 lines

What does exist is devices with a video input (e.g. composite) and a camera input: each input will have different flags. Since these vflip/hflip properties do not change they can be enumerated in advance and you know what each input supports.

Quoted 2 lines

Cons: Would require polling to support the case of a camera being turned toward / away from the user while streaming.

Polling applies only to the bits that tell the orientation of the camera. See below for a discussion of this.

Didn't we agree to separate orietation (as in sensor mounted upside down) and the user being able to rotate the camera (which i believe we called pivotting).

For the orientation case, which is the case with the sq905X, we do not need to poll, as for pivotting, here is my proposal:

Have both orientation and pivotting flags in the input flags. The pivotting flags will indicate the pivotting state at the moment of doing the ioctl (which may change later), so these indeed may need to be polled, unlike the orientation flags which will never change.

Quoted 4 lines
Quoted 1 line

Can't easily identify drivers that don't support it.

Not too difficult to add through the use of a capability bit. Either in v4l2_input or (perhaps) v4l2_capability.

+1

Quoted 8 lines

Another Pro is that this approach will also work for v4l2_output in the case of, say, rotated LCD displays. Using camera orientation bits in v4l2_buffer while capturing will work, but using similar bits for output will fail since the data is going in the wrong direction.

Quoted 6 lines

The interest in detecting if a driver provides this informnation is to allow libv4l to know when it should use the driver provided information and when it should use its internal table (which needs to be retained for backward compatibility). With no detection capability the driver provided info should be ignored for USB IDs in the built in table.

Quoted 4 lines
Quoted 1 line

Thoughts please

Is polling bad in this case? It is not something that needs immediate attention IMHO. The overhead for checking once every X seconds is quite low.

Agreed, but it still is something which should be avoided if possible, implementing polling also means adding a lot of IMHO unneeded code on the userspace side.

I would prefer to make the "realtime" pivotting state available to the app by adding flags containing the pivotting state at frame capture to the v4l2_buf flags.

But if people dislike this, libv4l can simple poll the input now and then.

Quoted 2 lines

Furthermore, it is only needed on devices that cannot do v/hflipping in hardware.

Erm, no, since we decided we want to differentiate between sensor orientation and camera pivotting, I think the driver should not automagically do v/hflipping in the pivot case, this should be left up to userspace. Maybe the user actually wants to have an upside down picture ?

Quoted 2 lines

An alternative is to put some effort in a proper event interface. There is one implemented in include/linux/dvb/video.h and used by ivtv for video decoding. The idea is that the application registers events it wants to receive, and whenever such an event arrives the select() call will exit with a high-prio event (exception). The application then checks what happened.

No not another event interface please. Once could argue that we should export the pivotting info through the generic linux input system, but not a v4l specific event interface please. Actually I think making the pivotting sensor available through the generic input system in addition to making it available through input flags would be nice to have. Just like we need to make all those take a picture now buttons on webcams available through the generic input system.

Regards,

Hans

#6 lore raw
Quoted 10 lines · On Sunday 22 February 2009 12:17:58 Hans de Goede wrote:
Quoted 5 lines · Hans Verkuil
Quoted 2 lines

Cons: Would require polling to support the case of a camera being turned toward / away from the user while streaming.

Polling applies only to the bits that tell the orientation of the camera. See below for a discussion of this.

Didn't we agree to separate orietation (as in sensor mounted upside down) and the user being able to rotate the camera (which i believe we called pivotting).

Yes, I wrote this more as a clarification since I thought that the original text didn't make this distinction clearly enough.

Quoted 7 lines

For the orientation case, which is the case with the sq905X, we do not need to poll, as for pivotting, here is my proposal:

Have both orientation and pivotting flags in the input flags. The pivotting flags will indicate the pivotting state at the moment of doing the ioctl (which may change later), so these indeed may need to be polled, unlike the orientation flags which will never change.

Ack.

I propose to take one of the reserved fields from v4l2_input and either turn it into two __u16 fields (capabilities and flags) or one __u32 field (flags). I'm not sure whether the alignment of two __u16 fields will be done correctly on 32 and 64 bit systems since v4l2_input cannot change in size. Something to test first.

Quoted 31 lines
Quoted 4 lines
Quoted 1 line

Can't easily identify drivers that don't support it.

Not too difficult to add through the use of a capability bit. Either in v4l2_input or (perhaps) v4l2_capability.

+1

Quoted 12 lines

Another Pro is that this approach will also work for v4l2_output in the case of, say, rotated LCD displays. Using camera orientation bits in v4l2_buffer while capturing will work, but using similar bits for output will fail since the data is going in the wrong direction.

Quoted 7 lines

The interest in detecting if a driver provides this informnation is to allow libv4l to know when it should use the driver provided information and when it should use its internal table (which needs to be retained for backward compatibility). With no detection capability the driver provided info should be ignored for USB IDs in the built in table.

Quoted 7 lines
Quoted 1 line

Thoughts please

Is polling bad in this case? It is not something that needs immediate attention IMHO. The overhead for checking once every X seconds is quite

low.

Agreed, but it still is something which should be avoided if possible, implementing polling also means adding a lot of IMHO unneeded code on the userspace side.

Quoted 6 lines

I would prefer to make the "realtime" pivotting state available to the app by adding flags containing the pivotting state at frame capture to the v4l2_buf flags.

But if people dislike this, libv4l can simple poll the input now and then.

I think this should be prototyped. Compare polling vs. putting the state in the v4l2_buf flags and see if that makes a lot of difference in the user experience compared to polling once a second. If it clearly improves things for the user, then I have no objections to adding bits to v4l2_buf.

Quoted 7 lines
Quoted 2 lines

Furthermore, it is only needed on devices that cannot do v/hflipping in hardware.

Erm, no, since we decided we want to differentiate between sensor orientation and camera pivotting, I think the driver should not automagically do v/hflipping in the pivot case, this should be left up to userspace. Maybe the user actually wants to have an upside down picture ?

Sorry, you're right. I'd forgotten that.

Quoted 7 lines
Quoted 6 lines

An alternative is to put some effort in a proper event interface. There is one implemented in include/linux/dvb/video.h and used by ivtv for video decoding. The idea is that the application registers events it wants to receive, and whenever such an event arrives the select() call will exit with a high-prio event (exception). The application then checks what happened.

Quoted 7 lines

No not another event interface please. Once could argue that we should export the pivotting info through the generic linux input system, but not a v4l specific event interface please. Actually I think making the pivotting sensor available through the generic input system in addition to making it available through input flags would be nice to have. Just like we need to make all those take a picture now buttons on webcams available through the generic input system.

Agreed. I'm not up to date when it comes to the state of event interfaces in linux, and this is a solution for the longer term anyway. I guess someone needs to pick this up and research it to see what works best.

Regards,

Hans

Signature · 1 line
Hans Verkuil - video4linux developer - sponsored by TANDBERG
#7 lore raw
Quoted 1 line · Hans Verkuil

On Sunday 22 February 2009 12:17:58 Hans de Goede wrote:

<snipped a large part we agree upon (hurray!) >

Quoted 4 lines
Quoted 4 lines

Agreed, but it still is something which should be avoided if possible, implementing polling also means adding a lot of IMHO unneeded code on the userspace side.

Quoted 9 lines
Quoted 6 lines

I would prefer to make the "realtime" pivotting state available to the app by adding flags containing the pivotting state at frame capture to the v4l2_buf flags.

But if people dislike this, libv4l can simple poll the input now and then.

I think this should be prototyped. Compare polling vs. putting the state in the v4l2_buf flags and see if that makes a lot of difference in the user experience compared to polling once a second. If it clearly improves things for the user, then I have no objections to adding bits to v4l2_buf.

This is were the difficulties start, doing this 1 per second requires making clock() calls and comparing timestamps (always tricky to get right). We could do this once every X frames, but framerates can vary wildly with webcams.

My biggest concern is not the user experience, but the fact that needing to poll uglyfies the userspace code. However thinking about this more, I guess we could just poll every frame. That makes the polling code simple and should give a good user experience.

<snip of more things we agree up on>

Regards,

Hans

#8 lore raw
In reply to Hans de Goede
Quoted 104 lines · Hans de Goede
Quoted 4 lines · Hans Verkuil

Hi Adam,

Sorry for the late reply, it's been very busy.

Same here :)

Quoted 16 lines · On Wednesday 18 February 2009 01:30:52 Adam Baker wrote:
Quoted 15 lines

(linux-omap included in distribution as lots of omap systems include cameras so this could be relevant there.)

Background

A number of the webcams now supported by v4l have sensors that are mounted upside down. Up to now this has been handled by having a table in libv4l of the USB IDs of affected cameras. This approach however fails to address two known cases (and probably more as yet unknown ones) where the USB ID is insufficient to determine the sensor orientation.

In one of those cases (SQ-905) USB commands must be issued to the camera at probe time) to determine what sensor is fitted and in the other case (Genesys gl860) the camera can be pointed towards or away from the user
and it swaps orientation when it is changed.

Quoted 7 lines
Quoted 4 lines

It is possible that there are cameras that can use gravity sensors or similar to report how they are being held but such user driven orientation which may be intended for creative effect should probably be separated from this hardware related issue.

Yes, I strongly agree with this.

+1

Quoted 9 lines
Quoted 6 lines

3) Use an extra couple of bits in V4L2_BUF_FLAGS
Pros: Simple to implement. Can change per frame without needing polling.
Cons: Doesn't work for non libv4l apps that try to use the read() interface. Can't easily identify drivers that don't support it (does 0 mean not rotated or just not implemented). Can only be read when streaming (does that matter?)

I think that matters, yes.

I too can see it being usefull to get the orientation when not streaming.

Quoted 11 lines
Quoted 10 lines

4) Use some reserved bits from the v4l2_capability structure
Pros: Less overhead than controls.
Cons: Would require polling to support the case of a camera being turned toward / away from the user while streaming. Can't easily identify drivers that don't support it.

5) Use some reserved bits from the v4l2_input structure (or possibly the status word but that is normally only valid for current input)
Pros: Less overhead than controls. Could support multiple sensors in one camera if such a beast exists.

Quoted 7 lines

What does exist is devices with a video input (e.g. composite) and a camera input: each input will have different flags. Since these vflip/hflip properties do not change they can be enumerated in advance and you know what each input supports.

Quoted 2 lines

Cons: Would require polling to support the case of a camera being turned toward / away from the user while streaming.

Polling applies only to the bits that tell the orientation of the camera. See below for a discussion of this.

Didn't we agree to separate orietation (as in sensor mounted upside down) and the user being able to rotate the camera (which i believe we called pivotting).

For the orientation case, which is the case with the sq905X, we do not need to poll, as for pivotting, here is my proposal:

Have both orientation and pivotting flags in the input flags. The pivotting flags will indicate the pivotting state at the moment of doing the ioctl (which may change later), so these indeed may need to be polled, unlike the orientation flags which will never change.

Quoted 4 lines
Quoted 1 line

Can't easily identify drivers that don't support it.

Not too difficult to add through the use of a capability bit. Either in v4l2_input or (perhaps) v4l2_capability.

+1

Quoted 8 lines

Another Pro is that this approach will also work for v4l2_output in the case of, say, rotated LCD displays. Using camera orientation bits in v4l2_buffer while capturing will work, but using similar bits for output will fail since the data is going in the wrong direction.

Quoted 6 lines

The interest in detecting if a driver provides this informnation is to allow libv4l to know when it should use the driver provided information and when it should use its internal table (which needs to be retained for backward compatibility). With no detection capability the driver provided info should be ignored for USB IDs in the built in table.

Quoted 4 lines
Quoted 1 line

Thoughts please

Is polling bad in this case? It is not something that needs immediate attention IMHO. The overhead for checking once every X seconds is quite low.

Agreed, but it still is something which should be avoided if possible, implementing polling also means adding a lot of IMHO unneeded code on the userspace side.

I would prefer to make the "realtime" pivotting state available to the app by adding flags containing the pivotting state at frame capture to the v4l2_buf flags.

But if people dislike this, libv4l can simple poll the input now and then.

Quoted 2 lines

Furthermore, it is only needed on devices that cannot do v/hflipping in hardware.

Erm, no, since we decided we want to differentiate between sensor orientation and camera pivotting, I think the driver should not automagically do v/hflipping in the pivot case, this should be left up to userspace. Maybe the user actually wants to have an upside down picture ?

Quoted 9 lines
Quoted 2 lines

An alternative is to put some effort in a proper event interface. There is one implemented in include/linux/dvb/video.h and used by ivtv for video decoding. The idea is that the application registers events it wants to receive, and whenever such an event arrives the select() call will exit with a high-prio event (exception). The application then checks what happened.

No not another event interface please. Once could argue that we should export the pivotting info through the generic linux input system, but not a v4l specific event interface please. Actually I think making the pivotting sensor available through the generic input system in addition to making it available through input flags would be nice to have. Just like we need to make all those take a picture now buttons on webcams available through the generic input system.

Regards,

Hans

Hans and Adam,

I am not sure how it fits into the above discussion, but perhaps it is relevant to point out that flags can be toggled. Here is what I mean:

Suppose that we have two flags 01 and 10 (i.e. 2), and 01 signifies VFLIP and 10 signifies HFLIP.

Then for an "ordinary" camera in ordinary position these are initialized as 00. If the "ordinary" camera is turned in some funny way (and it is possible to know that) then one or both of these flags gets turned off.

But if it is a "funny" camera like some of the SQ905s the initial values are 1 and 1, because the sensor is in fact mounted upside down. Now, suppose that there is some camera in the future which, just like this, has the sensor upside down, and suppose that said hypothetical camera also has the ability to "know" that it has been turned over so what was upside down is now right side up. Well, all that one has to do is to flip the two bits from whatever they were to have instead the opposite values!

Observe that this would take care of the orientation problem both for cameras which had the sensor mounted right in the first place, and for cameras which had the sensor mounted wrong in the first place. Just use the same two bits to describe the sensor orientation, and if there is any reason (based upon some ability to know that the camera orientation is now different) that the orientation should change, then just flip the bits as appropriate.

Then it would be the job of the support module to provide proper initial values only for these bits, and everything else could be done later on, in userspace.

Theodore Kilgore

#9 lore raw
Quoted 1 line · [email protected] wrote:

<snip>

Quoted 8 lines

Hans and Adam,

I am not sure how it fits into the above discussion, but perhaps it is relevant to point out that flags can be toggled. Here is what I mean:

Suppose that we have two flags 01 and 10 (i.e. 2), and 01 signifies VFLIP and 10 signifies HFLIP.

Then for an "ordinary" camera in ordinary position these are initialized as 00. If the "ordinary" camera is turned in some funny way (and it is possible to know that) then one or both of these flags gets turned off.

Quoted 7 lines

But if it is a "funny" camera like some of the SQ905s the initial values are 1 and 1, because the sensor is in fact mounted upside down. Now, suppose that there is some camera in the future which, just like this, has the sensor upside down, and suppose that said hypothetical camera also has the ability to "know" that it has been turned over so what was upside down is now right side up. Well, all that one has to do is to flip the two bits from whatever they were to have instead the opposite values!

Observe that this would take care of the orientation problem both for cameras which had the sensor mounted right in the first place, and for cameras which had the sensor mounted wrong in the first place. Just use the same two bits to describe the sensor orientation, and if there is any reason (based upon some ability to know that the camera orientation is now different) that the orientation should change, then just flip the bits as appropriate.

Then it would be the job of the support module to provide proper initial values only for these bits, and everything else could be done later on, in userspace.

Theodore Kilgore

Theodore,

We want to be able to differentiate between a cam which has its sensor mounted upside down, and a cam which can be pivotted and happens to be upside down at the moment, in case of any upside down mounted sensor, we will always want to compentsate, in case of a pivotting camera wether we compensate or not could be a user preference.

So in you example of an upside down mounted sensor in a pivotting encasing and the encasing is pivotted 180 degrees we would have the hflip and vflip bits set for sensor orientation and we would have the pivotted 180 degrees bit set. If the user has choosen to compensate for pivotting the default, we would do nothing. But it is important to be able to differentiate between the 2.

Regards,

Hans

#10 lore raw
Quoted 21 lines · Hans de Goede
Quoted 1 line · [email protected] wrote:

<snip>

Quoted 14 lines

Hans and Adam,

I am not sure how it fits into the above discussion, but perhaps it is relevant to point out that flags can be toggled. Here is what I mean:

Suppose that we have two flags 01 and 10 (i.e. 2), and 01 signifies VFLIP and 10 signifies HFLIP.

Then for an "ordinary" camera in ordinary position these are initialized as 00. If the "ordinary" camera is turned in some funny way (and it is possible to know that) then one or both of these flags gets turned off.

But if it is a "funny" camera like some of the SQ905s the initial values are 1 and 1, because the sensor is in fact mounted upside down. Now, suppose that there is some camera in the future which, just like this, has the sensor upside down, and suppose that said hypothetical camera also has the ability to "know" that it has been turned over so what was upside down is now right side up. Well, all that one has to do is to flip the two bits from whatever they were to have instead the opposite values!

Observe that this would take care of the orientation problem both for cameras which had the sensor mounted right in the first place, and for cameras which had the sensor mounted wrong in the first place. Just use the same two bits to describe the sensor orientation, and if there is any reason (based upon some ability to know that the camera orientation is now different) that the orientation should change, then just flip the bits as appropriate.

Then it would be the job of the support module to provide proper initial values only for these bits, and everything else could be done later on, in userspace.

Quoted 7 lines
Quoted 1 line

Theodore Kilgore

Theodore,

We want to be able to differentiate between a cam which has its sensor mounted upside down, and a cam which can be pivotted and happens to be upside down at the moment, in case of any upside down mounted sensor, we will always want to compentsate, in case of a pivotting camera wether we compensate or not could be a user preference.

So in you example of an upside down mounted sensor in a pivotting encasing and the encasing is pivotted 180 degrees we would have the hflip and vflip bits set for sensor orientation and we would have the pivotted 180 degrees bit set. If the user has choosen to compensate for pivotting the default, we would do nothing. But it is important to be able to differentiate between the 2.

Hans,

I am not sure if we are talking past each other, or what. But I was pointing out that the initial values of two bits can indicate the "default" orientation of the sensor, and this can be done permanently in the module, which transmits the initial setting of those two bits to anything up the line which is interested or curious to know those initial values. The information in those two bits will definitely tell whether the sensor is mounted upside down in the camera. For example, if it is mounted upside down, then they are both set in the module to "on" and exported therefrom. But if the sensor is mounted correctly, then both of them are set to "off" and similarly exported.

Now if any application for any reason (such as "knowing" that the camera is upside down or is pointing in the opposite direction, or into a mirror) wants to change the defaults, all it has to do is to toggle the bits.

But, hmmm. Perhaps there is the question about how the app "knows" that the camera is upside down or is pointed in another direction. If the camera has a gyroscope inside, for example, then it could be the camera which needs to tell to the app about the current orientation, or else the app would not have any way to know ... Is this the problem, then? For that kind of thing, one might need more than two bits in order to pass the needed information.

Incidentally, I obviously agree about the need to pass the initial sensor orientation. So anything I say about this should be taken in that context.

Theodore Kilgore

#11 lore raw
Quoted 33 lines · [email protected] wrote:
Quoted 21 lines · Hans de Goede
Quoted 1 line · [email protected] wrote:

<snip>

Quoted 14 lines

Hans and Adam,

I am not sure how it fits into the above discussion, but perhaps it is relevant to point out that flags can be toggled. Here is what I mean:

Suppose that we have two flags 01 and 10 (i.e. 2), and 01 signifies VFLIP and 10 signifies HFLIP.

Then for an "ordinary" camera in ordinary position these are initialized as 00. If the "ordinary" camera is turned in some funny way (and it is possible to know that) then one or both of these flags gets turned off.

But if it is a "funny" camera like some of the SQ905s the initial values are 1 and 1, because the sensor is in fact mounted upside down. Now, suppose that there is some camera in the future which, just like this, has the sensor upside down, and suppose that said hypothetical camera also has the ability to "know" that it has been turned over so what was upside down is now right side up. Well, all that one has to do is to flip the two bits from whatever they were to have instead the opposite values!

Observe that this would take care of the orientation problem both for cameras which had the sensor mounted right in the first place, and for cameras which had the sensor mounted wrong in the first place. Just use the same two bits to describe the sensor orientation, and if there is any reason (based upon some ability to know that the camera orientation is now different) that the orientation should change, then just flip the bits as appropriate.

Then it would be the job of the support module to provide proper initial values only for these bits, and everything else could be done later on, in userspace.

Quoted 7 lines
Quoted 1 line

Theodore Kilgore

Theodore,

We want to be able to differentiate between a cam which has its sensor mounted upside down, and a cam which can be pivotted and happens to be upside down at the moment, in case of any upside down mounted sensor, we will always want to compentsate, in case of a pivotting camera wether we compensate or not could be a user preference.

So in you example of an upside down mounted sensor in a pivotting encasing and the encasing is pivotted 180 degrees we would have the hflip and vflip bits set for sensor orientation and we would have the pivotted 180 degrees bit set. If the user has choosen to compensate for pivotting the default, we would do nothing. But it is important to be able to differentiate between the 2.

Quoted 8 lines

Hans,

I am not sure if we are talking past each other, or what. But I was pointing out that the initial values of two bits can indicate the "default" orientation of the sensor, and this can be done permanently in the module, which transmits the initial setting of those two bits to anything up the line which is interested or curious to know those initial values. The information in those two bits will definitely tell whether the sensor is mounted upside down in the camera. For example, if it is mounted upside down, then they are both set in the module to "on" and exported therefrom. But if the sensor is mounted correctly, then both of them are set to "off" and similarly exported.

Now if any application for any reason (such as "knowing" that the camera is upside down or is pointing in the opposite direction, or into a mirror) wants to change the defaults, all it has to do is to toggle the bits.

But, hmmm. Perhaps there is the question about how the app "knows" that the camera is upside down or is pointed in another direction. If the camera has a gyroscope inside, for example, then it could be the camera which needs to tell to the app about the current orientation, or else the app would not have any way to know ... Is this the problem, then? For that kind of thing, one might need more than two bits in order to pass the needed information.

Yes that is what we are talking about, the camera having a gravity switch (usually nothing as advanced as a gyroscope). Also the bits we are talking about are in a struct which communicates information one way, from the camera to userspace, so there is no way to clear the bits to make the camera do something.

Regards,

Hans

#12 lore raw

On Sun, 22 Feb 2009, Hans de Goede wrote:

<big snip>

Quoted 5 lines

Yes that is what we are talking about, the camera having a gravity switch (usually nothing as advanced as a gyroscope). Also the bits we are talking about are in a struct which communicates information one way, from the camera to userspace, so there is no way to clear the bits to make the camera do something.

Regards,

Hans

Well, of course the bits are in a struct which is communicates information one way from the camera to userspace. But userspace can do what it deems appropriate with those bits. My point was that if userspace wants to turn the camera upside down in software, all it has to do is to negate those bits. For that purpose, it does not matter whether the bits were originally set "on" (indicating that the sensor is upside down in the camera) or whether they were set "off" (indicating the sensor is right side up in the camera). Now, if it is a question of passing along a changing camera orientation, it is obvious that has to be done with some other mechanism. So, unless my not terribly profound observation about the ability to negate bits was confusing, we are not in disagreement.

Theodore Kilgore

#13 lore raw
In reply to Hans de Goede
Quoted 4 lines · Hans de Goede

Yes that is what we are talking about, the camera having a gravity switch (usually nothing as advanced as a gyroscope). Also the bits we are talking about are in a struct which communicates information one way, from the camera to userspace, so there is no way to clear the bits to make the camera do something.

First, I'd like to say I agree with most that the installed orientation of the camera sensor really is a different concept than the current value of a gravity sensor. It's not necessary, and maybe not even desirable, to handle them in the same way.

I do not see the advantage of using reserved bits instead of controls.

The are a limited number of reserved bits. In some structures there are only a few left. They will run out. Then what? Packing non-standard sensor attributes and camera sensor meta-data into a few reserved bits is not a sustainable policy.

Controls on the other card are not limited and won't run out.

Currently reserved bits and previously reserved but now in use bits look exactly the same. How does an app know if the bits are valid or not? More reserved bits?

The control API is already designed to allow for enumeration of controls.

Reserved bits have no meta-data to document them. An app sees that some bits in the input struct which were reserved when it was written are now set. What does this tell the app? Nothing. What can the app tell the user? Nothing.

The control API provides for control meta-data. We have a means of enumeration, name, min, max, step size, and various flags to tell us about a control that wasn't already handled. Does this new gravity sensor have 45 degree resolution? The control's step size can show this even if only 90 degree rotations had been considered when gravity sensor support was first added to some other driver. At the very least, the app (and various existing utilities) can provide information about the controls to the user.

#14 lore raw
Quoted 11 lines · Trent Piepho
Quoted 4 lines · Hans de Goede

Yes that is what we are talking about, the camera having a gravity switch (usually nothing as advanced as a gyroscope). Also the bits we are talking about are in a struct which communicates information one way, from the camera to userspace, so there is no way to clear the bits to make the camera do something.

First, I'd like to say I agree with most that the installed orientation of the camera sensor really is a different concept than the current value of a gravity sensor. It's not necessary, and maybe not even desirable, to handle them in the same way.

Quoted 9 lines

I do not see the advantage of using reserved bits instead of controls.

The are a limited number of reserved bits. In some structures there are only a few left. They will run out. Then what? Packing non-standard sensor attributes and camera sensor meta-data into a few reserved bits is not a sustainable policy.

Controls on the other card are not limited and won't run out.

Yes but these things are *not* controls, end of discussion. The control API is for controls, not to stuff all kind of cruft in.

Regards,

Hans

#15 lore raw
Quoted 19 lines · Hans de Goede
Quoted 18 lines · Trent Piepho
Quoted 4 lines · Hans de Goede

Yes that is what we are talking about, the camera having a gravity switch (usually nothing as advanced as a gyroscope). Also the bits we are talking about are in a struct which communicates information one way, from the camera to userspace, so there is no way to clear the bits to make the camera do something.

First, I'd like to say I agree with most that the installed orientation of the camera sensor really is a different concept than the current value of a gravity sensor. It's not necessary, and maybe not even desirable, to handle them in the same way.

I do not see the advantage of using reserved bits instead of controls.

The are a limited number of reserved bits. In some structures there are only a few left. They will run out. Then what? Packing non-standard sensor attributes and camera sensor meta-data into a few reserved bits is not a sustainable policy.

Quoted 5 lines
Quoted 2 lines

Controls on the other card are not limited and won't run out.

Yes but these things are *not* controls, end of discussion. The control API is for controls, not to stuff all kind of cruft in.

All kind of cruft belongs in the reserved bits of whatever field it can be stuffed in? Until the bits run out and then it belongs where?

What is the difference? Why does it matter? Performance? Maintenance? Is there something that's not possible? I do not find "end of discussion" to be a very convincing argument.

#16 lore raw
Quoted 19 lines · Trent Piepho
Quoted 18 lines · Hans de Goede
Quoted 17 lines · Trent Piepho
Quoted 4 lines · Hans de Goede

Yes that is what we are talking about, the camera having a gravity switch (usually nothing as advanced as a gyroscope). Also the bits we are talking about are in a struct which communicates information one way, from the camera to userspace, so there is no way to clear the bits to make the camera do something.

First, I'd like to say I agree with most that the installed orientation of the camera sensor really is a different concept than the current value of a gravity sensor. It's not necessary, and maybe not even desirable, to handle them in the same way.

I do not see the advantage of using reserved bits instead of controls.

The are a limited number of reserved bits. In some structures there are only a few left. They will run out. Then what? Packing non-standard sensor attributes and camera sensor meta-data into a few reserved bits is not a sustainable policy.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Controls on the other card are not limited and won't run out.

Yes but these things are *not* controls, end of discussion. The control API is for controls, not to stuff all kind of cruft in.

All kind of cruft belongs in the reserved bits of whatever field it can be stuffed in?

Not whatever field, these are input properties which happen to also be pretty binary so putting them in the input flags field makes plenty of sense.

Quoted 3 lines

What is the difference? Why does it matter? Performance? Maintenance? Is there something that's not possible? I do not find "end of discussion" to be a very convincing argument.

Well they are not controls, that is the difference, the control interface is for controls (and only for controls, end of discussion if you ask me). These are not controls but properties, they do not have a default min and max value, they have only one *unchanging* value, there is nothing the application can control / change. It has been suggested to make them readonly, but that does not fix the ugliness. A proper written v4l2 application will enumerate all the controls, and then the user will see a grayed out control saying: "your cam is upside down" what is there to control ? will this be a grayed out slider? or a grayed out checkbox "your cam is upside down", or maybe a not grayed out dropdown: where the user can select: "my sensor is upside down", "I deny my sensor is upside down", "I don't care my sensor is upside down", "WTF is this doing in my webcam control panel?", "nwod edispu si rosnes yM"

Do you know I have an idea, lets get rid of the S2 API for DVB and put all that in controls too. Oh, and think like standards for video formats, surely that can be a control too, and ... and, ...

Yes we can stuff almost anything in a control, that does not make it a good idea.

Regards,

Hans

#17 lore raw
Quoted 31 lines · Hans de Goede
Quoted 19 lines · Trent Piepho
Quoted 18 lines · Hans de Goede
Quoted 17 lines · Trent Piepho
Quoted 4 lines · Hans de Goede

Yes that is what we are talking about, the camera having a gravity switch (usually nothing as advanced as a gyroscope). Also the bits we are talking about are in a struct which communicates information one way, from the camera to userspace, so there is no way to clear the bits to make the camera do something.

First, I'd like to say I agree with most that the installed orientation of the camera sensor really is a different concept than the current value of a gravity sensor. It's not necessary, and maybe not even desirable, to handle them in the same way.

I do not see the advantage of using reserved bits instead of controls.

The are a limited number of reserved bits. In some structures there are only a few left. They will run out. Then what? Packing non-standard sensor attributes and camera sensor meta-data into a few reserved bits is not a sustainable policy.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Controls on the other card are not limited and won't run out.

Yes but these things are *not* controls, end of discussion. The control API is for controls, not to stuff all kind of cruft in.

All kind of cruft belongs in the reserved bits of whatever field it can be stuffed in?

Not whatever field, these are input properties which happen to also be pretty binary so putting them in the input flags field makes plenty of sense.

Quoted 7 lines
Quoted 3 lines

What is the difference? Why does it matter? Performance? Maintenance? Is there something that's not possible? I do not find "end of discussion" to be a very convincing argument.

Well they are not controls, that is the difference, the control interface is for controls (and only for controls, end of discussion if you ask me). These are not controls but properties, they do not have a default min and max value,

Camera pivot sensor ranges from 0 to 270. How is that not a min and max?

Quoted 1 line

they have only one *unchanging* value, there is nothing the application can

Camera sensors don't have an unchanging value.

And who says scan order can't change? Suppose the camera returns raw bayer format data top to bottom, but if you request yuv then an image processing section needs to kick in and that returns the data bottom to top.

Quoted 8 lines

control / change. It has been suggested to make them readonly, but that does not fix the ugliness. A proper written v4l2 application will enumerate all the controls, and then the user will see a grayed out control saying: "your cam is upside down" what is there to control ? will this be a grayed out slider? or a grayed out checkbox "your cam is upside down", or maybe a not grayed out
dropdown: where the user can select: "my sensor is upside down", "I deny my sensor is upside down", "I don't care my sensor is upside down", "WTF is this doing in my webcam control panel?", "nwod edispu si rosnes yM"

Why is there a read-only flag for controls if the concept is so mind blowing to users? Have there been complaints about it?

Quoted 3 lines

Do you know I have an idea, lets get rid of the S2 API for DVB and put all that in controls too. Oh, and think like standards for video formats, surely that can be a control too, and ... and, ...

Good point. The S2 API is much more like the control interface than the previous API. Enumerated attributes which can be set one at a time or in groups. More can be added. There is some meta data about them. The old API used the a limited number of fixed structs, a few reserved bits, no meta-data, and a query/set everything at once API.

I think the camera meta-data and camera sensor API should look more like S2.

#18 lore raw
Quoted 39 lines · Trent Piepho
Quoted 29 lines · Hans de Goede
Quoted 19 lines · Trent Piepho
Quoted 18 lines · Hans de Goede
Quoted 17 lines · Trent Piepho
Quoted 4 lines · Hans de Goede

Yes that is what we are talking about, the camera having a gravity switch (usually nothing as advanced as a gyroscope). Also the bits we are talking about are in a struct which communicates information one way, from the camera to userspace, so there is no way to clear the bits to make the camera do something.

First, I'd like to say I agree with most that the installed orientation of the camera sensor really is a different concept than the current value of a gravity sensor. It's not necessary, and maybe not even desirable, to handle them in the same way.

I do not see the advantage of using reserved bits instead of controls.

The are a limited number of reserved bits. In some structures there are only a few left. They will run out. Then what? Packing non-standard sensor attributes and camera sensor meta-data into a few reserved bits is not a sustainable policy.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Controls on the other card are not limited and won't run out.

Yes but these things are *not* controls, end of discussion. The control API is for controls, not to stuff all kind of cruft in.

All kind of cruft belongs in the reserved bits of whatever field it can be stuffed in?

Not whatever field, these are input properties which happen to also be pretty binary so putting them in the input flags field makes plenty of sense.

Quoted 6 lines
Quoted 3 lines

What is the difference? Why does it matter? Performance? Maintenance? Is there something that's not possible? I do not find "end of discussion" to be a very convincing argument.

Well they are not controls, that is the difference, the control interface is for controls (and only for controls, end of discussion if you ask me). These are not controls but properties, they do not have a default min and max value,

Camera pivot sensor ranges from 0 to 270. How is that not a min and max?

Quoted 8 lines
Quoted 1 line

they have only one *unchanging* value, there is nothing the application can

Camera sensors don't have an unchanging value.

And who says scan order can't change? Suppose the camera returns raw bayer format data top to bottom, but if you request yuv then an image processing section needs to kick in and that returns the data bottom to top.

Yes, because hardware designers like throwing away lots of transistors to memory so they are going to put memory in the controller to buffer an entire frame and then scan out the memory buffer in different order then the sensor gave them the data, so they cannot do FIFO, so they will actually need 2 frames of memory.

If the sensor is soldered upside down on the PCB that is a very much unchanging value, and an input property if you ask me.

So new proposal: use 2 bits in the input flags to indicate if the input is hardwired vflipped and/or hflipped.

Create a new class of controls for querying possible changing camera properties like pivoting and aperture.

Regards,

Hans

#19 lore raw
In reply to Hans de Goede
Quoted 20 lines · On Sunday 22 February 2009 23:54:42 Hans de Goede wrote:
Quoted 19 lines · Trent Piepho
Quoted 5 lines · Hans de Goede

Yes that is what we are talking about, the camera having a gravity switch (usually nothing as advanced as a gyroscope). Also the bits we are talking about are in a struct which communicates information one way, from the camera to userspace, so there is no way to clear the bits to make the camera do something.

First, I'd like to say I agree with most that the installed orientation of the camera sensor really is a different concept than the current value of a gravity sensor. It's not necessary, and maybe not even desirable, to handle them in the same way.

I do not see the advantage of using reserved bits instead of controls.

The are a limited number of reserved bits. In some structures there are only a few left. They will run out. Then what? Packing non-standard sensor attributes and camera sensor meta-data into a few reserved bits is not a sustainable policy.

Quoted 4 lines
Quoted 1 line

Controls on the other card are not limited and won't run out.

Yes but these things are *not* controls, end of discussion. The control API is for controls, not to stuff all kind of cruft in.

I agree, these are not controls.

There is an option to use the current status field. There are enough bits free, that's not the problem. But the spec is explicit about the fact that these bits apply to the current input only, and that's not true for these new bits. We can change the spec in this regard of course, but then you have to document each bit of the status field whether it is valid for the current input only, or also if this isn't the current input. It's all a bit messy.

In addition, there are 4 reserved fields here and it is the first time in a very long time that we actually need one. And after all, that's why they are there in the first place.

I see three options:

1) stuff them into the status field after all.
2) take one of the reserved fields and make it a single 'flags' field.
3) take one of the reserved fields and make it a u16 capabilities and u16 flags field.

Trent does have a point that we need to be careful not to add fields without a good reason. Choosing option 1 fits the bill, and the orientation also fits the 'status' name. Only the sensor mount orientation is not really a status. Although with some creative naming we might come close :-)

Hmm, let's see:

V4L2_IN_ST_HAS_SENSOR_INFO 0x00000010
V4L2_IN_ST_SENSOR_HFLIPPED 0x00000020
V4L2_IN_ST_SENSOR_VFLIPPED 0x00000040

V4L2_IN_ST_HAS_PIVOT_INFO 0x00001000
V4L2_IN_ST_PIVOT_0 0x00000000
V4L2_IN_ST_PIVOT_90 0x00002000
V4L2_IN_ST_PIVOT_180 0x00004000
V4L2_IN_ST_PIVOT_270 0x00006000
V4L2_IN_ST_PIVOT_MSK 0x00006000

Actually, that's not too bad.

Regards,

Hans

Signature · 1 line
Hans Verkuil - video4linux developer - sponsored by TANDBERG
#20 lore raw
Quoted 36 lines · Hans Verkuil
Quoted 20 lines · On Sunday 22 February 2009 23:54:42 Hans de Goede wrote:
Quoted 19 lines · Trent Piepho
Quoted 5 lines · Hans de Goede

Yes that is what we are talking about, the camera having a gravity switch (usually nothing as advanced as a gyroscope). Also the bits we are talking about are in a struct which communicates information one way, from the camera to userspace, so there is no way to clear the bits to make the camera do something.

First, I'd like to say I agree with most that the installed orientation of the camera sensor really is a different concept than the current value of a gravity sensor. It's not necessary, and maybe not even desirable, to handle them in the same way.

I do not see the advantage of using reserved bits instead of controls.

The are a limited number of reserved bits. In some structures there are only a few left. They will run out. Then what? Packing non-standard sensor attributes and camera sensor meta-data into a few reserved bits is not a sustainable policy.

Quoted 4 lines
Quoted 1 line

Controls on the other card are not limited and won't run out.

Yes but these things are *not* controls, end of discussion. The control API is for controls, not to stuff all kind of cruft in.

I agree, these are not controls.

There is an option to use the current status field. There are enough bits free, that's not the problem. But the spec is explicit about the fact that these bits apply to the current input only, and that's not true for these new bits. We can change the spec in this regard of course, but then you have to document each bit of the status field whether it is valid for the current input only, or also if this isn't the current input. It's all a bit messy.

Quoted 3 lines

In addition, there are 4 reserved fields here and it is the first time in a very long time that we actually need one. And after all, that's why they are there in the first place.

v4l2_capability: 5 of 32 cap bits left, 4 reserved words
v4l2_fmtdesc: 31 flag bits, 4 reserved words
v4l2_buffer: 22 flags bits, 1 reserved word
v4l2_framebuffer: 25 cap bits, 26 flag bits, no reserved words
v4l2_input: 4 reserved words
v4l2_output: 4 reserved words
v4l2_tuner: 27 cap bits, 4 reserved words

Quoted 11 lines

Trent does have a point that we need to be careful not to add fields without a good reason. Choosing option 1 fits the bill, and the orientation also fits the 'status' name. Only the sensor mount orientation is not really a status. Although with some creative naming we might come close :-)

Hmm, let's see:

V4L2_IN_ST_HAS_SENSOR_INFO 0x00000010
V4L2_IN_ST_SENSOR_HFLIPPED 0x00000020
V4L2_IN_ST_SENSOR_VFLIPPED 0x00000040

Quoted 6 lines

V4L2_IN_ST_HAS_PIVOT_INFO 0x00001000
V4L2_IN_ST_PIVOT_0 0x00000000
V4L2_IN_ST_PIVOT_90 0x00002000
V4L2_IN_ST_PIVOT_180 0x00004000
V4L2_IN_ST_PIVOT_270 0x00006000
V4L2_IN_ST_PIVOT_MSK 0x00006000

One of my other points what the controls include meta-data. What happens when someone has an orientation sensor with 45 degree resolution? A control would just change the step size from 90 to 45. Existing software would already know what it means. With this method you have to add:

V4L2_IN_ST_HAS_PIVOT_45_INFO 0x00008000
V4L2_IN_ST_PIVOT_45 0x00010000
V4L2_IN_ST_PIVOT_135 0x00012000
V4L2_IN_ST_PIVOT_225 0x00014000
V4L2_IN_ST_PIVOT_315 0x00016000
V4L2_IN_ST_PIVOT_45_MSK 0x00016000

Interpreting the pivot bits becomes more fun now. Existing software can do nothing with these extra bits. It can tell the user nothing.

Next, add camera aperture. f/0.7 to f/64 in both half stop and third stop increments. Getting low on bits already.

Suppose I have just three or four sensors on my camera. That's not all that much. A typical digital camera has more than that. They're probably connected by something slow, like I2C. Maybe non-trivial calculations are needed to use the data, like most demod's SNR registers. Each time I query the input the driver must query and calculate the value of ALL the sensors. Even though I only can about one of them. Controls can be queried individually.

#21 lore raw
Quoted 81 lines · On Monday 23 February 2009 00:56:40 Trent Piepho wrote:
Quoted 37 lines · Hans Verkuil
Quoted 21 lines · On Sunday 22 February 2009 23:54:42 Hans de Goede wrote:
Quoted 20 lines · Trent Piepho
Quoted 5 lines · Hans de Goede

Yes that is what we are talking about, the camera having a gravity switch (usually nothing as advanced as a gyroscope). Also the bits we are talking about are in a struct which communicates information one way, from the camera to userspace, so there is no way to clear the bits to make the camera do something.

First, I'd like to say I agree with most that the installed orientation of the camera sensor really is a different concept than the current value of a gravity sensor. It's not necessary, and maybe not even desirable, to handle them in the same way.

I do not see the advantage of using reserved bits instead of controls.

The are a limited number of reserved bits. In some structures there are only a few left. They will run out. Then what? Packing non-standard sensor attributes and camera sensor meta-data into a few reserved bits is not a sustainable policy.

Quoted 4 lines
Quoted 1 line

Controls on the other card are not limited and won't run out.

Yes but these things are *not* controls, end of discussion. The control API is for controls, not to stuff all kind of cruft in.

I agree, these are not controls.

There is an option to use the current status field. There are enough bits free, that's not the problem. But the spec is explicit about the fact that these bits apply to the current input only, and that's not true for these new bits. We can change the spec in this regard of course, but then you have to document each bit of the status field whether it is valid for the current input only, or also if this isn't the current input. It's all a bit messy.

Quoted 3 lines

In addition, there are 4 reserved fields here and it is the first time in a very long time that we actually need one. And after all, that's why they are there in the first place.

v4l2_capability: 5 of 32 cap bits left, 4 reserved words
v4l2_fmtdesc: 31 flag bits, 4 reserved words
v4l2_buffer: 22 flags bits, 1 reserved word
v4l2_framebuffer: 25 cap bits, 26 flag bits, no reserved words
v4l2_input: 4 reserved words
v4l2_output: 4 reserved words
v4l2_tuner: 27 cap bits, 4 reserved words

Quoted 12 lines

Trent does have a point that we need to be careful not to add fields without a good reason. Choosing option 1 fits the bill, and the orientation also fits the 'status' name. Only the sensor mount orientation is not really a status. Although with some creative naming we might come close :-)

Hmm, let's see:

V4L2_IN_ST_HAS_SENSOR_INFO 0x00000010
V4L2_IN_ST_SENSOR_HFLIPPED 0x00000020
V4L2_IN_ST_SENSOR_VFLIPPED 0x00000040

Quoted 6 lines

V4L2_IN_ST_HAS_PIVOT_INFO 0x00001000
V4L2_IN_ST_PIVOT_0 0x00000000
V4L2_IN_ST_PIVOT_90 0x00002000
V4L2_IN_ST_PIVOT_180 0x00004000
V4L2_IN_ST_PIVOT_270 0x00006000
V4L2_IN_ST_PIVOT_MSK 0x00006000

One of my other points what the controls include meta-data. What happens when someone has an orientation sensor with 45 degree resolution? A control would just change the step size from 90 to 45. Existing software would already know what it means. With this method you have to add:

V4L2_IN_ST_HAS_PIVOT_45_INFO 0x00008000
V4L2_IN_ST_PIVOT_45 0x00010000
V4L2_IN_ST_PIVOT_135 0x00012000
V4L2_IN_ST_PIVOT_225 0x00014000
V4L2_IN_ST_PIVOT_315 0x00016000
V4L2_IN_ST_PIVOT_45_MSK 0x00016000

Quoted 2 lines

Interpreting the pivot bits becomes more fun now. Existing software can do nothing with these extra bits. It can tell the user nothing.

These bits deal exclusively with the sensor position and for the sole purpose of deciding whether the image has to be rotated in hard/software to get it the right-way up. In most cases this is limited to rotating 180 degrees, but omap has some memory management tricks to do the 90/270 degree cases as well.

This is not the same as having a full positioning 2D or 3D system in a camera that can give you exact angles. I do not think that should be part of v4l2. There isn't anything that v4l2 can do with that. An application might use this information to setup special transformations to do such rotates, and if the hardware can do that then that would be part of an effects API or something like that. With non-90 degree angle you just run into a totally different set of problems that are definitely out-of-scope.

Quoted 10 lines

Next, add camera aperture. f/0.7 to f/64 in both half stop and third stop increments. Getting low on bits already.

Suppose I have just three or four sensors on my camera. That's not all that much. A typical digital camera has more than that. They're probably connected by something slow, like I2C. Maybe non-trivial calculations are needed to use the data, like most demod's SNR registers.
Each time I query the input the driver must query and calculate the
value of ALL the sensors. Even though I only can about one of them. Controls can be queried individually.

Note that I am not averse to this idea in general. It would be easy to setup a separate class of 'controls' containing meta data that work the same as controls, but are just called differently. It has always been in the back of my mind that it would be a pretty good match, functionality-wise.

But in this case it just fits very well in the existing v4l2_input struct. These bits are specific to each input, so VIDIOC_ENUMINPUTS is the perfect match for that. If you want to do this with controls, then you would end up with a control for each input. Hardly elegant.

Regards,

Hans

Signature · 1 line
Hans Verkuil - video4linux developer - sponsored by TANDBERG
#22 lore raw
Quoted 91 lines · On Mon, 23 Feb 2009 08:34:08 +0100 Hans Verkuil <[email protected]> wrote:
Quoted 81 lines · On Monday 23 February 2009 00:56:40 Trent Piepho wrote:
Quoted 37 lines · Hans Verkuil
Quoted 21 lines · On Sunday 22 February 2009 23:54:42 Hans de Goede wrote:
Quoted 20 lines · Trent Piepho
Quoted 5 lines · Hans de Goede

Yes that is what we are talking about, the camera having a gravity switch (usually nothing as advanced as a gyroscope). Also the bits we are talking about are in a struct which communicates information one way, from the camera to userspace, so there is no way to clear the bits to make the camera do something.

First, I'd like to say I agree with most that the installed orientation of the camera sensor really is a different concept than the current value of a gravity sensor. It's not necessary, and maybe not even desirable, to handle them in the same way.

I do not see the advantage of using reserved bits instead of controls.

The are a limited number of reserved bits. In some structures there are only a few left. They will run out. Then what? Packing non-standard sensor attributes and camera sensor meta-data into a few reserved bits is not a sustainable policy.

Quoted 4 lines
Quoted 1 line

Controls on the other card are not limited and won't run out.

Yes but these things are *not* controls, end of discussion. The control API is for controls, not to stuff all kind of cruft in.

I agree, these are not controls.

There is an option to use the current status field. There are enough bits free, that's not the problem. But the spec is explicit about the fact that these bits apply to the current input only, and that's not true for these new bits. We can change the spec in this regard of course, but then you have to document each bit of the status field whether it is valid for the current input only, or also if this isn't the current input. It's all a bit messy.

Quoted 3 lines

In addition, there are 4 reserved fields here and it is the first time in a very long time that we actually need one. And after all, that's why they are there in the first place.

v4l2_capability: 5 of 32 cap bits left, 4 reserved words
v4l2_fmtdesc: 31 flag bits, 4 reserved words
v4l2_buffer: 22 flags bits, 1 reserved word
v4l2_framebuffer: 25 cap bits, 26 flag bits, no reserved words
v4l2_input: 4 reserved words
v4l2_output: 4 reserved words
v4l2_tuner: 27 cap bits, 4 reserved words

Quoted 12 lines

Trent does have a point that we need to be careful not to add fields without a good reason. Choosing option 1 fits the bill, and the orientation also fits the 'status' name. Only the sensor mount orientation is not really a status. Although with some creative naming we might come close :-)

Hmm, let's see:

V4L2_IN_ST_HAS_SENSOR_INFO 0x00000010
V4L2_IN_ST_SENSOR_HFLIPPED 0x00000020
V4L2_IN_ST_SENSOR_VFLIPPED 0x00000040

Quoted 6 lines

V4L2_IN_ST_HAS_PIVOT_INFO 0x00001000
V4L2_IN_ST_PIVOT_0 0x00000000
V4L2_IN_ST_PIVOT_90 0x00002000
V4L2_IN_ST_PIVOT_180 0x00004000
V4L2_IN_ST_PIVOT_270 0x00006000
V4L2_IN_ST_PIVOT_MSK 0x00006000

One of my other points what the controls include meta-data. What happens when someone has an orientation sensor with 45 degree resolution? A control would just change the step size from 90 to 45. Existing software would already know what it means. With this method you have to add:

V4L2_IN_ST_HAS_PIVOT_45_INFO 0x00008000
V4L2_IN_ST_PIVOT_45 0x00010000
V4L2_IN_ST_PIVOT_135 0x00012000
V4L2_IN_ST_PIVOT_225 0x00014000
V4L2_IN_ST_PIVOT_315 0x00016000
V4L2_IN_ST_PIVOT_45_MSK 0x00016000

Quoted 2 lines

Interpreting the pivot bits becomes more fun now. Existing software can do nothing with these extra bits. It can tell the user nothing.

These bits deal exclusively with the sensor position and for the sole purpose of deciding whether the image has to be rotated in hard/software to get it the right-way up. In most cases this is limited to rotating 180 degrees, but omap has some memory management tricks to do the 90/270 degree cases as well.

Quoted 7 lines

This is not the same as having a full positioning 2D or 3D system in a camera that can give you exact angles. I do not think that should be part of v4l2. There isn't anything that v4l2 can do with that. An application might use this information to setup special transformations to do such rotates, and if the hardware can do that then that would be part of an effects API or something like that. With non-90 degree angle you just run into a totally different set of problems that are definitely out-of-scope.

We are trying to address 2 different situations using the same approach. After thinking more about it, I suspect that we need two separate approaches.

For the static info that a sensor is mounted on a different position on a notebook case, for example. The same camera could be mounted on different position for different hardwares and kernel will never know for sure, if the vendor didn't care to change the USB ID for the rotated camera.

All kernel could provide is a HINT info, for some known cases, but it should be possible to override this in userspace. For this case, the better userspace api seems to be by using some flags at v4l2_input.

However, I'm starting to agree with one of the Hans de Goede opinions that this information should be stored at userspace, since, on a large amount of cases, like surveillance mounted cameras, or some hardware with a fixed commercial webcam, like an ATM machine, this info is not stored anyware. The only way is to allow users to edit a file pointing the webcam orientation for that device.

The second case is something like a gravity sensor. This can provide us 2D/3D angles with 90 degrees stepping, but I don't doubt that more sophisticated cameras will start to appear, having a more precise mechanism, to be used by applications that could compensate a 2D/3D rotation on software.

In the trivial case of a 90 degrees step, IMO, the event interface seems to be the better approach. However, if we really want to do motion compensation in software, the only alternative is to have this information together with the streaming meta-data.

Cheers,
Mauro

author
#23 lore raw
In reply to Hans de Goede
Quoted 5 lines · Hans de Goede

We want to be able to differentiate between a cam which has its sensor mounted upside down, and a cam which can be pivotted and happens to be upside down at the moment, in case of any upside down mounted sensor, we will always want to compentsate, in case of a pivotting camera wether we compensate or not could be a user preference.

If we take Olivier Lorin's gl-860 case though, how do we define what is the normal orientation and what is pivoted, it is likely we'd just decide the direction where the sensor output is the right way up is normal and the other is pivoted and then what info have you got from having multiple flags.

In order to explain what I mean it is probably best to refer to rotations in terms of pitch, yaw and roll (as per the definitions at
http://en.wikipedia.org/wiki/Flight_dynamics) where the forward direction is the shooting direction.

When still cameras are fitted with gravity sensors they are normally set up with the intention of measuring roll and will often get confused by 90 degrees of pitch. If a laptop is fitted with a camera that can either record the user or the view looking away from the user then the camera needs to be able to either pitch or yaw but not roll. If the camera yaws then no correction is needed but if it pitches then the resulting image needs rotating to be the correct way up (and in that scenario it is improbable that the user doesn't want the correction applied). Other than the fact that one needs correcting and the other doesn't these options appear identical to the user and so manufacturers provide one or the other but not both.

If a video camera had a roll sensor (or even, as a believe some specialist tripods can manage, a full set of roll, pitch and yaw measurements) then a substantially different mechanism is needed to provide access to that data but in the absence of anyone having access to such equipment I don't think we can design the interface now.

Adam

#24 lore raw
In reply to Hans Verkuil
Quoted 3 lines · On Sat, 21 Feb 2009 12:53:57 +0100 Hans Verkuil <[email protected]> wrote:

Hi Adam,

Sorry for the late reply, it's been very busy.

Me too.

Quoted 6 lines
Quoted 6 lines

1) Reuse the existing HFLIP and VFLIP controls, marking them as read-only Pros : No change needed to videodev2.h
Cons: It is confusing to have controls that have a subtly different meaning if they are read only. Existing apps that support those controls might get confused. Would require polling to support the case of a camera being turned toward / away from the user while streaming.

Reusing an existing control for a different meaning seems wrong. What happens when some cam has the capability of doing hardware flipping, and have the cam flipped?

Quoted 7 lines
Quoted 7 lines

2) Introduce a new orientation control (possibly in a new CAMERA_PROPERTIES class)
Pros: libv4l can easily tell if the driver supports the control.
Cons: It is really a property, not a control so calling it a control is wrong. Controls add lots of overhead in terms of driver code. Would require polling to support the case of a camera being turned toward / away from the user while streaming.

I think this could be a good idea, but I agree with Hans Verkuil comments: since this is a characteristics for a given input, using a control here would mean that the driver will return it based on the selected input. Seems a little messy. Also, a mounted characteristics of the device, is not really a sensor, as Hans de Goede pointed.

Quoted 8 lines
Quoted 6 lines

3) Use an extra couple of bits in V4L2_BUF_FLAGS
Pros: Simple to implement. Can change per frame without needing polling.
Cons: Doesn't work for non libv4l apps that try to use the read() interface. Can't easily identify drivers that don't support it (does 0 mean not rotated or just not implemented). Can only be read when streaming (does that matter?)

I think that matters, yes.

I don't think this is a good idea. The metadata at the frame polling is meant to return the stream info. We shouldn't mix sensor position here.

Quoted 14 lines
Quoted 9 lines

4) Use some reserved bits from the v4l2_capability structure
Pros: Less overhead than controls.
Cons: Would require polling to support the case of a camera being turned toward / away from the user while streaming. Can't easily identify drivers that don't support it.
5) Use some reserved bits from the v4l2_input structure (or possibly the status word but that is normally only valid for current input)
Pros: Less overhead than controls. Could support multiple sensors in one camera if such a beast exists.

What does exist is devices with a video input (e.g. composite) and a camera
input: each input will have different flags. Since these vflip/hflip properties do not change they can be enumerated in advance and you know what each input supports.

Quoted 5 lines
Quoted 2 lines

Cons: Would require polling to support the case of a camera being turned toward / away from the user while streaming.

Polling applies only to the bits that tell the orientation of the camera. See below for a discussion of this.

Analog tv does polling for signal strength, since userspace apps do mute and stops presenting video, if the signal is too weak. IMO, a similar mechanism should be used by pivoting.

IMO, this would be better addressed as a property of v4l2_input. So, I think that (5) is better than (4).

Quoted 17 lines
Quoted 1 line

Can't easily identify drivers that don't support it.

Not too difficult to add through the use of a capability bit. Either in v4l2_input or (perhaps) v4l2_capability.

Another Pro is that this approach will also work for v4l2_output in the case of, say, rotated LCD displays. Using camera orientation bits in v4l2_buffer while capturing will work, but using similar bits for output will fail since the data is going in the wrong direction.

Quoted 7 lines

The interest in detecting if a driver provides this informnation is to allow libv4l to know when it should use the driver provided information and when it should use its internal table (which needs to be retained for backward compatibility). With no detection capability the driver provided info should be ignored for USB IDs in the built in table.

Thoughts please

There is a case that we should think: at libv4l, we may need to override the "default" orientation, by a custom one. For example: Surveillance systems have cameras mounted on a fixed position. Depending on the camera, and the desired position, some cameras may needed to be mounted rotated (the same case also applies to some embedded hardware like ATM machines, where a webcam maybe mounted with 180 degrees, due to hardware constraints).

Ok, this is nothing that kernel needs to handle, but, at userspace, we need to have a file where the user could edit and store the camera position, to override whatever we have in kernel.

Quoted 18 lines

Is polling bad in this case? It is not something that needs immediate attention IMHO. The overhead for checking once every X seconds is quite low. Furthermore, it is only needed on devices that cannot do v/hflipping in hardware.

An alternative is to put some effort in a proper event interface. There is one implemented in include/linux/dvb/video.h and used by ivtv for video decoding. The idea is that the application registers events it wants to receive, and whenever such an event arrives the select() call will exit with a high-prio event (exception). The application then checks what happened.

The video.h implementation is pretty crappy, but we can make something more useful. Alternatively, I believe there are other event mechanisms as well in the kernel. I know some work was done on that, but I don't know what the end result was.

Quoted 4 lines

Note that this does not apply to sensor mount information. I think that v4l2_input is the perfect place for that. It is after all an input-specific property and static property, and it maps 1-on-1 to the output case as well, should we need it there as well.

I think we will need an event interface sooner or later. I had some discussions in the past about servo mechanisms to control the webcam position. Let's consider a webcam with movement servos at vertical and horizontal, and that we want to follow an object, like a person face. After sending a command to the servo, it will start moving, but it will take some time until it reaches the point. An event interface seems interesting to advice userspace that the movement already happened.

Maybe we could use this approach for the gravity sensors.

Cheers,
Mauro

author
#25 lore raw
Quoted 16 lines · Mauro Carvalho Chehab

On Sat, 21 Feb 2009 12:53:57 +0100

Quoted 3 lines · Hans Verkuil

Hi Adam,

Sorry for the late reply, it's been very busy.

Me too.

Quoted 6 lines
Quoted 6 lines

1) Reuse the existing HFLIP and VFLIP controls, marking them as read-only Pros : No change needed to videodev2.h
Cons: It is confusing to have controls that have a subtly different meaning if they are read only. Existing apps that support those controls might get confused. Would require polling to support the case of a camera being turned toward / away from the user while streaming.

Quoted 3 lines

Reusing an existing control for a different meaning seems wrong. What happens when some cam has the capability of doing hardware flipping, and have the cam flipped?

I thought that case had already been agreed, implement the flip controls but set the flip flag in the hardware to the opposite of what the control says.

Quoted 47 lines
Quoted 16 lines
Quoted 10 lines

4) Use some reserved bits from the v4l2_capability structure
Pros: Less overhead than controls.
Cons: Would require polling to support the case of a camera being turned toward / away from the user while streaming. Can't easily identify drivers that don't support it.
5) Use some reserved bits from the v4l2_input structure (or possibly the status word but that is normally only valid for current input)
Pros: Less overhead than controls. Could support multiple sensors in one camera if such a beast exists.

What does exist is devices with a video input (e.g. composite) and a camera input: each input will have different flags. Since these vflip/hflip properties do not change they can be enumerated in advance and you know what each input supports.

Quoted 5 lines
Quoted 2 lines

Cons: Would require polling to support the case of a camera being turned toward / away from the user while streaming.

Polling applies only to the bits that tell the orientation of the camera. See below for a discussion of this.

Analog tv does polling for signal strength, since userspace apps do mute and stops presenting video, if the signal is too weak. IMO, a similar mechanism should be used by pivoting.

IMO, this would be better addressed as a property of v4l2_input. So, I think that (5) is better than (4).

Quoted 17 lines
Quoted 1 line

Can't easily identify drivers that don't support it.

Not too difficult to add through the use of a capability bit. Either in v4l2_input or (perhaps) v4l2_capability.

Another Pro is that this approach will also work for v4l2_output in the case of, say, rotated LCD displays. Using camera orientation bits in v4l2_buffer while capturing will work, but using similar bits for output will fail since the data is going in the wrong direction.

Quoted 7 lines

The interest in detecting if a driver provides this informnation is to allow libv4l to know when it should use the driver provided information and when it should use its internal table (which needs to be retained for backward compatibility). With no detection capability the driver provided info should be ignored for USB IDs in the built in table.

Thoughts please

Quoted 7 lines

There is a case that we should think: at libv4l, we may need to override the "default" orientation, by a custom one. For example: Surveillance systems have cameras mounted on a fixed position. Depending on the camera, and the desired position, some cameras may needed to be mounted rotated (the same case also applies to some embedded hardware like ATM machines, where a webcam maybe mounted with 180 degrees, due to hardware constraints).

Agreed, Hans de Geode pointed out the similar case that 2 laptops may use the same camera but one mount it upside down so hardware info unrelated to the camera indicates the orientation.

Quoted 4 lines

Ok, this is nothing that kernel needs to handle, but, at userspace, we need to have a file where the user could edit and store the camera position, to override whatever we have in kernel.

Unfortunately what that doesn't address is the problem that first started this discussion. A camera where the orientation information is contained in the USB messages from the camera so the driver is the only thing that can reasonably access it.

Note also that for sensor orientation I doubt that 90 or 270 degrees rotation will be seen but I do know that the case of data being flipped on just one axis does exist.

Quoted 23 lines
Quoted 18 lines

Is polling bad in this case? It is not something that needs immediate attention IMHO. The overhead for checking once every X seconds is quite low. Furthermore, it is only needed on devices that cannot do v/hflipping in hardware.

An alternative is to put some effort in a proper event interface. There is one implemented in include/linux/dvb/video.h and used by ivtv for video decoding. The idea is that the application registers events it wants to receive, and whenever such an event arrives the select() call will exit with a high-prio event (exception). The application then checks what happened.

The video.h implementation is pretty crappy, but we can make something more useful. Alternatively, I believe there are other event mechanisms as well in the kernel. I know some work was done on that, but I don't know what the end result was.

Quoted 4 lines

Note that this does not apply to sensor mount information. I think that v4l2_input is the perfect place for that. It is after all an input-specific property and static property, and it maps 1-on-1 to the output case as well, should we need it there as well.

Quoted 7 lines

I think we will need an event interface sooner or later. I had some discussions in the past about servo mechanisms to control the webcam position. Let's consider a webcam with movement servos at vertical and horizontal, and that we want to follow an object, like a person face. After sending a command to the servo, it will start moving, but it will take some time until it reaches the point. An event interface seems interesting to advice userspace that the movement already happened.

I suspect it would make most sense in that sort of scenario to consider the servos as a separate device - they might easily be part of a mount that is usable independent of the camera fitted to it.

Adam

#26 lore raw
Quoted 9 lines · Adam Baker
Quoted 8 lines · Mauro Carvalho Chehab

On Sat, 21 Feb 2009 12:53:57 +0100

Quoted 3 lines · Hans Verkuil

Hi Adam,

Sorry for the late reply, it's been very busy.

Me too.

<big snip>

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

The interest in detecting if a driver provides this informnation is to allow libv4l to know when it should use the driver provided information and when it should use its internal table (which needs to be retained for backward compatibility). With no detection capability the driver provided info should be ignored for USB IDs in the built in table.

<snip>

Quoted 10 lines
Quoted 5 lines

Ok, this is nothing that kernel needs to handle, but, at userspace, we need to have a file where the user could edit and store the camera position, to override whatever we have in kernel.

Unfortunately what that doesn't address is the problem that first started this discussion. A camera where the orientation information is contained in the USB messages from the camera so the driver is the only thing that can reasonably access it.

Alas, this is so true. What started the entire discussion about passing the info about sensor orientation is a set of cameras all of which have the same Vendor:Product ID but require different handling of the data (vflip and hflip, or just vflip) depending upon information which can only be obtained by communication with the camera, *not* by just knowing its Vendor:Product ID.

Therefore, it is a little bit disheartening to see discussions -- again -- which come back to some kind of "internal table" in V4L, or which come back to things like "have a file where the user could edit and store the camera position, to override whatever we have in kernel."

Repeating the obvious, which apparently still needs to be repeated because not all of the participants in the discussion "get it":

1. The "internal table" in V4L could not handle this problem, because the internal table would be based upon what information? The USB Vendor:Product number? For that matter, no other table could work, either. Well, actually, a table could work. It would have to be inside the module supporting the camera, and the matter of which table entry corresponds to what camera would have to be settled by passing a USB command to the camera and then parsing the camera's response. So now the question is how to get the information out of the module, which can only be collected and analysed inside the module.

2. The "have a file where the user could edit" idea may seem attractive to some, because it shoves the whole problem of agreeing on the appropriate way to get needed information out of a kernel module and into userspace onto someone not present during the current discussion, the user. However, this is not a solution, and thinking about it just a little bit ought to make that totally obvious. This is a strongly worded statement, so I will proceed to explain why the matter is so obvious.

Let us assume the very best, and assume that every app which is V4L conformant has a "one time" initialization step and creates a directory $HOME/.app containing stored settings. So we might have a file called $HOME/.app/sq905. This file gets automatically written when the hapless user hooks up his sq905 camera the first time, and has to go through a choice routine to decide which side of the frame is up and which side is to the left and to the right. One could even take serious steps to make this otherwise unnecessary and silly sequence to be as really nice and "user friendly" as it could possibly be, and set this all up to be done with a sequence of mouse clicks and then the file, very kindly, gets written automatically. Those of you who are thinking that this "file a user could edit" is the way to go are, presumably, thinking along these lines. Well there are at least four things which are obviously wrong with this "solution":

2.1. The user is forced to deal with something which the user should not even have to confront. The user is called upon to remedy an omission and a deficiency which was ignored at a lower level, because a bunch of developers could not come together on a reasonable course of action. Well, some don't like to see this one, so there are three more reasons.

2.2 Every camera is going to require a file for itself in $HOME/.app even if there is nothing that the user needs to do. Many of the supported cameras need nothing of the kind, so this would be kind of silly.

2.3 The user has two apps for dealing with webcams. So now the user needs to have another directory called $HOME/.app2 with similar files in it?

2.4 (and this one is the worst of all) The user has two cameras which are both powered by the same kernel module, and the two cameras need two different things done. Now what??? Both cameras can not simultaneously have a valid entry in $HOME/.app/sq905 which tells what to do with the data out of the camera. Because what is right for one of them is wrong for the other. Further to add to this, one of the tests which was run on this very module was to see if it will run two of the same kind of camera simultaneously. It passed the test. Now someone wants the data to be right-side up from both cameras, and correctly oriented left-to right. Is that an unreasonable desire on the part of a user? Of course not. Again, now what????

There is no way out of this series of dilemmas and absurdities with the "file a user could edit" approach. None.

So, please, could we get serious and actually come up with a reasonable solution? To put the matter completely in perspective, the support for the affected set of devices is otherwise quite complete, and the only thing remaining is to come up with a reasonable solution, which would provide a pleasing experience to users.

Theodore Kilgore

#27 lore raw
Quoted 60 lines · On Mon, 23 Feb 2009, [email protected] wrote:
Quoted 9 lines · Adam Baker
Quoted 8 lines · Mauro Carvalho Chehab

On Sat, 21 Feb 2009 12:53:57 +0100

Quoted 3 lines · Hans Verkuil

Hi Adam,

Sorry for the late reply, it's been very busy.

Me too.

<big snip>

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

The interest in detecting if a driver provides this informnation is to
allow libv4l to know when it should use the driver provided information
and when it should use its internal table (which needs to be retained
for backward compatibility). With no detection capability the driver
provided info should be ignored for USB IDs in the built in table.

<snip>

Quoted 8 lines
Quoted 7 lines

Ok, this is nothing that kernel needs to handle, but, at userspace, we need
to have a file where the user could edit and store the camera position, to
override whatever we have in kernel.

Quoted 6 lines

Unfortunately what that doesn't address is the problem that first started this
discussion. A camera where the orientation information is contained in the
USB messages from the camera so the driver is the only thing that can reasonably access it.

Alas, this is so true. What started the entire discussion about passing the info about sensor orientation is a set of cameras all of which have the same Vendor:Product ID but require different handling of the data (vflip and hflip, or just vflip) depending upon information which can only be obtained by communication with the camera, *not* by just knowing its Vendor:Product ID.

Therefore, it is a little bit disheartening to see discussions -- again -- which come back to some kind of "internal table" in V4L, or which come back to things like "have a file where the user could edit and store the camera position, to override whatever we have in kernel."

Repeating the obvious, which apparently still needs to be repeated because not all of the participants in the discussion "get it":

1. The "internal table" in V4L could not handle this problem, because the internal table would be based upon what information? The USB Vendor:Product number? For that matter, no other table could work, either. Well, actually, a table could work. It would have to be inside the module supporting the camera, and the matter of which table entry corresponds to what camera would have to be settled by passing a USB command to the camera and then parsing the camera's response. So now the question is how to get the information out of the module, which can only be collected and analysed inside the module.

2. The "have a file where the user could edit" idea may seem attractive to some, because it shoves the whole problem of agreeing on the appropriate way to get needed information out of a kernel module and into userspace onto someone not present during the current discussion, the user. However, this is not a solution, and thinking about it just a little bit ought to make that totally obvious. This is a strongly worded statement, so I will proceed to explain why the matter is so obvious.

Let us assume the very best, and assume that every app which is V4L conformant has a "one time" initialization step and creates a directory $HOME/.app containing stored settings. So we might have a file called $HOME/.app/sq905. This file gets automatically written when the hapless user hooks up his sq905 camera the first time, and has to go through a choice routine to decide which side of the frame is up and which side is to the left and to the right. One could even take serious steps to make this otherwise unnecessary and silly sequence to be as really nice and "user friendly" as it could possibly be, and set this all up to be done with a sequence of mouse clicks and then the file, very kindly, gets written automatically. Those of you who are thinking that this "file a user could edit" is the way to go are, presumably, thinking along these lines. Well there are at least four things which are obviously wrong with this "solution":

2.1. The user is forced to deal with something which the user should not even have to confront. The user is called upon to remedy an omission and a deficiency which was ignored at a lower level, because a bunch of developers could not come together on a reasonable course of action. Well, some don't like to see this one, so there are three more reasons.

2.2 Every camera is going to require a file for itself in $HOME/.app even if there is nothing that the user needs to do. Many of the supported cameras need nothing of the kind, so this would be kind of silly.

Quoted 7 lines

2.3 The user has two apps for dealing with webcams. So now the user needs to have another directory called $HOME/.app2 with similar files in it?

2.4 (and this one is the worst of all) The user has two cameras which are both powered by the same kernel module, and the two cameras need two different things done. Now what??? Both cameras can not simultaneously have a valid entry in $HOME/.app/sq905 which tells what to do with the data out of the camera. Because what is right for one of them is wrong for the other. Further to add to this, one of the tests which was run on this very module was to see if it will run two of the same kind of camera simultaneously. It passed the test. Now someone wants the data to be right-side up from both cameras, and correctly oriented left-to right. Is that an unreasonable desire on the part of a user? Of course not. Again, now what????

There is no way out of this series of dilemmas and absurdities with the "file a user could edit" approach. None.

So, please, could we get serious and actually come up with a reasonable solution? To put the matter completely in perspective, the support for the affected set of devices is otherwise quite complete, and the only thing remaining is to come up with a reasonable solution, which would provide a pleasing experience to users.

Theodore,

You're considering just one subset of the V4L usages: notebook webcams. If you think about non-notebook webcams [1] or about security cams, you'll see that the only way for you to have the flipping information is inside some userspace file.

For example, I have here one video input bttv board with 16 cameras,
connected on 4 bttv chips. I have also a Pelco camera, that has just one support socket. Depending on the place you mount it, the camera has to be rotated by 180 degrees. It can also be useful to mount it rotated by 90 degrees.

After mounting the cameras, no matter what apps you are using (a streaming video app to broadcast it via internet, a security app, etc), the rotation information for that input, on that particular PCI, bus won't change.

As we've standardized the VIDIOC_QUERYCAP bus_info, now, the information for the camera at input 3 of bttv hardware at PCI addres 03:00.3 is unique. It is also unique the same information for a notebook mounted webcam (since the USB bus ID won't change for that devices).

So, if we standardize where this information is stored, for example, at /etc/libv4l and/or at ~/.libv4lrc, and let libv4l handle such info, this will be something consistent for all applications using libv4l. Other apps that might not want to trust on libv4l can also read the info at the same file.

So, I really think that this should be part of the approach.

I agree that we should have a way to get a hint about a camera rotation, if this is somehow provided by reading at the hardware.

[1] Normal webcams can be mounted on some hardware and have some orientations that would be different than expected. I've seen this before on some PC-based harware where the camera is enclosed inside the clause, like on Automatic Transfer Machines. Also, the user may want to use the camera on an inverted position, to make easy to fix it somewhere.

--
Cheers,
Mauro Carvalho Chehab
http://linuxtv.org
[email protected]

#28 lore raw
Quoted 1 line · Mauro Carvalho Chehab

On Mon, 23 Feb 2009, [email protected] wrote:

<big snip>

Quoted 3 lines

Theodore,

You're considering just one subset of the V4L usages: notebook webcams.

Actually, the sq905 cameras are not "notebook webcams." They are cheap, consumer entry level dual-mode cameras. They can be used as hand-held still cameras, to shoot still photos, and they can also be used as webcams. When sold, they usually came with some kind of mounting device that could hold them rigidly for webcam use. There are lots of similar cameras. Mercifully, not all of those others have the problems of the sq905, which have led to the present impasse.

If

Quoted 1 line

you think about non-notebook webcams [1] or about security cams, you'll see that the only way for you to have the flipping information is inside some userspace file.

However, there are obvious differences. For those cameras the question might well come up about how to control the movement of the camera, or, at least, to be aware of which way the camera is pointing. For these, the topic is an inherent property of the particular model of the camera -- or a defect, if someone wants to say so. Since the property is not determined by USB number, it is inherently impossible outside of the module to create a table which contains the needed information.

My intention here was to re-focus attention on the original problem which brought up the current discussion, and to that end the problem must be clearly understood. To have the flipping information inside some userspace file might solve some other problem and may be a generally very good idea. But it will not, can not, and never will be able to solve this problem.

Quoted 3 lines

For example, I have here one video input bttv board with 16 cameras,
connected on 4 bttv chips. I have also a Pelco camera, that has just one support socket. Depending on the place you mount it, the camera has to be rotated by 180 degrees. It can also be useful to mount it rotated by 90 degrees.

Good. So one needs external controls and userspace tools. Did I ever say that such things should never be done? No. All I said was that there is a problem, presently on the table, and those kinds of things are not, can not be, never were, and never will be solutions for _this_ problem.

Quoted 5 lines

After mounting the cameras, no matter what apps you are using (a streaming video app to broadcast it via internet, a security app, etc), the rotation information for that input, on that particular PCI, bus won't change.

As we've standardized the VIDIOC_QUERYCAP bus_info, now, the information for the camera at input 3 of bttv hardware at PCI addres 03:00.3 is unique. It is also unique the same information for a notebook mounted webcam (since the USB bus ID won't change for that devices).

Errrm... Again, the cameras in question here are not notebook mounted webcams.

Quoted 2 lines

So, if we standardize where this information is stored, for example, at /etc/libv4l and/or at ~/.libv4lrc, and let libv4l handle such info, this will be something consistent for all applications using libv4l. Other apps that might not want to trust on libv4l can also read the info at the same file.

Sorry, this will not work here. It may solve some other problem, but not this problem. Or, if one wants to "store" the information there, I don't care, really, but then there needs to be a way to get the information from the module, where it is, and get written into said table, which is where you want it, and this needs to happen every time an sq905 camera gets plugged in -- without pestering the user about the matter every time that such a camera gets hooked up.

Comparison: I have tossed a coin. Is it going to come up heads? Or tails? It is possible to know which, because the coin has been tossed. It would not even be cheating to look at it, or allow someone who did look at it, to pass to us the information. But we are not going to look, and if someone tells us we will not listen because we have not agreed on what language to use for communication. Instead, we will put a guess about the outcome into a table. We will make it a nice table, which can be revised using nice GUI tools, so it is easy for the user. So if our guess is wrong let the user fix it. Then next time we toss the coin the table entry will be right because either it was right before, or now someone fixed it???

Quoted 2 lines

So, I really think that this should be part of the approach.

I was not even addressing what should or should not be part of the approach to some other problems. My point was that such discussion is not germane to the problem of how to pass on the correct orientation of the sensor, for the sq905 cameras. There are lots of other problems out there to solve. No denying that.

Also an overview is often very helpful. Also trying to visualize what might be needed in the future is helpful. All of this can be extremely helpful. But not everyone can see or imagine every possible thing. For example, it seems that some of the best minds in the business are stunned when confronted with the fact that some manufacturer of cheap electronics in Taiwan has produced a lot of mass-market cameras with the sensors turned upside down, along with some other cameras having the same USB ID with different sensors, which act a bit differently. Clearly, if such a thing happened once it can happen again. So how to deal with such a problem? Something similar or worse will surely come up again.

Quoted 5 lines

I agree that we should have a way to get a hint about a camera rotation, if this is somehow provided by reading at the hardware.

[1] Normal webcams can be mounted on some hardware and have some orientations that would be different than expected. I've seen this before on some PC-based harware where the camera is enclosed inside the clause, like on Automatic Transfer Machines. Also, the user may want to use the camera on an inverted position, to make easy to fix it somewhere.

Again, these are in fact separate issues. The fundamental issue required for supporting the sq905 cameras is that an agreed-upon method must exist by which precisely two pieces of information can be passed along by the module. Which of the two pieces of information is relevant and needs to be sent along is, alas, only known from within the module. That is the backdrop for the entire discussion. As far as I know, this thread would not exist if that need had not come up. These two pieces of information are, precisely, "frame data requires flipping across a horizontal axis unless camera is upside down" and "frame data requires flipping across a vertical axis unless camera is held up to a mirror"

So, what do these two deep questions, which confound the assembled wisdom of an entire list of Linux video developers, have to do with tables in userspace? None that I can see, unless someone wants to provide a mechanism for the information, having been collected in the module, to be available to the table in userspace.

Theodore Kilgore

#29 lore raw
Quoted 79 lines · On Tue, 24 Feb 2009, [email protected] wrote:
Quoted 1 line · Mauro Carvalho Chehab

On Mon, 23 Feb 2009, [email protected] wrote:

<big snip>

Quoted 3 lines

Theodore,

You're considering just one subset of the V4L usages: notebook webcams.

Actually, the sq905 cameras are not "notebook webcams." They are cheap, consumer entry level dual-mode cameras. They can be used as hand-held still cameras, to shoot still photos, and they can also be used as webcams. When sold, they usually came with some kind of mounting device that could hold them rigidly for webcam use. There are lots of similar cameras. Mercifully, not all of those others have the problems of the sq905, which have led to the present impasse.

If

Quoted 3 lines

you think about non-notebook webcams [1] or about security cams, you'll see that the only way for you to have the flipping information is inside some userspace file.

However, there are obvious differences. For those cameras the question might well come up about how to control the movement of the camera, or, at least, to be aware of which way the camera is pointing. For these, the topic is an inherent property of the particular model of the camera -- or a defect, if someone wants to say so. Since the property is not determined by USB number, it is inherently impossible outside of the module to create a table which contains the needed information.

My intention here was to re-focus attention on the original problem which brought up the current discussion, and to that end the problem must be clearly understood. To have the flipping information inside some userspace file might solve some other problem and may be a generally very good idea. But it will not, can not, and never will be able to solve this problem.

Quoted 6 lines

For example, I have here one video input bttv board with 16 cameras, connected on 4 bttv chips. I have also a Pelco camera, that has just one support socket. Depending on the place you mount it, the camera has to be rotated by 180 degrees. It can also be useful to mount it rotated by 90 degrees.

Good. So one needs external controls and userspace tools. Did I ever say that such things should never be done? No. All I said was that there is a problem, presently on the table, and those kinds of things are not, can not be, never were, and never will be solutions for _this_ problem.

Quoted 10 lines

After mounting the cameras, no matter what apps you are using (a streaming video app to broadcast it via internet, a security app, etc), the rotation
information for that input, on that particular PCI, bus won't change.

As we've standardized the VIDIOC_QUERYCAP bus_info, now, the information for the camera at input 3 of bttv hardware at PCI addres 03:00.3 is unique. It is also unique the same information for a notebook mounted webcam (since the USB bus ID won't change for that devices).

Errrm... Again, the cameras in question here are not notebook mounted webcams.

Quoted 6 lines

So, if we standardize where this information is stored, for example, at /etc/libv4l and/or at ~/.libv4lrc, and let libv4l handle such info, this will be something consistent for all applications using libv4l. Other apps that might not want to trust on libv4l can also read the info at the same file.

Sorry, this will not work here. It may solve some other problem, but not this problem. Or, if one wants to "store" the information there, I don't care, really, but then there needs to be a way to get the information from the module, where it is, and get written into said table, which is where you want it, and this needs to happen every time an sq905 camera gets plugged in -- without pestering the user about the matter every time that such a camera
gets hooked up.

Comparison: I have tossed a coin. Is it going to come up heads? Or tails? It is possible to know which, because the coin has been tossed. It would not even be cheating to look at it, or allow someone who did look at it, to pass to us the information. But we are not going to look, and if someone tells us we will not listen because we have not agreed on what language to use for communication. Instead, we will put a guess about the outcome into a table. We will make it a nice table, which can be revised using nice GUI tools, so it is easy for the user. So if our guess is wrong let the user fix it. Then next time we toss the coin the table entry will be right because either it was right before, or now someone fixed it???

Quoted 2 lines

So, I really think that this should be part of the approach.

I was not even addressing what should or should not be part of the approach to some other problems. My point was that such discussion is not germane to the problem of how to pass on the correct orientation of the sensor, for the sq905 cameras. There are lots of other problems out there to solve. No denying that.

Also an overview is often very helpful. Also trying to visualize what might be needed in the future is helpful. All of this can be extremely helpful. But not everyone can see or imagine every possible thing. For example, it seems that some of the best minds in the business are stunned when confronted with the fact that some manufacturer of cheap electronics in Taiwan has produced a lot of mass-market cameras with the sensors turned upside down, along with some other cameras having the same USB ID with different sensors, which act a bit differently. Clearly, if such a thing happened once it can happen again. So how to deal with such a problem? Something similar or worse will surely come up again.

Quoted 9 lines

I agree that we should have a way to get a hint about a camera rotation, if this is somehow provided by reading at the hardware.

[1] Normal webcams can be mounted on some hardware and have some orientations that would be different than expected. I've seen this before on some PC-based harware where the camera is enclosed inside the clause, like on Automatic Transfer Machines. Also, the user may want to use the camera on an inverted position, to make easy to fix it somewhere.

Quoted 3 lines

Again, these are in fact separate issues. The fundamental issue required for supporting the sq905 cameras is that an agreed-upon method must exist by which precisely two pieces of information can be passed along by the module. Which of the two pieces of information is relevant and needs to be sent along is, alas, only known from within the module. That is the backdrop for the entire discussion. As far as I know, this thread would not exist if that need had not come up. These two pieces of information are, precisely, "frame data requires flipping across a horizontal axis unless camera is upside down" and "frame data requires flipping across a vertical axis unless camera is held up to a mirror"

So, what do these two deep questions, which confound the assembled wisdom of an entire list of Linux video developers, have to do with tables in userspace? None that I can see, unless someone wants to provide a mechanism for the information, having been collected in the module, to be available to the table in userspace.

I'm not saying that userspace tables would solve all problems. I'm just saying that this should be part of the solution.

For sure we need to have a way for retrieving this information for devices like the sq905 cameras, where the information can't be currently be determined by userspace.

In the case of sq905, this information is static, right? If so, IMO, the better approach is to use a flag at the v4l2_input, as already discussed in this thread.

Signature · 4 lines
Cheers,
Mauro Carvalho Chehab
http://linuxtv.org
[email protected]
#30 lore raw
Quoted 79 lines · Mauro Carvalho Chehab
Quoted 77 lines · On Tue, 24 Feb 2009, [email protected] wrote:
Quoted 1 line · Mauro Carvalho Chehab

On Mon, 23 Feb 2009, [email protected] wrote:

<big snip>

Quoted 3 lines

Theodore,

You're considering just one subset of the V4L usages: notebook webcams.

Actually, the sq905 cameras are not "notebook webcams." They are cheap, consumer entry level dual-mode cameras. They can be used as hand-held still cameras, to shoot still photos, and they can also be used as webcams. When sold, they usually came with some kind of mounting device that could hold them rigidly for webcam use. There are lots of similar cameras. Mercifully, not all of those others have the problems of the sq905, which have led to the present impasse.

If

Quoted 3 lines

you think about non-notebook webcams [1] or about security cams, you'll see that the only way for you to have the flipping information is inside some userspace file.

However, there are obvious differences. For those cameras the question might well come up about how to control the movement of the camera, or, at least, to be aware of which way the camera is pointing. For these, the topic is an inherent property of the particular model of the camera -- or a defect, if someone wants to say so. Since the property is not determined by USB number, it is inherently impossible outside of the module to create a table which contains the needed information.

My intention here was to re-focus attention on the original problem which brought up the current discussion, and to that end the problem must be clearly understood. To have the flipping information inside some userspace file might solve some other problem and may be a generally very good idea. But it will not, can not, and never will be able to solve this problem.

Quoted 6 lines

For example, I have here one video input bttv board with 16 cameras, connected on 4 bttv chips. I have also a Pelco camera, that has just one support socket. Depending on the place you mount it, the camera has to be rotated by 180 degrees. It can also be useful to mount it rotated by 90 degrees.

Good. So one needs external controls and userspace tools. Did I ever say that such things should never be done? No. All I said was that there is a problem, presently on the table, and those kinds of things are not, can not be, never were, and never will be solutions for _this_ problem.

Quoted 10 lines

After mounting the cameras, no matter what apps you are using (a streaming video app to broadcast it via internet, a security app, etc), the rotation
information for that input, on that particular PCI, bus won't change.

As we've standardized the VIDIOC_QUERYCAP bus_info, now, the information for the camera at input 3 of bttv hardware at PCI addres 03:00.3 is unique. It is also unique the same information for a notebook mounted webcam (since the USB bus ID won't change for that devices).

Errrm... Again, the cameras in question here are not notebook mounted webcams.

Quoted 6 lines

So, if we standardize where this information is stored, for example, at /etc/libv4l and/or at ~/.libv4lrc, and let libv4l handle such info, this will be something consistent for all applications using libv4l. Other apps that might not want to trust on libv4l can also read the info at the same file.

Sorry, this will not work here. It may solve some other problem, but not this problem. Or, if one wants to "store" the information there, I don't care, really, but then there needs to be a way to get the information from the module, where it is, and get written into said table, which is where you want it, and this needs to happen every time an sq905 camera gets plugged in -- without pestering the user about the matter every time that such a camera gets hooked up.

Comparison: I have tossed a coin. Is it going to come up heads? Or tails? It is possible to know which, because the coin has been tossed. It would not even be cheating to look at it, or allow someone who did look at it, to pass to us the information. But we are not going to look, and if someone tells us we will not listen because we have not agreed on what language to use for communication. Instead, we will put a guess about the outcome into a table. We will make it a nice table, which can be revised using nice GUI tools, so it is easy for the user. So if our guess is wrong let the user fix it. Then next time we toss the coin the table entry will be right because either it was right before, or now someone fixed it???

Quoted 2 lines

So, I really think that this should be part of the approach.

I was not even addressing what should or should not be part of the approach to some other problems. My point was that such discussion is not germane to the problem of how to pass on the correct orientation of the sensor, for the sq905 cameras. There are lots of other problems out there to solve. No denying that.

Also an overview is often very helpful. Also trying to visualize what might be needed in the future is helpful. All of this can be extremely helpful. But not everyone can see or imagine every possible thing. For example, it seems that some of the best minds in the business are stunned when confronted with the fact that some manufacturer of cheap electronics in Taiwan has produced a lot of mass-market cameras with the sensors turned upside down, along with some other cameras having the same USB ID with different sensors, which act a bit differently. Clearly, if such a thing happened once it can happen again. So how to deal with such a problem? Something similar or worse will surely come up again.

Quoted 9 lines

I agree that we should have a way to get a hint about a camera rotation, if this is somehow provided by reading at the hardware.

[1] Normal webcams can be mounted on some hardware and have some orientations that would be different than expected. I've seen this before on some PC-based harware where the camera is enclosed inside the clause, like on Automatic Transfer Machines. Also, the user may want to use the camera on an inverted position, to make easy to fix it somewhere.

Quoted 5 lines
Quoted 3 lines

Again, these are in fact separate issues. The fundamental issue required for supporting the sq905 cameras is that an agreed-upon method must exist by which precisely two pieces of information can be passed along by the module. Which of the two pieces of information is relevant and needs to be sent along is, alas, only known from within the module. That is the backdrop for the entire discussion. As far as I know, this thread would not exist if that need had not come up. These two pieces of information are, precisely, "frame data requires flipping across a horizontal axis unless camera is upside down" and "frame data requires flipping across a vertical axis unless camera is held up to a mirror"

So, what do these two deep questions, which confound the assembled wisdom of an entire list of Linux video developers, have to do with tables in userspace? None that I can see, unless someone wants to provide a mechanism for the information, having been collected in the module, to be available to the table in userspace.

I'm not saying that userspace tables would solve all problems. I'm just saying that this should be part of the solution.

I meant what I said. I do not see how that tables in userspace carry any relevance at all to this particular problem. Perhaps I am dense. Might tables in userspace help with the solution of some other problems? Well, yes. But then we are discussing those problems, not this one.

Quoted 2 lines

For sure we need to have a way for retrieving this information for devices like the sq905 cameras, where the information can't be currently be determined by userspace.

Yes, indeed. Except for just one word, "currently." It does not fit here. The matter is not one of present inability, just because we are not clever enough. The logic of the situation is inexorable. Well, in part I take that back. If the operating system were fundamentally redesigned, making it possible to run a video device completely in userspace, then and only then the needed information could be determined by userspace. But I do not expect that will happen very soon.

Quoted 2 lines

In the case of sq905, this information is static, right?

Yes. Exactly. You get the data the way it came out. That's it. No choices about that are possible. Precisely how the data came out depends only on the camera. An inquiry to the camera before streaming is started can provide the relevant information.

If so, IMO, the

Quoted 1 line

better approach is to use a flag at the v4l2_input, as already discussed in this thread.

OK.

Theodore Kilgore

#31 lore raw
Quoted 5 lines · On Tue, 24 Feb 2009 20:12:00 -0600 (CST) [email protected] wrote:
Quoted 3 lines

For sure we need to have a way for retrieving this information for devices like the sq905 cameras, where the information can't be currently be determined by userspace.

Yes, indeed. Except for just one word, "currently." It does not fit here.

Yes, it fits, since, after implementing the API, the userspace will have this information by using the agreed API.

Quoted 5 lines
Quoted 3 lines

If so, IMO, the
better approach is to use a flag at the v4l2_input, as already discussed in this thread.

OK.

As it seems to be a consensus that the better is to use a flag inside v4l2_input, could you please provide us an RFC patch to implement it and update the V4L2 spec?

Cheers,
Mauro

#32 lore raw
Quoted 16 lines · Mauro Carvalho Chehab
Quoted 5 lines · On Tue, 24 Feb 2009 20:12:00 -0600 (CST) [email protected] wrote:
Quoted 3 lines

For sure we need to have a way for retrieving this information for devices like the sq905 cameras, where the information can't be currently be determined by userspace.

Yes, indeed. Except for just one word, "currently." It does not fit here.

Yes, it fits, since, after implementing the API, the userspace will have this information by using the agreed API.

Quoted 4 lines
Quoted 3 lines

If so, IMO, the
better approach is to use a flag at the v4l2_input, as already discussed in this thread.

Quoted 5 lines
Quoted 1 line

OK.

As it seems to be a consensus that the better is to use a flag inside v4l2_input, could you please provide us an RFC patch to implement it and update the V4L2 spec?

With respect, I think that it would be better to consult with some of the other people who are most closely involved in the issue. This would include Jean-Francois (one obviously does not want to come up with some scheme which would interfere with other camera drivers, and he has a good overview on that) and Hans de Goede (v4l will use the new method) and Adam Baker. I will be happy to participate in discussions with them and work something up.

Thank you very much for helping to come to an apparent consensus about what kind of characteristics that the solution has to have. I appreciate your attention to the issue, and at this point I hope for a good and a fairly quick resolution.

Theodore Kilgore

#33 lore raw
In reply to Mauro Carvalho Chehab
Quoted 2 lines
Quoted 2 lines

Also an overview is often very helpful. Also trying to visualize what might be needed in the future is helpful. All of this can be extremely helpful. But not everyone can see or imagine every possible thing. For example, it seems that some of the best minds in the business are stunned when confronted with the fact that some manufacturer of cheap electronics in Taiwan has produced a lot of mass-market cameras with the sensors turned upside down, along with some other cameras having the same USB ID with different sensors, which act a bit differently. Clearly, if such a thing happened once it can happen again. So how to deal with such a problem?

Actually, this happens and is happening!

Just step back a get an other view.

These consumer products are manly produced for the Windoz audience.

After introduction of Win XP the consumer where told that USB device will run out of the box in Win XP, which is sometimes true, but .....

But on all (Windowz) Webcams (are Linux Webcams available?) I buy, I find a sticker which tells me to first insert the driver CD before connecting the cam to the PC. When you do, like instructed, your cam works like you expected!

Evan the USB ID is the same like the other webcam from the other vendor, you are (more or less) forced to install the driver from this particular vendor, you get a new driver! Doesn't matter if the sensor is mounted upside down, the "new" driver takes care about this. So, it looks like the cam in the Windowz World just works because you were forced to install the driver from the CD.

So I guess the Windoz diver just knows more then the USB ID.

In the Linux World most of the drive are re-engineered, we don't know how to detect how the sensor is mounted, do we?

Actually, what I try to say, is that only the cam can know how the sensor is mounted. Thus, the kernel module has to provide this information to user space (by query the hardware).

The "pivot" is an other thing.

Thomas

#34 lore raw
Quoted 6 lines · Thomas Kaiser
Quoted 2 lines
Quoted 2 lines

Also an overview is often very helpful. Also trying to visualize what might be needed in the future is helpful. All of this can be extremely helpful. But not everyone can see or imagine every possible thing. For example, it seems that some of the best minds in the business are stunned when confronted with the fact that some manufacturer of cheap electronics in Taiwan has produced a lot of mass-market cameras with the sensors turned upside down, along with some other cameras having the same USB ID with different sensors, which act a bit differently. Clearly, if such a thing happened once it can happen again. So how to deal with such a problem?

Actually, this happens and is happening!

Just step back a get an other view.

I had plenty of other views. SQ905, SQ905C, MR97310, SN9C2028, and one or two more. And, oh yes, the GT8290 chip on which Grandtech went broke, which was intended for mass storage still cameras but had a 32-bit CBW instead of a 31-bit CBW (off by one error and nobody caught it).

Quoted 2 lines

These consumer products are manly produced for the Windoz audience.

Very true. Usually, it even says on the package that it will not work on Mac. But since Linux is unmentionable they did not say anything about it. So we make it work anyway.

Quoted 2 lines

After introduction of Win XP the consumer where told that USB device will run out of the box in Win XP,

Not all devices. Perhaps what they really meant is now you don't need to install two drivers if you have two Mass Storage Transparent Scsi Bulk Transport flash drives. To that extent, it seems that they were truthful. There are lots of proprietary devices out there for which AFAICT the drivers are still not included in Windows. Some of these are unfinished projects of mine, too, such as the JL2005C cameras. The big bugbear with those is the compression algorithm. It is a horror.

Quoted 3 lines

which is sometimes true, but .....

But on all (Windowz) Webcams (are Linux Webcams available?) I buy, I find a sticker which tells me to first insert the driver CD before connecting the cam to the PC. When you do, like instructed, your cam works like you expected!

Of course. Hardware will not work without a driver.

Quoted 2 lines

Evan the USB ID is the same like the other webcam from the other vendor, you are (more or less) forced to install the driver from this particular vendor, you get a new driver!

Not true. Windows, even back in the days of Windows 98, searched for the Vendor:Product number to look for the driver. There were several places to search for the number. For example, the INF files. And I have a box full of SQ905 cameras for which I will personally guarantee that every one of them will work on Windows 98 with the Windows driver from the CD that came with any other one. Now, if the picture is always facing in the right direction, that is another question, naturally.

Doesn't matter if the sensor is mounted upside down,

Quoted 1 line

the "new" driver takes care about this. So, it looks like the cam in the Windowz World just works because you were forced to install the driver from the CD.

And while we are on that topic, I could definitely assert that not always did the "manufacturer" get this right, either. I, recall, developed the original SQ905 stillcam driver for Linux. I got letters from users and sample photos, too, where the photos came out of the camera and lettering on a sign visible in the photo was bass ackwards. They were cheap cameras, and the people who sold some of them were just a little bit sloppy. Their virtue, as cheap cameras go, was comparatively good optics in some of them. I have seen lots of cameras about the same price, which had much more features and much worse pictures.

Quoted 2 lines

So I guess the Windoz diver just knows more then the USB ID.

No, not really. See above.

Quoted 2 lines

In the Linux World most of the drive are re-engineered, we don't know how to detect how the sensor is mounted, do we?

Well, yes, we do. And that is what this discussion was about. How to use that knowledge constructively while writing a kernel driver.

Quoted 2 lines

Actually, what I try to say, is that only the cam can know how the sensor is mounted. Thus, the kernel module has to provide this information to user space (by query the hardware).

Well, that is more like it. Yes, one does have to ask the camera. But the camera will tell its answer truthfully. And this is before any streaming and image processing has started, too. What a deal.

Quoted 2 lines

The "pivot" is an other thing.

Very true. And worth paying attention to. But it is not the same issue.

Quoted 3 lines

Thomas

Theodore Kilgore

#35 lore raw
Quoted 1 line
Quoted 1 line

Actually, this happens and is happening!

I got OT. I just needed to vent!

Sorry for the spam.

For the sensor mounting, I think the cam knows how the sensor is mounted and therefor the driver knows (driver -> first abstraction layer between software and hardware). Therefor the drive has to report the sensor orientation to user space (read only).

Thomas

PS: May be, I am OT all ready.

#36 lore raw
In reply to Mauro Carvalho Chehab

<really big snip>

Quoted 8 lines
Quoted 1 line

So, what do these two deep questions, which confound the assembled wisdom of an entire list of Linux video developers, have to do with tables in userspace? None that I can see, unless someone wants to provide a mechanism for the information, having been collected in the module, to be available to the table in userspace.

I'm not saying that userspace tables would solve all problems. I'm just saying that this should be part of the solution.

For sure we need to have a way for retrieving this information for devices like the sq905 cameras, where the information can't be currently be determined by userspace.

In the case of sq905, this information is static, right? If so, IMO, the better approach is to use a flag at the v4l2_input, as already discussed in this thread.

Yes we all seem to agree on this. Adam, since you started this thread can you write a small RFC with that solution worked out with proposed videodev2.h changes?

Note that I'm only talking about input flags for the orientation problem and not the pivotting problem. I think the pivotting problem may need some more discussion. But since we all seem to be in agreement wrt to orientation problem and specifically the sq905 problem, lets do one more RFC, then everyone does a +1 to that and we move forward with this as a solution for the orientation problem.

Regards,

Hans

p.s.

For the pivotting problem I'm tending towards a special control class which contains read-only controls which are really camera properties. This will allow us to cope with any granularity of pivoting sensors. This could then also be used for in example aperture. But lets start a new thread for that.

#37 lore raw

PATCH 1/1 libv4lconvert support for SQ905C decompression

Hans,

Below is a patch for libv4lconvert, to support the decompression used by the SQ905C cameras (0x2770:0x905C) and some other related cameras. There is at the moment no support module for these cameras in streaming mode, but I intend to submit one.

This contribution was created in whole by me, based upon code in libgphoto2 which was created in whole by me, and which was licensed for libgphoto2 under the LGPL license.

The content of the file sq905c.patch follows -----------------------------------------------------------------------

Makefile +1 -1
diff -uprN libv4lconvert-old/Makefile libv4lconvert-new/Makefile
@@ -12,7 +12,7 @@ endif
 
 CONVERT_OBJS  = libv4lconvert.o tinyjpeg.o sn9c10x.o sn9c20x.o pac207.o \
 		mr97310a.o flip.o crop.o jidctflt.o spca561-decompress.o \
-		rgbyuv.o spca501.o bayer.o
+		rgbyuv.o spca501.o sq905c.o bayer.o
 TARGETS       = $(CONVERT_LIB) libv4lconvert.pc
 INCLUDES      = ../include/libv4lconvert.h
 
libv4lconvert-priv.h +7 -0
diff -uprN libv4lconvert-old/libv4lconvert-priv.h libv4lconvert-new/libv4lconvert-priv.h
@@ -47,6 +47,10 @@
 #define V4L2_PIX_FMT_MR97310A v4l2_fourcc('M','3','1','0')
 #endif
 
+#ifndef V4L2_PIX_FMT_SQ905C
+#define V4L2_PIX_FMT_SQ905C v4l2_fourcc('9','0','5','C')
+#endif
+
 #ifndef V4L2_PIX_FMT_PJPG
 #define V4L2_PIX_FMT_PJPG v4l2_fourcc('P', 'J', 'P', 'G')
 #endif
@@ -180,6 +184,9 @@ void v4lconvert_decode_pac207(const unsi
 void v4lconvert_decode_mr97310a(const unsigned char *src, unsigned char *dst,
   int width, int height);
 
+void v4lconvert_decode_sq905c(const unsigned char *src, unsigned char *dst,
+  int width, int height);
+
 void v4lconvert_bayer_to_rgb24(const unsigned char *bayer,
   unsigned char *rgb, int width, int height, unsigned int pixfmt);
 
libv4lconvert.c +6 -0
diff -uprN libv4lconvert-old/libv4lconvert.c libv4lconvert-new/libv4lconvert.c
@@ -61,6 +61,7 @@ static const struct v4lconvert_pixfmt su
   { V4L2_PIX_FMT_SN9C10X,      V4LCONVERT_COMPRESSED },
   { V4L2_PIX_FMT_PAC207,       V4LCONVERT_COMPRESSED },
   { V4L2_PIX_FMT_MR97310A,     V4LCONVERT_COMPRESSED },
+  { V4L2_PIX_FMT_SQ905C,       V4LCONVERT_COMPRESSED },
   { V4L2_PIX_FMT_PJPG,         V4LCONVERT_COMPRESSED },
 };
 
@@ -608,6 +609,7 @@ static int v4lconvert_convert_pixfmt(str
     case V4L2_PIX_FMT_SN9C10X:
     case V4L2_PIX_FMT_PAC207:
     case V4L2_PIX_FMT_MR97310A:
+    case V4L2_PIX_FMT_SQ905C:
     {
       unsigned char *tmpbuf;
 
@@ -633,6 +635,10 @@ static int v4lconvert_convert_pixfmt(str
 	  v4lconvert_decode_mr97310a(src, tmpbuf, width, height);
 	  src_pix_fmt = V4L2_PIX_FMT_SBGGR8;
 	  break;
+	case V4L2_PIX_FMT_SQ905C:
+	  v4lconvert_decode_sq905c(src, tmpbuf, width, height);
+	  src_pix_fmt = V4L2_PIX_FMT_SRGGB8;
+	  break;
       }
       src = tmpbuf;
       /* Deliberate fall through to raw bayer fmt code! */
sq905c.c +191 -0

191 lines not on this page · raw message