Adding a control for Sensor Orientation
Hi all,
Hans Verkuil put forward a convincing argument that sensor orientation shouldn't be part of the buffer flags as then it would be unavailable to clients that use read() so it looks like implementing a read only control is the only appropriate option.
It seems that Sensor Orientation is an attribute that many cameras may want to expose so it shouldn't be a private control. Olivier Lorin's example patch created a new CAMERA_PROPERTIES class. I'm not sure that a new class is really justified so would like to hear other views on where the control should live (and also if everyone is happy with Hans Verkuil's suggested name of SENSOR_ORIENTATION which I prefer to Olivier Lorin's SENSOR_UPSIDE_DOWN as we want to represent HFLIP and VFLIP as well as upside down (which as currently implemented means 180 degree rotation.))
Assuming that it is considered inappropriate to add a new control as an "Old-style 'user' control" then it is also, I presume, necessary to extend gspca to support VIDIOC_G_EXT_CTRLS as at the moment it requires all control access to use VIDIOC_G_CTRL. Would doing this conflict with anything anyone else may be working on such as conversion to use v4l2_device.
Thoughts please.
Adam Baker
Quoted 17 lines · On Saturday 14 February 2009 21:48:51 Adam Baker wrote:
Hi all,
Hans Verkuil put forward a convincing argument that sensor orientation shouldn't be part of the buffer flags as then it would be unavailable to clients that use
read()so it looks like implementing a read only control is the only appropriate option.It seems that Sensor Orientation is an attribute that many cameras may want to expose so it shouldn't be a private control. Olivier Lorin's example patch created a new
CAMERA_PROPERTIESclass. I'm not sure that a new class is really justified so would like to hear other views on where the control should live (and also if everyone is happy with Hans Verkuil's suggested name ofSENSOR_ORIENTATIONwhich I prefer to Olivier Lorin'sSENSOR_UPSIDE_DOWNas we want to represent HFLIP and VFLIP as well as upside down (which as currently implemented means 180 degree rotation.))
Quoted 8 lines
Assuming that it is considered inappropriate to add a new control as an "Old-style 'user' control" then it is also, I presume, necessary to extend gspca to support
VIDIOC_G_EXT_CTRLSas at the moment it requires all control access to useVIDIOC_G_CTRL. Would doing this conflict with anything anyone else may be working on such as conversion to usev4l2_device.Thoughts please.
This is definitely a camera control, so I guess you need to add this new camera control and the g_ext_ctrls ioctl. It's a bit overkill, and I have lots of ideas on how to drastically improve control handling in drivers, but that's a few months in the future once all drivers are converted to v4l2_device. Hmm, if you are working in gspca anyway, are you interested in adding v4l2_device to it? It's trivial for that driver.
Regards,
Hans
Signature · 1 line
Hans Verkuil - video4linux developer - sponsored by TANDBERG
Quoted 3 lines · Adam Baker
Hi all,
Hans Verkuil put forward a convincing argument that sensor orientation shouldn't be part of the buffer flags as then it would be unavailable to clients that use
read()
Yes and this is a bogus argument, clients using read also do not get things like timestamps, and vital information like which field is in the read buffer when dealing with interleaved sources. read() is a simple interface for simple applications. Given that the only user of these flags will likely be libv4l I *strongly* object to having this info in some control, it is not a control, it is a per frame (on some cams) information about how to interpret that frame, the buffer flags is a very logical place, *the* logical place even for this!
The fact that there is no way to transport metadata about a frame like flags, but also timestamp and field! Is a problem with the read interface in general, iow read() is broken wrt to this. If people care add some ioctl or something which users of read() can use to get the buffer metadata from the last read() buffer, stuffing buffer metadata in a control (barf), because of read() brokenness is a very *bad* idea, and won't work in general due to synchronization problems.
Doing this as a control will be a pain to implement both at the driver level, see the discussion this is causing, and in libv4l. For libv4l this will basicly mean polling the control. And hello polling is lame and something from the 1980-ies.
Please just make this a buffer flag.
Thank you,
Hans
Quoted 24 lines · On Saturday 14 February 2009 22:55:39 Hans de Goede wrote:
Quoted 5 lines · Adam Baker
Hi all,
Hans Verkuil put forward a convincing argument that sensor orientation shouldn't be part of the buffer flags as then it would be unavailable to clients that use
read()Yes and this is a bogus argument, clients using read also do not get things like timestamps, and vital information like which field is in the read buffer when dealing with interleaved sources.
read()is a simple interface for simple applications. Given that the only user of these flags will likely be libv4l I *strongly* object to having this info in some control, it is not a control, it is a per frame (on some cams) information about how to interpret that frame, the buffer flags is a very logical place, *the* logical place even for this!The fact that there is no way to transport metadata about a frame like flags, but also timestamp and field! Is a problem with the read interface in general, iow
read()is broken wrt to this. If people care add some ioctl or something which users ofread()can use to get the buffer metadata from the lastread()buffer, stuffing buffer metadata in a control (barf), because ofread()brokenness is a very *bad* idea, and won't work in general due to synchronization problems.
Quoted 6 lines
Doing this as a control will be a pain to implement both at the driver level, see the discussion this is causing, and in libv4l. For libv4l this will basicly mean polling the control. And hello polling is lame and something from the 1980-ies.
Please just make this a buffer flag.
OK, make it a buffer flag. I've got to agree that it makes more sense to do it that way.
Regards,
Hans
Signature · 1 line
Hans Verkuil - video4linux developer - sponsored by TANDBERG
Quoted 30 lines · Hans Verkuil
Quoted 29 lines · On Saturday 14 February 2009 22:55:39 Hans de Goede wrote:
Quoted 5 lines · Adam Baker
Hi all,
Hans Verkuil put forward a convincing argument that sensor orientation shouldn't be part of the buffer flags as then it would be unavailable to clients that use
read()Yes and this is a bogus argument, clients using read also do not get things like timestamps, and vital information like which field is in the read buffer when dealing with interleaved sources.
read()is a simple interface for simple applications. Given that the only user of these flags will likely be libv4l I *strongly* object to having this info in some control, it is not a control, it is a per frame (on some cams) information about how to interpret that frame, the buffer flags is a very logical place, *the* logical place even for this!The fact that there is no way to transport metadata about a frame like flags, but also timestamp and field! Is a problem with the read interface in general, iow
read()is broken wrt to this. If people care add some ioctl or something which users ofread()can use to get the buffer metadata from the lastread()buffer, stuffing buffer metadata in a control (barf), because ofread()brokenness is a very *bad* idea, and won't work in general due to synchronization problems.Doing this as a control will be a pain to implement both at the driver level, see the discussion this is causing, and in libv4l. For libv4l this will basicly mean polling the control. And hello polling is lame and something from the 1980-ies.
Quoted 12 lines
Quoted 1 line
Please just make this a buffer flag.
OK, make it a buffer flag. I've got to agree that it makes more sense to do it that way.
Regards,
Hans
Signature · 1 line
Hans Verkuil - video4linux developer - sponsored by TANDBERG
Let me take a moment to remind everyone what the problem is that brought this discussion up. Adam Baker and I are working on a driver for a certain camera. Or, better stated, for a set of various cameras, which all have the same USB Vendor:Product number. Various cameras which all have this ID have different capabilities and need different treatment of the frame data.
The most particular problem is that some of the cameras require byte reversal of the frame data string, which would rotate the image 180 degrees around its center. Others of these cameras require reversal of the horizontal lines in the image (vertical 180 degree rotation of the image across a horizontal axis).
The point is, one can not tell from the Vendor:Product number which of these actions is required. However, one *is* able to tell immediately after the camera is initialized, which of these actions is required. Namely, one reads and parses the response to the first USB command sent to the camera.
So, for us (Adam and me) the question is simply to know how everyone will agree that the information about the image orientation can be sent from the module to V4L. When this issue is resolved, we can finish writing the sq905 camera driver. From this rather narrow point of view, the issue is not which method ought to be adopted. Rather, the issue is that no method has been adopted. It is rather difficult to write module code which will obey a non-existent standard.
Theodore Kilgore
Quoted 47 lines · [email protected] wrote:
Quoted 30 lines · Hans Verkuil
Quoted 29 lines · On Saturday 14 February 2009 22:55:39 Hans de Goede wrote:
Quoted 5 lines · Adam Baker
Hi all,
Hans Verkuil put forward a convincing argument that sensor orientation shouldn't be part of the buffer flags as then it would be unavailable to clients that use
read()Yes and this is a bogus argument, clients using read also do not get things like timestamps, and vital information like which field is in the read buffer when dealing with interleaved sources.
read()is a simple interface for simple applications. Given that the only user of these flags will likely be libv4l I *strongly* object to having this info in some control, it is not a control, it is a per frame (on some cams) information about how to interpret that frame, the buffer flags is a very logical place, *the* logical place even for this!The fact that there is no way to transport metadata about a frame like flags, but also timestamp and field! Is a problem with the read interface in general, iow
read()is broken wrt to this. If people care add some ioctl or something which users ofread()can use to get the buffer metadata from the lastread()buffer, stuffing buffer metadata in a control (barf), because ofread()brokenness is a very *bad* idea, and won't work in general due to synchronization problems.Doing this as a control will be a pain to implement both at the driver level, see the discussion this is causing, and in libv4l. For libv4l this will basicly mean polling the control. And hello polling is lame and something from the 1980-ies.
Quoted 12 lines
Quoted 1 line
Please just make this a buffer flag.
OK, make it a buffer flag. I've got to agree that it makes more sense to do it that way.
Regards,
Hans
Signature · 1 line
Hans Verkuil - video4linux developer - sponsored by TANDBERG
Quoted 8 lines
Let me take a moment to remind everyone what the problem is that brought this discussion up. Adam Baker and I are working on a driver for a certain camera. Or, better stated, for a set of various cameras, which all have the same USB Vendor:Product number. Various cameras which all have this ID have different capabilities and need different treatment of the frame data.
The most particular problem is that some of the cameras require byte reversal of the frame data string, which would rotate the image 180 degrees around its center. Others of these cameras require reversal of the horizontal lines in the image (vertical 180 degree rotation of the image across a horizontal axis).
The point is, one can not tell from the Vendor:Product number which of these actions is required. However, one *is* able to tell immediately after the camera is initialized, which of these actions is required. Namely, one reads and parses the response to the first USB command sent to the camera.
So, for us (Adam and me) the question is simply to know how everyone will agree that the information about the image orientation can be sent from the module to V4L. When this issue is resolved, we can finish writing the sq905 camera driver. From this rather narrow point of view, the issue is not which method ought to be adopted. Rather, the issue is that no method has been adopted. It is rather difficult to write module code which will obey a non-existent standard.
Ack, but the problem later was extended by the fact that it turns out some cams have a rotation detection (gravity direction) switch, which means you can flip the cam on its socket while streaming, and then the cam will tell you its rotation has changed, that makes this a per frame property rather then a static property of the cam. Which lead to this discussion, but we (the 2 Hans 's) agree now that using the flags field in the buffer struct is the best way forward. So there is a standard now, simply add 2 buffer flags to videodev2.h, one for content is h-flipped and one for content is v-flipped and you are done.
Regards,
Hans
Quoted 85 lines · On Sunday 15 February 2009 10:08:04 Hans de Goede wrote:
Quoted 66 lines · [email protected] wrote:
Quoted 39 lines · Hans Verkuil
Quoted 26 lines · On Saturday 14 February 2009 22:55:39 Hans de Goede wrote:
Quoted 5 lines · Adam Baker
Hi all,
Hans Verkuil put forward a convincing argument that sensor orientation shouldn't be part of the buffer flags as then it would be unavailable to clients that use
read()Yes and this is a bogus argument, clients using read also do not get things like timestamps, and vital information like which field is in the read buffer when dealing with interleaved sources.
read()is a simple interface for simple applications. Given that the only user of these flags will likely be libv4l I *strongly* object to having this info in some control, it is not a control, it is a per frame (on some cams) information about how to interpret that frame, the buffer flags is a very
logical place, *the* logical place even for this!The fact that there is no way to transport metadata about a frame like flags, but also timestamp and field! Is a problem with the read interface
in general, iowread()is broken wrt to this. If people care add some ioctl or something which users ofread()can use to get the buffer metadata from the lastread()buffer, stuffing buffer metadata in a control (barf), because ofread()brokenness is a very *bad* idea, and won't work in general due to synchronization problems.Quoted 7 lines
Doing this as a control will be a pain to implement both at the driver level, see the discussion this is causing, and in libv4l. For libv4l this
will basicly mean polling the control. And hello polling is lame and something from the 1980-ies.Please just make this a buffer flag.
OK, make it a buffer flag. I've got to agree that it makes more sense to do
it that way.Quoted 6 lines
Regards,
Hans
--
Hans Verkuil - video4linux developer - sponsored by TANDBERGLet me take a moment to remind everyone what the problem is that brought this discussion up. Adam Baker and I are working on a driver for a certain camera. Or, better stated, for a set of various cameras, which all have the same USB Vendor:Product number. Various cameras which all have this ID have different capabilities and need different treatment of the frame data.
The most particular problem is that some of the cameras require byte reversal of the frame data string, which would rotate the image 180 degrees around its center. Others of these cameras require reversal of the horizontal lines in the image (vertical 180 degree rotation of the image across a horizontal axis).
The point is, one can not tell from the Vendor:Product number which of these actions is required. However, one *is* able to tell immediately after the camera is initialized, which of these actions is required. Namely, one reads and parses the response to the first USB command sent to the camera.
Quoted 7 lines
So, for us (Adam and me) the question is simply to know how everyone will agree that the information about the image orientation can be sent from the module to V4L. When this issue is resolved, we can finish writing the sq905 camera driver. From this rather narrow point of view, the issue is not which method ought to be adopted. Rather, the issue is that no method has been adopted. It is rather difficult to write module code which will obey a non-existent standard.
Ack, but the problem later was extended by the fact that it turns out some cams have a rotation detection (gravity direction) switch, which means you can flip the cam on its socket while streaming, and then the cam will tell you its rotation has changed, that makes this a per frame property rather then a static property of the cam. Which lead to this discussion, but we (the 2 Hans 's) agree now that using the flags field in the buffer struct is the best way forward. So there is a standard now, simply add 2 buffer flags to
videodev2.h, one for content is h-flipped and one for content is v-flipped and you are done.
Quoted 3 lines
Regards,
Hans
I think we should also be able to detect 90 and 270 degree rotations. Or at the very least prepare for it. It's a safe bet to assume that webcams will arrive that can detect portrait vs landscape orientation.
Regards,
Hans
Signature · 1 line
Hans Verkuil - video4linux developer - sponsored by TANDBERG
Quoted 82 lines · Hans Verkuil
Quoted 81 lines · On Sunday 15 February 2009 10:08:04 Hans de Goede wrote:
Quoted 63 lines · [email protected] wrote:
Quoted 37 lines · Hans Verkuil
Quoted 25 lines · On Saturday 14 February 2009 22:55:39 Hans de Goede wrote:
Quoted 5 lines · Adam Baker
Hi all,
Hans Verkuil put forward a convincing argument that sensor orientation shouldn't be part of the buffer flags as then it would be unavailable to clients that use
read()Yes and this is a bogus argument, clients using read also do not get things like timestamps, and vital information like which field is in the read buffer when dealing with interleaved sources.
read()is a simple interface for simple applications. Given that the only user of these flags will likely be libv4l I *strongly* object to having this info in some control, it is not a control, it is a per frame (on some cams) information about how to interpret that frame, the buffer flags is a very
logical place, *the* logical place even for this!The fact that there is no way to transport metadata about a frame like flags, but also timestamp and field! Is a problem with the read interface
in general, iowread()is broken wrt to this. If people care add some ioctl or something which users ofread()can use to get the buffer metadata from the lastread()buffer, stuffing buffer metadata in a control (barf), because ofread()brokenness is a very *bad* idea, and won't work in general due to synchronization problems.Quoted 7 lines
Doing this as a control will be a pain to implement both at the driver level, see the discussion this is causing, and in libv4l. For libv4l this
will basicly mean polling the control. And hello polling is lame and something from the 1980-ies.Please just make this a buffer flag.
OK, make it a buffer flag. I've got to agree that it makes more sense to do
it that way.Quoted 6 lines
Regards,
Hans
--
Hans Verkuil - video4linux developer - sponsored by TANDBERGLet me take a moment to remind everyone what the problem is that brought this discussion up. Adam Baker and I are working on a driver for a certain camera. Or, better stated, for a set of various cameras, which all have the same USB Vendor:Product number. Various cameras which all have this ID have different capabilities and need different treatment of the frame data.
The most particular problem is that some of the cameras require byte reversal of the frame data string, which would rotate the image 180 degrees around its center. Others of these cameras require reversal of the horizontal lines in the image (vertical 180 degree rotation of the image across a horizontal axis).
The point is, one can not tell from the Vendor:Product number which of these actions is required. However, one *is* able to tell immediately after the camera is initialized, which of these actions is required. Namely, one reads and parses the response to the first USB command sent to the camera.
Quoted 7 lines
So, for us (Adam and me) the question is simply to know how everyone will agree that the information about the image orientation can be sent from the module to V4L. When this issue is resolved, we can finish writing the sq905 camera driver. From this rather narrow point of view, the issue is not which method ought to be adopted. Rather, the issue is that no method has been adopted. It is rather difficult to write module code which will obey a non-existent standard.
Ack, but the problem later was extended by the fact that it turns out some cams have a rotation detection (gravity direction) switch, which means you can flip the cam on its socket while streaming, and then the cam will tell you its rotation has changed, that makes this a per frame property rather then a static property of the cam. Which lead to this discussion, but we (the 2 Hans 's) agree now that using the flags field in the buffer struct is the best way forward. So there is a standard now, simply add 2 buffer flags to
videodev2.h, one for content is h-flipped and one for content is v-flipped and you are done.
Quoted 6 lines
Quoted 3 lines
Regards,
Hans
I think we should also be able to detect 90 and 270 degree rotations. Or at the very least prepare for it. It's a safe bet to assume that webcams will arrive that can detect portrait vs landscape orientation.
Handling those (esp on the fly) will be rather hard as width and height then get swapped. So lets worry about those when we need to. We will need an additional flag for those cases anyways.
Regards,
Hans
Quoted 32 lines · Hans de Goede
Quoted 31 lines · Hans Verkuil
Quoted 29 lines · On Sunday 15 February 2009 10:08:04 Hans de Goede wrote:
Quoted 13 lines · [email protected] wrote:
Quoted 6 lines · Hans Verkuil
Quoted 1 line · On Saturday 14 February 2009 22:55:39 Hans de Goede wrote:
Adam Baker wrote:
OK, make it a buffer flag. I've got to agree that it makes more sense to do
it that way.The most particular problem is that some of the cameras require byte reversal of the frame data string, which would rotate the image 180 degrees around its center. Others of these cameras require reversal of the horizontal lines in the image (vertical 180 degree rotation of the image across a horizontal axis).
Quoted 5 lines
The point is, one can not tell from the Vendor:Product number which of these actions is required. However, one *is* able to tell immediately after the camera is initialized, which of these actions is required. Namely, one reads and parses the response to the first USB command sent to the camera.
Ack, but the problem later was extended by the fact that it turns out some cams have a rotation detection (gravity direction) switch, which means you can flip the cam on its socket while streaming, and then the cam will tell you its rotation has changed, that makes this a per frame property rather then a static property of the cam. Which lead to this discussion, but we (the 2 Hans 's) agree now that using the flags field in the buffer struct is the best way forward. So there is a standard now, simply add 2 buffer flags to
videodev2.h, one for content is h-flipped and one for content is v-flipped and you are done.
Quoted 7 lines
Quoted 3 lines
I think we should also be able to detect 90 and 270 degree rotations. Or at the very least prepare for it. It's a safe bet to assume that webcams will arrive that can detect portrait vs landscape orientation.
Handling those (esp on the fly) will be rather hard as width and height then get swapped. So lets worry about those when we need to. We will need an additional flag for those cases anyways.
Why would you need to worry about width and height getting swapped? Meta-data about the frame would indicate it's now in portrait mode vs landscape mode, but the dimentions would be unchanged.
Quoted 35 lines · Trent Piepho
Quoted 34 lines · Hans de Goede
Quoted 33 lines · Hans Verkuil
Quoted 20 lines · On Sunday 15 February 2009 10:08:04 Hans de Goede wrote:
Quoted 13 lines · [email protected] wrote:
Quoted 6 lines · Hans Verkuil
Quoted 1 line · On Saturday 14 February 2009 22:55:39 Hans de Goede wrote:
Adam Baker wrote:
OK, make it a buffer flag. I've got to agree that it makes more sense to do
it that way.The most particular problem is that some of the cameras require byte reversal of the frame data string, which would rotate the image 180 degrees around its center. Others of these cameras require reversal of the horizontal lines in the image (vertical 180 degree rotation of the image across a horizontal axis).
Quoted 5 lines
The point is, one can not tell from the Vendor:Product number which of these actions is required. However, one *is* able to tell immediately after the camera is initialized, which of these actions is required. Namely, one reads and parses the response to the first USB command sent to the camera.
Quoted 9 lines
Ack, but the problem later was extended by the fact that it turns out some cams have a rotation detection (gravity direction) switch, which means you can flip the cam on its socket while streaming, and then the cam will tell you its rotation has changed, that makes this a per frame property rather then a static property of the cam. Which lead to this discussion, but we (the 2 Hans 's) agree now that using the flags field in the buffer struct is the best way forward. So there is a standard now, simply add 2 buffer flags to
videodev2.h, one for content is h-flipped and one for content is v-flipped and you are done.I think we should also be able to detect 90 and 270 degree rotations. Or at the very least prepare for it. It's a safe bet to assume that webcams will arrive that can detect portrait vs landscape orientation.
Quoted 7 lines
Quoted 3 lines
Handling those (esp on the fly) will be rather hard as width and height then get swapped. So lets worry about those when we need to. We will need an additional flag for those cases anyways.
Why would you need to worry about width and height getting swapped? Meta-data about the frame would indicate it's now in portrait mode vs landscape mode, but the dimentions would be unchanged.
Yes, unless ofcourse you want to display a proper picture and not one on its side, when the camera is rotated 90 degrees, so somewere you need to rotate the picture 90 degrees, and the lower down in the stack you do that, the bigger the chance you do not need to duplicate the rotation code in every single app. however the app will mostlikely become unhappy when you start out pushing frames whith a changed width / height.
Regards,
Hans
Quoted 44 lines · Hans de Goede
Quoted 35 lines · Trent Piepho
Quoted 34 lines · Hans de Goede
Quoted 33 lines · Hans Verkuil
Quoted 20 lines · On Sunday 15 February 2009 10:08:04 Hans de Goede wrote:
Quoted 13 lines · [email protected] wrote:
Quoted 6 lines · Hans Verkuil
Quoted 1 line · On Saturday 14 February 2009 22:55:39 Hans de Goede wrote:
Adam Baker wrote:
OK, make it a buffer flag. I've got to agree that it makes more sense to do
it that way.The most particular problem is that some of the cameras require byte reversal of the frame data string, which would rotate the image 180 degrees around its center. Others of these cameras require reversal of the horizontal lines in the image (vertical 180 degree rotation of the image across a horizontal axis).
Quoted 5 lines
The point is, one can not tell from the Vendor:Product number which of these actions is required. However, one *is* able to tell immediately after the camera is initialized, which of these actions is required. Namely, one reads and parses the response to the first USB command sent to the camera.
Quoted 9 lines
Ack, but the problem later was extended by the fact that it turns out some cams have a rotation detection (gravity direction) switch, which means you can flip the cam on its socket while streaming, and then the cam will tell you its rotation has changed, that makes this a per frame property rather then a static property of the cam. Which lead to this discussion, but we (the 2 Hans 's) agree now that using the flags field in the buffer struct is the best way forward. So there is a standard now, simply add 2 buffer flags to
videodev2.h, one for content is h-flipped and one for content is v-flipped and you are done.I think we should also be able to detect 90 and 270 degree rotations. Or at the very least prepare for it. It's a safe bet to assume that webcams will arrive that can detect portrait vs landscape orientation.
Quoted 7 lines
Quoted 3 lines
Handling those (esp on the fly) will be rather hard as width and height then get swapped. So lets worry about those when we need to. We will need an additional flag for those cases anyways.
Why would you need to worry about width and height getting swapped? Meta-data about the frame would indicate it's now in portrait mode vs landscape mode, but the dimentions would be unchanged.
Quoted 6 lines
Yes, unless ofcourse you want to display a proper picture and not one on its side, when the camera is rotated 90 degrees, so somewere you need to rotate the picture 90 degrees, and the lower down in the stack you do that, the bigger the chance you do not need to duplicate the rotation code in every single app. however the app will mostlikely become unhappy when you start out pushing frames whith a changed width / height.
It seems that image rotation, like format conversion, is something that is best done in userspace. It could be done in hardware with opengl or faster software using MMX or SSE based code that can't be used in the kernel.
Quoted 52 lines · Trent Piepho
Quoted 44 lines · Hans de Goede
Quoted 35 lines · Trent Piepho
Quoted 34 lines · Hans de Goede
Quoted 33 lines · Hans Verkuil
Quoted 20 lines · On Sunday 15 February 2009 10:08:04 Hans de Goede wrote:
Quoted 13 lines · [email protected] wrote:
Quoted 6 lines · Hans Verkuil
Quoted 1 line · On Saturday 14 February 2009 22:55:39 Hans de Goede wrote:
Adam Baker wrote:
OK, make it a buffer flag. I've got to agree that it makes more sense to do
it that way.The most particular problem is that some of the cameras require byte reversal of the frame data string, which would rotate the image 180 degrees around its center. Others of these cameras require reversal of the horizontal lines in the image (vertical 180 degree rotation of the image across a horizontal axis).
Quoted 5 lines
The point is, one can not tell from the Vendor:Product number which of these actions is required. However, one *is* able to tell immediately after the camera is initialized, which of these actions is required. Namely, one reads and parses the response to the first USB command sent to the camera.
Quoted 9 lines
Ack, but the problem later was extended by the fact that it turns out some cams have a rotation detection (gravity direction) switch, which means you can flip the cam on its socket while streaming, and then the cam will tell you its rotation has changed, that makes this a per frame property rather then a static property of the cam. Which lead to this discussion, but we (the 2 Hans 's) agree now that using the flags field in the buffer struct is the best way forward. So there is a standard now, simply add 2 buffer flags to
videodev2.h, one for content is h-flipped and one for content is v-flipped and you are done.I think we should also be able to detect 90 and 270 degree rotations. Or at the very least prepare for it. It's a safe bet to assume that webcams will arrive that can detect portrait vs landscape orientation.
Quoted 7 lines
Quoted 3 lines
Handling those (esp on the fly) will be rather hard as width and height then get swapped. So lets worry about those when we need to. We will need an additional flag for those cases anyways.
Why would you need to worry about width and height getting swapped? Meta-data about the frame would indicate it's now in portrait mode vs landscape mode, but the dimentions would be unchanged.
Quoted 6 lines
Yes, unless ofcourse you want to display a proper picture and not one on its side, when the camera is rotated 90 degrees, so somewere you need to rotate the picture 90 degrees, and the lower down in the stack you do that, the bigger the chance you do not need to duplicate the rotation code in every single app. however the app will mostlikely become unhappy when you start out pushing frames whith a changed width / height.
Quoted 4 lines
It seems that image rotation, like format conversion, is something that is best done in userspace. It could be done in hardware with opengl or faster software using MMX or SSE based code that can't be used in the kernel.
My impression is that nobody here disagrees with this. Certainly, I do not. As I understand, there is a general intention to avoid writing new modules which do such things and to try to rewrite old ones. The reasons are, presumably, very similar to the reasons you give.
Therefore,
1. Everyone seems to agree that the kernel module itself is not going to do things like rotate or flip data even if a given supported device always needs that done.
However, this decision has a consequence:
2. Therefore, the module must send the information about what is needed out of the module, to whatever place is going to deal with it. Information which is known to the module but unknown anywere else must be transmitted somehow.
Now there is a further consequence:
3. In view of (1) and (2) there has to be a way agreed upon for the module to pass the relevant information onward.
It is precisely on item 3 that we are stuck right now. There is an immediate need, not a theoretical need but an immediate need. However, there is no agreed-upon method or convention for communication.
Some might argue that it is sufficient to know some ID of the device (USB Vendor:Product number, for example), but it is not:
4. The idea of relying on the USB ID of the supported device to decide what should be done with the frame data is inadequate. Sometimes, preliminary communication with the device is the only possible way to learn what is needed. Again, this is not a theoretical problem. It is an actual problem. A known device exists. Go back to item (3).
There are of course related problems. But it strikes me that the related problems are not so very related at all. As I understand, it is visualized that a camera could be put on a pivot, with control mechanism which would permit various rotations and then the question becomes how to support a camera and to make the stream come out "right" no matter which way the camera is pointed. A far-seeing project design will certainly think of things like that before they happen and will try to anticipate what to do.
Why do I say then that these problems are not related at all to the present problem? Because we are dealing with cameras that always present the data upside-down or mirrored, unless corrective action is taken. So, again, either the module has to do the correction inside itself (and everyone agrees that it should not!) or it has to have a standard protocol to pass that information onward. to be dealt with appropriately. It would seem to me best to separate a problem like this from discussions about tilting or physically rotating the camera.
Theodore Kilgore
Quoted 9 lines · On Sun, 15 Feb 2009 10:29:03 +0100 Hans de Goede <[email protected]> wrote:
Quoted 4 lines
I think we should also be able to detect 90 and 270 degree rotations. Or at the very least prepare for it. It's a safe bet to assume that webcams will arrive that can detect portrait vs landscape orientation.
Handling those (esp on the fly) will be rather hard as width and height then get swapped. So lets worry about those when we need to. We will need an additional flag for those cases anyways.
The camera rotation is something that is already needed, at least on some embedded devices, like those cellular phones whose display changes when you rotate the device.
By looking at the v4l2_buffer struct, we currently have 4 reserved bytes. It
has also one flags field, with several bits not used. I can see 2 possibilities to extend the API:
1) adding V4L2_BUF_FLAG_HFLIP and V4L2_BUF_FLAG_VFLIP flags. This would work for 90, 180 and 270 rotation;
2) spend a few bytes for storing the camera angle. If a camera have a precise sensor, this could be used by some software to reduce the angle movements of a moving camera. However, if we consider this case, we would also need to consider other camera movements.
So, IMO, the two flags could be more appropriate.
Cheers,
Mauro
Hi guys,
Am Sonntag, den 15.02.2009, 19:46 -0600 schrieb
[email protected]:
Quoted 114 lines · Trent Piepho
Quoted 53 lines · Hans de Goede
Quoted 44 lines
Quoted 35 lines · Trent Piepho
Quoted 34 lines · Hans de Goede
Quoted 33 lines · Hans Verkuil
Quoted 20 lines · On Sunday 15 February 2009 10:08:04 Hans de Goede wrote:
Quoted 13 lines · [email protected] wrote:
Quoted 6 lines · Hans Verkuil
Quoted 1 line · On Saturday 14 February 2009 22:55:39 Hans de Goede wrote:
Adam Baker wrote:
OK, make it a buffer flag. I've got to agree that it makes more sense to do
it that way.The most particular problem is that some of the cameras require byte reversal of the frame data string, which would rotate the image 180 degrees around its center. Others of these cameras require reversal of the horizontal lines in the image (vertical 180 degree rotation of the image across a horizontal axis).
Quoted 5 lines
The point is, one can not tell from the Vendor:Product number which of these actions is required. However, one *is* able to tell immediately after the camera is initialized, which of these actions is required. Namely, one reads and parses the response to the first USB command sent to the camera.
Quoted 9 lines
Ack, but the problem later was extended by the fact that it turns out some cams have a rotation detection (gravity direction) switch, which means you can flip the cam on its socket while streaming, and then the cam will tell you its rotation has changed, that makes this a per frame property rather then a static property of the cam. Which lead to this discussion, but we (the 2 Hans 's) agree now that using the flags field in the buffer struct is the best way forward. So there is a standard now, simply add 2 buffer flags to
videodev2.h, one for content is h-flipped and one for content is v-flipped and you are done.I think we should also be able to detect 90 and 270 degree rotations. Or at the very least prepare for it. It's a safe bet to assume that webcams will arrive that can detect portrait vs landscape orientation.
Quoted 7 lines
Quoted 3 lines
Handling those (esp on the fly) will be rather hard as width and height then get swapped. So lets worry about those when we need to. We will need an additional flag for those cases anyways.
Why would you need to worry about width and height getting swapped? Meta-data about the frame would indicate it's now in portrait mode vs landscape mode, but the dimentions would be unchanged.
Quoted 6 lines
Yes, unless ofcourse you want to display a proper picture and not one on its side, when the camera is rotated 90 degrees, so somewere you need to rotate the picture 90 degrees, and the lower down in the stack you do that, the bigger the chance you do not need to duplicate the rotation code in every single app. however the app will mostlikely become unhappy when you start out pushing frames whith a changed width / height.
Quoted 4 lines
It seems that image rotation, like format conversion, is something that is best done in userspace. It could be done in hardware with opengl or faster software using MMX or SSE based code that can't be used in the kernel.
My impression is that nobody here disagrees with this. Certainly, I do not. As I understand, there is a general intention to avoid writing new modules which do such things and to try to rewrite old ones. The reasons are, presumably, very similar to the reasons you give.
Therefore,
1. Everyone seems to agree that the kernel module itself is not going to do things like rotate or flip data even if a given supported device always needs that done.
However, this decision has a consequence:
2. Therefore, the module must send the information about what is needed out of the module, to whatever place is going to deal with it. Information which is known to the module but unknown anywere else must be transmitted somehow.
Now there is a further consequence:
3. In view of (1) and (2) there has to be a way agreed upon for the module to pass the relevant information onward.
It is precisely on item 3 that we are stuck right now. There is an immediate need, not a theoretical need but an immediate need. However, there is no agreed-upon method or convention for communication.
Some might argue that it is sufficient to know some ID of the device (USB Vendor:Product number, for example), but it is not:
4. The idea of relying on the USB ID of the supported device to decide what should be done with the frame data is inadequate. Sometimes, preliminary communication with the device is the only possible way to learn what is needed. Again, this is not a theoretical problem. It is an actual problem. A known device exists. Go back to item (3).
There are of course related problems. But it strikes me that the related problems are not so very related at all. As I understand, it is visualized that a camera could be put on a pivot, with control mechanism which would permit various rotations and then the question becomes how to support a camera and to make the stream come out "right" no matter which way the camera is pointed. A far-seeing project design will certainly think of things like that before they happen and will try to anticipate what to do.
Why do I say then that these problems are not related at all to the present problem? Because we are dealing with cameras that always present the data upside-down or mirrored, unless corrective action is taken. So, again, either the module has to do the correction inside itself (and everyone agrees that it should not!) or it has to have a standard protocol to pass that information onward. to be dealt with appropriately. It would seem to me best to separate a problem like this from discussions about tilting or physically rotating the camera.
Quoted 1 line
Theodore Kilgore
we are still lacking women here.
About cams, I vote for to have them on a separate list until they have something ready for API inclusion.
But to come to a point, we are making some sort of cam history and can't excuse anymore with that others do it already anyway since long.
I recommend to read an already old book about pictures and video.
It is by Paul Virilio and in German titled "Rasender Stillstand".
Original "L'inertie polaire" by editor Christian Bourgois, Paris 1990.
When I joined this sort of stuff here, it was common to point to something eventually related, but seems old fashioned these days ...
Cheers,
Hermann
Quoted 35 lines · On Sun, 15 Feb 2009 [email protected] wrote:
Quoted 19 lines · Trent Piepho
Quoted 11 lines · Hans de Goede
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
I think we should also be able to detect 90 and 270 degree rotations. Or at the very least prepare for it. It's a safe bet to assume that webcams will arrive that can detect portrait vs landscape orientation.
Handling those (esp on the fly) will be rather hard as width and height then get swapped. So lets worry about those when we need to. We will need an additional flag for those cases anyways.
Why would you need to worry about width and height getting swapped? Meta-data about the frame would indicate it's now in portrait mode vs landscape mode, but the dimentions would be unchanged.
Quoted 6 lines
Yes, unless ofcourse you want to display a proper picture and not one on its side, when the camera is rotated 90 degrees, so somewere you need to rotate the picture 90 degrees, and the lower down in the stack you do that, the bigger the chance you do not need to duplicate the rotation code in every single app. however the app will mostlikely become unhappy when you start out pushing frames whith a changed width / height.
Quoted 3 lines
It seems that image rotation, like format conversion, is something that is best done in userspace. It could be done in hardware with opengl or faster software using MMX or SSE based code that can't be used in the kernel.
1. Everyone seems to agree that the kernel module itself is not going to do things like rotate or flip data even if a given supported device always needs that done.
However, this decision has a consequence:
2. Therefore, the module must send the information about what is needed out of the module, to whatever place is going to deal with it. Information which is known to the module but unknown anywere else must be transmitted somehow.
Quoted 8 lines
Now there is a further consequence:
3. In view of (1) and (2) there has to be a way agreed upon for the module to pass the relevant information onward.
It is precisely on item 3 that we are stuck right now. There is an immediate need, not a theoretical need but an immediate need. However, there is no agreed-upon method or convention for communication.
There are already controls being added for HFLIP, VFLIP, and rotation, so that's certainly one way. They're being added as writable controls for someone's hardware that can do these image manipulations. But it seems like a different driver could just as easily provide them as read-only controls to inform software that it has a non-standard image layout.
It sounds like per frame rotation metadata would be useful for cameras that have an orientation sensor.
Quoted 4 lines
problems are not so very related at all. As I understand, it is visualized that a camera could be put on a pivot, with control mechanism which would permit various rotations and then the question becomes how to support a camera and to make the stream come out "right" no matter which way the
I think we already have controls for camera motors, i.e. pan, tilt, and rotation. That's totally different.
Quoted 15 lines · Mauro Carvalho Chehab
Quoted 9 lines · On Sun, 15 Feb 2009 10:29:03 +0100 Hans de Goede <[email protected]> wrote:
Quoted 4 lines
I think we should also be able to detect 90 and 270 degree rotations. Or at the very least prepare for it. It's a safe bet to assume that webcams will arrive that can detect portrait vs landscape orientation.
Handling those (esp on the fly) will be rather hard as width and height then get swapped. So lets worry about those when we need to. We will need an additional flag for those cases anyways.
The camera rotation is something that is already needed, at least on some embedded devices, like those cellular phones whose display changes when you rotate the device.
Quoted 5 lines
By looking at the
v4l2_bufferstruct, we currently have 4 reserved bytes. It
has also one flags field, with several bits not used. I can see 2 possibilities to extend the API:1) adding
V4L2_BUF_FLAG_HFLIPandV4L2_BUF_FLAG_VFLIPflags. This would work for 90, 180 and 270 rotation;
HFLIP and VFLIP are only good for 0 and 180 degrees. 90 and 270 isn't the same as flipping.
The problem I'm seeing here is that as people are using v4l2 for digital cameras instead of tv capture there is more and more meta-data available. Things like shutter speed, aperture, focus distance, and so on. Just look at all the EXIF data a digital camera provides. Four bytes and two flags are going to run out very quickly at this rate.
It's a shame there are not 8 bytes left, as then they could be used for a pointer to an extended meta-data structure.
Quoted 27 lines · On Monday 16 February 2009 05:04:40 Trent Piepho wrote:
Quoted 16 lines · Mauro Carvalho Chehab
On Sun, 15 Feb 2009 10:29:03 +0100
Quoted 8 lines · Hans de Goede
Quoted 4 lines
I think we should also be able to detect 90 and 270 degree rotations. Or at the very least prepare for it. It's a safe bet to assume that webcams will arrive that can detect portrait vs landscape orientation.
Handling those (esp on the fly) will be rather hard as width and height then get swapped. So lets worry about those when we need to. We will need an additional flag for those cases anyways.
The camera rotation is something that is already needed, at least on some embedded devices, like those cellular phones whose display changes when you rotate the device.
Quoted 6 lines
By looking at the
v4l2_bufferstruct, we currently have 4 reserved bytes. It has also one flags field, with several bits not used. I can see 2 possibilities to extend the API:1) adding
V4L2_BUF_FLAG_HFLIPandV4L2_BUF_FLAG_VFLIPflags. This would work for 90, 180 and 270 rotation;HFLIP and VFLIP are only good for 0 and 180 degrees. 90 and 270 isn't the same as flipping.
Quoted 8 lines
The problem I'm seeing here is that as people are using v4l2 for digital cameras instead of tv capture there is more and more meta-data available. Things like shutter speed, aperture, focus distance, and so on. Just look at all the EXIF data a digital camera provides. Four bytes and two flags are going to run out very quickly at this rate.
It's a shame there are not 8 bytes left, as then they could be used for a pointer to an extended meta-data structure.
I think we have to distinguish between two separate types of data: fixed ('the sensor is mounted upside-down', or 'the sensor always requires a hflip/vflip') and dynamic ('the user pivoted the camera 270 degrees').
The first is static data and I think we can just reuse the existing HFLIP/VFLIP controls: just make them READONLY to tell libv4l that libv4l needs to do the flipping.
The second is dynamic data and should be passed through v4l2_buffer since this can change on a per-frame basis. In this case add two bits to the v4l2_buffer's flags field:
V4L2_BUF_FLAG_ROTATION_MSK 0x0c00V4L2_BUF_FLAG_ROTATION_0 0x0000V4L2_BUF_FLAG_ROTATION_90 0x0400V4L2_BUF_FLAG_ROTATION_180 0x0800V4L2_BUF_FLAG_ROTATION_270 0x0c00
No need to use the reserved field.
This makes a lot more sense to me: static (or rarely changing) data does not belong to v4l2_buffer, that's what controls are for. And something dynamic like pivoting belongs to v4l2_buffer. This seems like a much cleaner API to me.
Regards,
Hans
Signature · 1 line
Hans Verkuil - video4linux developer - sponsored by TANDBERG
Quoted 1 line · [email protected] wrote:
<huge snip>
Quoted 6 lines
Therefore,
1. Everyone seems to agree that the kernel module itself is not going to do things like rotate or flip data even if a given supported device always needs that done.
However, this decision has a consequence:
Quoted 8 lines
2. Therefore, the module must send the information about what is needed out of the module, to whatever place is going to deal with it. Information which is known to the module but unknown anywere else must be transmitted somehow.
Now there is a further consequence:
3. In view of (1) and (2) there has to be a way agreed upon for the module to pass the relevant information onward.
It is precisely on item 3 that we are stuck right now. There is an immediate need, not a theoretical need but an immediate need. However, there is no agreed-upon method or convention for communication.
We are no longer stuck here, the general agreement is adding 2 new buffer flags, one to indicate the driver knows the data in the buffer is vflipped and one for hflip. Then we can handle v-flipped, h-flipped and 180 degrees cameras
This is agreed up on, Trent is arguing we may need more flags in the future, but that is something for the future, all we need know is these 2 flags and Hans Verkuil who AFAIK was the only one objecting to doing this with buffer flags has agreed this is the best solution.
So Adam, kilgota, please ignore the rest of this thread and move forward with the driver, just add the necessary buffer flags to videodev2.h as part of your patch (It is usually to submit new API stuff with the same patch which introduces the first users of this API.
I welcome libv4l patches to use these flags.
Regards,
Hans
Quoted 46 lines · Hans Verkuil
Quoted 24 lines · On Monday 16 February 2009 05:04:40 Trent Piepho wrote:
Quoted 14 lines · Mauro Carvalho Chehab
On Sun, 15 Feb 2009 10:29:03 +0100
Quoted 7 lines · Hans de Goede
Quoted 4 lines
I think we should also be able to detect 90 and 270 degree rotations. Or at the very least prepare for it. It's a safe bet to assume that webcams will arrive that can detect portrait vs landscape orientation.
Handling those (esp on the fly) will be rather hard as width and height then get swapped. So lets worry about those when we need to. We will need an additional flag for those cases anyways.
The camera rotation is something that is already needed, at least on some embedded devices, like those cellular phones whose display changes when you rotate the device.
Quoted 6 lines
By looking at the
v4l2_bufferstruct, we currently have 4 reserved bytes. It has also one flags field, with several bits not used. I can see 2 possibilities to extend the API:1) adding
V4L2_BUF_FLAG_HFLIPandV4L2_BUF_FLAG_VFLIPflags. This would work for 90, 180 and 270 rotation;HFLIP and VFLIP are only good for 0 and 180 degrees. 90 and 270 isn't the same as flipping.
Quoted 8 lines
The problem I'm seeing here is that as people are using v4l2 for digital cameras instead of tv capture there is more and more meta-data available. Things like shutter speed, aperture, focus distance, and so on. Just look at all the EXIF data a digital camera provides. Four bytes and two flags are going to run out very quickly at this rate.
It's a shame there are not 8 bytes left, as then they could be used for a pointer to an extended meta-data structure.
I think we have to distinguish between two separate types of data: fixed ('the sensor is mounted upside-down', or 'the sensor always requires a hflip/vflip') and dynamic ('the user pivoted the camera 270 degrees').
The first is static data and I think we can just reuse the existing HFLIP/VFLIP controls: just make them READONLY to tell libv4l that libv4l needs to do the flipping.
The second is dynamic data and should be passed through
v4l2_buffersince this can change on a per-frame basis. In this case add two bits to the v4l2_buffer's flags field:
V4L2_BUF_FLAG_ROTATION_MSK0x0c00V4L2_BUF_FLAG_ROTATION_00x0000V4L2_BUF_FLAG_ROTATION_900x0400V4L2_BUF_FLAG_ROTATION_1800x0800V4L2_BUF_FLAG_ROTATION_2700x0c00
Quoted 3 lines
No need to use the reserved field.
This makes a lot more sense to me: static (or rarely changing) data does not belong to
v4l2_buffer, that's what controls are for. And something dynamic like pivoting belongs tov4l2_buffer. This seems like a much cleaner API to me.
I agree that we have static and dynamic camera properties, and that we may want
to have 2 API's for them. I disagree the control API is the proper API to
expose static properties, many existing applications will not handle this well.
More over in the case we are discussing now, we have one type of data (sensor orientation) which can be both static or dynamic depending on the camera having 2 API's for this is just plain silly. Thus unnecessarily complicates things if some camera property can be static in some cases and dynamic in others then we should just always treat it as dynamic. This way we will only have one code path to deal with instead of two (with very interesting interactions also, what if both API's sat rotate 180 degrees, should we then not rotate at all?). This way lies madness.
My conclusion:
1) Since rotation can be dynamic store it in the buffer flags
2) In the future we will most likely need an API to be able to query camera properties
Regards,
Hans