v4l: Add camera voice coil lens control class, current control

Add a V4L2 control class for voice coil lens driver devices. These are simple devices that are used to move a camera lens from its resting position.

Signed-off-by Sakari Ailus <[email protected]>

---
 Documentation/media/uapi/v4l/extended-controls.rst | 28 ++++++++++++++++++++++
 include/uapi/linux/v4l2-controls.h                 |  8 +++++++
 2 files changed, 36 insertions(+)

Documentation/media/uapi/v4l/extended-controls.rst +28 -0

index abb1057..a75451a 100644
@@ -3022,6 +3022,34 @@ Image Process Control IDs
     driver specific and are documented in :ref:`v4l-drivers`.
 
 
+.. _voice-coil-lens-controls:
+
+Voice Coil Lens Control Reference
+=================================
+
+The Voice Coil class controls are used to control voice coil lens
+devices. These are very simple devices that consist of a voice coil, a
+spring and a lens. The current applied on the voice coil is used to
+move the lens away from the resting position which typically is (close
+to) infinity. The higher the current applied, the closer the lens is
+typically focused.
+
+.. _voice-coil-lens-control-is:
+
+Voice Coil Lens Control IDs
+---------------------------
+
+``V4L2_CID_VOICE_COIL_CLASS (class)``
+    The VOICE_COIL class descriptor.
+
+``V4L2_CID_VOICE_COIL_CURRENT (integer)``
+    Current applied on a voice coil. The more current is applied, the
+    more is the position of the lens moved from its resting position.
+    Do note that there may be a ringing effect; the lens will
+    oscillate after changing the current applied unless the device
+    implements ringing compensation.
+
+
 .. _dv-controls:
 
 Digital Video Control Reference

include/uapi/linux/v4l2-controls.h +8 -0

index 0d2e1e0..9ef152b 100644
@@ -62,6 +62,7 @@
 #define V4L2_CTRL_CLASS_FM_RX		0x00a10000	/* FM Receiver controls */
 #define V4L2_CTRL_CLASS_RF_TUNER	0x00a20000	/* RF tuner controls */
 #define V4L2_CTRL_CLASS_DETECT		0x00a30000	/* Detection controls */
+#define V4L2_CTRL_CLASS_VOICE_COIL	0x00a40000	/* Voice coil lens driver controls */
 
 /* User-class control IDs */
 

@@ -894,6 +895,13 @@ enum v4l2_jpeg_chroma_subsampling {
 #define V4L2_CID_TEST_PATTERN			(V4L2_CID_IMAGE_PROC_CLASS_BASE + 3)
 #define V4L2_CID_DEINTERLACING_MODE		(V4L2_CID_IMAGE_PROC_CLASS_BASE + 4)
 
+/* Voice coil lens driver controls */
+
+#define V4L2_CID_VOICE_COIL_CLASS_BASE		(V4L2_CTRL_CLASS_VOICE_COIL | 0x900)
+#define V4L2_CID_VOICE_COIL_CLASS		(V4L2_CTRL_CLASS_VOICE_COIL | 1)
+
+#define V4L2_CID_VOICE_COIL_CURRENT		(V4L2_CID_VOICE_COIL_CLASS_BASE + 1)
+
 
 /*  DV-class control IDs defined by V4L2 */
 #define V4L2_CID_DV_CLASS_BASE			(V4L2_CTRL_CLASS_DV | 0x900)

Quoted 4 lines · Sakari Ailus

Add a V4L2 control class for voice coil lens driver devices. These are simple devices that are used to move a camera lens from its resting position.

Quoted 1 line

Signed-off-by Sakari Ailus <[email protected]>

Looks good to me.

I wonder... should we somehow expose the range of diopters to userspace? I believe userland camera application will need that information.

Quoted 5 lines · Sakari Ailus

Quoted 4 lines

Add a V4L2 control class for voice coil lens driver devices. These are simple devices that are used to move a camera lens from its resting position.

Quoted 7 lines

Quoted 1 line

Signed-off-by Sakari Ailus <[email protected]>

Looks good to me.

I wonder... should we somehow expose the range of diopters to userspace? I believe userland camera application will need that information.

It'd certainly be useful to be able to provide more information.

The question is: where to store it, and how? It depends on the voice coil, the spring constant, the lens and the distance of the lens from the sensor --- at least. Probably the sensor size as well.

On voice coil lenses it is also somewhat inexact.

Hi!

Quoted 14 lines · Pavel Machek

Quoted 5 lines · Sakari Ailus

Quoted 4 lines

Add a V4L2 control class for voice coil lens driver devices. These are simple devices that are used to move a camera lens from its resting position.

Quoted 7 lines

Quoted 1 line

Signed-off-by Sakari Ailus <[email protected]>

Looks good to me.

I wonder... should we somehow expose the range of diopters to userspace? I believe userland camera application will need that information.

Quoted 7 lines

It'd certainly be useful to be able to provide more information.

The question is: where to store it, and how? It depends on the voice coil, the spring constant, the lens and the distance of the lens from the sensor --- at least. Probably the sensor size as well.

On voice coil lenses it is also somewhat inexact.

I was thinking read-only attribute providing minimum and maximum diopters in case there's linear relationship as on N900.

Quoted 12 lines

Quoted 11 lines

Quoted 3 lines

I wonder... should we somehow expose the range of diopters to userspace? I believe userland camera application will need that information.

It'd certainly be useful to be able to provide more information.

The question is: where to store it, and how? It depends on the voice coil, the spring constant, the lens and the distance of the lens from the sensor --- at least. Probably the sensor size as well.

On voice coil lenses it is also somewhat inexact.

Review

Quoted 5 lines

I was thinking read-only attribute providing minimum and maximum diopters in case there's linear relationship as on N900.

(quoted diff) +2 -0

+#define V4L2_CID_VOICE_DIOPTERS_AT_REST (V4L2_CID_VOICE_COIL_CLASS_BASE + 2)
+#define V4L2_CID_VOICE_DIOPTERS_AT_MAX (V4L2_CID_VOICE_COIL_CLASS_BASE + 3)

Where do you store that information and how? Should the user be also told how the applied current affects the value?

I also wonder whether that's the best way to provide the information to the user --- we have things such as devices that are a part of a camera module and telling the user on which side of the device the camera is located.

We've been planning to have a property API for this to provide the user with a tree of key-value pairs, with details unsettled as of yet, so it's certainly nothing that could be used yet.

Do you have a user application that could make use of such information?

Add a V4L2 control class for voice coil lens driver devices. These are simple devices that are used to move a camera lens from its resting position.

From some past threads with this patch, you mentioned that:

Signed-off-by Sakari Ailus <[email protected]>

---
 Documentation/media/uapi/v4l/extended-controls.rst | 28 ++++++++++++++++++++++
 include/uapi/linux/v4l2-controls.h                 |  8 +++++++
 2 files changed, 36 insertions(+)

Documentation/media/uapi/v4l/extended-controls.rst +28 -0

index abb1057..a75451a 100644
@@ -3022,6 +3022,34 @@ Image Process Control IDs
     driver specific and are documented in :ref:`v4l-drivers`.
 
 
+.. _voice-coil-lens-controls:
+
+Voice Coil Lens Control Reference
+=================================
+
+The Voice Coil class controls are used to control voice coil lens
+devices. These are very simple devices that consist of a voice coil, a
+spring and a lens. The current applied on the voice coil is used to
+move the lens away from the resting position which typically is (close
+to) infinity. The higher the current applied, the closer the lens is
+typically focused.
+
+.. _voice-coil-lens-control-is:
+
+Voice Coil Lens Control IDs
+---------------------------
+
+``V4L2_CID_VOICE_COIL_CLASS (class)``
+    The VOICE_COIL class descriptor.
+
+``V4L2_CID_VOICE_COIL_CURRENT (integer)``
+    Current applied on a voice coil. The more current is applied, the
+    more is the position of the lens moved from its resting position.
+    Do note that there may be a ringing effect; the lens will
+    oscillate after changing the current applied unless the device
+    implements ringing compensation.
+
+
 .. _dv-controls:
 
 Digital Video Control Reference

include/uapi/linux/v4l2-controls.h +1 -0

index 0d2e1e0..9ef152b 100644
@@ -62,6 +62,7 @@
 #define V4L2_CTRL_CLASS_FM_RX		0x00a10000	/* FM Receiver controls */
 #define V4L2_CTRL_CLASS_RF_TUNER	0x00a20000	/* RF tuner controls */
 #define V4L2_CTRL_CLASS_DETECT		0x00a30000	/* Detection controls */
+#define V4L2_CTRL_CLASS_VOICE_COIL	0x00a40000	/* Voice coil lens driver controls */
 
 /* User-class control IDs */
 

@@ -894,6 +895,13 @@ enum v4l2_jpeg_chroma_subsampling {
 #define V4L2_CID_TEST_PATTERN			(V4L2_CID_IMAGE_PROC_CLASS_BASE + 3)
 #define V4L2_CID_DEINTERLACING_MODE		(V4L2_CID_IMAGE_PROC_CLASS_BASE + 4)
 

Worse than that, patch 2/2 gives the false sensation that both controls are equal.

Ok, I understand that they need to be identical on the existing driver, in order to keep backward compatibility, but I'm afraid that, without a clear distinction between them at the documentation, people may just clone the existing code on other drivers.

Hi Sakari,

Em Tue, 14 Feb 2017 14:20:22 +0200
Sakari Ailus <[email protected]> escreveu:

Quoted 3 lines

Add a V4L2 control class for voice coil lens driver devices. These are simple devices that are used to move a camera lens from its resting position.

From some past threads with this patch, you mentioned that:

"The FOCUS_ABSOLUTE control really is not a best control ID to
control a voice coil driver's current."

However, I'm not seeing any explanation there at the thread, at the patch description or at the documentation explaining why, and, more important, when someone should use the focus control or the camera voice coil control.

Worse than that, patch 2/2 gives the false sensation that both controls are equal.

Ok, I understand that they need to be identical on the existing driver, in order to keep backward compatibility, but I'm afraid that, without a clear distinction between them at the documentation, people may just clone the existing code on other drivers.

So, please add more details to patch 1/2.

Hi Mauro,

Quoted 14 lines · Mauro Carvalho Chehab

Hi Sakari,

Em Tue, 14 Feb 2017 14:20:22 +0200
Sakari Ailus <[email protected]> escreveu:

Quoted 3 lines

Add a V4L2 control class for voice coil lens driver devices. These are simple devices that are used to move a camera lens from its resting position.

From some past threads with this patch, you mentioned that:

"The FOCUS_ABSOLUTE control really is not a best control ID to
control a voice coil driver's current."

Quoted 4 lines

However, I'm not seeing any explanation there at the thread, at the patch description or at the documentation explaining why, and, more important, when someone should use the focus control or the camera voice coil control.

It should be available in the thread.

Nevertheless, V4L2_CID_FOCUS_ABSOLUTE
is documented as follows (emphasis mine):

This control sets the *focal point* of the camera to the specified position. The unit is undefined. Positive values set the focus closer to the camera, negative values towards infinity.

What you control in voice coil devices is current (in Ampères) and the current only has a relatively loose relation to the focal point.

Additionally, increasing the current brings the focus closer, not farther.

I anticipate adding controls for ringing compensation in the future. Virtually all other devices except this one do ringing compensation and there's some control to be done for that.

How about adding such an explanation added to the commit message?

Quoted 9 lines

Worse than that, patch 2/2 gives the false sensation that both controls are equal.

Ok, I understand that they need to be identical on the existing driver, in order to keep backward compatibility, but I'm afraid that, without a clear distinction between them at the documentation, people may just clone the existing code on other drivers.

Indeed. The only reason that I'm not just replacing FOCUS_ABSOLUTE with the new contorol is backwards compatibility. But as Pavel pointed out, he's likely the sole user of this device that can only be found (as far as we commonly are aware) in the Nokia N900.

I'm happy to just switch the control, and Pavel mentioned he's happy with that. It would avoid copying the code in new drivers --- which I would most certainly point out anyway.

Quoted 2 lines

So, please add more details to patch 1/2.

Let me know if you're happy with the above.

Hi!

For focus to be useful, we need autofocus implmented somewhere. Unfortunately, v4l framework does not seem to provide good place where to put autofocus. I believe, long-term, we'll need some kind of "video server" providing this kind of services.

Anyway, we probably don't want autofocus in kernel (even through some cameras do it in hardware), and we probably don't want autofocus in each and every user application.

So what remains is libv4l2.

Now, this is in no way clean or complete,
and functionality provided by sdl.c and asciicam.c probably _should_ be in application, but... I'd like to get the code out there.

Oh and yes, I've canibalized decode_tm6000.c application instead of introducing my own. Autotools scare me, sorry.

Regards,
Pavel

lib/libv4l2/asciicam.c new file +7 -0

new file mode 100644
index 0000000..5388967
@@ -0,0 +1,63 @@
+/* gcc asciicam.c /usr/lib/i386-linux-gnu/libv4l2.so.0.0.0 -o asciicam
+   gcc asciicam.c /usr/lib/arm-linux-gnueabi/libv4l2.so.0 -o asciicam
+
+gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -fvisibility=hidden -I../../lib/include -Wall -Wpointer-arith -D_GNU_SOURCE -I../../include -g -O2 asciicam.c libv4l2.c /usr/lib/arm-linux-gnueabi/libv4lconvert.so.0 log.c v4l2convert.c v4l2-plugin.c -o asciicam
+ */
+#include <stdio.h>
+#include <sys/types.h>

lib/libv4l2/asciicam.c new file +31 -0

+  int i;
+  static struct v4l2_format fmt;
+
+#if 1
+  fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+  fmt.fmt.pix.pixelformat = V4L2_PIX_FMT_RGB24;
+  fmt.fmt.pix.field = V4L2_FIELD_INTERLACED;
+  fmt.fmt.pix.width = 640;
+  fmt.fmt.pix.height = 480;
+  if (fmt.fmt.pix.pixelformat != 'RGB3') /* v4l2_fourcc('R', 'G', 'B', '3'); */
+    printf("hmm. strange format?\n");
+  
+  printf("ioctl = %d\n", v4l2_ioctl(fd, VIDIOC_S_FMT, &fmt));
+#endif
+
+  for (i=0; i<500; i++) {
+    int num = v4l2_read(fd, buf, SIZE);
+    int y,x;
+    
+    printf("%d\n", num);
+#if 0
+    for (y = 0; y < 25; y++) {
+      for (x = 0; x < 80; x++) {
+	int y1 = y * 480/25;
+	int x1 = x * 640/80;
+	int c = buf[fmt.fmt.pix.width*3*y1 + 3*x1] +
+	  buf[fmt.fmt.pix.width*3*y1 + 3*x1 + 1] +
+	  buf[fmt.fmt.pix.width*3*y1 + 3*x1 + 2];
+
+	if (c < 30) c = ' ';
+	else if (c < 60) c = '.';

lib/libv4l2/asciicam.c new file +3 -0

+#endif    
+  }
+}

lib/libv4l2/libv4l2-priv.h +17 -0

index 343db5e..af740a7 100644
@@ -1,3 +1,4 @@
+/* -*- c-file-style: "linux" -*- */
 /*
 #             (C) 2008 Hans de Goede <[email protected]>
 

@@ -70,6 +71,14 @@
 	} while (0)
 
 #define MIN(a, b) (((a) < (b)) ? (a) : (b))
+#define V4L2_MAX_SUBDEVS 16
+
+#define V4L2_MAX_FOCUS 10
+struct v4l2_focus_step {
+	int num;
+	int sharpness[V4L2_MAX_FOCUS];
+	int position[V4L2_MAX_FOCUS];
+};
 
 struct v4l2_dev_info {
 	int fd;

@@ -104,6 +113,14 @@ struct v4l2_dev_info {
 	void *plugin_library;
 	void *dev_ops_priv;
 	const struct libv4l_dev_ops *dev_ops;
+        int subdev_fds[V4L2_MAX_SUBDEVS];
+  	int exposure; 
+  	int frame;
+  	int focus;
+
+	/* Autofocus parameters */
+	int focus_frame, focus_exposure, focus_step, sh_prev;
+	struct v4l2_focus_step focus_step_params;
 };
 
 /* From v4l2-plugin.c */

lib/libv4l2/libv4l2.c +0 -0

index 0ba0a88..3b84437 100644

lib/libv4l2/libv4l2.c +0 -0

 /*
 #             (C) 2008 Hans de Goede <[email protected]>
 

lib/libv4l2/libv4l2.c +15 -1

 static int v4l2_ensure_convert_mmap_buf(int index)
 {
 	if (devices[index].convert_mmap_buf != MAP_FAILED) {

@@ -311,7 +319,409 @@ static int v4l2_queue_read_buffer(int index, int buffer_index)
 	return 0;
 }
 
-static int v4l2_dequeue_and_convert(int index, struct v4l2_buffer *buf,
+static void v4l2_paint(char *buf, int x1, int y1, const struct v4l2_format *fmt)
+{
+  int y, x;
+  int width = fmt->fmt.pix.width;
+  printf("width = %d\n", width);
+    
+    for (y = 0; y < 25; y++) {
+      for (x = 0; x < 80; x++) {
+	int c = buf[width*4*y1 + 4*x1] +
+	  buf[width*4*y1 + 4*x1 + 1] +
+	  buf[width*4*y1 + 4*x1 + 2];
+
+	if (c < 30) c = ' ';
+	else if (c < 60) c = '.';
+	else if (c < 120) c = '_';

lib/libv4l2/libv4l2.c +33 -0

+
+#define SX 80
+#define SY 25
+#define BUCKETS 20
+
+static void v4l2_histogram(unsigned char *buf, int cdf[], struct v4l2_format *fmt)
+{
+    for (int y = 0; y < fmt->fmt.pix.height; y+=19)
+      for (int x = 0; x < fmt->fmt.pix.width; x+=19) {
+	pixel p = buf_pixel(fmt, buf, x, y);
+	
+	int b;
+	/* HACK: we divide green by 2 to have nice picture, undo it here. */
+	b = p.r + 2*p.g + p.b;
+	b = (b * BUCKETS)/(256);
+	cdf[b]++;
+      }
+}
+
+static long v4l2_sharpness(unsigned char *buf, struct v4l2_format *fmt)
+{
+  int h = fmt->fmt.pix.height;
+  int w = fmt->fmt.pix.width;
+  long r = 0;
+
+    for (int y = h/3; y < h-h/3; y+=h/9)
+      for (int x = w/3; x < w-w/3; x++) {
+	pixel p1 = buf_pixel(fmt, buf, x, y);
+	pixel p2 = buf_pixel(fmt, buf, x+2, y);
+	
+	int b1, b2;
+	/* HACK: we divide green by 2 to have nice picture, undo it here. */
+	b1 = p1.r + 2*p1.g + p1.b;

lib/libv4l2/libv4l2.c +9 -0

+
+int v4l2_set_exposure(int fd, int exposure)
+{
+	int index = v4l2_get_index(fd);
+
+	if (index == -1 || devices[index].convert == NULL) {
+		V4L2_LOG_ERR("v4l2_set_exposure called with invalid fd: %d\n", fd);
+		errno = EBADF;
+		return -1;

lib/libv4l2/libv4l2.c +65 -0

+	ctrl.value = gain;
+	if (ioctl(devices[index].subdev_fds[0], VIDIOC_S_CTRL, &ctrl) < 0) {
+	  printf("Could not set exposure\n");
+	}
+	return 0;
+}
+
+int v4l2_set_focus(int fd, int diopt)
+{
+	int index = v4l2_get_index(fd);
+
+	if (index == -1 || devices[index].convert == NULL) {
+		V4L2_LOG_ERR("v4l2_set_focus called with invalid fd: %d\n", fd);
+		errno = EBADF;
+		return -1;
+	}
+
+	struct v4l2_control ctrl;
+	ctrl.id = V4L2_CID_FOCUS_ABSOLUTE;
+	ctrl.value = diopt;
+	if (ioctl(devices[index].subdev_fds[1], VIDIOC_S_CTRL, &ctrl) < 0) {
+		printf("Could not set focus\n");
+	}
+	return 0;
+}
+
+#define LESS 1
+#define MID 0
+#define MORE 2
+
+static void v4l2_start_focus_step(struct v4l2_focus_step *fs, int focus, int step)
+{
+	int i;
+
+	fs->num = 3;
+	for (i=0; i<V4L2_MAX_FOCUS; i++) {
+		fs->sharpness[i] = -1;
+		fs->position[i] = -1;
+	}
+
+	fs->position[LESS] = focus - step;
+	fs->position[MID] = focus;
+	fs->position[MORE] = focus + step;
+}
+
+static int v4l2_focus_get_best(struct v4l2_focus_step *fs)
+{
+	int i, max = -1, maxi = -1;
+	
+	for (i=0; i<fs->num; i++)
+		if (max < fs->sharpness[i]) {
+			max = fs->sharpness[i];
+			maxi = i;
+		}
+
+	return maxi;
+}
+
+static void v4l2_auto_focus_continuous(struct v4l2_dev_info *m, int sh)
+{
+	int f = m->frame - m->focus_frame;
+	int step;
+	const int max_step = 300, min_step = 20;
+	struct v4l2_focus_step *fs = &m->focus_step_params;
+

lib/libv4l2/libv4l2.c +29 -0

+		printf("Focused at %d\n", m->focus);
+		exit(0);
+	}
+
+	step = m->focus_step;
+
+	if (m->exposure != m->focus_exposure) {
+		m->focus_frame = m->frame;
+		m->focus_exposure = m->exposure;
+		v4l2_start_focus_step(fs, m->focus, m->focus_step);
+		return;
+	}
+	if (m->focus < step) {
+		m->focus = step;
+	}
+
+	const int every = 3;
+	if (f%every)
+		return;
+
+	{
+		int i = f/every;
+
+		if (i == 0) {
+			printf("Can not happen?\n");
+			return;
+		}
+		i--;
+		if (i < fs->num)

lib/libv4l2/libv4l2.c +205 -3

+	int best = v4l2_focus_get_best(fs);
+	if ((fs->sharpness[best] < m->sh_prev) && (m->focus_step < max_step)) {
+		m->focus_step *=2;
+		m->sh_prev = m->sh_prev * 0.9;
+		printf("step up %d\n", m->focus_step);
+	} else if (best == LESS) {
+		printf("less ");
+		m->focus -= step;
+	} else if (best == MORE) {
+		printf("more ");
+		m->focus += step;
+	} else {
+		m->sh_prev = fs->sharpness[MID];
+		m->focus_step = m->focus_step / 2;
+		printf("step %d ", m->focus_step);
+	}
+	m->focus_frame = m->frame;
+	v4l2_start_focus_step(fs, m->focus, m->focus_step);
+	printf("Focus now %d\n", m->focus);
+}
+
+static void v4l2_start_focus_sweep(struct v4l2_focus_step *fs, int focus, int step)
+{
+	int i;
+
+	fs->num = V4L2_MAX_FOCUS;
+	for (i=0; i<V4L2_MAX_FOCUS; i++) {
+		fs->sharpness[i] = -1;
+		fs->position[i] = -1;
+	}
+
+	int f = focus;
+	for (i=0; i<V4L2_MAX_FOCUS; i++) {
+		fs->position[i] = f;
+		f += step;
+	}
+}
+
+static void v4l2_auto_focus_single(struct v4l2_dev_info *m, int sh)
+{
+	int f = m->frame - m->focus_frame;
+	int step;
+	struct v4l2_focus_step *fs = &m->focus_step_params;
+
+	if (m->focus_step == 0) {
+		printf("step reset -- max\n");
+		m->focus_step = 1;
+	}
+
+	if (m->exposure != m->focus_exposure) {
+		m->focus_frame = m->frame;
+		m->focus_exposure = m->exposure;
+		v4l2_start_focus_sweep(fs, 0, 100);
+		return;
+	}
+
+	const int every = 3;
+	if (f%every)
+		return;
+
+	{
+		int i = f/every;
+
+		if (i == 0) {
+			printf("Can not happen?\n");
+			return;
+		}
+		i--;
+		if (i < fs->num)
+			v4l2_set_focus(m->fd, fs->position[i]);
+		if (i > 0)
+			fs->sharpness[i-1] = sh;
+		if (i < fs->num)
+			return;
+	}
+#if 0
+	int i;
+	for (i=0; i<fs->num; i++) {
+		printf("%d: %d | ", fs->position[i], fs->sharpness[i]);
+	}
+#endif
+	int best = v4l2_focus_get_best(fs);
+	m->focus_frame = m->frame;
+	switch (m->focus_step) {
+	case 1:
+		printf("Best now %d / %d\n", fs->position[best], fs->sharpness[best]);
+		v4l2_start_focus_sweep(fs, fs->position[best] - 50, 10);
+		m->focus_step = 2;
+		break;
+	case 2:
+		printf("Best now %d / %d\n", fs->position[best], fs->sharpness[best]);
+		v4l2_start_focus_sweep(fs, fs->position[best] - 10, 2);
+		m->focus_step = 3;
+		break;
+	case 3:
+		printf("Best now %d / %d\n", fs->position[best], fs->sharpness[best]);
+		printf("done.\n");
+		exit(0);
+	}
+}
+
+
+static void v4l2_auto_exposure(int index, struct v4l2_buffer *buf)
+{
+	struct v4l2_format *fmt;
+	int cdf[BUCKETS] = { 0, };
+	int i;
+
+	fmt = &devices[index].src_fmt;
+
+	v4l2_histogram(devices[index].frame_pointers[buf->index], cdf, fmt);
+
+#if 0
+	printf("hist: ");
+	for (i = 0; i<BUCKETS; i++)
+		printf("%d ", cdf[i]);
+	printf("\n");
+#endif
+	for (i=1; i<BUCKETS; i++)
+		cdf[i] += cdf[i-1];
+
+	int b = BUCKETS;
+	int brightPixels = cdf[b-1] - cdf[b-8];
+	int targetBrightPixels = cdf[b-1]/50;
+	int maxSaturatedPixels = cdf[b-1]/200;
+	int saturatedPixels = cdf[b-1] - cdf[b-2];
+	// how much should I change brightness by
+	float adjustment = 1.0f;
+#if 0
+	printf( "AutoExposure: totalPixels: %d,"
+		"brightPixels: %d, targetBrightPixels: %d,"
+		"saturatedPixels: %d, maxSaturatedPixels: %d\n",
+		cdf[b-1], brightPixels, targetBrightPixels,
+		saturatedPixels, maxSaturatedPixels);
+#endif
+	  
+	if (saturatedPixels > maxSaturatedPixels) {
+		// first don't let things saturate too much
+		adjustment = 1.0f - ((float)(saturatedPixels - maxSaturatedPixels))/cdf[b-1];
+	} else if (brightPixels < (targetBrightPixels - (saturatedPixels * 4))) {
+		// increase brightness to try and hit the desired number of well exposed pixels
+		int l = b-6;
+		while (brightPixels < targetBrightPixels && l > 0) {
+			brightPixels += cdf[l];
+			brightPixels -= cdf[l-1];
+			l--;
+		}
+
+		// that level is supposed to be at b-11;
+		adjustment = ((float) (b-6+1))/(l+1);
+	} else {
+		// we're not oversaturated, and we have enough bright pixels. Do nothing.
+	}
+
+	{
+		float limit = 4;
+		if (adjustment > limit) { adjustment = limit; }
+		if (adjustment < 1/limit) { adjustment = 1/limit; }
+	}
+	  
+	if (!devices[index].exposure)
+		devices[index].exposure = 1;
+	devices[index].exposure *= adjustment;
+	if (adjustment != 1.)
+		printf( "AutoExposure: adjustment: %f exposure %d\n", adjustment, devices[index].exposure);
+
+	v4l2_set_exposure(devices[index].fd, devices[index].exposure);
+}
+
+static void v4l2_statistics(int index, struct v4l2_buffer *buf)
+{
+	unsigned char *b;
+	struct v4l2_format *fmt;
+
+	fmt = &devices[index].src_fmt;
+	b = devices[index].frame_pointers[buf->index];
+
+	if (!(devices[index].frame%3))
+		v4l2_auto_exposure(index, buf);
+	
+	int sh = v4l2_sharpness(b, fmt);
+	v4l2_auto_focus_single(&devices[index], sh);
+	
+	devices[index].frame++;
+	if (!(devices[index].frame%4))
+		sdl_render(&sdl, b, fmt);
+}
+
+int v4l2_dequeue_and_convert(int index, struct v4l2_buffer *buf,
 		unsigned char *dest, int dest_size)
 {
 	const int max_tries = V4L2_IGNORE_FIRST_FRAME_ERRORS + 1;

@@ -345,6 +755,13 @@ static int v4l2_dequeue_and_convert(int index, struct v4l2_buffer *buf,
 			errno = -EINVAL;
 			return -1;
 		}
+		
+#if 1
+		v4l2_statistics(index, buf);
+#endif
+#if 0
+		/* This is rather major eater of CPU time. CPU time goes from 80% to 4% 
+		   when conversion is disabled. */
 
 		result = v4lconvert_convert(devices[index].convert,
 				&devices[index].src_fmt, &devices[index].dest_fmt,

@@ -352,7 +769,7 @@ static int v4l2_dequeue_and_convert(int index, struct v4l2_buffer *buf,
 				buf->bytesused, dest ? dest : (devices[index].convert_mmap_buf +
 					buf->index * devices[index].convert_mmap_frame_size),
 				dest_size);
-
+#endif
 		if (devices[index].first_frame) {
 			/* Always treat convert errors as EAGAIN during the first few frames, as
 			   some cams produce bad frames at the start of the stream

@@ -789,18 +1206,24 @@ no_capture:
 
 	/* Note we always tell v4lconvert to optimize src fmt selection for
 	   our default fps, the only exception is the app explicitly selecting
-	   a fram erate using the S_PARM ioctl after a S_FMT */
+	   a frame rate using the S_PARM ioctl after a S_FMT */
 	if (devices[index].convert)
 		v4lconvert_set_fps(devices[index].convert, V4L2_DEFAULT_FPS);
 	v4l2_update_fps(index, &parm);
 
+	devices[index].subdev_fds[0] = SYS_OPEN("/dev/video_sensor", O_RDWR, 0);
+	devices[index].subdev_fds[1] = SYS_OPEN("/dev/video_focus", O_RDWR, 0);
+
+	printf("Sensor: %d, focus: %d\n", devices[index].subdev_fds[0], 
+	       devices[index].subdev_fds[1]);
+
 	V4L2_LOG("open: %d\n", fd);
 
 	return fd;
 }
 
 /* Is this an fd for which we are emulating v4l1 ? */
-static int v4l2_get_index(int fd)
+int v4l2_get_index(int fd)
 {
 	int index;
 

lib/libv4l2/libv4l2.c +62 -0

 	index = v4l2_get_index(fd);
 	if (index == -1)
 		return SYS_CLOSE(fd);

@@ -1782,3 +2209,65 @@ int v4l2_get_control(int fd, int cid)
 			(qctrl.maximum - qctrl.minimum) / 2) /
 		(qctrl.maximum - qctrl.minimum);
 }
+
+void v4l2_debug(void)
+{
+	printf("debug\n");
+}
+
+/* ------------------------------------------------------------------ */
+
+#define SIZE 10*1024*1024
+
+char buf[SIZE];
+
+void my_main(void)
+{
+	int fd = v4l2_open("/dev/video2", O_RDWR);
+	int i;
+	static struct v4l2_format fmt;
+
+	fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+	fmt.fmt.pix.pixelformat = V4L2_PIX_FMT_RGB24;
+	fmt.fmt.pix.field = V4L2_FIELD_INTERLACED;
+	fmt.fmt.pix.width = 640;
+	fmt.fmt.pix.height = 480;
+
+	v4l2_set_gain(fd, 300);
+	v4l2_set_exposure(fd, 10000);
+	v4l2_set_focus(fd, 0);
+
+
+	printf("ioctl = %d\n", v4l2_ioctl(fd, VIDIOC_S_FMT, &fmt));
+
+	printf("capture is %d, %d\n", fmt.fmt.pix.width, fmt.fmt.pix.height);
+	/* factor == 2 still fits window, but very slow. factor == 6 .. */
+	/* factor needs to be odd, otherwise ... fun with BA10 format. */
+	sdl_init(&sdl, fmt.fmt.pix.width, fmt.fmt.pix.height, 5);
+  
+	v4l2_debug();
+
+	/* In 800x600 "raw" mode, this should take cca 17.8 seconds (without
+	   sdl output. CPU usage should be cca 5% without conversion). That's 28 fps.
+	   (benchmark with i<500)
+	*/
+	for (i=0; i<50000; i++) {
+		int num = v4l2_read(fd, buf, SIZE);
+
+		if (i==490) {
+			printf("Focus to closest.... -------------------\n");
+			v4l2_set_focus(fd, 99999);
+		}
+    
+#if 0
+		v4l2_paint(buf, 640/80, 480/25, &fmt);
+#endif
+		/* Over USB connection, rendering every single frame slows
+		   execution down from 23 seconds to 36 seconds. */
+#if 0
+		if (!(i%4))
+			sdl_render(&sdl, buf, &fmt);
+#endif
+	}
+  
+}

lib/libv4l2/sdl.c new file +0 -0

new file mode 100644
index 0000000..17a8d24

utils/decode_tm6000/decode_tm6000.c +5 -347

@@ -1,365 +1,16 @@
-/*
-   decode_tm6000.c - decode multiplexed format from TM5600/TM6000 USB
+/* gcc asciicam.c /usr/lib/i386-linux-gnu/libv4l2.so.0.0.0 -o asciicam
+   gcc asciicam.c /usr/lib/arm-linux-gnueabi/libv4l2.so.0 -o asciicam
 
-   Copyright (C) 2007 Mauro Carvalho Chehab <[email protected]>
-
-   This program is free software; you can redistribute it and/or modify
-   it under the terms of the GNU General Public License as published by
-   the Free Software Foundation version 2.
-
-   This program is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-   GNU General Public License for more details.
-
-   You should have received a copy of the GNU General Public License
-   along with this program; if not, write to the Free Software
-   Foundation, Inc., 51 Franklin Street, Suite 500, Boston, MA 02110-1335 USA.
+gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -fvisibility=hidden -I../../lib/include -Wall -Wpointer-arith -D_GNU_SOURCE -I../../include -g -O2 asciicam.c libv4l2.c /usr/lib/arm-linux-gnueabi/libv4lconvert.so.0 log.c v4l2convert.c v4l2-plugin.c -o asciicam
  */
-#include "../libv4l2util/v4l2_driver.h"
 #include <stdio.h>
-#include <string.h>
-#include <argp.h>
-#include <unistd.h>
 #include <sys/types.h>
 #include <sys/stat.h>
 #include <fcntl.h>
-#include <stdlib.h>
-#include <errno.h>
-
-const char *argp_program_version="decode_tm6000 version 0.0.1";
-const char *argp_program_bug_address="Mauro Carvalho Chehab <[email protected]>";
-const char doc[]="Decodes tm6000 proprietary format streams";
-const struct argp_option options[] = {
-	{"verbose",	'v',	0,	0,	"enables debug messages", 0},
-	{"device",	'd',	"DEV",	0,	"uses device for reading", 0},
-	{"output",	'o',	"FILE",	0,	"outputs raw stream to a file", 0},
-	{"input",	'i',	"FILE",	0,	"parses a file, instead of a device", 0},
-	{"freq",	'f',	"Freq",	0,	"station frequency, in MHz (default is 193.25)", 0},
-	{"nbufs",	'n',	"quant",0,	"number of video buffers", 0},
-	{"audio",	'a',	0,	0,	"outputs audio on stdout", 0},
-	{"read",	'r',	0,	0,	"use read() instead of mmap method", 0},
-	{ 0, 0, 0, 0, 0, 0 }
-};
-
-static char outbuf[692224];
-static int debug=0, audio=0, use_mmap=1, nbufs=4;
-static float freq_mhz=193.25;
-static char *devicename="/dev/video0";
-static char *filename=NULL;
-static enum {
-	NORMAL,
-	INPUT,
-	OUTPUT
-} mode = NORMAL;
-
-static FILE *fout;
-
-//const char args_doc[]="ARG1 ARG2";
-
-static error_t parse_opt (int key, char *arg, struct argp_state *state)
-{
-	switch (key) {
-	case 'a':
-		audio++;
-		break;
-	case 'r':
-		use_mmap=0;
-		break;
-	case 'v':
-		debug++;
-		break;
-	case 'd':
-		devicename=arg;
-		break;
-	case 'i':
-	case 'o':
-		if (mode!=NORMAL) {
-			argp_error(state,"You can't use input/output options simultaneously.\n");
-			break;
-		}
-		if (key=='i')
-			mode=INPUT;
-		else
-			mode=OUTPUT;
-
-		filename=arg;
-		break;
-	case 'f':
-		freq_mhz=atof(arg);
-		break;
-	case 'n':
-		nbufs=atoi(arg);
-		if  (nbufs<2)
-			nbufs=2;
-		break;
-	default:
-		return ARGP_ERR_UNKNOWN;
-	}
-	return 0;
-}
-
-static struct argp argp = {
-	.options=options,
-	.parser=parse_opt,
-	.args_doc=NULL,
-	.doc=doc,
-};
 
-#define TM6000_URB_MSG_LEN 180
-enum {
-	TM6000_URB_MSG_VIDEO=1,
-	TM6000_URB_MSG_AUDIO,
-	TM6000_URB_MSG_VBI,
-	TM6000_URB_MSG_PTS,
-	TM6000_URB_MSG_ERR,
-};
-
-const char *tm6000_msg_type[]= {
-	"unknown(0)",	//0
-	"video",	//1
-	"audio",	//2
-	"vbi",		//3
-	"pts",		//4
-	"err",		//5
-	"unknown(6)",	//6
-	"unknown(7)",	//7
-};
-
-#define dprintf(fmt,arg...) \
-	if (debug) fprintf(stderr, fmt, ##arg)
-
-static int recebe_buffer (struct v4l2_buffer *v4l2_buf, struct v4l2_t_buf *buf)
-{
-	dprintf("Received %zd bytes\n", buf->length);
-fflush(stdout);
-	memcpy (outbuf,buf->start,buf->length);
-	return buf->length;
-}
-
-
-static int prepare_read (struct v4l2_driver *drv)
-{
-	struct v4l2_format fmt;
-	double freq;
-	int rc;
-
-	memset (drv,0,sizeof(*drv));
-
-	if (v4l2_open (devicename, 1,drv)<0) {
-		perror ("Error opening dev");
-		return -1;
-	}
-
-	memset (&fmt,0,sizeof(fmt));
-
-	uint32_t pixelformat=V4L2_PIX_FMT_TM6000;
-
-	if (v4l2_gettryset_fmt_cap (drv,V4L2_SET,&fmt, 720, 480,
-				    pixelformat,V4L2_FIELD_ANY)) {
-		perror("set_input to tm6000 raw format");
-		return -1;
-	}
-
-	if (freq_mhz) {
-		freq=freq_mhz * 1000 * 1000;
-		rc=v4l2_getset_freq (drv,V4L2_SET, &freq);
-		if (rc<0)
-			printf ("Cannot set freq to %.3f MHz\n",freq_mhz);
-	}
-
-	if (use_mmap) {
-		printf("Preparing for receiving frames on %d buffers...\n",nbufs);
-		fflush (stdout);
-
-		rc=v4l2_mmap_bufs(drv, nbufs);
-		if (rc<0) {
-			printf ("Cannot mmap %d buffers\n",nbufs);
-			return -1;
-		}
-
-//		v4l2_stop_streaming(&drv);
-		rc=v4l2_start_streaming(drv);
-		if (rc<0) {
-			printf ("Cannot start streaming\n");
-			return -1;
-		}
-	}
-	printf("Waiting for frames...\n");
-
-	return 0;
-}
+#include <linux/videodev2.h>
 
-static int read_stream (struct v4l2_driver *drv, int fd)
+void main(void)
 {
-	if (use_mmap) {
-		fd_set fds;
-		struct timeval tv;
-		int r;
-
-		FD_ZERO (&fds);
-		FD_SET (fd, &fds);
-
-		/* Timeout. */
-		tv.tv_sec = 2;
-		tv.tv_usec = 0;
-
-		r = select (fd + 1, &fds, NULL, NULL, &tv);
-		if (-1 == r) {
-			if (EINTR == errno) {
-				perror ("select");
-				return -errno;
-			}
-		}
-
-		if (0 == r) {
-			fprintf (stderr, "select timeout\n");
-			return -errno;
-		}
-
-		return v4l2_rcvbuf(drv, recebe_buffer);
-	} else {
-		int size=read(fd, outbuf, sizeof(outbuf));
-		return size;
-	}
-
-	return 0;
-}
-
-static int read_char (struct v4l2_driver *drv, int fd)
-{
-	static int sizebuf=0;
-	static unsigned char *p=NULL;
-	unsigned char c;
-
-	if (sizebuf<=0) {
-		sizebuf=read_stream(drv,fd);
-		if (sizebuf<=0)
-			return -1;
-		p=(unsigned char *)outbuf;
-	}
-	c=*p;
-	p++;
-	sizebuf--;
-
-	return c;
-}
-
-
-int main (int argc, char*argv[])
-{
-	int fd;
-	unsigned int i;
-	unsigned char buf[TM6000_URB_MSG_LEN], img[720*2*480];
-	unsigned int  cmd, size, field, block, line, pos=0;
-	unsigned long header=0;
-	int           linesize=720*2,skip=0;
-	struct v4l2_driver drv;
-
-	argp_parse (&argp, argc, argv, 0, 0, 0);
-
-	if (mode!=INPUT) {
-		if (prepare_read (&drv)<0)
-			return -1;
-		fd=drv.fd;
-	} else {
-		/*mode == INPUT */
-
-		fd=open(filename,O_RDONLY);
-		if (fd<0) {
-			perror ("error opening a file for parsing");
-			return -1;
-		}
-		dprintf("file %s opened for parsing\n",filename);
-		use_mmap=0;
-	}
-
-	if (mode==OUTPUT) {
-		fout=fopen(filename,"w");
-		if (!fout) {
-			perror ("error opening a file to write");
-			return -1;
-		}
-		dprintf("file %s opened for output\n",filename);
-
-		do {
-			size=read_stream (&drv,fd);
-
-			if (size<=0) {
-				close (fd);
-				return -1;
-			}
-			dprintf("writing %d bytes\n",size);
-			fwrite(outbuf,1, size,fout);
-//			fflush(fout);
-		} while (1);
-	}
-
-
-	while (1) {
-		skip=0;
-		header=0;
-		do {
-			int c;
-			c=read_char (&drv,fd);
-			if (c<0) {
-				perror("read");
-				return -1;
-			}
-
-			header=(header>>8)&0xffffff;  
-			header=header|(c<<24);
-			skip++;
-		} while ( (((header>>24)&0xff) != 0x47) );
-
-		/* split the header fields */
-		size  = (((header & 0x7e)<<1) -1) * 4;
-		block = (header>>7) & 0xf;
-		field = (header>>11) & 0x1;
-		line  = (header>>12) & 0x1ff;
-		cmd   = (header>>21) & 0x7;
-
-		/* Read the remaining buffer */
-		for (i=0;i<sizeof(buf);i++) {
-			int c;
-			c=read_char (&drv,fd);
-			if (c<0) {
-				perror("read");
-				return -1;
-			}
-			buf[i]=c;
-		}
-
-		/* FIXME: Mounts the image as field0+field1
-			* It should, instead, check if the user selected
-			* entrelaced or non-entrelaced mode
-			*/
-		pos=((line<<1)+field)*linesize+
-					block*TM6000_URB_MSG_LEN;
-
-			/* Prints debug info */
-		dprintf("0x%08x (skip %d), %s size=%d, line=%d, field=%d, block=%d\n",
-				(unsigned int)header, skip,
-				 tm6000_msg_type[cmd],
-				 size, line, field, block);
-
-		/* Don't allow to write out of the buffer */
-		if (pos+sizeof(buf) > sizeof(img))
-			cmd = TM6000_URB_MSG_ERR;
-
-		/* handles each different URB message */
-		switch(cmd) {
-		case TM6000_URB_MSG_VIDEO:
-			/* Fills video buffer */
-			memcpy(buf,&img[pos],sizeof(buf));
-		case TM6000_URB_MSG_AUDIO:
-			if (audio)

Quoted 5 lines

For focus to be useful, we need autofocus implmented somewhere. Unfortunately, v4l framework does not seem to provide good place where to put autofocus. I believe, long-term, we'll need some kind of "video server" providing this kind of services.

Quoted 5 lines

Anyway, we probably don't want autofocus in kernel (even through some cameras do it in hardware), and we probably don't want autofocus in each and every user application.

So what remains is libv4l2.

IMO, the best place for autofocus is at libv4l2. Putting it on a separate "video server" application looks really weird for me.

Btw, libv4l2 already has some autotools for auto gain and auto white balance. See the implementation under:
lib/libv4lconvert/processing

The libv4l internal controls can be seen at:
lib/libv4lconvert/control/libv4lcontrol.h

The ones implemented by the processing part of the library are:

Quoted 6 lines

Now, this is in no way clean or complete,
and functionality provided by sdl.c and asciicam.c probably _should_ be in application, but... I'd like to get the code out there.

Oh and yes, I've canibalized decode_tm6000.c application instead of introducing my own. Autotools scare me, sorry.

Why replace decode_tm6000.c by something else? If you want to add another test application, just place it on a new file.

I added a few notes together with the code, pointing the main things I think it require changes, in order for me to do a better review at the code. I didn't test nor tried to check the algorithms inside, as the code, on its current state, requires rework and code cleanup.

Please don't add a new application under lib/. It is fine if you want some testing application, if the ones there aren't enough, but please place it under contrib/test/.

You should likely take a look at v4l2grab first, as it could have almost everything you would need.

IMHO, it would be better to use aalib. Btw, xawtv3 has a code example using it, under:
console/ttv.c

As it already uses libv4l, prhaps you could use it, instead of adding a new ascii app.

lib/libv4l2/libv4l2.c static struct v4l2_dev_info devices[V4L2_MAX_DEVICES] = { · line 110

Quoted 7 lines

lib/libv4l2/libv4l2.c +7 -0

+#include "sdl.c"
+
+static struct sdl sdl;
+
+int v4l2_get_index(int fd);
+void my_main(void);
+

The above looks really odd. Why do you want to make libv4l2 dependent on sdl?

Quoted 33 lines

lib/libv4l2/libv4l2.c +33 -0

+static void v4l2_histogram(unsigned char *buf, int cdf[], struct v4l2_format *fmt)
+{
+    for (int y = 0; y < fmt->fmt.pix.height; y+=19)
+      for (int x = 0; x < fmt->fmt.pix.width; x+=19) {
+	pixel p = buf_pixel(fmt, buf, x, y);
+	
+	int b;
+	/* HACK: we divide green by 2 to have nice picture, undo it here. */
+	b = p.r + 2*p.g + p.b;
+	b = (b * BUCKETS)/(256);
+	cdf[b]++;
+      }
+}
+
+static long v4l2_sharpness(unsigned char *buf, struct v4l2_format *fmt)
+{
+  int h = fmt->fmt.pix.height;
+  int w = fmt->fmt.pix.width;
+  long r = 0;
+
+    for (int y = h/3; y < h-h/3; y+=h/9)
+      for (int x = w/3; x < w-w/3; x++) {
+	pixel p1 = buf_pixel(fmt, buf, x, y);
+	pixel p2 = buf_pixel(fmt, buf, x+2, y);
+	
+	int b1, b2;
+	/* HACK: we divide green by 2 to have nice picture, undo it here. */
+	b1 = p1.r + 2*p1.g + p1.b;
+	b2 = p2.r + 2*p2.g + p2.b;
+
+	int v;
+	v = (b1-b2)*(b1-b2);
+	if (v > 36)

lib/libv4l2/libv4l2.c static int v4l2_queue_read_buffer(int index, int buffer_index) · line 389

Quoted 5 lines

lib/libv4l2/libv4l2.c +5 -0

+		r+=v;
+      }
+
+    return r;
+}

IMO, the above belongs to a separate processing module under
lib/libv4lconvert/processing/

Quoted 15 lines

lib/libv4l2/libv4l2.c +15 -0

+
+int v4l2_set_exposure(int fd, int exposure)
+{
+	int index = v4l2_get_index(fd);
+
+	if (index == -1 || devices[index].convert == NULL) {
+		V4L2_LOG_ERR("v4l2_set_exposure called with invalid fd: %d\n", fd);
+		errno = EBADF;
+		return -1;
+	}
+
+	struct v4l2_control ctrl;
+	ctrl.id = V4L2_CID_EXPOSURE;
+	ctrl.value = exposure;
+	if (ioctl(devices[index].subdev_fds[0], VIDIOC_S_CTRL, &ctrl) < 0) {

lib/libv4l2/libv4l2.c static int v4l2_queue_read_buffer(int index, int buffer_index) · line 408

Quoted 4 lines

lib/libv4l2/libv4l2.c +4 -0

+	  printf("Could not set exposure\n");
+	}
+	return 0;
+}

Shouldn't it be together with lib/libv4lconvert/processing/autogain.c, perhaps as an alternative implementation, if what's there is not enough?

lib/libv4l2/libv4l2.c int v4l2_close(int fd) · line 1252

Quoted 8 lines

lib/libv4l2/libv4l2.c +4 -0

@@ -823,6 +1246,10 @@ int v4l2_close(int fd)
 {
 	int index, result;
 
+	if (fd == -2) {
+	  my_main();
+	}
+

That looks a hack!

lib/libv4l2/sdl.c line 8

Quoted 2 lines

lib/libv4l2/sdl.c new file +2 -0

+#include <SDL2/SDL.h>
+#include <SDL2/SDL_image.h>

If you're adding a SDL-specific application, you'll need to add the needed autoconf bits to detect if SDL devel package is installed, auto-disabling it if not.

Yet, I don't think that SDL should be part of the library, but, instead, part of some application.

utils/decode_tm6000/decode_tm6000.c

Quoted 3 lines

index 4bffbdd..fda7e3b 100644

utils/decode_tm6000/decode_tm6000.c +0 -0

Everything below it is completely wrong!

Hi!

Quoted 5 lines

Quoted 5 lines

For focus to be useful, we need autofocus implmented somewhere. Unfortunately, v4l framework does not seem to provide good place where to put autofocus. I believe, long-term, we'll need some kind of "video server" providing this kind of services.

Quoted 8 lines

Quoted 5 lines

Anyway, we probably don't want autofocus in kernel (even through some cameras do it in hardware), and we probably don't want autofocus in each and every user application.

So what remains is libv4l2.

IMO, the best place for autofocus is at libv4l2. Putting it on a separate "video server" application looks really weird for me.

Well... let me see. libraries are quite limited -- it is hard to open files, or use threads/have custom main loop. It may be useful to switch resolutions -- do autofocus/autogain at lower resolution, then switch to high one for taking picture. It would be good to have that in "system" code, but I'm not at all sure libv4l2 design will allow that.

It would be good if application could say "render live camera into this window" and only care about user interface, then say "give me a high resolution jpeg". But that would require main loop in the library...

It would be nice if more than one application could be accessing the camera at the same time... (I.e. something graphical running preview then using command line tool to grab a picture.) This one is definitely not solveable inside a library...

Quoted 2 lines

The above looks really odd. Why do you want to make libv4l2 dependent on sdl?

I don't, but I had some nasty problems with linker; this should really go into application but it refused to link. Scary libtool.

Quoted 6 lines

If you're adding a SDL-specific application, you'll need to add the needed autoconf bits to detect if SDL devel package is installed, auto-disabling it if not.

Yet, I don't think that SDL should be part of the library, but, instead, part of some application.

Agreed. libtool prevented me from doing the right thing.

Quoted 14 lines

Quoted 5 lines

Quoted 5 lines

For focus to be useful, we need autofocus implmented somewhere. Unfortunately, v4l framework does not seem to provide good place where to put autofocus. I believe, long-term, we'll need some kind of "video server" providing this kind of services.

Quoted 8 lines

Quoted 5 lines

Anyway, we probably don't want autofocus in kernel (even through some cameras do it in hardware), and we probably don't want autofocus in each and every user application.

So what remains is libv4l2.

IMO, the best place for autofocus is at libv4l2. Putting it on a separate "video server" application looks really weird for me.

Quoted 6 lines

Well... let me see. libraries are quite limited -- it is hard to open files, or use threads/have custom main loop. It may be useful to switch resolutions -- do autofocus/autogain at lower resolution, then switch to high one for taking picture. It would be good to have that in "system" code, but I'm not at all sure libv4l2 design will allow that.

I don't see why it would be hard to open files or have threads inside a library. There are several libraries that do that already, specially the ones designed to be used on multimidia apps.

Quoted 4 lines

It would be good if application could say "render live camera into this window" and only care about user interface, then say "give me a high resolution jpeg". But that would require main loop in the library...

Nothing prevents writing an upper layer on the top of libv4l in order to provide such kind of functions.

Quoted 4 lines

It would be nice if more than one application could be accessing the camera at the same time... (I.e. something graphical running preview then using command line tool to grab a picture.) This one is definitely not solveable inside a library...

Someone once suggested to have something like pulseaudio for V4L. For such usage, a server would be interesting. Yet, I would code it in a way that applications using libv4l will talk with such daemon in a transparent way.

Quoted 5 lines

Quoted 4 lines

It would be nice if more than one application could be accessing the camera at the same time... (I.e. something graphical running preview then using command line tool to grab a picture.) This one is definitely not solveable inside a library...

Quoted 4 lines

Someone once suggested to have something like pulseaudio for V4L. For such usage, a server would be interesting. Yet, I would code it in a way that applications using libv4l will talk with such daemon in a transparent way.

Yes, we need something like pulseaudio for V4L. And yes, we should make it transparent for applications using libv4l.

Quoted 10 lines · On Tuesday 25 April 2017 10:05:38 Pavel Machek wrote:

Quoted 9 lines

Quoted 4 lines

It would be nice if more than one application could be accessing the camera at the same time... (I.e. something graphical running preview then using command line tool to grab a picture.) This one is definitely not solveable inside a library...

Someone once suggested to have something like pulseaudio for V4L. For such usage, a server would be interesting. Yet, I would code it in a way that applications using libv4l will talk with such daemon in a transparent way.

Quoted 2 lines

Yes, we need something like pulseaudio for V4L. And yes, we should make it transparent for applications using libv4l.

IIRC there is already some effort in writing such "video" server which would support accessing more application into webcam video, like pulseaudio server for accessing more applications to microphone input.

Hi!

Quoted 11 lines · Pali Roh?r

Quoted 10 lines · On Tuesday 25 April 2017 10:05:38 Pavel Machek wrote:

Quoted 9 lines

Quoted 4 lines

It would be nice if more than one application could be accessing the camera at the same time... (I.e. something graphical running preview then using command line tool to grab a picture.) This one is definitely not solveable inside a library...

Someone once suggested to have something like pulseaudio for V4L. For such usage, a server would be interesting. Yet, I would code it in a way that applications using libv4l will talk with such daemon in a transparent way.

Quoted 6 lines

Quoted 2 lines

Yes, we need something like pulseaudio for V4L. And yes, we should make it transparent for applications using libv4l.

IIRC there is already some effort in writing such "video" server which would support accessing more application into webcam video, like pulseaudio server for accessing more applications to microphone input.

Do you have project name / url / something?

Quoted 20 lines · On Tuesday 25 April 2017 13:23:30 Pavel Machek wrote:

Hi!

Quoted 17 lines · Pali Roh?r

Quoted 5 lines · On Tuesday 25 April 2017 10:05:38 Pavel Machek wrote:

Quoted 5 lines

Quoted 4 lines

It would be nice if more than one application could be accessing the camera at the same time... (I.e. something graphical running preview then using command line tool to grab a picture.) This one is definitely not solveable inside a library...

Quoted 7 lines

Quoted 4 lines

Someone once suggested to have something like pulseaudio for V4L. For such usage, a server would be interesting. Yet, I would code it in a way that applications using libv4l will talk with such daemon in a transparent way.

Yes, we need something like pulseaudio for V4L. And yes, we should make it transparent for applications using libv4l.

IIRC there is already some effort in writing such "video" server which would support accessing more application into webcam video, like pulseaudio server for accessing more applications to microphone input.

Quoted 1 line

Do you have project name / url / something?

Pinos (renamed from PulseVideo)

https://blogs.gnome.org/uraeus/2015/06/30/introducing-pulse-video/
https://cgit.freedesktop.org/~wtay/pinos/

But from git history it looks like it is probably dead now...

Quoted 25 lines · Pali Roh?r

Quoted 13 lines · On Tuesday 25 April 2017 13:23:30 Pavel Machek wrote:

Hi!

Quoted 11 lines · Pali Roh?r

Quoted 10 lines · On Tuesday 25 April 2017 10:05:38 Pavel Machek wrote:

Quoted 9 lines

Quoted 4 lines

It would be nice if more than one application could be accessing the camera at the same time... (I.e. something graphical running preview then using command line tool to grab a picture.) This one is definitely not solveable inside a library...

Someone once suggested to have something like pulseaudio for V4L. For such usage, a server would be interesting. Yet, I would code it in a way that applications using libv4l will talk with such daemon in a transparent way.

Quoted 8 lines

Quoted 6 lines

Quoted 2 lines

Yes, we need something like pulseaudio for V4L. And yes, we should make it transparent for applications using libv4l.

IIRC there is already some effort in writing such "video" server which would support accessing more application into webcam video, like pulseaudio server for accessing more applications to microphone input.

Do you have project name / url / something?

Pinos (renamed from PulseVideo)

Quoted 4 lines

https://blogs.gnome.org/uraeus/2015/06/30/introducing-pulse-video/
https://cgit.freedesktop.org/~wtay/pinos/

But from git history it looks like it is probably dead now...

Actually, last commit is an hour ago on "work" branch. Seems alive to me ;-).

Thanks for pointer...
Pavel

Pinos (renamed from PulseVideo)

https://blogs.gnome.org/uraeus/2015/06/30/introducing-pulse-video/
https://cgit.freedesktop.org/~wtay/pinos/

But from git history it looks like it is probably dead now...

Quoted 14 lines

Quoted 13 lines

Quoted 6 lines

It would be nice if more than one application could be accessing the
camera at the same time... (I.e. something graphical running preview
then using command line tool to grab a picture.) This one is definitely not solveable inside a library...

Someone once suggested to have something like pulseaudio for V4L. For such usage, a server would be interesting. Yet, I would code it
in a way that applications using libv4l will talk with such daemon
in a transparent way.

Quoted 2 lines

Yes, we need something like pulseaudio for V4L. And yes, we should make it transparent for applications using libv4l.

IIRC there is already some effort in writing such "video" server which
would support accessing more application into webcam video, like pulseaudio server for accessing more applications to microphone input.

Well, fd's are hard, because application can do fork() and now interesting stuff happens. Threads are tricky, because now you have locking etc.

libv4l2 is designed to be LD_PRELOADED. That is not really feasible with "complex" library.

Le mardi 25 avril 2017 ? 10:05 +0200, Pavel Machek a ?crit?:

Quoted 4 lines

Well, fd's are hard, because application can do fork() and now interesting stuff happens. Threads are tricky, because now you have locking etc.

Quoted 2 lines

libv4l2 is designed to be LD_PRELOADED. That is not really feasible with "complex" library.

That is incorrect. The library propose an API where you simply replace certain low level calls, like ioctl -> v4l2_ioctl, open -> v4l2_open(). You have to do that explicitly in your existing code. It does not abstract the API itself unlike libdrm.

Quoted 9 lines

Quoted 2 lines

IMO, the best place for autofocus is at libv4l2. Putting it on a separate "video server" application looks really weird for me.

Well... let me see. libraries are quite limited -- it is hard to open files, or use threads/have custom main loop. It may be useful to switch resolutions -- do autofocus/autogain at lower resolution, then switch to high one for taking picture. It would be good to have that in "system" code, but I'm not at all sure libv4l2 design will allow that.

I don't see why it would be hard to open files or have threads inside a library. There are several libraries that do that already, specially the ones designed to be used on multimidia apps.

Hi!

Quoted 10 lines

Quoted 9 lines

Quoted 2 lines

IMO, the best place for autofocus is at libv4l2. Putting it on a separate "video server" application looks really weird for me.

Well... let me see. libraries are quite limited -- it is hard to open files, or use threads/have custom main loop. It may be useful to switch resolutions -- do autofocus/autogain at lower resolution, then switch to high one for taking picture. It would be good to have that in "system" code, but I'm not at all sure libv4l2 design will allow that.

Quoted 3 lines

I don't see why it would be hard to open files or have threads inside a library. There are several libraries that do that already, specially the ones designed to be used on multimidia apps.

Well, This is what the libv4l2 says:

This file implements libv4l2, which offers v4l2_ prefixed versions of
open/close/etc. The API is 100% the same as directly opening
/dev/videoX
using regular open/close/etc, the big difference is that format
conversion

but if I open additional files in v4l2_open(), API is no longer the same, as unix open() is defined to open just one file descriptor.

Now. There is autogain support in libv4lconvert, but it expects to use same fd for camera and for the gain... which does not work with subdevs.

Of course, opening subdevs by name like this is not really acceptable. But can you suggest a method that is?

Thanks,
Pavel

commit 4cf9d10ead014c0db25452e4bb9cd144632407c3

Author: Pavel <[email protected]>
Date:   Wed Apr 26 11:38:04 2017 +0200

Add subdevices.

lib/libv4l2/libv4l2-priv.h +2 -0

index 343db5e..a6bc48e 100644
@@ -26,6 +26,7 @@
 #include "../libv4lconvert/libv4lsyscall-priv.h"
 
 #define V4L2_MAX_DEVICES 16
+#define V4L2_MAX_SUBDEVS 8
 /* Warning when making this larger the frame_queued and frame_mapped members of
    the v4l2_dev_info struct can no longer be a bitfield, so the code needs to
    be adjusted! */

@@ -104,6 +105,7 @@ struct v4l2_dev_info {
 	void *plugin_library;
 	void *dev_ops_priv;
 	const struct libv4l_dev_ops *dev_ops;
+        int subdev_fds[V4L2_MAX_SUBDEVS];
 };
 
 /* From v4l2-plugin.c */

lib/libv4l2/libv4l2.c +10 -2

index 0ba0a88..edc9642 100644
@@ -1,3 +1,4 @@
+/* -*- c-file-style: "linux" -*- */
 /*
 #             (C) 2008 Hans de Goede <[email protected]>
 

@@ -789,18 +790,25 @@ no_capture:
 
 	/* Note we always tell v4lconvert to optimize src fmt selection for
 	   our default fps, the only exception is the app explicitly selecting
-	   a fram erate using the S_PARM ioctl after a S_FMT */
+	   a frame rate using the S_PARM ioctl after a S_FMT */
 	if (devices[index].convert)
 		v4lconvert_set_fps(devices[index].convert, V4L2_DEFAULT_FPS);
 	v4l2_update_fps(index, &parm);
 
+	devices[index].subdev_fds[0] = SYS_OPEN("/dev/video_sensor", O_RDWR, 0);
+	devices[index].subdev_fds[1] = SYS_OPEN("/dev/video_focus", O_RDWR, 0);
+	devices[index].subdev_fds[2] = -1;
+
+	printf("Sensor: %d, focus: %d\n", devices[index].subdev_fds[0], 
+	       devices[index].subdev_fds[1]);
+
 	V4L2_LOG("open: %d\n", fd);
 
 	return fd;
 }
 
 /* Is this an fd for which we are emulating v4l1 ? */
-static int v4l2_get_index(int fd)
+int v4l2_get_index(int fd)
 {
 	int index;
 

commit 1d6a9ce121f53e8f2e38549eed597a3c3dea5233

Author: Pavel <[email protected]>
Date:   Wed Apr 26 12:34:04 2017 +0200

Enable ioctl propagation.

lib/libv4l2/libv4l2.c +20 -0

index edc9642..6dab661 100644
@@ -1064,6 +1064,23 @@ static int v4l2_s_fmt(int index, struct v4l2_format *dest_fmt)
 	return 0;
 }
 
+static int v4l2_propagate_ioctl(int index, unsigned long request, void *arg)
+{
+	int i = 0;
+	int result;
+	while (1) {
+		if (devices[index].subdev_fds[i] == -1)
+			return -1;
+		printf("g_ctrl failed, trying...\n");
+		result = SYS_IOCTL(devices[index].subdev_fds[i], request, arg);
+		printf("subdev %d result %d\n", i, result);
+		if (result == 0)
+			return 0;
+		i++;
+	}
+	return -1;
+}
+
 int v4l2_ioctl(int fd, unsigned long int request, ...)
 {
 	void *arg;

@@ -1193,14 +1210,20 @@ no_capture_request:
 	switch (request) {
 	case VIDIOC_QUERYCTRL:
 		result = v4lconvert_vidioc_queryctrl(devices[index].convert, arg);
+		if (result == -1)
+			result = v4l2_propagate_ioctl(index, request, arg);
 		break;
 
 	case VIDIOC_G_CTRL:
 		result = v4lconvert_vidioc_g_ctrl(devices[index].convert, arg);
+		if (result == -1)

Quoted 20 lines

Quoted 3 lines

I don't see why it would be hard to open files or have threads inside a library. There are several libraries that do that already, specially the ones designed to be used on multimidia apps.

Well, This is what the libv4l2 says:

This file implements libv4l2, which offers v4l2_ prefixed versions of
open/close/etc. The API is 100% the same as directly opening
/dev/videoX
using regular open/close/etc, the big difference is that format
conversion

but if I open additional files in v4l2_open(), API is no longer the same, as unix open() is defined to open just one file descriptor.

Now. There is autogain support in libv4lconvert, but it expects to use same fd for camera and for the gain... which does not work with subdevs.

Quoted 2 lines

Of course, opening subdevs by name like this is not really acceptable. But can you suggest a method that is?

There are two separate things here:

1) Autofoucs for a device that doesn't use subdev API
2) libv4l2 support for devices that require MC and subdev API

Hi!

Quoted 20 lines

Quoted 20 lines

Quoted 3 lines

I don't see why it would be hard to open files or have threads inside a library. There are several libraries that do that already, specially the ones designed to be used on multimidia apps.

Well, This is what the libv4l2 says:

This file implements libv4l2, which offers v4l2_ prefixed versions of
open/close/etc. The API is 100% the same as directly opening
/dev/videoX
using regular open/close/etc, the big difference is that format
conversion

but if I open additional files in v4l2_open(), API is no longer the same, as unix open() is defined to open just one file descriptor.

Now. There is autogain support in libv4lconvert, but it expects to use same fd for camera and for the gain... which does not work with subdevs.

Quoted 7 lines

Quoted 2 lines

Of course, opening subdevs by name like this is not really acceptable. But can you suggest a method that is?

There are two separate things here:

1) Autofoucs for a device that doesn't use subdev API
2) libv4l2 support for devices that require MC and subdev API

Actually there are three: 0) autogain. Unfortunately, I need autogain first before autofocus has a chance...

And that means... bayer10 support for autogain.

Plus, I changed avg_lum to long long. Quick calculation tells me int could overflow with few megapixel sensor.

Oh, btw http://ytse.tricolour.net/docs/LowLightOptimization.html no longer works.

Regards,
Pavel

lib/libv4lconvert/processing/autogain.c +0 -0

index c6866d6..0b52d0f 100644
@@ -68,6 +71,41 @@ static void autogain_adjust(struct v4l2_queryctrl *ctrl, int *value,
 	}
 }
 

lib/libv4lconvert/processing/autogain.c +8 -0

+			avg_lum += *buf++;
+		buf += fmt->fmt.pix.bytesperline - fmt->fmt.pix.width / 2;
+	}
+	avg_lum /= fmt->fmt.pix.height * fmt->fmt.pix.width / 4;
+	avg_lum /= 4;
+	return avg_lum;
+}
+

lib/libv4lconvert/processing/autogain.c +15 -9

+			avg_lum += *buf++;
+		buf += fmt->fmt.pix.bytesperline - fmt->fmt.pix.width / 2;
+	}
+	avg_lum /= fmt->fmt.pix.height * fmt->fmt.pix.width / 4;
+	return avg_lum;
+}
+
 /* auto gain and exposure algorithm based on the knee algorithm described here:
 http://ytse.tricolour.net/docs/LowLightOptimization.html */
 static int autogain_calculate_lookup_tables(

@@ -100,17 +142,16 @@ static int autogain_calculate_lookup_tables(
 	switch (fmt->fmt.pix.pixelformat) {
+	case V4L2_PIX_FMT_SGBRG10:
+	case V4L2_PIX_FMT_SGRBG10:
+	case V4L2_PIX_FMT_SBGGR10:
+	case V4L2_PIX_FMT_SRGGB10:
+		avg_lum = get_luminosity_bayer10((void *) buf, fmt);
+		break;
+
 	case V4L2_PIX_FMT_SGBRG8:
 	case V4L2_PIX_FMT_SGRBG8:
 	case V4L2_PIX_FMT_SBGGR8:
 	case V4L2_PIX_FMT_SRGGB8:
-		buf += fmt->fmt.pix.height * fmt->fmt.pix.bytesperline / 4 +
-			fmt->fmt.pix.width / 4;
-
-		for (y = 0; y < fmt->fmt.pix.height / 2; y++) {
-			for (x = 0; x < fmt->fmt.pix.width / 2; x++)
-				avg_lum += *buf++;
-			buf += fmt->fmt.pix.bytesperline - fmt->fmt.pix.width / 2;
-		}
-		avg_lum /= fmt->fmt.pix.height * fmt->fmt.pix.width / 4;
+		avg_lum = get_luminosity_bayer8(buf, fmt);
 		break;
 
 	case V4L2_PIX_FMT_RGB24:

lib/libv4lconvert/processing/libv4lprocessing.c +4 -0

index b061f50..b98d024 100644
@@ -164,6 +165,10 @@ void v4lprocessing_processing(struct v4lprocessing_data *data,
 	case V4L2_PIX_FMT_SGRBG8:
 	case V4L2_PIX_FMT_SBGGR8:
 	case V4L2_PIX_FMT_SRGGB8:
+	case V4L2_PIX_FMT_SGBRG10:
+	case V4L2_PIX_FMT_SGRBG10:
+	case V4L2_PIX_FMT_SBGGR10:
+	case V4L2_PIX_FMT_SRGGB10:
 	case V4L2_PIX_FMT_RGB24:
 	case V4L2_PIX_FMT_BGR24:
 		break;

Quoted 27 lines

Quoted 4 lines

There are two separate things here:

1) Autofoucs for a device that doesn't use subdev API
2) libv4l2 support for devices that require MC and subdev API

Actually there are three: 0) autogain. Unfortunately, I need autogain first before autofocus has a chance...

And that means... bayer10 support for autogain.

Plus, I changed avg_lum to long long. Quick calculation tells me int could overflow with few megapixel sensor.

Oh, btw http://ytse.tricolour.net/docs/LowLightOptimization.html no longer works.

Regards,
Pavel

lib/libv4lconvert/processing/autogain.c +0 -0

index c6866d6..0b52d0f 100644
@@ -68,6 +71,41 @@ static void autogain_adjust(struct v4l2_queryctrl *ctrl, int *value,
 	}
 }
 

lib/libv4lconvert/processing/autogain.c static void autogain_adjust(struct v4l2_queryctrl *ctrl, int *value, · line 83

Quoted 10 lines

lib/libv4lconvert/processing/autogain.c +10 -0

+static int get_luminosity_bayer10(uint16_t *buf, const struct v4l2_format *fmt)
+{
+	long long avg_lum = 0;
+	int x, y;
+	
+	buf += fmt->fmt.pix.height * fmt->fmt.pix.bytesperline / 4 +
+		fmt->fmt.pix.width / 4;
+
+	for (y = 0; y < fmt->fmt.pix.height / 2; y++) {
+		for (x = 0; x < fmt->fmt.pix.width / 2; x++)

That would take some time :). AIUI, we have NEON support in ARM kernels (CONFIG_KERNEL_MODE_NEON), I wonder if it makes sense (me) to convert the above loop to NEON-optimized when it comes to it? Are there any drawbacks in using NEON code in kernel?

Hi!

Quoted 27 lines

Quoted 27 lines

Quoted 4 lines

There are two separate things here:

1) Autofoucs for a device that doesn't use subdev API
2) libv4l2 support for devices that require MC and subdev API

Actually there are three: 0) autogain. Unfortunately, I need autogain first before autofocus has a chance...

And that means... bayer10 support for autogain.

Plus, I changed avg_lum to long long. Quick calculation tells me int could overflow with few megapixel sensor.

Oh, btw http://ytse.tricolour.net/docs/LowLightOptimization.html no longer works.

Regards,
Pavel

lib/libv4lconvert/processing/autogain.c +0 -0

index c6866d6..0b52d0f 100644
@@ -68,6 +71,41 @@ static void autogain_adjust(struct v4l2_queryctrl *ctrl, int *value,
 	}
 }
 

Quoted 15 lines

lib/libv4lconvert/processing/autogain.c static void autogain_adjust(struct v4l2_queryctrl *ctrl, int *value, · line 83

Quoted 10 lines

lib/libv4lconvert/processing/autogain.c +10 -0

+static int get_luminosity_bayer10(uint16_t *buf, const struct v4l2_format *fmt)
+{
+	long long avg_lum = 0;
+	int x, y;
+	
+	buf += fmt->fmt.pix.height * fmt->fmt.pix.bytesperline / 4 +
+		fmt->fmt.pix.width / 4;
+
+	for (y = 0; y < fmt->fmt.pix.height / 2; y++) {
+		for (x = 0; x < fmt->fmt.pix.width / 2; x++)

That would take some time :). AIUI, we have NEON support in ARM kernels (CONFIG_KERNEL_MODE_NEON), I wonder if it makes sense (me) to convert the above loop to NEON-optimized when it comes to it? Are there any drawbacks in using NEON code in kernel?

Well, thanks for offer. This is actualy libv4l2.

But I'd say NEON conversion is not neccessary anytime soon. First, this is just trying to get average luminosity. We can easily skip quite a lot of pixels, and still get reasonable answer.

Second, omap3isp actually has a hardware block computing statistics for us. We just don't use it for simplicity.

(But if you want to play with camera, I'll get you patches; there's ton of work to be done, both kernel and userspace :-).

Oh, somehow I got confused that this is kernel code :)

Quoted 7 lines

But I'd say NEON conversion is not neccessary anytime soon. First, this is just trying to get average luminosity. We can easily skip quite a lot of pixels, and still get reasonable answer.

Second, omap3isp actually has a hardware block computing statistics for us. We just don't use it for simplicity.

Right, I forgot about that.

Quoted 2 lines

(But if you want to play with camera, I'll get you patches; there's ton of work to be done, both kernel and userspace :-).

Well, I saw a low hanging fruit I thought I can convert to NEON in a day or two, while having some rest from the huge "project" I am devoting all my spare time recently (rebasing hildon/maemo 5 on top of devuan Jessie). Still, if there is something relatively small to be done, just email me and I'll have a look.

Quoted 7 lines

lib/libv4lconvert/processing/autogain.c +7 -0

+static int get_luminosity_bayer10(uint16_t *buf, const struct v4l2_format *fmt)
+{
+	long long avg_lum = 0;
+	int x, y;
+	
+	buf += fmt->fmt.pix.height * fmt->fmt.pix.bytesperline / 4 +
+		fmt->fmt.pix.width / 4;

lib/libv4lconvert/processing/autogain.c static void autogain_adjust(struct v4l2_queryctrl *ctrl, int *value, · line 83

Quoted 3 lines

lib/libv4lconvert/processing/autogain.c +3 -0

+
+	for (y = 0; y < fmt->fmt.pix.height / 2; y++) {
+		for (x = 0; x < fmt->fmt.pix.width / 2; x++)

That would take some time :). AIUI, we have NEON support in ARM kernels (CONFIG_KERNEL_MODE_NEON), I wonder if it makes sense (me) to convert the above loop to NEON-optimized when it comes to it? Are there any drawbacks in using NEON code in kernel?

Quoted 7 lines · Ivaylo Dimitrov

Quoted 7 lines

lib/libv4lconvert/processing/autogain.c +7 -0

+static int get_luminosity_bayer10(uint16_t *buf, const struct v4l2_format *fmt)
+{
+	long long avg_lum = 0;
+	int x, y;
+	
+	buf += fmt->fmt.pix.height * fmt->fmt.pix.bytesperline / 4 +
+		fmt->fmt.pix.width / 4;

Quoted 8 lines

lib/libv4lconvert/processing/autogain.c static void autogain_adjust(struct v4l2_queryctrl *ctrl, int *value, · line 83

Quoted 3 lines

lib/libv4lconvert/processing/autogain.c +3 -0

+
+	for (y = 0; y < fmt->fmt.pix.height / 2; y++) {
+		for (x = 0; x < fmt->fmt.pix.width / 2; x++)

That would take some time :). AIUI, we have NEON support in ARM kernels (CONFIG_KERNEL_MODE_NEON), I wonder if it makes sense (me) to convert the above loop to NEON-optimized when it comes to it? Are there any drawbacks in using NEON code in kernel?

Using neon without the VFP state saved and restored corrupts userspace's FP state. So, you have to save the entire VFP state to use neon in kernel mode. There are helper functions for this: kernel_neon_begin() and kernel_neon_end().