Software-only image processing for Intel "complex" cameras
Hi!
On Nokia N900, I have similar problems as Intel IPU3 hardware.
Meeting notes say that pure software implementation is not fast enough, but that it may be useful for debugging. It would be also useful for me on N900, and probably useful for processing "raw" images from digital cameras.
There is sensor part, and memory-to-memory part, right? What is the format of data from the sensor part? What operations would be expensive on the CPU? If we did everthing on the CPU, what would be maximum resolution where we could still manage it in real time?
Would it be possible to get access to machine with IPU3, or would there be someone willing to test libv4l2 patches?
Thanks and best regards,
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 10 lines · Pavel Machek
Hi!
On Nokia N900, I have similar problems as Intel IPU3 hardware.
Meeting notes say that pure software implementation is not fast enough, but that it may be useful for debugging. It would be also useful for me on N900, and probably useful for processing "raw" images
from digital cameras.
Quoted 4 lines
There is sensor part, and memory-to-memory part, right? What is the format of data from the sensor part? What operations would be expensive on the CPU? If we did everthing on the CPU, what would be maximum resolution where we could still manage it in real time?
The IPU3 sensor produce a vendor specific form of bayer. If we manage to implement support for this format, it would likely be done in software. I don't think anyone can answer your other questions has no one have ever implemented this, hence measure performance.
Quoted 8 lines
Would it be possible to get access to machine with IPU3, or would there be someone willing to test libv4l2 patches?
Thanks and best regards,
Pavel
Hi!
Quoted 13 lines
Quoted 8 lines
On Nokia N900, I have similar problems as Intel IPU3 hardware.
Meeting notes say that pure software implementation is not fast enough, but that it may be useful for debugging. It would be also useful for me on N900, and probably useful for processing "raw" images
from digital cameras.Quoted 4 lines
There is sensor part, and memory-to-memory part, right? What is the format of data from the sensor part? What operations would be expensive on the CPU? If we did everthing on the CPU, what would be maximum resolution where we could still manage it in real time?
Quoted 4 lines
The IPU3 sensor produce a vendor specific form of bayer. If we manage to implement support for this format, it would likely be done in software. I don't think anyone can answer your other questions has no one have ever implemented this, hence measure performance.
I believe Intel has some estimates.
What is the maximum resolution of camera in the current Dell systems?
Thanks,
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 23 lines
-----Original Message-----
From: Pavel Machek [mailto:[email protected]]
Sent: Wednesday, June 20, 2018 4:12 PM
To: Nicolas Dufresne
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Limonciello, Mario
Subject: Re: Software-only image processing for Intel "complex" camerasHi!
Quoted 13 lines
Quoted 12 lines
On Nokia N900, I have similar problems as Intel IPU3 hardware.
Meeting notes say that pure software implementation is not fast enough, but that it may be useful for debugging. It would be also useful for me on N900, and probably useful for processing "raw" images
from digital cameras.There is sensor part, and memory-to-memory part, right? What is the format of data from the sensor part? What operations would be expensive on the CPU? If we did everthing on the CPU, what would be maximum resolution where we could still manage it in real time?
Quoted 9 lines
Quoted 4 lines
The IPU3 sensor produce a vendor specific form of bayer. If we manage to implement support for this format, it would likely be done in software. I don't think anyone can answer your other questions has no one have ever implemented this, hence measure performance.
I believe Intel has some estimates.
What is the maximum resolution of camera in the current Dell systems?
5M camera sensor HW spec:
2592x1944
8M camera sensor HW spec:
3264x2448
Thanks,
Em Thu, 21 Jun 2018 13:41:37 +0000
<[email protected]> escreveu:
Quoted 30 lines
Quoted 30 lines
-----Original Message-----
From: Pavel Machek [mailto:[email protected]]
Sent: Wednesday, June 20, 2018 4:12 PM
To: Nicolas Dufresne
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Limonciello, Mario
Subject: Re: Software-only image processing for Intel "complex" camerasHi!
Quoted 13 lines
Quoted 8 lines
On Nokia N900, I have similar problems as Intel IPU3 hardware.
Meeting notes say that pure software implementation is not fast enough, but that it may be useful for debugging. It would be also useful for me on N900, and probably useful for processing "raw" images
from digital cameras.Quoted 4 lines
There is sensor part, and memory-to-memory part, right? What is the format of data from the sensor part? What operations would be expensive on the CPU? If we did everthing on the CPU, what would be maximum resolution where we could still manage it in real time?
Quoted 4 lines
The IPU3 sensor produce a vendor specific form of bayer. If we manage to implement support for this format, it would likely be done in software. I don't think anyone can answer your other questions has no one have ever implemented this, hence measure performance.
I believe Intel has some estimates.
Quoted 8 lines
Quoted 2 lines
What is the maximum resolution of camera in the current Dell systems?
5M camera sensor HW spec:
2592x19448M camera sensor HW spec:
3264x2448
Looking at the ipu3 driver, I'm wandering how the library would identify the system components. The way VIDIOC_QUERYCAP is currently implemented doesn't help at all:
static int cio2_v4l2_querycap(struct file *file, void *fh,
struct v4l2_capability *cap)
{
struct cio2_device *cio2 = video_drvdata(file);
strlcpy(cap->driver, CIO2_NAME, sizeof(cap->driver));
strlcpy(cap->card, CIO2_DEVICE_NAME, sizeof(cap->card));
snprintf(cap->bus_info, sizeof(cap->bus_info),
"PCI:%s", pci_name(cio2->pci_dev));
return 0;
}In order to allow the library to know more about the hardware, it would likely need to expose some model number to userspace. Ok, userspace could always call dmidecode, but that requires root privileges. We really don't want media apps to require root.
So, perhaps caps->cap (or a MC caps equivalent call) should, instead be filled with values obtained from the BIOS DMI tables with some logic based on enum dmi_field:
enum dmi_field {
DMI_NONE,
DMI_BIOS_VENDOR,
DMI_BIOS_VERSION,
DMI_BIOS_DATE,
DMI_SYS_VENDOR,
DMI_PRODUCT_NAME,
DMI_PRODUCT_VERSION,
DMI_PRODUCT_SERIAL,
DMI_PRODUCT_UUID,
DMI_PRODUCT_FAMILY,
DMI_BOARD_VENDOR,
DMI_BOARD_NAME,
DMI_BOARD_VERSION,
DMI_BOARD_SERIAL,
DMI_BOARD_ASSET_TAG,
DMI_CHASSIS_VENDOR,
DMI_CHASSIS_TYPE,
DMI_CHASSIS_VERSION,
DMI_CHASSIS_SERIAL,
DMI_CHASSIS_ASSET_TAG,
DMI_STRING_MAX,
};
e. g. something like:
board_vendor = dmi_get_system_info(DMI_BOARD_VENDOR); board_name = dmi_get_system_info(DMI_BOARD_NAME); board_version = dmi_get_system_info(DMI_BOARD_NAME); product_name = dmi_get_system_info(DMI_PRODUCT_NAME); product_version = dmi_get_system_info(DMI_PRODUCT_VERSION); sprintf(dev->cap, "%s:%s:%s:%s", board_vendor, board_name, board_version, product_name, product_version);
(the real code should check if the values are filled, as not all BIOS vendors use the same DMI fields)
With that, the library can auto-adjust without needing to run anything as root.
Cheers,
Mauro
Quoted 110 lines
-----Original Message-----
From: Mauro Carvalho Chehab [mailto:[email protected]]
Sent: Thursday, June 21, 2018 1:50 PM
To: Limonciello, Mario
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: Software-only image processing for Intel "complex" camerasEm Thu, 21 Jun 2018 13:41:37 +0000
<[email protected]> escreveu:Quoted 30 lines
Quoted 30 lines
-----Original Message-----
From: Pavel Machek [mailto:[email protected]]
Sent: Wednesday, June 20, 2018 4:12 PM
To: Nicolas Dufresne
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Limonciello, Mario
Subject: Re: Software-only image processing for Intel "complex" camerasHi!
Quoted 13 lines
Quoted 8 lines
On Nokia N900, I have similar problems as Intel IPU3 hardware.
Meeting notes say that pure software implementation is not fast enough, but that it may be useful for debugging. It would be also useful for me on N900, and probably useful for processing "raw" images
from digital cameras.Quoted 4 lines
There is sensor part, and memory-to-memory part, right? What is the format of data from the sensor part? What operations would be expensive on the CPU? If we did everthing on the CPU, what would be maximum resolution where we could still manage it in real time?
Quoted 4 lines
The IPU3 sensor produce a vendor specific form of bayer. If we manage to implement support for this format, it would likely be done in software. I don't think anyone can answer your other questions has no one have ever implemented this, hence measure performance.
I believe Intel has some estimates.
Quoted 8 lines
Quoted 2 lines
What is the maximum resolution of camera in the current Dell systems?
5M camera sensor HW spec:
2592x19448M camera sensor HW spec:
3264x2448Looking at the ipu3 driver, I'm wandering how the library would identify the system components. The way
VIDIOC_QUERYCAPis currently implemented doesn't help at all:static int cio2_v4l2_querycap(struct file *file, void *fh,
structv4l2_capability*cap){ struct cio2_device *cio2 = video_drvdata(file); strlcpy(cap->driver, CIO2_NAME, sizeof(cap->driver)); strlcpy(cap->card, CIO2_DEVICE_NAME, sizeof(cap->card)); snprintf(cap->bus_info, sizeof(cap->bus_info), "PCI:%s", pci_name(cio2->pci_dev)); return 0; }In order to allow the library to know more about the hardware, it would likely need to expose some model number to userspace. Ok, userspace could always call dmidecode, but that requires root privileges. We really don't want media apps to require root.
So, perhaps
caps->cap(or a MC caps equivalent call) should, instead be filled with values obtained from the BIOS DMI tables with some logic based on enumdmi_field:enum
dmi_field{
DMI_NONE,
DMI_BIOS_VENDOR,
DMI_BIOS_VERSION,
DMI_BIOS_DATE,
DMI_SYS_VENDOR,
DMI_PRODUCT_NAME,
DMI_PRODUCT_VERSION,
DMI_PRODUCT_SERIAL,
DMI_PRODUCT_UUID,
DMI_PRODUCT_FAMILY,
DMI_BOARD_VENDOR,
DMI_BOARD_NAME,
DMI_BOARD_VERSION,
DMI_BOARD_SERIAL,
DMI_BOARD_ASSET_TAG,
DMI_CHASSIS_VENDOR,
DMI_CHASSIS_TYPE,
DMI_CHASSIS_VERSION,
DMI_CHASSIS_SERIAL,
DMI_CHASSIS_ASSET_TAG,
DMI_STRING_MAX,
};e. g. something like:
board_vendor = dmi_get_system_info(DMI_BOARD_VENDOR); board_name = dmi_get_system_info(DMI_BOARD_NAME); board_version = dmi_get_system_info(DMI_BOARD_NAME); product_name = dmi_get_system_info(DMI_PRODUCT_NAME); product_version = dmi_get_system_info(DMI_PRODUCT_VERSION); sprintf(dev->cap, "%s:%s:%s:%s", board_vendor, board_name,
Quoted 8 lines
board_version,product_name,product_version);(the real code should check if the values are filled, as not all BIOS vendors use the same DMI fields)
With that, the library can auto-adjust without needing to run anything as root.
Well actually most of those fields you're interested in are already exposed to userspace through sysfs /sys/class/dmi/id/
Can't the library just pull them from there?
The one field that isn't exposed is actually the one I think you should key off of though:
Product SKU number. So I would propose as part of this change that should start to get exposed to userspace too.
The reasoning is I'm a little concerned in taking an approach that goes off of marketing model number specifically because it's creating an assumption that all systems with that model number have the exact same components.
It's possible for two systems to have the same model number but to second source for example. This might not affect complex cameras, but I just want to make sure it's taken into consideration. At least going off of Product SKU will better narrow it down.
Em Thu, 21 Jun 2018 18:58:37 +0000
<[email protected]> escreveu:
Quoted 115 lines
Quoted 115 lines
-----Original Message-----
From: Mauro Carvalho Chehab [mailto:[email protected]]
Sent: Thursday, June 21, 2018 1:50 PM
To: Limonciello, Mario
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: Software-only image processing for Intel "complex" camerasEm Thu, 21 Jun 2018 13:41:37 +0000
<[email protected]> escreveu:Quoted 30 lines
Quoted 30 lines
-----Original Message-----
From: Pavel Machek [mailto:[email protected]]
Sent: Wednesday, June 20, 2018 4:12 PM
To: Nicolas Dufresne
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Limonciello, Mario
Subject: Re: Software-only image processing for Intel "complex" camerasHi!
Quoted 13 lines
Quoted 8 lines
On Nokia N900, I have similar problems as Intel IPU3 hardware.
Meeting notes say that pure software implementation is not fast enough, but that it may be useful for debugging. It would be also useful for me on N900, and probably useful for processing "raw" images
from digital cameras.Quoted 4 lines
There is sensor part, and memory-to-memory part, right? What is the format of data from the sensor part? What operations would be expensive on the CPU? If we did everthing on the CPU, what would be maximum resolution where we could still manage it in real time?
Quoted 4 lines
The IPU3 sensor produce a vendor specific form of bayer. If we manage to implement support for this format, it would likely be done in software. I don't think anyone can answer your other questions has no one have ever implemented this, hence measure performance.
I believe Intel has some estimates.
Quoted 8 lines
Quoted 2 lines
What is the maximum resolution of camera in the current Dell systems?
5M camera sensor HW spec:
2592x19448M camera sensor HW spec:
3264x2448Looking at the ipu3 driver, I'm wandering how the library would identify the system components. The way
VIDIOC_QUERYCAPis currently implemented doesn't help at all:static int cio2_v4l2_querycap(struct file *file, void *fh,
structv4l2_capability*cap){ struct cio2_device *cio2 = video_drvdata(file); strlcpy(cap->driver, CIO2_NAME, sizeof(cap->driver)); strlcpy(cap->card, CIO2_DEVICE_NAME, sizeof(cap->card)); snprintf(cap->bus_info, sizeof(cap->bus_info), "PCI:%s", pci_name(cio2->pci_dev)); return 0; }In order to allow the library to know more about the hardware, it would likely need to expose some model number to userspace. Ok, userspace could always call dmidecode, but that requires root privileges. We really don't want media apps to require root.
So, perhaps
caps->cap(or a MC caps equivalent call) should, instead be filled with values obtained from the BIOS DMI tables with some logic based on enumdmi_field:enum
dmi_field{
DMI_NONE,
DMI_BIOS_VENDOR,
DMI_BIOS_VERSION,
DMI_BIOS_DATE,
DMI_SYS_VENDOR,
DMI_PRODUCT_NAME,
DMI_PRODUCT_VERSION,
DMI_PRODUCT_SERIAL,
DMI_PRODUCT_UUID,
DMI_PRODUCT_FAMILY,
DMI_BOARD_VENDOR,
DMI_BOARD_NAME,
DMI_BOARD_VERSION,
DMI_BOARD_SERIAL,
DMI_BOARD_ASSET_TAG,
DMI_CHASSIS_VENDOR,
DMI_CHASSIS_TYPE,
DMI_CHASSIS_VERSION,
DMI_CHASSIS_SERIAL,
DMI_CHASSIS_ASSET_TAG,
DMI_STRING_MAX,
};e. g. something like:
board_vendor = dmi_get_system_info(DMI_BOARD_VENDOR); board_name = dmi_get_system_info(DMI_BOARD_NAME); board_version = dmi_get_system_info(DMI_BOARD_NAME); product_name = dmi_get_system_info(DMI_PRODUCT_NAME); product_version = dmi_get_system_info(DMI_PRODUCT_VERSION); sprintf(dev->cap, "%s:%s:%s:%s", board_vendor, board_name,
board_version,product_name,product_version);(the real code should check if the values are filled, as not all BIOS vendors use the same DMI fields)
Quoted 7 lines
Quoted 3 lines
With that, the library can auto-adjust without needing to run anything as root.
Well actually most of those fields you're interested in are already exposed to userspace through sysfs
/sys/class/dmi/id/Can't the library just pull them from there?
Good point. Yeah, the library could use them.
Quoted 11 lines
The one field that isn't exposed is actually the one I think you should key off of though:
Product SKU number. So I would propose as part of this change that should start to get exposed to userspace too.The reasoning is I'm a little concerned in taking an approach that goes off of marketing model number specifically because it's creating an assumption that all systems with that model number have the exact same components.
It's possible for two systems to have the same model number but to second source for example. This might not affect complex cameras, but I just want to make sure it's taken into consideration. At least going off of Product SKU will better narrow it down.
Yeah, the library should be able to uniquely match whatever components are inside a system, and just the marketing model number might not be enough.
Right now, only dmidecode seems to get SKU number.
On a quick look, it seems that a patch adding support for it was already proposed:
https://lkml.org/lkml/2018/4/24/1166
And got accepted:
https://lkml.org/lkml/2018/4/27/96
So, it seems we'll be covered for newer Kernel versions.
Cheers,
Mauro
Em Fri, 22 Jun 2018 06:08:50 +0900
Mauro Carvalho Chehab <[email protected]> escreveu:
Quoted 145 lines
Em Thu, 21 Jun 2018 18:58:37 +0000
<[email protected]> escreveu:Quoted 115 lines
Quoted 115 lines
-----Original Message-----
From: Mauro Carvalho Chehab [mailto:[email protected]]
Sent: Thursday, June 21, 2018 1:50 PM
To: Limonciello, Mario
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: Software-only image processing for Intel "complex" camerasEm Thu, 21 Jun 2018 13:41:37 +0000
<[email protected]> escreveu:Quoted 30 lines
Quoted 30 lines
-----Original Message-----
From: Pavel Machek [mailto:[email protected]]
Sent: Wednesday, June 20, 2018 4:12 PM
To: Nicolas Dufresne
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Limonciello, Mario
Subject: Re: Software-only image processing for Intel "complex" camerasHi!
Quoted 13 lines
Quoted 8 lines
On Nokia N900, I have similar problems as Intel IPU3 hardware.
Meeting notes say that pure software implementation is not fast enough, but that it may be useful for debugging. It would be also useful for me on N900, and probably useful for processing "raw" images
from digital cameras.Quoted 4 lines
There is sensor part, and memory-to-memory part, right? What is the format of data from the sensor part? What operations would be expensive on the CPU? If we did everthing on the CPU, what would be maximum resolution where we could still manage it in real time?
Quoted 4 lines
The IPU3 sensor produce a vendor specific form of bayer. If we manage to implement support for this format, it would likely be done in software. I don't think anyone can answer your other questions has no one have ever implemented this, hence measure performance.
I believe Intel has some estimates.
Quoted 8 lines
Quoted 2 lines
What is the maximum resolution of camera in the current Dell systems?
5M camera sensor HW spec:
2592x19448M camera sensor HW spec:
3264x2448Looking at the ipu3 driver, I'm wandering how the library would identify the system components. The way
VIDIOC_QUERYCAPis currently implemented doesn't help at all:static int cio2_v4l2_querycap(struct file *file, void *fh,
structv4l2_capability*cap){ struct cio2_device *cio2 = video_drvdata(file); strlcpy(cap->driver, CIO2_NAME, sizeof(cap->driver)); strlcpy(cap->card, CIO2_DEVICE_NAME, sizeof(cap->card)); snprintf(cap->bus_info, sizeof(cap->bus_info), "PCI:%s", pci_name(cio2->pci_dev)); return 0; }In order to allow the library to know more about the hardware, it would likely need to expose some model number to userspace. Ok, userspace could always call dmidecode, but that requires root privileges. We really don't want media apps to require root.
So, perhaps
caps->cap(or a MC caps equivalent call) should, instead be filled with values obtained from the BIOS DMI tables with some logic based on enumdmi_field:enum
dmi_field{
DMI_NONE,
DMI_BIOS_VENDOR,
DMI_BIOS_VERSION,
DMI_BIOS_DATE,
DMI_SYS_VENDOR,
DMI_PRODUCT_NAME,
DMI_PRODUCT_VERSION,
DMI_PRODUCT_SERIAL,
DMI_PRODUCT_UUID,
DMI_PRODUCT_FAMILY,
DMI_BOARD_VENDOR,
DMI_BOARD_NAME,
DMI_BOARD_VERSION,
DMI_BOARD_SERIAL,
DMI_BOARD_ASSET_TAG,
DMI_CHASSIS_VENDOR,
DMI_CHASSIS_TYPE,
DMI_CHASSIS_VERSION,
DMI_CHASSIS_SERIAL,
DMI_CHASSIS_ASSET_TAG,
DMI_STRING_MAX,
};e. g. something like:
board_vendor = dmi_get_system_info(DMI_BOARD_VENDOR); board_name = dmi_get_system_info(DMI_BOARD_NAME); board_version = dmi_get_system_info(DMI_BOARD_NAME); product_name = dmi_get_system_info(DMI_PRODUCT_NAME); product_version = dmi_get_system_info(DMI_PRODUCT_VERSION); sprintf(dev->cap, "%s:%s:%s:%s", board_vendor, board_name,
board_version,product_name,product_version);(the real code should check if the values are filled, as not all BIOS vendors use the same DMI fields)
Quoted 7 lines
Quoted 3 lines
With that, the library can auto-adjust without needing to run anything as root.
Well actually most of those fields you're interested in are already exposed to userspace through sysfs
/sys/class/dmi/id/Can't the library just pull them from there?
Good point. Yeah, the library could use them.
Quoted 11 lines
The one field that isn't exposed is actually the one I think you should key off of though:
Product SKU number. So I would propose as part of this change that should start to get exposed to userspace too.The reasoning is I'm a little concerned in taking an approach that goes off of marketing model number specifically because it's creating an assumption that all systems with that model number have the exact same components.
It's possible for two systems to have the same model number but to second source for example. This might not affect complex cameras, but I just want to make sure it's taken into consideration. At least going off of Product SKU will better narrow it down.
Yeah, the library should be able to uniquely match whatever components are inside a system, and just the marketing model number might not be enough.
Right now, only dmidecode seems to get SKU number.
Quoted 8 lines
On a quick look, it seems that a patch adding support for it was already proposed:
https://lkml.org/lkml/2018/4/24/1166And got accepted:
https://lkml.org/lkml/2018/4/27/96So, it seems we'll be covered for newer Kernel versions.
Just occurred to me that this is a major difference about how complex camera hardware detection should happen. On ARM-based devices, the hardware topology comes from Open Firmware. It uniquely describes how the board is wired. That reflects the way the media controller exposes the hardware, as each platform driver reads a set of properties via of_property_foo() functions.
On "normal" PCI drivers (bttv, cx88, ...), there's a PCI table filled with PCI vendor IDs and some complex logic with uses that in order to identify the hardware components and to load the associated drivers.
With IPU3, the device appears as a PCI device. The driver doesn't do any of_property_foo() call, nor it has a device table with would be used to identify what sensor is associated to what device.
So, on a quick look there, I was unable to see how it detects what sensors are connected to an specific hardware instance and how it would load the associated sensor drivers.
For us to be able to stick with an example, let's assume the Dell Latitude 5285 laptop [1]. What sensors are present on it? What's its PCI ID?
[1] https://www.spinics.net/lists/linux-usb/msg167478.html
https://www.spinics.net/lists/linux-usb/msg163617.html
Jerry/Jian,
Could you please shed a light about how a Dell 5285 hardware would be detected by the IPU3 driver and what MC graph is created by the current driver?
Cheers,
Mauro
Hi Mauro,
Quoted 13 lines
-----Original Message-----
From: [email protected] [mailto:linux-media- [email protected]] On Behalf Of Mauro Carvalho Chehab
Sent: Friday, June 22, 2018 5:41 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Hu, Jerry W <[email protected]>; Zheng, Jian Xu <[email protected]>
Subject: Re: Software-only image processing for Intel "complex" camerasEm Fri, 22 Jun 2018 06:08:50 +0900
Mauro Carvalho Chehab <[email protected]> escreveu:
Quoted 8 lines
Quoted 3 lines
Em Thu, 21 Jun 2018 18:58:37 +0000
<[email protected]> escreveu:Jerry/Jian,
Could you please shed a light about how a Dell 5285 hardware would be detected by the IPU3 driver and what MC graph is created by the current driver?
Sure, Mauro. I need to check the information on the Dell 5285. IPU3 driver are detected by PCI vendor id and device id.
IPU3 CIO2 MC graph is:
Sensor A -> IPU3 CSI2 0(subdev) -> IPU3 CIO2 0 (video node) Sensor B -> IPU3 CSI2 1(subdev) -> IPU3 CIO2 1 (video node)
IPU3 IMGu MC graph is:
IMGu subdev inputs:
ipu3-imgu input [Pad0] => ipu3-imgu [Pad0]
ipu3-imgu parameters [Pad0] => ipu3-imgu [Pad1]
IMGu subdev outputs:
ipu3-imgu [Pad2] => ipu3-imgu output [Pad0] ipu3-imgu [Pad3] => ipu3-imgu viewfinder [Pad0] ipu3-imgu [Pad4] => ipu3-imgu postview [Pad0] ipu3-imgu [Pad5] => ipu3-imgu 3a stat [Pad0]
Best Regards,
Jianxu(Clive) Zheng
Em Mon, 25 Jun 2018 09:48:56 +0000
"Zheng, Jian Xu" <[email protected]> escreveu:
Quoted 24 lines
Hi Mauro,
Quoted 13 lines
-----Original Message-----
From: [email protected] [mailto:linux-media- [email protected]] On Behalf Of Mauro Carvalho Chehab
Sent: Friday, June 22, 2018 5:41 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Hu, Jerry W <[email protected]>; Zheng, Jian Xu <[email protected]>
Subject: Re: Software-only image processing for Intel "complex" camerasEm Fri, 22 Jun 2018 06:08:50 +0900
Mauro Carvalho Chehab <[email protected]> escreveu:Quoted 8 lines
Quoted 3 lines
Em Thu, 21 Jun 2018 18:58:37 +0000
<[email protected]> escreveu:Jerry/Jian,
Could you please shed a light about how a Dell 5285 hardware would be detected by the IPU3 driver and what MC graph is created by the current driver?
Quoted 6 lines
Sure, Mauro. I need to check the information on the Dell 5285. IPU3 driver are detected by PCI vendor id and device id.
IPU3 CIO2 MC graph is:
Sensor A -> IPU3 CSI2 0(subdev) -> IPU3 CIO2 0 (video node) Sensor B -> IPU3 CSI2 1(subdev) -> IPU3 CIO2 1 (video node)
How does it detect what driver should be loaded for Sensor A and B?
Does the ACPI table identifies the sensors? If so, how the driver associates an specific PCI vendor ID with the corresponding sensor settings?
Quoted 6 lines
IPU3 IMGu MC graph is:
IMGu subdev inputs:
ipu3-imgu input [Pad0] => ipu3-imgu [Pad0]
ipu3-imgu parameters [Pad0] => ipu3-imgu [Pad1]
Quoted 8 lines
IMGu subdev outputs:
ipu3-imgu [Pad2] => ipu3-imgu output [Pad0] ipu3-imgu [Pad3] => ipu3-imgu viewfinder [Pad0] ipu3-imgu [Pad4] => ipu3-imgu postview [Pad0] ipu3-imgu [Pad5] => ipu3-imgu 3a stat [Pad0]Best Regards,
Jianxu(Clive) Zheng
Thanks,
Mauro
Hi Mauro,
Quoted 34 lines · Mauro Carvalho Chehab
Em Mon, 25 Jun 2018 09:48:56 +0000
"Zheng, Jian Xu" <[email protected]> escreveu:Quoted 30 lines
Hi Mauro,
Quoted 13 lines
-----Original Message-----
From: [email protected] [mailto:linux-media- [email protected]] On Behalf Of Mauro Carvalho Chehab
Sent: Friday, June 22, 2018 5:41 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Hu, Jerry W <[email protected]>; Zheng, Jian Xu <[email protected]>
Subject: Re: Software-only image processing for Intel "complex" camerasEm Fri, 22 Jun 2018 06:08:50 +0900
Mauro Carvalho Chehab <[email protected]> escreveu:Quoted 8 lines
Quoted 3 lines
Em Thu, 21 Jun 2018 18:58:37 +0000
<[email protected]> escreveu:Jerry/Jian,
Could you please shed a light about how a Dell 5285 hardware would be detected by the IPU3 driver and what MC graph is created by the current driver?
Sure, Mauro. I need to check the information on the Dell 5285. IPU3 driver are detected by PCI vendor id and device id.
IPU3 CIO2 MC graph is:
Sensor A -> IPU3 CSI2 0(subdev) -> IPU3 CIO2 0 (video node) Sensor B -> IPU3 CSI2 1(subdev) -> IPU3 CIO2 1 (video node)
I don't think your questions below were answered. I'll try to answer since I've been looking at this, but Sakari or others can correct me if I got something wrong.
Quoted 2 lines
How does it detect what driver should be loaded for Sensor A and B?
It depends on the firmware interface used to describe the hardware topology, for ACPI the sensors are described in a DSDT table and match using an entry in the struct acpi_device_id table in the camera driver. For DT the sensors are described in a DT node with a compatible string that matches an entry in the struct of_device_id table, or an entry in the struct i2c_device_id table as a fallback.
Quoted 4 lines
Does the ACPI table identifies the sensors? If so, how the driver associates an specific PCI vendor ID with the corresponding sensor settings?
Camera sensors are usually not PCI devices, but I2C devices. And yes, these are described in an ACPI table (DSDT) as mentioned before. For example in a laptop I've the following on an disassembled DSDT table:
Device (CAM0)
{
Name (_ADR, Zero) // _ADR: Address
Name (_HID, "OVTI5648") // _HID: Hardware ID
Name (_CID, "OVTI5648") // _CID: Compatible ID
Name (_DDN, "OV5648-CRDD") // _DDN: DOS Device Name
Name (_UID, Zero) // _UID: Unique ID
Name (_DEP, Package (0x02) // _DEP: Dependencies
{
PMI0,
I2C4
})Now how the drivers get loaded / registered and how the media entities are associated is somewhat orthogonal (although the information to do both is usually encoded in the same place).
In the case of OF, the relation between the bridge device and the sensors is encoded in the DT using the video interface / graph binding [0, 1]. For ACPI, the same information is encoded in an ACPI table using a _DSD extension [2], which is quite similar to the DT binding.
Since both hardware descriptions use the same properties (port, endpoint, etc) and semantics, drivers can use the V4L2 fwnode kAPI to parse this information regardless of the underlying firmware interface used to encoded this relationship. This allows the same camera drivers to be used with either DT or ACPI.
So a bridge driver will register a v4l2 async notifier on probe and camera drivers will register the subdevices using the v4l2 async interface. That way the v4l2 async layer can bind the two and call the bridge device .bound (each time that a pending device is registered) and .complete (when there are no pending devices remaining to be registered) callbacks.
A problem I found though is that [2] is Linux specific, so the firmware for a machine meant to be used with Windows (i.e: Microsoft Surface 2-in-1 computers) will have a DSDT table that's not compatible with the v4l2 fwnode layer. One option is to override [4] the DSDT table with one that contains the information expected by Linux, for example by having the table in the initramfs [5]. The problem is that it seems the distros (at least in the Fedora case) don't want to get into the business of shipping machine specific ACPI tables.
Another option could be to have a shim layer in the kernel that translates the information in the DSDT as used by Windows to what's expected by Linux. This is an option that I haven't investigated yet, but wanted to know your opinion if this could be a feasible approach. Or if you have any other suggestions on how we should proceed with this.
[0]: https://www.kernel.org/doc/Documentation/devicetree/bindings/media/video-interfaces.txt
[1]: https://www.kernel.org/doc/Documentation/devicetree/bindings/graph.txt
[2]: https://www.kernel.org/doc/Documentation/acpi/dsd/graph.txt
[3]: https://linuxtv.org/downloads/v4l-dvb-apis/kapi/v4l2-fwnode.html
[4]: https://www.kernel.org/doc/Documentation/acpi/dsdt-override.txt
[5]: https://www.kernel.org/doc/Documentation/acpi/initrd_table_override.txt
Best regards,
Javier
Hi!
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
e. g. something like:
board_vendor = dmi_get_system_info(DMI_BOARD_VENDOR); board_name = dmi_get_system_info(DMI_BOARD_NAME); board_version = dmi_get_system_info(DMI_BOARD_NAME); product_name = dmi_get_system_info(DMI_PRODUCT_NAME); product_version = dmi_get_system_info(DMI_PRODUCT_VERSION); sprintf(dev->cap, "%s:%s:%s:%s", board_vendor, board_name,
board_version,product_name,product_version);(the real code should check if the values are filled, as not all BIOS vendors use the same DMI fields)
With that, the library can auto-adjust without needing to run anything as root.
Quoted 6 lines
Quoted 4 lines
Well actually most of those fields you're interested in are already exposed to userspace through sysfs
/sys/class/dmi/id/Can't the library just pull them from there?
Good point. Yeah, the library could use them.
This could be done, but it would be better if libraries could query neccessary information from v4l2 drivers.
If DMI was used, _library_ would need to know about new hardware, which is not desirable.
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Em Wed, 20 Jun 2018 16:57:59 -0400
Nicolas Dufresne <[email protected]> escreveu:
Quoted 4 lines
The IPU3 sensor produce a vendor specific form of bayer. If we manage to implement support for this format, it would likely be done in software. I don't think anyone can answer your other questions has no one have ever implemented this, hence measure performance.
Decoding that bayer-like specific format is something that we currently do at libv4lconvert. IMHO, we should have support for it inside the library, even if the device is capable of decoding it via a mem2mem device. If not for usage in "production", being able to read it from the sensor directly helps to test it. It might also be used on some applications where there would be no need for 3A algorithms to run (for example, on industrial cameras where the light and the distance of the objects don't change).
Cheers,
Mauro
Hi Pavel,
Quoted 10 lines · Pavel Machek
Hi!
On Nokia N900, I have similar problems as Intel IPU3 hardware.
Meeting notes say that pure software implementation is not fast enough, but that it may be useful for debugging. It would be also useful for me on N900, and probably useful for processing "raw" images from digital cameras.
Quoted 4 lines
There is sensor part, and memory-to-memory part, right? What is the format of data from the sensor part? What operations would be expensive on the CPU? If we did everthing on the CPU, what would be maximum resolution where we could still manage it in real time?
We can still use the memory-to-memory part (IMGU), even without 3A. It would just do demosaicing at default parameters and give us a YUV (NV12) frame. We could use some software component to analyze the YUV output and adjust sensor parameters accordingly. Possibly the part we already have in libv4l2 could just work?
The expensive operation would be analyzing the frame itself. I suppose you need to build some histogram representing brightness and white balance of the frame and then infer necessary sensor adjustments from that.
Quoted 3 lines
Would it be possible to get access to machine with IPU3, or would there be someone willing to test libv4l2 patches?
I should be able to help with some basic testing, preferably limited to command line tools (but I might be able to create a test environment for X11 tools if really necessary).
Best regards,
Tomasz
Hi!
Quoted 12 lines
Quoted 11 lines
On Nokia N900, I have similar problems as Intel IPU3 hardware.
Meeting notes say that pure software implementation is not fast enough, but that it may be useful for debugging. It would be also useful for me on N900, and probably useful for processing "raw" images from digital cameras.
There is sensor part, and memory-to-memory part, right? What is the format of data from the sensor part? What operations would be expensive on the CPU? If we did everthing on the CPU, what would be maximum resolution where we could still manage it in real time?
Quoted 5 lines
We can still use the memory-to-memory part (IMGU), even without 3A. It would just do demosaicing at default parameters and give us a YUV (NV12) frame. We could use some software component to analyze the YUV output and adjust sensor parameters accordingly. Possibly the part we already have in libv4l2 could just work?
As soon as you get YUV, yes, libv4l2 should be able to work with that.
OTOH using the memory-to-memory part is going to be tricky. What format is the data before demosaicing? Something common like BGGR10?
Quoted 4 lines
The expensive operation would be analyzing the frame itself. I suppose you need to build some histogram representing brightness and white balance of the frame and then infer necessary sensor adjustments from that.
That does not really have to be expensive. You can sample ... say 10000 pixels from the image, and get good-enough data for 3A.
Quoted 6 lines
Quoted 2 lines
Would it be possible to get access to machine with IPU3, or would there be someone willing to test libv4l2 patches?
I should be able to help with some basic testing, preferably limited to command line tools (but I might be able to create a test environment for X11 tools if really necessary).
Could you just compile libv4l2 with sdlcam demo on the target, and then ssh -X there from some sort of reasonable system?
Best regards,
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Hi Pavel,
Quoted 21 lines · Pavel Machek
Hi!
Quoted 12 lines
Quoted 11 lines
On Nokia N900, I have similar problems as Intel IPU3 hardware.
Meeting notes say that pure software implementation is not fast enough, but that it may be useful for debugging. It would be also useful for me on N900, and probably useful for processing "raw" images from digital cameras.
There is sensor part, and memory-to-memory part, right? What is the format of data from the sensor part? What operations would be expensive on the CPU? If we did everthing on the CPU, what would be maximum resolution where we could still manage it in real time?
Quoted 5 lines
We can still use the memory-to-memory part (IMGU), even without 3A. It would just do demosaicing at default parameters and give us a YUV (NV12) frame. We could use some software component to analyze the YUV output and adjust sensor parameters accordingly. Possibly the part we already have in libv4l2 could just work?
Quoted 3 lines
As soon as you get YUV, yes, libv4l2 should be able to work with that.
OTOH using the memory-to-memory part is going to be tricky.
Why? I don't see any reason why it would be tricky. You just feed the right CAPTURE node with YUV buffers and the right OUTPUT node with raw buffers and that should work.
Quoted 2 lines
What
format is the data before demosaicing? Something common like BGGR10?
It's documented in detail in V4L2 documentation:
https://www.kernel.org/doc/html/latest/media/uapi/v4l/pixfmt-srggb10-ipu3.html
Quoted 8 lines
Quoted 5 lines
The expensive operation would be analyzing the frame itself. I suppose you need to build some histogram representing brightness and white balance of the frame and then infer necessary sensor adjustments from that.
That does not really have to be expensive. You can sample ... say 10000 pixels from the image, and get good-enough data for 3A.
Yes, but you need to do it relatively frequently to react for scene changes and also even 10000 pixels (depending on distribution ) might mean fetching 10000*cacheline bytes of data.
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
Would it be possible to get access to machine with IPU3, or would there be someone willing to test libv4l2 patches?
I should be able to help with some basic testing, preferably limited to command line tools (but I might be able to create a test environment for X11 tools if really necessary).
Could you just compile libv4l2 with sdlcam demo on the target, and then ssh -X there from some sort of reasonable system?
Yes, that should work I guess. That would be a Chrome OS device (which doesn't include X11), so that would mean compiling some X11 libs (and probably some more dependencies) as well, but that's not impossible.
Best regards,
Tomasz
Hi!
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 11 lines
On Nokia N900, I have similar problems as Intel IPU3 hardware.
Meeting notes say that pure software implementation is not fast enough, but that it may be useful for debugging. It would be also useful for me on N900, and probably useful for processing "raw" images from digital cameras.
There is sensor part, and memory-to-memory part, right? What is the format of data from the sensor part? What operations would be expensive on the CPU? If we did everthing on the CPU, what would be maximum resolution where we could still manage it in real time?
We can still use the memory-to-memory part (IMGU), even without 3A. It would just do demosaicing at default parameters and give us a YUV (NV12) frame. We could use some software component to analyze the YUV output and adjust sensor parameters accordingly. Possibly the part we already have in libv4l2 could just work?
Quoted 7 lines
Quoted 3 lines
As soon as you get YUV, yes, libv4l2 should be able to work with that.
OTOH using the memory-to-memory part is going to be tricky.
Why? I don't see any reason why it would be tricky. You just feed the right CAPTURE node with YUV buffers and the right OUTPUT node with raw buffers and that should work.
I have seen insides of libv4l2. I believe unpacking by hand will be simpler / less tricky than passing buffers around.
Of course, long term both will be needed.
Quoted 5 lines
Quoted 2 lines
What
format is the data before demosaicing? Something common like BGGR10?It's documented in detail in V4L2 documentation:
Ah, thanks for pointer. That's not too bad.
Quoted 11 lines
Quoted 7 lines
Quoted 4 lines
The expensive operation would be analyzing the frame itself. I suppose you need to build some histogram representing brightness and white balance of the frame and then infer necessary sensor adjustments from that.
That does not really have to be expensive. You can sample ... say 10000 pixels from the image, and get good-enough data for 3A.
Yes, but you need to do it relatively frequently to react for scene changes and also even 10000 pixels (depending on distribution ) might mean fetching 10000*cacheline bytes of data.
Yeah, there are tradeoffs...
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
Would it be possible to get access to machine with IPU3, or would there be someone willing to test libv4l2 patches?
I should be able to help with some basic testing, preferably limited to command line tools (but I might be able to create a test environment for X11 tools if really necessary).
Quoted 6 lines
Quoted 2 lines
Could you just compile libv4l2 with sdlcam demo on the target, and then ssh -X there from some sort of reasonable system?
Yes, that should work I guess. That would be a Chrome OS device (which doesn't include X11), so that would mean compiling some X11 libs (and probably some more dependencies) as well, but that's not impossible.
I believe you want to put Debian chroot in there... or get machine with IPU3 where you can install Debian directly.
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html