media: atmel: atmel-isc: implement media controller
Subsystems (5)
v4 cover media: atmel: atmel-isc: implement media controller
This series is the v4 series that attempts to support media controller in the atmel ISC and XISC drivers.
The CSI2DC driver was accepted thus removed from the patch series, together with other patches.
Important note: this series applies on top of current media_staging tree, as it relies on previous patches in the series which were accepted.
Changes in v4:
-> as reviewed by Hans, added new patch to remove the 'stop' variable and reworked one patch that was using it
-> as reviewed by Jacopo, reworked some parts of the media controller implementation
Thanks to everyone who reviewed my work !
Eugen
Previous cover letter from v3:
This series is the v3 of the series that attempts to support media controller in the atmel ISC and XISC drivers.
This series also includes the csi2dc driver which was previously sent in a separate series:
https://www.spinics.net/lists/linux-media/msg181042.html
https://www.spinics.net/lists/linux-media/msg181044.html
The driver now addresses comments received in latest v5 series from last year.
The series includes some minor changes and fixes that improve the isc common code base, like removing the enum frameintervals VIDIOC, fixing bytesperline for planar formats, etc.
Many thanks to folks from libcamera who helped a lot with understanding how a media controller driver should behave.
Feedback is welcome !
Changes in v3:
- change in bindings, small fixes in csi2dc driver and conversion to mc for the isc-base.
- removed some MAINTAINERS patches and used patterns in MAINTAINERS
Changes in v2:
- integrated many changes suggested by Jacopo in the review of the v1 series.
- add a few new patches
Eugen Hristev (11):
media: atmel: atmel-isc: replace 'stop' variable with vb2 calls
media: atmel: atmel-isc-base: use streaming status when queueing
buffers
media: atmel: atmel-isc: implement media controller
media: atmel: atmel-sama5d2-isc: fix wrong mask in YUYV format check
media: atmel: atmel-isc-base: use mutex to lock awb workqueue from
streaming
media: atmel: atmel-isc: compact the controller formats list
media: atmel: atmel-isc: change format propagation to subdev into only
verification
dt-bindings: media: microchip,xisc: add bus-width of 14
ARM: dts: at91: sama7g5: add nodes for video capture
ARM: configs: at91: sama7: add xisc and csi2dc
ARM: multi_v7_defconfig: add atmel video pipeline modules
.../bindings/media/microchip,xisc.yaml | 2 +- arch/arm/boot/dts/sama7g5.dtsi | 49 ++ arch/arm/configs/multi_v7_defconfig | 3 + arch/arm/configs/sama7_defconfig | 2 + drivers/media/platform/atmel/Makefile | 2 +- drivers/media/platform/atmel/atmel-isc-base.c | 455 +++++++++++------- .../media/platform/atmel/atmel-isc-scaler.c | 250 ++++++++++ drivers/media/platform/atmel/atmel-isc.h | 43 +- .../media/platform/atmel/atmel-sama5d2-isc.c | 67 ++- .../media/platform/atmel/atmel-sama7g5-isc.c | 72 ++- 10 files changed, 694 insertions(+), 251 deletions(-) create mode 100644 drivers/media/platform/atmel/atmel-isc-scaler.c
Signature · 1 line
2.25.1
PATCH v4 1/11 media: atmel: atmel-isc: replace 'stop' variable with vb2 calls
The stop variable is redundant as the state of the streaming can be obtained by calling vb2_start_streaming_called(&isc->vb2_vidq) or by calling vb2_is_busy(&isc->vb2_vidq).
Thus, remove the stop variable completely.
---
Changes in v4:
- new patch
drivers/media/platform/atmel/atmel-isc-base.c | 12 +++++------- drivers/media/platform/atmel/atmel-isc.h | 2 -- 2 files changed, 5 insertions(+), 9 deletions(-)
drivers/media/platform/atmel/atmel-isc-base.c +5 -7
@@ -362,7 +362,6 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)spin_lock_irqsave (&isc->dma_queue_lock, flags); isc->sequence =0 ; - isc->stop = false;reinit_completion (&isc->comp); isc->cur_frm = list_first_entry(&isc->dma_queue,
@@ -403,8 +402,6 @@ static void isc_stop_streaming(struct vb2_queue *vq)v4l2_ctrl_activate (isc->do_wb_ctrl, false); - isc->stop = true; -/* Wait until the end of the current frame */ if (isc->cur_frm && !wait_for_completion_timeout (&isc->comp,5 *HZ )) v4l2_err(&isc->v4l2_dev,
@@ -1029,7 +1026,7 @@ static int isc_s_fmt_vid_cap(struct file *file, void *priv, {struct isc_device *isc =video_drvdata (file); -if (vb2_is_streaming (&isc->vb2_vidq)) +if (vb2_is_busy (&isc->vb2_vidq))return -EBUSY ;return isc_set_fmt (isc, f);
@@ -1236,7 +1233,8 @@ irqreturn_t isc_interrupt(int irq, void *dev_id) isc->cur_frm =NULL ; } -if (!list_empty (&isc->dma_queue) && !isc->stop) { +if (!list_empty (&isc->dma_queue) && +vb2_start_streaming_called (&isc->vb2_vidq)) { isc->cur_frm =list_first_entry (&isc->dma_queue, struct isc_buffer, list);list_del (&isc->cur_frm->list);
@@ -1244,7 +1242,7 @@ irqreturn_t isc_interrupt(int irq, void *dev_id)isc_start_dma (isc); } -if (isc->stop) +if (!vb2_start_streaming_called (&isc->vb2_vidq))complete (&isc->comp); ret =IRQ_HANDLED ;
@@ -1398,7 +1396,7 @@ static void isc_awb_work(struct work_struct *w)int ret;/* streaming is not active anymore */ -if (isc->stop) +if (!vb2_start_streaming_called (&isc->vb2_vidq))return ;if (ctrls->hist_stat !=HIST_ENABLED )
drivers/media/platform/atmel/atmel-isc.h +0 -2
@@ -201,7 +201,6 @@ struct isc_reg_offsets {* @dma_queue: the queue for dma buffers * @cur_frm: current isc frame/buffer * @sequence: current frame number -* @stop: true if isc is not streaming, false if streaming * @comp: completion reference that signals frame completion * * @fmt: current v42l format
@@ -276,7 +275,6 @@ struct isc_device {struct list_head dma_queue;struct isc_buffer *cur_frm;unsigned int sequence; -bool stop;struct completion comp;struct v4l2_format fmt;
Signature · 1 line
2.25.1
Re: [PATCH v4 01/11] media: atmel: atmel-isc: replace 'stop' variable with vb2 calls
Hi Eugen,
Quoted 7 lines · Eugen Hristev
The stop variable is redundant as the state of the streaming can be obtained by calling
vb2_start_streaming_called(&isc->vb2_vidq)or by callingvb2_is_busy(&isc->vb2_vidq).
Thus, remove the stop variable completely.Suggested-by Hans Verkuil <[email protected]>Signed-off-by Eugen Hristev <[email protected]>
I trust yours and Hans' judgment here.
The patch looks sane
Thanks
Quoted 87 lines
---
Changes in v4:
- new patchdrivers/media/platform/atmel/atmel-isc-base.c | 12 +++++------- drivers/media/platform/atmel/atmel-isc.h | 2 -- 2 files changed, 5 insertions(+), 9 deletions(-)drivers/media/platform/atmel/atmel-isc-base.c +5
-7@@ -362,7 +362,6 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)spin_lock_irqsave (&isc->dma_queue_lock, flags); isc->sequence =0 ; - isc->stop = false;reinit_completion (&isc->comp); isc->cur_frm = list_first_entry(&isc->dma_queue,@@ -403,8 +402,6 @@ static void isc_stop_streaming(struct vb2_queue *vq)v4l2_ctrl_activate (isc->do_wb_ctrl, false); - isc->stop = true; -/* Wait until the end of the current frame */ if (isc->cur_frm && !wait_for_completion_timeout (&isc->comp,5 *HZ )) v4l2_err(&isc->v4l2_dev,@@ -1029,7 +1026,7 @@ static int isc_s_fmt_vid_cap(struct file *file, void *priv, {struct isc_device *isc =video_drvdata (file); -if (vb2_is_streaming (&isc->vb2_vidq)) +if (vb2_is_busy (&isc->vb2_vidq))return -EBUSY ;return isc_set_fmt (isc, f);@@ -1236,7 +1233,8 @@ irqreturn_t isc_interrupt(int irq, void *dev_id) isc->cur_frm =NULL ; } -if (!list_empty (&isc->dma_queue) && !isc->stop) { +if (!list_empty (&isc->dma_queue) && +vb2_start_streaming_called (&isc->vb2_vidq)) { isc->cur_frm =list_first_entry (&isc->dma_queue, struct isc_buffer, list);list_del (&isc->cur_frm->list);@@ -1244,7 +1242,7 @@ irqreturn_t isc_interrupt(int irq, void *dev_id)isc_start_dma (isc); } -if (isc->stop) +if (!vb2_start_streaming_called (&isc->vb2_vidq))complete (&isc->comp); ret =IRQ_HANDLED ;@@ -1398,7 +1396,7 @@ static void isc_awb_work(struct work_struct *w)int ret;/* streaming is not active anymore */ -if (isc->stop) +if (!vb2_start_streaming_called (&isc->vb2_vidq))return ;if (ctrls->hist_stat !=HIST_ENABLED )drivers/media/platform/atmel/atmel-isc.h +0
-3@@ -201,7 +201,6 @@ struct isc_reg_offsets {* @dma_queue: the queue for dma buffers * @cur_frm: current isc frame/buffer * @sequence: current frame number -* @stop: true if isc is not streaming, false if streaming * @comp: completion reference that signals frame completion * * @fmt: current v42l format @@ -276,7 +275,6 @@ struct isc_device {struct list_head dma_queue;struct isc_buffer *cur_frm;unsigned int sequence; -bool stop;struct completion comp;struct v4l2_format fmt; --
Quoted 2 lines
2.25.1
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v4 01/11] media: atmel: atmel-isc: replace 'stop' variable with vb2 calls
Quoted 68 lines · Jacopo Mondi
Hi Eugen,
Quoted 7 lines · Eugen Hristev
The stop variable is redundant as the state of the streaming can be obtained by calling
vb2_start_streaming_called(&isc->vb2_vidq)or by callingvb2_is_busy(&isc->vb2_vidq).
Thus, remove the stop variable completely.Suggested-by Hans Verkuil <[email protected]>Signed-off-by Eugen Hristev <[email protected]>I trust yours and Hans' judgment here.
The patch looks saneReviewed-by Jacopo Mondi <[email protected]>Thanks
Quoted 52 lines
---
Changes in v4:
- new patchdrivers/media/platform/atmel/atmel-isc-base.c | 12 +++++------- drivers/media/platform/atmel/atmel-isc.h | 2 -- 2 files changed, 5 insertions(+), 9 deletions(-)drivers/media/platform/atmel/atmel-isc-base.c +3
-5@@ -362,7 +362,6 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)spin_lock_irqsave (&isc->dma_queue_lock, flags); isc->sequence =0 ; - isc->stop = false;reinit_completion (&isc->comp); isc->cur_frm = list_first_entry(&isc->dma_queue,@@ -403,8 +402,6 @@ static void isc_stop_streaming(struct vb2_queue *vq)v4l2_ctrl_activate (isc->do_wb_ctrl, false); - isc->stop = true; -/* Wait until the end of the current frame */ if (isc->cur_frm && !wait_for_completion_timeout (&isc->comp,5 *HZ )) v4l2_err(&isc->v4l2_dev,@@ -1029,7 +1026,7 @@ static int isc_s_fmt_vid_cap(struct file *file, void *priv, {struct isc_device *isc =video_drvdata (file); -if (vb2_is_streaming (&isc->vb2_vidq)) +if (vb2_is_busy (&isc->vb2_vidq))return -EBUSY ;return isc_set_fmt (isc, f);@@ -1236,7 +1233,8 @@ irqreturn_t isc_interrupt(int irq, void *dev_id) isc->cur_frm =NULL ; } -if (!list_empty (&isc->dma_queue) && !isc->stop) { +if (!list_empty (&isc->dma_queue) && +vb2_start_streaming_called (&isc->vb2_vidq)) { isc->cur_frm =list_first_entry (&isc->dma_queue, struct isc_buffer, list);list_del (&isc->cur_frm->list);@@ -1244,7 +1242,7 @@ irqreturn_t isc_interrupt(int irq, void *dev_id)isc_start_dma (isc); }
Quoted 2 lines
drivers/media/platform/atmel/atmel-isc-base.c irqreturn_t isc_interrupt(int irq, void *dev_id) · line 1245
Quoted 2 lines
drivers/media/platform/atmel/atmel-isc-base.c +1
-1-if (isc->stop) +if (!vb2_start_streaming_called (&isc->vb2_vidq))
Hello Hans, Jacopo,
This patch has a problem here. In the IRQ context, I cannot tell whether the stop streaming has been called or not.
When stop streaming is called, vb2_start_streaming_called still returns true.
So in the IRQ context I cannot know if I have to program the next buffer and start the next capture (next DMA), or I have to stop, and call complete on the isc completion , such that stop streaming will release all the other buffers.
To know this, it was previously done with the 'stop' variable which was set when stop_streaming was called.
Any idea how to handle this with vb2_ calls ?
Or I would have to rework this patch and keep the 'stop' internal variable ?
Thanks,
Eugen
Quoted 31 lines
Quoted 31 lines
drivers/media/platform/atmel/atmel-isc-base.c +1
-1complete (&isc->comp); ret =IRQ_HANDLED ;@@ -1398,7 +1396,7 @@ static void isc_awb_work(struct work_struct *w)int ret;/* streaming is not active anymore */ -if (isc->stop) +if (!vb2_start_streaming_called (&isc->vb2_vidq))return ;if (ctrls->hist_stat !=HIST_ENABLED )drivers/media/platform/atmel/atmel-isc.h +0
-2@@ -201,7 +201,6 @@ struct isc_reg_offsets {* @dma_queue: the queue for dma buffers * @cur_frm: current isc frame/buffer * @sequence: current frame number -* @stop: true if isc is not streaming, false if streaming * @comp: completion reference that signals frame completion * * @fmt: current v42l format @@ -276,7 +275,6 @@ struct isc_device {struct list_head dma_queue;struct isc_buffer *cur_frm;unsigned int sequence; -bool stop;struct completion comp;
Quoted 4 lines
Quoted 4 lines
drivers/media/platform/atmel/atmel-isc.h +0
-1struct v4l2_format fmt; --2.25.1
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v4 2/11 media: atmel: atmel-isc-base: use streaming status when queueing buffers
In reply to Eugen HristevDuring experiments with libcamera, it looks like vb2_is_streaming returns true before our start streaming is called.
Order of operations is streamon -> queue -> start_streaming
ISC would have started the DMA immediately when a buffer is being added to the vbqueue if the queue is streaming.
It is more safe to start the DMA after the start streaming of the driver is called.
Thus, even if vb2queue is streaming, add the buffer to the dma queue of the driver instead of actually starting the DMA process, if the start streaming has not been called yet.
Tho achieve this, we have to use vb2_start_streaming_called instead of vb2_is_streaming.
---
Changes in v4:
- changed to using vb2_start_streaming_called instead of stop variable
drivers/media/platform/atmel/atmel-isc-base.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/media/platform/atmel/atmel-isc-base.c +3 -3
@@ -439,7 +439,7 @@ static void isc_buffer_queue(struct vb2_buffer *vb)spin_lock_irqsave (&isc->dma_queue_lock, flags);if (!isc->cur_frm &&list_empty (&isc->dma_queue) && -vb2_is_streaming (vb->vb2_queue)) { +vb2_start_streaming_called (vb->vb2_queue)) { isc->cur_frm = buf;isc_start_dma (isc); }else
@@ -1532,7 +1532,7 @@ static int isc_s_awb_ctrl(struct v4l2_ctrl *ctrl)isc_update_awb_ctrls (isc); -if (vb2_is_streaming (&isc->vb2_vidq)) { +if (vb2_start_streaming_called (&isc->vb2_vidq)) {/* * If we are streaming, we can update profile to * have the new settings in place.
@@ -1549,7 +1549,7 @@ static int isc_s_awb_ctrl(struct v4l2_ctrl *ctrl)/* if we have autowhitebalance on, start histogram procedure */ if (ctrls->awb ==ISC_WB_AUTO && -vb2_is_streaming (&isc->vb2_vidq) && +vb2_start_streaming_called (&isc->vb2_vidq) &&ISC_IS_FORMAT_RAW (isc->config.sd_format->mbus_code))isc_set_histogram (isc, true);
Signature · 1 line
2.25.1
PATCH v4 3/11 media: atmel: atmel-isc: implement media controller
In reply to Eugen HristevImplement the support for media-controller.
This means that the capabilities of the driver have changed and now it also advertises the IO_MC .
The driver will register it's media device, and add the video entity to this media device. The subdevices are registered to the same media device. The ISC will have a base entity which is auto-detected as atmel_isc_base. It will also register a subdevice that allows cropping of the incoming frame to the maximum frame size supported by the ISC.
The ISC will create a link between the subdevice that is asynchronously registered and the atmel_isc_scaler entity.
Then, the atmel_isc_scaler and atmel_isc_base are connected through another link.
---
Changes in v4:
As suggested by Jacopo:
- renamed atmel_isc_mc to atmel_isc_scaler.c
- moved init_mc/clean_mc to isc_base file
Changes in v2:
- implement try formats
drivers/media/platform/atmel/Makefile | 2 +- drivers/media/platform/atmel/atmel-isc-base.c | 73 +++++- .../media/platform/atmel/atmel-isc-scaler.c | 245 ++++++++++++++++++ drivers/media/platform/atmel/atmel-isc.h | 37 +++ .../media/platform/atmel/atmel-sama5d2-isc.c | 14 +- .../media/platform/atmel/atmel-sama7g5-isc.c | 12 +- 6 files changed, 375 insertions(+), 8 deletions(-) create mode 100644 drivers/media/platform/atmel/atmel-isc-scaler.c
drivers/media/platform/atmel/Makefile +1 -1
@@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only atmel-isc-objs = atmel-sama5d2-isc.o atmel-xisc-objs = atmel-sama7g5-isc.o -atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o +atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o atmel-isc-scaler.o obj-$(CONFIG_VIDEO_ATMEL_ISI) += atmel-isi.o obj-$(CONFIG_VIDEO_ATMEL_ISC_BASE) += atmel-isc-common.o
drivers/media/platform/atmel/atmel-isc-base.c +69 -4
123 lines not on this page · raw message
drivers/media/platform/atmel/atmel-isc-scaler.c new file +245 -0
245 lines not on this page · raw message
drivers/media/platform/atmel/atmel-isc.h +37 -0
@@ -183,6 +183,17 @@ struct isc_reg_offsets {u32 his_entry; }; +enum isc_mc_pads { +ISC_PAD_SINK =0 , +ISC_PADS_NUM =1 , +}; + +enum isc_scaler_pads { +ISC_SCALER_PAD_SINK =0 , +ISC_SCALER_PAD_SOURCE =1 , +ISC_SCALER_PADS_NUM =2 , +}; +/* * struct isc_device - ISC device driver data/config struct * @regmap: Register map
@@ -257,6 +268,12 @@ struct isc_reg_offsets {* be used as an input to the controller * @controller_formats_size: size of controller_formats array * @formats_list_size: size of formats_list array +* @pads: media controller pads for isc video entity +* @mdev: media device that is registered by the isc +* @remote_pad: remote pad on the connected subdevice +* @scaler_sd: subdevice for the scaler that isc registers +* @scaler_pads: media controller pads for the scaler subdevice +* @scaler_format: current format for the scaler subdevice */ struct isc_device {struct regmap *regmap;
@@ -344,6 +361,19 @@ struct isc_device {struct isc_format *formats_list;u32 controller_formats_size;u32 formats_list_size; + +struct { +struct media_pad pads[ISC_PADS_NUM ]; +struct media_device mdev; + +u32 remote_pad; + }; + +struct { +struct v4l2_subdev scaler_sd; +struct media_pad scaler_pads[ISC_SCALER_PADS_NUM ]; +struct v4l2_mbus_framefmt scaler_format; + }; };extern const struct regmap_config isc_regmap_config;
@@ -355,4 +385,11 @@ int isc_clk_init(struct isc_device *isc);void isc_subdev_cleanup (struct isc_device *isc);void isc_clk_cleanup (struct isc_device *isc); +int isc_scaler_link (struct isc_device *isc); +int isc_scaler_init (struct isc_device *isc); +int isc_mc_init (struct isc_device *isc,u32 ver); +void isc_mc_cleanup (struct isc_device *isc); + +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index);#endif
drivers/media/platform/atmel/atmel-sama5d2-isc.c +12 -2
@@ -553,6 +553,12 @@ static int atmel_isc_probe(struct platform_device *pdev)break ; } +regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver); + + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev);
@@ -562,7 +568,7 @@ static int atmel_isc_probe(struct platform_device *pdev) ret =clk_prepare_enable (isc->ispck);if (ret) {dev_err (dev,"failed to enable ispck: %d\n" , ret); - goto cleanup_subdev; + goto isc_probe_mc_init_err; }/* ispck should be greater or equal to hclock */
@@ -572,7 +578,6 @@ static int atmel_isc_probe(struct platform_device *pdev) goto unprepare_clk; } -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip ISC version %x\n" , ver);return 0 ;
@@ -580,6 +585,9 @@ static int atmel_isc_probe(struct platform_device *pdev) unprepare_clk:clk_disable_unprepare (isc->ispck); +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);
@@ -600,6 +608,8 @@ static int atmel_isc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev);
drivers/media/platform/atmel/atmel-sama7g5-isc.c +11 -1
@@ -547,15 +547,23 @@ static int microchip_xisc_probe(struct platform_device *pdev)break ; } +regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver); + + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev); -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip XISC version %x\n" , ver);return 0 ; +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);
@@ -576,6 +584,8 @@ static int microchip_xisc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev);
Signature · 1 line
2.25.1
Hi Eugen
Quoted 132 lines · Eugen Hristev
Implement the support for media-controller.
This means that the capabilities of the driver have changed and now it also advertises theIO_MC.
The driver will register it's media device, and add the video entity to this media device. The subdevices are registered to the same media device. The ISC will have a base entity which is auto-detected asatmel_isc_base. It will also register a subdevice that allows cropping of the incoming frame to the maximum frame size supported by the ISC.
The ISC will create a link between the subdevice that is asynchronously registered and theatmel_isc_scalerentity.
Then, theatmel_isc_scalerandatmel_isc_baseare connected through another link.Signed-off-by Eugen Hristev <[email protected]>---
Changes in v4:
As suggested by Jacopo:
- renamedatmel_isc_mctoatmel_isc_scaler.c
- moved init_mc/clean_mc toisc_basefileChanges in v2:
- implement try formatsdrivers/media/platform/atmel/Makefile | 2 +- drivers/media/platform/atmel/atmel-isc-base.c | 73 +++++- .../media/platform/atmel/atmel-isc-scaler.c | 245 ++++++++++++++++++ drivers/media/platform/atmel/atmel-isc.h | 37 +++ .../media/platform/atmel/atmel-sama5d2-isc.c | 14 +- .../media/platform/atmel/atmel-sama7g5-isc.c | 12 +- 6 files changed, 375 insertions(+), 8 deletions(-) create mode 100644 drivers/media/platform/atmel/atmel-isc-scaler.cdrivers/media/platform/atmel/Makefile +1
-1@@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only atmel-isc-objs = atmel-sama5d2-isc.o atmel-xisc-objs = atmel-sama7g5-isc.o -atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o +atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o atmel-isc-scaler.o obj-$(CONFIG_VIDEO_ATMEL_ISI) += atmel-isi.o obj-$(CONFIG_VIDEO_ATMEL_ISC_BASE) += atmel-isc-common.odrivers/media/platform/atmel/atmel-isc-base.c +27
-4@@ -1710,6 +1710,7 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, struct isc_device, v4l2_dev);struct isc_subdev_entity *subdev_entity = container_of(notifier, struct isc_subdev_entity, notifier); +int pad;if (video_is_registered (&isc->video_dev)) {v4l2_err (&isc->v4l2_dev,"only supports one sub-device.\n" );@@ -1718,6 +1719,16 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, subdev_entity->sd = subdev; + pad =media_entity_get_fwnode_pad (&subdev->entity, asd->match.fwnode, +MEDIA_PAD_FL_SOURCE ); +if (pad <0 ) { +v4l2_err (&isc->v4l2_dev,"failed to find pad for %s\n" , + subdev->name); +return pad; + } + + isc->remote_pad = pad; +return 0 ; }@@ -1732,8 +1743,8 @@ static void isc_async_unbind(struct v4l2_async_notifier *notifier,v4l2_ctrl_handler_free (&isc->ctrls.handler); } -static struct isc_format *find_format_by_code (struct isc_device *isc, -unsigned int code,int *index) +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index) {struct isc_format *fmt = &isc->formats_list[0 ];unsigned int i;@@ -1749,6 +1760,7 @@ static struct isc_format *find_format_by_code(struct isc_device *isc,return NULL ; } +EXPORT_SYMBOL_GPL (isc_find_format_by_code);static int isc_formats_init (struct isc_device *isc) {@@ -1765,7 +1777,7 @@ static int isc_formats_init(struct isc_device *isc) NULL, &mbus_code)) { mbus_code.index++; - fmt =find_format_by_code (isc, mbus_code.code, &i); + fmt =isc_find_format_by_code (isc, mbus_code.code, &i);if (!fmt) {v4l2_warn (&isc->v4l2_dev,"Mbus code %x not supported\n" , mbus_code.code);@@ -1891,7 +1903,8 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) vdev->queue = q; vdev->lock = &isc->lock; vdev->ctrl_handler = &isc->ctrls.handler; - vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE ; + vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE | +V4L2_CAP_IO_MC ;video_set_drvdata (vdev, isc); ret =video_register_device (vdev,VFL_TYPE_VIDEO ,-1 );@@ -1901,8 +1914,18 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) goto isc_async_complete_err; } + ret =isc_scaler_link (isc); +if (ret <0 ) + goto isc_async_complete_unregister_device; + + ret =media_device_register (&isc->mdev); +if (ret <0 ) + goto isc_async_complete_unregister_device;return 0 ; +isc_async_complete_unregister_device: +video_unregister_device (vdev); + isc_async_complete_err:mutex_destroy (&isc->lock);return ret;
drivers/media/platform/atmel/atmel-isc-base.c int isc_pipeline_init(struct isc_device *isc) · line 2001
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-base.c +7
-0@@ -1969,6 +1992,48 @@ int isc_pipeline_init(struct isc_device *isc) }EXPORT_SYMBOL_GPL (isc_pipeline_init); +int isc_mc_init (struct isc_device *isc,u32 ver) +{ +const struct of_device_id *match; +int ret; + + isc->video_dev.entity.function =MEDIA_ENT_F_IO_V4L ; + isc->video_dev.entity.flags =MEDIA_ENT_FL_DEFAULT ;
Should you set entity.ops.link_validate = v4l2_subdev_link_validate to be able to have the media pipeline validated at media_pipeline_start() time ?
Quoted 44 lines
drivers/media/platform/atmel/atmel-isc-base.c +35
-0+ isc->pads[ISC_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + + ret =media_entity_pads_init (&isc->video_dev.entity,ISC_PADS_NUM , + isc->pads); +if (ret <0 ) { +dev_err (isc->dev,"media entity init failed\n" ); +return ret; + } + + isc->mdev.dev = isc->dev; + + match =of_match_node (isc->dev->driver->of_match_table, + isc->dev->of_node); + +strscpy (isc->mdev.driver_name,KBUILD_MODNAME , +sizeof (isc->mdev.driver_name)); +strscpy (isc->mdev.model, match->compatible,sizeof (isc->mdev.model)); +snprintf (isc->mdev.bus_info,sizeof (isc->mdev.bus_info),"platform:%s" , + isc->v4l2_dev.name); + isc->mdev.hw_revision = ver; + +media_device_init (&isc->mdev); + + isc->v4l2_dev.mdev = &isc->mdev; + +return isc_scaler_init (isc); +} +EXPORT_SYMBOL_GPL (isc_mc_init); + +void isc_mc_cleanup (struct isc_device *isc) +{ +media_entity_cleanup (&isc->video_dev.entity); +} +EXPORT_SYMBOL_GPL (isc_mc_cleanup); +/* regmap configuration */ #define ATMEL_ISC_REG_MAX 0xd5cconst struct regmap_config isc_regmap_config = {drivers/media/platform/atmel/atmel-isc-scaler.c new file +0
-0
drivers/media/platform/atmel/atmel-isc-scaler.c line 5
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0@@ -0,0 +1,245 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Microchip Image Sensor Controller (ISC) Scaler entity support +* +* Copyright (C) 2021 Microchip Technology, Inc.
Time flies! It's 2022 already :)
Quoted 19 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +19
-0+* +* Author: Eugen Hristev <[email protected]> +* +*/ + +#include <media/media-device.h> +#include <media/media-entity.h> +#include <media/v4l2-device.h> +#include <media/v4l2-subdev.h> + +#include "atmel-isc-regs.h" +#include "atmel-isc.h" + +static int isc_scaler_get_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *format) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt;
drivers/media/platform/atmel/atmel-isc-scaler.c line 34
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +if (format->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + format->pad); + format->format = *v4l2_try_fmt; + +return 0 ; + } + + format->format = isc->scaler_format;
isc->scaler_format is only used inside this file if I'm not mistaken. I wonder why it lives in the isc_device struct.
Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +11
-0+ +return 0 ; +} + +static int isc_scaler_set_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *req_fmt) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; +struct isc_format *fmt;
drivers/media/platform/atmel/atmel-isc-scaler.c line 55
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+unsigned int i; + +if (req_fmt->pad ==ISC_SCALER_PAD_SOURCE ) +v4l_bound_align_image + (&req_fmt->format.width,16 , isc->max_width,0 , + &req_fmt->format.height,16 , isc->max_height,0 ,0 ); +else +v4l_bound_align_image + (&req_fmt->format.width,16 ,10000 ,0 , + &req_fmt->format.height,16 ,10000 ,0 ,0 );
Where does 10000 come from ?
drivers/media/platform/atmel/atmel-isc-scaler.c line 63
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +8
-0+ + req_fmt->format.colorspace =V4L2_COLORSPACE_SRGB ; + req_fmt->format.field =V4L2_FIELD_NONE ; + req_fmt->format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + req_fmt->format.quantization =V4L2_QUANTIZATION_DEFAULT ; + req_fmt->format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + fmt =isc_find_format_by_code (isc, req_fmt->format.code, &i);
So you rely on the isc format list for the scaler as well ? I think it's fine as long as they are identical
Quoted 17 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +17
-0+ +if (!fmt) + fmt = &isc->formats_list[0 ]; + + req_fmt->format.code = fmt->mbus_code; + +if (req_fmt->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + req_fmt->pad); + *v4l2_try_fmt = req_fmt->format; +/* Trying on the pad sink makes the source sink change too */ +if (req_fmt->pad ==ISC_SCALER_PAD_SINK ) { + v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state, +ISC_SCALER_PAD_SOURCE ); + *v4l2_try_fmt = req_fmt->format; +
drivers/media/platform/atmel/atmel-isc-scaler.c line 90
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+v4l_bound_align_image (&v4l2_try_fmt->width, +16 , isc->max_width,0 , + &v4l2_try_fmt->height, +16 , isc->max_height,0 ,0 ); + } +/* if we are just trying, we are done */ +return 0 ; + } + + isc->scaler_format = req_fmt->format;
No per-pad format but a global scaler_format ? How do you configure scaling ?
Actually, I would like to know more about the scaler device capabilities. What functions can this IP perform ? Does it do cropping, can it also do (down)scaling or even composition ?
I think it is worth to read
https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-subdev.html#v4l2-subdev-selections
where it is reported how cropping and scaling are implemented by using the selection API.
Figure 4.5 is particularly helpful to explain the simple crop case, for which you need to implement support to by adding s_selection on the sink pad TGT_CROP target.
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; +} + +static int isc_scaler_enum_mbus_code(struct v4l2_subdev *sd,
drivers/media/platform/atmel/atmel-isc-scaler.c line 105
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+struct v4l2_subdev_state *sd_state, + struct v4l2_subdev_mbus_code_enum *code) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +int supported_index =0 ; +int i; + +for (i =0 ; i < isc->formats_list_size; i++) { +if (!isc->formats_list[i].sd_support) +continue ;
The sd_support flag still doesn't click in my head.
Shouldn't the enumeration of available formats on the scaler do not depend on the sensor supproted formats ?
Quoted 15 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +15
-0+if (supported_index == code->index) { + code->code = isc->formats_list[i].mbus_code; +return 0 ; + } + supported_index++; + } + +return -EINVAL ; +} + +static int isc_scaler_g_sel (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_selection *sel) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd);
drivers/media/platform/atmel/atmel-isc-scaler.c line 130
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +if (sel->pad ==ISC_SCALER_PAD_SOURCE ) +return -EINVAL ; + +if (sel->target !=V4L2_SEL_TGT_CROP_BOUNDS && + sel->target !=V4L2_SEL_TGT_CROP ) +return -EINVAL ; + + sel->r.height = isc->max_height; + sel->r.width = isc->max_width;
The CROP_BOUNDS should be set to the same size as the sink pad image format, as it represents the maximum valid crop rectangle.
TGT_CROP should report the configured crop rectangle which can be intiialized to the same size as CROP_BOUNDS, if my understanding of the spec is correct
Quoted 15 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +15
-0+ + sel->r.left =0 ; + sel->r.top =0 ; + +return 0 ; +} + +static int isc_scaler_init_cfg (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state) +{ +struct v4l2_mbus_framefmt *v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state,0 ); + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); + + *v4l2_try_fmt = isc->scaler_format;
drivers/media/platform/atmel/atmel-isc-scaler.c line 155
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +return 0 ; +} + +static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = { + .enum_mbus_code = isc_scaler_enum_mbus_code, + .set_fmt = isc_scaler_set_fmt, + .get_fmt = isc_scaler_get_fmt, + .get_selection = isc_scaler_g_sel, + .init_cfg = isc_scaler_init_cfg,
.link_validate = v4l2_subdev_link_validate_default,
To have the formats at the end of links that point to this entity validated (I think the framework already calls it if not set though, please check v4l2-subdev.c:v4l2_subdev_link_validate())
Quoted 6 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +6
-0+}; + +static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = { + .pad = &isc_scaler_pad_ops, +}; +
drivers/media/platform/atmel/atmel-isc-scaler.c line 171
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+int isc_scaler_init (struct isc_device *isc) +{ +int ret; + +v4l2_subdev_init (&isc->scaler_sd, &xisc_scaler_subdev_ops); + + isc->scaler_sd.owner =THIS_MODULE ; + isc->scaler_sd.dev = isc->dev; +snprintf (isc->scaler_sd.name,sizeof (isc->scaler_sd.name), +"atmel_isc_scaler" );
I would drop 'atmel' for brevity, unless other entities have this prefix set already
Quoted 63 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +63
-0+ + isc->scaler_sd.flags |=V4L2_SUBDEV_FL_HAS_DEVNODE ; + isc->scaler_sd.entity.function =MEDIA_ENT_F_PROC_VIDEO_SCALER ; + isc->scaler_pads[ISC_SCALER_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + isc->scaler_pads[ISC_SCALER_PAD_SOURCE ].flags =MEDIA_PAD_FL_SOURCE ; + + isc->scaler_format.height = isc->max_height; + isc->scaler_format.width = isc->max_width; + isc->scaler_format.code = isc->formats_list[0 ].mbus_code; + isc->scaler_format.colorspace =V4L2_COLORSPACE_SRGB ; + isc->scaler_format.field =V4L2_FIELD_NONE ; + isc->scaler_format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + isc->scaler_format.quantization =V4L2_QUANTIZATION_DEFAULT ; + isc->scaler_format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + ret =media_entity_pads_init (&isc->scaler_sd.entity, +ISC_SCALER_PADS_NUM , + isc->scaler_pads); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd media entity init failed\n" ); +return ret; + } + ret =v4l2_device_register_subdev (&isc->v4l2_dev, &isc->scaler_sd); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd failed to register subdev\n" ); +return ret; + } + +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_init); + +int isc_scaler_link (struct isc_device *isc) +{ +int ret; + + ret =media_create_pad_link (&isc->current_subdev->sd->entity, + isc->remote_pad, &isc->scaler_sd.entity, +ISC_SCALER_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->current_subdev->sd->entity.name, + isc->scaler_sd.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , + isc->current_subdev->sd->name, isc->remote_pad); + + ret =media_create_pad_link (&isc->scaler_sd.entity, +ISC_SCALER_PAD_SOURCE , + &isc->video_dev.entity,ISC_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { + v4l2_err(&isc->v4l2_dev, + "Failed to create pad link: %s to %s\n", + isc->scaler_sd.entity.name,
drivers/media/platform/atmel/atmel-isc-scaler.c line 244
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ isc->video_dev.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , isc->scaler_sd.name, +ISC_SCALER_PAD_SOURCE ); + +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_link);
From the DT graph point of view, the ISC appears as a single block with an CSI-2 input port. It is then in charge of parsing the .dts and add to its notifier the image sensor remote subdevice.
When the sensor subdev registers and the ISC notifier completes, the scaler entity which was initialized at isc_probe() time is linked in between the ISC and the image sensor immutably.
I think it is fine for now, but I wonder if you plan to plumb more components between the ISC video node and the sensor, if it's not worth changing the DT bindings and their parsing logic to separate the CSI-2 receiver from the ISC, whcih can create its media graph at probe time and have the CSI-2 receiver entity as downstream remote. I think I need to get to know the ISC better to really have an idea. For now, this seems ok to me, but please check with maintainers if this is fine with them.
Quoted 73 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +1
-0+drivers/media/platform/atmel/atmel-isc.h +37
-0@@ -183,6 +183,17 @@ struct isc_reg_offsets {u32 his_entry; }; +enum isc_mc_pads { +ISC_PAD_SINK =0 , +ISC_PADS_NUM =1 , +}; + +enum isc_scaler_pads { +ISC_SCALER_PAD_SINK =0 , +ISC_SCALER_PAD_SOURCE =1 , +ISC_SCALER_PADS_NUM =2 , +}; +/* * struct isc_device - ISC device driver data/config struct * @regmap: Register map @@ -257,6 +268,12 @@ struct isc_reg_offsets {* be used as an input to the controller * @controller_formats_size: size of controller_formats array * @formats_list_size: size of formats_list array +* @pads: media controller pads for isc video entity +* @mdev: media device that is registered by the isc +* @remote_pad: remote pad on the connected subdevice +* @scaler_sd: subdevice for the scaler that isc registers +* @scaler_pads: media controller pads for the scaler subdevice +* @scaler_format: current format for the scaler subdevice */ struct isc_device {struct regmap *regmap;@@ -344,6 +361,19 @@ struct isc_device {struct isc_format *formats_list;u32 controller_formats_size;u32 formats_list_size; + +struct { +struct media_pad pads[ISC_PADS_NUM ]; +struct media_device mdev; + +u32 remote_pad; + }; + +struct { +struct v4l2_subdev scaler_sd; +struct media_pad scaler_pads[ISC_SCALER_PADS_NUM ]; +struct v4l2_mbus_framefmt scaler_format; + }; };extern const struct regmap_config isc_regmap_config;@@ -355,4 +385,11 @@ int isc_clk_init(struct isc_device *isc);void isc_subdev_cleanup (struct isc_device *isc);void isc_clk_cleanup (struct isc_device *isc); +int isc_scaler_link (struct isc_device *isc); +int isc_scaler_init (struct isc_device *isc); +int isc_mc_init (struct isc_device *isc,u32 ver); +void isc_mc_cleanup (struct isc_device *isc); + +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index);#endif drivers/media/platform/atmel/atmel-sama5d2-isc.c +0
-0
drivers/media/platform/atmel/atmel-sama5d2-isc.c static int atmel_isc_probe(struct platform_device *pdev) · line 556
Quoted 4 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c +1
-0@@ -553,6 +553,12 @@ static int atmel_isc_probe(struct platform_device *pdev)break ; } +regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);
I am surprised you can read a register before runtime_pm is intialized!
Thanks
j
Quoted 83 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c +11
-2+ + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev);@@ -562,7 +568,7 @@ static int atmel_isc_probe(struct platform_device *pdev) ret =clk_prepare_enable (isc->ispck);if (ret) {dev_err (dev,"failed to enable ispck: %d\n" , ret); - goto cleanup_subdev; + goto isc_probe_mc_init_err; }/* ispck should be greater or equal to hclock */ @@ -572,7 +578,6 @@ static int atmel_isc_probe(struct platform_device *pdev) goto unprepare_clk; } -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip ISC version %x\n" , ver);return 0 ;@@ -580,6 +585,9 @@ static int atmel_isc_probe(struct platform_device *pdev) unprepare_clk:clk_disable_unprepare (isc->ispck); +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -600,6 +608,8 @@ static int atmel_isc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev);drivers/media/platform/atmel/atmel-sama7g5-isc.c +11
-2@@ -547,15 +547,23 @@ static int microchip_xisc_probe(struct platform_device *pdev)break ; } +regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver); + + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev); -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip XISC version %x\n" , ver);return 0 ; +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -576,6 +584,8 @@ static int microchip_xisc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev); --
Quoted 2 lines
2.25.1
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 138 lines · Jacopo Mondi
Hi Eugen
Quoted 135 lines · Eugen Hristev
Implement the support for media-controller.
This means that the capabilities of the driver have changed and now it also advertises theIO_MC.
The driver will register it's media device, and add the video entity to this media device. The subdevices are registered to the same media device. The ISC will have a base entity which is auto-detected asatmel_isc_base. It will also register a subdevice that allows cropping of the incoming frame to the maximum frame size supported by the ISC.
The ISC will create a link between the subdevice that is asynchronously registered and theatmel_isc_scalerentity.
Then, theatmel_isc_scalerandatmel_isc_baseare connected through another link.Signed-off-by Eugen Hristev <[email protected]>---
Changes in v4:
As suggested by Jacopo:
- renamedatmel_isc_mctoatmel_isc_scaler.c
- moved init_mc/clean_mc toisc_basefileChanges in v2:
- implement try formatsdrivers/media/platform/atmel/Makefile | 2 +- drivers/media/platform/atmel/atmel-isc-base.c | 73 +++++- .../media/platform/atmel/atmel-isc-scaler.c | 245 ++++++++++++++++++ drivers/media/platform/atmel/atmel-isc.h | 37 +++ .../media/platform/atmel/atmel-sama5d2-isc.c | 14 +- .../media/platform/atmel/atmel-sama7g5-isc.c | 12 +- 6 files changed, 375 insertions(+), 8 deletions(-)create mode 100644
drivers/media/platform/atmel/atmel-isc-scaler.cdrivers/media/platform/atmel/Makefile +1
-1@@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only atmel-isc-objs = atmel-sama5d2-isc.o atmel-xisc-objs = atmel-sama7g5-isc.o -atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o +atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o atmel-isc-scaler.o obj-$(CONFIG_VIDEO_ATMEL_ISI) += atmel-isi.o obj-$(CONFIG_VIDEO_ATMEL_ISC_BASE) += atmel-isc-common.odrivers/media/platform/atmel/atmel-isc-base.c +27
-4@@ -1710,6 +1710,7 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, struct isc_device, v4l2_dev);struct isc_subdev_entity *subdev_entity = container_of(notifier, struct isc_subdev_entity, notifier); +int pad;if (video_is_registered (&isc->video_dev)) {v4l2_err (&isc->v4l2_dev,"only supports one sub-device.\n" );@@ -1718,6 +1719,16 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, subdev_entity->sd = subdev; + pad =media_entity_get_fwnode_pad (&subdev->entity, asd->match.fwnode, +MEDIA_PAD_FL_SOURCE ); +if (pad <0 ) { +v4l2_err (&isc->v4l2_dev,"failed to find pad for %s\n" , + subdev->name); +return pad; + } + + isc->remote_pad = pad; +return 0 ; }@@ -1732,8 +1743,8 @@ static void isc_async_unbind(struct v4l2_async_notifier *notifier,v4l2_ctrl_handler_free (&isc->ctrls.handler); } -static struct isc_format *find_format_by_code (struct isc_device *isc, -unsigned int code,int *index) +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index) {struct isc_format *fmt = &isc->formats_list[0 ];unsigned int i;@@ -1749,6 +1760,7 @@ static struct isc_format *find_format_by_code(struct isc_device *isc,return NULL ; } +EXPORT_SYMBOL_GPL (isc_find_format_by_code);static int isc_formats_init (struct isc_device *isc) {@@ -1765,7 +1777,7 @@ static int isc_formats_init(struct isc_device *isc) NULL, &mbus_code)) { mbus_code.index++; - fmt =find_format_by_code (isc, mbus_code.code, &i); + fmt =isc_find_format_by_code (isc, mbus_code.code, &i);if (!fmt) {v4l2_warn (&isc->v4l2_dev,"Mbus code %x not supported\n" , mbus_code.code);@@ -1891,7 +1903,8 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) vdev->queue = q; vdev->lock = &isc->lock; vdev->ctrl_handler = &isc->ctrls.handler; - vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE ; + vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE | +V4L2_CAP_IO_MC ;video_set_drvdata (vdev, isc); ret =video_register_device (vdev,VFL_TYPE_VIDEO ,-1 );@@ -1901,8 +1914,18 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) goto isc_async_complete_err; } + ret =isc_scaler_link (isc); +if (ret <0 ) + goto isc_async_complete_unregister_device; + + ret =media_device_register (&isc->mdev); +if (ret <0 ) + goto isc_async_complete_unregister_device;return 0 ; +isc_async_complete_unregister_device: +video_unregister_device (vdev); + isc_async_complete_err:mutex_destroy (&isc->lock);return ret;@@ -1969,6 +1992,48 @@ int isc_pipeline_init(struct isc_device *isc) }EXPORT_SYMBOL_GPL (isc_pipeline_init);
Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-base.c int isc_pipeline_init(struct isc_device *isc) · line 2001
Quoted 7 lines
drivers/media/platform/atmel/atmel-isc-base.c +7
-0+int isc_mc_init (struct isc_device *isc,u32 ver) +{ +const struct of_device_id *match; +int ret; + + isc->video_dev.entity.function =MEDIA_ENT_F_IO_V4L ; + isc->video_dev.entity.flags =MEDIA_ENT_FL_DEFAULT ;Should you set entity.ops.link_validate =
v4l2_subdev_link_validateto be able to have the media pipeline validated atmedia_pipeline_start()time ?
Hi,
I am doing that in a subsequent patch. Things are not completely ready at this moment, because ISC still relies on the old mechanism to call v4l2_subdev_call with set_fmt on the subdevice (which subsequent patch removes and adds checks to the link validate ).
Quoted 77 lines
Quoted 45 lines
drivers/media/platform/atmel/atmel-isc-base.c +35
-0+ isc->pads[ISC_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + + ret =media_entity_pads_init (&isc->video_dev.entity,ISC_PADS_NUM , + isc->pads); +if (ret <0 ) { +dev_err (isc->dev,"media entity init failed\n" ); +return ret; + } + + isc->mdev.dev = isc->dev; + + match =of_match_node (isc->dev->driver->of_match_table, + isc->dev->of_node); + +strscpy (isc->mdev.driver_name,KBUILD_MODNAME , +sizeof (isc->mdev.driver_name)); +strscpy (isc->mdev.model, match->compatible,sizeof (isc->mdev.model)); +snprintf (isc->mdev.bus_info,sizeof (isc->mdev.bus_info),"platform:%s" , + isc->v4l2_dev.name); + isc->mdev.hw_revision = ver; + +media_device_init (&isc->mdev); + + isc->v4l2_dev.mdev = &isc->mdev; + +return isc_scaler_init (isc); +} +EXPORT_SYMBOL_GPL (isc_mc_init); + +void isc_mc_cleanup (struct isc_device *isc) +{ +media_entity_cleanup (&isc->video_dev.entity); +} +EXPORT_SYMBOL_GPL (isc_mc_cleanup); +/* regmap configuration */ #define ATMEL_ISC_REG_MAX 0xd5cconst struct regmap_config isc_regmap_config = {drivers/media/platform/atmel/atmel-isc-scaler.c new file +0
-0drivers/media/platform/atmel/atmel-isc-scaler.c line 5
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0@@ -0,0 +1,245 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Microchip Image Sensor Controller (ISC) Scaler entity support +* +* Copyright (C) 2021 Microchip Technology, Inc. Time flies! It's 2022 already :)
Quoted 24 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +24
-0+* +* Author: Eugen Hristev <[email protected]> +* +*/ + +#include <media/media-device.h> +#include <media/media-entity.h> +#include <media/v4l2-device.h> +#include <media/v4l2-subdev.h> + +#include "atmel-isc-regs.h" +#include "atmel-isc.h" + +static int isc_scaler_get_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *format) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; + +if (format->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + format->pad); + format->format = *v4l2_try_fmt;
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 34
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; + } + + format->format = isc->scaler_format;
isc->scaler_formatis only used inside this file if I'm not mistaken. I wonder why it lives in theisc_devicestruct.
isc_device is a placeholder for all isc things.
I would not create a separate struct in this file that would have to be allocated etc... just for one two things. So I preferred to have it in the same place as all the other things.
Quoted 16 lines
Quoted 16 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +15
-0+ +return 0 ; +} + +static int isc_scaler_set_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *req_fmt) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; +struct isc_format *fmt; +unsigned int i; + +if (req_fmt->pad ==ISC_SCALER_PAD_SOURCE ) + v4l_bound_align_image
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 55
Quoted 6 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +6
-0+ (&req_fmt->format.width,16 , isc->max_width,0 , + &req_fmt->format.height,16 , isc->max_height,0 ,0 ); + else + v4l_bound_align_image + (&req_fmt->format.width, 16, 10000, 0, + &req_fmt->format.height, 16, 10000, 0, 0);Where does 10000 come from ?
It's a random number. Do you have any suggestion for a better one ? Maybe this would be much more clear with my comments below (where I will explain what the isc scaler does )
In short, it allows the other entity in the link to 'sink' a huge format into this 'scaler' . Because the scaler can crop anything basically down to the biggest format size the ISC could handle.
Quoted 12 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 63
Quoted 9 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +8
-0+ + req_fmt->format.colorspace =V4L2_COLORSPACE_SRGB ; + req_fmt->format.field =V4L2_FIELD_NONE ; + req_fmt->format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + req_fmt->format.quantization =V4L2_QUANTIZATION_DEFAULT ; + req_fmt->format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + fmt =isc_find_format_by_code (isc, req_fmt->format.code, &i);So you rely on the isc format list for the scaler as well ? I think it's fine as long as they are identical
Yes, the scaler is kind of a dummy entity , but it's required by the media controller validation of the pipeline.
Quoted 35 lines
Quoted 18 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +17
-0+ +if (!fmt) + fmt = &isc->formats_list[0 ]; + + req_fmt->format.code = fmt->mbus_code; + +if (req_fmt->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + req_fmt->pad); + *v4l2_try_fmt = req_fmt->format; +/* Trying on the pad sink makes the source sink change too */ +if (req_fmt->pad ==ISC_SCALER_PAD_SINK ) { + v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state, +ISC_SCALER_PAD_SOURCE ); + *v4l2_try_fmt = req_fmt->format; +drivers/media/platform/atmel/atmel-isc-scaler.c line 90
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+v4l_bound_align_image (&v4l2_try_fmt->width, +16 , isc->max_width,0 , + &v4l2_try_fmt->height, +16 , isc->max_height,0 ,0 ); + } +/* if we are just trying, we are done */ +return 0 ; + } + + isc->scaler_format = req_fmt->format;No per-pad format but a global
scaler_format? How do you configure scaling ?Actually, I would like to know more about the scaler device capabilities. What functions can this IP perform ? Does it do cropping, can it also do (down)scaling or even composition ?
Quoted 8 lines
I think it is worth to read
https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-subdev.html#v4l2-subdev-selections
where it is reported how cropping and scaling are implemented by using the selection API.Figure 4.5 is particularly helpful to explain the simple crop case, for which you need to implement support to by adding
s_selectionon the sink padTGT_CROPtarget.
The scaler is kind of a dummy entity.
The problem is that different subdevices cannot be connected directly to a v4l entity, because the ISC does cropping on the incoming frame. The ISC has a limit on the total number of pixels per frame, thus it will do a crop.
So, if I allow e.g. an entity that sends 3280x2464 , when the ISC can only handle 3264x2464 , then we have two choices:
1/ crop it and output the 3264x2464 -> cannot, because userspace expects 3280x2464, and the whole pipeline fails to configure, there is a mismatch between /dev/video output and what the whole pipeline produces 2/ deny the link, and ask the subdevice to produce the 3264x2464 which the ISC can handle at most -> cannot , because the sensor let's say can *only* produce 3280x2464 and *any* other frame size returns an error.
The only solution to make both worlds happy, is to have a dummy entity called 'scaler' which in fact now it only performs a simple crop. It accepts any frame size at input (hence the 10000x10000 which you saw earlier), and outputs at most 3264x2464 the isc_max_height and isc_max_width.
So to answer your question, the isc scaler is a software model for a simple cropping, that would make media controller happy, and capture software happy.
Here it how it looks :
Device topology
- entity 1: atmel_isc_scaler (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev0
pad0: Sink
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb
crop.bounds:(0,0)/3264x2464
crop:(0,0)/3264x2464]
<- "csi2dc":1 [ENABLED,IMMUTABLE]
pad1: Source
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb]
-> "atmel_isc_common":0 [ENABLED,IMMUTABLE]- entity 4: csi2dc (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev1
pad0: Sink
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb]
<- "dw-csi.0":1 [ENABLED]
pad1: Source
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb]
-> "atmel_isc_scaler":0 [ENABLED,IMMUTABLE]- entity 7: dw-csi.0 (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev2
pad0: Sink
[fmt:SRGGB10_1X10/3280x2464]
<- "imx219 1-0010":0 [ENABLED]
pad1: Source
[fmt:SRGGB10_1X10/3280x2464]
-> "csi2dc":0 [ENABLED]- entity 12: imx219 1-0010 (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0
device node name /dev/v4l-subdev3
pad0: Source
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb
xfer:srgb ycbcr:601 quantization:full-range
crop.bounds:(8,8)/3280x2464
crop:(8,8)/3280x2464]
-> "dw-csi.0":0 [ENABLED]- entity 24: atmel_isc_common (1 pad, 1 link)
type Node subtype V4L flags 1
device node name /dev/video0
pad0: Sink
<- "atmel_isc_scaler":1 [ENABLED,IMMUTABLE]
Scaler does this one cute little thing :
pad0: Sink
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb
crop.bounds:(0,0)/3264x2464
crop:(0,0)/3264x2464]
<- "csi2dc":1 [ENABLED,IMMUTABLE]
pad1: Source
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb]
-> "atmel_isc_common":0 [ENABLED,IMMUTABLE]Which is what we needed.
Quoted 19 lines
Quoted 6 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; +} + +static int isc_scaler_enum_mbus_code(struct v4l2_subdev *sd,drivers/media/platform/atmel/atmel-isc-scaler.c line 105
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+struct v4l2_subdev_state *sd_state, + struct v4l2_subdev_mbus_code_enum *code) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +int supported_index =0 ; +int i; + +for (i =0 ; i < isc->formats_list_size; i++) { +if (!isc->formats_list[i].sd_support) +continue ;The
sd_supportflag still doesn't click in my head.
Quoted 2 lines
Shouldn't the enumeration of available formats on the scaler do not depend on the sensor supproted formats ?
You're right. I will have to check it again.
Quoted 30 lines
Quoted 16 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +15
-0+if (supported_index == code->index) { + code->code = isc->formats_list[i].mbus_code; +return 0 ; + } + supported_index++; + } + +return -EINVAL ; +} + +static int isc_scaler_g_sel (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_selection *sel) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd);drivers/media/platform/atmel/atmel-isc-scaler.c line 130
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +if (sel->pad ==ISC_SCALER_PAD_SOURCE ) +return -EINVAL ; + +if (sel->target !=V4L2_SEL_TGT_CROP_BOUNDS && + sel->target !=V4L2_SEL_TGT_CROP ) +return -EINVAL ; + + sel->r.height = isc->max_height; + sel->r.width = isc->max_width;The
CROP_BOUNDSshould be set to the same size as the sink pad image format, as it represents the maximum valid crop rectangle.
Quoted 3 lines
TGT_CROPshould report the configured crop rectangle which can be intiialized to the same size asCROP_BOUNDS, if my understanding of the spec is correct
So you would like to have this differentiated, and report the CROP_BOUNDS to whatever is on the sink pad, and the TGT_CROP to what is reported now, the maximum size of the ISC frame .
My understanding is correct ?
Quoted 44 lines
Quoted 16 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +15
-0+ + sel->r.left =0 ; + sel->r.top =0 ; + +return 0 ; +} + +static int isc_scaler_init_cfg (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state) +{ +struct v4l2_mbus_framefmt *v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state,0 ); + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); + + *v4l2_try_fmt = isc->scaler_format;drivers/media/platform/atmel/atmel-isc-scaler.c line 155
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +return 0 ; +} + +static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = { + .enum_mbus_code = isc_scaler_enum_mbus_code, + .set_fmt = isc_scaler_set_fmt, + .get_fmt = isc_scaler_get_fmt, + .get_selection = isc_scaler_g_sel, + .init_cfg = isc_scaler_init_cfg,.link_validate =
v4l2_subdev_link_validate_default,To have the formats at the end of links that point to this entity validated (I think the framework already calls it if not set though, please check v4l2-subdev.c:v4l2_subdev_link_validate())
Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +11
-0+}; + +static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = { + .pad = &isc_scaler_pad_ops, +}; + +int isc_scaler_init (struct isc_device *isc) +{ +int ret; + +v4l2_subdev_init (&isc->scaler_sd, &xisc_scaler_subdev_ops);
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 171
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ + isc->scaler_sd.owner =THIS_MODULE ; + isc->scaler_sd.dev = isc->dev; +snprintf (isc->scaler_sd.name,sizeof (isc->scaler_sd.name), +"atmel_isc_scaler" );I would drop 'atmel' for brevity, unless other entities have this prefix set already
The v4l entity takes it's name from the module name, which is atmel_isc_common, so I thought to keep the prefix
Quoted 70 lines
Quoted 70 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +69
-0+ + isc->scaler_sd.flags |=V4L2_SUBDEV_FL_HAS_DEVNODE ; + isc->scaler_sd.entity.function =MEDIA_ENT_F_PROC_VIDEO_SCALER ; + isc->scaler_pads[ISC_SCALER_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + isc->scaler_pads[ISC_SCALER_PAD_SOURCE ].flags =MEDIA_PAD_FL_SOURCE ; + + isc->scaler_format.height = isc->max_height; + isc->scaler_format.width = isc->max_width; + isc->scaler_format.code = isc->formats_list[0 ].mbus_code; + isc->scaler_format.colorspace =V4L2_COLORSPACE_SRGB ; + isc->scaler_format.field =V4L2_FIELD_NONE ; + isc->scaler_format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + isc->scaler_format.quantization =V4L2_QUANTIZATION_DEFAULT ; + isc->scaler_format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + ret =media_entity_pads_init (&isc->scaler_sd.entity, +ISC_SCALER_PADS_NUM , + isc->scaler_pads); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd media entity init failed\n" ); +return ret; + } + ret =v4l2_device_register_subdev (&isc->v4l2_dev, &isc->scaler_sd); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd failed to register subdev\n" ); +return ret; + } + +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_init); + +int isc_scaler_link (struct isc_device *isc) +{ +int ret; + + ret =media_create_pad_link (&isc->current_subdev->sd->entity, + isc->remote_pad, &isc->scaler_sd.entity, +ISC_SCALER_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->current_subdev->sd->entity.name, + isc->scaler_sd.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , + isc->current_subdev->sd->name, isc->remote_pad); + + ret =media_create_pad_link (&isc->scaler_sd.entity, +ISC_SCALER_PAD_SOURCE , + &isc->video_dev.entity,ISC_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->scaler_sd.entity.name, + isc->video_dev.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , isc->scaler_sd.name, +ISC_SCALER_PAD_SOURCE );
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 244
Quoted 4 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +4
-0+ +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_link);From the DT graph point of view, the ISC appears as a single block
with an CSI-2 input port. It is then in charge of parsing the .dts and add to its notifier the image sensor remote subdevice.
Actually it's only a parallel port.
And it can be connected either to a sensor directly or to the csi2dc bridge. (the csi2dc bridge can be connected to a CSI-2 receiver)
Quoted 4 lines
When the sensor subdev registers and the ISC notifier completes, the scaler entity which was initialized at
isc_probe()time is linked in between the ISC and the image sensor immutably.
yes, because this cannot change at runtime, and usually, it can't change without altering the board hardware. (at least this is what my understanding of immutability is )
Quoted 9 lines
I think it is fine for now, but I wonder if you plan to plumb more components between the ISC video node and the sensor, if it's not worth changing the DT bindings and their parsing logic to separate the CSI-2 receiver from the ISC, whcih can create its media graph at probe time and have the CSI-2 receiver entity as downstream remote. I think I need to get to know the ISC better to really have an idea. For now, this seems ok to me, but please check with maintainers if this is fine with them.
In the XISC case (sama7g5-isc), the csi2dc receiver is the subdev (so, the remote subdev).
The ISC will register a scaler, and connect the subdev to this scaler first, and then, the scaler to the isc itself (the v4l entity).
Quoted 45 lines
Quoted 45 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +1
-0+drivers/media/platform/atmel/atmel-isc.h +21
-0@@ -183,6 +183,17 @@ struct isc_reg_offsets {u32 his_entry; }; +enum isc_mc_pads { +ISC_PAD_SINK =0 , +ISC_PADS_NUM =1 , +}; + +enum isc_scaler_pads { +ISC_SCALER_PAD_SINK =0 , +ISC_SCALER_PAD_SOURCE =1 , +ISC_SCALER_PADS_NUM =2 , +}; +/* * struct isc_device - ISC device driver data/config struct * @regmap: Register map @@ -257,6 +268,12 @@ struct isc_reg_offsets {* be used as an input to the controller * @controller_formats_size: size of controller_formats array * @formats_list_size: size of formats_list array +* @pads: media controller pads for isc video entity +* @mdev: media device that is registered by the isc +* @remote_pad: remote pad on the connected subdevice +* @scaler_sd: subdevice for the scaler that isc registers +* @scaler_pads: media controller pads for the scaler subdevice +* @scaler_format: current format for the scaler subdevice */ struct isc_device {struct regmap *regmap;@@ -344,6 +361,19 @@ struct isc_device {struct isc_format *formats_list;u32 controller_formats_size;u32 formats_list_size; + +struct { +struct media_pad pads[ISC_PADS_NUM ]; +struct media_device mdev;
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc.h struct isc_device { · line 375
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc.h +8
-0+ +u32 remote_pad; + }; + +struct { +struct v4l2_subdev scaler_sd; +struct media_pad scaler_pads[ISC_SCALER_PADS_NUM ]; +struct v4l2_mbus_framefmt scaler_format;
Here are the scaler stuff which I added, in the same bulk struct for the whole isc device
Quoted 24 lines
Quoted 24 lines
drivers/media/platform/atmel/atmel-isc.h +8
-0+ }; };extern const struct regmap_config isc_regmap_config;@@ -355,4 +385,11 @@ int isc_clk_init(struct isc_device *isc);void isc_subdev_cleanup (struct isc_device *isc);void isc_clk_cleanup (struct isc_device *isc); +int isc_scaler_link (struct isc_device *isc); +int isc_scaler_init (struct isc_device *isc); +int isc_mc_init (struct isc_device *isc,u32 ver); +void isc_mc_cleanup (struct isc_device *isc); + +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index);#endif drivers/media/platform/atmel/atmel-sama5d2-isc.c +0
-0@@ -553,6 +553,12 @@ static int atmel_isc_probe(struct platform_device *pdev)break ; }
Quoted 5 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c static int atmel_isc_probe(struct platform_device *pdev) · line 556
Quoted 1 line
drivers/media/platform/atmel/atmel-sama5d2-isc.c +1
-0+regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);I am surprised you can read a register before
runtime_pmis intialized!
Actually I can read any register after the moment of starting the clock on the hardware block.
Maybe the starting and stopping of the clock needs to be moved to the runtime_pm calls, but this is another story, not related to the media controller.
I moved this line because I had to pass the version to the isc_mc_init call
Quoted 2 lines
Thanks
j
Thanks for reviewing !
Eugen
Quoted 79 lines
Quoted 79 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c +11
-2+ + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev);@@ -562,7 +568,7 @@ static int atmel_isc_probe(struct platform_device *pdev) ret =clk_prepare_enable (isc->ispck);if (ret) {dev_err (dev,"failed to enable ispck: %d\n" , ret); - goto cleanup_subdev; + goto isc_probe_mc_init_err; }/* ispck should be greater or equal to hclock */ @@ -572,7 +578,6 @@ static int atmel_isc_probe(struct platform_device *pdev) goto unprepare_clk; } -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip ISC version %x\n" , ver);return 0 ;@@ -580,6 +585,9 @@ static int atmel_isc_probe(struct platform_device *pdev) unprepare_clk:clk_disable_unprepare (isc->ispck); +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -600,6 +608,8 @@ static int atmel_isc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev);drivers/media/platform/atmel/atmel-sama7g5-isc.c +9
-1@@ -547,15 +547,23 @@ static int microchip_xisc_probe(struct platform_device *pdev)break ; } +regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver); + + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev); -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip XISC version %x\n" , ver);return 0 ; +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -576,6 +584,8 @@ static int microchip_xisc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev);
Quoted 8 lines
Quoted 8 lines
drivers/media/platform/atmel/atmel-sama7g5-isc.c +2
-1+isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev); --2.25.1
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Hi Eugen
Quoted 151 lines · On Mon, Feb 07, 2022 at 01:40:31PM +0000, [email protected] wrote:
Quoted 138 lines · Jacopo Mondi
Hi Eugen
Quoted 135 lines · Eugen Hristev
Implement the support for media-controller.
This means that the capabilities of the driver have changed and now it also advertises theIO_MC.
The driver will register it's media device, and add the video entity to this media device. The subdevices are registered to the same media device. The ISC will have a base entity which is auto-detected asatmel_isc_base. It will also register a subdevice that allows cropping of the incoming frame to the maximum frame size supported by the ISC.
The ISC will create a link between the subdevice that is asynchronously registered and theatmel_isc_scalerentity.
Then, theatmel_isc_scalerandatmel_isc_baseare connected through another link.Signed-off-by Eugen Hristev <[email protected]>---
Changes in v4:
As suggested by Jacopo:
- renamedatmel_isc_mctoatmel_isc_scaler.c
- moved init_mc/clean_mc toisc_basefileChanges in v2:
- implement try formatsdrivers/media/platform/atmel/Makefile | 2 +- drivers/media/platform/atmel/atmel-isc-base.c | 73 +++++- .../media/platform/atmel/atmel-isc-scaler.c | 245 ++++++++++++++++++ drivers/media/platform/atmel/atmel-isc.h | 37 +++ .../media/platform/atmel/atmel-sama5d2-isc.c | 14 +- .../media/platform/atmel/atmel-sama7g5-isc.c | 12 +- 6 files changed, 375 insertions(+), 8 deletions(-)create mode 100644
drivers/media/platform/atmel/atmel-isc-scaler.cdrivers/media/platform/atmel/Makefile +1
-1@@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only atmel-isc-objs = atmel-sama5d2-isc.o atmel-xisc-objs = atmel-sama7g5-isc.o -atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o +atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o atmel-isc-scaler.o obj-$(CONFIG_VIDEO_ATMEL_ISI) += atmel-isi.o obj-$(CONFIG_VIDEO_ATMEL_ISC_BASE) += atmel-isc-common.odrivers/media/platform/atmel/atmel-isc-base.c +27
-4@@ -1710,6 +1710,7 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, struct isc_device, v4l2_dev);struct isc_subdev_entity *subdev_entity = container_of(notifier, struct isc_subdev_entity, notifier); +int pad;if (video_is_registered (&isc->video_dev)) {v4l2_err (&isc->v4l2_dev,"only supports one sub-device.\n" );@@ -1718,6 +1719,16 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, subdev_entity->sd = subdev; + pad =media_entity_get_fwnode_pad (&subdev->entity, asd->match.fwnode, +MEDIA_PAD_FL_SOURCE ); +if (pad <0 ) { +v4l2_err (&isc->v4l2_dev,"failed to find pad for %s\n" , + subdev->name); +return pad; + } + + isc->remote_pad = pad; +return 0 ; }@@ -1732,8 +1743,8 @@ static void isc_async_unbind(struct v4l2_async_notifier *notifier,v4l2_ctrl_handler_free (&isc->ctrls.handler); } -static struct isc_format *find_format_by_code (struct isc_device *isc, -unsigned int code,int *index) +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index) {struct isc_format *fmt = &isc->formats_list[0 ];unsigned int i;@@ -1749,6 +1760,7 @@ static struct isc_format *find_format_by_code(struct isc_device *isc,return NULL ; } +EXPORT_SYMBOL_GPL (isc_find_format_by_code);static int isc_formats_init (struct isc_device *isc) {@@ -1765,7 +1777,7 @@ static int isc_formats_init(struct isc_device *isc) NULL, &mbus_code)) { mbus_code.index++; - fmt =find_format_by_code (isc, mbus_code.code, &i); + fmt =isc_find_format_by_code (isc, mbus_code.code, &i);if (!fmt) {v4l2_warn (&isc->v4l2_dev,"Mbus code %x not supported\n" , mbus_code.code);@@ -1891,7 +1903,8 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) vdev->queue = q; vdev->lock = &isc->lock; vdev->ctrl_handler = &isc->ctrls.handler; - vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE ; + vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE | +V4L2_CAP_IO_MC ;video_set_drvdata (vdev, isc); ret =video_register_device (vdev,VFL_TYPE_VIDEO ,-1 );@@ -1901,8 +1914,18 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) goto isc_async_complete_err; } + ret =isc_scaler_link (isc); +if (ret <0 ) + goto isc_async_complete_unregister_device; + + ret =media_device_register (&isc->mdev); +if (ret <0 ) + goto isc_async_complete_unregister_device;return 0 ; +isc_async_complete_unregister_device: +video_unregister_device (vdev); + isc_async_complete_err:mutex_destroy (&isc->lock);return ret;@@ -1969,6 +1992,48 @@ int isc_pipeline_init(struct isc_device *isc) }EXPORT_SYMBOL_GPL (isc_pipeline_init);Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-base.c int isc_pipeline_init(struct isc_device *isc) · line 2001
Quoted 7 lines
drivers/media/platform/atmel/atmel-isc-base.c +7
-0+int isc_mc_init (struct isc_device *isc,u32 ver) +{ +const struct of_device_id *match; +int ret; + + isc->video_dev.entity.function =MEDIA_ENT_F_IO_V4L ; + isc->video_dev.entity.flags =MEDIA_ENT_FL_DEFAULT ;Should you set entity.ops.link_validate =
v4l2_subdev_link_validateto be able to have the media pipeline validated atmedia_pipeline_start()time ?
Quoted 7 lines
Hi,
I am doing that in a subsequent patch. Things are not completely ready at this moment, because ISC still relies on the old mechanism to call
v4l2_subdev_callwithset_fmton the subdevice (which subsequent patch removes and adds checks to the link validate ).
I see.. the subsequent patches are not part of this series, right ?
I think patches 1, 2, 4, 5 and 6 from this series do not depend on the introduction of media controller or the scaler, right ? (I'm not sure if the DTS patches do).
If that's the case, would it make sense to to fast-track them (as some of them are fixes) and then on top plumb the media controller infrastructure with this patch and the ones you have been mentioning ?
Quoted 117 lines
Quoted 77 lines
Quoted 45 lines
drivers/media/platform/atmel/atmel-isc-base.c +35
-0+ isc->pads[ISC_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + + ret =media_entity_pads_init (&isc->video_dev.entity,ISC_PADS_NUM , + isc->pads); +if (ret <0 ) { +dev_err (isc->dev,"media entity init failed\n" ); +return ret; + } + + isc->mdev.dev = isc->dev; + + match =of_match_node (isc->dev->driver->of_match_table, + isc->dev->of_node); + +strscpy (isc->mdev.driver_name,KBUILD_MODNAME , +sizeof (isc->mdev.driver_name)); +strscpy (isc->mdev.model, match->compatible,sizeof (isc->mdev.model)); +snprintf (isc->mdev.bus_info,sizeof (isc->mdev.bus_info),"platform:%s" , + isc->v4l2_dev.name); + isc->mdev.hw_revision = ver; + +media_device_init (&isc->mdev); + + isc->v4l2_dev.mdev = &isc->mdev; + +return isc_scaler_init (isc); +} +EXPORT_SYMBOL_GPL (isc_mc_init); + +void isc_mc_cleanup (struct isc_device *isc) +{ +media_entity_cleanup (&isc->video_dev.entity); +} +EXPORT_SYMBOL_GPL (isc_mc_cleanup); +/* regmap configuration */ #define ATMEL_ISC_REG_MAX 0xd5cconst struct regmap_config isc_regmap_config = {drivers/media/platform/atmel/atmel-isc-scaler.c new file +0
-0drivers/media/platform/atmel/atmel-isc-scaler.c line 5
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0@@ -0,0 +1,245 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Microchip Image Sensor Controller (ISC) Scaler entity support +* +* Copyright (C) 2021 Microchip Technology, Inc. Time flies! It's 2022 already :)
Quoted 24 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +24
-0+* +* Author: Eugen Hristev <[email protected]> +* +*/ + +#include <media/media-device.h> +#include <media/media-entity.h> +#include <media/v4l2-device.h> +#include <media/v4l2-subdev.h> + +#include "atmel-isc-regs.h" +#include "atmel-isc.h" + +static int isc_scaler_get_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *format) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; + +if (format->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + format->pad); + format->format = *v4l2_try_fmt;Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 34
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; + } + + format->format = isc->scaler_format;
isc->scaler_formatis only used inside this file if I'm not mistaken. I wonder why it lives in theisc_devicestruct.
isc_deviceis a placeholder for all isc things.I would not create a separate struct in this file that would have to be allocated etc... just for one two things. So I preferred to have it in the same place as all the other things.
Quoted 16 lines
Quoted 16 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +15
-0+ +return 0 ; +} + +static int isc_scaler_set_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *req_fmt) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; +struct isc_format *fmt; +unsigned int i; + +if (req_fmt->pad ==ISC_SCALER_PAD_SOURCE ) + v4l_bound_align_imageQuoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 55
Quoted 6 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +6
-0+ (&req_fmt->format.width,16 , isc->max_width,0 , + &req_fmt->format.height,16 , isc->max_height,0 ,0 ); + else + v4l_bound_align_image + (&req_fmt->format.width, 16, 10000, 0, + &req_fmt->format.height, 16, 10000, 0, 0);Where does 10000 come from ?
Quoted 1 line
It's a random number. Do you have any suggestion for a better one ?
An actual HW limit ? :)
Quoted 6 lines
Maybe this would be much more clear with my comments below (where I will explain what the isc scaler does )
In short, it allows the other entity in the link to 'sink' a huge format into this 'scaler' . Because the scaler can crop anything basically down to the biggest format size the ISC could handle.
doesn't it have any documented input size limit ?
Quoted 13 lines
Quoted 12 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 63
Quoted 9 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +8
-0+ + req_fmt->format.colorspace =V4L2_COLORSPACE_SRGB ; + req_fmt->format.field =V4L2_FIELD_NONE ; + req_fmt->format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + req_fmt->format.quantization =V4L2_QUANTIZATION_DEFAULT ; + req_fmt->format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + fmt =isc_find_format_by_code (isc, req_fmt->format.code, &i);So you rely on the isc format list for the scaler as well ? I think it's fine as long as they are identical
Quoted 3 lines
Yes, the scaler is kind of a dummy entity , but it's required by the media controller validation of the pipeline.
Quoted 44 lines
Quoted 35 lines
Quoted 18 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +17
-0+ +if (!fmt) + fmt = &isc->formats_list[0 ]; + + req_fmt->format.code = fmt->mbus_code; + +if (req_fmt->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + req_fmt->pad); + *v4l2_try_fmt = req_fmt->format; +/* Trying on the pad sink makes the source sink change too */ +if (req_fmt->pad ==ISC_SCALER_PAD_SINK ) { + v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state, +ISC_SCALER_PAD_SOURCE ); + *v4l2_try_fmt = req_fmt->format; +drivers/media/platform/atmel/atmel-isc-scaler.c line 90
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+v4l_bound_align_image (&v4l2_try_fmt->width, +16 , isc->max_width,0 , + &v4l2_try_fmt->height, +16 , isc->max_height,0 ,0 ); + } +/* if we are just trying, we are done */ +return 0 ; + } + + isc->scaler_format = req_fmt->format;No per-pad format but a global
scaler_format? How do you configure scaling ?Actually, I would like to know more about the scaler device capabilities. What functions can this IP perform ? Does it do cropping, can it also do (down)scaling or even composition ?
Quoted 8 lines
I think it is worth to read
https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-subdev.html#v4l2-subdev-selections
where it is reported how cropping and scaling are implemented by using the selection API.Figure 4.5 is particularly helpful to explain the simple crop case, for which you need to implement support to by adding
s_selectionon the sink padTGT_CROPtarget.
Quoted 5 lines
The scaler is kind of a dummy entity.
The problem is that different subdevices cannot be connected directly to a v4l entity, because the ISC does cropping on the incoming frame. The ISC has a limit on the total number of pixels per frame, thus it will do a crop.
What is this limit related to ? I don't know the ISC internals, but is the limitation due to the overall image size, or maybe it's due to some line buffers being limited in the size they can transfer to memory ? Is it a limitation on the size of the image in pixels, or is it due to a limitation of the processing bandwidth on the input interface and thus depends on the actual size of the image in bytes ?
Quoted 5 lines
So, if I allow e.g. an entity that sends 3280x2464 , when the ISC can only handle 3264x2464 , then we have two choices:
1/ crop it and output the 3264x2464 -> cannot, because userspace expects 3280x2464, and the whole pipeline fails to configure, there is a mismatch between/dev/videooutput and what the whole pipeline produces
I'm confused in first place. If the ISC cannot output anything larger than 3264x2464, then if userspace sets a 3280x2464 size on the ISC, this should be adjusted to 3264x2464.
Or is the ISC limitation on the -input- side ?
Quoted 3 lines
2/ deny the link, and ask the subdevice to produce the 3264x2464 which the ISC can handle at most -> cannot , because the sensor let's say can *only* produce 3280x2464 and *any* other frame size returns an error.
buggy sensor I would say :)
Quoted 6 lines
The only solution to make both worlds happy, is to have a dummy entity called 'scaler' which in fact now it only performs a simple crop. It accepts any frame size at input (hence the 10000x10000 which you saw earlier), and outputs at most 3264x2464 the
isc_max_heightandisc_max_width.
I have a maybe dumb question: can the cropping operation be modeled by applying a TGT_CROP rectangle (whose _BOUND size is 3280x2464) to the atmel_isc_common entity sink pad and avoid the scaler completely ?
Quoted 59 lines
So to answer your question, the isc scaler is a software model for a simple cropping, that would make media controller happy, and capture software happy.
Here it how it looks :
Device topology
- entity 1:atmel_isc_scaler(2 pads, 2 links)type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev0 pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb crop.bounds:(0,0)/3264x2464 crop:(0,0)/3264x2464] <- "csi2dc":1 [ENABLED,IMMUTABLE] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] -> "atmel_isc_common":0 [ENABLED,IMMUTABLE]- entity 4: csi2dc (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev1 pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] <- "dw-csi.0":1 [ENABLED] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] -> "atmel_isc_scaler":0 [ENABLED,IMMUTABLE]- entity 7: dw-csi.0 (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev2 pad0: Sink [fmt:SRGGB10_1X10/3280x2464] <- "imx219 1-0010":0 [ENABLED] pad1: Source [fmt:SRGGB10_1X10/3280x2464] -> "csi2dc":0 [ENABLED]- entity 12: imx219 1-0010 (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0
device node name/dev/v4l-subdev3
pad0: Source
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb
xfer:srgb ycbcr:601 quantization:full-rangecrop.bounds:(8,8)/3280x2464 crop:(8,8)/3280x2464] -> "dw-csi.0":0 [ENABLED]- entity 24:
atmel_isc_common(1 pad, 1 link)
type Node subtype V4L flags 1
device node name/dev/video0
pad0: Sink
<- "atmel_isc_scaler":1 [ENABLED,IMMUTABLE]Scaler does this one cute little thing :
Quoted 7 lines
pad0: Sink
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb
crop.bounds:(0,0)/3264x2464
crop:(0,0)/3264x2464]
<- "csi2dc":1 [ENABLED,IMMUTABLE]
pad1: Source
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb]Shouldn't this be 3264x2464 as that's what the entity outputs after the cropping ? And shouldn't then be 3264x2464 the size output from the video device too ?
Quoted 59 lines
-> "atmel_isc_common":0 [ENABLED,IMMUTABLE]Which is what we needed.
Quoted 19 lines
Quoted 6 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; +} + +static int isc_scaler_enum_mbus_code(struct v4l2_subdev *sd,drivers/media/platform/atmel/atmel-isc-scaler.c line 105
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+struct v4l2_subdev_state *sd_state, + struct v4l2_subdev_mbus_code_enum *code) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +int supported_index =0 ; +int i; + +for (i =0 ; i < isc->formats_list_size; i++) { +if (!isc->formats_list[i].sd_support) +continue ;The
sd_supportflag still doesn't click in my head.Quoted 2 lines
Shouldn't the enumeration of available formats on the scaler do not depend on the sensor supproted formats ?
You're right. I will have to check it again.
Quoted 30 lines
Quoted 16 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +15
-0+if (supported_index == code->index) { + code->code = isc->formats_list[i].mbus_code; +return 0 ; + } + supported_index++; + } + +return -EINVAL ; +} + +static int isc_scaler_g_sel (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_selection *sel) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd);drivers/media/platform/atmel/atmel-isc-scaler.c line 130
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +if (sel->pad ==ISC_SCALER_PAD_SOURCE ) +return -EINVAL ; + +if (sel->target !=V4L2_SEL_TGT_CROP_BOUNDS && + sel->target !=V4L2_SEL_TGT_CROP ) +return -EINVAL ; + + sel->r.height = isc->max_height; + sel->r.width = isc->max_width;The
CROP_BOUNDSshould be set to the same size as the sink pad image format, as it represents the maximum valid crop rectangle.
Quoted 9 lines
Quoted 3 lines
TGT_CROPshould report the configured crop rectangle which can be intiialized to the same size asCROP_BOUNDS, if my understanding of the spec is correctSo you would like to have this differentiated, and report the
CROP_BOUNDSto whatever is on the sink pad, and theTGT_CROPto what is reported now, the maximum size of the ISC frame .
My understanding is correct ?
I didn't know you have an HW limitation, so your _BOUNDS is not the input image size but rather 3264x2464 ( == max_width x max_height).
What I meant is that _BOUNDS should report the maximum rectangle size that can be applied to the _CROP target. In you case you have an HW limitation 3264x2464 and that's the largest rectangle you can apply. TGT_CROP can be initialized to the same as _BOUND, but if you implement s_selection it should report what has been there applied. But as you don't implement s_selection yet, I think this is fine for now. Maybe a little comment ?
Also, is set->r zeroed by the framework before getting here ? Otherwise you should set r.left and r.top to 0
Quoted 53 lines
Quoted 44 lines
Quoted 16 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +15
-0+ + sel->r.left =0 ; + sel->r.top =0 ; + +return 0 ; +} + +static int isc_scaler_init_cfg (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state) +{ +struct v4l2_mbus_framefmt *v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state,0 ); + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); + + *v4l2_try_fmt = isc->scaler_format;drivers/media/platform/atmel/atmel-isc-scaler.c line 155
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +return 0 ; +} + +static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = { + .enum_mbus_code = isc_scaler_enum_mbus_code, + .set_fmt = isc_scaler_set_fmt, + .get_fmt = isc_scaler_get_fmt, + .get_selection = isc_scaler_g_sel, + .init_cfg = isc_scaler_init_cfg,.link_validate =
v4l2_subdev_link_validate_default,To have the formats at the end of links that point to this entity validated (I think the framework already calls it if not set though, please check v4l2-subdev.c:v4l2_subdev_link_validate())
Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +11
-0+}; + +static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = { + .pad = &isc_scaler_pad_ops, +}; + +int isc_scaler_init (struct isc_device *isc) +{ +int ret; + +v4l2_subdev_init (&isc->scaler_sd, &xisc_scaler_subdev_ops);Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 171
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ + isc->scaler_sd.owner =THIS_MODULE ; + isc->scaler_sd.dev = isc->dev; +snprintf (isc->scaler_sd.name,sizeof (isc->scaler_sd.name), +"atmel_isc_scaler" );I would drop 'atmel' for brevity, unless other entities have this prefix set already
Quoted 3 lines
The v4l entity takes it's name from the module name, which is
atmel_isc_common, so I thought to keep the prefix
Ack!
Quoted 189 lines
Quoted 70 lines
Quoted 70 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +69
-0+ + isc->scaler_sd.flags |=V4L2_SUBDEV_FL_HAS_DEVNODE ; + isc->scaler_sd.entity.function =MEDIA_ENT_F_PROC_VIDEO_SCALER ; + isc->scaler_pads[ISC_SCALER_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + isc->scaler_pads[ISC_SCALER_PAD_SOURCE ].flags =MEDIA_PAD_FL_SOURCE ; + + isc->scaler_format.height = isc->max_height; + isc->scaler_format.width = isc->max_width; + isc->scaler_format.code = isc->formats_list[0 ].mbus_code; + isc->scaler_format.colorspace =V4L2_COLORSPACE_SRGB ; + isc->scaler_format.field =V4L2_FIELD_NONE ; + isc->scaler_format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + isc->scaler_format.quantization =V4L2_QUANTIZATION_DEFAULT ; + isc->scaler_format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + ret =media_entity_pads_init (&isc->scaler_sd.entity, +ISC_SCALER_PADS_NUM , + isc->scaler_pads); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd media entity init failed\n" ); +return ret; + } + ret =v4l2_device_register_subdev (&isc->v4l2_dev, &isc->scaler_sd); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd failed to register subdev\n" ); +return ret; + } + +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_init); + +int isc_scaler_link (struct isc_device *isc) +{ +int ret; + + ret =media_create_pad_link (&isc->current_subdev->sd->entity, + isc->remote_pad, &isc->scaler_sd.entity, +ISC_SCALER_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->current_subdev->sd->entity.name, + isc->scaler_sd.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , + isc->current_subdev->sd->name, isc->remote_pad); + + ret =media_create_pad_link (&isc->scaler_sd.entity, +ISC_SCALER_PAD_SOURCE , + &isc->video_dev.entity,ISC_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->scaler_sd.entity.name, + isc->video_dev.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , isc->scaler_sd.name, +ISC_SCALER_PAD_SOURCE );Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 244
Quoted 4 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +4
-0+ +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_link);From the DT graph point of view, the ISC appears as a single block
with an CSI-2 input port. It is then in charge of parsing the .dts and add to its notifier the image sensor remote subdevice.Actually it's only a parallel port.
And it can be connected either to a sensor directly or to the csi2dc bridge. (the csi2dc bridge can be connected to a CSI-2 receiver)Quoted 4 lines
When the sensor subdev registers and the ISC notifier completes, the scaler entity which was initialized at
isc_probe()time is linked in between the ISC and the image sensor immutably.yes, because this cannot change at runtime, and usually, it can't change without altering the board hardware. (at least this is what my understanding of immutability is )
Quoted 9 lines
I think it is fine for now, but I wonder if you plan to plumb more components between the ISC video node and the sensor, if it's not worth changing the DT bindings and their parsing logic to separate the CSI-2 receiver from the ISC, whcih can create its media graph at probe time and have the CSI-2 receiver entity as downstream remote. I think I need to get to know the ISC better to really have an idea. For now, this seems ok to me, but please check with maintainers if this is fine with them.
In the XISC case (sama7g5-isc), the csi2dc receiver is the subdev (so, the remote subdev).
The ISC will register a scaler, and connect the subdev to this scaler first, and then, the scaler to the isc itself (the v4l entity).Quoted 45 lines
Quoted 45 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +1
-0+drivers/media/platform/atmel/atmel-isc.h +21
-0@@ -183,6 +183,17 @@ struct isc_reg_offsets {u32 his_entry; }; +enum isc_mc_pads { +ISC_PAD_SINK =0 , +ISC_PADS_NUM =1 , +}; + +enum isc_scaler_pads { +ISC_SCALER_PAD_SINK =0 , +ISC_SCALER_PAD_SOURCE =1 , +ISC_SCALER_PADS_NUM =2 , +}; +/* * struct isc_device - ISC device driver data/config struct * @regmap: Register map @@ -257,6 +268,12 @@ struct isc_reg_offsets {* be used as an input to the controller * @controller_formats_size: size of controller_formats array * @formats_list_size: size of formats_list array +* @pads: media controller pads for isc video entity +* @mdev: media device that is registered by the isc +* @remote_pad: remote pad on the connected subdevice +* @scaler_sd: subdevice for the scaler that isc registers +* @scaler_pads: media controller pads for the scaler subdevice +* @scaler_format: current format for the scaler subdevice */ struct isc_device {struct regmap *regmap;@@ -344,6 +361,19 @@ struct isc_device {struct isc_format *formats_list;u32 controller_formats_size;u32 formats_list_size; + +struct { +struct media_pad pads[ISC_PADS_NUM ]; +struct media_device mdev;Quoted 8 lines
drivers/media/platform/atmel/atmel-isc.h struct isc_device { · line 375
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc.h +8
-0+ +u32 remote_pad; + }; + +struct { +struct v4l2_subdev scaler_sd; +struct media_pad scaler_pads[ISC_SCALER_PADS_NUM ]; +struct v4l2_mbus_framefmt scaler_format;Here are the scaler stuff which I added, in the same bulk struct for the whole isc device
Quoted 24 lines
Quoted 24 lines
drivers/media/platform/atmel/atmel-isc.h +8
-0+ }; };extern const struct regmap_config isc_regmap_config;@@ -355,4 +385,11 @@ int isc_clk_init(struct isc_device *isc);void isc_subdev_cleanup (struct isc_device *isc);void isc_clk_cleanup (struct isc_device *isc); +int isc_scaler_link (struct isc_device *isc); +int isc_scaler_init (struct isc_device *isc); +int isc_mc_init (struct isc_device *isc,u32 ver); +void isc_mc_cleanup (struct isc_device *isc); + +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index);#endif drivers/media/platform/atmel/atmel-sama5d2-isc.c +0
-0@@ -553,6 +553,12 @@ static int atmel_isc_probe(struct platform_device *pdev)break ; }
Quoted 8 lines
Quoted 5 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c static int atmel_isc_probe(struct platform_device *pdev) · line 556
Quoted 1 line
drivers/media/platform/atmel/atmel-sama5d2-isc.c +1
-0+regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);I am surprised you can read a register before
runtime_pmis intialized!Actually I can read any register after the moment of starting the clock on the hardware block.
Ah right then
Quoted 3 lines
Maybe the starting and stopping of the clock needs to be moved to the
runtime_pmcalls, but this is another story, not related to the media controller.
It's fine, but maybe worth recording with a todo ?
Thanks
j
Quoted 95 lines
I moved this line because I had to pass the version to the
isc_mc_initcallQuoted 2 lines
Thanks
jThanks for reviewing !
EugenQuoted 87 lines
Quoted 87 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c +11
-2+ + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev);@@ -562,7 +568,7 @@ static int atmel_isc_probe(struct platform_device *pdev) ret =clk_prepare_enable (isc->ispck);if (ret) {dev_err (dev,"failed to enable ispck: %d\n" , ret); - goto cleanup_subdev; + goto isc_probe_mc_init_err; }/* ispck should be greater or equal to hclock */ @@ -572,7 +578,6 @@ static int atmel_isc_probe(struct platform_device *pdev) goto unprepare_clk; } -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip ISC version %x\n" , ver);return 0 ;@@ -580,6 +585,9 @@ static int atmel_isc_probe(struct platform_device *pdev) unprepare_clk:clk_disable_unprepare (isc->ispck); +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -600,6 +608,8 @@ static int atmel_isc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev);drivers/media/platform/atmel/atmel-sama7g5-isc.c +11
-2@@ -547,15 +547,23 @@ static int microchip_xisc_probe(struct platform_device *pdev)break ; } +regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver); + + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev); -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip XISC version %x\n" , ver);return 0 ; +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -576,6 +584,8 @@ static int microchip_xisc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev); --2.25.1
Quoted 156 lines · Jacopo Mondi
Hi Eugen
Quoted 153 lines · On Mon, Feb 07, 2022 at 01:40:31PM +0000, [email protected] wrote:
Quoted 138 lines · Jacopo Mondi
Hi Eugen
Quoted 135 lines · Eugen Hristev
Implement the support for media-controller.
This means that the capabilities of the driver have changed and now it also advertises theIO_MC.
The driver will register it's media device, and add the video entity to this media device. The subdevices are registered to the same media device. The ISC will have a base entity which is auto-detected asatmel_isc_base. It will also register a subdevice that allows cropping of the incoming frame to the maximum frame size supported by the ISC.
The ISC will create a link between the subdevice that is asynchronously registered and theatmel_isc_scalerentity.
Then, theatmel_isc_scalerandatmel_isc_baseare connected through another link.Signed-off-by Eugen Hristev <[email protected]>---
Changes in v4:
As suggested by Jacopo:
- renamedatmel_isc_mctoatmel_isc_scaler.c
- moved init_mc/clean_mc toisc_basefileChanges in v2:
- implement try formatsdrivers/media/platform/atmel/Makefile | 2 +- drivers/media/platform/atmel/atmel-isc-base.c | 73 +++++- .../media/platform/atmel/atmel-isc-scaler.c | 245 ++++++++++++++++++ drivers/media/platform/atmel/atmel-isc.h | 37 +++ .../media/platform/atmel/atmel-sama5d2-isc.c | 14 +- .../media/platform/atmel/atmel-sama7g5-isc.c | 12 +- 6 files changed, 375 insertions(+), 8 deletions(-)create mode 100644
drivers/media/platform/atmel/atmel-isc-scaler.cdrivers/media/platform/atmel/Makefile +1
-1@@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only atmel-isc-objs = atmel-sama5d2-isc.o atmel-xisc-objs = atmel-sama7g5-isc.o -atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o +atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o atmel-isc-scaler.o obj-$(CONFIG_VIDEO_ATMEL_ISI) += atmel-isi.o obj-$(CONFIG_VIDEO_ATMEL_ISC_BASE) += atmel-isc-common.odrivers/media/platform/atmel/atmel-isc-base.c +27
-4@@ -1710,6 +1710,7 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, struct isc_device, v4l2_dev);struct isc_subdev_entity *subdev_entity = container_of(notifier, struct isc_subdev_entity, notifier); +int pad;if (video_is_registered (&isc->video_dev)) {v4l2_err (&isc->v4l2_dev,"only supports one sub-device.\n" );@@ -1718,6 +1719,16 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, subdev_entity->sd = subdev; + pad =media_entity_get_fwnode_pad (&subdev->entity, asd->match.fwnode, +MEDIA_PAD_FL_SOURCE ); +if (pad <0 ) { +v4l2_err (&isc->v4l2_dev,"failed to find pad for %s\n" , + subdev->name); +return pad; + } + + isc->remote_pad = pad; +return 0 ; }@@ -1732,8 +1743,8 @@ static void isc_async_unbind(struct v4l2_async_notifier *notifier,v4l2_ctrl_handler_free (&isc->ctrls.handler); } -static struct isc_format *find_format_by_code (struct isc_device *isc, -unsigned int code,int *index) +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index) {struct isc_format *fmt = &isc->formats_list[0 ];unsigned int i;@@ -1749,6 +1760,7 @@ static struct isc_format *find_format_by_code(struct isc_device *isc,return NULL ; } +EXPORT_SYMBOL_GPL (isc_find_format_by_code);static int isc_formats_init (struct isc_device *isc) {@@ -1765,7 +1777,7 @@ static int isc_formats_init(struct isc_device *isc) NULL, &mbus_code)) { mbus_code.index++; - fmt =find_format_by_code (isc, mbus_code.code, &i); + fmt =isc_find_format_by_code (isc, mbus_code.code, &i);if (!fmt) {v4l2_warn (&isc->v4l2_dev,"Mbus code %x not supported\n" , mbus_code.code);@@ -1891,7 +1903,8 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) vdev->queue = q; vdev->lock = &isc->lock; vdev->ctrl_handler = &isc->ctrls.handler; - vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE ; + vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE | +V4L2_CAP_IO_MC ;video_set_drvdata (vdev, isc); ret =video_register_device (vdev,VFL_TYPE_VIDEO ,-1 );@@ -1901,8 +1914,18 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) goto isc_async_complete_err; } + ret =isc_scaler_link (isc); +if (ret <0 ) + goto isc_async_complete_unregister_device; + + ret =media_device_register (&isc->mdev); +if (ret <0 ) + goto isc_async_complete_unregister_device;return 0 ; +isc_async_complete_unregister_device: +video_unregister_device (vdev); + isc_async_complete_err:mutex_destroy (&isc->lock);return ret;@@ -1969,6 +1992,48 @@ int isc_pipeline_init(struct isc_device *isc) }EXPORT_SYMBOL_GPL (isc_pipeline_init);Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-base.c int isc_pipeline_init(struct isc_device *isc) · line 2001
Quoted 7 lines
drivers/media/platform/atmel/atmel-isc-base.c +7
-0+int isc_mc_init (struct isc_device *isc,u32 ver) +{ +const struct of_device_id *match; +int ret; + + isc->video_dev.entity.function =MEDIA_ENT_F_IO_V4L ; + isc->video_dev.entity.flags =MEDIA_ENT_FL_DEFAULT ;Should you set entity.ops.link_validate =
v4l2_subdev_link_validateto be able to have the media pipeline validated atmedia_pipeline_start()time ?Hi,
Quoted 7 lines
Quoted 5 lines
I am doing that in a subsequent patch. Things are not completely ready at this moment, because ISC still relies on the old mechanism to call
v4l2_subdev_callwithset_fmton the subdevice (which subsequent patch removes and adds checks to the link validate ).I see.. the subsequent patches are not part of this series, right ?
No. Patch 7 does this. In that patch, the link validation is created and all the logic of format propagation is removed and reworked, and creates the need for the link_validate call
Could you have also a look at that patch ?
Quoted 8 lines
I think patches 1, 2, 4, 5 and 6 from this series do not depend on the introduction of media controller or the scaler, right ? (I'm not sure if the DTS patches do).
If that's the case, would it make sense to to fast-track them (as some of them are fixes) and then on top plumb the media controller infrastructure with this patch and the ones you have been mentioning ?
It would make sense.
Quoted 118 lines
Quoted 118 lines
Quoted 78 lines
Quoted 46 lines
drivers/media/platform/atmel/atmel-isc-base.c +35
-0+ isc->pads[ISC_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + + ret =media_entity_pads_init (&isc->video_dev.entity,ISC_PADS_NUM , + isc->pads); +if (ret <0 ) { +dev_err (isc->dev,"media entity init failed\n" ); +return ret; + } + + isc->mdev.dev = isc->dev; + + match =of_match_node (isc->dev->driver->of_match_table, + isc->dev->of_node); + +strscpy (isc->mdev.driver_name,KBUILD_MODNAME , +sizeof (isc->mdev.driver_name)); +strscpy (isc->mdev.model, match->compatible,sizeof (isc->mdev.model)); +snprintf (isc->mdev.bus_info,sizeof (isc->mdev.bus_info),"platform:%s" , + isc->v4l2_dev.name); + isc->mdev.hw_revision = ver; + +media_device_init (&isc->mdev); + + isc->v4l2_dev.mdev = &isc->mdev; + +return isc_scaler_init (isc); +} +EXPORT_SYMBOL_GPL (isc_mc_init); + +void isc_mc_cleanup (struct isc_device *isc) +{ +media_entity_cleanup (&isc->video_dev.entity); +} +EXPORT_SYMBOL_GPL (isc_mc_cleanup); +/* regmap configuration */ #define ATMEL_ISC_REG_MAX 0xd5cconst struct regmap_config isc_regmap_config = {drivers/media/platform/atmel/atmel-isc-scaler.c new file +0
-0drivers/media/platform/atmel/atmel-isc-scaler.c line 5
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0@@ -0,0 +1,245 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Microchip Image Sensor Controller (ISC) Scaler entity support +* +* Copyright (C) 2021 Microchip Technology, Inc. Time flies! It's 2022 already :)
Quoted 24 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +24
-0+* +* Author: Eugen Hristev <[email protected]> +* +*/ + +#include <media/media-device.h> +#include <media/media-entity.h> +#include <media/v4l2-device.h> +#include <media/v4l2-subdev.h> + +#include "atmel-isc-regs.h" +#include "atmel-isc.h" + +static int isc_scaler_get_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *format) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; + +if (format->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + format->pad); + format->format = *v4l2_try_fmt;Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 34
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; + } + + format->format = isc->scaler_format;
isc->scaler_formatis only used inside this file if I'm not mistaken. I wonder why it lives in theisc_devicestruct.
isc_deviceis a placeholder for all isc things.I would not create a separate struct in this file that would have to be allocated etc... just for one two things. So I preferred to have it in the same place as all the other things.
Quoted 24 lines
Quoted 12 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +11
-0+ +return 0 ; +} + +static int isc_scaler_set_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *req_fmt) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; +struct isc_format *fmt;drivers/media/platform/atmel/atmel-isc-scaler.c line 55
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+unsigned int i; + +if (req_fmt->pad ==ISC_SCALER_PAD_SOURCE ) +v4l_bound_align_image + (&req_fmt->format.width,16 , isc->max_width,0 , + &req_fmt->format.height,16 , isc->max_height,0 ,0 ); +else +v4l_bound_align_image + (&req_fmt->format.width,16 ,10000 ,0 , + &req_fmt->format.height,16 ,10000 ,0 ,0 );Where does 10000 come from ?
Quoted 3 lines
Quoted 1 line
It's a random number. Do you have any suggestion for a better one ?
An actual HW limit ? :)
There is no hardware limit. The ISC just stops sampling pixels once the crop limit is reached. The element that generates pixels could go on forever , or until the next HBLANK/VBLANK. So what limit could I place here ?
The cropping is mandatory, otherwise there could be an overflow w.r.t. the buffer size if the ISC would sample more pixels than the software expects it to.
Quoted 9 lines
Quoted 7 lines
Maybe this would be much more clear with my comments below (where I will explain what the isc scaler does )
In short, it allows the other entity in the link to 'sink' a huge format into this 'scaler' . Because the scaler can crop anything basically down to the biggest format size the ISC could handle.doesn't it have any documented input size limit ?
As stated above, ISC handles a pixel stream and stores it in an internal SRAM. ISC stops sampling when this SRAM is full or when the cropping limit is reached.
After storing the pixels in the SRAM (which is limited to the ISC maximum frame size), the ISC will work on the pixels and then DMA the frame to another RAM.
Quoted 70 lines
Quoted 14 lines
Quoted 13 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 63
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +8
-0+ + req_fmt->format.colorspace =V4L2_COLORSPACE_SRGB ; + req_fmt->format.field =V4L2_FIELD_NONE ; + req_fmt->format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + req_fmt->format.quantization =V4L2_QUANTIZATION_DEFAULT ; + req_fmt->format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + fmt =isc_find_format_by_code (isc, req_fmt->format.code, &i);So you rely on the isc format list for the scaler as well ? I think it's fine as long as they are identical
Quoted 3 lines
Yes, the scaler is kind of a dummy entity , but it's required by the media controller validation of the pipeline.
Quoted 44 lines
Quoted 35 lines
Quoted 18 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +17
-0+ +if (!fmt) + fmt = &isc->formats_list[0 ]; + + req_fmt->format.code = fmt->mbus_code; + +if (req_fmt->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + req_fmt->pad); + *v4l2_try_fmt = req_fmt->format; +/* Trying on the pad sink makes the source sink change too */ +if (req_fmt->pad ==ISC_SCALER_PAD_SINK ) { + v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state, +ISC_SCALER_PAD_SOURCE ); + *v4l2_try_fmt = req_fmt->format; +drivers/media/platform/atmel/atmel-isc-scaler.c line 90
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+v4l_bound_align_image (&v4l2_try_fmt->width, +16 , isc->max_width,0 , + &v4l2_try_fmt->height, +16 , isc->max_height,0 ,0 ); + } +/* if we are just trying, we are done */ +return 0 ; + } + + isc->scaler_format = req_fmt->format;No per-pad format but a global
scaler_format? How do you configure scaling ?Actually, I would like to know more about the scaler device capabilities. What functions can this IP perform ? Does it do cropping, can it also do (down)scaling or even composition ?
Quoted 8 lines
I think it is worth to read
https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-subdev.html#v4l2-subdev-selections
where it is reported how cropping and scaling are implemented by using the selection API.Figure 4.5 is particularly helpful to explain the simple crop case, for which you need to implement support to by adding
s_selectionon the sink padTGT_CROPtarget.Quoted 5 lines
The scaler is kind of a dummy entity.
The problem is that different subdevices cannot be connected directly to a v4l entity, because the ISC does cropping on the incoming frame. The ISC has a limit on the total number of pixels per frame, thus it will do a crop.
Quoted 6 lines
What is this limit related to ? I don't know the ISC internals, but is the limitation due to the overall image size, or maybe it's due to some line buffers being limited in the size they can transfer to memory ? Is it a limitation on the size of the image in pixels, or is it due to a limitation of the processing bandwidth on the input interface and thus depends on the actual size of the image in bytes ?
Limitation is both on line size and full image size. From my understanding, is that the internal SRAM is limited (8 Mpixels with a small safe buffer on top in the case of sama7g5 )
This SRAM could be used as 800000x10 for example, or 3200x2464 for another example. However, I am not sure if a real line of 800,000 pixels make sense, or if there is another internal mechanism in the ISC that stops it.
In any case, the ISC is cropping the incoming frame from aaaXbbb up to the maximum frame that it can handle
Quoted 10 lines
Quoted 6 lines
So, if I allow e.g. an entity that sends 3280x2464 , when the ISC can only handle 3264x2464 , then we have two choices:
1/ crop it and output the 3264x2464 -> cannot, because userspace expects 3280x2464, and the whole pipeline fails to configure, there is a mismatch between/dev/videooutput and what the whole pipeline producesI'm confused in first place. If the ISC cannot output anything larger than 3264x2464, then if userspace sets a 3280x2464 size on the ISC, this should be adjusted to 3264x2464.
yes, that is true. However, without an entity that says 'I am doing cropping', the media pipeline will fail, because previous entity will output 3280x2464 .
The conclusion to create a scaler entitity was done during my last year discussions on the IRC. I could not find a way to perform this cropping at the v4l entity side, and one reason is that the v4l entity does not support what I needed , or at least that was my understanding. You claim that the v4l entity could perform the cropping in the same entity ?
Let's assume for a minute that it could do this. Even so, I would prefer to have a separate scaler entity to do this , for several reasons:
-> I would like to be able to crop the frame arbitrarily if I want to, and the scaler would allow me to do this easier
-> it would be more obvious that the frame is cropped by having this entity there
-> in fact some of the ISC versions really have a down scaler, that I have not implemented yet. So it would be useful to already have this in place to be able to scale down with it at a later time.
Quoted 8 lines
Or is the ISC limitation on the -input- side ?
Quoted 3 lines
2/ deny the link, and ask the subdevice to produce the 3264x2464 which the ISC can handle at most -> cannot , because the sensor let's say can *only* produce 3280x2464 and *any* other frame size returns an error.
buggy sensor I would say :)
I cannot tell this to the customers, and I cannot fail the whole pipeline because the sensor does not crop 16 pixels, when in fact I can do this very easily on the ISC side.
Quoted 11 lines
Quoted 7 lines
The only solution to make both worlds happy, is to have a dummy entity called 'scaler' which in fact now it only performs a simple crop. It accepts any frame size at input (hence the 10000x10000 which you saw earlier), and outputs at most 3264x2464 the
isc_max_heightandisc_max_width.I have a maybe dumb question: can the cropping operation be modeled by applying a
TGT_CROPrectangle (whose_BOUNDsize is 3280x2464) to theatmel_isc_commonentity sink pad and avoid the scaler completely ?
I am not sure. This is what I wanted initially. But then I thought about the three reasons stated above for the use of the scaler entity, and discussed with folks on the IRC, and come up with the solution for the scaler entity
Quoted 68 lines
Quoted 60 lines
So to answer your question, the isc scaler is a software model for a simple cropping, that would make media controller happy, and capture software happy.
Here it how it looks :
Device topology
- entity 1:atmel_isc_scaler(2 pads, 2 links)type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev0 pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb crop.bounds:(0,0)/3264x2464 crop:(0,0)/3264x2464] <- "csi2dc":1 [ENABLED,IMMUTABLE] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] -> "atmel_isc_common":0 [ENABLED,IMMUTABLE]- entity 4: csi2dc (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev1 pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] <- "dw-csi.0":1 [ENABLED] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] -> "atmel_isc_scaler":0 [ENABLED,IMMUTABLE]- entity 7: dw-csi.0 (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev2 pad0: Sink [fmt:SRGGB10_1X10/3280x2464] <- "imx219 1-0010":0 [ENABLED] pad1: Source [fmt:SRGGB10_1X10/3280x2464] -> "csi2dc":0 [ENABLED]- entity 12: imx219 1-0010 (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0 device node name/dev/v4l-subdev3
pad0: Source
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb
xfer:srgb ycbcr:601 quantization:full-rangecrop.bounds:(8,8)/3280x2464 crop:(8,8)/3280x2464] -> "dw-csi.0":0 [ENABLED]- entity 24:
atmel_isc_common(1 pad, 1 link)
type Node subtype V4L flags 1
device node name/dev/video0
pad0: Sink
<- "atmel_isc_scaler":1 [ENABLED,IMMUTABLE]Scaler does this one cute little thing :
Quoted 7 lines
pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb crop.bounds:(0,0)/3264x2464 crop:(0,0)/3264x2464] <- "csi2dc":1 [ENABLED,IMMUTABLE] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb]
Quoted 3 lines
Shouldn't this be 3264x2464 as that's what the entity outputs after the cropping ? And shouldn't then be 3264x2464 the size output from the video device too ?
That's right.
I don't know why the source format for the scaler is still 3280x2464. Maybe there is a bug on g_fmt for it.. have to check it.
Anyway, the video format is like this :
# v4l2-ctl --get-fmt-video
Format Video Capture:
Width/Height : 3264/2464
Pixel Format : 'RGBP' (16-bit RGB 5-6-5)
Field : None
Bytes per Line : 6528
Size Image : 16084992
Colorspace : sRGB
Transfer Function : Default (maps to sRGB)
YCbCr/HSV Encoding: Default (maps to ITU-R 601)
Quantization : Default (maps to Full Range)
Flags :
#and initializing the pipeline looks fine from user perspective. The scaler is in fact the solution to make this pipeline work with libcamera.
I found this cropping to be an issue in media controller when trying it with libcamera. Otherwise, the other user space apps which I was using never complained that anything was wrong
Libcamera simply refuses to acknowledge the pipeline if the video output is 3264x2464 but there is no entity that changes the format from 3280x2464 down
Quoted 71 lines
Quoted 61 lines
-> "atmel_isc_common":0 [ENABLED,IMMUTABLE]Which is what we needed.
Quoted 19 lines
Quoted 6 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; +} + +static int isc_scaler_enum_mbus_code(struct v4l2_subdev *sd,drivers/media/platform/atmel/atmel-isc-scaler.c line 105
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+struct v4l2_subdev_state *sd_state, + struct v4l2_subdev_mbus_code_enum *code) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +int supported_index =0 ; +int i; + +for (i =0 ; i < isc->formats_list_size; i++) { +if (!isc->formats_list[i].sd_support) +continue ;The
sd_supportflag still doesn't click in my head.Quoted 2 lines
Shouldn't the enumeration of available formats on the scaler do not depend on the sensor supproted formats ?
You're right. I will have to check it again.
Quoted 30 lines
Quoted 16 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +15
-0+if (supported_index == code->index) { + code->code = isc->formats_list[i].mbus_code; +return 0 ; + } + supported_index++; + } + +return -EINVAL ; +} + +static int isc_scaler_g_sel (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_selection *sel) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd);drivers/media/platform/atmel/atmel-isc-scaler.c line 130
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +if (sel->pad ==ISC_SCALER_PAD_SOURCE ) +return -EINVAL ; + +if (sel->target !=V4L2_SEL_TGT_CROP_BOUNDS && + sel->target !=V4L2_SEL_TGT_CROP ) +return -EINVAL ; + + sel->r.height = isc->max_height; + sel->r.width = isc->max_width;The
CROP_BOUNDSshould be set to the same size as the sink pad image format, as it represents the maximum valid crop rectangle.Quoted 9 lines
Quoted 3 lines
TGT_CROPshould report the configured crop rectangle which can be intiialized to the same size asCROP_BOUNDS, if my understanding of the spec is correctSo you would like to have this differentiated, and report the
CROP_BOUNDSto whatever is on the sink pad, and theTGT_CROPto what is reported now, the maximum size of the ISC frame .
My understanding is correct ?
Quoted 6 lines
I didn't know you have an HW limitation, so your
_BOUNDSis not the input image size but rather 3264x2464 ( ==max_widthxmax_height).What I meant is that
_BOUNDSshould report the maximum rectangle size that can be applied to the_CROPtarget. In you case you have an HW limitation 3264x2464 and that's the largest rectangle you can apply.
So the CROP should be at 3264x2464
Quoted 2 lines
TGT_CROPcan be initialized to the same as_BOUND, but if you implements_selectionit should report what has been there applied.
and BOUND to actual frame size ?
Quoted 2 lines
But as you don't implement
s_selectionyet, I think this is fine for now. Maybe a little comment ?
It could also be the case where e.g. the sensor is outputting 1920x1080, in this case the scaler would do nothing.
CROP is still 3264x2464 and BOUND in this case is 1920x1080 ? If the sensor is outputting 1920x1080, this format comes directly from the sensor (the sensor is cropping it from 3280x2464 or not... it's the sensor's problem)
Quoted 3 lines
Also, is
set->rzeroed by the framework before getting here ? Otherwise you should set r.left and r.top to 0
If these are redundant, no problem to remove them
Quoted 260 lines
Quoted 54 lines
Quoted 45 lines
Quoted 17 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +15
-0+ + sel->r.left =0 ; + sel->r.top =0 ; + +return 0 ; +} + +static int isc_scaler_init_cfg (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state) +{ +struct v4l2_mbus_framefmt *v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state,0 ); + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); + + *v4l2_try_fmt = isc->scaler_format;drivers/media/platform/atmel/atmel-isc-scaler.c line 155
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +return 0 ; +} + +static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = { + .enum_mbus_code = isc_scaler_enum_mbus_code, + .set_fmt = isc_scaler_set_fmt, + .get_fmt = isc_scaler_get_fmt, + .get_selection = isc_scaler_g_sel, + .init_cfg = isc_scaler_init_cfg,.link_validate =
v4l2_subdev_link_validate_default,To have the formats at the end of links that point to this entity validated (I think the framework already calls it if not set though, please check v4l2-subdev.c:v4l2_subdev_link_validate())
Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +11
-0+}; + +static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = { + .pad = &isc_scaler_pad_ops, +}; + +int isc_scaler_init (struct isc_device *isc) +{ +int ret; + +v4l2_subdev_init (&isc->scaler_sd, &xisc_scaler_subdev_ops);Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 171
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ + isc->scaler_sd.owner =THIS_MODULE ; + isc->scaler_sd.dev = isc->dev; +snprintf (isc->scaler_sd.name,sizeof (isc->scaler_sd.name), +"atmel_isc_scaler" );I would drop 'atmel' for brevity, unless other entities have this prefix set already
Quoted 3 lines
The v4l entity takes it's name from the module name, which is
atmel_isc_common, so I thought to keep the prefixAck!
Quoted 189 lines
Quoted 70 lines
Quoted 70 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +69
-0+ + isc->scaler_sd.flags |=V4L2_SUBDEV_FL_HAS_DEVNODE ; + isc->scaler_sd.entity.function =MEDIA_ENT_F_PROC_VIDEO_SCALER ; + isc->scaler_pads[ISC_SCALER_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + isc->scaler_pads[ISC_SCALER_PAD_SOURCE ].flags =MEDIA_PAD_FL_SOURCE ; + + isc->scaler_format.height = isc->max_height; + isc->scaler_format.width = isc->max_width; + isc->scaler_format.code = isc->formats_list[0 ].mbus_code; + isc->scaler_format.colorspace =V4L2_COLORSPACE_SRGB ; + isc->scaler_format.field =V4L2_FIELD_NONE ; + isc->scaler_format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + isc->scaler_format.quantization =V4L2_QUANTIZATION_DEFAULT ; + isc->scaler_format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + ret =media_entity_pads_init (&isc->scaler_sd.entity, +ISC_SCALER_PADS_NUM , + isc->scaler_pads); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd media entity init failed\n" ); +return ret; + } + ret =v4l2_device_register_subdev (&isc->v4l2_dev, &isc->scaler_sd); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd failed to register subdev\n" ); +return ret; + } + +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_init); + +int isc_scaler_link (struct isc_device *isc) +{ +int ret; + + ret =media_create_pad_link (&isc->current_subdev->sd->entity, + isc->remote_pad, &isc->scaler_sd.entity, +ISC_SCALER_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->current_subdev->sd->entity.name, + isc->scaler_sd.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , + isc->current_subdev->sd->name, isc->remote_pad); + + ret =media_create_pad_link (&isc->scaler_sd.entity, +ISC_SCALER_PAD_SOURCE , + &isc->video_dev.entity,ISC_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->scaler_sd.entity.name, + isc->video_dev.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , isc->scaler_sd.name, +ISC_SCALER_PAD_SOURCE );Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 244
Quoted 4 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +4
-0+ +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_link);From the DT graph point of view, the ISC appears as a single block
with an CSI-2 input port. It is then in charge of parsing the .dts and add to its notifier the image sensor remote subdevice.Actually it's only a parallel port.
And it can be connected either to a sensor directly or to the csi2dc bridge. (the csi2dc bridge can be connected to a CSI-2 receiver)Quoted 4 lines
When the sensor subdev registers and the ISC notifier completes, the scaler entity which was initialized at
isc_probe()time is linked in between the ISC and the image sensor immutably.yes, because this cannot change at runtime, and usually, it can't change without altering the board hardware. (at least this is what my understanding of immutability is )
Quoted 9 lines
I think it is fine for now, but I wonder if you plan to plumb more components between the ISC video node and the sensor, if it's not worth changing the DT bindings and their parsing logic to separate the CSI-2 receiver from the ISC, whcih can create its media graph at probe time and have the CSI-2 receiver entity as downstream remote. I think I need to get to know the ISC better to really have an idea. For now, this seems ok to me, but please check with maintainers if this is fine with them.
In the XISC case (sama7g5-isc), the csi2dc receiver is the subdev (so, the remote subdev).
The ISC will register a scaler, and connect the subdev to this scaler first, and then, the scaler to the isc itself (the v4l entity).Quoted 45 lines
Quoted 45 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +1
-0+drivers/media/platform/atmel/atmel-isc.h +21
-0@@ -183,6 +183,17 @@ struct isc_reg_offsets {u32 his_entry; }; +enum isc_mc_pads { +ISC_PAD_SINK =0 , +ISC_PADS_NUM =1 , +}; + +enum isc_scaler_pads { +ISC_SCALER_PAD_SINK =0 , +ISC_SCALER_PAD_SOURCE =1 , +ISC_SCALER_PADS_NUM =2 , +}; +/* * struct isc_device - ISC device driver data/config struct * @regmap: Register map @@ -257,6 +268,12 @@ struct isc_reg_offsets {* be used as an input to the controller * @controller_formats_size: size of controller_formats array * @formats_list_size: size of formats_list array +* @pads: media controller pads for isc video entity +* @mdev: media device that is registered by the isc +* @remote_pad: remote pad on the connected subdevice +* @scaler_sd: subdevice for the scaler that isc registers +* @scaler_pads: media controller pads for the scaler subdevice +* @scaler_format: current format for the scaler subdevice */ struct isc_device {struct regmap *regmap;@@ -344,6 +361,19 @@ struct isc_device {struct isc_format *formats_list;u32 controller_formats_size;u32 formats_list_size; + +struct { +struct media_pad pads[ISC_PADS_NUM ]; +struct media_device mdev;Quoted 8 lines
drivers/media/platform/atmel/atmel-isc.h struct isc_device { · line 375
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc.h +8
-0+ +u32 remote_pad; + }; + +struct { +struct v4l2_subdev scaler_sd; +struct media_pad scaler_pads[ISC_SCALER_PADS_NUM ]; +struct v4l2_mbus_framefmt scaler_format;Here are the scaler stuff which I added, in the same bulk struct for the whole isc device
Quoted 24 lines
Quoted 24 lines
drivers/media/platform/atmel/atmel-isc.h +8
-0+ }; };extern const struct regmap_config isc_regmap_config;@@ -355,4 +385,11 @@ int isc_clk_init(struct isc_device *isc);void isc_subdev_cleanup (struct isc_device *isc);void isc_clk_cleanup (struct isc_device *isc); +int isc_scaler_link (struct isc_device *isc); +int isc_scaler_init (struct isc_device *isc); +int isc_mc_init (struct isc_device *isc,u32 ver); +void isc_mc_cleanup (struct isc_device *isc); + +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index);#endif drivers/media/platform/atmel/atmel-sama5d2-isc.c +0
-0@@ -553,6 +553,12 @@ static int atmel_isc_probe(struct platform_device *pdev)break ; }Quoted 8 lines
Quoted 5 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c static int atmel_isc_probe(struct platform_device *pdev) · line 556
Quoted 1 line
drivers/media/platform/atmel/atmel-sama5d2-isc.c +1
-0+regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);I am surprised you can read a register before
runtime_pmis intialized!Actually I can read any register after the moment of starting the clock on the hardware block.
Ah right then
Quoted 104 lines
Quoted 3 lines
Maybe the starting and stopping of the clock needs to be moved to the
runtime_pmcalls, but this is another story, not related to the media controller.It's fine, but maybe worth recording with a todo ?
Thanks
jQuoted 95 lines
I moved this line because I had to pass the version to the
isc_mc_initcallQuoted 2 lines
Thanks
jThanks for reviewing !
EugenQuoted 87 lines
Quoted 87 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c +11
-2+ + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev);@@ -562,7 +568,7 @@ static int atmel_isc_probe(struct platform_device *pdev) ret =clk_prepare_enable (isc->ispck);if (ret) {dev_err (dev,"failed to enable ispck: %d\n" , ret); - goto cleanup_subdev; + goto isc_probe_mc_init_err; }/* ispck should be greater or equal to hclock */ @@ -572,7 +578,6 @@ static int atmel_isc_probe(struct platform_device *pdev) goto unprepare_clk; } -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip ISC version %x\n" , ver);return 0 ;@@ -580,6 +585,9 @@ static int atmel_isc_probe(struct platform_device *pdev) unprepare_clk:clk_disable_unprepare (isc->ispck); +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -600,6 +608,8 @@ static int atmel_isc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev);drivers/media/platform/atmel/atmel-sama7g5-isc.c +11
-2@@ -547,15 +547,23 @@ static int microchip_xisc_probe(struct platform_device *pdev)break ; } +regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver); + + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev); -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip XISC version %x\n" , ver);return 0 ; +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -576,6 +584,8 @@ static int microchip_xisc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev); --2.25.1
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Hi Eugen
Quoted 163 lines · On Tue, Feb 08, 2022 at 10:33:31AM +0000, [email protected] wrote:
Quoted 162 lines · Jacopo Mondi
Hi Eugen
Quoted 158 lines · On Mon, Feb 07, 2022 at 01:40:31PM +0000, [email protected] wrote:
Quoted 138 lines · Jacopo Mondi
Hi Eugen
Quoted 135 lines · Eugen Hristev
Implement the support for media-controller.
This means that the capabilities of the driver have changed and now it also advertises theIO_MC.
The driver will register it's media device, and add the video entity to this media device. The subdevices are registered to the same media device. The ISC will have a base entity which is auto-detected asatmel_isc_base. It will also register a subdevice that allows cropping of the incoming frame to the maximum frame size supported by the ISC.
The ISC will create a link between the subdevice that is asynchronously registered and theatmel_isc_scalerentity.
Then, theatmel_isc_scalerandatmel_isc_baseare connected through another link.Signed-off-by Eugen Hristev <[email protected]>---
Changes in v4:
As suggested by Jacopo:
- renamedatmel_isc_mctoatmel_isc_scaler.c
- moved init_mc/clean_mc toisc_basefileChanges in v2:
- implement try formatsdrivers/media/platform/atmel/Makefile | 2 +- drivers/media/platform/atmel/atmel-isc-base.c | 73 +++++- .../media/platform/atmel/atmel-isc-scaler.c | 245 ++++++++++++++++++ drivers/media/platform/atmel/atmel-isc.h | 37 +++ .../media/platform/atmel/atmel-sama5d2-isc.c | 14 +- .../media/platform/atmel/atmel-sama7g5-isc.c | 12 +- 6 files changed, 375 insertions(+), 8 deletions(-)create mode 100644
drivers/media/platform/atmel/atmel-isc-scaler.cdrivers/media/platform/atmel/Makefile +1
-1@@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only atmel-isc-objs = atmel-sama5d2-isc.o atmel-xisc-objs = atmel-sama7g5-isc.o -atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o +atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o atmel-isc-scaler.o obj-$(CONFIG_VIDEO_ATMEL_ISI) += atmel-isi.o obj-$(CONFIG_VIDEO_ATMEL_ISC_BASE) += atmel-isc-common.odrivers/media/platform/atmel/atmel-isc-base.c +27
-4@@ -1710,6 +1710,7 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, struct isc_device, v4l2_dev);struct isc_subdev_entity *subdev_entity = container_of(notifier, struct isc_subdev_entity, notifier); +int pad;if (video_is_registered (&isc->video_dev)) {v4l2_err (&isc->v4l2_dev,"only supports one sub-device.\n" );@@ -1718,6 +1719,16 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, subdev_entity->sd = subdev; + pad =media_entity_get_fwnode_pad (&subdev->entity, asd->match.fwnode, +MEDIA_PAD_FL_SOURCE ); +if (pad <0 ) { +v4l2_err (&isc->v4l2_dev,"failed to find pad for %s\n" , + subdev->name); +return pad; + } + + isc->remote_pad = pad; +return 0 ; }@@ -1732,8 +1743,8 @@ static void isc_async_unbind(struct v4l2_async_notifier *notifier,v4l2_ctrl_handler_free (&isc->ctrls.handler); } -static struct isc_format *find_format_by_code (struct isc_device *isc, -unsigned int code,int *index) +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index) {struct isc_format *fmt = &isc->formats_list[0 ];unsigned int i;@@ -1749,6 +1760,7 @@ static struct isc_format *find_format_by_code(struct isc_device *isc,return NULL ; } +EXPORT_SYMBOL_GPL (isc_find_format_by_code);static int isc_formats_init (struct isc_device *isc) {@@ -1765,7 +1777,7 @@ static int isc_formats_init(struct isc_device *isc) NULL, &mbus_code)) { mbus_code.index++; - fmt =find_format_by_code (isc, mbus_code.code, &i); + fmt =isc_find_format_by_code (isc, mbus_code.code, &i);if (!fmt) {v4l2_warn (&isc->v4l2_dev,"Mbus code %x not supported\n" , mbus_code.code);@@ -1891,7 +1903,8 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) vdev->queue = q; vdev->lock = &isc->lock; vdev->ctrl_handler = &isc->ctrls.handler; - vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE ; + vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE | +V4L2_CAP_IO_MC ;video_set_drvdata (vdev, isc); ret =video_register_device (vdev,VFL_TYPE_VIDEO ,-1 );@@ -1901,8 +1914,18 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) goto isc_async_complete_err; } + ret =isc_scaler_link (isc); +if (ret <0 ) + goto isc_async_complete_unregister_device; + + ret =media_device_register (&isc->mdev); +if (ret <0 ) + goto isc_async_complete_unregister_device;return 0 ; +isc_async_complete_unregister_device: +video_unregister_device (vdev); + isc_async_complete_err:mutex_destroy (&isc->lock);return ret;@@ -1969,6 +1992,48 @@ int isc_pipeline_init(struct isc_device *isc) }EXPORT_SYMBOL_GPL (isc_pipeline_init);Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-base.c int isc_pipeline_init(struct isc_device *isc) · line 2001
Quoted 7 lines
drivers/media/platform/atmel/atmel-isc-base.c +7
-0+int isc_mc_init (struct isc_device *isc,u32 ver) +{ +const struct of_device_id *match; +int ret; + + isc->video_dev.entity.function =MEDIA_ENT_F_IO_V4L ; + isc->video_dev.entity.flags =MEDIA_ENT_FL_DEFAULT ;Should you set entity.ops.link_validate =
v4l2_subdev_link_validateto be able to have the media pipeline validated atmedia_pipeline_start()time ?Hi,
I am doing that in a subsequent patch. Things are not completely ready at this moment, because ISC still relies on the old mechanism to call
v4l2_subdev_callwithset_fmton the subdevice (which subsequent patch removes and adds checks to the link validate ).
Quoted 6 lines
Quoted 1 line
I see.. the subsequent patches are not part of this series, right ?
No. Patch 7 does this. In that patch, the link validation is created and all the logic of format propagation is removed and reworked, and creates the need for the
link_validatecall
Could you have also a look at that patch ?
Ah ups, 7 was not in my inbox with the rest of the series. Just noticed. I had even reviewed the previous version, I should have remembered link_validate was added later
Quoted 4 lines
Quoted 4 lines
I think patches 1, 2, 4, 5 and 6 from this series do not depend on the introduction of media controller or the scaler, right ? (I'm not sure if the DTS patches do).
Do DTS patches depend on media-controller plumbing ?
Quoted 7 lines
Quoted 4 lines
If that's the case, would it make sense to to fast-track them (as some of them are fixes) and then on top plumb the media controller infrastructure with this patch and the ones you have been mentioning ?
It would make sense.
Once the other discussion with Hans about the stop state it might make sense to fastrack the first part of the series ?
Quoted 138 lines
Quoted 118 lines
Quoted 118 lines
Quoted 78 lines
Quoted 46 lines
drivers/media/platform/atmel/atmel-isc-base.c +35
-0+ isc->pads[ISC_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + + ret =media_entity_pads_init (&isc->video_dev.entity,ISC_PADS_NUM , + isc->pads); +if (ret <0 ) { +dev_err (isc->dev,"media entity init failed\n" ); +return ret; + } + + isc->mdev.dev = isc->dev; + + match =of_match_node (isc->dev->driver->of_match_table, + isc->dev->of_node); + +strscpy (isc->mdev.driver_name,KBUILD_MODNAME , +sizeof (isc->mdev.driver_name)); +strscpy (isc->mdev.model, match->compatible,sizeof (isc->mdev.model)); +snprintf (isc->mdev.bus_info,sizeof (isc->mdev.bus_info),"platform:%s" , + isc->v4l2_dev.name); + isc->mdev.hw_revision = ver; + +media_device_init (&isc->mdev); + + isc->v4l2_dev.mdev = &isc->mdev; + +return isc_scaler_init (isc); +} +EXPORT_SYMBOL_GPL (isc_mc_init); + +void isc_mc_cleanup (struct isc_device *isc) +{ +media_entity_cleanup (&isc->video_dev.entity); +} +EXPORT_SYMBOL_GPL (isc_mc_cleanup); +/* regmap configuration */ #define ATMEL_ISC_REG_MAX 0xd5cconst struct regmap_config isc_regmap_config = {drivers/media/platform/atmel/atmel-isc-scaler.c new file +0
-0drivers/media/platform/atmel/atmel-isc-scaler.c line 5
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0@@ -0,0 +1,245 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Microchip Image Sensor Controller (ISC) Scaler entity support +* +* Copyright (C) 2021 Microchip Technology, Inc. Time flies! It's 2022 already :)
Quoted 24 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +24
-0+* +* Author: Eugen Hristev <[email protected]> +* +*/ + +#include <media/media-device.h> +#include <media/media-entity.h> +#include <media/v4l2-device.h> +#include <media/v4l2-subdev.h> + +#include "atmel-isc-regs.h" +#include "atmel-isc.h" + +static int isc_scaler_get_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *format) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; + +if (format->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + format->pad); + format->format = *v4l2_try_fmt;Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 34
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; + } + + format->format = isc->scaler_format;
isc->scaler_formatis only used inside this file if I'm not mistaken. I wonder why it lives in theisc_devicestruct.
isc_deviceis a placeholder for all isc things.I would not create a separate struct in this file that would have to be allocated etc... just for one two things. So I preferred to have it in the same place as all the other things.
Quoted 24 lines
Quoted 12 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +11
-0+ +return 0 ; +} + +static int isc_scaler_set_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *req_fmt) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; +struct isc_format *fmt;drivers/media/platform/atmel/atmel-isc-scaler.c line 55
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+unsigned int i; + +if (req_fmt->pad ==ISC_SCALER_PAD_SOURCE ) +v4l_bound_align_image + (&req_fmt->format.width,16 , isc->max_width,0 , + &req_fmt->format.height,16 , isc->max_height,0 ,0 ); +else +v4l_bound_align_image + (&req_fmt->format.width,16 ,10000 ,0 , + &req_fmt->format.height,16 ,10000 ,0 ,0 );Where does 10000 come from ?
Quoted 3 lines
Quoted 1 line
It's a random number. Do you have any suggestion for a better one ?
An actual HW limit ? :)
There is no hardware limit. The ISC just stops sampling pixels once the crop limit is reached. The element that generates pixels could go on forever , or until the next HBLANK/VBLANK. So what limit could I place here ?
The cropping is mandatory, otherwise there could be an overflow w.r.t. the buffer size if the ISC would sample more pixels than the software expects it to.Quoted 8 lines
Quoted 7 lines
Maybe this would be much more clear with my comments below (where I will explain what the isc scaler does )
In short, it allows the other entity in the link to 'sink' a huge format into this 'scaler' . Because the scaler can crop anything basically down to the biggest format size the ISC could handle.
Quoted 9 lines
Quoted 1 line
doesn't it have any documented input size limit ?
As stated above, ISC handles a pixel stream and stores it in an internal SRAM. ISC stops sampling when this SRAM is full or when the cropping limit is reached.
After storing the pixels in the SRAM (which is limited to the ISC maximum frame size), the ISC will work on the pixels and then DMA the frame to another RAM.
I understand...
So whatever the input size is, only the first n-th bytes are actually dumped to memory, the following ones are discarded.
So the crop is always applied on the top-leftmost corner as a consequence, right ?
I understand why you had difficulties picking a default :)
This is not easy to handle, as the scaler doesn't actually applies a crop but rather stops capturing after the destination buffer has been saturated. So the limiting factor is the full image size, not one of its dimensions, like in example the line length. This makes it hard to adjust cropping to something meaningful for most use cases.
In your below example you suggest that an image size of (800000 x 10) would theoretically be captured correctly. What if the input is of size (800000 x 10 + 1). What size would you set the crop rectangle to ?
I wouldn't however be too much concerned for the moment, but I would record this in a comment ?
Quoted 98 lines
Quoted 70 lines
Quoted 14 lines
Quoted 13 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 63
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +8
-0+ + req_fmt->format.colorspace =V4L2_COLORSPACE_SRGB ; + req_fmt->format.field =V4L2_FIELD_NONE ; + req_fmt->format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + req_fmt->format.quantization =V4L2_QUANTIZATION_DEFAULT ; + req_fmt->format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + fmt =isc_find_format_by_code (isc, req_fmt->format.code, &i);So you rely on the isc format list for the scaler as well ? I think it's fine as long as they are identical
Quoted 3 lines
Yes, the scaler is kind of a dummy entity , but it's required by the media controller validation of the pipeline.
Quoted 44 lines
Quoted 35 lines
Quoted 18 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +17
-0+ +if (!fmt) + fmt = &isc->formats_list[0 ]; + + req_fmt->format.code = fmt->mbus_code; + +if (req_fmt->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + req_fmt->pad); + *v4l2_try_fmt = req_fmt->format; +/* Trying on the pad sink makes the source sink change too */ +if (req_fmt->pad ==ISC_SCALER_PAD_SINK ) { + v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state, +ISC_SCALER_PAD_SOURCE ); + *v4l2_try_fmt = req_fmt->format; +drivers/media/platform/atmel/atmel-isc-scaler.c line 90
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+v4l_bound_align_image (&v4l2_try_fmt->width, +16 , isc->max_width,0 , + &v4l2_try_fmt->height, +16 , isc->max_height,0 ,0 ); + } +/* if we are just trying, we are done */ +return 0 ; + } + + isc->scaler_format = req_fmt->format;No per-pad format but a global
scaler_format? How do you configure scaling ?Actually, I would like to know more about the scaler device capabilities. What functions can this IP perform ? Does it do cropping, can it also do (down)scaling or even composition ?
Quoted 8 lines
I think it is worth to read
https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-subdev.html#v4l2-subdev-selections
where it is reported how cropping and scaling are implemented by using the selection API.Figure 4.5 is particularly helpful to explain the simple crop case, for which you need to implement support to by adding
s_selectionon the sink padTGT_CROPtarget.Quoted 5 lines
The scaler is kind of a dummy entity.
The problem is that different subdevices cannot be connected directly to a v4l entity, because the ISC does cropping on the incoming frame. The ISC has a limit on the total number of pixels per frame, thus it will do a crop.Quoted 6 lines
What is this limit related to ? I don't know the ISC internals, but is the limitation due to the overall image size, or maybe it's due to some line buffers being limited in the size they can transfer to memory ? Is it a limitation on the size of the image in pixels, or is it due to a limitation of the processing bandwidth on the input interface and thus depends on the actual size of the image in bytes ?
Limitation is both on line size and full image size. From my understanding, is that the internal SRAM is limited (8 Mpixels with a small safe buffer on top in the case of sama7g5 )
This SRAM could be used as 800000x10 for example, or 3200x2464 for another example. However, I am not sure if a real line of 800,000 pixels make sense, or if there is another internal mechanism in the ISC that stops it.
In any case, the ISC is cropping the incoming frame from aaaXbbb up to the maximum frame that it can handleQuoted 10 lines
Quoted 6 lines
So, if I allow e.g. an entity that sends 3280x2464 , when the ISC can only handle 3264x2464 , then we have two choices:
1/ crop it and output the 3264x2464 -> cannot, because userspace expects 3280x2464, and the whole pipeline fails to configure, there is a mismatch between/dev/videooutput and what the whole pipeline producesI'm confused in first place. If the ISC cannot output anything larger than 3264x2464, then if userspace sets a 3280x2464 size on the ISC, this should be adjusted to 3264x2464.
Quoted 9 lines
yes, that is true. However, without an entity that says 'I am doing cropping', the media pipeline will fail, because previous entity will output 3280x2464 .
The conclusion to create a scaler entitity was done during my last year discussions on the IRC. I could not find a way to perform this cropping at the v4l entity side, and one reason is that the v4l entity does not support what I needed , or at least that was my understanding. You claim that the v4l entity could perform the cropping in the same entity ?
I thought so, but I haven't tried to be honest
Quoted 10 lines
Let's assume for a minute that it could do this. Even so, I would prefer to have a separate scaler entity to do this , for several reasons:
-> I would like to be able to crop the frame arbitrarily if I want to, and the scaler would allow me to do this easier
-> it would be more obvious that the frame is cropped by having this entity there
-> in fact some of the ISC versions really have a down scaler, that I have not implemented yet. So it would be useful to already have this in place to be able to scale down with it at a later time.
I think this last reason is enough to have an entity plumbed in already
Quoted 8 lines
Quoted 8 lines
Or is the ISC limitation on the -input- side ?
Quoted 3 lines
2/ deny the link, and ask the subdevice to produce the 3264x2464 which the ISC can handle at most -> cannot , because the sensor let's say can *only* produce 3280x2464 and *any* other frame size returns an error.
Quoted 6 lines
Quoted 1 line
buggy sensor I would say :)
I cannot tell this to the customers, and I cannot fail the whole pipeline because the sensor does not crop 16 pixels, when in fact I can do this very easily on the ISC side.
Fair enough :)
Quoted 85 lines
Quoted 11 lines
Quoted 7 lines
The only solution to make both worlds happy, is to have a dummy entity called 'scaler' which in fact now it only performs a simple crop. It accepts any frame size at input (hence the 10000x10000 which you saw earlier), and outputs at most 3264x2464 the
isc_max_heightandisc_max_width.I have a maybe dumb question: can the cropping operation be modeled by applying a
TGT_CROPrectangle (whose_BOUNDsize is 3280x2464) to theatmel_isc_commonentity sink pad and avoid the scaler completely ?I am not sure. This is what I wanted initially. But then I thought about the three reasons stated above for the use of the scaler entity, and discussed with folks on the IRC, and come up with the solution for the scaler entity
Quoted 68 lines
Quoted 67 lines
So to answer your question, the isc scaler is a software model for a simple cropping, that would make media controller happy, and capture software happy.
Here it how it looks :
Device topology
- entity 1:atmel_isc_scaler(2 pads, 2 links)type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev0 pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb crop.bounds:(0,0)/3264x2464 crop:(0,0)/3264x2464] <- "csi2dc":1 [ENABLED,IMMUTABLE] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] -> "atmel_isc_common":0 [ENABLED,IMMUTABLE]- entity 4: csi2dc (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev1 pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] <- "dw-csi.0":1 [ENABLED] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] -> "atmel_isc_scaler":0 [ENABLED,IMMUTABLE]- entity 7: dw-csi.0 (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev2 pad0: Sink [fmt:SRGGB10_1X10/3280x2464] <- "imx219 1-0010":0 [ENABLED] pad1: Source [fmt:SRGGB10_1X10/3280x2464] -> "csi2dc":0 [ENABLED]- entity 12: imx219 1-0010 (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0 device node name/dev/v4l-subdev3
pad0: Source
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb
xfer:srgb ycbcr:601 quantization:full-rangecrop.bounds:(8,8)/3280x2464 crop:(8,8)/3280x2464] -> "dw-csi.0":0 [ENABLED]- entity 24:
atmel_isc_common(1 pad, 1 link)
type Node subtype V4L flags 1
device node name/dev/video0
pad0: Sink
<- "atmel_isc_scaler":1 [ENABLED,IMMUTABLE]Scaler does this one cute little thing :
pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb crop.bounds:(0,0)/3264x2464 crop:(0,0)/3264x2464] <- "csi2dc":1 [ENABLED,IMMUTABLE] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb]
Quoted 7 lines
Quoted 3 lines
Shouldn't this be 3264x2464 as that's what the entity outputs after the cropping ? And shouldn't then be 3264x2464 the size output from the video device too ?
That's right.
I don't know why the source format for the scaler is still 3280x2464. Maybe there is a bug ong_fmtfor it.. have to check it.
Thanks, I think this should be fixed ten
Quoted 17 lines
Anyway, the video format is like this :
# v4l2-ctl --get-fmt-video Format Video Capture: Width/Height : 3264/2464 Pixel Format : 'RGBP' (16-bit RGB 5-6-5) Field : None Bytes per Line : 6528 Size Image : 16084992 Colorspace : sRGB Transfer Function : Default (maps to sRGB) YCbCr/HSV Encoding: Default (maps to ITU-R 601) Quantization : Default (maps to Full Range) Flags : #
Quoted 9 lines
and initializing the pipeline looks fine from user perspective. The scaler is in fact the solution to make this pipeline work with libcamera.
I found this cropping to be an issue in media controller when trying it with libcamera. Otherwise, the other user space apps which I was using never complained that anything was wrong
Libcamera simply refuses to acknowledge the pipeline if the video output is 3264x2464 but there is no entity that changes the format from 3280x2464 down
Not sure why libcamera should play a role there. Isn't it the media pipeline validation that complains and returns an -EPIPE ?
Quoted 92 lines
Quoted 72 lines
Quoted 62 lines
-> "atmel_isc_common":0 [ENABLED,IMMUTABLE]Which is what we needed.
Quoted 19 lines
Quoted 6 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; +} + +static int isc_scaler_enum_mbus_code(struct v4l2_subdev *sd,drivers/media/platform/atmel/atmel-isc-scaler.c line 105
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+struct v4l2_subdev_state *sd_state, + struct v4l2_subdev_mbus_code_enum *code) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +int supported_index =0 ; +int i; + +for (i =0 ; i < isc->formats_list_size; i++) { +if (!isc->formats_list[i].sd_support) +continue ;The
sd_supportflag still doesn't click in my head.Quoted 2 lines
Shouldn't the enumeration of available formats on the scaler do not depend on the sensor supproted formats ?
You're right. I will have to check it again.
Quoted 30 lines
Quoted 16 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +15
-0+if (supported_index == code->index) { + code->code = isc->formats_list[i].mbus_code; +return 0 ; + } + supported_index++; + } + +return -EINVAL ; +} + +static int isc_scaler_g_sel (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_selection *sel) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd);drivers/media/platform/atmel/atmel-isc-scaler.c line 130
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +if (sel->pad ==ISC_SCALER_PAD_SOURCE ) +return -EINVAL ; + +if (sel->target !=V4L2_SEL_TGT_CROP_BOUNDS && + sel->target !=V4L2_SEL_TGT_CROP ) +return -EINVAL ; + + sel->r.height = isc->max_height; + sel->r.width = isc->max_width;The
CROP_BOUNDSshould be set to the same size as the sink pad image format, as it represents the maximum valid crop rectangle.Quoted 9 lines
Quoted 3 lines
TGT_CROPshould report the configured crop rectangle which can be intiialized to the same size asCROP_BOUNDS, if my understanding of the spec is correctSo you would like to have this differentiated, and report the
CROP_BOUNDSto whatever is on the sink pad, and theTGT_CROPto what is reported now, the maximum size of the ISC frame .
My understanding is correct ?Quoted 6 lines
I didn't know you have an HW limitation, so your
_BOUNDSis not the input image size but rather 3264x2464 ( ==max_widthxmax_height).What I meant is that
_BOUNDSshould report the maximum rectangle size that can be applied to the_CROPtarget. In you case you have an HW limitation 3264x2464 and that's the largest rectangle you can apply.So the CROP should be at 3264x2464
Quoted 2 lines
TGT_CROPcan be initialized to the same as_BOUND, but if you implements_selectionit should report what has been there applied.and BOUND to actual frame size ?
Quoted 2 lines
But as you don't implement
s_selectionyet, I think this is fine for now. Maybe a little comment ?It could also be the case where e.g. the sensor is outputting 1920x1080, in this case the scaler would do nothing.
CROP is still 3264x2464 and BOUND in this case is 1920x1080 ? If the sensor is outputting 1920x1080, this format comes directly from the sensor (the sensor is cropping it from 3280x2464 or not... it's the sensor's problem)Quoted 1 line
Quoted 4 lines
Quoted 2 lines
Also, is
set->rzeroed by the framework before getting here ? Otherwise you should set r.left and r.top to 0If these are redundant, no problem to remove them
I was actually confused. I was suggesting you to add...
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 133
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +3
-0+ + sel->r.left =0 ; + sel->r.top =0 ;
These ^
So nothing to change. Sorry I've missed them
Thanks
j
Quoted 360 lines
Quoted 359 lines
Quoted 49 lines
Quoted 40 lines
Quoted 12 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +12
-0+ +return 0 ; +} + +static int isc_scaler_init_cfg (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state) +{ +struct v4l2_mbus_framefmt *v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state,0 ); + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); + + *v4l2_try_fmt = isc->scaler_format;drivers/media/platform/atmel/atmel-isc-scaler.c line 155
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +return 0 ; +} + +static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = { + .enum_mbus_code = isc_scaler_enum_mbus_code, + .set_fmt = isc_scaler_set_fmt, + .get_fmt = isc_scaler_get_fmt, + .get_selection = isc_scaler_g_sel, + .init_cfg = isc_scaler_init_cfg,.link_validate =
v4l2_subdev_link_validate_default,To have the formats at the end of links that point to this entity validated (I think the framework already calls it if not set though, please check v4l2-subdev.c:v4l2_subdev_link_validate())
Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +11
-0+}; + +static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = { + .pad = &isc_scaler_pad_ops, +}; + +int isc_scaler_init (struct isc_device *isc) +{ +int ret; + +v4l2_subdev_init (&isc->scaler_sd, &xisc_scaler_subdev_ops);Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 171
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ + isc->scaler_sd.owner =THIS_MODULE ; + isc->scaler_sd.dev = isc->dev; +snprintf (isc->scaler_sd.name,sizeof (isc->scaler_sd.name), +"atmel_isc_scaler" );I would drop 'atmel' for brevity, unless other entities have this prefix set already
Quoted 3 lines
The v4l entity takes it's name from the module name, which is
atmel_isc_common, so I thought to keep the prefixAck!
Quoted 189 lines
Quoted 70 lines
Quoted 70 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +69
-0+ + isc->scaler_sd.flags |=V4L2_SUBDEV_FL_HAS_DEVNODE ; + isc->scaler_sd.entity.function =MEDIA_ENT_F_PROC_VIDEO_SCALER ; + isc->scaler_pads[ISC_SCALER_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + isc->scaler_pads[ISC_SCALER_PAD_SOURCE ].flags =MEDIA_PAD_FL_SOURCE ; + + isc->scaler_format.height = isc->max_height; + isc->scaler_format.width = isc->max_width; + isc->scaler_format.code = isc->formats_list[0 ].mbus_code; + isc->scaler_format.colorspace =V4L2_COLORSPACE_SRGB ; + isc->scaler_format.field =V4L2_FIELD_NONE ; + isc->scaler_format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + isc->scaler_format.quantization =V4L2_QUANTIZATION_DEFAULT ; + isc->scaler_format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + ret =media_entity_pads_init (&isc->scaler_sd.entity, +ISC_SCALER_PADS_NUM , + isc->scaler_pads); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd media entity init failed\n" ); +return ret; + } + ret =v4l2_device_register_subdev (&isc->v4l2_dev, &isc->scaler_sd); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd failed to register subdev\n" ); +return ret; + } + +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_init); + +int isc_scaler_link (struct isc_device *isc) +{ +int ret; + + ret =media_create_pad_link (&isc->current_subdev->sd->entity, + isc->remote_pad, &isc->scaler_sd.entity, +ISC_SCALER_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->current_subdev->sd->entity.name, + isc->scaler_sd.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , + isc->current_subdev->sd->name, isc->remote_pad); + + ret =media_create_pad_link (&isc->scaler_sd.entity, +ISC_SCALER_PAD_SOURCE , + &isc->video_dev.entity,ISC_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->scaler_sd.entity.name, + isc->video_dev.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , isc->scaler_sd.name, +ISC_SCALER_PAD_SOURCE );Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 244
Quoted 4 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +4
-0+ +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_link);From the DT graph point of view, the ISC appears as a single block
with an CSI-2 input port. It is then in charge of parsing the .dts and add to its notifier the image sensor remote subdevice.Actually it's only a parallel port.
And it can be connected either to a sensor directly or to the csi2dc bridge. (the csi2dc bridge can be connected to a CSI-2 receiver)Quoted 4 lines
When the sensor subdev registers and the ISC notifier completes, the scaler entity which was initialized at
isc_probe()time is linked in between the ISC and the image sensor immutably.yes, because this cannot change at runtime, and usually, it can't change without altering the board hardware. (at least this is what my understanding of immutability is )
Quoted 9 lines
I think it is fine for now, but I wonder if you plan to plumb more components between the ISC video node and the sensor, if it's not worth changing the DT bindings and their parsing logic to separate the CSI-2 receiver from the ISC, whcih can create its media graph at probe time and have the CSI-2 receiver entity as downstream remote. I think I need to get to know the ISC better to really have an idea. For now, this seems ok to me, but please check with maintainers if this is fine with them.
In the XISC case (sama7g5-isc), the csi2dc receiver is the subdev (so, the remote subdev).
The ISC will register a scaler, and connect the subdev to this scaler first, and then, the scaler to the isc itself (the v4l entity).Quoted 45 lines
Quoted 45 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +1
-0+drivers/media/platform/atmel/atmel-isc.h +21
-0@@ -183,6 +183,17 @@ struct isc_reg_offsets {u32 his_entry; }; +enum isc_mc_pads { +ISC_PAD_SINK =0 , +ISC_PADS_NUM =1 , +}; + +enum isc_scaler_pads { +ISC_SCALER_PAD_SINK =0 , +ISC_SCALER_PAD_SOURCE =1 , +ISC_SCALER_PADS_NUM =2 , +}; +/* * struct isc_device - ISC device driver data/config struct * @regmap: Register map @@ -257,6 +268,12 @@ struct isc_reg_offsets {* be used as an input to the controller * @controller_formats_size: size of controller_formats array * @formats_list_size: size of formats_list array +* @pads: media controller pads for isc video entity +* @mdev: media device that is registered by the isc +* @remote_pad: remote pad on the connected subdevice +* @scaler_sd: subdevice for the scaler that isc registers +* @scaler_pads: media controller pads for the scaler subdevice +* @scaler_format: current format for the scaler subdevice */ struct isc_device {struct regmap *regmap;@@ -344,6 +361,19 @@ struct isc_device {struct isc_format *formats_list;u32 controller_formats_size;u32 formats_list_size; + +struct { +struct media_pad pads[ISC_PADS_NUM ]; +struct media_device mdev;Quoted 8 lines
drivers/media/platform/atmel/atmel-isc.h struct isc_device { · line 375
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc.h +8
-0+ +u32 remote_pad; + }; + +struct { +struct v4l2_subdev scaler_sd; +struct media_pad scaler_pads[ISC_SCALER_PADS_NUM ]; +struct v4l2_mbus_framefmt scaler_format;Here are the scaler stuff which I added, in the same bulk struct for the whole isc device
Quoted 24 lines
Quoted 24 lines
drivers/media/platform/atmel/atmel-isc.h +8
-0+ }; };extern const struct regmap_config isc_regmap_config;@@ -355,4 +385,11 @@ int isc_clk_init(struct isc_device *isc);void isc_subdev_cleanup (struct isc_device *isc);void isc_clk_cleanup (struct isc_device *isc); +int isc_scaler_link (struct isc_device *isc); +int isc_scaler_init (struct isc_device *isc); +int isc_mc_init (struct isc_device *isc,u32 ver); +void isc_mc_cleanup (struct isc_device *isc); + +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index);#endif drivers/media/platform/atmel/atmel-sama5d2-isc.c +0
-0@@ -553,6 +553,12 @@ static int atmel_isc_probe(struct platform_device *pdev)break ; }Quoted 8 lines
Quoted 5 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c static int atmel_isc_probe(struct platform_device *pdev) · line 556
Quoted 1 line
drivers/media/platform/atmel/atmel-sama5d2-isc.c +1
-0+regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);I am surprised you can read a register before
runtime_pmis intialized!Actually I can read any register after the moment of starting the clock on the hardware block.
Ah right then
Quoted 3 lines
Maybe the starting and stopping of the clock needs to be moved to the
runtime_pmcalls, but this is another story, not related to the media controller.It's fine, but maybe worth recording with a todo ?
Thanks
jQuoted 95 lines
I moved this line because I had to pass the version to the
isc_mc_initcallQuoted 2 lines
Thanks
jThanks for reviewing !
EugenQuoted 87 lines
Quoted 87 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c +11
-2+ + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev);@@ -562,7 +568,7 @@ static int atmel_isc_probe(struct platform_device *pdev) ret =clk_prepare_enable (isc->ispck);if (ret) {dev_err (dev,"failed to enable ispck: %d\n" , ret); - goto cleanup_subdev; + goto isc_probe_mc_init_err; }/* ispck should be greater or equal to hclock */ @@ -572,7 +578,6 @@ static int atmel_isc_probe(struct platform_device *pdev) goto unprepare_clk; } -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip ISC version %x\n" , ver);return 0 ;@@ -580,6 +585,9 @@ static int atmel_isc_probe(struct platform_device *pdev) unprepare_clk:clk_disable_unprepare (isc->ispck); +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -600,6 +608,8 @@ static int atmel_isc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev);drivers/media/platform/atmel/atmel-sama7g5-isc.c +11
-2@@ -547,15 +547,23 @@ static int microchip_xisc_probe(struct platform_device *pdev)break ; } +regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver); + + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev); -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip XISC version %x\n" , ver);return 0 ; +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -576,6 +584,8 @@ static int microchip_xisc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev); --2.25.1
Quoted 178 lines · Jacopo Mondi
Hi Eugen
Quoted 163 lines · On Tue, Feb 08, 2022 at 10:33:31AM +0000, [email protected] wrote:
Quoted 162 lines · Jacopo Mondi
Hi Eugen
Quoted 158 lines · On Mon, Feb 07, 2022 at 01:40:31PM +0000, [email protected] wrote:
Quoted 138 lines · Jacopo Mondi
Hi Eugen
Quoted 135 lines · Eugen Hristev
Implement the support for media-controller.
This means that the capabilities of the driver have changed and now it also advertises theIO_MC.
The driver will register it's media device, and add the video entity to this media device. The subdevices are registered to the same media device. The ISC will have a base entity which is auto-detected asatmel_isc_base. It will also register a subdevice that allows cropping of the incoming frame to the maximum frame size supported by the ISC.
The ISC will create a link between the subdevice that is asynchronously registered and theatmel_isc_scalerentity.
Then, theatmel_isc_scalerandatmel_isc_baseare connected through another link.Signed-off-by Eugen Hristev <[email protected]>---
Changes in v4:
As suggested by Jacopo:
- renamedatmel_isc_mctoatmel_isc_scaler.c
- moved init_mc/clean_mc toisc_basefileChanges in v2:
- implement try formatsdrivers/media/platform/atmel/Makefile | 2 +- drivers/media/platform/atmel/atmel-isc-base.c | 73 +++++- .../media/platform/atmel/atmel-isc-scaler.c | 245 ++++++++++++++++++ drivers/media/platform/atmel/atmel-isc.h | 37 +++ .../media/platform/atmel/atmel-sama5d2-isc.c | 14 +- .../media/platform/atmel/atmel-sama7g5-isc.c | 12 +- 6 files changed, 375 insertions(+), 8 deletions(-)create mode 100644
drivers/media/platform/atmel/atmel-isc-scaler.cdrivers/media/platform/atmel/Makefile +1
-1@@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only atmel-isc-objs = atmel-sama5d2-isc.o atmel-xisc-objs = atmel-sama7g5-isc.o -atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o +atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o atmel-isc-scaler.o obj-$(CONFIG_VIDEO_ATMEL_ISI) += atmel-isi.o obj-$(CONFIG_VIDEO_ATMEL_ISC_BASE) += atmel-isc-common.odrivers/media/platform/atmel/atmel-isc-base.c +27
-4@@ -1710,6 +1710,7 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, struct isc_device, v4l2_dev);struct isc_subdev_entity *subdev_entity = container_of(notifier, struct isc_subdev_entity, notifier); +int pad;if (video_is_registered (&isc->video_dev)) {v4l2_err (&isc->v4l2_dev,"only supports one sub-device.\n" );@@ -1718,6 +1719,16 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, subdev_entity->sd = subdev; + pad =media_entity_get_fwnode_pad (&subdev->entity, asd->match.fwnode, +MEDIA_PAD_FL_SOURCE ); +if (pad <0 ) { +v4l2_err (&isc->v4l2_dev,"failed to find pad for %s\n" , + subdev->name); +return pad; + } + + isc->remote_pad = pad; +return 0 ; }@@ -1732,8 +1743,8 @@ static void isc_async_unbind(struct v4l2_async_notifier *notifier,v4l2_ctrl_handler_free (&isc->ctrls.handler); } -static struct isc_format *find_format_by_code (struct isc_device *isc, -unsigned int code,int *index) +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index) {struct isc_format *fmt = &isc->formats_list[0 ];unsigned int i;@@ -1749,6 +1760,7 @@ static struct isc_format *find_format_by_code(struct isc_device *isc,return NULL ; } +EXPORT_SYMBOL_GPL (isc_find_format_by_code);static int isc_formats_init (struct isc_device *isc) {@@ -1765,7 +1777,7 @@ static int isc_formats_init(struct isc_device *isc) NULL, &mbus_code)) { mbus_code.index++; - fmt =find_format_by_code (isc, mbus_code.code, &i); + fmt =isc_find_format_by_code (isc, mbus_code.code, &i);if (!fmt) {v4l2_warn (&isc->v4l2_dev,"Mbus code %x not supported\n" , mbus_code.code);@@ -1891,7 +1903,8 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) vdev->queue = q; vdev->lock = &isc->lock; vdev->ctrl_handler = &isc->ctrls.handler; - vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE ; + vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE | +V4L2_CAP_IO_MC ;video_set_drvdata (vdev, isc); ret =video_register_device (vdev,VFL_TYPE_VIDEO ,-1 );@@ -1901,8 +1914,18 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) goto isc_async_complete_err; } + ret =isc_scaler_link (isc); +if (ret <0 ) + goto isc_async_complete_unregister_device; + + ret =media_device_register (&isc->mdev); +if (ret <0 ) + goto isc_async_complete_unregister_device;return 0 ; +isc_async_complete_unregister_device: +video_unregister_device (vdev); + isc_async_complete_err:mutex_destroy (&isc->lock);return ret;@@ -1969,6 +1992,48 @@ int isc_pipeline_init(struct isc_device *isc) }EXPORT_SYMBOL_GPL (isc_pipeline_init);Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-base.c int isc_pipeline_init(struct isc_device *isc) · line 2001
Quoted 7 lines
drivers/media/platform/atmel/atmel-isc-base.c +7
-0+int isc_mc_init (struct isc_device *isc,u32 ver) +{ +const struct of_device_id *match; +int ret; + + isc->video_dev.entity.function =MEDIA_ENT_F_IO_V4L ; + isc->video_dev.entity.flags =MEDIA_ENT_FL_DEFAULT ;Should you set entity.ops.link_validate =
v4l2_subdev_link_validateto be able to have the media pipeline validated atmedia_pipeline_start()time ?Hi,
I am doing that in a subsequent patch. Things are not completely ready at this moment, because ISC still relies on the old mechanism to call
v4l2_subdev_callwithset_fmton the subdevice (which subsequent patch removes and adds checks to the link validate ).Quoted 6 lines
Quoted 1 line
I see.. the subsequent patches are not part of this series, right ?
No. Patch 7 does this. In that patch, the link validation is created and all the logic of format propagation is removed and reworked, and creates the need for the
link_validatecall
Could you have also a look at that patch ?Ah ups, 7 was not in my inbox with the rest of the series. Just noticed. I had even reviewed the previous version, I should have remembered
link_validatewas added laterQuoted 1 line
Quoted 1 line
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
I think patches 1, 2, 4, 5 and 6 from this series do not depend on the introduction of media controller or the scaler, right ? (I'm not sure if the DTS patches do).
Do DTS patches depend on media-controller plumbing ?
I think not, but, I have not tested them without the media controller.
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
If that's the case, would it make sense to to fast-track them (as some of them are fixes) and then on top plumb the media controller infrastructure with this patch and the ones you have been mentioning ?
It would make sense.
Once the other discussion with Hans about the stop state it might make sense to fastrack the first part of the series ?
The other three patches (4,5,6) are quite independent and can go faster.
Quoted 149 lines
Quoted 139 lines
Quoted 119 lines
Quoted 119 lines
Quoted 79 lines
Quoted 47 lines
drivers/media/platform/atmel/atmel-isc-base.c +35
-0+ isc->pads[ISC_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + + ret =media_entity_pads_init (&isc->video_dev.entity,ISC_PADS_NUM , + isc->pads); +if (ret <0 ) { +dev_err (isc->dev,"media entity init failed\n" ); +return ret; + } + + isc->mdev.dev = isc->dev; + + match =of_match_node (isc->dev->driver->of_match_table, + isc->dev->of_node); + +strscpy (isc->mdev.driver_name,KBUILD_MODNAME , +sizeof (isc->mdev.driver_name)); +strscpy (isc->mdev.model, match->compatible,sizeof (isc->mdev.model)); +snprintf (isc->mdev.bus_info,sizeof (isc->mdev.bus_info),"platform:%s" , + isc->v4l2_dev.name); + isc->mdev.hw_revision = ver; + +media_device_init (&isc->mdev); + + isc->v4l2_dev.mdev = &isc->mdev; + +return isc_scaler_init (isc); +} +EXPORT_SYMBOL_GPL (isc_mc_init); + +void isc_mc_cleanup (struct isc_device *isc) +{ +media_entity_cleanup (&isc->video_dev.entity); +} +EXPORT_SYMBOL_GPL (isc_mc_cleanup); +/* regmap configuration */ #define ATMEL_ISC_REG_MAX 0xd5cconst struct regmap_config isc_regmap_config = {drivers/media/platform/atmel/atmel-isc-scaler.c new file +0
-0drivers/media/platform/atmel/atmel-isc-scaler.c line 5
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0@@ -0,0 +1,245 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Microchip Image Sensor Controller (ISC) Scaler entity support +* +* Copyright (C) 2021 Microchip Technology, Inc. Time flies! It's 2022 already :)
Quoted 24 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +24
-0+* +* Author: Eugen Hristev <[email protected]> +* +*/ + +#include <media/media-device.h> +#include <media/media-entity.h> +#include <media/v4l2-device.h> +#include <media/v4l2-subdev.h> + +#include "atmel-isc-regs.h" +#include "atmel-isc.h" + +static int isc_scaler_get_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *format) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; + +if (format->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + format->pad); + format->format = *v4l2_try_fmt;Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 34
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; + } + + format->format = isc->scaler_format;
isc->scaler_formatis only used inside this file if I'm not mistaken. I wonder why it lives in theisc_devicestruct.
isc_deviceis a placeholder for all isc things.I would not create a separate struct in this file that would have to be allocated etc... just for one two things. So I preferred to have it in the same place as all the other things.
Quoted 24 lines
Quoted 12 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +11
-0+ +return 0 ; +} + +static int isc_scaler_set_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *req_fmt) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; +struct isc_format *fmt;drivers/media/platform/atmel/atmel-isc-scaler.c line 55
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+unsigned int i; + +if (req_fmt->pad ==ISC_SCALER_PAD_SOURCE ) +v4l_bound_align_image + (&req_fmt->format.width,16 , isc->max_width,0 , + &req_fmt->format.height,16 , isc->max_height,0 ,0 ); +else +v4l_bound_align_image + (&req_fmt->format.width,16 ,10000 ,0 , + &req_fmt->format.height,16 ,10000 ,0 ,0 );Where does 10000 come from ?
Quoted 3 lines
Quoted 1 line
It's a random number. Do you have any suggestion for a better one ?
An actual HW limit ? :)
There is no hardware limit. The ISC just stops sampling pixels once the crop limit is reached. The element that generates pixels could go on forever , or until the next HBLANK/VBLANK. So what limit could I place here ?
The cropping is mandatory, otherwise there could be an overflow w.r.t. the buffer size if the ISC would sample more pixels than the software expects it to.Quoted 8 lines
Quoted 7 lines
Maybe this would be much more clear with my comments below (where I will explain what the isc scaler does )
In short, it allows the other entity in the link to 'sink' a huge format into this 'scaler' . Because the scaler can crop anything basically down to the biggest format size the ISC could handle.Quoted 9 lines
Quoted 1 line
doesn't it have any documented input size limit ?
As stated above, ISC handles a pixel stream and stores it in an internal SRAM. ISC stops sampling when this SRAM is full or when the cropping limit is reached.
After storing the pixels in the SRAM (which is limited to the ISC maximum frame size), the ISC will work on the pixels and then DMA the frame to another RAM.
Quoted 4 lines
I understand...
So whatever the input size is, only the first n-th bytes are actually dumped to memory, the following ones are discarded.
Yes ! That's the behavior.
Quoted 3 lines
So the crop is always applied on the top-leftmost corner as a consequence, right ?
Right. Pixels come in order. The ISC does not really know how many pixels are coming until its memory is full (or counting until the crop limit is reached). After that it stops sampling pixels, and waits for vblank to finish the frame.
Quoted 8 lines
I understand why you had difficulties picking a default :)
This is not easy to handle, as the scaler doesn't actually applies a crop but rather stops capturing after the destination buffer has been saturated. So the limiting factor is the full image size, not one of its dimensions, like in example the line length. This makes it hard to adjust cropping to something meaningful for most use cases.
I didn't know how to name this except crop. It 'crops out' what does not fit in the internal memory of the ISC
Quoted 7 lines
In your below example you suggest that an image size of (800000 x 10) would theoretically be captured correctly. What if the input is of size (800000 x 10 + 1). What size would you set the crop rectangle to ?
I wouldn't however be too much concerned for the moment, but I would record this in a comment ?
I discussed a bit more with designer and looked in datasheet. It looks like the horizontal line length is not a hard limit, but it makes the ISC internal ISP stop working, so I would keep the horizontal limit as well for now, unless an important use case (I am coming back to that). The vertical limit looks to be a hard requirement.
So in short, we have to impose a vertical limit of 2464 , we cannot have more lines than these. So if the horizontal is 100 pixels e.g., we can capture only 100x2464 .
In practice we could capture [16,3264] on the horizontal, and [16,2464] on the vertical, and in theory horizontal could go more like [16, 4000], but we have to limit the vertical, and without the internal ISP. So I would not haphazardly go in this direction. Let's stick to the datasheet maximum frame 3264x2464 .
According to this , I changed the frame size of the ISC to be a continuous size from [16,16] to [3264,2464]
About that use case that I talked about earlier, we have one sensor , imx274, which is more wide, and outputs something like 3800x2100 . With the cropping and limitations we have in place now, this will be limited to 3264x2100 , which is in line with datasheet and what the ISC supports now.
As in theory we could capture 3800x2100 as long as the SRAM doesn't get full, for now I would like to avoid capturing this size (and disable the internal ISP), until we have someone really asking for such kind of capture and with some good reasons.
In short I would like to have the crop rectangle to 3264x2464 and the bounds would be limited to the incoming frame but no H>3264 and no V>2464. Does that make sense ?
Quoted 265 lines
Quoted 99 lines
Quoted 71 lines
Quoted 15 lines
Quoted 14 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 63
Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +8
-0+ + req_fmt->format.colorspace =V4L2_COLORSPACE_SRGB ; + req_fmt->format.field =V4L2_FIELD_NONE ; + req_fmt->format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + req_fmt->format.quantization =V4L2_QUANTIZATION_DEFAULT ; + req_fmt->format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + fmt =isc_find_format_by_code (isc, req_fmt->format.code, &i);So you rely on the isc format list for the scaler as well ? I think it's fine as long as they are identical
Quoted 3 lines
Yes, the scaler is kind of a dummy entity , but it's required by the media controller validation of the pipeline.
Quoted 44 lines
Quoted 35 lines
Quoted 18 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +17
-0+ +if (!fmt) + fmt = &isc->formats_list[0 ]; + + req_fmt->format.code = fmt->mbus_code; + +if (req_fmt->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + req_fmt->pad); + *v4l2_try_fmt = req_fmt->format; +/* Trying on the pad sink makes the source sink change too */ +if (req_fmt->pad ==ISC_SCALER_PAD_SINK ) { + v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state, +ISC_SCALER_PAD_SOURCE ); + *v4l2_try_fmt = req_fmt->format; +drivers/media/platform/atmel/atmel-isc-scaler.c line 90
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+v4l_bound_align_image (&v4l2_try_fmt->width, +16 , isc->max_width,0 , + &v4l2_try_fmt->height, +16 , isc->max_height,0 ,0 ); + } +/* if we are just trying, we are done */ +return 0 ; + } + + isc->scaler_format = req_fmt->format;No per-pad format but a global
scaler_format? How do you configure scaling ?Actually, I would like to know more about the scaler device capabilities. What functions can this IP perform ? Does it do cropping, can it also do (down)scaling or even composition ?
Quoted 8 lines
I think it is worth to read
https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-subdev.html#v4l2-subdev-selections
where it is reported how cropping and scaling are implemented by using the selection API.Figure 4.5 is particularly helpful to explain the simple crop case, for which you need to implement support to by adding
s_selectionon the sink padTGT_CROPtarget.Quoted 5 lines
The scaler is kind of a dummy entity.
The problem is that different subdevices cannot be connected directly to a v4l entity, because the ISC does cropping on the incoming frame. The ISC has a limit on the total number of pixels per frame, thus it will do a crop.Quoted 6 lines
What is this limit related to ? I don't know the ISC internals, but is the limitation due to the overall image size, or maybe it's due to some line buffers being limited in the size they can transfer to memory ? Is it a limitation on the size of the image in pixels, or is it due to a limitation of the processing bandwidth on the input interface and thus depends on the actual size of the image in bytes ?
Limitation is both on line size and full image size. From my understanding, is that the internal SRAM is limited (8 Mpixels with a small safe buffer on top in the case of sama7g5 )
This SRAM could be used as 800000x10 for example, or 3200x2464 for another example. However, I am not sure if a real line of 800,000 pixels make sense, or if there is another internal mechanism in the ISC that stops it.
In any case, the ISC is cropping the incoming frame from aaaXbbb up to the maximum frame that it can handleQuoted 10 lines
Quoted 6 lines
So, if I allow e.g. an entity that sends 3280x2464 , when the ISC can only handle 3264x2464 , then we have two choices:
1/ crop it and output the 3264x2464 -> cannot, because userspace expects 3280x2464, and the whole pipeline fails to configure, there is a mismatch between/dev/videooutput and what the whole pipeline producesI'm confused in first place. If the ISC cannot output anything larger than 3264x2464, then if userspace sets a 3280x2464 size on the ISC, this should be adjusted to 3264x2464.
Quoted 9 lines
yes, that is true. However, without an entity that says 'I am doing cropping', the media pipeline will fail, because previous entity will output 3280x2464 .
The conclusion to create a scaler entitity was done during my last year discussions on the IRC. I could not find a way to perform this cropping at the v4l entity side, and one reason is that the v4l entity does not support what I needed , or at least that was my understanding. You claim that the v4l entity could perform the cropping in the same entity ?I thought so, but I haven't tried to be honest
Quoted 10 lines
Let's assume for a minute that it could do this. Even so, I would prefer to have a separate scaler entity to do this , for several reasons:
-> I would like to be able to crop the frame arbitrarily if I want to, and the scaler would allow me to do this easier
-> it would be more obvious that the frame is cropped by having this entity there
-> in fact some of the ISC versions really have a down scaler, that I have not implemented yet. So it would be useful to already have this in place to be able to scale down with it at a later time.I think this last reason is enough to have an entity plumbed in already
Quoted 8 lines
Quoted 8 lines
Or is the ISC limitation on the -input- side ?
Quoted 3 lines
2/ deny the link, and ask the subdevice to produce the 3264x2464 which the ISC can handle at most -> cannot , because the sensor let's say can *only* produce 3280x2464 and *any* other frame size returns an error.
Quoted 6 lines
Quoted 1 line
buggy sensor I would say :)
I cannot tell this to the customers, and I cannot fail the whole pipeline because the sensor does not crop 16 pixels, when in fact I can do this very easily on the ISC side.
Fair enough :)
Quoted 85 lines
Quoted 11 lines
Quoted 7 lines
The only solution to make both worlds happy, is to have a dummy entity called 'scaler' which in fact now it only performs a simple crop. It accepts any frame size at input (hence the 10000x10000 which you saw earlier), and outputs at most 3264x2464 the
isc_max_heightandisc_max_width.I have a maybe dumb question: can the cropping operation be modeled by applying a
TGT_CROPrectangle (whose_BOUNDsize is 3280x2464) to theatmel_isc_commonentity sink pad and avoid the scaler completely ?I am not sure. This is what I wanted initially. But then I thought about the three reasons stated above for the use of the scaler entity, and discussed with folks on the IRC, and come up with the solution for the scaler entity
Quoted 68 lines
Quoted 67 lines
So to answer your question, the isc scaler is a software model for a simple cropping, that would make media controller happy, and capture software happy.
Here it how it looks :
Device topology
- entity 1:atmel_isc_scaler(2 pads, 2 links)type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev0 pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb crop.bounds:(0,0)/3264x2464 crop:(0,0)/3264x2464] <- "csi2dc":1 [ENABLED,IMMUTABLE] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] -> "atmel_isc_common":0 [ENABLED,IMMUTABLE]- entity 4: csi2dc (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev1 pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] <- "dw-csi.0":1 [ENABLED] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] -> "atmel_isc_scaler":0 [ENABLED,IMMUTABLE]- entity 7: dw-csi.0 (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev2 pad0: Sink [fmt:SRGGB10_1X10/3280x2464] <- "imx219 1-0010":0 [ENABLED] pad1: Source [fmt:SRGGB10_1X10/3280x2464] -> "csi2dc":0 [ENABLED]- entity 12: imx219 1-0010 (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0 device node name/dev/v4l-subdev3
pad0: Source
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb
xfer:srgb ycbcr:601 quantization:full-rangecrop.bounds:(8,8)/3280x2464 crop:(8,8)/3280x2464] -> "dw-csi.0":0 [ENABLED]- entity 24:
atmel_isc_common(1 pad, 1 link)
type Node subtype V4L flags 1
device node name/dev/video0
pad0: Sink
<- "atmel_isc_scaler":1 [ENABLED,IMMUTABLE]Scaler does this one cute little thing :
pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb crop.bounds:(0,0)/3264x2464 crop:(0,0)/3264x2464] <- "csi2dc":1 [ENABLED,IMMUTABLE] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb]Quoted 7 lines
Quoted 3 lines
Shouldn't this be 3264x2464 as that's what the entity outputs after the cropping ? And shouldn't then be 3264x2464 the size output from the video device too ?
That's right.
I don't know why the source format for the scaler is still 3280x2464. Maybe there is a bug ong_fmtfor it.. have to check it.Thanks, I think this should be fixed ten
Quoted 17 lines
Anyway, the video format is like this :
# v4l2-ctl --get-fmt-video Format Video Capture: Width/Height : 3264/2464 Pixel Format : 'RGBP' (16-bit RGB 5-6-5) Field : None Bytes per Line : 6528 Size Image : 16084992 Colorspace : sRGB Transfer Function : Default (maps to sRGB) YCbCr/HSV Encoding: Default (maps to ITU-R 601) Quantization : Default (maps to Full Range) Flags : #Quoted 9 lines
and initializing the pipeline looks fine from user perspective. The scaler is in fact the solution to make this pipeline work with libcamera.
I found this cropping to be an issue in media controller when trying it with libcamera. Otherwise, the other user space apps which I was using never complained that anything was wrong
Libcamera simply refuses to acknowledge the pipeline if the video output is 3264x2464 but there is no entity that changes the format from 3280x2464 down
Quoted 2 lines
Not sure why libcamera should play a role there. Isn't it the media pipeline validation that complains and returns an
-EPIPE?
I tried this before the media_pipeline_start .
Perhaps libcamera was doing some similar checks, anyway, it was complaining and not adding the stream, and reported no usable streams/camera
Once I will rework some of the things according to your review I will run libcamera again to see what it complains about
Quoted 474 lines
Quoted 93 lines
Quoted 73 lines
Quoted 63 lines
-> "atmel_isc_common":0 [ENABLED,IMMUTABLE]Which is what we needed.
Quoted 19 lines
Quoted 6 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; +} + +static int isc_scaler_enum_mbus_code(struct v4l2_subdev *sd,drivers/media/platform/atmel/atmel-isc-scaler.c line 105
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+struct v4l2_subdev_state *sd_state, + struct v4l2_subdev_mbus_code_enum *code) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +int supported_index =0 ; +int i; + +for (i =0 ; i < isc->formats_list_size; i++) { +if (!isc->formats_list[i].sd_support) +continue ;The
sd_supportflag still doesn't click in my head.Quoted 2 lines
Shouldn't the enumeration of available formats on the scaler do not depend on the sensor supproted formats ?
You're right. I will have to check it again.
Quoted 30 lines
Quoted 16 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +15
-0+if (supported_index == code->index) { + code->code = isc->formats_list[i].mbus_code; +return 0 ; + } + supported_index++; + } + +return -EINVAL ; +} + +static int isc_scaler_g_sel (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_selection *sel) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd);drivers/media/platform/atmel/atmel-isc-scaler.c line 130
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +if (sel->pad ==ISC_SCALER_PAD_SOURCE ) +return -EINVAL ; + +if (sel->target !=V4L2_SEL_TGT_CROP_BOUNDS && + sel->target !=V4L2_SEL_TGT_CROP ) +return -EINVAL ; + + sel->r.height = isc->max_height; + sel->r.width = isc->max_width;The
CROP_BOUNDSshould be set to the same size as the sink pad image format, as it represents the maximum valid crop rectangle.Quoted 9 lines
Quoted 3 lines
TGT_CROPshould report the configured crop rectangle which can be intiialized to the same size asCROP_BOUNDS, if my understanding of the spec is correctSo you would like to have this differentiated, and report the
CROP_BOUNDSto whatever is on the sink pad, and theTGT_CROPto what is reported now, the maximum size of the ISC frame .
My understanding is correct ?Quoted 6 lines
I didn't know you have an HW limitation, so your
_BOUNDSis not the input image size but rather 3264x2464 ( ==max_widthxmax_height).What I meant is that
_BOUNDSshould report the maximum rectangle size that can be applied to the_CROPtarget. In you case you have an HW limitation 3264x2464 and that's the largest rectangle you can apply.So the CROP should be at 3264x2464
Quoted 2 lines
TGT_CROPcan be initialized to the same as_BOUND, but if you implements_selectionit should report what has been there applied.and BOUND to actual frame size ?
Quoted 2 lines
But as you don't implement
s_selectionyet, I think this is fine for now. Maybe a little comment ?It could also be the case where e.g. the sensor is outputting 1920x1080, in this case the scaler would do nothing.
CROP is still 3264x2464 and BOUND in this case is 1920x1080 ? If the sensor is outputting 1920x1080, this format comes directly from the sensor (the sensor is cropping it from 3280x2464 or not... it's the sensor's problem)Quoted 1 line
Quoted 4 lines
Quoted 2 lines
Also, is
set->rzeroed by the framework before getting here ? Otherwise you should set r.left and r.top to 0If these are redundant, no problem to remove them
I was actually confused. I was suggesting you to add...
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 133
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +3
-0+ + sel->r.left =0 ; + sel->r.top =0 ;These ^
So nothing to change. Sorry I've missed them
Thanks
jQuoted 360 lines
Quoted 359 lines
Quoted 49 lines
Quoted 40 lines
Quoted 12 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +12
-0+ +return 0 ; +} + +static int isc_scaler_init_cfg (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state) +{ +struct v4l2_mbus_framefmt *v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state,0 ); + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); + + *v4l2_try_fmt = isc->scaler_format;drivers/media/platform/atmel/atmel-isc-scaler.c line 155
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +return 0 ; +} + +static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = { + .enum_mbus_code = isc_scaler_enum_mbus_code, + .set_fmt = isc_scaler_set_fmt, + .get_fmt = isc_scaler_get_fmt, + .get_selection = isc_scaler_g_sel, + .init_cfg = isc_scaler_init_cfg,.link_validate =
v4l2_subdev_link_validate_default,To have the formats at the end of links that point to this entity validated (I think the framework already calls it if not set though, please check v4l2-subdev.c:v4l2_subdev_link_validate())
Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +11
-0+}; + +static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = { + .pad = &isc_scaler_pad_ops, +}; + +int isc_scaler_init (struct isc_device *isc) +{ +int ret; + +v4l2_subdev_init (&isc->scaler_sd, &xisc_scaler_subdev_ops);Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 171
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ + isc->scaler_sd.owner =THIS_MODULE ; + isc->scaler_sd.dev = isc->dev; +snprintf (isc->scaler_sd.name,sizeof (isc->scaler_sd.name), +"atmel_isc_scaler" );I would drop 'atmel' for brevity, unless other entities have this prefix set already
Quoted 3 lines
The v4l entity takes it's name from the module name, which is
atmel_isc_common, so I thought to keep the prefixAck!
Quoted 189 lines
Quoted 70 lines
Quoted 70 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +69
-0+ + isc->scaler_sd.flags |=V4L2_SUBDEV_FL_HAS_DEVNODE ; + isc->scaler_sd.entity.function =MEDIA_ENT_F_PROC_VIDEO_SCALER ; + isc->scaler_pads[ISC_SCALER_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + isc->scaler_pads[ISC_SCALER_PAD_SOURCE ].flags =MEDIA_PAD_FL_SOURCE ; + + isc->scaler_format.height = isc->max_height; + isc->scaler_format.width = isc->max_width; + isc->scaler_format.code = isc->formats_list[0 ].mbus_code; + isc->scaler_format.colorspace =V4L2_COLORSPACE_SRGB ; + isc->scaler_format.field =V4L2_FIELD_NONE ; + isc->scaler_format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + isc->scaler_format.quantization =V4L2_QUANTIZATION_DEFAULT ; + isc->scaler_format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + ret =media_entity_pads_init (&isc->scaler_sd.entity, +ISC_SCALER_PADS_NUM , + isc->scaler_pads); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd media entity init failed\n" ); +return ret; + } + ret =v4l2_device_register_subdev (&isc->v4l2_dev, &isc->scaler_sd); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd failed to register subdev\n" ); +return ret; + } + +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_init); + +int isc_scaler_link (struct isc_device *isc) +{ +int ret; + + ret =media_create_pad_link (&isc->current_subdev->sd->entity, + isc->remote_pad, &isc->scaler_sd.entity, +ISC_SCALER_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->current_subdev->sd->entity.name, + isc->scaler_sd.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , + isc->current_subdev->sd->name, isc->remote_pad); + + ret =media_create_pad_link (&isc->scaler_sd.entity, +ISC_SCALER_PAD_SOURCE , + &isc->video_dev.entity,ISC_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->scaler_sd.entity.name, + isc->video_dev.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , isc->scaler_sd.name, +ISC_SCALER_PAD_SOURCE );Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 244
Quoted 4 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +4
-0+ +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_link);From the DT graph point of view, the ISC appears as a single block
with an CSI-2 input port. It is then in charge of parsing the .dts and add to its notifier the image sensor remote subdevice.Actually it's only a parallel port.
And it can be connected either to a sensor directly or to the csi2dc bridge. (the csi2dc bridge can be connected to a CSI-2 receiver)Quoted 4 lines
When the sensor subdev registers and the ISC notifier completes, the scaler entity which was initialized at
isc_probe()time is linked in between the ISC and the image sensor immutably.yes, because this cannot change at runtime, and usually, it can't change without altering the board hardware. (at least this is what my understanding of immutability is )
Quoted 9 lines
I think it is fine for now, but I wonder if you plan to plumb more components between the ISC video node and the sensor, if it's not worth changing the DT bindings and their parsing logic to separate the CSI-2 receiver from the ISC, whcih can create its media graph at probe time and have the CSI-2 receiver entity as downstream remote. I think I need to get to know the ISC better to really have an idea. For now, this seems ok to me, but please check with maintainers if this is fine with them.
In the XISC case (sama7g5-isc), the csi2dc receiver is the subdev (so, the remote subdev).
The ISC will register a scaler, and connect the subdev to this scaler first, and then, the scaler to the isc itself (the v4l entity).Quoted 45 lines
Quoted 45 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +1
-0+drivers/media/platform/atmel/atmel-isc.h +21
-0@@ -183,6 +183,17 @@ struct isc_reg_offsets {u32 his_entry; }; +enum isc_mc_pads { +ISC_PAD_SINK =0 , +ISC_PADS_NUM =1 , +}; + +enum isc_scaler_pads { +ISC_SCALER_PAD_SINK =0 , +ISC_SCALER_PAD_SOURCE =1 , +ISC_SCALER_PADS_NUM =2 , +}; +/* * struct isc_device - ISC device driver data/config struct * @regmap: Register map @@ -257,6 +268,12 @@ struct isc_reg_offsets {* be used as an input to the controller * @controller_formats_size: size of controller_formats array * @formats_list_size: size of formats_list array +* @pads: media controller pads for isc video entity +* @mdev: media device that is registered by the isc +* @remote_pad: remote pad on the connected subdevice +* @scaler_sd: subdevice for the scaler that isc registers +* @scaler_pads: media controller pads for the scaler subdevice +* @scaler_format: current format for the scaler subdevice */ struct isc_device {struct regmap *regmap;@@ -344,6 +361,19 @@ struct isc_device {struct isc_format *formats_list;u32 controller_formats_size;u32 formats_list_size; + +struct { +struct media_pad pads[ISC_PADS_NUM ]; +struct media_device mdev;Quoted 8 lines
drivers/media/platform/atmel/atmel-isc.h struct isc_device { · line 375
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc.h +8
-0+ +u32 remote_pad; + }; + +struct { +struct v4l2_subdev scaler_sd; +struct media_pad scaler_pads[ISC_SCALER_PADS_NUM ]; +struct v4l2_mbus_framefmt scaler_format;Here are the scaler stuff which I added, in the same bulk struct for the whole isc device
Quoted 24 lines
Quoted 24 lines
drivers/media/platform/atmel/atmel-isc.h +8
-0+ }; };extern const struct regmap_config isc_regmap_config;@@ -355,4 +385,11 @@ int isc_clk_init(struct isc_device *isc);void isc_subdev_cleanup (struct isc_device *isc);void isc_clk_cleanup (struct isc_device *isc); +int isc_scaler_link (struct isc_device *isc); +int isc_scaler_init (struct isc_device *isc); +int isc_mc_init (struct isc_device *isc,u32 ver); +void isc_mc_cleanup (struct isc_device *isc); + +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index);#endif drivers/media/platform/atmel/atmel-sama5d2-isc.c +0
-0@@ -553,6 +553,12 @@ static int atmel_isc_probe(struct platform_device *pdev)break ; }Quoted 8 lines
Quoted 5 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c static int atmel_isc_probe(struct platform_device *pdev) · line 556
Quoted 1 line
drivers/media/platform/atmel/atmel-sama5d2-isc.c +1
-0+regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);I am surprised you can read a register before
runtime_pmis intialized!Actually I can read any register after the moment of starting the clock on the hardware block.
Ah right then
Quoted 3 lines
Maybe the starting and stopping of the clock needs to be moved to the
runtime_pmcalls, but this is another story, not related to the media controller.It's fine, but maybe worth recording with a todo ?
Thanks
jQuoted 95 lines
I moved this line because I had to pass the version to the
isc_mc_initcallQuoted 2 lines
Thanks
jThanks for reviewing !
EugenQuoted 87 lines
Quoted 87 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c +11
-2+ + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev);@@ -562,7 +568,7 @@ static int atmel_isc_probe(struct platform_device *pdev) ret =clk_prepare_enable (isc->ispck);if (ret) {dev_err (dev,"failed to enable ispck: %d\n" , ret); - goto cleanup_subdev; + goto isc_probe_mc_init_err; }/* ispck should be greater or equal to hclock */ @@ -572,7 +578,6 @@ static int atmel_isc_probe(struct platform_device *pdev) goto unprepare_clk; } -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip ISC version %x\n" , ver);return 0 ;@@ -580,6 +585,9 @@ static int atmel_isc_probe(struct platform_device *pdev) unprepare_clk:clk_disable_unprepare (isc->ispck); +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -600,6 +608,8 @@ static int atmel_isc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev);drivers/media/platform/atmel/atmel-sama7g5-isc.c +11
-2@@ -547,15 +547,23 @@ static int microchip_xisc_probe(struct platform_device *pdev)break ; } +regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver); + + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev); -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip XISC version %x\n" , ver);return 0 ; +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -576,6 +584,8 @@ static int microchip_xisc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev); --2.25.1
Hi Eugen
Quoted 404 lines · On Wed, Feb 09, 2022 at 06:59:37AM +0000, [email protected] wrote:
Quoted 178 lines · Jacopo Mondi
Hi Eugen
Quoted 163 lines · On Tue, Feb 08, 2022 at 10:33:31AM +0000, [email protected] wrote:
Quoted 162 lines · Jacopo Mondi
Hi Eugen
Quoted 158 lines · On Mon, Feb 07, 2022 at 01:40:31PM +0000, [email protected] wrote:
Quoted 138 lines · Jacopo Mondi
Hi Eugen
Quoted 135 lines · Eugen Hristev
Implement the support for media-controller.
This means that the capabilities of the driver have changed and now it also advertises theIO_MC.
The driver will register it's media device, and add the video entity to this media device. The subdevices are registered to the same media device. The ISC will have a base entity which is auto-detected asatmel_isc_base. It will also register a subdevice that allows cropping of the incoming frame to the maximum frame size supported by the ISC.
The ISC will create a link between the subdevice that is asynchronously registered and theatmel_isc_scalerentity.
Then, theatmel_isc_scalerandatmel_isc_baseare connected through another link.Signed-off-by Eugen Hristev <[email protected]>---
Changes in v4:
As suggested by Jacopo:
- renamedatmel_isc_mctoatmel_isc_scaler.c
- moved init_mc/clean_mc toisc_basefileChanges in v2:
- implement try formatsdrivers/media/platform/atmel/Makefile | 2 +- drivers/media/platform/atmel/atmel-isc-base.c | 73 +++++- .../media/platform/atmel/atmel-isc-scaler.c | 245 ++++++++++++++++++ drivers/media/platform/atmel/atmel-isc.h | 37 +++ .../media/platform/atmel/atmel-sama5d2-isc.c | 14 +- .../media/platform/atmel/atmel-sama7g5-isc.c | 12 +- 6 files changed, 375 insertions(+), 8 deletions(-)create mode 100644
drivers/media/platform/atmel/atmel-isc-scaler.cdrivers/media/platform/atmel/Makefile +1
-1@@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only atmel-isc-objs = atmel-sama5d2-isc.o atmel-xisc-objs = atmel-sama7g5-isc.o -atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o +atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o atmel-isc-scaler.o obj-$(CONFIG_VIDEO_ATMEL_ISI) += atmel-isi.o obj-$(CONFIG_VIDEO_ATMEL_ISC_BASE) += atmel-isc-common.odrivers/media/platform/atmel/atmel-isc-base.c +27
-4@@ -1710,6 +1710,7 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, struct isc_device, v4l2_dev);struct isc_subdev_entity *subdev_entity = container_of(notifier, struct isc_subdev_entity, notifier); +int pad;if (video_is_registered (&isc->video_dev)) {v4l2_err (&isc->v4l2_dev,"only supports one sub-device.\n" );@@ -1718,6 +1719,16 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, subdev_entity->sd = subdev; + pad =media_entity_get_fwnode_pad (&subdev->entity, asd->match.fwnode, +MEDIA_PAD_FL_SOURCE ); +if (pad <0 ) { +v4l2_err (&isc->v4l2_dev,"failed to find pad for %s\n" , + subdev->name); +return pad; + } + + isc->remote_pad = pad; +return 0 ; }@@ -1732,8 +1743,8 @@ static void isc_async_unbind(struct v4l2_async_notifier *notifier,v4l2_ctrl_handler_free (&isc->ctrls.handler); } -static struct isc_format *find_format_by_code (struct isc_device *isc, -unsigned int code,int *index) +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index) {struct isc_format *fmt = &isc->formats_list[0 ];unsigned int i;@@ -1749,6 +1760,7 @@ static struct isc_format *find_format_by_code(struct isc_device *isc,return NULL ; } +EXPORT_SYMBOL_GPL (isc_find_format_by_code);static int isc_formats_init (struct isc_device *isc) {@@ -1765,7 +1777,7 @@ static int isc_formats_init(struct isc_device *isc) NULL, &mbus_code)) { mbus_code.index++; - fmt =find_format_by_code (isc, mbus_code.code, &i); + fmt =isc_find_format_by_code (isc, mbus_code.code, &i);if (!fmt) {v4l2_warn (&isc->v4l2_dev,"Mbus code %x not supported\n" , mbus_code.code);@@ -1891,7 +1903,8 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) vdev->queue = q; vdev->lock = &isc->lock; vdev->ctrl_handler = &isc->ctrls.handler; - vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE ; + vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE | +V4L2_CAP_IO_MC ;video_set_drvdata (vdev, isc); ret =video_register_device (vdev,VFL_TYPE_VIDEO ,-1 );@@ -1901,8 +1914,18 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) goto isc_async_complete_err; } + ret =isc_scaler_link (isc); +if (ret <0 ) + goto isc_async_complete_unregister_device; + + ret =media_device_register (&isc->mdev); +if (ret <0 ) + goto isc_async_complete_unregister_device;return 0 ; +isc_async_complete_unregister_device: +video_unregister_device (vdev); + isc_async_complete_err:mutex_destroy (&isc->lock);return ret;@@ -1969,6 +1992,48 @@ int isc_pipeline_init(struct isc_device *isc) }EXPORT_SYMBOL_GPL (isc_pipeline_init);Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-base.c int isc_pipeline_init(struct isc_device *isc) · line 2001
Quoted 7 lines
drivers/media/platform/atmel/atmel-isc-base.c +7
-0+int isc_mc_init (struct isc_device *isc,u32 ver) +{ +const struct of_device_id *match; +int ret; + + isc->video_dev.entity.function =MEDIA_ENT_F_IO_V4L ; + isc->video_dev.entity.flags =MEDIA_ENT_FL_DEFAULT ;Should you set entity.ops.link_validate =
v4l2_subdev_link_validateto be able to have the media pipeline validated atmedia_pipeline_start()time ?Hi,
I am doing that in a subsequent patch. Things are not completely ready at this moment, because ISC still relies on the old mechanism to call
v4l2_subdev_callwithset_fmton the subdevice (which subsequent patch removes and adds checks to the link validate ).Quoted 6 lines
Quoted 1 line
I see.. the subsequent patches are not part of this series, right ?
No. Patch 7 does this. In that patch, the link validation is created and all the logic of format propagation is removed and reworked, and creates the need for the
link_validatecall
Could you have also a look at that patch ?Ah ups, 7 was not in my inbox with the rest of the series. Just noticed. I had even reviewed the previous version, I should have remembered
link_validatewas added laterQuoted 1 line
Quoted 1 line
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
I think patches 1, 2, 4, 5 and 6 from this series do not depend on the introduction of media controller or the scaler, right ? (I'm not sure if the DTS patches do).
Do DTS patches depend on media-controller plumbing ?
I think not, but, I have not tested them without the media controller.
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
If that's the case, would it make sense to to fast-track them (as some of them are fixes) and then on top plumb the media controller infrastructure with this patch and the ones you have been mentioning ?
It would make sense.
Once the other discussion with Hans about the stop state it might make sense to fastrack the first part of the series ?
The other three patches (4,5,6) are quite independent and can go faster.
Quoted 149 lines
Quoted 139 lines
Quoted 119 lines
Quoted 119 lines
Quoted 79 lines
Quoted 47 lines
drivers/media/platform/atmel/atmel-isc-base.c +35
-0+ isc->pads[ISC_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + + ret =media_entity_pads_init (&isc->video_dev.entity,ISC_PADS_NUM , + isc->pads); +if (ret <0 ) { +dev_err (isc->dev,"media entity init failed\n" ); +return ret; + } + + isc->mdev.dev = isc->dev; + + match =of_match_node (isc->dev->driver->of_match_table, + isc->dev->of_node); + +strscpy (isc->mdev.driver_name,KBUILD_MODNAME , +sizeof (isc->mdev.driver_name)); +strscpy (isc->mdev.model, match->compatible,sizeof (isc->mdev.model)); +snprintf (isc->mdev.bus_info,sizeof (isc->mdev.bus_info),"platform:%s" , + isc->v4l2_dev.name); + isc->mdev.hw_revision = ver; + +media_device_init (&isc->mdev); + + isc->v4l2_dev.mdev = &isc->mdev; + +return isc_scaler_init (isc); +} +EXPORT_SYMBOL_GPL (isc_mc_init); + +void isc_mc_cleanup (struct isc_device *isc) +{ +media_entity_cleanup (&isc->video_dev.entity); +} +EXPORT_SYMBOL_GPL (isc_mc_cleanup); +/* regmap configuration */ #define ATMEL_ISC_REG_MAX 0xd5cconst struct regmap_config isc_regmap_config = {drivers/media/platform/atmel/atmel-isc-scaler.c new file +0
-0drivers/media/platform/atmel/atmel-isc-scaler.c line 5
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0@@ -0,0 +1,245 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Microchip Image Sensor Controller (ISC) Scaler entity support +* +* Copyright (C) 2021 Microchip Technology, Inc. Time flies! It's 2022 already :)
Quoted 24 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +24
-0+* +* Author: Eugen Hristev <[email protected]> +* +*/ + +#include <media/media-device.h> +#include <media/media-entity.h> +#include <media/v4l2-device.h> +#include <media/v4l2-subdev.h> + +#include "atmel-isc-regs.h" +#include "atmel-isc.h" + +static int isc_scaler_get_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *format) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; + +if (format->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + format->pad); + format->format = *v4l2_try_fmt;Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 34
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; + } + + format->format = isc->scaler_format;
isc->scaler_formatis only used inside this file if I'm not mistaken. I wonder why it lives in theisc_devicestruct.
isc_deviceis a placeholder for all isc things.I would not create a separate struct in this file that would have to be allocated etc... just for one two things. So I preferred to have it in the same place as all the other things.
Quoted 24 lines
Quoted 12 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +11
-0+ +return 0 ; +} + +static int isc_scaler_set_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *req_fmt) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; +struct isc_format *fmt;drivers/media/platform/atmel/atmel-isc-scaler.c line 55
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+unsigned int i; + +if (req_fmt->pad ==ISC_SCALER_PAD_SOURCE ) +v4l_bound_align_image + (&req_fmt->format.width,16 , isc->max_width,0 , + &req_fmt->format.height,16 , isc->max_height,0 ,0 ); +else +v4l_bound_align_image + (&req_fmt->format.width,16 ,10000 ,0 , + &req_fmt->format.height,16 ,10000 ,0 ,0 );Where does 10000 come from ?
Quoted 3 lines
Quoted 1 line
It's a random number. Do you have any suggestion for a better one ?
An actual HW limit ? :)
There is no hardware limit. The ISC just stops sampling pixels once the crop limit is reached. The element that generates pixels could go on forever , or until the next HBLANK/VBLANK. So what limit could I place here ?
The cropping is mandatory, otherwise there could be an overflow w.r.t. the buffer size if the ISC would sample more pixels than the software expects it to.Quoted 8 lines
Quoted 7 lines
Maybe this would be much more clear with my comments below (where I will explain what the isc scaler does )
In short, it allows the other entity in the link to 'sink' a huge format into this 'scaler' . Because the scaler can crop anything basically down to the biggest format size the ISC could handle.Quoted 9 lines
Quoted 1 line
doesn't it have any documented input size limit ?
As stated above, ISC handles a pixel stream and stores it in an internal SRAM. ISC stops sampling when this SRAM is full or when the cropping limit is reached.
After storing the pixels in the SRAM (which is limited to the ISC maximum frame size), the ISC will work on the pixels and then DMA the frame to another RAM.Quoted 4 lines
I understand...
So whatever the input size is, only the first n-th bytes are actually dumped to memory, the following ones are discarded.
Yes ! That's the behavior.
Quoted 3 lines
So the crop is always applied on the top-leftmost corner as a consequence, right ?
Right. Pixels come in order. The ISC does not really know how many pixels are coming until its memory is full (or counting until the crop limit is reached). After that it stops sampling pixels, and waits for vblank to finish the frame.
Quoted 8 lines
I understand why you had difficulties picking a default :)
This is not easy to handle, as the scaler doesn't actually applies a crop but rather stops capturing after the destination buffer has been saturated. So the limiting factor is the full image size, not one of its dimensions, like in example the line length. This makes it hard to adjust cropping to something meaningful for most use cases.
I didn't know how to name this except crop. It 'crops out' what does not fit in the internal memory of the ISC
Quoted 7 lines
In your below example you suggest that an image size of (800000 x 10) would theoretically be captured correctly. What if the input is of size (800000 x 10 + 1). What size would you set the crop rectangle to ?
I wouldn't however be too much concerned for the moment, but I would record this in a comment ?
I discussed a bit more with designer and looked in datasheet. It looks like the horizontal line length is not a hard limit, but it makes the ISC internal ISP stop working, so I would keep the horizontal limit as well for now, unless an important use case (I am coming back to that). The vertical limit looks to be a hard requirement.
So in short, we have to impose a vertical limit of 2464 , we cannot have more lines than these. So if the horizontal is 100 pixels e.g., we can capture only 100x2464 .
In practice we could capture [16,3264] on the horizontal, and [16,2464] on the vertical, and in theory horizontal could go more like [16, 4000], but we have to limit the vertical, and without the internal ISP. So I would not haphazardly go in this direction. Let's stick to the datasheet maximum frame 3264x2464 .
According to this , I changed the frame size of the ISC to be a continuous size from [16,16] to [3264,2464]
About that use case that I talked about earlier, we have one sensor , imx274, which is more wide, and outputs something like 3800x2100 . With the cropping and limitations we have in place now, this will be limited to 3264x2100 , which is in line with datasheet and what the ISC
Quoted 8 lines
supports now.
As in theory we could capture 3800x2100 as long as the SRAM doesn't get full, for now I would like to avoid capturing this size (and disable the internal ISP), until we have someone really asking for such kind of capture and with some good reasons.
In short I would like to have the crop rectangle to 3264x2464 and the bounds would be limited to the incoming frame but no H>3264 and no V>2464. Does that make sense ?
Just to make sure I got it: is the ISC actually able to crop on the line length ? If you apply a 3264 limit, will the last 536 pixels of each line will be dropped if the input frame is 3800x2100 as your above example ?
Or rather, is the SRAM size limit the issue and the lines gets captured in full lenght no matter what, until the buffer gets filled up and all the successive pixels dropped.
In other words, do your cropped image look like:
3264
+--------------------------------X-----------+ 3800
|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |
|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |
|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |
|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |
|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |
|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |
|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |
|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |
|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |
|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |
|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |
+--------------------------------------------+ 2100Or rather
3264
+--------------------------------X-----------+ 3800
|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
|xxxxxxxxxxxxxxxxx| |
| |
+--------------------------------------------+ 2100I assume the first, as otherwise you wouldn't be able to interpret the image as 3264x2100.
I'm asking because you had the un-cropped 3280x2464 size as the ISC source pad size in the paste of the media graph you provided below, which might mean the output image is actually full size in lines lenght and it's maybe missing pixels at the end, in case the ISC operates as in the above second case.
Quoted 751 lines
Quoted 266 lines
Quoted 100 lines
Quoted 72 lines
Quoted 16 lines
Quoted 4 lines
Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 63
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +8
-0+ + req_fmt->format.colorspace =V4L2_COLORSPACE_SRGB ; + req_fmt->format.field =V4L2_FIELD_NONE ; + req_fmt->format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + req_fmt->format.quantization =V4L2_QUANTIZATION_DEFAULT ; + req_fmt->format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + fmt =isc_find_format_by_code (isc, req_fmt->format.code, &i);So you rely on the isc format list for the scaler as well ? I think it's fine as long as they are identical
Quoted 3 lines
Yes, the scaler is kind of a dummy entity , but it's required by the media controller validation of the pipeline.
Quoted 44 lines
Quoted 35 lines
Quoted 18 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +17
-0+ +if (!fmt) + fmt = &isc->formats_list[0 ]; + + req_fmt->format.code = fmt->mbus_code; + +if (req_fmt->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + req_fmt->pad); + *v4l2_try_fmt = req_fmt->format; +/* Trying on the pad sink makes the source sink change too */ +if (req_fmt->pad ==ISC_SCALER_PAD_SINK ) { + v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state, +ISC_SCALER_PAD_SOURCE ); + *v4l2_try_fmt = req_fmt->format; +drivers/media/platform/atmel/atmel-isc-scaler.c line 90
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+v4l_bound_align_image (&v4l2_try_fmt->width, +16 , isc->max_width,0 , + &v4l2_try_fmt->height, +16 , isc->max_height,0 ,0 ); + } +/* if we are just trying, we are done */ +return 0 ; + } + + isc->scaler_format = req_fmt->format;No per-pad format but a global
scaler_format? How do you configure scaling ?Actually, I would like to know more about the scaler device capabilities. What functions can this IP perform ? Does it do cropping, can it also do (down)scaling or even composition ?
Quoted 8 lines
I think it is worth to read
https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-subdev.html#v4l2-subdev-selections
where it is reported how cropping and scaling are implemented by using the selection API.Figure 4.5 is particularly helpful to explain the simple crop case, for which you need to implement support to by adding
s_selectionon the sink padTGT_CROPtarget.Quoted 5 lines
The scaler is kind of a dummy entity.
The problem is that different subdevices cannot be connected directly to a v4l entity, because the ISC does cropping on the incoming frame. The ISC has a limit on the total number of pixels per frame, thus it will do a crop.Quoted 6 lines
What is this limit related to ? I don't know the ISC internals, but is the limitation due to the overall image size, or maybe it's due to some line buffers being limited in the size they can transfer to memory ? Is it a limitation on the size of the image in pixels, or is it due to a limitation of the processing bandwidth on the input interface and thus depends on the actual size of the image in bytes ?
Limitation is both on line size and full image size. From my understanding, is that the internal SRAM is limited (8 Mpixels with a small safe buffer on top in the case of sama7g5 )
This SRAM could be used as 800000x10 for example, or 3200x2464 for another example. However, I am not sure if a real line of 800,000 pixels make sense, or if there is another internal mechanism in the ISC that stops it.
In any case, the ISC is cropping the incoming frame from aaaXbbb up to the maximum frame that it can handleQuoted 10 lines
Quoted 6 lines
So, if I allow e.g. an entity that sends 3280x2464 , when the ISC can only handle 3264x2464 , then we have two choices:
1/ crop it and output the 3264x2464 -> cannot, because userspace expects 3280x2464, and the whole pipeline fails to configure, there is a mismatch between/dev/videooutput and what the whole pipeline producesI'm confused in first place. If the ISC cannot output anything larger than 3264x2464, then if userspace sets a 3280x2464 size on the ISC, this should be adjusted to 3264x2464.
Quoted 9 lines
yes, that is true. However, without an entity that says 'I am doing cropping', the media pipeline will fail, because previous entity will output 3280x2464 .
The conclusion to create a scaler entitity was done during my last year discussions on the IRC. I could not find a way to perform this cropping at the v4l entity side, and one reason is that the v4l entity does not support what I needed , or at least that was my understanding. You claim that the v4l entity could perform the cropping in the same entity ?I thought so, but I haven't tried to be honest
Quoted 10 lines
Let's assume for a minute that it could do this. Even so, I would prefer to have a separate scaler entity to do this , for several reasons:
-> I would like to be able to crop the frame arbitrarily if I want to, and the scaler would allow me to do this easier
-> it would be more obvious that the frame is cropped by having this entity there
-> in fact some of the ISC versions really have a down scaler, that I have not implemented yet. So it would be useful to already have this in place to be able to scale down with it at a later time.I think this last reason is enough to have an entity plumbed in already
Quoted 8 lines
Quoted 8 lines
Or is the ISC limitation on the -input- side ?
Quoted 3 lines
2/ deny the link, and ask the subdevice to produce the 3264x2464 which the ISC can handle at most -> cannot , because the sensor let's say can *only* produce 3280x2464 and *any* other frame size returns an error.
Quoted 6 lines
Quoted 1 line
buggy sensor I would say :)
I cannot tell this to the customers, and I cannot fail the whole pipeline because the sensor does not crop 16 pixels, when in fact I can do this very easily on the ISC side.
Fair enough :)
Quoted 85 lines
Quoted 11 lines
Quoted 7 lines
The only solution to make both worlds happy, is to have a dummy entity called 'scaler' which in fact now it only performs a simple crop. It accepts any frame size at input (hence the 10000x10000 which you saw earlier), and outputs at most 3264x2464 the
isc_max_heightandisc_max_width.I have a maybe dumb question: can the cropping operation be modeled by applying a
TGT_CROPrectangle (whose_BOUNDsize is 3280x2464) to theatmel_isc_commonentity sink pad and avoid the scaler completely ?I am not sure. This is what I wanted initially. But then I thought about the three reasons stated above for the use of the scaler entity, and discussed with folks on the IRC, and come up with the solution for the scaler entity
Quoted 68 lines
Quoted 67 lines
So to answer your question, the isc scaler is a software model for a simple cropping, that would make media controller happy, and capture software happy.
Here it how it looks :
Device topology
- entity 1:atmel_isc_scaler(2 pads, 2 links)type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev0 pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb crop.bounds:(0,0)/3264x2464 crop:(0,0)/3264x2464] <- "csi2dc":1 [ENABLED,IMMUTABLE] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] -> "atmel_isc_common":0 [ENABLED,IMMUTABLE]- entity 4: csi2dc (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev1 pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] <- "dw-csi.0":1 [ENABLED] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] -> "atmel_isc_scaler":0 [ENABLED,IMMUTABLE]- entity 7: dw-csi.0 (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev2 pad0: Sink [fmt:SRGGB10_1X10/3280x2464] <- "imx219 1-0010":0 [ENABLED] pad1: Source [fmt:SRGGB10_1X10/3280x2464] -> "csi2dc":0 [ENABLED]- entity 12: imx219 1-0010 (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0 device node name/dev/v4l-subdev3
pad0: Source
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb
xfer:srgb ycbcr:601 quantization:full-rangecrop.bounds:(8,8)/3280x2464 crop:(8,8)/3280x2464] -> "dw-csi.0":0 [ENABLED]- entity 24:
atmel_isc_common(1 pad, 1 link)
type Node subtype V4L flags 1
device node name/dev/video0
pad0: Sink
<- "atmel_isc_scaler":1 [ENABLED,IMMUTABLE]Scaler does this one cute little thing :
pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb crop.bounds:(0,0)/3264x2464 crop:(0,0)/3264x2464] <- "csi2dc":1 [ENABLED,IMMUTABLE] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb]Quoted 7 lines
Quoted 3 lines
Shouldn't this be 3264x2464 as that's what the entity outputs after the cropping ? And shouldn't then be 3264x2464 the size output from the video device too ?
That's right.
I don't know why the source format for the scaler is still 3280x2464. Maybe there is a bug ong_fmtfor it.. have to check it.Thanks, I think this should be fixed ten
Quoted 17 lines
Anyway, the video format is like this :
# v4l2-ctl --get-fmt-video Format Video Capture: Width/Height : 3264/2464 Pixel Format : 'RGBP' (16-bit RGB 5-6-5) Field : None Bytes per Line : 6528 Size Image : 16084992 Colorspace : sRGB Transfer Function : Default (maps to sRGB) YCbCr/HSV Encoding: Default (maps to ITU-R 601) Quantization : Default (maps to Full Range) Flags : #Quoted 9 lines
and initializing the pipeline looks fine from user perspective. The scaler is in fact the solution to make this pipeline work with libcamera.
I found this cropping to be an issue in media controller when trying it with libcamera. Otherwise, the other user space apps which I was using never complained that anything was wrong
Libcamera simply refuses to acknowledge the pipeline if the video output is 3264x2464 but there is no entity that changes the format from 3280x2464 downQuoted 2 lines
Not sure why libcamera should play a role there. Isn't it the media pipeline validation that complains and returns an
-EPIPE?I tried this before the
media_pipeline_start.
Perhaps libcamera was doing some similar checks, anyway, it was complaining and not adding the stream, and reported no usable streams/cameraOnce I will rework some of the things according to your review I will run libcamera again to see what it complains about
Quoted 474 lines
Quoted 93 lines
Quoted 73 lines
Quoted 63 lines
-> "atmel_isc_common":0 [ENABLED,IMMUTABLE]Which is what we needed.
Quoted 19 lines
Quoted 6 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; +} + +static int isc_scaler_enum_mbus_code(struct v4l2_subdev *sd,drivers/media/platform/atmel/atmel-isc-scaler.c line 105
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+struct v4l2_subdev_state *sd_state, + struct v4l2_subdev_mbus_code_enum *code) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +int supported_index =0 ; +int i; + +for (i =0 ; i < isc->formats_list_size; i++) { +if (!isc->formats_list[i].sd_support) +continue ;The
sd_supportflag still doesn't click in my head.Quoted 2 lines
Shouldn't the enumeration of available formats on the scaler do not depend on the sensor supproted formats ?
You're right. I will have to check it again.
Quoted 30 lines
Quoted 16 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +15
-0+if (supported_index == code->index) { + code->code = isc->formats_list[i].mbus_code; +return 0 ; + } + supported_index++; + } + +return -EINVAL ; +} + +static int isc_scaler_g_sel (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_selection *sel) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd);drivers/media/platform/atmel/atmel-isc-scaler.c line 130
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +if (sel->pad ==ISC_SCALER_PAD_SOURCE ) +return -EINVAL ; + +if (sel->target !=V4L2_SEL_TGT_CROP_BOUNDS && + sel->target !=V4L2_SEL_TGT_CROP ) +return -EINVAL ; + + sel->r.height = isc->max_height; + sel->r.width = isc->max_width;The
CROP_BOUNDSshould be set to the same size as the sink pad image format, as it represents the maximum valid crop rectangle.Quoted 9 lines
Quoted 3 lines
TGT_CROPshould report the configured crop rectangle which can be intiialized to the same size asCROP_BOUNDS, if my understanding of the spec is correctSo you would like to have this differentiated, and report the
CROP_BOUNDSto whatever is on the sink pad, and theTGT_CROPto what is reported now, the maximum size of the ISC frame .
My understanding is correct ?Quoted 6 lines
I didn't know you have an HW limitation, so your
_BOUNDSis not the input image size but rather 3264x2464 ( ==max_widthxmax_height).What I meant is that
_BOUNDSshould report the maximum rectangle size that can be applied to the_CROPtarget. In you case you have an HW limitation 3264x2464 and that's the largest rectangle you can apply.So the CROP should be at 3264x2464
Quoted 2 lines
TGT_CROPcan be initialized to the same as_BOUND, but if you implements_selectionit should report what has been there applied.and BOUND to actual frame size ?
Quoted 2 lines
But as you don't implement
s_selectionyet, I think this is fine for now. Maybe a little comment ?It could also be the case where e.g. the sensor is outputting 1920x1080, in this case the scaler would do nothing.
CROP is still 3264x2464 and BOUND in this case is 1920x1080 ? If the sensor is outputting 1920x1080, this format comes directly from the sensor (the sensor is cropping it from 3280x2464 or not... it's the sensor's problem)Quoted 1 line
Quoted 4 lines
Quoted 2 lines
Also, is
set->rzeroed by the framework before getting here ? Otherwise you should set r.left and r.top to 0If these are redundant, no problem to remove them
I was actually confused. I was suggesting you to add...
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 133
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +3
-0+ + sel->r.left =0 ; + sel->r.top =0 ;These ^
So nothing to change. Sorry I've missed them
Thanks
jQuoted 360 lines
Quoted 359 lines
Quoted 49 lines
Quoted 40 lines
Quoted 12 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +12
-0+ +return 0 ; +} + +static int isc_scaler_init_cfg (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state) +{ +struct v4l2_mbus_framefmt *v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state,0 ); + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); + + *v4l2_try_fmt = isc->scaler_format;drivers/media/platform/atmel/atmel-isc-scaler.c line 155
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +return 0 ; +} + +static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = { + .enum_mbus_code = isc_scaler_enum_mbus_code, + .set_fmt = isc_scaler_set_fmt, + .get_fmt = isc_scaler_get_fmt, + .get_selection = isc_scaler_g_sel, + .init_cfg = isc_scaler_init_cfg,.link_validate =
v4l2_subdev_link_validate_default,To have the formats at the end of links that point to this entity validated (I think the framework already calls it if not set though, please check v4l2-subdev.c:v4l2_subdev_link_validate())
Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +11
-0+}; + +static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = { + .pad = &isc_scaler_pad_ops, +}; + +int isc_scaler_init (struct isc_device *isc) +{ +int ret; + +v4l2_subdev_init (&isc->scaler_sd, &xisc_scaler_subdev_ops);Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 171
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ + isc->scaler_sd.owner =THIS_MODULE ; + isc->scaler_sd.dev = isc->dev; +snprintf (isc->scaler_sd.name,sizeof (isc->scaler_sd.name), +"atmel_isc_scaler" );I would drop 'atmel' for brevity, unless other entities have this prefix set already
Quoted 3 lines
The v4l entity takes it's name from the module name, which is
atmel_isc_common, so I thought to keep the prefixAck!
Quoted 189 lines
Quoted 70 lines
Quoted 70 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +69
-0+ + isc->scaler_sd.flags |=V4L2_SUBDEV_FL_HAS_DEVNODE ; + isc->scaler_sd.entity.function =MEDIA_ENT_F_PROC_VIDEO_SCALER ; + isc->scaler_pads[ISC_SCALER_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + isc->scaler_pads[ISC_SCALER_PAD_SOURCE ].flags =MEDIA_PAD_FL_SOURCE ; + + isc->scaler_format.height = isc->max_height; + isc->scaler_format.width = isc->max_width; + isc->scaler_format.code = isc->formats_list[0 ].mbus_code; + isc->scaler_format.colorspace =V4L2_COLORSPACE_SRGB ; + isc->scaler_format.field =V4L2_FIELD_NONE ; + isc->scaler_format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + isc->scaler_format.quantization =V4L2_QUANTIZATION_DEFAULT ; + isc->scaler_format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + ret =media_entity_pads_init (&isc->scaler_sd.entity, +ISC_SCALER_PADS_NUM , + isc->scaler_pads); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd media entity init failed\n" ); +return ret; + } + ret =v4l2_device_register_subdev (&isc->v4l2_dev, &isc->scaler_sd); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd failed to register subdev\n" ); +return ret; + } + +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_init); + +int isc_scaler_link (struct isc_device *isc) +{ +int ret; + + ret =media_create_pad_link (&isc->current_subdev->sd->entity, + isc->remote_pad, &isc->scaler_sd.entity, +ISC_SCALER_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->current_subdev->sd->entity.name, + isc->scaler_sd.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , + isc->current_subdev->sd->name, isc->remote_pad); + + ret =media_create_pad_link (&isc->scaler_sd.entity, +ISC_SCALER_PAD_SOURCE , + &isc->video_dev.entity,ISC_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->scaler_sd.entity.name, + isc->video_dev.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , isc->scaler_sd.name, +ISC_SCALER_PAD_SOURCE );Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 244
Quoted 4 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +4
-0+ +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_link);From the DT graph point of view, the ISC appears as a single block
with an CSI-2 input port. It is then in charge of parsing the .dts and add to its notifier the image sensor remote subdevice.Actually it's only a parallel port.
And it can be connected either to a sensor directly or to the csi2dc bridge. (the csi2dc bridge can be connected to a CSI-2 receiver)Quoted 4 lines
When the sensor subdev registers and the ISC notifier completes, the scaler entity which was initialized at
isc_probe()time is linked in between the ISC and the image sensor immutably.yes, because this cannot change at runtime, and usually, it can't change without altering the board hardware. (at least this is what my understanding of immutability is )
Quoted 9 lines
I think it is fine for now, but I wonder if you plan to plumb more components between the ISC video node and the sensor, if it's not worth changing the DT bindings and their parsing logic to separate the CSI-2 receiver from the ISC, whcih can create its media graph at probe time and have the CSI-2 receiver entity as downstream remote. I think I need to get to know the ISC better to really have an idea. For now, this seems ok to me, but please check with maintainers if this is fine with them.
In the XISC case (sama7g5-isc), the csi2dc receiver is the subdev (so, the remote subdev).
The ISC will register a scaler, and connect the subdev to this scaler first, and then, the scaler to the isc itself (the v4l entity).Quoted 45 lines
Quoted 45 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +1
-0+drivers/media/platform/atmel/atmel-isc.h +21
-0@@ -183,6 +183,17 @@ struct isc_reg_offsets {u32 his_entry; }; +enum isc_mc_pads { +ISC_PAD_SINK =0 , +ISC_PADS_NUM =1 , +}; + +enum isc_scaler_pads { +ISC_SCALER_PAD_SINK =0 , +ISC_SCALER_PAD_SOURCE =1 , +ISC_SCALER_PADS_NUM =2 , +}; +/* * struct isc_device - ISC device driver data/config struct * @regmap: Register map @@ -257,6 +268,12 @@ struct isc_reg_offsets {* be used as an input to the controller * @controller_formats_size: size of controller_formats array * @formats_list_size: size of formats_list array +* @pads: media controller pads for isc video entity +* @mdev: media device that is registered by the isc +* @remote_pad: remote pad on the connected subdevice +* @scaler_sd: subdevice for the scaler that isc registers +* @scaler_pads: media controller pads for the scaler subdevice +* @scaler_format: current format for the scaler subdevice */ struct isc_device {struct regmap *regmap;@@ -344,6 +361,19 @@ struct isc_device {struct isc_format *formats_list;u32 controller_formats_size;u32 formats_list_size; + +struct { +struct media_pad pads[ISC_PADS_NUM ]; +struct media_device mdev;Quoted 8 lines
drivers/media/platform/atmel/atmel-isc.h struct isc_device { · line 375
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc.h +8
-0+ +u32 remote_pad; + }; + +struct { +struct v4l2_subdev scaler_sd; +struct media_pad scaler_pads[ISC_SCALER_PADS_NUM ]; +struct v4l2_mbus_framefmt scaler_format;Here are the scaler stuff which I added, in the same bulk struct for the whole isc device
Quoted 24 lines
Quoted 24 lines
drivers/media/platform/atmel/atmel-isc.h +8
-0+ }; };extern const struct regmap_config isc_regmap_config;@@ -355,4 +385,11 @@ int isc_clk_init(struct isc_device *isc);void isc_subdev_cleanup (struct isc_device *isc);void isc_clk_cleanup (struct isc_device *isc); +int isc_scaler_link (struct isc_device *isc); +int isc_scaler_init (struct isc_device *isc); +int isc_mc_init (struct isc_device *isc,u32 ver); +void isc_mc_cleanup (struct isc_device *isc); + +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index);#endif drivers/media/platform/atmel/atmel-sama5d2-isc.c +0
-0@@ -553,6 +553,12 @@ static int atmel_isc_probe(struct platform_device *pdev)break ; }Quoted 8 lines
Quoted 5 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c static int atmel_isc_probe(struct platform_device *pdev) · line 556
Quoted 1 line
drivers/media/platform/atmel/atmel-sama5d2-isc.c +1
-0+regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);I am surprised you can read a register before
runtime_pmis intialized!Actually I can read any register after the moment of starting the clock on the hardware block.
Ah right then
Quoted 3 lines
Maybe the starting and stopping of the clock needs to be moved to the
runtime_pmcalls, but this is another story, not related to the media controller.It's fine, but maybe worth recording with a todo ?
Thanks
jQuoted 95 lines
I moved this line because I had to pass the version to the
isc_mc_initcallQuoted 2 lines
Thanks
jThanks for reviewing !
EugenQuoted 87 lines
Quoted 87 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c +11
-2+ + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev);@@ -562,7 +568,7 @@ static int atmel_isc_probe(struct platform_device *pdev) ret =clk_prepare_enable (isc->ispck);if (ret) {dev_err (dev,"failed to enable ispck: %d\n" , ret); - goto cleanup_subdev; + goto isc_probe_mc_init_err; }/* ispck should be greater or equal to hclock */ @@ -572,7 +578,6 @@ static int atmel_isc_probe(struct platform_device *pdev) goto unprepare_clk; } -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip ISC version %x\n" , ver);return 0 ;@@ -580,6 +585,9 @@ static int atmel_isc_probe(struct platform_device *pdev) unprepare_clk:clk_disable_unprepare (isc->ispck); +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -600,6 +608,8 @@ static int atmel_isc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev);drivers/media/platform/atmel/atmel-sama7g5-isc.c +11
-2@@ -547,15 +547,23 @@ static int microchip_xisc_probe(struct platform_device *pdev)break ; } +regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver); + + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev); -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip XISC version %x\n" , ver);return 0 ; +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -576,6 +584,8 @@ static int microchip_xisc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev); --2.25.1
Quoted 416 lines · Jacopo Mondi
Hi Eugen
Quoted 404 lines · On Wed, Feb 09, 2022 at 06:59:37AM +0000, [email protected] wrote:
Quoted 178 lines · Jacopo Mondi
Hi Eugen
Quoted 163 lines · On Tue, Feb 08, 2022 at 10:33:31AM +0000, [email protected] wrote:
Quoted 162 lines · Jacopo Mondi
Hi Eugen
Quoted 158 lines · On Mon, Feb 07, 2022 at 01:40:31PM +0000, [email protected] wrote:
Quoted 138 lines · Jacopo Mondi
Hi Eugen
Quoted 135 lines · Eugen Hristev
Implement the support for media-controller.
This means that the capabilities of the driver have changed and now it also advertises theIO_MC.
The driver will register it's media device, and add the video entity to this media device. The subdevices are registered to the same media device. The ISC will have a base entity which is auto-detected asatmel_isc_base. It will also register a subdevice that allows cropping of the incoming frame to the maximum frame size supported by the ISC.
The ISC will create a link between the subdevice that is asynchronously registered and theatmel_isc_scalerentity.
Then, theatmel_isc_scalerandatmel_isc_baseare connected through another link.Signed-off-by Eugen Hristev <[email protected]>---
Changes in v4:
As suggested by Jacopo:
- renamedatmel_isc_mctoatmel_isc_scaler.c
- moved init_mc/clean_mc toisc_basefileChanges in v2:
- implement try formatsdrivers/media/platform/atmel/Makefile | 2 +- drivers/media/platform/atmel/atmel-isc-base.c | 73 +++++- .../media/platform/atmel/atmel-isc-scaler.c | 245 ++++++++++++++++++ drivers/media/platform/atmel/atmel-isc.h | 37 +++ .../media/platform/atmel/atmel-sama5d2-isc.c | 14 +- .../media/platform/atmel/atmel-sama7g5-isc.c | 12 +- 6 files changed, 375 insertions(+), 8 deletions(-)create mode 100644
drivers/media/platform/atmel/atmel-isc-scaler.cdrivers/media/platform/atmel/Makefile +1
-1@@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only atmel-isc-objs = atmel-sama5d2-isc.o atmel-xisc-objs = atmel-sama7g5-isc.o -atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o +atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o atmel-isc-scaler.o obj-$(CONFIG_VIDEO_ATMEL_ISI) += atmel-isi.o obj-$(CONFIG_VIDEO_ATMEL_ISC_BASE) += atmel-isc-common.odrivers/media/platform/atmel/atmel-isc-base.c +27
-4@@ -1710,6 +1710,7 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, struct isc_device, v4l2_dev);struct isc_subdev_entity *subdev_entity = container_of(notifier, struct isc_subdev_entity, notifier); +int pad;if (video_is_registered (&isc->video_dev)) {v4l2_err (&isc->v4l2_dev,"only supports one sub-device.\n" );@@ -1718,6 +1719,16 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, subdev_entity->sd = subdev; + pad =media_entity_get_fwnode_pad (&subdev->entity, asd->match.fwnode, +MEDIA_PAD_FL_SOURCE ); +if (pad <0 ) { +v4l2_err (&isc->v4l2_dev,"failed to find pad for %s\n" , + subdev->name); +return pad; + } + + isc->remote_pad = pad; +return 0 ; }@@ -1732,8 +1743,8 @@ static void isc_async_unbind(struct v4l2_async_notifier *notifier,v4l2_ctrl_handler_free (&isc->ctrls.handler); } -static struct isc_format *find_format_by_code (struct isc_device *isc, -unsigned int code,int *index) +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index) {struct isc_format *fmt = &isc->formats_list[0 ];unsigned int i;@@ -1749,6 +1760,7 @@ static struct isc_format *find_format_by_code(struct isc_device *isc,return NULL ; } +EXPORT_SYMBOL_GPL (isc_find_format_by_code);static int isc_formats_init (struct isc_device *isc) {@@ -1765,7 +1777,7 @@ static int isc_formats_init(struct isc_device *isc) NULL, &mbus_code)) { mbus_code.index++; - fmt =find_format_by_code (isc, mbus_code.code, &i); + fmt =isc_find_format_by_code (isc, mbus_code.code, &i);if (!fmt) {v4l2_warn (&isc->v4l2_dev,"Mbus code %x not supported\n" , mbus_code.code);@@ -1891,7 +1903,8 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) vdev->queue = q; vdev->lock = &isc->lock; vdev->ctrl_handler = &isc->ctrls.handler; - vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE ; + vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE | +V4L2_CAP_IO_MC ;video_set_drvdata (vdev, isc); ret =video_register_device (vdev,VFL_TYPE_VIDEO ,-1 );@@ -1901,8 +1914,18 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) goto isc_async_complete_err; } + ret =isc_scaler_link (isc); +if (ret <0 ) + goto isc_async_complete_unregister_device; + + ret =media_device_register (&isc->mdev); +if (ret <0 ) + goto isc_async_complete_unregister_device;return 0 ; +isc_async_complete_unregister_device: +video_unregister_device (vdev); + isc_async_complete_err:mutex_destroy (&isc->lock);return ret;@@ -1969,6 +1992,48 @@ int isc_pipeline_init(struct isc_device *isc) }EXPORT_SYMBOL_GPL (isc_pipeline_init);Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-base.c int isc_pipeline_init(struct isc_device *isc) · line 2001
Quoted 7 lines
drivers/media/platform/atmel/atmel-isc-base.c +7
-0+int isc_mc_init (struct isc_device *isc,u32 ver) +{ +const struct of_device_id *match; +int ret; + + isc->video_dev.entity.function =MEDIA_ENT_F_IO_V4L ; + isc->video_dev.entity.flags =MEDIA_ENT_FL_DEFAULT ;Should you set entity.ops.link_validate =
v4l2_subdev_link_validateto be able to have the media pipeline validated atmedia_pipeline_start()time ?Hi,
I am doing that in a subsequent patch. Things are not completely ready at this moment, because ISC still relies on the old mechanism to call
v4l2_subdev_callwithset_fmton the subdevice (which subsequent patch removes and adds checks to the link validate ).Quoted 6 lines
Quoted 1 line
I see.. the subsequent patches are not part of this series, right ?
No. Patch 7 does this. In that patch, the link validation is created and all the logic of format propagation is removed and reworked, and creates the need for the
link_validatecall
Could you have also a look at that patch ?Ah ups, 7 was not in my inbox with the rest of the series. Just noticed. I had even reviewed the previous version, I should have remembered
link_validatewas added laterQuoted 1 line
Quoted 1 line
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
I think patches 1, 2, 4, 5 and 6 from this series do not depend on the introduction of media controller or the scaler, right ? (I'm not sure if the DTS patches do).
Do DTS patches depend on media-controller plumbing ?
I think not, but, I have not tested them without the media controller.
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
If that's the case, would it make sense to to fast-track them (as some of them are fixes) and then on top plumb the media controller infrastructure with this patch and the ones you have been mentioning ?
It would make sense.
Once the other discussion with Hans about the stop state it might make sense to fastrack the first part of the series ?
The other three patches (4,5,6) are quite independent and can go faster.
Quoted 149 lines
Quoted 139 lines
Quoted 119 lines
Quoted 119 lines
Quoted 79 lines
Quoted 47 lines
drivers/media/platform/atmel/atmel-isc-base.c +35
-0+ isc->pads[ISC_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + + ret =media_entity_pads_init (&isc->video_dev.entity,ISC_PADS_NUM , + isc->pads); +if (ret <0 ) { +dev_err (isc->dev,"media entity init failed\n" ); +return ret; + } + + isc->mdev.dev = isc->dev; + + match =of_match_node (isc->dev->driver->of_match_table, + isc->dev->of_node); + +strscpy (isc->mdev.driver_name,KBUILD_MODNAME , +sizeof (isc->mdev.driver_name)); +strscpy (isc->mdev.model, match->compatible,sizeof (isc->mdev.model)); +snprintf (isc->mdev.bus_info,sizeof (isc->mdev.bus_info),"platform:%s" , + isc->v4l2_dev.name); + isc->mdev.hw_revision = ver; + +media_device_init (&isc->mdev); + + isc->v4l2_dev.mdev = &isc->mdev; + +return isc_scaler_init (isc); +} +EXPORT_SYMBOL_GPL (isc_mc_init); + +void isc_mc_cleanup (struct isc_device *isc) +{ +media_entity_cleanup (&isc->video_dev.entity); +} +EXPORT_SYMBOL_GPL (isc_mc_cleanup); +/* regmap configuration */ #define ATMEL_ISC_REG_MAX 0xd5cconst struct regmap_config isc_regmap_config = {drivers/media/platform/atmel/atmel-isc-scaler.c new file +0
-0drivers/media/platform/atmel/atmel-isc-scaler.c line 5
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0@@ -0,0 +1,245 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Microchip Image Sensor Controller (ISC) Scaler entity support +* +* Copyright (C) 2021 Microchip Technology, Inc. Time flies! It's 2022 already :)
Quoted 24 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +24
-0+* +* Author: Eugen Hristev <[email protected]> +* +*/ + +#include <media/media-device.h> +#include <media/media-entity.h> +#include <media/v4l2-device.h> +#include <media/v4l2-subdev.h> + +#include "atmel-isc-regs.h" +#include "atmel-isc.h" + +static int isc_scaler_get_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *format) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; + +if (format->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + format->pad); + format->format = *v4l2_try_fmt;Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 34
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; + } + + format->format = isc->scaler_format;
isc->scaler_formatis only used inside this file if I'm not mistaken. I wonder why it lives in theisc_devicestruct.
isc_deviceis a placeholder for all isc things.I would not create a separate struct in this file that would have to be allocated etc... just for one two things. So I preferred to have it in the same place as all the other things.
Quoted 24 lines
Quoted 12 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +11
-0+ +return 0 ; +} + +static int isc_scaler_set_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *req_fmt) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; +struct isc_format *fmt;drivers/media/platform/atmel/atmel-isc-scaler.c line 55
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+unsigned int i; + +if (req_fmt->pad ==ISC_SCALER_PAD_SOURCE ) +v4l_bound_align_image + (&req_fmt->format.width,16 , isc->max_width,0 , + &req_fmt->format.height,16 , isc->max_height,0 ,0 ); +else +v4l_bound_align_image + (&req_fmt->format.width,16 ,10000 ,0 , + &req_fmt->format.height,16 ,10000 ,0 ,0 );Where does 10000 come from ?
Quoted 3 lines
Quoted 1 line
It's a random number. Do you have any suggestion for a better one ?
An actual HW limit ? :)
There is no hardware limit. The ISC just stops sampling pixels once the crop limit is reached. The element that generates pixels could go on forever , or until the next HBLANK/VBLANK. So what limit could I place here ?
The cropping is mandatory, otherwise there could be an overflow w.r.t. the buffer size if the ISC would sample more pixels than the software expects it to.Quoted 8 lines
Quoted 7 lines
Maybe this would be much more clear with my comments below (where I will explain what the isc scaler does )
In short, it allows the other entity in the link to 'sink' a huge format into this 'scaler' . Because the scaler can crop anything basically down to the biggest format size the ISC could handle.Quoted 9 lines
Quoted 1 line
doesn't it have any documented input size limit ?
As stated above, ISC handles a pixel stream and stores it in an internal SRAM. ISC stops sampling when this SRAM is full or when the cropping limit is reached.
After storing the pixels in the SRAM (which is limited to the ISC maximum frame size), the ISC will work on the pixels and then DMA the frame to another RAM.Quoted 4 lines
I understand...
So whatever the input size is, only the first n-th bytes are actually dumped to memory, the following ones are discarded.
Yes ! That's the behavior.
Quoted 3 lines
So the crop is always applied on the top-leftmost corner as a consequence, right ?
Right. Pixels come in order. The ISC does not really know how many pixels are coming until its memory is full (or counting until the crop limit is reached). After that it stops sampling pixels, and waits for vblank to finish the frame.
Quoted 8 lines
I understand why you had difficulties picking a default :)
This is not easy to handle, as the scaler doesn't actually applies a crop but rather stops capturing after the destination buffer has been saturated. So the limiting factor is the full image size, not one of its dimensions, like in example the line length. This makes it hard to adjust cropping to something meaningful for most use cases.
I didn't know how to name this except crop. It 'crops out' what does not fit in the internal memory of the ISC
Quoted 7 lines
In your below example you suggest that an image size of (800000 x 10) would theoretically be captured correctly. What if the input is of size (800000 x 10 + 1). What size would you set the crop rectangle to ?
I wouldn't however be too much concerned for the moment, but I would record this in a comment ?
I discussed a bit more with designer and looked in datasheet. It looks like the horizontal line length is not a hard limit, but it makes the ISC internal ISP stop working, so I would keep the horizontal limit as well for now, unless an important use case (I am coming back to that). The vertical limit looks to be a hard requirement.
So in short, we have to impose a vertical limit of 2464 , we cannot have more lines than these. So if the horizontal is 100 pixels e.g., we can capture only 100x2464 .
In practice we could capture [16,3264] on the horizontal, and [16,2464] on the vertical, and in theory horizontal could go more like [16, 4000], but we have to limit the vertical, and without the internal ISP. So I would not haphazardly go in this direction. Let's stick to the datasheet maximum frame 3264x2464 .
According to this , I changed the frame size of the ISC to be a continuous size from [16,16] to [3264,2464]
About that use case that I talked about earlier, we have one sensor , imx274, which is more wide, and outputs something like 3800x2100 . With the cropping and limitations we have in place now, this will be limited to 3264x2100 , which is in line with datasheet and what the ISCQuoted 8 lines
supports now.
As in theory we could capture 3800x2100 as long as the SRAM doesn't get full, for now I would like to avoid capturing this size (and disable the internal ISP), until we have someone really asking for such kind of capture and with some good reasons.
In short I would like to have the crop rectangle to 3264x2464 and the bounds would be limited to the incoming frame but no H>3264 and no V>2464. Does that make sense ?
Quoted 4 lines
Just to make sure I got it: is the ISC actually able to crop on the line length ? If you apply a 3264 limit, will the last 536 pixels of each line will be dropped if the input frame is 3800x2100 as your above example ?
Yes. Last 536 pixels will be dropped.
After the 3264 limit, no more pixels are being sampled until HBLANK.
Quoted 39 lines
Or rather, is the SRAM size limit the issue and the lines gets captured in full lenght no matter what, until the buffer gets filled up and all the successive pixels dropped.
In other words, do your cropped image look like:
3264 +--------------------------------X-----------+ 3800 |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | +--------------------------------------------+ 2100Or rather
3264 +--------------------------------X-----------+ 3800 |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxx| | | | +--------------------------------------------+ 2100
Quoted 2 lines
I assume the first, as otherwise you wouldn't be able to interpret the image as 3264x2100.
Yes, the first.
Quoted 6 lines
I'm asking because you had the un-cropped 3280x2464 size as the ISC source pad size in the paste of the media graph you provided below, which might mean the output image is actually full size in lines lenght and it's maybe missing pixels at the end, in case the ISC operates as in the above second case.
No, the image won't be padded with blanks on the right side. Actually this cannot happen due to a very big reason: the frame is being written to the main memory by a DMA controller. This DMA controller will write all the lines at consecutive addresses, and it will write exactly how many pixels are being captured by the module from the ISC named 'parallel front end' which does the cropping. Thus, the DMA will never output empty padding bytes at the end of each line, and it will not add a certain stride to each line if not instructed to. In fact the DMA can add a stride offset for the address of each line, which can be used for composing or copying the frame into another frame, but this is a different story. ( I tried this for fun, but not implemented in the driver at the moment)
The cropping is also mandatory even if the captured frame is less than the maximum size, because the ISC has no configuration related to frame size in fact. The parallel front end captures pixels and the dma engine will write them to memory. There is no control over this except the cropping.
So for example if the software expects 640x480 but somehow someone sends more pixels, it can lead to buffer overflow if the parallel front end is not configured to ignore(crop) anything beyond 640x480 .
With this being said, or having an understanding about the sink pad of the ISC, how should I configure the sink frame size ?
Is there any possibility to avoid limits on the frame size and have it arbitrary ?
I am 99% sure that v4l2-compliance will not be happy if there is no limit on the frame on entity pads (hence, I bound-aligned the frame to 16x16 up to 10k x 10k )
Quoted 752 lines
Quoted 752 lines
Quoted 267 lines
Quoted 101 lines
Quoted 73 lines
Quoted 17 lines
Quoted 5 lines
Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 63
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +8
-0+ + req_fmt->format.colorspace =V4L2_COLORSPACE_SRGB ; + req_fmt->format.field =V4L2_FIELD_NONE ; + req_fmt->format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + req_fmt->format.quantization =V4L2_QUANTIZATION_DEFAULT ; + req_fmt->format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + fmt =isc_find_format_by_code (isc, req_fmt->format.code, &i);So you rely on the isc format list for the scaler as well ? I think it's fine as long as they are identical
Quoted 3 lines
Yes, the scaler is kind of a dummy entity , but it's required by the media controller validation of the pipeline.
Quoted 44 lines
Quoted 35 lines
Quoted 18 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +17
-0+ +if (!fmt) + fmt = &isc->formats_list[0 ]; + + req_fmt->format.code = fmt->mbus_code; + +if (req_fmt->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + req_fmt->pad); + *v4l2_try_fmt = req_fmt->format; +/* Trying on the pad sink makes the source sink change too */ +if (req_fmt->pad ==ISC_SCALER_PAD_SINK ) { + v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state, +ISC_SCALER_PAD_SOURCE ); + *v4l2_try_fmt = req_fmt->format; +drivers/media/platform/atmel/atmel-isc-scaler.c line 90
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+v4l_bound_align_image (&v4l2_try_fmt->width, +16 , isc->max_width,0 , + &v4l2_try_fmt->height, +16 , isc->max_height,0 ,0 ); + } +/* if we are just trying, we are done */ +return 0 ; + } + + isc->scaler_format = req_fmt->format;No per-pad format but a global
scaler_format? How do you configure scaling ?Actually, I would like to know more about the scaler device capabilities. What functions can this IP perform ? Does it do cropping, can it also do (down)scaling or even composition ?
Quoted 8 lines
I think it is worth to read
https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-subdev.html#v4l2-subdev-selections
where it is reported how cropping and scaling are implemented by using the selection API.Figure 4.5 is particularly helpful to explain the simple crop case, for which you need to implement support to by adding
s_selectionon the sink padTGT_CROPtarget.Quoted 5 lines
The scaler is kind of a dummy entity.
The problem is that different subdevices cannot be connected directly to a v4l entity, because the ISC does cropping on the incoming frame. The ISC has a limit on the total number of pixels per frame, thus it will do a crop.Quoted 6 lines
What is this limit related to ? I don't know the ISC internals, but is the limitation due to the overall image size, or maybe it's due to some line buffers being limited in the size they can transfer to memory ? Is it a limitation on the size of the image in pixels, or is it due to a limitation of the processing bandwidth on the input interface and thus depends on the actual size of the image in bytes ?
Limitation is both on line size and full image size. From my understanding, is that the internal SRAM is limited (8 Mpixels with a small safe buffer on top in the case of sama7g5 )
This SRAM could be used as 800000x10 for example, or 3200x2464 for another example. However, I am not sure if a real line of 800,000 pixels make sense, or if there is another internal mechanism in the ISC that stops it.
In any case, the ISC is cropping the incoming frame from aaaXbbb up to the maximum frame that it can handleQuoted 10 lines
Quoted 6 lines
So, if I allow e.g. an entity that sends 3280x2464 , when the ISC can only handle 3264x2464 , then we have two choices:
1/ crop it and output the 3264x2464 -> cannot, because userspace expects 3280x2464, and the whole pipeline fails to configure, there is a mismatch between/dev/videooutput and what the whole pipeline producesI'm confused in first place. If the ISC cannot output anything larger than 3264x2464, then if userspace sets a 3280x2464 size on the ISC, this should be adjusted to 3264x2464.
Quoted 9 lines
yes, that is true. However, without an entity that says 'I am doing cropping', the media pipeline will fail, because previous entity will output 3280x2464 .
The conclusion to create a scaler entitity was done during my last year discussions on the IRC. I could not find a way to perform this cropping at the v4l entity side, and one reason is that the v4l entity does not support what I needed , or at least that was my understanding. You claim that the v4l entity could perform the cropping in the same entity ?I thought so, but I haven't tried to be honest
Quoted 10 lines
Let's assume for a minute that it could do this. Even so, I would prefer to have a separate scaler entity to do this , for several reasons:
-> I would like to be able to crop the frame arbitrarily if I want to, and the scaler would allow me to do this easier
-> it would be more obvious that the frame is cropped by having this entity there
-> in fact some of the ISC versions really have a down scaler, that I have not implemented yet. So it would be useful to already have this in place to be able to scale down with it at a later time.I think this last reason is enough to have an entity plumbed in already
Quoted 8 lines
Quoted 8 lines
Or is the ISC limitation on the -input- side ?
Quoted 3 lines
2/ deny the link, and ask the subdevice to produce the 3264x2464 which the ISC can handle at most -> cannot , because the sensor let's say can *only* produce 3280x2464 and *any* other frame size returns an error.
Quoted 6 lines
Quoted 1 line
buggy sensor I would say :)
I cannot tell this to the customers, and I cannot fail the whole pipeline because the sensor does not crop 16 pixels, when in fact I can do this very easily on the ISC side.
Fair enough :)
Quoted 85 lines
Quoted 11 lines
Quoted 7 lines
The only solution to make both worlds happy, is to have a dummy entity called 'scaler' which in fact now it only performs a simple crop. It accepts any frame size at input (hence the 10000x10000 which you saw earlier), and outputs at most 3264x2464 the
isc_max_heightandisc_max_width.I have a maybe dumb question: can the cropping operation be modeled by applying a
TGT_CROPrectangle (whose_BOUNDsize is 3280x2464) to theatmel_isc_commonentity sink pad and avoid the scaler completely ?I am not sure. This is what I wanted initially. But then I thought about the three reasons stated above for the use of the scaler entity, and discussed with folks on the IRC, and come up with the solution for the scaler entity
Quoted 68 lines
Quoted 67 lines
So to answer your question, the isc scaler is a software model for a simple cropping, that would make media controller happy, and capture software happy.
Here it how it looks :
Device topology
- entity 1:atmel_isc_scaler(2 pads, 2 links)type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev0 pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb crop.bounds:(0,0)/3264x2464 crop:(0,0)/3264x2464] <- "csi2dc":1 [ENABLED,IMMUTABLE] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] -> "atmel_isc_common":0 [ENABLED,IMMUTABLE]- entity 4: csi2dc (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev1 pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] <- "dw-csi.0":1 [ENABLED] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] -> "atmel_isc_scaler":0 [ENABLED,IMMUTABLE]- entity 7: dw-csi.0 (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev2 pad0: Sink [fmt:SRGGB10_1X10/3280x2464] <- "imx219 1-0010":0 [ENABLED] pad1: Source [fmt:SRGGB10_1X10/3280x2464] -> "csi2dc":0 [ENABLED]- entity 12: imx219 1-0010 (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0 device node name/dev/v4l-subdev3
pad0: Source
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb
xfer:srgb ycbcr:601 quantization:full-rangecrop.bounds:(8,8)/3280x2464 crop:(8,8)/3280x2464] -> "dw-csi.0":0 [ENABLED]- entity 24:
atmel_isc_common(1 pad, 1 link)
type Node subtype V4L flags 1
device node name/dev/video0
pad0: Sink
<- "atmel_isc_scaler":1 [ENABLED,IMMUTABLE]Scaler does this one cute little thing :
pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb crop.bounds:(0,0)/3264x2464 crop:(0,0)/3264x2464] <- "csi2dc":1 [ENABLED,IMMUTABLE] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb]Quoted 7 lines
Quoted 3 lines
Shouldn't this be 3264x2464 as that's what the entity outputs after the cropping ? And shouldn't then be 3264x2464 the size output from the video device too ?
That's right.
I don't know why the source format for the scaler is still 3280x2464. Maybe there is a bug ong_fmtfor it.. have to check it.Thanks, I think this should be fixed ten
Quoted 17 lines
Anyway, the video format is like this :
# v4l2-ctl --get-fmt-video Format Video Capture: Width/Height : 3264/2464 Pixel Format : 'RGBP' (16-bit RGB 5-6-5) Field : None Bytes per Line : 6528 Size Image : 16084992 Colorspace : sRGB Transfer Function : Default (maps to sRGB) YCbCr/HSV Encoding: Default (maps to ITU-R 601) Quantization : Default (maps to Full Range) Flags : #Quoted 9 lines
and initializing the pipeline looks fine from user perspective. The scaler is in fact the solution to make this pipeline work with libcamera.
I found this cropping to be an issue in media controller when trying it with libcamera. Otherwise, the other user space apps which I was using never complained that anything was wrong
Libcamera simply refuses to acknowledge the pipeline if the video output is 3264x2464 but there is no entity that changes the format from 3280x2464 downQuoted 2 lines
Not sure why libcamera should play a role there. Isn't it the media pipeline validation that complains and returns an
-EPIPE?I tried this before the
media_pipeline_start.
Perhaps libcamera was doing some similar checks, anyway, it was complaining and not adding the stream, and reported no usable streams/cameraOnce I will rework some of the things according to your review I will run libcamera again to see what it complains about
Quoted 474 lines
Quoted 93 lines
Quoted 73 lines
Quoted 63 lines
-> "atmel_isc_common":0 [ENABLED,IMMUTABLE]Which is what we needed.
Quoted 19 lines
Quoted 6 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; +} + +static int isc_scaler_enum_mbus_code(struct v4l2_subdev *sd,drivers/media/platform/atmel/atmel-isc-scaler.c line 105
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+struct v4l2_subdev_state *sd_state, + struct v4l2_subdev_mbus_code_enum *code) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +int supported_index =0 ; +int i; + +for (i =0 ; i < isc->formats_list_size; i++) { +if (!isc->formats_list[i].sd_support) +continue ;The
sd_supportflag still doesn't click in my head.Quoted 2 lines
Shouldn't the enumeration of available formats on the scaler do not depend on the sensor supproted formats ?
You're right. I will have to check it again.
Quoted 30 lines
Quoted 16 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +15
-0+if (supported_index == code->index) { + code->code = isc->formats_list[i].mbus_code; +return 0 ; + } + supported_index++; + } + +return -EINVAL ; +} + +static int isc_scaler_g_sel (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_selection *sel) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd);drivers/media/platform/atmel/atmel-isc-scaler.c line 130
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +if (sel->pad ==ISC_SCALER_PAD_SOURCE ) +return -EINVAL ; + +if (sel->target !=V4L2_SEL_TGT_CROP_BOUNDS && + sel->target !=V4L2_SEL_TGT_CROP ) +return -EINVAL ; + + sel->r.height = isc->max_height; + sel->r.width = isc->max_width;The
CROP_BOUNDSshould be set to the same size as the sink pad image format, as it represents the maximum valid crop rectangle.Quoted 9 lines
Quoted 3 lines
TGT_CROPshould report the configured crop rectangle which can be intiialized to the same size asCROP_BOUNDS, if my understanding of the spec is correctSo you would like to have this differentiated, and report the
CROP_BOUNDSto whatever is on the sink pad, and theTGT_CROPto what is reported now, the maximum size of the ISC frame .
My understanding is correct ?Quoted 6 lines
I didn't know you have an HW limitation, so your
_BOUNDSis not the input image size but rather 3264x2464 ( ==max_widthxmax_height).What I meant is that
_BOUNDSshould report the maximum rectangle size that can be applied to the_CROPtarget. In you case you have an HW limitation 3264x2464 and that's the largest rectangle you can apply.So the CROP should be at 3264x2464
Quoted 2 lines
TGT_CROPcan be initialized to the same as_BOUND, but if you implements_selectionit should report what has been there applied.and BOUND to actual frame size ?
Quoted 2 lines
But as you don't implement
s_selectionyet, I think this is fine for now. Maybe a little comment ?It could also be the case where e.g. the sensor is outputting 1920x1080, in this case the scaler would do nothing.
CROP is still 3264x2464 and BOUND in this case is 1920x1080 ? If the sensor is outputting 1920x1080, this format comes directly from the sensor (the sensor is cropping it from 3280x2464 or not... it's the sensor's problem)Quoted 1 line
Quoted 4 lines
Quoted 2 lines
Also, is
set->rzeroed by the framework before getting here ? Otherwise you should set r.left and r.top to 0If these are redundant, no problem to remove them
I was actually confused. I was suggesting you to add...
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 133
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +3
-0+ + sel->r.left =0 ; + sel->r.top =0 ;These ^
So nothing to change. Sorry I've missed them
Thanks
jQuoted 360 lines
Quoted 359 lines
Quoted 49 lines
Quoted 40 lines
Quoted 12 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +12
-0+ +return 0 ; +} + +static int isc_scaler_init_cfg (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state) +{ +struct v4l2_mbus_framefmt *v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state,0 ); + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); + + *v4l2_try_fmt = isc->scaler_format;drivers/media/platform/atmel/atmel-isc-scaler.c line 155
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +return 0 ; +} + +static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = { + .enum_mbus_code = isc_scaler_enum_mbus_code, + .set_fmt = isc_scaler_set_fmt, + .get_fmt = isc_scaler_get_fmt, + .get_selection = isc_scaler_g_sel, + .init_cfg = isc_scaler_init_cfg,.link_validate =
v4l2_subdev_link_validate_default,To have the formats at the end of links that point to this entity validated (I think the framework already calls it if not set though, please check v4l2-subdev.c:v4l2_subdev_link_validate())
Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +11
-0+}; + +static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = { + .pad = &isc_scaler_pad_ops, +}; + +int isc_scaler_init (struct isc_device *isc) +{ +int ret; + +v4l2_subdev_init (&isc->scaler_sd, &xisc_scaler_subdev_ops);Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 171
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ + isc->scaler_sd.owner =THIS_MODULE ; + isc->scaler_sd.dev = isc->dev; +snprintf (isc->scaler_sd.name,sizeof (isc->scaler_sd.name), +"atmel_isc_scaler" );I would drop 'atmel' for brevity, unless other entities have this prefix set already
Quoted 3 lines
The v4l entity takes it's name from the module name, which is
atmel_isc_common, so I thought to keep the prefixAck!
Quoted 189 lines
Quoted 70 lines
Quoted 70 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +69
-0+ + isc->scaler_sd.flags |=V4L2_SUBDEV_FL_HAS_DEVNODE ; + isc->scaler_sd.entity.function =MEDIA_ENT_F_PROC_VIDEO_SCALER ; + isc->scaler_pads[ISC_SCALER_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + isc->scaler_pads[ISC_SCALER_PAD_SOURCE ].flags =MEDIA_PAD_FL_SOURCE ; + + isc->scaler_format.height = isc->max_height; + isc->scaler_format.width = isc->max_width; + isc->scaler_format.code = isc->formats_list[0 ].mbus_code; + isc->scaler_format.colorspace =V4L2_COLORSPACE_SRGB ; + isc->scaler_format.field =V4L2_FIELD_NONE ; + isc->scaler_format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + isc->scaler_format.quantization =V4L2_QUANTIZATION_DEFAULT ; + isc->scaler_format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + ret =media_entity_pads_init (&isc->scaler_sd.entity, +ISC_SCALER_PADS_NUM , + isc->scaler_pads); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd media entity init failed\n" ); +return ret; + } + ret =v4l2_device_register_subdev (&isc->v4l2_dev, &isc->scaler_sd); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd failed to register subdev\n" ); +return ret; + } + +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_init); + +int isc_scaler_link (struct isc_device *isc) +{ +int ret; + + ret =media_create_pad_link (&isc->current_subdev->sd->entity, + isc->remote_pad, &isc->scaler_sd.entity, +ISC_SCALER_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->current_subdev->sd->entity.name, + isc->scaler_sd.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , + isc->current_subdev->sd->name, isc->remote_pad); + + ret =media_create_pad_link (&isc->scaler_sd.entity, +ISC_SCALER_PAD_SOURCE , + &isc->video_dev.entity,ISC_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->scaler_sd.entity.name, + isc->video_dev.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , isc->scaler_sd.name, +ISC_SCALER_PAD_SOURCE );Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 244
Quoted 4 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +4
-0+ +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_link);From the DT graph point of view, the ISC appears as a single block
with an CSI-2 input port. It is then in charge of parsing the .dts and add to its notifier the image sensor remote subdevice.Actually it's only a parallel port.
And it can be connected either to a sensor directly or to the csi2dc bridge. (the csi2dc bridge can be connected to a CSI-2 receiver)Quoted 4 lines
When the sensor subdev registers and the ISC notifier completes, the scaler entity which was initialized at
isc_probe()time is linked in between the ISC and the image sensor immutably.yes, because this cannot change at runtime, and usually, it can't change without altering the board hardware. (at least this is what my understanding of immutability is )
Quoted 9 lines
I think it is fine for now, but I wonder if you plan to plumb more components between the ISC video node and the sensor, if it's not worth changing the DT bindings and their parsing logic to separate the CSI-2 receiver from the ISC, whcih can create its media graph at probe time and have the CSI-2 receiver entity as downstream remote. I think I need to get to know the ISC better to really have an idea. For now, this seems ok to me, but please check with maintainers if this is fine with them.
In the XISC case (sama7g5-isc), the csi2dc receiver is the subdev (so, the remote subdev).
The ISC will register a scaler, and connect the subdev to this scaler first, and then, the scaler to the isc itself (the v4l entity).Quoted 45 lines
Quoted 45 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +1
-0+drivers/media/platform/atmel/atmel-isc.h +21
-0@@ -183,6 +183,17 @@ struct isc_reg_offsets {u32 his_entry; }; +enum isc_mc_pads { +ISC_PAD_SINK =0 , +ISC_PADS_NUM =1 , +}; + +enum isc_scaler_pads { +ISC_SCALER_PAD_SINK =0 , +ISC_SCALER_PAD_SOURCE =1 , +ISC_SCALER_PADS_NUM =2 , +}; +/* * struct isc_device - ISC device driver data/config struct * @regmap: Register map @@ -257,6 +268,12 @@ struct isc_reg_offsets {* be used as an input to the controller * @controller_formats_size: size of controller_formats array * @formats_list_size: size of formats_list array +* @pads: media controller pads for isc video entity +* @mdev: media device that is registered by the isc +* @remote_pad: remote pad on the connected subdevice +* @scaler_sd: subdevice for the scaler that isc registers +* @scaler_pads: media controller pads for the scaler subdevice +* @scaler_format: current format for the scaler subdevice */ struct isc_device {struct regmap *regmap;@@ -344,6 +361,19 @@ struct isc_device {struct isc_format *formats_list;u32 controller_formats_size;u32 formats_list_size; + +struct { +struct media_pad pads[ISC_PADS_NUM ]; +struct media_device mdev;Quoted 8 lines
drivers/media/platform/atmel/atmel-isc.h struct isc_device { · line 375
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc.h +8
-0+ +u32 remote_pad; + }; + +struct { +struct v4l2_subdev scaler_sd; +struct media_pad scaler_pads[ISC_SCALER_PADS_NUM ]; +struct v4l2_mbus_framefmt scaler_format;Here are the scaler stuff which I added, in the same bulk struct for the whole isc device
Quoted 24 lines
Quoted 24 lines
drivers/media/platform/atmel/atmel-isc.h +8
-0+ }; };extern const struct regmap_config isc_regmap_config;@@ -355,4 +385,11 @@ int isc_clk_init(struct isc_device *isc);void isc_subdev_cleanup (struct isc_device *isc);void isc_clk_cleanup (struct isc_device *isc); +int isc_scaler_link (struct isc_device *isc); +int isc_scaler_init (struct isc_device *isc); +int isc_mc_init (struct isc_device *isc,u32 ver); +void isc_mc_cleanup (struct isc_device *isc); + +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index);#endif drivers/media/platform/atmel/atmel-sama5d2-isc.c +0
-0@@ -553,6 +553,12 @@ static int atmel_isc_probe(struct platform_device *pdev)break ; }Quoted 8 lines
Quoted 5 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c static int atmel_isc_probe(struct platform_device *pdev) · line 556
Quoted 1 line
drivers/media/platform/atmel/atmel-sama5d2-isc.c +1
-0+regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);I am surprised you can read a register before
runtime_pmis intialized!Actually I can read any register after the moment of starting the clock on the hardware block.
Ah right then
Quoted 3 lines
Maybe the starting and stopping of the clock needs to be moved to the
runtime_pmcalls, but this is another story, not related to the media controller.It's fine, but maybe worth recording with a todo ?
Thanks
jQuoted 95 lines
I moved this line because I had to pass the version to the
isc_mc_initcallQuoted 2 lines
Thanks
jThanks for reviewing !
EugenQuoted 87 lines
Quoted 87 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c +11
-2+ + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev);@@ -562,7 +568,7 @@ static int atmel_isc_probe(struct platform_device *pdev) ret =clk_prepare_enable (isc->ispck);if (ret) {dev_err (dev,"failed to enable ispck: %d\n" , ret); - goto cleanup_subdev; + goto isc_probe_mc_init_err; }/* ispck should be greater or equal to hclock */ @@ -572,7 +578,6 @@ static int atmel_isc_probe(struct platform_device *pdev) goto unprepare_clk; } -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip ISC version %x\n" , ver);return 0 ;@@ -580,6 +585,9 @@ static int atmel_isc_probe(struct platform_device *pdev) unprepare_clk:clk_disable_unprepare (isc->ispck); +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -600,6 +608,8 @@ static int atmel_isc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev);drivers/media/platform/atmel/atmel-sama7g5-isc.c +11
-2@@ -547,15 +547,23 @@ static int microchip_xisc_probe(struct platform_device *pdev)break ; } +regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver); + + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev); -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip XISC version %x\n" , ver);return 0 ; +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -576,6 +584,8 @@ static int microchip_xisc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev); --2.25.1
Hi Eugen
Quoted 475 lines · On Fri, Feb 11, 2022 at 10:32:43AM +0000, [email protected] wrote:
Quoted 416 lines · Jacopo Mondi
Hi Eugen
Quoted 404 lines · On Wed, Feb 09, 2022 at 06:59:37AM +0000, [email protected] wrote:
Quoted 178 lines · Jacopo Mondi
Hi Eugen
Quoted 163 lines · On Tue, Feb 08, 2022 at 10:33:31AM +0000, [email protected] wrote:
Quoted 162 lines · Jacopo Mondi
Hi Eugen
Quoted 158 lines · On Mon, Feb 07, 2022 at 01:40:31PM +0000, [email protected] wrote:
Quoted 138 lines · Jacopo Mondi
Hi Eugen
Quoted 135 lines · Eugen Hristev
Implement the support for media-controller.
This means that the capabilities of the driver have changed and now it also advertises theIO_MC.
The driver will register it's media device, and add the video entity to this media device. The subdevices are registered to the same media device. The ISC will have a base entity which is auto-detected asatmel_isc_base. It will also register a subdevice that allows cropping of the incoming frame to the maximum frame size supported by the ISC.
The ISC will create a link between the subdevice that is asynchronously registered and theatmel_isc_scalerentity.
Then, theatmel_isc_scalerandatmel_isc_baseare connected through another link.Signed-off-by Eugen Hristev <[email protected]>---
Changes in v4:
As suggested by Jacopo:
- renamedatmel_isc_mctoatmel_isc_scaler.c
- moved init_mc/clean_mc toisc_basefileChanges in v2:
- implement try formatsdrivers/media/platform/atmel/Makefile | 2 +- drivers/media/platform/atmel/atmel-isc-base.c | 73 +++++- .../media/platform/atmel/atmel-isc-scaler.c | 245 ++++++++++++++++++ drivers/media/platform/atmel/atmel-isc.h | 37 +++ .../media/platform/atmel/atmel-sama5d2-isc.c | 14 +- .../media/platform/atmel/atmel-sama7g5-isc.c | 12 +- 6 files changed, 375 insertions(+), 8 deletions(-)create mode 100644
drivers/media/platform/atmel/atmel-isc-scaler.cdrivers/media/platform/atmel/Makefile +1
-1@@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only atmel-isc-objs = atmel-sama5d2-isc.o atmel-xisc-objs = atmel-sama7g5-isc.o -atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o +atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o atmel-isc-scaler.o obj-$(CONFIG_VIDEO_ATMEL_ISI) += atmel-isi.o obj-$(CONFIG_VIDEO_ATMEL_ISC_BASE) += atmel-isc-common.odrivers/media/platform/atmel/atmel-isc-base.c +27
-4@@ -1710,6 +1710,7 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, struct isc_device, v4l2_dev);struct isc_subdev_entity *subdev_entity = container_of(notifier, struct isc_subdev_entity, notifier); +int pad;if (video_is_registered (&isc->video_dev)) {v4l2_err (&isc->v4l2_dev,"only supports one sub-device.\n" );@@ -1718,6 +1719,16 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, subdev_entity->sd = subdev; + pad =media_entity_get_fwnode_pad (&subdev->entity, asd->match.fwnode, +MEDIA_PAD_FL_SOURCE ); +if (pad <0 ) { +v4l2_err (&isc->v4l2_dev,"failed to find pad for %s\n" , + subdev->name); +return pad; + } + + isc->remote_pad = pad; +return 0 ; }@@ -1732,8 +1743,8 @@ static void isc_async_unbind(struct v4l2_async_notifier *notifier,v4l2_ctrl_handler_free (&isc->ctrls.handler); } -static struct isc_format *find_format_by_code (struct isc_device *isc, -unsigned int code,int *index) +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index) {struct isc_format *fmt = &isc->formats_list[0 ];unsigned int i;@@ -1749,6 +1760,7 @@ static struct isc_format *find_format_by_code(struct isc_device *isc,return NULL ; } +EXPORT_SYMBOL_GPL (isc_find_format_by_code);static int isc_formats_init (struct isc_device *isc) {@@ -1765,7 +1777,7 @@ static int isc_formats_init(struct isc_device *isc) NULL, &mbus_code)) { mbus_code.index++; - fmt =find_format_by_code (isc, mbus_code.code, &i); + fmt =isc_find_format_by_code (isc, mbus_code.code, &i);if (!fmt) {v4l2_warn (&isc->v4l2_dev,"Mbus code %x not supported\n" , mbus_code.code);@@ -1891,7 +1903,8 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) vdev->queue = q; vdev->lock = &isc->lock; vdev->ctrl_handler = &isc->ctrls.handler; - vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE ; + vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE | +V4L2_CAP_IO_MC ;video_set_drvdata (vdev, isc); ret =video_register_device (vdev,VFL_TYPE_VIDEO ,-1 );@@ -1901,8 +1914,18 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) goto isc_async_complete_err; } + ret =isc_scaler_link (isc); +if (ret <0 ) + goto isc_async_complete_unregister_device; + + ret =media_device_register (&isc->mdev); +if (ret <0 ) + goto isc_async_complete_unregister_device;return 0 ; +isc_async_complete_unregister_device: +video_unregister_device (vdev); + isc_async_complete_err:mutex_destroy (&isc->lock);return ret;@@ -1969,6 +1992,48 @@ int isc_pipeline_init(struct isc_device *isc) }EXPORT_SYMBOL_GPL (isc_pipeline_init);Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-base.c int isc_pipeline_init(struct isc_device *isc) · line 2001
Quoted 7 lines
drivers/media/platform/atmel/atmel-isc-base.c +7
-0+int isc_mc_init (struct isc_device *isc,u32 ver) +{ +const struct of_device_id *match; +int ret; + + isc->video_dev.entity.function =MEDIA_ENT_F_IO_V4L ; + isc->video_dev.entity.flags =MEDIA_ENT_FL_DEFAULT ;Should you set entity.ops.link_validate =
v4l2_subdev_link_validateto be able to have the media pipeline validated atmedia_pipeline_start()time ?Hi,
I am doing that in a subsequent patch. Things are not completely ready at this moment, because ISC still relies on the old mechanism to call
v4l2_subdev_callwithset_fmton the subdevice (which subsequent patch removes and adds checks to the link validate ).Quoted 6 lines
Quoted 1 line
I see.. the subsequent patches are not part of this series, right ?
No. Patch 7 does this. In that patch, the link validation is created and all the logic of format propagation is removed and reworked, and creates the need for the
link_validatecall
Could you have also a look at that patch ?Ah ups, 7 was not in my inbox with the rest of the series. Just noticed. I had even reviewed the previous version, I should have remembered
link_validatewas added laterQuoted 1 line
Quoted 1 line
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
I think patches 1, 2, 4, 5 and 6 from this series do not depend on the introduction of media controller or the scaler, right ? (I'm not sure if the DTS patches do).
Do DTS patches depend on media-controller plumbing ?
I think not, but, I have not tested them without the media controller.
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
If that's the case, would it make sense to to fast-track them (as some of them are fixes) and then on top plumb the media controller infrastructure with this patch and the ones you have been mentioning ?
It would make sense.
Once the other discussion with Hans about the stop state it might make sense to fastrack the first part of the series ?
The other three patches (4,5,6) are quite independent and can go faster.
Quoted 149 lines
Quoted 139 lines
Quoted 119 lines
Quoted 119 lines
Quoted 79 lines
Quoted 47 lines
drivers/media/platform/atmel/atmel-isc-base.c +35
-0+ isc->pads[ISC_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + + ret =media_entity_pads_init (&isc->video_dev.entity,ISC_PADS_NUM , + isc->pads); +if (ret <0 ) { +dev_err (isc->dev,"media entity init failed\n" ); +return ret; + } + + isc->mdev.dev = isc->dev; + + match =of_match_node (isc->dev->driver->of_match_table, + isc->dev->of_node); + +strscpy (isc->mdev.driver_name,KBUILD_MODNAME , +sizeof (isc->mdev.driver_name)); +strscpy (isc->mdev.model, match->compatible,sizeof (isc->mdev.model)); +snprintf (isc->mdev.bus_info,sizeof (isc->mdev.bus_info),"platform:%s" , + isc->v4l2_dev.name); + isc->mdev.hw_revision = ver; + +media_device_init (&isc->mdev); + + isc->v4l2_dev.mdev = &isc->mdev; + +return isc_scaler_init (isc); +} +EXPORT_SYMBOL_GPL (isc_mc_init); + +void isc_mc_cleanup (struct isc_device *isc) +{ +media_entity_cleanup (&isc->video_dev.entity); +} +EXPORT_SYMBOL_GPL (isc_mc_cleanup); +/* regmap configuration */ #define ATMEL_ISC_REG_MAX 0xd5cconst struct regmap_config isc_regmap_config = {drivers/media/platform/atmel/atmel-isc-scaler.c new file +0
-0drivers/media/platform/atmel/atmel-isc-scaler.c line 5
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0@@ -0,0 +1,245 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Microchip Image Sensor Controller (ISC) Scaler entity support +* +* Copyright (C) 2021 Microchip Technology, Inc. Time flies! It's 2022 already :)
Quoted 24 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +24
-0+* +* Author: Eugen Hristev <[email protected]> +* +*/ + +#include <media/media-device.h> +#include <media/media-entity.h> +#include <media/v4l2-device.h> +#include <media/v4l2-subdev.h> + +#include "atmel-isc-regs.h" +#include "atmel-isc.h" + +static int isc_scaler_get_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *format) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; + +if (format->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + format->pad); + format->format = *v4l2_try_fmt;Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 34
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; + } + + format->format = isc->scaler_format;
isc->scaler_formatis only used inside this file if I'm not mistaken. I wonder why it lives in theisc_devicestruct.
isc_deviceis a placeholder for all isc things.I would not create a separate struct in this file that would have to be allocated etc... just for one two things. So I preferred to have it in the same place as all the other things.
Quoted 24 lines
Quoted 12 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +11
-0+ +return 0 ; +} + +static int isc_scaler_set_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *req_fmt) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; +struct isc_format *fmt;drivers/media/platform/atmel/atmel-isc-scaler.c line 55
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+unsigned int i; + +if (req_fmt->pad ==ISC_SCALER_PAD_SOURCE ) +v4l_bound_align_image + (&req_fmt->format.width,16 , isc->max_width,0 , + &req_fmt->format.height,16 , isc->max_height,0 ,0 ); +else +v4l_bound_align_image + (&req_fmt->format.width,16 ,10000 ,0 , + &req_fmt->format.height,16 ,10000 ,0 ,0 );Where does 10000 come from ?
Quoted 3 lines
Quoted 1 line
It's a random number. Do you have any suggestion for a better one ?
An actual HW limit ? :)
There is no hardware limit. The ISC just stops sampling pixels once the crop limit is reached. The element that generates pixels could go on forever , or until the next HBLANK/VBLANK. So what limit could I place here ?
The cropping is mandatory, otherwise there could be an overflow w.r.t. the buffer size if the ISC would sample more pixels than the software expects it to.Quoted 8 lines
Quoted 7 lines
Maybe this would be much more clear with my comments below (where I will explain what the isc scaler does )
In short, it allows the other entity in the link to 'sink' a huge format into this 'scaler' . Because the scaler can crop anything basically down to the biggest format size the ISC could handle.Quoted 9 lines
Quoted 1 line
doesn't it have any documented input size limit ?
As stated above, ISC handles a pixel stream and stores it in an internal SRAM. ISC stops sampling when this SRAM is full or when the cropping limit is reached.
After storing the pixels in the SRAM (which is limited to the ISC maximum frame size), the ISC will work on the pixels and then DMA the frame to another RAM.Quoted 4 lines
I understand...
So whatever the input size is, only the first n-th bytes are actually dumped to memory, the following ones are discarded.
Yes ! That's the behavior.
Quoted 3 lines
So the crop is always applied on the top-leftmost corner as a consequence, right ?
Right. Pixels come in order. The ISC does not really know how many pixels are coming until its memory is full (or counting until the crop limit is reached). After that it stops sampling pixels, and waits for vblank to finish the frame.
Quoted 8 lines
I understand why you had difficulties picking a default :)
This is not easy to handle, as the scaler doesn't actually applies a crop but rather stops capturing after the destination buffer has been saturated. So the limiting factor is the full image size, not one of its dimensions, like in example the line length. This makes it hard to adjust cropping to something meaningful for most use cases.
I didn't know how to name this except crop. It 'crops out' what does not fit in the internal memory of the ISC
Quoted 7 lines
In your below example you suggest that an image size of (800000 x 10) would theoretically be captured correctly. What if the input is of size (800000 x 10 + 1). What size would you set the crop rectangle to ?
I wouldn't however be too much concerned for the moment, but I would record this in a comment ?
I discussed a bit more with designer and looked in datasheet. It looks like the horizontal line length is not a hard limit, but it makes the ISC internal ISP stop working, so I would keep the horizontal limit as well for now, unless an important use case (I am coming back to that). The vertical limit looks to be a hard requirement.
So in short, we have to impose a vertical limit of 2464 , we cannot have more lines than these. So if the horizontal is 100 pixels e.g., we can capture only 100x2464 .
In practice we could capture [16,3264] on the horizontal, and [16,2464] on the vertical, and in theory horizontal could go more like [16, 4000], but we have to limit the vertical, and without the internal ISP. So I would not haphazardly go in this direction. Let's stick to the datasheet maximum frame 3264x2464 .
According to this , I changed the frame size of the ISC to be a continuous size from [16,16] to [3264,2464]
About that use case that I talked about earlier, we have one sensor , imx274, which is more wide, and outputs something like 3800x2100 . With the cropping and limitations we have in place now, this will be limited to 3264x2100 , which is in line with datasheet and what the ISCQuoted 8 lines
supports now.
As in theory we could capture 3800x2100 as long as the SRAM doesn't get full, for now I would like to avoid capturing this size (and disable the internal ISP), until we have someone really asking for such kind of capture and with some good reasons.
In short I would like to have the crop rectangle to 3264x2464 and the bounds would be limited to the incoming frame but no H>3264 and no V>2464. Does that make sense ?Quoted 4 lines
Just to make sure I got it: is the ISC actually able to crop on the line length ? If you apply a 3264 limit, will the last 536 pixels of each line will be dropped if the input frame is 3800x2100 as your above example ?
Yes. Last 536 pixels will be dropped.
After the 3264 limit, no more pixels are being sampled until HBLANK.Quoted 39 lines
Or rather, is the SRAM size limit the issue and the lines gets captured in full lenght no matter what, until the buffer gets filled up and all the successive pixels dropped.
In other words, do your cropped image look like:
3264 +--------------------------------X-----------+ 3800 |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | +--------------------------------------------+ 2100Or rather
3264 +--------------------------------X-----------+ 3800 |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxx| | | | +--------------------------------------------+ 2100Quoted 2 lines
I assume the first, as otherwise you wouldn't be able to interpret the image as 3264x2100.
Yes, the first.
Quoted 6 lines
I'm asking because you had the un-cropped 3280x2464 size as the ISC source pad size in the paste of the media graph you provided below, which might mean the output image is actually full size in lines lenght and it's maybe missing pixels at the end, in case the ISC operates as in the above second case.
Quoted 12 lines
No, the image won't be padded with blanks on the right side. Actually this cannot happen due to a very big reason: the frame is being written to the main memory by a DMA controller. This DMA controller will write all the lines at consecutive addresses, and it will write exactly how many pixels are being captured by the module from the ISC named 'parallel front end' which does the cropping. Thus, the DMA will never output empty padding bytes at the end of each line, and it will not add a certain stride to each line if not instructed to. In fact the DMA can add a stride offset for the address of each line, which can be used for composing or copying the frame into another frame, but this is a different story. ( I tried this for fun, but not implemented in the driver at the moment)
Oh that will really be interesting and it's indeed an argument for hainvg a scaler subdevice indeed
Quoted 10 lines
The cropping is also mandatory even if the captured frame is less than the maximum size, because the ISC has no configuration related to frame size in fact. The parallel front end captures pixels and the dma engine will write them to memory. There is no control over this except the cropping.
So for example if the software expects 640x480 but somehow someone sends more pixels, it can lead to buffer overflow if the parallel front end is not configured to ignore(crop) anything beyond 640x480 .
I see and I think you have correctly plumbed that with the scaler sink pad crop rectangle. I don't seem to find where the crop rectangle sizes are actually applied to the ISC in this series ? Could you point me to it for sake of my better understanding ?
Quoted 2 lines
With this being said, or having an understanding about the sink pad of the ISC, how should I configure the sink frame size ?
I don't think you should, the video device format is set through the usual S_FMT as it happens already.
What I think it's missing in your implementation is only to propagate the crop rectangle size of the scaler sink pad to the scaler source pad format (as you don't have any composition rectangles yet) [1]
With that done, you should be able to capture from your video device in the cropped dimensions.
The (simplified) media pipeline will look something like
sensor
type: Subdev
pad 0 (sink) (3280, 2462)
-> scaler:0scaler
type: Subdev
pad 0 (sink) (3280, 2464)
crop: (3262, 2464)
<- sensor:0
pad 1 (source) (3264, 2464)
-> isc_commonisc_common
type V4L2
pad 0 (sink)
<- scaler:1
And the video device format set to (3264, 2464)
Then later you might want to implement s_selection on the scaler to have the crop size configurable.
One additional question: can the crop be panned ? What I mean is that you explained you can instruct the ISC to drop pixels after a certain line lenght. This will cause the crop not to be centered on the input image. Can the ISC be configured to ignore data -before- and -after- a certain number of pixels or lines ? In that case you might want to handle the top,left corner of the crop rectangle too. Anyway, something for later..
Quoted 6 lines
Is there any possibility to avoid limits on the frame size and have it arbitrary ?
I am 99% sure that v4l2-compliance will not be happy if there is no limit on the frame on entity pads (hence, I bound-aligned the frame to 16x16 up to 10k x 10k )
That I'm not sure, but I think having size limits is required ?
Anyway, from my side, the next version should be good to go. Thanks for sticking to all my comments and questions so far!
Thanks
j
Quoted 753 lines
Quoted 752 lines
Quoted 752 lines
Quoted 267 lines
Quoted 101 lines
Quoted 73 lines
Quoted 17 lines
Quoted 5 lines
Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 63
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +8
-0+ + req_fmt->format.colorspace =V4L2_COLORSPACE_SRGB ; + req_fmt->format.field =V4L2_FIELD_NONE ; + req_fmt->format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + req_fmt->format.quantization =V4L2_QUANTIZATION_DEFAULT ; + req_fmt->format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + fmt =isc_find_format_by_code (isc, req_fmt->format.code, &i);So you rely on the isc format list for the scaler as well ? I think it's fine as long as they are identical
Quoted 3 lines
Yes, the scaler is kind of a dummy entity , but it's required by the media controller validation of the pipeline.
Quoted 44 lines
Quoted 35 lines
Quoted 18 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +17
-0+ +if (!fmt) + fmt = &isc->formats_list[0 ]; + + req_fmt->format.code = fmt->mbus_code; + +if (req_fmt->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + req_fmt->pad); + *v4l2_try_fmt = req_fmt->format; +/* Trying on the pad sink makes the source sink change too */ +if (req_fmt->pad ==ISC_SCALER_PAD_SINK ) { + v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state, +ISC_SCALER_PAD_SOURCE ); + *v4l2_try_fmt = req_fmt->format; +drivers/media/platform/atmel/atmel-isc-scaler.c line 90
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+v4l_bound_align_image (&v4l2_try_fmt->width, +16 , isc->max_width,0 , + &v4l2_try_fmt->height, +16 , isc->max_height,0 ,0 ); + } +/* if we are just trying, we are done */ +return 0 ; + } + + isc->scaler_format = req_fmt->format;No per-pad format but a global
scaler_format? How do you configure scaling ?Actually, I would like to know more about the scaler device capabilities. What functions can this IP perform ? Does it do cropping, can it also do (down)scaling or even composition ?
Quoted 8 lines
I think it is worth to read
https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-subdev.html#v4l2-subdev-selections
where it is reported how cropping and scaling are implemented by using the selection API.Figure 4.5 is particularly helpful to explain the simple crop case, for which you need to implement support to by adding
s_selectionon the sink padTGT_CROPtarget.Quoted 5 lines
The scaler is kind of a dummy entity.
The problem is that different subdevices cannot be connected directly to a v4l entity, because the ISC does cropping on the incoming frame. The ISC has a limit on the total number of pixels per frame, thus it will do a crop.Quoted 6 lines
What is this limit related to ? I don't know the ISC internals, but is the limitation due to the overall image size, or maybe it's due to some line buffers being limited in the size they can transfer to memory ? Is it a limitation on the size of the image in pixels, or is it due to a limitation of the processing bandwidth on the input interface and thus depends on the actual size of the image in bytes ?
Limitation is both on line size and full image size. From my understanding, is that the internal SRAM is limited (8 Mpixels with a small safe buffer on top in the case of sama7g5 )
This SRAM could be used as 800000x10 for example, or 3200x2464 for another example. However, I am not sure if a real line of 800,000 pixels make sense, or if there is another internal mechanism in the ISC that stops it.
In any case, the ISC is cropping the incoming frame from aaaXbbb up to the maximum frame that it can handleQuoted 10 lines
Quoted 6 lines
So, if I allow e.g. an entity that sends 3280x2464 , when the ISC can only handle 3264x2464 , then we have two choices:
1/ crop it and output the 3264x2464 -> cannot, because userspace expects 3280x2464, and the whole pipeline fails to configure, there is a mismatch between/dev/videooutput and what the whole pipeline producesI'm confused in first place. If the ISC cannot output anything larger than 3264x2464, then if userspace sets a 3280x2464 size on the ISC, this should be adjusted to 3264x2464.
Quoted 9 lines
yes, that is true. However, without an entity that says 'I am doing cropping', the media pipeline will fail, because previous entity will output 3280x2464 .
The conclusion to create a scaler entitity was done during my last year discussions on the IRC. I could not find a way to perform this cropping at the v4l entity side, and one reason is that the v4l entity does not support what I needed , or at least that was my understanding. You claim that the v4l entity could perform the cropping in the same entity ?I thought so, but I haven't tried to be honest
Quoted 10 lines
Let's assume for a minute that it could do this. Even so, I would prefer to have a separate scaler entity to do this , for several reasons:
-> I would like to be able to crop the frame arbitrarily if I want to, and the scaler would allow me to do this easier
-> it would be more obvious that the frame is cropped by having this entity there
-> in fact some of the ISC versions really have a down scaler, that I have not implemented yet. So it would be useful to already have this in place to be able to scale down with it at a later time.I think this last reason is enough to have an entity plumbed in already
Quoted 8 lines
Quoted 8 lines
Or is the ISC limitation on the -input- side ?
Quoted 3 lines
2/ deny the link, and ask the subdevice to produce the 3264x2464 which the ISC can handle at most -> cannot , because the sensor let's say can *only* produce 3280x2464 and *any* other frame size returns an error.
Quoted 6 lines
Quoted 1 line
buggy sensor I would say :)
I cannot tell this to the customers, and I cannot fail the whole pipeline because the sensor does not crop 16 pixels, when in fact I can do this very easily on the ISC side.
Fair enough :)
Quoted 85 lines
Quoted 11 lines
Quoted 7 lines
The only solution to make both worlds happy, is to have a dummy entity called 'scaler' which in fact now it only performs a simple crop. It accepts any frame size at input (hence the 10000x10000 which you saw earlier), and outputs at most 3264x2464 the
isc_max_heightandisc_max_width.I have a maybe dumb question: can the cropping operation be modeled by applying a
TGT_CROPrectangle (whose_BOUNDsize is 3280x2464) to theatmel_isc_commonentity sink pad and avoid the scaler completely ?I am not sure. This is what I wanted initially. But then I thought about the three reasons stated above for the use of the scaler entity, and discussed with folks on the IRC, and come up with the solution for the scaler entity
Quoted 68 lines
Quoted 67 lines
So to answer your question, the isc scaler is a software model for a simple cropping, that would make media controller happy, and capture software happy.
Here it how it looks :
Device topology
- entity 1:atmel_isc_scaler(2 pads, 2 links)type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev0 pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb crop.bounds:(0,0)/3264x2464 crop:(0,0)/3264x2464] <- "csi2dc":1 [ENABLED,IMMUTABLE] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] -> "atmel_isc_common":0 [ENABLED,IMMUTABLE]- entity 4: csi2dc (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev1 pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] <- "dw-csi.0":1 [ENABLED] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] -> "atmel_isc_scaler":0 [ENABLED,IMMUTABLE]- entity 7: dw-csi.0 (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev2 pad0: Sink [fmt:SRGGB10_1X10/3280x2464] <- "imx219 1-0010":0 [ENABLED] pad1: Source [fmt:SRGGB10_1X10/3280x2464] -> "csi2dc":0 [ENABLED]- entity 12: imx219 1-0010 (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0 device node name/dev/v4l-subdev3
pad0: Source
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb
xfer:srgb ycbcr:601 quantization:full-rangecrop.bounds:(8,8)/3280x2464 crop:(8,8)/3280x2464] -> "dw-csi.0":0 [ENABLED]- entity 24:
atmel_isc_common(1 pad, 1 link)
type Node subtype V4L flags 1
device node name/dev/video0
pad0: Sink
<- "atmel_isc_scaler":1 [ENABLED,IMMUTABLE]Scaler does this one cute little thing :
pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb crop.bounds:(0,0)/3264x2464 crop:(0,0)/3264x2464] <- "csi2dc":1 [ENABLED,IMMUTABLE] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb]Quoted 7 lines
Quoted 3 lines
Shouldn't this be 3264x2464 as that's what the entity outputs after the cropping ? And shouldn't then be 3264x2464 the size output from the video device too ?
That's right.
I don't know why the source format for the scaler is still 3280x2464. Maybe there is a bug ong_fmtfor it.. have to check it.Thanks, I think this should be fixed ten
Quoted 17 lines
Anyway, the video format is like this :
# v4l2-ctl --get-fmt-video Format Video Capture: Width/Height : 3264/2464 Pixel Format : 'RGBP' (16-bit RGB 5-6-5) Field : None Bytes per Line : 6528 Size Image : 16084992 Colorspace : sRGB Transfer Function : Default (maps to sRGB) YCbCr/HSV Encoding: Default (maps to ITU-R 601) Quantization : Default (maps to Full Range) Flags : #Quoted 9 lines
and initializing the pipeline looks fine from user perspective. The scaler is in fact the solution to make this pipeline work with libcamera.
I found this cropping to be an issue in media controller when trying it with libcamera. Otherwise, the other user space apps which I was using never complained that anything was wrong
Libcamera simply refuses to acknowledge the pipeline if the video output is 3264x2464 but there is no entity that changes the format from 3280x2464 downQuoted 2 lines
Not sure why libcamera should play a role there. Isn't it the media pipeline validation that complains and returns an
-EPIPE?I tried this before the
media_pipeline_start.
Perhaps libcamera was doing some similar checks, anyway, it was complaining and not adding the stream, and reported no usable streams/cameraOnce I will rework some of the things according to your review I will run libcamera again to see what it complains about
Quoted 474 lines
Quoted 93 lines
Quoted 73 lines
Quoted 63 lines
-> "atmel_isc_common":0 [ENABLED,IMMUTABLE]Which is what we needed.
Quoted 19 lines
Quoted 6 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; +} + +static int isc_scaler_enum_mbus_code(struct v4l2_subdev *sd,drivers/media/platform/atmel/atmel-isc-scaler.c line 105
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+struct v4l2_subdev_state *sd_state, + struct v4l2_subdev_mbus_code_enum *code) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +int supported_index =0 ; +int i; + +for (i =0 ; i < isc->formats_list_size; i++) { +if (!isc->formats_list[i].sd_support) +continue ;The
sd_supportflag still doesn't click in my head.Quoted 2 lines
Shouldn't the enumeration of available formats on the scaler do not depend on the sensor supproted formats ?
You're right. I will have to check it again.
Quoted 30 lines
Quoted 16 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +15
-0+if (supported_index == code->index) { + code->code = isc->formats_list[i].mbus_code; +return 0 ; + } + supported_index++; + } + +return -EINVAL ; +} + +static int isc_scaler_g_sel (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_selection *sel) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd);drivers/media/platform/atmel/atmel-isc-scaler.c line 130
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +if (sel->pad ==ISC_SCALER_PAD_SOURCE ) +return -EINVAL ; + +if (sel->target !=V4L2_SEL_TGT_CROP_BOUNDS && + sel->target !=V4L2_SEL_TGT_CROP ) +return -EINVAL ; + + sel->r.height = isc->max_height; + sel->r.width = isc->max_width;The
CROP_BOUNDSshould be set to the same size as the sink pad image format, as it represents the maximum valid crop rectangle.Quoted 9 lines
Quoted 3 lines
TGT_CROPshould report the configured crop rectangle which can be intiialized to the same size asCROP_BOUNDS, if my understanding of the spec is correctSo you would like to have this differentiated, and report the
CROP_BOUNDSto whatever is on the sink pad, and theTGT_CROPto what is reported now, the maximum size of the ISC frame .
My understanding is correct ?Quoted 6 lines
I didn't know you have an HW limitation, so your
_BOUNDSis not the input image size but rather 3264x2464 ( ==max_widthxmax_height).What I meant is that
_BOUNDSshould report the maximum rectangle size that can be applied to the_CROPtarget. In you case you have an HW limitation 3264x2464 and that's the largest rectangle you can apply.So the CROP should be at 3264x2464
Quoted 2 lines
TGT_CROPcan be initialized to the same as_BOUND, but if you implements_selectionit should report what has been there applied.and BOUND to actual frame size ?
Quoted 2 lines
But as you don't implement
s_selectionyet, I think this is fine for now. Maybe a little comment ?It could also be the case where e.g. the sensor is outputting 1920x1080, in this case the scaler would do nothing.
CROP is still 3264x2464 and BOUND in this case is 1920x1080 ? If the sensor is outputting 1920x1080, this format comes directly from the sensor (the sensor is cropping it from 3280x2464 or not... it's the sensor's problem)Quoted 1 line
Quoted 4 lines
Quoted 2 lines
Also, is
set->rzeroed by the framework before getting here ? Otherwise you should set r.left and r.top to 0If these are redundant, no problem to remove them
I was actually confused. I was suggesting you to add...
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 133
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +3
-0+ + sel->r.left =0 ; + sel->r.top =0 ;These ^
So nothing to change. Sorry I've missed them
Thanks
jQuoted 360 lines
Quoted 359 lines
Quoted 49 lines
Quoted 40 lines
Quoted 12 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +12
-0+ +return 0 ; +} + +static int isc_scaler_init_cfg (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state) +{ +struct v4l2_mbus_framefmt *v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state,0 ); + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); + + *v4l2_try_fmt = isc->scaler_format;drivers/media/platform/atmel/atmel-isc-scaler.c line 155
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +return 0 ; +} + +static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = { + .enum_mbus_code = isc_scaler_enum_mbus_code, + .set_fmt = isc_scaler_set_fmt, + .get_fmt = isc_scaler_get_fmt, + .get_selection = isc_scaler_g_sel, + .init_cfg = isc_scaler_init_cfg,.link_validate =
v4l2_subdev_link_validate_default,To have the formats at the end of links that point to this entity validated (I think the framework already calls it if not set though, please check v4l2-subdev.c:v4l2_subdev_link_validate())
Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +11
-0+}; + +static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = { + .pad = &isc_scaler_pad_ops, +}; + +int isc_scaler_init (struct isc_device *isc) +{ +int ret; + +v4l2_subdev_init (&isc->scaler_sd, &xisc_scaler_subdev_ops);Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 171
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ + isc->scaler_sd.owner =THIS_MODULE ; + isc->scaler_sd.dev = isc->dev; +snprintf (isc->scaler_sd.name,sizeof (isc->scaler_sd.name), +"atmel_isc_scaler" );I would drop 'atmel' for brevity, unless other entities have this prefix set already
Quoted 3 lines
The v4l entity takes it's name from the module name, which is
atmel_isc_common, so I thought to keep the prefixAck!
Quoted 189 lines
Quoted 70 lines
Quoted 70 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +69
-0+ + isc->scaler_sd.flags |=V4L2_SUBDEV_FL_HAS_DEVNODE ; + isc->scaler_sd.entity.function =MEDIA_ENT_F_PROC_VIDEO_SCALER ; + isc->scaler_pads[ISC_SCALER_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + isc->scaler_pads[ISC_SCALER_PAD_SOURCE ].flags =MEDIA_PAD_FL_SOURCE ; + + isc->scaler_format.height = isc->max_height; + isc->scaler_format.width = isc->max_width; + isc->scaler_format.code = isc->formats_list[0 ].mbus_code; + isc->scaler_format.colorspace =V4L2_COLORSPACE_SRGB ; + isc->scaler_format.field =V4L2_FIELD_NONE ; + isc->scaler_format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + isc->scaler_format.quantization =V4L2_QUANTIZATION_DEFAULT ; + isc->scaler_format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + ret =media_entity_pads_init (&isc->scaler_sd.entity, +ISC_SCALER_PADS_NUM , + isc->scaler_pads); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd media entity init failed\n" ); +return ret; + } + ret =v4l2_device_register_subdev (&isc->v4l2_dev, &isc->scaler_sd); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd failed to register subdev\n" ); +return ret; + } + +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_init); + +int isc_scaler_link (struct isc_device *isc) +{ +int ret; + + ret =media_create_pad_link (&isc->current_subdev->sd->entity, + isc->remote_pad, &isc->scaler_sd.entity, +ISC_SCALER_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->current_subdev->sd->entity.name, + isc->scaler_sd.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , + isc->current_subdev->sd->name, isc->remote_pad); + + ret =media_create_pad_link (&isc->scaler_sd.entity, +ISC_SCALER_PAD_SOURCE , + &isc->video_dev.entity,ISC_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->scaler_sd.entity.name, + isc->video_dev.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , isc->scaler_sd.name, +ISC_SCALER_PAD_SOURCE );Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 244
Quoted 4 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +4
-0+ +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_link);From the DT graph point of view, the ISC appears as a single block
with an CSI-2 input port. It is then in charge of parsing the .dts and add to its notifier the image sensor remote subdevice.Actually it's only a parallel port.
And it can be connected either to a sensor directly or to the csi2dc bridge. (the csi2dc bridge can be connected to a CSI-2 receiver)Quoted 4 lines
When the sensor subdev registers and the ISC notifier completes, the scaler entity which was initialized at
isc_probe()time is linked in between the ISC and the image sensor immutably.yes, because this cannot change at runtime, and usually, it can't change without altering the board hardware. (at least this is what my understanding of immutability is )
Quoted 9 lines
I think it is fine for now, but I wonder if you plan to plumb more components between the ISC video node and the sensor, if it's not worth changing the DT bindings and their parsing logic to separate the CSI-2 receiver from the ISC, whcih can create its media graph at probe time and have the CSI-2 receiver entity as downstream remote. I think I need to get to know the ISC better to really have an idea. For now, this seems ok to me, but please check with maintainers if this is fine with them.
In the XISC case (sama7g5-isc), the csi2dc receiver is the subdev (so, the remote subdev).
The ISC will register a scaler, and connect the subdev to this scaler first, and then, the scaler to the isc itself (the v4l entity).Quoted 45 lines
Quoted 45 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +1
-0+drivers/media/platform/atmel/atmel-isc.h +21
-0@@ -183,6 +183,17 @@ struct isc_reg_offsets {u32 his_entry; }; +enum isc_mc_pads { +ISC_PAD_SINK =0 , +ISC_PADS_NUM =1 , +}; + +enum isc_scaler_pads { +ISC_SCALER_PAD_SINK =0 , +ISC_SCALER_PAD_SOURCE =1 , +ISC_SCALER_PADS_NUM =2 , +}; +/* * struct isc_device - ISC device driver data/config struct * @regmap: Register map @@ -257,6 +268,12 @@ struct isc_reg_offsets {* be used as an input to the controller * @controller_formats_size: size of controller_formats array * @formats_list_size: size of formats_list array +* @pads: media controller pads for isc video entity +* @mdev: media device that is registered by the isc +* @remote_pad: remote pad on the connected subdevice +* @scaler_sd: subdevice for the scaler that isc registers +* @scaler_pads: media controller pads for the scaler subdevice +* @scaler_format: current format for the scaler subdevice */ struct isc_device {struct regmap *regmap;@@ -344,6 +361,19 @@ struct isc_device {struct isc_format *formats_list;u32 controller_formats_size;u32 formats_list_size; + +struct { +struct media_pad pads[ISC_PADS_NUM ]; +struct media_device mdev;Quoted 8 lines
drivers/media/platform/atmel/atmel-isc.h struct isc_device { · line 375
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc.h +8
-0+ +u32 remote_pad; + }; + +struct { +struct v4l2_subdev scaler_sd; +struct media_pad scaler_pads[ISC_SCALER_PADS_NUM ]; +struct v4l2_mbus_framefmt scaler_format;Here are the scaler stuff which I added, in the same bulk struct for the whole isc device
Quoted 24 lines
Quoted 24 lines
drivers/media/platform/atmel/atmel-isc.h +8
-0+ }; };extern const struct regmap_config isc_regmap_config;@@ -355,4 +385,11 @@ int isc_clk_init(struct isc_device *isc);void isc_subdev_cleanup (struct isc_device *isc);void isc_clk_cleanup (struct isc_device *isc); +int isc_scaler_link (struct isc_device *isc); +int isc_scaler_init (struct isc_device *isc); +int isc_mc_init (struct isc_device *isc,u32 ver); +void isc_mc_cleanup (struct isc_device *isc); + +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index);#endif drivers/media/platform/atmel/atmel-sama5d2-isc.c +0
-0@@ -553,6 +553,12 @@ static int atmel_isc_probe(struct platform_device *pdev)break ; }Quoted 8 lines
Quoted 5 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c static int atmel_isc_probe(struct platform_device *pdev) · line 556
Quoted 1 line
drivers/media/platform/atmel/atmel-sama5d2-isc.c +1
-0+regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);I am surprised you can read a register before
runtime_pmis intialized!Actually I can read any register after the moment of starting the clock on the hardware block.
Ah right then
Quoted 3 lines
Maybe the starting and stopping of the clock needs to be moved to the
runtime_pmcalls, but this is another story, not related to the media controller.It's fine, but maybe worth recording with a todo ?
Thanks
jQuoted 95 lines
I moved this line because I had to pass the version to the
isc_mc_initcallQuoted 2 lines
Thanks
jThanks for reviewing !
EugenQuoted 87 lines
Quoted 87 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c +11
-2+ + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev);@@ -562,7 +568,7 @@ static int atmel_isc_probe(struct platform_device *pdev) ret =clk_prepare_enable (isc->ispck);if (ret) {dev_err (dev,"failed to enable ispck: %d\n" , ret); - goto cleanup_subdev; + goto isc_probe_mc_init_err; }/* ispck should be greater or equal to hclock */ @@ -572,7 +578,6 @@ static int atmel_isc_probe(struct platform_device *pdev) goto unprepare_clk; } -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip ISC version %x\n" , ver);return 0 ;@@ -580,6 +585,9 @@ static int atmel_isc_probe(struct platform_device *pdev) unprepare_clk:clk_disable_unprepare (isc->ispck); +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -600,6 +608,8 @@ static int atmel_isc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev);drivers/media/platform/atmel/atmel-sama7g5-isc.c +11
-2@@ -547,15 +547,23 @@ static int microchip_xisc_probe(struct platform_device *pdev)break ; } +regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver); + + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev); -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip XISC version %x\n" , ver);return 0 ; +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -576,6 +584,8 @@ static int microchip_xisc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev); --2.25.1
Quoted 505 lines · Jacopo Mondi
Hi Eugen
Quoted 475 lines · On Fri, Feb 11, 2022 at 10:32:43AM +0000, [email protected] wrote:
Quoted 416 lines · Jacopo Mondi
Hi Eugen
Quoted 404 lines · On Wed, Feb 09, 2022 at 06:59:37AM +0000, [email protected] wrote:
Quoted 178 lines · Jacopo Mondi
Hi Eugen
Quoted 163 lines · On Tue, Feb 08, 2022 at 10:33:31AM +0000, [email protected] wrote:
Quoted 162 lines · Jacopo Mondi
Hi Eugen
Quoted 158 lines · On Mon, Feb 07, 2022 at 01:40:31PM +0000, [email protected] wrote:
Quoted 138 lines · Jacopo Mondi
Hi Eugen
Quoted 135 lines · Eugen Hristev
Implement the support for media-controller.
This means that the capabilities of the driver have changed and now it also advertises theIO_MC.
The driver will register it's media device, and add the video entity to this media device. The subdevices are registered to the same media device. The ISC will have a base entity which is auto-detected asatmel_isc_base. It will also register a subdevice that allows cropping of the incoming frame to the maximum frame size supported by the ISC.
The ISC will create a link between the subdevice that is asynchronously registered and theatmel_isc_scalerentity.
Then, theatmel_isc_scalerandatmel_isc_baseare connected through another link.Signed-off-by Eugen Hristev <[email protected]>---
Changes in v4:
As suggested by Jacopo:
- renamedatmel_isc_mctoatmel_isc_scaler.c
- moved init_mc/clean_mc toisc_basefileChanges in v2:
- implement try formatsdrivers/media/platform/atmel/Makefile | 2 +- drivers/media/platform/atmel/atmel-isc-base.c | 73 +++++- .../media/platform/atmel/atmel-isc-scaler.c | 245 ++++++++++++++++++ drivers/media/platform/atmel/atmel-isc.h | 37 +++ .../media/platform/atmel/atmel-sama5d2-isc.c | 14 +- .../media/platform/atmel/atmel-sama7g5-isc.c | 12 +- 6 files changed, 375 insertions(+), 8 deletions(-)create mode 100644
drivers/media/platform/atmel/atmel-isc-scaler.cdrivers/media/platform/atmel/Makefile +1
-1@@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only atmel-isc-objs = atmel-sama5d2-isc.o atmel-xisc-objs = atmel-sama7g5-isc.o -atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o +atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o atmel-isc-scaler.o obj-$(CONFIG_VIDEO_ATMEL_ISI) += atmel-isi.o obj-$(CONFIG_VIDEO_ATMEL_ISC_BASE) += atmel-isc-common.odrivers/media/platform/atmel/atmel-isc-base.c +27
-4@@ -1710,6 +1710,7 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, struct isc_device, v4l2_dev);struct isc_subdev_entity *subdev_entity = container_of(notifier, struct isc_subdev_entity, notifier); +int pad;if (video_is_registered (&isc->video_dev)) {v4l2_err (&isc->v4l2_dev,"only supports one sub-device.\n" );@@ -1718,6 +1719,16 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, subdev_entity->sd = subdev; + pad =media_entity_get_fwnode_pad (&subdev->entity, asd->match.fwnode, +MEDIA_PAD_FL_SOURCE ); +if (pad <0 ) { +v4l2_err (&isc->v4l2_dev,"failed to find pad for %s\n" , + subdev->name); +return pad; + } + + isc->remote_pad = pad; +return 0 ; }@@ -1732,8 +1743,8 @@ static void isc_async_unbind(struct v4l2_async_notifier *notifier,v4l2_ctrl_handler_free (&isc->ctrls.handler); } -static struct isc_format *find_format_by_code (struct isc_device *isc, -unsigned int code,int *index) +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index) {struct isc_format *fmt = &isc->formats_list[0 ];unsigned int i;@@ -1749,6 +1760,7 @@ static struct isc_format *find_format_by_code(struct isc_device *isc,return NULL ; } +EXPORT_SYMBOL_GPL (isc_find_format_by_code);static int isc_formats_init (struct isc_device *isc) {@@ -1765,7 +1777,7 @@ static int isc_formats_init(struct isc_device *isc) NULL, &mbus_code)) { mbus_code.index++; - fmt =find_format_by_code (isc, mbus_code.code, &i); + fmt =isc_find_format_by_code (isc, mbus_code.code, &i);if (!fmt) {v4l2_warn (&isc->v4l2_dev,"Mbus code %x not supported\n" , mbus_code.code);@@ -1891,7 +1903,8 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) vdev->queue = q; vdev->lock = &isc->lock; vdev->ctrl_handler = &isc->ctrls.handler; - vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE ; + vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE | +V4L2_CAP_IO_MC ;video_set_drvdata (vdev, isc); ret =video_register_device (vdev,VFL_TYPE_VIDEO ,-1 );@@ -1901,8 +1914,18 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) goto isc_async_complete_err; } + ret =isc_scaler_link (isc); +if (ret <0 ) + goto isc_async_complete_unregister_device; + + ret =media_device_register (&isc->mdev); +if (ret <0 ) + goto isc_async_complete_unregister_device;return 0 ; +isc_async_complete_unregister_device: +video_unregister_device (vdev); + isc_async_complete_err:mutex_destroy (&isc->lock);return ret;@@ -1969,6 +1992,48 @@ int isc_pipeline_init(struct isc_device *isc) }EXPORT_SYMBOL_GPL (isc_pipeline_init);Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-base.c int isc_pipeline_init(struct isc_device *isc) · line 2001
Quoted 7 lines
drivers/media/platform/atmel/atmel-isc-base.c +7
-0+int isc_mc_init (struct isc_device *isc,u32 ver) +{ +const struct of_device_id *match; +int ret; + + isc->video_dev.entity.function =MEDIA_ENT_F_IO_V4L ; + isc->video_dev.entity.flags =MEDIA_ENT_FL_DEFAULT ;Should you set entity.ops.link_validate =
v4l2_subdev_link_validateto be able to have the media pipeline validated atmedia_pipeline_start()time ?Hi,
I am doing that in a subsequent patch. Things are not completely ready at this moment, because ISC still relies on the old mechanism to call
v4l2_subdev_callwithset_fmton the subdevice (which subsequent patch removes and adds checks to the link validate ).Quoted 6 lines
Quoted 1 line
I see.. the subsequent patches are not part of this series, right ?
No. Patch 7 does this. In that patch, the link validation is created and all the logic of format propagation is removed and reworked, and creates the need for the
link_validatecall
Could you have also a look at that patch ?Ah ups, 7 was not in my inbox with the rest of the series. Just noticed. I had even reviewed the previous version, I should have remembered
link_validatewas added laterQuoted 1 line
Quoted 1 line
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
I think patches 1, 2, 4, 5 and 6 from this series do not depend on the introduction of media controller or the scaler, right ? (I'm not sure if the DTS patches do).
Do DTS patches depend on media-controller plumbing ?
I think not, but, I have not tested them without the media controller.
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
If that's the case, would it make sense to to fast-track them (as some of them are fixes) and then on top plumb the media controller infrastructure with this patch and the ones you have been mentioning ?
It would make sense.
Once the other discussion with Hans about the stop state it might make sense to fastrack the first part of the series ?
The other three patches (4,5,6) are quite independent and can go faster.
Quoted 149 lines
Quoted 139 lines
Quoted 119 lines
Quoted 119 lines
Quoted 79 lines
Quoted 47 lines
drivers/media/platform/atmel/atmel-isc-base.c +35
-0+ isc->pads[ISC_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + + ret =media_entity_pads_init (&isc->video_dev.entity,ISC_PADS_NUM , + isc->pads); +if (ret <0 ) { +dev_err (isc->dev,"media entity init failed\n" ); +return ret; + } + + isc->mdev.dev = isc->dev; + + match =of_match_node (isc->dev->driver->of_match_table, + isc->dev->of_node); + +strscpy (isc->mdev.driver_name,KBUILD_MODNAME , +sizeof (isc->mdev.driver_name)); +strscpy (isc->mdev.model, match->compatible,sizeof (isc->mdev.model)); +snprintf (isc->mdev.bus_info,sizeof (isc->mdev.bus_info),"platform:%s" , + isc->v4l2_dev.name); + isc->mdev.hw_revision = ver; + +media_device_init (&isc->mdev); + + isc->v4l2_dev.mdev = &isc->mdev; + +return isc_scaler_init (isc); +} +EXPORT_SYMBOL_GPL (isc_mc_init); + +void isc_mc_cleanup (struct isc_device *isc) +{ +media_entity_cleanup (&isc->video_dev.entity); +} +EXPORT_SYMBOL_GPL (isc_mc_cleanup); +/* regmap configuration */ #define ATMEL_ISC_REG_MAX 0xd5cconst struct regmap_config isc_regmap_config = {drivers/media/platform/atmel/atmel-isc-scaler.c new file +0
-0drivers/media/platform/atmel/atmel-isc-scaler.c line 5
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0@@ -0,0 +1,245 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Microchip Image Sensor Controller (ISC) Scaler entity support +* +* Copyright (C) 2021 Microchip Technology, Inc. Time flies! It's 2022 already :)
Quoted 24 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +24
-0+* +* Author: Eugen Hristev <[email protected]> +* +*/ + +#include <media/media-device.h> +#include <media/media-entity.h> +#include <media/v4l2-device.h> +#include <media/v4l2-subdev.h> + +#include "atmel-isc-regs.h" +#include "atmel-isc.h" + +static int isc_scaler_get_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *format) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; + +if (format->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + format->pad); + format->format = *v4l2_try_fmt;Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 34
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; + } + + format->format = isc->scaler_format;
isc->scaler_formatis only used inside this file if I'm not mistaken. I wonder why it lives in theisc_devicestruct.
isc_deviceis a placeholder for all isc things.I would not create a separate struct in this file that would have to be allocated etc... just for one two things. So I preferred to have it in the same place as all the other things.
Quoted 24 lines
Quoted 12 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +11
-0+ +return 0 ; +} + +static int isc_scaler_set_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *req_fmt) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; +struct isc_format *fmt;drivers/media/platform/atmel/atmel-isc-scaler.c line 55
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+unsigned int i; + +if (req_fmt->pad ==ISC_SCALER_PAD_SOURCE ) +v4l_bound_align_image + (&req_fmt->format.width,16 , isc->max_width,0 , + &req_fmt->format.height,16 , isc->max_height,0 ,0 ); +else +v4l_bound_align_image + (&req_fmt->format.width,16 ,10000 ,0 , + &req_fmt->format.height,16 ,10000 ,0 ,0 );Where does 10000 come from ?
Quoted 3 lines
Quoted 1 line
It's a random number. Do you have any suggestion for a better one ?
An actual HW limit ? :)
There is no hardware limit. The ISC just stops sampling pixels once the crop limit is reached. The element that generates pixels could go on forever , or until the next HBLANK/VBLANK. So what limit could I place here ?
The cropping is mandatory, otherwise there could be an overflow w.r.t. the buffer size if the ISC would sample more pixels than the software expects it to.Quoted 8 lines
Quoted 7 lines
Maybe this would be much more clear with my comments below (where I will explain what the isc scaler does )
In short, it allows the other entity in the link to 'sink' a huge format into this 'scaler' . Because the scaler can crop anything basically down to the biggest format size the ISC could handle.Quoted 9 lines
Quoted 1 line
doesn't it have any documented input size limit ?
As stated above, ISC handles a pixel stream and stores it in an internal SRAM. ISC stops sampling when this SRAM is full or when the cropping limit is reached.
After storing the pixels in the SRAM (which is limited to the ISC maximum frame size), the ISC will work on the pixels and then DMA the frame to another RAM.Quoted 4 lines
I understand...
So whatever the input size is, only the first n-th bytes are actually dumped to memory, the following ones are discarded.
Yes ! That's the behavior.
Quoted 3 lines
So the crop is always applied on the top-leftmost corner as a consequence, right ?
Right. Pixels come in order. The ISC does not really know how many pixels are coming until its memory is full (or counting until the crop limit is reached). After that it stops sampling pixels, and waits for vblank to finish the frame.
Quoted 8 lines
I understand why you had difficulties picking a default :)
This is not easy to handle, as the scaler doesn't actually applies a crop but rather stops capturing after the destination buffer has been saturated. So the limiting factor is the full image size, not one of its dimensions, like in example the line length. This makes it hard to adjust cropping to something meaningful for most use cases.
I didn't know how to name this except crop. It 'crops out' what does not fit in the internal memory of the ISC
Quoted 7 lines
In your below example you suggest that an image size of (800000 x 10) would theoretically be captured correctly. What if the input is of size (800000 x 10 + 1). What size would you set the crop rectangle to ?
I wouldn't however be too much concerned for the moment, but I would record this in a comment ?
I discussed a bit more with designer and looked in datasheet. It looks like the horizontal line length is not a hard limit, but it makes the ISC internal ISP stop working, so I would keep the horizontal limit as well for now, unless an important use case (I am coming back to that). The vertical limit looks to be a hard requirement.
So in short, we have to impose a vertical limit of 2464 , we cannot have more lines than these. So if the horizontal is 100 pixels e.g., we can capture only 100x2464 .
In practice we could capture [16,3264] on the horizontal, and [16,2464] on the vertical, and in theory horizontal could go more like [16, 4000], but we have to limit the vertical, and without the internal ISP. So I would not haphazardly go in this direction. Let's stick to the datasheet maximum frame 3264x2464 .
According to this , I changed the frame size of the ISC to be a continuous size from [16,16] to [3264,2464]
About that use case that I talked about earlier, we have one sensor , imx274, which is more wide, and outputs something like 3800x2100 . With the cropping and limitations we have in place now, this will be limited to 3264x2100 , which is in line with datasheet and what the ISCQuoted 8 lines
supports now.
As in theory we could capture 3800x2100 as long as the SRAM doesn't get full, for now I would like to avoid capturing this size (and disable the internal ISP), until we have someone really asking for such kind of capture and with some good reasons.
In short I would like to have the crop rectangle to 3264x2464 and the bounds would be limited to the incoming frame but no H>3264 and no V>2464. Does that make sense ?Quoted 4 lines
Just to make sure I got it: is the ISC actually able to crop on the line length ? If you apply a 3264 limit, will the last 536 pixels of each line will be dropped if the input frame is 3800x2100 as your above example ?
Yes. Last 536 pixels will be dropped.
After the 3264 limit, no more pixels are being sampled until HBLANK.Quoted 39 lines
Or rather, is the SRAM size limit the issue and the lines gets captured in full lenght no matter what, until the buffer gets filled up and all the successive pixels dropped.
In other words, do your cropped image look like:
3264 +--------------------------------X-----------+ 3800 |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | +--------------------------------------------+ 2100Or rather
3264 +--------------------------------X-----------+ 3800 |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxx| | | | +--------------------------------------------+ 2100Quoted 2 lines
I assume the first, as otherwise you wouldn't be able to interpret the image as 3264x2100.
Yes, the first.
Quoted 6 lines
I'm asking because you had the un-cropped 3280x2464 size as the ISC source pad size in the paste of the media graph you provided below, which might mean the output image is actually full size in lines lenght and it's maybe missing pixels at the end, in case the ISC operates as in the above second case.
Quoted 12 lines
No, the image won't be padded with blanks on the right side. Actually this cannot happen due to a very big reason: the frame is being written to the main memory by a DMA controller. This DMA controller will write all the lines at consecutive addresses, and it will write exactly how many pixels are being captured by the module from the ISC named 'parallel front end' which does the cropping. Thus, the DMA will never output empty padding bytes at the end of each line, and it will not add a certain stride to each line if not instructed to. In fact the DMA can add a stride offset for the address of each line, which can be used for composing or copying the frame into another frame, but this is a different story. ( I tried this for fun, but not implemented in the driver at the moment)
Oh that will really be interesting and it's indeed an argument for hainvg a scaler subdevice indeed
Quoted 10 lines
The cropping is also mandatory even if the captured frame is less than the maximum size, because the ISC has no configuration related to frame size in fact. The parallel front end captures pixels and the dma engine will write them to memory. There is no control over this except the cropping.
So for example if the software expects 640x480 but somehow someone sends more pixels, it can lead to buffer overflow if the parallel front end is not configured to ignore(crop) anything beyond 640x480 .
Quoted 4 lines
I see and I think you have correctly plumbed that with the scaler sink pad crop rectangle. I don't seem to find where the crop rectangle sizes are actually applied to the ISC in this series ? Could you point me to it for sake of my better understanding ?
The ISC is already doing that for some time now:
https://elixir.bootlin.com/linux/latest/source/drivers/media/platform/atmel/atmel-isc-base.c#L459
Initially it was more like a safety feature, because I discovered that there was no way of limiting the incoming pixels, and if some malevolent entity would just keep sending pixels, it could lead to exploits.
Now, the scaler is just exposing this to the user.
Quoted 39 lines
Quoted 3 lines
With this being said, or having an understanding about the sink pad of the ISC, how should I configure the sink frame size ?
I don't think you should, the video device format is set through the usual
S_FMTas it happens already.What I think it's missing in your implementation is only to propagate the crop rectangle size of the scaler sink pad to the scaler source pad format (as you don't have any composition rectangles yet) [1]
With that done, you should be able to capture from your video device in the cropped dimensions.
The (simplified) media pipeline will look something like
sensor
type: Subdev pad 0 (sink) (3280, 2462) -> scaler:0scaler
type: Subdev pad 0 (sink) (3280, 2464) crop: (3262, 2464) <- sensor:0 pad 1 (source) (3264, 2464) -> isc_common
isc_common
type V4L2
pad 0 (sink)
<- scaler:1And the video device format set to (3264, 2464)
Then later you might want to implement
s_selectionon the scaler to have the crop size configurable.
Quoted 7 lines
One additional question: can the crop be panned ? What I mean is that you explained you can instruct the ISC to drop pixels after a certain line lenght. This will cause the crop not to be centered on the input image. Can the ISC be configured to ignore data -before- and -after- a certain number of pixels or lines ? In that case you might want to handle the top,left corner of the crop rectangle too. Anyway, something for later..
Looking over this, it looks possible. I have not tried it. It looks like configuring a minimum row /col number would make the ISC drop the first n pixels/ first n lines.
I will have to try this, but if it works, user could configure the cropping area, that would be interesting ! Anyway I can't implement this right now, but it's a nice thing to work on later.
Quoted 10 lines
Quoted 6 lines
Is there any possibility to avoid limits on the frame size and have it arbitrary ?
I am 99% sure that v4l2-compliance will not be happy if there is no limit on the frame on entity pads (hence, I bound-aligned the frame to 16x16 up to 10k x 10k )
Quoted 4 lines
That I'm not sure, but I think having size limits is required ?
Anyway, from my side, the next version should be good to go. Thanks for sticking to all my comments and questions so far!
What should I do about the 10000 x 10000 pixels ?
Thanks for reviewing !
Eugen
Quoted 758 lines
Thanks
jQuoted 753 lines
Quoted 752 lines
Quoted 752 lines
Quoted 267 lines
Quoted 101 lines
Quoted 73 lines
Quoted 17 lines
Quoted 5 lines
Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 63
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +8
-0+ + req_fmt->format.colorspace =V4L2_COLORSPACE_SRGB ; + req_fmt->format.field =V4L2_FIELD_NONE ; + req_fmt->format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + req_fmt->format.quantization =V4L2_QUANTIZATION_DEFAULT ; + req_fmt->format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + fmt =isc_find_format_by_code (isc, req_fmt->format.code, &i);So you rely on the isc format list for the scaler as well ? I think it's fine as long as they are identical
Quoted 3 lines
Yes, the scaler is kind of a dummy entity , but it's required by the media controller validation of the pipeline.
Quoted 44 lines
Quoted 35 lines
Quoted 18 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +17
-0+ +if (!fmt) + fmt = &isc->formats_list[0 ]; + + req_fmt->format.code = fmt->mbus_code; + +if (req_fmt->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + req_fmt->pad); + *v4l2_try_fmt = req_fmt->format; +/* Trying on the pad sink makes the source sink change too */ +if (req_fmt->pad ==ISC_SCALER_PAD_SINK ) { + v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state, +ISC_SCALER_PAD_SOURCE ); + *v4l2_try_fmt = req_fmt->format; +drivers/media/platform/atmel/atmel-isc-scaler.c line 90
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+v4l_bound_align_image (&v4l2_try_fmt->width, +16 , isc->max_width,0 , + &v4l2_try_fmt->height, +16 , isc->max_height,0 ,0 ); + } +/* if we are just trying, we are done */ +return 0 ; + } + + isc->scaler_format = req_fmt->format;No per-pad format but a global
scaler_format? How do you configure scaling ?Actually, I would like to know more about the scaler device capabilities. What functions can this IP perform ? Does it do cropping, can it also do (down)scaling or even composition ?
Quoted 8 lines
I think it is worth to read
https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-subdev.html#v4l2-subdev-selections
where it is reported how cropping and scaling are implemented by using the selection API.Figure 4.5 is particularly helpful to explain the simple crop case, for which you need to implement support to by adding
s_selectionon the sink padTGT_CROPtarget.Quoted 5 lines
The scaler is kind of a dummy entity.
The problem is that different subdevices cannot be connected directly to a v4l entity, because the ISC does cropping on the incoming frame. The ISC has a limit on the total number of pixels per frame, thus it will do a crop.Quoted 6 lines
What is this limit related to ? I don't know the ISC internals, but is the limitation due to the overall image size, or maybe it's due to some line buffers being limited in the size they can transfer to memory ? Is it a limitation on the size of the image in pixels, or is it due to a limitation of the processing bandwidth on the input interface and thus depends on the actual size of the image in bytes ?
Limitation is both on line size and full image size. From my understanding, is that the internal SRAM is limited (8 Mpixels with a small safe buffer on top in the case of sama7g5 )
This SRAM could be used as 800000x10 for example, or 3200x2464 for another example. However, I am not sure if a real line of 800,000 pixels make sense, or if there is another internal mechanism in the ISC that stops it.
In any case, the ISC is cropping the incoming frame from aaaXbbb up to the maximum frame that it can handleQuoted 10 lines
Quoted 6 lines
So, if I allow e.g. an entity that sends 3280x2464 , when the ISC can only handle 3264x2464 , then we have two choices:
1/ crop it and output the 3264x2464 -> cannot, because userspace expects 3280x2464, and the whole pipeline fails to configure, there is a mismatch between/dev/videooutput and what the whole pipeline producesI'm confused in first place. If the ISC cannot output anything larger than 3264x2464, then if userspace sets a 3280x2464 size on the ISC, this should be adjusted to 3264x2464.
Quoted 9 lines
yes, that is true. However, without an entity that says 'I am doing cropping', the media pipeline will fail, because previous entity will output 3280x2464 .
The conclusion to create a scaler entitity was done during my last year discussions on the IRC. I could not find a way to perform this cropping at the v4l entity side, and one reason is that the v4l entity does not support what I needed , or at least that was my understanding. You claim that the v4l entity could perform the cropping in the same entity ?I thought so, but I haven't tried to be honest
Quoted 10 lines
Let's assume for a minute that it could do this. Even so, I would prefer to have a separate scaler entity to do this , for several reasons:
-> I would like to be able to crop the frame arbitrarily if I want to, and the scaler would allow me to do this easier
-> it would be more obvious that the frame is cropped by having this entity there
-> in fact some of the ISC versions really have a down scaler, that I have not implemented yet. So it would be useful to already have this in place to be able to scale down with it at a later time.I think this last reason is enough to have an entity plumbed in already
Quoted 8 lines
Quoted 8 lines
Or is the ISC limitation on the -input- side ?
Quoted 3 lines
2/ deny the link, and ask the subdevice to produce the 3264x2464 which the ISC can handle at most -> cannot , because the sensor let's say can *only* produce 3280x2464 and *any* other frame size returns an error.
Quoted 6 lines
Quoted 1 line
buggy sensor I would say :)
I cannot tell this to the customers, and I cannot fail the whole pipeline because the sensor does not crop 16 pixels, when in fact I can do this very easily on the ISC side.
Fair enough :)
Quoted 85 lines
Quoted 11 lines
Quoted 7 lines
The only solution to make both worlds happy, is to have a dummy entity called 'scaler' which in fact now it only performs a simple crop. It accepts any frame size at input (hence the 10000x10000 which you saw earlier), and outputs at most 3264x2464 the
isc_max_heightandisc_max_width.I have a maybe dumb question: can the cropping operation be modeled by applying a
TGT_CROPrectangle (whose_BOUNDsize is 3280x2464) to theatmel_isc_commonentity sink pad and avoid the scaler completely ?I am not sure. This is what I wanted initially. But then I thought about the three reasons stated above for the use of the scaler entity, and discussed with folks on the IRC, and come up with the solution for the scaler entity
Quoted 68 lines
Quoted 67 lines
So to answer your question, the isc scaler is a software model for a simple cropping, that would make media controller happy, and capture software happy.
Here it how it looks :
Device topology
- entity 1:atmel_isc_scaler(2 pads, 2 links)type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev0 pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb crop.bounds:(0,0)/3264x2464 crop:(0,0)/3264x2464] <- "csi2dc":1 [ENABLED,IMMUTABLE] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] -> "atmel_isc_common":0 [ENABLED,IMMUTABLE]- entity 4: csi2dc (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev1 pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] <- "dw-csi.0":1 [ENABLED] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] -> "atmel_isc_scaler":0 [ENABLED,IMMUTABLE]- entity 7: dw-csi.0 (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev2 pad0: Sink [fmt:SRGGB10_1X10/3280x2464] <- "imx219 1-0010":0 [ENABLED] pad1: Source [fmt:SRGGB10_1X10/3280x2464] -> "csi2dc":0 [ENABLED]- entity 12: imx219 1-0010 (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0 device node name/dev/v4l-subdev3
pad0: Source
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb
xfer:srgb ycbcr:601 quantization:full-rangecrop.bounds:(8,8)/3280x2464 crop:(8,8)/3280x2464] -> "dw-csi.0":0 [ENABLED]- entity 24:
atmel_isc_common(1 pad, 1 link)
type Node subtype V4L flags 1
device node name/dev/video0
pad0: Sink
<- "atmel_isc_scaler":1 [ENABLED,IMMUTABLE]Scaler does this one cute little thing :
pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb crop.bounds:(0,0)/3264x2464 crop:(0,0)/3264x2464] <- "csi2dc":1 [ENABLED,IMMUTABLE] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb]Quoted 7 lines
Quoted 3 lines
Shouldn't this be 3264x2464 as that's what the entity outputs after the cropping ? And shouldn't then be 3264x2464 the size output from the video device too ?
That's right.
I don't know why the source format for the scaler is still 3280x2464. Maybe there is a bug ong_fmtfor it.. have to check it.Thanks, I think this should be fixed ten
Quoted 17 lines
Anyway, the video format is like this :
# v4l2-ctl --get-fmt-video Format Video Capture: Width/Height : 3264/2464 Pixel Format : 'RGBP' (16-bit RGB 5-6-5) Field : None Bytes per Line : 6528 Size Image : 16084992 Colorspace : sRGB Transfer Function : Default (maps to sRGB) YCbCr/HSV Encoding: Default (maps to ITU-R 601) Quantization : Default (maps to Full Range) Flags : #Quoted 9 lines
and initializing the pipeline looks fine from user perspective. The scaler is in fact the solution to make this pipeline work with libcamera.
I found this cropping to be an issue in media controller when trying it with libcamera. Otherwise, the other user space apps which I was using never complained that anything was wrong
Libcamera simply refuses to acknowledge the pipeline if the video output is 3264x2464 but there is no entity that changes the format from 3280x2464 downQuoted 2 lines
Not sure why libcamera should play a role there. Isn't it the media pipeline validation that complains and returns an
-EPIPE?I tried this before the
media_pipeline_start.
Perhaps libcamera was doing some similar checks, anyway, it was complaining and not adding the stream, and reported no usable streams/cameraOnce I will rework some of the things according to your review I will run libcamera again to see what it complains about
Quoted 474 lines
Quoted 93 lines
Quoted 73 lines
Quoted 63 lines
-> "atmel_isc_common":0 [ENABLED,IMMUTABLE]Which is what we needed.
Quoted 19 lines
Quoted 6 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; +} + +static int isc_scaler_enum_mbus_code(struct v4l2_subdev *sd,drivers/media/platform/atmel/atmel-isc-scaler.c line 105
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+struct v4l2_subdev_state *sd_state, + struct v4l2_subdev_mbus_code_enum *code) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +int supported_index =0 ; +int i; + +for (i =0 ; i < isc->formats_list_size; i++) { +if (!isc->formats_list[i].sd_support) +continue ;The
sd_supportflag still doesn't click in my head.Quoted 2 lines
Shouldn't the enumeration of available formats on the scaler do not depend on the sensor supproted formats ?
You're right. I will have to check it again.
Quoted 30 lines
Quoted 16 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +15
-0+if (supported_index == code->index) { + code->code = isc->formats_list[i].mbus_code; +return 0 ; + } + supported_index++; + } + +return -EINVAL ; +} + +static int isc_scaler_g_sel (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_selection *sel) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd);drivers/media/platform/atmel/atmel-isc-scaler.c line 130
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +if (sel->pad ==ISC_SCALER_PAD_SOURCE ) +return -EINVAL ; + +if (sel->target !=V4L2_SEL_TGT_CROP_BOUNDS && + sel->target !=V4L2_SEL_TGT_CROP ) +return -EINVAL ; + + sel->r.height = isc->max_height; + sel->r.width = isc->max_width;The
CROP_BOUNDSshould be set to the same size as the sink pad image format, as it represents the maximum valid crop rectangle.Quoted 9 lines
Quoted 3 lines
TGT_CROPshould report the configured crop rectangle which can be intiialized to the same size asCROP_BOUNDS, if my understanding of the spec is correctSo you would like to have this differentiated, and report the
CROP_BOUNDSto whatever is on the sink pad, and theTGT_CROPto what is reported now, the maximum size of the ISC frame .
My understanding is correct ?Quoted 6 lines
I didn't know you have an HW limitation, so your
_BOUNDSis not the input image size but rather 3264x2464 ( ==max_widthxmax_height).What I meant is that
_BOUNDSshould report the maximum rectangle size that can be applied to the_CROPtarget. In you case you have an HW limitation 3264x2464 and that's the largest rectangle you can apply.So the CROP should be at 3264x2464
Quoted 2 lines
TGT_CROPcan be initialized to the same as_BOUND, but if you implements_selectionit should report what has been there applied.and BOUND to actual frame size ?
Quoted 2 lines
But as you don't implement
s_selectionyet, I think this is fine for now. Maybe a little comment ?It could also be the case where e.g. the sensor is outputting 1920x1080, in this case the scaler would do nothing.
CROP is still 3264x2464 and BOUND in this case is 1920x1080 ? If the sensor is outputting 1920x1080, this format comes directly from the sensor (the sensor is cropping it from 3280x2464 or not... it's the sensor's problem)Quoted 1 line
Quoted 4 lines
Quoted 2 lines
Also, is
set->rzeroed by the framework before getting here ? Otherwise you should set r.left and r.top to 0If these are redundant, no problem to remove them
I was actually confused. I was suggesting you to add...
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 133
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +3
-0+ + sel->r.left =0 ; + sel->r.top =0 ;These ^
So nothing to change. Sorry I've missed them
Thanks
jQuoted 360 lines
Quoted 359 lines
Quoted 49 lines
Quoted 40 lines
Quoted 12 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +12
-0+ +return 0 ; +} + +static int isc_scaler_init_cfg (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state) +{ +struct v4l2_mbus_framefmt *v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state,0 ); + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); + + *v4l2_try_fmt = isc->scaler_format;drivers/media/platform/atmel/atmel-isc-scaler.c line 155
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +return 0 ; +} + +static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = { + .enum_mbus_code = isc_scaler_enum_mbus_code, + .set_fmt = isc_scaler_set_fmt, + .get_fmt = isc_scaler_get_fmt, + .get_selection = isc_scaler_g_sel, + .init_cfg = isc_scaler_init_cfg,.link_validate =
v4l2_subdev_link_validate_default,To have the formats at the end of links that point to this entity validated (I think the framework already calls it if not set though, please check v4l2-subdev.c:v4l2_subdev_link_validate())
Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +11
-0+}; + +static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = { + .pad = &isc_scaler_pad_ops, +}; + +int isc_scaler_init (struct isc_device *isc) +{ +int ret; + +v4l2_subdev_init (&isc->scaler_sd, &xisc_scaler_subdev_ops);Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 171
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ + isc->scaler_sd.owner =THIS_MODULE ; + isc->scaler_sd.dev = isc->dev; +snprintf (isc->scaler_sd.name,sizeof (isc->scaler_sd.name), +"atmel_isc_scaler" );I would drop 'atmel' for brevity, unless other entities have this prefix set already
Quoted 3 lines
The v4l entity takes it's name from the module name, which is
atmel_isc_common, so I thought to keep the prefixAck!
Quoted 189 lines
Quoted 70 lines
Quoted 70 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +69
-0+ + isc->scaler_sd.flags |=V4L2_SUBDEV_FL_HAS_DEVNODE ; + isc->scaler_sd.entity.function =MEDIA_ENT_F_PROC_VIDEO_SCALER ; + isc->scaler_pads[ISC_SCALER_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + isc->scaler_pads[ISC_SCALER_PAD_SOURCE ].flags =MEDIA_PAD_FL_SOURCE ; + + isc->scaler_format.height = isc->max_height; + isc->scaler_format.width = isc->max_width; + isc->scaler_format.code = isc->formats_list[0 ].mbus_code; + isc->scaler_format.colorspace =V4L2_COLORSPACE_SRGB ; + isc->scaler_format.field =V4L2_FIELD_NONE ; + isc->scaler_format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + isc->scaler_format.quantization =V4L2_QUANTIZATION_DEFAULT ; + isc->scaler_format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + ret =media_entity_pads_init (&isc->scaler_sd.entity, +ISC_SCALER_PADS_NUM , + isc->scaler_pads); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd media entity init failed\n" ); +return ret; + } + ret =v4l2_device_register_subdev (&isc->v4l2_dev, &isc->scaler_sd); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd failed to register subdev\n" ); +return ret; + } + +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_init); + +int isc_scaler_link (struct isc_device *isc) +{ +int ret; + + ret =media_create_pad_link (&isc->current_subdev->sd->entity, + isc->remote_pad, &isc->scaler_sd.entity, +ISC_SCALER_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->current_subdev->sd->entity.name, + isc->scaler_sd.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , + isc->current_subdev->sd->name, isc->remote_pad); + + ret =media_create_pad_link (&isc->scaler_sd.entity, +ISC_SCALER_PAD_SOURCE , + &isc->video_dev.entity,ISC_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->scaler_sd.entity.name, + isc->video_dev.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , isc->scaler_sd.name, +ISC_SCALER_PAD_SOURCE );Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 244
Quoted 4 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +4
-0+ +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_link);From the DT graph point of view, the ISC appears as a single block
with an CSI-2 input port. It is then in charge of parsing the .dts and add to its notifier the image sensor remote subdevice.Actually it's only a parallel port.
And it can be connected either to a sensor directly or to the csi2dc bridge. (the csi2dc bridge can be connected to a CSI-2 receiver)Quoted 4 lines
When the sensor subdev registers and the ISC notifier completes, the scaler entity which was initialized at
isc_probe()time is linked in between the ISC and the image sensor immutably.yes, because this cannot change at runtime, and usually, it can't change without altering the board hardware. (at least this is what my understanding of immutability is )
Quoted 9 lines
I think it is fine for now, but I wonder if you plan to plumb more components between the ISC video node and the sensor, if it's not worth changing the DT bindings and their parsing logic to separate the CSI-2 receiver from the ISC, whcih can create its media graph at probe time and have the CSI-2 receiver entity as downstream remote. I think I need to get to know the ISC better to really have an idea. For now, this seems ok to me, but please check with maintainers if this is fine with them.
In the XISC case (sama7g5-isc), the csi2dc receiver is the subdev (so, the remote subdev).
The ISC will register a scaler, and connect the subdev to this scaler first, and then, the scaler to the isc itself (the v4l entity).Quoted 45 lines
Quoted 45 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +1
-0+drivers/media/platform/atmel/atmel-isc.h +21
-0@@ -183,6 +183,17 @@ struct isc_reg_offsets {u32 his_entry; }; +enum isc_mc_pads { +ISC_PAD_SINK =0 , +ISC_PADS_NUM =1 , +}; + +enum isc_scaler_pads { +ISC_SCALER_PAD_SINK =0 , +ISC_SCALER_PAD_SOURCE =1 , +ISC_SCALER_PADS_NUM =2 , +}; +/* * struct isc_device - ISC device driver data/config struct * @regmap: Register map @@ -257,6 +268,12 @@ struct isc_reg_offsets {* be used as an input to the controller * @controller_formats_size: size of controller_formats array * @formats_list_size: size of formats_list array +* @pads: media controller pads for isc video entity +* @mdev: media device that is registered by the isc +* @remote_pad: remote pad on the connected subdevice +* @scaler_sd: subdevice for the scaler that isc registers +* @scaler_pads: media controller pads for the scaler subdevice +* @scaler_format: current format for the scaler subdevice */ struct isc_device {struct regmap *regmap;@@ -344,6 +361,19 @@ struct isc_device {struct isc_format *formats_list;u32 controller_formats_size;u32 formats_list_size; + +struct { +struct media_pad pads[ISC_PADS_NUM ]; +struct media_device mdev;Quoted 8 lines
drivers/media/platform/atmel/atmel-isc.h struct isc_device { · line 375
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc.h +8
-0+ +u32 remote_pad; + }; + +struct { +struct v4l2_subdev scaler_sd; +struct media_pad scaler_pads[ISC_SCALER_PADS_NUM ]; +struct v4l2_mbus_framefmt scaler_format;Here are the scaler stuff which I added, in the same bulk struct for the whole isc device
Quoted 24 lines
Quoted 24 lines
drivers/media/platform/atmel/atmel-isc.h +8
-0+ }; };extern const struct regmap_config isc_regmap_config;@@ -355,4 +385,11 @@ int isc_clk_init(struct isc_device *isc);void isc_subdev_cleanup (struct isc_device *isc);void isc_clk_cleanup (struct isc_device *isc); +int isc_scaler_link (struct isc_device *isc); +int isc_scaler_init (struct isc_device *isc); +int isc_mc_init (struct isc_device *isc,u32 ver); +void isc_mc_cleanup (struct isc_device *isc); + +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index);#endif drivers/media/platform/atmel/atmel-sama5d2-isc.c +0
-0@@ -553,6 +553,12 @@ static int atmel_isc_probe(struct platform_device *pdev)break ; }Quoted 8 lines
Quoted 5 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c static int atmel_isc_probe(struct platform_device *pdev) · line 556
Quoted 1 line
drivers/media/platform/atmel/atmel-sama5d2-isc.c +1
-0+regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);I am surprised you can read a register before
runtime_pmis intialized!Actually I can read any register after the moment of starting the clock on the hardware block.
Ah right then
Quoted 3 lines
Maybe the starting and stopping of the clock needs to be moved to the
runtime_pmcalls, but this is another story, not related to the media controller.It's fine, but maybe worth recording with a todo ?
Thanks
jQuoted 95 lines
I moved this line because I had to pass the version to the
isc_mc_initcallQuoted 2 lines
Thanks
jThanks for reviewing !
EugenQuoted 87 lines
Quoted 87 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c +11
-2+ + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev);@@ -562,7 +568,7 @@ static int atmel_isc_probe(struct platform_device *pdev) ret =clk_prepare_enable (isc->ispck);if (ret) {dev_err (dev,"failed to enable ispck: %d\n" , ret); - goto cleanup_subdev; + goto isc_probe_mc_init_err; }/* ispck should be greater or equal to hclock */ @@ -572,7 +578,6 @@ static int atmel_isc_probe(struct platform_device *pdev) goto unprepare_clk; } -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip ISC version %x\n" , ver);return 0 ;@@ -580,6 +585,9 @@ static int atmel_isc_probe(struct platform_device *pdev) unprepare_clk:clk_disable_unprepare (isc->ispck); +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -600,6 +608,8 @@ static int atmel_isc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev);drivers/media/platform/atmel/atmel-sama7g5-isc.c +11
-2@@ -547,15 +547,23 @@ static int microchip_xisc_probe(struct platform_device *pdev)break ; } +regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver); + + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev); -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip XISC version %x\n" , ver);return 0 ; +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -576,6 +584,8 @@ static int microchip_xisc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev); --2.25.1
Hi Eugen
Quoted 586 lines · On Fri, Feb 11, 2022 at 12:32:14PM +0000, [email protected] wrote:
Quoted 505 lines · Jacopo Mondi
Hi Eugen
Quoted 475 lines · On Fri, Feb 11, 2022 at 10:32:43AM +0000, [email protected] wrote:
Quoted 416 lines · Jacopo Mondi
Hi Eugen
Quoted 404 lines · On Wed, Feb 09, 2022 at 06:59:37AM +0000, [email protected] wrote:
Quoted 178 lines · Jacopo Mondi
Hi Eugen
Quoted 163 lines · On Tue, Feb 08, 2022 at 10:33:31AM +0000, [email protected] wrote:
Quoted 162 lines · Jacopo Mondi
Hi Eugen
Quoted 158 lines · On Mon, Feb 07, 2022 at 01:40:31PM +0000, [email protected] wrote:
Quoted 138 lines · Jacopo Mondi
Hi Eugen
Quoted 135 lines · Eugen Hristev
Implement the support for media-controller.
This means that the capabilities of the driver have changed and now it also advertises theIO_MC.
The driver will register it's media device, and add the video entity to this media device. The subdevices are registered to the same media device. The ISC will have a base entity which is auto-detected asatmel_isc_base. It will also register a subdevice that allows cropping of the incoming frame to the maximum frame size supported by the ISC.
The ISC will create a link between the subdevice that is asynchronously registered and theatmel_isc_scalerentity.
Then, theatmel_isc_scalerandatmel_isc_baseare connected through another link.Signed-off-by Eugen Hristev <[email protected]>---
Changes in v4:
As suggested by Jacopo:
- renamedatmel_isc_mctoatmel_isc_scaler.c
- moved init_mc/clean_mc toisc_basefileChanges in v2:
- implement try formatsdrivers/media/platform/atmel/Makefile | 2 +- drivers/media/platform/atmel/atmel-isc-base.c | 73 +++++- .../media/platform/atmel/atmel-isc-scaler.c | 245 ++++++++++++++++++ drivers/media/platform/atmel/atmel-isc.h | 37 +++ .../media/platform/atmel/atmel-sama5d2-isc.c | 14 +- .../media/platform/atmel/atmel-sama7g5-isc.c | 12 +- 6 files changed, 375 insertions(+), 8 deletions(-)create mode 100644
drivers/media/platform/atmel/atmel-isc-scaler.cdrivers/media/platform/atmel/Makefile +1
-1@@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only atmel-isc-objs = atmel-sama5d2-isc.o atmel-xisc-objs = atmel-sama7g5-isc.o -atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o +atmel-isc-common-objs = atmel-isc-base.o atmel-isc-clk.o atmel-isc-scaler.o obj-$(CONFIG_VIDEO_ATMEL_ISI) += atmel-isi.o obj-$(CONFIG_VIDEO_ATMEL_ISC_BASE) += atmel-isc-common.odrivers/media/platform/atmel/atmel-isc-base.c +27
-4@@ -1710,6 +1710,7 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, struct isc_device, v4l2_dev);struct isc_subdev_entity *subdev_entity = container_of(notifier, struct isc_subdev_entity, notifier); +int pad;if (video_is_registered (&isc->video_dev)) {v4l2_err (&isc->v4l2_dev,"only supports one sub-device.\n" );@@ -1718,6 +1719,16 @@ static int isc_async_bound(struct v4l2_async_notifier *notifier, subdev_entity->sd = subdev; + pad =media_entity_get_fwnode_pad (&subdev->entity, asd->match.fwnode, +MEDIA_PAD_FL_SOURCE ); +if (pad <0 ) { +v4l2_err (&isc->v4l2_dev,"failed to find pad for %s\n" , + subdev->name); +return pad; + } + + isc->remote_pad = pad; +return 0 ; }@@ -1732,8 +1743,8 @@ static void isc_async_unbind(struct v4l2_async_notifier *notifier,v4l2_ctrl_handler_free (&isc->ctrls.handler); } -static struct isc_format *find_format_by_code (struct isc_device *isc, -unsigned int code,int *index) +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index) {struct isc_format *fmt = &isc->formats_list[0 ];unsigned int i;@@ -1749,6 +1760,7 @@ static struct isc_format *find_format_by_code(struct isc_device *isc,return NULL ; } +EXPORT_SYMBOL_GPL (isc_find_format_by_code);static int isc_formats_init (struct isc_device *isc) {@@ -1765,7 +1777,7 @@ static int isc_formats_init(struct isc_device *isc) NULL, &mbus_code)) { mbus_code.index++; - fmt =find_format_by_code (isc, mbus_code.code, &i); + fmt =isc_find_format_by_code (isc, mbus_code.code, &i);if (!fmt) {v4l2_warn (&isc->v4l2_dev,"Mbus code %x not supported\n" , mbus_code.code);@@ -1891,7 +1903,8 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) vdev->queue = q; vdev->lock = &isc->lock; vdev->ctrl_handler = &isc->ctrls.handler; - vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE ; + vdev->device_caps =V4L2_CAP_STREAMING |V4L2_CAP_VIDEO_CAPTURE | +V4L2_CAP_IO_MC ;video_set_drvdata (vdev, isc); ret =video_register_device (vdev,VFL_TYPE_VIDEO ,-1 );@@ -1901,8 +1914,18 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) goto isc_async_complete_err; } + ret =isc_scaler_link (isc); +if (ret <0 ) + goto isc_async_complete_unregister_device; + + ret =media_device_register (&isc->mdev); +if (ret <0 ) + goto isc_async_complete_unregister_device;return 0 ; +isc_async_complete_unregister_device: +video_unregister_device (vdev); + isc_async_complete_err:mutex_destroy (&isc->lock);return ret;@@ -1969,6 +1992,48 @@ int isc_pipeline_init(struct isc_device *isc) }EXPORT_SYMBOL_GPL (isc_pipeline_init);Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-base.c int isc_pipeline_init(struct isc_device *isc) · line 2001
Quoted 7 lines
drivers/media/platform/atmel/atmel-isc-base.c +7
-0+int isc_mc_init (struct isc_device *isc,u32 ver) +{ +const struct of_device_id *match; +int ret; + + isc->video_dev.entity.function =MEDIA_ENT_F_IO_V4L ; + isc->video_dev.entity.flags =MEDIA_ENT_FL_DEFAULT ;Should you set entity.ops.link_validate =
v4l2_subdev_link_validateto be able to have the media pipeline validated atmedia_pipeline_start()time ?Hi,
I am doing that in a subsequent patch. Things are not completely ready at this moment, because ISC still relies on the old mechanism to call
v4l2_subdev_callwithset_fmton the subdevice (which subsequent patch removes and adds checks to the link validate ).Quoted 6 lines
Quoted 1 line
I see.. the subsequent patches are not part of this series, right ?
No. Patch 7 does this. In that patch, the link validation is created and all the logic of format propagation is removed and reworked, and creates the need for the
link_validatecall
Could you have also a look at that patch ?Ah ups, 7 was not in my inbox with the rest of the series. Just noticed. I had even reviewed the previous version, I should have remembered
link_validatewas added laterQuoted 1 line
Quoted 1 line
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
I think patches 1, 2, 4, 5 and 6 from this series do not depend on the introduction of media controller or the scaler, right ? (I'm not sure if the DTS patches do).
Do DTS patches depend on media-controller plumbing ?
I think not, but, I have not tested them without the media controller.
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
If that's the case, would it make sense to to fast-track them (as some of them are fixes) and then on top plumb the media controller infrastructure with this patch and the ones you have been mentioning ?
It would make sense.
Once the other discussion with Hans about the stop state it might make sense to fastrack the first part of the series ?
The other three patches (4,5,6) are quite independent and can go faster.
Quoted 149 lines
Quoted 139 lines
Quoted 119 lines
Quoted 119 lines
Quoted 79 lines
Quoted 47 lines
drivers/media/platform/atmel/atmel-isc-base.c +35
-0+ isc->pads[ISC_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + + ret =media_entity_pads_init (&isc->video_dev.entity,ISC_PADS_NUM , + isc->pads); +if (ret <0 ) { +dev_err (isc->dev,"media entity init failed\n" ); +return ret; + } + + isc->mdev.dev = isc->dev; + + match =of_match_node (isc->dev->driver->of_match_table, + isc->dev->of_node); + +strscpy (isc->mdev.driver_name,KBUILD_MODNAME , +sizeof (isc->mdev.driver_name)); +strscpy (isc->mdev.model, match->compatible,sizeof (isc->mdev.model)); +snprintf (isc->mdev.bus_info,sizeof (isc->mdev.bus_info),"platform:%s" , + isc->v4l2_dev.name); + isc->mdev.hw_revision = ver; + +media_device_init (&isc->mdev); + + isc->v4l2_dev.mdev = &isc->mdev; + +return isc_scaler_init (isc); +} +EXPORT_SYMBOL_GPL (isc_mc_init); + +void isc_mc_cleanup (struct isc_device *isc) +{ +media_entity_cleanup (&isc->video_dev.entity); +} +EXPORT_SYMBOL_GPL (isc_mc_cleanup); +/* regmap configuration */ #define ATMEL_ISC_REG_MAX 0xd5cconst struct regmap_config isc_regmap_config = {drivers/media/platform/atmel/atmel-isc-scaler.c new file +0
-0drivers/media/platform/atmel/atmel-isc-scaler.c line 5
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0@@ -0,0 +1,245 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Microchip Image Sensor Controller (ISC) Scaler entity support +* +* Copyright (C) 2021 Microchip Technology, Inc. Time flies! It's 2022 already :)
Quoted 24 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +24
-0+* +* Author: Eugen Hristev <[email protected]> +* +*/ + +#include <media/media-device.h> +#include <media/media-entity.h> +#include <media/v4l2-device.h> +#include <media/v4l2-subdev.h> + +#include "atmel-isc-regs.h" +#include "atmel-isc.h" + +static int isc_scaler_get_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *format) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; + +if (format->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + format->pad); + format->format = *v4l2_try_fmt;Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 34
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; + } + + format->format = isc->scaler_format;
isc->scaler_formatis only used inside this file if I'm not mistaken. I wonder why it lives in theisc_devicestruct.
isc_deviceis a placeholder for all isc things.I would not create a separate struct in this file that would have to be allocated etc... just for one two things. So I preferred to have it in the same place as all the other things.
Quoted 24 lines
Quoted 12 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +11
-0+ +return 0 ; +} + +static int isc_scaler_set_fmt (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_format *req_fmt) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +struct v4l2_mbus_framefmt *v4l2_try_fmt; +struct isc_format *fmt;drivers/media/platform/atmel/atmel-isc-scaler.c line 55
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+unsigned int i; + +if (req_fmt->pad ==ISC_SCALER_PAD_SOURCE ) +v4l_bound_align_image + (&req_fmt->format.width,16 , isc->max_width,0 , + &req_fmt->format.height,16 , isc->max_height,0 ,0 ); +else +v4l_bound_align_image + (&req_fmt->format.width,16 ,10000 ,0 , + &req_fmt->format.height,16 ,10000 ,0 ,0 );Where does 10000 come from ?
Quoted 3 lines
Quoted 1 line
It's a random number. Do you have any suggestion for a better one ?
An actual HW limit ? :)
There is no hardware limit. The ISC just stops sampling pixels once the crop limit is reached. The element that generates pixels could go on forever , or until the next HBLANK/VBLANK. So what limit could I place here ?
The cropping is mandatory, otherwise there could be an overflow w.r.t. the buffer size if the ISC would sample more pixels than the software expects it to.Quoted 8 lines
Quoted 7 lines
Maybe this would be much more clear with my comments below (where I will explain what the isc scaler does )
In short, it allows the other entity in the link to 'sink' a huge format into this 'scaler' . Because the scaler can crop anything basically down to the biggest format size the ISC could handle.Quoted 9 lines
Quoted 1 line
doesn't it have any documented input size limit ?
As stated above, ISC handles a pixel stream and stores it in an internal SRAM. ISC stops sampling when this SRAM is full or when the cropping limit is reached.
After storing the pixels in the SRAM (which is limited to the ISC maximum frame size), the ISC will work on the pixels and then DMA the frame to another RAM.Quoted 4 lines
I understand...
So whatever the input size is, only the first n-th bytes are actually dumped to memory, the following ones are discarded.
Yes ! That's the behavior.
Quoted 3 lines
So the crop is always applied on the top-leftmost corner as a consequence, right ?
Right. Pixels come in order. The ISC does not really know how many pixels are coming until its memory is full (or counting until the crop limit is reached). After that it stops sampling pixels, and waits for vblank to finish the frame.
Quoted 8 lines
I understand why you had difficulties picking a default :)
This is not easy to handle, as the scaler doesn't actually applies a crop but rather stops capturing after the destination buffer has been saturated. So the limiting factor is the full image size, not one of its dimensions, like in example the line length. This makes it hard to adjust cropping to something meaningful for most use cases.
I didn't know how to name this except crop. It 'crops out' what does not fit in the internal memory of the ISC
Quoted 7 lines
In your below example you suggest that an image size of (800000 x 10) would theoretically be captured correctly. What if the input is of size (800000 x 10 + 1). What size would you set the crop rectangle to ?
I wouldn't however be too much concerned for the moment, but I would record this in a comment ?
I discussed a bit more with designer and looked in datasheet. It looks like the horizontal line length is not a hard limit, but it makes the ISC internal ISP stop working, so I would keep the horizontal limit as well for now, unless an important use case (I am coming back to that). The vertical limit looks to be a hard requirement.
So in short, we have to impose a vertical limit of 2464 , we cannot have more lines than these. So if the horizontal is 100 pixels e.g., we can capture only 100x2464 .
In practice we could capture [16,3264] on the horizontal, and [16,2464] on the vertical, and in theory horizontal could go more like [16, 4000], but we have to limit the vertical, and without the internal ISP. So I would not haphazardly go in this direction. Let's stick to the datasheet maximum frame 3264x2464 .
According to this , I changed the frame size of the ISC to be a continuous size from [16,16] to [3264,2464]
About that use case that I talked about earlier, we have one sensor , imx274, which is more wide, and outputs something like 3800x2100 . With the cropping and limitations we have in place now, this will be limited to 3264x2100 , which is in line with datasheet and what the ISCQuoted 8 lines
supports now.
As in theory we could capture 3800x2100 as long as the SRAM doesn't get full, for now I would like to avoid capturing this size (and disable the internal ISP), until we have someone really asking for such kind of capture and with some good reasons.
In short I would like to have the crop rectangle to 3264x2464 and the bounds would be limited to the incoming frame but no H>3264 and no V>2464. Does that make sense ?Quoted 4 lines
Just to make sure I got it: is the ISC actually able to crop on the line length ? If you apply a 3264 limit, will the last 536 pixels of each line will be dropped if the input frame is 3800x2100 as your above example ?
Yes. Last 536 pixels will be dropped.
After the 3264 limit, no more pixels are being sampled until HBLANK.Quoted 39 lines
Or rather, is the SRAM size limit the issue and the lines gets captured in full lenght no matter what, until the buffer gets filled up and all the successive pixels dropped.
In other words, do your cropped image look like:
3264 +--------------------------------X-----------+ 3800 |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| | +--------------------------------------------+ 2100Or rather
3264 +--------------------------------X-----------+ 3800 |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx| |xxxxxxxxxxxxxxxxx| | | | +--------------------------------------------+ 2100Quoted 2 lines
I assume the first, as otherwise you wouldn't be able to interpret the image as 3264x2100.
Yes, the first.
Quoted 6 lines
I'm asking because you had the un-cropped 3280x2464 size as the ISC source pad size in the paste of the media graph you provided below, which might mean the output image is actually full size in lines lenght and it's maybe missing pixels at the end, in case the ISC operates as in the above second case.
Quoted 12 lines
No, the image won't be padded with blanks on the right side. Actually this cannot happen due to a very big reason: the frame is being written to the main memory by a DMA controller. This DMA controller will write all the lines at consecutive addresses, and it will write exactly how many pixels are being captured by the module from the ISC named 'parallel front end' which does the cropping. Thus, the DMA will never output empty padding bytes at the end of each line, and it will not add a certain stride to each line if not instructed to. In fact the DMA can add a stride offset for the address of each line, which can be used for composing or copying the frame into another frame, but this is a different story. ( I tried this for fun, but not implemented in the driver at the moment)
Oh that will really be interesting and it's indeed an argument for hainvg a scaler subdevice indeed
Quoted 10 lines
The cropping is also mandatory even if the captured frame is less than the maximum size, because the ISC has no configuration related to frame size in fact. The parallel front end captures pixels and the dma engine will write them to memory. There is no control over this except the cropping.
So for example if the software expects 640x480 but somehow someone sends more pixels, it can lead to buffer overflow if the parallel front end is not configured to ignore(crop) anything beyond 640x480 .Quoted 4 lines
I see and I think you have correctly plumbed that with the scaler sink pad crop rectangle. I don't seem to find where the crop rectangle sizes are actually applied to the ISC in this series ? Could you point me to it for sake of my better understanding ?
The ISC is already doing that for some time now:
https://elixir.bootlin.com/linux/latest/source/drivers/media/platform/atmel/atmel-isc-base.c#L459
Initially it was more like a safety feature, because I discovered that there was no way of limiting the incoming pixels, and if some malevolent entity would just keep sending pixels, it could lead to exploits.
Now, the scaler is just exposing this to the user.
Quoted 39 lines
Quoted 3 lines
With this being said, or having an understanding about the sink pad of the ISC, how should I configure the sink frame size ?
I don't think you should, the video device format is set through the usual
S_FMTas it happens already.What I think it's missing in your implementation is only to propagate the crop rectangle size of the scaler sink pad to the scaler source pad format (as you don't have any composition rectangles yet) [1]
With that done, you should be able to capture from your video device in the cropped dimensions.
The (simplified) media pipeline will look something like
sensor
type: Subdev pad 0 (sink) (3280, 2462) -> scaler:0scaler
type: Subdev pad 0 (sink) (3280, 2464) crop: (3262, 2464) <- sensor:0 pad 1 (source) (3264, 2464) -> isc_common
isc_common
type V4L2
pad 0 (sink)
<- scaler:1And the video device format set to (3264, 2464)
Then later you might want to implement
s_selectionon the scaler to have the crop size configurable.Quoted 7 lines
One additional question: can the crop be panned ? What I mean is that you explained you can instruct the ISC to drop pixels after a certain line lenght. This will cause the crop not to be centered on the input image. Can the ISC be configured to ignore data -before- and -after- a certain number of pixels or lines ? In that case you might want to handle the top,left corner of the crop rectangle too. Anyway, something for later..
Looking over this, it looks possible. I have not tried it. It looks like configuring a minimum row /col number would make the ISC drop the first n pixels/ first n lines.
I will have to try this, but if it works, user could configure the cropping area, that would be interesting ! Anyway I can't implement this right now, but it's a nice thing to work on later.
Quoted 10 lines
Quoted 6 lines
Is there any possibility to avoid limits on the frame size and have it arbitrary ?
I am 99% sure that v4l2-compliance will not be happy if there is no limit on the frame on entity pads (hence, I bound-aligned the frame to 16x16 up to 10k x 10k )
Quoted 7 lines
Quoted 4 lines
That I'm not sure, but I think having size limits is required ?
Anyway, from my side, the next version should be good to go. Thanks for sticking to all my comments and questions so far!
What should I do about the 10000 x 10000 pixels ?
If only used for aligning in
v4l_bound_align_image
(&req_fmt->format.width, 16, 10000, 0,
&req_fmt->format.height, 16, 10000, 0, 0);I think you can use UINT_MAX to make it more explicit ?
Quoted 762 lines
Thanks for reviewing !
Eugen
Quoted 758 lines
Thanks
jQuoted 753 lines
Quoted 752 lines
Quoted 752 lines
Quoted 267 lines
Quoted 101 lines
Quoted 73 lines
Quoted 17 lines
Quoted 5 lines
Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 63
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +8
-0+ + req_fmt->format.colorspace =V4L2_COLORSPACE_SRGB ; + req_fmt->format.field =V4L2_FIELD_NONE ; + req_fmt->format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + req_fmt->format.quantization =V4L2_QUANTIZATION_DEFAULT ; + req_fmt->format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + fmt =isc_find_format_by_code (isc, req_fmt->format.code, &i);So you rely on the isc format list for the scaler as well ? I think it's fine as long as they are identical
Quoted 3 lines
Yes, the scaler is kind of a dummy entity , but it's required by the media controller validation of the pipeline.
Quoted 44 lines
Quoted 35 lines
Quoted 18 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +17
-0+ +if (!fmt) + fmt = &isc->formats_list[0 ]; + + req_fmt->format.code = fmt->mbus_code; + +if (req_fmt->which ==V4L2_SUBDEV_FORMAT_TRY ) { + v4l2_try_fmt =v4l2_subdev_get_try_format (sd, sd_state, + req_fmt->pad); + *v4l2_try_fmt = req_fmt->format; +/* Trying on the pad sink makes the source sink change too */ +if (req_fmt->pad ==ISC_SCALER_PAD_SINK ) { + v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state, +ISC_SCALER_PAD_SOURCE ); + *v4l2_try_fmt = req_fmt->format; +drivers/media/platform/atmel/atmel-isc-scaler.c line 90
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+v4l_bound_align_image (&v4l2_try_fmt->width, +16 , isc->max_width,0 , + &v4l2_try_fmt->height, +16 , isc->max_height,0 ,0 ); + } +/* if we are just trying, we are done */ +return 0 ; + } + + isc->scaler_format = req_fmt->format;No per-pad format but a global
scaler_format? How do you configure scaling ?Actually, I would like to know more about the scaler device capabilities. What functions can this IP perform ? Does it do cropping, can it also do (down)scaling or even composition ?
Quoted 8 lines
I think it is worth to read
https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-subdev.html#v4l2-subdev-selections
where it is reported how cropping and scaling are implemented by using the selection API.Figure 4.5 is particularly helpful to explain the simple crop case, for which you need to implement support to by adding
s_selectionon the sink padTGT_CROPtarget.Quoted 5 lines
The scaler is kind of a dummy entity.
The problem is that different subdevices cannot be connected directly to a v4l entity, because the ISC does cropping on the incoming frame. The ISC has a limit on the total number of pixels per frame, thus it will do a crop.Quoted 6 lines
What is this limit related to ? I don't know the ISC internals, but is the limitation due to the overall image size, or maybe it's due to some line buffers being limited in the size they can transfer to memory ? Is it a limitation on the size of the image in pixels, or is it due to a limitation of the processing bandwidth on the input interface and thus depends on the actual size of the image in bytes ?
Limitation is both on line size and full image size. From my understanding, is that the internal SRAM is limited (8 Mpixels with a small safe buffer on top in the case of sama7g5 )
This SRAM could be used as 800000x10 for example, or 3200x2464 for another example. However, I am not sure if a real line of 800,000 pixels make sense, or if there is another internal mechanism in the ISC that stops it.
In any case, the ISC is cropping the incoming frame from aaaXbbb up to the maximum frame that it can handleQuoted 10 lines
Quoted 6 lines
So, if I allow e.g. an entity that sends 3280x2464 , when the ISC can only handle 3264x2464 , then we have two choices:
1/ crop it and output the 3264x2464 -> cannot, because userspace expects 3280x2464, and the whole pipeline fails to configure, there is a mismatch between/dev/videooutput and what the whole pipeline producesI'm confused in first place. If the ISC cannot output anything larger than 3264x2464, then if userspace sets a 3280x2464 size on the ISC, this should be adjusted to 3264x2464.
Quoted 9 lines
yes, that is true. However, without an entity that says 'I am doing cropping', the media pipeline will fail, because previous entity will output 3280x2464 .
The conclusion to create a scaler entitity was done during my last year discussions on the IRC. I could not find a way to perform this cropping at the v4l entity side, and one reason is that the v4l entity does not support what I needed , or at least that was my understanding. You claim that the v4l entity could perform the cropping in the same entity ?I thought so, but I haven't tried to be honest
Quoted 10 lines
Let's assume for a minute that it could do this. Even so, I would prefer to have a separate scaler entity to do this , for several reasons:
-> I would like to be able to crop the frame arbitrarily if I want to, and the scaler would allow me to do this easier
-> it would be more obvious that the frame is cropped by having this entity there
-> in fact some of the ISC versions really have a down scaler, that I have not implemented yet. So it would be useful to already have this in place to be able to scale down with it at a later time.I think this last reason is enough to have an entity plumbed in already
Quoted 8 lines
Quoted 8 lines
Or is the ISC limitation on the -input- side ?
Quoted 3 lines
2/ deny the link, and ask the subdevice to produce the 3264x2464 which the ISC can handle at most -> cannot , because the sensor let's say can *only* produce 3280x2464 and *any* other frame size returns an error.
Quoted 6 lines
Quoted 1 line
buggy sensor I would say :)
I cannot tell this to the customers, and I cannot fail the whole pipeline because the sensor does not crop 16 pixels, when in fact I can do this very easily on the ISC side.
Fair enough :)
Quoted 85 lines
Quoted 11 lines
Quoted 7 lines
The only solution to make both worlds happy, is to have a dummy entity called 'scaler' which in fact now it only performs a simple crop. It accepts any frame size at input (hence the 10000x10000 which you saw earlier), and outputs at most 3264x2464 the
isc_max_heightandisc_max_width.I have a maybe dumb question: can the cropping operation be modeled by applying a
TGT_CROPrectangle (whose_BOUNDsize is 3280x2464) to theatmel_isc_commonentity sink pad and avoid the scaler completely ?I am not sure. This is what I wanted initially. But then I thought about the three reasons stated above for the use of the scaler entity, and discussed with folks on the IRC, and come up with the solution for the scaler entity
Quoted 68 lines
Quoted 67 lines
So to answer your question, the isc scaler is a software model for a simple cropping, that would make media controller happy, and capture software happy.
Here it how it looks :
Device topology
- entity 1:atmel_isc_scaler(2 pads, 2 links)type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev0 pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb crop.bounds:(0,0)/3264x2464 crop:(0,0)/3264x2464] <- "csi2dc":1 [ENABLED,IMMUTABLE] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] -> "atmel_isc_common":0 [ENABLED,IMMUTABLE]- entity 4: csi2dc (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev1 pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] <- "dw-csi.0":1 [ENABLED] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb] -> "atmel_isc_scaler":0 [ENABLED,IMMUTABLE]- entity 7: dw-csi.0 (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev2 pad0: Sink [fmt:SRGGB10_1X10/3280x2464] <- "imx219 1-0010":0 [ENABLED] pad1: Source [fmt:SRGGB10_1X10/3280x2464] -> "csi2dc":0 [ENABLED]- entity 12: imx219 1-0010 (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0 device node name/dev/v4l-subdev3
pad0: Source
[fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb
xfer:srgb ycbcr:601 quantization:full-rangecrop.bounds:(8,8)/3280x2464 crop:(8,8)/3280x2464] -> "dw-csi.0":0 [ENABLED]- entity 24:
atmel_isc_common(1 pad, 1 link)
type Node subtype V4L flags 1
device node name/dev/video0
pad0: Sink
<- "atmel_isc_scaler":1 [ENABLED,IMMUTABLE]Scaler does this one cute little thing :
pad0: Sink [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb crop.bounds:(0,0)/3264x2464 crop:(0,0)/3264x2464] <- "csi2dc":1 [ENABLED,IMMUTABLE] pad1: Source [fmt:SRGGB10_1X10/3280x2464 field:none colorspace:srgb]Quoted 7 lines
Quoted 3 lines
Shouldn't this be 3264x2464 as that's what the entity outputs after the cropping ? And shouldn't then be 3264x2464 the size output from the video device too ?
That's right.
I don't know why the source format for the scaler is still 3280x2464. Maybe there is a bug ong_fmtfor it.. have to check it.Thanks, I think this should be fixed ten
Quoted 17 lines
Anyway, the video format is like this :
# v4l2-ctl --get-fmt-video Format Video Capture: Width/Height : 3264/2464 Pixel Format : 'RGBP' (16-bit RGB 5-6-5) Field : None Bytes per Line : 6528 Size Image : 16084992 Colorspace : sRGB Transfer Function : Default (maps to sRGB) YCbCr/HSV Encoding: Default (maps to ITU-R 601) Quantization : Default (maps to Full Range) Flags : #Quoted 9 lines
and initializing the pipeline looks fine from user perspective. The scaler is in fact the solution to make this pipeline work with libcamera.
I found this cropping to be an issue in media controller when trying it with libcamera. Otherwise, the other user space apps which I was using never complained that anything was wrong
Libcamera simply refuses to acknowledge the pipeline if the video output is 3264x2464 but there is no entity that changes the format from 3280x2464 downQuoted 2 lines
Not sure why libcamera should play a role there. Isn't it the media pipeline validation that complains and returns an
-EPIPE?I tried this before the
media_pipeline_start.
Perhaps libcamera was doing some similar checks, anyway, it was complaining and not adding the stream, and reported no usable streams/cameraOnce I will rework some of the things according to your review I will run libcamera again to see what it complains about
Quoted 474 lines
Quoted 93 lines
Quoted 73 lines
Quoted 63 lines
-> "atmel_isc_common":0 [ENABLED,IMMUTABLE]Which is what we needed.
Quoted 19 lines
Quoted 6 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ +return 0 ; +} + +static int isc_scaler_enum_mbus_code(struct v4l2_subdev *sd,drivers/media/platform/atmel/atmel-isc-scaler.c line 105
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+struct v4l2_subdev_state *sd_state, + struct v4l2_subdev_mbus_code_enum *code) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); +int supported_index =0 ; +int i; + +for (i =0 ; i < isc->formats_list_size; i++) { +if (!isc->formats_list[i].sd_support) +continue ;The
sd_supportflag still doesn't click in my head.Quoted 2 lines
Shouldn't the enumeration of available formats on the scaler do not depend on the sensor supproted formats ?
You're right. I will have to check it again.
Quoted 30 lines
Quoted 16 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +15
-0+if (supported_index == code->index) { + code->code = isc->formats_list[i].mbus_code; +return 0 ; + } + supported_index++; + } + +return -EINVAL ; +} + +static int isc_scaler_g_sel (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state, +struct v4l2_subdev_selection *sel) +{ + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd);drivers/media/platform/atmel/atmel-isc-scaler.c line 130
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +if (sel->pad ==ISC_SCALER_PAD_SOURCE ) +return -EINVAL ; + +if (sel->target !=V4L2_SEL_TGT_CROP_BOUNDS && + sel->target !=V4L2_SEL_TGT_CROP ) +return -EINVAL ; + + sel->r.height = isc->max_height; + sel->r.width = isc->max_width;The
CROP_BOUNDSshould be set to the same size as the sink pad image format, as it represents the maximum valid crop rectangle.Quoted 9 lines
Quoted 3 lines
TGT_CROPshould report the configured crop rectangle which can be intiialized to the same size asCROP_BOUNDS, if my understanding of the spec is correctSo you would like to have this differentiated, and report the
CROP_BOUNDSto whatever is on the sink pad, and theTGT_CROPto what is reported now, the maximum size of the ISC frame .
My understanding is correct ?Quoted 6 lines
I didn't know you have an HW limitation, so your
_BOUNDSis not the input image size but rather 3264x2464 ( ==max_widthxmax_height).What I meant is that
_BOUNDSshould report the maximum rectangle size that can be applied to the_CROPtarget. In you case you have an HW limitation 3264x2464 and that's the largest rectangle you can apply.So the CROP should be at 3264x2464
Quoted 2 lines
TGT_CROPcan be initialized to the same as_BOUND, but if you implements_selectionit should report what has been there applied.and BOUND to actual frame size ?
Quoted 2 lines
But as you don't implement
s_selectionyet, I think this is fine for now. Maybe a little comment ?It could also be the case where e.g. the sensor is outputting 1920x1080, in this case the scaler would do nothing.
CROP is still 3264x2464 and BOUND in this case is 1920x1080 ? If the sensor is outputting 1920x1080, this format comes directly from the sensor (the sensor is cropping it from 3280x2464 or not... it's the sensor's problem)Quoted 1 line
Quoted 4 lines
Quoted 2 lines
Also, is
set->rzeroed by the framework before getting here ? Otherwise you should set r.left and r.top to 0If these are redundant, no problem to remove them
I was actually confused. I was suggesting you to add...
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 133
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +3
-0+ + sel->r.left =0 ; + sel->r.top =0 ;These ^
So nothing to change. Sorry I've missed them
Thanks
jQuoted 360 lines
Quoted 359 lines
Quoted 49 lines
Quoted 40 lines
Quoted 12 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +12
-0+ +return 0 ; +} + +static int isc_scaler_init_cfg (struct v4l2_subdev *sd, +struct v4l2_subdev_state *sd_state) +{ +struct v4l2_mbus_framefmt *v4l2_try_fmt = +v4l2_subdev_get_try_format (sd, sd_state,0 ); + struct isc_device *isc = container_of(sd, struct isc_device, scaler_sd); + + *v4l2_try_fmt = isc->scaler_format;drivers/media/platform/atmel/atmel-isc-scaler.c line 155
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +10
-0+ +return 0 ; +} + +static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = { + .enum_mbus_code = isc_scaler_enum_mbus_code, + .set_fmt = isc_scaler_set_fmt, + .get_fmt = isc_scaler_get_fmt, + .get_selection = isc_scaler_g_sel, + .init_cfg = isc_scaler_init_cfg,.link_validate =
v4l2_subdev_link_validate_default,To have the formats at the end of links that point to this entity validated (I think the framework already calls it if not set though, please check v4l2-subdev.c:v4l2_subdev_link_validate())
Quoted 11 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +11
-0+}; + +static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = { + .pad = &isc_scaler_pad_ops, +}; + +int isc_scaler_init (struct isc_device *isc) +{ +int ret; + +v4l2_subdev_init (&isc->scaler_sd, &xisc_scaler_subdev_ops);Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 171
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +5
-0+ + isc->scaler_sd.owner =THIS_MODULE ; + isc->scaler_sd.dev = isc->dev; +snprintf (isc->scaler_sd.name,sizeof (isc->scaler_sd.name), +"atmel_isc_scaler" );I would drop 'atmel' for brevity, unless other entities have this prefix set already
Quoted 3 lines
The v4l entity takes it's name from the module name, which is
atmel_isc_common, so I thought to keep the prefixAck!
Quoted 189 lines
Quoted 70 lines
Quoted 70 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +69
-0+ + isc->scaler_sd.flags |=V4L2_SUBDEV_FL_HAS_DEVNODE ; + isc->scaler_sd.entity.function =MEDIA_ENT_F_PROC_VIDEO_SCALER ; + isc->scaler_pads[ISC_SCALER_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; + isc->scaler_pads[ISC_SCALER_PAD_SOURCE ].flags =MEDIA_PAD_FL_SOURCE ; + + isc->scaler_format.height = isc->max_height; + isc->scaler_format.width = isc->max_width; + isc->scaler_format.code = isc->formats_list[0 ].mbus_code; + isc->scaler_format.colorspace =V4L2_COLORSPACE_SRGB ; + isc->scaler_format.field =V4L2_FIELD_NONE ; + isc->scaler_format.ycbcr_enc =V4L2_YCBCR_ENC_DEFAULT ; + isc->scaler_format.quantization =V4L2_QUANTIZATION_DEFAULT ; + isc->scaler_format.xfer_func =V4L2_XFER_FUNC_DEFAULT ; + + ret =media_entity_pads_init (&isc->scaler_sd.entity, +ISC_SCALER_PADS_NUM , + isc->scaler_pads); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd media entity init failed\n" ); +return ret; + } + ret =v4l2_device_register_subdev (&isc->v4l2_dev, &isc->scaler_sd); +if (ret <0 ) { +dev_err (isc->dev,"scaler sd failed to register subdev\n" ); +return ret; + } + +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_init); + +int isc_scaler_link (struct isc_device *isc) +{ +int ret; + + ret =media_create_pad_link (&isc->current_subdev->sd->entity, + isc->remote_pad, &isc->scaler_sd.entity, +ISC_SCALER_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->current_subdev->sd->entity.name, + isc->scaler_sd.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , + isc->current_subdev->sd->name, isc->remote_pad); + + ret =media_create_pad_link (&isc->scaler_sd.entity, +ISC_SCALER_PAD_SOURCE , + &isc->video_dev.entity,ISC_PAD_SINK , +MEDIA_LNK_FL_ENABLED | +MEDIA_LNK_FL_IMMUTABLE ); + +if (ret <0 ) { +v4l2_err (&isc->v4l2_dev, +"Failed to create pad link: %s to %s\n" , + isc->scaler_sd.entity.name, + isc->video_dev.entity.name); +return ret; + } + +dev_dbg (isc->dev,"link with %s pad: %d\n" , isc->scaler_sd.name, +ISC_SCALER_PAD_SOURCE );Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-scaler.c line 244
Quoted 4 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +4
-0+ +return ret; +} +EXPORT_SYMBOL_GPL (isc_scaler_link);From the DT graph point of view, the ISC appears as a single block
with an CSI-2 input port. It is then in charge of parsing the .dts and add to its notifier the image sensor remote subdevice.Actually it's only a parallel port.
And it can be connected either to a sensor directly or to the csi2dc bridge. (the csi2dc bridge can be connected to a CSI-2 receiver)Quoted 4 lines
When the sensor subdev registers and the ISC notifier completes, the scaler entity which was initialized at
isc_probe()time is linked in between the ISC and the image sensor immutably.yes, because this cannot change at runtime, and usually, it can't change without altering the board hardware. (at least this is what my understanding of immutability is )
Quoted 9 lines
I think it is fine for now, but I wonder if you plan to plumb more components between the ISC video node and the sensor, if it's not worth changing the DT bindings and their parsing logic to separate the CSI-2 receiver from the ISC, whcih can create its media graph at probe time and have the CSI-2 receiver entity as downstream remote. I think I need to get to know the ISC better to really have an idea. For now, this seems ok to me, but please check with maintainers if this is fine with them.
In the XISC case (sama7g5-isc), the csi2dc receiver is the subdev (so, the remote subdev).
The ISC will register a scaler, and connect the subdev to this scaler first, and then, the scaler to the isc itself (the v4l entity).Quoted 45 lines
Quoted 45 lines
drivers/media/platform/atmel/atmel-isc-scaler.c new file +1
-0+drivers/media/platform/atmel/atmel-isc.h +21
-0@@ -183,6 +183,17 @@ struct isc_reg_offsets {u32 his_entry; }; +enum isc_mc_pads { +ISC_PAD_SINK =0 , +ISC_PADS_NUM =1 , +}; + +enum isc_scaler_pads { +ISC_SCALER_PAD_SINK =0 , +ISC_SCALER_PAD_SOURCE =1 , +ISC_SCALER_PADS_NUM =2 , +}; +/* * struct isc_device - ISC device driver data/config struct * @regmap: Register map @@ -257,6 +268,12 @@ struct isc_reg_offsets {* be used as an input to the controller * @controller_formats_size: size of controller_formats array * @formats_list_size: size of formats_list array +* @pads: media controller pads for isc video entity +* @mdev: media device that is registered by the isc +* @remote_pad: remote pad on the connected subdevice +* @scaler_sd: subdevice for the scaler that isc registers +* @scaler_pads: media controller pads for the scaler subdevice +* @scaler_format: current format for the scaler subdevice */ struct isc_device {struct regmap *regmap;@@ -344,6 +361,19 @@ struct isc_device {struct isc_format *formats_list;u32 controller_formats_size;u32 formats_list_size; + +struct { +struct media_pad pads[ISC_PADS_NUM ]; +struct media_device mdev;Quoted 8 lines
drivers/media/platform/atmel/atmel-isc.h struct isc_device { · line 375
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc.h +8
-0+ +u32 remote_pad; + }; + +struct { +struct v4l2_subdev scaler_sd; +struct media_pad scaler_pads[ISC_SCALER_PADS_NUM ]; +struct v4l2_mbus_framefmt scaler_format;Here are the scaler stuff which I added, in the same bulk struct for the whole isc device
Quoted 24 lines
Quoted 24 lines
drivers/media/platform/atmel/atmel-isc.h +8
-0+ }; };extern const struct regmap_config isc_regmap_config;@@ -355,4 +385,11 @@ int isc_clk_init(struct isc_device *isc);void isc_subdev_cleanup (struct isc_device *isc);void isc_clk_cleanup (struct isc_device *isc); +int isc_scaler_link (struct isc_device *isc); +int isc_scaler_init (struct isc_device *isc); +int isc_mc_init (struct isc_device *isc,u32 ver); +void isc_mc_cleanup (struct isc_device *isc); + +struct isc_format *isc_find_format_by_code (struct isc_device *isc, +unsigned int code,int *index);#endif drivers/media/platform/atmel/atmel-sama5d2-isc.c +0
-0@@ -553,6 +553,12 @@ static int atmel_isc_probe(struct platform_device *pdev)break ; }Quoted 8 lines
Quoted 5 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c static int atmel_isc_probe(struct platform_device *pdev) · line 556
Quoted 1 line
drivers/media/platform/atmel/atmel-sama5d2-isc.c +1
-0+regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);I am surprised you can read a register before
runtime_pmis intialized!Actually I can read any register after the moment of starting the clock on the hardware block.
Ah right then
Quoted 3 lines
Maybe the starting and stopping of the clock needs to be moved to the
runtime_pmcalls, but this is another story, not related to the media controller.It's fine, but maybe worth recording with a todo ?
Thanks
jQuoted 95 lines
I moved this line because I had to pass the version to the
isc_mc_initcallQuoted 2 lines
Thanks
jThanks for reviewing !
EugenQuoted 87 lines
Quoted 87 lines
drivers/media/platform/atmel/atmel-sama5d2-isc.c +11
-2+ + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev);@@ -562,7 +568,7 @@ static int atmel_isc_probe(struct platform_device *pdev) ret =clk_prepare_enable (isc->ispck);if (ret) {dev_err (dev,"failed to enable ispck: %d\n" , ret); - goto cleanup_subdev; + goto isc_probe_mc_init_err; }/* ispck should be greater or equal to hclock */ @@ -572,7 +578,6 @@ static int atmel_isc_probe(struct platform_device *pdev) goto unprepare_clk; } -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip ISC version %x\n" , ver);return 0 ;@@ -580,6 +585,9 @@ static int atmel_isc_probe(struct platform_device *pdev) unprepare_clk:clk_disable_unprepare (isc->ispck); +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -600,6 +608,8 @@ static int atmel_isc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev);drivers/media/platform/atmel/atmel-sama7g5-isc.c +11
-2@@ -547,15 +547,23 @@ static int microchip_xisc_probe(struct platform_device *pdev)break ; } +regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver); + + ret =isc_mc_init (isc, ver); +if (ret <0 ) + goto isc_probe_mc_init_err; +pm_runtime_set_active (dev);pm_runtime_enable (dev);pm_request_idle (dev); -regmap_read (isc->regmap,ISC_VERSION + isc->offsets.version, &ver);dev_info (dev,"Microchip XISC version %x\n" , ver);return 0 ; +isc_probe_mc_init_err: +isc_mc_cleanup (isc); + cleanup_subdev:isc_subdev_cleanup (isc);@@ -576,6 +584,8 @@ static int microchip_xisc_remove(struct platform_device *pdev)pm_runtime_disable (&pdev->dev); +isc_mc_cleanup (isc); +isc_subdev_cleanup (isc);v4l2_device_unregister (&isc->v4l2_dev); --2.25.1
PATCH v4 4/11 media: atmel: atmel-sama5d2-isc: fix wrong mask in YUYV format check
In reply to Eugen HristevWhile this does not happen in production, this check should be done versus the mask, as checking with the YCYC value may not include some bits that may be set.
Is it correct and safe to check the whole mask.
--- drivers/media/platform/atmel/atmel-sama5d2-isc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/media/platform/atmel/atmel-sama5d2-isc.c +1 -1
@@ -291,7 +291,7 @@ static void isc_sama5d2_config_rlp(struct isc_device *isc)* Thus, if the YCYC mode is selected, replace it with the * sama5d2-compliant mode which is YYCC . */ -if ((rlp_mode &ISC_RLP_CFG_MODE_YCYC ) ==ISC_RLP_CFG_MODE_YCYC ) { +if ((rlp_mode &ISC_RLP_CFG_MODE_MASK ) ==ISC_RLP_CFG_MODE_YCYC ) { rlp_mode &= ~ISC_RLP_CFG_MODE_MASK ; rlp_mode |=ISC_RLP_CFG_MODE_YYCC ; }
Signature · 1 line
2.25.1
PATCH v4 5/11 media: atmel: atmel-isc-base: use mutex to lock awb workqueue from streaming
In reply to Eugen HristevThe AWB workqueue runs in a kernel thread and needs to be synchronized w.r.t. the streaming status.
It is possible that streaming is stopped while the AWB workq is running. In this case it is likely that the check for vb2_start_streaming_called is done at one point in time, but the AWB computations are done later, including a call to isc_update_profile, which requires streaming to be started. Thus , isc_update_profile will fail if during this operation sequence the streaming was stopped.
To solve this issue, a mutex is added, that will serialize the awb work and streaming stopping, with the mention that either streaming is stopped completely including termination of the last frame is done, and after that the AWB work can check stream status and stop; either first AWB work is completed and after that the streaming can stop correctly.
The awb spin lock cannot be used since this spinlock is taken in the same context and using it in the stop streaming will result in a recursion BUG.
--- drivers/media/platform/atmel/atmel-isc-base.c | 31 ++++++++++++++++--- drivers/media/platform/atmel/atmel-isc.h | 1 + 2 files changed, 28 insertions(+), 4 deletions(-)
drivers/media/platform/atmel/atmel-isc-base.c +27 -4
@@ -400,6 +400,7 @@ static void isc_stop_streaming(struct vb2_queue *vq)struct isc_buffer *buf;int ret; +mutex_lock (&isc->awb_mutex);v4l2_ctrl_activate (isc->do_wb_ctrl, false);/* Wait until the end of the current frame */
@@ -407,6 +408,8 @@ static void isc_stop_streaming(struct vb2_queue *vq)v4l2_err (&isc->v4l2_dev,"Timeout waiting for end of the capture\n" ); +mutex_unlock (&isc->awb_mutex); +/* Disable DMA interrupt */ regmap_write (isc->regmap,ISC_INTDIS ,ISC_INT_DDONE );
@@ -1395,10 +1398,6 @@ static void isc_awb_work(struct work_struct *w)u32 min, max;int ret; -/* streaming is not active anymore */ -if (!vb2_start_streaming_called (&isc->vb2_vidq)) -return ; -if (ctrls->hist_stat !=HIST_ENABLED )return ;
@@ -1453,7 +1452,24 @@ static void isc_awb_work(struct work_struct *w) }regmap_write (regmap,ISC_HIS_CFG + isc->offsets.his, hist_id | baysel |ISC_HIS_CFG_RAR ); + +/* +* We have to make sure the streaming has not stopped meanwhile. +* ISC requires a frame to clock the internal profile update. +* To avoid issues, lock the sequence with a mutex +*/ +mutex_lock (&isc->awb_mutex); + +/* streaming is not active anymore */ +if (!vb2_start_streaming_called (&isc->vb2_vidq)) { +mutex_unlock (&isc->awb_mutex); +return ; + }; +isc_update_profile (isc); + +mutex_unlock (&isc->awb_mutex); +/* if awb has been disabled, we don't need to start another histogram */ if (ctrls->awb)regmap_write (regmap,ISC_CTRLEN ,ISC_CTRL_HISREQ );
@@ -1532,6 +1548,8 @@ static int isc_s_awb_ctrl(struct v4l2_ctrl *ctrl)isc_update_awb_ctrls (isc); +mutex_lock (&isc->awb_mutex); +if (vb2_start_streaming_called (&isc->vb2_vidq)) {/* * If we are streaming, we can update profile to
@@ -1546,6 +1564,7 @@ static int isc_s_awb_ctrl(struct v4l2_ctrl *ctrl)*/ v4l2_ctrl_activate (isc->do_wb_ctrl, false); } +mutex_unlock (&isc->awb_mutex);/* if we have autowhitebalance on, start histogram procedure */ if (ctrls->awb == ISC_WB_AUTO &&
@@ -1738,6 +1757,7 @@ static void isc_async_unbind(struct v4l2_async_notifier *notifier, {struct isc_device *isc =container_of (notifier->v4l2_dev, struct isc_device, v4l2_dev); +mutex_destroy (&isc->awb_mutex);cancel_work_sync (&isc->awb_work);video_unregister_device (&isc->video_dev);v4l2_ctrl_handler_free (&isc->ctrls.handler);
@@ -1848,6 +1868,8 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier) isc->current_subdev =container_of (notifier, struct isc_subdev_entity, notifier);mutex_init (&isc->lock); +mutex_init (&isc->awb_mutex); +init_completion (&isc->comp);/* Initialize videobuf2 queue */
@@ -1927,6 +1949,7 @@ static int isc_async_complete(struct v4l2_async_notifier *notifier)video_unregister_device (vdev); isc_async_complete_err: +mutex_destroy (&isc->awb_mutex);mutex_destroy (&isc->lock);return ret; }
drivers/media/platform/atmel/atmel-isc.h +1 -0
@@ -305,6 +305,7 @@ struct isc_device {struct work_struct awb_work;struct mutex lock;/* serialize access to file operations */ +struct mutex awb_mutex;/* serialize access to streaming status from awb work queue */ spinlock_t awb_lock;/* serialize access to DMA buffers from awb work queue */ struct regmap_field *pipeline[ISC_PIPE_LINE_NODE_NUM ];
Signature · 1 line
2.25.1
PATCH v4 6/11 media: atmel: atmel-isc: compact the controller formats list
In reply to Eugen HristevCompact the list array to be more readable.
No other changes, only cosmetic.
--- .../media/platform/atmel/atmel-sama5d2-isc.c | 51 ++++++---------- .../media/platform/atmel/atmel-sama7g5-isc.c | 60 +++++++------------ 2 files changed, 37 insertions(+), 74 deletions(-)
drivers/media/platform/atmel/atmel-sama5d2-isc.c +17 -34
@@ -60,56 +60,39 @@static const struct isc_format sama5d2_controller_formats[] = { { .fourcc =V4L2_PIX_FMT_ARGB444 , - }, - { + }, { .fourcc =V4L2_PIX_FMT_ARGB555 , - }, - { + }, { .fourcc =V4L2_PIX_FMT_RGB565 , - }, - { + }, { .fourcc =V4L2_PIX_FMT_ABGR32 , - }, - { + }, { .fourcc =V4L2_PIX_FMT_XBGR32 , - }, - { + }, { .fourcc =V4L2_PIX_FMT_YUV420 , - }, - { + }, { .fourcc =V4L2_PIX_FMT_YUYV , - }, - { + }, { .fourcc =V4L2_PIX_FMT_YUV422P , - }, - { + }, { .fourcc =V4L2_PIX_FMT_GREY , - }, - { + }, { .fourcc =V4L2_PIX_FMT_Y10 , - }, - { + }, { .fourcc =V4L2_PIX_FMT_SBGGR8 , - }, - { + }, { .fourcc =V4L2_PIX_FMT_SGBRG8 , - }, - { + }, { .fourcc =V4L2_PIX_FMT_SGRBG8 , - }, - { + }, { .fourcc =V4L2_PIX_FMT_SRGGB8 , - }, - { + }, { .fourcc =V4L2_PIX_FMT_SBGGR10 , - }, - { + }, { .fourcc =V4L2_PIX_FMT_SGBRG10 , - }, - { + }, { .fourcc =V4L2_PIX_FMT_SGRBG10 , - }, - { + }, { .fourcc =V4L2_PIX_FMT_SRGGB10 , }, };
drivers/media/platform/atmel/atmel-sama7g5-isc.c +20 -40
85 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v4 7/11 media: atmel: atmel-isc: change format propagation to subdev into only verification
In reply to Eugen HristevAs a top MC video driver, the atmel-isc should not propagate the format to the subdevice.
It should rather check at start_streaming() time if the subdev is properly configured with a compatible format.
Removed the whole format finding logic, and reworked the format verification at start_streaming time, such that the ISC will return an error if the subdevice is not properly configured. To achieve this, media_pipeline_start is called and a link_validate callback is created to check the formats. With this being done, the module parameter 'sensor_preferred' makes no sense anymore. The ISC should not decide which format the sensor is using. The ISC should only cope with the situation and inform userspace if the streaming is possible in the current configuration.
The redesign of the format propagation has also risen the question of the enumfmt callback. If enumfmt is called with an mbus_code, the enumfmt handler should only return the formats that are supported for this mbus_code. To make it more easy to understand the formats, changed the report order to report first the native formats, and after that the formats that the ISC can convert to.
---
Changes in v4:
- moved validation code into link_validate and used media_pipeline_start
- merged this patch with the enum_fmt patch which was previously in v3 of the series
Changes in v3:
- clamp to maximum resolution once the frame size from the subdev is found
drivers/media/platform/atmel/atmel-isc-base.c | 335 ++++++++++-------- .../media/platform/atmel/atmel-isc-scaler.c | 5 + drivers/media/platform/atmel/atmel-isc.h | 3 + 3 files changed, 191 insertions(+), 152 deletions(-)
drivers/media/platform/atmel/atmel-isc-base.c +183 -152
@@ -36,11 +36,6 @@ static unsigned int debug;module_param (debug, int,0644 );MODULE_PARM_DESC (debug,"debug level (0-2)" ); -static unsigned int sensor_preferred =1 ; -module_param (sensor_preferred, uint,0644 ); -MODULE_PARM_DESC (sensor_preferred, -"Sensor is preferred to output the specified format (1-on 0-off), default 1" ); -#define ISC_IS_FORMAT_RAW (mbus_code) \ (((mbus_code) & 0xf000) == 0x3000)
@@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)unsigned long flags;int ret; + ret =media_pipeline_start (&isc->video_dev.entity, &isc->mpipe); +if (ret) + goto err_pipeline_start; +/* Enable stream on the sub device */ ret =v4l2_subdev_call (isc->current_subdev->sd, video, s_stream,1 );if (ret && ret != -ENOIOCTLCMD ) {
@@ -384,6 +383,9 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)v4l2_subdev_call (isc->current_subdev->sd, video, s_stream,0 ); err_start_stream: +media_pipeline_stop (&isc->video_dev.entity); + +err_pipeline_start:spin_lock_irqsave (&isc->dma_queue_lock, flags);list_for_each_entry (buf, &isc->dma_queue, list)vb2_buffer_done (&buf->vb.vb2_buf,VB2_BUF_STATE_QUEUED );
@@ -420,6 +422,9 @@ static void isc_stop_streaming(struct vb2_queue *vq)if (ret && ret != -ENOIOCTLCMD )v4l2_err (&isc->v4l2_dev,"stream off failed in subdev\n" ); +/* Stop media pipeline */ +media_pipeline_stop (&isc->video_dev.entity); +/* Release all active buffers */ spin_lock_irqsave (&isc->dma_queue_lock, flags);if (unlikely (isc->cur_frm)) {
@@ -496,21 +501,56 @@ static int isc_enum_fmt_vid_cap(struct file *file, void *priv,u32 index = f->index;u32 i, supported_index; -if (index < isc->controller_formats_size) { - f->pixelformat = isc->controller_formats[index].fourcc; -return 0 ; + supported_index =0 ; + +for (i =0 ; i < isc->formats_list_size; i++) { +if (!isc->formats_list[i].sd_support) +continue ; +/* +* If specific mbus_code is requested, provide only +* supported formats with this mbus code +*/ +if (f->mbus_code && f->mbus_code != + isc->formats_list[i].mbus_code) +continue ; +if (supported_index == index) { + f->pixelformat = isc->formats_list[i].fourcc; +return 0 ; + } + supported_index++; } - index -= isc->controller_formats_size; +/* +* If the sensor does not support this mbus_code whatsoever, +* there is no reason to advertise any of our output formats +*/ +if (supported_index ==0 ) +return -EINVAL ; +/* +* If the sensor uses a format that is not raw, then we cannot +* convert it to any of the formats that we usually can with a +* RAW sensor. Thus, do not advertise them. +*/ +if (isc->config.sd_format && + !ISC_IS_FORMAT_RAW (isc->config.sd_format->mbus_code)) +return -EINVAL ; + +/* +* Iterate again through the formats that we can convert to. +* However, to avoid duplicates, skip the formats that +* the sensor already supports directly +*/ + index -= supported_index; supported_index =0 ; -for (i =0 ; i < isc->formats_list_size; i++) { -if (!ISC_IS_FORMAT_RAW (isc->formats_list[i].mbus_code) || - !isc->formats_list[i].sd_support) +for (i =0 ; i < isc->controller_formats_size; i++) { +/* if this format is already supported by sensor, skip it */ +if (find_format_by_fourcc (isc, isc->controller_formats[i].fourcc))continue ;if (supported_index == index) { - f->pixelformat = isc->formats_list[i].fourcc; + f->pixelformat = + isc->controller_formats[i].fourcc;return 0 ; } supported_index++;
@@ -581,20 +621,30 @@ static int isc_try_validate_formats(struct isc_device *isc)break ;default :/* any other different formats are not supported */ +v4l2_err (&isc->v4l2_dev,"Requested unsupported format.\n" ); ret = -EINVAL ; }v4l2_dbg (1 , debug, &isc->v4l2_dev,"Format validation, requested rgb=%u, yuv=%u, grey=%u, bayer=%u\n" , rgb, yuv, grey, bayer); -/* we cannot output RAW if we do not receive RAW */ -if ((bayer) && !ISC_IS_FORMAT_RAW (isc->try_config.sd_format->mbus_code)) +if ((bayer) && + !ISC_IS_FORMAT_RAW (isc->try_config.sd_format->mbus_code)) { +v4l2_err (&isc->v4l2_dev,"Cannot output RAW if we do not receive RAW.\n" );return -EINVAL ; + } -/* we cannot output GREY if we do not receive RAW/GREY */ if (grey && !ISC_IS_FORMAT_RAW (isc->try_config.sd_format->mbus_code) && - !ISC_IS_FORMAT_GREY (isc->try_config.sd_format->mbus_code)) + !ISC_IS_FORMAT_GREY (isc->try_config.sd_format->mbus_code)) { +v4l2_err (&isc->v4l2_dev,"Cannot output GREY if we do not receive RAW/GREY.\n" ); +return -EINVAL ; + } + +if ((rgb || bayer || yuv) && +ISC_IS_FORMAT_GREY (isc->try_config.sd_format->mbus_code)) { +v4l2_err (&isc->v4l2_dev,"Cannot convert GREY to another format.\n" );return -EINVAL ; + }return ret; }
@@ -822,7 +872,7 @@ static void isc_try_fse(struct isc_device *isc,* If we do not know yet which format the subdev is using, we cannot * do anything. */ -if (!isc->try_config.sd_format) +if (!isc->config.sd_format)return ; fse.code = isc->try_config.sd_format->mbus_code;
@@ -843,180 +893,141 @@ static void isc_try_fse(struct isc_device *isc, } } -static int isc_try_fmt (struct isc_device *isc,struct v4l2_format *f, -u32 *code) +static int isc_try_fmt (struct isc_device *isc,struct v4l2_format *f) { -int i; -struct isc_format *sd_fmt =NULL , *direct_fmt =NULL ;struct v4l2_pix_format *pixfmt = &f->fmt.pix; -struct v4l2_subdev_pad_config pad_cfg = {}; -struct v4l2_subdev_state pad_state = { - .pads = &pad_cfg - }; -struct v4l2_subdev_format format = { - .which =V4L2_SUBDEV_FORMAT_TRY , - }; -u32 mbus_code; -int ret; -bool rlp_dma_direct_dump = false; +unsigned int i;if (f->type !=V4L2_BUF_TYPE_VIDEO_CAPTURE )return -EINVAL ; -/* Step 1: find a RAW format that is supported */ -for (i =0 ; i < isc->num_user_formats; i++) { -if (ISC_IS_FORMAT_RAW (isc->user_formats[i]->mbus_code)) { - sd_fmt = isc->user_formats[i]; + isc->try_config.fourcc = isc->user_formats[0 ]->fourcc; + +/* find if the format requested is supported */ +for (i =0 ; i < isc->controller_formats_size; i++) +if (isc->controller_formats[i].fourcc == pixfmt->pixelformat) { + isc->try_config.fourcc = pixfmt->pixelformat;break ; } - } -/* Step 2: We can continue with this RAW format, or we can look -* for better: maybe sensor supports directly what we need. -*/ - direct_fmt =find_format_by_fourcc (isc, pixfmt->pixelformat); - -/* Step 3: We have both. We decide given the module parameter which -* one to use. -*/ -if (direct_fmt && sd_fmt && sensor_preferred) - sd_fmt = direct_fmt; - -/* Step 4: we do not have RAW but we have a direct format. Use it. */ -if (direct_fmt && !sd_fmt) - sd_fmt = direct_fmt; - -/* Step 5: if we are using a direct format, we need to package -* everything as 8 bit data and just dump it -*/ -if (sd_fmt == direct_fmt) - rlp_dma_direct_dump = true; - -/* Step 6: We have no format. This can happen if the userspace -* requests some weird/invalid format. -* In this case, default to whatever we have -*/ -if (!sd_fmt && !direct_fmt) { - sd_fmt = isc->user_formats[isc->num_user_formats -1 ]; -v4l2_dbg (1 , debug, &isc->v4l2_dev, -"Sensor not supporting %.4s, using %.4s\n" , - (char *)&pixfmt->pixelformat, (char *)&sd_fmt->fourcc); - } - -if (!sd_fmt) { - ret = -EINVAL ; - goto isc_try_fmt_err; - } - -/* Step 7: Print out what we decided for debugging */ -v4l2_dbg (1 , debug, &isc->v4l2_dev, -"Preferring to have sensor using format %.4s\n" , - (char *)&sd_fmt->fourcc); - -/* Step 8: at this moment we decided which format the subdev will use */ - isc->try_config.sd_format = sd_fmt; - -/* Limit to Atmel ISC hardware capabilities */ -if (pixfmt->width > isc->max_width) - pixfmt->width = isc->max_width; -if (pixfmt->height > isc->max_height) - pixfmt->height = isc->max_height; - -/* -* The mbus format is the one the subdev outputs. -* The pixels will be transferred in this format Sensor -> ISC -*/ - mbus_code = sd_fmt->mbus_code; - -/* -* Validate formats. If the required format is not OK, default to raw. -*/ - - isc->try_config.fourcc = pixfmt->pixelformat; - -if (isc_try_validate_formats (isc)) { - pixfmt->pixelformat = isc->try_config.fourcc = sd_fmt->fourcc; -/* Re-try to validate the new format */ - ret =isc_try_validate_formats (isc); -if (ret) - goto isc_try_fmt_err; - } - - ret =isc_try_configure_rlp_dma (isc, rlp_dma_direct_dump); -if (ret) - goto isc_try_fmt_err; - - ret =isc_try_configure_pipeline (isc); -if (ret) - goto isc_try_fmt_err; -/* Obtain frame sizes if possible to have crop requirements ready */ -isc_try_fse (isc, &pad_state); - -v4l2_fill_mbus_format (&format.format, pixfmt, mbus_code); - ret =v4l2_subdev_call (isc->current_subdev->sd, pad, set_fmt, - &pad_state, &format); -if (ret <0 ) - goto isc_try_fmt_subdev_err; +/* If we did not find the requested format, we will fallback here */ + pixfmt->pixelformat = isc->try_config.fourcc; + pixfmt->colorspace =V4L2_COLORSPACE_SRGB ; + pixfmt->field =V4L2_FIELD_NONE ; -v4l2_fill_pix_format (pixfmt, &format.format); +isc_try_configure_rlp_dma (isc, false);/* Limit to Atmel ISC hardware capabilities */ -if (pixfmt->width > isc->max_width) - pixfmt->width = isc->max_width; -if (pixfmt->height > isc->max_height) - pixfmt->height = isc->max_height; +v4l_bound_align_image (&pixfmt->width,16 , isc->max_width,0 , + &pixfmt->height,16 , isc->max_height,0 ,0 ); pixfmt->field =V4L2_FIELD_NONE ; pixfmt->bytesperline = (pixfmt->width * isc->try_config.bpp_v4l2) >>3 ; pixfmt->sizeimage = ((pixfmt->width * isc->try_config.bpp) >>3 ) * pixfmt->height; -if (code) - *code = mbus_code; + isc->try_fmt = *f;return 0 ; +} -isc_try_fmt_err: -v4l2_err (&isc->v4l2_dev,"Could not find any possible format for a working pipeline\n" ); -isc_try_fmt_subdev_err: -memset (&isc->try_config,0 ,sizeof (isc->try_config)); +static int isc_set_fmt (struct isc_device *isc,struct v4l2_format *f) +{ +isc_try_fmt (isc, f); -return ret; +/* make the try configuration active */ + isc->config = isc->try_config; + isc->fmt = isc->try_fmt; + +v4l2_dbg (1 , debug, &isc->v4l2_dev,"ISC set_fmt to %.4s @%dx%d\n" , + (char *)&f->fmt.pix.pixelformat, + f->fmt.pix.width, f->fmt.pix.height); + +return 0 ; } -static int isc_set_fmt (struct isc_device *isc,struct v4l2_format *f) +static int isc_validate (struct isc_device *isc) { +int ret; +int i; +struct isc_format *sd_fmt =NULL ; +struct v4l2_pix_format *pixfmt = &isc->fmt.fmt.pix;struct v4l2_subdev_format format = { .which =V4L2_SUBDEV_FORMAT_ACTIVE , + .pad = isc->remote_pad, + }; +struct v4l2_subdev_pad_config pad_cfg = {}; +struct v4l2_subdev_state pad_state = { + .pads = &pad_cfg, }; -u32 mbus_code =0 ; -int ret; - ret =isc_try_fmt (isc, f, &mbus_code); +/* Get current format from subdev */ + ret =v4l2_subdev_call (isc->current_subdev->sd, pad, get_fmt,NULL , + &format);if (ret)return ret; -v4l2_fill_mbus_format (&format.format, &f->fmt.pix, mbus_code); - ret =v4l2_subdev_call (isc->current_subdev->sd, pad, - set_fmt,NULL , &format); -if (ret <0 ) -return ret; +/* Identify the subdev's format configuration */ +for (i =0 ; i < isc->num_user_formats; i++) +if (isc->user_formats[i]->mbus_code == format.format.code) { + sd_fmt = isc->user_formats[i]; +break ; + } + +/* Check if the format is not supported */ +if (!sd_fmt) { +v4l2_err (&isc->v4l2_dev, +"Current subdevice is streaming a media bus code that is not supported 0x%x\n" , + format.format.code); +return -EPIPE ; + } + +/* At this moment we know which format the subdev will use */ + isc->try_config.sd_format = sd_fmt; + +/* If the sensor is not RAW, we can only do a direct dump */ +if (!ISC_IS_FORMAT_RAW (isc->try_config.sd_format->mbus_code)) +isc_try_configure_rlp_dma (isc, true);/* Limit to Atmel ISC hardware capabilities */ -if (f->fmt.pix.width > isc->max_width) - f->fmt.pix.width = isc->max_width; -if (f->fmt.pix.height > isc->max_height) - f->fmt.pix.height = isc->max_height; +v4l_bound_align_image (&format.format.width,16 , isc->max_width,0 , + &format.format.height,16 , isc->max_height,0 ,0 ); + +/* Check if the frame size is the same. Otherwise we may overflow */ +if (pixfmt->height != format.format.height || + pixfmt->width != format.format.width) { +v4l2_err (&isc->v4l2_dev, +"ISC not configured with the proper frame size: %dx%d\n" , + format.format.width, format.format.height); +return -EPIPE ; + } - isc->fmt = *f; +v4l2_dbg (1 , debug, &isc->v4l2_dev, +"Identified subdev using format %.4s with %dx%d %d bpp\n" , + (char *)&sd_fmt->fourcc, pixfmt->width, pixfmt->height, + isc->try_config.bpp); +/* Reset and restart AWB if the subdevice changed the format */ if (isc->try_config.sd_format && isc->config.sd_format && isc->try_config.sd_format != isc->config.sd_format) { isc->ctrls.hist_stat =HIST_INIT ;isc_reset_awb_ctrls (isc);isc_update_v4l2_ctrls (isc); } -/* make the try configuration active */ + +/* Validate formats */ + ret =isc_try_validate_formats (isc); +if (ret) +return ret; + +/* Obtain frame sizes if possible to have crop requirements ready */ +isc_try_fse (isc, &pad_state); + +/* Configure ISC pipeline for the config */ + ret =isc_try_configure_pipeline (isc); +if (ret) +return ret; + isc->config = isc->try_config;v4l2_dbg (1 , debug, &isc->v4l2_dev,"New ISC configuration in place\n" );
@@ -1040,7 +1051,7 @@ static int isc_try_fmt_vid_cap(struct file *file, void *priv, {struct isc_device *isc =video_drvdata (file); -return isc_try_fmt (isc, f,NULL ); +return isc_try_fmt (isc, f); } static int isc_enum_input(struct file *file, void *priv,
@@ -1841,7 +1852,7 @@ static int isc_set_default_fmt(struct isc_device *isc) };int ret; - ret =isc_try_fmt (isc, &f,NULL ); + ret =isc_try_fmt (isc, &f);if (ret)return ret;
@@ -2015,6 +2026,24 @@ int isc_pipeline_init(struct isc_device *isc) }EXPORT_SYMBOL_GPL (isc_pipeline_init); +int isc_link_validate (struct media_link *link) +{ +struct video_device *vdev = +media_entity_to_video_device (link->sink->entity); +struct isc_device *isc =video_get_drvdata (vdev); +int ret; + + ret =v4l2_subdev_link_validate (link); +if (ret) +return ret; + +return isc_validate (isc); +} + +static const struct media_entity_operations isc_entity_operations = { + .link_validate = isc_link_validate, +}; +int isc_mc_init (struct isc_device *isc,u32 ver) {const struct of_device_id *match;
@@ -2022,6 +2051,8 @@ int isc_mc_init(struct isc_device *isc, u32 ver) isc->video_dev.entity.function =MEDIA_ENT_F_IO_V4L ; isc->video_dev.entity.flags =MEDIA_ENT_FL_DEFAULT ; + isc->video_dev.entity.ops = &isc_entity_operations; + isc->pads[ISC_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; ret = media_entity_pads_init(&isc->video_dev.entity, ISC_PADS_NUM,
drivers/media/platform/atmel/atmel-isc-scaler.c +5 -0
@@ -155,6 +155,10 @@ static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = { .init_cfg = isc_scaler_init_cfg, }; +static const struct media_entity_operations isc_scaler_entity_ops = { + .link_validate = v4l2_subdev_link_validate, +}; +static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = { .pad = &isc_scaler_pad_ops, };
@@ -172,6 +176,7 @@ int isc_scaler_init(struct isc_device *isc) isc->scaler_sd.flags |=V4L2_SUBDEV_FL_HAS_DEVNODE ; isc->scaler_sd.entity.function =MEDIA_ENT_F_PROC_VIDEO_SCALER ; + isc->scaler_sd.entity.ops = &isc_scaler_entity_ops; isc->scaler_pads[ISC_SCALER_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; isc->scaler_pads[ISC_SCALER_PAD_SOURCE ].flags =MEDIA_PAD_FL_SOURCE ;
drivers/media/platform/atmel/atmel-isc.h +3 -0
@@ -270,6 +270,7 @@ enum isc_scaler_pads {* @formats_list_size: size of formats_list array * @pads: media controller pads for isc video entity * @mdev: media device that is registered by the isc +* @mpipe: media device pipeline used by the isc * @remote_pad: remote pad on the connected subdevice * @scaler_sd: subdevice for the scaler that isc registers * @scaler_pads: media controller pads for the scaler subdevice
@@ -295,6 +296,7 @@ struct isc_device {struct completion comp;struct v4l2_format fmt; +struct v4l2_format try_fmt;struct isc_format **user_formats;unsigned int num_user_formats;
@@ -366,6 +368,7 @@ struct isc_device {struct {struct media_pad pads[ISC_PADS_NUM ];struct media_device mdev; +struct media_pipeline mpipe;u32 remote_pad; };
Signature · 1 line
2.25.1
Re: [PATCH v4 07/11] media: atmel: atmel-isc: change format propagation to subdev into only verification
Hi Eugen
Quoted 4 lines · Eugen Hristev
As a top MC video driver, the atmel-isc should not propagate the format to the subdevice.
It should rather check atstart_streaming()time if the subdev is properly configured with a compatible format.
Quoted 8 lines
Removed the whole format finding logic, and reworked the format verification at
start_streamingtime, such that the ISC will return an error if the subdevice is not properly configured. To achieve this,media_pipeline_startis called and alink_validatecallback is created to check the formats. With this being done, the module parameter 'sensor_preferred' makes no sense anymore. The ISC should not decide which format the sensor is using. The ISC should only cope with the situation and inform userspace if the streaming is possible in the current configuration.
This sounds right!
Quoted 71 lines
The redesign of the format propagation has also risen the question of the enumfmt callback. If enumfmt is called with an
mbus_code, the enumfmt handler should only return the formats that are supported for thismbus_code. To make it more easy to understand the formats, changed the report order to report first the native formats, and after that the formats that the ISC can convert to.Signed-off-by Eugen Hristev <[email protected]>---
Changes in v4:
- moved validation code intolink_validateand usedmedia_pipeline_start
- merged this patch with theenum_fmtpatch which was previously in v3 of the seriesChanges in v3:
- clamp to maximum resolution once the frame size from the subdev is founddrivers/media/platform/atmel/atmel-isc-base.c | 335 ++++++++++-------- .../media/platform/atmel/atmel-isc-scaler.c | 5 + drivers/media/platform/atmel/atmel-isc.h | 3 + 3 files changed, 191 insertions(+), 152 deletions(-)drivers/media/platform/atmel/atmel-isc-base.c +10
-5@@ -36,11 +36,6 @@ static unsigned int debug;module_param (debug, int,0644 );MODULE_PARM_DESC (debug,"debug level (0-2)" ); -static unsigned int sensor_preferred =1 ; -module_param (sensor_preferred, uint,0644 ); -MODULE_PARM_DESC (sensor_preferred, -"Sensor is preferred to output the specified format (1-on 0-off), default 1" ); -#define ISC_IS_FORMAT_RAW (mbus_code) \ (((mbus_code) & 0xf000) == 0x3000)@@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)unsigned long flags;int ret; + ret =media_pipeline_start (&isc->video_dev.entity, &isc->mpipe); +if (ret) + goto err_pipeline_start; +/* Enable stream on the sub device */ ret =v4l2_subdev_call (isc->current_subdev->sd, video, s_stream,1 );if (ret && ret != -ENOIOCTLCMD ) {@@ -384,6 +383,9 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)v4l2_subdev_call (isc->current_subdev->sd, video, s_stream,0 ); err_start_stream: +media_pipeline_stop (&isc->video_dev.entity); + +err_pipeline_start:spin_lock_irqsave (&isc->dma_queue_lock, flags);list_for_each_entry (buf, &isc->dma_queue, list)vb2_buffer_done (&buf->vb.vb2_buf,VB2_BUF_STATE_QUEUED );@@ -420,6 +422,9 @@ static void isc_stop_streaming(struct vb2_queue *vq)if (ret && ret != -ENOIOCTLCMD )v4l2_err (&isc->v4l2_dev,"stream off failed in subdev\n" ); +/* Stop media pipeline */ +media_pipeline_stop (&isc->video_dev.entity); +/* Release all active buffers */ spin_lock_irqsave (&isc->dma_queue_lock, flags);if (unlikely (isc->cur_frm)) {@@ -496,21 +501,56 @@ static int isc_enum_fmt_vid_cap(struct file *file, void *priv,u32 index = f->index;
drivers/media/platform/atmel/atmel-isc-base.c static int isc_enum_fmt_vid_cap(struct file *file, void *priv, · line 508
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-base.c +5
-3u32 i, supported_index; -if (index < isc->controller_formats_size) { - f->pixelformat = isc->controller_formats[index].fourcc; -return 0 ; + supported_index =0 ; + +for (i =0 ; i < isc->formats_list_size; i++) { +if (!isc->formats_list[i].sd_support) +continue ;
Sorry to ask again about sd_support, but if the ISC formats enumeration is not constrained to a specific mbus_code, shouldn't all the formats be enumerated regardless if they're supported from the sensor or not ?
drivers/media/platform/atmel/atmel-isc-base.c static int isc_enum_fmt_vid_cap(struct file *file, void *priv, · line 514
Quoted 6 lines
drivers/media/platform/atmel/atmel-isc-base.c +6
-0+/* +* If specific mbus_code is requested, provide only +* supported formats with this mbus code +*/ +if (f->mbus_code && f->mbus_code != + isc->formats_list[i].mbus_code)
nit: for clarity
if (f->mbus_code &&
f->mbus_code != isc->formats_list[i].mbus_code)Quoted 34 lines
drivers/media/platform/atmel/atmel-isc-base.c +27
-2+continue ; +if (supported_index == index) { + f->pixelformat = isc->formats_list[i].fourcc; +return 0 ; + } + supported_index++; } - index -= isc->controller_formats_size; +/* +* If the sensor does not support this mbus_code whatsoever, +* there is no reason to advertise any of our output formats +*/ +if (supported_index ==0 ) +return -EINVAL ; +/* +* If the sensor uses a format that is not raw, then we cannot +* convert it to any of the formats that we usually can with a +* RAW sensor. Thus, do not advertise them. +*/ +if (isc->config.sd_format && + !ISC_IS_FORMAT_RAW (isc->config.sd_format->mbus_code)) +return -EINVAL ; + +/* +* Iterate again through the formats that we can convert to. +* However, to avoid duplicates, skip the formats that +* the sensor already supports directly +*/ + index -= supported_index; supported_index =0 ; -for (i =0 ; i < isc->formats_list_size; i++) {
drivers/media/platform/atmel/atmel-isc-base.c static int isc_enum_fmt_vid_cap(struct file *file, void *priv, · line 553
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-base.c +5
-3-if (!ISC_IS_FORMAT_RAW (isc->formats_list[i].mbus_code) || - !isc->formats_list[i].sd_support) +for (i =0 ; i < isc->controller_formats_size; i++) { +/* if this format is already supported by sensor, skip it */ +if (find_format_by_fourcc (isc, isc->controller_formats[i].fourcc))continue ;if (supported_index == index) { - f->pixelformat = isc->formats_list[i].fourcc; + f->pixelformat = + isc->controller_formats[i].fourcc;
I trust this last part to work correctly, as it's not 100% clear to me :)
Quoted 7 lines
drivers/media/platform/atmel/atmel-isc-base.c +0
-0return 0 ; } supported_index++;@@ -581,20 +621,30 @@ static int isc_try_validate_formats(struct isc_device *isc)break ;default :/* any other different formats are not supported */
drivers/media/platform/atmel/atmel-isc-base.c static int isc_try_validate_formats(struct isc_device *isc) · line 631
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-base.c +2
-2+v4l2_err (&isc->v4l2_dev,"Requested unsupported format.\n" ); ret = -EINVAL ; }v4l2_dbg (1 , debug, &isc->v4l2_dev,"Format validation, requested rgb=%u, yuv=%u, grey=%u, bayer=%u\n" , rgb, yuv, grey, bayer); -/* we cannot output RAW if we do not receive RAW */ -if ((bayer) && !ISC_IS_FORMAT_RAW (isc->try_config.sd_format->mbus_code)) + if ((bayer) &&
just 'bayer' ?
Quoted 166 lines
drivers/media/platform/atmel/atmel-isc-base.c +26
-110+ !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code)) { +v4l2_err (&isc->v4l2_dev,"Cannot output RAW if we do not receive RAW.\n" );return -EINVAL ; + } -/* we cannot output GREY if we do not receive RAW/GREY */ if (grey && !ISC_IS_FORMAT_RAW (isc->try_config.sd_format->mbus_code) && - !ISC_IS_FORMAT_GREY (isc->try_config.sd_format->mbus_code)) + !ISC_IS_FORMAT_GREY (isc->try_config.sd_format->mbus_code)) { +v4l2_err (&isc->v4l2_dev,"Cannot output GREY if we do not receive RAW/GREY.\n" ); +return -EINVAL ; + } + +if ((rgb || bayer || yuv) && +ISC_IS_FORMAT_GREY (isc->try_config.sd_format->mbus_code)) { +v4l2_err (&isc->v4l2_dev,"Cannot convert GREY to another format.\n" );return -EINVAL ; + }return ret; }@@ -822,7 +872,7 @@ static void isc_try_fse(struct isc_device *isc,* If we do not know yet which format the subdev is using, we cannot * do anything. */ -if (!isc->try_config.sd_format) +if (!isc->config.sd_format)return ; fse.code = isc->try_config.sd_format->mbus_code;@@ -843,180 +893,141 @@ static void isc_try_fse(struct isc_device *isc, } } -static int isc_try_fmt (struct isc_device *isc,struct v4l2_format *f, -u32 *code) +static int isc_try_fmt (struct isc_device *isc,struct v4l2_format *f) { -int i; -struct isc_format *sd_fmt =NULL , *direct_fmt =NULL ;struct v4l2_pix_format *pixfmt = &f->fmt.pix; -struct v4l2_subdev_pad_config pad_cfg = {}; -struct v4l2_subdev_state pad_state = { - .pads = &pad_cfg - }; -struct v4l2_subdev_format format = { - .which =V4L2_SUBDEV_FORMAT_TRY , - }; -u32 mbus_code; -int ret; -bool rlp_dma_direct_dump = false; +unsigned int i;if (f->type !=V4L2_BUF_TYPE_VIDEO_CAPTURE )return -EINVAL ; -/* Step 1: find a RAW format that is supported */ -for (i =0 ; i < isc->num_user_formats; i++) { -if (ISC_IS_FORMAT_RAW (isc->user_formats[i]->mbus_code)) { - sd_fmt = isc->user_formats[i]; + isc->try_config.fourcc = isc->user_formats[0 ]->fourcc; + +/* find if the format requested is supported */ +for (i =0 ; i < isc->controller_formats_size; i++) +if (isc->controller_formats[i].fourcc == pixfmt->pixelformat) { + isc->try_config.fourcc = pixfmt->pixelformat;break ; } - } -/* Step 2: We can continue with this RAW format, or we can look -* for better: maybe sensor supports directly what we need. -*/ - direct_fmt =find_format_by_fourcc (isc, pixfmt->pixelformat); - -/* Step 3: We have both. We decide given the module parameter which -* one to use. -*/ -if (direct_fmt && sd_fmt && sensor_preferred) - sd_fmt = direct_fmt; - -/* Step 4: we do not have RAW but we have a direct format. Use it. */ -if (direct_fmt && !sd_fmt) - sd_fmt = direct_fmt; - -/* Step 5: if we are using a direct format, we need to package -* everything as 8 bit data and just dump it -*/ -if (sd_fmt == direct_fmt) - rlp_dma_direct_dump = true; - -/* Step 6: We have no format. This can happen if the userspace -* requests some weird/invalid format. -* In this case, default to whatever we have -*/ -if (!sd_fmt && !direct_fmt) { - sd_fmt = isc->user_formats[isc->num_user_formats -1 ]; -v4l2_dbg (1 , debug, &isc->v4l2_dev, -"Sensor not supporting %.4s, using %.4s\n" , - (char *)&pixfmt->pixelformat, (char *)&sd_fmt->fourcc); - } - -if (!sd_fmt) { - ret = -EINVAL ; - goto isc_try_fmt_err; - } - -/* Step 7: Print out what we decided for debugging */ -v4l2_dbg (1 , debug, &isc->v4l2_dev, -"Preferring to have sensor using format %.4s\n" , - (char *)&sd_fmt->fourcc); - -/* Step 8: at this moment we decided which format the subdev will use */ - isc->try_config.sd_format = sd_fmt; - -/* Limit to Atmel ISC hardware capabilities */ -if (pixfmt->width > isc->max_width) - pixfmt->width = isc->max_width; -if (pixfmt->height > isc->max_height) - pixfmt->height = isc->max_height; - -/* -* The mbus format is the one the subdev outputs. -* The pixels will be transferred in this format Sensor -> ISC -*/ - mbus_code = sd_fmt->mbus_code; - -/* -* Validate formats. If the required format is not OK, default to raw. -*/ - - isc->try_config.fourcc = pixfmt->pixelformat; - -if (isc_try_validate_formats (isc)) { - pixfmt->pixelformat = isc->try_config.fourcc = sd_fmt->fourcc; -/* Re-try to validate the new format */ - ret =isc_try_validate_formats (isc); -if (ret) - goto isc_try_fmt_err; - } - - ret =isc_try_configure_rlp_dma (isc, rlp_dma_direct_dump); -if (ret) - goto isc_try_fmt_err; - - ret =isc_try_configure_pipeline (isc); -if (ret) - goto isc_try_fmt_err; -/* Obtain frame sizes if possible to have crop requirements ready */ -isc_try_fse (isc, &pad_state); - -v4l2_fill_mbus_format (&format.format, pixfmt, mbus_code); - ret =v4l2_subdev_call (isc->current_subdev->sd, pad, set_fmt, - &pad_state, &format); -if (ret <0 ) - goto isc_try_fmt_subdev_err; +/* If we did not find the requested format, we will fallback here */ + pixfmt->pixelformat = isc->try_config.fourcc; + pixfmt->colorspace =V4L2_COLORSPACE_SRGB ; + pixfmt->field =V4L2_FIELD_NONE ; -v4l2_fill_pix_format (pixfmt, &format.format); +isc_try_configure_rlp_dma (isc, false);/* Limit to Atmel ISC hardware capabilities */ -if (pixfmt->width > isc->max_width)
drivers/media/platform/atmel/atmel-isc-base.c static void isc_try_fse(struct isc_device *isc, · line 927
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-base.c +2
-3- pixfmt->width = isc->max_width; -if (pixfmt->height > isc->max_height) - pixfmt->height = isc->max_height; +v4l_bound_align_image (&pixfmt->width,16 , isc->max_width,0 , + &pixfmt->height,16 , isc->max_height,0 ,0 ); pixfmt->field =V4L2_FIELD_NONE ; pixfmt->bytesperline = (pixfmt->width * isc->try_config.bpp_v4l2) >>3 ; pixfmt->sizeimage = ((pixfmt->width * isc->try_config.bpp) >>3 ) * pixfmt->height;
Seems pixfmt->field is set twice.
Could you move the assignment of all pixfmt to a single block ?
Quoted 150 lines
drivers/media/platform/atmel/atmel-isc-base.c +80
-24-if (code) - *code = mbus_code; + isc->try_fmt = *f;return 0 ; +} -isc_try_fmt_err: -v4l2_err (&isc->v4l2_dev,"Could not find any possible format for a working pipeline\n" ); -isc_try_fmt_subdev_err: -memset (&isc->try_config,0 ,sizeof (isc->try_config)); +static int isc_set_fmt (struct isc_device *isc,struct v4l2_format *f) +{ +isc_try_fmt (isc, f); -return ret; +/* make the try configuration active */ + isc->config = isc->try_config; + isc->fmt = isc->try_fmt; + +v4l2_dbg (1 , debug, &isc->v4l2_dev,"ISC set_fmt to %.4s @%dx%d\n" , + (char *)&f->fmt.pix.pixelformat, + f->fmt.pix.width, f->fmt.pix.height); + +return 0 ; } -static int isc_set_fmt (struct isc_device *isc,struct v4l2_format *f) +static int isc_validate (struct isc_device *isc) { +int ret; +int i; +struct isc_format *sd_fmt =NULL ; +struct v4l2_pix_format *pixfmt = &isc->fmt.fmt.pix;struct v4l2_subdev_format format = { .which =V4L2_SUBDEV_FORMAT_ACTIVE , + .pad = isc->remote_pad, + }; +struct v4l2_subdev_pad_config pad_cfg = {}; +struct v4l2_subdev_state pad_state = { + .pads = &pad_cfg, }; -u32 mbus_code =0 ; -int ret; - ret =isc_try_fmt (isc, f, &mbus_code); +/* Get current format from subdev */ + ret =v4l2_subdev_call (isc->current_subdev->sd, pad, get_fmt,NULL , + &format);if (ret)return ret; -v4l2_fill_mbus_format (&format.format, &f->fmt.pix, mbus_code); - ret =v4l2_subdev_call (isc->current_subdev->sd, pad, - set_fmt,NULL , &format); -if (ret <0 ) -return ret; +/* Identify the subdev's format configuration */ +for (i =0 ; i < isc->num_user_formats; i++) +if (isc->user_formats[i]->mbus_code == format.format.code) { + sd_fmt = isc->user_formats[i]; +break ; + } + +/* Check if the format is not supported */ +if (!sd_fmt) { +v4l2_err (&isc->v4l2_dev, +"Current subdevice is streaming a media bus code that is not supported 0x%x\n" , + format.format.code); +return -EPIPE ; + } + +/* At this moment we know which format the subdev will use */ + isc->try_config.sd_format = sd_fmt; + +/* If the sensor is not RAW, we can only do a direct dump */ +if (!ISC_IS_FORMAT_RAW (isc->try_config.sd_format->mbus_code)) +isc_try_configure_rlp_dma (isc, true);/* Limit to Atmel ISC hardware capabilities */ -if (f->fmt.pix.width > isc->max_width) - f->fmt.pix.width = isc->max_width; -if (f->fmt.pix.height > isc->max_height) - f->fmt.pix.height = isc->max_height; +v4l_bound_align_image (&format.format.width,16 , isc->max_width,0 , + &format.format.height,16 , isc->max_height,0 ,0 ); + +/* Check if the frame size is the same. Otherwise we may overflow */ +if (pixfmt->height != format.format.height || + pixfmt->width != format.format.width) { +v4l2_err (&isc->v4l2_dev, +"ISC not configured with the proper frame size: %dx%d\n" , + format.format.width, format.format.height); +return -EPIPE ; + } - isc->fmt = *f; +v4l2_dbg (1 , debug, &isc->v4l2_dev, +"Identified subdev using format %.4s with %dx%d %d bpp\n" , + (char *)&sd_fmt->fourcc, pixfmt->width, pixfmt->height, + isc->try_config.bpp); +/* Reset and restart AWB if the subdevice changed the format */ if (isc->try_config.sd_format && isc->config.sd_format && isc->try_config.sd_format != isc->config.sd_format) { isc->ctrls.hist_stat =HIST_INIT ;isc_reset_awb_ctrls (isc);isc_update_v4l2_ctrls (isc); } -/* make the try configuration active */ + +/* Validate formats */ + ret =isc_try_validate_formats (isc); +if (ret) +return ret; + +/* Obtain frame sizes if possible to have crop requirements ready */ +isc_try_fse (isc, &pad_state); + +/* Configure ISC pipeline for the config */ + ret =isc_try_configure_pipeline (isc); +if (ret) +return ret; + isc->config = isc->try_config;v4l2_dbg (1 , debug, &isc->v4l2_dev,"New ISC configuration in place\n" );@@ -1040,7 +1051,7 @@ static int isc_try_fmt_vid_cap(struct file *file, void *priv, {struct isc_device *isc =video_drvdata (file); -return isc_try_fmt (isc, f,NULL ); +return isc_try_fmt (isc, f); } static int isc_enum_input(struct file *file, void *priv,@@ -1841,7 +1852,7 @@ static int isc_set_default_fmt(struct isc_device *isc) };int ret; - ret =isc_try_fmt (isc, &f,NULL ); + ret =isc_try_fmt (isc, &f);if (ret)return ret;@@ -2015,6 +2026,24 @@ int isc_pipeline_init(struct isc_device *isc) }EXPORT_SYMBOL_GPL (isc_pipeline_init);
drivers/media/platform/atmel/atmel-isc-base.c int isc_pipeline_init(struct isc_device *isc) · line 2038
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-base.c +10
-0+int isc_link_validate (struct media_link *link) +{ +struct video_device *vdev = +media_entity_to_video_device (link->sink->entity); +struct isc_device *isc =video_get_drvdata (vdev); +int ret; + + ret =v4l2_subdev_link_validate (link); +if (ret) +return ret;
Doesn't v4l2_subdev_link_validate() call v4l2_subdev_link_validate_default() if you don't define a .link_validate() pad operation ? v4l2_subdev_link_validate_default() compares format and sizes of both ends to the links and want them to be identical, something that defeat the purpose of your custom isc_validate() ?
Thanks
j
drivers/media/platform/atmel/atmel-isc-base.c int isc_pipeline_init(struct isc_device *isc) · line 2046
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-base.c +8
-0+ +return isc_validate (isc); +} + +static const struct media_entity_operations isc_entity_operations = { + .link_validate = isc_link_validate, +}; +
};
Quoted 66 lines
drivers/media/platform/atmel/atmel-isc-base.c +2
-0int isc_mc_init (struct isc_device *isc,u32 ver) {const struct of_device_id *match;@@ -2022,6 +2051,8 @@ int isc_mc_init(struct isc_device *isc, u32 ver) isc->video_dev.entity.function =MEDIA_ENT_F_IO_V4L ; isc->video_dev.entity.flags =MEDIA_ENT_FL_DEFAULT ; + isc->video_dev.entity.ops = &isc_entity_operations; + isc->pads[ISC_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; ret = media_entity_pads_init(&isc->video_dev.entity, ISC_PADS_NUM,drivers/media/platform/atmel/atmel-isc-scaler.c +5
-0@@ -155,6 +155,10 @@ static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = { .init_cfg = isc_scaler_init_cfg, }; +static const struct media_entity_operations isc_scaler_entity_ops = { + .link_validate = v4l2_subdev_link_validate, +}; +static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = { .pad = &isc_scaler_pad_ops, };@@ -172,6 +176,7 @@ int isc_scaler_init(struct isc_device *isc) isc->scaler_sd.flags |=V4L2_SUBDEV_FL_HAS_DEVNODE ; isc->scaler_sd.entity.function =MEDIA_ENT_F_PROC_VIDEO_SCALER ; + isc->scaler_sd.entity.ops = &isc_scaler_entity_ops; isc->scaler_pads[ISC_SCALER_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; isc->scaler_pads[ISC_SCALER_PAD_SOURCE ].flags =MEDIA_PAD_FL_SOURCE ;drivers/media/platform/atmel/atmel-isc.h +3
-1@@ -270,6 +270,7 @@ enum isc_scaler_pads {* @formats_list_size: size of formats_list array * @pads: media controller pads for isc video entity * @mdev: media device that is registered by the isc +* @mpipe: media device pipeline used by the isc * @remote_pad: remote pad on the connected subdevice * @scaler_sd: subdevice for the scaler that isc registers * @scaler_pads: media controller pads for the scaler subdevice @@ -295,6 +296,7 @@ struct isc_device {struct completion comp;struct v4l2_format fmt; +struct v4l2_format try_fmt;struct isc_format **user_formats;unsigned int num_user_formats;@@ -366,6 +368,7 @@ struct isc_device {struct {struct media_pad pads[ISC_PADS_NUM ];struct media_device mdev; +struct media_pipeline mpipe;u32 remote_pad; }; --2.25.1
Re: [PATCH v4 07/11] media: atmel: atmel-isc: change format propagation to subdev into only verification
Quoted 91 lines · Jacopo Mondi
Hi Eugen
Quoted 4 lines · Eugen Hristev
As a top MC video driver, the atmel-isc should not propagate the format to the subdevice.
It should rather check atstart_streaming()time if the subdev is properly configured with a compatible format.Quoted 8 lines
Removed the whole format finding logic, and reworked the format verification at
start_streamingtime, such that the ISC will return an error if the subdevice is not properly configured. To achieve this,media_pipeline_startis called and alink_validatecallback is created to check the formats. With this being done, the module parameter 'sensor_preferred' makes no sense anymore. The ISC should not decide which format the sensor is using. The ISC should only cope with the situation and inform userspace if the streaming is possible in the current configuration.This sounds right!
Quoted 73 lines
The redesign of the format propagation has also risen the question of the enumfmt callback. If enumfmt is called with an
mbus_code, the enumfmt handler should only return the formats that are supported for thismbus_code. To make it more easy to understand the formats, changed the report order to report first the native formats, and after that the formats that the ISC can convert to.Signed-off-by Eugen Hristev <[email protected]>---
Changes in v4:
- moved validation code intolink_validateand usedmedia_pipeline_start
- merged this patch with theenum_fmtpatch which was previously in v3 of the seriesChanges in v3:
- clamp to maximum resolution once the frame size from the subdev is founddrivers/media/platform/atmel/atmel-isc-base.c | 335 ++++++++++-------- .../media/platform/atmel/atmel-isc-scaler.c | 5 + drivers/media/platform/atmel/atmel-isc.h | 3 + 3 files changed, 191 insertions(+), 152 deletions(-)drivers/media/platform/atmel/atmel-isc-base.c +10
-5@@ -36,11 +36,6 @@ static unsigned int debug;module_param (debug, int,0644 );MODULE_PARM_DESC (debug,"debug level (0-2)" ); -static unsigned int sensor_preferred =1 ; -module_param (sensor_preferred, uint,0644 ); -MODULE_PARM_DESC (sensor_preferred, -"Sensor is preferred to output the specified format (1-on 0-off), default 1" ); -#define ISC_IS_FORMAT_RAW (mbus_code) \ (((mbus_code) & 0xf000) == 0x3000)@@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)unsigned long flags;int ret; + ret =media_pipeline_start (&isc->video_dev.entity, &isc->mpipe); +if (ret) + goto err_pipeline_start; +/* Enable stream on the sub device */ ret =v4l2_subdev_call (isc->current_subdev->sd, video, s_stream,1 );if (ret && ret != -ENOIOCTLCMD ) {@@ -384,6 +383,9 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)v4l2_subdev_call (isc->current_subdev->sd, video, s_stream,0 ); err_start_stream: +media_pipeline_stop (&isc->video_dev.entity); + +err_pipeline_start:spin_lock_irqsave (&isc->dma_queue_lock, flags);list_for_each_entry (buf, &isc->dma_queue, list)vb2_buffer_done (&buf->vb.vb2_buf,VB2_BUF_STATE_QUEUED );@@ -420,6 +422,9 @@ static void isc_stop_streaming(struct vb2_queue *vq)if (ret && ret != -ENOIOCTLCMD )v4l2_err (&isc->v4l2_dev,"stream off failed in subdev\n" ); +/* Stop media pipeline */ +media_pipeline_stop (&isc->video_dev.entity); +/* Release all active buffers */ spin_lock_irqsave (&isc->dma_queue_lock, flags);if (unlikely (isc->cur_frm)) {@@ -496,21 +501,56 @@ static int isc_enum_fmt_vid_cap(struct file *file, void *priv,u32 index = f->index;u32 i, supported_index;
Quoted 13 lines
drivers/media/platform/atmel/atmel-isc-base.c static int isc_enum_fmt_vid_cap(struct file *file, void *priv, · line 508
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-base.c +5
-3-if (index < isc->controller_formats_size) { - f->pixelformat = isc->controller_formats[index].fourcc; -return 0 ; + supported_index =0 ; + +for (i =0 ; i < isc->formats_list_size; i++) { +if (!isc->formats_list[i].sd_support) +continue ;Sorry to ask again about
sd_support, but if the ISC formats enumeration is not constrained to a specificmbus_code, shouldn't all the formats be enumerated regardless if they're supported from the sensor or not ?
Yes. this slipped a little. I have to rework a bit the format reporting to user space.
Quoted 57 lines
drivers/media/platform/atmel/atmel-isc-base.c static int isc_enum_fmt_vid_cap(struct file *file, void *priv, · line 514
Quoted 7 lines
drivers/media/platform/atmel/atmel-isc-base.c +6
-0+/* +* If specific mbus_code is requested, provide only +* supported formats with this mbus code +*/ +if (f->mbus_code && f->mbus_code != + isc->formats_list[i].mbus_code)nit: for clarity
if (f->mbus_code && f->mbus_code != isc->formats_list[i].mbus_code)Quoted 34 lines
drivers/media/platform/atmel/atmel-isc-base.c +27
-2+continue ; +if (supported_index == index) { + f->pixelformat = isc->formats_list[i].fourcc; +return 0 ; + } + supported_index++; } - index -= isc->controller_formats_size; +/* +* If the sensor does not support this mbus_code whatsoever, +* there is no reason to advertise any of our output formats +*/ +if (supported_index ==0 ) +return -EINVAL ; +/* +* If the sensor uses a format that is not raw, then we cannot +* convert it to any of the formats that we usually can with a +* RAW sensor. Thus, do not advertise them. +*/ +if (isc->config.sd_format && + !ISC_IS_FORMAT_RAW (isc->config.sd_format->mbus_code)) +return -EINVAL ; + +/* +* Iterate again through the formats that we can convert to. +* However, to avoid duplicates, skip the formats that +* the sensor already supports directly +*/ + index -= supported_index; supported_index =0 ; -for (i =0 ; i < isc->formats_list_size; i++) {drivers/media/platform/atmel/atmel-isc-base.c static int isc_enum_fmt_vid_cap(struct file *file, void *priv, · line 553
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-base.c +5
-3-if (!ISC_IS_FORMAT_RAW (isc->formats_list[i].mbus_code) || - !isc->formats_list[i].sd_support) +for (i =0 ; i < isc->controller_formats_size; i++) { +/* if this format is already supported by sensor, skip it */ +if (find_format_by_fourcc (isc, isc->controller_formats[i].fourcc))continue ;if (supported_index == index) { - f->pixelformat = isc->formats_list[i].fourcc; + f->pixelformat = + isc->controller_formats[i].fourcc;
Quoted 2 lines
I trust this last part to work correctly, as it's not 100% clear to me :)
I think I will remove this and just report all the formats to userspace. Let libcamera handle this ;)
Quoted 14 lines
Quoted 14 lines
drivers/media/platform/atmel/atmel-isc-base.c +1
-0return 0 ; } supported_index++;@@ -581,20 +621,30 @@ static int isc_try_validate_formats(struct isc_device *isc)break ;default :/* any other different formats are not supported */ +v4l2_err (&isc->v4l2_dev,"Requested unsupported format.\n" ); ret = -EINVAL ; }v4l2_dbg (1 , debug, &isc->v4l2_dev,"Format validation, requested rgb=%u, yuv=%u, grey=%u, bayer=%u\n" , rgb, yuv, grey, bayer);
Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-base.c static int isc_try_validate_formats(struct isc_device *isc) · line 631
Quoted 3 lines
drivers/media/platform/atmel/atmel-isc-base.c +1
-2-/* we cannot output RAW if we do not receive RAW */ -if ((bayer) && !ISC_IS_FORMAT_RAW (isc->try_config.sd_format->mbus_code)) + if ((bayer) &&just 'bayer' ?
not sure what you mean ?
Have to check if we can output raw bayer , and we can only do that if we receive raw bayer from the sensor.
Quoted 173 lines
Quoted 173 lines
drivers/media/platform/atmel/atmel-isc-base.c +28
-113+ !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code)) { +v4l2_err (&isc->v4l2_dev,"Cannot output RAW if we do not receive RAW.\n" );return -EINVAL ; + } -/* we cannot output GREY if we do not receive RAW/GREY */ if (grey && !ISC_IS_FORMAT_RAW (isc->try_config.sd_format->mbus_code) && - !ISC_IS_FORMAT_GREY (isc->try_config.sd_format->mbus_code)) + !ISC_IS_FORMAT_GREY (isc->try_config.sd_format->mbus_code)) { +v4l2_err (&isc->v4l2_dev,"Cannot output GREY if we do not receive RAW/GREY.\n" ); +return -EINVAL ; + } + +if ((rgb || bayer || yuv) && +ISC_IS_FORMAT_GREY (isc->try_config.sd_format->mbus_code)) { +v4l2_err (&isc->v4l2_dev,"Cannot convert GREY to another format.\n" );return -EINVAL ; + }return ret; }@@ -822,7 +872,7 @@ static void isc_try_fse(struct isc_device *isc,* If we do not know yet which format the subdev is using, we cannot * do anything. */ -if (!isc->try_config.sd_format) +if (!isc->config.sd_format)return ; fse.code = isc->try_config.sd_format->mbus_code;@@ -843,180 +893,141 @@ static void isc_try_fse(struct isc_device *isc, } } -static int isc_try_fmt (struct isc_device *isc,struct v4l2_format *f, -u32 *code) +static int isc_try_fmt (struct isc_device *isc,struct v4l2_format *f) { -int i; -struct isc_format *sd_fmt =NULL , *direct_fmt =NULL ;struct v4l2_pix_format *pixfmt = &f->fmt.pix; -struct v4l2_subdev_pad_config pad_cfg = {}; -struct v4l2_subdev_state pad_state = { - .pads = &pad_cfg - }; -struct v4l2_subdev_format format = { - .which =V4L2_SUBDEV_FORMAT_TRY , - }; -u32 mbus_code; -int ret; -bool rlp_dma_direct_dump = false; +unsigned int i;if (f->type !=V4L2_BUF_TYPE_VIDEO_CAPTURE )return -EINVAL ; -/* Step 1: find a RAW format that is supported */ -for (i =0 ; i < isc->num_user_formats; i++) { -if (ISC_IS_FORMAT_RAW (isc->user_formats[i]->mbus_code)) { - sd_fmt = isc->user_formats[i]; + isc->try_config.fourcc = isc->user_formats[0 ]->fourcc; + +/* find if the format requested is supported */ +for (i =0 ; i < isc->controller_formats_size; i++) +if (isc->controller_formats[i].fourcc == pixfmt->pixelformat) { + isc->try_config.fourcc = pixfmt->pixelformat;break ; } - } -/* Step 2: We can continue with this RAW format, or we can look -* for better: maybe sensor supports directly what we need. -*/ - direct_fmt =find_format_by_fourcc (isc, pixfmt->pixelformat); - -/* Step 3: We have both. We decide given the module parameter which -* one to use. -*/ -if (direct_fmt && sd_fmt && sensor_preferred) - sd_fmt = direct_fmt; - -/* Step 4: we do not have RAW but we have a direct format. Use it. */ -if (direct_fmt && !sd_fmt) - sd_fmt = direct_fmt; - -/* Step 5: if we are using a direct format, we need to package -* everything as 8 bit data and just dump it -*/ -if (sd_fmt == direct_fmt) - rlp_dma_direct_dump = true; - -/* Step 6: We have no format. This can happen if the userspace -* requests some weird/invalid format. -* In this case, default to whatever we have -*/ -if (!sd_fmt && !direct_fmt) { - sd_fmt = isc->user_formats[isc->num_user_formats -1 ]; -v4l2_dbg (1 , debug, &isc->v4l2_dev, -"Sensor not supporting %.4s, using %.4s\n" , - (char *)&pixfmt->pixelformat, (char *)&sd_fmt->fourcc); - } - -if (!sd_fmt) { - ret = -EINVAL ; - goto isc_try_fmt_err; - } - -/* Step 7: Print out what we decided for debugging */ -v4l2_dbg (1 , debug, &isc->v4l2_dev, -"Preferring to have sensor using format %.4s\n" , - (char *)&sd_fmt->fourcc); - -/* Step 8: at this moment we decided which format the subdev will use */ - isc->try_config.sd_format = sd_fmt; - -/* Limit to Atmel ISC hardware capabilities */ -if (pixfmt->width > isc->max_width) - pixfmt->width = isc->max_width; -if (pixfmt->height > isc->max_height) - pixfmt->height = isc->max_height; - -/* -* The mbus format is the one the subdev outputs. -* The pixels will be transferred in this format Sensor -> ISC -*/ - mbus_code = sd_fmt->mbus_code; - -/* -* Validate formats. If the required format is not OK, default to raw. -*/ - - isc->try_config.fourcc = pixfmt->pixelformat; - -if (isc_try_validate_formats (isc)) { - pixfmt->pixelformat = isc->try_config.fourcc = sd_fmt->fourcc; -/* Re-try to validate the new format */ - ret =isc_try_validate_formats (isc); -if (ret) - goto isc_try_fmt_err; - } - - ret =isc_try_configure_rlp_dma (isc, rlp_dma_direct_dump); -if (ret) - goto isc_try_fmt_err; - - ret =isc_try_configure_pipeline (isc); -if (ret) - goto isc_try_fmt_err; -/* Obtain frame sizes if possible to have crop requirements ready */ -isc_try_fse (isc, &pad_state); - -v4l2_fill_mbus_format (&format.format, pixfmt, mbus_code); - ret =v4l2_subdev_call (isc->current_subdev->sd, pad, set_fmt, - &pad_state, &format); -if (ret <0 ) - goto isc_try_fmt_subdev_err; +/* If we did not find the requested format, we will fallback here */ + pixfmt->pixelformat = isc->try_config.fourcc; + pixfmt->colorspace =V4L2_COLORSPACE_SRGB ; + pixfmt->field =V4L2_FIELD_NONE ; -v4l2_fill_pix_format (pixfmt, &format.format); +isc_try_configure_rlp_dma (isc, false);/* Limit to Atmel ISC hardware capabilities */ -if (pixfmt->width > isc->max_width) - pixfmt->width = isc->max_width; -if (pixfmt->height > isc->max_height) - pixfmt->height = isc->max_height; +v4l_bound_align_image (&pixfmt->width,16 , isc->max_width,0 , + &pixfmt->height,16 , isc->max_height,0 ,0 );
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-base.c static void isc_try_fse(struct isc_device *isc, · line 927
Quoted 4 lines
drivers/media/platform/atmel/atmel-isc-base.c +0
-0pixfmt->field =V4L2_FIELD_NONE ; pixfmt->bytesperline = (pixfmt->width * isc->try_config.bpp_v4l2) >>3 ; pixfmt->sizeimage = ((pixfmt->width * isc->try_config.bpp) >>3 ) * pixfmt->height;Seems
pixfmt->fieldis set twice.Could you move the assignment of all pixfmt to a single block ?
okay
Quoted 151 lines
Quoted 151 lines
drivers/media/platform/atmel/atmel-isc-base.c +80
-24-if (code) - *code = mbus_code; + isc->try_fmt = *f;return 0 ; +} -isc_try_fmt_err: -v4l2_err (&isc->v4l2_dev,"Could not find any possible format for a working pipeline\n" ); -isc_try_fmt_subdev_err: -memset (&isc->try_config,0 ,sizeof (isc->try_config)); +static int isc_set_fmt (struct isc_device *isc,struct v4l2_format *f) +{ +isc_try_fmt (isc, f); -return ret; +/* make the try configuration active */ + isc->config = isc->try_config; + isc->fmt = isc->try_fmt; + +v4l2_dbg (1 , debug, &isc->v4l2_dev,"ISC set_fmt to %.4s @%dx%d\n" , + (char *)&f->fmt.pix.pixelformat, + f->fmt.pix.width, f->fmt.pix.height); + +return 0 ; } -static int isc_set_fmt (struct isc_device *isc,struct v4l2_format *f) +static int isc_validate (struct isc_device *isc) { +int ret; +int i; +struct isc_format *sd_fmt =NULL ; +struct v4l2_pix_format *pixfmt = &isc->fmt.fmt.pix;struct v4l2_subdev_format format = { .which =V4L2_SUBDEV_FORMAT_ACTIVE , + .pad = isc->remote_pad, + }; +struct v4l2_subdev_pad_config pad_cfg = {}; +struct v4l2_subdev_state pad_state = { + .pads = &pad_cfg, }; -u32 mbus_code =0 ; -int ret; - ret =isc_try_fmt (isc, f, &mbus_code); +/* Get current format from subdev */ + ret =v4l2_subdev_call (isc->current_subdev->sd, pad, get_fmt,NULL , + &format);if (ret)return ret; -v4l2_fill_mbus_format (&format.format, &f->fmt.pix, mbus_code); - ret =v4l2_subdev_call (isc->current_subdev->sd, pad, - set_fmt,NULL , &format); -if (ret <0 ) -return ret; +/* Identify the subdev's format configuration */ +for (i =0 ; i < isc->num_user_formats; i++) +if (isc->user_formats[i]->mbus_code == format.format.code) { + sd_fmt = isc->user_formats[i]; +break ; + } + +/* Check if the format is not supported */ +if (!sd_fmt) { +v4l2_err (&isc->v4l2_dev, +"Current subdevice is streaming a media bus code that is not supported 0x%x\n" , + format.format.code); +return -EPIPE ; + } + +/* At this moment we know which format the subdev will use */ + isc->try_config.sd_format = sd_fmt; + +/* If the sensor is not RAW, we can only do a direct dump */ +if (!ISC_IS_FORMAT_RAW (isc->try_config.sd_format->mbus_code)) +isc_try_configure_rlp_dma (isc, true);/* Limit to Atmel ISC hardware capabilities */ -if (f->fmt.pix.width > isc->max_width) - f->fmt.pix.width = isc->max_width; -if (f->fmt.pix.height > isc->max_height) - f->fmt.pix.height = isc->max_height; +v4l_bound_align_image (&format.format.width,16 , isc->max_width,0 , + &format.format.height,16 , isc->max_height,0 ,0 ); + +/* Check if the frame size is the same. Otherwise we may overflow */ +if (pixfmt->height != format.format.height || + pixfmt->width != format.format.width) { +v4l2_err (&isc->v4l2_dev, +"ISC not configured with the proper frame size: %dx%d\n" , + format.format.width, format.format.height); +return -EPIPE ; + } - isc->fmt = *f; +v4l2_dbg (1 , debug, &isc->v4l2_dev, +"Identified subdev using format %.4s with %dx%d %d bpp\n" , + (char *)&sd_fmt->fourcc, pixfmt->width, pixfmt->height, + isc->try_config.bpp); +/* Reset and restart AWB if the subdevice changed the format */ if (isc->try_config.sd_format && isc->config.sd_format && isc->try_config.sd_format != isc->config.sd_format) { isc->ctrls.hist_stat =HIST_INIT ;isc_reset_awb_ctrls (isc);isc_update_v4l2_ctrls (isc); } -/* make the try configuration active */ + +/* Validate formats */ + ret =isc_try_validate_formats (isc); +if (ret) +return ret; + +/* Obtain frame sizes if possible to have crop requirements ready */ +isc_try_fse (isc, &pad_state); + +/* Configure ISC pipeline for the config */ + ret =isc_try_configure_pipeline (isc); +if (ret) +return ret; + isc->config = isc->try_config;v4l2_dbg (1 , debug, &isc->v4l2_dev,"New ISC configuration in place\n" );@@ -1040,7 +1051,7 @@ static int isc_try_fmt_vid_cap(struct file *file, void *priv, {struct isc_device *isc =video_drvdata (file); -return isc_try_fmt (isc, f,NULL ); +return isc_try_fmt (isc, f); } static int isc_enum_input(struct file *file, void *priv,@@ -1841,7 +1852,7 @@ static int isc_set_default_fmt(struct isc_device *isc) };int ret; - ret =isc_try_fmt (isc, &f,NULL ); + ret =isc_try_fmt (isc, &f);if (ret)return ret;@@ -2015,6 +2026,24 @@ int isc_pipeline_init(struct isc_device *isc) }EXPORT_SYMBOL_GPL (isc_pipeline_init);
Quoted 16 lines
drivers/media/platform/atmel/atmel-isc-base.c int isc_pipeline_init(struct isc_device *isc) · line 2038
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-base.c +10
-0+int isc_link_validate (struct media_link *link) +{ +struct video_device *vdev = +media_entity_to_video_device (link->sink->entity); +struct isc_device *isc =video_get_drvdata (vdev); +int ret; + + ret =v4l2_subdev_link_validate (link); +if (ret) +return ret;Doesn't
v4l2_subdev_link_validate()callv4l2_subdev_link_validate_default()if you don't define a .link_validate() pad operation ?v4l2_subdev_link_validate_default()compares format and sizes of both ends to the links and want them to be identical, something that defeat the purpose of your customisc_validate()?
I think so. However if I replace the validate call with my custom one, I thought that the old default should be called as well.
Meaning, that maybe my custom validation does certain isc-related checks, while the v4l2_subdev_link_validate does exactly what you said. So, by defining a custom link_validate, do I lose the checks that are done by the v4l2_subdev_link_validate ?
That's what I wanted to avoid. Chip in my checks, but keep the v4l2 standard validation in place.
Does that make sense ?
Quoted 78 lines
Thanks
jdrivers/media/platform/atmel/atmel-isc-base.c int isc_pipeline_init(struct isc_device *isc) · line 2046
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-base.c +8
-0+ +return isc_validate (isc); +} + +static const struct media_entity_operations isc_entity_operations = { + .link_validate = isc_link_validate, +}; +};
Quoted 66 lines
drivers/media/platform/atmel/atmel-isc-base.c +2
-0int isc_mc_init (struct isc_device *isc,u32 ver) {const struct of_device_id *match;@@ -2022,6 +2051,8 @@ int isc_mc_init(struct isc_device *isc, u32 ver) isc->video_dev.entity.function =MEDIA_ENT_F_IO_V4L ; isc->video_dev.entity.flags =MEDIA_ENT_FL_DEFAULT ; + isc->video_dev.entity.ops = &isc_entity_operations; + isc->pads[ISC_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; ret = media_entity_pads_init(&isc->video_dev.entity, ISC_PADS_NUM,drivers/media/platform/atmel/atmel-isc-scaler.c +5
-0@@ -155,6 +155,10 @@ static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = { .init_cfg = isc_scaler_init_cfg, }; +static const struct media_entity_operations isc_scaler_entity_ops = { + .link_validate = v4l2_subdev_link_validate, +}; +static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = { .pad = &isc_scaler_pad_ops, };@@ -172,6 +176,7 @@ int isc_scaler_init(struct isc_device *isc) isc->scaler_sd.flags |=V4L2_SUBDEV_FL_HAS_DEVNODE ; isc->scaler_sd.entity.function =MEDIA_ENT_F_PROC_VIDEO_SCALER ; + isc->scaler_sd.entity.ops = &isc_scaler_entity_ops; isc->scaler_pads[ISC_SCALER_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; isc->scaler_pads[ISC_SCALER_PAD_SOURCE ].flags =MEDIA_PAD_FL_SOURCE ;drivers/media/platform/atmel/atmel-isc.h +3
-1@@ -270,6 +270,7 @@ enum isc_scaler_pads {* @formats_list_size: size of formats_list array * @pads: media controller pads for isc video entity * @mdev: media device that is registered by the isc +* @mpipe: media device pipeline used by the isc * @remote_pad: remote pad on the connected subdevice * @scaler_sd: subdevice for the scaler that isc registers * @scaler_pads: media controller pads for the scaler subdevice @@ -295,6 +296,7 @@ struct isc_device {struct completion comp;struct v4l2_format fmt; +struct v4l2_format try_fmt;struct isc_format **user_formats;unsigned int num_user_formats;@@ -366,6 +368,7 @@ struct isc_device {struct {struct media_pad pads[ISC_PADS_NUM ];struct media_device mdev; +struct media_pipeline mpipe;u32 remote_pad; }; --2.25.1
Re: [PATCH v4 07/11] media: atmel: atmel-isc: change format propagation to subdev into only verification
Hi Eugen,
Quoted 166 lines · On Fri, Feb 11, 2022 at 10:38:56AM +0000, [email protected] wrote:
Quoted 91 lines · Jacopo Mondi
Hi Eugen
Quoted 4 lines · Eugen Hristev
As a top MC video driver, the atmel-isc should not propagate the format to the subdevice.
It should rather check atstart_streaming()time if the subdev is properly configured with a compatible format.Quoted 8 lines
Removed the whole format finding logic, and reworked the format verification at
start_streamingtime, such that the ISC will return an error if the subdevice is not properly configured. To achieve this,media_pipeline_startis called and alink_validatecallback is created to check the formats. With this being done, the module parameter 'sensor_preferred' makes no sense anymore. The ISC should not decide which format the sensor is using. The ISC should only cope with the situation and inform userspace if the streaming is possible in the current configuration.This sounds right!
Quoted 73 lines
The redesign of the format propagation has also risen the question of the enumfmt callback. If enumfmt is called with an
mbus_code, the enumfmt handler should only return the formats that are supported for thismbus_code. To make it more easy to understand the formats, changed the report order to report first the native formats, and after that the formats that the ISC can convert to.Signed-off-by Eugen Hristev <[email protected]>---
Changes in v4:
- moved validation code intolink_validateand usedmedia_pipeline_start
- merged this patch with theenum_fmtpatch which was previously in v3 of the seriesChanges in v3:
- clamp to maximum resolution once the frame size from the subdev is founddrivers/media/platform/atmel/atmel-isc-base.c | 335 ++++++++++-------- .../media/platform/atmel/atmel-isc-scaler.c | 5 + drivers/media/platform/atmel/atmel-isc.h | 3 + 3 files changed, 191 insertions(+), 152 deletions(-)drivers/media/platform/atmel/atmel-isc-base.c +10
-5@@ -36,11 +36,6 @@ static unsigned int debug;module_param (debug, int,0644 );MODULE_PARM_DESC (debug,"debug level (0-2)" ); -static unsigned int sensor_preferred =1 ; -module_param (sensor_preferred, uint,0644 ); -MODULE_PARM_DESC (sensor_preferred, -"Sensor is preferred to output the specified format (1-on 0-off), default 1" ); -#define ISC_IS_FORMAT_RAW (mbus_code) \ (((mbus_code) & 0xf000) == 0x3000)@@ -337,6 +332,10 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)unsigned long flags;int ret; + ret =media_pipeline_start (&isc->video_dev.entity, &isc->mpipe); +if (ret) + goto err_pipeline_start; +/* Enable stream on the sub device */ ret =v4l2_subdev_call (isc->current_subdev->sd, video, s_stream,1 );if (ret && ret != -ENOIOCTLCMD ) {@@ -384,6 +383,9 @@ static int isc_start_streaming(struct vb2_queue *vq, unsigned int count)v4l2_subdev_call (isc->current_subdev->sd, video, s_stream,0 ); err_start_stream: +media_pipeline_stop (&isc->video_dev.entity); + +err_pipeline_start:spin_lock_irqsave (&isc->dma_queue_lock, flags);list_for_each_entry (buf, &isc->dma_queue, list)vb2_buffer_done (&buf->vb.vb2_buf,VB2_BUF_STATE_QUEUED );@@ -420,6 +422,9 @@ static void isc_stop_streaming(struct vb2_queue *vq)if (ret && ret != -ENOIOCTLCMD )v4l2_err (&isc->v4l2_dev,"stream off failed in subdev\n" ); +/* Stop media pipeline */ +media_pipeline_stop (&isc->video_dev.entity); +/* Release all active buffers */ spin_lock_irqsave (&isc->dma_queue_lock, flags);if (unlikely (isc->cur_frm)) {@@ -496,21 +501,56 @@ static int isc_enum_fmt_vid_cap(struct file *file, void *priv,u32 index = f->index;u32 i, supported_index;Quoted 13 lines
drivers/media/platform/atmel/atmel-isc-base.c static int isc_enum_fmt_vid_cap(struct file *file, void *priv, · line 508
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-base.c +5
-3-if (index < isc->controller_formats_size) { - f->pixelformat = isc->controller_formats[index].fourcc; -return 0 ; + supported_index =0 ; + +for (i =0 ; i < isc->formats_list_size; i++) { +if (!isc->formats_list[i].sd_support) +continue ;Sorry to ask again about
sd_support, but if the ISC formats enumeration is not constrained to a specificmbus_code, shouldn't all the formats be enumerated regardless if they're supported from the sensor or not ?Yes. this slipped a little. I have to rework a bit the format reporting to user space.
Quoted 57 lines
drivers/media/platform/atmel/atmel-isc-base.c static int isc_enum_fmt_vid_cap(struct file *file, void *priv, · line 514
Quoted 7 lines
drivers/media/platform/atmel/atmel-isc-base.c +6
-0+/* +* If specific mbus_code is requested, provide only +* supported formats with this mbus code +*/ +if (f->mbus_code && f->mbus_code != + isc->formats_list[i].mbus_code)nit: for clarity
if (f->mbus_code && f->mbus_code != isc->formats_list[i].mbus_code)Quoted 44 lines
drivers/media/platform/atmel/atmel-isc-base.c +32
-5+continue ; +if (supported_index == index) { + f->pixelformat = isc->formats_list[i].fourcc; +return 0 ; + } + supported_index++; } - index -= isc->controller_formats_size; +/* +* If the sensor does not support this mbus_code whatsoever, +* there is no reason to advertise any of our output formats +*/ +if (supported_index ==0 ) +return -EINVAL ; +/* +* If the sensor uses a format that is not raw, then we cannot +* convert it to any of the formats that we usually can with a +* RAW sensor. Thus, do not advertise them. +*/ +if (isc->config.sd_format && + !ISC_IS_FORMAT_RAW (isc->config.sd_format->mbus_code)) +return -EINVAL ; + +/* +* Iterate again through the formats that we can convert to. +* However, to avoid duplicates, skip the formats that +* the sensor already supports directly +*/ + index -= supported_index; supported_index =0 ; -for (i =0 ; i < isc->formats_list_size; i++) { -if (!ISC_IS_FORMAT_RAW (isc->formats_list[i].mbus_code) || - !isc->formats_list[i].sd_support) +for (i =0 ; i < isc->controller_formats_size; i++) { +/* if this format is already supported by sensor, skip it */ +if (find_format_by_fourcc (isc, isc->controller_formats[i].fourcc))continue ;if (supported_index == index) { - f->pixelformat = isc->formats_list[i].fourcc; + f->pixelformat = + isc->controller_formats[i].fourcc;
Quoted 6 lines
Quoted 2 lines
I trust this last part to work correctly, as it's not 100% clear to me :)
I think I will remove this and just report all the formats to userspace. Let libcamera handle this ;)
I think so, the driver will be greatly simplified with this change.
Quoted 20 lines
Quoted 14 lines
Quoted 14 lines
drivers/media/platform/atmel/atmel-isc-base.c +1
-0return 0 ; } supported_index++;@@ -581,20 +621,30 @@ static int isc_try_validate_formats(struct isc_device *isc)break ;default :/* any other different formats are not supported */ +v4l2_err (&isc->v4l2_dev,"Requested unsupported format.\n" ); ret = -EINVAL ; }v4l2_dbg (1 , debug, &isc->v4l2_dev,"Format validation, requested rgb=%u, yuv=%u, grey=%u, bayer=%u\n" , rgb, yuv, grey, bayer);Quoted 5 lines
drivers/media/platform/atmel/atmel-isc-base.c static int isc_try_validate_formats(struct isc_device *isc) · line 631
Quoted 3 lines
drivers/media/platform/atmel/atmel-isc-base.c +1
-2-/* we cannot output RAW if we do not receive RAW */ -if ((bayer) && !ISC_IS_FORMAT_RAW (isc->try_config.sd_format->mbus_code)) + if ((bayer) &&just 'bayer' ?
Quoted 4 lines
not sure what you mean ?
Have to check if we can output raw bayer , and we can only do that if we receive raw bayer from the sensor.
I just meant that you could remove () from (bayer) ?
Sorry for not being clear :)
Quoted 351 lines
Quoted 173 lines
Quoted 173 lines
drivers/media/platform/atmel/atmel-isc-base.c +28
-113+ !ISC_IS_FORMAT_RAW(isc->try_config.sd_format->mbus_code)) { +v4l2_err (&isc->v4l2_dev,"Cannot output RAW if we do not receive RAW.\n" );return -EINVAL ; + } -/* we cannot output GREY if we do not receive RAW/GREY */ if (grey && !ISC_IS_FORMAT_RAW (isc->try_config.sd_format->mbus_code) && - !ISC_IS_FORMAT_GREY (isc->try_config.sd_format->mbus_code)) + !ISC_IS_FORMAT_GREY (isc->try_config.sd_format->mbus_code)) { +v4l2_err (&isc->v4l2_dev,"Cannot output GREY if we do not receive RAW/GREY.\n" ); +return -EINVAL ; + } + +if ((rgb || bayer || yuv) && +ISC_IS_FORMAT_GREY (isc->try_config.sd_format->mbus_code)) { +v4l2_err (&isc->v4l2_dev,"Cannot convert GREY to another format.\n" );return -EINVAL ; + }return ret; }@@ -822,7 +872,7 @@ static void isc_try_fse(struct isc_device *isc,* If we do not know yet which format the subdev is using, we cannot * do anything. */ -if (!isc->try_config.sd_format) +if (!isc->config.sd_format)return ; fse.code = isc->try_config.sd_format->mbus_code;@@ -843,180 +893,141 @@ static void isc_try_fse(struct isc_device *isc, } } -static int isc_try_fmt (struct isc_device *isc,struct v4l2_format *f, -u32 *code) +static int isc_try_fmt (struct isc_device *isc,struct v4l2_format *f) { -int i; -struct isc_format *sd_fmt =NULL , *direct_fmt =NULL ;struct v4l2_pix_format *pixfmt = &f->fmt.pix; -struct v4l2_subdev_pad_config pad_cfg = {}; -struct v4l2_subdev_state pad_state = { - .pads = &pad_cfg - }; -struct v4l2_subdev_format format = { - .which =V4L2_SUBDEV_FORMAT_TRY , - }; -u32 mbus_code; -int ret; -bool rlp_dma_direct_dump = false; +unsigned int i;if (f->type !=V4L2_BUF_TYPE_VIDEO_CAPTURE )return -EINVAL ; -/* Step 1: find a RAW format that is supported */ -for (i =0 ; i < isc->num_user_formats; i++) { -if (ISC_IS_FORMAT_RAW (isc->user_formats[i]->mbus_code)) { - sd_fmt = isc->user_formats[i]; + isc->try_config.fourcc = isc->user_formats[0 ]->fourcc; + +/* find if the format requested is supported */ +for (i =0 ; i < isc->controller_formats_size; i++) +if (isc->controller_formats[i].fourcc == pixfmt->pixelformat) { + isc->try_config.fourcc = pixfmt->pixelformat;break ; } - } -/* Step 2: We can continue with this RAW format, or we can look -* for better: maybe sensor supports directly what we need. -*/ - direct_fmt =find_format_by_fourcc (isc, pixfmt->pixelformat); - -/* Step 3: We have both. We decide given the module parameter which -* one to use. -*/ -if (direct_fmt && sd_fmt && sensor_preferred) - sd_fmt = direct_fmt; - -/* Step 4: we do not have RAW but we have a direct format. Use it. */ -if (direct_fmt && !sd_fmt) - sd_fmt = direct_fmt; - -/* Step 5: if we are using a direct format, we need to package -* everything as 8 bit data and just dump it -*/ -if (sd_fmt == direct_fmt) - rlp_dma_direct_dump = true; - -/* Step 6: We have no format. This can happen if the userspace -* requests some weird/invalid format. -* In this case, default to whatever we have -*/ -if (!sd_fmt && !direct_fmt) { - sd_fmt = isc->user_formats[isc->num_user_formats -1 ]; -v4l2_dbg (1 , debug, &isc->v4l2_dev, -"Sensor not supporting %.4s, using %.4s\n" , - (char *)&pixfmt->pixelformat, (char *)&sd_fmt->fourcc); - } - -if (!sd_fmt) { - ret = -EINVAL ; - goto isc_try_fmt_err; - } - -/* Step 7: Print out what we decided for debugging */ -v4l2_dbg (1 , debug, &isc->v4l2_dev, -"Preferring to have sensor using format %.4s\n" , - (char *)&sd_fmt->fourcc); - -/* Step 8: at this moment we decided which format the subdev will use */ - isc->try_config.sd_format = sd_fmt; - -/* Limit to Atmel ISC hardware capabilities */ -if (pixfmt->width > isc->max_width) - pixfmt->width = isc->max_width; -if (pixfmt->height > isc->max_height) - pixfmt->height = isc->max_height; - -/* -* The mbus format is the one the subdev outputs. -* The pixels will be transferred in this format Sensor -> ISC -*/ - mbus_code = sd_fmt->mbus_code; - -/* -* Validate formats. If the required format is not OK, default to raw. -*/ - - isc->try_config.fourcc = pixfmt->pixelformat; - -if (isc_try_validate_formats (isc)) { - pixfmt->pixelformat = isc->try_config.fourcc = sd_fmt->fourcc; -/* Re-try to validate the new format */ - ret =isc_try_validate_formats (isc); -if (ret) - goto isc_try_fmt_err; - } - - ret =isc_try_configure_rlp_dma (isc, rlp_dma_direct_dump); -if (ret) - goto isc_try_fmt_err; - - ret =isc_try_configure_pipeline (isc); -if (ret) - goto isc_try_fmt_err; -/* Obtain frame sizes if possible to have crop requirements ready */ -isc_try_fse (isc, &pad_state); - -v4l2_fill_mbus_format (&format.format, pixfmt, mbus_code); - ret =v4l2_subdev_call (isc->current_subdev->sd, pad, set_fmt, - &pad_state, &format); -if (ret <0 ) - goto isc_try_fmt_subdev_err; +/* If we did not find the requested format, we will fallback here */ + pixfmt->pixelformat = isc->try_config.fourcc; + pixfmt->colorspace =V4L2_COLORSPACE_SRGB ; + pixfmt->field =V4L2_FIELD_NONE ; -v4l2_fill_pix_format (pixfmt, &format.format); +isc_try_configure_rlp_dma (isc, false);/* Limit to Atmel ISC hardware capabilities */ -if (pixfmt->width > isc->max_width) - pixfmt->width = isc->max_width; -if (pixfmt->height > isc->max_height) - pixfmt->height = isc->max_height; +v4l_bound_align_image (&pixfmt->width,16 , isc->max_width,0 , + &pixfmt->height,16 , isc->max_height,0 ,0 );Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-base.c static void isc_try_fse(struct isc_device *isc, · line 927
Quoted 4 lines
drivers/media/platform/atmel/atmel-isc-base.c +0
-0pixfmt->field =V4L2_FIELD_NONE ; pixfmt->bytesperline = (pixfmt->width * isc->try_config.bpp_v4l2) >>3 ; pixfmt->sizeimage = ((pixfmt->width * isc->try_config.bpp) >>3 ) * pixfmt->height;Seems
pixfmt->fieldis set twice.Could you move the assignment of all pixfmt to a single block ?
okay
Quoted 151 lines
Quoted 151 lines
drivers/media/platform/atmel/atmel-isc-base.c +80
-24-if (code) - *code = mbus_code; + isc->try_fmt = *f;return 0 ; +} -isc_try_fmt_err: -v4l2_err (&isc->v4l2_dev,"Could not find any possible format for a working pipeline\n" ); -isc_try_fmt_subdev_err: -memset (&isc->try_config,0 ,sizeof (isc->try_config)); +static int isc_set_fmt (struct isc_device *isc,struct v4l2_format *f) +{ +isc_try_fmt (isc, f); -return ret; +/* make the try configuration active */ + isc->config = isc->try_config; + isc->fmt = isc->try_fmt; + +v4l2_dbg (1 , debug, &isc->v4l2_dev,"ISC set_fmt to %.4s @%dx%d\n" , + (char *)&f->fmt.pix.pixelformat, + f->fmt.pix.width, f->fmt.pix.height); + +return 0 ; } -static int isc_set_fmt (struct isc_device *isc,struct v4l2_format *f) +static int isc_validate (struct isc_device *isc) { +int ret; +int i; +struct isc_format *sd_fmt =NULL ; +struct v4l2_pix_format *pixfmt = &isc->fmt.fmt.pix;struct v4l2_subdev_format format = { .which =V4L2_SUBDEV_FORMAT_ACTIVE , + .pad = isc->remote_pad, + }; +struct v4l2_subdev_pad_config pad_cfg = {}; +struct v4l2_subdev_state pad_state = { + .pads = &pad_cfg, }; -u32 mbus_code =0 ; -int ret; - ret =isc_try_fmt (isc, f, &mbus_code); +/* Get current format from subdev */ + ret =v4l2_subdev_call (isc->current_subdev->sd, pad, get_fmt,NULL , + &format);if (ret)return ret; -v4l2_fill_mbus_format (&format.format, &f->fmt.pix, mbus_code); - ret =v4l2_subdev_call (isc->current_subdev->sd, pad, - set_fmt,NULL , &format); -if (ret <0 ) -return ret; +/* Identify the subdev's format configuration */ +for (i =0 ; i < isc->num_user_formats; i++) +if (isc->user_formats[i]->mbus_code == format.format.code) { + sd_fmt = isc->user_formats[i]; +break ; + } + +/* Check if the format is not supported */ +if (!sd_fmt) { +v4l2_err (&isc->v4l2_dev, +"Current subdevice is streaming a media bus code that is not supported 0x%x\n" , + format.format.code); +return -EPIPE ; + } + +/* At this moment we know which format the subdev will use */ + isc->try_config.sd_format = sd_fmt; + +/* If the sensor is not RAW, we can only do a direct dump */ +if (!ISC_IS_FORMAT_RAW (isc->try_config.sd_format->mbus_code)) +isc_try_configure_rlp_dma (isc, true);/* Limit to Atmel ISC hardware capabilities */ -if (f->fmt.pix.width > isc->max_width) - f->fmt.pix.width = isc->max_width; -if (f->fmt.pix.height > isc->max_height) - f->fmt.pix.height = isc->max_height; +v4l_bound_align_image (&format.format.width,16 , isc->max_width,0 , + &format.format.height,16 , isc->max_height,0 ,0 ); + +/* Check if the frame size is the same. Otherwise we may overflow */ +if (pixfmt->height != format.format.height || + pixfmt->width != format.format.width) { +v4l2_err (&isc->v4l2_dev, +"ISC not configured with the proper frame size: %dx%d\n" , + format.format.width, format.format.height); +return -EPIPE ; + } - isc->fmt = *f; +v4l2_dbg (1 , debug, &isc->v4l2_dev, +"Identified subdev using format %.4s with %dx%d %d bpp\n" , + (char *)&sd_fmt->fourcc, pixfmt->width, pixfmt->height, + isc->try_config.bpp); +/* Reset and restart AWB if the subdevice changed the format */ if (isc->try_config.sd_format && isc->config.sd_format && isc->try_config.sd_format != isc->config.sd_format) { isc->ctrls.hist_stat =HIST_INIT ;isc_reset_awb_ctrls (isc);isc_update_v4l2_ctrls (isc); } -/* make the try configuration active */ + +/* Validate formats */ + ret =isc_try_validate_formats (isc); +if (ret) +return ret; + +/* Obtain frame sizes if possible to have crop requirements ready */ +isc_try_fse (isc, &pad_state); + +/* Configure ISC pipeline for the config */ + ret =isc_try_configure_pipeline (isc); +if (ret) +return ret; + isc->config = isc->try_config;v4l2_dbg (1 , debug, &isc->v4l2_dev,"New ISC configuration in place\n" );@@ -1040,7 +1051,7 @@ static int isc_try_fmt_vid_cap(struct file *file, void *priv, {struct isc_device *isc =video_drvdata (file); -return isc_try_fmt (isc, f,NULL ); +return isc_try_fmt (isc, f); } static int isc_enum_input(struct file *file, void *priv,@@ -1841,7 +1852,7 @@ static int isc_set_default_fmt(struct isc_device *isc) };int ret; - ret =isc_try_fmt (isc, &f,NULL ); + ret =isc_try_fmt (isc, &f);if (ret)return ret;@@ -2015,6 +2026,24 @@ int isc_pipeline_init(struct isc_device *isc) }EXPORT_SYMBOL_GPL (isc_pipeline_init);Quoted 16 lines
drivers/media/platform/atmel/atmel-isc-base.c int isc_pipeline_init(struct isc_device *isc) · line 2038
Quoted 10 lines
drivers/media/platform/atmel/atmel-isc-base.c +10
-0+int isc_link_validate (struct media_link *link) +{ +struct video_device *vdev = +media_entity_to_video_device (link->sink->entity); +struct isc_device *isc =video_get_drvdata (vdev); +int ret; + + ret =v4l2_subdev_link_validate (link); +if (ret) +return ret;Doesn't
v4l2_subdev_link_validate()callv4l2_subdev_link_validate_default()if you don't define a .link_validate() pad operation ?v4l2_subdev_link_validate_default()compares format and sizes of both ends to the links and want them to be identical, something that defeat the purpose of your customisc_validate()?
Quoted 3 lines
I think so. However if I replace the validate call with my custom one, I thought that the old default should be called as well.
I was actually suggesting to remove the call to v4l2_subdev_link_validate() completely as I thought you were performing the same checks in your custom validate. If you're not please ignore this comment, and even if you are my comment is wrong, as I should have suggested you to remove the duplicated code and let the framework do that :)
Quoted 8 lines
Meaning, that maybe my custom validation does certain isc-related checks, while the
v4l2_subdev_link_validatedoes exactly what you said. So, by defining a customlink_validate, do I lose the checks that are done by thev4l2_subdev_link_validate?
That's what I wanted to avoid. Chip in my checks, but keep the v4l2 standard validation in place.Does that make sense ?
Absolutely, sorry for bothering and if the pipeline validation works please go ahead with what you have!
Thanks
j
Quoted 80 lines
Quoted 79 lines
Thanks
jdrivers/media/platform/atmel/atmel-isc-base.c int isc_pipeline_init(struct isc_device *isc) · line 2046
Quoted 8 lines
drivers/media/platform/atmel/atmel-isc-base.c +8
-0+ +return isc_validate (isc); +} + +static const struct media_entity_operations isc_entity_operations = { + .link_validate = isc_link_validate, +}; +};
Quoted 66 lines
drivers/media/platform/atmel/atmel-isc-base.c +2
-0int isc_mc_init (struct isc_device *isc,u32 ver) {const struct of_device_id *match;@@ -2022,6 +2051,8 @@ int isc_mc_init(struct isc_device *isc, u32 ver) isc->video_dev.entity.function =MEDIA_ENT_F_IO_V4L ; isc->video_dev.entity.flags =MEDIA_ENT_FL_DEFAULT ; + isc->video_dev.entity.ops = &isc_entity_operations; + isc->pads[ISC_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; ret = media_entity_pads_init(&isc->video_dev.entity, ISC_PADS_NUM,drivers/media/platform/atmel/atmel-isc-scaler.c +5
-0@@ -155,6 +155,10 @@ static const struct v4l2_subdev_pad_ops isc_scaler_pad_ops = { .init_cfg = isc_scaler_init_cfg, }; +static const struct media_entity_operations isc_scaler_entity_ops = { + .link_validate = v4l2_subdev_link_validate, +}; +static const struct v4l2_subdev_ops xisc_scaler_subdev_ops = { .pad = &isc_scaler_pad_ops, };@@ -172,6 +176,7 @@ int isc_scaler_init(struct isc_device *isc) isc->scaler_sd.flags |=V4L2_SUBDEV_FL_HAS_DEVNODE ; isc->scaler_sd.entity.function =MEDIA_ENT_F_PROC_VIDEO_SCALER ; + isc->scaler_sd.entity.ops = &isc_scaler_entity_ops; isc->scaler_pads[ISC_SCALER_PAD_SINK ].flags =MEDIA_PAD_FL_SINK ; isc->scaler_pads[ISC_SCALER_PAD_SOURCE ].flags =MEDIA_PAD_FL_SOURCE ;drivers/media/platform/atmel/atmel-isc.h +3
-1@@ -270,6 +270,7 @@ enum isc_scaler_pads {* @formats_list_size: size of formats_list array * @pads: media controller pads for isc video entity * @mdev: media device that is registered by the isc +* @mpipe: media device pipeline used by the isc * @remote_pad: remote pad on the connected subdevice * @scaler_sd: subdevice for the scaler that isc registers * @scaler_pads: media controller pads for the scaler subdevice @@ -295,6 +296,7 @@ struct isc_device {struct completion comp;struct v4l2_format fmt; +struct v4l2_format try_fmt;struct isc_format **user_formats;unsigned int num_user_formats;@@ -366,6 +368,7 @@ struct isc_device {struct {struct media_pad pads[ISC_PADS_NUM ];struct media_device mdev; +struct media_pipeline mpipe;u32 remote_pad; }; --2.25.1
PATCH v4 8/11 dt-bindings: media: microchip,xisc: add bus-width of 14
In reply to Eugen HristevThe Microchip XISC supports a bus width of 14 bits. Add it to the supported bus widths.
--- Documentation/devicetree/bindings/media/microchip,xisc.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Documentation/devicetree/bindings/media/microchip,xisc.yaml +1 -1
@@ -67,7 +67,7 @@ properties: remote-endpoint: true bus-width: - enum: [8, 9, 10, 11, 12] + enum: [8, 9, 10, 11, 12, 14] default: 12 hsync-active:
Signature · 1 line
2.25.1
Re: [PATCH v4 08/11] dt-bindings: media: microchip,xisc: add bus-width of 14
Quoted 8 lines · Eugen Hristev
The Microchip XISC supports a bus width of 14 bits. Add it to the supported bus widths.
Signed-off-by Eugen Hristev <[email protected]>--- Documentation/devicetree/bindings/media/microchip,xisc.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
PATCH v4 9/11 ARM: dts: at91: sama7g5: add nodes for video capture
In reply to Eugen HristevAdd node for the XISC (eXtended Image Sensor Controller) and CSI2DC (csi2 demux controller).
These nodes represent the top level of the video capture hardware pipeline and are directly connected in hardware.
---
Changes in v4:
- add mandatory property bus type
Changes in v3:
- change bus width for endpoints to the default 14
arch/arm/boot/dts/sama7g5.dtsi | 49 ++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+)
arch/arm/boot/dts/sama7g5.dtsi +49 -0
@@ -236,6 +236,55 @@ sdmmc2: mmc@e120c000 { status = "disabled"; }; + csi2dc: csi2dc@e1404000 { + compatible = "microchip,sama7g5-csi2dc"; + reg = <0xe1404000 0x500>; + clocks = <&pmc PMC_TYPE_PERIPHERAL 34>, <&xisc>; + clock-names = "pclk", "scck"; + assigned-clocks = <&xisc>; + assigned-clock-rates = <266000000>; + + ports { + #address-cells = <1>; + #size-cells = <0>; + port@0 { + reg = <0>; + csi2dc_in: endpoint { + }; + }; + + port@1 { + reg = <1>; + csi2dc_out: endpoint { + bus-width = <14>; + hsync-active = <1>; + vsync-active = <1>; + remote-endpoint = <&xisc_in>; + }; + }; + }; + }; + + xisc: xisc@e1408000 { + compatible = "microchip,sama7g5-isc"; + reg = <0xe1408000 0x2000>; + interrupts = <GIC_SPI 56 IRQ_TYPE_LEVEL_HIGH>; + clocks = <&pmc PMC_TYPE_PERIPHERAL 56>; + clock-names = "hclock"; + #clock-cells = <0>; + clock-output-names = "isc-mck"; + + port { + xisc_in: endpoint { + bus-type = <5>; /* Parallel */ + bus-width = <14>; + hsync-active = <1>; + vsync-active = <1>; + remote-endpoint = <&csi2dc_out>; + }; + }; + }; + pwm: pwm@e1604000 { compatible = "microchip,sama7g5-pwm", "atmel,sama5d2-pwm"; reg = <0xe1604000 0x4000>;
Signature · 1 line
2.25.1
PATCH v4 10/11 ARM: configs: at91: sama7: add xisc and csi2dc
In reply to Eugen HristevEnable XISC and CSI2DC drivers.
--- arch/arm/configs/sama7_defconfig | 2 ++ 1 file changed, 2 insertions(+)
arch/arm/configs/sama7_defconfig +2 -0
@@ -126,6 +126,8 @@ CONFIG_MEDIA_SUPPORT_FILTER=y CONFIG_MEDIA_CAMERA_SUPPORT=y CONFIG_MEDIA_PLATFORM_SUPPORT=y CONFIG_V4L_PLATFORM_DRIVERS=y +CONFIG_VIDEO_ATMEL_XISC=y +CONFIG_VIDEO_MICROCHIP_CSI2DC=y CONFIG_VIDEO_IMX219=m CONFIG_VIDEO_IMX274=m CONFIG_VIDEO_OV5647=m
Signature · 1 line
2.25.1
PATCH v4 11/11 ARM: multi_v7_defconfig: add atmel video pipeline modules
In reply to Eugen HristevAdd drivers for the atmel video capture pipeline: atmel isc, xisc and microchip csi2dc.
--- arch/arm/configs/multi_v7_defconfig | 3 +++ 1 file changed, 3 insertions(+)
arch/arm/configs/multi_v7_defconfig +3 -0
@@ -638,7 +638,10 @@ CONFIG_VIDEO_S5P_MIPI_CSIS=m CONFIG_VIDEO_EXYNOS_FIMC_LITE=m CONFIG_VIDEO_EXYNOS4_FIMC_IS=m CONFIG_VIDEO_RCAR_VIN=m +CONFIG_VIDEO_ATMEL_ISC=m +CONFIG_VIDEO_ATMEL_XISC=m CONFIG_VIDEO_ATMEL_ISI=m +CONFIG_VIDEO_MICROCHIP_CSI2DC=m CONFIG_V4L_MEM2MEM_DRIVERS=y CONFIG_VIDEO_SAMSUNG_S5P_JPEG=m CONFIG_VIDEO_SAMSUNG_S5P_MFC=m
Signature · 1 line
2.25.1