Skip to contentSkip to conversation

imx8m-mini csi-2 limitations

DISCUSSION Petko Manolov 4 participants 8 messages Aug 15, 2022 – Sep 1, 2022 Media (V4L/DVB)

lore
Share & tools
lore
#0 lore raw

Hello guys,

I'm developing a driver for Sony imx492 sensor and i'm using imx8m-mini (Variscite SoM on Symphony carrier board) platform. Thanks to Laurent i dumped the vendor mipi-csi and csi-bridge drivers in favor of mainline (v5.19-rc6 atm) and parts started to click in place. However, there are still a few rough edges.

The sensor is 47mpix, and is only capable of streaming over all four CSI-2 lanes. Each lane is 1.728gbit, which makes 6.912gbps total bandwidth. imx8m-mini's d-phy can handle (on paper) up to 6gbps.

I guess the main question here is: what's the fastest transfer rate that's known to work with this SoC? Does anyone have experience with imx8m-mini in this regard?

Even with the settings that i'm able to obtain a (distorted) test pattern image with, i keep getting tons of:

[99910.847591] imx-mipi-csis 32e30000.mipi-csi: FIFO Overflow Error events: 1353347

IRQs also look a suspicious on the 'csi' part:

214:    2653716          0          0          0     GICv3  49 Level     32e30000.mipi-csi
215:          2          0          0          0     GICv3  48 Level     csi

That's with this clock setup:

 sys_pll2                             1        1        0  1000000000          0     0  50000         Y
    sys_pll2_out                      5        5        0  1000000000          0     0  50000         Y
       sys_pll2_1000m                 3        3        0  1000000000          0     0  50000         Y
          csi1_phy_ref                1        1        0  1000000000          0     0  50000         Y
          csi1_core                   1        1        0   500000000          0     0  50000         Y
             csi1_root_clk            1        1        0   500000000          0     0  50000         Y

However, if i change the latter two (csi1_core & csi1_root_clk) to 1ghz, there are still mipi-csi IRQs coming in, but nothing from the csi-bridge.

Any advice on where i shall start looking first?

thanks,
Petko

#1 lore raw

Hello again,

After digging down NXP's documentation and reading some comments within kernel drivers (sorry Jacopo, you've left some nice comments in imx7-media-csi.c, hence you're spammed now :) i came to the conclusion that mipi-csi to csi-bridge data path can only be 8bits in raw12 format. The only exception is YUV422 format, which get BIT_MIPI_DOUBLE_CMPNT flag set.

So i guess there are two questions:

a) can the csi-bridge (and mipi-csi) be persuaded to do two-bytes for raw12 format;

b) what's the maximum frequency for csi-bridge PIXCLK?

I've increased PIXCLK up to 1ghz (as opposed to the default 500mhz) and this seem to hung the bridge.

cheers,
Petko

Quoted 44 lines · Petko Manolov

Hello guys,

I'm developing a driver for Sony imx492 sensor and i'm using imx8m-mini (Variscite SoM on Symphony carrier board) platform. Thanks to Laurent i dumped the vendor mipi-csi and csi-bridge drivers in favor of mainline (v5.19-rc6 atm) and parts started to click in place. However, there are still a few rough edges.

The sensor is 47mpix, and is only capable of streaming over all four CSI-2 lanes. Each lane is 1.728gbit, which makes 6.912gbps total bandwidth. imx8m-mini's d-phy can handle (on paper) up to 6gbps.

I guess the main question here is: what's the fastest transfer rate that's known to work with this SoC? Does anyone have experience with imx8m-mini in this regard?

Even with the settings that i'm able to obtain a (distorted) test pattern image with, i keep getting tons of:

[99910.847591] imx-mipi-csis 32e30000.mipi-csi: FIFO Overflow Error events: 1353347

IRQs also look a suspicious on the 'csi' part:

214:    2653716          0          0          0     GICv3  49 Level     32e30000.mipi-csi
215:          2          0          0          0     GICv3  48 Level     csi

That's with this clock setup:

 sys_pll2                             1        1        0  1000000000          0     0  50000         Y
    sys_pll2_out                      5        5        0  1000000000          0     0  50000         Y
       sys_pll2_1000m                 3        3        0  1000000000          0     0  50000         Y
          csi1_phy_ref                1        1        0  1000000000          0     0  50000         Y
          csi1_core                   1        1        0   500000000          0     0  50000         Y
             csi1_root_clk            1        1        0   500000000          0     0  50000         Y

However, if i change the latter two (csi1_core & csi1_root_clk) to 1ghz, there are still mipi-csi IRQs coming in, but nothing from the csi-bridge.

Any advice on where i shall start looking first?

thanks,
Petko

#2 lore raw

Hi Petko

Quoted 5 lines · Petko Manolov

Hello again,

After digging down NXP's documentation and reading some comments within kernel drivers (sorry Jacopo, you've left some nice comments in imx7-media-csi.c, hence you're spammed now :) i came to the conclusion that mipi-csi to csi-bridge data

No worries.

I tried re-reading all the discussions, but memory still fails me here.

I recall I had an issue with the CSIS bandwidth and somewhere I suggested that it might be due to the CSI-2 specification version that the peripheral implemented. I can't find that email anymore though :/

Quoted 3 lines

path can only be 8bits in raw12 format. The only exception is YUV422 format, which get BIT_MIPI_DOUBLE_CMPNT flag set.

Yes, there has been quite some discussions about how to mix-and-match the settings between the CSIS (mipi-csi) and the CSI (csi-bridge) [*]

If I recall correctly there are at least three configurations at play:

- The CSIS pixel mode selection

MIPI_CSI_ISP_CONFIG[13:12]

The driver currently forces dual pixel sampling mode for YUV422 and report a TODO:

* TODO: Verify which other formats require DUAL (or QUAD) modes.

Here you're trying to transfer RAW12 data, which is indicated in the register documentation as one of the formats suitable for DUAL/QUAD mode transfer

- I'm actually not sure how and if RAW12 gets expanded to 16 bit samples by filling the 16-bit word with 0s or it either gets packed in 6 bytes with no paddings. Figure "13-33. Pixel alignment" seems to suggest that, but I'm not sure I really got that diagram

- The CSI-bridge

- CSI_CR3[3] SENSOR_16BITS
- CSI_CR18[20] MIPI_DOUBLE_CMP

The comment in imx7_csi_configure() explains how different versions of YUYV (2X8 and 1X16) are handled, and why SENSOR_16BITS and MIPI_DOUBLE_CMP have to be enabled when the csi-bridge has to be instructed to sample 16 bits samples from the RX queue.

I guess the key would be here to find out what combination of SINGLE/DUAL/QUAD sampling is opportune for RAW12 (I would try with DUAL first and then QUAD) and then configure the csi-bridge bus sampling mode accordingly (likely in the same was as it is done for YUYV_1X16)

When it comes to bandwidth limitation, can you try to reduce the sensor output size to make sure you chase one issue at the time ?

[*] Is "CSI" the most abused acronym of history ?

Quoted 4 lines

So i guess there are two questions:

	a) can the csi-bridge (and mipi-csi) be persuaded to do two-bytes for
	raw12 format;

See above :)

Quoted 2 lines

b) what's the maximum frequency for csi-bridge PIXCLK?

Not sure about this one.

I've anyway cc-ed Laurent and Xavier, which can surely be helpful here

Thanks
j

Quoted 53 lines

I've increased PIXCLK up to 1ghz (as opposed to the default 500mhz) and this seem to hung the bridge.

cheers,
Petko

Quoted 44 lines · Petko Manolov

Hello guys,

I'm developing a driver for Sony imx492 sensor and i'm using imx8m-mini (Variscite SoM on Symphony carrier board) platform. Thanks to Laurent i dumped the vendor mipi-csi and csi-bridge drivers in favor of mainline (v5.19-rc6 atm) and parts started to click in place. However, there are still a few rough edges.

The sensor is 47mpix, and is only capable of streaming over all four CSI-2 lanes. Each lane is 1.728gbit, which makes 6.912gbps total bandwidth. imx8m-mini's d-phy can handle (on paper) up to 6gbps.

I guess the main question here is: what's the fastest transfer rate that's known to work with this SoC? Does anyone have experience with imx8m-mini in this regard?

Even with the settings that i'm able to obtain a (distorted) test pattern image with, i keep getting tons of:

[99910.847591] imx-mipi-csis 32e30000.mipi-csi: FIFO Overflow Error events: 1353347

IRQs also look a suspicious on the 'csi' part:

214:    2653716          0          0          0     GICv3  49 Level     32e30000.mipi-csi
215:          2          0          0          0     GICv3  48 Level     csi

That's with this clock setup:

 sys_pll2                             1        1        0  1000000000          0     0  50000         Y
    sys_pll2_out                      5        5        0  1000000000          0     0  50000         Y
       sys_pll2_1000m                 3        3        0  1000000000          0     0  50000         Y
          csi1_phy_ref                1        1        0  1000000000          0     0  50000         Y
          csi1_core                   1        1        0   500000000          0     0  50000         Y
             csi1_root_clk            1        1        0   500000000          0     0  50000         Y

However, if i change the latter two (csi1_core & csi1_root_clk) to 1ghz, there are still mipi-csi IRQs coming in, but nothing from the csi-bridge.

Any advice on where i shall start looking first?

thanks,
Petko

#3 lore raw

Hi Petko,

Am Donnerstag, 18. August 2022, 12:51:31 CEST schrieb Jacopo Mondi:

Quoted 61 lines

Hi Petko

Quoted 6 lines · Petko Manolov

Hello again,

After digging down NXP's documentation and reading some comments within kernel drivers (sorry Jacopo, you've left some nice comments in imx7-media-csi.c, hence you're spammed now :) i came to the conclusion that mipi-csi to csi-bridge data

No worries.

I tried re-reading all the discussions, but memory still fails me here.

I recall I had an issue with the CSIS bandwidth and somewhere I suggested that it might be due to the CSI-2 specification version that the peripheral implemented. I can't find that email anymore though :/

Quoted 2 lines

path can only be 8bits in raw12 format. The only exception is YUV422 format, which get BIT_MIPI_DOUBLE_CMPNT flag set.

Yes, there has been quite some discussions about how to mix-and-match the settings between the CSIS (mipi-csi) and the CSI (csi-bridge) [*]

If I recall correctly there are at least three configurations at play:

- The CSIS pixel mode selection

MIPI_CSI_ISP_CONFIG[13:12]

The driver currently forces dual pixel sampling mode for YUV422 and report a TODO:

* TODO: Verify which other formats require DUAL (or QUAD) modes.

Here you're trying to transfer RAW12 data, which is indicated in the register documentation as one of the formats suitable for DUAL/QUAD mode transfer

- I'm actually not sure how and if RAW12 gets expanded to 16 bit samples by filling the 16-bit word with 0s or it either gets packed in 6 bytes with no paddings. Figure "13-33. Pixel alignment" seems to suggest that, but I'm not sure I really got that diagram

- The CSI-bridge

- CSI_CR3[3] SENSOR_16BITS
- CSI_CR18[20] MIPI_DOUBLE_CMP

The comment in imx7_csi_configure() explains how different versions of YUYV (2X8 and 1X16) are handled, and why SENSOR_16BITS and MIPI_DOUBLE_CMP have to be enabled when the csi-bridge has to be
instructed to sample 16 bits samples from the RX queue.

I guess the key would be here to find out what combination of SINGLE/DUAL/QUAD sampling is opportune for RAW12 (I would try with DUAL first and then QUAD) and then configure the csi-bridge bus sampling mode accordingly (likely in the same was as it is done for YUYV_1X16)

Quoted 7 lines

When it comes to bandwidth limitation, can you try to reduce the sensor output size to make sure you chase one issue at the time ?

[*] Is "CSI" the most abused acronym of history ?

Quoted 2 lines

So i guess there are two questions:
a) can the csi-bridge (and mipi-csi) be persuaded to do two-bytes

for

Quoted 5 lines
Quoted 1 line
	raw12 format;

See above :)

Quoted 1 line

b) what's the maximum frequency for csi-bridge PIXCLK?

While it does not exactly answer your question, the i.MX8M Mini Datasheet (both for Commercial and Industrial) mentions "four-lane MIPI camera serial interfaces, which operates up to a maximum bit rate of 1.5 Gbps". I would assume the bridge is capable of handling the full link speed on four lanes, otherwise this would be quite useless.

Best regards,
Alexander

#4 lore raw
Quoted 1 line · Alexander Stein

Hi Petko,

Hi Alexander,

<snip>

Quoted 5 lines

While it does not exactly answer your question, the i.MX8M Mini Datasheet (both for Commercial and Industrial) mentions "four-lane MIPI camera serial interfaces, which operates up to a maximum bit rate of 1.5 Gbps". I would assume the bridge is capable of handling the full link speed on four lanes, otherwise this would be quite useless.

Ahem, if that the case (useless), it won't be a first.

In theory the d-phy should support 2.5gbps, but in practice it is only 1.5gbps. If CSIS to CSI (bridge) can't be made to transfer 16bit words for 12 bit raw format and if the frequency between both can go up to 500mhz, this leaves us with 4gbps (8bit x 500mhz) max bandwidth.

If the above calculation is correct, this leaves us with only one option - limit the speed of the camera sensor.

cheers,
Petko

#5 lore raw
In reply to Jacopo Mondi
Quoted 10 lines · Jacopo Mondi

Hi Petko

Quoted 6 lines · Petko Manolov

Hello again,

After digging down NXP's documentation and reading some comments within kernel drivers (sorry Jacopo, you've left some nice comments in imx7-media-csi.c, hence you're spammed now :) i came to the conclusion that mipi-csi to csi-bridge data

Quoted 7 lines

No worries.

I tried re-reading all the discussions, but memory still fails me here.

I recall I had an issue with the CSIS bandwidth and somewhere I suggested that it might be due to the CSI-2 specification version that the peripheral implemented. I can't find that email anymore though :/

Well, only bumping up the frequency (to 1ghz) between the CSIS and bridge does not work out of the box. The latter stopped generating IRQs, which kind of ruined the day for me. It would be better if i know if that's at all possible, before start debugging the issue.

Quoted 8 lines

Yes, there has been quite some discussions about how to mix-and-match the settings between the CSIS (mipi-csi) and the CSI (csi-bridge) [*]

If I recall correctly there are at least three configurations at play:

- The CSIS pixel mode selection

MIPI_CSI_ISP_CONFIG[13:12]

Got this one.

Quoted 14 lines

The driver currently forces dual pixel sampling mode for YUV422 and report a TODO:

* TODO: Verify which other formats require DUAL (or QUAD) modes.

Here you're trying to transfer RAW12 data, which is indicated in the register documentation as one of the formats suitable for DUAL/QUAD mode transfer

- I'm actually not sure how and if RAW12 gets expanded to 16 bit samples by filling the 16-bit word with 0s or it either gets packed in 6 bytes with no paddings. Figure "13-33. Pixel alignment" seems to suggest that, but I'm not sure I really got that diagram

Quoted 4 lines

- The CSI-bridge

- CSI_CR3[3] SENSOR_16BITS
- CSI_CR18[20] MIPI_DOUBLE_CMP

Ah, i did not set CSI_CR3 when i (quickly) tried to go to 16bit data width. I'll give it another try based on the above.

Quoted 4 lines

The comment in imx7_csi_configure() explains how different versions of YUYV (2X8 and 1X16) are handled, and why SENSOR_16BITS and MIPI_DOUBLE_CMP have to be enabled when the csi-bridge has to be instructed to sample 16 bits samples from the RX queue.

OK.

Quoted 4 lines

I guess the key would be here to find out what combination of SINGLE/DUAL/QUAD sampling is opportune for RAW12 (I would try with DUAL first and then QUAD) and then configure the csi-bridge bus sampling mode accordingly (likely in the same was as it is done for YUYV_1X16)

I've come across this comment in imx-mipi-csis.c left by you earlier this year:

"Neither of the IP cores connected to the CSIS in i.MX SoCs (CSI bridge or ISI) support quad pixel mode, so this will never work in practice."

I guess, if true, it doesn't make sense to waste time on quad mode.

Quoted 2 lines

When it comes to bandwidth limitation, can you try to reduce the sensor output size to make sure you chase one issue at the time ?

I am at Sony's mercy here. The datasheed it full of holes and i get feeded small bits of information at irregular intervals. I asked them to provide me with register setup for much slower data rate & pixel clock (at full speed the damn thing is streaming at almost 7gbps), but i'm still waiting...

Quoted 1 line

[*] Is "CSI" the most abused acronym of history ?

Maybe. I already hate spelling this acronym even in my mind. :)

Quoted 5 lines
Quoted 4 lines

So i guess there are two questions:

	a) can the csi-bridge (and mipi-csi) be persuaded to do two-bytes for
	raw12 format;
Quoted 8 lines

See above :)

Quoted 2 lines

b) what's the maximum frequency for csi-bridge PIXCLK?

Not sure about this one.

I've anyway cc-ed Laurent and Xavier, which can surely be helpful here

Thanks a bunch, the above information is very helpful.

cheers,
Petko

#6 lore raw

Hello,

Quoted 5 lines · Petko Manolov
Quoted 4 lines · Jacopo Mondi
Quoted 3 lines · Petko Manolov

Hello again,

After digging down NXP's documentation and reading some comments within

Please note that the documentation is sometimes confusing and sometimes incorrect (it would be too easy otherwise, wouldn't it ?).

How to configure the CSIS output bus depends on the SoC integration. If I recall correctly, all i.MX SoCs that connect the CSIS to a CSI bridge have the same integration, but when the CSIS is connected to other IP cores, configuration requirements may vary.

Quoted 20 lines
Quoted 11 lines
Quoted 3 lines

kernel drivers (sorry Jacopo, you've left some nice comments in imx7-media-csi.c, hence you're spammed now :) i came to the conclusion that mipi-csi to csi-bridge data

No worries.

I tried re-reading all the discussions, but memory still fails me here.

I recall I had an issue with the CSIS bandwidth and somewhere I suggested that it might be due to the CSI-2 specification version that the peripheral implemented. I can't find that email anymore though :/

Well, only bumping up the frequency (to 1ghz) between the CSIS and bridge does not work out of the box. The latter stopped generating IRQs, which kind of ruined the day for me. It would be better if i know if that's at all possible, before start debugging the issue.

Quoted 3 lines

Yes, there has been quite some discussions about how to mix-and-match the settings between the CSIS (mipi-csi) and the CSI (csi-bridge) [*]

Quoted 21 lines
Quoted 5 lines

If I recall correctly there are at least three configurations at play:

- The CSIS pixel mode selection

MIPI_CSI_ISP_CONFIG[13:12]

Got this one.

Quoted 13 lines

The driver currently forces dual pixel sampling mode for YUV422 and report a TODO:

* TODO: Verify which other formats require DUAL (or QUAD) modes.

Here you're trying to transfer RAW12 data, which is indicated in the register documentation as one of the formats suitable for DUAL/QUAD mode transfer

- I'm actually not sure how and if RAW12 gets expanded to 16 bit samples by filling the 16-bit word with 0s or it either gets packed in 6 bytes with no paddings. Figure "13-33. Pixel alignment" seems to suggest that, but I'm not sure I really got that diagram

That would imply a 48-bit bus between the CSIS and the CSI bridge, and I don't think that's the case.

When reading the CSIS documentation, keep in mind that the IP comes from a third party (Samsung). The CSIS documentation in the i.MX8MM reference manual comes from a generic documentation of the IP core, which offers different output bus widths (among other integration options). It doesn't precisely describe the synthesis options uses in the i.MX8MM.

Quoted 20 lines
Quoted 4 lines

- The CSI-bridge

- CSI_CR3[3] SENSOR_16BITS
- CSI_CR18[20] MIPI_DOUBLE_CMP

Ah, i did not set CSI_CR3 when i (quickly) tried to go to 16bit data width. I'll give it another try based on the above.

Quoted 4 lines

The comment in imx7_csi_configure() explains how different versions of YUYV (2X8 and 1X16) are handled, and why SENSOR_16BITS and MIPI_DOUBLE_CMP have to be enabled when the csi-bridge has to be instructed to sample 16 bits samples from the RX queue.

OK.

Quoted 4 lines

I guess the key would be here to find out what combination of SINGLE/DUAL/QUAD sampling is opportune for RAW12 (I would try with DUAL first and then QUAD) and then configure the csi-bridge bus sampling mode accordingly (likely in the same was as it is done for YUYV_1X16)

Quoted 6 lines

I've come across this comment in imx-mipi-csis.c left by you earlier this year:

"Neither of the IP cores connected to the CSIS in i.MX SoCs (CSI bridge or ISI) support quad pixel mode, so this will never work in practice."

I guess, if true, it doesn't make sense to waste time on quad mode.

To the best of my recollection, that's accurate.

Quoted 27 lines
Quoted 2 lines

When it comes to bandwidth limitation, can you try to reduce the sensor output size to make sure you chase one issue at the time ?

I am at Sony's mercy here. The datasheed it full of holes and i get feeded small bits of information at irregular intervals. I asked them to provide me with register setup for much slower data rate & pixel clock (at full speed the damn thing is streaming at almost 7gbps), but i'm still waiting...

Quoted 1 line

[*] Is "CSI" the most abused acronym of history ?

Maybe. I already hate spelling this acronym even in my mind. :)

Quoted 5 lines
Quoted 4 lines

So i guess there are two questions:

	a) can the csi-bridge (and mipi-csi) be persuaded to do two-bytes for
	raw12 format;
Quoted 8 lines

See above :)

Quoted 2 lines

b) what's the maximum frequency for csi-bridge PIXCLK?

Not sure about this one.

I've anyway cc-ed Laurent and Xavier, which can surely be helpful here

Thanks a bunch, the above information is very helpful.

Signature · 3 lines
Regards,

Laurent Pinchart
#7 lore raw
In reply to Jacopo Mondi
Quoted 1 line · Jacopo Mondi

<snip>

Quoted 5 lines

If I recall correctly there are at least three configurations at play:

- The CSIS pixel mode selection

MIPI_CSI_ISP_CONFIG[13:12]

Funny enough adding MIPI_CSIS_ISPCFG_PIXEL_MODE_QUAD and MIPI_CSIS_ISPCFG_ALIGN_32BIT to the config(0) register actually produced an image. Highly distorted image, but did not break the image acquisition.

MIPI_CSIS_ISPCFG_ALIGN_32BIT seems to be the wrong flag here, but it without it the streaming got blocked.

Quoted 15 lines

The driver currently forces dual pixel sampling mode for YUV422 and report a TODO:

* TODO: Verify which other formats require DUAL (or QUAD) modes.

Here you're trying to transfer RAW12 data, which is indicated in the register documentation as one of the formats suitable for DUAL/QUAD mode transfer

- I'm actually not sure how and if RAW12 gets expanded to 16 bit samples by filling the 16-bit word with 0s or it either gets packed in 6 bytes with no paddings. Figure "13-33. Pixel alignment" seems to suggest that, but I'm not sure I really got that diagram

Quoted 3 lines

- The CSI-bridge

- CSI_CR3[3] SENSOR_16BITS

Actually the driver is always setting this one for raw10/12 formats.

Quoted 1 line

- CSI_CR18[20] MIPI_DOUBLE_CMP

Enabling this bit for raw10/12 formats kills the transmission. According to the documentation this flag transfers full pixel per clock cycle as opposed to just half a pixel.

Quoted 4 lines

The comment in imx7_csi_configure() explains how different versions of YUYV (2X8 and 1X16) are handled, and why SENSOR_16BITS and MIPI_DOUBLE_CMP have to be enabled when the csi-bridge has to be instructed to sample 16 bits samples from the RX queue.

Seems like i have to find a way to make the mipi-csi stream full pixel (iow 16bit), if any such thing exists. And hope this configuration is supported between both parts, mipi & bridge.

Quoted 4 lines

I guess the key would be here to find out what combination of SINGLE/DUAL/QUAD sampling is opportune for RAW12 (I would try with DUAL first and then QUAD) and then configure the csi-bridge bus sampling mode accordingly (likely in the same was as it is done for YUYV_1X16)

In fact i was mimicking MEDIA_BUS_FMT_YUYV8_1X16 case, but as mentioned, this didn't work. Seems like raw10/12 formats are not so well (if at all) handled by the bridge.

Quoted 2 lines

When it comes to bandwidth limitation, can you try to reduce the sensor output size to make sure you chase one issue at the time ?

I used insanely big vblank and hblank values. Obtaining single frame took about 90 seconds. This indeed reduced "FIFO Overflow Error" and "Frame Size Error" to four digit number (the lowest rate i've seen so far), but there was no visible improvement in the captured test pattern image.

I doubt i can make the sensor stream slower and i also think it doesn't make sense to go down this path - there are still errors.

Any advice on how to debug the CSI bridge? NXP support is as useless as ever.

cheers,
Petko