Possible bug on wilc1000 [Klartext]

EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe

Hello,

I tested the wireless chip wilc1000 with the 5.16.5 Kernel and the firmware v15.4.1
(https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/atmel/wilc1000_wifi_firmware-1.bin)
on an i.MX6 QUAD with iperf3:

# iperf3 -c IP_ADDR -P 16 -t 0

After a while the test gets stuck and I got the following kernel messages:
mmc0: Timeout waiting for hardware interrupt.
mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
mmc0: sdhci: Sys addr: 0x138f0200 | Version: 0x00000002
mmc0: sdhci: Blk size: 0x00000158 | Blk cnt: 0x00000001
mmc0: sdhci: Argument: 0x14000158 | Trn mode: 0x00000013
mmc0: sdhci: Present: 0x01d88a0a | Host ctl: 0x00000013
mmc0: sdhci: Power: 0x00000002 | Blk gap: 0x00000080
mmc0: sdhci: Wake-up: 0x00000008 | Clock: 0x0000009f
mmc0: sdhci: Timeout: 0x0000008f | Int stat: 0x00000000
mmc0: sdhci: Int enab: 0x107f100b | Sig enab: 0x107f100b
mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000003
mmc0: sdhci: Caps: 0x07eb0000 | Caps_1: 0x0000a000
mmc0: sdhci: Cmd: 0x0000353a | Max curr: 0x00ffffff
mmc0: sdhci: Resp[0]: 0x00001000 | Resp[1]: 0x00000000
mmc0: sdhci: Resp[2]: 0x00000000 | Resp[3]: 0x00000000
mmc0: sdhci: Host ctl2: 0x00000000
mmc0: sdhci: ADMA Err: 0x00000007 | ADMA Ptr: 0x4c041200
mmc0: sdhci-esdhc-imx: ========= ESDHC IMX DEBUG STATUS DUMP =========
mmc0: sdhci-esdhc-imx: cmd debug status: 0x2100
mmc0: sdhci-esdhc-imx: data debug status: 0x2200
mmc0: sdhci-esdhc-imx: trans debug status: 0x2300
mmc0: sdhci-esdhc-imx: dma debug status: 0x2402
mmc0: sdhci-esdhc-imx: adma debug status: 0x25b4
mmc0: sdhci-esdhc-imx: fifo debug status: 0x2610
mmc0: sdhci-esdhc-imx: async fifo debug status: 0x2751
mmc0: sdhci: ============================================ wilc1000_sdio mmc0:0001:1: wilc_sdio_cmd53..failed, err(-110) wilc1000_sdio mmc0:0001:1: Failed cmd53 [0], bytes read...

I tried to reduce the clock speed to 20MHz in the devicetree with max-frequency = <20000000>;
but the problem then also occurs.

Is this a possible bug?

Quoted 39 lines

Hello,

I tested the wireless chip wilc1000 with the 5.16.5 Kernel and the firmware v15.4.1
(https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/atmel/wilc1000_wifi_firmware-1.bin)
on an i.MX6 QUAD with iperf3:

# iperf3 -c IP_ADDR -P 16 -t 0

After a while the test gets stuck and I got the following kernel messages:
mmc0: Timeout waiting for hardware interrupt.
mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
mmc0: sdhci: Sys addr: 0x138f0200 | Version: 0x00000002
mmc0: sdhci: Blk size: 0x00000158 | Blk cnt: 0x00000001
mmc0: sdhci: Argument: 0x14000158 | Trn mode: 0x00000013
mmc0: sdhci: Present: 0x01d88a0a | Host ctl: 0x00000013
mmc0: sdhci: Power: 0x00000002 | Blk gap: 0x00000080
mmc0: sdhci: Wake-up: 0x00000008 | Clock: 0x0000009f
mmc0: sdhci: Timeout: 0x0000008f | Int stat: 0x00000000
mmc0: sdhci: Int enab: 0x107f100b | Sig enab: 0x107f100b
mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000003
mmc0: sdhci: Caps: 0x07eb0000 | Caps_1: 0x0000a000
mmc0: sdhci: Cmd: 0x0000353a | Max curr: 0x00ffffff
mmc0: sdhci: Resp[0]: 0x00001000 | Resp[1]: 0x00000000
mmc0: sdhci: Resp[2]: 0x00000000 | Resp[3]: 0x00000000
mmc0: sdhci: Host ctl2: 0x00000000
mmc0: sdhci: ADMA Err: 0x00000007 | ADMA Ptr: 0x4c041200
mmc0: sdhci-esdhc-imx: ========= ESDHC IMX DEBUG STATUS DUMP =========
mmc0: sdhci-esdhc-imx: cmd debug status: 0x2100
mmc0: sdhci-esdhc-imx: data debug status: 0x2200
mmc0: sdhci-esdhc-imx: trans debug status: 0x2300
mmc0: sdhci-esdhc-imx: dma debug status: 0x2402
mmc0: sdhci-esdhc-imx: adma debug status: 0x25b4
mmc0: sdhci-esdhc-imx: fifo debug status: 0x2610
mmc0: sdhci-esdhc-imx: async fifo debug status: 0x2751
mmc0: sdhci: ============================================ wilc1000_sdio mmc0:0001:1: wilc_sdio_cmd53..failed, err(-110) wilc1000_sdio mmc0:0001:1: Failed cmd53 [0], bytes read...

Quoted 7 lines

I tried to reduce the clock speed to 20MHz in the devicetree with max-frequency = <20000000>;
but the problem then also occurs.

Is this a possible bug?

The bus error seems to be specific to the host during the SDIO transfer. How long does it take to reproduce it? Does the issue also happen without "-P 16" iPerf3 option?

It might be worth trying with power-save mode disabled. The PS mode can be disabled either by setting WID_POWER_MANAGEMENT WID to 0(WILC_FW_NO_POWERSAVE) in "wilc_set_power_mgmt()" or compile the kernel with CFG80211_DEFAULT_PS disabled.

From: [email protected] [mailto:[email protected]]
Sent: Wednesday, February 9, 2022 3:37 PM

Quoted 39 lines · Christoph Niedermaier

Quoted 38 lines

Hello,

I tested the wireless chip wilc1000 with the 5.16.5 Kernel and the firmware v15.4.1
(https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/atmel/wilc1000_wifi_firmware-1.bin)
on an i.MX6 QUAD with iperf3:

# iperf3 -c IP_ADDR -P 16 -t 0

After a while the test gets stuck and I got the following kernel messages:
mmc0: Timeout waiting for hardware interrupt.
mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
mmc0: sdhci: Sys addr: 0x138f0200 | Version: 0x00000002
mmc0: sdhci: Blk size: 0x00000158 | Blk cnt: 0x00000001
mmc0: sdhci: Argument: 0x14000158 | Trn mode: 0x00000013
mmc0: sdhci: Present: 0x01d88a0a | Host ctl: 0x00000013
mmc0: sdhci: Power: 0x00000002 | Blk gap: 0x00000080
mmc0: sdhci: Wake-up: 0x00000008 | Clock: 0x0000009f
mmc0: sdhci: Timeout: 0x0000008f | Int stat: 0x00000000
mmc0: sdhci: Int enab: 0x107f100b | Sig enab: 0x107f100b
mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000003
mmc0: sdhci: Caps: 0x07eb0000 | Caps_1: 0x0000a000
mmc0: sdhci: Cmd: 0x0000353a | Max curr: 0x00ffffff
mmc0: sdhci: Resp[0]: 0x00001000 | Resp[1]: 0x00000000
mmc0: sdhci: Resp[2]: 0x00000000 | Resp[3]: 0x00000000
mmc0: sdhci: Host ctl2: 0x00000000
mmc0: sdhci: ADMA Err: 0x00000007 | ADMA Ptr: 0x4c041200
mmc0: sdhci-esdhc-imx: ========= ESDHC IMX DEBUG STATUS DUMP =========
mmc0: sdhci-esdhc-imx: cmd debug status: 0x2100
mmc0: sdhci-esdhc-imx: data debug status: 0x2200
mmc0: sdhci-esdhc-imx: trans debug status: 0x2300
mmc0: sdhci-esdhc-imx: dma debug status: 0x2402
mmc0: sdhci-esdhc-imx: adma debug status: 0x25b4
mmc0: sdhci-esdhc-imx: fifo debug status: 0x2610
mmc0: sdhci-esdhc-imx: async fifo debug status: 0x2751
mmc0: sdhci: ============================================ wilc1000_sdio mmc0:0001:1: wilc_sdio_cmd53..failed, err(-110) wilc1000_sdio mmc0:0001:1: Failed cmd53 [0], bytes read...

Quoted 7 lines

Quoted 7 lines

I tried to reduce the clock speed to 20MHz in the devicetree with max-frequency = <20000000>;
but the problem then also occurs.

Is this a possible bug?

Hi Ajay,
Thanks for the answer.

Quoted 3 lines

The bus error seems to be specific to the host during the SDIO transfer. How long does it take to reproduce it? Does the issue also happen without "-P 16" iPerf3 option?

It takes about 10s (something a bit longer) till I got this kernel error messages and it doesn't matter if I use it with "-P 16" or without.

Quoted 54 lines

From: [email protected] [mailto:[email protected]]
Sent: Wednesday, February 9, 2022 3:37 PM

Quoted 39 lines · Christoph Niedermaier

Quoted 38 lines

Hello,

I tested the wireless chip wilc1000 with the 5.16.5 Kernel and the firmware v15.4.1
(https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/atmel/wilc1000_wifi_firmware-1.bin)
on an i.MX6 QUAD with iperf3:

# iperf3 -c IP_ADDR -P 16 -t 0

After a while the test gets stuck and I got the following kernel messages:
mmc0: Timeout waiting for hardware interrupt.
mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
mmc0: sdhci: Sys addr: 0x138f0200 | Version: 0x00000002
mmc0: sdhci: Blk size: 0x00000158 | Blk cnt: 0x00000001
mmc0: sdhci: Argument: 0x14000158 | Trn mode: 0x00000013
mmc0: sdhci: Present: 0x01d88a0a | Host ctl: 0x00000013
mmc0: sdhci: Power: 0x00000002 | Blk gap: 0x00000080
mmc0: sdhci: Wake-up: 0x00000008 | Clock: 0x0000009f
mmc0: sdhci: Timeout: 0x0000008f | Int stat: 0x00000000
mmc0: sdhci: Int enab: 0x107f100b | Sig enab: 0x107f100b
mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000003
mmc0: sdhci: Caps: 0x07eb0000 | Caps_1: 0x0000a000
mmc0: sdhci: Cmd: 0x0000353a | Max curr: 0x00ffffff
mmc0: sdhci: Resp[0]: 0x00001000 | Resp[1]: 0x00000000
mmc0: sdhci: Resp[2]: 0x00000000 | Resp[3]: 0x00000000
mmc0: sdhci: Host ctl2: 0x00000000
mmc0: sdhci: ADMA Err: 0x00000007 | ADMA Ptr: 0x4c041200
mmc0: sdhci-esdhc-imx: ========= ESDHC IMX DEBUG STATUS DUMP =========
mmc0: sdhci-esdhc-imx: cmd debug status: 0x2100
mmc0: sdhci-esdhc-imx: data debug status: 0x2200
mmc0: sdhci-esdhc-imx: trans debug status: 0x2300
mmc0: sdhci-esdhc-imx: dma debug status: 0x2402
mmc0: sdhci-esdhc-imx: adma debug status: 0x25b4
mmc0: sdhci-esdhc-imx: fifo debug status: 0x2610
mmc0: sdhci-esdhc-imx: async fifo debug status: 0x2751
mmc0: sdhci: ============================================ wilc1000_sdio mmc0:0001:1: wilc_sdio_cmd53..failed, err(-110) wilc1000_sdio mmc0:0001:1: Failed cmd53 [0], bytes read...

Quoted 7 lines

Quoted 7 lines

I tried to reduce the clock speed to 20MHz in the devicetree with max-frequency = <20000000>;
but the problem then also occurs.

Is this a possible bug?

Hi Ajay,
Thanks for the answer.

Quoted 3 lines

The bus error seems to be specific to the host during the SDIO transfer. How long does it take to reproduce it? Does the issue also happen without "-P 16" iPerf3 option?

Quoted 2 lines

It takes about 10s (something a bit longer) till I got this kernel error messages and it doesn't matter if I use it with "-P 16" or without.

I did not observe the issue with my setup(SAMA5D4 XPLAINED + WILC1000 SDIO) when tested iPerf for a longer duration(~1000sec). I suspect the issue could be related to the SDHCI host controller.
Try to debug the host controller side for the possible cause of timeout.

On 2/10/22 17:19, [email protected] wrote:

Hi,

Quoted 65 lines · Christoph Niedermaier

Quoted 62 lines

From: [email protected] [mailto:[email protected]]
Sent: Wednesday, February 9, 2022 3:37 PM

Quoted 43 lines · Christoph Niedermaier

Quoted 42 lines

Hello,

I tested the wireless chip wilc1000 with the 5.16.5 Kernel and the firmware v15.4.1
(https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/atmel/wilc1000_wifi_firmware-1.bin)

on an i.MX6 QUAD with iperf3:

# iperf3 -c IP_ADDR -P 16 -t 0

After a while the test gets stuck and I got the following kernel messages:
mmc0: Timeout waiting for hardware interrupt.
mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
mmc0: sdhci: Sys addr:  0x138f0200 | Version:  0x00000002
mmc0: sdhci: Blk size:  0x00000158 | Blk cnt:  0x00000001
mmc0: sdhci: Argument:  0x14000158 | Trn mode: 0x00000013
mmc0: sdhci: Present:   0x01d88a0a | Host ctl: 0x00000013
mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000009f
mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
mmc0: sdhci: Int enab:  0x107f100b | Sig enab: 0x107f100b
mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000003
mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x0000a000
mmc0: sdhci: Cmd:       0x0000353a | Max curr: 0x00ffffff
mmc0: sdhci: Resp[0]:   0x00001000 | Resp[1]:  0x00000000
mmc0: sdhci: Resp[2]:   0x00000000 | Resp[3]:  0x00000000
mmc0: sdhci: Host ctl2: 0x00000000
mmc0: sdhci: ADMA Err:  0x00000007 | ADMA Ptr: 0x4c041200
mmc0: sdhci-esdhc-imx: ========= ESDHC IMX DEBUG STATUS DUMP
=========
mmc0: sdhci-esdhc-imx: cmd debug status:  0x2100
mmc0: sdhci-esdhc-imx: data debug status:  0x2200
mmc0: sdhci-esdhc-imx: trans debug status:  0x2300
mmc0: sdhci-esdhc-imx: dma debug status:  0x2402
mmc0: sdhci-esdhc-imx: adma debug status:  0x25b4
mmc0: sdhci-esdhc-imx: fifo debug status:  0x2610
mmc0: sdhci-esdhc-imx: async fifo debug status:  0x2751
mmc0: sdhci: ============================================ wilc1000_sdio mmc0:0001:1: wilc_sdio_cmd53..failed, err(-110) wilc1000_sdio mmc0:0001:1: Failed cmd53 [0], bytes read...

Quoted 7 lines

Quoted 7 lines

I tried to reduce the clock speed to 20MHz in the devicetree with max-frequency = <20000000>;
but the problem then also occurs.

Is this a possible bug?

Hi Ajay,
Thanks for the answer.

Quoted 4 lines

The bus error seems to be specific to the host during the SDIO transfer.
How long does it take to reproduce it? Does the issue also happen without "-P 16" iPerf3 option?

It takes about 10s (something a bit longer) till I got this kernel error
messages and it doesn't matter if I use it with "-P 16" or without.

Quoted 4 lines

I did not observe the issue with my setup(SAMA5D4 XPLAINED + WILC1000 SDIO) when tested iPerf for a longer duration(~1000sec). I suspect the issue could be related to the SDHCI host controller.
Try to debug the host controller side for the possible cause of timeout.

It seems the timeout happens because the card fails to respond to SDIO command 53, right ?

Is there some error logging/tracing functionality in the WILC1000 firmware which can provide further information why the card did not respond ?

Could it be the card suffered some sort of FIFO overflow ? The MX6Q is a bit more performant than the CA7 (I think?) SAMA5D4, so maybe that plays some role ?

Quoted 75 lines · Marek Vasut

On 2/10/22 17:19, [email protected] wrote:

Hi,

Quoted 69 lines · Christoph Niedermaier

Quoted 55 lines

From: [email protected] [mailto:[email protected]]
Sent: Wednesday, February 9, 2022 3:37 PM

Quoted 43 lines · Christoph Niedermaier

Quoted 42 lines

Hello,

I tested the wireless chip wilc1000 with the 5.16.5 Kernel and the firmware v15.4.1
(https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/atmel/wilc1000_wifi_firmware-1.bin)

on an i.MX6 QUAD with iperf3:

# iperf3 -c IP_ADDR -P 16 -t 0

After a while the test gets stuck and I got the following kernel messages:
mmc0: Timeout waiting for hardware interrupt.
mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
mmc0: sdhci: Sys addr:  0x138f0200 | Version:  0x00000002
mmc0: sdhci: Blk size:  0x00000158 | Blk cnt:  0x00000001
mmc0: sdhci: Argument:  0x14000158 | Trn mode: 0x00000013
mmc0: sdhci: Present:   0x01d88a0a | Host ctl: 0x00000013
mmc0: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
mmc0: sdhci: Wake-up:   0x00000008 | Clock:    0x0000009f
mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
mmc0: sdhci: Int enab:  0x107f100b | Sig enab: 0x107f100b
mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000003
mmc0: sdhci: Caps:      0x07eb0000 | Caps_1:   0x0000a000
mmc0: sdhci: Cmd:       0x0000353a | Max curr: 0x00ffffff
mmc0: sdhci: Resp[0]:   0x00001000 | Resp[1]:  0x00000000
mmc0: sdhci: Resp[2]:   0x00000000 | Resp[3]:  0x00000000
mmc0: sdhci: Host ctl2: 0x00000000
mmc0: sdhci: ADMA Err:  0x00000007 | ADMA Ptr: 0x4c041200
mmc0: sdhci-esdhc-imx: ========= ESDHC IMX DEBUG STATUS DUMP
=========
mmc0: sdhci-esdhc-imx: cmd debug status:  0x2100
mmc0: sdhci-esdhc-imx: data debug status:  0x2200
mmc0: sdhci-esdhc-imx: trans debug status:  0x2300
mmc0: sdhci-esdhc-imx: dma debug status:  0x2402
mmc0: sdhci-esdhc-imx: adma debug status:  0x25b4
mmc0: sdhci-esdhc-imx: fifo debug status:  0x2610
mmc0: sdhci-esdhc-imx: async fifo debug status:  0x2751
mmc0: sdhci: ============================================ wilc1000_sdio mmc0:0001:1: wilc_sdio_cmd53..failed, err(-110) wilc1000_sdio mmc0:0001:1: Failed cmd53 [0], bytes read...

Quoted 7 lines

Quoted 7 lines

I tried to reduce the clock speed to 20MHz in the devicetree with max-frequency = <20000000>;
but the problem then also occurs.

Is this a possible bug?

Hi Ajay,
Thanks for the answer.

Quoted 7 lines

Quoted 4 lines

The bus error seems to be specific to the host during the SDIO transfer.
How long does it take to reproduce it? Does the issue also happen without "-P 16" iPerf3 option?

It takes about 10s (something a bit longer) till I got this kernel error
messages and it doesn't matter if I use it with "-P 16" or without.

I did not observe the issue with my setup(SAMA5D4 XPLAINED + WILC1000 SDIO) when tested iPerf for a longer duration(~1000sec). I suspect the issue could be related to the SDHCI host controller.
Try to debug the host controller side for the possible cause of timeout.

Quoted 3 lines

It seems the timeout happens because the card fails to respond to SDIO command 53, right ?

Yes, the timeout could be for any reason like either the CMD53 has not reached to chip or response not received correctly at host end.

Quoted 3 lines

Is there some error logging/tracing functionality in the WILC1000 firmware which can provide further information why the card did not respond ?

WILC1000 SD module has UART serial debug port for firmware logs but I don't think it would be useful here because it needs to be debug/probe at SDIO bus level.

Quoted 3 lines

Could it be the card suffered some sort of FIFO overflow ? The MX6Q is a bit more performant than the CA7 (I think?) SAMA5D4, so maybe that plays some role ?

As I understand, the issue is observed with basic iPerf testing(less throughput) so not sure if the host performance will have such an impact. IIRC few of the customers are using the same host(i.MX6) though I am not sure if it's over SPI or SDIO bus. Till now, I have not come across such limitations with the specific host.

EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe

Quoted 78 lines · On 2/11/22 11:15, [email protected] wrote:

Quoted 77 lines · Marek Vasut

On 2/10/22 17:19, [email protected] wrote:

Hi,

Quoted 71 lines · Christoph Niedermaier

Quoted 56 lines

From: [email protected] [mailto:[email protected]]
Sent: Wednesday, February 9, 2022 3:37 PM

Quoted 44 lines · Christoph Niedermaier

Quoted 43 lines

Hello,

I tested the wireless chip wilc1000 with the 5.16.5 Kernel and the firmware v15.4.1
(https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/atmel/wilc1000_wifi_firmware-1.bin)

on an i.MX6 QUAD with iperf3:

# iperf3 -c IP_ADDR -P 16 -t 0

After a while the test gets stuck and I got the following kernel messages:
mmc0: Timeout waiting for hardware interrupt.
mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
mmc0: sdhci: Sys addr:  0x138f0200 | Version: 0x00000002
mmc0: sdhci: Blk size:  0x00000158 | Blk cnt: 0x00000001
mmc0: sdhci: Argument:  0x14000158 | Trn mode: 0x00000013
mmc0: sdhci: Present:   0x01d88a0a | Host ctl: 0x00000013
mmc0: sdhci: Power:     0x00000002 | Blk gap: 0x00000080
mmc0: sdhci: Wake-up:   0x00000008 | Clock: 0x0000009f
mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
mmc0: sdhci: Int enab:  0x107f100b | Sig enab: 0x107f100b
mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000003
mmc0: sdhci: Caps:      0x07eb0000 | Caps_1: 0x0000a000
mmc0: sdhci: Cmd:       0x0000353a | Max curr: 0x00ffffff
mmc0: sdhci: Resp[0]:   0x00001000 | Resp[1]: 0x00000000
mmc0: sdhci: Resp[2]:   0x00000000 | Resp[3]: 0x00000000
mmc0: sdhci: Host ctl2: 0x00000000
mmc0: sdhci: ADMA Err:  0x00000007 | ADMA Ptr: 0x4c041200
mmc0: sdhci-esdhc-imx: ========= ESDHC IMX DEBUG STATUS DUMP
=========
mmc0: sdhci-esdhc-imx: cmd debug status:  0x2100
mmc0: sdhci-esdhc-imx: data debug status:  0x2200
mmc0: sdhci-esdhc-imx: trans debug status:  0x2300
mmc0: sdhci-esdhc-imx: dma debug status:  0x2402
mmc0: sdhci-esdhc-imx: adma debug status:  0x25b4
mmc0: sdhci-esdhc-imx: fifo debug status:  0x2610
mmc0: sdhci-esdhc-imx: async fifo debug status: 0x2751
mmc0: sdhci: ============================================ wilc1000_sdio mmc0:0001:1: wilc_sdio_cmd53..failed, err(-110) wilc1000_sdio mmc0:0001:1: Failed cmd53 [0], bytes read...

Quoted 7 lines

Quoted 7 lines

I tried to reduce the clock speed to 20MHz in the devicetree with max-frequency = <20000000>;
but the problem then also occurs.

Is this a possible bug?

Hi Ajay,
Thanks for the answer.

Quoted 7 lines

Quoted 4 lines

The bus error seems to be specific to the host during the SDIO transfer.
How long does it take to reproduce it? Does the issue also happen without "-P 16" iPerf3 option?

It takes about 10s (something a bit longer) till I got this kernel error
messages and it doesn't matter if I use it with "-P 16" or without.

I did not observe the issue with my setup(SAMA5D4 XPLAINED + WILC1000 SDIO) when tested iPerf for a longer duration(~1000sec). I suspect the issue could be related to the SDHCI host controller.
Try to debug the host controller side for the possible cause of timeout.

Quoted 6 lines

Quoted 3 lines

It seems the timeout happens because the card fails to respond to SDIO command 53, right ?

Yes, the timeout could be for any reason like either the CMD53 has not reached to chip or response not received correctly at host end.

The problem happens seconds or tens of seconds into the test, so there must've been CMD53 which reached the card before the problem occurred, and there must have been a lot of those CMD53 before the problem happened too, since CMD53 seems to be some data transfer CMD ?

Quoted 8 lines

Quoted 3 lines

Is there some error logging/tracing functionality in the WILC1000 firmware which can provide further information why the card did not respond ?

WILC1000 SD module has UART serial debug port for firmware logs but I don't think it would be useful here because it needs to be debug/probe at SDIO bus level.

Is there some other kind of logging which can tell us more details on where to look for this problem ?

Maybe we can try monitoring the SDIO traffic with ftrace ?

Any other options, short of taking the hardware apart ?

Quoted 11 lines

Quoted 5 lines

Could it be the card suffered some sort of FIFO overflow ? The MX6Q is a
bit more performant than the CA7 (I think?) SAMA5D4, so maybe that plays
some role ?

As I understand, the issue is observed with basic iPerf testing(less throughput) so not sure if the host performance will have such an impact. IIRC few of the customers are using the same host(i.MX6) though I am not sure if it's over SPI or SDIO bus. Till now, I have not come across such limitations with the specific host.

That iperf -P 16 hammers the chip with a lot of short packets, the problem does not occur during iperf3 -P 1 run or UDP iperf3 run (that's the one with low traffic). Here the interface is saturated, that's why I speculate some sort of FIFO overrun is happening.

I have also noticed there are some wilc1000 downstream drivers with huge stacks of patches, but I never really figured out whether those are still relevant or whether the upstream wilc1000 driver is perfectly fine. I would like to believe it is the later, is it ?

Quoted 122 lines · Marek Vasut

EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe

Quoted 78 lines · On 2/11/22 11:15, [email protected] wrote:

Quoted 77 lines · Marek Vasut

On 2/10/22 17:19, [email protected] wrote:

Hi,

Quoted 71 lines · Christoph Niedermaier

Quoted 56 lines

From: [email protected] [mailto:[email protected]]
Sent: Wednesday, February 9, 2022 3:37 PM

Quoted 44 lines · Christoph Niedermaier

Quoted 43 lines

Hello,

I tested the wireless chip wilc1000 with the 5.16.5 Kernel and the firmware v15.4.1
(https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/atmel/wilc1000_wifi_firmware-1.bin)

on an i.MX6 QUAD with iperf3:

# iperf3 -c IP_ADDR -P 16 -t 0

After a while the test gets stuck and I got the following kernel messages:
mmc0: Timeout waiting for hardware interrupt.
mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
mmc0: sdhci: Sys addr:  0x138f0200 | Version: 0x00000002
mmc0: sdhci: Blk size:  0x00000158 | Blk cnt: 0x00000001
mmc0: sdhci: Argument:  0x14000158 | Trn mode: 0x00000013
mmc0: sdhci: Present:   0x01d88a0a | Host ctl: 0x00000013
mmc0: sdhci: Power:     0x00000002 | Blk gap: 0x00000080
mmc0: sdhci: Wake-up:   0x00000008 | Clock: 0x0000009f
mmc0: sdhci: Timeout:   0x0000008f | Int stat: 0x00000000
mmc0: sdhci: Int enab:  0x107f100b | Sig enab: 0x107f100b
mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000003
mmc0: sdhci: Caps:      0x07eb0000 | Caps_1: 0x0000a000
mmc0: sdhci: Cmd:       0x0000353a | Max curr: 0x00ffffff
mmc0: sdhci: Resp[0]:   0x00001000 | Resp[1]: 0x00000000
mmc0: sdhci: Resp[2]:   0x00000000 | Resp[3]: 0x00000000
mmc0: sdhci: Host ctl2: 0x00000000
mmc0: sdhci: ADMA Err:  0x00000007 | ADMA Ptr: 0x4c041200
mmc0: sdhci-esdhc-imx: ========= ESDHC IMX DEBUG STATUS DUMP
=========
mmc0: sdhci-esdhc-imx: cmd debug status:  0x2100
mmc0: sdhci-esdhc-imx: data debug status:  0x2200
mmc0: sdhci-esdhc-imx: trans debug status:  0x2300
mmc0: sdhci-esdhc-imx: dma debug status:  0x2402
mmc0: sdhci-esdhc-imx: adma debug status:  0x25b4
mmc0: sdhci-esdhc-imx: fifo debug status:  0x2610
mmc0: sdhci-esdhc-imx: async fifo debug status: 0x2751
mmc0: sdhci: ============================================ wilc1000_sdio mmc0:0001:1: wilc_sdio_cmd53..failed, err(-110) wilc1000_sdio mmc0:0001:1: Failed cmd53 [0], bytes read...

Quoted 7 lines

Quoted 7 lines

I tried to reduce the clock speed to 20MHz in the devicetree with max-frequency = <20000000>;
but the problem then also occurs.

Is this a possible bug?

Hi Ajay,
Thanks for the answer.

Quoted 7 lines

Quoted 4 lines

The bus error seems to be specific to the host during the SDIO transfer.
How long does it take to reproduce it? Does the issue also happen without "-P 16" iPerf3 option?

It takes about 10s (something a bit longer) till I got this kernel error
messages and it doesn't matter if I use it with "-P 16" or without.

I did not observe the issue with my setup(SAMA5D4 XPLAINED + WILC1000 SDIO) when tested iPerf for a longer duration(~1000sec). I suspect the issue could be related to the SDHCI host controller.
Try to debug the host controller side for the possible cause of timeout.

Quoted 6 lines

Quoted 3 lines

It seems the timeout happens because the card fails to respond to SDIO command 53, right ?

Yes, the timeout could be for any reason like either the CMD53 has not reached to chip or response not received correctly at host end.

The problem happens seconds or tens of seconds into the test, so there must've been CMD53 which reached the card before the problem occurred, and there must have been a lot of those CMD53 before the problem happened too, since CMD53 seems to be some data transfer CMD ?

Quoted 8 lines

Quoted 3 lines

Is there some error logging/tracing functionality in the WILC1000 firmware which can provide further information why the card did not respond ?

WILC1000 SD module has UART serial debug port for firmware logs but I don't think it would be useful here because it needs to be debug/probe at SDIO bus level.

Is there some other kind of logging which can tell us more details on where to look for this problem ?

Maybe we can try monitoring the SDIO traffic with ftrace ?

Any other options, short of taking the hardware apart ?

Quoted 11 lines

Quoted 5 lines

Could it be the card suffered some sort of FIFO overflow ? The MX6Q is a
bit more performant than the CA7 (I think?) SAMA5D4, so maybe that plays
some role ?

As I understand, the issue is observed with basic iPerf testing(less throughput) so not sure if the host performance will have such an impact. IIRC few of the customers are using the same host(i.MX6) though I am not sure if it's over SPI or SDIO bus. Till now, I have not come across such limitations with the specific host.

Quoted 5 lines

That iperf -P 16 hammers the chip with a lot of short packets, the problem does not occur during iperf3 -P 1 run or UDP iperf3 run (that's the one with low traffic). Here the interface is saturated, that's why I speculate some sort of FIFO overrun is happening.

But earlier it was mentioned that the problem doesn't matter with or without "-P" option. So it seems the issue happens during stress test.

Quoted 4 lines

I have also noticed there are some wilc1000 downstream drivers with huge stacks of patches, but I never really figured out whether those are still relevant or whether the upstream wilc1000 driver is perfectly fine. I would like to believe it is the later, is it ?

Yes, wilc1000 mainline driver has all the required changes except few minor bug fixes and new features but none related to this scenario.