PDK:20.11.1: net/ena crash while fetching xstats

https://bugs.dpdk.org/show_bug.cgi?id=996

            Bug ID: 996
           Summary: PDK:20.11.1: net/ena crash while fetching xstats
           Product: DPDK
           Version: 20.11
          Hardware: x86
                OS: Linux
            Status: UNCONFIRMED
          Severity: major
          Priority: Normal
         Component: ethdev
          Assignee: [email protected]
          Reporter: [email protected]

Target Milestone: ---

I am facing one issue in net/ena driver while fetching extended stats (xstats). The DPDK seems to segfault with below backtrace.

DPDK Version: 20.11.1
ENA version: 2.2.1

Using host libthread_db library "/lib64/libthread_db.so.1".

Core was generated by `/opt/dpfs/usr/local/bin/brdagent'.

Program terminated with signal SIGSEGV, Segmentation fault.

#0 __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:232

232 VMOVU %VEC(0), (%rdi)

[Current thread is 1 (Thread 0x7fffed93a400 (LWP 5060))]

Thread 1 (Thread 0x7fffed93a400 (LWP 5060)):

#0 __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:232

#1 0x00007ffff3c246df in ena_com_handle_admin_completion () from ../lib64/../../lib64/libdpdk.so.20

#2 0x00007ffff3c1e7f5 in ena_interrupt_handler_rte () from ../lib64/../../lib64/libdpdk.so.20

#3 0x00007ffff3519902 in eal_intr_thread_main () from /../lib64/../../lib64/libdpdk.so.20

#4 0x00007ffff510714a in start_thread (arg=<optimized out>) at pthread_create.c:479

#5 0x00007ffff561ff23 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Background:

This used to work fine with DPDK-19.11.3 , that means there was no crash observed with the 19.11.3 DPDK version, but now after upgrading to DPDK 20.11.1, DPDK is crashing with the above trace. It looks to me as a DPDK issue.

I could see multiple fixes/patches in the net/ena area, but not able to identify which patch would exactly fix this issue.

For example:
http://git.dpdk.org/dpdk/diff/?h=releases&id=aab58857330bb4bd03f6699bf1ee716f72993774
https://inbox.dpdk.org/dev/[email protected]/T/#me99457c706718bb236d1fd8006ee7a0319ce76fc

Could you please help here and let me know what patch could fix this issue.

You are receiving this mail because:
You are the assignee for the bug.

https://bugs.dpdk.org/show_bug.cgi?id=996

Michal Krawczyk ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

             Status|UNCONFIRMED                 |RESOLVED
         Resolution|---                         |WONTFIX
                 CC|                            |[email protected]

--- Comment #1 from Michal Krawczyk ([email protected]) --- Hi Amiya,

this is associated with lack of the proper MP support of the older ENA PMD (it supports MP properly starting from DPDK v22.03).

The xstats API uses ENA admin queue in DPDK v20.11, and admin queue cannot be used safely from the secondary process.

Those commits were improving the MP support starting from v20.11 to v22.03:

net/ena: make ethdev references multi-process safe aab58857330bb4bd03f6699bf1ee716f72993774

net/ena: disable ops not supported by secondary process 39ecdd3dfa15d5ac591ce8d77d362480bff32355

net/ena: proxy AQ calls to primary process (this is the critical patch) e3595539e0e03f0dbb81904f8edaaef0447a4f62

net/ena: enable stats for multi-process mode
3aa3fa851f58873457bdc5c387d0e5956f812322

net/ena/base: make IO memzone unique per port
850e1bb1c72b3d1163b2857ab7a02af11ba29c40

Thanks,
Michal

You are receiving this mail because:
You are the assignee for the bug.

https://bugs.dpdk.org/show_bug.cgi?id=996

Michal Krawczyk ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

         Resolution|WONTFIX                     |---
             Status|RESOLVED                    |UNCONFIRMED

--- Comment #3 from Michal Krawczyk ([email protected]) --- Hey Amiya,

sorry for the late reply, I was OOO for one week.

Thank you for providing us with more details.

If you aren't calling any API that needs to use the ENA admin queue from the secondary process, the situation you're seeing shouldn't happen.

I just executed simple application on DPDK v20.11.1 in MP mode - the main process is fetching the xstats, the secondary process is simply performing the packets forwarding. The application is not crashing for my case.

From what I understand, the crash happens, because:

1. The ENA admin queue is not using the shared memory
2. The secondary process sends the request and saves it in the secondary process memory
3. The primary process receives the interrupt and executes the completion handler
4. The completion handler cannot find the relevant request (as it's in the secondary process memory) and the app crashes.

Please double check if:

1. The xstats aren't being fetched from the secondary process
2. You aren't calling any of API below from the secondary process, which also uses the ENA admin queue:
- rte_eth_dev_set_mtu()
- rte_eth_dev_rss_reta_update()
- rte_eth_dev_rss_reta_query()

The point 1. is much more likely as you've described it's a regression in v20.11, and indeed - the xstats were extended after v19.11 release.

If none of the above is true, every other information that could potentially get us closer to the core of the issue may be helpful (we can't reproduce this on our side).

Thanks,
Michal

You are receiving this mail because:
You are the assignee for the bug.

https://bugs.dpdk.org/show_bug.cgi?id=996

[email protected] changed:

What |Removed |Added
----------------------------------------------------------------------------

         Resolution|---                         |FIXED
             Status|UNCONFIRMED                 |RESOLVED
                 CC|                            |[email protected]

--- Comment #5 from [email protected] ---
ENA PMD 2.6.0 and later (available since DPDK 22.03) can be used safely from the secondary process to retrieve xstats

You are receiving this mail because:
You are the assignee for the bug.