eal interrupt: fd error conditions not handled

http://bugs.dpdk.org/show_bug.cgi?id=1873

            Bug ID: 1873
           Summary: eal interrupt: fd error conditions not handled
           Product: DPDK
           Version: 26.03
          Hardware: All
                OS: All
            Status: UNCONFIRMED
          Severity: normal
          Priority: Normal
         Component: core
          Assignee: [email protected]
          Reporter: [email protected]

Target Milestone: ---

The DPDK interrupt handling thread enters a busy-loop when epoll error conditions (EPOLLERR, EPOLLHUP, EPOLLRDHUP) occur on interrupt file descriptors. This means that the dpdk-intr thread will run at 100% cpu consumption.

When an interrupt file descriptor enters an error state or experiences a hangup, the interrupt handling code in eal_intr_process_interrupts() does not properly detect or handle these conditions. The epoll events EPOLLERR, EPOLLHUP, and EPOLLRDHUP indicate critical issues with the file descriptor, but the current code attempts to read from the fd without first checking for these error conditions.

This results in:
1. The interrupt continuing to fire repeatedly
2. The error condition never being cleared
3. The interrupt thread entering a busy-loop, consuming 100% CPU on a core

This is generic issue that could happen to any interrupt read directly in the eal interrupt code or passed to a registered handler for external processing.

I have reproduced this issue in multiple ways such as unbinding a device, bonding a device, manually deleting the fd.

To give an example of a normal use case where this might occur:
- A DPDK application starts, eal init and there are some mlx devices probed.
- These devices are not used by DPDK, but later are used to form a linux bond.
- As part of LAG setup, the mlx5_core kernel driver removes the current devices and fd's.
- Removal of the interrupt fd triggers a mlx devx interrupt with EPOLLRDHUP in DPDK.
- This is passed to the mlx devx interrupt handler, which does a read and gets an EAGAIN
- As the condition is not cleared, the next epoll_wait returns immediately with the same event and so we loop

This issue can also occur with devices that are used in DPDK, but it would not be expected to unbind a mlx device from the mlx5_core kernel driver when in use.

Testpmd has some additional handling of RTE_ETH_EVENT_INTR_RMV which in some cases may break the busy-loop by detaching the device, but this will not work in every case and an application should not have to register for this event and rely on it triggering in order to resolve the issue.

Example to reproduce:

# dpdk-testpmd -l 8,9 -- -i
EAL: Detected CPU lcores: 56
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
Interactive-mode selected
testpmd: Flow tunnel offload support might be limited or unavailable on port 0
testpmd: Flow tunnel offload support might be limited or unavailable on port 1
testpmd: create a new mbuf pool <mb_pool_0>: n=155456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
testpmd: create a new mbuf pool <mb_pool_1>: n=155456, size=2176, socket=1
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 0)
Port 0: 04:3F:72:C2:07:B8
Configuring Port 1 (socket 0)
Port 1: 04:3F:72:C2:07:B9
Checking link statuses...
Done

testpmd>

# top -H -p $(pidof dpdk-testpmd)
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND  
 164848 root      20   0  128.5g 434112  75264 S   0.0   0.2   0:01.08
dpdk-testpmd                                   
 164849 root      20   0  128.5g 434112  75264 S   0.0   0.2   0:00.00
dpdk-intr                                      
 164850 root      20   0  128.5g 434112  75264 S   0.0   0.2   0:00.00
dpdk-mp-msg                                    
 164851 root      20   0  128.5g 434112  75264 S   0.0   0.2   0:00.00
dpdk-worker9                                   
 164852 root      20   0  128.5g 434112  75264 S   0.0   0.2   0:00.00
dpdk-telemet-v2      

testpmd> port stop 0

# top -H -p $(pidof dpdk-testpmd)
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND  
 164848 root      20   0  128.5g 434112  75264 S   0.0   0.2   0:01.08
dpdk-testpmd                                   
 164849 root      20   0  128.5g 434112  75264 S   0.0   0.2   0:00.00
dpdk-intr                                      
 164850 root      20   0  128.5g 434112  75264 S   0.0   0.2   0:00.00
dpdk-mp-msg                                    
 164851 root      20   0  128.5g 434112  75264 S   0.0   0.2   0:00.00
dpdk-worker9                                   
 164852 root      20   0  128.5g 434112  75264 S   0.0   0.2   0:00.00
dpdk-telemet-v2     

# echo "0000:06:00.0" > /sys/bus/pci/drivers/mlx5_core/unbind

# top -H -p $(pidof dpdk-testpmd)
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND  
 164849 root      20   0  128.5g 434112  75264 R  99.9   0.2   0:07.87
dpdk-intr                                      
 164848 root      20   0  128.5g 434112  75264 S   0.0   0.2   0:01.08
dpdk-testpmd                                   
 164850 root      20   0  128.5g 434112  75264 S   0.0   0.2   0:00.00
dpdk-mp-msg                                    
 164851 root      20   0  128.5g 434112  75264 S   0.0   0.2   0:00.00
dpdk-worker9                                   
 164852 root      20   0  128.5g 434112  75264 S   0.0   0.2   0:00.00
dpdk-telemet-v2   

# strace -p $(ps -T -p $(pidof dpdk-testpmd) | grep dpdk-intr | awk '{print
$2}')

epoll_wait(6, [{events=EPOLLIN|EPOLLRDHUP, data={u32=28, u64=28}}], 8, -1) = 1
read(28, 0x7f1f5c7fc2f0, 40) = -1 EAGAIN (Resource temporarily
unavailable)
epoll_wait(6, [{events=EPOLLIN|EPOLLRDHUP, data={u32=28, u64=28}}], 8, -1) = 1
read(28, 0x7f1f5c7fc2f0, 40) = -1 EAGAIN (Resource temporarily
unavailable)
epoll_wait(6, [{events=EPOLLIN|EPOLLRDHUP, data={u32=28, u64=28}}], 8, -1) = 1
read(28, 0x7f1f5c7fc2f0, 40) = -1 EAGAIN (Resource temporarily
unavailable)
epoll_wait(6, [{events=EPOLLIN|EPOLLRDHUP, data={u32=28, u64=28}}], 8, -1) = 1
read(28, 0x7f1f5c7fc2f0, 40) = -1 EAGAIN (Resource temporarily
unavailable)
epoll_wait(6, [{events=EPOLLIN|EPOLLRDHUP, data={u32=28, u64=28}}], 8, -1) = 1
read(28, 0x7f1f5c7fc2f0, 40) = -1 EAGAIN (Resource temporarily
unavailable)
epoll_wait(6, [{events=EPOLLIN|EPOLLRDHUP, data={u32=28, u64=28}}], 8, -1) = 1 <snip>

I will send shortly an eal patch that will check for (EPOLLERR, EPOLLHUP, EPOLLRDHUP) in the fd events and remove any interrupts that have these events, preventing a busy loop.

You are receiving this mail because:
You are the assignee for the bug.