Skip to contentSkip to conversation

New: kernel hang occasionally while running the sample of xdpsock

Bug 208275 Stephen Hemminger 3 participants 6 messages Jun 22, 2020 – Jun 23, 2020 BPF

lore
Share & tools
lore
#0 lore raw

Begin forwarded message:

Date: Mon, 22 Jun 2020 10:13:52 +0000
From: [email protected]
To: [email protected]
Subject: [Bug 208275] New: kernel hang occasionally while running the sample of xdpsock

https://bugzilla.kernel.org/show_bug.cgi?id=208275

            Bug ID: 208275
           Summary: kernel hang occasionally while running the sample of
                    xdpsock
           Product: Networking
           Version: 2.5
    Kernel Version: 5.7.0
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: high
          Priority: P1
         Component: Other
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Distribution:
5.7.0-1.el7.centos.x86_64

Hardware Environment:
Dell Inc. PowerEdge R730/0WCJNT, BIOS 2.1.7 06/16/2016

Software Environment:

Problem Description:
kernel hang occasionally while running the sample of xdpsock

Steps to reproduce:

I want to test the rx performace of AF_XDP. I change the nic to 4 queues by cmd ethtool -L p6p1 combined 4, then I will create 1 socket for every queue.

for ((i=0; i<4; i++));
do
./xdpsock -r -z -i p6p1 -m -q $i &
done

I run the xdpsock in samples/bpf using the shell command above. And occasionally the kernel hang, so I have to power off and on.

Additonal information:

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#1 lore raw
Quoted 10 lines · Stephen Hemminger

Begin forwarded message:

Date: Mon, 22 Jun 2020 10:13:52 +0000
From: [email protected]
To: [email protected]
Subject: [Bug 208275] New: kernel hang occasionally while running the sample of xdpsock

Thanks for forwarding, Stephen.

I'll have a look!

Björn

Quoted 50 lines

https://bugzilla.kernel.org/show_bug.cgi?id=208275

            Bug ID: 208275
           Summary: kernel hang occasionally while running the sample of
                    xdpsock
           Product: Networking
           Version: 2.5
    Kernel Version: 5.7.0
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: high
          Priority: P1
         Component: Other
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Distribution:
5.7.0-1.el7.centos.x86_64

Hardware Environment:
Dell Inc. PowerEdge R730/0WCJNT, BIOS 2.1.7 06/16/2016

Software Environment:

Problem Description:
kernel hang occasionally while running the sample of xdpsock

Steps to reproduce:

I want to test the rx performace of AF_XDP. I change the nic to 4 queues by cmd ethtool -L p6p1 combined 4, then I will create 1 socket for every queue.

for ((i=0; i<4; i++));
do
./xdpsock -r -z -i p6p1 -m -q $i &
done

I run the xdpsock in samples/bpf using the shell command above. And occasionally the kernel hang, so I have to power off and on.

Additonal information:

--
You are receiving this mail because:
You are the assignee for the bug.

#2 lore raw
Quoted 14 lines · Bj?rn T?pel
Quoted 10 lines · Stephen Hemminger

Begin forwarded message:

Date: Mon, 22 Jun 2020 10:13:52 +0000
From: bugzilla-daemon at bugzilla.kernel.org
To: stephen at networkplumber.org
Subject: [Bug 208275] New: kernel hang occasionally while running the sample of xdpsock

Quoted 4 lines

Thanks for forwarding, Stephen.

I'll have a look!

Intel ixgbe splat. Adding intel-wired-lan to To:.

#3 lore raw

Hi, Bjorn, Thank your response.
Could you describe it more clearly? I can not get it exactly. Thx.

Bj?rn T?pel <[email protected]> ?2020?6?23??? ??1:50???

Quoted 16 lines · Bj?rn T?pel
Quoted 6 lines
Quoted 3 lines · Stephen Hemminger
Quoted 6 lines
Quoted 6 lines

Begin forwarded message:

Date: Mon, 22 Jun 2020 10:13:52 +0000
From: bugzilla-daemon at bugzilla.kernel.org
To: stephen at networkplumber.org
Subject: [Bug 208275] New: kernel hang occasionally while running the

sample of xdpsock

Quoted 2 lines
Quoted 1 line
Quoted 7 lines
Quoted 4 lines

Thanks for forwarding, Stephen.

I'll have a look!

Intel ixgbe splat. Adding intel-wired-lan to To:.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20200623/75f6ccfb/attachment.html>

#4 lore raw
Quoted 5 lines · Yahui Chen

Hi, Bjorn, Thank your response.
Could you describe it more clearly? I can not get it exactly. Thx.

When XDP is enabled, the ixgbe NIC does a (somewhat heavy)
reconfiguration. During the reconfiguration, for some reason, the rx_buffer->page is NULL in the following call chain:
ixgbe_down()->ixgbe_clean_all_rx_rings()->ixgbe_clean_rx_ring()->__page_frag_cache_drain()

This results in that when __page_frag_cache_drain() want to touch the reference counter, you get a NULL pointer dereference.

[277994.329145] BUG: kernel NULL pointer dereference, address: 0000000000000034
...
[277994.329428] RIP: 0010:__page_frag_cache_drain+0x5/0x40
[277994.329463] Code: d2 ff ff 31 f6 84 c0 74 04 0f b6 73 51 48 89 df

e8 70 ff ff ff eb dc 48 83 eb 01 eb d0 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 <f0> 29 77 34 74 01 c3 48 8b 07 55 48 89 e5 a9 00 00 01 00 74 0f 0f

2a:* f0 29 77 34 lock sub %esi,0x34(%rdi) <--
trapping instruction

I tried to reproduce the issue, but without success so far. I'll keep looking for the bug. Hopefully someone from Intel with better insight into ixgbe can help!

Bj?rn

#5 lore raw
In reply to Stephen Hemminger

Hi, Bjorn, Thx for your response.
I can not get it exactly. Could you describe the details more clearly? Thx.

Quoted 60 lines · Stephen Hemminger <[email protected]> 于2020年6月23日周二 上午12:08写道:

Begin forwarded message:

Date: Mon, 22 Jun 2020 10:13:52 +0000
From: [email protected]
To: [email protected]
Subject: [Bug 208275] New: kernel hang occasionally while running the sample of xdpsock

https://bugzilla.kernel.org/show_bug.cgi?id=208275

            Bug ID: 208275
           Summary: kernel hang occasionally while running the sample of
                    xdpsock
           Product: Networking
           Version: 2.5
    Kernel Version: 5.7.0
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: high
          Priority: P1
         Component: Other
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Distribution:
5.7.0-1.el7.centos.x86_64

Hardware Environment:
Dell Inc. PowerEdge R730/0WCJNT, BIOS 2.1.7 06/16/2016

Software Environment:

Problem Description:
kernel hang occasionally while running the sample of xdpsock

Steps to reproduce:

I want to test the rx performace of AF_XDP. I change the nic to 4 queues by cmd ethtool -L p6p1 combined 4, then I will create 1 socket for every queue.

for ((i=0; i<4; i++));
do
./xdpsock -r -z -i p6p1 -m -q $i &
done

I run the xdpsock in samples/bpf using the shell command above. And occasionally the kernel hang, so I have to power off and on.

Additonal information:

--
You are receiving this mail because:
You are the assignee for the bug.