Skip to contentSkip to conversation

New: kernel panic ip_route_input

Bugme-new Andrew Morton 4 participants 6 messages Mar 25, 2008 – Mar 26, 2008 1 message not in the archive

Subsystems (2)
Networking Network drivers
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]/

Andrew Morton

author
#1 lore raw

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 17 lines · On Tue, 25 Mar 2008 12:48:37 -0700 (PDT) [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=10324

           Summary: kernel panic ip_route_input
           Product: Networking
           Version: 2.5
     KernelVersion: 2.6.25-rc6-git6
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: IPV4
        AssignedTo: [email protected]
        ReportedBy: [email protected]
Quoted 2 lines

Latest working kernel version: 2.6.24.x
Earliest failing kernel version: 2.6.25-rc6

A post-2.6.24 regression.

Quoted 18 lines

Distribution: Custom
Hardware Environment: Dual CPU, e100 + e1000
Software Environment: Custom build. FIB_TRIE enabled
Problem Description: Kernel panic after few seconds running network on heavy traffic (around 350 Mbps).

Steps to reproduce: It is complicated configuration. I can provide whole image of system, around 128MB (it is semi-embedded), but it will need at least few traffic generators similar to mine. So it is difficult to reproduce. I can apply any patches required, but highly prefferable if they dont make server unreachable. I have panic and oops autoreboot sysctl options, nmi-watchdog and software watchdog enabled.
I think it is critical to fix this bug before releasing 2.6.25.

Here is latest message with panic:
CONFIG_DEBUG_RODATA, 4K stacks, Frame pointers, CONFIG_DEBUG_STACKOVERFLOW - enabled.

Quoted 7 lines

It reboots recently after bringing up networking. Sometimes even not reaching place where i am loading netconsole.

Always in ip_route_input. This is 2.6.25-rc6-git6, 2 panic's. They are easy to reproduce, but i cannot do bisect, cause this machine dont have power switch and in difficult to reach area.

I've snipped the log output - it was such a wordwrapped mess :(

Can you please resend it in a reply-to-all to this email, after ensuring that it won't be wordwrapped?

Thanks.

#2 lore raw

I upload whole netconsole file at
http://www.nuclearcat.com/files/ip_route_input_crash.txt
There is two crashes.

On Tue, 25 Mar 2008 13:03:02 -0700, Andrew Morton wrote

Quoted 54 lines

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 17 lines · On Tue, 25 Mar 2008 12:48:37 -0700 (PDT) [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=10324

           Summary: kernel panic ip_route_input
           Product: Networking
           Version: 2.5
     KernelVersion: 2.6.25-rc6-git6
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: IPV4
        AssignedTo: [email protected]
        ReportedBy: [email protected]
Quoted 2 lines

Latest working kernel version: 2.6.24.x
Earliest failing kernel version: 2.6.25-rc6

A post-2.6.24 regression.

Quoted 18 lines

Distribution: Custom
Hardware Environment: Dual CPU, e100 + e1000
Software Environment: Custom build. FIB_TRIE enabled
Problem Description: Kernel panic after few seconds running network on heavy traffic (around 350 Mbps).

Steps to reproduce: It is complicated configuration. I can provide whole image of system, around 128MB (it is semi-embedded), but it will need at least few traffic generators similar to mine. So it is difficult to reproduce. I can apply any patches required, but highly prefferable if they dont make server unreachable. I have panic and oops autoreboot sysctl options, nmi-watchdog and software watchdog enabled.
I think it is critical to fix this bug before releasing 2.6.25.

Here is latest message with panic:
CONFIG_DEBUG_RODATA, 4K stacks, Frame pointers, CONFIG_DEBUG_STACKOVERFLOW - enabled.

Quoted 7 lines

It reboots recently after bringing up networking. Sometimes even not reaching place where i am loading netconsole.

Always in ip_route_input. This is 2.6.25-rc6-git6, 2 panic's. They are easy to reproduce, but i cannot do bisect, cause this machine dont have power switch and in difficult to reach area.

Quoted 6 lines

I've snipped the log output - it was such a wordwrapped mess :(

Can you please resend it in a reply-to-all to this email, after ensuring that it won't be wordwrapped?

Thanks.

--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.

#3 lore raw
In reply to Andrew Morton

Changing from FIB_TRIE to FIB_HASH, and it is not panicing anymore. It seems recent patches to FIB_TRIE broke something (i have complicated setup with ifb, maybe it will help). If you can point me to any patch, i can reverse it and try again with TRIE.

On Tue, 25 Mar 2008 13:03:02 -0700, Andrew Morton wrote

Quoted 54 lines

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 17 lines · On Tue, 25 Mar 2008 12:48:37 -0700 (PDT) [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=10324

           Summary: kernel panic ip_route_input
           Product: Networking
           Version: 2.5
     KernelVersion: 2.6.25-rc6-git6
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: IPV4
        AssignedTo: [email protected]
        ReportedBy: [email protected]
Quoted 2 lines

Latest working kernel version: 2.6.24.x
Earliest failing kernel version: 2.6.25-rc6

A post-2.6.24 regression.

Quoted 18 lines

Distribution: Custom
Hardware Environment: Dual CPU, e100 + e1000
Software Environment: Custom build. FIB_TRIE enabled
Problem Description: Kernel panic after few seconds running network on heavy traffic (around 350 Mbps).

Steps to reproduce: It is complicated configuration. I can provide whole image of system, around 128MB (it is semi-embedded), but it will need at least few traffic generators similar to mine. So it is difficult to reproduce. I can apply any patches required, but highly prefferable if they dont make server unreachable. I have panic and oops autoreboot sysctl options, nmi-watchdog and software watchdog enabled.
I think it is critical to fix this bug before releasing 2.6.25.

Here is latest message with panic:
CONFIG_DEBUG_RODATA, 4K stacks, Frame pointers, CONFIG_DEBUG_STACKOVERFLOW - enabled.

Quoted 7 lines

It reboots recently after bringing up networking. Sometimes even not reaching place where i am loading netconsole.

Always in ip_route_input. This is 2.6.25-rc6-git6, 2 panic's. They are easy to reproduce, but i cannot do bisect, cause this machine dont have power switch and in difficult to reach area.

Quoted 6 lines

I've snipped the log output - it was such a wordwrapped mess :(

Can you please resend it in a reply-to-all to this email, after ensuring that it won't be wordwrapped?

Thanks.

--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.

maintainer
#4 lore raw

From: "Denys" <[email protected]>
Date: Wed, 26 Mar 2008 01:12:01 +0200

Quoted 4 lines

Changing from FIB_TRIE to FIB_HASH, and it is not panicing anymore. It seems recent patches to FIB_TRIE broke something (i have complicated setup with ifb, maybe it will help). If you can point me to any patch, i can reverse it and try again with TRIE.

Unfortunately there were 39 changes made to fib_trie during this time. So it's not like we can give you one patch to try reverting :-/

Really, the fastest way to track this down would be to do a bisect, focused in on net/ipv4/fib_trie.c which git bisect does allow you to do.

#5 lore raw
In reply to Denys
Quoted 4 lines · On Wed, 26 Mar 2008 01:12:01 +0200 "Denys" <[email protected]> wrote:

Changing from FIB_TRIE to FIB_HASH, and it is not panicing anymore. It seems recent patches to FIB_TRIE broke something (i have complicated setup with ifb, maybe it will help). If you can point me to any patch, i can reverse it and try again with TRIE.

Bisect?

#6 lore raw

I have to wait when power switch will be installed in that location then.

On Tue, 25 Mar 2008 16:49:09 -0700, Stephen Hemminger wrote

Quoted 4 lines
Quoted 1 line · On Wed, 26 Mar 2008 01:12:01 +0200 "Denys" <[email protected]> wrote:

Changing from FIB_TRIE to FIB_HASH, and it is not panicing anymore. It

seems

Quoted 1 line
Quoted 1 line

recent patches to FIB_TRIE broke something (i have complicated setup with

ifb,

Quoted 1 line
Quoted 1 line

maybe it will help). If you can point me to any patch, i can reverse it

and

Quoted 7 lines
Quoted 1 line

try again with TRIE.

Bisect?
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.