possible bug x86 2.4.2 SMP in IP receive stack
With dual x86 processors running 2.4.2, if I blast a UDP stream at the machine using netperf, I can easily cause the kernel to panic with the message below.
Feb 23 12:42:30 rcc2 kernel: Warning: kfree_skb passed an skb still on a list (from c01f58dc).
I'm going to pop out one processor on the receiver and see if that makes the problem go away.
Note that this is using a Myrinet network that is able to get more than 1.5Gigabit/sec UDP transfers on single-processor x86 2.4.0 linux. Perhaps this is reproducible with good GigE cards with jumbo MTU turned on.
I'm also upping the socket limits
echo "1048576" > /proc/sys/net/core/rmem_max echo "1048576" > /proc/sys/net/core/wmem_max echo "1048576" > /proc/sys/net/core/wmem_default echo "1048576" > /proc/sys/net/core/rmem_default echo "1048576" > /proc/sys/net/core/optmem_max
Feb 23 12:42:30 rcc2 kernel: Warning: kfree_skb passed an skb still on a list (from c01f58dc).
Looking up the "from c01f58dc" in the ksyms shows that
ip_rcv
is the caller.
c01f3d38 ip_route_input_Rsmp_0a99f032c01f44f8 ip_route_output_key_Rsmp_4ce6fe49c01f5170 inet_add_protocol_Rsmp_a27098bdc01f51f0 inet_del_protocol_Rsmp_0c8ae503c01f5538 ip_rcv_Rsmp_587335e5c01f58dc ERROR LOCATION (kfree_skb passed an skb still on a list (from c01f58dc)) c01f61dc ip_defrag_Rsmp_5532f3a2c01f6b34 ip_options_compile_Rsmp_b8621391c01f70ec ip_options_undo_Rsmp_9721f12fc01f8650 ip_fragment_Rsmp_41bc67d3c01f8bb0 ip_send_check_Rsmp_a37b7441c01f8bf8 ip_finish_output_Rsmp_5b565e28
On a different machine I have seen this.
Feb 23 12:32:20 rcc kernel: KERNEL: assertion (del_timer(&qp->timer) == 0) failed at ip_fragment.c(163):ip_frag_destroy
CONFIG_X86=yCONFIG_ISA=yCONFIG_UID16=yCONFIG_EXPERIMENTAL=yCONFIG_MODULES=yCONFIG_MODVERSIONS=yCONFIG_KMOD=yCONFIG_MPENTIUMIII=yCONFIG_X86_WP_WORKS_OK=yCONFIG_X86_INVLPG=yCONFIG_X86_CMPXCHG=yCONFIG_X86_BSWAP=yCONFIG_X86_POPAD_OK=yCONFIG_X86_TSC=yCONFIG_X86_GOOD_APIC=yCONFIG_X86_PGE=yCONFIG_X86_USE_PPRO_CHECKSUM=yCONFIG_NOHIGHMEM=yCONFIG_SMP=yCONFIG_HAVE_DEC_LOCK=yCONFIG_NET=yCONFIG_X86_IO_APIC=yCONFIG_X86_LOCAL_APIC=yCONFIG_PCI=yCONFIG_PCI_GOANY=yCONFIG_PCI_BIOS=yCONFIG_PCI_DIRECT=yCONFIG_PCI_NAMES=yCONFIG_HOTPLUG=yCONFIG_SYSVIPC=yCONFIG_SYSCTL=yCONFIG_KCORE_ELF=yCONFIG_BINFMT_AOUT=yCONFIG_BINFMT_ELF=yCONFIG_BINFMT_MISC=yCONFIG_PM=yCONFIG_ACPI=yCONFIG_PNP=yCONFIG_ISAPNP=yCONFIG_BLK_DEV_FD=yCONFIG_PACKET=yCONFIG_PACKET_MMAP=yCONFIG_UNIX=yCONFIG_INET=yCONFIG_IDE=yCONFIG_BLK_DEV_IDE=yCONFIG_BLK_DEV_IDEDISK=yCONFIG_BLK_DEV_IDECD=yCONFIG_BLK_DEV_CMD640=yCONFIG_BLK_DEV_RZ1000=yCONFIG_BLK_DEV_IDEPCI=yCONFIG_IDEPCI_SHARE_IRQ=yCONFIG_BLK_DEV_IDE_MODES=yCONFIG_NETDEVICES=yCONFIG_NET_ETHERNET=yCONFIG_NET_PCI=yCONFIG_EEPRO100=yCONFIG_VT=yCONFIG_VT_CONSOLE=yCONFIG_SERIAL=yCONFIG_UNIX98_PTYS=yCONFIG_MOUSE=yCONFIG_PSMOUSE=yCONFIG_DRM=yCONFIG_DRM_TDFX=yCONFIG_AUTOFS_FS=yCONFIG_AUTOFS4_FS=yCONFIG_FAT_FS=yCONFIG_MSDOS_FS=yCONFIG_ISO9660_FS=yCONFIG_PROC_FS=yCONFIG_DEVPTS_FS=yCONFIG_EXT2_FS=yCONFIG_NFS_FS=yCONFIG_NFS_V3=yCONFIG_NFSD=yCONFIG_NFSD_V3=yCONFIG_SUNRPC=yCONFIG_LOCKD=yCONFIG_LOCKD_V4=yCONFIG_MSDOS_PARTITION=yCONFIG_NLS=yCONFIG_VGA_CONSOLE=y
------------------------------------------------------------------
Bob Felderman (626) 821-5555 Director of Software Development (626) 821-5316 fax Myricom Inc. [email protected] 325 N. Santa Anita Ave. http://www.myri.com
Arcadia, CA 91006
------------------------------------------------------------------
Sounds like a bug wrt. SKB allocations in the Myrinet driver.
You're the author of most of that code, so I'm sure you're the best one to audit it :-)
Later,
David S. Miller
[email protected]