Skip to contentSkip to conversation

Unable to handle kernel NULL pointer dereference at virtual address 00000000

DISCUSSION Ray Van Dolson 2 participants 5 messages Dec 15, 2004 – Jan 3, 2005

lore
Share & tools
lore
#0 lore raw

Hoping someone can shed some light on this, or help me out in narrowing the problem down.

Our servers exhibit these error messages (at bottom) off and on and eventually appears to lead to system instability. I guess the three possibilities are that this is either a) a hardware problem, b) a problem with poptop, pppd or the mppe modules, or c) a problem with the kernel

This happens on every single one of our DL140's, so if it's a hardware problem, then it is widespread on this platform.

If it's a problem with poptop, pppd or mppe modules I'd at least like to be able to give some evidence to that effect...

If it's c, I'm hoping someone here can give some suggestions on how to resolve this. :-) Please let me know if I can provide further information. I have a box on which this happens fairly regularly, so I can reproduce it pretty easily.

Thanks in advance!

System Specs:
HP DL140
Dual 2.4GHz P4 Xeon w/ Hyperthreading
Fedora Core 2
2.6.9 kernel (static compile, downloaded from ftp.kernel.org with the
following patches):
- http://marc.theaimsgroup.com/?l=linux-kernel&m=109926628920398
This fixes an oops I was getting in mm/prio_tree.c
- http://mdomsch.bkbits.net:8080/linux-2.6-mppe
This provides MPPE support for clients connecting via PoPToP 1.2.1.

Intel E100 Pro Dual Port NIC (Using built-in e100 driver). Onboard Broadcom BCM5700 Gigabit NIC's are *not* in use (no driver support
compiled in)

Software:
Poptop 1.2.1
PPP 2.4.3

Kernel config:

CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_EXPERIMENTAL=y
CONFIG_CLEAN_COMPILE=y
CONFIG_LOCALVERSION=""
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_SYSCTL=y
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_LOG_BUF_SHIFT=17
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_EXTRA_PASS=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_SHMEM=y
CONFIG_X86_PC=y
CONFIG_MPENTIUM4=y
CONFIG_X86_GENERIC=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=7
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_SMP=y
CONFIG_NR_CPUS=8
CONFIG_SCHED_SMT=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_TSC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
CONFIG_X86_MCE_P4THERMAL=y
CONFIG_MICROCODE=y
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
CONFIG_HIGHMEM4G=y
CONFIG_HIGHMEM=y
CONFIG_HIGHPTE=y
CONFIG_MTRR=y
CONFIG_IRQBALANCE=y
CONFIG_HAVE_DEC_LOCK=y
CONFIG_ACPI=y
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_INTERPRETER=y
CONFIG_ACPI_AC=y
CONFIG_ACPI_BATTERY=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_FAN=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_ASUS=y
CONFIG_ACPI_TOSHIBA=y
CONFIG_ACPI_BLACKLIST_YEAR=2001
CONFIG_ACPI_BUS=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_PCI=y
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
CONFIG_PCI=y
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_LEGACY_PROC=y
CONFIG_PCI_NAMES=y
CONFIG_ISA=y
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_MISC=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_PNP=y
CONFIG_ISAPNP=y
CONFIG_BLK_DEV_FD=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_CRYPTOLOOP=y
CONFIG_BLK_DEV_NBD=y
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_SIZE=16384
CONFIG_BLK_DEV_INITRD=y
CONFIG_LBD=y
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
CONFIG_BLK_DEV_IDECD=y
CONFIG_BLK_DEV_IDESCSI=y
CONFIG_IDE_GENERIC=y
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
CONFIG_BLK_DEV_GENERIC=y
CONFIG_BLK_DEV_IDEDMA_PCI=y
CONFIG_IDEDMA_PCI_AUTO=y
CONFIG_BLK_DEV_SVWKS=y
CONFIG_BLK_DEV_IDEDMA=y
CONFIG_IDEDMA_AUTO=y
CONFIG_SCSI=y
CONFIG_BLK_DEV_SD=y
CONFIG_BLK_DEV_SR=y
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=y
CONFIG_SCSI_SPI_ATTRS=y
CONFIG_SCSI_FC_ATTRS=y
CONFIG_SCSI_QLA2XXX=y
CONFIG_I2O=y
CONFIG_I2O_CONFIG=y
CONFIG_I2O_BLOCK=y
CONFIG_I2O_SCSI=y
CONFIG_I2O_PROC=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_NETLINK_DEV=y
CONFIG_UNIX=y
CONFIG_NET_KEY=y
CONFIG_INET=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_FWMARK=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_NET_IPIP=y
CONFIG_NET_IPGRE=y
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=y
CONFIG_INET_ESP=y
CONFIG_INET_IPCOMP=y
CONFIG_INET_TUNNEL=y
CONFIG_NETFILTER=y
CONFIG_IP_NF_CONNTRACK=y
CONFIG_IP_NF_CT_ACCT=y
CONFIG_IP_NF_CT_PROTO_SCTP=y
CONFIG_IP_NF_FTP=y
CONFIG_IP_NF_IRC=y
CONFIG_IP_NF_TFTP=y
CONFIG_IP_NF_AMANDA=y
CONFIG_IP_NF_QUEUE=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_MATCH_LIMIT=y
CONFIG_IP_NF_MATCH_IPRANGE=y
CONFIG_IP_NF_MATCH_MAC=y
CONFIG_IP_NF_MATCH_PKTTYPE=y
CONFIG_IP_NF_MATCH_MARK=y
CONFIG_IP_NF_MATCH_MULTIPORT=y
CONFIG_IP_NF_MATCH_TOS=y
CONFIG_IP_NF_MATCH_RECENT=y
CONFIG_IP_NF_MATCH_ECN=y
CONFIG_IP_NF_MATCH_DSCP=y
CONFIG_IP_NF_MATCH_AH_ESP=y
CONFIG_IP_NF_MATCH_LENGTH=y
CONFIG_IP_NF_MATCH_TTL=y
CONFIG_IP_NF_MATCH_TCPMSS=y
CONFIG_IP_NF_MATCH_HELPER=y
CONFIG_IP_NF_MATCH_STATE=y
CONFIG_IP_NF_MATCH_CONNTRACK=y
CONFIG_IP_NF_MATCH_OWNER=y
CONFIG_IP_NF_MATCH_ADDRTYPE=y
CONFIG_IP_NF_MATCH_REALM=y
CONFIG_IP_NF_MATCH_SCTP=y
CONFIG_IP_NF_MATCH_COMMENT=y
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_TARGET_REJECT=y
CONFIG_IP_NF_TARGET_LOG=y
CONFIG_IP_NF_TARGET_ULOG=y
CONFIG_IP_NF_TARGET_TCPMSS=y
CONFIG_IP_NF_NAT=y
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=y
CONFIG_IP_NF_TARGET_REDIRECT=y
CONFIG_IP_NF_TARGET_NETMAP=y
CONFIG_IP_NF_TARGET_SAME=y
CONFIG_IP_NF_NAT_LOCAL=y
CONFIG_IP_NF_NAT_SNMP_BASIC=y
CONFIG_IP_NF_NAT_IRC=y
CONFIG_IP_NF_NAT_FTP=y
CONFIG_IP_NF_NAT_TFTP=y
CONFIG_IP_NF_NAT_AMANDA=y
CONFIG_IP_NF_MANGLE=y
CONFIG_IP_NF_TARGET_TOS=y
CONFIG_IP_NF_TARGET_ECN=y
CONFIG_IP_NF_TARGET_DSCP=y
CONFIG_IP_NF_TARGET_MARK=y
CONFIG_IP_NF_TARGET_CLASSIFY=y
CONFIG_IP_NF_RAW=y
CONFIG_IP_NF_TARGET_NOTRACK=y
CONFIG_IP_NF_ARPTABLES=y
CONFIG_IP_NF_ARPFILTER=y
CONFIG_IP_NF_ARP_MANGLE=y
CONFIG_XFRM=y
CONFIG_XFRM_USER=y
CONFIG_NET_DIVERT=y
CONFIG_NET_SCHED=y
CONFIG_NET_SCH_CLK_JIFFIES=y
CONFIG_NET_SCH_CBQ=y
CONFIG_NET_SCH_HTB=y
CONFIG_NET_SCH_HFSC=y
CONFIG_NET_SCH_PRIO=y
CONFIG_NET_SCH_RED=y
CONFIG_NET_SCH_SFQ=y
CONFIG_NET_SCH_TEQL=y
CONFIG_NET_SCH_TBF=y
CONFIG_NET_SCH_GRED=y
CONFIG_NET_SCH_DSMARK=y
CONFIG_NET_SCH_NETEM=y
CONFIG_NET_SCH_INGRESS=y
CONFIG_NET_QOS=y
CONFIG_NET_ESTIMATOR=y
CONFIG_NET_CLS=y
CONFIG_NET_CLS_TCINDEX=y
CONFIG_NET_CLS_ROUTE4=y
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=y
CONFIG_NET_CLS_U32=y
CONFIG_CLS_U32_PERF=y
CONFIG_NET_CLS_IND=y
CONFIG_NET_CLS_RSVP=y
CONFIG_NET_CLS_RSVP6=y
CONFIG_NET_CLS_POLICE=y
CONFIG_NETPOLL=y
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_NETDEVICES=y
CONFIG_DUMMY=y
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
CONFIG_NET_PCI=y
CONFIG_E100=y
CONFIG_PPP=y
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=y
CONFIG_PPP_SYNC_TTY=y
CONFIG_PPP_DEFLATE=y
CONFIG_PPP_BSDCOMP=y
CONFIG_PPP_MPPE=y
CONFIG_NETCONSOLE=y
CONFIG_INPUT=y
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_SOUND_GAMEPORT=y
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_SERIAL=y
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=y
CONFIG_INPUT_UINPUT=y
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_8250_NR_UARTS=1024
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_DETECT_IRQ=y
CONFIG_SERIAL_8250_MULTIPORT=y
CONFIG_SERIAL_8250_RSA=y
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_IPMI_HANDLER=y
CONFIG_IPMI_DEVICE_INTERFACE=y
CONFIG_IPMI_SI=y
CONFIG_IPMI_WATCHDOG=y
CONFIG_IPMI_POWEROFF=y
CONFIG_HW_RANDOM=y
CONFIG_NVRAM=y
CONFIG_RTC=y
CONFIG_AGP=y
CONFIG_AGP_ATI=y
CONFIG_AGP_INTEL=y
CONFIG_HANGCHECK_TIMER=y
CONFIG_I2C=y
CONFIG_I2C_CHARDEV=y
CONFIG_I2C_ALGOBIT=y
CONFIG_I2C_ALGOPCF=y
CONFIG_I2C_ALGOPCA=y
CONFIG_I2C_I801=y
CONFIG_I2C_I810=y
CONFIG_I2C_ISA=y
CONFIG_I2C_PIIX4=y
CONFIG_I2C_SENSOR=y
CONFIG_SENSORS_ADM1021=y
CONFIG_SENSORS_ADM1025=y
CONFIG_SENSORS_ADM1031=y
CONFIG_SENSORS_ASB100=y
CONFIG_SENSORS_DS1621=y
CONFIG_SENSORS_FSCHER=y
CONFIG_SENSORS_GL518SM=y
CONFIG_SENSORS_IT87=y
CONFIG_SENSORS_LM75=y
CONFIG_SENSORS_LM77=y
CONFIG_SENSORS_LM78=y
CONFIG_SENSORS_LM80=y
CONFIG_SENSORS_LM83=y
CONFIG_SENSORS_LM85=y
CONFIG_SENSORS_LM90=y
CONFIG_SENSORS_MAX1619=y
CONFIG_SENSORS_SMSC47M1=y
CONFIG_SENSORS_VIA686A=y
CONFIG_SENSORS_W83781D=y
CONFIG_SENSORS_W83L785TS=y
CONFIG_SENSORS_W83627HF=y
CONFIG_SENSORS_EEPROM=y
CONFIG_SENSORS_PCF8574=y
CONFIG_SENSORS_PCF8591=y
CONFIG_SENSORS_RTC8564=y
CONFIG_VIDEO_DEV=y
CONFIG_VIDEO_SELECT=y
CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_JBD=y
CONFIG_FS_MBCACHE=y
CONFIG_FS_POSIX_ACL=y
CONFIG_ROMFS_FS=y
CONFIG_AUTOFS_FS=y
CONFIG_AUTOFS4_FS=y
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_ZISOFS_FS=y
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_RAMFS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
CONFIG_NFS_V4=y
CONFIG_NFS_DIRECTIO=y
CONFIG_NFSD=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V4=y
CONFIG_NFSD_TCP=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=y
CONFIG_SUNRPC=y
CONFIG_SUNRPC_GSS=y
CONFIG_RPCSEC_GSS_KRB5=y
CONFIG_SMB_FS=y
CONFIG_CIFS=y
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
CONFIG_PARTITION_ADVANCED=y
CONFIG_OSF_PARTITION=y
CONFIG_MAC_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
CONFIG_SGI_PARTITION=y
CONFIG_SUN_PARTITION=y
CONFIG_EFI_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_CODEPAGE_737=y
CONFIG_NLS_CODEPAGE_775=y
CONFIG_NLS_CODEPAGE_850=y
CONFIG_NLS_CODEPAGE_852=y
CONFIG_NLS_CODEPAGE_855=y
CONFIG_NLS_CODEPAGE_857=y
CONFIG_NLS_CODEPAGE_860=y
CONFIG_NLS_CODEPAGE_861=y
CONFIG_NLS_CODEPAGE_862=y
CONFIG_NLS_CODEPAGE_863=y
CONFIG_NLS_CODEPAGE_864=y
CONFIG_NLS_CODEPAGE_865=y
CONFIG_NLS_CODEPAGE_866=y
CONFIG_NLS_CODEPAGE_869=y
CONFIG_NLS_CODEPAGE_936=y
CONFIG_NLS_CODEPAGE_950=y
CONFIG_NLS_CODEPAGE_932=y
CONFIG_NLS_CODEPAGE_949=y
CONFIG_NLS_CODEPAGE_874=y
CONFIG_NLS_ISO8859_8=y
CONFIG_NLS_CODEPAGE_1250=y
CONFIG_NLS_CODEPAGE_1251=y
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=y
CONFIG_NLS_ISO8859_2=y
CONFIG_NLS_ISO8859_3=y
CONFIG_NLS_ISO8859_4=y
CONFIG_NLS_ISO8859_5=y
CONFIG_NLS_ISO8859_6=y
CONFIG_NLS_ISO8859_7=y
CONFIG_NLS_ISO8859_9=y
CONFIG_NLS_ISO8859_13=y
CONFIG_NLS_ISO8859_14=y
CONFIG_NLS_ISO8859_15=y
CONFIG_NLS_KOI8_R=y
CONFIG_NLS_KOI8_U=y
CONFIG_NLS_UTF8=y
CONFIG_DEBUG_KERNEL=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_EARLY_PRINTK=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_DEBUG_STACK_USAGE=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_CRYPTO=y
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_NULL=y
CONFIG_CRYPTO_MD4=y
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=y
CONFIG_CRYPTO_SHA512=y
CONFIG_CRYPTO_WP512=y
CONFIG_CRYPTO_DES=y
CONFIG_CRYPTO_BLOWFISH=y
CONFIG_CRYPTO_TWOFISH=y
CONFIG_CRYPTO_SERPENT=y
CONFIG_CRYPTO_AES_586=y
CONFIG_CRYPTO_CAST5=y
CONFIG_CRYPTO_CAST6=y
CONFIG_CRYPTO_TEA=y
CONFIG_CRYPTO_ARC4=y
CONFIG_CRYPTO_KHAZAD=y
CONFIG_CRYPTO_DEFLATE=y
CONFIG_CRYPTO_MICHAEL_MIC=y
CONFIG_CRYPTO_CRC32C=y
CONFIG_CRC_CCITT=y
CONFIG_CRC32=y
CONFIG_LIBCRC32C=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_X86_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y
CONFIG_PC=y

lspci output:

00:00.0 Host bridge: ServerWorks GCNB-LE Host Bridge (rev 32)
00:00.1 Host bridge: ServerWorks GCNB-LE Host Bridge
00:03.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27) 00:0f.0 ISA bridge: ServerWorks CSB6 South Bridge (rev a0)
00:0f.1 IDE interface: ServerWorks CSB6 RAID/IDE Controller (rev a0)
00:0f.2 USB Controller: ServerWorks CSB6 OHCI USB Controller (rev 05)
00:0f.3 Host bridge: ServerWorks GCLE-2 Host Bridge
00:10.0 Host bridge: ServerWorks CIOB-E I/O Bridge with Gigabit Ethernet (rev 12) 00:10.2 Host bridge: ServerWorks CIOB-E I/O Bridge with Gigabit Ethernet (rev 12) 01:06.0 PCI bridge: Digital Equipment Corporation DECchip 21152 (rev 03) 02:04.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 05) 02:05.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 05) 03:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 02) 03:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 02)

ksymoops output of problem:

ksymoops 2.4.9 on i686 2.6.9. Options used
-V (default)
-K (specified)
-l /proc/modules (default)
-O (specified)
-m /boot/System.map (specified)

No ksyms, skipping lsmod

Unable to handle kernel NULL pointer dereference<7>divert: not allocating divert_blk for non-ethernet device ppp440
00000000
*pde = 00000000
Oops: 0000 [#1]
CPU:    2
EIP:    0060:[<00000000>]    Not tainted VLI

Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010286 (2.6.9)
eax: ed13b000 ebx: d1d0a000 ecx: c029e9de edx: f795ef40
esi: d1d0a000 edi: 00000000 ebp: e2f30080 esp: d2b0dea0
ds: 007b es: 007b ss: 0068

Stack: c02a205a ed13b000 00000000 c02a122c d1d0a000 13208a2e c040956f d1d0a000 
       d1d0a00c e2f30080 00000000 c029cda9 d1d0a000 e2f30080 00000000 c01552cd 
       e2f30080 00000010 00000004 00000004 c0166aa0 e2f30080 00000000 00000000 
Call Trace:
 [<c02a205a>] pty_chars_in_buffer+0x2c/0x49
 [<c02a122c>] normal_poll+0xed/0x150
 [<c040956f>] schedule_timeout+0x75/0xbf
 [<c029cda9>] tty_poll+0xa0/0xb0
 [<c01552cd>] fget+0x49/0x5e
 [<c0166aa0>] do_select+0x269/0x2c6
 [<c0166691>] __pollwait+0x0/0xc7
 [<c0166dd5>] sys_select+0x2b3/0x4c6
 [<c0105971>] sysenter_past_esp+0x52/0x71
Code:  Bad EIP value.
Quoted 9 lines
Quoted 9 lines

EIP; 00000000 Before first symbol

eax; ed13b000 <pg0+2ca80000/3f943400>
ebx; d1d0a000 <pg0+1164f000/3f943400>
ecx; c029e9de <n_tty_chars_in_buffer+0/78>
edx; f795ef40 <pg0+372a3f40/3f943400>
esi; d1d0a000 <pg0+1164f000/3f943400>
ebp; e2f30080 <pg0+22875080/3f943400>
esp; d2b0dea0 <pg0+12452ea0/3f943400>

Trace; c02a205a <pty_chars_in_buffer+2c/49>
Trace; c02a122c <normal_poll+ed/150>
Trace; c040956f <schedule_timeout+75/bf>
Trace; c029cda9 <tty_poll+a0/b0>
Trace; c01552cd <fget+49/5e>
Trace; c0166aa0 <do_select+269/2c6>
Trace; c0166691 <__pollwait+0/c7>
Trace; c0166dd5 <sys_select+2b3/4c6>
Trace; c0105971 <sysenter_past_esp+52/71>

Yet another ksymoops output:

ksymoops 2.4.9 on i686 2.6.9. Options used
-V (default)
-K (specified)
-l /proc/modules (default)
-O (specified)
-m /boot/System.map (specified)

No ksyms, skipping lsmod

Unable to handle kernel NULL pointer dereference<7>divert: not allocating divert_blk for non-ethernet device ppp504
00000000
*pde = 00000000
Oops: 0000 [#2]
CPU:    2
EIP:    0060:[<00000000>]    Not tainted VLI

Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010286 (2.6.9)
eax: cc8ae000 ebx: d46ca000 ecx: c029e9de edx: f7801880
esi: d46ca000 edi: 00000000 ebp: e16a1d80 esp: d2907ea0
ds: 007b es: 007b ss: 0068

Stack: c02a205a cc8ae000 00000000 c02a122c d46ca000 1da86df5 c040956f d46ca000 
       d46ca00c e16a1d80 00000000 c029cda9 d46ca000 e16a1d80 00000000 c01552cd 
       e16a1d80 00000010 00000004 00000004 c0166aa0 e16a1d80 00000000 00000000 
Call Trace:
 [<c02a205a>] pty_chars_in_buffer+0x2c/0x49
 [<c02a122c>] normal_poll+0xed/0x150
 [<c040956f>] schedule_timeout+0x75/0xbf
 [<c029cda9>] tty_poll+0xa0/0xb0
 [<c01552cd>] fget+0x49/0x5e
 [<c0166aa0>] do_select+0x269/0x2c6
 [<c0166691>] __pollwait+0x0/0xc7
 [<c0166dd5>] sys_select+0x2b3/0x4c6
 [<c0105971>] sysenter_past_esp+0x52/0x71
Code:  Bad EIP value.
Quoted 9 lines
Quoted 9 lines

EIP; 00000000 Before first symbol

eax; cc8ae000 <pg0+c1f3000/3f943400>
ebx; d46ca000 <pg0+1400f000/3f943400>
ecx; c029e9de <n_tty_chars_in_buffer+0/78>
edx; f7801880 <pg0+37146880/3f943400>
esi; d46ca000 <pg0+1400f000/3f943400>
ebp; e16a1d80 <pg0+20fe6d80/3f943400>
esp; d2907ea0 <pg0+1224cea0/3f943400>

Trace; c02a205a <pty_chars_in_buffer+2c/49>
Trace; c02a122c <normal_poll+ed/150>
Trace; c040956f <schedule_timeout+75/bf>
Trace; c029cda9 <tty_poll+a0/b0>
Trace; c01552cd <fget+49/5e>
Trace; c0166aa0 <do_select+269/2c6>
Trace; c0166691 <__pollwait+0/c7>
Trace; c0166dd5 <sys_select+2b3/4c6>
Trace; c0105971 <sysenter_past_esp+52/71>

#1 lore raw
Quoted 41 lines · Ray Van Dolson

Hoping someone can shed some light on this, or help me out in narrowing the problem down.

Our servers exhibit these error messages (at bottom) off and on and eventually appears to lead to system instability. I guess the three possibilities are that this is either a) a hardware problem, b) a problem with poptop, pppd or the mppe modules, or c) a problem with the kernel

This happens on every single one of our DL140's, so if it's a hardware problem, then it is widespread on this platform.

If it's a problem with poptop, pppd or mppe modules I'd at least like to be able to give some evidence to that effect...

If it's c, I'm hoping someone here can give some suggestions on how to resolve this. :-) Please let me know if I can provide further information. I have a box on which this happens fairly regularly, so I can reproduce it pretty easily.

Thanks in advance!

System Specs:
HP DL140
Dual 2.4GHz P4 Xeon w/ Hyperthreading
Fedora Core 2
2.6.9 kernel (static compile, downloaded from ftp.kernel.org with the
following patches):
- http://marc.theaimsgroup.com/?l=linux-kernel&m=109926628920398
This fixes an oops I was getting in mm/prio_tree.c
- http://mdomsch.bkbits.net:8080/linux-2.6-mppe
This provides MPPE support for clients connecting via PoPToP 1.2.1.

Intel E100 Pro Dual Port NIC (Using built-in e100 driver). Onboard Broadcom BCM5700 Gigabit NIC's are *not* in use (no driver support
compiled in)

Software:
Poptop 1.2.1
PPP 2.4.3

Kernel config:

[snip]

Quoted 1 line

lspci output:

[snip]

Quoted 1 line

<7>divert: not allocating divert_blk for non-ethernet device ppp440

These messages I have gotten for years. They're debugging messages, as a result of:
register_netdevice()
alloc_divert_blk()
(sees it's not an ethernet device, printk's the message)

So I believe they're harmless.

Quoted 12 lines

ksymoops output of problem:

Unable to handle kernel NULL pointer dereference
00000000
*pde = 00000000
Oops: 0000 [#1]
CPU:    2
EIP:    0060:[<00000000>]    Not tainted VLI

Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010286 (2.6.9)
eax: ed13b000 ebx: d1d0a000 ecx: c029e9de edx: f795ef40
esi: d1d0a000 edi: 00000000 ebp: e2f30080 esp: d2b0dea0
ds: 007b es: 007b ss: 0068

Quoted 14 lines
Stack: c02a205a ed13b000 00000000 c02a122c d1d0a000 13208a2e c040956f d1d0a000 
       d1d0a00c e2f30080 00000000 c029cda9 d1d0a000 e2f30080 00000000 c01552cd 
       e2f30080 00000010 00000004 00000004 c0166aa0 e2f30080 00000000 00000000 
Call Trace:
 [<c02a205a>] pty_chars_in_buffer+0x2c/0x49
 [<c02a122c>] normal_poll+0xed/0x150
 [<c040956f>] schedule_timeout+0x75/0xbf
 [<c029cda9>] tty_poll+0xa0/0xb0
 [<c01552cd>] fget+0x49/0x5e
 [<c0166aa0>] do_select+0x269/0x2c6
 [<c0166691>] __pollwait+0x0/0xc7
 [<c0166dd5>] sys_select+0x2b3/0x4c6
 [<c0105971>] sysenter_past_esp+0x52/0x71
Code:  Bad EIP value.

It looks like pty_chars_in_buffer() dereferenced a NULL function pointer, but I don't see how that can be, the one deference is tested for NULL before doing so.

I can't rule out the ppp_mppe code, but I haven't seen this crash before myself. Does this happen on simlar systems that aren't running poptop?

Thanks,
Matt

Signature · 4 lines
Matt Domsch
Sr. Software Engineer, Lead Engineer
Dell Linux Solutions linux.dell.com & www.dell.com/linux
Linux on Dell mailing lists @ http://lists.us.dell.com
#2 lore raw
Quoted 6 lines · Matt Domsch

These messages I have gotten for years. They're debugging messages, as a result of:
register_netdevice( alloc_divert_blk( (sees it's not an ethernet device, )
printk's the message )

So I believe they're harmless.

This is what I was thinking too. I see plenty of these messages that do not result in crashes.

Quoted 27 lines
Quoted 12 lines

ksymoops output of problem:

Unable to handle kernel NULL pointer dereference
00000000
*pde = 00000000
Oops: 0000 [#1]
CPU: 2

EIP: 0060:[<00000000>] Not tainted VLI
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010286 (2.6.9)
eax: ed13b000 ebx: d1d0a000 ecx: c029e9de edx: f795ef40
esi: d1d0a000 edi: 00000000 ebp: e2f30080 esp: d2b0dea0
ds: 007b es: 007b ss: 0068

Quoted 10 lines
Stack: c02a205a ed13b000 00000000 c02a122c d1d0a000 13208a2e c040956f       
d1d0a000 d1d0a00c e2f30080 00000000 c029cda9 d1d0a000 e2f30080 00000000     
c01552cd e2f30080 00000010 00000004 00000004 c0166aa0 e2f30080 00000000     
00000000                                                                    
Call Trace: [<c02a205a>] pty_chars_in_buffer+0x2c/0x49 [<c02a122c>]
normal_poll+0xed/0x150 [<c040956f>] schedule_timeout+0x75/0xbf
[<c029cda9>] tty_poll+0xa0/0xb0 [<c01552cd>] fget+0x49/0x5e [<c0166aa0>]
do_select+0x269/0x2c6 [<c0166691>] __pollwait+0x0/0xc7 [<c0166dd5>]
sys_select+0x2b3/0x4c6 [<c0105971>] sysenter_past_esp+0x52/0x71
Code: Bad EIP value.

It looks like pty_chars_in_buffer() dereferenced a NULL function pointer, but I don't see how that can be, the one deference is tested for NULL before doing so.

Quoted 5 lines

I can't rule out the ppp_mppe code, but I haven't seen this crash before myself. Does this happen on simlar systems that aren't running poptop?

Thanks,
Matt

Fortunately we use the DL140 server pretty widely here. None of the non-poptop servers exhibit this problem. Every single one of them that has been used as a Poptop server however does exhibit this issue. It seems to happen more frequenly on the machines with a higher number of users connected.

I have switched one of the servers (the one with the highest load) to use Jan Dubiec's MPPC/MPPE patches for ppp 2.4.3 and Kernel 2.6.9. We'll see how that goes.

Any information I can provide or something I can do on my end to better trace this down?

Ray

#3 lore raw

Just a follow-up on this. Things seem to run better (no crashes yet after two days of runtime) after booting with nosmp noapic.

Any issues with the MPPE or CryptoAPI code that might only be triggered when running in SMP mode?

Ray

Quoted 56 lines · Ray Van Dolson
Quoted 6 lines · Matt Domsch

These messages I have gotten for years. They're debugging messages, as a result of:
register_netdevice( alloc_divert_blk( (sees it's not an ethernet device, ) printk's the message )

So I believe they're harmless.

This is what I was thinking too. I see plenty of these messages that do not result in crashes.

Quoted 26 lines
Quoted 12 lines

ksymoops output of problem:

Unable to handle kernel NULL pointer dereference
00000000
*pde = 00000000
Oops: 0000 [#1]
CPU: 2

EIP: 0060:[<00000000>] Not tainted VLI
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010286 (2.6.9)
eax: ed13b000 ebx: d1d0a000 ecx: c029e9de edx: f795ef40
esi: d1d0a000 edi: 00000000 ebp: e2f30080 esp: d2b0dea0
ds: 007b es: 007b ss: 0068

Quoted 10 lines
Stack: c02a205a ed13b000 00000000 c02a122c d1d0a000 13208a2e c040956f
d1d0a000 d1d0a00c e2f30080 00000000 c029cda9 d1d0a000 e2f30080 00000000
c01552cd e2f30080 00000010 00000004 00000004 c0166aa0 e2f30080 00000000
00000000
Call Trace: [<c02a205a>] pty_chars_in_buffer+0x2c/0x49 [<c02a122c>]
normal_poll+0xed/0x150 [<c040956f>] schedule_timeout+0x75/0xbf
[<c029cda9>] tty_poll+0xa0/0xb0 [<c01552cd>] fget+0x49/0x5e [<c0166aa0>]
do_select+0x269/0x2c6 [<c0166691>] __pollwait+0x0/0xc7 [<c0166dd5>]
sys_select+0x2b3/0x4c6 [<c0105971>] sysenter_past_esp+0x52/0x71
Code: Bad EIP value.

It looks like pty_chars_in_buffer() dereferenced a NULL function pointer, but I don't see how that can be, the one deference is tested for NULL before doing so.

Quoted 6 lines

I can't rule out the ppp_mppe code, but I haven't seen this crash before myself. Does this happen on simlar systems that aren't running poptop?

Thanks,
Matt

Fortunately we use the DL140 server pretty widely here. None of the non-poptop servers exhibit this problem. Every single one of them that has been used as a Poptop server however does exhibit this issue. It seems to happen more frequenly on the machines with a higher number of users connected.

I have switched one of the servers (the one with the highest load) to use Jan Dubiec's MPPC/MPPE patches for ppp 2.4.3 and Kernel 2.6.9. We'll see how that goes.

Any information I can provide or something I can do on my end to better trace this down?

Ray

#4 lore raw
In reply to Matt Domsch
Quoted 26 lines · Matt Domsch
Quoted 12 lines

ksymoops output of problem:

Unable to handle kernel NULL pointer dereference
00000000
*pde = 00000000
Oops: 0000 [#1]
CPU: 2

EIP: 0060:[<00000000>] Not tainted VLI
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010286 (2.6.9)
eax: ed13b000 ebx: d1d0a000 ecx: c029e9de edx: f795ef40
esi: d1d0a000 edi: 00000000 ebp: e2f30080 esp: d2b0dea0
ds: 007b es: 007b ss: 0068

Quoted 10 lines
Stack: c02a205a ed13b000 00000000 c02a122c d1d0a000 13208a2e c040956f       
d1d0a000 d1d0a00c e2f30080 00000000 c029cda9 d1d0a000 e2f30080 00000000     
c01552cd e2f30080 00000010 00000004 00000004 c0166aa0 e2f30080 00000000     
00000000                                                                    
Call Trace: [<c02a205a>] pty_chars_in_buffer+0x2c/0x49 [<c02a122c>]
normal_poll+0xed/0x150 [<c040956f>] schedule_timeout+0x75/0xbf
[<c029cda9>] tty_poll+0xa0/0xb0 [<c01552cd>] fget+0x49/0x5e [<c0166aa0>]
do_select+0x269/0x2c6 [<c0166691>] __pollwait+0x0/0xc7 [<c0166dd5>]
sys_select+0x2b3/0x4c6 [<c0105971>] sysenter_past_esp+0x52/0x71
Code: Bad EIP value.

It looks like pty_chars_in_buffer() dereferenced a NULL function pointer, but I don't see how that can be, the one deference is tested for NULL before doing so.

Quoted 5 lines

I can't rule out the ppp_mppe code, but I haven't seen this crash before myself. Does this happen on simlar systems that aren't running poptop?

Thanks,
Matt

Yet another follow-up on this. After completely disabling SMP/Hyperthreading on these systems, we have had zero NULL pointer dereferences. Obviously can't say for sure that this points to problems with the MPPE module--the only true test would be to allow our customers to connect without MPPE and reenable SMP/HT. Obviously not an option in a live environment. :)

Guess we just won't be using SMP/HT on our busier systems!

Ray