Unable to handle kernel NULL pointer dereference at virtual address 00000000
Hoping someone can shed some light on this, or help me out in narrowing the problem down.
Our servers exhibit these error messages (at bottom) off and on and eventually appears to lead to system instability. I guess the three possibilities are that this is either a) a hardware problem, b) a problem with poptop, pppd or the mppe modules, or c) a problem with the kernel
This happens on every single one of our DL140's, so if it's a hardware problem, then it is widespread on this platform.
If it's a problem with poptop, pppd or mppe modules I'd at least like to be able to give some evidence to that effect...
If it's c, I'm hoping someone here can give some suggestions on how to resolve this. :-) Please let me know if I can provide further information. I have a box on which this happens fairly regularly, so I can reproduce it pretty easily.
Thanks in advance!
System Specs:
HP DL140
Dual 2.4GHz P4 Xeon w/ Hyperthreading
Fedora Core 2
2.6.9 kernel (static compile, downloaded from ftp.kernel.org with the
following patches):
- http://marc.theaimsgroup.com/?l=linux-kernel&m=109926628920398
This fixes an oops I was getting in mm/prio_tree.c
- http://mdomsch.bkbits.net:8080/linux-2.6-mppe
This provides MPPE support for clients connecting via PoPToP 1.2.1.
Intel E100 Pro Dual Port NIC (Using built-in e100 driver). Onboard Broadcom BCM5700 Gigabit NIC's are *not* in use (no driver support
compiled in)
Software:
Poptop 1.2.1
PPP 2.4.3
Kernel config:
CONFIG_X86=yCONFIG_MMU=yCONFIG_UID16=yCONFIG_GENERIC_ISA_DMA=yCONFIG_GENERIC_IOMAP=yCONFIG_EXPERIMENTAL=yCONFIG_CLEAN_COMPILE=yCONFIG_LOCALVERSION=""CONFIG_SWAP=yCONFIG_SYSVIPC=yCONFIG_POSIX_MQUEUE=yCONFIG_BSD_PROCESS_ACCT=yCONFIG_SYSCTL=yCONFIG_AUDIT=yCONFIG_AUDITSYSCALL=yCONFIG_LOG_BUF_SHIFT=17CONFIG_IKCONFIG=yCONFIG_IKCONFIG_PROC=yCONFIG_KALLSYMS=yCONFIG_KALLSYMS_EXTRA_PASS=yCONFIG_FUTEX=yCONFIG_EPOLL=yCONFIG_IOSCHED_NOOP=yCONFIG_IOSCHED_AS=yCONFIG_IOSCHED_DEADLINE=yCONFIG_IOSCHED_CFQ=yCONFIG_SHMEM=yCONFIG_X86_PC=yCONFIG_MPENTIUM4=yCONFIG_X86_GENERIC=yCONFIG_X86_CMPXCHG=yCONFIG_X86_XADD=yCONFIG_X86_L1_CACHE_SHIFT=7CONFIG_RWSEM_XCHGADD_ALGORITHM=yCONFIG_X86_WP_WORKS_OK=yCONFIG_X86_INVLPG=yCONFIG_X86_BSWAP=yCONFIG_X86_POPAD_OK=yCONFIG_X86_GOOD_APIC=yCONFIG_X86_INTEL_USERCOPY=yCONFIG_X86_USE_PPRO_CHECKSUM=yCONFIG_HPET_TIMER=yCONFIG_HPET_EMULATE_RTC=yCONFIG_SMP=yCONFIG_NR_CPUS=8CONFIG_SCHED_SMT=yCONFIG_X86_LOCAL_APIC=yCONFIG_X86_IO_APIC=yCONFIG_X86_TSC=yCONFIG_X86_MCE=yCONFIG_X86_MCE_NONFATAL=yCONFIG_X86_MCE_P4THERMAL=yCONFIG_MICROCODE=yCONFIG_X86_MSR=yCONFIG_X86_CPUID=yCONFIG_HIGHMEM4G=yCONFIG_HIGHMEM=yCONFIG_HIGHPTE=yCONFIG_MTRR=yCONFIG_IRQBALANCE=yCONFIG_HAVE_DEC_LOCK=yCONFIG_ACPI=yCONFIG_ACPI_BOOT=yCONFIG_ACPI_INTERPRETER=yCONFIG_ACPI_AC=yCONFIG_ACPI_BATTERY=yCONFIG_ACPI_BUTTON=yCONFIG_ACPI_FAN=yCONFIG_ACPI_PROCESSOR=yCONFIG_ACPI_THERMAL=yCONFIG_ACPI_ASUS=yCONFIG_ACPI_TOSHIBA=yCONFIG_ACPI_BLACKLIST_YEAR=2001CONFIG_ACPI_BUS=yCONFIG_ACPI_EC=yCONFIG_ACPI_POWER=yCONFIG_ACPI_PCI=yCONFIG_ACPI_SYSTEM=yCONFIG_X86_PM_TIMER=yCONFIG_PCI=yCONFIG_PCI_GOANY=yCONFIG_PCI_BIOS=yCONFIG_PCI_DIRECT=yCONFIG_PCI_MMCONFIG=yCONFIG_PCI_LEGACY_PROC=yCONFIG_PCI_NAMES=yCONFIG_ISA=yCONFIG_BINFMT_ELF=yCONFIG_BINFMT_MISC=yCONFIG_STANDALONE=yCONFIG_PREVENT_FIRMWARE_BUILD=yCONFIG_PNP=yCONFIG_ISAPNP=yCONFIG_BLK_DEV_FD=yCONFIG_BLK_DEV_LOOP=yCONFIG_BLK_DEV_CRYPTOLOOP=yCONFIG_BLK_DEV_NBD=yCONFIG_BLK_DEV_RAM=yCONFIG_BLK_DEV_RAM_SIZE=16384CONFIG_BLK_DEV_INITRD=yCONFIG_LBD=yCONFIG_IDE=yCONFIG_BLK_DEV_IDE=yCONFIG_BLK_DEV_IDEDISK=yCONFIG_IDEDISK_MULTI_MODE=yCONFIG_BLK_DEV_IDECD=yCONFIG_BLK_DEV_IDESCSI=yCONFIG_IDE_GENERIC=yCONFIG_BLK_DEV_IDEPCI=yCONFIG_IDEPCI_SHARE_IRQ=yCONFIG_BLK_DEV_GENERIC=yCONFIG_BLK_DEV_IDEDMA_PCI=yCONFIG_IDEDMA_PCI_AUTO=yCONFIG_BLK_DEV_SVWKS=yCONFIG_BLK_DEV_IDEDMA=yCONFIG_IDEDMA_AUTO=yCONFIG_SCSI=yCONFIG_BLK_DEV_SD=yCONFIG_BLK_DEV_SR=yCONFIG_BLK_DEV_SR_VENDOR=yCONFIG_CHR_DEV_SG=yCONFIG_SCSI_SPI_ATTRS=yCONFIG_SCSI_FC_ATTRS=yCONFIG_SCSI_QLA2XXX=yCONFIG_I2O=yCONFIG_I2O_CONFIG=yCONFIG_I2O_BLOCK=yCONFIG_I2O_SCSI=yCONFIG_I2O_PROC=yCONFIG_NET=yCONFIG_PACKET=yCONFIG_PACKET_MMAP=yCONFIG_NETLINK_DEV=yCONFIG_UNIX=yCONFIG_NET_KEY=yCONFIG_INET=yCONFIG_IP_ADVANCED_ROUTER=yCONFIG_IP_MULTIPLE_TABLES=yCONFIG_IP_ROUTE_FWMARK=yCONFIG_IP_ROUTE_MULTIPATH=yCONFIG_IP_ROUTE_VERBOSE=yCONFIG_NET_IPIP=yCONFIG_NET_IPGRE=yCONFIG_SYN_COOKIES=yCONFIG_INET_AH=yCONFIG_INET_ESP=yCONFIG_INET_IPCOMP=yCONFIG_INET_TUNNEL=yCONFIG_NETFILTER=yCONFIG_IP_NF_CONNTRACK=yCONFIG_IP_NF_CT_ACCT=yCONFIG_IP_NF_CT_PROTO_SCTP=yCONFIG_IP_NF_FTP=yCONFIG_IP_NF_IRC=yCONFIG_IP_NF_TFTP=yCONFIG_IP_NF_AMANDA=yCONFIG_IP_NF_QUEUE=yCONFIG_IP_NF_IPTABLES=yCONFIG_IP_NF_MATCH_LIMIT=yCONFIG_IP_NF_MATCH_IPRANGE=yCONFIG_IP_NF_MATCH_MAC=yCONFIG_IP_NF_MATCH_PKTTYPE=yCONFIG_IP_NF_MATCH_MARK=yCONFIG_IP_NF_MATCH_MULTIPORT=yCONFIG_IP_NF_MATCH_TOS=yCONFIG_IP_NF_MATCH_RECENT=yCONFIG_IP_NF_MATCH_ECN=yCONFIG_IP_NF_MATCH_DSCP=yCONFIG_IP_NF_MATCH_AH_ESP=yCONFIG_IP_NF_MATCH_LENGTH=yCONFIG_IP_NF_MATCH_TTL=yCONFIG_IP_NF_MATCH_TCPMSS=yCONFIG_IP_NF_MATCH_HELPER=yCONFIG_IP_NF_MATCH_STATE=yCONFIG_IP_NF_MATCH_CONNTRACK=yCONFIG_IP_NF_MATCH_OWNER=yCONFIG_IP_NF_MATCH_ADDRTYPE=yCONFIG_IP_NF_MATCH_REALM=yCONFIG_IP_NF_MATCH_SCTP=yCONFIG_IP_NF_MATCH_COMMENT=yCONFIG_IP_NF_FILTER=yCONFIG_IP_NF_TARGET_REJECT=yCONFIG_IP_NF_TARGET_LOG=yCONFIG_IP_NF_TARGET_ULOG=yCONFIG_IP_NF_TARGET_TCPMSS=yCONFIG_IP_NF_NAT=yCONFIG_IP_NF_NAT_NEEDED=yCONFIG_IP_NF_TARGET_MASQUERADE=yCONFIG_IP_NF_TARGET_REDIRECT=yCONFIG_IP_NF_TARGET_NETMAP=yCONFIG_IP_NF_TARGET_SAME=yCONFIG_IP_NF_NAT_LOCAL=yCONFIG_IP_NF_NAT_SNMP_BASIC=yCONFIG_IP_NF_NAT_IRC=yCONFIG_IP_NF_NAT_FTP=yCONFIG_IP_NF_NAT_TFTP=yCONFIG_IP_NF_NAT_AMANDA=yCONFIG_IP_NF_MANGLE=yCONFIG_IP_NF_TARGET_TOS=yCONFIG_IP_NF_TARGET_ECN=yCONFIG_IP_NF_TARGET_DSCP=yCONFIG_IP_NF_TARGET_MARK=yCONFIG_IP_NF_TARGET_CLASSIFY=yCONFIG_IP_NF_RAW=yCONFIG_IP_NF_TARGET_NOTRACK=yCONFIG_IP_NF_ARPTABLES=yCONFIG_IP_NF_ARPFILTER=yCONFIG_IP_NF_ARP_MANGLE=yCONFIG_XFRM=yCONFIG_XFRM_USER=yCONFIG_NET_DIVERT=yCONFIG_NET_SCHED=yCONFIG_NET_SCH_CLK_JIFFIES=yCONFIG_NET_SCH_CBQ=yCONFIG_NET_SCH_HTB=yCONFIG_NET_SCH_HFSC=yCONFIG_NET_SCH_PRIO=yCONFIG_NET_SCH_RED=yCONFIG_NET_SCH_SFQ=yCONFIG_NET_SCH_TEQL=yCONFIG_NET_SCH_TBF=yCONFIG_NET_SCH_GRED=yCONFIG_NET_SCH_DSMARK=yCONFIG_NET_SCH_NETEM=yCONFIG_NET_SCH_INGRESS=yCONFIG_NET_QOS=yCONFIG_NET_ESTIMATOR=yCONFIG_NET_CLS=yCONFIG_NET_CLS_TCINDEX=yCONFIG_NET_CLS_ROUTE4=yCONFIG_NET_CLS_ROUTE=yCONFIG_NET_CLS_FW=yCONFIG_NET_CLS_U32=yCONFIG_CLS_U32_PERF=yCONFIG_NET_CLS_IND=yCONFIG_NET_CLS_RSVP=yCONFIG_NET_CLS_RSVP6=yCONFIG_NET_CLS_POLICE=yCONFIG_NETPOLL=yCONFIG_NET_POLL_CONTROLLER=yCONFIG_NETDEVICES=yCONFIG_DUMMY=yCONFIG_NET_ETHERNET=yCONFIG_MII=yCONFIG_NET_PCI=yCONFIG_E100=yCONFIG_PPP=yCONFIG_PPP_MULTILINK=yCONFIG_PPP_FILTER=yCONFIG_PPP_ASYNC=yCONFIG_PPP_SYNC_TTY=yCONFIG_PPP_DEFLATE=yCONFIG_PPP_BSDCOMP=yCONFIG_PPP_MPPE=yCONFIG_NETCONSOLE=yCONFIG_INPUT=yCONFIG_INPUT_MOUSEDEV=yCONFIG_INPUT_MOUSEDEV_SCREEN_X=1024CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768CONFIG_SOUND_GAMEPORT=yCONFIG_SERIO=yCONFIG_SERIO_I8042=yCONFIG_INPUT_KEYBOARD=yCONFIG_KEYBOARD_ATKBD=yCONFIG_INPUT_MOUSE=yCONFIG_MOUSE_PS2=yCONFIG_MOUSE_SERIAL=yCONFIG_INPUT_MISC=yCONFIG_INPUT_PCSPKR=yCONFIG_INPUT_UINPUT=yCONFIG_VT=yCONFIG_VT_CONSOLE=yCONFIG_HW_CONSOLE=yCONFIG_SERIAL_8250=yCONFIG_SERIAL_8250_CONSOLE=yCONFIG_SERIAL_8250_NR_UARTS=1024CONFIG_SERIAL_8250_EXTENDED=yCONFIG_SERIAL_8250_SHARE_IRQ=yCONFIG_SERIAL_8250_DETECT_IRQ=yCONFIG_SERIAL_8250_MULTIPORT=yCONFIG_SERIAL_8250_RSA=yCONFIG_SERIAL_CORE=yCONFIG_SERIAL_CORE_CONSOLE=yCONFIG_UNIX98_PTYS=yCONFIG_IPMI_HANDLER=yCONFIG_IPMI_DEVICE_INTERFACE=yCONFIG_IPMI_SI=yCONFIG_IPMI_WATCHDOG=yCONFIG_IPMI_POWEROFF=yCONFIG_HW_RANDOM=yCONFIG_NVRAM=yCONFIG_RTC=yCONFIG_AGP=yCONFIG_AGP_ATI=yCONFIG_AGP_INTEL=yCONFIG_HANGCHECK_TIMER=yCONFIG_I2C=yCONFIG_I2C_CHARDEV=yCONFIG_I2C_ALGOBIT=yCONFIG_I2C_ALGOPCF=yCONFIG_I2C_ALGOPCA=yCONFIG_I2C_I801=yCONFIG_I2C_I810=yCONFIG_I2C_ISA=yCONFIG_I2C_PIIX4=yCONFIG_I2C_SENSOR=yCONFIG_SENSORS_ADM1021=yCONFIG_SENSORS_ADM1025=yCONFIG_SENSORS_ADM1031=yCONFIG_SENSORS_ASB100=yCONFIG_SENSORS_DS1621=yCONFIG_SENSORS_FSCHER=yCONFIG_SENSORS_GL518SM=yCONFIG_SENSORS_IT87=yCONFIG_SENSORS_LM75=yCONFIG_SENSORS_LM77=yCONFIG_SENSORS_LM78=yCONFIG_SENSORS_LM80=yCONFIG_SENSORS_LM83=yCONFIG_SENSORS_LM85=yCONFIG_SENSORS_LM90=yCONFIG_SENSORS_MAX1619=yCONFIG_SENSORS_SMSC47M1=yCONFIG_SENSORS_VIA686A=yCONFIG_SENSORS_W83781D=yCONFIG_SENSORS_W83L785TS=yCONFIG_SENSORS_W83627HF=yCONFIG_SENSORS_EEPROM=yCONFIG_SENSORS_PCF8574=yCONFIG_SENSORS_PCF8591=yCONFIG_SENSORS_RTC8564=yCONFIG_VIDEO_DEV=yCONFIG_VIDEO_SELECT=yCONFIG_VGA_CONSOLE=yCONFIG_DUMMY_CONSOLE=yCONFIG_EXT2_FS=yCONFIG_EXT2_FS_XATTR=yCONFIG_EXT2_FS_POSIX_ACL=yCONFIG_EXT2_FS_SECURITY=yCONFIG_EXT3_FS=yCONFIG_EXT3_FS_XATTR=yCONFIG_EXT3_FS_POSIX_ACL=yCONFIG_EXT3_FS_SECURITY=yCONFIG_JBD=yCONFIG_FS_MBCACHE=yCONFIG_FS_POSIX_ACL=yCONFIG_ROMFS_FS=yCONFIG_AUTOFS_FS=yCONFIG_AUTOFS4_FS=yCONFIG_ISO9660_FS=yCONFIG_JOLIET=yCONFIG_ZISOFS=yCONFIG_ZISOFS_FS=yCONFIG_UDF_FS=yCONFIG_UDF_NLS=yCONFIG_PROC_FS=yCONFIG_PROC_KCORE=yCONFIG_SYSFS=yCONFIG_TMPFS=yCONFIG_RAMFS=yCONFIG_NFS_FS=yCONFIG_NFS_V3=yCONFIG_NFS_V4=yCONFIG_NFS_DIRECTIO=yCONFIG_NFSD=yCONFIG_NFSD_V3=yCONFIG_NFSD_V4=yCONFIG_NFSD_TCP=yCONFIG_LOCKD=yCONFIG_LOCKD_V4=yCONFIG_EXPORTFS=yCONFIG_SUNRPC=yCONFIG_SUNRPC_GSS=yCONFIG_RPCSEC_GSS_KRB5=yCONFIG_SMB_FS=yCONFIG_CIFS=yCONFIG_CIFS_XATTR=yCONFIG_CIFS_POSIX=yCONFIG_PARTITION_ADVANCED=yCONFIG_OSF_PARTITION=yCONFIG_MAC_PARTITION=yCONFIG_MSDOS_PARTITION=yCONFIG_BSD_DISKLABEL=yCONFIG_MINIX_SUBPARTITION=yCONFIG_SOLARIS_X86_PARTITION=yCONFIG_UNIXWARE_DISKLABEL=yCONFIG_SGI_PARTITION=yCONFIG_SUN_PARTITION=yCONFIG_EFI_PARTITION=yCONFIG_NLS=yCONFIG_NLS_DEFAULT="utf8"CONFIG_NLS_CODEPAGE_437=yCONFIG_NLS_CODEPAGE_737=yCONFIG_NLS_CODEPAGE_775=yCONFIG_NLS_CODEPAGE_850=yCONFIG_NLS_CODEPAGE_852=yCONFIG_NLS_CODEPAGE_855=yCONFIG_NLS_CODEPAGE_857=yCONFIG_NLS_CODEPAGE_860=yCONFIG_NLS_CODEPAGE_861=yCONFIG_NLS_CODEPAGE_862=yCONFIG_NLS_CODEPAGE_863=yCONFIG_NLS_CODEPAGE_864=yCONFIG_NLS_CODEPAGE_865=yCONFIG_NLS_CODEPAGE_866=yCONFIG_NLS_CODEPAGE_869=yCONFIG_NLS_CODEPAGE_936=yCONFIG_NLS_CODEPAGE_950=yCONFIG_NLS_CODEPAGE_932=yCONFIG_NLS_CODEPAGE_949=yCONFIG_NLS_CODEPAGE_874=yCONFIG_NLS_ISO8859_8=yCONFIG_NLS_CODEPAGE_1250=yCONFIG_NLS_CODEPAGE_1251=yCONFIG_NLS_ASCII=yCONFIG_NLS_ISO8859_1=yCONFIG_NLS_ISO8859_2=yCONFIG_NLS_ISO8859_3=yCONFIG_NLS_ISO8859_4=yCONFIG_NLS_ISO8859_5=yCONFIG_NLS_ISO8859_6=yCONFIG_NLS_ISO8859_7=yCONFIG_NLS_ISO8859_9=yCONFIG_NLS_ISO8859_13=yCONFIG_NLS_ISO8859_14=yCONFIG_NLS_ISO8859_15=yCONFIG_NLS_KOI8_R=yCONFIG_NLS_KOI8_U=yCONFIG_NLS_UTF8=yCONFIG_DEBUG_KERNEL=yCONFIG_MAGIC_SYSRQ=yCONFIG_EARLY_PRINTK=yCONFIG_DEBUG_STACKOVERFLOW=yCONFIG_DEBUG_STACK_USAGE=yCONFIG_X86_FIND_SMP_CONFIG=yCONFIG_X86_MPPARSE=yCONFIG_CRYPTO=yCONFIG_CRYPTO_HMAC=yCONFIG_CRYPTO_NULL=yCONFIG_CRYPTO_MD4=yCONFIG_CRYPTO_MD5=yCONFIG_CRYPTO_SHA1=yCONFIG_CRYPTO_SHA256=yCONFIG_CRYPTO_SHA512=yCONFIG_CRYPTO_WP512=yCONFIG_CRYPTO_DES=yCONFIG_CRYPTO_BLOWFISH=yCONFIG_CRYPTO_TWOFISH=yCONFIG_CRYPTO_SERPENT=yCONFIG_CRYPTO_AES_586=yCONFIG_CRYPTO_CAST5=yCONFIG_CRYPTO_CAST6=yCONFIG_CRYPTO_TEA=yCONFIG_CRYPTO_ARC4=yCONFIG_CRYPTO_KHAZAD=yCONFIG_CRYPTO_DEFLATE=yCONFIG_CRYPTO_MICHAEL_MIC=yCONFIG_CRYPTO_CRC32C=yCONFIG_CRC_CCITT=yCONFIG_CRC32=yCONFIG_LIBCRC32C=yCONFIG_ZLIB_INFLATE=yCONFIG_ZLIB_DEFLATE=yCONFIG_X86_SMP=yCONFIG_X86_HT=yCONFIG_X86_BIOS_REBOOT=yCONFIG_X86_TRAMPOLINE=yCONFIG_PC=y
lspci output:
00:00.0 Host bridge: ServerWorks GCNB-LE Host Bridge (rev 32)
00:00.1 Host bridge: ServerWorks GCNB-LE Host Bridge
00:03.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27) 00:0f.0 ISA bridge: ServerWorks CSB6 South Bridge (rev a0)
00:0f.1 IDE interface: ServerWorks CSB6 RAID/IDE Controller (rev a0)
00:0f.2 USB Controller: ServerWorks CSB6 OHCI USB Controller (rev 05)
00:0f.3 Host bridge: ServerWorks GCLE-2 Host Bridge
00:10.0 Host bridge: ServerWorks CIOB-E I/O Bridge with Gigabit Ethernet (rev 12) 00:10.2 Host bridge: ServerWorks CIOB-E I/O Bridge with Gigabit Ethernet (rev 12) 01:06.0 PCI bridge: Digital Equipment Corporation DECchip 21152 (rev 03) 02:04.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 05) 02:05.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 05) 03:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 02) 03:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 02)
ksymoops output of problem:
ksymoops 2.4.9 on i686 2.6.9. Options used
-V (default)
-K (specified)
-l /proc/modules (default)
-O (specified)
-m /boot/System.map (specified)
No ksyms, skipping lsmod
Unable to handle kernel NULL pointer dereference<7>divert: not allocating divert_blk for non-ethernet device ppp440 00000000 *pde = 00000000 Oops: 0000 [#1] CPU: 2 EIP: 0060:[<00000000>] Not tainted VLI
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010286 (2.6.9)
eax: ed13b000 ebx: d1d0a000 ecx: c029e9de edx: f795ef40
esi: d1d0a000 edi: 00000000 ebp: e2f30080 esp: d2b0dea0
ds: 007b es: 007b ss: 0068
Stack: c02a205a ed13b000 00000000 c02a122c d1d0a000 13208a2e c040956f d1d0a000
d1d0a00c e2f30080 00000000 c029cda9 d1d0a000 e2f30080 00000000 c01552cd
e2f30080 00000010 00000004 00000004 c0166aa0 e2f30080 00000000 00000000
Call Trace:
[<c02a205a>] pty_chars_in_buffer+0x2c/0x49
[<c02a122c>] normal_poll+0xed/0x150
[<c040956f>] schedule_timeout+0x75/0xbf
[<c029cda9>] tty_poll+0xa0/0xb0
[<c01552cd>] fget+0x49/0x5e
[<c0166aa0>] do_select+0x269/0x2c6
[<c0166691>] __pollwait+0x0/0xc7
[<c0166dd5>] sys_select+0x2b3/0x4c6
[<c0105971>] sysenter_past_esp+0x52/0x71
Code: Bad EIP value.Quoted 9 lines
Quoted 9 lines
EIP; 00000000 Before first symbol
eax;
ed13b000<pg0+2ca80000/3f943400>
ebx;d1d0a000<pg0+1164f000/3f943400>
ecx;c029e9de<n_tty_chars_in_buffer+0/78>
edx;f795ef40<pg0+372a3f40/3f943400>
esi;d1d0a000<pg0+1164f000/3f943400>
ebp;e2f30080<pg0+22875080/3f943400>
esp;d2b0dea0<pg0+12452ea0/3f943400>
Trace; c02a205a <pty_chars_in_buffer+2c/49>
Trace; c02a122c <normal_poll+ed/150>
Trace; c040956f <schedule_timeout+75/bf>
Trace; c029cda9 <tty_poll+a0/b0>
Trace; c01552cd <fget+49/5e>
Trace; c0166aa0 <do_select+269/2c6>
Trace; c0166691 <__pollwait+0/c7>
Trace; c0166dd5 <sys_select+2b3/4c6>
Trace; c0105971 <sysenter_past_esp+52/71>
Yet another ksymoops output:
ksymoops 2.4.9 on i686 2.6.9. Options used
-V (default)
-K (specified)
-l /proc/modules (default)
-O (specified)
-m /boot/System.map (specified)
No ksyms, skipping lsmod
Unable to handle kernel NULL pointer dereference<7>divert: not allocating divert_blk for non-ethernet device ppp504 00000000 *pde = 00000000 Oops: 0000 [#2] CPU: 2 EIP: 0060:[<00000000>] Not tainted VLI
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010286 (2.6.9)
eax: cc8ae000 ebx: d46ca000 ecx: c029e9de edx: f7801880
esi: d46ca000 edi: 00000000 ebp: e16a1d80 esp: d2907ea0
ds: 007b es: 007b ss: 0068
Stack: c02a205a cc8ae000 00000000 c02a122c d46ca000 1da86df5 c040956f d46ca000
d46ca00c e16a1d80 00000000 c029cda9 d46ca000 e16a1d80 00000000 c01552cd
e16a1d80 00000010 00000004 00000004 c0166aa0 e16a1d80 00000000 00000000
Call Trace:
[<c02a205a>] pty_chars_in_buffer+0x2c/0x49
[<c02a122c>] normal_poll+0xed/0x150
[<c040956f>] schedule_timeout+0x75/0xbf
[<c029cda9>] tty_poll+0xa0/0xb0
[<c01552cd>] fget+0x49/0x5e
[<c0166aa0>] do_select+0x269/0x2c6
[<c0166691>] __pollwait+0x0/0xc7
[<c0166dd5>] sys_select+0x2b3/0x4c6
[<c0105971>] sysenter_past_esp+0x52/0x71
Code: Bad EIP value.Quoted 9 lines
Quoted 9 lines
EIP; 00000000 Before first symbol
eax;
cc8ae000<pg0+c1f3000/3f943400>
ebx;d46ca000<pg0+1400f000/3f943400>
ecx;c029e9de<n_tty_chars_in_buffer+0/78>
edx;f7801880<pg0+37146880/3f943400>
esi;d46ca000<pg0+1400f000/3f943400>
ebp;e16a1d80<pg0+20fe6d80/3f943400>
esp;d2907ea0<pg0+1224cea0/3f943400>
Trace; c02a205a <pty_chars_in_buffer+2c/49>
Trace; c02a122c <normal_poll+ed/150>
Trace; c040956f <schedule_timeout+75/bf>
Trace; c029cda9 <tty_poll+a0/b0>
Trace; c01552cd <fget+49/5e>
Trace; c0166aa0 <do_select+269/2c6>
Trace; c0166691 <__pollwait+0/c7>
Trace; c0166dd5 <sys_select+2b3/4c6>
Trace; c0105971 <sysenter_past_esp+52/71>
Quoted 41 lines · Ray Van Dolson
Hoping someone can shed some light on this, or help me out in narrowing the problem down.
Our servers exhibit these error messages (at bottom) off and on and eventually appears to lead to system instability. I guess the three possibilities are that this is either a) a hardware problem, b) a problem with poptop, pppd or the mppe modules, or c) a problem with the kernel
This happens on every single one of our DL140's, so if it's a hardware problem, then it is widespread on this platform.
If it's a problem with poptop, pppd or mppe modules I'd at least like to be able to give some evidence to that effect...
If it's c, I'm hoping someone here can give some suggestions on how to resolve this. :-) Please let me know if I can provide further information. I have a box on which this happens fairly regularly, so I can reproduce it pretty easily.
Thanks in advance!
System Specs:
HP DL140
Dual 2.4GHz P4 Xeon w/ Hyperthreading
Fedora Core 2
2.6.9 kernel (static compile, downloaded from ftp.kernel.org with the
following patches):
- http://marc.theaimsgroup.com/?l=linux-kernel&m=109926628920398
This fixes an oops I was getting inmm/prio_tree.c
- http://mdomsch.bkbits.net:8080/linux-2.6-mppe
This provides MPPE support for clients connecting via PoPToP 1.2.1.Intel E100 Pro Dual Port NIC (Using built-in e100 driver). Onboard Broadcom BCM5700 Gigabit NIC's are *not* in use (no driver support
compiled in)Software:
Poptop 1.2.1
PPP 2.4.3Kernel config:
[snip]
Quoted 1 line
lspci output:
[snip]
Quoted 1 line
<7>divert: not allocating
divert_blkfor non-ethernet device ppp440
These messages I have gotten for years. They're debugging messages, as a result of:register_netdevice()
alloc_divert_blk()
(sees it's not an ethernet device, printk's the message)
So I believe they're harmless.
Quoted 12 lines
ksymoops output of problem:
Unable to handle kernel NULL pointer dereference 00000000 *pde = 00000000 Oops: 0000 [#1] CPU: 2 EIP: 0060:[<00000000>] Not tainted VLIUsing defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010286 (2.6.9)
eax:ed13b000ebx:d1d0a000ecx:c029e9deedx:f795ef40
esi:d1d0a000edi: 00000000 ebp:e2f30080esp:d2b0dea0
ds: 007b es: 007b ss: 0068
Quoted 14 lines
Stack: c02a205a ed13b000 00000000 c02a122c d1d0a000 13208a2e c040956f d1d0a000
d1d0a00c e2f30080 00000000 c029cda9 d1d0a000 e2f30080 00000000 c01552cd
e2f30080 00000010 00000004 00000004 c0166aa0 e2f30080 00000000 00000000
Call Trace:
[<c02a205a>] pty_chars_in_buffer+0x2c/0x49
[<c02a122c>] normal_poll+0xed/0x150
[<c040956f>] schedule_timeout+0x75/0xbf
[<c029cda9>] tty_poll+0xa0/0xb0
[<c01552cd>] fget+0x49/0x5e
[<c0166aa0>] do_select+0x269/0x2c6
[<c0166691>] __pollwait+0x0/0xc7
[<c0166dd5>] sys_select+0x2b3/0x4c6
[<c0105971>] sysenter_past_esp+0x52/0x71
Code: Bad EIP value.It looks like pty_chars_in_buffer() dereferenced a NULL function pointer, but I don't see how that can be, the one deference is tested for NULL before doing so.
I can't rule out the ppp_mppe code, but I haven't seen this crash before myself. Does this happen on simlar systems that aren't running poptop?
Thanks,
Matt
Signature · 4 lines
Matt Domsch Sr. Software Engineer, Lead Engineer Dell Linux Solutions linux.dell.com & www.dell.com/linux Linux on Dell mailing lists @ http://lists.us.dell.com
Quoted 6 lines · Matt Domsch
These messages I have gotten for years. They're debugging messages, as a result of:
register_netdevice( alloc_divert_blk( (sees it's not an ethernet device, )
printk's the message )So I believe they're harmless.
This is what I was thinking too. I see plenty of these messages that do not result in crashes.
Quoted 27 lines
Quoted 12 lines
ksymoops output of problem:
Unable to handle kernel NULL pointer dereference 00000000 *pde = 00000000 Oops: 0000 [#1] CPU: 2EIP: 0060:[<00000000>] Not tainted VLI
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010286 (2.6.9)
eax:ed13b000ebx:d1d0a000ecx:c029e9deedx:f795ef40
esi:d1d0a000edi: 00000000 ebp:e2f30080esp:d2b0dea0
ds: 007b es: 007b ss: 0068Quoted 10 lines
Stack: c02a205a ed13b000 00000000 c02a122c d1d0a000 13208a2e c040956f d1d0a000 d1d0a00c e2f30080 00000000 c029cda9 d1d0a000 e2f30080 00000000 c01552cd e2f30080 00000010 00000004 00000004 c0166aa0 e2f30080 00000000 00000000 Call Trace: [<c02a205a>] pty_chars_in_buffer+0x2c/0x49 [<c02a122c>] normal_poll+0xed/0x150 [<c040956f>] schedule_timeout+0x75/0xbf [<c029cda9>] tty_poll+0xa0/0xb0 [<c01552cd>] fget+0x49/0x5e [<c0166aa0>] do_select+0x269/0x2c6 [<c0166691>] __pollwait+0x0/0xc7 [<c0166dd5>] sys_select+0x2b3/0x4c6 [<c0105971>] sysenter_past_esp+0x52/0x71 Code: Bad EIP value.It looks like
pty_chars_in_buffer()dereferenced a NULL function pointer, but I don't see how that can be, the one deference is tested for NULL before doing so.
Quoted 5 lines
I can't rule out the
ppp_mppecode, but I haven't seen this crash before myself. Does this happen on simlar systems that aren't running poptop?Thanks,
Matt
Fortunately we use the DL140 server pretty widely here. None of the non-poptop servers exhibit this problem. Every single one of them that has been used as a Poptop server however does exhibit this issue. It seems to happen more frequenly on the machines with a higher number of users connected.
I have switched one of the servers (the one with the highest load) to use Jan Dubiec's MPPC/MPPE patches for ppp 2.4.3 and Kernel 2.6.9. We'll see how that goes.
Any information I can provide or something I can do on my end to better trace this down?
Ray
Just a follow-up on this. Things seem to run better (no crashes yet after two days of runtime) after booting with nosmp noapic.
Any issues with the MPPE or CryptoAPI code that might only be triggered when running in SMP mode?
Ray
Quoted 56 lines · Ray Van Dolson
Quoted 6 lines · Matt Domsch
These messages I have gotten for years. They're debugging messages, as a result of:
register_netdevice(alloc_divert_blk( (sees it's not an ethernet device, ) printk's the message )So I believe they're harmless.
This is what I was thinking too. I see plenty of these messages that do not result in crashes.
Quoted 26 lines
Quoted 12 lines
ksymoops output of problem:
Unable to handle kernel NULL pointer dereference 00000000 *pde = 00000000 Oops: 0000 [#1] CPU: 2EIP: 0060:[<00000000>] Not tainted VLI
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010286 (2.6.9)
eax:ed13b000ebx:d1d0a000ecx:c029e9deedx:f795ef40
esi:d1d0a000edi: 00000000 ebp:e2f30080esp:d2b0dea0
ds: 007b es: 007b ss: 0068Quoted 10 lines
Stack: c02a205a ed13b000 00000000 c02a122c d1d0a000 13208a2e c040956f d1d0a000 d1d0a00c e2f30080 00000000 c029cda9 d1d0a000 e2f30080 00000000 c01552cd e2f30080 00000010 00000004 00000004 c0166aa0 e2f30080 00000000 00000000 Call Trace: [<c02a205a>] pty_chars_in_buffer+0x2c/0x49 [<c02a122c>] normal_poll+0xed/0x150 [<c040956f>] schedule_timeout+0x75/0xbf [<c029cda9>] tty_poll+0xa0/0xb0 [<c01552cd>] fget+0x49/0x5e [<c0166aa0>] do_select+0x269/0x2c6 [<c0166691>] __pollwait+0x0/0xc7 [<c0166dd5>] sys_select+0x2b3/0x4c6 [<c0105971>] sysenter_past_esp+0x52/0x71 Code: Bad EIP value.It looks like
pty_chars_in_buffer()dereferenced a NULL function pointer, but I don't see how that can be, the one deference is tested for NULL before doing so.Quoted 6 lines
I can't rule out the
ppp_mppecode, but I haven't seen this crash before myself. Does this happen on simlar systems that aren't running poptop?Thanks,
MattFortunately we use the DL140 server pretty widely here. None of the non-poptop servers exhibit this problem. Every single one of them that has been used as a Poptop server however does exhibit this issue. It seems to happen more frequenly on the machines with a higher number of users connected.
I have switched one of the servers (the one with the highest load) to use Jan Dubiec's MPPC/MPPE patches for ppp 2.4.3 and Kernel 2.6.9. We'll see how that goes.
Any information I can provide or something I can do on my end to better trace this down?
Ray
Quoted 26 lines · Matt Domsch
Quoted 12 lines
ksymoops output of problem:
Unable to handle kernel NULL pointer dereference 00000000 *pde = 00000000 Oops: 0000 [#1] CPU: 2EIP: 0060:[<00000000>] Not tainted VLI
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010286 (2.6.9)
eax:ed13b000ebx:d1d0a000ecx:c029e9deedx:f795ef40
esi:d1d0a000edi: 00000000 ebp:e2f30080esp:d2b0dea0
ds: 007b es: 007b ss: 0068Quoted 10 lines
Stack: c02a205a ed13b000 00000000 c02a122c d1d0a000 13208a2e c040956f d1d0a000 d1d0a00c e2f30080 00000000 c029cda9 d1d0a000 e2f30080 00000000 c01552cd e2f30080 00000010 00000004 00000004 c0166aa0 e2f30080 00000000 00000000 Call Trace: [<c02a205a>] pty_chars_in_buffer+0x2c/0x49 [<c02a122c>] normal_poll+0xed/0x150 [<c040956f>] schedule_timeout+0x75/0xbf [<c029cda9>] tty_poll+0xa0/0xb0 [<c01552cd>] fget+0x49/0x5e [<c0166aa0>] do_select+0x269/0x2c6 [<c0166691>] __pollwait+0x0/0xc7 [<c0166dd5>] sys_select+0x2b3/0x4c6 [<c0105971>] sysenter_past_esp+0x52/0x71 Code: Bad EIP value.It looks like
pty_chars_in_buffer()dereferenced a NULL function pointer, but I don't see how that can be, the one deference is tested for NULL before doing so.
Quoted 5 lines
I can't rule out the
ppp_mppecode, but I haven't seen this crash before myself. Does this happen on simlar systems that aren't running poptop?Thanks,
Matt
Yet another follow-up on this. After completely disabling SMP/Hyperthreading on these systems, we have had zero NULL pointer dereferences. Obviously can't say for sure that this points to problems with the MPPE module--the only true test would be to allow our customers to connect without MPPE and reenable SMP/HT. Obviously not an option in a live environment. :)
Guess we just won't be using SMP/HT on our busier systems!
Ray