linux 2.2.19pre and thttpd (VM-global problem?)

Hi all,

I'm expiriencing a problem with thttpd web server (www.acme.com/software/thttpd) on recent linux 2.2 kernels with Andrea's VM-global patches. Without the patch server runs normally with its usual dose of complaints on the linux platform (it's being developed on BSD afaik), but with the patch it runs ok for about three days (depends on traffic i guess), then enters into some state where it reports 'out of memory' for every larger file (>1Mb) it starts serving and dies. When it comes back up it dies again whitin 10 seconds. As this is not happening on a stock kernel and the restart of the server itself has no efect, i conclude it has to be something there in the VM-global that thttpd doesnt really like. As the VM-global seems to be the only cure for the VM_do_try_to_free_pages problem, which is an issue for me too, i'd really like to hear some official words on this before 2.2.19 comes out with VM-global ... and while i'm at it, can we expect ide patch in 2.2.19 too?

Pegasus
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Before the -7 revision the VM-global was sharing a memory corruption bug with vanilla 2.2.x. VM-global was incidentally hiding such bug extremely well though (but with heavy load and using LVM snapshotting I triggered it even under VM-global-6, so then I spotted such last leftover too and the strict fix for such MM corruption bug got merged into 2.2.18pre2x too, and at the same time I released VM-global-7)

I need to know exactly which kernel sources you were using (and also the compiler of course ;)

out of memory looks an userspace message, so it looks like malloc request was too large (it could happen because of an userspace corruption in the 'size' parameter for example).

Even if you are using VM-global-7 with only this information it's non obvious that VM-global-7 (the one included 19pre3) is the source of your problem.

So if you were using the -7 revision please check the logs and the console for Oopses and try to strace the daemon to see how it dies. It's also not clear what it means that restarting the server itself has no effect, as far I can tell it could even be an userspace issue (some rc script that doesn't cope with an unclean exit of the server?). And besides the webserver, how does the rest of the system behaves when the problem appears? There are too many possibilities, we need to restrict our search.

You were using -6 revision as first thing you can try to reproduce with 2.2.19pre3 after checking the compiler. If you send me your .config I can send you a binary image to test.

Thanks for the feedback,
Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

I downloaded the sources right now to see what 'out of memory' means. I assume you were using version 2.20b.

My guess was right, it's not possible to know where it came from though:

andrea@athlon:~/devel/thttpd-2.20b > grep 'out of' *.c fdwatch.c:** or 0 if the timeout expired, or -1 on errors. A timeout of INFTIM means

libhttpd.c:     syslog( LOG_CRIT, "out of memory" );
libhttpd.c:         syslog( LOG_CRIT, "out of memory" );
libhttpd.c:         syslog( LOG_CRIT, "out of memory" );
libhttpd.c:     syslog( LOG_CRIT, "out of memory" );
libhttpd.c:         syslog( LOG_CRIT, "out of memory" );
libhttpd.c:         syslog( LOG_CRIT, "out of memory" );
libhttpd.c:     syslog( LOG_ERR, "out of memory" );
libhttpd.c:     ** since it's impossible to get out of the tree.  However, we still
libhttpd.c:                     syslog( LOG_ERR, "out of memory" );
libhttpd.c:     syslog( LOG_ERR, "out of memory" );
thttpd.c:       syslog( LOG_CRIT, "out of memory" );
thttpd.c:       syslog( LOG_CRIT, "out of memory" );
thttpd.c:       (void) fprintf( stderr, "%s: out of memory\n", argv0 );
thttpd.c:               syslog( LOG_CRIT, "out of memory" );
thttpd.c:               (void) fprintf( stderr, "out of memory\n" );
thttpd.c:           syslog( LOG_CRIT, "out of memory" );
thttpd.c:           (void) fprintf( stderr, "out of memory\n" );
thttpd.c:               syslog( LOG_CRIT, "out of memory" );

andrea@athlon:~/devel/thttpd-2.20b >

But luckily it always happens after some kind of memory allocation, so it's going to be the overcommit check of mmap that is complaining as I guessed (assuming glibc isn't buggy in your system).

I was thinking I have a minor fix (in aa patchkit) that can help if the problem happens while you are quite near to use all the memory (swap included) and you have very low level of buffercache and pagecache. But I don't think you were near the out of memory condition, right? (can you monitor via vmstat 1 >log to be sure?) It addresses more a correctness issue than a real life problem. And if it was a real life problem then you can workaround it with `echo 1

recompile the kernel (but again: I don't think this is the problem, setting overcommit_memory to 1 would probably only temporarly hide the problem, you would probably get the task killed by the kernel some time later anyways)

BTW, after checking the sources we at least know it wasn't due memory fragmentation (fork complains with a different message).

Andrea

PS. I'm very suprised thttpd isn't threaded, it should really be threaded at
least on the x86 family to run fast.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

This is one of the main thttpd design points: run in a select() loop. Since it is intended for mainly static workloads, it performs quite well...

--
Petru Paler, mailto:[email protected]
http://www.ppetru.net - ICQ: 41817235
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

At the moment it is 2.2.18 + all pathces that can be found at www.linuxraid.org and ide patch 2.2.18.1221, but i first spotted this problem on a similary configured 2.2.17 with VM-global patch 3. gcc --version returns egcs-2.91.66. The box is an old p200mmx with 128mb ram running redhat 6.0 with lots of patches and customisations ... glibc rpm packet is glibc-2.1.3-21.

Sample vmstat output:
procs memory swap io
system cpu

 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us 
sy  id
 0  0  0   5804   2716   2624  92188   3   1    50     9  446   441  
7   3  90

Straceing the server shows that it indeed quits after a munmap:

[snip]
select(9, [5], [6 7 8], NULL, {72, 164000}) = 1 (out [7], left {71, 960000})
gettimeofday({978077877, 462325}, NULL) = 0
write(7, "\364\362\222\210\311\336\323\206"..., 243498437) = 20272 select(9, [5], [6 7 8], NULL, {71, 944000}) = 1 (in [5], left {71, 770000})
gettimeofday({978077877, 645302}, NULL) = 0
accept(5, {sin_family=AF_INET, sin_port=htons(1444), sin_addr=inet_addr("24.11.189.33")}, [16]) = 9

fcntl(9, F_SETFD, FD_CLOEXEC)           = 0
fcntl(9, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(9, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
brk(0x8079000)                          = 0x8078000
brk(0x8079000)                          = 0x8078000
time([978077877])                       = 978077877
getpid()                                = 22024

send(3, "<26>Dec 29 09:17:57 thttpd[22024"..., 68, 0) = 68
munmap(0x125000, 4096) = 0
_exit(1) = ?

Hmm ... this looks like it dies when another request comes in when one download is already in progress ... let me try again:
[snip]
select(7, [5], [6], NULL, {98, 652000}) = 1 (out [6], left {98, 560000}) gettimeofday({978078108, 982991}, NULL) = 0
write(6, "O\327\372R\220\354r\317[\231T\0\340"..., 246365477) = 15928 select(7, [5], [6], NULL, {98, 547000}) = 1 (in [5], left {98, 210000}) gettimeofday({978078109, 324744}, NULL) = 0
accept(5, {sin_family=AF_INET, sin_port=htons(3687), sin_addr=inet_addr("62.76.36.242")}, [16]) = 7

fcntl(7, F_SETFD, FD_CLOEXEC)           = 0
fcntl(7, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(7, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
brk(0x8078000)                          = 0x8077000
brk(0x8078000)                          = 0x8077000
time([978078109])                       = 978078109
getpid()                                = 22061

send(3, "<27>Dec 29 09:21:49 thttpd[22061"..., 69, 0) = 69
munmap(0x125000, 4096) = 0
_exit(1) = ?

Am i right? My IP was 213.143.68.212 in both runs.

I just tried the 'echo 1 >/proc/sys/vm/overcommit_memory' trick ... and no, it's still dieing at about the same rate.

My .config on 2.2.18-cdhs :

#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y

#
# Processor type and features
#
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
CONFIG_M586TSC=y
# CONFIG_M686 is not set
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_TSC=y
# CONFIG_MICROCODE is not set
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m
CONFIG_1GB=y
# CONFIG_2GB is not set
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
# CONFIG_SMP is not set

#
# Loadable module support
#
CONFIG_MODULES=y
CONFIG_MODVERSIONS=y
CONFIG_KMOD=y
#
# General setup
#
CONFIG_NET=y
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_QUIRKS=y
# CONFIG_PCI_OPTIMIZE is not set
CONFIG_PCI_OLD_PROC=y
# CONFIG_MCA is not set
# CONFIG_VISWS is not set
CONFIG_SYSVIPC=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_SYSCTL=y
CONFIG_BINFMT_AOUT=m
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_MISC=m
# CONFIG_BINFMT_JAVA is not set
# CONFIG_PARPORT is not set
# CONFIG_APM is not set
# CONFIG_TOSHIBA is not set

#
# Plug and Play support
#
# CONFIG_PNP is not set

#
# Block devices
#
CONFIG_BLK_DEV_FD=m
CONFIG_BLK_DEV_IDE=y
# CONFIG_BLK_DEV_HD_IDE is not set
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
# CONFIG_IDEDISK_STROKE is not set
CONFIG_BLK_DEV_IDECD=m
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_BLK_DEV_IDEFLOPPY is not set
# CONFIG_BLK_DEV_IDESCSI is not set
# CONFIG_IDE_TASK_IOCTL is not set
# CONFIG_PKT_TASK_IOCTL is not set
# CONFIG_IDE_TASK_IOCTL_DEBUG is not set
# CONFIG_BLK_DEV_CMD640 is not set
# CONFIG_BLK_DEV_RZ1000 is not set
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
CONFIG_BLK_DEV_IDEDMA=y
CONFIG_IDEDMA_AUTO=y
# CONFIG_IDEDMA_NEW_DRIVE_LISTINGS is not set
CONFIG_IDEDMA_PCI_EXPERIMENTAL=y
# CONFIG_IDEDMA_PCI_WIP is not set
CONFIG_BLK_DEV_OFFBOARD=y
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
# CONFIG_BLK_DEV_AMD7409 is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_CY82C693 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_HPT34X is not set
CONFIG_BLK_DEV_HPT366=y
CONFIG_BLK_DEV_PIIX=y
CONFIG_PIIX_TUNING=y
# CONFIG_BLK_DEV_OPTI621 is not set
# CONFIG_BLK_DEV_PDC202XX is not set
# CONFIG_BLK_DEV_SIS5513 is not set
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_OSB4 is not set
# CONFIG_BLK_DEV_TRM290 is not set
# CONFIG_BLK_DEV_VIA82CXXX is not set
# CONFIG_IDE_CHIPSETS is not set
# CONFIG_IDEDMA_IVB is not set
CONFIG_BLK_DEV_LOOP=m
# CONFIG_BLK_DEV_NBD is not set
CONFIG_BLK_DEV_MD=y
CONFIG_AUTODETECT_RAID=y
CONFIG_MD_LINEAR=y
CONFIG_MD_STRIPED=y
CONFIG_MD_MIRRORING=y
CONFIG_MD_RAID5=y
# CONFIG_MD_TRANSLUCENT is not set
# CONFIG_MD_HSM is not set
CONFIG_MD_BOOT=y
CONFIG_BLK_DEV_RAM=m
CONFIG_BLK_DEV_RAM_SIZE=4096
# CONFIG_BLK_DEV_XD is not set
# CONFIG_BLK_DEV_DAC960 is not set
CONFIG_PARIDE_PARPORT=y
# CONFIG_PARIDE is not set
CONFIG_BLK_DEV_IDE_MODES=y
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_HD is not set

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_NETLINK=y
CONFIG_RTNETLINK=y
CONFIG_NETLINK_DEV=y
CONFIG_FIREWALL=y
CONFIG_FILTER=y
CONFIG_UNIX=y
CONFIG_INET=y
# CONFIG_IP_MULTICAST is not set
# CONFIG_IP_ADVANCED_ROUTER is not set
# CONFIG_IP_PNP is not set
CONFIG_IP_FIREWALL=y
CONFIG_IP_FIREWALL_NETLINK=y
CONFIG_NETLINK_DEV=y
# CONFIG_IP_TRANSPARENT_PROXY is not set
# CONFIG_IP_MASQUERADE is not set
# CONFIG_IP_ROUTER is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE=m
CONFIG_IP_ALIAS=y
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
# CONFIG_INET_RARP is not set
CONFIG_SKB_LARGE=y
# CONFIG_IPV6 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_BRIDGE is not set
# CONFIG_NET_DIVERT is not set
# CONFIG_LLC is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_NET_FASTROUTE is not set
# CONFIG_NET_HW_FLOWCONTROL is not set
# CONFIG_CPU_IS_SLOW is not set

#
# QoS and/or fair queueing
#
# CONFIG_NET_SCHED is not set

#
# Telephony Support
#
# CONFIG_PHONE is not set
# CONFIG_PHONE_IXJ is not set

#
# SCSI support
#
# CONFIG_SCSI is not set

#
# I2O device support
#
# CONFIG_I2O is not set
# CONFIG_I2O_PCI is not set
# CONFIG_I2O_BLOCK is not set
# CONFIG_I2O_SCSI is not set

#
# Network device support
#
CONFIG_NETDEVICES=y

#
# ARCnet devices
#
# CONFIG_ARCNET is not set
CONFIG_DUMMY=m
CONFIG_BONDING=m
# CONFIG_EQUALIZER is not set
# CONFIG_ETHERTAP is not set
# CONFIG_NET_SB1000 is not set

#
# Ethernet (10 or 100Mbit)
#
CONFIG_NET_ETHERNET=y
CONFIG_NET_VENDOR_3COM=y
# CONFIG_EL1 is not set
# CONFIG_EL2 is not set
# CONFIG_ELPLUS is not set
# CONFIG_EL16 is not set
CONFIG_EL3=m
# CONFIG_3C515 is not set
CONFIG_VORTEX=m
# CONFIG_LANCE is not set
# CONFIG_NET_VENDOR_SMC is not set
# CONFIG_NET_VENDOR_RACAL is not set
# CONFIG_RTL8139 is not set
# CONFIG_RTL8139TOO is not set
# CONFIG_NET_ISA is not set
CONFIG_NET_EISA=y
# CONFIG_PCNET32 is not set
# CONFIG_AC3200 is not set
# CONFIG_APRICOT is not set
# CONFIG_CS89x0 is not set
# CONFIG_DM9102 is not set
# CONFIG_DE4X5 is not set
# CONFIG_DEC_ELCP is not set
# CONFIG_DEC_ELCP_OLD is not set
# CONFIG_DGRS is not set
CONFIG_EEXPRESS_PRO100=m
# CONFIG_LNE390 is not set
# CONFIG_NE3210 is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_TLAN is not set
# CONFIG_VIA_RHINE is not set
# CONFIG_SIS900 is not set
# CONFIG_ES3210 is not set
# CONFIG_EPIC100 is not set
# CONFIG_ZNET is not set
# CONFIG_NET_POCKET is not set

#
# Ethernet (1000 Mbit)
#
# CONFIG_ACENIC is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_SK98LIN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_NET_RADIO is not set

#
# Token ring devices
#
# CONFIG_TR is not set
# CONFIG_NET_FC is not set
# CONFIG_RCPCI is not set
# CONFIG_SHAPER is not set

#
# Wan interfaces
#
# CONFIG_HOSTESS_SV11 is not set
# CONFIG_COSA is not set
# CONFIG_SEALEVEL_4021 is not set
# CONFIG_SYNCLINK_SYNCPPP is not set
# CONFIG_LANMEDIA is not set
# CONFIG_COMX is not set
# CONFIG_HDLC is not set
# CONFIG_DLCI is not set
# CONFIG_XPEED is not set
# CONFIG_SBNI is not set

#
# Amateur Radio support
#
# CONFIG_HAMRADIO is not set

#
# IrDA (infrared) support
#
# CONFIG_IRDA is not set

#
# ISDN subsystem
#
# CONFIG_ISDN is not set

#
# Old CD-ROM drivers (not SCSI, not IDE)
#
# CONFIG_CD_NO_IDESCSI is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_SERIAL=y
# CONFIG_SERIAL_CONSOLE is not set
# CONFIG_SERIAL_EXTENDED is not set
# CONFIG_SERIAL_NONSTANDARD is not set
CONFIG_UNIX98_PTYS=y
CONFIG_UNIX98_PTY_COUNT=2048
CONFIG_MOUSE=y

#
# Mice
#
# CONFIG_ATIXL_BUSMOUSE is not set
# CONFIG_BUSMOUSE is not set
# CONFIG_MS_BUSMOUSE is not set
CONFIG_PSMOUSE=y
# CONFIG_82C710_MOUSE is not set
# CONFIG_PC110_PAD is not set

#
# Joysticks
#
# CONFIG_JOYSTICK is not set
# CONFIG_QIC02_TAPE is not set
# CONFIG_WATCHDOG is not set
CONFIG_NVRAM=m
CONFIG_RTC=y
# CONFIG_INTEL_RNG is not set
# CONFIG_AGP is not set
# CONFIG_DRM is not set

#
# Video For Linux
#
# CONFIG_VIDEO_DEV is not set
# CONFIG_DTLK is not set

#
# Ftape, the floppy tape device driver
#
# CONFIG_FTAPE is not set

#
# USB support
#
# CONFIG_USB is not set

#
# Filesystems
#
CONFIG_QUOTA=y
# CONFIG_AUTOFS_FS is not set
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
CONFIG_FAT_FS=m
# CONFIG_MSDOS_FS is not set
# CONFIG_UMSDOS_FS is not set
CONFIG_VFAT_FS=m
CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
# CONFIG_MINIX_FS is not set
# CONFIG_NTFS_FS is not set
# CONFIG_HPFS_FS is not set
CONFIG_PROC_FS=y
CONFIG_DEVPTS_FS=y
# CONFIG_QNX4FS_FS is not set
# CONFIG_ROMFS_FS is not set
CONFIG_EXT2_FS=y
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
# CONFIG_EFS_FS is not set

#
# Network File Systems
#
# CONFIG_CODA_FS is not set
CONFIG_NFS_FS=m
# CONFIG_NFS_V3 is not set
CONFIG_NFSD=m
# CONFIG_NFSD_V3 is not set
# CONFIG_NFSD_TCP is not set
CONFIG_SUNRPC=m
CONFIG_LOCKD=m
# CONFIG_SMB_FS is not set
# CONFIG_NCP_FS is not set

#
# Partition Types
#
# CONFIG_BSD_DISKLABEL is not set
# CONFIG_MAC_PARTITION is not set
# CONFIG_SMD_DISKLABEL is not set
# CONFIG_SOLARIS_X86_PARTITION is not set
# CONFIG_UNIXWARE_DISKLABEL is not set
CONFIG_NLS=y

#
# Native Language Support
#
CONFIG_NLS_DEFAULT="cp437"
# CONFIG_NLS_CODEPAGE_437 is not set
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_ISO8859_1 is not set
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
# CONFIG_NLS_KOI8_R is not set

#
# Console drivers
#
CONFIG_VGA_CONSOLE=y
CONFIG_VIDEO_SELECT=y
# CONFIG_MDA_CONSOLE is not set
# CONFIG_FB is not set

#
# Sound
#
# CONFIG_SOUND is not set

#
# Security options
#
CONFIG_SECURE_STACK=y
CONFIG_SECURE_STACK_SMART=y
CONFIG_SECURE_LINK=y
CONFIG_SECURE_FIFO=y
CONFIG_SECURE_PROC=y
CONFIG_SECURE_FD_0_1_2=y
CONFIG_SECURE_RLIMIT_NPROC=y
# CONFIG_SECURE_SHM is not set

#
# Kernel hacking
#
CONFIG_MAGIC_SYSRQ=y

Pegasus
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

If you applied raid-0.90 and VM_global did you sort out the fact both of them try to use the same free bit for extra buffer flags ?
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Good. Can you try to reproduce with 2.2.19pre3? (if you absolutely need raid 0.90 you can try again with 2.2.19pre3aa3 after backing out 04_wake-one-3 that introduces a deadlocks spotted by Chris and that I'm debugging right now)

	num_ready = fdwatch( tmr_mstimeout( &tv ) );
	if ( num_ready < 0 )
	    {
	    if ( errno == EINTR )
		continue;       /* try again */
	    syslog( LOG_ERR, "fdwatch - %m" );
	    exit( 1 );
	    }

	(void) gettimeofday( &tv, (struct timezone*) 0 );

httpd_get_conn( httpd_server* hs, int listen_fd, httpd_conn* hc ) [..]

    hc->conn_fd = accept( listen_fd, &sa.sa, &sz );

    (void) fcntl( hc->conn_fd, F_SETFD, 1 );

	flags = fcntl( c->hc->conn_fd, F_GETFL, 0 );

	else if ( fcntl( c->hc->conn_fd, F_SETFL, flags | O_NDELAY ) < 0 )

for some unknown reason it doesn't notice it has to handle the new connection.

    shut_down();
    syslog( LOG_NOTICE, "exiting" );

However according to the latest sources (2.20b) shut_down() should at least call gettimeofday at once and that's not the case for you:

shut_down( void )

    {
    int cnum;
    struct timeval tv;

#ifdef STATS_TIME
    show_stats( JunkClientData, (struct timeval*) 0 );
#endif /* STATS_TIME */
    (void) gettimeofday( &tv, (struct timezone*) 0 );

So you aren't using thttpd version 2.20b (or it's not the vanilla source).

Well, this would look like an userspace bug, but I understand it's strange that it works well for three days.... Can you try to upgrade thttpd to the latest version compiled from vanilla sources?

But regardless of the userspace upgrade, I still recommend to upgrade to 2.2.19pre3 or 2.2.19pre3aa3 minus 04_wake-one-3 (once I'll fix wake-one-3 I'll release -4 revision and pre3aa4).

Thanks,
Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

It can't scale in SMP.

Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

No one said it does, but it works nicely on UP.

--
Petru Paler, mailto:[email protected]
http://www.ppetru.net - ICQ: 41817235
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

What ?

The TCP stack is threaded, so things like checksum calculation will take advantage of multiple processors - right ?

Thes rest of the work is roughly copying data that isn't already cached from the disk into memory. Well, you have one disk so threads will buy you zero there.

Unless you do blocking I/O on the files or on the sockets, I fail to see how threads could possibly boost the performance on a web server that serves *static*only* pages.

99% of the real work is done by the kernel, so whether you have your user-space application threaded or not should not be an issue - the way that I see it...

Andrea, or someone else, would you care to enlighten me on that ?

(The reason I'm curious is because I'm about a month away from implementing
something that would run high-bandwidth TCP transfers and I'm planning on keeping it single-threaded - unless someone can tell me that's a bad idea)

Thanks,

................................................................
:   [email protected]   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Maybe you got me wrong (my english isnt that good): I said that it does not scale on SMP, but it works just fine on UP.

Wrong. "Threaded" TCP/IP stack -> fine grained locking, not "multiple threads".

They do boost performance on SMP (because you can have N (N=nr. of CPUs) threads serving data).

Keep it single threaded if you run on UP only...

--
Petru Paler, mailto:[email protected]
http://www.ppetru.net - ICQ: 41817235
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Your cgi will keep the other CPU occupied, or run two of them. thttpd has superb scaling properties compared to say apache.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Depends on memory bandwidth, caches, locking overhead and a million other issues.

Theory - SMP needs threading
Practice is generally a little different

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

I think with 8 CPUs and 8 NICs (usual benchmark setup) you want more than 1 cpu serving static data and it should be more efficient if it's threaded and sleeping in accept() instead of running eight of them (starting from sharing tlb entries and avoiding flushes probably without the need of CPU binding).

Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

BTW, it could as well use TCP_CORK + sendfile that will become truly zero copy eventually.

Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

That's a good benchmark setup when the benchmark requires a single machine.

In the real world it often makes a lot of sense though to use a cluster of cheap single CPU machines behind a load balancer (gives you better fault tolerance and is likely cheaper)

For these thttpd is a nice web server.

-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Ok - my point was (and maybe I'm just misunderstanding something):

Multiple threads do send() etc. -> this gets serialized by the kernel anyway, as TCP checksums are done in the kernel as well as the NIC
driving etc. If the TCP layer cannot parallelize checksum calculation.

One thread does many send()'s -> this also gets serialized by the kernel.

Or: one thread vs. many threads reading from the disk -> makes no difference since there is *one* buffer layer, one driver, one disk etc. etc.

My question is: Does it change anything in the way the TCP stack
can crank out packets, if I change the number of threads in user-space making the system calls ?

Does the fine-grained locking allow multiple user-space threads to actually have TCP checksums etc. done on multiple CPUs concurrently, and is it true that this will not happen if I have one user-space thread doing non-blocking send() calls ?

Thanks,

................................................................
:   [email protected]   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

hey, nobody sane runs an 8 CPU box with 8 NICs for a production webserver. 8 single CPU boxes, or 4 dual boxes behind a load balancer. now that's more common, more scalable, more robust. :)

oh yeah they all run perl, java, or php too :) i've seen sites with more than 100 dynamic front-ends and a pair of 350Mhz x86 boxes in the corner handling all the static needs (running apache even!). a pair only 'cause of redundancy reasons, not because of load reasons.

100Mbits/s of *transit* internet bandwidth costs US$75000 per month in typical datacenters btw. obviously there's cheaper bandwidth if you push out to the edges of the net, into the central offices.

this isn't to say that nobody will ever want phat bandwidth on a single webserver... but i'd say linux and most everyone else is at least 3 years ahead of the long-haul networks in terms of ability to pump data. the true value of zero-copy TCP will be more apparent in the upcoming age of 1 Mbit/s video thanks to MPEG4. i'd expect to see the cable/dsl conglomerates start doing this in their central offices soon.

the RAM and CPU hungry dynamic content languages tend to dictate the sheer quantity of machines required to handle even small web volumes -- folks quickly exceed the reasonably priced SMP systems available. the cost per CPU, and per Mb RAM are the limiting factors (these go up quickly when you put all CPUs and RAM on the bus). that plus the desire for reliability / lack of a single failure point mean that web development tools have grown up expecting to be on a distributed network of nodes rather than on a large SMP system.

the 8 CPU monsters run stuff like NFS/filesystems and oracle. some day someone will build a horizontally scalable database which works, with a reasonable licensing policy and then we'll see even more value in commodity hardware/networking. this is an area i'm currently quite interested in, pointers to research / interesting projects appreciated. i know about globalfilesystem.org, veritas clustered fs, oracle parallel server, and IBM DB2 EEE already -- none are quite to the point where i'd use them to scale something across 100 nodes yet though.

-dean

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

				    ^^^^^^^^^^^^^^^^^^^^^

and it also provides high avaibility that is mandatory in a setup that needs high performance anyways.

and zope+python as well indeed.

Exactly. I totally agree with you (everybody who talked with me about those issues or attended my last two talks can confirm I agree ;).

But you're talking about real world. I was only talking about benchmarks. Don't complain me if usual benchmark setup is done with one server N-way SMP + N-Gbit-NICs (infact I can't run any usual webserving benchmark at home).

Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/