khttpd newbie problem

I'm having an oops with khttpd on an embedded 2.4.17 ppc405 system, so I thought I'd try it out on my pc. But I can't get khttpd to serve any requests.

I built khttpd into the kernel with vanilla 2.4.17smp on Intel on Red Hat 7.2, then turned it on as follows:

echo /home/dank/stress > /proc/sys/net/khttpd/documentroot echo 80 > /proc/sys/net/khttpd/serverport
echo 8000 > /proc/sys/net/khttpd/maxconnect
echo 1 > /proc/sys/net/khttpd/start

I also made sure there was an index.html in /home/dank/stress, turned off the firewall, did /etc/init.c/ipchains restart, and made sure netstat reported port 80 as listening.

But... when I try to fetch http://localhost/index.html,
it just sits there. Likewise, when I telnet to port 80, even from a different machine, it just accepts bytes forever; no matter what I type, it just echoes the bytes right back at me.

If I do
echo 1 > /proc/sys/net/khttpd/stop
port 80 stops listening, and any open connections are closed.

I must be doing something silly... surely khttpd works? Is it because I'm running SMP, perhaps?
- Dan

Hi Dan,

Any reason for not using tux? Its been tested heavily on ppc64, the same patches should work on ppc32.

Anton

That's an excellent suggestion. It certainly seems that khttpd is no longer production quality (if it ever was), and tux is.

I'm on an embedded system, so if tux is much larger, I'll be annoyed; but the system does have 64 MB, so it's not *that* cramped. And working is much better than crashing.
- Dan

Here are two sets of patches, one for the tux userspace utilities and one for the kernel - apply it after the tux patch.

I'll push the relevant changes to Ingo and Paulus.

Anton

kernel patch:

diff -ru --exclude-from=/home/anton/ppc64/exclude linux-2.4_tux/arch/ppc/kernel/misc.S linux-2.4_tux_work/arch/ppc/kernel/misc.S
@@ -1162,6 +1162,30 @@
 	.long sys_mincore
 	.long sys_gettid
 	.long sys_tkill
+	.long sys_ni_syscall	/* sys_setxattr */
+	.long sys_ni_syscall	/* sys_lsetxattr */    /* 210 */
+	.long sys_ni_syscall	/* sys_fsetxattr */
+	.long sys_ni_syscall	/* sys_getxattr */
+	.long sys_ni_syscall	/* sys_lgetxattr */
+	.long sys_ni_syscall	/* sys_fgetxattr */
+	.long sys_ni_syscall	/* sys_listxattr */    /* 215 */
+	.long sys_ni_syscall	/* sys_llistxattr */
+	.long sys_ni_syscall	/* sys_flistxattr */
+	.long sys_ni_syscall	/* sys_removexattr */
+	.long sys_ni_syscall	/* sys_lremovexattr */
+	.long sys_ni_syscall	/* sys_fremovexattr */  /* 220 */
+	.long sys_ni_syscall	/* sys_futex */
+	.long sys_ni_syscall	/* sys_sched_setaffinity */
+	.long sys_ni_syscall	/* sys_sched_getaffinity */
+#ifdef CONFIG_TUX
+	.long __sys_tux
+#else
+# ifdef CONFIG_TUX_MODULE
+	.long sys_tux
+# else
+	.long sys_ni_syscall
+# endif
+#endif
 	.rept NR_syscalls-(.-sys_call_table)/4
 		.long sys_ni_syscall
 	.endr

diff -ru --exclude-from=/home/anton/ppc64/exclude linux-2.4_tux/include/asm-ppc/fcntl.h linux-2.4_tux_work/include/asm-ppc/fcntl.h
@@ -23,6 +23,7 @@
 #define O_NOFOLLOW      0100000	/* don't follow links */
 #define O_LARGEFILE     0200000
 #define O_DIRECT	0400000	/* direct disk access hint */
+#define O_ATOMICLOOKUP 01000000	/* do atomic file lookup */
 
 #define F_DUPFD		0	/* dup */
 #define F_GETFD		1	/* get close_on_exec */

diff -ru --exclude-from=/home/anton/ppc64/exclude linux-2.4_tux/include/asm-ppc/unistd.h linux-2.4_tux_work/include/asm-ppc/unistd.h
@@ -216,6 +216,22 @@
 #define __NR_mincore		206
 #define __NR_gettid		207
 #define __NR_tkill		208
+#define __NR_setxattr		209
+#define __NR_lsetxattr		210
+#define __NR_fsetxattr		211
+#define __NR_getxattr		212
+#define __NR_lgetxattr		213
+#define __NR_fgetxattr		214
+#define __NR_listxattr		215
+#define __NR_llistxattr		216
+#define __NR_flistxattr		217
+#define __NR_removexattr	218
+#define __NR_lremovexattr	219
+#define __NR_fremovexattr	220
+#define __NR_futex		221
+#define __NR_sched_setaffinity  222
+#define __NR_sched_getaffinity  223
+#define __NR_tux		224
 
 #define __NR(n)	#n
 

diff -ru --exclude-from=/home/anton/ppc64/exclude linux-2.4_tux/net/tux/accept.c linux-2.4_tux_work/net/tux/accept.c
@@ -64,7 +64,13 @@
 #define IP(n) ((unsigned char *)&addr)[n]
 	err = sock->ops->bind(sock, (struct sockaddr*)&sin, sizeof(sin));
 	if (err < 0) {
-		printk(KERN_ERR "TUX: error %d binding socket. This means that probably some other process is (or was a short time ago) using addr %s://%d.%d.%d.%d:%d.\n", err, proto->name, IP(3), IP(2), IP(1), IP(0), port);
+		printk(KERN_ERR "TUX: error %d binding socket. This means that probably some other process is (or was a short time ago) using addr %s://%d.%d.%d.%d:%d.\n", err, proto->name,
+#ifdef __BIG_ENDIAN
+			IP(0), IP(1), IP(2), IP(3), 
+#else
+			IP(3), IP(2), IP(1), IP(0), 
+#endif
+			port);
 		goto err;
 	}
 

@@ -87,7 +93,13 @@
 	}
 
 	printk(KERN_NOTICE "TUX: thread %d listens on %s://%d.%d.%d.%d:%d.\n",
-		nr, proto->name, IP(3), IP(2), IP(1), IP(0), port);
+		nr, proto->name, 
+#ifdef __BIG_ENDIAN
+		IP(0), IP(1), IP(2), IP(3), 
+#else
+		IP(3), IP(2), IP(1), IP(0), 
+#endif
+		port);
 	return sock;
 
 err:

diff -ru --exclude-from=/home/anton/ppc64/exclude linux-2.4_tux/net/tux/proc.c linux-2.4_tux_work/net/tux/proc.c
@@ -1005,7 +1005,12 @@
 #define IP(n) ((unsigned char *)&listen->ip)[n]
 
 	return sprintf (page, "%s://%u.%u.%u.%u:%hu\n", listen->proto->name,
-		IP(3), IP(2), IP(1), IP(0), listen->port);
+#ifdef __BIG_ENDIAN
+		IP(0), IP(1), IP(2), IP(3), 
+#else
+		IP(3), IP(2), IP(1), IP(0), 
+#endif
+		listen->port);
 }
 
 static int listen_write_proc (struct file *file, const char *buffer,

diff -ru --exclude-from=/home/anton/ppc64/exclude linux-2.4_tux/net/tux/proto_ftp.c linux-2.4_tux_work/net/tux/proto_ftp.c
@@ -602,8 +602,10 @@
 	}
 #if CONFIG_TUX_DEBUG
 	req->bytes_expected = 0;
+#ifndef __powerpc__
 	if (tux_TDprintk)
 		show_stack(NULL);
+#endif
 #endif
 	req->in_file.f_pos = 0;
 	TDprintk("zapping, data sock state: %d (err: %d, urg: %d)\n",

tux userspace patch:

diff -ru tux-2.2.5/tux.c tux-2.2.5_work/tux.c
@@ -33,7 +33,13 @@
 }
 
 #else
+
+#ifdef __powerpc__
+#define __NR_tux 224
+#else
 #define __NR_tux 222
+#endif
+
 _syscall2 (int, tux, unsigned int, action, user_req_t *, req)
 #endif
 

diff -ru tux-2.2.5/tux2w3c.c tux-2.2.5_work/tux2w3c.c
@@ -35,6 +35,23 @@
 #ifdef __ia64__
 #define O_LARGEFILE    0100000
 #endif
+
+#ifdef __powerpc__
+#define O_LARGEFILE 0200000
+#endif
+
+#ifdef _BIG_ENDIAN
+#define LOG_ID1 0x1111beef
+#define LOG_ID2 0x2222beef
+#define LOG_ID3 0x1112beef
+#define LOG_ID4 0x2223beef
+#else
+#define LOG_ID1 0xefbe1111
+#define LOG_ID2 0xefbe2222
+#define LOG_ID3 0xefbe1211
+#define LOG_ID4 0xefbe2322
+#endif
+
 typedef struct standard_tux_record {
 	__u32 ip_addr;
 	__u32 time;

@@ -147,14 +164,14 @@
 		if (c == EOF)
 			break;
 		id = (id << 8) | c;
-		if ((id != 0xefbe1111) && (id != 0xefbe2222) &&
-				(id != 0xefbe1211) && (id != 0xefbe2322))
+		if ((id != LOG_ID1) && (id != LOG_ID2) &&
+				(id != LOG_ID3) && (id != LOG_ID4))
 			continue;
 		extended_rec = 0;
-		if ((id == 0xefbe2222) || (id == 0xefbe2322))
+		if ((id == LOG_ID2) || (id == LOG_ID4))
 			extended_rec = 1;
 		referer_log = 0;
-		if ((id == 0xefbe2322) || (id == 0xefbe1211))
+		if ((id == LOG_ID4) || (id == LOG_ID3))
 			referer_log = 1;
 		if (!extended_rec) {
 			if (my_read(file, &std_rec, sizeof(std_rec)) != sizeof(std_rec))

On Sat, May 04, 2002 at 10:55:09PM -0700, Dan Kegel wrote:

| Anton Blanchard wrote:
| > > I'm having an oops with khttpd on an embedded 2.4.17 ppc405
| > > system, so I thought I'd try it out on my pc.  But I can't
| > > get khttpd to serve any requests.
| > 
| > Any reason for not using tux? Its been tested heavily on ppc64,
| > the same patches should work on ppc32.
| 
| That's an excellent suggestion.  It certainly seems that khttpd
| is no longer production quality (if it ever was), and tux is.

khttpd is very much production quality on IA32, and has been since 2.4.0-test1. TUX2 is not, however, since under load it enters a 99% CPU busy loop. You may not have enough load to cause TUX2 to do this, and TUX1 may not have this problem.

| I'm on an embedded system, so if tux is much larger, I'll
| be annoyed; but the system does have 64 MB, so it's not *that*
| cramped.  And working is much better than crashing.

khttpd is extremely dependent on alignment and data sizes -- the filename extension handling is deeply unfunny* for example. khttpd most likely has a problem with PPC (endian, etc). Are you applying any other patches that could conflict?

Ken.
[email protected]

* phrase plagiarized from ac

| - Dan
| -
| To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
| the body of a message to [email protected]
| More majordomo info at  http://vger.kernel.org/majordomo-info.html
| Please read the FAQ at  http://www.tux.org/lkml/

It oopses readily on start/stop -- see
http://marc.theaimsgroup.com/?l=linux-kernel&m=102068445316516&w=2
in which DecodeHeader is called with no buffer allocated. Happened quite frequently during my tests. Since my patch I can't get it to oops on x86, but it's still oopsing on ppc405. So perhaps now it's production quality... but it still needs a bit too much babying on stop. (It'd be fairly easy to fix the unreliable way it senses the stop command.)

I checked the tux list and found the post you're taling about:
http://marc.theaimsgroup.com/?l=tux-list&m=101420257421009&w=2
Hmm. Well, at least it doesn't oops :-) Thanks for the warning.

We're running fairly vanilla linuxppc_2_4_devel. It worked quite well with 2.4.2 from Montavista, but developed stability problems when we moved to 2.4.17 (and now 2.4.19-pre8).

I'm trying to put together a minimal test case that shows the crash. Here's the script I run that does it reliably:
#!/bin/msh
echo 1 > /proc/sys/net/khttpd/threads
echo pwd > /proc/sys/net/khttpd/documentroot
echo 80 > /proc/sys/net/khttpd/serverport
echo 8000 > /proc/sys/net/khttpd/maxconnect
while /bin/true; do
echo 1 > /proc/sys/net/khttpd/start
./foo_load duration=7
echo 1 > /proc/sys/net/khttpd/stop
sleep 1
done

where foo_load is a hacked version of http_load (I think) which is fetching a single 128KB file over and over. I tried reproducing this with acme.com's http_load, without success so far.

In the 'aw, for what it's worth' category, here are some of the oopses I've seen on the ppc with 2.4.19-pre8 from the linuxppc_2_4_devel tree plus the bare minimum patches needed to run on our board.

With sleep of 1 second, I usually see the crash in foo_load:

Trace; c0084488 <kfree_skbmem+20/94>
Trace; c00b21a4 <tcp_v4_hnd_req+38/1e0>
Trace; c00b250c <tcp_v4_do_rcv+b8/16c>
Trace; c00b2a6c <tcp_v4_rcv+4ac/778>
Trace; c00973c0 <ip_local_deliver+f0/1ac>
Trace; c0097840 <ip_rcv+3c4/470>
Trace; c0088f48 <net_rx_action+204/334>
Trace; c00199b4 <do_softirq+88/100>
Trace; c00887ec <dev_queue_xmit+248/334>
Trace; c009a168 <ip_output+110/174>
Trace; c009a5c0 <ip_queue_xmit+3f4/56c>
Trace; c00ab298 <tcp_transmit_skb+528/614>
Trace; c00adce4 <tcp_connect+458/518>
Trace; c00b0b28 <tcp_v4_connect+340/40c>
Trace; c00bf2d8 <inet_stream_connect+120/2b8>
Trace; c00818e8 <sys_connect+68/8c>
Trace; c0082378 <sys_socketcall+f0/1e0>
Trace; c00060dc <ret_from_syscall_1+0/b4>

Here a backtrace with sleep=2 seconds, inside khttpd:

Trace; 00000300 Before first symbol
Trace; c0083c3c <alloc_skb+110/204>
Trace; c009f00c <tcp_sendmsg+1f8/122c>
Trace; c00befd8 <inet_sendmsg+5c/70>
Trace; c008058c <sock_sendmsg+9c/cc>
Trace; c800e520 <END_OF_CODE+7eedd10/???
Trace; c800dcf4 <END_OF_CODE+7eed4e4/???
Trace; c800d268 <END_OF_CODE+7eeca58/???
Trace; c00088f8 <kernel_thread+2c/38>

A backtrace with sleep=3 seconds, in foo_load:

Trace; c0031614 <__free_pages+38/48>
Trace; c004a230 <sys_poll+340/360>
Trace; c00060dc <ret_from_syscall_1+0/b4>

A backtrace with sleep=3 seconds, in khttpd:

Trace; c0bdad18 <END_OF_CODE+aba508/???
Trace; c0083c58 <alloc_skb+12c/204>
Trace; c00ad8fc <tcp_send_ack+2c/11c>
Trace; c00ae638 <tcp_delack_timer+1c4/230>
Trace; c001e52c <timer_bh+2c8/420>
Trace; c0019eb4 <bh_action+3c/84>
Trace; c0019d74 <tasklet_hi_action+a0/e0>
Trace; c00199b4 <do_softirq+88/100>
Trace; c0007c48 <timer_interrupt+244/25c>
Trace; c0006310 <ret_from_intercept+0/8>
Trace; c0034c50 <shmem_getpage+a0/118>
Trace; c0035488 <do_shmem_file_read+94/14c>
Trace; c00355a0 <shmem_file_read+60/98>
Trace; c800dcd8 <END_OF_CODE+7eed4c8/???
Trace; c800d268 <END_OF_CODE+7eeca58/???
Trace; c00088f8 <kernel_thread+2c/38>

Yukky. Makes me want to go work with user-mode web servers instead.

- Dan

On Wed, May 08, 2002 at 10:26:14PM -0700, Dan Kegel wrote:

| Ken Brownfield wrote:
| > khttpd is very much production quality on IA32, and has been since
| > 2.4.0-test1. 
| 
| It oopses readily on start/stop -- see 
| http://marc.theaimsgroup.com/?l=linux-kernel&m=102068445316516&w=2
| in which DecodeHeader is called with no buffer allocated.
| Happened quite frequently during my tests.  Since my patch I
| can't get it to oops on x86, but it's still oopsing on ppc405.
| So perhaps now it's production quality... but it still needs
| a bit too much babying on stop.  (It'd be fairly easy to fix
| the unreliable way it senses the stop command.)

Hmm. I've had it running *hard* for two years, never seen a single oops or glitch of any sort, kernels 2.4.0-test1 through 2.4.18. O(1), preempt, low-latency all on at various times. All static images though, with no pass-through.

| > TUX2 is not, however, since under load it enters a 99% CPU
| > busy loop.
| 
| I checked the tux list and found the post you're taling about:
| http://marc.theaimsgroup.com/?l=tux-list&m=101420257421009&w=2
| Hmm.  Well, at least it doesn't oops :-)  Thanks for the warning.

Heh, no prob. The most recent version might have fixed it, although the last time I tried TUX2 I got a hard hang... prolly O(1) but I never had time to narrow it down.

[...]

| where foo_load is a hacked version of http_load (I think)
| which is fetching a single 128KB file over and over.
| I tried reproducing this with acme.com's http_load, 
| without success so far.

Odd, seems identical to my standard traffic, though my images are in the 4-64k range. What you're seeing seems to be an interaction problem -- something different in the non-mainstream, non-x86 kernel that's allowing khttpd to hang itself. No argument that khttpd is frightening.

[...]
| Yukky. Makes me want to go work with user-mode web servers instead.

Yeah, good luck tracking down X15. I've wanted to create a heavily-modified apache MPM, but my time is unfortunately finite...

Apache2 on top of TUX2 would be ideal, but alas neither is quite prime-time yet.

Ken.
[email protected]

| 
| - Dan

Tux2 has been stable for me on very large specweb runs. Ingo has been able to fix all the bugs I couldnt - a detailed bug report here would be helpful.

Anton

Try cycling it up and down in a loop. That's what triggers the bug my recent patch fixed. (Actually, just starting it once triggers it, depending on a race condition between the management thread and the worker threads.)

I have a copy, actually. It uses Linux's rtsig scheme. Can't use it, though, as its license prohibits commercial use. I am tempted to add rtsig support to thttpd, though.

- Dan