Skip to contentSkip to conversation

epoll oops on 2.6.x

DISCUSSION Louay Gammo 2 participants 6 messages May 13, 2004 – May 17, 2004

lore
Share & tools
lore
#0 lore raw

Hi,

I keep getting these oopses and kernel panics when I overload my Itanium-2 box with http requests. I had these oopses in 2.6.1 through 2.6.5 and it is consistently happening while using epoll, but not select. Also, these oopses happen only on IA-64.

I am including some of these oopses as attachements. I am not sure what the 'normal' bug reporting mechanism is for IA-64 kernels.

Thanks

Louay Gammo

#1 lore raw
Quoted 20 lines · Randy.Dunlap

Davide,

Quoted 1 line

From the linux-ia64 mailing list, in case you didn't see it.....

Begin forwarded message:

Date: Thu, 13 May 2004 17:39:02 -0400
From: Louay Gammo <[email protected]>
To: [email protected]
Subject: epoll oops on 2.6.x

Hi,

I keep getting these oopses and kernel panics when I overload my Itanium-2 box with http requests. I had these
oopses in 2.6.1 through 2.6.5 and it is consistently happening while using epoll, but not select. Also, these oopses
happen only on IA-64.

Quoted 3 lines

I am including some of these oopses as attachements. I am not sure what the 'normal' bug reporting mechanism is
for IA-64 kernels.

I'm getting this from Randy, since I was not subscribed to linux-ia64. Now I am. I took a quick look at it and so far the clue is completely missing. Only one of the oops actually happen inside epoll, even if the other ones comes typically from __wake_up_common() (others from vfs_read(), that is quite unrelated), that let me think that it might be epoll related (epoll has a wait queue item dropped in). Can you try a few experiments?

1) Disable pre-emption (if enabled)
2) Load the server thru loopback
3) #define DEBUG_EPOLL 10

(Your machine does not look too much in shape looking at INIT messages though)

- Davide

#2 lore raw
In reply to Louay Gammo
Quoted 5 lines · Davide Libenzi
Quoted 4 lines

including some of these oopses as attachements. I am not sure what the 'normal' bug reporting mechanism is for IA-64 kernels.

Quoted 7 lines

I'm getting this from Randy, since I was not subscribed to linux-ia64. Now I am. I took a quick look at it and so far the clue is completely missing. Only one of the oops actually happen inside epoll, even if the other ones comes typically from __wake_up_common() (others from vfs_read(), that is quite unrelated), that let me think that it might be epoll related (epoll has a wait queue item dropped in). Can you try a few experiments?

1) Disable pre-emption (if enabled)
2) Load the server thru loopback
3) #define DEBUG_EPOLL 10

I did #1 and #2 and ran my original experiment. I did not get a panic but
I got 0 replies from the server. So I undid the #2 and I got a kernel panic (included below). So I am going to try to overload the server on loopback interface (I think this is what you are saying) and I will let you know of the results.

I am enclosing the kernel config just in case that might clue anyone to a configuration problem.

Thanks,

Louay Gammo

tissimo:~# Unable to handle kernel paging request at virtual address 000000000001003e
userver[1284]: Oops 8804682956800 [1]

Pid: 1284, CPU 0, comm: userver
psr : 0000101008022018 ifs : 8000000000000003 ip : [<a000000100080620>] Not tainted
ip is at prepare_to_wait_exclusive+0x80/0xe0
unat: 0000000000000000 pfs : 0000000000000288 rsc : 0000000000000003
rnat: 0000000000000000 bsps: 0000000000000000 pr : 0000000000158659
ldrs: 0000000000000000 ccv : 0000000000000000 fpsr: 0009804c0270033f csd : 0000000000000000 ssd : 0000000000000000
b0 : a000000100530360 b6 : a000000100002d70 b7 : a000000100581980 f6 : 000000000000000000000 f7 : 000000000000000000000 f8 : 000000000000000000000 f9 : 000000000000000000000 f10 : 000000000000000000000 f11 : 000000000000000000000 r1 : a000000100990000 r2 : e0000040f5f5fd58 r3 : 0000000000000001 r8 : 0000000000000000 r9 : 000000000001003e r10 : e0000040f5f5fd88 r11 : a000000100722900 r12 : e0000040f5f5fd30 r13 : e0000040f5f58000 r14 : e000004043d26160 r15 : e0000040f5f5fd58 r16 : e0000040f5f58000 r17 : e000004043880d48 r18 : 0000001008026018 r19 : 0000000000000000 r20 : 000000003fffff00 r21 : e0000040f5f5fd58 r22 : ffffffffffff0020 r23 : a0000001008e9ab0 r24 : e0000040f5f5fd78 r25 : a0000001008ea198 r26 : e0000040f5f5fd70 r27 : e000004042206088 r28 : e000004042206090 r29 : e0000040422060a8 r30 : e0000040422060b0 r31 : e0000040422060b8

Call Trace:
[<a000000100017ae0>] show_stack+0x80/0xa0
                               sp=e0000040f5f5f900 bsp=e0000040f5f59378
[<a00000010003a330>] die+0x130/0x1a0
                               sp=e0000040f5f5fad0 bsp=e0000040f5f59340
[<a000000100054cd0>] ia64_do_page_fault+0x350/0x920
                               sp=e0000040f5f5fad0 bsp=e0000040f5f592d8
[<a0000001000118a0>] ia64_leave_kernel+0x0/0x260
                               sp=e0000040f5f5fb60 bsp=e0000040f5f592d8
[<a000000100080620>] prepare_to_wait_exclusive+0x80/0xe0
                               sp=e0000040f5f5fd30 bsp=e0000040f5f592c0
[<a000000100530360>] __lock_sock+0x100/0x1a0
                               sp=e0000040f5f5fd30 bsp=e0000040f5f59298
[<a000000100530f70>] lock_sock+0x50/0xa0
                               sp=e0000040f5f5fd90 bsp=e0000040f5f59278
[<a0000001005c7610>] inet_accept+0xb0/0x1e0
                               sp=e0000040f5f5fd90 bsp=e0000040f5f59248
[<a00000010052bdf0>] sys_accept+0x170/0x300
                               sp=e0000040f5f5fda0 bsp=e0000040f5f591a0
[<a000000100011720>] ia64_ret_from_syscall+0x0/0x20
                               sp=e0000040f5f5fe30 bsp=e0000040f5f591a0

<0>Kernel panic: Aiee, killing interrupt handler!
In interrupt handler - not syncing

Quoted 12 lines

(Your machine does not look too much in shape looking at INIT messages though)

- Davide

-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
In reply to Louay Gammo
Quoted 8 lines · Louay Gammo

I did #1 and #2 and ran my original experiment. I did not get a panic but I got 0 replies from the server. So I undid the #2 and I got a kernel panic (included below).
So I am going to try to overload the server on loopback interface (I think this is what you are saying) and I will let you know of the results.

I am enclosing the kernel config just in case that might clue anyone to a configuration problem.

Can you try also to run the appended test program, like:

# ./pipetest -n 2000 -a 3 -w 20000

- Davide

/*
* PipeTest by Davide Libenzi ( Epoll performace tester )
* Copyright (C) 1999,..,2003 Davide Libenzi
*

 *  This program is free software; you can redistribute it and/or modify
 *  it under the terms of the GNU General Public License as published by
 *  the Free Software Foundation; either version 2 of the License, or
 *  (at your option) any later version.
 *
 *  This program is distributed in the hope that it will be useful,
 *  but WITHOUT ANY WARRANTY; without even the implied warranty of
 *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 *  GNU General Public License for more details.
 *
 *  You should have received a copy of the GNU General Public License
 *  along with this program; if not, write to the Free Software
 *  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA

*
* Davide Libenzi <[email protected]>
*
*
* You need either a never glibc or the epoll library available here :
*
* http://www.xmailserver.org/linux-patches/nio-improve.html#sys_epoll
*
* to build this source file. To build :
*
* gcc -o pipetest pipetest.c -lepoll
*
*/

#include <sys/types.h>
#include <sys/stat.h>
#include <sys/time.h>
#include <sys/socket.h>
#include <sys/signal.h>
#include <sys/resource.h>
#include <fcntl.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>

#include <sys/epoll.h>

#define RUNTIMES 16

static int count, writes, fired;
static int *pipes;
static int num_pipes, num_active, num_writes;
static int epfd;
static struct epoll_event *events;

unsigned long long getustime(void) {
	struct timeval tm;

	gettimeofday(&tm, NULL);
	return (unsigned long long) tm.tv_sec * 1000000ULL + (unsigned long long) tm.tv_usec;
}
void read_cb(int fd, int idx) {
	int widx = idx + num_active + 1;
	u_char ch;

	if (read(fd, &ch, sizeof(ch)))
		count++;
	else
		fprintf(stderr, "false read event: fd=%d idx=%d\n", fd, idx);
	if (writes) {
		if (widx >= num_pipes)
			widx -= num_pipes;
		write(pipes[2 * widx + 1], "e", 1);
		writes--;
		fired++;
	}
}
int run_once(long *work, unsigned long long *tr) {
	int i, res;
	unsigned long long ts, te;

	fired = 0;
	for (i = 0; i < num_active; i++, fired++)
		write(pipes[i * 2 + 1], "e", 1);

	count = 0;
	writes = num_writes;

	ts = getustime();
	do {
		res = epoll_wait(epfd, events, num_pipes, 0);
		for (i = 0; i < res; i++)
			read_cb(pipes[2 * events[i].data.u32], events[i].data.u32);
	} while (count != fired);
	te = getustime();

	*tr = te - ts;
	*work = count;

	return (0);
}

int main (int argc, char **argv) {

	struct rlimit rl;
	int i, c;
	long work;
	unsigned long long tr;
	int *cp;
	struct epoll_event ev;
	extern char *optarg;

	num_pipes = 100;
	num_active = 1;
	num_writes = num_pipes;
	while ((c = getopt(argc, argv, "n:a:w:")) != -1) {
		switch (c) {
		case 'n':
			num_pipes = atoi(optarg);
			break;
		case 'a':
			num_active = atoi(optarg);
			break;
		case 'w':
			num_writes = atoi(optarg);
			break;
		default:
			fprintf(stderr, "Illegal argument \"%c\"\n", c);
			exit(1);
		}
	}

	rl.rlim_cur = rl.rlim_max = num_pipes * 2 + 50;
	if (setrlimit(RLIMIT_NOFILE, &rl) = -1) {
		perror("setrlimit"); 
		exit(1);
	}

	events = calloc(num_pipes, sizeof(struct epoll_event));
	pipes = calloc(num_pipes * 2, sizeof(int));
	if (events = NULL || pipes = NULL) {
		perror("malloc");
		exit(1);
	}

	if ((epfd = epoll_create(num_pipes)) = -1) {
		perror("epoll_create");
		exit(1);
	}

	for (cp = pipes, i = 0; i < num_pipes; i++, cp += 2) {
		if (pipe(cp) = -1) {
			perror("pipe");
			exit(1);
		}
		fcntl(cp[0], F_SETFL, fcntl(cp[0], F_GETFL) | O_NONBLOCK);
	}

	for (cp = pipes, i = 0; i < num_pipes; i++, cp += 2) {
		ev.events = EPOLLIN | EPOLLET;
		ev.data.u32 = i;
		if (epoll_ctl(epfd, EPOLL_CTL_ADD, cp[0], &ev) < 0) {
			perror("epoll_ctl");
			exit(1);
		}
	}

	for (i = 0; i < RUNTIMES; i++) {
		run_once(&work, &tr);
		if (!work)
			exit(1);
		fprintf(stdout, "%lf\n", (double) tr / (double) work);
	}

	exit(0);
}
#4 lore raw
In reply to Louay Gammo
Quoted 10 lines · Davide Libenzi
Quoted 6 lines · Louay Gammo

I did #1 and #2 and ran my original experiment. I did not get a panic but I got 0 replies from the server. So I undid the #2 and I got a kernel panic (included below).
So I am going to try to overload the server on loopback interface (I think this is what you are saying) and I will let you know of the results.

Quoted 11 lines
Quoted 4 lines

I am enclosing the kernel config just in case that might clue anyone to a configuration problem.

Can you try also to run the appended test program, like:

# ./pipetest -n 2000 -a 3 -w 20000

Ok, and I got this:
~/pipetest> ./a.out -n 2000 -a 3 -w 20000
3.917762
3.868220
3.860521
3.855072
3.853522
3.853422
3.854822
3.853072
3.853972
3.854972
3.856022
3.855172
3.856572
3.860621
3.861171
3.855672

Does this help?

Quoted 7 lines

- Davide

Louay Gammo

#5 lore raw
In reply to Louay Gammo
Quoted 40 lines · Louay Gammo
Quoted 8 lines · Davide Libenzi
Quoted 6 lines · Louay Gammo

I did #1 and #2 and ran my original experiment. I did not get a panic but I got 0 replies from the server. So I undid the #2 and I got a kernel panic (included below).
So I am going to try to overload the server on loopback interface (I think this is what you are saying) and I will let you know of the results.

Quoted 11 lines
Quoted 4 lines

I am enclosing the kernel config just in case that might clue anyone to a configuration problem.

Can you try also to run the appended test program, like:

# ./pipetest -n 2000 -a 3 -w 20000

Ok, and I got this:
~/pipetest> ./a.out -n 2000 -a 3 -w 20000
3.917762
3.868220
3.860521
3.855072
3.853522
3.853422
3.854822
3.853072
3.853972
3.854972
3.856022
3.855172
3.856572
3.860621
3.861171
3.855672

Quoted 1 line

Does this help?

This proves that it works fine with pipes. Can you try to put your server listening on localhost and load it thru localhost?

- Davide