Skip to contentSkip to conversation

Kernel hangs in SMP + VMware environment.

DISCUSSION Roland 2 participants 2 messages May 13, 2008 – May 14, 2008

lore
Share & tools
lore

Roland

author
#0 lore raw

maybe related to http://bugzilla.kernel.org/show_bug.cgi?id=9834 ?

you say "recent" , so this does happen from 2.6.21 to 2.6.26rc2 ? does that happen only on a dedicated vmware box, or on different ones? vmware-tools active? ->stop -> different ?

could you provide some more information about your hardware/vmware environment ? does that happen on esx or on hosted products (workstation, server, player..) ?

regards
roland

List:       linux-kernel
Subject:    Kernel hangs in SMP + VMware environment.
From:       Tetsuo Handa <penguin-kernel () I-love ! SAKURA ! ne ! jp>
Date:       2008-05-12 21:41:34
Message-ID: 200805130641.CDG56299.QStFOLVOOJFHMF () I-love ! SAKURA ! ne ! jp

[Download message RAW]

I'm experiencing hang up problem with recent kernels in VMware environment.

Here are two examples.

http://I-love.SAKURA.ne.jp/tmp/messages.1 (203kB)
http://I-love.SAKURA.ne.jp/tmp/messages.2 (1.7MB)

The messages.1 is a log when "tar" stopped processing while extracting a .tar.bz2 file by
"rpmbuild -bb --target i586 --with baseonly kernel.spec". I got this log in runlevel 3 of Fedora 8.

The messages.2 is a log when the compiler processes (e.g. "cc1") seem to be hanged up (no compiler messages appear for minutes, which unlikely happen).
I got this log in runlevel 1 of Fedora 8
by "rpmbuild -bb --target i586 --with baseonly kernel.spec" after starting rsyslog and stopping anacron.

I experience this problem in many distro (e.g. Fedora, Ubuntu, SuSE), which use kenel (I think) around 2.6.21 and later.

I experience this problem only in VMware.
I have never experienced this problem in native environment.

I experience this problem only when I assign 2 CPUs to VMware. I have never experienced this problem with 1 CPU.

May be something scheduler related in SMP + VMware environment.

The a.out process in the log files are http://lkml.org/lkml/2008/5/12/130
What other information should I dump for identifying the location of hang up?

Regards.

#1 lore raw

Hello.

Quoted 1 line · Roland

maybe related to http://bugzilla.kernel.org/show_bug.cgi?id=9834 ?

Thank you for URL.
My bug seems to be timer related.

Quoted 1 line

you say "recent" , so this does happen from 2.6.21 to 2.6.26rc2 ?

I don't know exact version, but I don't experience this problem in earlier kernels (e.g. Fedora Core 5 which uses kernel 2.6.20).

Quoted 2 lines

does that happen only on a dedicated vmware box, or on different ones? vmware-tools active? ->stop -> different ?

vmware-tools is not installed for my Fedora 8.

Quoted 4 lines

could you provide some more information about your hardware/vmware environment ?
does that happen on esx or on hosted products (workstation, server, player..) ?

Hardware: ThinkPad X60 (Intel Core 2 Duo, 2048MB RAM, No swap partition) VMware host environment: CentOS 5.1 (x86_64)
VMware version: VMware Workstation 6.0.2 (x86_64)
VMware guest environment: many distro using recent kernels (all i386)

I don't have ESX server environment.

Today, I tried to reproduce this problem using 2.6.24.5-85.fc8 kernel and I got 2 patterns.

http://I-love.SAKURA.ne.jp/tmp/hangup-3.png (10kB)
http://I-love.SAKURA.ne.jp/tmp/messages-3.txt (174kB)

hangup-3.png is the screenshot of hang up and messages-3.txt is the sysrq logs. Funny thing is that "tar" process sleeps for minutes at blk_remove_plug() (while "tar" finishes within a minute if 1 CPU).

http://I-love.SAKURA.ne.jp/tmp/dmesg-4.txt (120kB)

dmesg-4.txt is a partial output of "dmesg".
Since rsyslog sometimes cannot save logs to /var/log/messages by some reason, I tried to directly save from /proc/kmsg using "a.out",
but "a.out" couldn't save logs neither.
Funny thing is that "a.out" process sleeps for minutes at getnstimeofday(). The source code of "a.out" is

  #include <stdio.h>
  #include <unistd.h>
  #include <time.h>
  #include <sys/types.h>
  #include <sys/stat.h>
  #include <fcntl.h>
  #include <string.h>
  
  int main(int argc, char *argv[]) {
  	FILE *fp = fopen("/proc/sys/kernel/sysrq", "w");
  	if (!fp) return 1;
  	fprintf(fp, "1\n");
  	fclose(fp);
  	fp = fopen("/proc/sysrq-trigger", "w");
  	if (!fp) return 1;
  	if (fork() == 0) {
  		int fd_r = open("/proc/kmsg", O_RDONLY);
  		int fd_w = open("/root/messages", O_WRONLY | O_TRUNC | O_CREAT, 0600);
  		char buffer[4096];
  		char timebuf[80];
  		memset(timebuf, 0, sizeof(timebuf));
  		memset(buffer, 0, sizeof(buffer));
  		while (1) {
  			const int len = read(fd_r, buffer, sizeof(buffer));
  			static time_t prev = 0;
  			const time_t now = time(NULL);
  			if (now != prev) {
  				static int counter = 0;
  				prev = now;
  				snprintf(timebuf, sizeof(timebuf) - 1, "****************************** %d %lu ******************************\n", counter++, now);
  				write(fd_w, timebuf, strlen(timebuf));
  			}
  			write(fd_w, buffer, len);
  		}
  		_exit(0);
  	}
  	while (1) {
  		sleep(60);
  		fprintf(fp, "t\n");
  		fflush(fp);
  	}
  	return 0;
  }

Other patterns that hanged up are

(1) # time sync

(2) # top

(3) # dmesg > ~/log

Regards.