Kernel hangs in SMP + VMware environment.
maybe related to http://bugzilla.kernel.org/show_bug.cgi?id=9834 ?
you say "recent" , so this does happen from 2.6.21 to 2.6.26rc2 ? does that happen only on a dedicated vmware box, or on different ones? vmware-tools active? ->stop -> different ?
could you provide some more information about your hardware/vmware environment ? does that happen on esx or on hosted products (workstation, server, player..) ?
regards
roland
List: linux-kernel Subject: Kernel hangs in SMP + VMware environment. From: Tetsuo Handa <penguin-kernel () I-love ! SAKURA ! ne ! jp> Date: 2008-05-12 21:41:34 Message-ID: 200805130641.CDG56299.QStFOLVOOJFHMF () I-love ! SAKURA ! ne ! jp
[Download message RAW]
I'm experiencing hang up problem with recent kernels in VMware environment.
Here are two examples.
http://I-love.SAKURA.ne.jp/tmp/messages.1 (203kB)
http://I-love.SAKURA.ne.jp/tmp/messages.2 (1.7MB)
The messages.1 is a log when "tar" stopped processing while extracting a .tar.bz2 file by
"rpmbuild -bb --target i586 --with baseonly kernel.spec". I got this log in runlevel 3 of Fedora 8.
The messages.2 is a log when the compiler processes (e.g. "cc1") seem to be hanged up (no compiler messages appear for minutes, which unlikely happen).
I got this log in runlevel 1 of Fedora 8
by "rpmbuild -bb --target i586 --with baseonly kernel.spec" after starting rsyslog and stopping anacron.
I experience this problem in many distro (e.g. Fedora, Ubuntu, SuSE), which use kenel (I think) around 2.6.21 and later.
I experience this problem only in VMware.
I have never experienced this problem in native environment.
I experience this problem only when I assign 2 CPUs to VMware. I have never experienced this problem with 1 CPU.
May be something scheduler related in SMP + VMware environment.
The a.out process in the log files are http://lkml.org/lkml/2008/5/12/130
What other information should I dump for identifying the location of hang up?
Regards.
Hello.
Quoted 1 line · Roland
maybe related to http://bugzilla.kernel.org/show_bug.cgi?id=9834 ?
Thank you for URL.
My bug seems to be timer related.
Quoted 1 line
you say "recent" , so this does happen from 2.6.21 to 2.6.26rc2 ?
I don't know exact version, but I don't experience this problem in earlier kernels (e.g. Fedora Core 5 which uses kernel 2.6.20).
Quoted 2 lines
does that happen only on a dedicated vmware box, or on different ones? vmware-tools active? ->stop -> different ?
vmware-tools is not installed for my Fedora 8.
Quoted 4 lines
could you provide some more information about your hardware/vmware environment ?
does that happen on esx or on hosted products (workstation, server, player..) ?
Hardware: ThinkPad X60 (Intel Core 2 Duo, 2048MB RAM, No swap partition) VMware host environment: CentOS 5.1 (x86_64)
VMware version: VMware Workstation 6.0.2 (x86_64)
VMware guest environment: many distro using recent kernels (all i386)
I don't have ESX server environment.
Today, I tried to reproduce this problem using 2.6.24.5-85.fc8 kernel and I got 2 patterns.
http://I-love.SAKURA.ne.jp/tmp/hangup-3.png (10kB)
http://I-love.SAKURA.ne.jp/tmp/messages-3.txt (174kB)
hangup-3.png is the screenshot of hang up and messages-3.txt is the sysrq logs. Funny thing is that "tar" process sleeps for minutes at blk_remove_plug() (while "tar" finishes within a minute if 1 CPU).
http://I-love.SAKURA.ne.jp/tmp/dmesg-4.txt (120kB)
dmesg-4.txt is a partial output of "dmesg".
Since rsyslog sometimes cannot save logs to /var/log/messages by some reason, I tried to directly save from /proc/kmsg using "a.out",
but "a.out" couldn't save logs neither.
Funny thing is that "a.out" process sleeps for minutes at getnstimeofday(). The source code of "a.out" is
#include <stdio.h>
#include <unistd.h>
#include <time.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <string.h>
int main(int argc, char *argv[]) {
FILE *fp = fopen("/proc/sys/kernel/sysrq", "w");
if (!fp) return 1;
fprintf(fp, "1\n");
fclose(fp);
fp = fopen("/proc/sysrq-trigger", "w");
if (!fp) return 1;
if (fork() == 0) {
int fd_r = open("/proc/kmsg", O_RDONLY);
int fd_w = open("/root/messages", O_WRONLY | O_TRUNC | O_CREAT, 0600);
char buffer[4096];
char timebuf[80];
memset(timebuf, 0, sizeof(timebuf));
memset(buffer, 0, sizeof(buffer));
while (1) {
const int len = read(fd_r, buffer, sizeof(buffer));
static time_t prev = 0;
const time_t now = time(NULL);
if (now != prev) {
static int counter = 0;
prev = now;
snprintf(timebuf, sizeof(timebuf) - 1, "****************************** %d %lu ******************************\n", counter++, now);
write(fd_w, timebuf, strlen(timebuf));
}
write(fd_w, buffer, len);
}
_exit(0);
}
while (1) {
sleep(60);
fprintf(fp, "t\n");
fflush(fp);
}
return 0;
}Other patterns that hanged up are
(1) # time sync
(2) # top
(3) # dmesg > ~/log
Regards.