VMWare crashes

I realize this may be a VMWare problem, but I just waited to bring this to the attention of the developers in case it was related to the kernel and to also see if anyone else is having the same problem. VMWare dies under load with all kernel versions up to and including ac versions after 2.4.6. Kernel version up to and including 2.4.5-ac15 I know all run fine. Somewhere between 2.4.5-ac15 and 2.4.6 is where the problem started. I have backed up to 2.4.5 now and VMWare is rock solid.

Unable to handle kernel NULL pointer dereference at virtual address 00000070
 printing eip:
e1af85e1
Call Trace: [<c0203fe4>] [<c0203fe4>] [<c011722f>] [<c012eb26>] [<c0106dc3>]

Quoted 1 line

EIP; e1af85e1 <[vmnet]VNetHubCycleDetect+69/7c> <=====

Quoted 9 lines · Gary White (Network Administr

I realize this may be a VMWare problem, but I just waited to bring this to the attention of the developers in case it was related to the kernel and to also see if anyone else is having the same problem. VMWare dies under load with all kernel versions up to and including ac versions after 2.4.6. Kernel version up to and including 2.4.5-ac15 I know all run fine. Somewhere between 2.4.5-ac15 and 2.4.6 is where the problem started. I have backed up to 2.4.5 now and VMWare is rock solid.

Quoted 4 lines

Unable to handle kernel NULL pointer dereference at virtual address 00000070
 printing eip:
e1af85e1
Call Trace: [<c0203fe4>] [<c0203fe4>] [<c011722f>] [<c012eb26>] [<c0106dc3>]

Could you feed these oopeses through ksymoops?

I'm now running vmware 24h/day with linux and win98 as guest, doing network transfers between host and guest, and I did not noticed any problems.

It worked fine with 2.4.5-ac24 from wednesday to sunday, and yesterday I upgraded to 2.4.6-ac2, and it still works. Kernel compiled with Debian's gcc-3.0-3 or gcc-3.0-4, Asus A7V, KT133, 1GHz Athlon, and Chaintech 6BTM, 440BX, 300MHz Celeron... I did not tested Linus's kernel for more than 6 months now, so I cannot tell whether it works with Linus's 2.4.6, or not...
Best regards,
Petr Vandrovec [email protected]

Here are the results of ksymoops...

Code; e1af85e1 <[vmnet]VNetHubCycleDetect+69/7c> <=====
0: 8b 42 70 mov 0x70(%edx),%eax <=====

Quoted 1 line

EIP; e1af9639 <[vmnet]VNetBridgeReceiveFromVNet+191/20c> <=====

Quoted 9 lines · Gary White (Network Administr

I realize this may be a VMWare problem, but I just waited to bring this to the attention of the developers in case it was related to the kernel and to also see if anyone else is having the same problem. VMWare dies under load with all kernel versions up to and including ac versions after 2.4.6. Kernel version up to and including 2.4.5-ac15 I know all run fine. Somewhere between 2.4.5-ac15 and 2.4.6 is where the problem started. I have backed up to 2.4.5 now and VMWare is rock solid.

Quoted 4 lines

Unable to handle kernel NULL pointer dereference at virtual address 00000070
 printing eip:
e1af85e1
Call Trace: [<c0203fe4>] [<c0203fe4>] [<c011722f>] [<c012eb26>] [<c0106dc3>]

Could you feed these oopeses through ksymoops?

I'm now running vmware 24h/day with linux and win98 as guest, doing network transfers between host and guest, and I did not noticed any problems.

It worked fine with 2.4.5-ac24 from wednesday to sunday, and yesterday I upgraded to 2.4.6-ac2, and it still works. Kernel compiled with Debian's gcc-3.0-3 or gcc-3.0-4, Asus A7V, KT133, 1GHz Athlon, and Chaintech 6BTM, 440BX, 300MHz Celeron... I did not tested Linus's kernel for more than 6 months now, so I cannot tell whether it works with Linus's 2.4.6, or not...
Best regards,
Petr Vandrovec [email protected]

Quoted 4 lines · Gary White (Network Administrator)

Here are the results of ksymoops...

Code; e1af85e1 <[vmnet]VNetHubCycleDetect+69/7c> <=====
0: 8b 42 70 mov 0x70(%edx),%eax <=====

Thanks, meanwhile I found simillar report on VMware newsgroups, so there must be something really real (it was with example how to reproduce it, so I received fine oops too).

Following patch fixes oopses, at least for me. Due to some changes in tasklets and/or in networking there is now very large quantum of skbs in flight from one part of vmnet (packet written to /dev/vmnet*) through netif_rx to another (packet received on eth0 interface). This trigerred 'history buffer overflow' message, which then started cleaning history buffer. And if two CPUs started cleaning at the same moment, one of them did kfree_skb(NULL) sooner or later...

So only increasing VNET_BRIDGE_HISTORY from 8 to 48 fixes problem, but as I do not want oopses, rest of this patch just fixes oopses themselves. If you'll apply patch except VNET_BRIDGE_HISTORY line, it will work, but you'll get large stream of 'history buffer full' messages when doing TCP transfers between guest and host. (48 is apparently enough for dual PIII/800, I did not tested lower values (48 is next multiple of 8 which ends on 8, this saved one keystroke and so on...))

For those unfamiliar with patch (I'm sure there are no on linux-kernel, but there can be some in VMware newsgroups) I put updated vmnet.tar.gz at ftp://platan.vc.cvut.cz/pub/vmware/vmnet-204-for-2.4.6.tar.gz

Best regards,
Petr Vandrovec [email protected]

diff -urN vmnet-only.orig/bridge.c vmnet-only/bridge.c

bridge.c Mon Jul 9 21:50:36 2001 renamed from bridge.c Thu Apr 26 19:59:28 2001 +15 -3

@@ -44,7 +44,7 @@
 #include "vnetInt.h"
 
 
-#define VNET_BRIDGE_HISTORY    8
+#define VNET_BRIDGE_HISTORY    48
 
 typedef struct VNetBridge VNetBridge;
 

@@ -58,6 +58,7 @@
    Bool                     savedPromisc;
    struct sk_buff          *history[VNET_BRIDGE_HISTORY];
    VNetPort                 port;
+   spinlock_t              historyLock;
 };
 
 

@@ -130,6 +131,7 @@
       goto out;
    }
    memset(bridge, 0, sizeof *bridge);
+   spin_lock_init(&bridge->historyLock);
    memcpy(bridge->name, devName, sizeof bridge->name);
 
    /*

@@ -391,6 +393,8 @@
         unsigned long flags;
         int i;
         SKB_INCREF(clone);
+
+        spin_lock_irqsave(&bridge->historyLock, flags);
         // XXX need to lock history
         for (i = 0; i < VNET_BRIDGE_HISTORY; i++) {
            if (bridge->history[i] == NULL) {

@@ -417,11 +421,15 @@
            for (i = 0; i < VNET_BRIDGE_HISTORY; i++) {
               struct sk_buff *s = bridge->history[i];
               bridge->history[i] = NULL;
-              KFREE_SKB(s, FREE_WRITE);
+              if (s) {
+                 spin_unlock_irqrestore(&bridge->historyLock, flags);
+                 KFREE_SKB(s, FREE_WRITE);
+                 spin_lock_irqsave(&bridge->historyLock, flags);
+              }
            }
            bridge->history[0] = clone;
         }
-
+         spin_unlock_irqrestore(&bridge->historyLock, flags);
         clone->dev = dev;
         clone->protocol = eth_type_trans(clone, dev);
         save_flags(flags);

@@ -773,6 +781,7 @@
 {
    VNetBridge *bridge = *(VNetBridge**)&((struct sock *)pt->data)->protinfo;
    int i;
+   unsigned long flags;
 
    if (bridge->dev == NULL) {
       LOG(3, (KERN_DEBUG "bridge-%s: received %d closed\n",

@@ -782,11 +791,13 @@
    }
 
    // XXX need to lock history
+   spin_lock_irqsave(&bridge->historyLock, flags);
    for (i = 0; i < VNET_BRIDGE_HISTORY; i++) {
       struct sk_buff *s = bridge->history[i];
       if (s != NULL &&
          (s == skb || SKB_IS_CLONE_OF(skb, s))) {
         bridge->history[i] = NULL;
+        spin_unlock_irqrestore(&bridge->historyLock, flags);
         KFREE_SKB(s, FREE_WRITE);
         LOG(3, (KERN_DEBUG "bridge-%s: receive %d self %d\n",
                 bridge->name, (int) skb->len, i));

@@ -795,6 +806,7 @@
         return 0;
       }
    }
+   spin_unlock_irqrestore(&bridge->historyLock, flags);
 
 #  if LOGLEVEL >= 4
    {