Skip to contentSkip to conversation

SKAS race condition status?

uml-devel Christopher S. Aker 4 participants 8 messages Oct 17, 2005 – Oct 25, 2005 User-mode Linux (UML)

lore
Share & tools
lore
#0 lore raw

Has there been any progress on the race in SKAS? I've been dealing with 2-4 hosts crashing a week. Same thing happens on my old SKAS kernel, 2.6.7 with skas3-v4 or so.

From:
http://marc.theaimsgroup.com/?t=112376981300002&r=1&w=2

Unable to handle kernel paging request at virtual address 00002000
  printing eip:
00000000
*pde = 2a771001
Oops: 0000 [#1]
SMP
Modules linked in:
CPU:    2
EIP:    0200:[<00000000>]    Not tainted VLI
EFLAGS: 00010292   (2.6.11.11-1-bigmem)

EIP is at 0x0
eax: ef66b784 ebx: ef7ba000 ecx: ef60f540 edx: 00000000
esi: 00000001 edi: 00000000 ebp: ef7bbf14 esp: ef7bbf9c
ds: 007b es: 007b ss: 0068
Process 2.4.29-linode39 (pid: 11195, threadinfo=ef7ba000 task=ef60f540)

Stack: ef7bbfc4 ef7ba000 c0106fd7 00000005 be7ff890 c0104934 be7ff768
be7ff760
        00000000 c0102639 be7ff768 00000000 40029d00 be7ff760 00000000
be7ff928
        00000000 0000007b 0000007b 0000004e 40100fb1 00000073 00000246
be7ff74c
Call Trace:
  [<c0106fd7>] do_syscall_trace+0x97/0x10e
  [<c0104934>] math_state_restore+0x24/0x40
  [<c0102639>] syscall_trace_entry+0x11/0x2a
Code:  Bad EIP value.

Is there code I could add there to OOPs rather than panic? I'd prefer to have just one UML go down rather than the entire host.

Thanks!
-Chris

-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions, and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

#1 lore raw
Quoted 1 line · Christopher S. Aker

Has there been any progress on the race in SKAS? I've been dealing with 2-4 hosts crashing a week. Same thing happens on my old SKAS kernel, 2.6.7 with skas3-v4 or so.

Jeff and I realized that this only happens on Xeon processors where the model number is > 2. None of my model 2 Xeon hosts crash, but my model 3 and model 4 xeons do. grep model /proc/cpuinfo for info. I've had skas3-v4 and skas3-v8.2 generate the same panic, so it's something processor model specific.

Peter, can you verify the problems you've had are on hosts with a model 3 or 4 Xeon processor? Also, with regard to 2.6.12, I'm not sure we're looking at the same issue, as I had problems with that kernel too but wasn't able to verify it was the same panic. I have had 2.6.13 working, and it does generate the same SKAS panic eventually.

Thanks,
-Chris

-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

#2 lore raw

/proc/cpuinfo -- model 3
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 3
model name : Intel(R) Xeon(TM) CPU 2.80GHz
stepping : 4
cpu MHz : 2801.996
cache size : 1024 KB
physical id : 0
siblings : 2
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm pni monitor ds_cpl cid xtpr
bogomips : 5521.40

/proc/cpuinfo -- model 4
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name : Intel(R) Xeon(TM) CPU 3.20GHz
stepping : 3
cpu MHz : 3200.832
cache size : 2048 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips : 6409.33

-Chris

-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

#3 lore raw
Quoted 1 line · Christopher S. Aker

/proc/cpuinfo -- model 3

After some researches, it seems that the matching errata document is this one:

http://www.intel.com/design/xeon/specupdt/302402.htm

which would imply you have a 64-bit enabled processor...

Note 1: that's not the only document matching, but the other is for 64-bit processors too

Note 2: I couldn't match steppings between those docs and cpuinfo...

Signature · 23 lines
Inform me of my mistakes, so I can keep imitating Homer Simpson's "Doh!".
Paolo Giarrusso, aka Blaisorblade (Skype ID "PaoloGiarrusso", ICQ 215621894)
http://www.user-mode-linux.org/~blaisorblade

	

	
		
___________________________________ 
Yahoo! Mail: gratis 1GB per i messaggi e allegati da 10MB 
http://mail.yahoo.it



-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course
Free Certification Exam for All Training Attendees Through End of 2005
Visit http://www.jboss.com/services/certification for more information
_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
#4 lore raw
Quoted 9 lines · Blaisorblade

After some researches, it seems that the matching errata document is this one:

http://www.intel.com/design/xeon/specupdt/302402.htm

which would imply you have a 64-bit enabled processor...

Note 1: that's not the only document matching, but the other is for 64-bit processors too

Note 2: I couldn't match steppings between those docs and cpuinfo...

OK. I matched up some purchase orders with Intel part numbers:

BX80546KG3200FU <-- model 4
BX80546KG3200EU <-- model 3 (I think)

I'll start disabling HyperThreading on hosts that crash next.

-Chris

-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

#5 lore raw
In reply to Christopher S. Aker

I have skas3-v8 host servers with model 4 processes and good (100+ day) uptimes. The host server I had issues with (host38) got converted to Xen and is still having problems. I am seeing issues with host34 that require the odd restart. That server runs the same CPU model and kernel code as other servers of mine with good uptime.

My conclusion is that I must be seeing hardware issues rather than a skas bug.

- Peter

# for ((i=34;i<41;i++)); do echo host$i $(ssh host$i 'echo $(cat /proc/cpuinfo  | grep model | grep -v name | head -n 1) $(uname -r) $(uptime)'); done
host34 model : 2 2.6.11.10-skas3-v8 21:52:37 up 15:19, 0 users, load average: 1.64, 1.60, 1.32
host35 model : 4 2.6.11.7-skas3-v8 21:49:21 up 42 days, 12:15, 0 users, load average: 13.72, 9.66, 5.21
host36 model : 4 2.6.11.12-skas3-v8.2 21:49:23 up 61 days, 0 min, 0 users, load average: 1.26, 1.44, 1.78
host37 model : 4 2.6.11.7-skas3-v8 21:49:25 up 121 days, 9:48, 0 users, load average: 0.91, 1.04, 1.14
host38 model : 2 2.6.11.12-xen0 21:49:26 up 7 days, 13:28, 0 users, load average: 0.02, 0.05, 0.01
host39 model : 4 2.6.11.7-skas3-v8 21:49:28 up 106 days, 2:46, 0 users, load average: 0.59, 0.65, 0.70
host40 model : 4 2.6.11.7-skas3-v8 21:49:39 up 121 days, 9:47, 0 users, load average: 0.27, 0.43, 0.47
Quoted 11 lines · Christopher S. Aker
Quoted 1 line · Christopher S. Aker

Has there been any progress on the race in SKAS? I've been dealing with 2-4 hosts crashing a week. Same thing happens on my old SKAS kernel, 2.6.7 with skas3-v4 or so.

Jeff and I realized that this only happens on Xeon processors where the model number is > 2. None of my model 2 Xeon hosts crash, but my model 3 and model 4 xeons do. grep model /proc/cpuinfo for info. I've had skas3-v4 and skas3-v8.2 generate the same panic, so it's something processor model specific.

Peter, can you verify the problems you've had are on hosts with a model 3 or 4 Xeon processor? Also, with regard to 2.6.12, I'm not sure we're looking at the same issue, as I had problems with that kernel too but wasn't able to verify it was the same panic. I have had 2.6.13 working, and it does generate the same SKAS panic eventually.

Thanks,
-Chris

-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

#6 lore raw
Quoted 3 lines · Peter

I have skas3-v8 host servers with model 4 processes and good (100+ day) uptimes. The host server I had issues with (host38) got converted to Xen and is still having problems. I am seeing issues with host34 that require the odd restart. That server runs the same CPU model and kernel code as other servers of mine with good uptime.

My conclusion is that I must be seeing hardware issues rather than a skas bug.

By chance, do you have Hyperthreading disabled on your model 4 Xeon boxes?

-Chris

-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

#7 lore raw

HT is not disabled on any of my boxes.

Quoted 10 lines · Christopher S. Aker
Quoted 3 lines · Peter

I have skas3-v8 host servers with model 4 processes and good (100+ day) uptimes. The host server I had issues with (host38) got converted to Xen and is still having problems. I am seeing issues with host34 that require the odd restart. That server runs the same CPU model and kernel code as other servers of mine with good uptime.

My conclusion is that I must be seeing hardware issues rather than a skas bug.

By chance, do you have Hyperthreading disabled on your model 4 Xeon boxes?

-Chris

-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel