Solaris 10 doesn't work under KVM

I tried to install Solaris 10 under KVM, without success, so Avi told me to post the details here. My CPU is an AMD X2, and I am using KVM trunk on a 2.6.19.2, started with the following command-line:

kvm -hda solaris.qcow -cdrom sol-10-u3-ga-x86-dvd.iso -boot d -m 512 -no-acpi

(I tried with and without -no-acpi).

If I follow the normal installation path, I soon get messages like:

svc.startd[7]: svc:/milestone/single-user:default: Method "/sbin/rcS start" failed due to signal ILL

-no-acpi should not be necessary for Solaris/x86.

I've seen SIGILLs on AMD (Opteron 8218) as well. It worked on an Intel CPU, however.

Is your host running in 32 or 64 bit mode? I used 64 bit mode (host OS was SLES 10).

Now, how to reproduce my setup:
- Browse http://www.sun.com/software/solaris/get.jsp
- Click "Download Solaris 10"
- If asked for a user/pass, use bugmenot0/bugmenot (or make your own account)
- Select the medium at the top of the page. You can reproduce this bug using only the first CD (308MB), rather than downloading it all.
- Unzip the ISO somewhere
- Create a temporary disk image and boot.

Please let me know if you need any additional information. Thank you much in advance,

Quoted 10 lines · Waba

Now, how to reproduce my setup:
- Browse http://www.sun.com/software/solaris/get.jsp
- Click "Download Solaris 10"
- If asked for a user/pass, use bugmenot0/bugmenot (or make your own account)
- Select the medium at the top of the page. You can reproduce this bug using only the first CD (308MB), rather than downloading it all.
- Unzip the ISO somewhere
- Create a temporary disk image and boot.

Quoted 3 lines

Please let me know if you need any additional information. Thank you much in advance,

The qemu BIOS complains that it can't boot from the CDROM (sol-10-u3-ga-x86-v1.iso). This happens with and without kvm. Do I need some other image?

'file' says:
/data/images/solaris/sol-10-u3-ga-x86-v1.iso: ISO 9660 CD-ROM filesystem
data 'SOL_10_1106_X86 ' (bootable)

The qemu BIOS complains that it can't boot from the CDROM (sol-10-u3-ga-x86-v1.iso). This happens with and without kvm. Do I need some other image?

'file' says:
/data/images/solaris/sol-10-u3-ga-x86-v1.iso: ISO 9660 CD-ROM filesystem
data 'SOL_10_1106_X86 ' (bootable)

Quoted 8 lines · Avi Kivity

The qemu BIOS complains that it can't boot from the CDROM (sol-10-u3-ga-x86-v1.iso). This happens with and without kvm. Do I need some other image?

'file' says:
/data/images/solaris/sol-10-u3-ga-x86-v1.iso: ISO 9660 CD-ROM filesystem
data 'SOL_10_1106_X86 ' (bootable)

That's strange. That looks like the same file that I got here, and it works for me (past the GRUB screen at least). Just in case, here is the MD5 sum of my version:

c6f82c36195609b8dad67f61a927c7bc sol-10-u3-ga-x86-v1.iso

Quoted 15 lines · Waba

Quoted 8 lines · Avi Kivity

The qemu BIOS complains that it can't boot from the CDROM (sol-10-u3-ga-x86-v1.iso). This happens with and without kvm. Do I need some other image?

'file' says:
/data/images/solaris/sol-10-u3-ga-x86-v1.iso: ISO 9660 CD-ROM
filesystem data 'SOL_10_1106_X86 ' (bootable)

That's strange. That looks like the same file that I got here, and it works for me (past the GRUB screen at least). Just in case, here is the MD5 sum of my version:

Quoted 3 lines

c6f82c36195609b8dad67f61a927c7bc sol-10-u3-ga-x86-v1.iso

Mine's different: I ran out of disk space.

I'll re-download and try again (I killed the .zip).

Hello,

I tried to install Solaris 10 under KVM, without success, so Avi told me to post the details here. My CPU is an AMD X2, and I am using KVM trunk on a 2.6.19.2, started with the following command-line:

kvm -hda solaris.qcow -cdrom sol-10-u3-ga-x86-dvd.iso -boot d -m 512 -no-acpi

(I tried with and without -no-acpi).

If I follow the normal installation path, I soon get messages like:

svc.startd[7]: svc:/milestone/single-user:default: Method "/sbin/rcS start" failed due to signal ILL

The attached patch should fix it.

Quoted 1 line · Avi Kivity

The attached patch should fix it.

Update for those who wouldn't have followed IRC meanwhile: this patch does fix the bug for Opteron-based systems, but doesn't improve anything on my X2 4600+. Avi therefore suggests that someone (_joro?) with knowledge and access to an X2 has a look at it.

Quoted 9 lines · Waba

Quoted 2 lines · Avi Kivity

The attached patch should fix it.

Update for those who wouldn't have followed IRC meanwhile: this patch does fix the bug for Opteron-based systems, but doesn't improve anything on my X2 4600+. Avi therefore suggests that someone (_joro?) with knowledge and access to an X2 has a look at it.

I was able to reproduce the bug on a SVM machine here and did some deeper research. I intercepted the #UD exception and printed out the opcode. This opcode was all zero in the first time and changed randomly to other undefined values. I also saved the last exit code before the UD intercept and that was a PF intercept. The guest is in 32 bit PAE mode when this happens.
Regarding this research I assume this bug is not SVM related, I think something in the MMU goes wrong here.

Waba, can you apply the attached patch and post dmesg after the error occurs? (it also has a small fix which may help).

Quoted 3 lines

Waba, can you apply the attached patch and post dmesg after the error occurs? (it also has a small fix which may help).

No luck with the fix, it stills SIGILLs :( Here is the dmesg:

[ 4800.373717] cr0_wp: 1 (before)
[ 4808.442199] kvm: emulating exchange as write

ok. please keep the patch applied, and an addition:

- change '#undef AUDIT' to '#define AUDIT' in mmu.c
- in the same file, change 'static int dbg = 1;' to 'static int dbg = 0;'
- 'echo 9 > /proc/sysrq-trigger'

and run again. Note that kvm will be very slow with this. Watch dmesg for errors from kvm.

Quoted 9 lines · Avi Kivity

ok. please keep the patch applied, and an addition:

- change '#undef AUDIT' to '#define AUDIT' in mmu.c
- in the same file, change 'static int dbg = 1;' to 'static int dbg = 0;'
- 'echo 9 > /proc/sysrq-trigger'

and run again. Note that kvm will be very slow with this. Watch dmesg for errors from kvm.

When you say 'very slow', you mean it :) I couldn't reproduce the bug in the usual fashion because the involved processes timed out and were killed by the service manager.

However, simply invoking "svccfg" at the prompt seems to bring in Java or whatever heavy stuff is causing our problem, and I got a nice SIGILL this way.

Sadly, nothing interesting appeared in the log:

[ 8544.402301] Loglevel set to 9
[ 8548.697286] cr0_wp: 1 (before)
[ 9868.532303] kvm: emulating exchange as write
[16785.771040] SysRq : Changing Loglevel
[16785.771049] Loglevel set to 0

Please tell me if there is anything else I can try out.

Quoted 29 lines · Waba

Quoted 9 lines · Avi Kivity

ok. please keep the patch applied, and an addition:

- change '#undef AUDIT' to '#define AUDIT' in mmu.c
- in the same file, change 'static int dbg = 1;' to 'static int dbg = 0;'
- 'echo 9 > /proc/sysrq-trigger'

and run again. Note that kvm will be very slow with this. Watch dmesg for errors from kvm.

When you say 'very slow', you mean it :) I couldn't reproduce the bug in the usual fashion because the involved processes timed out and were killed by the service manager.

However, simply invoking "svccfg" at the prompt seems to bring in Java or whatever heavy stuff is causing our problem, and I got a nice SIGILL this way.

Sadly, nothing interesting appeared in the log:

[ 8544.402301] Loglevel set to 9
[ 8548.697286] cr0_wp: 1 (before)
[ 9868.532303] kvm: emulating exchange as write
[16785.771040] SysRq : Changing Loglevel
[16785.771049] Loglevel set to 0

Quoted 2 lines

Please tell me if there is anything else I can try out.

Can you try to isolate the process which fails?

e.g. install with qemu, start in kvm in single user mode, and use truss or something similar.

Quoted 37 lines · Avi Kivity

Quoted 23 lines · Waba

Quoted 10 lines · Avi Kivity

ok. please keep the patch applied, and an addition:

- change '#undef AUDIT' to '#define AUDIT' in mmu.c
- in the same file, change 'static int dbg = 1;' to 'static int dbg = 0;'
- 'echo 9 > /proc/sysrq-trigger'

and run again. Note that kvm will be very slow with this. Watch dmesg for errors from kvm.

When you say 'very slow', you mean it :) I couldn't reproduce the bug in the usual fashion because the involved processes timed out and were killed by the service manager.

However, simply invoking "svccfg" at the prompt seems to bring in Java or whatever heavy stuff is causing our problem, and I got a nice SIGILL this way.

Sadly, nothing interesting appeared in the log:

Quoted 10 lines

[ 8544.402301] Loglevel set to 9
[ 8548.697286] cr0_wp: 1 (before)
[ 9868.532303] kvm: emulating exchange as write
[16785.771040] SysRq : Changing Loglevel
[16785.771049] Loglevel set to 0

Please tell me if there is anything else I can try out.

Can you try to isolate the process which fails?

Quoted 3 lines

e.g. install with qemu, start in kvm in single user mode, and use truss or something similar.

When I tried it I got a the SIGILL ever I invoked a new program. The first SIGILL is in the starting process which is killed then. Afterwards I get a shell. Every simple command there (like ls for example) caused a SIGILL then.

Quoted 40 lines · Joerg Roedel

Quoted 37 lines · Avi Kivity

Quoted 21 lines · Waba

Quoted 9 lines · Avi Kivity

ok. please keep the patch applied, and an addition:

- change '#undef AUDIT' to '#define AUDIT' in mmu.c
- in the same file, change 'static int dbg = 1;' to 'static int dbg = 0;'
- 'echo 9 > /proc/sysrq-trigger'

and run again. Note that kvm will be very slow with this. Watch dmesg for errors from kvm.

When you say 'very slow', you mean it :) I couldn't reproduce the bug in the usual fashion because the involved processes timed out and were killed by the service manager.

However, simply invoking "svccfg" at the prompt seems to bring in Java or whatever heavy stuff is causing our problem, and I got a nice SIGILL this way.

Sadly, nothing interesting appeared in the log:

Quoted 9 lines

[ 8544.402301] Loglevel set to 9
[ 8548.697286] cr0_wp: 1 (before)
[ 9868.532303] kvm: emulating exchange as write
[16785.771040] SysRq : Changing Loglevel
[16785.771049] Loglevel set to 0

Please tell me if there is anything else I can try out.

Can you try to isolate the process which fails?

e.g. install with qemu, start in kvm in single user mode, and use truss or something similar.

Quoted 6 lines

When I tried it I got a the SIGILL ever I invoked a new program. The first SIGILL is in the starting process which is killed then. Afterwards I get a shell. Every simple command there (like ls for example) caused a SIGILL then.

Was this with the cr0.wp patch applied (commit 4348)?

Quoted 54 lines · Avi Kivity

Quoted 52 lines · Joerg Roedel

Quoted 33 lines · Avi Kivity

Quoted 30 lines · Waba

Quoted 10 lines · Avi Kivity

ok. please keep the patch applied, and an addition:

- change '#undef AUDIT' to '#define AUDIT' in mmu.c
- in the same file, change 'static int dbg = 1;' to 'static int dbg = 0;'
- 'echo 9 > /proc/sysrq-trigger'

and run again. Note that kvm will be very slow with this. Watch dmesg for errors from kvm.

When you say 'very slow', you mean it :) I couldn't reproduce the bug in the usual fashion because the involved processes timed out and were killed by the service manager.

However, simply invoking "svccfg" at the prompt seems to bring in Java or whatever heavy stuff is causing our problem, and I got a nice SIGILL this way.

Sadly, nothing interesting appeared in the log:

[ 8544.402301] Loglevel set to 9
[ 8548.697286] cr0_wp: 1 (before)
[ 9868.532303] kvm: emulating exchange as write
[16785.771040] SysRq : Changing Loglevel
[16785.771049] Loglevel set to 0

Quoted 9 lines

Quoted 3 lines

Please tell me if there is anything else I can try out.

Can you try to isolate the process which fails?

e.g. install with qemu, start in kvm in single user mode, and use truss or something similar.

When I tried it I got a the SIGILL ever I invoked a new program. The first SIGILL is in the starting process which is killed then. Afterwards I get a shell. Every simple command there (like ls for example) caused a SIGILL then.

Quoted 2 lines

Was this with the cr0.wp patch applied (commit 4348)?

Yes. It was with kvm-current from last friday. The patch was already in there.

Quoted 59 lines · Joerg Roedel

Quoted 56 lines · Avi Kivity

Quoted 41 lines · Joerg Roedel

Quoted 38 lines · Avi Kivity

Quoted 23 lines · Waba

Quoted 10 lines · Avi Kivity

ok. please keep the patch applied, and an addition:

- change '#undef AUDIT' to '#define AUDIT' in mmu.c
- in the same file, change 'static int dbg = 1;' to 'static int dbg = 0;'
- 'echo 9 > /proc/sysrq-trigger'

and run again. Note that kvm will be very slow with this. Watch dmesg for errors from kvm.

When you say 'very slow', you mean it :) I couldn't reproduce the bug in the usual fashion because the involved processes timed out and were killed by the service manager.

However, simply invoking "svccfg" at the prompt seems to bring in Java or whatever heavy stuff is causing our problem, and I got a nice SIGILL this way.

Sadly, nothing interesting appeared in the log:

Quoted 10 lines

[ 8544.402301] Loglevel set to 9
[ 8548.697286] cr0_wp: 1 (before)
[ 9868.532303] kvm: emulating exchange as write
[16785.771040] SysRq : Changing Loglevel
[16785.771049] Loglevel set to 0

Please tell me if there is anything else I can try out.

Can you try to isolate the process which fails?

Quoted 11 lines

Quoted 4 lines

e.g. install with qemu, start in kvm in single user mode, and use truss or something similar.

When I tried it I got a the SIGILL ever I invoked a new program. The first SIGILL is in the starting process which is killed then. Afterwards I get a shell. Every simple command there (like ls for example) caused a SIGILL then.

Was this with the cr0.wp patch applied (commit 4348)?

Quoted 3 lines

Yes. It was with kvm-current from last friday. The patch was already in there.

Can this be sysenter support? i.e. is sysenter supported on opteron but not on athlon x2?

Quoted 3 lines

Can this be sysenter support? i.e. is sysenter supported on opteron but not on athlon x2?

AFAIK, on both Opteron an Athlon X2 Solaris uses syscall, rather than sysenter.
The catch here is that it starts with the least optimized libc at boot, and at some
point SMF uses moe(1) to find out most suitable version of libc and lofs-mount it
on top of the original. From this point on AMD machine will use syscall.

Quoted 5 lines · Cyril Plisko

Quoted 3 lines

Can this be sysenter support? i.e. is sysenter supported on opteron but not on athlon x2?

Quoted 8 lines

AFAIK, on both Opteron an Athlon X2 Solaris uses syscall, rather than sysenter.
The catch here is that it starts with the least optimized libc at boot, and at some
point SMF uses moe(1) to find out most suitable version of libc and lofs-mount it
on top of the original. From this point on AMD machine will use syscall.

Is this true on 32-bit solaris as well?

While the host is 64-bit, the guest is 32 in this case IIRC.

--
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Quoted 5 lines · Cyril Plisko

Quoted 3 lines

Can this be sysenter support? i.e. is sysenter supported on opteron but not on athlon x2?

Quoted 7 lines

AFAIK, on both Opteron an Athlon X2 Solaris uses syscall, rather than sysenter. The catch here is that it starts with the least optimized libc at boot, and at some
point SMF uses moe(1) to find out most suitable version of libc and lofs-mount it
on top of the original. From this point on AMD machine will use syscall.

Is this true on 32-bit solaris as well?

While the host is 64-bit, the guest is 32 in this case IIRC.

Quoted 14 lines · Avi Kivity

Quoted 12 lines · Cyril Plisko

Quoted 3 lines

Can this be sysenter support? i.e. is sysenter supported on opteron but not on athlon x2?

AFAIK, on both Opteron an Athlon X2 Solaris uses syscall, rather than sysenter. The catch here is that it starts with the least optimized libc at boot, and at some
point SMF uses moe(1) to find out most suitable version of libc and lofs-mount it
on top of the original. From this point on AMD machine will use syscall.

Quoted 3 lines

Is this true on 32-bit solaris as well?

While the host is 64-bit, the guest is 32 in this case IIRC.

Yeah, the guest is running in 32bit PAE when the SIGILLs happen.

Joerg

--
Joerg Roedel
Operating System Research Center
AMD Saxony LLC & Co. KG

It took me a while, but I figured it out... nearly!

Everything SIGILLs after the fs-root service is started. Its start method does several things, but the problematic bit is replacing the libc with an optimised version (namely, /usr/lib/libc/libc_hwcap1.so.1, which makes use of the SSE, MMX, CMOV, SEP and FPU instruction sets according to file(1)).

All these flags are indeed advertised in the CPUID (isainfo -v: sse2 sse fxsr mmx cmov sep cx8 tsc fpu)). If the amd_sysc bit had been present, the hwcap2 version would have been selected by moe(1), I guess (adds SSE2 support and replaces SEP by AMD_SYSC).

Disabling the libc replacement in /lib/svc/method/fs-root entirely workarounds the problem.

Further investigating, I tricked ls(1) into using the optimised libc through LD_LIBRARY_PATH and obtained a core. mdb(1) told me that the culprit was hiding at libc`memset+0x74. And finally, dis(1) revealed that the faulty instruction is "movups (%esp), %xmm0", a SSE feature. The %xmm0 register is apparently for storage purposes only, as the only instructions used to access it are movups, movntps and movaps.

At this point I hope that it makes a lot of sense to you, because I have no idea why it works fine on Avi's Opteron, etc.

Let me know if you need any additional debugging.

Can you post the host's /proc/cpuinfo? I'll compare it with my opteron.

Quoted 2 lines · Avi Kivity

Can you post the host's /proc/cpuinfo? I'll compare it with my opteron.

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow pni cx16 lahf_lm cmp_legacy svm cr8_legacy

It took me a while, but I figured it out... nearly!

Everything SIGILLs after the fs-root service is started. Its start method does several things, but the problematic bit is replacing the libc with an optimised version (namely, /usr/lib/libc/libc_hwcap1.so.1, which makes use of the SSE, MMX, CMOV, SEP and FPU instruction sets according to file(1)).

All these flags are indeed advertised in the CPUID (isainfo -v: sse2 sse fxsr mmx cmov sep cx8 tsc fpu)). If the amd_sysc bit had been present, the hwcap2 version would have been selected by moe(1), I guess (adds SSE2 support and replaces SEP by AMD_SYSC).

Disabling the libc replacement in /lib/svc/method/fs-root entirely workarounds the problem.

Further investigating, I tricked ls(1) into using the optimised libc through LD_LIBRARY_PATH and obtained a core. mdb(1) told me that the culprit was hiding at libc`memset+0x74. And finally, dis(1) revealed that the faulty instruction is "movups (%esp), %xmm0", a SSE feature. The %xmm0 register is apparently for storage purposes only, as the only instructions used to access it are movups, movntps and movaps.

At this point I hope that it makes a lot of sense to you, because I have no idea why it works fine on Avi's Opteron, etc.

Let me know if you need any additional debugging.

Quoted 1 line

Quoted 1 line

On Mon, Feb 12, 2007 at 4:48 AM, in message <45D03801.4040006-atKUWr5tajBWk0Htik3J/[email protected]>,

Quoted 23 lines · Waba

It took me a while, but I figured it out... nearly!

Everything SIGILLs after the fs- root service is started. Its start method does several things, but the problematic bit is replacing the libc with an optimised version (namely, /usr/lib/libc/libc_hwcap1.so.1, which makes use of the SSE, MMX, CMOV, SEP and FPU instruction sets according to file(1)).

All these flags are indeed advertised in the CPUID (isainfo - v: sse2 sse fxsr mmx cmov sep cx8 tsc fpu)). If the amd_sysc bit had been present, the hwcap2 version would have been selected by moe(1), I guess (adds SSE2 support and replaces SEP by AMD_SYSC).

Disabling the libc replacement in /lib/svc/method/fs- root entirely workarounds the problem.

Further investigating, I tricked ls(1) into using the optimised libc through LD_LIBRARY_PATH and obtained a core. mdb(1) told me that the culprit was hiding at libc`memset+0x74. And finally, dis(1) revealed that the faulty instruction is "movups (%esp), %xmm0", a SSE feature. The %xmm0 register is apparently for storage purposes only, as the only instructions used to access it are movups, movntps and movaps.

Quoted 5 lines

At this point I hope that it makes a lot of sense to you, because I have no idea why it works fine on Avi's Opteron, etc.

Let me know if you need any additional debugging.

Let's look at the control registers at the time of the SIGILL. Can you reproduce the error with the attached patch and send dmesg?

Quoted 6 lines · Avi Kivity

Quoted 5 lines · Waba

It took me a while, but I figured it out... nearly!

Everything SIGILLs after the fs- root service is started. Its start method does several things, but the problematic bit is replacing the libc with an optimised version (namely,

Quoted 4 lines

Quoted 4 lines

which makes use of the SSE, MMX, CMOV, SEP and FPU instruction sets according to file(1)).

All these flags are indeed advertised in the CPUID (isainfo - v:

sse

Quoted 1 line

Quoted 1 line

fxsr mmx cmov sep cx8 tsc fpu)). If the amd_sysc bit had been

Quoted 10 lines

Quoted 10 lines

the hwcap2 version would have been selected by moe(1), I guess (adds SSE2 support and replaces SEP by AMD_SYSC).

Disabling the libc replacement in /lib/svc/method/fs- root entirely workarounds the problem.

Further investigating, I tricked ls(1) into using the optimised libc through LD_LIBRARY_PATH and obtained a core. mdb(1) told me that the culprit was hiding at libc`memset+0x74. And finally, dis(1) revealed that the faulty instruction is "movups (%esp), %xmm0", a SSE

Quoted 1 line

Quoted 1 line

The %xmm0 register is apparently for storage purposes only, as the

Quoted 9 lines

Quoted 7 lines

instructions used to access it are movups, movntps and movaps.

At this point I hope that it makes a lot of sense to you, because I have no idea why it works fine on Avi's Opteron, etc.

Let me know if you need any additional debugging.

Let's look at the control registers at the time of the SIGILL. Can

Quoted 1 line

reproduce the error with the attached patch and send dmesg?

Hi Avi,
I have a sneaking suspicion that this may be the same root-cause of

findings with #UD on SLES. I wrote a program that allows you to take

sum pages of a running program's text sections and compare them. I

compared the output of GRUB running on bare-metal and as a KVM guest

they were identical (except for the expected text that is affected by relocation). This was not what I was expecting since we speculated MMU corruption. Admittedly the test is not conclusive since the page

could surely be different under the load of the target apps execution verses the delta program. But I was hoping for a smoking gun ;)

Note that I am seeing #UD under other apps as well (Firefox for

If there were a disparity between the advertised and actual CPUID flags

SLES is using libraries that interpret the flags, that could explain

behavior here. Note that grub is blowing up in libc for me as well. I will explore a CPUID disparity as a possibility next and report back. What I did notice is that KVM seems to report the CPU as an AMD, even though I am running on a Woodcrest. I would speculate that the problem

that some AMD specific flag (e.g. amd_sysc) is set when it should not

Note that I am currently being pulled off KVM work for about a week so

will be silent for a bit.

-Greg

-----------------------------------------------------------------------

Using Tomcat but need to do more? Need to support web services,

Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache

http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=12164

_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Let's look at the control registers at the time of the SIGILL. Can you reproduce the error with the attached patch and send dmesg?

Quoted 2 lines · Avi Kivity

Let's look at the control registers at the time of the SIGILL. Can you reproduce the error with the attached patch and send dmesg?

#ud: cr0 8005002b (8005003b) cr4 b8 (b8)

Qemu also aborted with "unhandled vm exit: 08" or similar, but I guess that the important part is the printk.

bit 9 of cr4 (osfxsr) is clear, which according to the docs generates #ud on any sse instruction.

Quoted 3 lines · Avi Kivity

bit 9 of cr4 (osfxsr) is clear, which according to the docs generates #ud on any sse instruction.

I still have no idea why this bit was not set when running on my CPU, but with the register set up this way, no more SIGILL. I re-enabled the libc mount and rebooted the guest to be sure, and it does work!

Well, there's probably an emulator bug somewhere.

Can you add a printk() to set_cr4() in kvm_main.c and see what the guest does? The documentation states that it's up to the OS to enable the bit, so I can't just apply the previous patch, even though it fixes the problem.

Quoted 7 lines · Avi Kivity

Well, there's probably an emulator bug somewhere.

Can you add a printk() to set_cr4() in kvm_main.c and see what the guest does? The documentation states that it's up to the OS to enable the bit, so I can't just apply the previous patch, even though it fixes the problem.

set_cr4: set 0000000000000010
set_cr4: set 0000000000000090
set_cr4: set 0000000000000098
set_cr4: set 00000000000000b8

Here is it. If I get it right, the guest never sets 0x200 then ?

-Waba.

PS: I'll be on holidays and unable to read my mail until Thursday.

It took me a while, but I figured it out... nearly!

Further investigating, I tricked ls(1) into using the optimised libc through LD_LIBRARY_PATH and obtained a core. mdb(1) told me that the culprit was hiding at libc`memset+0x74. And finally, dis(1) revealed that the faulty instruction is "movups (%esp), %xmm0", a SSE feature. The %xmm0 register is apparently for storage purposes only, as the only instructions used to access it are movups, movntps and movaps.

At this point I hope that it makes a lot of sense to you, because I have no idea why it works fine on Avi's Opteron, etc.