Setting nx bit in virtual CPU

Hello,

Summary: How can I have a virtual CPU with the nx bit set whilst enjoying KVM acceleration?

My Host - AMD Athlon(tm) 64 Processor 3200+ running Gentoo My VM - KVM running hardened Gentoo
My KVM version - 0.12.3
My Task - Implement restricted secure VM to handle services exposed to internet.
My Command - kvm -hda /dev/mapper/vols-andrew -kernel ./bzImage -append root=/dev/hda2 -cpu host -runas xxx -net nic -net user -m 256 -k en-gb -vnc :1 -monitor stdio

In order to maximise the security of my VM, I have enabled PaX which is supposed to prevent various address space attacks. Sadly, when I run 'paxtest' it reports that my VM is still vulnerable. I have concluded that the problem is most likely caused by the virtual CPU not having the nx bit set.

Flags in virtual CPU: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall mmxext fxsr_opt lm rep_good pni cx16 lahf_lm

Flags in host CPU: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good nopl pni cx16 lahf_lm svm extapic cr8_legacy

As you can see, despite using the '-cpu host' command, several host flags, including nx, are missing in the VM. Setting '-cpu host,+nx' doesn't make any difference.

If however, I remove the '-cpu host' option and add the '-no-kvm' option the virtual CPU has the nx flag and paxtest reports that my VM is secure. Of course the down side is that everything runs much slower.

Confusingly, the following page about tuning KVM
(http://www.linux-kvm.org/page/Tuning_KVM) lists the flags for the default qemu64 cpu and nx is clearly included. But, when I set '-cpu qemu64' I get a model name of QEMU Virtual CPU, but no sign of an nx bit.

So, is there any way of having the nx bit and the benefits of KVM acceleration.

Thank you.

WFM here (both current git tree and 0.12.3) w/ either -cpu host or -cpu qemu64. The code definitly does what you'd expect in both those cases.

thanks,
-chris

OK, thanks for that. Clearly something wrong with my installation. At least now I know it is possible I can keep fiddling until it works.

Richard

Are you running a 32-bit non-pae host kernel? In that case, nx is disabled both for the guest and host. Switch to a pae (or 64-bit) kernel and all should be well.

error compiling committee.c: too many arguments to function

Nope, both Kernels are 64 bit.

uname -a Host: Linux gordon 2.6.27-gentoo-r8 #5 Sat Mar 14 18:01:59 GMT 2009 x86_64 AMD Athlon(tm) 64 Processor 3200+ AuthenticAMD GNU/Linux

uname -a Guest: Linux andrew 2.6.28-hardened-r9 #4 Mon Jan 18 22:39:31 GMT 2010 x86_64 AMD Athlon(tm) 64 Processor 3200+ AuthenticAMD GNU/Linux

As you can see, both kernels are a little old, and I have been wondering if that might be part of the problem. The Guest one is old because that is the latest stable hardened version in Gentoo. The host one is old because of:

(gordon:~) rs10% uptime
22:01:37 up 374 days, 23:29, 1 user, load average: 1.09, 0.42, 0.18

Now that I have managed to smash the psychologically important 1 year uptime for the first time ever (Woo!) I shall probably upgrade the host kernel in the near future. Of course, it is important to remember that with the --no-kvm switch it works just fine (only slowly) with exactly the same two kernels.

Thanks

2.6.27 should be plenty fine for nx. Really the important bit is that the host kernel has nx enabled. Can you check if that is so?

Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Umm, could you give me a clue about how to do that. It is some time since I configured the host kernel, but I do have a /proc/config.gz. Could I check by looking in that?

Thanks

The attached script should verify it.

Do not meddle in the internals of kernels, for they are subtle and quick to panic.

#!/usr/bin/python

class msr(object):

    def __init__(self):
        try:
            self.f = file('/dev/cpu/0/msr')
        except:
            self.f = file('/dev/msr0')
    def read(self, index, default = None):
        import struct
        self.f.seek(index)
        try:
            return struct.unpack('Q', self.f.read(8))[0]
        except:
            return default

efer = msr().read(0xc0000080, 0)
nx = (efer >> 11) & 1

if nx:
print 'nx: enabled'
else:
print 'nx: disabled'

rs% ./check-nx
Traceback (most recent call last):
File "./check-nx", line 17, in <module>

    efer = msr().read(0xc0000080, 0)
  File "./check-nx", line 8, in __init__
    self.f = file('/dev/msr0')

IOError: [Errno 2] No such file or directory: '/dev/msr0'

Sorry!

Run as root, please. And check first that you have a file named /dev/cpu/0/msr.

I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Doh!

gordon Code # ./check-nx
nx: enabled
gordon Code #

OK, seems to be enabled just fine. Any other ideas? I am beginning to get that horrible feeling that there isn't a real problem and it is just me being dumb!

I really hope so, because I am out of ideas... :)

Can you verify check-nx returns disabled on the guest? Does /proc/cpuinfo show nx in the guest?

I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

OK, time for a summary:

Host: /proc/cpuinfo shows 'nx' and check-nx shows 'enabled'

Guest: /proc/cpuinfo doesn't show nx and check-nx shows 'disabled'

Guest (with -no-kvm option): /proc/cpuinfo shows 'nx', but check-nx still shows 'disabled'

Below I have included all the listings which I think might be useful, but if you would like to see anything else then please ask.

HOST:

/proc/cpuinfo

processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 79
model name : AMD Athlon(tm) 64 Processor 3200+
stepping : 2
cpu MHz : 1000.000
cache size : 512 KB
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good nopl pni cx16 lahf_lm svm extapic cr8_legacy
bogomips : 2000.06
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc

GUEST with command line - kvm -hda /dev/mapper/vols-andrew -kernel ./bzImage -append root=/dev/hda2 -cpu host -runas xx -net nic -net user -m 256 -k en-gb -vnc :1 -monitor stdio

/proc/cpuinfo

processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 79
model name : AMD Athlon(tm) 64 Processor 3200+
stepping : 2
cpu MHz : 10000.330
cache size : 512 KB
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall mmxext fxsr_opt lm rep_good pni cx16 lahf_lm
bogomips : 2000.06
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

Results of paxtest

PaXtest - Copyright(c) 2003,2004 by Peter Busser <[email protected]> Released under the GNU Public Licence version 2 or later

Mode: kiddie
Linux andrew 2.6.28-hardened-r9 #4 Mon Jan 18 22:39:31 GMT 2010 x86_64 AMD Athlon(tm) 64 Processor 3200+ AuthenticAMD GNU/Linux

Executable anonymous mapping             : Vulnerable
Executable bss                           : Vulnerable
Executable data                          : Vulnerable
Executable heap                          : Vulnerable
Executable stack                         : Vulnerable
Executable anonymous mapping (mprotect)  : Vulnerable
Executable bss (mprotect)                : Vulnerable
Executable data (mprotect)               : Vulnerable
Executable heap (mprotect)               : Vulnerable
Executable stack (mprotect)              : Vulnerable
Executable shared library bss (mprotect) : Vulnerable
Executable shared library data (mprotect): Vulnerable
Writable text segments                   : Killed
Anonymous mapping randomisation test     : 33 bits (guessed)
Heap randomisation test (ET_EXEC)        : 13 bits (guessed)
Heap randomisation test (ET_DYN)         : 40 bits (guessed)
Main executable randomisation (ET_EXEC)  : No randomisation
Main executable randomisation (ET_DYN)   : 12 bits (guessed)
Shared library randomisation test        : 33 bits (guessed)
Stack randomisation test (SEGMEXEC)      : 40 bits (guessed)
Stack randomisation test (PAGEEXEC)      : 40 bits (guessed)
Return to function (strcpy)              : paxtest: bad luck, try

different compiler options.
Return to function (memcpy) : *** buffer overflow detected
***: rettofunc2 - terminated
rettofunc2: buffer overflow attack in function <unknown> - terminated Report to http://bugs.gentoo.org/
Killed
Return to function (strcpy, RANDEXEC) : paxtest: bad luck, try
different compiler options.
Return to function (memcpy, RANDEXEC) : *** buffer overflow detected
***: rettofunc2x - terminated
rettofunc2x: buffer overflow attack in function <unknown> - terminated Report to http://bugs.gentoo.org/
Killed
Executable shared library bss : Killed
Executable shared library data : Killed

GUEST with command line - kvm -hda /dev/mapper/vols-andrew -kernel ./bzImage -append root=/dev/hda2 -no-kvm -runas xx -net nic -net user -m 256 -k en-gb -vnc :1 -monitor stdio

/proc/cpuinfo

processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 2
model name : QEMU Virtual CPU version 0.12.3
stepping : 3
cpu MHz : 1998.067
cache size : 512 KB
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm pni cx16 lahf_lm svm abm sse4a
bogomips : 3996.13
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

Results of paxtest

PaXtest - Copyright(c) 2003,2004 by Peter Busser <[email protected]> Released under the GNU Public Licence version 2 or later

Mode: kiddie
Linux andrew 2.6.28-hardened-r9 #4 Mon Jan 18 22:39:31 GMT 2010 x86_64 QEMU Virtual CPU version 0.12.3 AuthenticAMD GNU/Linux

Executable anonymous mapping             : Killed
Executable bss                           : Killed
Executable data                          : Killed
Executable heap                          : Killed
Executable stack                         : Killed
Executable anonymous mapping (mprotect)  : Killed
Executable bss (mprotect)                : Killed
Executable data (mprotect)               : Killed
Executable heap (mprotect)               : Killed
Executable stack (mprotect)              : Killed
Executable shared library bss (mprotect) : Killed
Executable shared library data (mprotect): Killed
Writable text segments                   : Killed
Anonymous mapping randomisation test     : 33 bits (guessed)
Heap randomisation test (ET_EXEC)        : 13 bits (guessed)
Heap randomisation test (ET_DYN)         : 40 bits (guessed)
Main executable randomisation (ET_EXEC)  : No randomisation
Main executable randomisation (ET_DYN)   : 12 bits (guessed)
Shared library randomisation test        : 33 bits (guessed)
Stack randomisation test (SEGMEXEC)      : 40 bits (guessed)
Stack randomisation test (PAGEEXEC)      : 40 bits (guessed)
Return to function (strcpy)              : paxtest: bad luck, try

different compiler options.
Return to function (memcpy) : *** buffer overflow detected
***: rettofunc2 - terminated
rettofunc2: buffer overflow attack in function <unknown> - terminated Report to http://bugs.gentoo.org/
Killed
Return to function (strcpy, RANDEXEC) : paxtest: bad luck, try
different compiler options.
Return to function (memcpy, RANDEXEC) : *** buffer overflow detected
***: rettofunc2x - terminated
rettofunc2x: buffer overflow attack in function <unknown> - terminated Report to http://bugs.gentoo.org/
Killed
Executable shared library bss : Killed
Executable shared library data : Killed

Strange. Can you hack qemu-kvm's cpuid code where it issues the ioctl KVM_SET_CPUID2 to show what the data is? I'm not where that code is in your version of qemu-kvm.

error compiling committee.c: too many arguments to function

Gad, the last time I tried to mess around with this sort of low level code was many years ago when I was a keen young bachelor burning the midnight oil trying to get the weird IDE controller on my Alpha to work properly! Anyway, I have tried to give it a go.

I found a file called qemu-kvm-x86.c

It contained a function called kvm_setup_cpuid2 which I modified as follows:

int kvm_setup_cpuid2(CPUState *env, int nent,
struct kvm_cpuid_entry2 *entries)

{
        struct kvm_cpuid2 *cpuid;
        int r, i;
        fprintf(stderr, "cpuid=nent %d\n", nent);
        for (i=0; i < nent; i++) {
                fprintf(stderr, "%x %x %x %x %x %x %x\n",
entries[i].function, entries[i].index, entries[i].flags, entries[i].eax,
entries[i].ebx, entries[i].ecx, entries[i].edx);
                }
        cpuid = qemu_malloc(sizeof(*cpuid) + nent * sizeof(*entries));

        cpuid->nent = nent;
        memcpy(cpuid->entries, entries, nent * sizeof(*entries));
        r = kvm_vcpu_ioctl(env, KVM_SET_CPUID2, cpuid);
        free(cpuid);
        return r;
}

So, basically I go round a loop and print out the contents of each kvm_cpuid_entry2 structure.

Results below, using Andre Przywara's handy nano-kernel. I do hope that some of this makes some kind of sense!

qemu-kvm -kernel cpuid_mb -vnc :0

cpuid=nent 21
40000000 0 0 0 4b4d564b 564b4d56 4d
40000001 0 0 7 0 0 0
0 0 0 4 68747541 444d4163 69746e65
1 0 0 623 800 80002001 78bfbfd
2 0 0 1 0 0 2c307d
3 0 0 0 0 0 0
4 0 1 121 1c0003f 3f 1
4 1 1 122 1c0003f 3f 1
4 2 1 143 3c0003f fff 1
4 3 1 0 0 0 0
80000000 0 0 8000000a 68747541 444d4163 69746e65
80000001 0 0 623 0 1 2181abfd
80000002 0 0 554d4551 72695620 6c617574 55504320
80000003 0 0 72657620 6e6f6973 312e3020 332e32
80000004 0 0 0 0 0 0
80000005 0 0 1ff01ff 1ff01ff 40020140 40020140
80000006 0 0 0 42004200 2008140 0
80000007 0 0 0 0 0 0
80000008 0 0 3028 0 0 0
80000009 0 0 0 0 0 0
8000000a 0 0 1 10 0 0

qemu-kvm -kernel cpuid_mb -cpu host -vnc :0

cpuid=nent 29
40000000 0 0 0 4b4d564b 564b4d56 4d
40000001 0 0 7 0 0 0
0 0 0 1 68747541 444d4163 69746e65
1 0 0 40ff2 800 80002001 78bfbff
80000000 0 0 80000018 68747541 444d4163 69746e65
80000001 0 0 40ff2 0 1 23c3fbff
80000002 0 0 20444d41 6c687441 74286e6f 3620296d
80000003 0 0 72502034 7365636f 20726f73 30303233
80000004 0 0 2b 0 0 0
80000005 0 0 1ff01ff 1ff01ff 40020140 40020140
80000006 0 0 0 42004200 2008140 0
80000007 0 0 0 0 0 0
80000008 0 0 3028 0 0 0
80000009 0 0 0 0 0 0
8000000a 0 0 1 10 0 0
8000000b 0 0 0 0 0 0
8000000c 0 0 0 0 0 0
8000000d 0 0 0 0 0 0
8000000e 0 0 0 0 0 0
8000000f 0 0 0 0 0 0
80000010 0 0 0 0 0 0
80000011 0 0 0 0 0 0
80000012 0 0 0 0 0 0
80000013 0 0 0 0 0 0
80000014 0 0 0 0 0 0
80000015 0 0 0 0 0 0
80000016 0 0 0 0 0 0
80000017 0 0 0 0 0 0
80000018 0 0 0 0 0 0

If I try with -no-kvm then nothing gets printed, presumably because this is a kvm specific function and doesn't get called in that case.

80000001 edx bit 20 is NX, which is cleared. So it is qemu-kvm at fault here.

I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Can you try to boot the attached multiboot "kernel", which just outputs a brief CPUID dump?

$ qemu-kvm -kernel cpuid_mb -vnc :0
(Unfortunately I have no serial console support in there yet, so you either have to write the values down or screenshot it).
In the 4th line from the button it should print NX (after SYSCALL).

Regards,
Andre.

Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 448-3567-12

OK, that was fun! Resulting screen shots are attached.

...default.png With command line above.
...cpu_host.png With -cpu host option added.
...no_kvm.png With -no-kvm option added.

I hope that helps!

Richard

OK, AFAIK there are several flags missing. I dimly remember there was a bug with masking the CPUID bits in older kernels, so I guess you have to celebrate your uptime for the last time and then give it a reboot with a more up-to-date host kernel. (I also rebooted my desktop after I made the one year and now am gone green with turning it off over night ;-)
Maybe you get around with rebuilding fixed versions of kvm.ko and kvm_amd.ko, I can provide a fix for you if you wish (please point me to a way to get the actual kernel source you use).
The userspace was up-to-date? (qemu-kvm 0.12.3)?

Regards,
Andre.

Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 488-3567-12

Well, its funny how these things turn out. On Saturday, my better half insisted that I replace a mains socket. It wasn't on the same ring main as the server so I just switched off the relevant circuit breaker and got on with it. Unfortunately, whilst I was doing the work the main RCD tripped, probably because I let the Neutral and Earth touch which can cause that to happen sometimes. Anyway, by the time I realised there was a problem the old batteries in my UPS had given up and my uptime was over with a grand total of 382 days, 12:07:24! If this had happened a few weeks ago I would probably have thrown myself from the roof! In my previous house, I installed a special circuit for the server not protected by an RCD especially to avoid problems of this sort. Adding one in my current house has now moved a long way up my "To Do" list!

The server has started again, but a few things still aren't happy. As soon as I have them sorted I will step up to the latest kernel and let you know what difference it makes.

Thanks,

Richard