x86: Fixed Unable to execute init process since glibc version 2.33
PATCH 1/1 x86: Fixed Unable to execute init process since glibc version 2.33
From: Dongli Si <[email protected]>
glibc detected invalid CPU Vendor name will cause an error:
[ 0.450127] Run /sbin/init as init process /lib64/libc.so.6: CPU ISA level is lower than required [ 0.451931] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00 [ 0.452117] CPU: 0 PID: 1 Comm: init Not tainted 5.17.0-rc1 #72
--- x86/cpuid.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
x86/cpuid.c +9 -5
@@ -2,6 +2,7 @@#include "kvm/kvm.h" #include "kvm/util.h" +#include "kvm/cpufeature.h" #include <sys/ioctl.h> #include <stdlib.h>
@@ -10,7 +11,7 @@static void filter_cpuid (struct kvm_cpuid2 *kvm_cpuid) { -unsigned int signature[3 ]; +struct cpuid_regs regs;unsigned int i;/*
@@ -22,10 +23,13 @@ static void filter_cpuid(struct kvm_cpuid2 *kvm_cpuid)switch (entry->function) {case 0 :/* Vendor name */ -memcpy (signature,"LKVMLKVMLKVM" ,12 ); - entry->ebx = signature[0 ]; - entry->ecx = signature[1 ]; - entry->edx = signature[2 ]; + regs = (struct cpuid_regs ) { + .eax =0x00 , + }; +host_cpuid (®s); + entry->ebx = regs.ebx; + entry->ecx = regs.ecx; + entry->edx = regs.edx;break ;case 1 :/* Set X86_FEATURE_HYPERVISOR */
Signature · 1 line
2.32.0
Hi,
Quoted 8 lines · Dongli Si
From: Dongli Si <[email protected]>
glibc detected invalid CPU Vendor name will cause an error:
[ 0.450127] Run /sbin/init as init process /lib64/libc.so.6: CPU ISA level is lower than required [ 0.451931] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00 [ 0.452117] CPU: 0 PID: 1 Comm: init Not tainted 5.17.0-rc1 #72
This looks bad. I'll try to reproduce it and I'll give my Tested-by tag after I test it.
Thanks,
Alex
Quoted 48 lines
Signed-off-by Dongli Si <[email protected]>--- x86/cpuid.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)x86/cpuid.c +9
-5@@ -2,6 +2,7 @@#include "kvm/kvm.h" #include "kvm/util.h" +#include "kvm/cpufeature.h" #include <sys/ioctl.h> #include <stdlib.h> @@ -10,7 +11,7 @@static void filter_cpuid (struct kvm_cpuid2 *kvm_cpuid) { -unsigned int signature[3 ]; +struct cpuid_regs regs;unsigned int i;/* @@ -22,10 +23,13 @@ static void filter_cpuid(struct kvm_cpuid2 *kvm_cpuid)switch (entry->function) {case 0 :/* Vendor name */ -memcpy (signature,"LKVMLKVMLKVM" ,12 ); - entry->ebx = signature[0 ]; - entry->ecx = signature[1 ]; - entry->edx = signature[2 ]; + regs = (struct cpuid_regs ) { + .eax =0x00 , + }; +host_cpuid (®s); + entry->ebx = regs.ebx; + entry->ecx = regs.ecx; + entry->edx = regs.edx;break ;case 1 :/* Set X86_FEATURE_HYPERVISOR */ Signature · 1 line
2.32.0
Hi,
Hi, I was unable to apply the patch as is because your base is missing commit 20b93be583f6 ("x86: Set the correct APIC ID"). Changing it is so it applies was trivial though.
I tested the patch with a VM image using glibc 2.36 and Linux v5.17-rc7, the glibc error went away and I was able to boot to userspace; as an added bonus, this kernel message went away:
[ 0.000000] CPU: vendor_id 'LKVMLKVMLKVM' unknown, using generic init. [ 0.000000] CPU: Your system may be unstable.
But this warning is new:
[ 0.000000] [Firmware Bug]: TSC doesn't count with P0 frequency!
I don't know what causes it (kvmtool's bios implementation is not feature complete with regards to my CPU features?), and doesn't look like it's something introduced by this patch, so as far as I'm concerned the patch is working as intended:
Thanks,
Alex
Quoted 56 lines · Dongli Si
From: Dongli Si <[email protected]>
glibc detected invalid CPU Vendor name will cause an error:
[ 0.450127] Run /sbin/init as init process /lib64/libc.so.6: CPU ISA level is lower than required [ 0.451931] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00 [ 0.452117] CPU: 0 PID: 1 Comm: init Not tainted 5.17.0-rc1 #72Signed-off-by Dongli Si <[email protected]>--- x86/cpuid.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)x86/cpuid.c +9
-5@@ -2,6 +2,7 @@#include "kvm/kvm.h" #include "kvm/util.h" +#include "kvm/cpufeature.h" #include <sys/ioctl.h> #include <stdlib.h> @@ -10,7 +11,7 @@static void filter_cpuid (struct kvm_cpuid2 *kvm_cpuid) { -unsigned int signature[3 ]; +struct cpuid_regs regs;unsigned int i;/* @@ -22,10 +23,13 @@ static void filter_cpuid(struct kvm_cpuid2 *kvm_cpuid)switch (entry->function) {case 0 :/* Vendor name */ -memcpy (signature,"LKVMLKVMLKVM" ,12 ); - entry->ebx = signature[0 ]; - entry->ecx = signature[1 ]; - entry->edx = signature[2 ]; + regs = (struct cpuid_regs ) { + .eax =0x00 , + }; +host_cpuid (®s); + entry->ebx = regs.ebx; + entry->ecx = regs.ecx; + entry->edx = regs.edx;break ;case 1 :/* Set X86_FEATURE_HYPERVISOR */ Signature · 1 line
2.32.0
On Sat, 26 Feb 2022 14:00:48 +0800
Dongli Si <[email protected]> wrote:
Hi,
Quoted 40 lines
From: Dongli Si <[email protected]>
glibc detected invalid CPU Vendor name will cause an error:
[ 0.450127] Run /sbin/init as init process /lib64/libc.so.6: CPU ISA level is lower than required [ 0.451931] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00 [ 0.452117] CPU: 0 PID: 1 Comm: init Not tainted 5.17.0-rc1 #72Signed-off-by Dongli Si <[email protected]>--- x86/cpuid.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)x86/cpuid.c +2
-2@@ -2,6 +2,7 @@#include "kvm/kvm.h" #include "kvm/util.h" +#include "kvm/cpufeature.h" #include <sys/ioctl.h> #include <stdlib.h> @@ -10,7 +11,7 @@static void filter_cpuid (struct kvm_cpuid2 *kvm_cpuid) { -unsigned int signature[3 ]; +struct cpuid_regs regs;unsigned int i;/* @@ -22,10 +23,13 @@ static void filter_cpuid(struct kvm_cpuid2 *kvm_cpuid)switch (entry->function) {case 0 :/* Vendor name */ -memcpy (signature,"LKVMLKVMLKVM" ,12 );
x86/cpuid.c static void filter_cpuid(struct kvm_cpuid2 *kvm_cpuid) · line 32
Quoted 10 lines
x86/cpuid.c +7
-3- entry->ebx = signature[0 ]; - entry->ecx = signature[1 ]; - entry->edx = signature[2 ]; + regs = (struct cpuid_regs ) { + .eax =0x00 , + }; +host_cpuid (®s); + entry->ebx = regs.ebx; + entry->ecx = regs.ecx; + entry->edx = regs.edx;
But that's redundant, isn't it? We already get the host vendor ID in the three registers in entry, and the current code is just there to overwrite this. So just removing the whole "case 0:" part should do the trick.
Also please be aware that there was a reason for this fixup, as explained in commit bc0b99a2a740 ("kvm tools: Filter out CPU vendor string").
Alex, did you boot this on an AMD box, to spot if this is still an issue?
Cheers,
Andre
Quoted 3 lines
x86/cpuid.c +0
-0break ;case 1 :/* Set X86_FEATURE_HYPERVISOR */
Hi,
Quoted 60 lines · Andre Przywara
On Sat, 26 Feb 2022 14:00:48 +0800
Dongli Si <[email protected]> wrote:Hi,
Quoted 40 lines
From: Dongli Si <[email protected]>
glibc detected invalid CPU Vendor name will cause an error:
[ 0.450127] Run /sbin/init as init process /lib64/libc.so.6: CPU ISA level is lower than required [ 0.451931] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00 [ 0.452117] CPU: 0 PID: 1 Comm: init Not tainted 5.17.0-rc1 #72Signed-off-by Dongli Si <[email protected]>--- x86/cpuid.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)x86/cpuid.c +2
-2@@ -2,6 +2,7 @@#include "kvm/kvm.h" #include "kvm/util.h" +#include "kvm/cpufeature.h" #include <sys/ioctl.h> #include <stdlib.h> @@ -10,7 +11,7 @@static void filter_cpuid (struct kvm_cpuid2 *kvm_cpuid) { -unsigned int signature[3 ]; +struct cpuid_regs regs;unsigned int i;/* @@ -22,10 +23,13 @@ static void filter_cpuid(struct kvm_cpuid2 *kvm_cpuid)switch (entry->function) {case 0 :/* Vendor name */ -memcpy (signature,"LKVMLKVMLKVM" ,12 );x86/cpuid.c static void filter_cpuid(struct kvm_cpuid2 *kvm_cpuid) · line 32
Quoted 10 lines
x86/cpuid.c +7
-3- entry->ebx = signature[0 ]; - entry->ecx = signature[1 ]; - entry->edx = signature[2 ]; + regs = (struct cpuid_regs ) { + .eax =0x00 , + }; +host_cpuid (®s); + entry->ebx = regs.ebx; + entry->ecx = regs.ecx; + entry->edx = regs.edx;But that's redundant, isn't it? We already get the host vendor ID in the three registers in entry, and the current code is just there to overwrite this. So just removing the whole "case 0:" part should do the trick.
Quoted 4 lines
Also please be aware that there was a reason for this fixup, as explained in commit
bc0b99a2a740("kvm tools: Filter out CPU vendor string").Alex, did you boot this on an AMD box, to spot if this is still an issue?
I did a boot on an AMD Ryzen 3900x, didn't find any issues. But I don't think a sample of one CPU is representative, so I'm not sure if the error will not manifest with other models which exist now, or be released in the future.
From what I can tell, kvmtool doesn't use KVM_X86_SET_MSR_FILTER, and the default behaviour for KVM is to try to emulate the accesses to the MSRs in the kernel instead of reflecting them to userspace. So I guess if the user is running kvmtool on a very new AMD or Intel CPU (one of the CPUs mentioned in the commit message for the fix was an engineering sample, for example) for which KVM doesn't have full support, the error can manifest again.
I'm not sure adding code to emulate a specific CPU is the right solution for kvmtool. So I'm thinking either use the host CPU and tolerate the KVM error messages, the frequency of which depends on how fast new MSRs are added to KVM (I have no clue about that), or choose a very simple CPU model that can be emulated by a particular version of KVM.
Thoughts?
Thanks,
Alex
On Tue, 8 Mar 2022 17:31:25 +0000
Andre Przywara <[email protected]> wrote:
Hi,
I did some digging on this issue, see below:
Quoted 61 lines
On Sat, 26 Feb 2022 14:00:48 +0800
Dongli Si <[email protected]> wrote:Hi,
Quoted 41 lines
From: Dongli Si <[email protected]>
glibc detected invalid CPU Vendor name will cause an error:
[ 0.450127] Run /sbin/init as init process /lib64/libc.so.6: CPU ISA level is lower than required [ 0.451931] Kernel panic - not syncing: Attempted to kill init!exitcode=0x00007f00 [ 0.452117] CPU: 0 PID: 1 Comm: init Not
tainted 5.17.0-rc1 #72Signed-off-by Dongli Si <[email protected]>--- x86/cpuid.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)x86/cpuid.c +2
-2@@ -2,6 +2,7 @@#include "kvm/kvm.h" #include "kvm/util.h" +#include "kvm/cpufeature.h" #include <sys/ioctl.h> #include <stdlib.h> @@ -10,7 +11,7 @@static void filter_cpuid (struct kvm_cpuid2 *kvm_cpuid) { -unsigned int signature[3 ]; +struct cpuid_regs regs;unsigned int i;/* @@ -22,10 +23,13 @@ static void filter_cpuid(struct kvm_cpuid2 *kvm_cpuid) switch (entry->function) {case 0 :/* Vendor name */ -memcpy (signature,"LKVMLKVMLKVM" ,12 );x86/cpuid.c static void filter_cpuid(struct kvm_cpuid2 · line 32
Quoted 10 lines
x86/cpuid.c +7
-3- entry->ebx = signature[0 ]; - entry->ecx = signature[1 ]; - entry->edx = signature[2 ]; + regs = (struct cpuid_regs ) { + .eax =0x00 , + }; +host_cpuid (®s); + entry->ebx = regs.ebx; + entry->ecx = regs.ecx; + entry->edx = regs.edx;But that's redundant, isn't it? We already get the host vendor ID in the three registers in entry, and the current code is just there to overwrite this. So just removing the whole "case 0:" part should do the trick.
Quoted 3 lines
Also please be aware that there was a reason for this fixup, as explained in commit
bc0b99a2a740("kvm tools: Filter out CPU vendor string").
So I had a closer look, this is some background:
1) x86 is in the process of stepping up the minimum requirements for the ISA level, so older x86-64 CPUs might not be supported anymore by some distros. As a part of this, glibc allows to set a minimum required CPU, and has a runtime check to verify compatibility:
https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86/cpu-features.c;h=514226b37889;hb=HEAD#l398
This routine first checks the vendor string, and only does very basic capability checks if an unknown vendor (not AMD/Centaur/Intel) is detected. The AMD manual (APM Vol. 3, 24594 Rev 3.3), CPUID instruction, states:
===============
For AMD processors, the string is AuthenticAMD. This string informs software that it should follow the AMD CPUID definition for subsequent CPUID function calls. If the function returns another vendor’s string, software must use that vendor’s CPUID definition when interpreting the results of subsequent CPUID function calls.
===============
So kvmtool using "LKVMLKVMLKVM" as the vendor string will probably end up as detecting only the minimum CPU ISA level, which means glibc's built to a higher standard will fail, as reported.
On top of that glibc problem the kernel also does various CPU checks, and will deny features if an unknown vendor is detected. There are already warnings in the kernel boot log today because of this.
2) The above mentioned kvmtool commit bc0b99a2a740 switched away from keeping the host's vendor ID, because certain errata workarounds triggered when the guest saw the host CPU vendor/family/model/stepping values. This led to random MSR accesses, which KVM could not deal well with at the time. KVM has improved since, and has code to deal with #GP injections due to not-emulated MSR accesses. The particular first issue mentioned in the above commit for instance is addressed by Linux commit d47cc0db8fd6:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d47cc0db8fd6
In general I remember that using random vendor strings was dismissed as a good idea years ago, and other VMMs and hypervisors tend to inject either the host's vendor string or at least a well-known value. So any Linux guest issues due to certain vendor strings would apply to other VMMs or HVs as well, and are probably fixed already.
So the problem with glibc is out there, and is there to stay. The algorithm of checking for the vendor string first is correct by the books, although arguably technically not strictly needed. But we cannot fix this, so have to deal with it.
Quoted 1 line
Alex, did you boot this on an AMD box, to spot if this is still an issue?
I gave it a try on an old AMD box similar to the one in mentioned in the commit message, and it worked fine with the native vendor string.
So I would suggest to just revert kvmtool commit bc0b99a2a740. I will send a patch to that effect, unless someone objects now.
Cheers,
Andre
Hi,
Quoted 117 lines · Andre Przywara
On Tue, 8 Mar 2022 17:31:25 +0000
Andre Przywara <[email protected]> wrote:Hi,
I did some digging on this issue, see below:
Quoted 61 lines
On Sat, 26 Feb 2022 14:00:48 +0800
Dongli Si <[email protected]> wrote:Hi,
Quoted 41 lines
From: Dongli Si <[email protected]>
glibc detected invalid CPU Vendor name will cause an error:
[ 0.450127] Run /sbin/init as init process /lib64/libc.so.6: CPU ISA level is lower than required [ 0.451931] Kernel panic - not syncing: Attempted to kill init!exitcode=0x00007f00 [ 0.452117] CPU: 0 PID: 1 Comm: init Not
tainted 5.17.0-rc1 #72Signed-off-by Dongli Si <[email protected]>--- x86/cpuid.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)x86/cpuid.c +2
-2@@ -2,6 +2,7 @@#include "kvm/kvm.h" #include "kvm/util.h" +#include "kvm/cpufeature.h" #include <sys/ioctl.h> #include <stdlib.h> @@ -10,7 +11,7 @@static void filter_cpuid (struct kvm_cpuid2 *kvm_cpuid) { -unsigned int signature[3 ]; +struct cpuid_regs regs;unsigned int i;/* @@ -22,10 +23,13 @@ static void filter_cpuid(struct kvm_cpuid2 *kvm_cpuid) switch (entry->function) {case 0 :/* Vendor name */ -memcpy (signature,"LKVMLKVMLKVM" ,12 );x86/cpuid.c static void filter_cpuid(struct kvm_cpuid2 · line 32
Quoted 10 lines
x86/cpuid.c +7
-3- entry->ebx = signature[0 ]; - entry->ecx = signature[1 ]; - entry->edx = signature[2 ]; + regs = (struct cpuid_regs ) { + .eax =0x00 , + }; +host_cpuid (®s); + entry->ebx = regs.ebx; + entry->ecx = regs.ecx; + entry->edx = regs.edx;But that's redundant, isn't it? We already get the host vendor ID in the three registers in entry, and the current code is just there to overwrite this. So just removing the whole "case 0:" part should do the trick.
Quoted 3 lines
Also please be aware that there was a reason for this fixup, as explained in commit
bc0b99a2a740("kvm tools: Filter out CPU vendor string").So I had a closer look, this is some background:
1) x86 is in the process of stepping up the minimum requirements for the ISA level, so older x86-64 CPUs might not be supported anymore by some distros. As a part of this, glibc allows to set a minimum required CPU, and has a runtime check to verify compatibility:
https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86/cpu-features.c;h=514226b37889;hb=HEAD#l398
This routine first checks the vendor string, and only does very basic capability checks if an unknown vendor (not AMD/Centaur/Intel) is detected. The AMD manual (APM Vol. 3, 24594 Rev 3.3), CPUID instruction, states:
===============
For AMD processors, the string is AuthenticAMD. This string informs software that it should follow the AMD CPUID definition for subsequent CPUID function calls. If the function returns another vendor’s string, software must use that vendor’s CPUID definition when interpreting the results of subsequent CPUID function calls.
===============So kvmtool using "LKVMLKVMLKVM" as the vendor string will probably end up as detecting only the minimum CPU ISA level, which means glibc's built to a higher standard will fail, as reported.
On top of that glibc problem the kernel also does various CPU checks, and will deny features if an unknown vendor is detected. There are already warnings in the kernel boot log today because of this.2) The above mentioned kvmtool commit
bc0b99a2a740switched away from keeping the host's vendor ID, because certain errata workarounds triggered when the guest saw the host CPU vendor/family/model/stepping values. This led to random MSR accesses, which KVM could not deal well with at the time. KVM has improved since, and has code to deal with #GP injections due to not-emulated MSR accesses. The particular first issue mentioned in the above commit for instance is addressed by Linux commitd47cc0db8fd6:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d47cc0db8fd6In general I remember that using random vendor strings was dismissed as a good idea years ago, and other VMMs and hypervisors tend to inject either the host's vendor string or at least a well-known value. So any Linux guest issues due to certain vendor strings would apply to other VMMs or HVs as well, and are probably fixed already.
So the problem with glibc is out there, and is there to stay. The algorithm of checking for the vendor string first is correct by the books, although arguably technically not strictly needed. But we cannot fix this, so have to deal with it.
Quoted 7 lines
Quoted 1 line
Alex, did you boot this on an AMD box, to spot if this is still an issue?
I gave it a try on an old AMD box similar to the one in mentioned in the commit message, and it worked fine with the native vendor string.
So I would suggest to just revert kvmtool commit
bc0b99a2a740. I will send a patch to that effect, unless someone objects now.
Thank you for the detailed explanation!
I too think that reverting the commit that added the custom vendor string is the best way to fix the glibc errors in a guest.
Thanks,
Alex