Skip to contentSkip to conversation

regression: massive trouble with fpu rework

DISCUSSION Mike Galbraith 6 participants 28 messages Jun 27, 2015 – Jul 9, 2015 x86

lore
Share & tools
lore

Mike Galbraith

author Reviewed-by Borislav Petkov
#0 lore raw

Hi Ingo,

My i7-4790 box is having one hell of a time with this merge window, is dead in the water. The netconsole log below is v4.1-7254-gc13c81006314, but trouble begins at bisected point much earlier. If I turn off kvm, such that I can kinda sorta boot, systemd says many services "enter failed state", box is pretty much a doorstop. Though I can get to a prompt, I can't login. If kvm is enabled, it explodes as soon as it autoloads (wtf does it do that when it's not being used?)

Bisecting to the beginning of my woes takes me to the below. Before that, it doesn't matter if kvm is enabled or not, all is well. Below the current gripage with kvm disabled, find the kvm explosion, and another explosion as I approach the beginning of my box's woes.

067051ccd209623cb56152cf4cb06616ee2bcc5c is the first bad commit commit 067051ccd209623cb56152cf4cb06616ee2bcc5c

Author: Ingo Molnar <[email protected]>
Date:   Sat Apr 25 08:27:44 2015 +0200

x86/fpu: Do system-wide setup from fpu__detect()

fpu__cpu_init() is called on every CPU, so it is the wrong place to call fpu__init_system() from. Call it from fpu__detect():
this is early CPU init code, but we already have CPU features detected, so we can call the system-wide FPU init code from here.

Cc Andy Lutomirski <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Fenghua Yu <[email protected]>
Cc H. Peter Anvin <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Oleg Nesterov <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>

:040000 040000 b85f9a16eea19a4ec974c00ea53144a3d11d4e99 697905b73e3f8931a28588c728da3ab1a7289afa M arch

git bisect start
# good: [cfe3eceb7a2eb91284d5605c5315249bb165e9d3] Merge branch 'x86-efi-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good cfe3eceb7a2eb91284d5605c5315249bb165e9d3
# bad: [650ec5a6bd5df4ab0c9ef38d05b94cd82fb99ad8] Merge branch 'x86-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad 650ec5a6bd5df4ab0c9ef38d05b94cd82fb99ad8
# bad: [c4d6ee6e2e52ec604cc1d76877791f8e8f5c79b5] x86/fpu: Remove failure paths from fpstate-alloc low level functions git bisect bad c4d6ee6e2e52ec604cc1d76877791f8e8f5c79b5
# good: [db2b1d3ad1cdae9f268d6db54b6127b09933da3d] x86/fpu: Use 'struct fpu' in fpstate_alloc_init()
git bisect good db2b1d3ad1cdae9f268d6db54b6127b09933da3d
# good: [e9dbfd673a9cc9d3ef90e3ea4e136a833dab3674] x86/fpu: Move legacy check to fpu__init_system_xstate()
git bisect good e9dbfd673a9cc9d3ef90e3ea4e136a833dab3674
# bad: [7218e8b723dc9ceb71cf2fbd3d019e9e11b7d3cf] x86/fpu: Factor out fpu__init_system_generic()
git bisect bad 7218e8b723dc9ceb71cf2fbd3d019e9e11b7d3cf
# good: [064e51e3c8aee6cfb03ab75e9f9551db3924eb07] x86/fpu: Clean up eager_fpu_init() and rename it to fpu__ctx_switch_init() git bisect good 064e51e3c8aee6cfb03ab75e9f9551db3924eb07
# good: [3960fccf2e22c3b3d61bd5a45b6172e66e660d8b] x86/fpu: Call fpu__init_cpu_ctx_switch() from fpu__init_cpu()
git bisect good 3960fccf2e22c3b3d61bd5a45b6172e66e660d8b
# bad: [7202ab46f7392265c1ecbf03f600393bf32a8bdf] x86/fpu: Remove fpu__init_cpu_ctx_switch() call from fpu__init_system() git bisect bad 7202ab46f7392265c1ecbf03f600393bf32a8bdf
# bad: [067051ccd209623cb56152cf4cb06616ee2bcc5c] x86/fpu: Do system-wide setup from fpu__detect()
git bisect bad 067051ccd209623cb56152cf4cb06616ee2bcc5c

# first bad commit: [067051ccd209623cb56152cf4cb06616ee2bcc5c] x86/fpu: Do system-wide setup from fpu__detect()
[    2.411280] ------------[ cut here ]------------
[    2.412198] WARNING: CPU: 3 PID: 1 at ./arch/x86/include/asm/fpu/internal.h:366 copy_kernel_to_xregs.part.8+0x1e/0x27()
[    2.413132] Modules linked in:
[    2.414003] CPU: 3 PID: 1 Comm: init Tainted: G        W       4.1.0-master #279
[    2.414872] Hardware name: MEDION MS-7848/MS-7848, BIOS M7848W08.20C 09/23/2013
[    2.415745]  ffffffff817f8db8 ffff88040c91fea8 ffffffff8159726c 0000000000000001
[    2.416633]  0000000000000000 ffff88040c91fee8 ffffffff81052bca 00000000ffffffff
[    2.417524]  ffff88040ca18600 ffff88040ca18600 0000000000000000 00007f77d5254148
[    2.418411] Call Trace:
[    2.419286]  [<ffffffff8159726c>] dump_stack+0x45/0x57
[    2.420159]  [<ffffffff81052bca>] warn_slowpath_common+0x8a/0xc0
[    2.421017]  [<ffffffff81052cba>] warn_slowpath_null+0x1a/0x20
[    2.421860]  [<ffffffff81591bf5>] copy_kernel_to_xregs.part.8+0x1e/0x27
[    2.422702]  [<ffffffff8100dfe9>] fpu__restore+0xf9/0x100
[    2.423536]  [<ffffffff810045c3>] do_device_not_available+0x33/0x40
[    2.424356]  [<ffffffff8159f178>] device_not_available+0x18/0x20
[    2.425159] ---[ end trace cb88537fdc8fa202 ]---
[    2.426492] ------------[ cut here ]------------
[    2.427289] WARNING: CPU: 3 PID: 1 at ./arch/x86/include/asm/fpu/internal.h:337 __switch_to+0x586/0x5d0()
[    2.428100] Modules linked in:
[    2.428907] CPU: 3 PID: 1 Comm: systemd Tainted: G        W       4.1.0-master #279
[    2.429943] Hardware name: MEDION MS-7848/MS-7848, BIOS M7848W08.20C 09/23/2013
[    2.431370]  ffffffff817f8db8 ffff880409a0fd10 ffffffff8159726c 0000000000000001
[    2.432801]  0000000000000000 ffff880409a0fd50 ffffffff81052bca ffff88040ca185c0
[    2.434233]  ffff880409a01c80 ffff88040ca18000 ffff88041ecd2080 0000000000000003
[    2.435662] Call Trace:
[    2.437067]  <UNK> 
[    2.437077] ---[ end trace cb88537fdc8fa203 ]---
[    2.440704] random: systemd urandom read with 49 bits of entropy available
[    2.441642] systemd[1]: systemd 210 running in system mode. (+PAM -LIBWRAP +AUDIT +SELINUX -IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ +SECCOMP +APPARMOR)
[    2.442612] systemd[1]: Detected architecture 'x86-64'.
[    2.443411] systemd[1]: Running in initial RAM disk.
[    2.447158] systemd[1]: Inserted module 'autofs4'
[    2.448014] systemd[1]: Set hostname to <homer>.
[    2.449122] ------------[ cut here ]------------
[    2.449964] WARNING: CPU: 2 PID: 1 at ./arch/x86/include/asm/fpu/internal.h:337 fpu__copy+0xd3/0x120()
[    2.450785] Modules linked in: autofs4
[    2.451591] CPU: 2 PID: 1 Comm: systemd Tainted: G        W       4.1.0-master #279
[    2.452404] Hardware name: MEDION MS-7848/MS-7848, BIOS M7848W08.20C 09/23/2013
[    2.453215]  ffffffff817f8db8 ffff88040c91fd58 ffffffff8159726c 0000000000000001
[    2.454025]  0000000000000000 ffff88040c91fd98 ffffffff81052bca 00000000ffffffff
[    2.454810]  ffff8800dca5a280 ffff88040ca18000 0000000000000000 ffff8800dca59c80
[    2.455590] Call Trace:
[    2.456363]  [<ffffffff8159726c>] dump_stack+0x45/0x57
[    2.457121]  [<ffffffff81052bca>] warn_slowpath_common+0x8a/0xc0
[    2.457867]  [<ffffffff81052cba>] warn_slowpath_null+0x1a/0x20
[    2.458595]  [<ffffffff8100e193>] fpu__copy+0xd3/0x120
[    2.459312]  [<ffffffff8100d48e>] arch_dup_task_struct+0x2e/0x40
[    2.460015]  [<ffffffff810509a0>] copy_process.part.42+0x120/0x1710
[    2.460721]  [<ffffffff8116df05>] ? __sb_end_write+0x35/0x70
[    2.461418]  [<ffffffff8117d69a>] ? iterate_dir+0x5a/0x110
[    2.462114]  [<ffffffff81052145>] do_fork+0xd5/0x330
[    2.462798]  [<ffffffff8117db4d>] ? SyS_getdents+0x9d/0xe0
[    2.463475]  [<ffffffff8117d820>] ? fillonedir+0xd0/0xd0
[    2.464144]  [<ffffffff81052426>] SyS_clone+0x16/0x20
[    2.464815]  [<ffffffff8159d9db>] entry_SYSCALL_64_fastpath+0x16/0x6e
[    2.465479] ---[ end trace cb88537fdc8fa204 ]---
[    2.469752] ------------[ cut here ]------------
[    2.470483] WARNING: CPU: 1 PID: 98 at ./arch/x86/include/asm/fpu/internal.h:366 copy_kernel_to_xregs.part.0+0x1e/0x27()
[    2.471173] Modules linked in: autofs4
[    2.471834] CPU: 1 PID: 98 Comm: dracut-rootfs-g Tainted: G        W       4.1.0-master #279
[    2.472502] Hardware name: MEDION MS-7848/MS-7848, BIOS M7848W08.20C 09/23/2013
[    2.473169]  ffffffff817f8db8 ffff880409be3da0 ffffffff8159726c 0000000000000001
[    2.473844]  0000000000000000 ffff880409be3de0 ffffffff81052bca 0000000000000003
[    2.474515]  00007ffdf76bc580 ffff880409b75b80 ffff880409b75b80 ffff880409b75580
[    2.475182] Call Trace:
[    2.475841]  [<ffffffff8159726c>] dump_stack+0x45/0x57
[    2.476505]  [<ffffffff81052bca>] warn_slowpath_common+0x8a/0xc0
[    2.477177]  [<ffffffff81052cba>] warn_slowpath_null+0x1a/0x20
[    2.477838]  [<ffffffff81591c43>] copy_kernel_to_xregs.part.0+0x1e/0x27
[    2.478500]  [<ffffffff8100ef11>] __fpu__restore_sig+0x251/0x3f0
[    2.479162]  [<ffffffff810551ca>] ? do_wait+0x11a/0x220
[    2.479822]  [<ffffffff8105cbef>] ? recalc_sigpending+0x1f/0x60
[    2.480485]  [<ffffffff8100f2cd>] fpu__restore_sig+0x4d/0x60
[    2.481148]  [<ffffffff81003282>] restore_sigcontext+0x132/0x150
[    2.481796]  [<ffffffff81003f6d>] sys_rt_sigreturn+0x9d/0xc0
[    2.482437]  [<ffffffff8159dd33>] stub_rt_sigreturn+0x23/0x23
[    2.483071]  [<ffffffff8159d9db>] ? entry_SYSCALL_64_fastpath+0x16/0x6e
[    2.483702] ---[ end trace cb88537fdc8fa205 ]---
[    2.501200] ------------[ cut here ]------------
[    2.501925] WARNING: CPU: 2 PID: 0 at ./arch/x86/include/asm/fpu/internal.h:366 __switch_to+0x3dc/0x5d0()
[    2.502603] Modules linked in: autofs4
[    2.503242] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W       4.1.0-master #279
[    2.503905] Hardware name: MEDION MS-7848/MS-7848, BIOS M7848W08.20C 09/23/2013
[    2.504576]  ffffffff817f8db8 ffff88040c91fd00 ffffffff8159726c 0000000000000001
[    2.505253]  0000000000000000 ffff88040c91fd40 ffffffff81052bca 00000000c0000100
[    2.505901]  ffff88040ca18000 ffff88040ca5b900 ffff88041ec92080 ffff88040ca18600
[    2.506530] Call Trace:
[    2.507149]  <UNK> 
[    2.507155] ---[ end trace cb88537fdc8fa206 ]---
[    2.510183] systemd[1]: Expecting device dev-disk-by\x2duuid-63d8c9f4\x2dc094\x2d4dd8\x2d96cd\x2d23c5e91d0322.device...
[    2.511523] systemd[1]: Starting Timers.
[    2.512962] systemd[1]: Reached target Timers.
[    2.513608] systemd[1]: Starting -.slice.
[    2.514900] systemd[1]: Created slice -.slice.
[    2.515520] systemd[1]: Starting udev Control Socket.
[    2.516849] systemd[1]: Listening on udev Control Socket.
[    2.517473] systemd[1]: Starting udev Kernel Socket.
[    2.518737] systemd[1]: Listening on udev Kernel Socket.
[    2.519351] systemd[1]: Starting Journal Socket.
[    2.520683] systemd[1]: Listening on Journal Socket.
[    2.521321] systemd[1]: Starting Sockets.
[    2.522595] systemd[1]: Reached target Sockets.
[    2.523218] systemd[1]: Starting System Slice.
[    2.524566] systemd[1]: Created slice System Slice.
[    2.525247] systemd[1]: Starting dracut cmdline hook...
[    2.526651] systemd[1]: Starting system-systemd\x2dfsck.slice.
[    2.528034] systemd[1]: Created slice system-systemd\x2dfsck.slice.
[    2.528792] systemd[1]: Starting Load Kernel Modules...
[    2.530325] systemd[1]: Starting Create list of required static device nodes for the current kernel...
[    2.531235] sd 0:0:0:0: Attached scsi generic sg0 type 0
[    2.531278] sd 1:0:0:0: Attached scsi generic sg1 type 0
[    2.531297] sd 4:0:0:0: Attached scsi generic sg2 type 0
[    2.531316] sd 5:0:0:0: Attached scsi generic sg3 type 0
[    2.534490] systemd[1]: Starting Journal Service...
[    2.536022] systemd[1]: Starting Slices.
[    2.537140] systemd-journald[154]: Failed to set file attributes: Inappropriate ioctl for device
[    2.538285] systemd[1]: Reached target Slices.
[    2.538946] systemd[1]: Starting Setup Virtual Console...
[    2.540465] systemd[1]: Starting Swap.
[    2.542612] systemd[1]: Reached target Swap.
[    2.543709] systemd[1]: Starting Local File Systems.
[    2.545112] systemd[1]: Reached target Local File Systems.
[    2.546961] systemd[1]: Started Load Kernel Modules.
[    2.548995] systemd[1]: Started Create list of required static device nodes for the current kernel.
[    2.550572] systemd[1]: Started Setup Virtual Console.
[    2.552908] systemd[1]: Starting Create static device nodes in /dev...
[    2.554608] systemd[1]: Starting Apply Kernel Variables...
[    2.556948] systemd[1]: Started Create static device nodes in /dev.
[    2.558383] systemd[1]: Started Apply Kernel Variables.
[    2.559749] systemd[1]: Started dracut cmdline hook.
[    2.561210] systemd[1]: Started Journal Service.
[    2.585519] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input7
[    2.586779] ACPI: Power Button [PWRB]
[    2.587983] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input8
[    2.588018] ACPI: Power Button [PWRF]
[    2.592267] [drm] Initialized drm 1.1.0 20060810
[    2.594196] wmi: Mapper loaded
[    2.640007] clocksource: Switched to clocksource tsc
[    2.868141] random: nonblocking pool is initialized
[    2.937645] EXT4-fs (sdc3): mounted filesystem with ordered data mode. Opts: data=ordered
[    3.082729] systemd-journald[154]: Received SIGTERM from PID 1 (systemd).
[    4.944246] device-mapper: uevent: version 1.0.3
[    4.944309] device-mapper: ioctl: 4.31.0-ioctl (2015-3-12) initialised: [email protected]
[    5.540586] systemd-readahead[399]: Bumped block_nr parameter of 8:32 to 20480. This is a temporary hack and should be removed one day.
[    6.851888] ACPI Warning: SystemIO range 0x000000000000F000-0x000000000000F01F conflicts with OpRegion 0x000000000000F000-0x000000000000F00F (\_SB_.PCI0.SBUS.SMBI) (20150515/utaddress-254)
[    6.852656] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[    6.931752] thermal LNXTHERM:00: registered as thermal_zone0
[    6.932515] ACPI: Thermal Zone [TZ00] (28 C)
[    6.933477] thermal LNXTHERM:01: registered as thermal_zone1
[    6.934206] ACPI: Thermal Zone [TZ01] (30 C)
[    7.151702] iTCO_vendor_support: vendor-support=0
[    7.152822] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
[    7.153543] iTCO_wdt: Found a Lynx Point TCO device (Version=2, TCOBASE=0x1860)
[    7.154304] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
[    7.561298] EXT4-fs (sdc3): re-mounted. Opts: acl,user_xattr
[    7.828308] EXT4-fs (sdd1): mounted filesystem with writeback data mode. Opts: data=writeback,acl,user_xattr
[    8.105380] AVX2 version of gcm_enc/dec engaged.
[    8.106197] AES CTR mode by8 optimization enabled
[    8.371083] Adding 2104316k swap on /dev/sdc2.  Priority:-1 extents:1 across:2104316k FS
[    8.372290] snd_hda_intel 0000:00:1b.0: enabling device (0000 -> 0002)
[    8.372369] snd_hda_intel 0000:01:00.1: enabling device (0000 -> 0002)
[    8.372392] snd_hda_intel 0000:01:00.1: Disabling MSI
[    8.372395] snd_hda_intel 0000:01:00.1: Handle VGA-switcheroo audio client
[    8.504246] snd_hda_codec_realtek hdaudioC0D0: autoconfig for ALC887-VD: line_outs=1 (0x14/0x0/0x0/0x0/0x0) type:line
[    8.504247] snd_hda_codec_realtek hdaudioC0D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[    8.504248] snd_hda_codec_realtek hdaudioC0D0:    hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)
[    8.504248] snd_hda_codec_realtek hdaudioC0D0:    mono: mono_out=0x0
[    8.504249] snd_hda_codec_realtek hdaudioC0D0:    inputs:
[    8.504250] snd_hda_codec_realtek hdaudioC0D0:      Front Mic=0x19
[    8.504250] snd_hda_codec_realtek hdaudioC0D0:      Rear Mic=0x18
[    8.504251] snd_hda_codec_realtek hdaudioC0D0:      Line=0x1a
[    8.515534] input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input9
[    8.515563] input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input10
[    8.515587] input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1b.0/sound/card0/input11
[    8.515612] input: HDA Intel PCH Line Out as /devices/pci0000:00/0000:00:1b.0/sound/card0/input12
[    8.515636] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card0/input13
[    9.053824] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input14
[    9.055403] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input15
[    9.056681] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input16
[    9.057436] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input17
[    9.115047] FAT-fs (sdc1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[   10.043873] systemd-journald[403]: Received request to flush runtime journal from PID 1
[   10.340931] systemd[1]: Unit apparmor.service entered failed state.
[   11.072589] systemd[1]: set_kthread_prio.service: control process exited, code=killed status=11
[   11.096816] systemd[1]: Unit set_kthread_prio.service entered failed state.
[   11.133099] systemd[1]: Unit kdump.service entered failed state.
[   11.180579] systemd[1]: Unit SuSEfirewall2_init.service entered failed state.
[   11.344489] systemd[1]: display-manager.service: control process exited, code=killed status=11
[   11.368512] systemd[1]: Unit display-manager.service entered failed state.
[   12.062671] NET: Registered protocol family 17
[   12.517609] systemd[1]: Unit wickedd-dhcp4.service entered failed state.
[   12.760012] systemd[1]: wickedd-dhcp4.service holdoff time over, scheduling restart.
[   13.008371] systemd[1]: Unit wickedd.service entered failed state.
[   13.259794] systemd[1]: wickedd.service holdoff time over, scheduling restart.
[   14.324202] systemd[1]: Unit wickedd-nanny.service entered failed state.
[   14.396138] systemd[1]: Unit wickedd-auto4.service entered failed state.
[   14.504142] systemd[1]: Unit wickedd-dhcp6.service entered failed state.
[   25.551726] sysrq: SysRq : Power Off
[   25.567346] sd 5:0:0:0: [sdd] Synchronizing SCSI cache
[   25.568069] sd 5:0:0:0: [sdd] Stopping disk
[   25.979200] sd 4:0:0:0: [sdc] Synchronizing SCSI cache
[   26.044734] sd 4:0:0:0: [sdc] Stopping disk
[   26.523508] sd 1:0:0:0: [sdb] Synchronizing SCSI cache
[   26.524253] sd 1:0:0:0: [sdb] Stopping disk
[   26.928454] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[   26.933453] sd 0:0:0:0: [sda] Stopping disk
[    9.422011] invalid opcode: 0000 [#1] SMP 
[    9.422806] Modules linked in: kvm_intel(+) kvm snd_hda_codec_hdmi(+) crct10dif_pclmul snd_hda_codec_realtek snd_hda_codec_generic crc32_pclmul crc32c_intel ghash_clmulni_intel snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_hda_core snd_pcm aesni_intel aes_x86_64 lrw gf128mul snd_timer iTCO_wdt tpm_infineon snd glue_helper mei_me iTCO_vendor_support mei i2c_i801 tpm_tis lpc_ich joydev ablk_helper thermal battery processor mfd_core fan tpm soundcore cryptd intel_smartconnect serio_raw dm_mod efivarfs nouveau mxm_wmi wmi i2c_algo_bit drm_kms_helper ttm drm video thermal_sys button sg autofs4
[    9.425350] CPU: 1 PID: 480 Comm: systemd-udevd Tainted: G        W       4.1.0-master #268
[    9.426174] Hardware name: MEDION MS-7848/MS-7848, BIOS M7848W08.20C 09/23/2013
[    9.427000] task: ffff88040a658000 ti: ffff88040a660000 task.ti: ffff88040a660000
[    9.427828] RIP: 0010:[<ffffffffa041c811>]  [<ffffffffa041c811>] kvm_arch_init+0xf1/0x1c0 [kvm]
[    9.428679] RSP: 0018:ffff88040a663c68  EFLAGS: 00010246
[    9.429517] RAX: 0000000077fafbff RBX: 0000000000000000 RCX: 0000000000000000
[    9.430358] RDX: 0000000080000001 RSI: ffffffffa0455c60 RDI: ffffffffa0455c40
[    9.431198] RBP: ffff88040a663c88 R08: 0000000000000080 R09: 0000000000000000
[    9.432037] R10: 00000000000006c8 R11: 000000000001cf69 R12: ffffffffa0920160
[    9.432879] R13: 0000000000000000 R14: 0000000000004480 R15: ffffffffa09207e0
[    9.433715] FS:  00007f24f12c7880(0000) GS:ffff88041ec40000(0000) knlGS:0000000000000000
[    9.434557] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    9.435397] CR2: 00007f24f12d2000 CR3: 000000040a64f000 CR4: 00000000001007e0
[    9.436242] Stack:
[    9.437059]  ffffffffa09207e0 ffffffff81a12040 ffff8800dbc06e00 0000000000000040
[    9.437876]  ffff88040a663cd8 ffffffffa0407004 ffff88040a663d18 ffffffff00000018
[    9.438684]  ffff88040a663cf8 ffffffff81a12040 ffff8800dbc06e00 0000000000000000
[    9.439489] Call Trace:
[    9.440281]  [<ffffffffa0407004>] kvm_init+0x24/0x2b0 [kvm]
[    9.441074]  [<ffffffffa0341c5a>] ? vmx_check_processor_compat+0x60/0x60 [kvm_intel]
[    9.441874]  [<ffffffffa0341c7b>] vmx_init+0x21/0x3a6 [kvm_intel]
[    9.442661]  [<ffffffff810002fc>] do_one_initcall+0xbc/0x1f0
[    9.443436]  [<ffffffff8116ecf5>] ? kmem_cache_alloc_trace+0x35/0x130
[    9.444210]  [<ffffffff815c2a74>] do_init_module+0x61/0x1cc
[    9.444979]  [<ffffffff810c6693>] load_module+0x1cc3/0x2590
[    9.445744]  [<ffffffff810c2590>] ? store_uevent+0x40/0x40
[    9.446503]  [<ffffffff810c7136>] SyS_finit_module+0x86/0xb0
[    9.447262]  [<ffffffff815cd29b>] system_call_fastpath+0x16/0x6e
[    9.448018] Code: 25 25 2e 04 00 e8 00 38 00 00 e8 cb 4f ff ff 48 c7 c7 40 5c 45 a0 e8 af 81 cf e0 48 8b 05 dc bd 6b e1 a9 00 00 00 04 74 15 31 c9 <0f> 01 d0 48 c1 e2 20 89 c0 48 01 d0 48 89 05 44 00 04 00 e8 27 
[    9.449791] RIP  [<ffffffffa041c811>] kvm_arch_init+0xf1/0x1c0 [kvm]
[    9.450595]  RSP <ffff88040a663c68>
[    9.451403] ---[ end trace cb88537fdc8fa207 ]---
[    9.452740] Kernel panic - not syncing: Fatal exception
[    9.453576] Kernel Offset: disabled
[    9.454381] Rebooting in 60 seconds..
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000] ------------[ cut here ]------------
[    0.000000] WARNING: CPU: 0 PID: 0 at arch/x86/kernel/fpu/xsave.c:646 fpu__init_system_xstate+0x1da/0x440()
[    0.000000] x86/fpu: XSTATE_CPUID missing!
[    0.000000] Modules linked in:
[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.1.0-master #270
[    0.000000]  ffffffff817f0b3b ffffffff81a03dc8 ffffffff815c5088 000000000000002e
[    0.000000]  ffffffff81a03e18 ffffffff81a03e08 ffffffff8105432a 0000000000000000
[    0.000000]  0000000000000008 ffffffff81603f80 0000000001000000 00000000cbf3b260
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815c5088>] dump_stack+0x45/0x57
[    0.000000]  [<ffffffff8105432a>] warn_slowpath_common+0x8a/0xc0
[    0.000000]  [<ffffffff810543a6>] warn_slowpath_fmt+0x46/0x50
[    0.000000]  [<ffffffff8100f7aa>] fpu__init_system_xstate+0x1da/0x440
[    0.000000]  [<ffffffff8100d680>] fpu__init_system+0x90/0x1c0
[    0.000000]  [<ffffffff81afffed>] early_cpu_init+0xe6/0x144
[    0.000000]  [<ffffffff81afc293>] setup_arch+0x95/0xc4b
[    0.000000]  [<ffffffff8109b8af>] ? vprintk_default+0x1f/0x30
[    0.000000]  [<ffffffff81af9c4b>] start_kernel+0x90/0x41f
[    0.000000]  [<ffffffff81b49b2f>] ? memblock_reserve+0x4c/0x51
[    0.000000]  [<ffffffff81af95ad>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81af969a>] x86_64_start_kernel+0xeb/0xef
[    0.000000] ---[ end trace cb88537fdc8fa200 ]---
[    0.000000] x86/fpu: Using 'lazy' FPU context switches.

....

[    2.454087] console [netcon0] enabled
[    2.454902] netconsole: network logging started
[    2.455758] rtc_cmos 00:02: setting system clock to 2015-06-27 04:58:45 UTC (1435381125)
[    2.456816] Freeing unused kernel memory: 1164K (ffffffff81ae2000 - ffffffff81c05000)
[    2.457625] Write protecting the kernel read-only data: 10240k
[    2.458682] Freeing unused kernel memory: 180K (ffff8800015d3000 - ffff880001600000)
[    2.459728] Freeing unused kernel memory: 1208K (ffff8800018d2000 - ffff880001a00000)
[    2.460638] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    2.460638] 
[    2.462272] CPU: 6 PID: 1 Comm: init Tainted: G        W       4.1.0-master #277
[    2.463086] Hardware name: MEDION MS-7848/MS-7848, BIOS M7848W08.20C 09/23/2013
[    2.463906]  ffff880409a78000 ffff88040ca1bc88 ffffffff815c50d2 00000000000000ea
[    2.464735]  ffffffff817fe7e8 ffff88040ca1bd08 ffffffff815c0a7e ffff88040ca1bce8
[    2.465553]  ffff880400000010 ffff88040ca1bd18 ffff88040ca1bcb8 ffff88040ca10000
[    2.466361] Call Trace:
[    2.467152]  [<ffffffff815c50d2>] dump_stack+0x45/0x57
[    2.467935]  [<ffffffff815c0a7e>] panic+0xbb/0x1df
[    2.468701]  [<ffffffff81055f86>] do_exit+0xac6/0xad0
[    2.469462]  [<ffffffff81056c3f>] do_group_exit+0x3f/0xa0
[    2.470215]  [<ffffffff810615f7>] get_signal+0x1d7/0x5d0
[    2.470953]  [<ffffffff8105f435>] ? complete_signal+0x105/0x210
[    2.471679]  [<ffffffff81002da0>] do_signal+0x20/0x120
[    2.472396]  [<ffffffff8105f998>] ? __send_signal.constprop.25+0x138/0x3e0
[    2.473115]  [<ffffffff81060434>] ? force_sig_info+0xd4/0xf0
[    2.473830]  [<ffffffff81002f08>] do_notify_resume+0x68/0x80
[    2.474544]  [<ffffffff815cc5f3>] retint_signal+0x3d/0x7a
[    2.475285] Kernel Offset: disabled
[    2.476011] Rebooting in 60 seconds..
#1 lore raw
Quoted 4 lines · Mike Galbraith

Hi Ingo,

My i7-4790 box is having one hell of a time with this merge window, is dead in the water.

BIOS setting "Limit CPUID Maximum" upsets new fpu code mightily.

-Mike

#2 lore raw
Quoted 7 lines · Mike Galbraith
Quoted 4 lines · Mike Galbraith

Hi Ingo,

My i7-4790 box is having one hell of a time with this merge window, is dead in the water.

BIOS setting "Limit CPUID Maximum" upsets new fpu code mightily.

Ok, that's interesting. Mind explaining it a bit more verbosely - which setting is causing what?

This would suggest sensitivity on CPUID details, i.e. that doing fpu__init_system() before other CPU init sequences is causing the bug.

Does the patch below perhaps make a difference? (I'd suggest to apply it _without_ the other patch I sent.)

Thanks,

Ingo

 arch/x86/kernel/cpu/common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/common.c +1 -1
index 9fc5e3d9d9c8..922c5e0cea4c 100644
@@ -742,7 +742,6 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
 	cpu_detect(c);
 	get_cpu_vendor(c);
 	get_cpu_cap(c);
-	fpu__init_system(c);
 
 	if (this_cpu->c_early_init)
 		this_cpu->c_early_init(c);
@@ -754,6 +753,7 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
 		this_cpu->c_bsp_init(c);
 
 	setup_force_cpu_cap(X86_FEATURE_ALWAYS);
+	fpu__init_system(c);
 }
 
 void __init early_cpu_init(void)
#3 lore raw
Quoted 5 lines · Ingo Molnar
Quoted 3 lines · Mike Galbraith
Quoted 2 lines · Mike Galbraith

Hi Ingo,

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

My i7-4790 box is having one hell of a time with this merge window, is dead in the water.

BIOS setting "Limit CPUID Maximum" upsets new fpu code mightily.

Ok, that's interesting. Mind explaining it a bit more verbosely - which setting is causing what?

That BIOS setting is annotated with the helpful text "Disabled for Windows XP". It makes box say interesting things during boot, like...

x86/fpu: XSTATE_CPUID missing!

..or with HEAD, it triggers warning..

        if (boot_cpu_data.cpuid_level < XSTATE_CPUID) {
                WARN_ON_FPU(1);
                return;
        }

..and all kinds of bad juju follows. I have no idea what the thing does beyond what I can interpolate from the word 'limit'.

Quoted 5 lines

This would suggest sensitivity on CPUID details, i.e. that doing fpu__init_system() before other CPU init sequences is causing the bug.

Does the patch below perhaps make a difference? (I'd suggest to apply it _without_ the other patch I sent.)

Yup, that made it not care about the BIOS setting.. again.

Quoted 27 lines

Thanks,

Ingo

 arch/x86/kernel/cpu/common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/common.c +1 -1
index 9fc5e3d9d9c8..922c5e0cea4c 100644
@@ -742,7 +742,6 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
 	cpu_detect(c);
 	get_cpu_vendor(c);
 	get_cpu_cap(c);
-	fpu__init_system(c);
 
 	if (this_cpu->c_early_init)
 		this_cpu->c_early_init(c);
@@ -754,6 +753,7 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
 		this_cpu->c_bsp_init(c);
 
 	setup_force_cpu_cap(X86_FEATURE_ALWAYS);
+	fpu__init_system(c);
 }
 
 void __init early_cpu_init(void)
maintainer
#4 lore raw
Quoted 1 line · Mike Galbraith

Yup, that made it not care about the BIOS setting.. again.

Does it say

	"x86/fpu: Legacy x87 FPU detected."

with Ingo's patch?

Or do you see that "x86/fpu: Enabled xstate features... " print out from the end of fpu__init_system_xstate()?

Signature · 5 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--
#5 lore raw
Quoted 8 lines · Borislav Petkov
Quoted 1 line · Mike Galbraith

Yup, that made it not care about the BIOS setting.. again.

Does it say

	"x86/fpu: Legacy x87 FPU detected."

with Ingo's patch?

Nope.

Quoted 2 lines

Or do you see that "x86/fpu: Enabled xstate features... " print out from the end of fpu__init_system_xstate()?

[    0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100
[    0.000000] x86/fpu: Supporting XSAVE feature 0x01: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x02: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x04: 'AVX registers'
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 0x340 bytes, using 'standard' format.
[    0.000000] x86/fpu: Using 'eager' FPU context switches.
#6 lore raw
In reply to Mike Galbraith
Quoted 10 lines · Mike Galbraith
Quoted 1 line
Quoted 1 line

BIOS setting "Limit CPUID Maximum" upsets new fpu code mightily.

That BIOS setting is annotated with the helpful text "Disabled for Windows XP". It makes box say interesting things during boot, like...

x86/fpu: XSTATE_CPUID missing!

..or with HEAD, it triggers warning..

Quoted 7 lines
        if (boot_cpu_data.cpuid_level < XSTATE_CPUID) {
                WARN_ON_FPU(1);
                return;
        }

..and all kinds of bad juju follows. I have no idea what the thing does beyond what I can interpolate from the word 'limit'.

Well, it is supposed to disable CPUID levels >= 0x04. This thing should *NEVER* be enabled, the last operating system that required it to be enabled was Windows 98.

Can/do we override that crap during cpu init? If we cannot/don't, maybe instead of limping along with CPUID crippled, it would be better to either output a very nasty warning, or outright stop booting [with an appropriate error message] ?

Signature · 4 lines
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
#7 lore raw
Quoted 19 lines · Henrique de Moraes Holschuh
Quoted 10 lines · Mike Galbraith
Quoted 1 line
Quoted 1 line

BIOS setting "Limit CPUID Maximum" upsets new fpu code mightily.

That BIOS setting is annotated with the helpful text "Disabled for Windows XP". It makes box say interesting things during boot, like...

x86/fpu: XSTATE_CPUID missing!

..or with HEAD, it triggers warning..

Quoted 7 lines
        if (boot_cpu_data.cpuid_level < XSTATE_CPUID) {
                WARN_ON_FPU(1);
                return;
        }

..and all kinds of bad juju follows. I have no idea what the thing does beyond what I can interpolate from the word 'limit'.

Quoted 8 lines

Well, it is supposed to disable CPUID levels >= 0x04. This thing should *NEVER* be enabled, the last operating system that required it to be enabled was Windows 98.

Can/do we override that crap during cpu init? If we cannot/don't, maybe instead of limping along with CPUID crippled, it would be better to either output a very nasty warning, or outright stop booting [with an appropriate error message] ?

Why get all upset? We didn't even notice before, nor did/does that other OS. A casual "BTW, your BIOS sucks.." should suffice, no?

-Mike

#8 lore raw
Quoted 6 lines · Mike Galbraith
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

BIOS setting "Limit CPUID Maximum" upsets new fpu code mightily.

Well, it is supposed to disable CPUID levels >= 0x04. This thing should *NEVER* be enabled, the last operating system that required it to be enabled was Windows 98.

Quoted 7 lines
Quoted 4 lines

Can/do we override that crap during cpu init? If we cannot/don't, maybe instead of limping along with CPUID crippled, it would be better to either output a very nasty warning, or outright stop booting [with an appropriate error message] ?

Why get all upset? We didn't even notice before, nor did/does that other OS. A casual "BTW, your BIOS sucks.." should suffice, no?

Oh, I am not upset, although I suppose my reply did look like it. Sorry about that.

It is just that this kind of breakage should not be subtle if we can help it, because people will use a crippled system for years without noticing...

Signature · 4 lines
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
#9 lore raw
Quoted 12 lines · Henrique de Moraes Holschuh
Quoted 11 lines · Mike Galbraith
Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

BIOS setting "Limit CPUID Maximum" upsets new fpu code mightily.

Well, it is supposed to disable CPUID levels >= 0x04. This thing should *NEVER* be enabled, the last operating system that required it to be enabled was Windows 98.

Can/do we override that crap during cpu init? If we cannot/don't, maybe instead of limping along with CPUID crippled, it would be better to either output a very nasty warning, or outright stop booting [with an appropriate error message] ?

Quoted 5 lines
Quoted 2 lines

Why get all upset? We didn't even notice before, nor did/does that other OS. A casual "BTW, your BIOS sucks.." should suffice, no?

Oh, I am not upset, although I suppose my reply did look like it. Sorry about that.

I didn't mean you personally of course, I meant the kernel ;-)

Quoted 2 lines

It is just that this kind of breakage should not be subtle if we can help it, because people will use a crippled system for years without noticing...

If you can use it without noticing for years, it ain't crippled, or? My point being that severity seems more akin to the box having a zit behind its left ear, in which case lobotomizing it seems a tad extreme.

-Mike

#10 lore raw
Quoted 7 lines · Mike Galbraith
Quoted 2 lines · Henrique de Moraes Holschuh

It is just that this kind of breakage should not be subtle if we can help it, because people will use a crippled system for years without noticing...

If you can use it without noticing for years, it ain't crippled, or? My point being that severity seems more akin to the box having a zit behind its left ear, in which case lobotomizing it seems a tad extreme.

Noted. However if it does boot with cpuid limited (and we don't "unlimit" it somehow) on a recent processor, at *best* the user paid good money for a lot of stuff that is going to not be used to enhance system performance and system security.

It is not nice to the user to just limp along silently about this.

Signature · 4 lines
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
#11 lore raw
In reply to Mike Galbraith
Quoted 18 lines · Mike Galbraith
Quoted 5 lines

This would suggest sensitivity on CPUID details, i.e. that doing fpu__init_system() before other CPU init sequences is causing the bug.

Does the patch below perhaps make a difference? (I'd suggest to apply it _without_ the other patch I sent.)

Yup, that made it not care about the BIOS setting.. again.

Quoted 10 lines
arch/x86/kernel/cpu/common.c +0 -1
index 9fc5e3d9d9c8..922c5e0cea4c 100644
@@ -742,7 +742,6 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
 	cpu_detect(c);
 	get_cpu_vendor(c);
 	get_cpu_cap(c);
-	fpu__init_system(c);
 
Quoted 8 lines

arch/x86/kernel/cpu/common.c static void __init early_identify_cpu(struct cpuinfo_x86 *c) · line 757

Quoted 8 lines
arch/x86/kernel/cpu/common.c +1 -0
 	if (this_cpu->c_early_init)
 		this_cpu->c_early_init(c);
@@ -754,6 +753,7 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
 		this_cpu->c_bsp_init(c);
 
 	setup_force_cpu_cap(X86_FEATURE_ALWAYS);
+	fpu__init_system(c);
 }

Ok, so could you please move the fpu__init_system() further up and see which position is that starts breaking with the BIOS option set?

here's the current, broken layout of the code:

        get_cpu_cap(c);

[0] fpu__init_system(c);

        if (this_cpu->c_early_init)
                this_cpu->c_early_init(c);

[1]

        c->cpu_index = 0;
[2]
        filter_cpuid_features(c, false);

[3]

        if (this_cpu->c_bsp_init)
                this_cpu->c_bsp_init(c);

[4]

        setup_force_cpu_cap(X86_FEATURE_ALWAYS);

[5]
}

and we know it from your testing that moving [0] to [5] fixes the crash.

The question is, can we move it to [4], [3], [2] or even [1] instead, without breaking the system?

I still don't see where the breakage comes from, but this would help us narrow it down.

Thanks,

Ingo

#12 lore raw
Quoted 26 lines · Ingo Molnar

*
Ok, so could you please move the fpu__init_system() further up and see which position is that starts breaking with the BIOS option set?

here's the current, broken layout of the code:

        get_cpu_cap(c);

[0] fpu__init_system(c);

        if (this_cpu->c_early_init)
                this_cpu->c_early_init(c);

[1]

        c->cpu_index = 0;
[2]
        filter_cpuid_features(c, false);

[3]

        if (this_cpu->c_bsp_init)
                this_cpu->c_bsp_init(c);

[4]

        setup_force_cpu_cap(X86_FEATURE_ALWAYS);

[5]
}

Quoted 7 lines

and we know it from your testing that moving [0] to [5] fixes the crash.

The question is, can we move it to [4], [3], [2] or even [1] instead, without breaking the system?

I still don't see where the breakage comes from, but this would help us narrow it down.

[0] is the only spot that breaks box.

-Mike

maintainer
#13 lore raw
Quoted 7 lines · Mike Galbraith
Quoted 6 lines · Ingo Molnar

*
Ok, so could you please move the fpu__init_system() further up and see which position is that starts breaking with the BIOS option set?

here's the current, broken layout of the code:

Quoted 7 lines
Quoted 5 lines
        get_cpu_cap(c);

[0] fpu__init_system(c);

        if (this_cpu->c_early_init)
                this_cpu->c_early_init(c);

[0] is the only spot that breaks box.

I bet it is that

        /* Unmask CPUID levels if masked: */
        if (c->x86 > 6 || (c->x86 == 6 && c->x86_model >= 0xd)) {
                if (msr_clear_bit(MSR_IA32_MISC_ENABLE,
                                  MSR_IA32_MISC_ENABLE_LIMIT_CPUID_BIT) > 0) {
                        c->cpuid_level = cpuid_eax(0);
                        get_cpu_cap(c);
                }
        }

in early_init_intel(). If you feel like playing, you might comment it out to see what happens.

:-)

Signature · 5 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--
#14 lore raw
Quoted 27 lines · Borislav Petkov
Quoted 7 lines · Mike Galbraith
Quoted 6 lines · Ingo Molnar

*
Ok, so could you please move the fpu__init_system() further up and see which position is that starts breaking with the BIOS option set?

here's the current, broken layout of the code:

Quoted 7 lines
Quoted 5 lines
        get_cpu_cap(c);

[0] fpu__init_system(c);

        if (this_cpu->c_early_init)
                this_cpu->c_early_init(c);

[0] is the only spot that breaks box.

I bet it is that

        /* Unmask CPUID levels if masked: */
        if (c->x86 > 6 || (c->x86 == 6 && c->x86_model >= 0xd)) {
                if (msr_clear_bit(MSR_IA32_MISC_ENABLE,
                                  MSR_IA32_MISC_ENABLE_LIMIT_CPUID_BIT) > 0) {
                        c->cpuid_level = cpuid_eax(0);
                        get_cpu_cap(c);
                }
        }
Quoted 2 lines

in early_init_intel(). If you feel like playing, you might comment it out to see what happens.

I'll poke after I get back from physical tort^Wtherapy.

-Mike

#15 lore raw
In reply to Borislav Petkov
Quoted 27 lines · Borislav Petkov
Quoted 7 lines · Mike Galbraith
Quoted 6 lines · Ingo Molnar

*
Ok, so could you please move the fpu__init_system() further up and see which position is that starts breaking with the BIOS option set?

here's the current, broken layout of the code:

Quoted 7 lines
Quoted 5 lines
        get_cpu_cap(c);

[0] fpu__init_system(c);

        if (this_cpu->c_early_init)
                this_cpu->c_early_init(c);

[0] is the only spot that breaks box.

I bet it is that

        /* Unmask CPUID levels if masked: */
        if (c->x86 > 6 || (c->x86 == 6 && c->x86_model >= 0xd)) {
                if (msr_clear_bit(MSR_IA32_MISC_ENABLE,
                                  MSR_IA32_MISC_ENABLE_LIMIT_CPUID_BIT) > 0) {
                        c->cpuid_level = cpuid_eax(0);
                        get_cpu_cap(c);
                }
        }
Quoted 4 lines

in early_init_intel(). If you feel like playing, you might comment it out to see what happens.

:-)

Indeed, I bet that makes a difference!

I wish that 'unmasking' logic came with more comments:

- Why do BIOSen ever mask CPUIDs?

- Why do we unmask the masking?

- Why doesn't the kernel keep on working just fine even if certain CPUID aspects are turned off?

Thanks,

Ingo

maintainer
#16 lore raw
Quoted 3 lines · Ingo Molnar

I wish that 'unmasking' logic came with more comments:

- Why do BIOSen ever mask CPUIDs?

Doesn't say a thing why:

066941bd4eeb ("x86: unmask CPUID levels on Intel CPUs")

SDM doesn't say why either:

"Limit CPUID Maxval (R/W)

When this bit is set to 1, CPUID.00H returns a maximum value in EAX[7:0] of 3. BIOS should contain a setup question that allows users to specify when the installed OS does not support CPUID functions greater than 3.

...

Setting this bit may cause unexpected behavior in software that depends on the availability of CPUID leaves greater than 3."

In the case of hiding XSAVE from windoze ninety-old, probably something was exploding there, there wasn't a fix to the software so the hardware had to become soft.

Purely hypothetical, of course.

The last sentence from the SDM quote above also explains why the Linux workaround of clearing that bit again, exists.

Quoted 1 line

- Why do we unmask the masking?

Also purely hypothetical: because Linux doesn't have the windoze problem.

Quoted 2 lines

- Why doesn't the kernel keep on working just fine even if certain CPUID aspects are turned off?

I guess that should be doable but one has to get such a box, enable that BIOS feature and fix all the fallout that happens. Provided it can be fixed. hpa went and reenabled those CPUID leafs instead in 066941bd4eeb1.

I guess we simply shouldn't do any CPUID-dependent stuff before:

        if (this_cpu->c_early_init)
                this_cpu->c_early_init(c);

and slap a big fat comment above it.

Signature · 5 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--
#17 lore raw
In reply to Ingo Molnar
Quoted 7 lines · Ingo Molnar

Indeed, I bet that makes a difference!

I wish that 'unmasking' logic came with more comments:

- Why do BIOSen ever mask CPUIDs?

To work around bugs in legacy operating systems.

Quoted 1 line

- Why do we unmask the masking?

Because we don't have those specific bugs.

Quoted 2 lines

- Why doesn't the kernel keep on working just fine even if certain CPUID aspects are turned off?

Because it exercises code paths that are otherwise impossible, for example, it exposes the XSAVE capability without exposing the XSAVE information in higher CPUID leaves.

The other option would be to have a list of CPU features that should be turned off whenever the CPUID leaf maximum is too low, but it gives a better user experience to just override the BIOS capping and then we have fewer code paths in the kernel to worry about.

-hpa

#18 lore raw
Quoted 6 lines · H. Peter Anvin
Quoted 5 lines · Ingo Molnar

Indeed, I bet that makes a difference!

I wish that 'unmasking' logic came with more comments:

Quoted 8 lines
Quoted 2 lines

- Why do BIOSen ever mask CPUIDs?

To work around bugs in legacy operating systems.

Quoted 1 line

- Why do we unmask the masking?

Because we don't have those specific bugs.

Great - would be nice to put those reasons between /* */ markers, to keep future generations (and overworked maintainers!) from wondering.

Quoted 11 lines
Quoted 2 lines

- Why doesn't the kernel keep on working just fine even if certain CPUID aspects are turned off?

Because it exercises code paths that are otherwise impossible, for example, it exposes the XSAVE capability without exposing the XSAVE information in higher CPUID leaves.

The other option would be to have a list of CPU features that should be turned off whenever the CPUID leaf maximum is too low, but it gives a better user experience to just override the BIOS capping and then we have fewer code paths in the kernel to worry about.

1)

As a side note, I think we should generally be robust enough to recognize pretty much any CPUID 'mischief' and at minimum not crash.

2)

But this FPU crash is different, here the reason for the crash is the following bug in the FPU code:

	fpu__init_system(); /* inits the FPU based on masked CPUID */

	... CPUID *extends* ...

	fpu__init_cpu(); /* Actually uses the FPU now based on the expanded CPUID */

        *KABOOM*

I.e. we (obviously) should not base half on the FPU logic on different CPUID bits than the other half of the FPU logic.

I'll queue up the fix, which is to do the early FPU init after our CPUID state stabilizes. (i.e. the second patch I sent to Mike.)

Thanks,

Ingo

#19 lore raw
In reply to Borislav Petkov
Quoted 11 lines · Borislav Petkov

I bet it is that

        /* Unmask CPUID levels if masked: */
        if (c->x86 > 6 || (c->x86 == 6 && c->x86_model >= 0xd)) {
                if (msr_clear_bit(MSR_IA32_MISC_ENABLE,
                                  MSR_IA32_MISC_ENABLE_LIMIT_CPUID_BIT) > 0) {
                        c->cpuid_level = cpuid_eax(0);
                        get_cpu_cap(c);
                }
        }
Quoted 2 lines

in early_init_intel(). If you feel like playing, you might comment it out to see what happens.

With it commented out, and fpu__init_system() either back at previously booting position [5] or at original [0], doesn't matter, box is dead, but differently. It stalls after setting clocksource to tsc, and just sits there.

-Mike

maintainer
#20 lore raw
Quoted 4 lines · Mike Galbraith

With it commented out, and fpu__init_system() either back at previously booting position [5] or at original [0], doesn't matter, box is dead, but differently. It stalls after setting clocksource to tsc, and just sits there.

... which means that unmasking the CPUID features is absolutely needed on Linux. Not unmasking probably triggers this original bug which

066941bd4eeb ("x86: unmask CPUID levels on Intel CPUs")

fixed.

Signature · 5 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--
#21 lore raw
Quoted 6 lines · Borislav Petkov
Quoted 4 lines · Mike Galbraith

With it commented out, and fpu__init_system() either back at previously booting position [5] or at original [0], doesn't matter, box is dead, but differently. It stalls after setting clocksource to tsc, and just sits there.

Quoted 6 lines

... which means that unmasking the CPUID features is absolutely needed on Linux. Not unmasking probably triggers this original bug which

066941bd4eeb ("x86: unmask CPUID levels on Intel CPUs")

fixed.

Yes.

And I'd consider us hanging a separate (but not high prio) bug: the kernel should be robust as long as the CPUID data is stable. In that sense the original fix is right (we really want to unmask all available CPUID leaves), but it also masked another (less severe) kernel bug.

For example virtualization is known to tweak CPUID details creatively, and firmware (as this example shows it) can mess it up a well, so we generally want to treat it as untrusted input data that needs to be validated.

Thanks,

Ingo

#22 lore raw
Quoted 18 lines · Ingo Molnar
Quoted 6 lines · Borislav Petkov
Quoted 4 lines · Mike Galbraith

With it commented out, and fpu__init_system() either back at previously booting position [5] or at original [0], doesn't matter, box is dead, but differently. It stalls after setting clocksource to tsc, and just sits there.

Quoted 6 lines

... which means that unmasking the CPUID features is absolutely needed on Linux. Not unmasking probably triggers this original bug which

066941bd4eeb ("x86: unmask CPUID levels on Intel CPUs")

fixed.

Yes.

Quoted 9 lines

And I'd consider us hanging a separate (but not high prio) bug: the kernel should be robust as long as the CPUID data is stable. In that sense the original fix is right (we really want to unmask all available CPUID leaves), but it also masked another (less severe) kernel bug.

For example virtualization is known to tweak CPUID details creatively, and firmware (as this example shows it) can mess it up a well, so we generally want to treat it as untrusted input data that needs to be validated.

Well, that is not *entirely* possible, since if the data is just plain wrong, we're screwed no matter what.

However, we could deal with CPUID level capping. The best way to do that is probably to have a table of CPU features and the minimum required CPUID level for each. If maximum CPUID level < that level, disable that feature.

-hpa

#23 lore raw
In reply to Ingo Molnar
Quoted 8 lines · Ingo Molnar

And I'd consider us hanging a separate (but not high prio) bug: the kernel should be robust as long as the CPUID data is stable. In that sense the original fix is right (we really want to unmask all available CPUID leaves), but it also masked another (less severe) kernel bug.

For example virtualization is known to tweak CPUID details creatively, and firmware (as this example shows it) can mess it up a well, so we generally want to treat it as untrusted input data that needs to be validated.

Processor microcode updates can also change cpuid information, at least on Intel. There are Intel microcode updates in the field that do this.

Specific Intel MSR writes *should* be able to change cpuid information as well, as they enable/disable features that are reflected by a cpuid bit.

I have no idea about AMD, though.

Signature · 4 lines
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
#24 lore raw
In reply to Ingo Molnar
Quoted 26 lines · Ingo Molnar

Ok, so could you please move the fpu__init_system() further up and see which position is that starts breaking with the BIOS option set?

here's the current, broken layout of the code:

        get_cpu_cap(c);

[0] fpu__init_system(c);

        if (this_cpu->c_early_init)
                this_cpu->c_early_init(c);

[1]

        c->cpu_index = 0;
[2]
        filter_cpuid_features(c, false);

[3]

        if (this_cpu->c_bsp_init)
                this_cpu->c_bsp_init(c);

[4]

        setup_force_cpu_cap(X86_FEATURE_ALWAYS);

[5]
}

Quoted 8 lines

and we know it from your testing that moving [0] to [5] fixes the crash.

The question is, can we move it to [4], [3], [2] or even [1] instead, without breaking the system?

I still don't see where the breakage comes from, but this would help us narrow it down.

It should be moved to [4] or [5]. I would argue that the line setting X86_FEATURE_ALWAYS should moved up and then fpu__init_system(c) should be moved after the c_bsp_init() line.

-hpa

#25 lore raw
Quoted 32 lines · H. Peter Anvin
Quoted 31 lines · Ingo Molnar

Ok, so could you please move the fpu__init_system() further up and see which position is that starts breaking with the BIOS option set?

here's the current, broken layout of the code:

        get_cpu_cap(c);

[0] fpu__init_system(c);

        if (this_cpu->c_early_init)
                this_cpu->c_early_init(c);

[1]

        c->cpu_index = 0;
[2]
        filter_cpuid_features(c, false);

[3]

        if (this_cpu->c_bsp_init)
                this_cpu->c_bsp_init(c);

[4]

        setup_force_cpu_cap(X86_FEATURE_ALWAYS);

[5]
}

and we know it from your testing that moving [0] to [5] fixes the crash.

The question is, can we move it to [4], [3], [2] or even [1] instead, without breaking the system?

Quoted 7 lines
Quoted 3 lines

I still don't see where the breakage comes from, but this would help us narrow it down.

It should be moved to [4] or [5]. I would argue that the line setting X86_FEATURE_ALWAYS should moved up and then fpu__init_system(c) should be moved after the c_bsp_init() line.

Yeah, so the patch I sent to Mike (and which solved the bug) moved it to [5].

Thanks,

Ingo

#26 lore raw

[tip:x86/urgent] x86/fpu: Fix FPU related boot regression when CPUID masking BIOS feature is enabled

In reply to Ingo Molnar
x86/fpu: Fix FPU related boot regression when CPUID masking BIOS feature is enabled
Commit-ID:  db52ef74b35dcb91fd154fa52c618bdd1b90e28e
Gitweb:     http://git.kernel.org/tip/db52ef74b35dcb91fd154fa52c618bdd1b90e28e
Author:     Ingo Molnar <[email protected]>
AuthorDate: Sat, 27 Jun 2015 10:25:14 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 30 Jun 2015 07:22:10 +0200

x86/fpu: Fix FPU related boot regression when CPUID masking BIOS feature is enabled

Mike Galbraith reported:

" My i7-4790 box is having one hell of a time with this merge
window, dead in the water.

BIOS setting "Limit CPUID Maximum" upsets new fpu code mightily. "

It turns out that Linux does a double workaround here, as per:

066941bd4eeb ("x86: unmask CPUID levels on Intel CPUs")

it undoes the BIOS workaround - but as a side effect the CPUID state is not completely constant during early init anymore, and the new FPU init code did not take this into account.

So what happened is that the xstate init code did not have full CPUID available, which broke subsequent attempts to use xstate features.

Fix this by ordering the early FPU init code to after we've stabilized the CPUID state.

Reported-bisected-and-tested-by Mike Galbraith <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc H. Peter Anvin <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 arch/x86/kernel/cpu/common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/common.c +1 -1
index 9fc5e3d..922c5e0 100644
@@ -742,7 +742,6 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
 	cpu_detect(c);
 	get_cpu_vendor(c);
 	get_cpu_cap(c);
-	fpu__init_system(c);
 
 	if (this_cpu->c_early_init)
 		this_cpu->c_early_init(c);
@@ -754,6 +753,7 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
 		this_cpu->c_bsp_init(c);
 
 	setup_force_cpu_cap(X86_FEATURE_ALWAYS);
+	fpu__init_system(c);
 }
 
 void __init early_cpu_init(void)
#27 lore raw
In reply to Mike Galbraith
Quoted 27 lines · Mike Galbraith

Hi Ingo,

My i7-4790 box is having one hell of a time with this merge window, is dead in the water. The netconsole log below is v4.1-7254-gc13c81006314, but trouble begins at bisected point much earlier. If I turn off kvm, such that I can kinda sorta boot, systemd says many services "enter failed state", box is pretty much a doorstop. Though I can get to a prompt, I can't login. If kvm is enabled, it explodes as soon as it autoloads (wtf does it do that when it's not being used?)

Bisecting to the beginning of my woes takes me to the below. Before that, it doesn't matter if kvm is enabled or not, all is well. Below the current gripage with kvm disabled, find the kvm explosion, and another explosion as I approach the beginning of my box's woes.

067051ccd209623cb56152cf4cb06616ee2bcc5c is the first bad commit commit 067051ccd209623cb56152cf4cb06616ee2bcc5c

Author: Ingo Molnar <[email protected]>
Date:   Sat Apr 25 08:27:44 2015 +0200

x86/fpu: Do system-wide setup from fpu__detect()

fpu__cpu_init() is called on every CPU, so it is the wrong place to call fpu__init_system() from. Call it from fpu__detect():
this is early CPU init code, but we already have CPU features detected, so we can call the system-wide FPU init code from here.

Quoted 10 lines
Cc Andy Lutomirski <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Fenghua Yu <[email protected]>
Cc H. Peter Anvin <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Oleg Nesterov <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>

Just as a quick workaround, if you add back a per CPU init fpu__init_system() call, as per the disgusting hack below, do things get happier?

( You might trigger a few WARN_ON_ONCE() whinges, especially if you have
CONFIG_X86_DEBUG_FPU=y, but those should be one-time warnings that are not fatal. )

Totally untested, unfortunately.

My theory of the bug is that there is something that needs to be set up per CPU, which is a side effect of fpu__init_system(), and which the new fpu__init_cpu() does not capture. If this patch helps then the real fix would be to figure out that side effect.

Thanks,

Ingo

 arch/x86/kernel/fpu/init.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
arch/x86/kernel/fpu/init.c +12 -3
index fc878fee6a51..421babb08fe6 100644
@@ -4,6 +4,9 @@
 #include <asm/fpu/internal.h>
 #include <asm/tlbflush.h>
 
+#undef __init
+#define __init
+
 /*
  * Initialize the TS bit in CR0 according to the style of context-switches
  * we are using:
@@ -44,13 +47,18 @@ static void fpu__init_cpu_generic(void)
 /*
  * Enable all supported FPU features. Called when a CPU is brought online:
  */
-void fpu__init_cpu(void)
+static void __fpu__init_cpu(void)
 {
 	fpu__init_cpu_generic();
 	fpu__init_cpu_xstate();
 	fpu__init_cpu_ctx_switch();
 }
 
+void fpu__init_cpu(void)
+{
+	fpu__init_system(NULL);
+}
+
 /*
  * The earliest FPU detection code.
  *
@@ -267,13 +275,14 @@ static void __init fpu__init_system_ctx_switch(void)
  */
 void __init fpu__init_system(struct cpuinfo_x86 *c)
 {
-	fpu__init_system_early_generic(c);
+	if (c)
+		fpu__init_system_early_generic(c);
 
 	/*
 	 * The FPU has to be operational for some of the
 	 * later FPU init activities:
 	 */
-	fpu__init_cpu();
+	__fpu__init_cpu();
 
 	/*
 	 * But don't leave CR0::TS set yet, as some of the FPU setup