PROBLEM: linux 3.16 APIC and bhyve won't boot
Hello,
[1.] One line summary of the problem:
On Linux 3.16 a custom-built kernel with bhyve won't boot and will hang on the APIC timer calibration. [2.] Full description of the problem/report:
I'm booting a 3.16 in bhyve (FreeBSD hypervisor) and according to my tests lapic_cal_handler never gets called. This prevents the lapic_cal_loops
counter from being incremented and therefore the APIC calibration never
finishes. It is supposed to be called 25 times (APIC_CAL_LOOPS
constant). I added some apic_printk's to check for these infos (not a
best practice but I don't have access to a debugger in this specific
context).
[3.] Keywords (i.e., modules, networking, kernel):
kernel, apic
[4.] Kernel version (from /proc/version):
n/a (won't boot)
Linux version 3.16.1-bhyve (root@localhost) (gcc version 4.8.2 (Funtoo 4.8.2-r2) ) #22 SMP Fri Aug 29 12:23:32 Local
[5.] Output of Oops.. message (if applicable) with symbolic information
resolved (see Documentation/oops-tracing.txt)
N/A
[6.] A small shell script or example program which triggers the
problem (if possible)
N/A
[7.] Environment
[7.1.] Software (add the output of the ver_linux script here) If some fields are empty or look unusual you may have an old version. Compare to the current minimal requirements in Documentation/Changes.
Linux localhost 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64 Intel(R) Xeon(R) CPU E3-1240 v3 @ 3.40GHz GenuineIntel GNU/Linux
Gnu C 4.8.2 Gnu make 4.0 binutils 2.23.2 1.0 2.23.2 util-linux scripts/ver_linux: line 23: fdformat: command not found mount assert module-init-tools 15 e2fsprogs 1.42.10 xfsprogs 3.1.11 Linux C Library 2.18 Dynamic linker (ldd) 2.18 Procps 3.3.9 Net-tools 1.60_p20130513023548 Kbd 2.0.1 Sh-utils 8.21 Modules Loaded ext4 crc16 jbd2 mbcache sg sr_mod cdrom virtio_blk virtio_net ahci libahci libata scsi_mod virtio_pci virtio_ring virtio
[7.2.] Processor information (from /proc/cpuinfo):
[snip]
processor : 3vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel(R) Xeon(R) CPU E3-1240 v3 @ 3.40GHz
stepping : 3
cpu MHz : 3388.196
cache size : 0 KB
fpu : yesfpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae cx8 apic sep pge cmov pat pse36 clflush mmx fxsr sse sse2 ss pbe syscall nx pdpe1gb lm constant_tsc rep_good nopl nonstop_tsc pni pclmulqdq dtes64 ds_cpl smx ssse3 fma cx16 xtpr pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm xsaveopt fsgsbase erms
bogomips : 6776.74
clflush size : 64cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
[7.3.] Module information (from /proc/modules):
N/A
[7.4.] Loaded driver and hardware information (/proc/ioports, /proc/iomem) /proc/ioports
0000-0cf7 : PCI Bus 0000:00
0000-001f : dma1
0020-0021 : pic1
0040-0043 : timer0
0050-0053 : timer1
0060-0060 : keyboard
0064-0064 : keyboard
0080-008f : dma page reg
00a0-00a1 : pic2
00c0-00df : dma2
00f0-00ff : fpu
0220-0223 : pnp 00:01
0224-0227 : pnp 00:01
02f8-02ff : serial
03f8-03ff : serial
0400-0407 : pnp 00:01
0400-0403 : ACPI PM1a_EVT_BLK
0404-0405 : ACPI PM1a_CNT_BLK 0408-040b : ACPI PM_TMR
04d0-04d1 : pnp 00:01
0cf8-0cff : PCI conf1
0d00-1fff : PCI Bus 0000:00
2000-209f : PCI Bus 0000:00
2000-203f : 0000:00:01.0
2000-203f : virtio-pci
2040-205f : 0000:00:02.0
2040-205f : virtio-pci
2060-207f : 0000:00:03.0
2060-207f : virtio-pci/proc/iomem
00000000-0000ffff : reserved
00010000-0009fffe : System RAM
000f0000-000fffff : System ROM
00100000-bfffffff : System RAM
01000000-01359d65 : Kernel code
01359d66-01694f7f : Kernel data
0172a000-01807fff : Kernel bss
c0000000-c01fffff : PCI Bus 0000:00
c0000000-c0001fff : 0000:00:01.0
c0000000-c0001fff : virtio-pci
c0002000-c0003fff : 0000:00:02.0
c0002000-c0003fff : virtio-pci
c0004000-c0005fff : 0000:00:03.0
c0004000-c0005fff : virtio-pci
c0006000-c00063ff : 0000:00:04.0
c0006000-c00063ff : ahci c0006800-c0006fff : 0000:00:01.0
c0007000-c00077ff : 0000:00:02.0
c0007800-c0007fff : 0000:00:03.0
c0008000-c00087ff : 0000:00:04.0
c0008800-c0008fff : 0000:00:0f.0
e0000000-efffffff : PCI MMCONFIG 0000 [bus 00-ff]
e0000000-efffffff : pnp 00:01
fec00000-fec003ff : IOAPIC 0
fed00000-fed003ff : HPET 0
fee00000-fee00fff : Local APIC
100000000-13fffffff : System RAM
[7.5.] PCI information ('lspci -vvv' as root)
root@localhost:/# lspci -vvv 00:00.0 Host bridge: Network Appliance Corporation Device 1275 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF+ FastB2B+ ParErr+ DEVSEL=?? >TAbort+ <TAbort+ <MAbort+ >SERR+ <PERR+ INTx+ Latency: 64 Bus: primary=00, secondary=00, subordinate=00, sec-latency=0 I/O behind bridge: 00000000-00000fff Memory behind bridge: 00000000-000fffff Prefetchable memory behind bridge: 00000000-000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot-), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE- FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Latency L0 <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Not Supported, TimeoutDis- ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
00:01.0 SCSI storage controller: Red Hat, Inc Virtio block device
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64 Interrupt: pin A routed to IRQ 16 Region 0: I/O ports at 2000 [size=64] Region 1: Memory at c0000000 (32-bit, non-prefetchable) [size=8K] [virtual] Expansion ROM at c0006800 [disabled] [size=2K] Capabilities: [40] MSI-X: Enable+ Count=2 Masked- Vector table: BAR=1 offset=00000000 PBA: BAR=1 offset=00001000 Capabilities: [4c] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000
00:02.0 Ethernet controller: Red Hat, Inc Virtio network device
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64 Interrupt: pin A routed to IRQ 17 Region 0: I/O ports at 2040 [size=32] Region 1: Memory at c0002000 (32-bit, non-prefetchable) [size=8K] [virtual] Expansion ROM at c0007000 [disabled] [size=2K] Capabilities: [40] MSI-X: Enable+ Count=3 Masked- Vector table: BAR=1 offset=00000000 PBA: BAR=1 offset=00001000 Capabilities: [4c] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000
00:03.0 Ethernet controller: Red Hat, Inc Virtio network device
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64 Interrupt: pin A routed to IRQ 18 Region 0: I/O ports at 2060 [size=32] Region 1: Memory at c0004000 (32-bit, non-prefetchable) [size=8K] [virtual] Expansion ROM at c0007800 [disabled] [size=2K] Capabilities: [40] MSI-X: Enable+ Count=3 Masked- Vector table: BAR=1 offset=00000000 PBA: BAR=1 offset=00001000 Capabilities: [4c] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000
00:04.0 SATA controller: Intel Corporation 82801HR/HO/HH (ICH8R/DO/DH) 6 port SATA Controller [AHCI mode] (prog-if 01 [AHCI 1.0])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64 Interrupt: pin A routed to IRQ 44 Region 5: Memory at c0006000 (32-bit, non-prefetchable) [size=1K] [virtual] Expansion ROM at c0008000 [disabled] [size=2K] Capabilities: [40] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00000 Data: 4091
00:0f.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
[virtual] Expansion ROM at c0008800 [disabled] [size=2K]
root@localhost:/#
[7.6.] SCSI information (from /proc/scsi/scsi)
N/A
[7.7.] Other information that might be relevant to the problem
A stock Debian 3.2 kernel will boot. I don't have the possibility to build such a bloated kernel to see what's missing. My goal is to identify what is the minimal set of functionalities needed to have a kernel start up in a bhyve context. I have several other Linux VMs able to run on this hypervisor, none with such a recent kernel.
Yours sincerely,
Signature · 2 lines
Chloé Desoutter C[A-Z]O, Atasta NET
Hello,
Quoted 4 lines · Chloé Desoutter
[1.] One line summary of the problem:
On Linux 3.16 a custom-built kernel with bhyve won't boot and will hang on the APIC timer calibration.
Wow I haven't seen a report in this style in a while...
Quoted 8 lines
[2.] Full description of the problem/report:
I'm booting a 3.16 in bhyve (FreeBSD hypervisor) and according to my testslapic_cal_handlernever gets called. This prevents thelapic_cal_loopscounter from being incremented and therefore the APIC calibration never finishes. It is supposed to be called 25 times (APIC_CAL_LOOPSconstant). I added some apic_printk's to check for these infos (not a best practice but I don't have access to a debugger in this specific context).
<snip>
Quoted 6 lines
[7.7.] Other information that might be relevant to the problem A stock Debian 3.2 kernel will boot. I don't have the possibility to build such a bloated kernel to see what's missing. My goal is to identify what is the minimal set of functionalities needed to have a kernel start up in a bhyve context. I have several other Linux VMs able to run on this hypervisor, none with such a recent kernel.
Without being able to email explicit people in addition to the general LKML there's a good chance this will be overlooked. I can only guess the problem has something to do with arch/x86/kernel/apic/apic.c but I don't really know...
What will help a lot is if you can try your cut down 3.16 but modify it using make oldconfig on an old 3.2 kernel and see if that boots. If it does, then you can use git bisect v3.16 v3.2 to narrow down the exact commit that introduced the problem.
Signature · 1 line
Sitsofe | http://sucs.org/~sits/