drm/mgag200: doesn't work in panic context
Hi all,
I'm here to report two panics which hang forever (the machine cannot reboot). It is because mgag200 doesn't work in panic context. It sleeps and allocates memory non-atomically.
These were triggered while injecting machine checks using einj.
1)
[321381.466885] ------------[ cut here ]------------ [321381.472144] WARNING: CPU: 136 PID: 0 at kernel/time/timer.c:1098 del_timer_sync+0x36/0x60() [321381.481571] Modules linked in: einj(E) nmioe(E) iscsi_ibft(E) iscsi_boot_sysfs(E) af_packet(E) x86_pkg_temp_thermal(E) btrfs(E) intel_powerclamp(E) coretemp(E) kvm(E) xor(E) crct10dif_pclmul(E) raid6_pq(E) crc32_pclmul(E) crc32c_intel(E) ghash_clmulni_intel(E) iTCO_wdt(E) iTCO_vendor_support(E) joydev(E) aesni_intel(E) lpc_ich(E) aes_x86_64(E) lrw(E) gf128mul(E) glue_helper(E) sb_edac(E) ablk_helper(E) cryptd(E) pcspkr(E) mfd_core(E) i2c_i801(E) wmi(E) edac_core(E) shpchp(E) ipmi_si(E) ipmi_msghandler(E) processor(E) acpi_pad(E) button(E) dm_mod(E) ext4(E) crc16(E) mbcache(E) jbd2(E) hid_generic(E) usbhid(E) sr_mod(E) cdrom(E) sd_mod(E) mgag200(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) ehci_pci(E) ehci_hcd(E) drm_kms_helper(E) ixgbe(E) ahci(E) igb(E) mdio(E) ttm(E) libahci(E) ptp(E) i2c_algo_bit(E) usbcore(E) pps_core(E) drm(E) libata(E) megaraid_sas(E) usb_common(E) dca(E) sg(E) scsi_mod(E) autofs4(E) [321381.572300] CPU: 136 PID: 0 Comm: swapper/136 Tainted: G W E 4.1.0-rc8-7-default+ #4 [321381.582117] Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0059.R00.1501081238 01/08/2015 [321381.593777] ffffffff81818089 ffff88047fc88808 ffffffff8157d67e 0000000000000000 [321381.602184] 0000000000000000 ffff88047fc88848 ffffffff810637fa ffff88046e4bc740 [321381.610595] ffff88047fc888a8 ffff88047fc888a8 0000000104c6f0f8 ffff88047f5cdb00 [321381.619006] Call Trace: [321381.621834] <#MC> [<ffffffff8157d67e>] dump_stack+0x4c/0x65 [321381.628358] [<ffffffff810637fa>] warn_slowpath_common+0x8a/0xc0 [321381.635168] [<ffffffff810638ea>] warn_slowpath_null+0x1a/0x20 [321381.641775] [<ffffffff810cb316>] del_timer_sync+0x36/0x60 [321381.647995] [<ffffffff81582bf0>] schedule_timeout+0x150/0x280 [321381.654611] [<ffffffff812cc9fb>] ? idr_alloc+0x7b/0xe0 [321381.660547] [<ffffffff810c9c90>] ? internal_add_timer+0x80/0x80 [321381.667359] [<ffffffff810cb85c>] msleep+0x3c/0x50 [321381.672812] [<ffffffffa0145607>] mga_crtc_prepare+0x167/0x370 [mgag200] [321381.680404] [<ffffffffa04f38b6>] drm_crtc_helper_set_mode+0x2d6/0x530 [drm_kms_helper] [321381.689453] [<ffffffffa04f4896>] drm_crtc_helper_set_config+0x856/0xa70 [drm_kms_helper] [321381.698706] [<ffffffffa00a3318>] drm_mode_set_config_internal+0x68/0x100 [drm] [321381.706971] [<ffffffffa04fe8b2>] restore_fbdev_mode+0xc2/0xf0 [drm_kms_helper] [321381.715244] [<ffffffffa04feaa3>] drm_fb_helper_force_kernel_mode+0x73/0xb0 [drm_kms_helper] [321381.724780] [<ffffffffa04ff6f9>] drm_fb_helper_panic+0x29/0x30 [drm_kms_helper] [321381.733144] [<ffffffff8108270d>] notifier_call_chain+0x4d/0x80 [321381.739859] [<ffffffff81082791>] atomic_notifier_call_chain+0x21/0x30 [321381.747252] [<ffffffff815792d4>] panic+0xee/0x1f5 [321381.752704] [<ffffffff8102d272>] mce_panic+0x1e2/0x200 [321381.758640] [<ffffffff8102d303>] mce_timed_out+0x73/0x80 [321381.764762] [<ffffffff8102e8a1>] do_machine_check+0x5f1/0xae0 [321381.771377] [<ffffffff81348eaf>] ? intel_idle+0xbf/0x130 [321381.777499] [<ffffffff81585d49>] machine_check+0x29/0x50 [321381.783630] [<ffffffff81348eaf>] ? intel_idle+0xbf/0x130 [321381.789760] <<EOE>> [<ffffffff81450170>] cpuidle_enter_state+0x70/0x1f0 [321381.797457] [<ffffffff81450327>] cpuidle_enter+0x17/0x20 [321381.803586] [<ffffffff810a5968>] cpu_startup_entry+0x308/0x390 [321381.810297] [<ffffffff8103a203>] start_secondary+0x143/0x170 [321381.816814] ---[ end trace 9f2a977c4a9be24e ]--- [321381.822068] bad: scheduling from the idle thread! [321381.827421] CPU: 136 PID: 0 Comm: swapper/136 Tainted: G W E 4.1.0-rc8-7-default+ #4 [321381.837238] Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0059.R00.1501081238 01/08/2015 [321381.848898] ffff88046e4bc740 ffff88047fc887a8 ffffffff8157d67e 0000000000000000 [321381.857305] ffff88047fc95300 ffff88047fc887c8 ffffffff81093675 ffff88047fc88808 [321381.865713] ffff88047fc95300 ffff88047fc887f8 ffffffff8108796c 0000000100000000 [321381.874124] Call Trace: [321381.876951] <#MC> [<ffffffff8157d67e>] dump_stack+0x4c/0x65 [321381.883483] [<ffffffff81093675>] dequeue_task_idle+0x35/0x50 [321381.890001] [<ffffffff8108796c>] dequeue_task+0x5c/0x80 [321381.896027] [<ffffffff8108c56b>] deactivate_task+0x2b/0x30 [321381.902352] [<ffffffff8157fcea>] __schedule+0x64a/0x910 [321381.908385] [<ffffffff8157ffee>] schedule+0x3e/0x90 [321381.914030] [<ffffffff81582be8>] schedule_timeout+0x148/0x280 [321381.920636] [<ffffffff812cc9fb>] ? idr_alloc+0x7b/0xe0 [321381.926570] [<ffffffff810c9c90>] ? internal_add_timer+0x80/0x80 [321381.933382] [<ffffffff810cb85c>] msleep+0x3c/0x50 [321381.938835] [<ffffffffa0145607>] mga_crtc_prepare+0x167/0x370 [mgag200] [321381.946428] [<ffffffffa04f38b6>] drm_crtc_helper_set_mode+0x2d6/0x530 [drm_kms_helper] [321381.955478] [<ffffffffa04f4896>] drm_crtc_helper_set_config+0x856/0xa70 [drm_kms_helper] [321381.964731] [<ffffffffa00a3318>] drm_mode_set_config_internal+0x68/0x100 [drm] [321381.973004] [<ffffffffa04fe8b2>] restore_fbdev_mode+0xc2/0xf0 [drm_kms_helper] [321381.981277] [<ffffffffa04feaa3>] drm_fb_helper_force_kernel_mode+0x73/0xb0 [drm_kms_helper] [321381.990811] [<ffffffffa04ff6f9>] drm_fb_helper_panic+0x29/0x30 [drm_kms_helper] [321381.999174] [<ffffffff8108270d>] notifier_call_chain+0x4d/0x80 [321382.005887] [<ffffffff81082791>] atomic_notifier_call_chain+0x21/0x30 [321382.013280] [<ffffffff815792d4>] panic+0xee/0x1f5 [321382.018731] [<ffffffff8102d272>] mce_panic+0x1e2/0x200 [321382.024660] [<ffffffff8102d303>] mce_timed_out+0x73/0x80 [321382.030787] [<ffffffff8102e8a1>] do_machine_check+0x5f1/0xae0 [321382.037404] [<ffffffff81348eaf>] ? intel_idle+0xbf/0x130 [321382.043533] [<ffffffff81585d49>] machine_check+0x29/0x50 [321382.049665] [<ffffffff81348eaf>] ? intel_idle+0xbf/0x130 [321382.055794] <<EOE>> [<ffffffff81450170>] cpuidle_enter_state+0x70/0x1f0 [321382.063491] [<ffffffff81450327>] cpuidle_enter+0x17/0x20 [321382.069623] [<ffffffff810a5968>] cpu_startup_entry+0x308/0x390 [321382.076335] [<ffffffff8103a203>] start_secondary+0x143/0x170 [321382.082877] ------------[ cut here ]------------
2)
bkd04sdp:~ # [58109.056018] Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handler
[58109.056058] mce: [Hardware Error]: Machine check events logged [58110.109873] Shutting down cpus with NMI [58110.176778] Kernel Offset: disabled [58110.180667] drm_kms_helper: panic occurred, switching back to text console [58110.188367] mga_delay choosing mdelay... [58110.242399] mga_delay choosing mdelay... [58110.266768] ------------[ cut here ]------------ [58110.271926] kernel BUG at mm/vmalloc.c:1335! [58110.276695] invalid opcode: 0000 [#1] SMP [58110.281289] Modules linked in: einj(E) nmioe(E) iscsi_ibft(E) iscsi_boot_sysfs(E) af_packet(E) btrfs(E) xor(E) raid6_pq(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm(E) joydev(E) iTCO_wdt(E) iTCO_vendor_support(E) crct10dif_pclmul(E) crc32_pclmul(E) crc32c_intel(E) ghash_clmulni_intel(E) aesni_intel(E) aes_x86_64(E) lrw(E) gf128mul(E) glue_helper(E) sb_edac(E) ablk_helper(E) lpc_ich(E) cryptd(E) pcspkr(E) edac_core(E) mfd_core(E) i2c_i801(E) shpchp(E) wmi(E) ipmi_si(E) ipmi_msghandler(E) acpi_pad(E) processor(E) button(E) dm_mod(E) ext4(E) crc16(E) mbcache(E) jbd2(E) hid_generic(E) usbhid(E) sr_mod(E) cdrom(E) sd_mod(E) mgag200(E) syscopyarea(E) sysfillrect(E) ahci(E) ehci_pci(E) sysimgblt(E) drm_kms_helper(E) ehci_hcd(E) ixgbe(E) igb(E) ttm(E) libahci(E) mdio(E) ptp(E) usbcore(E) pps_core(E) drm(E) libata(E) i2c_algo_bit(E) usb_common(E) dca(E) megaraid_sas(E) sg(E) scsi_mod(E) autofs4(E) [58110.371884] CPU: 75 PID: 0 Comm: swapper/75 Tainted: G E 4.1.0-rc8-7-default+ #10 [58110.381506] Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0059.R00.1501081238 01/08/2015 [58110.393063] task: ffff88046ea6d580 ti: ffff88046ea70000 task.ti: ffff88046ea70000 [58110.401422] RIP: 0010:[<ffffffff81189c65>] [<ffffffff81189c65>] __get_vm_area_node+0x155/0x160 [58110.411156] RSP: 0018:ffff88047f7284b8 EFLAGS: 00010006 [58110.417091] RAX: 0000000080010003 RBX: 0000000091000000 RCX: ffffc90000000000 [58110.425065] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 00000000000eb000 [58110.433038] RBP: ffff88047f7284f8 R08: ffffe8ffffffffff R09: 00000000ffffffff [58110.441010] R10: ffff880036a6a700 R11: ffff880460ab69c0 R12: 00000000910eb000 [58110.448983] R13: 0000000000000001 R14: 0000000091000000 R15: 00000000000eb000 [58110.456955] FS: 0000000000000000(0000) GS:ffff88047f720000(0000) knlGS:0000000000000000 [58110.465994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [58110.472413] CR2: 00007f5bdb9ac095 CR3: 0000000001a0b000 CR4: 00000000001407e0 [58110.480386] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [58110.488358] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [58110.496331] Stack: [58110.498577] ffff88047f728518 ffffc90000000000 00000000910eafff 0000000091000000 [58110.506885] 00000000910eb000 0000000000000001 0000000091000000 00000000000eb000 [58110.515192] ffff88047f728518 ffffffff8118ad20 00000000000000d0 ffffffffa0334f78 [58110.523499] Call Trace: [58110.526229] <#MC> [58110.528382] [<ffffffff8118ad20>] get_vm_area_caller+0x40/0x50 [58110.535111] [<ffffffffa0334f78>] ? ttm_mem_reg_ioremap+0xc8/0x110 [ttm] [58110.542607] [<ffffffff81050f18>] __ioremap_caller+0x188/0x390 [58110.549127] [<ffffffff812d59b9>] ? find_next_bit+0x19/0x20 [58110.555353] [<ffffffff81051177>] ioremap_wc+0x17/0x20 [58110.561099] [<ffffffffa0334f78>] ttm_mem_reg_ioremap+0xc8/0x110 [ttm] [58110.568398] [<ffffffffa0335351>] ttm_bo_move_memcpy+0xd1/0x700 [ttm] [58110.575598] [<ffffffff811a6c55>] ? __kmalloc+0x4b5/0x4c0 [58110.581632] [<ffffffffa01d9b48>] mgag200_bo_move+0x18/0x20 [mgag200] [58110.588830] [<ffffffffa0332ea0>] ttm_bo_handle_move_mem+0x260/0x590 [ttm] [58110.596514] [<ffffffffa03337d2>] ? ttm_bo_mem_space+0xd2/0x320 [ttm] [58110.603705] [<ffffffffa0333eb2>] ttm_bo_validate+0x1c2/0x1d0 [ttm] [58110.610711] [<ffffffff8113f681>] ? irq_work_queue+0x11/0x90 [58110.617037] [<ffffffffa01da3d3>] mgag200_bo_push_sysram+0x93/0xe0 [mgag200] [58110.624915] [<ffffffffa01d5a26>] mga_crtc_do_set_base.isra.8.constprop.21+0x76/0x410 [mgag200] [58110.634636] [<ffffffffa01d6e02>] mga_crtc_mode_set+0x1042/0x2140 [mgag200] [58110.642416] [<ffffffffa01d5492>] ? mga_crtc_prepare+0x132/0x370 [mgag200] [58110.650106] [<ffffffffa04ec8db>] drm_crtc_helper_set_mode+0x2fb/0x530 [drm_kms_helper] [58110.659052] [<ffffffffa04ed896>] drm_crtc_helper_set_config+0x856/0xa70 [drm_kms_helper] [58110.668217] [<ffffffffa00b5318>] drm_mode_set_config_internal+0x68/0x100 [drm] [58110.676388] [<ffffffffa04f78b2>] restore_fbdev_mode+0xc2/0xf0 [drm_kms_helper] [58110.684558] [<ffffffffa04f7aa3>] drm_fb_helper_force_kernel_mode+0x73/0xb0 [drm_kms_helper] [58110.693989] [<ffffffffa04f86f9>] drm_fb_helper_panic+0x29/0x30 [drm_kms_helper] [58110.702260] [<ffffffff81081bad>] notifier_call_chain+0x4d/0x80 [58110.708873] [<ffffffff81081c31>] atomic_notifier_call_chain+0x21/0x30 [58110.716169] [<ffffffff8156f954>] panic+0xee/0x1f5 [58110.721530] [<ffffffff8102d232>] mce_panic+0x1e2/0x200 [58110.727366] [<ffffffff8102d2c3>] mce_timed_out+0x73/0x80 [58110.733396] [<ffffffff8102e861>] do_machine_check+0x5f1/0xae0 [58110.739915] [<ffffffff8133f52f>] ? intel_idle+0xbf/0x130 [58110.745952] [<ffffffff8157c3c9>] machine_check+0x29/0x50 [58110.751984] [<ffffffff8133f52f>] ? intel_idle+0xbf/0x130 [58110.758017] <<EOE>> [58110.760362] [<ffffffff814467f0>] cpuidle_enter_state+0x70/0x1f0 [58110.767275] [<ffffffff814469a7>] cpuidle_enter+0x17/0x20 [58110.773309] [<ffffffff810a4e18>] cpu_startup_entry+0x308/0x390 [58110.779916] [<ffffffff8103a163>] start_secondary+0x143/0x170 [58110.786325] Code: 00 00 48 0f bd cf 83 c1 01 83 f9 0c 0f 4c c8 b0 1e 83 f9 1e 0f 4f c8 49 d3 e6 e9 f8 fe ff ff 48 89 df e8 9f a8 01 00 31 c0 eb b8 <0f> 0b 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 49 89 c8 41 [58110.808146] RIP [<ffffffff81189c65>] __get_vm_area_node+0x155/0x160 [58110.815257] RSP <ffff88047f7284b8> [58110.820218] ---[ end trace ab0c230901a0ee95 ]---
Thanks
Rui
Quoted 3 lines · Rui Wang
Hi all,
I'm here to report two panics which hang forever (the machine cannot reboot). It is because mgag200 doesn't work in panic context. It sleeps and allocates memory non-atomically.
This is the same for all drm drivers, the drm atomic handling with fbcon/fbdev is totally broken. It would be serious work to fix this properly.
-Daniel
Quoted 162 lines
These were triggered while injecting machine checks using einj.
1)
[321381.466885] ------------[ cut here ]------------ [321381.472144] WARNING: CPU: 136 PID: 0 at kernel/time/timer.c:1098 del_timer_sync+0x36/0x60() [321381.481571] Modules linked in: einj(E) nmioe(E) iscsi_ibft(E) iscsi_boot_sysfs(E) af_packet(E) x86_pkg_temp_thermal(E) btrfs(E) intel_powerclamp(E) coretemp(E) kvm(E) xor(E) crct10dif_pclmul(E) raid6_pq(E) crc32_pclmul(E) crc32c_intel(E) ghash_clmulni_intel(E) iTCO_wdt(E) iTCO_vendor_support(E) joydev(E) aesni_intel(E) lpc_ich(E) aes_x86_64(E) lrw(E) gf128mul(E) glue_helper(E) sb_edac(E) ablk_helper(E) cryptd(E) pcspkr(E) mfd_core(E) i2c_i801(E) wmi(E) edac_core(E) shpchp(E) ipmi_si(E) ipmi_msghandler(E) processor(E) acpi_pad(E) button(E) dm_mod(E) ext4(E) crc16(E) mbcache(E) jbd2(E) hid_generic(E) usbhid(E) sr_mod(E) cdrom(E) sd_mod(E) mgag200(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) ehci_pci(E) ehci_hcd(E) drm_kms_helper(E) ixgbe(E) ahci(E) igb(E) mdio(E) ttm(E) libahci(E) ptp(E) i2c_algo_bit(E) usbcore(E) pps_core(E) drm(E) libata(E) megaraid_sas(E) usb_common(E) dca(E) sg(E) scsi_mod(E) autofs4(E) [321381.572300] CPU: 136 PID: 0 Comm: swapper/136 Tainted: G W E 4.1.0-rc8-7-default+ #4 [321381.582117] Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0059.R00.1501081238 01/08/2015 [321381.593777] ffffffff81818089 ffff88047fc88808 ffffffff8157d67e 0000000000000000 [321381.602184] 0000000000000000 ffff88047fc88848 ffffffff810637fa ffff88046e4bc740 [321381.610595] ffff88047fc888a8 ffff88047fc888a8 0000000104c6f0f8 ffff88047f5cdb00 [321381.619006] Call Trace: [321381.621834] <#MC> [<ffffffff8157d67e>] dump_stack+0x4c/0x65 [321381.628358] [<ffffffff810637fa>] warn_slowpath_common+0x8a/0xc0 [321381.635168] [<ffffffff810638ea>] warn_slowpath_null+0x1a/0x20 [321381.641775] [<ffffffff810cb316>] del_timer_sync+0x36/0x60 [321381.647995] [<ffffffff81582bf0>] schedule_timeout+0x150/0x280 [321381.654611] [<ffffffff812cc9fb>] ? idr_alloc+0x7b/0xe0 [321381.660547] [<ffffffff810c9c90>] ? internal_add_timer+0x80/0x80 [321381.667359] [<ffffffff810cb85c>] msleep+0x3c/0x50 [321381.672812] [<ffffffffa0145607>] mga_crtc_prepare+0x167/0x370 [mgag200] [321381.680404] [<ffffffffa04f38b6>] drm_crtc_helper_set_mode+0x2d6/0x530 [drm_kms_helper] [321381.689453] [<ffffffffa04f4896>] drm_crtc_helper_set_config+0x856/0xa70 [drm_kms_helper] [321381.698706] [<ffffffffa00a3318>] drm_mode_set_config_internal+0x68/0x100 [drm] [321381.706971] [<ffffffffa04fe8b2>] restore_fbdev_mode+0xc2/0xf0 [drm_kms_helper] [321381.715244] [<ffffffffa04feaa3>] drm_fb_helper_force_kernel_mode+0x73/0xb0 [drm_kms_helper] [321381.724780] [<ffffffffa04ff6f9>] drm_fb_helper_panic+0x29/0x30 [drm_kms_helper] [321381.733144] [<ffffffff8108270d>] notifier_call_chain+0x4d/0x80 [321381.739859] [<ffffffff81082791>] atomic_notifier_call_chain+0x21/0x30 [321381.747252] [<ffffffff815792d4>] panic+0xee/0x1f5 [321381.752704] [<ffffffff8102d272>] mce_panic+0x1e2/0x200 [321381.758640] [<ffffffff8102d303>] mce_timed_out+0x73/0x80 [321381.764762] [<ffffffff8102e8a1>] do_machine_check+0x5f1/0xae0 [321381.771377] [<ffffffff81348eaf>] ? intel_idle+0xbf/0x130 [321381.777499] [<ffffffff81585d49>] machine_check+0x29/0x50 [321381.783630] [<ffffffff81348eaf>] ? intel_idle+0xbf/0x130 [321381.789760] <<EOE>> [<ffffffff81450170>] cpuidle_enter_state+0x70/0x1f0 [321381.797457] [<ffffffff81450327>] cpuidle_enter+0x17/0x20 [321381.803586] [<ffffffff810a5968>] cpu_startup_entry+0x308/0x390 [321381.810297] [<ffffffff8103a203>] start_secondary+0x143/0x170 [321381.816814] ---[ end trace 9f2a977c4a9be24e ]--- [321381.822068] bad: scheduling from the idle thread! [321381.827421] CPU: 136 PID: 0 Comm: swapper/136 Tainted: G W E 4.1.0-rc8-7-default+ #4 [321381.837238] Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0059.R00.1501081238 01/08/2015 [321381.848898] ffff88046e4bc740 ffff88047fc887a8 ffffffff8157d67e 0000000000000000 [321381.857305] ffff88047fc95300 ffff88047fc887c8 ffffffff81093675 ffff88047fc88808 [321381.865713] ffff88047fc95300 ffff88047fc887f8 ffffffff8108796c 0000000100000000 [321381.874124] Call Trace: [321381.876951] <#MC> [<ffffffff8157d67e>] dump_stack+0x4c/0x65 [321381.883483] [<ffffffff81093675>] dequeue_task_idle+0x35/0x50 [321381.890001] [<ffffffff8108796c>] dequeue_task+0x5c/0x80 [321381.896027] [<ffffffff8108c56b>] deactivate_task+0x2b/0x30 [321381.902352] [<ffffffff8157fcea>] __schedule+0x64a/0x910 [321381.908385] [<ffffffff8157ffee>] schedule+0x3e/0x90 [321381.914030] [<ffffffff81582be8>] schedule_timeout+0x148/0x280 [321381.920636] [<ffffffff812cc9fb>] ? idr_alloc+0x7b/0xe0 [321381.926570] [<ffffffff810c9c90>] ? internal_add_timer+0x80/0x80 [321381.933382] [<ffffffff810cb85c>] msleep+0x3c/0x50 [321381.938835] [<ffffffffa0145607>] mga_crtc_prepare+0x167/0x370 [mgag200] [321381.946428] [<ffffffffa04f38b6>] drm_crtc_helper_set_mode+0x2d6/0x530 [drm_kms_helper] [321381.955478] [<ffffffffa04f4896>] drm_crtc_helper_set_config+0x856/0xa70 [drm_kms_helper] [321381.964731] [<ffffffffa00a3318>] drm_mode_set_config_internal+0x68/0x100 [drm] [321381.973004] [<ffffffffa04fe8b2>] restore_fbdev_mode+0xc2/0xf0 [drm_kms_helper] [321381.981277] [<ffffffffa04feaa3>] drm_fb_helper_force_kernel_mode+0x73/0xb0 [drm_kms_helper] [321381.990811] [<ffffffffa04ff6f9>] drm_fb_helper_panic+0x29/0x30 [drm_kms_helper] [321381.999174] [<ffffffff8108270d>] notifier_call_chain+0x4d/0x80 [321382.005887] [<ffffffff81082791>] atomic_notifier_call_chain+0x21/0x30 [321382.013280] [<ffffffff815792d4>] panic+0xee/0x1f5 [321382.018731] [<ffffffff8102d272>] mce_panic+0x1e2/0x200 [321382.024660] [<ffffffff8102d303>] mce_timed_out+0x73/0x80 [321382.030787] [<ffffffff8102e8a1>] do_machine_check+0x5f1/0xae0 [321382.037404] [<ffffffff81348eaf>] ? intel_idle+0xbf/0x130 [321382.043533] [<ffffffff81585d49>] machine_check+0x29/0x50 [321382.049665] [<ffffffff81348eaf>] ? intel_idle+0xbf/0x130 [321382.055794] <<EOE>> [<ffffffff81450170>] cpuidle_enter_state+0x70/0x1f0 [321382.063491] [<ffffffff81450327>] cpuidle_enter+0x17/0x20 [321382.069623] [<ffffffff810a5968>] cpu_startup_entry+0x308/0x390 [321382.076335] [<ffffffff8103a203>] start_secondary+0x143/0x170 [321382.082877] ------------[ cut here ]------------2)
bkd04sdp:~ # [58109.056018] Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handler
[58109.056058] mce: [Hardware Error]: Machine check events logged [58110.109873] Shutting down cpus with NMI [58110.176778] Kernel Offset: disabled [58110.180667] drm_kms_helper: panic occurred, switching back to text console [58110.188367] mga_delay choosing mdelay... [58110.242399] mga_delay choosing mdelay... [58110.266768] ------------[ cut here ]------------ [58110.271926] kernel BUG at mm/vmalloc.c:1335! [58110.276695] invalid opcode: 0000 [#1] SMP [58110.281289] Modules linked in: einj(E) nmioe(E) iscsi_ibft(E) iscsi_boot_sysfs(E) af_packet(E) btrfs(E) xor(E) raid6_pq(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm(E) joydev(E) iTCO_wdt(E) iTCO_vendor_support(E) crct10dif_pclmul(E) crc32_pclmul(E) crc32c_intel(E) ghash_clmulni_intel(E) aesni_intel(E) aes_x86_64(E) lrw(E) gf128mul(E) glue_helper(E) sb_edac(E) ablk_helper(E) lpc_ich(E) cryptd(E) pcspkr(E) edac_core(E) mfd_core(E) i2c_i801(E) shpchp(E) wmi(E) ipmi_si(E) ipmi_msghandler(E) acpi_pad(E) processor(E) button(E) dm_mod(E) ext4(E) crc16(E) mbcache(E) jbd2(E) hid_generic(E) usbhid(E) sr_mod(E) cdrom(E) sd_mod(E) mgag200(E) syscopyarea(E) sysfillrect(E) ahci(E) ehci_pci(E) sysimgblt(E) drm_kms_helper(E) ehci_hcd(E) ixgbe(E) igb(E) ttm(E) libahci(E) mdio(E) ptp(E) usbcore(E) pps_core(E) drm(E) libata(E) i2c_algo_bit(E) usb_common(E) dca(E) megaraid_sas(E) sg(E) scsi_mod(E) autofs4(E) [58110.371884] CPU: 75 PID: 0 Comm: swapper/75 Tainted: G E 4.1.0-rc8-7-default+ #10 [58110.381506] Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0059.R00.1501081238 01/08/2015 [58110.393063] task: ffff88046ea6d580 ti: ffff88046ea70000 task.ti: ffff88046ea70000 [58110.401422] RIP: 0010:[<ffffffff81189c65>] [<ffffffff81189c65>] __get_vm_area_node+0x155/0x160 [58110.411156] RSP: 0018:ffff88047f7284b8 EFLAGS: 00010006 [58110.417091] RAX: 0000000080010003 RBX: 0000000091000000 RCX: ffffc90000000000 [58110.425065] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 00000000000eb000 [58110.433038] RBP: ffff88047f7284f8 R08: ffffe8ffffffffff R09: 00000000ffffffff [58110.441010] R10: ffff880036a6a700 R11: ffff880460ab69c0 R12: 00000000910eb000 [58110.448983] R13: 0000000000000001 R14: 0000000091000000 R15: 00000000000eb000 [58110.456955] FS: 0000000000000000(0000) GS:ffff88047f720000(0000) knlGS:0000000000000000 [58110.465994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [58110.472413] CR2: 00007f5bdb9ac095 CR3: 0000000001a0b000 CR4: 00000000001407e0 [58110.480386] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [58110.488358] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [58110.496331] Stack: [58110.498577] ffff88047f728518 ffffc90000000000 00000000910eafff 0000000091000000 [58110.506885] 00000000910eb000 0000000000000001 0000000091000000 00000000000eb000 [58110.515192] ffff88047f728518 ffffffff8118ad20 00000000000000d0 ffffffffa0334f78 [58110.523499] Call Trace: [58110.526229] <#MC> [58110.528382] [<ffffffff8118ad20>] get_vm_area_caller+0x40/0x50 [58110.535111] [<ffffffffa0334f78>] ? ttm_mem_reg_ioremap+0xc8/0x110 [ttm] [58110.542607] [<ffffffff81050f18>] __ioremap_caller+0x188/0x390 [58110.549127] [<ffffffff812d59b9>] ? find_next_bit+0x19/0x20 [58110.555353] [<ffffffff81051177>] ioremap_wc+0x17/0x20 [58110.561099] [<ffffffffa0334f78>] ttm_mem_reg_ioremap+0xc8/0x110 [ttm] [58110.568398] [<ffffffffa0335351>] ttm_bo_move_memcpy+0xd1/0x700 [ttm] [58110.575598] [<ffffffff811a6c55>] ? __kmalloc+0x4b5/0x4c0 [58110.581632] [<ffffffffa01d9b48>] mgag200_bo_move+0x18/0x20 [mgag200] [58110.588830] [<ffffffffa0332ea0>] ttm_bo_handle_move_mem+0x260/0x590 [ttm] [58110.596514] [<ffffffffa03337d2>] ? ttm_bo_mem_space+0xd2/0x320 [ttm] [58110.603705] [<ffffffffa0333eb2>] ttm_bo_validate+0x1c2/0x1d0 [ttm] [58110.610711] [<ffffffff8113f681>] ? irq_work_queue+0x11/0x90 [58110.617037] [<ffffffffa01da3d3>] mgag200_bo_push_sysram+0x93/0xe0 [mgag200] [58110.624915] [<ffffffffa01d5a26>] mga_crtc_do_set_base.isra.8.constprop.21+0x76/0x410 [mgag200] [58110.634636] [<ffffffffa01d6e02>] mga_crtc_mode_set+0x1042/0x2140 [mgag200] [58110.642416] [<ffffffffa01d5492>] ? mga_crtc_prepare+0x132/0x370 [mgag200] [58110.650106] [<ffffffffa04ec8db>] drm_crtc_helper_set_mode+0x2fb/0x530 [drm_kms_helper] [58110.659052] [<ffffffffa04ed896>] drm_crtc_helper_set_config+0x856/0xa70 [drm_kms_helper] [58110.668217] [<ffffffffa00b5318>] drm_mode_set_config_internal+0x68/0x100 [drm] [58110.676388] [<ffffffffa04f78b2>] restore_fbdev_mode+0xc2/0xf0 [drm_kms_helper] [58110.684558] [<ffffffffa04f7aa3>] drm_fb_helper_force_kernel_mode+0x73/0xb0 [drm_kms_helper] [58110.693989] [<ffffffffa04f86f9>] drm_fb_helper_panic+0x29/0x30 [drm_kms_helper] [58110.702260] [<ffffffff81081bad>] notifier_call_chain+0x4d/0x80 [58110.708873] [<ffffffff81081c31>] atomic_notifier_call_chain+0x21/0x30 [58110.716169] [<ffffffff8156f954>] panic+0xee/0x1f5 [58110.721530] [<ffffffff8102d232>] mce_panic+0x1e2/0x200 [58110.727366] [<ffffffff8102d2c3>] mce_timed_out+0x73/0x80 [58110.733396] [<ffffffff8102e861>] do_machine_check+0x5f1/0xae0 [58110.739915] [<ffffffff8133f52f>] ? intel_idle+0xbf/0x130 [58110.745952] [<ffffffff8157c3c9>] machine_check+0x29/0x50 [58110.751984] [<ffffffff8133f52f>] ? intel_idle+0xbf/0x130 [58110.758017] <<EOE>> [58110.760362] [<ffffffff814467f0>] cpuidle_enter_state+0x70/0x1f0 [58110.767275] [<ffffffff814469a7>] cpuidle_enter+0x17/0x20 [58110.773309] [<ffffffff810a4e18>] cpu_startup_entry+0x308/0x390 [58110.779916] [<ffffffff8103a163>] start_secondary+0x143/0x170 [58110.786325] Code: 00 00 48 0f bd cf 83 c1 01 83 f9 0c 0f 4c c8 b0 1e 83 f9 1e 0f 4f c8 49 d3 e6 e9 f8 fe ff ff 48 89 df e8 9f a8 01 00 31 c0 eb b8 <0f> 0b 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 49 89 c8 41 [58110.808146] RIP [<ffffffff81189c65>] __get_vm_area_node+0x155/0x160 [58110.815257] RSP <ffff88047f7284b8> [58110.820218] ---[ end trace ab0c230901a0ee95 ]---Thanks
Rui
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Quoted 5 lines
Quoted 1 line
I'm here to report two panics which hang forever (the machine cannot reboot). It is because mgag200 doesn't work in panic context. It sleeps and allocates memory non-atomically.
This is the same for all drm drivers, the drm atomic handling with fbcon/fbdev is totally broken. It would be serious work to fix this properly.
It's a serious problem when a server crashes ... even worse when it hangs while doing so because we have to rely on some other agent to notice the hung server and go poke it with a stick.
If it is too hard to fix all of the drivers, is it possible to attack this in the allocator?
-Tony ���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m����������?�I�
Quoted 11 lines · Tony
Quoted 5 lines
Quoted 1 line
I'm here to report two panics which hang forever (the machine cannot reboot). It is because mgag200 doesn't work in panic context. It sleeps and allocates memory non-atomically.
This is the same for all drm drivers, the drm atomic handling with fbcon/fbdev is totally broken. It would be serious work to fix this properly.
It's a serious problem when a server crashes ... even worse when it hangs while doing so because we have to rely on some other agent to notice the hung server and go poke it with a stick.
If it is too hard to fix all of the drivers, is it possible to attack this in the allocator?
Hm, what do you mean by fixing this in the allocator? I've made some rough sketch of the problem space in
http://www.x.org/wiki/DRMJanitors/ under "Make panic handling work". Problem is that the folks which know what to do (drm hackers) have zero incentive to fix it (since if you blow up a drm driver any kind of fbcon panic handling is hopeless anyway).
The other problem is is that this is a serious effort with tons of little things all over to consider. My gut estimate is that probably it'll take something of the order of a man year to fix this for real. David Herrmann has supplied parts of the required puzzle to actually be able to somewhat reliably show panics on drm modesetting drivers, but that didn't contain any of the work to make fbdev not totally suck at panic handling first. And I guess for general distros and servers that's needed - developers simply disable all of fbdev to be able to debug kms hangs.
-Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Quoted 6 lines · Daniel Vetter
Hm, what do you mean by fixing this in the allocator? I've made some rough sketch of the problem space in
http://www.x.org/wiki/DRMJanitors/ under "Make panic handling work". Problem is that the folks which know what to do (drm hackers) have zero incentive to fix it (since if you blow up a drm driver any kind of fbcon panic handling is hopeless anyway).
Ok, silly question: switching the GPU output to the simplest, supported-by-all-GPUs mode before panicking is not that easy too, right?
Or does that mean, one needs to reinit GPU in order to even show something...
Thanks.
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Quoted 10 lines · Borislav Petkov
Quoted 6 lines · Daniel Vetter
Hm, what do you mean by fixing this in the allocator? I've made some rough sketch of the problem space in
http://www.x.org/wiki/DRMJanitors/ under "Make panic handling work". Problem is that the folks which know what to do (drm hackers) have zero incentive to fix it (since if you blow up a drm driver any kind of fbcon panic handling is hopeless anyway).Ok, silly question: switching the GPU output to the simplest, supported-by-all-GPUs mode before panicking is not that easy too, right?
Anything that resembles doing a modeset (and the precise mode is totally irrelevant) will be almost impossible: There's clocks to tune, various links to train, a bunch of sidebind communication and all that takes time, needs to happen in a very precise order and also needs lots of mutexes (crossing subsystems sometimes even) to avoid trampling on maybe the thread that just died and caused the panic.
In short you need specially-written codepaths for panic, which means only the most minimal thing has a chance to even work. fbdev/fbcon and the current drm panic handler are anything but that. Doing a full modeset isn't it either. Heck even the just updating the displayed buffer might need massive reconfiguration of the memory fetch watermarks if e.g. the screen is display high bpc yuv and you want to display the rgb fbcon buffer. And if you'd just try to take what's currently being displayed there's the annoying problem that often providing a cpu-contiguous view of that needs a vmalloc area, if not first moving the buffer around a bit in vram with the copy engine. That all involves lots of really complex code.
Quoted 2 lines
Or does that mean, one needs to reinit GPU in order to even show something...
If the gpu is runtime suspended, then yes you'd get to even init the entire thing first before you can display anything.
fbdev panic handling was designed for a world where you'd bash a few values into a few registers and never wait for the CRTC to reach stable timings and output a useful signal. Which could all happen very much after the kernel made it's dying sigh. Display hw has long stopped being this simple and display drivers also. The only vestige is that we still call the display pipeline object in drm CRTC for cathode ray tube controller ...
-Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
On Sat, Jun 27, 2015 at 07:56:19PM +0200, Daniel Vetter wrote:
<snip very useful explanation for !GPU people>
Quoted 3 lines
Which could all happen very much after the kernel made it's dying sigh. Display hw has long stopped being this simple and display drivers also.
Thanks for the explanation. I was fearing that it would go in such direction though. :\ Oh well.
So there are two observations to be ventured here:
* there's still no robust and simple way to save/show oops messages on laptops. When a laptop crashes and the user has only this one machine, catching the oops can be a serious PITA. Laptop most likely doesn't have serial, modeset is hard to do, as you explained, and saving it into persistent storage (UEFI) might brick it in some cases or whatever the firmware will do to it. I guess we can keep wishing here for something better.
* The server aspect might not really need GPU modesetting IMHO because realistically, servers have serial so the oops can go out there. Which begs the other question:
Shouldn't drm_fb_helper_panic() and that path below it simply be removed?
I mean, if it needs to do all that hw-reconf on the panic path and if that reconfiguration is not completely reliable, maybe we shouldn't do it all?
This way, we will basically remain as quiet as possible in order not to muddle the system unreasonably when we're about to die and the only thing we want to catch is an oops or maybe kexec the crash kernel...
Oops will go out on serial anyway and server will kexec - GPU/fb stuff doesn't do anything. Yes, no?
Hmmm.
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Quoted 13 lines · Borislav Petkov
On Sat, Jun 27, 2015 at 07:56:19PM +0200, Daniel Vetter wrote:
<snip very useful explanation for !GPU people>
Quoted 3 lines
Which could all happen very much after the kernel made it's dying sigh. Display hw has long stopped being this simple and display drivers also.
Thanks for the explanation. I was fearing that it would go in such direction though. :\ Oh well.
So there are two observations to be ventured here:
Quoted 7 lines
* there's still no robust and simple way to save/show oops messages on laptops. When a laptop crashes and the user has only this one machine, catching the oops can be a serious PITA. Laptop most likely doesn't have serial, modeset is hard to do, as you explained, and saving it into persistent storage (UEFI) might brick it in some cases or whatever the firmware will do to it. I guess we can keep wishing here for something better.
As long as the display is up and running we should have a fair stab at showing the oops - it's just that no one has seriously bothered with the necessary infastructure, automated testing (it won't work otherwise) and driver work.
Quoted 11 lines
* The server aspect might not really need GPU modesetting IMHO because realistically, servers have serial so the oops can go out there. Which begs the other question:
Shouldn't
drm_fb_helper_panic()and that path below it simply be removed?I mean, if it needs to do all that hw-reconf on the panic path and if that reconfiguration is not completely reliable, maybe we shouldn't do it all?
Quoted 6 lines
This way, we will basically remain as quiet as possible in order not to muddle the system unreasonably when we're about to die and the only thing we want to catch is an oops or maybe kexec the crash kernel...
Oops will go out on serial anyway and server will kexec - GPU/fb stuff doesn't do anything. Yes, no?
drm_fb_helper_panic isn't the only panic handler - fbdev/fbcon have their own. They interfere, and fbdev blissfully assumes that it can call almost any driver hook from hardirq context. Which means you'd also need to consolidate the various hand-rolled (in drivers instead of the fbdev helper) bits to offload fbdev callbacks to workers and make sure we don't go boom in panics in there either.
It's really a big mess unfortunately :(
-Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Quoted 2 lines · Daniel Vetter
As long as the display is up and running we should have a fair stab at showing the oops
Yeah, that has the same problem as all the other methods for showing oops - *if* it is still healthly. Like, for example, trying to catch an oops in networking with netconsole :)
Quoted 3 lines
- it's just that no one has seriously bothered with the necessary infastructure, automated testing (it won't work otherwise) and driver work.
Right.
Quoted 6 lines
drm_fb_helper_panicisn't the only panic handler - fbdev/fbcon have their own. They interfere, and fbdev blissfully assumes that it can call almost any driver hook from hardirq context. Which means you'd also need to consolidate the various hand-rolled (in drivers instead of the fbdev helper) bits to offload fbdev callbacks to workers and make sure we don't go boom in panics in there either.
Hmmm, this would mean to go and tell them through the panic notifier to not do anything "risky" anymore. Perhaps even stop them from doing regular tasks which could become risky in the case of panic. I guess this would mean, we should be doing the same in other subsystems too.
Hmm.
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Quoted 11 lines · Borislav Petkov
Quoted 6 lines
drm_fb_helper_panicisn't the only panic handler - fbdev/fbcon have their own. They interfere, and fbdev blissfully assumes that it can call almost any driver hook from hardirq context. Which means you'd also need to consolidate the various hand-rolled (in drivers instead of the fbdev helper) bits to offload fbdev callbacks to workers and make sure we don't go boom in panics in there either.Hmmm, this would mean to go and tell them through the panic notifier to not do anything "risky" anymore. Perhaps even stop them from doing regular tasks which could become risky in the case of panic. I guess this would mean, we should be doing the same in other subsystems too.
Not sure that applies to other subsystems, it's really just history. fbdev was designed for a world where register banging and some writes to a gart was all you had to do for graphics, and you can very well do that from hardirq context. And fbdev will take care of some simple locking itself if needed. The problem really is that drm drivers are anything but simple, and there's a bit of room to share more code and tricks (and fix up even more corner-cases) between drm drivers in the shared fbdev emulation library.
-Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch