Identify faulting CPU in Windows crash dumps
v2 cover Identify faulting CPU in Windows crash dumps
When a Windows guest crashes and generates a memory dump, the dump file does not indicate which CPU triggered the panic. This series adds support for recording the crash_occurred flag in the QEMUCPUState structure and using it in elf2dmp to identify the faulting CPU.
Thanks to Akihiko Odaki for reviewing v1 and pointing out these issues.
Changes since v1:
- Revert QEMUCPUSTATE_VERSION bump (tools check size, not version)
- Fix -Werror=unused-variable on i386 by moving crash_occurred check outside #ifdef
- Add size-based field detection in elf2dmp for forward compatibility
- Set header context before best-effort guest memory operations
- Use LIVE_SYSTEM_DUMP when BugcheckCode is zero
- Use PRIx32 for BugcheckCode format specifier
- Use linux.alibaba.com identity consistently
Zhengrong Li (3):
runstate: set crash_occurred on guest crashloaded
dump: add crash_occurred flag to QEMUCPUState
elf2dmp: fill ContextBuffer with faulting CPU context
contrib/elf2dmp/kdbg.h | 9 +++++++ contrib/elf2dmp/main.c | 50 ++++++++++++++++++++++++++++++++++++-- contrib/elf2dmp/qemu_elf.c | 20 +++++++++------ contrib/elf2dmp/qemu_elf.h | 3 +++ system/runstate.c | 5 ++++ target/i386/arch_dump.c | 6 +++++ 6 files changed, 84 insertions(+), 9 deletions(-)
Signature · 1 line
2.43.0
PATCH v2 1/3 runstate: set crash_occurred on guest crashloaded
Windows pvpanic driver writes PVPANIC_CRASH_LOADED (bit 1) on bugcheck, not PVPANIC_PANICKED (bit 0). As a result, qemu_system_guest_panicked() is never called and crash_occurred stays false for Windows guests.
Set crash_occurred in qemu_system_guest_crashloaded() as well, so the faulting CPU can be identified in the guest dump.
--- system/runstate.c | 5 +++++ 1 file changed, 5 insertions(+)
system/runstate.c +5 -0
@@ -724,6 +724,11 @@ void qemu_system_guest_panicked(GuestPanicInformation *info)void qemu_system_guest_crashloaded (GuestPanicInformation *info) {qemu_log_mask (LOG_GUEST_ERROR ,"Guest crash loaded" ); + +if (current_cpu) { + current_cpu->crash_occurred = true; + } +qapi_event_send_guest_crashloaded (GUEST_PANIC_ACTION_RUN , info);qapi_free_GuestPanicInformation (info); }
Signature · 1 line
2.43.0
Re: [PATCH v2 1/3] runstate: set crash_occurred on guest crashloaded
Hi,
Quoted 18 lines · Zhengrong Li
Windows pvpanic driver writes
PVPANIC_CRASH_LOADED(bit 1) on bugcheck, notPVPANIC_PANICKED(bit 0). As a result,qemu_system_guest_panicked()is never called andcrash_occurredstays false for Windows guests.Set
crash_occurredinqemu_system_guest_crashloaded()as well, so the faulting CPU can be identified in the guest dump.Signed-off-by Zhengrong Li <[email protected]>--- system/runstate.c | 5 +++++ 1 file changed, 5 insertions(+)system/runstate.c +0
-0
system/runstate.c void qemu_system_guest_panicked(GuestPanicInformation *info) · line 729
Quoted 6 lines
system/runstate.c +3
-0@@ -724,6 +724,11 @@ void qemu_system_guest_panicked(GuestPanicInformation *info)void qemu_system_guest_crashloaded (GuestPanicInformation *info) {qemu_log_mask (LOG_GUEST_ERROR ,"Guest crash loaded" ); + +if (current_cpu) { + current_cpu->crash_occurred = true;
This is generic code, I'm surprised we need to update the vCPU thread here. Could it be the reponsibility of the caller? As you described, pvpanic so add this change in handle_event()? I'm also surprised qemu_system_guest_panicked() does that -- the change comes from commit bac05aa9a77 ("cpu: Add crash_occurred flag into CPUState") which doesn't explain much. Unfortunately the pvpanic code doesn't have any maintainer listed...
Cc'ing some developers active in this code...
$ git grep @ $(git ls-files|fgrep pvpanic) hw/misc/pvpanic-pci.c:7: * Mihai Carabas <[email protected]> hw/misc/pvpanic.c:7: * Wen Congyang <[email protected]> hw/misc/pvpanic.c:8: * Hu Tao <[email protected]>
Quoted 5 lines
system/runstate.c +2
-0+ } +qapi_event_send_guest_crashloaded (GUEST_PANIC_ACTION_RUN , info);qapi_free_GuestPanicInformation (info); }
Re: [PATCH v2 1/3] runstate: set crash_occurred on guest crashloaded
Hi Philippe,
Thanks for the review.
I followed the existing pattern in qemu_system_guest_panicked(), which already sets crash_occurred on current_cpu when a guest panic is reported. The intent was to keep the panicked and crash-loaded paths consistent.
Putting it in the generic function also means any future crash notification mechanism (beyond pvpanic) would automatically benefit from identifying the faulting CPU in guest dumps.
I agree the original commit bac05aa9a77 ("cpu: Add crash_occurred flag into CPUState") doesn't explain the design choice well.
If you prefer moving it to handle_event() in the caller, I'm happy to adjust. What do you think?
Regards,
Zhengrong Li
Re: [PATCH v2 1/3] runstate: set crash_occurred on guest crashloaded
Quoted 13 lines · Zhengrong Li
Hi Philippe,
Thanks for the review.
I followed the existing pattern in
qemu_system_guest_panicked(), which already setscrash_occurredoncurrent_cpuwhen a guest panic is reported. The intent was to keep the panicked and crash-loaded paths consistent.Putting it in the generic function also means any future crash notification mechanism (beyond pvpanic) would automatically benefit from identifying the faulting CPU in guest dumps.
Quoted 5 lines
I agree the original commit
bac05aa9a77("cpu: Addcrash_occurredflag into CPUState") doesn't explain the design choice well.If you prefer moving it to
handle_event()in the caller, I'm happy to adjust. What do you think?
I find your reasoning makes sense. It is a common requirement of qemu_system_guest_panicked() callers to update current_cpu, and having the logic to update the CPU in the same function will de-duplicate that logic. While there is only one caller of qemu_system_guest_crashloaded(), consistency and future prospect justifies to do the same.
On the other hand, it is also true that a function that is prefixed qemu_system_* implicitly updating current_cpu is obscure. I think the intent would be clearer if the functions explicitly take the CPU via a parameter. In fact, most callers of qemu_system_guest_panicked() explicitly take the CPU via a parameter. e.g., kvm_cpu_exec(). qemu_system_guest_panicked() breaks the propagation of the CPU value and potentially attributes the panic to a wrong CPU.
We are adding a similar function and its call, so it is a good timing to perform such refactoring. I still don't require you to do that since updating the signature of qemu_system_guest_panicked() requires a simple yet codebase-wide edit that is not strictly necessary for the feature we are adding.
Regards,
Akihiko Odaki
Re: [PATCH v2 1/3] runstate: set crash_occurred on guest crashloaded
Hi Akihiko,
Thanks for the detailed analysis. I agree that explicitly passing the CPU as a parameter would be cleaner than relying on current_cpu implicitly. That sounds like a good standalone refactoring to pursue separately.
For this series, I'll keep the current approach.
Regards,
Zhengrong Li
PATCH v2 2/3 dump: add crash_occurred flag to QEMUCPUState
In reply to Zhengrong LiAdd is_crash_occurred_cpu field to QEMUCPUState so that dump-guest-memory can record which CPU triggered the guest panic.
The new field is appended after kernel_gs_base, so existing tools that check the 'size' field can safely ignore it when reading older dumps.
Changes since v1:
- Revert QEMUCPUSTATE_VERSION bump (not needed, tools check size)
- Move crash_occurred check outside #ifdef TARGET_X86_64 to fix -Werror=unused-variable on i386 builds (reported by Akihiko Odaki)
- Use linux.alibaba.com identity
--- contrib/elf2dmp/qemu_elf.h | 2 ++ target/i386/arch_dump.c | 6 ++++++ 2 files changed, 8 insertions(+)
contrib/elf2dmp/qemu_elf.h +2 -0
@@ -27,6 +27,8 @@ typedef struct QEMUCPUState {QEMUCPUSegment ldt, tr, gdt, idt;uint64_t cr[5 ];uint64_t kernel_gs_base; +uint8_t is_crash_occurred_cpu; +uint8_t pad[7 ]; } QEMUCPUState;int is_system (QEMUCPUState *s);
target/i386/arch_dump.c +6 -0
@@ -264,6 +264,8 @@ struct QEMUCPUState {* by checking 'size' field. */ uint64_t kernel_gs_base; +uint8_t is_crash_occurred_cpu; +uint8_t pad[7 ]; };typedef struct QEMUCPUState QEMUCPUState ;
@@ -279,6 +281,7 @@ static void copy_segment(QEMUCPUSegment *d, SegmentCache *s)static void qemu_get_cpustate (QEMUCPUState *s,CPUX86State *env) { +CPUState *cs =env_cpu (env);memset (s,0 ,sizeof (QEMUCPUState)); s->version =QEMUCPUSTATE_VERSION ;
@@ -325,6 +328,9 @@ static void qemu_get_cpustate(QEMUCPUState *s, CPUX86State *env)#ifdef TARGET_X86_64 s->kernel_gs_base = env->kernelgsbase;#endif +if (cs->crash_occurred) { + s->is_crash_occurred_cpu =1 ; + } } static inline int cpu_write_qemu_note(WriteCoreDumpFunction f,
Signature · 1 line
2.43.0
Re: [PATCH v2 2/3] dump: add crash_occurred flag to QEMUCPUState
Quoted 25 lines · Zhengrong Li
Add
is_crash_occurred_cpufield to QEMUCPUState so that dump-guest-memory can record which CPU triggered the guest panic.The new field is appended after
kernel_gs_base, so existing tools that check the 'size' field can safely ignore it when reading older dumps.Changes since v1:
- RevertQEMUCPUSTATE_VERSIONbump (not needed, tools check size)
- Movecrash_occurredcheck outside #ifdefTARGET_X86_64to fix -Werror=unused-variable on i386 builds (reported by Akihiko Odaki)
- Use linux.alibaba.com identitySigned-off-by Zhengrong Li <[email protected]>--- contrib/elf2dmp/qemu_elf.h | 2 ++ target/i386/arch_dump.c | 6 ++++++ 2 files changed, 8 insertions(+)contrib/elf2dmp/qemu_elf.h +0
-0
contrib/elf2dmp/qemu_elf.h typedef struct QEMUCPUState { · line 31
Quoted 5 lines
contrib/elf2dmp/qemu_elf.h +2
-0@@ -27,6 +27,8 @@ typedef struct QEMUCPUState {QEMUCPUSegment ldt, tr, gdt, idt;uint64_t cr[5 ];uint64_t kernel_gs_base; +uint8_t is_crash_occurred_cpu; +uint8_t pad[7 ];
Growing QEMUCPUState from 440 to 448 bytes confuses init_states() in contrib/elf2dmp/qemu_elf.c and makes valid version-1 notes appear to lack kernel_gs_base, despite that field occupying bytes 432–439.
Patch 3 fixes homogeneous old dumps, but that field-specific size detection belongs in this patch for bisectability and to support its compatibility claim. Please see docs/devel/submitting-a-patch.rst for the bisect requirement.
Regards,
Akihiko Odaki
Quoted 34 lines
contrib/elf2dmp/qemu_elf.h +0
-0} QEMUCPUState;int is_system (QEMUCPUState *s);target/i386/arch_dump.c +6
-0@@ -264,6 +264,8 @@ struct QEMUCPUState {* by checking 'size' field. */ uint64_t kernel_gs_base; +uint8_t is_crash_occurred_cpu; +uint8_t pad[7 ]; };typedef struct QEMUCPUState QEMUCPUState ;@@ -279,6 +281,7 @@ static void copy_segment(QEMUCPUSegment *d, SegmentCache *s)static void qemu_get_cpustate (QEMUCPUState *s,CPUX86State *env) { +CPUState *cs =env_cpu (env);memset (s,0 ,sizeof (QEMUCPUState)); s->version =QEMUCPUSTATE_VERSION ;@@ -325,6 +328,9 @@ static void qemu_get_cpustate(QEMUCPUState *s, CPUX86State *env)#ifdef TARGET_X86_64 s->kernel_gs_base = env->kernelgsbase;#endif +if (cs->crash_occurred) { + s->is_crash_occurred_cpu =1 ; + } } static inline int cpu_write_qemu_note(WriteCoreDumpFunction f,
PATCH v2 3/3 elf2dmp: fill ContextBuffer with faulting CPU context
In reply to Zhengrong LiWhen is_crash_occurred_cpu is set in QEMUCPUState, copy that CPU's context into WinDumpHeader64 ContextBuffer so WinDbg opens on the faulting CPU instead of CPU 0.
Also extract KiBugCheckData from the guest kernel to populate BugcheckCode and BugcheckParameters in the dump header.
Changes since v1:
- Add size-based field detection in init_states() for forward compatibility
- Use has_is_crash_occurred_cpu flag instead of version check
- Set header context before best-effort guest memory operations to avoid losing faulting CPU context on partial dump failures
- Use LIVE_SYSTEM_DUMP when BugcheckCode is zero
- Use PRIx32 for BugcheckCode format specifier
- Use linux.alibaba.com identity
--- contrib/elf2dmp/kdbg.h | 9 +++++++ contrib/elf2dmp/main.c | 50 ++++++++++++++++++++++++++++++++++++-- contrib/elf2dmp/qemu_elf.c | 20 +++++++++------ contrib/elf2dmp/qemu_elf.h | 1 + 4 files changed, 71 insertions(+), 9 deletions(-)
contrib/elf2dmp/kdbg.h +9 -0
@@ -195,4 +195,13 @@ typedef struct KDDEBUGGER_DATA64 { uint16_t OffsetPrcbContext; }KDDEBUGGER_DATA64 ; +typedef struct KIBUGCHECK_INFO { +uint32_t BugcheckCode; +uint32_t unused0; +uint64_t BugcheckParameter1; +uint64_t BugcheckParameter2; +uint64_t BugcheckParameter3; +uint64_t BugcheckParameter4; +}KIBUGCHECK_INFO ; +#endif /* KDBG_H */
contrib/elf2dmp/main.c +48 -2
72 lines not on this page · raw message
contrib/elf2dmp/qemu_elf.c +13 -7
@@ -71,7 +71,8 @@ static bool init_states(QEMU_Elf *qe)return false; } - qe->has_kernel_gs_base =1 ; + qe->has_kernel_gs_base =0 ; + qe->has_is_crash_occurred_cpu =0 ; offset = phdr[0 ].p_offset; states =g_ptr_array_new ();
@@ -103,14 +104,19 @@ static bool init_states(QEMU_Elf *qe) nhdr->n_descsz >= offsetof(QEMUCPUState, kernel_gs_base)) { state_size =MIN (state->size, nhdr->n_descsz); -if (state_size <sizeof (*state)) { +if (state_size >= offsetof(QEMUCPUState , kernel_gs_base) + +sizeof (state->kernel_gs_base)) { + qe->has_kernel_gs_base =1 ; + } + +if (state_size >= offsetof(QEMUCPUState , is_crash_occurred_cpu) + +sizeof (state->is_crash_occurred_cpu)) { + qe->has_is_crash_occurred_cpu =1 ; + } + +if (state_size !=sizeof (*state)) {eprintf ("CPU #%u: QEMU CPU state size %u doesn't match\n" , states->len, state_size); -/* -* We assume either every QEMU CPU state has KERNEL_GS_BASE or -* no one has. -*/ - qe->has_kernel_gs_base =0 ; }g_ptr_array_add (states, state); }
contrib/elf2dmp/qemu_elf.h +1 -0
@@ -42,6 +42,7 @@ typedef struct QEMU_Elf {QEMUCPUState **state;size_t state_nr;int has_kernel_gs_base; +int has_is_crash_occurred_cpu; } QEMU_Elf;bool QEMU_Elf_init (QEMU_Elf *qe,const char *filename);
Signature · 1 line
2.43.0
Re: [PATCH v2 3/3] elf2dmp: fill ContextBuffer with faulting CPU context
Quoted 55 lines · Zhengrong Li
When
is_crash_occurred_cpuis set in QEMUCPUState, copy that CPU's context into WinDumpHeader64 ContextBuffer so WinDbg opens on the faulting CPU instead of CPU 0.Also extract KiBugCheckData from the guest kernel to populate BugcheckCode and BugcheckParameters in the dump header.
Changes since v1:
- Add size-based field detection ininit_states()for forward compatibility
- Usehas_is_crash_occurred_cpuflag instead of version check
- Set header context before best-effort guest memory operations to avoid losing faulting CPU context on partial dump failures
- UseLIVE_SYSTEM_DUMPwhen BugcheckCode is zero
- Use PRIx32 for BugcheckCode format specifier
- Use linux.alibaba.com identitySigned-off-by Zhengrong Li <[email protected]>--- contrib/elf2dmp/kdbg.h | 9 +++++++ contrib/elf2dmp/main.c | 50 ++++++++++++++++++++++++++++++++++++-- contrib/elf2dmp/qemu_elf.c | 20 +++++++++------ contrib/elf2dmp/qemu_elf.h | 1 + 4 files changed, 71 insertions(+), 9 deletions(-)contrib/elf2dmp/kdbg.h +9
-0@@ -195,4 +195,13 @@ typedef struct KDDEBUGGER_DATA64 { uint16_t OffsetPrcbContext; }KDDEBUGGER_DATA64 ; +typedef struct KIBUGCHECK_INFO { +uint32_t BugcheckCode; +uint32_t unused0; +uint64_t BugcheckParameter1; +uint64_t BugcheckParameter2; +uint64_t BugcheckParameter3; +uint64_t BugcheckParameter4; +}KIBUGCHECK_INFO ; +#endif /* KDBG_H */ contrib/elf2dmp/main.c +1
-1@@ -338,11 +338,26 @@ static bool fill_header(WinDumpHeader64 *hdr, struct pa_space *ps,* A dump may still contain valuable information even if it lacks contexts of * some CPUs due to dump corruption or a failure before starting CPUs. */ -static void fill_context (KDDEBUGGER_DATA64 *kdbg, +static void fill_context (WinDumpHeader64 *hdr,KDDEBUGGER_DATA64 *kdbg,struct va_space *vs,QEMU_Elf *qe) {int i;
contrib/elf2dmp/main.c static bool fill_header(WinDumpHeader64 *hdr, struct pa_space *ps, · line 354
Quoted 10 lines
contrib/elf2dmp/main.c +9
-0+int fault_cpu =-1 ; +/* First pass: identify faulting CPU and set header context early */ +for (i =0 ; i < qe->state_nr; i++) { +QEMUCPUState *s = qe->state[i]; +if (qe->has_is_crash_occurred_cpu && s->is_crash_occurred_cpu) { +WinContext64 ctx; +win_context_init_from_qemu_cpu_state (&ctx, s); +memcpy (hdr->ContextBuffer, &ctx,sizeof (ctx)); + fault_cpu = i;
fault_cpu is assigned but never read. Consequently:
error: variable ‘fault_cpu’ set but not used
[-Werror=unused-but-set-variable]
make -j8 contrib/elf2dmp/elf2dmp fails.
Quoted 84 lines
contrib/elf2dmp/main.c +38
-1+printf ("Faulting CPU identified: #%d\n" , i); +break ; + } + } + +/* Second pass: fill context for all CPUs (best-effort) */ for (i =0 ; i < qe->state_nr; i++) {uint64_t Prcb;uint64_t Context;@@ -512,6 +527,7 @@ int main(int argc, char *argv[])uint64_t KdVersionBlock;bool kernel_found = false;OMFSignatureRSDS rsds; +uint64_t KiBugCheckData;if (argc !=3 ) {eprintf ("usage:\n\t%s elf_file dmp_file\n" , argv[0 ]);@@ -611,7 +627,37 @@ int main(int argc, char *argv[]) goto out_kdbg; } -fill_context (kdbg, &vs, &qemu_elf); +if (!SYM_RESOLVE (KernBase, &pdb, KiBugCheckData)) { +eprintf ("Failed to get KiBugCheckData.\n" ); + }else { +KIBUGCHECK_INFO data = {0 }; +if (va_space_rw (&vs, KiBugCheckData, &data,sizeof (data),0 )) { + header.BugcheckCode = data.BugcheckCode; + header.BugcheckParameter1 = data.BugcheckParameter1; + header.BugcheckParameter2 = data.BugcheckParameter2; + header.BugcheckParameter3 = data.BugcheckParameter3; + header.BugcheckParameter4 = data.BugcheckParameter4; + +/* +* If BugcheckCode wasn't saved, we consider guest OS as alive. +*/ +if (!header.BugcheckCode) { + header.BugcheckCode =LIVE_SYSTEM_DUMP ; + } + +printf ("KiBugCheckData: 0x%016" PRIx64 +", BugcheckCode: 0x%08" PRIx32", Args:" +" 0x%016" PRIx64" 0x%016" PRIx64 +" 0x%016" PRIx64" 0x%016" PRIx64"\n" , + KiBugCheckData, header.BugcheckCode, + data.BugcheckParameter1, data.BugcheckParameter2, + data.BugcheckParameter3, data.BugcheckParameter4); + }else { +eprintf ("Failed to va_space_rw KiBugCheckData.\n" ); + } + } + +fill_context (&header, kdbg, &vs, &qemu_elf);if (!write_dump (&ps, &header, argv[2 ])) {eprintf ("Failed to save dump\n" );contrib/elf2dmp/qemu_elf.c +11
-2@@ -71,7 +71,8 @@ static bool init_states(QEMU_Elf *qe)return false; } - qe->has_kernel_gs_base =1 ; + qe->has_kernel_gs_base =0 ; + qe->has_is_crash_occurred_cpu =0 ; offset = phdr[0 ].p_offset; states =g_ptr_array_new ();@@ -103,14 +104,19 @@ static bool init_states(QEMU_Elf *qe) nhdr->n_descsz >= offsetof(QEMUCPUState, kernel_gs_base)) { state_size =MIN (state->size, nhdr->n_descsz); -if (state_size <sizeof (*state)) { +if (state_size >= offsetof(QEMUCPUState , kernel_gs_base) + +sizeof (state->kernel_gs_base)) { + qe->has_kernel_gs_base =1 ; + } + +if (state_size >= offsetof(QEMUCPUState , is_crash_occurred_cpu) + +sizeof (state->is_crash_occurred_cpu)) { + qe->has_is_crash_occurred_cpu =1 ; + }
contrib/elf2dmp/qemu_elf.c static bool init_states(QEMU_Elf *qe) · line 120
Quoted 10 lines
contrib/elf2dmp/qemu_elf.c +2
-5+ +if (state_size !=sizeof (*state)) {eprintf ("CPU #%u: QEMU CPU state size %u doesn't match\n" , states->len, state_size); -/* -* We assume either every QEMU CPU state has KERNEL_GS_BASE or -* no one has. -*/ - qe->has_kernel_gs_base =0 ; }
Field availability is ORed across all CPU notes and then applied to every note. With a 440-byte CPU 0 note followed by a 448-byte CPU 1 note, CPU 1 sets has_is_crash_occurred_cpu, after which fill_context() reads byte 440 beyond CPU 0’s descriptor. That byte normally belongs to the next note header and is nonzero, so CPU 0 can be falsely selected. The same aggregation can make fix_dtb() read a missing kernel_gs_base. Please follow what's previously done for kernel_gs_base.
Regards,
Akihiko Odaki
Quoted 14 lines
contrib/elf2dmp/qemu_elf.c +0
-0g_ptr_array_add (states, state); }contrib/elf2dmp/qemu_elf.h +1
-0@@ -42,6 +42,7 @@ typedef struct QEMU_Elf {QEMUCPUState **state;size_t state_nr;int has_kernel_gs_base; +int has_is_crash_occurred_cpu; } QEMU_Elf;bool QEMU_Elf_init (QEMU_Elf *qe,const char *filename);