purgatory: Say last words in kexec on panic case
Subsystems (2)
v2 cover purgatory: Say last words in kexec on panic case
This is the version 2 of RFC patch series previously named "purgatory: Add basic support for IPMI command execution."
Changes since RFC v1:
- Add --ipmi-kcs-ports option to specify I/O ports for KCS I/F
- Add --ipmi-handle-panic option to generate an event on BMC to inform and record about the panic
- Add --output-cpu-ip option to output IP registers to console and BMC's SEL
...and various cleanups and improvements
I performed some Web searches, and I found that major BMCs still support KCS I/F. Although this patch series supports only KCS I/F, it should cover many recent servers with BMC. However, I/O ports used for KCS I/F are varies among servers, so I added an option to specify port numbers for KCS.
About error handling for KCS protocol, I re-checked IPMI specification, and I found that simply retrying on error is sufficient. So I didn't change the logic for the simplicity. Please see PATCH 3/7 for details.
Background and purpose
======================
If the second kernel for crash dumping hangs up while booting, no information related to the first kernel will be saved. This makes crash cause analysis difficult. Some enterprise users don't permit the same faults happen multiple times, so we want to save minimal information before booting the second kernel.
Approaches
==========
One of the approaches is (1) to use panic notifier call or kmsg dump feature. For example, a panic notifier callback registered by IPMI driver can save the panic message to BMC's SEL before booting the second kernel. Similarly, kmsg dump saves kernel logs to a non- volatile memory on the server. This approach covers multiple hardware/firmware implementation. However, doing many things in crashed kernel will reduce the reliability. Additionally, a part of the code is also used for normal operation and still evolving. This would makes it difficult to keep stable.
Another approach is (2) to save minimal information to BMC's SEL in purgatory. It is difficult to do complicate things in purgatory, but fortunately IPMI specification defines a simple interface, KCS (Keyboard Controller Style) I/F. KCS is controlled by two I/O ports and supported by most of major BMCs.
Here, we want more reliable one for the purpose, we adopt (2).
What are provided?
==================
This patch series provides multiple RAS features other than the main purpose described above.
- Timeout mechanism relying on polling RTC
- API to access BMC via KCS I/F
- Command line option to start/stop BMC's watchdog timer in purgatory
- Command line option to write the value of RIP registers to SEL and/or serial console (useful for kernel hang-up cases)
- Command line option to generate a plantform event on BMC (useful for server monitoring or HA clustering; you can make the BMC issue an SNMP trap)
- Command line option to change the default I/O ports of KCS I/F
Limitations of RFC version
==========================
This patch serires is incomplete, and there are some limitations.
- Related codes are unconditionally built into the kexec binary
- Implemented only for x86_64 (it may break the build for other architectures)
- Timeout value for IPMI operations is hard-coded
Future plan
===========
- Add an option to save the panic message to BMC's SEL (this requires some kernel modifications)
---
Hidehiro Kawai (7):
purgatory: Introduce timeout API
purgatory/x86: Support CMOS RTC
purgatory/ipmi: Support IPMI KCS interface
purgatory/ipmi: Support BMC watchdog timer start/stop in purgatory purgatory/ipmi: Add an option for purgatory to handle panic event with IPMI purgatory/x86: Add an option to output IP registers to console in panic case purgatory/ipmi/x86: Support logging of IP registers into SEL
kexec/arch/i386/crashdump-x86.c | 10 - kexec/arch/i386/include/arch/options.h | 4 kexec/arch/i386/kexec-x86.h | 1 kexec/arch/x86_64/kexec-x86_64.c | 15 + kexec/ipmi.h | 9 + kexec/kexec.c | 51 ++++ kexec/kexec.h | 11 + purgatory/Makefile | 6 purgatory/arch/i386/Makefile | 1 purgatory/arch/i386/purgatory-x86.h | 2 purgatory/arch/i386/rtc_cmos.c | 107 ++++++++ purgatory/arch/x86_64/Makefile | 2 purgatory/arch/x86_64/purgatory-elf-x86_64.c | 82 ++++++ purgatory/arch/x86_64/purgatory-x86_64.c | 35 ++- purgatory/include/purgatory-elf.h | 13 + purgatory/include/purgatory.h | 9 + purgatory/include/time.h | 33 ++ purgatory/ipmi.c | 361 ++++++++++++++++++++++++++ purgatory/purgatory-elf-core.c | 64 +++++ purgatory/time.c | 59 ++++ 20 files changed, 866 insertions(+), 9 deletions(-) create mode 100644 kexec/ipmi.h create mode 100644 purgatory/arch/i386/rtc_cmos.c create mode 100644 purgatory/arch/x86_64/purgatory-elf-x86_64.c create mode 100644 purgatory/include/purgatory-elf.h create mode 100644 purgatory/include/time.h create mode 100644 purgatory/ipmi.c create mode 100644 purgatory/purgatory-elf-core.c create mode 100644 purgatory/time.c
Signature · 9 lines
Hidehiro Kawai Hitachi, Ltd. Research & Development Group _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
RFC PATCH v2 1/7 purgatory: Introduce timeout API
This patch introduces timeout API for purgatory code.
Because we don't want to use interrupts in purgatory, this timeout API is based on polling of the clock.
If you want to start a timer, call init_timeout() with seconds to the timeout and an instance of struct timeout_info which manages the timer virtually. Then, you call EXIT_ON_TIMEOUT() macro periodically to check if the timeout happens. If it happens, the control jumps to timed_out label to handle it. This means you need to place timed_out label in each caller function.
To make this timeout API work, an actual implementation of get_unix_time() which returns the current UNIX epoch time is needed.
--- purgatory/Makefile | 4 ++- purgatory/include/time.h | 33 ++++++++++++++++++++++++++ purgatory/time.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 95 insertions(+), 1 deletion(-) create mode 100644 purgatory/include/time.h create mode 100644 purgatory/time.c
purgatory/Makefile +3 -1
@@ -12,10 +12,12 @@ PURGATORY_SRCS = PURGATORY_SRCS += purgatory/purgatory.c PURGATORY_SRCS += purgatory/printf.c PURGATORY_SRCS += purgatory/string.c +PURGATORY_SRCS += purgatory/time.c PURGATORY_MAP = purgatory/purgatory.map dist += purgatory/Makefile $(PURGATORY_SRCS) \ - purgatory/include/purgatory.h purgatory/include/string.h + purgatory/include/purgatory.h purgatory/include/string.h \ + purgatory/include/time.h include $(srcdir)/purgatory/arch/alpha/Makefile include $(srcdir)/purgatory/arch/arm/Makefile
purgatory/include/time.h new file +33 -0
@@ -0,0 +1,33 @@ +#ifndef TIME_H +#define TIME_H + +typedef long long time64_t ; + +struct timeout_info { +time64_t start;/* Start time in second */ +time64_t end;/* Expiration time in second */ +int timed_out;/* Fast check flag. Set to 1 on the timeout */ +}; + +time64_t date2unix (int year,int mon,int day,int hour,int min,int sec); +int check_timeout (struct timeout_info *toi); +void init_timeout (struct timeout_info *toi,time64_t left); + +/** +* Check if the timeout happens, and if so, jump to timed_out label. +* +* @toi: an instance of struct timeout_info initialized by init_timeout(). +* @check_clock: if this value is 0, this macro doesn't check the actual +* clock. This is useful for the exit path after timeout. +*/ +#define EXIT_ON_TIMEOUT (toi, check_clock) \ +do { \ + if ((toi)->timed_out) \ + goto timed_out; \ + if ((check_clock) && check_timeout((toi))) { \ + (toi)->timed_out = 1; \ + goto timed_out; \ + } \ +} while (0) + +#endif /* TIME_H */
purgatory/time.c new file +59 -0
59 lines not on this page · raw message
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RFC PATCH v2 2/7 purgatory/x86: Support CMOS RTC
In reply to Hidehiro KawaiSupport CMOS RTC based on MC148618A RTC by overriding get_unix_time(). This is needed for the timeout API previously added.
--- purgatory/arch/i386/Makefile | 1 purgatory/arch/i386/rtc_cmos.c | 107 ++++++++++++++++++++++++++++++++++++++++ purgatory/arch/x86_64/Makefile | 1 3 files changed, 109 insertions(+) create mode 100644 purgatory/arch/i386/rtc_cmos.c
purgatory/arch/i386/Makefile +1 -0
@@ -13,6 +13,7 @@ i386_PURGATORY_SRCS += purgatory/arch/i386/console-x86.c i386_PURGATORY_SRCS += purgatory/arch/i386/vga.c i386_PURGATORY_SRCS += purgatory/arch/i386/pic.c i386_PURGATORY_SRCS += purgatory/arch/i386/crashdump_backup.c +i386_PURGATORY_SRCS += purgatory/arch/i386/rtc_cmos.c dist += purgatory/arch/i386/Makefile $(i386_PURGATORY_SRCS) \ purgatory/arch/i386/purgatory-x86.h \
purgatory/arch/i386/rtc_cmos.c new file +107 -0
107 lines not on this page · raw message
purgatory/arch/x86_64/Makefile +1 -0
@@ -22,5 +22,6 @@ x86_64_PURGATORY_SRCS += purgatory/arch/i386/crashdump_backup.c x86_64_PURGATORY_SRCS += purgatory/arch/i386/console-x86.c x86_64_PURGATORY_SRCS += purgatory/arch/i386/vga.c x86_64_PURGATORY_SRCS += purgatory/arch/i386/pic.c +x86_64_PURGATORY_SRCS += purgatory/arch/i386/rtc_cmos.c x86_64_PURGATORY_EXTRA_CFLAGS = -mcmodel=large
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RFC PATCH v2 3/7 purgatory/ipmi: Support IPMI KCS interface
In reply to Hidehiro KawaiThis patch adds an interface to BMC for purgatory. While IPMI defines various interfaces to BMC, this patch supports only KCS (Keyboard Style Controller) interface which is implemented in the most of recent enterprise class servers with BMC.
Normally, I/O port 0xca2 and 0xca3 are used for KCS interface, but it varies depending on servers. In this case, you can specify the I/O ports by --ipmi-kcs-ports option.
To guarantee that the second kernel starts to boot in fixed seconds, polling-based timeout check is performed. Timeout counting starts when calling ipmi_init_timeout(), and it expires after 5 seconds by default.
--- kexec/kexec.c | 22 ++++ kexec/kexec.h | 4 + purgatory/Makefile | 1 purgatory/ipmi.c | 256 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 282 insertions(+), 1 deletion(-) create mode 100644 purgatory/ipmi.c
kexec/kexec.c +22 -0
@@ -57,6 +57,7 @@ static unsigned long kexec_flags = 0;/* Flags for kexec file (fd) based syscall */ static unsigned long kexec_file_flags =0 ;int kexec_debug =0 ; +int opt_ipmi_ports[2 ];void dbgprint_mem_range (const char *prefix,struct memory_range *mr,int nr_mr) {
@@ -643,6 +644,14 @@ static void update_purgatory(struct kexec_info *info)if (!info->rhdr.e_shdr) {return ; } + +if (opt_ipmi_ports[0 ] !=0 && opt_ipmi_ports[1 ] !=0 ) { +elf_rel_set_symbol (&info->rhdr,"kcs_port_data" , + &opt_ipmi_ports[0 ],sizeof (opt_ipmi_ports[0 ])); +elf_rel_set_symbol (&info->rhdr,"kcs_port_cmd" , + &opt_ipmi_ports[1 ],sizeof (opt_ipmi_ports[1 ])); + } +arch_update_purgatory (info);memset (region,0 ,sizeof (region));sha256_starts (&ctx);
@@ -969,6 +978,8 @@ void usage(void)" preserve context)\n" " to original kernel.\n" " -s, --kexec-file-syscall Use file based syscall for kexec operation\n" +" --ipmi-kcs-ports=<port1>,<port2> Specify Data_In/Out port to port1 and\n" +" Status/Command port to port2. Default is 0xca2,0xca3.\n" " -d, --debug Enable debugging to help spot a failure.\n" "\n" "Supported kernel file types and options: \n");
@@ -1215,6 +1226,7 @@ int main(int argc, char *argv[]) { 0, 0, 0, 0}, };static const char short_options[] =KEXEC_ALL_OPT_STR ; +int ret;/* * First check if --use-kexec-file-syscall is set. That changes lot of
@@ -1345,6 +1357,16 @@ int main(int argc, char *argv[])case OPT_KEXEC_FILE_SYSCALL :/* We already parsed it. Nothing to do. */ break ; +case OPT_IPMI_KCS_PORTS : + ret =sscanf (optarg,"%i,%i" , + &opt_ipmi_ports[0 ], &opt_ipmi_ports[1 ]); +if (ret !=2 ) { +fprintf (stderr,"Bad option value in " +"--ipmi-kcs-ports=%s\n" , optarg); +usage (); +return 1 ; + } +break ;default :break ; }
kexec/kexec.h +3 -1
@@ -224,7 +224,8 @@ extern int file_types;#define OPT_LOAD_PRESERVE_CONTEXT 259#define OPT_LOAD_JUMP_BACK_HELPER 260#define OPT_ENTRY 261 -#define OPT_MAX 262 +#define OPT_IPMI_KCS_PORTS 262 +#define OPT_MAX 263#define KEXEC_OPTIONS \ { "help", 0, 0, OPT_HELP }, \ { "version", 0, 0, OPT_VERSION }, \
@@ -244,6 +245,7 @@ extern int file_types; { "reuseinitrd", 0, 0, OPT_REUSE_INITRD }, \ { "kexec-file-syscall", 0, 0, OPT_KEXEC_FILE_SYSCALL }, \ { "debug", 0, 0, OPT_DEBUG }, \ + { "ipmi-kcs-ports", 1, 0, OPT_IPMI_KCS_PORTS }, \ #define KEXEC_OPT_STR "h?vdfxyluet:ps"
purgatory/Makefile +1 -0
@@ -9,6 +9,7 @@ PURGATORY = purgatory/purgatory.ro PURGATORY_SRCS = +PURGATORY_SRCS += purgatory/ipmi.c PURGATORY_SRCS += purgatory/purgatory.c PURGATORY_SRCS += purgatory/printf.c PURGATORY_SRCS += purgatory/string.c
purgatory/ipmi.c new file +256 -0
256 lines not on this page · raw message
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RFC PATCH v2 4/7 purgatory/ipmi: Support BMC watchdog timer start/stop in purgatory
In reply to Hidehiro KawaiThis patch adds a functionality to start/stop BMC watchdog timer in purgatory. Starting the watchdog timer is useful to automatically reboot the server when we fail to boot the second kernel. Stopping the watchdog timer is useful to prevent the second kernel from being stopped by the watchdog timer enabled while the first kernel is running.
If you specify --ipmi-wdt-start or --ipmi-wdt-stop option to kexec command, BMC's watchdog timer will start or stop respectively while executing purgatory. You can't specify the both options at the same time. The start operation doesn't change the parameters of the watchdog timer such as initial counter and action. You need to set those parameters in the first OS beforehand. On the other hand, the stop operation changes the parameters. You need to reset them when you reuse the watchdog timer later.
--- kexec/ipmi.h | 9 +++++ kexec/kexec.c | 21 +++++++++++ kexec/kexec.h | 6 +++ purgatory/arch/x86_64/purgatory-x86_64.c | 10 +++++ purgatory/include/purgatory.h | 4 ++ purgatory/ipmi.c | 58 ++++++++++++++++++++++++++++++ 6 files changed, 106 insertions(+), 2 deletions(-) create mode 100644 kexec/ipmi.h
kexec/ipmi.h new file +9 -0
@@ -0,0 +1,9 @@ +#ifndef IPMI_H +#define IPMI_H + +/* Options for IPMI code excuted in purgatory */ +#define IPMI_WDT_DO_NOTHING 0 +#define IPMI_WDT_START (1 << 0) +#define IPMI_WDT_STOP (1 << 1) + +#endif /* IPMI_H */
kexec/kexec.c +21 -0
@@ -49,6 +49,7 @@#include "kexec-sha256.h" #include "kexec-zlib.h" #include "kexec-lzma.h" +#include "ipmi.h" #include <arch/options.h> unsigned long long mem_min =0 ;
@@ -57,6 +58,7 @@ static unsigned long kexec_flags = 0;/* Flags for kexec file (fd) based syscall */ static unsigned long kexec_file_flags =0 ;int kexec_debug =0 ; +int opt_ipmi_wdt =IPMI_WDT_DO_NOTHING ;int opt_ipmi_ports[2 ];void dbgprint_mem_range (const char *prefix,struct memory_range *mr,int nr_mr)
@@ -645,6 +647,9 @@ static void update_purgatory(struct kexec_info *info)return ; } +elf_rel_set_symbol (&info->rhdr,"ipmi_wdt" , &opt_ipmi_wdt, +sizeof (opt_ipmi_wdt)); +if (opt_ipmi_ports[0 ] !=0 && opt_ipmi_ports[1 ] !=0 ) {elf_rel_set_symbol (&info->rhdr,"kcs_port_data" , &opt_ipmi_ports[0 ],sizeof (opt_ipmi_ports[0 ]));
@@ -980,6 +985,10 @@ void usage(void)" -s, --kexec-file-syscall Use file based syscall for kexec operation\n" " --ipmi-kcs-ports=<port1>,<port2> Specify Data_In/Out port to port1 and\n" " Status/Command port to port2. Default is 0xca2,0xca3.\n" +" --ipmi-wdt-start Start BMC's watchdog timer in panic case before\n" +" starting the second kernel\n" +" --ipmi-wdt-stop Stop BMC's watchdog timer in panic case before starting\n" +" the second kernel\n" " -d, --debug Enable debugging to help spot a failure.\n" "\n" "Supported kernel file types and options: \n");
@@ -1357,6 +1366,12 @@ int main(int argc, char *argv[])case OPT_KEXEC_FILE_SYSCALL :/* We already parsed it. Nothing to do. */ break ; +case OPT_IPMI_WDT_START : + opt_ipmi_wdt |=IPMI_WDT_START ; +break ; +case OPT_IPMI_WDT_STOP : + opt_ipmi_wdt |=IPMI_WDT_STOP ; +break ;case OPT_IPMI_KCS_PORTS : ret =sscanf (optarg,"%i,%i" , &opt_ipmi_ports[0 ], &opt_ipmi_ports[1 ]);
@@ -1392,6 +1407,12 @@ int main(int argc, char *argv[]) "\"--mem-max\" parameter\n"); } +if ((opt_ipmi_wdt &IPMI_WDT_START ) && + (opt_ipmi_wdt &IPMI_WDT_STOP )) { +die ("You can't specify both --ipmi-wdt-start and " +"--ipmi-wdt-stop\n" ); + } + fileind = optind;/* Reset getopt for the next pass; called in other source modules */ opterr =1 ;
kexec/kexec.h +5 -1
@@ -225,7 +225,9 @@ extern int file_types;#define OPT_LOAD_JUMP_BACK_HELPER 260#define OPT_ENTRY 261#define OPT_IPMI_KCS_PORTS 262 -#define OPT_MAX 263 +#define OPT_IPMI_WDT_START 263 +#define OPT_IPMI_WDT_STOP 264 +#define OPT_MAX 265#define KEXEC_OPTIONS \ { "help", 0, 0, OPT_HELP }, \ { "version", 0, 0, OPT_VERSION }, \
@@ -246,6 +248,8 @@ extern int file_types; { "kexec-file-syscall", 0, 0, OPT_KEXEC_FILE_SYSCALL }, \ { "debug", 0, 0, OPT_DEBUG }, \ { "ipmi-kcs-ports", 1, 0, OPT_IPMI_KCS_PORTS }, \ + { "ipmi-wdt-start", 0, 0, OPT_IPMI_WDT_START }, \ + { "ipmi-wdt-stop", 0, 0, OPT_IPMI_WDT_STOP }, \ #define KEXEC_OPT_STR "h?vdfxyluet:ps"
purgatory/arch/x86_64/purgatory-x86_64.c +9 -1
@@ -25,6 +25,14 @@ void x86_setup_jump_back_entry(void)/* This function can be used to execute after the SHA256 verification. */ void post_verification_setup_arch (void ) { -if (panic_kernel)crashdump_backup_memory (); +if (panic_kernel) { +crashdump_backup_memory (); + +if (ipmi_wdt) { +ipmi_init_timeout (); +ipmi_wdt_start_stop (); + } + } +if (jump_back_entry)x86_setup_jump_back_entry (); }
purgatory/include/purgatory.h +4 -0
@@ -1,11 +1,15 @@#ifndef PURGATORY_H #define PURGATORY_H +extern int ipmi_wdt; +void putchar (int ch);void sprintf (char *buffer,const char *fmt, ...) __attribute__ ((format (printf,2 ,3 )));void printf (const char *fmt, ...) __attribute__ ((format (printf,1 ,2 )));void setup_arch (void );void post_verification_setup_arch (void ); +void ipmi_init_timeout (void ); +void ipmi_wdt_start_stop (void );#endif /* PURGATORY_H */
purgatory/ipmi.c +58 -0
77 lines not on this page · raw message
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RFC PATCH v2 5/7 purgatory/ipmi: Add an option for purgatory to handle panic event with IPMI
In reply to Hidehiro KawaiThis patch adds --ipmi-handle-panic option which enables panic event handling in purgatory. At this point, if you specify the option, an "OS Stop" event is generated in the panic case. As the result, BMC may log the event to SEL. Furthermore, you can make the BMC issue an SNMP trap by configuring PEF alerting of the BMC in advance.
In the future, other actions to inform or record the panic event will be added.
--- kexec/kexec.c | 8 ++++++++ kexec/kexec.h | 4 +++- purgatory/arch/x86_64/purgatory-x86_64.c | 13 +++++++++++-- purgatory/include/purgatory.h | 4 ++++ purgatory/ipmi.c | 23 +++++++++++++++++++++++ 5 files changed, 49 insertions(+), 3 deletions(-)
kexec/kexec.c +8 -0
@@ -59,6 +59,7 @@ static unsigned long kexec_flags = 0;static unsigned long kexec_file_flags =0 ;int kexec_debug =0 ;int opt_ipmi_wdt =IPMI_WDT_DO_NOTHING ; +uint8_t opt_ipmi_handle_panic =0 ;int opt_ipmi_ports[2 ];void dbgprint_mem_range (const char *prefix,struct memory_range *mr,int nr_mr)
@@ -649,6 +650,9 @@ static void update_purgatory(struct kexec_info *info)elf_rel_set_symbol (&info->rhdr,"ipmi_wdt" , &opt_ipmi_wdt,sizeof (opt_ipmi_wdt)); +elf_rel_set_symbol (&info->rhdr,"ipmi_handle_panic" , + &opt_ipmi_handle_panic, +sizeof (opt_ipmi_handle_panic));if (opt_ipmi_ports[0 ] !=0 && opt_ipmi_ports[1 ] !=0 ) { elf_rel_set_symbol(&info->rhdr, "kcs_port_data",
@@ -989,6 +993,7 @@ void usage(void)" starting the second kernel\n" " --ipmi-wdt-stop Stop BMC's watchdog timer in panic case before starting\n" " the second kernel\n" +" --ipmi-handle-panic Inform/record the panic event via IPMI\n" " -d, --debug Enable debugging to help spot a failure.\n" "\n" "Supported kernel file types and options: \n");
@@ -1382,6 +1387,9 @@ int main(int argc, char *argv[])return 1 ; }break ; +case OPT_IPMI_HANDLE_PANIC : + opt_ipmi_handle_panic =1 ; +break ;default :break ; }
kexec/kexec.h +3 -1
@@ -227,7 +227,8 @@ extern int file_types;#define OPT_IPMI_KCS_PORTS 262#define OPT_IPMI_WDT_START 263#define OPT_IPMI_WDT_STOP 264 -#define OPT_MAX 265 +#define OPT_IPMI_HANDLE_PANIC 265 +#define OPT_MAX 266#define KEXEC_OPTIONS \ { "help", 0, 0, OPT_HELP }, \ { "version", 0, 0, OPT_VERSION }, \
@@ -250,6 +251,7 @@ extern int file_types; { "ipmi-kcs-ports", 1, 0, OPT_IPMI_KCS_PORTS }, \ { "ipmi-wdt-start", 0, 0, OPT_IPMI_WDT_START }, \ { "ipmi-wdt-stop", 0, 0, OPT_IPMI_WDT_STOP }, \ + { "ipmi-handle-panic", 0, 0, OPT_IPMI_HANDLE_PANIC }, \ #define KEXEC_OPT_STR "h?vdfxyluet:ps"
purgatory/arch/x86_64/purgatory-x86_64.c +11 -2
@@ -28,10 +28,19 @@ void post_verification_setup_arch(void)if (panic_kernel) {crashdump_backup_memory (); -if (ipmi_wdt) { +if (ipmi_handle_panic || ipmi_wdt)ipmi_init_timeout (); + +/* +* You can make the BMC send an SNMP trap here by configuring +* PEF alerting beforehand. Do this first to inform the +* panic event ASAP. +*/ +if (ipmi_handle_panic) +ipmi_send_panic_event (); + +if (ipmi_wdt)ipmi_wdt_start_stop (); - } }if (jump_back_entry)x86_setup_jump_back_entry ();
purgatory/include/purgatory.h +4 -0
@@ -1,7 +1,10 @@#ifndef PURGATORY_H #define PURGATORY_H +#include <stdint.h> +extern int ipmi_wdt; +extern uint8_t ipmi_handle_panic;void putchar (int ch);void sprintf (char *buffer,const char *fmt, ...)
@@ -11,5 +14,6 @@ void setup_arch(void);void post_verification_setup_arch (void );void ipmi_init_timeout (void );void ipmi_wdt_start_stop (void ); +void ipmi_send_panic_event (void );#endif /* PURGATORY_H */
purgatory/ipmi.c +23 -0
@@ -26,6 +26,7 @@int kcs_port_cmd =KCS_PORT_CMD_DEFAULT ;int kcs_port_data =KCS_PORT_DATA_DEFAULT ;int ipmi_wdt =IPMI_WDT_DO_NOTHING ; +uint8_t ipmi_handle_panic =0 ;/* IPMI command to start BMC watchdog timer */ const unsigned char cmd_start_wdt[] = {
@@ -45,6 +46,23 @@ const unsigned char cmd_stop_wdt[] = { 0xff, }; +/* +* IPMI command to generate an event to inform a kernel panic happend. +* All fixed parameters are same as that of IPMI driver for Linux. +*/ +const unsigned char cmd_panic_event[] = { +0x04 <<2 ,/* Sensor/Event */ +0x02 ,/* Platform Event Command */ +0x41 ,/* Generator ID: System Management Software */ +0x03 ,/* Event Message Rev.: comply with IPMI 1.0 */ +0x20 ,/* Sensor Type: OS Stop */ +0x00 ,/* Sensor # */ +0x6f ,/* Event Type: Sensor specific */ +0xa1 ,/* Data 1: Run-time Stop. Data 2 & 3 are OEM defined */ +0x00 ,/* Data 2 */ +0x00 ,/* Data 3 */ +}; +/* Total timeout for IPMI operations */ static struct timeout_info ipmi_to = { .end =INT64_MAX ,/* never time out */
@@ -312,3 +330,8 @@ void ipmi_wdt_start_stop(void)else if (ipmi_wdt &IPMI_WDT_STOP )do_stop_wdt (); } + +void ipmi_send_panic_event (void ) +{ +issue_ipmi_cmd (cmd_panic_event,sizeof (cmd_panic_event)); +}
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RFC PATCH v2 6/7 purgatory/x86: Add an option to output IP registers to console in panic case
In reply to Hidehiro KawaiThis patch adds --output-cpu-ip option which outpus the values of IP registers to VGA/serial console. To take effect, you need to specify both or one of --console-vga and --console-serial options, too.
The values of IP registers are taken from the ELF core header prepared by the first kernel. To do this, the address of the header is passed to the purgatory through elfcorehdr variable.
--- kexec/arch/i386/crashdump-x86.c | 10 ++-- kexec/arch/i386/include/arch/options.h | 4 +- kexec/arch/i386/kexec-x86.h | 1 kexec/arch/x86_64/kexec-x86_64.c | 15 ++++++ kexec/kexec.h | 1 purgatory/Makefile | 1 purgatory/arch/i386/purgatory-x86.h | 1 purgatory/arch/x86_64/Makefile | 1 purgatory/arch/x86_64/purgatory-elf-x86_64.c | 49 ++++++++++++++++++++ purgatory/arch/x86_64/purgatory-x86_64.c | 5 ++ purgatory/include/purgatory-elf.h | 13 +++++ purgatory/purgatory-elf-core.c | 64 ++++++++++++++++++++++++++ 12 files changed, 159 insertions(+), 6 deletions(-) create mode 100644 purgatory/arch/x86_64/purgatory-elf-x86_64.c create mode 100644 purgatory/include/purgatory-elf.h create mode 100644 purgatory/purgatory-elf-core.c
kexec/arch/i386/crashdump-x86.c +5 -5
@@ -867,7 +867,7 @@ int load_crashdump_segments(struct kexec_info *info, char* mod_cmdline, unsigned long max_addr, unsigned long min_base) { void *tmp; -unsigned long sz, bufsz, memsz, elfcorehdr; +unsigned long sz, bufsz, memsz;int nr_ranges =0 , nr_memmap =0 , align =1024 , i;struct memory_range *mem_range, *memmap_p;struct crash_elf_info elf_info;
@@ -990,15 +990,15 @@ int load_crashdump_segments(struct kexec_info *info, char* mod_cmdline, }else { memsz = bufsz; } - elfcorehdr =add_buffer (info, tmp, bufsz, memsz, align, min_base, + info->elfcorehdr =add_buffer (info, tmp, bufsz, memsz, align, min_base, max_addr,-1 ); -dbgprintf ("Created elf header segment at 0x%lx\n" , elfcorehdr); -if (delete_memmap (memmap_p, &nr_memmap, elfcorehdr, memsz) <0 ) +dbgprintf ("Created elf header segment at 0x%lx\n" , info->elfcorehdr); +if (delete_memmap (memmap_p, &nr_memmap, info->elfcorehdr, memsz) <0 )return -1 ;if (!bzImage_support_efi_boot || arch_options.noefi || !sysfs_efi_runtime_map_exist ())cmdline_add_efi (mod_cmdline); -cmdline_add_elfcorehdr (mod_cmdline, elfcorehdr); +cmdline_add_elfcorehdr (mod_cmdline, info->elfcorehdr);/* Inform second kernel about the presence of ACPI tables. */ for (i =0 ; i <CRASH_MAX_MEMORY_RANGES ; i++) {
kexec/arch/i386/include/arch/options.h +3 -1
@@ -17,7 +17,8 @@#define OPT_CONSOLE_SERIAL (OPT_MAX+4)#define OPT_ELF32_CORE (OPT_MAX+5)#define OPT_ELF64_CORE (OPT_MAX+6) -#define OPT_ARCH_MAX (OPT_MAX+7) +#define OPT_OUTPUT_CPU_IP (OPT_MAX+7) +#define OPT_ARCH_MAX (OPT_MAX+8)#define OPT_APPEND (OPT_ARCH_MAX+0)#define OPT_REUSE_CMDLINE (OPT_ARCH_MAX+1)
@@ -45,6 +46,7 @@ { "elf64-core-headers", 0, 0, OPT_ELF64_CORE }, \ { "pass-memmap-cmdline", 0, 0, OPT_PASS_MEMMAP_CMDLINE }, \ { "noefi", 0, 0, OPT_NOEFI}, \ + { "output-cpu-ip", 0, 0, OPT_OUTPUT_CPU_IP }, \ #define KEXEC_ARCH_OPT_STR KEXEC_OPT_STR ""
kexec/arch/i386/kexec-x86.h +1 -0
@@ -52,6 +52,7 @@ struct arch_options_t {enum coretype core_header_type;uint8_t pass_memmap_cmdline;uint8_t noefi; +uint8_t output_cpu_ip; };int multiboot_x86_probe (const char *buf,off_t len);
kexec/arch/x86_64/kexec-x86_64.c +15 -0
@@ -55,6 +55,8 @@ void arch_usage(void)" --console-serial Enable the serial console\n" " --pass-memmap-cmdline Pass memory map via command line in kexec on panic case\n" " --noefi Disable efi support\n" +" --output-cpu-ip Output RIP for each CPU in kexec on panic\n" +" case. You should use with --console-* options\n" ); }
@@ -67,6 +69,7 @@ struct arch_options_t arch_options = { .core_header_type = CORE_TYPE_ELF64, .pass_memmap_cmdline = 0, .noefi = 0, + .output_cpu_ip = 0, };int arch_process_options (int argc,char **argv)
@@ -136,8 +139,15 @@ int arch_process_options(int argc, char **argv)case OPT_NOEFI : arch_options.noefi =1 ;break ; +case OPT_OUTPUT_CPU_IP : + arch_options.output_cpu_ip =1 ; +break ; } } + +if ((arch_options.console_vga | arch_options.console_serial) ==0 ) + arch_options.output_cpu_ip =0 ; +/* Reset getopt for the next pass; called in other source modules */ opterr =1 ; optind =1 ;
@@ -177,6 +187,11 @@ void arch_update_purgatory(struct kexec_info *info) panic_kernel =1 ;elf_rel_set_symbol (&info->rhdr,"backup_start" , &info->backup_start,sizeof (info->backup_start)); +elf_rel_set_symbol (&info->rhdr,"elfcorehdr" , + &info->elfcorehdr,sizeof (info->elfcorehdr)); +elf_rel_set_symbol (&info->rhdr,"output_cpu_ip" , + &arch_options.output_cpu_ip, +sizeof (arch_options.output_cpu_ip)); }elf_rel_set_symbol (&info->rhdr,"panic_kernel" , &panic_kernel,sizeof (panic_kernel));
kexec/kexec.h +1 -0
@@ -158,6 +158,7 @@ struct kexec_info {unsigned long kexec_flags;unsigned long backup_src_start;unsigned long backup_src_size; +unsigned long elfcorehdr;/* Set to 1 if we are using kexec file syscall */ unsigned long file_mode :1;
purgatory/Makefile +1 -0
@@ -14,6 +14,7 @@ PURGATORY_SRCS += purgatory/purgatory.c PURGATORY_SRCS += purgatory/printf.c PURGATORY_SRCS += purgatory/string.c PURGATORY_SRCS += purgatory/time.c +PURGATORY_SRCS += purgatory/purgatory-elf-core.c PURGATORY_MAP = purgatory/purgatory.map dist += purgatory/Makefile $(PURGATORY_SRCS) \
purgatory/arch/i386/purgatory-x86.h +1 -0
@@ -5,5 +5,6 @@ void x86_reset_vga(void);void x86_setup_legacy_pic (void );void x86_setup_legacy_timer (void );void crashdump_backup_memory (void ); +void print_ip (void );#endif /* PURGATORY_X86_H */
purgatory/arch/x86_64/Makefile +1 -0
@@ -7,6 +7,7 @@ x86_64_PURGATORY_SRCS_native += purgatory/arch/x86_64/entry64.S x86_64_PURGATORY_SRCS_native += purgatory/arch/x86_64/setup-x86_64.S x86_64_PURGATORY_SRCS_native += purgatory/arch/x86_64/stack.S x86_64_PURGATORY_SRCS_native += purgatory/arch/x86_64/purgatory-x86_64.c +x86_64_PURGATORY_SRCS_native += purgatory/arch/x86_64/purgatory-elf-x86_64.c x86_64_PURGATORY_SRCS += $(x86_64_PURGATORY_SRCS_native)
purgatory/arch/x86_64/purgatory-elf-x86_64.c new file +49 -0
49 lines not on this page · raw message
purgatory/arch/x86_64/purgatory-x86_64.c +5 -0
@@ -1,11 +1,13 @@#include <stdint.h> #include <stddef.h> #include <purgatory.h> +#include <purgatory-elf.h> #include "purgatory-x86_64.h" uint8_t reset_vga =0 ;uint8_t legacy_pic =0 ;uint8_t panic_kernel =0 ; +uint8_t output_cpu_ip =0 ;unsigned long jump_back_entry =0 ;char *cmdline_end =NULL ;
@@ -41,6 +43,9 @@ void post_verification_setup_arch(void)if (ipmi_wdt)ipmi_wdt_start_stop (); + +if (output_cpu_ip &&have_valid_elf_header ()) +print_ip (); }if (jump_back_entry)x86_setup_jump_back_entry ();
purgatory/include/purgatory-elf.h new file +13 -0
@@ -0,0 +1,13 @@ +#ifndef PURGATORY_ELF_H +#define PURGATORY_ELF_H + +#include <elf.h> + +#define _ALIGN_MASK (addr, mask) (((addr) + (mask)) & ~(mask)) +#define _ALIGN (addr, size) _ALIGN_MASK((addr), (typeof(addr))(size) - 1) + +extern int have_valid_elf_header (void ); +extern char *get_note_contents (Elf64_Phdr *phdr); +extern void process_elf_notes (void (*fn)(Elf64_Phdr *)); + +#endif /* PURGATORY_ELF_H */
purgatory/purgatory-elf-core.c new file +64 -0
64 lines not on this page · raw message
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
RFC PATCH v2 7/7 purgatory/ipmi/x86: Support logging of IP registers into SEL
In reply to Hidehiro KawaiIf you specify both --output-cpu-ip and --ipmi-handle-panic, the values of IP registers are logged to BMC's SEL in the panic case. This kind of information is useful when the first kernel hanged up and panicked by NMI but booting the second kernel failed.
--- purgatory/arch/i386/purgatory-x86.h | 1 + purgatory/arch/x86_64/purgatory-elf-x86_64.c | 33 ++++++++++++++++++++++++++ purgatory/arch/x86_64/purgatory-x86_64.c | 13 +++++++++- purgatory/include/purgatory.h | 1 + purgatory/ipmi.c | 24 +++++++++++++++++++ 5 files changed, 71 insertions(+), 1 deletion(-)
purgatory/arch/i386/purgatory-x86.h +1 -0
@@ -6,5 +6,6 @@ void x86_setup_legacy_pic(void);void x86_setup_legacy_timer (void );void crashdump_backup_memory (void );void print_ip (void ); +void write_ip_to_sel (void );#endif /* PURGATORY_X86_H */
purgatory/arch/x86_64/purgatory-elf-x86_64.c +33 -0
@@ -43,7 +43,40 @@ static void fn_print_ip(Elf64_Phdr *phdr)printf ("RIP: %lx\n" ,get_ip_from_crash_note (phdr)); } +static void fn_write_ip_to_sel (Elf64_Phdr *phdr) +{ +static int seqnum =0 ; +unsigned long rip; +unsigned char *src, *dst; +int i; +struct { +uint8_t reserved[3 ]; +uint16_t le_seqnum; +uint64_t be_ip; + } __attribute__((packed)) sel_data; + +memset (&sel_data,0 ,sizeof (sel_data)); + +if (is_valid_crash_note (phdr)) { + sel_data.le_seqnum = seqnum++; + rip =get_ip_from_crash_note (phdr); + +/* Convert to big-endian for convenience */ + src = (unsigned char *)&rip; + dst = (unsigned char *)&sel_data.be_ip; +for (i=0 ; i <sizeof (rip); i++) + dst[i] = src[7 - i]; + +ipmi_sel_write ((unsigned char *)&sel_data,sizeof (sel_data)); + } +} +void print_ip (void ) {process_elf_notes (fn_print_ip); } + +void write_ip_to_sel (void ) +{ +process_elf_notes (fn_write_ip_to_sel); +}
purgatory/arch/x86_64/purgatory-x86_64.c +12 -1
@@ -27,8 +27,11 @@ void x86_setup_jump_back_entry(void)/* This function can be used to execute after the SHA256 verification. */ void post_verification_setup_arch (void ) { +int valid_elf =0 ; +if (panic_kernel) {crashdump_backup_memory (); + valid_elf =have_valid_elf_header ();if (ipmi_handle_panic || ipmi_wdt)ipmi_init_timeout ();
@@ -44,8 +47,16 @@ void post_verification_setup_arch(void)if (ipmi_wdt)ipmi_wdt_start_stop (); -if (output_cpu_ip &&have_valid_elf_header ()) +if (output_cpu_ip && valid_elf) { +/* +* Since this may fill up all SEL entries on many core +* machine, do it last. +*/ +if (ipmi_handle_panic) +write_ip_to_sel (); +print_ip (); + } }if (jump_back_entry)x86_setup_jump_back_entry ();
purgatory/include/purgatory.h +1 -0
@@ -15,5 +15,6 @@ void post_verification_setup_arch(void);void ipmi_init_timeout (void );void ipmi_wdt_start_stop (void );void ipmi_send_panic_event (void ); +void ipmi_sel_write (unsigned char *buf,int size);#endif /* PURGATORY_H */
purgatory/ipmi.c +24 -0
@@ -1,4 +1,5 @@#include <stdint.h> +#include <string.h> #include <sys/io.h> #include <purgatory.h> #include "../kexec/ipmi.h"
@@ -63,6 +64,16 @@ const unsigned char cmd_panic_event[] = { 0x00, /* Data 3 */ }; +unsigned char cmd_add_sel_entry[18 ] = { +0x0a <<2 ,/* Storage */ +0x44 ,/* Add SEL Entry */ +0x00 ,0x00 ,/* Record ID: 0 for Add SEL Entry */ +0xf1 ,/* Record Type: OEM event without timestamp */ +/* Remaining 13 bytes convey any data */ +}; + +#define SEL_DATA_START 5 +/* Total timeout for IPMI operations */ static struct timeout_info ipmi_to = { .end =INT64_MAX ,/* never time out */
@@ -335,3 +346,16 @@ void ipmi_send_panic_event(void) {issue_ipmi_cmd (cmd_panic_event,sizeof (cmd_panic_event)); } + +void ipmi_sel_write (unsigned char *buf,int size) +{ +/* We can write less than 13 bytes at once */ +if (size >sizeof (cmd_add_sel_entry) -SEL_DATA_START ) +return ; + +memset (&cmd_add_sel_entry[SEL_DATA_START ],0 , +sizeof (cmd_add_sel_entry) -SEL_DATA_START ); +memcpy (&cmd_add_sel_entry[SEL_DATA_START ], buf, size); + +issue_ipmi_cmd (cmd_add_sel_entry,sizeof (cmd_add_sel_entry)); +}
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Hello,
Are there any comments especially about the direction? Currently, there are two opinions.
Corey Minyard (IPMI driver maintainer) prefers to do this kind of works in the 1st kernel because the IPMI driver already supports various BMC implementations. His comments can be found here with my RFC v1 patch set:
http://thread.gmane.org/gmane.linux.kernel.kexec/15131
And, preliminary discussion related to this patch set with Eric Biederman can be found here:
https://lkml.org/lkml/2015/8/4/301
If there is no strong objection, I just complete the missing parts...
Best regards,
Hidehiro Kawai
Quoted 133 lines
From: kexec [mailto:[email protected]] On Behalf Of Hidehiro Kawai This is the version 2 of RFC patch series previously named "purgatory: Add basic support for IPMI command execution."
Changes since RFC v1:
- Add --ipmi-kcs-ports option to specify I/O ports for KCS I/F
- Add --ipmi-handle-panic option to generate an event on BMC to inform and record about the panic
- Add --output-cpu-ip option to output IP registers to console and BMC's SEL
...and various cleanups and improvementsI performed some Web searches, and I found that major BMCs still support KCS I/F. Although this patch series supports only KCS I/F, it should cover many recent servers with BMC. However, I/O ports used for KCS I/F are varies among servers, so I added an option to specify port numbers for KCS.
About error handling for KCS protocol, I re-checked IPMI specification, and I found that simply retrying on error is sufficient. So I didn't change the logic for the simplicity. Please see PATCH 3/7 for details.
Background and purpose
======================If the second kernel for crash dumping hangs up while booting, no information related to the first kernel will be saved. This makes crash cause analysis difficult. Some enterprise users don't permit the same faults happen multiple times, so we want to save minimal information before booting the second kernel.
Approaches
==========One of the approaches is (1) to use panic notifier call or kmsg dump feature. For example, a panic notifier callback registered by IPMI driver can save the panic message to BMC's SEL before booting the second kernel. Similarly, kmsg dump saves kernel logs to a non- volatile memory on the server. This approach covers multiple hardware/firmware implementation. However, doing many things in crashed kernel will reduce the reliability. Additionally, a part of the code is also used for normal operation and still evolving. This would makes it difficult to keep stable.
Another approach is (2) to save minimal information to BMC's SEL in purgatory. It is difficult to do complicate things in purgatory, but fortunately IPMI specification defines a simple interface, KCS (Keyboard Controller Style) I/F. KCS is controlled by two I/O ports and supported by most of major BMCs.
Here, we want more reliable one for the purpose, we adopt (2).
What are provided?
==================This patch series provides multiple RAS features other than the main purpose described above.
- Timeout mechanism relying on polling RTC
- API to access BMC via KCS I/F
- Command line option to start/stop BMC's watchdog timer in purgatory
- Command line option to write the value of RIP registers to SEL and/or serial console (useful for kernel hang-up cases)
- Command line option to generate a plantform event on BMC (useful for server monitoring or HA clustering; you can make the BMC issue an SNMP trap)
- Command line option to change the default I/O ports of KCS I/FLimitations of RFC version
==========================This patch serires is incomplete, and there are some limitations.
- Related codes are unconditionally built into the kexec binary
- Implemented only forx86_64(it may break the build for other architectures)
- Timeout value for IPMI operations is hard-codedFuture plan
===========- Add an option to save the panic message to BMC's SEL (this requires some kernel modifications)
---
Hidehiro Kawai (7):
purgatory: Introduce timeout API
purgatory/x86: Support CMOS RTC
purgatory/ipmi: Support IPMI KCS interface
purgatory/ipmi: Support BMC watchdog timer start/stop in purgatory purgatory/ipmi: Add an option for purgatory to handle panic event with IPMI purgatory/x86: Add an option to output IP registers to console in panic case purgatory/ipmi/x86: Support logging of IP registers into SELkexec/arch/i386/crashdump-x86.c | 10 - kexec/arch/i386/include/arch/options.h | 4 kexec/arch/i386/kexec-x86.h | 1 kexec/arch/x86_64/kexec-x86_64.c | 15 + kexec/ipmi.h | 9 + kexec/kexec.c | 51 ++++ kexec/kexec.h | 11 + purgatory/Makefile | 6 purgatory/arch/i386/Makefile | 1 purgatory/arch/i386/purgatory-x86.h | 2 purgatory/arch/i386/rtc_cmos.c | 107 ++++++++ purgatory/arch/x86_64/Makefile | 2 purgatory/arch/x86_64/purgatory-elf-x86_64.c | 82 ++++++ purgatory/arch/x86_64/purgatory-x86_64.c | 35 ++- purgatory/include/purgatory-elf.h | 13 + purgatory/include/purgatory.h | 9 + purgatory/include/time.h | 33 ++ purgatory/ipmi.c | 361 ++++++++++++++++++++++++++ purgatory/purgatory-elf-core.c | 64 +++++ purgatory/time.c | 59 ++++ 20 files changed, 866 insertions(+), 9 deletions(-) create mode 100644 kexec/ipmi.h create mode 100644 purgatory/arch/i386/rtc_cmos.c create mode 100644 purgatory/arch/x86_64/purgatory-elf-x86_64.c create mode 100644 purgatory/include/purgatory-elf.h create mode 100644 purgatory/include/time.h create mode 100644 purgatory/ipmi.c create mode 100644 purgatory/purgatory-elf-core.c create mode 100644 purgatory/time.c
Quoted 3 lines
--
Hidehiro Kawai
Hitachi, Ltd. Research & Development Group
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec