exit: Put an upper limit on how often we can oops
Subsystems (9)
v3 cover exit: Put an upper limit on how often we can oops
Hi,
This builds on Jann's v1 patch[1]. Changes in v3:
- fix #if/#ifdef confusion (Bill)
- rename from "reason" or "origin" and add it to the warn output (Marco)
v2: https://lore.kernel.org/lkml/[email protected]/
Thanks,
-Kees
[1] https://lore.kernel.org/lkml/[email protected]
Jann Horn (1):
exit: Put an upper limit on how often we can oops
Kees Cook (5):
panic: Separate sysctl logic from CONFIG_SMP
exit: Expose "oops_count" to sysfs
panic: Consolidate open-coded panic_on_warn checks
panic: Introduce warn_limit
panic: Expose "warn_count" to sysfs
.../ABI/testing/sysfs-kernel-oops_count | 6 ++ .../ABI/testing/sysfs-kernel-warn_count | 6 ++ Documentation/admin-guide/sysctl/kernel.rst | 17 ++++++ MAINTAINERS | 2 + include/linux/panic.h | 1 + kernel/exit.c | 60 +++++++++++++++++++ kernel/kcsan/report.c | 3 +- kernel/panic.c | 45 +++++++++++++- kernel/sched/core.c | 3 +- lib/ubsan.c | 3 +- mm/kasan/report.c | 4 +- mm/kfence/report.c | 3 +- 12 files changed, 140 insertions(+), 13 deletions(-) create mode 100644 Documentation/ABI/testing/sysfs-kernel-oops_count create mode 100644 Documentation/ABI/testing/sysfs-kernel-warn_count
Signature · 1 line
2.34.1
PATCH v3 1/6 panic: Separate sysctl logic from CONFIG_SMP
In preparation for adding more sysctls directly in kernel/panic.c, split CONFIG_SMP from the logic that adds sysctls.
--- kernel/panic.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/panic.c +3 -1
@@ -75,8 +75,9 @@ ATOMIC_NOTIFIER_HEAD(panic_notifier_list);EXPORT_SYMBOL (panic_notifier_list); -#if defined(CONFIG_SMP) && defined(CONFIG_SYSCTL) +#ifdef CONFIG_SYSCTL static struct ctl_table kern_panic_table[] = { +#ifdef CONFIG_SMP { .procname = "oops_all_cpu_backtrace", .data = &sysctl_oops_all_cpu_backtrace,
@@ -86,6 +87,7 @@ static struct ctl_table kern_panic_table[] = { .extra1 = SYSCTL_ZERO, .extra2 = SYSCTL_ONE, }, +#endif { } };
Signature · 1 line
2.34.1
PATCH v3 2/6 exit: Put an upper limit on how often we can oops
In reply to Kees CookFrom: Jann Horn <[email protected]>
Many Linux systems are configured to not panic on oops; but allowing an attacker to oops the system **really** often can make even bugs that look completely unexploitable exploitable (like NULL dereferences and such) if each crash elevates a refcount by one or a lock is taken in read mode, and this causes a counter to eventually overflow.
The most interesting counters for this are 32 bits wide (like open-coded refcounts that don't use refcount_t). (The ldsem reader count on 32-bit platforms is just 16 bits, but probably nobody cares about 32-bit platforms that much nowadays.)
So let's panic the system if the kernel is constantly oopsing.
The speed of oopsing 2^32 times probably depends on several factors, like how long the stack trace is and which unwinder you're using; an empirically important one is whether your console is showing a graphical environment or a text console that oopses will be printed to.
In a quick single-threaded benchmark, it looks like oopsing in a vfork() child with a very short stack trace only takes ~510 microseconds per run when a graphical console is active; but switching to a text console that oopses are printed to slows it down around 87x, to ~45 milliseconds per run.
(Adding more threads makes this faster, but the actual oops printing happens under &die_lock on x86, so you can maybe speed this up by a factor of around 2 and then any further improvement gets eaten up by lock contention.)
It looks like it would take around 8-12 days to overflow a 32-bit counter with repeated oopsing on a multi-core X86 system running a graphical environment; both me (in an X86 VM) and Seth (with a distro kernel on normal hardware in a standard configuration) got numbers in that ballpark.
12 days aren't *that* short on a desktop system, and you'd likely need much longer on a typical server system (assuming that people don't run graphical desktop environments on their servers), and this is a *very* noisy and violent approach to exploiting the kernel; and it also seems to take orders of magnitude longer on some machines, probably because stuff like EFI pstore will slow it down a ton if that's active.
--- Documentation/admin-guide/sysctl/kernel.rst | 8 ++++ kernel/exit.c | 42 +++++++++++++++++++++ 2 files changed, 50 insertions(+)
Documentation/admin-guide/sysctl/kernel.rst +8 -0
@@ -667,6 +667,14 @@ This is the default behavior. an oops event is detected. +oops_limit +========== + +Number of kernel oopses after which the kernel should panic when +``panic_on_oops`` is not set. Setting this to 0 or 1 has the same effect +as setting ``panic_on_oops=1``. + + osrelease, ostype & version ===========================
kernel/exit.c +42 -0
60 lines not on this page · raw message
Signature · 1 line
2.34.1
Hello,
Quoted 34 lines · On Thu, 17 Nov 2022 15:43:22 -0800 Kees Cook <[email protected]> wrote:
From: Jann Horn <[email protected]>
Many Linux systems are configured to not panic on oops; but allowing an attacker to oops the system **really** often can make even bugs that look completely unexploitable exploitable (like NULL dereferences and such) if each crash elevates a refcount by one or a lock is taken in read mode, and this causes a counter to eventually overflow.
The most interesting counters for this are 32 bits wide (like open-coded refcounts that don't use
refcount_t). (The ldsem reader count on 32-bit platforms is just 16 bits, but probably nobody cares about 32-bit platforms that much nowadays.)So let's panic the system if the kernel is constantly oopsing.
The speed of oopsing 2^32 times probably depends on several factors, like how long the stack trace is and which unwinder you're using; an empirically important one is whether your console is showing a graphical environment or a text console that oopses will be printed to.
In a quick single-threaded benchmark, it looks like oopsing in avfork()child with a very short stack trace only takes ~510 microseconds per run when a graphical console is active; but switching to a text console that oopses are printed to slows it down around 87x, to ~45 milliseconds per run.
(Adding more threads makes this faster, but the actual oops printing happens under&die_lockon x86, so you can maybe speed this up by a factor of around 2 and then any further improvement gets eaten up by lock contention.)It looks like it would take around 8-12 days to overflow a 32-bit counter with repeated oopsing on a multi-core X86 system running a graphical environment; both me (in an X86 VM) and Seth (with a distro kernel on normal hardware in a standard configuration) got numbers in that ballpark.
Quoted 6 lines
12 days aren't *that* short on a desktop system, and you'd likely need much longer on a typical server system (assuming that people don't run graphical desktop environments on their servers), and this is a *very* noisy and violent approach to exploiting the kernel; and it also seems to take orders of magnitude longer on some machines, probably because stuff like EFI pstore will slow it down a ton if that's active.
I found a blog article[1] recommending LTS kernels to backport this as below.
While this patch is already upstream, it is important that distributed kernels also inherit this oops limit and backport it to LTS releases if we want to avoid treating such null-dereference bugs as full-fledged security issues in the future.
Do you have a plan to backport this into upstream LTS kernels?
[1] https://googleprojectzero.blogspot.com/2023/01/exploiting-null-dereferences-in-linux.html
Thanks,
SJ
Quoted 5 lines
Signed-off-by Jann Horn <[email protected]>Reviewed-by Luis Chamberlain <[email protected]>Signed-off-by Kees Cook <[email protected]>
Quoted 1 line
Do you have a plan to backport this into upstream LTS kernels?
As I understand, the answer is "hopefully yes" with the big presumption that all stakeholders are on board for the change. There is *definitely* a plan to *submit* backports to the stable trees, but ofc it will require some approvals.
Quoted 66 lines · SeongJae Park
Hello,
Quoted 34 lines · On Thu, 17 Nov 2022 15:43:22 -0800 Kees Cook <[email protected]> wrote:
From: Jann Horn <[email protected]>
Many Linux systems are configured to not panic on oops; but allowing an attacker to oops the system **really** often can make even bugs that look completely unexploitable exploitable (like NULL dereferences and such) if each crash elevates a refcount by one or a lock is taken in read mode, and this causes a counter to eventually overflow.
The most interesting counters for this are 32 bits wide (like open-coded refcounts that don't use
refcount_t). (The ldsem reader count on 32-bit platforms is just 16 bits, but probably nobody cares about 32-bit platforms that much nowadays.)So let's panic the system if the kernel is constantly oopsing.
The speed of oopsing 2^32 times probably depends on several factors, like how long the stack trace is and which unwinder you're using; an empirically important one is whether your console is showing a graphical environment or a text console that oopses will be printed to.
In a quick single-threaded benchmark, it looks like oopsing in avfork()child with a very short stack trace only takes ~510 microseconds per run when a graphical console is active; but switching to a text console that oopses are printed to slows it down around 87x, to ~45 milliseconds per run.
(Adding more threads makes this faster, but the actual oops printing happens under&die_lockon x86, so you can maybe speed this up by a factor of around 2 and then any further improvement gets eaten up by lock contention.)It looks like it would take around 8-12 days to overflow a 32-bit counter with repeated oopsing on a multi-core X86 system running a graphical environment; both me (in an X86 VM) and Seth (with a distro kernel on normal hardware in a standard configuration) got numbers in that ballpark.
Quoted 6 lines
12 days aren't *that* short on a desktop system, and you'd likely need much longer on a typical server system (assuming that people don't run graphical desktop environments on their servers), and this is a *very* noisy and violent approach to exploiting the kernel; and it also seems to take orders of magnitude longer on some machines, probably because stuff like EFI pstore will slow it down a ton if that's active.
I found a blog article[1] recommending LTS kernels to backport this as below.
While this patch is already upstream, it is important that distributed kernels also inherit this oops limit and backport it to LTS releases if we want to avoid treating such null-dereference bugs as full-fledged security issues in the future.
Do you have a plan to backport this into upstream LTS kernels?
[1] https://googleprojectzero.blogspot.com/2023/01/exploiting-null-dereferences-in-linux.html
Thanks,
SJQuoted 5 lines
Signed-off-by Jann Horn <[email protected]>Reviewed-by Luis Chamberlain <[email protected]>Signed-off-by Kees Cook <[email protected]>
Quoted 6 lines · Seth Jenkins
Quoted 1 line
Do you have a plan to backport this into upstream LTS kernels?
As I understand, the answer is "hopefully yes" with the big presumption that all stakeholders are on board for the change. There is *definitely* a plan to *submit* backports to the stable trees, but ofc it will require some approvals.
I've asked for at least v6.1.x (it's a clean cherry-pick). Earlier kernels will need some non-trivial backporting. Is there anyone that would be interested in stepping up to do that?
https://lore.kernel.org/lkml/202301191532.AEEC765@keescook
-Kees
Signature · 1 line
Kees Cook
Quoted 8 lines · Kees Cook
Quoted 6 lines · Seth Jenkins
Quoted 1 line
Do you have a plan to backport this into upstream LTS kernels?
As I understand, the answer is "hopefully yes" with the big presumption that all stakeholders are on board for the change. There is *definitely* a plan to *submit* backports to the stable trees, but ofc it will require some approvals.
Quoted 6 lines
I've asked for at least v6.1.x (it's a clean cherry-pick). Earlier kernels will need some non-trivial backporting. Is there anyone that would be interested in stepping up to do that?
I've sent out a backport to 5.15:
https://lore.kernel.org/stable/[email protected]/T/#t
- Eric
Quoted 13 lines · Eric Biggers
Quoted 12 lines · Kees Cook
Quoted 6 lines · Seth Jenkins
Quoted 1 line
Do you have a plan to backport this into upstream LTS kernels?
As I understand, the answer is "hopefully yes" with the big presumption that all stakeholders are on board for the change. There is *definitely* a plan to *submit* backports to the stable trees, but ofc it will require some approvals.
I've asked for at least v6.1.x (it's a clean cherry-pick). Earlier kernels will need some non-trivial backporting. Is there anyone that would be interested in stepping up to do that?
Quoted 5 lines
I've sent out a backport to 5.15:
https://lore.kernel.org/stable/[email protected]/T/#t
Also 5.10, which wasn't too hard after doing 5.15:
https://lore.kernel.org/stable/[email protected]/T/#t
- Eric
Kees Cook
maintainerQuoted 14 lines · Eric Biggers
Quoted 13 lines · Eric Biggers
Quoted 12 lines · Kees Cook
Quoted 6 lines · Seth Jenkins
Quoted 1 line
Do you have a plan to backport this into upstream LTS kernels?
As I understand, the answer is "hopefully yes" with the big presumption that all stakeholders are on board for the change. There is *definitely* a plan to *submit* backports to the stable trees, but ofc it will require some approvals.
I've asked for at least v6.1.x (it's a clean cherry-pick). Earlier kernels will need some non-trivial backporting. Is there anyone that would be interested in stepping up to do that?
Quoted 8 lines
Quoted 5 lines
I've sent out a backport to 5.15:
https://lore.kernel.org/stable/[email protected]/T/#tAlso 5.10, which wasn't too hard after doing 5.15:
https://lore.kernel.org/stable/[email protected]/T/#t
Oh excellent! Thank you very much!
-Kees
Signature · 1 line
Kees Cook
PATCH v3 3/6 exit: Expose "oops_count" to sysfs
In reply to Kees CookSince Oops count is now tracked and is a fairly interesting signal, add the entry /sys/kernel/oops_count to expose it to userspace.
--- .../ABI/testing/sysfs-kernel-oops_count | 6 +++++ MAINTAINERS | 1 + kernel/exit.c | 22 +++++++++++++++++-- 3 files changed, 27 insertions(+), 2 deletions(-) create mode 100644 Documentation/ABI/testing/sysfs-kernel-oops_count
Documentation/ABI/testing/sysfs-kernel-oops_count new file +6 -0
@@ -0,0 +1,6 @@ +What: /sys/kernel/oops_count +Date: November 2022 +KernelVersion: 6.2.0 +Contact: Linux Kernel Hardening List <[email protected]> +Description: + Shows how many times the system has Oopsed since last boot.
MAINTAINERS +1 -0
@@ -11106,6 +11106,7 @@ M: Kees Cook <[email protected]> L: [email protected] S: Supported T: git git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git for-next/hardening +F: Documentation/ABI/testing/sysfs-kernel-oops_count F: include/linux/overflow.h F: include/linux/randomize_kstack.h F: mm/usercopy.c
kernel/exit.c +20 -2
@@ -67,6 +67,7 @@#include <linux/io_uring.h> #include <linux/kprobes.h> #include <linux/rethook.h> +#include <linux/sysfs.h> #include <linux/uaccess.h> #include <asm/unistd.h>
@@ -99,6 +100,25 @@ static __init int kernel_exit_sysctls_init(void)late_initcall (kernel_exit_sysctls_init);#endif +static atomic_t oops_count =ATOMIC_INIT (0 ); + +#ifdef CONFIG_SYSFS +static ssize_t oops_count_show (struct kobject *kobj,struct kobj_attribute *attr, +char *page) +{ +return sysfs_emit (page,"%d\n" ,atomic_read (&oops_count)); +} + +static struct kobj_attribute oops_count_attr =__ATTR_RO (oops_count); + +static __init int kernel_exit_sysfs_init (void ) +{ +sysfs_add_file_to_group (kernel_kobj, &oops_count_attr.attr,NULL ); +return 0 ; +} +late_initcall (kernel_exit_sysfs_init); +#endif +static void __unhash_process (struct task_struct *p,bool group_dead) { nr_threads--;
@@ -901,8 +921,6 @@ void __noreturn do_exit(long code)void __noreturn make_task_dead (int signr) { -static atomic_t oops_count =ATOMIC_INIT (0 ); -/* * Take the task off the cpu after something catastrophic has * happened.
Signature · 1 line
2.34.1
PATCH v3 4/6 panic: Consolidate open-coded panic_on_warn checks
In reply to Kees CookSeveral run-time checkers (KASAN, UBSAN, KFENCE, KCSAN, sched) roll their own warnings, and each check "panic_on_warn". Consolidate this into a single function so that future instrumentation can be added in a single location.
--- include/linux/panic.h | 1 + kernel/kcsan/report.c | 3 +-- kernel/panic.c | 9 +++++++-- kernel/sched/core.c | 3 +-- lib/ubsan.c | 3 +-- mm/kasan/report.c | 4 ++-- mm/kfence/report.c | 3 +-- 7 files changed, 14 insertions(+), 12 deletions(-)
include/linux/panic.h +1 -0
@@ -11,6 +11,7 @@ extern long (*panic_blink)(int state); __printf(1, 2)void panic (const char *fmt, ...)__noreturn __cold ;void nmi_panic (struct pt_regs *regs,const char *msg); +void check_panic_on_warn (const char *origin);extern void oops_enter (void );extern void oops_exit (void );extern bool oops_may_print (void );
kernel/kcsan/report.c +1 -2
@@ -492,8 +492,7 @@ static void print_report(enum kcsan_value_change value_change,dump_stack_print_info (KERN_DEFAULT );pr_err ("==================================================================\n" ); -if (panic_on_warn) -panic ("panic_on_warn set ...\n" ); +check_panic_on_warn ("KCSAN" ); }static void release_report (unsigned long *flags,struct other_info *other_info)
kernel/panic.c +7 -2
@@ -201,6 +201,12 @@ static void panic_print_sys_info(bool console_flush)ftrace_dump (DUMP_ALL ); } +void check_panic_on_warn (const char *origin) +{ +if (panic_on_warn) +panic ("%s: panic_on_warn set ...\n" , origin); +} +/** * panic - halt the system * @fmt: The text string to print
@@ -619,8 +625,7 @@ void __warn(const char *file, int line, void *caller, unsigned taint,if (regs)show_regs (regs); -if (panic_on_warn) -panic ("panic_on_warn set ...\n" ); +check_panic_on_warn ("kernel" );if (!regs)dump_stack ();
kernel/sched/core.c +1 -2
@@ -5729,8 +5729,7 @@ static noinline void __schedule_bug(struct task_struct *prev)pr_err ("Preemption disabled at:" );print_ip_sym (KERN_ERR , preempt_disable_ip); } -if (panic_on_warn) -panic ("scheduling while atomic\n" ); +check_panic_on_warn ("scheduling while atomic" );dump_stack ();add_taint (TAINT_WARN ,LOCKDEP_STILL_OK );
lib/ubsan.c +1 -2
@@ -154,8 +154,7 @@ static void ubsan_epilogue(void) current->in_ubsan--; -if (panic_on_warn) -panic ("panic_on_warn set ...\n" ); +check_panic_on_warn ("UBSAN" ); }void __ubsan_handle_divrem_overflow (void *_data,void *lhs,void *rhs)
mm/kasan/report.c +2 -2
@@ -164,8 +164,8 @@ static void end_report(unsigned long *flags, void *addr) (unsigned long)addr);pr_err ("==================================================================\n" );spin_unlock_irqrestore (&report_lock, *flags); -if (panic_on_warn && !test_bit (KASAN_BIT_MULTI_SHOT , &kasan_flags)) -panic ("panic_on_warn set ...\n" ); +if (!test_bit (KASAN_BIT_MULTI_SHOT , &kasan_flags)) +check_panic_on_warn ("KASAN" );if (kasan_arg_fault ==KASAN_ARG_FAULT_PANIC )panic ("kasan.fault=panic set ...\n" );add_taint (TAINT_BAD_PAGE ,LOCKDEP_NOW_UNRELIABLE );
mm/kfence/report.c +1 -2
@@ -268,8 +268,7 @@ void kfence_report_error(unsigned long address, bool is_write, struct pt_regs *rlockdep_on (); -if (panic_on_warn) -panic ("panic_on_warn set ...\n" ); +check_panic_on_warn ("KFENCE" );/* We encountered a memory safety error, taint the kernel! */ add_taint (TAINT_BAD_PAGE ,LOCKDEP_STILL_OK );
Signature · 1 line
2.34.1
Re: [PATCH v3 4/6] panic: Consolidate open-coded panic_on_warn checks
Quoted 28 lines · Kees Cook
Several run-time checkers (KASAN, UBSAN, KFENCE, KCSAN, sched) roll their own warnings, and each check "
panic_on_warn". Consolidate this into a single function so that future instrumentation can be added in a single location.Cc Marco Elver <[email protected]>Cc Dmitry Vyukov <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Juri Lelli <[email protected]>Cc Vincent Guittot <[email protected]>Cc Dietmar Eggemann <[email protected]>Cc Steven Rostedt <[email protected]>Cc Ben Segall <[email protected]>Cc Mel Gorman <[email protected]>Cc Daniel Bristot de Oliveira <[email protected]>Cc Valentin Schneider <[email protected]>Cc Andrey Ryabinin <[email protected]>Cc Alexander Potapenko <[email protected]>Cc Andrey Konovalov <[email protected]>Cc Vincenzo Frascino <[email protected]>Cc Andrew Morton <[email protected]>Cc David Gow <[email protected]>Cc tangmeng <[email protected]>Cc Jann Horn <[email protected]>Cc Shuah Khan <[email protected]>Cc Petr Mladek <[email protected]>
Quoted 8 lines
Cc "Paul E. McKenney" <[email protected]>Cc Sebastian Andrzej Siewior <[email protected]>Cc "Guilherme G. Piccoli" <[email protected]>Cc Tiezhu Yang <[email protected]>Reviewed-by Luis Chamberlain <[email protected]>Signed-off-by Kees Cook <[email protected]>
Quoted 123 lines
--- include/linux/panic.h | 1 + kernel/kcsan/report.c | 3 +-- kernel/panic.c | 9 +++++++-- kernel/sched/core.c | 3 +-- lib/ubsan.c | 3 +-- mm/kasan/report.c | 4 ++-- mm/kfence/report.c | 3 +-- 7 files changed, 14 insertions(+), 12 deletions(-)include/linux/panic.h +1
-0@@ -11,6 +11,7 @@ extern long (*panic_blink)(int state); __printf(1, 2)void panic (const char *fmt, ...)__noreturn __cold ;void nmi_panic (struct pt_regs *regs,const char *msg); +void check_panic_on_warn (const char *origin);extern void oops_enter (void );extern void oops_exit (void );extern bool oops_may_print (void );kernel/kcsan/report.c +1
-2@@ -492,8 +492,7 @@ static void print_report(enum kcsan_value_change value_change,dump_stack_print_info (KERN_DEFAULT );pr_err ("==================================================================\n" ); -if (panic_on_warn) -panic ("panic_on_warn set ...\n" ); +check_panic_on_warn ("KCSAN" ); }static void release_report (unsigned long *flags,struct other_info *other_info)kernel/panic.c +7
-2@@ -201,6 +201,12 @@ static void panic_print_sys_info(bool console_flush)ftrace_dump (DUMP_ALL ); } +void check_panic_on_warn (const char *origin) +{ +if (panic_on_warn) +panic ("%s: panic_on_warn set ...\n" , origin); +} +/** * panic - halt the system * @fmt: The text string to print @@ -619,8 +625,7 @@ void __warn(const char *file, int line, void *caller, unsigned taint,if (regs)show_regs (regs); -if (panic_on_warn) -panic ("panic_on_warn set ...\n" ); +check_panic_on_warn ("kernel" );if (!regs)dump_stack ();kernel/sched/core.c +1
-2@@ -5729,8 +5729,7 @@ static noinline void __schedule_bug(struct task_struct *prev)pr_err ("Preemption disabled at:" );print_ip_sym (KERN_ERR , preempt_disable_ip); } -if (panic_on_warn) -panic ("scheduling while atomic\n" ); +check_panic_on_warn ("scheduling while atomic" );dump_stack ();add_taint (TAINT_WARN ,LOCKDEP_STILL_OK );lib/ubsan.c +1
-2@@ -154,8 +154,7 @@ static void ubsan_epilogue(void) current->in_ubsan--; -if (panic_on_warn) -panic ("panic_on_warn set ...\n" ); +check_panic_on_warn ("UBSAN" ); }void __ubsan_handle_divrem_overflow (void *_data,void *lhs,void *rhs)mm/kasan/report.c +2
-2@@ -164,8 +164,8 @@ static void end_report(unsigned long *flags, void *addr) (unsigned long)addr);pr_err ("==================================================================\n" );spin_unlock_irqrestore (&report_lock, *flags); -if (panic_on_warn && !test_bit (KASAN_BIT_MULTI_SHOT , &kasan_flags)) -panic ("panic_on_warn set ...\n" ); +if (!test_bit (KASAN_BIT_MULTI_SHOT , &kasan_flags)) +check_panic_on_warn ("KASAN" );if (kasan_arg_fault ==KASAN_ARG_FAULT_PANIC )panic ("kasan.fault=panic set ...\n" );add_taint (TAINT_BAD_PAGE ,LOCKDEP_NOW_UNRELIABLE );mm/kfence/report.c +1
-3@@ -268,8 +268,7 @@ void kfence_report_error(unsigned long address, bool is_write, struct pt_regs *rlockdep_on (); -if (panic_on_warn) -panic ("panic_on_warn set ...\n" ); +check_panic_on_warn ("KFENCE" );/* We encountered a memory safety error, taint the kernel! */ add_taint (TAINT_BAD_PAGE ,LOCKDEP_STILL_OK ); --2.34.1
Re: [PATCH v3 4/6] panic: Consolidate open-coded panic_on_warn checks
In reply to Kees CookQuoted 157 lines · Kees Cook
Several run-time checkers (KASAN, UBSAN, KFENCE, KCSAN, sched) roll their own warnings, and each check "
panic_on_warn". Consolidate this into a single function so that future instrumentation can be added in a single location.Cc Marco Elver <[email protected]>Cc Dmitry Vyukov <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Juri Lelli <[email protected]>Cc Vincent Guittot <[email protected]>Cc Dietmar Eggemann <[email protected]>Cc Steven Rostedt <[email protected]>Cc Ben Segall <[email protected]>Cc Mel Gorman <[email protected]>Cc Daniel Bristot de Oliveira <[email protected]>Cc Valentin Schneider <[email protected]>Cc Andrey Ryabinin <[email protected]>Cc Alexander Potapenko <[email protected]>Cc Andrey Konovalov <[email protected]>Cc Vincenzo Frascino <[email protected]>Cc Andrew Morton <[email protected]>Cc David Gow <[email protected]>Cc tangmeng <[email protected]>Cc Jann Horn <[email protected]>Cc Shuah Khan <[email protected]>Cc Petr Mladek <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Sebastian Andrzej Siewior <[email protected]>Cc "Guilherme G. Piccoli" <[email protected]>Cc Tiezhu Yang <[email protected]>Reviewed-by Luis Chamberlain <[email protected]>Signed-off-by Kees Cook <[email protected]>--- include/linux/panic.h | 1 + kernel/kcsan/report.c | 3 +-- kernel/panic.c | 9 +++++++-- kernel/sched/core.c | 3 +-- lib/ubsan.c | 3 +-- mm/kasan/report.c | 4 ++-- mm/kfence/report.c | 3 +-- 7 files changed, 14 insertions(+), 12 deletions(-)include/linux/panic.h +1
-0@@ -11,6 +11,7 @@ extern long (*panic_blink)(int state); __printf(1, 2)void panic (const char *fmt, ...)__noreturn __cold ;void nmi_panic (struct pt_regs *regs,const char *msg); +void check_panic_on_warn (const char *origin);extern void oops_enter (void );extern void oops_exit (void );extern bool oops_may_print (void );kernel/kcsan/report.c +1
-2@@ -492,8 +492,7 @@ static void print_report(enum kcsan_value_change value_change,dump_stack_print_info (KERN_DEFAULT );pr_err ("==================================================================\n" ); -if (panic_on_warn) -panic ("panic_on_warn set ...\n" ); +check_panic_on_warn ("KCSAN" ); }static void release_report (unsigned long *flags,struct other_info *other_info)kernel/panic.c +7
-2@@ -201,6 +201,12 @@ static void panic_print_sys_info(bool console_flush)ftrace_dump (DUMP_ALL ); } +void check_panic_on_warn (const char *origin) +{ +if (panic_on_warn) +panic ("%s: panic_on_warn set ...\n" , origin); +} +/** * panic - halt the system * @fmt: The text string to print @@ -619,8 +625,7 @@ void __warn(const char *file, int line, void *caller, unsigned taint,if (regs)show_regs (regs); -if (panic_on_warn) -panic ("panic_on_warn set ...\n" ); +check_panic_on_warn ("kernel" );if (!regs)dump_stack ();kernel/sched/core.c +1
-2@@ -5729,8 +5729,7 @@ static noinline void __schedule_bug(struct task_struct *prev)pr_err ("Preemption disabled at:" );print_ip_sym (KERN_ERR , preempt_disable_ip); } -if (panic_on_warn) -panic ("scheduling while atomic\n" ); +check_panic_on_warn ("scheduling while atomic" );dump_stack ();add_taint (TAINT_WARN ,LOCKDEP_STILL_OK );lib/ubsan.c +1
-2@@ -154,8 +154,7 @@ static void ubsan_epilogue(void) current->in_ubsan--; -if (panic_on_warn) -panic ("panic_on_warn set ...\n" ); +check_panic_on_warn ("UBSAN" ); }void __ubsan_handle_divrem_overflow (void *_data,void *lhs,void *rhs)mm/kasan/report.c +2
-2@@ -164,8 +164,8 @@ static void end_report(unsigned long *flags, void *addr) (unsigned long)addr);pr_err ("==================================================================\n" );spin_unlock_irqrestore (&report_lock, *flags); -if (panic_on_warn && !test_bit (KASAN_BIT_MULTI_SHOT , &kasan_flags)) -panic ("panic_on_warn set ...\n" ); +if (!test_bit (KASAN_BIT_MULTI_SHOT , &kasan_flags)) +check_panic_on_warn ("KASAN" );if (kasan_arg_fault ==KASAN_ARG_FAULT_PANIC )panic ("kasan.fault=panic set ...\n" );add_taint (TAINT_BAD_PAGE ,LOCKDEP_NOW_UNRELIABLE );mm/kfence/report.c +1
-3@@ -268,8 +268,7 @@ void kfence_report_error(unsigned long address, bool is_write, struct pt_regs *rlockdep_on (); -if (panic_on_warn) -panic ("panic_on_warn set ...\n" ); +check_panic_on_warn ("KFENCE" );/* We encountered a memory safety error, taint the kernel! */ add_taint (TAINT_BAD_PAGE ,LOCKDEP_STILL_OK ); --
Quoted 2 lines
2.34.1
PATCH v3 5/6 panic: Introduce warn_limit
In reply to Kees CookLike oops_limit, add warn_limit for limiting the number of warnings when panic_on_warn is not set.
--- Documentation/admin-guide/sysctl/kernel.rst | 9 +++++++++ kernel/panic.c | 14 ++++++++++++++ 2 files changed, 23 insertions(+)
Documentation/admin-guide/sysctl/kernel.rst +9 -0
@@ -1508,6 +1508,15 @@ entry will default to 2 instead of 0. 2 Unprivileged calls to ``bpf()`` are disabled = ============================================================= + +warn_limit +========== + +Number of kernel warnings after which the kernel should panic when +``panic_on_warn`` is not set. Setting this to 0 or 1 has the same effect +as setting ``panic_on_warn=1``. + + watchdog ========
kernel/panic.c +14 -0
@@ -58,6 +58,7 @@ bool crash_kexec_post_notifiers;int panic_on_warn__read_mostly ;unsigned long panic_on_taint;bool panic_on_taint_nousertaint = false; +static unsigned int warn_limit__read_mostly =10000 ;int panic_timeout =CONFIG_PANIC_TIMEOUT ;EXPORT_SYMBOL_GPL (panic_timeout);
@@ -88,6 +89,13 @@ static struct ctl_table kern_panic_table[] = { .extra2 = SYSCTL_ONE, },#endif + { + .procname = "warn_limit", + .data = &warn_limit, + .maxlen = sizeof(warn_limit), + .mode = 0644, + .proc_handler = proc_douintvec, + }, { } };
@@ -203,8 +211,14 @@ static void panic_print_sys_info(bool console_flush)void check_panic_on_warn (const char *origin) { +static atomic_t warn_count =ATOMIC_INIT (0 ); +if (panic_on_warn)panic ("%s: panic_on_warn set ...\n" , origin); + +if (atomic_inc_return (&warn_count) >=READ_ONCE (warn_limit)) +panic ("%s: system warned too often (kernel.warn_limit is %d)" , + warn_limit); }/**
Signature · 1 line
2.34.1
Re: [PATCH v3 5/6] panic: Introduce warn_limit
Quoted 70 lines · Kees Cook
Like
oops_limit, addwarn_limitfor limiting the number of warnings whenpanic_on_warnis not set.Cc Jonathan Corbet <[email protected]>Cc Andrew Morton <[email protected]>Cc Baolin Wang <[email protected]>Cc "Jason A. Donenfeld" <[email protected]>Cc Eric Biggers <[email protected]>Cc Huang Ying <[email protected]>Cc Petr Mladek <[email protected]>Cc tangmeng <[email protected]>Cc "Guilherme G. Piccoli" <[email protected]>Cc Tiezhu Yang <[email protected]>Cc Sebastian Andrzej Siewior <[email protected]>Reviewed-by Luis Chamberlain <[email protected]>Signed-off-by Kees Cook <[email protected]>--- Documentation/admin-guide/sysctl/kernel.rst | 9 +++++++++ kernel/panic.c | 14 ++++++++++++++ 2 files changed, 23 insertions(+)Documentation/admin-guide/sysctl/kernel.rst +9
-0@@ -1508,6 +1508,15 @@ entry will default to 2 instead of 0. 2 Unprivileged calls to ``bpf()`` are disabled = ============================================================= + +warn_limit +========== + +Number of kernel warnings after which the kernel should panic when +``panic_on_warn`` is not set. Setting this to 0 or 1 has the same effect +as setting ``panic_on_warn=1``. + + watchdog ========kernel/panic.c +8
-0@@ -58,6 +58,7 @@ bool crash_kexec_post_notifiers;int panic_on_warn__read_mostly ;unsigned long panic_on_taint;bool panic_on_taint_nousertaint = false; +static unsigned int warn_limit__read_mostly =10000 ;int panic_timeout =CONFIG_PANIC_TIMEOUT ;EXPORT_SYMBOL_GPL (panic_timeout);@@ -88,6 +89,13 @@ static struct ctl_table kern_panic_table[] = { .extra2 = SYSCTL_ONE, },#endif + { + .procname = "warn_limit", + .data = &warn_limit, + .maxlen = sizeof(warn_limit), + .mode = 0644, + .proc_handler = proc_douintvec, + }, { } };@@ -203,8 +211,14 @@ static void panic_print_sys_info(bool console_flush)
kernel/panic.c static void panic_print_sys_info(bool console_flush) · line 221
Quoted 10 lines
kernel/panic.c +6
-0void check_panic_on_warn (const char *origin) { +static atomic_t warn_count =ATOMIC_INIT (0 ); +if (panic_on_warn)panic ("%s: panic_on_warn set ...\n" , origin); + +if (atomic_inc_return (&warn_count) >=READ_ONCE (warn_limit)) +panic ("%s: system warned too often (kernel.warn_limit is %d)" , + warn_limit);
Bah. This should be: origin, warn_limit.
Signature · 1 line
Kees Cook
PATCH v3 6/6 panic: Expose "warn_count" to sysfs
In reply to Kees CookSince Warn count is now tracked and is a fairly interesting signal, add the entry /sys/kernel/warn_count to expose it to userspace.
--- .../ABI/testing/sysfs-kernel-warn_count | 6 +++++ MAINTAINERS | 1 + kernel/panic.c | 22 +++++++++++++++++-- 3 files changed, 27 insertions(+), 2 deletions(-) create mode 100644 Documentation/ABI/testing/sysfs-kernel-warn_count
Documentation/ABI/testing/sysfs-kernel-warn_count new file +6 -0
@@ -0,0 +1,6 @@ +What: /sys/kernel/oops_count +Date: November 2022 +KernelVersion: 6.2.0 +Contact: Linux Kernel Hardening List <[email protected]> +Description: + Shows how many times the system has Warned since last boot.
MAINTAINERS +1 -0
@@ -11107,6 +11107,7 @@ L: [email protected] S: Supported T: git git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git for-next/hardening F: Documentation/ABI/testing/sysfs-kernel-oops_count +F: Documentation/ABI/testing/sysfs-kernel-warn_count F: include/linux/overflow.h F: include/linux/randomize_kstack.h F: mm/usercopy.c
kernel/panic.c +20 -2
@@ -32,6 +32,7 @@#include <linux/bug.h> #include <linux/ratelimit.h> #include <linux/debugfs.h> +#include <linux/sysfs.h> #include <trace/events/error_report.h> #include <asm/sections.h>
@@ -107,6 +108,25 @@ static __init int kernel_panic_sysctls_init(void)late_initcall (kernel_panic_sysctls_init);#endif +static atomic_t warn_count =ATOMIC_INIT (0 ); + +#ifdef CONFIG_SYSFS +static ssize_t warn_count_show (struct kobject *kobj,struct kobj_attribute *attr, +char *page) +{ +return sysfs_emit (page,"%d\n" ,atomic_read (&warn_count)); +} + +static struct kobj_attribute warn_count_attr =__ATTR_RO (warn_count); + +static __init int kernel_panic_sysfs_init (void ) +{ +sysfs_add_file_to_group (kernel_kobj, &warn_count_attr.attr,NULL ); +return 0 ; +} +late_initcall (kernel_panic_sysfs_init); +#endif +static long no_blink (int state) {return 0 ;
@@ -211,8 +231,6 @@ static void panic_print_sys_info(bool console_flush)void check_panic_on_warn (const char *origin) { -static atomic_t warn_count =ATOMIC_INIT (0 ); -if (panic_on_warn)panic ("%s: panic_on_warn set ...\n" , origin);
Signature · 1 line
2.34.1