Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF5FEECAAD3 for ; Thu, 1 Sep 2022 23:09:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233501AbiIAXJC (ORCPT ); Thu, 1 Sep 2022 19:09:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56192 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233295AbiIAXJB (ORCPT ); Thu, 1 Sep 2022 19:09:01 -0400 Received: from smtp-fw-80006.amazon.com (smtp-fw-80006.amazon.com [99.78.197.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A169B7B783 for ; Thu, 1 Sep 2022 16:08:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1662073740; x=1693609740; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=vd4ox5QrAJBFsfY4S7WjRffr/Dgr7ng7C7plv7yCvC8=; b=J/a9H+8bYYGE3sTVyDl2lR+P1NTlFDIwBxaKvEKdTKi9I9arIZOUhJYA c0+Tu6U5Vfrbs6ZBGDSkcIFXVGFUpaAJobIQ1GshbWQjbSVES7qMVxYnn 6/fht0QuRRZ07WNDKj7q/lPD/vYmQ4CpheOrygxSXWBp1bp7y7Sb0hRU/ U=; X-IronPort-AV: E=Sophos;i="5.93,281,1654560000"; d="scan'208";a="125876936" Received: from pdx4-co-svc-p1-lb2-vlan2.amazon.com (HELO email-inbound-relay-pdx-2c-7d0c7241.us-west-2.amazon.com) ([10.25.36.210]) by smtp-border-fw-80006.pdx80.corp.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Sep 2022 23:08:59 +0000 Received: from EX13MTAUWB001.ant.amazon.com (pdx1-ws-svc-p6-lb9-vlan3.pdx.amazon.com [10.236.137.198]) by email-inbound-relay-pdx-2c-7d0c7241.us-west-2.amazon.com (Postfix) with ESMTPS id EEEFF4513F; Thu, 1 Sep 2022 23:08:58 +0000 (UTC) Received: from EX19D004ANA001.ant.amazon.com (10.37.240.138) by EX13MTAUWB001.ant.amazon.com (10.43.161.207) with Microsoft SMTP Server (TLS) id 15.0.1497.38; Thu, 1 Sep 2022 23:08:55 +0000 Received: from 88665a182662.ant.amazon.com (10.43.161.228) by EX19D004ANA001.ant.amazon.com (10.37.240.138) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1118.12; Thu, 1 Sep 2022 23:08:52 +0000 From: Kuniyuki Iwashima To: CC: , , , , , , , , Subject: Re: [PATCH stable 5.4 5.10 5.15] kprobes: don't call disarm_kprobe() for disabled kprobes Date: Thu, 1 Sep 2022 16:08:44 -0700 Message-ID: <20220901230844.18015-1-kuniyu@amazon.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220901223205.11648-1-kuniyu@amazon.com> References: <20220901223205.11648-1-kuniyu@amazon.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.43.161.228] X-ClientProxiedBy: EX13D03UWA001.ant.amazon.com (10.43.160.141) To EX19D004ANA001.ant.amazon.com (10.37.240.138) Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org This can be applied to 4.19 too. Could you queue this up for 4.19 as well? Thank you. From: Kuniyuki Iwashima Date: Thu, 1 Sep 2022 22:32:05 +0000 > commit 9c80e79906b4ca440d09e7f116609262bb747909 upstream. > > The assumption in __disable_kprobe() is wrong, and it could try to disarm > an already disarmed kprobe and fire the WARN_ONCE() below. [0] We can > easily reproduce this issue. > > 1. Write 0 to /sys/kernel/debug/kprobes/enabled. > > # echo 0 > /sys/kernel/debug/kprobes/enabled > > 2. Run execsnoop. At this time, one kprobe is disabled. > > # /usr/share/bcc/tools/execsnoop & > [1] 2460 > PCOMM PID PPID RET ARGS > > # cat /sys/kernel/debug/kprobes/list > ffffffff91345650 r __x64_sys_execve+0x0 [FTRACE] > ffffffff91345650 k __x64_sys_execve+0x0 [DISABLED][FTRACE] > > 3. Write 1 to /sys/kernel/debug/kprobes/enabled, which changes > kprobes_all_disarmed to false but does not arm the disabled kprobe. > > # echo 1 > /sys/kernel/debug/kprobes/enabled > > # cat /sys/kernel/debug/kprobes/list > ffffffff91345650 r __x64_sys_execve+0x0 [FTRACE] > ffffffff91345650 k __x64_sys_execve+0x0 [DISABLED][FTRACE] > > 4. Kill execsnoop, when __disable_kprobe() calls disarm_kprobe() for the > disabled kprobe and hits the WARN_ONCE() in __disarm_kprobe_ftrace(). > > # fg > /usr/share/bcc/tools/execsnoop > ^C > > Actually, WARN_ONCE() is fired twice, and __unregister_kprobe_top() misses > some cleanups and leaves the aggregated kprobe in the hash table. Then, > __unregister_trace_kprobe() initialises tk->rp.kp.list and creates an > infinite loop like this. > > aggregated kprobe.list -> kprobe.list -. > ^ | > '.__.' > > In this situation, these commands fall into the infinite loop and result > in RCU stall or soft lockup. > > cat /sys/kernel/debug/kprobes/list : show_kprobe_addr() enters into the > infinite loop with RCU. > > /usr/share/bcc/tools/execsnoop : warn_kprobe_rereg() holds kprobe_mutex, > and __get_valid_kprobe() is stuck in > the loop. > > To avoid the issue, make sure we don't call disarm_kprobe() for disabled > kprobes. > > [0] > Failed to disarm kprobe-ftrace at __x64_sys_execve+0x0/0x40 (error -2) > WARNING: CPU: 6 PID: 2460 at kernel/kprobes.c:1130 __disarm_kprobe_ftrace.isra.19 (kernel/kprobes.c:1129) > Modules linked in: ena > CPU: 6 PID: 2460 Comm: execsnoop Not tainted 5.19.0+ #28 > Hardware name: Amazon EC2 c5.2xlarge/, BIOS 1.0 10/16/2017 > RIP: 0010:__disarm_kprobe_ftrace.isra.19 (kernel/kprobes.c:1129) > Code: 24 8b 02 eb c1 80 3d c4 83 f2 01 00 75 d4 48 8b 75 00 89 c2 48 c7 c7 90 fa 0f 92 89 04 24 c6 05 ab 83 01 e8 e4 94 f0 ff <0f> 0b 8b 04 24 eb b1 89 c6 48 c7 c7 60 fa 0f 92 89 04 24 e8 cc 94 > RSP: 0018:ffff9e6ec154bd98 EFLAGS: 00010282 > RAX: 0000000000000000 RBX: ffffffff930f7b00 RCX: 0000000000000001 > RDX: 0000000080000001 RSI: ffffffff921461c5 RDI: 00000000ffffffff > RBP: ffff89c504286da8 R08: 0000000000000000 R09: c0000000fffeffff > R10: 0000000000000000 R11: ffff9e6ec154bc28 R12: ffff89c502394e40 > R13: ffff89c502394c00 R14: ffff9e6ec154bc00 R15: 0000000000000000 > FS: 00007fe800398740(0000) GS:ffff89c812d80000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 000000c00057f010 CR3: 0000000103b54006 CR4: 00000000007706e0 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > PKRU: 55555554 > Call Trace: > > __disable_kprobe (kernel/kprobes.c:1716) > disable_kprobe (kernel/kprobes.c:2392) > __disable_trace_kprobe (kernel/trace/trace_kprobe.c:340) > disable_trace_kprobe (kernel/trace/trace_kprobe.c:429) > perf_trace_event_unreg.isra.2 (./include/linux/tracepoint.h:93 kernel/trace/trace_event_perf.c:168) > perf_kprobe_destroy (kernel/trace/trace_event_perf.c:295) > _free_event (kernel/events/core.c:4971) > perf_event_release_kernel (kernel/events/core.c:5176) > perf_release (kernel/events/core.c:5186) > __fput (fs/file_table.c:321) > task_work_run (./include/linux/sched.h:2056 (discriminator 1) kernel/task_work.c:179 (discriminator 1)) > exit_to_user_mode_prepare (./include/linux/resume_user_mode.h:49 kernel/entry/common.c:169 kernel/entry/common.c:201) > syscall_exit_to_user_mode (./arch/x86/include/asm/jump_label.h:55 ./arch/x86/include/asm/nospec-branch.h:384 ./arch/x86/include/asm/entry-common.h:94 kernel/entry/common.c:133 kernel/entry/common.c:296) > do_syscall_64 (arch/x86/entry/common.c:87) > entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) > RIP: 0033:0x7fe7ff210654 > Code: 15 79 89 20 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb be 0f 1f 00 8b 05 9a cd 20 00 48 63 ff 85 c0 75 11 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 3a f3 c3 48 83 ec 18 48 89 7c 24 08 e8 34 fc > RSP: 002b:00007ffdbd1d3538 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 > RAX: 0000000000000000 RBX: 0000000000000008 RCX: 00007fe7ff210654 > RDX: 0000000000000000 RSI: 0000000000002401 RDI: 0000000000000008 > RBP: 0000000000000000 R08: 94ae31d6fda838a4 R0900007fe8001c9d30 > R10: 00007ffdbd1d34b0 R11: 0000000000000246 R12: 00007ffdbd1d3600 > R13: 0000000000000000 R14: fffffffffffffffc R15: 00007ffdbd1d3560 > > > Link: https://lkml.kernel.org/r/20220813020509.90805-1-kuniyu@amazon.com > Fixes: 69d54b916d83 ("kprobes: makes kprobes/enabled works correctly for optimized kprobes.") > Signed-off-by: Kuniyuki Iwashima > Reported-by: Ayushman Dutta > Cc: "Naveen N. Rao" > Cc: Anil S Keshavamurthy > Cc: "David S. Miller" > Cc: Masami Hiramatsu > Cc: Wang Nan > Cc: Kuniyuki Iwashima > Cc: Kuniyuki Iwashima > Cc: Ayushman Dutta > Cc: > Signed-off-by: Andrew Morton > --- > kernel/kprobes.c | 9 +++++---- > 1 file changed, 5 insertions(+), 4 deletions(-) > > diff --git a/kernel/kprobes.c b/kernel/kprobes.c > index 3a3c0166bd1f..ed3f24a81549 100644 > --- a/kernel/kprobes.c > +++ b/kernel/kprobes.c > @@ -1705,11 +1705,12 @@ static struct kprobe *__disable_kprobe(struct kprobe *p) > /* Try to disarm and disable this/parent probe */ > if (p == orig_p || aggr_kprobe_disabled(orig_p)) { > /* > - * If kprobes_all_disarmed is set, orig_p > - * should have already been disarmed, so > - * skip unneed disarming process. > + * Don't be lazy here. Even if 'kprobes_all_disarmed' > + * is false, 'orig_p' might not have been armed yet. > + * Note arm_all_kprobes() __tries__ to arm all kprobes > + * on the best effort basis. > */ > - if (!kprobes_all_disarmed) { > + if (!kprobes_all_disarmed && !kprobe_disabled(orig_p)) { > ret = disarm_kprobe(orig_p, true); > if (ret) { > p->flags &= ~KPROBE_FLAG_DISABLED; > -- > 2.37.1 >