Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D748713B59B for ; Sun, 20 Apr 2025 15:19:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745162369; cv=none; b=jAj8WCDUR+zvSHOO2onbShm9r+e5Hx6V09aV6rhE1ejaghCPxDuM5PnR/APhkFKoOMV7LCHN8//3/AIiuRcMKDRMkTbse4cDuWXp/qU+RNYylK2rXFlYfg+ssbf24QUe5gFQjIjb853nlUbjC0R3bjf7I4Quga5H9f4kPmNqYoM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745162369; c=relaxed/simple; bh=VTfm+lb/GucPpDaGYxeCG2VwmOg+MyGsXFXU8YjGR7Q=; h=Message-ID:Date:MIME-Version:Subject:To:References:Cc:From: In-Reply-To:Content-Type; b=dl3O/XiYdAWbxovk9AzBYjkcmNnCNS1KeyD4RUA0TDN4YlBhELdq8hm+pGZt0+1iq62l6BiyzM4LJU/XMU+dmTK3oG4veiWEHSafbu1gNzVN5VHY6OeZQtNQz3zxNalLmkHxnEAGHKjohuRPChXimqa7x6ft/m1EcwNgXQhuJ5c= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=dzg08OVQ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="dzg08OVQ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6D4A1C4CEE2; Sun, 20 Apr 2025 15:19:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745162369; bh=VTfm+lb/GucPpDaGYxeCG2VwmOg+MyGsXFXU8YjGR7Q=; h=Date:Subject:To:References:Cc:From:In-Reply-To:From; b=dzg08OVQnvbRNOAtzd3hVT30qrsjsvpnrLHZZKrkAFYwgBxzf9QPQuHhpTwKiNBYZ wkMYEQ8OW7V642gnafbcifcWWTX0bLBtYBQcNcZhgzxFh4b2nlAsLV6+A4IcskBZLI uaH9mkFk29oStc+JgOuSCPgJfO7gBQKL07kBWUfXGqXTuIkztVbGGpsVkPhcW1tloE ybkjUzP/ARibhO0oIGMdHWQ6XlV7C8tPLl4vrkCCUszOITEgt8xCg/+P5q03ICXwEC s8Fhgjka543NPiBAof1EJ2ORJzU5/X4C+bsTg2W2eJmz6UhHDVrLS9+qO4sXSa6dFA ZNQSe40Utddzg== Message-ID: Date: Sun, 20 Apr 2025 11:19:28 -0400 Precedence: bulk X-Mailing-List: kdevops@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 0/5] crash: provide a crash watchdog To: Luis Chamberlain References: <20250420054822.533987-1-mcgrof@kernel.org> Content-Language: en-US Cc: kdevops@lists.linux.dev, Daniel Gomez From: Chuck Lever Organization: kernel.org In-Reply-To: <20250420054822.533987-1-mcgrof@kernel.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit On 4/20/25 1:48 AM, Luis Chamberlain wrote: > One of the biggest pains we've suffered with CIs has been > crashes, filesystem corruptions. Although we have a feature > which is *supposed* to do that, its obviosly not working. > Fix this by adding support for this stuff, and leveraging for > our CIs. We will start with our fstests CIs and can scale this > out to the other ones easily now. Hi Luis - I haven't looked at the crash detection/reporting logic before. Where can I find documentation for developers who want to add this support to their workflows, or for test engineers who want to enable it? Once I'm a little more educated, my review perspective will be thinking about how this works for cloud kdevops. > Luis Chamberlain (5): > systemd-remote: use ip address for systemd-remote journal > crash: add kernel crash watchdog library > fstests_watchdog.py: use the new crash watchdog library > crash_watchdog.py: add generic crash watchdog > crash_report.py: add a crash report > > scripts/workflows/fstests/fstests_watchdog.py | 89 ++- > scripts/workflows/generic/crash_report.py | 109 +++ > scripts/workflows/generic/crash_watchdog.py | 186 +++++ > scripts/workflows/generic/get_console.py | 1 + > scripts/workflows/generic/lib | 1 + > scripts/workflows/lib/crash.py | 724 ++++++++++++++++++ > scripts/workflows/lib/systemd_remote.py | 19 +- > 7 files changed, 1081 insertions(+), 48 deletions(-) > create mode 100755 scripts/workflows/generic/crash_report.py > create mode 100755 scripts/workflows/generic/crash_watchdog.py > create mode 120000 scripts/workflows/generic/get_console.py > create mode 120000 scripts/workflows/generic/lib > create mode 100755 scripts/workflows/lib/crash.py > -- Chuck Lever