fstests: integrate with coredump capturing
PATCH 3/3 fstests: integrate with coredump capturing
Hi all,
Integrate fstests with coredump capturing tools such as systemd-coredump.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
With a bit of luck, this should all go splendidly. Comments and questions are, as always, welcome.
--D
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=coredump-capture
---
Commits in this patchset:
* fsstress: don't abort when stat(".") returns EIO
* check: collect core dumps from systemd-coredump
--- check | 2 ++ common/rc | 44 ++++++++++++++++++++++++++++++++++++++++++++ ltp/fsstress.c | 15 ++++++++++++++- 3 files changed, 60 insertions(+), 1 deletion(-)
PATCH 1/2 fsstress: don't abort when stat(".") returns EIO
From: Darrick J. Wong <[email protected]>
First, start with the premise that fstests is run with a nonzero limit on the size of core dumps so that we can capture the state of misbehaving fs utilities like fsck and scrub if they crash.
When fsstress is compiled with DEBUG defined (which is the default), it will periodically call check_cwd to ensure that the current working directory hasn't changed out from underneath it.
If the filesystem is XFS and it shuts down, the stat64() calls will start returning EIO. In this case, we follow the out: label and call abort() to exit the program. Historically this did not produce any core dumps because $PWD is on the dead filesystem and the write fails.
However, modern systems are often configured to capture coredumps using
some external mechanism, e.g. abrt/systemd-coredump. In this case, the
capture tool will succeeds in capturing every crashed process, which fills the crash dump directory with a lot of useless junk. Worse, if the capture tool is configured to pass the dumps to fstests, it will flag the test as failed because something dumped core.
This is really silly, because basic stat requests for the current working directory can be satisfied from the inode cache without a disk access. In this narrow situation, EIO only happens when the fs has shut down, so just exit the program.
We really should have a way to query if a filesystem is shut down that isn't conflated with (possibly transient) EIO errors. But for now this is what we have to do. :(
--- ltp/fsstress.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
ltp/fsstress.c +14 -1
@@ -1049,8 +1049,21 @@ check_cwd(void) ret =stat64 ("." , &statbuf);if (ret !=0 ) { +int error = errno; +fprintf (stderr,"fsstress: check_cwd stat64() returned %d with errno: %d (%s)\n" , - ret, errno,strerror (errno)); + ret, error,strerror (error)); + +/* +* The current working directory is pinned in memory, which +* means that stat should not have had to do any disk accesses +* to retrieve stat information. Treat an EIO as an indication +* that the filesystem shut down and exit instead of dumping +* core like the abort() below does. +*/ +if (error ==EIO ) +exit (1 ); + goto out; }
Re: [PATCH 1/2] fsstress: don't abort when stat(".") returns EIO
Quoted 5 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
First, start with the premise that fstests is run with a nonzero limit on the size of core dumps so that we can capture the state of misbehaving fs utilities like fsck and scrub if they crash.
Can you explain what this has to do with core dumping?
I'm just really confused between this patch content and the subject of this patch and the entire series..
Quoted 4 lines
This is really silly, because basic stat requests for the current working directory can be satisfied from the inode cache without a disk access. In this narrow situation, EIO only happens when the fs has shut down, so just exit the program.
If we think it's silly we can trivially drop the xfs_is_shutdown check in xfs_vn_getattr. But is it really silly? We've tried to basically make every file system operation consistently fail on shut down file systems,
Quoted 3 lines
We really should have a way to query if a filesystem is shut down that isn't conflated with (possibly transient) EIO errors. But for now this is what we have to do. :(
Well, a new STATX_ flag would work, assuming stat doesn't actually fail :) Otherwise a new ioctl/fcntl would make sense, especially as the shutdown concept has spread beyond XFS.
Re: [PATCH 1/2] fsstress: don't abort when stat(".") returns EIO
Quoted 11 lines · Christoph Hellwig
Quoted 5 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
First, start with the premise that fstests is run with a nonzero limit on the size of core dumps so that we can capture the state of misbehaving fs utilities like fsck and scrub if they crash.
Can you explain what this has to do with core dumping?
I'm just really confused between this patch content and the subject of this patch and the entire series..
It's a bugfix ahead of new behaviors introduced in patch 2. I clearly didn't explain this well enough, so I'll try again.
Before abrt/systemd-coredump, FS_IOC_SHUTDOWN fsstress tests would do something like the following:
1. start fsstress, which chdirs to $TEST_DIR
2. shut down the filesystem
3. fsstress tries to stat($TEST_DIR), fails, and calls abort
4. abort triggers coredump
5. kernel fails to write "core" to $TEST_DIR (because fs is shut down)
6. test finishes, no core files written to $here, test passes
Once you install systemd-coredump, that changes to:
same 1-4 above
5. kernel pipes core file to coredumpctl, which writes it to /var/crash
6. test finishes, no core files written to $here, test passes
And then with patch 2 of this series, that becomes:
same 1-4 above
5. kernel pipes core file to coredumpctl, which writes it to /var/crash
6. test finishes, ./check queries coredumpctl for any new coredumps, and copies them to $here
7. ./check finds core files written to $here, test fails
Now we've caused a test failure where there was none before, simply because the crash reporting improved.
Therefore this patch changes fsstress not to call abort() from check_cwd when it has a reasonable suspicion that the fs has died.
(Did that help? /me is still pre-coffee...)
Quoted 9 lines
Quoted 4 lines
This is really silly, because basic stat requests for the current working directory can be satisfied from the inode cache without a disk access. In this narrow situation, EIO only happens when the fs has shut down, so just exit the program.
If we think it's silly we can trivially drop the
xfs_is_shutdowncheck inxfs_vn_getattr. But is it really silly? We've tried to basically make every file system operation consistently fail on shut down file systems,
No no, "really silly" refers to failing tests that we didn't used to fail.
Quoted 7 lines
Quoted 3 lines
We really should have a way to query if a filesystem is shut down that isn't conflated with (possibly transient) EIO errors. But for now this is what we have to do. :(
Well, a new
STATX_flag would work, assuming stat doesn't actually fail :) Otherwise a new ioctl/fcntl would make sense, especially as the shutdown concept has spread beyond XFS.
I think we ought to add a new ioctl or something so that callers can positively identify a shut down filesystem. bfoster I think was asking about that for fstests some years back, and ended up coding a bunch of grep heuristics to work around the lack of a real call.
I think we can't drop the "stat{,x} returns EIO on shutdown fs" behavior because I know of a few, uh, users whose heartbeat monitor periodically queries statx($PWD) and reboots the node if it returns errno.
--D
PATCH 2/2 check: collect core dumps from systemd-coredump
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
On modern RHEL (>=8) and Debian KDE systems, systemd-coredump can be installed to capture core dumps from crashed programs. If this is the case, we would like to capture core dumps from programs that crash during the test. Set up an (admittedly overwrought) pipeline to extract dumps created during the test and then capture them the same way that we pick up "core" and "core.$pid" files.
--- check | 2 ++ common/rc | 44 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+)
check +2 -0
@@ -924,6 +924,7 @@ function run_section() $1 == "'$seqnum'" {lasttime=" " $2 "s ... "; exit} \ END {printf "%s", lasttime}' "$check.time" rm -f core $seqres.notrun + _start_coredumpctl_collection start=`_wallclock` $timestamp && _timestamp
@@ -957,6 +958,7 @@ function run_section() # just "core". Use globbing to find the most common patterns, # assuming there are no other coredump capture packages set up. local cores=0 + _finish_coredumpctl_collection for i in core core.*; do test -f "$i" || continue if ((cores++ == 0)); then
common/rc +44 -0
50 lines not on this page · raw message
Re: [PATCH 2/2] check: collect core dumps from systemd-coredump
Quoted 46 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
On modern RHEL (>=8) and Debian KDE systems, systemd-coredump can be installed to capture core dumps from crashed programs. If this is the case, we would like to capture core dumps from programs that crash during the test. Set up an (admittedly overwrought) pipeline to extract dumps created during the test and then capture them the same way that we pick up "core" and "core.$pid" files.
Signed-off-by "Darrick J. Wong" <[email protected]>--- check | 2 ++ common/rc | 44 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+)check +2
-0@@ -924,6 +924,7 @@ function run_section() $1 == "'$seqnum'" {lasttime=" " $2 "s ... "; exit} \ END {printf "%s", lasttime}' "$check.time" rm -f core $seqres.notrun + _start_coredumpctl_collection start=`_wallclock` $timestamp && _timestamp@@ -957,6 +958,7 @@ function run_section() # just "core". Use globbing to find the most common patterns, # assuming there are no other coredump capture packages set up. local cores=0 + _finish_coredumpctl_collection for i in core core.*; do test -f "$i" || continue if ((cores++ == 0)); thencommon/rc +2
-0@@ -5034,6 +5034,50 @@ _check_kmemleak() fi } +# Current timestamp, in a format that systemd likes +_systemd_now() {
common/rc _check_kmemleak() · line 5048
Quoted 10 lines
common/rc +10
-0+ timedatectl show --property=TimeUSec --value +} + +# Do what we need to do to capture core dumps from coredumpctl +_start_coredumpctl_collection() { + command -v coredumpctl &>/dev/null || return + command -v timedatectl &>/dev/null || return + command -v jq &>/dev/null || return + + sysctl kernel.core_pattern | grep -q systemd-coredump || return
# rpm -qf `which coredumpctl` systemd-udev-252-53.el9.x86_64 # rpm -qf `which timedatectl` systemd-252-53.el9.x86_64 # rpm -qf `which jq` jq-1.6-17.el9.x86_64 # rpm -qf /usr/lib/systemd/systemd-coredump systemd-udev-252-53.el9.x86_64
So we have 3 optional running dependences, how about metion that in README?
Thanks,
Zorro
Quoted 36 lines
common/rc +32
-0+ COREDUMPCTL_START_TIMESTAMP="$(_systemd_now)" +} + +# Capture core dumps from coredumpctl. +# +# coredumpctl list only supports json output as a machine-readable format. The +# human-readable format intermingles spaces from the timestamp with actual +# column separators, so we cannot parse that sanely. The json output is an +# array of: +# { +# "time" : 1749744847150926, +# "pid" : 2297, +# "uid" : 0, +# "gid" : 0, +# "sig" : 6, +# "corefile" : "present", +# "exe" : "/run/fstests/e2fsprogs/fuse2fs", +# "size" : 47245 +# }, +# So we use jq to filter out lost corefiles, then print the pid and exe +# separated by a pipe and hope that nobody ever puts a pipe in an executable +# name. +_finish_coredumpctl_collection() { + test -n "$COREDUMPCTL_START_TIMESTAMP" || return + + coredumpctl list --since="$COREDUMPCTL_START_TIMESTAMP" --json=short 2>/dev/null | \ + jq --raw-output 'map(select(.corefile == "present")) | map("\(.pid)|\(.exe)") | .[]' | while IFS='|' read pid exe; do + test -e "core.$pid" || coredumpctl dump --output="core.$pid" "$pid" "$exe" &>> $seqres.full + done + unset COREDUMPCTL_START_TIMESTAMP +} + # don't check dmesg log after test _disable_dmesg_check() {
Re: [PATCH 2/2] check: collect core dumps from systemd-coredump
Quoted 67 lines · Zorro Lang
Quoted 46 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
On modern RHEL (>=8) and Debian KDE systems, systemd-coredump can be installed to capture core dumps from crashed programs. If this is the case, we would like to capture core dumps from programs that crash during the test. Set up an (admittedly overwrought) pipeline to extract dumps created during the test and then capture them the same way that we pick up "core" and "core.$pid" files.
Signed-off-by "Darrick J. Wong" <[email protected]>--- check | 2 ++ common/rc | 44 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+)check +2
-0@@ -924,6 +924,7 @@ function run_section() $1 == "'$seqnum'" {lasttime=" " $2 "s ... "; exit} \ END {printf "%s", lasttime}' "$check.time" rm -f core $seqres.notrun + _start_coredumpctl_collection start=`_wallclock` $timestamp && _timestamp@@ -957,6 +958,7 @@ function run_section() # just "core". Use globbing to find the most common patterns, # assuming there are no other coredump capture packages set up. local cores=0 + _finish_coredumpctl_collection for i in core core.*; do test -f "$i" || continue if ((cores++ == 0)); thencommon/rc +2
-0@@ -5034,6 +5034,50 @@ _check_kmemleak() fi } +# Current timestamp, in a format that systemd likes +_systemd_now() {common/rc _check_kmemleak() · line 5048
Quoted 10 lines
common/rc +10
-0+ timedatectl show --property=TimeUSec --value +} + +# Do what we need to do to capture core dumps from coredumpctl +_start_coredumpctl_collection() { + command -v coredumpctl &>/dev/null || return + command -v timedatectl &>/dev/null || return + command -v jq &>/dev/null || return + + sysctl kernel.core_pattern | grep -q systemd-coredump || return# rpm -qf `which coredumpctl` systemd-udev-252-53.el9.x86_64 # rpm -qf `which timedatectl` systemd-252-53.el9.x86_64 # rpm -qf `which jq` jq-1.6-17.el9.x86_64 # rpm -qf /usr/lib/systemd/systemd-coredump systemd-udev-252-53.el9.x86_64
Quoted 1 line
So we have 3 optional running dependences, how about metion that in README?
Done.
--D
Quoted 41 lines
Thanks,
ZorroQuoted 36 lines
common/rc +32
-0+ COREDUMPCTL_START_TIMESTAMP="$(_systemd_now)" +} + +# Capture core dumps from coredumpctl. +# +# coredumpctl list only supports json output as a machine-readable format. The +# human-readable format intermingles spaces from the timestamp with actual +# column separators, so we cannot parse that sanely. The json output is an +# array of: +# { +# "time" : 1749744847150926, +# "pid" : 2297, +# "uid" : 0, +# "gid" : 0, +# "sig" : 6, +# "corefile" : "present", +# "exe" : "/run/fstests/e2fsprogs/fuse2fs", +# "size" : 47245 +# }, +# So we use jq to filter out lost corefiles, then print the pid and exe +# separated by a pipe and hope that nobody ever puts a pipe in an executable +# name. +_finish_coredumpctl_collection() { + test -n "$COREDUMPCTL_START_TIMESTAMP" || return + + coredumpctl list --since="$COREDUMPCTL_START_TIMESTAMP" --json=short 2>/dev/null | \ + jq --raw-output 'map(select(.corefile == "present")) | map("\(.pid)|\(.exe)") | .[]' | while IFS='|' read pid exe; do + test -e "core.$pid" || coredumpctl dump --output="core.$pid" "$pid" "$exe" &>> $seqres.full + done + unset COREDUMPCTL_START_TIMESTAMP +} + # don't check dmesg log after test _disable_dmesg_check() {
PATCH v2 2/2 check: collect core dumps from systemd-coredump
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
On modern RHEL (>=8) and Debian KDE systems, systemd-coredump can be installed to capture core dumps from crashed programs. If this is the case, we would like to capture core dumps from programs that crash during the test. Set up an (admittedly overwrought) pipeline to extract dumps created during the test and then capture them the same way that we pick up "core" and "core.$pid" files.
---
v2: update reamde
--- README | 20 ++++++++++++++++++++ check | 2 ++ common/rc | 44 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 66 insertions(+)
README +20 -0
@@ -109,6 +109,11 @@ Ubuntu or Debian $ sudo apt-get install exfatprogs f2fs-tools ocfs2-tools udftools xfsdump \ xfslibs-dev +3. Install packages for optional features: + + systemd coredump capture: + $ sudo apt install systemd-coredump systemd jq + Fedora ------
@@ -124,6 +129,11 @@ Fedora $ sudo yum install btrfs-progs exfatprogs f2fs-tools ocfs2-tools xfsdump \ xfsprogs-devel +3. Install packages for optional features: + + systemd coredump capture: + $ sudo yum install systemd systemd-udev jq + RHEL or CentOS --------------
@@ -159,6 +169,11 @@ RHEL or CentOS For ocfs2 build and install: - see https://github.com/markfasheh/ocfs2-tools +5. Install packages for optional features: + + systemd coredump capture: + $ sudo yum install systemd systemd-udev jq + SUSE Linux Enterprise or openSUSE ---------------------------------
@@ -176,6 +191,11 @@ SUSE Linux Enterprise or openSUSE For XFS install: $ sudo zypper install xfsdump xfsprogs-devel +3. Install packages for optional features: + + systemd coredump capture: + $ sudo yum install systemd systemd-coredump jq + Build and install test, libs and utils --------------------------------------
check +2 -0
@@ -924,6 +924,7 @@ function run_section() $1 == "'$seqnum'" {lasttime=" " $2 "s ... "; exit} \ END {printf "%s", lasttime}' "$check.time" rm -f core $seqres.notrun + _start_coredumpctl_collection start=`_wallclock` $timestamp && _timestamp
@@ -957,6 +958,7 @@ function run_section() # just "core". Use globbing to find the most common patterns, # assuming there are no other coredump capture packages set up. local cores=0 + _finish_coredumpctl_collection for i in core core.*; do test -f "$i" || continue if ((cores++ == 0)); then
common/rc +44 -0
50 lines not on this page · raw message