e2fsprogs: e2scrub cleanups
cover e2fsprogs: e2scrub cleanups
The primary changes are in the device probe loop refactor for e2scrub_all.
There were also some documentation changes for the -n option.
Darrick J. Wong (1):
e2scrub_all: refactor device probe loop
Theodore Ts'o (8):
e2scrub: check to make sure lvm2 is installed
debian: drop lvm2 from the recommends line
Fix "make install-strip"
e2scrub: fix up "make install-strip" support
e2scrub: add the -n option which shows what commands e2scrub would
execute
e2scrub_all: add the -n option which shows what e2scrub_all would do
e2scrub_all: make sure there's enough free space for a snapshot e2scrub,e2scrub_all: print a (more understandable) error if not run as
root
Signature · 1 line
2.19.1
PATCH 1/9 e2scrub: check to make sure lvm2 is installed
Not all systems will have the lvm2 package installed, so check for that. Pretty much all systems should have util-linux installed, but check for that as well.
Of course, if lvm2 is installed we shouldn't find any LVM devices --- but eventually the Demon Murphy will find a way to make it happen. :-)
Also, set the PATH so we don't have to worry about the script failing due to /sbin not being in the path.
--- scrub/e2scrub.in | 12 ++++++++++++ scrub/e2scrub_all.in | 16 ++++++++++++++++ 2 files changed, 28 insertions(+)
scrub/e2scrub.in +12 -0
@@ -23,6 +23,8 @@ # check filesystems in VGs that have at least 256MB (or so) of # free space. +PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin + snap_size_mb=256 fstrim=0 reap=0
@@ -82,6 +84,16 @@ if [ -z "${arg}" ]; then exitcode 1 fi +if ! type lsblk >& /dev/null ; then + echo "e2scrub: can't find lsblk --- is util-linux installed?" + exitcode 1 +fi + +if ! type lvcreate >& /dev/null ; then + echo "e2scrub: can't find lvcreate --- is lvm2 installed?" + exitcode 1 +fi + # Find the device for a given mountpoint dev_from_mount() { local mountpt="$(realpath "$1")"
scrub/e2scrub_all.in +16 -0
@@ -18,6 +18,8 @@ # along with this program; if not, write the Free Software Foundation, # Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA. +PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin + scrub_all=0 conffile="@root_sysconfdir@/e2scrub.conf"
@@ -68,6 +70,20 @@ while getopts "ArV" opt; do done shift "$((OPTIND - 1))" +# If some prerequisite packages are not installed, exit with a code +# indicating success to avoid spamming the sysadmin with fail messages +# when e2scrub_all is run out of cron or a systemd timer. + +if ! type lsblk >& /dev/null ; then + echo "e2scrub_all: can't find lsblk --- is util-linux installed?" + exitcode 0 +fi + +if ! type lvcreate >& /dev/null ; then + echo "e2scrub_all: can't find lvcreate --- is lvm2 installed?" + exitcode 0 +fi + # Find scrub targets, make sure we only do this once. ls_scrub_targets() { lsblk -o NAME,FSTYPE,MOUNTPOINT -p -P -n | while read vars; do
Signature · 1 line
2.19.1
PATCH 2/9 debian: drop lvm2 from the recommends line
In reply to Theodore Ts'oIf the user doesn't intend to use lvm2, and it's not installed, installing e2fsprogs shouldn't drag it (and all of its dependencies) into the system.
Addresses-Debian-Bug: 924275
--- debian/control | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
debian/control +1 -1
@@ -191,7 +191,7 @@ XB-Important: yes Pre-Depends: ${shlibs:Depends}, ${misc:Depends}, libblkid1, libuuid1 Multi-Arch: foreign Suggests: gpart, parted, fuse2fs, e2fsck-static -Recommends: e2fsprogs-l10n, lvm2 +Recommends: e2fsprogs-l10n Architecture: any Description: ext2/ext3/ext4 file system utilities The ext2, ext3 and ext4 file systems are successors of the original ext
Signature · 1 line
2.19.1
PATCH 3/9 Fix "make install-strip"
In reply to Theodore Ts'o--- lib/Makefile.bsd-lib | 2 +- lib/Makefile.darwin-lib | 2 +- lib/Makefile.elf-lib | 2 +- lib/Makefile.library | 2 ++ lib/Makefile.solaris-lib | 2 +- 5 files changed, 6 insertions(+), 4 deletions(-)
lib/Makefile.bsd-lib +1 -1
@@ -36,7 +36,7 @@ install-shlibs install:: $(BSD_LIB) install-strip: install -install-shlibs-strip: install-shlibs +install-shlibs-strip:: install-shlibs uninstall-shlibs uninstall:: $(RM) -f $(DESTDIR)$(BSDLIB_INSTALL_DIR)/$(BSD_LIB)
lib/Makefile.darwin-lib +1 -1
@@ -39,7 +39,7 @@ install-shlibs install:: $(BSD_LIB) install-strip: install -install-shlibs-strip: install-shlibs +install-shlibs-strip:: install-shlibs uninstall-shlibs uninstall:: $(RM) -f $(DESTDIR)$(BSDLIB_INSTALL_DIR)/$(BSD_LIB)
lib/Makefile.elf-lib +1 -1
@@ -58,7 +58,7 @@ install-strip: install $(Q) $(STRIP) --strip-unneeded --remove-section=.comment \ --remove-section=.note $(DESTDIR)$(ELF_INSTALL_DIR)/$(ELF_LIB) -install-shlibs-strip: install-shlibs +install-shlibs-strip:: install-shlibs $(E) " STRIP-LIB $(ELF_INSTALL_DIR)/$(ELF_LIB)" $(Q) $(STRIP) --strip-unneeded --remove-section=.comment \ --remove-section=.note $(DESTDIR)$(ELF_INSTALL_DIR)/$(ELF_LIB)
lib/Makefile.library +2 -0
@@ -1,5 +1,7 @@ all:: subdirs $(LIBRARY).a +install-shlibs-strip:: + install-shlibs:: uninstall-shlibs::
lib/Makefile.solaris-lib +1 -1
@@ -50,7 +50,7 @@ install-shlibs install:: $(ELF_LIB) installdirs-elf-lib install-strip: install $(STRIP) -x $(DESTDIR)$(ELF_INSTALL_DIR)/$(ELF_LIB) -install-shlibs-strip: install-shlibs +install-shlibs-strip:: install-shlibs $(STRIP) -x $(DESTDIR)$(ELF_INSTALL_DIR)/$(ELF_LIB) uninstall-shlibs uninstall::
Signature · 1 line
2.19.1
PATCH 4/9 e2scrub: fix up "make install-strip" support
In reply to Theodore Ts'o--- scrub/Makefile.in | 2 ++ 1 file changed, 2 insertions(+)
scrub/Makefile.in +2 -0
@@ -129,6 +129,8 @@ install-systemd: $(SERVICE_FILES) $(INSTALL_DATA) $$i $(DESTDIR)$(SYSTEMD_SYSTEM_UNIT_DIR)/$$i; \ done +install-strip: install + install: $(PROGS) $(MANPAGES) $(FMANPAGES) installdirs $(INSTALL_TGT) $(Q) for i in $(PROGS); do \ $(ES) " INSTALL $(root_sbindir)/$$i"; \
Signature · 1 line
2.19.1
PATCH 5/9 e2scrub: add the -n option which shows what commands e2scrub would execute
In reply to Theodore Ts'oAlso fix the copyright symbol so it is properly formatted when processed into postscript or pdf.
--- scrub/e2scrub.8.in | 11 ++++++++++- scrub/e2scrub.in | 6 ++++-- 2 files changed, 14 insertions(+), 3 deletions(-)
scrub/e2scrub.8.in +10 -1
@@ -38,6 +38,15 @@ If the filesystem is not repaired, will be run before the next mount. .SH OPTIONS .TP +\fB-n\fR +Print what commands +.B e2scrub +would execute to check the file system. (Note: the commands will not +actually be executed; however, since +.B e2scrub +needs to run some commands to query the system to determine what +commands would be executed, it still needs to be run as root.) +.TP \fB-r\fR Remove the e2scrub snapshot and exit without checking anything. .TP
@@ -57,4 +66,4 @@ The exit codes are the same as in .SH AUTHOR Darrick J. Wong <[email protected]> .SH COPYRIGHT -Copyright ©2018 Oracle. License is GPLv2+. <http://www.gnu.org/licenses/gpl-2.0.html> +Copyright \[co]2018 Oracle. License is GPLv2+. <http://www.gnu.org/licenses/gpl-2.0.html>
scrub/e2scrub.in +4 -2
@@ -37,6 +37,7 @@ print_help() { echo "Usage: $0 [OPTIONS] mountpoint | device" echo echo "mountpoint must be on a LVM-managed block device" + echo "-n: Show what commands e2scrub would execute." echo "-r: Remove e2scrub snapshot and exit, do not check anything." echo "-t: Run fstrim if successful." echo "-V: Print version information and exit."
@@ -68,8 +69,9 @@ exitcode() { exit "${ret}" } -while getopts "rtV" opt; do - case "${opt}" in +while getopts "nrtV" opt; do + case "${opt}" in + "n") DBG="echo Would execute: " ;; "r") reap=1;; "t") fstrim=1;; "V") print_version; exitcode 0;;
Signature · 1 line
2.19.1
PATCH 6/9 e2scrub_all: add the -n option which shows what e2scrub_all would do
In reply to Theodore Ts'oAlso fix the copyright symbol so it is properly formatted when processed into postscript or pdf.
--- scrub/e2scrub_all.8.in | 17 ++++++++++++++--- scrub/e2scrub_all.in | 8 +++++--- 2 files changed, 19 insertions(+), 6 deletions(-)
scrub/e2scrub_all.8.in +14 -3
@@ -20,12 +20,23 @@ See the manual page for more information about how the checking is performed. .SH OPTIONS .TP -\fB-A\fR -Scrub all ext[234] filesystems even if they are not mounted. +\fB-n\fR +Print what commands +.B e2scrub_all +would execute to initiate the e2scrub operations. +(Note: these commands will not actually be executed; however, since +.B e2scrub_all +needs to run some additional, privileged commands to query the +system to determine which +.B e2scrub +commands would be executed, it still needs to be run as root.) .TP \fB-r\fR Remove e2scrub snapshots but do not check anything. .TP +\fB-A\fR +Scrub all ext[234] filesystems even if they are not mounted. +.TP \fB-V\fR Print version information and exit. .SH SEE ALSO
@@ -33,4 +44,4 @@ Print version information and exit. .SH AUTHOR Darrick J. Wong <[email protected]> .SH COPYRIGHT -Copyright ©2018 Oracle. License is GPLv2+. <http://www.gnu.org/licenses/gpl-2.0.html> +Copyright \[co]2018 Oracle. License is GPLv2+. <http://www.gnu.org/licenses/gpl-2.0.html>
scrub/e2scrub_all.in +5 -3
@@ -29,8 +29,9 @@ scrub_args="" print_help() { echo "Usage: $0 [OPTIONS]" - echo " -A: Scrub all ext[234] filesystems even if not mounted." + echo " -n: Show what commands e2scrub_all would execute." echo " -r: Remove e2scrub snapshots." + echo " -A: Scrub all ext[234] filesystems even if not mounted." echo " -V: Print version information and exit." }
@@ -60,10 +61,11 @@ exitcode() { exit "${ret}" } -while getopts "ArV" opt; do +while getopts "nrAV" opt; do case "${opt}" in - "A") scrub_all=1;; + "n") DBG="echo Would execute: " ;; "r") scrub_args="${scrub_args} -r";; + "A") scrub_all=1;; "V") print_version; exitcode 0;; *) print_help; exitcode 2;; esac
Signature · 1 line
2.19.1
PATCH 7/9 e2scrub_all: make sure there's enough free space for a snapshot
In reply to Theodore Ts'oIf there isn't, skip the volume so we don't spam the system administrator with error messages. It's quite commkon that there is is zero free space in the volume group.
Addresses-Debian-Bug: #924301
--- scrub/e2scrub_all.in | 4 ++++ 1 file changed, 4 insertions(+)
scrub/e2scrub_all.in +4 -0
@@ -21,6 +21,7 @@ PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin scrub_all=0 +snap_size_mb=256 conffile="@root_sysconfdir@/e2scrub.conf" test -f "${conffile}" && . "${conffile}"
@@ -108,6 +109,9 @@ ls_scrub_targets() { eval "${lvm_vars}" echo "${LVM2_LV_ROLE}" | grep -q "snapshot" && continue + free_space="$(vgs -o vg_free --units m --noheadings --no-suffix "${LVM2_VG_NAME}" 2> /dev/null | sed -e 's/\..*//')" + test "${snap_size_mb}" -gt "${free_space}" && continue + if [ -n "${MOUNTPOINT}" ]; then echo "${MOUNTPOINT}" else
Signature · 1 line
2.19.1
PATCH 8/9 e2scrub_all: refactor device probe loop
In reply to Theodore Ts'oFrom: "Darrick J. Wong" <[email protected]>
Paul Menzel reported that the e2scrub_all reaper service that runs at startup takes a long time to run, and Ted Ts'o pointed out that we could do a lot less work by using lvs as the outer loop in the ext4 filesystem probe function so that we only have to lsblk the lvm devices containing ext4 filesystems.
Therefore, refactor the loops to put lvs first, which should boost speed a bit.
[ Made some of the further optimizations suggested by Lukas Czerner. -- TYT ]
--- scrub/e2scrub_all.in | 49 ++++++++++++++++++++++---------------------- 1 file changed, 24 insertions(+), 25 deletions(-)
scrub/e2scrub_all.in +24 -25
@@ -22,6 +22,7 @@ PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin scrub_all=0 snap_size_mb=256 +reap=0 conffile="@root_sysconfdir@/e2scrub.conf" test -f "${conffile}" && . "${conffile}"
@@ -65,7 +66,7 @@ exitcode() { while getopts "nrAV" opt; do case "${opt}" in "n") DBG="echo Would execute: " ;; - "r") scrub_args="${scrub_args} -r";; + "r") scrub_args="${scrub_args} -r"; reap=1;; "A") scrub_all=1;; "V") print_version; exitcode 0;; *) print_help; exitcode 2;;
@@ -88,36 +89,34 @@ if ! type lvcreate >& /dev/null ; then fi # Find scrub targets, make sure we only do this once. -ls_scrub_targets() { - lsblk -o NAME,FSTYPE,MOUNTPOINT -p -P -n | while read vars; do - eval "${vars}" - +ls_scan_targets() { + for NAME in $(lvs -o lv_path --noheadings \ + -S "lv_active=active,lv_role=public,lv_role!=snapshot,vg_free>${snap_size_mb}") ; do # Skip non-ext[234] - case "${FSTYPE}" in + case "$(blkid -o value -s TYPE ${NAME})" in ext[234]) ;; *) continue;; esac - # Skip unmounted filesystems unless -A - if [ "${scrub_all}" -eq 0 ] && [ -z "${MOUNTPOINT}" ]; then - continue; + if [ "${scrub_all}" -eq 1 ] || + [ -n "$(lsblk -o MOUNTPOINT --noheadings ${NAME})" ]; then + echo ${NAME} fi + done | sort | uniq +} - # Skip non-lvm devices and lvm snapshots - lvm_vars="$(lvs --nameprefixes -o vg_name,lv_name,lv_role --noheadings "${NAME}" 2> /dev/null)" - test $? -ne 0 && continue - eval "${lvm_vars}" - echo "${LVM2_LV_ROLE}" | grep -q "snapshot" && continue - - free_space="$(vgs -o vg_free --units m --noheadings --no-suffix "${LVM2_VG_NAME}" 2> /dev/null | sed -e 's/\..*//')" - test "${snap_size_mb}" -gt "${free_space}" && continue +# Find leftover scrub snapshots +ls_reap_targets() { + lvs -o lv_path -S lv_role=snapshot -S lv_name=~\(e2scrub$\) --noheadings +} - if [ -n "${MOUNTPOINT}" ]; then - echo "${MOUNTPOINT}" - else - echo "${NAME}" - fi - done | sort | uniq +# Figure out what we're targeting +ls_targets() { + if [ "${reap}" -eq 1 ]; then + ls_reap_targets + else + ls_scan_targets + fi } # systemd doesn't know to do path escaping on the instance variable we pass
@@ -140,10 +139,10 @@ escape_path_for_systemd() { # Scrub any mounted fs on lvm by creating a snapshot and fscking that. stdin="$(realpath /dev/stdin)" -ls_scrub_targets | while read tgt; do +ls_targets | while read tgt; do # If we're not reaping and systemd is present, try invoking the # systemd service. - if [ -z "${scrub_args}" ] && type systemctl > /dev/null 2>&1; then + if [ "${reap}" -ne 1 ] && type systemctl > /dev/null 2>&1; then tgt_esc="$(escape_path_for_systemd "${tgt}")" ${DBG} systemctl start "e2scrub@${tgt_esc}" 2> /dev/null < "${stdin}" res=$?
Signature · 1 line
2.19.1
Re: [PATCH 8/9] e2scrub_all: refactor device probe loop
Quoted 45 lines · Theodore Ts'o
From: "Darrick J. Wong" <[email protected]>
Paul Menzel reported that the
e2scrub_allreaper service that runs at startup takes a long time to run, and Ted Ts'o pointed out that we could do a lot less work by using lvs as the outer loop in the ext4 filesystem probe function so that we only have to lsblk the lvm devices containing ext4 filesystems.Therefore, refactor the loops to put lvs first, which should boost speed a bit.
[ Made some of the further optimizations suggested by Lukas Czerner. -- TYT ]
Reported-by Paul Menzel <[email protected]>Signed-off-by Darrick J. Wong <[email protected]>Signed-off-by Theodore Ts'o <[email protected]>--- scrub/e2scrub_all.in | 49 ++++++++++++++++++++++---------------------- 1 file changed, 24 insertions(+), 25 deletions(-)scrub/e2scrub_all.in +2
-1@@ -22,6 +22,7 @@ PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin scrub_all=0 snap_size_mb=256 +reap=0 conffile="@root_sysconfdir@/e2scrub.conf" test -f "${conffile}" && . "${conffile}"@@ -65,7 +66,7 @@ exitcode() { while getopts "nrAV" opt; do case "${opt}" in "n") DBG="echo Would execute: " ;; - "r") scrub_args="${scrub_args} -r";; + "r") scrub_args="${scrub_args} -r"; reap=1;; "A") scrub_all=1;; "V") print_version; exitcode 0;; *) print_help; exitcode 2;;@@ -88,36 +89,34 @@ if ! type lvcreate >& /dev/null ; then fi # Find scrub targets, make sure we only do this once.
scrub/e2scrub_all.in if ! type lvcreate >& /dev/null ; then · line 96
Quoted 10 lines
scrub/e2scrub_all.in +4
-5-ls_scrub_targets() { - lsblk -o NAME,FSTYPE,MOUNTPOINT -p -P -n | while read vars; do - eval "${vars}" - +ls_scan_targets() { + for NAME in $(lvs -o lv_path --noheadings \ + -S "lv_active=active,lv_role=public,lv_role!=snapshot,vg_free>${snap_size_mb}") ; do # Skip non-ext[234] - case "${FSTYPE}" in + case "$(blkid -o value -s TYPE ${NAME})" in
as I said in previous email for some reason blkid makes it terribly slow for me.
lsblk -o FSTYPE -n ${NAME}lsblk makes is almost 9 times faster on my system. Not really sure what's going on with blkid in my setup.
scrub/e2scrub_all.in if ! type lvcreate >& /dev/null ; then · line 105
Quoted 12 lines
scrub/e2scrub_all.in +4
-3ext[234]) ;; *) continue;; esac - # Skip unmounted filesystems unless -A - if [ "${scrub_all}" -eq 0 ] && [ -z "${MOUNTPOINT}" ]; then - continue; + if [ "${scrub_all}" -eq 1 ] || + [ -n "$(lsblk -o MOUNTPOINT --noheadings ${NAME})" ]; then + echo ${NAME} fi + done | sort | uniq
It's already sorted and unique from lvs.
-Lukas
Quoted 47 lines
scrub/e2scrub_all.in +14
-16+} - # Skip non-lvm devices and lvm snapshots - lvm_vars="$(lvs --nameprefixes -o vg_name,lv_name,lv_role --noheadings "${NAME}" 2> /dev/null)" - test $? -ne 0 && continue - eval "${lvm_vars}" - echo "${LVM2_LV_ROLE}" | grep -q "snapshot" && continue - - free_space="$(vgs -o vg_free --units m --noheadings --no-suffix "${LVM2_VG_NAME}" 2> /dev/null | sed -e 's/\..*//')" - test "${snap_size_mb}" -gt "${free_space}" && continue +# Find leftover scrub snapshots +ls_reap_targets() { + lvs -o lv_path -S lv_role=snapshot -S lv_name=~\(e2scrub$\) --noheadings +} - if [ -n "${MOUNTPOINT}" ]; then - echo "${MOUNTPOINT}" - else - echo "${NAME}" - fi - done | sort | uniq +# Figure out what we're targeting +ls_targets() { + if [ "${reap}" -eq 1 ]; then + ls_reap_targets + else + ls_scan_targets + fi } # systemd doesn't know to do path escaping on the instance variable we pass@@ -140,10 +139,10 @@ escape_path_for_systemd() { # Scrub any mounted fs on lvm by creating a snapshot and fscking that. stdin="$(realpath /dev/stdin)" -ls_scrub_targets | while read tgt; do +ls_targets | while read tgt; do # If we're not reaping and systemd is present, try invoking the # systemd service. - if [ -z "${scrub_args}" ] && type systemctl > /dev/null 2>&1; then + if [ "${reap}" -ne 1 ] && type systemctl > /dev/null 2>&1; then tgt_esc="$(escape_path_for_systemd "${tgt}")" ${DBG} systemctl start "e2scrub@${tgt_esc}" 2> /dev/null < "${stdin}" res=$?Signature · 1 line
2.19.1
PATCH 9/9 e2scrub,e2scrub_all: print a (more understandable) error if not run as root
In reply to Theodore Ts'o--- scrub/e2scrub.in | 5 +++++ scrub/e2scrub_all.in | 5 +++++ 2 files changed, 10 insertions(+)
scrub/e2scrub.in +5 -0
@@ -25,6 +25,11 @@ PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin +if (( $EUID != 0 )); then + echo "e2scrub must be run as root" + exit 1 +fi + snap_size_mb=256 fstrim=0 reap=0
scrub/e2scrub_all.in +5 -0
@@ -20,6 +20,11 @@ PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin +if (( $EUID != 0 )); then + echo "e2scrub_all must be run as root" + exit 1 +fi + scrub_all=0 snap_size_mb=256 reap=0
Signature · 1 line
2.19.1