Skip to contentSkip to conversation

xfstest 285: userspace vs. fragmented multiblock dir2

PATCH In review Eric Sandeen 6 participants 6 messages May 21, 2012 – Nov 20, 2012

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xfstest 285: userspace vs. fragmented multiblock dir2

Current xfs_repair is unhappy with fragmented multiblock v2 directories. This test shows it ... patches to fix it soon to follow.

---

285 new file +128 -0
new file mode 100755
index 0000000..11a66f6
@@ -0,0 +1,128 @@
+#! /bin/bash
+# FS QA Test No. 285
+#
+# Test userspace on a fragmented multi-block dir2 filesystem
+#
+#-----------------------------------------------------------------------
+# Copyright (c) Red Hat, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+# creator
[email protected]
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+    cd /
+    rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fs xfs
+_supported_os IRIX Linux
+
+# real QA test starts here
+rm -f $seq.full
+_require_scratch
+_scratch_mkfs_xfs -n size=16k -d size=128m >> $seq.full 2>&1
+_scratch_mount
+
+# First we cause very badly fragmented freespace, then
+# make some multiblock directories in the filesystem, which
+# should come out very fragmented as a result
+
+# Step 1: Cause badly fragmented free space
+mkdir $SCRATCH_MNT/fragdir
+for I in `seq 0 27200`; do
+        (echo data > $SCRATCH_MNT/fragdir/f$I) >> $seq.full 2>&1
+done
+sync
+
+for I in `seq 0 2 27200`; do
+        rm -f $SCRATCH_MNT/fragdir/f$I >> $seq.full 2>&1
+done
+sync
+
+# Soak up any remaining freespace
+xfs_io -f -c "pwrite 0 16m" -c "fsync" $SCRATCH_MNT/space_file.large >> $seq.full 2>&1
+
+# Take a look at freespace for any post-mortem on the test
+_scratch_unmount
+xfs_db -c freesp $SCRATCH_DEV >> $seq.full 2>&1
+_scratch_mount
+
+# Step 2: Make a bunch of (hopefully fragmented) multiblock
+# dir2 directories
+
+# - FMT_LOCAL
+mkdir $SCRATCH_MNT/S_IFDIR.FMT_LOCAL
+touch $SCRATCH_MNT/S_IFDIR.FMT_LOCAL/localdirfile
+
+# - FMT_EXTENTS
+mkdir $SCRATCH_MNT/S_IFDIR.FMT_EXTENTS
+for I in `seq 1 100`; do
+	touch $SCRATCH_MNT/S_IFDIR.FMT_EXTENTS/extent_dir_file_$I
+done
+# With a few missing
+for I in `seq 10 2 20` 100; do
+	rm -f $SCRATCH_MNT/S_IFDIR.FMT_EXTENTS/extent_dir_file_$I
+done
+
+# - FMT_BTREE
+mkdir $SCRATCH_MNT/S_IFDIR.FMT_BTREE
+for I in `seq 1 1000`; do
+	touch $SCRATCH_MNT/S_IFDIR.FMT_BTREE/btree_dir_file_$I
+done
+# With a few missing
+for I in `seq 10 2 20` 1000; do
+	rm -f $SCRATCH_MNT/S_IFDIR.FMT_BTREE/btree_dir_file_$I
+done
+
+# Dave's special hack - grow freespace tree
+mkdir $SCRATCH_MNT/S_IFDIR.FMT_BTREE2
+for I in `seq 1 5000`; do
+	touch $SCRATCH_MNT/S_IFDIR.FMT_BTREE2/btree2_dir_file_$I
+done
+# Remove every other (odds)
+for I in `seq 1 2 5000`; do
+	rm -f $SCRATCH_MNT/S_IFDIR.FMT_BTREE2/btree2_dir_file_$I
+done
+
+_scratch_unmount
+# Can xfs_repair and xfs_check cope with this monster?
+_scratch_xfs_repair >> $seq.full 2>&1 || _fail "xfs_repair failed"
+xfs_check $SCRATCH_DEV >> $seq.full 2>&1 || _fail "xfs_check failed"
+
+# Yes they can!  Now...
+# Can xfs_metadump cope with this monster?
+xfs_metadump $SCRATCH_DEV $tmp.metadump || _fail "xfs_metadump failed"
+xfs_mdrestore $tmp.metadump $tmp.img || _fail "xfs_mdrestore failed"
+xfs_repair $tmp.img >> $seq.full 2>&1 || _fail "xfs_repair of metadump failed"
+
+# Yes it can; success, all done
+status=0
+exit
+
group +1 -0
index 104ed35..17afdcd 100644
@@ -403,3 +403,4 @@ deprecated
 282 dump ioctl auto quick
 283 dump ioctl auto quick
 284 auto
+285 repair dir auto

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw

Hey Eric,

Quoted 5 lines · Eric Sandeen

Current xfs_repair is unhappy with fragmented multiblock v2 directories. This test shows it ... patches to fix it soon to follow.

This failed on my system... looks like it's just PEBKAC. My partitions are too small?

# ./check 287
No patches applied
KERNEL_TOPDIR -- /root/xfs
BRANCH        -- refs/heads/master2
DESC          -- v3.4-10158-gf8f5701
PATCHES:
FSTYP         -- xfs (debug)
PLATFORM      -- Linux/x86_64 nfs10 3.5.0-rc1-0.7-default+
MKFS_OPTIONS  -- -f -bsize=4096 /dev/sdb4
MOUNT_OPTIONS -- /dev/sdb4 /mnt/scratch

287 - no qualified output

Ran: 287
Failures: 287
Failed 1 of 1 tests

Regards,
Ben

P.S. Here's the contents of $seq.full.

meta-data=/dev/sdb4              isize=256    agcount=4, agsize=8192 blks
         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=32768, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=16384  ascii-ci=0
log      =internal log           bsize=4096   blocks=1904, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

./287: line 60: /mnt/scratch/fragdir/f27196: No space left on device ./287: line 60: /mnt/scratch/fragdir/f27197: No space left on device ./287: line 60: /mnt/scratch/fragdir/f27198: No space left on device ./287: line 60: /mnt/scratch/fragdir/f27199: No space left on device ./287: line 60: /mnt/scratch/fragdir/f27200: No space left on device wrote 16777216/16777216 bytes at offset 0
16 MiB, 4096 ops; 0.0000 sec (565.311 MiB/sec and 144719.6410 ops/sec)
from to extents blocks pct

      1       1   11197   11197 100.00

Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
- scan filesystem freespace and inode maps...
- found root inode chunk
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
Phase 5 - rebuild AG headers and trees...
- reset superblock...
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
- traversal finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done
xfs_db got a bbmap for 127248
xfs_db got a bbmap for 195224
xfs_db got a bbmap for 195352
xfs_db got a bbmap for 195288
xfs_db got a bbmap for 248080
xfs_db got a bbmap for 247952
xfs_db got a bbmap for 247872
xfs_db got a bbmap for 247808
xfs_db got a bbmap for 247744
xfs_db got a bbmap for 247680
xfs_db got a bbmap for 247616
xfs_db got a bbmap for 247520
xfs_db got a bbmap for 247456
xfs_db got a bbmap for 247392
xfs_db got a bbmap for 248016
xfs_db got a bbmap for 248192
xfs_db got a bbmap for 248256
xfs_db got a bbmap for 248384
xfs_db got a bbmap for 248128
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
- scan filesystem freespace and inode maps...
- found root inode chunk
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 1
- agno = 0
- agno = 2
- agno = 3
Phase 5 - rebuild AG headers and trees...
- reset superblock...
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
- traversal finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw
Quoted 23 lines · Ben Myers

Hey Eric,

Quoted 5 lines · Eric Sandeen

Current xfs_repair is unhappy with fragmented multiblock v2 directories. This test shows it ... patches to fix it soon to follow.

This failed on my system... looks like it's just PEBKAC. My partitions are too small?

# ./check 287
No patches applied
KERNEL_TOPDIR -- /root/xfs
BRANCH        -- refs/heads/master2
DESC          -- v3.4-10158-gf8f5701
PATCHES:
FSTYP         -- xfs (debug)
PLATFORM      -- Linux/x86_64 nfs10 3.5.0-rc1-0.7-default+
MKFS_OPTIONS  -- -f -bsize=4096 /dev/sdb4
MOUNT_OPTIONS -- /dev/sdb4 /mnt/scratch
Quoted 1 line

287 - no qualified output

It failed because there is not 287.out file in the patch. Every test has to have a golden image to compare the result against...

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#3 lore raw
Quoted 24 lines · Dave Chinner
Quoted 23 lines · Ben Myers

Hey Eric,

Quoted 5 lines · Eric Sandeen

Current xfs_repair is unhappy with fragmented multiblock v2 directories. This test shows it ... patches to fix it soon to follow.

This failed on my system... looks like it's just PEBKAC. My partitions are too small?

# ./check 287
No patches applied
KERNEL_TOPDIR -- /root/xfs
BRANCH        -- refs/heads/master2
DESC          -- v3.4-10158-gf8f5701
PATCHES:
FSTYP         -- xfs (debug)
PLATFORM      -- Linux/x86_64 nfs10 3.5.0-rc1-0.7-default+
MKFS_OPTIONS  -- -f -bsize=4096 /dev/sdb4
MOUNT_OPTIONS -- /dev/sdb4 /mnt/scratch
Quoted 5 lines
Quoted 1 line

287 - no qualified output

It failed because there is not 287.out file in the patch. Every test has to have a golden image to compare the result against...

Argh.....

Sorry.

Quoted 6 lines

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#4 lore raw

PATCH v2 1/1 xfstest 287: userspace vs. fragmented multiblock dir2

In reply to Eric Sandeen

Current xfs_repair is unhappy with fragmented multiblock v2 directories. This test shows it ... patches to fix it soon to follow.

---

V2: Renumbered, now with even more 287.out-ness.
(hand-renamed to 287, should apply...)
((we need to merge the 2 git trees ... ))

287 new file +128 -0
new file mode 100755
index 0000000..11a66f6
@@ -0,0 +1,128 @@
+#! /bin/bash
+# FS QA Test No. 287
+#
+# Test xfs_repair on fragmented multi-block dir2 fs
+#
+#-----------------------------------------------------------------------
+# Copyright (c) Red Hat, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+# creator
[email protected]
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+    cd /
+    rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fs xfs
+_supported_os IRIX Linux
+
+# real QA test starts here
+rm -f $seq.full
+_require_scratch
+_scratch_mkfs_xfs -n size=16k -d size=128m >> $seq.full 2>&1
+_scratch_mount
+
+# First we cause very badly fragmented freespace, then
+# make some multiblock directories in the filesystem, which
+# should come out very fragmented as a result
+
+# Step 1: Cause badly fragmented free space
+mkdir $SCRATCH_MNT/fragdir
+for I in `seq 0 27200`; do
+        (echo data > $SCRATCH_MNT/fragdir/f$I) >> $seq.full 2>&1
+done
+sync
+
+for I in `seq 0 2 27200`; do
+        rm -f $SCRATCH_MNT/fragdir/f$I >> $seq.full 2>&1
+done
+sync
+
+# Soak up any remaining freespace
+xfs_io -f -c "pwrite 0 16m" -c "fsync" $SCRATCH_MNT/space_file.large >> $seq.full 2>&1
+
+# Take a look at freespace for any post-mortem on the test
+_scratch_unmount
+xfs_db -c freesp $SCRATCH_DEV >> $seq.full 2>&1
+_scratch_mount
+
+# Step 2: Make a bunch of (hopefully fragmented) multiblock
+# dir2 directories
+
+# - FMT_LOCAL
+mkdir $SCRATCH_MNT/S_IFDIR.FMT_LOCAL
+touch $SCRATCH_MNT/S_IFDIR.FMT_LOCAL/localdirfile
+
+# - FMT_EXTENTS
+mkdir $SCRATCH_MNT/S_IFDIR.FMT_EXTENTS
+for I in `seq 1 100`; do
+	touch $SCRATCH_MNT/S_IFDIR.FMT_EXTENTS/extent_dir_file_$I
+done
+# With a few missing
+for I in `seq 10 2 20` 100; do
+	rm -f $SCRATCH_MNT/S_IFDIR.FMT_EXTENTS/extent_dir_file_$I
+done
+
+# - FMT_BTREE
+mkdir $SCRATCH_MNT/S_IFDIR.FMT_BTREE
+for I in `seq 1 1000`; do
+	touch $SCRATCH_MNT/S_IFDIR.FMT_BTREE/btree_dir_file_$I
+done
+# With a few missing
+for I in `seq 10 2 20` 1000; do
+	rm -f $SCRATCH_MNT/S_IFDIR.FMT_BTREE/btree_dir_file_$I
+done
+
+# Dave's special hack - grow freespace tree
+mkdir $SCRATCH_MNT/S_IFDIR.FMT_BTREE2
+for I in `seq 1 5000`; do
+	touch $SCRATCH_MNT/S_IFDIR.FMT_BTREE2/btree2_dir_file_$I
+done
+# Remove every other (odds)
+for I in `seq 1 2 5000`; do
+	rm -f $SCRATCH_MNT/S_IFDIR.FMT_BTREE2/btree2_dir_file_$I
+done
+
+_scratch_unmount
+# Can xfs_repair and xfs_check cope with this monster?
+_scratch_xfs_repair >> $seq.full 2>&1 || _fail "xfs_repair failed"
+xfs_check $SCRATCH_DEV >> $seq.full 2>&1 || _fail "xfs_check failed"
+
+# Yes they can!  Now...
+# Can xfs_metadump cope with this monster?
+xfs_metadump $SCRATCH_DEV $tmp.metadump || _fail "xfs_metadump failed"
+xfs_mdrestore $tmp.metadump $tmp.img || _fail "xfs_mdrestore failed"
+xfs_repair $tmp.img >> $seq.full 2>&1 || _fail "xfs_repair of metadump failed"
+
+# Yes it can; success, all done
+status=0
+exit
+
287.out new file +1 -0
new file mode 100644
index 0000000..58759aa
@@ -0,0 +1 @@
+QA output created by 287
group +1 -0
index 104ed35..17afdcd 100644
@@ -403,3 +403,4 @@ deprecated
 284 auto
 285 auto rw
 286 other
+287 repair

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#5 lore raw

Re: [PATCH V2] xfstest 287: userspace vs. fragmented multiblock dir2

Quoted 163 lines · Eric Sandeen

Current xfs_repair is unhappy with fragmented multiblock v2 directories. This test shows it ... patches to fix it soon to follow.

---

V2: Renumbered, now with even more 287.out-ness.
(hand-renamed to 287, should apply...)
((we need to merge the 2 git trees ... ))

287 new file +128 -0
new file mode 100755
index 0000000..11a66f6
@@ -0,0 +1,128 @@
+#! /bin/bash
+# FS QA Test No. 287
+#
+# Test xfs_repair on fragmented multi-block dir2 fs
+#
+#-----------------------------------------------------------------------
+# Copyright (c) Red Hat, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+# creator
[email protected]
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+    cd /
+    rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fs xfs
+_supported_os IRIX Linux
+
+# real QA test starts here
+rm -f $seq.full
+_require_scratch
+_scratch_mkfs_xfs -n size=16k -d size=128m>>  $seq.full 2>&1
+_scratch_mount
+
+# First we cause very badly fragmented freespace, then
+# make some multiblock directories in the filesystem, which
+# should come out very fragmented as a result
+
+# Step 1: Cause badly fragmented free space
+mkdir $SCRATCH_MNT/fragdir
+for I in `seq 0 27200`; do
+        (echo data>  $SCRATCH_MNT/fragdir/f$I)>>  $seq.full 2>&1
+done
+sync
+
+for I in `seq 0 2 27200`; do
+        rm -f $SCRATCH_MNT/fragdir/f$I>>  $seq.full 2>&1
+done
+sync
+
+# Soak up any remaining freespace
+xfs_io -f -c "pwrite 0 16m" -c "fsync" $SCRATCH_MNT/space_file.large>>  $seq.full 2>&1
+
+# Take a look at freespace for any post-mortem on the test
+_scratch_unmount
+xfs_db -c freesp $SCRATCH_DEV>>  $seq.full 2>&1
+_scratch_mount
+
+# Step 2: Make a bunch of (hopefully fragmented) multiblock
+# dir2 directories
+
+# - FMT_LOCAL
+mkdir $SCRATCH_MNT/S_IFDIR.FMT_LOCAL
+touch $SCRATCH_MNT/S_IFDIR.FMT_LOCAL/localdirfile
+
+# - FMT_EXTENTS
+mkdir $SCRATCH_MNT/S_IFDIR.FMT_EXTENTS
+for I in `seq 1 100`; do
+	touch $SCRATCH_MNT/S_IFDIR.FMT_EXTENTS/extent_dir_file_$I
+done
+# With a few missing
+for I in `seq 10 2 20` 100; do
+	rm -f $SCRATCH_MNT/S_IFDIR.FMT_EXTENTS/extent_dir_file_$I
+done
+
+# - FMT_BTREE
+mkdir $SCRATCH_MNT/S_IFDIR.FMT_BTREE
+for I in `seq 1 1000`; do
+	touch $SCRATCH_MNT/S_IFDIR.FMT_BTREE/btree_dir_file_$I
+done
+# With a few missing
+for I in `seq 10 2 20` 1000; do
+	rm -f $SCRATCH_MNT/S_IFDIR.FMT_BTREE/btree_dir_file_$I
+done
+
+# Dave's special hack - grow freespace tree
+mkdir $SCRATCH_MNT/S_IFDIR.FMT_BTREE2
+for I in `seq 1 5000`; do
+	touch $SCRATCH_MNT/S_IFDIR.FMT_BTREE2/btree2_dir_file_$I
+done
+# Remove every other (odds)
+for I in `seq 1 2 5000`; do
+	rm -f $SCRATCH_MNT/S_IFDIR.FMT_BTREE2/btree2_dir_file_$I
+done
+
+_scratch_unmount
+# Can xfs_repair and xfs_check cope with this monster?
+_scratch_xfs_repair>>  $seq.full 2>&1 || _fail "xfs_repair failed"
+xfs_check $SCRATCH_DEV>>  $seq.full 2>&1 || _fail "xfs_check failed"
+
+# Yes they can!  Now...
+# Can xfs_metadump cope with this monster?
+xfs_metadump $SCRATCH_DEV $tmp.metadump || _fail "xfs_metadump failed"
+xfs_mdrestore $tmp.metadump $tmp.img || _fail "xfs_mdrestore failed"
+xfs_repair $tmp.img>>  $seq.full 2>&1 || _fail "xfs_repair of metadump failed"
+
+# Yes it can; success, all done
+status=0
+exit
+
287.out new file +1 -0
new file mode 100644
index 0000000..58759aa
@@ -0,0 +1 @@
+QA output created by 287
group +1 -0
index 104ed35..17afdcd 100644
@@ -403,3 +403,4 @@ deprecated
  284 auto
  285 auto rw
  286 other
+287 repair
Quoted 4 lines

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

This patch has been committed to git://oss.sgi.com/xfs/cmds/xfstests master branch, commit 2a4ed (test 291).

--Mark.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs