suspiciously good fsck times?
Quoted 14 lines · Theodore Tso
Quoted 3 lines · Eric Sandeen
Val & I talked about this a little, and came to the conclusion that directory fragmentation might be a pretty big part of it.
Hmm, could be. Let's see. Ric said 46.5 million files, I don't know how big the filenames were, but let's assume a directory entry size of 32, so that means if we assume perfect packing, 128 directory entries per 4k block. Let's use 100 directory entries/blok just to make the math easyer, so that's 465,000 blocks. If we assume a 10ms seek time, and that the blocks are totally scattered, that's 4650 seconds, or 1.29 hours. So that's roughly within the ballpark that Ric measured.
Quoted 2 lines
- Ted
(changing cc to the real list instead of ext4 owner - sorry!)
The file names are 40 bytes long, (6 initial bytes of time stamp with 24 random bytes at end of name. For example:
451aeb61ead89~~~DYASX8LYL4NAUWK3WI187VRP
The 4 threads chose the target subdirectory based on the time stamp, rotating into a new subdirectory every 3 minutes or so.
ric
Quoted 15 lines · Theodore Tso
Quoted 8 lines · Ric Wheeler
Just to be mean, I have been trying to test the fsck speed of ext4 with lots of small files. The test I ran uses
fs_markto fill a 1TB Seagate drive with 45.6 million 20k files (distributed between 256 subdirectories).Running on ext3, "fsck -f" takes about one hour.
Running on ext4, with
uninit_bg, the same fsck is finished in a bit over 5 minutes - more than 10x faster. (Withoutuninit_bg, the fsck takes about 10 minutes).Is this too good to be true? Below is the fsck run itself, the tree is Ted's latest git tree and his 1.41 WIP tools,
Wow. My guess is that
flex_bgis making the difference. What we would want to compare is the I/O read statistics line:
Quoted 9 lines
Quoted 2 lines
I/O read: 14198MB, write: 1MB, rate: 46.77MB/s
That's pretty good, and indicates we've avoided a *lot* of seeking. The e2fsck -t -t output for ext3 should show roughly the same mount of I/O read (with 20k files, there would be no advantage towards using extents), but the I/O rate is probably *much* lower, indicating a lot more seeking is going on.
We did run fsck through seekwatcher & saw a significant reduction in seeks/sec for ext4. Eric has the pretty pictures that he can share.
Quoted 3 lines
Can you send the full e2fsck -t -t output of the ext3 run? And what is the hdparm -t -t results of the disk?
I didn't run the ext3 test with -t -t (but can refill and rerun, takes about 12 hours).
This disk is a relatively new Seagate 1TB drive, specs at:
http://www.seagate.com/ww/v/index.jsp?vgnextoid=0732f141e7f43110VgnVCM100000f5ee0a0aRCRD
hdparm test:
[root@localhost rwheeler]# /sbin/hdparm -t -t /dev/sdb
/dev/sdb:
Timing buffered disk reads: 186 MB in 3.03 seconds = 61.33 MB/sec
Quoted 4 lines
If I'm right, if you create the filesystem with mke2fs -t ext4dev -O ^flex_bg,^uninit_bg, you should see performance back to the old ext3 levels.
With uninit_bg off, it ran about 10 minutes, but it would be interesting to run without either.
Quoted 5 lines
- Ted
P.S. We probably do want to examine the block allocation layout withflex_bgto make sure that the filesystem ages well in the long term.
Testing aged file systems is always the holy grail - this workload is a fairly artificial one and was laid down with 4 threads currently writing to a shared subdirectory.
ric