From: Jan Kara Subject: Re: [PATCH v2] writeback: Do not sync data dirtied after sync start Date: Fri, 27 Sep 2013 11:37:45 +0200 Message-ID: <20130927093745.GA31068@quack.suse.cz> References: <1380223438-26381-1-git-send-email-jack@suse.cz> <20130927005553.GV26872@dastard> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: Jan Kara , Al Viro , Wu Fengguang , linux-fsdevel@vger.kernel.org To: Dave Chinner Return-path: Received: from cantor2.suse.de ([195.135.220.15]:45650 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752155Ab3I0Jht (ORCPT ); Fri, 27 Sep 2013 05:37:49 -0400 Content-Disposition: inline In-Reply-To: <20130927005553.GV26872@dastard> Sender: linux-fsdevel-owner@vger.kernel.org List-ID: On Fri 27-09-13 10:55:53, Dave Chinner wrote: > On Thu, Sep 26, 2013 at 09:23:58PM +0200, Jan Kara wrote: > > When there are processes heavily creating small files while sync(2)= is > > running, it can easily happen that quite some new files are created > > between WB_SYNC_NONE and WB_SYNC_ALL pass of sync(2). That can happ= en > > especially if there are several busy filesystems (remember that syn= c > > traverses filesystems sequentially and waits in WB_SYNC_ALL phase o= n one > > fs before starting it on another fs). Because WB_SYNC_ALL pass is s= low > > (e.g. causes a transaction commit and cache flush for each inode in > > ext3), resulting sync(2) times are rather large. >=20 > Yup, that can be a problem. >=20 > Build warning form the patch: >=20 > In file included from include/trace/ftrace.h:575:0, > from include/trace/define_trace.h:90, > from include/trace/events/writeback.h:603, > from fs/fs-writeback.c:89: > include/trace/events/writeback.h: In function =BFftrace_raw_event_wri= teback_queue_io=BF: > include/trace/events/writeback.h:277:1: warning: initialization makes= pointer from integer without a cast [enabled by default] > In file included from include/trace/ftrace.h:711:0, > from include/trace/define_trace.h:90, > from include/trace/events/writeback.h:603, > from fs/fs-writeback.c:89: > include/trace/events/writeback.h: In function =BFperf_trace_writeback= _queue_io=BF: > include/trace/events/writeback.h:277:1: warning: initialization makes= pointer from integer without a cast [enabled by default] Thanks for catching this. I'll send v3 in a minute. > > The following script reproduces the problem: > >=20 > > function run_writers > > { > > for (( i =3D 0; i < 10; i++ )); do > > mkdir $1/dir$i > > for (( j =3D 0; j < 40000; j++ )); do > > dd if=3D/dev/zero of=3D$1/dir$i/$j bs=3D4k count=3D4 &>/dev/n= ull > > done & > > done > > } > >=20 > > for dir in "$@"; do > > run_writers $dir > > done > >=20 > > sleep 40 > > time sync > > =3D=3D=3D=3D=3D=3D > >=20 > > Fix the problem by disregarding inodes dirtied after sync(2) was ca= lled > > in the WB_SYNC_ALL pass. To allow for this, sync_inodes_sb() now ta= kes a > > time stamp when sync has started which is used for setting up work = for > > flusher threads. > >=20 > > To give some numbers, when above script is run on two ext4 filesyst= ems on > > simple SATA drive, the average sync time from 10 runs is 267.549 se= conds > > with standard deviation 104.799426. With the patched kernel, the av= erage > > sync time from 10 runs is 2.995 seconds with standard deviation 0.0= 96. >=20 > Hmmmm. 2.8 seconds on my XFS perf VM without the patch. Ok, try a > smaller VM backed by single spindle of spinning rust rather than > SSDs. Over 10 runs I see: >=20 > kernel min max av > vanilla 0.18s 4.46s 1.63s > patched 0.14s 0.45s 0.28s >=20 > Definitely an improvement, but nowhere near the numbers you are > seeing for ext4 - maybe XFS isn't as susceptible to this problem > as ext4. Nope, ext4 on an unpatched kernel gives 1.66/6.81/3.12s, > (which is less than your patched kernel results :) but means > so it must be something else configuration/hardware related. Have you really used *two* (or more) busy filesystems? That makes the problem an order of magnitude worse for me. The numbers I've posted are= for such situation... Honza --=20 Jan Kara SUSE Labs, CR -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel= " in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html