From: Bernd Schubert Subject: Re: deadlock balance_dirty_pages() to be expected? Date: Fri, 07 Oct 2011 16:30:18 +0200 Message-ID: <4E8F0CFA.3010205@itwm.fraunhofer.de> References: <4E8EF1D9.8050700@itwm.fraunhofer.de> <20111007133733.GA6399@localhost> <4E8F07C6.6000808@itwm.fraunhofer.de> <20111007142126.GA11800@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "linux-fsdevel@vger.kernel.org" , Jan Kara , Peter Zijlstra To: Wu Fengguang Return-path: Received: from mailgw1.uni-kl.de ([131.246.120.220]:49134 "EHLO mailgw1.uni-kl.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S965451Ab1JGOaV (ORCPT ); Fri, 7 Oct 2011 10:30:21 -0400 Received: from itwm2.itwm.fhg.de (itwm2.itwm.fhg.de [131.246.191.3]) by mailgw1.uni-kl.de (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id p97EUJrO016430 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NOT) for ; Fri, 7 Oct 2011 16:30:19 +0200 In-Reply-To: <20111007142126.GA11800@localhost> Sender: linux-fsdevel-owner@vger.kernel.org List-ID: On 10/07/2011 04:21 PM, Wu Fengguang wrote: > On Fri, Oct 07, 2011 at 10:08:06PM +0800, Bernd Schubert wrote: >> Hello Fengguang, >> >> On 10/07/2011 03:37 PM, Wu Fengguang wrote: >>> Hi Bernd, >>> >>> On Fri, Oct 07, 2011 at 08:34:33PM +0800, Bernd Schubert wrote: >>>> Hello, >>>> >>>> while I'm working on the page cached mode in FhGFS (*) I noticed a >>>> deadlock in balance_dirty_pages(). >>>> >>>> sysrq-w showed that it never started background write-out due to >>>> >>>> if (bdi_nr_reclaimable> bdi_thresh) { >>>> pages_written += writeback_inodes_wb(&bdi->wb, >>>> (write_chunk); >>>> >>>> >>>> and therefore also did not leave that loop with >>>> >>>> if (pages_written>= write_chunk) >>>> break; /* We've done our duty */ >>>> >>>> >>>> So my process stay in uninterruptible D-state forever. >>> >>> If writeback_inodes_wb() is not triggered, the process should still be >>> able to proceed, presumably with longer delays, but never stuck forever. >>> That's because the flusher thread should still be cleaning the pages >>> in the background which will knock down the dirty pages and eventually >>> unthrottle the dirtier process. >> >> Hmm, that does not seem to work: >> >> 1330 pts/0 D+ 0:13 dd if=/dev/zero of=/mnt/fhgfs/testfile bs=1M >> count=100 > > That's normal: dd will be in D state in the vast majority time, but > the point is, one single balance_dirty_pages() call should not take > forever time, and dd should be able to go out of the D state (and > re-enter it almost immediately) from time to time. > >> So the process is in D state ever since I wrote the first mail, just for >> 100MB writes. Even if it still would do something, it would be extremely >> slow. Sysrq-w then shows: > > So it's normal to catch such trace for 99% times. But do you mean the > writeout bandwidth is lower than expected? If it really is still doing something, it is *ways* slower. Once I added bdi support, it finishes to write the 100MB file in my kvm test instance within a few seconds. Right now it is running for hours already... As I added a dump_stack() to our writepages() method, I also see that this function is never called. > >>> [ 6727.616976] SysRq : Show Blocked State >>> [ 6727.617575] task PC stack pid father >>> [ 6727.618252] dd D 0000000000000000 3544 1330 1306 0x00000000 >>> [ 6727.619002] ffff88000ddfb9a8 0000000000000046 ffffffff81398627 0000000000000046 >>> [ 6727.620157] 0000000000000000 ffff88000ddfa000 ffff88000ddfa000 ffff88000ddfbfd8 >>> [ 6727.620466] ffff88000ddfa010 ffff88000ddfa000 ffff88000ddfbfd8 ffff88000ddfa000 >>> [ 6727.620466] Call Trace: >>> [ 6727.620466] [] ? __schedule+0x697/0x7e0 >>> [ 6727.620466] [] ? trace_hardirqs_on_caller+0x20/0x1b0 >>> [ 6727.620466] [] schedule+0x3f/0x60 >>> [ 6727.620466] [] schedule_timeout+0x164/0x2f0 >>> [ 6727.620466] [] ? lock_timer_base+0x70/0x70 >>> [ 6727.620466] [] io_schedule_timeout+0x69/0x90 >>> [ 6727.620466] [] balance_dirty_pages_ratelimited_nr+0x234/0x640 >>> [ 6727.620466] [] ? iov_iter_copy_from_user_atomic+0xaf/0x180 >>> [ 6727.620466] [] generic_file_buffered_write+0x1ce/0x270 >>> [ 6727.620466] [] ? generic_file_aio_write+0x5c/0xf0 >>> [ 6727.620466] [] __generic_file_aio_write+0x238/0x460 >>> [ 6727.620466] [] ? generic_file_aio_write+0x5c/0xf0 >>> [ 6727.620466] [] generic_file_aio_write+0x78/0xf0 >>> [ 6727.620466] [] FhgfsOps_aio_write+0xdc/0x144 [fhgfs] >>> [ 6727.620466] [] do_sync_write+0xda/0x120 >>> [ 6727.620466] [] ? might_fault+0x9c/0xb0 >>> [ 6727.620466] [] vfs_write+0xc8/0x180 >>> [ 6727.620466] [] sys_write+0x51/0x90 >>> [ 6727.620466] [] system_call_fastpath+0x16/0x1b >>> [ 6727.620466] Sched Debug Version: v0.10, 3.1.0-rc9+ #47 >> >> >>> >>>> Once I added basic inode->i_data.backing_dev_info bdi support to our >>>> file system, the deadlock did not happen anymore. >>> >>> What's the workload and change exactly? >> >> I wish I could simply send the patch, but until all the paper work is >> done I'm not allowed to :( >> >> The basic idea is: >> >> 1) During mount and setting the super block from >> >> static struct file_system_type fhgfs_fs_type = >> { >> .mount = fhgfs_mount, >> } >> >> Then in fhgfs_mount(): >> >> bdi_setup_and_register(&sbInfo->bdi, "fhgfs", BDI_CAP_MAP_COPY); >> sb->s_bdi =&sbInfo->bdi; >> >> >> >> 2) When new (S_IFREG) inodes are allocated, for example from >> >> static struct inode_operations fhgfs_dir_inode_ops >> { >> .lookup, >> .create, >> .link >> } >> >> inode->i_data.backing_dev_info =&sbInfo->bdi; > > Ah when you didn't register the "fhgfs" bdi, there should be no > dedicated flusher thread for doing the writeout. Which is obviously > suboptimal. > >>>> So my question is simply if we should expect this deadlock, if the file >>>> system does not set up backing device information and if so, shouldn't >>>> this be documented? >>> >>> Such deadlock is not expected.. >> >> Ok thanks, then we should figure out why it happens. Due to a network >> outage here I won't have time before Monday to track down which kernel >> version introduced it, though. > > It's long time ago when the per-bdi writeback is introduced, I suspect. Ok, I can start to test if 2.6.32 also already deadlocks. Thanks, Bernd