From: Bernd Schubert Subject: Re: deadlock balance_dirty_pages() to be expected? Date: Fri, 07 Oct 2011 16:08:06 +0200 Message-ID: <4E8F07C6.6000808@itwm.fraunhofer.de> References: <4E8EF1D9.8050700@itwm.fraunhofer.de> <20111007133733.GA6399@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Jan Kara , Peter Zijlstra To: Wu Fengguang , linux-fsdevel@vger.kernel.org Return-path: Received: from mailgw1.uni-kl.de ([131.246.120.220]:59652 "EHLO mailgw1.uni-kl.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752663Ab1JGOIK (ORCPT ); Fri, 7 Oct 2011 10:08:10 -0400 Received: from itwm2.itwm.fhg.de (itwm2.itwm.fhg.de [131.246.191.3]) by mailgw1.uni-kl.de (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id p97E88pl004939 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NOT) for ; Fri, 7 Oct 2011 16:08:08 +0200 In-Reply-To: <20111007133733.GA6399@localhost> Sender: linux-fsdevel-owner@vger.kernel.org List-ID: Hello Fengguang, On 10/07/2011 03:37 PM, Wu Fengguang wrote: > Hi Bernd, > > On Fri, Oct 07, 2011 at 08:34:33PM +0800, Bernd Schubert wrote: >> Hello, >> >> while I'm working on the page cached mode in FhGFS (*) I noticed a >> deadlock in balance_dirty_pages(). >> >> sysrq-w showed that it never started background write-out due to >> >> if (bdi_nr_reclaimable> bdi_thresh) { >> pages_written += writeback_inodes_wb(&bdi->wb, >> (write_chunk); >> >> >> and therefore also did not leave that loop with >> >> if (pages_written>= write_chunk) >> break; /* We've done our duty */ >> >> >> So my process stay in uninterruptible D-state forever. > > If writeback_inodes_wb() is not triggered, the process should still be > able to proceed, presumably with longer delays, but never stuck forever. > That's because the flusher thread should still be cleaning the pages > in the background which will knock down the dirty pages and eventually > unthrottle the dirtier process. Hmm, that does not seem to work: 1330 pts/0 D+ 0:13 dd if=/dev/zero of=/mnt/fhgfs/testfile bs=1M count=100 So the process is in D state ever since I wrote the first mail, just for 100MB writes. Even if it still would do something, it would be extremely slow. Sysrq-w then shows: > [ 6727.616976] SysRq : Show Blocked State > [ 6727.617575] task PC stack pid father > [ 6727.618252] dd D 0000000000000000 3544 1330 1306 0x00000000 > [ 6727.619002] ffff88000ddfb9a8 0000000000000046 ffffffff81398627 0000000000000046 > [ 6727.620157] 0000000000000000 ffff88000ddfa000 ffff88000ddfa000 ffff88000ddfbfd8 > [ 6727.620466] ffff88000ddfa010 ffff88000ddfa000 ffff88000ddfbfd8 ffff88000ddfa000 > [ 6727.620466] Call Trace: > [ 6727.620466] [] ? __schedule+0x697/0x7e0 > [ 6727.620466] [] ? trace_hardirqs_on_caller+0x20/0x1b0 > [ 6727.620466] [] schedule+0x3f/0x60 > [ 6727.620466] [] schedule_timeout+0x164/0x2f0 > [ 6727.620466] [] ? lock_timer_base+0x70/0x70 > [ 6727.620466] [] io_schedule_timeout+0x69/0x90 > [ 6727.620466] [] balance_dirty_pages_ratelimited_nr+0x234/0x640 > [ 6727.620466] [] ? iov_iter_copy_from_user_atomic+0xaf/0x180 > [ 6727.620466] [] generic_file_buffered_write+0x1ce/0x270 > [ 6727.620466] [] ? generic_file_aio_write+0x5c/0xf0 > [ 6727.620466] [] __generic_file_aio_write+0x238/0x460 > [ 6727.620466] [] ? generic_file_aio_write+0x5c/0xf0 > [ 6727.620466] [] generic_file_aio_write+0x78/0xf0 > [ 6727.620466] [] FhgfsOps_aio_write+0xdc/0x144 [fhgfs] > [ 6727.620466] [] do_sync_write+0xda/0x120 > [ 6727.620466] [] ? might_fault+0x9c/0xb0 > [ 6727.620466] [] vfs_write+0xc8/0x180 > [ 6727.620466] [] sys_write+0x51/0x90 > [ 6727.620466] [] system_call_fastpath+0x16/0x1b > [ 6727.620466] Sched Debug Version: v0.10, 3.1.0-rc9+ #47 > >> Once I added basic inode->i_data.backing_dev_info bdi support to our >> file system, the deadlock did not happen anymore. > > What's the workload and change exactly? I wish I could simply send the patch, but until all the paper work is done I'm not allowed to :( The basic idea is: 1) During mount and setting the super block from static struct file_system_type fhgfs_fs_type = { .mount = fhgfs_mount, } Then in fhgfs_mount(): bdi_setup_and_register(&sbInfo->bdi, "fhgfs", BDI_CAP_MAP_COPY); sb->s_bdi = &sbInfo->bdi; 2) When new (S_IFREG) inodes are allocated, for example from static struct inode_operations fhgfs_dir_inode_ops { .lookup, .create, .link } inode->i_data.backing_dev_info = &sbInfo->bdi; > >> So my question is simply if we should expect this deadlock, if the file >> system does not set up backing device information and if so, shouldn't >> this be documented? > > Such deadlock is not expected.. Ok thanks, then we should figure out why it happens. Due to a network outage here I won't have time before Monday to track down which kernel version introduced it, though. Thanks, Bernd