From: Al Boldi Subject: Re: Allocation strategy - dynamic zone for small files Date: Tue, 14 Nov 2006 14:21:09 +0300 Message-ID: <200611141421.09823.a1426z@gawab.com> References: <1163426218.3343.10.camel@avishay.fsl.cs.sunysb.edu.fsl.cs.sunysb.edu> <20061114010235.GB6454@thunk.org> Mime-Version: 1.0 Content-Type: text/plain; charset="windows-1256" Content-Transfer-Encoding: 7bit Return-path: Received: from [212.33.164.255] ([212.33.164.255]:22656 "EHLO raad.intranet") by vger.kernel.org with ESMTP id S1754774AbWKNLR7 (ORCPT ); Tue, 14 Nov 2006 06:17:59 -0500 Received: from localhost ([10.0.0.111]) by raad.intranet (8.8.7/8.8.7) with ESMTP id OAA04294 for ; Tue, 14 Nov 2006 14:18:38 +0300 To: linux-fsdevel@vger.kernel.org In-Reply-To: <20061114010235.GB6454@thunk.org> Content-Disposition: inline Sender: linux-fsdevel-owner@vger.kernel.org List-Id: linux-fsdevel.vger.kernel.org Theodore Tso wrote: > > More thoughts: > > 1) It's not just about storage efficiency, but also about transfer > efficiency. Disk drives generally like to transfer hunks of data in > 16k to 64k at a time. So getting related pieces of small hunks of > data read at the same time, we can win big on performance. BUT, it's > extremely hard to do this at the filesystem level, since the > application is much more likely to know which micro-file of 16 bytes > is likely to be needed at the same time as some other micro-file which > is only 16 bytes long. > > 2) If you have millions of separate files, each 16 bytes long, and you > need to read a huge number of them, you can end up getting killed on > system call overhead. > > I remember having this argument with Hans Reiser at one point. His > argument was that parsing was evil; and should never have to be done. > (And if anyone has ever seen the vast quanties of garbage which is > generated when you implement an XML parser in Java, and the resulting > GC overhead I can't blame them for thinking this...) So his argument > was that instead of parsing a file like /etc/inetd.conf, there should > be an /etc/inetd.conf.d directory, and in that directory there might > be directory called telnet, and another one called ssh, and yet > another called smtp, and then you might have files such as: > > FILENAME CONTENTS > =============================================================== > > /etc/inetd.conf.d/telnet/port 23 > /etc/inetd.conf.d/telnet/protocol tcp > /etc/inetd.conf.d/telnet/flags nowait > /etc/inetd.conf.d/telnet/user root > /etc/inetd.conf.d/telnet/daemon /sbin/telnetd > > /etc/inetd.conf.d/ssh/port 22 > /etc/inetd.conf.d/ssh/protocol tcp > /etc/inetd.conf.d/ssh/flags nowait > /etc/inetd.conf.d/ssh/user root > /etc/inetd.conf.d/ssh/daemon /sbin/sshd > > etc. When I pointed out the system call overhead that would result > since instead of an open, read, close to read /etc/inetd.conf, you > would now need perhaps a hundred or more system calls do the > opendir/readir loop, and then individually opening, reading, and > closing each file, Hans had a solution ---- I have a different solution. Plugins into the VFS that handle special situations. I could do something like this manually in userland via loop/FUSE, but a more integrated solution could prove more useful. Thanks! -- Al