single-file file system? (fwd)
A message that is not in the archive (sent off-list or not yet received): Pine.GSO.4.53.0409302256180.27202@compserv1
Hello all,
Quoted 5 lines
iirc there was one long ago from Miquel van Smoorenburg for 2.4 (or maybe 2.2 i don't remember). But if you want it for 2.6 it would be probably easier to just rewrite it.
-Andi
I found the file system that Andi Kleen talked about: rawfs, written by Miquel van Smoorenburg. It was last tested on 2.4.20, and abstracts a disk partition as a single file in a simple mountable file system.
I ran a benchmark on both, the file exported by rawfs, and the raw disk, which I shall proceed to describe. The benchmark opens the file passed to it, randomly seeks to a position in the file, and randomly chooses to either read or write a 4k block from or to the file.
The mean times from both the experiments are as follows. What is being shown is the time taken for the process to run on the file exported by rawfs, and the raw disk partition. Both the experiments were run enough times to reduce standard deviation to a minimum.
-> RAWFS exporting a 2GB partition:
Time MEAN (sec)
---- ----------
Elapsed 66.9925
User 0.0370
System 0.2510
I/O 66.7045
-> Plain 2GB partition (/dev/hdXY):
Time MEAN (sec)
---- ----------
Elapsed 74.1280
User 0.0330
System 0.2990
I/O 73.7960
The numbers above show a 10% overhead in I/O time for the plain partition over rawfs.
My question is, I wonder how rawfs performs better on the benchmark over directly accessing the raw partition, when both of them have very similar operations when reading or writing data.
Any ideas on how this could be?
Thanks,
Aditya
Quoted 1 line
Any ideas on how this could be?
The raw partition does not do any readahead or write behind. All IO is synchronous. Normally it is used by applications (databases) that do that by itself.
-Andi
Quoted 3 lines · Andi Kleen
Quoted 1 line
Any ideas on how this could be?
The raw partition does not do any readahead or write behind.
Reasonable explanation but IMHO not behavior. Is it indeed true? Why would this exception exist?
Quoted 1 line
All IO is synchronous.
There is O_SYNC for that.
Your explanation also contradicts somehow to my findings, sent here earlier:
http://marc.theaimsgroup.com/?l=linux-fsdevel&m=108981315716397
See the raw vs filesystems comparisons for both 2.4 and 2.6 kernels:
Fs_kernel_method usr sys real CPU
ext2_24_seek 1.97 17.50 8:15.98 3% raw_24_seek 1.65 17.88 8:17.69 3% xfs_24_seek 1.72 20.77 8:25.17 4% ext3_24_seek 1.55 23.48 8:35.42 4% reiserfs_24_seek 1.71 28.20 8:50.02 5% jfs_24_seek 1.76 31.11 11:34.68 4%
Fs_kernel_method usr sys real CPU
reiser4_26_seek 1.81 36.66 8:06.12 7% raw_26_seek 1.80 33.49 8:27.02 6% ext2_26_seek 1.80 29.77 8:27.29 6% reiserfs_26_seek 1.85 41.14 8:30.26 8% xfs_26_seek 1.76 34.10 8:36.17 6% ext3_26_seek 1.84 35.55 8:43.41 7% jfs_26_seek 1.48 36.46 14:23.51 4%
So, except ext2 in 2.4 and reiser4 in 2.6 kernels, operations on raw partition were faster than with any other filesystems.
I doubt these filesystems would have so much houskeeping overhead that they would overrule the benefits of their performance optimizations. Raw partitions without readahead and write behind would perform even worse than jfs with both kernels, I would expect.
Szaka
Quoted 9 lines · Szakacsits Szabolcs
Quoted 3 lines · Andi Kleen
Quoted 1 line
Any ideas on how this could be?
The raw partition does not do any readahead or write behind.
Reasonable explanation but IMHO not behavior. Is it indeed true? Why would this exception exist?
Both need a OS managed cache. And RAW devices by definition don't have a cache in the OS, otherwise they wouldn't be very raw. O_DIRECT acts the same way.
-Andi
Quoted 9 lines · Andi Kleen
Quoted 5 lines · Szakacsits Szabolcs
Quoted 2 lines
The raw partition does not do any readahead or write behind.
Reasonable explanation but IMHO not behavior. Is it indeed true? Why would this exception exist?
Both need a OS managed cache. And RAW devices by definition don't have a cache in the OS
This is theory and it contradicts to observations, both on 2.4 and 2.6 kernels.
You didn't explain why is random, non-cached raw access faster than cached and optimized access for the most common used filesystems (see my former email, for example, for more details).
Szaka
Quoted 3 lines
Quoted 1 line
Any ideas on how this could be?
The raw partition does not do any readahead or write behind.
There's some terminology confusion here. These are not raw devices in Linux terminology. They are block devices. I can tell by the device special file name (/dev/hdXX) and by the fact that they're about as fast as filesystem access.
The block device does have readahead and writebehind. The raw device (because it has no readahead or writebehind) would be much, much slower with these 4K reads and writes.
Quoted 5 lines · Bryan Henderson
Quoted 2 lines
The raw partition does not do any readahead or write behind.
There's some terminology confusion here. These are not raw devices in Linux terminology. They are block devices.
Yes, this is reasonable explanation and quite probably it's the truth too.
I didn't include but I've run my former mentioned benchmarks on FreeBSD too (didn't have block devices) and the results were the worst. Even worse than jfs. So that must have been really a raw device (and what I referred as "raw" were indeed block devices).
Szaka
Quoted 10 lines
Quoted 1 line
The raw partition does not do any readahead or write behind.
There's some terminology confusion here. These are not raw devices in Linux terminology. They are block devices. I can tell by the device special file name (
/dev/hdXX) and by the fact that they're about as fast as filesystem access.The block device does have readahead and writebehind. The raw device (because it has no readahead or writebehind) would be much, much slower with these 4K reads and writes.
That is correct. What I put down as raw devices are block devices on a system running the 2.4.24 Linux kernel.
Why then the 10% discrepancy in time, when both interfaces have readahead and writebehind while transferring data?
Thanks,
Aditya
Quoted 4 lines · Bryan Henderson
Quoted 3 lines
Quoted 1 line
Any ideas on how this could be?
The raw partition does not do any readahead or write behind.
Quoted 8 lines
There's some terminology confusion here. These are not raw devices in Linux terminology. They are block devices. I can tell by the device special file name (
/dev/hdXX) and by the fact that they're about as fast as filesystem access.The block device does have readahead and writebehind. The raw device (because it has no readahead or writebehind) would be much, much slower with these 4K reads and writes.
I've looked at the source, and I don't see where the block device (e.g., /dev/hda6) should do synchronous writes, but I am sure that they do.
If I do 10,000 4K random writes on Ext2 it takes 0.34 seconds. If I do the same random writes on a partition, it takes 58 seconds. With O_SYNC, Ext2 takes 80 seconds.
I also ran this same test on rawfs. When O_SYNC is used, the device and rawfs are within 2%. Without O_SYNC, rawfs finishes in 0.16 seconds.
Here is a summary of the times in seconds:
Block Device Ext2 RawfsO_SYNC 63.5 80.9 64.3
~O_SYNC 58.5 0.35 0.17
Another mini-proof is the following:
[rcwright@t4 cwright]# time dd of=/tmp/t4 if=/dev/zero bs=1M count=100 100+0 records in
100+0 records out
real 0m0.316s user 0m0.000s sys 0m0.300s
[rcwright@t4 cwright]# time dd of=/dev/hda6 if=/dev/zero bs=1M count=100 100+0 records in
100+0 records out
real 0m6.038s user 0m0.000s sys 0m0.220s
Why would it take 6 seconds for the block device, and only 0.36 for a file system if block devices did not have synchronous writes?
Does anyone know what in the kernel source causes opening a block device to be synchronous? I've looked but don't see anything obvious.
Charles
BTW, I'm using a 2.4 kernel.
Someone told me a long time ago that block device close does a sync. It's illogical that it would do so, but I can see the practical advantage.
(You can determine whether your syncs are happening at write time or close time by trying the dd with varying buffer sizes. If it runs as fast with a 4K buffer size as with a 1M buffer size, you know you're not doing synchronous writes).