Skip to contentSkip to conversation

single-file file system? (fwd)

DISCUSSION Aditya Kashyap 5 participants 10 messages Oct 1, 2004 – Oct 5, 2004 1 message not in the archive VFS

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): Pine.GSO.4.53.0409302256180.27202@compserv1

#1 lore raw

Hello all,

Quoted 5 lines

iirc there was one long ago from Miquel van Smoorenburg for 2.4 (or maybe 2.2 i don't remember). But if you want it for 2.6 it would be probably easier to just rewrite it.

-Andi

I found the file system that Andi Kleen talked about: rawfs, written by Miquel van Smoorenburg. It was last tested on 2.4.20, and abstracts a disk partition as a single file in a simple mountable file system.

I ran a benchmark on both, the file exported by rawfs, and the raw disk, which I shall proceed to describe. The benchmark opens the file passed to it, randomly seeks to a position in the file, and randomly chooses to either read or write a 4k block from or to the file.

The mean times from both the experiments are as follows. What is being shown is the time taken for the process to run on the file exported by rawfs, and the raw disk partition. Both the experiments were run enough times to reduce standard deviation to a minimum.

-> RAWFS exporting a 2GB partition:
Time MEAN (sec)
---- ----------
Elapsed 66.9925
User 0.0370
System 0.2510
I/O 66.7045

-> Plain 2GB partition (/dev/hdXY):
Time MEAN (sec)
---- ----------
Elapsed 74.1280
User 0.0330
System 0.2990
I/O 73.7960

The numbers above show a 10% overhead in I/O time for the plain partition over rawfs.

My question is, I wonder how rawfs performs better on the benchmark over directly accessing the raw partition, when both of them have very similar operations when reading or writing data.

Any ideas on how this could be?

Thanks,
Aditya

#2 lore raw
Quoted 1 line

Any ideas on how this could be?

The raw partition does not do any readahead or write behind. All IO is synchronous. Normally it is used by applications (databases) that do that by itself.

-Andi

#3 lore raw
Quoted 3 lines · Andi Kleen
Quoted 1 line

Any ideas on how this could be?

The raw partition does not do any readahead or write behind.

Reasonable explanation but IMHO not behavior. Is it indeed true? Why would this exception exist?

Quoted 1 line

All IO is synchronous.

There is O_SYNC for that.

Your explanation also contradicts somehow to my findings, sent here earlier:

http://marc.theaimsgroup.com/?l=linux-fsdevel&m=108981315716397

See the raw vs filesystems comparisons for both 2.4 and 2.6 kernels:

Fs_kernel_method usr sys real CPU

ext2_24_seek         1.97    17.50    8:15.98    3%
raw_24_seek          1.65    17.88    8:17.69    3%
xfs_24_seek          1.72    20.77    8:25.17    4%
ext3_24_seek         1.55    23.48    8:35.42    4%
reiserfs_24_seek     1.71    28.20    8:50.02    5%
jfs_24_seek          1.76    31.11   11:34.68    4%

Fs_kernel_method usr sys real CPU

reiser4_26_seek      1.81    36.66    8:06.12    7%
raw_26_seek          1.80    33.49    8:27.02    6%
ext2_26_seek         1.80    29.77    8:27.29    6%
reiserfs_26_seek     1.85    41.14    8:30.26    8%
xfs_26_seek          1.76    34.10    8:36.17    6%
ext3_26_seek         1.84    35.55    8:43.41    7%
jfs_26_seek          1.48    36.46   14:23.51    4%

So, except ext2 in 2.4 and reiser4 in 2.6 kernels, operations on raw partition were faster than with any other filesystems.

I doubt these filesystems would have so much houskeeping overhead that they would overrule the benefits of their performance optimizations. Raw partitions without readahead and write behind would perform even worse than jfs with both kernels, I would expect.

Szaka

#4 lore raw
Quoted 9 lines · Szakacsits Szabolcs
Quoted 3 lines · Andi Kleen
Quoted 1 line

Any ideas on how this could be?

The raw partition does not do any readahead or write behind.

Reasonable explanation but IMHO not behavior. Is it indeed true? Why would this exception exist?

Both need a OS managed cache. And RAW devices by definition don't have a cache in the OS, otherwise they wouldn't be very raw. O_DIRECT acts the same way.

-Andi

#5 lore raw
Quoted 9 lines · Andi Kleen
Quoted 5 lines · Szakacsits Szabolcs
Quoted 2 lines

The raw partition does not do any readahead or write behind.

Reasonable explanation but IMHO not behavior. Is it indeed true? Why would this exception exist?

Both need a OS managed cache. And RAW devices by definition don't have a cache in the OS

This is theory and it contradicts to observations, both on 2.4 and 2.6 kernels.

You didn't explain why is random, non-cached raw access faster than cached and optimized access for the most common used filesystems (see my former email, for example, for more details).

Szaka

#6 lore raw
In reply to Andi Kleen
Quoted 3 lines
Quoted 1 line

Any ideas on how this could be?

The raw partition does not do any readahead or write behind.

There's some terminology confusion here. These are not raw devices in Linux terminology. They are block devices. I can tell by the device special file name (/dev/hdXX) and by the fact that they're about as fast as filesystem access.

The block device does have readahead and writebehind. The raw device (because it has no readahead or writebehind) would be much, much slower with these 4K reads and writes.

#7 lore raw
Quoted 5 lines · Bryan Henderson
Quoted 2 lines

The raw partition does not do any readahead or write behind.

There's some terminology confusion here. These are not raw devices in Linux terminology. They are block devices.

Yes, this is reasonable explanation and quite probably it's the truth too.

I didn't include but I've run my former mentioned benchmarks on FreeBSD too (didn't have block devices) and the results were the worst. Even worse than jfs. So that must have been really a raw device (and what I referred as "raw" were indeed block devices).

Szaka

#8 lore raw
In reply to Bryan Henderson
Quoted 10 lines
Quoted 1 line

The raw partition does not do any readahead or write behind.

There's some terminology confusion here. These are not raw devices in Linux terminology. They are block devices. I can tell by the device special file name (/dev/hdXX) and by the fact that they're about as fast as filesystem access.

The block device does have readahead and writebehind. The raw device (because it has no readahead or writebehind) would be much, much slower with these 4K reads and writes.

That is correct. What I put down as raw devices are block devices on a system running the 2.4.24 Linux kernel.

Why then the 10% discrepancy in time, when both interfaces have readahead and writebehind while transferring data?

Thanks,
Aditya

#9 lore raw
In reply to Bryan Henderson
Quoted 4 lines · Bryan Henderson
Quoted 3 lines
Quoted 1 line

Any ideas on how this could be?

The raw partition does not do any readahead or write behind.

Quoted 8 lines

There's some terminology confusion here. These are not raw devices in Linux terminology. They are block devices. I can tell by the device special file name (/dev/hdXX) and by the fact that they're about as fast as filesystem access.

The block device does have readahead and writebehind. The raw device (because it has no readahead or writebehind) would be much, much slower with these 4K reads and writes.

I've looked at the source, and I don't see where the block device (e.g., /dev/hda6) should do synchronous writes, but I am sure that they do.

If I do 10,000 4K random writes on Ext2 it takes 0.34 seconds. If I do the same random writes on a partition, it takes 58 seconds. With O_SYNC, Ext2 takes 80 seconds.

I also ran this same test on rawfs. When O_SYNC is used, the device and rawfs are within 2%. Without O_SYNC, rawfs finishes in 0.16 seconds.

Here is a summary of the times in seconds:
Block Device Ext2 Rawfs
O_SYNC 63.5 80.9 64.3
~O_SYNC 58.5 0.35 0.17

Another mini-proof is the following:
[rcwright@t4 cwright]# time dd of=/tmp/t4 if=/dev/zero bs=1M count=100 100+0 records in
100+0 records out

real    0m0.316s
user    0m0.000s
sys     0m0.300s

[rcwright@t4 cwright]# time dd of=/dev/hda6 if=/dev/zero bs=1M count=100 100+0 records in
100+0 records out

real    0m6.038s
user    0m0.000s
sys     0m0.220s

Why would it take 6 seconds for the block device, and only 0.36 for a file system if block devices did not have synchronous writes?

Does anyone know what in the kernel source causes opening a block device to be synchronous? I've looked but don't see anything obvious.

Charles

BTW, I'm using a 2.4 kernel.

#10 lore raw

Someone told me a long time ago that block device close does a sync. It's illogical that it would do so, but I can see the practical advantage.

(You can determine whether your syncs are happening at write time or close time by trying the dd with varying buffer sizes. If it runs as fast with a 4K buffer size as with a 1M buffer size, you know you're not doing synchronous writes).