Skip to contentSkip to conversation

benchmarks of O_STREAMING in 2.5

DISCUSSION Robert Love 4 participants 6 messages Oct 17, 2002 – Oct 21, 2002

lore
Share & tools
lore

Robert Love

author
#0 lore raw

I gave the O_STREAMING in Andrew's 2.5-mm tree the treatment..

Short summary: It works.

The streaming read test in the following benchmarks is simply a read() in 64KB byte chunks of an 800MB file.

First test. Show the cache effects are indeed as we intend and the overhead is negligible. Here, mem=2G.

	O_STREAMING?	Wall time	Cache Delta
	Yes		21.827s		0
	No		21.734s		+800MB

Second test. Same deal, but mem=8M. There was slight swapping, so I suspect the reduced VM pressure is why the O_STREAMING run is faster.

	O_STREAMING?	Wall time	Cache Delta
	Yes		22.303s		0
	No		28.812s		+1MB

Third and final test. Kernel compile (make -j2) with a couple streaming reads in the background. Again, mem=2G. This shows that actually saving the pagecache from the horrid waste is useful.

	O_STREAMING	Wall time to complete Kernel compile
	Yes		5m30.494s
	No		4m59.661s

So, uh, Andrew's 2.5 code works ;-)

Someone buy me a dual Xeon,

Robert Love

#1 lore raw
Quoted 7 lines

I gave the O_STREAMING in Andrew's 2.5-mm tree the treatment..

      O_STREAMING     Wall time to complete Kernel compile
      Yes             5m30.494s
      No              4m59.661s

So, uh, Andrew's 2.5 code works ;-)

?? O_STREAMING makes it slower or is it a typo ?

Bye.

author
#2 lore raw
Quoted 9 lines · Giuliano Pochini
Quoted 7 lines

I gave the O_STREAMING in Andrew's 2.5-mm tree the treatment..

      O_STREAMING     Wall time to complete Kernel compile
      Yes             5m30.494s
      No              4m59.661s

So, uh, Andrew's 2.5 code works ;-)

?? O_STREAMING makes it slower or is it a typo ?

Yes, typo, those should be switched.

Robert Love

#3 lore raw
In reply to Robert Love
Quoted 6 lines · Robert Love

I gave the O_STREAMING in Andrew's 2.5-mm tree the treatment..

Short summary: It works.

The streaming read test in the following benchmarks is simply a read() in 64KB byte chunks of an 800MB file.

All you now need to do is make it work with an API thats usable by the other 99% of real world apps, is extensible and sensible ways and therefore can be used.

#4 lore raw
Quoted 4 lines · Alan Cox
Quoted 2 lines · Robert Love

I gave the O_STREAMING in Andrew's 2.5-mm tree the treatment..

Quoted 8 lines
Quoted 4 lines

Short summary: It works.

The streaming read test in the following benchmarks is simply a read() in 64KB byte chunks of an 800MB file.

All you now need to do is make it work with an API thats usable by the other 99% of real world apps, is extensible and sensible ways and therefore can be used.

I'm confused. Isn't this just an O_STREAM flag on open/fcntl ? How could it be simpler? I suppose the VM could do better than it currently does automatically but there is no harm in the app giving a hint like this thus allowing stuff to be dropped from the cache more aggresively?

Pádraig.

#5 lore raw
Quoted 5 lines · Padraig Brady

I'm confused. Isn't this just an O_STREAM flag on open/fcntl ? How could it be simpler? I suppose the VM could do better than it currently does automatically but there is no harm in the app giving a hint like this thus allowing stuff to be dropped from the cache more aggresively?

Which bit are you streaming, what kind of things are you doing with the data ?

Take a database table are you - accessing it sequentially with caching, accessing it randomly without readahead or streaming it and discarding data. All three are valid answers, for the same data, and potentially on a per table basis in the same file.

See the fadvise proposal. O_STREAMING is basically a subset of what is needed