Skip to contentSkip to conversation

Adaptive read-ahead V12

DISCUSSION Wu Fengguang 2 participants 3 messages May 26, 2006 – Jun 1, 2006

lore
Share & tools
lore
#0 lore raw

----- Forwarded message from Iozone <[email protected]> -----

Subject: Adaptive read-ahead V12
From: Iozone <[email protected]>
To: Wu Fengguang <[email protected]>
X-Mailer: Microsoft Outlook Express 6.00.2900.2670
Date: Thu, 25 May 2006 11:44:37 -0500

Wu Fengguang,

I see that Andrew M. is giving you some pushback....
His argument is that the application could do a better job of scheduling its own read-ahead. ( I've heard this one before)

My thoughts on this argument would be along the lines of:

Indeed the application might be able to do a better job, however expecting, or demanding, the rewrite
of all applications to behave better might be an unreasonable
expectation. Rewriting all the weather codes, all the
structrual analysis codes, all the data-bases, video streaming and so on, would solve the problem but the timeline for such an re-codification may approach infinity, while systems that have the ability to silently increase performance maintain an advantage in the market.
Note: Many vendors already have sophisticated A.I. read-ahead algorithms. Examples: Microsoft, Veritas HP, and Isolon.

Linux already does read-ahead that does not require
the application to be involved. It just does it in
a simpler way that could be improved for handling
a wider range of applications. This new technology is
a simple evolutionary step in that direction. Linux read-ahead and is currently 8 years behind existing technologies in this area. Isn't it time to catch up a bit ?

Note: You probably will need to reword this before replying to Andrew, as I tend to be somewhat blunt. :-)

Hint: I took me a while to convince the kernel folks in my company to accept such a technology. But when the prototype accelerated one of their key partner's code by 30 % wall clock, and another showed 50X speedup in file I/O across a wide stripe, that did the trick.

Enjoy,
Don Capps

[...]
----- End forwarded message -----

#1 lore raw
Quoted 15 lines · Wu Fengguang

----- Forwarded message from Iozone <[email protected]> -----

Subject: Adaptive read-ahead V12
From: Iozone <[email protected]>
To: Wu Fengguang <[email protected]>
X-Mailer: Microsoft Outlook Express 6.00.2900.2670
Date: Thu, 25 May 2006 11:44:37 -0500

Wu Fengguang,

I see that Andrew M. is giving you some pushback....
His argument is that the application could do a better job of scheduling its own read-ahead. ( I've heard this one before)

Quoted 7 lines

My thoughts on this argument would be along the lines of:

Indeed the application might be able to do a better job, however expecting, or demanding, the rewrite of all applications to behave better might be an unreasonable expectation.

A reasonable expectation would be that the application would have a way to tell the kernel to ignore readahead for a given file, other than changing the behavior of the kernel as a whole for all processes on the machine. This smart application could then use aio or some other similar method to do preread itself.

My personal opinion is that the kernel only does a good job reading ahead for sequential access, and since that's the common case only a means of preventing that effort need be provided.

I can think of several ways the kernel might be smarter about the way it reads (or not) random access, but since I have no way to test them on applications other than my own I won't belabor them here.

Signature · 5 lines
Bill Davidsen <[email protected]>
  Obscure bug of 2004: BASH BUFFER OVERFLOW - if bash is being run by a
normal user and is setuid root, with the "vi" line edit mode selected,
and the character set is "big5," an off-by-one errors occurs during
wildcard (glob) expansion.
#2 lore raw
Quoted 24 lines · Bill Davidsen
Quoted 15 lines · Wu Fengguang

----- Forwarded message from Iozone <[email protected]> -----

Subject: Adaptive read-ahead V12
From: Iozone <[email protected]>
To: Wu Fengguang <[email protected]>
X-Mailer: Microsoft Outlook Express 6.00.2900.2670
Date: Thu, 25 May 2006 11:44:37 -0500

Wu Fengguang,

I see that Andrew M. is giving you some pushback....
His argument is that the application could do a better job of scheduling its own read-ahead. ( I've heard this one before)

Quoted 7 lines

My thoughts on this argument would be along the
lines of:

Indeed the application might be able to do a better job, however expecting, or demanding, the rewrite of all applications to behave better might be an unreasonable expectation.

Quoted 5 lines

A reasonable expectation would be that the application would have a way to tell the kernel to ignore readahead for a given file, other than changing the behavior of the kernel as a whole for all processes on the machine. This smart application could then use aio or some other similar method to do preread itself.

Sure. The adaptive readahead works fine with user level readahead via the readahead() or posix_fadvise() syscall. I.e. if a program do necessary readahead() calls that can cover all the file pages requested by the following read() calls, it avoids triggering unnecessary kernel readaheads.

Quoted 3 lines

My personal opinion is that the kernel only does a good job reading ahead for sequential access, and since that's the common case only a means of preventing that effort need be provided.

posix_fadvise(fd, ..., POSIX_FADV_RANDOM) can do the trick for a file. blockdev --setra 0 /dev/hda does so for a device.

Wu