Skip to contentSkip to conversation

LVM, raw-io, devfs

linux-lvm Fernando Dammous 8 participants 16 messages Oct 1, 1999 – Oct 4, 1999

lore
Share & tools
lore
#0 lore raw

Hi all,

I would like help from the more experts in this matter, because I fear that I can in some way to lose or to corrupt my FS under LVM.

LVM 0.7 is installed (built into the kernel, which is still in the version 2.0.36) and working perfectly, with the root filesystem and swap in a LV (boot with initrd).

Now, I would like to install the devfs and mainly the raw-io.

Can I install them in a kernel 2.0.36, which is already patched with LVM?

Should I apply the patches in which order (first devfs, later the raw-io, or vice versa)?

I intend to install Oracle8 using raw-io.
The devfs isn't so important for me (should it be ?).

If anyone knows about restrictions or some clues I would thank a lot.

Thanks in advance

Fernando

#1 lore raw
Quoted 9 lines · Fernando Dammous

LVM 0.7 is installed (built into the kernel, which is still in the version 2.0.36) and working perfectly, with the root filesystem and swap in a LV (boot with initrd).

Now, I would like to install the devfs and mainly the raw-io.

Can I install them in a kernel 2.0.36, which is already patched with LVM?

Should I apply the patches in which order (first devfs, later the raw-io, or vice versa)?

I intend to install Oracle8 using raw-io.

well, I'm still not sure whether Oracle software is compatible with raw-io patch in linux kernel, please let me know how are you doing with that.

Quoted 1 line

The devfs isn't so important for me (should it be ?).

devfs is a very usable patch compatible with most of devices (including LVM) and software (I can only think of vmware that doesn't run with vmware). it saves a lot of inodes in /dev and contains only devices really present in the system and uses intelligent backward-compatible naming scheme (so you can still use inodes like /dev/sda etc) present in many modern UN*X flavors (like Solaris, e.g.). You should give it at lease a chance before saying you use it only because of raw-devices ;)

Quoted 1 line

If anyone knows about restrictions or some clues I would thank a lot.

Regards,

Signature · 6 lines
                                             freddy

----==-- _                      Adrien "freddy" Farkas
---==---(_)__  __ ____  __       [email protected]
--==---/ / _ \/ // /\ \/ /  We all know Linux is great... It
-=====/_/_//_/\___/ /_/\_\  does infinite loops in 5 seconds
#2 lore raw
In reply to Fernando Dammous

Hi,

Quoted 4 lines · Fernando Dammous

Now, I would like to install the devfs and mainly the raw-io.

Can I install them in a kernel 2.0.36, which is already patched with LVM?

I've never tried the raw IO code on 2.0: let me know how it goes! In principle, there is no reason why raw IO on LVM should not work.

--Stephen

#3 lore raw

Hi, just checking a fact. :)

IBM's ADSM(*) (well, TSM now, thanks to Tivoli) uses raw logical
volumes on a fairly native level as storage pools, etc. I'm curious, having never looked into raw IO under Linux, whether it's the same principles - and, that said, whether or not IBM will be porting the TSM server to Linux?

Regardless, LVM is a huge jump in the right direction if we want IBM to
release more schtuff for Linux.

Now, I've just got to find a Linux-HSM(**) project to control my
optical jukebox. :) Any ideas whether this exists?

(apologies for the footnotes, they're there for the benefit of people on the list who don't know these acronyms)
*: ADSM: ADSTAR Distributed Storage Management. A highly-cool enterprise backup system, scheduling, etc - IBM's version of Veritas. **: HSM: Heirarchial Storage Management. Basically, take a tape library or jukebox, make a database of everything on every cartridge, and mount that database as a filesystem. Copy something out of it, the library goes and gets the appropriate media and feeds it to the drive. With a 20-cart optical jukebox, this is basically a 20*1.3G filesystem (albiet a fairly slow one, but with a script to copy the file into a temp space, perfect as a low-use fileserver. ADSM does this, but the cost is hefty, to say the least.)

Cheers,
- Drew.
:wq

Quoted 15 lines · Stephen C. Tweedie

Hi,

Quoted 4 lines · Fernando Dammous

Now, I would like to install the devfs and mainly the raw-io.

Can I install them in a kernel 2.0.36, which is already patched with LVM?

I've never tried the raw IO code on 2.0: let me know how it goes! In principle, there is no reason why raw IO on LVM should not work.

--Stephen

#4 lore raw
Quoted 2 lines · Drew Smith

Now, I've just got to find a Linux-HSM(**) project to control my
optical jukebox. :) Any ideas whether this exists?

Ingo Molnar is doing HSM as part of linux-raid,
not working yet afaik

L.

Signature · 2 lines
Luca Berra -- [email protected]
    Communications Media & Services S.r.l.
#5 lore raw
In reply to Drew Smith

Hi,

Quoted 2 lines · Drew Smith

Now, I've just got to find a Linux-HSM(**) project to control my
optical jukebox. :) Any ideas whether this exists?

Not yet, but XFS has all of the support necessary for us to do a kernel DMAPI implementation, which would open up all sorts of HSM possibilities.

--Stephen

#6 lore raw

I've been using the Linux ADSM v3 _client_ for some time but I've never seen a server project. Even the client is still listed as unsupported from IBM although you can get it from them. It's awesome...where's the server?!?

--Eric

Quoted 13 lines · Stephen C. Tweedie

Hi,

Quoted 2 lines · Drew Smith

Now, I've just got to find a Linux-HSM(**) project to control my
optical jukebox. :) Any ideas whether this exists?

Not yet, but XFS has all of the support necessary for us to do a kernel DMAPI implementation, which would open up all sorts of HSM possibilities.

--Stephen

#8 lore raw
In reply to Stephen C. Tweedie

HI Stephen and everybody,

Quoted 5 lines

I've never tried the raw IO code on 2.0: let me know how it goes! In principle, there is no reason why raw IO on LVM should not work.

--Stephen

I prefer to update my kernel (to 2.3.18 or even to wait for the 2.4), before and then to use RAW-IO.
I just hope I can simply alter Oracle to use RAW-IO instead of having reinstall it.
Using Oracle with LVM is already very good, with RAW-IO it will be better (I believe).
In any way, I thank in advance for your interest (and of all people of here) in helping me in this case.

Thank you very much

Fernando

#9 lore raw

On Sat, Oct 02, 1999 at 02:08:09PM -0300, Fernando Dammous wrote:
<stuff deleted>

Quoted 6 lines

I prefer to update my kernel (to 2.3.18 or even to wait for the 2.4), before and then to use RAW-IO.
I just hope I can simply alter Oracle to use RAW-IO instead of having reinstall it.
Using Oracle with LVM is already very good, with RAW-IO it will be better (I believe).

I do question that. Sybase somehow needs (or better prefers) RAW-IO to ensure it's structural integrity. If the buffer cache is between the data transfer of the LOG and the physical device it is possible that not all of the log that should be written out is written out, however Sybase thinks it has been written to disk. RAW-IO/direct IO ensures that the LOG that has been ordered to be written on disk will be written to disk. I would like to emphasize that not using RAW-IO will not impose corrupting your database. It imposes that there's a risk of loosing your data in case of a failure.

Oracle has always been using filesystems for it's datastorage. Only for the parrallel server configuration they demand RAW IO. Like Sybase 11.9.x under Linux it probably uses some O_SYNC flag to write to log, so the write call only returns when the data has been really written to the physical device. I don't think you'll see a performance enhancement when using Oracle and RAW/IO. There's even a chance, due to the fact you mis the buffer cache optimisations, that you'll see a performance degradation.

This is a common misunderstanding.

Best regards,

Pascal van Dam

Quoted 8 lines

In any way, I thank in advance for your interest (and of all people of here) in helping me in this case.

Thank you very much

Fernando

#7 lore raw
In reply to Drew Smith
Quoted 5 lines · Drew Smith

IBM's ADSM(*) (well, TSM now, thanks to Tivoli) uses raw logical
volumes on a fairly native level as storage pools, etc. I'm curious, having never looked into raw IO under Linux, whether it's the same principles - and, that said, whether or not IBM will be porting the TSM server to Linux?

I'm working on that one. Ummm... should we launch a petition? I'd be willing to host a web page.

Signature · 1 line
Don't you feel more like you do now than you did when you came in?
#12 lore raw
In reply to Fernando Dammous
Quoted 5 lines · Fernando Dammous

HI Stephen and everybody,

Quoted 3 lines

I've never tried the raw IO code on 2.0: let me know how it goes! In principle, there is no reason why raw IO on LVM should not work.

Quoted 7 lines
Quoted 2 lines

--Stephen

I prefer to update my kernel (to 2.3.18 or even to wait for the 2.4), before and then to use RAW-IO.
I just hope I can simply alter Oracle to use RAW-IO instead of having reinstall it.

Forget it, Linux port of Oracle DB doesn't support raw-io.

Quoted 2 lines

Using Oracle with LVM is already very good, with RAW-IO it will be better (I believe).

Signature · 6 lines
                                             freddy

----==-- _                      Adrien "freddy" Farkas
---==---(_)__  __ ____  __       [email protected]
--==---/ / _ \/ // /\ \/ /  We all know Linux is great... It
-=====/_/_//_/\___/ /_/\_\  does infinite loops in 5 seconds
#13 lore raw

Hi (again!),

From: Adrien Farkas <[email protected]>

Quoted 1 line

Forget it, Linux port of Oracle DB doesn't support raw-io.

I admit that I'm not a deep expert of Oracle, mainly regarding interaction with the I/O devices and with the operating system.

However, I have been finding in the documentation of Oracle for Linux, several references of utilization of RAW-DEVICES (among others). I'm not, here, wanting to create controversy, because it is dealt with a matter extra-LVM and being you a member of Oracle (I saw for your e-mail address [email protected]), you must have knowledge of this subject. Then I ask:
What does mean the references about RAW-DEVICES in documentation?
To illustrate, I separated some from :

Oracle8 Administrator's Reference for LINUX
Release 8.0.5
A66585-01

<SNIP>
- Tuning of I/O Loads across all Drives
I/O loads are tuned across all drives, including drives storing Oracle data in raw devices
<SNIP>
- Raw Device Sizes
Choose a small set of standard sizes for all raw devices that may be used to store Oracle database files.

In general, standardizing on a single size is recommended. If a single size is used, raw files can be moved from one partition to another safely. The size should be small enough so that a fairly large number can be created, but large enough to be convenient.

For example, a 2 GB drive could be divided into 10 partitions of 200 MB each-a good balance between size and number. Any tablespace using raw devices should stripe them across several drives. If possible, the striping should be done with a Logical Volume Manager."
<SNIP>
- Tuning the LINUX Buffer Cache Size
To take full advantage of raw devices, adjust the size of the Oracle8 buffer cache and, if memory is limited, the LINUX buffer cache.

The LINUX buffer cache is provided by the operating system. It holds blocks of data in memory while they are being transferred from memory to disk, or vice versa.

The Oracle8 buffer cache is the area in memory that stores the Oracle database buffers. Since Oracle8 can use raw devices, it does not need to use the LINUX buffer cache.

When moving to raw devices, increase the size of the Oracle8 buffer cache. If the amount of memory on the system is limited, make a corresponding decrease in the LINUX buffer cache size.

The LINUX command vmstat may help you determine which buffer caches should be increased or decreased.
<SNIP>
------------------------

Cheers and sorry for the long mail!
Fernando

#14 lore raw
Quoted 17 lines · Fernando Dammous

Hi (again!),

From: Adrien Farkas <[email protected]>

Quoted 1 line

Forget it, Linux port of Oracle DB doesn't support raw-io.

I admit that I'm not a deep expert of Oracle, mainly regarding interaction with the I/O devices and with the operating system.

However, I have been finding in the documentation of Oracle for Linux, several references of utilization of RAW-DEVICES (among others). I'm not, here, wanting to create controversy, because it is dealt with a matter extra-LVM and being you a member of Oracle (I saw for your e-mail address [email protected]), you must have knowledge of this subject. Then I ask:
What does mean the references about RAW-DEVICES in documentation?
To illustrate, I separated some from :

Quoted 3 lines

Oracle8 Administrator's Reference for LINUX
Release 8.0.5
A66585-01

Oh, I'm sorry, Linux port of Oracle _does_ support raw io, but it depends on Oracle8 release, 8.0.5 does not, 8.0.5.1 (not released for public for free, afaik) and 8i (aka 8.1.5) both do. But definitely, I've never tried it. The last time I was interested in using raw partitions was in the time of 8.0.5, so prehaps that's why I still thought Oracle doesn't support them.

Quoted 43 lines

<SNIP>
- Tuning of I/O Loads across all Drives
I/O loads are tuned across all drives, including drives storing Oracle data in raw devices
<SNIP>
- Raw Device Sizes
Choose a small set of standard sizes for all raw devices that may be used to store Oracle database files.

In general, standardizing on a single size is recommended. If a single size is used, raw files can be moved from one partition to another safely. The size should be small enough so that a fairly large number can be created, but large enough to be convenient.

For example, a 2 GB drive could be divided into 10 partitions of 200 MB each-a good balance between size and number. Any tablespace using raw devices should stripe them across several drives. If possible, the striping should be done with a Logical Volume Manager."
<SNIP>
- Tuning the LINUX Buffer Cache Size
To take full advantage of raw devices, adjust the size of the Oracle8 buffer cache and, if memory is limited, the LINUX buffer cache.

The LINUX buffer cache is provided by the operating system. It holds blocks of data in memory while they are being transferred from memory to disk, or vice versa.

The Oracle8 buffer cache is the area in memory that stores the Oracle database buffers. Since Oracle8 can use raw devices, it does not need to use the LINUX buffer cache.

When moving to raw devices, increase the size of the Oracle8 buffer cache. If the amount of memory on the system is limited, make a corresponding decrease in the LINUX buffer cache size.

The LINUX command vmstat may help you determine which buffer caches should be increased or decreased.
<SNIP>
------------------------

Cheers and sorry for the long mail!
Fernando

Signature · 6 lines
                                             freddy

----==-- _                      Adrien "freddy" Farkas
---==---(_)__  __ ____  __       [email protected]
--==---/ / _ \/ // /\ \/ /  We all know Linux is great... It
-=====/_/_//_/\___/ /_/\_\  does infinite loops in 5 seconds
#10 lore raw
In reply to [email protected]

Hi,

Quoted 8 lines · On Sun, 3 Oct 1999 07:56:44 +0200, [email protected] said:

Oracle has always been using filesystems for it's datastorage. Only for the parrallel server configuration they demand RAW IO. Like Sybase 11.9.x under Linux it probably uses some O_SYNC flag to write to log, so the write call only returns when the data has been really written to the physical device. I don't think you'll see a performance enhancement when using Oracle and RAW/IO. There's even a chance, due to the fact you mis the buffer cache optimisations, that you'll see a performance degradation.

It depends. Raw IO uses less CPU --- much less in some cases. It also avoids polluting memory with extra copies of data in the buffer cache. As a result, just measuring the pure disk bandwidth of raw IO versus buffered IO does not tell you the whole story about performance: you really need to measure the application's performance itself, including the effect of CPU and memory pressure.

--Stephen

#11 lore raw
In reply to Stephen C. Tweedie
Quoted 13 lines · Stephen C. Tweedie

Hi,

Quoted 8 lines · On Sun, 3 Oct 1999 07:56:44 +0200, [email protected] said:

Oracle has always been using filesystems for it's datastorage. Only for the parrallel server configuration they demand RAW IO. Like Sybase 11.9.x under Linux it probably uses some O_SYNC flag to write to log, so the write call only returns when the data has been really written to the physical device. I don't think you'll see a performance enhancement when using Oracle and RAW/IO. There's even a chance, due to the fact you mis the buffer cache optimisations, that you'll see a performance degradation.

Quoted 6 lines

It depends. Raw IO uses less CPU --- much less in some cases. It also avoids polluting memory with extra copies of data in the buffer cache. As a result, just measuring the pure disk bandwidth of raw IO versus buffered IO does not tell you the whole story about performance: you really need to measure the application's performance itself, including the effect of CPU and memory pressure.

You're right. I did oversee the buffercache pollution issue. Thanks for pointing it out.

Best regards,

Pascal

Quoted 2 lines

--Stephen