Virtual File Drivers for a VoD(Video on Demand) application Project

Kind Attn: Heinz Mauelshagen

Dear Sir,

I was going through the information posted by you on the Internet regarding the Heinz Mauelshagen's LVM (Logical Volume Manager) for Linux.
Hence I am assuming that a lot of work has been done from you guys in this direction.

We are a Bangalore, India based Software development company looking
at options to develop the Virtual File Driver for a VoD (Video on Demand) application. This Driver has to communicate with a Windows Media Player 9 on one end and a Temporary buffer at the other end.

I have to stress the point that we need to use a Striping File system.

Some of the File System specifications are as follows:

Block size : 4096 bytes
Large File size: 1 G byte
Number of Maximum Large files: 1M
Number of Maximum nodes: 4096 nodes

We are looking for open source options. Please let us know your
comments and suggestions about the availabilty of such Software and also the licensing terms and conditions.

Thanks & Regards,
Bala

Never heard of a Striping File System.

I have heard of File Systems which can be optimized (or tuned) to be layered above a Striped Logical Volume.

Typically the striping is implemented in a hardware or software raid solution.

In addition, LVM can stripe a LV (Logical Volume) over multiple PVs (Physical Volumes).

Once you have your logical volume created with the striping in place via one of the above 3 choices, you do the mkfs and create/install the filesystem.

1 GB per file with a million files is an Exabyte. That is certainly pushing the limits of todays technology and is not supported in any released vanilla kernel.org kernel. I believe it is supported by the 2.6.0-test series of kernels, but I have not heard of anyone building a filesystem that big yet.

I don't know how large DM and LVM2 can scale in the 2.6.0-test7 kernel. I believe 2 TB has been the traditional barrier with the 2.4 kernel.

How big of a single FS are you really taking about?

What do you mean by a node?

If that is 4096 computers, I would look at Lustre.

From their website "The 1.0 release of Lustre will happen in late 2003 and will target clusters of up to 1,000 nodes with 100 TBs of storage."

IIRC, the US DOD is sponsoring Lustre and HP is doing most of the implementation.

I don't know if Lustre uses any LVM or DM technology, but personally I doubt it.

For a single computer solution, the XFS filesystem is generally considered the best Linux FS for Video stream distribution.

I have spec'ed a 6 TB XFS filesystem on a single computer for my companies R&D work. It looks very doable from a hardware perspective, but now I have to get the money to buy all the hardware and see if it works in reality.

My concept would be to use hardware raid to create 6 1 TB volumes. Then use either software raid or DM to stripe those together into a single 6 TB volume.

FYI: I'm not familiar with any hardware Raid controller that supports volumes larger than 2 TB., and my gut feel is that sticking to 1 TB and below is safer than trying to push the technology.

Alos, if the DM in the 2.6.0-test kernel can support 6TBs, then I will likely have it in the mix as well.

All of the above is GPL'ed to my knowledge.

IANAL:
It sounds like you want to be a service provider, so I don't know what impact the GPL will have on you. Certainly you can use the above at no cost, but you may be required to make accessible to your customers the original source code plus any patches to the above you incorporate.

If so, your patches would also have to covered by the GPL, and thus could be incorporated back into the original at the discretion of the individual package maintainers.

Greg

Greg Freemyer

Le 15.10.2003 18:46, Greg Freemyer a �crit�:

I think 1Gb per file with 1M files is "only" one Petabyte?

--
Regards
- Jean-Luc

Your right, I got my terms confused.

Unfortunately I have to live in the world of GB most of the time, with the occasional TB.

Petabytes and Exabytes are both beyond my reality.