Skip to contentSkip to conversation

Developing a Distributed File System

DISCUSSION UZAIR LAKHANI 2 participants 4 messages Jul 10, 2006 – Jul 11, 2006 VFS

lore
Share & tools
lore
#0 lore raw

Hello All,

Our team want to develop a distributed file system, a file system like NFS or in other words a remote file access system.

In our design we have multiple clients and a single server. I want to identify main issues involved. What I found after doing some study is this:

(1) Handling of multiple clients (concurrent read/write problem)

(2) Continuation of (1) i.e. the distributed lock management

(3) For performance reasons caching on the client side and the cache coherency problem
(this can be avoided for our prototype release)

(4) Data integrity problem.

To solve these issues, I think I have to study NFS, AFS and coda. Also I am doing general googling for finding out distributed file system issues.

Since both linux and fs (especially a distributed one) are new to me, I want help in this regard that how I start this development of a prototype and which documents/books to study for this purpose.

Thanks,
Uzair Lakhani,
Karachi, Pakistan.

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

#1 lore raw
Quoted 3 lines · UZAIR LAKHANI

Our team want to develop a distributed file system, a file system like NFS or in other words a remote file access system.

First you need to explain why you want to develop a new distributed file system? What is wrong with any of the existing filesystems?

Quoted 4 lines

Since both linux and fs (especially a distributed one) are new to me, I want help in this regard that how I start this development of a prototype and which documents/books to study for this purpose.

This is by no means an easy task to start with. How about fixing some bug or simple feature to implement on an existing filesystem (even a local one) before tackling the hardest problem first?

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

#2 lore raw
Quoted 10 lines · --- Andreas Dilger
Quoted 1 line · UZAIR LAKHANI

Our team want to develop a distributed file

system, a

Quoted 1 line

file system like NFS or in other words a remote

file

Quoted 1 line

access system.

First you need to explain why you want to develop a new distributed file system? What is wrong with any of the existing filesystems?

Hello All,
Thanks for the reply. Basically we want to move towards developing a fail-over and high performance cluster file system. Although in the simplest design in which there will be a master server and a backup server.

Therefore we first have to figure out the issues involved in a distributed file system (single server and multiple clients).

You may have noticed earlier that I had asked questions regarding the definition of a cluster file system. This was for the same reason.

Quoted 15 lines
Quoted 2 lines

Since both linux and fs (especially a distributed

one)

Quoted 1 line

are new to me, I want help in this regard that how

I

Quoted 2 lines

start this development of a prototype and which
documents/books to study for this purpose.

This is by no means an easy task to start with. How about
fixing some bug or simple feature to implement on an existing filesystem (even a local one) before tackling
the hardest problem first?

Quoted 5 lines

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Thanks,
Uzair Lakhani,
Karachi, Pakistan.

Quoted 8 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

#3 lore raw
Quoted 9 lines · UZAIR LAKHANI

Thanks for the reply. Basically we want to move towards developing a fail-over and high performance cluster file system. Although in the simplest design in which there will be a master server and a backup server.

Therefore we first have to figure out the issues involved in a distributed file system (single server and multiple clients).

I hate to state the obvious, but this is a VERY HARD PROBLEM.

I may be biased in sugessing this, but why not spend your time and efforts in improving some existing cluster filesystem instead of spending 20 work years of effort just to get a decently usable system? That way, instead of the ever-popular process of having 20 mediocre implementations of something Linux instead gains a small number of very good implementations?

Possible targets of your efforts might include OCFS2, Lustre, GFS2.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.