Skip to contentSkip to conversation

Multiplexing filesystem

DISCUSSION Mark Richards 8 participants 12 messages Nov 27, 2001 – Dec 2, 2001

lore
Share & tools
lore
#0 lore raw

Quick question, which I suspect has a long answer.

I would like to write a multiplexing filesystem. The idea is as follows:

The filesystem would ideally wrap another filesystem, such as nfs or smbfs or ext2. Most operations would just be passed to the native fs call. However, for some files, selectable at run time by some control singal, would actually reside on another file system. The other filesystem would have to be mounted.

The idea is for a version controlling filesystem. The server would be a network server (hence the desire to wrap nfs) which presents a 'view' of the source code. When the user reserves a file for editing, the file is copied to the local disk. From that point on, the local file is referred to until the user commits the change or unreserves the file. Ideally, the local copy of the file could be on any file system, not one that is necessarily local. And this has to be totally transparent to the user, except for the step where the user 'reserves' the file.

I've thought about two ways to do this. One is to wrap the 'versioning' file system with a multiplexor that checks fs calls to see if they are referring to a file that is on a different fs. The other approach is to intercept calls to the VFS to do the same trick.

I'm new to the whole filesystem-coding thing, so bear with me if what i've just said makes no sense. So, my question (I guess it wasn't quick after all) is:
Can it be done, and are either of my two approaches feasible? Any suggestions or tips?

Thanks,
Mark Richards

PS please CC me if possible.

#1 lore raw

You may want to look at FS filter drivers in Windows 2000 as a good model for something like this. Intermezzo also seems to do something a lot like this, but not to the Windows 2000 degree. W2K has something called a filter driver interface, which is super handy for encryption, and going remote.

The filter driver idea is pretty cool, and MS has been able to implement some pretty cool stuff with it, like snap in virus detection software across different file systems. They also did something called a "mini-port" file system, which was a filter driver on top of their very complex IFS interface which made writing remote file systems, like network clients, as lot easier and faster.

The disadvantage to their approach was that several new and nasty "viruses" showed up as filter drivers and used this interface to trash file systems.

Food for thought.

:-)

Jeff

Quoted 38 lines · Mark Richards

Quick question, which I suspect has a long answer.

I would like to write a multiplexing filesystem. The idea is as follows:

The filesystem would ideally wrap another filesystem, such as nfs or smbfs or ext2. Most operations would just be passed to the native fs call. However, for some files, selectable at run time by some control singal, would actually reside on another file system. The other filesystem would have to be mounted.

The idea is for a version controlling filesystem. The server would be a network server (hence the desire to wrap nfs) which presents a 'view' of the source code. When the user reserves a file for editing, the file is copied to the local disk. From that point on, the local file is referred to until the user commits the change or unreserves the file. Ideally, the local copy of the file could be on any file system, not one that is necessarily local. And this has to be totally transparent to the user, except for the step where the user 'reserves' the file.

I've thought about two ways to do this. One is to wrap the 'versioning' file system with a multiplexor that checks fs calls to see if they are referring to a file that is on a different fs. The other approach is to intercept calls to the VFS to do the same trick.

I'm new to the whole filesystem-coding thing, so bear with me if what i've just said makes no sense. So, my question (I guess it wasn't quick after all) is:
Can it be done, and are either of my two approaches feasible? Any suggestions or tips?

Thanks,
Mark Richards

PS please CC me if possible.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#2 lore raw
In reply to Mark Richards

Oh yeah. The MS Distributed File System in W2K is actually a filter driver on top of their existings FS's. So is the mail slot interface.

Jeff

Quoted 66 lines · Jeff V. Merkey

You may want to look at FS filter drivers in Windows 2000 as a good model for something like this. Intermezzo also seems to do something a lot like this, but not to the Windows 2000 degree. W2K has something called a filter driver interface, which is super handy for encryption, and going remote.

The filter driver idea is pretty cool, and MS has been able to implement some pretty cool stuff with it, like snap in virus detection software across different file systems. They also did something called a "mini-port" file system, which was a filter driver on top of their very complex IFS interface which made writing remote file systems, like network clients, as lot easier and faster.

The disadvantage to their approach was that several new and nasty "viruses" showed up as filter drivers and used this interface to trash file systems.

Food for thought.

:-)

Jeff

Quoted 38 lines · Mark Richards

Quick question, which I suspect has a long answer.

I would like to write a multiplexing filesystem. The idea is as follows:

The filesystem would ideally wrap another filesystem, such as nfs or smbfs or ext2. Most operations would just be passed to the native fs call. However, for some files, selectable at run time by some control singal, would actually reside on another file system. The other filesystem would have to be mounted.

The idea is for a version controlling filesystem. The server would be a network server (hence the desire to wrap nfs) which presents a 'view' of the source code. When the user reserves a file for editing, the file is copied to the local disk. From that point on, the local file is referred to until the user commits the change or unreserves the file. Ideally, the local copy of the file could be on any file system, not one that is necessarily local. And this has to be totally transparent to the user, except for the step where the user 'reserves' the file.

I've thought about two ways to do this. One is to wrap the 'versioning' file system with a multiplexor that checks fs calls to see if they are referring to a file that is on a different fs. The other approach is to intercept calls to the VFS to do the same trick.

I'm new to the whole filesystem-coding thing, so bear with me if what i've just said makes no sense. So, my question (I guess it wasn't quick after all) is:
Can it be done, and are either of my two approaches feasible? Any suggestions or tips?

Thanks,
Mark Richards

PS please CC me if possible.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#3 lore raw
In reply to Mark Richards

What about implimenting something like KDE's IOSlaves at kernel level? you could then create a symlink to any recognized URL. This would also have plenty of other uses.

Quoted 32 lines · Mark Richards

Quick question, which I suspect has a long answer.

I would like to write a multiplexing filesystem. The idea is as follows:

The filesystem would ideally wrap another filesystem, such as nfs or smbfs or ext2. Most operations would just be passed to the native fs call. However, for some files, selectable at run time by some control singal, would actually reside on another file system. The other filesystem would have to be mounted.

The idea is for a version controlling filesystem. The server would be a network server (hence the desire to wrap nfs) which presents a 'view' of the source code. When the user reserves a file for editing, the file is copied to the local disk. From that point on, the local file is referred to until the user commits the change or unreserves the file. Ideally, the local copy of the file could be on any file system, not one that is necessarily local. And this has to be totally transparent to the user, except for the step where the user 'reserves' the file.

I've thought about two ways to do this. One is to wrap the 'versioning' file system with a multiplexor that checks fs calls to see if they are referring to a file that is on a different fs. The other approach is to intercept calls to the VFS to do the same trick.

I'm new to the whole filesystem-coding thing, so bear with me if what i've just said makes no sense. So, my question (I guess it wasn't quick after all) is: Can it be done, and are either of my two approaches feasible? Any suggestions or tips?

Thanks,
Mark Richards

Quoted 7 lines

PS please CC me if possible.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

#4 lore raw
In reply to Mark Richards
Quoted 10 lines · Mark Richards

Quick question, which I suspect has a long answer.

I would like to write a multiplexing filesystem. The idea is as follows:

The filesystem would ideally wrap another filesystem, such as nfs or smbfs or ext2. Most operations would just be passed to the native fs call. However, for some files, selectable at run time by some control singal, would actually reside on another file system. The other filesystem would have to be mounted.

Quoted 8 lines

The idea is for a version controlling filesystem. The server would be a network server (hence the desire to wrap nfs) which presents a 'view' of the source code. When the user reserves a file for editing, the file is copied to the local disk. From that point on, the local file is referred to until the user commits the change or unreserves the file. Ideally, the local copy of the file could be on any file system, not one that is necessarily local. And this has to be totally transparent to the user, except for the step where the user 'reserves' the file.

Coda already do what you want:
Files are kept on a server, and copied to your local disk when you use it. You may even disconnect when working on the local copy - your changes will be propagated back to the server whenever you reconnect to the network.
The copying is indeed completely transparent.

If you need reservation - use the permission system.
A suid program simply makes _you_ owner, and only
the owner gets write permission. This is your check-out program. Check-in consists of changing ownership
back to root (or some userid allocated to the versioning system)

Helge Hafting

#5 lore raw
In reply to Mark Richards
Quoted 8 lines · Helge Hafting

Coda already do what you want:
Files are kept on a server, and copied to your local disk when you use it. You may even disconnect when working on the local copy - your changes will be propagated back to the server whenever you reconnect to the network.
The copying is indeed completely transparent.

Quoted 7 lines

If you need reservation - use the permission system.
A suid program simply makes _you_ owner, and only
the owner gets write permission. This is your check-out program. Check-in consists of changing ownership
back to root (or some userid allocated to the versioning system)

Helge Hafting

I'll look into Coda, but ideally I wouldn't have to copy each file to the local workstation when I use it, only when it is reserved for editing. Also, I'd like to be able to store the local copy anywhere on the filesystem, if possible.

Mark

#6 lore raw
In reply to Mark Richards
Quoted 24 lines · Mark Richards

Quick question, which I suspect has a long answer.

I would like to write a multiplexing filesystem. The idea is as follows:

The filesystem would ideally wrap another filesystem, such as nfs or smbfs or ext2. Most operations would just be passed to the native fs call. However, for some files, selectable at run time by some control singal, would actually reside on another file system. The other filesystem would have to be mounted.

The idea is for a version controlling filesystem. The server would be a network server (hence the desire to wrap nfs) which presents a 'view' of the source code. When the user reserves a file for editing, the file is copied to the local disk. From that point on, the local file is referred to until the user commits the change or unreserves the file. Ideally, the local copy of the file could be on any file system, not one that is necessarily local. And this has to be totally transparent to the user, except for the step where the user 'reserves' the file.

I've thought about two ways to do this. One is to wrap the 'versioning' file system with a multiplexor that checks fs calls to see if they are referring to a file that is on a different fs. The other approach is to intercept calls to the VFS to do the same trick.

Quoted 4 lines

I'm new to the whole filesystem-coding thing, so bear with me if what i've just said makes no sense. So, my question (I guess it wasn't quick after all) is:
Can it be done, and are either of my two approaches feasible? Any suggestions or tips?

I've heard of 'transparent' or 'stackable' filesystems that let you mount a set of filesystems on top of each other, and changes are made to the top level filesystem. (I think the BSDs might have (had) that...) If you were to use something like that, the 'reserve' step would be transparent to the user, but the 'commit' (to the lower layer fs) would not be. (Mount the lower layer in some other place as well and write to it for your commit.)

You might want to consider that approach since I think you would find a number of other people would have other uses for the same filesystem. (And thus you might get more help, and have a chance of getting it into (one of) the standard kernel(s).)
Hmm... it just occurred to me... multisession CDs do something similar... That might have some benefits for those who do multisession cd mastering--and that gives you more potential contributors.

Just some thoughts... have fun!

Eli
--------------------. Real Users find the one combination of bizarre
Eli Carter \ input values that shuts down the system for days.
eli.carter(a)inet.com `-------------------------------------------------

#7 lore raw
In reply to Mark Richards
Quoted 16 lines · Mark Richards
Quoted 14 lines · Helge Hafting

Coda already do what you want:
Files are kept on a server, and copied to your local disk when you use it. You may even disconnect when working on the local copy - your changes will be propagated back to the server whenever you reconnect to the network.
The copying is indeed completely transparent.

If you need reservation - use the permission system.
A suid program simply makes _you_ owner, and only
the owner gets write permission. This is your check-out program. Check-in consists of changing ownership
back to root (or some userid allocated to the versioning system)

Quoted 6 lines
Quoted 1 line

Helge Hafting

I'll look into Coda, but ideally I wouldn't have to copy each file to the local workstation when I use it, only when it is reserved for editing. Also, I'd like to be able to store the local copy anywhere on the filesystem, if possible.

Don't know much about coda, but...

Helge has basically said that the files *are* copied to your system transparently (probably to a configurable location) and once you are finished it will be synced with the coda server.

If you need *exclusive* access, just check out (change permissions to you as owner, and chmod 0600), and back when you're done. This is probably done with root suid tools to be able to chown.

MF

#8 lore raw
In reply to Mark Richards

You may want to look into utilizing CVS for the check-in/check-out mechanism and just virtualizing that layer on top of whatever FS is being used....

This may be a good place to start looking.
http://sourceforge.net/projects/cvsfs/

If you don't need all the sophistication of CVS perhaps RCS would do.

The way Coda does it is by "hoarding" files, you specify what files you want to hoard, and it refers to those as local only, when you "un-hoard" them it syncs them with the server (I think this is the process, it's been a while since I played with it.)

I think intermezzo has something similar, but neither does any revision control.

Quoted 38 lines · Mark Richards

Quick question, which I suspect has a long answer.

I would like to write a multiplexing filesystem. The idea is as follows:

The filesystem would ideally wrap another filesystem, such as nfs or smbfs or ext2. Most operations would just be passed to the native fs call. However, for some files, selectable at run time by some control singal, would actually reside on another file system. The other filesystem would have to be mounted.

The idea is for a version controlling filesystem. The server would be a network server (hence the desire to wrap nfs) which presents a 'view' of the source code. When the user reserves a file for editing, the file is copied to the local disk. From that point on, the local file is referred to until the user commits the change or unreserves the file. Ideally, the local copy of the file could be on any file system, not one that is necessarily local. And this has to be totally transparent to the user, except for the step where the user 'reserves' the file.

I've thought about two ways to do this. One is to wrap the 'versioning' file system with a multiplexor that checks fs calls to see if they are referring to a file that is on a different fs. The other approach is to intercept calls to the VFS to do the same trick.

I'm new to the whole filesystem-coding thing, so bear with me if what i've just said makes no sense. So, my question (I guess it wasn't quick after all) is:
Can it be done, and are either of my two approaches feasible? Any suggestions or tips?

Thanks,
Mark Richards

PS please CC me if possible.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 5 lines
Blake Barnett (bdb)  <[email protected]>
Sr. Unix Administrator
DevelopOnline.com                 office: 480-377-6816

"Do, or do not.  There is no try." --Yoda
#9 lore raw
In reply to Mark Richards

Mark Richards wrote:
[...]

Quoted 3 lines

I'll look into Coda, but ideally I wouldn't have to copy each file to the local workstation when I use it, only when it is reserved for editing. Also, I'd like to be able to store the local copy anywhere on the filesystem, if possible.

Worried your drive will fill up? The files copied to your drive is merely copied as a "caching" operation. They still seem to reside at the server - this is totally transparent. And of course you can limit this caching - if too many files is cached some is simply thrown away. (Or sent back
if they were changed.) They will be re-
loaded automatically if you ever need them again.

If you really want to store them where you want instead of transparent access, why bother with a new FS at all? (I believe coda lets you specify where the caching will happen, if you have several partitions/drives)

Simply run a script that reserves the file (by using the permission system) and *copy* it to
where you want. Check-in will consist of copying the altered file back, and restore normal permissions.

You might even want to run a system like CVS, unless there is some special reason for not doing that.

Helge Hafting

#10 lore raw
In reply to Mark Richards
Quoted 19 lines · Helge Hafting

Mark Richards wrote:
[...]

Quoted 3 lines

I'll look into Coda, but ideally I wouldn't have to copy each file to the local workstation when I use it, only when it is reserved for editing. Also, I'd like to be able to store the local copy anywhere on the filesystem, if possible.

Worried your drive will fill up? The files copied to your drive is merely copied as a "caching" operation. They still seem to reside at the server - this is totally transparent. And of course you can limit this caching - if too many files is cached some is simply thrown away. (Or sent back
if they were changed.) They will be re-
loaded automatically if you ever need them again.

If you really want to store them where you want instead of transparent access, why bother with a new FS at all? (I believe coda lets you specify where the caching will happen, if you have several partitions/drives)

Quoted 7 lines

Simply run a script that reserves the file (by using the permission system) and *copy* it to
where you want. Check-in will consist of copying the altered file back, and restore normal permissions.

You might even want to run a system like CVS, unless there is some special reason for not doing that.

Heh, it's funny that you mention that, because one of the goals of this project is to access a CVS-like repository as if it were a filesystem. I suppose that i'm not opposed to on-disk caching of files; this may improve performance in some cases. However, the specific feature I need right now is that a file being modified by one person is explicitly not seen by other people (they see the original version). The parts that need to be transparent to the user are the copying/moving of the file from one place (network) to another (somewhere else). Also, files created locally should not be propagated back to the network unless the users explicitly specifies that they should. So, this system is not totally transparent to the user (because we can't read the user's mind to figure out what files (s)he would like to edit or 'check in'), but the filesystem part of it should be. Plus, it's important that local files and network files appear in the same directory side by side, as if they were in the same directory on the server.

I still haven't had a chance to look at Coda; maybe it does solve my problem, but probably not quite.

Mark

Quoted 8 lines

Helge Hafting
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#11 lore raw
In reply to Mark Richards

Hi!

Quoted 3 lines

What about implimenting something like KDE's IOSlaves at kernel level? you could then create a symlink to any recognized URL. This would also have plenty of other uses.

uservfs.sf.net

Signature · 2 lines
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.