Skip to contentSkip to conversation

Visionary ideas for SQL file systems

DISCUSSION Marc Perkel 5 participants 6 messages Jun 14, 2006

lore
Share & tools
lore
#0 lore raw

I'm going to throw this idea out there just to get people thinking. There's nothing in reality that is like this except maybe some of the ReiserFS ideas, but I want to take the idea farther. the idea is ......

Why not put an SQL filesystem directly on a block devices where files are really blobs within the filesystem and file names and file attributes are all indexed data withing the SQL database. The operating system will have SQL built in.

Right now we have a variety of name spaces, file attributes, cluster sises, inodes and other nasty stuff that are too exposed. Suppose that you could add any fileds you want, any keys you want. Suppose that users and groups could have any number of fields. Suppose you wanted to add more levels like "managers" and some of the fancy Novell stuff. With a database the user could create any kind of an interface to access files that they want.

Picture this. When listing a directory how do we determine who gets to see what file names? I siggest that the rule be an SQL query that the system owner can configure anyway they want. That way you can set it up so that the users only see the file names that they have access to and you could emulate Linux, or you could emulate Windows, or you could emulate Netware, or each directory could have it's own embedded rules that are itself stored within the database.

Now - we hav files that we can read, write, lock, create, delete, etc. The file appear to be a colection of bytes, but what if they aren't really a collection of bytes? Suppose what appeard as a text file was really the output of a query that created what looks like a file but eack line was a record in an SQL database? Writing the file might not be storing bytes but rather storing rows in a database. The reading and writing of the file would be controlled by the files read query and write query, So if you want it to work like today's files then you are reading and writing a blob. But that would be just one of many options.

For example, if you are using an embedded query to read and write files the lines of one file might also be lines in a different file that has a query that intersects the same data. So if you write a line in one file you could change the corisponding line in another file that includes the same data element. So if your writing a program and you change the name of an include file then everything that references that file changes the moment you rename it.

A tar file wouldn't be a separate file. It might just be a query that creates a tar file view of other existing files. By creating a name with a .tar extension and pointing it to a directory makes a tar view of an existing directory. But you can edit the contents of the tar file by editing the files withing the directory that the tar query points to.

So - this is totally outside the bix thinking but use you imagination and envision what could be done if we lose the file system paradyme and embrace the SQL based data paradhyme.

Will it be faster? Doing only what we are limited to today, no. Doing what we would be able to do, yes. This is a radically new concept and you should be very stoned to fully appreciate it. I just wanted to throw the idea out there so that people can start rolling it around and thinking about it. It's an idea that is similar in some ways to the /proc filesystem where things appear as files that aren't

My 2 cents ....

#1 lore raw
Quoted 3 lines · Marc Perkel

I'm going to throw this idea out there just to get people thinking. There's nothing in reality that is like this except maybe some of the ReiserFS ideas, but I want to take the idea farther. the idea is ......

I personally wish for an in-kernel Quake, and would very much like somebody else to code it.
More seriously, your visonary idea has been discussed regularly on this list, and the outcome is always the same: it's better done in userspace.

Xav

#2 lore raw
In reply to Marc Perkel
Quoted 5 lines

Why not put an SQL filesystem directly on a block devices where files are really blobs within the filesystem and file names and file attributes are all indexed data withing the SQL database. The operating system will have SQL built in.

Yet another potential security hole. A kernel oops/panic or even exploit is more critical than a userspace exploit. Hey, mysql for example runs as unprivilegued, but can your kernel do that? Hardly, I suppose.

Quoted 9 lines

Right now we have a variety of name spaces, file attributes, cluster sises, inodes and other nasty stuff that are too exposed. Suppose that you could add any fileds you want, any keys you want. Suppose that users and groups could have any number of fields. Suppose you wanted to add more levels like "managers" and some of the fancy Novell stuff. With a database the user could create any kind of an interface to access files that they want.

Picture this. When listing a directory how do we determine who gets to see what file names?

Why not just add another directory which is read-protected for all unknown users? At least that's what suffices for me.

Quoted 9 lines

I siggest that the rule be an SQL query that the system owner can configure anyway they want. That way you can set it up so that the users only see the file names that they have access to and you could emulate Linux, or you could emulate Windows, or you could emulate Netware, or each directory could have it's own embedded rules that are itself stored within the database.

Now - we hav files that we can read, write, lock, create, delete, etc. The file appear to be a colection of bytes, but what if they aren't really a collection of bytes?

Unpredictable application behavior may arise.

Quoted 7 lines

Suppose what appeard as a text file was really the output of a query that created what looks like a file but eack line was a record in an SQL database? Writing the file might not be storing bytes but rather storing rows in a database. The reading and writing of the file would be controlled by the files read query and write query, So if you want it to work like today's files then you are reading and writing a blob. But that would be just one of many options.

IIRC databases are better off storing simple data rather than blobs. Too bad most files are rather big.

Quoted 4 lines

For example, if you are using an embedded query to read and write files the lines of one file might also be lines in a different file that has a query that intersects the same data. So if you write a line in one file you could change the corisponding line in another file that includes the same data element.

Unpredictable application behavior may arise.

Quoted 8 lines

So if your writing a program and you change the name of an include file then everything that references that file changes the moment you rename it.

A tar file wouldn't be a separate file. It might just be a query that creates a tar file view of other existing files. By creating a name with a .tar extension and pointing it to a directory makes a tar view of an existing directory. But you can edit the contents of the tar file by editing the files withing the directory that the tar query points to.

An SQL server generating tar archives? Goes against Unix philosophy that one thing should only do one thing, and do that thing well.

Quoted 10 lines

So - this is totally outside the bix thinking but use you imagination and envision what could be done if we lose the file system paradyme and embrace the SQL based data paradhyme.

Will it be faster? Doing only what we are limited to today, no. Doing what we would be able to do, yes. This is a radically new concept and you should be very stoned to fully appreciate it. I just wanted to throw the idea out there so that people can start rolling it around and thinking about it. It's an idea that is similar in some ways to the /proc filesystem where things appear as files that aren't

that are not what? Real? That is because it lists information about dynamic "things", processors, memory state, etc. But files are mostly static (esp. file and ftp servers). The SQL approach for such fileservers seems to me like the outer maximum opposite idea to XFS realtime.

Jan Engelhardt

#3 lore raw
In reply to Marc Perkel
Quoted 3 lines · Marc Perkel

I'm going to throw this idea out there just to get people thinking. There's nothing in reality that is like this except maybe some of the ReiserFS ideas, but I want to take the idea farther. the idea is ......

Why not put an SQL filesystem directly on a block devices where files are really blobs within the filesystem and file names and file attributes are all indexed data withing the SQL database. The operating system will have SQL built in.

Maybe it is better done by fuse? So, did you consider contributing to RelFS or something like it? Try FUSE.sf.net to find out the present fs-in-usermode projects.

#4 lore raw
Quoted 6 lines · Michael Raskin
Quoted 3 lines · Marc Perkel

I'm going to throw this idea out there just to get people thinking. There's nothing in reality that is like this except maybe some of the ReiserFS ideas, but I want to take the idea farther. the idea is ......

Why not put an SQL filesystem directly on a block devices where files are really blobs within the filesystem and file names and file attributes are all indexed data withing the SQL database. The operating system will have SQL built in.

Maybe it is better done by fuse? So, did you consider contributing to RelFS or something like it? Try FUSE.sf.net to find out the present fs-in-usermode projects.

That certianly looks like a good tool to start experimenting with.

#5 lore raw
In reply to Marc Perkel
Quoted 3 lines · Marc Perkel

I'm going to throw this idea out there just to get people thinking. There's nothing in reality that is like this except maybe some of the ReiserFS ideas, but I want to take the idea farther. the idea is ......

recommend reading hans' future_vision.html paper at namesys.com, as well as dominic's "FS design with the Be file system" which is a pdf somewhere online now. some intense study of ADT's is also recommended

Quoted 4 lines

Why not put an SQL filesystem directly on a block devices where files are really blobs within the filesystem and file names and file attributes are all indexed data withing the SQL database. The operating system will have SQL built in.

after going over the suggested reading above, you'll see why SQL is less than ideal

Quoted 7 lines

Right now we have a variety of name spaces, file attributes, cluster sises, inodes and other nasty stuff that are too exposed. Suppose that you could add any fileds you want, any keys you want. Suppose that users and groups could have any number of fields. Suppose you wanted to add more levels like "managers" and some of the fancy Novell stuff. With a database the user could create any kind of an interface to access files that they want.

this goal is very worthwhile, but it's a lot harder than you imagine, because of compatability, performance, security, and backwards compatability.

<snip examples of nifty applications of the above>

Quoted 7 lines

So - this is totally outside the bix thinking but use you imagination and envision what could be done if we lose the file system paradyme and embrace the SQL based data paradhyme.

Will it be faster? Doing only what we are limited to today, no. Doing what we would be able to do, yes. This is a radically new concept and you should be very stoned to fully appreciate it. I just wanted to throw

lol

Quoted 3 lines

the idea out there so that people can start rolling it around and thinking about it. It's an idea that is similar in some ways to the /proc filesystem where things appear as files that aren't

others have suggested that all this belongs in user-space. hans and i argue that point every once in a while still. i think it comes down to -ENOPATCH

NATE