OCFS2

Hello,

This is OCFS2, a shared disk cluster file system which we hope will be included in the kernel.

We think OCFS2 has many qualities which make it particularly interesting as a cluster file system:

-We've tried very hard to avoid the abstraction layers which tend to
plague other cluster file systems. Wherever possible, OCFS2 sits directly atop the VFS. Journaling is done via JBD, including node recovery.

-Though OCFS2 is an extent based file system, much of the disk layout
design has been taken from Ext3. As mentioned before, journal formats are JBD. Though OCFS2 does not yet support htree, directory data looks nearly identical to that of Ext3, with the exception that OCFS2 inode numbers are 64 bits wide. Data and meta data are allocated from dynamically growable "chain allocators" whose chain items look very similar to block groups.

-OCFS2 has a strong data locking model, which includes a shared mmap
implementation (shared writeable mappings are not yet supported) and full AIO support.

-OCFS2 has a very low overhead direct I/O path.

Within the file system, the cluster stack below us has been abstracted out as much as possible:

-Heartbeat events are delivered to us via a simple callback mechanism.

-File system network communication has only minimal knowledge of node
information.

-File system cluster locking, including lock caching, is handled
through a "dlm glue" layer which abstracts away the actual dlm implementation beneath (though it assumes a VMS-like API with a minimal set of features).

Included in OCFS2 is a small cluster stack. It should be noted that we do *not* propose this as a generic cluster stack. While we hope the cluster stack can be useful to other projects it was certainly implemented with only the requirements of a cluster file system in mind. Looking towards the future, we would like to either extend this stack to be more generic or plug in a more generic stack - assuming it gives us similar ease of use and performance and has undergone the appropriate community discussion. For now the cluster stack and dlm components have been placed in their respective directories within the "ocfs2" subdirectory.

A major goal in the design of OCFS2 was to make configuration of the entire stack as painless as possible. There is only one configuration file required which describes all nodes in an OCFS2 cluster (/etc/ocfs2/cluster.conf). The cluster.conf file is the same for all nodes so that it can be easily re-copied when changed.

Anyone wanting to use OCFS2 should download the tool chain from
http://oss.oracle.com/projects/ocfs2-tools/source.html The current OCFS2-tools package includes a full set of standard file system utilities (mkfs.ocfs2, fsck.ocfs2, mount.ocfs2, tunefs.ocfs2, debugfs.ocfs). Pulling down the latest subversion tree is usually best. Setup is quite simple, after software installation, either hand create a configuration or use the "ocfs2console" program to generate one for you.

Currently the code should be considered beta quality. The OCFS2 team is deep in a cycle of bug fixing and performance evaluation. As a proof of concept, we've been able to bring up a 12 node shared root cluster. Already, performance on the file system looks promising. A parallel kernel build (make -j4 bzImage on each node within their own trees) compares well with ext3:

        Ext3           OCFS2
node1   13m4s          13m16s
node2   10m27s         10m38s

In case there is any concern about code size, a quick comparison shows the file system and cluster stacks combined size to be not significantly larger than reiserfs and 1/3 the size of xfs.

For people using git, these changes can be pulled from:
http://oss.oracle.com/git/ocfs2.git/

A full patch can be downloaded from:
http://oss.oracle.com/projects/ocfs2/dist/files/patches/2.6.12-rc4/complete/ocfs2-configfs-all.patch

Broken out versions of the patches can be found at:
http://oss.oracle.com/projects/ocfs2/dist/files/patches/2.6.12-rc4/broken-out/

A short description of each patch follows. The same descriptions can be found at the top of each patch file.

http://oss.oracle.com/projects/ocfs2/dist/files/patches/2.6.12-rc4/broken-out/01_configfs.patch
Configfs, a file system for userspace-driven kernel object configuration. The OCFS2 stack makes extensive use of this for propagation of cluster configuration information into kernel.

http://oss.oracle.com/projects/ocfs2/dist/files/patches/2.6.12-rc4/broken-out/02_mlog.patch
Very simple printk wrapper which adds the ability to enable various sets of debug messages at run-time.

http://oss.oracle.com/projects/ocfs2/dist/files/patches/2.6.12-rc4/broken-out/03_nm.patch
A simple node information service, filled and updated from userspace. The rest of the stack queries this service for simple node information.

http://oss.oracle.com/projects/ocfs2/dist/files/patches/2.6.12-rc4/broken-out/04_heartbeat.patch
Disk based heartbeat. Configured and started from userspace, the kernel component handles I/O submission and event generation via callback mechanism.

http://oss.oracle.com/projects/ocfs2/dist/files/patches/2.6.12-rc4/broken-out/05_messaging.patch
Node messaging via tcp. Used by the dlm and the file system for point to point communication between nodes.

http://oss.oracle.com/projects/ocfs2/dist/files/patches/2.6.12-rc4/broken-out/06_dlm.patch
A distributed lock manager built with the cluster file system use case in mind. The OCFS2 dlm exposes a VMS style API, though things have been simplified internally. The only lock levels implemented currently are NLMODE, PRMODE and EXMODE.

http://oss.oracle.com/projects/ocfs2/dist/files/patches/2.6.12-rc4/broken-out/07_dlmfs.patch
dlmfs: A minimal dlm userspace interface implemented via a virtual file system.
Most of the OCFS2 tools make use of this to take cluster locks when doing operations on the file system.

http://oss.oracle.com/projects/ocfs2/dist/files/patches/2.6.12-rc4/broken-out/08_ocfs2.patch
The OCFS2 file system module.

http://oss.oracle.com/projects/ocfs2/dist/files/patches/2.6.12-rc4/broken-out/09_build.patch
Link the code into the kernel build system. OCFS2 is marked as experimental and we only enable builds on x86, x86-64 and ia64 - endianness consistency is a work in progress.

When built, the stack will result in 5 modules: configfs.ko, ocfs2_nodemanager.ko, ocfs2_dlm.ko, ocfs2.ko and ocfs2_dlmfs.ko

--
Mark Fasheh
Senior Software Developer, Oracle
[email protected]

Some comments on this patch:
- there's no reason to make JBD user-visible
- is there any reason why CONFIGFS_FS is user-visible?
- some global code might become static:
run "make namespacecheck" after compiling the kernel and check the configfs and ocfs2 parts of the output

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Sure, the only reason I made it visible was because of the comment in there:

# CONFIG_JBD could be its own option (even modular), but until there are

# other users than ext3, we will simply make it be the same as CONFIG_EXT3_FS

I don't really have a preference either way.

It's a generic mechanism for userspace driven configuration of kernel functionality. There's nothing specific to OCFS2 about it. Other kernel subsystems/projects could use it too, for their own configuration mechanisms. More details are in configfs.txt, which is included in the above patch. Note the example used in the documentation text is an NBD driver.

Yeah, there's some stuff that that scripts catches. Thanks.

-Manish

I'd say the comment is wrong.

If other subsystems use it, they should select it.

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

While attempting to unload the kernel module which has created a kernel thread whoch runs perfectly i get this oops:

Unable to handle kernel paging request at virtual address d08f364c
 printing eip:
c011d6ff
*pde = 01af0067
*pte = 00000000
Oops: 0002
parport_pc lp parport autofs pcnet_cs 8390 crc32 ds yenta_socket pcmcia_core
floppy microcode ext3 jbd
CPU:    0
EIP:    0060:[<c011d6ff>]    Not tainted
EFLAGS: 00010092

EIP is at interruptible_sleep_on_timeout [kernel] 0x4f (2.4.21-4.EL)
eax: d08f3648 ebx: 00000286 ecx: 00000286 edx: d08f3648
esi: 000001c3 edi: 00000000 ebp: cc445fdc esp: cc445fb4
ds: 0068 es: 0068 ss: 0068
Process enigma (pid: 1622, stackpage=cc445000)

Stack: 00000000 cc444000 d08f3648 d08f3648 00000000 00000003 00000656
00000656
       d08f3060 00000000 cc445fec d08f310d 00000004 cc444000 00000000
c010945d
       d08f363c 00000000 00000000
Call Trace:   [<c010945d>] kernel_thread_helper [kernel] 0x5 (0xcc445ff0)

Code: 89 50 04 89 02 c7 45 e4 00 00 00 00 c7 45 e0 00 00 00 00 53

Kernel panic: Fatal exception

The module is being unloaded with the following procedure:
lock_kernel();
flag=1; /* This flag does a break in the for(;;) loop of the thread after the interruptible_sleep_on_timeout , after the break current=NULL;*/ mb();
kill_proc(pid_thread,SIGKILL,1);
unlock_kernel();
kill_proc(2,SIGCHLD,1); /* assuming that keventd is running with PID 2 */

you failed to put in an URL to the sourcecode of your module.

Your messages keep showing up as replies to other threads. It looks like you're just hitting 'reply' to the latest message in your lkml mailbox in order to post a new message (at least the headers of your mail contain references to unrelated messages). Please don't do this. Just create a new message with '[email protected]' in the To field. People will be more likely to see your message this way since it won't be buried in a thread they may or may not be interested in.

Thanks,
Jesse

I too am quick to cite XFS when anyone complains about the code size of anything we write.;-)

Congrats on getting a filesystem to start to work, I know how long that road is, and best of luck to you all,

Hans

[...snip...]

Does this include support for UNIX sockets and named pipes? One of the issues I have with filesystems like AFS and NFS is that they should theoretically make such things possible, but the code is not implemented yet and does not appear it ever will be. This is very useful for home directories when using programs like "links" that rely on being able to create UNIX sockets in .links/socket or similar.

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r !y?(-)
------END GEEK CODE BLOCK------

Unfortunately no, at least not on a clustered level. Local node, those should work as normal. If you absolutely need this functionality, I believe OpenSSI currently supports it.

If you're looking for an easy way to share small amounts data between applications in an OCFS2 cluster, you might be interested in looking at dlmfs. Once mounted, it's quite trivial to create dlm domains and lock resources between nodes via regular file system operations (open, close, etc). File read and write will return and set the resource LVB for you, currently 64 bytes. Though this can all be controlled from shell, a small library (libo2cb) to take advantage of this is provided in the current toolchain distribution.

	--Mark

--
Mark Fasheh
Senior Software Developer, Oracle
[email protected]

+ if (recovery &&
+ (!dlm_is_recovery_lock(name, strlen(name)) || convert) ) {
+ goto error;
+ }

Zero terminated strings for lock names is bad taste. It generates a bunch of useless strlen executions and you force an ascii namespace for no apparent reason. Add a 9th parameter, namelen, to the lock call maybe?

Regards,

Daniel

Or perhaps pass in a qstr? Anyway I have to agree. That shouldn't be difficult to fix up.

	--Mark

--
Mark Fasheh
Senior Software Developer, Oracle
[email protected]

Hi Mark,

Qstr would be nice, either that or an explicit string length. Either way, the compiler will catch any missed fixups.

Regards,

Daniel

What's wrong with ascii strings?

We call those 'UTF8' these days.

Jeff

I think you just answered your own question.

Lee

Incorrect. Read the quoted context.

Jeff

The masklog code looks potentially useful outside of ocfs2, as a general kernel facility. Any chance of splitting it out?

Quibbles:

- A lot of the macros should probably be replaced with static inlines, like OCFS2_IS_VALID_DINODE.

- Approx. 80 typedefs. ouch.

James Morris
<[email protected]>

Hi James,

Thanks for looking over this.

Absolutely. We've found that stuff invaluable in debugging issues for a while now, and as far as I recall, Zach wrote it with the idea that it could be useful to other folks too. I think the only issue to resolve there would be how other subsystems reserve a debugging bit...

Heh, we're slowly removing those :)

	--Mark

--
Mark Fasheh
Senior Software Developer, Oracle
[email protected]

Responding to your post, but I guess it's really Oracle I'm asking:

On my first glance just now at the git-ocfs patch, I was surprised that what seemed to be separable facilities were combined into one patch. I see a file system, a lock manager (is that what DLM stands for - please spell out acronyms) and a configuration file system.

+configfs/
+ - directory containing configfs documentation and example code.
...
+dlmfs.txt
+ - info on the userspace interface to the OCFS2 DLM.
...
+ocfs2.txt
+ - info and mount options for the OCFS2 clustered filesystem.

These combine to make a 45 thousand line patch. That's a big patch. Only the netdev and reiser4 patches (and the combined linus.patch) are bigger.

Shouldn't these be 3 patches, or more?

                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401

They are. I believe you're looking at the combined patch. If you want them broken out, we keep those at
http://oss.oracle.com/projects/ocfs2/files/patches/

My original patch mail also had links to the broken out ones...

	--Mark

--
Mark Fasheh
Senior Software Developer, Oracle
[email protected]

Ok - good.

                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401

A couple more random questions:

1) How does this lock manager relate to the Red Hat lock manager, the dlm-* patches?

Note as an aside that dlm-* has 23 patches for a total of 14 thousand lines, or about 600 lines per patch. This is rather different than burying a lock manager into a 45 thousand line new file system patch ;)

2) Would a diffstat be a useful addition to this patch?

                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401