HSM, DMAPI, and external media filesystems

I would like to discuss how to get DMAPI accepted into the mainline kernel. DMAPI is or was supported by a number of filesystems which are included in linux (including XFS) and also is supported by a number of proprietary filesystems. However, the DMAPI implementation has not yet been accepted into the main line.

- What are the barriers to inclusion and how can they be resolved?

- What FOSS HSM projects are available to use the interface?
* xfstests/dmapi/sample_hsm
* openhsm (opensms)
* tinyhsm (unfinished, unreleased)

- Who are the interested parties (if any), and how could we split up the work?

In the end I'd like to come up with a plan that seems reasonable toward getting DMAPI included mainline. I know that Alex Elder and others have done work in this area and I would like to see it bear fruit.

---

Secondly, I'd enjoy a discussion of an old idea from back in the day when I worked as a sysadmin and became a storage software hobbyist. Having recently lost a great deal of data due to disk failure and failed backups, I struck upon the idea of turning the backup paradigm on it's head:

I'd rather have the authoritative copy of my data on an external medium and only cache it on spinning rust. The filesystem would log every transaction (WORM style) to removable media and the set of discs or tapes would be replayable onto new hard disks when I had a failure. Checkpoints would ensure that the number of discs/tapes necessary to replay the filesystem remained manageable, and the write performance problem would be mitigated by maintaining a temporary log on spinning disk to spool up enough data to keep the tape from having to rewind. One could choose to recycle media from before a checkpoint, or save it off for posterity.

My choice of medium at the time was cd-rom since I couldn't afford a tape drive. I went about trying to implement such a thing using DMAPI, which isn't suitable (you can't set an event mask on a file quickly enough to log the first writes). Then I tried a stacking filesystem, fuse, got a job, and lost interest.

This is almost certainly not a new idea. If there is already something like this implemented, I'd like to hear about it. If there isn't, maybe we could come up with a rough design on a cocktail napkin and pique enough interest to make a start.

Regards,
Ben

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

* ohsm (unfinished, unreleased) (I think)
http://code.google.com/p/fscops/
Presented at OLS 2009

Greg
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Latest one was at http://sourceforge.net/projects/ohsm/

Thanks -
Manish

The grotty DMAPI interface. That includes both the high-level ioctl implementations as well as lower level details.

In the end most of it boils down to three things:

- a notification API telling the daemon about changes, and allowing it to block them. That's in some ways fairly similar to fanotify, which should be reused. Preferably including the interface, but at the very least the underlying implementation.
- a handle based I/O method. We now have open by handle at the VFS, so it might just need a few additions like the invisible I/O flag.
- ways to make extents offline in the fs, and set various HSM specific flags. This is the actual filesystem-specific piece of code, but we'll need to think about an interface to query and set these bits. filesystem-specific piece of code.

That actually seems to be the only one.

While that one has HSM in the name it actually seems to work quite differently from the traditional partially offline HSM model, nevermind that it seems abandoned.

pointers?

I'm not really sure discussing this grand highlevel design without backing code really fits for LSF. Might be a fine topic for a BOF if enough interested people are around.