xfs: minimize DMAPI footprint
cover xfs: minimize DMAPI footprint
Back in the early part of May, Christoph Hellwig sent a patch to the list proposing to drop the remaining DMAPI support code from XFS. He said:
Dmapi support was never merged upstream, but we still have a lot of hooks bloating XFS for it, all over the fast pathes of the filesystem.
This patch drops over 700 lines of dmapi overhead. If we'll ever get HSM support in mainline at least the namespace events can be done much saner in the VFS instead of the individual filesystem, so it's not like this is much help for future work.
SGI has a product that uses the DMAPI support code that's included in mainline XFS, along with some additional code (the "never merged" stuff Christoph refers to) that we maintain separately. To our customers that need it, this is an extremely important feature.
I was concerned that the outright removal of the code would make it very difficult for us to continue to offer our DMAPI support, especially given the rate at which code has been getting refactored in XFS lately. Specifically, DMAPI relies on the file system to send it messages when certain events occur, and I don't want to lose track of the specific spots in the XFS code where this needs to occur.
I agree with Christoph that the DMAPI-specific definitions, etc., have no place in the top-of-tree code right now. However I wanted to provide an alternative to Christoph's patch, one which preserves some very specific hooks that allow DMAPI support to remain, but which otherwise minimizes the footprint and visibility of DMAPI definitions in environments where it is not used (and which furthermore incurs little or no run-time overhead in non-DMAPI builds).
What follows is a set of patches that I think accomplishes these goals. The net result of these changes is:
- DMAPI calls, along with almost all supporting logic, have been replaced with a set of 21 function calls that are used to signal events. The information supplied to these functions is exactly what is required--no more--and their syntactic interfaces do not rely on any internal DMAPI definitions.
- Two files remain in top-of-tree code to support DMAPI:
"xfs_dmops.c" and "xfs_dmapi.h". The former still defines the two functions called at mount and unmount time to attach/detach a vector of DMAPI operations to the mount point. I've simplified this though, so a null pointer is used if DMAPI is not present. The header file now only declares the two functions, and defines a set of no-op macros to be used for DMAPI event generation when DMAPI is not configured in the build. Everything else will now be defined elsewhere.
The "dmapi" mount option remains, but the rest of the details have been moved into separate patches that we can continue to maintain out-of-tree, and I've been able to verify that with those patches applied, the result still implements the DMAPI support we need.
I would like to get some feedback on these changes, and hopefully get agreement that the result is an acceptable compromise that *nearly* eliminates the DMAPI code from top-of-tree XFS. I think there are a few more small things that are DMAPI-specific that may need to be pulled out, but what I have here gives a pretty good idea of what I've set out to do.
-Alex
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 5 lines
SGI has a product that uses the DMAPI support code that's included in mainline XFS, along with some additional code (the "never merged" stuff Christoph refers to) that we maintain separately. To our customers that need it, this is an extremely important feature.
So why don't you bother to get HSM support upstream properly, or at least maintain it somewhere where you can get at it? What sourcxe tree do those important customers use it?
Quoted 2 lines
What follows is a set of patches that I think accomplishes these goals. The net result of these changes is:
While this is a lot better than the old DMAPI supoort, it's still lots of dead code in the mainline tree, that won't ever be used there, as proper HSM suport if it ever was merged would sit at the VFS layer.
In addition to that the people who effectively maintain XFS for both the community and lots of paying customers have done a large amount of work ontop of the DMAPI removal of the last 1 1/2 month. So I'd say rebase your changes over
http://git.kernel.org/?p=linux/kernel/git/dgc/xfsdev.git;a=shortlog;h=refs/heads/for-2.6.36
and keep them in a separate branch dmapi-dev branch where SGI can pull the code for it's customers from. This branch could also include the actual dmapi code and core kernel modifications, so that people that want dmapi support actually have chance to find a complete kernel tree for it. I'd also hope you have fixed grave bugs like the oops on unmount after multiple mount one I pointed out to SGI about two years ago, but which still wasn't fixed inthe last dmapi enabled tree.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 10 lines · Christoph Hellwig
Quoted 5 lines
SGI has a product that uses the DMAPI support code that's included in mainline XFS, along with some additional code (the "never merged" stuff Christoph refers to) that we maintain separately. To our customers that need it, this is an extremely important feature.
So why don't you bother to get HSM support upstream properly, or at least maintain it somewhere where you can get at it? What sourcxe tree do those important customers use it?
Quoted 7 lines
Quoted 2 lines
What follows is a set of patches that I think accomplishes these goals. The net result of these changes is:
While this is a lot better than the old DMAPI supoort, it's still lots of dead code in the mainline tree, that won't ever be used there, as proper HSM suport if it ever was merged would sit at the VFS layer.
My question about the DMAPI hooks also still stands - if we leave the hooks in mainline, how are we supposed to test that they are still placed correctly for the out-of-tree patches to function correctly? I can't see that we can actually do this, so I question the value of even leaving minimal hooks in place....
Quoted 5 lines
In addition to that the people who effectively maintain XFS for both the community and lots of paying customers have done a large amount of work ontop of the DMAPI removal of the last 1 1/2 month. So I'd say rebase your changes over
Quoted 7 lines
http://git.kernel.org/?p=linux/kernel/git/dgc/xfsdev.git;a=shortlog;h=refs/heads/for-2.6.36and keep them in a separate branch dmapi-dev branch where SGI can pull the code for it's customers from. This branch could also include the actual dmapi code and core kernel modifications, so that people that want dmapi support actually have chance to find a complete kernel tree for it.
This makes a lot of sense to me. I'd prefer an all-or-nothing approach to supporting DMAPI (and any other out-of-tree enabling functionality for that matter) and putting it all in separate branch would give us both all and nothing. ;)
It would also help us test the DMAPI infrastructure without needing a HSM as the xfsqa test suite does a pretty good job of testing it. And, of course, we could also help clean it up if it is testable. As such, I'd be quite happy to maintain a dmapi-dev branch in the above repo if the eventual goal is to try to move the code towards being more acceptible for mainline inclusion....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
(Responding to Dave, since it basically included all of Christoph's response as well.)
Quick summary: we'll go ahead with the suggestion to maintain the DMAPI code (all of it) in a separate branch.
Quoted 11 lines · Dave Chinner
Quoted 10 lines · Christoph Hellwig
Quoted 5 lines
SGI has a product that uses the DMAPI support code that's included in mainline XFS, along with some additional code (the "never merged" stuff Christoph refers to) that we maintain separately. To our customers that need it, this is an extremely important feature.
So why don't you bother to get HSM support upstream properly, or at least maintain it somewhere where you can get at it? What sourcxe tree do those important customers use it?
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
What follows is a set of patches that I think accomplishes these goals. The net result of these changes is:
While this is a lot better than the old DMAPI supoort, it's still lots of dead code in the mainline tree, that won't ever be used there, as proper HSM suport if it ever was merged would sit at the VFS layer.
I agree that HSM support would be better implemented at the VFS layer, allowing other file systems to benefit. I do not agree that after my patch series is applied it's "lots" of dead code in the mainline tree. I also think this is a case where it's better to preserve something that works and evolve it into something better than to throw it all away and start over.
That being said, until now I personally have spent almost no time on the DMAPI code itself, and I don't claim any special knowledge about the best way to provide kernel support for HSM's. I also acknowledge that there's a bunch of other required DMAPI code that's not in the mainline tree (but there was no intent to conceal it).
Quoted 5 lines
My question about the DMAPI hooks also still stands - if we leave the hooks in mainline, how are we supposed to test that they are still placed correctly for the out-of-tree patches to function correctly? I can't see that we can actually do this, so I question the value of even leaving minimal hooks in place....
Point taken. I (or SGI I guess) would much rather have all of the code in the mainline tree but it's not. (I don't know any details on the history of that, by the way.) Maintaining the other parts separately was done of necessity.
Quoted 13 lines
Quoted 5 lines
In addition to that the people who effectively maintain XFS for both the community and lots of paying customers have done a large amount of work ontop of the DMAPI removal of the last 1 1/2 month. So I'd say rebase your changes over
Quoted 7 lines
http://git.kernel.org/?p=linux/kernel/git/dgc/xfsdev.git;a=shortlog;h=refs/heads/for-2.6.36and keep them in a separate branch dmapi-dev branch where SGI can pull the code for it's customers from. This branch could also include the actual dmapi code and core kernel modifications, so that people that want dmapi support actually have chance to find a complete kernel tree for it.
Quoted 4 lines
This makes a lot of sense to me. I'd prefer an all-or-nothing approach to supporting DMAPI (and any other out-of-tree enabling functionality for that matter) and putting it all in separate branch would give us both all and nothing. ;)
I think this is a good solution, allowing all of the code to be published and available--and complete--while not forcing the mainline code to be polluted by the clutter of non-operative code.
Quoted 6 lines
It would also help us test the DMAPI infrastructure without needing a HSM as the xfsqa test suite does a pretty good job of testing it. And, of course, we could also help clean it up if it is testable. As such, I'd be quite happy to maintain a dmapi-dev branch in the above repo if the eventual goal is to try to move the code towards being more acceptible for mainline inclusion....
Here is what I plan to do:
- Accept Christoph's "drop dmapi hooks" patch into the XFS master branch. Once that's in, there's a huge cascade of subsequent work that need to be pulled in and I'll get those out there also.
- Work on creating a complete and working dmapi branch. Once we've got a fairly well-integrated whole, I will publish that branch on the XFS git tree. We will plan to maintain that in parallel with the XFS master branch, re-basing it on the master branch periodically.
- I'll make an announcement on the list about the availability of the new branch.
-Alex
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 4 lines · Alex Elder
Here is what I plan to do:
- Accept Christoph's "drop dmapi hooks" patch into the XFS master branch. Once that's in, there's a huge cascade of subsequent work that need to be pulled in
Quoted 8 lines
and I'll get those out there also.
- Work on creating a complete and working dmapi branch. Once we've got a fairly well-integrated whole, I will publish that branch on the XFS git tree. We will plan to maintain that in parallel with the XFS master branch, re-basing it on the master branch periodically.
- I'll make an announcement on the list about the availability of the new branch.
Thanks. I think I still have a few old patches for dmapi code from a few years ago that I can repost to simplify that code.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs