A new NFSv4 server...
Subsystems (2)
In case some developers are interested... I'm poking at a from-scratch userland NFSv4 server, as a side project.
In my personal opinion, version 4 of NFS is a quantum-leap improvement over previous versions. While I used NFS v3 extensively, I always felt it was a crappy protocol, and unworthy of serious development effort. That changed with v4.
I chose to use NFSv4 as the basis for experiments (hopefully yielding production software) that I've long wanted to do in reliable filesystems, distributed filesystems, and other fun areas.
In the first step down this long path, I've created an NFSv4 userland server from scratch. Currently it merely serves data straight from RAM, but the long term goal is to permit modular storage backends. Thus you could implement a simple RAM backend, an sqlite-based backend or a complex distributed storage backend.
As this is a first-mention developer-only announcement, I didn't bother to create source tarballs. Here is the git repo:
git://git.kernel.org/pub/scm/daemon/nfs/nfs4-ram.git
This is the home page, but it's mainly a stub pointing to the git repo:
http://linux.yyz.us/projects/nfsv4.html
The server will
* serve data from RAM, with NFSv4 persistent filehandles and FILE_SYNC4
* destroy all data, when the process exits
* pass 97% of the useful pynfs tests (cvs latest)
* pass fsx-linux stress testing, with Linux NFSv4 client (2.6.recent)
* pass kernel build stress testing, with Linux NFSv4 client (2.6.recent)
It will not, at the present time,
* store any data or metadata in stable storage
* do RPCSEC_GSS (thus, not yet RFC-compliant)
* do delegations (thus, with reduced caching and increased revalidations, can be slower than disk-based storage)
At this point, I'm quite interested to hear feedback on how the server works with other NFSv4 clients. I'm interested in making sure the server is portable to FreeBSD and other OS's, even though it was developed and tested only on Linux. I also intend to use some Linux-specific syscalls, most notably sync_file_range(2) and sendfile/tee/splice/vmsplice, so that will have to be glossed over by portability code.
Finally, this is a spare time project, something I've mostly been poking at while having idle time on a not-Internet-connected laptop. Technically its sponsored by Red Hat, since RH pays my salary for all my open source work, but this is largely a personal project done for personal reasons. I just hope others find it interesting or useful, as it progresses.
Jeff
Quoted 2 lines · Jeff Garzik
At this point, I'm quite interested to hear feedback on how the server works with other NFSv4 clients.
Any possibility of making Connectathon in May?:
The major NFSv4 implementors normally have their clients there, so it's usually the quickest way to find and solve any interoperability problems. Almost all the work will probably be on sessions and pNFS, but people should be willing to do basic 4.0 testing too.
Glad to hear you're still working on this--it sounds interesting.
--b.
Quoted 11 lines · J. Bruce Fields
Quoted 1 line · Jeff Garzik
At this point, I'm quite interested to hear feedback on how the server works with other NFSv4 clients.
Any possibility of making Connectathon in May?:
The major NFSv4 implementors normally have their clients there, so it's usually the quickest way to find and solve any interoperability problems. Almost all the work will probably be on sessions and pNFS, but people should be willing to do basic 4.0 testing too.
As this isn't an official RH project, I would probably have to pay my own way, which makes it doubtful :)
Plus, surely in this day and age, we can figure out something better than waiting for face-to-face events to test something. Maybe somebody could arrange a donation of some slice of a grid (Amazon EC2?), make various OS images available, and give engineers some way to request a selection of tests, with a selection of OS images?
Quoted 1 line
Glad to hear you're still working on this--it sounds interesting.
Certainly pNFS parallels some of the work I want to do... NFSv4.1 is so darned complex though. I am torn as to whether or not I want to take my server down that path.
I really wish the entire wire protocol were scrapped and replaced with something more sane, and easier to parse. The variable-length structures passed to PCI hardware these days [as seen in the kernel drivers I hack on, IOW] are just as compact, if not more so, but are designed to be parsed quickly in large chunks, rather than the "next XDR may be your last!" approach :)
Sessions are IMO a tad overdone, too... largely due to necessities forced upon NFSv4.1 by the legacy RPC protocol assumptions. If you simply /assume/ basic properties of TCP or SCTP, it's a lot easier to do multi-channel or multi-homed messaging. Multi-channel _isn't_ really that hard, and we've been doing it since the earliest days of NNTP and the Usenet Top 1000 pissing contest, if not longer.
It's tempting to see what would arise from a clean-slate wire protocol effort, something that is otherwise compatible with NFS 4.x operations, objects, and data model.
Jeff
Quoted 8 lines · Jeff Garzik
Quoted 4 lines · J. Bruce Fields
As this isn't an official RH project, I would probably have to pay my own way, which makes it doubtful :)
It's entirely possible someone might run your server code on a spare machine,
given functioning install packages and easy instructions.
Quoted 6 lines
Plus, surely in this day and age, we can figure out something better than waiting for face-to-face events to test something. Maybe somebody could arrange a donation of some slice of a grid (Amazon EC2?), make various OS images available, and give engineers some way to request a selection of tests, with a selection of OS images?
Vendors turn up to cthon with proprietary and unreleased software and hardware
which they most certainly are not going to let anyone else run for them. Also,
being in the same hall with all those vendors' technical folks tends to make bugs
shallow. It's a very valuable exercise for any organisation making a living from
NFS.
Quoted 2 lines
I really wish the entire wire protocol were scrapped and replaced with something more sane, and easier to parse.
You had me worried there for a moment, I thought you might be the first person to admit to liking the NFS4 protocol design.
Quoted 4 lines
It's tempting to see what would arise from a clean-slate wire protocol effort, something that is otherwise compatible with NFS 4.x operations, objects, and data model.
Much like the old phone system, the primary value of protocols like NFS is the
widespread presence of reliable conformant implementations. Most of the rest of
the NFS is problematic. I would argue that some aspects of the NFS operations,
objects, and data model is rather more busted than the XDR encoding.
The classic persistent file handles, for example, could be considered a major
design flaw. Firstly it makes the inode# -> dentry lookup a performance path
for the underlying filesystem, which it isn't in any local load. Secondly, it's
inherently insecure if you export anything less than an entire filesystem, unless
you use a slow, buggy, and non-conformant hack like subtree_check.
Another major flaw is putting the client in control of when unstable data is written to disk, but not providing any way for the client to find out how to do that optimally.
Then there's the NFS4 approach to extended attributes.
Signature · 3 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group. The cake is *not* a lie. I don't speak for SGI.
A message that is not in the archive (sent off-list or not yet received): 477DD11B.40909-cP1dWloDopni96+mSzHFpQC/[email protected]
Quoted 3 lines · Greg Banks
It's entirely possible someone might run your server code on a spare machine,
given functioning install packages and easy instructions.
Easy enough to do...
Quoted 11 lines
Quoted 2 lines
Plus, surely in this day and age, we can figure out something better than waiting for face-to-face events to test something. Maybe somebody could arrange a donation of some slice of a grid (Amazon EC2?), make various OS images available, and give engineers some way to request a selection of tests, with a selection of OS images?
Vendors turn up to cthon with proprietary and unreleased software and hardware
which they most certainly are not going to let anyone else run for them. Also,
being in the same hall with all those vendors' technical folks tends to make bugs
shallow. It's a very valuable exercise for any organisation making a living from
NFS.
Certainly, but I could see a grid of released, non-proprietary software as quite a valuable resource in addition to f2f events. Quality can only increase, if the [Linux | *BSD | OpenSolaris |...] NFS clients could run regression tests against several different NFS servers, each time an NFS client receives a set of changes.
Even if it's only the open source operating systems that wish to participate, having a mix of OS's and platforms would be useful. A permanent, virtual cthon.
Quoted 5 lines
Quoted 1 line
I really wish the entire wire protocol were scrapped and replaced with something more sane, and easier to parse.
You had me worried there for a moment, I thought you might be the first person to admit to liking the NFS4 protocol design.
Quoted 1 line
It's tempting to see what would arise from a clean-slate wire protocol effort, something that is otherwise compatible with NFS 4.x operations, objects, and data model.
It's more like v4 is a vast relative improvement over prior NFS. Given the huge number of NFS users and sites, IMO v4 is a huge improvement for Unix file sharing overall.
But if you are dreaming of a truly clean slate protocol... I've got a long wish list too :)
Quoted 8 lines
Much like the old phone system, the primary value of protocols like NFS is the
widespread presence of reliable conformant implementations. Most of the rest of
the NFS is problematic. I would argue that some aspects of the NFS operations,
objects, and data model is rather more busted than the XDR encoding.
Quoted 5 lines
The classic persistent file handles, for example, could be considered a major
design flaw. Firstly it makes the inode# -> dentry lookup a performance path
for the underlying filesystem, which it isn't in any local load.
Oh, certainly. I was mainly thinking a replacement of the wire protocol would be an easier step for people to swallow than a new protocol.
But if you are implying there is enough momentum to simply rewrite NFS from scratch, I'll cheer and help out with coding :) Or maybe Zach Brown will do it for us with CRFS: http://linux.conf.au/programme/detail?TalkID=247
A big feature of NFS today is its high Just Works(tm) value (ease of configuration and some minimum level of fault tolerance), so any replacement would need to have similar attributes.
Quoted 5 lines
Another major flaw is putting the client in control of when unstable data is written to disk, but not providing any way for the client to find out how to do that optimally.
Then there's the NFS4 approach to extended attributes.
Ugh. Don't get me started. That's not in my server yet, but I can already see the mess ahead.
Jeff
Quoted 48 lines · Jeff Garzik
Quoted 8 lines
Quoted 2 lines
I really wish the entire wire protocol were scrapped and replaced with something more sane, and easier to parse.
You had me worried there for a moment, I thought you might be the first person to admit to liking the NFS4 protocol design.
Quoted 3 lines
It's tempting to see what would arise from a clean-slate wire protocol effort, something that is otherwise compatible with NFS 4.x operations, objects, and data model.
It's more like v4 is a vast relative improvement over prior NFS. Given the huge number of NFS users and sites, IMO v4 is a huge improvement for Unix file sharing overall.
But if you are dreaming of a truly clean slate protocol... I've got a long wish list too :)
Quoted 8 lines
Much like the old phone system, the primary value of protocols like NFS is the
widespread presence of reliable conformant implementations. Most of the rest of
the NFS is problematic. I would argue that some aspects of the NFS operations,
objects, and data model is rather more busted than the XDR encoding.Quoted 5 lines
The classic persistent file handles, for example, could be considered a major
design flaw. Firstly it makes the inode# -> dentry lookup a performance path
for the underlying filesystem, which it isn't in any local load.Oh, certainly. I was mainly thinking a replacement of the wire protocol would be an easier step for people to swallow than a new protocol.
But if you are implying there is enough momentum to simply rewrite NFS from scratch, I'll cheer and help out with coding :) Or maybe Zach Brown will do it for us with CRFS:
http://linux.conf.au/programme/detail?TalkID=247A big feature of NFS today is its high Just Works(tm) value (ease of configuration and some minimum level of fault tolerance), so any replacement would need to have similar attributes.
Quoted 4 lines
Another major flaw is putting the client in control of when unstable data is written to disk, but not providing any way for the client to find out how to do that optimally.
Quoted 7 lines
Quoted 1 line
Then there's the NFS4 approach to extended attributes.
Ugh. Don't get me started. That's not in my server yet, but I can already see the mess ahead.
Jeff
Jeff, taking into account the amount of effort people and different organizations have already put into NFSv4 and NFSv4.1 I wish you could tunnel your inventive energy into making NFSv4.1 better rather than trying to reinvent NFS/RPC/XDR.
Although It's rather late in the process since the NFSv4 working group is close to putting the NFSv4.1 Internet-draft up for last call, we would certainly appreciate more implementation feedback.
Benny
Quoted 8 lines · Benny Halevy
Jeff, taking into account the amount of effort people and different organizations have already put into NFSv4 and NFSv4.1 I wish you could tunnel your inventive energy into making NFSv4.1 better rather than trying to reinvent NFS/RPC/XDR.
Although It's rather late in the process since the NFSv4 working group is close to putting the NFSv4.1 Internet-draft up for last call, we would certainly appreciate more implementation feedback.
I am more than happy to give feedback, though (as you say) it is probably too late for substantial feedback to have any large effect.
My general engineering opinions of pNFS:
* Fills an obvious need: eliminating the need to copy data through the metadata interface to backend storage. Many clear, tangible benefits here.
* pNFS major issue #1: client storage protocol
Storing and retrieving blobs over the network, with strong authentication/integrity/security, is a solved problem.
Pick ONE client storage protocol (HTTP? iSCSI OSD2?), and stick to it. Or maybe HTTP|SCSI but nothing more. Heck, even BitTorrent w/ auth extensions would be better than yet another protocol for similar purposes (not that I'm advocating BT, just saying...).
Maximize reuse of existing software and mindshare.
* pNFS major issue #2: abandons NFS's "one true generic" path
I believe pNFS violates the "spirit of NFS" by deviating from a defacto assumption found in earlier versions: data transfer is simple, arbitrary blobs, addressed in the same manner, and sent via the same protocol.
Pick ONE layout type, and stick to it. Banish all other layout types to other software layers.
Protocol conversion servers, firmware, and other softwares can easily convert from a generic layout to something more exotic like OSD or [insert site specific protocol here].
NFS itself should not be delving into low-level storage details like this. Clients should not need to know low-level details (like stripe sizes). In Linux, we call this a layering violation.
Working on kernel storage drivers as I do, I can see the attraction of wanting to do things this way... but we invented layering and abstraction in computer science for good reasons :)
* pNFS major issue #3: no longer a "closed loop" protocol
By permitting multiple layout types, and in particular undefined (site-specific) layout types, it is by definition _impossible_ for anyone to claim full protocol interoperability with other implementations.
The number of possible combinations approaches infinity, with obvious consequences on testing, and production software quality.
And when a marketing department advertises "fully NFSv4.1 compliant!" on ther company's appliance, it is trivial for any engineer to construct another "fully NFSv4.1 compliant" setup -- with equivalent authentication, metadata and data sets -- that is not interoperable except via the fallback case (copy through the metadata server).
Such interoperability breakdowns are IMO not in the spirit of NFS.
Jeff
Quoted 15 lines · Jeff Garzik
Quoted 8 lines · Benny Halevy
Jeff, taking into account the amount of effort people and different organizations have already put into NFSv4 and NFSv4.1 I wish you could tunnel your inventive energy into making NFSv4.1 better rather than trying to reinvent NFS/RPC/XDR.
Although It's rather late in the process since the NFSv4 working group is close to putting the NFSv4.1 Internet-draft up for last call, we would certainly appreciate more implementation feedback.
I am more than happy to give feedback, though (as you say) it is probably too late for substantial feedback to have any large effect.
My general engineering opinions of pNFS:
Quoted 8 lines
* Fills an obvious need: eliminating the need to copy data through the metadata interface to backend storage. Many clear, tangible benefits here.
* pNFS major issue #1: client storage protocol
Storing and retrieving blobs over the network, with strong authentication/integrity/security, is a solved problem.
Pick ONE client storage protocol (HTTP? iSCSI OSD2?), and stick to it. Or maybe HTTP|SCSI but nothing more. Heck, even BitTorrent w/ auth extensions would be better than yet another protocol for similar purposes (not that I'm advocating BT, just saying...).
Well, two things about that: maybe if pnfs started from scratch this was the approach that could have been taken but one of the motivating factors for pNFS (and actually one that I believe will help make it successful) was the desire to replace existing proprietary file system protocols from several vendors such as EMC, IBM, or Panasas that used different storage protocols. Second, providing support for several kinds of storage is better for customers having existing storage they want to harness together with pNFS.
Quoted 2 lines
Maximize reuse of existing software and mindshare.
That's always good.
Quoted 7 lines
* pNFS major issue #2: abandons NFS's "one true generic" path
I believe pNFS violates the "spirit of NFS" by deviating from a defacto assumption found in earlier versions: data transfer is simple, arbitrary blobs, addressed in the same manner, and sent via the same protocol.
Pick ONE layout type, and stick to it. Banish all other layout types to other software layers.
I agree wholeheartedly that we should have had one layout data structure ("layout type" is a loaded term in the spec...). This was my position from day one (and even before) but unfortunately it wasn't accepted and each "layout type" got to define it's own layout data structure while we could have defined one generic data structure for mapping files onto all different kinds of storage devices while keeping only the device addressing information private to the "layout type" (== storage protocol class), plus some other data that's internal to the layout type, e.g. OSD capabilities.
Quoted 4 lines
Protocol conversion servers, firmware, and other softwares can easily convert from a generic layout to something more exotic like OSD or [insert site specific protocol here].
NFS itself should not be delving into low-level storage details like this. Clients should not need to know low-level details (like stripe sizes). In Linux, we call this a layering violation.
This is the essence of the layout type concept, implemented as a layout driver in the linux nfsv4.1 client implementation. The fact that the files layout type definition are part of the nfsv4.1 protocol are a mere fact that the files based layout type uses NFSv4.1 as the storage protocol. It could very well be defined in a separate RFC exactly like the blocks and objects layout type specifications and that can keep its internal data structures opaque to the generic NFSv4.1 protocol (which behaves as a transport protocol for the layout-type specific data)
Quoted 2 lines
Working on kernel storage drivers as I do, I can see the attraction of wanting to do things this way... but we invented layering and abstraction in computer science for good reasons :)
Yup. I think that the layout driver is the software layer you're looking for.
Quoted 7 lines
* pNFS major issue #3: no longer a "closed loop" protocol
By permitting multiple layout types, and in particular undefined (site-specific) layout types, it is by definition _impossible_ for anyone to claim full protocol interoperability with other implementations.
The number of possible combinations approaches infinity, with obvious consequences on testing, and production software quality.
The non-standard layout types are defined as experimental. To claim interoperability one would need to publish a suitable specification of the new layout type (see "Defining new layout types", section 22.4 of http://www.nfsv4-editor.org/draft-18/draft-ietf-nfsv4-minorversion1-18.html#pnfsiana).
Though I agree that allowing a single file to be accessed with multiple layout types (in theory) complicates testing typically there will be at most one layout type per file system.
Quoted 2 lines
And when a marketing department advertises "fully NFSv4.1 compliant!" on ther company's appliance, it is trivial for any engineer to construct another "fully NFSv4.1 compliant" setup -- with equivalent authentication, metadata and data sets -- that is not interoperable except via the fallback case (copy through the metadata server).
The compliance with NFSv4.1 is indeed tied with the legacy I/O path, and for pnfs, with the files layout type, as it is a part of NFSv4.1. Other implementation of NFSv4.1 with pNFS over non-files layout types will have to claim compliance with their respective standards. For example, Panasas's implementation will need to comply with the OSD standard, iSCSI (or FC), the Object-based pNFS RFC, and finally NFSv4.1
Quoted 1 line
Such interoperability breakdowns are IMO not in the spirit of NFS.
That's a part of making progress IMO...
Benny
Quoted 4 lines
Jeff
Quoted 9 lines · Jeff Garzik
Quoted 4 lines · Benny Halevy
Although It's rather late in the process since the NFSv4 working group is close to putting the NFSv4.1 Internet-draft up for last call, we would certainly appreciate more implementation feedback.
I am more than happy to give feedback, though (as you say) it is probably too late for substantial feedback to have any large effect.
About five years too late. Witness the uncomfortable hacks required to retrofit the extra Sessions fields into the protocol without changing the basic COMPOUND arguments and results structures, which a minor version doesn't allow you to do.
Quoted 16 lines
My general engineering opinions of pNFS:
* Fills an obvious need: eliminating the need to copy data through the metadata interface to backend storage. Many clear, tangible benefits here.
*[...]
Pick ONE client storage protocol [...] and stick to it.* [...]
Pick ONE layout type, and stick to it.
* pNFS major issue #3: no longer a "closed loop" protocol
Quoted 7 lines
And when a marketing department advertises "fully NFSv4.1 compliant!" on ther company's appliance, it is trivial for any engineer to construct another "fully NFSv4.1 compliant" setup -- with equivalent authentication, metadata and data sets -- that is not interoperable except via the fallback case (copy through the metadata server).
Such interoperability breakdowns are IMO not in the spirit of NFS.
Strongly agreed with all the above. It's difficult to avoid the conclusion that the current pNFS spec is designed to allow the existing parallel filesystem vendors to sell "standards compliant" solutions that only work with their client software and where all the MDS and DS machines need to be bought from the same vendor. If the spec defined a single layout type, and all three protocols involved were variants of NFSv4.1 and were defined in the spec, we would have a true open standard.
Signature · 3 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group. The cake is *not* a lie. I don't speak for SGI.
Quoted 10 lines · Greg Banks
Strongly agreed with all the above. It's difficult to avoid the conclusion that the current pNFS spec is designed to allow the existing parallel filesystem vendors to sell "standards compliant" solutions that only work with their client software and where all the MDS and DS machines need to be bought from the same vendor. If the spec defined a single layout type, and all three protocols involved were variants of NFSv4.1 and were defined in the spec, we would have a true open standard.
Greg, I'm afraid your conclusion is just wrong. What exactly is it based on?
I'd appreciate if you could look again at the current Internet Drafts comprising
NFSv4.1 and layout types and please raise any issues you see in the specs that would jeopardize interoperability between different client software vendors
and different server / storage vendors.
http://tools.ietf.org/html/draft-ietf-nfsv4-minorversion1
http://tools.ietf.org/html/draft-ietf-nfsv4-pnfs-obj
http://tools.ietf.org/html/draft-ietf-nfsv4-pnfs-block
There is indeed a third protocol in the overall architecture used by the MDS to manage
the storage devices and this protocol is outside the scope of NFSv4.1. This may lead
to non-interoperable implementations of server/storage systems but that definitely was
not the intent of the design decision to leave the storage management protocol unspecified
in NFSv4.1.
The object-based layout type. for example. is based on using the standard OSD protocol
between the MDS and the OSDs for control as well as between the clients and the OSDs
for data transfer. How does that preclude interoperability between different client, MDS,
and DS vendors if the MDS, OSDs, and clients comply with T-10 OSD and MDS and
client comply with NFSv4.1 and the pnfs-obj RFC (when it becomes one)?
Benny
Quoted 4 lines · Benny Halevy
Jeff, taking into account the amount of effort people and different organizations have already put into NFSv4 and NFSv4.1 I wish you could tunnel your inventive energy into making NFSv4.1 better rather than trying to reinvent NFS/RPC/XDR.
It's also important to have fun. Imagining what you could do from a clean slate is, if nothing else, a fun exercise. And it may end up with ideas that turn out to be implementable without starting from scratch.
--b.
Quoted 8 lines · J. Bruce Fields
Quoted 5 lines · Benny Halevy
Jeff, taking into account the amount of effort people and different organizations have already put into NFSv4 and NFSv4.1 I wish you could tunnel your inventive energy into making NFSv4.1 better rather than trying to reinvent NFS/RPC/XDR.
Quoted 6 lines
It's also important to have fun. Imagining what you could do from a clean slate is, if nothing else, a fun exercise. And it may end up with ideas that turn out to be implementable without starting from scratch.
--b.
Absolutely :)
And I certainly hope it will benefit all of us.
Benny
[About v4]
Quoted 4 lines · Jeff Garzik
Quoted 4 lines
Quoted 2 lines
I really wish the entire wire protocol were scrapped and replaced with something more sane, and easier to parse.
You had me worried there for a moment, I thought you might be the first person to admit to liking the NFS4 protocol design.
Couldn't agree more.
Quoted 4 lines
In my personal opinion, version 4 of NFS is a quantum-leap improvement over previous versions. While I used NFS v3 extensively, I always felt it was a crappy protocol, and unworthy of serious development effort. That changed with v4.
...
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
It's tempting to see what would arise from a clean-slate wire protocol effort, something that is otherwise compatible with NFS 4.x operations, objects, and data model.
It's more like v4 is a vast relative improvement over prior NFS. Given the huge number of NFS users and sites, IMO v4 is a huge improvement for Unix file sharing overall.
Many years ago, before NFSv4 was finished, I felt the same. I was waiting for v4 and thought that everything would be so much better. I wanted to help and started the "pynfs" project. Today, I have a different opinion. I think v3 is a fairly good protocol, if you use it correctly. For example, many people don't realize that you don't need the portmapper, that you can use a single well-known TCP port, that you can use RPCSEC_GSS and so forth, even with v3.
I think v4 has a few valuable improvements, but it comes with a very high price. v3 has a minimalistic beauty which v4 lacks. For example, take a look at the OPEN operation with 7 arguments, of which many are complex data structures:
(cfh), seqid, share_access, share_deny, owner, openhow, claim -> (cfh), stateid, cinfo, rflags, open_confirm, attrset delegation
Not pretty...
Quoted 2 lines
Oh, certainly. I was mainly thinking a replacement of the wire protocol would be an easier step for people to swallow than a new protocol.
I've been thinking of trying to put together something like NFS v3.5. Some parts of v4 are nice, but the complexity is too high.
Regards,
---
Peter Åstrand ThinLinc Chief Developer
Cendio AB http://www.cendio.se
Wallenbergs gata 4
583 30 Linköping Phone: +46-13-21 46 00
Quoted 7 lines · On Friday January 4, [email protected] wrote:
Quoted 3 lines
Oh, certainly. I was mainly thinking a replacement of the wire protocol would be an easier step for people to swallow than a new protocol.
I've been thinking of trying to put together something like NFS v3.5. Some parts of v4 are nice, but the complexity is too high.
That is soooooooo tempting.
Of course we would need to be clear on how it is better than NFS, CIFS, 9P, and CRFS to name but a few. And it would not be a small undertaking.
But it is very tempting.
NeilBrown
A message that is not in the archive (sent off-list or not yet received): Pine.LNX.4.64.0801040954070.5004-K9BqGu7AvB3wj5YHdwD3Ga2PxDmRETKR@public.gmane.org
Quoted 7 lines · On Fri, Jan 04, 2008 at 10:15:00AM +0100, Peter =C5strand wrote:
=20
[About v4]
=20
=20
On Fri, 4 Jan 2008, Jeff Garzik wrote:
=20Quoted 1 line
Quoted 1 line
Quoted 1 line
I really wish the entire wire protocol were scrapped and replac=
ed with
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
something more sane, and easier to parse.=20
You had me worried there for a moment, I thought you might be the=
first
Quoted 7 lines
Quoted 1 line
Quoted 1 line
person to admit to liking the NFS4 protocol design.
=20
Couldn't agree more.=20
=20
=20
=20Quoted 1 line
In my personal opinion, version 4 of NFS is a quantum-leap improvem=
ent over
Quoted 1 line
Quoted 1 line
previous versions. While I used NFS v3 extensively, I always felt =
it was a
Quoted 1 line
Quoted 1 line
crappy protocol, and unworthy of serious development effort. That c=
hanged with
Quoted 3 lines
Quoted 1 line
v4.
...
Quoted 1 line
Quoted 1 line
Quoted 1 line
It's tempting to see what would arise from a clean-slate wire p=
rotocol
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
effort, something that is otherwise compatible with NFS 4.x ope=
rations,
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
objects, and data model.
=20
It's more like v4 is a vast relative improvement over prior NFS. G=
iven the
Quoted 1 line
Quoted 1 line
huge number of NFS users and sites, IMO v4 is a huge improvement fo=
r Unix file
Quoted 3 lines
Quoted 1 line
sharing overall.
=20
Many years ago, before NFSv4 was finished, I felt the same. I was wai=
ting=20
Quoted 1 line
for v4 and thought that everything would be so much better. I wanted =
to=20
Quoted 1 line
help and started the "pynfs" project. Today, I have a different opini=
on. I=20
Quoted 1 line
think v3 is a fairly good protocol, if you use it correctly. For exam=
ple,=20
Quoted 1 line
many people don't realize that you don't need the portmapper, that yo=
u can=20
Quoted 2 lines
use a single well-known TCP port, that you can use
RPCSEC_GSSand so=20 forth, even with v3.=20
Somehow this reminds me of IPv4 vs. IPv6. IIRC some protocol features h= ave
in a sense been "backported" to IPv4.
--=20
=46rank
Quoted 1 line · Peter =C3=85strand wrote:
Many years ago, before NFSv4 was finished, I felt the same. I was wai=
ting=20
Quoted 1 line
for v4 and thought that everything would be so much better. I wanted =
to=20
Quoted 1 line
help and started the "pynfs" project. Today, I have a different opini=
on. I=20
<grin>
Quoted 1 line
think v3 is a fairly good protocol, if you use it correctly. For exam=
ple,=20
Quoted 1 line
many people don't realize that you don't need the portmapper, that yo=
u can=20
Quoted 2 lines
use a single well-known TCP port, that you can use
RPCSEC_GSSand so=20 forth, even with v3.=20
Absolutely... But still, I think integrated mount protocol (aka pseudo= =20
filesystem namespace) and integrated locking were big steps forward.=20 You really shouldn't need more than one protocol.
Speaking of RPCSEC_GSS: I would love to see a much more straightforward= =20
authentication process, something /not/ buried inside special behaviors= =20
triggered by opcodes found in an opaque cred struct :/ RPCSEC_GSS=20 context creation, the special casing around the 'null' procedure, and=20 the overloading of the RPC data portion of things is a huge pain to=20 implement.
Authentication and security should be simple, tough to screw up. I=20 would tend to prefer an ASCII-based authentication/security negotiation= =20
at the start of a [SCTP|TCP] stream.
Use TLS to give most people what they want: AUTH_SYS with encryption.=20 GSSAPI is fine as a "required option" but you shouldn't need GSSAPI to=20 do simple wire encryption between IP-authenticated hosts.
Quoted 1 line
I think v4 has a few valuable improvements, but it comes with a very =
high=20
Quoted 1 line
price. v3 has a minimalistic beauty which v4 lacks. For example, take=
a=20
Quoted 1 line
look at the OPEN operation with 7 arguments, of which many are comple=
x=20
Quoted 6 lines
data structures:
=20
(cfh), seqid,share_access,share_deny, owner, openhow, claim -> (cfh), stateid, cinfo, rflags,open_confirm, attrset delegation
=20
Not pretty... =20
heh, tell me about it. First I started out using rpcgen, then rewrote=20 everything to do raw XDR decoding. OPEN is huge.
IMO, OPEN should be split into multiple operations, probably one for=20 each "OPEN arm". It's not like new opcode numbers are expensive.
Or, hope of hopes, simplify OPEN in some other manner, like delegating=20 tasks to other operations.
Quoted 1 line
Quoted 1 line
Oh, certainly. I was mainly thinking a replacement of the wire prot=
ocol would
Quoted 3 lines
Quoted 1 line
be an easier step for people to swallow than a new protocol.
=20
I've been thinking of trying to put together something like NFS v3.5.=
Some=20
Quoted 1 line
parts of v4 are nice, but the complexity is too high.=20
Agreed that's it's quite complex.
One of my personal desires is for a high level of cache coherence=20 throughout the system for all clients (though perhaps an admin could=20 optionally relax this requirement). I'm a fan of Google's "Chubby", a=20 distributed reliable filesystem that stalls client writes until cache=20 invalidations for the associated byte range are processed for all=20 interested clients.
And anything approaching cache coherence requires some complexity :/
Another thing I like about NFSv4 is that batching sequences into chunks= =20
of fine-grained operations is generally a useful practice. So while th= e=20
end result (COMPOUND) is a bit of a pain, bundling a sequence of=20 operations into a single unit is useful.
Jeff
Quoted 9 lines
Quoted 5 lines
think v3 is a fairly good protocol, if you use it correctly. For example, many people don't realize that you don't need the portmapper, that you can use a single well-known TCP port, that you can use
RPCSEC_GSSand so forth, even with v3.Absolutely... But still, I think integrated mount protocol (aka pseudo filesystem namespace) and integrated locking were big steps forward. You really shouldn't need more than one protocol.
I don't think the mount protocol is a big problem. Either you can continue to use the mount protocol and just have one export (/) and export all file systems below it ("nohide"). Modern clients (such as modern Linux kernels) will automatically create "sub mounts" when traversing and discovering new fsids.
Or, you can get rid of the mount protocol by using WebNFS and the public filehandle.
File locking support is harder, though.
Quoted 7 lines
Authentication and security should be simple, tough to screw up. I would tend to prefer an ASCII-based authentication/security negotiation at the start of a [SCTP|TCP] stream.
Use TLS to give most people what they want:
AUTH_SYSwith encryption. GSSAPI is fine as a "required option" but you shouldn't need GSSAPI to do simple wire encryption between IP-authenticated hosts.
SSH is another option if you just want encryption, but my impression is that AUTH_SYS is a very big problem as well.
Quoted 8 lines
heh, tell me about it. First I started out using rpcgen, then rewrote everything to do raw XDR decoding. OPEN is huge.
IMO, OPEN should be split into multiple operations, probably one for each "OPEN arm". It's not like new opcode numbers are expensive.
Or, hope of hopes, simplify OPEN in some other manner, like delegating tasks to other operations.
Or perhaps aiming for something less than perfect. Remember, the perfect is the enemy of the good.
Regards,
---
Peter Åstrand ThinLinc Chief Developer
Cendio AB http://www.cendio.se
Wallenbergs gata 4
583 30 Linköping Phone: +46-13-21 46 00
_______________________________________________
NFSv4 mailing list
[email protected]
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4
A message that is not in the archive (sent off-list or not yet received): Pine.LNX.4.64.0801042030380.18738-K9BqGu7AvB3wj5YHdwD3Ga2PxDmRETKR@public.gmane.org
Quoted 6 lines · Peter Åstrand
Quoted 3 lines
Use TLS to give most people what they want:
AUTH_SYSwith encryption. GSSAPI is fine as a "required option" but you shouldn't need GSSAPI to do simple wire encryption between IP-authenticated hosts.SSH is another option if you just want encryption, but my impression is that
AUTH_SYSis a very big problem as well.
I've been looking into this recently, essentially ending up down a very similar track to the SSiLKey proposal presented at IETF67:
http://www3.ietf.org/proceedings/06nov/slides/spkm-5/spkm-5.ppt
The basic idea in SSiLKey is to boostrap an RPCSEC_GSS session with TLS and then layer LIPKEY on top.
It seems to me that SSH might be preferrable to TLS as a low infrastucture mechanism, as many people already have ssh keys (and use them), there's no need for a HTTP server, and SSH already supports a variety of authentication mechanisms.
In the SSH case, I'm not sure yet whether LIPKEY would be the most appropriate mechanism to utilize, and whether this scheme might in fact be cleaner overall without using GSS at this level. i.e. GSS can be used directly by SSH itself if desired, and there's also PAM.
There's also a patch for SSH to utilize GPG keys (raising the possibility of utilizing existing webs of trust), although it does not seem to be current.
- James
Signature · 2 lines
James Morris <[email protected]>
I like it. NFS 3.5. Drop pNFS into the v3 protocol (a greatly simplified = pNFS compared to what we ended up with in 4.1), and you'd have yourself a s= weet little distributed fs. The important part is that any such effort be = called "NFS x.y". Naming is important, or we'd be running AFS/DFS instead = of NFS 4.0 (please, try to resist flaming me on the hyperbole).
-Dan
-----Original Message-----
From: Peter =C5strand [mailto:[email protected]] =
Sent: Friday, January 04, 2008 1:15 AM
To: Jeff Garzik
Cc: NFS list; [email protected]
Subject: Re: A new NFSv4 server...
[About v4]
Quoted 6 lines · Jeff Garzik
Quoted 6 lines
Quoted 3 lines
I really wish the entire wire protocol were scrapped and replaced =
with something more sane, and easier to parse.
You had me worried there for a moment, I thought you might be the =
first person to admit to liking the NFS4 protocol design.
Couldn't agree more. =
Quoted 7 lines
In my personal opinion, version 4 of NFS is a quantum-leap improvement =
over previous versions. While I used NFS v3 extensively, I always =
felt it was a crappy protocol, and unworthy of serious development =
effort. That changed with v4.
...
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
It's tempting to see what would arise from a clean-slate wire =
protocol effort, something that is otherwise compatible with NFS =
Quoted 8 lines
Quoted 1 line
Quoted 1 line
4.x operations, objects, and data model.
=
It's more like v4 is a vast relative improvement over prior NFS. =
Given the huge number of NFS users and sites, IMO v4 is a huge =
improvement for Unix file sharing overall.
Many years ago, before NFSv4 was finished, I felt the same. I was waiting f= or v4 and thought that everything would be so much better. I wanted to help=
and started the "pynfs" project. Today, I have a different opinion. I thin=
k v3 is a fairly good protocol, if you use it correctly. For example, many = people don't realize that you don't need the portmapper, that you can use a=
single well-known TCP port, that you can use RPCSEC_GSS and so forth, even= with v3. =
I think v4 has a few valuable improvements, but it comes with a very high p= rice. v3 has a minimalistic beauty which v4 lacks. For example, take a look=
at the OPEN operation with 7 arguments, of which many are complex data str=
uctures:
(cfh), seqid, share_access, share_deny, owner, openhow, claim -> (cfh), stateid, cinfo, rflags, open_confirm, attrset delegation
Not pretty... =
Quoted 3 lines
Oh, certainly. I was mainly thinking a replacement of the wire =
protocol would be an easier step for people to swallow than a new protoco=
l.
I've been thinking of trying to put together something like NFS v3.5. Some = parts of v4 are nice, but the complexity is too high. =
Regards, =
---
Peter =C5strand ThinLinc Chief Developer
Cendio AB http://www.cendio.se
Wallenbergs gata 4
583 30 Link=F6ping Phone: +46-13-21 46 00
[About your implementation]
Quoted 2 lines
In case some developers are interested... I'm poking at a from-scratch userland NFSv4 server, as a side project.
Cool! Being one of the unfs3 developers, I believe that userland NFS servers are very useful.
Quoted 2 lines
In the first step down this long path, I've created an NFSv4 userland server from scratch. Currently it merely serves data straight from RAM, but the long
Do you know about the n4 project
(http://cvs.samba.org/cgi-bin/cvsweb/n4/) ? It was abandon many years ago, but might have some usefulness.
Quoted 3 lines
term goal is to permit modular storage backends. Thus you could implement a simple RAM backend, an sqlite-based backend or a complex distributed storage backend.
unfs3 has a basic modular backend system. I wonder if it would be possible to merge your server with unfs3, and still have something that's readable. If you are interested, take a look at
http://cvs.lysator.liu.se/viewcvs/viewcvs.cgi/unfs3/?root=unfs3.
Regards,
---
Peter Åstrand ThinLinc Chief Developer
Cendio AB http://www.cendio.se
Wallenbergs gata 4
583 30 Linköping Phone: +46-13-21 46 00
Quoted 3 lines · Peter =C5strand wrote:
Do you know about the n4 project
(http://cvs.samba.org/cgi-bin/cvsweb/n4/) ? It was abandon many years ago, but might have some usefulness. =
Nope, I'll definitely take look. Already imported it into git using =
git-cvsimport. :)
Quoted 1 line
Quoted 1 line
term goal is to permit modular storage backends. Thus you could impleme=
nt a
Quoted 1 line
Quoted 1 line
simple RAM backend, an sqlite-based backend or a complex distributed sto=
rage
Quoted 7 lines
Quoted 1 line
backend.
=
unfs3 has a basic modular backend system. I wonder if it would be possible to merge your server with unfs3, and still have something that's readable. If you are interested, take a look at
http://cvs.lysator.liu.se/viewcvs/viewcvs.cgi/unfs3/?root=3Dunfs3.
I'm always interested in [legally] stealing useful ideas and code, so I =
will definitely take a look.
I was sorta thinking about implementing a couple backends, and seeing =
what API organically appears. That's sorta how Linux kernel API =
"design" happens, and it tends to produce something useful and compact, =
if not a bit unique :)
I can imagine that some backends may wish that the server handle some =
details of state and locking, while other backends may wish to record =
all that information into a database in stable storage. So it's =
difficult to forecast how all that will fall out in the end. unfs3 =
probably has many lessons to teach me...
So far my best resource for NFS technical "folklore" is generally =
google, which turns up a wealth of useful mailing list discussions =
involving neilb, meisler, and others.
Jeff
P.S. cvsps, the util git-cvsimport uses, doesn't seem to like the unfs3 =
CVS repository. Any ideas?
Running cvsps...
connect error: Network is unreachable
cvs rlog: Logging unfs3
cvs rlog: Logging unfs3/Config
cvs rlog: Logging unfs3/Extras
cvs rlog: Logging unfs3/contrib
cvs rlog: Logging unfs3/contrib/nfsotpclient cvs rlog: Logging unfs3/contrib/nfsotpclient/mountclient cvs rlog: Logging unfs3/contrib/rpcproxy
cvs rlog: Logging unfs3/doc
Fetching LICENSE v 1.1
New LICENSE: 1416 bytes
Fetching Makefile.in v 1.1
Unknown: error
The same command works just fine with 99% of cvs repositories out there, =
pserver, ssh, or whatever.
And a regular CVS checkout works just fine, I am able to check out and =
browse files and look for gems.
Quoted 2 lines · Jeff Garzik
P.S. cvsps, the util git-cvsimport uses, doesn't seem to like the unfs3 CVS repository. Any ideas?
Seems to work for me. I tested with:
$ cvs -d :pserver:[email protected]:/cvsroot/unfs3 co unfs3 $ cd unfs3 $ cvsps
Using cvsps version 2.1.
Rgds,
---
Peter Åstrand ThinLinc Chief Developer
Cendio AB http://www.cendio.se
Wallenbergs gata 4
583 30 Linköping Phone: +46-13-21 46 00