Minneapolis Cluster Summit, July 29-30
gfs needs to run in the kernel. dlm should run in the kernel since gfs uses it so heavily. cman is the clustering subsystem on top of which both of those are built and on which both depend quite critically. It simply makes most sense to put cman in the kernel for what we're doing with it. That's not a dogmatic position, just a practical one based on our experience.
This isn't really acceptable. We've spent a long time throwing things out of the kernel so you really need a good justification for putting things in again. "it makes sense" and "its just practical" aren't sufficient.
You also face two other additional hurdles:
1) GFS today uses a user space DLM. What critical problems does this have that you suddenly need to move it all into the kernel?
2) We have numerous other clustering products for Linux, none of which (well except the Veritas one) has any requirement at all on having pieces in the kernel. If all the others operate in user space, why does yours need to be in the kernel?
So do you have a justification for requiring these as kernel components?
James
Quoted 10 lines · James Bottomley
gfs needs to run in the kernel. dlm should run in the kernel since gfs uses it so heavily. cman is the clustering subsystem on top of which both of those are built and on which both depend quite critically. It simply makes most sense to put cman in the kernel for what we're doing with it. That's not a dogmatic position, just a practical one based on our experience.
This isn't really acceptable. We've spent a long time throwing things out of the kernel so you really need a good justification for putting things in again. "it makes sense" and "its just practical" aren't sufficient.
The "it" refers to gfs. This means gfs doesn't make a lot of sense and isn't very practical without it. I'm not the one to speculate on what gfs would become otherwise, others would do that better.
Quoted 4 lines
You also face two other additional hurdles:
1) GFS today uses a user space DLM. What critical problems does this have that you suddenly need to move it all into the kernel?
GFS does not use a user space dlm today. GFS uses the client-server gulm lock manager for which the client (gfs) side runs in the kernel and the gulm server runs in userspace on some other node. People have naturally been averse to using servers like this with gfs for a long time and we've finally created the serverless dlm (a la VMS clusters). For many people this is the only option that makes gfs interesting; it's also what the opengfs group was doing.
This is a revealing discussion. We've worked hard to make gfs's lock manager independent from gfs itself so it could be useful to others and make gfs less monolithic. We could have left it embedded within the file system itself -- that's what most other cluster file systems do. If we'd done that we would have avoided this objection altogether but with an inferior design. The fact that there's an independent lock manager to point at and question illustrates our success. The same goes for the cluster manager. (We could, of course, do some simple glueing together and make a monlithic system again :-)
Quoted 4 lines
2) We have numerous other clustering products for Linux, none of which (well except the Veritas one) has any requirement at all on having pieces in the kernel. If all the others operate in user space, why does yours need to be in the kernel?
If you want gfs in user space you don't want gfs; you want something different.
Signature · 1 line
Dave Teigland <[email protected]>
Quoted 3 lines · David Teigland
The "it" refers to gfs. This means gfs doesn't make a lot of sense and isn't very practical without it. I'm not the one to speculate on what gfs would become otherwise, others would do that better.
Quoted 2 lines · This is what you actually
It simply makes most sense to put cman in the kernel for what we're doing with it.
I interpret that to mean you think cman (your cluster manager) should be in the kernel. Is this incorrect?
Quoted 6 lines
Quoted 5 lines
You also face two other additional hurdles:
1) GFS today uses a user space DLM. What critical problems does this have that you suddenly need to move it all into the kernel?
Quoted 6 lines
GFS does not use a user space dlm today. GFS uses the client-server gulm lock manager for which the client (gfs) side runs in the kernel and the gulm server runs in userspace on some other node. People have naturally been averse to using servers like this with gfs for a long time and we've finally created the serverless dlm (a la VMS clusters). For many people this is the only option that makes gfs interesting; it's also what the opengfs group was doing.
OK, whatever you choose to call it, the previous lock manager used by gfs was userspace.
OK, so why is a kernel based DLM the only option that makes GFS interesting? What are the concrete advantages you achieve with a kernel based DLM that you don't get with a user space one? There are plenty of symmetric serverless userspace DLM implementations that follow the old VMS (and even updated by Oracle) spec.
Steve Dake has already given a pretty compelling list of why you shouldn't put the DLM and clustering in the kernel, what is the more compelling list of reasons why it should be?
Quoted 8 lines
This is a revealing discussion. We've worked hard to make gfs's lock manager independent from gfs itself so it could be useful to others and make gfs less monolithic. We could have left it embedded within the file system itself -- that's what most other cluster file systems do. If we'd done that we would have avoided this objection altogether but with an inferior design. The fact that there's an independent lock manager to point at and question illustrates our success. The same goes for the cluster manager. (We could, of course, do some simple glueing together and make a monlithic system again :-)
I'm not questioning your goal, merely your in-kernel implementation. Sharing is good. However things which are shared don't automatically have to be in-kernel.
Quoted 6 lines
Quoted 4 lines
2) We have numerous other clustering products for Linux, none of which (well except the Veritas one) has any requirement at all on having pieces in the kernel. If all the others operate in user space, why does yours need to be in the kernel?
If you want gfs in user space you don't want gfs; you want something different.
I didn't say GFS, I said "cluster products". That's the DLM and CMAN pieces of your architecture.
Once you can convince us that CMAN et al should be in the kernel, the next stage of the discussion would be the API. Several groups (like GGL, SAF and OCF) have done API work for clusters. They were mostly careful to select APIs that avoided mandating cluster policy. You seem to have chosen a particular policy (voting quorate) to implement. Again, that's a red flag. Policy should not be in the kernel; if we all agree there should be in-kernel APIs for clustering then they should be sufficiently abstracted to support all current cluster policies.
James