Non-GPL export of invalidate_mmap_range

The attached patch to make invalidate_mmap_range() non-GPL exported seems to have been lost somewhere between 2.6.1-mm4 and 2.6.1-mm5. It still applies cleanly. Could you please take it up again?

Hello, Andrew,

The attached patch to make invalidate_mmap_range() non-GPL exported seems to have been lost somewhere between 2.6.1-mm4 and 2.6.1-mm5. It still applies cleanly. Could you please take it up again?

Quoted 5 lines · Paul E. McKenney

Hello, Andrew,

The attached patch to make invalidate_mmap_range() non-GPL exported seems to have been lost somewhere between 2.6.1-mm4 and 2.6.1-mm5. It still applies cleanly. Could you please take it up again?

And there's still no reason to ease IBM's GPL violations by exporting deep VM internals. The GPLed DFS you claimed you needed this for still hasn't shown up but instead you want to change the export all the time.

Tells a lot about IBMs promises..

IBM shipped the promised SAN Filesystem some months ago.

Quoted 2 lines · Paul E. McKenney

IBM shipped the promised SAN Filesystem some months ago.

Neat, but it's hard to see the relevance of this to your patch.

I don't see any licensing issues with the patch because the filesystem which needs it clearly meets Linus's "this is not a derived work" criteria.

Quoted 8 lines · Andrew Morton

Quoted 2 lines · Paul E. McKenney

IBM shipped the promised SAN Filesystem some months ago.

Neat, but it's hard to see the relevance of this to your patch.

I don't see any licensing issues with the patch because the filesystem which needs it clearly meets Linus's "this is not a derived work" criteria.

it does?

It needed no changes to work on linux?

it only uses "core unix" apis ?

it needs no changes to the core kernel? *buzz*

It doesn't require knowledge of deep and changing internals ? *buzz*

It doesn't need changing for various kernel versions ?

I remember this baby overriding syscalls and the like not too long ago...

The word "clearly" isn't correct imo. Just because something has a few lines of code that started on another OS doesn't make it "clearly" not a derived work, at least not in my eyes.

There is a small shim layer required, but the bulk of the code implementing GPFS is common between AIX and Linux. It was on AIX first by quite a few years.

Quoted 1 line

it only uses "core unix" apis ?

If they are made available, yes. That is the point of this patch, after all. ;-)

Quoted 1 line

It doesn't require knowledge of deep and changing internals ? *buzz*

That is indeed the idea.

Quoted 3 lines · Paul E. McKenney

There is a small shim layer required, but the bulk of the code implementing GPFS is common between AIX and Linux. It was on AIX first by quite a few years.

Small glue layer? Unfortunately ibm took it off the website, but the thing is damn huge.

Quoted 4 lines

Quoted 1 line

it only uses "core unix" apis ?

If they are made available, yes. That is the point of this patch, after all. ;-)

No, that's wrong. It patches the syscall table and plays evilish tricks with lowlevel MM code.

Quoted 3 lines

Quoted 1 line

It doesn't require knowledge of deep and changing internals ? *buzz*

That is indeed the idea.

The one on the ibm website a little ago did. You're free to upload a new one that clearly doesn't need all this, but..

The sys_call_table stuff was under #ifdef, and was intended for use by a research project that was later put out of its misery. This stuff has since been removed from the source tree.

As to the evilish tricks with lowlevel MM code, the whole point of the mmap_invalidate_range() patch is to be able to rid GPFS of exactly these evilish tricks.

I don't understand why IBM is pushing this dubious change right now,

GPL violation and thus copyright violation issues in Linux is the last thing IBM wants to see in the press with the current mess going on, right?

a) Does the export make technical sense? Do filesystems have legitimate need for access to this symbol?

(really, a) is sufficient grounds, but for real-world reasons:)

b) Does the IBM filsystem meet the kernel's licensing requirements?

It appears that the answers are a): yes and b) probably.

Please, feel free to add additional criteria. We could also ask "do we want to withhold this symbols to encourage IBM to GPL the filesystem" or "do we simply refuse to export any symbol which is not used by any GPL software" (if so, why?).

But at the end of the day, if we decide to not export this symbol, we owe Paul a good, solid reason, yes?

Quoted 8 lines · Andrew Morton

a) Does the export make technical sense? Do filesystems have legitimate need for access to this symbol?

(really, a) is sufficient grounds, but for real-world reasons:)

b) Does the IBM filsystem meet the kernel's licensing requirements?

Quoted 1 line

It appears that the answers are a): yes and b) probably.

Well, the answer to b) is most likely not. I see it very hard to argue to have something like gpfs not beeing a derived work. The glue code they had online certainly looked very much like a derived work, and if the new version got better they wouldn't have any reason to remove it from the website, right?

Quoted 4 lines

Please, feel free to add additional criteria. We could also ask "do we want to withhold this symbols to encourage IBM to GPL the filesystem" or "do we simply refuse to export any symbol which is not used by any GPL software" (if so, why?).

Yes. Andrew, please read the GPL, it's very clear about derived works. Then please tell me why you think gpfs is not a derived work.

Quoted 2 lines

But at the end of the day, if we decide to not export this symbol, we owe Paul a good, solid reason, yes?

Yes. We've traditionally not exported symbols unless we had an intree user, and especially not if it's for a module that's not GPL licensed.

We had this discussion with Linus a few time, maybe he can comment again to make it clear.

Yes. Andrew, please read the GPL, it's very clear about derived works. Then please tell me why you think gpfs is not a derived work.

Quoted 3 lines · Christoph Hellwig

Yes. Andrew, please read the GPL, it's very clear about derived works. Then please tell me why you think gpfs is not a derived work.

OK, so I looked at the wrapper. It wasn't a tremendously pleasant experience. It is huge, and uses fairly standard-looking filesytem interfaces and locking primitives. Also some awareness of NFSV4 for some reason.

Still, the wrapper is GPL so this is not relevant. Its only use is to tell us whether or not the non-GPL bits are "derived" from Linux, and it doesn't do that.

OK, so I looked at the wrapper. It wasn't a tremendously pleasant experience. It is huge, and uses fairly standard-looking filesytem interfaces and locking primitives. Also some awareness of NFSV4 for some reason.

Still, the wrapper is GPL so this is not relevant.

Its only use is to tell
us whether or not the non-GPL bits are "derived" from Linux, and it doesn't do that.

Why do you believe that GPFS represents a kernel licensing violation?

Quoted 4 lines · Andrew Morton

OK, so I looked at the wrapper. It wasn't a tremendously pleasant experience. It is huge, and uses fairly standard-looking filesytem interfaces and locking primitives. Also some awareness of NFSV4 for some reason.

And pokes deep into internal structures that it shouldn't.

Quoted 1 line

Still, the wrapper is GPL so this is not relevant.

It's BSD licensed - they couldn't distribute it together with GPFS if it was GPL.

Quoted 3 lines

Its only use is to tell
us whether or not the non-GPL bits are "derived" from Linux, and it doesn't do that.

Well, something that needs an almost one megabyte big wrapper per defintion is not a standalone work but something that's deeply interwinded with the kernel. The tons of kernel version checks certainly show it's poking deeper than it should.

Quoted 1 line

Why do you believe that GPFS represents a kernel licensing violation?

See above. Something that pokes deep into internal structures and even needs new exports certainly is a derived work. There's a few different interpretations of the derived works clause in the GPL around, the FSF one wouldn't allow binary modules at all, and Linus' one is also pretty strict.

Quoted 1 line

And pokes deep into internal structures that it shouldn't.

Again, the point of the patch is to get rid of such poking.

Only if we can settle this, we can answer this export question. If we want to allow them, the export is a perfectly reasonable thing to ask for. If not, we probably need to add a few more _GPL barriers.

A rule of thumb might be whether any code in the tree uses a given export, and if not, prune it. Anything which even we don't use or export across the user-land boundary certainly qualifies as a kernel interna.

Currently, no kernel module seems to use this export. So I'd think such a point could certainly be made.

Quoted 10 lines · Lars Marowsky-Bree

Only if we can settle this, we can answer this export question. If we want to allow them, the export is a perfectly reasonable thing to ask for. If not, we probably need to add a few more _GPL barriers.

A rule of thumb might be whether any code in the tree uses a given export, and if not, prune it. Anything which even we don't use or export across the user-land boundary certainly qualifies as a kernel interna.

Currently, no kernel module seems to use this export. So I'd think such a point could certainly be made.

I'm not sure. I'm all for trimming the export list, but the real questions are
* does that export make sense?

* does it impose extra restrictions on what we can do with core
code? (without breaking it, that is)

* is it needed in the first place? If it's redundant - to hell it
goes.

Note that majority of the exported symbols fail at least one of the above and _that_ is why they should be killed. Whether their users are GPL or not doesn't matter - if they don't make sense, they must die, no matter what b0rken code might be using them.

IMNSHO the questions above should be answered first and AFAICS they hadn't been even discussed in that case.

A rule of thumb might be whether any code in the tree uses a given export, and if not, prune it. Anything which even we don't use or export across the user-land boundary certainly qualifies as a kernel interna.

Quoted 6 lines · Andrew Morton

OK, so I looked at the wrapper. It wasn't a tremendously pleasant experience. It is huge, and uses fairly standard-looking filesytem interfaces and locking primitives. Also some awareness of NFSV4 for some reason.

Still, the wrapper is GPL so this is not relevant.

Well, something that needs an almost one megabyte big wrapper per defintion is not a standalone work but something that's deeply interwinded with the kernel. The tons of kernel version checks certainly show it's poking deeper than it should.
...

Something that pokes deep into internal structures and even needs new exports certainly is a derived work.

Christoph Hellwig <[email protected]> wrote:

And the "But when you distribute..." part is what the Linus doctrine rubs out. Because it is unreasonable to say that a large piece of work such as this is "derived" from Linux.

Yes. Andrew, please read the GPL, it's very clear about derived works. Then please tell me why you think gpfs is not a derived work.

Quoted 2 lines

But at the end of the day, if we decide to not export this symbol, we owe Paul a good, solid reason, yes?

Yes. We've traditionally not exported symbols unless we had an intree user, and especially not if it's for a module that's not GPL licensed.

Quoted 2 lines

Yes. We've traditionally not exported symbols unless we had an intree user, and especially not if it's for a module that's not GPL licensed.

That's certainly a good rule of thumb and we (and I) have used it before.

What is the reasoning behind it?

Quoted 3 lines · Andrew Morton

Quoted 2 lines

Yes. We've traditionally not exported symbols unless we had an intree user, and especially not if it's for a module that's not GPL licensed.

Quoted 3 lines

That's certainly a good rule of thumb and we (and I) have used it before.

What is the reasoning behind it?

The reason is that someone who wants to distribute a binary only module has to show it's module is not a derived work, and someone who needs new core in the kernel and new exports pretty much shows his work is deeply integrated with the kernel.

inter-node cache consistency. Other distributed filesystems will need this and probably AIX already provides it.

Quoted 2 lines · Andrew Morton

inter-node cache consistency. Other distributed filesystems will need this and probably AIX already provides it.

You've probably not seen the AIX VM architecture. Good for you as it's not good for your stomache. I did when I still was SCAldera and although my NDAs don't allow me to go into details I can tell you that the AIX VM architecture is deeply tied into the segment architecture of the Power CPU and signicicantly different from any other UNIX variant.

So porting code from AIX that touches anything VM related is a complete rewrite.

You've probably not seen the AIX VM architecture. Good for you as it's not good for your stomache. I did when I still was SCAldera and although my NDAs don't allow me to go into details I can tell you that the AIX VM architecture is deeply tied into the segment architecture of the Power CPU and signicicantly different from any other UNIX variant.

Quoted 6 lines

You've probably not seen the AIX VM architecture. Good for you as it's not good for your stomache. I did when I still was SCAldera and although my NDAs don't allow me to go into details I can tell you that the AIX VM architecture is deeply tied into the segment architecture of the Power CPU and signicicantly different from any other UNIX variant.

Interesting, what version of AIX did you get access to? And how can you be sure thats still the case?

Quoted 2 lines · Christoph Hellwig

I don't understand why IBM is pushing this dubious change right now,

It isn't a dubious change, on technical grounds. It is reasonable for a distributed filesystem to want to be able to shoot down pte's which map sections of pagecache. Just as it is reasonable for the filesystem to be able to shoot down the pagecache itself.

We've exported much lower-level stuff than this, because some in-kernel module happened to use it.

We need to give Paul a reasoned and logically consistent answer to his request. For that we need to establish some sort of framework against which to make a decision and then make the decision.

One approach is a fait-accomplis from the top-level maintainer. Here, we're trying to do it in a different way.

I have proposed two criteria upon which this should be judged:

a) Does the export make technical sense? Do filesystems have legitimate need for access to this symbol?

(really, a) is sufficient grounds, but for real-world reasons:)

b) Does the IBM filsystem meet the kernel's licensing requirements?

It appears that the answers are a): yes and b) probably.

Please, feel free to add additional criteria. We could also ask "do we want to withhold this symbols to encourage IBM to GPL the filesystem" or "do we simply refuse to export any symbol which is not used by any GPL software" (if so, why?). Over to you.

But at the end of the day, if we decide to not export this symbol, we owe Paul a good, solid reason, yes?

Quoted 9 lines · Andrew Morton

Quoted 2 lines · Christoph Hellwig

I don't understand why IBM is pushing this dubious change right now,

It isn't a dubious change, on technical grounds. It is reasonable for a distributed filesystem to want to be able to shoot down pte's which map sections of pagecache. Just as it is reasonable for the filesystem to be able to shoot down the pagecache itself.

Quoted 2 lines

We've exported much lower-level stuff than this, because some in-kernel module happened to use it.

Probably not always the right choice, though... I highly suspect we far to much of our intestines are easily available.

Quoted 14 lines · David Weinehall

Quoted 11 lines · Andrew Morton

Quoted 1 line · Christoph Hellwig

I don't understand why IBM is pushing this dubious change right now,

It isn't a dubious change, on technical grounds. It is reasonable for a distributed filesystem to want to be able to shoot down pte's which map sections of pagecache. Just as it is reasonable for the filesystem to be able to shoot down the pagecache itself.

We've exported much lower-level stuff than this, because some in-kernel module happened to use it.

Quoted 2 lines

Probably not always the right choice, though... I highly suspect we far to much of our intestines are easily available.

Again, the whole point of the patch is to -reduce- the degree of intestinal export.

Thanx, Paul

a) Does the export make technical sense? Do filesystems have legitimate need for access to this symbol?

(really, a) is sufficient grounds, but for real-world reasons:)

b) Does the IBM filsystem meet the kernel's licensing requirements?

Quoted 4 lines · Andrew Morton

a) Does the export make technical sense? Do filesystems have legitimate need for access to this symbol?

(really, a) is sufficient grounds, but for real-world reasons:)

Technically, I assume both OCFS, Lustre, (OpenGFS), PolyServe and basically /everyone/ doing a cluster file system, proprietary or not, will eventually need this capability. Vendors have included hooks for this in 2.4 already anyway.

So on technical grounds, I'm strongly inclined to support it, but I would like to suggest that it is ensured that the hook is sufficient for all of the named CFS.

Paul, have you spoken with them?

Quoted 1 line

b) Does the IBM filsystem meet the kernel's licensing requirements?

If you are worried about this one, you can export it GPL-only, which as an Open Source developer I'd appreciate, but from a real-world business perspective would be unhappy about ;-)

Sincerely,
Lars Marowsky-Bree <[email protected]>

Signature · 3 lines

High Availability & Clustering	      \ ever tried. ever failed. no matter.
SUSE Labs			      | try again. fail again. fail better.
Research & Development, SUSE LINUX AG \ 	-- Samuel Beckett

Quoted 1 line

b) Does the IBM filsystem meet the kernel's licensing requirements?

If you are worried about this one, you can export it GPL-only, which as an Open Source developer I'd appreciate, but from a real-world business perspective would be unhappy about ;-)

It already is exported GPL-only, this is all about changing it to be for linking bin only modules as well...

Quoted 4 lines · Andrew Morton

Neat, but it's hard to see the relevance of this to your patch. I don't see any licensing issues with the patch because the filesystem which needs it clearly meets Linus's "this is not a derived work" criteria.

it does?

it needs no changes to the core kernel? *buzz*

I don't see any licensing issues with the patch because the filesystem which needs it clearly meets Linus's "this is not a derived work" criteria.

And I don't see a technical problem with the export: given that we export truncate_inode_pages() it makes sense to also export the corresponding pagetable shootdown function.

Yes, this is a sensitive issue. Can we please evaluate it strictly according to technical and licensing considerations?

Having said that, what concerns issues remain with Paul's patch?

OpenGFS and Sistina GFS use zap_page_range directly, essentially doing the same as invalidate_mmap_range but skipping any vmas belonging to MAP_PRIVATE mmaps.

Hi,

Quoted 3 lines · Daniel Phillips

OpenGFS and Sistina GFS use zap_page_range directly, essentially doing the same as invalidate_mmap_range but skipping any vmas belonging to MAP_PRIVATE mmaps.

Well, MAP_PRIVATE maps can contain shared pages too --- any page in a MAP_PRIVATE map that has been mapped but not yet written to is still shared, and still needs shot down on truncate().

Hi Stephen,

Quoted 7 lines · Stephen C. Tweedie

Hi,

Quoted 3 lines · Daniel Phillips

OpenGFS and Sistina GFS use zap_page_range directly, essentially doing the same as invalidate_mmap_range but skipping any vmas belonging to MAP_PRIVATE mmaps.

Quoted 3 lines

Well, MAP_PRIVATE maps can contain shared pages too --- any page in a MAP_PRIVATE map that has been mapped but not yet written to is still shared, and still needs shot down on truncate().

Exactly, and we ought to take this opportunity to do that properly, which is easy. I'm just curious how GPFS deals with this issue, or if it simply doesn't support MAP_PRIVATE.

GPFS supports MAP_PRIVATE, but does not specify the behavior if you change the underlying file. There are a number of things one can do, but one must keep in mind that different processes can MAP_PRIVATE the same file at different times, and that some processes might MAP_SHARED it at the same time that others MAP_PRIVATE it. Here are the alternatives I can imagine:

1. Any time a file changes, create a copy of the old version
for any MAP_PRIVATE vmas. This would essentially create a point-in-time copy of any file that a process mapped MAP_PRIVATE. This is arguably the most intuitive from the user's standpoint, but (a) it would not be a small change and
(b) I haven't heard of anyone coming up with a good use for it. Please enlighten me if I am missing a simple implementation or compelling uses.

2. Modify invalidate_mmap_range() to leave MAP_PRIVATE vmas.
as suggested by Daniel.

This would mean that a
process that had mapped a file MAP_PRIVATE and faulted in parts of it would see different versions of the file in different pages. This should be straightforward to implement, but in what situation is this skewed view of the file useful?

3. Modify invalidate_mmap_range() to leave MAP_PRIVATE vmas,
but invalidate those pages in the vma that have not yet been modified (that are not anonymous) as suggested by Stephen. This would mean that a process that had mapped a file MAP_PRIVATE and written on parts of it would see different versions of the file in different pages.

Again, in what situation is this skewed view of the file useful?

5. The current behavior, where the process's writes do not
flow through to the file, but all changes to the file are visible to the writing process.

6. Requiring that MAP_PRIVATE be applied only to unchanging
files, so that (for example) any change to the underlying file removes that file from any MAP_PRIVATE address spaces. Subsequent accesses would get a SEGV, rather than a surprise from silently changing data.

So, please help me out here... What do applications that MAP_PRIVATE changing files really expect to happen?

Quoted 7 lines · Paul E. McKenney

GPFS supports MAP_PRIVATE, but does not specify the behavior if you change the underlying file. There are a number of things one can do, but one must keep in mind that different processes can MAP_PRIVATE the same file at different times, and that some processes might MAP_SHARED it at the same time that others MAP_PRIVATE it. Here are the alternatives I can imagine:

Quoted 8 lines

1. Any time a file changes, create a copy of the old version
for any MAP_PRIVATE vmas. This would essentially create a point-in-time copy of any file that a process mapped MAP_PRIVATE. This is arguably the most intuitive from the user's standpoint, but (a) it would not be a small change and
(b) I haven't heard of anyone coming up with a good use for it. Please enlighten me if I am missing a simple implementation or compelling uses.

This is MAP_COPY I think. Even if somebody did manage to sneak it by Linus one day it would certainly not be under the guise of MAP_PRIVATE.

Quoted 2 lines

2. Modify invalidate_mmap_range() to leave MAP_PRIVATE vmas.
as suggested by Daniel.

I did not suggest that, rather I described the existing practice in OpenGFS and Sistina GFS, which at least does not destroy anonymous data. The correct behaviour is the one you describe in option 3, and we are perfectly willing to change GFS to obtain that behaviour. To be precise: I suggest we change invalidate_mmap_range to skip anon pages, and change vmtruncate to use something else, having the current semantics.

As a historical note: the behavior GFS obtains from option 2 is Posix-compliant, but falls short of Linus-compliance, who insists on completely accurate invalidation behavior as is right and proper.

Quoted 6 lines

This would mean that a
process that had mapped a file MAP_PRIVATE and faulted in parts of it would see different versions of the file in different pages. This should be straightforward to implement, but in what situation is this skewed view of the file useful?

You've got me there ;) However, Posix explicitly blesses this sloppy behaviour. I suppose that with additional user space locking, applications could make it work reliably. But it's still sloppy, and worse, it's different from Linux's local filesystem behaviour.

Quoted 6 lines

3. Modify invalidate_mmap_range() to leave MAP_PRIVATE vmas,
but invalidate those pages in the vma that have not yet been modified (that are not anonymous) as suggested by Stephen. This would mean that a process that had mapped a file MAP_PRIVATE and written on parts of it would see different versions of the file in different pages.

This is the correct behaviour and is the current behaviour for local filesystems. In particular, all processes on all nodes will see the current contents of any file page that they have not yet faulted in, as of the last time any process wrote that file page via mmap or otherwise.

Our goal for GFS, and the goal I'd like to hold up as definitive for any distributed filesystem, is to imitate local filesystem semantics exactly, even across the cluster.

Quoted 1 line

Again, in what situation is this skewed view of the file useful?

It's not skewed in any way that I can see. Though I am no linker expert, I dimly recall that these are precisely the semantics ld relies on.

Quoted 3 lines

5. The current behavior, where the process's writes do not
flow through to the file, but all changes to the file are visible to the writing process.

We all agree that's broken, I hope.

Quoted 5 lines

6. Requiring that MAP_PRIVATE be applied only to unchanging
files, so that (for example) any change to the underlying file removes that file from any MAP_PRIVATE address spaces. Subsequent accesses would get a SEGV, rather than a surprise from silently changing data.

Creative :) Well, data that changes "silently" is a fact of life whenever data is shared. It's up to applications to ensure that shared data changes predictably.

Quoted 2 lines

So, please help me out here... What do applications that MAP_PRIVATE changing files really expect to happen?

Number 3, is that ok with you? Incidently, your list doesn't include the semantics we'd get by just exporting and using invalidate_mmap_range. I presume that is because you agree it's not correct (it will clobber CoWed anonymous pages).

OK, I surrender. I got some private email agreeing with this viewpoint. Any dissenters, speak soon, or...

Quoted 2 lines · Paul E. McKenney

OK, I surrender. I got some private email agreeing with this viewpoint. Any dissenters, speak soon, or...

An implementation is going to look something like the patch below. Unfortunately I don't think there is a way around passing an extra parameter all the way down the unmap call chain. Doubly unfortunately, this doesn't give any benefit at all to anybody who doesn't use a clustered filesystem (which is nearly everybody) while there is a marginal cost. Do you know a better way? Anyway, this is the price of correct MAP_PRIVATE semantics for clustered filesystems. At least I have quantified it so we can decide if it's worth it. (My opinion: correctness is always worth it.)

Regards,

Daniel

include/linux/mm.h +1 -2

 void clear_page_tables(struct mmu_gather *tlb, unsigned long first, int nr);
 int copy_page_range(struct mm_struct *dst, struct mm_struct *src,
 			struct vm_area_struct *vma);

@@ -444,8 +442,7 @@
 			unsigned long size, pgprot_t prot);
 
 extern void invalidate_mmap_range(struct address_space *mapping,
-				  loff_t const holebegin,
-				  loff_t const holelen);
+			  loff_t const holebegin,  loff_t const holelen, int zap);
 extern int vmtruncate(struct inode * inode, loff_t offset);
 extern pmd_t *FASTCALL(__pmd_alloc(struct mm_struct *mm, pgd_t *pgd, unsigned long address));
 extern pte_t *FASTCALL(pte_alloc_kernel(struct mm_struct *mm, pmd_t *pmd, unsigned long address));

mm/memory.c +1 -1

@@ -386,7 +386,7 @@
 
 static void
 zap_pte_range(struct mmu_gather *tlb, pmd_t * pmd,
-		unsigned long address, unsigned long size)
+		unsigned long address, unsigned long size, int zap)
 {
 	unsigned long offset;
 	pte_t *ptep;

mm/memory.c +8 -8

 					if (pte_dirty(pte))
 						set_page_dirty(page);
 					if (page->mapping && pte_young(pte) &&

@@ -436,7 +436,7 @@
 
 static void
 zap_pmd_range(struct mmu_gather *tlb, pgd_t * dir,
-		unsigned long address, unsigned long size)
+		unsigned long address, unsigned long size, int zap)
 {
 	pmd_t * pmd;
 	unsigned long end;

@@ -453,14 +453,14 @@
 	if (end > ((address + PGDIR_SIZE) & PGDIR_MASK))
 		end = ((address + PGDIR_SIZE) & PGDIR_MASK);
 	do {
-		zap_pte_range(tlb, pmd, address, end - address);
-		address = (address + PMD_SIZE) & PMD_MASK; 
+		zap_pte_range(tlb, pmd, address, end - address, zap);
+		address = (address + PMD_SIZE) & PMD_MASK;
 		pmd++;
 	} while (address < end);
 }
 
-void unmap_page_range(struct mmu_gather *tlb, struct vm_area_struct *vma,
-			unsigned long address, unsigned long end)
+static void unmap_page_range(struct mmu_gather *tlb, struct vm_area_struct *vma,
+			unsigned long address, unsigned long end, int zap)
 {
 	pgd_t * dir;
 

@@ -474,7 +474,7 @@
 	dir = pgd_offset(vma->vm_mm, address);
 	tlb_start_vma(tlb, vma);
 	do {
-		zap_pmd_range(tlb, dir, address, end - address);
+		zap_pmd_range(tlb, dir, address, end - address, zap);
 		address = (address + PGDIR_SIZE) & PGDIR_MASK;
 		dir++;
 	} while (address && (address < end));

@@ -524,7 +524,7 @@
  */
 int unmap_vmas(struct mmu_gather **tlbp, struct mm_struct *mm,
 		struct vm_area_struct *vma, unsigned long start_addr,
-		unsigned long end_addr, unsigned long *nr_accounted)
+		unsigned long end_addr, unsigned long *nr_accounted, int zap)
 {
 	unsigned long zap_bytes = ZAP_BLOCK_SIZE;
 	unsigned long tlb_start = 0;	/* For tlb_finish_mmu */

@@ -568,7 +568,7 @@
 				tlb_start_valid = 1;
 			}
 
-			unmap_page_range(*tlbp, vma, start, start + block);
+			unmap_page_range(*tlbp, vma, start, start + block, zap);
 			start += block;
 			zap_bytes -= block;
 			if ((long)zap_bytes > 0)

mm/memory.c +10 -3

+			unsigned long address, unsigned long size, int zap)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	struct mmu_gather *tlb;

@@ -612,11 +612,17 @@
 	lru_add_drain();
 	spin_lock(&mm->page_table_lock);
 	tlb = tlb_gather_mmu(mm, 0);
-	unmap_vmas(&tlb, mm, vma, address, end, &nr_accounted);
+	unmap_vmas(&tlb, mm, vma, address, end, &nr_accounted, zap);
 	tlb_finish_mmu(tlb, address, end);
 	spin_unlock(&mm->page_table_lock);
 }
 
+void zap_page_range(struct vm_area_struct *vma,
+			unsigned long address, unsigned long size)
+{
+	invalidate_page_range(vma, address, size, 1);
+}
+
 /*
  * Do a quick page-table lookup for a single page.
  * mm->page_table_lock must be held.

@@ -1095,9 +1101,9 @@
 		    	continue;	/* Mapping disjoint from hole. */
 		zba = (hba <= vba) ? vba : hba;
 		zea = (vea <= hea) ? vea : hea;
-		zap_page_range(vp,
+		invalidate_page_range(vp,
 			       ((zba - vba) << PAGE_SHIFT) + vp->vm_start,
-			       (zea - zba + 1) << PAGE_SHIFT);
+			       (zea - zba + 1) << PAGE_SHIFT, 1);
 	}
 }
 

mm/memory.c +1 -1

@@ -1156,7 +1162,7 @@
 	if (inode->i_size < offset)
 		goto do_expand;
 	i_size_write(inode, offset);
-	invalidate_mmap_range(mapping, offset + PAGE_SIZE - 1, 0);
+	invalidate_mmap_range(mapping, offset + PAGE_SIZE - 1, 0, 1);
 	truncate_inode_pages(mapping, offset);
 	goto out_truncate;
 

mm/mmap.c +1 -1

@@ -1134,7 +1134,7 @@
 
 	lru_add_drain();
 	tlb = tlb_gather_mmu(mm, 0);
-	unmap_vmas(&tlb, mm, vma, start, end, &nr_accounted);
+	unmap_vmas(&tlb, mm, vma, start, end, &nr_accounted, 1);
 	vm_unacct_memory(nr_accounted);
 
 	if (is_hugepage_only_range(start, end - start))

I cannot think of any reasonable alternative to passing the parameter down either, as it certainly does not be reasonable to duplicate the code...

How about something like "private_too" instead of "zap"?

mm/memory.c line 597

Quoted 3 lines

mm/memory.c +1 -2

-void zap_page_range(struct vm_area_struct *vma,
-			unsigned long address, unsigned long size)
+void invalidate_page_range(struct vm_area_struct *vma,

Would it be useful for this to be inline? (Wouldn't seem so, zapping mappings has enough overhead that an extra level of function call should be deep down in the noise...)

Doesn't the new argument need to be passed down through invalidate_mmap_range_list()?

Hi Paul,

Quoted 3 lines

I cannot think of any reasonable alternative to passing the parameter down either, as it certainly does not be reasonable to duplicate the code...

Yes, it's simply the (small) price that has to be paid in order to be able to boast about our accurate semantics.

Quoted 1 line

How about something like "private_too" instead of "zap"?

How about just "all", which is what we mean.

Quoted 7 lines

mm/memory.c line 597

Quoted 3 lines

mm/memory.c +1 -2

-void zap_page_range(struct vm_area_struct *vma,
-			unsigned long address, unsigned long size)
+void invalidate_page_range(struct vm_area_struct *vma,

Would it be useful for this to be inline? (Wouldn't seem so, zapping mappings has enough overhead that an extra level of function call should be deep down in the noise...)

Yes, it doesn't seem worth it just to save a stack frame.

Actually, I erred there in that invalidate_mmap_range should not export the flag, because it never makes sense to pass in non-zero from a DFS.

Quoted 2 lines

Doesn't the new argument need to be passed down through invalidate_mmap_range_list()?

It does, thanks for the catch. Please bear with me for a moment while I reroll this, then hopefully we can move on to the more interesting discussion of whether it's worth it. (Yes it is :)

Quoted 2 lines · Daniel Phillips

Actually, I erred there in that invalidate_mmap_range should not export the flag, because it never makes sense to pass in non-zero from a DFS.

Doesn't vmtruncate() want to pass non-zero "all" in to invalidate_mmap_range() in order to maintain compatibility with existing Linux semantics?

Quoted 7 lines · Paul E. McKenney

Quoted 2 lines · Daniel Phillips

Actually, I erred there in that invalidate_mmap_range should not export the flag, because it never makes sense to pass in non-zero from a DFS.

Doesn't vmtruncate() want to pass non-zero "all" in to invalidate_mmap_range() in order to maintain compatibility with existing Linux semantics?

That comes from inside. The DFS's truncate interface should just be vmtruncate. If I missed something, please shout.

Your earlier patch has a call to invalidate_mmap_range() within vmtruncate(), which passes "1" to the last arg, so as to get rid of all mappings to the truncated portion of the file. So either invalidate_mmap_range() needs to keep the fourth arg or needs to be a wrapper for an underlying function that vmtruncate() can call, or some such.

The latter may be what you intended to do.

Hi Paul et al,

Here is an updated patch. The name of the exported function is changed to "invalidate_filemap_range" to reflect the fact that only file-backed pages are invalidated, and to distinguish the three parameter flavour from the four parameter version called from vmtruncate. The inner loop in zap_pte_range is hopefully correct now.

While I'm in here, why is the assignment "pte =" at line 411 of memory.c not redundant?

   http://lxr.linux.no/source/mm/memory.c?v=2.6.1#L411

As far as I can see, the ->filemap spinlock protects the pte from modification and pte was already assigned at line 405.

Anyway, we can now see that the full cost of this DFS-specific feature in the inner loop is a single (unlikely) branch.

I'll repeat my proposition here: providing local filesystem semantics for MAP_PRIVATE on any distributed filesystem requires these decorations on the unmap path. Though there is no benefit for local filesystems, the cost is insignificant.

Regards,

Daniel

include/linux/mm.h +9 -9

@@ -430,23 +430,23 @@
 void shmem_lock(struct file * file, int lock);
 int shmem_zero_setup(struct vm_area_struct *);
 
-void zap_page_range(struct vm_area_struct *vma, unsigned long address,
-			unsigned long size);
 int unmap_vmas(struct mmu_gather **tlbp, struct mm_struct *mm,
 		struct vm_area_struct *start_vma, unsigned long start_addr,
-		unsigned long end_addr, unsigned long *nr_accounted);
-void unmap_page_range(struct mmu_gather *tlb, struct vm_area_struct *vma,
-			unsigned long address, unsigned long size);
+		unsigned long end_addr, unsigned long *nr_accounted, int zap);
 void clear_page_tables(struct mmu_gather *tlb, unsigned long first, int nr);
 int copy_page_range(struct mm_struct *dst, struct mm_struct *src,
 			struct vm_area_struct *vma);
 int zeromap_page_range(struct vm_area_struct *vma, unsigned long from,
 			unsigned long size, pgprot_t prot);
-
-extern void invalidate_mmap_range(struct address_space *mapping,
-				  loff_t const holebegin,
-				  loff_t const holelen);
+extern void invalidate_filemap_range(struct address_space *mapping, loff_t const start, loff_t const length);
 extern int vmtruncate(struct inode * inode, loff_t offset);
+void invalidate_page_range(struct vm_area_struct *vma, unsigned long address, unsigned long size, int all);
+
+static inline void zap_page_range(struct vm_area_struct *vma, ulong address, ulong size)
+{
+	invalidate_page_range(vma, address, size, 1);
+}
+
 extern pmd_t *FASTCALL(__pmd_alloc(struct mm_struct *mm, pgd_t *pgd, unsigned long address));
 extern pte_t *FASTCALL(pte_alloc_kernel(struct mm_struct *mm, pmd_t *pmd, unsigned long address));
 extern pte_t *FASTCALL(pte_alloc_map(struct mm_struct *mm, pmd_t *pmd, unsigned long address));

mm/memory.c +39 -30

@@ -384,9 +384,13 @@
 	return -ENOMEM;
 }
 
-static void
-zap_pte_range(struct mmu_gather *tlb, pmd_t * pmd,
-		unsigned long address, unsigned long size)
+static inline int is_anon(struct page *page)
+{
+	return !page->mapping || PageSwapCache(page);
+}
+
+static void zap_pte_range(struct mmu_gather *tlb, pmd_t * pmd,
+		unsigned long address, unsigned long size, int all)
 {
 	unsigned long offset;
 	pte_t *ptep;

@@ -409,7 +413,8 @@
 			continue;
 		if (pte_present(pte)) {
 			unsigned long pfn = pte_pfn(pte);
-
+			if (unlikely(!all) && is_anon(pfn_to_page(pfn)))
+				continue;
 			pte = ptep_get_and_clear(ptep);
 			tlb_remove_tlb_entry(tlb, ptep, address+offset);
 			if (pfn_valid(pfn)) {

@@ -426,7 +431,7 @@
 				}
 			}
 		} else {
-			if (!pte_file(pte))
+			if (!pte_file(pte) && all)
 				free_swap_and_cache(pte_to_swp_entry(pte));
 			pte_clear(ptep);
 		}

@@ -434,9 +439,8 @@
 	pte_unmap(ptep-1);
 }
 
-static void
-zap_pmd_range(struct mmu_gather *tlb, pgd_t * dir,
-		unsigned long address, unsigned long size)
+static void zap_pmd_range(struct mmu_gather *tlb, pgd_t * dir,
+		unsigned long address, unsigned long size, int all)
 {
 	pmd_t * pmd;
 	unsigned long end;

@@ -453,14 +457,14 @@
 	if (end > ((address + PGDIR_SIZE) & PGDIR_MASK))
 		end = ((address + PGDIR_SIZE) & PGDIR_MASK);
 	do {
-		zap_pte_range(tlb, pmd, address, end - address);
-		address = (address + PMD_SIZE) & PMD_MASK; 
+		zap_pte_range(tlb, pmd, address, end - address, all);
+		address = (address + PMD_SIZE) & PMD_MASK;
 		pmd++;
 	} while (address < end);
 }
 
-void unmap_page_range(struct mmu_gather *tlb, struct vm_area_struct *vma,
-			unsigned long address, unsigned long end)
+static void unmap_page_range(struct mmu_gather *tlb, struct vm_area_struct *vma,
+		unsigned long address, unsigned long end, int all)
 {
 	pgd_t * dir;
 

@@ -474,7 +478,7 @@
 	dir = pgd_offset(vma->vm_mm, address);
 	tlb_start_vma(tlb, vma);
 	do {
-		zap_pmd_range(tlb, dir, address, end - address);
+		zap_pmd_range(tlb, dir, address, end - address, all);
 		address = (address + PGDIR_SIZE) & PGDIR_MASK;
 		dir++;
 	} while (address && (address < end));

@@ -524,7 +528,7 @@
  */
 int unmap_vmas(struct mmu_gather **tlbp, struct mm_struct *mm,
 		struct vm_area_struct *vma, unsigned long start_addr,
-		unsigned long end_addr, unsigned long *nr_accounted)
+		unsigned long end_addr, unsigned long *nr_accounted, int all)
 {
 	unsigned long zap_bytes = ZAP_BLOCK_SIZE;
 	unsigned long tlb_start = 0;	/* For tlb_finish_mmu */

@@ -568,7 +572,7 @@
 				tlb_start_valid = 1;
 			}
 
-			unmap_page_range(*tlbp, vma, start, start + block);
+			unmap_page_range(*tlbp, vma, start, start + block, all);
 			start += block;
 			zap_bytes -= block;
 			if ((long)zap_bytes > 0)

@@ -594,8 +598,8 @@
  * @address: starting address of pages to zap
  * @size: number of bytes to zap
  */
-void zap_page_range(struct vm_area_struct *vma,
-			unsigned long address, unsigned long size)
+void invalidate_page_range(struct vm_area_struct *vma,
+		unsigned long address, unsigned long size, int all)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	struct mmu_gather *tlb;

@@ -612,7 +616,7 @@
 	lru_add_drain();
 	spin_lock(&mm->page_table_lock);
 	tlb = tlb_gather_mmu(mm, 0);
-	unmap_vmas(&tlb, mm, vma, address, end, &nr_accounted);
+	unmap_vmas(&tlb, mm, vma, address, end, &nr_accounted, all);
 	tlb_finish_mmu(tlb, address, end);
 	spin_unlock(&mm->page_table_lock);
 }

@@ -1071,10 +1075,8 @@
  * Both hba and hlen are page numbers in PAGE_SIZE units.
  * An hlen of zero blows away the entire portion file after hba.
  */
-static void
-invalidate_mmap_range_list(struct list_head *head,
-			   unsigned long const hba,
-			   unsigned long const hlen)
+static void invalidate_mmap_range_list(struct list_head *head,
+		 unsigned long const hba,  unsigned long const hlen, int all)
 {
 	struct list_head *curr;
 	unsigned long hea;	/* last page of hole. */

@@ -1095,9 +1097,9 @@
 		    	continue;	/* Mapping disjoint from hole. */
 		zba = (hba <= vba) ? vba : hba;
 		zea = (vea <= hea) ? vea : hea;
-		zap_page_range(vp,
+		invalidate_page_range(vp,
 			       ((zba - vba) << PAGE_SHIFT) + vp->vm_start,
-			       (zea - zba + 1) << PAGE_SHIFT);
+			       (zea - zba + 1) << PAGE_SHIFT, all);
 	}
 }
 

@@ -1115,8 +1117,8 @@
  * up to a PAGE_SIZE boundary.  A holelen of zero truncates to the
  * end of the file.
  */
-void invalidate_mmap_range(struct address_space *mapping,
-		      loff_t const holebegin, loff_t const holelen)
+static void invalidate_mmap_range(struct address_space *mapping,
+		loff_t const holebegin, loff_t const holelen, int all)
 {
 	unsigned long hba = holebegin >> PAGE_SHIFT;
 	unsigned long hlen = (holelen + PAGE_SIZE - 1) >> PAGE_SHIFT;

@@ -1133,12 +1135,19 @@
 	/* Protect against page fault */
 	atomic_inc(&mapping->truncate_count);
 	if (unlikely(!list_empty(&mapping->i_mmap)))
-		invalidate_mmap_range_list(&mapping->i_mmap, hba, hlen);
+		invalidate_mmap_range_list(&mapping->i_mmap, hba, hlen, all);
 	if (unlikely(!list_empty(&mapping->i_mmap_shared)))
-		invalidate_mmap_range_list(&mapping->i_mmap_shared, hba, hlen);
+		invalidate_mmap_range_list(&mapping->i_mmap_shared, hba, hlen, all);
 	up(&mapping->i_shared_sem);
 }
-EXPORT_SYMBOL_GPL(invalidate_mmap_range);
+
+ void invalidate_filemap_range(struct address_space *mapping,
+		loff_t const start, loff_t const length)
+{
+	invalidate_mmap_range(mapping, start, length, 0);
+}
+
+EXPORT_SYMBOL_GPL(invalidate_filemap_range);
 
 /*
  * Handle all mappings that got truncated by a "truncate()"

@@ -1156,7 +1165,7 @@
 	if (inode->i_size < offset)
 		goto do_expand;
 	i_size_write(inode, offset);
-	invalidate_mmap_range(mapping, offset + PAGE_SIZE - 1, 0);
+	invalidate_mmap_range(mapping, offset + PAGE_SIZE - 1, 0, 1);
 	truncate_inode_pages(mapping, offset);
 	goto out_truncate;
 

mm/mmap.c +1 -1

@@ -1134,7 +1134,7 @@
 
 	lru_add_drain();
 	tlb = tlb_gather_mmu(mm, 0);
-	unmap_vmas(&tlb, mm, vma, start, end, &nr_accounted);
+	unmap_vmas(&tlb, mm, vma, start, end, &nr_accounted, 1);
 	vm_unacct_memory(nr_accounted);
 
 	if (is_hugepage_only_range(start, end - start))

This is the function formerly known as invalidate_mmap_range, with the addition of a new code path in the zap_ call chain to handle MAP_PRIVATE properly. This function by itself is enough to support a crude but useful form of distributed mmap where a shared file is cached only on one cluster node at a time.

To use this, the distributed filesystem has to hook do_no_page to intercept page faults and carry out the needed global locking. The locking itself does not require any new kernel hooks. In brief, the patch here and another patch to be presented for the do_no_page hook, together provide the core kernel API for a simplified, distributed mmap. (Note that there may be a workaround for the lack of a do_no_page hook, but certainly not as simple and robust.)

To put this in perspective, I'll mention the two big limitations of the simplified API:

1) Invalidation is always a whole file at a time

I'd like to take this opportunity to apologize to Paul for derailing his more modest proposal, but unfortunately, the semantics that could be obtained that way are fatally flawed: private mmaps just won't work. What I've written here is about the minimum that supports acceptable mmap semantics.

mm/memory.c line 420

Quoted 6 lines · Daniel Phillips

mm/memory.c +3 -1

-			pte = ptep_get_and_clear(ptep);
+			if (unlikely(!all) && is_anon(pfn_to_page(pfn)))
+				continue;
+			pte = ptep_get_and_clear(ptep); /* get dirty bit atomically */
 			tlb_remove_tlb_entry(tlb, ptep, address+offset);
 			if (pfn_valid(pfn)) {

I think you need to check pfn_valid() before running is_anon(pfn_to_page())

Quoted 9 lines · Andrew Morton

mm/memory.c line 420

Quoted 6 lines · Daniel Phillips

mm/memory.c +3 -1

-			pte = ptep_get_and_clear(ptep);
+			if (unlikely(!all) && is_anon(pfn_to_page(pfn)))
+				continue;
+			pte = ptep_get_and_clear(ptep); /* get dirty bit atomically */
 			tlb_remove_tlb_entry(tlb, ptep, address+offset);
 			if (pfn_valid(pfn)) {

I think you need to check pfn_valid() before running is_anon(pfn_to_page())

Easy enough:

	if (unlikely(!all) && pfn_valid(pfn) && is_anon(pfn_to_page(pfn)))

but how can we legitimately get !pfn_valid there?

Quoted 1 line

but how can we legitimately get !pfn_valid there?

A mapping of some I/O region?

I think you need to check pfn_valid() before running is_anon(pfn_to_page())

Here is a rearranged zap_pte_range that avoids any operations for out-of-range pfns.

Quoted 2 lines · Daniel Phillips

Here is a rearranged zap_pte_range that avoids any operations for out-of-range pfns.

Please remind us why Linux needs this patch?

I forget what `all' does? anon+swapcache as well as pagecache?

A bit of API documentation here would be appropriate.

Quoted 5 lines · Andrew Morton

Quoted 2 lines · Daniel Phillips

Here is a rearranged zap_pte_range that avoids any operations for out-of-range pfns.

Please remind us why Linux needs this patch?

The is purely to support mmap, including MAP_PRIVATE, accurately on distributed filesystems, where "accurately" is defined as "with local filesystem semantics".

If the same file region is mmapped by more than one node, only one of them is allowed to have a given page of the mmap valid in the page tables at any time. When a memory write occurs on one of the other nodes, it must fault so that the distributed filesystem can arrange for exclusive ownership of the file page (or as GFS currently implements it, the whole file) to change from one node to the other. At this time, any pages already faulted in must be unmapped so that future memory accesses will properly fault. This unmapping is done by zap_page_range, which has nearly the semantics we want except that it will also unmap private pages of a MAP_PRIVATE mapping, destroying the only copy of that data. A user would observe the privately written data spontaneously revert to the current file contents. The purpose of this patch is to fix that.

This patch allows a distributed filesystem to unmap file-backed memory without unmapping anonymous pages or deleting swap cache, avoiding the above data destruction. Since zap_page_range is the only function that knows how to unmap memory, it needs to be taught how to skip anonymous pages.

An alternative to this patch is simply to export zap_page_range, then the distributed filesystem can walk the lists of mmapped vmas itself, skipping any that are MAP_PRIVATE. This achieves Posix local filesystem semantics, but not Linux local filesystem semantics, because updates to the mmap from other nodes become visible unpredictably. Earlier this year, Linus said that he wants tighter semantics for distributed MAP_PRIVATE.

This patch presses zap_page_range into service in a way that was not originally intended, that is, for invalidation as opposed to destruction of memory regions. The requirements are identical except for the MAP_PRIVATE detail. Forking the whole zap_ chain would be even more distasteful than grafting on this option flag. It's also impractical to implement a zap_ variant within a dfs module because of the heavy use of per-arch APIs. As far I can see, this patch is the minimum cost of having accurate semantics for distributed MAP_PRIVATE mmap.

I'll take the opportunity to beat my chest a once again about the fact that this doesn't benefit anything other than distributed filesystems. On the other hand, the cost is miniscule: 54 bytes, a little stack and likely no measureable cpu.

Quoted 1 line

I forget what `all' does? anon+swapcache as well as pagecache?

Yes

Quoted 1 line

A bit of API documentation here would be appropriate.

Oops, sorry:

/**
* zap_page_range - remove user pages in a given range
* @vma: vm_area_struct holding the applicable pages
* @address: starting address of pages to zap
* @size: number of bytes to zap
* @all: also unmap anonymous pages
*/
void zap_page_range(struct vm_area_struct *vma,
unsigned long address, unsigned long size, int all)

Regards,

Daniel

It does, thanks for the catch. Please bear with me for a moment while I reroll this, then hopefully we can move on to the more interesting discussion of whether it's worth it. (Yes it is :)

Quoted 3 lines · Daniel Phillips

It does, thanks for the catch. Please bear with me for a moment while I reroll this, then hopefully we can move on to the more interesting discussion of whether it's worth it. (Yes it is :)

What about to the more interesting question who needs it. It think this whole discussion who needs what and which approach is better is pretty much moot as long as we don't have an intree users.

Instead of wasting your time on different designs you should hurry of getting your filesystems encumbrance-reviewed, cleaned up and merged - with intree users we have a chance of finding the right API. And your newly started dicussion shows pretty much that with only out of tree users we'll never get a sane API.

Again, we (everybody who cared to jump in) now agree on what is sane here, it's quite logical. As for supplying background material so this makes sense to a wider group of people, sorry it's been on my to-do list for a while. Getting a DFS, namely Sistina GFS, into the tree is underway as you know from the press release, however turning the ship takes time. Meanwhile, the api discussion can't wait because the rudder on that ship is even smaller.

Surely, such a GFS needs a cluster infrastructure - membership, messaging, DLM - in the kernel.

Can you or anyone else from Sistina/RHAT clarify on the details of this?

I presume you meant "DFS". I can't comment on the details of the plan for GFS just now, however consider OpenGFS: yes, it needs and has a cluster infrastructure. The kernel does not dictate anything about that infrastructure. Each DFS is free to implement its own infrastructure, possibly involving kernel extensions.

Quoted 2 lines · On 2004-02-20T22:16:09, I said:

Each DFS is free to implement its own infrastructure, possibly involving kernel extensions.

Yes. Though I do reserve the right to find this highly silly, that we might end up with multiple hooks for clustering infrastructure in the kernel...

So, how does OpenGFS/GFS achieve the communication? How does it interact with the infrastructure (which, I infere from your above comments, is meant to reside in user-space)?

But the one true clustering infrastructure hasn't been developed yet.

from being able to define that sensibly now. It's better to implement exactly what a given DFS needs for the time being.

Quoted 3 lines

So, how does OpenGFS/GFS achieve the communication? How does it interact with the infrastructure (which, I infere from your above comments, is meant to reside in user-space)?

It's done both ways, actually. No new kernel hooks are used in either case.

Quoted 6 lines

Quoted 3 lines

So, how does OpenGFS/GFS achieve the communication? How does it interact with the infrastructure (which, I infere from your above comments, is meant to reside in user-space)?

It's done both ways, actually. No new kernel hooks are used in either case.

That doesn't answer my question how you are doing the user-space / kernel communication ;-)

And will the user-space infrastructure to go with GFS be open source too? Questions over questions.

These URLs do require that you register, but there is no cost nor any agreement other than the GPL itself. The Linux client has not been shipped as product yet. The code is still quite rough, which is one reason that it has not be submitted to, for example, LKML. ;-)

Harsh, when you consider the biggest protagonist in the Linux market place is suing their arse off speficically for contributing stuff to Linux. Lawyer overload is bound to be involved in the delays.

Hello, Andrew,

The attached patch to make invalidate_mmap_range() non-GPL exported seems to have been lost somewhere between 2.6.1-mm4 and 2.6.1-mm5. It still applies cleanly. Could you please take it up again?

Thanx, Paul

------------------------------------------------------------------------

It was EXPORT_SYMBOL_GPL(), however IBM's GPFS is not GPL.