VM subsystem bug in 2.4.0 ?

Hi all!

I have a problem with 2.4.0

I'm testing Informix IIF-2000 database server running on dual Intel Pentium II - 233. When I run 'make -j30 bzImage' in the kernel source, my
Linux box hangs without any messages. This occurs when Informix is running. When I stoped Informix and tryed to do the same, all passed ok!

I think this is bug in kernel ( VM subsystem ) code.

Informix allocate about to 50% of memory as LOCKED shared memory segments.

I'm thinking the reason in this. Kernel wants, but can't to swap out locked shm's segments.

Thank you.

Sergey E. Volkov
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

You are right. I have seen this bug before with the kernel moving unswappable pages from the active list to the inactive_dirty list and back.

We need a check in deactivate_page() to prevent the kernel from moving pages from locked shared memory segments to the inactive_dirty list.

Christoph? Linus?

regards,

Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...

		http://www.surriel.com/

http://www.conectiva.com/ http://distro.conectiva.com.br/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

The only solution I see is something like a "active_immobile" list, and add entries to that list whenever "writepage()" returns 1 - instead of just moving them to the active list.

Seems to be a simple enough change. The main worry would be getting the pages _off_ such a list: anything that unlocks a shared memory segment (can you even do that? If the only way to unlock is to remove, we have no problems) would have to have a special function to move all pages from the immobile list back to the active list (and then they'd get moved back if they were for another segment that is still locked).

Linus

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Just marking them with a special "do not deactivate me" bit seems to work fine enough. When this special bit is set, we simply move the page to the back of the active list instead of deactivating.

And when the bit changes again, the page can be evicted from memory just fine. In the mean time, the locked pages will also have undergone normal page aging and at unlock time we know whether to swap out the page or not.

I agree that this scheme has a higher overhead than your idea, but it also seems to be a bit more flexible and simple. Alternatively, we could just scan the wired_list once a minute and move the unwired pages to the active list.

regards,

Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...

		http://www.surriel.com/

http://www.conectiva.com/ http://distro.conectiva.com.br/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Hi Rik,

At IPC_UNLOCK there is no reference to the pages locked by this segment available. We could perhaps move the whole locked list to the active list if we unlock any segment.

Second point: How do we handle out of swap? I do not think that we should lock these pages but keep them in the active list.

Greetings
Christoph

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Hi,

But again, how do you clear the bit? Locking is a per-vma property, not per-page. I can mmap a file twice and mlock just one of the mappings. If you get a munlock(), how are you to know how many other locked mappings still exist?

--Stephen
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Note that this would be solved very cleanly if the SHM code would use the "VM_LOCKED" flag, and actually lock the pages in the VM, instead of trying to lock them down for writepage().

That would mean that such a segment would still get swapped out when it is not mapped anywhere, but I wonder if that semantic difference really matters.

If the vma is marked VM_LOCKED, the VM subsystem will do the right thing (the page will never get removed from the page tables, so it won't ever make it into that back-and-forth bounce between the active and the inactive lists).

Linus

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

here comes the patch. (lightly tested)

Greetings
Christoph

diff -uNr 2.4.0/include/linux/shmem_fs.h c/include/linux/shmem_fs.h
@@ -22,7 +22,6 @@
 	swp_entry_t	i_direct[SHMEM_NR_DIRECT]; /* for the first blocks */
 	swp_entry_t   **i_indirect; /* doubly indirect blocks */
 	unsigned long	swapped;
-	int		locked;     /* into memory */
 	struct list_head	list;
 };
 

diff -uNr 2.4.0/mm/shmem.c c/mm/shmem.c
@@ -201,8 +201,6 @@
 	swp_entry_t *entry, swap;
 
 	info = &page->mapping->host->u.shmem_i;
-	if (info->locked)
-		return 1;
 	swap = __get_swap_page(2);
 	if (!swap.val)
 		return 1;

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

I'd really like an opinion on whether this is truly legal or not? After all, it does change the behaviour to mean "pages are locked only if they have been mapped into virtual memory". Which is not what it used to mean.

Arguably the new semantics are perfectly valid semantics on their own, but I'm not sure they are acceptable.

In contrast, the PG_realdirty approach would give the old behaviour of truly locked-down shm segments, with not significantly different complexity behaviour.

What do other UNIXes do for shm_lock()?

The Linux man-page explicitly states for SHM_LOCK that

The user must fault in any pages that are required to be present after locking is enabled.

which kind of implies to me that the VM_LOCKED implementation is ok. HOWEVER, looking at the HP-UX man-page, for example, certainly implies that the PG_realdirty approach is the correct one. The IRIX man-pages in contrast say

Locking occurs per address space;
multiple processes or sprocs mapping the area at different addresses each need to issue the lock (this is primarily an issue with the per-process page tables).

which again implies that they've done something akin to a VM_LOCKED implementation.

Does anybody have any better pointers, ideas, or opinions?

Linus

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Hi Linus,

I just checked SuS and they do not list SHM_LOCK as command at all.

Yes.

Yes.

So Irix does something quite different. For Irix SHM_LOCK is a special version of mlock...

I think the VM_LOCKED approach is the best:

- SuS does not specify anything, the different vendors do different things. So people using SHM_LOCK have to be aware that the details differ.
- Technically this is the fastest approach for attached segments: We do not scan the relevent vmas at all and by doing so we keep the overhead lowest. And I do not see a reason to use SHM_LOCK besides performance.

BTW I also have a patch appended which bumps the page count. Works also, is also small, but we will have a higher soft fault rate with that.

Greetings
Christoph

diff -uNr 2.4.0/ipc/shm.c c/ipc/shm.c
@@ -121,6 +121,7 @@
 {
 	shm_tot -= (shp->shm_segsz + PAGE_SIZE - 1) >> PAGE_SHIFT;
 	shm_rmid (shp->id);
+	shmem_lock(shp->shm_file, 0);
 	fput (shp->shm_file);
 	kfree (shp);
 }

@@ -467,10 +468,10 @@
 		if(err)
 			goto out_unlock;
 		if(cmd==SHM_LOCK) {
-			shp->shm_file->f_dentry->d_inode->u.shmem_i.locked = 1;
+			shmem_lock(shp->shm_file, 1);
 			shp->shm_flags |= SHM_LOCKED;
 		} else {
-			shp->shm_file->f_dentry->d_inode->u.shmem_i.locked = 0;
+			shmem_lock(shp->shm_file, 0);
 			shp->shm_flags &= ~SHM_LOCKED;
 		}
 		shm_unlock(shmid);

diff -uNr 2.4.0/mm/shmem.c c/mm/shmem.c
@@ -310,6 +310,8 @@
 	}
 	/* We have the page */
 	SetPageUptodate (page);
+	if (info->locked)
+		page_cache_get(page);
 
 cached_page:
 	UnlockPage (page);

@@ -399,6 +401,32 @@
 	spin_unlock (&sb->u.shmem_sb.stat_lock);
 	buf->f_namelen = 255;
 	return 0;
+}
+
+void shmem_lock(struct file * file, int lock)
+{
+	struct inode * inode = file->f_dentry->d_inode;
+	struct shmem_inode_info * info = &inode->u.shmem_i;
+	struct page * page;
+	unsigned long idx, size;
+
+	if (info->locked == lock)
+		return;
+	down(&inode->i_sem);
+	info->locked = lock;
+	size = (inode->i_size + PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT;
+	for (idx = 0; idx < size; idx++) {
+		page = find_lock_page(inode->i_mapping, idx);
+		if (!page)
+			continue;
+		if (!lock) {
+			/* release the extra count and our reference */
+			page_cache_release(page);
+			page_cache_release(page);
+		}
+		UnlockPage(page);
+	}
+	up(&inode->i_sem);
 }
 
 /*

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Hi Linus,

It appears that the fine-detail semantics vary across the board. DYNIX/ptx supports two forms of SysV shm locking - soft and hard. Soft-locking (the default) merely makes the pages sticky, so if you fault them in, they stay in your resident set, but don't count against it. If, however the process swaps, they're all evicted, and when the process is swapped back in, you get to fault the back in all over again. Hard locking pins the segment into physical memory until such time as it's destroyed. It stays there even if there are currently no attaches. Again, such pages are not counted against the process RSS.

SVR4 only support one form. It faults all the pages in and locks them into memory, but doesn't treat the especially wrt rss/paging, which seems none too clever - if they're locked into memory, you might as well use them :-)

[Details of the differing approches omitted]

I don't know if there are any arguments in favour of making both approaches available. Gut feel says that's overkill. We ended up with two by historical accident. The soft-locking was always there (althought semantically different to SVR4), and the hard-locking stuff was added to boost performance with a certain six-letter RDBMS that attaches an SGA to each process. They all get to attach it "for free", and since it doesn't count towards the RSS, it allowed tuning a fairly small RSS across the system without having the RDMBS processes spent all their time (soft) faulting SGA pages in and out of their RSS.

Tim

Tim Wright - [email protected] or [email protected] or [email protected]
IBM Linux Technology Center, Beaverton, Oregon
"Nobody ever said I was charming, they said "Rimmer, you're a git!"" RD VI
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Hi Stephen,

It's worse: The issue we are talking about is SYSV IPC_LOCK. This is a per segment thing. A user can (un)lock a segment at any time. But we do not have the references to the vmas attached to the segemnts or to the pages allocated.

Greetings
Christoph

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Hi,

The issue is locked VA pages. SysV is just one of the ways in which it can happen: the solution has got to address both that and mlock()/mlockall().

Why not? Won't the address space mmap* lists give you this?

--Stephen
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

No, mlock() and mlockall() already works. Exactly because mlocked pages will never be removed from the VM, the VM layer knows how to deal with them (or "not deal with them" as the case is more properly stated). They won't ever get on the inactive list, and because refill_inactive_scan() won't be able to move handle them (the count is elevated by the VM mappings), the VM will correctly and gracefully fall back on scanning the page tables to find some _other_ blocks.

So mlock works fine.

The reason shm locked segments do _not_ work fine is exactly because they are not locked down in the VM, and for that reason they can end up being detached from everything and thus moved to the inactive list. That counts as "progress" as far as the VM is concerned, so we get into this endless loop where we move the page to the inactive list, then try to write it out, fail, and move it back to the active list again. The VM _thinks_ it is making progress, but obviously isn't.

End result: lockup.

Marking the pages with a magic flag would solve it. Then we could just make refill_inactive_scan() ignore such pages. Something like "PG_Reallydirty".

Linus

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Hi Stephen,

AFAIU mlock'ed pages would never get deactivated since the ptes do not get dropped.

OK. We could go from shmid_kernel->file->dentry->inode->mapping We had to scan all mappings for pages in the page tables and in the page cache. Doesn't look really nice :-(

Greetings
Christoph

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Hi,

D'oh, right --- so can't you lock a segment just by bumping page_count on its pages?

--Stephen
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Hi Stephen,

Looks like a good idea.

Oh, and my last posting was partly bogus: I can directly get the pages with page cache lookups on the file.

Greetings
Christoph

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Even more appropriately, you have the inode->i_mapping lists that you can use directly (no need to do lookups, just walk the list).

Note that bumping the counts is _NOT_ as easy as you seem to think. The
problem: vmtruncate() and friends. It's much easier to just have a flag that gets cleared on truncate.

Linus

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Putting this together with an idea from Linus:

Call it 'pinned'... the pinned list would have pages with use count = 2 or more. A page gets off the pinned list when its use count goes to 1 in put_page.

--
Daniel
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

I don't even want to start thinking about how this would screw up the (already fragile) page aging balance...

Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...

		http://www.surriel.com/

http://www.conectiva.com/ http://distro.conectiva.com.br/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Hi Rik,

As of 2.4.1-pre we pin the pages by increasing the page count for locked segments. No special list needed.

Greetings
Christoph

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

Sure no special list is needed. But without a special list to park those pages on they will just circulate on the active/inactive lists, wasting CPU cycles and trashing cache. A special list would be an improvement, but is no burning issue.

--
Daniel
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/