2.6.30-rc3-mmotm-090428-1814 -- bogus pointer deref

I'm seeing this on an ia64 platform--HP rx8640--running the numactl package regression test. On ia64 a "NaT Consumption" [NaT = "not a thing"] usually means a bogus pointer. I verified that it also occurs on 2.6.30-rc3-mmotm-090424-1814. The regression test runs to completion on a 4-node x86_64 platform for both the 04/27 and 04/28 mmotm kernels.

The bug occurs right after the test suite issues the message:

"testing numactl --interleave=all memhog 15728640"

-------------------------------
Console log:

numactl[7821]: NaT consumption 2216203124768 [2]

Modules linked in: ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc vfat fat dm_mirror dm_multipath scsi_dh pci_slot parport_pc lp parport sg sr_mod cdrom button e1000 tg3 libphy dm_region_hash dm_log dm_mod sym53c8xx mptspi mptscsih mptbase scsi_transport_spi sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table]

Pid: 7821, CPU 25, comm: numactl
psr : 0000121008022038 ifs : 8000000000000004 ip : [<a00000010014ec91>] Not tainted (2.6.30-rc3-mmotm-090428-1631)
ip is at next_zones_zonelist+0x31/0x120
unat: 0000000000000000 pfs : 000000000000038b rsc : 0000000000000003
rnat: 0000001008022038 bsps: e0000702c094e000 pr : 0000000000659a65
ldrs: 0000000000000000 ccv : 0000000000000000 fpsr: 0009804c0270033f csd : 0000000000000000 ssd : 0000000000000000
b0 : a00000010018dd30 b6 : a0000001000414c0 b7 : a000000100474660 f6 : 1003e3d64d824615cef9c f7 : 1003e9e3779b97f4a7c16 f8 : 1003e0a00000010071e2a f9 : 1003e0000000000000001 f10 : 1003e0000000000000003 f11 : 1003e8208208208208209 r1 : a000000100c647c0 r2 : 0000000000001f90 r3 : 0000000000000000 r8 : 0000000000001f88 r9 : 0000000000000000 r10 : 0000000000000000 r11 : 0000000000000000 r12 : e0000803859eaad0 r13 : e0000803859e8000 r14 : 0000000000000000 r15 : ffffffffffffff80 r16 : e000078000113c38 r17 : 0000000000000000 r18 : 0000000000000006 r19 : e000078000113c08 r20 : 0000000000000000 r21 : e000078000113c00 r22 : 0000000000000000 r23 : ffffffffffff04d8 r24 : e0000803859e8cb8 r25 : 000000000000033c r26 : e0000803859eab60 r27 : e0000803859eab54 r28 : e0000803859eab58 r29 : 000000007fffffff r30 : 000000000000033c r31 : 000000000000033c

Call Trace:
 [<a000000100015660>] show_stack+0x40/0xa0
                                sp=e0000803859ec690 bsp=e0000803859ec670
 [<a000000100015f90>] show_regs+0x870/0x8c0
                                sp=e0000803859ec860 bsp=e0000803859ec618
 [<a0000001000398b0>] die+0x1b0/0x2c0
                                sp=e0000803859ec860 bsp=e0000803859ec5c8
 [<a000000100039a10>] die_if_kernel+0x50/0x80
                                sp=e0000803859ec860 bsp=e0000803859ec598

This is all I see. Apparently system locked up.

------------------

Here's the memory init debug info:

Zone PFN ranges:
DMA 0x00000001 -> 0x00040000
Normal 0x00040000 -> 0x220ff000
Movable zone start PFN for each node
early_node_map[6] active PFN ranges

    4: 0x00000001 -> 0x00007ffa
    0: 0x1c008000 -> 0x1c0fec00
    1: 0x1e000000 -> 0x1e0ff000
    2: 0x20000000 -> 0x200ff000
    3: 0x22000000 -> 0x220fef67
    3: 0x220fefa0 -> 0x220feff6

mminit::pageflags_layout_widths Section 0 Node 6 Zone 2 Flags 23 mminit::pageflags_layout_shifts Section 20 Node 6 Zone 2 mminit::pageflags_layout_offsets Section 0 Node 58 Zone 56 mminit::pageflags_layout_zoneid Zone ID: 56 -> 64 mminit::pageflags_layout_usage location: 64 -> 56 unused 56 -> 23 flags 23 -> 0 On node 0 totalpages: 1010688
Normal zone: 3948 pages used for memmap
Normal zone: 1006740 pages, LIFO batch:7
mminit::memmap_init Initialising map node 0 zone 1 pfns 469794816 -> 470805504 On node 1 totalpages: 1044480
Normal zone: 4080 pages used for memmap
Normal zone: 1040400 pages, LIFO batch:7
mminit::memmap_init Initialising map node 1 zone 1 pfns 503316480 -> 504360960 On node 2 totalpages: 1044480
Normal zone: 4080 pages used for memmap
Normal zone: 1040400 pages, LIFO batch:7
mminit::memmap_init Initialising map node 2 zone 1 pfns 536870912 -> 537915392 On node 3 totalpages: 1044413
Normal zone: 4080 pages used for memmap
Normal zone: 1040333 pages, LIFO batch:7
mminit::memmap_init Initialising map node 3 zone 1 pfns 570425344 -> 571469814 On node 4 totalpages: 32761
DMA zone: 128 pages used for memmap
DMA zone: 0 pages reserved
DMA zone: 32633 pages, LIFO batch:7
mminit::memmap_init Initialising map node 4 zone 0 pfns 1 -> 32762 mminit::zonelist general 0:Normal = 0:Normal 1:Normal 2:Normal 3:Normal 4:DMA mminit::zonelist thisnode 0:Normal = 0:Normal
mminit::zonelist general 1:Normal = 1:Normal 2:Normal 3:Normal 0:Normal 4:DMA mminit::zonelist thisnode 1:Normal = 1:Normal
mminit::zonelist general 2:Normal = 2:Normal 3:Normal 0:Normal 1:Normal 4:DMA mminit::zonelist thisnode 2:Normal = 2:Normal
mminit::zonelist general 3:Normal = 3:Normal 0:Normal 1:Normal 2:Normal 4:DMA mminit::zonelist thisnode 3:Normal = 3:Normal
mminit::zonelist general 4:DMA = 4:DMA
mminit::zonelist thisnode 4:DMA = 4:DMA
Built 5 zonelists in Zone order, mobility grouping on. Total pages: 4160506

Note that this platform has a small [~512MB] pseudo-node #4 that contains DMA only. Here's the 'numactl --hardware' output:

available: 5 nodes (0-4)
node 0 size: 15792 MB
node 0 free: 14908 MB
node 1 size: 16320 MB
node 1 free: 15985 MB
node 2 size: 16320 MB
node 2 free: 16106 MB
node 3 size: 16318 MB
node 3 free: 16146 MB
node 4 size: 511 MB
node 4 free: 495 MB
node distances:
node 0 1 2 3 4

  0:  10  17  17  17  14 
  1:  17  10  17  17  14 
  2:  17  17  10  17  14 
  3:  17  17  17  10  14 
  4:  14  14  14  14  10 

If I create a cpuset with "mems" 0-3 -- i.e., eliminate the dma-only node 4 -- I do not hit the this "Nat Consumption" bug. The x86_64 test platform doesn't have this "feature".

I suspect that the page alloc optimizations are making assumptions that
aren't true for this platform. I know we had to muck around quite a
bit to get this all to work in the "memoryless nodes" and "two zonelist" patches a while back.

I'll try to bisect to specific patch--probably tomorrow.

Regards,
Lee

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

What line is this?

What is a pseudo-node?

Based on the timing of the bug, the most likely explanation
is that there is a problem in there. I went through the zonelist-walker changes but didn't spot anything. Could you try reverting page-allocator-do-not-check-numa-node-id-when-the-caller-knows-the-node-is-valid please? It has a few changes with repect to NUMA and ia-64 and the error might be in there somewhere.

Is it only the interleave policy that is affected or are other NUMA placement policies with node 4 causing trouble as well? If it's only interleave, are you aware of any recent changes to the interleave policy in -mm that might also explain this problem?

Can you also try with this minimal debugging patch applied and the full console log please? I'll keep thinking on it and hopefully I'll get inspired

index 4e0e265..82e17bb 100644
@@ -41,8 +41,6 @@ void mminit_verify_zonelist(void)
 			listid = i / MAX_NR_ZONES;
 			zonelist = &pgdat->node_zonelists[listid];
 			zone = &pgdat->node_zones[zoneid];
-			if (!populated_zone(zone))
-				continue;
 
 			/* Print information about the zonelist */
 			printk(KERN_DEBUG "mminit::zonelist %s %d:%s = ",

index 16ce8b9..c8c54d1 100644
@@ -57,6 +57,10 @@ struct zoneref *next_zones_zonelist(struct zoneref *z,
 					nodemask_t *nodes,
 					struct zone **zone)
 {
+	/* Should be impossible, check for NULL or near-NULL values for z */
+	BUG_ON(!z);
+	BUG_ON((unsigned long )z < PAGE_SIZE);
+
 	/*
 	 * Find the next suitable zone to use for the allocation.
 	 * Only filter based on nodemask if it's set

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi, Mel:

Sorry for the delay. Swamped. Was building incrementally patched kernels and took a while to get back to where I could [sort of] answer this. Below I've included part of the disassembly of the mmzone.o for this kernel.

<snip>

It's an artifact of the firmware and platform architecture. It's a memory-only node at physical address zero that contains memory that is hardware-interleaved across a small slice of the real, physical nodes' memory. It shows up in the ACPI SRAT/SLIT tables as a separate 'PXM' [proximity domain] that Linux treats as a "node". Because it's <4G [on my test platform], it's all ia64 dma zone.

Yeah, I've built some kernels to test. Best to build them all at once, since it takes a while to reboot this beast. I'll let you know.

I've tried to find a simple reproducer using memtoy, but haven't found one yet. The numactl package regression test hit is every time. I tried running a 'membind' test, and it doesn't seem to occur, altho I do hit oom:

numactl --membind=4 ./memhog $(scale 16G)

But, when I try interleave, it hits the bug:

numactl --interleave=all ./memhog $(scale 16G)

That hits the bug.

Will do. I'll send the results.

Here's a section of the disassembly of mmzone.o. Looks like the fault is in the inlined zonelist_zone_idx() -- 0x1a0 + 0x31 ~= 0x1d0/1d6. Dereferencing the zoneref pointer.

mm/mmzone.o: file format elf64-ia64-little

Disassembly of section .text:

0000000000000000 <next_online_pgdat>:

	return NODE_DATA(first_online_node);
}

<snip>

00000000000001a0 <next_zones_zonelist>:

static inline int zref_in_nodemask(struct zoneref *zref, nodemask_t *nodes) {

#ifdef CONFIG_NUMA
	return node_isset(zonelist_node_idx(zref), *nodes);
#else
	return 1;

#endif /* CONFIG_NUMA */
}

/* Returns the next zone at or below highest_zoneidx in a zonelist */ struct zoneref *next_zones_zonelist(struct zoneref *z,
enum zone_type highest_zoneidx, nodemask_t *nodes, struct zone **zone)
{
1a0: 10 40 00 40 00 21 [MIB] mov r8=r32

	/*
	 * Find the next suitable zone to use for the allocation.
	 * Only filter based on nodemask if it's set
	 */
	if (likely(nodes == NULL))
 1a6:	60 00 88 0e 72 03 	            cmp.eq p6,p7=0,r34
 1ac:	30 00 00 40       	      (p06) br.cond.sptk.few 1d0 <next_zones_zonelist+0x30>
 1b0:	11 00 00 00 01 00 	[MIB]       nop.m 0x0
 1b6:	00 00 00 02 00 00 	            nop.i 0x0
 1bc:	60 00 00 40       	            br.few 210 <next_zones_zonelist+0x70>;;
		while (zonelist_zone_idx(z) > highest_zoneidx)
			z++;
 1c0:	09 40 40 10 00 21 	[MMI]       adds r8=16,r8
 1c6:	00 00 00 02 00 00 	            nop.m 0x0
 1cc:	00 00 04 00       	            nop.i 0x0;;

}

static inline int zonelist_zone_idx(struct zoneref *zoneref)

{
	return zoneref->zone_idx;
 1d0:	0b 10 20 10 00 21 	[MMI]       adds r2=8,r8;;
 1d6:	e0 00 08 20 20 00 	            ld4 r14=[r2]	<<<< ???
 1dc:	00 00 04 00       	            nop.i 0x0;;
 1e0:	10 00 00 00 01 00 	[MIB]       nop.m 0x0
 1e6:	80 08 39 12 69 04 	            cmp4.ltu p8,p9=r33,r14
 1ec:	e0 ff ff 4a       	      (p08) br.cond.dptk.few 1c0 <next_zones_zonelist+0x20>
 1f0:	10 00 00 00 01 00 	[MIB]       nop.m 0x0
 1f6:	00 00 00 02 00 00 	            nop.i 0x0
 1fc:	b0 00 00 40       	            br.few 2a0 <next_zones_zonelist+0x100>
	else
		while (zonelist_zone_idx(z) > highest_zoneidx ||
				(z->zone && !zref_in_nodemask(z, nodes)))
			z++;
 200:	09 40 40 10 00 21 	[MMI]       adds r8=16,r8
 206:	00 00 00 02 00 00 	            nop.m 0x0
 20c:	00 00 04 00       	            nop.i 0x0;;

}

static inline int zonelist_zone_idx(struct zoneref *zoneref)

{
	return zoneref->zone_idx;
 210:	0b 48 20 10 00 21 	[MMI]       adds r9=8,r8;;
 216:	30 00 24 20 20 00 	            ld4 r3=[r9]
 21c:	00 00 04 00       	            nop.i 0x0;;
 220:	10 00 00 00 01 00 	[MIB]       nop.m 0x0
 226:	a0 08 0d 16 69 05 	            cmp4.ltu p10,p11=r33,r3

22c: e0 ff ff 4a (p10) br.cond.dptk.few 200 <next_zones_zonelist+0x60>
230: 09 00 00 00 01 00 [MMI] nop.m 0x0
static inline int zonelist_node_idx(struct zoneref *zoneref) {

#ifdef CONFIG_NUMA
	/* zone_to_nid not available in this context */
	return zoneref->zone->node;
 236:	a0 00 20 30 20 00 	            ld8 r10=[r8]
 23c:	00 00 04 00       	            nop.i 0x0;;
 240:	11 78 c0 14 00 21 	[MIB]       adds r15=48,r10
 246:	c0 00 28 1a 72 06 	            cmp.eq p12,p13=0,r10
 24c:	60 00 00 43       	      (p12) br.cond.dpnt.few 2a0 <next_zones_zonelist+0x100>;;
static inline int zonelist_node_idx(struct zoneref *zoneref)
{
#ifdef CONFIG_NUMA
	/* zone_to_nid not available in this context */
	return zoneref->zone->node;

250: 02 a0 00 1e 10 10 [MII] ld4 r20=[r15]

static __inline__ int
test_bit (int nr, const volatile void *addr)

{
	return 1 & (((const volatile __u32 *) addr)[nr >> 5] >> (nr & 31));
 256:	00 00 00 02 00 60 	            nop.i 0x0;;
 25c:	b2 a0 68 52       	            extr r19=r20,5,27
 260:	02 00 00 00 01 00 	[MII]       nop.m 0x0

static __inline__ int
test_bit (int nr, const volatile void *addr)

{
	return 1 & (((const volatile __u32 *) addr)[nr >> 5] >> (nr & 31));
 266:	f0 f8 50 58 40 00 	            and r15=31,r20;;
 26c:	00 00 04 00       	            nop.i 0x0
 270:	0b 90 4c 44 11 20 	[MMI]       shladd r18=r19,2,r34;;
 276:	10 01 48 60 21 00 	            ld4.acq r17=[r18]
 27c:	00 00 04 00       	            nop.i 0x0;;
 280:	03 00 00 00 01 00 	[MII]       nop.m 0x0
 286:	00 89 00 10 40 60 	            addp4 r16=r17,r0;;
 28c:	f1 80 00 79       	            shr.u r11=r16,r15;;
 290:	10 00 00 00 01 00 	[MIB]       nop.m 0x0
 296:	e0 00 2c 1e 28 07 	            tbit.z p14,p15=r11,0

29c: 70 ff ff 4a (p14) br.cond.dptk.few 200 <next_zones_zonelist+0x60>
else

static inline struct zone *zonelist_zone(struct zoneref *zoneref)

{
	return zoneref->zone;

2a0: 09 00 00 00 01 00 [MMI] nop.m 0x0

	*zone = zonelist_zone(z);
 2a6:	50 01 20 30 20 00 	            ld8 r21=[r8]
 2ac:	00 00 04 00       	            nop.i 0x0;;
 2b0:	11 00 54 46 98 11 	[MIB]       st8 [r35]=r21
	return z;
}

2b6: 00 00 00 02 00 80 nop.i 0x0
2bc: 08 00 84 00 br.ret.sptk.many b0;;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

<snip>

Mel: I think you can rest easy. I've duplicated the problem with a kernel that truncates the mmotm 04/28 series just before your patches. Hope it's not my cpuset-mm fix that occurs just before that! I'll let you know.

Did hit one or your BUG_ON's, tho'. See below.

Ok, I can rest a little easier but I won't that much. I've mucked around enough in there over the last while that it might still be something I did.

I don't think so because it was in mmotm before my patchset was and you didn't spot any problems.

This implies that z was NULL when it was passed to the iterator

#define for_each_zone_zonelist_nodemask(zone, z, zlist, highidx, nodemask) \
        for (z = first_zones_zonelist(zlist, highidx, nodemask, &zone); \
                zone; \
                z = next_zones_zonelist(++z, highidx, nodemask, &zone)) \

and we ended up with z == ++NULL;

Can you send the full dmesg and what your bisection point was? Maybe I can spot something. The implication is that a corrupt or badly constructed zonelist is being passed into the page allocator so I'd like to see where it is coming from.

Thanks

Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

[Note: we're experiencing some e-mail problems at HP today. I've added a Reply-to: that should work, but my normal hp address will probably bounce until this gets resolved.]

Mel:

I've tracked it down to this patch:

cpusetmm-update-tasks-mems_allowed-in-time.patch

Without that patch, the numademo command:

	numactl --interleave=all ./memhog $(scale 16G)

runs to completion w/o error. With this patch pushed, I capture this trace today:

numactl[7408]: NaT consumption 17179869216 [1]

Modules linked in: ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc vfat fat dm_mirror dm_multipath scsi_dh pci_slot parport_pc lp parport sg sr_mod cdrom button tg3 e1000 libphy dm_region_hash dm_log dm_mod sym53c8xx mptspi mptscsih mptbase scsi_transport_spi sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table]

Pid: 7408, CPU 2, comm: numactl
psr : 0000101008526030 ifs : 8000000000000897 ip : [<a0000001001335f0>] Not tainted (2.6.30-rc3-mmotm-090428-1631+zonelist-debug-7)
ip is at __alloc_pages_internal+0x110/0x720
unat: 0000000000000000 pfs : 0000000000000897 rsc : 0000000000000003
rnat: 0000000000000000 bsps: 00000000001cca31 pr : 0000000500555a69
ldrs: 0000000000000000 ccv : 0000000000000000 fpsr: 0009804c8a70433f csd : 0000000000000000 ssd : 0000000000000000
b0 : a0000001001335d0 b6 : a000000100038fa0 b7 : a00000010000bbc0 f6 : 1003e000000000000003e f7 : 1003e000000003ffffffe f8 : 100178208141befbf5f00 f9 : 1003effffffffffffffc1 f10 : 1003e00000000001dbeff f11 : 1003e0044b82fa09b5a53 r1 : a000000100c844c0 r2 : e00007038a7a0ce0 r3 : e00007038a7a0cd0 r8 : 0000000000000000 r9 : 0000000000020000 r10 : 0000000000000000 r11 : 0000000000000000 r12 : e00007038a7a7de0 r13 : e00007038a7a0000 r14 : 0000000000000000 r15 : e00007038a7a0cf8 r16 : e000070020103b88 r17 : 0000000000000000 r18 : e000070020103b80 r19 : 0000000000000000 r20 : 0000000000000018 r21 : a000000100a22ec8 r22 : a000000100a22ec0 r23 : a000000100a852f0 r24 : e000070380031808 r25 : e00007038003180c r26 : 0000000000000000 r27 : 0000000000000000 r28 : 0000000000004000 r29 : 0000000000004000 r30 : 0000000000000000 r31 : e0000783821c9900

Call Trace:
 [<a000000100015720>] show_stack+0x40/0xa0
                                sp=e00007038a7a7830 bsp=e00007038a7a13d0
 [<a000000100016050>] show_regs+0x870/0x8c0
                                sp=e00007038a7a7a00 bsp=e00007038a7a1378
 [<a0000001000399d0>] die+0x1b0/0x2c0
                                sp=e00007038a7a7a00 bsp=e00007038a7a1330
 [<a000000100039b30>] die_if_kernel+0x50/0x80
                                sp=e00007038a7a7a00 bsp=e00007038a7a1300
 [<a000000100733980>] ia64_fault+0x1140/0x1260
                                sp=e00007038a7a7a00 bsp=e00007038a7a12a8
 [<a00000010000c3c0>] ia64_native_leave_kernel+0x0/0x270
                                sp=e00007038a7a7c10 bsp=e00007038a7a12a8
 [<a0000001001335f0>] __alloc_pages_internal+0x110/0x720
                                sp=e00007038a7a7de0 bsp=e00007038a7a11e8
 [<a000000100181b60>] alloc_page_interleave+0xa0/0x160
                                sp=e00007038a7a7df0 bsp=e00007038a7a11a8
 [<a000000100182060>] alloc_page_vma+0x120/0x220
                                sp=e00007038a7a7df0 bsp=e00007038a7a1170
 [<a000000100156a50>] handle_mm_fault+0x330/0xf60
                                sp=e00007038a7a7df0 bsp=e00007038a7a1100
 [<a000000100157b80>] __get_user_pages+0x500/0x820
                                sp=e00007038a7a7e00 bsp=e00007038a7a1090
 [<a000000100157f00>] get_user_pages+0x60/0x80
                                sp=e00007038a7a7e10 bsp=e00007038a7a1038
 [<a0000001001ad3d0>] get_arg_page+0x50/0x160
                                sp=e00007038a7a7e10 bsp=e00007038a7a1008
 [<a0000001001ad940>] copy_strings+0x200/0x3a0
                                sp=e00007038a7a7e20 bsp=e00007038a7a0f78
 [<a0000001001adb20>] copy_strings_kernel+0x40/0x60
                                sp=e00007038a7a7e20 bsp=e00007038a7a0f40
 [<a0000001001b0b80>] do_execve+0x320/0x5e0
                                sp=e00007038a7a7e20 bsp=e00007038a7a0ee0
 [<a000000100014780>] sys_execve+0x60/0xa0
                                sp=e00007038a7a7e30 bsp=e00007038a7a0ea8
 [<a00000010000b910>] ia64_execve+0x30/0x140
                                sp=e00007038a7a7e30 bsp=e00007038a7a0e58
 [<a00000010000c240>] ia64_ret_from_syscall+0x0/0x20
                                sp=e00007038a7a7e30 bsp=e00007038a7a0e58
 [<a000000000010720>] _stext+0xffffffff00010720/0x400
                                sp=e00007038a7a8000 bsp=e00007038a7a0e58
---------------------------------------------------------

So, I'm off trying to figure what's happening with interleaved allocations with this patch.

Later,
Lee

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>