Some issues + [PATCH] kanoj-mm8-2.2.9 Show statistics on alloc/free requests for each pagefree list

Attached is a patch to mm/page_alloc.c that will report the cumulative number of alloc and free requests for pages of each size via the MagicSysRq 'm' command. To turn on the display, you need to add a

#define FREELIST_STAT

to mm/page_alloc.c before the #ifdef FREELIST_STAT line.

On my HP-Kayak 2p ia32 system, the relevant output right after I get a login prompt is:

2*4kB (19651, 33223) 1*8kB (373, 299) 1*16kB (2, 0) 0*32kB (2, 0) 1*64kB (0, 0) 0*128kB (0, 0) 0*256kB (1, 0) 0*512kB (0, 0) 0*1024kB (0, 0) 26*2048kB (0, 0) = 53344kB)

And after running a 2.2.9 kernel compile:

183*4kB (510767, 515934) 19*8kB (2480, 2323) 10*16kB (2, 0) 3*32kB (2, 0) 0*64kB (0, 0) 0*128kB (0, 0) 2*256kB (1, 0) 2*512kB (0, 0) 0*1024kB (0, 0) 8*2048kB (0, 0) = 19060kB)

The first number in the bracketed pair is the number of alloc requests, the second is the number of free requests. (Yes, don't ask me how the frees outnumber the allocs for 4K pages, probably some code is asking for bigger pages and freeing single pages).

Anyway, this raises some interesting questions about the buddy algorithm. Is it really worth aggressively coalescing pages on each free? Wouldn't it be better to lazily coalesce pages (maybe by a kernel thread), or even on demand? By far, the most number of requests are coming for the 4K pages, followed by 8K (task/stack pair). A kernel compile is no representative app, but I would be surprised if there are too many apps/drivers which will force bigger page requests, once kernel initialization is complete. Wouldn't it be better to optimize the more common case?

Not to mention that if we do not do aggressive coalescing, we could think about maintaining pages freed by shrink_mmap in the SwapCache/filecache, so that those pages could be reclaimed from the free list on re-reference.

Kanoj
[email protected]

@@ -36,11 +36,32 @@
 #define NR_MEM_LISTS 10
 #endif
 
+#define FREELIST_STAT
+#ifdef FREELIST_STAT
+#define	alloc_stat_field	unsigned long alloc_stat;
+#define free_stat_field		unsigned long free_stat;
+#define init_stat(area)		(area)->alloc_stat = (area)->free_stat = 0
+#define alloc_stat_inc(area)	(area)->alloc_stat++
+#define free_stat_inc(area)	(area)->free_stat++
+#define alloc_stat_get(area)	(area)->alloc_stat
+#define free_stat_get(area)	(area)->free_stat
+#else
+#define	alloc_stat_field
+#define	free_stat_field
+#define	init_stat(area)
+#define alloc_stat_inc(area)
+#define free_stat_inc(area)
+#define alloc_stat_get(area)	(unsigned long)0
+#define free_stat_get(area)	(unsigned long)0
+#endif
+
 /* The start of this MUST match the start of "struct page" */
 struct free_area_struct {
 	struct page *next;
 	struct page *prev;
 	unsigned int * map;
+	alloc_stat_field
+	free_stat_field
 };
 
 #define memory_head(x) ((struct page *)(x))

@@ -51,6 +72,7 @@
 {
 	head->next = memory_head(head);
 	head->prev = memory_head(head);
+	init_stat(head);
 }
 
 static inline void add_mem_queue(struct free_area_struct * head, struct page * entry)

@@ -99,6 +121,8 @@
 
 	spin_lock_irqsave(&page_alloc_lock, flags);
 
+	free_stat_inc(free_area + order);
+
 #define list(x) (mem_map+(x))
 
 	map_nr &= mask;

@@ -236,6 +260,7 @@
 	}
 ok_to_allocate:
 	spin_lock_irqsave(&page_alloc_lock, flags);
+	alloc_stat_inc(free_area + order);
 	RMQUEUE(order, gfp_mask);
 	spin_unlock_irqrestore(&page_alloc_lock, flags);
 

@@ -277,7 +302,7 @@
 			nr ++;
 		}
 		total += nr * ((PAGE_SIZE>>10) << order);
-		printk("%lu*%lukB ", nr, (unsigned long)((PAGE_SIZE>>10) << order));
+		printk("%lu*%lukB (%lu, %lu) ", nr, (unsigned long)((PAGE_SIZE>>10) << order), alloc_stat_get(free_area + order), free_stat_get(free_area + order));
 	}
 	spin_unlock_irqrestore(&page_alloc_lock, flags);
 	printk("= %lukB)\n", total);

--
To unsubscribe, send a message with 'unsubscribe linux-mm my@address' in the body to [email protected]. For more info on Linux MM,
see: http://humbolt.geo.uu.nl/Linux-MM/

There is a important case ATM that needs bigger blocks allocated from bottom half context: NFS packet defragmenting. For a 8K wsize it needs even 16K blocks (8K payload + the IP/UDP header forces it to the next buddy size). I guess your statistics would look very different on a nfsroot machine. Until lazy defragmenting is supported for UDP it is probably better not to change it.

-Andi
--
To unsubscribe, send a message with 'unsubscribe linux-mm my@address' in the body to [email protected]. For more info on Linux MM,
see: http://humbolt.geo.uu.nl/Linux-MM/

This is the experiment I tried: using automount, I cd'ed into a nfs mounted directiory, and copied kernel sources over to the local (client) machine. The statistics before and after the copy on the client:

Before:

10*4kB (20993, 34343) 3*8kB (398, 319) 0*16kB (2, 0) 0*32kB (2, 0) 0*64kB (0, 0) 1*128kB (0, 0) 0*256kB (1, 0) 0*512kB (0, 0) 1*1024kB (0, 0) 25*2048kB (0, 0) = 52416kB)

After:

192*4kB (88737, 89889) 27*8kB (744, 405) 3*16kB (2, 0) 0*32kB (2, 0) 0*64kB (0,
0) 0*128kB (0, 0) 0*256kB (1, 0) 1*512kB (0, 0) 0*1024kB (0, 0) 0*2048kB (0, 0) = 1544kB)

I am not sure about the wsize though ... maybe someone with access to a nfsroot machine can try a quick experiment and publish the results?

Btw, if the nfs defrag code is coming from bottom half, it probably has logic to handle allocation failures? Andi, could you please send me a pointer to the relevant code?

Thanks.

Kanoj

--
To unsubscribe, send a message with 'unsubscribe linux-mm my@address' in the body to [email protected]. For more info on Linux MM,
see: http://humbolt.geo.uu.nl/Linux-MM/

You probably used the default of 4K (=8K blocks). 8K wsize often performs better against other Linux servers.

BTW, I am a bit surprised that you don't have a nfsroot or at least nfs-/usr machine - it is really handy for experimental kernel testing:
no fscks, no fs corruptions ..

The relevant code in net/ipv4/ip_fragment.c; called from the IP input path running in net_bh (net/core/dev.c:net_bh() -> net/ipv4/ip_input.c)

It simply drops the packet then. Remember that IP is unreliable so this works always.

In theory it could set a a short retry timer and try again later (because the smaller fragments in the frag queue are still there), but this would need some complicated backlog refeed logic.

Fixing defragmenting to directly defragment into the target buffer is on my list for 2.3; I assume it is on David's list too so I it'll probably change.

-Andi

This is like TV. I don't like TV.
--
To unsubscribe, send a message with 'unsubscribe linux-mm my@address'
in the body to [email protected].  For more info on Linux MM,
see: http://humbolt.geo.uu.nl/Linux-MM/