allocate page caches pages in round robin fasion

On a NUMA machine, page cache pages should be spread out across the system since they're generally global in nature and can eat up whole nodes worth of memory otherwise. This can end up hurting performance since jobs will have to make off node references for much or all of their non-file data.

The patch works by adding an alloc_page_round_robin routine that simply allocates on successive nodes each time its called, based on the value of a per-cpu variable modulo the number of nodes. The variable is per-cpu to avoid cacheline contention when many cpus try to do page cache allocations at once.

After dd if=/dev/zero of=/tmp/bigfile bs=1G count=2 on a stock kernel:

Node 7 MemUsed:         49248 kB
Node 6 MemUsed:         42176 kB
Node 5 MemUsed:        316880 kB
Node 4 MemUsed:         36160 kB
Node 3 MemUsed:         45152 kB
Node 2 MemUsed:         50000 kB
Node 1 MemUsed:         68704 kB
Node 0 MemUsed:       2426256 kB

and after the patch:

Node 7 MemUsed:        328608 kB
Node 6 MemUsed:        319424 kB
Node 5 MemUsed:        318608 kB
Node 4 MemUsed:        321600 kB
Node 3 MemUsed:        319648 kB
Node 2 MemUsed:        327504 kB
Node 1 MemUsed:        389504 kB
Node 0 MemUsed:        744752 kB

Thanks,
Jesse

Wouldn't this be painful for any workload that accesses a unique set of files on each node? If an application knows that it is touching truly shared data which every node could possibly access, then they can use the NUMA API to cause round-robin allocations to occur.

Maybe a per-node watermark on page cache usage would be more useful. Once a node starts to get full, and it's past the watermark, we can go and shoot down some of the node's page cache. If the data access is truly global, then it has a good chance of being brought in on a different node.

-- Dave