Cache memory never gets released

Yes, I am absolutely sure I have a problem. The cached memory gets lost so I can't ever use it again over time. I have a simple hog.c program that just allocs blocks of 1M and fills them with data:
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>

/* This program just hogs a lot of memory */

#define SIZE 0x100000
int main(int argc,char **argv)
{
char *p;

int i=0;
        for(;;)
        {
                p=malloc(SIZE);
                if(!p) break;
                memset(p,0,SIZE);
                i+=SIZE;printf("Allocated %dM\n",i/1000000);
//              sleep(1);
        }
        exit(0);
}

On a pristine system I can work the cached line output by "free" down to around 6M. After usage, over time, the cached amount works its way up to be basically all of system memory, and I can't ever work it down. The hog program above runs but the kernel kills it after only a small amount of memory can be allocated.

I'm not doing anything mysterious. The kernel is stock 2.4.26 except for some changes to these files:
Documentation/Configure.help
Makefile
arch/i386/config.in
drivers/sound/via82cxxx_audio.c
fs/proc/array.c
fs/proc/proc_misc.c
include/asm-i386/param.h
kernel/signal.c
kernel/sys.c
net/ipv4/icmp.c

Nothing related to caching/memory management has been touched. I've applied the variable HZ patch so ticks happen 500/second up from 100/second.

Is there any valid reason why cached memory would stay locked at 54M for example and never go lower, even when there are almost no processes running on the system?

Thanks--
Dave
PS Is there some way that cached blocks can somehow form interdependencies? For example block A can't be released until block B is, and vice-versa? And that these can build up over time? This is the kind of behaviour I'm seeing.