Skip to contentSkip to conversation

help with next generation bkbits please

DISCUSSION Larry McVoy 2 participants 3 messages Sep 21, 2004

lore
Share & tools
lore
#0 lore raw

Hi,

we're trying to upgrade bkbits.net for your pushing&pulling pleasure and we're having some problems.

We wanted to throw lots of memory at the problem so we went with an ASUS SK8V motherboard, opteron 148, 4 x 1GB registered / ECC dimms. We thought we would be careful so we bought dimms that ASUS claims works.

We can't get the system to stabilize and we're looking for either
a) information on how to do that or
b) a suggestion for a machine which will support 4GB or more

What we are currently seeing looks like a cache writeback problem. I have a simple memory scrubber, see below, which just cycles through a series of patterns, verifying the previous one and writing a new one, switch pattern, repeat until pattern list is exhausted, then loop. We cycle through the offset into the array, 0xdeadbeef, 0x50505050, 0x0a0a0a0a, 0x55555555, 0xaaaaaaaa, 0, 0xffffffff.

What we see is that for 16x4 bytes in a row we will get errors where what we get is the previous value. In other words, we just went through a loop that verified that all the data is 0xdeadbeef and then set it to 0x50505050, and then in the next loop 16 values will be 0xdeadbeef. In other words, it looks like the cache writeback didn't work, it's as if the dirty bits were cleared for some reason.

Does anyone have any idea what might be causing this and what we need to do to fix it? We'd really like to get you guys a nicer machine.

Signature · 2 lines
---
Larry McVoy                lm at bitmover.com           http://www.bitkeeper.com
#1 lore raw

Forgot memory scrubber, here it is.

#include <stdio.h>
#include <stdlib.h>
#include <strings.h>
#include <unistd.h>

#define unless(x) if (!(x))
#define u32 unsigned int
#define status(s) write(1, s, 1); newline = 1;

int newline;
void doit(u32 bs);

int
main(int ac, char **av)

{
	if (ac != 2) {
		printf("Usage: %s amount_in_MB\n", av[0]);
		exit(0);
	}
	doit(atoi(av[1])<<20);
	return (0);
}

void
bad(u32 *start, u32 *p, u32 want)

{
	u32	off = (char*)p - (char*)start;
	u32	got = *p;

	if (newline) printf("\n");
	newline = 0;
	printf("WANT=0x%08x GOT=0x%08x DIFF=0x%08x OFF=0x%08x ADDR=%p\n",
	    want, got, want - got, off, (void*)p);
}

void
doit(u32 bs)

{
	u32	*buf = malloc(bs);
	u32	*end;
	u32	*p;
	u32	off;

	fprintf(stderr, "Scrub %u bytes\n", bs);
	unless (buf) {
		perror("malloc");
		exit(1);
	}
	end = (u32*)((char*)buf + bs);
	bzero(buf, bs);
	unless (sizeof(int) == 4) {
		fprintf(stderr, "Expected 4 byte ints\n");
		exit(1);
	}
	for (off = 0, p = buf; p < end; *p++ = off, off += 4);
	for (;;) {
		for (off = 0, p = buf; p < end; off += 4) {
			if (*p != off) bad(buf, p, off);
			*p++ = 0xdeadbeef;
		}
		status("d");
		for (off = 0, p = buf; p < end; off += 4) {
			if (*p != 0xdeadbeef) bad(buf, p, 0xdeadbeef);
			*p++ = 0x50505050;
		}
		status("5");
		for (off = 0, p = buf; p < end; off += 4) {
			if (*p != 0x50505050) bad(buf, p, 0x50505050);
			*p++ = 0x0a0a0a0a;
		}
		status("a");
		for (off = 0, p = buf; p < end; off += 4) {
			if (*p != 0x0a0a0a0a) bad(buf, p, 0x0a0a0a0a);
			*p++ = 0x55555555;
		}
		status("-");
		for (off = 0, p = buf; p < end; off += 4) {
			if (*p != 0x55555555) bad(buf, p, 0x55555555);
			*p++ = 0xaaaaaaaa;
		}
		status("A");
		for (off = 0, p = buf; p < end; off += 4) {
			if (*p != 0xaaaaaaaa) bad(buf, p, 0xaaaaaaaa);
			*p++ = 0x0;
		}
		status("0");
		for (off = 0, p = buf; p < end; off += 4) {
			if (*p != 0x0) bad(buf, p, 0);
			*p++ = 0xffffffff;
		}
		status("f");
		for (off = 0, p = buf; p < end; off += 4) {
			if (*p != 0xffffffff) bad(buf, p, 0xffffffff);
			*p++ = off;
		}
		status("o");
	}
}
#2 lore raw
In reply to Larry McVoy
Quoted 20 lines · Larry McVoy

Hi,

we're trying to upgrade bkbits.net for your pushing&pulling pleasure and we're having some problems.

We wanted to throw lots of memory at the problem so we went with an ASUS SK8V motherboard, opteron 148, 4 x 1GB registered / ECC dimms. We thought we would be careful so we bought dimms that ASUS claims works.

We can't get the system to stabilize and we're looking for either
a) information on how to do that or
b) a suggestion for a machine which will support 4GB or more

What we are currently seeing looks like a cache writeback problem. I have a simple memory scrubber, see below, which just cycles through a series of patterns, verifying the previous one and writing a new one, switch pattern, repeat until pattern list is exhausted, then loop. We cycle through the offset into the array, 0xdeadbeef, 0x50505050, 0x0a0a0a0a, 0x55555555, 0xaaaaaaaa, 0, 0xffffffff.

Quoted 2 lines

What we see is that for 16x4 bytes in a row we will get errors where what we get is the previous value. In other words, we just went through

Show the output of scrubber. Does it happen on random addresses? Same address? With which sizes (L1/L2/main RAM) does it happen? etc...

Quoted 4 lines

a loop that verified that all the data is 0xdeadbeef and then set it to 0x50505050, and then in the next loop 16 values will be 0xdeadbeef. In other words, it looks like the cache writeback didn't work, it's as if the dirty bits were cleared for some reason.

64 bytes is a cacheline size for Opteron. You may have a faulty CPU. --
vda