Skip to contentSkip to conversation

Wow! Is memory ever cheap!

DISCUSSION Larry McVoy 6 participants 14 messages May 5, 2001 – May 7, 2001

lore
Share & tools
lore
#0 lore raw

This is a 750Mhz K7 system with 1.5GB of memory in 3 512MB DIMMS. The DIMMS are not ECC, but we use BitKeeper here and it tells us when we have bad DIMMS.

Guess what the memory cost? $396.58 shipped to my door, second day air, with a lifetime warranty. I got it at www.memory4less.com which I found using www.pricewatch.com. I have no association with either of those places other than being a customer (i.e., this isn't advertising spam).

I'm burning it in right now, I wrote a little program which fills it with different test patterns and then reads them back to make sure they don't lose any bits. Seems to be working, it's done about 30 passes.

1.5GB for $400. Amazing. No more whining from you guys that BitKeeper uses too much memory :-)

$ hinv
Main memory size: 1535.9375 Mbytes
1 AuthenticAMD  processor
1 1.44M floppy drive
1 vga+ graphics device
1 keyboard
IDE devices:
    /dev/hda is a ST310211A, 9541MB w/512kB Cache, CHS=1216/255/63
SCSI devices:
    /dev/sda is a 3ware disk, model 3w-xxxx 74.541 GB
PCI bus devices:
    Host bridge: VIA Technologies VT 82C691 Apollo Pro (rev 2).
    PCI bridge: VIA Technologies VT 82C598 Apollo MVP3 AGP (rev 0).
    ISA bridge: VIA Technologies VT 82C686 Apollo Super (rev 34).
    IDE interface: VIA Technologies VT 82C586 Apollo IDE (rev 16).
    Host bridge: VIA Technologies VT 82C686 Apollo Super ACPI (rev 48).
    Ethernet controller: 3Com 3C905B 100bTX (rev 48).
    Ethernet controller: 3Com 3C905B 100bTX (rev 48).
    Ethernet controller: 3Com 3C905B 100bTX (rev 48).
    Ethernet controller: 3Com 3C905B 100bTX (rev 48).
    RAID storage controller: Unknown vendor Unknown device (rev 18).
    VGA compatible controller: Matrox Matrox G200 AGP (rev 1).
Signature · 2 lines
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 
#1 lore raw

I'll frickin' whine if I want to :-). I still use bitkeeper on a Solaris 2.6 machine with 32MB of memory.

Quoted 47 lines · Larry McVoy

This is a 750Mhz K7 system with 1.5GB of memory in 3 512MB DIMMS. The DIMMS are not ECC, but we use BitKeeper here and it tells us when we have bad DIMMS.

Guess what the memory cost? $396.58 shipped to my door, second day air, with a lifetime warranty. I got it at www.memory4less.com which I found using www.pricewatch.com. I have no association with either of those places other than being a customer (i.e., this isn't advertising spam).

I'm burning it in right now, I wrote a little program which fills it with different test patterns and then reads them back to make sure they don't lose any bits. Seems to be working, it's done about 30 passes.

1.5GB for $400. Amazing. No more whining from you guys that BitKeeper uses too much memory :-)

$ hinv
Main memory size: 1535.9375 Mbytes
1 AuthenticAMD  processor
1 1.44M floppy drive
1 vga+ graphics device
1 keyboard
IDE devices:
    /dev/hda is a ST310211A, 9541MB w/512kB Cache, CHS=1216/255/63
SCSI devices:
    /dev/sda is a 3ware disk, model 3w-xxxx 74.541 GB
PCI bus devices:
    Host bridge: VIA Technologies VT 82C691 Apollo Pro (rev 2).
    PCI bridge: VIA Technologies VT 82C598 Apollo MVP3 AGP (rev 0).
    ISA bridge: VIA Technologies VT 82C686 Apollo Super (rev 34).
    IDE interface: VIA Technologies VT 82C586 Apollo IDE (rev 16).
    Host bridge: VIA Technologies VT 82C686 Apollo Super ACPI (rev 48).
    Ethernet controller: 3Com 3C905B 100bTX (rev 48).
    Ethernet controller: 3Com 3C905B 100bTX (rev 48).
    Ethernet controller: 3Com 3C905B 100bTX (rev 48).
    Ethernet controller: 3Com 3C905B 100bTX (rev 48).
    RAID storage controller: Unknown vendor Unknown device (rev 18).
    VGA compatible controller: Matrox Matrox G200 AGP (rev 1).
Signature · 7 lines
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
#2 lore raw
In reply to Larry McVoy

I'm burning it in right now, I wrote a little program which fills it with different test patterns and then reads them back to make sure they don't lose any bits. Seems to be working, it's done about 30 passes.

I wrote something similar to test an Alpha with a flakey L2 cache; it didn't find anything. However, a script that did kernel compiles in a loop soon finds errors.

I don't know much about memory testing, other than it is hard, really hard -- and there is some magic in the way gcc access memory that seems to trigger nasties.

It has been suggested that a good thesis would be to distill whatever magic gcc has for testing memory and study that :)

1.5GB for $400. Amazing. No more whining from you guys that BitKeeper uses too much memory :-)

1.5GB without ECC? Seems like a disater waiting to happen? Is ECC memory much more expensive?

--cw

#3 lore raw
In reply to Larry McVoy
Quoted 2 lines · Chris Wedgwood

1.5GB without ECC? Seems like a disater waiting to happen? Is ECC memory much more expensive?

Almost twice as expensive for 512MB dimms.

I used to be a die hard ECC fan but that changed since what we do here is BitKeeper and BitKeeper checksums everything. It tells us right away when we have problems (to date it has found bad memory dimms, NFS corruption, and a SPARC/Linux cache aliasing bug). So I've given up in ECC, we don't need it.

On the other hand, if your apps don't have built in integrity checks then ECC is pretty much a requirement.

By the way, the integrity checks don't need to be complicated, BK uses a horrible 16 bit ignore the overflow checksum to be compat with SCCS and it seems to have caught everything that much more sophisticated and CPU intensive checksums have caught. In other words, anything is much much better than nothing.

Signature · 2 lines
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 
#4 lore raw
In reply to Larry McVoy
Followup to:  <[email protected]>
By author:    Larry McVoy <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 7 lines · Chris Wedgwood
Quoted 2 lines

1.5GB without ECC? Seems like a disater waiting to happen? Is ECC memory much more expensive?

Almost twice as expensive for 512MB dimms.

Quoted 9 lines

I used to be a die hard ECC fan but that changed since what we do here is BitKeeper and BitKeeper checksums everything. It tells us right away when we have problems (to date it has found bad memory dimms, NFS corruption, and a SPARC/Linux cache aliasing bug). So I've given up in ECC, we don't need it.

On the other hand, if your apps don't have built in integrity checks then ECC is pretty much a requirement.

Isn't this pretty much saying "if you're willing to dedicate your system to running nothing but Bitkeeper, you can run it really fast?"

-hpa

Signature · 3 lines
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt
#5 lore raw
In reply to Larry McVoy
Quoted 16 lines · H. Peter Anvin
Followup to:  <[email protected]>
By author:    Larry McVoy <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 13 lines · Chris Wedgwood
Quoted 2 lines

1.5GB without ECC? Seems like a disater waiting to happen? Is ECC memory much more expensive?

Almost twice as expensive for 512MB dimms.

I used to be a die hard ECC fan but that changed since what we do here is BitKeeper and BitKeeper checksums everything. It tells us right away when we have problems (to date it has found bad memory dimms, NFS corruption, and a SPARC/Linux cache aliasing bug). So I've given up in ECC, we don't need it.

Quoted 5 lines
Quoted 2 lines

On the other hand, if your apps don't have built in integrity checks then ECC is pretty much a requirement.

Isn't this pretty much saying "if you're willing to dedicate your system to running nothing but Bitkeeper, you can run it really fast?"

A) Fast has nothing to do with it, ECC runs at the same speed as non-ECC;
B) As I said above, "if your apps don't have built in integrity checks then ECC is pretty much a requirement";
C) As I said above, we use our systems for BK development, so this choice makes sense for us.

I think the point you are really missing is that it is not an either/or choice. All you really need in practice is one application which is both heavily used and has integrity checks. It could be BitKeeper or something else, all that matters is that it will detect memory problems.

That application will flush out your memory problems. Yeah, you could get burned before that app finds them and if you are worried about that, then run ECC. I think it's an interesting data point, however, that I care deeply about data integrity and I've transitioned from insisting on ECC to not caring. If my choice wasn't working for me, I would still be using ECC. In other words, I'm a lot closer to your way of thinking than you might expect.

Signature · 2 lines
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 
#6 lore raw
In reply to Larry McVoy
Quoted 8 lines · Larry McVoy
Quoted 6 lines
Quoted 3 lines

On the other hand, if your apps don't have built in integrity checks then ECC is pretty much a requirement.

Isn't this pretty much saying "if you're willing to dedicate your system to running nothing but Bitkeeper, you can run it really fast?"

A) Fast has nothing to do with it, ECC runs at the same speed as non-ECC;

"It" meaning BitKeeper.

Quoted 9 lines

B) As I said above, "if your apps don't have built in integrity checks then ECC is pretty much a requirement";
C) As I said above, we use our systems for BK development, so this choice makes sense for us.

I think the point you are really missing is that it is not an either/or choice. All you really need in practice is one application which is both heavily used and has integrity checks. It could be BitKeeper or something else, all that matters is that it will detect memory problems.

This is not true in my experience. YES, it will detect bad memory configurations, but with over 2^34 memory cells to worry about -- each of them carrying a charge of a few dozen electrons only -- you WILL have random failures, especially if the memory is allowed to remain stale for extended periods of time, as is very likely in a configuration like this (think disk cache.)

Bad memory configurations is bad. However, good memory configurations are not necessarily perfect.

-hpa

Signature · 3 lines
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt
#7 lore raw
In reply to Larry McVoy
Quoted 7 lines · H. Peter Anvin
Quoted 4 lines · Larry McVoy
Quoted 2 lines

Isn't this pretty much saying "if you're willing to dedicate your system to running nothing but Bitkeeper, you can run it really fast?"

A) Fast has nothing to do with it, ECC runs at the same speed as non-ECC;

"It" meaning BitKeeper.

What does BitKeeper have to do with this conversation?

s/BitKeeper/any_app_which_has_integrity_checks/

Whether that app runs fast or not has nothing to do with ECC/non-ECC, right? And while whether that app runs fast or not may have something to do with other apps that you run along side of it, that's true for all apps, right? So why the focus on BitKeeper? Am I missing something?

Quoted 6 lines

This is not true in my experience. YES, it will detect bad memory configurations, but with over 2^34 memory cells to worry about -- each of them carrying a charge of a few dozen electrons only -- you WILL have random failures, especially if the memory is allowed to remain stale for extended periods of time, as is very likely in a configuration like this (think disk cache.)

BitKeeper, at least, runs the integrity checks every time it accesses the data. So it doesn't matter if it is in the disk cache or not. The same could be true of any other application.

Quoted 2 lines

Bad memory configurations is bad. However, good memory configurations are not necessarily perfect.

No, they most certainly aren't. You can have all the ECC you want and if the disk or the bus or some or the part of the path corrupts the data then you are hosed.

Dave Clark made the point that you _MUST_ have end to end checks if you care about your data. He would argue, correctly, in my opinion, that it doesn't matter if you have ECC, something else can screw you.

And in fact, while we were having this discussion I was running a disk scrubber to see if I had bad disks or not:

[root@disks /u1]# df -m .
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/hdg2 17099 0 17099 0% /u1
[root@disks /u1]# ~lm/tmp/a.out 17000
000000000000000001111111111
BAD @ 1045602304 3e50b000:3e52a000
[root@disks /u1]#

and from dmesg:

hdh: irq timeout: status=0x50 { DriveReady SeekComplete }
hdg: timeout waiting for DMA
ide_dmaproc: chipset supported ide_dma_timeout func only: 14

But my application was NEVER notified that the drive subsystem was hosed, the operating system (this is 2.4.4, by the way, steaming hot bits), never told the application that the data was probably bad. And it isn't a memory problem, I ran a memory scrubber and the system memory is just fine, it's the disk subsystem that went out to lunch. Without telling me, by the way. If this happened inside of SUN the guy reponsible would be getting a new orifice courtesy of systems group. Not cool to pass the data up when it is bad.

So explain to me how ECC is enough? It's clearly not. People have made compelling arguments for end to end checks for at least 25 years, the internet works largely because of these end to end checks (turn off checksums and find out if I'm right or not, you'll see), ECC isn't end to end, so what's the point? Yeah, it's better than nothing but to argue that it even remotely approaches enough is just flat out wrong, and it is is _inherently_ unable to be part of a solution which is correct, it's simply one place that the data lands.

Signature · 2 lines
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 
#8 lore raw
In reply to Larry McVoy
Quoted 5 lines · Larry McVoy
Quoted 4 lines
Quoted 2 lines

A) Fast has nothing to do with it, ECC runs at the same speed as non-ECC;

"It" meaning BitKeeper.

Quoted 9 lines

What does BitKeeper have to do with this conversation?

s/BitKeeper/any_app_which_has_integrity_checks/

Whether that app runs fast or not has nothing to do with ECC/non-ECC, right? And while whether that app runs fast or not may have something to do with other apps that you run along side of it, that's true for all apps, right? So why the focus on BitKeeper? Am I missing something?

Because your original post was "yeah, Bitkeeper is a memory hog but you can get really cheap non-ECC RAM so just stuff your system with crappy RAM and be happy." Doing so dedicates my system to running a small set of applications, which I am utterly uninterested in.

-hpa

Signature · 3 lines
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt
#9 lore raw
In reply to Larry McVoy
Quoted 7 lines · H. Peter Anvin
Quoted 1 line · Larry McVoy

What does BitKeeper have to do with this conversation?

Because your original post was "yeah, Bitkeeper is a memory hog but you can get really cheap non-ECC RAM so just stuff your system with crappy RAM and be happy." Doing so dedicates my system to running a small set of applications, which I am utterly uninterested in.

.. BitKeeper isn't a memory hog, the kernel is bloated. Over 100MB of
source last I checked. BitKeeper is incredibly good at _NOT_ being a memory hog, it uses the page cache as its memory pool. If things fit in the cache, they go fast, if they don't, they don't. BitKeeper is just like diff in that respect. If you think BitKeeper is a memory hog, then you must hate diff too. How about netscape? Don't run that either? Give me a break.

.. It's great that you aren't interested in running that set of small
applications, I'm sure the entire kernel list is happy to learn that.

.. You can get really cheap ECC ram as well, even if it were 2x as expensive,
that's still really cheap, less than 50 cents a megabyte, so what's your problem? Go get some ECC memory and be happy.

Signature · 2 lines
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 
#10 lore raw
In reply to Larry McVoy
Quoted 10 lines · Larry McVoy
Quoted 7 lines · H. Peter Anvin
Quoted 1 line · Larry McVoy

What does BitKeeper have to do with this conversation?

Because your original post was "yeah, Bitkeeper is a memory hog but you can get really cheap non-ECC RAM so just stuff your system with crappy RAM and be happy." Doing so dedicates my system to running a small set of applications, which I am utterly uninterested in.

Quoted 8 lines

.. BitKeeper isn't a memory hog, the kernel is bloated. Over 100MB of
source last I checked. BitKeeper is incredibly good at _NOT_ being a memory hog, it uses the page cache as its memory pool. If things fit in the cache, they go fast, if they don't, they don't. BitKeeper is just like diff in that respect. If you think BitKeeper is a memory hog, then you must hate diff too. How about netscape? Don't run that either? Give me a break.

I wasn't the one who said it, you did. I don't have any evidence either way.

Quoted 2 lines

.. It's great that you aren't interested in running that set of small
applications, I'm sure the entire kernel list is happy to learn that.

I believe the same is true for most people, with the major exceptions being the embedded systems and server farm people.

-hpa

Signature · 3 lines
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt
#11 lore raw
In reply to Larry McVoy
Quoted 7 lines · H. Peter Anvin
Quoted 3 lines · Larry McVoy
Quoted 3 lines

Because your original post was "yeah, Bitkeeper is a memory hog but you can get really cheap non-ECC RAM so just stuff your system with crappy RAM and be happy."

I wasn't the one who said it, you did. I don't have any evidence either way.

Err, Peter, it's starting to sound like you have some ax to grind that I don't know about. So I'll bow out of this conversation.

For the record, however, I never stated that BitKeeper is a memory hog, that's a conclusion you drew. Somehow, if I had said "look, for very little money you can fit all of the kernel source, revision history, and objects in memory", would you have translated that into "BitKeeper is a memory hog"? It seems that way.

BitKeeper has nothing to do with it, it's all about how big the data set is that the application is working on. BitKeeper is better than most applications, it mmaps the data and uses the page cache so that it doesn't cache it twice. Contrast that with most other apps, you'll see they have their own internal cache of data when they should just use mmap.

Anyway, I think we've beaten this to death, so let's move on.

Signature · 2 lines
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 
#12 lore raw
In reply to Larry McVoy
Quoted 7 lines · Larry McVoy
Quoted 4 lines · H. Peter Anvin
Quoted 3 lines · Larry McVoy
Quoted 3 lines

Because your original post was "yeah, Bitkeeper is a memory hog but you can get really cheap non-ECC RAM so just stuff your system with crappy RAM and be happy."

Quoted 6 lines
Quoted 2 lines

I wasn't the one who said it, you did. I don't have any evidence either way.

Err, Peter, it's starting to sound like you have some ax to grind that I don't know about. So I'll bow out of this conversation.

The only axe I have to grind was the obvious application myopia of your original post... "my application is the only one that matters." That's all.

-hpa

Signature · 3 lines
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt
#13 lore raw
In reply to Larry McVoy
Quoted 17 lines · H. Peter Anvin
Quoted 8 lines · Larry McVoy
Quoted 6 lines · H. Peter Anvin
Quoted 4 lines · Larry McVoy
Quoted 4 lines

Because your original post was "yeah, Bitkeeper is a memory hog but you can get really cheap non-ECC RAM so just stuff your system with crappy RAM and be happy."

Quoted 6 lines
Quoted 3 lines

I wasn't the one who said it, you did. I don't have any evidence either way.

Err, Peter, it's starting to sound like you have some ax to grind that I don't know about. So I'll bow out of this conversation.

Quoted 6 lines

The only axe I have to grind was the obvious application myopia of your original post... "my application is the only one that matters." That's all.

-hpa

<quote>

This is a 750Mhz K7 system with 1.5GB of memory in 3 512MB DIMMS. The DIMMS are not ECC, but we use BitKeeper here and it tells us when we have bad DIMMS.

Guess what the memory cost? $396.58 shipped to my door, second day air,
with a lifetime warranty. I got it at www.memory4less.com <http://www.memory4less.com> which I found using www.pricewatch.com <http://www.pricewatch.com>. I have no association with either of those places other than being a customer (i.e., this isn't advertising spam).

I'm burning it in right now, I wrote a little program which fills it with different test patterns and then reads them back to make sure they don't lose any bits. Seems to be working, it's done about 30 passes.

1.5GB for $400. Amazing. No more whining from you guys that BitKeeper uses too much memory [:-)]

$ hinv
Main memory size: 1535.9375 Mbytes
1 AuthenticAMD  processor
1 1.44M floppy drive
1 vga+ graphics device
1 keyboard
IDE devices:
   /dev/hda is a ST310211A, 9541MB w/512kB Cache, CHS=1216/255/63
SCSI devices:
   /dev/sda is a 3ware disk, model 3w-xxxx 74.541 GB
PCI bus devices:
   Host bridge: VIA Technologies VT 82C691 Apollo Pro (rev 2).
   PCI bridge: VIA Technologies VT 82C598 Apollo MVP3 AGP (rev 0).
   ISA bridge: VIA Technologies VT 82C686 Apollo Super (rev 34).
   IDE interface: VIA Technologies VT 82C586 Apollo IDE (rev 16).
   Host bridge: VIA Technologies VT 82C686 Apollo Super ACPI (rev 48).
   Ethernet controller: 3Com 3C905B 100bTX (rev 48).
   Ethernet controller: 3Com 3C905B 100bTX (rev 48).
   Ethernet controller: 3Com 3C905B 100bTX (rev 48).
   Ethernet controller: 3Com 3C905B 100bTX (rev 48).
   RAID storage controller: Unknown vendor Unknown device (rev 18).quote
   VGA compatible controller: Matrox Matrox G200 AGP (rev 1).
-- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm

</quote>

Lets move on now.

Signature · 5 lines
I'd rather listen to Newton than to Mundie [MS flunkie who made a speech on
the evil-ness of open source]. He may have been dead for almost three
hundred years, but despite that he stinks up the room less.

Linus