Skip to contentSkip to conversation

DRAM unreliable under specific access patern

DISCUSSION Pavel Machek 4 participants 9 messages Dec 24, 2014 – Dec 29, 2014

lore
Share & tools
lore
#0 lore raw

Hi!

It seems that it is easy to induce DRAM bit errors by doing repeated reads from adjacent memory cells on common hw. Details are at

https://www.ece.cmu.edu/~safari/pubs/kim-isca14.pdf

. Older memory modules seem to work better, and ECC should detect this. Paper has inner loop that should trigger this.

Workarounds seem to be at hardware level, and tricky, too.

Does anyone have implementation of detector? Any ideas how to work around it in software?

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#1 lore raw

Hi!

(I added original researches to the list).

I see you have FPGA-based detector, and probably PC based detector, too. Would it be possible to share sources of the PC based one?

Thanks,
Pavel

Quoted 16 lines · Pavel Machek

Hi!

It seems that it is easy to induce DRAM bit errors by doing repeated reads from adjacent memory cells on common hw. Details are at

https://www.ece.cmu.edu/~safari/pubs/kim-isca14.pdf

. Older memory modules seem to work better, and ECC should detect this. Paper has inner loop that should trigger this.

Workarounds seem to be at hardware level, and tricky, too.

Does anyone have implementation of detector? Any ideas how to work around it in software?

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#2 lore raw
In reply to Pavel Machek
Quoted 11 lines · Pavel Machek

Hi!

It seems that it is easy to induce DRAM bit errors by doing repeated reads from adjacent memory cells on common hw. Details are at

https://www.ece.cmu.edu/~safari/pubs/kim-isca14.pdf

. Older memory modules seem to work better, and ECC should detect this. Paper has inner loop that should trigger this.

Workarounds seem to be at hardware level, and tricky, too.

One mostly-effective solution would be to stop buying computers without ECC. Unfortunately, no one seems to sell non-server chips that can do ECC.

Quoted 4 lines

Does anyone have implementation of detector? Any ideas how to work around it in software?

Platform-dependent page coloring with very strict, and impossible to implement fully correctly, page allocation constraints?

--Andy

Quoted 4 lines

Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Signature · 2 lines
Andy Lutomirski
AMA Capital Management, LLC
#3 lore raw
Quoted 8 lines · Andy Lutomirski
Quoted 7 lines · Pavel Machek

Hi!

It seems that it is easy to induce DRAM bit errors by doing repeated reads from adjacent memory cells on common hw. Details are at

https://www.ece.cmu.edu/~safari/pubs/kim-isca14.pdf

Quoted 8 lines
Quoted 4 lines

. Older memory modules seem to work better, and ECC should detect this. Paper has inner loop that should trigger this.

Workarounds seem to be at hardware level, and tricky, too.

One mostly-effective solution would be to stop buying computers without ECC. Unfortunately, no one seems to sell non-server chips that can do ECC.

Or keep using old computers :-).

Quoted 6 lines
Quoted 3 lines

Does anyone have implementation of detector? Any ideas how to work around it in software?

Platform-dependent page coloring with very strict, and impossible to implement fully correctly, page allocation constraints?

This seems to be at cacheline level, not at page level, if I understand it correctly.

So the problem would is: I have something mapped read-only, and I can still cause bitflips in it.

Hmm. So it is pretty obviously a security problem, no need for java. Just do some bit flips in binary root is going to run, and it will crash for him. You can map binaries read-only, so you have enough access.

As far as I understand it, attached program could reproduce it on affected machines?
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

/* -*- linux-c -*-
*
* Try to trigger DRAM disturbance errors, as described in *
* https://www.ece.cmu.edu/~safari/pubs/kim-isca14.pdf
*
* Copyright 2014 Pavel Machek <[email protected]>, GPLv2+. */

#include <stdlib.h>
#include <stdio.h>
#include <string.h>

void disturb(char *where)

{
	unsigned int i;
	for (i=0; i<0x1000000; i++) {
		__asm__ __volatile__(
			"movl 0(%0), %%eax 		\n"	\
			"movl 64(%0), %%eax		\n"	\
			"clflush 0(%0)			\n"	\
			"clflush 64(%0)			\n"	\
			"mfence"
			:: "r" (where)
			: "eax"
			);
	}
}

int main(int argc, char *argv[])

{
	long size = 1*1024*1024;
	long i;
	unsigned char *mem;
	
	mem = malloc(size);
	memset(mem, 0xff, size);

	for (i=0; i<128; i+=4)
		disturb(mem+i);

	for (i=0; i<size; i++)
		if (mem[i] != 0xff)
			printf("At %lx, got %x\n", i, mem[i]);
}
#4 lore raw
Quoted 30 lines · Pavel Machek
Quoted 8 lines · Andy Lutomirski
Quoted 7 lines · Pavel Machek

Hi!

It seems that it is easy to induce DRAM bit errors by doing repeated reads from adjacent memory cells on common hw. Details are at

https://www.ece.cmu.edu/~safari/pubs/kim-isca14.pdf

Quoted 8 lines
Quoted 4 lines

. Older memory modules seem to work better, and ECC should detect this. Paper has inner loop that should trigger this.

Workarounds seem to be at hardware level, and tricky, too.

One mostly-effective solution would be to stop buying computers without ECC. Unfortunately, no one seems to sell non-server chips that can do ECC.

Or keep using old computers :-).

Quoted 6 lines
Quoted 3 lines

Does anyone have implementation of detector? Any ideas how to work around it in software?

Platform-dependent page coloring with very strict, and impossible to implement fully correctly, page allocation constraints?

This seems to be at cacheline level, not at page level, if I understand it correctly.

Quoted 7 lines

So the problem would is: I have something mapped read-only, and I can still cause bitflips in it.

Hmm. So it is pretty obviously a security problem, no need for java. Just do some bit flips in binary root is going to run, and it will crash for him. You can map binaries read-only, so you have enough access.

Right. So we're mostly screwed.

Quoted 3 lines

As far as I understand it, attached program could reproduce it on affected machines?

I thought that article suggested using addresses 8M (is that 8 megabytes?) apart for the two accesses.

--Andy

Quoted 4 lines

Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Signature · 2 lines
Andy Lutomirski
AMA Capital Management, LLC
#5 lore raw
Quoted 34 lines · Andy Lutomirski
Quoted 33 lines · Pavel Machek
Quoted 8 lines · Andy Lutomirski
Quoted 7 lines · Pavel Machek

Hi!

It seems that it is easy to induce DRAM bit errors by doing repeated reads from adjacent memory cells on common hw. Details are at

https://www.ece.cmu.edu/~safari/pubs/kim-isca14.pdf

Quoted 8 lines
Quoted 4 lines

. Older memory modules seem to work better, and ECC should detect this. Paper has inner loop that should trigger this.

Workarounds seem to be at hardware level, and tricky, too.

One mostly-effective solution would be to stop buying computers without ECC. Unfortunately, no one seems to sell non-server chips that can do ECC.

Or keep using old computers :-).

Quoted 6 lines
Quoted 3 lines

Does anyone have implementation of detector? Any ideas how to work around it in software?

Platform-dependent page coloring with very strict, and impossible to implement fully correctly, page allocation constraints?

This seems to be at cacheline level, not at page level, if I understand it correctly.

So the problem would is: I have something mapped read-only, and I can still cause bitflips in it.

Quoted 6 lines
Quoted 4 lines

Hmm. So it is pretty obviously a security problem, no need for java. Just do some bit flips in binary root is going to run, and it will crash for him. You can map binaries read-only, so you have enough access.

Right. So we're mostly screwed.

Well... We could periodically scrub (every few miliseconds) pages mapped to userspace. We might be able to do some magic and disallow cache flushes to userspace programs. We might be able to use performance metrics to detect heavy readers.

We might be able to reprogram DRAM controller to refresh more often.

Or we may switch to AMD systems as they seem to be less suspectible :-).
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#6 lore raw
Quoted 2 lines · Pavel Machek

Well... We could periodically scrub (every few miliseconds) pages mapped to userspace.

I.e. implement ECC in software. Would be extremely slow though.

Quoted 2 lines

We might be able to do some magic and disallow cache flushes to userspace programs.

My understanding is that cflush is not strictly necessary, it only makes the issue more likely to trigger.

If you modify the pattern so that it neraly fits into cacheline (but not really), you would be able to produce similar (if not the same) cache eviction pattern as if without explicit cflush. Right?

Signature · 2 lines
Jiri Kosina
SUSE Labs
#7 lore raw
Quoted 6 lines · Jiri Kosina
Quoted 2 lines · Pavel Machek

Well... We could periodically scrub (every few miliseconds) pages mapped to userspace.

I.e. implement ECC in software. Would be extremely slow though.

No, not really. If you read the cells that are about to go bad, you'll update them. Agreed on extremely slow.

Quoted 5 lines
Quoted 2 lines

We might be able to do some magic and disallow cache flushes to userspace programs.

My understanding is that cflush is not strictly necessary, it only makes the issue more likely to trigger.

Umm. Not really, AFAICT.

So, the memory can take "certain ammount" of "neighboring accesses". You need to do that ammount before next refresh.

Quoted 3 lines

If you modify the pattern so that it neraly fits into cacheline (but not really), you would be able to produce similar (if not the same) cache eviction pattern as if without explicit cflush. Right?

No, I don't think so.

Well.. you need to generate certain ammount of traffic on the address lines, and it corrupts "neighboring" cells. I wish I knew more about DRAM... If you'll read a cache line, you can't "break" it as reads refreshes it. You need to do few miliseconds worth of reads, AFAICT.

If you'll just keep reading cachelines, the cachelines you read will not be "neighboring" enough to the "target" cells you want to break.

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#8 lore raw
In reply to Pavel Machek

Hi Pavel,

Quoted 6 lines · Pavel Machek

Hi!

It seems that it is easy to induce DRAM bit errors by doing repeated reads from adjacent memory cells on common hw. Details are at

https://www.ece.cmu.edu/~safari/pubs/kim-isca14.pdf

Extremely interesting stuff. I've always wondered if such modules were *that* reliable given how picky they are about all timings.

Quoted 7 lines

. Older memory modules seem to work better, and ECC should detect this. Paper has inner loop that should trigger this.

Workarounds seem to be at hardware level, and tricky, too.

Does anyone have implementation of detector? Any ideas how to work around it in software?

Maybe reserve some memory "canary" that is periodically scanned and observe changes there. That will not tell you for sure that something has not been done, but it will tell you for sure that bits were flipped.

Also I'm wondering whether perf counters on certain CPUs could be used to detect the abnormal number of clflushes or even the memory access pattern (will not work in multi-socket environments if a user has one dedicated CPU though).

Thanks for sharing the link!
Willy