EDAC on arm64
Hi Rob,
Have you considered reviving the patch you posted previously for EDAC support (the atomic_scrub read/write test piece dependency)?
http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
Jon.
Quoted 4 lines · Jon Masters
Have you considered reviving the patch you posted previously for EDAC support (the
atomic_scrubread/write test piece dependency)?http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
Well, we'd need a way to handle the non-coherent DMA case and it's really not clear how to fix that.
Will
Quoted 8 lines · Will Deacon
Quoted 4 lines · Jon Masters
Have you considered reviving the patch you posted previously for EDAC support (the
atomic_scrubread/write test piece dependency)?http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
Well, we'd need a way to handle the non-coherent DMA case and it's really not clear how to fix that.
I agree, that's where the discussions stopped. Basically the EDAC memory writing is racy with any non-cacheable memory accesses (by CPU or device). The only way we could safely use this is only if all the devices are coherent *and* KVM is disabled. With KVM, guests may access the memory uncached, so we hit the same problem.
Signature · 1 line
Catalin
Quoted 7 lines · Catalin Marinas
Quoted 6 lines · Will Deacon
Quoted 4 lines · Jon Masters
Have you considered reviving the patch you posted previously for EDAC support (the
atomic_scrubread/write test piece dependency)?http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
Quoted 8 lines
Quoted 2 lines
Well, we'd need a way to handle the non-coherent DMA case and it's really not clear how to fix that.
I agree, that's where the discussions stopped. Basically the EDAC memory writing is racy with any non-cacheable memory accesses (by CPU or device). The only way we could safely use this is only if all the devices are coherent *and* KVM is disabled. With KVM, guests may access the memory uncached, so we hit the same problem.
Scrubbing only prevents repeated error reporting of correctable errors which only repeat on a cache miss. Perhaps we should just add an empty version that is a nop. At least then we can enable EDAC and be able to report errors. Otherwise, they may just be silently ignored (fine for correctable, but not so good to do with uncorrectable errors).
Anyway, I'll leave it to someone who still has this itch to sort out. My itch was cured.
Rob
Quoted 3 lines
--
Catalin
Quoted 17 lines · Rob Herring
Quoted 7 lines · Catalin Marinas
Quoted 6 lines · Will Deacon
Quoted 4 lines · Jon Masters
Have you considered reviving the patch you posted previously for EDAC support (the
atomic_scrubread/write test piece dependency)?http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
Quoted 8 lines
Quoted 2 lines
Well, we'd need a way to handle the non-coherent DMA case and it's really not clear how to fix that.
I agree, that's where the discussions stopped. Basically the EDAC memory writing is racy with any non-cacheable memory accesses (by CPU or device). The only way we could safely use this is only if all the devices are coherent *and* KVM is disabled. With KVM, guests may access the memory uncached, so we hit the same problem.
Quoted 5 lines
Scrubbing only prevents repeated error reporting of correctable errors which only repeat on a cache miss. Perhaps we should just add an empty version that is a nop. At least then we can enable EDAC and be able to report errors. Otherwise, they may just be silently ignored (fine for correctable, but not so good to do with uncorrectable errors).
This looks fine to me.
Quoted 1 line
Anyway, I'll leave it to someone who still has this itch to sort out.
Maybe Jon? ;)
Signature · 1 line
Catalin
Quoted 17 lines · Rob Herring
Quoted 7 lines · Catalin Marinas
Quoted 6 lines · Will Deacon
Quoted 4 lines · Jon Masters
Have you considered reviving the patch you posted previously for EDAC support (the
atomic_scrubread/write test piece dependency)?http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
Quoted 8 lines
Quoted 2 lines
Well, we'd need a way to handle the non-coherent DMA case and it's really not clear how to fix that.
I agree, that's where the discussions stopped. Basically the EDAC memory writing is racy with any non-cacheable memory accesses (by CPU or device). The only way we could safely use this is only if all the devices are coherent *and* KVM is disabled. With KVM, guests may access the memory uncached, so we hit the same problem.
Quoted 3 lines
Scrubbing only prevents repeated error reporting of correctable errors which only repeat on a cache miss. Perhaps we should just add an empty version that is a nop.
Can the error be cleared by a cache clean&invalidate by VA?
Signature · 1 line
Catalin
Quoted 18 lines · Catalin Marinas
Quoted 17 lines · Rob Herring
Quoted 15 lines · Catalin Marinas
Quoted 8 lines · Will Deacon
Quoted 4 lines · Jon Masters
Have you considered reviving the patch you posted previously for EDAC support (the
atomic_scrubread/write test piece dependency)?http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
Well, we'd need a way to handle the non-coherent DMA case and it's really not clear how to fix that.
I agree, that's where the discussions stopped. Basically the EDAC memory writing is racy with any non-cacheable memory accesses (by CPU or device). The only way we could safely use this is only if all the devices are coherent *and* KVM is disabled. With KVM, guests may access the memory uncached, so we hit the same problem.
Quoted 5 lines
Quoted 3 lines
Scrubbing only prevents repeated error reporting of correctable errors which only repeat on a cache miss. Perhaps we should just add an empty version that is a nop.
Can the error be cleared by a cache clean&invalidate by VA?
Yes, but only if the line is in fact dirty. If the line is clean, it will just make the error repeat on every access. You can't make the line dirty for the same reasons you've mentioned.
Rob
Quoted 19 lines · Rob Herring
Quoted 18 lines · Catalin Marinas
Quoted 17 lines · Rob Herring
Quoted 15 lines · Catalin Marinas
Quoted 8 lines · Will Deacon
Quoted 4 lines · Jon Masters
Have you considered reviving the patch you posted previously for EDAC support (the
atomic_scrubread/write test piece dependency)?http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
Well, we'd need a way to handle the non-coherent DMA case and it's really not clear how to fix that.
I agree, that's where the discussions stopped. Basically the EDAC memory writing is racy with any non-cacheable memory accesses (by CPU or device). The only way we could safely use this is only if all the devices are coherent *and* KVM is disabled. With KVM, guests may access the memory uncached, so we hit the same problem.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Scrubbing only prevents repeated error reporting of correctable errors which only repeat on a cache miss. Perhaps we should just add an empty version that is a nop.
Can the error be cleared by a cache clean&invalidate by VA?
Yes, but only if the line is in fact dirty. If the line is clean, it will just make the error repeat on every access.
So is the error reported even if the line is removed from the cache entirely via a (clean _and_) invalidate operation? Once removed from the cache, an access cannot hit the line as it's no longer there.
Signature · 1 line
Catalin
Quoted 24 lines · Catalin Marinas
Quoted 23 lines · Rob Herring
Quoted 22 lines · Catalin Marinas
Quoted 20 lines · Rob Herring
Quoted 7 lines · Catalin Marinas
Quoted 6 lines · Will Deacon
Quoted 4 lines · Jon Masters
Have you considered reviving the patch you posted previously for EDAC support (the
atomic_scrubread/write test piece dependency)?http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
Quoted 8 lines
Quoted 2 lines
Well, we'd need a way to handle the non-coherent DMA case and it's really not clear how to fix that.
I agree, that's where the discussions stopped. Basically the EDAC memory writing is racy with any non-cacheable memory accesses (by CPU or device). The only way we could safely use this is only if all the devices are coherent *and* KVM is disabled. With KVM, guests may access the memory uncached, so we hit the same problem.
Scrubbing only prevents repeated error reporting of correctable errors which only repeat on a cache miss. Perhaps we should just add an empty version that is a nop.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Can the error be cleared by a cache clean&invalidate by VA?
Yes, but only if the line is in fact dirty. If the line is clean, it will just make the error repeat on every access.
So is the error reported even if the line is removed from the cache entirely via a (clean _and_) invalidate operation? Once removed from the cache, an access cannot hit the line as it's no longer there.
But when there is a miss and you fetch the line from memory again, the error will be triggered again. Invalidating the cache would work would work if you are doing ECC checks in the caches, but I'm thinking more of the scenario where the memory controller is doing the checking. An error is going to be generated whenever the memory controller fetches a word (64-bits) with an error. The error could be induced either on the read in which case it is one time transient error or on the write in which case it will repeat until you write that location again. I don't think you typically know which case it is. Assuming that location is only read, then you will get errors reported each time there is a cache miss (or DMA reads the location).
BTW, some processors do ECC calc and checks in the caches and maintain that all the way to memory. Then the memory contents are protected in the caches and internal buses and you're not doing ECC twice (in the L2/L3 and memory ctrlr). Of course, you can't have non-coherent DMA in that case.
Rob
Quoted 3 lines
--
Catalin
Quoted 7 lines · On Monday 02 March 2015 14:58:41 Catalin Marinas wrote:
Quoted 6 lines · Will Deacon
Quoted 4 lines · Jon Masters
Have you considered reviving the patch you posted previously for EDAC support (the
atomic_scrubread/write test piece dependency)?http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
Quoted 8 lines
Quoted 2 lines
Well, we'd need a way to handle the non-coherent DMA case and it's really not clear how to fix that.
I agree, that's where the discussions stopped. Basically the EDAC memory writing is racy with any non-cacheable memory accesses (by CPU or device). The only way we could safely use this is only if all the devices are coherent *and* KVM is disabled. With KVM, guests may access the memory uncached, so we hit the same problem.
Is this a setting of the host, or does the guest always have this capability?
If a guest can influence the caching of a page it has access to, I can imagine all sorts of security problems with malicious guests regardless of EDAC.
Arnd
Quoted 11 lines · Arnd Bergmann
Quoted 10 lines · On Monday 02 March 2015 14:58:41 Catalin Marinas wrote:
Quoted 8 lines · Will Deacon
Quoted 4 lines · Jon Masters
Have you considered reviving the patch you posted previously for EDAC support (the
atomic_scrubread/write test piece dependency)?http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
Well, we'd need a way to handle the non-coherent DMA case and it's really not clear how to fix that.
Quoted 7 lines
Quoted 5 lines
I agree, that's where the discussions stopped. Basically the EDAC memory writing is racy with any non-cacheable memory accesses (by CPU or device). The only way we could safely use this is only if all the devices are coherent *and* KVM is disabled. With KVM, guests may access the memory uncached, so we hit the same problem.
Is this a setting of the host, or does the guest always have this capability?
The guest can always make it stricter than what the host set in stage 2 (i.e. from Normal Cacheable -> NonCacheable -> Device) but never in the other direction.
Quoted 3 lines
If a guest can influence the caching of a page it has access to, I can imagine all sorts of security problems with malicious guests regardless of EDAC.
Not as long as the host is aware of this. Basically it needs to flush the cache on a page when it is mapped into the guest address space (IPA) and flush it again when reading a page from guest.
Signature · 1 line
Catalin
Quoted 18 lines · On Monday 02 March 2015 22:25:16 Catalin Marinas wrote:
Quoted 17 lines · Arnd Bergmann
Quoted 15 lines · On Monday 02 March 2015 14:58:41 Catalin Marinas wrote:
Quoted 8 lines · Will Deacon
Quoted 4 lines · Jon Masters
Have you considered reviving the patch you posted previously for EDAC support (the
atomic_scrubread/write test piece dependency)?http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
Well, we'd need a way to handle the non-coherent DMA case and it's really not clear how to fix that.
I agree, that's where the discussions stopped. Basically the EDAC memory writing is racy with any non-cacheable memory accesses (by CPU or device). The only way we could safely use this is only if all the devices are coherent *and* KVM is disabled. With KVM, guests may access the memory uncached, so we hit the same problem.
Quoted 5 lines
Quoted 1 line
Is this a setting of the host, or does the guest always have this capability?
The guest can always make it stricter than what the host set in stage 2 (i.e. from Normal Cacheable -> NonCacheable -> Device) but never in the other direction.
Do you have an idea what the purpose of this is? Why would a guest even want to mark pages as noncachable that are mapped into the host as cachable and that might have dirty cache lines?
Quoted 7 lines
Quoted 3 lines
If a guest can influence the caching of a page it has access to, I can imagine all sorts of security problems with malicious guests regardless of EDAC.
Not as long as the host is aware of this. Basically it needs to flush the cache on a page when it is mapped into the guest address space (IPA) and flush it again when reading a page from guest.
You have to flush and invalidate the cache line, but of course nobody wants to do that because it totally destroys performance.
Arnd
Quoted 21 lines · Arnd Bergmann
Quoted 20 lines · On Monday 02 March 2015 22:25:16 Catalin Marinas wrote:
Quoted 18 lines · Arnd Bergmann
Quoted 7 lines · On Monday 02 March 2015 14:58:41 Catalin Marinas wrote:
Quoted 6 lines · Will Deacon
Quoted 4 lines · Jon Masters
Have you considered reviving the patch you posted previously for EDAC support (the
atomic_scrubread/write test piece dependency)?http://lists.infradead.org/pipermail/linux-arm-kernel/2014-April/249039.html
Quoted 8 lines
Quoted 2 lines
Well, we'd need a way to handle the non-coherent DMA case and it's really not clear how to fix that.
I agree, that's where the discussions stopped. Basically the EDAC memory writing is racy with any non-cacheable memory accesses (by CPU or device). The only way we could safely use this is only if all the devices are coherent *and* KVM is disabled. With KVM, guests may access the memory uncached, so we hit the same problem.
Is this a setting of the host, or does the guest always have this capability?
Quoted 7 lines
Quoted 3 lines
The guest can always make it stricter than what the host set in stage 2 (i.e. from Normal Cacheable -> NonCacheable -> Device) but never in the other direction.
Do you have an idea what the purpose of this is? Why would a guest even want to mark pages as noncachable that are mapped into the host as cachable and that might have dirty cache lines?
The stage 1 / stage 2 attributes combination works such that the hypervisor can impose more stricter attributes or none at all (in which case it is up to the guest to decide what it needs). So for example devices mapped into the guest address space (e.g. the GIC) are marked as Device memory in stage 2 so that the guest could never map them as Normal Cacheable memory (with some bad consequences).
The other direction is that the guest may want to create a stricter mapping than what the host wants. A possible reason is some frame buffer or anything else where the guest assumes that by creating a non-cacheable mapping it won't need to do cache maintenance. That's why when KVM maps a page into the guest address space, it flushes the cache so there are no dirty lines. Since the host would not write to such page again, it won't dirty the cache (and if the guest does, it needs to deal with it itself).
There are some scenarios where this does not work well: a virtual frame buffer emulated by Qemu where the guest thinks it is non-cacheable and Qemu maps it as cacheable. The only sane solution here is to tell the guest that the (virtual) frame buffer device is DMA coherent and that it should use a cacheable mapping. But I don't think the host should somehow (transparently) upgrade the cacheability that the guest thinks it has.
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
If a guest can influence the caching of a page it has access to, I can imagine all sorts of security problems with malicious guests regardless of EDAC.
Not as long as the host is aware of this. Basically it needs to flush the cache on a page when it is mapped into the guest address space (IPA) and flush it again when reading a page from guest.
You have to flush and invalidate the cache line, but of course nobody wants to do that because it totally destroys performance.
There are two cases when the cache needs flushing: (1) when a page is mapped into the guest address space (done lazily via the page faulting mechanism) and (2) when the host reads a page already mapped in the guest address space (e.g. swapping out). They indeed take some time but none of them are on a critical path.
(and maybe at some point we'll get fully transparent caches on ARM as well, so we don't have to worry about cache maintenance)
Signature · 1 line
Catalin