Skip to contentSkip to conversation

whymb: fix description in DMA coherence

PATCH In review Hao Lee 2 participants 11 messages Apr 17, 2022 – May 7, 2022

lore
Share & tools
lore

Hao Lee

author
#0 lore raw

PATCH 1/1 whymb: fix description in DMA coherence

CPU caches need to be flushed before DMA operations to avoid data overwriting.

---
 appendix/whymb/whymemorybarriers.tex | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
appendix/whymb/whymemorybarriers.tex +1 -1
index b49d5fe8..4ec27962 100644
@@ -1640,7 +1640,7 @@ future such problems:
 	in any DMA buffer before presenting that buffer to the
 	I/O device.
 	Similarly, you need to flush the CPU caches of any location
-	in any DMA buffer after DMA to that buffer completes.
+	in any DMA buffer before DMA to that buffer completes.
 	And even then, you need to be \emph{very} careful to avoid
 	pointer bugs, as even a misplaced read to an input buffer
 	can result in corrupting the data input!
Signature · 1 line
2.21.0
#1 lore raw
Quoted 13 lines · Hao Lee

CPU caches need to be flushed before DMA operations to avoid data overwriting.

---
 appendix/whymb/whymemorybarriers.tex | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
appendix/whymb/whymemorybarriers.tex +0 -0
index b49d5fe8..4ec27962 100644

appendix/whymb/whymemorybarriers.tex future such problems: · line 1641

Quoted 2 lines
appendix/whymb/whymemorybarriers.tex +0 -0
@@ -1640,7 +1640,7 @@ future such problems:
 	in any DMA buffer before presenting that buffer to the
 	I/O device.

The paragraph above is before the I/O starts. Any changes to the buffer need to be flushed to memory so that the I/O device's DMA actually sees them.

The next paragraph is the other end of I/O request, after it completes. So we cannot have "before". But maybe we need "invalidate" instead of "flush". With a note that itcan be more efficient to both flush and invalidate prior to presenting the DMA buffer to the device.

The point is that any pre-DMA data in the CPU caches must be gotten rid of in order for the CPUs to see the data that was actually DMA'ed into memory.

appendix/whymb/whymemorybarriers.tex future such problems: · line 1646

Quoted 6 lines
appendix/whymb/whymemorybarriers.tex +1 -1
 	Similarly, you need to flush the CPU caches of any location
-	in any DMA buffer after DMA to that buffer completes.
+	in any DMA buffer before DMA to that buffer completes.
 	And even then, you need to be \emph{very} careful to avoid
 	pointer bugs, as even a misplaced read to an input buffer
 	can result in corrupting the data input!

And this last clause could just wordsmithed a bit.

Thoughts?

Thanx, Paul

author
#2 lore raw
Quoted 21 lines · Paul E. McKenney
Quoted 13 lines · Hao Lee

CPU caches need to be flushed before DMA operations to avoid data overwriting.

---
 appendix/whymb/whymemorybarriers.tex | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
appendix/whymb/whymemorybarriers.tex +0 -0
index b49d5fe8..4ec27962 100644

appendix/whymb/whymemorybarriers.tex future such problems: · line 1641

Quoted 2 lines
appendix/whymb/whymemorybarriers.tex +0 -0
@@ -1640,7 +1640,7 @@ future such problems:
 	in any DMA buffer before presenting that buffer to the
 	I/O device.

The paragraph above is before the I/O starts. Any changes to the buffer need to be flushed to memory so that the I/O device's DMA actually sees them.

Quoted 3 lines

The next paragraph is the other end of I/O request, after it completes. So we cannot have "before". But maybe we need "invalidate" instead of "flush".

Got it!

Quoted 6 lines

With a note that itcan be more efficient to both flush and invalidate prior to presenting the DMA buffer to the device.

The point is that any pre-DMA data in the CPU caches must be gotten rid of in order for the CPUs to see the data that was actually DMA'ed into memory.

Great explanation! Now I know we need something like WBINVD instruction for the former case and INVD instruction for the latter.

Quoted 2 lines

appendix/whymb/whymemorybarriers.tex future such problems: · line 1642

Quoted 2 lines
appendix/whymb/whymemorybarriers.tex +0 -0
 	Similarly, you need to flush the CPU caches of any location

Should we change the word "flush" to "invalidate" to differentiate these two opposite DMA operations?

Quoted 7 lines

appendix/whymb/whymemorybarriers.tex future such problems: · line 1646

Quoted 5 lines
appendix/whymb/whymemorybarriers.tex +1 -1
-	in any DMA buffer after DMA to that buffer completes.
+	in any DMA buffer before DMA to that buffer completes.
 	And even then, you need to be \emph{very} careful to avoid
 	pointer bugs, as even a misplaced read to an input buffer
 	can result in corrupting the data input!

And this last clause could just wordsmithed a bit.

Maybe it's not quite rigorous?

If I understand correctly, a misplaced read will load the data from input buffer area to cache line. If the cpu doesn't write to this cache line, I think this case can only result in that the cpu reads stale data in subsequent reads but won't corrupt the input buffer given that this cache line is clean and won't writeback to memory.

Thanks,
Hao Lee

Quoted 4 lines

Thoughts?

Thanx, Paul

#3 lore raw
Quoted 31 lines · Hao Lee
Quoted 21 lines · Paul E. McKenney
Quoted 13 lines · Hao Lee

CPU caches need to be flushed before DMA operations to avoid data overwriting.

---
 appendix/whymb/whymemorybarriers.tex | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
appendix/whymb/whymemorybarriers.tex +0 -0
index b49d5fe8..4ec27962 100644

appendix/whymb/whymemorybarriers.tex future such problems: · line 1641

Quoted 2 lines
appendix/whymb/whymemorybarriers.tex +0 -0
@@ -1640,7 +1640,7 @@ future such problems:
 	in any DMA buffer before presenting that buffer to the
 	I/O device.

The paragraph above is before the I/O starts. Any changes to the buffer need to be flushed to memory so that the I/O device's DMA actually sees them.

Quoted 3 lines

The next paragraph is the other end of I/O request, after it completes. So we cannot have "before". But maybe we need "invalidate" instead of "flush".

Got it!

Quoted 3 lines

With a note that itcan be more efficient to both flush and invalidate prior to presenting the DMA buffer to the device.

Quoted 6 lines
Quoted 3 lines

The point is that any pre-DMA data in the CPU caches must be gotten rid of in order for the CPUs to see the data that was actually DMA'ed into memory.

Great explanation! Now I know we need something like WBINVD instruction for the former case and INVD instruction for the latter.

Glad that it helped!

Quoted 4 lines
Quoted 1 line

appendix/whymb/whymemorybarriers.tex future such problems: · line 1642

Quoted 1 line
appendix/whymb/whymemorybarriers.tex +0 -0
 	Similarly, you need to flush the CPU caches of any location

Should we change the word "flush" to "invalidate" to differentiate these two opposite DMA operations?

If after the DMA, we need to care about the difference, hasn't the CPU incorrectly written to part of the DMA buffer during the DMA operation? If you are thinking of some other scenario, could you please list the events leading up to that scenrio?

Quoted 8 lines
Quoted 7 lines

appendix/whymb/whymemorybarriers.tex future such problems: · line 1646

Quoted 5 lines
appendix/whymb/whymemorybarriers.tex +1 -1
-	in any DMA buffer after DMA to that buffer completes.
+	in any DMA buffer before DMA to that buffer completes.
 	And even then, you need to be \emph{very} careful to avoid
 	pointer bugs, as even a misplaced read to an input buffer
 	can result in corrupting the data input!

And this last clause could just wordsmithed a bit.

Quoted 7 lines

Maybe it's not quite rigorous?

If I understand correctly, a misplaced read will load the data from input buffer area to cache line. If the cpu doesn't write to this cache line, I think this case can only result in that the cpu reads stale data in subsequent reads but won't corrupt the input buffer given that this cache line is clean and won't writeback to memory.

If the user code that accesses the buffer runs on this same CPU, it will look to that CPU as if the DMA buffer was corrupted, correct? After all, that user code will see the value loaded by the CPU, not the value actually DMAed into memory.

Hmmm... It might be necessary to spell out these DMA-corruption scenarios. Thoughts?

Thanx, Paul

author
#4 lore raw
Quoted 41 lines · Paul E. McKenney
Quoted 31 lines · Hao Lee
Quoted 21 lines · Paul E. McKenney
Quoted 13 lines · Hao Lee

CPU caches need to be flushed before DMA operations to avoid data overwriting.

---
 appendix/whymb/whymemorybarriers.tex | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
appendix/whymb/whymemorybarriers.tex +0 -0
index b49d5fe8..4ec27962 100644

appendix/whymb/whymemorybarriers.tex future such problems: · line 1641

Quoted 2 lines
appendix/whymb/whymemorybarriers.tex +0 -0
@@ -1640,7 +1640,7 @@ future such problems:
 	in any DMA buffer before presenting that buffer to the
 	I/O device.

The paragraph above is before the I/O starts. Any changes to the buffer need to be flushed to memory so that the I/O device's DMA actually sees them.

Quoted 3 lines

The next paragraph is the other end of I/O request, after it completes. So we cannot have "before". But maybe we need "invalidate" instead of "flush".

Got it!

Quoted 3 lines

With a note that itcan be more efficient to both flush and invalidate prior to presenting the DMA buffer to the device.

Quoted 6 lines
Quoted 3 lines

The point is that any pre-DMA data in the CPU caches must be gotten rid of in order for the CPUs to see the data that was actually DMA'ed into memory.

Great explanation! Now I know we need something like WBINVD instruction for the former case and INVD instruction for the latter.

Glad that it helped!

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

appendix/whymb/whymemorybarriers.tex future such problems: · line 1642

Quoted 1 line
appendix/whymb/whymemorybarriers.tex +0 -0
 	Similarly, you need to flush the CPU caches of any location

Should we change the word "flush" to "invalidate" to differentiate these two opposite DMA operations?

If after the DMA, we need to care about the difference, hasn't the CPU incorrectly written to part of the DMA buffer during the DMA operation?

Sorry, I can't understand this passage... Maybe I didn't make myself clear. I find a great example in arch/powerpc/include/asm/cacheflush.h

Before DMA provides output buffer to devices, we should call flush_dcache_range or clean_dcache_range to let the devices see the latest data generated by the CPU. After DMA writes data into the input buffer, we should use invalidate_dcache_range to invalidate the CPU cache so the CPU can read the latest data written by devices.

Should we change the word "flush" in the following sentence to "invalidate"?

Similarly, you need to flush the CPU caches of any location in any DMA buffer after DMA to that buffer completes.

Quoted 13 lines

If you are thinking of some other scenario, could you please list the events leading up to that scenrio?

Quoted 10 lines
Quoted 7 lines

appendix/whymb/whymemorybarriers.tex future such problems: · line 1646

Quoted 5 lines
appendix/whymb/whymemorybarriers.tex +1 -1
-	in any DMA buffer after DMA to that buffer completes.
+	in any DMA buffer before DMA to that buffer completes.
 	And even then, you need to be \emph{very} careful to avoid
 	pointer bugs, as even a misplaced read to an input buffer
 	can result in corrupting the data input!

And this last clause could just wordsmithed a bit.

Maybe it's not quite rigorous?

Quoted 8 lines
Quoted 5 lines

If I understand correctly, a misplaced read will load the data from input buffer area to cache line. If the cpu doesn't write to this cache line, I think this case can only result in that the cpu reads stale data in subsequent reads but won't corrupt the input buffer given that this cache line is clean and won't writeback to memory.

If the user code that accesses the buffer runs on this same CPU, it will look to that CPU as if the DMA buffer was corrupted, correct?

Ah, this indeed makes sense!

Quoted 6 lines

After
all, that user code will see the value loaded by the CPU, not the value actually DMAed into memory.

Hmmm... It might be necessary to spell out these DMA-corruption scenarios. Thoughts?

Yes. I think spelling out these scenarios should make this passage easier to understand.

Thanks,
Hao Lee

Quoted 2 lines

Thanx, Paul

author
#5 lore raw
Quoted 60 lines · Hao Lee
Quoted 41 lines · Paul E. McKenney
Quoted 31 lines · Hao Lee
Quoted 21 lines · Paul E. McKenney
Quoted 13 lines · Hao Lee

CPU caches need to be flushed before DMA operations to avoid data overwriting.

---
 appendix/whymb/whymemorybarriers.tex | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
appendix/whymb/whymemorybarriers.tex +0 -0
index b49d5fe8..4ec27962 100644

appendix/whymb/whymemorybarriers.tex future such problems: · line 1641

Quoted 2 lines
appendix/whymb/whymemorybarriers.tex +0 -0
@@ -1640,7 +1640,7 @@ future such problems:
        in any DMA buffer before presenting that buffer to the
        I/O device.

The paragraph above is before the I/O starts. Any changes to the buffer need to be flushed to memory so that the I/O device's DMA actually sees them.

Quoted 3 lines

The next paragraph is the other end of I/O request, after it completes. So we cannot have "before". But maybe we need "invalidate" instead of "flush".

Got it!

Quoted 3 lines

With a note that itcan be more efficient to both flush and invalidate prior to presenting the DMA buffer to the device.

Quoted 6 lines
Quoted 3 lines

The point is that any pre-DMA data in the CPU caches must be gotten rid of in order for the CPUs to see the data that was actually DMA'ed into memory.

Great explanation! Now I know we need something like WBINVD instruction for the former case and INVD instruction for the latter.

Glad that it helped!

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

appendix/whymb/whymemorybarriers.tex future such problems: · line 1642

Quoted 1 line
appendix/whymb/whymemorybarriers.tex +0 -0
        Similarly, you need to flush the CPU caches of any location

Should we change the word "flush" to "invalidate" to differentiate these two opposite DMA operations?

If after the DMA, we need to care about the difference, hasn't the CPU incorrectly written to part of the DMA buffer during the DMA operation?

Sorry, I can't understand this passage... Maybe I didn't make myself clear. I find a great example in arch/powerpc/include/asm/cacheflush.h

Before DMA provides output buffer to devices, we should call flush_dcache_range or clean_dcache_range to let the devices see the latest data generated by the CPU. After DMA writes data into the input buffer, we should use invalidate_dcache_range to invalidate the CPU cache so the CPU can read the latest data written by devices.

Quoted 5 lines

Should we change the word "flush" in the following sentence to "invalidate"?

Similarly, you need to flush the CPU caches of any location in any DMA buffer after DMA to that buffer completes.

Hello,

My above reply may be missed :)

Thanks,
Hao Lee

Quoted 40 lines
Quoted 14 lines

If you are thinking of some other scenario, could you please list the events leading up to that scenrio?

Quoted 10 lines
Quoted 7 lines

appendix/whymb/whymemorybarriers.tex future such problems: · line 1646

Quoted 5 lines
appendix/whymb/whymemorybarriers.tex +1 -1
-       in any DMA buffer after DMA to that buffer completes.
+       in any DMA buffer before DMA to that buffer completes.
        And even then, you need to be \emph{very} careful to avoid
        pointer bugs, as even a misplaced read to an input buffer
        can result in corrupting the data input!

And this last clause could just wordsmithed a bit.

Maybe it's not quite rigorous?

Quoted 8 lines
Quoted 5 lines

If I understand correctly, a misplaced read will load the data from input buffer area to cache line. If the cpu doesn't write to this cache line, I think this case can only result in that the cpu reads stale data in subsequent reads but won't corrupt the input buffer given that this cache line is clean and won't writeback to memory.

If the user code that accesses the buffer runs on this same CPU, it will look to that CPU as if the DMA buffer was corrupted, correct?

Ah, this indeed makes sense!

Quoted 6 lines

After
all, that user code will see the value loaded by the CPU, not the value actually DMAed into memory.

Hmmm... It might be necessary to spell out these DMA-corruption scenarios. Thoughts?

Yes. I think spelling out these scenarios should make this passage easier to understand.

Thanks,
Hao Lee

Quoted 2 lines

Thanx, Paul

#6 lore raw
Quoted 63 lines · Hao Lee
Quoted 62 lines · Hao Lee
Quoted 41 lines · Paul E. McKenney
Quoted 31 lines · Hao Lee
Quoted 21 lines · Paul E. McKenney
Quoted 13 lines · Hao Lee

CPU caches need to be flushed before DMA operations to avoid data overwriting.

---
 appendix/whymb/whymemorybarriers.tex | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
appendix/whymb/whymemorybarriers.tex +0 -0
index b49d5fe8..4ec27962 100644

appendix/whymb/whymemorybarriers.tex future such problems: · line 1641

Quoted 2 lines
appendix/whymb/whymemorybarriers.tex +0 -0
@@ -1640,7 +1640,7 @@ future such problems:
        in any DMA buffer before presenting that buffer to the
        I/O device.

The paragraph above is before the I/O starts. Any changes to the buffer need to be flushed to memory so that the I/O device's DMA actually sees them.

Quoted 3 lines

The next paragraph is the other end of I/O request, after it completes. So we cannot have "before". But maybe we need "invalidate" instead of "flush".

Got it!

Quoted 3 lines

With a note that itcan be more efficient to both flush and invalidate prior to presenting the DMA buffer to the device.

Quoted 6 lines
Quoted 3 lines

The point is that any pre-DMA data in the CPU caches must be gotten rid of in order for the CPUs to see the data that was actually DMA'ed into memory.

Great explanation! Now I know we need something like WBINVD instruction for the former case and INVD instruction for the latter.

Glad that it helped!

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

appendix/whymb/whymemorybarriers.tex future such problems: · line 1642

Quoted 1 line
appendix/whymb/whymemorybarriers.tex +0 -0
        Similarly, you need to flush the CPU caches of any location

Should we change the word "flush" to "invalidate" to differentiate these two opposite DMA operations?

If after the DMA, we need to care about the difference, hasn't the CPU incorrectly written to part of the DMA buffer during the DMA operation?

Sorry, I can't understand this passage... Maybe I didn't make myself clear. I find a great example in arch/powerpc/include/asm/cacheflush.h

Before DMA provides output buffer to devices, we should call flush_dcache_range or clean_dcache_range to let the devices see the latest data generated by the CPU. After DMA writes data into the input buffer, we should use invalidate_dcache_range to invalidate the CPU cache so the CPU can read the latest data written by devices.

Should we change the word "flush" in the following sentence to "invalidate"?

Quoted 7 lines
Quoted 3 lines

Similarly, you need to flush the CPU caches of any location in any DMA buffer after DMA to that buffer completes.

Hello,

My above reply may be missed :)

No, not missed, just slowness on my part. For whatever reason, the past few months have seen unusually high levels of activity for Linux-kernel RCU [1]. Plus I am presenting at the LSF/MM summit next week.

But I will get to this, I promise!

In the meantime, one of the things that needs to be done to Appendix C to bring it up to chapter status is to provide a state table for each of the sequences of events, then converting the enumerated steps to paragraphs. See for example Table 15.1 and the corresponding discussion. Would you be interested in constructing such tables? That would speed things up!

Thanx, Paul

[1] https://docs.google.com/document/d/1GCdQC8SDbb54W1shjEXqGZ0Rq8a6kIeYutdSIajfpLA/edit?usp=sharing

Quoted 43 lines

Thanks,
Hao Lee

Quoted 40 lines
Quoted 14 lines

If you are thinking of some other scenario, could you please list the events leading up to that scenrio?

Quoted 10 lines
Quoted 7 lines

appendix/whymb/whymemorybarriers.tex future such problems: · line 1646

Quoted 5 lines
appendix/whymb/whymemorybarriers.tex +1 -1
-       in any DMA buffer after DMA to that buffer completes.
+       in any DMA buffer before DMA to that buffer completes.
        And even then, you need to be \emph{very} careful to avoid
        pointer bugs, as even a misplaced read to an input buffer
        can result in corrupting the data input!

And this last clause could just wordsmithed a bit.

Maybe it's not quite rigorous?

Quoted 8 lines
Quoted 5 lines

If I understand correctly, a misplaced read will load the data from input buffer area to cache line. If the cpu doesn't write to this cache line, I think this case can only result in that the cpu reads stale data in subsequent reads but won't corrupt the input buffer given that this cache line is clean and won't writeback to memory.

If the user code that accesses the buffer runs on this same CPU, it will look to that CPU as if the DMA buffer was corrupted, correct?

Ah, this indeed makes sense!

Quoted 6 lines

After
all, that user code will see the value loaded by the CPU, not the value actually DMAed into memory.

Hmmm... It might be necessary to spell out these DMA-corruption scenarios. Thoughts?

Yes. I think spelling out these scenarios should make this passage easier to understand.

Thanks,
Hao Lee

Quoted 2 lines

Thanx, Paul

author
#7 lore raw
Quoted 71 lines · Paul E. McKenney
Quoted 69 lines · Hao Lee
Quoted 60 lines · Hao Lee
Quoted 41 lines · Paul E. McKenney
Quoted 31 lines · Hao Lee
Quoted 21 lines · Paul E. McKenney
Quoted 13 lines · Hao Lee

CPU caches need to be flushed before DMA operations to avoid data overwriting.

---
 appendix/whymb/whymemorybarriers.tex | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
appendix/whymb/whymemorybarriers.tex +0 -0
index b49d5fe8..4ec27962 100644

appendix/whymb/whymemorybarriers.tex future such problems: · line 1641

Quoted 2 lines
appendix/whymb/whymemorybarriers.tex +0 -0
@@ -1640,7 +1640,7 @@ future such problems:
        in any DMA buffer before presenting that buffer to the
        I/O device.

The paragraph above is before the I/O starts. Any changes to the buffer need to be flushed to memory so that the I/O device's DMA actually sees them.

Quoted 3 lines

The next paragraph is the other end of I/O request, after it completes. So we cannot have "before". But maybe we need "invalidate" instead of "flush".

Got it!

Quoted 3 lines

With a note that itcan be more efficient to both flush and invalidate prior to presenting the DMA buffer to the device.

Quoted 6 lines
Quoted 3 lines

The point is that any pre-DMA data in the CPU caches must be gotten rid of in order for the CPUs to see the data that was actually DMA'ed into memory.

Great explanation! Now I know we need something like WBINVD instruction for the former case and INVD instruction for the latter.

Glad that it helped!

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

appendix/whymb/whymemorybarriers.tex future such problems: · line 1642

Quoted 1 line
appendix/whymb/whymemorybarriers.tex +0 -0
        Similarly, you need to flush the CPU caches of any location

Should we change the word "flush" to "invalidate" to differentiate these two opposite DMA operations?

If after the DMA, we need to care about the difference, hasn't the CPU incorrectly written to part of the DMA buffer during the DMA operation?

Sorry, I can't understand this passage... Maybe I didn't make myself clear. I find a great example in arch/powerpc/include/asm/cacheflush.h

Before DMA provides output buffer to devices, we should call flush_dcache_range or clean_dcache_range to let the devices see the latest data generated by the CPU. After DMA writes data into the input buffer, we should use invalidate_dcache_range to invalidate the CPU cache so the CPU can read the latest data written by devices.

Quoted 5 lines

Should we change the word "flush" in the following sentence to "invalidate"?

Similarly, you need to flush the CPU caches of any location in any DMA buffer after DMA to that buffer completes.

Hello,

Quoted 7 lines
Quoted 1 line

My above reply may be missed :)

No, not missed, just slowness on my part. For whatever reason, the past few months have seen unusually high levels of activity for Linux-kernel RCU [1]. Plus I am presenting at the LSF/MM summit next week.

But I will get to this, I promise!

Oh, thanks! Take your time.

Quoted 6 lines

In the meantime, one of the things that needs to be done to Appendix C to bring it up to chapter status is to provide a state table for each of the sequences of events, then converting the enumerated steps to paragraphs. See for example Table 15.1 and the corresponding discussion. Would you be interested in constructing such tables? That would speed things up!

No problem! I'm glad I can contribute to this book!

Thanks,
Hao Lee

Quoted 48 lines

Thanx, Paul

[1] https://docs.google.com/document/d/1GCdQC8SDbb54W1shjEXqGZ0Rq8a6kIeYutdSIajfpLA/edit?usp=sharing

Quoted 43 lines

Thanks,
Hao Lee

Quoted 40 lines
Quoted 14 lines

If you are thinking of some other scenario, could you please list the events leading up to that scenrio?

Quoted 10 lines
Quoted 7 lines

appendix/whymb/whymemorybarriers.tex future such problems: · line 1646

Quoted 5 lines
appendix/whymb/whymemorybarriers.tex +1 -1
-       in any DMA buffer after DMA to that buffer completes.
+       in any DMA buffer before DMA to that buffer completes.
        And even then, you need to be \emph{very} careful to avoid
        pointer bugs, as even a misplaced read to an input buffer
        can result in corrupting the data input!

And this last clause could just wordsmithed a bit.

Maybe it's not quite rigorous?

Quoted 8 lines
Quoted 5 lines

If I understand correctly, a misplaced read will load the data from input buffer area to cache line. If the cpu doesn't write to this cache line, I think this case can only result in that the cpu reads stale data in subsequent reads but won't corrupt the input buffer given that this cache line is clean and won't writeback to memory.

If the user code that accesses the buffer runs on this same CPU, it will look to that CPU as if the DMA buffer was corrupted, correct?

Ah, this indeed makes sense!

Quoted 6 lines

After
all, that user code will see the value loaded by the CPU, not the value actually DMAed into memory.

Hmmm... It might be necessary to spell out these DMA-corruption scenarios. Thoughts?

Yes. I think spelling out these scenarios should make this passage easier to understand.

Thanks,
Hao Lee

Quoted 2 lines

Thanx, Paul

#8 lore raw
Quoted 83 lines · Hao Lee
Quoted 71 lines · Paul E. McKenney
Quoted 69 lines · Hao Lee
Quoted 60 lines · Hao Lee
Quoted 41 lines · Paul E. McKenney
Quoted 31 lines · Hao Lee
Quoted 21 lines · Paul E. McKenney
Quoted 13 lines · Hao Lee

CPU caches need to be flushed before DMA operations to avoid data overwriting.

---
 appendix/whymb/whymemorybarriers.tex | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
appendix/whymb/whymemorybarriers.tex +0 -0
index b49d5fe8..4ec27962 100644

appendix/whymb/whymemorybarriers.tex future such problems: · line 1641

Quoted 2 lines
appendix/whymb/whymemorybarriers.tex +0 -0
@@ -1640,7 +1640,7 @@ future such problems:
        in any DMA buffer before presenting that buffer to the
        I/O device.

The paragraph above is before the I/O starts. Any changes to the buffer need to be flushed to memory so that the I/O device's DMA actually sees them.

Quoted 3 lines

The next paragraph is the other end of I/O request, after it completes. So we cannot have "before". But maybe we need "invalidate" instead of "flush".

Got it!

Quoted 3 lines

With a note that itcan be more efficient to both flush and invalidate prior to presenting the DMA buffer to the device.

Quoted 6 lines
Quoted 3 lines

The point is that any pre-DMA data in the CPU caches must be gotten rid of in order for the CPUs to see the data that was actually DMA'ed into memory.

Great explanation! Now I know we need something like WBINVD instruction for the former case and INVD instruction for the latter.

Glad that it helped!

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

appendix/whymb/whymemorybarriers.tex future such problems: · line 1642

Quoted 1 line
appendix/whymb/whymemorybarriers.tex +0 -0
        Similarly, you need to flush the CPU caches of any location

Should we change the word "flush" to "invalidate" to differentiate these two opposite DMA operations?

If after the DMA, we need to care about the difference, hasn't the CPU incorrectly written to part of the DMA buffer during the DMA operation?

Sorry, I can't understand this passage... Maybe I didn't make myself clear. I find a great example in arch/powerpc/include/asm/cacheflush.h

Before DMA provides output buffer to devices, we should call flush_dcache_range or clean_dcache_range to let the devices see the latest data generated by the CPU. After DMA writes data into the input buffer, we should use invalidate_dcache_range to invalidate the CPU cache so the CPU can read the latest data written by devices.

Quoted 5 lines

Should we change the word "flush" in the following sentence to "invalidate"?

Similarly, you need to flush the CPU caches of any location in any DMA buffer after DMA to that buffer completes.

Hello,

Quoted 7 lines
Quoted 1 line

My above reply may be missed :)

No, not missed, just slowness on my part. For whatever reason, the past few months have seen unusually high levels of activity for Linux-kernel RCU [1]. Plus I am presenting at the LSF/MM summit next week.

But I will get to this, I promise!

Oh, thanks! Take your time.

Quoted 1 line
Quoted 7 lines
Quoted 5 lines

In the meantime, one of the things that needs to be done to Appendix C to bring it up to chapter status is to provide a state table for each of the sequences of events, then converting the enumerated steps to paragraphs. See for example Table 15.1 and the corresponding discussion. Would you be interested in constructing such tables? That would speed things up!

No problem! I'm glad I can contribute to this book!

Very good, thank you! I look forward to seeing patches providing tables.

Thanx, Paul

Quoted 51 lines

Thanks,
Hao Lee

Quoted 48 lines

Thanx, Paul

[1] https://docs.google.com/document/d/1GCdQC8SDbb54W1shjEXqGZ0Rq8a6kIeYutdSIajfpLA/edit?usp=sharing

Quoted 43 lines

Thanks,
Hao Lee

Quoted 40 lines
Quoted 14 lines

If you are thinking of some other scenario, could you please list the events leading up to that scenrio?

Quoted 10 lines
Quoted 7 lines

appendix/whymb/whymemorybarriers.tex future such problems: · line 1646

Quoted 5 lines
appendix/whymb/whymemorybarriers.tex +1 -1
-       in any DMA buffer after DMA to that buffer completes.
+       in any DMA buffer before DMA to that buffer completes.
        And even then, you need to be \emph{very} careful to avoid
        pointer bugs, as even a misplaced read to an input buffer
        can result in corrupting the data input!

And this last clause could just wordsmithed a bit.

Maybe it's not quite rigorous?

Quoted 8 lines
Quoted 5 lines

If I understand correctly, a misplaced read will load the data from input buffer area to cache line. If the cpu doesn't write to this cache line, I think this case can only result in that the cpu reads stale data in subsequent reads but won't corrupt the input buffer given that this cache line is clean and won't writeback to memory.

If the user code that accesses the buffer runs on this same CPU, it will look to that CPU as if the DMA buffer was corrupted, correct?

Ah, this indeed makes sense!

Quoted 6 lines

After
all, that user code will see the value loaded by the CPU, not the value actually DMAed into memory.

Hmmm... It might be necessary to spell out these DMA-corruption scenarios. Thoughts?

Yes. I think spelling out these scenarios should make this passage easier to understand.

Thanks,
Hao Lee

Quoted 2 lines

Thanx, Paul

#9 lore raw
In reply to Hao Lee
Quoted 40 lines · Hao Lee
Quoted 39 lines · Paul E. McKenney
Quoted 37 lines · Hao Lee
Quoted 21 lines · Paul E. McKenney
Quoted 13 lines · Hao Lee

CPU caches need to be flushed before DMA operations to avoid data overwriting.

---
 appendix/whymb/whymemorybarriers.tex | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
appendix/whymb/whymemorybarriers.tex +0 -0
index b49d5fe8..4ec27962 100644

appendix/whymb/whymemorybarriers.tex future such problems: · line 1641

Quoted 2 lines
appendix/whymb/whymemorybarriers.tex +0 -0
@@ -1640,7 +1640,7 @@ future such problems:
 	in any DMA buffer before presenting that buffer to the
 	I/O device.

The paragraph above is before the I/O starts. Any changes to the buffer need to be flushed to memory so that the I/O device's DMA actually sees them.

Quoted 3 lines

The next paragraph is the other end of I/O request, after it completes. So we cannot have "before". But maybe we need "invalidate" instead of "flush".

Got it!

Quoted 6 lines

With a note that itcan be more efficient to both flush and invalidate prior to presenting the DMA buffer to the device.

The point is that any pre-DMA data in the CPU caches must be gotten rid of in order for the CPUs to see the data that was actually DMA'ed into memory.

Great explanation! Now I know we need something like WBINVD instruction for the former case and INVD instruction for the latter.

Quoted 6 lines
Quoted 6 lines

Glad that it helped!

Quoted 4 lines
Quoted 1 line

appendix/whymb/whymemorybarriers.tex future such problems: · line 1642

Quoted 1 line
appendix/whymb/whymemorybarriers.tex +0 -0
 	Similarly, you need to flush the CPU caches of any location

Should we change the word "flush" to "invalidate" to differentiate these two opposite DMA operations?

I did use "invalidate" after an input DMA, but added a footnote about the incorrect store operation.

Quoted 12 lines
Quoted 2 lines

If after the DMA, we need to care about the difference, hasn't the CPU incorrectly written to part of the DMA buffer during the DMA operation?

Sorry, I can't understand this passage... Maybe I didn't make myself clear. I find a great example in arch/powerpc/include/asm/cacheflush.h

Before DMA provides output buffer to devices, we should call flush_dcache_range or clean_dcache_range to let the devices see the latest data generated by the CPU. After DMA writes data into the input buffer, we should use invalidate_dcache_range to invalidate the CPU cache so the CPU can read the latest data written by devices.

Quoted 4 lines

Should we change the word "flush" in the following sentence to "invalidate"?

Similarly, you need to flush the CPU caches of any location in any DMA buffer after DMA to that buffer completes.

The trick is that "invalidate" and "flush" normally do the same thing for cachelines in the MESI exclusive and shared states. In both cases, memory is considered to be up to date, so the data can simply be dropped from the cache. There is a difference in the MESI modified state, but the flush prior to the DMA will have ensured that none of the cache lines corresponding to the DMA buffer are in exclusive state.

So the only way that a post-DMA flush can be different than a post-DMA invalidation is if the CPU (incorrectly!) stored to the DMA buffer while the DMA was happening.

And you should never store to a DMA buffer while the DMA is happening, at least not if you value the data baing DMA'ed. ;-)

Quoted 28 lines
Quoted 13 lines

If you are thinking of some other scenario, could you please list the events leading up to that scenrio?

Quoted 10 lines
Quoted 7 lines

appendix/whymb/whymemorybarriers.tex future such problems: · line 1646

Quoted 5 lines
appendix/whymb/whymemorybarriers.tex +1 -1
-	in any DMA buffer after DMA to that buffer completes.
+	in any DMA buffer before DMA to that buffer completes.
 	And even then, you need to be \emph{very} careful to avoid
 	pointer bugs, as even a misplaced read to an input buffer
 	can result in corrupting the data input!

And this last clause could just wordsmithed a bit.

Maybe it's not quite rigorous?

Quoted 8 lines
Quoted 5 lines

If I understand correctly, a misplaced read will load the data from input buffer area to cache line. If the cpu doesn't write to this cache line, I think this case can only result in that the cpu reads stale data in subsequent reads but won't corrupt the input buffer given that this cache line is clean and won't writeback to memory.

If the user code that accesses the buffer runs on this same CPU, it will look to that CPU as if the DMA buffer was corrupted, correct?

Ah, this indeed makes sense!

Quoted 4 lines

After
all, that user code will see the value loaded by the CPU, not the value actually DMAed into memory.

Quoted 5 lines
Quoted 2 lines

Hmmm... It might be necessary to spell out these DMA-corruption scenarios. Thoughts?

Yes. I think spelling out these scenarios should make this passage easier to understand.

How does the diff shown below look?

Thanx, Paul

------------------------------------------------------------------------

commit 5b1df1e1d99d59ee73286fded86276495d4cb138

Author: Paul E. McKenney <[email protected]>
Date:   Fri May 6 12:49:36 2022 -0700

appendix/whymb: Clarify DMA data-corruption scenarios

appendix/whymb/whymemorybarriers.tex +21 -2
index 2140eb8a..f2cca5d6 100644
@@ -1739,11 +1739,30 @@ future such problems:
 	you must carefully flush the CPU caches of any location
 	in any DMA buffer before presenting that buffer to the
 	I/O device.
-	Similarly, you need to flush the CPU caches of any location
-	in any DMA buffer after DMA to that buffer completes.
+	Otherwise, a store from one of the CPUs might not be
+	accounted for in the data DMAed out through the device.
+	This is a form of data corruption, which is an extremely
+	serious bug.
+
+	Similarly, you need to invalidate\footnote{
+		Why not flush?
+		If there is a difference, then a CPU must have incorrectly
+		stored to the DMA buffer in the midst of the DMA operation.}
+	the CPU caches corresponding to any location in any DMA buffer
+	after DMA to that buffer completes.
+	Otherwise, a given CPU might see the old data still residing in
+	its cache instead of the newly DMAed data that it was supposed
+	to see.
+	This is another form of data corruption.
+
 	And even then, you need to be \emph{very} careful to avoid
 	pointer bugs, as even a misplaced read to an input buffer
 	can result in corrupting the data input!
+	One way to avoid this is to invalidate all of the caches of
+	all of the CPUs once the DMA completes, but it is much easier
+	and more efficient if the device DMA participates in the
+	cache-coherence protocol, making all of this flushing and
+	invalidating unnecessory.
 
 \item	External busses that fail to transmit cache-coherence data.
 
author
#10 lore raw
Quoted 43 lines · Paul E. McKenney
Quoted 42 lines · Hao Lee
Quoted 41 lines · Paul E. McKenney
Quoted 31 lines · Hao Lee
Quoted 21 lines · Paul E. McKenney
Quoted 13 lines · Hao Lee

CPU caches need to be flushed before DMA operations to avoid data overwriting.

---
 appendix/whymb/whymemorybarriers.tex | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
appendix/whymb/whymemorybarriers.tex +0 -0
index b49d5fe8..4ec27962 100644

appendix/whymb/whymemorybarriers.tex future such problems: · line 1641

Quoted 2 lines
appendix/whymb/whymemorybarriers.tex +0 -0
@@ -1640,7 +1640,7 @@ future such problems:
 	in any DMA buffer before presenting that buffer to the
 	I/O device.

The paragraph above is before the I/O starts. Any changes to the buffer need to be flushed to memory so that the I/O device's DMA actually sees them.

Quoted 3 lines

The next paragraph is the other end of I/O request, after it completes. So we cannot have "before". But maybe we need "invalidate" instead of "flush".

Got it!

Quoted 3 lines

With a note that itcan be more efficient to both flush and invalidate prior to presenting the DMA buffer to the device.

Quoted 6 lines
Quoted 3 lines

The point is that any pre-DMA data in the CPU caches must be gotten rid of in order for the CPUs to see the data that was actually DMA'ed into memory.

Great explanation! Now I know we need something like WBINVD instruction for the former case and INVD instruction for the latter.

Glad that it helped!

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

appendix/whymb/whymemorybarriers.tex future such problems: · line 1642

Quoted 1 line
appendix/whymb/whymemorybarriers.tex +0 -0
 	Similarly, you need to flush the CPU caches of any location

Should we change the word "flush" to "invalidate" to differentiate these two opposite DMA operations?

I did use "invalidate" after an input DMA, but added a footnote about the incorrect store operation.

I see in the following patch. Thanks!

Quoted 25 lines
Quoted 13 lines
Quoted 3 lines

If after the DMA, we need to care about the difference, hasn't the CPU incorrectly written to part of the DMA buffer during the DMA operation?

Sorry, I can't understand this passage... Maybe I didn't make myself clear. I find a great example in arch/powerpc/include/asm/cacheflush.h

Before DMA provides output buffer to devices, we should call flush_dcache_range or clean_dcache_range to let the devices see the latest data generated by the CPU. After DMA writes data into the input buffer, we should use invalidate_dcache_range to invalidate the CPU cache so the CPU can read the latest data written by devices.

Quoted 4 lines

Should we change the word "flush" in the following sentence to "invalidate"?

Similarly, you need to flush the CPU caches of any location in any DMA buffer after DMA to that buffer completes.

The trick is that "invalidate" and "flush" normally do the same thing for cachelines in the MESI exclusive and shared states. In both cases, memory is considered to be up to date, so the data can simply be dropped from the cache. There is a difference in the MESI modified state, but the flush prior to the DMA will have ensured that none of the cache lines corresponding to the DMA buffer are in exclusive state.

Quoted 6 lines

So the only way that a post-DMA flush can be different than a post-DMA invalidation is if the CPU (incorrectly!) stored to the DMA buffer while the DMA was happening.

And you should never store to a DMA buffer while the DMA is happening, at least not if you value the data baing DMA'ed. ;-)

This explanation is pretty clear. Many thanks!

Quoted 29 lines
Quoted 29 lines
Quoted 14 lines

If you are thinking of some other scenario, could you please list the events leading up to that scenrio?

Quoted 10 lines
Quoted 7 lines

appendix/whymb/whymemorybarriers.tex future such problems: · line 1760

Quoted 5 lines
appendix/whymb/whymemorybarriers.tex +1 -1
-	in any DMA buffer after DMA to that buffer completes.
+	in any DMA buffer before DMA to that buffer completes.
 	And even then, you need to be \emph{very} careful to avoid
 	pointer bugs, as even a misplaced read to an input buffer
 	can result in corrupting the data input!

And this last clause could just wordsmithed a bit.

Maybe it's not quite rigorous?

Quoted 8 lines
Quoted 5 lines

If I understand correctly, a misplaced read will load the data from input buffer area to cache line. If the cpu doesn't write to this cache line, I think this case can only result in that the cpu reads stale data in subsequent reads but won't corrupt the input buffer given that this cache line is clean and won't writeback to memory.

If the user code that accesses the buffer runs on this same CPU, it will look to that CPU as if the DMA buffer was corrupted, correct?

Ah, this indeed makes sense!

Quoted 4 lines

After
all, that user code will see the value loaded by the CPU, not the value actually DMAed into memory.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Hmmm... It might be necessary to spell out these DMA-corruption scenarios. Thoughts?

Yes. I think spelling out these scenarios should make this passage easier to understand.

How does the diff shown below look?

Perfect!

Thanks,
Hao Lee

Quoted 51 lines

Thanx, Paul

------------------------------------------------------------------------

commit 5b1df1e1d99d59ee73286fded86276495d4cb138

Author: Paul E. McKenney <[email protected]>
Date:   Fri May 6 12:49:36 2022 -0700

appendix/whymb: Clarify DMA data-corruption scenarios

appendix/whymb/whymemorybarriers.tex +21 -2
index 2140eb8a..f2cca5d6 100644
@@ -1739,11 +1739,30 @@ future such problems:
 	you must carefully flush the CPU caches of any location
 	in any DMA buffer before presenting that buffer to the
 	I/O device.
-	Similarly, you need to flush the CPU caches of any location
-	in any DMA buffer after DMA to that buffer completes.
+	Otherwise, a store from one of the CPUs might not be
+	accounted for in the data DMAed out through the device.
+	This is a form of data corruption, which is an extremely
+	serious bug.
+
+	Similarly, you need to invalidate\footnote{
+		Why not flush?
+		If there is a difference, then a CPU must have incorrectly
+		stored to the DMA buffer in the midst of the DMA operation.}
+	the CPU caches corresponding to any location in any DMA buffer
+	after DMA to that buffer completes.
+	Otherwise, a given CPU might see the old data still residing in
+	its cache instead of the newly DMAed data that it was supposed
+	to see.
+	This is another form of data corruption.
+
 	And even then, you need to be \emph{very} careful to avoid
 	pointer bugs, as even a misplaced read to an input buffer
 	can result in corrupting the data input!
+	One way to avoid this is to invalidate all of the caches of
+	all of the CPUs once the DMA completes, but it is much easier
+	and more efficient if the device DMA participates in the
+	cache-coherence protocol, making all of this flushing and
+	invalidating unnecessory.
 
 \item	External busses that fail to transmit cache-coherence data.