guest assigned device MMIO maps with WC: does this work correctly?

DISCUSSION Michael S. Tsirkin 2 participants 11 messages Nov 20, 2013 – Nov 21, 2013 KVM

lore

Share & tools

lore

Michael S. Tsirkin

author

2013-11-20 19:55 UTC #0 lore raw

I see this in kvm:

static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)
{
u64 ret;

/* For VT-d and EPT combination
* 1. MMIO: always map as UC
* 2. EPT with VT-d:
* a. VT-d without snooping control feature: can't guarantee
* the
* result, try to trust guest.
* b. VT-d with snooping control feature: snooping control
* feature of
* VT-d engine can guarantee the cache correctness. Just
* set it
* to WB to keep consistent with host. So the same as item
* 3.
* 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
* consistent with host MTRR
*/
if (is_mmio)
ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;

...
}

does this mean that even if guest maps BAR for an assigned device as write combined (or configures such using an MTRR), host will override this and use uncacheable in practice?

I think there are some drivers that map MMIO as write-combined for performance (but don't have such hardware) so
I'd like to figure out from code whether this will work correctly.

Signature · 1 line

MST

Gleb Natapov

2013-11-21 07:18 UTC #1 lore raw

Quoted 25 lines · Michael S. Tsirkin

I see this in kvm:
static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)
{
u64 ret;
/* For VT-d and EPT combination
* 1. MMIO: always map as UC
* 2. EPT with VT-d:
* a. VT-d without snooping control feature: can't guarantee
* the
* result, try to trust guest.
* b. VT-d with snooping control feature: snooping control
* feature of
* VT-d engine can guarantee the cache correctness. Just
* set it
* to WB to keep consistent with host. So the same as item
* 3.
* 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
* consistent with host MTRR
*/
if (is_mmio)
ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;

Quoted 8 lines

...
}
does this mean that even if guest maps BAR for an assigned device as write combined (or configures such using an MTRR), host will override this and use uncacheable in practice?

No, it does not mean that. I already answered this once (my previous answer included below): effective memory type is a combination of MTRR (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2 in SDM on how effective memory type is calculated.

Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table (that what ioremap_wc does), everything works as it should. If guest maps MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will be by default) combined memory type will be UC, so also fine. If guest maps MMIO range as WB and fixes mtrr for this region to be WB then memory type will be incorrect in a guest, but I found only one place that does it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache either remap RAM or used to get whatever memory type configured in MTRR.

--
Gleb.

Michael S. Tsirkin

author

2013-11-21 07:58 UTC #2 lore raw

Quoted 30 lines · Gleb Natapov

Quoted 29 lines · Michael S. Tsirkin
I see this in kvm:
static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)
{
u64 ret;
/* For VT-d and EPT combination
* 1. MMIO: always map as UC
* 2. EPT with VT-d:
* a. VT-d without snooping control feature: can't guarantee
* the
* result, try to trust guest.
* b. VT-d with snooping control feature: snooping control
* feature of
* VT-d engine can guarantee the cache correctness. Just
* set it
* to WB to keep consistent with host. So the same as item
* 3.
* 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
* consistent with host MTRR
*/
if (is_mmio)
ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
...
}

Quoted 8 lines

Quoted 4 lines
does this mean that even if guest maps BAR for an assigned device as write combined (or configures such using an MTRR), host will override this and use uncacheable in practice?
No, it does not mean that. I already answered this once (my previous answer included below): effective memory type is a combination of MTRR (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2 in SDM

Can you quote chapter name please?
My SDM has
11.5.2.2 Denormal-Operand Exception (#D)

Quoted 8 lines

on how effective memory type is calculated.
Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table (that what ioremap_wc does), everything works as it should. If guest maps MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will be by default) combined memory type will be UC, so also fine. If guest maps MMIO range as WB and fixes mtrr for this region to be WB then memory type will be incorrect in a guest,

Meaning MTRR in guest is ignored in this case?

Quoted 6 lines

but I found only one place that does
it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache either remap RAM or used to get whatever memory type configured in MTRR.
--
Gleb.

Gleb Natapov

2013-11-21 08:01 UTC #3 lore raw

Quoted 41 lines · Michael S. Tsirkin

Quoted 30 lines · Gleb Natapov
Quoted 29 lines · Michael S. Tsirkin
I see this in kvm:
static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)
{
u64 ret;
/* For VT-d and EPT combination
* 1. MMIO: always map as UC
* 2. EPT with VT-d:
* a. VT-d without snooping control feature: can't guarantee
* the
* result, try to trust guest.
* b. VT-d with snooping control feature: snooping control
* feature of
* VT-d engine can guarantee the cache correctness. Just
* set it
* to WB to keep consistent with host. So the same as item
* 3.
* 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
* consistent with host MTRR
*/
if (is_mmio)
ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
...
}
Quoted 8 lines
Quoted 4 lines
does this mean that even if guest maps BAR for an assigned device as write combined (or configures such using an MTRR), host will override this and use uncacheable in practice?
No, it does not mean that. I already answered this once (my previous answer included below): effective memory type is a combination of MTRR (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2 in SDM

Quoted 4 lines

Can you quote chapter name please?
My SDM has
11.5.2.2 Denormal-Operand Exception (#D)

Either your or mine is out of date:
Selecting Memory Types for Pentium III and More Recent Processor Families

Quoted 10 lines

Quoted 8 lines
on how effective memory type is calculated.
Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table (that what ioremap_wc does), everything works as it should. If guest maps MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will be by default) combined memory type will be UC, so also fine. If guest maps MMIO range as WB and fixes mtrr for this region to be WB then memory type will be incorrect in a guest,
Meaning MTRR in guest is ignored in this case?

Yes.

Quoted 7 lines

Quoted 7 lines
but I found only one place that does
it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache either remap RAM or used to get whatever memory type configured in MTRR.
--
Gleb.

--
Gleb.

Michael S. Tsirkin

author

2013-11-21 08:13 UTC #4 lore raw

Quoted 42 lines · Gleb Natapov

Quoted 41 lines · Michael S. Tsirkin
Quoted 38 lines · Gleb Natapov
Quoted 25 lines · Michael S. Tsirkin
I see this in kvm:
static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)
{
u64 ret;
/* For VT-d and EPT combination
* 1. MMIO: always map as UC
* 2. EPT with VT-d:
* a. VT-d without snooping control feature: can't guarantee
* the
* result, try to trust guest.
* b. VT-d with snooping control feature: snooping control
* feature of
* VT-d engine can guarantee the cache correctness. Just
* set it
* to WB to keep consistent with host. So the same as item
* 3.
* 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
* consistent with host MTRR
*/
if (is_mmio)
ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
Quoted 8 lines
...
}
does this mean that even if guest maps BAR for an assigned device as write combined (or configures such using an MTRR), host will override this and use uncacheable in practice?
No, it does not mean that. I already answered this once (my previous answer included below): effective memory type is a combination of MTRR (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2 in SDM

Quoted 6 lines

Quoted 4 lines
Can you quote chapter name please?
My SDM has
11.5.2.2 Denormal-Operand Exception (#D)
Either your or mine is out of date:
Selecting Memory Types for Pentium III and More Recent Processor Families

OK this one I'm familiar with, it describes how PAT interacts with MTRR. But how does this interact with EPT? do you remember where's that described?

Quoted 22 lines

Quoted 10 lines
Quoted 8 lines
on how effective memory type is calculated.
Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table (that what ioremap_wc does), everything works as it should. If guest maps MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will be by default) combined memory type will be UC, so also fine. If guest maps MMIO range as WB and fixes mtrr for this region to be WB then memory type will be incorrect in a guest,
Meaning MTRR in guest is ignored in this case?
Yes.
Quoted 7 lines
Quoted 7 lines
but I found only one place that does
it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache either remap RAM or used to get whatever memory type configured in MTRR.
--
Gleb.
--
Gleb.

Michael S. Tsirkin

author

2013-11-21 08:15 UTC #5 lore raw

Quoted 47 lines · Michael S. Tsirkin

Quoted 46 lines · Gleb Natapov
Quoted 45 lines · Michael S. Tsirkin
Quoted 30 lines · Gleb Natapov
Quoted 29 lines · Michael S. Tsirkin
I see this in kvm:
static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)
{
u64 ret;
/* For VT-d and EPT combination
* 1. MMIO: always map as UC
* 2. EPT with VT-d:
* a. VT-d without snooping control feature: can't guarantee
* the
* result, try to trust guest.
* b. VT-d with snooping control feature: snooping control
* feature of
* VT-d engine can guarantee the cache correctness. Just
* set it
* to WB to keep consistent with host. So the same as item
* 3.
* 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
* consistent with host MTRR
*/
if (is_mmio)
ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
...
}
Quoted 8 lines
Quoted 4 lines
does this mean that even if guest maps BAR for an assigned device as write combined (or configures such using an MTRR), host will override this and use uncacheable in practice?
No, it does not mean that. I already answered this once (my previous answer included below): effective memory type is a combination of MTRR (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2 in SDM
Can you quote chapter name please?
My SDM has
11.5.2.2 Denormal-Operand Exception (#D)

Quoted 6 lines

Quoted 2 lines
Either your or mine is out of date:
Selecting Memory Types for Pentium III and More Recent Processor Families
OK this one I'm familiar with, it describes how PAT interacts with MTRR. But how does this interact with EPT? do you remember where's that described?

Found it 28.2.4 EPT and Memory Typing
sorry about the noise.

Quoted 22 lines

Quoted 22 lines
Quoted 10 lines
Quoted 8 lines
on how effective memory type is calculated.
Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table (that what ioremap_wc does), everything works as it should. If guest maps MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will be by default) combined memory type will be UC, so also fine. If guest maps MMIO range as WB and fixes mtrr for this region to be WB then memory type will be incorrect in a guest,
Meaning MTRR in guest is ignored in this case?
Yes.
Quoted 7 lines
Quoted 7 lines
but I found only one place that does
it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache either remap RAM or used to get whatever memory type configured in MTRR.
--
Gleb.
--
Gleb.

Gleb Natapov

2013-11-21 08:16 UTC #10 lore raw

In reply to Michael S. Tsirkin

Quoted 47 lines · Michael S. Tsirkin

Quoted 46 lines · Gleb Natapov
Quoted 45 lines · Michael S. Tsirkin
Quoted 30 lines · Gleb Natapov
Quoted 29 lines · Michael S. Tsirkin
I see this in kvm:
static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)
{
u64 ret;
/* For VT-d and EPT combination
* 1. MMIO: always map as UC
* 2. EPT with VT-d:
* a. VT-d without snooping control feature: can't guarantee
* the
* result, try to trust guest.
* b. VT-d with snooping control feature: snooping control
* feature of
* VT-d engine can guarantee the cache correctness. Just
* set it
* to WB to keep consistent with host. So the same as item
* 3.
* 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
* consistent with host MTRR
*/
if (is_mmio)
ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
...
}
Quoted 8 lines
Quoted 4 lines
does this mean that even if guest maps BAR for an assigned device as write combined (or configures such using an MTRR), host will override this and use uncacheable in practice?
No, it does not mean that. I already answered this once (my previous answer included below): effective memory type is a combination of MTRR (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2 in SDM
Can you quote chapter name please?
My SDM has
11.5.2.2 Denormal-Operand Exception (#D)

Quoted 7 lines

Quoted 2 lines
Either your or mine is out of date:
Selecting Memory Types for Pentium III and More Recent Processor Families
OK this one I'm familiar with, it describes how PAT interacts with MTRR. But how does this interact with EPT? do you remember where's that described?

28.2.5.2 Memory Type Used for Translated Guest-Physical Addresses Substitute MTRR with EPT MT and go to the same table.

--
Gleb.

Michael S. Tsirkin

author

2013-11-21 08:18 UTC #6 lore raw

In reply to Michael S. Tsirkin

Quoted 51 lines · Michael S. Tsirkin

Quoted 50 lines · Michael S. Tsirkin
Quoted 48 lines · Gleb Natapov
Quoted 41 lines · Michael S. Tsirkin
Quoted 30 lines · Gleb Natapov
Quoted 29 lines · Michael S. Tsirkin
I see this in kvm:
static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)
{
u64 ret;
/* For VT-d and EPT combination
* 1. MMIO: always map as UC
* 2. EPT with VT-d:
* a. VT-d without snooping control feature: can't guarantee
* the
* result, try to trust guest.
* b. VT-d with snooping control feature: snooping control
* feature of
* VT-d engine can guarantee the cache correctness. Just
* set it
* to WB to keep consistent with host. So the same as item
* 3.
* 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
* consistent with host MTRR
*/
if (is_mmio)
ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
...
}
Quoted 8 lines
Quoted 4 lines
does this mean that even if guest maps BAR for an assigned device as write combined (or configures such using an MTRR), host will override this and use uncacheable in practice?
No, it does not mean that. I already answered this once (my previous answer included below): effective memory type is a combination of MTRR (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2 in SDM
Quoted 4 lines
Can you quote chapter name please?
My SDM has
11.5.2.2 Denormal-Operand Exception (#D)
Either your or mine is out of date:
Selecting Memory Types for Pentium III and More Recent Processor Families

Quoted 6 lines

Quoted 3 lines
OK this one I'm familiar with, it describes how PAT interacts with MTRR. But how does this interact with EPT? do you remember where's that described?
Found it 28.2.4 EPT and Memory Typing
sorry about the noise.

OK and that's explicit:
If CR0.CD = 0, the effective memory type depends upon the value of bit 6 of the last EPT paging-structure entry:
— If the value is 0, the effective memory type is the combination of the EPT memory type and the PAT memory type specified in Table 11-7 in Section 11.5.2.2, using the EPT memory type in place of the MTRR memory type.

— If the value is 1, the memory type used for the access is the EPT memory
type. The PAT memory type is ignored.

If CR0.CD = 1, the effective memory type is UC.

So it's simple. EPT replaces guest's MTRR.

Quoted 23 lines

Quoted 23 lines
Quoted 23 lines
Quoted 11 lines
Quoted 9 lines
on how effective memory type is calculated.
Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table (that what ioremap_wc does), everything works as it should. If guest maps MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will be by default) combined memory type will be UC, so also fine. If guest maps MMIO range as WB and fixes mtrr for this region to be WB then memory type will be incorrect in a guest,
Meaning MTRR in guest is ignored in this case?
Yes.
Quoted 7 lines
Quoted 7 lines
but I found only one place that does
it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache either remap RAM or used to get whatever memory type configured in MTRR.
--
Gleb.
--
Gleb.

Michael S. Tsirkin

author

2013-11-21 08:22 UTC #7 lore raw

Quoted 67 lines · Michael S. Tsirkin

Quoted 51 lines · Michael S. Tsirkin
Quoted 50 lines · Michael S. Tsirkin
Quoted 48 lines · Gleb Natapov
Quoted 41 lines · Michael S. Tsirkin
Quoted 30 lines · Gleb Natapov
Quoted 29 lines · Michael S. Tsirkin
I see this in kvm:
static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)
{
u64 ret;
/* For VT-d and EPT combination
* 1. MMIO: always map as UC
* 2. EPT with VT-d:
* a. VT-d without snooping control feature: can't guarantee
* the
* result, try to trust guest.
* b. VT-d with snooping control feature: snooping control
* feature of
* VT-d engine can guarantee the cache correctness. Just
* set it
* to WB to keep consistent with host. So the same as item
* 3.
* 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
* consistent with host MTRR
*/
if (is_mmio)
ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
...
}
Quoted 8 lines
Quoted 4 lines
does this mean that even if guest maps BAR for an assigned device as write combined (or configures such using an MTRR), host will override this and use uncacheable in practice?
No, it does not mean that. I already answered this once (my previous answer included below): effective memory type is a combination of MTRR (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2 in SDM
Quoted 4 lines
Can you quote chapter name please?
My SDM has
11.5.2.2 Denormal-Operand Exception (#D)
Either your or mine is out of date:
Selecting Memory Types for Pentium III and More Recent Processor Families
Quoted 6 lines
Quoted 3 lines
OK this one I'm familiar with, it describes how PAT interacts with MTRR. But how does this interact with EPT? do you remember where's that described?
Found it 28.2.4 EPT and Memory Typing
sorry about the noise.
OK and that's explicit:
If CR0.CD = 0, the effective memory type depends upon the value of bit 6 of the last EPT paging-structure entry:
— If the value is 0, the effective memory type is the combination of the EPT memory type and the PAT memory type specified in Table 11-7 in Section 11.5.2.2, using the EPT memory type in place of the MTRR memory type.

Quoted 7 lines

— If the value is 1, the memory type used for the access is the EPT memory
type. The PAT memory type is ignored.
If CR0.CD = 1, the effective memory type is UC.
So it's simple. EPT replaces guest's MTRR.

And that in turn means that since we set UC in EPT, VCPU will always work as if it's UC except for guests using WC - WC takes precedence.

Thanks!

Quoted 23 lines

Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 11 lines
Quoted 9 lines
on how effective memory type is calculated.
Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table (that what ioremap_wc does), everything works as it should. If guest maps MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will be by default) combined memory type will be UC, so also fine. If guest maps MMIO range as WB and fixes mtrr for this region to be WB then memory type will be incorrect in a guest,
Meaning MTRR in guest is ignored in this case?
Yes.
Quoted 7 lines
Quoted 7 lines
but I found only one place that does
it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache either remap RAM or used to get whatever memory type configured in MTRR.
--
Gleb.
--
Gleb.

Gleb Natapov

2013-11-21 08:25 UTC #8 lore raw

Quoted 72 lines · Michael S. Tsirkin

Quoted 71 lines · Michael S. Tsirkin
Quoted 51 lines · Michael S. Tsirkin
Quoted 50 lines · Michael S. Tsirkin
Quoted 48 lines · Gleb Natapov
Quoted 41 lines · Michael S. Tsirkin
Quoted 30 lines · Gleb Natapov
Quoted 29 lines · Michael S. Tsirkin
I see this in kvm:
static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)
{
u64 ret;
/* For VT-d and EPT combination
* 1. MMIO: always map as UC
* 2. EPT with VT-d:
* a. VT-d without snooping control feature: can't guarantee
* the
* result, try to trust guest.
* b. VT-d with snooping control feature: snooping control
* feature of
* VT-d engine can guarantee the cache correctness. Just
* set it
* to WB to keep consistent with host. So the same as item
* 3.
* 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
* consistent with host MTRR
*/
if (is_mmio)
ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
...
}
Quoted 8 lines
Quoted 4 lines
does this mean that even if guest maps BAR for an assigned device as write combined (or configures such using an MTRR), host will override this and use uncacheable in practice?
No, it does not mean that. I already answered this once (my previous answer included below): effective memory type is a combination of MTRR (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2 in SDM
Quoted 4 lines
Can you quote chapter name please?
My SDM has
11.5.2.2 Denormal-Operand Exception (#D)
Either your or mine is out of date:
Selecting Memory Types for Pentium III and More Recent Processor Families
Quoted 6 lines
Quoted 3 lines
OK this one I'm familiar with, it describes how PAT interacts with MTRR. But how does this interact with EPT? do you remember where's that described?
Found it 28.2.4 EPT and Memory Typing
sorry about the noise.
OK and that's explicit:
If CR0.CD = 0, the effective memory type depends upon the value of bit 6 of the last EPT paging-structure entry:
— If the value is 0, the effective memory type is the combination of the EPT memory type and the PAT memory type specified in Table 11-7 in Section 11.5.2.2, using the EPT memory type in place of the MTRR memory type.
— If the value is 1, the memory type used for the access is the EPT memory
type. The PAT memory type is ignored.

Quoted 9 lines

Quoted 3 lines
If CR0.CD = 1, the effective memory type is UC.
So it's simple. EPT replaces guest's MTRR.
And that in turn means that since we set UC in EPT, VCPU will always work as if it's UC except for guests using WC - WC takes precedence.

That's what I wrote below, no? :)

Quoted 25 lines

Thanks!
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 11 lines
Quoted 9 lines
on how effective memory type is calculated.
Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table (that what ioremap_wc does), everything works as it should. If guest maps MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will be by default) combined memory type will be UC, so also fine. If guest maps MMIO range as WB and fixes mtrr for this region to be WB then memory type will be incorrect in a guest,
Meaning MTRR in guest is ignored in this case?
Yes.
Quoted 7 lines
Quoted 7 lines
but I found only one place that does
it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache either remap RAM or used to get whatever memory type configured in MTRR.
--
Gleb.
--
Gleb.

--
Gleb.

Michael S. Tsirkin

author

2013-11-21 08:30 UTC #9 lore raw

Quoted 75 lines · Gleb Natapov

Quoted 74 lines · Michael S. Tsirkin
Quoted 73 lines · Michael S. Tsirkin
Quoted 51 lines · Michael S. Tsirkin
Quoted 50 lines · Michael S. Tsirkin
Quoted 48 lines · Gleb Natapov
Quoted 41 lines · Michael S. Tsirkin
Quoted 30 lines · Gleb Natapov
Quoted 29 lines · Michael S. Tsirkin
I see this in kvm:
static u64 vmx_get_mt_mask(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio)
{
u64 ret;
/* For VT-d and EPT combination
* 1. MMIO: always map as UC
* 2. EPT with VT-d:
* a. VT-d without snooping control feature: can't guarantee
* the
* result, try to trust guest.
* b. VT-d with snooping control feature: snooping control
* feature of
* VT-d engine can guarantee the cache correctness. Just
* set it
* to WB to keep consistent with host. So the same as item
* 3.
* 3. EPT without VT-d: always map as WB and set IPAT=1 to keep
* consistent with host MTRR
*/
if (is_mmio)
ret = MTRR_TYPE_UNCACHABLE << VMX_EPT_MT_EPTE_SHIFT;
...
}
Quoted 8 lines
Quoted 4 lines
does this mean that even if guest maps BAR for an assigned device as write combined (or configures such using an MTRR), host will override this and use uncacheable in practice?
No, it does not mean that. I already answered this once (my previous answer included below): effective memory type is a combination of MTRR (EPT MT bits in case of a guest) and PAT bits. See section 11.5.2.2 in SDM
Quoted 4 lines
Can you quote chapter name please?
My SDM has
11.5.2.2 Denormal-Operand Exception (#D)
Either your or mine is out of date:
Selecting Memory Types for Pentium III and More Recent Processor Families
Quoted 6 lines
Quoted 3 lines
OK this one I'm familiar with, it describes how PAT interacts with MTRR. But how does this interact with EPT? do you remember where's that described?
Found it 28.2.4 EPT and Memory Typing
sorry about the noise.
OK and that's explicit:
If CR0.CD = 0, the effective memory type depends upon the value of bit 6 of the last EPT paging-structure entry:
— If the value is 0, the effective memory type is the combination of the EPT memory type and the PAT memory type specified in Table 11-7 in Section 11.5.2.2, using the EPT memory type in place of the MTRR memory type.
— If the value is 1, the memory type used for the access is the EPT memory
type. The PAT memory type is ignored.
If CR0.CD = 1, the effective memory type is UC.

Quoted 8 lines

Quoted 7 lines
Quoted 1 line
So it's simple. EPT replaces guest's MTRR.
And that in turn means that since we set UC in EPT, VCPU will always work as if it's UC except for guests using WC - WC takes precedence.
That's what I wrote below, no? :)

Yes I just added that this applies to other memory types like WT (that probably nobody uses?).

Quoted 28 lines

Quoted 25 lines
Thanks!
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 11 lines
Quoted 9 lines
on how effective memory type is calculated.
Since MTRR UC + PAT WC = WC, if guest maps MMIO as WC in a page table (that what ioremap_wc does), everything works as it should. If guest maps MMIO as WB (ioremap_cache) and MTRR says MMIO is UC (like any MMIO will be by default) combined memory type will be UC, so also fine. If guest maps MMIO range as WB and fixes mtrr for this region to be WB then memory type will be incorrect in a guest,
Meaning MTRR in guest is ignored in this case?
Yes.
Quoted 7 lines
Quoted 7 lines
but I found only one place that does
it in Linux: drivers/video/vesafb.c. All other uses of ioremap_cache either remap RAM or used to get whatever memory type configured in MTRR.
--
Gleb.
--
Gleb.
--
Gleb.