Add enlightenments for vGPU
Subsystems (2)
cover Add enlightenments for vGPU
Intel GVT-g (previously known as XenGT), is a complete GPU virtualization solution with mediated pass-through for 4th generation Intel Core processors - Haswell platform. This technology presents a virtual full-fledged GPU to each Virtual Machine (VM). VMs can directly access performance-critical resources, without intervention from the hypervisor in most cases, while privileged operations from VMs are trap-and-emulated at minimal cost. For detail, please refer to
https://01.org/xen/blogs/wangbo85/2014/intel-gvt-gxengt-pubic-release
This patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, are included in another patchset, and will be sent out later.
The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory w/o hypervisor's intervention, e.g. filling textures and queuing commands. However w/ the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses w/o complexity of translation.
Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (w/ some faulting mechanism).
The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource XenGT also partitions them among VMs.
Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.
Yu Zhang (8):
drm/i915: Introduce a PV INFO page structure for Intel GVT-g. drm/i915: Adds graphic address space ballooning logic drm/i915: Partition the fence registers for vgpu in i915 driver drm/i915: Disable framebuffer compression for i915 driver in VM drm/i915: Add the display switch logic for vgpu in i915 driver drm/i915: Disable power management for i915 driver in VM drm/i915: Create vgpu specific write MMIO to reduce traps drm/i915: Support alias ppgtt in VM if ppgtt is enabled
drivers/gpu/drm/i915/i915_dma.c | 22 ++++ drivers/gpu/drm/i915/i915_drv.h | 53 +++++++++ drivers/gpu/drm/i915/i915_gem.c | 4 + drivers/gpu/drm/i915/i915_gem_gtt.c | 214 +++++++++++++++++++++++++++++++++-- drivers/gpu/drm/i915/i915_reg.h | 20 ++++ drivers/gpu/drm/i915/intel_display.c | 10 ++ drivers/gpu/drm/i915/intel_pm.c | 6 + drivers/gpu/drm/i915/intel_uncore.c | 24 ++++ 8 files changed, 345 insertions(+), 8 deletions(-)
Signature · 1 line
1.9.1
PATCH 1/8 drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
From: Yu Zhang <[email protected]>
Introduce a PV INFO structure, to facilitate the Intel GVT-g technology, which is a GPU virtualization solution with mediated pass-through(previously known as XenGT). This page contains the shared information between i915 driver and the mediator. For now, this structure utilizes an area of 4K bypes on HSW GPU's unused MMIO space to support existing production. Future hardware will have the reserved window architecturally defined, and layout of the page will be added in future BSpec.
The i915 driver load routine detects if it is running in a VM by reading the contents of this PV INFO page. Thereafter a flag, vgpu_active is set, and intel_vgpu_active() is used by checking this flag to conclude if gpu is virtualized with Intel GVT-g. By now, it will return true only when the driver is running in the XenGT environment on HSW.
--- drivers/gpu/drm/i915/i915_dma.c | 2 ++ drivers/gpu/drm/i915/i915_drv.h | 52 +++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_gem_gtt.c | 21 +++++++++++++++ drivers/gpu/drm/i915/i915_reg.h | 13 ++++++++++ 4 files changed, 88 insertions(+)
drivers/gpu/drm/i915/i915_dma.c +2 -0
@@ -1739,6 +1739,8 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags) goto out_freewq; } +i915_check_vgpu (dev); +intel_irq_init (dev);intel_uncore_sanitize (dev);
drivers/gpu/drm/i915/i915_drv.h +52 -0
70 lines not on this page · raw message
drivers/gpu/drm/i915/i915_gem_gtt.c +21 -0
@@ -30,6 +30,27 @@#include "i915_trace.h" #include "intel_drv.h" +void i915_check_vgpu (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv = dev->dev_private; +uint64_t magic; +uint32_t version; + + dev_priv->vgpu_active = false; + +if (!IS_HASWELL (dev)) +return ; + + magic =I915_READ64 (vgt_info_off (magic)); +if (magic !=VGT_MAGIC ) +return ; + + version = (I915_READ16 (vgt_info_off (version_major)) <<16 ) | +I915_READ16 (vgt_info_off (version_minor)); +if (version ==INTEL_VGT_IF_VERSION ) + dev_priv->vgpu_active = true; +} +static void bdw_setup_private_ppat (struct drm_i915_private *dev_priv);static void chv_setup_private_ppat (struct drm_i915_private *dev_priv);
drivers/gpu/drm/i915/i915_reg.h +13 -0
@@ -6661,4 +6661,17 @@ enum punit_power_well {#define _PALETTE_A (dev_priv->info.display_mmio_offset + 0xa000)#define _PALETTE_B (dev_priv->info.display_mmio_offset + 0xa800) +/* The MMIO offset of the shared info between i915 and vgt driver */ +#define VGT_PVINFO_PAGE 0x78000 +#define VGT_PVINFO_SIZE 0x1000 + +#define VGT_MAGIC 0x4776544776544776/* 'vGTvGTvG' */ +#define VGT_VERSION_MAJOR 1 +#define VGT_VERSION_MINOR 0 +#define INTEL_VGT_IF_VERSION ((VGT_VERSION_MAJOR << 16) | VGT_VERSION_MINOR) + +#define vgt_info_off (x) \ + (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x) + +#endif /* _I915_REG_H_ */
Signature · 1 line
1.9.1
Re: [PATCH 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
Quoted 80 lines · Jike Song
From: Yu Zhang <[email protected]>
Introduce a PV INFO structure, to facilitate the Intel GVT-g technology, which is a GPU virtualization solution with mediated pass-through(previously known as XenGT). This page contains the shared information between i915 driver and the mediator. For now, this structure utilizes an area of 4K bypes on HSW GPU's unused MMIO space to support existing production. Future hardware will have the reserved window architecturally defined, and layout of the page will be added in future BSpec.
The i915 driver load routine detects if it is running in a VM by reading the contents of this PV INFO page. Thereafter a flag,
vgpu_activeis set, andintel_vgpu_active()is used by checking this flag to conclude if gpu is virtualized with Intel GVT-g. By now, it will return true only when the driver is running in the XenGT environment on HSW.Signed-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Eddie Dong <[email protected]>--- drivers/gpu/drm/i915/i915_dma.c | 2 ++ drivers/gpu/drm/i915/i915_drv.h | 52 +++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_gem_gtt.c | 21 +++++++++++++++ drivers/gpu/drm/i915/i915_reg.h | 13 ++++++++++ 4 files changed, 88 insertions(+)drivers/gpu/drm/i915/i915_dma.c +2
-0@@ -1739,6 +1739,8 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags) goto out_freewq; } +i915_check_vgpu (dev); +intel_irq_init (dev);intel_uncore_sanitize (dev);drivers/gpu/drm/i915/i915_drv.h +31
-0@@ -1442,6 +1442,48 @@ struct i915_frontbuffer_tracking {unsigned flip_bits; }; +/* +* The following structure pages are defined in GEN MMIO space +* for virtualization. (One page for now) +*/ + +struct vgt_if { +uint64_t magic;/* VGT_MAGIC */ +uint16_t version_major; +uint16_t version_minor; +uint32_t vgt_id;/* ID of vGT instance */ +uint32_t rsv1[12 ];/* pad to offset 0x40 */ +/* +* Data structure to describe the balooning info of resources. +* Each VM can only have one portion of continuous area for now. +* (May support scattered resource in future) +* (starting from offset 0x40) +*/ +struct { +/* Aperture register balooning */ +struct { +uint32_t my_base; +uint32_t my_size; + } low_gmadr;/* aperture */ +/* GMADR register balooning */ +struct { +uint32_t my_base; +uint32_t my_size; + } high_gmadr;/* non aperture */ +/* allowed fence registers */ +uint32_t fence_num; +uint32_t rsv2[3 ];
drivers/gpu/drm/i915/i915_drv.h struct i915_frontbuffer_tracking { · line 1485
Quoted 10 lines
drivers/gpu/drm/i915/i915_drv.h +10
-0+ } avail_rs;/* available/assigned resource */ +uint32_t rsv3[0x200 -24 ];/* pad to half page */ +/* +* The bottom half page is for response from Gfx driver to hypervisor. +* Set to reserved fields temporarily by now. +*/ +uint32_t rsv4; +uint32_t display_ready;/* ready for display owner switch */ +uint32_t rsv5[0x200 -2 ];/* pad to one page */ +};
Where is the other half of this if? Shouldn't this be defined there and included here?
drivers/gpu/drm/i915/i915_drv.h struct drm_i915_private { · line 1499
Quoted 9 lines
drivers/gpu/drm/i915/i915_drv.h +2
-0+struct drm_i915_private {struct drm_device *dev;struct kmem_cache *slab;@@ -1454,6 +1496,8 @@ struct drm_i915_private {struct intel_uncore uncore; +bool vgpu_active;
struct i915_virtual_gpu {
bool active;
} vgu;will be tidier and future proof.
Quoted 5 lines
drivers/gpu/drm/i915/i915_drv.h +1
-0+struct intel_gmbus gmbus[GMBUS_NUM_PORTS ];
drivers/gpu/drm/i915/i915_drv.h extern void intel_uncore_check_errors(struct drm_device *dev); · line 2341
Quoted 9 lines
drivers/gpu/drm/i915/i915_drv.h +6
-0@@ -2289,6 +2333,14 @@ extern void intel_uncore_check_errors(struct drm_device *dev);extern void intel_uncore_fini (struct drm_device *dev);extern void intel_uncore_forcewake_reset (struct drm_device *dev,bool restore); +extern void i915_check_vgpu (struct drm_device *dev); +static inline bool intel_vgpu_active (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv = dev->dev_private; + +return dev_priv->vgpu_active;
return to_i915(dev)->vgpu_active;
Or just use struct drm_i915_private and drop all the pointer dancing.
Quoted 10 lines
drivers/gpu/drm/i915/i915_drv.h +2
-0+} +void i915_enable_pipestat (struct drm_i915_private *dev_priv,enum pipe pipe,u32 status_mask);drivers/gpu/drm/i915/i915_gem_gtt.c +0
-0
drivers/gpu/drm/i915/i915_gem_gtt.c line 37
Quoted 8 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +5
-0@@ -30,6 +30,27 @@#include "i915_trace.h" #include "intel_drv.h" +void i915_check_vgpu (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv = dev->dev_private; +uint64_t magic; +uint32_t version;
BUILD_BUG_ON(sizeof(struct vgt_if) != VGT_PVINFO_SIZE);
drivers/gpu/drm/i915/i915_gem_gtt.c line 39
Quoted 2 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +2
-0+ + dev_priv->vgpu_active = false;
Already false on entry.
drivers/gpu/drm/i915/i915_gem_gtt.c line 51
Quoted 12 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +12
-0+ +if (!IS_HASWELL (dev)) +return ; + + magic =I915_READ64 (vgt_info_off (magic)); +if (magic !=VGT_MAGIC ) +return ; + + version = (I915_READ16 (vgt_info_off (version_major)) <<16 ) | +I915_READ16 (vgt_info_off (version_minor)); +if (version ==INTEL_VGT_IF_VERSION ) + dev_priv->vgpu_active = true;
A debug trace would be very useful here, maybe even INFO since it is useful information to the user (when enabled).
What should the failure path be? Presumably you want to disable the driver under a virtual environment with an abi mismatch.
Quoted 11 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +2
-0+} +static void bdw_setup_private_ppat (struct drm_i915_private *dev_priv);static void chv_setup_private_ppat (struct drm_i915_private *dev_priv);drivers/gpu/drm/i915/i915_reg.h +0
-0@@ -6661,4 +6661,17 @@ enum punit_power_well {#define _PALETTE_A (dev_priv->info.display_mmio_offset + 0xa000)
drivers/gpu/drm/i915/i915_reg.h enum punit_power_well { · line 6671
Quoted 10 lines
drivers/gpu/drm/i915/i915_reg.h +8
-0#define _PALETTE_B (dev_priv->info.display_mmio_offset + 0xa800) +/* The MMIO offset of the shared info between i915 and vgt driver */ +#define VGT_PVINFO_PAGE 0x78000 +#define VGT_PVINFO_SIZE 0x1000 + +#define VGT_MAGIC 0x4776544776544776/* 'vGTvGTvG' */ +#define VGT_VERSION_MAJOR 1 +#define VGT_VERSION_MINOR 0 +#define INTEL_VGT_IF_VERSION ((VGT_VERSION_MAJOR << 16) | VGT_VERSION_MINOR)
#define INTEL_VGT_IF_VERSION_ENCODE(major, minor) ((major) << 16 | (minor)) #define INTEL_VGT_IF_VERSION \
INTEL_VGT_IF_VERSION_ENCODE(VGT_VERSION_MAJOR, VGT_VERSION_MiN)
and then
version = INTEL_VGT_IF_VERSION_ENCODE(I915_READ16(vgt_info_off(version_major)),
I915_READ16(vgt_info_off(version_minor));
drivers/gpu/drm/i915/i915_reg.h enum punit_power_well { · line 6674
Quoted 3 lines
drivers/gpu/drm/i915/i915_reg.h +3
-0+ +#define vgt_info_off (x) \ + (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x)
(VGT_PVINFO_PAGE + offsetof(struct vgt_if, x))
vgt_info_off feels a little redundant given usage. Perhaps vgt_reg(x).
Now, given that these are simply trapped memory access, wouldn't it be simply to have:
struct i915_virtual_gpu {
struct vgt_if *if;
} vgu;static inline bool intel_vgpu_active(struct drm_i915_private *i915) { return i915->vgpu.if; }
then you have constructs like
void i915_check_vgpu(struct drm_i915_private *i915)
{
struct vgt_if *if;
if = i915->regs + VGT_PVINFO_PAGE;
if (if->magic != VGT_MAGIC)
return;
if (INTEL_VGT_IF_VERSION !=
INTEL_VGT_IF_VERSION_ENCODE(if->version_major,
if->version_minor))
return;
i915->vgpu.if = if;
}And later, for example:
if (intel_vgpu_active(dev_priv))
dev_priv->num_fence_regs = dev_priv->vgpu.if->fence_num;
-Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
Re: [PATCH 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
Hi Chris, thanks very much for your detailed comments! I'll try to address them as well as in your other emails.
Quoted 86 lines · Chris Wilson
Quoted 85 lines · Jike Song
From: Yu Zhang <[email protected]>
Introduce a PV INFO structure, to facilitate the Intel GVT-g technology, which is a GPU virtualization solution with mediated pass-through(previously known as XenGT). This page contains the shared information between i915 driver and the mediator. For now, this structure utilizes an area of 4K bypes on HSW GPU's unused MMIO space to support existing production. Future hardware will have the reserved window architecturally defined, and layout of the page will be added in future BSpec.
The i915 driver load routine detects if it is running in a VM by reading the contents of this PV INFO page. Thereafter a flag,
vgpu_activeis set, andintel_vgpu_active()is used by checking this flag to conclude if gpu is virtualized with Intel GVT-g. By now, it will return true only when the driver is running in the XenGT environment on HSW.Signed-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Eddie Dong <[email protected]>--- drivers/gpu/drm/i915/i915_dma.c | 2 ++ drivers/gpu/drm/i915/i915_drv.h | 52 +++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_gem_gtt.c | 21 +++++++++++++++ drivers/gpu/drm/i915/i915_reg.h | 13 ++++++++++ 4 files changed, 88 insertions(+)drivers/gpu/drm/i915/i915_dma.c +2
-0@@ -1739,6 +1739,8 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags) goto out_freewq; } +i915_check_vgpu (dev); +intel_irq_init (dev);intel_uncore_sanitize (dev);drivers/gpu/drm/i915/i915_drv.h +36
-0@@ -1442,6 +1442,48 @@ struct i915_frontbuffer_tracking {unsigned flip_bits; }; +/* +* The following structure pages are defined in GEN MMIO space +* for virtualization. (One page for now) +*/ + +struct vgt_if { +uint64_t magic;/* VGT_MAGIC */ +uint16_t version_major; +uint16_t version_minor; +uint32_t vgt_id;/* ID of vGT instance */ +uint32_t rsv1[12 ];/* pad to offset 0x40 */ +/* +* Data structure to describe the balooning info of resources. +* Each VM can only have one portion of continuous area for now. +* (May support scattered resource in future) +* (starting from offset 0x40) +*/ +struct { +/* Aperture register balooning */ +struct { +uint32_t my_base; +uint32_t my_size; + } low_gmadr;/* aperture */ +/* GMADR register balooning */ +struct { +uint32_t my_base; +uint32_t my_size; + } high_gmadr;/* non aperture */ +/* allowed fence registers */ +uint32_t fence_num; +uint32_t rsv2[3 ]; + } avail_rs;/* available/assigned resource */ +uint32_t rsv3[0x200 -24 ];/* pad to half page */ +/* +* The bottom half page is for response from Gfx driver to hypervisor. +* Set to reserved fields temporarily by now.
Quoted 9 lines
drivers/gpu/drm/i915/i915_drv.h struct i915_frontbuffer_tracking { · line 1485
Quoted 5 lines
drivers/gpu/drm/i915/i915_drv.h +5
-0+*/ +uint32_t rsv4; +uint32_t display_ready;/* ready for display owner switch */ +uint32_t rsv5[0x200 -2 ];/* pad to one page */ +};Where is the other half of this if? Shouldn't this be defined there and included here?
Among the bottom half page, current only display_ready is used by i915.
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/i915/i915_drv.h +1
-0+struct drm_i915_private {struct drm_device *dev;struct kmem_cache *slab;@@ -1454,6 +1496,8 @@ struct drm_i915_private {struct intel_uncore uncore;
Quoted 7 lines
drivers/gpu/drm/i915/i915_drv.h struct drm_i915_private { · line 1499
Quoted 1 line
drivers/gpu/drm/i915/i915_drv.h +1
-0+bool vgpu_active;struct i915_virtual_gpu { bool active; } vgu;will be tidier and future proof.
Okay.
Quoted 16 lines
Quoted 5 lines
drivers/gpu/drm/i915/i915_drv.h +1
-0+struct intel_gmbus gmbus[GMBUS_NUM_PORTS ];drivers/gpu/drm/i915/i915_drv.h extern void intel_uncore_check_errors(struct drm_device *dev); · line 2341
Quoted 9 lines
drivers/gpu/drm/i915/i915_drv.h +6
-0@@ -2289,6 +2333,14 @@ extern void intel_uncore_check_errors(struct drm_device *dev);extern void intel_uncore_fini (struct drm_device *dev);extern void intel_uncore_forcewake_reset (struct drm_device *dev,bool restore); +extern void i915_check_vgpu (struct drm_device *dev); +static inline bool intel_vgpu_active (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv = dev->dev_private; + +return dev_priv->vgpu_active;return to_i915(dev)->vgpu_active;
Quoted 2 lines
Or just use struct
drm_i915_privateand drop all the pointer dancing.
Thanks for 2 good points, I prefer the former one :)
Quoted 13 lines
Quoted 13 lines
drivers/gpu/drm/i915/i915_drv.h +2
-0+} +void i915_enable_pipestat (struct drm_i915_private *dev_priv,enum pipe pipe,u32 status_mask);drivers/gpu/drm/i915/i915_gem_gtt.c +0
-0@@ -30,6 +30,27 @@#include "i915_trace.h" #include "intel_drv.h"
Quoted 8 lines
drivers/gpu/drm/i915/i915_gem_gtt.c line 37
Quoted 5 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +5
-0+void i915_check_vgpu (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv = dev->dev_private; +uint64_t magic; +uint32_t version;
BUILD_BUG_ON(sizeof(struct vgt_if) != VGT_PVINFO_SIZE);
Okay
Quoted 5 lines
drivers/gpu/drm/i915/i915_gem_gtt.c line 39
Quoted 2 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +2
-0+ + dev_priv->vgpu_active = false;Already false on entry.
Okay
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +7
-0+ +if (!IS_HASWELL (dev)) +return ; + + magic =I915_READ64 (vgt_info_off (magic)); +if (magic !=VGT_MAGIC ) +return ;
Quoted 9 lines
drivers/gpu/drm/i915/i915_gem_gtt.c line 51
Quoted 5 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +5
-0+ + version = (I915_READ16 (vgt_info_off (version_major)) <<16 ) | +I915_READ16 (vgt_info_off (version_minor)); +if (version ==INTEL_VGT_IF_VERSION ) + dev_priv->vgpu_active = true;A debug trace would be very useful here, maybe even INFO since it is useful information to the user (when enabled).
You mean a DRM_INFO? Okay.
Quoted 3 lines
What should the failure path be? Presumably you want to disable the driver under a virtual environment with an abi mismatch.
For the failure path, we want to simply let the i915 driver think that it's running in a native environment.
Quoted 26 lines
Quoted 11 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +2
-0+} +static void bdw_setup_private_ppat (struct drm_i915_private *dev_priv);static void chv_setup_private_ppat (struct drm_i915_private *dev_priv);drivers/gpu/drm/i915/i915_reg.h +0
-0@@ -6661,4 +6661,17 @@ enum punit_power_well {#define _PALETTE_A (dev_priv->info.display_mmio_offset + 0xa000)drivers/gpu/drm/i915/i915_reg.h enum punit_power_well { · line 6671
Quoted 10 lines
drivers/gpu/drm/i915/i915_reg.h +8
-0#define _PALETTE_B (dev_priv->info.display_mmio_offset + 0xa800) +/* The MMIO offset of the shared info between i915 and vgt driver */ +#define VGT_PVINFO_PAGE 0x78000 +#define VGT_PVINFO_SIZE 0x1000 + +#define VGT_MAGIC 0x4776544776544776/* 'vGTvGTvG' */ +#define VGT_VERSION_MAJOR 1 +#define VGT_VERSION_MINOR 0 +#define INTEL_VGT_IF_VERSION ((VGT_VERSION_MAJOR << 16) | VGT_VERSION_MINOR)#define
INTEL_VGT_IF_VERSION_ENCODE(major, minor)((major) << 16 | (minor)) #defineINTEL_VGT_IF_VERSION\
INTEL_VGT_IF_VERSION_ENCODE(VGT_VERSION_MAJOR, VGT_VERSION_MiN)
Quoted 5 lines
and then
version = INTEL_VGT_IF_VERSION_ENCODE(I915_READ16(vgt_info_off(version_major)),
I915_READ16(vgt_info_off(version_minor));
Okay.
Quoted 8 lines
drivers/gpu/drm/i915/i915_reg.h enum punit_power_well { · line 6674
Quoted 3 lines
drivers/gpu/drm/i915/i915_reg.h +3
-0+ +#define vgt_info_off (x) \ + (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x)(
VGT_PVINFO_PAGE+offsetof(struct vgt_if, x))
vgt_info_offfeels a little redundant given usage. Perhapsvgt_reg(x).
Okay. What about vgtif_reg(x)?
Quoted 9 lines
Now, given that these are simply trapped memory access, wouldn't it be simply to have:
struct i915_virtual_gpu { struct vgt_if *if; } vgu;static inline bool
intel_vgpu_active(struct drm_i915_private *i915){ returni915->vgpu.if; }
Thanks for your demo code, can I do some minor change?
static inline bool intel_vgpu_active(struct drm_device *dev)
{
return to_i915(dev)->vgpu.if != NULL;
}Quoted 18 lines
then you have constructs like
voidi915_check_vgpu(struct drm_i915_private *i915){ struct vgt_if *if; if = i915->regs + VGT_PVINFO_PAGE; if (if->magic != VGT_MAGIC) return; if (INTEL_VGT_IF_VERSION != INTEL_VGT_IF_VERSION_ENCODE(if->version_major, if->version_minor)) return; i915->vgpu.if = if; }
Quoted 4 lines
And later, for example:
if (
intel_vgpu_active(dev_priv))dev_priv->num_fence_regs = dev_priv->vgpu.if->fence_num;
Thanks again for the demo code!
Quoted 3 lines
-Chris
--
Thanks,
Jike
Re: [PATCH 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
In reply to Chris WilsonQuoted 27 lines · Chris Wilson
Now, given that these are simply trapped memory access, wouldn't it be simply to have:
struct i915_virtual_gpu { struct vgt_if *if; } vgu;static inline bool
intel_vgpu_active(struct drm_i915_private *i915){ returni915->vgpu.if; }then you have constructs like
voidi915_check_vgpu(struct drm_i915_private *i915){ struct vgt_if *if; if = i915->regs + VGT_PVINFO_PAGE; if (if->magic != VGT_MAGIC) return; if (INTEL_VGT_IF_VERSION != INTEL_VGT_IF_VERSION_ENCODE(if->version_major, if->version_minor)) return; i915->vgpu.if = if; }
Quoted 5 lines
And later, for example:
if (
intel_vgpu_active(dev_priv))dev_priv->num_fence_regs = dev_priv->vgpu.if->fence_num;
Hi Chris, sorry that I didn't understand you correctly. After discussion with Yu today, I realized that unfortunately, the vgt_if can't be dereferenced directly.
There are several reasons:
- dereferencing a MMIO address will be complained by sparse(1)
- for Guest VM, such accesses will be trapped by hypervisor, and hence emulated correctly; However this doesn't work for Host(e.g. Domain 0 of Xen, the Linux host KVM resides in). For host, we used a proactive mechanism to redirect i915 MMIO accesses to vgt, the GPU device-model, for the sake of central management & sharing among VMs, including host.
Given that, though technically your code works for Guest, but after the integration of host support of iGVT, we still need to use I915_READ/I915_WRITE then. The host patches is soon to posted for your review :)
I should have realized that earlier, sorry!
Quoted 2 lines
-Chris
--
Thanks,
Jike
Re: [PATCH 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
Quoted 50 lines · Jike Song
Quoted 28 lines · Chris Wilson
Now, given that these are simply trapped memory access, wouldn't it be simply to have:
struct i915_virtual_gpu { struct vgt_if *if; } vgu;static inline bool
intel_vgpu_active(struct drm_i915_private *i915){ returni915->vgpu.if; }then you have constructs like
voidi915_check_vgpu(struct drm_i915_private *i915){ struct vgt_if *if; if = i915->regs + VGT_PVINFO_PAGE; if (if->magic != VGT_MAGIC) return; if (INTEL_VGT_IF_VERSION != INTEL_VGT_IF_VERSION_ENCODE(if->version_major, if->version_minor)) return; i915->vgpu.if = if; }Quoted 5 lines
And later, for example:
if (
intel_vgpu_active(dev_priv))dev_priv->num_fence_regs = dev_priv->vgpu.if->fence_num;Hi Chris, sorry that I didn't understand you correctly. After discussion with Yu today, I realized that unfortunately, the
vgt_ifcan't be dereferenced directly.There are several reasons:
- dereferencing a MMIO address will be complained by sparse(1)
- for Guest VM, such accesses will be trapped by hypervisor, and hence emulated correctly; However this doesn't work for Host(e.g. Domain 0 of Xen, the Linux host KVM resides in). For host, we used a proactive mechanism to redirect i915 MMIO accesses to vgt, the GPU device-model, for the sake of central management & sharing among VMs, including host.
Quoted 7 lines
Given that, though technically your code works for Guest, but after the integration of host support of iGVT, we still need to use I915_READ/I915_WRITE
then. The host patches is soon to posted for your review :)I should have realized that earlier, sorry!
Hi Chris,
Sorry, I also should have noticed this issue earlier.
To my understanding, the reason you proposed to use the "structvgt_if *if" in struct vgpu, to replace the previous vgpu_active, is to simplify the mmio accesses in our patches. This suggestion works fine from the guest & native point of view. However, just like Jike's mail said, this change may not work for the host side, which also need to visit the PVINFO page from time to time. So, could we still keep the vgpu_active flag when detecting virtual gpu, and read the mmio registers in PVINFO structure by I915_READ?
Thanks
Yu
Quoted 13 lines
Quoted 3 lines
-Chris
--
Thanks,
Jike
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
In reply to Jike SongQuoted 40 lines · Jike Song
Quoted 27 lines · Chris Wilson
Now, given that these are simply trapped memory access, wouldn't it be simply to have:
struct i915_virtual_gpu { struct vgt_if *if; } vgu;static inline bool
intel_vgpu_active(struct drm_i915_private *i915){ returni915->vgpu.if; }then you have constructs like
voidi915_check_vgpu(struct drm_i915_private *i915){ struct vgt_if *if; if = i915->regs + VGT_PVINFO_PAGE; if (if->magic != VGT_MAGIC) return; if (INTEL_VGT_IF_VERSION != INTEL_VGT_IF_VERSION_ENCODE(if->version_major, if->version_minor)) return; i915->vgpu.if = if; }Quoted 5 lines
And later, for example:
if (
intel_vgpu_active(dev_priv))dev_priv->num_fence_regs = dev_priv->vgpu.if->fence_num;Hi Chris, sorry that I didn't understand you correctly. After discussion with Yu today, I realized that unfortunately, the
vgt_ifcan't be dereferenced directly.There are several reasons:
Quoted 8 lines
- dereferencing a MMIO address will be complained by sparse(1)
- for Guest VM, such accesses will be trapped by hypervisor, and hence emulated correctly; However this doesn't work for Host(e.g. Domain 0 of Xen, the Linux host KVM resides in). For host, we used a proactive mechanism to redirect i915 MMIO accesses to vgt, the GPU device-model, for the sake of central management & sharing among VMs, including host.
You only need to be careful during vgpu detection. After that you know everything is safe. If you do the detection during intel_uncore_init(), or similar, you can use raw mmio access and explict sparse annotations to do the right thing.
-Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
Re: [PATCH 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
Quoted 50 lines · Chris Wilson
Quoted 40 lines · Jike Song
Quoted 27 lines · Chris Wilson
Now, given that these are simply trapped memory access, wouldn't it be simply to have:
struct i915_virtual_gpu { struct vgt_if *if; } vgu;static inline bool
intel_vgpu_active(struct drm_i915_private *i915){ returni915->vgpu.if; }then you have constructs like
voidi915_check_vgpu(struct drm_i915_private *i915){ struct vgt_if *if; if = i915->regs + VGT_PVINFO_PAGE; if (if->magic != VGT_MAGIC) return; if (INTEL_VGT_IF_VERSION != INTEL_VGT_IF_VERSION_ENCODE(if->version_major, if->version_minor)) return; i915->vgpu.if = if; }Quoted 5 lines
And later, for example:
if (
intel_vgpu_active(dev_priv))dev_priv->num_fence_regs = dev_priv->vgpu.if->fence_num;Hi Chris, sorry that I didn't understand you correctly. After discussion with Yu today, I realized that unfortunately, the
vgt_ifcan't be dereferenced directly.There are several reasons:
Quoted 8 lines
- dereferencing a MMIO address will be complained by sparse(1)
- for Guest VM, such accesses will be trapped by hypervisor, and hence emulated correctly; However this doesn't work for Host(e.g. Domain 0 of Xen, the Linux host KVM resides in). For host, we used a proactive mechanism to redirect i915 MMIO accesses to vgt, the GPU device-model, for the sake of central management & sharing among VMs, including host.
Quoted 5 lines
You only need to be careful during vgpu detection. After that you know everything is safe. If you do the detection during
intel_uncore_init(), or similar, you can use raw mmio access and explict sparse annotations to do the right thing.
-Chris
Hi Chris,
Thanks for your suggestion, however, it addressed issue #1 but not issue #2. I'd like to give a detailed explanation :)
For Guest i915 driver, when accessing a MMIO reg, it goes:
whatever I915_READ/readl or direct dereferencing(addr)
addr is translated to gpa(guest physical address) by guest paging structure
hypervisor trapped the gpa access and forward it to vgt
vgt_emulate_read32(vgt instance of guest, gpa)
The real problem is, when running as the host gpu driver, i915 MMIO/GTT accesses are:
1) every difficult to trap, say the domain 0 of Xen hypervisor; or
2) impossible to trap, say KVM(In KVM, the host i915 is running in the very same privilege level and root mode as KVM hypervisor.)
So, for Host i915 driver, when accessing a MMIO reg, it goes:
I915_READ(addr) gen6_read32(addr) offset = addr - dev_priv->regs vgt_mmio_readl(the vgt instance of host, offset) .. some processing .. if (passed-throuth) readl(offset + dev_priv->regs) else pa = BAR0 + offset vgt_emulate_read32(vgt instance of host, pa)
The vgt_if corresponds to the PVINFO page, not passed-through(actually it doesn't physically exist), should fall into the "else" above.
Given that ... Yes we can dereference pointers for guest here, but consider that we'll deal with host in the future ...
--
Thanks,
Jike
Re: [PATCH 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
Quoted 95 lines · Jike Song
Quoted 51 lines · Chris Wilson
Quoted 41 lines · Jike Song
Quoted 28 lines · Chris Wilson
Now, given that these are simply trapped memory access, wouldn't it be simply to have:
struct i915_virtual_gpu { struct vgt_if *if; } vgu;static inline bool
intel_vgpu_active(struct drm_i915_private *i915){ returni915->vgpu.if; }then you have constructs like
voidi915_check_vgpu(struct drm_i915_private *i915){ struct vgt_if *if; if = i915->regs + VGT_PVINFO_PAGE; if (if->magic != VGT_MAGIC) return; if (INTEL_VGT_IF_VERSION != INTEL_VGT_IF_VERSION_ENCODE(if->version_major, if->version_minor)) return; i915->vgpu.if = if; }Quoted 5 lines
And later, for example:
if (
intel_vgpu_active(dev_priv))dev_priv->num_fence_regs = dev_priv->vgpu.if->fence_num;Hi Chris, sorry that I didn't understand you correctly. After discussion with Yu today, I realized that unfortunately, the
vgt_ifcan't be dereferenced directly.There are several reasons:
Quoted 8 lines
- dereferencing a MMIO address will be complained by sparse(1)
- for Guest VM, such accesses will be trapped by hypervisor, and hence emulated correctly; However this doesn't work for Host(e.g. Domain 0 of Xen, the Linux host KVM resides in). For host, we used a proactive mechanism to redirect i915 MMIO accesses to vgt, the GPU device-model, for the sake of central management & sharing among VMs, including host.
Quoted 5 lines
You only need to be careful during vgpu detection. After that you know everything is safe. If you do the detection during
intel_uncore_init(), or similar, you can use raw mmio access and explict sparse annotations to do the right thing.
-ChrisHi Chris,
Thanks for your suggestion, however, it addressed issue #1 but not issue #2.
I'd like to give a detailed explanation :)For Guest i915 driver, when accessing a MMIO reg, it goes:
whatever I915_READ/readl or direct dereferencing(addr)
addr is translated to gpa(guest physical address) by guest
paging structure
hypervisor trapped the gpa access and forward it to vgt
vgt_emulate_read32(vgt instance of guest, gpa)The real problem is, when running as the host gpu driver, i915 MMIO/GTT accesses are:
1) every difficult to trap, say the domain 0 of Xen hypervisor; or
2) impossible to trap, say KVM(In KVM, the host i915 is running in
the very same
privilege level and root mode as KVM hypervisor.)So, for Host i915 driver, when accessing a MMIO reg, it goes:
I915_READ(addr)
gen6_read32(addr)
offset = addr -dev_priv->regsvgt_mmio_readl(the vgt instance of host, offset)
.. some processing ..
if (passed-throuth)
readl(offset + dev_priv->regs)
else
pa = BAR0 + offsetvgt_emulate_read32(vgt instance of host, pa)
Quoted 8 lines
The
vgt_ifcorresponds to the PVINFO page, not passed-through(actually it doesn't physically exist),
should fall into the "else" above.Given that ... Yes we can dereference pointers for guest here, but consider that we'll
deal with host in the future ...
Hi Chris,
Any comments on this issue?
The host side patches were sent out by Jike last week. As you can see
in his patch, the I915_READ/WRITEs are redefined, and the MMIO accesses
are mediated to vgt, by offset of the regs, instead of by read/write (dev_priv->regs + offset) for host i915.
So, in order to let i915 in the host side access the PVINFO page
successfully, could we still use the I915_READ/WRTE, and keep thevgpu_active flag when detecting virtual gpu? :)
Thanks
Yu
Quoted 10 lines
--
Thanks,
Jike
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
In reply to Jike SongQuoted 80 lines · Jike Song
From: Yu Zhang <[email protected]>
Introduce a PV INFO structure, to facilitate the Intel GVT-g technology, which is a GPU virtualization solution with mediated pass-through(previously known as XenGT). This page contains the shared information between i915 driver and the mediator. For now, this structure utilizes an area of 4K bypes on HSW GPU's unused MMIO space to support existing production. Future hardware will have the reserved window architecturally defined, and layout of the page will be added in future BSpec.
The i915 driver load routine detects if it is running in a VM by reading the contents of this PV INFO page. Thereafter a flag,
vgpu_activeis set, andintel_vgpu_active()is used by checking this flag to conclude if gpu is virtualized with Intel GVT-g. By now, it will return true only when the driver is running in the XenGT environment on HSW.Signed-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Eddie Dong <[email protected]>--- drivers/gpu/drm/i915/i915_dma.c | 2 ++ drivers/gpu/drm/i915/i915_drv.h | 52 +++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_gem_gtt.c | 21 +++++++++++++++ drivers/gpu/drm/i915/i915_reg.h | 13 ++++++++++ 4 files changed, 88 insertions(+)drivers/gpu/drm/i915/i915_dma.c +2
-0@@ -1739,6 +1739,8 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags) goto out_freewq; } +i915_check_vgpu (dev); +intel_irq_init (dev);intel_uncore_sanitize (dev);drivers/gpu/drm/i915/i915_drv.h +31
-0@@ -1442,6 +1442,48 @@ struct i915_frontbuffer_tracking {unsigned flip_bits; }; +/* +* The following structure pages are defined in GEN MMIO space +* for virtualization. (One page for now) +*/ + +struct vgt_if { +uint64_t magic;/* VGT_MAGIC */ +uint16_t version_major; +uint16_t version_minor; +uint32_t vgt_id;/* ID of vGT instance */ +uint32_t rsv1[12 ];/* pad to offset 0x40 */ +/* +* Data structure to describe the balooning info of resources. +* Each VM can only have one portion of continuous area for now. +* (May support scattered resource in future) +* (starting from offset 0x40) +*/ +struct { +/* Aperture register balooning */ +struct { +uint32_t my_base; +uint32_t my_size; + } low_gmadr;/* aperture */ +/* GMADR register balooning */ +struct { +uint32_t my_base; +uint32_t my_size; + } high_gmadr;/* non aperture */ +/* allowed fence registers */ +uint32_t fence_num; +uint32_t rsv2[3 ];
drivers/gpu/drm/i915/i915_drv.h struct i915_frontbuffer_tracking { · line 1485
Quoted 10 lines
drivers/gpu/drm/i915/i915_drv.h +10
-0+ } avail_rs;/* available/assigned resource */ +uint32_t rsv3[0x200 -24 ];/* pad to half page */ +/* +* The bottom half page is for response from Gfx driver to hypervisor. +* Set to reserved fields temporarily by now. +*/ +uint32_t rsv4; +uint32_t display_ready;/* ready for display owner switch */ +uint32_t rsv5[0x200 -2 ];/* pad to one page */ +};
Needs __packed to make sure we exactly match the hw layout. We do the same with the vbt headers, mostly as documentation.
Quoted 69 lines
drivers/gpu/drm/i915/i915_drv.h +11
-0+struct drm_i915_private {struct drm_device *dev;struct kmem_cache *slab;@@ -1454,6 +1496,8 @@ struct drm_i915_private {struct intel_uncore uncore; +bool vgpu_active; +struct intel_gmbus gmbus[GMBUS_NUM_PORTS ];@@ -2289,6 +2333,14 @@ extern void intel_uncore_check_errors(struct drm_device *dev);extern void intel_uncore_fini (struct drm_device *dev);extern void intel_uncore_forcewake_reset (struct drm_device *dev,bool restore); +extern void i915_check_vgpu (struct drm_device *dev); +static inline bool intel_vgpu_active (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv = dev->dev_private; + +return dev_priv->vgpu_active; +} +void i915_enable_pipestat (struct drm_i915_private *dev_priv,enum pipe pipe,u32 status_mask);drivers/gpu/drm/i915/i915_gem_gtt.c +21
-0@@ -30,6 +30,27 @@#include "i915_trace.h" #include "intel_drv.h" +void i915_check_vgpu (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv = dev->dev_private; +uint64_t magic; +uint32_t version; + + dev_priv->vgpu_active = false; + +if (!IS_HASWELL (dev)) +return ; + + magic =I915_READ64 (vgt_info_off (magic)); +if (magic !=VGT_MAGIC ) +return ; + + version = (I915_READ16 (vgt_info_off (version_major)) <<16 ) | +I915_READ16 (vgt_info_off (version_minor)); +if (version ==INTEL_VGT_IF_VERSION ) + dev_priv->vgpu_active = true; +} +static void bdw_setup_private_ppat (struct drm_i915_private *dev_priv);static void chv_setup_private_ppat (struct drm_i915_private *dev_priv);drivers/gpu/drm/i915/i915_reg.h +1
-0@@ -6661,4 +6661,17 @@ enum punit_power_well {#define _PALETTE_A (dev_priv->info.display_mmio_offset + 0xa000)#define _PALETTE_B (dev_priv->info.display_mmio_offset + 0xa800) +/* The MMIO offset of the shared info between i915 and vgt driver */
drivers/gpu/drm/i915/i915_reg.h enum punit_power_well { · line 6674
Quoted 10 lines
drivers/gpu/drm/i915/i915_reg.h +10
-0+#define VGT_PVINFO_PAGE 0x78000 +#define VGT_PVINFO_SIZE 0x1000 + +#define VGT_MAGIC 0x4776544776544776/* 'vGTvGTvG' */ +#define VGT_VERSION_MAJOR 1 +#define VGT_VERSION_MINOR 0 +#define INTEL_VGT_IF_VERSION ((VGT_VERSION_MAJOR << 16) | VGT_VERSION_MINOR) + +#define vgt_info_off (x) \ + (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x)
I think a new header file i915_vgt.h for these definitions would be good. i915_reg.h is giant already ...
-Daniel
Quoted 10 lines
drivers/gpu/drm/i915/i915_reg.h +2
-0+ +#endif /* _I915_REG_H_ */ Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Re: [PATCH 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
Quoted 2 lines · Daniel Vetter
I think a new header file
i915_vgt.hfor these definitions would be good.i915_reg.his giant already ...
Separate header would also lend itself well for some kerneldoc DOC:
overview section and for properly documenting all the vgt_ interface functions. See e.g. the cmd parser for a nice example of how to integrate a documentation for new features into our drm driver docbook. -Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Re: [PATCH 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
In reply to Daniel VetterQuoted 21 lines
From: Intel-gfx [mailto:[email protected]] On Behalf Of Daniel Vetter
Review
Quoted 1 line
(quoted diff) +0
-0
b/drivers/gpu/drm/i915/i915_reg.hQuoted 5 lines
index b65bdfc..a70f12e 100644
drivers/gpu/drm/i915/i915_reg.h +0
-0@@ -6661,4 +6661,17 @@ enum punit_power_well {#define _PALETTE_A (dev_priv->info.display_mmio_offset + 0xa000)drivers/gpu/drm/i915/i915_reg.h enum punit_power_well { · line 6671
Quoted 10 lines
drivers/gpu/drm/i915/i915_reg.h +8
-0#define _PALETTE_B (dev_priv->info.display_mmio_offset + 0xa800) +/* The MMIO offset of the shared info between i915 and vgt driver */ +#define VGT_PVINFO_PAGE 0x78000 +#define VGT_PVINFO_SIZE 0x1000 + +#define VGT_MAGIC 0x4776544776544776/* 'vGTvGTvG' */ +#define VGT_VERSION_MAJOR 1 +#define VGT_VERSION_MINOR 0 +#define INTEL_VGT_IF_VERSION ((VGT_VERSION_MAJOR << 16) |
VGT_VERSION_MINOR)
Quoted 8 lines
drivers/gpu/drm/i915/i915_reg.h enum punit_power_well { · line 6674
Quoted 3 lines
drivers/gpu/drm/i915/i915_reg.h +3
-0+ +#define vgt_info_off (x) \ + (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x)I think a new header file
i915_vgt.hfor these definitions would be good.i915_reg.his giant already ...
-Daniel
sure we can follow this suggestion. Actually our original implementation has the definition separately defined. Just because we saw current i915 has a single register definition header, so thought we should adapt to that. :-)
Thanks
Kevin
PATCH 2/8 drm/i915: Adds graphic address space ballooning logic
In reply to Jike SongFrom: Yu Zhang <[email protected]>
In XenGT, the global graphic memory space is partitioned by multiple vgpu instances in different VMs. The ballooning code is added in i915_gem_setup_global_gtt(), utilizing the drm mm allocator APIs to mark the graphic address space which are partitioned out to other vgpus as reserved.
One special treatment for this scenario is that the guard page is added at the end of both aperture and non-aperture spaces. This is to follow the behavior pattern in native case. However here we have a question: does the prefetch issue mentioned at the begining of i915_gem_setup_global_gtt() happens for all the GTT entries, or just at then end of the BAR0 space? If all entries may have prefetch issues, then this special guard page is necessary, to protect unexpected accesses into GTT entries partitioned out by other VMs. Otherwise, we may only need one guard page at the end of the physical GTT space.
--- drivers/gpu/drm/i915/i915_gem_gtt.c | 179 ++++++++++++++++++++++++++++++++++-- 1 file changed, 171 insertions(+), 8 deletions(-)
drivers/gpu/drm/i915/i915_gem_gtt.c +171 -8
@@ -30,6 +30,159 @@#include "i915_trace.h" #include "intel_drv.h" +struct _balloon_info_ { +/* +* There are up to 2 regions per low/high GM that +* might be ballooned. Here, index 0/1 is for low +* GM, 2/3 for high GM. +*/ +struct drm_mm_node space[4 ]; +} bl_info; + +void intel_vgt_deballoon (void ) +{ +int i; + +DRM_INFO ("VGT deballoon.\n" ); + +for (i =0 ; i <4 ; i++) { +if (bl_info.space[i].allocated) +drm_mm_remove_node (&bl_info.space[i]); + } + +memset (&bl_info,0 ,sizeof (bl_info)); +} + +static int vgt_balloon_space (struct drm_mm *mm, +struct drm_mm_node *node, +unsigned long start,unsigned long end) +{ +unsigned long size = end - start; + +if (start == end) +return -EINVAL ; + +DRM_INFO ("balloon space: range [ 0x%lx - 0x%lx ] %lu KB.\n" , + start, end, size /1024 ); + +return drm_mm_insert_node_in_range_generic (mm, node, size,0 ,0 , + start, end, +DRM_MM_SEARCH_DEFAULT , +DRM_MM_CREATE_DEFAULT ); +} + +static int intel_vgt_balloon (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_address_space *ggtt_vm = &dev_priv->gtt.base; + +unsigned long low_gm_base, low_gm_size, low_gm_end; +unsigned long high_gm_base, high_gm_size, high_gm_end; +int ret; + + low_gm_base =I915_READ (vgt_info_off (avail_rs.low_gmadr.my_base)); + low_gm_size =I915_READ (vgt_info_off (avail_rs.low_gmadr.my_size)); + high_gm_base =I915_READ (vgt_info_off (avail_rs.high_gmadr.my_base)); + high_gm_size =I915_READ (vgt_info_off (avail_rs.high_gmadr.my_size)); + + low_gm_end = low_gm_base + low_gm_size; + high_gm_end = high_gm_base + high_gm_size; + +DRM_INFO ("VGT ballooning configuration:\n" ); +DRM_INFO ("Low GM: base 0x%lx size %ldKB\n" , + low_gm_base, low_gm_size /1024 ); +DRM_INFO ("High GM: base 0x%lx size %ldKB\n" , + high_gm_base, high_gm_size /1024 ); + +if (low_gm_base < ggtt_vm->start + || low_gm_end > dev_priv->gtt.mappable_end + || high_gm_base < dev_priv->gtt.mappable_end + || high_gm_end > ggtt_vm->start + ggtt_vm->total) { +DRM_ERROR ("Invalid ballooning configuration!\n" ); +return -EINVAL ; + } + +memset (&bl_info,0 ,sizeof (bl_info)); + +/* High GM ballooning */ +if (high_gm_base > dev_priv->gtt.mappable_end) { + ret =vgt_balloon_space (&ggtt_vm->mm, + &bl_info.space[2 ], + dev_priv->gtt.mappable_end, + high_gm_base); + +if (ret) + goto err; + } + +if (high_gm_end <= ggtt_vm->start + ggtt_vm->total) { +/* +* We need a guard page per low/high GM for each vgpu. +* Here, leave one page at the end of the allocated high +* GM reserved to the scratch page, and balloon it out, +* to serve as guard page. +*/ +if (high_gm_size >PAGE_SIZE ) { + high_gm_size -=PAGE_SIZE ; + ggtt_vm->clear_range (ggtt_vm, + high_gm_base + high_gm_size, +PAGE_SIZE , true); + }else { +BUG_ON (high_gm_size !=0 ); + } + + ret =vgt_balloon_space (&ggtt_vm->mm, + &bl_info.space[3 ], + high_gm_base + high_gm_size, + ggtt_vm->start + ggtt_vm->total); +if (ret) + goto err; + } + +/* Low GM ballooning */ +if (low_gm_base > ggtt_vm->start) { + ret =vgt_balloon_space (&ggtt_vm->mm, + &bl_info.space[0 ], + ggtt_vm->start, low_gm_base); + +if (ret) + goto err; + } + +if (low_gm_end <= dev_priv->gtt.mappable_end) { +/* +* We need a guard page per low/high GM for each vgpu. +* Here, leave one page at the end of the allocated low +* GM reserved to the scratch page, and balloon it out, +* to serve as guard page. +*/ +if (low_gm_size >PAGE_SIZE ) { + low_gm_size -=PAGE_SIZE ; + ggtt_vm->clear_range (ggtt_vm, + low_gm_base + low_gm_size, +PAGE_SIZE , true); + }else { +BUG_ON (low_gm_size !=0 ); + } + + ret =vgt_balloon_space (&ggtt_vm->mm, + &bl_info.space[1 ], + low_gm_base + low_gm_size, + dev_priv->gtt.mappable_end); + +if (ret) + goto err; + } + +DRM_INFO ("VGT balloon successfully\n" ); +return 0 ; + +err: +DRM_ERROR ("VGT balloon fail\n" ); +intel_vgt_deballoon (); +return -ENOMEM ; +} +void i915_check_vgpu (struct drm_device *dev) {struct drm_i915_private *dev_priv = dev->dev_private;
@@ -1682,8 +1835,21 @@ int i915_gem_setup_global_gtt(struct drm_device *dev,BUG_ON (mappable_end > end); -/* Subtract the guard page ... */ -drm_mm_init (&ggtt_vm->mm, start, end - start -PAGE_SIZE ); + dev_priv->gtt.base.start = start; + dev_priv->gtt.base.total = end - start; + +if (intel_vgpu_active (dev)) { +drm_mm_init (&ggtt_vm->mm, start, end - start); + ret =intel_vgt_balloon (dev); +if (ret) +return ret; + }else { +/* Subtract the guard page ... */ +drm_mm_init (&ggtt_vm->mm, start, end - start -PAGE_SIZE ); +/* Clear the reserved guard page */ + ggtt_vm->clear_range (ggtt_vm, end -PAGE_SIZE ,PAGE_SIZE , true); + } +if (!HAS_LLC (dev)) dev_priv->gtt.base.mm.color_adjust = i915_gtt_color_adjust;
@@ -1703,9 +1869,6 @@ int i915_gem_setup_global_gtt(struct drm_device *dev, obj->has_global_gtt_mapping =1 ; } - dev_priv->gtt.base.start = start; - dev_priv->gtt.base.total = end - start; -/* Clear any non-preallocated blocks */ drm_mm_for_each_hole (entry, &ggtt_vm->mm, hole_start, hole_end) { DRM_DEBUG_KMS("clearing unused GTT space: [%lx, %lx]\n",
@@ -1714,9 +1877,6 @@ int i915_gem_setup_global_gtt(struct drm_device *dev, hole_end - hole_start, true); } -/* And finally clear the reserved guard page */ - ggtt_vm->clear_range (ggtt_vm, end -PAGE_SIZE ,PAGE_SIZE , true); -if (USES_PPGTT (dev) && !USES_FULL_PPGTT (dev)) {struct i915_hw_ppgtt *ppgtt;
@@ -1757,6 +1917,9 @@ void i915_global_gtt_cleanup(struct drm_device *dev) }if (drm_mm_initialized (&vm->mm)) { +if (intel_vgpu_active (dev)) +intel_vgt_deballoon (); +drm_mm_takedown (&vm->mm);list_del (&vm->global_link); }
Signature · 1 line
1.9.1
Re: [PATCH 2/8] drm/i915: Adds graphic address space ballooning logic
Quoted 13 lines · Jike Song
From: Yu Zhang <[email protected]>
In XenGT, the global graphic memory space is partitioned by multiple vgpu instances in different VMs. The ballooning code is added in
i915_gem_setup_global_gtt(), utilizing the drm mm allocator APIs to mark the graphic address space which are partitioned out to other vgpus as reserved.
One special treatment for this scenario is that the guard page is added at the end of both aperture and non-aperture spaces. This is to follow the behavior pattern in native case. However here we have a question: does the prefetch issue mentioned at the begining ofi915_gem_setup_global_gtt()happens for all the GTT entries, or just at then end of the BAR0 space?
The CS prefetcher happens everywhere and so can read from the end of one range into the beginning of another clients, so requires consideration if not actualy a guard page. The very last entry in the whole GTT must be a guard page (or allocated to something not prefetchable).
Quoted 4 lines
If all entries may have prefetch issues,
then this special guard page is necessary, to protect unexpected accesses into GTT entries partitioned out by other VMs. Otherwise, we may only need one guard page at the end of the physical GTT space.
I am a bit dubious how this works when userspace still believes that it can access the whole mappable aperture, and then how every driver attempts to pin its own planes, rings and whatnot (since it still believes that it is talking to the actual hardware and that the hardware requires access to its virtual address). The host should be able to move the ranges around in order to accommodate userspace in any particular guest (hence a balloon interface I presume). But I don't see how that is possible, and you don't explain it either.
The implementation also looks backwards. To work correctly with the GTT allocator, you need to preallocate the reserved space such that it can only allocate from the allowed ranges. Similarly, it should evict any conflicting nodes when deballooning.
-Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
Re: [PATCH 2/8] drm/i915: Adds graphic address space ballooning logic
Quoted 7 lines · Chris Wilson
Quoted 5 lines · Jike Song
From: Yu Zhang <[email protected]>
If all entries may have prefetch issues,
then this special guard page is necessary, to protect unexpected accesses into GTT entries partitioned out by other VMs. Otherwise, we may only need one guard page at the end of the physical GTT space.
Quoted 8 lines
I am a bit dubious how this works when userspace still believes that it can access the whole mappable aperture, and then how every driver attempts to pin its own planes, rings and whatnot (since it still believes that it is talking to the actual hardware and that the hardware requires access to its virtual address). The host should be able to move the ranges around in order to accommodate userspace in any particular guest (hence a balloon interface I presume). But I don't see how that is possible, and you don't explain it either.
Yeah this is something we need to fix, either by pimping i915 fault support to be able to split up a big bo into chunks, or by telling userspace about the massively reduced contiguous mapping size. It should be tracked somewhere as a todo task ...
-Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Re: [PATCH 2/8] drm/i915: Adds graphic address space ballooning logic
In reply to Chris WilsonQuoted 18 lines
From: Chris Wilson
Sent: Friday, September 19, 2014 1:06 AMQuoted 13 lines · Jike Song
From: Yu Zhang <[email protected]>
In XenGT, the global graphic memory space is partitioned by multiple vgpu instances in different VMs. The ballooning code is added in
i915_gem_setup_global_gtt(), utilizing the drm mm allocator APIs to mark the graphic address space which are partitioned out to other vgpus as reserved.
One special treatment for this scenario is that the guard page is added at the end of both aperture and non-aperture spaces. This is to follow the behavior pattern in native case. However here we have a question: does the prefetch issue mentioned at the begining ofi915_gem_setup_global_gtt()happens for all the GTT entries, or just at then end of the BAR0 space?
Quoted 4 lines
The CS prefetcher happens everywhere and so can read from the end of one range into the beginning of another clients, so requires consideration if not actualy a guard page. The very last entry in the whole GTT must be a guard page (or allocated to something not prefetchable).
So this only applies to the very last page of the whole GTT, right? for normal non-present GTT entries, suppose CS prefetcher should behave correctly since that can happen anywhere.
If it's true, possibly we don't need reserve a guard page for each partition, but only reserve one for the last partition at end of the whole GTT. Or keep current way is also fine, which has the unified policy for all partitions, and avoid undesired effect (not sure any) due to bad setting of the 1st GTT entry in adjacent partition.
Quoted 6 lines
Quoted 5 lines
If all entries may have prefetch issues,
then this special guard page is necessary, to protect unexpected accesses into GTT entries partitioned out by other VMs. Otherwise, we may only need one guard page at the end of the physical GTT space.
Quoted 8 lines
I am a bit dubious how this works when userspace still believes that it can access the whole mappable aperture, and then how every driver attempts to pin its own planes, rings and whatnot (since it still believes that it is talking to the actual hardware and that the hardware requires access to its virtual address). The host should be able to move the ranges around in order to accommodate userspace in any particular guest (hence a balloon interface I presume). But I don't see how that is possible, and you don't explain it either.
for this we discussed with Jesse/Daniel earlier. In an ideal world user space should assume hardware resource knowledge, i.e. it should get all available resource from KMD. In such case once ballooning is completed, all the clients will only use allocated portion. However as you said now there's assumption in user space, which will fail in XenGT say assuming 256MB aperture but only have 64MB allocated. Jesse/Daniel discussed two ways to improve this situation, either add a lightweight page fault mechanism to catch the fault, or change mesa to remove the assumption. That can be an orthogonal effort with this patch set.
for displays, we do have a problem for multi-monitor support, if framebuffer size exceeds allocated memory. But that's the same situation on bare metal, just the matter of different level of capability limitation.
and our balloon is static, i.e. everything settled down in the driver load phase. dynamic ballooning is very complex, since the specific page may be referenced in kernel/user level, in different structures, etc. That's a good research topic, but let's pursue simple solution first. :-)
Quoted 5 lines
The implementation also looks backwards. To work correctly with the GTT allocator, you need to preallocate the reserved space such that it can only allocate from the allowed ranges. Similarly, it should evict any conflicting nodes when deballooning.
Could you elaborate a bit for above suggestion?
Quoted 8 lines
-Chris
--
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH 2/8] drm/i915: Adds graphic address space ballooning logic
Quoted 7 lines · Kevin
Quoted 5 lines
From: Chris Wilson
The implementation also looks backwards. To work correctly with the GTT allocator, you need to preallocate the reserved space such that it can only allocate from the allowed ranges. Similarly, it should evict any conflicting nodes when deballooning.Could you elaborate a bit for above suggestion?
My expectation was that the dev_priv->gtt.base.vm would contain exactly two holes after setup (in the mappable and non-mappable range). To do that you would explicitly reserve everything barred from this client using a set of drm_mm_reserve_node()
-Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
Re: [PATCH 2/8] drm/i915: Adds graphic address space ballooning logic
Quoted 7 lines · Chris Wilson
Quoted 6 lines · Kevin
Quoted 5 lines
From: Chris Wilson
The implementation also looks backwards. To work correctly with the GTT allocator, you need to preallocate the reserved space such that it can only allocate from the allowed ranges. Similarly, it should evict any conflicting nodes when deballooning.
Quoted 6 lines
Quoted 1 line
Could you elaborate a bit for above suggestion?
My expectation was that the
dev_priv->gtt.base.vmwould contain exactly two holes after setup (in the mappable and non-mappable range). To do that you would explicitly reserve everything barred from this client using a set ofdrm_mm_reserve_node()
Essentially a reserve_node implements what you open-code with insert_node_range right now.
One issue aside with both this and with the PDE reservations for gen7 is that there are now other thins in the ggtt drm_mm allocator than just gem objects. Which means our debugfs files are now less useful.
It might be useful to augment that dumper with one that dumps everything. We could add a few bits of driver-private tags in drm_mm_node (there's space) to figure out what kind of object it is. Would be a great follow-up task.
-Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Re: [PATCH 2/8] drm/i915: Adds graphic address space ballooning logic
Quoted 10 lines · Daniel Vetter
Quoted 9 lines · Chris Wilson
Quoted 7 lines · Kevin
Quoted 5 lines
From: Chris Wilson
The implementation also looks backwards. To work correctly with the GTT allocator, you need to preallocate the reserved space such that it can only allocate from the allowed ranges. Similarly, it should evict any conflicting nodes when deballooning.Could you elaborate a bit for above suggestion?
Quoted 7 lines
Quoted 4 lines
My expectation was that the
dev_priv->gtt.base.vmwould contain exactly two holes after setup (in the mappable and non-mappable range). To do that you would explicitly reserve everything barred from this client using a set ofdrm_mm_reserve_node()Essentially a
reserve_nodeimplements what you open-code withinsert_node_rangeright now.
Heh, there is a big difference. One inserts exactly where you ask and fails if it conflicts, the other inserts where it feels like within that range.
Quoted 8 lines
One issue aside with both this and with the PDE reservations for gen7 is that there are now other thins in the ggtt
drm_mmallocator than just gem objects. Which means our debugfs files are now less useful.It might be useful to augment that dumper with one that dumps everything. We could add a few bits of driver-private tags in
drm_mm_node(there's space) to figure out what kind of object it is. Would be a great follow-up task.
I think moving the other way and making them all objects so that we can tie them into evection and the shrinker, use more interesting allocation strategies, improve integration with debugging etc.
-Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
Re: [PATCH 2/8] drm/i915: Adds graphic address space ballooning logic
Quoted 16 lines · Chris Wilson
Quoted 15 lines · Daniel Vetter
Quoted 13 lines · Chris Wilson
Quoted 7 lines · Kevin
Quoted 5 lines
From: Chris Wilson
The implementation also looks backwards. To work correctly with the GTT allocator, you need to preallocate the reserved space such that it can only allocate from the allowed ranges. Similarly, it should evict any conflicting nodes when deballooning.Could you elaborate a bit for above suggestion?
My expectation was that the
dev_priv->gtt.base.vmwould contain exactly two holes after setup (in the mappable and non-mappable range). To do that you would explicitly reserve everything barred from this client using a set ofdrm_mm_reserve_node()
Quoted 6 lines
Quoted 2 lines
Essentially a
reserve_nodeimplements what you open-code withinsert_node_rangeright now.Heh, there is a big difference. One inserts exactly where you ask and fails if it conflicts, the other inserts where it feels like within that range.
Well if the the requested size matches the range exactly then it will be the same. Which iirc is what's going on here I think.
Quoted 4 lines
Quoted 4 lines
One issue aside with both this and with the PDE reservations for gen7 is that there are now other thins in the ggtt
drm_mmallocator than just gem objects. Which means our debugfs files are now less useful.
Quoted 8 lines
Quoted 4 lines
It might be useful to augment that dumper with one that dumps everything. We could add a few bits of driver-private tags in
drm_mm_node(there's space) to figure out what kind of object it is. Would be a great follow-up task.I think moving the other way and making them all objects so that we can tie them into evection and the shrinker, use more interesting allocation strategies, improve integration with debugging etc.
Hm, not sure yet since it will be a lot of work at least. But I guess we could untangle the meaning of obj->pin a bit and add an unbind vfunc which adds some magic. But there's a lot of stuff attached to a gem bo that just doesn't make a lot of sense really, so maybe a better option would be to subclass a struct i915_ggtt_vma with special magic. Dunno really. -Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Re: [PATCH 2/8] drm/i915: Adds graphic address space ballooning logic
Hi Daniel & Chris,
Thank you very much for your comments, And sorry for my late reply.:)
I was focusing on other tasks previously.
See my questions below:
Quoted 17 lines · Daniel Vetter
Quoted 16 lines · Chris Wilson
Quoted 15 lines · Daniel Vetter
Quoted 13 lines · Chris Wilson
Quoted 7 lines · Kevin
Quoted 5 lines
From: Chris Wilson
The implementation also looks backwards. To work correctly with the GTT allocator, you need to preallocate the reserved space such that it can only allocate from the allowed ranges. Similarly, it should evict any conflicting nodes when deballooning.Could you elaborate a bit for above suggestion?
My expectation was that the
dev_priv->gtt.base.vmwould contain exactly two holes after setup (in the mappable and non-mappable range). To do that you would explicitly reserve everything barred from this client using a set ofdrm_mm_reserve_node()
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Essentially a
reserve_nodeimplements what you open-code withinsert_node_rangeright now.Heh, there is a big difference. One inserts exactly where you ask and fails if it conflicts, the other inserts where it feels like within that range.
Do you mean drm_mm_search_free_in_range_generic() may not get reserve the exact range we are expecting to? Is this why you'd prefer the drm_mm_reserve_node()?
Besides, the ggtt_vm->mm is just initialized right before the ballooning code in routine i915_gem_setup_global_gtt(), so is there any chance the range to be partitioned out is already reserved by someone else?
Quoted 17 lines
Well if the the requested size matches the range exactly then it will be the same. Which iirc is what's going on here I think.
Quoted 4 lines
Quoted 4 lines
One issue aside with both this and with the PDE reservations for gen7 is that there are now other thins in the ggtt
drm_mmallocator than just gem objects. Which means our debugfs files are now less useful.Quoted 8 lines
Quoted 4 lines
It might be useful to augment that dumper with one that dumps everything. We could add a few bits of driver-private tags in
drm_mm_node(there's space) to figure out what kind of object it is. Would be a great follow-up task.I think moving the other way and making them all objects so that we can tie them into evection and the shrinker, use more interesting allocation strategies, improve integration with debugging etc.
Quoted 5 lines
Hm, not sure yet since it will be a lot of work at least. But I guess we could untangle the meaning of
obj->pina bit and add an unbind vfunc which adds some magic. But there's a lot of stuff attached to a gem bo that just doesn't make a lot of sense really, so maybe a better option would be to subclass a structi915_ggtt_vmawith special magic. Dunno really.
Sorry, not sure what these comments are about. :) I'll need time to read the code. Could you please elaborate a bit? Thanks!
P.S. about the guard page: for now, the current logic reserves a guard page between different guests and at the very last entry of the whole physical GTT. the previous comments says: "The CS prefetcher happens everywhere and so can read from the end of one range into the beginning of another clients". So I guess the guard page in current patch is necessary, right?
Quoted 2 lines
-Daniel
Thanks
Yu
Re: [PATCH 2/8] drm/i915: Adds graphic address space ballooning logic
Quoted 31 lines · Yu
Hi Daniel & Chris,
Thank you very much for your comments, And sorry for my late
reply.:) I was focusing on other tasks previously.
See my questions below:Quoted 17 lines · Daniel Vetter
Quoted 16 lines · Chris Wilson
Quoted 15 lines · Daniel Vetter
Quoted 13 lines · Chris Wilson
Quoted 7 lines · Kevin
Quoted 5 lines
From: Chris Wilson
The implementation also looks backwards. To work correctly with the GTT allocator, you need to preallocate the reserved space such that it can only allocate from the allowed ranges. Similarly, it should evict any conflicting nodes when deballooning.Could you elaborate a bit for above suggestion?
My expectation was that the
dev_priv->gtt.base.vmwould contain exactly two holes after setup (in the mappable and non-mappable range). To do that you would explicitly reserve everything barred from this client using a set ofdrm_mm_reserve_node()Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Essentially a
reserve_nodeimplements what you open-code withinsert_node_rangeright now.Heh, there is a big difference. One inserts exactly where you ask and fails if it conflicts, the other inserts where it feels like within that range.
Quoted 8 lines
Do you mean
drm_mm_search_free_in_range_generic()may not get reserve the exact range we are expecting to? Is this why you'd prefer thedrm_mm_reserve_node()?Besides, the
ggtt_vm->mmis just initialized right before the ballooning code in routinei915_gem_setup_global_gtt(), so is there any chance the range to be partitioned out is already reserved by someone else?
No. It is just that drm_mm_reserve_node() was added to have the precise semantics expected here with the strict checks. And should work better with dynamic ballooning...
-Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
Re: [PATCH 2/8] drm/i915: Adds graphic address space ballooning logic
Hi Chris & Daniel,
Thanks for your comments. Following are my understandings about the
changes needed for this patch:
1> We do not need the guard page anymore between different VMs. For the very last physical GTT entry, let's keep it pointing to a guard page.
2> To reserve the GMs in our balloon code, drm_mm_reserve_node() is preferred than the current drm_mm_insert_node_in_range_generic().
Am I correct? Please correct me if anything wrong. Thanks! :)
Yu
Quoted 46 lines · Chris Wilson
Quoted 31 lines · Yu
Hi Daniel & Chris,
Thank you very much for your comments, And sorry for my late
reply.:) I was focusing on other tasks previously.
See my questions below:Quoted 17 lines · Daniel Vetter
Quoted 16 lines · Chris Wilson
Quoted 15 lines · Daniel Vetter
Quoted 13 lines · Chris Wilson
Quoted 7 lines · Kevin
Quoted 5 lines
From: Chris Wilson
The implementation also looks backwards. To work correctly with the GTT allocator, you need to preallocate the reserved space such that it can only allocate from the allowed ranges. Similarly, it should evict any conflicting nodes when deballooning.Could you elaborate a bit for above suggestion?
My expectation was that the
dev_priv->gtt.base.vmwould contain exactly two holes after setup (in the mappable and non-mappable range). To do that you would explicitly reserve everything barred from this client using a set ofdrm_mm_reserve_node()Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Essentially a
reserve_nodeimplements what you open-code withinsert_node_rangeright now.Heh, there is a big difference. One inserts exactly where you ask and fails if it conflicts, the other inserts where it feels like within that range.
Quoted 8 lines
Do you mean
drm_mm_search_free_in_range_generic()may not get reserve the exact range we are expecting to? Is this why you'd prefer thedrm_mm_reserve_node()?Besides, the
ggtt_vm->mmis just initialized right before the ballooning code in routinei915_gem_setup_global_gtt(), so is there any chance the range to be partitioned out is already reserved by someone else?No. It is just that
drm_mm_reserve_node()was added to have the precise semantics expected here with the strict checks. And should work better with dynamic ballooning...
-Chris
Re: [PATCH 2/8] drm/i915: Adds graphic address space ballooning logic
Quoted 5 lines · Yu
Hi Chris & Daniel,
Thanks for your comments. Following are my understandings about
the changes needed for this patch:
Quoted 8 lines
1> We do not need the guard page anymore between different VMs. For the very last physical GTT entry, let's keep it pointing to a guard page.
2> To reserve the GMs in our balloon code,
drm_mm_reserve_node()is preferred than the currentdrm_mm_insert_node_in_range_generic().Am I correct? Please correct me if anything wrong. Thanks! :)
Agreed. If you care to respin the patch, I'll see if anything else alarms me ;-)
-Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
Re: [PATCH 2/8] drm/i915: Adds graphic address space ballooning logic
Quoted 10 lines · Chris Wilson
Quoted 9 lines · Yu
Hi Chris & Daniel,
Thanks for your comments. Following are my understandings about
the changes needed for this patch:1> We do not need the guard page anymore between different VMs. For the very last physical GTT entry, let's keep it pointing to a guard page.
Quoted 7 lines
Quoted 4 lines
2> To reserve the GMs in our balloon code,
drm_mm_reserve_node()is preferred than the currentdrm_mm_insert_node_in_range_generic().Am I correct? Please correct me if anything wrong. Thanks! :)
Agreed. If you care to respin the patch, I'll see if anything else alarms me ;-)
Got it. Thank you, Chris.
Quoted 2 lines
-Chris
Re: [PATCH 2/8] drm/i915: Adds graphic address space ballooning logic
In reply to Zhang, YuQuoted 31 lines · Yu
Hi Daniel & Chris,
Thank you very much for your comments, And sorry for my late reply.:) I
was focusing on other tasks previously.
See my questions below:Quoted 17 lines · Daniel Vetter
Quoted 16 lines · Chris Wilson
Quoted 15 lines · Daniel Vetter
Quoted 13 lines · Chris Wilson
Quoted 7 lines · Kevin
Quoted 5 lines
From: Chris Wilson
The implementation also looks backwards. To work correctly with the GTT allocator, you need to preallocate the reserved space such that it can only allocate from the allowed ranges. Similarly, it should evict any conflicting nodes when deballooning.Could you elaborate a bit for above suggestion?
My expectation was that the
dev_priv->gtt.base.vmwould contain exactly two holes after setup (in the mappable and non-mappable range). To do that you would explicitly reserve everything barred from this client using a set ofdrm_mm_reserve_node()Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Essentially a
reserve_nodeimplements what you open-code withinsert_node_rangeright now.Heh, there is a big difference. One inserts exactly where you ask and fails if it conflicts, the other inserts where it feels like within that range.
Quoted 7 lines
Do you mean
drm_mm_search_free_in_range_generic()may not get reserve the exact range we are expecting to? Is this why you'd prefer thedrm_mm_reserve_node()?Besides, the
ggtt_vm->mmis just initialized right before the ballooning code in routinei915_gem_setup_global_gtt(), so is there any chance the range to be partitioned out is already reserved by someone else?
Not really, but simply using drm_mm_reserve_node makes the intent much clearer. Also if we go around to runtime ballooning failing hard is probably what we want anyway.
Quoted 16 lines
Quoted 16 lines
Well if the the requested size matches the range exactly then it will be the same. Which iirc is what's going on here I think.
Quoted 12 lines
Quoted 8 lines
One issue aside with both this and with the PDE reservations for gen7 is that there are now other thins in the ggtt
drm_mmallocator than just gem objects. Which means our debugfs files are now less useful.It might be useful to augment that dumper with one that dumps everything. We could add a few bits of driver-private tags in
drm_mm_node(there's space) to figure out what kind of object it is. Would be a great follow-up task.I think moving the other way and making them all objects so that we can tie them into evection and the shrinker, use more interesting allocation strategies, improve integration with debugging etc.
Quoted 8 lines
Quoted 5 lines
Hm, not sure yet since it will be a lot of work at least. But I guess we could untangle the meaning of
obj->pina bit and add an unbind vfunc which adds some magic. But there's a lot of stuff attached to a gem bo that just doesn't make a lot of sense really, so maybe a better option would be to subclass a structi915_ggtt_vmawith special magic. Dunno really.Sorry, not sure what these comments are about. :) I'll need time to read the code. Could you please elaborate a bit? Thanks!
We're discussing different options to unify the objects allocated from the ggtt drm_mm. Not terribly relevant for your work directly.
Quoted 5 lines
P.S. about the guard page: for now, the current logic reserves a guard page between different guests and at the very last entry of the whole physical GTT. the previous comments says: "The CS prefetcher happens everywhere and so can read from the end of one range into the beginning of another clients". So I guess the guard page in current patch is necessary, right?
tbh I'm not really sure whether we even need the guard page at the very end of the gtt any more on recent platforms. And we definitely don't need any guard page between different allocations on hsw, so I think we can just remove that.
-Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
PATCH 3/8 drm/i915: Partition the fence registers for vgpu in i915 driver
In reply to Jike SongFrom: Yu Zhang <[email protected]>
In XenGT, the fence registers are partitioned by multiple vgpu instances in different VMs. Routine i915_gem_load() is modified to reset the num_fence_regs, when the driver detects it's runing in a VM. And the allocated fence numbers is provided in PV INFO page structure.
--- drivers/gpu/drm/i915/i915_gem.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +4 -0
@@ -5038,6 +5038,10 @@ i915_gem_load(struct drm_device *dev)else dev_priv->num_fence_regs =8 ; +if (intel_vgpu_active (dev)) + dev_priv->num_fence_regs = +I915_READ (vgt_info_off (avail_rs.fence_num)); +/* Initialize fence registers to zero */ INIT_LIST_HEAD (&dev_priv->mm.fence_list);i915_gem_restore_fences (dev);
Signature · 1 line
1.9.1
PATCH 4/8 drm/i915: Disable framebuffer compression for i915 driver in VM
In reply to Jike SongFrom: Yu Zhang <[email protected]>
Framebuffer compression is disabled when driver detects it's running in XenGT VM, because XenGT does not provide emulations for FBC related operations, and we do not expose stolen memory to the VM.
--- drivers/gpu/drm/i915/i915_dma.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/gpu/drm/i915/i915_dma.c +4 -0
@@ -1741,6 +1741,10 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags)i915_check_vgpu (dev); +/* disable framebuffer compression in vgt */ +if (intel_vgpu_active (dev)) + i915.enable_fbc =0 ; +intel_irq_init (dev);intel_uncore_sanitize (dev);
Signature · 1 line
1.9.1
Re: [PATCH 4/8] drm/i915: Disable framebuffer compression for i915 driver in VM
Quoted 19 lines · Jike Song
From: Yu Zhang <[email protected]>
Framebuffer compression is disabled when driver detects it's running in XenGT VM, because XenGT does not provide emulations for FBC related operations, and we do not expose stolen memory to the VM.
Signed-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Zhiyuan Lv <[email protected]>--- drivers/gpu/drm/i915/i915_dma.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/gpu/drm/i915/i915_dma.c +0
-0
drivers/gpu/drm/i915/i915_dma.c int i915_driver_load(struct drm_device *dev, unsigned long flags) · line 1746
Quoted 6 lines
drivers/gpu/drm/i915/i915_dma.c +3
-0@@ -1741,6 +1741,10 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags)i915_check_vgpu (dev); +/* disable framebuffer compression in vgt */ +if (intel_vgpu_active (dev)) + i915.enable_fbc =0 ;
This should be done inside intel_enable_fbc() so that the correct reason is given as to why it is disabled.
-Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
Re: [PATCH 4/8] drm/i915: Disable framebuffer compression for i915 driver in VM
Quoted 21 lines · Chris Wilson
Quoted 20 lines · Jike Song
From: Yu Zhang <[email protected]>
Framebuffer compression is disabled when driver detects it's running in XenGT VM, because XenGT does not provide emulations for FBC related operations, and we do not expose stolen memory to the VM.
Signed-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Zhiyuan Lv <[email protected]>--- drivers/gpu/drm/i915/i915_dma.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/gpu/drm/i915/i915_dma.c +0
-0@@ -1741,6 +1741,10 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags)
Quoted 8 lines
drivers/gpu/drm/i915/i915_dma.c int i915_driver_load(struct drm_device *dev, unsigned long flags) · line 1746
Quoted 5 lines
drivers/gpu/drm/i915/i915_dma.c +3
-0i915_check_vgpu (dev); +/* disable framebuffer compression in vgt */ +if (intel_vgpu_active (dev)) + i915.enable_fbc =0 ;This should be done inside
intel_enable_fbc()so that the correct reason is given as to why it is disabled.
I'm sorry, but do you mean intel_update_fbc()?
Quoted 2 lines
-Chris
--
Thanks,
Jike
Re: [PATCH 4/8] drm/i915: Disable framebuffer compression for i915 driver in VM
Quoted 5 lines · Jike Song
Quoted 2 lines · Chris Wilson
This should be done inside
intel_enable_fbc()so that the correct reason is given as to why it is disabled.I'm sorry, but do you mean
intel_update_fbc()?
Yes. But don't let me stop you trying to bring sanity to the FBC code either ;-)
-Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
PATCH 5/8 drm/i915: Add the display switch logic for vgpu in i915 driver
In reply to Jike SongFrom: Yu Zhang <[email protected]>
Display switch logic is added to notify the vgt mediator that current vgpu have a valid surface to show. It does so by writing the display_ready field in PV INFO page, and then will be handled in vgt mediator. This is useful to avoid trickiness when the VM's framebuffer is being accessed in the middle of VM modesetting, e.g. compositing the framebuffer in the host side
--- drivers/gpu/drm/i915/i915_dma.c | 8 ++++++++ drivers/gpu/drm/i915/i915_reg.h | 7 +++++++ drivers/gpu/drm/i915/intel_display.c | 10 ++++++++++ 3 files changed, 25 insertions(+)
drivers/gpu/drm/i915/i915_dma.c +8 -0
@@ -1790,6 +1790,14 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags) }else {/* Start out suspended in ums mode. */ dev_priv->ums.mm_suspended =1 ; +if (intel_vgpu_active (dev)) { +/* +* Notify a valid surface after modesetting, +* when running inside a VM. +*/ +I915_WRITE (vgt_info_off (display_ready), +VGT_DRV_DISPLAY_READY ); + } }i915_setup_sysfs (dev);
drivers/gpu/drm/i915/i915_reg.h +7 -0
@@ -6673,5 +6673,12 @@ enum punit_power_well {#define vgt_info_off (x) \ (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x) +/* +* The information set by the guest gfx driver, through the display_ready field +*/ +enum vgt_display_status { +VGT_DRV_DISPLAY_NOT_READY =0 , +VGT_DRV_DISPLAY_READY ,/* ready for display switch */ +};#endif /* _I915_REG_H_ */
drivers/gpu/drm/i915/intel_display.c +10 -0
@@ -11642,6 +11642,16 @@ static int intel_crtc_set_config(struct drm_mode_set *set)intel_modeset_check_state (set->crtc->dev); } +if (intel_vgpu_active (dev)) { +/* +* Notify a valid surface after modesetting, +* when running inside a VM. +*/ +struct drm_i915_private *dev_priv = dev->dev_private; + +I915_WRITE (vgt_info_off (display_ready),VGT_DRV_DISPLAY_READY ); + } +if (ret) {DRM_DEBUG_KMS ("failed to set mode on [CRTC:%d], err = %d\n" , set->crtc->base.id, ret);
Signature · 1 line
1.9.1
Re: [PATCH 5/8] drm/i915: Add the display switch logic for vgpu in i915 driver
Quoted 23 lines · Jike Song
From: Yu Zhang <[email protected]>
Display switch logic is added to notify the vgt mediator that current vgpu have a valid surface to show. It does so by writing the
display_readyfield in PV INFO page, and then will be handled in vgt mediator. This is useful to avoid trickiness when the VM's framebuffer is being accessed in the middle of VM modesetting, e.g. compositing the framebuffer in the host sideSigned-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Zhiyuan Lv <[email protected]>--- drivers/gpu/drm/i915/i915_dma.c | 8 ++++++++ drivers/gpu/drm/i915/i915_reg.h | 7 +++++++ drivers/gpu/drm/i915/intel_display.c | 10 ++++++++++ 3 files changed, 25 insertions(+)drivers/gpu/drm/i915/i915_dma.c +0
-0
drivers/gpu/drm/i915/i915_dma.c int i915_driver_load(struct drm_device *dev, unsigned long flags) · line 1799
Quoted 10 lines
drivers/gpu/drm/i915/i915_dma.c +7
-0@@ -1790,6 +1790,14 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags) }else {/* Start out suspended in ums mode. */ dev_priv->ums.mm_suspended =1 ; +if (intel_vgpu_active (dev)) { +/* +* Notify a valid surface after modesetting, +* when running inside a VM. +*/ +I915_WRITE (vgt_info_off (display_ready), +VGT_DRV_DISPLAY_READY );
That's interesting. As i915.ko is definitely not ready at this point.
Quoted 27 lines
drivers/gpu/drm/i915/i915_dma.c +1
-0+ } }i915_setup_sysfs (dev);drivers/gpu/drm/i915/i915_reg.h +7
-0@@ -6673,5 +6673,12 @@ enum punit_power_well {#define vgt_info_off (x) \ (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x) +/* +* The information set by the guest gfx driver, through the display_ready field +*/ +enum vgt_display_status { +VGT_DRV_DISPLAY_NOT_READY =0 , +VGT_DRV_DISPLAY_READY ,/* ready for display switch */ +};#endif /* _I915_REG_H_ */ drivers/gpu/drm/i915/intel_display.c +0
-0@@ -11642,6 +11642,16 @@ static int intel_crtc_set_config(struct drm_mode_set *set)intel_modeset_check_state (set->crtc->dev);
drivers/gpu/drm/i915/intel_display.c static int intel_crtc_set_config(struct drm_mode_set *set) · line 11652
Quoted 10 lines
drivers/gpu/drm/i915/intel_display.c +8
-0} +if (intel_vgpu_active (dev)) { +/* +* Notify a valid surface after modesetting, +* when running inside a VM. +*/ +struct drm_i915_private *dev_priv = dev->dev_private; + +I915_WRITE (vgt_info_off (display_ready),VGT_DRV_DISPLAY_READY );
Should there not be some coordination before we start the modeset? -Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
Re: [PATCH 5/8] drm/i915: Add the display switch logic for vgpu in i915 driver
Hi Chris:
Thanks for the comment. See my comments below.
-----Original Message-----
From: Intel-gfx [mailto:[email protected]] On Behalf Of Chris Wilson
Sent: Friday, September 19, 2014 4:15 PM
To: Song, Jike
Cc: Vetter, Daniel; [email protected]
Subject: Re: [Intel-gfx] [PATCH 5/8] drm/i915: Add the display switch logic for vgpu in i915 driver
Quoted 23 lines · Jike Song
From: Yu Zhang <[email protected]>
Display switch logic is added to notify the vgt mediator that current vgpu have a valid surface to show. It does so by writing the
display_readyfield in PV INFO page, and then will be handled in vgt mediator. This is useful to avoid trickiness when the VM's framebuffer is being accessed in the middle of VM modesetting, e.g. compositing the framebuffer in the host sideSigned-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Zhiyuan Lv <[email protected]>--- drivers/gpu/drm/i915/i915_dma.c | 8 ++++++++ drivers/gpu/drm/i915/i915_reg.h | 7 +++++++ drivers/gpu/drm/i915/intel_display.c | 10 ++++++++++ 3 files changed, 25 insertions(+)(quoted diff) +0
-0
b/drivers/gpu/drm/i915/i915_dma.cindex bb4ad52..d9462e1 100644
drivers/gpu/drm/i915/i915_dma.c int i915_driver_load(struct drm_device *dev, unsigned long flags) · line 1799
Quoted 10 lines
drivers/gpu/drm/i915/i915_dma.c +7
-0@@ -1790,6 +1790,14 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags) }else {/* Start out suspended in ums mode. */ dev_priv->ums.mm_suspended =1 ; +if (intel_vgpu_active (dev)) { +/* +* Notify a valid surface after modesetting, +* when running inside a VM. +*/ +I915_WRITE (vgt_info_off (display_ready), +VGT_DRV_DISPLAY_READY );
That's interesting. As i915.ko is definitely not ready at this point.
Zhi: Will remove this piece in V2, as we have already had a similar logic in intel_crtc_set_config.
Quoted 27 lines
drivers/gpu/drm/i915/i915_dma.c +1
-0+ } }i915_setup_sysfs (dev);(quoted diff) +0
-0
b/drivers/gpu/drm/i915/i915_reg.hindex a70f12e..38d606c 100644drivers/gpu/drm/i915/i915_reg.h +6
-0@@ -6673,5 +6673,12 @@ enum punit_power_well { #definevgt_info_off (x) \ (VGT_PVINFO_PAGE + (long )&((struct vgt_if *)NULL )->x) +/* +* The information set by the guest gfx driver, through the +display_ready field */ enum vgt_display_status { +VGT_DRV_DISPLAY_NOT_READY =0 , +VGT_DRV_DISPLAY_READY ,/* ready for display switch */ +};#endif /* _I915_REG_H_ */ drivers/gpu/drm/i915/intel_display.c +0
-0@@ -11642,6 +11642,16 @@ static int intel_crtc_set_config(struct drm_mode_set *set)intel_modeset_check_state (set->crtc->dev);
drivers/gpu/drm/i915/intel_display.c static int intel_crtc_set_config(struct drm_mode_set *set) · line 11652
Quoted 10 lines
drivers/gpu/drm/i915/intel_display.c +8
-0} +if (intel_vgpu_active (dev)) { +/* +* Notify a valid surface after modesetting, +* when running inside a VM. +*/ +struct drm_i915_private *dev_priv = dev->dev_private; + +I915_WRITE (vgt_info_off (display_ready),VGT_DRV_DISPLAY_READY );
Should there not be some coordination before we start the modeset?
Zhi: In the design of GVT-g, there will be only one VM can occupy the display monitor at one time, which is called "Foreground VM". Switching foreground VM is implemented by switching only a part of display registers e.g. DSPCNTR/DSPSURF/DISPSTRIDE/DSPLINOFF/DSPTILEOFF. Other display mode set registers are virtualized to VM. So we may not care about the beginning of a modeset sequence here.
But we really expect to know when a VM can be switched to foreground, which is, VM has already configured those registers to valid values. If we switch a VM without valid values in those registers to foreground e.g. during VM booting time, I'm afraid that's not what we expect...
-Chris
--
Chris Wilson, Intel Open Source Technology Centre _______________________________________________ Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH 5/8] drm/i915: Add the display switch logic for vgpu in i915 driver
In reply to Jike SongQuoted 61 lines · Jike Song
From: Yu Zhang <[email protected]>
Display switch logic is added to notify the vgt mediator that current vgpu have a valid surface to show. It does so by writing the
display_readyfield in PV INFO page, and then will be handled in vgt mediator. This is useful to avoid trickiness when the VM's framebuffer is being accessed in the middle of VM modesetting, e.g. compositing the framebuffer in the host sideSigned-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Zhiyuan Lv <[email protected]>--- drivers/gpu/drm/i915/i915_dma.c | 8 ++++++++ drivers/gpu/drm/i915/i915_reg.h | 7 +++++++ drivers/gpu/drm/i915/intel_display.c | 10 ++++++++++ 3 files changed, 25 insertions(+)drivers/gpu/drm/i915/i915_dma.c +8
-0@@ -1790,6 +1790,14 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags) }else {/* Start out suspended in ums mode. */ dev_priv->ums.mm_suspended =1 ; +if (intel_vgpu_active (dev)) { +/* +* Notify a valid surface after modesetting, +* when running inside a VM. +*/ +I915_WRITE (vgt_info_off (display_ready), +VGT_DRV_DISPLAY_READY ); + } }i915_setup_sysfs (dev);drivers/gpu/drm/i915/i915_reg.h +7
-0@@ -6673,5 +6673,12 @@ enum punit_power_well {#define vgt_info_off (x) \ (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x) +/* +* The information set by the guest gfx driver, through the display_ready field +*/ +enum vgt_display_status { +VGT_DRV_DISPLAY_NOT_READY =0 , +VGT_DRV_DISPLAY_READY ,/* ready for display switch */ +};#endif /* _I915_REG_H_ */ drivers/gpu/drm/i915/intel_display.c +0
-0@@ -11642,6 +11642,16 @@ static int intel_crtc_set_config(struct drm_mode_set *set)intel_modeset_check_state (set->crtc->dev); }
drivers/gpu/drm/i915/intel_display.c static int intel_crtc_set_config(struct drm_mode_set *set) · line 11653
Quoted 10 lines
drivers/gpu/drm/i915/intel_display.c +9
-0+if (intel_vgpu_active (dev)) { +/* +* Notify a valid surface after modesetting, +* when running inside a VM. +*/ +struct drm_i915_private *dev_priv = dev->dev_private; + +I915_WRITE (vgt_info_off (display_ready),VGT_DRV_DISPLAY_READY ); + }
Since very shortly we now have the frontbuffer tracking support code. Should we move it there? See intel_frontbuffer_flush&flip. -Daniel
Quoted 11 lines
drivers/gpu/drm/i915/intel_display.c +1
-0+if (ret) {DRM_DEBUG_KMS ("failed to set mode on [CRTC:%d], err = %d\n" , set->crtc->base.id, ret);Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Re: [PATCH 5/8] drm/i915: Add the display switch logic for vgpu in i915 driver
Hi Daniel,
Quoted 63 lines · Daniel Vetter
Quoted 62 lines · Jike Song
From: Yu Zhang <[email protected]>
Display switch logic is added to notify the vgt mediator that current vgpu have a valid surface to show. It does so by writing the
display_readyfield in PV INFO page, and then will be handled in vgt mediator. This is useful to avoid trickiness when the VM's framebuffer is being accessed in the middle of VM modesetting, e.g. compositing the framebuffer in the host sideSigned-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Zhiyuan Lv <[email protected]>--- drivers/gpu/drm/i915/i915_dma.c | 8 ++++++++ drivers/gpu/drm/i915/i915_reg.h | 7 +++++++ drivers/gpu/drm/i915/intel_display.c | 10 ++++++++++ 3 files changed, 25 insertions(+)drivers/gpu/drm/i915/i915_dma.c +8
-0@@ -1790,6 +1790,14 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags) }else {/* Start out suspended in ums mode. */ dev_priv->ums.mm_suspended =1 ; +if (intel_vgpu_active (dev)) { +/* +* Notify a valid surface after modesetting, +* when running inside a VM. +*/ +I915_WRITE (vgt_info_off (display_ready), +VGT_DRV_DISPLAY_READY ); + } }i915_setup_sysfs (dev);drivers/gpu/drm/i915/i915_reg.h +7
-0@@ -6673,5 +6673,12 @@ enum punit_power_well {#define vgt_info_off (x) \ (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x) +/* +* The information set by the guest gfx driver, through the display_ready field +*/ +enum vgt_display_status { +VGT_DRV_DISPLAY_NOT_READY =0 , +VGT_DRV_DISPLAY_READY ,/* ready for display switch */ +};#endif /* _I915_REG_H_ */ drivers/gpu/drm/i915/intel_display.c +0
-0@@ -11642,6 +11642,16 @@ static int intel_crtc_set_config(struct drm_mode_set *set)intel_modeset_check_state (set->crtc->dev); }
Quoted 12 lines
drivers/gpu/drm/i915/intel_display.c static int intel_crtc_set_config(struct drm_mode_set *set) · line 11653
Quoted 9 lines
drivers/gpu/drm/i915/intel_display.c +9
-0+if (intel_vgpu_active (dev)) { +/* +* Notify a valid surface after modesetting, +* when running inside a VM. +*/ +struct drm_i915_private *dev_priv = dev->dev_private; + +I915_WRITE (vgt_info_off (display_ready),VGT_DRV_DISPLAY_READY ); + }Since very shortly we now have the frontbuffer tracking support code. Should we move it there? See intel_frontbuffer_flush&flip.
Thank you very much for the comments and sorry for the delayed reply!
For virtual machines we only need such notification once for guest system booting, would that be too heavy to add code into the flush&flip function, consider that they are to be called many times?
Going forward, we want to use the same "display_ready" for i915 running in host machines so that we can capture display status changes. Would that be good to keep the code in "intel_crtc_set_config"? Appreciate your inputs. Thanks!
Regards,
-Zhiyuan
Quoted 22 lines
-Daniel
Quoted 11 lines
drivers/gpu/drm/i915/intel_display.c +1
-0+if (ret) {DRM_DEBUG_KMS ("failed to set mode on [CRTC:%d], err = %d\n" , set->crtc->base.id, ret);Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfxSignature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH 5/8] drm/i915: Add the display switch logic for vgpu in i915 driver
Quoted 81 lines · Zhiyuan Lv
Hi Daniel,
Quoted 63 lines · Daniel Vetter
Quoted 62 lines · Jike Song
From: Yu Zhang <[email protected]>
Display switch logic is added to notify the vgt mediator that current vgpu have a valid surface to show. It does so by writing the
display_readyfield in PV INFO page, and then will be handled in vgt mediator. This is useful to avoid trickiness when the VM's framebuffer is being accessed in the middle of VM modesetting, e.g. compositing the framebuffer in the host sideSigned-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Zhiyuan Lv <[email protected]>--- drivers/gpu/drm/i915/i915_dma.c | 8 ++++++++ drivers/gpu/drm/i915/i915_reg.h | 7 +++++++ drivers/gpu/drm/i915/intel_display.c | 10 ++++++++++ 3 files changed, 25 insertions(+)drivers/gpu/drm/i915/i915_dma.c +8
-0@@ -1790,6 +1790,14 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags) }else {/* Start out suspended in ums mode. */ dev_priv->ums.mm_suspended =1 ; +if (intel_vgpu_active (dev)) { +/* +* Notify a valid surface after modesetting, +* when running inside a VM. +*/ +I915_WRITE (vgt_info_off (display_ready), +VGT_DRV_DISPLAY_READY ); + } }i915_setup_sysfs (dev);drivers/gpu/drm/i915/i915_reg.h +7
-0@@ -6673,5 +6673,12 @@ enum punit_power_well {#define vgt_info_off (x) \ (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x) +/* +* The information set by the guest gfx driver, through the display_ready field +*/ +enum vgt_display_status { +VGT_DRV_DISPLAY_NOT_READY =0 , +VGT_DRV_DISPLAY_READY ,/* ready for display switch */ +};#endif /* _I915_REG_H_ */ drivers/gpu/drm/i915/intel_display.c +0
-0@@ -11642,6 +11642,16 @@ static int intel_crtc_set_config(struct drm_mode_set *set)intel_modeset_check_state (set->crtc->dev); }Quoted 12 lines
drivers/gpu/drm/i915/intel_display.c static int intel_crtc_set_config(struct drm_mode_set *set) · line 11653
Quoted 9 lines
drivers/gpu/drm/i915/intel_display.c +9
-0+if (intel_vgpu_active (dev)) { +/* +* Notify a valid surface after modesetting, +* when running inside a VM. +*/ +struct drm_i915_private *dev_priv = dev->dev_private; + +I915_WRITE (vgt_info_off (display_ready),VGT_DRV_DISPLAY_READY ); + }Since very shortly we now have the frontbuffer tracking support code. Should we move it there? See intel_frontbuffer_flush&flip.
Thank you very much for the comments and sorry for the delayed reply!
Quoted 7 lines
For virtual machines we only need such notification once for guest system booting, would that be too heavy to add code into the flush&flip function, consider that they are to be called many times?
Going forward, we want to use the same "
display_ready" for i915 running in host machines so that we can capture display status changes. Would that be good to keep the code in "intel_crtc_set_config"? Appreciate your inputs. Thanks!
I guess the question is what exactly you want to signal to the hyporvisor with this. I guess I need to dig a bit into the sourcecode for the hyperviros part, and you need to make a really clear specification of when guests should call this and what the hyporvisor must to in reaction of this.
I don't think we'll have any issues with a bit of overhead in the frontbuffer flip/flush functions, they're not called too often. Aside:
There's no nice kerneldoc for this stuff:
http://people.freedesktop.org/~danvet/drm/drmI915.html#idp54709056
Cheers, Daniel
Quoted 26 lines
Regards,
-ZhiyuanQuoted 22 lines
-Daniel
Quoted 11 lines
drivers/gpu/drm/i915/intel_display.c +1
-0+if (ret) {DRM_DEBUG_KMS ("failed to set mode on [CRTC:%d], err = %d\n" , set->crtc->base.id, ret);Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfxSignature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Re: [PATCH 5/8] drm/i915: Add the display switch logic for vgpu in i915 driver
Hi Daniel,
Quoted 90 lines · Daniel Vetter
Quoted 81 lines · Zhiyuan Lv
Hi Daniel,
Quoted 63 lines · Daniel Vetter
Quoted 62 lines · Jike Song
From: Yu Zhang <[email protected]>
Display switch logic is added to notify the vgt mediator that current vgpu have a valid surface to show. It does so by writing the
display_readyfield in PV INFO page, and then will be handled in vgt mediator. This is useful to avoid trickiness when the VM's framebuffer is being accessed in the middle of VM modesetting, e.g. compositing the framebuffer in the host sideSigned-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Zhiyuan Lv <[email protected]>--- drivers/gpu/drm/i915/i915_dma.c | 8 ++++++++ drivers/gpu/drm/i915/i915_reg.h | 7 +++++++ drivers/gpu/drm/i915/intel_display.c | 10 ++++++++++ 3 files changed, 25 insertions(+)drivers/gpu/drm/i915/i915_dma.c +8
-0@@ -1790,6 +1790,14 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags) }else {/* Start out suspended in ums mode. */ dev_priv->ums.mm_suspended =1 ; +if (intel_vgpu_active (dev)) { +/* +* Notify a valid surface after modesetting, +* when running inside a VM. +*/ +I915_WRITE (vgt_info_off (display_ready), +VGT_DRV_DISPLAY_READY ); + } }i915_setup_sysfs (dev);drivers/gpu/drm/i915/i915_reg.h +7
-0@@ -6673,5 +6673,12 @@ enum punit_power_well {#define vgt_info_off (x) \ (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x) +/* +* The information set by the guest gfx driver, through the display_ready field +*/ +enum vgt_display_status { +VGT_DRV_DISPLAY_NOT_READY =0 , +VGT_DRV_DISPLAY_READY ,/* ready for display switch */ +};#endif /* _I915_REG_H_ */ drivers/gpu/drm/i915/intel_display.c +0
-0@@ -11642,6 +11642,16 @@ static int intel_crtc_set_config(struct drm_mode_set *set)intel_modeset_check_state (set->crtc->dev); }Quoted 12 lines
drivers/gpu/drm/i915/intel_display.c static int intel_crtc_set_config(struct drm_mode_set *set) · line 11653
Quoted 9 lines
drivers/gpu/drm/i915/intel_display.c +9
-0+if (intel_vgpu_active (dev)) { +/* +* Notify a valid surface after modesetting, +* when running inside a VM. +*/ +struct drm_i915_private *dev_priv = dev->dev_private; + +I915_WRITE (vgt_info_off (display_ready),VGT_DRV_DISPLAY_READY ); + }Since very shortly we now have the frontbuffer tracking support code. Should we move it there? See intel_frontbuffer_flush&flip.
Thank you very much for the comments and sorry for the delayed reply!
Quoted 7 lines
For virtual machines we only need such notification once for guest system booting, would that be too heavy to add code into the flush&flip function, consider that they are to be called many times?
Going forward, we want to use the same "
display_ready" for i915 running in host machines so that we can capture display status changes. Would that be good to keep the code in "intel_crtc_set_config"? Appreciate your inputs. Thanks!
Quoted 5 lines
I guess the question is what exactly you want to signal to the hyporvisor with this. I guess I need to dig a bit into the sourcecode for the hyperviros part, and you need to make a really clear specification of when guests should call this and what the hyporvisor must to in reaction of this.
In this patchset for i915 driver running inside a virtual machine, the code here is to signal vgt that guest framebuffer address in plane surface register is ready to be written into physical registers. vgt's reaction is to make policy change. After that point, VM's write to surface MMIO will go into physical register directly. Here we do not need notifications for every guest display surface MMIO write, but just the time we can do the policy change.
The usage of this is for guest system booting. At the very beginning, we block the guest surface MMIO writes so that the monitor still shows up host contents, then we rely on this notification to show up guest VM's screen.
We have another change for i915 driver to notify vgt the modesetting. But that is for i915 running as a host driver. Sorry that it may be confusing to mix two things together. This can be discussed later.
Quoted 7 lines
I don't think we'll have any issues with a bit of overhead in the frontbuffer flip/flush functions, they're not called too often. Aside:
There's no nice kerneldoc for this stuff:http://people.freedesktop.org/~danvet/drm/drmI915.html#idp54709056
Thanks for the info! The doc is helpful for me to understand the new feature. Since intel_frontbuffer_flush() will be called every time there is page flip request, that may not be what we want: just one time notification to do the policy change. Thanks!
Regards,
-Zhiyuan
Quoted 37 lines
Cheers, Daniel
Quoted 26 lines
Regards,
-ZhiyuanQuoted 22 lines
-Daniel
Quoted 11 lines
drivers/gpu/drm/i915/intel_display.c +1
-0+if (ret) {DRM_DEBUG_KMS ("failed to set mode on [CRTC:%d], err = %d\n" , set->crtc->base.id, ret);Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfxSignature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfxSignature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH 5/8] drm/i915: Add the display switch logic for vgpu in i915 driver
Quoted 14 lines · Zhiyuan Lv
Quoted 5 lines · Daniel Vetter
I guess the question is what exactly you want to signal to the hyporvisor with this. I guess I need to dig a bit into the sourcecode for the hyperviros part, and you need to make a really clear specification of when guests should call this and what the hyporvisor must to in reaction of this.
In this patchset for i915 driver running inside a virtual machine, the code here is to signal vgt that guest framebuffer address in plane surface register is ready to be written into physical registers. vgt's reaction is to make policy change. After that point, VM's write to surface MMIO will go into physical register directly. Here we do not need notifications for every guest display surface MMIO write, but just the time we can do the policy change.
Quoted 3 lines
The usage of this is for guest system booting. At the very beginning, we block the guest surface MMIO writes so that the monitor still shows up host contents, then we rely on this notification to show up guest VM's screen.
This sounds more like we need one call at the end of the i915 driver load sequence to signal to the hypervisor that everything is now down and a proper driver is in charge of the virtual machine gfx.
Quoted 3 lines
We have another change for i915 driver to notify vgt the modesetting. But that is for i915 running as a host driver. Sorry that it may be confusing to mix two things together. This can be discussed later.
Yeah, better split that out and move it to the host-side vgt integration series.
Quoted 11 lines
Quoted 6 lines
I don't think we'll have any issues with a bit of overhead in the frontbuffer flip/flush functions, they're not called too often. Aside:
There's no nice kerneldoc for this stuff:http://people.freedesktop.org/~danvet/drm/drmI915.html#idp54709056
Thanks for the info! The doc is helpful for me to understand the new feature. Since
intel_frontbuffer_flush()will be called every time there is page flip request, that may not be what we want: just one time notification to do the policy change. Thanks!
Always nice when my documentation work proves useful ;-)
Cheers, Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
PATCH 6/8 drm/i915: Disable power management for i915 driver in VM
In reply to Jike SongFrom: Yu Zhang <[email protected]>
In XenGT, GPU power management is controlled by host i915 driver, so there is no need to provide virtualized GPU PM support. In the future it might be useful to gather VM input for freq boost, but now let's disable it simply.
--- drivers/gpu/drm/i915/intel_pm.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/gpu/drm/i915/intel_pm.c +6 -0
@@ -5419,6 +5419,12 @@ void intel_enable_gt_powersave(struct drm_device *dev)mutex_unlock (&dev->struct_mutex); }else if (INTEL_INFO (dev)->gen >=6 ) {/* +* Powersaving is disabled when running inside a VM. +*/ +if (intel_vgpu_active (dev)) +return ; + +/* * PCU communication is slow and this doesn't need to be * done at any specific time, so do this out of our fast path * to make resume and init faster.
Signature · 1 line
1.9.1
Re: [PATCH 6/8] drm/i915: Disable power management for i915 driver in VM
Quoted 18 lines · Jike Song
From: Yu Zhang <[email protected]>
In XenGT, GPU power management is controlled by host i915 driver, so there is no need to provide virtualized GPU PM support. In the future it might be useful to gather VM input for freq boost, but now let's disable it simply.
Signed-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>--- drivers/gpu/drm/i915/intel_pm.c | 6 ++++++ 1 file changed, 6 insertions(+)drivers/gpu/drm/i915/intel_pm.c +0
-0
drivers/gpu/drm/i915/intel_pm.c void intel_enable_gt_powersave(struct drm_device *dev) · line 5425
Quoted 7 lines
drivers/gpu/drm/i915/intel_pm.c +4
-0@@ -5419,6 +5419,12 @@ void intel_enable_gt_powersave(struct drm_device *dev)mutex_unlock (&dev->struct_mutex); }else if (INTEL_INFO (dev)->gen >=6 ) {/* +* Powersaving is disabled when running inside a VM. +*/ +if (intel_vgpu_active (dev)) +return ;
Don't special case this to gen6+, it always applies because the comment is not accurate, whereas the changelog is a much more apt description.
/* Powersaving is controlled by the host when inside a VM */ if (intel_vgpu_active(dev))
return;
-Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
PATCH 7/8 drm/i915: Create vgpu specific write MMIO to reduce traps
In reply to Jike SongFrom: Yu Zhang <[email protected]>
In the virtualized environment, forcewake operations are not necessory for the driver, because mmio accesses will be trapped and emulated by the host side, and real forcewake operations are also done in the host. New mmio write handlers are added to directly call the __raw_i915_write, therefore will reduce many traps and increase the overall performance for drivers runing in the VM with Intel GVT-g enhancement
--- drivers/gpu/drm/i915/i915_dma.c | 7 +++++-- drivers/gpu/drm/i915/i915_drv.h | 1 + drivers/gpu/drm/i915/intel_uncore.c | 24 ++++++++++++++++++++++++ 3 files changed, 30 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_dma.c +5 -2
@@ -1741,10 +1741,13 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags)i915_check_vgpu (dev); -/* disable framebuffer compression in vgt */ -if (intel_vgpu_active (dev)) +if (intel_vgpu_active (dev)) { +/* disable framebuffer compression in vgt */ i915.enable_fbc =0 ; +intel_vgpu_reg_write_init (dev); + } +intel_irq_init (dev);intel_uncore_sanitize (dev);
drivers/gpu/drm/i915/i915_drv.h +1 -0
@@ -2334,6 +2334,7 @@ extern void intel_uncore_fini(struct drm_device *dev);extern void intel_uncore_forcewake_reset (struct drm_device *dev,bool restore);extern void i915_check_vgpu (struct drm_device *dev); +extern void intel_vgpu_reg_write_init (struct drm_device *dev);static inline bool intel_vgpu_active (struct drm_device *dev) {struct drm_i915_private *dev_priv = dev->dev_private;
drivers/gpu/drm/i915/intel_uncore.c +24 -0
@@ -722,6 +722,14 @@ hsw_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace)REG_WRITE_FOOTER ; \ } +#define __vgpu_write (x) \ +static void \ +vgpu_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace) { \ + REG_WRITE_HEADER; \ + __raw_i915_write##x(dev_priv, reg, val); \ + REG_WRITE_FOOTER; \ +} +static const u32 gen8_shadowed_regs[] = {FORCEWAKE_MT ,GEN6_RPNSWREQ ,
@@ -816,6 +824,10 @@ __gen4_write(8) __gen4_write(16) __gen4_write(32) __gen4_write(64) +__vgpu_write(8) +__vgpu_write(16) +__vgpu_write(32) +__vgpu_write(64)#undef __chv_write#undef __gen8_write
@@ -823,9 +835,21 @@ __gen4_write(64)#undef __gen6_write#undef __gen5_write#undef __gen4_write +#undef __vgpu_write#undef REG_WRITE_FOOTER#undef REG_WRITE_HEADER +void intel_vgpu_reg_write_init (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv = dev->dev_private; + + dev_priv->uncore.funcs.mmio_writeb = vgpu_write8; + dev_priv->uncore.funcs.mmio_writew = vgpu_write16; + dev_priv->uncore.funcs.mmio_writel = vgpu_write32; + dev_priv->uncore.funcs.mmio_writeq = vgpu_write64; + +} +void intel_uncore_init (struct drm_device *dev) {struct drm_i915_private *dev_priv = dev->dev_private;
Signature · 1 line
1.9.1
Re: [PATCH 7/8] drm/i915: Create vgpu specific write MMIO to reduce traps
Quoted 9 lines · Jike Song
From: Yu Zhang <[email protected]>
In the virtualized environment, forcewake operations are not necessory for the driver, because mmio accesses will be trapped and emulated by the host side, and real forcewake operations are also done in the host. New mmio write handlers are added to directly call the
__raw_i915_write, therefore will reduce many traps and increase the overall performance for drivers runing in the VM with Intel GVT-g enhancement
Explain why this is not done inside uncore_init.
-Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
Re: [PATCH 7/8] drm/i915: Create vgpu specific write MMIO to reduce traps
[adding cc. By mistake I set suppresscc=all in my ~/.gitconfig]
Quoted 11 lines · Chris Wilson
Quoted 9 lines · Jike Song
From: Yu Zhang <[email protected]>
In the virtualized environment, forcewake operations are not necessory for the driver, because mmio accesses will be trapped and emulated by the host side, and real forcewake operations are also done in the host. New mmio write handlers are added to directly call the
__raw_i915_write, therefore will reduce many traps and increase the overall performance for drivers runing in the VM with Intel GVT-g enhancement
Quoted 2 lines
Explain why this is not done inside
uncore_init.
-Chris
Hi Chris,
The gpu_active flag is set by intel_detect_vgpu(), which uses I915_READxx, should be after intel_uncore_init.
--
Thanks,
Jike
PATCH 8/8 drm/i915: Support alias ppgtt in VM if ppgtt is enabled
In reply to Jike SongFrom: Yu Zhang <[email protected]>
The current XenGT only supports alias ppgtt. And the emulation is done in XenGT host by first trapping PP_DIR_BASE mmio accesses. Updating PP_DIR_BASE by using instructions such as MI_LOAD_REGISTER_IMM are hard to emulate and are not supported in current XenGT. Therefore this patch also added a new callback routine - vgpu_mm_switch() to set the PP_DIR_BASE by mmio writes.
--- drivers/gpu/drm/i915/i915_dma.c | 5 +++++ drivers/gpu/drm/i915/i915_gem_gtt.c | 14 ++++++++++++++ 2 files changed, 19 insertions(+)
drivers/gpu/drm/i915/i915_dma.c +5 -0
@@ -1746,6 +1746,11 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags) i915.enable_fbc =0 ;intel_vgpu_reg_write_init (dev); + +if (USES_FULL_PPGTT (dev)) { +DRM_INFO ("Only support alias ppgtt for now in VM.\n" ); + i915.enable_ppgtt =1 ; + } }intel_irq_init (dev);
drivers/gpu/drm/i915/i915_gem_gtt.c +14 -0
@@ -898,6 +898,17 @@ static int hsw_mm_switch(struct i915_hw_ppgtt *ppgtt,return 0 ; } +static int vgpu_mm_switch (struct i915_hw_ppgtt *ppgtt, +struct intel_engine_cs *ring) +{ +struct drm_device *dev = ppgtt->base.dev; +struct drm_i915_private *dev_priv = dev->dev_private; + +I915_WRITE (RING_PP_DIR_DCLV (ring),PP_DIR_DCLV_2G ); +I915_WRITE (RING_PP_DIR_BASE (ring),get_pd_offset (ppgtt)); +return 0 ; +} +static int gen7_mm_switch (struct i915_hw_ppgtt *ppgtt,struct intel_engine_cs *ring) {
@@ -1224,6 +1235,9 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt) }else BUG (); +if (intel_vgpu_active (dev)) + ppgtt->switch_mm = vgpu_mm_switch; + ret =gen6_ppgtt_alloc (ppgtt);if (ret)return ret;
Signature · 1 line
1.9.1
Re: [PATCH 8/8] drm/i915: Support alias ppgtt in VM if ppgtt is enabled
Quoted 21 lines · Jike Song
From: Yu Zhang <[email protected]>
The current XenGT only supports alias ppgtt. And the emulation is done in XenGT host by first trapping
PP_DIR_BASEmmio accesses. UpdatingPP_DIR_BASEby using instructions such asMI_LOAD_REGISTER_IMMare hard to emulate and are not supported in current XenGT. Therefore this patch also added a new callback routine -vgpu_mm_switch()to set thePP_DIR_BASEby mmio writes.Signed-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>--- drivers/gpu/drm/i915/i915_dma.c | 5 +++++ drivers/gpu/drm/i915/i915_gem_gtt.c | 14 ++++++++++++++ 2 files changed, 19 insertions(+)drivers/gpu/drm/i915/i915_dma.c +0
-0
drivers/gpu/drm/i915/i915_dma.c int i915_driver_load(struct drm_device *dev, unsigned long flags) · line 1753
Quoted 8 lines
drivers/gpu/drm/i915/i915_dma.c +5
-0@@ -1746,6 +1746,11 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags) i915.enable_fbc =0 ;intel_vgpu_reg_write_init (dev); + +if (USES_FULL_PPGTT (dev)) { +DRM_INFO ("Only support alias ppgtt for now in VM.\n" ); + i915.enable_ppgtt =1 ; + }
This should be moved to sanitize_enable_ppgtt(), probably by expanding HAS_PPGTT(), e.g.:
drivers/gpu/drm/i915/i915_drv.h +0 -2
@@ -2139,8 +2139,6 @@ struct drm_i915_cmd_table {#define HAS_HW_CONTEXTS (dev) (INTEL_INFO(dev)->gen >= 6)#define HAS_LOGICAL_RING_CONTEXTS (dev) (INTEL_INFO(dev)->gen >= 8) -#define HAS_ALIASING_PPGTT (dev) (INTEL_INFO(dev)->gen >= 6) -#define HAS_PPGTT (dev) (INTEL_INFO(dev)->gen >= 7 && !IS_GEN8(dev))#define USES_PPGTT (dev) (i915.enable_ppgtt)#define USES_FULL_PPGTT (dev) (i915.enable_ppgtt == 2)
drivers/gpu/drm/i915/i915_gem_gtt.c +13 -4
@@ -35,13 +35,23 @@ static void chv_setup_private_ppat(struct drm_i915_private *dev_priv);static int sanitize_enable_ppgtt (struct drm_device *dev,int enable_ppgtt) { -if (enable_ppgtt ==0 || !HAS_ALIASING_PPGTT (dev)) +bool has_aliasing_ppgtt; +bool has_full_ppgtt; + + has_aliasing_ppgtt =INTEL_INFO (dev)->gen >=6 ; + has_full_ppgtt =INTEL_INFO (dev)->gen >=7 ; +if (IS_GEN8 (dev)) + has_full_ppgtt = false;/* XXX why? */ +if (intel_vgpu_active (dev)) + has_full_ppgtt = false;/* emulation is too hard */ + +if (enable_ppgtt ==0 || !has_aliasing_ppgtt)return 0 ;if (enable_ppgtt ==1 )return 1 ; -if (enable_ppgtt ==2 &&HAS_PPGTT (dev)) +if (enable_ppgtt ==2 && has_full_ppgtt)return 2 ;#ifdef CONFIG_INTEL_IOMMU
@@ -59,7 +69,7 @@ static int sanitize_enable_ppgtt(struct drm_device *dev, int enable_ppgtt)return 0 ; } -return HAS_PPGTT (dev) ?2 :HAS_ALIASING_PPGTT (dev) ?1 :0 ; +return has_full_ppgtt ?2 : has_aliasing_ppgtt ?1 :0 ; } -Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
Re: [PATCH 8/8] drm/i915: Support alias ppgtt in VM if ppgtt is enabled
Quoted 48 lines · Chris Wilson
This should be moved to
sanitize_enable_ppgtt(), probably by expandingHAS_PPGTT(), e.g.:drivers/gpu/drm/i915/i915_drv.h +0
-2@@ -2139,8 +2139,6 @@ struct drm_i915_cmd_table {#define HAS_HW_CONTEXTS (dev) (INTEL_INFO(dev)->gen >= 6)#define HAS_LOGICAL_RING_CONTEXTS (dev) (INTEL_INFO(dev)->gen >= 8) -#define HAS_ALIASING_PPGTT (dev) (INTEL_INFO(dev)->gen >= 6) -#define HAS_PPGTT (dev) (INTEL_INFO(dev)->gen >= 7 && !IS_GEN8(dev))#define USES_PPGTT (dev) (i915.enable_ppgtt)#define USES_FULL_PPGTT (dev) (i915.enable_ppgtt == 2)drivers/gpu/drm/i915/i915_gem_gtt.c +12
-2@@ -35,13 +35,23 @@ static void chv_setup_private_ppat(struct drm_i915_private *dev_priv);static int sanitize_enable_ppgtt (struct drm_device *dev,int enable_ppgtt) { -if (enable_ppgtt ==0 || !HAS_ALIASING_PPGTT (dev)) +bool has_aliasing_ppgtt; +bool has_full_ppgtt; + + has_aliasing_ppgtt =INTEL_INFO (dev)->gen >=6 ; + has_full_ppgtt =INTEL_INFO (dev)->gen >=7 ; +if (IS_GEN8 (dev)) + has_full_ppgtt = false;/* XXX why? */ +if (intel_vgpu_active (dev)) + has_full_ppgtt = false;/* emulation is too hard */ + +if (enable_ppgtt ==0 || !has_aliasing_ppgtt)return 0 ;if (enable_ppgtt ==1 )return 1 ; -if (enable_ppgtt ==2 &&HAS_PPGTT (dev)) +if (enable_ppgtt ==2 && has_full_ppgtt)return 2 ;#ifdef CONFIG_INTEL_IOMMU
drivers/gpu/drm/i915/i915_gem_gtt.c static int sanitize_enable_ppgtt(struct drm_device *dev, int enable_ppgtt) · line 73
Quoted 6 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +1
-1@@ -59,7 +69,7 @@ static int sanitize_enable_ppgtt(struct drm_device *dev, int enable_ppgtt)return 0 ; } -return HAS_PPGTT (dev) ?2 :HAS_ALIASING_PPGTT (dev) ?1 :0 ; +return has_full_ppgtt ?2 : has_aliasing_ppgtt ?1 :0 ; }
Thanks for the demo. Currently sanitize_enable_ppgtt() is called by i915_gem_gtt_init(), before intel_check_vgpu().
I'm trying to detect VGPU as early as possible, maybe between intel_detect_pch() and intel_uncore_init(), using bare readq/readw instead of I915_READxx.
drivers/gpu/drm/i915/i915_gem_gtt.c static int sanitize_enable_ppgtt(struct drm_device *dev, int enable_ppgtt) · line 74
Quoted 3 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +0
-1-Chris
--
Thanks,
Jike
Re: [PATCH 8/8] drm/i915: Support alias ppgtt in VM if ppgtt is enabled
Quoted 5 lines · Jike Song
Thanks for the demo. Currently
sanitize_enable_ppgtt()is called byi915_gem_gtt_init(), beforeintel_check_vgpu().I'm trying to detect VGPU as early as possible, maybe between
intel_detect_pch()andintel_uncore_init(), using bare readq/readw instead ofI915_READxx.
I am thinking that the right level for check_vgpu is uncore_init. Using raw mmio is acceptable at that point, and once you know you have a vgpu you can forgo all the mmio emulation.
-Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre