Add enlightenments for vGPU
Subsystems (2)
v2 cover Add enlightenments for vGPU
Intel GVT-g (previously known as XenGT), is a complete GPU virtualization solution with mediated pass-through for 4th generation Intel Core processors - Haswell platform. This technology presents a virtual full-fledged GPU to each Virtual Machine (VM). VMs can directly access performance-critical resources, without intervention from the hypervisor in most cases, while privileged operations from VMs are trap-and-emulated at minimal cost. For detail, please refer to
https://01.org/xen/blogs/wangbo85/2014/intel-gvt-gxengt-pubic-release
This patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, were sent out in another patchset.
The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory w/o hypervisor's intervention, e.g. filling textures and queuing commands. However w/ the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses w/o complexity of translation.
Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (w/ some faulting mechanism).
The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource XenGT also partitions them among VMs.
Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.
Yu Zhang (8):
drm/i915: Introduce a PV INFO page structure for Intel GVT-g. drm/i915: Adds graphic address space ballooning logic drm/i915: Partition the fence registers for vgpu in i915 driver drm/i915: Disable framebuffer compression for i915 driver in VM drm/i915: Add the display switch logic for vgpu in i915 driver drm/i915: Disable power management for i915 driver in VM drm/i915: Create vgpu specific write MMIO to reduce traps drm/i915: Support alias ppgtt in VM if ppgtt is enabled
drivers/gpu/drm/i915/i915_dma.c | 11 +++ drivers/gpu/drm/i915/i915_drv.h | 12 +++ drivers/gpu/drm/i915/i915_gem.c | 5 + drivers/gpu/drm/i915/i915_gem_gtt.c | 186 +++++++++++++++++++++++++++++++++++- drivers/gpu/drm/i915/i915_vgt_if.h | 93 ++++++++++++++++++ drivers/gpu/drm/i915/intel_pm.c | 8 ++ drivers/gpu/drm/i915/intel_uncore.c | 22 +++++ 7 files changed, 334 insertions(+), 3 deletions(-) create mode 100644 drivers/gpu/drm/i915/i915_vgt_if.h
Signature · 1 line
1.9.1
PATCH v2 1/8 drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
Introduce a PV INFO structure, to facilitate the Intel GVT-g technology, which is a GPU virtualization solution with mediated pass-through(previously known as XenGT). This page contains the shared information between i915 driver and the mediator. For now, this structure utilizes an area of 4K bypes on HSW GPU's unused MMIO space to support existing production. Future hardware will have the reserved window architecturally defined, and layout of the page will be added in future BSpec.
The i915 driver load routine detects if it is running in a VM by reading the contents of this PV INFO page. If true, the pointer, vgpu.vgt_info is initialized, and intel_vgpu_active() is used by checking this pointer to conclude if gpu is virtualized with Intel GVT-g. By now, it will return true only when the driver is running in the XenGT environment on HSW.
v2:
take Chris' comments:
- call the i915_check_vgpu() in intel_uncore_init()
- sanitize i915_check_vgpu() by adding BUILD_BUG_ON() and debug info
take Daniel's comments:
- put the definition of PV INFO into a new header - i915_vgt_if.h
other changes:
- access mmio regs by readq/readw in i915_check_vgpu()
--- drivers/gpu/drm/i915/i915_drv.h | 12 ++++++ drivers/gpu/drm/i915/i915_gem_gtt.c | 26 ++++++++++++ drivers/gpu/drm/i915/i915_vgt_if.h | 85 +++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/intel_uncore.c | 2 + 4 files changed, 125 insertions(+) create mode 100644 drivers/gpu/drm/i915/i915_vgt_if.h
drivers/gpu/drm/i915/i915_drv.h +12 -0
@@ -1448,6 +1448,10 @@ struct i915_frontbuffer_tracking {unsigned flip_bits; }; +struct i915_virtual_gpu { +bool active; +}; +struct drm_i915_private {struct drm_device *dev;struct kmem_cache *slab;
@@ -1460,6 +1464,8 @@ struct drm_i915_private {struct intel_uncore uncore; +struct i915_virtual_gpu vgpu; +struct intel_gmbus gmbus[GMBUS_NUM_PORTS ];
@@ -2303,6 +2309,12 @@ extern void intel_uncore_check_errors(struct drm_device *dev);extern void intel_uncore_fini (struct drm_device *dev);extern void intel_uncore_forcewake_reset (struct drm_device *dev,bool restore); +extern void i915_check_vgpu (struct drm_device *dev); +static inline bool intel_vgpu_active (struct drm_device *dev) +{ +return to_i915 (dev)->vgpu.active; +} +void i915_enable_pipestat (struct drm_i915_private *dev_priv,enum pipe pipe,u32 status_mask);
drivers/gpu/drm/i915/i915_gem_gtt.c +26 -0
@@ -26,10 +26,36 @@#include <linux/seq_file.h> #include <drm/drmP.h> #include <drm/i915_drm.h> +#include "i915_vgt_if.h" #include "i915_drv.h" #include "i915_trace.h" #include "intel_drv.h" +void i915_check_vgpu (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv =to_i915 (dev); +uint64_t magic; +uint32_t version; + +BUILD_BUG_ON (sizeof (struct vgt_if ) !=VGT_PVINFO_SIZE ); + +if (!IS_HASWELL (dev)) +return ; + + magic =readq (dev_priv->regs +vgtif_reg (magic)); +if (magic !=VGT_MAGIC ) +return ; + + version =INTEL_VGT_IF_VERSION_ENCODE ( +readw (dev_priv->regs +vgtif_reg (version_major)), +readw (dev_priv->regs +vgtif_reg (version_minor))); +if (version !=INTEL_VGT_IF_VERSION ) +return ; + + dev_priv->vgpu.active = true; +DRM_INFO ("Virtual GPU for Intel GVT-g detected.\n" ); +} +static void bdw_setup_private_ppat (struct drm_i915_private *dev_priv);static void chv_setup_private_ppat (struct drm_i915_private *dev_priv);
drivers/gpu/drm/i915/i915_vgt_if.h new file +85 -0
85 lines not on this page · raw message
drivers/gpu/drm/i915/intel_uncore.c +2 -0
@@ -827,6 +827,8 @@ void intel_uncore_init(struct drm_device *dev) {struct drm_i915_private *dev_priv = dev->dev_private; +i915_check_vgpu (dev); +setup_timer (&dev_priv->uncore.force_wake_timer, gen6_force_wake_timer, (unsigned long )dev_priv);
Signature · 1 line
1.9.1
PATCH v2 2/8 drm/i915: Adds graphic address space ballooning logic
In reply to Yu ZhangIn XenGT, the global graphic memory space is partitioned by multiple vgpu instances in different VMs. The ballooning code is added in i915_gem_setup_global_gtt(), utilizing the drm mm allocator APIs to mark the graphic address space which are partitioned out to other vgpus as reserved.
v2:
take Chris and Daniel's comments:
- no guard page between different VMs
- use drm_mm_reserve_node() to do the reservation for ballooning, instead of the previous drm_mm_insert_node_in_range_generic()
--- drivers/gpu/drm/i915/i915_gem_gtt.c | 144 +++++++++++++++++++++++++++++++++++- 1 file changed, 141 insertions(+), 3 deletions(-)
drivers/gpu/drm/i915/i915_gem_gtt.c +141 -3
@@ -31,6 +31,134 @@#include "i915_trace.h" #include "intel_drv.h" +struct _balloon_info_ { +/* +* There are up to 2 regions per low/high GM that +* might be ballooned. Here, index 0/1 is for low +* GM, 2/3 for high GM. +*/ +struct drm_mm_node space[4 ]; +} bl_info; + +void intel_vgt_deballoon (void ) +{ +int i; + +DRM_INFO ("VGT deballoon.\n" ); + +for (i =0 ; i <4 ; i++) { +if (bl_info.space[i].allocated) +drm_mm_remove_node (&bl_info.space[i]); + } + +memset (&bl_info,0 ,sizeof (bl_info)); +} + +static int vgt_balloon_space (struct drm_mm *mm, +struct drm_mm_node *node, +unsigned long start,unsigned long end) +{ +unsigned long size = end - start; + +if (start == end) +return -EINVAL ; + +DRM_INFO ("balloon space: range [ 0x%lx - 0x%lx ] %lu KB.\n" , + start, end, size /1024 ); + + node->start = start; + node->size = size; + +return drm_mm_reserve_node (mm, node); +} + +static int intel_vgt_balloon (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv =to_i915 (dev); +struct i915_address_space *ggtt_vm = &dev_priv->gtt.base; +unsigned long ggtt_vm_end = ggtt_vm->start + ggtt_vm->total; + +unsigned long low_gm_base, low_gm_size, low_gm_end; +unsigned long high_gm_base, high_gm_size, high_gm_end; +int ret; + + low_gm_base =I915_READ (vgtif_reg (avail_rs.low_gmadr.my_base)); + low_gm_size =I915_READ (vgtif_reg (avail_rs.low_gmadr.my_size)); + high_gm_base =I915_READ (vgtif_reg (avail_rs.high_gmadr.my_base)); + high_gm_size =I915_READ (vgtif_reg (avail_rs.high_gmadr.my_size)); + + low_gm_end = low_gm_base + low_gm_size; + high_gm_end = high_gm_base + high_gm_size; + +DRM_INFO ("VGT ballooning configuration:\n" ); +DRM_INFO ("Low GM: base 0x%lx size %ldKB\n" , + low_gm_base, low_gm_size /1024 ); +DRM_INFO ("High GM: base 0x%lx size %ldKB\n" , + high_gm_base, high_gm_size /1024 ); + +if (low_gm_base < ggtt_vm->start + || low_gm_end > dev_priv->gtt.mappable_end + || high_gm_base < dev_priv->gtt.mappable_end + || high_gm_end > ggtt_vm_end) { +DRM_ERROR ("Invalid ballooning configuration!\n" ); +return -EINVAL ; + } + +memset (&bl_info,0 ,sizeof (bl_info)); + +/* High GM ballooning */ +if (high_gm_base > dev_priv->gtt.mappable_end) { + ret =vgt_balloon_space (&ggtt_vm->mm, + &bl_info.space[2 ], + dev_priv->gtt.mappable_end, + high_gm_base); + +if (ret) + goto err; + } + +/* +* No need to partition out the last physical page, +* because it is reserved to the guard page. +*/ +if (high_gm_end < ggtt_vm_end -PAGE_SIZE ) { + ret =vgt_balloon_space (&ggtt_vm->mm, + &bl_info.space[3 ], + high_gm_end, + ggtt_vm_end -PAGE_SIZE ); +if (ret) + goto err; + } + +/* Low GM ballooning */ +if (low_gm_base > ggtt_vm->start) { + ret =vgt_balloon_space (&ggtt_vm->mm, + &bl_info.space[0 ], + ggtt_vm->start, low_gm_base); + +if (ret) + goto err; + } + +if (low_gm_end < dev_priv->gtt.mappable_end) { + ret =vgt_balloon_space (&ggtt_vm->mm, + &bl_info.space[1 ], + low_gm_end, + dev_priv->gtt.mappable_end); + +if (ret) + goto err; + } + +DRM_INFO ("VGT balloon successfully\n" ); +return 0 ; + +err: +DRM_ERROR ("VGT balloon fail\n" ); +intel_vgt_deballoon (); +return ret; +} +void i915_check_vgpu (struct drm_device *dev) {struct drm_i915_private *dev_priv =to_i915 (dev);
@@ -1709,6 +1837,16 @@ int i915_gem_setup_global_gtt(struct drm_device *dev,/* Subtract the guard page ... */ drm_mm_init (&ggtt_vm->mm, start, end - start -PAGE_SIZE ); + + dev_priv->gtt.base.start = start; + dev_priv->gtt.base.total = end - start; + +if (intel_vgpu_active (dev)) { + ret =intel_vgt_balloon (dev); +if (ret) +return ret; + } +if (!HAS_LLC (dev)) dev_priv->gtt.base.mm.color_adjust = i915_gtt_color_adjust;
@@ -1728,9 +1866,6 @@ int i915_gem_setup_global_gtt(struct drm_device *dev, obj->has_global_gtt_mapping =1 ; } - dev_priv->gtt.base.start = start; - dev_priv->gtt.base.total = end - start; -/* Clear any non-preallocated blocks */ drm_mm_for_each_hole (entry, &ggtt_vm->mm, hole_start, hole_end) { DRM_DEBUG_KMS("clearing unused GTT space: [%lx, %lx]\n",
@@ -1782,6 +1917,9 @@ void i915_global_gtt_cleanup(struct drm_device *dev) }if (drm_mm_initialized (&vm->mm)) { +if (intel_vgpu_active (dev)) +intel_vgt_deballoon (); +drm_mm_takedown (&vm->mm);list_del (&vm->global_link); }
Signature · 1 line
1.9.1
PATCH v2 3/8 drm/i915: Partition the fence registers for vgpu in i915 driver
In reply to Yu ZhangIn XenGT, the fence registers are partitioned by multiple vgpu instances in different VMs. Routine i915_gem_load() is modified to reset the num_fence_regs, when the driver detects it's runing in a VM. And the allocated fence numbers is provided in PV INFO page structure.
--- drivers/gpu/drm/i915/i915_gem.c | 5 +++++ 1 file changed, 5 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +5 -0
@@ -28,6 +28,7 @@#include <drm/drmP.h> #include <drm/drm_vma_manager.h> #include <drm/i915_drm.h> +#include "i915_vgt_if.h" #include "i915_drv.h" #include "i915_trace.h" #include "intel_drv.h"
@@ -4988,6 +4989,10 @@ i915_gem_load(struct drm_device *dev)else dev_priv->num_fence_regs =8 ; +if (intel_vgpu_active (dev)) + dev_priv->num_fence_regs = +I915_READ (vgtif_reg (avail_rs.fence_num)); +/* Initialize fence registers to zero */ INIT_LIST_HEAD (&dev_priv->mm.fence_list);i915_gem_restore_fences (dev);
Signature · 1 line
1.9.1
PATCH v2 4/8 drm/i915: Disable framebuffer compression for i915 driver in VM
In reply to Yu ZhangFramebuffer compression is disabled when driver detects it's running in XenGT VM, because XenGT does not provide emulations for FBC related operations, and we do not expose stolen memory to the VM.
v2:
take Chris' comments:
- move the code into intel_update_fbc()
--- drivers/gpu/drm/i915/intel_pm.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/gpu/drm/i915/intel_pm.c +4 -0
@@ -544,6 +544,10 @@ void intel_update_fbc(struct drm_device *dev)return ; } +/* disable framebuffer compression in vgt */ +if (intel_vgpu_active (dev)) + i915.enable_fbc =0 ; +/* * If FBC is already on, we just have to verify that we can * keep it that way...
Signature · 1 line
1.9.1
PATCH v2 5/8 drm/i915: Add the display switch logic for vgpu in i915 driver
In reply to Yu ZhangDisplay switch logic is added to notify the vgt mediator that current vgpu have a valid surface to show. It does so by writing the display_ready field in PV INFO page, and then will be handled in vgt mediator. This is useful to avoid trickiness when the VM's framebuffer is being accessed in the middle of VM modesetting, e.g. compositing the framebuffer in the host side.
v2:
- move the notification code outside the 'else' in load sequence
- remove the notification code in intel_crtc_set_config()
--- drivers/gpu/drm/i915/i915_dma.c | 11 +++++++++++ drivers/gpu/drm/i915/i915_vgt_if.h | 8 ++++++++ 2 files changed, 19 insertions(+)
drivers/gpu/drm/i915/i915_dma.c +11 -0
@@ -35,6 +35,7 @@#include <drm/drm_legacy.h> #include "intel_drv.h" #include <drm/i915_drm.h> +#include "i915_vgt_if.h" #include "i915_drv.h" #include "i915_trace.h" #include <linux/pci.h>
@@ -1780,6 +1781,16 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags) dev_priv->ums.mm_suspended =1 ; } +if (intel_vgpu_active (dev)) { +/* +* Notify a valid surface after modesetting, +* when running inside a VM. +*/ +struct drm_i915_private *dev_priv =to_i915 (dev); + +I915_WRITE (vgtif_reg (display_ready),VGT_DRV_DISPLAY_READY ); + } +i915_setup_sysfs (dev);if (INTEL_INFO (dev)->num_pipes) {
drivers/gpu/drm/i915/i915_vgt_if.h +8 -0
@@ -82,4 +82,12 @@ struct vgt_if {#define vgtif_reg (x) \ (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x) +/* +* The information set by the guest gfx driver, through the display_ready field +*/ +enum vgt_display_status { +VGT_DRV_DISPLAY_NOT_READY =0 , +VGT_DRV_DISPLAY_READY ,/* ready for display switch */ +}; +#endif /* _I915_VGT_IF_H_ */
Signature · 1 line
1.9.1
PATCH v2 6/8 drm/i915: Disable power management for i915 driver in VM
In reply to Yu ZhangIn XenGT, GPU power management is controlled by host i915 driver, so there is no need to provide virtualized GPU PM support. In the future it might be useful to gather VM input for freq boost, but now let's disable it simply.
v2:
take Chris' comments:
- do not special case this to gen6+
--- drivers/gpu/drm/i915/intel_pm.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/gpu/drm/i915/intel_pm.c +4 -0
@@ -5304,6 +5304,10 @@ void intel_enable_gt_powersave(struct drm_device *dev) {struct drm_i915_private *dev_priv = dev->dev_private; +/* Powersaving is controlled by the host when inside a VM */ +if (intel_vgpu_active (dev)) +return ; +if (IS_IRONLAKE_M (dev)) {mutex_lock (&dev->struct_mutex);ironlake_enable_drps (dev);
Signature · 1 line
1.9.1
PATCH v2 7/8 drm/i915: Create vgpu specific write MMIO to reduce traps
In reply to Yu ZhangIn the virtualized environment, forcewake operations are not necessory for the driver, because mmio accesses will be trapped and emulated by the host side, and real forcewake operations are also done in the host. New mmio write handlers are added to directly call the __raw_i915_write, therefore will reduce many traps and increase the overall performance for drivers runing in the VM with Intel GVT-g enhancement.
v2:
take Chris' comments:
- register the mmio hooks in intel_uncore_init()
--- drivers/gpu/drm/i915/intel_uncore.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
drivers/gpu/drm/i915/intel_uncore.c +20 -0
@@ -719,6 +719,14 @@ hsw_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace)REG_WRITE_FOOTER ; \ } +#define __vgpu_write (x) \ +static void \ +vgpu_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace) { \ + REG_WRITE_HEADER; \ + __raw_i915_write##x(dev_priv, reg, val); \ + REG_WRITE_FOOTER; \ +} +static const u32 gen8_shadowed_regs[] = {FORCEWAKE_MT ,GEN6_RPNSWREQ ,
@@ -813,6 +821,10 @@ __gen4_write(8) __gen4_write(16) __gen4_write(32) __gen4_write(64) +__vgpu_write(8) +__vgpu_write(16) +__vgpu_write(32) +__vgpu_write(64)#undef __chv_write#undef __gen8_write
@@ -820,6 +832,7 @@ __gen4_write(64)#undef __gen6_write#undef __gen5_write#undef __gen4_write +#undef __vgpu_write#undef REG_WRITE_FOOTER#undef REG_WRITE_HEADER
@@ -950,6 +963,13 @@ void intel_uncore_init(struct drm_device *dev) dev_priv->uncore.funcs.mmio_readq = gen4_read64;break ; } + +if (intel_vgpu_active (dev)) { + dev_priv->uncore.funcs.mmio_writeb = vgpu_write8; + dev_priv->uncore.funcs.mmio_writew = vgpu_write16; + dev_priv->uncore.funcs.mmio_writel = vgpu_write32; + dev_priv->uncore.funcs.mmio_writeq = vgpu_write64; + } }void intel_uncore_fini (struct drm_device *dev)
Signature · 1 line
1.9.1
PATCH v2 8/8 drm/i915: Support alias ppgtt in VM if ppgtt is enabled
In reply to Yu ZhangThe current XenGT only supports alias ppgtt. And the emulation is done in XenGT host by first trapping PP_DIR_BASE mmio accesses. Updating PP_DIR_BASE by using instructions such as MI_LOAD_REGISTER_IMM are hard to detect and are not supported in current XenGT. Therefore this patch also added a new callback routine - vgpu_mm_switch() to set the PP_DIR_BASE by mmio writes.
v2:
take Chris' comments:
- move the code into sanitize_enable_ppgtt()
--- drivers/gpu/drm/i915/i915_gem_gtt.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
drivers/gpu/drm/i915/i915_gem_gtt.c +16 -0
@@ -197,6 +197,9 @@ static int sanitize_enable_ppgtt(struct drm_device *dev, int enable_ppgtt)if (IS_GEN8 (dev)) has_full_ppgtt = false;/* XXX why? */ +if (intel_vgpu_active (dev)) + has_full_ppgtt = false;/* emulation is too hard */ +if (enable_ppgtt ==0 || !has_aliasing_ppgtt)return 0 ;
@@ -886,6 +889,16 @@ static int hsw_mm_switch(struct i915_hw_ppgtt *ppgtt,return 0 ; } +static int vgpu_mm_switch (struct i915_hw_ppgtt *ppgtt, +struct intel_engine_cs *ring) +{ +struct drm_i915_private *dev_priv =to_i915 (ppgtt->base.dev); + +I915_WRITE (RING_PP_DIR_DCLV (ring),PP_DIR_DCLV_2G ); +I915_WRITE (RING_PP_DIR_BASE (ring),get_pd_offset (ppgtt)); +return 0 ; +} +static int gen7_mm_switch (struct i915_hw_ppgtt *ppgtt,struct intel_engine_cs *ring) {
@@ -1212,6 +1225,9 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt) }else BUG (); +if (intel_vgpu_active (dev)) + ppgtt->switch_mm = vgpu_mm_switch; + ret =gen6_ppgtt_alloc (ppgtt);if (ret)return ret;
Signature · 1 line
1.9.1
Quoted 59 lines · Yu Zhang
Intel GVT-g (previously known as XenGT), is a complete GPU virtualization solution with mediated pass-through for 4th generation Intel Core processors - Haswell platform. This technology presents a virtual full-fledged GPU to each Virtual Machine (VM). VMs can directly access performance-critical resources, without intervention from the hypervisor in most cases, while privileged operations from VMs are trap-and-emulated at minimal cost. For detail, please refer to
https://01.org/xen/blogs/wangbo85/2014/intel-gvt-gxengt-pubic-releaseThis patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, were sent out in another patchset.
The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory w/o hypervisor's intervention, e.g. filling textures and queuing commands. However w/ the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses w/o complexity of translation.
Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (w/ some faulting mechanism).
The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource XenGT also partitions them among VMs.
Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.
Yu Zhang (8):
drm/i915: Introduce a PV INFO page structure for Intel GVT-g. drm/i915: Adds graphic address space ballooning logic drm/i915: Partition the fence registers for vgpu in i915 driver drm/i915: Disable framebuffer compression for i915 driver in VM drm/i915: Add the display switch logic for vgpu in i915 driver drm/i915: Disable power management for i915 driver in VM drm/i915: Create vgpu specific write MMIO to reduce traps drm/i915: Support alias ppgtt in VM if ppgtt is enabled
Quoted 9 lines
drivers/gpu/drm/i915/i915_dma.c | 11 +++ drivers/gpu/drm/i915/i915_drv.h | 12 +++ drivers/gpu/drm/i915/i915_gem.c | 5 + drivers/gpu/drm/i915/i915_gem_gtt.c | 186 +++++++++++++++++++++++++++++++++++- drivers/gpu/drm/i915/i915_vgt_if.h | 93 ++++++++++++++++++ drivers/gpu/drm/i915/intel_pm.c | 8 ++ drivers/gpu/drm/i915/intel_uncore.c | 22 +++++ 7 files changed, 334 insertions(+), 3 deletions(-) create mode 100644 drivers/gpu/drm/i915/i915_vgt_if.h
I seem to have two v2 versions of this patch series. Anything changed or why the resend? I didn't see any comment on the older version ... -Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Quoted 70 lines · Daniel Vetter
Quoted 59 lines · Yu Zhang
Intel GVT-g (previously known as XenGT), is a complete GPU virtualization solution with mediated pass-through for 4th generation Intel Core processors - Haswell platform. This technology presents a virtual full-fledged GPU to each Virtual Machine (VM). VMs can directly access performance-critical resources, without intervention from the hypervisor in most cases, while privileged operations from VMs are trap-and-emulated at minimal cost. For detail, please refer to
https://01.org/xen/blogs/wangbo85/2014/intel-gvt-gxengt-pubic-releaseThis patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, were sent out in another patchset.
The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory w/o hypervisor's intervention, e.g. filling textures and queuing commands. However w/ the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses w/o complexity of translation.
Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (w/ some faulting mechanism).
The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource XenGT also partitions them among VMs.
Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.
Yu Zhang (8):
drm/i915: Introduce a PV INFO page structure for Intel GVT-g. drm/i915: Adds graphic address space ballooning logic drm/i915: Partition the fence registers for vgpu in i915 driver drm/i915: Disable framebuffer compression for i915 driver in VM drm/i915: Add the display switch logic for vgpu in i915 driver drm/i915: Disable power management for i915 driver in VM drm/i915: Create vgpu specific write MMIO to reduce traps drm/i915: Support alias ppgtt in VM if ppgtt is enabledQuoted 9 lines
drivers/gpu/drm/i915/i915_dma.c | 11 +++ drivers/gpu/drm/i915/i915_drv.h | 12 +++ drivers/gpu/drm/i915/i915_gem.c | 5 + drivers/gpu/drm/i915/i915_gem_gtt.c | 186 +++++++++++++++++++++++++++++++++++- drivers/gpu/drm/i915/i915_vgt_if.h | 93 ++++++++++++++++++ drivers/gpu/drm/i915/intel_pm.c | 8 ++ drivers/gpu/drm/i915/intel_uncore.c | 22 +++++ 7 files changed, 334 insertions(+), 3 deletions(-) create mode 100644 drivers/gpu/drm/i915/i915_vgt_if.h
Quoted 2 lines
I seem to have two v2 versions of this patch series. Anything changed or why the resend? I didn't see any comment on the older version ...
Well, looked through it anyway. On a high level this looks good for the vgt integration for guests. I think we need some polish though still, specifically for documentation.
- Please extract all the various intel_vgt_* functions spread all over the tree into a new i915_vgt_if.c (or intel_vgt.c, but then the header should be changed, too I think).
- Please add kerneldoc to all the functions which are non-static and so used by the driver outside of your kernel module.
- Please add a DOC: section detailing some of the high-level design considerations of vGT and also put that into the new file. I think in the future we should also keep the guest<->host abi revisions in there (i.e. the stuff in PV_INFO).
- Please pull all the new documentation together and integrate it into the i915 section of the drm docbook. A good place is probably a new subsection "Paravirtualized Guest Support (vGPU)" under the driver core section.
There's quite a few i915 driver subsystems which are already nicely documented like this, but I also have a small howto:
http://blog.ffwll.ch/2014/06/documentation-for-drmi915.html
Cheers, Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Quoted 82 lines · Daniel Vetter
Quoted 70 lines · Daniel Vetter
Quoted 67 lines · Yu Zhang
Intel GVT-g (previously known as XenGT), is a complete GPU virtualization solution with mediated pass-through for 4th generation Intel Core processors - Haswell platform. This technology presents a virtual full-fledged GPU to each Virtual Machine (VM). VMs can directly access performance-critical resources, without intervention from the hypervisor in most cases, while privileged operations from VMs are trap-and-emulated at minimal cost. For detail, please refer to
https://01.org/xen/blogs/wangbo85/2014/intel-gvt-gxengt-pubic-releaseThis patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, were sent out in another patchset.
The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory w/o hypervisor's intervention, e.g. filling textures and queuing commands. However w/ the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses w/o complexity of translation.
Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (w/ some faulting mechanism).
The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource XenGT also partitions them among VMs.
Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.
Yu Zhang (8):
drm/i915: Introduce a PV INFO page structure for Intel GVT-g. drm/i915: Adds graphic address space ballooning logic drm/i915: Partition the fence registers for vgpu in i915 driver drm/i915: Disable framebuffer compression for i915 driver in VM drm/i915: Add the display switch logic for vgpu in i915 driver drm/i915: Disable power management for i915 driver in VM drm/i915: Create vgpu specific write MMIO to reduce traps drm/i915: Support alias ppgtt in VM if ppgtt is enableddrivers/gpu/drm/i915/i915_dma.c | 11 +++ drivers/gpu/drm/i915/i915_drv.h | 12 +++ drivers/gpu/drm/i915/i915_gem.c | 5 + drivers/gpu/drm/i915/i915_gem_gtt.c | 186 +++++++++++++++++++++++++++++++++++- drivers/gpu/drm/i915/i915_vgt_if.h | 93 ++++++++++++++++++ drivers/gpu/drm/i915/intel_pm.c | 8 ++ drivers/gpu/drm/i915/intel_uncore.c | 22 +++++ 7 files changed, 334 insertions(+), 3 deletions(-)Quoted 1 line
create mode 100644
drivers/gpu/drm/i915/i915_vgt_if.hQuoted 2 lines
I seem to have two v2 versions of this patch series. Anything changed or why the resend? I didn't see any comment on the older version ...
Well, looked through it anyway. On a high level this looks good for the vgt integration for guests. I think we need some polish though still, specifically for documentation.
- Please extract all the various intel_vgt_* functions spread all over the tree into a new
i915_vgt_if.c(orintel_vgt.c, but then the header should be changed, too I think).
Quoted 7 lines
- Please add kerneldoc to all the functions which are non-static and so used by the driver outside of your kernel module.
- Please add a DOC: section detailing some of the high-level design considerations of vGT and also put that into the new file. I think in the future we should also keep the guest<->host abi revisions in there (i.e. the stuff in
PV_INFO).
Sure. Thanks!
Quoted 5 lines
- Please pull all the new documentation together and integrate it into the i915 section of the drm docbook. A good place is probably a new subsection "Paravirtualized Guest Support (vGPU)" under the driver core section.
How about subsection name "Intel GVT-g Guest Support(vGPU)"? :)
Quoted 5 lines
There's quite a few i915 driver subsystems which are already nicely documented like this, but I also have a small howto:
Thanks! This blog helps a lot. :)
Quoted 3 lines
Cheers, Daniel
Quoted 6 lines · Zhang
Quoted 4 lines · Daniel Vetter
- Please pull all the new documentation together and integrate it into the i915 section of the drm docbook. A good place is probably a new subsection "Paravirtualized Guest Support (vGPU)" under the driver core section.
How about subsection name "Intel GVT-g Guest Support(vGPU)"? :)
Ack. Sounds a bit like marketing though ;-)
Cheers, Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Hi Daniel,
Thanks a lot for your reply. Indeed, I sent two v2 patches, because
the format of the first v2 patchset is incorrect - I forgot to add the what changed part in those messages. :)
Yu
Quoted 74 lines · Daniel Vetter
Quoted 67 lines · Yu Zhang
Intel GVT-g (previously known as XenGT), is a complete GPU virtualization solution with mediated pass-through for 4th generation Intel Core processors - Haswell platform. This technology presents a virtual full-fledged GPU to each Virtual Machine (VM). VMs can directly access performance-critical resources, without intervention from the hypervisor in most cases, while privileged operations from VMs are trap-and-emulated at minimal cost. For detail, please refer to
https://01.org/xen/blogs/wangbo85/2014/intel-gvt-gxengt-pubic-releaseThis patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, were sent out in another patchset.
The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory w/o hypervisor's intervention, e.g. filling textures and queuing commands. However w/ the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses w/o complexity of translation.
Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (w/ some faulting mechanism).
The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource XenGT also partitions them among VMs.
Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.
Yu Zhang (8):
drm/i915: Introduce a PV INFO page structure for Intel GVT-g. drm/i915: Adds graphic address space ballooning logic drm/i915: Partition the fence registers for vgpu in i915 driver drm/i915: Disable framebuffer compression for i915 driver in VM drm/i915: Add the display switch logic for vgpu in i915 driver drm/i915: Disable power management for i915 driver in VM drm/i915: Create vgpu specific write MMIO to reduce traps drm/i915: Support alias ppgtt in VM if ppgtt is enableddrivers/gpu/drm/i915/i915_dma.c | 11 +++ drivers/gpu/drm/i915/i915_drv.h | 12 +++ drivers/gpu/drm/i915/i915_gem.c | 5 + drivers/gpu/drm/i915/i915_gem_gtt.c | 186 +++++++++++++++++++++++++++++++++++- drivers/gpu/drm/i915/i915_vgt_if.h | 93 ++++++++++++++++++ drivers/gpu/drm/i915/intel_pm.c | 8 ++ drivers/gpu/drm/i915/intel_uncore.c | 22 +++++ 7 files changed, 334 insertions(+), 3 deletions(-)Quoted 1 line
create mode 100644
drivers/gpu/drm/i915/i915_vgt_if.hI seem to have two v2 versions of this patch series. Anything changed or why the resend? I didn't see any comment on the older version ... -Daniel