Add enlightenments for vGPU
Subsystems (3)
v4 cover Add enlightenments for vGPU
This patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, will be sent out in other patches.
The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory without hypervisor's intervention, e.g. filling textures or queuing commands. However with the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses without complexity of translation.
Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (with some faulting mechanism).
The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource, XenGT also partitions them among VMs.
Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.
Yu Zhang (8):
drm/i915: Introduce a PV INFO page structure for Intel GVT-g. drm/i915: Adds graphic address space ballooning logic drm/i915: Partition the fence registers for vGPU in i915 driver drm/i915: Disable framebuffer compression for i915 driver in VM drm/i915: Add the display switch logic for vGPU in i915 driver drm/i915: Disable power management for i915 driver in VM drm/i915: Create vGPU specific MMIO operations to reduce traps drm/i915: Support alias ppgtt in VM if ppgtt is enabled
Documentation/DocBook/drm.tmpl | 5 + drivers/gpu/drm/i915/Makefile | 3 + drivers/gpu/drm/i915/i915_dma.c | 8 ++ drivers/gpu/drm/i915/i915_drv.h | 10 ++ drivers/gpu/drm/i915/i915_gem.c | 5 + drivers/gpu/drm/i915/i915_gem_gtt.c | 33 ++++- drivers/gpu/drm/i915/i915_vgpu.c | 264 ++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_vgpu.h | 91 +++++++++++++ drivers/gpu/drm/i915/intel_fbc.c | 4 + drivers/gpu/drm/i915/intel_pm.c | 4 + drivers/gpu/drm/i915/intel_uncore.c | 34 +++++ 11 files changed, 458 insertions(+), 3 deletions(-) create mode 100644 drivers/gpu/drm/i915/i915_vgpu.c create mode 100644 drivers/gpu/drm/i915/i915_vgpu.h
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v4 1/8 drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
Introduce a PV INFO structure, to facilitate the Intel GVT-g technology, which is a GPU virtualization solution with mediated pass-through. This page contains the shared information between i915 driver and the host emulator. For now, this structure utilizes an area of 4K bytes on HSW GPU's unused MMIO space. Future hardware will have the reserved window architecturally defined, and layout of the page will be added in future BSpec.
The i915 driver load routine detects if it is running in a VM by reading the contents of this PV INFO page. Thereafter a flag, vgpu.active is set, and intel_vgpu_active() is used by checking this flag to conclude if GPU is virtualized with Intel GVT-g. By now, intel_vgpu_active() will return true, only when the driver is running as a guest in the Intel GVT-g enhanced environment on HSW platform.
v2:
take Chris' comments:
- call the i915_check_vgpu() in intel_uncore_init()
- sanitize i915_check_vgpu() by adding BUILD_BUG_ON() and debug info
take Daniel's comments:
- put the definition of PV INFO into a new header - i915_vgt_if.h
other changes:
- access mmio regs by readq/readw in i915_check_vgpu()
v3:
take Daniel's comments:
- move the i915/vgt interfaces into a new i915_vgpu.c
- update makefile
- add kerneldoc to functions which are non-static
- add a DOC: section describing some of the high-level design
- update drm docbook
other changes:
- rename i915_vgt_if.h to i915_vgpu.h
v4:
take Tvrtko's comments:
- fix a typo in commit message
- add debug message when vgt version mismatches
- rename low_gmadr/high_gmadr to mappable/non-mappable in PV INFO structure
--- Documentation/DocBook/drm.tmpl | 5 +++ drivers/gpu/drm/i915/Makefile | 3 ++ drivers/gpu/drm/i915/i915_drv.h | 10 +++++ drivers/gpu/drm/i915/i915_vgpu.c | 86 +++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_vgpu.h | 85 ++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/intel_uncore.c | 3 ++ 6 files changed, 192 insertions(+) create mode 100644 drivers/gpu/drm/i915/i915_vgpu.c create mode 100644 drivers/gpu/drm/i915/i915_vgpu.h
Documentation/DocBook/drm.tmpl +5 -0
@@ -3979,6 +3979,11 @@ int num_ioctls;</synopsis> !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_disable_interrupts !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_enable_interrupts </sect2> + <sect2> + <title>Intel GVT-g Guest Support(vGPU)</title> +!Pdrivers/gpu/drm/i915/i915_vgpu.c Intel GVT-g guest support +!Idrivers/gpu/drm/i915/i915_vgpu.c + </sect2> </sect1> <sect1> <title>Display Hardware Handling</title>
drivers/gpu/drm/i915/Makefile +3 -0
@@ -83,6 +83,9 @@ i915-y += dvo_ch7017.o \ intel_sdvo.o \ intel_tv.o +# virtual gpu code +i915-y += i915_vgpu.o + # legacy horrors i915-y += i915_dma.o \ i915_ums.o
drivers/gpu/drm/i915/i915_drv.h +10 -0
@@ -1640,6 +1640,10 @@ struct i915_workarounds {u32 count; }; +struct i915_virtual_gpu { +bool active; +}; +struct drm_i915_private {struct drm_device *dev;struct kmem_cache *slab;
@@ -1652,6 +1656,8 @@ struct drm_i915_private {struct intel_uncore uncore; +struct i915_virtual_gpu vgpu; +struct intel_gmbus gmbus[GMBUS_NUM_PORTS ];
@@ -2583,6 +2589,10 @@ void intel_uncore_forcewake_get(struct drm_i915_private *dev_priv,void intel_uncore_forcewake_put (struct drm_i915_private *dev_priv,enum forcewake_domains domains);void assert_forcewakes_inactive (struct drm_i915_private *dev_priv); +static inline bool intel_vgpu_active (struct drm_device *dev) +{ +return to_i915 (dev)->vgpu.active; +} void i915_enable_pipestat(struct drm_i915_private *dev_priv, enum pipe pipe,
drivers/gpu/drm/i915/i915_vgpu.c new file +86 -0
86 lines not on this page · raw message
drivers/gpu/drm/i915/i915_vgpu.h new file +85 -0
85 lines not on this page · raw message
drivers/gpu/drm/i915/intel_uncore.c +3 -0
@@ -23,6 +23,7 @@#include "i915_drv.h" #include "intel_drv.h" +#include "i915_vgpu.h" #include <linux/pm_runtime.h>
@@ -1082,6 +1083,8 @@ void intel_uncore_init(struct drm_device *dev) {struct drm_i915_private *dev_priv = dev->dev_private; +i915_check_vgpu (dev); +intel_uncore_ellc_detect (dev);intel_uncore_fw_domains_init (dev);__intel_uncore_early_sanitize (dev, false);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Dave Airlie
maintainerRe: [PATCH v4 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
drivers/gpu/drm/i915/i915_vgpu.h line 35
Quoted 5 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +5
-0+/* +* The following structure pages are defined in GEN MMIO space +* for virtualization. (One page for now) +*/ +#define VGT_MAGIC 0x4776544776544776/* 'vGTvGTvG' */
one of my dusty 32-bit compilers dislikes this,
CC [M] drivers/gpu/drm/i915/i915_vgpu.o/home/airlied/kernel/drm-2.6/drivers/gpu/drm/i915/i915_vgpu.c: In function ‘i915_check_vgpu’:/home/airlied/kernel/drm-2.6/drivers/gpu/drm/i915/i915_vgpu.c:73:
warning: integer constant is too large for ‘long’ type
Dave.
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v4 2/8 drm/i915: Adds graphic address space ballooning logic
In reply to Yu ZhangWith Intel GVT-g, the global graphic memory space is partitioned by multiple vGPU instances in different VMs. The ballooning code is called in i915_gem_setup_global_gtt(), utilizing the drm mm allocator APIs to mark the graphic address space which are partitioned out to other vGPUs as reserved. With ballooning, host side does not need to translate a grahpic address from guest view to host view. By now, current implementation only support the static ballooning, but in the future, with more cooperation from guest driver, the same interfaces can be extended to grow/shrink the guest graphic memory dynamically.
v2:
take Chris and Daniel's comments:
- no guard page between different VMs
- use drm_mm_reserve_node() to do the reservation for ballooning, instead of the previous drm_mm_insert_node_in_range_generic()
v3:
take Daniel's comments:
- move ballooning functions into i915_vgpu.c
- add kerneldoc to ballooning functions
v4:
take Tvrtko's comments:
- more accurate comments and commit message
--- drivers/gpu/drm/i915/i915_gem_gtt.c | 17 +++- drivers/gpu/drm/i915/i915_vgpu.c | 178 ++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_vgpu.h | 2 + 3 files changed, 194 insertions(+), 3 deletions(-)
drivers/gpu/drm/i915/i915_gem_gtt.c +14 -3
@@ -27,6 +27,7 @@#include <drm/drmP.h> #include <drm/i915_drm.h> #include "i915_drv.h" +#include "i915_vgpu.h" #include "i915_trace.h" #include "intel_drv.h"
@@ -1756,6 +1757,16 @@ static int i915_gem_setup_global_gtt(struct drm_device *dev,/* Subtract the guard page ... */ drm_mm_init (&ggtt_vm->mm, start, end - start -PAGE_SIZE ); + + dev_priv->gtt.base.start = start; + dev_priv->gtt.base.total = end - start; + +if (intel_vgpu_active (dev)) { + ret =intel_vgt_balloon (dev); +if (ret) +return ret; + } +if (!HAS_LLC (dev)) dev_priv->gtt.base.mm.color_adjust = i915_gtt_color_adjust;
@@ -1775,9 +1786,6 @@ static int i915_gem_setup_global_gtt(struct drm_device *dev, vma->bound |=GLOBAL_BIND ; } - dev_priv->gtt.base.start = start; - dev_priv->gtt.base.total = end - start; -/* Clear any non-preallocated blocks */ drm_mm_for_each_hole (entry, &ggtt_vm->mm, hole_start, hole_end) { DRM_DEBUG_KMS("clearing unused GTT space: [%lx, %lx]\n",
@@ -1829,6 +1837,9 @@ void i915_global_gtt_cleanup(struct drm_device *dev) }if (drm_mm_initialized (&vm->mm)) { +if (intel_vgpu_active (dev)) +intel_vgt_deballoon (); +drm_mm_takedown (&vm->mm);list_del (&vm->global_link); }
drivers/gpu/drm/i915/i915_vgpu.c +178 -0
181 lines not on this page · raw message
drivers/gpu/drm/i915/i915_vgpu.h +2 -0
@@ -81,5 +81,7 @@ struct vgt_if { (VGT_PVINFO_PAGE + (long)&((struct vgt_if *)NULL)->x)extern void i915_check_vgpu (struct drm_device *dev); +extern int intel_vgt_balloon (struct drm_device *dev); +extern void intel_vgt_deballoon (void );#endif /* _I915_VGPU_H_ */
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v4 3/8 drm/i915: Partition the fence registers for vGPU in i915 driver
In reply to Yu ZhangWith Intel GVT-g, the fence registers are partitioned by multiple vGPU instances in different VMs. Routine i915_gem_load() is modified to reset the num_fence_regs, when the driver detects it's running in a VM. Accesses to the fence registers from vGPU will be trapped and remapped by the host side. And the allocated fence number is provided in PV INFO page structure. By now, the value of fence number is fixed, but in the future we can relax this limitation, to allocate the fence registers dynamically from host side.
--- drivers/gpu/drm/i915/i915_gem.c | 5 +++++ 1 file changed, 5 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +5 -0
@@ -29,6 +29,7 @@#include <drm/drm_vma_manager.h> #include <drm/i915_drm.h> #include "i915_drv.h" +#include "i915_vgpu.h" #include "i915_trace.h" #include "intel_drv.h" #include <linux/oom.h>
@@ -4987,6 +4988,10 @@ i915_gem_load(struct drm_device *dev)else dev_priv->num_fence_regs =8 ; +if (intel_vgpu_active (dev)) + dev_priv->num_fence_regs = +I915_READ (vgtif_reg (avail_rs.fence_num)); +/* Initialize fence registers to zero */ INIT_LIST_HEAD (&dev_priv->mm.fence_list);i915_gem_restore_fences (dev);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v4 4/8 drm/i915: Disable framebuffer compression for i915 driver in VM
In reply to Yu ZhangFramebuffer compression is disabled when driver detects it's running in a Intel GVT-g enlightened VM, because FBC is not emulated and there is no stolen memory for a vGPU.
v2:
take Chris' comments:
- move the code into intel_update_fbc()
v4:
take Tvrtko's comments:
- rebase the code into intel_fbc_update()
--- drivers/gpu/drm/i915/intel_fbc.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/gpu/drm/i915/intel_fbc.c +4 -0
@@ -505,6 +505,10 @@ void intel_fbc_update(struct drm_device *dev)if (!HAS_FBC (dev))return ; +/* disable framebuffer compression in vGPU */ +if (intel_vgpu_active (dev)) + i915.enable_fbc =0 ; +if (i915.enable_fbc <0 ) {if (set_no_fbc_reason (dev_priv,FBC_CHIP_DEFAULT ))DRM_DEBUG_KMS ("disabled per chip default\n" );
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v4 5/8 drm/i915: Add the display switch logic for vGPU in i915 driver
In reply to Yu ZhangDisplay switch logic is added to notify the host side that current vGPU have a valid surface to show. It does so by writing the display_ready field in PV INFO page, and then will be handled in the host side. This is useful to avoid trickiness when the VM's framebuffer is being accessed in the middle of VM modesetting, e.g. compositing the framebuffer in the host side.
v2:
- move the notification code outside the 'else' in load sequence
- remove the notification code in intel_crtc_set_config()
v4:
- code rebase, no need to define another dev_priv
- use #define instead of enum for display readiness
--- drivers/gpu/drm/i915/i915_dma.c | 8 ++++++++ drivers/gpu/drm/i915/i915_vgpu.h | 4 ++++ 2 files changed, 12 insertions(+)
drivers/gpu/drm/i915/i915_dma.c +8 -0
@@ -36,6 +36,7 @@#include "intel_drv.h" #include <drm/i915_drm.h> #include "i915_drv.h" +#include "i915_vgpu.h" #include "i915_trace.h" #include <linux/pci.h> #include <linux/console.h>
@@ -842,6 +843,13 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags) } } +/* +* Notify a valid surface after modesetting, +* when running inside a VM. +*/ +if (intel_vgpu_active (dev)) +I915_WRITE (vgtif_reg (display_ready),VGT_DRV_DISPLAY_READY ); +i915_setup_sysfs (dev);if (INTEL_INFO (dev)->num_pipes) {
drivers/gpu/drm/i915/i915_vgpu.h +4 -0
@@ -80,6 +80,10 @@ struct vgt_if {#define vgtif_reg (x) \ (VGT_PVINFO_PAGE + (long)&((struct vgt_if *)NULL)->x) +/* vGPU display status to be used by the host side */ +#define VGT_DRV_DISPLAY_NOT_READY 0 +#define VGT_DRV_DISPLAY_READY 1/* ready for display switch */ +extern void i915_check_vgpu (struct drm_device *dev);extern int intel_vgt_balloon (struct drm_device *dev);extern void intel_vgt_deballoon (void );
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v4 6/8 drm/i915: Disable power management for i915 driver in VM
In reply to Yu ZhangWith Intel GVT-g, GPU power management is controlled by host driver, so there is no need to provide virtualized GPU PM support. In the future it might be useful to gather VM input for freq boost, but now let's disable it simply.
v2:
take Chris' comments:
- do not special case this to gen6+
--- drivers/gpu/drm/i915/intel_pm.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/gpu/drm/i915/intel_pm.c +4 -0
@@ -5623,6 +5623,10 @@ void intel_enable_gt_powersave(struct drm_device *dev) {struct drm_i915_private *dev_priv = dev->dev_private; +/* Powersaving is controlled by the host when inside a VM */ +if (intel_vgpu_active (dev)) +return ; +if (IS_IRONLAKE_M (dev)) {mutex_lock (&dev->struct_mutex);ironlake_enable_drps (dev);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v4 7/8 drm/i915: Create vGPU specific MMIO operations to reduce traps
In reply to Yu ZhangIn the virtualized environment, forcewake operations are not necessary for the driver, because mmio accesses will be trapped and emulated by the host side, and real forcewake operations are also done in the host. New mmio access handlers are added to directly call the __raw_i915_read/write, therefore will reduce many traps and increase the overall performance for drivers running in the VM with Intel GVT-g enhancement.
v2:
take Chris' comments:
- register the mmio hooks in intel_uncore_init()
v3:
take Daniel's comments:
- use macros to assign mmio write functions for vGPU
v4:
take Tvrtko's comments:
- also use mmio hooks for read operations
--- drivers/gpu/drm/i915/intel_uncore.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
drivers/gpu/drm/i915/intel_uncore.c +31 -0
@@ -641,6 +641,14 @@ static inline void __force_wake_get(struct drm_i915_private *dev_priv, dev_priv->uncore.funcs.force_wake_get (dev_priv, fw_domains); } +#define __vgpu_read (x) \ +static u##x \ +vgpu_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \ + GEN6_READ_HEADER(x); \ + val = __raw_i915_read##x(dev_priv, reg); \ + GEN6_READ_FOOTER; \ +} +#define __gen6_read (x) \ static u##x \ gen6_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
@@ -704,6 +712,10 @@ gen9_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \GEN6_READ_FOOTER ; \ } +__vgpu_read(8) +__vgpu_read(16) +__vgpu_read(32) +__vgpu_read(64) __gen9_read(8) __gen9_read(16)__gen9_read (32 )
@@ -725,6 +737,7 @@ __gen6_read(64)#undef __chv_read#undef __vlv_read#undef __gen6_read +#undef __vgpu_read#undef GEN6_READ_FOOTER#undef GEN6_READ_HEADER
@@ -808,6 +821,14 @@ hsw_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace)GEN6_WRITE_FOOTER ; \ } +#define __vgpu_write (x) \ +static void vgpu_write##x(struct drm_i915_private *dev_priv, \ + off_t reg, u##x val, bool trace) { \ + GEN6_WRITE_HEADER; \ + __raw_i915_write##x(dev_priv, reg, val); \ + GEN6_WRITE_FOOTER; \ +} +static const u32 gen8_shadowed_regs[] = {FORCEWAKE_MT ,GEN6_RPNSWREQ ,
@@ -925,12 +946,17 @@ __gen6_write(8) __gen6_write(16) __gen6_write(32) __gen6_write(64) +__vgpu_write(8) +__vgpu_write(16) +__vgpu_write(32) +__vgpu_write(64)#undef __gen9_write#undef __chv_write#undef __gen8_write#undef __hsw_write#undef __gen6_write +#undef __vgpu_write#undef GEN6_WRITE_FOOTER#undef GEN6_WRITE_HEADER
@@ -1133,6 +1159,11 @@ void intel_uncore_init(struct drm_device *dev)break ; } +if (intel_vgpu_active (dev)) { +ASSIGN_WRITE_MMIO_VFUNCS (vgpu); +ASSIGN_READ_MMIO_VFUNCS (vgpu); + } +i915_check_and_clear_faults (dev); }#undef ASSIGN_WRITE_MMIO_VFUNCS
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v4 8/8 drm/i915: Support alias ppgtt in VM if ppgtt is enabled
In reply to Yu ZhangThe current Intel GVT-g only supports alias ppgtt. And the emulation is done in the host by first trapping PP_DIR_BASE mmio accesses. Updating PP_DIR_BASE by using instructions such as MI_LOAD_REGISTER_IMM are hard to detect and are not supported in current code. Therefore this patch also adds a new callback routine - vgpu_mm_switch() to set the PP_DIR_BASE by mmio writes.
v2:
take Chris' comments:
- move the code into sanitize_enable_ppgtt()
v4:
take Tvrtko's comments:
- fix the parenthesis alignment warning
--- drivers/gpu/drm/i915/i915_gem_gtt.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
drivers/gpu/drm/i915/i915_gem_gtt.c +16 -0
@@ -104,6 +104,9 @@ static int sanitize_enable_ppgtt(struct drm_device *dev, int enable_ppgtt) has_aliasing_ppgtt =INTEL_INFO (dev)->gen >=6 ; has_full_ppgtt =INTEL_INFO (dev)->gen >=7 ; +if (intel_vgpu_active (dev)) + has_full_ppgtt = false;/* emulation is too hard */ +/* * We don't allow disabling PPGTT for gen9+ as it's a requirement for * execlists, the sole mechanism available to submit work.
@@ -798,6 +801,16 @@ static int hsw_mm_switch(struct i915_hw_ppgtt *ppgtt,return 0 ; } +static int vgpu_mm_switch (struct i915_hw_ppgtt *ppgtt, +struct intel_engine_cs *ring) +{ +struct drm_i915_private *dev_priv =to_i915 (ppgtt->base.dev); + +I915_WRITE (RING_PP_DIR_DCLV (ring),PP_DIR_DCLV_2G ); +I915_WRITE (RING_PP_DIR_BASE (ring),get_pd_offset (ppgtt)); +return 0 ; +} +static int gen7_mm_switch (struct i915_hw_ppgtt *ppgtt,struct intel_engine_cs *ring) {
@@ -1127,6 +1140,9 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt) }else BUG (); +if (intel_vgpu_active (dev)) + ppgtt->switch_mm = vgpu_mm_switch; + ret =gen6_ppgtt_alloc (ppgtt);if (ret)return ret;
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 40 lines · Yu Zhang
This patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, will be sent out in other patches.
The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory without hypervisor's intervention, e.g. filling textures or queuing commands. However with the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses without complexity of translation.
Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (with some faulting mechanism).
The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource, XenGT also partitions them among VMs.
Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.
Quoted 9 lines
Yu Zhang (8):
drm/i915: Introduce a PV INFO page structure for Intel GVT-g. drm/i915: Adds graphic address space ballooning logic drm/i915: Partition the fence registers for vGPU in i915 driver drm/i915: Disable framebuffer compression for i915 driver in VM drm/i915: Add the display switch logic for vGPU in i915 driver drm/i915: Disable power management for i915 driver in VM drm/i915: Create vGPU specific MMIO operations to reduce traps drm/i915: Support alias ppgtt in VM if ppgtt is enabled
All my comments have been addressed (and I especially like the ASCII diagram of the memory space!) so you can put my r-b on all the patches from this series:
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 51 lines · Tvrtko Ursulin
Quoted 40 lines · Yu Zhang
This patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, will be sent out in other patches.
The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory without hypervisor's intervention, e.g. filling textures or queuing commands. However with the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses without complexity of translation.
Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (with some faulting mechanism).
The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource, XenGT also partitions them among VMs.
Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.
Quoted 9 lines
Yu Zhang (8):
drm/i915: Introduce a PV INFO page structure for Intel GVT-g. drm/i915: Adds graphic address space ballooning logic drm/i915: Partition the fence registers for vGPU in i915 driver drm/i915: Disable framebuffer compression for i915 driver in VM drm/i915: Add the display switch logic for vGPU in i915 driver drm/i915: Disable power management for i915 driver in VM drm/i915: Create vGPU specific MMIO operations to reduce traps drm/i915: Support alias ppgtt in VM if ppgtt is enabled
Quoted 5 lines
All my comments have been addressed (and I especially like the ASCII diagram of the memory space!) so you can put my r-b on all the patches from this series:
Reviewed-by Tvrtko Ursulin <[email protected]>
Thank you very much, Tvrtko. This is great. :-)
Btw, should I resend another version patch series, which add the "Reviewed-by" line, or will Daniel directly merge these patches and add the "Reviewed-by" at that time? I'm willing to take whatever actions necessary, but I'm not familiar with this process. Thanks again. :)
Yu
Quoted 6 lines
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 51 lines · Tvrtko Ursulin
Quoted 40 lines · Yu Zhang
This patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, will be sent out in other patches.
The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory without hypervisor's intervention, e.g. filling textures or queuing commands. However with the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses without complexity of translation.
Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (with some faulting mechanism).
The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource, XenGT also partitions them among VMs.
Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.
Quoted 9 lines
Yu Zhang (8):
drm/i915: Introduce a PV INFO page structure for Intel GVT-g. drm/i915: Adds graphic address space ballooning logic drm/i915: Partition the fence registers for vGPU in i915 driver drm/i915: Disable framebuffer compression for i915 driver in VM drm/i915: Add the display switch logic for vGPU in i915 driver drm/i915: Disable power management for i915 driver in VM drm/i915: Create vGPU specific MMIO operations to reduce traps drm/i915: Support alias ppgtt in VM if ppgtt is enabled
Quoted 5 lines
All my comments have been addressed (and I especially like the ASCII diagram of the memory space!) so you can put my r-b on all the patches from this series:
Reviewed-by Tvrtko Ursulin <[email protected]>
All merged, thanks for patches&review.
-Daniel
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 52 lines · Daniel Vetter
Quoted 51 lines · Tvrtko Ursulin
Quoted 49 lines · Yu Zhang
This patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, will be sent out in other patches.
The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory without hypervisor's intervention, e.g. filling textures or queuing commands. However with the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses without complexity of translation.
Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (with some faulting mechanism).
The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource, XenGT also partitions them among VMs.
Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.
Yu Zhang (8):
drm/i915: Introduce a PV INFO page structure for Intel GVT-g. drm/i915: Adds graphic address space ballooning logic drm/i915: Partition the fence registers for vGPU in i915 driver drm/i915: Disable framebuffer compression for i915 driver in VM drm/i915: Add the display switch logic for vGPU in i915 driver drm/i915: Disable power management for i915 driver in VM drm/i915: Create vGPU specific MMIO operations to reduce traps drm/i915: Support alias ppgtt in VM if ppgtt is enabled
Quoted 8 lines
Quoted 5 lines
All my comments have been addressed (and I especially like the ASCII diagram of the memory space!) so you can put my r-b on all the patches from this series:
Reviewed-by Tvrtko Ursulin <[email protected]>All merged, thanks for patches&review.
-Daniel
Got it. Thank you, Daniel! :)
Yu
Quoted 1 line
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx