Skip to contentSkip to conversation

Add enlightenments for vGPU

PATCH In review Yu Zhang 4 participants 14 messages Feb 10, 2015 – Mar 11, 2015

Subsystems (3)
lore
Share & tools
lore

Yu Zhang

author
#0 lore raw

v4 cover Add enlightenments for vGPU

This patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, will be sent out in other patches.

The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory without hypervisor's intervention, e.g. filling textures or queuing commands. However with the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses without complexity of translation.

Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (with some faulting mechanism).

The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource, XenGT also partitions them among VMs.

Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.

Yu Zhang (8):
drm/i915: Introduce a PV INFO page structure for Intel GVT-g. drm/i915: Adds graphic address space ballooning logic drm/i915: Partition the fence registers for vGPU in i915 driver drm/i915: Disable framebuffer compression for i915 driver in VM drm/i915: Add the display switch logic for vGPU in i915 driver drm/i915: Disable power management for i915 driver in VM drm/i915: Create vGPU specific MMIO operations to reduce traps drm/i915: Support alias ppgtt in VM if ppgtt is enabled

 Documentation/DocBook/drm.tmpl      |   5 +
 drivers/gpu/drm/i915/Makefile       |   3 +
 drivers/gpu/drm/i915/i915_dma.c     |   8 ++
 drivers/gpu/drm/i915/i915_drv.h     |  10 ++
 drivers/gpu/drm/i915/i915_gem.c     |   5 +
 drivers/gpu/drm/i915/i915_gem_gtt.c |  33 ++++-
 drivers/gpu/drm/i915/i915_vgpu.c    | 264 ++++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_vgpu.h    |  91 +++++++++++++
 drivers/gpu/drm/i915/intel_fbc.c    |   4 +
 drivers/gpu/drm/i915/intel_pm.c     |   4 +
 drivers/gpu/drm/i915/intel_uncore.c |  34 +++++
 11 files changed, 458 insertions(+), 3 deletions(-)
 create mode 100644 drivers/gpu/drm/i915/i915_vgpu.c
 create mode 100644 drivers/gpu/drm/i915/i915_vgpu.h
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#1 lore raw

PATCH v4 1/8 drm/i915: Introduce a PV INFO page structure for Intel GVT-g.

Introduce a PV INFO structure, to facilitate the Intel GVT-g technology, which is a GPU virtualization solution with mediated pass-through. This page contains the shared information between i915 driver and the host emulator. For now, this structure utilizes an area of 4K bytes on HSW GPU's unused MMIO space. Future hardware will have the reserved window architecturally defined, and layout of the page will be added in future BSpec.

The i915 driver load routine detects if it is running in a VM by reading the contents of this PV INFO page. Thereafter a flag, vgpu.active is set, and intel_vgpu_active() is used by checking this flag to conclude if GPU is virtualized with Intel GVT-g. By now, intel_vgpu_active() will return true, only when the driver is running as a guest in the Intel GVT-g enhanced environment on HSW platform.

v2:
take Chris' comments:
- call the i915_check_vgpu() in intel_uncore_init()
- sanitize i915_check_vgpu() by adding BUILD_BUG_ON() and debug info
take Daniel's comments:
- put the definition of PV INFO into a new header - i915_vgt_if.h
other changes:
- access mmio regs by readq/readw in i915_check_vgpu()

v3:
take Daniel's comments:
- move the i915/vgt interfaces into a new i915_vgpu.c
- update makefile
- add kerneldoc to functions which are non-static
- add a DOC: section describing some of the high-level design
- update drm docbook
other changes:
- rename i915_vgt_if.h to i915_vgpu.h

v4:
take Tvrtko's comments:
- fix a typo in commit message
- add debug message when vgt version mismatches
- rename low_gmadr/high_gmadr to mappable/non-mappable in PV INFO structure

---
 Documentation/DocBook/drm.tmpl      |  5 +++
 drivers/gpu/drm/i915/Makefile       |  3 ++
 drivers/gpu/drm/i915/i915_drv.h     | 10 +++++
 drivers/gpu/drm/i915/i915_vgpu.c    | 86 +++++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_vgpu.h    | 85 ++++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/intel_uncore.c |  3 ++
 6 files changed, 192 insertions(+)
 create mode 100644 drivers/gpu/drm/i915/i915_vgpu.c
 create mode 100644 drivers/gpu/drm/i915/i915_vgpu.h
Documentation/DocBook/drm.tmpl +5 -0
index 03f1985..249f0c9 100644
@@ -3979,6 +3979,11 @@ int num_ioctls;</synopsis>
 !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_disable_interrupts
 !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_enable_interrupts
       </sect2>
+      <sect2>
+        <title>Intel GVT-g Guest Support(vGPU)</title>
+!Pdrivers/gpu/drm/i915/i915_vgpu.c Intel GVT-g guest support
+!Idrivers/gpu/drm/i915/i915_vgpu.c
+      </sect2>
     </sect1>
     <sect1>
       <title>Display Hardware Handling</title>
drivers/gpu/drm/i915/Makefile +3 -0
index f019225..f025e7f 100644
@@ -83,6 +83,9 @@ i915-y += dvo_ch7017.o \
 	  intel_sdvo.o \
 	  intel_tv.o
 
+# virtual gpu code
+i915-y += i915_vgpu.o
+
 # legacy horrors
 i915-y += i915_dma.o \
 	  i915_ums.o
drivers/gpu/drm/i915/i915_drv.h +10 -0
index c0b8644..9bc0c9e 100644
@@ -1640,6 +1640,10 @@ struct i915_workarounds {
 	u32 count;
 };
 
+struct i915_virtual_gpu {
+	bool active;
+};
+
 struct drm_i915_private {
 	struct drm_device *dev;
 	struct kmem_cache *slab;
@@ -1652,6 +1656,8 @@ struct drm_i915_private {
 
 	struct intel_uncore uncore;
 
+	struct i915_virtual_gpu vgpu;
+
 	struct intel_gmbus gmbus[GMBUS_NUM_PORTS];
 
 
@@ -2583,6 +2589,10 @@ void intel_uncore_forcewake_get(struct drm_i915_private *dev_priv,
 void intel_uncore_forcewake_put(struct drm_i915_private *dev_priv,
 				enum forcewake_domains domains);
 void assert_forcewakes_inactive(struct drm_i915_private *dev_priv);
+static inline bool intel_vgpu_active(struct drm_device *dev)
+{
+	return to_i915(dev)->vgpu.active;
+}
 
 void
 i915_enable_pipestat(struct drm_i915_private *dev_priv, enum pipe pipe,
drivers/gpu/drm/i915/i915_vgpu.c new file +86 -0

86 lines not on this page · raw message

drivers/gpu/drm/i915/i915_vgpu.h new file +85 -0

85 lines not on this page · raw message

drivers/gpu/drm/i915/intel_uncore.c +3 -0
index c47a3ba..d95419d 100644
@@ -23,6 +23,7 @@
 
 #include "i915_drv.h"
 #include "intel_drv.h"
+#include "i915_vgpu.h"
 
 #include <linux/pm_runtime.h>
 
@@ -1082,6 +1083,8 @@ void intel_uncore_init(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
+	i915_check_vgpu(dev);
+
 	intel_uncore_ellc_detect(dev);
 	intel_uncore_fw_domains_init(dev);
 	__intel_uncore_early_sanitize(dev, false);
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
maintainer
#2 lore raw

Re: [PATCH v4 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.

drivers/gpu/drm/i915/i915_vgpu.h line 35

Quoted 5 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +5 -0
+/*
+ * The following structure pages are defined in GEN MMIO space
+ * for virtualization. (One page for now)
+ */
+#define VGT_MAGIC         0x4776544776544776   /* 'vGTvGTvG' */

one of my dusty 32-bit compilers dislikes this,

CC [M] drivers/gpu/drm/i915/i915_vgpu.o
/home/airlied/kernel/drm-2.6/drivers/gpu/drm/i915/i915_vgpu.c: In function ‘i915_check_vgpu’:
/home/airlied/kernel/drm-2.6/drivers/gpu/drm/i915/i915_vgpu.c:73:
warning: integer constant is too large for ‘long’ type

Dave.
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#3 lore raw

PATCH v4 2/8 drm/i915: Adds graphic address space ballooning logic

In reply to Yu Zhang

With Intel GVT-g, the global graphic memory space is partitioned by multiple vGPU instances in different VMs. The ballooning code is called in i915_gem_setup_global_gtt(), utilizing the drm mm allocator APIs to mark the graphic address space which are partitioned out to other vGPUs as reserved. With ballooning, host side does not need to translate a grahpic address from guest view to host view. By now, current implementation only support the static ballooning, but in the future, with more cooperation from guest driver, the same interfaces can be extended to grow/shrink the guest graphic memory dynamically.

v2:
take Chris and Daniel's comments:
- no guard page between different VMs
- use drm_mm_reserve_node() to do the reservation for ballooning, instead of the previous drm_mm_insert_node_in_range_generic()

v3:
take Daniel's comments:
- move ballooning functions into i915_vgpu.c
- add kerneldoc to ballooning functions

v4:
take Tvrtko's comments:
- more accurate comments and commit message

---
 drivers/gpu/drm/i915/i915_gem_gtt.c |  17 +++-
 drivers/gpu/drm/i915/i915_vgpu.c    | 178 ++++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_vgpu.h    |   2 +
 3 files changed, 194 insertions(+), 3 deletions(-)
drivers/gpu/drm/i915/i915_gem_gtt.c +14 -3
index b48b586..057f905 100644
@@ -27,6 +27,7 @@
 #include <drm/drmP.h>
 #include <drm/i915_drm.h>
 #include "i915_drv.h"
+#include "i915_vgpu.h"
 #include "i915_trace.h"
 #include "intel_drv.h"
 
@@ -1756,6 +1757,16 @@ static int i915_gem_setup_global_gtt(struct drm_device *dev,
 
 	/* Subtract the guard page ... */
 	drm_mm_init(&ggtt_vm->mm, start, end - start - PAGE_SIZE);
+
+	dev_priv->gtt.base.start = start;
+	dev_priv->gtt.base.total = end - start;
+
+	if (intel_vgpu_active(dev)) {
+		ret = intel_vgt_balloon(dev);
+		if (ret)
+			return ret;
+	}
+
 	if (!HAS_LLC(dev))
 		dev_priv->gtt.base.mm.color_adjust = i915_gtt_color_adjust;
 
@@ -1775,9 +1786,6 @@ static int i915_gem_setup_global_gtt(struct drm_device *dev,
 		vma->bound |= GLOBAL_BIND;
 	}
 
-	dev_priv->gtt.base.start = start;
-	dev_priv->gtt.base.total = end - start;
-
 	/* Clear any non-preallocated blocks */
 	drm_mm_for_each_hole(entry, &ggtt_vm->mm, hole_start, hole_end) {
 		DRM_DEBUG_KMS("clearing unused GTT space: [%lx, %lx]\n",
@@ -1829,6 +1837,9 @@ void i915_global_gtt_cleanup(struct drm_device *dev)
 	}
 
 	if (drm_mm_initialized(&vm->mm)) {
+		if (intel_vgpu_active(dev))
+			intel_vgt_deballoon();
+
 		drm_mm_takedown(&vm->mm);
 		list_del(&vm->global_link);
 	}
drivers/gpu/drm/i915/i915_vgpu.c +178 -0

181 lines not on this page · raw message

drivers/gpu/drm/i915/i915_vgpu.h +2 -0
index aa72e14..3ed01a7 100644
@@ -81,5 +81,7 @@ struct vgt_if {
 	(VGT_PVINFO_PAGE + (long)&((struct vgt_if *)NULL)->x)
 
 extern void i915_check_vgpu(struct drm_device *dev);
+extern int intel_vgt_balloon(struct drm_device *dev);
+extern void intel_vgt_deballoon(void);
 
 #endif /* _I915_VGPU_H_ */
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#4 lore raw

PATCH v4 3/8 drm/i915: Partition the fence registers for vGPU in i915 driver

In reply to Yu Zhang

With Intel GVT-g, the fence registers are partitioned by multiple vGPU instances in different VMs. Routine i915_gem_load() is modified to reset the num_fence_regs, when the driver detects it's running in a VM. Accesses to the fence registers from vGPU will be trapped and remapped by the host side. And the allocated fence number is provided in PV INFO page structure. By now, the value of fence number is fixed, but in the future we can relax this limitation, to allocate the fence registers dynamically from host side.

---
 drivers/gpu/drm/i915/i915_gem.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +5 -0
index c26d36c..1765989 100644
@@ -29,6 +29,7 @@
 #include <drm/drm_vma_manager.h>
 #include <drm/i915_drm.h>
 #include "i915_drv.h"
+#include "i915_vgpu.h"
 #include "i915_trace.h"
 #include "intel_drv.h"
 #include <linux/oom.h>
@@ -4987,6 +4988,10 @@ i915_gem_load(struct drm_device *dev)
 	else
 		dev_priv->num_fence_regs = 8;
 
+	if (intel_vgpu_active(dev))
+		dev_priv->num_fence_regs =
+				I915_READ(vgtif_reg(avail_rs.fence_num));
+
 	/* Initialize fence registers to zero */
 	INIT_LIST_HEAD(&dev_priv->mm.fence_list);
 	i915_gem_restore_fences(dev);
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#5 lore raw

PATCH v4 4/8 drm/i915: Disable framebuffer compression for i915 driver in VM

In reply to Yu Zhang

Framebuffer compression is disabled when driver detects it's running in a Intel GVT-g enlightened VM, because FBC is not emulated and there is no stolen memory for a vGPU.

v2:
take Chris' comments:
- move the code into intel_update_fbc()

v4:
take Tvrtko's comments:
- rebase the code into intel_fbc_update()

---
 drivers/gpu/drm/i915/intel_fbc.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/gpu/drm/i915/intel_fbc.c +4 -0
index 7341e87..ee65731 100644
@@ -505,6 +505,10 @@ void intel_fbc_update(struct drm_device *dev)
 	if (!HAS_FBC(dev))
 		return;
 
+	/* disable framebuffer compression in vGPU */
+	if (intel_vgpu_active(dev))
+		i915.enable_fbc = 0;
+
 	if (i915.enable_fbc < 0) {
 		if (set_no_fbc_reason(dev_priv, FBC_CHIP_DEFAULT))
 			DRM_DEBUG_KMS("disabled per chip default\n");
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#6 lore raw

PATCH v4 5/8 drm/i915: Add the display switch logic for vGPU in i915 driver

In reply to Yu Zhang

Display switch logic is added to notify the host side that current vGPU have a valid surface to show. It does so by writing the display_ready field in PV INFO page, and then will be handled in the host side. This is useful to avoid trickiness when the VM's framebuffer is being accessed in the middle of VM modesetting, e.g. compositing the framebuffer in the host side.

v2:
- move the notification code outside the 'else' in load sequence
- remove the notification code in intel_crtc_set_config()

v4:
- code rebase, no need to define another dev_priv
- use #define instead of enum for display readiness

---
 drivers/gpu/drm/i915/i915_dma.c  | 8 ++++++++
 drivers/gpu/drm/i915/i915_vgpu.h | 4 ++++
 2 files changed, 12 insertions(+)
drivers/gpu/drm/i915/i915_dma.c +8 -0
index 1a46787..5804aa5 100644
@@ -36,6 +36,7 @@
 #include "intel_drv.h"
 #include <drm/i915_drm.h>
 #include "i915_drv.h"
+#include "i915_vgpu.h"
 #include "i915_trace.h"
 #include <linux/pci.h>
 #include <linux/console.h>
@@ -842,6 +843,13 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags)
 		}
 	}
 
+	/*
+	 * Notify a valid surface after modesetting,
+	 * when running inside a VM.
+	 */
+	if (intel_vgpu_active(dev))
+		I915_WRITE(vgtif_reg(display_ready), VGT_DRV_DISPLAY_READY);
+
 	i915_setup_sysfs(dev);
 
 	if (INTEL_INFO(dev)->num_pipes) {
drivers/gpu/drm/i915/i915_vgpu.h +4 -0
index 3ed01a7..0db9ccf 100644
@@ -80,6 +80,10 @@ struct vgt_if {
 #define vgtif_reg(x) \
 	(VGT_PVINFO_PAGE + (long)&((struct vgt_if *)NULL)->x)
 
+/* vGPU display status to be used by the host side */
+#define VGT_DRV_DISPLAY_NOT_READY 0
+#define VGT_DRV_DISPLAY_READY     1  /* ready for display switch */
+
 extern void i915_check_vgpu(struct drm_device *dev);
 extern int intel_vgt_balloon(struct drm_device *dev);
 extern void intel_vgt_deballoon(void);
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#7 lore raw

PATCH v4 6/8 drm/i915: Disable power management for i915 driver in VM

In reply to Yu Zhang

With Intel GVT-g, GPU power management is controlled by host driver, so there is no need to provide virtualized GPU PM support. In the future it might be useful to gather VM input for freq boost, but now let's disable it simply.

v2:
take Chris' comments:
- do not special case this to gen6+

---
 drivers/gpu/drm/i915/intel_pm.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/gpu/drm/i915/intel_pm.c +4 -0
index 3c64810..177ff97 100644
@@ -5623,6 +5623,10 @@ void intel_enable_gt_powersave(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
+	/* Powersaving is controlled by the host when inside a VM */
+	if (intel_vgpu_active(dev))
+		return;
+
 	if (IS_IRONLAKE_M(dev)) {
 		mutex_lock(&dev->struct_mutex);
 		ironlake_enable_drps(dev);
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#8 lore raw

PATCH v4 7/8 drm/i915: Create vGPU specific MMIO operations to reduce traps

In reply to Yu Zhang

In the virtualized environment, forcewake operations are not necessary for the driver, because mmio accesses will be trapped and emulated by the host side, and real forcewake operations are also done in the host. New mmio access handlers are added to directly call the __raw_i915_read/write, therefore will reduce many traps and increase the overall performance for drivers running in the VM with Intel GVT-g enhancement.

v2:
take Chris' comments:
- register the mmio hooks in intel_uncore_init()
v3:
take Daniel's comments:
- use macros to assign mmio write functions for vGPU
v4:
take Tvrtko's comments:
- also use mmio hooks for read operations

---
 drivers/gpu/drm/i915/intel_uncore.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
drivers/gpu/drm/i915/intel_uncore.c +31 -0
index d95419d..1b8d8ce 100644
@@ -641,6 +641,14 @@ static inline void __force_wake_get(struct drm_i915_private *dev_priv,
 		dev_priv->uncore.funcs.force_wake_get(dev_priv, fw_domains);
 }
 
+#define __vgpu_read(x) \
+static u##x \
+vgpu_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
+	GEN6_READ_HEADER(x); \
+	val = __raw_i915_read##x(dev_priv, reg); \
+	GEN6_READ_FOOTER; \
+}
+
 #define __gen6_read(x) \
 static u##x \
 gen6_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
@@ -704,6 +712,10 @@ gen9_read##x(struct drm_i915_private *dev_priv, off_t reg, bool trace) { \
 	GEN6_READ_FOOTER; \
 }
 
+__vgpu_read(8)
+__vgpu_read(16)
+__vgpu_read(32)
+__vgpu_read(64)
 __gen9_read(8)
 __gen9_read(16)
 __gen9_read(32)
@@ -725,6 +737,7 @@ __gen6_read(64)
 #undef __chv_read
 #undef __vlv_read
 #undef __gen6_read
+#undef __vgpu_read
 #undef GEN6_READ_FOOTER
 #undef GEN6_READ_HEADER
 
@@ -808,6 +821,14 @@ hsw_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace)
 	GEN6_WRITE_FOOTER; \
 }
 
+#define __vgpu_write(x) \
+static void vgpu_write##x(struct drm_i915_private *dev_priv, \
+			  off_t reg, u##x val, bool trace) { \
+	GEN6_WRITE_HEADER; \
+	__raw_i915_write##x(dev_priv, reg, val); \
+	GEN6_WRITE_FOOTER; \
+}
+
 static const u32 gen8_shadowed_regs[] = {
 	FORCEWAKE_MT,
 	GEN6_RPNSWREQ,
@@ -925,12 +946,17 @@ __gen6_write(8)
 __gen6_write(16)
 __gen6_write(32)
 __gen6_write(64)
+__vgpu_write(8)
+__vgpu_write(16)
+__vgpu_write(32)
+__vgpu_write(64)
 
 #undef __gen9_write
 #undef __chv_write
 #undef __gen8_write
 #undef __hsw_write
 #undef __gen6_write
+#undef __vgpu_write
 #undef GEN6_WRITE_FOOTER
 #undef GEN6_WRITE_HEADER
 
@@ -1133,6 +1159,11 @@ void intel_uncore_init(struct drm_device *dev)
 		break;
 	}
 
+	if (intel_vgpu_active(dev)) {
+		ASSIGN_WRITE_MMIO_VFUNCS(vgpu);
+		ASSIGN_READ_MMIO_VFUNCS(vgpu);
+	}
+
 	i915_check_and_clear_faults(dev);
 }
 #undef ASSIGN_WRITE_MMIO_VFUNCS
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#9 lore raw

PATCH v4 8/8 drm/i915: Support alias ppgtt in VM if ppgtt is enabled

In reply to Yu Zhang

The current Intel GVT-g only supports alias ppgtt. And the emulation is done in the host by first trapping PP_DIR_BASE mmio accesses. Updating PP_DIR_BASE by using instructions such as MI_LOAD_REGISTER_IMM are hard to detect and are not supported in current code. Therefore this patch also adds a new callback routine - vgpu_mm_switch() to set the PP_DIR_BASE by mmio writes.

v2:
take Chris' comments:
- move the code into sanitize_enable_ppgtt()
v4:
take Tvrtko's comments:
- fix the parenthesis alignment warning

---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
drivers/gpu/drm/i915/i915_gem_gtt.c +16 -0
index 057f905..e54b2a0 100644
@@ -104,6 +104,9 @@ static int sanitize_enable_ppgtt(struct drm_device *dev, int enable_ppgtt)
 	has_aliasing_ppgtt = INTEL_INFO(dev)->gen >= 6;
 	has_full_ppgtt = INTEL_INFO(dev)->gen >= 7;
 
+	if (intel_vgpu_active(dev))
+		has_full_ppgtt = false; /* emulation is too hard */
+
 	/*
 	 * We don't allow disabling PPGTT for gen9+ as it's a requirement for
 	 * execlists, the sole mechanism available to submit work.
@@ -798,6 +801,16 @@ static int hsw_mm_switch(struct i915_hw_ppgtt *ppgtt,
 	return 0;
 }
 
+static int vgpu_mm_switch(struct i915_hw_ppgtt *ppgtt,
+			  struct intel_engine_cs *ring)
+{
+	struct drm_i915_private *dev_priv = to_i915(ppgtt->base.dev);
+
+	I915_WRITE(RING_PP_DIR_DCLV(ring), PP_DIR_DCLV_2G);
+	I915_WRITE(RING_PP_DIR_BASE(ring), get_pd_offset(ppgtt));
+	return 0;
+}
+
 static int gen7_mm_switch(struct i915_hw_ppgtt *ppgtt,
 			  struct intel_engine_cs *ring)
 {
@@ -1127,6 +1140,9 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	} else
 		BUG();
 
+	if (intel_vgpu_active(dev))
+		ppgtt->switch_mm = vgpu_mm_switch;
+
 	ret = gen6_ppgtt_alloc(ppgtt);
 	if (ret)
 		return ret;
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Reviewed-by on the v4 series
#10 lore raw
In reply to Yu Zhang
Quoted 40 lines · Yu Zhang

This patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, will be sent out in other patches.

The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory without hypervisor's intervention, e.g. filling textures or queuing commands. However with the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses without complexity of translation.

Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (with some faulting mechanism).

The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource, XenGT also partitions them among VMs.

Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.

Quoted 9 lines

Yu Zhang (8):
drm/i915: Introduce a PV INFO page structure for Intel GVT-g. drm/i915: Adds graphic address space ballooning logic drm/i915: Partition the fence registers for vGPU in i915 driver drm/i915: Disable framebuffer compression for i915 driver in VM drm/i915: Add the display switch logic for vGPU in i915 driver drm/i915: Disable power management for i915 driver in VM drm/i915: Create vGPU specific MMIO operations to reduce traps drm/i915: Support alias ppgtt in VM if ppgtt is enabled

All my comments have been addressed (and I especially like the ASCII diagram of the memory space!) so you can put my r-b on all the patches from this series:

Regards,

Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#11 lore raw
Quoted 51 lines · Tvrtko Ursulin
Quoted 40 lines · Yu Zhang

This patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, will be sent out in other patches.

The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory without hypervisor's intervention, e.g. filling textures or queuing commands. However with the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses without complexity of translation.

Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (with some faulting mechanism).

The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource, XenGT also partitions them among VMs.

Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.

Quoted 9 lines

Yu Zhang (8):
drm/i915: Introduce a PV INFO page structure for Intel GVT-g. drm/i915: Adds graphic address space ballooning logic drm/i915: Partition the fence registers for vGPU in i915 driver drm/i915: Disable framebuffer compression for i915 driver in VM drm/i915: Add the display switch logic for vGPU in i915 driver drm/i915: Disable power management for i915 driver in VM drm/i915: Create vGPU specific MMIO operations to reduce traps drm/i915: Support alias ppgtt in VM if ppgtt is enabled

Quoted 5 lines

All my comments have been addressed (and I especially like the ASCII diagram of the memory space!) so you can put my r-b on all the patches from this series:

Thank you very much, Tvrtko. This is great. :-)

Btw, should I resend another version patch series, which add the "Reviewed-by" line, or will Daniel directly merge these patches and add the "Reviewed-by" at that time? I'm willing to take whatever actions necessary, but I'm not familiar with this process. Thanks again. :)

Yu

Quoted 6 lines

Regards,

Tvrtko

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#12 lore raw
In reply to Tvrtko Ursulin
Quoted 51 lines · Tvrtko Ursulin
Quoted 40 lines · Yu Zhang

This patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, will be sent out in other patches.

The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory without hypervisor's intervention, e.g. filling textures or queuing commands. However with the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses without complexity of translation.

Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (with some faulting mechanism).

The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource, XenGT also partitions them among VMs.

Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.

Quoted 9 lines

Yu Zhang (8):
drm/i915: Introduce a PV INFO page structure for Intel GVT-g. drm/i915: Adds graphic address space ballooning logic drm/i915: Partition the fence registers for vGPU in i915 driver drm/i915: Disable framebuffer compression for i915 driver in VM drm/i915: Add the display switch logic for vGPU in i915 driver drm/i915: Disable power management for i915 driver in VM drm/i915: Create vGPU specific MMIO operations to reduce traps drm/i915: Support alias ppgtt in VM if ppgtt is enabled

Quoted 5 lines

All my comments have been addressed (and I especially like the ASCII diagram of the memory space!) so you can put my r-b on all the patches from this series:

All merged, thanks for patches&review.
-Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#13 lore raw
Quoted 52 lines · Daniel Vetter
Quoted 51 lines · Tvrtko Ursulin
Quoted 49 lines · Yu Zhang

This patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, will be sent out in other patches.

The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory without hypervisor's intervention, e.g. filling textures or queuing commands. However with the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses without complexity of translation.

Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (with some faulting mechanism).

The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource, XenGT also partitions them among VMs.

Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.

Yu Zhang (8):
drm/i915: Introduce a PV INFO page structure for Intel GVT-g. drm/i915: Adds graphic address space ballooning logic drm/i915: Partition the fence registers for vGPU in i915 driver drm/i915: Disable framebuffer compression for i915 driver in VM drm/i915: Add the display switch logic for vGPU in i915 driver drm/i915: Disable power management for i915 driver in VM drm/i915: Create vGPU specific MMIO operations to reduce traps drm/i915: Support alias ppgtt in VM if ppgtt is enabled

Quoted 8 lines
Quoted 5 lines

All my comments have been addressed (and I especially like the ASCII diagram of the memory space!) so you can put my r-b on all the patches from this series:

All merged, thanks for patches&review.
-Daniel

Got it. Thank you, Daniel! :)

Yu

Quoted 1 line

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx